Empowering Career Decision-Making: A Practical Strategy for Using Occupational Information.

Complex changes in the labor market continue to transform the way employment is conducted in a pervasive ripple effect that began in urban centers of business and moved relentlessly toward making an impact on rural enterprises. Efforts by businesses to become more efficient and the burgeoning growth of a contingent workforce have diversified the marketplace, driving the rapid development of new mechanisms for conducting business. The rate and extent of these labor market changes increase the need for comprehensive occupational information.

Placement specialists and counselors whose case management responsibilities include placement need to acquire detailed information in ways that conserve time, energy, and agency resources. The need to evaluate an individual's ability to perform the essential functions of a job, as defined by the Americans with Disabilities Act, increases the need for reliable occupational information. In addition, persons with disabilities commonly lack two valuable resources for career decision-making: (a) work experience and (b) networks within the employment community (Johnson, 1993). They depend on rehabilitation professionals to mitigate these employment experience and networking deficits with reliable occupational information.

The Dictionary of Occupational Titles (DOT, 1977) has long provided rehabilitation professionals with the best available information of its kind to use in critical rehabilitation activities (e.g., determining eligibility, identifying transferable skills, and job-matching) (Cain & Treiman, 1981; Miller, Treiman, Cain & Roos, 1980; Spenner, 1990). Uniquely comprehensive, the DOT can be used to rapidly identify aptitude requirements, physical demands, education and training requirements, as well as other occupational prerequisites. While this resource contains more than 100 different characteristics for each of 12,000 jobs, it does not contain occupational information that is perfectly reliable (e.g., Cain & Treiman, 1981; Geyer, Hice, Hawk, Brannon, & Boese, 1989; Spenner, 1990). Nevertheless, the increased need for occupational information makes the DOT too valuable a resource to ignore or underutilize. A strategy is needed to maximize effective and efficient use of the document to support confident selection of employment options.

The user-friendly strategy presented in this paper will enable rehabilitation professionals to construct error bands around estimated DOT occupational ratings in order to identify the level of confidence placed on a particular occupational prerequisite. The RE-SEM strategy discussed here uses reliability estimates (RE) and Standard Error of Measurement (SEM) coefficients to facilitate career decisions when a match is sought between occupational requirements and the abilities and interests of job-seekers.

The Process of Generating DOT Ratings

DOT ratings are generated by a two-step process. Initially, information is gathered directly by full-time, highly trained job analysts who observe and interview job incumbents and supervisors working in a specific occupation (an occupation is a class of jobs with common requirements). For example, specific jobs within the lawyer occupation might include corporate lawyer, divorce lawyer, or criminal prosecutor. Observation and interview data then are documented in a Job Analysis Report (JAR) and used by other job analysts to prepare summaries and ratings for the DOT document. Quality control checks are conducted at various points in this process. Although these highly skilled job analysts have experience and follow very specific procedures, human error can occur regardless of the rater's training and experience level.

Inaccurate Information in the DOT

Error in the DOT's occupational information has been widely documented (Cain & Green, 1983; Fine, 1955a, 1955b; Fine & Heinz, 1958; Geyer, Hice, Hawk, Boese, & Brannon, 1989: Newman & Fine, 1957; Trattner, Fine, & Kubis, 1955; Webb, Shavelson, Shea, & Morello, 1981). Studies show that rater (human) error is the primary source of error in the DOT (Cain & Green, 1983; Webb, Shavelson, Shea, & Morello, 1981).

Although human error occurs in many forms, some evidence suggests that errors may result at least in part from job analysts' relative inexperience with rating the occupational characteristics of service occupations compared to manufacturing occupations. The DOT provides more information about manufacturing occupations relative to service occupations, thus job analysts may not be as experienced with rating service sector occupations (Miller, Treiman, Cain & Roos, 1980). This lack of experience may explain why a study by Cain and Green (1983) showed that analysts' ratings of DOT occupational characteristics were more reliable for manufacturing sector occupations than for service occupations. Other reasons for errors might include lack of detail in written job descriptions (contained in the JAR) prepared by job analysts (Cain & Green, 1983), or perhaps fatigue, distraction, recording errors, miscommunication during the interviews with job incumbents, misinterpretation of occupational requirements, and more (Geyer & Hunter, 1992).

Estimating the reliability of DOT occupational characteristics involves measuring the degree to which DOT ratings are free of errors created by individual raters. The most comprehensive and current estimates of the reliability of specific occupational characteristics in the DOT have been reported by Geyer, et al. (1989). They have published comprehensive reliability estimates for nearly all occupational characteristics in the DOT. (Other studies cited earlier have been limited in scope or have used nonstandard, outmoded United States Employment Service operating procedures). Their reliability estimates are presented and explained below to facilitate further dissemination of the findings and because subsequent illustrations are derived from their data.

The Geyer, et al. (1989) study was designed to assess the reliability of ratings for each occupational characteristic in the DOT. To assess reliability, the study used procedures and personnel that duplicated as nearly as possible those used to generate the original DOT ratings. Ratings of occupational characteristics from four job analysts were used to compute Cronbach's coefficient alpha reliability estimates (REs). The Spearman-Brown Formula was then applied to the estimates to derive REs to be expected from ratings made by a single job analyst. The derived REs appear in Table 1. The formulas for REs and Spearman-Brown are reported in Anatasi (1968). Further details on the study's research methodology are available in Geyer, et al. (1989).

Table 1
Reliability and SEM Estimates for DOT Occupational Characteristics

              Item                   Scoring     Mean    Std Dev
Worker Functions
  People                             0 to 8      5.91      2.70
  Data                               0 to 6      3.64      2.14
  Things                             0 to 7      4.09      2.44

General Educational Development
(GED)
  Reasoning                         1 to 6(*)    3.14      1.33
  Language                          1 to 6(*)    2.74      1.38
  Mathematical                      1 to 6(*)    2.41      1.23

Specific vocational preparation      1 to 9      4.95      2.73
Aptitudes
  Verbal                            1 to 5(*)    3.16      1.05
  General learning ability          1 to 5(*)    3.08      1.08
  Color discrimination              1 to 5(*)    4.39       .96
  Spatial                           1 to 5(*)    3.50      1.07
  Numerical                         1 to 5(*)    3.60       .91
  Form perception                   1 to 5(*)    3.46      1.02
  Clerical perception               1 to 5(*)    3.64       .92
  Finger dexterity                  1 to 5(*)    3.64       .93
  Motor coordination                1 to 5(*)    3.40       .91
  Eye-hand-foot coordination        1 to 5(*)    4.51       .67
  Manual dexterity                  1 to 5(*)    3.31       .82

Temperaments
  Repetitive                         0 to 1       .38       .49
  People                             0 to 1       .38       .49
  Judgments                          0 to 1       .41       .50
  Tolerances                         0 to 1       .29       .46
  Under specific instructions        0 to 1       .34       .48
  Directing                          0 to 1       .18       .39
  Variety                            0 to 1       .42       .50

Interests
  Industrial                         0 to 1       .30       .46
  Mechanical                         0 to 1       .34       .48
  Humanitarian                       0 to 1       .15       .36

Physical Demands
  Sitting                            0 to 1     20.91     28.22
  Standing                           0 to 1     43.36     26.79
  Strength                           1 to 5      2.74      1.13
  Walking                            0 to 1     35.60     21.78

Physical demands (criticality)
  Talking                            0 to 1       .52       .50
  Hearing                            0 to 1       .54       .50
  Near acuity                        0 to 1       .65       .48
  Crouching                          0 to 1       .20       .40
  Far acuity                         0 to 1       .22       .42
  Stooping                           0 to 1       .34       .48
  Depth perception                   0 to 1       .48       .50
  Color vision                       0 to 1       .30       .46
  Fingering                          0 to 1       .61       .49

Physical demands (criticality)
  Accommodation                      0 to 1       .29       .46

Physical demands (frequency)
  Talking                            0 to 4      1.44      1.03
  Climbing                           0 to 4       .36       .53
  Hearing                            0 to 4      1.51       .99
  Feeling                            0 to 4       .40       .69
  Color vision                       0 to 4       .58       .76
  Reaching                           0 to 4      2.20       .78
  Balancing                          0 to 4       .16       .40
  Crouching                          0 to 4       .45       .59
  Field of vision                    0 to 4       .31       .77
  Depth perception                   0 to 4       .99      1.08
  Kneeling                           0 to 4       .39       .58
  Handling                           0 to 4      2.29       .77
  Stooping                           0 to 4       .82       .74
  Near acuity                        0 to 4      1.36      1.01
  Far acuity                         0 to 4       .39       .70
  Tasting/smelling                   0 to 4       .21       .47
  Fingering                          0 to 4      1.31       .84
  Accommodation                      0 to 4       .52       .80

Hazards
  Moving mechanical pans             0 to 1       .22       .42

Education and Training
  Apprenticeship                                 2.55     10.53
  Complexity (experimental scale)     0 to 5     2.30      1.37
  College courses                                 .84      1.72
  General education                              8.52      5.26
  Performance other jobs                        15.05     22.03
  On-the-job training                            4.18      7.85

              Item                     RE        SEM
Worker Functions
  People                               .90       0.85
  Data                                 .81       0.93
  Things                               .81       1.06

General Educational Development
(GED)
  Reasoning                            .92       0.38
  Language                             .90       0.44
  Mathematical                         .83       0.51

Specific vocational preparation        .82       1.16
Aptitudes
  Verbal                               .93       0.28
  General learning ability             .88       0.37
  Color discrimination                 .85       0.37
  Spatial                              .78       0.50
  Numerical                            .75       0.46
  Form perception                      .72       0.54
  Clerical perception                  .70       0.50
  Finger dexterity                     .68       0.53
  Motor coordination                   .65       0.54
  Eye-hand-foot coordination           .57       0.44
  Manual dexterity                     .51       0.57

Temperaments
  Repetitive                           .83       0.20
  People                               .75       0.25
  Judgments                            .72       0.26
  Tolerances                           .54       0.31
  Under specific instructions          .50       0.34
  Directing                            .49       0.28
  Variety                              .41       0.38

Interests
  Industrial                           .81       0.20
  Mechanical                           .80       0.21
  Humanitarian                         .67       0.21

Physical Demands
  Sitting                              .79      12.93
  Standing                             .64      16.07
  Strength                             .63       0.69
  Walking                              .46      16.00

Physical demands (criticality)
  Talking                              .69       0.28
  Hearing                              .53       0.34
  Near acuity                          .43       0.36
  Crouching                            .33       0.33
  Far acuity                           .25       0.36
  Stooping                             .25       0.42
  Depth perception                     .22       0.44
  Color vision                         .21       0.41
  Fingering                            .18       0.44

Physical demands (criticality)
  Accommodation                        .17       0.42

Physical demands (frequency)
  Talking                              .71       0.55
  Climbing                             .68       0.30
  Hearing                              .60       0.63
  Feeling                              .56        46
  Color vision                         .53       0.52
  Reaching                             .51       0.55
  Balancing                            .48       0.29
  Crouching                            .44       0.44
  Field of vision                      .43       0.58
  Depth perception                     .37       0.86
  Kneeling                             .35       0.47
  Handling                             .33       0.63
  Stooping                             .30       0.62
  Near acuity                          .29       0.85
  Far acuity                           .22       0.62
  Tasting/smelling                     .22       0.42
  Fingering                            .18       0.76
  Accommodation                        .14       0.74

Hazards
  Moving mechanical pans               .39       0.33

Education and Training
  Apprenticeship                       .99       1.05
  Complexity (experimental scale)      .94       0.34
  College courses                      .87       0.62
  General education                    .70       2.88
  Performance other jobs               .55      14.78
  On-the-job training                  .48       5.66

(*) Higher scores correspond to lower values; otherwise higher scores correspond to higher values

Reliability estimates (REs) ranged from a low of. 14 to a high of .99 on a scale from zero to one (see Table 1). Occupational characteristics with the highest reliability (i.e., exceeding .8) were Worker Functions, General Educational Development (GED) characteristics, Specific Vocational Preparation (SVP), and certain Aptitudes, Temperaments, Interests, and Education and Training items. Conversely, REs for the group of Physical Demands were some of the lowest in the study (i.e., most were below .25). In fact, the degree of error associated with some physical demands may appear to some users to be too low to support their use in career counseling. Nevertheless, for reasons explained below, each of the ratings may still be useful.

Using Imperfect Ratings

A practical approach to dealing with concerns about reliability is to use Standard Error of Measurement (SEM) coefficients to construct error bands around an estimated DOT occupational rating to determine the range of values for a characteristic which may be considered to be accurate. SEM coefficients measuring the extent of the error have been calculated to compliment the reliability estimates in Table 1. These coefficients were calculated using the standard deviation for ratings of a specific occupation on a specific occupational characteristic and the RE across occupations for the specific occupational characteristic (a formula for SEM is given below).

Simultaneous consideration of SEM coefficients and REs can empower counselors and job placement specialists who are hesitant to rely on DOT ratings with low levels of observed reliability. Exclusive use of REs without consideration of SEM coefficients can inappropriately reduce confidence in DOT ratings and result in underutilization of this comprehensive resource. Therefore, the RE-SEM method can facilitate occupational decision-making.

Application of the RE-SEM Method

The counselor in the following example has a client who is interested in computer programming. The local labor market has a strong demand for computer programmers and the client could qualify for a grant to attend a programming school. In addition, the client demonstrates average intelligence, an attribute associated with DOT ratings such as job complexity and verbal ability. In order to facilitate decision-making, the counselor can initiate the RE-SEM method by looking up the level of job complexity for a computer programmer.

Job complexity has been scaled so that average complexity is 100, with 16% of computer programming jobs above 110 and below 90, and 2% above 120 and below 80. Assume that the counselor finds a value in the DOT for computer programmer of 120, a value considerably higher than the average complexity of 100, where the client would have a 50% competition level. Measurement error may have caused the complexity level to be overestimated. What is the likely range of error? How likely is it that the actual complexity level of computer programmer could be average? Or as low as 110?

Table 1 shows that the reliability of the complexity measure is an extremely high .94. The equation for the standard error of measurement (SEM) is SEM=SDx [square root of 1-r]. SDx equals the standard deviation for the ratings of a specific occupation on a specific occupational characteristic. The reliability estimate across occupations for the specific occupational characteristic equals r (Anatasi, 1968). Applying this SEM formula, the SEM for complexity in our example would then be .34. A common error band used in contemporary science is the 68% error band obtained by adding or subtracting the standard error from the observed value. In this example, the error band would be from 120 - .34 = 119.66 to 120 + .34 = 120.34. Thus, the counselor can be 68% sure that the complexity level for computer programmer is between 119.34 and 120.34. If the counselor goes up or down by two standard errors, he or she would have a 95% error band, ranging between 119.32 and 120.68. With this 95% error band, the probability that the job is average is truly remote. Indeed the probability that the true level could be as low as 110 is one in 10,000. The client would not be competitive for this occupation based on these figures.

However, there are many characteristics where the reliability is far short of .94. Suppose the reliability of the complexity rating were as low as .25. The standard error would be 1.19 and the 68% error band would be from 118.81 to 121.19. The 95% error band would be from 117.62 to 122.38. The chance that the true level could be as low as average (100) is still remote. The client still would have little chance of being competitive as a computer programmer. Thus even with a reliability coefficient as low as .25, the occupational rating would be useful in determining the viability of computer programming as an option for this client. Application of the RE-SEM method contributed valuable information to the decision-making process in this example, providing a sense of confidence in the DOT rating even when the reliability estimate was very low. Indeed, judging from SEM coefficients given in Table 1, they will nearly always confirm use of the single DOT rating.

Utilization of the RE-SEM Method During Case Management

Vocational analysis is a critical phase in the case management process. Having acquired intake interview information and/or vocational evaluation findings, the counselor is ready to think diagnostically and prognostically to generate potential vocational alternatives. Factors in the case, including (1)physical, (2)educational-vocational, (3) psychosocial, (4) economic, and (5) personal vocational choice considerations, may create questions regarding the appropriateness of the client's initial vocational goal (Rubin & Roessler, 1992). The vocational goal, at this point, may result from a lack of awareness regarding more suitable alternatives congruent with these five factors. Hence, the counselor's identification of more appropriate vocational alternatives for the client becomes highly significant.

From the DOT, the counselor can learn about the job function of the client's initial vocational goal. To identify additional occupational possibilities, the counselor can review the broad category of occupations under which the initial vocational goal is subsumed to identify job possibilities with working conditions that will not exacerbate the disability. The detailed list of occupations may contain several potential job possibilities congruent with ability and interest. At this point in the process, a counselor typically will have multiple alternatives. A strategy with accuracy and power is needed to evaluate each of the alternatives and narrow the field of options to a most optimal, second, and third vocational goal as recommended in the literature (Farley & Rubin. 1980; Ruben & Roessler, 1992). At this commonly occurring juncture, the RE-SEM method can make a valuable contribution to the case management process.

Utilization of the quick and easy RE-SEM strategy can lend confidence to the selection and/or prioritization of vocational alternatives. Thus, RE-SEM can make an important contribution to the identification of an appropriate vocational goal. However. it is not intended to be the primary source of information in selecting occupational alternatives, and cannot replace clinical judgment, knowledge of client motivation and work experience, nor the counselor's experience with occupational information and job markets.

Summary

Changes in the labor market create an increased need for comprehensive information to use in the delivery of rehabilitation services. At the same time, diminishing resources have increased the need to base career decision-making on reliable information. The Department of Labor endeavors to meet these needs through an effort begun in 1994 to create on-line access to occupational information. It can be updated rapidly and utilized by anyone with access to the Internet. The emerging American Jobs Bank and corresponding Talent Bank build on the DOT foundation and will be familiar to DOT users who will want to use them in combination with the DOT document. The development of and transition to on-line occupational resources will take time before they are broadly available to rehabilitation personnel. Therefore, the DOT document continues to be the best available information of its kind to use in critical rehabilitation activities.

Each of these changes increase the need for reliable occupational information to use in career decision-making. The practical RE-SEM strategy utilizes a reliability estimate and the Standard Error of Measurement to increase the utility of occupational information in the DOT. RE-SEM provides a quick and easy method for guiding critical rehabilitation activities (e.g., determining eligibility, identifying transferable skills, and job-matching) where occupational information plays a vital role.

References

Anatasi, A. (1968). Psychological Testing (3rd Ed). New York: MacMillan Company.

Cain, P.S., Greene, B.F. (1983). Reliabilities of selected ratings available from the Dictionary of Occupational Titles. Journal of Applied Psychology. 68, 155-165.

Cain, P.S., & Treiman, D.J. (1981). The Dictionary of Occupational Titles as a source of occupational data. American Sociological Review, 46, 253-278.

Farley, R.C., & Rubin, S.E. (1980). Systematic Interviewing Skills. Fayetteville, AR: University of Arkansas Research & Training Center in Vocational Rehabilitation.

Fine, S.A. (Spring, 1955). Functional job analysis. Journal of Personnel Administration and Industrial Relations(1), 1-16.

Fine, S.A. (1955). A structure of worker functions. Personnel and Guidance Journal, 34, 66-72.

Fine, S.A. & Heinz, C.A. (1958). The functional occupational classification structure. Personnel and Guidance Journal, November, 180-192.

Geyer, P.D., Hice, J., Hawk, J., Boese, R., & Brannon, Y. (1989). Reliability of ratings available from the Dictionary of Occupational Titles. Personnel Psychology, 42, 547-560.

Geyer, P., & Hunter, J. (1992). Issues of reliability and validity in ratings of Occupational Characteristics in the Dictionary of Occupational Titles. Unpublished Technical Report.

Johnson, V.A. (1993). Factors impacting the job retention and advancement of workers who are deaf. The Volta Review, 95(4), 341-354.

Miller, A.R., Treiman, D.J., Cain, P.S., & Rots. P.A. (1980). Work, jobs, and occupations: A critical review of the Dictionary of Occupational Titles. Washington, D.C.: National Academy Press.

Newman, J., & Fine, S.A. (1957). Validity of job descriptions for physical requirements and work condition information. Personnel Psychology, 10, 181-189.

Ruben, S. E. & Roessler, R. (1992). Case Management and Rehabilitation Counseling. (4th Ed). Austin: TX, PRO-ED.

Spenner, K.I. (1990). Skill: Meanings, methods, and measures. Work & Occupations, 17(4), 399-421.

Trattner, M.H., Fine, S.A., & Kubis, J.F. (1955). A comparison of worker requirement ratings made by reading job descriptions and by direct job observation. Personnel Psychology, 8, 183-194.

U. S. Department of Labor. (1977). Dictionary of Occupational Titles (4th ed.). Washington, D. C.: U. S. Government Printing Office.

Webb, N.M., Shavelson, R.J., Shea, J., & Morello, E. (1981). Generalizability of general education development ratings of jobs in the United States. Journal of Applied Psychology, 66(2), 186-192.

Paul D. Geyer

Virginia Anne Johnson

University of Arkansas

John E. Hunter

Michigan State University

Paul D. Geyer, Rehabilitation Research Training Center, 4601 West Markham, University of Arkansas, Little Rock, AR 72205. Email: [e-mail]