Validation of a Work Capacity Evaluation for Individuals with Psychiatric Disorders.

Vocational disability is frequently associated with psychiatric disorders, and comprises an important element in the diagnostic criteria for many mental disorders (Diagnostic and Statistical Manual of Mental Disorders, 4th ed., p. xxi; 1994). Anthony and Blanch (1987) found that less than 15% of individuals with serious mental illness receiving community treatment were engaged in gainful employment. A review of longitudinal studies revealed rates of gainful employment ranging from 6%, to 41% for individuals with schizophrenia (McGlashan, 1988), with most rates between 10% and 20%. It has been estimated that between 1.7 to 2.4 million Americans have secondary disabilities accompanying severe psychiatric impairments (Goldman, Gattozzi, & Taube, 1981). Moreover, individuals with severe mental disorders are notorious for having the worst employment outcomes of the various disability groups served by state and federal rehabilitation programs (Marshak, Bostich, & Turton, 1990).

Despite the broad scope of psychiatric disability, there have been few instruments validated to measure the capacity of individuals with psychiatric disorders to perform entry level work (Bell & Lysaker, 1995; Bell, Milstein, & Lysaker, 1993; Goldman, Skodol, & Lave, 1992; Holstein & Harding, 1992; Institute of Medicine, 1991; Strauss & Carpenter, 1974). Nor has there been a systematic examination of the specific nature of work disability in individuals with serious mental disorders (Lysaker et al., 1993). Studies of the work capacity of psychiatric patients have relied mainly on global ratings of work, post-hospital employment (Cheadle & Morgan, 1972; Di Stefano & Pryer, 1970; Griffiths, 1974), or psychiatrists' ratings of clinical reports (Pincus & Kennedy, 1991). Such ratings may lack sensitivity to the diverse character of work capacity present in patients with various mental disorders, symptom severity, work histories, or work disincentives.

The lack of a standardized (i.e., normalized, reliable, and valid) and operationalized method for measuring the ability of people with mental impairments to perform entry level work has hampered professionals and public agencies in making diagnoses, conducting functional assessments, determining disability, and designing effective rehabilitation programs. In 1984, the Social Security Administration revised its method of determining psychiatric disability, using a two step criteria: (1) psychiatric impairment as evaluated through diagnosis by DSM-III (1980) and (2) functional disabilities in activities of daily living, work and stress tolerance, and social functioning. The question arose whether these most recent standards and procedures were valid measures of determining disability in persons with psychiatric disorders. This research was designed to answer that question.

A range of work-related measures, taken during actual or simulated work sessions, has been considered the preferred mode for obtaining an objective evaluation of work capacity (Anthony & Jansen, 1984; Bond, 1992). These measures include productivity level, quality of output, work tolerance, responses to work place stressors, promptness, attendance, and the ability to solve work-related and interpersonal problems. The current investigators developed an objective and operationalized protocol for the functional evaluation of entry level work capacity for individuals with psychiatric impairments (Massel et al., 1990). Termed the Work Capacity Evaluation (WCE), this instrument was implemented in a naturalistic work setting and included direct observations of relevant job skills and deficits. The WCE was developed as a central component of a study addressing the relationship between psychiatric symptoms and the capacity to perform entry level work. In that study the tasks on which participants worked were designed to coincide with Social Security Administration's (SSA) description of "basic work activities" for jobs that are available in the community. These "basic work activities" determine the threshold below which an individual can be considered as having a disability. The current manuscript describes the validation of the tasks selected, as a first step toward the standardization of the WCE protocol.

For the tasks to be valid indicators of entry level work, they had to be considered representative of work available in the community. Also, persons without psychiatric impairments and disabilities should perform better on these jobs than persons with these conditions. Similarly, those with psychiatric disabilities who had been adjudicated as unable to work by the SSA or the Department of Veteran's Affairs (VA) should perform more poorly than those with psychiatric disorders who had not been so adjudicated.

The validity of the tasks in the WCE was established in two studies. The first study determined the social validity of the WCE tasks, viewed by employers and rehabilitation experts as representative of entry level work. A second study determined the construct validity of the WCE tasks by comparing the work performance of patients with psychiatric disorders, both adjudicated as disabled by the SSA and not so adjudicated, and individuals without psychiatric impairments.

Social Validity of the WCE Tasks

Social validation refers to the value and importance to society (i.e., consumers, their relatives, professionals) of treatment goals, assessment procedures, and treatment effects. Therefore, social validity is an important consideration in establishing the usefulness and acceptability of assessment and treatment procedures (Kazdin, 1982; Minkin et al., 1976; Wolf, 1978). Social validity, in the context of the current study, refers to the relevance of the WCE protocol (i.e., work tasks) to "real world" entry level work. The protocol must be representative of the spectrum of work activities inherent in entry-level jobs in the community. In conducting this type of validation, experts were consulted for their opinions at every stage of the development and refinement of the work tasks.

Initially, the directors of six vocational rehabilitation programs in the Los Angeles area were consulted regarding the basic skills that should be targeted for testing capacity to perform entry-level work. The subsequent stage involved convening a panel of rehabilitation and vocational counselors (N=8) to plan and design the work tasks which would incorporate those basic skills previously identified. All panel members had experience in vocational counseling or supervising individuals with mental impairments in rehabilitation settings. Based on their recommendations, eight sample tasks were developed. Descriptions of the tasks are provided in Table 1. These tasks measured an array of entry-level work skills and activities, such as fine and large motor dexterity, sequencing, neatness, accuracy, clerical skills, and the ability to operate uncomplicated machinery.

Table 1. Description of work tasks in the initial spectrum "entry-level work activities" for the Work Capacity Evaluation

Task                              Description

Toilet tank assembly              Assembling the flush mechanism in
                                  toilet bowls
Electronic circuit assembly       Assembling an electronic circuit
                                  using soldering
Filing index cards                Sorting and filing cards according
                                  to alphabetical order as in a
                                  clerical task
Towel folding and packaging       Folding, packing, and heat-sealing
                                  towels for use on airplanes
Sorting nuts and bolts            Using a calibrator to sort
                                  assorted sizes of nuts and bolts
Packaging funnels                 Placing different size funnels
                                  into each other, then heat-sealing
                                  them in plastic bags
Assembling a rod                  Attaching various nuts, bolts,
                                  springs, and assorted other parts
                                  onto a rod
Sorting and assembling nuts       Sorting nuts and bolts and then
and bolts                         assembling them to fit a
                                  pre-planned model

The next stage in task development/validation was for each task to be cross-validated by other vocational and rehabilitation counselors (N=26), all of whom worked with persons with mental impairments. The raters used a seven-point scale (i.e., Very relevant = 7; Somewhat relevant = 4; Not relevant = 1). The 26 counselors were instructed to rate the vocational relevance of the tasks, based on a three-item questionnaire:

   1. How relevant to entry level work available in the community is this
   task?

   2. How would performance on this task be relevant to readiness to engage in
   entry level work?

   3. How relevant is this task for the Work Capacity Evaluation work sample?

Tasks were ranked on these items according to the endorsements of these experts. The four tasks ranked the highest were selected for the WCE. The selected tasks, in rank order, were: Filing index cards, assembling toilet tank flush mechanisms, folding and packaging hand towels, and circuit board assembly.

The last stage in the social validation of the work tasks arose from the realization that the expertise of those involved in the process of task development and validation was limited to working with individuals with mental impairments. However, the goal for the tasks was to create a test that would be relevant to entry-level work available in the community at large, not just for persons with mental impairments. Ratings of the relevance of the tasks to entry-level jobs in the community were then solicited from supervisors of unskilled laborers working in ordinary work settings (N-29). These individuals were in a position to know if the work tasks were representative of entry-level work available in the community.

These community-based work supervisors were asked the same questions as the vocational and rehabilitation counselors regarding the eight tasks selected for use in the WCE. On the critical questions of how relevant the tasks are to entry-level work available in the community (question 1), and how relevant the tasks are to readiness for entry-level jobs (question 2), both groups agreed that the filing index cards task was highly relevant, while the electronic circuit assembly task was somewhat relevant. The remaining two tasks (i.e., towel folding and packaging and toilet tank assembly) were rated between mostly and highly relevant. These results are depicted in Figure 1 Ratings for each task on both question 1 and 2 were almost identical. Therefore, the two scores were combined. When ratings of the counselors were compared to ratings of the work supervisors, no significant differences were found for any of the four tasks with the highest ratings. Together, these results indicated that all four tasks were relevant for an evaluation of entry-level work.

[Figure 1 ILLUSTRATION OMITTED]

Construct Validity of the WCE Tasks

To establish the construct validity of an assessment protocol, it is essential to determine that the instrument reflects relevant characteristics of the concept or purpose that the instrument is designed to assess (Campbell, 1960). Thus, a measurement instrument with adequate construct validity yields scores that allow an adequate estimate of its functions. In the context of the construct validity of the WCE tasks, the scores obtained by different groups with disabilities and disorders should reflect their different work capacity status. Consequently, control individuals without a psychiatric disorder and with a current work history would be expected to perform significantly better than those with psychiatric disorders, especially since most of the with psychiatric disorders reported meager rates of employment. Among individuals with psychiatric disorders, those adjudicated as vocationally disabled would be expected to perform more poorly than those not so adjudicated. Furthermore, individuals with more severe mental disorders (e.g., those with schizophrenia and bipolar disorder) would be expected to perform more poorly than those with less severe disorders (e.g., anxiety or depressive disorders).

To determine the construct validity of the WCE tasks, the performance of 366 patients with psychiatric disorders was examined. These patients were recruited broadly from mental health centers, outpatient clinics, a VA hospital, and residential care homes. They were diagnosed using DSM-III (1980) criteria and the Expanded Present State Examination, a structured diagnostic interview. When the project was underway, the DSM III was the prevailing mode of diagnosis. Their primary diagnoses were schizophrenia, depression, bipolar disorder, anxiety disorder, substance abuse or dependence, and personality disorder. The study also included 76 individuals as control participants with no psychiatric disorder, who were carefully screened using the same structured diagnostic interviews. The control participants were employed at least part-time during the six months before entering the project. Control participants worked in blue collar, entry-level jobs such as assembly-line factory work, unskilled laborers in the construction and landscaping trades, and basic electronic work that involved tasks similar to those in the work capacity evaluation.

The work evaluations were conducted in a factory setting that housed a sheltered-workshop on the grounds of the West Los Angeles Veterans Administration Medical Center. The work environment was designed to simulate an actual job milieu as much as possible. Participants reported each morning to a supervisor and were assigned to a work station. They were expected to work seven hours per day, including coffee and lunch breaks, for three days. A subset of individuals participated in a 15-day WCE to determine the optimal duration for assessing the work capacity of individuals with mental illnesses. During the WCE period, each participant was instructed to work on all four tasks, following a standard, rotating protocol.

Productivity on the work tasks was evaluated through a Work Performance Index (WPI), which was the total number of units produced during each work session minus the number of defective units per amount of time worked. Scores on each task were standardized to the performance of the 76 nonpsychiatric control participants in the study, who completed all six sessions over the three-day WCE. Eighty-eight nonpsychiatric control participants were initially included; however, 12 dropped out before completing the evaluation. Although the reasons for dropping out were not formally assessed, these participants were debriefed upon departing the study. Six of the 12 departed because of obtaining employment, four attributed dropping out because of boredom, and the remaining two provided no reasons. Each participant's WPI total score was the cumulative average of his/her performance over all work sessions.

The data were analyzed in a simple one-way analysis of variance of the five participant groups according to their psychiatric condition and disability status (psychotic-disabled, psychotic-nondisabled, nonpsychotic disabled, nonpsychotic-nondisabled, nonpsychiatric controls). The two groups with psychotic disorders were defined by the DSM III (1980) diagnoses of schizophrenia, bipolar disorder, and major depression with psychotic features. The two disability groups were defined by participants' receipt of Social Security or Veterans Affairs disability pensions (e.g., SSI, SSDI, 50% or greater Service Connected or Non-Service Connected adjudicated disability). Pairwise comparisons were evaluated with simple follow-up t-tests, performed both with and without Bonferroni adjustment (for 10 contrasts). All of the unadjusted pairwise contrasts that were statistically significant were so beyond the p = .0001 level, and thus all remained significant at p = .001 or better even after adjustment for multiple comparison.

The overall analysis of variance revealed significant group differences (F = 18.12, df= 4,437, p = .0001). The nonpsychiatric control participants and those participants with a nonpsychotic-nondisabled psychiatric condition demonstrated the best WPI scores, and did not differ significantly from each other (t = 1.69, df = 437, unadjusted p = .10, adjusted p = .92). Both of these groups significantly outperformed each of the other three groups (all unadjusted p-values [is less than] .0001; all adjusted p [is less than] .001). As shown in Table 2, none of the three pairwise contrasts among the participants with a psychotic disorder, whether with a disability or not, and those participants with a nonpsychotic-disabled condition, were statistically significant. Thus, adjudicated disability status was not associated with significantly poorer WPI scores among individuals with psychotic conditions. Individuals with a psychotic disorder had generally low WPI scores (averaging a little more than one standard deviation below the "normal" mean), whether they were adjudicated disabled or not. However, among those participants with a nonpsychotic psychiatric disorder, disability status was associated with significant WPI performance impairment relative both to patients with a nondisabled-nonpsychotic psychiatric condition (t = 4.26, df = 437,p = .0001) or control nonpsychiatric participants (t = 5.22, df = 437, p = .0001).

Table 2. Work Performance Index in five participant groups with different conditions. Groups joined with the same vertical bar did not differ significantly (see text)

Group                            N    Mean      SD      Pairwise
                                                      Significance

Psychotic-Disabled              87    -1.36    1.16
Psychotic-Nondisabled           83    -1.03    1.07
Nonpsychotic-Disabled           59    -1.16    1.40
Nonpsychotic-Nondisabled       137    -0.39    1.16
Nonpsychiatric Participants     76    -0.11    1.01

Since the WPI was normalized against the performance of the nonpsychiatric control participants, rating close to 0.0 reflected higher task performance and fewer errors. As shown in Table 3, individuals without psychiatric disorders performed significantly better on the WPI than those with psychiatric disorders. As predicted, participants who had psychotic disorders were less productive than their counterparts without psychotic disorders (F = 24.96, p [is less than] .0001). It was of interest that individuals with substance abuse disorders (who were detoxified and not intoxicated at the time of their participation in the WCE) performed no differently than participants who did not have a mental disorder. Participants who had been adjudicated as disabled by the Social Security or the Department of Veterans Affairs performed significantly more poorly than those without adjudicated disability (F = 18.77, p [is less than] .0001).

Table 3. Work Performance Index ratings of participants with different diagnoses, and disability status. The closer the Index comes to zero, the better the work performance of the subjects in a particular diagnostic group,

Participant Population            Number      Work Performance Index

Nonpsychiatric Participants          76               -0.11
Schizophrenia                       107               -1.30
Bipolar                              49               -1.37
Depression                          125                -.64
Substance Abuse                      53                -.23
Disabled                            146               -1.28
Nondisabled                         220                -.63
Disabled-Psychotic                   87               -1.36
Disabled-Nonpsychotic                59               -1.16
Nondisabled-Psychotic                83               -1.03
Nondisabled-Nonpsychotic            137                -.39

Duration for Valid Work Capacity Evaluation

One of the questions faced by vocational evaluators who are called upon to conduct functional work capacity assessments on individuals with disabilities is how lengthy an assessment needs to be in order to generate valid results. In the current study, we designed WCE's from three days to three weeks in an attempt to answer that question for the assessment of individuals with mental disabilities. As shown in Figure 2, the Work Performance Index stabilized very quickly, with 78% of the variance in this index accounted for by the end of the first day (initial two sessions) of the WCE. By the end of the second day, 94% of the variance could be explained, suggesting that work capacity evaluations of persons with a wide variety of mental disorders can be accomplished in a very short time period -- if the evaluation uses empirically validated methodology.

[Figure 2 ILLUSTRATION OMITTED]

Moreover, we found that a single task in the WCE, the index card sorting, correlated at a statistically significant level with all four tasks. A review of the 127 most important clerical and kindred occupations, as described by Maze (1991) in her Enhanced Guide for Occupational Exploration, revealed that skills of the type required to perform the index card sorting task are integral to 63% of the occupations. The list of relevant occupations includes Inventory Clerk, Job Tracer, Administrative Clerk, Classifier, Billing Clerk, Order Caller, and Invoice Control Clerk. In summary, the WCE is a highly cost-effective method of determining psychiatric disability, representative of entry level jobs, and could be used as a supplement to the standard disability determination procedures of the Social Security Administration.

Discussion

The results provide evidence of adequate social and construct validity of the WCE tasks. These findings are in accord with our predictions and show acceptable discriminant validity and sensitivity of the WCE tasks for detecting differences among diagnostic disability groups. This represents a step towards the standardization of the WCE, since the work tasks form the foundation of the WCE protocol. In contrast, previous instruments developed to measure work functioning have lacked the robust validity of the WCE and relied on global, nonoperationalized subjective ratings of work performance that are not easily replicable in different settings, with different raters or participants (Black, 1988; Bond & Friedmeyer, 1987; Cheadle, Cushing, Drew, & Morgan, 1967; Cheadle & Morgan, 1972; Griffiths, 1973). Moreover, as highlighted in the results, the WCE can be efficiently carried out in less than one day using a single task from the work battery (i.e., index card sorting task).

A major advantage of the objective, standardized, and operationalized WCE approach is the potential utility of assessment scores on individuals for comparability in a wide variety of settings -- both for rating rehabilitation readiness and rehabilitation outcomes. For example, the work capacity of various disability groups -- physical and psychiatric -- can be compared with one another as they enter or complete vocational rehabilitation services. Priorities for determining disability by the Social Security Administration can be extracted from these standardized and objective ratings. In fact, this has been a policy consequence of the current research in terms of disability pensions for individuals with substance abuse disorders. This study found that the work capacity of detoxified individuals from alcohol or drug addictions was no different from that of nonpsychiatric control participants. After the final report of the study had been made to the Social Security Administration, it was decided that individuals with alcohol or drug addictions (without concurrent comorbidity from other psychiatric or medical disorders) would no longer be eligible for psychiatric disability payments. Congress passed and the President signed into law an amendment to the Social Security Act that discontinued persons with addictions from the disability rolls as of January 1, 1997.

Moreover, an individual's score on the WPI and on the other measures of the WCE battery can be compared to normative scores (these are currently being derived), to determine the relative work capacity of that individual. Another important step in this development is that the WCE is sufficiently portable and efficient for use in a variety of settings. The materials used for the tasks (see Table 1) are inexpensive, easily transportable, and require no more than a five-foot-wide work station for their operation. In fact, we found that the card sorting task alone discriminated among the participant populations tested, as well as the entire WCE, Thus, three of the four tasks consuming time and resources may be eliminated from the WCE battery without diminishing the sensitivity and specificity of the protocol.

There has been a controversy in the literature regarding the role of psychiatric symptoms and diagnosis in employability and work capacity. Anthony and his colleagues have argued that the dimensions of psychopathology and work are not substantially related to employment (Anthony & Jansen, 1984; Dion & Anthony, 1987), although many of the studies reviewed by this group were conducted in the pre-DSM III (1980) era when diagnosis and symptom assessment were unreliable (Massel et al., 1990). Moreover, there have been extraordinarily few empirical studies of psychopathology, diagnosis and work capacity when all three of these dimensions were measured concurrently -- as was the case in the current research. Because symptoms improve with treatment over time, one would not necessarily expect a very high correlation between symptoms measured at time 1 and vocational functioning measured at time 2. In the current study, the strong relationships found between simultaneously measured symptom-based diagnosis and work capacity are congruent with more recent research using sensitive and reliable measures of psychopathology (Breier et al., 1993; Hoffmann & Kupper, 1997; Solinski, Jackson & Bell, 1992; Westermeyer & Harrow, 1987). In a subsequent publication, our research team shall report on the predictive validity of the severity of psychopathology -- as measured by scores on the Expanded Brief Psychiatric Rating Scale (Ventura et al., 1993) -- on measures of employability and sustained employment, using data from the current study.

In the WCE, participants' performance was examined over either three or fifteen days of assessment. Results indicated that a brief assessment period of even one-half to one day might be sufficient to establish the instrument's validity in measuring the capacity to work of psychiatric patients. Most of the total variance in participants' performance on the WCE battery was accounted for in this span of time. This makes the WCE a highly cost-effective tool for measuring psychiatric disability. A major goal of the WCE project was to establish an objective, cost-effective and standardized method of assessing the work capacity of persons with psychiatric impairments. The social and construct validation of the work tasks of the WCE protocol is a major step towards the fulfillment of that goal.

Acknowledgement

This research was co-sponsored by UCLA Department of Psychiatry and the West Los Angeles Veterans Affairs Medical Center. This research was supported by research grant number 10-P-98193-9 from the Social Security Administration and NIMH Clinical Research Center grant MH 30911. The findings and conclusions do not reflect the official policy of the Department of Veterans Affairs, University of California, or the Social Security Administration.

The authors wish to acknowledge the technical assistance of Mary Jane Robertson, Carol Giannini, and Sun Hwang in the data collection and analyses.

References

American Psychiatric Association. (1980). Diagnostic and statistical manual of mental disorders (3rd ed.). Washington, DC: Author.

American Psychiatric Association. (1994). Diagnostic and statistical manual of mental disorders (4th ed.). Washington, DC: Author.

Anthony, W. A., & Blanch, A. (1987). Supported employment for persons who are psychiatrically disabled: A historical and conceptual perspective. Psychosocial Rehabilitation Journal. 11, 5-23.

Anthony, W. A., & Jansen, M.A. (1984). Predicting the vocational capacity of the chronically mentally ill: Research and policy implications. American Psychologist, 39, 537-544.

Bell, M. D. & Lysaker, P. H. (1995). Psychiatric symptoms and work performance among persons with severe mental illness. Psychiatric Services, 46, 508-510.

Bell, M.D., Milstein, R. M., & Lysaker, P. H. (1993). Pay as an incentive in work participation by patients with severe mental illness. Hospital & Community Psychiatry, 44, 684-686.

Black, B. J. (1988). Work and mental illness: Transitions to employment. Baltimore, MD: Johns Hopkins Press.

Bond, G. R. (1992). Vocational rehabilitation. In R. P. Liberman (Ed.), Handbook of psychiatric rehabilitation, 244-275. Boston: Allyn and Bacon.

Bond, G. R., & Friedmeyer, M. H. (1987). Predictive validity of situational assessment at a psychiatric rehabilitation center. Rehabilitation Psychology, 32, 99-112.

Breier A., Schreiber, J. L., Dyer, J. & Pickar, D. (1991) National Institute of Mental Health longitudinal study of chronic schizophrenia: Prognosis and predictors of outcome. Archives of General Psychiatry, 48, (3):239-246.

Campbell, D.T. (1960). Recommendations for APA test standards regarding construct, trait, and discriminant validity. American Psychologist, 15, 546-53.

Cheadle, J. A. & Morgan, R. (1972). The measurement of the work performance of psychiatric patients: A reappraisal. British Journal of Psychiatry, 120, 437-441.

Cheadle, J. A., Cushing, D., Drew, C. D. A. & Morgan, R. (1967). The measurement of the work performance of psychiatric patients. British Journal of Psychiatry, 113, 841-846.

Cook, J. A. & Rosenberg, H. (1993). Predicting community employment among persons with psychiatric disability. Journal of Rehabilitation Administration, 18, 6-22.

Di Stefano, M. K. & Pryer, M.W. (1970). Vocational evaluation and successful placement of psychiatric clients in a vocational rehabilitation program. American Journal of Occupational Therapy, 24, 205-207.

Dion, G. L. & Anthony, W.A. (1987). Research in psychiatric rehabilitation. Rehabilitation Counseling Bulletin, 30, 177-203.

Goldman, H. H., Gattozzi, A. A., & Taube, C.A. (1981). Defining and counting the chronically mentally ill. Hospital and Community Psychiatry, 32, 21-27.

Goldman, H. H., Skodol, A. E., & Lave, T. R. (1992). Revising Axis V for DSM-IV: A review of measures of social functioning. American Journal of Psychiatry, 149, 1148-1156.

Griffiths, R. (1973). A standardized assessment of the work behavior of psychiatric patients. British Journal of Psychiatry, 123, 403-408.

Griffiths, R. (1974). Rehabilitation of chronic psychotic patients. Psychological Medicine, 4, 316-325.

Hoffmann H., & Kupper, Z. (1997, in press). Relationship between social competence, psychopathology, and work performance and their predictive value for vocational rehabilitation of schizophrenic outpatients. Schizophrenia Research.

Holstein, A. R., & Harding, C. M. (1992). Omissions in assessment of work roles: Implications for evaluating and social functioning in mental illness. Journal of Orthopsychiatry, 62, 469-474.

Institute of Medicine. (1991). Disability in America: Toward a national agenda for prevention. A. M. Pope & A. R. Tarlov (Eds.) Washington, D. C.: National Academy Press.

Jacobs, H. E., Wissusik, D., Collier, R., Stackman, D., & Burkeman, D. (1992). Correlations between psychiatric disabilities and vocational outcome. Hospital & Community Psychiatry, 43, 365-369.

Kazdin, A. E. (1982). Methodological strategies in behavior-therapy research. In G. T. Wilson, & C. M. Franks (Eds.), Contemporary Behavior Therapy: Conceptual and Empirical Foundations, 403-439. New York: Guilford Press.

Lysaker, F., Bell, M., Milstein, R., Bryson, G., Shestopal, A., & Beam, J. (1993). Work capacity in schizophrenia. Hospital and Community Psychiatry, 44, 278-280.

Marshak, L. E., Bostich, D., & Turton, L. J. (1990). Closure outcomes for clients with psychiatric disabilities served by the rehabilitation system. Rehabilitation Counseling Bulletin, 33, 247-250.

Massel, H. K., Liberman, R. P., Mintz, J., Jacobs, H. E., Rush, T., Gianinni, C. A., & Zarate, R. (1990). Evaluating the capacity to work of the mentally ill. Psychiatry, 53, 31-42.

McGlashan, T. H. (1988). A selective review of recent North American long-term follow-up studies of schizophrenia. Schizophrenia Bulletin, 14, 515-542.

Minkin, N., Braukmann, C. J., Minkin, B. I Timbers, G. D., Timbers, B. J., Fixsen, D. L., Phillips, E. I., & Wolf, M. (1976). The social validation and training of conversational skills. Journal of Applied Behavior Analysis, 9, 127-139.

Pincus, H. A., Kennedy, C. (1991). Determining disability due to mental impairment: APA's evaluation of Social Security Administration guideline. American Journal of Psychiatry, August, 148, 1037-1043.

Solinski, S., Jackson, H. J., & Bell, N. M. (1992). Prediction of employability in schizophrenic patients. Schizophrenia Research, 7, 141-148.

Strauss, J. S., & Carpenter, W. T. (1974). The prediction of outcome in schizophrenia. Archives of General Psychiatry, 31. 37-42.

Ventura, J., Green, M. F., Shaner, A., & Liberman, R. P. (1993). Training and quality assurance in the use of the Brief Psychiatric Rating Scale: The "Drift Busters". International Journal of Methods in Psychiatric Research, 3,221-244.

Westermeyer, J. F. & Harrow, M. (1987). Factors associated with work impairments in schizophrenic and nonschizophrenic patients. In R. R. Grinker & M. Harrow (Eds.), Clinical Research in Schizophrenia: A Multidimensional Approach, 142-160. New York:Plenum.

Wolf, M. M. (1978). Social validity: The case for subjective measurement or how applied behavior analysis is finding its heart. Journal of Applied Behavior Analysis, 11,203-214.

Roberto Zarate

Robert Paul Liberman

Jim Mintz

H. Keith Massel

University of California, Los Angeles

Dr. R.P. Liberman, Community & Rehabilitative Psychiatry (116AR), West Los Angeles, VA Medical Center, 11301 Wilshire Blvd., Los Angeles, CA 90073.