Work Personality Profile: Validation within the Supported Employment Environment.

The Rehabilitation Act Amendments of 1992 mandated that ongoing support services be provided at least twice monthly, for individuals in supported employment placement, and that such support include comprehensive assessment. The goal of support services is the continuation of employment, with possible additional supports consisting of (a) skilled job trainers to accompany the individual for skill training at the job site (e.g., job coaches); (b) job development and placement; (c) social skills training; (d) follow-up contact with family, friends, employers, and advocates; and (e) supervision and observation (29 USCA, Sec. 706(33)).

While mandated by federal legislation, current comprehensive assessments used in providing supported employment support are criticized for their subjectivity, limited utility across settings, and failure to include items of value in the world of work. Menchetti and Udvari-Solner (1990) believed that traditional assessment techniques (i.e., psychological, aptitude, and intelligence tests) are neither reliable nor valid for this purpose and that they are philosophically incompatible with supported employment assumptions. Such measures lack content and predictive validity because they do not adequately reflect work behavior in real work settings. Menchetti and Udvari-Solner (1990) also believed that traditional assessments are based on a readiness model (i.e., demonstration prior to job placement of appropriate work behaviors) which does not accurately reflect the status of clients entering supported work. Several studies (DeStefano, 1991; Olney & Salomone, 1992; Szymanski, 1991) stated that traditional assessment instruments are inappropriate for use in supported employment because they lack accurate documentation of program service needs and validation within the supported employment environment.

Rogan and Hagner (1990) recommended post-employment evaluation of individuals receiving supported employment services and that these assessments should continue for the duration of employment. Such evaluations should be used by supported employment programs for job or task matching. Menchetti and Rusch (1988); Parker, Szymanski, and Hanley-Maxwell (1989); and Wehman (1981) believed that the best predictor of long-term performance within a supported employment setting is the actual performance in that setting (i.e., a real job) for an adequate period of time. Szymanski, Dunn, and Parker (1989) described this framework as ecological assessment reflecting the presumption that people interact with their environment and that both change as a result of this interaction.

Ecological assessment examines not only the individual but also the potential environment, the perceptions of all individuals involved, and the interactions between individuals and their environment. Menchetti and Udvari-Solner (1990) believed that ecological assessment goes beyond analyzing the individual to include analysis of the workplace and the people involved. According to some experts, ecological assessment begins with measuring worker abilities. Parker et al. (1989) expressed the view that assessment of the individual and, in particular, the assessment of current work related skills/behaviors is an important step in operationalizing ecological assessment.

Hypotheses

Assessments used in supported employment currently lack objectivity and ecological validity. The Work Personality Profile (WPP) developed by Bolton and Roessler (1986) is a behavioral rating instrument for use in employment settings that provides a broad assessment of the vocational circumstance. This study hypothesized that the WPP is a valid instrument for use within the supported employment environment by establishing: (a) content validity through feedback from expert raters, (b) concurrent validity demonstrated by a positive relationship with job satisfactoriness, (c) construct validity demonstrated by a positive relationship with vocational maturity, and (d) predictive validity demonstrated by a positive relationship with job tenure.

Method

Participants

Study participants were individuals receiving supported employment services. Eight non-profit service providers incorporated the WPP into their provision of services. Programs were selected based on their supported employment model of service delivery and size. Only two models (i.e., individual and enclave) were being implemented by programs providing supported employment services in the State in which this study was conducted. The range of participants per program ranged from 4 to 74. All employed program participants participate in the study. One hundred and forty-nine participants were employed at the beginning of the study, with an 51 additional study participants gaining employment during the length of the study. Signed consent was obtained and participant information was confidential. Total sample size was 200.

Study participants included 55% Male and 45% female, with 68% of the participants with mental retardation, 14% with mental illness, 6% with a visual or hearing impairment, 4% with multiple disabilities, 3.5% with an orthopedic disability, and 3% with a learning disability. Not all disabilities were documented by medical or psychological evaluations but, rather, in many cases were documented subjectively by human services counselors, family members, or case workers. The average age was 29 and the ethnicity was 66% Caucasian, 32% African-American, 1% Hispanic, and 1% Asian. Placement was 63% in individual and 37% in enclave.

Expert Raters

The expert raters who gathered the data for this study included eleven job coaches and seven staff supervisors (N = 18). Program directors participating in this study reported that their job coaches or staff supervisors either had a background in job development, job coaching, and service coordination, or were trained in these three areas prior to working with program participants.

Instrumentation

Work Personality Profile (WPP) The WPP is an observational work behavior rating instrument (Bolton & Roessler, 1986). Bolton and Roessler (1986) felt that the WPP measures those abilities that satisfy work role requirements (i.e., work attributes essential to maintain employment). Consisting of 58 behavior rating items, the WPP uses a 4-point scale and can be completed in less than 15 minutes. Scoring can be done either by hand or with a computer. Results are reported as a profile of work behavior in terms of individual scores, scores on 11 rational scales, and scores on the five factor analytic scales. Field testing and factor analysis resulted in 58 items and isolated five empirical scales (Roessler & Bolton, 1983).

Reliability. Internal reliability for the 11 work performance scales ranged from .71 to .92, and the 5 factor-analytic scales had reliability coefficients ranging from .83 to .91. Interrater reliabilities for the 11 work performance scales were as follows after specific time periods: (a) 2 weeks, median of .48, (b) 4 weeks, median of .55, and (c) 6 weeks, median of .62. Median interrater reliability coefficients for the 5 factor-analytic scales were .58 and .66 after 4 and 6 weeks. The authors suggest using 2 to 4 raters to determine an average rating. Re-rating reliabilities average above .80 on all scales, which suggests that raters use the WPP in a consistent manner.

Validity. Concurrent and predictive validity were determined by correlating the WPP with the General Aptitude Test Battery (GATB) service outcomes. The sample used to determine validity consisted of 181 clients at a comprehensive rehabilitation center. Average interscale correlation with the GATB was .78. Clients completing various vocational training programs (e.g., laundry, file clerk, or custodial) had higher WPP scores than participants who dropped out of the training program. Four (i.e., 1, 4, 7, & 8) of the eight scales were significant predictors (p [is less than] .05) of vocational competence. In addition, scales 4, 7, and 8 were significant (p [is less than] .05) predictors of vocational training program completion.

Two research studies support the importance of acquiring and maintaining these eleven work attributes. Hanley-Maxwell, Rusch, Chadsey-Rusch, and Renzaglia (1986) reported that the most frequent reasons cited for job loss for individuals with severe disabilities involved behaviors related to character (e.g., walked off job, stealing, tardiness, lack of attendance and cooperation). Character was closely followed by several behaviors related to production (e.g., lack of independence, quality, inflexibility, and inability to do the job). Although character and production were the most common individual reasons cited for job termination, combination factors were frequently cited. Character/production and lack of social awareness (e.g., poor grooming)/production were the most commonly cited combinations of reasons for termination. Social problems were cited more often than non-social problems.

In a second study, Crimando, Belcher, and Riggar (1986) reported that the major reasons for job loss among individuals with severe disabilities were lack of interpersonal social skills and productivity. Job coaching, co-worker awareness training, on-the-job and follow-up services were identified as the most effective methods for maintaining successful job placement.

Minnesota Satisfactoriness Scale (MSS) The MSS (Gibson, Weiss, Dawis, & Lofquist, 1970) is a 28 item behavioral observation questionnaire designed to assess the satisfactoriness of an individual as a worker. As described in the Minnesota Theory of Work Adjustment (MTWA), satisfactoriness is a measure of workers' on-the-job.

The MSS has been widely used as a concurrent measure for on-the-job evaluation. Wexley and Pulakos (1982) used the MSS as a concurrent measure to determine performance ratings. Hollender (1974) used the MSS to evaluate the success of a work adjustment program for adolescent males with mental retardation and Bolton and Brookings (1991) used the MSS as an instrument for comparing employees with and without a disability.

Employability Maturity Interview (EMI) The EMI is a rationally derived scale of 10 items from Manuele's (1983) 120-item Adult Vocational Maturity Assessment Interview (AVMAI). The 10 items were selected from four of the eight scales comprising the AVMAI and are concerned with orientation to work, concern with choice, self-appraisal: interests and abilities, and self appraisal: personality characteristics (Morelock, Roessler, & Bolton, 1987).

The EMI is a structured interview assessing self-knowledge and occupational information aspects of the world of work (Morelock et al., 1987), which Manuele (1983) considers to be vocational maturity. Manuele (1983) stated that vocational maturity is a function of competently coping with the demands of the various career development stages. The interview is conducted orally, requiring no reading or writing from the respondent. Scoring guidelines and normative data are provided with each interview form (Roessler & Bolton, 1987).

The EMI was appropriate for this study because (a) the AVMAI was developed and validated on individuals who were educationally and economically disadvantaged and the sample included many individuals with disabilities (Manuele, 1983), (b) the EMI was normed on a sample of individuals with disabilities (Roessler & Bolton, 1987), and (c) work behavior and vocational maturity are constructs of competently coping with career development (Manuele, 1983; Super, 1974).

Content Validity Rating of WPP. In this study, 18 expert raters provided content validity ratings for the WPP. These expert raters were asked how well items in the WPP represented behaviors needed for: (a) training for a job, (b) maintaining employment, and (c) demonstrating promotability.

Procedure

Expert raters collected data on study participants using the WPP, MSS, and EMI within thirty days of study initiation. Prior to data collection, meetings were held with program directors and expert raters to incorporate the instruments within ongoing program service delivery and to train the expert raters on the proper use of the instruments. All the expert raters volunteered to collect data for this study. Additional data were collected at three and six months from the initial data gathering date. Expert raters completed the WPP and MSS on each participant after observing the participant, at the job site and working, for at least two weeks. The EMI was completed on each participant during break periods or regularly scheduled individual monthly meetings. The content validity rating was completed at six months, study length, from initial assessment. Gender, race, age, type of disability and placement, number of months on the job, currently employed, and hourly wage rate data were collected at end of study.

Design

Pearson Product-Moment correlation coefficients were obtained between total scores on the WPP and MSS, EMI, job tenure (i.e., currently on the job, number of job changes, days absent, wage level, recommendations for promotion, salary increase, and job mobility), and related data. To determine, if any, the effect of time on the job and job tenure, partial correlations, controlling for time on the job, were obtained between WPP scores and job tenure.

Results

Validity Data

The results generally supported the validity and usefulness of the WPP. Specific validity results are as follows:

Content validity. Mean scores of 3.0 or greater indicate support for content validity of the WPP (A rating of 3 indicates that the WPP is at least somewhat useful). In all three categories training for a job (M = 3.00, SD = 1.33); maintaining employment (M = 3.28, SD = 1.32); and demonstrating promotability (M = 3.06, SD = 1.21); expert raters reported that items in the WPP represented work behaviors needed in these three employment phases.

Description of Variables for Table 1. Table 1 reports the intercorrelations between WPP scores and data related to the study hypotheses (e.g, indicators of concurrent, construct, and predictive validity, and job tenure). MSS 1 to MSS3 are scores on questions 14, 15, & 16 on the MSS and indicate job advancement (i.e., vertical mobility and job tenure). These questions ask raters to make hypothetical decisions (i.e., Give him/her a raise? Transfer him/her to a job at a higher level? Promote him/her to a position of more responsibility?). The WPP data were collected at study initiation or at job placement. WPP3 and WPP6 data were collected at three and six months after WPP data collection. Job tenure data (e.g., MSS 1, hourly wage) were collected at the end of the study and at least two months after the WPP, WPP3, and WPP6 data were collected.

Table 1

Intercorrelations between WPP Scores and Related Data

                     WPP          WPP3         WPP6
                     (n = 194)    (n = 169)    (n = 145)

MSS                  .57(**)
EMI                  .49(**)
MSS1                 .49(**)      .50(**)       .47(**)
MSS2                 .42(**)      .60(**)       .50(**)
MSS3                 .38(**)      .57(**)       .58(**)
Employed            -.13         -.28(**)      -.25(**)
# of Job Changes     .07          .02           .00
Avg. Weekly Hours    .16          .14           .13
Days Absent          .02          .05          -.10
Hourly Wage          .43(**)      .46(**)       .38(**)
Time on the Job      .09          .08           .13

(**) 12 < .01

The variable employed" was coded as 1 for employed and 2 for not employed at time of data collection, hence a negative correlation indicates a relationship between a high WPP score and employment.

Concurrent validity. The WPP correlated significantly with the Minnesota Satisfactoriness Scale (MSS), a measure of job satisfactoriness (r = .57, p = [is less than] .01; see Table 1). Concurrent validity is supported by the strong relationship between the WPP and the MSS.

Construct validity. Manuele (1983) stated that individuals who effectively master work tasks similar to those Super (1957) included in the trial work period (e.g., adjusting to work tasks, routine, and tempo) are said to be vocationally mature. The WPP correlated significantly with the Employability Maturity Interview (EMI) supporting construct validity (r = .49, p = [is less than] .01; see also Table 1). Construct validity is supported by the strong relationship between the WPP and the EMI.

Predictive validity. Initial WPP scores correlated significantly with MSS3 scores (r = .38, p = [is less than] .01; see also Table 1). As previously reported, MSS3 scores, collected 3 months after initial data collection, are answers to three questions concerning vertical mobility and job tenure. WPP scores correlated significantly with hourly wage (r = .43, p = [is less than] .01). This dependent variable was collected six months from initial WPP. Also, WPP scores collected at three and six months both correlated significantly with employment (i.e., is the participant employed at the end of the study; r = -.28, p = [is less than] .01).

The WPP did not correlate significantly with number of job changes since placement, number of days absent from work, or time on the job. Description of the sample indicates that a high percent of participants did not change jobs (84%) nor were they typically absent from work (Mode = .00). In addition, subsequent WPP data (i.e., WPP3 and WPP6) had stronger correlational values with the outcome variables that the initial WPP rating (see Table 1)

Partial correlational analysis controlling for time on the job also addressed the predictive validity of the WPP, as time could be a selection factor that contributes to work performance. However, the WPP continued to demonstrate a significant relationship with hourly wages and vertical mobility (hourly wage, r = .42, p [is less than] .01; MSS1, r = .47, p [is less than] .01; MSS2, r = .42, p = [is less than] .01; MSS3, r = .38, p, .01; see Table 2).

Table 2

Partial Correlation Coefficients Controlling for Time

WPP
(n = 164)

MSS1           .47(**)
MSS2           .42(**)
MSS3           .38(**)
Days Absent   -.04
employed       .01
Age            .05
Hours          .13
Wage           .42(**)

(**) 12 < .01

Predictive validity is supported by the relationship between scores on the WPP and indicators of job tenure and related data (i.e, job advancement, employment, average weekly hours, and hourly wage).

Discussion

Limitations of Study

The supported employment programs for this study were chosen for their supported employment model and size. This sample represented two (i.e., individual and enclave) of the four models of supported employment placement. The other two models are work crews, that travel to jobs and perform specific contract duties, and entrepreneurial operations, that perform product or service specific subcontract work (Kregel, Wehman, & Banks, 1989).

Program sites used a variety of record keeping methods, and program service planning varied from programs conducting weekly staffings of participants to individual supervisors responsible for participant evaluation. Readers are also reminded that the same raters completed all three instruments, a fact which could have affected the relationship found between the instruments.

Implications of Study Findings

Roessler and Greenwood (1987) stated that vocational evaluations/assessments, within the broad scope of providing services to individuals with a disability, should consider the stable aspects of an individual (e.g., aptitudes, interests, and needs) as well as those aspects that affect motivation and choice making (e.g., self image and decision and problem-solving skills). In addition, assessment measures need to address an individual's ability to meet the work role, retain employment, seek a job, and make reasonable vocational choices.

Most current assessment systems within the supported employment setting are subjective (e.g., committee staffings), have limited utility, and fail to include items of value in the world of work. Data reported in this study support the use of the WPP as a valid instrument for use with individuals receiving supported employment services. This study fulfills an important step in operationalizing the WPP as an ecological assessment. The results demonstrate the validity of the WPP by assessing an individual's work related skills/behaviors within the actual employment setting (Parker et al., 1989; Szymanski et al., 1989).

By using the WPP, job coaches and other supervisory staff would be able to make decisions regarding service delivery options based on more objective information. Program staff would be able to target specific work related needs and job related behavior deficits. In the face shrinking resources the use of the WPP as an ongoing assessment could be a cost effective method to facilitate planning of needed support services.

Information from the WPP can also be used to develop intervention techniques. As findings support a relationship between WPP scores and indicators of job performance, it would be beneficial to individuals receiving supported employment to receive services related to areas of identified weaknesses. A skills training model could be implemented alongside the implementation of the WPP as an ongoing assessment (Marr & Roessler, 1986).

A study would be beneficial to develop best practice" procedures to implement the WPP into supported employment assessment, planning, and job placement procedures. Findings provided a valuable rationale for programs throughout the nation to incorporate the WPP in their service delivery systems. Research should consider strategies to overcome barriers to implementation (e.g., type of placement model, lack of administrative support and mission statement, and staff attitude towards using an objective assessment). The WPP not only provides supported employment service providers with valid information, such information could be useful in providing prompt and cost effective services.

References

Bolton, B., & Roessler, R. (1986). Manual for the work personality profile. Fayetteville, AR: University of Arkansas Research & Training Center in Vocational Rehabilitation.

Crimando, W., Belcher, K., & Riggar, T. F. (1986). Job retention problems of clients served in rehabilitation facilities. Journal of Job Placement, 2(1), 10-12.

DeStefano, L. (1991). A historical overview of research in supported employment: Implications for the next decade. Journal of Disability Policy Studies, 2(1), 10-12.

Gibson, D. L., Weiss, D. J., Dawis, R. V., & Lofquist, L. H. (1970). Manual for the Minnesota Satisfactoriness Scale. Minneapolis, MN: University of Minnesota Industrial Relations Center.

Hanley-Maxwell, C., Rusch, F. R., Chadsey-Rusch, J., & Renzaglia, A. (1986). Reported factors contributing to job terminations of individuals with severe disabilities. Journal of the Association for Persons with Severe Handicaps, 11 (1), 45-52.

Hollender, J. W. (1974). Prediction of work adjustment for adolescent male educable retardates. Journal of Counseling Psychology, 21, 164-165.

Manuele, C. A. (1983). The development of a measure to assess vocational maturity in adults with delayed career development. Journal of Vocational Behavior, 23, 45-63.

Marr, J., & Roessler, R. T. (1986). Behavior management in work settings. Fayetteville, AR: Arkansas Research & Training Center in Vocational Rehabilitation.

Menchetti, B. M., & Rusch, F. R. (1988). Vocational evaluation and eligibility for rehabilitation services. In P. Wehman & M. S. Moon (Eds.). Vocational rehabilitation and supported employment (pp. 79-90). Baltimore: Brookes.

Morelock, K., Roessler, R., & Bolton, B. (1987). The employability maturity interview: Reliability and construct validity. Vocational Adjustment and Work Evaluation Bulletin, 20, 53-59.

Olney, M. F., & Salomone, P. R. (1992). Empowerment and choice in supported employment: Helping people to help themselves. Journal of Applied Rehabilitation Counseling, 23(3), 41-44.

Parker, R. M., Szymanski, E. M., & Hanley-Maxwell, C. (1989). Ecological assessment in supported employment. Journal of Applied Rehabilitation Counseling, 20(3), 36-43.

Roessler, R., & Bolton, B. (1983). Assessment and enhancement of functional vocational capabilities: A five-year research strategy. Richard J. Baker Memorial Monograph Series, 1. Vocational Evaluation & Work Adjustment Association.

Roessler, R., & Bolton, B. (1987). Manual for the employability maturity interview. Fayetteville, AR: University of Arkansas Research & Training Center in Vocational Rehabilitation.

Roessler, R., & Greenwood, R. (1987). Vocational evaluation. In B. Bolton (Ed.), Handbook of Measurement and Evaluation in Rehabilitation (2nd ed., pp. 151-168). Baltimore: Paul H. Brookes Publishing Co.

Rogan, P., & Hagner, D. (1990). Vocational evaluation in supported employment. Journal of Rehabilitation, 56(1), 45-51.

Super, D. E. (1974). Vocational maturity theory. In D. E. Super (ed.), Measuring vocational maturity for counseling and education (pp. 9-23). Washington, DC: National Vocational Guidance Association.

Szymanski, E. M. (1991). Quantitative research: Common myths and cautions for utilization in supported employment. Journal of Disability Policy Studies, 2(1), 115-126. Press.

Szymanski, E. M., Dunn, C., & Parker, R. M. (1989). Rehabilitation of persons with learning disabilities: An ecological framework. Rehabilitation Counseling Bulletin, 33, 38-53. 29 United States Code Annotated 601-800 pp. 204-209 (West Supplement, 1993)

Wehman, P. (1981). Competitive employment: New horizons for severely disabled individuals. Baltimore: Brookes.

Wexley, K. N., & Pulakos, E. D. (1982). Sex effects on performance ratings in manager-subordinate dyads: A field study. Journal of Applied Psychology, 67,433-439.

Ed R. Williams, Ph.D., LPC, CRC, University of Arkansas at Little Rock, Ross Hall #412, 2801 South University Avenue, Little Rock, AR 72204-1099.3