Employment Outcome Expectancies: Consensus Among Consumers, Providers, and Funding Agents of Community Rehabilitation Programs.

Securing and maintaining employment are perhaps the most important and tangible outcomes of successful vocational rehabilitation programs. They also serve as a means of examining the effectiveness of these programs (Johnston & Granger, 1994; Thomas & Menz, 1997). The outcomes achieved are often used to substantiate the time, human resources, and dollars invested in rehabilitation efforts (Berkowitz, 1985; Johnston, Hall, Carnavale, & Boak, 1995). Outcomes data are also used to substantiate claims of the relative effectiveness of specific approaches or programs developed to serve persons with disabilities (Gibbs, 1991). A review of employment outcome literature, however, reveals that a standard means of reporting employment outcomes is not presently used by the field (McAweeney, Forchheimer, & Tate, 1997; Thomas & Menz, 1997). In fact, a recent analysis of the literature dealing with employment out comes found wide variability in the nature of the data reported, making any type of meaningful inter-program comparisons difficult to impossible (Cifu et al., 1997).

Employment outcomes have often been used as a benchmark of program success for community-based rehabilitation programs (CRPs). All parties having a stake in the nature and quality of the outcomes expected for consumers from CRPs were believed to have similar ideas of what they considered as a reasonable measure of a successful employment outcome. However, no studies were found that confirmed that a consensus existed. Therefore, because a standard employment outcome reporting mechanism is not presently used for persons exiting CRPs, comparisons of programs, program models, or treatment effects cannot be reliably and validly accomplished.

Key aspects of employment outcome information are not routinely reported. Hours worked, wages earned, benefits received, and the employment models used (supported, transitional, independent, enclave) are reputedly important to interpreting the success of the outcomes achieved (CARF, 1997). These aspects, however, are frequently not specified in the employment outcome literature. The impact of selectively reporting certain aspects of employment outcome information can be significant. Such information can intentionally or inadvertently mislead and, therefore, requires serious attention and rigorous investigation (Cook & Rosenberg, 1994). The development of guidelines for reporting outcomes could assist in development of a means of more accurately documenting the nature and extent of successes achieved by CRPs.

In addition to inconsistent data reporting many outcome studies do not examine satisfaction with the services received (Moseley, 1988; Schwab, DiNitto, Simmons, & Smith, 1999). The opportunities provided for making informed choices in the rehabilitation process are also seldom addressed (Gibbs, 1991; Kosciulek, 1999). The compatibility of jobs obtained with consumer expectancies, goals, and needs, while of great importance to the consumer, is also not commonly addressed in the outcomes reported in the literature (Inge, Banks, Wehman, Hill, & Shafer, 1988). Despite the movement to empower consumers with rights to choose and be actively involved in rehabilitation planning and goal selection (Rogers, Anthony, Cohen, & Davies, 1997), few contemporary studies report data on these variables in relation to employment outcomes resulting from involvement in CRPs (Kosciulek, 1999).

Although employment is a primary goal of community-based rehabilitation programs and is often the standard of success by which programs are evaluated (Dean, 1991; Kruse, 1997), several factors have rendered employment outcomes more difficult to accurately define and report. These include the advent of various new vocational rehabilitation strategies (Inge & Tilson, 1997), technology interventions (Butterworth, Whitney-Thomas, & Shaw, 1997), and work-related supports available for maintaining persons in community-based employment following the provision of vocational rehabilitation services (Byboe, Mowbray, & McCroham, 1995). Because employment outcomes vary widely as a function of unemployment rates, severity of disability, and available support services, and a host of other moderating variables, attempting to compare programs becomes even more complex (Hughes & Scott, 1997).

Employment outcome studies presented in the rehabilitation literature often fail to define even the least technical aspects of employment (Kay, 1993). For example, researchers often do not consistently specify whether employment outcomes achieved were permanent or temporary, full time or part time, and whether employment obtained was in part subsidized by other entitlement programs or government initiatives (Johnston, 1991). All of these issues are important and related to quality of outcomes. A review of recent vocational outcome publications for persons with disabilities reveals that researchers publishing in this area tend to not use the same criteria when reporting job success rates, thus making comparisons between programs difficult. Researchers and policy makers do agree, however, that efforts are needed to begin to standardize employment outcome reporting (Thomas & Menz, 1997).

Employment characteristics that are important to examine when evaluating the nature and success of employment outcomes of persons who obtain services from CRPs have been widely cited in the research literature, but are not consistently reported. Some of the many factors cited as important to report in outcome reporting include (a) number of hours worked and wages earned (Stodden & Browder, 1986), (b) job satisfaction (Koch & Merz, 1995), (c) employment benefits received (Bell, Lysaker, & Milstein, 1996), (d) capability of advancing on the job (Sands, Kozleski, & Goodwin, 1992), (e) the degree to which a person was empowered with the right to choose employment goals and services needed to achieve those goals (Rumrill & Garnette, 1997), and (f) whether the person with a disability being served has been accommodated adequately and reasonably at the work site to make them as independent as possible (Zuckerman, 1993). Improvement in a person's quality of life following involvement in vocational rehabilitation has also been cited as an important outcome (Fabian, 1991; Felce & Perry, 1995).

Another important variable for investigation is when outcomes should be reported. Employment outcome data presented in project reports, outcome summaries, and book and journal publications suggest that a standard time frame for reporting is often not used, a factor that is important with certain disability groups. Persons with brain injuries, for example, have been cited as often capable of quickly finding work or returning to a former job, but are likely to have trouble sustaining employment after the standard 90-day follow-up period without support or work accommodations (Bollingmo, 1997; Johnston et al., 1995; Kay, 1993). Persons with persistent mental illness are more likely to have cyclic patterns of rehabilitation need during periods of more florid symptoms that are more likely to affect employment at irregular intervals (Bond & McDonel, 1991; Marshak, Bostick, & Turton, 1990). This issue of when outcome measures should be reported is perhaps as critical as any other aspect of outcomes reporting and may in fact require disability specific standards for reporting.

This study was conducted to determine whether consensus between various stakeholders (i.e., consumers, providers, and funders) on identification of the most important employment outcome characteristics exists. Opinions of matched sets of consumers, funding agents, and CRP providers were surveyed to determine the degree of consensus that existed regarding employment outcome expectancies. The outcome expectancies included the factors traditionally reported in the vocational rehabilitation literature such as wages, benefits, and hours worked as well as a wide range of other variables. These variables included less tangible items as specified in federal legislation, such as opportunities for informed choice, active involvement in all aspects of employment planning, consumer satisfaction, and suitability of work and living environments. More specifically, this research was conducted to address the following research questions relative to 66 variables derived from a review of vocational outcomes literature and considered as important by a consistency advisory committee (see method section for details):

1. What do various stakeholders (i.e., consumers, providers, and funders) perceive to be the most important employment outcome characteristics?

2. Does a consensus exist among stakeholders regarding the most important employment outcome characteristics?

3. What do stakeholders believe are the most appropriate time frames to collect employment outcome information?

Note that subjects were instructed to give their opinions of the importance of all factors identified in the survey instrument. Subjects were not asked to offer an opinion as to whether the CRP or the employment achieved actually met the stated criteria.

Method

Identification of Variables

In order to address the larger issue of defining what should be included in an employment outcomes instrument, this study began by examining the characteristics judged to be most important to the primary stakeholders. As a preliminary step, a Constituency Advisory Committee (CAC) was convened to provide advisement, input, and direction to this study. This group represented people with significant experience in consumer advocacy, outcome research methodology, CRP vocational programming, or combination of these areas. The CAC consisted of 14 persons: 4 senior level researchers; 2 representatives of a national accrediting organization; 3 persons with disabilities, associated with national advocacy groups; 3 CRP directors; 1 director of a state association of rehabilitation facilities; and 1 director of a state vocational rehabilitation agency. This approach used Participatory Action Research (Elden, 1983) methodology to insure input and involvement of important constituents.

Potential employment-related domains and content were initially identified through a literature review. Content items were developed based on literature related to employment benefits and conditions, informed consumer choice, rehabilitation partnerships, consumer satisfaction, and federal legislative requirements. These issues were reportedly related to the consumers' perspectives regarding quality of life and expectancies of how vocational rehabilitation could improve their independence and self-sufficiency. Potential disability-specific characteristics related to employment outcomes were not examined. A taxonomy of variables was organized and then revised as a result of CAC review and input. The CAC subsequently reviewed the revised taxonomy to establish scalings for ratings of importance of the variables included in the consensus study. The survey questionnaire was then revised and mailed to a selected audience of responders using a random selection of CRPs as described herein.

Participants

The subjects of this study included triads of CRP providers, funders, and consumer representatives. The CRPs were believed to be representative of CRPs nationally. The CRPs were randomly selected from mailing lists and the funders and representatives were identified by the CRP. Equal numbers of CRPs accredited by the Commission on Accreditation of Rehabilitation Facilities (CARF) and non-CARF accredited facilities were randomly selected from the most recent CARF mailing list and from a mailing list of non-CARF accredited CRPs that offer employment-related services. A sample of programs were randomly selected from the hundreds of programs identified until 200 unduplicated and valid locations were obtained for each category.

Both the CARF accredited programs and programs that were non-CARF accredited were verified to assure that they provided community-based vocational rehabilitation programs. This verification was done either by Internet search of the yellow page listings or by a direct telephone call to the program. Additional CARF and non-CARF accredited programs were identified as alternative sites in the event that an adequate sample was not obtained.

CRPs selected for study were asked to identify a representative of their primary funding source and a consumer or consumer advocate representative of the persons typically served by their program, to respond to parallel versions of a survey. It was suggested that the last person to obtain community-based employment who is representative of their persons served by the CRP be selected.

Survey Instrument

The survey used a 7-point, Likert-type scale ranging from "not important" to "extremely important." Respondents were asked to rate the importance of 66 variables identified in the literature and suggested by the CAC as important to employment outcomes or important as employment outcomes.

The first six survey items examined pre-employment service factors. The items pertained to employment planning (e.g., "work interests were explored and identified" and "barriers to achieving job goals were identified"). An overall rating of the importance of the planning components to enhancing employment outcomes was also included.

A section focusing upon skill training and work preparation identified four important characteristics, including whether rehabilitation programs developed specific marketable job skills, developed job seeking and keeping skills, and identified work options.

The importance of direct work benefits, job delimiters, and worker expectations from employment and job placement service variables were also rated. Direct work benefits included such factors as hours worked, wages earned, and benefits received. Job delimiters and worker expectations categories included such variables as travel time and distance between home and work, regularity of hours per pay period, and safety of work conditions.

The perceptions of high quality services were also addressed in the survey. For example, respondents were asked to rate such characteristics as the competence of staff, quality of services, opportunities for consumers to exercise informed choice, and respect for consumers.

Another section of the questionnaire identified benchmark times for which to follow up on employment services. The responders were asked to identify, from their perspective, the optimal time frames to follow up on outcomes from vocational assessment and planning, skill training, and employment services.

Initial Mailing

In order to involve willing participant sites and to obtain information regarding the demographics of each of the CRP subjects, a cover letter was sent to each of the 400 CRPs. The letter introduced the purposed study, requested the program's assistance in identifying demographic information, and requested participation in a survey to determine the importance of employment outcome characteristics.

Of the 400 surveys mailed to solicit potential research sites, 177 were returned, with 164 providing data regarding the services offered and the population served. Of this group, however, only 79 indicated a willingness to participate in the consensus study.

These 79 CRPs agreed to respond to a subsequent survey and to forward questionnaires to a funding agent and a consumer representative. Each of the three versions of the questionnaires included directions that all stakeholders were to independently complete the questionnaires. Separate mailing envelopes were sent to each respondent. The CRP provided names and addresses of the identified funder and advocate. Funders and advocates were notified that the researchers would attempt to follow up with people who did not return a questionnaire.

Survey Mailings

Executive directors at each site were asked to identify an individual within their organization to serve as a responder. It was suggested that the rater preferably select the last person who obtained community-based employment if the rater believed that the subject was representative of the people typically served by the CRP. The directors were informed that they could complete the rating themselves or they could give it to a rehabilitation director or other appropriate staff to complete. This person was then asked to convey one copy of the survey to a funding agent and another copy to a consumer or the CRP's consumer advocate, depending on the person's ability to participate in the process. Surveys were color coded to identify who was completing the survey (provider, funder, consumer).

Results

Sample Characteristics of Participating CRPs

Although 79 CRPs agreed to participate in the research, only 42 usable sets of matched groups were obtained. The majority identified persons with developmental disabilities as the primary disability group served. Persons with mental illness, physical disabilities, and brain injury were identified less frequently as the primary disability served by CRPs. The most frequently selected secondary disability groups served included persons with mental illness followed by persons with physical disabilities.

Primary services provided by the participating CRPs were identified as employment- related services, job seeking/keeping skills, vocational evaluation, work adjustment, and skill training. The majority also provided supported and CRP-based employment, work activities, and day activities. Transportation, recreation, and residential services were often provided by the CRPs surveyed as well. The high incidence of offering employment-related services and job-seeking skills training and proportionately lower incidence of vocational evaluations and skill training may suggest a trend towards less emphasis on assessment and employment preparation services. The finding that supported employment is provided more frequently than sheltered employment, work activities, and day activities is also suggestive of a trend in vocational rehabilitation service delivery.

Although an attempt was made to obtain equal numbers of CARF and non-CARF accredited programs, a larger percentage of the responses received were obtained from CRPs with CARF accreditation. Of the final 56 programs that had three stakeholders represented, only 9 were not CARF accredited in employment-related services. Because of the small number of non-CARF respondents, meaningful differences between CARF accredited programs and non-CARF accredited programs could not be determined.

Analysis of Consensus Survey

In order to provide the reader with the content of the survey, a shortened version of the content items are listed in Table 1. The items are grouped in the manner suggested by the CAC for presentation in the survey.

Table 1: 66 Outcome Variables

* Work interests explored and identified
* Work options detailed - assets and transferable work skills
  identified
* Vocational guidance used to explore work options and goals
* Barriers to achieving job goals identified
* Plans developed to attain job goals
* Trial work opportunities used to determine suitability of
  jobs if appropriate
* Specific marketable job skills developed
* Job seeking and keeping skills developed
* Career potential improved
* Work or job options identified
* Hours work weekly/annually
* Hourly/weekly/annual earnings
* Earnings in relation to a comparable job or industry
  standard
* Minimum wage
* Level needed to be self-sufficient
* Reduced reliance on government subsidies or entitlement
  programs
* Affordable housing after job placement
* Primary medical coverage
* Dental coverage
* Company sponsored retirement program
* Paid sick days
* Paid holidays
* Paid vacation
* Personal days
* Long-term disability coverage
* Competitive employment
* Transition employment
* Supported employment
* Sheltered employment
* Education or additional training
* Alternative productive activities
* Potential for stable, long-term employment
* Potential for future earning, advancement, and development
  of additional skills
* Positive return on money invested in work training or
  preparation realized following job placement
* Child care availability
* Travel time and distance between home and work
* Transportation availability for work
* Regularity of hours worked each pay period
* Choice of work schedule or shift
* Flexible work schedule (e.g., time-off when needed
* Job matches skills, abilities, employment goals, and values
* Support of rehabilitation staff
* Clearly understand work assignments
* Readily accessible employment services if job is in trouble
  or lost
* Reasonable work accommodations and assistive technology
  needs considered
* Training for co-workers who provide supports or assistance
  is given
* Opportunity for active participation in job search
* Attractiveness of work environment
* Opportunity to socialize at work
* Work that is valued by others
* Safe working conditions
* Work with people who do not have a disability
* Work that is not stressful
* Opportunity to feel valued
* Work that is enjoyable or satisfying
* Opportunity to work independently
* Productive use of individual's time
* Independent living options are promoted
* Participation in social and recreational activities of
  community is possible
* Competence of service staff
* Organization's reputation for quality services
* Service setting is non-stigmatizing
* Opportunity for informed choice and to exercise personal
  choice, and active participation in all rehabilitation phases
  exists
* Persons served are treated with respect and sensitive issues
  are dealt with respectfully
* Price charged for services
* Time from referral to placement outcome

Because only complete matched data sets were used, 14 respondents were eliminated from the database, thus a total of 42 useful sets of matched groups of providers, funders, and consumer advocates were obtained. Data from these matched groups were analyzed to determine whether there was a consensus among matched sets of respondents. An analysis of variance (ANOVA) followed by a Bonferroni post-hoc analysis was computed for each item to determine whether significant differences between the groups existed. Significant differences were identified across only three items (see Table 2). These items indicated the tendencies of the:

1. Funding agents to put less emphasis on the importance of obtaining a job with earnings above minimum wage than did consumers.

2. Consumers to place greater importance on personal days than funders.

3. Funders to place more significance on the competence of service staff than providers.

Table 2: Significant Results Indicating Items in Which There
Was a Potential Lack of Consensus

                                              Grand
Item                   Group Means            Means    p Value

                Provider   Payor   Advocate

Minimum wage      5.21      5.72     5.02      5.30     P=.006
Personal days     4.28      4.58     4.96      4.57     P=.027
Competence of
service days      6.08      6.52     6.29      6.52     P=.021

In addition, grand means were calculated and used to rank order the importance of the variables. The three highest ranked items in each vocational category are listed in Table 3. Survey items listed in Table 3 are ranked by means within each category. The means for all three groups taken collectively were ranked by their ordinal position within each of the subscales.

Table 3: Items Judged To Be Most Important Within Categories

                                                   Group Means

              Sections and Items

                                                 Provider   Payor

Employment Planning

Work interests explored and identified             6.35     6.18

Work options detailed - assets and                 6.23     6.29
transferable work skills identified

Plans developed to attain job goals                5.99     6.05

Employment Preparation

Job seeking and keeping skills developed           6.04     5.98

Work or job options identified                     5.79     5.81

Specific marketable job skills developed           5.65     5.95

Employment Descriptors

Employment Formats
- Competitive                                      6.35     6.43

- Supported                                        6.13     6.06

- Potential for stable,
long-term employment                               6.05     6.15

Work Supports and Accommodations

Transportation availability for work               6.99     6.44

Support of rehabilitation staff                    5.78     5.96

Reasonable work accommodations
and assistive technology needs considered          6.00     6.33

Worker Expectations

Safe working conditions                            6.36     6.42

Productive use of individual's time                5.99     6.30

Opportunity to feel valued                         6.06     5.84

Provider's Quality of Service

Persons served are treated with respect and
sensitive issues are dealt with respectfully       6.63     6.72

Opportunity for informed choice and to
exercise personal choice, and active partici-
pation in all rehabilitation phases exists         6.38     6.30

Competence of service staff                        6.08     6.52

                                                 Group Means   Grand
              Sections and Items                               Means

                                                   Advocate

Employment Planning

Work interests explored and identified               6.30      6.28

Work options detailed - assets and                   6.00      6.18
transferable work skills identified

Plans developed to attain job goals                  6.13      6.05

Employment Preparation

Job seeking and keeping skills developed             5.91      5.98

Work or job options identified                       5.69      5.77

Specific marketable job skills developed             5.67      5.74

Employment Descriptors

Employment Formats
- Competitive                                        5.96      6.25

- Supported                                          6.12      6.11

- Potential for stable,
long-term employment                                 6.11      6.10

Work Supports and Accommodations

Transportation availability for work                 6.30      6.62

Support of rehabilitation staff                      6.07      6.18

Reasonable work accommodations
and assistive technology needs considered            6.21      6.16

Worker Expectations

Safe working conditions                              6.44      6.40

Productive use of individual's time                  6.04      6.09

Opportunity to feel valued                           6.28      6.06

Provider's Quality of Service

Persons served are treated with respect and
sensitive issues are dealt with respectfully         6.55      6.63

Opportunity for informed choice and to
exercise personal choice, and active partici-
pation in all rehabilitation phases exists           6.38      6.35

Competence of service staff                          6.29      6.27

                                                  Ranked
              Sections and Items                  Within
                                                 Subscale

Employment Planning

Work interests explored and identified              1

Work options detailed - assets and                  2
transferable work skills identified

Plans developed to attain job goals                 3

Employment Preparation

Job seeking and keeping skills developed            1

Work or job options identified                      2

Specific marketable job skills developed            3

Employment Descriptors

Employment Formats
- Competitive                                       1

- Supported                                         2

- Potential for stable,
  long-term employment                              3

Work Supports and Accommodations

Transportation availability for work                1

Support of rehabilitation staff                     2

Reasonable work accommodations
and assistive technology needs considered           3

Worker Expectations

Safe working conditions                             1

Productive use of individual's time                 2

Opportunity to feel valued                          3

Provider's Quality of Service

Persons served are treated with respect and
sensitive issues are dealt with respectfully        1

Opportunity for informed choice and to
exercise personal choice, and active partici-
pation in all rehabilitation phases exists          2

Competence of service staff                         3

The 10 highest ranked items over all categories are listed in Table 4. Survey items listed in Table 4 are ranked by grand means for all three groups taken collectively.

Table 4: The 10 Items Judged To Be Most Important Across All Categories

                                                    Group Means

              Sections and Items

                                                 Provider   Payor

Persons served are treated with respect
& sensitive issues are dealt with respectfully     6.63      6.72

Transportation availability for work               6.99      6.44

Safe working conditions                            6.36      6.42

Opportunity for informed choice and to
exercise personal choice, and active partici-
pation in all rehabilitation phases exists         6.38      6.30

Work interests explored and identified             6.35      6.18

Competence of service staff                        6.08      6.52

Competitive                                        6.35      6.43

Support of rehabilitation staff                    5.78      5.96

Work options detailed - assets
and transferable work skills identified            6.23      6.29

Reasonable work accommodations and
assistive technology needs considered              6.00      6.33

                                                 Group Means   Grand
              Sections and Items                               Means

                                                   Advocate

Persons served are treated with respect
& sensitive issues are dealt with respectfully       6.55       6.63

Transportation availability for work                 6.30       6.62

Safe working conditions                              6.44       6.40

Opportunity for informed choice and to
exercise personal choice, and active partici-
pation in all rehabilitation phases exists           6.38       6.35

Work interests explored and identified               6.30       6.28

Competence of service staff                          6.29       6.27

Competitive                                          5.96       6.25

Support of rehabilitation staff                      6.07       6.18

Work options detailed - assets
and transferable work skills identified              6.00       6.18

Reasonable work accommodations and
assistive technology needs considered                6.21       6.16

                                                 Overall
              Sections and Items                 Ranking

Persons served are treated with respect
& sensitive issues are dealt with respectfully      1

Transportation availability for work                2

Safe working conditions                             3

Opportunity for informed choice and to
exercise personal choice, and active partici-
pation in all rehabilitation phases exists          4

Work interests explored and identified              5

Competence of service staff                         6

Competitive                                         7

Support of rehabilitation staff                     8

Work options detailed - assets
and transferable work skills identified             9

Reasonable work accommodations and
assistive technology needs considered              10

Although 3 of the 66 variables studied did show a significant difference, the variability, in a practical sense was minimal. Thus, this supports the contention that, in general, a consensus of opinion exists as to the most important outcome expectancies.

The study also revealed that the top concerns among all three groups were not wage and benefits expectations, as may have been expected. Instead, the top concerns were related to quality of employment, quality of services, and opportunity for informed choice. Of interest are the low rankings associated with "hours worked" (overall ranking 62 of 66), "hourly/weekly earnings" (overall ranking 54 of 66), and "earning minimum wage" (overall ranking 40 of 66).

Time Frames for Outcomes

In order to determine whether a consensus could be reached regarding the most appropriate times to collect information regarding employment outcomes, subjects were given several options of benchmark dates for documenting outcomes in relation to an individual's participation in services. Then, they were asked to give an open-ended response to the questions as to the number of days following the benchmark reference point when a follow-up for outcomes reporting should take place. Questions were posed as to the appropriate timeframes for the following: (a) follow-up on vocational assessment, (b) skill training and employment preparation, and (c) employment and job placement services. Benchmark and follow-up periods were calculated using only the data sets that included responses from all three stakeholders, using grand means for all matched respondents.

For vocational assessment services, the majority (55.4%) indicated that an appropriate time to assess satisfaction with services was after evaluation when an agreed upon plan is developed. Another 19.8% suggested the date that the person exits service is an appropriate benchmark, and 24.0% identified the first date of employment as most appropriate (.8% were not able to be entered because a range was given as a response). In terms of when a follow-up should be conducted, 51% of the respondents suggested that 30 days after the benchmark date would be appropriate. The other most frequently selected time frames were 60 days (12.1%), 90 days (10.3%), and 0 days (10.3%). It is assumed that the latter did not feel that a follow-up on this service element was important. Additionally, 16.3% gave other responses.

For skill training and employment services, the majority (60%) identified the day that the person completes training as the most appropriate benchmark date. Of those not selecting the exit date, the majority (nearly 40%) identified the first day of employment as most appropriate as a benchmark date. The majority (52.3%) reported that 30 days was an appropriate time after the benchmark date to conduct an initial follow-up, 15% reported 90 days, and the remainder reported a wide variety of follow-up times.

For employment services, the majority (55.5%) reported that the first day of the new job was an appropriate benchmark date, while 19.5% cited the date that a job was accepted was an appropriate benchmark. The majority (53.3%) reported that 30 days was the appropriate time for a first formal follow-up. A small percent cited other benchmark dates. The most popular times for subsequent follow-ups were at 90 days (38.5%), 180 days (16.5%), and 365 days (11.9%).

Discussion

Currently no valid, consensus-based instruments exist with which to measure CRP employment outcomes. The absence of a measure makes the task of evaluating the nature and successes of employment achieved by consumers served by CRPs difficult. Furthermore, no previous studies have been identified in the literature that investigated what the principal stakeholders believe are the most important employment outcomes to pursue when assessing employment outcomes for persons with disabilities served by CRPs. The results of this research are an important first step in addressing this research problem.

Results of this study suggest that the three stakeholders identified as having vested interests in the employment outcomes resulting for CRPs (consumer, provider, and funder) are generally in consensus on the relative importance of the variables studied. Due to the large number of variables that the groups defined as important, a vital first step in subsequent research will be to reorganize the domains and content into an instrument suitable for use by the field. Also, subsequent research could include pilot and validation studies of the new instrument. Given the high rate of agreement, a subsequent research study that examines employment outcomes in relation to these important characteristics is also warranted.

The results indicate that the top concerns among all three groups were related to quality of employment, quality of services, and opportunity for informed choice. Contrary to previous research (Bell, Lysaker, & Milstein, 1996; Stodden & Browder, 1986), wage and benefit expectations were not top concerns. These findings must be interpreted in the context of the population served by CRPs. The low rankings associated with hours worked and earnings may be indicative of the severity of disability of CRP consumers. These findings appear to reflect the severe nature of the functional limitations inherent in the persons served by CRPs. Until a comprehensive study regarding employment outcomes of contemporary CRPs can be completed, it will be difficult to address the issue of the quality of outcomes achieved in relation to the expectancies of the persons served and their advocates, service providers, and funding agents.

Results of the questions pertaining to time frames indicate that there may be limited consensus regarding optimal time frames for outcome reporting. Further research is warranted in this area. It is of interest that there does seem to be support for long-term and open-ended time frames. Again, these findings may be indicative of the severity of disability of CRP consumers.

Limitations

This study included a relatively small number of subject participants. The authors recognize that with a sample size of 42 sets of data, it is possible that small effect sizes may have been present even when significant differences between the groups were not found. Additionally, other variables that could further define consumer status were not investigated (i.e., age, education, disability status), although an ardent effort was made to provide an adequate content validation of the extensive list of variables studied using a constituency advisory counsel. Finally, there may have been a potential for selection bias introduced by the method used to secure subjects. Although participating CRPs were randomly selected, there may have been bias introduced by the manner in which subjects were identified. To minimize the potential for selection bias, the executive directors, who were the recipients of the survey at each CRP, were asked to identify the most appropriate person in their organization to serve as the primary contact person. The primary contact was asked to identify as a subject, the last person who obtained community-based employment, if that person indeed was representative of the people served by the CRP. Due to the aforementioned limitations, future research is warranted investigating additional variables with a larger sample.

Conclusion

In order to build a solid foundation upon which to report the employment outcomes, an instrument is needed that assesses outcomes against standards relevant to employment services typically provided by CRPs and meaningful for people with disabilities, providers, and funders. An instrument is needed, based on consensus on what should be included in the broad picture of employment outcomes from CRP services if we are to validly compare alternatives with any confidence. The results of this study may be used to define the important characteristics to examine in a standardized measure of employment outcomes. This research has found consensus on the domains and content of variables to assess. Future research should develop and validate a standard means of collecting data on employment outcomes. It is hoped that such instrumentation will be flexible enough to accommodate individual community needs, yet structured enough to be applied in a variety of situations.

Notes

This research has been supported in part by the National Institute on Disability and Rehabilitation Research (NIDRR), U.S. Department of Education, grant # H 133B980040 Complete copies of the final report are available through the Research and Training Center, University of Wisconsin-Stout, Menomonie, WI 54751. The web site address is: http://www.rtc.uwstout.edu

References

Bell, M. D., Lysaker, P. H., & Milstein, R. M. (1996). Clinical benefits of paid work activity in schizophrenia. Schizophrenia Bulletin, 22(1), 51-67.

Berkowitz, M. (1985). Benefit cost analysis. In S. Moon, P. Goodhall, & P. Wehman (Eds.), Critical issues related to supported competitive employment: Proceedings from the first RRTC symposium on employment for citizens who are mentally retarded. Richmond: Virginia Commonwealth University, Rehabilitation Research and Training Center.

Bollingmo, G. (1997). Survey of employment services and vocational outcomes for individuals with mental retardation in Norway. Journal of Vocational Rehabilitation, 8, 269-283.

Bond, G., & McDonel, E. C. (1991). Vocational rehabilitation outcomes for persons with psychiatric disabilities: An update. Journal of Vocational Rehabilitation, 1(3), 9-20.

Butterworth, J., Whitney-Thomas, J., & Shaw, D. (1997). The changing role of community based instruction: Strategies for facilitating workplace supports. Journal of Vocational Rehabilitation, 8, 9-20.

Byboe, D., Mowbray, C., & McCroham, N. (1995). Towards zero inclusion in vocational services for persons with psychiatric disabilities: Predication of service receipt in a hybrid vocational case management service program. Psychosocial Rehabilitation Journal, 18(4), 74-93.

CARF The Rehabilitation Accreditation Commission. (1997). Managing outcomes: Customer-driven outcomes measurement & management systems. Tucson, AZ: Author.

Cifu, D. X., Keyser-Marcus, L., Lopez, E., Wehman, P., Kreutzer, J. S., Englander, J., & High, W. (1997, February). Acute predictors of successful return to work 1 year after traumatic brain injury: A multi-center analysis. Archives of Physical Medical Rehabilitation, 78, 125-131.

Cook, J. A., & Rosenberg, H. (1994). Predicting community employment among persons with psychiatric disability: A logistic regression analysis. Journal of Rehabilitation Administration, 18(1), 6-22.

Dean, D. H. (1991, Spring). Comparing employment-related outcomes of the vocational rehabilitation program using longitudinal earnings. American Rehabilitation, 5-9.

Elden, M. (1983). Participatory research at work. Journal of Occupational Behavioral, 4(1), 21-34.

Fabian, E. S. (1991). Using quality-of-life indicators in rehabilitation program evaluation. Rehabilitation Counseling Bulletin, 36(4), 344-356.

Felce, D., & Perry, J. (1995). Quality of life: Its definition and measurement. Research in Developmental Disabilities, 16(1), 51-74.

Gibbs, W. E. (1991, Spring). Vocational rehabilitation outcome measures: The probability of employment and the duration of periods of employment. American Rehabilitation 40, 10-13.

Hughes, C., & Scott, S. V. (1997). Teaching self-management in employment settings. Journal of Vocational Rehabilitation, 8, 43-53.

Inge, K. J., & Tilson, G. P. (1997). Ensuring support systems that work: Getting beyond the natural supports versus job coach controversy. Journal of Vocational Rehabilitation, 9, 133-142.

Inge, J., Banks, P. D., Wehman, P., Hill, J. W., & Shafer, M. S. (1988). Quality of life for individuals who are labeled mentally retarded: Evaluating competitive employment versus sheltered workshop employment. Education and Training in Mental Retardation, 23(2), 97-104.

Johnston, M. V. (1991). The outcome of community re-entry programs for brain injury survivors. Part 2: Independent living and productive activities. Brain Injury, 5, 155-168.

Johnston, M. V., & Granger, C. V. (1994). Outcome research in medical rehabilitation: A primer and introduction to a series. American Journal of Physical Medicine and Rehabilitation, 73, 296-303.

Johnston, M. V., Hall, K., Carnavale, G., & Boak, C. (1995). Functional assessment and outcome evaluation in TBI rehabilitation. In I. J. Horn & N. D. Zasler (Eds.), Medical rehabilitation of traumatic brain injury. Philadelphia, PA: Hanley & Belfus.

Kay, T. (1993). Selection and outcome criteria for community-based employment: Perspectives, methodological problems and options. In D. F. Thomas, F. E. Menz, & D. C. McAlees (Eds.), Community-based employment following traumatic brain injury. Menomonie: University of Wisconsin-Stout, Rehabilitation Research and Training Center.

Koch, L. C., & Merz, M. A. (1995). Assessing client satisfaction in vocational rehabilitation program evaluation: A review of instrumentation. Journal of Rehabilitation, 61(4), 24-30.

Kosciulek, J. (1999). The consumer-directed theory of empowerment. Rehabilitation Counseling Bulletin, 42(3), 196-213.

Kruse, D. (1997, September). Employment and disability: Characteristics of employed and non-employed people with disabilities. A report to the U. S. Department of Labor. Rutgers University: New Jersey.

Marshak, L. E., Bostick, D., & Turton, L. J. (1990). Closure outcomes for clients with psychiatric disabilities served by the vocational rehabilitation system. Rehabilitation Counseling Bulletin, 33(3), 247-250.

McAweeney, M. J., Forchheimer, M., & Tare, D. G. (1997, Summer). Improving outcome research in rehabilitation psychology: Some methodological recommendations. Rehabilitation Psychology, 42(2), 125-135.

Moseley, C. R. (1988). Job satisfaction research: Implications for supported employment. Journal of the Association for Persons With Severe Handicaps, 13, 211-219.

Rogers, E. S., Anthony, W. A., Cohen, M., & Davies, R. R. (1997). Prediction of vocational outcome based on clinical and demographic indicators among vocationally ready clients. Community Mental Health Journal, 33(2), 99-112.

Rumrill, P. D., Jr., & Garnette, M. R. (1997). Career adjustment via reasonable accommodations: The effects of an employee-empowerment intervention for people with disabilities. Work: A Journal of Prevention, Assessment & Rehabilitation, 9, 57-64.

Sands, D. J., Kozleski, E. B., & Goodwin, L. D. (1992). Quality of life for workers with developmental disabilities: Fact or fiction? Career Development for Exceptional Individuals, 15(2), 157-177.

Schwab, J., DiNitto, W. A., Simmons, J. F., & Smith, T. W., (1999). The dimensions of client satisfaction with rehabilitation services. Journal of Vocational Rehabilitation, 13, 183-194.

Stodden, R. A., & Browder, P. M. (1986). Community based competitive employment preparation of developmentally disabled persons: A program description and evaluations. Journal of Education and Training of the Mentally Retarded, 21(1), 43-53.

Thomas, D. F., & Menz, F. E. (1997). Validation of the vocational assessment protocol. Journal of Head Trauma Rehabilitation, 12(5), 72-87.

Zuckerman, D. (1993). Reasonable accommodations for people with mental illness under the ADA. Mental and Physical Disability Law Reporter, 17(3), 311-320.

Dale Thomas, Ph.D, ABPP, Senior Research Scientist, Research and Training Center, University of Wisconsin-Stout, Education and Human Services Building, 5th Floor West, Menomonie, WI 54751. Visit the web site: www.rtc.uwstout.edu