On the Congruence of Evaluation, Training, and Placement.

Given current national economic and health care reform forces, state and federal programs currently exist in a climate of accountability. Programs and services funded by the Rehabilitation Services Administration must demonstrate fiscal responsibility and be able to account for every dollar in this era of increasingly tight fiscal constraints. State VR agencies must manage programmatic responsibilities in such a way as to cut unnecessary spending and target already scarce resources to the areas where they are most needed (Queen, 1993). As such, the efficiency and effectiveness of services provided by state VR programs will continue to be subject to close public scrutiny.

Further, recent legislative mandates i.e., Americans with Disabilities Act (ADA) and 1992 Rehabilitation Act Amendments have placed the emphasis of rehabilitation services on client choice, empowerment, and involvement in the rehabilitation process. Combined, the ADA and 1992 Rehabilitation Act Amendments establish new policies; toward people with disabilities. Policies and programs are no longer to be based on pity and paternalism, but on respect for individual dignity; not on dependency and care-taking, but on independence and self-sufficiency (Richards, 1993). The implications of tighter fiscal constraints and a new national policy toward people with disabilities means that services provided by state agencies must be more efficient and accountable than ever. This combination of fiscal and legislative forces thus results in the necessity to evaluate the effectiveness of VR programs. The congruence between vocational evaluation, training, and job placement can be perceived as one aspect of case service provision important in evaluating VR program effectiveness and accountability.

The purpose of the current investigation was to assess the congruence between vocational evaluation (VE) job recommendations, vocational skills training, and jobs obtained by success, fully rehabilitated (status 26) state VR clients. By examining the efficiency of the entire continuum of vocational rehabilitation services, it was anticipated that the results of this study would provide clues about where to direct resources and VR process problem areas related to evaluation and training to address. It was also anticipated that by promoting the effective allocation of resources that study results would provide guidance for facilitating optimal employment outcomes among consumers of vocational rehabilitation services.

The following four research questions were developed to accomplish the objectives stated above:

1. What is the congruence between jobs obtained and vocational evaluation job recommendations for successfully rehabilitated VR clients?

2. What is the congruence between vocational skills training and vocational evaluation job recommendations for successfully rehabilitated VR clients?

3. What is the congruence between jobs obtained and vocational skills training for successfully rehabilitated VR clients?

4. What is the congruence between vocational evaluation job recommendations, vocational skills training, and jobs obtained for successfully rehabilitated VR clients?

Method

Subjects

The population of interest in this study was individuals served by one state-federal vocational rehabilitation agency. Subjects were selected from cases that were closed successfully rehabilitated (status 26) by the VR agency during FY 1993. In addition to a successful closure, the selection procedure required that former clients received a vocational evaluation and participated in specialized vocational skills training as part of their VR program. Work adjustment training and on-the-job training were not considered specialized training in this study. Seventy-eight of 320 successfully-closed cases met the criteria for inclusion in the final sample.

The client sample group represented urban, suburban, rural, and semi-rural regions of a state in the southeastern United States. Two-thirds of the cases were male. Over half were never married and one-fourth were married. Three-fourths of the sample group was White, and one-fourth was African-American. Subject disabilities were primarily orthopedic disabilities (35%), developmental disabilities (22%), or mental illness (17%). Primary disabilities for the remainder of the sample group were distributed across neurologic (9%), visual impairment/blindness (8%), hearing impairment/deafness (4%), chronic medical conditions (3%), and alcoholism/substance abuse (2%). The average education level of the participants was 11.6 years with a range from six to 18 years. Participant ages ranged from 18 to 53 with a mean of 30.4 years.

Procedure

Two sources were utilized to gather data for this investigation. First, cases were identified through the client information system of one state VR agency. Seventy-eight of 320 clients successfully rehabilitated during FY 1993 met the criteria for inclusion in the final sample (i.e., received a vocational evaluation and participated in specialized vocational skills training). The client information system was also used to collect participant demographic information. Second, closed client case files were reviewed to collect the data on subject vocational evaluation job recommendations, training area, and the job title of the position in which the subject was closed successful.

The relationships between vocational evaluation job recommendations, vocational skills training, and jobs obtained at closure were classified as congruent (hit) or incongruent (miss), using the first three digits of the occupational classification code of the Dictionary of Occupational Titles (5th ed.). The first three digits of the code indicate the occupational group in which a job falls. If the occupational code for VE job recommendations, training, or job obtained were the same, the two were considered congruent. All three codes were required to be the same for congruence to exist across evaluation recommendations, training, and jobs obtained. Levels of congruence (i.e., percentages of hits and misses) were calculated for each research question.

Results

The purpose of this study was to assess the congruence between vocational evaluation job recommendations, vocational skills training, and jobs obtained by successfully rehabilitated clients of one state VR agency. Results are described in terms of their relationship to each research question. In looking at the similarity between jobs obtained and vocational evaluation job recommendations (Research Question 1), 36 relationships (46.2%) were classified as hits and 42 relationships (53.8%) were classified as misses. This result shows that less than half of the subjects obtained jobs that were congruent with the jobs recommended in their vocational evaluation reports.

Examination of the similarity between vocational skills training and VE job recommendations (Research Question 2) indicated that 48 relationships (61.5%) were hits and 30 relationships (38.5%) were misses. This finding shows that six of every ten subjects entered vocational skills training programs congruent with the jobs recommended in their vocational evaluation reports.

Research Question 3 dealt with the congruence between jobs obtained by subjects and vocational skills training received. Thirty-nine (50%) of these relationships were classified as hits and 39 (50%) were classified as misses. This result shows that one-half of the subjects obtained jobs congruent with their vocational skills training.

Examination of the similarity between vocational evaluation job recommendations, vocational skills training, and jobs obtained (Research Question 4) indicated that 31 relationships (39.7%) were hits and 47 relationships (60.3%) were misses. This finding shows that there was congruence across the continuum of VR services (vocational evaluation-training-placement) for four of every ten subjects.

Discussion

The current investigation evaluated the efficiency and effectiveness of the vocational rehabilitation process by assessing the congruence between vocational evaluation job recommendations, vocational skills training, and jobs obtained by successfully rehabilitated clients. It was anticipated that the results would facilitate the effective allocation of state VR agency resources and provide data useful for increasing consistency in the rehabilitation process.

Prior to the discussion of results, a caveat regarding limitations should be noted. This caveat relates to external validity, or the degree to which the findings of this investigation can be generalized across persons, times, and settings (Parker, 1993). Given the nonprobability nature of the sample selection and cross-sectional data collection procedure, results cannot be generalized beyond the sample used at the time of the study.

Results indicate that many successfully rehabilitated clients were placed in occupational areas they were not trained for (e.g., trained in auto mechanics and placed in horticulture). Only one-half (50%) of the subjects obtained jobs congruent with the training they received. Thus, the vocational training offered did not seem to shape the vocational future of the subjects in a significant manner.

Less than half of the subjects obtained jobs that were congruent with the jobs recommended in their vocational evaluation reports. Further, training received matched vocational evaluation job recommendations for six of every ten subjects. Thus, as reflected in subject training areas and jobs obtained, vocational evaluation job recommendations were implemented on a limited basis.

The present study found a 40% congruence rate across the entire continuum of vocational rehabilitation services. That is, occupational groups of jobs recommended, training, and jobs obtained were the same for four of every ten subjects. This finding, combined with the congruence levels between evaluation and training (61.5%), evaluation and placement (46.2%), and training and placement (50%), indicates that there was as much dissimilarity as similarity between outcomes of evaluation, training, and placement services for this sample.

A limited amount of literature was available to compare the findings of the current investigation to previous research. The 50% congruence rate between training and placement found in this study is identical to that found by Chun and Growick (1983) for successfully rehabilitated clients served by another state agency. The finding that VE recommendations were followed approximately only 50% of the time is also consistent with findings by Caston and Watson (1990) and Cook (1978) which suggest that little relationship exists between vocational evaluation recommendations made and eventual rehabilitation outcomes. This result takes on added significance given that other researchers (e.g., Kosciulek, 1991; Marut & Bullis, 1985) have found that completion of VE recommendations relates favorably to positive employment outcomes.

No studies have examined the congruence across the entire continuum of VR services in the same client sample. In this study, congruence across evaluation, training, and placement existed for four of every ten participants (i.e., 40% of the time). This data thus provides an important addition to the literature on the efficiency and effectiveness of vocational rehabilitation services.

Whatever the reasons for the discrepancy rate between evaluation recommendations, training, and jobs obtained for subjects in this study, stricter accountability in the provision of VR services will be required as resources become more limited and the new national disability policy of inclusion, empowerment, and choice expands. The results of this study thus have two important implications for state VR agency service delivery. First, state VR personnel should be aware that one-half of the subjects obtained jobs congruent with the training they received. If similar training-placement congruency rates are detected in other states, a review and modification of the VR case service process in an effort to improve the consistency between training and jobs obtained would be warranted.

Second, vocational evaluation job recommendations were implemented approximately 50% of the time. This finding raises the question as to how vocational evaluations are being used by state VR counselors. If VE's are being used primarily as screening devices to aid in eligibility determination, specific job recommendations may not be the most important information desired by counselors. If the purpose of VE's, however, is to determine a specific vocational goal for use in Individualized Written Rehabilitation Plan development, specific job recommendations should be very useful to counselors.

One avenue for increasing the congruence levels between vocational evaluation and other phases of the rehabilitation process is to encourage more community-based vocational evaluations (i.e., geared toward actual training and jobs in local labor markets). Demanding higher levels of ecological (real-world) validity in vocational evaluations would facilitate an increased awareness of training and job opportunities among vocational evaluators.

The results of this investigation also provide direction for future research designed to improve the effectiveness of the VR process. First, replication of this study in other states would allow for direct comparison to determine if similar congruency rates exist in other regions of the country. A second study may involve qualitative follow-up (i.e., interviewing) of counselors and successfully rehabilitated clients to identify factors affecting congruency in the rehabilitation process (e.g., consumer choice options, labor market conditions, the counseling process, availability of funds and services). The results of these studies combined with the findings of the current investigation would provide data useful for improving the congruence between vocational evaluation, training, and placement. Such information would, in turn, assist in facilitating optimal employment outcomes among consumers of vocational rehabilitation services.

References

Caston, H. L., & Watson, A. L. (1990). Vocational Assessment and rehabilitation outcomes. Rehabilitation Counseling Bulletin, 34 (1), 61-66.

Chun, R. T., & Growick, B. S. (1983). On the con training and placement. Rehabilitation Counseling Bulletin, 27 (2), 113-116.

Cook, D. W. (1978). Effectiveness of vocational evaluation training recommendations. Vocational Evaluation and Work Adjustment Bulletin, II (3), 8-13.

Cook, D. W., & Brookings, J. B. (1980). The relationship of rehabilitation client vocational appraisal to training outcome and employment. Journal of Applied Rehabilitation Counseling, 11 (1), 32-35.

Kosciulek, J. F. (1991). The relationship between vocational evaluation recommendations and rehabilitation outcomes for persons with head injuries: An empirical analysis. In R. Fry (Ed.), The issues papers. Fifth national forum on issues in vocational assessment (pp. 277-283). Menomonie, WI: University of Wisconsin-Stout, Materials Development Center.

Marut, P., & Bullis, M. (1985). An analysis of the relationship of vocational evaluation recommendations to rehabilitation outcomes for deaf people. Vocational Evaluation and Work Adjustment Bulletin, 18(2), 66-70.

Parker, R. M. (1993). Threats to the validity of research [Editorial]. Rehabilitation Counseling Bulletin, 36, 130-138.

Queen, A. (1993). RSA's vision of progress and excellence. In R. N. Pacinelli & S. N. Gunn (Eds.), Renewing the commitment, meeting the challenge, and shaping the future (pp. 92-97). Proceedings of the 1993 NCRE, RSA, and CSAVR conference. Washington, D.C.: Rehabilitation Services Administration.

Richards, C. (1993). A new national policy: Understanding the relationship between the Americans with Disabilities Act and newly-revised Rehabilitation Act. Journal of Job Placement, 9 (3), 18-20.

Received: August 1994

Revision: November 1994

Acceptance: December 1994

John F. Kosciulek University of Missouri-Columbia

Linda A. Prozonic Georgia Division of Rehabilitation Services

Deborah Bell University of Georgia 

John F. Kosciulek, Rehabilitation Counseling Program, Department of Educational & Counseling Psychology, 16 Hill Hall, University of Missouri Columbia, Columbia, Missouri 65211.