Vocational rehabilitation outcome measures: the probability of employment and the duration of periods of employment.

W Ernest Gibbs, Ph.D.

This article reviews two labor market outcomes which can be used to evaluate the impact of vocational rehabilitation services. While there has been much analysis concerning the effect vocational rehabilitation services have on a client's earnings, two additional outcomes measures are: the probability of employment and the duration of periods of employment. Procedures and models are introduced which elaborate how to estimate these outcome measures and how to use them to evaluate the impact of intervention by the vocational rehabilitation program.

The federal/state vocational rehabilitation (VR) program provides services to people with disabilities to enable them to enter or reenter the labor force. The traditional measure of "success" of the program is placement in the competitive labor market.(1) In particular, there are three labor market outcomes which can be influenced by VR services. In turn, each of these outcomes can then be used to evaluate the success of the VR program. One effect is a change in earnings, conditional on employment after closure from the VR program. A second effect addresses a change in the employment probability. A third labor market result is a change in the expected length of period of employment.

To date, much of the analysis of the VR program has focused on the effect of VR services on the participants' expected earnings. Researchers have used cross sectional data to compute earnings gains and to estimate benefits of the VR program Bellante, 1972; Conley, 1969; and Worrall, 1978). More recently, Dean and Dolan (1986) have used longitudinal data to analyze the earnings effect of the VR program on its participants. Many of the models used to evaluate the program have been adapted from models which were designed to analyze manpower training programs.(2) However, these types of models might not fully measure the impacts of the VR program, which provides its clients with a variety of services beyond those provided by other manpower training programs.(3)

The VR program serves a clientele which is different in many ways than the clientele of the typical manpower training program. One difference between the clientele centers on the purpose of seeking assistance. It is possible that people who enroll into the VR program have different reasons for doing so, when compared to people who enter a training program. It is assumed that people enter training programs to acquire additional human capital-the productive skills, talent and knowledge which are directly related to a person's productive capacity. Thus, participants of training programs will invest in the optimal amount of human capital which will maximize their lifetime earnings (Ashenfelter, 1978). However, these assumptions may not be true for people with disabilities, who may enter the VR program not only to receive services which increase their human capital (i.e., vocational training), but also to receive other services directly related to their disabling condition (i.e., physical restoration).

Additionally, VR participants may have different labor market goals than the participants of manpower training programs. When searching for employment, VR participants may not necessarily choose the position which will maximize their potential earnings. Non-wage benefits may play a more important role in the employment decision. For instance, VR recipients may choose jobs which offer more job security, a better work environment or other favorable conditions. Traditional models which only examine wage or earnings of participants can not measure these types of benefits. Thus, to fully evaluate the VR program additional outcome measures need to be examined.

The goal of the VR program is rehabilitation--the placement or restoration of people who are disabled into the labor force. This goal seems to imply that the labor market outcomes which are related to whether the participant is employed should be of primary importance. These employment outcomes include the probability of being employed and the duration of periods of employment. The program would be successful in achieving its stated objective if receipt of VR services increased periods of employment or decreased the probability that an employed person would become unemployed at a later date. Given the fluctuations in the labor market and the fact that many people will lose their jobs, an examination of any ensuing period of unemployment may become necessary. Accordingly, the program would be successful if participation in the program decreased periods of unemployment or increased the probability that an unemployed person would find employment.

Study Design

Much of the benefit-cost analyses of the VR program conducted in studies have used a pre- and post-program type of evaluation design (Bellante, 1972; Conley, 1969; and Worran, 1978). However, many problems arose with this measurement. It is widely recognized that people who enter manpower training programs may suffer an earnings loss just before starting the program. This phenomenon is commonly known as "pre-program dip." This shift might be a temporary movement off the client's age/earnings profile, or it may reflect a permanent decline in earning power. For VR applicants this dip might occur partly because of the onset of a disability and they may never regain pre-onset earnings. This change also implies that the measure of benefit (the difference in earnings at referral and closure) will be sensitive to the choice of the time period chosen to measure pre-program earnings. Another problem is that many people (as many as 90 percent) report zero earnings at referral. A decision has to be made if this a true reflection of their human capital or if some level of earnings should be imputed for them.

These studies were all hampered because of the lack of a comparison group. The participant's wage gain could not be compared with what the participant or a similarly disabled person would have experienced without the intervention of VR.

A more favorable quasi-experimental design is the pre-treatment/post-treatment design with nonequivalent groups. In this design, information is obtained for a group of clients prior to and after the introduction of treatment and, also, for a comparison group of similar people who did not receive treatment. In recent years, several studies have examined the comparison group issue.(4) The tentative conclusion of these studies was that people who were accepted into the VR program but left before receiving any VR services would provide the most suitable comparison group for program evaluation.

Statistical Models

To evaluate the effect of VR services on periods of employment and employment probabilities requires statistical models. Survival or duration models, which were developed by researchers in the biomedical sciences and engineering fields, can be used for this analysis. Biostatisticians have used these models to examine treatment effectiveness and to analyze survival times resulting from laboratory studies of animals or from clinical studies of humans who have acute diseases.(5) Also examined was the conditional probability that a patient would die in a given period, hence the term, hazard rate. Engineers have used these models to observe the time to failure of manufactured items, such as mechanical or electronic components.

Social scientists have borrowed heavily from these models to explain many social phenomena. Survival models have been used to explain timing of births (Rodriguez and Hobcraft, 1984), migration (Chang, 1984), length of marriages, and child mortality (Trussell and Hammerslough, 1983). Economists have used these models to analyze labor market conditions (Fenn, 1981; Butler and Worrall, 1985).

The two outcome measures of interest in this paper can be described in the terminology of survival analysis. The survival function is defined as the probability that the person remains employed longer than a given time period. Estimation of this function will give the expected period of employment for a person. The hazard rate or hazard function is the conditional probability that a person leaves employment in a very short time period, given that the person remains employed until that time period. Estimation of this function will yield the expected probability of being employed.

There are two main approaches available to estimate these functions, one parametric, the other nonparametric. In the nonparametric approach, distribution-free techniques are used to estimate empirical survivor functions and hazard rates. Two commonly used nonparametric methods are the actuarial life-table estimator and the Kaplan-Meier product limit estimator. The estimators are quite similar; the main difference is that the actuarial estimator uses a fixed sequence of time intervals, while the product limit estimator uses individual survival times. The product-limit method is useful when the sample size is relatively small. When the sample size is large, say, over 100 observations, the life-table method may be more appropriate.

The nonparametric analysis provides a good estimator of empirical survival functions using observed data. However, this type of analysis is best suited for comparing homogeneous groups, where there is little or no observed or unobserved heterogeneity between groups. To determine the relationship between certain attributes and the survival times it is necessary to stratify the observations into cohorts or subgroups according to individual characteristics.(6) The survival function and hazard rate are then estimated separately for each cohort. With this univariate technique it is possible to examine the effects of observed characteristics on expected periods of employment and the expected probability of employment. However, it is not possible to control for unobserved differences among individuals or between groups in this method.

If a more sophisticated analysis is required or if there is a great deal of heterogeneity between the groups, it may be more appropriate to use parametric models. The first step in estimating these parametric models is to specify a likelihood function. This function gives the relation between the variable of interest (length of employment) and the other variables which are assumed to influence the employment time. In this manner, the model can allow for observed characteristics, such as age, race, gender, and education. Also a binary variable can be included to indicate whether the person received VR services. Next, full information maximum likelihood techniques are used to estimate values of the parameters which maximize the likelihood (probability) of observing the data.(7) These maximum likelihood methods make the parametric models more efficient than the nonparametric models, but the estimation of the parametric model is also more computationally burdensome.

Data Requirements

To estimate the parametric and nonparametric survival models requires event history data. Information concerning the demographic characteristics and the work histories of individuals in both the service and comparison groups is needed. The demographic variables required are those which are related to the labor market outcomes.(8) It is assumed that the labor market outcomes of individual clients will depend upon the characteristics that the individual brings to the program, such as age at referral, race, gender, and education. It is recognized that a person's employment outcome may also be related to level of health and degree of functioning. However, data on functioning is not readily available, but information pertaining to the disabling condition can be used in the analysis. Disabilities may be subdivided by type of disabling condition into three broad classifications-physical, mental and emotional.(9) This stratification may be necessary because the type of disability is a major factor in determining the type, duration and intensity of the treatment a client will receive.

Besides the demographic data, information concerning the work history of the client after leaving the program is required for the evaluation. It is necessary to follow the employment flows (i.e., movements between employment and unemployment) for people in both the service and treatment group. For survival analysis it is important to determine the length of periods of employment and unemployment. At the present time, this employment information is not recorded by the VR program, but it is possible to obtain this employment history from the client's employment insurance records.

Summary and Discussion

The VR program, which provides services to people with disabilities so they may obtain or regain employment, traditionally measures its success by the number of people it "rehabilitates." A client is classified as a rehabilitant (closure status 26) if he or she is employed for more than 60 days. However, the program usually does not follow rehabilitants for longer than the 60-day period.

The practice of not following clients after closure has hindered past efforts to evaluate the program. Analysts using a benefit-cost framework have examined the post-closure earnings of successful rehabilitants to estimate the benefits of the program. However, without the luxury of longitudinal data for participants, researchers were forced to make assumptions about the path of future earnings streams. Earnings projections or age/earnings profiles had to be constructed using only one period of earnings for each person. Recognizing that it is unrealistic to assume that all rehabilitants will remain employed for the rest of their working lives, the analysts had to adjust these estimated future earnings streams for the possibility of unemployment. For the most part, they used some constant rate of unemployment over the expected work life of the rehabilitant.

Some of these previous studies were also restricted by the lack of a control or comparison group. To determine the impact of the VR program on the outcome measures (earnings, employment durations, employment probabilities) of its participants requires knowledge of what these outcomes would have been if the participants had not received services. Since this impact cannot be estimated directly and since true control groups are not available, a comparison group can be used as a proxy for what the participants' outcomes would have been if they did not receive VR services. Many past studies did not use comparison groups, and those which did suffered from a lack of longitudinal earnings information.

Recent developments have made new and better data available and help solve some of the past problems of VR program evaluation. It is now feasible to link the VR program client information (R-300 data set) with employment insurance records. Clients can be followed for longer periods after closure and more information concerning the clients' pre-referral employment histories can be provided. This longitudinal information is available not only for the successful rehabilitants, but also for clients who were closed as "not rehabilitated." One group, those who were accepted into the program but did not receive any services (closure status 30), can serve as a viable comparison group for successful rehabilitants.

This longitudinal data has made it possible to use new techniques to examine the efficiency of the VR program. While previous efforts have concentrated on examining earnings, the focus of this paper is to describe the practicality of analyzing outcome measures related to the employment patterns of its participants. These additional outcome measures include 1) the duration of periods of employment and unemployment and 2) the conditional probability that a participant will be employed or unemployed.(10)

To further explore the expected effect of the VR program requires the use of parametric models. These models specify a likelihood function and assume that survival times follow a family of distributions. This multivariate analysis can allow for measurable variables other then the receipt of VR services, which may influence the duration of employment or unemployment.

The two alternative outcome measures-the probability of employment and the duration of periods of employment-which are outlined in this article give novel approaches for analyzing the impact of VR services. In many instances, these outcome measures may offer a truer yardstick for analyzing the effectiveness of a VR program. The statistical techniques diagramed in this article offer a procedure which could be used to measure the effect of VR services on the new alternative labor market outcomes. Analyzing these alternative measures may be important, since, in many cases, examining solely the relationship between VR services and earnings may understate the actual impact of VR services on clients' labor market experiences. To be able to most efficiently allocate the resources of a VR program, it is imperative that administrators receive as much information as possible on the different labor market outcome measures.

Notes

1. Other measures of success such as placement as a homemaker or placement in sheltered workshops are not examined in this article.

2. These programs include the Manpower Development and Training Act (MDTA), Comprehensive Employment and Training Act (CETA) and the Job Training and Partnership Act (JTPA).

3. In addition to vocational training and job placement, some additional services provided include diagnosis, counseling, and physical restoration (surgery).

4. See Dean (1991) for a complete discussion of the history of the comparison group issue.

5. See Lee (1980) for a good introduction to survival analysis and the methods introduced in this paper. Since this type analysis originally examined subjects with terminal diseases, much of the terminology remains in that context. For our analysis "life" begins with closure from the program and with the start of the employment spell. "Death" occurs when a person leaves employment.

6. People can be divided into service and comparison groups and according to age, race, gender, level of education, etc.

7. Many statistical packages (i.e., SAS, BDMP, Limdep) have programs which will estimate both the nonparametric and parametric models.

8. The demographic data can be obtained from the R-300 or the R-911 data reporting system.

9. The physical disability group includes people with visual impairment, hearing impairment, orthopedic deformity, and / or amputation of major or minor members. The emotional disability group includes those with mental, psychoneurotic and personality disorders. The mental disability group includes people classified as mentally retarded.

10. See Gibbs (1990) for an application of the techniques presented in this paper. He estimates both parametric and nonparametric survival models using data from the Virginia Department of Rehabilitative Services.

The University of Central Florida Division of In-house Grants program provided funding for research for this paper. Additional funding was provided by a McKnight Junior Faculty Fellowship.

Bibliography

1. Ashenfelter, O. (1978). Estimating the effect of training programs on earnings. The Review of Economics and Statistics, 60, 47-57.

2. Bellante, D. (1972). A Multivariate Analysis of a Vocational Rehabilitation Program. The Journal of Human Resources, 7, 266-241.

3. Berkowitz, M. (Ed.). (1987). Measuring the efficiency of public programs: Costs and benefits in vocational rehabilitation, Philadelphia, PA: Temple University Press.

4. Butler, R. J., & Worrall, J. (1985). Work injury compensation and the duration of nonwork spells. Economic Journal, 95 (7),14-724.

5. Chang, C. (1984). A hazard rate analysis of family migration by using longitudinal duration data. mimeo. Rutgers University.

6. Conley, R. (1969). A benefit-cost analysis of the vocational rehabilitation program. The Journal of Human Resources, 4. 226-252.

7. Dean, D. (1991, Spring). Comparing employment-related outcomes of the vocational rehabilitation program using longitudinal earnings. American Rehabilitation, pp. 4-9.

8. Dean, D., & R. Dolan, (1986) Estimating earnings impacts of the vocational rehabilitation program. Unpublished manuscript.

9. Fenn, P. (1981). Sickness duration, residual disability, and income replacement: An empirical analysis. The Economic Journal, 91, 158-173.

10. Gibbs, W. E. (1990). Alternative measure to evaluate the impact of vocational rehabilitative services. Rehabilitation Counseling Bulletin,. 34, 33-43.

11. Lee, E. (1980). Statistical methods for survival data analysis. Belmont, CA: Lifetime Learning Publications.

12. Rodriguez, G., & Hobcraft J. (1980). Illustrative analysis: Life-table analysis of birth intervals in Colombia. Scientific Reports (No. 16). London: World Fertility Survey.

13. Trussell, J., & Hammerslough, C. (1983). A hazards-model analysis of the covariates of infant and child mortality in Sri Lanka. Demography. 20, 1-26.

14. Worrall, J. (1978). A benefit-cost analysis for the vocational rehabilitation program. The Journal of Human Resources. 13, 285-298.