Developing a screening tool to predict disability program participation.

The ability to model public benefits eligibility processes can aid in program planning, resource allocation, and budgeting. Given the complexities of most public program regulations, policies, and practices, developing a simple and accurate model of eligibility for public benefits is no simple task. The success of a demonstration project currently being developed by the federal government will depend largely on the ability to construct such a model. The United States Social Security Administration (SSA) is designing a demonstration project that will promote employment among applicants for Social Security Disability Insurance (SSDI). The project, Early Intervention (EI), is one of a series of ventures that SSA is undertaking that essentially challenge the established norms and practices of the organization toward programs for persons with disabilities (Mitra & Brucker, 2004). This paper will describe one of the methodologies that will be used to select participants for El. Prior to discussing the conceptual framework for and the details of the methodology, some background information on SSDI and the disability determination process is provided.

Social Security Disability Insurance

SSDI provides cash benefits and health care coverage to persons with work-preventing disabilities and their families. As of June 2005, average monthly payments to disabled workers were $897 (SSA, 2005c). To qualify for benefits, individuals must meet stringent eligibility conditions set by SSA, the federal agency responsible for administering SSDI. Applicants must first satisfy employment history thresholds, as SSDI was intended to only provide support to persons who have had a substantial attachment to the labor force. Certain disability eligibility criteria must be satisfied as well. SSA defines disability as an inability to participate in a certain level of employment due to a medical condition that is expected to last for at least 12 months or result in death. Individuals who are blind, over the age of 55 and unable to engage in certain levels of work activity are also eligible to receive benefits. Once awarded benefits, beneficiaries can expect to receive benefits until they either medically improve, return to work at a substantial level, transition to the retirement rolls, or die. On average, beneficiaries remain on the rolls for about eleven years (Chirikos, 1995).

SSA administered SSDI to over 6.1 million disabled workers in December 2004, paying out over $71 billion in cash benefits. These figures reflect an increase of 23% in the number of beneficiaries and an increase of 44% in benefit payments over the five-year period from 2000 to 2004 (SSA, 2005a, 2005b). Given that this rate of growth has been predicted to remain fairly steady (Daub, 2002), SSA has made a concerted effort to develop return to work programs and incentives that will increase the rate at which people exit the program. Thus far, however, the return to work rate of beneficiaries has remained low, at less than 1% (SSA, 2002).

A complementary approach to encouraging program exit is to develop a method for slowing the rate of entry to SSDI. SSA is currently designing a demonstration project that would promote employment as an alternative to entering the SSDI rolls. The Ticket to Work and Work Incentives Improvement Act (TWWIIA) of 1999 gives SSA the authority, for the first time, to provide return to work services and supports to applicants who meet the SSA definition of disability. The El demonstration is designed to test whether the early provision of temporary cash stipends, health care coverage, and employment services can efficiently return probable SSDI beneficiaries to work. The combination of immediate services and supports, coupled with an avoidance of the long, arduous disability determination process that is normally involved in securing SSDI benefits, may result in improved return to work rates for this population (Social Security Advisory Board, 2001a). A key to this approach is to develop a quick and accurate way of determining who, among all applicants, is likely to be approved for benefits.

Disability, Determination

To understand the challenges in developing an effective screening tool, it is important to gain an appreciation of the complexity of the current disability determination process. SSA determines eligibility for both SSDI and Supplemental Security Income (SSI), a means-tested income assistance program for persons with disabilities, in a series of five stages (SSA, 2002). A person interested in applying for disability benefits typically first contacts the local SSA field office, by phone or in person, and completes an application. SSA claims representatives perform the first stage of review, verifying that the applicant meets certain non-medical criteria, including the accrual of the requisite employment history and an absence of a certain level of employment. Ira person meets these two requirements, the application is sent to the Disability Determination Services (DDS) office, a state-specific entity responsible for assessing disability. If, at this first point of contact, claims representatives identify individuals with certain severe disability conditions, such as heart or liver transplant candidates, DDS can expedite the determination process.

The next four stages of the benefit eligibility process involve decisions by DDS. Claims representatives at the second stage assess the severity of applicants' medical conditions. Persons without a condition that interferes with basic work activities are denied benefits, while those with such conditions undergo further assessment. At the third stage, applicants are evaluated regarding the type of disabling medical condition. Applicants with a condition that either meets SSA specific listings of impairments or that is equally severe are allowed SSDI benefits. Otherwise, applications proceed to the next stage. The final two stages evaluate employment capacity. For the fourth stage, applicants considered eligible to perform their previous employment are denied. At the fifth stage, applicants who are not able to perform any type of employment in the national economy are allowed, while those deemed able to perform such work are denied SSDI benefits.

Applicants who are not awarded benefits through this initial disability determination process may pursue an appeal. Denied applicants have four appellate opportunities, starting with reconsideration at the DDS level and ending at the federal court. The average time involved for a DDS decision is 106 days, while for the appellate process, the time for a decision can range from 95 to 477 days, depending on the level of appeal (SSA, 2003a). In contrast, the EI demonstration needs to estimate the probability of benefit receipt the first day an applicant comes into contact with SSA.

Persons who are detached from the labor force for an extended period of time have been found to have a more difficult transition back to employment (Palmer, 1990). To achieve the highest level of success, El processes must support the identification of suitable participants at the earliest point of contact with SSA. While the thoroughness of the usual SSA disability determination process provides some obvious advantages in terms of accuracy, the wait that the process requires imposes a delay in return to work efforts.

Screening Tool Requirements

Certain real world constraints affect the development of a screening tool for the El demonstration. In designing the screening process, care must be taken to minimize changes to the usual flow of SSA field office business in order to avoid imposing greater burdens on the claims representatives and the timeliness of application processing. The El screening tool should have the capacity to be applied quickly by SSA claims representatives, using claimant reported information. Claimant reporting is necessary because no confirmation for either the condition or its severity will be gathered from medical professionals during this initial screening. The screening tool is limited to a small number of variables, preferably ones that are normally collected by SSA in the first phase of the SSDI disability determination process, so that claims representatives will not have to spend excessive time on the screening task. Despite these time and resource constraints, the screening tool must be able to assess applicants reliably and validly. Previous research on screening tools and their effectiveness can help to inform the development of the El screening tool.

As Waddell. Burton, and Main (2003) point out, screening tools have inherent limitations. The ability to identify persons likely to move to a long-term disability program may be condition specific or may depend on the timing of the screening instrument. Significant external events, such as unemployment rates and labor market conditions, may be difficult to model. Another source of error rests in the use of self-report measures to identify an individual's eligibility for a program, thereby giving an applicant the capacity to bias the screen towards acceptance.

The SSDI eligibility process is difficult to model for several reasons. As evidenced from the previous discussion, the SSA disability determination process has multiple stages and is dynamic, changing from year to year based on programmatic considerations and court decisions. Despite the process being a uniform system, the application or interpretation of determination policies and procedures is both region and state specific. For example, in 2000, state-level DDS approval rates for SSDI ranged from 31% to 65% (Social Security Advisory Board, 2001b). State residence has been identified by others as a variable that may affect the probability of SSDI receipt (Hu, Lahiri, Vaughan, & Wixon, 1997; Kreider, 1999).

The adjudication process further complicates the ability to model SSDI. Some denied individuals are more tenacious in appealing initial disability decisions. For instance, during the fiscal year 2002, of every 62 persons denied at the initial stage of determination, 22 had reconsiderations, another 19 appealed to the hearings level, and 5 submitted their cases to the appeals council review (SSA, 2003b). The approximate success rate at each of those levels was 14%, 63%, and 20%, respectively. The success rates may reflect the presentation of additional data or other factors rather than a deviation from the determination model used by SSA.

In sum, a number of constraints, including the realities of administering a screening tool in a public agency, the reliance on self-report data, the limitations inherent in screening tools, and the complexities of the eligibility process, provide challenges to the development of an effective screening tool.

Previous SSDI Prediction Models

A review of previously developed statistical models of the SSDI process can guide variable selection for the current analysis. Modeling the SSDI application and decision processes is not a new initiative. Rather than screening applicants for selection into a specific return to work program, however, the focus has previously been on developing models of the application and decision process or on evaluating policy options for the administration of SSDI. Data from national surveys, sometimes in tandem with SSA administrative data, has generally been used to examine those individuals who apply for and receive SSDI during a specified period. The advantage of using national data sets is the presence of variables that are not included in SSA administrative data.

Kreider (1999) examined the impact of SSDI on male labor force participation and the opportunity costs of applying for and receiving SSDI. He found that receipt of benefits was related to age, disability level, regional considerations, and being married at the onset of the disability. Another study focused on older working adults and the probability of SSDI acceptance (Kreider & Riphahn, 2000). Significant variables for SSDI receipt among males at either stage, initial application or appeal, included age at application, race, education, the proportion of applicants accepted in a region, residing in the northeast, the year of application, having a greater number of children, poor sight, and back or heart conditions. Among women, factors that affected SSDI receipt included the age at application, the proportion of applicants accepted in a region, marital status, the year of application, illnesses related to kidney, bladder, stomach, or ulcers, having a psychological condition, poor sight, and the number of children. A similar type of analysis focused on those with mental disorders (Bilder & Mechanic, 2003). Significant variables that influenced SSDI receipt among applicants included education, instrumental activities of daily living, restricted activity days, being unable to work, and having another SSDI or SSI recipient in the household.

Alternative modeling approaches have emphasized multistage modeling techniques (e.g., Dwyer, Hu, Vaughan, & Wixon, 2001; Hu et al., 1997; Lahiri, Vaughan, & Wixon, 1995). Hu et al., for example, used a four-stage sequential model with different factors considered at each stage, including over 50 variables to approximate the DDS decision process. The equations accounted for 16% to 46% of the variance of the disability decision at each stage, with good predictive validity in the resulting equations. When using only one equation for the disability decision, the model accounted for only 18% of the variance; the predicted allowance rate was also lower than the rate in the multi-stage equations. The development of just one equation eliminated certain factors, such as education and functional limitations, that play an important role within the SSA disability decision process and in fact were significant in the sequential equations.

Using a similar process, Benitez-Silva, Buchinsky, Chan, Rust, and Sheidvasser (1999) examined the application, receipt, and appeals process and developed two models: one that involved factors related to the four-step decision model for the first decision and a separate model for the appeals process. For both models, the variable with the strongest influence was a dichotomous measure that indicated a health limitation that prevented work. For the four-step decision model, they attempted to include variables at each step that were relevant to the SSA decision process. For instance, a model of the first step, being able to engage in substantial employment activity, included net wealth, total hours worked, average hours worked per week for a month period after application, previous SSDI or SSI applications, and earnings in the preceding year. Overall, significant variables included hours worked per week after application, previous applications to SSDI/SSI, having a stroke, the age at application, and the year of application.

An alternative approach in considering the decision process is to look at the differences between successful and unsuccessful SSDI applicants. Unsuccessful applicants have been found more likely to be female, younger, non-white, married, and educated (Kennedy, Olney, Richer, & Newsom, 2002). Health-related variables, such as functional limitations, activities of daily living, and health severity, were also found to be more severe for successful applicants than for unsuccessful applicants. However, this study compared the two groups alter the application process had occurred.

Several issues related to the above research limit the applicability of the models to the current problem of identifying SSDI recipients from the applicant pool. First, SSDI and SSI program involvement are often combined. While the disability determination process is the same for both programs and applicants may be considered or screened for both programs, practical differences may exist between those who apply for one versus the other. The El demonstration will reject those who apply for SSI, even if they are also applying for SSDI. Second, the number of variables used in the predictive equations is quite large. Such a large number is problematic when considering that an El screening procedure must be applied quickly and easily by claims representatives. The need for a brief screen comprised of only a few variables prohibits the use of a complicated multistage model. One must also assume that claims representatives will gather information differently from that obtained for public surveys. While data survey experts are trained to be fontal and uniform in their collection of data, claims representatives may prompt, direct, and clarify responses from applicants to facilitate the case. Another issue with public surveys is that several variables that were identified as significant, such as the year of application, are impossible to include in a useable predictive model for SSDI applicants. Finally, in addition to those persons who were awarded benefits at the DDS level, a screening tool must also predict which applicants would successfully navigate the adjudication process.

Method

Sample

Developing a model to predict SSDI benefit receipt requires both application information and benefit decision information. The primary source of data for this analysis was a set of 1996 disability benefits application case files collected from three SSA field offices (New Brunswick, New Jersey; Portland, Oregon; and Cheyenne, Wyoming). Applicant files were selected based primarily on administrative accessibility, not random selection criteria. The data contained no personal identifiers such as the applicant's Social Security number, name, or address. In total, information from 548 different applicants from the three offices was obtained, three cases of which were dropped because they were 65 years of age and therefore not eligible for SSDI. Data from the application files were entered into a database and linked, by SSA, with SSA administrative data containing benefit decision information. The case files that were collected from the field offices included a larger than normal percentage of cases that went on to receive approval for benefits. The data set had an applicant approval rate of 65%, far higher than the national 1996 average approval rate of 49% (SSA, 2003a), though this latter number may not include approvals through the appeals process.

The data included demographic, medical, education, and employment history information derived from SSA administrative records (SSA 831 electronic data file) and forms (SSA forms 16, 3367, 3368, and 3369). Some differences existed in the data available among regions, as some regions had adapted certain forms or collected additional information. Only information that was common to all regions was used for this analysis.

Measures

Variable selection was guided by previous research results and by the information available in the application files. The following independent variables were considered for inclusion in the model: age, education level, marital status, average annual earnings, mental and physical limitations, whether or not the onset of the disability coincided with work cessation, and the body systems affected by the disabling condition. All of these variables met two important criteria: they were available through self-report from the applicant and they were available as part of the normal application information collection process.

Rather than using a continuous variable, age was divided into five dichotomous variables based on SSA classifications: less than 45 years of age, between 45 and 49 years, between 50 and 54 years, between 55 and 59 years, and 60 years and over. Education level was represented as three dichotomous variables describing the highest level achieved: less than high school, high school degree or equivalent, or schooling beyond the high school level. The SSA application form was used to identify marital status. About half (48%) of the applicant sample was married.

Annual earnings information was limited to SSA generated earnings categories. SSA calculates earnings categories by excluding earnings from the year of application and the year prior to application, on the assumption that the illness or injury could negatively affect such income, and by averaging earnings from the previous five years. For example, for a person applying in 1996, income from 1995 and 1996 was excluded and income from 1990 to 1994 was included. This average was compared to the national average of earnings during the years considered. Earnings were placed into one of five dichotomous variables based on SSA classifications: marginal (equal to or less than 2080 (the number of hours of full-time employment in one year) multiplied by the minimum wage, or less than $10,712), low (more than marginal and up to 75% of the national average, or between $10,713 and $20,606), average (between 75% and 125% of the national average, or between $20,607 and $34,344), high (between 125% and 200% of the national average, or between $34,345 and $54,951), and very high (over 200% of the national average, or over $54,951). The largest proportion (46%) of applicants in this sample had a marginal level of earnings.

Limitations for activities of daily living, or functional limitations, are often used to categorize the extent of a disability. Ten categories of functional limitations were included in the SSA application data: reading, writing, answering, hearing, sitting, understanding, using hands, breathing, seeing, and walking1. Claims representatives were asked to note whether the applicant exhibited any of the above listed limitations. Because of the quality of the data (up to 55% of the individual limitations sections had missing information), dichotomous variables were used to indicate the presence or absence of a functional limitation along two dimensions: physical and mental limitations. Physical limitations include limitations with hearing, sitting, using hands, breathing, seeing, or walking, while mental limitations involve problems with reading, writing, answering, or understanding. Only 12% of the sample (21% of valid responses) had a record of a physical limitation and 10% (16% of valid responses) had a record of a mental limitation.

The data included information on when a person stopped working after their illness or injury began. This information was coded into a dichotomous variable to indicate whether the applicant had stopped work at the same time as the onset of the disability. Such information identifies an acute condition, causing immediate work cessation, rather than a chronic condition, where there is a sufficient lag in the time from the date of onset until the impairment led to the discontinuation of work. More than one-fourth (28%) of the applicants stopped work with the onset of their condition.

SSA categorizes impairments into one of fourteen different body systems. Although the data set included body systems as identified by a medical professional, SSA staff will not have access to full medical evidence or records during the EI screening process, and so the applicant's self-reported disability was used to assess the relevant body system. Applicants could list up to two types of affected body systems. Rather than use the first reported body system, primary and secondary systems were both considered. For analytical purposes, digestive, genito-urinary, hemic/lymphatic, multiple body systems, and skin body systems were combined into an "other" category because of the minimal reporting of these conditions. The body systems with the highest prevalence in the sample were musculoskeletal (41%), mental disorders (20%), and cardiovascular (15%).

As indicated above, the administrative data was incomplete for many cases. Because common methods of missing data imputation (such as complete case analysis, mean imputation, or hot-deck imputation) result in bias, a multiple imputation method was used (Allison, 2001). Multiple imputation is being increasingly employed in health and economic studies to account for missing data and has been demonstrated to be the best choice for imputation (e.g., Barzi & Woodward, 2004; Briggs, Clark, Wolstenholme, & Clarke, 2003; Ellis, Shannon, Cox, Aiken, & Fowler, 2004; Gu & Yi, 2004; Patrician, 2002). While this statistical method uses iterative algorithms to predict the missing value, its advantage is that it inserts a random element or uncertainty into the imputed value. Multiple imputed data sets were created, analyzed separately, and the results averaged into a final set of parameters using PROC MI and MIANALYZE procedures in SAS version 8.2.

Analysis

Those cases that met terminal or presumptive disability criteria were identified based on the applicant-reported condition. Terminal cases were those in which it was assumed that the illness or injury results in death, the person had a diagnosis of AIDS, or the applicant was receiving hospice care. Presumptive disability cases were those in which an applicant had an impairment considered untreatable. Examples of presumptive conditions include waiting for a heart or liver transplant, chronic pulmonary or heart failure, or being comatose for 30 days or more. For these cases, rather than including them in modeling SSDI receipt, an assumption was made that they would automatically be screened as eligible. These cases were therefore excluded from the initial modeling process.

After the terminal and presumptive cases were removed, a cross-validation technique was adapted from Pedhazur (1982), dividing the data into two groups: one to create the models (a scoring sample) and one to test the models (a validation sample). Eighty percent of the cases were placed in the scoring sample, with the remaining 20% used for validation. The validation sample also included a random sample of 20% of the terminal and presumptive cases, thereby simulating the actual applicant pool.

Three models for predicting SSDI receipt were developed using the modeling sample. The dependent variable for all models was a dichotomous variable of benefit allowance. Logistic regression analysis was used for two of the models. Whereas linear regression is more appropriate when the dependent variable is continuous, logistic regression is appropriate when the dependent variable is dichotomous. Logistic regression can be used to classify individuals into one of two populations, in this case, those who receive benefits and those who do not. The results provide a predicted probability that an individual belongs to a specified group (Afifi & Clark, 1997; Long, 1997). The first model used logistic regression and included all of the independent variables described above in the measures section. A second logistic regression model was based on variables that were significant in the first model at the p < .10 level. For the third model, a sum of scores scale was developed where each of the categorical variables was assigned values from 0 to 4 relative to the degree that individuals with the value obtained SSDI benefits (see, for example, Bellini, Neath, and Bolton, 1995). The advantage of this method is that the scale is easy to use and simple to interpret.

For each of the models, the validation sample was used to assess accuracy, and classification tables were developed based on predicted and actual receipt of SSDI. Comparison of the models was based on five measures that are commonly used to assess the accuracy of a screening tool (Howell, 1997; Lemeshow, Teres, Avrunin, & Pastides, 1988): sensitivity (the correct classification of those persons who received SSDI), specificity (the correct classification of those people who were denied SSDI benefits), predictive value of acceptance (the proportion of applicants predicted to receive SSDI who were accepted), predictive value of denial (the proportion of applicants predicted to be denied who were denied), and efficiency (the total proportion of applicants who were correctly classified). In addition, the proportion of SSDI recipients allowed by the model was calculated, as well as the phi coefficient, a measure of association for dichotomous variables. All analyses used SAS version 8.2.

Results

Table 1 shows the distribution of the sample applicants by variable, as well as the proportion of denials and allowances. The sample included a large portion of beneficiaries who were young, had a high school education, had marginal earnings, and did not have physical or mental limitations. Most of the sample did not stop work with the onset of illness. Musculoskeletal and mental disorders were the most commonly listed disorders. Applicants who received SSDI benefits were more likely to be aged 55 to 59, have more than a high school education, be married, have average or higher earnings, and have physical or mental limitations. Persons who experienced more acute conditions and were forced to stop working at the same time their disability started were less likely to receive SSDI. Applicants with a condition affecting the cardiovascular system had high levels of SSDI receipt, while applicants with musculoskeletal and mental disorders had lower levels.

In Table 2, the coefficients and standard errors of the logistic regression equations are presented. With Model 1, all variables were included on the presumption that, though they were not significant, they had been shown by previous research to be related to the application decision. Variables that were significant at the p < 0.10 level included having a mental limitation, low to high earnings, being between 55 and 59 years of age, having a condition whose onset did not coincide with leaving employment, and having a sense related impairment. Those variables formed the basis for the second model.

The variables and their values are listed for the sum of scores model in Table 3. Values for each variable were based on the ability of each variable to discriminate between SSDI acceptance and denial. Categories that related to a greater denial rate were coded as zero, while items related to acceptance rates were coded from one to four, depending upon the magnitude of the effect. For instance, the two variables assigned the highest codes, mental limitations and age between 55 and 59, had an allowance rate twice their denial rate. In contrast, being married was only a marginal indicator for receipt of SSDI, and so that variable carried a weight of two. Applicants within the modeling sample received scores ranging from one to 23.

To contrast the models, the actual rates of SSDI receipt were compared with the predicted rates. For Models 1 and 2, a probability of benefit receipt was calculated based on the logistic equation developed for each model. Though 0.50 is generally used as a cutoff for logistic models (e.g., Dwyer et al., 2001; Lemeshow et al., 1988), to obtain a more restrictive model, a probability of 0.60 or above was used as indicative of SSDI receipt. For Model 3, the sum of scores model, the rates of acceptance and denial for each score were examined, using as the criterion value the score above which applicants moved to the SSDI rolls in greater proportion than the general sample. Persons scoring eight or above were more likely to be allowed benefits.

Table 4 compares the three models on the ability to classify applicants on both the scoring and the validation samples. The acceptance rate, the proportion predicted to receive SSDI, ranged from 55% to 60% using the scoring sample and from 60% to 65% using the validation sample. Model 1, the logistic model which included all Variables, was not as accurate as either Model 2, the logistic model with significant variables, or Model 3, the sum of scores model. In both the scoring and validation samples, Model 2 had the highest acceptance rate and sensitivity (the ability to identify successful SSDI applicants), with a slight advantage for the predictive value of denial (i.e., the proportion of individuals who were predicted to be denied and were actually denied). Model 3 had the highest specificity (the ability to identify denied SSDI applicants) and predictive value of acceptance (i.e., the proportion of individuals who were predicted to be accepted and were actually accepted). Model 3 also had a slightly better phi coefficient, though the effect size was small in both samples. Both Models 2 and 3 were equally efficient, correctly predicting the SSDI outcome of 68% of the scoring sample and 62% of the validation sample.

All models were less accurate with the validation sample than the scoring sample. Though sensitivity was similar and predictive value of denial slightly better, other comparison measures decreased.

Discussion

The ability to model a benefits determination process is critical for many federal and state programs. With any model, accuracy is compromised by using simple formulae to model a complex eligibility process. When the process modeled involves disability, a concept with inherent difficulties in its definition, the issue becomes even more problematic. Despite these difficulties, two of the models developed here can reasonably identify those persons who would qualify for or be denied SSDI benefits. The sum of scores model was slightly better in predicting denied applicants, while the logistic model was slightly better in predicting successful applicants.

The legislation that authorized the El demonstration states that services can only be provided to applicants with impairments that may reasonably be presumed to be disabling (TWWIIA, 1999, Section 301). Each of the models discussed was successful about two-thirds of the time in predicting which applicants were awarded benefits. Accuracy for predicting who was denied benefits was not as high. The demonstration is intended to "do no harm," and persons who are inaccurately screened out will be able to continue pursuing SSDI benefits in the usual manner.

The implementation of any of the proposed models will allow SSA to assess, with a reasonable level of accuracy and in a relatively efficient manner, the likelihood of an applicant receiving benefits. Given the constraints on available data and the realities related to implementing this tool in a real world environment, the models developed here have obvious limitations. The data used to develop the models may not be representative of the SSDI applicant pool. As mentioned earlier, the interpretation of the disability determination decision processes may vary from region to region and such fluctuations could have influenced the data. Furthermore, the data were skewed towards applicants who had been approved for benefits. The amount of missing data, despite using an imputation approach, may also have influenced the models and the ability to predict awards and denials.

As a tool to identify SSDI recipients, the selection cut-off point is itself flexible and can be adjusted to meet the policy needs and objectives of the demonstration. This point is illustrated using the restricted logistic model (Model 2) on the validation sample. Using 0.60 as the probability selection cutoff, we could anticipate that of every 100 people in this sample who apply, 66 would be identified as likely to receive SSDI. However, based on the sensitivity and an actual acceptance rate of 54% (SSA, 2003b), of those 66 persons who would be flagged, 41 would actually receive SSDI and 25 would not, though they would be offered access to the EI program. This latter group represents the false positives or deadweight, persons who incorrectly receive the intervention.

The selection screen can be tailored to include a large number of individuals (emphasizing sensitivity) or minimize the false positive rate (increasing specificity), either by lowering or raising the cut-off point, respectively, but it is difficult for a screening tool to have both high sensitivity and specificity (Waddell, Burton, and Main, 2003). Projects or programs using a screen must decide which path to take. Continuing with the illustration, we could use a higher cut point (e.g., taking the 90th percentile probability score from the model sample for those who are denied benefits, or 0.7577) to reduce the deadweight. This criterion would result in values for sensitivity, specificity, predicted value of acceptance, and predicted value of denial for the restricted logistic model of 32%, 75%, 63%, and 45%, respectively, with a screening acceptance rate of 29%. As the example shows, the ability to predict those who will receive SSDI declines while the ability to predict who is denied increases, with no change in the predictive value of acceptance. Of 100 people who apply, 29 would be identified by the screening tool as SSDI candidates, of whom 17 would actually receive SSDI benefits and 12 would be false positives. While this change would decrease the deadweight, fewer actual beneficiaries would be offered services that could help them.

SSA will have the unique opportunity, through the EI process demonstration, to evaluate the screening tools on key criteria related to accuracy, efficiency, and equity. SSA will also have the ability to design a program that will be cost effective, no matter which criteria are used to select people into the program. Program cost forecasts can include estimates of deadweight so that program expenditures can be adjusted accordingly. The successful identification of probable beneficiaries is but one factor that will influence overall program costs-benefit ratios.

During the planned EI process evaluation, data will be collected to determine if the selection model correctly identifies SSDI beneficiaries, and the same data will be used to correct and improve the model. A significant aspect of the EI program evaluation process is whether or not a model identifies SSDI beneficiaries accurately, or if it excludes a portion or particular subset of beneficiaries. Finally, SSA will observe the use of a screening process in SSA field offices, thereby assessing the effects of its administration and how such a process affects the claims representative workload.

Author Note

Todd C. Honeycutt and Debra Brucker completed initial drafts of this article while working for the Program for Disability Research at Rutgers, the State University of New Jersey.

The work presented here was performed pursuant to a grant from the U.S. Social Security Administration (SSA) funded as part of the Disability Research Institute. The opinions and conclusions expressed are solely those of the authors and should not be construed as representing the opinions or policy of SSA or any agency of the Federal Government.

The authors wish to thank Monroe Berkowitz, John Hennessey, Sophie Mitra, Brett O'Hara, Joann Sim, Robert Weathers, and two anonymous reviewers for their assistance.

References

Afifi, A. A., & Clark, V. (1997). Computer-aided multivariate analysis. Boca Raton, FL: CRC Press.

Allison, P. (2001). Missing data. Thousand Oaks, CA: Sage.

Barzi, F., & Woodward, M. (2004). Imputations of missing values in practice: Results from imputations of serum cholesterol in 28 cohort studies. American Journal of Epidemiology, 160, 34-45.

Bellini, J., Neath, J., & Bolton, B. (1995). Development of a scale of social disadvantage for vocational rehabilitation. Journal of Rehabilitation Administration, 19(2), 107-118.

Benitez-Silva, H., Buchinsky, M., Chan, H. M., Rust, J., & Sheidvasser, S. (1999). An empirical analysis of the Social Security Disability application, appeal, and award process. Labour Economics, 6, 147-178.

Bilder, S., & Mechanic, D. (2003). Navigating the disability process: Persons with mental disorders applying for and receiving disability benefits. Milbank Quarterly, 81(1), 751-06.

Briggs, A., Clark, T., Wolstenholme, J., & Clarke, P. (2003). Missing ... presumed at random: cost-analysis of incomplete data. Health Economics, 12, 377-392.

Chirikos, T. N. (1995). Medicare and the Social Security Disability Insurance program. Health Affairs, Winter 1995, 244-252.

Daub, H. (2002). Reforming the Disability Insurance and Supplemental Security Income Disability programs (statement before the US House of Representatives Committee on Ways and Means, Subcommittee on Social Security).

Dwyer, D., Hu, J., Vaughan, D. R., & Wixon, B. (2001). Counting the disabled: Using survey self-reports to estimate medical eligibility for Social Security's disability programs (ORES Working Paper Series No. 90). Washington, DC: Social Security Administration.

Ellis, B. H., Shannon, E. D., Cox, J. K., Aiken, L., & Fowler, B. M. (2004). Chronic conditions: Results of the Medicare health outcomes survey, 1998-2000. Health Care Financing Review, 25, 75-91.

Gu, D., & Yi, Z. (2004). Sociodemographic effects on the onset and recovery of ADL disability among Chinese oldest-old. Demographic Research, 11, 1-42.

Howell, D. C. (1997). Statistical methods for psychology. Belmont, CA: Duxbury Press.

Hu, J., Lahiri, K., Vaughan, D. R., & Wixon, B. (1997). A structural model of Social Security's disability determination process (ORES Working Paper Series No. 72). Washington, DC: Social Security Administration.

Kennedy, J., Olney, M., Richer, T., & Newsom, M. (2002). A profile of SSDI applicants and beneficiaries, age 18-64: Estimates from the 1994 and 1995 National Health Interview Surveys on Disability. Retrieved March 22, 2003, from http:// www.als.uiuc.edu/dri/pubs/pubs.htm

Kreider, B. (1999). Social Security Disability Insurance: Applications, awards, and lifetime income flows. Journal of Labor Economics, 17(4), 784-827.

Kreider, B., & Riphahn, R. T. (2000). Explaining applications to the U.S. disability system. Journal of Human Resources, 35(1), 82-115.

Lahiri, K., Vaughan, D., & Wixon, B. (1995). Modeling SSA's sequential disability determination process using matched SIPP data. Social Security Bulletin, 58, 3-42.

Lemeshow, S., Teres, D., Avrunin, J. S., & Pastides, H. (1988). Predicting the outcome of intensive care unity patients. Journal of the American Statistical Association, 83, 348-356.

Long, J. S. (1997). Regression models for discrete and limited dependent variables. Thousand Oaks, CA: Sage.

Mitra, S., & Brucker, D. (2004). The early intervention project: An innovative initiative to return disability insurance applicants to work. Journal of Disability Policy Studies, 15, 159-167.

Palmer, E. (1990). Potential gains from early intervention in sickness absenteeism at the workplace: An overview of the current state of knowledge and trends in Sweden. In M. Berkowitz (Ed.), Forging Linkages: Modifying Disability Benefit Programs to Encourage Employment, 9-13. New York: Rehabilitation International.

Patrician, P. A. (2002). Multiple imputation for missing data. Research in Nursing and Health, 25, 76-84.

Pedhazur, E. J. (1982). Multiple regression in behavioral research. Fort Worth, TX: Harcourt Brace Jovanovich.

Social Security Administration. (2002). Annual statistical report on the Social Security Disability Insurance program. Baltimore, MD: Author.

Social Security Administration. (2003a). Disability process May 1, 2002. Retrieved July 31, 2003, from http://www.ssa.gov/ disability/disability_process_frameset.html

Social Security Administration. (2003b). Flow of cases through the disability process. Retrieved July 31, 2003, from http:// www.ssa.gov/disability/disability_process_welcome_2002.htm

Social Security Administration. (2005a). Disability insurance benefit payments. Retrieved August 30, 2005, from http:// www.ssa.gov/OACT/STATS/table4a6.html

Social Security Administration. (2005b). Number of disabled workers and their dependents receiving benefits on December 31, 1970-2004. Retrieved August 30, 2005, from http://www. ssa.gov/OACT/STATS/DIbenies.html

Social Security Administration. (2005c). Fact sheet on the Old-Age, Survivors and Disability Insurance Program. Retrieved August 30, 2005, from http://www.ssa.gov/OACT/FACTS/

Social Security Advisory Board. (2001a). Charting the future of SSA's disability programs: The need for fundamental change. Washington, DC: Author.

Social Security Advisory Board. (2001b). Disability decision making: Data and materials. Washington, DC: Author.

Ticket to Work and Work Incentive Improvement Act (TWWIIA) of 1999, PL 106-170, 42 U.S.C. [section] 1305, et seq.

Waddell, G., Burton, A. K., & Main, C. J. (2003). Screening to identify people at risk of long-term incapacity for work. London: Royal Society of Medicine.

Todd C. Honeycutt

Rutgers University

Debra Brucker

Rutgers University

Footnotes

(1) The SSA application form was changed in 1998 to include the following limitations: hearing, reading, breathing, understanding, coherency, concentrating, talking, answering, sitting, standing, walking, seeing, using hand(s), and writing.

Todd C. Honeycutt, 20 Woodland Terrace, High Bridge, NJ, 08829. Email: [e-mail]

Table 1
Sample Characteristics, Denial Rates, and Allowance Rates

                                      Total          Denied
                                      (% of total)   (% of denied)

N                                     545            191
Age Group
                  18-44               235 (43%)      103 (53%)
                  45-49               80 (15%)       25 (13%)
                  50-54               69 (13%)       21 (11%)
                  55-59               92 (17%)       20 (10%)
                  60+                 65 (12%)       22 (12%)
                  Missing             4 (1%)         0 (0%)
Education Level
                  Less than HS        109 (20%)      46 (24%)
                  High School         248 (46%)      85 (45%)
                  More than HS        146 (27%)      45 (23%)
                  Missing             42 (8%)        15 (8%)
Marital Status
                  Married             259 (48%)      79 (41%)
                  Not married         281 (51%)      110 (58%)
                  Missing             6 (1%)         2 (1%)
Earnings Level
                  Marginal            250 (46%)      111 (41%)
                  Low                 112 (20%)      39 (20%)
                  Average             91 (17%)       23 (12%)
                  High                62 (11%)       13 (7%)
                  Very high           26 (5%)        5 (3%)
                  Missing             4 (1%)         0 (0%)
Physical
Limitations
                  Yes                 64 (12%)       17 (9%)
                  No                  238 (43%)      88 (46%)
                  Missing             243 (45%)      86 (45%)
Mental
Limitations
                  Yes                 54 (10%)       8 (4%)
                  No                  251 (46%)      97 (51%)
                  Missing             240 (44%)      86 (45%)
Onset with Work
Cessation
                  Yes                 151 (28%)      58 (30%)
                  No                  334 (61%)      107 (56%)
                  Missing             60 (11%)       26 (14%)
Body System
                  Musculoskeletal     219 (40%)      90 (47%)
                  Senses/speech       25 (5%)        3 (2%)
                  Respiratory         38 (7%)        14 (7%)
                  Cardiovascular      82 (15%)       21 (11%)
                  Endocrine           37 (7%)        9 (5%)
                  Neurological        62 (11%)       21 (11%)
                  Mental disorders    109 (20%)      47 (24%)
                  Neoplastic          31 (6%)        4 (2%)
                  Immune deficiency   33 (6%)        6 (3%)
                  Other               29 (5%)        8 (4%)
                  Missing             30 (6%)        15 (8%)

                                      Allowed
                                      (% of allowed)

N                                     354
Age Group
                  18-44               132 (37%)
                  45-49               55 (16%)
                  50-54               48 (14%)
                  55-59               72 (20%)
                  60+                 43 (12%)
                  Missing             4 (1%)
Education Level
                  Less than HS        63 (18%)
                  High School         163 (46%)
                  More than HS        101 (28%)
                  Missing             27 (8%)
Marital Status
                  Married             181 (51%)
                  Not married         170 (48%)
                  Missing             4 (1%)
Earnings Level
                  Marginal            139 (39%)
                  Low                 73 (21%)
                  Average             68 (19%)
                  High                49 (14%)
                  Very high           21 (6%)
                  Missing             4 (1%)
Physical
Limitations
                  Yes                 47 (13%)
                  No                  150 (42%)
                  Missing             157 (44%)
Mental
Limitations
                  Yes                 46 (13%)
                  No                  154 (44%)
                  Missing             154 (44%)
Onset with Work
Cessation
                  Yes                 93 (26%)
                  No                  227 (64%)
                  Missing             34 (10%)
Body System
                  Musculoskeletal     129 (36%)
                  Senses/speech       22 (6%)
                  Respiratory         24 (7%)
                  Cardiovascular      61 (17%)
                  Endocrine           28 (8%)
                  Neurological        41 (12%)
                  Mental disorders    62 (18%)
                  Neoplastic          27 (8%)
                  Immune deficiency   27 (8%)
                  Other               21 (6%)
                  Missing             15 (4%)

Note. Persons had up to two affected body systems.

Table 2
Logistic Regression Results for Full and Restricted Models

                                    Model 1
                            (Full Logistic Model)

                            Coefficient       SE

Intercept                   -0.445           0.323
Age Group
  18-44 a
  45-49                      0.222           0.348
  50-54                      0.672           0.449
  55-59                      0.846 *         0.398
  60+                        0.445           0.400
Education Level
  Less than HS              -0.276           0.316
  High School (a)
  More than HS              -0.078           0.308
Marital Status
  Married                    0.336           0.249
Earnings Level
  Marginal (a)
  Low                        0.538 (+)       0.317
  Average                    1.083 **        0.362
  High                       1.235 **        0.467
  Very high                  1.047           0.667
Physical Limitations         0.178           0.426
Mental Limitations           1.135 *         0.461
Onset with Work Cessation   -0.547 (+)       0.295
Body System
  Musculoskeletal (a)
  Senses/speech              1.794 (+)       1.029
  Respiratory                0.494           0.538
  Cardiovascular             0.438           0.361
  Endocrine                  0.276           0.485
  Neurological               0.136           0.382
  Mental disorders          -0.098           0.326
  Neoplastic                 1.250           0.822
  Immune deficiency          1.100           0.771
  Other                      0.192           0.542

                                      Model 2
                            (Restricted Logistic Model)

                            Coefficient       SE

Intercept                    0.048           0.183
Age Group
  18-44 a
  45-49
  50-54
  55-59                      0.853 *         0.357
  60+
Education Level
  Less than HS
  High School (a)
  More than HS
Marital Status
  Married
Earnings Level
  Marginal (a)
  Low                        0.404           0.293
  Average                    1.067 **        0.336
  High                       1.176 **        0.440
  Very high
Physical Limitations
Mental Limitations           1.024 *         0.449
Onset with Work Cessation   -0.336           0.264
Body System
  Musculoskeletal (a)
  Senses/speech              1.582           1.021
  Respiratory
  Cardiovascular
  Endocrine
  Neurological
  Mental disorders
  Neoplastic
  Immune deficiency
  Other

Note. For purposes of estimation and comparison, the reference
categories are persons less than 45 years old, with marginal
earnings, HS education, and musculoskeletal disorders.

(a) Denotes reference group for Model 1.

(+) p < .10. * p < .05. ** p < .01.

Table 3
Sum of Scores Model Variables, Values, and Denial and Allowance Rates

Variable                                    Value   Denied %   Allowed

Age Group
  18-44                                       0       57%        38%
  45-54                                       2       23%        27%
  55-59                                       4        9%        22%
  60-65                                       1       11%        13%
Education Level
  Less than high school                       0       26%        21%
  High school or more than high school        1       74%        79%
Marital Status
  Not married                                 0       60%        47%
  Married                                     2       40%        53%
Earnings Level
  Marginal                                    0       59%        38%
  Low                                         1       20%        20%
  Average, high, or very high                 3       21%        42%
Physical Limitations
  Yes                                         2       18%        24%
  No                                          0       82%        76%
Mental Limitations
  Yes                                         4        9%        21%
  No                                          0       91%        79%
Onset with Work Cessation
  Yes                                         0       35%        32%
  No                                          1       65%        68%
Primary Condition Body System
  Musculoskeletal, multiple body systems,     0       79%        59%
  neurological, or mental
  All other conditions                        3       21%        41%
Secondary Condition Body System
  Respiratory or mental                       0       20%        14%
  None or any other disorder                  1       69%        67%
  Senses, cardiovascular, endocrine, or       3       11%        20%
  neurological
Total                                                 100%      100%

Table 4
Comparison of Models in Predicting SSDI Receipt, by Scoring and
Validation Samples

                                     Model 1      Model 2     Model 3
                                      (Full     (Restricted   (Sum of
                                    Logistic)    Logistic)    Scores)

Scoring Sample
  Acceptance Rate                      58%          60%         55%
  Sensitivity                          70%          72%         68%
  Specificity                          63%          62%         69%
  Predictive Value of Acceptance       78%          78%         80%
  Predictive Value of Denial           53%          54%         53%
  Efficiency                           67%          68%         68%
  Phi Coefficient                     0.32         0.32        0.35
Validation Sample
  Acceptance Rate                      65%          66%         60%
  Sensitivity                          70%          74%         69%
  Specificity                          42%          45%         53%
  Predictive Value of Acceptance       62%          64%         66%
  Predictive Value of Denial           51%          57%         56%
  Efficiency                           58%          62%         62%
  Phi Coefficient                     0.13         0.20        0.22