Assessing outcome in practice settings: a primer and example from an eating disorders program. (Research).

All segments of the behavioral healthcare system are increasingly being called upon to demonstrate the effectiveness of their treatment. This article outlines a type of research study, the effectiveness study, that can be implemented quite easily and inexpensively and can provide a measure of treatment success. The article discusses choice-points for engaging in an effectiveness study and gives suggestions for research decisions. A small-scale study conducted at a partial hospitalization unit for eating disorders is used as an example. The information on designing and implementing a research program and the example provided are intended to assist practitioners and program administrators who are considering engaging in their own outcome research.

**********

All segments of mental healthcare have felt the effects of changes in healthcare funding. Treatment programs are increasingly being called upon to demonstrate the effectiveness of their interventions in order to receive managed care contracts or insurance reimbursement for treatment (Lennox, 1995; Sexton, 1996). Programs and professionals who cannot demonstrate their successes may find themselves unable to survive in the current competitive environment (Burlingame, Lambert, & Reisinger, 1995).

In spite of these pressures, there has been some resistance among mental healthcare providers to demonstrate program effectiveness. Some providers have argued that the invasion of accountability into mental healthcare has negatively impacted therapeutic decision making (Sherman, 1992). Still others argue that time spent in evaluation could be better used in treatment or that the therapeutic process is simply not quantifiable (Mirin & Namerow, 1991). Finally, there is a growing voice in the literature that specific interventions do not lead to specific outcomes. Rather a common-factors or curative-factors approach supports the belief that transtheoretical therapeutic factors are primarily responsible for client change. From this perspective, research that leads to specific empirically supported treatments for specific disorders misses the mark (Wampold, 2001). Regardless of these philosophical objectives, the financial reality remains. Few practitioners could survive financially without managed care or other insurance contracts (Granello & Granello, 2001).

Of course, not all practitioners are resistant to demonstrations of effectiveness. However, even among those who are open to conducting such research, there appears to be a widespread reluctance to initiating such research. Practitioners may erroneously believe that the task will be overwhelming (Plante, Couchman, & Diaz, 1995), or that a program of research will necessarily be costly, complex, and time-consuming (Granello, Granello, & Lee, 1999). What has become apparent is that many mental health practitioners have not received the training they need to conduct such research. Research methods courses in university programs may focus on understanding laboratory research with true experimental designs that often are not possible in real-world assessment (Sandell, Blomberg, & Lazar, 1997). Thus, there are some practitioners who may be ill prepared to conduct their own outcome research, regardless of their willingness to do so.

Because of these problems with conducting research, many programs rely on published research studies to demonstrate treatment successes. Clearly, published research plays a significant role in determining appropriate treatment interventions and the efficacy of various modalities, but bridging the gap between research and practice is essential (Sexton, 2000). Research on one's own program effectiveness, in conjunction with already published research to support general program interventions, increases the quality of the information provided to practitioners, the public, and the funding sources (Granello & Granello, 2001).

Measuring treatment effectiveness need not be a difficult or cumbersome task. Simple measures of effectiveness can be implemented quite easily, and the demonstrated outcomes from such research can be a very effective tool to provide evidence of treatment success. Further, measures of program effectiveness can be used to enhance client satisfaction and, if used properly, can be used proactively to improve client care (Burlingame et al., 1995).

This article outlines a specialized type of research, the effectiveness study, that can be used by many different types of behavioral treatment facilities to demonstrate treatment success. The focus on this research was in an eating disorders program, but many of the components of this type of research are generalizable to other types of intervention programming. The purpose of this article is to assist mental health program administrators and clinicians who wish to demonstrate treatment effectiveness yet may be reluctant to engage in research that is costly, that requires manipulation of the treatment program, or that has experimentally complex designs. The effectiveness study is one approach that can be used to provide a measure of treatment success to the public, managed care and other insurance companies, and the clinical staff. This article outlines the choice-points for engaging in an effectiveness study and gives suggestions for research decisions. A small-scale study conducted at a partial hospitalization unit for eating disorders is used as an example. The information on designing and implementing a research program and the example provided are intended to assist program administrators who wish to begin their own outcome research.

METHODOLOGY

Effectiveness Research

Mental health administrators wishing to collect research data need to be aware of the differences between field or "effectiveness" research and laboratory or "efficacy" research. Administrators and clinicians working with patients in the field must face fiscal, ethical, and demographic realities that are often outside of their control. This reality differs significantly from laboratory research in which experimental manipulation is easier. The distinction is important for individuals wishing to conduct outcome research outside of the laboratory and merits a brief discussion.

Efficacy studies use random assignment to treatment and control groups, manualized treatment, and subjects who meet criteria for a single diagnosed disorder (Seligman, 1995; Wampold, 1997). They provide useful information and are appropriate designs for laboratory studies (Sandell et al., 1997) or settings in which highly controlled manipulation of variables is possible (Kazdin, 1980; Lambert, Huefner, & Nace, 1997). However, they are beyond the scope of most mental health treatment programs to implement.

Effectiveness studies such as the current one attempt to answer how well patients fare under treatment as it is actually practiced in the field, and yield useful and credible information that can empirically validate psychotherapy (Sandell et al., 1997). Effectiveness studies recognize that less than methodologically ideal situations exist in the field. These methodological limitations include: (a) therapy is not always of fixed duration; it usually keeps going until the patient improves, quits, or the insurance runs out; (b) psychotherapy is often eclectic rather than manualized, and is typically self-correcting--that is, if one technique is not working, another is usually tried; (c) and patients typically present with multiple problems, some sub-clinical and some diagnosable, rather than the pure diagnoses represented in efficacy studies (Seligman, 1995).

The nature of conducting research under field or uncontrolled conditions does not invalidate the results of that research but requires that the limitations of such data be understood in relation to interpreting and applying results to service delivery and program quality assurance. Effectiveness studies have limitations, particularly in the area of internal validity. The lack of a control group and randomization are noteworthy. This is consistent with much of the outcome research, particularly with patients in severe distress, given the ethical implications of denying treatment to research participants allocated to a control condition (McLeod, 1994). In addition, the lack of control over patient selection, over therapist adherence to treatment plans, and over psychotherapy modalities utilized also are limitations. What actually occurs in the therapy sessions is not controlled, and conclusions cannot be drawn regarding what, if any, specific interventions led to treatment success.

Overall, the effectiveness study, like the current one, meets the needs of programs wishing to provide evidence of their effectiveness to funding sources and to the public. It is not meant, however, to take the place of existing, published research studies on effective treatment interventions. Rather, studies like this one, conducted in one's own program, provide an adjunct measure of treatment success. Although all research has limitations, effectiveness and efficacy studies can provide a strong complement to one another, in general, balancing the high internal validity/low external validity of efficacy research with the high external validity/low internal validity of effectiveness research.

Research Design

The first and most basic decisions that need to be made are those regarding research design. What questions is the research intended to answer? At the partial hospitalization unit where the current research was conducted, there was no existing information on program demographics or treatment success. Therefore, the hospital was looking for basic data on patient characteristics and treatment outcome. Thus, the research design included a compilation of simple descriptive statistics and a pre-post measure of patient symptomatology. We wanted to answer: (a) what are the typical characteristics (demographics, diagnosis, Body Mass Index, etc.) of patients who attend the partial hospitalization unit? (b) do patients who receive treatment at our partial hospital program leave with significantly lower levels of symptomatology at discharge then they had at admission? and, (c) do any demographic variables (e.g., age, diagnosis, Body Mass Index) affect the success rate of the treatment? If more complex designs are required, then these decisions must be made early. The most common addition to the simple pre-post design is the inclusion of follow-up measures to assess whether treatment gains are maintained.

Population

Research ethical protocol states that participation in program effectiveness studies must be voluntary, and if individuals are under age 18, then a parent or guardian must give consent. Aside from ethical considerations, a primary choice-point for program administrators is which patients will be included in the research program. If a program has patients with Anorexia Nervosa (AN), Bulimia Nervosa (BN), and Eating Disorder: Not Otherwise Specified (ED:NOS), for example, will all clients participate in the research? If the program has multiple levels of intervention (e.g., in-patient, day treatment, outpatient), will all levels of treatment be included? What will be done with those patients who move between treatment levels during the course of their treatment? Will those who participate in treatment only irregularly be included? What about those who leave treatment Against Medical Advice (AMA)? Clearly, it is difficult to obtain post-treatment scores from these individuals. These decisions should be made up front, as they will have important implications for instrumentation, data collection, and data analysis.

In the current study, it was decided that all patients who were fully enrolled in the partial hospitalization program at some point during the 6-month time period in which data were collected would be asked to participate. We decided that patients who attended only a few days per week or a few hours per day would not be included in the analysis. In addition, patients were excluded from participation if they left AMA. Thus, the sample (n = 33) included patients with a diagnosis of AN, BN, and Eating Disorder, NOS (primarily having symptoms of Binge Eating Disorder, BED).

Instrument Selection

Selection of instruments decides the type of data that can be obtained, and thus the choices regarding instrumentation must be made with care. The basic research questions that are being investigated guide the instrument selection. Program administrators are strongly encouraged to use existing instruments whenever possible, rather than attempting to develop their own. Instruments developed independently require large commitments of time and resources to ensure reliability and validity, and once data are collected, no comparisons can be made with norming groups from existing research. When selecting from existing instruments, program administrators can consider the cost of the instruments, which includes personnel time to administer, score, and analyze the results. Further, it is important to consider a measure that is sensitive to meaningful change (Burlingame et al., 1995). Given the large differences in the breadth and depth of information that existing instruments yield, as well as differences in validity and reliability of existing instruments, administrators are cautioned to make instrumentation choices carefully. (For articles detailing methods to critically analyze outcome instruments, see: Ciarlo, Brown, Edward, Kiresuk, & Newman, 1986; Lambert, Ogles, & Masters, 1992; Newman, Hunter, & Irving, 1987).

Using a small battery of instruments, rather than just one, may provide the best information. However, administrators should take care not to overburden the patients or have so many instruments that they are overwhelmed with data. Two or three short instruments, plus a demographics questionnaire, may be sufficient (Granello, Granello, & Lee, 2000). Clinician ratings (e.g., a Global Assessment of Functioning score) can be an important component of treatment evaluation, as clinicians may be in a unique position to provide insight into patient progress. However, administrators are cautioned against using clinician ratings as a stand-alone measure of progress, as they have been criticized for their subjectivity (McLeod, 1994).

Measures of global functioning. These measures are used to assess the overall symptomatology of clients and provide mental health counselors and treatment programs with gross measures of how well their clients function in broad areas of their lives. In existing outcome research on patients with eating disorders, the most commonly used measures of global functioning are the Symptom Checklist 90-R (SCL-90-R; Derogatis, 1994), the Brief Symptom Inventory (BSI; Derogatis & Spencer, 1982), and the Minnesota Multiphasic Personality Inventory (MMPI; Hathaway & McKinley, 1948).

Measures of eating disorders. These measures are used to assess the specific symptoms that are often associated with disordered eating. Among the most commonly used assessments in eating disorders outcome research are the Eating Disorder Inventory-2 (EDI-2; Garner, 1991) and the Eating Attitudes Test (EAT; Garner, Olmsted, Bohr, & Garfinkel, 1982). A third instrument, the Three Factor Eating Questionnaire (TFEQ; Stunkard & Messick, 1985) also has been used, although less frequently than either the EDI or the EAT.

Symptom-specific instruments. Because eating disorders have been linked to a variety of psychological symptoms, including low self-esteem (O'Dea, 2000), depression (Tomori & Rus-Makovex, 2000), anxiety (Zaider, Johnson, & Cockrell, 2000), social anxiety (Striegel-Moore, Silberstein, & Rodin, 1993), obsessional thinking (Zubieta, Demitrack, Fenick, & Krahn, 1995), and problems with social adjustment (Keel, Mitchell, Miller, Davis, & Crow, 2000), some studies have made use of specific instruments to measure these constructs. Administrators in programs that focus particularly on symptom reduction in one of these areas may wish to select an instrument that is particularly sensitive to changes in that area.

Client satisfaction. Although published outcome research on patients with eating disorders does not tend to include measures of client satisfaction, administrators may wish to include such a measure in their research programs. Client satisfaction is of particular concern to both funding sources and program administrators. When assessing client satisfaction, it is important to measure the patients' attitudes about the specific treatment received from the program being assessed and not their attitudes about mental health treatment in general or treatment that they might have experienced in the past. The use of a standardized instrument that has reliability and validity data is preferable to an agency-developed patient satisfaction questionnaire (Granello et al., 1999). The most commonly used patient satisfaction questionnaire in the outcome research literature is the Client Satisfaction Questionnaire-8 (CSQ-8; Larsen, Attkisson, & Hargreaves, 1979).

Demographics. An essential component of data collection is the demographics questionnaire. This typically includes information on gender, age, race, marital status, occupational status, diagnosis, and age of onset. For patients with eating disorders, however, information on the symptoms of the disorder is particularly important. Often this includes questions regarding frequency of binges, purges, type of purges (laxative abuse, vomiting, excessive exercise), presence of amenorrhea, height, weight, and Body Mass Index (BMI). The demographics questionnaire, like the other instruments, must be given at all data collection points in order to monitor progress on the various symptoms of the eating disorder. Thus, administrators may wish to develop two versions of the demographics questionnaire. The one given at the first point of data collection must include both demographics and symptom-specific questions. Questionnaires given at subsequent data points need only include the information relevant to the eating disorder or other mental health symptoms, and not information on age, race, ethnicity, etc, as this information does not change. The information collected through the demographics questionnaire becomes important not only to measure progress (e.g., did the frequency of binges diminish over the course of treatment?), but also to group patients to compare treatment success (e.g., is the program more successful in treating patients who purge via vomiting than treating those who engage in excessive exercise?).

Often these demographic questionnaires are designed in-house by the treatment facility to glean information that is most relevant to their particular clients or to the research questions they have. For those programs that wish to make use of an existing questionnaire, the Eating Disorders Examination, questionnaire form (EDE; Fairburn & Cooper, 1993) can be used.

Instruments in the current study. In the current study, we used the SCL-90-R (Derogatis, 1994) and a self-made demographics questionnaire. We collected information on gender, age, race, marital and occupational status, age of onset, number of hospitalizations since onset, current medications, frequency of binges, frequency and type of purges, exercise, height, weight, BMI, and diagnosis.

Procedures

Administrators need to take into account the process by which data will be collected. An analysis of the work environment may be necessary to determine the points at which data can be most easily collected and by whom. Clinicians may be resistant to collection of data if they believe that it is adding to their workloads, reducing time spent in therapy, or if they do not see the benefits to their clients or the program. Administrators may wish to consider instruments and procedures that can be implemented by clerical personnel, as these individuals' time usually represents a lower fiscal cost to the program. Finally, consideration can be made for collecting research data at logical times in the patient's treatment, such as at admission or discharge planning session, when other data are being collected. If data are to be collected at follow-up (e.g., 3 or 6 months after the completion of the treatment), then a procedure will need to be developed and implemented to capture data from the patients at these times.

In the current study, data were collected within 24 hours of initial intake, typically when other intake paperwork was completed. By including a copy of the instruments in each standard intake packet, the completion rate for the paperwork was improved. Discharge data were collected on the day the patient left treatment, when other discharge paperwork was completed. Instruments were placed in folders prior to discharge to improve chances of completion.

Statistical Analyses

Program administrators can use the information collected in the demographics questionnaire to answer very basic questions about the patients in their programs. This information can be extremely useful in program planning.

Pre-post research design questions can typically be answered with a paired-sample (e.g., each person has data collected at admission and discharge) t-test. In this very simple design, comparisons are made between admission and discharge scores on a single instrument or subscale. Typically, to make comparisons on multiple pre-post subscale scores, a repeated measures MANOVA will be necessary. Subscales of the same instrument often have high inter-scale correlations that make a MANOVA, rather than a series of t-tests, statistically necessary (Stevens, 1992). MANOVAs are used when multiple dependent variables are statistically related to each other, as would be the case for most instrument subscales. In MANOVA, comparisons are made on all subscales simultaneously, and computerized statistical packages such as the SPSS[R] program (2000) used in this study automatically incorporate the correlational pattern of the dependent variables. Using the MANOVA also eliminates the need for a Bonferroni correction. In other words, when a series of tests are run, the alpha level must typically be set at a higher, more stringent, level in order to reduce the possibility of Type I errors (or finding significant differences that resulted from chance alone). This is not necessary with the MANOVA.

In the current study, data were available on only one male patient, and therefore, he was eliminated from the analysis. Thus, all participants included in the analysis were female (n = 32). Average age was 26.78, with a range from 16 to 40 (SD = 8.70). Participants were primarily Caucasian (n = 30), single (n = 17), and employed full-time (n = 18). Only 19% were on Disability Status at time of admission. Average age of onset of the eating disorders was 19.1 (Range = 10 to 36 years, SD = 5.92), and average number of hospitalization prior to current treatment was 1.1 (Range = 0 to 12, SD = 2.4).

We used the psychiatric outpatient norms available in the test manual to convert the SCL-90-R raw scores of our patients to T-scores (Derogatis, 1994). Administrators will need to decide whether to convert raw scores to T-scores, and which norming group most closely matches their sample. Conversion to T-scores allows for comparisons of the sample to the norming groups. After we converted our data, we could have used a simple paired-samples t-test on the Global Severity Index of the SCL-90-R (the index that measures global functioning on all scales of the SCL-90-R) to measure whether patients, on average, improved in their functioning between admission and discharge. The mean T-score at admission was 53.47 (compared to psychiatric outpatients), and at discharge, the mean T-score was 43.94. This represents a statistically significant overall reduction in symptomatology (t(31) = 7.45, p<.001, [[eta].sup.2] = .64, power >.99). Effect sizes are used to determine the strength of a relationship, and in this study, the effect size is measured by eta ([eta]), part of the r family of effect sizes (Rosenthal, 1994). An eta of .64 is considered large (Cohen, 1988). Thus, while the results of the t-test only reveal that the relationship is significantly larger than zero (rejecting the null hypothesis), effect sizes allow interpretations about the size of the change. The statistical power of this test also is useful, as it reveals the ability of the test to uncover relationships. Power ranges from zero (no ability to detect meaningful relationships) to 1.0 (absolute ability to detect meaningful relationships). In this case, the estimated post-hoc power of >.99 means that there was a greater than 99% chance of finding the effect size that we uncovered, if it existed in the population. Thus, the results of the paired-sample t-test in this study revealed a large difference in self-reported symptomatology between admission and discharge scores, as measured by the SCL-90-R Global Severity Index, for patients in the day treatment program. Although this information is useful for marketing, more complex statistical analyses can yield other clinically significant data.

A repeated measures MANOVA (that is, data were collected on each client at admission and discharge) was used to determine symptom reduction for all SCL-90-R subscales (rather than simply the Global Severity Index, which was measured through the paired-sample t-test). A MANOVA can be used to supplement the findings of the t-test. That is, the t-test measures overall change in the Global Severity Index while the MANOVA looks at changes in each of the separate subscales. MANOVA results indicated that the average patient attained statistically significant improvement (F(12,20) = 4.15, p<.01, [eta] = .87, power>.99). Note that because more information is included in this test than in the simple t-test (all subscales versus just the Global Severity Index), the effect size of the overall test is higher. To determine on which of the subscales significant improvements occurred, post-hoc univariate tests were performed. Results indicated significant improvement on all SCL-90-R subscales with an alpha level of p<.01 for all tests. Thus, patients in this study reported statistically and clinically significant improvement between admission and discharge on the following subscales: Somatization, Obsessive-Compulsive, Interpersonal Sensitivity, Depression, Anxiety, Hostility, Phobic Anxiety, Paranoid Ideation, and Psychoticism. Effect sizes for all subscales ranged from a low of .47 (Phobic Anxiety) to a high of .78 (Depression).

In the current study, diagnosis at intake did not affect treatment success. There were no significant differences on SCL-90-R scores between individuals with anorexia nervosa (n = 9), bulimia nervosa (n = 11), and eating disorder, NOS (n = 11) at admission (F = 1.25,p = .30) or discharge (F = .89, p = .60). We believed that it was unrealistic to expect that BMI would change significantly over the course of treatment in the partial hospitalization unit (average length of stay equals 12 days). Therefore, rather than examine a pre-post BMI, we investigated whether BMI at admission affected treatment results. Because both BMI and Global Severity Index scores are continuous rather than categorical variables, we used a correlation to determine their relationship. We found that BMI was not significantly correlated to Global Severity Index scores at admission (r = -.39 p>.05) or at discharge (r = -.39, p>.05).

Using the Results

With the results of the current study, the eating disorder partial hospitalization unit was able to answer the original research questions in the following way. Using demographic information, the partial hospitalization program was able to uncover information about the types of patients who use their program and the severity of their illnesses. Using either t-test or MANOVA results, we found that overall, patients made large treatment gains between admission and follow-up. The effect size (eta) of .87 means that these improvements are very large (as defined by Cohen, 1988) and represent clinical as well as statistical significance. Overall, patients made treatment gains in all areas measured by the SCL-90-R, with a moderate effect size for Phobic Anxiety, and large effect sizes for pre-post differences on the remaining subscales (Cohen). Finally, there were no significant differences in the treatment success at this partial hospitalization unit for patients in all three diagnostic categories of eating disorders, and no statistically significant relationship between treatment success and admission BMI. In other words, patients with Bulimia Nervosa improved as well as patients with Anorexia Nervosa and those with ED:NOS.

Further analyses could be conducted to answer specific questions. For example, although we did not conduct an analysis with other types of diagnoses, with a larger sample, we could have investigated whether a coexisting Axis I disorder (e.g., depression) or the existence of an Axis II disorder affected treatment results. Additionally, a pre-post t-test of number of binges/purges or comparisons of pre-post BMI (for programs with longer treatments) also could yield important program information.

It is important to note that clinical significance, more than statistical significance, is what should drive programmatic decisions. Effect sizes will help gauge this clinically significant progress. Mental health administrators and practitioners should emphasize the clinical significance of moderate or large effect sizes, even in the face of statistical non-significance, which is often caused by low statistical power that is typical of small scale studies.

IMPLICATIONS FOR MENTAL HEALTH COUNSELORS AND PROGRAM ADMINISTRATORS

A primary goal of clinicians and administrators in all behavioral healthcare programs is, of course, providing quality treatment to patients in order to improve their functioning. The current economic realities, if not adequately addressed, can negatively impact a program's ability to provide quality care, and in some cases, even to exist. The effectiveness study outlined in this article provides a mechanism through which programs can begin to collect data, and the results of the current research provided strong evidence of treatment success at the eating disorders partial hospitalization unit studied. The large effect sizes provided evidence of clinical as well as statistical improvement. Results can reassure both funding sources and the public of the clinical utility of the treatments provided.

The research has a second important purpose, beyond marketing. Data collected can provide a program with much-needed information to assist with continuous quality improvement efforts required by most accrediting bodies. Mental health counselors with access to data can use that data to improve their treatment interventions, and research has found that practitioners' efficacy improves when they are involved in research (Hauri, Sanborn, Corson, & Violette, 1988). Reports from agencies that make systematic attempts to investigate their outcomes indicate that once mental health clinicians become aware of variations in client outcomes, they are in a better position to generate ideas for improvement and hypotheses for further testing ("Authors pose 7 questions," 1997). Thus, data collection and analysis can have great clinical importance.

A third purpose of the research could be as a published piece in the literature. Although an outcome assessment at a particular agency may not typically be thought of as publishable, in certain circumstances, the results could add important contributions to the literature. In cases where little or no research exists in a particular area or when only efficacy studies have been published to date, the results of an effectiveness study could provide a starting point for other practitioners and researchers.

In the current study, we used the results in several important ways. First, although the clinicians in the program believed that the program was successful, it was extremely validating to an overworked staff to see evidence of this. Second, the results were used to demonstrate effectiveness to insurance companies and funding agencies. A third, unanticipated, result was that the administration of the hospital was so impressed by these results that they immediately approved funding to hire another mental health counselor for the program and used the results of this research in their media and marketing campaigns. Thus, this small-scale study had clear clinical and financial implications.

The results of effectiveness studies can be used to demonstrate overall treatment effectiveness and to make broad programmatic changes. However, they have limitations. Whereas aggregate data can help identify areas of particular strength for the treatment providers, it does not take into account outliers and problems with specific cases (McCarthy, Gelber, & Dugger, 1993). In addition, because the study is naturalistic, it cannot be said whether or not any specific treatments or interventions led to treatment success. Internal validity is also compromised, due to maturation. Lambert (1986) has demonstrated that the passage of time has a positive effect on many acute exacerbations of mental illness and may be a factor in the positive outcomes noted.

We agree with Seligman's (1995) assertions that in spite of their limitations, effectiveness studies are a complementary research method to efficacy studies. They provide mental health practitioners with research that is clinically useful, important for negotiating managed care or other insurance contracts, and allows for meaningful research to be conducted with minimal disruption to the existing program. It is important to note, however, that effectiveness studies can be conducted with more statistical controls than were used in the current study. Randomized assignment to alternative treatments, rather than the "wait-list" or "no treatment" controls typically used in efficacy studies, are a meaningful way of comparing treatment in effectiveness studies.

In spite of the limitations, effectiveness studies such as this one can yield important findings that can affect treatment interventions. Mental health administrators wishing to conduct outcome research in their own programs may wish to consult Ogles, Lambert, and Goates' (2002) book on outcome assessment for practitioners for a more comprehensive review. Based on our review of the literature and our experience in conducting outcome assessment, we encourage mental health administrators and practitioners to keep a few important suggestions in mind. First, effectiveness studies cannot be all things to all mental health administrators and practitioners. Complex designs with multiple administrations and a large number of instruments may so overwhelm the staff that they are never completed, or once completed, they are never statistically analyzed in a meaningful way. For programs just beginning to collect data, our recommendation is to keep the data collection and analysis small. In many cases, simple descriptive statistics, graphed or displayed appropriately, might be sufficient to understand client progress. Second, although outcome research need not be cost prohibitive, some foresight will be necessary to set aside sufficient funds for instruments, and if necessary, data analysis. The present study represented a university-agency collaboration, which although not necessary, can provide a symbiotic relationship (data for the university, data analysis for the agency). Third, as much as possible, integrate the collection of data into the everyday functioning of the agency (e.g., put pre-tests in admissions packets, assign a particular person to collect follow-up data, etc.). Finally, if your program is not currently collecting data, then any step, however small, is a step in the right direction. The demonstration of treatment effectiveness can provide both an external benefit in the way of marketing, and an internal benefit, by providing evidence of effectiveness to an overworked staff.

REFERENCES

Authors pose 7 questions to address in designing outcomes system. (1997, August). Behavioral Health Outcomes, 2(7), 9-10.

Burlingame, G. M., Lambert, M. J., & Reisinger, C. W. (1995). Pragmatics of tracking mental health outcomes in a managed care setting. The Journal of Mental Health Administration, 22, 226-236.

Ciarlo, J. A., Brown, T. R., Edwards, D. W., Kiresuk, T. K., & Newman, F. L. (1986). Assessing mental health treatment outcome measurement techniques. (DHHS Publication No. ADM 86-1301). U.S. Government Printing Office: Washington, DC.

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.

Derogatis, L. R. (1994). SCL-90-R: Administration, scoring, and procedures manual. Minneapolis: National Computer Systems.

Derogatis, L. R., & Spencer, M. S. (1982). The Brief Symptom Inventory (BSI): Administration, scoring, and procedures manual-I. Baltimore: Johns Hopkins University School of Medicine, Clinical Psychometrics Research Unit.

Fairburn, C. G., & Cooper, Z. (1993). The Eating Disorder Examination (12th ed.). In C. G. Fairburn & G. T. Wilson (Eds.), Binge eating: Nature, assessment, and treatment, (pp. 317-360). New York: Guilford.

Garner, D. M. (1991). Eating Disorder Inventory 2: Professional manual. Odessa, FL: Psychological Assessment Resources.

Garner, D. M., Olmsted, M. P., Bohr, Y., & Garfinkel, P. E. (1982). The Eating Attitudes Test: Psychometric features and clinical correlates. Psychological Medicine, 12, 871-878.

Granello, D. H., & Granello, P. F. (2001). Counseling outcome research: Making practical choices for real-world applications. In G. R. Walz & J. C. Bleuer (Eds.), Assessment: Issues and challenges for the millennium, (pp. 163-172). Greensboro, NC: ERIC/CASS.

Granello, D. H., Granello, P. F., & Lee, F. (1999). Measuring treatment outcomes and client satisfaction in a partial hospitalization program. The Journal of Behavioral Health Sciences and Research, 26, 50-63.

Granello, D. H., Granello, P. F., & Lee, F. (2000). Measuring treatment outcome in a child and adolescent partial hospitalization program. Administration and Policy in Mental Health, 27, 409-422,

Hathaway, S. R., & McKinley, J. C. (1948). The Minnesota Multiphasic Personality Inventory. New York: Psychological Corp.

Hauri, P., Sanborn, C., Corson, J., & Violette, J. (1988). Handbook for beginning mental health researchers. New York: Haworth.

Kazdin. A. E. (1980). Research design in clinical psychology. New York: Harper & Row.

Keel, P. K., Mitchell, J. E., Miller, K. B., Davis, T. C., & Crow, S. J. (2000). Social adjustment over 10 years following diagnosis with bulimia nervosa. International Journal of Eating Disorders, 27, 21-28.

Lambert, M. J. (1986). Some implications of psychotherapy research for eclectic practice. International Journal of Eclectic Psychotherapy, 5, 16-46.

Lambert, M. J., Huefner, J. C., & Nace, D. K. (1997). The promise and problems of psychotherapy research in a managed care setting. Psychotherapy Research, 7, 321-332.

Lambert, M. J., Ogles, B. M., & Masters, K. S. (1992). Choosing outcome assessment devices: An organizational and conceptual scheme. Journal of Counseling and Development, 70, 527-532.

Larsen, D. L., Attkisson, C. C., & Hargreaves, W. A. (1979). Assessment of client/patient satisfaction: Development of a general scale. Evaluation and Program Planning, 6, 299-324.

Lennox, R. D. (1995). Outcomes of mental health and drug abuse treatment. The Journal of Mental Health Administration, 22, 212-213.

McCarthy, P. R., Gelber, S., & Dugger, D. E. (1993). Outcome measurement to outcome management: The critical step. Administration and Policy in Mental Health, 21, 59-68.

McLeod, J. (1994). Doing counselling research. London: Sage.

Mirin, S. M., & Namerow, M. J. (1991). Why study treatment outcome? Hospital and Community Psychiatry, 42, 1007-1013.

Newman, F. L., Hunter, R. H., & Irving, D. (1987). Simple measures of progress and outcome in the evaluation of mental health services. Evaluation and Program Planning, 10, 209-218.

O'Dea, J. (2000). School-based interventions to prevent eating problems: First do no harm. Eating Disorders: The Journal of Treatment and Prevention, 8, 123-130.

Ogles, B. M., Lambert, M. J., & Fields, S. A. (2002). Essentials of outcome assessment. New York: Wiley.

Plante, T. G., Couchman, C. E., & Diaz, A. R. (1995). Measuring treatment outcome and client satisfaction among children and families. The Journal of Mental Health Administration, 22, 261-269.

Rosenthal, R. (1994). Parametric measures of effect size. In H. Cooper & L. V. Hedges, (Eds.), The handbook of research synthesis, (pp. 231-244). New York: Russell Sage Foundation.

Sandell, R., Blomberg, J, & Lazar, A. (1997). When reality doesn't fit the blueprint: Doing research on psychoanalysis and long-term psychotherapy in a public health service program. Psychotherapy Research, 7, 333-344.

Seligman, M. E. P. (1995). The effectiveness of psychotherapy: The Consumer Reports Study. American Psychologist, 50, 965-974.

Sexton, T. L. (1996). The relevance of counseling outcome research: Current trends and practical implications. Journal of Counseling and Development, 24, 590-600.

Sexton, T. L. (2000). Restructuring clinical training: In pursuit of evidence-based clinical training. Counselor Education and Supervision, 39, 218-227.

Sherman. C. F. (1992). Changing practice models in managed health care. Psychotherapy in Private Practice, 11, 29-32.

SPSS. (2000). SPSS[R] base 10.0 for Windows[TM] user's guide. Chicago: SPSS.

Stevens, J. (1992). Applied multivariate statistics for the social sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.

Striegel-Moore, R. H., Silberstein, L. R., & Rodin, J. (1993). The social self in bulimia nervosa: Public self-consciousness, social anxiety, and perceived fraudulence. Journal of Abnormal Psychology, 102, 297-303.

Stunkard, A. J., & Messick. S. (1985). The three-factor eating questionnaire to measure dietary restraint, disinhibition and hunger. Journal of Psychosomatic Research, 29, 71-83.

Tomori, M., & Rus-Makovex, M. (2000). Eating behavior, depression, and self-esteem in high school students. Journal of Adolescent Health, 26, 361-367.

Wampold, B. E. (1997). Methodological problems in identifying efficacious psychotherapies. Psychotherapy Research, 7, 21-43.

Wampold, B. E. (2001). The great psychotherapy debate: Models, methods, and findings. Mahwah, NJ: L. Erlbaum.

Zaider, T. I., Johnson, J. G., & Cockrell, S. J. (2000). Psychiatric comorbidity associated with eating disorder symptomatology among adolescents in the community. International Journal of Eating Disorders, 28, 58-67.

Zubieta, J. K., Demitrack, M. A., Fenick, A., & Krahn, D. D. (1995). Obsessionality in eating-disorder patients: Relationship to clinical presentation and two-year outcome. Journal of Psychiatric Research, 29, 333-342.

Darcy Haag Granello, Ph.D., LPCC, is an associate professor of Counselor Education in the School of Physical Activity and Educational Services at The Ohio State University. Email: [e-mail]. Laura Hill, Ph.D., is the CEO/CCO of The Center for Eating Disorders and Psychotherapy in Worthington, OH.