Benchmarking clinical outcomes: soliciting feedback from clients at intake and again at regular intervals can enable EAPs to benchmark care and outcomes and provide services when and where they can do the most good.

In 1999, PacifiCare Behavioral Health (PBH), a managed behavioral healthcare organization with 4 million members across the United States, implemented ALERT, a clinical information system for measuring and managing treatment outcomes during the course of outpatient psychotherapy ALERT is based on one basic principle--namely, that there is immense value in enabling clients to provide feedback on outcomes during the course of outpatient care. A growing body of evidence suggests that clinicians who have access to information from the outcome questionnaires are better able to assess for suicidal ideation and substance abuse problems and keep at-risk clients engaged in treatment until they improve.

The ALERT system monitors outcomes using the life status questionnaire (LSQ) for adults and the youth life status questionnaire (YLSQ) for children and adolescents (the abbreviation Y/LSQ will be used when referring to both measures simultaneously). These questionnaires are completed at set intervals by the client or, in the case of younger children, by a parent or other adult. Both measures demonstrate high reliability and validity and both have been subjected to extensive field testing with a wide variety of treatment populations and diagnoses, so their psychometric properties are well understood.

The questionnaires are brief (30 items), thereby providing an efficient means of gathering information directly from clients. Completed Y/LSQs are faxed to a central, toll-free number for processing. The ALERT system then notifies clinicians when clients appear at risk for a poor outcome or when risk factors such as suicidal thinking and substance abuse are identified.

PBH encourages clinicians to keep these high-risk clients engaged in treatment and offers to certify more intensive services as needed. This approach is grounded in the knowledge that the likelihood of improvement rises if clients remained engaged in treatment. ALERT thus uses data from the questionnaire to increase utilization where it can be expected to do the most good.

This article summarizes key findings from a comprehensive analysis of PBH's dataset on the effectiveness of the care provided by our network clinicians. For purposes of this article, outcomes are assessed based on intent to treat rather than using predetermined criteria for treatment completion--that is, all cases are included in the evaluation of effectiveness, even if the client leaves treatment after only three sessions. This encourages the clinician to keep the client engaged in treatment until adequate benefit is realized.

While the data cited in this article come primarily from outpatient psychotherapy funded through health plans, the findings also apply to EAP populations. In general, samples collected from clients receiving EAP services exhibit slightly less distress as measured by the outcome questionnaires than clients utilizing mental health benefits. Even so, a significant percentage of clients seeking psychotherapy services report relatively few and/or mild symptoms, a level of distress more characteristic of the general community population than of individuals seeking counseling services. After adjusting for differences in severity at intake, as is described below, EAP cases show patterns of improvement similar to those receiving outpatient psychotherapy, with most of the improvement occurring within the first few sessions.

BENCHMARKING EXPECTED OUTCOMES

Evaluating psychotherapy and counseling effectiveness involves benchmarking and measuring change using pre-treatment (intake) and post-treatment scores on the Y/LSQ. In this case, post-treatment scores are obtained from the last session for which an outcome questionnaire is available. Unlike controlled studies, however, the length of treatment varies greatly, from just a few days to many months.

Simple comparisons of pre- and post-therapy scores on the Y/LSQ can be misleading due to the variability in cases. This is particularly problematic when the types of diagnoses and severity of symptoms treated differ significantly from clinician to clinician. A client with a very mild case of adjustment disorder clearly cannot be expected to have the same outcome as a client with severe major depression.

Clients with high intake scores average more change than clients with low intake scores. This is partly a statistical artifact known as regression to the mean. The change measured within the PBH system, however, exceeds what is expected from regression to the mean, reflecting the fact that behavioral health treatments are designed to alleviate severe symptoms.

On the other hand, clients reporting mild symptoms that are consistent with normal variations in mood report relatively little improvement. In fact, clients with intake scores below the clinical cutoff--which indicates that the clients are more characteristic of a non-clinical sample--tend to experience higher scores over time, indicating increasing rather than decreasing symptoms.

As might be expected, there is a wide range in the severity of symptoms among clients seeking treatment. Almost one-third of the cases in the ALERT sample had scores below the clinical cut-off score on the Y/LSQ; at the other end of the continuum, 30 percent of adults and 25 percent of children/adolescents had intake scores classified as severe. (By way of comparison, fewer than one in 20 individuals from the normative nontreatment community sample have scores this high.) Fortunately, clients with scores in the severe range tend to experience rapid improvement.

Intake scores vary greatly depending on diagnosis. The most common diagnoses are adjustment disorders (26 percent) and depressive disorders excluding hi-polar disorder (46 percent). Adjustment disorders and depression account for approximately 70 percent of the diagnoses in each severity range, though the portion of each changes with severity. Among clients with scores below the clinical cutoff at intake, 39 percent are diagnosed with adjustment disorders and 30 percent with depression. Among those with scores in the severe range, 60 percent are diagnosed with depression and only 13 percent with an adjustment disorder.

Currently, more than 7,000 clinicians are using the Y/LSQ questionnaires and contributing data to the ALERT database. Because of the large sample sizes involved, it is possible to develop benchmarks for expected outcomes. Each client's change on the outcome questionnaire is compared to the expected change based on the results from all other cases with similar diagnoses and intake scores. Benchmarking outcomes permits PBH to monitor the progress of each individual as well as evaluate the clinicians in the network.

To account for this wide variance in intake scores and diagnoses, ALERT makes use of regression techniques and residualized change scores when benchmarking outcomes. The residualized change score for each client is the difference between the actual change (as measured by the difference between the pre- and post-treatment scores) and the expected change for comparable clients in the database.

Benchmarking the ALERT results against those obtained from controlled studies of psychotherapy and drug treatment presents additional challenges. Published research typically reports change as effect size, which is calculated by dividing the raw score change by the standard deviation of the scores at intake. Clinical trials, however, almost always have a more restricted sample than an outpatient sample due to the inclusion criteria for the studies. Clients who would score below the clinical cutoff on the Y/LSQ tend to be excluded from clinical trials because their symptom severity is not consistent with the psychiatric disorder under investigation.

Restricting the sample in this way leads to a smaller standard deviation of intake scores. Higher intake scores and a smaller standard deviation have the effect of greatly increasing the estimated effect size when compared to a much more heterogeneous sample, such as is found in the ALERT dataset.

To arrive at a more valid comparison between effect sizes within the PBH sample and the research literature, cases in the normal range were excluded from the analysis of effect size. Likewise, the standard deviation of intake scores from cases in the clinical range was used to calculate effect size.

PBH outcomes are quite positive: Across all age groups, the effect size for cases in the severe range is 0.93, which compares very favorably to results from published studies. Wampold's 2001 comprehensive analysis of outcomes concluded that "a reasonable and defensible point estimate for the efficacy of psychotherapy would be 0.80," and this is a relatively large effect size.

The overall effect size for all cases in the clinical range (as opposed to just the severe range) are somewhat more modest at 0.64 effect size, but it is important to bear in mind that the PBH clinical sample remains skewed toward the mild range, lowering overall effect sizes. If the sample contained a greater percentage of cases with at least moderate severity, the PBH effect size would be well within the range of effect sizes observed in successful clinical trials.

IDENTIFYING HIGH-VALUE CLINICIANS

The ALERT dataset provides a unique resource that permits exploration of various factors driving psychotherapy and counseling outcomes. Unlike data from clinical trails using experimental designs, the ALERT repository permits an understanding of the importance of the individual clinician in ensuring the best outcomes for clients.

It is clear that the single most important factor is the quality of the network, meaning the effectiveness of the clinicians. Clients treated by clinicians of above-average effectiveness achieve much more change at lower cost than those treated by relatively ineffective clinicians. In addition, multidisciplinary group practices appear to provide superior value when compared to solo practitioners, a finding that has been consistent since the initiation of the program. Group practices average more than 20 percent greater change per case than solo practitioners while also averaging fewer sessions. This results in a 30 percent lower cost per case for group practices.

The data also show that clinicians and clients do a good job (on average) of determining how much treatment is needed. As a result, PBH gives clinicians broad latitude when planning treatment, authorizing additional sessions each time a Y/LSQ is submitted.

While the choice of clinician is the single most important factor in determining outcome, the data provide evidence that outcomes for some clients may be improved by following recommendations contained in various treatment guidelines. Notably, clients with moderate to severe symptoms report more improvement with a combination of a medication and psychotherapy than with psychotherapy alone.

In the final analysis, however, outcomes will not be improved simply by encouraging adherence to treatment guidelines. For the most part, clients appear to be receiving treatment that is well suited to their individual needs. Overall, the ALERT data reveal little difference in outcomes based simply on method or duration of treatment, with the exception of some cases reporting symptoms in the severe range.

The individual clinician remains the primary driver of outcomes, with large and reliable differences existing between clinicians. In fact, a client treated by a clinician in the upper quartile for outcomes averages over 30 times as much improvement per dollar expended than those seen by a clinician in the bottom quartile. The ability to identify such high-value clinicians and to steer more referrals in their direction is the key to improving outcomes while managing costs effectively.

References

Brown, G.S., G.M. Burlingame, and M.J. Lambert. 2001. "Pushing the quality envelope: A new outcomes management system." Psychiatric Services, 52(7): 925-934.

Brown, G.S., R. Herman, E. Jones, and J. Wu. 2004. Improving substance abuse assessments in a managed care environment." Joint Commission Journal on Quality and Safety, 30(8): 448-454.

Brown, G.S., E. Jones, W. Betts, and J. Wu. 2003. "Improving suicide risk assessment in a managed-mare environment." Crisis, 24(2): 49-55.

Burlingame, G.M., B.W. Jasper, and G. Peterson. 2001. Administration and Scoring Manual for the YLSQ. Wilmington, Del: American Professional Credentialing Services.

Lambert, M.J., N.B. Hansen, and A.E. Finch. 2001. "Patient-focused research: Using patient outcome data to enhance treatment effects?' Journal of Consulting and Clinical Psychology, 69: 159-172.

Lambert, M.J., D.R. Hatfield, and D.A. Vermeersch. 2001. Administration and Scoring Manual for the LSQ. East Setauket, N.Y.: American Professional Credentialing Services.

Lambert, M.J., and B.J. Ogles. 2004. "The efficacy and effectiveness of psychotherapy." In Bergin and Garfield's Handbook of Psychotherapy and Behavior Change. New York: John Wiley & Sons, 139-193.

Lambert, M.J., J.L. Whipple, and D.W. Smart. 2001. "The effects of providing therapists with feedback on client progress during psychotherapy: Are outcomes enhanced?" Psychotherapy Research, 11: 49-68.

Vermeersch, D.A., M.J. Lambert, and G.M. Burlingame. 2002. "Outcome questionnaire: hem sensitivity to change." Journal of Personality Assessment, 74: 242-261.

Wampold, B.E. 2001. Great Psychotherapy Debate: Models, Methods and Findings. Mahwah, N.J.: Erlbaum.

Ed Jones is vice president and chief clinical officer for PacifiCare Behavioral Health (PBH), a managed behavioral healthcare organization with 4 million members. A licensed psychologist with 20-plus years of clinical experience, he has served in a wide range of direct service and management roles. He represents PBH on the board of the American Managed Behavioral Healthcare Association and is chairman of AMBHA for 2004.

G. S. (Jeb) Brown is founder and director of the Center for Clinical Informatics, a consulting firm specializing in outcomes management for behavioral healthcare. His current projects include the development of the ALERT outcomes management system for PacifiCare Behavioral Health and the Signal System for Resources for Living. He continues to see clients for psychotherapy a few hours a week, and he does measure his own outcomes.