Printer Friendly

Uniting practice-based evidence with evidence-based practice: Utah has brought all stakeholders together in a consumer-focused outcomes measurement system.

Administrators at Utah's Division of Substance Abuse and Mental Health (DSAMH) have implemented a bold initiative: managing all statewide mental health services on the basis of real-time outcomes of services provided by approximately 1,500 clinicians. Utah elected to implement a consumer-centered outcomes system repeatedly shown to improve outcomes and reduce treatment failures in six rigorous randomized clinical trials. (1-6)

Mark Payne, DSAMH director, states that the division's primary goal is to improve care and access to services by pushing beyond the mere provision of evidence-based practices (i.e., laboratory clinical trials). In the division's view, using evidence-based practices or "best practices" for specific treatments is only one way to maximize patient benefit and, by itself, is inadequate for ensuring quality of care for the individual consumer. A critical and complementary procedure is to manage care by examining practice-based evidence.

In short, practice-based evidence consists of real-time patient outcomes being delivered to clinicians immediately before treatment sessions and using outcome data on all patients to make decisions about effective interventions. DSAMH believes that a practitioner should know the consequences of behavioral healthcare for each individual receiving treatment. This belief is founded on solid evidence from clinical trials, in which feedback on individual patient progress during therapy improves the eventual outcome, produces more cost-effective treatment, and reduces waiting times for treatment. Progress data inform decisions to discontinue more expensive care when maximum benefit is reached (i.e., clinicians have more information to consider stepping up or stepping down care).

Using evidence from the real world and laboratory research, Utah partnered with a vendor of outcome measurements (OQ Measures, LLC) last year to implement a five-minute self-report outcome measurement across its entire care continuum (inpatient, outpatient, residential, etc.) and with all facilities contracting with the state, as well as state-run facilities. A significant advantage to this approach is the ability to monitor patients' progress regardless of the care setting.

When the partnership between the state and vendor began, two of the state's mental health centers already were having success with using the vendor's software package, which collects self-report assessments before each treatment session using a handheld PDA (figure) or computer kiosk, or paper survey. The software system provides realtime (five seconds from entry) reports to clinicians on a patient's "mental health vital signs," which are tracked based on three dimensions: adults' symptomatic distress, interpersonal relations, and functioning in daily activities. Adolescents and parents/guardians provide parallel information on age-normed questionnaires.

Most importantly, the scoring software applies empirically derived and tested algorithms that identify consumers whose treatment response is off-track for a positive outcome. The software system then alerts the clinician with an electronic or printed report that identifies clients who need particular attention--i.e., cases at risk for treatment failure.

A similar procedure is used in primary care to monitor blood pressure. A high blood pressure reading may result in an initial prescription of diet and exercise. If that works, there's nothing more to be done. If blood pressure remains high, the physician may intensify treatment by prescribing medication. Behavioral health organizations can systematically apply the same management procedures in mental health and substance abuse care. In Utah, when clients not on track for a positive outcome are identified (about 20%), the software provides additional assessments for problem solving, along with a decision tree and suggested interventions (clinical support tools) for such cases.


The most distinctive feature of DSAMH's system is its central focus on collecting data to provide clinicians with real-time, consumer-focused outcomes. Using real-time outcomes is in stark contrast with the typical practice of collecting and storing outcome data for quarterly or annual administrative reports. Nevertheless, the accumulated data can support subsequent practice research that examines which treatments or services are particularly effective, as well as support regular administrative reports. Thus, practice-based evidence ultimately drives initiatives to improve service delivery by examining treatment center and regional variations in outcomes. The greatest advantage of this type of research is that it relies on what actually takes place in clinical practice.

A core principle of practice-based evidence is that all services are assessed to determine if they have their intended impact--improving patient well-being. Since practice is the core driver, it meets both practitioners' and managers' desires to provide quality services assessed in part by repeated consumer-based outcome assessments. However, practitioners' adoption of practice-based evidence in service delivery is crucial, as they strive to generate solutions to improve outcomes at the local level with individual clients. This is why each Utah clinician and administrator receives brief training in how to use the software system to improve their patients' clinical outcomes.

Practice-based measurement systems need to realistically fit into daily clinical practice to ensure continuous use. Adopting routine measurement systems requires using information technology systems. Practice-based evidence makes the individual client central because predicting the course of progress in treatment is based, in part, upon his/her initial mental health vital signs. More specifically, the DSAMH's practice-based evidence relies upon sophisticated statistical procedures applied to large clinical data sets drawn from practice research in Utah and insurance companies that cover more than 17.5 million lives across the nation. These statistical procedures produce a predicted trajectory of change for clients who enter treatment at varying levels of distress. These predicted trajectories are used for triage or for comparison against a client's actual progress, supporting clinical decisions during treatment. In short, predicted change trajectories provide a continuous reference or benchmark against which to evaluate an individual patient's progress. Using statistical information on expected treatment response parallels practitioners' use of tacit clinical information to guide them through successive clinical decision-making processes, supplying them with greater predictive accuracy.

At the organizational service level, the focus moves to groups or clusters of clients rather than individual clients. Thus, rather than providing feedback on the individual client, aggregate feedback data are provided to assist in quality of service delivery decisions on a macro level.

For instance, the Utah State Hospital had used aggregate outcome data to calculate the average level of symptom distress for patients at discharge using multiyear in-house data. The hospital now uses target scores derived from self-and practitioner-completed outcome assessments as one "trigger" for initiating discharge planning discussions with patients, inpatient treatment teams, and the referring community mental health centers. Outcome data with target scores now provide a hospital-wide metric that reduces unit or practitioner preferences (e.g., length of stay, medication, etc.) based on patient level of functioning--outcomes provide common mental health vital sign metrics. Since every center referring to the hospital now uses the same mental health vital signs, historical trends in outcome scores (symptom distress) assist in decision making between facilities and, of course, for continuous monitoring.

When aggregate outcome data are available, program and client characteristics can be examined as potential mediators or moderators of outcomes within a service setting. One component within any service that has received relatively sparse attention is the practitioner. Traditional treatment-focused research aims largely to make therapist contributions irrelevant. However, Okiishi et al, using a database on 71 practitioners, found considerable variation between average rates of clients' change, with top therapists showing average rates of client recovery twice those of bottom-ranked therapists. (7) The top therapists also had half as many patients whose condition deteriorated. None of the differences could be attributed to the use of specific types of psychotherapy. These and similar findings underscore the urgent need to address the issue of therapist effectiveness, which is wholly consistent with Utah's new practice-based evidence approach. While it seems logical that some therapists are more effective than others, these differences typically are not measured and ignored.

Within practice-based research, the aim is to compare the processes and outcomes of targeted services against a standard--i.e., reference points that can be used to interpret data. In fact, the term benchmark originates from artisans who would mark a workbench to make measurement of work in progress easier. Benchmarking systems are sometimes derided because of the simplicity of the measures used. However, using consistent procedures within a culture that encourages outcomes monitoring will allow Utah to learn from regions of the state having unusual success. Benchmarking at the organizational service level not only requires comparing services with similar structure and function, but also organizations with similar cases. When cases across services are not comparable, information known to predict outcomes (such as relapse history, severity, and comorbidity) can be used to adjust comparisons between services. Such comparisons, given an appropriate organizational framework, can lead to improved practice and patient outcomes when administrators and practitioners learn from the collected data.

Benchmarking also can be used to compare an organization's effectiveness in a particular clinical setting against the outcomes obtained in controlled efficacy studies. This is a key component in bridge building between evidence-based practice and practice-based evidence. The aim should be to carry out benchmarkinginvolvingreal-world comparisons of services against specified benchmarks from clinical research.

Sherri Wittwer, executive director of the National Alliance on Mental Illness-Utah, states that "Utah's outcome system provides consumers with a real, tangible voice in their own treatment both verbally [through therapy] and in written form [through the software], making treatment decisions more self-directed."

NAMI-Utah, DSAMH, and the software vendor have partnered to apply the federal Substance Abuse and Mental Health Services Administration's Ten Fundamental Components of Recovery to the Utah system. Specifically, consumer focus groups developed ten recovery indicators that parallel the SAMHSA model (e.g., hope, satisfying relationships, adequate housing, employment, etc.) using their own "consumer language." These focus groups are working to identify items in the software instruments that tie their personal experience with SAMHSA's recovery components, with the goal of integrating outcome and recovery assessments to create the possibility of new mental health vital signs. Wittwer notes that the key question underlying their cooperative initiative is to ultimately understand how positive outcomes measured from a traditional perspective (i.e., the mental health vital signs measured by the software) relate to recovery indicators advocated by SAMHSA, NAMI, and other consumer advocates. Heretofore, there has been few links between the consumer-generated recovery movement and traditional outcome measures.

Monitoring patient treatment response, intervening with clinical support tools when problems occur, and monitoring the effects of clinical care at the program level are activities on the cutting edge of behavioral healthcare. They are the hallmark of administrators committed to excellence and attempting to create a culture of learning from data. The combined effort of administrators, clinicians, consumer advocates, and researchers in Utah is an unusual and promising development in wedding practice-based evidence and evidence-based practice within a large service delivery system.

Dr. Lambert is a Past-President of the Society for Psychotherapy Research and former Associate Editor of the Journal of Consulting and Clinical Psychology. He has published more than 175 peer-reviewed articles or books on the effects of psychological treatments and the measurement of treatment outcomes.

Dr. Burlingame's work in group psychotherapy was recognized with the Group Psychologist of the Year Award by the American Psychological Association in 2006 and the Outstanding Contributions to Group Psychotherapy Award from the American Group Psychotherapy Association in 2003. He also received in 2006 the Centennial Global Service Award from Loma Linda University. He has published more than 100 peer-reviewed publications and books in the area of group psychotherapy process and outcomes.

Drs. Lambert and Burlingame are principals in OQ Measures, LLC. To contact the authors, e-mail and


1. Harmon SC, Lambert MJ, Smart DW, et al. Enhancing outcome for potential treatment failures: Therapist-client feedback and clinical support tools. Psychother Res 2007;17:379-92.

2. Hawkins EJ, Lambert MJ, Vermeersch DA, et al. The effects of providing patient progress information to therapists and patients. Psychother Res 2004;31:308-27.

3. Lambert MJ, Whipple JL, Smart DW, et al. The effects of providing therapists with feedback on client progress during psychotherapy: Are outcomes enhanced? Psychother Res 2001;11:49-68.

4. Lambert MJ, Whipple JL, Vermeersch DA, et al. Enhancing psychotherapy outcomes via providing feedback on client progress: A replication. Clin Psychol Psychother 2002;9:91-103.

5. Slade K, Lambert MJ, Harmon SC, et al. Enhancing psychotherapy outcomes: The effects of immediate electronic feedback and Clinical Support Tools. Under review.

6. Whipple JL, Lambert MJ, Vermeersch DA, et al. Improving the effects of psychotherapy: The use of early identification of treatment failure and problem solving strategies in routine practice. J Couns Psychol 2003;58:59-68.

7. Okiishi JC, Lambert MJ, Eggett D, et al. An analysis of therapist treatment effects: Toward providing feedback to individual therapists on their clients' psychotherapy outcome. J Clin Psychol 2006;62(9):1157-72.



Michael J. Lambert, PhD, is a Professor of Psychology and holds an Endowed Chair in the College of Family, Home, and Social Sciences at Brigham Young University in Provo, Utah.


Gary M. Burlingame, PhD, is a Professor of Psychology at Brigham Young University.

COPYRIGHT 2007 Vendome Group LLC
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2007 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Title Annotation:perfromanc measurement in Division of Substance Abuse and Mental Health
Author:Lambert, Michael J.; Burlingame, Gary M.
Publication:Behavioral Healthcare
Article Type:Cover story
Geographic Code:1U8UT
Date:Oct 1, 2007
Previous Article:A plan for a national consumer memorial: it will rise on the grounds of Saint Elizabeths in Washington, D.C.
Next Article:Treatment in a new light: a New York City organization creates a life-affirming space for addiction detox and rehabilitation.

Terms of use | Copyright © 2016 Farlex, Inc. | Feedback | For webmasters