Printer Friendly

An implementation story: moving the GAIN from pilot project to statewide use.

One component of Missouri's Enhancing Supports for Substance Abusing Youth (ESSAY) project, which was funded by a Strengthening Communities for Youth grant from the Substance Abuse and Mental Health Services Administration, Center for Substance Abuse Treatment (SAMHSA-CSAT), was to pilot the Global Appraisal of Individual Needs assessment (GAIN; Dennis et al. 2006) at an adolescent substance abuse treatment agency. The grant awardee, the Missouri Department of Mental Health, Division of Alcohol and Drug Abuse (DMH-ADA), wanted to determine whether the GAIN might be a viable instrument to use statewide, since their then-current instrument was of unknown reliability and validity. The GAIN is an evidence-based assessment (EBA): an instrument that demonstrates validity, reliability, and utility for a particular diagnosis or purpose and that is sensitive to client change over time (Barlow 2005; Kazdin 2005; Mash & Hunsley 2005). After several years of piloting the GAIN and building consensus among providers across the state, the DMH-ADA mandated use of the GAIN at all state-contracted adolescent substance abuse programs. The DMH-ADA contracted with Chestnut Health Systems, a behavioral health care company that offers substance use and mental health treatment services as well as conducting training and research, to provide a comprehensive training and implementation package. This article describes the process of implementing the GAIN from the initial pilot site to statewide use.

IMPLEMENTING EVIDENCE-BASED ASSESSMENTS

Implementation refers to the process of installing new practices into treatment settings, ideally while maintaining their effectiveness (Gotham 2004). Putting a new program or practice in place is a lengthy, complex process that can take several years to complete fully (Fixsen et al. 2005). Implementing a new assessment system is a similarly complex process that requires much more than handing clinicians a paper copy of the instrument. Statewide implementation of a new assessment system adds levels of complexity requiring coordination between local treatment providers, purveyors of the assessment, and the state-level agency mandating the change.

Evidence-based implementation strategies are not established for evidence-based practices (EBPs) or EBAs (Gotham 2006). However, a number of implementation models have been derived from theory, expert consensus, and extant research (e.g., Simpson & Flynn 2007; Aarons & Carmazzi 2005; Fixsen et al. 2005; Greenhalgh et al. 2004; Frambach & Schillewaert 2002). Following an exhaustive review of implementation theory and research, Fixsen and colleagues (2005) developed two useful frameworks: stages of implementation and core implementation components. They describe implementation as having six stages (see Table 1). Exploration and adoption refers to identifying the need for a change, exploring options for new practices or assessments, and making the decision to implement the new practice or assessment. During program installation clinicians and agency administrators complete preparatory tasks such as training staff and making policy and procedure changes to fit the new practice or assessment. They may work with "purveyors," people or agencies with expertise in the practice or assessment (or in technology transfer) who assist or direct the implementation. Initial implementation refers to the first period in which the new practice or assessment is used and is often marked by logistical difficulties and personnel issues as staff and clients adjust to the change. When the new practice or assessment is fully in use, it is considered to be in full operation. Innovation refers to a stage, after all aspects of the original program are in place, when an EBP or EBA can be adapted to meet local needs. Finally, a program or agency can move into a sustainability stage where further resources, supervision, and training are required to continue long-term use of the practice or assessment.

Fixsen and colleagues (2005) also describe a set of core implementation components (see Figure 1) including staff selection, preservice and in-service training, consultation and coaching, staff evaluation, facilitative administration, and systems interventions. Core implementation components are essential areas in which change must take place in order to implement a new practice or assessment. Fixsen and colleagues also call these components "implementation drivers" because they push implementation forward, are integrated (influence one another), and are compensatory (strengths in one component can make up for weaknesses in other components).

Only a few studies have examined assessment implementation (e.g., Poulsen et al. 2005; Quick & Fonteyn 2005; de Rond, de Wit & van Dam 2001), and nothing suggests that the implementation of EBAs differs significantly from that of EBPs. This case study uses Fixsen and colleagues' (2005) stages of implementation and core implementation components to illustrate the process of implementing the GAIN in Missouri.

MISSOURI'S STATE-CONTRACTED ADOLESCENT SUBSTANCE ABUSE TREATMENT PROGRAMS AND THE ESSAY PROJECT

Missouri has a state-level Department of Mental Health that includes the Division of Alcohol and Drug Abuse (DMH-ADA). The DMH-ADA contracts with local treatment providers and sets policies and procedures for assessment and treatment that are funded through Missouri's Medicaid program and DMH-ADA's purchase-of-service system. At the time of this implementation, the DMH-ADA contracted with eight substance abuse treatment agencies to operate fifteen adolescent substance abuse treatment programs throughout Missouri. Except for three programs in urban St. Louis and Kansas City, the other programs were in suburban or rural locations. In total, the agencies admitted 2,872 clients in FY2007, and individual programs admitted between 80 and 1521 clients. Each program offered three levels of care: regular outpatient, intensive outpatient, and day treatment with or without residential support.

[FIGURE 1 OMITTED]

The ESSAY grant, awarded in late 2001, was designed to develop a comprehensive coordinated system of care in the St. Louis, Missouri area to ensure that substance-abusing adolescents received around-the-clock services and support from traditional service providers and natural caregivers. Project components included piloting an EBA, implementing wraparound care and several EBPs, and piloting an advanced management information system. The system was also designed to identify service gaps and facilitate changes at the local and state levels. ESSAY partners included: the Missouri Institute of Mental Health (MIMH), a research institute of the University of Missouri--Columbia School of Medicine, to provide evaluation and management information system services; New Beginnings, Inc., which operates an adolescent substance abuse treatment program in inner-city St. Louis, to pilot the project; the Substance Abuse Committee from the St. Louis System of Care Board to identify system issues that presented barriers to treatment or additional needed services; and the state-level Adolescent Provider Workgroup, comprised of representatives from DMH-ADA contracted adolescent substance abuse treatment providers, to track ESSAY project progress.

THE GAIN

The GAIN is a family of comprehensive, validated, semistructured assessments. It has been adopted in the United States and Canada by 10 state adolescent treatment systems, four counties, and over 500 individual agencies, including substance use and mental health treatment providers, justice agencies such as drug courts and probation departments, and child and family welfare departments. The full biopsychosocial intake assessment, the GAIN-Initial (GAIN-I; Dennis et al. 2006), collects information in eight areas (background, substance use, physical health, risk behaviors, mental health, environment, legal, and vocational) from adolescents and adults. Questions in each section cover the of recency of problems and breadth of symptoms as well as lifetime total, recency, and frequency of service use, all recorded in numbers of days or times. The Full GAIN-I takes approximately 120 minutes to administer. A Core version of the GAIN-I, which includes a smaller set of items, can take 60 to 90 minutes to administer, and specific projects design their own Core version that includes Core items plus other items from the Full GAIN-I. The follow-up version (M90) targets change during the past 90 days, with administration averaging between 45 and 60 minutes. The GAIN-Quick is a basic assessment to identify those in need of either a brief intervention or a longer, more detailed assessment and is used to assist substance use, mental health, and physical health treatment staff in making effective referral and placement decisions. (The GAIN-QM is a follow-up instrument for the GAIN-Q.) The GAIN Short Screener is a two-page, five-minute screening tool used with a general population to identify people likely to have an internalizing, externalizing, or substance use diagnosis.

The GAIN-I contains 103 scales, with the main scales having alphas over .90 and the subscales generally having alphas over .70. Substance use reporting methods were compared (self-report, urine, and saliva) to determine the validity of the GAIN's self-reported items, and each method was largely consistent with the combined estimate of any use (kappa of .59 for self-report, .69 for urine, and .56 for saliva) and was reliable in a test-retest study (rho = .94; Lennox et al. 2006). Self-reported treatment-utilization data from the GAIN are largely consistent with agency records (r = .78; Godley et al. 2002). Diagnoses based on the GAIN have good test-retest reliability for substance use disorders (kappa = .60; Dennis et al. 2006) and accurately predict independent and blind staff diagnoses of co-occurring psychiatric disorders (kappas 0.69-1.00; Shane, Jasiukaitis & Green 2003). Detailed norms and scale psychometrics of the GAIN for adults and adolescents overall and by level of care are available (e.g., Dennis et al. 2006).

Two important aspects of the GAIN system are the accompanying software and the clinical reports. The computerized version of the GAIN uses the Assessment Building System (ABS) software, which allows users either to administer the assessment on paper and later data-enter responses or administer the assessment interactively by asking questions as they appear on the computer screen and entering the client's responses in real time. Following data entry or interactive administration, ABS generates several clinical reports to assist in treatment planning and level-of-care placement. The GAIN Recommendation and Referral Summary (G-RRS) is a detailed narrative biopsychosocial report that comprehensively and descriptively summarizes GAIN information into a fully editable Microsoft Word document. The Individualized Clinical Profile (ICP) is a more detailed Microsoft Access report documenting the client responses (i.e., scale scores, interpretive ranges) that lead to the diagnostic impressions and comments reported in the G-RRS. The Personalized Feedback Report (PFR) is a worksheet listing the client's reported reasons for quitting and is used with the client to target areas that motivate the client to stop using. Finally, the validity report, used during or after the interview, flags responses in the assessment that are not consistent, prompting the interviewer to find out more about that area of the client's life.

SIX STAGES OF IMPLEMENTATION

The next sections describe the implementation of the GAIN from initial pilot site to statewide use following Fixsen and colleagues' six-stage model.

1. Exploration and Adoption

The exploration and adoption stage typically includes identifying needs for new practices, learning about available practices, and building consensus toward a decision to adopt a new practice (or assessment). After surveying assessment instruments, and on recommendation by SAMHSA-CSAT, ESSAY staff decided to implement the GAIN at New Beginnings. Shortly thereafter, ESSAY staff introduced the GAIN to the statewide Adolescent Provider Workgroup. At that time, adolescent treatment providers used the Missouri Adolescent Comprehensive Substance Assessment (MACSA), which was part of the state's internet-based data system. The MACSA is a rating scale developed through consensus of the adolescent providers; it is neither standardized nor comprehensive. Although DMH-ADA staff and some providers wanted an EBA, most providers wanted to retain the MACSA because it was home-grown, already in place, and quick to administer (20 to 30 minutes). Initial reactions to the possibility of adopting the GAIN were fairly negative, including concerns about the GAIN's length and validity.

Despite this early resistance from the provider network, the ESSAY project staff believed the GAIN to be the best available assessment, and it was implemented at New Beginnings in January 2003. A number of implementation issues arose during the pilot phase, mostly related to staff attitudes, values, and readiness to change. For example, clinicians were concerned about client rapport and engagement if their first interaction was a standardized assessment. It helped to remind the staff that the GAIN is a semistructured interview: items need to be asked in order and as written, but the interviewer can then explain items, answer client questions, and clarify responses. This method shows the interviewer cares about getting accurate information, and hearing about clients' lives helps establish rapport.

Clinicians were also concerned that clients would deny or underreport substance use, in part because most clients were in the criminal justice system. Conversations with clinicians emphasized that being up-front and direct with adolescents helps to establish rapport, leading to accurate self-reporting. Clients are shown respect for their opinions while being gently but firmly questioned when their responses seem inconsistent. Also, MIMH evaluation staff examined agreement between self-reported marijuana use on the GAIN and urine drug screen results for forty-three adolescents. The adolescents were informed that they would be urine tested prior to completion of the GAIN. The agreement rate between self-reported marijuana use and urine test results was 86% and the kappa coefficient was 0.70, indicating excellent agreement. Only one participant reported no use but had a positive screen, and five acknowledged marijuana use but had negative drug screens. Disclosure of marijuana use was not related to criminal justice status. This information assured staff of the accuracy and reliability of GAIN responses.

The GAIN's length is a frequent initial concern. It was helpful to remind clinicians that the techniques acquired during the GAIN certification process decrease administration time by 25% to 31% (White 2006). Also, clinicians were reluctant to offer breaks during the interview because they increased overall administration time. However, providing breaks, having snacks or small toys to occupy respondents' hands, and being truthful when clients ask "How much longer?" decrease overall administration time, since staff and clients focus better if given breaks and amenities. Some clinicians pointed to the MACSA's shorter administration time, but it was noted that the GAIN provides broader and deeper information from which to make clinical decisions, and the computer-generated clinical reports greatly assist with treatment planning. Over time New Beginnings' staff appreciated the comprehensive information provided by the GAIN, especially regarding co-occurring mental health issues. The G-RRS was used to develop treatment goals and outline wraparound service needs, and counselors used the PFR with clients during individual sessions. They reported that discussing clients' reasons for wanting to quit using substances as compiled on the PFR was helpful in motivating clients for treatment.

Over the following two years the New Beginnings clinical director regularly updated the Adolescent Provider Workgroup on the GAIN's utility and feasibility. The successful resolution of initial concerns, combined with other agencies' increasing knowledge about the GAIN, led to greater acceptance of the instrument. By spring 2005, DMH-ADA administrators were convinced of the GAIN's superiority over the MACSA, and several initially skeptical providers were convinced of the GAIN's benefits. DMH-ADA approached Chestnut Health Systems to assist in implementing the GAIN statewide. The political forces surrounding implementation of EBAs or EBPs can be complex and daunting and include players as diverse as consumers and families, treatment agencies, state funders, legislators, and academics and consultants. In Missouri, pressure to implement an EBA primarily resulted from DMH-ADA leaders, a few provider agencies, and consultants from MIMH. The ESSAY grant and suggestions from CSAT project officers provided an impetus to pilot a new EBA, leading the way for discussions across the state regarding changing the required assessment.

2. Program Installation

During program installation a clinician or agency prepares to adopt a new service by completing tasks such as training staff and making policy and procedure changes. Missouri's statewide GAIN implementation effort involved Fixsen and colleagues' core implementation components (2005; see Figure 1), and the GAIN Coordinating Center (GCC) served as the purveyors to direct and manage the implementation.

Consultation and coaching of state staff and provider agencies. Consultation and coaching occurred with state staff and provider agency directors, and with agency staff (see Consultation and Coaching of Staff, below). A GCC project coordinator oversaw implementation and installation and communicated with GCC staff (e.g., training, quality assurance [QA], and software staff). An implementation team consisted of the GCC assistant director and project coordinator; DMH-ADA division director, deputy director, director of clinical services, and training staff; and the ESSAY (MIMH) evaluation director, who led GAIN implementation at New Beginnings. Following several months of frequent communication through teleconferences and email, the GCC made an initial consultation visit, meeting with the implementation team and senior staff from the provider agencies. This served to disseminate basic information about the GAIN as well as to prepare for systems interventions required for statewide implementation.

Consultation included preparing agencies for the preservice training and quality assurance process. Checks for organizational readiness were conducted to ensure that agencies had necessary computer equipment. To help agencies identify staff for key roles and decide who to send to training, the GCC disseminated job descriptions that included important qualifications and duties for each role (e.g., GAIN Administrator, Local GAIN Trainer, and ABS Administrator). Another important step was to communicate fully in meetings and writing the expectations of providers and trainees (including "what ifs" such as "what if I miss part of the training?" or "what if I miss a deadline in the certification process after training?") and the help that would be available from the implementation team and the GCC. Preparing agencies for the training, including early communication of expectations and identification of potential barriers, was critical to creating and maintaining positive attitudes towards the change process.

Systems interventions. Fixsen and colleagues (2005: 29) state that "systems interventions are strategies to work with external systems to ensure the availability of the financial, organizational, and human resources required to support the work of the practitioners." In Missouri, system-level interventions included the DMH-ADA outlining the timeframe and requirements for implementation, specifying which items on the GAIN would be used, deciding how the GAIN would work with other required assessments, and revising billing limits for assessments. Originally, a nine-month implementation timeline was set, where personnel attended a training in September 2005, were given three- and six-month deadlines to complete different levels of certification (discussed later in Preservice Training), and were given time to incorporate the GAIN into their treatment system and set up accompanying software prior to the start deadline of July 1, 2006.

A workgroup including DMH-ADA staff, provider representatives, and the MIMH evaluation director reexamined the required assessment battery. The Full GAIN-I was reviewed to choose items for a Missouri-specific version, the MO GAIN Core. In addition to the MACSA, several ancillary questionnaires were deemed unnecessary because the information was part of the GAIN. The providers appreciated streamlining the assessment battery. Once the MO GAIN Core items were finalized, the GCC created a paper version and specialized software.

Although the MO GAIN Core was designed to take 60 to 90 minutes to administer and obviated the need for several ancillary assessments, it was longer than the previous assessment and required certified staff members. Thus, the DMH-ADA increased the assessment reimbursement, and tied it to completion of Administration certification (see later in Consultation and Coaching of Staff). In addition to appropriately compensating agencies for staff time, this monetary increase served two purposes. It decreased resistance by demonstrating consideration of clinicians' feelings about changing to the longer EBA, and it helped ensure that all staff members met the July 1, 2006 start date, since increased reimbursement was granted prior to the deadline for clinicians who achieved early certification.

Staff selection. Staff selection "is essential since it is at this level that evidence based practices and programs are actually carried out (or not)" (Fixsen et al. 2005: 37). A program implementing the GAIN typically selects staff to fill three roles: GAIN Administrator, Local GAIN Trainer, and ABS Administrator. Any number of personnel can be trained to administer the GAIN and there is no requirement for educational background or licensure, although a particular project may include one. Missouri initially required GAIN Administrators to be licensed substance abuse counselors.

Selecting Local Trainers is a more delicate process. Characteristics to look for include attention to organization and detail and a high level of respect within the agency. Local Trainers need to recognize areas for their trainees' improvement, provide constructive feedback, and have strong interpersonal skills. Achieving Local Trainer certification requires about eighty hours of training and QA work (which agency directors and clinical supervisors should take into account). The GCC recommends that each agency have at least two staff members certified in GAIN administration and trained to train others in case of staff turnover. Larger agencies may train staff members from each treatment site.

In Missouri's implementation each agency had at least two Local Trainers, which reduced each trainer's workload, and allowed each agency to sustain the GAIN if one trainer left. In two large agencies a larger pool of trainers was certified to respond to the organizations' needs and greater potential for turnover. The most challenging obstacle was the placement of Local Trainers in smaller and rural agencies. In agencies servicing a larger geographical area there was less access to trainers, and Local Trainers were more valuable and their loss more problematic for implementation.

Because the ABS software is an important part of implementation, appropriate staff members need to be selected at the state and local levels to manage ABS functions. Information technology (IT) skills and proficiency with software installation are essential for these roles. Missouri had a preexisting web-based data management system, so GCC support of ABS was limited to their specific requests. Each agency identified a local ABS Administrator to support software needs, upload data through a secure website, and communicate with the state-level data manager. The state-level data manager maintained contact between the agencies and the GCC regarding software needs and implementation, compiled and reviewed the data from across the state, and provided aggregate reports to the DMH-ADA.

Preservice training. The GCC provides specialized GAIN training. In September 2005, twenty-eight staff members from eight agencies and the DMH-ADA participated in GAIN training. This "train the trainer" model consists of three and a half days of instruction in which trainees learn and practice GAIN administration, quality assurance for certification, clinical interpretation and application of the GAIN, and how to use the ABS software. The GAIN training model follows best practices for training outlined by Fixsen and colleagues (2005: 41): "Effective training workshops appear to consist of presenting information (knowledge), providing demonstrations (live or taped) of the important aspects of the practice or program, and assuring opportunities to practice key skills in the training setting (behavior rehearsal)."

One example of the GCC's effective training practices is the use of "round robin" administration practice sessions. Trainees watch a live demonstration of a GAIN assessment (staff to staff, not a real client). Then trainees are assigned to small groups of five or six members to practice administering the GAIN to a trainer who acts as a client. Each trainee administers several pages of the assessment, and the trainer and other group members provide constructive advice on areas in which the trainee can improve and reinforce practices that the trainee followed correctly. These small-group sessions allow participants to become familiar with the instrument and ask questions about specific items and methods for administration.

Consultation and coaching of staff. The GCC certification process provides additional consultation and coaching after the initial training and a standardized staff evaluation process to ensure that trainees meet mastery criteria for administering the GAIN and training others to administer it. As described by Fixsen and colleagues (2005), coaching should include supervision, teaching while engaged in practice activities, and assessment and feedback.

The GCC uses a multilevel certification process. The first level, Coursework certification, is achieved upon completion of the initial GCC GAIN training. To reach the second level, Administration certification, GAIN trainees submit audiotaped interviews to the GCC QA team and receive detailed feedback, repeating the process until they achieve certification. The GCC QA reviewer evaluates whether the interviewer can add to or maintain the validity of the information collected from the client and rates the interviewer on four primary areas: documentation of client responses, instructions to the client, item administration, and engagement with the client. In the third level, Local Trainer certification, Administrators teach GAIN administration to staff at their agencies, provide QA feedback and coaching to these trainees, and recommend them to the GCC for Site Interviewer certification. During the Local Trainer certification process, Local Trainer candidates submit to the GCC QA team audiotaped submissions from their own trainees along with their written feedback. Certification is met when Local Trainers demonstrate that they can determine when their staff members have met mastery standards for GAIN administration.

Throughout the certification process the GCC QA team tracks each trainee's progress and sends deadline reminders via email and phone. Certification deadlines help ensure that trainees complete the process while information is fresh after training. In addition, the GAIN project coordinator makes agency calls, beginning about six weeks after training, to help with installation or other issues that may affect implementation based on the agency's clientele, levels of care, or other agency characteristics.

Of the twenty-eight trainees in the September 2005 training, twenty-two were assigned to become Local Trainers, and six were assigned to become Administrators. Those six were to serve as trainees for the Local Trainers at their agencies and have their submissions reviewed by those Local Trainer trainees instead of the GCC QA team. The twenty-two trainees were to send their first taped assessment to the QA team within two weeks of the training. They were assigned a three-month deadline (December 30, 2005) to reach Administration certification and three additional months to reach Local Trainer certification (March 30, 2005). Trainees who did not meet the requirements were required to repeat the entire training sequence unless a deadline extension was requested and granted. The six trainees who submitted tapes to Local Trainers were given a Site Interviewer certification deadline of three months following the submission of their first taped interview.

The success of the QA process is a combination of effective preparation, training, persistent consultation and coaching on the part of the QA team, flexibility on deadlines (by both the state and the GCC), and encouragement of senior staff at each agency. Twenty-seven of the twenty-eight staff members trained in September 2005 completed the QA process (one left her position), and the majority of trainees met assigned deadlines. An additional factor contributing to this excellent certification rate was that the DMH-ADA director of clinical services acted as a liaison to the trainees. Since he was also trained in the QA process, he was especially understanding of issues that trainees were experiencing.

Staff and program evaluation. Staff and program evaluations should include measures of context (required components or qualifications for a program to operate), compliance (use of the program or assessment), and competence (skill in providing the program or assessment) (Fixsen et al. 2005). At the agency level, staff evaluation can be sustained by having clinical supervisors or Local Trainers monitor GAIN administration over time. For example, staff members periodically submit an audiotaped assessment or tape every interview from which some are randomly reviewed. Local Trainers review the audiotapes and provide ongoing feedback. At the state level, the DMH-ADA developed several staff and program evaluation components, which are monitored by the state-level data manager. Measures of context in Missouri staff evaluations included a provision that only staff members who are GAIN certified may administer the GAIN and that all agencies must have at least one Local Trainer on staff. A measure of compliance was that agencies are able to bill only for intake assessments that have been uploaded to the DMH-ADA. Measures of competence were that agencies were required to submit GAIN certification documentation including training dates for their staff, and that agencies correct their data if asked to by the state data manager, who performs checks on the GAIN data submitted each month. (See Sustainability for more on the data process.)

3. Initial Implementation

The initial implementation stage is often full of logistical difficulties and personnel issues as staff and clients adjust to the change. By early June 2006 three of the eight agencies were using the GAIN for all intakes, and a few other sites were just beginning to use the GAIN. Several agencies asked for an extension of the July 1, 2006 deadline so that staff could have more time to make programmatic changes before putting the GAIN in place. Difficulties at this stage included having all Local Trainers and Site Interviewers complete certification; ensuring that ABS was fully installed and that staff felt comfortable using it and generating reports; integrating GAIN data and clinical reports into the treatment planning process; and developing procedures for and teaching ABS administrators how to upload data to the DMH-ADA.

In response, the DMH-ADA extended the implementation deadline to August 1, 2006. Also, the state-level data manager and ESSAY (MIMH) evaluation director conducted a series of four telephone conferences with representatives from each agency to discuss several of these issues including GAIN data editing and data entry procedures, troubleshooting ABS problems, and uploading data. One call included a clinical expert from the GCC who discussed using the G-RRS for treatment planning. By August, core agency personnel were trained and certified on the GAIN, ABS was installed, and state-level policies and procedures for using the GAIN as part of the EBA were in place.

4. Full Operation

Full operation, as defined by Fixsen and colleagues (2005), comes when the new practices and procedures are fully integrated into the program and clinicians have skill and expertise in their delivery. We found it a bit difficult to judge the line between initial implementation and full operation. By November 2006 all of the agencies in Missouri had been using the GAIN with all new clients for three months. In that sense, the GAIN was fully implemented; however, most agencies were still resolving implementation-related issues and continued to have significant concerns. In an informal survey completed by seven of the eight agencies, almost all reported that the GAIN provided more information about clients than the MACSA and that the clinical reports were very helpful in summarizing information and beginning treatment planning.

Ongoing challenges included administration and G-RRS report review time, staffing patterns for GAIN administration, and making the GAIN a seamless part of the assessment-to-treatment-planning process. More specifically, staff members and agency directors expressed ongoing concern about the assessment's length, although most reported that administration time was continuing to decrease as clinicians already certified in GAIN administration became even more familiar with the instrument and software. In addition, DMH-ADA offered on-site and telephone assistance from the state-level data manager regarding software issues. Regarding staffing patterns, New Beginnings, who had used the assessment for three years, suggested that smaller offices or agencies could assign a main clinician to administer the GAIN but have other interviewers administer the GAIN at least once per month so they could remain familiar with the assessment. Another site echoed this idea. Finally, revising the assessment-to-treatment-planning process is a long-term, complex change, so the DMH-ADA issued guidelines to clarify how agencies could make level-of-care determinations based on GAIN assessment profiles. Despite these solutions, the DMH-ADA decided to convene a meeting of the Adolescent Provider Workgroup in January 2007 to discuss helps and hindrances in implementation, and consider possible innovations. In retrospect, this time period was probably still initial implementation rather than full operation.

5. Innovation

Innovation should follow full operation. Only after all aspects of the original program or assessment are in place should agencies try to change or modify the program to meet local needs. In this case, innovation refers to the way in which the GAIN is implemented; the GAIN itself is not substantially changed. The GCC has found that attempts to modify the GAIN process during state and other large implementation projects typically come in two waves. First, prior to training, providers in a mandated situation may be resistant to changing the assessment and may call for changes to the GAIN or assessment plan. During and just after the training, trainees are excited about the GAIN as they discover its uses, but then can feel overwhelmed as they work toward certification. A second wave of calls for changes occurs closer to initial implementation as trainees push to complete certification. Encouragement by the implementation coordinator, agency administration, and state leaders is especially important to maintain trainee focus on certification and assure that the agency is preparing for procedural changes (e.g., using clinical reports in treatment planning). Generally, if allowed to vent, feel involved in the process, and given reassurance that the process will get easier, agencies make it through the more difficult early implementation stages and innovation does not occur until after full implementation.

Several times during initial implementation (in early summer 2006) and early full operation (October-November 2006), treatment provider staff brought concerns to the DMH-ADA, including the GAIN's length. They suggested innovations such as deleting some required items, allowing unlicensed staff to administer the GAIN, and having clients self-assess using ABS's interactive version. In November 2006, the GCC assistant director and project coordinator spoke with DMH-ADA staff and a few concerned providers to discuss possible adaptations. They asked agencies to keep track of possible item deletions and invited them to a GAIN roundtable in January 2007. This timeline allowed for six months of full operation (or initial implementation) before seriously discussing innovation.

In January 2007, the Adolescent Provider Workgroup met regarding possible innovations. After much discussion, three main changes were made to the assessment process. The DMH-ADA approved trained, GAIN certified technicians, rather than licensed substance abuse counselors, to administer the GAIN (echoing the GCC's standard). This change allowed higher-level staff to focus on reviewing assessment results and formulating the diagnostic summary and treatment plan. Also, the DMH-ADA expanded the required timeframe for assessment completion from three days or three visits to five days or five visits. This change allowed more flexibility in timing the assessment and more sensitivity to clients' needs (i.e., rapport building, crisis management). And, along with the timing change, the DMH-ADA stated that programs could begin billing for treatment services prior to completion of the assessment. Resolution of these three issues resulted in much greater provider acceptance of and satisfaction with the GAIN. Almost all providers dropped the issue of shortening the instrument.

6. Sustainability

Finally, in the sustainability stage, programs must address issues such as how to provide resources, supervision, and training over the long term in order to maintain and support the program or assessment (e.g., Fals-Stewart, Logsdon & Birchler 2004). In Missouri the seeds of this stage were planted throughout the implementation process: the policy changes enacted by DMH-ADA allow for increased reimbursement for the GAIN and increased staff training requirements, and the Local Trainer model ensures ongoing supervision and training of new GAIN Administrators. It is important for the DMH-ADA to maintain personnel who are GAIN trained as well. In addition to the keys to implementation already discussed, four supplemental pieces have shown early success in ensuring sustainability: (1) hiring a Regional Trainer who can train across agencies; (2) investing in a data management system that returns useful, aggregated information to agencies and the state; (3) providing advanced clinical interpretation training several months into full implementation; and (4) keeping a local implementation coordinator until the project has been sustained for several years.

Staff turnover can become an issue unless agencies have Local Trainers to train new interviewers. The GCC now offers a Regional Trainer certification in which a Local Trainer receives additional instruction (in person and through written and video materials) and supervised practice in training (as a trainer in a national train-the-trainer meeting). A Regional Trainer can train new interviewers across agencies (Local Trainers are limited to training at their own agency). States that have a Regional Trainer are better able to keep provider agencies staffed with certified Site Interviewers and thus boost the chances of sustainability.

States that invest in collecting and combining data across agencies have more ongoing success in sustaining the GAIN than those that do not. The GCC works with agencies to "clean" GAIN data by clearing up inconsistent or missing information and provides reports regarding the number of interviews completed and the characteristics of clients by agency and across the state. Agencies and the state can use the information for reports, presentations, and grant writing. Data management functions can be conducted by the GCC or a state data manager. In the latter case, the GCC has a data management training and certification program (typically requiring six to nine months).

Sometimes providers are overwhelmed by trying to complete certification and address early logistical issues when first implementing the GAIN, and while they print out the clinical reports, they may not use the GAIN clinically to its fullest potential. Thus implementation projects have provided advanced clinical training several months into full operation when clinicians have more experience using the GAIN. Advanced clinical training involves a two-and-a-half-day training on how to efficiently edit and then use the GAIN reports for staffings, level of care placement, and treatment planning. This training helps to fully incorporate the GAIN into clinical practice.

Finally, it is strongly recommended for a large statewide project such as this one that a local coordinator continue to monitor progress far into the sustainability stage. Since the exploration and adoption phase may take over a year, and the program installation and early implementation phases can last just as long, it is very important that a coordinator oversee the work through all stages of implementation rather than just until full operation.

IMPLICATIONS FOR IMPLEMENTING EBA

Further Recommendations for a Successful Implementation

The collaboration between the GCC and Missouri DMH-ADA highlights the concepts and principles discussed by Fixsen and colleagues as essential to the successful implementation of an EBP or EBA. In addition, Table 2 lists our recommendations for a successful implementation, most of which have been discussed throughout the article.

Although applied retrospectively, this implementation effort generally followed Fixsen and colleagues' (2005) stages of implementation. In this example, it was difficult to determine when the full operation stage began, and there was some cycling through the full operation and innovation stages. Based in part on practice-related barriers and some staff and director resistance to change, agencies revamped their processes following the innovations made by the DMH-ADA in January 2007, sending the implementation back toward initial implementation/full operation before moving toward sustainability. It would be extremely helpful to communicate the stages framework to agency and state-level administrators who might otherwise expect that implementation is a faster and easier process. Reminding providers that full integration of the GAIN can take at least six months assisted the DMH-ADA and GCC in managing concerns, anxiety, and the wish to innovate too early. Similarly, although efforts to change are met with barriers, resistance, and difficulty, when it is made clear from the onset that these are expected and will be dealt with openly, it can smooth the way for change.

Successful implementations do not happen without stewardship from all levels. This project had a well-defined implementation team that encompassed the GCC, several levels of staff at the DMH-ADA, and outside consultants. However, provider representatives should have been included in initial negotiating regarding the implementation process. Moreover, it is important to have a designated staff member from each treatment site as a change agent who is formally recognized as responsible for GAIN implementation and is included in implementation-related planning and ongoing consultation.

The unwavering support of upper administration is a key to successful implementation of evidence-based practices in the health services field (Aarons 2006), such as statewide GAIN implementation. This project had strong support by DMH-ADA staff over time, including a dedicated state-level data manager and significant effort by the director of clinical services. The GCC has found implementation to be most successful when a local (state level) project coordinator oversees the implementation on a day-to-day basis and acts as a liaison between agencies, the state, and the GCC. Also, it is critical for the local project coordinator and one or more upper-level administrators to demonstrate support by completing the training and certification process in order to identify with and respond to any provider concerns.

Organizational factors, including organizational readiness to change, leadership behaviors, and organizational culture and climate, are increasingly recognized as important to implementation efforts (e.g., Saldana et al. 2007; Simpson, Joe & Rowan-Szal 2007; Aarons 2006; Hemmelgarn, Glisson & James 2006). Organizational readiness includes program and training needs, institutional resources, staff attributes and organizational climate as measured by the Organizational Readiness to Change scale (ORC; Lehman, Greener & Simpson 2002), as well as EBA- or EBP-specific attitudes. The Missouri GAIN implementation may have benefited from more attention to the agencies. For example, the results of an ORC assessment with each agency prior to implementation may have highlighted areas that hindered the implementation efforts (e.g., perceptions of staff-supervisor communication, job stress). The project also might have met less resistance if the requirements for using the GAIN had been written out and the agency directors had agreed to and communicated those requirements to their staff prior to the installation stage.

Fixsen and colleagues (2005) stress the importance of using purveyors to initially lead the implementation process and later assist as the lead is turned over to the agency or site. A corollary is to contract with the purveyor for ongoing technical assistance at the agency level as well as training of front-line clinicians. The GCC, with its implementation, training, and quality assurance process, is an excellent example of a purveyor. Their staff's experience in implementing the GAIN across multiple states, counties, and individual sites provides them with a wealth of knowledge regarding which approaches may work in which settings. Moreover, the collaboration of the GCC with Missouri's DMH-ADA resources and personnel ensured that all of the core implementation components were included in the process.

Attention and planning needs to be directed toward how best to diffuse the implementation through a provider network or state. Having only a few agencies begin implementation at first can help work out initial problems. Early adopters become champions or opinion leaders, and providers are often more receptive to hearing about the benefits of an instrument like the GAIN from other providers. This happened in Missouri, where the pilot process helped work out early issues and garnered support for the EBA. Also, implementing the GAIN in a focused manner with a small number of agencies and sustained commitment to the process (compared to some states implementing in over 100 agencies across multiple systems) helped its success. Other states (e.g., Illinois) have chosen partial implementation whereby only strongly committed agencies are given support to implement the GAIN. Resistance to assessment, QA and implementation deadlines were significantly reduced in these partial implementation states compared to other statewide projects. However, sustainability in these states was not as strong as it was in states where there was a universal agreement to use the tool. Other statewide implementation efforts have reported that successful implementations are more likely if they arise from the bottom-up, as grassroots innovations (e.g., Munetz et al. 2006). The GCC's experiences with the GAIN have echoed this finding, with the caveat that some voluntarily spread implementations are only partially realized or become unsustainable, perhaps due to the need for policy change and top-down support in complex system-wide efforts.

GCC Changes Based on this Project

Chestnut Health Systems' GCC has made several improvements in its regional support package based on the implementation experiences of Missouri and other states. First, they created a regional Core version of the GAIN, designed for shorter administration time (60 to 70 minutes) and to meet most state reporting needs, which has been a good starting point for states to review when designing their own Core version. Second, the Regional Trainer certification and the data management training and certification process were both developed in response to the states' needs to have local support for sustainability rather than being tied to the GCC. Finally, the GCC hired and trained specific project coordinators prior to embarking on regional implementation projects, knowing that states would need a single point of contact to manage complex issues along with the regular support lines.

CONCLUSION

It is tempting to say that the decision to adopt an EBA or EBP is the easiest part of the implementation process. However, it might be more accurately stated that the adoption decision is actually not a finite occurrence but a process itself that is made and remade at every step of implementation. As seen in our experience of moving the GAIN from one pilot site to statewide use, commitment to the decision was needed throughout all stages of implementation. Reaffirmation to using the GAIN was evident in staff, treatment agency, and state level decisions about providing the initial funds and personnel resources for program installation activities and initial implementation through full operation and into sustainability. Without continuing support, any implementation project can fall apart (Fals-Stewart, Logsdon & Birchler 2004). Focusing implementation efforts at the individual staff, agency, and system levels can result in decreased staff and administrator stress and result in a smoother change process.

REFERENCES

Aarons, G.A. 2006. Transformational and transactional leadership: Association with attitudes toward evidence-based practices. Psychiatric Services 57 (8): 1162-69.

Aarons, G.A. & Carmazzi, A. 2005. Implementation of evidence-based practices: New models and the role of provider attitudes. Paper presented at annual meeting of the National Association of State Mental Health Program Directors, Baltimore, MD, February.

Barlow, D.H. 2005. What's new about evidence-based assessment? Psychological Assessment 17 (3): 308-11.

de Rond, M.; de Wit, R. & van Dam, F. 2001. The implementation of a pain monitoring programme for nurses in daily clinical practice: Results of a follow-up study in five hospitals. Journal of Advanced Nursing 35 (4): 590-98.

Dennis, M.L.; White, M.K.; Titus, J.C. & Unsicker, J. 2006. Global Appraisal of Individual Needs: Administration Guide for the GAIN and Related Measures. Bloomington, Illinois: Chestnut Health Systems.

Fals-Stewart, W.; Logsdon, T. & Birchler, G.R. 2004. Diffusion of an empirically supported treatment for substance abuse: An organizational autopsy of technology transfer success and failure. Clinical Psychology: Science and Practice 11 (2): 177-82.

Fixsen, D.L.; Naoom, S.F.; Blase, K.A.; Friedman, R.M. & Wallace, F. 2005. Implementation Research: A Synthesis of the Literature (Rep. No. FMHI Publication #231). Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network.

Frambach, R.T. & Schillewaert, N. 2002. Organizational innovation adoption: A multi-level framework of determinants and opportunities for future research. Journal of Business Research 55 (2): 163-76.

Godley, M.D.; Godley, S.H.; Dennis, M.L.; Funk, R. & Passetti, L.L. 2002. Preliminary outcomes from the assertive continuing care experiment for adolescents discharged from residential treatment. Journal of Substance Abuse Treatment 23 (1): 21-32.

Gotham, H.J. 2006. Advancing the implementation of EBPs into clinical practice: How do we get there from here? Professional Psychology: Research and Practice 37 (6): 606-13.

Gotham, H.J. 2004. Diffusion of mental health and substance abuse treatments: Development, dissemination, and implementation. Clinical Psychology: Science and Practice 11 (2): 161-76.

Greenhalgh, T.; Robert, G.; Macfarlane, F.; Bate, P. & Kyriakidou, O. 2004. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Quarterly 82 (4): 581-629.

Hemmelgarn, A.L.; Glisson, C. & James, L.R. 2006. Organizational culture and climate: Implications for services and interventions research. Clinical Psychology: Science and Practice 13 (1): 73-89.

Kazdin, A.E. 2005. Evidence-based assessment for children and adolescents: Issues in measurement development and clinical application. Journal of Clinical Child and Adolescent Psychology 34 (3): 548-58.

Lehman, W.E.K.; Greener, J.M. & Simpson, D.D. 2002. Assessing organizational readiness for change. Journal of Substance Abuse Treatment 22 (4): 197-209.

Lennox, R.; Dennis, M.L.; Scott, C.K. & Funk, R. 2006. Combining psychometric and biometric measures of substance use. Drug and Alcohol Dependence 83 (2): 95-103.

Mash, E.J. & Hunsley, J. 2005. Evidence-based assessment of child and adolescent disorders: Issues and challenges. Journal of Clinical Child and Adolescent Psychology 34 (3): 362-79.

Munetz, M.R.; Morrison, A.; Krake, J.; Young, B. & Woody, M. 2006. Statewide implementation of the crisis intervention team program: The Ohio model. Psychiatric Services 57 (1): 1569-71.

Poulsen, I.; Hesselbo, B.; Pietersen, I. & Schroll, M. 2005. Implementation of functional assessment scales in geriatric practice: A feasibility study. Scandinavian Journal of Public Health 33 (4): 292-99.

Quick, M. & Fonteyn, M. 2005. Development and implementation of a clinical survey for cancer-related fatigue assessment. Clinical Journal of Oncology Nursing 9 (4): 435-39.

Saldana, L.; Chapman, J.E.; Henggeler, S.W. & Rowland, M.D. 2007. The Organizational Readiness for Change scale in adolescent programs: Criterion validity. Journal of Substance Abuse Treatment 33 (2): 159-69.

Shane, P.A.; Jasiukaitis, P. & Green, R.S. 2003. Treatment outcomes among adolescents with substance abuse problems: The relationship between comorbidities and post-treatment substance involvement. Evaluation and Program Planning 26 (4): 393-402.

Simpson, D.D. & Flynn, P.M. 2007. Moving innovations into treatment: A stage-based approach to program change. Journal of Substance Abuse Treatment 33 (2): 111-20.

Simpson, D.D.; Joe, G.W. & Rowan-Szal, G.A. 2007. Linking the elements of change: Program and client responses to innovation. Journal of Substance Abuse Treatment 33 (2): 201209

White, M.K. 2006. The effects of a Quality Assurance (QA) and Certification Program on the quality and validity of adolescent substance abuse treatment data. Paper presented at the 2006 Joint Meeting on Adolescent Treatment Effectiveness, Washington, DC, March.

([dagger]) Financial assistance for this study was provided in part by SAMHSA's Center for Substance Abuse Treatment (CSAT) Grant No. TI13305 to the Missouri Department of Mental Health, Division of Alcohol and Drug Abuse (Barbara Keehn, Principal Investigator). Heather J. Gotham was previously at the Missouri Institute of Mental Health and served as Evaluation Director during this project. The authors would like to thank Terry Morris for his generous help in preparing this article.

Heather J. Gotham, Ph.D., Senior Manager of Evaluation, Mid-America Addiction Technology Transfer Center, University of Missouri-Kansas City, Kansas City, MO.

Michelle K. White, Ph.D., Research Scientist, Assistant Director, GAIN Coordinating Center, Lighthouse Institute, Chestnut Health Systems, Bloomington, IL.

Hannah S. Bergethon, B.A., GAIN Project Coordinator, Lighthouse Institute, Chestnut Health Systems, Bloomington, IL.

Tim Feeney, B.A., Technical Writing Coordinator, Lighthouse Institute, Chestnut Health Systems, Bloomington, IL.

Dong W. Cho, Ph.D., Associate Professor, Missouri Institute of Mental Health, St. Louis, MO.

Barbara Keehn, R.N., B.S.N., Deputy Director, Missouri Department of Mental Health, Division of Alcohol and Drug Abuse, Jefferson City, MO.

Please address correspondence and reprint requests to: Heather J. Gotham, Mid-America Addiction Technology Transfer Center, University of Missouri-Kansas City, 5100 Rockhill Road, Kansas City, MO 64110-2499; email: gothamhj@umkc.edu; phone: (816) 482-1135, fax (816) 482-1101
TABLE 1

Six Stages of the Implementation Process

1. Exploration and Adoption

2. Program Installation

3. Initial Implementation

4. Full Operation

5. Innovation

6. Sustainability

Source: Fixsen et al. 2005: 15

TABLE 2

Recommendations for a Successful Implementation

* Implementation is an ongoing multistage process that will include
barriers and resistance to change.

* An implementation team should be comprised of stakeholders, staff
and change agents at multiple levels.

* Support of upper-level staff, including firsthand experience with
instruments and processes being implemented, is important for
widespread acceptance.

* Organizational factors including organizational readiness to
change affect implementation and can be directly addressed.

* Purveyors should initially lead the implementation process and
later assist as the lead is turned over to the agency or site.

* Purveyor[s] should provide ongoing technical assistance at the
agency level as well as training of front-line clinicians.

* Attention should be paid to how to best diffuse the EBA into a
provider network (e.g., pilot in one agency, small implementation
in several agencies).
COPYRIGHT 2008 Haight-Ashbury Publications
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2008 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Title Annotation:Global Alliance for Improved Nutrition
Author:Gotham, Heather J.; White, Michelle K.; Bergethon, Hannah S.; Feeney, Tim; Cho, Dong W.; Keehn, Barb
Publication:Journal of Psychoactive Drugs
Geographic Code:1USA
Date:Mar 1, 2008
Words:8597
Previous Article:Clinical outcomes of traumatized youth in adolescent substance abuse treatment: a longitudinal multisite study.
Next Article:The link between recent sexual abuse and drug use among African American male college students: it's not just a female problem in and around campus.
Topics:

Terms of use | Privacy policy | Copyright © 2018 Farlex, Inc. | Feedback | For webmasters