Taking school-based substance abuse prevention to scale: district-wide implementation of Keep A Clear Mind.

Abstract

Public schools are under increased pressure to implement evidence-based substance abuse prevention programs. A number of model programs have been identified, but little research has examined the effectiveness of these programs when "brought to scale" or implemented district-wide. The current paper summarizes the application of the Adelman and Taylor's (1997) model for district-wide program implementation to the dissemination of an evidence-based parent-child drug education program called Keep A Clear Mind (KA CM; Werch & Young, 1990). In addition to documenting the partnership process used to scale-up the program to a district-level, evaluation results are presented from 2,677 fifth graders in 43 schools who participated in the KACM program. Pre-post comparisons from two consecutive cohorts of students indicated a significant reduction in students 'attitudes supporting alcohol use and a significant increase in parent/child communication about prevention, students 'perceived ability to resist peer pressure, and their belief that it is "wrong" to use alcohol, tobacco, and marijuana. Focus groups conducted with a subset of the KACM teachers indicated great support for the KACM program, the partnership approach, and the dissemination model. Findings provide support for Adelman and Taylor's (1997) model as a framework for collaborative district-wide implementation of substance-abuse prevention programs.

The increasing legislative demands on school administrators and district directors to provide safe and orderly school environments have resulted in a greater emphasis on the importance of using evidence-based substance abuse prevention programs. Federal agencies, such as the Substance Abuse and Mental Heath Services Administration (SAMHSA), the U.S. Office of Safe and Drug Free Schools, and the Office of Juvenile Justice and Delinquency Prevention have generated lists of promising, effective, and model substance abuse prevention programs based on careful review of multiple efficacy trials. Although the programs on these evidence-based lists have been shown to have a positive impact when executed in tightly controlled settings, the effectiveness of these programs when implemented by communities and on a large scale is unknown.

The need for more research on the transition from efficacy of school-based substance abuse prevention to effectiveness research (Flay, 1986) is highlighted by the trend in public education toward district-level coordination and implementation of programs. In an effort to cull limited financial resources, school districts often opt to implement one or two evidence-based programs that are widely disseminated by the school district. Coordinated large scale implementation of preventive efforts can potentially improve student outcomes and be more cost-effective for the school system (Greenberg, 2004). Several models for widespread program dissemination and implementation have been proposed (e.g., Adelman & Taylor, 1997; Backer, Lieberman, & Kuehnel, 1986; Fixsen, Naoom, Blase, Friedman, & Wallace, 2005; Cheung & Cheng, 1997; Feldman, Baler, & Penner, 1997; Zins & Illback, 1995); yet, there are few published examples of the application of these models to the successful implementation of substance abuse prevention efforts. The present paper describes the district level implementation of Keep A Clear Mind (KACM; Werch & Young, 1990), an evidence-based substance abuse prevention program, as an illustration of Adelman and Taylor's (1997) multi-phase framework for disseminating school-based programs.

We first describe the Adelman and Taylor dissemination model and show how it was followed by the school district to "roll out" KACM (i.e., systematically phasing the implementation by beginning with a small number of schools and adding more schools over time). Since few published studies have focused on the effects of these types of programs when implemented in a large number of sites, an important aspect of the current study is the presentation of evaluation findings from two consecutive cohorts of fifth graders who participated in KACM. Through this case example, we also highlight the need for more substance abuse prevention effectiveness research and provide recommendations for large scale dissemination efforts.

Multi-Phase Program Diffusion Model

One of the most widely cited models for dissemination of educational reforms was put forth by Adelman and Taylor (1997). Applying research and theories from the fields of community psychology and organizational change, this model outlines an efficient four-stage strategy for implementing programmatic change across large educational settings. We briefly summarize the four stages of this model and then describe how it was used to implement the KACM substance abuse prevention program.

Phase I: Creating Readiness. This first phase of the model stresses the importance of obtaining community and stakeholder "buy-in" (i.e., community support) and preparing the environment for change. One strategy for motivating the community is to provide information about the issue and its local impact (Hogan et al., 2003). This can occur by summarizing research on the potential health risks (e.g., causes and consequences of early substance use), reviewing local data on the severity of the problem (e.g., rates of substance abuse in the community, prevalence of liquor stores in the neighborhood), and listing the current programs or services available to the target population (or lack there of). These and other types of information can be particularly persuasive when there is currently a lack of services, but can also be employed when trying to replace an existing program with limited effectiveness with a new and potentially more efficacious program. Input from multiple stakeholders (e.g., parents, teachers, administrators, community members) at this stage can also ensure that the program is contextually and culturally appropriate and may help the community anticipate and respond to obstacles (Hogan et al., 2003). Implementation of new programs also typically necessitates other pragmatic or "fundamental changes in a school's culture" (Adelman & Taylor, 1997, p. 202), such as reallocating time, staff, resources, and materials.

Phase II: Initial Implementation. The second phase of Adelman and Taylor's (1997) model focuses on providing staff with support and guidance as they begin implementing the program. It IS strongly recommended that a district-level support person, or "coach," work with the schools (at least initially) to provide technical assistance, aid in problem solving, and maintain momentum and enthusiasm for the program (Elliott & Mihalic, 2004). It is advantageous if the coach is a local expert or has prior experience with the program; however, a "stakeholder" (i.e., person from one school) can act as a coach or mentor for another school. It is beneficial if the coach is not an employee of the school, as the outside perspective is often useful in pinpointing problems and confronting challenging staff.

Phase III: Institutionalization. This third phase focuses on maintaining the changes made to the system through encouraging ownership of the new program and responding to potential roadblocks to sustaining the program (Adelman & Taylor, 1997). During this stage, leadership roles typically shift from external sources to members of the community (i.e., school). However, it is critical that one person or agency continue to serve a centralized leadership role to provide overall coordination, but begin developing the necessary scaffolding to encourage and guide local problem-solving. Other important components of this phase include preparing for sustainability by creating an infrastructure for training new staff (or substitutes) when there is staff turnover, continuing to provide on-site technical support to ensure fidelity and local accountability, and forging collaborations with other related initiatives.

Phase IV: Ongoing Evolution and Renewal. The final phase of this model stresses the importance of continued program development and integration of new knowledge, as accomplished through ongoing program evaluation and data-based decision-making. While it is important to chart and track progress toward intended goals throughout the entire implementation process, Adelman and Taylor (1997) caution that getting too focused on program impact during the early stages of implementation can put undue strain on both financial resources and stakeholder investment. This highlights the need for both formative (process focused) and summative (outcomes focused) evaluation efforts (Patton, 1997). Adelman and Taylor describe three outcomes of interest: the success of the replication, the immediate effect of the program on student outcomes, and the long-term effect of the program on student outcomes. Much can be learned by conducting a formative evaluation of the fidelity of implementation, examining the systems-level changes that have occurred as a result of the program, and documenting the people involved in and potentially affected by the program. Only after the formative evaluation has been conducted should the student outcomes be evaluated (Patton, 1997). Common outcomes examined in substance abuse prevention trials include changes in attitude and behavior. Consistent with a theory of change approach to evaluation (Fulbright-Anderson, Kubisch, & Connell, 1998; Izzo, Connell, Gambone & Bradshaw, 2004), it is important not to prematurely focus on long-term outcomes without first considering the implementation fidelity and the impact on immediate and intermediate effects. Furthermore, assessing student outcomes too early and not finding the expected effects may negatively influence program decisions (Patton, 1997) and diminish the stakeholders' commitment to the program.

While the model proposed by Adelman and Taylor (1997) holds promise as a method for large-scale dissemination, there is a paucity of research documenting the model's efficacy as applied to substance abuse prevention or other school-based primary prevention efforts. In the following sections, we present a case example of the implementation of KACM in two elementary school cohorts (Anne Arundel County Public Schools (AACPS), 2006) as an illustration of the district-wide school reform process described by Adelman and Taylor. We begin with a brief overview of KACM and the context in which the model was implemented.

CASE EXAMPLE: KEEP A CLEAR MIND

Program Description

As mentioned above, the KACM drug education program is a SAMHSA (2005) "model" program which aims to prevent substance abuse in elementary school children by educating them and their parents about the dangers of substance abuse, and helping students develop strategies for resisting substance abuse. The program is unique in that most of the substance abuse prevention activities occur at home through lessons distributed by the teacher. The take-home drug education program targets upper elementary school students, aged 8 to 12 years, and their parents. The program includes four lessons (Alcohol, Tobacco, Marijuana, and Tools to Resist) which are distributed by teachers and completed by the parent and child at home. The student receives a small incentive (keychain, stickers) for returning the parent-signed lesson booklet to the administering teacher. As a follow-up to the lessons, the school mails out a series of ten weekly newsletters to participating parents on issues related to substance abuse prevention. The program was developed to address known risk factors for later substance abuse, as well as to involve parents in the implementation of the program, and to be easy and inexpensive, and to require little class time to implement (Werch & Young, 1990). Previous evaluations of KACM have demonstrated that the program positively influences students' attitudes, behaviors, and communication with their parents (see Werch et al., 1991 for more information on program efficacy); however, less is known about the effectiveness of the intervention when implemented at the district level. In the following sections, we describe the introduction of KACM across a large Maryland public school district following Adelman and Taylor's (1997) model of dissemination.

Phase I: Creating Readiness

The Anne Arundel County Public School System (AACPS) is a large school district located between Baltimore City and Washington, D.C., which services approximately 75,000 students m 117 public schools (suburban, urban, and rural). Consistent with the first phase of broadening the scope of implementation, the school district selected KACM and prepared their community of teachers, children, and families for its implementation. Representatives from the district's Safe and Drug-Free Schools Division worked in collaboration with the county's Local Management Board to select an evidence-based substance abuse prevention program. The Local Management Board is a community-based organization, including representation from all of the district's youth-serving agencies (e.g., juvenile justice, social services, community mental health) that provides funding to agencies to support new programs that benefit children and families. After reviewing multiple substance abuse prevention programs recommended by SAMHSA, the community collaborative selected KACM because it requires little class time, has a positive focus, and encourages parent participation in the prevention effort.

The district needed little preparation for implementation, as the schools were previously implementing Drug Abuse Resistance Education (D.A.R.E.), and KACM was intended to be its replacement program. While the district and administrators were already aware of the importance of substance abuse prevention, it was critical that the lead KACM Facilitator get buy-in for the KACM program as the replacement program for D.A.R.E. The KACM Program Facilitator, who was based out of the school district's Office of Safe and Drug Free Schools, attended a district-wide principals' meeting to gain buy-in from administrators. In order to do so, the facilitator presented some research on the impact and prevalence of early substance use, summarized the published research suggesting the district's primary substance abuse prevention program (D.A.R.E.) had questionable efficacy (Rosenbaum & Hanson, 1998), and provided an overview of the key features of KACM. The administrators' response to the KACM program was overwhelmingly positive.

KACM was first implemented in the district in the spring of the 2002-03 school year. Fifth-graders in 19 schools participated during the first year, and students from 33 schools participated in the second year of implementation. The third cohort of the initiative included 43 elementary schools, and the fourth wave included 27 schools. Participation at the school, classroom, and student levels was voluntary. Per the district's standard procedures, the district implemented a passive consent process in which parents were informed in writing about the purpose of the program and of their right to withdraw their children from the KACM program and completion of the pre- and post-intervention evaluation surveys of students' attitudes toward substance use and prior substances use experience. Each classroom that implemented the KACM program during the first three years received a $300 honorarium from the Safe and Drug Free Schools Division to purchase incentives, rewards, or other class-related materials. Due to a reduction in the funding on the district level, the schools in the fourth cohort were provided with tangible incentives (e.g., t-shirts, key chains), but no monetary honorarium.

Phase II: Initial Implementation

During this phase of implementation, support and guidance were provided to school staff by the district-wide KACM Facilitator. An attractive feature of the KACM program was that teachers implementing the program required relatively little training and needed only limited guidance and support beyond the initial training. The four-hour teacher training session was held in the fall of each year for all fifth grade teachers from participating elementary schools. At this training, the KACM materials were distributed and the process for program implementation was reviewed. Although the program is relatively straight-forward, the KACM Facilitator provided additional technical assistance and suggestions for better integration of the program into existing programs and curricula, thus fulfilling the role of program "coach." The facilitator also provided on-site technical assistance as needed. Finally, a plan was developed for the evaluation of KACM that focused on both process and short-term outcomes.

Phase III: Institutionalization

Adelman and Taylor's (1997) third phase entails shifting centralized leadership to community members and ensuring that measures to sustain the program beyond initial implementation are put into place. The community-based Local Management Board funded the district-wide KACM implementation for all four years of implementation, either directly or through a grant to the District's Office of Safe and Drug Free Schools. Management of the program (e.g., recruiting schools, disseminating program materials, training teachers, and collecting data for ongoing evaluation) was shifted to the school district after the second wave of implementation. In an effort to ensure that the program is financially sustainable, the $300 honorarium offered to participating schools was terminated, and the student incentives were purchased by the District and provided directly to the schools. Teachers responded positively to the elimination of monetary honorarium and most chose to continue implementing KACM without the financial incentive.

One very important element related to program sustainability is maintaining a level of enthusiasm about the program among its stakeholders, including district and local school-level staff and administrators, as well as among students and their families. Public events and media coverage which draw positive attention to the initiative have helped it remain visible among the District's numerous programs. These events can also potentially attract new sponsors for the program. The KACM Program Facilitator coordinated such a community event at an elementary school implementing KACM to publicize the program. The event was jointly sponsored by the school district, the Governor's Office of Crime Control and Prevention, and the Local Management Board. The Maryland State Governor's wife attended the event and discussed the importance of preventing early alcohol use. Students participating in KACM and several of their parents attended the event and gave testimonials about the impact of the program. Many parents shared stories about positive interactions they had with their children regarding the KACM take-home materials. Media events like this build excitement within a community and demonstrate to key stakeholders and potential funders the value of the program and the local impact.

Phase IV: Ongoing Evolution and Renewal

This final phase of the Adelman and Taylor (1997) implementation model involves engaging in an evaluation of the program, with a focus on continued implementation and optimization of fidelity and effectiveness. A core aspect of the district's effort was the evaluation of KACM, both in terms of process and outcomes (AACPS, 2006). A smaller evaluation focused on the second cohort of KACM schools, followed by a more extensive evaluation of the effect of the program among the third and fourth cohorts of KACM schools. Data were collected on the number of signed booklets as a measure of "dosage" of program exposure, as well as on the intended outcomes (e.g., frequency of parent/ child communication about substance use, students' attitudes and drug-related behaviors) using a pre/post-test design.

Process findings. A variety of methods was employed to document and evaluate the process aspects of the KACM initiative. The district developed a log which was completed by the teacher to document which lessons had been distributed to each student and which lessons the students had returned with a parent signature. A single school-level dosage score was calculated using these data by tabulating the percentage of students receiving and retuning signed lesson forms. Since high fidelity, or high dosage, is often defined as meeting or exceeding an 80% implementation level within the school (Homer et al., 2004), the scores were dichotomized into those reaching a high level (i.e., 80% or higher) of implementation and those that did not. Sixty-seven percent of the 43 cohort three schools implemented KACM with high fidelity, whereas 25% implemented KACM at low fidelity, and the remaining 8% of schools did not provide dosage logs. Thirty-seven percent of the 27 cohort four schools implemented KACM with high fidelity, whereas 53% implemented KACM at low fidelity, and the remaining 11% of schools did not provide dosage logs (AACPS, 2006). These findings suggest variability in either the recording of program dosage or the actual dosage levels.

To learn more about teachers' experience implementing the program, the third cohort KACM teachers were invited to participate in a focus group to discuss their experience implementing KACM and their perceptions of the impact of the program on intermediate outcomes, like parent/child communication and attitudes toward substance use. Fifteen cohort three teachers participated in the focus group (AACPS, 2006), during which they reported great satisfaction with the ease by which KACM was implemented, and reported a willingness to continue to use program. They appreciated that parents were involved in the program and reported positive interactions with both students and parents related to the KACM program. Several teachers reported successfully linking KACM to other school-based health programs, such as tobacco lessons in science class and other drug awareness events like Tobacco Week (AACPS, 2006).

These 15 teachers also completed a brief survey which was developed by the KACM Facilitator to assess teacher perceptions of the program (AACPS, 2006). The vast majority of teachers agreed that KACM was "very easy" to implement (86.7%), had a "moderate" to "significant" impact on parent/child communication (100.0%). Additionally, teachers reported that KACM had a significant impact on student attitudes toward drug use (46.7%) and had a "significant impact" on student behaviors (46.7%) (AACPS, 2006). All of the teachers indicated that they would choose to use KACM the next year and would recommend it to other teachers. It is important to note that the teacher data were only available for approximately 20% of the KACM teachers, so further work is needed with a larger sample of teachers to understand their perceptions and experiences with the program. Future evaluation efforts will also assess the attitudes of administrators, students, and parents regarding the program.

Effect on intermediate outcomes. In both the 2004-05 (cohort three) and 2005-06 (cohort four) school years, students completed an anonymous revised version of the pre/post intervention survey which was provided with the KACM program materials as an indicator of attitude and behavior change. Multivariate GLM analyses were conducted in SPSS on the pre/post-test data for these two cohorts. The results evinced several significant changes between the pre- and post-intervention regarding students' report of substance use related attitudes and behaviors in both the 2004-05 and 2005-06 school years (AACPS, 2006). Specifically, students reported speaking with their parents about drug use more frequently after participating in the program than before, F (1, 4966) = 4.983, p<.001. See Figure 1 for mean pre- and post-test subscale scores across both the 2005 and 2006 cohorts. In addition, students reported a significantly greater ability to resist the temptations of drugs after participating in the program than before, F (1, 4966) = 6.120, p<.05 (Figure 2). Students also reported believing substance use was significantly more harmful to their health after participating in the program, F (1, 4966) = 6.204, p<.05 (Figure 3), and similar effects were observed for students' perception that it is "wrong to use drugs", F (1, 4966) = 12.193, p<.001 (Figure 4). Finally, there were significant changes in the students' general attitudes toward alcohol use, F (1, 4966) = 10.225,p<.001, favoring the program (AACPS, 2006) (Figure 5).

Although there are several limitations of these data (e.g., the lack of a comparison group, the focus on self-reported short-term outcomes, and limited information on program fidelity and dosage), the findings suggest a positive short-term effect of the intervention. Further work is needed to determine if these effects persist over time and whether they translate into behavior changes.

DISCUSSION

Although coordinated district-wide substance abuse prevention programming efforts are becoming more common, there are relatively few published examples of successful district-wide dissemination of school-based substance abuse prevention programs. Several models exist that delineate strategies for implementing prevention programs (e.g., Adelman & Taylor, 1997; Fixsen et al., 2005; Zins & Illback, 1995), all of which share a similar concept for the stages of implementation. Adelman and Taylor's (1997) multi-phase program diffusion model holds great promise as a strategy for successful large scale implementation of educationally-based substance abuse prevention programs; however, most of the prior work on this and other similar models has been theoretical and formulated based on observations across numerous diverse programs. The current paper applied the four-stage Adelman and Taylor model of large-scale implementation to a community-initiated and managed implementation of the Keep A Clear Mind Program evidence-based substance abuse prevention program over four years.

Moving from efficacy to effectiveness research. This study also touches on the burgeoning field of effectiveness research, which aims to determine the outcomes of evidence-based interventions in "real-world" settings by community members, rather than researchers (Flay, 1986; Shadish, 2002). In these designs, researchers do not have direct control over the management of the program, nor do they provide the training or supervision of the individuals providing the intervention. Instead, community-based agencies assume the lead for both program coordination and management. Because effectiveness studies are more ecologically or contextually similar to the environments in which the interventions will likely be implemented, they often suffer some of the same methodological flaws as program evaluations, such as questionable implementation fidelity or unmeasured covariates. The outcomes observed in effectiveness trials represent a combination of the "true" latent impact of the program (i.e., program efficacy), program fidelity, and other process factors, such as program buy-in and engagement. Consequently, they often yield intervention effect sizes which are smaller than the efficacy studies. Regardless, effectiveness research is critical to determining the impact of programs which can be expected when the program is implemented in real-world settings by educators or communities, rather than researchers.

Limitations. While the current study represents a first step toward effectiveness research, there are several limitations to note when interpreting these findings. Although implemented across an entire district, the quantitative summative evaluation data were limited to the subset of schools which submitted sufficient data for analysis. As such, one must consider the small, and potentially selective, sample of schools included in these analyses. Other design limitations include the lack of a comparison group (i.e., control group not receiving the intervention) and limited information regarding long-term student outcomes. Additional information on program fidelity would clarify ambiguity regarding the dosage data, as it is unclear whether the classes with low or no dosage data failed to implement the program or had poor documentation procedures.

With regard to the application of the Adelman and Taylor model to the implementation of KACM, some aspects of the model were less challenging to follow for this particular program than they may be regarding other more intensive programs. For example, the community was already implementing a substance abuse prevention program and KACM was intended to be a replacement program; thus, relatively little preparation needed to occur. Similarly, although a district-level KACM coach facilitator was identified, the coaching and training responsibilities were lighter for this program than they may be for other programs. Furthermore, this paper focused on the application of this model to a single program in a single school district. Additional research is needed to determine the applicability of the Adelman and Taylor model for large-scale dissemination of multiple school-based programs in different contexts.

Conclusions and Recommendations

Following the Adelman and Taylor model, the district successfully implemented KACM in multiple schools, achieved positive student outcomes, and has sustained the program for multiple school years even in the face of budget cuts and management changes. First, the school district successfully secured buy-in from the administrators, teachers, and children and their families by sharing information with them about the program and the positive results previously reported in research. Second, the school district provided staff with training prior to the initial implementation of the program and ensured that staff had access to a coach and ongoing technical assistance. Third, the school system assumed responsibility for leadership of KACM from the County government, thereby providing the district with greater control and independence and allows the original leadership to focus on other community needs. Finally, the school district assessed program dosage and student outcomes annually as part of their evaluation process. This enabled the district to determine what changes in training and implementation need to be made to maximize program impacts. They also presented the evaluation findings annually to the district-level administrators, the principals and teachers at the participating KACM schools, and to the community partners at the Local Management Board. This helped the community partners work collaboratively to streamline program implementation to meet the school district's specific resources and needs. Acknowledging the methodological limitations noted above, the evaluation findings provide preliminary empirical support for the application of Adelman and Taylor's (1997) model of district-wide school reform to the implementation of KACM.

We strongly recommend that school districts implementing substance abuse prevention programs on a large scale use the Adelman and Taylor framework or other similar stage models to guide their work. By adopting a district-level model of program coordination, programming decisions are shifted from the individual schools to the district. This in turn may result in greater program fidelity, enhanced program outcomes, and increased sustainability; however, centralized coordination does present some administrative challenges. Maximizing buy-in from multiple stakeholders, community preparation, program fidelity, and sustainability will likely increase the success of the program when implemented across multiple schools.

Prior research on other school-based prevention models indicates a strong association among the quality of technical assistance, program fidelity, and student outcomes (Elliott & Mihalic, 2004); consequently, a coach that can both conduct the training and provide the ongoing technical support should be identified when implementing KACM and other more intensive curriculum-based substance abuse programs, such as the Life Skills Training Program (Botvin, Griffin, Paul, & Macaulay, 2003). The program should also include an evaluation to facilitate data-based decision-making. It is critical to collect information regarding both process and outcomes in order to determine what types of programmatic changes need to be made to maximize outcomes. Taken together, the findings of the current study provide preliminary support for the application of the Adelman and Taylor (1997) model to the district-wide dissemination of substance use prevention programs.

Correspondence concerning this article should be address to: Keri Jowers, M.Ed., Predoctoral Fellow, Department of Mental Health, Johns Hopkins Bloomberg School of Public Health, 624 N. Broadway, Baltimore, MD 21215, Telephone [phone], Fax [phone] and Email: [e-mail].

REFERENCES

Adelman, H. S., & Taylor, L. (1997). Toward a scale-up model for replicating new approaches to schooling. Journal of Educational and Psychological Consultation, 8, 197-230.

Anne Arundel County Public Schools (AACPS) (2006). Keep A Clear Mind Substance Abuse Prevention Program: Evaluation Summary for 2005 anal 2006. Technical report prepared for the Division of Safe & Drug-Free Schools, AACPS, Annapolis, MD.

Backer, T. E., Liberman, R. P., & Kuehnel, T. G. (1986). Dissemination and adoption of innovative psychosocial interventions. Journal of Consulting and Clinical Psychology, 54, 111-118

Botvin, G. J., Griffin, K. W., Paul, E., & Macaulay, A. P. (2003). Preventing tobacco and alcohol use among elementary school students through Life Skills Training. Journal of Child & Adolescent Substance Abuse, 12, 1-18.

Cheung, W. M., & Cheng, Y. C. (1997). The strategies for implementing multilevel self- management in schools. International Journal of Educational Management, 11, 159- 169.

Elliott, D., & Mihalic, S. (2004). Issues in disseminating and replicating effective prevention programs. Prevention Science, 5, 47-53.

Feldman, S., Baler, S., & Penner, S. (1997). The role of private-for-profit managed behavioral health in the public sector. Administration and Policy in Mental Health, 24, 379-390.

Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M. & Wallace, F. (2005). Implementation Research: A Synthesis of the Literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231).

Flay, B. R. (1986). Efficacy and effectiveness trials (and other phases of research) in the development of health promotion programs. Preventive Medicine, 15, 451-474.

Fulbright-Anderson, K., Kubisch, A. C., & Connell, J. P. (1998). New Approaches to evaluating community initiatives: Vol. 2. Theory, measurement, and analysis. Washington, DC: Aspen Institute.

Greenberg, M. T. (2004). Current and future challenges in school-based prevention: the researcher perspective. Prevention Science, 5, 5-13.

Hogan, J. A., Baca, I., Daley, C. Garcia, T., Jaker, J., Lowther, M., & Klitzner, M. (2003). Disseminating science-based prevention: Lessons learned from CSAP's and CAPTs. Journal of Drug Education, 33, 233-243.

Horner, R., Todd, A., Lewis-Palmer, T., Irvin, L., Sugai, G., & Boland, J. (2004). The school-wide evaluation tool (SET): A research instrument for assessing school-wide positive behavior support. Journal of Positive Behavior Interventions, 6, 3-12.

Izzo, C. V., Connell, J. P., Gambone, M. A., & Bradshaw, C. P. (2004). Understanding and improving youth development initiatives through evaluation. In S.F. Hamilton & M.A. Hamilton (Eds.), Youth development handbook: Coming of age in American communities (pp. 301-326). Thousand Oaks, CA: Sage.

Patton, M. Q. (1997). Utilization-focused evaluation: The new century text (3rd ed.). Thousand Oaks, CA: Sage.

Rosenbaum, D. P., & Hanson, G. S. (1998). Assessing the effects of school-based drug education: A six-year multilevel analysis of project D.A.R.E. Journal of Research in Crime and Delinquency, 35, 381-403.

Shadish, W. R. (2002). Revisiting field experimentation: Field notes for the future. Psychological Methods, 7, 3-18.

Substance Abuse and Mental Health Services Administration [On-line]. Model Programs. Retrieved on June 15, 2005: http:// modelprograms.samhsa.gov/template cf.cfm?page=model&pk ProgramID=8.

Werch, C., & Young, M. (1990). Keep A Clear Mind: A Drug Prevention Program for Children and Their Parents. The Health Education Projects Office, University of Arkansas: Fayetteville, AR.

Werch C. E., Young, M., Clark, M., Garrett, C., Hooks, S., & Kersten, C. (1991). Effects of a take-home drug prevention program on drug-related communication and beliefs of parents and children. Journal of School Health, 61, 346-350.

Zins, J. E., & Illback, R. J. (1995). Consulting to facilitate planned organizational change in schools. Journal of Educational and Psychological Consultation, 6, 237-245.

[FIGURE 1 OMITTED]

[FIGURE 2 OMITTED]

[FIGURE 3 OMITTED]

[FIGURE 4 OMITTED]

[FIGURE 5 OMITTED]

Keri L. Jowers, Catherine P. Bradshaw

Johns Hopkins Bloomberg School of Public Health, Johns Hopkins Center for the Prevention of Youth Violence

and

Sherry Gately

Office of Safe and Drug Free Schools, Anne Arundel County Public Schools