Enhancing prevention programs' credibility through the use of a logic model.

Dear Editor:

As park, recreation and sport management professionals become increasingly involved in youth development, they find themselves operating in a world requiring diverse and complementary program justification. There is an interest to move beyond the provision of facilities and equipment for traditional sports activities to the development of programs that target specific outcomes that benefit youth (Ellis, Braff, & Hutchinson, 2001). Program justification has become essential for credibility. Credibility depends on a sound programming process that clearly establishes a link between theory, the program model and subsequent behavioral changes in participating youth (McKenzie & Smeltzer, 1997). This pendulum shift has presented several challenges to park, recreation and sport management professionals.

While park, recreation and sport professionals are quick to tout the positive impact of their programs for at-risk youth in terms of increased self-esteem, improved school achievement, and better communication skills; however, the actual relationship between recreation and beneficial outcomes has not been adequately explained (Baker & Witt, 2000). Baldwin (2000) stated there has been little discussion of the theoretical foundation of park and recreation programming for at-risk youth, despite the need for scientific rigor to establish the leisure services and sport field's credibility of these programs.

Secondly, most professionals have not been trained in scientific evaluation or have not used their skills in years, consequently have not embraced evaluation as a programming process. These issues are further complicated because the evaluation process in park, recreation and sport often lacks specificity in the selection of possible program outcomes despite recent improvements in program planning. Many programs are dominated by short-term, unstructured activities, such as recreational basketball, that do not focus on behavioral changes. Kleiber (1999) stated that these traditional activities do not provide the "challenge" necessary to promote development in participants.

Recent research reveals that park and recreation practitioners are expressing a desire to understand theory and incorporate it into their programs (McKenzie & Smeltzer, 1999). Practitioners realize that such inclusion provides credence and a rationale for the intervention or program, as well as a foundation for the desired outcome. However, most park and recreation prevention programs have used broad/ vague program objectives, making it difficult to assess specific program outcomes (Baldwin, 2000). Baldwin asserted that identifying the behavioral framework of the recreation program context is particularly challenging for park and recreation professionals.

To advance professional use of evaluation models, the author developed a logic model to systematically illustrate and evaluate the relationship between theories, procedures, programmatic content, evaluation methods, and outcome of a multidisciplinary, community-based prevention project that was designed for "high at-risk" youth. The author hopes that this experience will bridge theory and practice, and assist in advancing funding and collaborative opportunities. While it is not the intent of this communication to report findings of the study, the author does provide general information regarding the results to explain the interrelated components of the model. The project examined the outcomes of recreation and health related activities to improve youth academic standing and reduce youth negative behaviors such as drugs and criminal activities. A multidisciplinary team of educators, prevention professionals, community organizations, college students, and parents undertook the project referred to as Project WISE-UP, which was funded by a two-year grant from the U.S. Department of Juvenile Justice and the Florida Department of Juvenile Justice. The goals of the project were: (a) to reduce the risk that youth will engage in dysfunctional behavior such as drugs and criminal activities; and (b) to improve participants' academic standing while reducing their in-school suspension rates. Specific objectives included increase participants' GPA and decrease aggregate absences, suspensions and referrals to the principal's office, and improve their leisure attitudes and behavior.

The WISE-UP Logic Model (See Figure 1) illustrated a logic-flow from program activities to the anticipated short-and long-term impact of the intervention.

Figure 1 shows the improved framework for the WISE-UP Logic Model. It consisted of five components: (1) Assumptions/Theoretical Premise, (2) Program Activities, (3) Immediate Outcomes/Evaluative Activities, (4) Intermediate Outcomes/Objectives, and (5) Final outcomes/Goals.

[FIGURE 1 OMITTED]

The assumption/theoretical premise, which included social development theory, provided a foundation to define and design the content and structure of intervention activities. Social development theory provided staff the means to understand the interrelatedness or covariates of risk behavior. Jessor (1991) discussed the importance of understanding adolescent risk behavior as a cluster instead of attempting to focus on an independent or separate behavior to develop effective prevention programs. According to Jessor, at-risk youths' lifestyle often involves a constellation of risk behaviors (driving after drinking, unprotected sexual intercourse, fighting, smoking, illicit drug use, etc). Therefore, the development of a single-focused program is not as effective as a comprehensive program designed to focus on the entire array of adolescent risk behaviors. A program that focuses on "Just say no to drugs" needs to concentrate on drug education, alcohol abuse, decision-making, self-esteem, and sexual precocity, as a minimal to be effective. Jessor's theoretical notion-Conceptual Framework for Adolescent Risk Behavior: Risk and Protective Factors, Risk Behavior, and Risk Outcomes-provided the means for staff to understand risk behavior in the context of a "web of causation" (Jessor, 1991). The conceptual framework also provided an understanding of the interrelated nature of risk factors and protective factors, risk behavior, and consequences of the risk behaviors. Drawing from this theoretical notion, the staff designed Project WISE-UP as a comprehensive, multidisciplinary program that included drug education, crime prevention, decision-making, tutoring, leisure education, recreation, family discussions, and life skills.

The assumption/theoretical premises component also relies on methods extrapolated from the Family Ecological Systems approach, which is derived from theory and research in human development, cognitive development, child and adolescent psychology, family therapy, and community mental health (Henggeler, 1982). Henggeler's perspective is that the adolescent is embedded in and interacts with various systems and subsystems, ranging from personal attributes, parents, siblings and extended family, to the school, church, and community organizations. The Family Ecological System designates five spheres of influence on an individual, including the following: personal attributes of the individual (intrapersonal system); interpersonal relations within the immediate settings in which an adolescent develops and interacts (microsytem); linkages between two or more elements in the microsystem; social structures that influence power over the adolescent indirectly (exosystem); and patterns of cultures, values and belief (macrosystem). It is within these systems that the individual's behavior and development are shaped. This theoretical notion aided the staff in understanding the interrelations among the adolescent, family, and extrafamilial systems. Given this theoretical premise, the author established a partnership with the adolescents (intrapersonal system); parents, extended family, siblings, peers, teachers and the recreation staff (microsystem & mesosystem); and community organizations and the school system (exosystem).

Figure 1 presents additional theoretical notions regarding decision-making and problem-solving knowledge and skills, constructive use of leisure activities, healthy lifestyles, and good study habits that guided the development of the program and evaluation activities. For example, one theoretical notion that influenced program development was the belief that constructive and wise use of a diverse range of enjoyable and rewarding leisure activities will deter drug use and criminal behavior. Additionally, research suggests that having a great deal of unstructured, unsupervised discretionary time can undermine positive youth development (Curruthers and Busser, 2000). From these assumptions, the author designed the programmatic activities for Project WISE-UP that included crime prevention, drug education, leisure education, life skills (decision-making, problem solving and conflict resolution), teen courts, a tutorial program, field trips, chess, and parental involvement. It was offered in two-hour sessions, four times a week for 32 weeks over two-years, using the university semester as a framework. All project activities were held in the participants' community, a federally subsidized housing project, where they resided and attended public school.

All middle school students, ages 11-14 and a younger sibling, were invited to participate in the program. Thirty-seven (37) students agreed to participate. Fifty-nine percent of participants were males and 41% were females. All participants were African Americans with 100% of them receiving free school lunches. Most of the participants were "high- at risk" students from households headed by a single parent whose household income fell below the state's poverty level. They were exposed to multiple risk factors: economic disadvantage, alcohol and drug experimentation, chronic failure in school, learning disabilities, et cetera. All participants had poor academic achievement records, poor school attendance, and very high in/out of school suspensions based on misconduct. Most participants had been administratively promoted for several grades.

Project staff tested these hypotheses and refined assumptions through various programmatic and evaluative activities. For instance, evaluative activities for the leisure education module included "pre" and "post" surveys to assess subjects' perceptions of satisfaction and value of leisure, combined with a self-inventory of leisure behavior. The immediate outcomes resulted directly from an activity and led to intermediate outcomes; for example, enriched knowledge of the harmful effects of drugs; advanced perceptions and attitudes toward leisure activities, decision-making and problem solving skills; increased family bonding; and improved study skills. Intermediate objectives were evaluated by comparing the participants' GPAs, school attendance, in/out of school suspensions, and disciplinary referrals at various intervals. Data were obtained from the school board, one of the project's partners. If the project was effective, then improved GPAs, reduced days of absence, reduced suspensions, and reduced referrals should have followed the improved knowledge and attitudes of drugs, improved leisure attitudes and perceptions, improved life skills, and improved family bonding and communication.

Table 1 presents a summary of the intermediate outcomes for Year 1 and Year 2. By the end of Year 1, data revealed participants' GPA decreased from 2.21 to 1.95; days of absence increased from 14.4 to 19.06 days; and school suspensions increased from 2.19 to 7.91 times.

The only area that suggested program effectiveness in Year 1 was disciplinary referrals to the principal or dean of student's office. It decreased from an average of 9.66 to 6.53 times. Year 2 data revealed improvements in all areas, consequently demonstrating the logical progression of the project's impact. By the end of Year 2, 39% (n = 12) of participants improved in their GPA. The average improvement for the project participants was .15. The number of referrals to the principal or dean of student's office decreased by an average of 2.3 encounters, which represented 55% of the participants showing a decrease in referrals. The average number of school suspensions decreased by 5.5 times, and 71% of the participants showed a decrease in the number of school suspensions. The average days of absences decreased by 1.7 days, with 38% of participants showing a decrease in absences.

At the end of Year 1, data on participants' drug and alcohol knowledge showed an 11% increase overall. Forty-six percent of participants (n = 14) improved their knowledge (See Table 2).

Thirty-six percent of participants (n = 11) improved their attitudes toward drugs and alcohol abuse. Fifty-five percent of participants (n = 17) improved their leisure attitudes. These were measured using a Likert-type scale using a range of 1 to 5. Data on participants' decision-making and conflict resolution knowledge revealed that 55% of participants (n = 17) reported an improvement. Data revealed 64% percent of participants (n = 20) improved their decision-making behaviors and forty-six (46%) (n = 14) of participants improved family bonding and communication scores.

Students were presented a series of questions, which were indicators of parental involvement in their completion of homework, problems, and social decisions. Data revealed 36% (n = 11) of the participants improved their study habits during Year 1. One obvious limitation of the study was an inability to control for other intervening variables that may have contributed to participants' success.

Park, recreation and sport professionals of youth-serving agencies are facing increasing challenges to align programs with policy directions that endorse a youth development approach. Such an approach requires program documentation of outcomes and the utilization of a sound programming process that clearly establishes a link between theory, program model and subsequent behavioral changes in participating youths. Documentation of the value of park, recreation and sport activities is the most difficult type of evaluation because it requires a systematic methodology for collecting information, a careful topology of activities, clearly defined goals and performance outcomes, and specific measures of outcomes and perceived benefits. Perhaps this is why only a few studies have been conducted that specifically evaluate program effectiveness (Baker & Witt, 1996). These challenges will require more dialogue of the theoretical foundation of park, recreation and sport programming for at-risk youth, funding, and staff training. Establishing programs on a sound theoretical base enhances the rigor and quality of programs and subsequent outcome evaluation (Baldwin, 2000). Through this research endeavor, the author hopes to advance professional use of evaluation models and to facilitate their use in park, recreation, sport, and leisure services. It has been the author's experience that most stakeholders perceive recreation programs as nonessential and intangible service. Most stakeholders understand the relationship between recreation and economic impact and/or personal fitness; but too few perceive recreation as a vital element in behavioral modification or youth development. The author found this to be the case in seeking funding for the discussed project. Prior to the state agency's approval of funding, the author was summoned to a meeting of prevention, law enforcement, and youth development personnel to discuss strategies to achieve such long-term goals. When the author noted the second goal of the project--to reduce youth dysfunctional behaviors, such as drugs and criminal activities, the question was raised: "How are you going to achieve it." The logic--model was a "saving grace" in explaining how the intervention had been conceptualized to enhance youth development. Having studied the essence of social learning theory, Jessor's Conceptual Framework for Adolescent Risk Behavior, and the Family Ecological Model (Jessor, 1991), the author was able to provide a "map" of critical program variables and their relationship to theory and practice.

In addition to the model providing a paradigm to systematically explain the relationship between theoretical premise, program intervention, immediate and long-term outcomes, it provided a logical explanation of the logic flow from program activities to the anticipated short and long-term impact of the intervention.

The results at the end of Year 1 were not as positive as those at the end of Year 2. After one year of intervention, the only area that suggested program effectiveness was disciplinary referrals. As noted, it decreased from an average of 9.66 to 6.53 times. For Year 1, data revealed that the project was effective only in slowing the rate of academic deterioration after the first year. This result was somewhat expected, recalling that these were high-risk youth, and many of them had been administratively promoted. It is the contention of the author that without the prevention program, the participants' dysfunctional behavior would have increased significantly in the absence of the project intervention. Second year data revealed improvements in all areas, consequently suggesting the logical progression of the project's impact. These findings support the literature, which holds youth development takes time and is most effectively addressed by programs that focus on an array of risk factors.

As illustrated in the logic model, immediate objective measures need to occur before indicators of intermediate outcome/objectives or final outcomes can be expected. Results of the data analysis support this logical progression. Participants' academic performance and disciplinary behaviors improved after they showed improvements in drug and alcohol knowledge and attitudes, leisure attitudes and perception, study habits, family bonding and communication, and in decision-making knowledge and skills.

In conclusions, there are several implications for this study. The modest improvements in the participants' academic performance and improved knowledge and skills in decision-making, drug education, and improved leisure attitudes and recreation participation suggest that park and recreation programs can make a positive contribution to youth development. The study results also give credence to park, recreation and sport prevention programs that target youth who are failing in the school system. The logic model suggests that agency effectiveness may be enhanced if there is a direct link between assumptions/theoretical notions, goals, programs and the valued outcomes, which give credence to Baldwin (2000), Cato, Chen and Corbett-Perez (1998), and McKenzie's (1997) assertions. The model also provides a strategy for managers of youth serving agencies who are encountering high demands for services, scrutiny from funders, and the need to go an extra step in justifying the benefits of recreation in youth development.

Despite this effort, as well as other advancements in the evaluation and documentation of park, recreation and sport prevention programs, there are still issues to be resolved. On-going research is required to further test the model and document the value of such programs in the behavior modification and youth development. If park, recreation and sport professionals want to move beyond traditional recreation programming for youth, to the development of programs that target specific outcomes for at-risk youth, they must provide diverse and complementary program strategies that are objective and measurable. Intervention programs that can document positive behavioral changes in participants enhance collaborative and funding opportunities.

References

Baker, D. A., & Witt, R A. (2000). Multiple stakeholders' views of the goals and content of two after-school enrichment programs. Journal of Park and Recreation Administration, 18, 68-86.

Baldwin, C. K. (2000). Theory, programs, and outcomes: Assessing the challenges of evaluating at-risk youth recreation programs. Journal of Park and Recreation Administration, 18, 19-33.

Cato, B., Chen, W., & Corbett-Perez, S. (1998). Logic model: A tool for planning and evaluating health and recreation prevention projects. Journal of Physical Education, Recreation and Dance, 69, 57-61.

Curruthers, C., & Busser, J. A. (2000). A qualitative outcome study of boys and girls club program leaders, club members, and parents. Journal of Park and Recreation Administration, 18, 5-67.

Ellis, J. M., Braff, E., & Hutchinson, S. L. (2001). Youth recreation and resiliency: Putting theory into practice in Fairfax Count. Therapeutic Recreation Journal, 35, 307-319.

Henggeler, S. W. (1982). The family-ecological systems theory. In S. W. Henggeler (Ed.), Delinquency and adolescent psychopathology: A family ecological systems approach. Boston: John Wright. PSG Inc.

Jessor, R. (1991). Risk behavior in adolescence: A psychological framework for understanding and action. Journal of Adolescent Health, 12, 597-605.

Kleiber, D. (1999). Leisure experience and human development: A dialectical interpretation. New York: Basic Books.

McKenzie, J. E, & Smeltzer, J. L. (1997). Planning, implementing, and evaluating health promotion programs: A primer (2nd ed.) Needham Heights, MA: Allyn and Bacon.

Bertha Cato, Department of Tourism, Recreation and Sports Management, College of Health & Human Performance, University of Florida 

Table 1
Project WISE-UP Intermediate Data
Year 1 and Year 2

              Beginning   Year 2       CHANGES
               Year 1                  End of
               N = 30     N = 32       Year 1

GPA (0-4)       2.21       1.95         -0.26

ABSENCES
(Days)          14.41      19.06        +4.65

REFERRALS
(Frequency)      9.66       6.53        -3.13

SCHOOL
SUSPENSIONS
(Frequency)      2.19       7.91        +5.72

               Year 2     CHANGES   % PARTICIPANT
                          End of      IMPROVED
               N = 31     Year 2     Year 1 & 2

GPA (0-4)       2.10       +0.15        38.71

ABSENCES
(Days)         17.36       -1.70        37.93

REFERRALS
(Frequency)     4.22       -2.31        55.17

SCHOOL
SUSPENSIONS
(Frequency)     2.42       -5.49        70.97

Table 2
Project WISE-UP Intermediate Outcome Data

Year 1                                                       % of
                                                          Participant
Outcome Measures          Pre-test  Post-test  Changes %   Improved

Drug & Alcohol
  Knowledge (20) *         10.68      11.86     +11.05       45.45
Drug & Alcohol
  Attitudes (20)           15.54      15.91      +2.38       36.36
Leisure Perception (40)    29.18      30.00      +2.81       50.00
Leisure Attitudes (40)     29.32      31.45       +7.3       54.55
Conflict Resolution (10)    5.55       6.09       +9.7       54.55
Decision Making (10)        7.45       8.18       +9.8       54.55
Decision Making
  Behavior (20)            14.54      16.36     +12.54       63.64
Family Bonding (60)        45.27      47.64       +5.2       45.45
Study Habits (28)          22.32      22.86       +2.4       36.36

* Total possible scores for each variable is presented in parenthesis