Evaluating after-school programs: early reports find positive gains but more research still needed.

In the early 1990s, the National Education Commission on Time and Learning released a report titled "Prisoners of Time." The authors recognized that time had been a missing element in the national debate about learning and thus called for universal extension of the school day.

About the same time, the Carnegie Corporation of New York issued its report "A Matter of Time: Risk and Opportunity in the Nonschool Hour," which noted that 40 percent of a young adolescent's time is unstructured, unsupervised and unproductive. It also pointed to nonschool hours as a highly promising opportunity to offer young adolescents useful experiences to promote health, growth and development.

Although each report focused more than a decade ago on different aspects of a child's development, together these reports represented the growing recognition that the hours beyond the traditional school day can be a key resource in improving learning and skill building, keeping children safe and helping working parents.

In the years since these reports, after-school programs have become part of the national education landscape and are a leading priority in many communities across the country. In 1997, the C.S. Mott Foundation partnered with the U.S. Department of Education in support of the 21st Century Community Learning Centers, also known as 21st CCLC, and other major after-school initiatives. From 1998 to 2001, the federal commitment to the 21st CCLC initiative skyrocketed from $1 million to $1 billion a year. State and local governments continue to increase after-school funding despite the largest state budget deficits since World War II.

Although after-school programs provide an opportunity to address a number of developmental outcomes, schools and communities are beginning to see after-school hours as an ideal time to reinforce children's learning gains, provide enrichment opportunities and supplement the academic curriculum offered at school.

Are They Effective?

As the interest in after-school programs grows, so does the need for reliable information about what works. Although the field is relatively new, a growing number of studies indicate that after-school programs do make a positive difference in the lives of their participants. Not only do after-school programs provide a safe haven during the nonschool hours, students who consistently participate in quality after-school activities have better grades, greater student engagement in school, increased homework completion, reduced absenteeism and tardiness, greater parent involvement, increased civic engagement and reduced crime and violence in the nonschool hours.

Several have pinpointed the academic gains. The UCLA Center for the Study of Evaluation studied 20,000 elementary school students enrolled in a program known as LA's BEST. The study, "A Decade of Results: The Impact of the LA's BEST After School Enrichment Program on Subsequent Student Achievement and Performance," released in June 2000, reported that students involved in the after-school enrichment program for more than four years experienced significantly improved standardized test scores in language arts, reading, and mathematics when compared to nonparticipants.

Another study, conducted by Stephen P. Klein and Roger Bolus of Gansk and Associates in 2001-2002, revealed that 4th graders who attended the Foundations Inc. School-Age Enrichment Program in the Philadelphia area gained an average of 38 points in reading and 45 points in math compared to 17 and 26 points, respectively, for nonparticipants during the 2001-2002 academic year.

Although these documented results of academic improvement support the value of promoting after-school programs, those referenced above are not an extension of the traditional school-day experience. Rather, these after-school programs provide young people with the opportunity to participate in various activities from which they can learn and grow, including art, dance, music and sports. These activities can contribute to students' overall well-being; improve problem solving, interpersonal, and communication skills; and raise their academic achievement.

Evaluations of after-school programs also indicate a positive impact on student behavior, safety and family life. Policy Studies Associates' evaluation of The After-School Corporation programs in New York City between 1998 and 2000 concluded that the elementary school-age participants' social skills improved, including their ability to maintain self-control, make constructive choices about their behavior and avoid fights. Parents shared that the program helped them balance work and family life and they missed fewer days of work because their children had someplace to go after school.

Current Research

However, other studies have reported no effects, and in some cases negative effects on program participation. These inconsistent findings underscore the relatively newness of the field and a need for a step-by-step progression of more rigorous research studies that can inform program improvement and assess the impact at appropriate junctures in a program's life cycle. One of the first studies needed is identifying the types of after-school programs or program characteristics that are most conducive to positive student outcomes.

Toward this goal, Deborah Vandell, a leading researcher at the University of Wisconsin in Madison, Center for Education Research, and Elizabeth Reisner, a principal at Washington, D.C.-based Policy Studies Associates, are conducting a three-year study of the effects of high-quality after-school programs on developmental and learning outcomes of students who are at high risk of school failure.

The national study draws from existing research, including a recent report by the National Research Council's Committee on Community Level Programs for Youth, which concludes that key program characteristics leading to positive student outcomes include physical and psychological safety, appropriate structures, supportive relationships, opportunities to belong, positive social norms, support for efficacy, opportunities for skill building, and integration of family, school and community efforts.

Early analyses have shown that the 19 elementary and 16 middle school after-school programs in this study have strong ties with community organizations, offer a wide variety of activities ranging from tutoring to supports, demonstrate the likelihood of financial sustainability over several years and indicate high levels of job satisfaction among staff.

Tentative conclusions suggest that high-quality after-school programs (alone and in combination with other after-school activities) can help offset the negative effects of unsupervised after-school hours. Elementary school students who attend after-school programs regularly report better work habits and fewer incidents of misconduct compared to students unsupervised during the after-school hours. Attending school-based after-school programs also was linked to teacher-reported gains in students' work habits, social skills, task persistence and academic performance relative to unsupervised students.

For middle school participants, attending high-quality after-school programs (alone or in combination with other activities) reduced self-reported misconduct and the use of drugs and alcohol in comparison to youth who spent substantial time unsupervised. Teachers also reported relative improvements in work habits among students who regularly attended after-school programs.

Future Research

Much work still needs to be done to identify and support effective after-school practices and programs. Lingering questions include these:

* How are active participation in after-school activities and student outcomes linked? For instance, which has more influence on academic achievement: new skills development, working with a mentor or family participation?

* What overall impact might after-school programs have on academics and how soon?

* How much program participation is enough to produce beneficial outcomes for participants? How much intensity, duration and breadth of activities are needed? How do we know?

* How do we measure, assess and develop programs to improve a range of outcomes for participants? For example, what minimum staff qualifications are necessary to ensure positive participant outcomes?

Many existing studies examined one program at one point in time. More rigorous research designs such as experimental, quasi-experimental and longitudinal studies are necessary to truly understand the impact of after-school programs. Using a theory of change can guide the thinking and implementation of program goals and elements linked to desired outcomes and conditions for optimal success.

In general, assessing short-term outcomes may be appropriate after a year of implementation. Long-term outcomes should be assessed only after the program has had the opportunity to implement its activities with qualified staff and resources, which may take two or three years.

Through continuous, rigorous research, we will have a better understanding of the access, equity and impact of after-school programs and what best practices can meet the diverse needs of participants from all backgrounds and ethnicities.

RELATED ARTICLE: A framework for effective programs.

A new resource, "A Framework for Successful Afterschool Programs," is a guide for improving and measuring the quality and effectiveness of after-school programs.

The web-based document (www. publicengagement.com/framework) grew out of a national summit on after-school programs convened two years ago by the U.S. Department of Education and Arnold Schwarzenegger, and sponsored by the C.S. Mott Foundation. The event drew researchers, program and policy experts, educators and government officials.

As a follow-up, a team of researchers and program experts developed the framework, which states that intentionally linking program goals, program elements, participant outcomes and evaluation is key to improving the effectiveness of after-school programs.

The framework describes a theory of change approach to guide the thinking and implementation of program goals and elements. It outlines a broad range of desired participant outcomes and discusses conditions most suitable for achieving positive results that meet the needs of after-school participants.

A key feature of the framework is a set of after-school participant outcome matrices. These matrices offer examples of program elements, outcomes and performance measures in the areas of academic, social/emotion. health and safety and community engagement goals.

Using this document, school leaders can engage after-school stakeholders in an informed and thoughtful process about developing, implementing and evaluating after-school programs at their schools.

--An-Me Chung

An-Me Chung is a program officer at the C.S. Mott Foundation, 503 S. Saginaw St., Suite 1200, Flint, MI 48502. E-mail: [e-mail]. Eugene Hillsman is a program assistant at the Mort Foundation.