Why evaluation matters: determining effective school counseling practices.

Why does evaluation matter so much in school counseling? Most importantly, when we evaluate our interventions and programs, we can be more certain that what we are doing is making a difference for our students. We have a professional responsibility to show that what we are doing is effective. As the following articles in this special issue of Professional School Counseling demonstrate, with evaluation there is a clearer picture of what works and what needs to be done differently. There is more clarity about the outcomes of our efforts. Additionally, evaluation results demonstrate the impact and value of our work to key stakeholders such as parents, administrators, and school boards, which can help justify resources for school counseling programs.

School counselors have been encouraged to evaluate programs for a variety of reasons for many years (Aubrey, 1982; Fairchild, 1993; Fairchild & Zins, 1986; Myrick, 1984), but the need is even more imperative today. Educators and mental health providers in all fields are under ever-increasing scrutiny to demonstrate evidence of effectiveness and accountability. "Because we think it works" or "because it has always been done this way" are no longer legitimate reasons to continue any educational or mental health intervention. In the accountability era,

What gets measured gets done. If you don't measure results, you can't tell success from failure. If you can't see success, you can't reward it. If you can't reward success you're probably rewarding failure. If you can't see success, you can't learn from it. If you can't recognize failure, you can't correct it. If you can demonstrate results, you can win public support. (Osborne & Gaebler, 1992, p. x)

The impetus to demonstrate accountability through evaluation is reflected in current education initiatives and funding. The U.S. Department of Education (2001) states that evaluation is crucial in all educational efforts, and that it informs project activities and practices, justifies expenditure of funds, enhances administrative planning and policy-making, assures that project objectives have been met, provides evidence for program achievements, monitors program implementation, notes unintended consequences, informs allocation of resources, and identifies problems and costs.

Perhaps most importantly, our country holds the premise that every child deserves the opportunity to have a meaningful education (Rebell & Wolff, 2008). School counselors have the ethical imperative of ensuring that our work contributes in substantial ways toward making it possible for all students to succeed. Ultimately, evaluation helps us to do our best work, with the greatest impact, most efficiently.

DEFINING EVALUATION

There are many definitions of evaluation. In general, it is the purposeful and systematic collection and analysis of data or information used for the purpose of documenting the effectiveness, impact, and outcomes of programs, establishing accountability, and identifying areas needing change and improvement (America's Career Resource Network, n.d.; Wall, Dimmitt, Wong, & Fuller, 2007). Evaluation asks, "Did this program or intervention make a difference for these kids, in this setting?" with a goal of providing useful feedback about a specific population in a unique context.

Evaluation is not the same thing as research. Both use the basic scientific method in that they ask a question, construct a hypothesis, test the hypothesis, analyze data, and draw conclusions based on that data. But research seeks to test broad, generalizable hypotheses such as "Cognitive behavioral group therapy is effective for people with anxiety disorders." A related evaluation hypothesis would be "Cognitive behavioral therapy as done by the school counselor in Jefferson Middle School is effective for a group of eight students who have test anxiety." We need the research--it provides crucial information about what is likely to work for most people in most settings. But it is the evaluation that will tell us whether the intervention worked here and now with our students.

Another way to think about the relationship between assessment and research is to metaphorically link them to assessment of learning (and indeed, assessment and evaluation are both measurements, but of different practices and outcomes). A specific teacher's test of math content knowledge is related to a nationally normed test such as the math SATs in the same way that evaluation is related to research. The local test is not applicable in all settings, it is unique to the students and teacher involved, and it provides constructive context-specific information. It may be useful for a larger audience--the teacher could post the test and see if it helps other people think about how to test their students' math knowledge--but it does not have to be. The local test is much easier and faster to create, and it needs to be "good enough." The SATs, on the other hand, because they are used across the country in many contexts, need to meet much more rigorous standards. They are nationally normed and have proven reliability and validity. Significantly more time and effort go into creating them and demonstrating their accuracy as measurement tools.

Research is more powerful than evaluation in large part because it uses specific research designs (double-blind studies, control groups, outcomes over time, multiple measurements of outcomes, the use of measurements that are reliable and valid, etc.) that allow for greater certainty in hypothesis testing. For instance, we are more likely to trust a medicine if we know it has been researched using a double-blind study to control for placebo effects. In research, the cost of an error (saying that something is effective when it actually is not) is very serious and costly. If the research about an educational or counseling intervention is wrong, then people would waste time and money implementing something that may not work very well (Dimmitt, Carey, & Hatch, 2007). If research about a medical intervention is wrong, people's health would be impacted.

There is less at stake with evaluation, because the aims are different. Evaluation is done to determine whether it is likely that a specific practice, intervention, or program is effective in a particular context. Evaluation also influences program decision-making at the local level (Dimmitt et al., 2007). It is especially useful for schools because most educational stakeholders (administrators, parents, and teachers) want to know if something is effective for their students or their child, and they are less invested in knowing that the results are generalizable to a broader population. While using strong study design is helpful in evaluation and can provide additional power to the results found, evaluations often use less rigorous designs to test the hypothesis because the goal of evaluation is different from the goal of research. Not insignificantly, research is also much more time-intensive and vastly more expensive than evaluation.

EVALUATING PROGRAMS OR INTERVENTIONS?

It is useful to evaluate both programs and interventions. An intervention is a specific activity or group of related activities that is designed to help people learn new information or new skills. It is usually part of a program, but it may be stand alone. A program is a structured set of interventions and services designed to accomplish specific goals. The ASCA National Model[R] (American School Counselor Association, 2005) is a program, whereas Second Step (Committee for Children, 2009) is an intervention. An ongoing social skills group, a classroom curriculum unit on postsecondary options, and a school-wide PeaceBuilders (2009) implementation are examples of interventions at the group, classroom, and school levels.

With interventions, we need to know what outcomes the intervention is designed to impact and need to have ways to measure those outcomes. If the intervention is designed to increase social skills, which social skills, and how will we know that the intervention has worked? Is it enough to ask the students if they know the information, or do we want to see behavioral evidence? Parents and teachers can be asked about changes they've seen as well as students themselves. It is obviously crucial that the instruments used to measure the outcomes are accurate assessments of change, otherwise your intervention might e working but the results are not being reflected in the evaluation.

With program evaluation, demonstrating impact is more complicated, because there are usually multiple interventions and services that make up the program. There are more evaluation questions that can be asked, in addition to whether the interventions are working. We can question if the program as a whole is accomplishing its goals and if it has the necessary resources to meet its goals. Are there additional interventions or program components that could improve the overall program impact? Is there any effort that could be eliminated to increase efficiency and effectiveness? And of course, for many program evaluations, the bottom line is whether the program can demonstrate its impact and cost-effectiveness (Wall et al., 2007).

A slightly different set of questions involves the extent to which students, parents, and school personnel believe that the school counseling program is meeting school and student needs (Dimmitt et al., 2007). Surveying high school seniors about school counseling program components such as college and career counseling, course advising, and general support provides valuable information about how the program is being used, how it is perceived, and how effective it is according to those it serves. Demonstrating impact in this way can build social capital in the system as nothing else can.

As the articles in this journal indicate, identifying the outcomes of school counseling programs, determining the best way to ascertain that student behaviors are changing in anticipated ways, and measuring those changes are all vital in evaluation. The practitioners who share their stories here are to be applauded for their efforts and their courage in providing a broad range of examples of how school counselors can use evaluation to demonstrate that the work does make a difference, and that it impacts key student outcomes.

TYPES OF EVALUATION

The strongest evaluation designs involve continuous data gathering for the duration of the intervention or program. The ongoing gathering of information allows for the clearest knowledge that the program is being implemented as planned, and that practices are having the intended impact. This level of measurement takes time and energy that practitioners may not have, however, so most evaluations do data gathering at several key points during the process.

Needs Assessment

The earliest level of data or information gathering that may be part of an evaluation is a needs assessment. Needs assessments can help determine who needs the intervention or program and how great the need is. Sometimes needs assessments are used to identify possible interventions as well. For instance, if a high school counselor wanted to develop some content-specific support groups for students, she could do a needs assessment survey with her students to determine what kinds of groups they think would be helpful (completing financial aid forms for college, coping with parental divorce, managing test anxiety, gay/lesbian concerns, study skills, etc.). The results will inform her about the concerns of her students and the extent of the need. Knowing that the groups component of her guidance program is designed with student needs in mind will support any evaluation she does of the program's impact and outcomes.

Using Outcome Research to Identify Possible Interventions

Another possible early type of information gathering in evaluation is a determination of which possible intervention or program component is most likely to work within your school. Outcome research can be used to identify what has generally been found to be effective, and then you can decide what is likely to work in the local setting, and how to evaluate it. When interventions that already have been found to be effective are chosen, it is easier to know what student outcomes to measure, because the research often contains valuable information about the intended results of the intervention and what student behaviors, knowledge, or skills or have typically been impacted (Dimmitt et al., 2007). One of the advantages to evaluation (compared to research) is that the school-specific information that school counselors typically hold about the unique qualities of the student population and the family and community factors that influence student outcomes all shape the choices that get made about what interventions to implement and what outcomes to evaluate. This allows intervention choices to be tailored to the distinctive needs of our students, which increases the likelihood of success (Clewell & Campbell, 2007).

Formative Evaluation

Once the intervention or program component has been chosen and is being implemented, formative or process evaluation can be used to check that it is having the initial hoped-for impact. Formative evaluation occurs during the process of the implementation and allows for modification for improvement if necessary. For instance, if after implementing Student Success Skills (Brigman & Webb, 2004) with some of the fifth graders in your school, your formative evaluation indicates that they are understanding the study skills being taught but not yet using them, then some adjustments to the intervention might be warranted to provide more practice time for the behaviors or to provide more positive reinforcement for students who are using the skills. Either way, the impact of your intervention at the end is likely be stronger when you improve the delivery based on formative assessment. Because it is your goal to improve students' study skills (and hence their academic outcomes), identifying how to do this most effectively at any point in the process should only enhance your eventual findings.

Formative evaluation also allows you to get a snapshot of how participants feel about the intervention. What are the students' thoughts and feelings about the intervention? Do they think they are learning new information and/or skills? If there are teachers or administrators involved, what do they think about the intervention so far? This can all be valuable feedback that provides you with ideas for improving your process in an ongoing way, with the goal of improving eventual outcomes.

Implementation Evaluation

Gathering information about how an intervention or program component was actually put into practice provides another set of useful data in evaluation. Implementation evaluation is concerned with the consistency and quality of the delivery of the program component or intervention. If more than one person is involved, did they all receive similar training? Did every class get the same amount of time on the materials? Did every group meet the same number of times?

This information can help us be more effective, because if the only students who benefited from the curriculum instruction were the ones who had the time to practice the skills, then that lets us know that all students need to practice the skills in order for this intervention to be successful with this group of students. It also can help us be more efficient, for if all students benefited from an intervention (the outcomes for all groups were similar) but some met six times and others met eight times, then we know that maybe we only need to meet six times for the intervention to be impactful.

Implementation evaluation and documentation also provide very useful records of what was done, for how long, and what was measured. This can prove invaluable for subsequent implementations and evaluations. It is much easier to replicate an effective intervention or to fix an ineffective intervention if there is documentation of what occurred.

Outcome Evaluation

When people think of evaluation, outcome evaluation is typically what they have in mind. This is the measurement of the impact of the intervention, and it is arguably what stakeholders are most interested in. It answers the questions about the extent to which the desired changes are occurring, what is and is not working, how student outcomes are being improved, and so on.

There are various ways to conceptualize the different types of outcomes that school counseling programs and interventions typically impact. The ASCA National Model (2005) differentiates among student achievement data, achievement-related data, and standards- or competency-based data. Standardized test scores (state-level and nationally normed), graduation rates, and college acceptance rates are examples of achievement data. The most important place to show intervention impact is in achievement data, because it reflects student learning outcomes that are of critical importance to stakeholders. But, because so many factors impact these key data outcomes (there are multiple school, school leadership, teacher, and curriculum factors that affect achievement outcomes; Marzano, 2003; Zigarelli, 1996), demonstrating that a school counseling intervention or program component has an impact is challenging, particularly with evaluation results.

Achievement-related data concern the multiple factors that can influence student achievement, such as attendance, course enrollment, drug/alcohol use, mental health challenges, and parent involvement. Research in education and school counseling has found that improvements in these areas promote student achievement outcomes (Wang, Haertel, & Walberg, 1993). Many school counseling interventions or program components are designed to improve achievement-related data.

Standards- or competency-related data are the direct, first-order student-based outcomes of an intervention or program component. Did students achieve the learning objectives of the curriculum intervention? What do students now know, or know how to do? Have students completed a college application, created a portfolio of career options, or demonstrated conflict resolution skills in a role play? Has test anxiety decreased for students who participated in the anxiety-reduction group? Did the friendship groups help students develop their targeted social skills? This is clearly an important type of data to collect, as it demonstrates immediate effectiveness (or not) of the school counselor's efforts. If the competency-related data aren't showing an improvement, odds are very good that the intervention isn't impacting achievement-related or achievement data. Most evaluation efforts will be targeted at demonstrating that there was a difference in student data at this level.

PARTNERING FOR SUCCESS

No successful effort is accomplished alone. As with all aspects of school counseling, collaboration and teaming make evaluation more doable and rewarding. You may have school colleagues who enjoy data collection or analysis, or there may be a parent in the community who can volunteer relevant skills. School counselor educators can support evaluation efforts through consultation and help with evaluation design and data analysis. School counseling interns can do evaluation during their time at the school, as part of their training, or for credit through their university. Evaluation of any school program or intervention is of interest to administrators, who can provide time and money for the effort.

Internet resources provide information about evaluation design, instruments that measure outcomes related to school counseling interventions, professional development about evaluation, and evaluation results from other schools. The American Evaluation Association (2009) has Web-based evaluation resources including lists of measures that can be used in evaluation, evaluation training, and related publications. The National Center for Education Evaluation and Regional Assistance (2009), sponsored by the U.S. Department of Education, provides technical support to educators and disseminates research and evaluation results and materials. The Center for School Counseling Outcome Research (Carey & Dimmitt, 2006) provides Web-based information, surveys, outcome research summaries of relevant interventions, and research briefs.

THE FUTURE OF EVALUATION IN SCHOOL COUNSELING

Evaluation is crucial to the school counseling profession in myriad ways. Demonstration of impact and effectiveness, school by school, program by program, child by child, builds evidence that school counseling contributes in important ways to the educational endeavor. In this way, community awareness and understanding of the role of school counselors develop. Local evaluation of interventions and programs implemented for district children matters to parents, administrators, and members of that community in ways that broader educational outcomes do not.

As we build our knowledge of which program components make the biggest difference to students in the most contexts, and as we learn that certain interventions seem to be effective when evaluated by many different practitioners, we generate useful information about what is likely to work. Ideally, we would have sound research about every intervention we use, but this is far from the case. The research base is being generated, but interventions are needed now, and in the absence of research, evaluation is much better than no assessment of impact at all. The authors of these articles are to be commended for taking these crucial first steps, and for providing evidence that evaluation is possible even for overworked practitioners who are new to the process. Evaluation is a work in progress, like any human endeavor. As we might say to our students, taking that first step in a new direction may be the hardest, but with others sharing the path and the knowledge that our goal is a worthy one, we can make the journey. I

References

America's Career Resource Network. (n.d.). Basic program evaluation. Retrieved April 19, 2009, from http://cte. ed.gov/acrn/evaluation/documents/presentation508.doc

American Evaluation Association. (2009). Home page. Retrieved April 28, 2009, from http://www.eval.org

American School Counselor Association. (2005). The ASCA national model: A framework for school counseling programs (2nd ed.). Alexandria, VA: Author.

Aubrey, R. F. (1982). Program planning and evaluation: Road map of the 1980s. Elementary School Guidance and Counseling, 17, 52-60.

Brigman, G., & Webb, L. (2004). Student Success Skills: Classroom manual. Boca Raton, FL: Atlantic Education Consultants.

Carey, J. C., & Dimmitt, C. (2006). Resources for school counselors and counselor educators: The Center for School Counseling Outcome Research. Professional School Counseling, 9, 416-420.

Clewell, B. C., & Campbell, R B. (2007). Good schools in poor neighborhoods: Defying demographics, achieving success. Washington, DC: Urban Institute Press.

Committee for Children. (2009). Second Step: A violence prevention curriculum. Retrieved April 29, 2009, from http://www.cfchildren.org

Dimmitt, C., Carey, J. C., & Hatch, T. (2007). Evidence-based school counseling: Making a difference with data-driven practices. Thousand Oaks, CA: Corwin Press.

Fairchild, T. N. (1993). Accountability practices of school counselors: 1990 national survey. The School Counselor, 40, 363-374.

Fairchild, T. N., & Zins, J. E. (1986). Accountability practices of school counselors: A national survey. Journal of Counseling and Development, 65, 196-199.

Marzano, R. J. (2003). What works in schools: Translating research into action. Alexandria, VA: Association for Supervision and Curriculum Development.

Myrick, R. D. (1984). Beyond issues of school counselor accountability. Measurement and Evaluation in Guidance, 16, 218-222.

National Center for Education Evaluation and Regional Assistance. (2009). Home page. Retrieved April 30, 2009, from http://ies.ed.gov/ncee/

Osborne, D. E., & Gaebler, T. (1992). Reinventing government: How the entrepreneurial spirit is transforming government. Reading, MA: Addison-Wesley.

PeaceBuilders. (2009). PeaceBuilders[R] for safe, positive learning environments. Retrieved April 30, 2009, from http://www.peacebuilders.com

Rebell, M. A., & Wolff, J. R. (2008). Moving every child ahead: From NCLB hype to meaningful educational opportunity. New York: Teachers College Press.

U.S. Department of Education. (2001). Fund for the Improvement of Postsecondary Education--Notes on evaluation design. Retrieved April 21, 2009, from http://www.ed.gov/print/about/offices/list/ope/fipse/ notes.html

Wall, J., Dimmitt, C., Wong, W., & Fuller, L. (2007). Useful evaluation of career development programs [Webcast]. Retrieved April 18, 2009, from http://cte.ed.gov/acrn/ webcast1/eval/viewer.html

Wang, M. C., Haertel, G. D., & Walberg, H. J. (1993). What helps students learn? Educational Leadership, 51(4), 74-80.

Zigarelli, M. A. (1996). An empirical test of conclusions from effective schools research. Journal of Educational Research, 90, 103-110.

Carey Dimmitt, Ph.D., is associate director, Center for School Counseling Outcome Research, and program coordinator and associate professor, School Counseling Program, University of Massachusetts, Amherst. E-mail: cdimmitt@ educ.umass.edu