Training school counselors in program evaluation.

Today's school counselors are faced with demands to demonstrate the impact and effectiveness of their counseling programs. Twenty-eight school counselors from a large Southwestern school district participated in a program evaluation training workshop designed to help them develop evaluation skills necessary for demonstrating program accountability. The majority of participants expressed high levels of interest in evaluating their programs but believed they needed more training in evaluation procedures. The authors discuss implications and make suggestions for future training and research on program evaluation in school counseling.

*********

In recent years, school counselor accountability has received considerable attention in the professional literature (Dahir & Stone, 2003; Fairchild, 1993; Fairchild & Seeley, 1995; Gysbers & Henderson, 2000; Isaacs, 2003; Myrick, 2003; Otwell & Mullis, 1997). Education reform measures, particularly the No Child Left Behind Act (U.S. Department of Education, 2001), have changed the landscape in which today's school counselors practice. Historically, school counselors have been connected with school reform movements (Herr, 2002). Yet, as noted by Dahir and Stone, school counselors have not often been held accountable for their contributions to student success. In the 21st century, however, school counselors are forging a professional identity that emphasizes leadership, collaboration, and fostering the academic achievement of all students (Adelman, 2002; House & Hayes, 2002; House & Scars, 2002). As a result, school counselors are increasingly being called upon to take the lead in implementing results-based programs with an emphasis on systematically evaluating their outcomes and impact on student success.

As the professional identity of school counselors has evolved, perspectives on demonstrating school counseling program accountability have changed as well. Myrick (1990) documented the growing interest in school counselor accountability, especially during the 1980s, and foreshadowed the current focus on accountability in schools. However, historical school counselor accountability activities, such as keeping detailed calendars and logs of services, are no longer considered sufficient measures of program success (Astramovich & Coker, 2003; Fairchild, 1993). Borders and Drury (1992) noted that school counseling accountability has increasingly shifted toward an emphasis on program outcomes rather than on reports of services delivered. In an effort to expand the repertoire of accountability measures, Fairchild and Seeley (1995) incorporated formal evaluation activities into their suggestions for school counseling accountability practices, specifically emphasizing needs assessments, data analysis, and student, teacher, and parent evaluations of the school counseling program. More recently, the ASCA National Model[R] (American School Counselor Association, 2003) for school counseling programs emphasized the use of data and evaluation for program improvement and professional accountability. In addition, Hosie (1994) highlighted how counseling students can benefit from training in program evaluation. The Council for Accreditation of Counseling and Related Educational Programs (2001) also incorporated evaluation methods into its standards for the counselor education curriculum.

Counseling program evaluation refers to the ongoing use of evaluation principles by counselors to assess and improve the effectiveness and impact of their programs and services (Astramovich & Coker, 2003). Rather than merely being a defensive measure against accountability pressures, program evaluations have intrinsic value in helping counselors monitor and evaluate the effectiveness of the services they provide to clients. According to Isaacs (2003), program evaluations can help school counselors determine the extent that programs are positively impacting students and can help identify barriers to student success, subsequently guiding counselors in designing effective programs for the students they serve. Gysbers, Hughey, Starr, and Lapan (1992) also stressed that feedback from evaluations can help counselors refine and modify components of their comprehensive school counseling programs.

More specifically, Studer and Sommers (2000) indicated that results evaluations allow school counselors to identify observable outcomes of their programs and interventions such as changes in students' behavior, academic performance, and personal/ social growth. In addition, benchmark evaluations of progress can aid the school counselor in determining the effectiveness of counseling outcomes not immediately measurable due to the process nature of counseling. In these instances, data can be collected over a multiyear period to determine program effectiveness over time. Overall, school counseling program evaluation may be viewed as a collaborative, planned process using multiple sources of data for assessing program effectiveness.

Although the ASCA National Model (2003) reinforces the importance of evaluation and accountability practices, school counselors generally have expressed little interest in, or experience with, program evaluation methods (Schmidt, 1995). Lack of appropriate training in research and evaluation has been suggested as a primary factor preventing school counselors from undertaking program evaluations (Fairchild, 1993). Other reasons that school counselors may avoid formal evaluation activities include uncertainties about the evaluation process, concerns that evaluations are too complicated and time-consuming, and fears about how the evaluation information will be used (Lusky & Hayes, 2001). According to Isaacs (2003), school counselors also may resist using program evaluation because they lack confidence in their ability to use data effectively and apply it meaningfully to their professional practice.

Considering the emphasis on accountability in education and the coinciding calls for school counselors to develop evaluation skills, a field-based introductory training session on evaluation methods was developed and provided to a small group of practicing school counselors. The purpose of the training was to explore initial attitudes toward accountability practices and program evaluation and to gauge receptiveness to conducting evaluations. Another goal of the training was to identify barriers or limitations that prevent school counselors from conducting program evaluations. Finally, the training was designed to provide school counselors with a foundation for evaluating their counseling services.

METHODS

Curriculum and Training

The program evaluation training curriculum was based on concepts outlined in the ASCA National Model (2003), particularly emphasizing the relationship between the management system and accountability components of comprehensive school counseling programs. Also included were accountability and program evaluation concepts as discussed by Dahir and Stone (2003), Gysbers and Henderson (2000), Lapan (2001), and Myrick (2003). Overall, the curriculum was designed to help participants (a) understand the role of accountability in today's educational environments; (b) understand the emphasis on accountability and program evaluation in the ASCA National Model; (c) define program evaluation; (d) understand the evaluation process including the role of needs assessment, program planning, program implementation, and assessing outcomes; and (e) plan to implement their own school counseling program evaluation.

The school counseling program evaluation training was delivered as a 3-hour workshop that included a didactic presentation followed by activities designed to help counselors begin planning for program evaluations. The didactic portion of the training included a PowerPoint presentation that specifically addressed the current focus on accountability in school counseling, the management system and accountability in the ASCA National Model (2003), definitions of program evaluation, the rationale for program evaluation in school counseling, categories of program evaluation (e.g., needs assessments, formative evaluations, process evaluations, and outcomes evaluations), and stages of program evaluation implementation. During the activity portion of the training, participants were guided in conceptualizing the program evaluation process to their particular school setting. Specifically, participants identified initial goals for a school counseling program evaluation at their school site, primary questions to be addressed, sources of data and information, key people involved, and a timeline for completion.

Participants

The program evaluation training was provided as a 3-hour workshop to a total of 28 school counselors from a large, urban school district in the Southwestern United States. A $250 Association for Counselor Education and Supervision (ACES) research grant was awarded to develop a training curriculum and provide program evaluation training to an initial group of 5 school counselors. The same training was subsequently provided to a second group of 23 school counselors at the request of the district's directors of guidance.

Group I. For the first training group, participants were selected from a convenience sample of practicing school counselors by the posting of an invitation to participate on the district's school counselor listserv. In order to ensure representation across grade levels, 5 participants were purposively selected (Gall, Borg, & Gall, 1996), including 1 elementary, 2 middle, and 2 high school counselors, with a mean of 5.6 (SD = 4.22; range 3-13) years of experience in school counseling. Members of the first group participated in the program evaluation training and then were asked to follow through on some of the evaluation activities and plans developed during the training. Two weeks following the training, on-site visits were arranged with each participant to follow up on their plans for program evaluations. An ACES grant funded a $50 participation stipend for members of the initial training group.

Group 2. After the initial training was completed, the school district directors of guidance requested that an additional training be conducted with a group of year-round elementary school counselors in the same district. Participants in the second training consisted of 23 elementary school counselors with a mean of 8.57 (SD = 6.86; range 1-28) years of experience in school counseling. The program evaluation training for the second group was delivered as part of a planned professional development day for school counselors. Participants in the second group did not receive a stipend for participation.

Survey Instrument

In order to assess initial attitudes toward accountability and program evaluation, a brief survey instrument was created for participants to complete prior to the training. The instrument consisted of demographic items including the current school setting, years of experience in school counseling, and amount of previous training in program evaluation during graduate school or at professional workshops and conferences. Participants then were asked to respond to 11 items using the following 5-point scale: 5 = strongly agree, 4 = agree, 3 = uncertain/ neutral, 2 = disagree, and 1 = strongly disagree. Four items addressed participants' views of the role of school counselors in the accountability movement, four items addressed participants' attitudes toward program evaluation, and three items addressed training in program evaluation. The last part of the instrument consisted of two open-ended questions eliciting general comments about program evaluations and identifying barriers to conducting program evaluations in school counseling. Specifically, participants were asked, "What are some of the primary concerns you have about conducting an evaluation of your school counseling program?" and "What would prevent you from conducting an evaluation of your school counseling program?"

RESULTS

Previous Experience with Program Evaluation

The majority of training participants, 15 (53.6%), did not receive training in program evaluation during their graduate coursework. Of the remaining participants, 12 (42.9%) indicated receiving some program evaluation training during graduate-level coursework, and 1 (3.6%) did not specify. Those participants receiving graduate training in evaluation identified a few specific courses that covered evaluation including K-12 Guidance, Elementary School Counseling, Research Methods, and Practicum/ Internship.

When asked if they had received program evaluation training at a workshop or a professional conference, 22 (78.5%) participants reported receiving no such training, 5 (17.9%) had received some training, and 1 (3.6%) did not specify. Of the 5 participants reporting some previous professional development or workshop training in program evaluation, sponsoring organizations identified included a state department of education, the Education Trust, and ASCA. Overall, the majority of participants had received no prior training in counseling program evaluation methods.

Initial Attitudes Toward Accountability

Participants responded to four survey items addressing their views on the role of school counselors in accountability. Of the trainees, 18 (64.3%) agreed or strongly agreed that accountability is currently a major focus in school counseling, while 6 (21.4%) were uncertain and 4 (14.3%) disagreed. In addition, 24 (85.7%) agreed or strongly agreed that school counselors should make time to demonstrate accountability of their programs, with 3 (10.7%) being uncertain and 1 (3.6%) disagreeing. Among the participants, 23 (82.1%) agreed or strongly agreed that outcome data on a school's counseling program enhance the school's accountability, while 4 (14.3%) were uncertain and 1 (3.6%) disagreed. Finally, 16 (57.2%) participants agreed or strongly agreed that they play a key role in accountability at their school; however, 8 (28.6%) were uncertain or neutral and 4 (14.2%) disagreed or strongly disagreed.

Initial Attitudes Toward Program Evaluation

Participants responded to four survey items addressing their initial attitudes about program evaluation. A total of 23 (82.2%) participants agreed or strongly agreed that they understood the rationale and need for conducting program evaluations, with 3 (10.7%) being uncertain and 1 (3.6%) disagreeing. Additionally, 26 (92.8%) expressed a willingness to conduct a school counseling evaluation, with 2 (7.1%) being uncertain. However, when asked if they understand the process of implementing an evaluation, only 13 (46.4%) agreed, while 10 (35.7%) were uncertain/neutral and 5 (17.9%) disagreed. Furthermore, when asked if program evaluations were too time-consuming to be conducted, participants' responses were split, with 9 (32.1%) agreeing or strongly agreeing, 9 (32.1%) expressing uncertainty/neutrality, and 10 (35.7%) disagreeing or strongly disagreeing.

Training Needs

Participants responded to three questions that addressed training needs in program evaluation. When asked if they needed training in program evaluation methods in order to evaluate their school counseling programs, 21 (75.0%) agreed or strongly agreed, 1 (3.6%) was uncertain or neutral, and 6 (21.5%) disagreed or strongly disagreed. However, when participants were asked if they believed they did not have enough research skills to conduct evaluations, 20 (71.4%) disagreed or strongly disagreed, 4 (14.3%) agreed, and 4 (14.3%) were uncertain. Finally, 26 (92.8%) participants agreed or strongly agreed that current students in school counseling should learn about program evaluation, while 2 (7.1%) were uncertain.

Concerns About School Counseling Program Evaluation

Participants were asked to answer an open-ended question focusing on their concerns over implementing program evaluations. Several themes emerged from their comments. First, some participants expressed a need for finding practical and simple evaluation methods for use in school counseling, and one participant indicated needing help narrowing down the components of the school counseling program to be evaluated. Another concern raised by participants involved the way that results would be used and who would have access to the findings. Some also indicated a concern that evaluation results would be used to label the success or failure of the school counseling program.

Participants also were asked to identify any potential barriers to their implementation of a program evaluation. The amount of time needed to conduct an evaluation was cited as a possible barrier. Participants also indicated that the cumbersomeness and overwhelming nature of evaluation might prevent them from engaging in evaluations. Another perceived barrier involved minimal support by administrators and the lack of support staff to help with the evaluation process. Finally, several participants indicated that lack of training and uncertainties about the tools and techniques of evaluation hampered their ability to conduct program evaluations.

Follow-Up with the First Group

On-site school visits were arranged with the first group of 5 trainees to follow up on their plans for implementing school counseling program evaluations. At the time of the visits, 2 counselors had initiated some evaluation activities since participating in the training. One of the middle school counselors was actively planning to evaluate the impact and effectiveness of an anger management group. The elementary counselor from the first group was exploring attendance data and developing interventions to target students with poor attendance rates.

The remaining participants from the first training group, 1 middle school counselor and 2 high school counselors, were unable to initiate any evaluation activities after the training. All 3 cited time constraints as their primary barrier to engaging in evaluation. Another barrier expressed by participants involved identifying the type of data and analysis necessary for answering their evaluation questions. For instance, one high school counselor expressed a desire to study college enrollment patterns among minority students but had difficulty in determining the types of questions to include on a needs assessment to evaluate students' plans for attending college.

Participants from the initial training group suggested that program evaluation training was valuable to them. They furthermore indicated that program evaluation training should be sequential and delivered in multiple, small segments to allow for application of skills between training sessions.

DISCUSSION

The majority of the school counselors who participated had not previously received training in evaluation methods during their graduate coursework or at professional development workshops. This finding is consistent with Schmidt's (1995) assertion that school counselors generally have little experience or training in program evaluation methods. Given the increasing emphasis on program accountability in education, school counselors and school counselor educators may need to reassess the importance placed on acquiring evaluation skills both during and after graduate training. Participants also indicated that graduate students in school counseling should receive training in program evaluation. These suggestions mirror Trevisan's (2000) call for counselors to receive focused evaluation training in addition to general research methods coursework. Counselor educators may therefore consider examining their programs' curriculum, ensuring that students have, at a minimum, foundational training in evaluation methods upon which to build. Program evaluation practice could be incorporated into practicum and internship experiences as a means for helping students acquire practical evaluation skills.

Participants generally believed that accountability is a major concern of school counselors and indicated that efforts to demonstrate accountability of school counseling programs are important. However, just under half of the trainees indicated they did not see themselves playing a key role in the accountability of their schools. This finding echoes Dahir and Stone's (2003) assertion that school counselors often have been left out of the accountability loop. Although interest and desires to demonstrate accountability may be high, helping counselors become actively involved in an accountability identity at their schools seems essential. Conducting program evaluations may therefore help school counselors develop a solid accountability role in their schools (Trevisan, 2000).

Participants from both groups understood the rationale and need for evaluating school counseling programs and most indicated they had ample research skills to do so. However, they specifically stated a need to learn about methods and implementation of program evaluations. Furthermore, some of the school counselors suggested that ongoing training and supervision in evaluation methods would give counselors an opportunity for hands-on practice and application of evaluation skills. Similar to calls made by Loesch (2001) and Lusky and Hayes (2001), the trainees also expressed a need for finding evaluation models that are practical and specific to the needs of school counselors. Lastly, participants stressed that school counseling evaluation methods must be time efficient and generate meaningful information for counselors. The development of counselor-specific evaluation methods therefore appears paramount in helping school counselors implement program evaluations.

Although many of the implications of the training clearly call upon counselor education programs to enhance the graduate preparation of counselors in program evaluation, several recommendations for practicing school counselors were evident. First, school counselors need to seek continuing education opportunities for furthering their skills in evaluation through university coursework and professional conferences and workshops. School counselors also should work with their district directors of school counseling to develop collaborative relationships with university faculty with expertise in evaluation in order to provide opportunities for training and hands-on practice in program evaluation methods. Finally, as suggested by the ASCA National Model (2003), school counselors should allocate time in their schedules for management system and accountability activities, including evaluations of their comprehensive school counseling programs.

CONCLUSION

Overall, this training of school counselors in program evaluation helped affirm school counselors' interest in acquiring evaluation skills. Specifically, participants desired practical, hands-on training in evaluation methods. As part of future training, school counselors also may benefit from seeing examples of counselor-implemented evaluations (e.g., Cook & Kaffenberger, 2003) and hearing success stories directly from school counselors who are engaged in evaluation practices to improve their programs (e.g., Fairchild, 1994). In order for program evaluations to become a routine practice, school counselors need to see the usefulness in doing evaluations and have practical methods for doing so. In addition, research on school counseling program evaluation may help clarify the concerns of school counselors in implementing evaluations and may help spur the development of counselor-specific evaluation methods. Ultimately, results from ongoing program evaluations have the potential to underscore the effectiveness of comprehensive school counseling programs and further enhance the professional identity of school counselors.

References

Adelman, H. S. (2002). School counselors and school reform: New directions. Professional School Counseling, 5, 235-248.

American School Counselor Association. (2003). The ASCA national model: A framework for school counseling programs. Alexandria, VA: Author.

Astramovich, R. L., & Coker, J. K. (2003, October). Program evaluation for school counselors: An exploratory study. Paper presented at the meeting of the Western Association for Counselor Education and Supervision, Tucson, AZ.

Borders, L. D., & Drury, S. M. (1992). Comprehensive school counseling programs: A review for policymakers and practitioners. Journal of Counseling and Development, 70, 487-498.

Cook, J. B., & Kaffenberger, C. J. (2003). Solution shop: A solution-focused counseling and study skills program for middle school. Professional School Counseling, 7, 116-123.

Council for Accreditation of Counseling and Related Educational Programs. (2001). Accreditation procedures manual and application. Alexandria, VA: Author.

Dahir, C. A., & Stone, C. B. (2003). Accountability: A M.E.A.S.U.R.E. of the impact school counselors have on student achievement. Professional School Counseling, 6, 214-221.

Fairchild, T. N. (1993). Accountability practices of school counselors: 1990 national survey. The School Counselor, 40, 363-374.

Fairchild, T. N. (1994). Evaluation of counseling services: Accountability in a rural elementary school. Elementary School Guidance and Counseling, 29, 28-37.

Fairchild, T. N., & Seeley, T. J. (1995). Accountability strategies for school counselors: A baker's dozen. The School Counselor, 42, 377-392.

Gall, M. D., Borg, W. R., & Gall, J. P. (1996). Educational research: An introduction (6th ed.).White Plains, NY: Longman.

Gysbers, N. C., & Henderson, P. (2000). Developing and managing your school guidance program (3rd ed.). Alexandria, VA: American Counseling Association.

Gysbers, N. C., Hughey, K., Start, M., & Lapan, R. T. (1992). Improving school guidance programs: A framework for program, personnel, and results evaluation. Journal of Counseling and Development, 70, 565-570.

Herr, E. L. (2002). School reform and perspectives on the role of school counselors: A century of proposals for change. Professional School Counseling, 5, 220-234.

Hosie, T. (1994). Program evaluation: A potential area of expertise for counselors. Counselor Education and Supervision, 33, 349-355.

House, R. M., & Hayes, R. L. (2002). School counselors: Becoming key players in school reform. Professional School Counseling, 5, 249-256.

House, R. M., & Sears, S. J. (2002). Preparing school counselors to be leaders and advocates: A critical need in the new millennium. Theory Into Practice, 41, 154-162.

Isaacs, M. L. (2003). Data-driven decision making: The engine of accountability. Professional School Counseling, 6, 288-295.

Lapan, R. T. (2001). Results-based comprehensive guidance and counseling programs: A framework for planning and evaluation. Professional School Counseling, 4, 289-299.

Loesch, L.C. (2001).Counseling program evaluation: Inside and outside the box. In D.C. Locke, J. E. Myers, & E. L. Herr (Eds.), The handbook of counseling (pp. 513-525). Thousand Oaks, CA: Sage.

Lusky, M. B., & Hayes, R. L. (2001). Collaborative consultation and program evaluation. Journal of Counseling and Development, 79, 26-38.

Myrick, R. D. (1990). Retrospective measurement: An accountability tool. Elementary School Guidance and Counseling, 25, 21-29.

Myrick, R. D. (2003). Accountability: Counselors count. Professional School Counseling, 6, 174-179.

Otwell, R S., & Mullis, F. (1997). Academic achievement and counselor accountability. Elementary School Counseling and Guidance, 31, 343-348.

Schmidt, J. J. (1995). Assessing school counseling programs through external reviews. The School Counselor, 43, 114-123.

Studer, J. R. & Sommers, J. A. (2000).The professional school counselor and accountability. NASSP Bulletin, 84, 93-99.

Trevisan, M. S. (2000).The status of program evaluation expectations in state school counselor certification requirements. American Journal of Evaluation, 21, 81-94.

U.S. Department of Education. (2001). No Child Left Behind. Available from ED.gov Web site, http://www.ed.gov/nclb/ landing.

Randall L. Astramovich, Ph.D., is an assistant professor of counseling, Department of Educational Psychology, University of Nevada, Las Vegas. E-mail: [e-mail]

J. Kelly Coker, Ph.D., is a licensed professional counselor, Harbin & Associates Psychotherapy, Fayetteville, NC.

Wendy J. Hoskins, Ph.D., is an assistant professor of counseling, University of Nevada, Las Vegas.

An Association for Counselor Education and Supervision grant funded a portion of this field training.