Producing evidence to show counseling effectiveness in the schools.

School counselors have been remiss in documenting how the school counseling program is an essential component to the growth of school-aged youth. As a result, school counseling programs and personnel either have been eliminated or school counselors have been assigned duties that do not reflect their training and education. Many school counselors complain that existing assessment instruments either are too expensive or do not adequately reflect the program objectives and goals. This article provides a step-by-step model for designing testing instruments that can be effective in producing data that reflect program or intervention results. An example of how a middle school counselor assessed her leadership program is included.

**********

School counselors can no longer sit back and hope that others will recognize the good things they are doing. With accountability a top priority in educational settings, programs are now under increased pressure to prove that what they are doing is successful and effective. School counseling programs are no exception. Historically, when school counselors were first making strides in the establishment and maintenance of vocational services in schools, there was some concern about showing evidence of the effectiveness of these efforts (Gysbers, 2004). In time, when a mental health model replaced the vocational emphasis, there was once again a renewed discussion on how counselors could prove that their interventions were helpful (Foster, Watson, Meeks, & Young, 2002). With today's educational reform initiatives and the emphasis on comprehensive, developmental school counseling programs, there is revived vigor to prove that what school counselors do is vital and essential to the growth of school-aged youth (Foster et al.). Unfortunately, because school counselors did not heed the early warnings to actively demonstrate success, programs and personnel were eliminated. Or, in some cases, the building principal assigned non-counseling duties to school counselors that had little relationship to their professional training. Unfortunately, these trends still continue.

Accountability means "being responsible for one's actions ... and documenting effectiveness through measures of professional activity outcomes" (Loesch & Ritchie, 2005, p. 2). Many school counselors provide enumerative data that document such things as student contacts, parent or teacher consultation, or the number of guidance classes taught. However, these numbers do not reveal how well the counselor performs in each of these activities.

Accountability is as important today as it was when guidance programs were first introduced into the schools. Yet school counselors continue to show reluctance to engage in accountability procedures, for numerous reasons:

1. Counselors sometimes view accountability as threatening in that positive results may lead to additional assignments in an already overburdened profession (Frith & Clark, 1982; Mecca, Rivera, & Esposito, 2000; Myrick, 1990).

2. Many counselors do not believe that they have the necessary skills to conduct research (Gladding, 2000) that they perceive as being better left to psychometricians.

3. Some school counselors view accountability as an activity that would negatively impact their performance, particularly if students do not show a significant improvement as a result of an intervention (Mytick, 1990).

4. Administrators often are reluctant to approve of data collection from school-aged youth due to litigation or the belief that students are not reliable sources for obtaining information (Frith & Clark, 1982).

5. Ethical considerations surrounding confidentiality, time constraints, and a lack of funds are additional reasons school counselors cite for not conducting research activities (Fairchild, 1993, as cited in Foster et al., 2002).

A new mindset is needed to convince school counselors to accept the challenge of demonstrating their effectiveness. School board members require a defensible rationale for justifying programs and personnel with concrete data. If professional school counselors do not take an active role in showing how school counseling programs make a positive difference in the lives of school-aged children and adolescents, programs and personnel will continue to be eliminated (Gibson & Mitchell, 1999).

Many professional school counselors who are convinced about the importance of data collection complain about the lack of relevant data collection instruments, and they rely on published standardized inventories. However, these instruments often are too costly and do not always reflect what the counselor needs. Furthermore, counselors hesitate to design an assessment instrument due to the fear that it would not stand up to the conditions of a well-constructed test (validity, reliability, and so on). Yet, it is possible to design assessment instruments that can produce dependable results that are easily interpreted and understood. The purpose of this article is to summarize assessment considerations, and to provide step-by-step guidelines for counselors to design an assessment inventory. In addition, several methodologies in which these instruments can be used are discussed.

ASSESSMENT CONSIDERATIONS

Selecting a testing instrument, orientating, administrating, and interpreting the results for outcome assessment are essential components of any program improvement process. An effective assessment instrument closely aligned with program goals or intervention objectives serves as a barometer for keeping an eye on the desired results. When the test is viewed as relevant to the counselor and the participants, the time spent on designing and taking a test will be more meaningful and more likely to be used appropriately. Test results may be used for raising public awareness, complementing fund-raising, and grant writing. Questions to consider in selecting and implementing an assessment inventory include the following:

1. Who is going to use the results?

2. What is being assessed?

3. Where will the assessment take place?

4. When will the assessment occur?

5. Why is the assessment being conducted?

Who Will Use the Results?

When counselors evaluate a program, counseling strategy, or activity, formative and summative evaluations are essential for determining what is going well and what needs to be revised. Formative evaluation includes analyzing data while the program or activity is being undertaken so that changes can be made prior to termination. Summative evaluation occurs at the conclusion of a program or intervention to assess the final effects of the counselor's efforts (Baker, 2000). Both of these assessment measures rely on each other. After obtaining the data, the counselor then can share the results with administrators and other stakeholders. The use of an easy-to-read chart that visually shows data and progress can attract the attention of policy makers (Hughes & James, 2001).

What Is Being Assessed?

For the most part, school counselors are concerned with the student, the "what" of assessment, and how a particular treatment will impact children and adolescents (Hood & Johnson, 2002). In assessing students, the counselor needs to consider whether to focus on affect (feelings), cognition (learning), behavior (acting), or a combination (Hood & Johnson). Professional school counselors primarily focus on affect, with the assumption that improvements in this area will lead to improvements in academic performance (Loesch & Ritchie, 2005).

Where Is the Assessment to Take Place?

Test interpretation requires attention to where the assessment occurred. Standardized tests are administered using established procedures in order to more accurately interpret results. Different testing environments may result in irregular testing results, leading to validity concerns (e.g., environmental factors that are not controlled). For instance, a middle school counselor led a self-esteem group for seventh graders during lunchtime in a room off the school cafeteria. The room was distracting due to the noise and people inadvertently entering the room while the group was in session. The group participants did not receive as much benefit from this group as they may have if it had been conducted in a room located in a quieter part of the school.

When Will the Assessment Occur?

In this age of testing, teachers often are reluctant to share classroom time with other educational personnel. Yet, teachers are more responsive to sharing students and more willing to collaborate with counselors when they are aware of the purposes of the assessment, how it will be administered and interpreted, and, more importantly, how it will benefit them. Specific times during the academic year may be better for assessment than others. For instance, students' lack of interest in inventories administered during the end of the school year could produce erroneous results. Administering a test before or after a holiday is an additional time period that may not produce the best assessment results.

Why Is the Assessment Being Used?

Counselors need to be vigilant in continually evaluating effectiveness. If a particular intervention or program did not work as anticipated, the counselor either can revise a component or may decide to take an entirely different approach to more successfully meet a program or intervention objective. When stakeholders are apprised of programs or interventions that meet the program objectives or goals, they are more likely to support the program and the personnel influential to the program.

DESIGNING THE INSTRUMENT

The process of determining an assessment procedure and choosing a testing instrument is as important as obtaining the end result. A good instrument consists of questions (also known as items) that are chosen with care, are age-appropriate, and are aligned with the program goals and objectives (Mecca et al., 2000). Information on available instruments can be found by going to the Internet and searching the National Information Center on Health Services Research and Health Care Technology; the Buros Center for Testing at the University of Nebraska, Lincoln; or the ERIC system (produced by the Education Resources Information Center) for studies that have been conducted with different testing instruments.

Unfortunately, there are few "off-the shelf" assessment instruments that completely match a particular program's objectives. In these cases, instruments need to be custom designed (Myford & Cline, 2001). When school counselors develop assessment instruments that reflect their objectives and goals, they obtain quantifiable, valid results (Loesch & Ritchie, 2005).

To begin the instrument designing process, a theoretical assessment foundation is necessary. The groundwork for the instrument can be established through a review of the literature to identify the variables that influence the concept being measured (Melchert & Sayger, 1998) and to provide instrument credibility to the questions (Jones & Kottler, 2006). If the instruments do not meet the developmental needs of the participants, or do not exactly mirror the needs of the research, they may be modified while continuing to embrace the integrity of the test.

What Procedures Are Needed for Collecting Data?

Conducting research using the instrument as an assessment tool is an area that creates apprehension among counselors. Action-based research is a systematic research method by which data are collected in order to improve counseling or performance (Creswell, 2002; Mills, 2003; see also Rowell's article in this issue, "Action Research and School Counseling: Closing the Gap Between Research and Practice"). Action-based research consists of a series of integrated, cyclical steps that include identifying an area of concern, collecting data, analyzing the data, and developing a plan (Creswell). Several examples of action-based research follow.

An action-based research study using pretest and posttest assessments can be conducted. The school counselor can have students complete a pretest before participating in an activity and a posttest when the activity is completed. For example, if the counselor is conducting a study skills group, at the first meeting all student participants would take a brief questionnaire to provide information about their study habits. Then, at the group conclusion, a posttest would be given to assess the effectiveness of the group in raising participants' awareness and/or behavior regarding study skills. The pretest-posttest assessments would be analyzed and compared for determining areas that need to be strengthened.

A single-participant (or single-subject) methodology is another method for signifying the value of counseling (Loesch & Ritchie, 2005; see also McDougall & Smith's article in this issue, "Recent Innovations in Small-N Designs for Research and Practice in Professional School Counseling"). For example, the counselor could determine the effectiveness of a theoretical approach with two students with a similar concern such as anger management. Baseline data would be collected by tracking incidents of anger during a predetermined period of time through defined, targeted behaviors associated with anger. With one student the counselor may decide to spend six sessions, for 1 hour per week, using Glasser's Choice Theory. The counselor may decide to provide the other student with counseling using an Adlerian Individual Psychology approach for six sessions, for 1 hour per week. At the end of the 6 weeks, anger incidents for each would again be tracked in the same manner as the baseline data were obtained. The school counselor could use these data to assess which approach worked better. The counselor probably would find that both theoretical counseling approaches helped the students deal with their anger issues.

Retrospective assessment is another approach for assessing students. This methodology attempts to re-create the past through the recollection of prior memories. This method is used when the results of a pretest are not available. Students rate each item based on how they felt prior to individual counseling or a guidance lesson, and then at the end of the counseling experience. Students use an "X" for beginning feelings and an "O" for the end response. If there is a positive change, it is designated with a "+" next to the item. If there is a negative trend, the counselor puts a dash next to the item. For example, suppose a counselor wanted to work with students who were having a difficulty transitioning from one grade to another. Using a solution-focused approach with these individuals, the counselor could ask the students to rate how they felt when they transitioned to a new grade the previous year, and how they felt about the school at the end of the year. Hopefully, the ratings would show students that they had similar feelings the previous year and their feelings changed, for the most part, in a positive direction (see Figure 1).

A true experimental design is when the school counselor randomly assigns students to either a treatment group or a comparison group (control). Random assignment means that each person has an equal chance of being assigned to either of the groups (Creswell, 2002). Experimental group members receive a specific intervention, and control group members do not participate in the intervention. For example, the counselor may want to find out if there is a relationship between grade point average (GPA) and self-esteem. The counselor may set up a study to determine whether counseling and tutoring, or tutoring alone, would make an impact on students' self-esteem. The counselor could randomly assign similar students to one of two experimental groups, or the control group. One experimental group of participants is to receive counseling and tutoring, the other experimental group is to receive only tutoring, and another group of students, composing the control group, receives neither intervention. All the group participants would take a pretest and posttest, and based on the results the counselor may determine that GPA increased in both experimental groups, but self-esteem increased only in the counseling-tutoring group, while neither GPA nor self-esteem increased in the control group. Based on these results the counselor may include counseling as well as tutoring in future groups.

What Types of Questions Would Provide the Best Results?

The framing of a question needs to take into consideration the needs and developmental level of the intended participants, the goal and objectives of the program (Myford & Cline, 2001), and a question format that provides the most accurate responses. Each question needs to be given the same serious consideration. Swenson (as cited in Thoms & Kellerman 1995) listed specific guidelines for question formation:

* Avoid ambiguous words. Specific words are more readily understood than ambiguous terms. (If possible, provide an explanation to make the meaning clearer.)

* Avoid negatively worded questions. Questions with double negatives are confusing. (For example, "Are you opposed to not having drug education in the curriculum?")

* Provide alternatives. If providing multiple responses to a question, giving the option of "other" offers information the researcher may not have considered. (For instance, "Which of the following do you feel should be included in the school counseling program? Drug use--, self-esteem--, study skills--, mediation skills--, other--.")

* Keep the questionnaire brief. Long questionnaires are frustrating and result in incomplete or inaccurate responses, or no response if they are mailed or are online. Write as few questions as is needed to get the information that is desired (Frary, 1996).

* Ask one question at a time. Sometimes a question is confusing when two items are asked. (For instance, "Do you believe the counselor-student ratio is adequate? What do you think an ideal ratio should be?")

* Underline words or make words bold for emphasis. If you have key words that need to be brought to the attention of the participants, underline them. (For instance, "Which of the following activities do you think a school counselor should not perform?")

* Provide a point of reference. Sometimes questions are asked and the participants are confused as to what is being compared. (For example, instead of "Do you believe parents have a better understanding of the school counseling program?", a better question is "Do you believe that parents have a better understanding of the school counseling program compared to 5 years ago?")

* Ask questions that provide more information. (For example, instead of asking the question "Do you feel students would benefit from a workshop on social skills?--yes,--no," a better question would be to ask participants to list the workshop topics that are of most interest to students.)

What Procedures Would Provide a Reliable Instrument?

Even though counselors may format a question a particular way, it is essential to receive feedback on the questions from a representative group of respondents (Frary, 1996). These individuals pilot-test the questionnaire and help the test developer to iron out any content errors and/or technical mistakes, as well as to offer feedback. This evaluative process increases the chances of having a high-quality instrument (Luoma & Tarnanen, 2003). Participants in the pilot study should have similar backgrounds to those for whom the instrument is designed, and they can be asked to provide suggestions for revision. In some cases, instruments that have already been developed and have a recognized level of validity and reliability can be analyzed, and questions can be revised to fit the needs and developmental level of the participants (U.S. Department of Health and Human Services, 2002).

Pilot-testing the instrument for gender/ethnic/ cultural groups provides reliability (i.e., the consistency of the respondents' responses) checks, and with adaptations the instrument could be appropriate for use with diverse groups. Even with established tests, reliability may need to be somewhat sacrificed in favor of local cultural adaptations. For instance, one government agency was concerned about the spread of HIV through contaminated needles in a Hispanic section of an urban area. A story about Kagan the Dragon was developed to educate the neighborhood children to avoid needles that they might find in their neighborhoods. It was not until much money and time had been put into the project that researchers discovered that one Spanish interpretation of the name Kagan is "pooper." Cultural integrity links the instrument to more reliable outcome measures (U.S. Department of Health and Human Services, 2002).

How Is the Information to Be Analyzed and Presented to Constituents?

After the data are analyzed, they should be presented in a manner that is clear and easy for all stakeholders (e.g., parents, teachers, administrators) to understand. Presenting the analysis in the form of a chart or graph will assist the school counselor in presenting the data in a format that distinctly shows the results and benefits of the programs or interventions. For example, after conducting a single-participant methodology, the counselor may want to present a bar graph to represent the students' changes and improvements from week to week. In addition, a line graph may be helpful when demonstrating the effectiveness of an activity through pretest-posttest assessment. The charts from Figure 1 are examples of results that present a visible picture of the benefits of counseling. These results can provide credence to grant writing while keeping track of excellence (Mecca et al., 2000). An example of how a middle school counselor applied these strategies for her work with middle school students is below, and a checklist of the steps she took to develop the assessment and the program is found in Appendix A.

PUTTING THEORY TO PRACTICE

A middle school counselor was given the task of developing a leadership group composed of 22 eighth-grade students. Because the counselor was not sure what topics were important for leadership training with this group of 13- and 14-year-olds, she decided to investigate similar groups in other schools through a review of the literature. This search provided her with several concepts that she wanted to include in her group: goal-setting, communication, public speaking, assertiveness, confidence in various areas, empathy, and understanding diversity. Based on her research and talking with other colleagues, she located an assertiveness test to explore this concept, but she developed a pretest-posttest measure to help her evaluate the effectiveness of the other concepts that would be presented in her group. After she developed this instrument, she asked several eighth graders to serve as the pilot group for the purpose of test revision.

After revising the test based on initial feedback, the counselor defined each of the topics and designed a Likert scale (i.e., potential answers for a question are arranged in a continuum from, say, 1 for lowest to 5 for highest). The group participants rated the extent to which each concept described them from 1 ("not at all") to 5 ("very much"). In addition, students were asked to identify a goal they wanted to accomplish while participating in the group, and strategies that would be used to reach this goal. The group met for 8 weeks, for 1 hour twice a week.

The instrument was administered during the initial meeting of the group, and again at the end of the group meetings. The data collected revealed a positive change in some areas, no change in others, and a slight downward trend in other areas. Rather than being discouraged that not all the trends showed an improvement, the counselor took this information as an opportunity to reflect on the reasons that some areas did not show the anticipated improvement, and to revise the materials or activities. A graph was easily designed in Microsoft Excel to share the results of the group with stakeholders (see Figure 2).

[FIGURE 2 OMITTED]

CONCLUSION

Professional school counselors have a responsibility to show how their programs, activities, or interventions make a difference in the lives of pre-K to Grade 12 youth. Enumerative data are not enough; producing data that concretely reveals effectiveness in these activities is needed. Many professional school counselors complain that available instruments either are too expensive or do not adequately reflect the purpose of the program or activity. In these cases, assessment inventories need to be developed so that data can he collected to concretely show student activities and programs. There are several action-based research methodologies that counselors use in conjunction with assessment instruments to produce tangible results. School counselors can present results in easy-to-read and understandable styles so that all program constituents can be informed as to the effectiveness of counseling in the schools.

APPENDIX A

Checklist for Developing an Assessment Instrument

This checklist can be used to assist in the assessment development process. Check each step as it is completed, and provide the necessary information.

1. -- Why is an assessment needed? (check all that apply)

a. program effectiveness b. strategy or intervention effectiveness c. personnel performance d. other

2. -- Who is going to be assessed?

a. students b. teachers c. parents d. administration e. other

3. -- Conduct a review of the literature.

a. Internet b. journals c. colleagues d. books C. other

4. -- Search resources for an instrument.

a. Internet b. colleagues c. Tests in Print d. Buros Center e. other

5. -- Determine strategy for data collection.

a. pretest-posttest b. single-participant c. experimental design d. retrospective design c. other

6. -- Do the questions address your presenting problem and population?

a. multiple-choice b. fill in the blanks c. Likert scale d. true/false e. other

7. -- Conduct a pilot study.

a. Are subjects of a similar background? b. Are cultural issues addressed? c. Are questions developmentally appropriate?

8. -- Revise the instrument according to feedback from the pilot study.

9. -- Where will the assessment occur?

a. classroom b. counselor's office c. cafeteria d. home e. other

10. -- When will the assessment occur?

a. during a school day in the fall b. during a school day in the winter c. during a school day in the spring d. after school e. at home after school f. other

11. -- Who are the results for?

a. parents b. administration c. students d. colleagues e. other

12. -- How will the results be presented?

a. graph b. chart c. written communication d. other

13. -- Are the results in an easily understood format?

a. geared toward the audience b. in acceptable language c. clearly marked d. other

References

Baker, S. (2000). School counseling for the twenty-first century (3rd ed.). Upper Saddle River, NJ: Merrill Prentice-Hall.

Creswell, J. W. (2002). Educational research: Planning, conducting, and evaluating quantitative and qualitative research. Upper Saddle River, NJ: Pearson.

Foster, L. H., Watson, T. S., Meeks, C., & Young, J. S. (2002). Single-subject research design for school counselors: Becoming an applied researcher [Electronic version]. Professional School Counseling, 6, 146-154.

Frary, R. B. (1996). Hints for designing effective questionnaires. Practical Assessment, Research & Evaluation, 5, Article 3. Retrieved September 3, 2005, from http://www.pareonline.net/getvn.asp?v=5&n=3

Frith, G. H., & Clark, R. (1982). Evaluating elementary counseling programs: 10 common myths of practitioners. Elementary School Guidance & Counseling, 17, 49-51.

Gibson, R. I., & Mitchell, M. H. (1999). Introduction to counseling and guidance (5th ed.). Upper Saddle River, NJ: Merrill Prentice-Hall.

Gladding, S.T. (2000). Counseling: A comprehensive profession (4th ed.). Upper Saddle River, NJ: Merrill Prentice-Hall.

Gysbers, N. C. (2004). Comprehensive guidance and counseling programs: The evolution of accountability. Professional School Counseling, 8, 1-15.

Hood, A. B., & Johnson, R. W. (2002). Assessment in counseling (3rd ed.). Alexandria, VA: American Counseling Association.

Hughes, D. K., & James, S. H. (2001). Using accountability data to protect a school counseling program: One counselor's experience. Professional School Counseling, 4, 306-309.

Jones, W. P., & Kottler, J. A. (2006). Understanding research: Becoming a competent and critical consumer. Upper Saddle River, NJ: Pearson.

Loesch, L. C., & Ritchie, M. H. (2005). The accountable school counselor. Austin,TX: Pro-Ed.

Luoma, S., & Tarnanen, M. (2003). Creating a self-rating instrument for second language writing: From idea to implementation. Language Testing, 20, 440-465.

Mecca, W. F., Rivera, A., & Esposito, A. J. (2000). Instituting an outcomes assessment effort: Lessons from the field. Families in Society, 81, 85-89.

Melchert, T. P., & Sayger, T.V. (1998).The development of an instrument for measuring memories of family of origin characteristics. Educational and Psychological Measurement, 58, 99-118.

Mills, G. (2003). Action research: A guide for the teacher researcher (2nd ed.). Upper Saddle River, NJ: Merrill.

Myford, C. M., & Cline, F. A. (2001). Designing assessment instruments to measure the impact of participation in short-term educational programs. Studies in Educational Evaluation, 27, 107-131.

Myrick, R. D. (1990). Retrospective measurement: An accountability tool. Elementary School Guidance & Counseling, 25, 21-29.

Thoms, K. J., & Kellerman, D. K. (1995). Ask a silly question, get a silly answer: Designing effective questionnaires for needs assessment instruments. Performance Improvement, 34, 4-6.

U.S. Department of Health and Human Services. (2002). Achieving outcomes: A practitioner's guide to effective prevention (Conference Edition). Washington, DC: Author.

Jeannine R. Studer, Ed.D., is an associate professor in the Department of Educational Psychology & Counseling, University of Tennessee, Knoxville. E-mail: [e-mail]

Aaron H. Oberman, Ph.D., is an assistant professor in the Department of Counseling & Educational Psychology, University of West Georgia, Carrollton.

Reagan H. Womack, M.S., is a professional school counselor at North Middle School, Lenoir City, TN.

Figure 1. Example of a retrospective assessment.
On the first chart, the student places an "X" on
the number that best describes how he or she
remembers feeling at the start of the previous year.
On the second chart, an "O" indicates how he or
she remembers feeling at the end of the previous
year. On the third chart, the changes are indicated
with a plus sign for a positive change, a negative
sign for a negative change, or no change.

Feelings at Start of Previous School Year

Frustrated        X
Worried                X
Anxious                     X
Nervous                X

             1    2    3    4    5

Feelings at End of Previous School Year

Frustrated   O
Worried                O
Anxious           O
Nervous                     O
             1    2    3    4    5

Key: 1 = "I hardly ever have this feeling."

5 = "I have this feeling almost all of the time."

Trend

Frustrated     +
Worried        No change
Anxious        +
Nervous        -