Using comparison groups in school counseling research: a primer.

This article describes comparison group research designs and discusses how such designs can be used in school counseling research to demonstrate the effectiveness of school counselors and school counseling interventions. The article includes a review of internal and external validity constructs as they relate to this approach to research. Examples of relevant research using this design are presented.

*********

The lack of a sound research base in the field of school counseling has been lamented for many years (Allen, 1992; Bauman, 2004; Cramer, Herr, Morris, & Frantz, 1970; Lee & Workman, 1992; Loesch, 1988; Whiston & Sexton, 1998; Wilson, 1985). The recent emphasis on research in No Child Left Behind legislation (2002) and the ASCA National Model[R] (American School Counselor Association, 2005) has moved the need for rigorous empirical research to the forefront. The ASCA National Model stresses that school counseling programs include learning objectives that are based on measurable student outcomes and that are data-driven and accountable for student outcomes. The focus on data and measurement makes clear that school counselors can no longer avoid conducting research and using empirical research to make decisions.

The nature and goals of such research are the subject of a recent debate. Brown and Trusty (2005) have contended that research should focus on demonstrating that well-designed and appropriate interventions used by school counselors are effective, and they further argued that research investigating whether comprehensive school counseling programs increase student academic achievement is not productive given the presence of numerous confounding influences. Sink (2005) disagreed, noting that school counselors are expected to contribute to the total educational effort to raise academic achievement. He advised that research to examine how school counselors influence achievement can be conducted using carefully selected methodologies, and while not definitively establishing causality, such research can provide strong evidence of the impact of comprehensive school counseling programs on student achievement.

In their review of school counseling outcome research from 1988 to 1995, Whiston and Sexton (1998) found that of the 50 published studies they located, most provided only descriptive data, used convenience samples, lacked control or comparison groups, used outcome measures of questionable reliability and validity, and did not monitor adherence to intervention protocol. Such studies do little to add to the knowledge base of the profession, and they do not meet established standards for scientific rigor.

In an era of limited resources for education and "accountability" becoming a watchword, counselors must demonstrate how they contribute to the academic success of students. Heartfelt letters of appreciation and positive comments by constituents, while sincere, will not convince stakeholders and holders of purse strings of the value of the profession. School counselors, occupied by providing services in schools, often neglect to demonstrate their importance until their positions are considered for reduction. This reactive approach is less likely to sway opinion than ongoing proactive efforts to use research effectively. Collecting, analyzing, and discriminating data that provide evidence of counselors' effectiveness are consistent with the professional goals and models that define the profession.

Under No Child Left Behind, school counselors (along with other education professionals) are called upon to demonstrate their effectiveness using quantitative data such as evidence of academic achievement, attendance and graduation rates, and measures of school safety (McGannon, Carey, & Dimmitt, 2005). No Child Left Behind and the ASCA National Model both emphasize the importance of scientific, rigorous, well-designed research as an essential component of modern school counseling programs. These guidelines indicate that conducting research is no longer a peripheral activity that a few counselors might attempt but is a central part of the role of all school counselors. The ASCA National Model says the following about data:

   Data analysis: Counselors analyze student
   achievement and counseling-program-related
   data to evaluate the counseling program, conduct
   research on activity outcomes and discover
   gaps that exist between different groups
   of students that need to be addressed. Data
   analysis also aids in the continued development
   and updating of the school counseling
   program and resources. School counselors
   share data and their interpretation with staff
   and administration to ensure each student has
   the opportunity to receive an optimal education.
   (ASCA, 2005, p. 44)

Although most school counselors have had a graduate course in research methods (and perhaps statistics), these introductory courses typically are designed to prepare students to be critical readers of research. Even among those few students who conduct research in their graduate training programs, it is the rare school counselor who continues to do research once he or she is a practicing school counselor. Responsibility for the absence of research from school counselors' job description lies not only with the counselors, but also with the administrators and district officials who do not require or value research. No Child Left Behind has raised the awareness of educators in all fields that accountability is expected; data are the foundation for educational decisions, including decisions about counselors. In this climate, there is greater support (some might say pressure) for research.

There are a number of different kinds of research, and a description of all relevant types is beyond the scope of this article. The purpose of this article is to provide a rationale for using control and comparison group designs in school counseling research. I begin by defining some basic terminology and reviewing the concept of validity, which is fundamental to all research. I then briefly discuss single-group pre-post research designs, which often are used in schools because they are relatively easy to conduct. The main focus of the article is comparison group designs, and these will be described in more detail. Finally, I provide a discussion of relevant research using comparison group designs as examples of this research strategy.

DEFINITIONS

Several technical terms are used in this discussion of research, and it is important that the reader be clear about their meaning. Researchers study variables that can assume different values. The independent variable is the intervention variable, or the variable manipulated by the researcher. The dependent variable is the outcome variable, the effect. In a study of the effect of participation in extracurricular activities on graduation rates, the independent variable is the participation (which could be defined as the number of activities, or the number of hours per week of involvement, or a yes/no category) and the graduation rate is the dependent variable. Researchers also may refer to moderator variables, which are variables that influence the relationship between the independent and dependent variables. Parental education might moderate the relationship between extracurricular participation and graduation rates, and it then would be a moderator variable. Statistical significance means that the obtained results are unlikely to have occurred by chance. If results are statistically significant at p < .05, the results would be obtained by chance in less than 5 out of every 100 cases. The counselor/researcher should keep in mind that with large samples, results might be statistically significant but not practically significant.

Let us imagine that a new program for elementary math skills were implemented in several schools in a large district. At the end of a school year, the difference between the achievement scores of those who used the program and those who continued the usual math program was statistically significant. One might conclude that the new approach is better. But what if the difference in scores were only .10 (grade equivalent)? Depending on the cost of the program, one might conclude that although the difference is statistically significant (p < .05), in practice the difference or gain is not substantial enough to justify a large expenditure on the new program. There are ways to describe the practical significance of the findings through the use of effect sizes; these are discussed in Sink and Stroh's article in this issue ("Practical Significance: The Use of Effect Sizes in School Counseling Research").

VALIDITY

Regardless of the design or method of research, school counselors must be concerned with the validity of the research they conduct or read. In general, validity refers to the degree of confidence we can have in the findings of a research study. If a study does not demonstrate adequate validity, the results are of questionable application and should not be the basis for decisions. Internal validity refers to whether the observed change in the dependent (outcome) variable is due to the independent variable and only the independent variable. For example, if we are interested in whether student attendance (our dependent variable) improved for high school freshmen when a new orientation program was conducted by the school counselors (our independent variable; the orientation program), we want to be sure that no other variables could explain the obtained results. If, in addition to the new orientation program for freshmen, the school employed additional truant officers, we could not be sure that the change in attendance was due only to the new orientation program and not to the truant officers' activities. The internal validity of the study would be compromised.

External validity refers to the degree that the results of one study can generalize to (apply to) other people in other places or times. School counselors reading the results of research in a journal want to know whether they can reasonably expect that the reported results would apply in their own setting with their own students. Researchers hope that their results will be useful to others in other locations and times. If the new orientation program improved attendance for students in one school or district, the issue of external validity asks the question of whether other schools are likely to achieve the same results with the same program. The threats to external validity are related to the population from which the sample was selected (was it representative, did it include members of all groups of interest?) and the context in which the study was conducted (was it in a laboratory or a school, did participants know they were involved in an experiment, did the researcher convey the hoped-for outcomes?). These two types of external validity often are referred to as population validity and ecological validity. A study conducted at a private school with European American upper-class students is of questionable validity for an inner-city school with a large percentage of minority students.

Another factor in external validity is the nature of the research itself. If the students in the experimental group were aware they were receiving a special program different from that of the control group, their efforts may have been changed by that knowledge. In addition, the researcher must incorporate a way to ensure that interventions delivered in a naturalistic school setting are faithful to the protocol of the experiment. If the intervention is a series of lessons, for example, the researcher must be sure that the lessons are delivered as described in the manual. If each teacher or counselor makes changes in the program, external validity is compromised by the absence of treatment (intervention) fidelity. Researchers can increase the external validity of their work by attending carefully to sample selection and to the conduct of the experiment.

Campbell and Stanley (1963) described important threats to internal validity. These are conditions that provide possible alternative explanations for obtained results, or ways that events or conditions other than the independent variable may explain observed changes in the dependent variable. The following is a brief review of those threats.

History

In this context, history refers to any event not planned or part of the research that occurs during the research. In the example above regarding the new orientation program, let's imagine that the principal decides to visit each freshman English class during the first week of school. Although not part of the research, this event (history) might be an alternative explanation for the difference in attendance rates. History is the greatest threat to internal validity when it affects only one group of research participants. If your research design used a comparison group (last year's freshmen) who had not experienced the historical event, your internal validity would be reduced. However, if you were studying whether the attendance of males vs. females increased when the new orientation program was implemented, and both males and females experienced the visits by the principal, internal validity would not be affected. When research is being conducted in schools, there are often events that occur outside the counselors' control, and the counselor must be alert to these competing explainers of results.

Maturation

Human beings change and develop over time. This means that some changes will occur independently of any intervention. For example, a middle school counselor might provide a series of guidance lessons on conflict resolution to seventh graders. If the counselor were to measure student attitude toward fighting, or the number of fights before and after the lessons, results might show a decrease in conflict after the lessons. However, maturation might be an alternative explanation for the results; students may be exhibiting less physical conflict because they are developing cognitively and socially, not because of the lessons. In the discussion of comparison group designs later in this article, I suggest designs that minimize the influence of this threat to internal validity.

Testing

Researchers may want to give participants a pretest to determine the base rate of whatever behavior or attitude is of interest. If a counselor were going to do a series of guidance activities to reduce racial/ethnic stereotyping in a school, he or she may want to get a measure of the degree of stereotyping that students do at the start of the project. However, the pretest may sensitize participants to the issue of stereotyping, and that may influence their scores on the posttest. This is called the testing effect.

Instrumentation

A counselor who is leading an anti-bullying program at her school wants to measure the effect of the program on student bullying behavior. She knows that much bullying occurs on the playground, and she uses a behavioral observation method to determine the frequency of playground bullying before the program begins, and after the program has been in place for a semester. The behavioral observation method requires several observers, and it may be that some observers are more alert than others. Or, the observers may become more adept with practice. If the observers are not the same at both measurement points, instrumentation is a threat to internal validity. The changes may not be a result of the children's behavior, but of the observers' skill.

Regression to the Mean

When a counselor is interested in extreme groups (students high or low in a particular characteristic), a pretest-posttest design is vulnerable to this threat. We know that on subsequent testing, both high and low scores tend to become closer to the mean (average score). So observed changes may be due to this tendency rather than any real change in the characteristic being measured.

Selection

In some research, counselors are studying more than one group of students (e.g., classes, genders). If you are trying a new program with one class and using another class as a comparison group, the groups might be different on some other factors (e.g., reading level, intelligence) that can affect the results.

Mortality

This threat to internal validity refers to loss of participants during the course of the study. In a comparison group design, this becomes a problem when mortality is greater in one group than in another. For example, in a study where the comparison group is another school, there might be asbestos discovered in one of the schools and many students transfer out of that school. That group would have greater mortality than the other group.

Selection Interaction

It is possible that one of the other threats to internal validity combines with selection. This means that one of the comparison groups is affected by those threats (e.g., history, maturation) differently than other groups in the study. For example, an elementary school counselor implements a new program to teach empathy skills to fifth-grade classes. Lessons are given throughout the school year, and a nearby school serves as a control group. On an outcome measure, the counselor finds that at the end of the year, girls show more improvement in empathy than boys do. It might be that those findings are because girls of this age tend to develop these skills naturally at this age, while boys develop them later. The findings may reflect a selection (grade level) maturation (girls faster than boys) effect.

School counselors doing research must be alert to potential threats to internal validity. While it is impossible to avoid all such threats, especially when doing the research with students in schools (rather than in a laboratory), if results are to be meaningful, the researcher must acknowledge them. In some cases, there are statistical methods to control for the influence of these threats.

One of the advantages to publishing the results of research is that school counselors do not have to reinvent the wheel. That is, we read about research in the hope that findings will generalize to other students, settings, and times. Generally, results should be replicated in other contexts so that it is not just a single study but a body of research that establishes the generalizibility of findings. Let us assume that the original study of the orientation program was conducted in a large, urban high school. Will the same program have the same outcome in a small, rural high school and with a different racial/ethnic composition?

SINGLE-GROUP PRETEST-POSTTEST DESIGN

School counselors have the advantage of conducting research in settings that reflect real students. Some programs of interest to counselors cannot be effectively studied in a laboratory; if they could be, we would question the external validity, of the findings. Conducting research in a school also has disadvantages, not the least of which is the inability to control many factors in a research process. For example, researchers may not be able to randomly assign students to classrooms, and they may have to contend with numerous historical events that occur during a research study. Nevertheless, the findings are clearly relevant and applicable to the school of interest.

One research strategy that is relatively uncomplicated to do is a single-group pretest-posttest design. In this design, the school counselor implements a program (a series of guidance lessons or counseling groups to address a particular topic). Prior to starting the program, the students take a pretest so their baseline levels can be determined. The program is delivered, and then the students take a posttest. The improvement in scores from pretest to posttest is used to measure the impact of the program. At first glance, that seems to be a logical approach. One advantage of the pretest-posttest design is that one does not have to include a control group, and the pretest information allows the counselor to determine differential effects (e.g., the lessons increased tolerance in boys more than in girls).

However, this design is particularly vulnerable to the threats to internal validity described above. How can the counselor demonstrate that it was the program that caused the change in scores, and not maturation, history, testing, or regression to the mean? For example, let us imagine that the lessons were developed to increase tolerance toward physically handicapped students. During the time the weekly lessons were being presented, there was a television special on the topic that many students watched. Or perhaps there were classroom disruptions during the time the lessons were presented. Were the observed changes the result of the TV program, the disruptions, or the lessons?

COMPARISON OR CONTROL GROUP DESIGN

A more rigorous design that avoids many of the threats to internal validity inherent in pretest-posttest designs is the control group or comparison group design. A control group is a group of participants who get no intervention; if a group gets a different intervention, then we call it a comparison group. History and maturation will affect both the experimental and the comparison groups, so any differences in the outcome variable cannot be biased by those threats. Testing and regression to the mean also are going to influence both groups, so observed differences can be attributed to the intervention rather than these alternative explanations. Of the 50 school counseling outcome studies published between 1988 and 1995, only 26% used this design (Whiston & Sexton, 1998). The authors of the review concluded that more research of this kind is needed, and they recommended the wait-list control group strategy used often in other counseling research. In the school setting, this means that classes, schools, or students who do not receive the intervention (program, activity) during the research period will receive it at a later time (the following semester, year, etc.).

There are several ways in which comparison groups can be created. The first is random assignment. That means that all eligible participants are randomly assigned to one of the experimental conditions (intervention, comparison group, control group). When this is not possible, the researcher can use preexisting groups (e.g., already formed or intact classes) that are matched on key variables, such as reading level or socioeconomic status. An investigation of the impact of a new "transition to kindergarten program" might use current students in kindergarten as the experimental group and students from a previous year (now first graders) at the same school as the comparison group. The assumption in this case is that previous students resemble current students on the relevant characteristics.

A final method would be to use pretest scores to ensure that the groups are matched on key variables prior to the introduction of the intervention. After creating matched groups, the researcher then can randomly assign the groups to the intervention conditions. If the intervention might have a differential effect based on levels of test anxiety, the researcher can administer a pretest of test anxiety; create groups of high-, average-, and low-anxiety students; and create two groups with equal representation from the different levels of anxiety. Once the groups are created, a random procedure can be used to assign one group to receive the intervention (e.g., instruction in progressive relaxation) and the other group to serve as the control group.

Random Assignment

The most rigorous comparison group design utilizes random assignment to condition (experimental group or control/comparison group). Random assignment means that every participant has an equal chance of being in the experimental condition. Most statistical software packages include features that allow the researcher to randomize assignment in a scientific manner. There are also sites on the Internet that a counselor might be able to locate and use if such software is not readily available. In most educational settings, it is usually not possible to randomly assign students to one or another class or program. However, random assignment can be accomplished by using classes or schools as sampling units. For example, in a study evaluating the effects of a new drug prevention curriculum for middle school students, if there is more than one school interested in participating, the schools can be randomly divided into two groups (using a number of randomization procedures), with one group designated as the experimental group (the schools receiving the curriculum) and the other as the comparison or control group (which will not receive the curriculum at this time). If only one school is going to participate, the same procedure can be applied to classrooms.

In some cases, it may not be possible to randomly assign classrooms to the intervention or nonintervention groups. If there are two schools or two classrooms that are potential participants, and only one is interested in testing the curriculum, the other can become the comparison group. The problem with this method is that the two groups (schools, classes) may be different prior to the curriculum implementation in ways that affect the outcome (e.g., intelligence, reading level). If, however, the researcher is able to administer the pretests and posttests to both groups (or obtain data on both groups), these differences can be identified and controlled for statistically. That means that the analyses can determine whether the obtained differences would exist over and above the influence of these potentially influential variables. If reading level were a possible confounding variable, the researcher can use a statistical analysis called analysis of covariance, in which reading level is designated the covariate. The results of this analysis will reveal whether observed differences in the dependent variable (the outcome) are significant when differences in reading level have been taken into account. The researcher can include more than one covariate if several attributes are potential competing explainers of results.

Measurement Concerns

How are variables measured? This is a basic question that researchers must address when designing the study. For the results to be valid, the measures must be reliable and valid. Reliability refers to the consistency of the scores, and validity relates to whether the measure measures what it purports to measure and for whom it does so. Researchers need to use considerable care in the selection of the instruments to be used. While researcher-designed questionnaires may be used, it is essential to establish the reliability and validity of such measures. Published measures generally will report such data so that the researcher can make informed decisions. If other methods of assessment are used (such as observation), those also must be evaluated prior to use, and they must meet the same standards of psychometric adequacy as paper-and-pencil measures. Some studies in the school counseling field have used self-reported student grades as outcome variables. A more precise measure would be to use actual recorded grades from student records. To take this a step further, grades may be influenced by the grading practices and standards of different teachers; achievement test scores might be a more valid measure to use as a dependent variable.

Data Analysis

The word statistics invokes fear and anxiety in many for whom research is not a frequent activity. Counselors need to know that in the age of computers, the task of analyzing data is far less daunting. Even without the specialized statistical programs used by most researchers, school counselors can utilize statistical features of Microsoft Excel and EZAnalyze (Poynton, 2005), an add-in for Excel. Using these tools, the school counselor can easily obtain descriptive data about the sample (including means and standard deviations) and can disaggregate data by group (e.g., by gender or ethnicity). In addition, the school counselor can perform several analyses, including correlations (to assess the strength of relationships between two variables such as math and reading test scores), t tests (to test the significance of differences between two groups or pretests and posttests for the same group), and analyses of variance (to test differences among more than two groups). The counselor also can obtain tables and graphs directly from the program, allowing for visual presentation of results.

For more complex analyses, many school districts have research departments that can help. And many counselor educators at universities are eager to assist, and can do so even when located at a distance, using e-mail to receive data. Consulting with university researchers is a good idea throughout a research project when school counselors are novice researchers, but it can be especially important during the data analysis and interpretation step. A very useful review of the various analysis options for comparison group designs can be found in Gliner, Morgan, and Harmon (2003).

Reporting Results

Whether the purpose of the research is for program improvement or to comply with mandatory reporting regulations, it is important to present the results clearly and accurately. Much of the audience will be unfamiliar with the terms used, so definitions are essential. It makes sense to begin by stating the research question at the outset, and then describing how you went about answering that question. For example, if the question was, "What is the effect of the new study skills group on student achievement?" you would begin by describing the study skills group and defining student achievement (e.g., overall grade point average, scores on achievement tests). Then the research design is described. For example, researchers often will write something like, "A pretest-posttest comparison group design was used, with last year's students serving as the comparison group [or another school with comparable demographic variables]." The next step is to present results clearly, using tables and graphs when they enhance the presentation. Finally, you want to provide the answer to your question, and discuss the implications of your findings. Any limitations of your research should be acknowledged.

School counselors need to disseminate their research to advance the profession. Presentations to stakeholders and administration are one method of doing so. A brief summary report of findings to the administration or school board, or a more detailed presentation, can educate these important groups about the interventions that counselors are using and will demonstrate to these stakeholders that programs have been scientifically evaluated for effectiveness. Writing articles for state and national journals in which you present a report of your research is an important contribution to the field. Knowledge builds on previous knowledge, and if research is not published, others will not have the benefit of the findings.

SOME EXAMPLES

To illustrate the control/comparison group approach to research, several studies have been selected for review. These will demonstrate the advantages and pitfalls of this strategy. In the interest of space, the details of statistical analyses will not be given below. Interested readers may refer to the original journal articles for that information.

Random Assignment--RIPP

A school-based violence prevention program, Responding in Peaceful and Positive Ways (RIPP), was studied in three public middle schools in a Southern city (see Farrell, Meyer, & White, 2001). All regular education sixth-grade classes were potential participants. Thirteen classes, including 305 students, were randomly assigned to the experimental group (receiving the program), and 14 classes (321 students) to the control group. The schools also implemented a school-wide peer mediation program, which was available to all students in the schools. The vast majority of participants were African American.

The RIPP program consisted of 25 sessions (50 minutes each) taught by trained prevention specialists who were African American men. The lessons were presented weekly during social studies or health education classes over one school year. A manual followed by the presenters was used to increase treatment fidelity; researchers observed the implementation and provided an additional check of treatment fidelity.

In addition to pretest data obtained in October and posttest data collected in May of that school year, follow-up data were gathered 6 and 12 months after the completion of the program. These measures were administered by research assistants who were "blind" to (did not know) the condition (intervention or control group) of the classes. Of the 626 students who began the study, complete pretest and posttest data were obtained from 474. Four hundred ten students were available at the 6-month follow-up, and 359 at the 12-month follow-up.

A variety of measures was used to assess the variables of interest: frequency scales for problem behaviors, violent behaviors, and drug use; a RIPP multiple-choice knowledge test, a problem situation inventory, and two scales assessing relevant attitudes (beliefs supporting aggression, attitudes toward conflict). Reliability and validity data are given for all measures, and all were acceptable. Demographic data also are available, include disciplinary code violations.

Analyses included a comparison of experimental and control groups on demographic variables of gender, age, and ethnicity. No differences were detected between the groups. Because some students did not complete the entire program, the researchers examined the effect of attrition on the two groups. Their analyses determined that attrition affected both the experimental and control groups in a similar way. Analyses also investigated differences on the pretests between experimental and control groups. There were no differences between the two groups on disciplinary referrals, although differences were detected by age and gender (the older the student, the higher the rate of disciplinary referrals; boys also were more likely than girls to have violations). No differences were found on specific violent behaviors. In fact, the only difference found at pretest was the higher incidence of positive attitudes toward nonviolence in the control group.

The researchers then analyzed the differences at posttest. Although many of the results were in the predicted direction, statistically significant differences between the intervention group (which received the RIPP curriculum) and the control group were found on the number of disciplinary violations for violent behavior (the control group had 2.2 times as many) and the rates of in-school suspensions (the control group had 5 times as many). The intervention participants used peer mediation more frequently at posttest than did the control participants and also reported fewer fight-related injuries at posttest. Differences at follow-up time periods (6 and 12 months post-intervention) were in the expected direction but were not significant, with the exception of the rate of in-school suspensions at 12-month follow-up, which was three times greater for boys only.

Further analysis revealed an important finding: The participants who reported high levels of violence on the pretest had lower scores at 6- and 12-month follow-up than did comparable participants in the control group. Results also indicated that while students in the intervention group demonstrated increased knowledge of the material in the curriculum, they did not show any change in their attitudes or on the use of nonviolent responses in hypothetical situations. The authors speculate that because knowledge improved but attitudes and skills did not, there may not have been adequate support for the use of these skills in the school environment. This is important for counselors: If they teach skills to students, the school community must endorse the use of the skills, reinforce the skills, and encourage students to apply them in the school setting.

Although this study was one of few conducted in schools using random assignment and a large sample, the research was somewhat compromised by the nature of the design. With some classes in each school receiving the curriculum, it is difficult to ensure that the students in the control condition were unaware of the program and did not learn some of the material via modeling from peers in the intervention condition. In those cases where the results "approached significance," this limitation may have had an effect, and it in fact may have influenced other results as well.

Random Assignment--SSS

Researchers (Brigman & Campbell, 2003) tested the Student Success Skills (SSS) on a random sample of 180 students in four grades at six schools in a Southern state. The SSS program is an intervention in which school counselors teach skills found to predict school success using effective teaching strategies for classroom guidance and group counseling components. Students in the participating schools who scored between the 25th and 50th percentile on a state assessment test were eligible to participate. Thirty students from each school were randomly selected for participation. The comparison group was created by matching participating schools with nonparticipating schools based on geography, race, and socioeconomic status and then randomly selecting students. In the study, the outcome (dependent) variables were math and reading scores on a standardized test and teacher ratings of classroom behavior. The authors reported the psychometric properties of all the measures used.

In order to ensure that the counselors were providing the program correctly, counselors received three days of training and three half-day follow-up training sessions. Half-day peer coaching sessions were scheduled during months when follow-up training did not occur. The group counseling component consisted of eight weekly 45-minute sessions and four booster sessions using a structured format for teaching cognitive, social, and self-management skills. The number and frequency of classroom guidance lessons were not specified for each site, but at least three classroom guidance lessons were to be delivered at each grade level. Five of the six schools met the established criteria for implementation guidelines.

Results showed statistically significant differences between the intervention and control groups on the math and reading posttest scores. In this case, the size of the difference between the two groups is large. The behavior scale was not administered to the comparison group, but authors noted that 70% of participants demonstrated an average of 22 percentile points improvement in behavior between September and April administrations of the behavior rating scale.

The use of the comparison group was essential for these results to have an impact, as any improvement found for the intervention group could have been the result of history, maturation, or regression to the mean. Although different idiosyncratic historical influences may have affected results, the findings are strengthened by the use of the comparison group. The researchers noted that the absence of the behavior ratings for the comparison group was a limitation. In addition, it is not known how this intervention compares to other interventions used to increase student achievement. This study demonstrates how counselors can use research to demonstrate that their interventions positively affect student achievement.

To further document the impact of this intervention, the researchers have published two replications of this study (for details, see Webb, Brigman, & Campbell, 2005, and Campbell & Brigman, 2005). Replication is a way to demonstrate that a research study conducted using the same intervention, procedures, and measures yields similar results in other settings or contexts. In this case, while results were similar in the replications, all three studies' participants were predominantly European American, and it remains to be demonstrated that the intervention is successful with more diverse students.

Using Both Comparison and Control Groups

Both a comparison group and a control group were used to investigate the effects of a social cognitive group on self-esteem and difficulties in peer relationships among adolescents aged 13-16 years (Barrett, Webster, & Wallis, 1999). In order to be certain that the effects of the intervention could be attributed to the intervention and not just the attention received as a result of being in a group, the researchers used a comparison group. This group received the special attention of being in a group, but instead of the program using skill development process and procedures, the comparison group was given similar information in a didactic fashion using lecture and films. A wait-list control group also was used.

To place the 51 participants (students recommended by teacher nomination) in one of the groups, the researchers matched the students on gender, age, and scores on the three pretests (measures of self-esteem, self-related cognitions, and social competence with peers). Thus, the design ensured that the three groups were comparable prior to the interventions. Measures were described and the psychometric properties were adequate.

The researchers used change scores from pretest to posttest to assess the impact of the intervention. The experimental group showed significantly greater improvement in self esteem and self-related cognitions and perceptions than did both the comparison and control groups. Also detected was a significant increase in the self-reported interpersonal difficulties with peers in the intervention group. The researchers speculated that the increased difficulties may reflect a concern with ecological validity. That is, while the participants may be able to use the new skills within the program, they were not able to transfer those skills to their daily environment. This is a useful finding: Programs in schools need to include components to help apply the skills in the participants' natural environment.

Although this study utilized a relatively small convenience sample from one school, and it did not incorporate a follow-up component, the employment of both comparison and control groups is a useful feature given the nature of the intervention, and one that counselors might consider for other research questions.

CONCLUSION

This article has discussed how comparison and control group designs strengthen research. Well-designed research studies that examine the effectiveness of school counseling programs and practices contribute to a growing body of evidence that supports the positive impact of school counselors on academic achievement. Although there has always been a need for school counselors to be accountable, the current emphasis on data in education makes this need particularly salient. The urgency of the situation is evident from Whiston's (2002) observation:

   In my opinion, this is a critical time for leaders
   in school counseling to invest in the future of
   the profession and support school counseling
   research. School counselors may believe they
   make a difference, but without "hard data" to
   support these claims, school counselors run
   the risks of losing their positions. (p. 153)

This article assists school counselors and researchers by describing an approach that produces the most robust findings.

To ensure both internal and external validity, school counselors must attend carefully to all the elements of their study: They must choose the best measures (e.g., achievement test scores vs. grades, objective testing vs. self-report when possible) that have adequate reliability and validity. They must ensure that data are analyzed in the most effective manner (e.g., using statistical methods to control for possible pre-intervention differences in groups). They must clearly describe the population and context from which participants were selected, explain procedures for assignment to groups (with random assignment being the ideal), and articulate how the researchers ensured that the intervention was delivered correctly in the same way to all groups.

School counselors have an ethical obligation to seek evidence of their effectiveness. Principle A.9.g of the Ethical Standards for School Counselors (ASCA, 2004) maintains that a professional school counselor assess "the effectiveness of his/her program in having an impact on students' academic, career and personal/social development through accountability measures especially examining efforts to close achievement, opportunity and attainment gaps." It may seem that the pressure to produce scientific evidence is unreasonable, given the many duties that most school counselors perform. It is important to remember that without such evidence, school counselors may have difficulty justifying their role in a system that is increasingly data-driven.

This is an optimal time for counselor educators and researchers to collaborate with school counselors in order to enhance the credibility of the school counseling profession. Together they can produce the high-quality research that is necessary for both accountability and advancement of the field.

References

Allen, J. (1992). Action-oriented research: Promoting school counselor advocacy and accountability (ERIC Digest No. ED347477). Retrieved June 9, 2002, from http://www.ed.gov/databases/ERIC_Digest/ed347477.html

American School Counselor Association. (2004). Ethical standards for school counselors. Alexandria, VA: Author. Retrieved August 21, 2005, from http://www.schoolcounselor.org/files/ethical%20standards.pdf

American School Counselor Association. (2005). The ASCA national model: A framework for school counseling programs (2nd ed.). Alexandria, VA: Author.

Barrett, P. M., Webster, H. M., & Wallis, J. R. (1999). Adolescent self-esteem and cognitive skills training: A school-based intervention. Journal of Child and Family Studies, 8, 217-227.

Bauman, S. (2004). School counselors and research revisited. Professional School Counseling, 7, 141-151.

Brigman, G., & Campbell, C. (2003). Helping students improve academic achievement and school success behavior. Professional School Counseling, 7, 91-98.

Brown, D., & Trusty, J. (2005). School counselors, comprehensive school counseling programs, and academic achievement: Are school counselors promising more than they can deliver? Professional School Counseling, 9, 1-8.

Campbell, C., & Brigman, G. (2005). Closing the achievement gap: A structured approach to group counseling. Journal for Specialists in Group Work, 30, 67-82.

Campbell, D.T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Chicago: Rand McNally.

Cramer, S. H., Herr, E. L., Morris, C. N., & Frantz, T.T. (1970). Research and the school counselor. Boston: Houghton Mifflin.

Farrell, A. D., Meyer, A. L., & White, K. S. (2001). Evaluation of Responding in Peaceful and Positive Ways (RIPP): A school-based prevention program for reducing violence among urban adolescents. Journal of Clinical Child Psychology, 30, 451-463.

Gliner, J. A., Morgan, G. A., & Harmon, R.J. (2003). Pretest-posttest comparison group designs: Analysis and interpretation. Journal of the Academy of Child and Adolescent Psychiatry, 42, 500-503.

Lee, C. C., & Workman, D. J. (1992). School counselors and research: Current status and future direction. School Counselor, 40, 15-19.

Loesch, L. C. (1988). Is "school counseling research" an oxymoron? In G. R. Walz (Ed.), Research and counseling: Building strong school counseling programs (pp. 169-180). Alexandria, VA: American School Counselor Association.

McGannon, W., Carey, J., & Dimmitt, C. (2005). The current status of school counseling outcome research (Research Monograph No. 2). Amherst, MA: University of Massachusetts, School of Education, Center for School Counseling Outcome Research.

No Child Left Behind Act of 2001, Pub. L. No. 107-110, 115 Stat. (2002).

Poynton, T. A. (2005). EZAnalyze (Version 2.0) [Computer software]. Retrieved August 21, 2005, from http://www. ezanalyze.com

Sink, C. A. (2005). Comprehensive school counseling programs and academic achievement--A rejoinder to Brown and Trusty. Professional School Counseling, 9, 9-12.

Webb, L. D., Brigman, G. A., & Campbell, C. (2005). Linking school counselors and student success: A replication of the Student Success Skills approach targeting the academic and social competence of students. Professional School Counseling, 8, 407-413.

Whiston, S. C. (2002). Response to the past, present, and future of school counseling: Raising some issues. Professional School Counseling, 5, 148-155.

Whiston, S. C., & Sexton, T. L. (1998). A review of school counseling outcome research: Implications for practice. Journal of Counseling & Development, 76, 412-426.

Wilson, N. S. (1985). School counselors and research: Obstacles and opportunities. School Counselor, 33, 111-119.

Sheri Bauman, Ph.D., is an assistant professor in the Department of Educational Psychology at the University of Arizona, Tucson. E-mail: [e-mail]