An assessment of the evidence-base for school-wide positive behavior support.
The use of SWPBS has increased quite rapidly across schools. This is happening against a backdrop of enthusiasm among policymakers, researchers and practitioners about the use of evidence-based practices in school settings. As SWPBS continues to attract the interest of school personnel it is necessary to look at this approach and examine its evidence base. This study was an attempt to extend previous work to that effect. Like previous efforts, this study demonstrated that although SWPBS has become quite popular, the evidence base may still be classified as promising. Research on SWPBS has to address many methodological limitations to strengthen its evidence base.
KEYWORDS: Positive behavior support, school-wide, evidence-based
As violent behavior in our schools reaches critical proportions (Safran & Oswald, 2003), policymakers, school administrators, teachers and families face a mammoth task to create violence-free school environments (Horner & Sugai, 2000; Taylor-Greene et al., 1997). This concern was heightened by the 1998 White House mandate that called for safe school environments (Dwyer, Osher & Warger, 1998; Horner & Sugai, 2000). Unfortunately, the traditional punitive and reactive practices, such as suspension or expulsion of the student perpetrators, have not been effective in addressing this problem (Morgan-D'Atrio, Northup, LaFleur, & Spera, 1996; Scheuerman & Hall, 2012; Taylor-Greene et al, 1997) prompting the search for alternative ways of handling dangerous behavior in schools. One of the approaches that has emerged out of this search is the use of school-wide positive behavior supports (SWPBS).
SWPBS is a systems approach that derives from the principles of applied behavior analysis. The approach aims to establish a safe school environment and a positive school culture that supports positive behavioral and academic outcomes for all students. According to Sugai and Horner (2002), SWPBS is based on data-driven decision making targeting measurable outcomes that can be attributed to specific practices implemented across settings capable of supporting these practices. The approach requires school personnel to "define, teach, and reward expected behaviors, develop peer support systems, and implement clear consequences for inappropriate behavior" (Taylor-Greene et al., 1997, p. 100) and is usually implemented across three tiers--primary, secondary and tertiary (see Lane, Robertson, & Graham-Bailey, 2006 as well as Sugai et al., 2000 for a description of these tiers).
There is a growing body of research supporting the efficacy of SWPBS in reducing challenging behavior in schools (Bradshaw, Mitchell & Leaf, 2010; Bradshaw, Reinke, Brown, Bevans, & Leaf, 2008; Taylor-Greene & Kartub, 2000; Taylor-Greene et al., 1997) and promoting the academic achievement of students (Horner et al., 2009; Lassen, Steele, & Sailor, 2006). Not surprisingly, at least 13,000 schools in the U.S. were adopting SWPBS by 2010 (Homer, Sugai, & Anderson, 2010). However, the recent No Child Left Behind Act calls for school personnel to use research validated practices in school settings. Consequently, some researchers are questioning the solidity of the evidence behind SWPBS (e.g., Lane et al, 2006). Lane et al. (2006) examined the methodological considerations of SWPBS studies published between 1990 and 2005 to determine the quality of that research. Specifically, Lane et al. (2006) reviewed 14 articles representing studies across 63 schools for secondary students. They examined SWPBS in relation to school characteristics, intervention focus and components, research design, reliability and validity issues, and intervention outcomes. They found several methodological limitations which included limited demographic information provided on the participating schools, insufficient description of the interventions that would promote replication of the studies, mostly descriptive research designs, and lack of reliability and validity information presented with the outcomes measures. As a result of these limitations, Lane et al. (2006) concluded that "although many of the [SWPBS] investigations produced desirable outcomes ... a number of methodological limitations limit the ability to draw accurate conclusions about intervention outcomes" (p. 186).
More recently, Horner et al. (2010) conducted a similar review of studies published between 2000 and 2009, albeit using a different approach, and reached a different conclusion. According to Horner et al. (2010) "the overall [SWPBS] approach carries sufficient experimental documentation to be classified as evidence based and to warrant large scale implementation" (p. 11). Clearly, there appears to be a discrepancy between these two conclusions. While Horner et al. (2010) stated that there is sufficient experimental evidence to support the efficacy of SWPBS, Lane et al. (2006) concluded that many "methodological limitations limit the ability to draw accurate conclusions about intervention outcomes" (p. 186). The question whether SWPBS can be considered evidence based is still largely unanswered. In trying to answer this question Horner et al. (2010) proposed criteria for determining whether a practice is evidence based or not, which they used in their review to reach at the afore-mentioned conclusion.
The criteria Horner et al. (2010) proposed included the following five elements: (a) the practice and participants are defined with operational precision, (b) the research employs valid and reliable measures, (c) the research is grounded in rigorous designs, (d) the research documents experimental effects without iatrogenic outcomes, and (e) the research documents effects. The criteria apply "primarily to the consideration of individual studies" (p. 6). However, Horner et al. (2010) applied the criteria to the SWPBS research body in concert and not to individual studies separately. Unfortunately/doing so may create a false impression about the solidity of the research behind SWPBS because studies that fulfill one of the quality indicators may be cited to support the body of research when those studies may actually have fundamental methodological flaws. Thus, as Horner et al. (2010) acknowledge, applying the criteria to individual studies may yield a better assessment of the evidence behind SWPBS.
This study sought to extend the work of Horner et al. (2010) in assessing the evidence base for SWPBS. However, unlike in the Horner et al. (2010) study, in this study the proposed criteria were applied to individual studies. It is anticipated that such an examination will contribute to scholarship on SWPBS by highlighting strengths and weaknesses of the research behind this approach. In doing so, we will be able to determine the extent to which SWPBS should be considered evidence based and possibly identify ways to strengthen that body of evidence.
Articles that were included in this review reported primary level (see Lane et al., 2006) behavioral interventions targeting all students in the school. The articles were experimental studies that reported student outcomes and were published in a peer-reviewed journal between the years 1990 and July 2011.
The articles were first located through an online search using ERIC, PsychInfo and EBSCO. Next, a list of journals publishing the articles identified from this search was generated and another online search of the journals' websites was conducted. Any missing journals were searched through collections of print copies. Articles included in this review came from the following journals: Psychology in the Schools, Journal of Positive Behavior Interventions, Journal of Emotional and Behavioral Disorders, Education and Treatment of Children, British Journal of Educational Psychology, and the American Educational Research Journal.
A matrix for coding the articles was developed based on the criteria proposed by Horner et al. (2010), that include the following five elements: (a) the practice and participants are defined with operational precision, (b) the research employs valid and reliable measures, (c) the research is grounded in rigorous designs, (d) the research documents experimental effects without iatrogenic outcomes, and (e) the research documents effects. Each of these criteria are defined below based on descriptions provided by Horner et al. (2010).
The practice and participants are defined with operational precision. According to Horner et al. (2010), a study that meets this criterion has to "define the practice (or component practices) with precision to allow replication" (p. 7). In addition, the study has to define both the children and adults (e.g., school personnel) in the study in operational terms that would allow replication. Thus, to determine whether the studies met this criterion each study was coded "yes", "no", or "not clear" in response to each of the three questions: Is the intervention defined with operational precision to allow replication of the study? Are the children in the study described operationally to allow replication of the study? Are the adults (e.g., school personnel) in the study described operationally to allow replication of the study?
The research employs valid and reliable measures. This criterion referred to whether a study used a combination of "standardized assessment measures and direct observation of student behavior to assess effects." (Horner et al., 2010, p. 7). Studies were therefore coded on whether the measures used would be considered valid and reliable (at least 80%) for the variables they measured. Also included in Horner et al.'s (2010) criterion were specific indices of fidelity, such as the School-wide Evaluation Tool (SET). Studies were coded "yes", "no", or "not clear" on whether they used a valid and reliable measure (e.g., SET) to assess fidelity. "Not clear" meant that reliability and/or validity measures were described, but did not report sufficient data to assess reliability and/or validity. Studies were also coded "yes", "no", or "not clear" on whether they used a combination of specific indices (e.g., direct observation, office discipline referrals [ODRs], and academic outcomes) to assess effect. "Not clear" meant that it could not be determined from the presented data whether the researchers used a combination of indices for outcomes.
The research is grounded in rigorous designs. Horner et al.'s (2010) definition included both randomized control group designs and single-case designs. Thus, studies were coded "yes," "no," or "not clear" on each of four indicators (i.e. whether they used randomized control trials, whether all possible threats to internal and external validity were controlled, and whether the data analysis techniques used were relevant and appropriate in addressing the research questions asked). "Not clear" meant that the researchers provided ambiguous information about the experimental groups and procedures used in the study, but was not sufficient to make a judgment about the type of design used.
The research documents experimental effects without iatrogenic outcomes. This criterion refers to whether a study demonstrates that implementation of SWPBS resulted in positive student outcomes without any negative effects. Studies were therefore coded "yes" if they reported positive student outcomes without any negative effects. If a study reported any negative effects it was coded "no". Further, effect sizes were calculated for each study to determine the social significance of the effects. "Not clear" was coded if the data provided were not sufficient to calculate effect size (i.e., the researchers did not report data such as the standard deviations and means for either baseline and intervention condition or control and treatment groups).
The research documents effects. In order to meet this criterion a study has to demonstrate two indicators namely sustainability which was determined by high implementation fidelity and consistent administrative support. Consistent administrative support is defined to mean administration involvement in data collection and decision-making as well as social validity measures reporting administrative satisfaction. Each study was coded "yes," "no," or "not clear" on whether there was consistent administrative support. For fidelity, if a fidelity index of at least 80% is reported and follows conventional procedures for calculating interrater-agreement it was coded "yes"; studies were coded "no" if they did not meet this condition. If fidelity is not reported as some index, but is explicitly measured, it was coded "not clear."
The first and second authors served as raters for inter-rater agreement. Prior to the coding, the raters discussed the criteria and reached an agreement on each to ensure uniform application of the criteria. Inter-rater agreement was calculated for all ratings by dividing the total number of agreements by the total number of agreements plus disagreements and multiplying the result by 100. Inter-rater agreement was 84.5%. After the inter-rater agreement was calculated, the raters discussed the disagreements and reached an agreement on each of them. The raters mostly disagreed on "not clear" which may reflect a not-so-clear definition of that category.
The search generated a total of 34 articles. Twenty four of the studies were descriptive non-experimental studies mostly based on the AB single-case design. Because descriptive studies are designed to provide descriptive information about SWPBS and not necessarily to determine experimental effects, the 24 studies were excluded from the analysis. As a result, only 10 studies were considered experimental and were included in the analysis.
Although most of the studies were conducted in elementary school settings, the settings also included middle school and high schools (both rural and urban). Table 1 displays the demographic characteristics of the studies.
Table 1 Characteristics of Studies Study School Research Design Dependent Characteristics Variable(s) Bradshaw, Rural and Longitudinal ODRs, school Mitchell, & Lea suburban randomized suspensions, (2010) elementary effectiveness academic schools study (37 schools achievement were matched on selected demographics and randomized to the treatment (21) and control group (16)) Colvin & Middle schools Pre-post design Office Kameenui (experimental and referrals (1993) control groups) Cook, Habib, Not urban Middle Randomized School climate Phillips, schools experiment (21 measures (e.g., Settersten, schools were quality of Shagle, & randomly assigned student Degirmencioglu to experimental relations with (1999) and control staff); Student groups) moderator and outcome measures (e.g., number of official days of absences, GPA, misbehavior) Gottfredson, Urban middle Nonequivalent Teacher ratings Gottfredson, & schools control group of classroom Hybl (1993) design (pre-post orderliness, differences) classroom organization, student disruptive behavior, student attendance to academic work. Student reports of frequency of misbehavior, clarity of rules, level of positive reinforcement. School discipline records. Horner, Sugai, Urban elementary Randomized Perceived Smolkowski, schools wait-list control schools safety, Eber, Nakasato, effectiveness ODRs, academic Todd, & trial achievement Esperanza (2009) Lane, Wehby, high schools Experimental Academic Robertson, & design i.e., 5 x 2 achievement, Rogers (2007) group (Group x attendance, Time) behavior, repeated-measures referrals, model access to reinforcement Metzler, intermediate & An AB design in Use of positive Biglan, Rusby, middle schools one community with reinforcement & Sprague within twenty a comparison for appropriate (2001) miles of a city community behavior, ODRs, student perception of safety, & student reports of harassment. Shapiro, middle and Pre- and Knowledge of Burgoon, elementary urban post-experimental psychosocial Welker, & public schools program assessment skills, Clough (2002) with comparison attitudes towards guns, aggression Sprague, Middle & Treatment ODRs, adults Walker, Golly, elementary comparison perceptions of White, Myers, & Suburban and analysis (nine school safety, Shannon (2001) urban community treatment and six perceptions of schools comparison status of groups) school discipline, & student social skills knowledge Stevens, primary and Experimental Levels of Bourdeaudhuij, secondary pre-test/posttest bullying and & Oost (2000) schools comparison being bullied, including a positive control group interactions among students Study Intervention Measure(s) of Findings Fidelity Bradshaw, Schoolwide School-wide Significant Mitchell, & Lea Positive Evaluation reduction in (2010) Behavior Tool; Effective ODRs and Interventions Behavior suspensions, and Supports. Support Survey improvements in The program standardized involved team test formation, achievement external scores behavioral support coach, definition of expectations for positive student behavior, teaching of expected behavior, schoolwide system of rewards for positive behavior, a system of responding to behavioral violations, data collection and analysis Colvin & School-wide None 50% decrease in Kameenui program called ODRs, decrease (1993) Project PREPARE. in suspensions, The program is a detentions and proactive office instructional consequences approach to managing problem behavior on a school-wide basis. Cook, Habib, The Comer's Implementation The program did Phillips, School questionnaire not have Settersten, Development completed each influence on Shagle, & Program. It year school climate Degirmencioglu seeks to enhance and student (1999) academic outcomes outcomes by improving the social relationships and social climate in a school via development of a school improvement plan, support from school community and progress monitoring. Gottfredson, School and Program Improvement in Gottfredson, & classroom wide Development student reports Hybl (1993) disciplinary Evaluation of classroom policy reviews, method (a form organization, behavior designed to classroom tracking system, increase the order, and rule classroom strength of clarity. organization and implementation Decrease in management, and ODRs positive reinforcement Horner, Sugai, School-wide School-wide Low rates of Smolkowski, Positive Evaluation ODRs, Eber, Nakasato, Behavior Tool improvement in Todd, & Support. The state reading Esperanza program involved standard (2009) defining behavioral expectations, teaching behavioral expectations, rewarding positive behavior, delivering predictable consequences for inappropriate behavior, data collection, administrator support, district support Lane, Wehby, SWPBS Program. Teacher Improvements in Robertson, & Involved 6 self-report GPA, decreases Rogers (2007) training (checklist), in unexcused sessions for Direct tardiness, school observation by decreases in personnel, research disciplinary behavioral assistants contacts. expectations, reinforcement procedures Metzler, The Effective Survey rating Increases in Biglan, Rusby, Behavior Support the frequency proportion of & Sprague as part of The with which students (2001) Community program receiving Builders strategies had positive Intervention. been reinforcement, The program implemented decreased ODRs, targets lower levels of increasing aggression appropriate social behavior in school settings by defining rules and expectations, teaching expected behaviors, providing praise, monitoring student behavior, enforcing rules, evaluating progress Shapiro, The Peacemakers Brief teacher Lower LS means, Burgoon, Program. The report decrease in Welker, & program includes questionnaire suspensions Clough (2002) instruction and remediation components Sprague, Effective Assessing Reduction in Walker, Golly, Behavioral Behavioral ODRs White, Myers, & Support Model. Support in Shannon (2001) The program Schools involves checklist defining problem and appropriate behaviors, teaching alternative behaviors, providing effective incentives to encourage behavior change, monitoring of implementation, staff training, & evaluation of effectiveness Stevens, The Flemish None Improved Bourdeaudhuij, school-based outcomes on & Oost (2000) anti-bullying bullying and intervention victimization program. The program involved restructuring the social environment via implementation of clear anti-bullying rules
Application of the Criteria
Tables 2 and 3 display the results for how the studies were coded.
Table 2 Whether Studies Operationally Defined Practice and Participants, Employed Valid and Reliable Measures, and Were Grounded in Rigorous Decisions Author (year) Practice & Participants Research is are Defined with Grounded in Operational Precision to Rigorous Designs promote Replication Adults Children Practice Randomized Threats to Control Internal Trials Validity Controlled Bradshaw, Y Y Y Y Y Mitchell, & Leaf (2010) Colvin & N N Y N Y Kameenui (1993) Cook, Habib, Y Y N Y Y Phillips, Settersten, Shagle, & Degirmencioglu (1999) Gottfredson, N NC Y N N Gottfredson, & Hybl (1993) Horner, Sugai, Y Y Y Y Y Smolkowski, Eber, Nakasato, Todd, & Esperanza (2009) Lane, Wehby, Y Y Y N Y Robertson, & Rogers (2007) Meteler, Y Y Y N N Biglan, Rusby, & Sprague (2001) Shapiro, N Y Y Y N Burgoon, Welker, & Clough (2002) Sprague, N Y Y N N Walker, Golly, White, Myers, & Shannon (2001) Stevens, NC NC Y Y N Bourdeaudhuij, & Oost (2000) Author (year) Research Employs Valid and Reliable Measures Threats to Appropriate Outcomes Treatment External Data Integrity Validity Analysis Controlled Techniques Bradshaw, Y Y Y Y Mitchell, & Leaf (2010) Colvin & N NC N N Kameenui (1993) Cook, Habib, Y Y Y Y Phillips, Settersten, Shagle, & Degirmencioglu (1999) Gottfredson, N NC Y Y Gottfredson, & Hybl (1993) Horner, Sugai, Y Y Y Y Smolkowski, Eber, Nakasato, Todd, & Esperanza (2009) Lane, Wehby, N Y Y Y Robertson, & Rogers (2007) Meteler, N N Y Y Biglan, Rusby, & Sprague (2001) Shapiro, N Y Y Y Burgoon, Welker, & Clough (2002) Sprague, N N Y Y Walker, Golly, White, Myers, & Shannon (2001) Stevens, N Y N N Bourdeaudhuij, & Oost (2000) Author (year) Table 3 How Studies Documented Effects Author Research Documents Research Documents (year) Experimental Effects Effects (high fidelity & Without Iatrogenic sustains administrative Outcomes support) Outcomes Effect Size High Administrative Fidelity Support Bradshaw, Y NC Y Y Mitchell, & Leaf (2010) Colvin & Y NC N Y Kameenui (1993) Cook, Habib, N NC N Y Phillips, Settersten, Shagle, & Degirmencioglu (1999) Gottfredson, NC Attending to NC N Gottfredson, & academic work Hybl (1993) = 0.01 Disrupting classroom =0.01 Rebellious behavior = 0.27 Horner, Sugai, Y ODRs = 0.03 Y Y Smolkowski, Eber, Nakasato, Todd, & Esperanza (2009) Lane, Wehby, Y GPA (across N NC Robertson, & groups) = Rogers (2007) -0.12 to 0.56 Unexcused tardiness (across groups) = 0.10 to 0.65 Suspensions = -0.38 to -0.04 Disciplinary contacts (across groups) = -0.25 to 0.52 Metzler, N ODRs = 0.90 N Y Biglan, Rusby, & Sprague (2001) Shapiro, Y Aggressive N N Burgoon, behavior = Welker, & -0.01; Clough (2002) Disciplinary incidences = -0.12; Conflict mediation referrals = -0.07; Suspensions = -0.11 Sprague, Y NC N Y Walker, Golly, White, Myers, & Shannon (2001) Stevens, Y Bullying = N N Bourdeaudhuij, -0.13; & Oost (2000) Being bullied = -0.27 & 0.00; Positive interactions = 0.29 & 0.00
The practice and participants are defined with operational precision. Four of the 10 studies (i.e. Bradshaw et al., 2010; Horner et al., 2009; Lane, Wehby, Robertson, & Rogers, 2007; Metzler, Biglan, Rusby, & Sprague, 2001) defined the participants (both adults and children) and practice with precision to allow replication. Although all the other 6 studies defined the practice with precision, each of them was either unclear or did not define either the adults or children (or both).
The research is grounded in rigorous research. Three of the 10 studies (i.e. Bradshaw et al., 2010; Cook et al, 1999; Horner et al., 2009) met this criterion; all 3 met the four indicators for this criterion (i.e., they used randomized control designs, controlled for both internal and external threats to validity and used appropriate data analysis techniques). Three studies (i.e. Gottfredson, Gottfredson & Hybl, 1993; Metzler et al., 2001; Sprague et al., 2001) did not meet any one of the four indicators. The rest of the studies (n = 4) met at least one of the four indicators.
The research employs valid and reliable measures. Eight of the 10 studies employed valid and reliable measures of both the outcomes and treatment integrity. Only 2 studies did not meet this criterion (i.e. Colvin & Kameenui, 1993; Stevens, De Bourdeaudhuij & Van Oost, 2000). Student outcome measures used in the 8 studies included academic achievement, ODRs and suspensions; treatment integrity measures included the SET, Effective Behavior Support Survey, Program Development Evaluation and direct observation.
The research documents experimental effects without iatrogenic out-comes. Seven out of the 10 studies documented experimental outcomes without any iatrogenic effects. Two studies did not meet this criterion (i.e. Cook et al., 1999; Metzler et al., 2001) while one study was coded as "not clear" (i.e. Gottfredson et al., 1993). Effect sizes were computed for 6 of the studies; the other 4 did not report adequate data to allow computation of effect sizes. The effect sizes ranged from 0.00 to 0.9. The effect sizes are listed in Table 3.
The research documents effects. Only 2 studies demonstrated both high fidelity and sustained administrative support (i.e. Bradshaw et al., 2010; Horner et al, 2009). This means they had a fidelity index of at least 80% and administration involvement in data collection and decision-making as well as social validity measures reporting administrative satisfaction. The rest of the studies failed to meet either of these indicators (high fidelity and sustained administrative support) or both.
Summary. Overall, only 2 of the 10 studies (i.e. Bradshaw et al., 2010; Horner et al., 2009) met all the five criteria for the evidence base for SWPBS; however, the effect size for the Bradshaw et al. (2010) study could not be computed. The 2 studies defined the practice and participants with operational precision to promote replication, employed valid and reliable measures of student outcomes and treatment fidelity, documented experimental effects without iatrogenic effects, demonstrated high fidelity and sustained administrative support, and were grounded in rigorous research designs.
This review was conducted to examine the evidence base behind SWPBS. The study sought to extend earlier work by Horner et al. (2010), who examined the evidence behind SWPBS studies published between 1990 and July 2011. Horner et al. (2010) used a five-criteria method which they applied to the body of SWPBS research. Although the five criteria "apply primarily to the consideration of individual studies" (Homer et al., 2010, p. 6), Horner et al. (2010) applied it to the body of SWPBS in concert. This study differs from that previous work in that it applies the five criteria to individual studies published over the past 20 years (1990-July 2011).
Overall, only 2 of the 10 studies that were analyzed met all the five criteria (i.e., Bradshaw et al., 2010; Horner et al., 2009). This finding is consistent with the findings of an earlier study by Lane et al. (2006) who, in examining methodological considerations of SW-PBS studies published between 1990 and 2005, found that many of the studies they reviewed lacked methodological rigor.
Although most studies (n = 9) defined the independent variable with operational precision that would allow replication, it is of concern that only half of the studies defined both the participants and practice with operational precision. Failure to define the participants limits the extent to which the findings can be generalized. It is particularly important for future studies to operationally describe both participating adults and children because, in addition to promoting external validity, knowing the teacher characteristics would help to identify factors that predict student outcomes related to SWPBS (Lane et al., 2006).
It is encouraging that most studies (n = 8) in this review used a combination of measures such as academic achievement and ODRs (e.g., Bradshaw et al., 2010; Horner et al., 2009; Lane et al., 2007) to evaluate outcomes. Lane et al. (2006) lamented the limited attention that is given to the selection of measures to evaluate intervention outcomes. According to Lane et al. (2006) "too often studies in school-wide primary interventions have relied exclusively on one outcome measure" (p. 192). Specifically, ODRs have been the most preferred measure of SWPBS outcomes in schools (Lassen et al., 2006). However, "this metric does not yield a complete picture of how well a school is functioning ... [and it does] not capture other important factors that PBS seeks to impact" (Lassen et al., 2006, p.701). Other researchers have found the use of this metric as a screening tool to have a high degree of error (Morgan-D'Atrio et al., 1996; Nelson, Benner, Reid, Epstein & Currin, 2002). Therefore, using this metric in combination with other measures such as academic achievement will yield better information about the impact of SWPBS in school settings than using it as a sole measure.
According to the results of this evaluation, most of the studies (n = 8) used acceptable measures of implementation fidelity. Previous studies found that most studies did not measure and report implementation fidelity data (e.g., Horner et al., 2010; Lane et al, 2006; Mclntyre, Gresham, DiGennaro, & Reed, 2007; Wheeler et al. 2009). For example, Mclntyre et al. (2007) examined the treatment integrity of school-based interventions in studies published between 1991 and 2005 and found that only 30% of the studies had reported treatment integrity data. Thus, it is encouraging that most of the studies in this review measured and reported data on implementation fidelity. As Lane et al. (2006) stated "the absence of treatment integrity data poses clear threats to the internal and external validity of any study" (p. 192).
However, even though it is encouraging that most of the studies in this review monitored and evaluated implementation fidelity, it is still problematic that only 2 of the 10 studies reported high fidelity. Measuring implementation fidelity is important because it ensures accurate application of the intervention (McIntyre et al., 2007) which helps to determine the "presence of a functional relationship between the dependent and independent variables (Wheeler, Baggett, Fox & Blevins, 2006, p.45). It is therefore, crucial for researchers to ensure that SWPBS interventions are implemented as intended as doing so would ensure reliable evaluation of outcomes.
Further, this evaluation indicated only 3 of the 10 studies were grounded in rigorous designs. This finding is also consistent with the findings of Lane et al. (2006) who reported that SWPBS studies published between 1990 and 2005 were predominantly descriptive and non-experimental. However, it is encouraging to note that 2 of the 3 studies (i.e., Bradshaw et al., 2010 and Horner et al., 2009) were published within the last 2 years. This development may be an indication that researchers are responding to earlier calls to address this limitation (e.g., by Kern & Manz, 2004; Lane et al., 2006). Thus, there is need for more experimental studies in order to bolster the evidence base for SWPBS.
Horner et al. (2010) reported no negative effects had been reported as a result of the implementation of SWPBS. Results of this study corroborate that finding as 7 of the 10 studies reported positive outcomes with no iatrogenic outcomes. Nevertheless, 2 studies did not show positive effects while the impact of the outcomes for 2 of the studies was "not clear." Two of the studies (i.e., Lane et al., 2007; Metzler et al., 2001) had large effect sizes of 0.9 and 0.5 (according to Cohen, 1988) respectively for at least one of the outcomes reported. The effect sizes for the other studies fell in the small to medium range (0.1 to 0.3) (Cohen, 1988) for at least one of the outcomes reported. It is important that, to date, SWPBS is mostly associated with positive student outcomes. Even though the SWPBS research is weak (Lane et al., 2006), the practice has demonstrated a level of potency that is important in promoting its widespread use in schools especially when the level of problem behavior in schools reaches alarming proportions (Safran & Oswald, 2003).
The sustainability of effects is important since it may take a few years to implement SWPBS effectively. Consistent administrative support has been identified as a key factor in promoting sustained implementation of SWPBS (Handler et al, 2007). It is therefore important that most of the studies (n = 6) in this review demonstrated sustained administrative support.
Finally, the most striking observation was that only 2 of the 10 studies met all the five criteria for the evidence-base for SWPBS. This finding appears to dampen the conclusion of Horner et al. (2010) that SWPBS "carries sufficient experimental documentation to be classified as evidence based ..." (p. 10). In fact results of this review, like those of Lane et al. (2006), suggest that SWPBS is quite a promising approach which, requires more inquiry with enhanced methodological rigor to be established as evidence based. Researchers should pay particular attention to the operational definitions of participants (both children and adults), use of experimental designs to examine the effects of SWPBS on student outcomes, and the evaluation of treatment fidelity. It is anticipated that addressing these issues will help to solidify the quite promising evidence base for SWPBS.
This study was an attempt to extend previous efforts aimed at examining the evidence base for SWPBS. However, the study is not without limitations. Horner et al.'s (2010) criteria use broad categories which made formulation of clear definitions for scoring difficult. For example, these researchers found it difficult to come up with a clear definition for "consistent administrative support". These challenges are evidenced by the moderate inter-rater agreement of 84.5%. As a result, replication of the study can be difficult given the lack of precision for some of the definitions.
Nevertheless, in spite of these limitations results of this review demonstrated that although there is evidence pointing to its efficacy, the research behind SWPBS is still weak. Many studies demonstrating the effective applications of SWPBS do not have the methodological rigor for evidence-based practices. This may be due to the applied nature of SWPBS. However, it is quite a promising approach which requires researchers to address these limitations in order for it to be classified as evidence-based. Future research on SWPBS should pay particular attention to issues like the rigor of the research designs used, treatment integrity, operational definition of participants and practice, and use of valid and reliable measures of student outcomes.
References marked with an asterisk indicate studies included in the review.
* Bradshaw, C. P., Mitchell, M. M., & Leaf, P. J. (2010). Examining the effects of schoolwide positive behavioral interventions and supports on student outcomes: Results from a randomized controlled effectiveness trial in elementary schools. Journal of Positive Behavior Interventions, 12, 133-148. doi:10.1177/1098300709334798
Bradshaw, C. P., Reinke, W. M, Brown, L. D., Bevans, K. B. & Leaf, P.J. (2008). Implementation of school-wide positive behavioral interventions and supports (PBIS) in elementary schools: Observations from a randomized trial. Education and Treatment of Children, 32(1), 1-26. Retrieved from http://www.education-andtreatmentofchildren.net/
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.
* Colvin, G., & Kameenui, E. J. (1993). Reconceptualizing behavior management and school-wide discipline and general education. Education and Treatment of Children, 16,361-382. Retrieved from http://www.educationandtreatmentofchildren.net/
* Cook, T. D., Habib, F., Phillips, M., Settersten, R. A., Shagle, S. C, & Degirrnencioglu, S. M. (1999). Comer's school development program in Prince George's County, Maryland: A theory-based evaluation. American Educational Research Journal 36, 543-597. doi:10.3102/00028312036003543
Dwyer, K. P., Osher, D., & Warger, W. (1998). Early warning timely response: A guide to safe schools. Washington, DC: U.S. Department of Education.
* Gottfredson, D. C, Gottfredson, G. D., & Hybl, L. G. (1993). Managing adolescent behavior a multiyear, multischool study. American Educational Research Journal, 30, 179-215. doi:10.3102/00028312030001179
Handler, M. W., Ray, J., Connell, J., Thier, K., Feinberg, A., & Putnam, R. (2007). Practical considerations in creating school-wide positive behavior support in public schools. Psychology in the Schools, 44, 29-38. doi:10.1002/pits
Horner, R. H., & Sugai, G. (2000). School-wide positive behavior support: An emerging initiative. Journal of Positive Behavior Interventions, 2, 231-232. Retrieved from http://pbi.sagepub.com/
Horner, R. H., Sugai, G., & Anderson, C. M. (2010). Examining the evidence base for school-wide positive behavior support. Focus on Exceptional Children, 42(8), 1-14. Retrieved from http://vnweb.hwwilsonweb.com/hww/Journals/getIssues.jhtml?sid-HWW:EDI&id=03212
* Horner, R. R, Sugai, G., Smolkowski, K., Eber, L., Nakasato, J., Todd, A. W., & Esperanza, J. (2009). A randomized, wait-list controlled effectiveness trial assessing school-wide positive behavior support in elementary schools. Journal of Positive Behavior Interventions, 11,133-144. doi:10.1177/1098300709332067
Kern, L., & Manz, P. (2004). A look at current validity issues of school-wide behavior support. Behavioral Disorders, 30, 47-59.
Lane, K. L., Robertson, E. J., & Graham-Bailey, M. A. L. (2006). An examination of school-wide interventions with primary level efforts conducted in secondary schools: Methodological considerations, applications of research methodology. Advances in Learning and Behavioral Disabilities, 19, 157-199. doi:10.1016/S0735-004X(06)19007-2
* Lane, K. L., Wehby, J. H., Robertson, E. J., & Rogers, L. A. (2007). How do different types of high school students respond to school-wide positive behavior support programs? Characteristics and responsiveness of teacher identified students. Journal of Emotional and Behavioral Disorders, 15, 3-20. Retrieved from http://ebx.sagepub.com/
Lassen, S. R., Steele, M. M., & Sailor, W. (2006). The relationship between positive behavior support to academic achievement in an urban middle school. Psychology in the Schools. 43(6), 701-712. doi:10.1002/pits.20177
Mclntyre, L. L., Gresham, F. M., DiGennaro, F. D., & Reed, D. D. (2007). Treatment integrity of school-based interventions with children in the Journal of Applied Behavior Analysis 1991-2005. Journal of Applied Behavior Analysis, 40, 659-672. doi:10.1901/jaba.2007.659-672
* Metzler, C. W., Biglan, A., Rusby, J. C., & Sprague, J. R. (2001). Evaluation of a comprehensive behavior management program to improve school-wide positive behavior support. Education and Treatment of Children, 24(4), 448-479. Retrieved from http://www.educationandtreatmentofchildren.net/
Morgan-D'Atrio, G, Northup, J., LaFleur, L., & Spera, S. (1996). Toward prescriptive alternatives to suspensions: A preliminary evaluation. Behavioral Disorders, 21,190-200.
Nelson, J. R., Benner, G. J., Reid, R. C., Epstein, M. H., & Currin, D. (2002). The convergent validity of office discipline referrals with the CBCL-TRF. Journal of Emotional and Behavioral Disorders, 10, 181-188. Retrieved from http://ebx.sagepub.com/content/10/3/181
Safran, S. P., & Oswald, K. (2003). Positive behavior supports: Can schools reshape disciplinary practices? Exceptional Children, 69, 361-373. Retrieved from http://www.cec.sped.org/Content/NavigationMenu/Publications2/ExceptionalChil-dren/default.htm
* Shapiro, J. P., Burgoon, J. D., Welker, C. J., & Clough, J. B. (2002). Evaluation of the peacemakers program: School-based violence prevention for students in grades four through eight. Psychology in the Schools, 39, 87-100. doi:10.1002/pits.l0040
Scheuerman, B. K., & Hall, J. A. (2012). Positive behavior supports for the classroom. Boston, MA: Pearson.
* Sprague, J., Walker, H., Golly, A., White, K., Myers, D. R., & Shannon, T. (2001). Translating research into effective practice: The effects of a universal staff and student intervention on indicators of discipline and school safety. Education and Treatment of Children, 24, 495-511. Retrieved from http://www.education-andtreatmentofchildren.net/
* Stevens, V., De Bourdeaudhuij, I., & Van Oost, P. (2000). Bullying in Flemish schools: An evaluation of anti-bullying intervention in primary and secondary schools. British Journal of Educational Psychology, 70,195-210. doi: 10.1348/000709900158056
Sugai, G., & Horner, R. (2002). The evolution of discipline practices: School-wide positive behavior supports. Child & Family Behavior Therapy, 24, 23-50. Retrieved from http://www.informa-world.com/smpp/title~content=t792303978~link=cover
Sugai, G., Horner, R. H., Dunlap, G., Hieneman, M., Lewis, T. J., Nelson, C. M., ... Ruef, M. (2000). Applying positive behavior support and functional behavioral assessment in schools. Journal of Positive Behavior Interventions, 2, 131-143. Retrieved from http://pbi.sagepub.com/
Taylor-Greene, S., Brown, D., Nelson, L., Longton, J., Gassman, T., Cohen, J., ... Hall, S. (1997). School-wide behavioral support: Starting the year off right. Journal of Behavioral Education, 7, 99-112. Retrieved from http://www.metapress.com/content/105719/?p=ef9622e70c3e4cf092685b679d8fb67c&pi=0
Taylor-Greene, S. J., & Kartub, D. T. (2000). Durable implementation of school-wide behavior support: The high five program. Journal of Positive Behavior Interventions, 2, 233-235. Retrieved from http://pbi.sagepub.com/
Wheeler, J. J., Baggett, B. A., Fox, J., & Blevins, L. (2006). Treatment integrity: A review of intervention studies conducted with children with autism. Focus on Autism and Other Developmental Disabilities, 21(1), 45-54.
Wheeler, J. J., Mayton, M. R., Carter, S. L., Chitiyo, M, Menendez, A. L., & Huang, A. (2009). An assessment of treatment integrity in behavioral intervention studies conducted with persons with mental retardation. Education and Training in Developmental Disabilities, 44, 187-195. Retrieved from http://www.dddcec.org/publications.htm#ETDD
Morgan Chitiyo and Michael E. May Southern Illinois University Carbondale
George Chitiyo Tennessee Technological University
Correspondence to Morgan Chitiyo, Southern Illinois University Carbondale, Educational Psychology & Special Education, 625 Wham Drive, Carbondale IL 62901-4618. email@example.com.
|Printer friendly Cite/link Email Feedback|
|Author:||Chitiyo, Morgan; May, Michael E.; Chitiyo, George|
|Publication:||Education & Treatment of Children|
|Article Type:||Clinical report|
|Date:||Feb 1, 2012|
|Previous Article:||An examination of typical classroom context and instruction for students with and without behavioral disorders.|
|Next Article:||A pilot study of a problem-solving model for team decision making.|