Printer Friendly

An assessment of the evidence-base for school-wide positive behavior support.

Abstract

The use of SWPBS has increased quite rapidly across schools. This is happening against a backdrop of enthusiasm among policymakers, researchers and practitioners about the use of evidence-based practices in school settings. As SWPBS continues to attract the interest of school personnel it is necessary to look at this approach and examine its evidence base. This study was an attempt to extend previous work to that effect. Like previous efforts, this study demonstrated that although SWPBS has become quite popular, the evidence base may still be classified as promising. Research on SWPBS has to address many methodological limitations to strengthen its evidence base.

KEYWORDS: Positive behavior support, school-wide, evidence-based

As violent behavior in our schools reaches critical proportions (Safran & Oswald, 2003), policymakers, school administrators, teachers and families face a mammoth task to create violence-free school environments (Horner & Sugai, 2000; Taylor-Greene et al., 1997). This concern was heightened by the 1998 White House mandate that called for safe school environments (Dwyer, Osher & Warger, 1998; Horner & Sugai, 2000). Unfortunately, the traditional punitive and reactive practices, such as suspension or expulsion of the student perpetrators, have not been effective in addressing this problem (Morgan-D'Atrio, Northup, LaFleur, & Spera, 1996; Scheuerman & Hall, 2012; Taylor-Greene et al, 1997) prompting the search for alternative ways of handling dangerous behavior in schools. One of the approaches that has emerged out of this search is the use of school-wide positive behavior supports (SWPBS).

SWPBS is a systems approach that derives from the principles of applied behavior analysis. The approach aims to establish a safe school environment and a positive school culture that supports positive behavioral and academic outcomes for all students. According to Sugai and Horner (2002), SWPBS is based on data-driven decision making targeting measurable outcomes that can be attributed to specific practices implemented across settings capable of supporting these practices. The approach requires school personnel to "define, teach, and reward expected behaviors, develop peer support systems, and implement clear consequences for inappropriate behavior" (Taylor-Greene et al., 1997, p. 100) and is usually implemented across three tiers--primary, secondary and tertiary (see Lane, Robertson, & Graham-Bailey, 2006 as well as Sugai et al., 2000 for a description of these tiers).

There is a growing body of research supporting the efficacy of SWPBS in reducing challenging behavior in schools (Bradshaw, Mitchell & Leaf, 2010; Bradshaw, Reinke, Brown, Bevans, & Leaf, 2008; Taylor-Greene & Kartub, 2000; Taylor-Greene et al., 1997) and promoting the academic achievement of students (Horner et al., 2009; Lassen, Steele, & Sailor, 2006). Not surprisingly, at least 13,000 schools in the U.S. were adopting SWPBS by 2010 (Homer, Sugai, & Anderson, 2010). However, the recent No Child Left Behind Act calls for school personnel to use research validated practices in school settings. Consequently, some researchers are questioning the solidity of the evidence behind SWPBS (e.g., Lane et al, 2006). Lane et al. (2006) examined the methodological considerations of SWPBS studies published between 1990 and 2005 to determine the quality of that research. Specifically, Lane et al. (2006) reviewed 14 articles representing studies across 63 schools for secondary students. They examined SWPBS in relation to school characteristics, intervention focus and components, research design, reliability and validity issues, and intervention outcomes. They found several methodological limitations which included limited demographic information provided on the participating schools, insufficient description of the interventions that would promote replication of the studies, mostly descriptive research designs, and lack of reliability and validity information presented with the outcomes measures. As a result of these limitations, Lane et al. (2006) concluded that "although many of the [SWPBS] investigations produced desirable outcomes ... a number of methodological limitations limit the ability to draw accurate conclusions about intervention outcomes" (p. 186).

More recently, Horner et al. (2010) conducted a similar review of studies published between 2000 and 2009, albeit using a different approach, and reached a different conclusion. According to Horner et al. (2010) "the overall [SWPBS] approach carries sufficient experimental documentation to be classified as evidence based and to warrant large scale implementation" (p. 11). Clearly, there appears to be a discrepancy between these two conclusions. While Horner et al. (2010) stated that there is sufficient experimental evidence to support the efficacy of SWPBS, Lane et al. (2006) concluded that many "methodological limitations limit the ability to draw accurate conclusions about intervention outcomes" (p. 186). The question whether SWPBS can be considered evidence based is still largely unanswered. In trying to answer this question Horner et al. (2010) proposed criteria for determining whether a practice is evidence based or not, which they used in their review to reach at the afore-mentioned conclusion.

The criteria Horner et al. (2010) proposed included the following five elements: (a) the practice and participants are defined with operational precision, (b) the research employs valid and reliable measures, (c) the research is grounded in rigorous designs, (d) the research documents experimental effects without iatrogenic outcomes, and (e) the research documents effects. The criteria apply "primarily to the consideration of individual studies" (p. 6). However, Horner et al. (2010) applied the criteria to the SWPBS research body in concert and not to individual studies separately. Unfortunately/doing so may create a false impression about the solidity of the research behind SWPBS because studies that fulfill one of the quality indicators may be cited to support the body of research when those studies may actually have fundamental methodological flaws. Thus, as Horner et al. (2010) acknowledge, applying the criteria to individual studies may yield a better assessment of the evidence behind SWPBS.

This study sought to extend the work of Horner et al. (2010) in assessing the evidence base for SWPBS. However, unlike in the Horner et al. (2010) study, in this study the proposed criteria were applied to individual studies. It is anticipated that such an examination will contribute to scholarship on SWPBS by highlighting strengths and weaknesses of the research behind this approach. In doing so, we will be able to determine the extent to which SWPBS should be considered evidence based and possibly identify ways to strengthen that body of evidence.

Method

Inclusion Criteria

Articles that were included in this review reported primary level (see Lane et al., 2006) behavioral interventions targeting all students in the school. The articles were experimental studies that reported student outcomes and were published in a peer-reviewed journal between the years 1990 and July 2011.

The articles were first located through an online search using ERIC, PsychInfo and EBSCO. Next, a list of journals publishing the articles identified from this search was generated and another online search of the journals' websites was conducted. Any missing journals were searched through collections of print copies. Articles included in this review came from the following journals: Psychology in the Schools, Journal of Positive Behavior Interventions, Journal of Emotional and Behavioral Disorders, Education and Treatment of Children, British Journal of Educational Psychology, and the American Educational Research Journal.

Procedure

A matrix for coding the articles was developed based on the criteria proposed by Horner et al. (2010), that include the following five elements: (a) the practice and participants are defined with operational precision, (b) the research employs valid and reliable measures, (c) the research is grounded in rigorous designs, (d) the research documents experimental effects without iatrogenic outcomes, and (e) the research documents effects. Each of these criteria are defined below based on descriptions provided by Horner et al. (2010).

The practice and participants are defined with operational precision. According to Horner et al. (2010), a study that meets this criterion has to "define the practice (or component practices) with precision to allow replication" (p. 7). In addition, the study has to define both the children and adults (e.g., school personnel) in the study in operational terms that would allow replication. Thus, to determine whether the studies met this criterion each study was coded "yes", "no", or "not clear" in response to each of the three questions: Is the intervention defined with operational precision to allow replication of the study? Are the children in the study described operationally to allow replication of the study? Are the adults (e.g., school personnel) in the study described operationally to allow replication of the study?

The research employs valid and reliable measures. This criterion referred to whether a study used a combination of "standardized assessment measures and direct observation of student behavior to assess effects." (Horner et al., 2010, p. 7). Studies were therefore coded on whether the measures used would be considered valid and reliable (at least 80%) for the variables they measured. Also included in Horner et al.'s (2010) criterion were specific indices of fidelity, such as the School-wide Evaluation Tool (SET). Studies were coded "yes", "no", or "not clear" on whether they used a valid and reliable measure (e.g., SET) to assess fidelity. "Not clear" meant that reliability and/or validity measures were described, but did not report sufficient data to assess reliability and/or validity. Studies were also coded "yes", "no", or "not clear" on whether they used a combination of specific indices (e.g., direct observation, office discipline referrals [ODRs], and academic outcomes) to assess effect. "Not clear" meant that it could not be determined from the presented data whether the researchers used a combination of indices for outcomes.

The research is grounded in rigorous designs. Horner et al.'s (2010) definition included both randomized control group designs and single-case designs. Thus, studies were coded "yes," "no," or "not clear" on each of four indicators (i.e. whether they used randomized control trials, whether all possible threats to internal and external validity were controlled, and whether the data analysis techniques used were relevant and appropriate in addressing the research questions asked). "Not clear" meant that the researchers provided ambiguous information about the experimental groups and procedures used in the study, but was not sufficient to make a judgment about the type of design used.

The research documents experimental effects without iatrogenic outcomes. This criterion refers to whether a study demonstrates that implementation of SWPBS resulted in positive student outcomes without any negative effects. Studies were therefore coded "yes" if they reported positive student outcomes without any negative effects. If a study reported any negative effects it was coded "no". Further, effect sizes were calculated for each study to determine the social significance of the effects. "Not clear" was coded if the data provided were not sufficient to calculate effect size (i.e., the researchers did not report data such as the standard deviations and means for either baseline and intervention condition or control and treatment groups).

The research documents effects. In order to meet this criterion a study has to demonstrate two indicators namely sustainability which was determined by high implementation fidelity and consistent administrative support. Consistent administrative support is defined to mean administration involvement in data collection and decision-making as well as social validity measures reporting administrative satisfaction. Each study was coded "yes," "no," or "not clear" on whether there was consistent administrative support. For fidelity, if a fidelity index of at least 80% is reported and follows conventional procedures for calculating interrater-agreement it was coded "yes"; studies were coded "no" if they did not meet this condition. If fidelity is not reported as some index, but is explicitly measured, it was coded "not clear."

Inter-rater Agreement

The first and second authors served as raters for inter-rater agreement. Prior to the coding, the raters discussed the criteria and reached an agreement on each to ensure uniform application of the criteria. Inter-rater agreement was calculated for all ratings by dividing the total number of agreements by the total number of agreements plus disagreements and multiplying the result by 100. Inter-rater agreement was 84.5%. After the inter-rater agreement was calculated, the raters discussed the disagreements and reached an agreement on each of them. The raters mostly disagreed on "not clear" which may reflect a not-so-clear definition of that category.

Results

The search generated a total of 34 articles. Twenty four of the studies were descriptive non-experimental studies mostly based on the AB single-case design. Because descriptive studies are designed to provide descriptive information about SWPBS and not necessarily to determine experimental effects, the 24 studies were excluded from the analysis. As a result, only 10 studies were considered experimental and were included in the analysis.

Although most of the studies were conducted in elementary school settings, the settings also included middle school and high schools (both rural and urban). Table 1 displays the demographic characteristics of the studies.
Table 1

Characteristics of Studies

Study School Research Design Dependent
 Characteristics Variable(s)

Bradshaw, Rural and Longitudinal ODRs, school
Mitchell, & Lea suburban randomized suspensions,
(2010) elementary effectiveness academic
 schools study (37 schools achievement
 were matched on
 selected
 demographics and
 randomized to the
 treatment (21) and
 control group
 (16))

Colvin & Middle schools Pre-post design Office
Kameenui (experimental and referrals
(1993) control groups)

Cook, Habib, Not urban Middle Randomized School climate
Phillips, schools experiment (21 measures (e.g.,
Settersten, schools were quality of
Shagle, & randomly assigned student
Degirmencioglu to experimental relations with
(1999) and control staff); Student
 groups) moderator and
 outcome
 measures (e.g.,
 number of
 official days
 of absences,
 GPA,
 misbehavior)

Gottfredson, Urban middle Nonequivalent Teacher ratings
Gottfredson, & schools control group of classroom
Hybl (1993) design (pre-post orderliness,
 differences) classroom
 organization,
 student
 disruptive
 behavior,
 student
 attendance to
 academic work.
 Student reports
 of frequency of
 misbehavior,
 clarity of
 rules, level of
 positive
 reinforcement.
 School
 discipline
 records.

Horner, Sugai, Urban elementary Randomized Perceived
Smolkowski, schools wait-list control schools safety,
Eber, Nakasato, effectiveness ODRs, academic
Todd, & trial achievement
Esperanza
(2009)

Lane, Wehby, high schools Experimental Academic
Robertson, & design i.e., 5 x 2 achievement,
Rogers (2007) group (Group x attendance,
 Time) behavior,
 repeated-measures referrals,
 model access to
 reinforcement

Metzler, intermediate & An AB design in Use of positive
Biglan, Rusby, middle schools one community with reinforcement
& Sprague within twenty a comparison for appropriate
(2001) miles of a city community behavior, ODRs,
 student
 perception of
 safety, &
 student reports
 of harassment.

Shapiro, middle and Pre- and Knowledge of
Burgoon, elementary urban post-experimental psychosocial
Welker, & public schools program assessment skills,
Clough (2002) with comparison attitudes
 towards guns,
 aggression

Sprague, Middle & Treatment ODRs, adults
Walker, Golly, elementary comparison perceptions of
White, Myers, & Suburban and analysis (nine school safety,
Shannon (2001) urban community treatment and six perceptions of
 schools comparison status of
 groups) school
 discipline, &
 student social
 skills
 knowledge

Stevens, primary and Experimental Levels of
Bourdeaudhuij, secondary pre-test/posttest bullying and
& Oost (2000) schools comparison being bullied,
 including a positive
 control group interactions
 among students

Study Intervention Measure(s) of Findings
 Fidelity

Bradshaw, Schoolwide School-wide Significant
Mitchell, & Lea Positive Evaluation reduction in
(2010) Behavior Tool; Effective ODRs and
 Interventions Behavior suspensions,
 and Supports. Support Survey improvements in
 The program standardized
 involved team test
 formation, achievement
 external scores
 behavioral
 support coach,
 definition of
 expectations for
 positive student
 behavior,
 teaching of
 expected
 behavior,
 schoolwide
 system of
 rewards for
 positive
 behavior, a
 system of
 responding to
 behavioral
 violations, data
 collection and
 analysis

Colvin & School-wide None 50% decrease in
Kameenui program called ODRs, decrease
(1993) Project PREPARE. in suspensions,
 The program is a detentions and
 proactive office
 instructional consequences
 approach to
 managing problem
 behavior on a
 school-wide
 basis.

Cook, Habib, The Comer's Implementation The program did
Phillips, School questionnaire not have
Settersten, Development completed each influence on
Shagle, & Program. It year school climate
Degirmencioglu seeks to enhance and student
(1999) academic outcomes
 outcomes by
 improving the
 social
 relationships
 and social
 climate in a
 school via
 development of a
 school
 improvement
 plan, support
 from school
 community and
 progress
 monitoring.

Gottfredson, School and Program Improvement in
Gottfredson, & classroom wide Development student reports
Hybl (1993) disciplinary Evaluation of classroom
 policy reviews, method (a form organization,
 behavior designed to classroom
 tracking system, increase the order, and rule
 classroom strength of clarity.
 organization and implementation Decrease in
 management, and ODRs
 positive
 reinforcement

Horner, Sugai, School-wide School-wide Low rates of
Smolkowski, Positive Evaluation ODRs,
Eber, Nakasato, Behavior Tool improvement in
Todd, & Support. The state reading
Esperanza program involved standard
(2009) defining
 behavioral
 expectations,
 teaching
 behavioral
 expectations,
 rewarding
 positive
 behavior,
 delivering
 predictable
 consequences for
 inappropriate
 behavior, data
 collection,
 administrator
 support,
 district
 support

Lane, Wehby, SWPBS Program. Teacher Improvements in
Robertson, & Involved 6 self-report GPA, decreases
Rogers (2007) training (checklist), in unexcused
 sessions for Direct tardiness,
 school observation by decreases in
 personnel, research disciplinary
 behavioral assistants contacts.
 expectations,
 reinforcement
 procedures

Metzler, The Effective Survey rating Increases in
Biglan, Rusby, Behavior Support the frequency proportion of
& Sprague as part of The with which students
(2001) Community program receiving
 Builders strategies had positive
 Intervention. been reinforcement,
 The program implemented decreased ODRs,
 targets lower levels of
 increasing aggression
 appropriate
 social behavior
 in school
 settings by
 defining rules
 and
 expectations,
 teaching
 expected
 behaviors,
 providing
 praise,
 monitoring
 student
 behavior,
 enforcing rules,
 evaluating
 progress

Shapiro, The Peacemakers Brief teacher Lower LS means,
Burgoon, Program. The report decrease in
Welker, & program includes questionnaire suspensions
Clough (2002) instruction and
 remediation
 components

Sprague, Effective Assessing Reduction in
Walker, Golly, Behavioral Behavioral ODRs
White, Myers, & Support Model. Support in
Shannon (2001) The program Schools
 involves checklist
 defining problem
 and appropriate
 behaviors,
 teaching
 alternative
 behaviors,
 providing
 effective
 incentives to
 encourage
 behavior change,
 monitoring of
 implementation,
 staff training,
 & evaluation of
 effectiveness

Stevens, The Flemish None Improved
Bourdeaudhuij, school-based outcomes on
& Oost (2000) anti-bullying bullying and
 intervention victimization
 program. The
 program involved
 restructuring
 the social
 environment via
 implementation
 of clear
 anti-bullying
 rules


Application of the Criteria

Tables 2 and 3 display the results for how the studies were coded.
Table 2

Whether Studies Operationally Defined Practice and
Participants, Employed Valid and Reliable Measures, and Were
Grounded in Rigorous Decisions

Author (year) Practice & Participants Research is
 are Defined with Grounded in
 Operational Precision to Rigorous Designs
 promote Replication

 Adults Children Practice Randomized Threats to
 Control Internal
 Trials Validity
 Controlled

Bradshaw, Y Y Y Y Y
Mitchell, &
Leaf (2010)

Colvin & N N Y N Y
Kameenui
(1993)

Cook, Habib, Y Y N Y Y
Phillips,
Settersten,
Shagle, &
Degirmencioglu
(1999)

Gottfredson, N NC Y N N
Gottfredson, &
Hybl (1993)

Horner, Sugai, Y Y Y Y Y
Smolkowski,
Eber, Nakasato,
Todd, &
Esperanza
(2009)

Lane, Wehby, Y Y Y N Y
Robertson, &
Rogers (2007)

Meteler, Y Y Y N N
Biglan, Rusby,
& Sprague
(2001)

Shapiro, N Y Y Y N
Burgoon,
Welker, &
Clough (2002)

Sprague, N Y Y N N
Walker, Golly,
White, Myers, &
Shannon (2001)

Stevens, NC NC Y Y N
Bourdeaudhuij,
& Oost (2000)

Author (year) Research Employs
 Valid and Reliable
 Measures

 Threats to Appropriate Outcomes Treatment
 External Data Integrity
 Validity Analysis
 Controlled Techniques

Bradshaw, Y Y Y Y
Mitchell, &
Leaf (2010)

Colvin & N NC N N
Kameenui
(1993)

Cook, Habib, Y Y Y Y
Phillips,
Settersten,
Shagle, &
Degirmencioglu
(1999)

Gottfredson, N NC Y Y
Gottfredson, &
Hybl (1993)

Horner, Sugai, Y Y Y Y
Smolkowski,
Eber, Nakasato,
Todd, &
Esperanza
(2009)

Lane, Wehby, N Y Y Y
Robertson, &
Rogers (2007)

Meteler, N N Y Y
Biglan, Rusby,
& Sprague
(2001)

Shapiro, N Y Y Y
Burgoon,
Welker, &
Clough (2002)

Sprague, N N Y Y
Walker, Golly,
White, Myers, &
Shannon (2001)

Stevens, N Y N N
Bourdeaudhuij,
& Oost (2000)
Author (year)

Table 3

How Studies Documented Effects

Author Research Documents Research Documents
(year) Experimental Effects Effects (high fidelity &
 Without Iatrogenic sustains administrative
 Outcomes support)

 Outcomes Effect Size High Administrative
 Fidelity Support

Bradshaw, Y NC Y Y
Mitchell, &
Leaf (2010)

Colvin & Y NC N Y
Kameenui
(1993)

Cook, Habib, N NC N Y
Phillips,
Settersten,
Shagle, &
Degirmencioglu
(1999)

Gottfredson, NC Attending to NC N
Gottfredson, & academic work
Hybl (1993) = 0.01
 Disrupting
 classroom
 =0.01
 Rebellious
 behavior =
 0.27

Horner, Sugai, Y ODRs = 0.03 Y Y
Smolkowski,
Eber, Nakasato,
Todd, &
Esperanza
(2009)

Lane, Wehby, Y GPA (across N NC
Robertson, & groups) =
Rogers (2007) -0.12 to
 0.56
 Unexcused
 tardiness
 (across
 groups) =
 0.10 to 0.65
 Suspensions =
 -0.38 to
 -0.04
 Disciplinary
 contacts
 (across
 groups) =
 -0.25 to
 0.52

Metzler, N ODRs = 0.90 N Y
Biglan, Rusby,
& Sprague
(2001)

Shapiro, Y Aggressive N N
Burgoon, behavior =
Welker, & -0.01;
Clough (2002) Disciplinary
 incidences =
 -0.12;
 Conflict
 mediation
 referrals =
 -0.07;
 Suspensions =
 -0.11

Sprague, Y NC N Y
Walker, Golly,
White, Myers, &
Shannon (2001)

Stevens, Y Bullying = N N
Bourdeaudhuij, -0.13;
& Oost (2000) Being bullied
 = -0.27 &
 0.00;
 Positive
 interactions
 = 0.29 &
 0.00


The practice and participants are defined with operational precision. Four of the 10 studies (i.e. Bradshaw et al., 2010; Horner et al., 2009; Lane, Wehby, Robertson, & Rogers, 2007; Metzler, Biglan, Rusby, & Sprague, 2001) defined the participants (both adults and children) and practice with precision to allow replication. Although all the other 6 studies defined the practice with precision, each of them was either unclear or did not define either the adults or children (or both).

The research is grounded in rigorous research. Three of the 10 studies (i.e. Bradshaw et al., 2010; Cook et al, 1999; Horner et al., 2009) met this criterion; all 3 met the four indicators for this criterion (i.e., they used randomized control designs, controlled for both internal and external threats to validity and used appropriate data analysis techniques). Three studies (i.e. Gottfredson, Gottfredson & Hybl, 1993; Metzler et al., 2001; Sprague et al., 2001) did not meet any one of the four indicators. The rest of the studies (n = 4) met at least one of the four indicators.

The research employs valid and reliable measures. Eight of the 10 studies employed valid and reliable measures of both the outcomes and treatment integrity. Only 2 studies did not meet this criterion (i.e. Colvin & Kameenui, 1993; Stevens, De Bourdeaudhuij & Van Oost, 2000). Student outcome measures used in the 8 studies included academic achievement, ODRs and suspensions; treatment integrity measures included the SET, Effective Behavior Support Survey, Program Development Evaluation and direct observation.

The research documents experimental effects without iatrogenic out-comes. Seven out of the 10 studies documented experimental outcomes without any iatrogenic effects. Two studies did not meet this criterion (i.e. Cook et al., 1999; Metzler et al., 2001) while one study was coded as "not clear" (i.e. Gottfredson et al., 1993). Effect sizes were computed for 6 of the studies; the other 4 did not report adequate data to allow computation of effect sizes. The effect sizes ranged from 0.00 to 0.9. The effect sizes are listed in Table 3.

The research documents effects. Only 2 studies demonstrated both high fidelity and sustained administrative support (i.e. Bradshaw et al., 2010; Horner et al, 2009). This means they had a fidelity index of at least 80% and administration involvement in data collection and decision-making as well as social validity measures reporting administrative satisfaction. The rest of the studies failed to meet either of these indicators (high fidelity and sustained administrative support) or both.

Summary. Overall, only 2 of the 10 studies (i.e. Bradshaw et al., 2010; Horner et al., 2009) met all the five criteria for the evidence base for SWPBS; however, the effect size for the Bradshaw et al. (2010) study could not be computed. The 2 studies defined the practice and participants with operational precision to promote replication, employed valid and reliable measures of student outcomes and treatment fidelity, documented experimental effects without iatrogenic effects, demonstrated high fidelity and sustained administrative support, and were grounded in rigorous research designs.

Discussion

This review was conducted to examine the evidence base behind SWPBS. The study sought to extend earlier work by Horner et al. (2010), who examined the evidence behind SWPBS studies published between 1990 and July 2011. Horner et al. (2010) used a five-criteria method which they applied to the body of SWPBS research. Although the five criteria "apply primarily to the consideration of individual studies" (Homer et al., 2010, p. 6), Horner et al. (2010) applied it to the body of SWPBS in concert. This study differs from that previous work in that it applies the five criteria to individual studies published over the past 20 years (1990-July 2011).

Overall, only 2 of the 10 studies that were analyzed met all the five criteria (i.e., Bradshaw et al., 2010; Horner et al., 2009). This finding is consistent with the findings of an earlier study by Lane et al. (2006) who, in examining methodological considerations of SW-PBS studies published between 1990 and 2005, found that many of the studies they reviewed lacked methodological rigor.

Although most studies (n = 9) defined the independent variable with operational precision that would allow replication, it is of concern that only half of the studies defined both the participants and practice with operational precision. Failure to define the participants limits the extent to which the findings can be generalized. It is particularly important for future studies to operationally describe both participating adults and children because, in addition to promoting external validity, knowing the teacher characteristics would help to identify factors that predict student outcomes related to SWPBS (Lane et al., 2006).

It is encouraging that most studies (n = 8) in this review used a combination of measures such as academic achievement and ODRs (e.g., Bradshaw et al., 2010; Horner et al., 2009; Lane et al., 2007) to evaluate outcomes. Lane et al. (2006) lamented the limited attention that is given to the selection of measures to evaluate intervention outcomes. According to Lane et al. (2006) "too often studies in school-wide primary interventions have relied exclusively on one outcome measure" (p. 192). Specifically, ODRs have been the most preferred measure of SWPBS outcomes in schools (Lassen et al., 2006). However, "this metric does not yield a complete picture of how well a school is functioning ... [and it does] not capture other important factors that PBS seeks to impact" (Lassen et al., 2006, p.701). Other researchers have found the use of this metric as a screening tool to have a high degree of error (Morgan-D'Atrio et al., 1996; Nelson, Benner, Reid, Epstein & Currin, 2002). Therefore, using this metric in combination with other measures such as academic achievement will yield better information about the impact of SWPBS in school settings than using it as a sole measure.

According to the results of this evaluation, most of the studies (n = 8) used acceptable measures of implementation fidelity. Previous studies found that most studies did not measure and report implementation fidelity data (e.g., Horner et al., 2010; Lane et al, 2006; Mclntyre, Gresham, DiGennaro, & Reed, 2007; Wheeler et al. 2009). For example, Mclntyre et al. (2007) examined the treatment integrity of school-based interventions in studies published between 1991 and 2005 and found that only 30% of the studies had reported treatment integrity data. Thus, it is encouraging that most of the studies in this review measured and reported data on implementation fidelity. As Lane et al. (2006) stated "the absence of treatment integrity data poses clear threats to the internal and external validity of any study" (p. 192).

However, even though it is encouraging that most of the studies in this review monitored and evaluated implementation fidelity, it is still problematic that only 2 of the 10 studies reported high fidelity. Measuring implementation fidelity is important because it ensures accurate application of the intervention (McIntyre et al., 2007) which helps to determine the "presence of a functional relationship between the dependent and independent variables (Wheeler, Baggett, Fox & Blevins, 2006, p.45). It is therefore, crucial for researchers to ensure that SWPBS interventions are implemented as intended as doing so would ensure reliable evaluation of outcomes.

Further, this evaluation indicated only 3 of the 10 studies were grounded in rigorous designs. This finding is also consistent with the findings of Lane et al. (2006) who reported that SWPBS studies published between 1990 and 2005 were predominantly descriptive and non-experimental. However, it is encouraging to note that 2 of the 3 studies (i.e., Bradshaw et al., 2010 and Horner et al., 2009) were published within the last 2 years. This development may be an indication that researchers are responding to earlier calls to address this limitation (e.g., by Kern & Manz, 2004; Lane et al., 2006). Thus, there is need for more experimental studies in order to bolster the evidence base for SWPBS.

Horner et al. (2010) reported no negative effects had been reported as a result of the implementation of SWPBS. Results of this study corroborate that finding as 7 of the 10 studies reported positive outcomes with no iatrogenic outcomes. Nevertheless, 2 studies did not show positive effects while the impact of the outcomes for 2 of the studies was "not clear." Two of the studies (i.e., Lane et al., 2007; Metzler et al., 2001) had large effect sizes of 0.9 and 0.5 (according to Cohen, 1988) respectively for at least one of the outcomes reported. The effect sizes for the other studies fell in the small to medium range (0.1 to 0.3) (Cohen, 1988) for at least one of the outcomes reported. It is important that, to date, SWPBS is mostly associated with positive student outcomes. Even though the SWPBS research is weak (Lane et al., 2006), the practice has demonstrated a level of potency that is important in promoting its widespread use in schools especially when the level of problem behavior in schools reaches alarming proportions (Safran & Oswald, 2003).

The sustainability of effects is important since it may take a few years to implement SWPBS effectively. Consistent administrative support has been identified as a key factor in promoting sustained implementation of SWPBS (Handler et al, 2007). It is therefore important that most of the studies (n = 6) in this review demonstrated sustained administrative support.

Finally, the most striking observation was that only 2 of the 10 studies met all the five criteria for the evidence-base for SWPBS. This finding appears to dampen the conclusion of Horner et al. (2010) that SWPBS "carries sufficient experimental documentation to be classified as evidence based ..." (p. 10). In fact results of this review, like those of Lane et al. (2006), suggest that SWPBS is quite a promising approach which, requires more inquiry with enhanced methodological rigor to be established as evidence based. Researchers should pay particular attention to the operational definitions of participants (both children and adults), use of experimental designs to examine the effects of SWPBS on student outcomes, and the evaluation of treatment fidelity. It is anticipated that addressing these issues will help to solidify the quite promising evidence base for SWPBS.

This study was an attempt to extend previous efforts aimed at examining the evidence base for SWPBS. However, the study is not without limitations. Horner et al.'s (2010) criteria use broad categories which made formulation of clear definitions for scoring difficult. For example, these researchers found it difficult to come up with a clear definition for "consistent administrative support". These challenges are evidenced by the moderate inter-rater agreement of 84.5%. As a result, replication of the study can be difficult given the lack of precision for some of the definitions.

Nevertheless, in spite of these limitations results of this review demonstrated that although there is evidence pointing to its efficacy, the research behind SWPBS is still weak. Many studies demonstrating the effective applications of SWPBS do not have the methodological rigor for evidence-based practices. This may be due to the applied nature of SWPBS. However, it is quite a promising approach which requires researchers to address these limitations in order for it to be classified as evidence-based. Future research on SWPBS should pay particular attention to issues like the rigor of the research designs used, treatment integrity, operational definition of participants and practice, and use of valid and reliable measures of student outcomes.

References

References marked with an asterisk indicate studies included in the review.

* Bradshaw, C. P., Mitchell, M. M., & Leaf, P. J. (2010). Examining the effects of schoolwide positive behavioral interventions and supports on student outcomes: Results from a randomized controlled effectiveness trial in elementary schools. Journal of Positive Behavior Interventions, 12, 133-148. doi:10.1177/1098300709334798

Bradshaw, C. P., Reinke, W. M, Brown, L. D., Bevans, K. B. & Leaf, P.J. (2008). Implementation of school-wide positive behavioral interventions and supports (PBIS) in elementary schools: Observations from a randomized trial. Education and Treatment of Children, 32(1), 1-26. Retrieved from http://www.education-andtreatmentofchildren.net/

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.

* Colvin, G., & Kameenui, E. J. (1993). Reconceptualizing behavior management and school-wide discipline and general education. Education and Treatment of Children, 16,361-382. Retrieved from http://www.educationandtreatmentofchildren.net/

* Cook, T. D., Habib, F., Phillips, M., Settersten, R. A., Shagle, S. C, & Degirrnencioglu, S. M. (1999). Comer's school development program in Prince George's County, Maryland: A theory-based evaluation. American Educational Research Journal 36, 543-597. doi:10.3102/00028312036003543

Dwyer, K. P., Osher, D., & Warger, W. (1998). Early warning timely response: A guide to safe schools. Washington, DC: U.S. Department of Education.

* Gottfredson, D. C, Gottfredson, G. D., & Hybl, L. G. (1993). Managing adolescent behavior a multiyear, multischool study. American Educational Research Journal, 30, 179-215. doi:10.3102/00028312030001179

Handler, M. W., Ray, J., Connell, J., Thier, K., Feinberg, A., & Putnam, R. (2007). Practical considerations in creating school-wide positive behavior support in public schools. Psychology in the Schools, 44, 29-38. doi:10.1002/pits

Horner, R. H., & Sugai, G. (2000). School-wide positive behavior support: An emerging initiative. Journal of Positive Behavior Interventions, 2, 231-232. Retrieved from http://pbi.sagepub.com/

Horner, R. H., Sugai, G., & Anderson, C. M. (2010). Examining the evidence base for school-wide positive behavior support. Focus on Exceptional Children, 42(8), 1-14. Retrieved from http://vnweb.hwwilsonweb.com/hww/Journals/getIssues.jhtml?sid-HWW:EDI&id=03212

* Horner, R. R, Sugai, G., Smolkowski, K., Eber, L., Nakasato, J., Todd, A. W., & Esperanza, J. (2009). A randomized, wait-list controlled effectiveness trial assessing school-wide positive behavior support in elementary schools. Journal of Positive Behavior Interventions, 11,133-144. doi:10.1177/1098300709332067

Kern, L., & Manz, P. (2004). A look at current validity issues of school-wide behavior support. Behavioral Disorders, 30, 47-59.

Lane, K. L., Robertson, E. J., & Graham-Bailey, M. A. L. (2006). An examination of school-wide interventions with primary level efforts conducted in secondary schools: Methodological considerations, applications of research methodology. Advances in Learning and Behavioral Disabilities, 19, 157-199. doi:10.1016/S0735-004X(06)19007-2

* Lane, K. L., Wehby, J. H., Robertson, E. J., & Rogers, L. A. (2007). How do different types of high school students respond to school-wide positive behavior support programs? Characteristics and responsiveness of teacher identified students. Journal of Emotional and Behavioral Disorders, 15, 3-20. Retrieved from http://ebx.sagepub.com/

Lassen, S. R., Steele, M. M., & Sailor, W. (2006). The relationship between positive behavior support to academic achievement in an urban middle school. Psychology in the Schools. 43(6), 701-712. doi:10.1002/pits.20177

Mclntyre, L. L., Gresham, F. M., DiGennaro, F. D., & Reed, D. D. (2007). Treatment integrity of school-based interventions with children in the Journal of Applied Behavior Analysis 1991-2005. Journal of Applied Behavior Analysis, 40, 659-672. doi:10.1901/jaba.2007.659-672

* Metzler, C. W., Biglan, A., Rusby, J. C., & Sprague, J. R. (2001). Evaluation of a comprehensive behavior management program to improve school-wide positive behavior support. Education and Treatment of Children, 24(4), 448-479. Retrieved from http://www.educationandtreatmentofchildren.net/

Morgan-D'Atrio, G, Northup, J., LaFleur, L., & Spera, S. (1996). Toward prescriptive alternatives to suspensions: A preliminary evaluation. Behavioral Disorders, 21,190-200.

Nelson, J. R., Benner, G. J., Reid, R. C., Epstein, M. H., & Currin, D. (2002). The convergent validity of office discipline referrals with the CBCL-TRF. Journal of Emotional and Behavioral Disorders, 10, 181-188. Retrieved from http://ebx.sagepub.com/content/10/3/181

Safran, S. P., & Oswald, K. (2003). Positive behavior supports: Can schools reshape disciplinary practices? Exceptional Children, 69, 361-373. Retrieved from http://www.cec.sped.org/Content/NavigationMenu/Publications2/ExceptionalChil-dren/default.htm

* Shapiro, J. P., Burgoon, J. D., Welker, C. J., & Clough, J. B. (2002). Evaluation of the peacemakers program: School-based violence prevention for students in grades four through eight. Psychology in the Schools, 39, 87-100. doi:10.1002/pits.l0040

Scheuerman, B. K., & Hall, J. A. (2012). Positive behavior supports for the classroom. Boston, MA: Pearson.

* Sprague, J., Walker, H., Golly, A., White, K., Myers, D. R., & Shannon, T. (2001). Translating research into effective practice: The effects of a universal staff and student intervention on indicators of discipline and school safety. Education and Treatment of Children, 24, 495-511. Retrieved from http://www.education-andtreatmentofchildren.net/

* Stevens, V., De Bourdeaudhuij, I., & Van Oost, P. (2000). Bullying in Flemish schools: An evaluation of anti-bullying intervention in primary and secondary schools. British Journal of Educational Psychology, 70,195-210. doi: 10.1348/000709900158056

Sugai, G., & Horner, R. (2002). The evolution of discipline practices: School-wide positive behavior supports. Child & Family Behavior Therapy, 24, 23-50. Retrieved from http://www.informa-world.com/smpp/title~content=t792303978~link=cover

Sugai, G., Horner, R. H., Dunlap, G., Hieneman, M., Lewis, T. J., Nelson, C. M., ... Ruef, M. (2000). Applying positive behavior support and functional behavioral assessment in schools. Journal of Positive Behavior Interventions, 2, 131-143. Retrieved from http://pbi.sagepub.com/

Taylor-Greene, S., Brown, D., Nelson, L., Longton, J., Gassman, T., Cohen, J., ... Hall, S. (1997). School-wide behavioral support: Starting the year off right. Journal of Behavioral Education, 7, 99-112. Retrieved from http://www.metapress.com/content/105719/?p=ef9622e70c3e4cf092685b679d8fb67c&pi=0

Taylor-Greene, S. J., & Kartub, D. T. (2000). Durable implementation of school-wide behavior support: The high five program. Journal of Positive Behavior Interventions, 2, 233-235. Retrieved from http://pbi.sagepub.com/

Wheeler, J. J., Baggett, B. A., Fox, J., & Blevins, L. (2006). Treatment integrity: A review of intervention studies conducted with children with autism. Focus on Autism and Other Developmental Disabilities, 21(1), 45-54.

Wheeler, J. J., Mayton, M. R., Carter, S. L., Chitiyo, M, Menendez, A. L., & Huang, A. (2009). An assessment of treatment integrity in behavioral intervention studies conducted with persons with mental retardation. Education and Training in Developmental Disabilities, 44, 187-195. Retrieved from http://www.dddcec.org/publications.htm#ETDD

Morgan Chitiyo and Michael E. May Southern Illinois University Carbondale

George Chitiyo Tennessee Technological University

Correspondence to Morgan Chitiyo, Southern Illinois University Carbondale, Educational Psychology & Special Education, 625 Wham Drive, Carbondale IL 62901-4618. mchitiyo@siu.edu.
COPYRIGHT 2012 West Virginia University Press, University of West Virginia
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2012 Gale, Cengage Learning. All rights reserved.

 Reader Opinion

Title:

Comment:



 

Article Details
Printer friendly Cite/link Email Feedback
Author:Chitiyo, Morgan; May, Michael E.; Chitiyo, George
Publication:Education & Treatment of Children
Article Type:Clinical report
Date:Feb 1, 2012
Words:6359
Previous Article:An examination of typical classroom context and instruction for students with and without behavioral disorders.
Next Article:A pilot study of a problem-solving model for team decision making.
Topics:

Terms of use | Copyright © 2015 Farlex, Inc. | Feedback | For webmasters