The impact of intensive positive behavioral supports on the behavioral functioning of students with emotional disturbance: how much does fidelity matter?
The task of addressing emotional and behavioral problems within school settings is complex. In the general school population, it is estimated approximately 20% of children and adolescents have serious emotional and behavioral problems (Hoagwood & Erwin, 1997; Department of Health and Human Services, 1999). These students present behaviors that range in both severity and kind, requiring a broad continuum of responses by school personnel in order to adequately address their behavioral needs. Many schools have adopted a three-tiered model of positive behavioral interventions and supports (PBIS) as a manner in which to decrease the occurrence of a range of challenging behaviors and to promote positive learning outcomes (Walker et al., 1996). The PBIS approach includes three tiers. Tier 1 implements primary prevention efforts that address school and classroom-wide universal interventions to support positive behavior for all students, Tier 2 implements secondary targeted interventions for use with students with at-risk behaviors, and Tier 3 implements tertiary or intensive individualized interventions designed for students with the most severe behavioral issues (Sugai & Horner, 2002).
School-wide PBIS is an evidence-based prevention model used to reduce challenging behavior and promote safe and healthy learning environments in schools (Sugai & Horner, 2002; McCurdy, Mannella, & Eldridge, 2003). Results of an analysis of the barriers and facilitators to the successful implementation of school-wide PBIS indicate that staff buy-in is one of the primary barriers for school-level implementation (Kincaid, Childs, Blase, & Wallace, 2007). The need for effective staff training has also been highlighted as a potential barrier or facilitator to successful implementation of PBIS. Staff buy-in and staff training are necessary but not sufficient indicators of successful PBIS implementation. Perhaps the most important variable to achieving positive and sustainable outcomes associated with PBIS is building the capacity of educators to implement evidence-based interventions within the PBIS model with fidelity. Sugai and Horner (2008), for example, refer to the importance of fidelity when implementing PBIS, and state that "schools are able to implement SWPBS [PBIS] with fidelity when supported by local trainers and coaches" (p. 71). Fidelity of implementation is traditionally defined as the extent to which the intervention is implemented as designed during an experimental study (e.g., Hord, Rutherford, Huling-Austin, & Hall, 1987). More specifically, fidelity of implementation can be differentiated into two primary categories: a) fidelity of structure (i.e., adherence and exposure), and b) fidelity of process (i.e., program differentiation, quality of delivery and responsiveness) (Dane & Schneider, 1998; Mowbray, Holter, Teague, & Bybee, 2003). Although widely reported in the health literature, the influence of fidelity of implementation is rarely reported in studies of K-12 core and intervention programs (O'Donnell, 2008).
For the majority of general educators, fidelity of the process of PBIS implementation will be most critical at the primary prevention level of intervention, including school-wide, classroom, and non-classroom settings, as the majority of students encountered will not be at risk or have an elevated risk of antisocial behavior (Walker, et al., 1996). In contrast, teachers of students with the most challenging behaviors (e.g., emotional disturbance), or displaying signs of antisocial behavior patterns, need to carry out the process of PBIS with fidelity comprehensively, or at all three levels of prevention. These levels of prevention include the structure and process of functional behavioral assessments, data-based decision making, individualized response to intervention systems, de-escalation of behavior events, linking community supports to families and youth needs, self-monitoring, and behavioral measurement (e.g., observation, rating scales) (Walker, et al.). Moreover, such students requiring intensive individualized behavioral supports may receive commercially available, scientifically-based approaches [e.g., The Incredible Years: Parents, Teachers, and Children Training Series (Webster-Stratton, 2001)] that must be delivered with fidelity to accomplish treatment goals. Thus, it may be concluded that the capacity of teachers to implement PBIS with fidelity appears to be important to positive social/emotional and academic outcomes of students requiring Tier III behavioral supports. We hypothesize that fidelity to PBIS delivered to students requiring intensive and individualized behavioral supports, particularly those receiving services under the special education category of emotional disturbance (ED), plays a statistically significant impact on the behavioral functioning of this population. Stated differently, we postulate that it will take fidelity of structure and process of PBIS to improve the responsiveness of students with ED and improve their social/emotional outcomes.
Students with ED, by definition of antisocial behavior, require intensive (i.e., Tier 3) intervention efforts in their educational programming (Walker, et al., 1996). However, improving behavioral functioning outcomes for these students has proven difficult for a number of reasons. First, the number of students provided special education services under the category of ED has grown rapidly over the last decade with nearly 500,000 students identified as of 2002 (U.S. Department of Education, 2002a; 2002b). Unfortunately, parallel growth has not occurred in the number of highly qualified special education teachers prepared to meet the needs of students with ED (Katsiyannis, Zhang, & Conroy, 2003). This special education teacher shortage has led school districts to increase their reliance on hiring teachers with emergency licensures and those prepared through alternative certification routes to fill vacancies in special education classrooms (Katisyannis, Zhang, & Conroy). Although this practice occurs across special education categories, it occurs disproportionately for positions serving student populations with ED (Office of Special Education Programs 2002a; 2002b). The end result is that many teachers in ED classrooms may not be adequately prepared to carry out the demands of planning and providing intensive behavioral interventions with fidelity. Second, although students with ED share the characteristics of marked and long standing problem behaviors, the specific nature of their behavioral issues can vary widely (e.g., physical aggression, impulsivity, depression, anxiety). This range of emotional and behavioral problems requires professionals to be well versed across a wide variety of intensive instructional and behavioral approaches, and coordinate implementation of scientifically-and evidence-based behavioral approaches with fidelity. Even teachers meeting highly qualified standards of preparation for working with students with special educational needs may not possess this wide array and depth of key skills (U.S. Department of Education, 2002b; Office of Special Education Programs, 2002b). Third, over 50% of students with ED are educated in a separate, self-contained setting (Office of Special Education Rehabilitative Services, 2002). So although PBIS is designed to be a school-wide model, often students in self-contained settings receive only intensive tertiary-level interventions (Hawken & O'Neill, 2006). Taken together, these challenges place school administrators in the likely position of having to provide additional training and support to their teachers of students with ED. Thus, the role of professional development is of critical importance to build the capacity of special education teachers to meet the needs of students with emotional and behavioral disabilities.
In the present study we utilized a pre-post naturalistic design to examine whether or not an intensive and sustained model of professional development in PBIS for teachers of students with ED would have a meaningful effect on the behavioral functioning of their students. We also explored whether teacher fidelity of PBIS implementation enhanced or constrained student changes in behavioral functioning (O'Donnell, 2008). In this context, the two purposes of the current study were to: 1) investigate the correlation of implementing PBIS and the behavioral functioning of students with ED served in self-contained settings; and 2) examine the extent to which teacher fidelity of PBIS implementation enhances or constrains student behavioral functioning.
Student participants. Thirty-seven public school students (29 males and 8 females) receiving special education services for ED in an urban, northwestern city participated in this study. The participant sample was drawn from self-contained classrooms serving students with ED across five different placements: one elementary school (n = 7), two middle schools (n = 12), one high school (n = 4), and one separate day school for 1st to 12th grade students (n = 14). In order to ensure that the teachers had a thorough knowledge of student strengths and problem behaviors, only students who had attended the self-contained classrooms for at least two months were included in the study. No students were excluded from the study due to this reason, however. Ethnic breakdowns were 65% Caucasian (n = 24), 30% African-American (n = 11), and 5% Hispanic (n = 2). Ages of students ranged from 7 to 16 years, with a mean of 13.2 (SD = 2.8). There were not statistically significant outcome differences across the five placements.
Teacher participants. Eight teachers of participating students completed ratings of student social and emotional problem behaviors. The numbers of teacher participants at the elementary, middle, high, and separate day school were 1 (13%), 2 (25%), 1 (13%), and 4 (50%), respectively. At the beginning of the study, all schools of participating teachers had been implementing PBIS for at least two years and continued implementation over the year of the present study, but with varying degrees of fidelity. Teacher fidelity to intensive PBIS practices (detailed below) ranged from 44 to 100%, with an average of 80% (SD = 19.9%). All of the teachers were female. The number of years teaching students with ED ranged from 1 to 19, with an average of 6.6 years (SD = 6.8). Teacher caseloads ranged from 6 to 20, with a mean of 11.5 (SD = 3.6) students. Caseload represents the number of students taught throughout the day, and does not necessarily equal class size. The number of participating students per teacher ranged from 2 to 7, with a mean of 4.6 (SD = 1.9).
All participating teachers held special education teaching endorsements. However, given that preparation requirements to teach in special education settings in Washington State are non-categorical, teachers are considered qualified to work with students with ED when they hold an endorsement in the general area of special education. The majority of teacher training programs in Washington State prepare educators to work with students with learning disabilities, rather than those with ED. Although some teachers of these students have coursework specific to this population, none of the teachers in the present investigation held categorical certification in the area of ED.
Trainers. The two trainers in this study were university level researchers, and are the two lead authors of the present study, with expertise in positive behavioral interventions and supports and emotional disturbance. The trainers had a combined 45 years of expertise in PBIS and designing supports to meet the academic and behavioral needs of youth with ED. They had successfully implemented and sustained PBIS systems in many professional roles over their careers as special educators serving students with ED, technical assistance providers to schools and mental health agencies, university instructors, and researchers. Given their expertise related to PBIS and ED, they received funding from the state of Washington Office of the Superintendent of Public Instruction (OSPI) to develop a model for building the capacity of teachers of students with ED to implement and sustain PBIS. They planned and conducted a training model in PBIS with the teacher participants over one school year. These trainers also served as observers and evaluators of teacher fidelity of PBIS implementation.
The Child Behavior Checklist: Teacher's Report Form (TRF; Achenbach, 2001) was used to measure the behavioral functioning of participants. The TRF consists of 113 problem items, such as difficulty following directions, disturbing other pupils, and disrupting class discipline. The teacher rates the child on each item and indicates the severity of the problem on a three-point Likert-type scale ranging from 0 (Not True) to 2 (Very True or Often True). The TRF scoring profile provides a total scale score (Total Problems), two broad-band scale scores (Internalizing and Externalizing), and eight syndrome subscale scores (Withdrawn, Somatic Complaints, Anxious/Depressed, Social Problems, Thought Problems, Attention Problems, Rule -Breaking Behavior, and Aggressive Behavior). The broad-band Internalizing scale score is based on the sum of the Withdrawn, Somatic Complaints, and Anxious/Depressed scale scores. The broad-band Externalizing scale score is based on the Rule -Breaking Behavior and Aggressive Behavior scale scores. The Social Problems, Thought Problems, and Attention Problems syndrome subscale scores are not included on either the broad-band Internalizing or Externalizing scale scores. The TRF test-retest and internal consistency values for the broad and syndrome scales are reported in the test manual as ranging from .62 to .96 and .72 to .95, respectively (Achenbach). In the present study, internal consistency of the scales was satisfactory with Cronbach's alpha noted as .76 for Internalizing, .84 for Externalizing, and .83 for Total Problems. Special education teachers serving students with ED in self-contained classrooms completed the TRF for each participating student in December 2005 and June of 2006. Teachers did not complete TRF protocols for students whom they had known for less than two months. A two-hour training session familiarized teachers with standardized assessment of student behavior and the structure (i.e., item formats) and specific instructions for completing the measure. Teachers were given two weeks to complete the TRF. Each student was rated independently by their own teacher who had received the PBIS training, and interobserver agreement was not assessed. A research assistant entered the scores into the computerized scoring program for the TRF. This program requires verification of every item through a double entry system, resulting in 100% agreement between the protocol and the final data entry confirmation.
A modified version of the Teacher Knowledge and Skills Survey (TKSS; Cheney, Walker, & Blum, 2004) was used to ascertain fidelity of implementation related to PBIS. The TKSS consisted of 25 items with a 5 point response scale. The TKSS was created as a self-report form; however for this study it was modified to make it an observational form. Item 9, for example, originally said, "I know what functional behavioral assessments are and how they are used to develop behavior intervention plans for students," and was changed to say: "Teacher knows what functional behavioral assessments are and how they are used to develop behavior intervention plans for students." The TKSS targets five teacher actions critical to strong implementation of PBIS with students with ED: 1) Specialized Behavior Support Strategies, 2) Behavior Screening Methods, Behavior Support Services, and Evaluation, 3) School-wide Discipline Process, 4) Individualized Curriculum and Modifications Supporting Students, and 5) Positive Classroom Environment. The first teacher skill (i.e., Specialized Behavior Support Strategies) focused on teacher skills related to students requiring intensive PBIS, including functional behavioral assessments, data-based decision making for behavior programs, de-escalation of behavior problems, outside services to link to families, and self-monitoring strategies. The second teacher skill (i.e., Behavior Screening Methods, Behavior Support Services, and Evaluation) measured knowledge of how to screen students for behavior problems, access pre-referral teams, counseling services and data-based decision-making. The third TKSS teacher skill area (i.e., School-wide Discipline Process) measures knowledge of the school-wide PBIS discipline policies, referral process, school-wide discipline team function, and goals and objectives. The fourth TKSS teacher skill area (i.e., Individualized Curriculum and Modifications Supporting Students) measures skills in modifying the curriculum, crisis support, collaboration with IEP teams, evaluation of individualized programs, and providing support for cultural or gender differences. The final TKSS teacher skill area (i.e., Positive Classroom Environment) measures skills necessary to provide a positive classroom environment (i.e., teaching and reinforcing expectations, social skills, prompts, and cues to remind students of behavioral expectations).
University trainers independently rated teacher level of mastery of knowledge and skill from one (none or little) to five (mastery of knowledge or skill) for each item. The trainer would then circle the appropriate number corresponding to teacher level of mastery with this item. Ratings were based on four classroom observations and notes from 20 training sessions with participating teachers over a one-year span. Independent TKSS ratings were conducted by the trainers in June 2006, after teachers had received all 40 hours of professional development and the trainers had conducted four observations of each teacher. The four observations were spaced throughout the time period between December 2005 and June 2006. Pearson Product Moment Correlations were used to estimate inter-rater reliability of the TKSS rating scores between the two PBIS trainers. The inter-rater agreement coefficient for the overall TKSS score was .94. The inter-rater reliability coefficients for the "Specialized Behavior Support Strategies," "Behavior Screening Methods, Behavior Support Services, and Evaluation," "School-wide Discipline Process," "Individualized Curriculum and Modifications Supporting Students," and "Positive Classroom Environment" teacher skills were .94, .96, 90, .93, and .92, respectively. Although strong inter-rater agreement was found, item disagreements were resolved through discussion and sharing evidence for ratings. After discussion and evidence sharing on item disagreements, raters reached consensus on the most accurate teacher rating on the individual item.
Training in Positive Behavioral Interventions and Supports. The present investigation represents a one-year professional development training project to increase the capacity of special education teachers to implement PBIS in self-contained programs for students with ED. Development of the training content was guided by three key sources. First, a needs assessment survey was conducted in December 2005 with participating special education teachers who served students with EBD in self-contained settings. The needs assessment was only used to guide instruction and was not conducted as a post-test. The survey was comprised of 10 items critical to the design and implementation of positive behavioral supports. Teachers rated their level of prior training on each item by indicating whether they had "no training," "some training," "sufficient training," or "extensive training." Of the 8 participating teachers, all described their level of training as less than "sufficient" in at least one of 8 important areas. These areas included designing effective reinforcers and consequences, conducting functional behavioral assessments, creating behavior intervention plans, teaching replacement behaviors, connecting individualized education plans (IEP) to positive behavior intervention plans (BIP), designing aversive therapy plans, conducting manifestation of determination reviews, and progress monitoring. Second, passage of recent legislation (e.g., Individuals with Disabilities Education Improvement Act of 2004; Elementary and Secondary Education Act--No Child Left Behind legislation of 2002) underscores the need to implement positive behavioral interventions and supports, conduct effective functional behavioral assessments, monitor progress, report progress to parents, and make adjustments to intervention to address any unexpected lack of progress. Third, the importance of building capacity for teachers of students with ED in the areas of PBIS, functional behavioral assessment, and progress monitoring has been highlighted in the professional literature (Katsiyannis, Landrum, Bullock, & Vinton, 1997; U.S. Department of
All eight self-contained ED teachers received the 20 training sessions together. The scope and sequence for the 20 sessions is provided in Appendix A. Training began with positive classroom management practices including the teaching of behavioral expectations and replacement behaviors (see Appendix A). Following this, training topics included assessing functions of behavior, developing positive behavior intervention plans that were explicitly linked to the functions of behavior, identifying and implementing research-based practices in the area of PBIS, and designing data collection procedures to inform instructional decisions. Training began in December 2005 and was carried out over the remainder of the school year, ending in early June 2006. The training consisted of 20 two-hour sessions presented by two university faculty members with expertise in the field of emotional and behavioral disorders (discussed previously). Each training session was comprised of three components. The first component was in lecture format and was a review of previously covered concepts. The second component was in lecture format and was the introduction of the new material. The third component was guided practice format, which was conducted in groups in which the teachers practiced the skills they had just learned. The mastery of the material was measured through TKSS scores, which were conducted by the two trainers during four classroom visitations. Teachers were paid a district-negotiated rate of pay for extra work for the 40 hours of training. In addition, teachers received payment for up to eight hours of extra pay for time spent beyond the workday to complete measures requested by the university trainers.
Non-parametric paired samples statistical analyses were used to analyze pre- and post-test TRF scores. Non-parametric tests were chosen due to the small sample size. Pearson product correlations were then utilized to determine any significant relationships between TRF change scores and TKSS scores. Finally, ANCOVAs were conducted to determine the influence of teacher knowledge on decreases of student problem behavior.
Two analyses were conducted to examine the impact of PBIS on the behavioral functioning of students with ED served in self-contained settings (Purpose 1). First, we conducted a non-parametric paired samples statistical analysis, the Wilcoxon Signed Ranks Test on the pre- and post-test TRF Scores of participants. The Wilcoxon Signed Ranks Test is more appropriate than paired samples t-tests in cases of unequal or small sample sizes, non-normal distributions, and unequal variances (Siegel & Castellan, 1988). Pre- and post-test TRF scores and Wilcoxon Signed Ranks Test Z scores are found in Table 1. Inspection of Table 1 reveals that statistically significant reductions on the TRF Externalizing Problems (p = .028) and Total Problems (p = .004) broadband scores were found from pre- to post-test. Further, statistically significant reductions were found on the Thought Problems (p = .034), Attention Problems (p = .003), and Aggressive Behavior (p = .014) syndrome scores from pre- to post-test.
Second, analyses were conducted to determine the percentage of students with ED (N = 37) who met normative criteria for clinically significant internalizing, externalizing, and total behavioral problems at pre-and post-test. The established Child Behavior Checklist: Teacher's Report Form criterion used to determine clinically significant behavioral problems was students' internalizing, externalizing, or total problems t-scores at or above 63. The percentage of students with ED who met this criterion prior to the intervention was compared to the percentage meeting the criterion after the intervention. Pre- and posttest differences were then compared. Statistically significant differences ([chi square] = 15.9, 1, N = 37,p < .01) were found in the percentages of students with clinically significant internalizing behavior problems at pre-test (n = 20, 54%) and post-test (n = 17, 46%). Similarly, statistically significant differences ([CHI SQUARE] = 12.6, 1, N = 37, p < .01) were found in the percentages of students with clinically significant externalizing behavior problems at pre-test (n = 28, 76%) and post-test (n = 25, 68%). Finally, statistically significant differences (X = 14.5, 1, N = 37, p < .01) were found in the percentages of students with clinically significant total behavior problems at pre-test (n = 28, 76%) and post-test (n = 23, 62%).
Two analyses were conducted to examine the extent to which teacher fidelity of PBIS implementation enhances or constrains student behavioral functioning (Purpose 2). First, analyses were conducted to examine the extent to which the five teacher knowledge domain scores as well as the teacher knowledge total score correlated with the change in student problem behavior. Change scores were calculated for the TRF Externalizing Problems, Internalizing Problems, and Total Problems by subtracting student pre-test from post-test scores. All 18 correlations were statistically significant (p < .05) and inverse in direction, as expected. The strength, or magnitude, of correlations was assessed using the scale developed by Cohen (1988). Correlations of .10 to .29, .30 to .49, and .50 and above were considered small, moderate, and large, respectively. As indicated in Table 2, four correlations (22%) were large and 14 (78%) moderate in magnitude. Large inverse correlations were found between overall TKSS and TRF Total Problems (r = -.52). Inspection of TKSS domains revealed that large inverse correlations were found between Behavior Screening Methods, Behavior Support Services and Evaluation (r = -.51) and School-wide Discipline Process (r = -.54) and TRF Total Problems. The magnitude of the relationships between all TKSS scores (overall and domain scores) and student changes in student Externalizing Problems were moderate, ranging from -.35 (School-wide Discipline Process) to -.42 (Behavior Screening Methods, Behavior Support Services, and Evaluation). The relationship between the TKSS School-wide Discipline Process score and student TRF Internalizing Problems change was large in magnitude (r = -.51, p < .01). The magnitude of all other relationships between all TKSS scores (overall and domain scores) and student changes in student Internalizing Problems were moderate, ranging from .33 (Individualized Curriculum and Modifications Supporting Students) to -.51 (School-wide Discipline Process). The correlation results are detailed in Table 2.
Second, analyses of covariance (ANCOVAs) were conducted for overall problem behavior to determine the influence of teacher knowledge on decreases of student problem behavior. Problem behavior was conceptualized as the TRF Externalizing Problems score and the TRF Internalizing Problems broad-band scores. The covariate measured in the ANCOVAs was the TKSS teacher knowledge total score. Although the problem behavior scores showed statistic ally significant decreases (p < .05) overall, results indicated statistically significant interaction effects of the overall TKSS teacher knowledge total score on changes in pre- to post-test problem behavior (F = 10.43, p = .003). Means and analyses of covariance interaction effects by TKSS Teacher Knowledge and Skills Survey domains and overall are presented in Table 3.
Identical ANCOVAs and post-hoc analyses were conducted to determine the interaction effects of the five teacher knowledge domain scores on the change of student problem behavior over time. The covariates measured in the ANCOVAs were the respective teacher knowledge domain scores. The results revealed statistically significant interaction effects between changes in pre- to post-test student problem behavior for all five teacher knowledge domain scores. The five teacher knowledge domain scores of Specialized Behavior Support Strategies; Behavior Screening Methods, Behavior Support Services, and Evaluation; School-wide Discipline Process; Individualized Curriculum and Modifications Supporting Students, and Positive Classroom Environment all showed statistically significant interaction effects on student problem behavior (p < .01). The results of the ANCOVAs are detailed in Table 3.
The two purposes of the current study were to: 1) investigate the impact of PBIS on the behavioral functioning of students with ED (N = 37) served in self-contained settings; and 2) examine the extent to which teacher fidelity of PBIS implementation enhances or constrains student behavioral functioning. Although an abundance of research has demonstrated PBIS as an effective school-wide model, there is a paucity of research on the effects of PBIS specifically on students in self-contained settings. Moreover, it remains unclear to what extent teacher capacity to implement PBIS with fidelity impacts student behavioral outcomes. We hypothesized that teacher fidelity to the structure and process of PBIS would reduce overall behavioral problems of students with ED served in self-contained settings.
Two main findings warrant discussion. The first finding is it appears PBIS is correlated with the behavioral functioning of students with ED served in self-contained settings. Statistically significant reductions were found in the pre- and post-test scores of students on the TRF syndrome scores of Thought Problems, Attention Problems, and Aggression. Further, statistically significant reductions in the number of students who met the criteria for clinically significant internalizing behavior problems, externalizing behavior problems, and total behavior problems were found. These data indicate that PBIS appeared to play a significant role in improving the behavioral functioning of students with ED. Although the impact of PBIS on the behavioral functioning of students with ED in self-contained settings has not yet been experimentally examined, to date, these data add to the small evidence-base for PBIS among students requiring intensive, individualized behavioral supports (Turnbull et al., 2002; Nelson, Martella, & Marchand-Martella, 2002).
The second finding that merits discussion is that teacher fidelity to the structure and process of PBIS played a significant role in reducing student problem behaviors. Indeed, results of ANCOVAs revealed a statistically significant interaction on problem behavior. As indicated in Table 3, all five teacher domains of the TKSS demonstrated statistically significant interaction effects on student problem behavior at the p < .01 level. Thus, it appears all five PBIS domains as measured by the TKSS were significantly important to improving the behavioral functioning of students with ED served in self-contained settings. When the magnitude of the relationship between teacher implementation of PBIS and student behavioral functioning was examined, a large inverse relationship (r = -.52) was found between teacher implementation of PBIS (overall TKKS score) and TRF Total Problems. Moderate to large inverse correlations were found between TRF Internalizing, Externalizing, and Total Problems and all of the five PBIS domains (see Table 2) measured by the TKSS. These data indicate that building the capacity of teachers of students with ED served in self-contained settings to implement PBIS with fidelity may play a large role in improving the responsiveness of this population to individualized and intensive behavioral interventions.
There are several implications of these findings. First, it appears that PBIS positively correlates with the behavior of students with ED in self-contained settings. Over 50% of students with ED are educated in a separate, self-contained setting (Office of Special Education Rehabilitative Services, 2002), and often receive only intensive tertiary-level interventions of PBIS (Hawken & O'Neill, 2006). Therefore, our findings underscore the importance of building the capacity of teachers of students with ED in all components of school-wide PBIS, including primary-level interventions (i.e., school-wide expectations and school-wide rewards), and secondary-level interventions (e.g., check-in, checkout system, positive adult attention) as were measured by the TKSS (Hawken & O'Neill).
Second, PBIS professional development activities and coaching of teachers of students with ED in self-contained settings appears to correlate with student behavioral functioning. Study findings demonstrate that the professional development activities were able to successfully build capacity of teachers, which resulted in improved student behavior outcomes. These findings have implications for elementary, middle, and high schools serving students with ED in self-contained settings. First, support when implementing a new program must be continued over an extended period of time. The professional development organization of PBIS components involved 20 two-hour sessions, plus in-classroom follow-up. Further, support for teachers will continue over the next school year, as it has been recommended that to successfully build capacity in a given area teacher support should continue for 12 to 15 months (National Institute of Child Health and Human Development, 2000). Second, the current shortage of well-prepared, highly qualified special education teachers, particularly those working with students with ED, makes professional development activities crucial (U. S. Department of Education, 2002b). Ongoing, high-quality training targeted to areas insufficient in teacher preparation must be conducted regularly and considered a high priority in order to increase teacher quality and therefore, student success.
This investigation was limited in several ways. The first and most debilitating limitation is the research design used to examine PBIS and related behavior outcomes, which was not experimental in nature. Researchers of future investigations should use higher quality designs including regression discontinuity, randomized experimental designs, or quasi-experimental investigations, which will allow causal inferences to be made. Second, the 37 students participating in this study were located in an urban, northwestern city. Therefore, the participants are not demographically representative of the general population and the generalizability of the findings is limited. Further, although the study encompassed students in elementary, middle, and high schools, the mean participant age of 13.2 years (SD = 2.8) emphasizes an overrepresentation of older students. Third, student behavior outcomes were measured solely in the form of teacher rating scales, specifically the TRF. Future studies should incorporate other measures of student behavior, including self-assessment, parent-assessment, and observation. Further, other common measures of PBS should also be collected, including office discipline referrals, suspensions, expulsions, etc. Fourth, we did not include schoolwide [e.g., Schoolwide Evaluation Tool, Version 2.1 (Sugai, Lewis-Palmer, Todd & Horner, 2005)] measures of PBIS implementation nor did we control for child factors (e.g., SES) that may have influenced treatment outcomes. It is probable, for example, that quality of schoolwide PBIS systems may have accounted for a significant amount of the variance in teacher fidelity to Tier III PBIS.
Our findings have implications for the challenge of moving PBIS to practice. Cook, Landrum, Tankersley, and Kauffman (2003) highlight that approaches may be rendered ineffective or counterproductive if not used with adequate dosage (amount of treatment) or when implemented without adequate fidelity. Placing this concern in context of the present investigation, despite almost a full year of training on PBIS, teachers implementing with less skill (i.e., low fidelity) tended to not experience large improvements in student behavioral functioning commensurate with their colleagues implementing with high levels of skill (i.e., high fidelity). Thus, these teachers may feel justified in concluding that either PBIS does not work very well or at all with students with ED served in self-contained settings. Limited or no consideration to fidelity of intervention is a large threat to internal validity; without consideration of level of fidelity it is difficult to ascertain whether the intervention was responsible for enhanced or constrained treatment outcomes. We draw upon and concur with the more than 30 year-old findings of Hall and Loucks (1977) who found that those implementing an innovation vary on adherence to the structure and quality of implementation of the innovation. Our findings underscore that student behavioral functioning outcomes varied significantly according to how well PBIS was delivered.
A related implication centers on the professional development needed to implement and sustain evidence-based approaches in schools. In a national study of university-based teacher education programs, Levine (2006) found that many students seem to be graduating from teacher education programs without the skills and knowledge they need to be effective teachers. He found that three out of five teacher education alumni surveyed (62%) indicated that schools of education do not prepare their graduates to cope with the realities of today's classrooms, particularly in the area of behavior management. Moreover, only 30% of principals said that their teachers are very or moderately well prepared to address the needs of students with disabilities. Cook and colleagues (2003) state it well: "Given the many pressures, time constraints, and limited resources under which teachers typically work, it is little wonder that contemporary teacher preparation programs and training results in many effective techniques being quickly abandoned, not used, or used incorrectly" (p. 352).
With the identification of specific gaps in teacher knowledge and skills regarding the provision of PBIS to students with ED, we were able to design training to target the specific needs of participating educators. The four features identified in the national study of special education personnel needs (discussed previously) provided the framework for training activities in this partnership (SPeNSE, 2002). These features guided this partnership in the following four ways. First, professional development was ongoing and included a broad range of specified topics. The needs assessment process used in the present partnership informed the design of the content of the training sessions so that multiple topics from areas identified by teachers of students with ED as needing further training were included. Second, training was immediately relevant and of high quality. Once areas of need were identified by the partnership participants, training was targeted to bring in evidenced based practices that met their current classroom challenges. Third, training time was allowed for planning and implementation. Time was allowed for the trainers to guide participants as they adapted session materials to best meet the specific needs of their unique classroom situations. Additionally, allotted time for planning and assistance ensured that those with the least amount of background training received the necessary assistance for correct implementation of the positive behavioral approaches. Fourth, opportunities for informal assistance were included. Visitations to classrooms provided an avenue for giving additional clarification regarding accuracy of implementation of training. Furthermore, informal observations were used to provide acknowledgement to teachers for a job well done.
In summary, the implementation of PBIS positively correlated with the behavior of elementary, middle, and high school students with ED in self-contained settings. Teacher fidelity to PBIS played a large and statistically significant role in improving the behavioral functioning of students with ED. Building the capacity of teachers serving self-contained students with ED to implement and sustain PBIS should involve not only workshops, but ongoing coaching and attention to fidelity of implementation. We conclude that such efforts may lead to significant reductions in the behavior problems of students with ED.
Appendix A: Scope and Sequence of 20 PBIS Training Sessions
Session 1: Getting started, overview of PBIS
Session 2: Overview of PBIS, evaluating your own capacity to implement and sustain PBIS
Session 3: Routines and Ecological Arrangements
Session 4: Expectations
Session 5: Developing a Plan for Responding to Problem Behavior
Session 6: Social Skills Instruction/Teaching replacement behaviors (Strategic and Intensive)
Session 7: Self-Management (Strategic and Intensive)
Session 8: Linking FBAs to IEPs & BIPs
Session 9: Using data to make decisions about PBS & BIPs, Helping others to make decisions
Session 10: Aversive Therapy and Manifestation Determination
Session 11: Pre-Functional Assessment Considerations, Making Sure Core Behavior Supports (e.g., routines, ecology, and expectations) are Strong
Session 12: Functional Behavioral Assessment I
Session 13: Functional Behavioral Assessment II
Session 14: Behavior Intervention Plans I
Session 15: Behavior Intervention Plans II
Session 16: Progress Monitoring: Academic
Session 17: Progress Monitoring: Behavioral
Session 18: Determining Responsiveness to Intervention
Session 19: Scientifically-Based Strategic and Intensive Behavioral Interventions
Session 20: Meeting the academic needs of students with ED
Achenbach, T. M. (2001). Teacher's Report Form for Ages 6-18. Burlington, VT: University of Vermont.
Cheney, D., Walker, B., & Blum, C. (2004). Teacher Knowledge and Skills Survey, V. 2.0. Seattle, WA: University of Washington.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd Ed.). Hillsdale, NJ: Lawrence Erlbaum.
Cook, B. G., Landrum, T. J., Tankersley, M., & Kauffman, J. K. (2003). Bringing research to bear on practice: Effecting evidence-based instruction for students with emotional or behavioral disorders. Education and Treatment of Children, 26(4), 345-361.
Dane, A. V. and Schneider, B. H. (1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review, 18, 23-45.
Department of Health and Human Services. (1999). Mental Health: A Report of the Surgeon General: Children and Mental Health (pp. 123-220). Washington, DC: Public Health Service.
Hall, G. E., & Loucks, S. F. (1977). A developmental model for determining whether the treatment is actually implemented. American Educational Research Journal, 14, 263-276.
Hawken, L. S., & O'Neill, R. E. (2006). Including students with severe disabilities in all levels of school-wide positive behavior support. Research & Practice for Persons with Severe Disabilities, 31(1), 46-53.
Hoagwood, K., & Erwin, H. (1997). Effectiveness of school-based mental health services for children: A 10-year research review. Journal of Child and Family Studies, 6(4), 435-451.
Hord, S.M., Rutherford, W.L., Huling-Austin, L., & Hall, G.E. (1987). Taking charge of change. Alexandria, VA: ASCD.
Katsiyannis, A., Landrum, T. J., Bullock, L., & Vinton, L. (1997). Certification requirements for teachers of students with emotional or behavioral disorders: A national survey. Behavioral Disorders, 22(3), 131-140.
Katsiyannis, A., Zhang, D., & Conroy, M. A. (2003). Availability of special education teachers: Trends and Issues. Remedial and Special Education, 24(4), 246-253.
Kincaid, D., Childs, K., Blase, K. A., & Wallace, F. (2007). Identifying barriers and facilitators in implementing schoolwide positive behavior support. Journal of Positive Behavior Interventions, 9, 174-184.
Levine, A. (2006). Educating school teachers. Washington, DC: The Education Schools Project.
McCurdy, B. L., Mannella, M. C., & Eldridge, N. (2003). Positive behavior support in urban schools: Can we prevent the escalation of antisocial behavior? Journal of Positive Behavior Interventions, 5, 158-170.
Mowbray, C., Holter, M. C., Teague, G. B., & Bybee, D. (2003). Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation, 24(3), 315-340.
National Institute of Child Health and Human Development. (2000). Report of the national reading panel. Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction: Reports of the subgroups (NIH Publication No. 00-4754). Washington, DC: U.S. Government Printing Office.
Nelson, J. R., Martella, R. C., & Marchand-Martella, N. E. (2002). Maximizing student learning: The effects of a comprehensive school-based program for preventing problem behaviors. Journal of Emotional and Behavioral Disorders, 10, 136-149.
O Donnell, C. L. (2008). Defining, conceptualizing, and measuring fidelity of implementation and its relationship to outcomes in K-12 curriculum intervention research. Review of Educational Research, 78(1), 33-84.
Office of Special Education Programs. (2002a). Beginning special educators: Characteristics, qualifications, and experiences (Study of Personnel Needs in Special Education summary sheet). Retrieved date from http://www.spense.org/results.html
Office of Special Education Programs. (2002b). Recruiting and retaining high quality teachers (Study of Personnel Needs in Special Education summary sheet). Retrieved date from http://www.spense.org/results.html
Office of Special Education Rehabilitative Services. (2002). Twenty-fourth Annual Report to Congress on the Implementation of the Individuals with Disabilities Education Act (IDEA). Washington, D.C.: OSERS.
Siegel, S., & Castellan, N.J. (1988). Nonparametric statistics for the behavioral sciences (2nd Ed.). New York, NY: McGraw-Hill.
SPeNSE. (July 2002). Key Findings for SPeNSE/ Study of Personnel Needs in Special Education. www.spense.org
Sugai, G., & Horner, R. H. (2002). The evolution of discipline practices: School-wide positive behavior supports. Child & Family Behavior Therapy, 24, 23-50.
Sugai, G., & Horner, R. H. (2008). What we know and need to know about preventing problem behavior in schools. Exceptionality, 16(2), 67-77.
Sugai, G., Lewis-Palmer, T., Todd, A., & Horner, R. (2005) School-wide Evaluation Tool, Version 2.1. University of Oregon, Educational and Community Supports.
Turnbull, A., Edmonson, H., Griggs, P., Wickham, D., Sailor, W., Freeman, R., Guess, D., Lassen, S., McCart, A., Park, J., Riffel, L., Turnbull, R., & Warren, J. (2002). A blueprint for schoolwide positive behavior support: Implementation of three components. Exceptional Children, 68, 377-402.
U. S. Department of Education (2002a). Implementation of the Individuals with Disabilities Education Act. Washington, DC: Author
U. S. Department of Education (2002b). The study of personnel needs in special education. Retrieved January 15, 2006, from http://www.spense.org
Walker, H. M., Horner, R. H., Sugai, G., Bullis, M., Sprague, J. R., Bricker, D., & Kaufman, M. J. (1996). Integrated approaches to preventing antisocial behavior patterns among school-age children and youth. Journal of Emotional and Behavioral Disorders, 4, 194-209.
Webster-Stratton, C. (2001). The Incredible Years: Parents, teachers, and children training series. Residential Treatment for Children and Youth, 18 (3), 31-4
This research was supported in part by a grant from United States Office of Special Education Programs (OSEP). Opinions expressed do not necessarily reflect the position of the U.S. Department of Education, and no endorsement should be inferred.
Author Contact information
Gregory J. Benner
University of Washington-Tacoma
1900 Commerce Street
Tacoma, WA 98402
Kathleen M. Beaudoin,
University of Washington-Tacoma
Tacoma, Washington, 85402.
E-mail: firstname.lastname@example.org .
University of Washington
Seattle, Washington, 98195
University of Washington
Seattle, Washington, 98195
Nicole C. Ralston
University of Washington-Tacoma
Tacoma, Washington, 85402.
Table 1 Pre- and Post-test TRF Scores and Wilcoxon Signed Ranks Test Z scores Measure Pre Post TRF Syndrome Scores Anxious/Depressed 64.2 (10.0) 62.8 (10.2) Withdrawn 62.1 (6.8) 60.2 (8.0) Somatic Complaints 57.4 (7.6) 56.1 (7.8) Social Problems 64.8 (8.8) 63.1 (8.3) Thought Problems 62.1 (10.9) 59.3 (10.4) Attention Problems 62.5 (8.2) 59.6 (7.9) Rule-Breaking Behavior 66.8 (10.8) 64.7 (10.6) Aggressive Behavior 69.7 (12.7) 66.9 (12.1) TRF Broad-Band Scores Internalizing 64.4 (7.8) 62.3 (9.7) Externalizing 68.4 (10.1) 66.1 (10.3) TRF Total Score Total Problems 67.8 (8.7) 64.4 (9.3) Measure Change Z Sig. TRF Syndrome Scores Anxious/Depressed -1.4 -1.3 .195 Withdrawn -1.9 -1.8 .067 Somatic Complaints -1.3 -1.0 .294 Social Problems -1.7 -1.2 .237 Thought Problems -2.8 -2.1 .034 * Attention Problems -2.9 -3.0 .003 ** Rule-Breaking Behavior -2.1 -1.6 .112 Aggressive Behavior -2.8 -2.5 .014 * TRF Broad-Band Scores Internalizing -2.1 -1.9 .062 Externalizing -2.3 -2.2 .028 * TRF Total Score Total Problems -3.4 -2.9 .004 ** Note. Standard deviations are in parentheses. * p < .05 and ** p < .01. Table 2 Correlations of TKSS Teacher Knowledge and Skills Survey Scores with Student TRF Externalizing, Internalizing, & Total Problems Change Scores TRF Change Scores TKSS domain/Overall Externalizing Internalizing Total Problems Problems Problems Specialized Behavior -.40 * -.35 * -.49 ** Support Strategies Behavior Screening Methods, - 42 ** -.40 * -.53 ** Behavior Support Services, and Evaluation School-wide -.35 * -.51 * -.54 ** Discipline Process Individualized Curriculum -.41 * -.33 * -.49 ** and Modifications Supporting Students Positive Classroom -.39 * -.34 * -.47 ** Environment Note. * p < .05 and ** p < .01. Table 3 Means and Analyses of Covariance Interaction Effects by TKSS Teacher Knowledge and Five Teacher Knowledge and Skills Survey Domains and Overall Interaction Effect on Problem Behavior (a) TKSS Fidelity Domain/Overall M SD F Sig. Specialized Behavior 31.8 10.2 9.31 .004 ** Support Strategies Behavior Screening Methods, 29.9 5.2 11.05 .002 ** Behavior Support Services, and Evaluation Discipline Process 16.3 1.8 9.22 .005 ** Individualized Curriculum and 18.8 7.2 9.54 .004 ** Modifications Supporting Students Positive Classroom Environment 17.5 4.1 8.37 .007 ** Total 114.1 27.1 10.43 .003 ** Note. (a) Interaction of TKSS x time x problem behavior. The construct of 'problem behavior' included TRF Internalizing Problems and TRF Externalizing Problems t-scores. ** p < .01.