Classwide PBIS for students with EBD: initial evaluation of an integrity tool.

Abstract

The purpose of this pilot project was to develop a tool and a process for providing performance feedback on evidence-based classroom management strategies to teachers of students in emotional support classrooms. The project was carried out with nine classroom teachers and descriptive results are discussed. Initial results found increases in treatment integrity after the first round of performance feedback and increases in student on-task behaviors. The evaluation procedures mirrored the screening approach taken within a response-to-intervention model (i.e., benchmark screening three times a year). The potential for "scaling-up" the techniques of assessing treatment integrity and providing performance feedback, in relation to the implementation of classroom management strategies, is discussed.

**********

Maintaining an adequate supply of highly qualified special educators is a challenge for most school districts. This concern is no more urgent than when it comes to meeting the needs of students with emotional and behavioral disorders (EBD). Although students with EBD comprise only about 8% of all students with disabilities, similar to other special education categories, that number continues to rise (U.S. Department of Education, 2006). Meeting this need requires a specifically trained workforce. Unfortunately, however, teachers of students with EBD tend to be less experienced, less credentialed, and less skilled in general instructional tasks than other special educators (Henderson, Klein, Gonzalez & Bradley, 2005).

It has been a number of years since the Peacock Hill Working Group's (1991) seminal publication defining the recommended practices for educators working in the field of EBD. Since that time, the field has faced even greater challenges such as the increased acuity-level of the student population (i.e., more violent, intractable) combined with the increased popularity of postmodern ideology (Kauffman, 2003; Walker, Zeller, Close, Webber, & Gresham, 1999). Nevertheless, despite these challenges, we have amassed a great deal of information about what works in educating students with EBD (Kauffman, 2003; Nelson, 2003). Recently, several researchers offered a compendium of research-based practices for educators of students with EBD that includes: posting, teaching and reviewing expectations; teacher praise and reinforcement; opportunities to respond during instruction; clear instructional strategies (e.g., direct instruction); maximizing structure; positive behavior support (e.g., function-based interventions and self-management); and a continuum of corrective consequences (Lewis, Hudson, Richter, & Johnson, 2004; Simonsen, Fairbanks, Briesch, Myers, & Sugai, 2008). Despite our acknowledgement of these critical research-based practices, there remains a gap in educators' capacity to deliver with a high degree of integrity.

The degree to which an intervention is carried out as intended is referred to as treatment integrity (Gresham, 1989). Treatment integrity has been identified as one of the most important aspects when considering behavior change. Prior to evaluating the impact of an intervention on student outcomes (A), it is imperative to evaluate the integrity with which the intervention has been implemented (B), because A is directly affected by B (Mortenson & Witt, 1998). With low integrity, the end result is a research-to-practice gap that interferes with achieving important educational outcomes (Fixen, Naoom, Blase, Friedman, & Wallace, 2005). A common approach to narrowing this gap is through in-service professional development combined with performance feedback (Sawka, McCurdy, & Mannella, 2002).

Performance feedback refers to a process of providing objective, data-based feedback with the aim to improve transfer or maintenance of newly acquired skills (Mortenson & Witt, 1998; Reinke, Lewis-Palmer & Martin, 2007). Many studies have documented a link between increases in treatment integrity and the addition of performance feedback to training or consultation.

Sutherland and colleagues investigated the used of performance feedback and goal setting to successfully increase a teacher's use of behavior specific praise with students with EBD (Sutherland & Wehby, 2001; Sutherland, Wehby & Copeland, 2000). Results documented a decrease in student disruptive behaviors as teacher praise increased. Similar outcomes were found in a study investigating the use of visual performance feedback on the rate of behavior specific praise provided to target students exhibiting behavior difficulties and confederate peers in the classroom (Reinke, Lewis-Palmer & Martin, 2007). Again, performance feedback resulted in higher, yet somewhat variable, rates of behavior specific praise toward target students. Finally, Noell and colleagues (2005) investigated the use of performance feedback on prereferral interventions for academic difficulties, social behavioral concerns, or both in the general education setting. The authors documented substantially superior levels of implementation under the performance feedback condition.

In summary, there is now a well established research base for the use of performance feedback in education. However, most studies evaluate performance feedback for a specific teacher behavior (e.g. behavior specific praise; Reinke et al., 2007; Sutherland & Wehby, 2001; Sutherland et al., 2000) or for monitoring the implementation of a specific individualized intervention (e.g. remedial instruction; Noell et al., 2005). Yet to be addressed is performance feedback for implementing a cluster of behaviors, such as a set of classroom management strategies. Given the reliance on brief in-service trainings, districts need efficient and effective procedures for providing performance feedback to teachers after such trainings. Viewing performance feedback in this broader, systemic context is borrowing from the fields of schoolwide positive behavior support (SWPBS) and response to intervention (RtI). Broadly speaking, both approaches are based on a three-tiered prevention and intervention logic that emphasize the need for (a) monitoring the integrity of interventions, (b) procedures for screening to evaluate the effectiveness of core/universal systems and (c) monitoring student progress at all levels (universal, secondary, tertiary) (Batsche and colleagues, 2006; Horner, Sugai, Todd & Lewis-Palmer, 2004).

The purpose of this pilot project was to develop a tool for improving the integrity of classroom management practices that could mirror an RtI approach. It was hypothesized that teachers' implementation of the classroom strategies would improve as a result of repeated performance feedback and that higher levels of implementation would be correlated with higher levels of student on-task behaviors.

Pilot Project

Participants and Setting

The project took place in a large (i.e., student enrollment of just over 20,000) suburban district in the Mid-Atlantic region of the United States. Eleven percent of the student population received special education services and the district scored sixth highest in the state on the annual statewide assessment.

Nine teachers of emotional support services participated in this pilot study. Four teachers were assigned to elementary level self-contained emotional support classrooms, and five teachers were assigned to middle school resource rooms. Six of the teachers were male, while three were female. Overall, this was a seasoned group of teachers with six teachers having over 20 years of experience, two having seven to eight years of experience, and one teacher in her first year of teaching. All teachers held a masters degree and certification in special education.

The elementary classrooms were comprised of six to eight students with one teacher and two educational assistants. These were primarily self-contained settings with 45 minute periods including lunch, recess and one non-academic subject area (e.g., art, music) each day. Curricula used in the classroom included Houghton Mifflin reading, Saxon Math, Everyday Math, and strategies to parallel the general curriculum. Middle school classrooms were instructed by one teacher and one educational assistant. Each class comprised 7 to 12 students with no more than 10 students at a time in class. The curricula used in the special education classroom parallel the general education curriculum. Most students spent one to three class periods per day in this setting.

Treatment Integrity Tool

This project was a collaboration between the school district and a local non-profit organization. The project started with a needs assessment of the district's emotional support classrooms. The needs assessment included a series of observations and interviews with staff from across the district's emotional support classrooms. One result of the needs assessment was to define a set of classroom management strategies that could be used across the emotional support classrooms in the district. Once these strategies were defined there was a need for a standardized approach for monitoring each teacher's implementation of the model.

After the needs assessment, the team identified the evidence based practices that would serve as the model for emotional support classwide behavior management strategies for the district. Pulling from existing literature, four core domains were identified: materials, classroom ecology, teaching expectations, and instructional management. Next, specific strategies were selected from the literature which aligned with the core domains. Once the essential classroom management strategies were established, a manual was drafted, teachers were trained during the beginning of the year inservice training period, and the team proceeded with developing an integrity tool. See Figure 1 for an excerpt of the tool (copies of the tool are available from the first author upon request). When developing the tool, the group decided to utilize a multimethod approach to evaluating the classroom system. Data collection techniques included permanent product review, direct observation of teacher and student behaviors, and interviews with teachers and students (see Table 1 for specific strategies and observation procedures).

[FIGURE 1 OMITTED]

Table 1 Classroom management strategies and observation procedures for
each core domain

Core Domain       Strategy                 Observation Procedure

Classroom      Teacher Praise             Direct observation
Ecology

               Contact for On-Task        Direct observation
               Behaviors

               Transitions *              Direct observation

               Schedule: Defined and      Permanent product
               Posted

               Schedule: Followed         Direct observation

Materials      Rules                      Permanent product

               Point sheets               Permanent Product

Teaching       Teacher reviews            Direct observation
Expectations   expectations

               Using point card           Permanent product

               Recognition system         Direct observation/Permanent
                                          product

               Students know              Interview
               expectations

               Students know recognition  Interview
               system

Instructional  Opportunities to respond   Direct observation
Management

               Correct academic           Direct observation
               responses

               Student on-task behavior   Direct observation

* Elementary classrooms only

Pilot Procedures

Treatment integrity. Treatment integrity was the primary outcome measure for this project. Teacher classroom management behaviors were measured using the integrity tool through direct observations, permanent product review, and interviews. Each integrity assessment took approximately 45 to 60 minutes to complete. Results from the tool were reported out as the percentage of steps completed (range = 0 to 100%).

Two of the authors completed the integrity observations. To complete the integrity check, the authors scheduled the observations during a time when the teacher was teaching a group lesson, preferable academic in nature. This was to ensure the opportunity to observe for correct academic responses and opportunities to respond. The observer would enter the classroom and briefly interview the teacher to get an understanding of the classroom expectation, recognition system, and schedule.

Integrity checks were conducted three times, in each classroom, across a calendar year (i.e., January, May, and October). The observation cycle mirrored the benchmarking schedule followed in a response-to-intervention model for student literacy. While technical adequacy of the sensitivity of the pilot tool to change over time has not been determined, the thought was that using the tool in this way would (a) mirror a familiar evaluation practice, (b) provide teachers with ongoing feedback regarding classroom management, and (c) provide district administrative staff with a preliminary structure to determine who and how to provide support to teaching staff.

Teachers were given a copy of the completed tool (i.e., written feedback) after each observation session. They also had the option to contact the observer to ask any follow-up questions or solicit further detail on the results of the observation. This served as the method for performance feedback between observation sessions.

Student behavior. Student on-task behavior served as a preliminary validation procedure for the tool. On-task behavior was defined as attending to the teacher or to assigned work, and included both active and passive forms of behaviors (e.g., raising hand, answering a question aloud, reading silently, looking at the blackboard during instruction). During the January integrity measures, nine concurrent direct observations of student behavior were collected in an effort to establish a case for concurrent validity. Whereas the treatment integrity tool includes a brief measure of student behavior (e.g., on-task behavior using 30-s interval recording), these were separate direct observations of student behavior. For these validity measures, on-task behavior was measured during a 20 minute direct observation, through momentary time-sampling on a 15 second interval recording schedule. A graduate level research assistant completed the student observations while one of the authors completed the integrity observation.

Interobserver Agreement and Treatment Acceptability

Interobserver agreement (IOA) was completed for 22% of the integrity observations during the initial round of feedback (e.g., January integrity checks). Agreement was calculated, using the frequency-ratio approach (smaller total divided by larger total multiplied by 100) (Kazdin, 1982), for the total score and each specific strategy. Mean IOA for the total score was 76.6% (range = 65 - 88%). Mean IOA scores of 83% or greater (range = 66.7 - 100%) were found for; materials, classroom ecology, behavior specific praise, behavior specific correction, using attention, transitions, and student interviews. Mean IOA scores of 73.3% or below (range = 0 - 87.5%) were found for: teaching expectations, non-specific praise, non-specific correction, opportunities to respond, correct academic responses.

Treatment acceptability was assessed for (a) the selected classroom management strategies and (b) the integrity measure/feedback process. Teachers were given the Intervention Rating Profile-15 (IRP-15; Martens, Witt, Elliott, & Darveaux, 1985) to assess the acceptability of the classroom management strategies. The 1RP-15 is a 15 item scale that has been found to be a reliable (a = .98) measure of general acceptability of an intervention (Martens et al., 1985). Three items were added to the IRP-15 to assess acceptability of the integrity measure/feedback process. The three added statements were: (1) I agree that external observation is important for evaluating my implementation of the behavioral system, (2) I feel the observation tool evaluates important components of the behavioral system, and (3) I found the scoring summary sheet to be useful in identifying my areas of strength. Three out of nine teachers completed the IRP-15 and found the manualized approach to be acceptable with an average rating of 73 (possible range = 15-90). They also found the integrity measure/feedback process to be acceptable with an average rating of 13.67 (possible range = 3 - 18).

Results

Treatment Integrity

Table 2 presents the results of the repeated measures of the integrity tool across three sessions. Total scores (percentage of steps completed) are reported for the elementary and middle school. The time-on-task data reported here is based on a brief observation that is included in the observation tool itself, and does not reflect the concurrent student observations conducted for the purpose of validation. Observation one yielded low teacher implementation of the classroom management strategies after the initial training. At the elementary level, there was steady improvement across observations. The middle school teachers showed improved implementation after the first integrity measure (i.e., between Observation 1 and Observation 2) but no growth after the second integrity measure. However, the range of implementation narrowed by the third session. Finally, as teacher implementation improved, so did student on-task behavior.

Table 2

Results from the Integrity Tool and Student On-Task Behavior Across
Three Observation Periods

                      Observation 1  Observation 2  Observation 3

Elementary
Mean                     67.5 *          74             82.2
Range                   (33-89)        (52-95)        (67-100)

Middle
Mean                     51 *            69             69
Range                   (48-54)        (38-90)        (57-81)

Student Time-On-Task
Mean                     73.8 **         86.3          85.7
Range                   (30-100)       (60-100)       (70-95)

* Total Score = Percentage of steps completed
** Percent of intervals with on-task behavior

Student Behavior

The nine concurrent observation sessions were compared to the individual domains and total score on the integrity measure. A Pearson correlation addressed the relationship between the total score, each domain score (i.e., classroom ecology, materials, teaching expectations, instructional management) and student on-task behavior. The correlation between the total score (M = 64.18, SD = 17.62) and student on-task (M = 76, SD = 13.25) behavior was found to be statistically significant, r = .673, p < .05. The correlation between the classroom ecology domain (M = 40.91, SD = 32.9) and student on-task behavior (M = 76, SD = 13.25) was found to be statistically significant, r = .784, p < .01, while the correlation between the three other domains (i.e., materials, teaching expectations, instructional management) and student on-task behavior was found to be non-significant. This indicates that the classroom ecology and total score on the tool and student on-task behavior are positively correlated, and that the scores on the other domains (i.e., materials, teaching expectations, and instructional management) and student on-task behavior are not related.

Discussion

To ensure that education has a meaningful impact on the outcomes of students with EBD, we must first select known evidence-based strategies. Second, teachers need to be trained on how to implement these practices in the classroom. Finally, teachers need to be provided with performance feedback after training to support the accurate implementation of these practices and maintenance of implementation over time.

This pilot project resulted in a manualized protocol of classroom management strategies, a proposed tool that can be used to provide written performance feedback to teachers, and preliminary data for the classroom tool. The classroom management strategies were selected from existing evidence-based practices. Treatment integrity was low after the initial in-service training. For the elementary group, treatment integrity increased across two additional measures and after written performance feedback was provided. On average, the elementary group met the criterion of 80% implementation after the second round of written performance feedback. This result is consistent with previous research on written performance feedback (Reinke et al., 2007). Treatment integrity improved after the first round of written performance feedback for the middle school group; however, no improvement for the group was documented after the second round of feedback. This could be due to a number of variables, one of which being the need for more intense performance feedback and/or coaching for this group. Additionally, as treatment integrity improved, so did student on-task behaviors. An improvement in student behavior associated with increased treatment integrity is also consistent with previous research (Noell et al., 2005; Sutherland & Wehby, 2001; Sutherland et al., 2000). Finally, a positive correlation was documented between independent time-on-task observations and the classroom ecology and total scores of the treatment integrity tool.

There are several limitations that need to be considered when reviewing the outcomes of this project. First, the descriptive nature of this project is a considerable limitation and therefore no functional relations or causal statements can be used when reviewing the results. These results are purely descriptive in nature and lead to more questions and areas for further investigation than to answers. Future investigations should employ experimental rigor to the procedures outlined in this project, either through single subject or large group methods. Second, the low number of participants (N = 9) further limits interpretation of the concurrent student time-on-task observations. The sample was not large enough to provide the power necessary to evaluate these correlations. Future studies around tool development should include a greater number of participants. Further research is also needed to determine the technical adequacy of the measure (e.g., reliability and validity). Finally, the low return of acceptability questionnaires limits our ability to make strong statements regarding the acceptability of the manualized strategies and the use of the tool for providing performance feedback.

A broader set of research is suggested to develop a model for applying an RtI approach to teacher behavior. In this project we followed the benchmarking model within an RtI framework and used the integrity tool to provide performance feedback three times a year. All teachers were provided written feedback, regardless of the level of implementation. The next steps would be to use a treatment integrity tool to identify teachers that need additional coaching on specific skills. For instance, performance falling between 80 and 100% would fall within the universal/core range and these teachers should be evaluated and provided with performance feedback during the next benchmarking period (e.g. three times per year). Performance falling between 21 and 79% would fall within the secondary/supplemental range and these teachers should be evaluated and provided performance feedback on a monthly basis. Some additional training and coaching might be needed. Performance falling between 0 and 20% would fall within the tertiary/intensive range and these teachers should be evaluated weekly and provided with performance feedback on a weekly basis. Additional training and coaching would be necessary. Future research is needed to develop integrity tools to match this model. Measurement studies could focus on developing performance feedback measures that are sensitive to change over short periods of time and that are efficient to administer.

Given the important role of treatment integrity in the outcomes of youths, it makes sense to consider applying treatment integrity and performance feedback procedures to monitor and support teachers in implementing classroom management strategies. To do so, it will be important to conduct demonstration projects and empirical studies on the systems necessary to support the practice of treatment integrity and performance feedback at this level. Much can be gleaned from the fields of SWPBS and RtI to guide future investigations. Some areas of inquiry can include: (a) How to establish buy-in from staff for screening and ongoing progress monitoring of classroom management implementation? (b) What are the current systems for professional development and how can we work within systems or implement change? (c) How do we identify or, if necessary, build the internal expertise for supporting empirically supported classroom management strategies?

In essence, we can borrow from the whole-school prevention logic of SWPBS and RtI. When considering the idea of adopting a whole staff screening process for monitoring the treatment integrity of classroom management strategies, questions of resources and feasibility surface. Future research could investigate if a three-tiered approach to staff development, as proposed in this discussion, is feasible to carry out in a school setting.

Conclusion

This purpose of this pilot project was to identify a process for providing performance feedback, to staff of emotional support classrooms, on the implementation of empirically-supported classroom management strategies. A component of this project was universal screening of the integrity with which classroom management strategies were being implemented across nine teachers. While purely descriptive in nature, this project served as an impetus for generating questions of "scaling-up" performance feedback efforts. We have a wealth of evidence supporting the use of performance feedback with teachers and the associated improved outcomes in student behavior, however; we mainly implement this approach one teacher at a time. With issues of treatment integrity continuing to exist, and with what we have learned from the fields of SWPBS and RtI, it seems timely that we start investigating whole-school (i.e., whole staff) efforts for addressing what often times is the very root of the problem: treatment integrity.

References

Batsche, G., Elliott, J., Grimes, J., Kovaleski, J. F., Prasse, D., Reschly, D. J., et al. (2006).

Response to intervention policy considerations and implementation. Alexandria, VA: National Association of State Directors of Special Education, Inc.

Fixen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231).

Gresham, F. M. (1989). Assessment of treatment integrity in school consultation and prereferral intervention. School Psychology Review, 18, 37-50.

Henderson, K., Klein, S., Gonzalez, P., & Bradley, R. (2005). Teachers of children with emotional disturbance: A national look at preparation, teaching conditions, and practices. Behavioral Disorders, 31, 6-17.

Horner, R. H., Sugai, G., Todd, A. W., & Lewis-Palmer, T. (2004). School-wide positive behavior support: An alternative approach to discipline in schools. In L. Bambara & L. Kern. (Eds.), Individualized Supports for Students with Problem Behavior: Designing Positive Behavior Plans. New York, NY: Guilford Press.

Kazdin, A. E. (1982). Single-case research design: Methods for clinical and applied settings. New York: Oxford University Press.

Kauffman, J.M. (2003). Reflections on the field. Behavioral Disorders, 28(3), 205-208.

Lewis, T.J., Hudson, S., Richter, M., & Johnson, N. (2004). Scientifically supported practices in emotional and behavioral disorders: A proposed approach and brief review of current practices. Behavioral Disorders, 29, 247-259.

Martens, B. K., Witt, J. C, Elliott, S. N. & Darveaux, D. X. (1985). Teacher judgments concerning the acceptability of school-based interventions. Professional Psychology-Research & Practice, 16, 191-198.

Mortenson, B. P. & Witt, J. C. (1998). The use of weekly performance feedback to increase teacher implementation of prereferral academic intervention. School Psychology Review, 27, 613-627.

Nelson, C.M. (2003). Through a glass darkly: Reflections on our field and its future. Behavioral Disorders, 28, 212-216.

Noell, G. H., Witt, J. C., Slider, N. J., Connell, J. E., Gatti, S. L., Williams, J. L., et al. (2005).

Treatment implementation following behavioral consultation in schools: A comparison of three follow-up strategies. School Psychology Review, 43, 87-106.

Peacock Hill Working Group. (1991). Problems and promises in special education and related services for children and youth with emotional and behavioral disorders. Behavioral Disorders, 16, 299-313.

Reinke, W. M., Lewis-Palmer, T., & Martin, E. (2007). The effects of visual performance feedback on teacher use of behavior-specific praise. Behavior Modification, 31, 247-263.

Sawka, K. D., McCurdy, B. M, & Mannella, M. (2002). Strengthening emotional support services: An empirically based model for training teachers of students with behavior disorders. Journal of Emotional and Behavioral Disorders, 10, 223-232.

Simonsen, B., Fairbanks, S., Briesch, A., Myers, D., & Sugai, G. (2008). Evidence-based practices in classroom management: Considerations for research to practice. Education and Treatment of Children, 31, 351-380.

Sutherland, K., & Wehby, J. (2001). The effect of self-evaluation on teaching behaviors in classrooms for students with emotional and behavioral disorders. The Journal of Special Education, 35, 161-171.

Sutherland, K., Wehby, J., & Copeland, S. (2000). Effects on varying rates of behavior-specific praise on the on-task behavior of students with EBD. Journal of Emotional and Behavioral Disorders, 8, 2-8.

U.S. Department of Education (2006). Twenty-eighth annual report to Congress on the implementation of the Individuals with Disabilities Education Act (IDEA). Washington D.C.: Author.

Walker, H. M., Zeller, R. W., Close, D. W., Webber, J., & Gresham, F. (1999). The present unwrapped: Change and challenge in the field of emotional disorders. Behavioral Disorders, 24, 293-304.

Jennifer L. Jeffrey

Barry L. McCurdy

Devereux Center for Effective Schools

Sam Ewing

Downingtown School District

Dustin Polis

Central Bucks School District 

Correspondence to Jennifer L. Jeffrey, 2012 Renaissance Boulevard, King of Prussia, PA 19406; e-mail: [e-mail].