Promoting academic success for all students.

Abstract

We evaluated a large-scale reform program with emphasis on curriculum-based instruction, ongoing progress monitoring, and systematic and continuing professional development. The design of the study included pre-, post-, and follow-up assessments of all students in reading and mathematics. In general, the treatment group performed statistically significantly better than the comparison group on all variables assessed. The substantial progress reflects the benefits of a balanced approach to school reform and the power of the School Renaissance program.

Introduction

Interest in improving the performance of all children gained renewed national prominence with the passage of the No Child Left Behind Act of 2001 which was signed into law on January 8, 2002. Heralded as "more than just a law," the act directs states and school districts to develop strong systems of accountability based upon student performance (U.S. Department of Education, 2002b). The law also gives states and school districts increased local control and flexibility, removing federal red tape and bureaucracy and putting decision-making in the hands of those at the local and state levels. To improve outcomes for children, No Child Left Behind provides parents of children from disadvantaged backgrounds the option to participate in public school choice programs or obtain supplemental services such as tutoring. To further support the achievement of improved results for all children, teachers are encouraged to use teaching methods based upon scientific research that shows they are effective (American Association of School Administrators, 2002; Dede, Honan, & Peters, 2005; U. S. Department of Education, 2002a,b, 2006).

The emphasis on using scientifically based research as a guide for deciding how to teach directs teachers in how to help the large numbers of students who are failing to profit in America's classrooms. It also serves as a way to address the disappointing results (cf. Walmsley & Allington, 1995) of many instructional programs, some of which even complicate the learning process by offering approaches that are philosophically different from those offered in the classroom (Santa & Hoien, 1999). Therefore, most poor readers never catch up with their peers in reading and writing abilities, and the gap between low and high readers broadens as children progress through the grades (Stanovich, 1986; 1991). These changes have generated an increased interest in programs designed to improve achievement levels, especially those of children performing below grade level. One such program, School Renaissance, promises to improve academic achievement for all children using proven teaching methods and technology for using on-going formative evaluation data.

School Renaissance is a commercially available K-12 comprehensive school reform program designed to help teachers use information to improve learning outcomes for their students (http://www.renlearn.com/). It is in the Catalog of School Reform Models produced jointly by the Northwest Regional Educational Laboratory and The Center for Comprehensive School Reform and Improvement. Criteria for selecting models include evidence of effectiveness in improving student academic achievement, extent of replication, implementation assistance provided to schools, and comprehensiveness. The average cost of a full implementation can range from $30,000 to $75,000 per school per year of a three-year plan. Using formative evaluation data, professionals in participating schools learn to make better instructional and curricular decisions and are in a better position to accelerate learning (cf. Nunnery, Ross, & Goldfeder, 2003; Paul, 1992, 1993; Paul, Swanson, Zheng, & Hehenberger, 1997; Paul, VanderZee, Rue, & Swanson, 1996; Peak & DeWalt, 1994; Ross & Nunnery, 2005; School Renaissance Institute, 2002; Smith & Clark, 2001). The model focuses on developing foundational skills in reading, writing, and mathematics. The program supports any curriculum, as teachers and principals integrate their textbooks, basal readers, and adopted curriculum into the model to create a comprehensive program. School Renaissance provides the information technology, professional development, consulting, and support needed to integrate information into all levels of differentiated instruction.

Electronic "learning information systems" or "curriculum-based monitoring systems" are used to store and report information on students' daily practice of academic tasks. Professional development, on-site consulting, and implementation and evaluation support are included to assist teachers in using this information to make sound instructional and curricular decisions (e.g., grouping certain students to teach a specific skill identified in the data as needing attention). Differentiation of instruction and practice aid students with special learning needs.

Data-driven decision-making is a daily process. Principals and teachers use research-based techniques to incorporate the data generated by the information technology into communication, common goals, and school culture. Using information to address individual needs becomes routine. In the initial stages of implementation, consultants help each school develop a leadership team and appoint a coordinator who supports other staff. When using School Renaissance district-wide, the coordinators at various schools collaborate. Continuous monitoring efforts provide a specific process for ongoing self-examination and continual renewal of key program expectations. While programs in America's preschools and schools have been effective, the need for evidence-based programs to support achievement of all children remains part of the country's education agenda. In this research, we set out to empirically determine the effects of School Renaissance on the academic performance of elementary school children.

Method

We designed this study to answer some important questions about a large-scale reform program implementation:

1. To what extent does School Renaissance help students learn to read?

2. To what extent do student participating in School Renaissance read better than comparable children using other programs already in place in the district?

3. To what extent does School Renaissance help students achieve in other academic areas? With these questions in mind, we conducted a district-level evaluation of the program.

Participants

Teachers and children in four elementary schools participated in this study. The treatment schools were designated as model School Renaissance sites, meaning that teachers were using the complete program. Although some of the components of School Renaissance were in the two Title I comparison schools, they were not used consistently, and teachers had received little, if any, professional development in the model. The comparison schools were matched to the treatment schools on percent of free and reduced lunch participation (an indication of lower socio-economic status), percent of majority students, and as close as possible, to geographic location. The grade organization in each school was similar, including preschool to fifth grade students in every school but one, and comparable overall enrollments (i.e., 350-500) in each school except one. Low (i.e., 1-8%) majority enrollments were evident in one treatment and one control school and high (i.e., 97-99%) majority enrollments were evident in the other two schools. Similarly, average (i.e., 41-52%) free and reduced lunch statistics were evident in one treatment and one control school while high numbers of students qualifying for free and reduced lunch (i.e., 97-99%) attended the other two schools. Students in one urban treatment and control school and one rural treatment and control school participated in the study.

Procedures

Participating professionals in treatment schools received the on-site, ongoing consulting involved in full School Renaissance implementation. Researchers visited each classroom in the four schools early and again towards the end of the school year. These general observations were made to record what was occurring at each of the schools.

Intervention description. School Renaissance is a K-12 comprehensive school reform modern that helps educators use information to improve learning outcomes for every child. With ongoing formative information on each student, classroom, and school, participating professionals are in stronger positions to make better instructional and curricular decisions and accelerate learning. The program provides the information technology, professional development, consulting, and support to help educators integrate information into all levels of school functioning. The School Renaissance model focuses on developing foundational skills in reading, writing, and mathematics. It supports any curriculum, as teachers and principals integrate their textbooks, basals, and adopted curriculum into the model to create a comprehensive program. Information technology, called "learning information systems" or "curriculum-based monitoring systems," is used to store and report information on students' daily practice of academic tasks. Professional development, on-site consulting, and implementation and evaluation support assist educators in using this information to make sound instructional and curricular decisions (e.g., grouping certain students to teach a specific skill identified in the data as needing attention). Differentiation of instruction and practice assist students with special learning needs.

In the initial stages of implementation, a Renaissance Learning Consultant helps each school develop a Leadership Team and appoint a Coordinator who supports other staff and is an important member of the Leadership Team. Such local representation is a characteristic of high quality school-wide reform efforts and essential for sustainability and long-term success (cf. Mid-Continent Research for Education and Learning, 2003; U. S. Department of Education, 2006; Walter, 2004). In district-wide implementations, the Renaissance Coordinator also collaborates with partners at other schools. Ongoing "certification" provides a specific process for ongoing sell-examination and continual renewal of the School Renaissance vision.

Treatment fidelity. To measure the implementation of the program, trained observers measured key aspects of the classroom and teacher-student interactions. For the reading classrooms, researchers used an observation checklist reflecting 24 expected behaviors. The mean average percent of behaviors observed for the treatment schools (M=53.00, SD 0.30) across grade levels and classrooms was significantly higher (F= 3.94, df=1,98, p < 0.05) than for the comparison schools (M=39.00, SD = 0.30). Similarly, for math, researchers used an observation checklist reflecting 20 desirable behaviors. The mean average percent of behaviors observed for the treatment schools (M=62.00, SD = 0.30) was significantly higher (F= 3.95, df=1,86, p < 0.05) for the comparison schools (M=47.00, SD = 0.30).

Design and Data Analysis

Scores were collected to use as a covariate from the Iowa Test of Basic Skills (ITBS) administered during the spring prior to the study for a single cohort of students with similar demographics to those participating in the evaluation. Pretest and posttest scores in reading, language arts, and mathematics from the state's end-of-grade assessment tests (CRCT) were compiled prior to and one and two years after implementation of the program and were used as the dependent variables in data analyses. Only students for whom scores were available for all years were included in the analyses. One hundred and thirty-two students participated in the control group. Scores were available for after the first year for 214 students and after the second year for 386 students in the treatment group.

Outcomes

Statistically significant differences were evident indicating that students in the treatment schools had outperformed their peers in the comparison schools in all three areas of achievement. When subsequent follow-up scores on the state's end-of-grade assessment tests were compared, the gains of the first year (M=328) were maintained through the second year (M=346). These analyses also reflected that not only were the gains of the treatment schools over the comparison schools maintained, but that the differences in reading increased during the second year (i.e., 2 pts vs. 22 pts). Effect sizes were conservatively estimated (Holmes, 1984) and all were positive indicating that the practical significance of the differences favored students participating in the School Renaissance program (i.e., the average overall effect was +.65, +.50 for reading, +.71 for language arts, and +.75 for mathematics). In general, the School Renaissance students outperformed their comparison group peers on the adjusted posttests by about 1/2 to 3/4 of a standard deviation over both years of implementation.

Conclusion

While many of the comparisons made in evaluating reform programs look at one year's third grade scores to the next year's third grade scores, a comparison that often is not valid, this study followed a cohort of children across three grades to evaluate the effects of implementation of School Renaissance on the progress of participating children. In all nine comparisons involving standardized test scores in reading, language arts, and mathematics, the children in "Renaissance" schools consistently outperformed their peers in other schools. Importantly, these outcomes were evident on a variety of measures and maintained over time for students similar to those typically at risk of school failure across the country.

The value of continuous monitoring of progress has long been included in lists of critical features of effective instruction for students at high-risk of school failure and recent large scale reviews have reaffirmed its importance in teaching basic skills (cf. Algozzine, Ysseldyke, & Elliott, 1997; American Association of School Administrators, 2002; Dede, Honan, & Peters, 2005; National Research Council, 1998; Northwest Regional Educational Laboratory, 2006; Nunnery, Ross, & Goldfeder, 2003; Ross & Nunnery, 2005; Ross, Nunnery, Avis, & Borek, 2005; Snow, Burns, & Griffin, 1998; Stringfield, Wayman, & Yakimowski, 2005; Topping & Sanders, 2000; U. S. Department of Education, 2005; Wayman, 2005; Wayman & Stringfield, 2005; Wayman, Midley, & Stringfield, 2005; Ysseldyke, Algozzine, & Thurlow, 2002). Key components of School Renaissance support this practice and the effects appear strong and durable. Electronic monitoring systems are used to store and report students' daily progress of academic tasks and this information is the basis for informing, modifying, and adjusting instruction over the course of lessons, units, and instructional periods.

The program also provides a clear benefit in linking to the natural curriculum in participating schools. Teachers in participating schools taught content prescribed by the state's department of education and it was included in all components of the School Renaissance implementation. Participants did not use alternate programs and special curriculum materials. Again, the benefits of curriculum-based measurement and instruction directly linked to the "scope-and-sequence" of the materials being used with all students have been widely documented (cf. Deno, 2003; Fuchs, 2004; Shapiro, Angello, & Eckert, 2004). School Renaissance incorporates the fundamentals of curriculum-based measurement practices: Technical adequacy, standard measurement tasks, prescriptive assessment materials supporting increased usefulness of outcomes for instructional planning, standardized specification of sample duration, administration, student directions, and scoring procedures, performance sampling, equivalent assessment materials, and time efficiency.

Regardless of the strength and consistency of our findings, limits of the research paradigm exist. School district constraints and practical limits prevented random assignment of conditions, students, and teachers. While these factors restrict the generalizability of the outcomes, our efforts to study progress in comparable schools have likely minimized their effects.

School Renaissance represents a balanced program that incorporates key features of effective instruction. It takes place across academic content areas and provides opportunities for teachers to adjust instruction with acceleration and/or remediation as needed. Children spend time actively engaged in materials that match their instructional levels and alterations are made based on regular, data-based decision-making activities that represent the best of what is accepted as effective school practice. The program also incorporates other components of direct instruction, such as frequent teacher explanations and demonstrations followed by increasingly independent student performance, supportive and corrective feedback to correct and sustain performance, and ongoing record keeping to ensure accurate appraisal of developing skills and needed instruction. The developer of School Renaissance, Renaissance Learning, provides a range of professional development and support components that sustain the program. In sum, School Renaissance makes sense and the outcomes of full-scale implementation are strong.

References

Algozzine, B., Ysseldyke, J. E., & Elliott, J. (1997). Strategies and tactics for effective instruction. Longmont, CO: Sopris West.

American Association of School Administrators. (2002). Using data to improve schools: What's working. Arlington, VA: Author.

Deno, S. L., (2003). Developments in curriculum-based measurement. The Journal of Special Education, 37, 184-192

Dede, C., Honan, J. & Peters, L. (Eds.). (2005). Scaling up success: Lessons learned from technology-based educational innovation. San Francisco: Jossey-Bass.

Fuchs, L. S. (2004). The past, present, and future of curriculum-based measurement research. School Psychology Review, 33, 188-192.

Holmes, C. T. (1984). Effect size estimation in recta-analysis. Journal of Experimental Education, 52(2), 106-109.

Mid-Continent Research for Education and Learning. (2003). Leadership folio series: Sustaining school improvement. Aurora, CO: Author.

National Research Council (1998). Preventing reading difficulties in young children. Washington;-DC: National Academy Press. [on-line]. Retrieved July 17, 2006, from http://books.nap.edu/books/030906418X/html/index.html.

Northwest Regional Educational Lab. (2006). The catalog of school reform models. Retrieved July 17, 2006 from http://www.nwrel.org/scpd/catalog/index.shtml

Nunnery, J. A., Ross, S. M., & Goldfeder, E. (2003). The effects of School Renaissance on TAAS scores in the McKinney ISD. Memphis, TN: University of Memphis, Center for Research in Educational Policy.

Paul, T. (1992). National reading study and theory of reading practice. Madison, WI: Institute for Academic Excellence.

Paul, T. (1993). National Study of Literature-Based Reading: How Literature-Based Reading Improves Both Reading and Math. Madison WI: Institute for Academic Excellence.

Paul, T., Swanson, S., Zhang, W., & Hehenberger, L. (1997). Learning information system effects on reading, language arts, math, science, and social studies. Madison, Wisconsin: Institute for Academic Excellence, Inc. (ERIC Document Reproduction Service No. ED 421 686)

Paul, T., VanderZee, D., Rue, T., & Swanson, S. (1996). Impact of the Accelerated Reader technology-based literacy program on overall academic achievement and school attendance. Paper presented at the National Reading Research Center Conference "Literacy and Technology for the 21st Century", Atlanta, GA, October 4, 1996. (ERIC Document Reproduction Service No. ED 421 684)

Peak, J., & Dewalt, M.W. (1994). Reading Achievement: Effects of Computerized Reading Management and Enrichment. ERS Spectrum, 12(1), 31-35.

Ross, S. M., & Nunnery, J.A. (2005). The effect of School Renaissance on student achievement in two Mississippi school districts. Memphis, TN: University of Memphis, Center for Research in Educational Policy.

Ross, S., Nunnery, J., Avis, A., & Borek, T. (2005). The effects of School Renaissance on sudent achievement in two Mississippi school districts: A longitudinal quasi-experimental study. Memphis, TN: University of Memphis, Center for Research in Educational Policy.

Santa, C. M., & Hoien, T. (1999). An assessment of Early Steps: A program for early intervention of reading problems. Reading Research Quarterly, 34, 54-79.

School Renaissance Institute. (2002). Idaho statewide implementation of reading renaissance: summary of second year's results. In Research Summary, p.44. Madison, WI: School Renaissance Institute.

Shapiro, E. S., Angello, L. M., & Eckert, T. L. (2004). Has curriculum-based assessment become a staple of school psychology practice? An update and extension of knowledge, use, and attitudes From 1990 to 2000. School Psychology Review, 33, 249-257.

Smith, E.G., & Clark, C. (2001). School Renaissance Comprehensive Model Evaluation. Madison, WI: School Renaissance Institute.

Snow, C., Bums, S., & Griffin, P. (1998). Preventing Reading Difficulties in Young Children. Washington, D.C.: National Academy Press.

Stanovich, K. E. (1986). Matthew effects in reading: Some consequences of individual differences in the acquisition of literacy. Reading Research Quarterly, 21, 360-406.

Stanovich, K. E. (1991). Changing models of reading and reading acquisition. In L. Riben & C.A. Perfetti (Eds.), Learning to read: Basic research and its implications (pp. 19-31). Hillsdale, NJ: Erlbaum.

Stringfield, S., Wayman, J. C., & Yakimowski, M. (2005). Scaling up data use in classrooms, schools and districts (pp. 133-152). In Dede, C., Honan, J. & Peters, L. (Eds.), Scaling up success: Lessons learned from technology-based educational innovation. San Francisco: Jossey-Bass.

Topping, K. J., & Sanders, W. L. (2000). Teacher effectiveness and computer assessment of reading: relating value-added and learning information system data. School Effectiveness and School Improvement, 11(3), 305-337.

U. S. Department of Education. (2002a). No Child Left Behind Web Site. http://www.ed.gov/offices/OESE/esea.

U. S. Department of Education. (2002b, August 7). Press Release: Paige Kicks Off New School Year In Florida, Says No Child Left Behind Is More Than A Law, It's A 'Revolution.' Washington, DC: Source.

U. S. Department of Education. (2006). Comprehensive school reform web site. http://www.ed.gov/programs/compreform/2pager.html

Walmsley, S., & Allington, R. (1995). Redefining and reforming instructional support programs for at-risk students. In R. Allington & S. Walmsley (Eds.), No quick fix: Rethinking literacy programs in America's elementary schools (pp. 19-44). Newark, DE: International Reading Association.

Walter, K. E. (2004). Making good choices: Sustainable school improvement. Napierville, IL: North Central Regional Education Laboratory.

Wayman, J. C. (2005). Involving teachers in data-based decision-making: Using computer data systems to support teacher inquiry and reflection. Journal of Education for Students Placed At Risk, 10(3), 295-308.

Wayman, J. C., & Stringfield, S. (2005, April). Teachers using data to improve instruction: Exemplary practices in using data warehouse and reporting systems. Paper presented at the 2005 Annual Meeting of the American Educational Research Association, Montreal, Canada.

Wayman, J. C, Midgley, S., & Stringfield, S. (2005, April). Collaborative teams to support data-based decision-making and instructional improvement. Paper presented at the 2005 Annual Meeting of the American Educational Research Association, Montreal, Canada.

Ysseldyke, J. E., Algozzine, B., & Thurlow, M. L. (2002). Critical issues in special education. Boston: MA: Houghton Mifflin.

C. Thomas Holmes, The University of Georgia 

Carvin L. Brown, The University of Georgia

Bob Aigozzine, University of North Carolina at Charlotte

C. Thomas Holmes and Carvin Brown are Professors in the Department of Educational Administration. Bob Algozzine is Director of the Behavior and Reading Improvement Center.