Continuous improvement: it takes more than test scores: analyzing state assessment results is only the beginning of effective data-driven decision making.

There is no question that the passage of No Child Left Behind in 2001 has impacted schools in at least two ways: First and foremost, NCLB has made the use of data to improve student achievement imperative; and second, NCLB has increased the need for continuous improvement processes within schools.

Summative data just the beginning

Schools in our country hear that data makes the difference in improving student achievement. Not all schools, however, have felt the positive impact from what they believe is data-driven decision making. The most common reason: Most school districts in this country believe they are being data-driven when they have analyzed the dickens out of their state assessment results.

Some school districts feel they are being data-driven when they analyze the dickens out of their state assessment results and use some formative assessments to help students prepare for the statewide test. Unfortunately, this is only the beginning of data-driven decision making.

Most states' assessment scores can speak volumes for what is going on in their schools and districts, and with student learning. Following assessment scores of the same groups of students (or the same students) over time can indicate the presence or absence of a continuum of learning that makes sense for the students. It can give information about student learning as well as the degree to which standards are being implemented at each grade level (provided the state assessment truly measures the state standards).

Looking at state assessment results by grade level over time can give information about the impact of the programs being implemented. Disaggregating state assessment results can tell us if our schools are meeting the needs of the students the schools are attempting to serve.

Test score analyses are important. In fact, in a perfect world, schools would use both formative and summative assessments to ensure that all students are learning. If only summative assessment data are studied, however, solutions for improving the scores can come out half-baked.

For example, when I started working with Lemon Middle School, the staff had determined that their students' scores in English/language arts and mathematics were lower than the previous years' scores. Since the math scores were the lowest, they decided to "focus" on math that year.

Gallant efforts

In their focus, they set up several strategies: remediation for the students not meeting proficiency standards, an afterschool program to assist students with their math homework, and a required math summer school program for any student not passing the state math assessment at the proficient or advanced levels.

Unfortunately, their gallant efforts did not lead to the test score improvements they had hoped for. Both math and English/language arts scores went down. They were devastated. As the staff and I reviewed their data and their solutions, we talked about establishing a continuous improvement plan. We set out to gather a bit more data to see if we could figure out how to work smarter, not harder, and get better results.

We had the student achievement data. We determined that other types of data, including demographic, perceptional and school process data, needed to be gathered and analyzed. What those data are and what we found in the data analysis at Lemon Middle School are discussed below.

Demographic data

Demographics can tell schools all about who they have as students, who they have as teachers, and how teachers are aligned to the students. Demographics begin to tell us about school processes and how the school is preparing to meet the needs of students. Demographics are important for setting the context of the school, and they are critical for understanding all other numbers.

Lemon Middle School's demographics showed that while they had a fairly diverse student population--60 percent Caucasian, 30 percent Hispanic/Latino and 10 percent African American--their teaching staff was 100 percent Caucasian and 80 percent female.

After a recent teacher "buy out" due to budget decreases, the current teaching staff had an average of six years of teaching experience. The math teachers were the least experienced, with only three years of teaching experience on average. The principal, a male, was in his third year at this school. The attendance rate was 94 percent for students and 91 percent for teachers. The district, in its concern about the inexperience of the teachers and the low test scores, assigned most of the specialists in the district to "watch over and help" Lemon teachers.

Perceptional data

Perceptional data can tell us about how students, staff and parents are feeling about the learning environment and give insight into what changes can be made to improve the learning environment and, ultimately, student learning.

Lemon Middle School students, staff and parents completed questionnaires about the learning environment. What the questionnaires told them was students did not feel teachers thought they could do the work, that quality work was not expected of them, and that teachers did not care about them as people--teachers only wanted students to get better test scores.

In their open-ended responses, several students wrote that all they liked about the school was being outside, their friends and sports. What they wished was different was that school would not be so boring, that they could learn cool things in cool ways, and that they would not get math and test scores shoved down their throats all the time.

The teacher questionnaire gave even more definitive information. Teachers felt that there was no vision for the school. There might have been a vision a couple of years ago, but that was before most of the current staff was hired. Teachers did not feel they had the support they needed to improve student learning, and they did not collaborate with each other to ensure a continuum of learning that made sense for the students.

In their open-ended responses, staff told about the inconsistencies in teaching throughout the organization, the fact that staff agreements were never enforced and that the district was always sending people in to "watch us, not help us."

Commitment from staff

The most often-stated responses to the question, "What would it take to improve student learning for all students at this school?" included this: All staff needs to commit to doing work in the same way, including teaching to the standards, assessing and using the results to effect improvement. Teachers who are not willing to put in the time and energy should not be allowed to keep their jobs, or they should have their pay docked.

Parents basically reiterated what the students told staff. Parents did not feel that the staff was truly committed to students' learning. Teachers did not seem to be making progress with what they were doing. Staff expected way too much help from the parents. Parents did not know how to help their children learn. Additionally, each parent worked one or two jobs. When could they help their children learn?

School processes

School processes are the curriculum, instruction and assessment strategies used to teach the content that students are expected to learn. It is important to understand what teachers are doing to get the results they are getting.

Lemon Middle School staff reviewed what they were doing to teach each of their subject areas. Most teachers were using the curriculum and instructional materials adopted by the district. What they could not tell each other was how much of the time they were teaching to the standards. In tact, they were not really sure what it would look like if they were teaching to the standards.

Their current processes for helping students were to "remediate" any student who did not score proficient or advanced on the state assessment. A lot of the remediation was simply re-teaching what had been taught before without determining what the students really needed.

Analyzing the data

These data--student learning, demographics, perceptions and school processes, separately and combined--tell school personnel what is going on in the school right now, give clues as to what needs to be done to improve, how to improve to get different results and even what is possible with respect to improvement.

Reviewing all the data gave Lemon Middle School staff some common thinking about what they needed to do to get different results. It also empowered them to get different results. Staff knew they needed a vision and commitment from every teacher to help every student learn. They knew they needed to become more familiar with and to implement content standards, and to learn how to make learning more active and fun. They also needed and wanted accountability processes and leadership to keep all the staff on the same page.

Because of the inexperience of the staff and principal, the district provided the dollars to help Lemon Middle School hire an outside facilitator to establish a clear and shared vision and a structure for continuous improvement. The structure included time to analyze their data and student work, and to develop strategies for improvement using the results of their analyses. Students' test scores in the following year were greatly improved.

The need for a continuous improvement process

True data-driven decision making is only partly about data. A clear and shared vision and leadership play major parts in data-driven decision making. If there is no focus or unified front in a school, there is also no continuum of learning that makes sense for students, and no structure to increase student achievement.

It takes strong leadership to inspire a shared vision and to ensure its implementation. It also takes a strong leader to ensure the analysis and use of data. A continuous improvement process can ensure that all professional development is focused on implementing the vision; that partners, such as parents, understand their roles in implementing the vision and helping students learn; and that there is continuous evaluation to know how to improve on an ongoing basis to reach school goals.

Inspiring a shared vision

Schools and school districts are under intense pressure to improve. With both limited resources and limited time to develop processes that allow them to move steadily upward, schools must use data to ensure that their improvement is effective and continuous.

It is vitally important that administrators--at school and district levels--lead the way in using data to inform decision making. Leaders must challenge processes through the study of school results, inspire a shared vision, enable others to act through planning and professional development, model the way through consistent actions, encourage the heart by reminding teachers of the purpose of school and why they got into teaching in the first place, and celebrate successes (Kouzes & Posner, 2002).

True data-driven decision making gives schools information about their current situation and clues about what would help them improve, as well as the leadership to see it through.

References

Bernhardt, V.L. (2003). Using Data to Improve Student Learning in Elementary Schools. Larchmont, NY: Eye on Education, Inc.

Bernhardt, V.L. (2004). Using Data to Improve Student Learning in Middle Schools. Larchmont, NY: Eye on Education, Inc.

Bernhardt, V.L. (2004). Using Data to Improve Student Learning in High Schools. Larchmont, NY: Eye on Education, Inc.

Bernhardt, V.L. (2005). Using Data to Improve Student Learning in School Districts. Larchmont, NY: Eye on Education, Inc.

Kouzes, J.M., & Posner, B.Z. (2002). The leadership challenge: How to keep getting extraordinary things done in organizations. (2nd Ed.). San Francisco, CA: Jossey-Bass Publishers.

Victoria L. Bernhardt has directed the Education for the Future Initiative since 1991, working with schools and districts across the country. Bernhardt is the author of 10 books, all published by Eye on Education, including the latest four-book series: "Using Data to Improve Student Learning."