Beyond data mania: educators must not only learn the skill of data analysis, but develop the disposition to seek and use evidence in decision-making throughout the school. This requires new habits and attitudes toward data use.

The newest merit badge for school leaders identifies them as "data-driven decision-makers." Data mania has given birth to an unprecedented number of professional development sessions and products aimed at helping administrators read and interpret student performance data. School leaders whose careers began in a time when annual test score reports were dutifully summarized and forgotten are suddenly led to believe that the answer to improving student achievement lies hidden within those scores.

WestEd's Western Assessment Collaborative worked for four years with a set of districts and schools to support their implementation of standards-based accountability. The effort resulted in promising changes in practice at these schools; but seldom, if ever, did principals uncover silver bullets in their data reports. Instead, these schools were characterized by ongoing, often messy and ambiguous processes of framing questions, examining and weighing evidence, taking action and discovering new questions.

But with these efforts have come new habits and attitudes toward data use. Growing numbers of administrators, teachers and even parents in these schools are no longer content to know "how many" students have reached certain standards, but seek to understand the factors or conditions that contribute to or inhibit improvement. They have developed skills for gathering and analyzing both quantifiable and more qualitative forms of evidence, and for using these in combination to guide their own work and the work of the school.

Perhaps most importantly, these new habits have begun to impact the school's own sense of efficacy in improving student performance.

A principal reports: "Teachers all know how to set goals and look at data and they all buy into the results. You can tell, because they are very excited when they get the results and they are very bummed when they don't reach their goal. You used to hear, `Its not my problem.' You just don't hear that anymore. There is real ownership of the challenge to improve student performance."

And teacher comments support the principal's observation: "There is real clarity about which students are learning and which ones aren't. We can all talk about how we know that, and what we are going to do. This has been a dramatic change for us."

So how are people supported to learn not only the skill of data analysis, but to develop the disposition to seek and use evidence in decision-making throughout the school? The Western Assessment Collaborative's work offers some "lessons learned" to guide others in this important work.

1 Help people to articulate the beliefs and assumptions that drive both the questions they explore and the interpretations they make of data.

Schools with the habit of considering data in their work begin by articulating the outcomes they want for their students, and see data analysis not as an end in itself, but simply as a signpost on the path toward those outcomes.

Data are meaningless outside the context of stated goals or standards for student achievement. It is our belief that girls need to know higher-level mathematics, and our assumption that they are just as capable as are boys of doing so, that drives our effort to find out why so many of our girls consistently score less well than boys in mathematics. If we bring a different set of beliefs to the table -- say that girls don't need to know higher-level mathematics, or that they are not as capable as boys -- then the data do not worry us and we are unlikely to pursue further questions about it.

Likewise, our interpretations of data are driven by our assumptions and beliefs. The principal who prepares attractive overheads of student performance data disaggregated by racial or ethnic sub-groups might be surprised by the wrath of parents of students of color in the audience, should she neglect to state her reason for cutting the data in this way. Audiences are just as likely to interpret the presentation of lower scores in the sub-groups as an attempt to blame those students for affecting schoolwide averages as they are to assume that the principal's purpose was to energize the community to better serve these students.

WAC's project taught participants to generate questions to explore based on their beliefs and assumptions about the purpose of the school and its vision for what students can and should learn. The first set of questions fell into the category that might be called, "Are things as good as they should be?"

For example: We believe that all students should be able to read at third grade level by the end of third grade. How many of our third grade students read at that level?

We believe that school should be a place where all students are comfortable and have opportunities to learn. How many of our students drop out? How many have poor attendance records?

The next generation of questions help school personnel to better understand whether the program works as well for some as it does for others:

If not all third grade students can read at third grade level, which students are not making adequate progress?

Even deeper disaggregations help to explore patterns that may point to strategies for improvement:

We believe the AVID program is a powerful way to help students of color close the achievement gap. Do the students of color in the AVID program progress faster than peers who are not in AVID? (And if so, how can we give more students access to this program?)

We believe some of our schools may be better than others at serving the needs of highly mobile students. Do highly mobile students do better at some schools than at others? (And if we find our belief to be confirmed, what's going on at these schools?)

Low-performing schools or schools with homogenous student populations may at first see little value in such exercises. "All our students (or all our schools) are low performing," they report in exasperation. But powerful data explorations can reveal to these districts or schools a variety of important information, such as individual teachers whose students consistently do better than other students in other classes, interventions that show promise, or schools with similar student populations that are making significant progress. When these patterns are revealed, personnel in low-performing schools are less able to claim that status as inevitable.

Our work also helped schools use less easily quantifiable sources of evidence to explore their assumptions about the factors contributing to poor student performance. When one school's students did poorly on a writing sample requiring a persuasive essay, several teachers speculated that different teachers held, and therefore taught to, very different standards for the quality of a persuasive essay.

The faculty agreed to explore that hunch by informally scoring a sample of papers from across classrooms and studying the pattern of their scores. When the hunch proved correct, the group then worked together to identify exemplars that all could use to calibrate their own judgments of student work.

2 Teach people to judge the quality of the evidence they have and to understand that no data are unassailable.

Much of the reluctance to use data in schools stems from historical disagreements about what data should be used to describe student performance. Educators criticize large-scale standardized measures as unfair, untimely and unrelated to what they teach. The information generated by these tests has little credibility to many administrators and classroom teachers and has limited utility in guiding the work they do on a day-to-day basis. Similarly, data that are valued by classroom teachers are often dismissed by parents, administrators and policymakers. Classroom or home-grown district or school-level assessments are criticized as unreliable, expensive and redundant in face of the state's system.

The quality of any data -- including data on student performance -- is judged by the user in terms of credibility and usefulness. Can I trust that the evidence I have is valid and reliable? And does it give me timely, comprehensible information to guide the work that I need to do? The all-to-often unacknowledged reality is that legislators, board members, administrators, teachers, parents and students each have different answers to those questions depending on their purpose for considering data in the first place.

Annual norm-referenced test scores are not useful -- in fact, are often misused -- to plan instructional programs for individual children. However, they are an efficient and accurate means of uncovering strengths or weaknesses in the broader instructional program over time.

Well-designed portfolios and performance tasks do provide rich diagnostic data to help classroom teachers plan instruction, but are too cumbersome and expensive to help policymakers understand patterns of performance among large groups of students.

WAC's project worked with educators and parents to help them understand the various purposes of assessment, how the inferences that could be drawn from any one assessment varied according to its design and administration, and to appreciate that every assessment was both valuable in illustrating a part of the story of student performance, and limited in the ways it could or should be used.

Understanding this, project participants demonstrated a greater willingness to use large-scale state assessment data and to guard against its inappropriate use. At the same time, they increased their investment in diagnostic classroom assessments and became more vigilant about the reliability of scores from these assessments.

What emerged was an understanding of the interdependence of high quality large-scale assessments and powerful classroom assessments that guide the day-to-day decision-making of classroom teachers.

One project teacher explained: "It's like when a new coach is hired to work with a baseball team. He wants to know the win-loss record and league standings because that lets him know what he's getting into. But if he wants to improve their game, those things don't help him much. He's going to need to diagnose what's going on with each player, what he does well, and what skills will need more work."

Likewise, one principal explained: "Our performance assessments have been really valuable in getting teachers to talk to each other about what we should expect from all kids. But it wasn't until we looked at our SAT-9 data that I could convince them that something was going on for our Latino kids. They do fine for a while, and then really nose-dive around fourth grade -- even more than other kids."

3 Foster "teaching with the end in mind."

Habits of using evidence in decision-making must extend to classroom, as well as school-level, practice. WAC's work introduces school leaders and classroom teachers to an instructional design process we call "backwards mapping from standards to instruction."

Instructional planning begins with the questions: "What evidence will students need to produce in order for me to be convinced that they have achieved this standard? On what criteria will I judge that performance?"

Teachers are confronted with questions about what it really means to know something and to provide credible and useful evidence of that learning. With the assessment and performance criteria to which students will be held accountable clearly in mind, teachers plan instruction to assure that students are provided the lessons and practice they need to in order to achieve desired results. Teachers are taught to use the evidence students produce on a daily basis -- their work -- to assess their progress and to diagnose both what they misunderstand and what they seem ready to learn.

WAC's most recent work with district office administrators, principals and teacher leaders illustrates how the evaluation and coaching process can be used to foster this form of evidence-based instruction. Supervisory planning conferences focus on helping teachers to articulate how they will know when students have achieved to the desired standard and what expectations for student performance will be used to guide lesson planning. Post-lesson observation conferences focus on what student work and student performance results reveal about the success of the lesson. The goal of WAC's leadership training is to help administrators and teacher leaders use evidence in creating the conditions that perpetuate continuous learning among all staff.

4 Share data with parents, the public and students and invite them to help you understand conditions that contribute to or inhibit achievement.

Our project required schools to conduct Accountability Dialogues -- sessions that engaged all members of the school community in establishing standards for student performance, looking at student performance data, exploring the factors that might be contributing to the school's performance and generating ideas for improvement.

Schools anticipated these sessions like one would a root-canal. One teacher reported, "I really wasn't looking forward to this. The thought of the faculty being in a fishbowl with a lot of parents looking at our work scared me to death."

Another offered, "There was a fear that if we shared the data, parents would say, `Do you guys know what you're doing?'"

But the sessions have paid off in helping schools build norms of using evidence in decision-making and in encouraging a sense of shared responsibility for improved student performance. Parents come to understand what is expected of students and are invited to suggest factors that may be contributing to or inhibiting improvement.

In one community, Latino parents at the dialogue were shocked to see how much lower their children's performance was, as a group, than their white peers. The session allowed their concern to turn to advocacy, and the district agreed to seek funds to start a parent education program to support these families.

Parents and community members respond to the schools' openness with appreciation and support. The following comment by one parent reflected the sentiments of most who participated in these sessions: "I had no idea the district would be so open and forthright. I felt valued here, like my opinion counted. I have a better sense that the school is trying to address the problems it has."

5 Invest in the technology and human resources necessary to facilitate data inquiry at the classroom, school and district level.

It will come as no surprise to any administrator reading this article that the current state of data technology in most schools cannot support the kinds of inquiry described above. When WAC's project began, school administrators who were beginning to value the power of data inquiry reported hours of laborious paper and pencil tallies in pursuit of answers to their questions. One assistant superintendent told the story of wanting to know whether middle school girls were opting out of higher-level math courses. To explore her question, she took the report cards of girls who had received A's or B's in seventh-grade math, laid them out on her living room floor and compared them to the same students' eighth-grade class schedule to see if they were taking algebra! Project principals were known to cajole parent volunteers into building and maintaining relational data-bases for the school.

While there has been some improvement over the last four years, things are not a whole lot better today. Although many districts have invested in new student data management systems, many have underestimated the manpower necessary to maintain these systems in ways that might be useful to schools.

The way data are entered or retrieved from the district system often frustrates school-level inquiry. School-based inquiry software proved valuable to several schools, but lay dormant in others where no one was available with the time or expertise to do data entry or run inquiries.

Recent leadership literature encourages us to become "data-driven decision-makers," but as with so many other catch-phrases before it, this one risks becoming trivialized in practice. Nothing much has been achieved just because an administrator can produce disaggregated state test data, and less than nothing is gained when data are used to justify inappropriate action.

Data are valuable, but they are also often ambiguous and even conflicting. Used alone, no one source of data tells the whole story and seldom "proves" anything. Data should inform human judgment, but should not replace it.

School improvement requires an organizational culture where the judicious use of evidence habitually drives the day-to-day practice of teachers and administrators, and the collective decision-making of faculty or community groups.

References

Holcomb, Edie L. (1999). Getting Excited about Data: How to combine people, passion and proof. Thousand Oaks, CA: Corwin Press.

Jamentz, Kate. (2001). Accountability dialogues: School communities creating demand from within. San Francisco: WestEd Publications.

Jamentz, Kate. (2001). Teaching to quality: The instructional demands of standards-based reform (Item No. 39-0186). New Jersey: American Federation of Teachers.

McCarthy, Betsy & Cathy Ringstaff. (2000). Evaluation of Kyosei: A partnership for standards-based reform (Year 3). Prepared by Western Assessment Collaborative at WestEd under grant from Stuart Foundation.

Olsen, L.; Jaramillo, A.; McCall-Perez, Z.; & White, J. (1999). Igniting Change for Immigrant Students: Portraits of three high schools. Oakland, CA: California Tomorrow.

Wellman, Bruce and Laura Lipton. (Winter 2000). "Navigation: Charting a path through a sea of information." Journal of Staff Development, Vol. 21.

Kate Jamentz is director of the Western Assessment Collaborative at WestEd.