Data digs: everybody's talking about data--getting it, using it, sharing it. Unearth the realities of data-driven decision-making and how it can be just what your district needs to help at-risk students. (Cover Story).

Growing up on a farm in Gilman, Wisc., a town with just 400 people, Tim Micke learned that experience is the best teacher.

Take the time he was out in a field and the tractor ran out of gas. As he walked the mile round trip to and from the gas tank, Micke knew he would never start farm work again without checking the gas gauge.

Now an associate principal at Prairie River Middle School in Merrill (Wisc.) Area Public School District, Micke views school data as he did the gas gauge. When there's a decision to make, he seeks numbers first. "With the data in front of me, I need to start digging in. I need to know the `why' of the [problem]." That dedication to data may explain why his colleagues have dubbed him "Data Man."

And Micke is a statistic himself--one of a growing number of administrators who rely heavily on data. "Traditionally, decisions have been made on informed intuition. [Data-driven decision-making] is relatively new," says Mike Parker, assistant director of the Center for Accountability Solutions at the American Association of School Administrators.

However, a leadership turnaround of this magnitude is not easy, and often even administrators sold on the idea don't use data well. Anthony Dallmann-Jones, director of the National At-Risk Education Network, says administrators "are so very weak in this area, not realizing that it would solve many of their dilemmas if they would just rely on `good data' for program planning, implementation, operations and modification. It sounds so simple, but many are truly not up to par in this area."

Testing the Soil

The problem is not a lack of data. "Schools have always had tons of data, but they had no way of organizing it and translating it into something ... useful at the classroom level," says Francis Barnes, superintendent of Palisades School District in Kintnersville, Pa. His district piloted Quality School Portfolio, a free decision support system, to gain skills in using disaggregated data. "It's far more complex than many people realize.... It isn't something that you adopt. It's something that becomes a part of existing processes in the district," Parker says.

And using data in this way has just not been part of education's culture, adds John Kline, director of planning, assessment and learning technologies at Fort Wayne (Ind.) Community Schools. "We considered ourselves artists more than scientists," he says.

For students who are at risk of not succeeding in school, the need for data-driven decisions is especially great. "I think schools just wish these kids would go away," says Dallmann-Jones. Teachers enrolled in his at-risk education graduate classes at Marian College, located in the Milwaukee area, often learn the hard truth about their districts: "I think [administrators would] be really happy if the bus would just pull up and take all the at-risk kids across town," he says.

When Dallmann-Jones evaluates alternative at-risk programs, he requests figures in several categories. Answers like: "We could probably find that but it's not at our fingertips" reflect administrator attitudes toward at-risk education, he says.

Adding to the problem is that "the confusion and cloudiness created by varied and poor defining" of at-risk makes it difficult to say whether the at-risk student population is growing, Dallmann-Jones says. If one common definition is used, dropout rates for non-Hispanic whites and for blacks have declined substantially between 1972 and 2000. The rates for Hispanic youth, meanwhile, have stayed the same.

Good news? Only if your definition of at-risk is narrow. Looking at an expanded use of the term--which includes anything that can impede a student's maximum social, emotional and intellectual growth and development--Dallmann-Jones says the numbers of students at-risk is on the rise. His answer is based on preschool participation, which the National Education Goals panel identified as an indicator of progress toward ensuring that all children start school ready to learn.

Beyond the definition issue, misconceptions about intervention also hurt at-risk students. "Some people think your risk status is well-known early on," says Margaret Terry Orr, an associate professor of educational administration at Columbia University's Teachers College. "But [it] can be influenced by ... personal or family circumstances that may shift a lot throughout your school career." Orr says educators also worry about students being "permanently labeled at-risk and then shuffled to options that narrow their future opportunities." Tracking dropout rates is another common fear, she says. "For a long time, people didn't want to look at dropout rates bemuse it signaled failure at the institution."

Planting the Seeds of Change

Once district leaders commit to using data to help all students achieve, data sources become a focus. Experts and administrators agree that academic, attendance and disciplinary data are especially telling.

Academic data might include grades, test results and SAT scores. Palisades also uses qualitative data from student interviews. The interview data tracks a different area each year, Barnes says. Reading analysis data, for example, was collected one year to determine how well students could make connections between a story and their own lives or the world.

When doing surveys, Dallmann-Jones cautions, "You need some numbers you can hang hats on." This might mean, for example: giving parents an assessment instrument or chart in September to help them accumulate useful data, such as how many hours a night their children do homework; or, asking parents to rate their feelings toward the school on a scale of 1 to 10. This can be more effective than simply asking an open-ended question.

The book Using Data to Close the Achievement Gap (Corwin Press, 2002) suggests these additional data sources and questions: teacher schedules (Which students get the most effective teachers?), teacher plan books (Are teachers planning instruction to meet standards and at what level?), student schedules (How much time is non-academic?), counselor records (What kinds of counseling are different student groups receiving?), meeting agendas (What does the school focus on?), homework time allocations (What is the amount of homework done at home or in school?), and budgets (How are resources used, and has it had an effect on student achievement?).

Every student considered at-risk should also have a student profile with a needs assessment that examines academic, social, psychological, health, emotional and behavioral needs, so instruction can be tailored, Dallmann-Jones says. If administrators start to gather data with the intent of developing a needs-based curriculum, "you ... see reparation start to happen down the line in schools," he adds.

With the influx of data at all levels of a district, it's important to make sure you're collecting useful data. "Last year's data is too old," Parker says. "It really [only] provides districts with a starting point for the upcoming school year." If educators look at the data two years from now, Dallmann-Jones says, "Will they be able to use it? Or will they just say, `Oh, that's interesting'?"

Growing the Data

Teachers in Whittier (Calif.) Union High School District have instant Web access to classroom data through their Educator's Assessment Data Management System, which disaggregates data on the ESL population and other subgroups. The system allows teachers "to get student profiles and make curricular and intervention decisions," says Nancy Bosserman, director of literacy, staff development and accounting.

In addition, Whittier has used My Access, an online writing assessment tool from Vantage Learning, on certain student populations during summer school for the past two years. Bosserman says the immediate feedback helps teachers in making instructional decisions and helps students in understanding their progress. This year, the district is expanding its use of the tool to track all students in grades 9 to 11 as part of its focus on literacy.

Collecting data that links to district goals is key, administrators and experts say. Other data-gathering tips:

* Develop a district plan for a data-driven culture. Don Viegut, director of curriculum and instruction in Merrill, notes that it takes time to create that culture. Four years ago, he says, "We more or less played school the way it was historically played." Since then, the use of data to make decisions has been promoted, and Viegut estimates that 70 percent of the district personnel are now on board. "Some people who were [initially] resistant are our best salespeople," he adds.

* Do an audit of data types, location, format and value in decision-making, says Parker. This can be the first task of a district-wide data focus group. Made up of teachers and administrators who are advocates of using data, the group can help institute a district data-use policy, Dallmann-Jones says.

* Start with a broad question, "What do we need to know?" Then ask what available data sheds light on that question. "If you start with just-the data, you'll get lost in a blizzard of information, and in fact you'll be doing something that is unscientific. You will look for facts to support your preconceptions," says Charles Ott, superintendent of School Administrative Unit 56 in Somersworth, N.H.

* Involve teachers. Micke uses aggregated data from Merrills teacher self-assessments to figure out where improvements are needed overall. The self-assessments include 60 questions in six areas-professionalism, planning, curriculum, preparation, learning strategies and assessment.

* Invest in professional development. Necessary data skills include how to analyze and interpret data, how to use a decision support system or other technology for analysis, and how to develop educators' abilities to respond appropriately to data, says Parker. Dallmann-Jones suggests having special education teachers show at-risk program teachers how they do student profiles. Whittier brought in educational specialists to talk to administrative teams about the power of data.

* Recognize that tracking at-risk students now requires action later. If you want to know what students are in the bottom 10 percent, for instance, Orr says you have to think of the intervention to follow. Otherwise "there's no sense in flagging [them]."

Using Your Data Garden

The three crucial areas for using data to help a-risk students are needs discovery, intervention and program development.

A county-wide survey of 6,000 children was used by Antigo (Wisc.) Unified School District to discover children's skill areas and lifestyle choices. Superintendent Lance Alwin reports that only about one in 10 children identified a meaningful relationship with an adult "It just blew our socks off," he says. Administrators disaggregated the budget and found at least $4 million being spent on remediation--and it was mostly the kids without the adult relationships who needed this extra academic attention. "These kids were falling not only through the cracks but falling off the face of the earth," Alwin says. By investing in early childhood education, the district hopes to see the remediation budget needs decrease over time.

Intervention is another area where data can help. In Fort Wayne, where Kline says data has been a major focus for more than five years, analysis of quarterly exams was especially beneficial at Indian Village Elementary School, which three years ago had been identified for probation by the state.

Using exam results, teachers identified students and their individual learning needs for summer school remediation, which grouped students by focus area. In addition, the school and each of its teachers and students set achievement goals. Assessment results for state-tested third graders have risen dramatically: from 52 percent passing the math exam in 1999 to 95 percent in 2001, and from 38 percent passing the English exam to 82 percent, Kline says. The school is now considered tops in the district and one of the top five in the region.

Fort Wayne middle and high school students may be recommended for one of two alternative programs, depending on what types of problems they've experienced. Those in the district's internal program, Kline says, are primarily those who would have been expelled for discipline reasons. The external program, which is contracted to a private high school and offers nontraditional afternoon and evening hours, helps dropouts who felt school wasn't right for them.

Data analysis can bring about program changes, as well. SAU 56's new Bridges to Success initiative, which Ott says was designed to end social promotion without relying on retention, was possible because of data collected as evidence of need. A year-long study that started with the question, "Who is failing?" uncovered failure rates by graduation class, gender, special education status, ESL and other factors. Disciplinary data told administrators not only who was getting in trouble but also what hours of the day, days of the week and weeks of the year saw spikes in disciplinary referrals.

The study helped the Bridges program obtain unanimous support from the school board, the city council and the community. "Some people were unhappy that we unearthed evidence that was not flattering to our school system. But it became an impetus, a galvanizing force to say, `How do we move forward in a direction that honestly and effectively addresses these problems?'" Ott says. Bridges to Success has three parts: expanded summer school (which started with about 20 students and includes 127 this year), the Home Base program for freshmen (which personalizes instruction for students who are entering high school ill-prepared) and full-credit afterschool courses (currently for students who have failed a course, but Ott hopes they'll change to enrichment courses as achievement increases).

Some advice for using data for needs discovery, intervention or program development:

* One number doesn't tell all. At Prairie River, a letter is automatically generated to inform parents of academic, behavior or attendance issues. Before it's sent, Micke looks for possible reasons for the issue, such as the child missing school for surgery. If it's something the school is aware of, he adds a note to the form letter. Or, he might call the parent to discuss the situation first. "We promote objectivity, but we also like using that gut feeling once in a while," Micke says.

* Consider ranking students on an at-risk scale. Micke does this on a 1 to 5 ranking system, where 5 is highly at-risk. ("Believe it or not, that ranking system I took from the U.S. Forest Service fire danger ranking!" he says.) Students with a 3 or higher receive extra attention from teachers and guidance counselors. The rank is just one item included in student intervention folders, which also contain information on anything the district has done to help a student during his or her school career-from parent meetings and tutoring to behavior modification plans and summer school.

* Look for patterns by disaggregating data. Question your assumptions about race, gender and ethnicity and ask yourself if something about the educational experience could be contributing to the identified achievement gaps, Orr says. "Be open to saying, `Is there something we can do differently with our educational resources, rather than [blaming] the child or the family?' "In Palisades, this kind of thinking led to the addition of a few school days throughout the year attended only by students who are not progressing at the level of the standards. The students were identified by disaggregating gender, race, grade level and achievement data, Barnes says.

* Don't get lost in the data. When Whittier's English language learner team met last year, Bosserman and the other organizers brought in data they thought would help. But for every figure shown, the participants wanted three or four more. "After we overwhelmed ourselves, we said, let's pull in the reins here. What do we really need to know? What are our goals?" she says.

Spreading the Wealth

Administrators who are well-versed in data agree that talking to others--in the district, in the community and in other districts--makes a big difference in capitalizing on the power of data.

In Merrill, administrators meet off-site for an annual "data retreat" to study test results and other data and make decisions about teaching and assessment. Viegut says this tells the "district's story" and helps in illuminating the big picture. Meetings are also held every two weeks throughout the year.

Merrill reaches out to the parent community through building-wide surveys and focus groups. Once a quarter, parents are invited to discuss school data in small group workshops. This gives administrators a chance to learn parent expectations for the school. In addition, Micke says that communication through these efforts and individual conversations help parents realize the importance of remaining involved in the schools, even as their children get older.

Ott uses the graphic reports of his district's QSP system to communicate data clearly to parents. For example, a speedometer graphic illustrated the district's goal of 90 percent of third graders reading on grade level. The speedometer needle, at 67 percent, showed how far the district had to go. "A major goal of ours [is] to get this information off of our desks and to the public. Our belief is that you become transparent-you communicate the good, the bad and the ugly openly and actively," he says.

To communicate with other districts, SAU 56 formed a quarterly regional consortium to share ideas for improving the management of information. "There's just so much information out there that you need partners to synthesize [it]," Ott says. "You start constructing meaning with other people who have a similar mission."

Micke continually seeks professional development opportunities to help him with data-driven decisions. "Keep making connections with other people and fine-tuning what you already have," he advises. "And if someone asks for help, you give it to them."

AT-RISK, ALL AROUND

While schools have traditionally defined at-risk students as those at risk of dropping out, experts say the definition should be broader. If the business of schools is to increase kids' chances of succeeding, then any student at risk of not succeeding should be considered at-risk, says Anthony Dallmann-Jones, director of the National At-Risk Education Network. "Not staying in school is one part of that. [Students can still be] in school but [have] diminished chances of succeeding," he says.

With this expanded definition in mind, these facts from The Children's Defense Funds' State of America's Children Yearbook 2002 may be startling:

1 child in 3:

* is born to unmarried parents

* will be poor at some point during childhood

* is behind a year or more in school

1 child in 4:

* lives with only one parent

1 child in 5:

* is born to a mother who did not graduate from high school

* under the age of three is poor now

1 child in 8:

* has no health insurance

* never graduates from high school

* is born to a teenage mother

* lives in a family receiving food stamps 

1 child in 12:

* has a disability

1 child in 24:

* lives with neither parent

1 child in 60:

* goes through a parent divorce in any year

DELVING INTO DATA: RESOURCES

* Center for Accountability Solutions, an American Association of School Administrators' group that helps school leaders gather, use and report meaningful data on student, school and district performance. Web resources include: Using Data to Improve Schools: What's Working; a report on training in data-driven decisions; lists of software tools (Quality School Portfolio and others), research and district data news; and information on a networking program for administrators interested in data. www.aasa.org/cas

* The National At-Risk Education Network, a non-profit agency dedicated to promoting the success of at-risk youth and supporting at-risk educators. Look for a self-assessment set of standards for at-risk education in November; districts that meet all nine standards can receive a certification award. Web resources include at-risk program profiles, news and a summary of the standards, www.atriskeducation.net

* Books on data and at-risk education (all from Corwin Press). Evaluating School Programs: An Educator's Guide (1992), Getting Excited About Data (1998), How to Build the Best Questionnaires in the Field of Education (1996), Programs for At-Risk Students: A Guide to Evaluation (1993), Special Education Programs: A Guide to Evaluation (1992), Tracking Your School's Success: A Guide to Sensible Evaluation (1992), and Using Data to Close the Achievement Gap (2002). www.corwinpress.com

Melissa Ezarik, [e-mail], is features editor.