Data Analysis by Walking Around.

One district's unique process for assessing the impact of teaching on student outcomes

What happens when you walk around schools intent on capturing the voices of roughly 1,300 students--half the student body in your district--not once, but twice a year through structured personal conversations? What do students' voices add to our knowledge about instruction and learning?

Palisades School District, located in a rural area north of Philadelphia, has a vision of becoming a data-driven decision-making district.

Our vision of data worth collecting extends beyond the traditional realm of scores on standardized tests. It embodies a cross-section of students' voices that is both broad and deep, providing a window into how students approach learning and how well they think they have mastered material. Our task is unique, we believe, because we invite educators from other districts as well as our own staff to gather feedback--one student at a time--in personal one-on-one conversations.

Over the past 2 1/2 years, we walked through our buildings culling data from short personal conversations with students to gain a qualitative dimension reflecting skills students in our district were learning and what knowledge they were applying.

Even the higher-ups in the district are not exempt from the walk-through process. Both of us personally have handled one-on-one conversations with more than 100 students during each fall or spring set of interviews and we lead orientations for the interviewers.

We knew that qualitative comments would not replace the numbers we have traditionally collected and continue to collect. But we suspected that conversations would help us gain a context for those numbers and test scores. Students' voices could help us to see better why and where some students are having trouble. Furthermore, they might help us pinpoint where students who might otherwise have been overlooked were not being sufficiently challenged.

Unfiltered Evidence

What we learned were not all surprises; students' unfiltered comments of their work and understandings enlightened us, enthused us and, in some cases, alerted us to instructional and testing issues that needed schoolwide and/or districtwide attention. We have no doubt that their comments have had an impact that will, in time, translate into higher achievement. We now have evidence that the ideals contained in our school district's mission statement are being realized.

Take, for instance, the rich responses students gave us during our walkthroughs when they were asked to tell us how they make predictions while reading. We asked: "What clues do you use to make predictions when you read a story?"

From a 1st-grader, we heard: "[I] guess what it will be about by looking at the cover and pages until the end." From a 3rd-grader: "I guess and then check in the next chapter to see if I'm right." And from a 5th grader: "I ask myself questions about what I think will happen." Our walk-through process has generated valuable data for use by teachers and administrators.

After the walk-through each fall, our teachers analyze the data to determine what skills and understandings are secure for children and which areas require increased attention for the remainder of the year. Then in our spring walk-throughs we gauge whether students have improved in their understanding of the specific areas in which they had trouble in the fall. Teachers, working in small teams and individually, use the data to target their instruction.

We continue to rely on other means of measuring students' understanding of what they have mastered by administering standardized tests. This leads to test score numbers that, with growing regularity, are published by the news media, reported by state education departments and placed on school system Web sites as a means of documenting student learning and holding educators and schools accountable.

That type of accountability will remain with us. However, we believe the walk-through process is an excellent tool for student assessment. It also helps satisfy those who argue there is more to a student than a standardized test score.

We also are making better use of test data by using the Quality School Portfolio software pilot program sponsored by the AASA Center for Accountability Solutions and UCLA's National Center for Research on Evaluation, Standards and Student Testing, also known as CRESST. While this program enables us to see trends and patterns that help us to make instructional modifications, we find the actual voices of students to be another important means of determining student understanding.

Initial Forays

The walk-through process has evolved over the past 30 months into its present form. When we initially considered the purpose of the walk-through we talked with staff at the Institute for Learning at the University of Pittsburgh. They helped us define the purpose of the walk-through.

The walk-through focuses the participants on improving the core of educational practice. Richard Elmore, a professor of educational administration, planning and social policy at the Harvard Graduate School of Education, defines core educational practice as how teachers understand the nature of knowledge; the student's role in learning; and how these ideas about knowledge and learning manifest themselves in teaching and class work.

In the initial year, Palisades teachers, administrators and parents conducted the walk-throughs. But because of the time-intensive nature of the process, in the second year we needed to expand the pool of adults to help with interviews.

We also realized that educators in neighboring districts could provide added talent and perspective during questioning. In fact, many districts are interested in participating in the process in our district with the idea that they may introduce something like it in their home district. Two neighboring districts now have walk-throughs in place, and one district is in the study phase of how to implement the process. As a result, we now receive more requests from outside educators to participate on walk-through teams than we can accommodate, so we maintain an ongoing waiting list.

Last fall we interviewed approximately 1,300 of our 2,200 students by inviting educators from outside the district to participate in walk-throughs with us. All of the response sheets had questions that were created by our teachers and given to participants to record the students' answers.

Teachers developed several goals as a result of their analysis. These included conducting more research on the use of questioning techniques and sharing successful practices with each other. We also scheduled a training session for faculty on how to help students make text-to-text and text-to-self connections. So the process keeps getting fine-tuned.

Our Process

Obviously school districts can take different approaches for implementing walk-throughs. Palisades uses a unique approach that requires teams of visiting educators (as well as some of our own educators, parents and board members) to engage students in a one-on-one conversation about what they are learning.

This interaction typically takes 15 minutes and is guided by a consistent set of questions that have been compiled jointly by our teachers and administrators and that is tailored to the students' general educational level (primary, intermediate, middle and high school).

Student responses are documented in a manner that lends itself to both a quantitative and qualitative analysis. However, the responses are used primarily for qualitative analysis of the student response data.

Approximately two weeks prior to the walk-through, participants within and outside the district receive a confirmation letter with some general information about the upcoming orientation and interviews.

On the day of orientation, each walk-through team of approximately 25 adult interviewers meets as a group with the principal to gain an overview of the building logistics. They then hear from the superintendent and assistant superintendent about the interview and data collection process and the schedule. Interviewers are assigned to rooms where they each randomly select and individually interview approximately 20 students.

The walk-through team reconvenes about an hour before dismissal to share observations, strengths and areas in need of attention. These comments are summarized and shared with the faculty at an end-of-the-day briefing. All the original data are given to the principal to use with the faculty as a basis for influencing instructional strategies for individual students, groups of students or professional development. We have tried to quantify the data, but the staff has indicated to us that the most valuable information is the unfiltered comments from the students.

Applying Principles

In the first two years, Palisades maintained a districtwide focus on "clear expectations," one of the Principles of Learning, developed at the Institute for Learning, University of Pittsburgh. This principle delineates the need to define clearly what students are expected to learn and then clearly communicate these expectations to all stakeholders--students, parents, staff and community members.

What this meant was that on our district s walk-throughs, we focused on clear expectations for writing and math problem solving. Participants looked for evidence of how well the students could describe four key elements: (1) their work as it related to writing standards and math problem-solving standards; (2) the rubric or process used to score their work; (3) their understanding of what they needed to do to improve their work; and (4) their understanding of how their work related to what they had previously learned and/or would be learning in the future.

In the third year, walk-throughs focused on reading analysis and interpretation. Participants are focusing on the Principles of Learning called "socializing intelligence" and "accountable talk." These principles support beliefs that students can become more intelligent in an environment where they are given opportunities to discuss their perspectives in small groups and where. they are taught terms, activities and strategies to use in this discussion-oriented process.

The walk-through process at Palisades is not a one-time event. At each of the five schools, baseline data is collected in the fall and then again in the spring to chart students' progress for a total of 10 walk-throughs districtwide each year. The recurring nature of the interview process has helped staff and administrators keep a focus on instruction all year long.

An important piece of this process then becomes the use of the walkthrough data by teams at the school. During the fall walk-throughs, principals work with either the entire faculty or a team of faculty members to review the data and develop action plans to put in place as soon as possible. This ensures that areas needing attention are addressed before the spring walk-through. Once the spring event is finished, the teams analyze the results to see if the interventions they put in place have made a difference.

From the feedback from student interviews, we have introduced four interventions to date that appear to have made a positive impact. At all grade levels, we learned that students needed a firmer grasp of mathematical terms related to problem solving. We also found that students felt they would benefit from seeing what high-quality work looked like. Today we have increased the amount of student work in all grades that is displayed publicly with accompanying standards and rubrics.

Grade-specific concerns also surfaced. In grades 6-8, we found students needed more help to move beyond text-to-self connections in reading to text-to-text connections. Palisades is devoting professional development to this area. In addition, high school students as a group voiced concerns about responding to so many different standards and rubrics. As a result, we developed schoolwide standards and rubrics for math problem solving and reading analysis and interpretation that are implemented in all disciplines.

Parent Communication

Bringing parents up-to-speed on the goals of the walk-through process is a continuing effort that begins at Back-to-School night in September. At that meeting, parents hear specifics on the district's prime academic focus concerning improved student performance on state and national New Standards Reference Exams. We inform them of the strategies we plan to use to achieve that goal, including how information from the walk-through has pinpointed strengths and areas of concern.

Some of our best communicators about the walk-through process and its relevance to our annual focus for the year are, not surprisingly, parents themselves if they have personally been involved in the process. The Back-to-School night presentation includes these active parents' firsthand accounts of the walk-through process to verify that the data gathered is important.

Parents also participate on building leadership teams where they help make decisions about the use of data. In accordance with our district's mission statement, parents are included on committees at all levels and in decision-making throughout the district (personnel, curriculum review, strategic and action plan teams, etc.).

Our first walk-through was in an elementary school, conducted by three central-office administrators. As could be expected, our staff had some anxiety, especially knowing that a superintendent, new to the district, would be conducting interviews with students. As one teacher said, "I'd feel more comfortable being interviewed than having students interviewed. You never know what they'll say!"

We implemented the process with only a few weeks' notice, and we didn't distribute the questions in advance, which increased staff concern. At the second walk-through in another elementary school, we did distribute the questions in advance and that helped to alleviate concerns. We are fortunate to have a talented staff whose anxiety stemmed from their conscientiousness and concern about having children prepared.

While some concern remains today, there's also an appreciation for the many positive comments made by visitors from outside the district. There's also an understanding that the focus is on student understanding, not on the instructional delivery, which could be confused with teacher evaluation.

Unexpected Pluses

If you visit one of our schools soon after a walk-through has taken place, you can feel something special in the school climate. We believe that positive vibes result from this process. Our administrative team believes that everyone in the school district, but most certainly students, are saying, "Notice me!" The walk-through process is the most powerful qualitative assessment instrument that we have used that recognizes individual students by placing them in one-on-one interaction with an adult to reflect expressively the knowledge and skills they have gained.

In the words of our students: "You get used to talking about your work. If you can't explain it to somebody, then it's not worth anything." "It's scary at first, because it feels like they are quizzing you, but then it feels good to be able to answer the questions." "It can make you realize how good you feel about math." "Smart students have rubrics in their minds; I don't have a rubric in my mind, but I can become smart by using a rubric."

Their words also can be humbling for adults. As one student put it: "It's good that they know what we're doing."

In addition, our school district has benefited in unanticipated ways. We have gained wonderful publicity through stories told by individuals from the community and other school systems who have visited our schools to participate on walk-through teams. Our community's understanding about our goals has grown along with their support. Our children have enjoyed having so much attention through the opportunities to talk to caring adults about their work.

The ultimate gain, of course, is improving student achievement and becoming more consistent with our district's mission statement: "Palisades School District is dedicated to the greatest possible development of all its students. Our mission is to enable all students to master the academic skills, concepts and knowledge to ensure their participation in our democratic society as life long learners and productive citizens. Working together as a community, we will support and encourage all to aim for excellence to think critically and creatively to take risks while solving problems, and to appreciate and respect other individuals and cultures."

Francis Barnes is superintendent of the Palisades School District, 39 Short Drive, Kintnersville, Pa. 18930. E-mail: [e-mail]. Marilyn Miller is assistant superintendent with the Palisades district.

Resources

Francis Barnes recommends the following works for those who would like to know more about the walkthrough process:

* Change Forces: The Sequel, by Michael Fullan, 1999, Falmer Press, Levittown, Pa.

* "Getting to Scale with Successful Educational Practice," by Richard F. Elmore in Rewards and Reform: Creating Educational Incentives That Work, edited by Susan H. Fuhrman and Jennifer A. O'Day, 1996, Jossey-Bass Publishers, San Francisco 

* "Walkthrough: Developing a Learning Community," Institute for Learning, Learning Research and Development Center, 1998, University of Pittsburgh, Pittsburgh, Pa.