Making magic: linking assessment and accountability: we must be able to articulate our assumptions about assessment and accountability and be able to test the value of those assumptions.

It's broken. The link between assessment and accountability is broken. In theory, if assessment and accountability are linked together, student achievement will rise. That has been the working theory of educational policymakers at the state and federal levels in recent years. Require annual assessments of student achievement and hold schools and districts accountable for demonstrating gains.

The theory worked for a while. Many--if not most--schools achieved annual increases in student achievement scores. California's Academic Performance Index, which measures gains in student achievement across the whole range of student performance levels, rose steadily. Some schools even saw gains in meeting No Child Left Behind's Annual Yearly Progress, although that only measures gains at the "proficient" level.

It seemed that the theory linking assessment and accountability was working, but the 2004 test results raise doubts about the theory. Newspapers reported the test results with headlines such as, "District fails exit exam," or "Test results flat." What happened? Assessment and accountability were linked, but achievement did not continue to rise. Is the theory wrong?

At the very least, the theory linking assessment and accountability is too simplistic, since it does not describe why or how the theory should work. In essence, it is a magician's black hat theory. Add assessments and accountability into a hat, pass the magician's wand over the hat three times, and out comes a rabbit--oops, I mean, achievement. The theory assumes that what happens in the hat is sufficient to produce achievement. But what was supposed to happen in the hat?

Awareness of the system

As educational leaders, we have operated on our beliefs about the contents of the magician's hat. The first item in the black hat was awareness of the assessment and accountability system. We began with awareness of the new statewide assessment tool, the SAT-9. We scrambled to find out what the test covered and to find tools that would raise awareness of the assessment for teachers and students. Some districts spent incredible sums on test prep companies that teach us how to teach our students to weed out incorrect answers and guess smartly. Was the magic in learning how to estimate knowledge?

Alignment to standards

Once the SAT-9 was administered and educators saw what it actually tested, we began to ask questions about alignment. How did the SAT-9 align with our newly adopted state standards in English/language arts, mathematics, history/social science and science?

The misalignment of the SAT-9 to our state standards was an obvious weak link in the assessment and accountability theory. The theory needed to be reworked. The magic must be in alignment. Creation of the California Standards Tests was the solution. The new working theory: Use standards-based tests as the foundation of an aligned assessment and accountability system and achievement will rise.

Achievement did rise--for some schools. We all were aware of the assessment and accountability system that was now based on a tight alignment between standards and the assessment, but still, not all schools attained achievement gains. So we asked ourselves, "Is there something else in the magician's hat?" Schools that had demonstrated gains answered affirmatively. "Yes, we have aligned our instruction to the standards and assessment."

Most of the gains were achieved at the early elementary grades and answers frequently focused on teaching beginning reading. However, since we were not all trained as first grade teachers, professional development must be the magic. With its full treasury, the state was able to offer professional development opportunities to train teachers to teach reading.

Professional development

The knowledge and expertise gained in these intense workshops and multi-year follow-up workshops and coaching achieved substantial increases in achievement for many students and schools, particularly at the elementary level, where teaching beginning reading is essential. Professional development in mathematics was also available to many teachers, with a strong emphasis on preparing students for success in algebra.

Few people questioned, however, which students studied algebra. In any case, most middle schools and high schools did not demonstrate substantial increases in student achievement. And professional development is expensive--both in dollars and time--and available to only a small portion of teachers.

Instructional materials

Then we asked, "Is there another way we can align teaching and the assessments that can be more widely available?" The usual answer included specific instructional materials. Proponents of one publisher's materials or another's argued strongly that the magic was in their favorite textbook.

For some schools there was magic in using instructional materials that aligned more closely to the state standards. However, other schools, even though the textbook lesson plans were faithfully followed, actually showed declines in student achievement with the new state-approved textbooks. Where was the magic for them?

What have we forgotten?

We have tried putting awareness, alignment, professional development and instructional materials into the magician's hat. What have we forgotten? Many districts tried time--increasing the time allotted for instruction on tested areas of the curriculum. Some districts tried focus--limiting instruction to only reading and mathematics at the elementary grades. Other districts tried leadership--removing principals if test scores did not increase. Often districts tried remediation and retention.

We cannot escape, however, the flat test scores published this past summer. We still have not made magic. What is our theory missing? What is missing from the magician's hat?

Some would argue that the theory is based on too few and too distant assessments. Tweak the theory to include curriculum-based assessments every six to nine weeks. Many districts have instituted this practice and have seen increases in student performance on these tests. Fewer districts have seen these increases consistently transfer or generalize to performance on the CSTs or the National Assessment of Educational Progress. Did students only learn to perform on the classroom assessments and not others also? Did the magic only happen in the classroom?

Others raise a concern that the state assessments are flawed. According to this position, the history of large-scale assessments shows repeatedly that test score gains level off after a few years. Awareness and alignment produce initial gains that become attenuated over time due to assessment limitations such as ceiling effects, limited feedback on student performance, unreasonable annual goals and weak calibration between test years and levels--particularly for tests such as the CSTs that change substantially from year to year.

While each of these issues may limit the effectiveness of assessment-based accountability systems, they are not insurmountable. With sufficient commitment of financial resources and lead time, the state could adhere to the most rigorous requirements for test development.

Performance results could be shared in ways that provide user-friendly and readily available charts that show student gains over time. Detailed item analysis information could be provided so that teachers and parents would have a clearer understanding of specific standards students need to master. Improvement targets could be based on individual student growth or gains over time.

We know all this could happen because it already does happen in some states, for some tests and for some accountability systems. We could develop the premier assessment and accountability system. Would that be the magic?

Opportunity and instruction

Developing and implementing a premier assessment and accountability system will help improve student achievement, but it is not a sufficient answer. In "What Works in Schools," Marzano (2003) provides a comprehensive review of the literature on the school-level, teacher-level and student-level elements of high student achievement.

As educators, we have direct responsibility over two of the three elements: school-level and teacher-level. The top school-level element is opportunity to learn; or, if you are the student, what you get to study. The No. 1 teacher-level element is instructional strategies; or, if you are the student, how you get to learn.

Opportunity and instruction, or what the student studies and how the student learns, are two critical missing elements for making magic. Opportunity to learn and instructional strategies are tightly bound together in the instructional choices and practices of our schools.

Access to instruction

Access to instruction is the first right for opportunity to learn. How often do we use prior performance as evidence that a student is not able to achieve-and by extension, should not be enrolled--in algebra, chemistry or an AP course? Seldom do we ask ourselves what changes in our instructional practices would better prepare the student to achieve in these advanced courses. Despite what appear to be our best student-centered intentions, we deny students access to opportunities to learn.

Comprehensive Instructional programs

Comprehensive instructional programs is the second right of opportunity to learn. In any given class, students may or may not have the opportunity to learn the subject well. For example, a reading program choice that de-emphasizes needed instruction in reading words hinders a student's opportunity to read with ease and fluency. An approach that de-emphasizes or delays instruction in comprehension strategies limits the student's opportunity to understand well the message of the reading.

Rather, what is more effective is a comprehensive program that teaches all essential elements of reading to students from the youngest ages. In the wild swings from whole-language to a phonics-based program, many have overlooked one aspect or another of effective reading instruction.

Beginning readers do need to pay attention to sounds, letters and words. And, even the youngest child on her mother's lap wants to know what happened after the wolf huffed and puffed and why the wolf wanted to blow down the little pig's house.

Other curricular dichotomies abound. Mathematics problem-solving without skill in number sense is difficult; number sense without problem-solving is meaningless. Scientific experimentation without a background of scientific facts and theories does not build on the knowledge already attained; past knowledge without experimentation and new applications is static.

Effective Instructional practice

Effective instructional practice is the third right for opportunity to learn. Marzano (2003) identifies and ranks nine instructional practices that have demonstrated effectiveness in improving student achievement. Identifying similarities and differences and summarizing and note-taking lead the list. Why? Each of these requires extensive effort on the part of the student to understand and make sense of what is being learned, i.e. to assume ownership, responsibility and a degree of self-direction of learning.

The student implemented effort to organize and synthesize knowledge is likely also the key element underpinning the effectiveness of teaching writing. Doug Reeves (2000) emphasizes that schools that implement a consistent focus on writing demonstrate increased student achievement, even in subjects beyond language arts. The task of writing relies heavily on the student's effort to plan, organize and synthesize information--the same types of skills used in identifying similarities and differences and in summarizing and note-taking.

Where students are taught, supported and expected to develop these capabilities, learning occurs, and its evidence is seen across multiple domains.

Initiative and control

In a recent paper, Richard Elmore (2004) argues that agency--or the initiative and control one takes to accomplish something--is critical to the leadership and learning demanded to achieve accountability goals. We learn when we take ownership of our learning. Teachers and students engage in a duet that facilitates not simply the transfer of knowledge, but the transfer from the teacher to the student of the responsibility of agency, i.e. the assumption by the student of the efforts to engage in the work of learning.

Elmore argues that agency is also critical to the learning of adults in an accountability system. Principals and teachers who recognize and implement effective instruction must lead the transfer and ownership of this knowledge to all the teachers at their site. District leaders who recognize effective schools must support the transfer of agency around this knowledge to school leaders. States that recognize effective districts must support the work of other districts to learn the approaches and reforms needed to achieve accountability goals.

But, particularly in the case of adult learning, transfer of agency is problematic. In contrast to much of school learning, the knowledge required to transform our educational systems--including what to do, how to do it, who should do it and how to help others do it--is elusive. we recognize aspects of excellence, but we are not experts in creating excellence. we see the magic, but we do not know how to create it.

The magician's wand

We must become educational learners in order to create the magic we desire. We need to work together to assume agency for our own learning from our experiences, and for the research and the experiences of others, so we can learn how to create schools where all students have the opportunity and the instruction to assume agency for their learning.

This is difficult, complex and challenging work that must occur across all levels of the educational system--state, district, school and classroom. It requires moving from designing accountability systems that assume a link between assessment and accountability to articulating our assumptions and testing their value. It requires that we learn to wave the magician's wand purposely, without having to say, "Hocus pocus."

You can write for Leadership magazine!

Share your expertise by contributing an article to one of these upcoming issues. E-mail your submissions to editor Susan Davis at [e-mail].

March/April 2005: High School Reform: How large high schools can be more responsive to student needs * Improving graduation rates * Providing ongoing professional learning for principals * Improving access to a quality curriculum * Creating partnerships. Deadline: Jan. 3, 2005

May/June 2005: Fostering a positive learning climate: Improving school culture * Enhancing staff morale * Finding the joy in administration * Visionary leadership * Negotiating win-win contracts that focus on student achivevement * Leading in times of crisis * Finding time to do it all * Building relationships * Strategies for making tough decisions. Deadline: Feb. 23, 2005

References

Elmore, Richard. (2004). Agency, Reciprocity, and Accountability in Democratic Education. Cambridge, MA: Graduate School of Education, Harvard University and Consortium for Policy Research in Education.

Marzano, Robert J. (2003). What works in schools. Translating research into action. Alexandria: VA: Association for Supervision and Curriculum Development.

Reeves, Douglas. (2000). Accountability in action: A blueprint for learning organizations, Denver, CO: Advanced Learning Press.

Linda R. G. Kaminski is the chief academic officer for the Santa Monica-Malibu Unified School District and president of ACSA's Curriculum, Instruction and Evaluation Council.