Separating growth from value added: two academic models offer different tools for different purposes--measuring individual learning and measuring what affects learning.

Could it be true that we were shortchanging our brightest students?

Over the years a handful of parents in the Rochester, N.H., schools had complained that their high-performing children were bored by our curriculum. Because these reports were small in number, we were unsure whether they signaled a teaching problem or reflected students' desire to be entertained. These students were performing well above the 90th percentile on our standardized assessments, and the number of students moving to the top achievement levels was increasing.

Until the school district started using Measures of Academic Progress, or MAP, a computer adaptive assessment system from Northwest Evaluation Association that reports both status and growth scores, we did not recognize a disturbing pattern among our students. Rochester appeared to be effective at moving its low-performing students forward, but the high achievement scores for our most advanced students masked an alarmingly low rate of academic growth.

Rochester adopted MAP shortly after the district implemented the Education Value Added Assessment System created in the 1980s and early '90s by William Sanders, then director of the Value-Added Assessment and Research Center at the University of Tennessee. In our first year of the value-added system, the data identified some teachers who seemed to be more effective in challenging the advanced students, and one team in particular that was generating extraordinary growth for nearly all of its students.

As we studied the growth-related data from these systems, one thing became clear to everyone: These two ways of examining academic growth offer extremely different data and serve widely different purposes. While the terms growth and value added often are used interchangeably, they do not represent the same concept.

Identifying Needs

Historically, the Rochester district had relied on annual administration of standardized tests to measure student and school performance, which became a source of frustration as we tried to identify a growth estimate and derive formative information from the test. Standardized tests of the day had almost no relationship to our curriculum, offered little help in identifying individual student needs and did not generate meaningful growth data.

For several years, we had encouraged Rochester teachers to incorporate student data into their instructional planning, but often were rebuffed with the very reasonable and supportable assertion that the data we provided were not useful in creating learning plans for individuals and small groups.

Rochester's introduction to growth measures was the turning point. Recognizing there would be some sequencing differences between our local curriculum and the assessment's growth scale, our teachers still found the information generated by the new test, as well as the instructional resources linked to MAP, gave them their first effective tool for identifying individual student academic needs and creating a learning sequence that made sense. Flexible grouping and differentiated instruction finally became understandable, feasible and useful.

We found a real bonus from MAP related to our outliers--those students performing significantly above or below grade level. Because each item is anchored to a vertically aligned scale covering all grades tested, the adaptive test for each student was not tied to grade-level content. Teachers, for the first time, were receiving information about the actual performance level of these students and about content appropriate for their needs.

The grade-level equivalent scores we received from traditional tests long had been a source of confusion because, in spite of frequent explanations, nearly all of our parents interpreted an 8.5 grade-level score for their 5th graders as evidence the students were capable of and should be working on 8th grade content. This myth often was perpetuated by teachers new to the school district who didn't yet understand that grade-level equivalent indicated simply that these students had responded to items from their own grade level similarly to average students in the higher grades.

In contrast, the adaptive test selected item difficulty and cognitive complexity based on each student's response pattern, moving outside of grade-level constraints as appropriate. With this information, our teachers finally could create an instructional plan that would support challenging and reasonable academic growth for every student, not just identify a few proficiency targets that were impossible for some and already achieved by others.

Individualized Learning

Perhaps the most exciting aspect of growth measures and computer adaptive testing is the capacity to involve students in creating their own learning plan and tracking their own progress. In many districts that use Measures of Academic Progress, students tested in the fall will meet with their teachers to review the results to plan the year ahead. Students are shown a typical growth rate for those at their grade and performance level so they can select a goal for spring testing. Most students choose targets beyond the typical growth norm.

Additionally, the students are introduced to a set of learning objectives, aligned with their state's curriculum standards, that must be learned to achieve the target score. This serves as an advance organizer for the students and helps them to recognize new concepts important for achievement of their learning goal. Often, the biggest distraction during a spring testing session in Rochester is the sudden and loud exclamations as students view final scores and realize they have achieved their growth goal.

The main focus of currently available growth measures is formative assessment--providing data to inform instructional planning. Large-scale growth assessments can be used in conjunction with other formative tools, including frequent, short diagnostic tests related to a limited number of learning objectives. Teacher-created diagnostic assessments, even when constructed from pools of validated items, do not constitute a growth measure. However, they can be used in conjunction with growth measures to provide a more complete picture of student learning for short-term instructional planning.

Growth assessments also can be correlated with statewide summative assessments. These can be highly predictive of performance on state exams and can identify learning objectives to move lower-achieving students toward proficiency.

Estimating Effects

Value-added assessment is not a student assessment system. Rather, it is a way of analyzing results from student assessments over time to estimate the relative effect teachers and schools have on student learning. Value-added models are not intended to track individual student growth for instructional planning.

Several statistical models have emerged with a host of names, such as simple and layered mixed effect models; simple fixed effect models; hierarchical linear models; co-variate adjustment models; multi-variate models; and gain score models. Often a single value-added model is described by several of these names.

The various approaches are based on different assumptions and often have been developed to address a theoretical or observed weakness in another model. Among the differences are assumptions about whether and how the effect of a teacher persists into future years, procedures to control for differences in student background and previous knowledge, whether to make prior adjustments to status scores before imputing gains in assessments that were not designed to measure growth, and how to address missing data.

Regardless of their differences and unique statistical complexities, all of the models share a couple of basics. First, they do not rely on a vertically aligned growth measure, such as MAP, to work. Growth can be imputed through statistical treatment of status scores from various student assessments, which may or may not include growth-based assessments. Second, they recognize that a teacher is not responsible for what students bring to the table on the first day of class, but assume that teachers will have a major impact on what and how much the students will learn while in their charge.

Fundamentals of the most commonly used value-added model involve comparing scores for all students tested in a school district over multiple years, then creating a predicted score for each student based on the district average and that student's test performance history. The student's actual score for the most recent test event is then compared with the predicted score. If the average difference between actual and predicted scores in a class, grade or school is high enough to exceed the effect of chance factors and measurement error, whether in a positive or negative direction, then the difference is attributed to teacher or school effect. If the value is too small to account for those factors with a sufficiently high degree of confidence, no significant difference is recorded. Thus the value-added model is intended to identify the outliers, not to document small differences in typical teacher performance.

Teacher Impact

Since the introduction of William Sanders' Tennessee Value-Added Assessment System in 1992, the concept has been the subject of debate, study and controversy among researchers and practitioners. Much of this debate centers on the appropriateness of value-added models for teacher evaluation and high-stakes accountability.

Although the idea of using objective, data-based criteria for justifying teacher rewards and penalties is attractive, many researchers have expressed concern about the viability of this approach. They cite questions about the basic assumptions in value-added modeling and, more importantly, the difficulty of establishing causality without random assignment of teachers and students.

In spite of the teacher evaluation controversy, value-added models can provide useful data as schools and teachers look at different aspects of their operations. In Rochester, with a guarantee that we would not evaluate using these data, our principals were able to engage teachers in analyzing the learning patterns among their students.

In some cases, teachers were surprised to see their own classroom reflected our general finding that the most gifted students were not experiencing the same rate of growth as their less advanced peers. Some teachers saw good performance for students near the middle, with significantly less growth for students at both ends of the spectrum. Others found differences based on gender or other student characteristics. In nearly all of these cases the value-added data caused the teacher to ask more questions and search for ways to change the pattern.

One of the most notable findings for the school district was that all of the teachers from a single grade level at one school were seeing very high growth among nearly all of their students. The high growth was evident regardless of students' initial achievement level, gender, socioeconomic status or any other factors considered in the value-added analysis.

Further, when a new teacher was brought onto that team, whether transferring from another school or coming straight out of college, the accelerated growth rate appeared within the teacher's first year and continued in future years.

The value-added data led us to look at a number of factors, including how the team organized itself for instruction, its student grouping practices and involvement of the team's lead teacher in shaping the practices on that team. In the end, we identified what we believed to be significant factors in the consistent success of the team and we were able to share those factors with other principals and instructional teams.

Exercising Caution

Growth measures and value-added data both can provide powerful information to educators for improving instruction and increasing student learning. Each, however, must be used with an understanding of its strengths and limitations.

First and perhaps most important, both of these tools focus on only one aspect of a student's educational experience--academic growth. Most often, even this is limited to a few core subjects. Neither growth measures nor value-added data should be considered a complete measure of student progress or school performance. Practitioners need to triangulate the assessment data with at least two other measures, and preferably more, that address student engagement and aspirations, physical and emotional well-being, perceptions of school climate and community support, and a host of other factors.

Educators also need to recognize that neither growth measures nor value-added analysis in their current forms are easily applied to high schools. The multiplicity of courses with their associated lack of a single, linear curriculum path, create noise. Additionally, multiple scheduling models, which may compress course content to a single quarter or expand it to two periods per day for a full academic year, make it difficult to assess all students at the same time relative to course completion.

Finally, much thought and care must be exercised before using assessment data for teacher evaluation. Technical debates aside, as school administrators encourage teachers to use data to inform their instructional planning and delivery, they may want to ask whether using value added or other student performance data as an accountability club will discourage the use of data by teachers for their own planning.

In Rochester, we found the value-added data supported what we already knew and had documented through careful classroom observation and use of effective evaluation procedures. We did not need student assessment data to make the case for teacher improvement, discipline or contract non-renewal, and we certainly didn't need an almost inevitable legal debate focused on data accuracy rather than classroom performance and student needs.

Both value-added analysis and growth measures have contributed to strengthening the public schools in Rochester. Use of data among teachers and administrators is no longer an abstract exercise for looking at curious and misunderstood relationships. Rather, the accurate, high-quality data available to the district's teachers now reveal individual student needs and serve as a source for discovering what is working and what can be strengthened. *

RELATED ARTICLE: Implementing a growth model in a school system.

BY RAY WILSON 

Making the shift from traditional, long-standing, institutional behavior is difficult for any organization. For those of us in public education, a shift that must be aggressively pursued is abandonment of our single-minded emphasis on an assessment model that relies almost entirely on measures of student status at a single point in time for a more balanced approach that also values student growth over time.

The traditional use of assessment in K-12 education as a tool to prove rather than improve student learning is deeply embedded in practice. Most certainly, No Child Left Behind and its onerous regulations requiring one-size-fits-all testing have reinforced these traditional assessment practices as state and federal agencies use test results to hold schools and their teachers accountable. Unfortunately, these practices do little to serve student needs. Ultimately, they force us to reconsider the role of assessment in education.

Schools and school districts nationwide are facing what Ronald A. Heifitz and Donald L. Laurie coined "adaptive challenges" in their Harvard Business Review article, "The Work of Leadership." These are challenges that force organizations to clarify their values and deeply held beliefs and adopt new ways of thinking and operating when they cease producing the results they require. "Adaptive problems are often systematic problems with no ready answers," Heifitz and Laurie wrote.

They suggest that to make change happen, the executives of an organization must break a long-standing behavior of their own: providing leadership in the form of solutions. The authors suggest that solutions to adaptive challenges usually reside in the people within the organization.

The Poway, Calif., Unified School District has embraced the concepts articulated by Heifitz and Laurie. In 2001, we launched a districtwide effort to implement a growth-based assessment model. Along the way, we learned we would have to challenge long-held beliefs and practices about assessment. In the end, we derived that the primary purpose of assessment should be to accelerate student achievement and not simply measure it.

Questioning Assessment

To challenge our own views on assessment, the first question we posed required us to reflect on one of the fundamental tenets of assessment: Why do we assess students? Were we embracing an outdated view of assessment? And if so, how would we articulate a contemporary and more productive view?

In looking back, we came up with a lengthy list of answers as to why we traditionally tested students. These could be summarized as the following fundamental reasons:

* to be publicly accountable;

* to justify or produce grades;

* to produce consequences (reward or punish);

* to evaluate programs; and

* to provide feedback to the learner.

We wrestled with the moral purposes of education in society and the reasons why we became educators. Ultimately, the conversation led us to develop our Guiding Principle of Assessment, stating the primary purpose of assessment is to accelerate student learning. We subsequently developed corollaries or core values that defined specific institutional behaviors. For example, if the primary purpose of assessment is to accelerate student learning, then the results of these assessments must be timely, meaningful to the learner, aligned with instruction and show growth of the student learning over time.

The superintendent at Poway facilitated the process of articulating these new value statements. Working closely with our teachers' union, we developed a memorandum of understanding, which in turn charged a study committee of teachers and principals with the task of researching formative and summative assessment and formally recommending a plan of action to move toward a growth model.

Simultaneously, we challenged a second institutional belief about how best we could effect change. The traditional assumption is that strong top-down leadership from the district office creates meaningful change. However, our research into organizational change led us to conclude we were dealing with an adaptive challenge best addressed at a grassroots level by the people within the organization seeking innovative solutions.

The study committee embraced the research that suggested school building staff would be the most effective and efficient unit of change and recommended faculty and administrators at each school be empowered to develop their own site-based assessment plans focused primarily on creating an articulated system of formative assessments from grade to grade. To accomplish this step, the study committee recommended that all district-mandated assessments be suspended for one-year to allow each school to examine beliefs, determine needs and draft their site-based assessment plan. This, we believed, was an adaptive plan implemented to achieve deep and lasting change.

Finally, we embraced the belief that feedback to professionals and staff inside the classroom must be as important as feedback to constituents outside the classroom. In apparent conflict with NCLB mandates, our district concluded the single-minded focus on the statewide tests for school accountability was contrary to our beliefs about the role of assessment in education. Indeed, we concluded these state tests would not yield our desired outcome of accelerating all students' learning.

A once-a-year "autopsy" by the state can provide meaningful insights into program effectiveness and alignment of instruction, but these insights were not in alignment with our primary purpose of assessment. In a nutshell, formative assessment became more important than summative. Operationally, this meant that our focus shifted to accountability for every student demonstrating growth over the year. NCLB/state-mandated testing, although important, became subordinate to showing students' progress during the year as well as year to year.

Choosing Assessment

The final piece of our growth model was the formal adoption in 2004 of a common benchmarking assessment that schools could employ to verify growth. We selected the computer adaptive Measures of Academic Progress by the Northwest Evaluation Association. It met four critical criteria essential to our growth model:

* Correlation between our benchmarking assessment and the annual state tests.

* Individual measures of growth during the year and over several years.

* Specific feedback on the subelements of the state standards. Because a general reading score does not provide the teacher or learner with adequate information for improvement, our benchmark assessment breaks out reading, math and language usage into subscores with specific skills associated with each.

* Local and national norms.

Is the Measures of Academic Progress mandated for use at all schools in our 33,000-student K-12 district? Absolutely not. Because the school is the most effective and efficient unit of change and each school is responsible for implementing its own site assessment plan, MAP has been an optional assessment provided by the district since 2001. At first, only a few schools elected to use it, but it soon became clear it provided information that teachers and students could use several times a year to monitor growth, set goals and initiate teaching to assure student learning. Teachers continue to develop their skills translating MAP test results into individual student growth goals.

Culture Shift

This change in culture has become manifest in our core processes.

First, goal conferences for our principals have begun, in the last two years, to focus on growth over status. Annual goal reviews have been replaced with mid-year updates based on growth data. School improvement plans use growth as the measure of success. Teachers who closely monitor student progress and modify instruction to ensure student success now get growth reports, delivered in fall, winter and spring. These same teachers have, since 2004, been collaborating with their students in the development and shared management of formally written class learning plans that include specific learnings objectives, action plans and measurable targets.

Meanwhile, an unexpected change in the culture of our district has been the active role of the students managing much of their own learning. This year, for the first time, every elementary student will have a personal growth goal.

Ray Wilson is executive director of assessment and accountability in the Poway Unified School District, 13626 Twin Peaks Road, Poway, CA 92064. E-mail: [e-mail]

Raymond Yeagley is chief operations officer of the Northwest Evaluation Association, 5885 S.W. Meadows Road, Suite 200, Lake Oswego, OR 97035. E-mail: [e-mail]. He served as superintendent in Rochester, N.H., for 17 years.