How We're Using Value-Added Assessment.

A Tennessee school district charts the effects of instructional staff on student outcomes

Teachers have little or no control over the type of students who walk into their classroom at the beginning of the year. They get what they get. Yet the notion that teachers should be held accountable for their students' progress during that year, regardless of the beginning achievement level, is absolutely reasonable.

Holding teachers responsible for their students' academic growth is more reasonable than expecting a class as a whole to be above the national average or above the 65th national percentile or whatever the particular assessment goal might be.

For decades, student achievement test scores have been averaged and reported by classroom, school, district and state. Such scores may be reported several different ways, but one of the most common ways is by national percentiles. Everyone knows that a school's mean or median national percentile scores tend to reflect the nature of the school's student population. Thus, higher scores tend to be found in more affluent middle- to upper-class communities, while lower scores tend to show up in less affluent inner cities and some rural communities. Put another way, one can predict relative test scores by surveying the school demographics--what essentially is prediction by environment.

Focus on Progress

When Tennessee's Value-Added Assessment System began reporting achievement test score gains from one year to the next, we discovered it was not possible to predict a school's performance by its location. Within the 52,000-student Knox County Schools, we had high-scoring schools with high gains and high-scoring schools with low gains. We had low-scoring schools with high gains and low-scoring schools with low gains.

Moreover, in almost every case the results we were seeing made sense to those individuals who were most familiar with the schools in the district. We firmly believe that the gain scores are far more indicative of the effects of the instructional staff in schools than the raw scores or their derivatives, such as national percentiles. The latter are more reflective of who the students are than how the schools are performing.

Earlier TVAAS studies had shown no difference in gains between schools that were primarily Caucasian and those that were primarily African-American. Perhaps more significantly, the same studies had shown no difference in gains between schools with low percentages of students on free and reduced price lunches versus those with high percentages of students qualifying for the lunch program. An attendance requirement (75 days per semester) must be met before a student's test scores are included in TVAAS calculations.

Upon receiving our first value-added results, we obviously were pleased to see some of our inner-city schools producing academic gains equal to or greater than the national norm gains. If one looked at the raw score histories of these same schools, one would see that their national percentile ranks had been well below 50 for decades. At the same time, district-level administrators knew that these schools were well staffed with competent and dedicated principals and teachers. At last, through value-added assessment, we had evidence to show what excellent results were being obtained in these schools.

Plotting Gains

Some critics argue that even though low-scoring schools may show good gains, most of them remain below average. From an individual student's viewpoint, however, a significant gain has to be cause for celebration, regardless of that student's beginning and ending points. And collectively, because exactly half of us are below the median by definition, some schools will be hard pressed to pull their averages above the national halfway point.

Schools never will be equal because the communities never will be equal, but in a particular school district the gap between the extremes can certainly be narrowed--and maybe by a lot. In a low-scoring school, if the national norm gain is achieved every year, their national percentile score never can be any lower than it was the first year. Moreover, any year the school manages to exceed the national norm gain, its national percentile rank will improve.

From the standpoint of instructional improvement, the most useful information to come out of value-added assessment is a TVAAS report that shows gains by student achievement groups. This means a teacher or administrator can compare relative gains among high-achieving students, average-achieving students and low-achieving students in each of five academic subjects. The report shows gains by grade and by subject for the most recent year, as well as a composite of the three previous years. A districtwide report covers each elementary and middle school, applicable to grades 3 through 8.

When the findings from these reports are plotted on a graph, the visual effect is especially dramatic. Gains are reported by scale score differences from one year to the next. A "target gain" is provided for each grade and subject. It is referred to as national norm gain and is based on a normal increase in scale scores, which maintains one's rank with respect to a distribution of scores of all students. For example, a student at the 50th national percentile at the end of the 6th grade must make a certain gain in scale score points in order to maintain her 50th percentile rank by the end of the 7th grade.

Gain patterns across achievement groups differ widely from school to school. The most common pattern is one that shows gains decreasing as achievement level increases. In other words, low achievers gain the most, average achievers the next most and high achievers the least. The fact that this is a common pattern is not surprising. High achievers may be performing reasonably well and making good grades. Many of them just are not being challenged. This is an easy trap for a teacher who is busy at the other end dealing with low achievers whose needs are more obvious.

Various Scenarios

A second pattern is the reverse of the first one, that is the low achievers make the least gains and the high achievers make the most gains. This pattern is less common than the first, but it does occur in many schools where the instructional focus is clearly on the more able to gifted students.

A third pattern shows the average students making the most gains while both the less able and more able student gains drop off on each side of the middle. This pattern is most likely to occur when the instructor teaches to the average students and lacks the energy, creativity or the will to appropriately address the needs of all students.

The fourth pattern--the reverse of the third--is one where the average students gain less than their peers on each side of them. While this tends to be the rarest pattern, one does see it occasionally. One possible explanation for this pattern could be a teacher most concerned with either low- or high-functioning students. But being aware of that fact, the teacher tries to compensate by giving equal time to the other end and, in the process, neglects the group in the middle.

The fifth scenario is a flat pattern where all achievement groups make about equal gains. If all groups have high gains (equal to or better than the national norm gain), then this is the ideal pattern indicating that the teacher is addressing the needs of all students, regardless of their starting points. This always has been recognized as good teaching. It also requires the most effort on the part of the teacher. On the other hand, if the pattern is flat and neither low-, average- or high-achieving groups are making good gains, this may be the worst-case scenario.

In our school district, we look for these patterns by subject, by grade and ultimately by teacher. The TVAAS report supplies the gains by subject in each grade so obviously one must approach the data differently when there are 10 teachers per grade or three teachers per subject instead of one classroom teacher per grade.

Most teachers have taken keen interest in the value-added approach, though many insist they are not surprised by what this new information has to say. Based on what they know of themselves and their school, the data make sense. Most teachers have tried to act on the information to address those achievement levels that have low gains by changing strategies, working smarter or doing something differently. Of course, a few teachers say they have no clue as to why the reports look as they do.

Central-Office Help

For the past six years, we have had an annual value-added assessment meeting for principals and selected elementary and middle school teachers. We schedule four distinct full-day meetings, two at each level, to work with smaller groups. Elementary principals are asked to select two teacher representatives from their schools--people they consider to be informal leaders who can relate well to their colleagues. Middle schools send department chairs for language arts, social studies, science and mathematics. Except for attrition, the same people participate each year.

The morning session offers an overview, a status report or an update on the Tennessee Value-Added Assessment System, depending on what has occurred since the previous meeting. In the afternoons the principals join their teachers to go over their schools' TVAAS reports and plan for addressing problem areas. Curriculum specialists and assessment personnel are present to answer questions and work with the teachers and principals in their school breakout sessions.

We have profiled our more successful schools and teachers. By that we mean those individuals and schools whose student gains have been consistently high both over time and over the range of student achievement.

The most successful teachers in the value-added analysis seem to be those with high energy and the ability to help all students learn--the low achievers, the average achievers and the high achievers. They tend to be hard workers who are willing to share their expertise and resources with each other. They keep themselves and their students on task. They tend to be among those who arrive early and stay late. Interestingly, they frequently are described as having a good sense of humor.

There are no big surprises here. The profile sounds like a description of an excellent teacher. The TVAAS data simply support that.

Positive Direction

By and large, we believe Tennessee's use of value-added assessment is having a positive effect on instruction in our district. Over time we have seen gain patterns change in a positive direction in schools where we know principals and teachers were working on the problems relating to student learning.

Samuel Bralton Jr. is coordinator of research and evaluation for the Knox County Schools

How I Respond to the Critics

Resistance to value-added assessment--perhaps resistance to anything new in education--may be traced to one of three conditions: ignorance, inertia and deception.

Ignorance is a lack of information, knowledge or foresight. Benign ignorance may be corrected with effort, but ignorance in a more virulent form presents a bigger challenge.

Inertia suggests resistance to change, which most of us harbor to some degree. This reluctance frequently results from a genuine fear of the untried and the unknown, although it may be compounded by apathy, laziness or contrariness.

Deception means resistance or opposition growing out of jealousy, avarice, hidden agendas or ax-grinding.

During the development and early implementation years of Tennessee's Value-Added Assessment System, we faced various charges from the critics. Here are some of the more common criticisms of value-added assessment and my brief response.

Critic: You cannot neutralize socioeconomic differences among students.

Bratton: You can when you focus on gain or improvement and give less attention to raw scores and their derivatives. The TVAAS data clearly show that when resources, including teachers, are comparable, students make similar progress no matter where they fall on the achievement scale relative to each other.

When calculating individual teacher effects, these results are further enhanced by excluding test scores of students who failed to come to school regularly (anyone attending 75 days or less per semester).

Critic: Standardized, multiple-choice tests are no good.

Bratton: First, that is an opinion that is not shared by everyone. More importantly (and with some irony), the TVAAS statistical package resolves many issues critics have raised concerning the use of standardized test data.

Critic: It's not fair to use student achievement test scores to evaluate teachers.

Bratton: First, student test data should not be the only factor in teacher evaluations. Other aspects to teaching must be examined as part of a comprehensive evaluation.

Second, to ensure fairness for teachers, several safeguards were put in place in Tennessee--some are grounded in law, others are technical and part of the statistical software used to study the test scores.

Third, a good deal of misinformation has been circulated about the relationship that needs to exist between what is taught (curriculum) and what is tested (assessment). With the TVAAS. software in place, the match must be reasonable; but it does not have to be perfect.

Critic: This must be very expensive.

Bratton: Accountability does not come free of charge but one reason standardized tests have flourished for decades without value-added assessment is they are less expensive than alternative assessments or so-called authentic assessments. The TVAAS statistical analysis that is appended to the achievement tests, which Tennessee funds (one test per student), is more than twice the per-student cost of the achievement tests and the value-added analysis combined.