Districts pilot value-added assessment: leaders in Ohio and Pennsylvania are making better sense of their school data.

Like a lot of superintendents, Pennsylvania's Steve Iovino believes in a simple axiom: the more student test data the better.

Warwick School District, a 4,100-student district located in southeastern Pennsylvania, tested its students even in years when state-mandated exams were not required.

But Iovino didn't know exactly what to make of all this test data his district was accumulating.

"We weren't doing much with longitudinal data," he says. In other words, the Warwick schools weren't tracking individual student achievement over time, but rather looking at how a particular grade scored on a test year after year--a snapshot assessment.

Iovino knew there were other ways of looking at his district's yearly test data. "A way that would predict how well a student would perform [in the future] and what kind of value was added to a student's achievement in a given year," he says.

He was right. Pennsylvania launched in 2002 the Pennsylvania Value-Added Assessment System with Warwick as one of the 32 pilot districts. The system is based on the value-added assessment model statistician William Sanders created in Tennessee in the 1980s.

As Iovino explains, value-added assessment carries two basic goals: to assess how much growth a student achieved from the beginning of the school year to the end and to predict a student's future growth. Value-added requires, though, a minimum of three consecutive years' worth of test data for an accurate assessment. When Sanders created the model few school districts in the country conducted annual testing.

But with the No Child Left Behind Act's yearly testing requirements, every school district in the country soon will have enough test data to use value-added assessment. Sanders and others hope they do.

Teacher Usage

"It took us a long time in education to realize we could do more with test scores than simply slice and average them," says June Rivers, who co-directs with Sanders, her husband, the Value-Added Assessment and Research Program at the SAS Institute in Cary, N.C. Rivers is working with officials in Pennsylvania to implement that state's value-added assessment system known as PVAAS.

In 2002 the state added a value-added component to its student assessment process and launched a pilot project with 32 districts that had been using yearly tests in grades three through eight. Last year, 30 more districts were added to the pilot and this year an additional 50 to 60 districts are participating, according to Kristin Lewald, a project manager for PVAAS, which is overseen by the state's department of education.

It's her job to work with superintendents such as Iovino to smooth the transition.

"What I tell them is that it's taking information they already have and using it in a different way. We use data they already have, without using another test," she says.

In Warwick, Iovino began the value-added assessment project by looking at his middle school students. "The plan was to take a look at total scores as well as student grouping," he says.

Iovino's teachers have used the value-added measures that allow a teacher to know in more detail where individual students stand academically and to regroup students before beginning a new curriculum topic. Iovino, in his explanation, cites a social studies unit on feudalism. "Let's say we have a group of 7th-grade students who aren't reading at grade level. That group receives a pre-teaching lesson before getting into the study of feudalism."

Warwick this year will begin value-added assessment at its four elementary schools. "I want to create a climate where this is something worth looking at and where we are developing future action for improvement," he says.

A Full Spectrum

An important aspect of value-added, according to both Lewald and Sanders, is that the information allows school officials to examine the progress of all students, not merely the students who are struggling. States have to go beyond the minimal NCLB requirements to ensure all students are making progress every year, Sanders says.

"What about kids from the other end of the spectrum? Kids not in jeopardy of falling behind?" Sanders asks rhetorically. "The next question is, 'Can we sustain that kid on [an upward] trajectory to get that kid in a position to go to a university and be successful?' We want sustained growth for all kids, not just the low-end students."

The Oak Hills School District in suburban Cincinnati is well aware of Sanders' views that challenge most state accountability tests that only show when kids are proficient.

"One of the things we're able to do with value-added is highlight areas where we're doing things really well," says Jay Kemen, Oak Hills' director of curriculum and instruction. The information is personalized and detailed at the student level, the grade level and the building level.

"Where we have very high performance based on growth, we can go in and say, 'What are you doing well?' It works in districts and schools that have developed a culture of learning," he says. "I think in part what this data and other sources give us is a broader picture."

In Oak Hills, Kemen says, the district might go into one building to look at the test scores and find that nine out of 10 students are doing well, but one isn't. "Is that student a lower-functioning student or is it a higher-functioning student? Are we not stretching them as we should?" he wonders.

Major Scaling

Like Pennsylvania, Ohio has been rolling out a value-added assessment system known as Project SOAR (Schools' On-line Achievement Reports) in pilot districts. The project is run by Battelle for Kids, a Columbus-based organization created in 2001 by the Ohio Business Roundtable. Jim Mahoney, a former superintendent, is Battelle's executive director. (See related story, page 16.)

Mahoney says the challenge at hand is to scale up from the current 78 participating districts to all 718 Ohio public school districts, a goal he expects to meet by August 2008.

At Westerly Elementary School, a school near Cleveland that houses only 3rd and 4th grades, Principal Sylvia Cooper says the annual state exam administered to her students was not especially valuable because each year it compared a different cohort of students.

"You're not comparing apples to apples," she says. "You don't know how well your program is affecting individual students."

The idea of value-added is appealing, she says, and her school has joined the pilot program. The previous assessment system, she says, looked at two different groups of students. Now, though, value-added will allow her to see how individual student perform over time. "We get a more specific picture of individual students progress and academic areas that need to be honed better," she says, adding that she and her staff have had little time to analyze the data enough to make curriculum adjustments.

Cooper is required to import her school's test data into a software program that will conduct the value-added measures. During the start-of-school crunch, the test data was put on the back burner.

The time crunch is one of the hurdles value-added advocates must clear to convince school leaders across the country on the benefits of the system. Another hurdle is convincing teachers.

Evaluating Teachers

Bill Sanders has been talking about a new way to assess school test data for more than 20 years. He first started looking at test scores in the mid-1980s as a researcher at the University of Tennessee at Knoxville. Then-Governor Lamar Alexander was looking for a way to evaluate teachers and Sanders thought he could find a solution.

Sanders, instead of trying to control for differences in a student's background, such as family income and parents' education levels, decided each student should act as his or her own control. In other words, the focus should be on gains rather than raw test scores so that each student's performance is measured not against that of similar students but against his or her own past performance.

The overall idea is that even if all students don't achieve at the same levels, schools and teachers should at least be adding value to each student's performance.

But Sanders' early work mostly fell on deaf ears. It wasn't until 1992 that state policymakers adopted his novel assessment notion. A key yet controversial aspect allows an examination of not just school-level performance but an individual teacher's performance, based on the gains made by his or her students.

Using Sanders' system it was easy for school officials in Tennessee to see which teachers were the biggest gainers and the state included the value-added measure in teacher evaluations. In Chattanooga, high-gaining teachers are rewarded with bonus pay and other districts throughout the state are considering following suit. But outside of Tennessee, the notion of using a student's test score gains to evaluate teachers is not exactly popular.

"Our teacher unions will never go for this," says David Plank, co-director of the Education Policy Center at Michigan State University. "Any move will be blocked."

Plank says Michigan, like every state, soon will have annual test data thanks to No Child Left Behind, making it possible to track classroom-level performance. "Technically it's feasible, politically it's not," he says.

Identifying Excellence

For states that have adopted a value-added assessment system, the way around the sticky politics of value-added has been to, thus far, pledge to steer clear of using it to evaluate teachers.

"PVAAS was not intended for teacher accountability," says Pennsylvania's Lewald.

Patricia Brenneman, Oak Hills superintendent, says neither is Project SOAR in Ohio. "We don't want our teachers being threatened by someone else's best practice," she says. If the data identifies a teacher who has increased student learning significantly, then "let's have that teacher do staff development with other teachers," says Brenneman.

Rather than fear value-added assessment, teachers in the Oak Hills district have embraced it, the superintendent says. "Data don't make change in the classroom, people make change."

Using the teachers who show the highest gains as best-practice tutors for other district teachers is exactly what officials in the Hamilton County, Tenn., schools, which include Chattanooga, have been doing for the past several years, says Dan Challener, president of Chattanooga's Public Education Foundation.

"Through precise research we've been able to identify about 75 teachers who consistently posted well above national and local average gains," he says. "And what we then created is a research network with those teachers."

The district also has created a set of classroom videos featuring the high-gaining teachers so that if other teachers want to study the classroom practices of those teachers, they can simply refer to the videotapes, Challener explains.

Sanders sees the teacher evaluation component inexorably tied to the value-added measure. "Once you make the investment of testing each kid, each year, then you have the opportunity to make these measurements at various levels," he says. "Now, I'm a strong advocate that districts should evolve to get [value-added] measurement at the classroom level," but he concedes he is aware of the reality and knows Ohio and Pennsylvania educators will not be using their value-added systems for teacher evaluation, at least not in the near future.

Last Hurdle?

So if states have removed the teacher evaluation component from value-added assessment, then what's holding education officials from fully embracing Sanders' model?

In Michigan, researcher and policy analyst David Plank is about to release a paper that examines potential pitfalls of value-added assessment.

For instance, he says, state tests used in value-added calculations typically produce an overall reading or language arts score that may be derived from several different skills, such as spelling and grammar. So if a 3rd-grade student scores a 30 on the spelling portion and a 90 on the grammar portion of that test, his or her overall score would be 60. If the next year that same student, scores a 90 on the spelling portion of the test and a 30 on the grammar portion his or her overall score would again be 60, showing no gain.

"I have been and continue to be a strong advocate for value added," says Plank. "Technically, though, it's even more difficult than any of us had realized in our early enthusiasms."

Plank believes one solution would be to increase the frequency of testing, an idea he realizes would prove costly and probably unpopular. Some school districts, though, already have taken precaution against potentially misleading value-added results by using a combination of tests.

Gerrita Postlewait, superintendent of the Horry County, S.C., School District, has been working with Sanders for several years. "He created a database using state- and national-normed test and Northwest Evaluation Association tests (see related story page 24) tests to help create a broad picture of value-added measures," she says.

The approach provides a depth of information as well, she says. Her 47-school district now can more accurately assess everything from student achievement to certain districtwide programs. "It's taken the guesswork out," she says.

Sanders is aware that his system is not perfect and concedes that if, like in Plank's example, the student's performance results on the spelling and grammar portions of a reading test are not separated out, no way exists to accurately measure student gains. "We can say that Mrs. Jones's students across the scale measured in the test made progress greater than the district average. Can we say more than that? Of course not because we don't have the data."

Still, Sanders and his SAS colleagues continue to refine the system, and Sanders believes the strength of value-added is what it reveals over time. As schools become increasingly data-rich, and collect year after year of test data, flaws in value-added will be greatly reduced, he says.

The key to value-added's future is having educators not only accept it as an assessment model but make the best use of the information it provides, he says. "You can do the best analysis in the world but if you don't have people trained and coached to use the information, not much is going to happen."

Getting to that point, though, requires a final leap of faith from the education community. "This is a paradigm shift in the way most educators think," Sanders says.

RELATED ARTICLE: The business of value added.

Standard and Poor's, a financial firm better known for its bond ratings worldwide, is also in the business of collecting and analyzing school data, from test score results to the cost of a school lunch program.

In 2001, Michigan and Pennsylvania hired S&P to organize and analyze school data for every school district. Each state paid $2 million for the service, according to Bob Durante, a director with S&P's School Evaluation Services.

Now, with funding from the Eli Broad Foundation and the U.S. Department of Education, S&P has embarked on a mission to assemble a soup-to-nuts collection of school data for all 50 states.

One piece of that project is nearly complete: Schoolresults.org, a website that organizes relevant statistical information for the No Child Left Behind Act, such as adequate yearly progress results, has data on 47 states as of early fall. The project's next step is to collect additional information, including school spending data, for all 50 states, says Durante.

The goal is for school officials to be able to compare their districts to state and county numbers as well as to other districts of similar size and demographics. Mike Flanagan, executive director of the Michigan Association of School Administrators, says he urged S&P to bring together comparable districts to share best practices for school staff.

Funding for the School Information Project has come from public coffers and private sources. The U.S. Department of Education gave a 2003 grant for $4.7 million to the Broad Foundation, and the Broad Foundation matched that funding, according to David Almacy, a spokesperson with the Department of Education. The estimated total cost for the project is $50 million, and the U.S. Department of Education is currently reviewing whether it will issue another grant to Broad for 2005, said Almacy.

Strong Booster 

Al Lonoconus, superintendent of the Southern Columbia School District in central Pennsylvania, was an early pioneer of the service S&P now is rolling out to the rest of the country. He began using S&P's online data analysis when he became schools chief three years ago.

At first, the big part was test score comparisons," he says. "We can go in and compare like school districts and like schools based on socioeconomic data. For similar schools that were doing as well or better than we were, we made calls to find out the types of things they were doing," he says. "We never before had that type of data."

Now, Lonoconus adds, his 1,400-student district is using S&P to examine economic data.

"We're looking at facilities and budgets and looking at districts within our county and districts in the state, and our ability to tax compared to other districts of similar size," he says. Lonoconus says his district is below state, county and comparable district averages in terms of its expenditures per student.

"The S&P analysis has pretty much confirmed what we already thought, but having an outside group come in and confirm it means it's no longer an assumption," says the superintendent. "That's very important. If we do have to ask for a tax increase, at least we can show we're not way out of line."

A Preferable Source

While Lonoconus has been active in his use of the S&P data, some education researchers question whether superintendents throughout the country will turn to S&P for data on their district.

With S&P in Michigan for the past three years, the schools I've dealt with almost never used it," says Bettie Landauer-Menchik, head of the data services unit for the K-12 outreach program at Michigan State University's Education Policy Center.

Landauer-Menchik says district administrators are more likely to turn to the state's school report cards--which contain the same data--than to the S&P site.

-- Brett Schaeffer

RELATED ARTICLE: Growth measures: don't call 'em 'value added'.

Gage Kingsbury, director of research at the Northwest Evaluation Association, takes great pains to explain the Portland, Ore.-based consulting firm does not conduct value-added assessment.

What NWEA does, says Kingsbury, is create its own computer-based tests, which are administered to students every nine weeks and which measure a student's academic growth.

So where value-added assessment is a process to measure yearly student progress, NWEA crafts tests called Measures of Academic Progress to track academic gains and losses quarter to quarter.

"We know at the end of every quarter how much growth has occurred," says Gerrita Postlewait, superintendent of Horry County Schools in South Carolina, which started using NWEA's tests systemwide last year following a pilot program the prior year. She also works with value-added pioneer William Sanders on analyzing her school district's results on state and national tests, giving her an abundance of data.

The NWEA reporting, though, provides her with information more quickly than other standardized tests.

Because the Measures of Academic Progress are computerized, teachers can see results the day after the test is given, says Kingsbury, who contends the average state test normally reports out 4 to 6 months after it's given.

"We can operate like a business and immediately address any problem areas," says Postlewait. "Our students then gain the benefit of a system that's responsive to their needs."

Several years ago, Postlewait says, her district tried to develop its own assessment-something akin to NWEA tests. But a lack of coherence and time, she says, thwarted the project. She turned to NWEA, at an annual cost of cost of $6 per student for her 30,000-student district, because she believes the state assessment exams, which are conducted each spring, offer no help for making immediate adjustments because the official results are not sent out by the South Carolina Department of Public Instruction until the end of September.

A Leveling Effect 

Formed in 1974 as a partnership between Portland-area school districts and the Seattle Public Schools, NWEA was incorporated as a nonprofit in 1977. It now works on a contractual basis with 1,300 districts in 40 states.

In addition to the rapid turnaround, NWEA's tests also offer district leaders more detailed information than many other standardized tests, says Linda Clark, superintendent of Joint School District 2 in Meridian, Idaho.

"We were looking for a testing system that was testing the curriculum we were teaching," says Clark, who spent 10 years as the district's director of student achievement before becoming superintendent this past July.

NWEA tests align with her district's curriculum, providing a level of detailed student assessment previously unavailable, she says.

"The ITBS (the Iowa Test of Basic Skills), which we had for many years, did not have the power to change and inform district practices," she says. "A system that can measure growth by quartile and look at disaggregated data is extremely powerful."

The 28,000-student Meridian district has been using NWEA's tests for nine years, starting with the organization's paper-and-pencil versions.

"Virtually all of our elementary schools are leveling," says Clark, meaning students are grouped based on their skills as gauged by the detailed NWEA tests rather than in grade levels. "A 9-year-old might not be in something called 4th grade. Instead, he may be taking middle school math or reading at a lower level," she says. By grouping students this way, Clark says, the district can maximize the abilities of instructional staff and provide students with an ongoing learning challenge.

The rest of Idaho's 126 public school districts, as well as the state's 14 charter districts and 15 private school districts have followed Clark's lead. Three years ago the Idaho Department of Public Instruction contracted with NWEA to develop a new statewide exam, the Idaho Standards Achievement Test.

Measurement Flaws

Ultimately, NWEA's Kingsbury doesn't oppose value-added assessment models, he simply sees flaws in them.

One problem with value-added assessment, he says, is that its outcomes are only as good as the data used. "The NAEP tests are an example of a test that is very broadly used in measuring student performance. But the NAEP tests wouldn't be good tests to look for an amount of value a school is adding to students [because] the test is not accurate at the student level," says Kingsbury.

Another issue, he says, is that fixating on gains can obscure the larger goal of getting students to an established standard. "A school can add a lot of value to a student's achievement without getting that student to a standard."

--Brett Schaeffer

Brett Schaeffer is a free-lance education writer based in Philadelphia. E-mail: [e-mail]