The Mismeasurement of Educational Quality.

School leaders have a duty to undo the harm being perpetrated on schools today

America's educators are being obliged to participate in a contest they can not win. The contest is called "the scoreboosting game" and its object is to have educators raise students' scores on standardized achievement tests.

The score-boosting game rests on the assumption that a set of high scores by students equals more successful instruction by educators. That assumption, however, is misguided because it clearly misapplies the information that can be gleaned from standardized tests.

Unfortunately, many educators' current efforts to succeed in this accountability-induced contest are causing serious educational harm to students. Today's evaluation of schoolwide performance, based on students' standardized test scores, must be ended. Fortunately, there are ways this brand of educational mismeasurement can be reduced.

Mischief Makers 

I am not writing this commentary to be provocative or to beef up my publication list. I am doing so because the country's top educational leadership has a responsibility to do something to mitigate the harm now being perpetuated among students. Superintendents and their colleagues in decision-making positions need to halt the significant educational mischief now going on in our schools because of policymakers' misguided approaches, inspired by ignorance, for holding schools accountable for academic performance.

In what follows, I will suggest why standardized achievement tests do not yield an accurate picture of a school staff's instructional effectiveness. I'll describe a few harmful educational practices that have been spurred by educators sometimes mindless pursuit of higher test scores. Finally, I'll set forth six specific actions that the nation's educational leaders could follow to diminish or expunge this increasingly widespread misuse of standardized achievement tests.

Simply put, bad things are happening to children in many schools because of an unsound accountability strategy rooted in the wrong type of tests. The strategy was initiated by educational policymakers who mistakenly believed that students' scores on a standardized achievement test tell us how well those students have been taught. Nor do I fault the architects of these well-intentioned but misguided testing programs. State legislators and state board of education officials were attempting to help children on the basis of their flawed knowledge about educational assessment.

Rather, I fault us. That is, I fault the education professionals (myself included) who sat on their hands while the score-boosting game was born and multiplied. Our apathy has allowed this mismeasurement of educational quality to flourish and led to a situation that has placed teachers and administrators in an untenable position. Even worse are the adverse educational consequences for our students.

Measurement Misuses

A standardized achievement test is any examination that is administered and scored in a standard, predetermined manner. The five commonly encountered standardized achievement tests are the California Achievement Tests, the Comprehensive Tests of Basic Skills, the Iowa Tests of Basic Skills, the Metropolitan Achievement Tests and the Stanford Achievement Tests. But in a number of states, we also see standardized achievement tests that have been built to mesh better with a state's curricular preferences.

Today's standardized achievement tests are modeled after the Army Alpha test, developed in 1918 to help identify suitable officer candidates for America's World War I forces. The Alpha was an intelligence test measuring an Army recruit's aptitude for success in an officer training program. The Alpha's comparative assessment strategy allowed recruits to be compared with a norm group (of previous Alpha test-takers) in order to isolate each recruit's relative aptitude.

Today's standardized achievement tests follow the Alpha assessment strategy. Therefore, today's standardized achievement tests must produce a sufficient degree of score spread so that fine-grained comparisons among students can be made.

I have no quarrel with standardized achievement tests if they are used properly. Both educators and parents can be helped by learning that a child scored at the 86th percentile in mathematics but only at the 30th percentile in language arts. That's a good use of such tests. But standardized achievement tests do not tell us how well students have been taught.

Here are three reasons why that's so.

First, because the makers of these tests must market their exams on the basis of a one-size-fits-all curriculum, what is taught in many localities does not match what is tested. Some studies suggest that more than 50 percent of the content on many standardized achievement tests is not taught, or not even supposed to be taught, in some school districts.

Second, because of the tests' relentless quest for score spread, test makers dare not include many items that fail to discriminate among students, Items that many students answer correctly tend to be excised from these oft-revised tests because they do not contribute to score spread. Yet items on which students score well often turn out to deal with content that, because of its importance, teachers stressed. Thus, items covering the most important things that teachers teach will tend to be jettisoned from standardized achievement tests.

Finally, if you spend any serious time reviewing the actual items on standardized achievement tests, you'll find there are at least three distinguishable categories of items. Some measure the kind of things taught in school. (However, as previously noted, in a given school setting there may be mismatches between what's tested and what's taught.)

A second category consists of those items linked to students' socioeconomic status. These items will be answered correctly more often by students from higher-SES families than by students from lower-SES families. Finally, some items will be answered correctly more frequently by students who were fortunate enough to inherit higher levels of academic aptitudes, that is, verbal, quantitative and spatial capacities.

The SES-linked and aptitude-linked items do a wonderful job in producing the kind of score spread so essential if standardized achievement tests are to fulfill their Alpha-like assessment mission. Yet, as a consequence of the inclusion of many such items (and there are many such items), standardized achievement tests measure to a large extent what students bring to school, not what they learn there.

As a result of these practices, students' scores on nationally standardized achievement tests do not accurately represent how well a school's staff has taught. Unfortunately, the same three problems also exist within many state-customized achievement tests, which, not surprisingly, are often built by the same Alpha-oriented companies that create and sell the five nationally standardized achievement tests.

If you're a superintendent in a district serving mostly children in high socioeconomic households, your test scores almost always will be high. If you're working with low-SES kids, however, your district is destined to get low scores. And that's not your fault, or the fault of your staff. It is the consequence of using the wrong test to evaluate instructional quality.

Harming Children

Given the requirement to use the wrong tests and the immense pressure to boost students' scores, many educators have begun to employ practices that clearly are not in children's best interests.

Important curricular content is often eliminated because it is not addressed by whatever high-stakes test is used locally. Children are subjected to relentless drilling on the content apt to be measured by the high-stakes test. Children are given interminably long practice sessions with items mirroring those on the test or, worse, with actual items surreptitiously copied from the high-stakes test.

The message sent to children in such settings is clear: "It's not learning that's important, it's higher test scores. And we are out to get them any way we can." Every week--or so it seems--we read horror stories about teachers or administrators who have been caught cheating on high-stakes tests. They cheat (and I'm not excusing it) because they simply can't play the score-boosting game and win.

We cannot allow this absurd score-boosting game to continue. It is a contest in which the chief indicator of success is flat-out wrong!

A Menu of Actions

What I'm suggesting, of course, is that if you are a school leader who allows the score-boosting game to continue as is, you really cannot succeed. If you serve children of low socioeconomic standing, you'll either be on the receiving end of poor student test scores or you will be obliged to endorse (or overlook) instructional practices that harm children. If you serve high-SES children, you run the risk of allowing your district's undeserved success to mask instructional practices that need improving.

Moreover, if the parents in a school district of higher socioeconomic status ever discover that the true reason your district's test scores are so high is largely due to your advantaged set of students, you may be in trouble.

Accordingly, I urge you to consider the following six options, described here briefly.

* Option 1: Provide an intense and comprehensive assessment-literacy program for teachers and administrators.

If educators do not understand why certain high-stakes tests yield invalid estimates of instructional quality, they cannot inform parents or policymakers of these problems. Moreover, assessment-illiterate educators cannot describe more appropriate evidence by which to evaluate instructional quality. Today, a staff of educators without fluency in assessment represents a clear liability for any educational leader and school system.

* Option 2: Offer carefully structured briefing sessions to educational policymakers regarding appropriate and inappropriate ways to evaluate schooling.

Concise explanatory sessions for local and state board members and legislators can be planned to explain why some kinds of tests yield faulty pictures of instructional quality. The use of actual items from currently used tests is especially helpful in getting policymakers to understand why certain high-stakes tests provide invalid estimates of educational quality.

* Option 3: implement a meaningful assessment literacy program for parents.

As soon as an assessment literacy program has been concluded for the staff, provide an outreach program tailored to the needs of parents. Parents will be supporters of what's good for children if they truly understand the key measurement concepts involved.

* Option 4: Encourage the creation of autonomous parent action groups.

Unfortunately, if educators protest the misuse of high-stakes testing programs, they will typically be regarded as self-serving, defensive and unbelievable. However, if nonpartisan parent groups protest poorly conceived high-stakes testing programs, the views of those parents will be given serious consideration by policymakers. I have great confidence in autonomous parent groups if the members of those groups are literate in assessment issues.

* Option 5: Review, under carefully controlled conditions, the actual items in the high-stakes tests being used.

Enormous insights can be gained by having educators and noneducators carefully analyze, one item at a time, the actual items in national or state standardized achievement tests. The protocols for such item reviews must be carefully designed, of course, but the results of rigorous reviews can be remarkably illuminating.

* Option 6: Devise and implement more valid, credible evaluative schemes.

There is nothing wrong with an accountability system to evaluate schooling if appropriate evaluative evidence is incorporated in the assessment. To reject evaluative programs based on the wrong data without replacing those evaluations with defensible data would clearly be both professionally and politically improper. (A related article on page 46 describes one kind of credible evidence that can be collected to reflect instructional quality.)

One of the nice things about a menu of options is that one need not choose only one item. Personally, when I find myself in a great cafeteria, my tray is soon laden with far more than one choice. (I find the use of two trays to be far less constraining.) If you really want to do something serious to stop the mismeasurement of educational quality, I encourage you to pursue more than one option. Indeed, why not go for all six?

Time to Act

To make up for our heretofore apathetic acceptance of the wrong high-stakes tests to measure educational quality, I believe the nation's educational leadership should commence without delay to foment a meaningful revolution.

The erroneous and educationally harmful appraisal of instructional quality via standardized achievement tests must be stopped. And you can help stop it.

James Papham is an emeritus professor in the UCLA Graduate School of Education and Information Studies. He can be reached at 1706 Keoniloa Place, Koloa, Hawaii 96756.

E-mail: [e-mail]

Resources

James Popham suggests the following materials for those who would like to pursue the recommendations in his article.

* Classroom Assessment: What Teachers Need to Know (for use with teachers); Modern Educational Measurement: Practical Guidelines for Educational Leaders (for use by administrators); and Testing! Testing!: What Every Parent Should Know About School Tests (for use with parents), all written by Popham and published by Allyn & Bacon, 800-278-3525 and www.ablongman.com.

* "Target: Assessment Literacy," a 19-video, 3-book staff development kit, available from IOX Assessment Associates, 310-822-3275 and [e-mail]