Living (or dying) with your NCLB tests: schools' ability to meet expectations will depend on tests' instructional sensitivity.

In a couple of years, America's public schools will be inundated by a flood of federally mandated accountability tests. Before the end of the 2005-06 school year, the substantially expanded assessment requirements of No Child Left Behind Act will kick in and, based on students' scores on those tests, every public school administrator will be regarded as a winner or a loser.

Talk about high stakes!

In some places, the deluge has begun. Several states are continuing to use the tests already in place by expanding that testing to all grade levels required by NCLB.

And while two years may seem like a ways off for those in other states, sensible school leaders recognize they soon will need to deal with the many schools certain to be labeled as "failing" for not meeting the requirement for adequate yearly progress on test performance under NCLB. School administrators who prefer to simply await their fate will have adopted an unsound survival strategy.

The time to dig into this issue is now. Unfortunately, the options for action are neither numerous nor nifty. But first a fundamental proposition needs to be raised--namely, that whether school leaders and their schools will be fairly or unfairly evaluated on the basis of students' test scores depends completely on the particular tests that have been adopted by their state to determine adequate yearly progress. If a state's NCLB tests are appropriate, the state's educators have a chance to succeed. If a state's NCLB tests are not appropriate, the state's educators don't have any chance at all. An appropriate NCLB test is one that has the capacity to detect genuine improvements in instruction.

Unrealistic Expectations

I do not intend here to describe NCLB's extensive assessment provisions or the demands inherent in the law's adequate yearly progress requirements. Because NCLB was signed into law by President Bush back in January 2002, most school leaders already have seen more than their share of explanations about the tests called for by the law and the levels of student improvement that must be demonstrated each year.

Regarding the law's adequate yearly progress requirements, I concur with Robert Linn, a University of Colorado professor of education, who observed during his April 2003 presidential address to the American Educational Research Association that the law's AYP requirements are altogether "unrealistic."

So as matters currently stand, every school administrator soon will be saddled with a host of new accountability tests as well as a set of fanciful AYP expectations regarding students' performances on those tests. Such a daunting reality might understandably encourage school leaders to simply toss in their administrative towels. But it's definitely not towel-tossing time yet. There still are a few options for action available to school leaders.

Tests Predominate

If a state's NCLB tests are instructionally sensitive, there's a chance that (at least until the law's unrealistic AYP expectations are adjusted) sufficient improvements will be made on those tests so the bulk of that state's schools and districts will be able to meet their AYP targets. An instructionally sensitive test is one that can, as its name implies, detect instructional improvements when such improvements take place.

On the other hand, if a state's NCLB tests are instructionally insensitive, there's almost no possibility that the state's educators can attain their AYP targets. That's because an instructionally insensitive test simply isn't able to help determine instructional quality. Improved teaching, even if it were pres-ent, will not be identified. In states where instructionally insensitive NCLB tests are in use, schools and districts deemed by the federal government to be "in need of improvement" will become the rule, not the exception.

Because students' NCLB test scores will be the dominant determinant of success or failure under NCLB, it should be obvious that the nature of those tests is crucial.

When I suggest that NCLB's unrealistic AYP expectations are likely to be adjusted, I know all too well what unbridled folly it is to try to predict the actions of our federal policymakers. However, I believe that as soon as more than half of our public schools nationwide are labeled ineffective because of the AYP requirements, those requirements will be modified. In at least a dozen states that have reported test results this fall, more than half of the schools have been labeled as not making adequate yearly progress.

It is absurd to contend so many of the nation's schools are performing that badly. We will witness a growing clamor from citizens who just don't believe their local schools are rotten. And ever-responsive to voter distress, elected officials will undoubtedly come up with a more realistic set of expectations for adequate yearly progress.

However, any such AYP alterations are unlikely to take place in the next year or two. Thus for several years public school leaders will need to play the accountability cards they've been dealt. And whether those cards are likely to help or harm a state's students depends almost exclusively on the nature of a state's NCLB tests. Let's look, therefore, at what determines the quality of such NCLB tests.

Instructionally Sensitive Tests

An instructionally sensitive test is one that can determine the presence of instructional improvement (or worsened instruction) if it has occurred. Clearly, the underlying accountability strategy inherent in NCLB rests on an assumption that if students are taught more effectively, those students will perform better on accountability tests.

This assumption makes a good deal of sense. If, however, the accountability tests being used are largely impervious to the impact of improved instruction, then NCLB's overriding accountability strategy clearly collapses. Consider the essential attributes of an instructionally sensitive NCLB test.

* Clear descriptions of what's assessed.

First, an instructionally sensitive NCLB test must be accompanied by sufficiently clear descriptions of the content standards (curricular aims) being assessed so that a state's educators can really understand what skills and knowledge are going to be measured. After reading such descriptions, teachers must truly understand what's to be assessed at a level of detail suitable for their classroom-planning purposes. Those descriptions, however, must be relatively brief and definitely teacher-palatable. We can't expect busy teachers to wade through lengthy, jargon-laden descriptions of what's to be tested in order to ferret out an accountability test's true assessment targets.

* A focus on a modest number of significant curricular aims.

Second, NCLB tests that are instructionally sensitive ought to measure only a small number of content standards. We dare not overwhelm teachers with seemingly endless lists of content standards. But because only a few content standards are to be measured, those curricular aims must be unarguably significant. That is, those content standards must be potent enough to incorporate any subskills or bodies of knowledge students need to achieve such a content standard.

An example of a potent curricular target would be a student's ability to compose original narrative, explanatory, descriptive or persuasive essays. This composition skill is clearly a significant one, and its mastery surely requires students to learn a host of subskills (for in stance, the mechanics of writing) and knowledge (such as the content to be incorporated in the essay). The modest number of content standards we end up selecting for an instructionally sensitive accountability test must be unquestionably potent.

* Instructionally informative results.

Finally, an instructionally sensitive NCLB test, after it has been administered, must report students' performances so that both teachers and parents can identify which content standards the child has or hasn't mastered. If test results are not reported according to students' mastery of each assessed content standard, teachers will never be able to identify the ineffective parts of their instructional program that need to be improved or the parts that are working well.

An instructionally sensitive NCLB test then will satisfy all three of these requirements by providing clear descriptions of what's assessed, focusing on a few significant curricular aims and reporting its results in an instructionally informative manner. If an NCLB test falls down on one or more of these three requirements, it will be instructionally insensitive.

Gauging Sensitivity

School administrators, therefore, need to discern immediately whether their state's NCLB tests (either as currently planned or already determined) are instructionally sensitive. What needs to be done, simply, is to appraise your state's NCLB tests according to the three requirements for such tests. Remember, if your state's NCLB tests are remiss on any one of these requirements, then school administrators in your state may be more likely to be labeled as ineffective. Even if those labels are grossly inaccurate, the world won't know it.

To make an accurate appraisal regarding the instructional sensitivity of your state's NCLB tests, you'll typically need to secure some information from your state's department of education. I'm not referring to lengthy technical manuals loaded with sufficient psychometric machinations to ward off all but the ardent. What you need is straightforward information regarding (1) how many and how demanding are the content standards ostensibly measured by your state's NCLB tests; (2) descriptions of the tests' assessed content standards; and (3) the reporting procedures used to describe group and individual student results.

Based on your common-sense consideration of such information, you'll then be able to determine whether your state's NCLB tests are instructionally sensitive. If you feel uncomfortable about arriving at this crucial judgment alone, join forces with other concerned school leaders in your state. Moreover, a group-based determination of an NCLB test's quality is likely to have more credibility than will an individual's judgment. The educational importance of drawing a conclusion about the instructional sensitivity of your state's NCLB tests is almost impossible to overestimate.

Options at Hand

Based on the judgment made about the instructional sensitivity of your state's NCLB tests, I believe you can undertake some relevant actions though your choices, not surprisingly, will depend directly on whether your state's NCLB tests are instructionally sensitive.

If you work in a state whose NCLB tests are instructionally sensitive, count your blessings. You'll be operating in a setting where, at least for the next several years, you have a genuine opportunity to walk the AYP gangplank without falling off. Whereas school administrators in almost all other states soon will be in for a blasting, you might just snare a bit of bliss.

I suggest that you try to do an efficient job in promoting your students' mastery of NCLB-tested curricular aims. If you do, this will allow sufficient instructional time so teachers can address other important skills and knowledge that are untested. There's clearly more stuff that students should learn than what's assessed on NCLB tests. Efficient promotion of students' attainment of NCLB-tested curricular targets will leave time to teach students other worthwhile things.

If your state's NCLB tests are instructionally insensitive, however, your options are neither numerous nor attractive. Let's face it, you are being required to play a no-win educational game in which there will be some winners and many losers. Faced with that uncomfortable scenario, some educators will be tempted to engage in activities that, unfortunately, end up eroding the quality of a student's education. For example, teachers may force their students to take part in literally months of practice-item drilling. Such drudgery may boost students' test scores a few points, but those students usually end up abhorring school. The price is way too high for the payoff.

I can really generate only two legitimate action steps for school administrators who work in a state where NCLB tests are instructionally insensitive.

First, it still may be possible for you and your colleagues to educate state policymakers about the need to install instructionally sensitive NCLB tests. Some seemingly irreversible decisions are, in fact, yet reversible, although many states that are changing their tests have already begun the process of writing requests for proposals.

If you push to get more suitable NCLB tests adopted in your state, you don't want to be seen as an educator who's fleeing from accountability. You need to forthrightly endorse the need for a rigorous program of educational accountability. An instructionally sensitive NCLB test will provide citizens and policymakers with accurate evidence of a school's instructional insensitive NCLB tests, however, are almost certain to be specious.

As intimidating as this option may appear to be, the replacement of inappropriate NCLB tests with better ones is the only course of action for a school leader who's concerned about the educational well-being of students. Instructionally insensitive NCLB tests will both yield inaccurate accountability evidence and will diminish educational quality. Such tests should be replaced.

A second option for school leaders is to initiate a powerful public information campaign about the potential harm to children that will be triggered by the wrong kinds of tests. (See related article, page 12.) This action can be conducted simultaneously with the "change-the-tests" action. Indeed, a solid public-information campaign describing the downside of instructionally insensitive NCLB tests might well precede or accompany a policy-level effort to replace existing NCLB tests.

The rationale for this second option is transparent. At present, many educators and most citizens are assessment illiterates. Such individuals can't be enlisted as allies in a fight to secure better NCLB tests because those individuals won't understand there's something wrong with a state's current tests. However, if parents, policymakers and other concerned citizens truly comprehend how an unsoundly implemented federal law can lower educational quality, then these individuals can be productive participants in any crusade for more suitable testing.

Passivity's Perils

During recent months I have been following with interest the reactions of both citizens and educators to state-level and district-level reports revealing the percentages of schools that flopped on AYP. Those AYP failure percentages, as we have seen, bounce all over the ballpark. In some states, such as Florida, almost 90 percent of the schools failed their AYP targets. In other states, such as Minnesota, truly small numbers of schools took an AYP tumble.

Citizens' responses, predictably, depend directly on the proportion of schools that failed. In settings where failure rates are relatively mild, most citizens display only mild displeasure, yet still urge educators to do an even better job. In locales where school failure rates are high, citizens tend to lash out at perceived educator ineptitude.

But what I find astonishing is the reaction of educators to these AYP-based failure reports. Remember, the chief determinant of AYP success is student performance on state NCLB tests. But almost without exception, educators accept the appropriateness of the tests being used in their states. As long as instructionally insensitive NCLB tests rule the day, then educational leaders better start signing up for workshops in "Humble Pie Made Edible."

I wish I could proffer a series of wonderful choices to school leaders working in states where instructionally insensitive NCLB tests are in place or soon will be. But, at the moment, the barrel of options is about empty. For your own success and, more importantly, for the educational well-being of the students entrusted to you, I urge you to tackle this problem with all the energy you can muster. You can live with appropriate NCLB tests.

RELATED ARTICLE: Public Campaigning for Better NCLB tests.

In the descriptions of action options for school leaders in states whose NCLB tests are instructionally insensitive, I suggest implementing a public information campaign about NCLB tests.

This is a straightforward, six-step blueprint for carrying out such a campaign.

* Step 1. Identify the campaign's audience(s). Decide whether the public awareness effort is aimed at parents, policymakers and/or citizens in general. Clearly, the campaign should be suitably tailored for its intended audience.

* Step 2. Decide on the campaign's objective(s). Is this to be an effort to enhance a target audience's assessment literacy in general or will there be only a focus on NCLB-related assessment issues? Is the campaign intended only to promote increases in audience knowledge or will there also be an effort to alter that audience's attitudes? Objectives should be stated as clearly as possible, preferably along with an identification of the indicators to be used to measure progress.

* Step 3. Form campaign staff. Recruit the individuals who will be responsible for carrying out the planned public information campaign.

* Step 4. Compile appropriate campaign resources. Choose suitable print or audiovisual materials that can be used by campaign staff. Keep abreast of breaking news coverage and solicit appropriate resources. For example, South Carolina Educational Television has recently produced a series of three assessment-relevant videos, each of which is available in either a 56-minute format (for showing to groups) or as 16-minute cassettes (suitable for take-home viewing by parents). Information about this reasonably priced series, "The Trouble With Testing," can be accessed at www.scetvstore.org or www.ioxassessment.com.

* Step 5. Carry out the campaign. Implement the campaign in full recognition that enhancing people's knowledge and influencing attitudes take ample time.

* Step 6. Evaluate the campaign's success. To determine whether the public information campaign was successful, return to the campaign's objectives, then collect credible evidence indicating whether those objectives were attained. Members of the target audience might be asked to complete anonymous cognitive and/or affective inventories. Ideally, such inventories also would have been completed prior to the campaign's beginning so that pre/post comparisons are possible. Personal interviews also can yield useful insights regarding whether the campaign was worth the effort.

--Jim Popham

REALTED ARTICLE: AASA's lead in test reform.

In summer 2001, before passage of the No Child Left Behind Act, AASA joined forces with four other national education associations to convene a national panel whose mission was to describe how a high-stakes accountability test could be built that would also support instruction.

All 10 members of the Commission on Instructionally Supportive Assessment agreed to carry out their deliberations without recompense on the condition that the five convening associations would not alter the commission's recommendations. Thus in a real sense this was an independent deliberative body.

During a three-month period in mid-2001, the commission identified nine requirements that an accountability test must satisfy if it were to be regarded as instructionally supportive. If a state's accountability tests satisfied these nine requirements, the commission's members believed those tests would support both instruction and accountability.

In October 2001, subsequent to extensive deliberations, two reports were released by the commission. The first, aimed at policymakers, describes the nine requirements for instructionally supportive accountability tests and provides a brief rationale for each requirement. The second report provides an illustrative request for proposals that might be used by those states that choose to solicit NCLB test-development services from an external contractor.

The model RFP in that second report, if issued by state officials, would lead to the construction of instructionally supportive NCLB tests. Both reports are downloadable from the AASA website at www.aasa.org/ issues_and_insights/assessment/Building _Tests.pdf.

A subsequent report considers the compatibility of the commission's pre-NCLB recommendations with the accountability provisions of the law as it was finally enacted. It can be obtained at www.nea.org/accountability/ images/02eseatesting.pdf.

Members of the Commission on Instructionally Supportive Assessment were: Eva L. Baker, UCLA; David C. Berliner, Arizona State University; Carol Camp-Yeakey, University of Virginia; James W. Pellegrino, University of Illinois at Chicago; W. James Popham, UCLA; Rachel F. Quenemoen, University of Minnesota; Flora V. Rodriguez-Brown, University of Illinois at Chicago; Paul D. Sandifer, South Carolina Department of Education; Stephen Sireci, University of Massachusetts, Amherst; and Martha L. Thurlow, University of Minnesota.

The associations that joined AASA in this effort were the National Association of Elementary School Principals, National Association of Secondary School Principals, National Middle Schools Association and the National Education Association.

--Jim Popham

James Popham is an emeritus professor of education at UCLA. He can be reached at 1706 Keoniloa Place, Koloa, HI 96756. E-mail: wpopham@ ucla.edu. He chaired the Commission on Instructionally Supportive Assessment, on which AASA participated.