What the FDA can teach us about evaluating programs.

In recent months, the U.S. Food and Drug Administration has taken action against drugs falling into a category of pain relievers, known as Cox-2 inhibitors.

This action by the federal agency can help educational evaluators see their task more clearly. Recall that these drugs were condemned despite their proven efficacy. What brought them down was evidence of adverse side effects, especially risk of heart attack and stroke.

Some might argue that educational evaluators need concern themselves only with efficacy. After all, when have you heard of a new curriculum or educational policy causing a stroke or heart attack (though panic attacks are another matter)? It's not surprising, then, that the over-whelming majority of educational evaluations focus exclusively on effectiveness.

To illustrate, consider a recent posting from the U.S. Department of Education's What Works Clearinghouse website on an innovative, computer-based algebra program. The clearinghouse claims that a randomized, controlled trial demonstrated 9th graders enrolled in the program earned significantly higher mean scores on an Educational Testing Service algebra test and higher grades than students using a conventional textbook alternative. Is there anything else we need to know?

While it's true students exposed to new algebra programs don't risk heart attacks, we have to remember that educational programs always produce what John Dewey called "collateral learning," and evaluators cannot afford to ignore it. Indeed, Dewey recognized, rightly in my view, the importance of such learning, especially the development of "enduring attitudes" toward learning itself. Such attitudes, Dewey argued, are "fundamentally what count in the future," long after the algebra lessons are forgotten.

Side Effects

Unfortunately, the evidence needed to assess side effects is more difficult and more expensive to obtain than that needed to judge efficacy. This is true in both medical and educational settings.

Of course, there's an important difference between the contexts of medicine and education. In the former, evaluators are looking for adverse physical effects. In the latter, they are trying to discern beneficial as well as harmful effects.

A few education evaluators appear to have understood the importance of "collateral learning," though their writings are unlikely to cite John Dewey. Among them is James Heckman, a Nobel laureate economist from the University of Chicago, whose recent work on educational investment stresses the importance of young people developing what he labels "noncognitive skills," just another name for Dewey's enduring attitudes.

Heckman and others have shown that intensive early childhood programs are a worthwhile investment, not primarily because of students' cognitive gains but because of the attitudes they foster. The manifestation of these attitudes can be seen decades later in higher rates of high school completion and lower incarceration rates.

Are we then to measure algebra programs by their success in keeping students out of jail? The idea is ludicrous, of course. What benefit is most important to capture? I think Dewey can help us here. He contends the most important attitude that can be formed is the desire to continue learning. How could that be measured? One way would be to survey students completing an algebra course about their attitudes toward algebra, but the results of such a survey would be unlikely to inspire much confidence.

Judging Attitudes

A more promising approach would be to track student course selection subsequent to taking the 9th-grade algebra course. Students face course selection choices in high school, and it is here the attitudes they've formed toward learning and toward their own capacities as learners are translated into action. If the students taking the innovative 9th-grade algebra course are also more likely to take three more years of mathematics as might be expected, that is an important benefit of the innovative program, especially because mathematics can be the gateway to remunerative occupations.

On the other hand, if these students, compared to their peers in the control group, are turned off from further study in math, or if an identifiable subgroup dropped out of high school before graduating, that would signal the possibility that our innovative program could be the educational equivalent of Vioxx.

To evaluate the collateral learning that stems from broader policies such as voucher programs, students might be asked several times over the years after completing the program to fill out surveys identifying the books they've read, the TV shows they watch regularly and so on. The idea would be to consider subsequent student choices of activities as expressions of enduring attitudes acquired earlier. Of course, the results of such longitudinal surveys will be meaningful only if the trial includes randomized or rigorously matched treatment and control groups.

Evaluators are right to follow the FDA lead in designing rigorous tests of efficacy. Now they need to design equally rigorous tests of the collateral learning that is the inevitable accompaniment of any educational program or policy.

Francis Schrag is a professor of educational policy studies at the University of Wisconsin, 1000 Bascom Mall, Madison, WI 53706. E-mail:[e-mail]

Coming Next Month in

THE SCHOOL ADMINISTRATOR

* The brave new world of blogging by superintendents

* 12 essential elements of a school district website

* A superintendents guest commentary on delaying teacher tenure decisions