Checking NYC's facts.

We take the essence of Eric Hanushek's article ("Pseudo Science and a Sound Basic Education," check the facts, Fall 2005) to be that existing means for appraising "adequate" school financing levels are, at best, inexact, and when in the hands of irresponsible advocates, often result simply in "junk science."

Perhaps surprising to some readers, the principals of Management Analysis & Planning (MAP) concur with his thesis.

Indeed, in several articles published elsewhere, we have stated that in pursuing the doctrine of "adequacy," the judicial system has far outstripped the capacity of social science to provide precise and verifiable answers.

How much does it take to enable a student from economically disadvantaged circumstances to achieve state-specified learning standards in mathematics, science, or language arts? What should high-school class sizes be? What are the characteristics of an effective teacher, one who achieves superior student outcomes?

Frankly, anyone who claims to know with certainty the answers to these crucial questions is usually an unabashed advocate for more money for schools and has little respect for policy analysis or research. We believe that qualified professionals who have demonstrated success providing instruction to diverse student populations can apply their knowledge to the discussion about what programs and processes are more likely to produce desired student outcomes. For that MAP offers no apologies. However, when professional judgment panel results are misused, that is indeed a problem, and it is in such circumstances Hanushek's argument deserves attention.

MAP has never claimed, in any of its adequacy studies, to have determined the amount below which expenditures are inadequate. We always offer estimates for policymakers to consider given assumptions specified in the analytic and estimation exercises. We had no control over how New York referees would use the report and its findings. We share Professor Hanushek's concern that the referees apparently relied on a bottom-line estimate without considering the assumptions and methodology underlying that estimate. Had they done so they would have discovered several assumptions and statistical procedures that served to inflate the estimate, but about which reasonable experts could disagree.

That said, Professor Hanushek's thesis falls a bit shy, and here is why.

Professor Hanushek's criticisms of the voodoo science of professional judgment and successful schools models of adequacy analysis, though justified when they are used to determine "the" number, lead then to reliance upon another analytic technique, cost-function analysis. But that, when applied by economists to the circumstances in New York, called for even more money than any of the professional judgment analyses specified.

In many ways, AIR/MAP (American Institutes for Research) figures were the median of all estimates if one includes those produced by William Duncombe and John Yinger of Syracuse. The latter use a cost-function analysis and claim to incorporate "efficiency" into their estimation models. Thus criticisms of the professional judgment studies without addressing cost-function estimates leave Hanushek's arguments incomplete and open to even higher estimates by advocates relying upon econometric methods.

MAP acknowledges and, when provided with an opportunity, has always insisted that any infusion of additional school money should be accompanied by other systemic reforms, including stronger accountability for school failure with an emphasis on performance above all else, changes in antiquated systems (for example, single-salary schedules), and a stronger focus on evaluations to determine success and failure of programs and individuals. We realize that money by itself is not an answer to the nation's education challenges.

Professor Hanushek does not declare how much money is the "right" amount for New York. That is good. He should not. No one knows the "right" amount. However, one can know the right process. The right process is to provide the political system with the best advice one can and then let the deliberative dynamics of representative government take hold.

JAMES R. SMITH 

President

JAMES W. GUTHRIE

Chairman, Management Analysis and Planning, Inc.

Eric Hanushek replies:

The MAP principals underscore a fundamental problem that runs throughout the use of costing-out studies, in New York and elsewhere. School finance policy is politics, in both the good and bad sense of the term. While science may provide guidance on some aspects of the problem, the costing-out studies are not scientific in the traditional sense. Even so, partisans in the dispute distort and misuse those studies under the banner of science. The authors, even with the best of intentions, unleash a process that they cannot control--and at least in this case, seem dismayed by.

As Smith and Guthrie point out, other methods have now found their way into the costing-out world, including the cost-function method. (Still another is the "state of the art" or evidence-based method.) Although they did not play a prominent role in the CFE judgment, such models have been introduced in a number of other state court cases and legislative deliberations. As Smith and Guthrie indicate, these alternatives are no better than the professional judgment or successful schools approaches--and may be worse in important dimensions.

It is clear, though, why costing-out studies are almost always commissioned by partisans in the school finance debates. Courts and legislatures are not going to make any nuanced use of them, but instead tend to extract a number, paint it as science, and use it for their purposes. Think alchemy, not science.