Looking in the wrong place: the flaw in the new federal charter school study.

As the proverbial story goes, a drunk, when asked where he had lost his keys, pointed off in the darkness, far from the lamppost under which he was searching. "But the light's better here," he explained.

So it is with the new federal study of charter schools. That study plans to look at 50 middle schools (and, perhaps, some older students at elementary schools), despite the fact that, according to our estimates based on data from the Center for Education Reform, only about 20% of the country's 3,400 charter schools are middle schools or have a middle school plus high school design; the rest serve students beginning in kindergarten or 9th grade. Even worse, two studies presented in this issue of Education Next appear to confirm what intuition suggests: charter schools are most effective for students who enter at an early age.

It's cheaper to look at middle-school kids, says the federal contractor, Mathematica Policy Research (MPR). By focusing on middle school, one can simply use the statewide data on a child's performance on tests for students in grades 3 through 8, as mandated by No Child Left Behind. For a solid study of elementary schools, MPR would have to administer its own tests.

In other words, the middle schools are where the cheap light is. Never mind that most kids are elsewhere.

Of course, no single study will ever tell us about all charter schools, because charters are inherently diverse. It no longer seems odd to encounter under the same moniker military schools and peace schools, schools for dancers and schools for those who have broken the law, boarding schools and schools that exist only in cyberspace. Indeed, that was the original intention: to free schools from suffocating regulation and give parents choices.

So it is ironic that the media treat charters as identical as they zero in on one overriding question: do students attending them learn more than students attending traditional public schools?

The issue is becoming all-consuming. Beginning with the New York Time's front-page splash about an American Federation of Teachers (AFT) study in August of 2004 ("Nation's Charter Schools Lagging Behind, U.S. Test Scores Reveal"), it seems that every study, no matter how problematic, has spawned a headline, simply because it talks about charters' effects on test scores.

Many studies, including the AFT one, are inherently suspect, inasmuch as they look only at student achievement levels at a single point in time, thereby telling us little or nothing about how much pupils are learning.

MPR, to its credit, plans to follow students over time. Unfortunately, by focusing mainly on the middle school, it is still headed down the wrong track.

Two high-quality studies presented in this issue of Education Next highlight the seriousness of the matter. In the research section (p. 52), Caroline Hoxby and Jonah Rockoff use a randomized experiment research design to evaluate three Chicago charter schools. Because these charters held a lottery to select their students, the research team could compare the achievement of applicants who were admitted with a similar group who remained in a traditional public school. The researchers found strong, positive effects of the charter school experience for younger children.

By contrast, Robert Bifulco and Helen Ladd (p. 60), use data from North Carolina to compare the test score gains of students in charter schools with their previous gains in traditional public schools. Students entering charter schools in grades 4-8 fell behind as a result of the move.

One explanation for differing results from the two studies is that they simply studied different schools. The more interesting possibility is that together they show charters to be most effective when they receive children at an early age.

It is thus unfortunate that the projected federal study of charters, unless quickly redesigned, will focus primarily on middle schools. Like the drunk, the feds are searching in the wrong place, simply because one light bulb is cheaper than another. The country could be left just as much in the dark at the end of the study as it is today.