The truths about charter schools: new research from Chicago and North Carolina.

With charter schools now serving approximately one million students nationwide, policymakers have been awaiting rigorous evaluations of their effects on student learning. The following articles help fill the gap. Caroline Hoxby and Jonah Rockoff present evidence from the first randomized evaluation of charter schools, focusing on three charter schools in Chicago. Robert Bifulco and Helen Ladd examine charter schools in the Tar Heel state, concentrating on those students whose progress can be compared in both charter and traditional public schools.

The picture that emerges is, to say the least, complex. But we learn some significant things: Charter schools appear to do better with young students who matriculate directly into these schools than with students who enter during the middle-school years. Students who remain in charter schools do better than those who migrate back and forth between sectors.

Findings from the City of Big Shoulders

Younger Students Learn More in Charter Schools

The number of charter schools has grown very rapidly in the United States, from essentially none in 1990 to more than 3,400 today. Supporters believe that the flexibility granted these new public schools allows them to be more innovative and responsive to student needs than traditional public schools are. And the fact that no student attends a charter school unless his parents want to keep him there means that families can "vote with their feet." When a parent leaves a charter, so does the funding associated with his child. Thus a charter school cannot survive without satisfied parents. But charter schools do not just answer to parents; they must also persuade an authorizer to recharter them every few years, and they must participate in statewide testing and accountability. Will this concoction of flexibility, answering to parents, and accountability to the government raise school quality? Bluntly put, do students in charter schools learn more than their counterparts in traditional public schools? More than they would have learned had they stayed put?

A Lottery-Based Evaluation of Charter Schools

Getting a reliable answer to these questions is vital to the current policy debate, but researchers who try to answer them face considerable obstacles. First and foremost, most charter schools are new and small. They just don't yet have enough results for researchers to draw conclusions. Second, although all charter schools share the features mentioned above, they are otherwise a diverse lot. Many set up shop in urban areas, serve minority and low-income students, and rely on a strategy and curriculum associated with an education management organization. However, some charter schools serve very rural, mostly white students. Some are run as start-ups by parent or community groups that do not associate themselves with a particular strategy or curriculum. Even within the world of education management organizations, approaches to learning can differ substantially. In short, an assessment of some charter schools is useful for learning about similar charter schools, but we should not expect it to inform us about all charter schools.

Even when researchers can evaluate charter schools that are large enough to contribute useful results to a study, old enough to have a track record, and representative of a substantial share of all charter schools, they face a daunting analytical challenge: finding students in the regular public schools who are truly comparable to the charter school students. Students who apply to attend charter schools are a self-selected group, and simply comparing them with all other students in local public schools is likely to be misleading. We do not even know whether to expect self-selection to work for or against charter schools. On the one hand, parents who try out charter schools may be especially motivated. On the other hand, parents whose children are doing well may avoid being "guinea pigs" in relatively untried schools.

In our study, we overcome this challenge by exploiting a feature common to most charter schools: the lottery that schools use to admit students when they have more applicants than spaces. Such lotteries present an opportunity, rare in education, to conduct randomized experiments of the type more commonly used in medical research.

We use this lottery-based approach to evaluate three schools managed by the Chicago Charter School Foundation (CCSF). Our treatment group (those who, in medicine, would receive the pill) comprises charter school applicants who drew a lottery number that earned them a place at one of the charter schools (lotteried in). Our control group (those who would receive the placebo) comprises the applicants who were lotteried out. All told, the study focuses on 2,448 students who are divided between the lotteried-in and lotteried-out groups. It's important to realize that all of the students in the study applied to charter schools, so self-selection is the same for all of them. All that distinguishes the groups is their randomly drawn lottery numbers, so we can be confident that the groups are comparable not only in observable ways (like race and income), but also in less tangible ways, such as motivation to succeed. Currently, we can compare the progress of both groups for up to four years following their application. We are continuing the study and will report further results as they become available.

Our results to date, which indicate clear positive effects of attending a charter school on the math and reading test scores of students who enter charter schools in kindergarten through 5th grade, represent the most credible evidence yet available on how charter schools affect student achievement. They are also uniquely informative for policymaking. In the long run, as charter schools become more established, almost all of their students will have entered in the early grades. Policymakers should therefore assign greater weight to studies that focus on such students than they do to studies that, because they lack experimental data, must focus on atypical students who enter charter schools when they are older.

The Chicago Charter School Foundation

Chicago is home to almost all the charter school students (8,817 of 9,980) in Illinois. Charter schools in Illinois are free to establish their own missions and curricula, but they participate in the state accountability system and must abide by personnel restrictions similar to those of regular public schools. In Chicago, charter schools receive a per-student fee equal to only 75 percent of the average per-pupil operating spending in traditional public schools. For the 2003-04 school year, it was $5,279.

The Chicago Charter School Foundation is a charitable organization that has been operating since 1997. It oversees five primary schools, one high school, and one K-12 school. Together, its schools enroll more than half of Chicago's charter school students. Seats in the charter schools are in demand. In the spring of 2004, CCSF schools had 2.4 applicants for every student they could admit. Most CCSF schools are run by nonprofit education management organizations, but one is run by a for-profit organization.

[ILLUSTRATION OMITTED]

Our current report relies on CCSF's oldest schools, all of which have been in operation since the late 1990s and have produced enough results to be evaluated: Longwood (K-12), with 1,200 students in Washington Heights; Bucktown (K-8), with 600 students in Logan Square; and Prairie (K-8), with 350 students in Roseland. Longwood is run by Edison Schools, which is for-profit, while Bucktown and Prairie are operated by the nonprofit American Quality Schools. Although these education management organizations differ somewhat, their strategies are fairly typical of organizations geared toward urban, disadvantaged children. They feature a structured school day and curriculum, combined with a family-oriented approach designed to get parents involved.

The charter schools we study are all located in neighborhoods where the population is disproportionately minority and poor, but the schools are not alike. Longwood is in a very black neighborhood, and 99 percent of its students are black. Bucktown and Prairie are in neighborhoods that are mixed ethnically, but they draw students who are disproportionately likely to be Hispanic and in need of bilingual education (see Figures 1a through 1c).

The charter school students are about as likely to be eligible for special education and for the free or reduced-price lunch program as are students in the regular Chicago public schools. It's very important to use the regular public schools' classifications of students into lunch program, special education, and bilingual education. Otherwise, the classifications could reflect differences in how often the charter schools place students in these programs rather than their students' traits.

The effects of attending a charter school reported in any study can only safely be extrapolated to students and schools like those included in the study. The students in our study are urban, dominated by racial and ethnic minorities, and largely disadvantaged. All of the students in our study applied to a charter school, so our results pertain to students who want to attend charter schools. Of course, these are precisely the students in whom policymakers are interested. No one suggests that students who do not want to attend charter schools should be forced to enroll in them, so learning whether they would have done better or worse in such schools is irrelevant.

The CCSF Lotteries

The charter school lotteries we study are pretty standard. A separate lottery is held for each school and grade. For example, if Bucktown has 60 kindergarten places available for 120 applicants and five 2nd-grade places available for 25 applicants, there would be a kindergarten lottery and a 2nd-grade lottery. After a charter school's first year of operating a particular grade, it is normal for the most places to be available in kindergarten. In each lottery, applications are assigned a random number and ordered according to it. Using this ordering, the places available in each grade in each school are filled. (If a student is lotteried in, then his or her siblings are also automatically granted a place if they apply in a subsequent year and in a grade for which there is space available.)

In this article, we focus on students who participated in the lotteries held in spring 2000, 2001, and 2002. The Consortium on Chicago School Research generously agreed to match as many of these students as was possible to the Chicago Public Schools' student database using their names, dates of birth, and the school and grade they reported attending when they applied. These data provide us with information on achievement, as measured by the Iowa Tests of Basic Skills (ITBS), before students applied and, even more crucially, with post-application achievement data for students who remained in Chicago's regular public schools.

All students who enrolled in a charter school were matched to a Chicago Public Schools record, as were 73 percent of the charter school applicants who applied but did not enroll. We ultimately limit our analysis to the 2,448 of these students who applied from a Chicago public school or applied to kindergarten (and thus were not in any school when they applied). We do this because the correct comparison for a student who applies from a private school is a lotteried-out student who would not appear in the Chicago Public Schools database. Our results should therefore be interpreted as the effect of attending a CCSF charter school on students who would otherwise be attending a regular public school, not the effect on students who would otherwise be attending a private school.

Enrollment in Regular Public Schools

One oft-stated concern about charter schools is that they will draw away the highest-achieving students from the regular public schools around them. We can address this issue by comparing the prior test scores of charter school applicants in our data with the test scores of students in regular public schools in their neighborhoods (within three miles). This exercise assumes that students would attend local schools if charter schools did not exist. If this is basically correct, the comparisons give a sense of how a charter school's existence affects regular public schools around it.

Longwood's applicants, before applying to the charter school, had similar reading scores but lower math scores (5 percentile points lower) than other students in neighboring regular public schools. Bucktown's applicants had similar reading scores but lower math scores (7 percentile points lower) compared with students in neighboring regular public schools. Applicants to Prairie score about the same in math as students in the neighboring regular public schools, but their reading scores are higher (4 percentile points higher). As we mentioned above, however, Prairie draws from a neighborhood with a smaller radius than the one we allow, and its students' earlier scores are typical of that smaller neighborhood. In short, the charter schools draw students who are, on average, somewhat lower achieving than public school students in the neighborhoods where the schools are located (see Figure 1d). While the differences in incoming achievement are not dramatic, they certainly do not support the theory that charter schools drain regular public schools of their best, most-advantaged students. Remember that the above differences in earlier achievement do not affect our results because they are between applicants and nonapplicants. For our control group, we use lotteried-out applicants, not nonapplicants.

Attrition and Noncompliance

Some charter-school applicants do not comply with the treatment that the lottery "assigns" them. A small share of lotteried-in students do not actually enroll in a charter school. Instead, they enroll in a private school, a public school in another district, or--most often--continue in a regular Chicago public school. Also, some lotteried-out students do not continue to attend the Chicago public school from which they applied. They switch to a private school, a public school in another district, or even a different charter school.

Students who remain somewhere in the Chicago public school system (including charter schools) appear in our database, making them "observed noncompliers." Accounting for observed noncompliance in a randomized experiment is a fairly simple matter; we can adjust the estimated effect of attending a charter school to reflect the fact that some lotteried-in students did not attend.

The remaining noncompliers, however, are not observed because they disappear from the database when, for instance, they move to a suburban school district or switch to a private school. Unobserved noncompliers ("attriters") are a problem in a randomized study if the characteristics of students who attrite among the lotteried in are different from the characteristics of students who attrite among the lotteried out. Fortunately, this problem does not arise in our study: the patterns of attrition are very similar among lotteried-in and lotteried-out students. Most important, the lotteried-out students who attrite are neither higher nor lower scoring than the lotteried-in students who attrite. Thus our results should not be affected by the fact that we are not able to track every student through every student through every postlottery school year.

Large Lotteries Work Best

CCSF followed careful procedures to ensure that their lotteries were truly random. But do randomized lotteries automatically generate treatment and control groups that are comparable? For the group of applicants as a whole, they apparently did. Looking at earlier test scores and demographic characteristics, we find no statistically significant differences between the lotteried-in and lotteried-out groups. And the fact that the groups are so similar in their outward traits suggests they are also similar in unobservable traits like motivation.

Yet in addition to checking whether the lotteried-in and lotteried-out students are comparable as whole groups, we also need to check that subgroups of students, sorted by the grade to which they applied, are comparable. That is, we need to check the comparability of lotteried-in and lotteried-out students who entered as, say, 2nd graders. The reason we need to check these subgroups is that separate lotteries were run for each grade of entry. To see this point, let's consider a concrete example. Suppose that a school held a lottery among 100 applicants for 50 kindergarten places and held a lottery among 20 applicants for two 6th-grade places. For the same reason that flipping a coin 100 times would probably result in about 50 percent heads, randomization would probably ensure comparability between the 50 lotteried-in and 50 lotteried-out kindergarteners. But we can be much less confident about the 6th-grade lottery, even if we know it is random, because there are so few applicants and places. Randomization could easily produce two lotteried-in students who just happen to be quite different from the 18 lotteried-out students.

In fact, the example is close to the truth. We find that randomization does ensure comparable groups in grades at which quite a few students are admitted. However, randomization is insufficient to ensure comparable groups in grades of entry that are rarely used. Since most students start in charter schools in early grades (kindergarten and 1st grade alone account for about 50 percent of new students), there are comparable groups for students who enter in kindergarten through grade 5. The 6th, 7th, and 8th grades account for, respectively, only 8, 5, and 4 percent of all CCSF admittees, and higher grades account for even tinier percentages. Thus it should be no surprise that the lotteried-in and lotteried-out groups are not comparable for grades of entry like 6 through 12.

In short, we confidently estimate the effect of attending a charter school for students who enter kindergarten through grade 5. We cannot use the lottery-based method with any confidence to estimate the effect of attending a charter school on students who enter in atypical grades, like grades 6 through 12. This is a limitation, because we might be intellectually curious about how charter schools affect the rare student who enters as, say, a 12th grader. However, it is a limitation that is largely irrelevant to policymakers. Most charter school students will, by definition, enter in grades that are typical grades of entry.

Charter Schools and Student Achievement

Because our evaluation is based on data from a randomized assignment, our analytic strategy is relatively simple. In essence, we simply compare the achievement of lotteried-in and lotteried-out applicants through the spring of 2004, or up to four years following their initial application. The results we report are adjusted to reflect the fact that not all lotteried-in students enrolled in charter schools. They therefore represent the effect of actually attending a charter school, not simply of drawing a lottery number low enough to gain admission. To refine the comparison, we account for the slight differences in the observable traits, including earlier test scores, that emerged by chance between lotteried-in and lotteried-out applicants. This refinement makes little difference in practice because randomization ensured that the groups were comparable.

[ILLUSTRATION OMITTED]

We find that students in charter schools outperformed a comparable group of lotteried-out students who remained in regular Chicago public schools by 5 to 6 percentile points in math and about 5 percentile points in reading. (See Figure 2.) These are the key results of our analysis, and they translate into gains of 2.5 to 3 points for each year spent in the charter schools. The results are based on students who enter charter schools in kindergarten through grade 5, the grades of entry for which we can confidently estimate effects. To put the gains in perspective, it may help to know that 5 to 6 percentile points is just under half of the gap between the average disadvantaged, minority student in Chicago public schools and the average middle-income, nonminority student in a suburban district. If the students continued to make such gains for each year they spent in charter schools (a big "if"), then the gap between the charter school students and their suburban counterparts would close entirely after about five years of school. Right now, such projections are necessarily very speculative, but they help to give some sense of the magnitude of the charter-school effect.

The Virtues of Randomized Experiments

While the small number of students entering charter schools in midstream grades, like grades 6 through 12, precludes our estimating effects for them, the resulting focus is on the whole desirable. After a charter school is established, the vast majority of its students enter in the early elementary grades; for the most part, places in higher grades become available only when a student leaves.

In contrast, the rareness of late-grade entry poses serious problems for value-added analyses of charter schools, such as that by Robert Bifulco and Helen Ladd, who study North Carolina charter schools (see "Results from the Tar Heel State," p. 60). Such studies, which compare the annual gains made by students in charter schools with the gains made by the same student while attending a traditional public school, draw only on the experiences of students who were tested for at least two years in the regular public schools before attending a charter school. Because they rely on state tests that are administered for the first time in the 3rd grade, almost all the students included entered charter schools in 5th grade or later. These students are most likely unrepresentative; after all, they are engaging in behavior that is rare.

The fact that 5th-grade entrants are rare is not accidental; it results from parents' hesitancy to move children between schools. Logic would suggest that students who are moved midstream are more likely to be struggling socially or academically, and any such differences would cause results based on their experience to be misleading. It is dangerous to apply such results to more typical charter-school students, and it is wrong to portray them as representative in the absence of independent evidence that they are.

Our own data set can provide some indication of the magnitude of the problem. Fifth-grade entrants comprise only 13 percent of CCSF's total admittees and only about 6 percent of the admittees in our analysis, which excludes applicants from private schools and does not include charter schools that are in their first year of operation. If we limit the analysis to the 5th-grade applicants for whom we can compute value-added estimates, the number of student-year observations included immediately falls by about 85 percent. If we use standard value-added methods to estimate the effects of attending a charter school for these students, the results do not match well with those of our lottery-based analysis. In short, studies that use value-added methods to evaluate charter schools are at best misleading. The students included are too atypical for the results to be interpreted in a straightforward way.

Conclusion

We have analyzed established charter schools in Chicago that are overseen by the Chicago Charter School Foundation. Our results demonstrate that, among students who enter in a typical grade, attending a charter school improves reading and math scores by an amount that is both statistically and substantively significant. We believe that these results can safely be extrapolated to similar schools that serve similar students. In particular, the results are most useful for understanding the effects of charter schools run by education-management organizations on student populations that comprise largely low-income and racial/ethnic minorities. We cannot confidently extrapolate the results to very different charter schools, students from very different backgrounds, or students who enter in atypical grades. Our results should be helpful for many policymakers who are concerned about urban students like those we study. However, we do not claim that the results are helpful for all policymakers.

Research on charter schools, like the schools themselves, is fairly new. We are not aware of any other studies that use lotteries to isolate the effects of attending a charter school. Standard value-added analyses, which are often used to evaluate charter schools, rely entirely on an unusual group of students who switch from regular public schools to charter schools late in their elementary-school careers. Our analysis confirms that estimates of the effects of attending a charter school that rely on this peculiar group of students differ dramatically from estimates that are representative of students who apply to charter schools.

These differences probably stem from the tendency of parents to move children in the middle of elementary school only if they are already struggling. Thus we doubt that value-added analysis will ever produce results that have relevance beyond the peculiar set of students on which they depend. Evaluations of charter schools should rely on students who are typical of charter school applicants, not on students who are atypical. Randomization provides us with estimates that are inherently better than those based on value-added analysis.

Caroline M. Hoxby is professor of economics, Harvard University. Jonah E. Rockoff is assistant professor of economics and finance, Columbia Business School.

BY CAROLINE M. HOXBY AND JONAH E. ROCKOFF

A Snapshot of Charters in Chicago (Figures 1a-1d)

The Chicago Charter School Foundation (CCSF) schools in this study are
disproportionately minority and poor compared to Chicago Public Schools
(CPS) overall. When measured against nearby CPS schools, however, it is
clear that CCSF schools are a reflection of their neighborhoods.

(Figure 1a) Composition of Charter Schools and Chicago Public Schools

                            Percentage of Students
                      CCSF Schools        All CPS Schools

Black                 74                  51
Hispanic              22                  36
Free/Reduced Lunch    81                  78
Special Education     10                  12
Bilingual Education   16                  14

(Figure 1b) Black Students in Charter Schools and Nearby Public Schools

                                Percentage
                 CCSF School  Nearby Schools  All CPS Schools

Bucktown         33           39
Longwood         99           92
Prairie          54           92
All CPS Schools                               51

(Figure 1c) Hispanic Students in Charter Schools and Nearby Public
Schools

                                 Percentage
                 CCSF School  Nearby Schools  All CPS Schools

Bucktown         53           47
Longwood          1            2
Prairie          44            4
All CPS Schools                               36

(Figure 1d) Initial Test Scores of Charter Schools and Nearby Public
Schools

                      Percentage
            Charter School  Nearby Schools

Math
Bucktown    46              47
Longwood    37              42
Prairie     43              41

Reading
Bucktown    46              47
Longwood    39              38
Prairie     40              36

Note: Nearby public schools are schools within a three-mile radius of
the respective charter school, with the exception of Prairie which draws
from an Hispanic area with a smaller radius.
SOURCES: Consortium on Chicago School Research data and National Center
for Education Statistics data

Note: Table made from bar graph.

The Elementary Effect of Charters (Figure 2)

Between 2000 and 2004, the lotteried-in CCSF school students in grades
K-5 scored significantly higher on both the math and reading segments of
the Iowa Tests of Basic Skills (ITBS) than the students who were
lotteried out.

Effect of CCSF Schools on ITBS Percentile Score of students in Grades
K-5

            Percentile

Math        6.4*
Reading     5.6**

* Significant at the 5% level
** Significant at the 10% level
SOURCE: Authors' calculations from Consortium on Chicago School Research
data

Note: Table made from bar graph.

Results from the Tar Heel State

Older Students Did Better When in Regular Public Schools

In this paper, we use an extensive student-level data set to evaluate the impact of charter schools in North Carolina on the math and reading performance of students in grades 4 through 8. We address three main questions: Do students attending charter schools in these grades make larger or smaller gains in achievement than they would have made in traditional public schools? If so, what accounts for the quality differences between charter schools and traditional public schools? And, finally, do students who attend traditional public schools subject to competition from charter schools make larger achievement gains than they would have in the absence of charter schools?

Controlled Choice

Legislation authorizing charter schools in North Carolina was passed in 1996, and the first charter schools opened in the fall of 1997. By capping the number of charter schools statewide, limiting the annual growth in the number of schools per district, and providing for input from the local district before approval of charter applications, North Carolina has exercised more control over the establishment of charter schools than some states. Nevertheless, the North Carolina legislation is quite permissive in that it allows any individual or group to apply for a charter and does not require local district approval of a charter application. North Carolina charters operate as independent nonprofit corporations, act as their own employers, are automatically exempted from several regulations, receive operating funding at the same level as traditional public schools, and are subject to the same state testing requirements as traditional public schools. Thus North Carolina's program includes many of the elements recommended by charter school advocates. However, the state does not provide charter schools any additional funding for facilities.

The number of charter schools in North Carolina grew steadily after 1997. By 2001-02 there were 93 charter schools and more than 18,000 charter school students; charter schools made up 4 percent of all schools and enrolled 1.4 percent of the state's 1.3 million students. Charters can be revoked for a number of reasons, including poor student performance and financial mismanagement. Overall, about 12 percent of the charter schools that have been opened are now closed, but in no case was the decision to revoke a charter or to close due primarily to low student performance. Growth in the number of charter schools has slowed markedly since 2001-02, primarily because the state law caps the number of charter schools at one hundred. Even so, only seven states had more charter schools than North Carolina as of 2002 and of those, only five had a greater concentration of charter schools: Arizona, Florida, Wisconsin, Michigan, and California. The 93 charter schools operating in 2001-02 were spread across 46 of North Carolina's 100 counties.

Our analysis of these schools is based on data from the North Carolina Education Research Data Center, a collaborative effort involving the North Carolina Department of Public Instruction, Duke University, and the University of North Carolina. The database contains individual-level information on test scores and background characteristics for all students in grades 3 through 8 in the state's public schools, charter and traditional. We use these data to measure the gains in student achievement made by individual students each year between 4th and 8th grade.

The mixture of 3rd through 8th graders in North Carolina charter schools in 2001-02 differed from that in traditional public schools (see Figure 1). Compared with traditional public schools, charter schools in North Carolina enrolled a larger percentage of black students and lower percentages of Hispanic and white students. Roughly 40 percent of charter school students in grades 3-8 were black, compared with 31 percent in traditional public schools. At the same time, charter schools served a higher percentage of students whose parents are college educated and a lower percentage of students whose parents are high school dropouts.

Despite the higher average education level of their parents, charter school students exhibit lower levels of performance on end-of-grade tests in both reading and math. The gap between students in charter schools and students in traditional public schools is. 12 standard deviations in reading and .22 standard deviations in math. Our analysis attempts to determine if any of this difference in performance can be attributed to the charter schools themselves.

Measuring Charter School Effectiveness

The primary challenge in determining how effective charter schools are in raising student achievement arises from the fact that charter school students are self-selected. Because they have chosen to attend a charter school, they are likely to differ in unobserved ways from otherwise similar students who choose to remain in traditional public schools.

In principle, the best way to determine how effectively charter schools raise student achievement is to conduct a randomized experiment. Studies adopting this approach take the students interested in attending a charter school, use a lottery to assign them randomly either to the charter school or to a control group of students who would not have access to that school, and then compare the achievement of the students given access to the charter school with that of the students in the control group. This approach, which is used by Caroline Hoxby and Jonah Rockoff in their study of charter schools in Chicago (see p. 52), is useful for determining if a particular charter school or the education program it offered is effective. However, the results of such experimental studies apply only to the programs offered by and the type of students who apply to the specific oversubscribed charter schools evaluated.

When studying an entire system of charter schools, including some that are not oversubscribed, it is not possible to conduct a true experiment. Thus we use a method that in effect compares the test-score gains of individual students in charter schools with the test-score gains made by the same students when they were in traditional public schools. This approach provides powerful protection against bias from self-selection. However, this protection comes at a cost. In the end, our analysis of charter school effectiveness is based on the experiences of only those students for whom we observe annual gains (whether positive or negative) in test scores at least once in a charter school and at least once in a traditional public school. Because we have information on students only in grades 3 through 8, they must have attended a traditional public school at least once between 4th and 8th grade to be included in our analysis. If such students are not representative of all students who attend charter schools, our analysis may not provide an accurate measure of the average effect of attending a charter school in these grades. As we show below, however, this limitation does not affect our basic conclusions.

The Effect of Attending a Charter School in North Carolina

Our data set includes information on five cohorts of students: those in 3rd grade in 1996, 1997, 1998, 1999, and 2000, some 495,000 students altogether. Each cohort contains all the students in 3rd grade in North Carolina public schools, including charter schools, during the specified year. Each student has a unique identifier that is consistent over time, which allows us to follow students from 3rd grade through the last year that they remain in North Carolina public schools, the year they complete 8th grade, or the 2001-02 school year, whichever comes first. To compare test-score gains of students from different grades, we first standardize their raw scores separately by grade and year to have a mean of zero and standard deviation of one and measure the change in each student's standardized score from one year to the next. A gain score of zero indicates that a student has kept pace with the average student in the state, while a student with a gain score of 0.25 standard deviations will have improved his or her performance by enough to exceed roughly 10 percent of the state's students.

We first compare the average gains made by all students in charter schools with the gains made by students in traditional public schools, taking into account differences in gender, ethnicity, and the highest level of education completed by their parents. Because moving between schools is known to have a negative impact on student achievement, we also control for whether the student changed schools in the current year and whether that change was structural (for instance, the student moved to a junior high school from its feeder elementary school).

The results of this initial analysis reveal that students in charter schools, on average, make annual gains that are 0.06 standard deviations less in reading and 0.08 standard deviations less in math than students in traditional public schools with similar observable characteristics. Because the students in charter schools are self-selected, however, this difference could be attributable to one or more unmeasured characteristics of the students rather than to the fact that they are in a charter school.

As explained above, we address the problem of self-selection by comparing the gains made by students the years they were in charter schools with the gains made by the same students the years they were in traditional public schools. Of the 8,745 students in our data set who attended a charter school for at least one year, 5,746 also attended a traditional public school at least once between 4th and 8th grade. The results of our analysis of these "switchers," which continues to take into account the difficulties associated with moving between schools, again indicate that students make smaller gains while enrolled in charter schools, by nearly 0.10 standard deviations in reading and 0.16 standard deviations in math. This pattern provides strong evidence that the smaller gains made by these charter school students are indeed due to the quality of the schools they attend rather than to any unobserved differences between charter school students and students in traditional public schools.

The difference in the rate of achievement growth between students enrolled in charter schools and students in traditional public schools is substantial. It is substantially larger than differences between the growth rates for children of high-school dropouts and the children of parents with graduate degrees as well as those between blacks and whites, differences that are the focus of considerable concern. For example, our analysis indicates that the annual gains made by children of high-school dropouts lag behind those of children of parents with graduate degrees by 0.03 standard deviations in reading and 0.06 standard deviations in math. The negative effects of attending a charter school, on average, for the students in grades 4 through 8 included in our analysis, are roughly three times this large.

How Representative Are the NC Charter Students?

Analyses of the effects of attending a charter school may be misleading if the students included are not representative of all students who actually attend charter schools. It is therefore important to consider how the 5,746 "switchers" included in our final analysis, those who attended both a charter school and a traditional public school in North Carolina between grades 4 and 8, differ from the state's full population of 8,745 charter school students in these grades. Although our data confirm that the two groups are quite similar demographically, two differences are worth noting.

The first is that students who leave charter schools before 8th grade to return to public schools are overrepresented in our analysis. Thirty-seven percent of the students for whom we observe test-score gains at least once in both sectors attended a traditional public school after they were in a charter school, while the same is true of only 30 percent of all students in charter schools. That is, charter school "exiters" are overrepresented in our analysis by nearly 25 percent. If these exiters left charter schools because they were not doing well there academically, our estimate of the effect of attending a charter school may be more negative than the true effect for all charter school students.

Looking separately at the effect of attending a charter school for exiters reveals that the effect of attending a charter school is, in fact, considerably more negative than for students who were observed first in a traditional public school and remained in a charter school throughout the study period (see Figure 2). We therefore calculated weighted averages of the effects for students observed only entering charter schools and the effects for students observed exiting charters, with the weights equal to the proportion of each group in the total population of charter school students. The corrected estimates are -0.09 standard deviations in reading and -0.15 standard deviations in math. In short, the overrepresentation of exiters matters, but it accounts for only a small fraction of the estimated negative effect of charter schools.

A second key difference between switchers and the larger group of students attending charter schools in any year is that students who entered charter schools in the younger grades are underrepresented. Although our data do not allow us to address this issue directly while still accounting for the self-selection of students into charter schools, simple comparisons indicate that students who entered charter schools in the later grades made smaller gains in math (but not reading) than students who entered earlier. Given the underrepresentation of students who enter during early grades, this difference suggests that the average effects of attending a charter school across all grades, 4 through 8, may be less negative than indicated by our final analysis, at least for math. Even so, we see no reason to suspect that the true average effects are not negative.

[ILLUSTRATION OMITTED]

New Schools with New Students

One potential explanation for these findings is that many of North Carolina's charter schools were in their first years of operation and thus were grappling with the challenges of starting a new school. We therefore reran our analysis, allowing for the effect of attending a charter school to vary with the number of years the charter school had been open. The results confirm that the negative effects of attending a charter school are considerably greater for students in newly opened charter schools than for students in charter schools that are more established. After again correcting for the overrepresentation of exiters, the effects of attending a newly opened charter school were -0.17 standard deviations in reading and -0.28 standard deviations in math, or almost twice the average effect reported above for all charter schools in the state. However, it is important to note that the complications associated with being a new school cannot fully explain the poor average performance of charter schools: the negative effects of attending a charter school in North Carolina remain greater than .10 standard deviations in both subjects, even for schools that have been operating for five years (see Figure 3a and 3b).

We next considered whether the effects of attending a charter school also varied with the length of time the specific student had been enrolled. The results proved informative. First, the negative average effects of attending a charter school are driven largely, but not entirely, by students making unusually small gains during their first year in a charter. This is true regardless of how long the school has been operating. For some reason, transferring into a charter school seems to have a much more negative effect on achievement than transferring into a traditional public school. Second, students who choose to remain in charter schools do not continue to make smaller gains than students in traditional public schools after their initial year in a charter school. This is reassuring, in that it justifies the decision of many parents to keep their children in charter schools once they are there; the disruptive effects of moving between schools would make the return to a traditional public school counterproductive. However, it is also clear that the initial achievement hit these students take is not offset by gains in subsequent years, so that even this group, which is harmed least by attending a charter school, still has lower levels of achievement as a result of attending a charter school. Finally, we see that students who ultimately leave charter schools two or more years after they enter continue to make smaller gains relative to those they would have made in a traditional public school even after their first year in a charter school.

One reason North Carolina's charter schools might have difficulty providing effective education programs is high rates of student turnover. Rapidly changing student populations make student grouping and scheduling more challenging, intake of new students can distract administrators from other tasks, and assessing and helping new students can place extra demands on teachers' time. On average, the percentage of students in a school in grades 4 through 8 that made a nonstructural transfer in the previous year is far higher in charter schools than in traditional public schools. While only 14 percent of students in traditional public schools made nonstructural transfers, the same is true of more than one-quarter of students in fifth-year charter schools and of an even larger share of students in newer charter schools. Taking into account the higher rates of student turnover in charter schools reduces the magnitude of the estimated negative effect of charter schools by 29 percent in reading and by 30 percent in math. Thus high rates of student turnover may account for as much as one-third of the negative impact charter schools have on student performance. Even so, the coefficients on the charter school variable remain statistically significant, suggesting that other factors also play a role in the poor performance of charter schools.

Effects of Competition on Traditional Public Schools

Charter schools have the potential to have broader effects on student achievement if traditional public schools respond to the threat of losing students to charter schools by improving the quality of their own education programs. Although the number of charter schools in North Carolina and the nation has grown rapidly over the past decade, they still represent only a fraction of the total number of public schools and are likely to remain so for a number of years. Still, if North Carolina's traditional public schools improved in response to their presence, the apparently negative effects of charter schools on the achievement of students who attend them could be offset by more positive statewide effects.

To estimate the effects of charter schools on students in traditional public schools, we use information on each school's distance from the nearest charter school to develop indicators of whether or not the traditional school faces competition from charter schools. How close does a charter school have to be located to a traditional public school to provide meaningful competition? For 90 percent of the 6,576 transfers in our database, the distance between the charter school where the student enrolled and the traditional public school the student attended the previous year is less than ten miles. Our data also indicate that schools within 2.5 miles of a charter school lose a higher percentage of students to charter schools, and hence appear to face more competition, on average, than do schools 2.5 to 5 miles from the nearest charter, and that the threat of losing students to a charter school depends also on the number of charter schools within a given radius of the school. Our analysis of competitive effects therefore investigates whether the effect of charter schools on traditional public schools varies with the number of nearby charter schools as well as with the distance to the nearest charter.

First, however, we must take into account the fact that the location of charter schools is not randomly determined. If charter schools were primarily established in response to dissatisfaction with traditional public schools, they would tend to be located in areas with low-quality traditional public schools where students would tend to make below-average test-score gains. Alternatively, charter schools might be more likely to attract students in areas where parents tend to be more motivated and more informed. In those areas, gains in student test scores might be higher than in other areas, even in the absence of charter schools.

[ILLUSTRATION OMITTED]

To address the problem of nonrandom location, we essentially measure the effect of charter school competition on test-score gains by comparing the gains made by students in each traditional public school before the establishment of a nearby charter with the gains those same students made in that school after the arrival of nearby charter schools. Our results suggest that traditional public schools did not respond to competition from charter schools by becoming more effective, at least as measured by the learning gains made by individual students in the years immediately following establishment of charter schools. Not only are none of the estimated effects statistically different from zero, but many point in the opposite of the expected direction. We emphasize, however, that the intensity of competition in North Carolina is not very great. Even schools located close to several charter schools are unlikely to lose a substantial percentage of students. Thus our finding should not be interpreted as a general statement about the potential of charter school competition to influence traditional public schools.

[ILLUSTRATION OMITTED]

Conclusions

We set out in this research to provide a comprehensive evaluation of the impact of charter schools on the math and reading performance of North Carolina students in grades 4 through 8. Our results can only be described as discouraging for charter school supporters. Students in these grades make considerably smaller achievement gains in charter schools than they would have in traditional public schools, and the negative effects are not limited to schools in their first year of operation. Nor are the negative effects of attending a charter school substantially offset by positive effects of charter schools on traditional public schools, a finding that may reflect the fact that North Carolina charter schools provide only a limited amount of competition.

However, for students who choose to remain in charter schools, the negative effects of attending a charter school are largely limited to their first year of attending a charter school. It is also important to note that our findings apply only to students who either entered a charter school after grade 4 or exited a charter school before grade 8. Our data do not allow us to comment on the experience of students who entered charter schools before grade 4 and attended them through the end of middle school.

We also provide evidence that high student turnover rates may account for about 30 percent of the difference between test-score gains made in charter schools and what we would expect the same students to make in traditional public schools. This finding suggests that student turnover can be an unintended negative side effect of school choice. Because school-choice plans lower the costs to families of switching schools, it is plausible that such plans will increase the movement of students across schools and thereby increase student turnover rates, to the detriment of all students.

However, charter schools in North Carolina exhibit negative effects on student achievement even after controlling for student turnover rates. Further investigation to determine whether the remaining negative effects are due to peer influence, resource inadequacies, or poor management would be useful. Whatever the reason for the low performance, the public interest is not well served when charter schools are ineffective in raising student achievement.

Robert Bifulco is assistant professor of public policy, University of Connecticut. Helen F. Ladd is professor of public policy studies and economics, Duke University.

BY ROBERT BIFULCO AND HELEN F. LADO 

Who Attends Charters? (Figure 1)

Charter schools are more likely to enroll black students than are
traditional public schools. At the same time, the parents of charter
school students are more educated than their public school counterparts.

Enrollment in Charter and Traditional Public Schools 2001-02

                                                  Percentage
                                                          Traditional
                                         Charter Schools  Public Schools

Changed schools in the last year         47               13
Parents have a four-year college degree  45               28
Parents did not finish high school        4               11
Black                                    40               31

SOURCE: Authors' presentation of data from North Carolina Education
Research Data Center

Note: Table made from bar graph.

The North Carolina Charter Lag (Figure 2)

The test-score gains of students who attended North Carolina charter
schools lagged behind those of their counterparts in traditional public
schools. This difference was more pronounced for students who
transferred back from a charter to a traditional public school
("exiting") than it was for those students who stayed in charter
schools ("entering").

Effect of Charter Schools on Test-Score Gains

               Percent of Standard Deviations
                    Reading      Math

Entering only       -.06*        -.1*
Exiting             -.16*        -.27*

* Significant at the 0.05 level.
SOURCE: Authors' calculations based on data from the North Carolina
Education Research Data Center

Note: Table made from bar graph.

Those First-Year Blues (Figure 3a)

The apparent negative impact of attending a charter school is largely
attributable to the adverse first year. Students who remained in
charters for more than one year kept pace with students in traditional
public schools.

Effect of Charter Attendance on Test-Score Gains of Students Who Were
Observed Entering School by Age of School

                              Standard Deviation
                       Students' first     Students'
                       year of charter     subsequent charter
                       attendance          attendance
Age of Charter School  Reading  Math       Reading  Math

One year               -.13*    -.21*
Two years              -.07*    -.11*      -.02     -.01
Three years            -.06*    -.09       3        0
Four years             -.11*    -.16*      0        -.01

(Figure 3b) Effect of Charter Schools on Test-Score Gains of Students
Who Were Observed Exiting School by Age of School

                          Standard Deviation
                       Students' first   Students'
                       year of charter   subsequent charter
                       attendance        attendance
Age of Charter School  Reading   Math    Reading  Math

One year               -.24*     -.48*
Two years              -.18*     -.29*   -.04     -.19*
Three years            -.54*     -.29*   -.11     -.22*
Four years             -.39*     -.19    -.11     -.14

* Significant at the 0.05 level.
SOURCE: Authors' calculations based on data from the North Carolina
Education Research Data Center

Note: Table made from bar graph.