Photo finish: certification doesn't guarantee a winner.

The July 2006 deadline came and went for states to comply with the No Child Left Behind (NCLB) mandate to have a "highly qualified" teacher in every classroom. To meet the standard, teachers must have a bachelor's degree, be state-certified, and prove they know the subjects they teach, either by satisfying minimum course-taking requirements or passing a test in the subject they teach. But will compliance ensure that students learn more? Does state certification make teachers better at fostering student learning? It is now possible to provide some answers to these questions by exploring the relative effectiveness of recently hired New York City public school teachers who entered the profession through alternate routes. Using a large data set provided by the New York City Department of Education (NYC DOE), we analyzed student test scores as well as information about the students, their teachers, classrooms, and schools. With this rich array of data, we compared the effectiveness of recently hired alternatively certified (AC) and uncertified teachers to that of their traditionally certified counterparts in improving student learning in math and reading during grades 4 through 8.

The results of our study of New York City public school teachers confirm a simple truth: some teachers are considerably better than others at helping students learn. For example, elementary-school students who have a teacher who performs in the top quartile of all elementary-school teachers learn 33 percent of a standard deviation more (substantially more) in math in a year than students who have a teacher who performs in the bottom quartile. Yet as we embrace this piece of conventional wisdom, we must discard another: the widespread sentiment that there are large differences in effectiveness between traditionally certified teachers and uncertified or alternatively certified teachers. The greatest potential for school districts to improve student achievement seems to rest not in regulating minimum qualifications for new teachers but in selectively retaining those teachers who are most effective during their first years of teaching.

New Routes to the Classroom

State laws were originally designed to regulate teacher quality by specifying minimum pre-service course-taking and exam performance. Reflecting these traditions, the majority of prospective public-school teachers have completed one or two years of full-time study in a university education program and passed exams such as the widely used PRAXIS tests to satisfy those requirements.

In the 1990s, that traditional system for regulating the flow into teaching began to break down. Faced with difficulties recruiting enough certified teachers, many school districts hired large numbers of uncertified teachers. When No Child Left Behind and state laws mandated a certified teacher in every classroom, districts turned to the growing number of teachers entering the profession through alternative certification (AC) programs. According to the National Center for Education Information (NCEI), 48 states and the District of Columbia currently have AC programs, and approximately 50,000 teachers became certified through alternative routes in 2004-05. NCEI estimates about one-third of all new teachers enter the field with alternative certification. Participants in the AC programs are usually required to possess a bachelor's degree and pass state licensing exams before entering the classroom and to enroll in a teacher-education program, taking coursework at night, during their first years in teaching.

We wanted to know whether staffing classrooms with uncertified and AC teachers shortchanges students. In other words, are students essentially paying the cost of training these teachers on the job, in the form of lost academic achievement?

Knowing the relative merits of traditionally certified, AC, and uncertified teachers is particularly important because AC and uncertified teachers are more likely to work in urban areas with low-income and low-achieving students, those most in need of a high-quality teacher who can foster their learning. This is one reason the New York City public schools are an ideal setting to explore how a new teacher's certification status might affect how much students learn. Besides being the largest and one of the most diverse school districts in the country, New York City is a major employer of uncertified and AC teachers. More than 50,000 new teachers were hired in New York schools between the 1999-2000 and 2004-05 school years. Uncertified and AC teachers accounted for, respectively, 34 percent and 20 percent of these new hires.

Teacher Certification in New York City

Recruiting a sufficient number of certified teachers has been a long-standing challenge for the New York City Department of Education. During the 1999-2000 school year, approximately 60 percent of all new teachers hired were uncertified. Recruiting difficulties have been more severe in schools with the lowest average achievement levels.

Since that time, the NYC DOE has taken a number of steps to decrease its use of uncertified personnel, one of which has been to expand its recruitment of alternatively certified teachers. Between the 1999-2000 and 2004-05 school years, uncertified hires fell from 60 percent to 7 percent of all hires, while the fraction who were alternatively certified rose from 2 percent to 36 percent. It is highly unlikely that this shift was a matter of previously uncertified teachers entering AC programs. The two populations--uncertified and AC teachers--differ in a number of ways: AC teachers are less likely to be black or Hispanic, tend to be several years younger when hired, and attended colleges with substantially higher median SAT scores (see Figure 1).

The major source of new AC teachers has been the New York City Teaching Fellows (NYCTF) program. Created in the summer of 2000, NYCTF is a partnership between the New York City Department of Education and The New Teacher Project (TNTP). TNTP is a national nonprofit organization, founded in 1997, that helps school districts recruit AC teachers. TNTP has also worked with school districts in Miami, New Orleans, Oakland, Philadelphia, Washington, D.C., and a number of other urban and rural communities.

The NYCTF program was created as a response to pressure from the state government to hire only certified teachers in the city's lowest-performing schools. After the NYC DOE failed to hire only certified teachers for these schools in 1999-2000, it was sued by state education commissioner Richard Mills. In 2000, additional motivation to produce and hire AC teachers was provided by a New York State law that required all teachers in the state to be certified in their subject of instruction by the 2003-04 school year. The number of teaching fellows hired grew from 350 in 2000-01 (less than 5 percent of new hires) to 2,500 in 2003-04 (more than 30 percent of new hires) and 2,000 in 2004-05 (more than 25 percent of new hires). Teaching fellows represent the vast majority of all AC teachers signed on in New York City since 1999. Between the 1999-2000 and 2004-05 school years, some 9,000 teaching fellows were hired.

New York recruits AC teachers through several other sources as well: the high-profile, nationwide Teach For America (TFA) program, the Peace Corps Fellows Program, and the Teaching Opportunity Program Scholars. (The latter two programs are relatively small.) New York City hired 1,544 new TFA teachers during the 1999-2000 to 2004-05 school years. Here, we limit our discussion to the findings on teaching fellows and the substantial number of teachers who come from the TFA program.

Measuring Teacher Effectiveness

It is a complicated task to determine how much difference a teacher makes in student achievement and whether or not that difference depends on how she entered the teaching profession. We need to compare the effectiveness of teachers in each certification group while separating out each of their students' baseline level of achievement (measured with test scores from the prior school year) and other characteristics of the student, classroom, school, and teacher that could affect student achievement.

We used data from the New York City Department of Education, which cover the 1998-99 to 2004-05 school years and grades 3 through 8, the grades in which students in New York City take standardized math and reading examinations. Because we need a prior year test score for each student in each grade in order to estimate the contribution made by the student's teacher, we can only study 4th- through 8th-grade teachers.

The student data include test scores, race and ethnicity, eligibility for the federal free and reduced-price lunch program, and status as an ESL or special-education student. We know about a student's prior attendance record and any suspensions from school. It is important to account for these student characteristics when evaluating an uncertified or AC teacher's contribution to student learning compared to that of a certified teacher. Uncertified teachers, teaching fellows, and TFA corps members all tend to teach in schools that, relative to those employing more certified teachers, have a higher percentage of minority students; more low-income, ESL, and special-education students; and students with lower achievement levels. We can place each student in a classroom and school so we know about the demographic and educational characteristics of students' schoolmates as well as the size of their class.

The NYC DOE data also include identification numbers for students' math and reading teachers, which were often the same for elementary-school students. This is a crucial piece of information, enabling us to match a student to his or her teacher. The NYC DOE payroll system was the source of information about each teacher's certification program and years of teaching experience. We are interested in evaluating teachers based on their certification at the time they are hired. Thus, uncertified teachers who later gain regular or alternative certification are still, in our study, considered uncertified. The study sample includes some 10,000 teachers in total, drawn from four groups: regular certified, regular uncertified, teaching fellows, and Teach For America.

Certification and Effectiveness

Simply put, a teacher's certification status matters little for student learning. We find no difference between teaching fellows and traditionally certified teachers or between uncertified and traditionally certified teachers in their impact on math achievement. Classrooms of students assigned to TFA teachers actually scored 2 percent of a standard deviation higher than students assigned to traditionally certified teachers. In reading, students assigned to teaching fellows did underperform students assigned to traditionally certified teachers by 1 percent of a standard deviation. These are the only instances in which we find that a teacher's initial certification status has statistically significant implications for student achievement. The picture of teacher effectiveness looked the same when we separately examined teachers in elementary schools, middle schools, and schools with above-and below-median test scores.

A Close Look at Teacher Experience

We also measured how teaching effectiveness improves with experience in New York City public schools. New York's teachers are no different from other teachers around the country. Teachers make long strides in their first three years, with very little experience-related improvement after that. The students of third-year teachers score 6 percent and 3 percent of a standard deviation higher in math and reading, respectively, than students of first-year teachers.

Since the first few years of experience are so important, we decided to take a closer look at how uncertified and AC teachers fare against traditionally certified teachers at different levels of teaching experience. We compared the effectiveness of teachers from among the certification groups in their first, second, and third years of teaching.

We found that teaching fellows, TFA corps members, and uncertified teachers may fare slightly worse as rookie teachers than certified teachers, but they quickly make up the lost ground (see Figure 2). For example, first-year teaching fellows underperform traditionally certified teachers in their first year by 1 percent of a standard deviation in math, but third-year teaching fellows outperform third-year traditionally certified teachers by 2 percent of a standard deviation. In reading, rookie teaching fellows underperform first-year teachers with traditional certification by 2 percent of a standard deviation. Yet by their third year of teaching, teaching fellows are eliciting student achievement as well as third-year traditionally certified teachers.

How do AC and uncertified teachers manage to catch up to traditionally certified teachers? This improvement need not reflect on-the-job learning. It could be that the propensity for ineffective teachers to leave could be higher among those uncertified and alternatively certified. This would improve the average performance of these groups relative to traditionally certified teachers, even though individual teachers did not improve at higher rates.

We can estimate the rate of improvement that is attributable to experience (as opposed to the departure of weak teachers) by following individual teachers and measuring the change in their performance from one year to the next. We can then see if these changes, say from year one to year three, tend to be greater or smaller for teachers from different certification groups.

There is some evidence that AC and uncertified teachers learn more from experience. Whether they teach math or reading, TFA teachers seem to learn at the same pace as traditionally certified teachers. The first two years of teaching bring substantially more improvement in both math and reading instruction for teaching fellows than for traditionally certified teachers. Given the same initial effectiveness as a traditionally certified teacher, our results indicate that, after two years on the job, a teaching fellow's students would score 3 percent of a standard deviation higher on average in math and reading. Uncertified math teachers' gains from experience also outpace those of traditionally certified teachers. Given the same initial effectiveness as a traditionally certified teacher, an uncertified third-year teacher's students would score 3 percent of a standard deviation higher, on average, in math.

Attrition

In debates over certification standards, alternative certification programs have been criticized for high turnover. Alternatively certified teachers may be just as good as traditionally certified teachers, the critics say, but they are more likely to leave teaching just when they are learning the ropes. Teach For America programs have been the focus of much of this criticism, since they ask their corps members for only a two-year commitment.

The critics may have a point. When faced with the choice of two teachers of equal initial effectiveness but differing expected turnover rates, a principal or district recruiter ought to choose the teacher with lower expected turnover. However, the strength of this preference depends on two things: the actual difference in turnover rates and the difference in effectiveness between an experienced and a novice teacher. Fortunately, both of these differences can be estimated with the NYC DOE data.

First, we set out to determine the average rate of attrition, by year of teaching experience, among teachers hired since the 1999-2000 school year. We used a statistical procedure that allows us to adjust for factors that might affect attrition, such as teacher age and the characteristics of the school in which they taught. We also adjusted for the state-mandated departure of teachers who remained uncertified at the end of the 2002-03 school year, which artificially raised the attrition of uncertified teachers in that year. (Starting in the fall of 2003, school districts in New York State were no longer permitted to employ uncertified teachers, although in schools with severe teacher shortages some continued to teach with two-year Modified Temporary Licenses.)

Perhaps surprisingly, after adjusting for these factors, teaching fellows have attrition rates similar to those of traditionally certified teachers. In fact, teaching fellows have slightly lower attrition rates in the first two years than traditionally certified teachers. After five years, approximately 50 percent of both groups still teach in the New York City public schools. Teachers who were initially uncertified are a little less likely to stick around than teaching fellows or traditionally certified teachers. Given the estimated rates of attrition, 45 percent would remain with the district in their fifth year.

Teach For America corps members do have much higher exit rates. By the fifth year, only about 18 percent of corps members would remain with the district. Much of the attrition for TFA corps members comes after two years of teaching. Presumably, this reflects the fact that TFA corps members sign up for a two-year teaching commitment.

We used these attrition rates to estimate the proportion of each group of teachers who would be in their first, second, and third year of teaching and so on, up to 30 years. Roughly 45 percent of TFA teachers would be in their first or second year of teaching during any given school year. In stark contrast, traditionally certified teaching and teaching fellows would be less than half as likely to be so green, with only 20 percent in their first or second year.

Despite the higher attrition among TFA corps members, the implications are not nearly as dire as some critics of the program suggest. The impact on student outcomes is negative but rather modest. When we compare teacher effectiveness in our simulated teacher pool, we estimated that TFA teachers would need to produce about 2 percent of a standard deviation in additional math and reading achievement in their students to offset the impact of the higher turnover rates. At least in math, the size of the Teach For America advantage during the first years of teaching is just large enough to offset the cost of higher turnover. (Although TFA corps members' effect on students' reading scores is not as great, the payoff to experience is also considerably lower in reading achievement.) In other words, both the critics of Teach For America, who point to high attrition rates, and the supporters of Teach For America, who maintain that the corps members are more effective than other novice teachers, are right. But on net, it is a wash.

Teachers Matter Even If Certification Does Not

The above discussion should not be taken to imply that teachers do not matter. Although it may not matter much whether a child is assigned to a certified or uncertified teacher, it certainly does matter to which teacher a student is assigned. Interestingly, the range of effectiveness within each certification group is wide and the distribution of effectiveness is roughly the same.

There are a number of potential sources of error in measurements of teacher effectiveness based on student achievement data. For instance, because many classrooms have 20 to 25 students, a few particularly talented or particularly disruptive students can radically change the classroom average. Moreover, there may be other factors, unrelated to class size, that lead to swings in classroom performance, such as a dog barking in the parking lot on the day of the test or special classroom "chemistry" reflecting a good match between a particular classroom of students and a particular teacher. We wanted to measure the part of a teacher's effectiveness that persists over time. We used a statistical method that separated our estimate of overall effectiveness into a fixed component, which is stable from year to year, and an idiosyncratic component that changes across classrooms and across years.

When we looked at the persistent component of teacher effectiveness, we found that the best teachers have a large positive impact on their students' academic performance relative to that of a less effective teacher. For example, the top quarter of elementary-school teachers improve student achievement in math by 33 percent of a standard deviation more than the bottom quarter of teachers do. Among middle-school teachers, the difference is slightly less but still important, at 20 percent of a standard deviation. To put this in perspective, the advantage of being the student of a teacher in the top quarter of effectiveness rather than the bottom quarter is roughly three times the advantage of being taught by an experienced teacher rather than by a novice, and more than ten times any advantage created by teacher certification!

Knowing When to Be Choosy

Traditionally, states and districts have regulated teacher quality by focusing on initial qualifications. In writing the No Child Left Behind Act, Congress followed that same logic, requiring states to live up to the minimum hiring standards they have established. But is a highly qualified teacher more likely to be a highly effective teacher? Our results suggest not. States and districts could learn a lot about teacher impacts on student achievement during the first few years on the job, as long as they assemble their student and teacher data to compare end-of-year performance among classrooms of students with similar baseline characteristics. That is an obvious first step. But states and districts also need to find ways to directly observe and rate in-class performance, using some combination of principals, peers, and external observers.

We believe states need to develop the infrastructure for assessing the performance of novice teachers during their first few years on the job. There are some models being developed, such as Connecticut's Beginning Educator Support and Training (BEST) program, Charlotte Danielson's performance evaluation rubrics, and the National Board for Professional Teaching Standards. These approaches are only beginning to be validated against student performance on standardized tests. That is a process that will take time, and it is probably too early to say that there is one best approach to assessing teacher effectiveness.

Admittedly, not all private employers terminate their less effective employees. Employees who do not perform up to expectations may remain in entry-level positions, left in the proverbial "mailroom." Schools are very different organizations from most private firms. Notably, there is no equivalent to the corporate mailroom. Less effective teachers, when they earn tenure, are assigned classrooms of students just like more effective teachers.

Nevertheless, there are some labor markets that operate in the way we propose. The market for faculty at top universities is one such example. Arguably, colleges and universities have a lot more information at the time they hire faculty than school districts do. They have a copy of several papers someone has written; they have spoken with a candidate's advisors, whom they often know personally; they invite the candidate to campus for a full day of interviews. Yet despite this intensive screening, the top departments remain at the top by selectively retaining only the most prolific researchers from among their junior faculty.

Ironically, current collective-bargaining agreements already give district leaders the flexibility to terminate ineffective teachers during their first two or three years on the job. For instance, in New York City, teachers' contracts may not be renewed if they receive an unsatisfactory rating from their principal during their first three years of teaching. Our estimates suggest that just one year yields a substantial amount of information on teachers' effectiveness. Three years would give a school district a lot of information.

Nevertheless, there are several potential impediments to districts' implementing this kind of policy. Individuals who believe they may be dismissed will be less attracted to teaching in a district with selective retention. Districts may have to make other changes, such as increasing salaries for teachers clearing the tenure hurdle, in order to recruit enough teachers to fill available positions. The "highly qualified teacher" requirements in the No Child Left Behind Act already make it hard for districts to hire sufficient numbers of novice teachers. Our proposal would require districts to hire even more. Those provisions will have to be rethought during the law's upcoming reauthorization.

By shifting the focus away from "qualifications," we are not proposing to open the floodgates into teaching. Nor are we intending to imply that teaching is based on innate talent rather than developed skill. Instead, we simply want to move the dam further downstream from the time of initial recruitment, and postpone assessments of teacher's effectiveness for a year or two until districts have much more useful information about which teachers are performing well and which are performing poorly. Only then will we have hope of living up to the aspirations embodied in the No Child Left Behind Act.

Thomas J. Kane is professor of education and economics at the Harvard Graduate School of Education. Jonah E. Rockoff is assistant professor of economics and finance at Columbia Business School. Douglas O. Staiger is professor of economics at Dartmouth College.

RELATED ARTICLE: Another Look

Further confirmation that neither traditional certification nor alternative routes to teaching guarantee teacher quality

One can have special confidence in a finding when independent research teams, working separately and using state-of-the-art methodologies, reach similar results. Education Next received a second study that closely resembles the analysis by Thomas Kane and his colleagues from a team of five scholars shortly after we put the Kane study into the peer review process. Not knowing the outcome of the Kane peer review, we did the same for the second study. Both were found to be of high quality.

The second study's authors, Donald Boyd, Pamela Grossman, Hamilton Lankford, Susanna Loeb, and James Wyckoff, also find few significant differences in effectiveness between traditionally certified New York City teachers and teachers entering through alternative pathways, such as Teach For America or the New York City Teaching Fellows program. More important, they find that the differences in teacher effectiveness within pathways far exceed the average differences between pathways. The similar methodology and findings in the two studies provide a rare example of replication in the social sciences.

Following the principle that the first paper to arrive receives precedence, we include here only the Kane study, but we encourage the interested reader to access the second study, which was published in Education Finance and Policy, at http://www.mitpressjournals.org/ toc/edfp/1/2.

Many Paths to the Classroom (Figure 1a)

New York City certified, alternatively certified, and uncertified
teachers differ in many respects.

Characteristics of New York City Teachers

                               Teach For  NYC Teaching
                    Certified  America    Fellow        Uncertified

Percentage
Black               11          9         19            31
Hispanic             9          9         11            18
Male                20         27         33            33
Graduate Education  36          4         14            15

Age
Median Age at Hire  27         23         27            29

(Figure 1b)

SAT Scores at Colleges Attended by New York City Teachers

                         Percentile of Median SAT Score
        Certified  Teach For America  NYC Teaching Fellow  Uncertified

Math    59         74                 68                   55
Verbal  63         79                 73                   59

Note: "Uncertified" teachers include those who are not participants in
Teach For America or the International Program (not shown).
SOURCE: Authors' calculations from 1999-2004 New York City Department of
Education data

Note: Table made from bar graph.

Experience Matters (Figure 2)

Whereas certification status has little impact on student test scores,
the impact of teaching experience is substantial, regardless of whether
the teacher is traditionally certified.

Note: "Uncertified" teachers include those who are not participants in
Teach For America or the International Program (not shown).
SOURCE: Authors' calculations from 1999-2004 New York City Department of
Education data