Whose Claims Are Valid?

Criteria for estimating the assertions of research support for reading programs, approaches and materials

It has become popular in recent years for advocates of particular approaches to beginning reading instruction to drape their claims in terms such as "research based," "empirically validated" and "rigorous scientific evaluation results." They tend to drop pie charts, bar graphs and other displays of supporting "data" into promotional materials and presentations.

The problem is everyone seems to have data that suggest the superiority of their preferred program, approach or materials. So who do you believe? And how can such claims be evaluated? This primer offers basic guidelines for assessing the adequacy of research-based assertions and determining your school district's need to change its current practices.

Converging Criteria

A series of studies conducted by different investigators in different sites that present similar findings is what we mean by converging evidence. The point to remember is that almost every approach worked somewhere for someone at some time. In the same vein, programs and products seem to work better in studies carried out by advocates rather than opponents or even neutral researchers.

Several characteristics of beginning reading instruction offer converging research evidence. For instance, developing adequate phonemic segmentation abilities in kindergarten and 1st grade is important (phonemic segmentation is the ability to "count" three sounds in cat). At the same time, little convergence can be found in the research on how best to organize instruction to assure this development in young learners.

Since the mid-1970s, several investigators have reported that from 80 to 85 percent of primary-grade children develop phonemic segmentation in the course of regular classroom instruction (offered typically by teachers who have never heard the term). Some studies also have demonstrated that most of the 15 to 20 percent of children experiencing difficulty with this skill can be taught phonemic segmentation and, once they do, many then begin to read. But some studies report on the use of tutorials, small-group interventions and whole-class interventions. Some programs provided 20 sessions, while others provided 100 or more. Some focused on games, some on particular drills, still others on inventive writing (or sound spelling) activities.

In short, we know it is important to foster phonemic segmentation in the primary grades, yet little research exists to recommend any specific instructional materials or approach for organizing the intervention as a tutorial, small group or whole-class effort.

In many respects, the same is true with phonics. A convergence of evidence points to the importance of providing children with useful decoding knowledge and strategies. But little convergence is found in the e research on how that is best accomplished. Some studies describe teachers effectively developing phonics strategies opportunistically in the context of authentic reading and writing activities. Others report effectively developing phonics skills in planned, sequential programs. Even more studies cite the sheer quantity of reading and writing activity as a primary influence not only on phonics learning but also on learning to read generally.

A convergence of evidence also exists on the need for children to have access to a rich and varied supply of books and stories that are of appropriate difficulty and engaging. In other words, attractive and well-stocked schools and classroom libraries are important factors in developing early reading.

While many claims are commonly made on research-based grounds, the converging evidence on the importance of teacher expertise often is ignored, even though the research routinely indicates that teacher quality is more critical to early literacy success than curriculum.

Peer Reviews

When looking at the research cited as evidence (and if it is not cited ask for the specific references), note how many citations are from peer-reviewed research journals. While an imperfect criterion (because even research journals vary in selectiveness), this is an important source of independent judgment about the quality of the study in hand. To be published in a peer-reviewed research journal usually means a report passed muster (anonymously) by three other researchers.

Only a handful of rigorous research journals routinely publish reading research. A short list would include Reading Research Quarterly, Journal of Literacy Research, American Educational Research Journal, Journal of Educational Psychology and Journal of Educational Research. While a well-designed reading research report might be published in any number of journals, most researchers will send their best quality work to one of these five journals first.

Sometimes, of easier access to educators are popular literacy products that could not meet the peer-review criterion. The promotional materials for these products are slick and slathered with reference to "research" support, including citations of unpublished dissertations, site reports and monographs. (There is a reason they remain unpublished.) In these cases, one must wonder why the reports have not been submitted to peer review or why they failed to gain the support of reviewers when submitted.

Comparability Factor

Even if the examination to this point suggests an adequacy of evidence, you still must consider the comparability guideline. Ask yourself these questions as you review a study:

* What sort of implementation is reported in the research cited?

* Are the demographics similar to students in your district?

* Is the same level of implementation support available at your site?

* Will the implementation effort needed be comparable given the current organizational an curricular framework of your school district?

* Will teachers and parents share the same level of enthusiasm for the innovation?

Compellingness Criteria

Making changes in early reading instruction requires that the evidence reviewed offers a generally compelling basis for modifying current practices. For instance, a single five-year qualitative study of three 1st-grade teachers hardly seems a compelling basis for establishing a federal, state or district educational policy. Likewise, a study of 375 children in 60 classrooms in a single school district would seem an unlikely candidate for altering policies concerning early reading instruction.

In neither case is the evidence compelling, though both reports may provide useful and important insights on beginning reading instruction.

Meeting the compellingness criteria in your own setting often will involve some self-study of current practices. In other words, it is difficult to imagine a study offering compelling evidence for change without good evidence on just what instruction is now offered locally. In many cases, good evidence for needed changes can be found but often the changes are needed in only some classrooms (pointing up that old problem of the importance of high-quality teachers).

Thirty years ago, in her classic book, Learning to Read: The Great Debate, Jeanne Chall noted, "I am not recommending a complete house cleaning-- an across-the-board abandonment of existing practices for new ones. Many teachers have developed methods of their own that are far superior to any that have been investigated and commercially published. In their quiet, unassuming way, these teachers are getting results that would be the envy of any of the authors of scientifically developed and tested methods." Recent studies of exemplary primary-grade teachers simply reinforce her view.

Consumer Awareness

Remember the old consumer protection axiom: If it sounds too good to be true, it probably is. When a product claims to be supported by research, just say, "Show me the studies!"

By doing so, you are attempting to establish a compellingness in the evidence available. But that takes homework and a skeptic's determination. It is easy for someone to assert that "hundreds of studies support..." Providing even a handful of relevant peer-reviewed, published studies is another matter, and it isn't too much to ask.

In the end, though, good schools simply employ larger numbers of high-quality teachers. So the final criterion is: Will this product, program or instructional material increase the number of high quality teachers in your schools?

Richard Allington is director of the National Research (enter an English Learning and Achievement, SUNY Albany, School of Education, Albany, N.Y. 12222. E-mail: [e-mail]