Publication bias: looking for missing data.

Publication bias: Looking for missing data

Not every experiment produces a clear-cut result -- and inconclusive studies are less likely to be published than those with more definitive results. This was the problem confronting biostatistician Colin B. Begg of the Dana-Farber Cancer Institute in Boston. To assess the efficacy of transplanting bone marrow for the treatment of leukemia, he wanted to compare his own statistical analyses of clinical trials at Dana-Farber with other published results. But he couldn't be sure the material reported in the medical literature was the full story.

"Many people will do studies and then not publish them," Begg says. "And it's hard to figure out which studies aren't published. The issue of the representativeness of published studies becomes a critical one."

Begg's experience led him and Jesse A. Berlin, now at the New England Research Institute in Watertown, Mass., to investigate the problem of "publication bias" more thoroughly. They report their findings in the current issue of the JOURNAL OF THE ROYAL STATISTICAL SOCIETY A. A paper examining the implications of this problem for cancer research will appear in the Jan. 18 JOURNAL OF THE NATIONAL CANCER INSTITUTE.

"Publication bias, the phenomenon in which studies with positive results are more likely to be published than studies with negative [inconclusive] results, is a serious problem in the interpretation of scientific research," Begg and Berlin contend. "It occurs because the decision to publish is often influenced by the results of the study."

The problem is particularly pronounced when researchers search for trends by aggregating the results of numerous independent studies. Because positive results are more likely to be reported, the overall picture may appear rosier than justified by all available evidence. In other words, abstraction of summary data from published reports is potentially misleading, affecting issues ranging from the health effects of environmental pollutants to the efficacy of medical treatments, several researchers say.

To illustrate the problem, one recent survey of investigators revealed that a substantial proportion of studies involving clinical trials of a certain, unspecified new therapy remained unpublished. Moreover, 55 percent of the published trials demonstrated a trend favoring the new therapy, whereas only 14 percent of the unpublished trials showed similarly positive results. Begg and Berlin have shown that the problem of bias is particularly severe when the studies involve small samples.

The issue of publication bias isn't new. Researchers, particularly in the behavioral sciences, have long discussed its potential impact. "Publication bias in the medical literature has been widely suspected for many years, but until recently there was little clear evidence," says Douglas G. Altman of the Imperial Cancer Research Fund in London, England. "Now, however, there can be no doubt that there is publication bias, and that it is a serious problem."

"It is important that, in the future, statisticians pay more attention to a phenomenon that many are aware of, but to which few have given serious attention," Begg says. Serious efforts to reduce the problem may require changes in the way researchers disseminate their results and in the methods used by statisticians. In particular, the emphasis on tests of "statistical significance" for "proving" theories may be misplaced.

In the case of cancer research funded by the National Cancer Institute, the design and conduct of clinical studies is already well organized and regulated. Descriptions of such studies go into a central registry. Begg wants this system extended to ensure the availability of the results of all studies, rather than just those that researchers choose to submit for publication and that eventually appear in journals.

"It would be a small step to mandate that results, in summarized form, be reported back to the registry," Begg says. "Then anyone who has access to the registry will also have access to the results."

Editors of medical journals are also aware of the problem, and the issue likely will come up at a meeting of such editors this spring. "I think there's a growing awareness that the journals are not ideal vehicles for disseminating this kind of information," Begg says. "However, how they should change is anyone's guess."

Whereas most scientists agree that publication bias occurs, some believe the problem is exaggerated. They argue that well-informed researchers know the problem and automatically treat published reports with appropriate caution. But more naive readers of published studies may not necessarily understand that need.

Begg and Berlin themselves add a cautionary note in their own report. "Although we have tried to be comprehensive in our review, there is a distinct possibility that the articles we cite are themselves subject to publication bias, in that such articles are likely to emphasize the magnitude of the problem," the researchers say. "Be that as it may, we believe that the various empirical and theoretical studies described provide sufficiently compelling evidence of the existence of a serious problem."