Improving grant application peer review for the NIEHS.

The goal of the extramural research program of the NIEHS is to fund the highest-quality science to understand how environmental exposures contribute to the development and progression of human diseases. To identify the most meritorious research proposals, all grant applications submitted to the NIEHS (as well as to other NIH institutes and centers) must undergo individual peer review. These reviews are typically performed by expert committees referred to as "study sections." Although some of these reviews take place at the individual institutes, approximately 70% of all applications are reviewed at the NIH Center for Scientific Review (CSR). Every year, CSR study sections review tens of thousands of applications, both solicited and investigator-initiated, ranging from small research projects that can be carried out in a short period of time with limited resources to large program projects in areas as diverse as basic molecular biology, nursing research, clinical trials, and bioengineering.

Since peer review plays such a vital role in providing information for institutes to use in deciding which proposals to fund, it is crucial that the process ensure the best possible evaluation of the science. NIEHS grantees and applicants have raised concerns over a restructuring at the CSR that brought about changes including the elimination of two study sections dealing mainly with toxicology. In response, members of the NIEHS Division of Extramural Research and Training have embarked on a more formalized and vigorous monitoring and analysis of outcomes of reviews of NIEHS applications by the CSR.

A number of important observations can now be made as a result of this evaluation, which is ongoing and expanding. Our initial evaluation suggests that the CSR restructuring does not appear to have had much of an overall effect on the outcomes of NIEHS applications, although applications in the oncological sciences are not scoring as well since the restructuring, and we are trying to determine if this is true in other topic areas as well. And other problems exist as well. Perhaps the most significant is that NIEHS applications have had less favorable review outcomes than applications assigned to most of the other NIH institutes and centers, in terms of both a larger percentage being "streamlined" (or identified as having less potential for success and therefore afforded a less intensive review process) and a smaller percentage ranking among the highest-scoring 20% of all applications.

There are several factors that may contribute to these outcomes. For instance, the NIEHS has some special characteristics that present challenges to the review of our applications. Our broad scientific mission contributes to applications being dispersed over a wide range of study section topics. We are also relatively small institute. As a result, on average, fewer applications are reviewed for the NIEHS per study section than for most other institutes and centers. Our data indicate that across the NIH, applications from any given institute generally have less favorable outcomes when reviewed in study sections in which they represent less than 5% of the total applications being reviewed. For example, an NIEHS application may be reviewed in a study section dealing with many aspects of a particular disease and one that might therefore be dominated by applications assigned to another institute. In this instance, if the number of NIEHS applications is small, especially if it represents less than 5% of the total, it is likely they would not fare as well. This is true regardless of the institute or center the application is assigned to, although it can have a greater overall impact on smaller institutes. Although it is unclear why this trend exists, as more information emerges from our evaluation we are confident that its impact can be minimized.

Achieving the best review process for all applications is a key part of the mission of the CSR. To this end, senior staff of the CSR and the NIEHS have had a number of discussions with researchers from the extramural community about the evaluation process, and are developing a plan to increase the percentage of NIEHS applications in any given study section. In addition, we will continue to work together to monitor and improve the peer-review process for NIEHS applications across all NIH study sections. The work of the NIEHS is critical to reducing the burden of disease, improving human health, and extending survival. An improved peer-review process will greatly facilitate our ability to identify and fund the best research to achieve these aims.

David A. Schwartz, MD

Director, NIEHS and NTP 

E-mail: [e-mail]

J. Patrick Mastin, PhD

Chief, Cellular, Organs, and Systems Pathobiology Branch

NIEHS

E-mail: [e-mail]

Michael Martin, PhD

Director, Division of Physiology and Pathology

Center for Scientific Review

E-mail: [e-mail]