Revamping peer review: the National Science Foundation will allow more peering into its reviews.

Revamping Peer Review

In 1977, a panel of scientists met to peer-review a series of research proposals submitted to the National Science Foundation (NSF). During discussions of a proposal to investigate archaeological remains in Ethiopia's Middle Awash Valley, rumors surfaced that a co-worker of the grant applicants--a geologist already leading 2k field researchers at the Ethiopian site -- was working undercover for the Central Intelligence Agency. NSF program officers did not alert the grant applicants to the rumor. Nor did they mention it to Jon E. Kalb, the subject of the allegation.

NSF rejected the proposal but has stated repeatedly that the decision was unrelated to Kalb. The alleged CIA tie, however, led the Ethiopian government to expel the geologist in 1978. Kalb, now a research associate at the University of Texas in Austin, filed a suit against NSF in 1986, contending his career had been damaged by NSF's failure to alert him to the rumor, which he says was unfounded. To document that the rumor circulated among NSF's peer reviewers and to understand why NSF never directly informed him or his colleagues of the allegation, he spent three years in law libraries and $10,000 in expenses to file some 150 inquiry forms with NSF under the Freedom of Information Act (FOIA).

As part of a December 1987 court settlement, NSF reimbursed Kalb for those expenses and another $10,000 in legal fees. But Kalb says his investigation of the rumor would have been simpler and less costly if NSF had complied in full with the 1974 Privacy Act.

Last July, with help from the nonprofit Public Citizen Litigation Group in Washington, D.C., he petitioned NSF to revise its procedures for creating and reviewing grant proposal files and to inform applicants about file materials they're entitled to see. On March 12, NSF General Counsel Charles H. Herz formally responded to that petition with an 18-page letter to attorney Eric R. Glitzenstein of Public Citizen.

The document announces that NSF will revamp its recordkeeping for peer reviews and its retrieval procedures for all grant application files. Through Herz's letter attributes most of these new procedures and policy clarifications to suggestions made by NSF Director Erich Block and the NSF staff, most bear a striking similarly to recommendations offered in the petition by Kalb and Public Citizen.

The petition indeed "led to some reform," says Daryl Chubin of the congressional Office of Technology Assessment, who recently began a study of peer review and alternative ways to handle research proposals submitted to six federal agencies funding basic research. Traditionally, NSF tends not "to examine the way they do things publicly," he adds. In Chubin's view, the petition prodded NSF officials "to exposed parts of their system that have never been exposed before" and "helped move NSE to recognize that they could not run a peer-review system on a collegial basis -- saying, 'Trust us.' They had to base it on some legal foundation. And that represents encouraging change."

Overall, he says, the reforms should assist NSF grant applicants in efforts to reconstruct the decision leading to a rejection.

Yet despite any shortcomings in NSF's traditional review procedures, Chubin says he suspects his study will show that the majority of researchers applying for grants there and at the National Institutes of Health (NIH) have always had an easier time finding out how their proposals were evaluated, thanks to the voluminous "paper trail" of key steps those two agencies generate.

In response to a request made in January by Sen. John Glenn (D-Ohio), the General Accounting Office has also begun two studies evaluating Privacy Act compliance and peer review procedures in federal agencies, with an eye toward identifying a potential model for all government agencies.

The Privacy Act entitles individuals, including scientists, to nonclassified information about themselves in U.S. government files. Applicants for research grants, for example, have a right not only to files on their peer reviews by outside scientists, but also to any diaries of phone coversations about those proposals or minutes of meetings at which their proposals were discussed, including the names of people present.

Such access was intended to give people an opportunity to identify and correct false or misleading information about themselves. For those interested in appealing an adverse judgment, such as a rejected grant proposal, it would also allow a review of the deliberations affecting the rejection.

Citing the Privacy Act, NIH has sent much of this information to grant applicants automatically since late 1978 and upon written request since 1974. But for years, getting the same information from NSF -- the federal government's single largest supporter of nonmedical basic research -- has proved more difficult, necessitating formal requests that might or might not elicit all the documents sought.

In his response to the Kalb petition, Herz sets forth a number of new NSF policy revisions. Among these is the plan to notify individuals, on the grant application form, that they have the right to review their file records under the Privacy Act.

Kalb asserts that his change -- one of the few that Herz attributes to the petition -- is more significant than it sounds. Until July 1988, according to an internal NSF report obtained by Kalb through one of his FOIA requests, NSF did not file materials relating to grant proposals under the applicants' names, but instead under th institutions for which they worked. Since the Privacy Act requires the release only of records filed under the inquirer's name, NSF's grant application files were exempt from the act, the report stated.

Moreover, the internal report documents that for at least 13 years NSF deliberately avoided filing grant proposals under the applicant's name so that it could block access to them via the Privacy Act. According to that report, dated April 1987 and entitled "Electronic Records: Legal and Policy Considerations," NSF "takes the position that portions of the grant proposal jackets [files], which often include names and comments of the peer reviewers, are not covered by the Privacy Act.... To support its position, NSF uses a [system] which prevents direct retrieval of a proposal file by the name of the submitter."

In fact, NSF could retrieve these files by an aplicant's name through indirect means. Kalb says -- and Herz has confirmed to SCIENCE NEWS--that a separate, cross-referenced file allowed the NSF staff to identify the institution name and file number for a proposal by looking up an applicant's name.

"We've checked with experts in the Department of Justice, Office of Management and Budget, the General Accounting Office, the Congressional Research Service and several congressional committees about this," Kalb told SCIENCE NEWS. "No one agrees that what NSF did conforms with the intent of the Privacy Act." He says NSF's inspector general told him he is currently investigating whether the two-tier filing system represented a criminal violation.

NSF will publish a notice in the Federal Register -- the government's document of record -- amending its current description of files subject to release under the Privacy Act, says NSF Deputy General Counsel Robert Andersen. The notice will explain that these files, indexed under grant applicants' names, now include all proposal documents going back at least to 1980 and files for funded projects going back into the 1950s. It will also point out that these files contain not only formal peer reviews but also "informal supplementary communication[s]" with the reviewers, such as informal letters and records of phone discussions.

Moreover, Herz writes, NSF will formalize its policy of automatically sending grant applicants the peer reviews from their files, with reviewers' names deleted. Such mailings have been routine, though not required, since 1984, Herz says, and were available to grant applicants upon written request for the seven years before that. NSF will now add to the automatically mailed package a more detailed summary of all meetings of outside peer reviewers discussing the proposal.

Grant applicants will receive "either a phone call or a personalized explanation of how the decision about their own proposal was made," says Jim McCullough, director of NSF's program-evaluation staff. When NSF rejects a proposal, he adds, "there will be much more feed-back," such as information on the number of proposals considered by an NSF program officer, the number of awards allowed under its budget and how well the applicant's proposal fared against its competition.

What Herz does not point out in his letter is that the automatic mailings will not include the aforementioned "informal, supplementary communications" in a proposal file. According to McCullough, a written request will produce these items and a listing of those present at review meetings.

Even the file materials mailed automatically will go out only after NSF accepts or rejects a proposal. To obtain them before a funding decision is made, applicants can send a written request. In contrast, NIH mails out such materials as they accrue.

In addressing the issue of rumors, Herz states that NSF will "encourage" program officers to "promptly notify a grant applicant if any nonscientific derogatory information or allegation, or any allegation of scientific misconduct, arises in the course of proposal review." NSF staff will also receive "clearer guidance" to ensure applicants can respond to any charges that could affect their proposals. McCullough told SCIENCE NEWS he hopes to have the new staff instructions in final form by the end of April.

In his letter, Herz says NSF's top management considered adopting "a formal rebuttal or clarification process" to examine derogatory charges or information that might harm a proposal's chances. This measure in similar to one recommended in Kalb's petition. However, Herz writes, "we concluded that the costs of such a formal and automatic process would outweigh the benefits."

The Privacy Act and the new NSF policies focus on protecting grant applicants. In Kalb's case, however, the rumor referred not to the applicants themselves but to one of their associates. Will NSF notify future applicants of charges directly defaming anyone other than themselves? "That's a good question; I don't know," NSF's Andersen told SCIENCE NEWS. Nor does he know whether the applicants' co-workers would be notified of any potentially damaging charges against them and given a chance to respond, he says. Without such notification, an episode like Kalb's could potentially recur.

Additional policy changes specified in the March 12 document include:

* In its letters confirming receipt of proposals, NSF will "invite" grant applicants to name scientists who, because of conflict of interest, might warrant exclusion from the peer review of their proposals. NSF currently asks applicants for the names of persons they regard as especially well qualified to evaluate a proposal.

* NSF will develop a computerized roster of outside researchers "on call" to peer-review its grant proposals. Once operational, the roster will enable grant applicants to review the several hundred thousand names in this file through Internet, a computer network widely used by researchers. Kalb had asked NSF to create lists of potential reviewers subdivided into the separate rosters that each NSF program officer might use in a given granting season. This would have reduced the names on each roster to serveral hundred. NSF will instead subdivide the master list into its 35 research divisions, which means that reviewing even its smallest files may require scanning thousands of names.

* NSF is broadening its grounds for grant-rejection appeals to include chllenges over "unaccounted-for conflict of interests or inappropriate consideration of records, information or rumor." In the past, few appeals have succeeded, according to nSF records obtained by Kalb under FOIA. Of the more than 13,000 proposals typically rejected each year, Kalb found, only about one is successfully appealed.

"We made commitments to make a lot of changes," Andersen told SCIENCE NEWS. "We will be diligent about following up on any commitment made."

McCullough adds that details outlining the most important of these revisions will be mailed as "Important Notices, our highest level of notification," to all gratee institutions within a few weeks.

Ironically, Kalb was out of fown when his copy of Herz's letter arrived, and so was among the last to learn of NSF's new plans. He was chasing down dinosaur remains in west texas--a hunt he perform to the paper chase.