Is your research improving health services?

Evaluating the impact of health research is an essential component of efforts to put research into practice. Donors and policy-makers are often not satisfied with research results alone. Increasingly, they are asking, "So what? What is the relevance of the research for improving health services and outcomes?"

This issue should ideally be considered when research questions are being formulated to ensure that the research will be relevant to practice. Certainly, it must be considered once research results are available and related promising interventions to improve health are being selected for implementation. The first challenge then is to identify interventions that "yield good value for money." Subsequently, as interventions or health research results are selected and moved into practice, the challenge is to determine whether they are enhancing the short-term performance of specific programs. Finally, and perhaps of greatest importance to donors and policymakers, is the need to determine whether research results and the selected interventions have a longer-term, broader impact.

The Handbook of Indicators for Family Planning Program Evaluation, (1) a publication produced by the Evaluation Project of the U.S. Agency for International Development (USAID), distinguishes shortterm performance evaluation from longterm, or impact evaluation. In performance evaluation, specific indicators may be gathered on program resources, activities, or results. In contrast, impact evaluation generally uses population-based data to measure the intermediate outcome (effect) or long-term impact of an effort.

Contraceptive prevalence is an example of an intermediate outcome of many family planning programs. In contrast, a change in fertility rates is an example of a program's long-term impact. A change in fertility rate, however, can take many years to occur. Furthermore, it is difficult to identify clearly the factors that produced such change. For these reasons, even longer-term evaluations tend to focus on intermediate outcomes rather than impact.

Research on a checklist to rule out pregnancy among nonmenstruating family planning clients (see article, page 21), serves as an example of looking at intermediate outcomes. A study conducted by FHI in Kenya demonstrated that the checklist correctly identified more than 99 percent of 1,629 nonmenstruating women visiting family planning clinics as not being pregnant. (2) When providers had training and good access to the checklists, refusals by providers to give contraceptives to nonmenstruating clients declined from 34 percent to 4 percent. (3)

Would introducing the checklist to rule out pregnancy have a long-term impact on fertility rates in Kenya? This is far more difficult to demonstrate. Nonetheless, the checklist's simplicity, low cost, and shortterm effect on provision of contraceptives in Kenya qualify it as an intervention likely to yield good value for money. Follow-up research in each of the settings where the checklist has been disseminated and adopted would clarify whether its effects there were similar to those observed in Kenya. Indeed, such research conducted by FHI in Senegal and Guatemala has yielded preliminary data showing marked declines in refusals to provide contraceptives to nonmenstruating family planning clients in clinics after the checklist became available there. But because the costs of replicating such research may outweigh its benefits, FHI has also chosen to simply distribute a follow-up questionnaire to those who requested the checklist for use in their programs. This will help document the extent to which the checklist has been used, accepted, and shared. It should also clarify the checklist's usefulness in these settings.

The findings generated by this questionnaire will not determine the long-term, broader impact of the checklist. But relatively simple findings like these are often what most helps program planners and policy-makers assess the value of research findings, ultimately clarifying the answer to the question of "So what?" and facilitating movement of research into practice.

By Susan McIntyre, MSW, MPH

Director of Evaluation of FHI's Contraceptive Technology Research Program.

McIntyre, who has worked with the Contraceptive Technology Researched Program for much of her FHI career, now works with colleagues to assess and document the programs's impact.

References

(1.) Bertrand J, Magnani R, Knowles J. Handbook of Indicators for Family Planning Program Evaluation. Chapel Hill, NC: The Evaluation Project, 1994,

(2.) Stanback J, Qureshi Z, Sekadde-Kigondu C, et al. Checklist for ruling out pregnancy among family-planning clients in primary care [letter]. Lancet 1999;354(9178):566-67.

(3.) Stanback J, Brechin SJG, Lynam P, et al, The Effectiveness of National Dissemination of Updated Reproductive Health/Family Planning Guidelines in Kenya. Final Report. Research Triangle Park, NC: Family Health International, 2001.