Measuring partnership activities: Partnerships in Environmental Public Health Evaluation Metrics Manual.
A key tenet of the PEPH is community engagement. In response to an NIEHS Request for Information in 2008, the community shared concerns about the lack of evaluation capacity and the need for tools and approaches to develop project specific evaluation metrics for public health-related program areas. In response, the NIEHS developed the PEPH Evaluation Metrics Manual (NIEHS 2012b) with significant input from PEPH grantees, program staff; and experts in the field, including input from > 250 individuals at > 30 professional meetings.
Evaluation of PEPH programs provides useful benefits to grantees, including the ability to a) identify program successes; b) determine whether a project worked and why (or why not); c) identify areas for program improvement and increased efficiency; d) describe expenditures and justify a need for additional funding; e) recognize and respond to public needs and wants; f) identify new audiences and applications for projects; and g) prioritize research and plan for the future. Evaluation also may help grantees find allies in other agencies, services, or sectors; publicize achievements in communities; and inform policy and other decision making. Evaluation metrics also provide a means for the NIEHS to evaluate the success of individual projects and the PEPH program as a whole.
Typical approaches to evaluating research outcomes involve analyzing publications. However, because many PEPH programs do not publish findings related to their community engagement, we worked with grantees and community members to identify appropriate metrics to measure and demonstrate success in five areas that are common to many PEPH grantees:
* Partnering (working with other organizations to conduct environmental public health activities)
* Leveraging (using the resources already available to a project to obtain additional resources)
* Disseminating findings (providing information about environmental public health issues and results of PEPH research)
* Training (developing programs that teach researchers, community members, workers, students, and others strategies for reducing hazardous environmental exposures)
* Capacity building (performing activities that improve an organization's ability to achieve its mission).
For each of these five areas, the NIEHS developed an illustrative logic model to demonstrate connections among project activities, outputs, and impacts; > 80 examples of metrics for each activity, output, and impact are provided as examples for grantees developing metrics for evaluating the progress and achievements of their own programs.
We present the partnership logic model (Figure 1; NIEHS 2012b) to illustrate how metrics can be developed from a logic model. When developing program logic models, it can be helpful to begin by working with partners to identify and articulate the desired impacts of the program. Once partners agree about "where they want to go" (impacts), discussions of "how to get there" (activities) naturally follow. Then partners can determine "how will we know we are there?" (outputs and metrics). The nouns, adjectives, verbs, and adverbs in the answers to these questions then serve as the basis for metrics.
Examples of metrics from grantee programs that address components of the partnership logic model include:
* Demonstrating success at identifying partners (activity 1), the University of Cincinnati's anti-idling campaign provided a description of the partners involved and the resources they bring to the project: Cincinnati Public Schools provided access to students and schools; the Cincinnati Health Department provided nursing services; a councilwoman provided credibility and the ability to attract attention to the project; and the Hamilton County Department of Environmental Services provided training and information to Cincinnati Public School staff and students.
* Demonstrating their success in involving community partners in research, the University of Florida, the Farmworker Association of Florida (FWAF), and Best Start Inc. (a social marketing research firm) described how partners were involved in the Together for Agricultural Safety Project (output 4). Members of the FWAF helped develop and lead focus groups and provided input into the survey development; researchers from the University of Florida and Best Start Inc. collected data; and 382 farmworkers provided data. All three partners also contributed to an article summarizing the process by which the project was implemented (Flocks a al. 2001).
* The Detroit Community-Academic Urban Research Center is a collaborative partnership that includes the University of Michigan Schools of Public Health, Nursing, and Social Work; the Detroit Department of Health and Wellness Promotion; eight community-based organizations; and the Henry Ford Health System. To demonstrate their success at expanding research collaborations, they provided details of the additional research opportunities that were generated as a result of the partnership (impact 3). These research opportunities included four grants from the NIEHS: $5 million for establishing a Children's Environmental Health Sciences Center; $2.4 million to conduct the Community Organizing Network for Environmental Health; $2.5 million for a household intervention to reduce asthma triggers; and $2.5 million for an epidemiologic project to characterize the effect of roadway-associated air pollution on the exacerbation of asthma in children.
PEPH grantees are encouraged to adapt metrics to fit the unique characteristics of their communities, and those with multisite initiatives may want to identify metrics that are applicable for all sites. Although the PEPH Evaluation Metrics Manual was designed with PEPH grantees in mind, it may be useful to community groups, advocates, and others working to address environmental public health issues. The principles outlined in the manual also may be useful to those interested in measuring the success of basic research programs.
We anticipate that the PEPH manual will be a living document that we will update periodically. Opportunities for expansion include new evaluation topics (e.g., cost--benefit analyses and econometric evaluations), as well as new approaches used in programs (e.g., social media) and new examples of metrics drawn from the ever-expanding network of PEPH grantees.
Program staff have been conducting training related to evaluation metrics and developing stand-alone materials that will be publically available through the PEPH website (NIEHS 2012a). NIEHS staff is available to conduct webinars related to the PEPH Evaluation Metrics Manual. We encourage those interested in learning more about the manual or about developing metrics for their program to explore the manual and contact the authors with any questions or concerns.
Developing the manual was a truly collaborative process and we offer great thanks to the grantees, community partners, colleagues, and NIEHS staficwho contributed.
The authors declare they have no actual or potential competing financial interests.
Flocks J, Clarke L, Albrecht S, Bryant C, Monaghan P, Baker H. (2001). Implementing a Community-Based Social Marketing Project to Improve Agricultural Worker Health. Environ I-teeth Perspect 109(suppl 3i:467-488.
NIEHS (National Institute of Environmental Health Sciences). 2012a. Partnerships for Environmental Public Health (PEPH). Available: http://www.niehs.nih.gov/peph [accessed 11 June 2012].
NIEHS (National Institute of Environmental Health Sciences). 2012b. Partnerships for Environmental Public Health Evaluation Metrics Manual. NIH Publication No. 12-7825. Available: http://www.niehs.nih.gov/pephmetrics [accessed 13 March 2012].
Christina H. Drew
Kristianna G. Pettibone
Liam R. O'Fallon
Gwen W. Coffman
Division of Extramural Research and Training
National Institute of Environmental Health Sciences National Institutes of Health
Department of Health and Human Services
Research Triangle Park, North Carolina
Linda S. Birnbaum
Director, National Institute of Environmental Health Sciences and National Toxicology Program
National Institutes of Health
Department of Health and Human Services
Research Triangle Park, North Carolina
Christina H. Drew, Chief of the Program Analysis Branch in the Division of Extramural Research and Training at NIEHS, manages evaluation activities for the extramural program. She has > 15 years of experience in information management for environmental health decision making, and she led the effort to develop metrics for the PEPH program.
Kristianna G. Pettibone, an evaluator with the Program Analysis Branch at NIEHS, has 13 years experience conducting evaluations of public health programs.
Liam R O'Fallon is a program analyst in the Division of Extramural Research and Training. He leads the Partnerships for Environmental Public Health Program and has > 12 years of experience in community-based program development.
Gwen W. Collman is director of the NIEHS Division of Extramural Research and Training where she oversees the areas of scientific program administration, peer review, and management and administration of the NIEHS grants portfolio. Prior to her current role, Collman served first as a program officer expanding the environmental and molecular epidemiology portfolio, and then as branch chief of the Susceptibility and Population Health Branch. Through her scientific program development work, Collman has provided strong leadership to programs that focus on community--researcher partnerships and public health impacts. She received a Ph.D. in Environmental Epidemiology from the University of North Carolina School of Public Health.
Linda S. Birnbaum, director of the NIEHS and the NTP, oversees a budget that funds multidisciplinary biomedical research programs and prevention and intervention efforts that encompass training, education, technology transfer, and community outreach. She recently received an honorary Doctor of Science from the University of Rochester, the distinguished alumna award from the University of Illinois, and was elected to the Institute of Medicine. She is the author of > 750 peer-reviewed publications, book chapters, abstracts, and reports. Birnbaum received her M.S. and Ph.D. in microbiology from the University of Illinois, Urbana. A board-certified toxicologist, she has served as a federal scientist for 32 years, 19 with the U.S. EPA Office of Research and Development, preceded by 10 years at the NIEHS as a senior staff fellow, a principal investigator, a research microbiologist, and a group leader for the institute's Chemical Disposition Group.
|Printer friendly Cite/link Email Feedback|
|Author:||Drew, Christina H.; Pettibone, Kristianna G.; O'Fllon, Liam R.; Collman, Gwen W.; Birnbaum, Linda S.|
|Publication:||Environmental Health Perspectives|
|Date:||Jul 1, 2012|
|Previous Article:||A research strategy to discover the environmental causes of autism and neurodevelopmental disabilities.|
|Next Article:||EHP Classic Paper of the Year, 2012.|