The use of the National Public Health Performance Standards to evaluate change in capacity to carry out the 10 essential services.

Introduction

The Allegheny County Health Department (ACHD) is a large (with a staff of 350) local health department located in southwestern Pennsylvania. The largest city in Allegheny county is Pittsburgh. The county is predominantly urban, with 130 municipalities and a population of 1,286,000. The health department offers programs in the following environmental areas: Air Quality, Public Drinking Water, Plumbing, Solid Waste and Waste Water Management, Housing and Community Environment, Vector Control, Childhood Lead Poisoning Prevention, Food Safety, Injury Prevention, Recycling, and Emergency Management.

In early 2002, the department was awarded a Centers for Disease Control and Prevention (CDC) National Center for Environmental Health (NCEH) Building Environmental Health Capacity cooperative agreement. This grant brought the department into the national discussion about the meaning of the concept "building environmental health capacity." Identification of an evaluation tool for assessing capacity-building efforts is critical to being able to answer this question. Before 1990, agencies used self-assessment instruments to measure public health services and their capacity to perform. Following recommendations made in the 1988 Institute of Medicine (IOM) report, The Future of Public Health (IOM, 1988), public health agencies began focusing on ways to measure change in capacity and examine the relationship between resources, services, and outcomes (Turnock & Handler, 1997). Therefore, since the National Public Health Performance Standards Program provides model standards and indicators for measurement of capacity, as well as examining the relationship between performance of services and outcomes, the department decided to use this assessment tool as a way of evaluating the capacity of its environmental health program to carry out the 10 essential services of public health. (For a list of the 10 essential services of public health, see the sidebar to the article "Using 10-Essential-Services Training to Revive, Refocus, and Strengthen Your Environmental Health Programs" on page 13 of this Journal.)

Once a baseline assessment of the department's capacity to carry out the 10 essential services was completed, problem areas were targeted with process improvement activities, and a reassessment was performed to determine the impact of these activities.

Methods

In May 2002, a series of three half-day meetings were held with 15 Allegheny County Health Department environmental public health program managers. These individuals oversee the day-to-day operations of the environmental public health programs. All environmental public health programs were represented at each meeting. In advance, a copy of the National Public Health Performance Standards (NPHPS) Local Public Health System Performance Assessment Instrument was mailed to each participant for review. Participants were asked to review the material in sections and prepare for discussion at each meeting.

The NPHPS assessment tool was used during the meetings. The model standard described in the tool was reviewed, and each indicator was scored. A department administrator facilitated each meeting.

For each indicator, participants were asked to assess, on a 4-point Likert scale, the extent to which the model standard was achieved. The scores the participants assigned reflect their assessment of the percentage attainment for each indicator. Table 1 gives the scoring indices that were used.

The facilitator read the model standard for each essential service, and respondents' scores were reported verbally to the group. If, for a particular indicator, the group could not reach consensus on a score, the scoring technique was modified; an average of the percentages assigned by all the participants was used. An overall average score for each essential service was obtained by averaging of the total indicator scores within each essential-service category. For example, Essential Service 1 had three indicators with three separate scores, which were averaged to determine an overall average score. (The indicators associated with each of the 10 essential services can be viewed in a table at www.achd.net/biostats/changes_to_capacity.html.)

The facilitator recorded all the final scores for each indicator. A report was developed that showed the indicator scores for each essential service.

An action plan focusing on increasing capacity was developed to improve those indicators for which attainment scores were in the "low partially" category or the "no" category (Table 1). The action plan was implemented in September 2002. Following a two-year time period, a second reassessment using the performance standard tool was conducted. This process resulted in two comparable assessments separated by two years of capacity-building activities. To measure the change, the difference was taken between the two overall scores for each essential service. The difference was then divided by the overall score from the first assessment period to obtain the percentage change resulting from the targeted capacity-building efforts.

Results

The results from the use of the National Public Health Performance Standards to evaluate change in capacity to carry out the 10 essential services were reported in a table titled "Changes to Capacity in Carrying Out the Ten (10) Essential Services," which can be found at www.achd.net/biostats/changes_to_capacity.html. The table gives the percentage attainment for each indicator, as well as the percentage change between the results from the first assessment period and those from the second.

In the first assessment period, the overall range of scores for essential services was 42.6-90.0 percent, with a mean of 65.2 percent. The first essential service, "Monitor health status to identify community health problems," received the lowest overall score, 42.6 percent, and the lowest reported scores for individual indicators. The participants reported that the ability of the department to carry out the activities described by essential services 5 and 9 were also limited for Indicator 5.3, "community health improvement process," and Indicator 9.1, "evaluating environmental health and service programs." The lower percentage-attainment scores reported for these indicators resulted in a lower overall score for their respective essential-service categories. The group also agreed that the local health department was limited in its preparation of the public health workforce in lifelong learning through continuing education, training and mentoring (Indicator 8.3) and in public health leadership development (Indicator 8.4).

A comparison of overall scores from the first assessment period and the second assessment period demonstrated improvement in activities for each essential-service category. For the second assessment period, overall scores for each essential-service category ranged from 71.8 percent to 100 percent, with a mean of 85.8 percent. Participants' judgment about the ability of the agency to carry out the activities described by the indicator scores found improvement, except with respect to Indicator 2.2, "plan for public health emergencies"; Indicator 2.3, "investigate and respond to public health emergencies"; Indicator 8.1, "workforce assessment"; and Indicator 9.3, "operations of a local public health system" (see the table at www.achd.net/biostats/changes_to_capacity.html). The greatest improvements were in the areas of essential services 1, 3, and 5, as evidenced by the percentage change values for those services. The improvements in the scores reflect the implementation of capacity-building efforts targeted at problem areas. The least amount of improvement was noted for Essential Service 2, toward which no interventions had been targeted. A decline in activity scores was reported between the two assessment periods for indicators 2.2 and 2.3. The refunding of the cooperative agreement provided the opportunity to allocate resources to target improvement activities for this essential service.

Discussion

The goal of the study was to determine how useful the NPHPS tool is for measurement of the capacity of a local public health system to carry out the 10 essential services of public health, as well as for demonstration of change in that capacity. This exercise demonstrated that, indeed, it was possible to use the NPHPS system in this way. In the first assessment period, the lowest score was found for Essential Service 1. Although monitoring the health status of the community was a priority for the health department, a registry for collecting and analyzing environmental health data did not exist. Staff also had no access to technical assistance or to technology that could assist them with managing, displaying, and analyzing the data. This inability of the staff to carry out these activities caused the low score for this essential service. In the second assessment period, capacity-building efforts focused on improving these problem areas. An epidemiologist was hired, who began developing a set of measures that could be used by environmental programs to prioritize environmental health issues, track programmatic goals and objectives, and monitor improvement in environmental health interventions over time. As a direct result of these activities, the greatest increase in percentage attainment was for this category. Areas that in the first assessment period received low percentage-attainment scores, essential services 3 and 5, showed significant improvement by the second reassessment period. A secondary benefit was the discussions among participants that were generated during use of the assessment tool. The process was very useful in increasing the participants' dialogue about the activities of each others' environmental public health programs. The participants were experts on their own programs, but they had had few prior opportunities to discuss issues being addressed by other environmental health programs. The assessment process encouraged such discussion and gave the managers insight into each others' program goals and issues.

Conclusion

The NPHPS Local Public Health System Performance Instrument is an effective tool for measurement of the capacity to perform the essential services of public health and for evaluating change in capacity. As was seen during the meetings, the tool also fosters cross-program communication of ideas and procedures.

Corresponding Author: Gerald Barron, Director, Pennsylvania Preparedness Leadership Institute, Center for Public Health Preparedness and Associate Professor, Department of Health Policy and Management, University of Pittsburgh, Graduate School of Public Health, 130 DeSoto Street, Pittsburgh, PA 15261. E-mail: [e-mail].

Acknowledgements and Disclaimer: Preparation of this article was supported in part by a Centers for Disease Control and Prevention (CDC) National Center for Environmental Health Cooperative Agreement (Cooperative Agreement CCU324327-03) to the Allegheny County Health Department, Pittsburgh, Pennsylvania. We gratefully acknowledge the assistance of the environmental program managers and supervisors of the Allegheny County Health Department who supported this initiative. The contents of the article are solely the responsibility of the authors and do not necessarily represent the official views of CDC.

REFERENCES

Centers for Disease Control and Prevention & National Association of County and City Health Officials. (n.d.) National Public Health Performance Standards Program: Local public health system performance assessment instrument (Version 1.0). Retrieved May 13, 2007, from http://www.astho.org/pubs/pmlocal.pdf.

Centers for Disease Control and Prevention, Office of the Director. (n.d.). National public health performance standards program. Retrieved November 11, 2006, from http://www.cdc.gov/od/ocphp/nphpsp/.

Institute of Medicine. (1988). The future of public health. Washington, DC: National Academy Press.

Turnock, B., & Handler, A. (1997, May). From measuring to improving public health practice. Annual Reviews of Public Health, 18, 261-282.

G. Barron, M.P.H.

J. Glad, M.P.H.

C. Vukotich, M.S.

TABLE 1 Extent to Which Standards Are Being Met--Scoring Scale

Likert Scale Points  Response Options  Response Score Levels

4                    Yes               76%-100% of the activity
                                       described in the question is met
                                       within the public health system.
3                    High partially    51%, but not >75% of the activity
                                       described in the question is met
                                       within the public health system.
2                    Low partially     26%, but not >50% of the activity
                                       described in the question is met
                                       within the public health system.
1                    No                0 but not >25% of the activity
                                       described in the question is met
                                       within the public health system.