Printer Friendly

The rush to measure outcomes: process evaluation and return on investment; how process evaluation has been used to enhance ROI in a statewide parenting project for youth at risk for tobacco, alcohol, and drug use. (Article).

In the face of scarce resources, funders and program administrators are keenly and understandably interested in demonstrating as soon as possible the outcomes and even the impact of services provided. All too often, this eagerness to measure outcomes yields disappointing results. Why? In the rush to offer proof that programs are working, opportunities for examining critical processes that underlie the achievement of outcomes may be missed. Simply stated, process evaluation raises the probability that programs will lead to the desired outcomes.

Process evaluation offers an opportunity to examine the expectations, the structure, the organization, the delivery flow, and the ease of use involved in effective program delivery. Most importantly, process evaluation provides a means of clarifying what needs to be adjusted to ensure that a program can be successful. An investment in process evaluation strengthens the context within which outcome evaluation takes place.

In this article, we have cited examples of process evaluation at the state level from the area of prevention programs for substance abuse, emphasizing parenting education.

What Is Process Evaluation?

Process evaluation is a methodical approach to examining and assessing the strategic systems and processes included in program delivery, the implementation patterns, and the procedures used to ensure service delivery. In describing the strategic and logistical components of program delivery, process evaluation identifies those approaches that work effectively, while highlighting those requiring improvement. Further, process evaluation provides information that directly impacts the possibility of program replication.

In a statewide project in Arizona seeking to upgrade parenting skills in parents of youth at risk for tobacco, alcohol, and drug use, a process evaluation was undertaken by the evaluation team prior to embarking on an outcome evaluation. The purpose of the process evaluation was to examine closely those factors that would put in place the framework needed to maintain strong parenting classes.

Among the components examined during the process evaluation were the following:

* identification of target families (e.g., those at risk);

* recruitment approaches used to bring parents into the program; and

* retention techniques to ensure that enrollees remained in the program.

Nitty Gritty Issues

Process evaluation emphasizes those areas that might be described as the "nitty gritty," or common sense elements that someone not in the field of prevention might ask. Among the questions that might emerge are these:

1. Is the program offering reaching the right people?

2. What methods are used to recruit the appropriate target population?

3. Once people enroll in the program, what methods are used to ensure that participants complete the course?

Answering such questions represents an important part of process evaluation, which seeks to establish that the program is happening, and that it is being delivered to the specifications agreed upon by funder and provider.

Two important areas that process evaluation can assess are verification of service provision and delivery efficacy.

Verification of Service Provision

Is the service reaching the intended population in the manner prescribed, and accomplishing what it purports to accomplish? In the case of statewide parenting programs, this is a critically important category of investigation. The funder wants to ensure that what has been agreed upon by the funded agency is actually being delivered to the clients. Once this has been determined, it will be possible to plan a review of outcomes planned and achieved. However, without first verifying that the program is actually being provided to clients, Outcome evaluation is not possible.

* What changes should be made in the identification of service population?

For example, are methods of identifying target families sufficiently broad? Do methods exclude families who need to be exposed to the program but would have no way of finding out about it?

* What recruitment practices are most successful in reaching the target population? From among the many possible ways of communicating the availability of parenting programs to parents, are person-to-person techniques such as referrals by school counselors, school psychologist, principals, or school social workers more effective than such mass-communication methods as sending home flyers with students, placing advertisements on local radio broadcasts, or advising church or synagogue groups of program availability. Do specific techniques work more effectively in some locations than in others? Process evaluation has an important place in helping discern what works best in the "real world."

* What practices are most/least cost effective? Recruitment practices can be costly or reasonable. Depending upon the number of parents that any given method generates, it may have value as an approach.

* What methods are most effective for ensuring that participants are retained in the program? Need alone does not determine whether people who in fact need the program will remain enrolled in the program until completion. For this reason, one major process evaluation concern in prevention programs involves an investigation into what makes people enroll as well as what keeps them coming back.

* To what extent does the program respond to the needs of the service population? Another core element of process evaluation involves examining the degree to which proven programs respond in a helpful way to the needs of actual participants who fit the profile of the target population. Methods of finding out the degree to which parents of at-risk families believe that parenting programs help them to perform their responsibilities more effectively are vital to a process evaluation.

* What changes can enhance the program's efficacy in responding to the needs of the service population? In the event that the program is insufficiently effective or complete in its response to service population needs, it is important for the process team to specify what needs to be done to address needs more directly and completely.

Delivery Efficacy

How effective and efficient are the service delivery mechanisms in reaching the client base, relative to costs and benefits? Regardless of how strong the program design, it is effectively only as good as its ability to reach the target population in an appropriate balance of the program's expense and the benefits that participants derive. Several critical questions follow, including examples based upon parent education.

* How easily can the target population access the services provided through the program? Does the program plan for parent education provide for parents to attend classes at times and places that are at least possible, and, preferably, convenient for them?

* What changes should be made in program delivery to facilitate ease of access? If for any reason parents find it difficult to attend programs, these reasons should be addressed and changes made to improve ease of access. Process evaluation will determine access challenges that require attention and action.

* To what extent are methods of program delivery efficient and cost effective? Process evaluation should investigate the degree to which program delivery methods are practical and "resource wise" for the type of population that the program seeks to serve. For example, if the vast majority of parents is not available for class participation on a series of week nights over a six to eight week period, but can be present on two successive Saturdays for a longer period of time, then it is wise for program coordinators to structure the plan to meet these needs.

* What, if any, changes in delivery method would enhance the efficiency of the program? If styles of facilitation, organization of lessons, appropriateness of materials, inclusion of participant discussion time, or other factors are perceived as possible elements of improvement, they can be shown to be valuable for strengthening program delivery. Techniques such as participant interviews, facilitator interviews, and facilitator or participant end-of-course surveys can be useful in responding to process evaluation questions identified at the beginning of the study.

Reality Test

The reality test of program practices that process evaluation is structured to accomplish provides valuable information about the extent to which services are being delivered as agreed. In the case of the statewide parenting program serving school communities, performing a process evaluation laid the groundwork for measuring the effectiveness of curricula for improving the efficacy of parenting processes. Without firmly establishing whether proper participant identification, recruitment, and retention processes were adequately in place, the outcome evaluation would have been handicapped.

Major Functions of Process Evaluation

The major functions of process evaluation are to:

* delineate practices that optimize the program's effectiveness;

* specify patterns of effectiveness, both quantitative and qualitative in nature;

* indicate issues and opportunities for improvement, based upon current implementation; and

* provide recommendations for subsequent action related to program support and implementation practices.

By its very nature, process evaluation is formative, identifying potential opportunities for improvement during the process of program delivery. Calling attention to areas that facilitate high functioning of a program paves the way for effective outcome evaluation by specifying approaches and procedures that raise the probability of program success. In effect, effective process evaluation can be considered a prerequisite to outcomes evaluation by virtue of:

* Furnishing the procedural infrastructure represented by support systems that facilitate the accomplishment of program objectives. For example, partnering with local community businesses to provide giveaways to parents enrolled in programs offered incentives that strengthened schools' ability to be successful in providing parenting programs. Similarly, establishing a system of principals issuing invitations to parents to join in "family-strengthening" sessions not only added to enrollment, but it added credibility to the process.

* Ensuring the presence and functioning of favorable practices that increase the probability that outputs will be delivered as required for achieving target outcomes. The aforementioned positioning of parenting classes as "family-strengthening" sessions, or discussion groups added prestige to the program perception and removed the stigma of the program as remedial in nature.

* Providing necessary course corrections early in the process to facilitate smooth delivery of programs. Where ineffective recruitment practices were being followed early in the program, an insufficient number of parent participants were enrolled in programs to establish critical mass in effective parenting processes. Ineffective practices, such as posting program information in places where parents were unlikely to see it, were replaced by one-on-one contact to at-risk families by school counselors and school social workers.

Key Elements to Examine and Report in Process Evaluation

Conducting process evaluation is initially descriptive in nature. The evaluation professional guiding the process seeks to assemble details that paint a picture of how the program is being implemented. This methodical mapping of the process, clarifying success points and specifying opportunities for improvement, assists program administrators in determining ways of adjusting the program to achieve the target objectives and attain target outcomes.

Areas important to process evaluation include, but are not limited to, the categories listed below.

Level of Effort Required by Staff

What level of effort is required of staff to deliver the product or services? Are there easier, more efficient ways of delivering the service? How does the staff perceive the efficacy of delivery methods? What recommendations do staff have for any needed improvement in service delivery? School-based parent program facilitators needed to be offered a suitable stipend for their efforts in coordinating and presenting classes on a routine basis. The process evaluation conducted for a particular statewide effort revealed the importance of providing child-care, food, or snacks for participants to enjoy during sessions, funds for transportation, and stipends for parents completing the program. In many cases, the presence of funds made available for such uses made it easier for facilitators to direct more of their attention to the participants and to program materials.

Staff Training Support

What staff training supports service delivery? Is the training sufficient for preparing staff to delivery the program? What additional training would be useful for enhancing staff knowledge and delivery?

Processes for Client Identification and Recruitment

What methods are in place for identifying the service population? What recruitment methods are in place for gaining clients? Are these methods culturally and community-appropriate for reaching the target population? When dealing with at-risk populations, a keen sensitivity to the myriad of cultures involved is essential.

Client Requirements for Accessing Service

What is required of clients who experience the service? To what extent are client requirements suited to the day-to-day life of those participating? Is the level of effort excessive, insufficient, or appropriate? Only a thorough process evaluation can provide the essential examination of how realistically suited to daily life the course requirements are. In addition, such key concerns as literacy level, language limitations, and learning capability need to be identified.

Client Response to Program

How do clients respond to the program? How do clients assess the quality of the program being provided? What, if anything, do clients request receiving more or less of in the program? What do clients ask to receive that they are not receiving now? Do clients indicate that they would recommend the program to their peers?

Staff Assessment of Program

What is the staff's assessment of the quality of program content and delivery? What are the strengths and opportunities of the program? What changes do staff recommend in program content, delivery, or materials? Facilitators' assessments of program quality and delivery offer yet another piece of input that provides clues as to the usefulness of the program. Many facilitators have excellent access to what participants respond to best, such as opportunities to bring in actual problems they were experiencing and ways of sharing ideas with one another. In addition, facilitators have a clear sense of which curricular components were useful and which were not.

Practices for Process Improvement

What processes are in place to improve the program? Who has participated in identifying these process improvements? Who is assigned to take part in the design of these processes? Identification of processes established for program improvement, including the enhancement of support for child-care and meals, prove useful in helping improve program structure and support. Equally important is the fact that administrators, teachers, staff, parents, and community members all have a voice in the design and voicing of potential program enhancements.

Summary

Process evaluation represents a source of invaluable information that can vastly improve the likelihood of a program's attaining its outcomes. Organizations that seize the opportunity to examine closely what is required to support and implement their programs typically reap the rewards of making this investment. The most successful school communities participating in the statewide parenting program were those that took advantage of and acted upon the process evaluation information provided to districts under the label "lessons learned." Typically, process evaluation yields a clearer understanding on the part of program staff and stakeholders of what a program requires and the adjustments needed to facilitate delivery. Until these measures are well understood and addressed, the attainment of outcomes will usually be subject to chance.

Sheila E. Murphy has served as president of Sheila Murphy Associates, a Phoenix-based firm specializing in program evaluation and organizational analysis, since 1993. Jane Dowling, is president of Wellington Consulting Group, Ltd., a Chicago-based research and evaluation firm established in 1992.
COPYRIGHT 2002 Bureaucrat, Inc.
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2002 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Author:Murphy, Sheila E.; Dowling, Jane
Publication:The Public Manager
Geographic Code:1USA
Date:Jun 22, 2002
Words:2453
Previous Article:The department of the Treasury moves into the workplace of the future: case study of the Virtual Resource Solution (VRS) launched by the Treasury...
Next Article:Private finance initiative: a better public-private partnership? (Article).
Topics:


Related Articles
Faith-based prevention model: a rural African-American case study.

Terms of use | Privacy policy | Copyright © 2018 Farlex, Inc. | Feedback | For webmasters