Communicating evaluation findings from offender programs.

With corrections implementing new offender programs across the country, public and private agencies, especially those that fund these programs, expect their investments to produce results that make a positive difference in offender behavior (for example, a reduction in recidivism). Increasingly, discussions and expectations have focused on evidence-based programming. "In general, evidence-based programs are theory driven, with program activities related to that theory, and have been reasonably well implemented and evaluated." (1) At a minimum, evidence-based programs should be grounded on a solid logic model. (2) Clearly defined, objective criteria for determining program effectiveness are important for evidence-based programming and should include:

* The degree to which the program is based on a well-defined theory or model;

* The degree to which the intended target population was actually served;

* The degree to which the intended target population received sufficient intervention;

* The quality and appropriateness of data collection and data analysis procedures; and

* The degree to which there is strong evidence of a cause and effect relationship (i.e., a high likelihood that the program caused or strongly contributed to the desired outcomes). (3)

Rigorous evaluations of correctional programs can increase our confidence that the interventions have met the standards and criteria for evidence-based programs. This process requires the collaboration of evaluators and program personnel, and such partnerships can sometimes generate anxiety and occasional resistance from both sides. (4) The American Evaluation Association (AEA), acknowledging the tensions inherent in the evaluator-evaluated relationship, developed a set of guiding principles to address potential misunderstandings in the partnership. These principles call for a clear understanding between all parties about:

* The purpose of the evaluation;

* The process to be employed in the study;

* The roles of the evaluator and the program staff;

* An expectation of the types of information to be provided during and at the end of the study; and

* The various audiences to whom the results will be presented. (5)

These are all essential parts of the formulation and evaluation of an evidenced-based approach. AEA also emphasizes that it is the responsibility of the evaluator to provide rigorous, systematic, competent, honest and respectful research. In the end, the evaluator joins the program administration in fulfilling the obligation to serve the public interest and good, especially when public monies fund the program and/or the evaluation.

Regardless of the anticipated outcomes of, or reactions to, an evaluation, a sense of unease may develop as the time for reporting the results of the evaluation approaches. Communicating the results, whether they show positive, negative or no differences in program outcomes, can be a challenge. If evaluation results are to be shared with a wide audience, some criticism will likely be experienced by both the evaluator and the program personnel. However, communication among stakeholders throughout the evaluation process can make the final stage of a project much less stressful. This column explores how to communicate evaluation findings and how communication throughout the stages of an evaluation is beneficial.

There are two phases of communicating findings. Phase one focuses on steps taken prior to the release of any official or final report. These steps eliminate some of the potential tension and criticisms intrinsic in evaluations. The second phase focuses on the findings, recommendations and dissemination of the final report.

Phase 1: Communicating Findings Before the Final Report

First, a thorough process evaluation of program activities should be conducted. This step provides feedback to all stakeholders that the program is being implemented and operated as designed and is consistent with local needs and contingencies. The disappointing results from the Project Greenlight evaluation, New York State Department of Correctional Services' (DOCS) reentry program, demonstrates that a process evaluation may have enabled program staff to address some major issues prior to the evaluation of the program outcomes. (6) If a program is not delivered as planned, positive outcomes should not be expected and an outcome evaluation should not be conducted. Additional work, including revisions to the program design and/or implementation activities, may be needed to get the program back on track before any outcome evaluation should be considered. A process evaluation increases confidence that the intervention produced the outcomes observed. However, it is not uncommon for agencies to focus only on the outcomes and forego a process evaluation in order to reduce the evaluation budget. This can be a costly and regrettable omission when programs are not functioning according to plan.

Second, evaluators should hold a series of planned meetings with all stakeholders throughout the stages of the evaluation in order to give and receive input important to the interests of the program. Program corrections and improvements should be made based on periodic process evaluation reports. This helps both the evaluator and the program staff. Even when evaluation findings are contrary to expectations, this forum eliminates unwelcome surprises in the final report. Patton, in Utilization-Focused Evaluation, suggests that "a 'good' evaluation process should build in feedback mechanisms to primary users that guarantee the relative predictability of the content of the final report." (7) This process is normally considered part of a formative evaluation, and formative feedback "is most useful as part of a process of thinking about a program rather than as a one-shot information dump." (8) Providing formative feedback helps shape the direction the program takes, prevents mission drift and leads to a more effective intervention.

Third, evaluators should share a rough draft of the evaluation report with all stakeholders, offering them a chance to assess and respond to the findings in case there are any inaccuracies that need to be corrected or alternative explanations that should be included. This gives them an opportunity to prepare for questions or criticisms and to formulate an action plan prior to the release of the evaluation report.

A fourth but often neglected step is to have experts in the field review the findings before they are released. Because of the negative outcomes of the Project Greenlight reentry program, the evaluators did two important things regarding expert review. First, they had the National Institute of Justice review their methodology to ensure that a rigorous methodological approach had been used. Although this occurred after the findings in Project Greenlight, ideally the methodology review should be completed during the planning stages of the evaluation. Second, the evaluators convened a panel of experts to discuss the findings and to explore what may have caused the negative outcomes. (9) These important steps increase the credibility of the methodology and strengthen forthcoming recommendations. Again, this is important in describing a program as evidence-based.

Phase 2: The Final Report

Evaluators typically want their findings and recommendations to be utilized as a learning tool to improve or enhance interventions instead of an absolute judgment of the value of the program as good or bad. (10) Final reports and the recommendations contained therein are an opportunity to influence important policy decisions. For example, the Pennsylvania Department of Corrections has an excellent practice of taking recommendations from an evaluation report and turning them into action plans. However, this requires that recommendations by evaluators be meaningful if they are to be taken seriously. Patton's "utilization-focused evaluation" approach offers several noteworthy guidelines for developing useful and practical recommendations:

* Clarify the kinds of recommendations expected by the stakeholders;

* Make recommendations that clearly follow from and are supported by the evaluation findings;

* Provide multiple options for a course of action (e.g., increase funding, reduce funding, terminate the program, etc.) and list the pros and cons of each recommendation;

* Focus on actions within the control of intended users;

* Exercise political sensitivity;

* Carefully and deliberately word evaluations and avoid using vague or obtuse language;

* Allow time to develop meaningful recommendations; and

* Develop strategies to get stakeholders to take ownership of the recommendations. (11)

Upon release of the final report, the next step involves disseminating the results to the targeted audiences. Government and corporate organizations often spend substantial resources planning for detailed and systematic dissemination of evaluation reports. Today, in corrections, it is common for funding sources to require full dissemination plans as part of the evaluation process. The essential elements of the plan address to whom the report will be released (correctional administrators, program staff, legislators, media, the general public) and the dissemination formats (conference presentations, Web sites, press releases, practitioner and academic publications). A thorough dissemination plan enhances the impact of the report and facilitates a coordinated response to inquiries from the target audiences and others.

Program development, evaluation, dissemination and replication are all part of a process for evidence-based programming. This column introduces the need for communication among program staff and evaluators from the beginning (ideally in the program development phase) to the end (the dissemination). Future columns in this space will address other aspects of what "evidence-based" means and how to conduct evidence-based programming.

ENDNOTES

(1) Chinman, M., P. Imm, and A. Wandersman. 2004. Getting to outcomes 2004: Promoting accountability through methods and tools for planning, implementation, and evaluation. p. 45. Santa Monica, Calif.: Rand. Available at http://www.fpg.unc.edu/~eco/pdfs/TR101.pdf.

(2) Stinchcomb, J. 2001. Using logic models to focus evaluation efforts: Translating operational theories into practical measures. Journal of Offender Rehabilitation, 33(2):47-65.

(3) Chinman, Imm and Wandersman, 2004. p. 44.

(4) Donaldson, S., L. Gooler and M. Scriven. 2002. Strategies for managing evaluation anxiety: Toward a psychology of program evaluation. American Journal of Evaluation, 23(3): 261-273.

(5) American Evaluation Association. 2006. The guiding principles of the American Evaluation Association. Available at www.eval.org/Publications/GuidingPrinciples.asp.

(6) Wilson, J. and R. Davis. 2006. Good intentions meet hard realities: An evaluation of the Project Greenlight Reentry Program. Criminology and Public Policy, 5(2): 303-338.

Rhine, E.E., T.L. Mawhorr and E.C. Parks. 2006. Implementation: The bane of effective correctional programs. Criminology and Public Policy, 5(2): 303-338

(7) Patton, M. 1997. Utilization-focused evaluation, edition three. p. 334. Thousand Oaks, Calif.: Sage Publications.

(8) Ibid.

(9) Brown, B. and R. Campbell (2005). Smoothing the path from prison to home: A roundtable discussion on the lessons of Project Greenlight. New York: Vera Institute.

(10) Patton, 1997.

(11) Patton, 1997. pp. 324-326.

Linda G. Smith, Ph.D., is a research consultant who has worked in corrections for more than 25 years as a practitioner, researcher and professor. Roberto Hugh Potter, Ph.D., has been working with corrections since 1977 as a therapist, researcher, trainer, educator and program coordinator.