Evaluating your camp program.

In the last several years, many agencies have been under increased pressure from funding sources to document the impact they have on the lives of the young people they serve. Camp Fire Boys and Girls in Des Moines, Iowa, undertook the following steps to develop a method to measure these outcomes. The evaluation method represents a collaboration between a social work professor skilled in program evaluation and the agency staff responsible for the administration of the day camp program. The product of this collaboration resulted in a methodology and assessment tool that could provide data for: 1) the agency administrators to document program effectiveness, and 2) camp counselors to identify specific behavior problems and then develop interventions.

The Programs

Initially, the day camp program was offered in one site, and through additional funding, another site was added. One site serves an average of ninety-five children each week and the other an average of forty-five youth per week. These programs were developed in response to several pressing needs within the neighborhoods they served:

* Parents needed a safe place for their children to play when they were not at home.

* Parents had difficulty getting their children to activities due to work obligations.

* Parents wanted programs close to their homes because of safety concerns.

* Children wanted to be involved in active programs so they would not be confined to home in their parents' absence.

Children in these programs came from low-income families and were between the ages of five and twelve. Like most summer day camp programs, the activities focused on self-reliance courses, development of outdoor and basic educational skills, field trips, creative activities such as arts and crafts and drama, and service projects. Breakfast, lunch, and snacks were provided to campers. College-aged youth were recruited as camp counselors to supervise groups of ten campers for ten weeks.

Developing the Evaluation

The evaluation of these two camp programs was launched in the summer of 1995 as a field test of the process and the instruments. The initial evaluation was a valuable learning experience in how the evaluation was to be conducted to ensure valid data. In its second year of use (summer 1996), the evaluation was better organized and the outcome information obtained from it appeared more valid.

The first difficulty that most agencies face in developing an evaluation for their programs is finding a skilled evaluator who is interested in their program. University faculty can be a good resource for agencies. Sometimes faculty take on these projects themselves and sometimes they assist graduate students who undertake the evaluations as part of their learning experience. The only disadvantage to graduate students is they sometimes have just one or two semesters to devote to a project. Often, they can help develop the process, but they may be gone by the time the data comes back.

The logistics of working with university faculty and/or their students must be discussed in advance of the project to ensure its timely completion. Private consultants in program evaluation, such as those who do grant writing, may also be available in your community. United Way staff may be able to help you identify professionals who do this type of work.

Funding program evaluation is another important consideration for agencies as they identify potential consultants. A reasonable budget must be set. If program evaluation is an important administrative responsibility within the agency, then those costs must be included in program budgets or other sources of assistance must be sought.

For this project, an agency board member agreed to underwrite the initial costs of developing the evaluation methodology and assessment tools. Subsequent costs of data entry, analysis, and report writing at the end of each program year have been written into the program budget.

After identifying an evaluator and negotiating the fee for service, the next step in the evaluation process was to orient the evaluation consultant to the aims of the camp, the types of activities offered, a profile of the campers, and an orientation to the challenges these camps offered to agency staff. This orientation process was conducted in two meetings with program administrators and was aided by grant materials that were provided to the evaluator before the meetings.

Identifying Program Objectives

Following these orientation meetings, the next task was to specify measurable program objectives that would satisfy funders' interests in outcomes for children. The evaluator and program staff focused on the following broad objectives:

* Promote personal responsibility among campers and focus on helping them decrease inappropriate personal behaviors.

* Promote positive peer interaction among campers.

* Promote positive camper-adult interaction.

* Promote increased knowledge of community recreational opportunities among campers.

* Promote enjoyment of the camp experience among campers. These objectives reflected anticipated changes that children who participated in these camps would make as a result of their involvement in camp activities over the ten-week program.

Choosing an Evaluation Method

Once the behavioral-focused objectives were refined and specific percentages of goal attainment for each were reached (for example: 75 percent of campers will demonstrate a decrease in negative peer interaction), the evaluation team began choosing a methodology to collect information about these program objectives. Two methods were considered: 1) a pre-test/post-test method in which information about campers would be collected at the beginning of camp and again at the end, and 2) a post-test-only measure in which information about client change would be collected only at the end of camp. The pre-test/post-test method - although more labor intensive than post-test only - was chosen as the most valid methodology to ascertain changes campers made as a result of program activities and counselor guidance.

A two-page instrument that would be used by camp counselors to record their observations on campers under their care was developed by the evaluation consultant with input from program staff. A draft of this instrument was reviewed by program administrators and changes in level of wording were made before supervisors and counselors were trained in its use.

Content of the Evaluation Instrument

At the beginning of camp, staff were asked to record basic demographic information about campers (age, sex, race, living situation, and annual household income) on the intake/pre-test form. Staff were also asked to record how many times the child had attended this camp previously and if the child took medication for a behavior problem (since administrators were interested in documenting the number of children who were already experiencing severe behavior problems that warranted medication). Much of this information was taken from enrollment forms completed by parents.

The next section of the form included three scales: personal behavior, interactions with peers, and interactions with adults. Each scale was comprised of five to thirteen items that identified behaviors in that domain. For example, the personal behavior scale listed thirteen items, including swears or uses inappropriate language, has difficulty paying attention, is easily frustrated with tasks, and has temper tantrums. The nine-item interactions with peers scale included the following behaviors: does not play with others/isolates, is easily taken advantage of by others, and interrupts the play of others. Statements in the five-item interactions with adults scale included: defiant when asked to comply, very attention-seeking, and appears fearful/hesitant with adult.

For each of these scale items, the staff member chose one of five responses to indicate how frequently the camper exhibited that behavior. The responses were: 1, never; 2, very infrequently; 3, sometimes; 4, frequently; and 5, very frequently. Low scores meant more appropriate behavior since the majority of scale Rems reflect negative behavior. Items that were positive in the original scales were reverse scored during the data analysis. Staff completed the same scales for each camper at program exit (post-test).

In addition to these scales, staff were asked to rate the child's knowledge of community resources from among four responses (1, poor, to 2, excellent) and how much enjoyment the child appeared to have for camp (on a continuum of 1, none, to 7, extreme). Again, these items were rated at both program intake and exit. Statistical reliability analysis was conducted on each of the three scales. Each scale was found to be very reliable.

Outcome Results

The evaluation of the two day camps in the summer of 1996 was based on eighty-six youth for whom pre-test and post-test data were available. Fifty-six of these youth (65 percent) attended one camp; the remaining thirty (35 percent) attended the other camp.

These youth ranged in age from five to twelve. The average age was eight years old. Fifty-three percent of the youth were female; 47 percent were male. Fifty-four percent (n = 46) of campers were African-Americans; four percent (n = 3) were Native Americans; 6 percent (n = 5) were multiracial; and 37 percent (n = 32) were white. Youth were predominantly from single parent, low-income families. Six of the youth (7 percent) were reportedly on medications for behavior problems.

Pre-test/Post-test Data Analysis

In analyzing the information from the pre-test/post-test assessment forms, changes in the behaviors of campers, their knowledge of community resources, and their observed enjoyment of camp was noted. This analysis was conducted using The Statistical Package for the Social Sciences: SPSS, a popular statistical software program. Based on the analysis, program administrators were able to document to funders that a statistically significant positive change occurred among campers who participated in these two summer programs.

Changes in peer interaction from camp beginning to exit were statistically significant among campers (p = 0.03) and personal behavior change among campers approached statistical significance (p = 0.09). A significant positive change in the youths' knowledge of community resources from program intake to exit (p = 0.02) was observed by counselors. Campers were found to enjoy the camp significantly more at the end than at the beginning as they had more opportunities for fun activities and friendship (p = 0.02). Although there was a positive change in their interaction with counselors from intake to exit, this change was not statistically significant (p = 0.34).

Useful Lessons

The validity of the data was improved from the first test in 1995 to the second evaluation in the summer of 1996. Staff needed training in the use of the instrument to ensure the validity and reliability of the data.

The pre-test/post-test form was useful because it had observable behavioral indicators in the three most common domains that the camp attempted to influence: personal behavior, peer interaction, and appropriate interaction with adults. Less specific (and consequently less valid) were the single-item questions, which focused on knowledge of community resources (a goal of the camp was to enhance knowledge of resources among youth) and enjoyment of camp by the children.

Nonetheless, this simple two-page form was useful as a guide not only for the overall evaluation of the camp, but also as a guide for intervention for specific identified problems of each camper. Unlike many behavioral checklists for children, the form developed for this program was comprised of just twenty-seven scale items. In using paraprofessionals to collect data, simplicity of effort is an important element to consider. If evaluation forms are too complex for staff to easily complete, the validity of the data is likely to suffer.

The evaluation proved very useful in documenting the effectiveness of the summer day camp program to funders as well as providing a useful tool for staff to assess and monitor specific problem behaviors of campers. After identifying specific problems, staff designed behavioral programs to address these identified problems. The effectiveness of these behavioral interventions was subsequently revealed in the post-test scores for each youth. This evaluation had both administrative and direct client benefits and would be useful for similar programs facing demands for outcome evaluations.

Edward Saunders, Ph.D., is an associate professor of social work at the University of Iowa. He has served as a consultant in program evaluation to many health and human service agencies in Iowa. He is the coauthor of Program Evaluation: A Primer for Human Service Agencies, a monograph published by United Way of Central Iowa.

Suz Welch, CCD, is the executive director of Camp Fire Boys and Girls, Des Moines, Iowa, and has been director of Camp Hantesa, in Boone, Iowa, since 1972. She is also the national vice-president for the American Camping Association.

DD Gass is assistant executive director and director of day camps for the Heart of the Hawkeye Council of Camp Fire Boys and Girls, Des Moines, Iowa.