Evaluation: what to measure and why?

* Two parents complained about the same thing. Should you be worried?

* Once again, the fifth-week summer slump hit the staff. Is there any way to avoid it?

* The campers just don't seem to be as enthused and involved as you would like. What do they really want?

* The organization is not running as smoothly as you would like. Where can you get ideas for improvement?

* Camper enrollment is down 15% since last season. What could be the cause of it and can the trend be reversed?

Anyone who is involved in the provision of camp programs is concerned with measuring the success of those experiences. Through a process called evaluation, camp professionals can verify what is intuitively understood about the benefits of camp.

All camp settings -- day camps, resident camps, and specialty camps -- require some form of evaluation to help determine successes and limitations. The success of camp environments may be measured in terms of the benefits derived by campers, staff performance, usefulness of program design, appropriateness of areas and facilities, use of financial resources, community support, public relations value and, in general, its relevance to the overarching goals of the organizing body.

Understanding evaluation

The process of evaluation begins when the camp organization clearly articulates its vision, mission, and philosophy, and questions how well it is functioning according to those statements. The following excerpt is from the vision statement of Camp Adventure [TM], an international youth-serving agency. The statement serves as the basis for all that Camp Adventure [TM] does and is the primary standard for measuring its success.

Camp Adventure [TM] will be an environment of magic and delight, wonder and discovery, fellowship and friendship, learning and sharing, and pride and achievement.

A vision statement is clear, concise, and compelling, and becomes the catalyst for all of the organization's efforts. Although the vision statement does not provide the criteria for evaluation, it is the beacon that provides direction and opportunity for measurement, control, and improvement of services.

Most camps are committed to their vision, feel that their program is unique and of high quality, and that their services make an impact upon those they serve. Evaluation provides those associated with camp the necessary information to make factual, objective judgments about program benefits, program design, staff effectiveness, areas and facilities, logistical support, and other information.

There are two basic types of evaluation: formative and summative. Formative evaluation occurs on a continuous basis throughout the life of a camp and provides opportunities for ongoing adjustments. One example would be mid-summer staff evaluations. Summative evaluations usually occur at the end of a particular program or camp season. For example, many camps interview campers or send an evaluation survey home to parents at the completion of the camp session. Summative evaluations are comprehensive in nature and serve as summaries of what was learned during the program or season that could be incorporated into future programs. For formative and summative evaluations to be effective, the reare affected: campers, parents, counselors, camp directors, camp administrators, governing boards and bodies, and others interested in the value and worth of camp programs.

What is evaluated?

The concept of evaluation is often overwhelming because there seems to be so much to examine. Effective organizations evaluate in four main areas: administration, facilities and grounds, programming, and staff.

* Administration: Typical areas of administrative evaluation include:

examining the mission statement -- Does it send the intended message? Is it being manifested in all camp operations?

organizational goals -- Are they in harmony with the mission statement? Are they attainable at some future point?

organizational structure -- Is the existing organizational chart effective? Is it efficient?

staff allocations -- Are staff members placed where they can best contribute and succeed? Are they where they are needed during peak times of the day?

and policies and procedures related to basic operations --

Are hiring/firing procedures equitable, just, and humanistic? Are severe weather policies based on sound information and logic? Are purchasing and budgeting procedures cost and quality effective?

* Facilities and grounds:

Many camps, particularly those that are accredited, go through an evaluation process to examine safety, function, and placement issues related to facilities. Other areas to be evaluated include environmental harmony issues; effective use of space; and layout and design of open space in relation to roads, service, program, and living areas. The types of questions asked should reflect the mission, values, and goals of the agency.

* Programming:

As camp professionals we want and need to know whether or not our programs are successful, well-liked, and safe. Program evaluation can be informal or formal.

Informal evaluations involve: observing behavior --

You notice that campers seem reluctant to register for a special program.

and listening to campers' aside comments --

"This was a great program!" one child says to another as they go into dinner.

Formal evaluations often involve standardized verbal or written questions asking campers what they thought about a particular program.

* Staff:

Frontline staff are critical to the success of any camp program; camper enjoyment, health, and safety are directly linked to the quality of staff employed in a program. In addition, senior camp staff and camp directors provide much in the form of mentorship to younger and newer staff members. Formative (periodic) staff evaluations throughout the season are a win-win situation: evaluating and sharing results with staff members gives them a chance to improve; campers receive the benefits of improved performance.

Evaluation Functions 

Each time the evaluation process is undertaken, various functions must be considered to meet the unique demands of a camp program. As noted in the following statement, the Camp Adventure [TM] staff is charged with the responsibility of pursuing multiple outcomes. Some of these outcomes are focused and technical, while others are broader in nature.

Camp Adventure [TM] staff will provide a range of enjoyable summer adventures designed to foster social interaction, facilitate personal growth, and develop educational and recreational skills. We strive to produce fun, joyful, exhilarating and treasured experiences.

While individual organizations may create or adapt evaluation activities that are unique to their campsetting, there are three broad functional areas of evaluation that every camp should consider.

* Human Relations:

Camps are human service providers, therefore, the human relations function is critical for all camp staff whether they be program staff, groundskeepers, kitchen staff, or office staff. All camp staff should be evaluated on their communication and active listening skills, their understanding of group dynamics, their enthusiasm and consideration, and their skill as group facilitators and team players. Key interpersonal skills can also be incorporated into a mission statement. For example, Camp Adventure [TM] emphasizes such things as social interaction and personal growth in the above statement.

* Technical Function:

The technical elements of leadership and programming directly relate to camps' concern for program quality. The conduct and implementation of camp tasks and programs (including everything from lawn maintenance to special activities) can be evaluated in terms of preparedness, handling of logistical issues, safety, public relations, and other aspects of program management.

* Conceptual Function:

This is a rather elusive, difficult to evaluate, yet important element that relates to the values, vision, philosophy, and professionalism of a camp. Camp Adventure [TM] addresses the conceptual function by striving to produce fun, joyful, exhilarating, and treasured experiences. Training staff members in this conceptual area is the best way to lay the foundation for evaluation. Sharing in detail your camp's vision will help all camp staff to know, understand, believe in, and exhibit behaviors that are in support of the total camp program and the mission statement and values of the agency.

Through evaluation of each of these functions, any camp can improve and adjust to changing needs. Figure 1 details more specific elements that can be addressed in the evaluation process for each of the three functions.

Evaluation Tools

Evaluation tools are the physical instruments used to collect information to answer questions about an element of camp. Those tools might be people (verbally asking questions) or paper (evaluation forms). Staff can ask campers questions throughout a program for continual feedback at each stage of an activity. End of season phone calls to parents and staff sharing sessions are also good verbal tools. When using a written evaluation, the format, aesthetics, layout, and design are all important considerations. An attractive, user-friendly form is more likely to be completed than one that appears busy and crowded. The form must also suit its audience (an evaluation form given to campers should be different from a form given to parents).

Whether using a verbal or a written evaluation, devise the questions to address all three functional areas. Also consider ways to share the results and to reward where applicable. For example, Camp Adventure [TM] operates from a perspective of wanting to "catch staff doing something right," therefore, its written evaluation form has a space for parents to identify staff members who made a special impact on their child. That staff member's name is then published in the camp newsletter, and the executive director sends the person a congratulatory note.

Sources of Information

Evaluations can be conducted and progress measured from a number of different perspectives. The four areas of camp: administration, facilities and grounds, program, and staff are each evaluated from all viewpoints in terms of camp vision, achievement of goals, developmental impact, and participant satisfaction. Customers evaluate staff, organization, and program. Staff evaluates organization and program, and conducts inter-functional-, peer-, and self-evaluations. Administrators and board members also evaluate staff and program. Remember that each evaluation procedure considers the human relations, technical, and conceptual aspects.

Benefits of Evaluation

The data collected from formal (and some informal) evaluations provides information that benefits many aspects of camp. Evaluation:

* Supports success in meeting needs/goals.

Objective data provides written "proof" that a camp is contemporary in the provision of programs, meeting the needs of campers within the current social climate, and on-track with organizational vision and goals. (This information is fodder for promotional materials and recruitment efforts.)

* Justifies funding/facilities.

In a day and age of accountability, statistics and anecdotes that can be quoted from customers and other stakeholders provide invaluable justification for additional funding, new facilities, or to simply remain in operation.

* Informs governing bodies.

It is important to keep governing bodies and organizational administrators apprised of camp processes. A formal evaluation system can provide needed information to keep those individuals well informed.

* Creates positive public relations.

An organization's sincere interest in obtaining and using input from parents, campers, the community, staff members, and others impacted by camp programs can generate positive public relations.

* Provides a forum for discussion.

The evaluation process opens the lines of communication between individuals who might not otherwise give and receive feedback. It offers an opportunity for dialogue about both problems and successes.

* Builds morale.

Having a system for meaningfully sharing problems, feelings, and thoughts with others in the organization is a constructive process that can build staff morale. To be of benefit, solicited information must be taken into consideration in future decision-making by those seeking the input.

*Enables preventative management/early problem-solving.

By soliciting input from parents, staff, and others, it is possible to identify and solve problems early, and to circumvent potential problems.

* Enables staff reward/recognition.

Results from evaluations may be used as a basis for recognizing and rewarding outstanding performance by staff members. The maintenance of a quality program requires on-going recognition of jobs well done.

* Provides information for future planning/marketing efforts.

Evaluation results can be used as a tool to better understand parent, camper, and community concerns, needs and desires. This information can be used when planning and marketing future programs.

* Determines cost effectiveness.

Some evaluations can be used to review and analyze costs associated with programs and facilities. For example, by calculating the break-even point versus cost-per-camper, the evaluator can determine a program's cost effectiveness.

* Establishes need for new direction.

Effective evaluation systems may produce results that indicate the need for a new direction by an organization. Needs for new or additional staff training, new programs, new facilities, and other areas may also emerge through the evaluation process.

Summary

Effective design and use of a rigorous evaluation system, though time-consuming and intense, provides a number of valuable benefits to camp. It may enable the organization to prevent problems, manage problems more effectively, and set a course of action for the future that is in tune with the feelings, thoughts, and needs of staff, campers, parents, and society. To gain the most from an evaluation system, camps should address at least these functions of evaluation: human relations, technical, and conceptual. Feedback can be solicited from a variety of individuals who have different connections to the camp program. A sincere attempt to gather information and respond to concerns will strengthen any camp and allow camp professionals to verify what is so often intuitively felt.

Evaluation Functions(*)

STAFF/PROGRAM PARENTS/CAMPERS

Human Relations Function

adjustment, burnout               caring
teamwork                          child inclusiveness
caring                            problem solving skills
communication                     making friends
sharing loads                     development of maturity
problem solving                   development of trust
handling campers                  fair treatment

(discipline, nurturing) MWR relationship

Technical Function

conduct of program                skill development
program leadership                learning new songs/activities
program safety                    session length
use of equipment, materials       excitement/boredom
conduct of staff meetings         leadership development
staff:camper ratio                safety
organizational structure          cost of program
housing arrangements              public relations

evaluation process marketing travel arrangements

Conceptual Function

share Camp Adventure vision       understand Camp Adventure
Camp Adventure philosophy         philosophy
(education)                       understand values of Camp Adventure
professionalism                   understand Camp Adventure vision

attitude

* University of Northern Iowa CAMP ADVENTURE [TM] Research and Evaluation Division Figure 1

Debra J. Jordan, Re.D., is an associate professor; Christopher R. Edginton, Ph.D., is a professor and director; and Susan R. Edginton is the program development coordinator of Camp Adventure [TM], all in the School of Health, Physical Education and Leisure Services at the University of Northern Iowa.