Assessing the Effects of Incorporating Service in Learning: The Search for a Comprehensive Process of Service Learning Evaluation.

Abstract

Service learning is rapidly becoming a popular teaching and learning tool, yet assessment of the effectiveness of the pedagogy is still developing. It is vital that we begin to systematically evaluate the extent to which service initiatives adhere to sound educational principles and advance the learning of our students. Furthermore, comprehensive assessment of service learning should address the interests of all stakeholders in the process, utilizing instruments that capture the interplay among faculty, students, community, and educational institution. Although evaluation of this pedagogy is still in its infancy, a careful and considered application of a variety of linked assessment mechanisms should provide insights into the true nature of service learning.

Introduction

Service learning, utilizing community service to provide students with real-world experiential learning, is gaining in popularity. Although certainly not a new methodology, it is currently experiencing a robust surge of implementation at colleges and universities all over the United States (Jackson, 1993). This renewed interest is due, in part, to curricular theorists and educational reformers who state that engaging students in service learning will help them realize that their developing skills and knowledge can address a multitude of social concerns (Kahne & Westheimer, 1996). In response to this position, many teachers are experimenting with the effect of service on learning, and educational institutions are beginning to integrate service requirements into their curriculums (Stukas, Snyder, & Clary, 1999).

This trend is in keeping with a call for higher education to become more involved in community service, something traditionally of low importance on the academic priority list (Clark, 1999). Increasingly, we are seeing partnerships forming between educational institutions and the communities in which they reside (Driscoll, Holland, Gelmon, & Kerrigan, 1996). Unlike the traditionally isolated academic model, these partnerships incorporate the concerns and ideas of the community in the decisions that shape the research, programs, teaching, and service of the educational institution (Clark, 1999). In a symbiotic transfer of information and growth, community resources and needs flow into the academy to ground theory, while academic resources and solutions flow back out into the community to address social issues.

Supporters of the service learning movement believe that making service an integral part of the educational experience helps students grow in personal, social, and civic ways and that this internalization of social values will ensure a vital source of volunteerism for the future (Stukas, Snyder, & Clary, 1999). In addition, it is argued that the authentic experiences of service learning reinforce students' performance abilities and build their self-esteem (Kahne & Westheimer, 1996). As interest and implementation of service learning projects have grown, however, concern has emerged about the educational integrity of the process and whether student involvement in service projects fulfills the basic mission of higher education institutions (Markus, Howard, & King, 1993).

Some critics wonder whether service learning projects can possibly have the dramatic effects described by its supporters. Others argue that service learning may, in fact, weaken curriculum and that time would be more wisely utilized in more traditional academic efforts (Gray, Ondaatje, Fricker, & Geschwind, 2000). Service learning requires a significant investment of time and resources, and detractors question the value of these costs and the extra workload for staff and faculty (Markus, Howard, & King, 1993). Although many of these issues may be valid, much more attention has been paid to implementing service-learning programs than to identifying exactly what they intend to accomplish and measuring their effectiveness. With resources increasingly being devoted to service learning, there comes a growing need to assess and document the strengths of the pedagogy.

This is an issue that confronts the University of Southern Maine's Lewiston-Auburn College. Service learning is a stated part of the mission of the institution and an intensive thread running throughout the undergraduate curriculums. The campus was created just over a decade ago, due to the efforts of area businesses and legislators who fought to have a public baccalaureate and graduate degree granting institution in their community. As a result, we have extremely strong ties to the area. However, despite our commitment to serve the needs of the community, we must ensure that we are adhering to sound educational principles and advancing the learning of our students. With these concerns in mind, we have spent the last few years searching for evaluation and assessment models that have proven useful for others involved in service learning. This discussion will describe the results of the search.

Why Do We Need to Assess Service Learning?

Obviously, it is always wise for educators to reflect on whether our teaching is achieving our goals. Furthermore, higher education is feeling increasing pressure to develop assessment techniques that truly measure the quality and cost effectiveness of the education we are providing. Some critics argue that higher education costs too much and delivers too little to the tax and tuition paying public. Many stakeholders (students, parents, business, government) are insisting that we supply a more valuable education experience and product (McDaniel, 1994). As academia faces stronger demands for enhanced learning outcomes, accountability has become a major concern (Coppola, 1999). If we are to continue to earn respect and patronage, we must show that the education we are providing merits this support.

These concerns have led to a new focus on teaching and learning (Hendley, 2000). This is especially true with a relatively unknown pedagogy such as service learning. At the moment, assessment of the effectiveness of this method is in an early developmental stage (Wang, Greathouse, & Falcinelle, 1998). With little evidence that service enhances student learning, teaching, academic institutions, or the communities served, objective arguments for maintaining and enlarging service learning programs are necessarily weakened. Data must be collected on the outcomes of using this tool to complement the wealth of publications on its implementation (Corbett & Kendall, 1999). Service learning will only be widely valued when it can be shown that there are direct benefits from these efforts.

Despite the notable growth in its popularity, relatively few educators, especially those on a tenure track, currently use service learning (Gray, Ondaatje, Fricker, & Geschwind, 2000). Faculty need to be convinced if they are to take on the new role of networking with the community, to allow the needs of the world outside academia to help shape the projects of the classroom. Likewise, administrators are unlikely to offer institutional support until they have data indicating positive impact. However, they must be persuaded if they are to use their power to direct resources toward these efforts and to modify reward structures to recognize this form of applied scholarship (Clark, 1999). Solid evaluation data showing how service learning effectively links theory and practice can go a long way toward building this necessary support.

Considerations in Assessment

Preliminary research indicates that effective service learning programs include monitoring and evaluating whether projects are meeting service and learning goals (Burns, 1998) and that exemplary programs integrate evaluation throughout the entire process (Shumer, 1997). This is particularly telling considering the participatory nature of the process and its potential effects on many parties. Ideally, a comprehensive assessment of service learning would consider the needs of all stakeholders--faculty, students, community, and educational institution (Jackson, 1993). It is with this in mind that we look at a breakdown of assessment's role in a variety of areas.

Perhaps the first perspective to consider is program and curricular needs. To be proactive educators, we must continually review and update the content of our programs and how we teach them (Clark, 1999). Service learning assessment should not be conducted without regard for the context of the larger programmatic goals that it should serve (Coppola, 1999). Well-designed assessment mechanisms should not only evaluate current service projects, but should also provide a base for designing and implementing improved service learning undertakings. Thus, evaluation from this perspective should focus on continuous improvement and be grounded in achieving the outcome variables of the curriculum as a whole.

In the process of conducting individual classes, assessment also plays a vital role. From the start, evaluation instruments nudge us to establish clear performance and learning expectations so that we may determine whether or not these objectives have been met (Falbo, 1997). Some advocate starting a collaborative evaluation process quite early in the term so that feedback can flow back and forth between students and faculty to help shape the course to achieve performance goals (Hatcher & Bringle, 1997). This is particularly important in a service learning classroom. Students are encouraged to take ownership of their own learning, and it can sometimes be difficult to find the right balance between student autonomy and faculty guidance, support, and feedback (New Hampshire Service Learning Assessment Study group, in press). It is therefore critical that the most central themes of the course are clearly identified and that emphasis be placed on understanding these topics.

Which brings us to a specific focus on individual student's learning. Service learning has been found to be most effective when students receive training and supervision, serve more than twenty (20) hours, and discuss their experiences regularly in class, closely linking discussion to course themes (Gray, Ondaatje, Fricker, & Geschwind, 2000). However, despite this shared learning experience, faculty need to be able to make a determination of each student's performance individually. This can be particularly challenging in the case of group projects when some students appear to be benefiting from other students' work and a notable variation in learning is suspected (Weaver, 1995). In addition, the learning due to the service element of the course should be looked at in isolation from the other aspects of the course. Assessment of students' learning through their service efforts should be based upon direct products of their service work that show their mastery of essential knowledge and skills (McDaniel, 1994; New Hampshire Service Learning Assessment Study group, in press). Thus, the extent to which each student both understands the course content and has been able to apply this information in real-world service situations must be evaluated.

Service learning also has constituents outside the classroom that merit involvement in the assessment process. It is important that the community client has say in how effectively the project has met the goals of the organization. When organizations are pleased with students' efforts, they will offer the community support that is essential for the long-term success and stability of a service learning initiative. Without community support, the program is likely to flounder (Gray, Ondaatje, Fricker, & Geschwind, 2000).

The final essential element of the assessment package is the instruments utilized by the educational institution as a whole. It is important for the institution to be able to compare service learning efforts across classes and curriculums. This can be tricky as the assessment tool must be flexible enough to provide feedback for the variety of ways service may be utilized throughout the institution. However, it is critical for the institution to be able to evaluate the role service learning plays in fulfilling its community mission (Driscoll, Holland, Gelmon, & Kerrigan, 1996). The assessment process must serve the institution's need to address the cultural, political, and social demands upon it.

Review of Service Learning Assessments

Existing service learning assessments are limited and tend to evaluate only one or two facets of service learning, or are slanted toward a few of the stakeholders. At Lewiston-Auburn College, we have identified six different perspectives on the service learning experience that should be included in a comprehensive assessment package: student self-assessment, client assessment of student work, faculty assessment of students, student assessment of courses, faculty assessment of courses, and institutional assessment of service learning initiatives.

Engaging students in reflective self-assessment of their learning is an essential element of the service learning process (Bums, 1998). The many forms of this type of assessment include journaling, impact statements, personal evaluation grids, experiential research papers, narratives based on personal data and vignettes written regularly during the course, and critical reviews of work done and personal development (Hatcher & `Bringer, 1997). The key is to utilize a vehicle for students to articulate what they learned, how they learned it, how this experience relates to the course content, and how this learning may be more largely applicable.

Community produced assessments of student work may also take a variety of forms. Probably the most common of these tools are supervisor or performance evaluation instruments (Cairn & Cairn, 1999). Other measures that have been used are interviews, focus groups, and pre and post surveys of the clients (Driscoll, Holland, Gelmon, & Kerrigan, 1996). Specific variables addressed have included assessing the organization's ability to improve services, comparing students' work to other volunteer and staff efforts, and collecting ratings of individual performance characteristics such as enthusiasm (Gray, Ondaatje, Fricker, & Geschwind, 2000). Of central importance from this perspective is to ensure that the client's needs and expectations have been met in a satisfactory manner.

Faculty assessment of students should be based on educational values and closely linked to the purposes, goals, and expectations of the service project (Falbo, 1997). Although identifying learning gained specifically from the service experience may be difficult (Cairn & Cairn, 1999), there are a variety of tools available for this function. Teachers can, of course, review all the mechanisms mentioned above, conduct service site visits, administer pre and post surveys to target specific variables, or examine work products from the service experience such as documents produced for the client. The latter would be appropriate for more extensive consultation projects when students work in groups to provide a thorough report for a community organization.

Another method that can provide extensive information about student learning is the portfolio. Portfolios can be constructed from a diversity of materials: logs, journals, contracts, photo essays, client projects, integrative essays, experiential research papers, and other self-assessments (Hatcher & Bringle, 1997). Portfolios can be extremely useful for in depth analysis; however, they can be extremely time consuming if an instructor wants to do a thorough evaluation. Regardless of the documentation chosen to evaluate student learning, the assessment should be firmly grounded in the learning objectives detailed prior to the learning experience.

Although student learning is generally considered of primary importance in the service process, measures of student performance are not the only tools that should be used. Evaluations of faculty effectiveness are very useful for improving teaching at various points throughout the term. In class, students may be asked to engage in evaluation techniques such as muddiest point, minute papers, or pro and con grids to provide valuable feedback for their instructors (Hatcher & Bringle, 1997). Outside of the classroom, students may be asked to take part in interviews or focus groups to tease out the practices that are more or less effective in the service learning classroom. Faculty themselves may also be interviewed, have classroom observations done, engage in journaling, and take part in pre and post surveys in a further attempt to determine ways that they may improve the learning environments they develop (Driscoll, Holland, Gelmon, & Kerrigan, 1996).

The final set of assessment tools is the mechanisms used by entire colleges and universities to evaluate service learning programs as a whole. One method is to analyze a body of course assessment instruments or alumni surveys so that the institution can determine the effectiveness of service efforts across courses and curriculums (Driscoll, Holland, Gelmon, & Kerrigan, 1996). Another technique is to utilize profiles of institutions that are recognized as having strong service learning programs to measure strengths and weaknesses comparatively. One such instrument (Maine Campus Compact, 2000) was developed from a study of twenty-seven (27) institutions nominated by service learning experts as having exemplary programs. In this instrument, thirteen (13) elements were noted to be present at all of the successful campuses: defining a service learning mission, tying this mission to the institutional mission, gauging needs and abilities before program expansion, gaining high-level administrative support, utilizing advisory boards, staying current on service learning literature, clarifying the potential for service learning in every discipline, supporting faculty investment, encouraging multiple service learning options, earning student support, seeking community involvement, trying innovative approaches, and assessing clearly defined outcomes. By conscientiously comparing their practices to efforts across the nation, institutions can evaluate where they may best utilize their resources to continuously improve their service initiatives.

Conclusions

Clearly, meaningful assessment of service learning must involve all constituencies and get each of their reactions to what happened in the process. In our search to identify evaluation tools that would best supply this information, we found that we need to develop a package of instruments that provides a comprehensive yet flexible toolbox appropriate for all service projects at the college. Despite the unique nature of each project, the assessment package must draw out specific information from students about their individual learning, from clients about their satisfaction with service products, and from faculty about student learning. By engaging all participants in the evaluation process, we will be able to capture a relatively complete image of the success of the project. When this data collection becomes routine, the institution will be able to conduct a thorough assessment of the effects of incorporating service in learning.

On reflection, the ideal assessment mechanism appears to be creating a portfolio for each service project, one that weaves the voices of the student, the teacher, and the client into a full representation of the effort. To this end, we will be creating guidelines that ask faculty to include five elements in their service course portfolios: students' impressions of their learning, documentation of students' work for the client, client evaluation of the work done, data from a standardized service course assessment tool, and the teacher's appraisal of the project's success. In this manner, we hope to create an evaluation process that is manageable, but inclusive enough to get a robust picture of the quality of service learning in each class and across the institution.

We have discovered that comprehensive assessment of service learning is quite complex. The many stakeholders in the service process possess a diversity of needs that must be considered if the evaluation is to be truly useful for the program. Although the assessment of this pedagogy is still in its infancy, a careful and considered application of a variety of linked assessment mechanisms should provide a fairly clear picture of the nature of service learning at an institution.

Bibliography 

Bums, L. (1998). Make sure it's service learning, not just community service. Educational Digest, 64 (2), 38-41.

Cairn, R. & Cairn, S. (1999). Service learning makes the grade. Educational Leadership, 56 (6), 66-68.

Clark, P. (1999). Service-learning education in community-academic partnerships: Implications for interdisciplinary geriatric training in the health professions. Educational Gerontology, 25 (7), 641-660.

Coppola, N. (1999). Setting the discourse community: Tasks and assessment for the new technical communication service course. Technical Communication Quarterly, 8 (3), 249-268.

Corbett, J., & Kendall, A. (1999). Evaluating service learning in the communication discipline. Journalism & Mass Communication Educator, 53 (4), 66-76.

Driscoll, A., Holland, B., Gelmon, S., & Kerrigan, S. (1996). An assessment model for service-learning: Comprehensive case studies of impact on faculty, students, community, and institution. Michigan Journal of Service Learning, 3 (1), 66-71.

Falbo, M. (1997). Serving to learn: A faculty guide to service learning. Marietta, OH: Ohio Campus Compact.

Gray, M., Ondaatje, E., Fricker, R., & Geschwind, S. (2000). Assessing service learning: Results from a survey of "Learn and Serve America, Higher Education." Change, 32 (2), 30-39.

Hatcher, J. & Bringle, R. (1997). Reflection: Bridging the gap between service and learning. College Teaching, 45 (4), 153-158.

Hendley, V. (2000). 30 years of higher education. AAHE Bulletin, 52 (7), 3-8.

Jackson, F. (1993). Evaluating service learning. In Kupiec, T. (Ed.), Rethinking tradition: Integrating service with academic study on college campuses (pp. 129-135). Denver, CO: Education Commission of the States.

Kahne, J. & Westheimer, J. (1996). In service of what? The politics of service learning. Phi Delta Kappan, 77 (9), 592-599.

McDaniel, T. (1994). College classrooms of the future. College Teaching, 42 (1), 27-31.

Markus, G., Howard, J., & King, D. (1993). Integrating community service and classroom instruction enhances learning: Results from an experiment. Educational Evaluation and Policy Analysis, 15 (4), 410-419.

New Hampshire Service Learning Assessment Study Group. (1999). Improving teaching and learning in New Hampshire through effective assessment of service-learning. In Learn and Serve America (Ed.) Service-learning and assessment.' A field guide for teachers. Washington, D.C.: Corporation for National Service, 8(1), 8-21.

Shumer, R. (1997). What research tells us about designing service learning programs. NASSP Bulletin, 81 (591), 18-24.

Stukas, A., Snyder, M., & Clary, E. (1999). The effects of "mandatory volunteerism" on intentions to volunteer. Psychological Science, 10 (1), 59-64.

Wang, J., Greathouse, B., & Falcinelle, V. (1998). An empirical investigation of self-esteem enhancement in a high-school challenge service-learning program. Education, 119 (1), 99-105.

Weaver, R. (1995). Bringing an applied research project to the classroom. Journal of Applied Sociology, 12 (2), 31-48.

Tara Grey Coste, University of Southern Maine Marvin Druker, Lewiston-Auburn College

Coste is a leadership professor who frequently uses service learning to empower students with an awareness of their adaptability and effectiveness <[e-mail]>. Druker is a leadership professor and associate dean at USM's Lewiston-Auburn College who utilizes service learning to increase community awareness and involvement. E-mail: <[e-mail]>.