Impact of reflection and training on S-L outcomes.

Abstract

A study was conducted to determine the impacts of Project SHINE (Students Helping in the Naturalization of Elders) on undergraduate and graduate student participants (N = 289). The results indicated that the frequency with which students' professors linked their coursework to SHINE, the importance placed on reflection, and the quality of students' training and ongoing support positively impacted on several outcome measures. The discussion emphasizes the importance of faculty and staff providing linkages between students' coursework and their service experiences to maximize positive student outcomes.

Introduction

Project SHINE (http://www.projectshine.org) is a national, multicultural, and intergenerational service-learning initiative based at Temple University's Center for Intergenerational Learning. SHINE links college students with older immigrants and refugees seeking to learn English and navigate the complex path to U.S. citizenship. Eighteen colleges and universities in fourteen cities across the U.S. currently participate in the program. SHINE students work with immigrant elders in small groups and one-on-one, creating individualized lessons in comfortable learning environments. Providing instruction in familiar settings such as churches, temples, community centers, and senior housing allows elders who may be unable to travel to established classes to access services they might not otherwise receive. Students also assist teachers in overcrowded classes, helping immigrant elders overcome learning difficulties associated with changes in memory, vision, hearing, and mobility and keep up with the pace of instruction.

At each institution, SHINE staff develop and maintain partnerships with local community organizations; recruit, train, place, and monitor students at community sites; provide ongoing support and technical assistance and opportunities for reflection and recognition at the end of the semester. Program staff also recruit and support faculty; providing individual consultations, making class presentations, facilitating reflection, organizing meetings with community partners, and encouraging peer support by bringing new and experienced faculty together. Tutor handbooks and a program manual developed by SHINE facilitate the replication of the program across the consortium. Although most students participate in the program through a service-learning class, students are encouraged to sustain service beyond one semester through work study or as volunteers. In addition to tutoring, student leaders are increasingly involved in many aspects of program implementation.

Theoretical Framework

SHINE's service-learning model exemplifies many of the "good practices" identified by the RAND Corporation in its evaluation of Learn & Service America Higher Education programs including training, support, and intensity of service (Gray, Ondaatje, & Zakaras, 1999). SHINE provides and requires participation in a comprehensive pre-service training co-facilitated by staff and community partners prior to placement. Program staff work in partnership with faculty, staff at community-based organizations including ESL/Citizenship instructors, and student leaders working as program assistants and site coordinators to support and supervise tutors/coaches in the field. Students are required to provide a minimum of 20 hours of direct service per semester.

Goals of SHINE for student participants include improved teaching/tutoring skills; increased knowledge of U.S. history and civics; greater awareness of community needs and how to address them; increased sense of social responsibility and civic engagement; increased academic interest, engagement, and performance; and a greater appreciation of the elderly and immigrants. The development of these goals was based on a growing body of research that indicates a variety of positive impacts of service-learning on college students. For example, service-learning programs have been shown to positively impact on students' ability to work well with others (e.g., Dalton & Petrie, 1997; Driscoll, Holland, Gelmon, & Kerrigan, 1996), sense of social responsibility and civic skills (e.g., Batchelder & Root, 1994; Billig, Meyer, & Hofschire, 2004; Keen & Keen, 1998), academic learning (e.g., Fenzel & Leary, 1997; Vogelgesang & Astin, 2000), and satisfaction with college (e.g., Astin & Sax, 1998; Berson & Younkin, 1998; Gallini & Moely, 2003). Therefore, the first goal of this study was to document the impacts of Project SHINE on participating students.

However, we were also interested in investigating what aspects of the service-learning experience and service-learning course were associated with these impacts. Service-learning research in both K-12 and higher education settings (e.g., Billig, 2004; Eyler & Giles, 1999) underscores the importance of incorporating high quality components in the service-learning experience to maximize outcomes. For example, the development of an intentional relationship between academic subject matter and service through frequent opportunities for reflection on the connection between service and academic content is critical in achieving positive student outcomes (Eyler & Giles, 1999). Overall, students who participate in well-integrated service-learning opportunities of greater duration develop better problem solving skills and a more complex understanding of social issues, abilities that are integral to the development of a lifelong ethic of service (Astin & Sax 1998; Eyler, Giles, & Braxton, 1997). However, many courses that incorporate a service component may do so only peripherally as a course option without a well-defined connection between academic subject matter and service. Opportunities for structured critical reflection and a systematic focus on ethical inquiry and civic engagement are also missing in many service-learning courses.

Service-learning researchers are beginning to focus on the importance of program characteristics such as the inclusion of structured reflection activities in determining whether service-learning positively impacts participants. For example, in a study of high school service-learning programs, Klute and Billig (2002) found that students experienced the most significant outcomes when they had direct interaction with community members and when service-learning activities were linked to academic standards. Similarly, research on service-learning at the college level indicates that factors such as placement quality (Eyler & Giles, 1999), quality and quantity of reflection activities (Gray et al., 1998), and receiving feedback from professors or other stakeholders (Greene, 1996; Subramony, 2000) significantly impacted student outcomes. To expand on these findings, our second goal in this study was to examine the influences of several Project SHINE course and program characteristics on participating students' academic and civic outcomes. These characteristics included the frequency with which students' professors linked their coursework to their Project SHINE experience, the importance that their professors placed on reflection activities, and the quality of students' Project SHINE training and ongoing support.

Method

The Sample

Students. All students that participated in Project SHINE during spring semester, 2004 were asked to take the pre- and post-surveys. Out of 592 total participants, 289 (49%) completed both surveys, and their data were used in the analyses. About 80% of the respondents attended four-year institutions. Seventy-nine percent of the students were undergraduates, and their median age was 22. About 70% were female, 25% were male, and 5% did not identify their gender. In terms of race, 27% were Asian, 27% were Hispanic/Latino, 22% were White, 7% were multi-ethnic/multi-racial, 13% were from other racial groups, and 4% did not respond to this question.

Faculty. All faculty that participated in Project SHINE during spring semester, 2004 were asked to complete a survey. Of 74 total participants, 35 (47%) completed the survey.

Instrumentation 

Students completed the pre-surveys at the beginning of the spring 2004 semester, prior to their Project SHINE training. The post-surveys were completed at the end of the spring 2004 semester after students' service had ended. The surveys were administered in an online format. The student survey measures are discussed below. Faculty completed an online survey at the end of the spring 2004 semester assessing their implementation of Project SHINE in their courses.

Data Analysis

Student survey. Factor analyses were performed to create scales and reduce the number of items for analysis. Those items that formed factors were averaged to create scales. The scales assessed the following outcomes (Cronbach's alphas for the pre- and post-survey scales are listed in parentheses):

1) School engagement (alpha = .82 (pre), .86 (post)]: This measure contained five items assessing students' engagement in school. These items were assessed on a 5-point scale from 1 = never/almost never to 5=always/almost always.

2) Civic knowledge (alpha = .89, .89): Civic knowledge was assessed with four items that measured students' knowledge of U.S. history and government on a 4-point scale from 1 = none to 4 = a lot.

3) Knowledge of U.S. immigration and naturalization processes (alpha = .89, .90): This measure assessed students' awareness of how U.S. immigration and naturalization processes work. It contained two items which were measured on a 4-point scale from 1 = none to 4 = a lot.

4) Civic skills (alpha = .78, .83): This six item measure assessed students' abilities to address civic issues. These items were measured on a 4-point agreement scale.

5) Civic dispositions (alpha = .67, .83): This measure contained four items assessing students' attitudes toward civic engagement. These items were assessed on a 4-point agreement scale.

6) Issue involvement (alpha = .73, .80): This measure was assessed with four items that asked about students' level of involvement with civic issues. Items were measured on a 4-point agreement scale.

7) Attitudes toward the elderly (alpha = .85, .85): This measure contained five items assessing students' attitudes toward working with the elderly. These items were assessed on a 4-point agreement scale.

8) Attitudes toward immigrants (alpha = .86, .86): Attitudes toward working with immigrants were assessed with five items measured on a 4-point agreement scale.

9) Civic intentions (alpha = .78, .78): This five item measure assessed students' future plans for civic involvement. These items were assessed on a 4-point scale from 1 = very unlikely to 4 = very likely.

10) Comfort with tutoring and teaching English (alpha = .87, .87): This four item measure assessed students' comfort level tutoring others and teaching English. Items were assessed on a 4-point scale from 1 = very uncomfortable to 4 = very comfortable.

11) Self-efficacy (alpha = .86, .85): This measure contained five items assessing students' beliefs that they had the ability to deal with a variety of situations. These items were measured on a 4-point agreement scale.

12) Voting intentions (alpha = .93, .91): Voting intentions were assessed with three items concerning how likely students were to vote in national, state, and local elections. These items were measured on a 4-point scale from 1 = very unlikely to 4 = very likely.

The post-survey contained additional measures assessing students' SHINE experience: Importance of reflection activities (alpha = .81): Three items assessed the importance placed on reflection activities in students' Project SHINE course. These items were assessed on a 4-point scale from 1 = not very/not at all to 4 = very important. Evaluation of program areas (alpha = .93): This eight item measure assessed students' perceptions of the quality of the pre-service training and ongoing support they received while participating in SHINE. Items were measured on a 4-point agreement scale.

Paired samples t tests were conducted to determine whether there were changes in the outcome measures over time. Independent samples t tests also were conducted to determine whether there were group differences for the key variables of the professor linking coursework to students' Project SHINE experience, the importance placed on reflection activities, and the impact of program training and support on student outcomes. To facilitate these analyses, students' responses concerning how frequently their professor linked their coursework to Project SHINE were collapsed into two categories: rarely/never/sometimes (n = 128) and often/very often (n = 103). For reflection activities, students were divided into two categories by placing those respondents who scored below the median (Mdn = 3) of this scale in the "low reflection" group (n = 103), and those who scored at or above the median in the "high reflection" group (n = 123). For evaluations of training and support, students' responses were divided into two categories by placing those respondents who scored below the median (Mdn = 3.25) in the "low ratings of training/support" group (n = 117), and those who scored at or above the median in the "high ratings of training/support" group (n = 122). Faculty survey. Descriptive statistics were computed for the faculty survey items. See issue website <http://rapidintellect.com/AEQweb/spr2005.htm>

Results

Student Survey

Results from paired samples t test analyses indicated several changes in the outcome measures over time. From the beginning to the end of the semester, students reported higher civic knowledge [t(266) = -7.14, p < .001] and knowledge of U.S. immigration and naturalization processes [t(264) = -8.91, p < .001], and higher comfort with teaching and tutoring English [t(266) = -2.44, p < .05]. However, some unintended effects were found as well. Over time students reported lower school engagement [t(262) = 4.03, p < .001], decreased civic dispositions [t(257) = 2.21, p < .05], and more negative attitudes toward immigrants [t(260) = 2.79, p < .01]. Descriptive statistics for these findings are presented in Table 1.

The results from the independent samples t test analyses indicated several significant differences between groups (see Tables 2, 3, and 4 for descriptive statistics). Those students whose professors linked their coursework to Project SHINE "often" or "very often" reported higher levels of school engagement than did those whose professors linked their coursework to Project SHINE "rarely/never" or "sometimes," t(225) = -2.24, p < .05. In terms of reflection activities, students in the "high reflection" group reported more positive attitudes toward immigrants [t(221) = -1.95, p < .05] and greater intentions to vote [t(223) = -2.01, p < .05] than did those in the "low reflection" group.

The quality of students' Project SHINE training and support was strongly related to student outcomes. Students in the "high ratings" group scored significantly higher than did students in the "low ratings" group on measures of school engagement [t(231) = -2.16, p < .05], civic knowledge [t(235) = -2.99, p < .01], knowledge of U.S. immigration and naturalization processes [t(234) = -1.95, p < .05], civic skills [t(234) = -3.82, p < .01], civic dispositions [t(233) = -5.40, p < .01], attitudes toward the elderly [t(233) = -5.62, p < .01] and immigrants [t(232) = -6.46, p < .01], civic intentions [t(231) = -3.99, p < .01], comfort teaching/tutoring [t(235) = -6.10, p < .01], self-efficacy [t(235) = -3.02, p < .01], and voting intentions [t(234) = -4.01, p < .01].

Faculty Survey

Almost half the faculty respondents (46%) indicated that they linked their coursework to students' Project SHINE experience "very often" or "often." About half (51%) indicated that they used reflection activities. These respondents were asked to identify the components of these activities. Most typically, faculty had students discuss their feelings about their Project SHINE experience (41%) or summarize the experience (41%). Close to one-third (31%) discussed issues regarding English language acquisition. About one-fourth discussed the culture from which the elders came (26%) or issues around teaching and use of different types of instruction (26%). Less than one-fourth discussed issues around civic responsibility/engagement (23%), issues around community involvement (20%), comparisons of cultures (20%), discussions of adaptation to American culture (20%), discussion of immigration issues (10%), or issues around aging (11%).

Discussion

Our findings were consistent with past research that indicates that faculty play a central role in the success of service-learning pedagogy, and that quality reflection is a key component of that success (e.g., Eyler & Giles, 1999; Hatcher, Bringle, & Muthiah, 2004). Students whose professors frequently connected their coursework to SHINE reported more positive attitudes toward immigrants and greater intentions to vote than those whose faculty did not. However, a little less than half of the faculty respondents reported linking their coursework to students' service experience "often" or "very often" and only half indicated that they used reflection activities. These findings may be due to variations in the implementation of SHINE across classrooms and campuses.

Incorporating SHINE as one of several options within a course including other service placements or in lieu of a paper or writing assignment, may make it more difficult for faculty to develop assignments and find themes that connect service placements in class discussions. The size of the class and the relative number of students opting for the program in comparison with other opportunities may also contribute to this difficulty in developing SHINE-specific activities. Further, those faculty that reported making those connections tended to select reflection activities that invited students to describe their service experience and their feelings about that experience. However, these activities were less likely to probe more deeply and examine issues around aging, immigration, cultural adaptation, cross cultural comparison, and community involvement.

The unintended effects of service-learning through SHINE such as lower school engagement, decreased civic dispositions, and more negative attitudes toward immigrants reported by students over time also underscore the importance of quality reflection and suggest that students may not be making those connections on their own. Students need to be provided with a framework from which to make sense of their experience within the context of the community they were serving. It is possible that students developed a more realistic understanding of the complexity of community and social problems and the need for systemic as well as individual solutions, and thus became more critical of the possibility of integrating immigrants into American life. The depth or absence of reflection activities may have also contributed to this outcome.

As established programs such as SHINE move beyond trailblazing faculty who are often at the cutting edge of new teaching/learning pedagogies to engage faculty who are new to the field, additional strategies for training and support are needed. Some options for this might include collaboration with professional development organizations such as Campus Compact, providing faculty or departmental stipends, or creating sample syllabi, discussion guides, and reflection activities to facilitate faculty integration. Hatcher et al.'s (2004) research provides a framework for creating reflection activities: They found that higher quality service-learning courses contained reflection activities that were "structured with clear guidelines and directions, were a regular part of the course, and clarified personal values" (p. 38). In addition, it would be particularly useful to create exercises that could integrate participants' experiences with the service experiences of other students. However, the success of such strategies might be hindered by obstacles such as limited faculty time, and the fact that service-learning is not rewarded in the promotion and tenure process at many institutions. These results also indicate that students need support not only in the classroom, but also in the community in order to maximize the positive outcomes of service-learning experiences. In fact, high quality training and support maximized positive student outcomes in our study. It is possible that these results were found because high quality training and support reinforce classroom learning by creating conditions for a positive service experience and/or, in the absence of meaningful reflection, provide an opportunity for students to analyze their SHINE experience as they respond to community needs and troubleshoot problems that arise. Further research is needed to better define quality, to determine more specifically which program areas have the greatest impact, and to identify how they promote learning.

Overall, the components of the SHINE experience (i.e., explicitly teaching citizenship/civic education as well as language and literacy skills, and interacting with individuals from diverse cultures) are well suited to promoting civic outcomes among participating students. Nevertheless, these results do not occur without the efforts of faculty and staff to create quality service and learning opportunities. Students need support in the classroom and the community to discover connections between theory and practice in authentic settings. It is critical that faculty and staff provide a structure for making these connections so that student outcomes may be maximized.

References

Astin, A.W., & Sax, L.J. (1998). How Undergraduates are Affected by Service Participation. Journal of College Student Development, 39(3), 251-263.

Batchelder, T.H., & Root, S. (1994). Effects of an Undergraduate Program to Integrate Academic Learning and Service: Cognitive, Prosocial Cognitive, and Identity Outcomes. Journal of Adolescence, 17, 341-355.

Berson, J.S., & Younkin, W.F. (1998). Doing Well by Doing Good: A Study of the Effects of a Service-Learning Experience on Student Success. Paper presented at the American Society of Higher Education, Miami, FL.

Billig, S. H. (2000, May). Research on K-12 School-Based Service-Learning: The Evidence Builds. Phi Delta Kappan, (81)9, 658-664.

Billig, S.H. (2004). Heads, Hearts, and Hands: The Research on K-12 Service-Learning. Growing to Greatness, State Farm.

Billig, S.H., Meyer, S., & Hofschire, L. (2004). National Civic Engagement Results from Three Campuses. RMC Research Corporation.

Dalton, J.C., & Petrie, A.M. (1997). The Power of Peer Culture. Educational Record, 78(3-4), 18-24.

Driscoll, A., Holland, B., Gelmon, S., & Kerrigan, S. (1996). An Assessment Model for Service-Learning: Comprehensive Case Studies of Impact on Faculty, Students, Community, and Institutions. Michigan Journal of Community Service Learning, 3, 66-71.

Eyler, J.S., & Giles, D.E., Jr. (1999). Where's the Learning in Service-Learning? San Francisco, CA: Jossey-Bass, Inc.

Eyler, J.S., Giles, D.E., Jr., & Braxton, J. (1997). The Impact of Service-Learning on College Students. Michigan Journal of Community Service Learning, 4,5-15.

Fenzel, L.M., & Leary, T.P. (1997). Evaluating Outcomes of Service-Learning Courses at a Parochial College. Paper presented at the Annual Meeting of the American Educational Research Association, Chicago, IL.

Gallini, S.M., & Moely, B.E. (2003). Service-Learning and Engagement, Academic Challenge and Retention. Michigan Journal of Community Service-Learning, 10.

Gray, M., Ondaatje, E. & Zakaras, L. (1999). Combining service learning in higher education: summary report. Rand Corporation.

Greene, D.P. (1996). Moral Reasoning, Student Development, Reciprocity and Quality of Life in a Service Learning Experiment. Unpublished Doctoral Dissertation, Colorado State University.

Hatcher, J.A., Bringle, R.G., & Muthiah, R. (2004). Designing Effective Reflection: What Matters to Service-Learning? Michigan Journal of Community Service Learning, 11, 38-46.

Keen, C., & Keen, J. (1998). Bonner Student Impact Survey. Bonner Foundation.

Klute, M. M., & Billig, S. H. (2002). The Impact of Service-Learning on MEAP: A Large-Scale Study of Michigan Learn and Serve Grantees. Denver, CO: RMC Research.

Subramony, M.V. (2000). The Relationship of Performance Feedback and Service-Learning. Michigan Journal of Community Service Learning, 7, 46-53.

Vogelgesang, L.J., & Astin, A.W. (2000). Comparing the Effects of Service-Learning and Community Service. Michigan Journal of Community Service Learning, 7, 25-34.

Tina Kluetmeier, Temple University, PA

Daryl Gordon, Temple University, PA

Linda Hofschire, RMC Research, PA

Tina Kluetmeier (M.S.Ed., 1995) is the National Director of Project SHINE; Daryl Gordon (Ph.D., 2004) is the Assistant National Director of Project SHINE and an adjunct professor; Linda Hofschire (Ph.D., 2003) is a Research Associate at RMC Research Corporation.