Exploratory study to measure the validity of the SELEB scale.

ABSTRACT

The SELEB scale (SErvice LEarning Benefit) was developed as a quantitative tool to assess the benefits of service learning. This research examines the ability of the SELEB Scale to discriminate between the effects of class projects that contain a service-learning component and class projects that do not. Two undergraduate marketing classes participated in the study. One class completed a class project with a for-profit organization while the other completed a service learning class project with a local nonprofit organization. The results indicate that the SELEB scale exhibits discriminant and convergent validity and captures differences in the experiences of the two groups.

1. INTRODUCTION

A significant body of research investigates the effects of service learning, and according to Campus Compact and reported by the National Service-Learning Clearinghouse, 6,272 service-learning classes were taught in the 1999-2000 academic year with over 700,000 students participating in those classes ("Service Learning is ..."). The SELEB scale (Toncar, Reid, Burns and Anderson 2003) has been developed to measure the benefits of service learning. However, the scale has yet to be rigorously validated. The purpose of this study is to begin to examine the validity of the scale.

2. SERVICE LEARNING

Service learning at the college and university level can be defined as a credit-bearing experience in which students participate in an organized service activity that meets community needs and then reflect on the service activity in such a way as to gain further understanding of course content, a broader appreciation of the discipline, and an enhanced sense of civic responsibility (Bringle and Hatcher 1996). More concisely, service learning is a process whereby students participate in course-relevant community service to enhance their learning experiences (Petkus 2000). Service-learning projects are seen as useful methods to help students to integrate their classroom activities with real-world experiences. Further, service learning allows students to make a real difference in their community. Though definitive statistics are not available, it is illustrative to note that over 900 two-and four-year colleges and universities, in 46 states and the District of Columbia are members of Campus Compact, a national organization that promotes community and service-learning initiatives.

Proponents of service learning provide a long list of benefits that can be derived by including a service-learning component into the curriculum. Service learning has been linked to increased grade point averages, retention, degree completion, civic responsibility, and life skills (interpersonal, leadership and confidence, among others), (Astin and Sax 1998). Students report greater satisfaction with courses that include a service-learning component, along with greater satisfaction with the instructor, reading assignments, and grades (Berson and Younkin 1998).

Previous research has linked service learning to increased personal development, social responsibility, interpersonal skills, tolerance, learning, application of learning (Eyler and Giles 1999), as well as intellectual and emotional growth (Blackwell 1996). Eyler, Giles, Stenson, and Gray (2001)identified the following personal outcomes from service learning including: (1) personal efficacy, personal identity, spiritual growth, and moral development and (2) interpersonal development, ability to work well with others, leadership skills, and communication skills.

There seems to be a consensus in the literature that service learning can provide substantial benefits to students. But all service-learning projects are not created equal, and all service-learning experiences are, to some degree, unique. The uniqueness has made it difficult to clearly identify what aspects of a service-learning experience make it "good." Eyler and Giles (1999) and others recognize the importance of the quality of service-learning programs. The placement quality, the strength of linkage between academic and service components, and the degree of written and oral reflection influence the effectiveness of service learning. This observation is echoed by Ikeda (2000) who reports that structured, intentional reflection is a key component of the service-learning process.

Batchelder and Root (1994) did compare students in a service-learning class with students in a traditional class and examined the effects of the service learning experiences on the students. They (1994) did find that the service-learning students had increased their pro-social decision-making, pro-social reasoning, and occupational identity processing skills. Berson and Younkin (1998) also evaluated students in classes with a service-learning component compared to students in class without a service-learning component. They (1998) found the service-learning students' grades in the courses were slightly higher than the students in classes without service-learning projects. The students in the classes with service-learning projects also reported a significantly greater level of satisfaction with the course, the instructor, the reading assignments, and the grading system. Litke (2002) looked at the differences between students who performed well in her service-learning class versus students who received a lower grade and found that both types of students experience positive effects from the class project.

The increased popularity of service learning is reflected in the relatively recent inclusion of service learning in business school curricula. Service learning has been effectively used in courses in accounting (Gujarathi and McQuade 2002), statistics (Root and Thorme 2001) and marketing (Easterling and Rudell 1997; Ekrich and Voorhees 2002; Petkus 2000). Notably, marketing courses are thought to be among the most appropriate among the business disciplines for experiential learning activities, such as service learning (Easterling and Rudell 1997). Petkus (2000) provides a useful blueprint for the implementation of service learning in a variety of marketing courses, suggesting that a service-learning component is appropriate for courses ranging from basic marketing to consumer behavior to marketing research.

Recently, the assessment of the benefits of service learning has become an important research issue. The SELEB scale, a 20-item instrument that measures the benefits of service learning as perceived by students, was introduced by Toncar, Reid, Burns and Anderson (2003). Subsequent research has utilized the SELEB scale to capture differences in the benefits of service learning that result from easier versus more difficult projects (Reid, Toncar and Anderson 2004), easier versus more difficult classes and differences in projects developed solely by instructors versus projects in which students played a role in the development of the project (Toncar, Reid and Anderson 2004). Based on the results reported by these authors, the SELEB scale appears to be an effective instrument for assessing the benefits of service-learning experiences.

However, at least two issues remain unresolved with regard to the validity of the SELEB scale. First, no attempt was made in the research reported above to assess the equivalence of the students in the classes that were compared. It is unclear whether the reported results were obtained as a result of the service-learning experience or because of pre-measurement differences among the groups. Therefore, the differences captured by the SELEB scale could have been the result of a priori differences among the student groups.

Second, and perhaps more problematic, it is not clear that the SELEB scale is measuring the benefits of service learning. While it is clearly measuring something, the authors have not established that the scale assesses the benefits of service-learning projects, to the exclusion of other class projects. Many of the reported benefits of service-learning projects may also be obtained using a class project that does not entail service learning. Therefore, it may be a misnomer to refer to the scale as a service-learning benefit measurement instrument. Instead, it may be more appropriate to refer to it as a class-project benefit measurement instrument.

The purpose of this research is to address these two issues. We consider two classes, both of which completed a class project as a course requirement. One class engaged in a service-learning project with a local nonprofit organization, while the other worked with a for-profit broadcasting company. We compare the classes prior to the project experience, to establish equivalence among the two student groups. We then use the SELEB scale to assess the benefits of both projects, to investigate whether the scale discriminates between the experiences of the two groups. Finally, we assess the benefits of the projects using a series of Likert-type scales and compare these responses to those obtained using the SELEB scale.

3. RESEARCH QUESTIONS

Previous research suggests that service-learning initiatives result in a variety of benefits to students. Implicit in these studies is that service-learning projects are more beneficial than class projects that do not have a service-learning component. This differential benefit should be captured by the SELEB scale. With this in mind, we are guided by the following premise:

Students in a class that engaged in a service-learning project will report greater benefits from the project, as measured by the SELEB scale, than students in a class engaged in a project with a for-profit organization.

4. METHODOLOGY

Two classes in Summer 2004 had class projects associated with them. In one class, the students were asked to develop an internal marketing plan for a nonprofit organization and were divided into teams with each team presenting their marketing plan to the nonprofit organization. In the other class, all of the students were asked to work with a local television station and administer a survey for them about economic conditions in the local area, analyze the data, and present the results to the organization.

Prior to being introduced to the projects in the class, both groups were asked to rate the importance of each of the 20 items on the SELEB scale using a seven-point scale with seven being very important and one being not at all important. At the end of the class, both groups were again asked to complete the SELEB scale. However, this time the instructions asked students to report how well the project had provided them with each item again on a seven-point scale. The students were then asked to rank the importance of each of the four dimensions of service learning represented by the SELEB scale (Toncar, Reid, Burns and Anderson 2003). They were also asked to respond to a series of statements about the course and the project again using seven-point scales. In the class that worked with a nonprofit organization, 26 students completed both the pretest and post-test questionnaires. In the class that worked with the TV station, 23 students completed both questionnaires.

5. RESULTS

5.1 Establishing Equivalency

To establish the equivalency of the two student groups, we first examined the demographic characteristics of the two classes to see how closely the two groups resembled each other. As presented in Table 1, the two classes were quite similar regarding the characteristics under consideration. Chi square tests revealed no significant differences among the groups on any of the measured characteristics.

We next examined how each group ranked the four dimensions of the SELEB scale on the pretest questionnaire, to learn if the groups differed in their perception of the relative importance of each dimension. The Spearman rank correlation test suggests that there was no significant difference in the responses of the two groups. The results of the Spearman correlation test appear in Table 2.

To summarize the examination of the equivalency of the two groups, the results suggest that the two classes are equivalent. Demographically the two are quite similar. The same can be said with respect to their rankings of the importance of the four dimensions of service learning.

5.2 Perceived Benefits of the Class Projects

To assess the effects of the class projects, we used the difference between the pretest and posttest SELEB scores. Table 3 presents t-test analyses between the pretest and posttest ratings of each SELEB scale item. To calculate the differences, the pretest measure was subtracted from the posttest measure, and then a t-test analysis was calculated for each difference.

With the exception of one item (community involvement), the classes differed significantly on all of the SELEB scale items. In every case, the mean pretest-posttest difference score for students who worked with the nonprofit organization was significantly greater than the mean difference score of the students who worked with the for-profit organization. This provides support for the proposition that the SELEB scale discriminates between the effects of different types of class projects. These results suggest that the students in the service-learning class perceived greater benefits to their project than did students in the for-profit project class.

We turn now to an examination of student responses to a series of questions in which students were asked to rate various aspects of their class and project. We would expect these results to be generally consistent with the results obtained using the SELEB scale. Table 4 presents the results of t-test analyses comparing the responses of the students in both classes.

Students in the two classes differed significantly with regard to two of the six statements. Students in the nonprofit project class believed more strongly than the for-profit project class that the class project made a meaningful contribution to the organization and that they enjoyed the class more because of the project.

A close inspection of Table 4 is somewhat intriguing. Both classes agreed that the class projects were useful in a pragmatic sense. Both provided an additional line on students' resumes, and both were effective at illustrating different concepts discussed in class. In this regard, the service-learning project appears to be no more effective than the for-profit project. However, students in the service-learning project course seemed to find the project more personally rewarding than the for profit class project group. This is clearly demonstrated by the significant differences reported above, and supported by the marginally significant differences in student responses to the items regarding the preference of the project over lectures and tests (p=.067), and whether students would recommend the class to a friend (p=.068). Taken together, this represents additional evidence of the benefit of service-learning projects over class projects with for-profit organizations.

6. DISCUSSION

The purpose of this research was to provide an initial test of the validity of the SELEB scale introduced by Toncar, Reid, Burns and Anderson (2003). If the SELEB scale measures what it purports to measure, then we should observe different results when the scale is administered to students who completed a service-learning class project, compared to students who complete a class project for a for-profit organization. Our results suggest this is the case. Overall, students who completed a service-learning project in their class reported greater satisfaction with the project and with the class. These results were reflected not only in the SELEB scale results but also in additional measures completed by both student groups.

To our knowledge, this is the first attempt to test the validity of the SELEB scale. Although we provide some support for the validity of the scale, much remains to be done. This investigation was narrow in scope, addressing the discriminant and convergent validity of the SELEB scale. Our results suggest that the SELEB scale effectively discriminates between different types of class projects and converges with another measure of a similar construct.

While the two classes were similar demographically and ranked the four dimensions of benefits to service learning similarly, differences did exist between them in their pretest-posttest scores. Given the differences, the students did seem to make the connection between the service-learning project in the class with the nonprofit project and the benefits that they derived in participating in that project. And, importantly, the SELEB scale captured this information. As with results of other studies, the students in the nonprofit project class seemed to rate the class somewhat higher than the students who worked on the project involving a company.

In summary, this research offers some evidence in support of the validity of the SELEB scale. It has face validity; it seems to be measuring what it purports to measure, and it exhibits some degree of both convergent and discriminant validity as well. Future research should continue to examine the validity of the SELEB scale.

7. LIMITATIONS

This study is certainly not the last word on the SELEB scale. This research involved a small sample of students, enrolled in different classes using different class projects taught by different faculty members. These variations alone provide ample opportunity for confounding the results of the study. We consider this research to be an exploratory first step toward assessing the validity of the SELEB scale. Additional studies with larger samples and different class projects are needed to examine the robustness of the scale.

REFERENCES

Astin, A. W. and L. J. Sax (1998), "How Undergraduates are Affected by Service Participation," Journal of College Student Development, 39 (3), 251-263.

Batchelder, T.H. and Root, S. (1994) "Effects of an Undergraduate Program to Integrate Academic Learning and Service: Cognitive, Prosocial cognitive, and Identity Outcomes," Journal of Adolescence, 17, 341-355.

Berson, J. S. and W. F. Younkin (1998), "Doing Well by Doing Good: A Study of the Effects of a Service Learning Experience on Student Success," American Society of Higher Education, Miami, FL.

Blackwell, A. P. (1996), "Students' Perceptions of Service Learning Participation in the College of Health and Human Sciences at The University of Southern Mississippi." Unpublished Dissertation, The University of Mississippi.

Bringle, Robert G. and Julie A. Hatcher (1996), "Institutionalization of Service Learning in Higher Education," Journal of Higher Education, 71 (3), 273-290.

"Campus Contact annual service statistics 2000," (2000) The Internet: http://www.compact.org/ newscc/stats2000/defaulty.html

Easterling, Debbie and Frederica Rudell (1997), "Rationale, Benefits and Methods of Service-Learning in Marketing Education," Journal of Education for Business, 73 (1), 58-61.

Ekrich, Donald, and Clay Voorhees (2002), "Expanding Experiential Learning in Marketing: The Synergy of Affiliation and Visitation," Proceedings American Marketing Association.

Eyler, Janet S. and Dwight. E. Giles (1999), "Where's the Learning in Service Learning?," San Francisco, CA. Jossey-Bass, Inc.

Eyler, Janet S., Dwight E. Giles, Christine M. Stenson, and Charlene J. Gray, (2001), "At a glance: What we know about the effects of service-learning on college students, faculty, institutions and communities, 1993-2000: Third Edition, Vanderbilt University, (August 31, 2001)

Gujarathi, Mahendra R. and Ralph J. McQuade (2002), "Service-Learning in Business Schools: A Case Study in an Intermediate Accounting Course," Journal of Education for Business, 144-150.

Ikeda, E. K. (2000), "How Does Service Enhance Learning? Toward an Understanding of the Process," Unpublished Dissertation, University of California, Los Angeles.

Litke, Rebecca A. (2002), "Do All Students 'Get It?': Comparing Students' Reflections to Course Performance," Michigan Journal of Community Service Learning, 8(2), pp. 27-34.

Mabry, J. B. (1998), "Pedagogical Variations in Service-Learning and Student Outcomes: How Time, Contact and Reflection Matter," Michigan Journal of Community Service Learning, 5, 32-47.

Petkus, Jr., Ed, (2000), "A Theoretical and Practical Framework for Service-Learning in Marketing: Kolb's Experiential Learning Cycle," Journal of Marketing Education, 22 (April), 64-70.

Reid, Jane S., Mark F. Toncar and Cynthia E. Anderson (2004), "All Service Learning Projects are not Perceived as Equal by Students," Proceedings, Association of Marketing Theory and Practice, Brenda Ponsford ed., March 25-27, 2004, session 6:1; p 1-5.

Root, Rob and Trisha Thorme (2001), "Community-Based projects in Applied Statistics: Using Service-Learning to Enhance Student Understanding," The American Statistician, 55 (4), 326-331.

"Service-Learning is ...," The Internet: http://www.servicelearning.org/article/archive/35

Mark F. Toncar, Youngstown State University, Youngstown, Ohio, USA

Jane S. Reid, Youngstown State University, Youngstown, Ohio, USA

Cynthia E. Anderson, Youngstown State University, Youngstown, Ohio, USA

Toncar, Mark F., Jane S. Reid and Cynthia E. Anderson (2004) "Student Perceptions of Service Learning Projects: Exploring the Impact of Project Ownership, Project Difficulty and Class Difficulty," Proceedings of the International Applied Business Research Conference, San Juan, Puerto Rico, March, 2004, article #272.

Toncar, Mark F., Jane S. Reid, David J. Burns, and Cynthia E. Anderson (2003)"The SELEB Scale: A Multidimensional Scale to Assess the Benefits of Service Learning," Proceedings of the Atlantic Marketing Association, Jerry W. Wilson, ed., 116-122.

TABLE 1. DEMOGRAPHICS OF THE TWO CLASSES

Class                 Nonprofit     For Profit      Chi-     Level of
Characteristic      Class Project  Class Project   Square  Significance

Percentage of           31/69          44/56        .848       .357
Males/Females
Percentage of
  Seniors/Juniors       85/15          87/13        .055       .815
Major: Percentage
Marketing/
  Management/Other    73/12/15        78/13/9       .512       .774
Class Required:
  Yes/No                81/19           91/9        1.11       .293

TABLE 2.
ANALYSIS OF RANKING OF SELEB DIMENSIONS BETWEEN STUDENTS IN
TWO CLASSES

SELEB Scale Dimension    Spearman Rho Correlation    Level of
                               Coeffiecient        significance

Practical skills                  0.163                0.263
Interpersonal skills              0.091                0.534
Citizenship skills                -.025                0.865
Personal responsibility           -.239                0.099

TABLE 3
ANALYSES OF DIFFERENCES BETWEEN PRETEST/POSTTEST RATINGS OF SELEB ITEMS

SELEB scale item           Nonprofit   For Profit  t-test  Level of
                             mean        mean      value     sig.
                           difference  difference

Personal growth               .462       -1.739     4.79     .000
Ability to work well          .654       -1.609     4.46     .000
with others
Leadership skills             .923       -1.391     4.84     .000
Communication skills          .692       -1.044     4.47     .000
Understanding cultural        .077       -1.652     3.19     .003
and racial differences
Social responsibility and     .462       -1.087     3.27     .002
citizenship skills
Community involvement         .308       -.304      1.48     .146
Applying knowledge to the     .808       -1.044     4.09     .000
real world
Problem analysis and         1.269       -.609      4.32     .000
critical thinking
Social self-confidence        .885       -1.435     3.63     .001
Conflict resolution          1.154       -1.304     4.16     .000
Ability to assume            1.039       -.913      3.27     .002
personal responsibility
Development of caring         .808       -1.826     5.06     .000
relationships
Gaining the trust of          .962       -1.478     4.05     .000
others
Empathy and sensitivity       .962       -.739      3.33     .002
to the plight of others
Workplace skills              .308       -1.087     2.61     .012
Ability to make a             .962       -.652      4.03     .000
difference in the
community
Skills in learning from       .963       -.739      3.67     .001
experience
Organizational skills         .808       -.913      3.22     .002
Connecting theory with        .654       -.870      3.27     .002
practice

TABLE 4
ANALYSES OF RATINGS OF CLASS PROJECTS

Statement             Nonprofit   For Profit   t-value   Level of
                      class mean  class mean               sig.

Class project was        4.96        4.65       0.593      .556
something to put on
resume
Class project helped     5.88        6.00      -0.367      .715
illustrate different
concepts discussed
in class
Class project made a     6.04        5.17       2.204      .032
meaningful
contribution to the
organization
Enjoyed the class a      5.23        3.91       2.556      .014
lot more because of
class project
Would have preferred     3.04        4.00      -1.874      .067
just lectures and
tests with no class
project
Would recommend this     5.04        4.22       1.868      .068
class to my friends