Student outcomes and honors programs: a longitudinal study of 172 honors students 2000-2004.
Since this edition of the JNCHC is dedicated to honors administration, it seems appropriate to offer a few introductory remarks about the usefulness of this study. College and university administrators participating in the accreditation process are well aware that assessing student learning is not the passing fad that some had suspected it might be. In the Southern Association of Colleges and Schools, for example, administrators are familiar with Core Requirement 2.1--the institution engages in ongoing, integrated, and institution-wide planning and evaluation processes that incorporate systematic review of programs and services (Handbook for reaffirmation of accreditation, 2004). All accreditation bodies in higher education now require systematic assessment of student learning.
Honors programs have been generally slow to adopt ongoing assessment strategies, and calls for intentional evaluation of honors education are not new. In a National Collegiate Honors Council monograph published in 1995, for example, Reilhman, Varhus and Whipple noted that "... the paucity of evaluations of honors programs is surprising" (p.2). A decade earlier, Randall and Collier (1985) observed, "examples of efforts to evaluate the effect of honors programs on the college career ... are extremely rare"(p. 2). Asearch of relevant literature today suggests that only marginal progress has been made toward providing substantive and scientifically gathered data about how student learning is enhanced as a result of participation in an honors program.
Until honors deans and directors make assessment and evaluation a priority, evidence that honors programs produce valuable outcomes for student learning will go largely unsubstantiated. The risks associated with such a condition will be especially amplified when resources are tight and financial officers are prowling for places to reduce costs. As Derek Bok (2005) admonishes, "Throughout undergraduate education, a great wall separates the world of research from the world of practice--even though practitioners involved are professors, trained in research, who would seem ideally prepared to take full advantage of whatever findings empirical investigations have to offer" (p. 9).
More important than proving the value of honors education and meeting accreditation guidelines, however, is the likelihood that improved assessment practices will unearth practical findings relevant to improving overall effectiveness of honors education. The following is a case in point and offers honors administrators a practical example of how the scientific method can assist in evaluating honors student outcomes.
BACKGROUND OF STUDY
This is a follow-up study of 172 honors students selected during their first year of college. The original study (Shushok, 2002) was designed to assess how students were affected by participation in the Honors College at a Carnegie-classification "Doctoral/Research Extensive" university in one of the Mid-Atlantic states. These 172 students were surveyed at the conclusion of their first year (2001) and again at the conclusion of their fourth year (2004). The 2001 study utilized a quantitative, quasi-experimental design using the College Student Experiences Questionnaire (CSEQ) as well as qualitative focus groups (Krueger, 1994) conducted after data analysis. The original study was funded, in part, by the National Collegiate Honors Council (NCHC). The 2004 study was funded by the participating institution.
SELECTION OF ORIGINAL STUDY PARTICIPANTS AND FINDINGS
For the purpose of this research, two groups of similarly credentialed students were selected and studied in April 2001 and again in April 2004. Half of these students (86) applied and were selected to participate in the Honors College. The other half (86) were equally qualified students who did not apply and therefore were not participants in this program. All students were beginning their first experience with postsecondary education (defined as having no more than nine semester hours of college-level work). Further, all students were of "traditional" age (defined as 17-22 years of age) and had achieved a high-school grade point average of at least 3.5 and a minimum combined SAT score of 1250. In addition to SAT scores and high-school grade point averages, each group was controlled to achieve a balance in race, gender, and place of residency (on- or off-campus housing), see Table 1.
To ensure that students in both groups were as similar as possible, caliper matching (Anderson et al., 1980) was utilized to match each of the randomly selected honors students with a non-honors student. As described in Anderson et al., "Caliper matching is a pair matching technique that attempts to achieve comparability of the treatment and comparison groups by defining two subjects to be a match if they differ on the value of the numerical confounding variable by no more than a small tolerance" (p.79). At the conclusion of the matching process, a perfect match was achieved between honors and non-honors students in the categories of race, gender and residency. Since finding an identical match in grade point average and SAT score was unlikely, differences within .15 of a standard deviation were considered acceptable. In the event that a student did not matriculate or declined to participate, another student was selected from the pool of students. In addition, when a match for an honors student could not be found, the honors participant was dropped from the study, and another student was selected.
To verify that the matching process had been successful, the SAT and GPA means were calculated for both groups, and an Independent Samples T-Test was utilized. After the matching procedure, the mean SAT score for the honors group was 1346 compared to 1339 for the non-honors group. The Independent Samples T-Test using an alpha level of .05 indicated no statistical difference in SAT scores between the two groups. Moreover, the two groups varied by only 11 percent of one standard deviation. The mean GPA for the honors group was 3.96 while the mean GPA for the non-honors group was 3.95. The Independent Samples T-Test using an alpha level of .05 reported no statistical difference in GPA between the groups. Additionally, the variation consisted of only three percent of one standard deviation. From the sample of 172 students (86 honors and 86 non-honors), 85.4 percent (147 of 172) returned usable data. Four students had left the university (two honors and two non-honors) at the time of data collection, see Table 2.
PRIMARY FINDINGS OF THE INITIAL 2001 STUDY
1. Honors students outperformed non-honors students as measured by cumulative grade point averages at the conclusion of the first year (Honors 3.41, Non-Honors 3.18). This study, therefore, suggested that participation in the Honors College had a positive effect on students' academic performance as measured by GPA during the first year.
2. Honors students had higher retention rates into the sophomore year (Honors 97 percent; Non-Honors 90 percent).
3. Honors and non-honors students engaged in extracurricular activities at similar rates (excluding statistical interactions). The measurement of perceived gains in the liberal arts, sciences, or technology, however, showed differences. Honors students estimated gains in these areas that were statistically significant when compared to those reported by non-honors students.
4. Honors programs may encourage outcomes for male students in a way that they do not for female students. For example, when considering interaction with faculty outside of the classroom, male honors students reported significantly higher levels of engagement than traditional students while female honors students reported essentially the same engagement with faculty when compared to traditional students. If participation in honors had a positive effect on student engagement with faculty, it was only for male students. Satisfaction with college followed a similar pattern and suggests that participating in honors has a large effect on this dependent variable for male students and no real effect for female students.
THE 2004 STUDY
In the April 2004, three years after data collection of the original study, the researcher returned to the university to study these students again. Of the original 172 students, 24 had left the institution without completing a degree (9 honors and 15 non-honors students). Of the 148 remaining students (including 6 honors and 7 non-honors students who had already received a degree from the institution), 104 students (70.2 percent) returned usable data. Of the 104 respondents, 90 completed both the 2001 survey and the 2004 survey (86.5 percent). Therefore, 14 participants, although selected to participate in 2001, only completed the 2004 questionnaire. For the calculation of grade point averages and retention data, however, all 148 students were considered.
Quantitative data for this study were collected using a self-designed 33-item survey to measure the type and frequency of student interaction with faculty members (6 questions); the type and frequency of participation in specified activities (15 questions); the level of student satisfaction with specified components of the learning environment (6 questions); and finally, student estimates of gains in specified areas (6 questions). Some questions were modeled after the College Student Experiences Questionnaire that was used in the initial 2001 study.
The questionnaire was reviewed and modified numerous times by a staff, faculty, and student steering committee on the researcher's campus. To estimate test reliability, 6 focus groups of 8 students completed the instrument twice over a four-week period. Based on the focus group administration of the instrument, test-retest correlation coefficients ranged between .72 and .89 while internal consistency (Cronbach's alpha) ranged between .79 and .90. As noted in Gall et al., (1996), reliability of .80 or higher suggests that results are generally suitable for most research purposes. At the request of the campus Honors College, data were also collected about students' intended graduation dates and post-graduation plans such as attending graduate school. Participants were contacted by way of their university electronic mail accounts. If a student agreed to participate, $5.00 cash was provided for a completed survey. The questionnaire was administered in the campus library.
A logistical regression was employed to analyze the ordinal data collected in the questionnaire and provide comparative statements by way of odds ratios. These describe differences in the attitudes or behaviors between honors and non-honors students. Comparisons also considered subsets of the surveyed group of students. These subsets included male, female, majority, and minority students. The ordinal responses represented opinions on a scale of 1, 2, 3, 4 and are depicted below in Table 3.
Focus group techniques (Krueger, 1994) were used to collect data for the qualitative phase of the study. Focus groups are discussions facilitated by a researcher in order to identify variables and patterns in perceptions. Three focus groups were conducted with six students in each. Focus groups were conducted in a library classroom on campus and lasted approximately 90 minutes.
RESULTS FOR 2004 STUDY
GRADE POINT AVERAGES
Table 4 indicates that honors students and non-honors students as measured by cumulative grade point average in the spring 2004 performed similarly. While the 79 remaining honors students (still enrolled or graduated) earned a mean grade point average of 3.46, the non-honors students earned a mean grade point average of 3.40. This difference is not statistically significant and is in contrast to what was found at the conclusion of the first year. As presented in Table 5, while honors students outperformed non-honors students at the conclusion of the first year, grade point averages had leveled by 2004. Evidence, therefore, would indicate that participation in an honors program had a positive effect on students' academic performance as measured by cumulative grade point average after the first year of study but not after four years of study.
Since the sample size was limited to 104 students, the levels were collapsed. Analysis, therefore, was performed on two levels of response. This allowed the researcher to make more valid conclusions. Table 6 represents how survey responses were collapsed. All odds ratios reported statistically significant at the .05 level and satisfied certain assumptions related to sample sizes and expected values. Odds ratios provided below represent survey questions that were statistically significant.
ODDS RATIOS (ALL SURVEYED STUDENTS)
Met with a Faculty Member During Office Hours
Students in the honors program were 2.5 times more likely than students in the non-honors program to meet with a faculty member during office hours very often/often (compared to occasionally/never).
Discussed Career Plans and Vocational Aspirations with a Faculty Member
Students in the honors program were 3.1 times more likely than students in the non-honors program to discuss career plans and vocational aspirations with a faculty member very often/often (compared to occasionally/never).
Discussed a Social Concern, Political Issue, or World Event with Another Student Outside of Class
Students in the honors program were 2.5 times more likely than students in the non-honors program to discuss a social concern, political issue, or world event with another student outside of class very often/often (compared to occasionally/never).
METHOD (MALE STUDENTS ONLY)
Discussed Career Plans and Vocational Aspirations with a Faculty Member
Male students in the honors program were 4.7 times more likely than male students in the non-honors program to discuss career plans and vocational aspirations with a faculty member very often/often (compared to occasionally/ never).
Discussed a Social Concern, Political Issue, or World Event with Another Student Outside of Class
Male students in the honors program were 5.3 times more likely than male students in the non-honors program to discuss a social concern, political issue, or world event with another student outside of class very often/often (compared to occasionally/never).
In Activities with an Academic Emphasis (Outside of Class)
Male students in the honors program were 3.6 times more likely than male students in the non-honors program to be very involved/involved (compared to somewhat involved/uninvolved) in activities with an academic emphasis (outside of class).
DISCUSSION OF RESULTS
Honors student retention and grade point average advantages detected at the end the first year appear to have leveled by the time students ended their fourth year of study. Of 148 students eligible for the 2004 study (79 honors and 69 non-honors), the grade point averages were 3.46 and 3.40 respectively. As mentioned earlier, of the 172 original students selected for the 2001 study, 21 students were retention casualties (9 honors and 15 non-honors). Neither the grade point average nor retention differences between honors and non-honors students were statistically significant in 2004. One might argue, however, that this is less important since the greatest risk for a student departing an institution takes place during the first year.
As in the 2001 study, there were generally no statistically significant differences in the type of activities in which honors and non-honors students participated with three notable exceptions.
Honors Students Appear to have a Quantitatively and Qualitatively Different Experience when Looking at the Type and Level of Interaction with Faculty Members. This is Especially Pronounced for Male Students.
Honors students were 2.5 times more likely than non-honors students to meet with a faculty member during office hours and 3.1 times more likely than non-honors students to discuss career plans and vocational aspirations with a faculty member. This difference appears to be even more pronounced for male honors students, who were 4.7 times more likely than non-honors males to meet with a faculty member during office hours and 5.3 times more likely than male non-honors students. This result supports evidence in the 2001 study that suggests honors programs may encourage outcomes for males in a way that it does not for female students. Especially for male students, both the 2001 and 2004 studies suggest that participation in an honors program has a statistically significant effect on student engagement with faculty. Focus group follow-up discussions exhibited the same pattern.
Honors Students Appear to be More Engaged with Peers in Discussing Contemporary Issues such as Social Concerns, Politics, and World Events. This is Especially Pronounced for Male Students.
Honors students were 2.5 times more likely than non-honors students to discuss a social concern, political issue, or world event with another student outside of class. Again, this difference appears to be more pronounced for male honors students, who are 5.3 times more likely than male non-honors students to discuss a social concern, political issues or world event with another student outside of class. Especially for male students, both the 2001 and 2004 studies suggest that participation in an honors program has a statistically significant effect on student engagement with peers as it relates to discussions about contemporary issues.
Male Honors Students are More Likely to Engage in Activities Outside of Class with an Academic Emphasis.
Male honors students were 3.6 times more likely than male non-honors students to be involved in activities outside of class with an academic interest. This difference again underscores the potential value of male student participation in the honors program.
KEY FOCUS GROUP RESULTS/THEMES
Focus group discussions suggested that the Honors College created an infrastructure for bright students to connect and feel comfortable with "like minded" students who, as one student suggested, "Value learning and feel that it's okay to be smart and study."
These discussions also supported quantitative data suggesting that this kind of connection may be especially important for male students, who found it "less culturally acceptable" to be academically oriented. As a result, participation in an honors program may facilitate male student entry into an environment where they feel safe developing academic inclinations and interests more readily. This may be a plausible theory for why males appear to benefit more substantially from honors participation when compared to females. Women, it may be, are able to find academically supportive peer groups outside of honors participation while men find such support more difficult to identify.
Focus group participants affirmed the notion that faculty contact is more frequent for those involved with the Honors College. In one exchange, a non-honors student listened to an honors student describe his strong contact with faculty; in response, the non-honors student said, "I really haven't had the access you had with the faculty."
Focus groups also revealed that many students either departed or became less involved in the Honors College during their sophomore, junior, and senior years. Students left or became less engaged either because they were asked to depart by the Honors College (for not meeting grade requirements, failing to take an honors class each semester) or, more frequently, because coordination of class schedules and degree requirements became too complicated. It could be suggested, therefore, that the strength of impact for the Honors College was most intense during the first year. This appears to be a reasonable theory for why the influence of honors participation on GPA and retention leveled between the years 2001 and 2004. Some investigation of "administrative obstacles" for returning honors students may help retention in honors programs and therefore learning outcomes.
The four-year study discussed here is but one example of the types of data that can be gathered to bolster support for honors education as well as improve it. The most important point, however, is that assessment efforts can uncover valuable information to improve the effectiveness of honors programs and the influence such efforts have on student learning outcomes. In this example, university officials will want to explore more thoroughly the possibilities for the differing impact on men and women. There are also more practical issues, including whether or not the institution can reduce unnecessary barriers that often prevent students from remaining in the program and thus being recipients of the influence such programs have on student learning. Since the urgency of daily administrative demands will always push assessment activities to the margins, honors deans and directors must be vigilant in demanding that such efforts be priorities. This may, however, require active involvement in assessment and research activities rather than delegation of them elsewhere.
Anderson, Sharon and et al. (1980). Statistical method for comparative studies: techniques for bias reduction. New York: Wiley and Sons.
Bok, Derek (2005). Our underachieving colleges: A candid look at how much students learn and why they should be learning more. Princeton: Princeton University Press.
Gall, M. et al. (1996). Education research: An introduction. New York: Longman.
Handbook for reaffirmation of accreditation (2nd ed.) (2004). Decatur, Georgia: Southern Association of Colleges and Schools.
Krueger, Richard A. (1994). Focus groups: A practical guide for applied research. Thousand Oaks: Sage.
Randall, C. and Spiller, N. (1985). Acomposite profile of the honors program in higher education. Forum for Honors 3, 20-30.
Reilman, J., Varhus, S. and Whipple, W. (1995). Evaluating honors programs: an outcomes approach. Monograph of the National Collegiate Honors Council.
Shushok, F., Jr. (2002). Educating the best and the brightest: Collegiate honors programs and the intellectual, social and psychological development of students. Ph.D. Dissertation. University of Maryland, College Park.
The author may be contacted at Frank_Shushok@baylor.edu.
FRANK SHUSHOK, JR.
Table 1. Comparison of Original 172 Study Participants Honors Non-Honors All New Group Group Students Mean SAT 1346 1339 1186 Mean GPA 3.96 3.95 3.45 Female 51.1% 51.1% 49% Male 48.9% 48.9% 51% On-Campus Living 83.8% 83.8% 67% Off-Campus Living 16.2% 16.2% 33% White 81.4% 81.4% 66% Black 4.7% 4.7% 15% Asian 11.6% 11.6% 16% Hispanic 2.3% 2.3% 3% Table 2. T-Test Indicating No Significant Difference between Honors and Non-Honors Study Groups after the Matching Process Standard Group N Mean Deviation Significance SAT Honors 86 1346.16 62.36 .784 Non-Honors 86 1339.18 62.14 GPA Honors 86 3.96 .27 .944 Non-Honors 86 3.94 .27 * p>.05 Table 3. Survey Ordinal Responses Questions Level 1 Level 2 Level 3 Level 4 1 to 13 Very Often Often Occasionally Never Very Somewhat 14 to 21 Involved Involved Involved Uninvolved Extremely Extremely 22 to 27 Satisfied Satisfied Dissatisfied Dissatisfied 28 to 33 Very Much Quite A Bit Some Very Little Table 4. T-Test Results Comparing Grade Point Averages of 2004 Study Standard Group N Mean Deviation Significance GPA Honors (all) 79 3.46 .47 .68 Non-Honors 69 3.40 .40 * p<.05 Table 5. T-Test Results Comparing Grade Point Averages of Study Participants after the Completion of First Academic Year (2001) Standard Group N Mean Deviation Significance GPA Honors (all) 86 3.41 .59 .07 Non-Honors 86 3.18 .75 * p<.05 Table 6. Collapse Method Questions Compares 1 to 13 1&2 to 3&4 14 to 21 1&2 to 3&4 22 to 27 1&2 to 3&4 28 to 33 1&2 to 3&4
|Printer friendly Cite/link Email Feedback|
|Title Annotation:||Research Essays|
|Author:||Shushok, Frank, Jr.|
|Publication:||Journal of the National Collegiate Honors Council|
|Date:||Sep 22, 2006|
|Previous Article:||Major forerunners to honors education at the collegiate level.|
|Next Article:||Dedication: Anne Ponder and Sam Schuman.|