The inequivalence of an online and classroom based general psychology course.

One-hundred seventy-five students enrolled in either a traditional classroom lecture section of General Psychology or in an online section of the same course were compared on exam performance. When covariates of high school grade point average and SAT composite scores were entered into the analysis, students enrolled in the classroom based lecture course fared significantly better than the online course students. These results are contradictory to prior investigations of online and classroom based courses.

**********

While distance learning is not a new concept, the delivery of content through a dynamic and interactive media is fairly recent. The delivery of a distance course over the world wide web has existed since about the mid-1990s when the availability of high speed internet connections, html editing tools, and personal computers of sufficient power made an online course practical. While some will argue with the utility of online courses, most would agree that they, are here to stay. Either the demand for services from students or the perceived cost effectiveness of online courses has made them a mainstay in academia. Virtually every academic institution offers some form of an online course, and in most cases offers dozens (Newman & Scurry, 2001). What is less clearly understood is how these online courses compare to the traditional classroom based sections.

The conclusions comparing online and classroom sections of courses are ambiguous. Some studies have reported an exam outcome advantage for the online course students (Maki, Maki, Patterson & Whittaker, 2000: Poirier and Feldman 2004). Another researcher reported no differences between online and classroom sections when students self selected themselves into sections or were randomly assigned to sections (Waschull, 2001). In a hedge to the contradictory outcomes already noted, Kinney (2001) found no differences in exam performance between online and classroom based students on four of five exams. The remaining exam score favored students in the online course section. And finally, Wang and Newlin (2000) report that students in a traditional classroom setting performed better than online peers on a course final exam and had overall higher course grades. To further cloud the issue, the conclusions from these studies are suspect with small sample sizes, in some cases as small as n = 9. failure of random assignment or failing to control for non-random assignment, and/or failure to proctor the examinations. What will be described here is one paradigm for an online course delivery system, and how students fared compared to a traditional classroom section of the same course. Furthermore, the previously identified design flaws will be remedied.

The classroom section of General Psychology consisted of a fifteen week semester with traditional topics covered, usually one chapter or topic per week. Students received vocabulary sheets ("Terms & Concepts") to guide their studying and to identify critical concepts to know for exams. Course delivery consisted of textbook readings, instructor lead lectures and some discussion as appropriate and as student interest dictated. Assessment in the course included three modular examinations over the immediately presented one-third of the course. The last exam was not comprehensive. All exams were proctored in the classroom by the instructor.

The online section of General Psychology was also conducted over fifteen weeks, and run concurrently with the traditional semester calendar. Instead of a lecture for each topic, students received a series of Question and Answer (Q&A) html links derived from the classroom lecture. One page listed all of the question links for a topic, and after posing the particular question, students could follow the link to see the answer. For example, one question link was, "What are some professional areas of psychology?" The answer page would include names and descriptions of clinical, counseling, developmental, cognitive, social, physiological, and human factors psychology. If anyone attended the classroom section, and also saw these online Q&A links, the overlap of content would be obvious and extremely redundant. In addition to the Q&A for each topic, there was a Short Essay Question (SEQ) for each topic which required students to generate applied written responses. For example, one SEQ was, "You friend tells you that her IQ is 95 and she is devastated that she is "below average'. What is your learned response to her, including some history about intelligence testing'?" The intended purpose of the SEQ was to simulate thought provoking answers that occur in class. Student responses generally consisted of one or two paragraph answers, and were overwhelmingly accurate. Students in the online course also had the same Terms & Concepts vocabulary sheets and textbook reading assignments as the classroom based students. Assessment in the course consisted of the same three exams as were received by the classroom based students, each of the three exams over the immediately completed one-third of the course, and proctored in a classroom by the instructor.

In summary, students in both sections received the same reading assignments. Terms & Concepts vocabulary sheets, similar versions of a "lecture" or "Q&A", and completed the same three exams. An additional task required of the online section students included the SEQ to simulate classroom discussion.

The purpose of this study is to examine whether or not these two delivery systems resulted in similar student outcomes. Student performance in the online section and the classroom based section will be compared by examining exam scores for each group. It is expected that no differences will exist between the groups, the delivery systems resulting in equivalent student exam performance.

Method

Participants

One hundred seventy-live students enrolled in the author's General Psychology courses from Fall 1998 to Fall 2003 at Ursuline College were employed for this study. Only students who completed the course were included in the data set. Ursuline College is a small liberal arts Catholic women's college located in north-eastern Ohio.

Materials & Procedure

Archival institutional records were examined to collect standardized achievement scores from students who completed General Psychology. Composite scores from the Scholastic Aptitude Test (SATC), high school grade point average (HSGPA) and the sum of the three exam scores for each student in General Psychology were collected. All students used the same textbook (e.g., Coon, 2003), although the edition changed for everyone during the sampling period.

Design

This a single factor between subjects quasi-experimental design. The quasi independent variable is Course Delivery with two levels (Online, Classroom). Two covariates were introduced into the design, SATC and HSGPA. The dependent variable is the sum of the three exam scores earned in the General Psychology course expressed as a percentage. Students self-selected themselves into either the online or classroom section of the course.

Results

Course grades expressed as a percentage were analyzed with a single factor between subjects analysis of covariance. The quasi-independent variable was Course Delivery with two levels (Online, Classroom). Covariates introduced into the analysis were HSGPA and SATC scores. A conservative p-value of .01 was used to determine significance due to violation of random assignment of participants to conditions.

There were significant effects of the covariates HSGPA (F(1,171) = 22.997, p < .001) and SATC (F(1,171) = 54.622, p < .001). After the effects of HSGPA and SATC were removed, there was also a significant effect of Course Delivery (F(1,171)= 29.436, p < .001). Students who completed the classroom section performed significantly better than those in the online section. See Table 1.

A secondary analysis investigated the characteristics of students" entrance scores. Participants' SATC and HSGPA were analyzed in a 2 x 5 between subjects MANOVA. The first factor, Course Delivery has two levels: Online, Classroom. The second factor, Course Grade has five levels: A, B, C, D, F. A conservative p-value of .01 was used for violation of random assignment of participants to conditions.

SATC Analysis

The was a significant effect of Course Delivery system (F(1,165) = 48.703, p = .001) with students enrolled in the online course having higher average SATC scores than those students enrolled in the classroom section. There was also a significant effect of Course Grades (F(4,165) = 23.701, p < .001). SATC scores were consistently ascending in value from grades of F through A. There was not a significant interaction of Course Dell very and Course Grade (F(4,165) = .563. p = .690). See Table 2.

HSGPA Analysis

The was a significant effect of Course Delivery system (F(1,165) = 22.804, p = .001) with students enrolled in the online course having higher average high school grade point averages than those students enrolled in the classroom section. There was also a significant effect of Course Grades (F(4,165) = 14.315, p < .001). Students" HSGPA consistently ascended in value from F through A exam grades. There was not a significant interaction of Course Delivery and Course Grade (F(4,165 = .321, p = .864). See Table 3.

Discussion

Exam scores were compared from students enrolled in either a General Psychology online course or a traditional classroom setting. On the face of it, students in each class performed similarly as evidenced by the unadjusted mean exam scores for each group. However, when consideration was given to likely covariates of their academic ability (SATC, HSGPA), students in the classroom section performed better than students in the online course (review Table 1). Despite considerable efforts to make the two General Psychology course delivery systems equivalent, in the end, they were not. Furthermore, for a student enrolled in the online course to earn a similar exam score as a classroom based student, the online course student needed a higher SAT composite score and a higher high school grade point average (review Tables 2 & 3). This is not terribly surprising. Online classes tend to require more independent work, and perhaps are actually more difficult for students to complete and be academically successful. This conclusion is very tenuous since other, non-examined factors could be contributing to academic course achievement, viz., motivation or g. One could create a very compelling argument that those students who are of higher ability (g) or motivation are likely to achieve regardless of the situation in which they are placed. Indeed, if Frey and Detterman (2004) are right and the SAT is basically an intelligence test, then the SAT scores used in this study are a measure of g. More g means that academic tasks will probably be easier to complete, and a given g should somewhat predict academic achievement. In this study, SAT composite scores did systematically differ with respect to exam scorns and also with respect to the course delivery system in which the student was enrolled.

Since adjusted exam score means revealed that the online students performed poorer than the classroom section students, a knee jerk response would be to abolish the online section. After all, why should we put students into situations where they are more likely to under perform? The thoughtful response is just the opposite. This study demonstrated the inequivalence of these two particular course delivery systems. This particular online course paradigm just isn't the way to do it if we want students of similar ability to perform equally well in the online and classroom based delivery systems. It doesn't mean that the course delivery systems cannot be eventually comparable. Considerable retooling of delivery of content and assignments is needed for the online course to become more "learner friendly".

On the other hand, there are plenty of other situations where students are less likely to optimally perform and we simply caution them and allow them to make an informed decision. Students will frequently enroll in course overloads to accelerate their class standing, graduation date or to maintain eligibility for financial aid. They may heavily load a single day with several hours of classes to free up other week days for study or employment. Additionally, we are more frequently faced with students who not only are enrolled full time, but also employed full time and perhaps even have substantial family responsibilities. In the end, we advise and caution students about their course choices, but the choices are theirs. For some students, some times, it may be more important to earn the course credit through distance learning with a lesser grade than to not have the credit at all.

This study addressed identified criticisms of prior studies where small sample sizes (Poirier & Feldman, 2004; Waschull, 2001), not controlling for non-random assignment (Kinney, 2001; Wang & Newlin, 2000; Waschull, 2001) and comparing student performance on unproctored exams (Kinney, 2001; Maki, Maki, Patterson & Whittaker, 2000; Wang & Newlin, 2000; Waschull, 2001) cast doubt on earlier experiments comparing online and classroom based courses. In this study, a sample size of 175 was used, likely covariates of course performance were employed (SATC, HSGPA) to control for non-random assignment, and proctored exams were used in both course sections. The future directions for additional research include revising the online course to include additional study tools, and making several interative investigations of the course outcomes, comparing the online course students and those in the classroom section. In the mean time, valuable data exist to help us guide students into the appropriate course delivery system based on incoming high school grades and SAT scores. While there will always be exceptions, these data will provide valuable assistance in helping us and students make informed decisions about course selection and delivery systems.

Author Note

This research was supported in part by a sabbatical leave from Ursuline College during the spring semester of 2005. I am grateful to Sr. Christine DeVinne, Sr. Patricia McCaffrey, JoAnne Podis, Sr. Diana Stano and members of the President's Council for their encouragement to complete this project.

References

Coon, D. (2003). Essentials of Psychology (9th Ed). New York: Wadsworth.

Frey, M. C. & Detterman, D. K. (2004). Scholastic assessment or g? The relationship between the scholastic assessment test and general cognitive ability. Psychological Science, 15(6), 373-378.

Kinney, N. E. (2001). A guide to design and testing in online psychology courses. Psychology Learning and Teaching, 1, 16-20.

Maki, R. H., Maki, W. S., Patterson, M., & Whittaker, P. D. (2000). Evaluation of a Web-based introductory psychology course: I. Learning and satisfaction in on-line versus lecture courses. Behavior Research Methods, Instruments, & Computers, 32, 230-239.

Newman, F., & Scurry, J. (2001, July 13). Online technology pushes pedagogy to the forefront. The Chronicle of Higher Education, 47, B7-B10.

Poirier, C. R. & Feldman, R. S. (2004). Teaching in cyberspace: Online versus traditional instruction using a waiting list experimental design. Teaching of Psychology, 31, 59-62.

Wang, A. Y., & Newlin, M. H. (2000). Characteristics of students who enroll and succeed in psychology web-based classes. Journal of Educational Psychology, 92, 137-143.

Waschull, S. B. (2001). The online delivery of psychology courses: Attrition, performance, and evaluation. Teaching of Psychology, 28, 143-147.

Christopher L. Edmonds, Ph.D., Associate Professor and Chairperson, Department of Psychology, Ursuline College.

Correspondence concerning this article should be addressed to Christopher L. Edmonds at [e-mail].

Table 1
Adjusted and Unadjusted Means of Exam Scores
for Each Delivery System

                     Unadjusted   Adjusted
                       Means       Means

Online (n = 104)       78.6%       73.6%
Classroom (n = 71)     77.5%       81.1%

Table 2
Mean and standard deviation SATC scores
as a function exam grades
and course delivery system

                     F        D          C

Online              898     1014       1117
                  (149)    (164)      (183)
Classroom           773      820        893
                   (63)    (126)      (133)
Marginal Means      851      863        973
                  (134)    (156)      (186)

                     B        A    Marginal
                                     Means

Online             1197     1383       1152
                  (123)    (125)      (181)
Classroom          1017     1118        940
                  (103)    (149)      (164)
Marginal Means     1119     1187
                  (145)    (184)

Table 3
Mean and standard deviation high school grade point
averages as a function exam grades and course
delivery system (A = 4.0)

                     F        D          C

Online             3.14     3.27       3.40
                 (.186)   (.594)     (.413)
Classroom          2.47     2.83       3.09
                 (.371)   (.349)     (.476)
Marginal           2.89     2.93       3.20
                 (.425)   (.445)     (.476)

                     B        A    Marginal
                                     Means

Online             3.73     3.96       3.58
                 (.261)   (.366)     (.417)
Classroom          3.36     3.62       3.17
                 (.534)   (.502)     (.545)
Marginal           3.57     3.71
                 (.439)   (.487)