Evaluating student use of Web-based course material.

Course management software and on-line course material are becoming more available.. As a result, many instructors have developed WebCT course Web sites to deliver on-line course material that supplements lecture topics. The present study examined associations between student characteristics, course performance, and access of Web-CT course material. We tracked and coded patterns of Web usage by 154 students in a Psychology as a Profession course. Frequency of student access to Web-based material correlated positively with grades on course assignments. Females accessed the home page more often than males. Self-reports from students evaluated the course Website as highly valuable. These results suggest important advantages in supplementing lecture courses with on-line material.

**********

A national survey revealed that psychology faculty view the Internet as an effective teaching tool (Vodanovich & Piotrowski, 2001). The present study aimed to determine what features of Web-based course material relate to subsequent course performance and whether access to the material varied by student characteristics. Published examples describe the incorporation of Web-based assignments and course supplements to enhance introductory psychology (Waschull, 2001), social psychology (Goldstein 1998; Lawson, 2000; Sherman 1998) and experimental (Goolkasian, Wallendael, & Gaultney, 2003) or research methods courses (Couch 1997; Rosen & Petty 1997). Unlike the attendance roster in lecture or classroom courses, it is not always certain with Web-based course material how often students "attend" the Web site or interact directly with the on-line material.

One might ask from a skeptical point of view what value, if any, is associated with student utilization of a course Website. Couch (1997) designed a research methods course Web site to present lecture notes, handouts, and study guides. Previously, students had purchased a course packet containing these supplementary materials at the start of the semester. The Web site offered an immediate advantage to students, who no longer had to pay for a course packet or possibly lose it. Likewise, the instructor no longer needed to prepare the packet in advance: the instructor could add or revise materials on the Web site throughout the semester. Based on course evaluation results, students reported they accessed the course Web site an average of three times per week. All of the students liked having access to the lecture notes, and 95% of the students felt that other instructors should develop similar course Web sites (Couch 1997).

In another study, Rosen and Petty (1997) found that students who enrolled in a Web-supplemented research methods course rated on-line databases as more useful at the end of the semester than at the beginning of the semester. Goolkasian et al. (2003) intensely studied the response of students enrolled in a cognitive science course designed around 12 Web-based course material modules. In general, students favorably rated the Web site and the "readability" and "usefulness" of the modules. A high percentage (54%) of the students indicated that the Web site was more useful than a traditional textbook.

Other research has found that Web-based course material can be as effective as traditional teaching methods. In a research methods course, student response to an interactive, Web-based tutorial was compared to an in-class lecture demonstration of the sampling distribution (Aberson, Berger, Healy, Kyle, & Romero 2000). Students in both the in-class and Web tutorial conditions improved equally on pre/post-test quizzes on the sampling distribution, and students in the Web tutorial condition rated the tutorial as useful and educational as students in the in-class lecture evaluated the demonstration.

Other research has analyzed patterns of student use of Web-based course material to examine how it relates to student course performance or individual difference variables. Wang and Newlin (2000) examined characteristics of psychology students who chose to enroll in a psychology research methods course taught entirely on the Internet. They found that final course grades were predicted by the number of times a students accessed the homepage as well as personality variables (i.e., need for cognition and internal locus of control). Meyer's (review of 30 studies comparing Web-based versus traditional classes recognized the strong influence of individual differences on learning outcomes.

In a study with dental students, supplemental quizzes were provided on a Web-CT course Web site (Henley, 2003). Students accessed the quizzes less frequently as the semester progressed, ranging from a high of 90% of students at the beginning of the semester to 50% during the final week. Students who earned the highest course grades accessed the site twice as often as students who earned the lowest course grades. On a course evaluation form, 80% of the students rated the supplemental quizzes as "helpful."

In a study conducted with students enrolled in a developmental biology course, Stith (2000) reported equivocal findings between Web site usage and course performance. Although total page hits did not correlate with final grade, the number of articles students read on the Web site bulletin board showed a consistent positive pattern with grades earned in the course.

Likewise, Goolkasian et al. (2003) did not find any systematic relationship between final course grade and time spent viewing the course Web site. However, the authors were cautious about this finding due to the relatively small number of psychology students enrolled. They concluded that in-class attendance was a better predictor of course performance than on-line material access.

As an extension of previous research, we examined the relationship between course performance, student characteristics, and use of Web-based supplemental course material among psychology students. Unlike some of the previous studies in which the authors assessed performance of students who self-select their enrollment into a Web-based version of a course, we include data from all enrolled students. To further consider the generalizability of previous findings, we also examined the patterns of Web usage within a professional development course, rather than a laboratory or statistics course. We obtained demographic characteristics from students at the time of their enrollment in the course: year in school, grade point average, and gender.

Method

Participants

During the Spring 2003 semester, 168 undergraduate students enrolled in a Psychology as a Profession course (Psychology 201) at a large public university. Data from 154 students were included in the data analysis, excluding 14 students who withdrew. A power analysis showed that a sample of 154 participants affords sufficient power (>.90) to detect correlational relationships of rho=.26 or greater at an alpha= .05, two-tailed test.

All psychology pre-majors must pass this one-credit, pass-fail course to enter the major. The course content includes undergraduate advising information, job search strategies, employment opportunities, and interaction with professional guest speakers. We developed a WebCT (//http:www.WebCT. com) course Web site, as described below, to present supplemental course material (Cohen, Futoran, Thorn, & Karraker, 2000) and pilot tested the Web site over several semesters (Cohen & Murray, 2000).

A high percentage of the 133 students passed (87%) with grading based on the course performance measures described below. The majority of students were female (62%), undergraduates (36% freshman, 40 % sophomore, 30% junior, and 4% senior). Most students were pre-psychology majors (88%), and, when asked, 87% planned to declare a major in psychology, and 13% remained uncertain whether they would major in psychology.

Measures

Course performance. Grading was based on out-of-class assignments, participation, final exam, and extra credit. Students needed to earn a minimum passing numerical score usually 70% of the maximum possible points available for each course requirement.

Students completed three assignments, worth 20 points each. Information to assist students with preparing assignments was posted on the Web-CT site. The first assignment, the four-year plan, involved completion of the psychology department's advising booklet, in which the student planned courses to take in order to meet university, college, and department graduation requirements for a psychology degree. The second assignment involved preparation of a one-page resume. The final assignment was a career exploration assignment that involved researching a bachelor-level career and a graduate-level career in psychology or related fields

The cumulative final exam consisted of 50 multiple-choice items including 35 items based on the lecture and reading questions. Answers to the lecture and reading questions were posted to the course WebCT site as a final exam study guide.

Students could optionally attend up to three extra credit lectures, presented by the upper-level undergraduate teaching assistants. Extra credit applied in the event a student failed the final exam or missed more than 3 of the 15 required lectures.

Course Evaluation Data. During the final week of class, students completed a course evaluation on WebCT to provide the instructor with feedback about various aspects of the course. WebCT-related items included 5-point Likert-scale ratings (1=very low through 5=very high) of the value of the course WebCT site and value of readings on the Web site (a separate item assessed the value of textbook readings). On another item, students indicated whether their general Internet use increased (i.e., a lot, somewhat, or stayed the same) as a result of their enrollment in Psychology 201. A final item assessed how frequently students ever accessed the Internet (i.e., never, monthly, weekly, or daily).

Web-CTAccess Data. A WebCT tracking feature stores records (each occurrence) of when an individual student accessed the homepage and the various "content pages" on the WebCT site. We converted this tracking information into the number of times each student accessed the course homepage and the following content pages:

1. Syllabus

2. Answers to Reading/Lecture Questions, which served as a final exam study guide

3. Schedule/Reading, which included lectures topics, assignments due dates, and a link to Web-based reading assignments

4. Announcements

5. Required Assignment information (i.e., four year plan, resume, career exploration)

6. Instructor and undergraduate teaching assistant biographies and e-mail address.

7. Guest Speaker information, which included biographical information for each of four guest speakers.

8. Link to university resources, such as the campus newspaper, psychology department, student services center.

9. Link to Psychology-related Web sites, such as Professional Organizations, Graduate School Information, Forensic Psychology Links, On-line Writing Help

Results

Descriptive Information

The mean number of hits per student to the course homepage on the WebCT site over the 15-week semester was 28.9 (SD=20.1) with a range from 2 to 152. Table 1 shows the mean number of hits per students and range of hits for each of the content pages. Table 1 lists the content pages ordered from most to least hit. As shown in the table, the Schedule/Reading link received the highest mean hits per students.

Relation Between Demographics and Web Site Access

Year in school and university grade point average (GPA) did not significantly differentiate Web utilization in terms of mean homepage hits. However, gender was related to Web utilization. Table 2 shows a gender comparison of average number of hits per student to the course homepage and content pages. Females accessed the course homepage more often than males, F(1,147)=4.20, [n.sup.2]=.03, p<.04. Of the content pages, females more frequently than males accessed the Syllabus, F(1,147)=5.68, [n.sup.2]=.04. p<.02 and the Schedule/Readings, F(1,147)=6.53, [n.sup.2]=.04, p<.01. There were no gender differences for hits to the other content pages.

Relations Between Course Performance and Web Site Access

We compared Web site access between students who passed and students who failed the course on several performance measures: final exam score, total assignments score (four year plan, resume, career explorations), extra credit, and participation points.

Table 3 shows a comparison of Web site access across pages by final grade. Students who passed the course accessed the course homepage more frequently than students who failed, but the difference was not statistically significant. Of the content pages, students who passed, compared to students who failed, more frequently accessed Answers to Reading and Lecture Questions, F(1,154) =5.71, [n.sup.2]=.04, p < .02 and Schedule/Readings, F(1, 154) =3.93, [n.sup.2]=.02, p < .05. There were no significant differences between students who passed and students who failed on number of hits to the other content pages. Total hits to the course homepage correlated significantly with all course performance measures except Extra Credit (r=.08, NS). Specifically, homepage hits correlated with Final Exam score(r=. 19, p<.02), Four-Year Plan score (r=.27,p<.001), Resume score (r=.27,p<. 001), Career Exploration score (r=.19,p<.02), and Participation Points (r=.31. p<.001).

Final exam score correlated significantly with hits to the Schedule/Readings content page (r=.23, p<.005) and Answers to Reading and Lecture Questions page (r=.36, p<.001). The latter page served as a final exam study guide. Similarly, required assignment grades correlated with hits to the Required Assignment content page, which provided instructions for completing the assignments: Four-Year Plan (r=.17, p<.03), Resume (r=.25, p<.002), and Career Exploration (r=.18, p<.03).

Course Evaluation Survey

Students completed a final course survey, using the WebCT survey tool, to evaluate the course material and their experience with the Web site. On a five-point scale (1 = very low value through 5=very high value), they rated the course as moderately valuable (M=3.8, SD= 1.0). Most (86%) agreed that the department should continue to require the course in the major. Additionally, students rated the course Web site as valuable (M=3.9, SD= 1.0). A sample of comments from the survey include: "... actually, the site is better than most I've used for other classes" and "... it has plenty of information on there for the course." Students gave moderate ratings to the readings posted on the Website (M=3.0, SD=.9) and rated the textbook (M=3.21, SD =1.1) significantly, but only slightly higher, t(150)=2.30, [n.sup.2]=.03, p<.02.

Nearly 90% of students reported daily Internet use. As a result of their participation in the course, 27% of students reported that their Internet use "increased somewhat" or "increased a lot." As one student said, "I access the Web-CT site at least once a day, and I really like how it is set up. Everything is easy to get to and simple to follow."

Discussion

We examined variables associated with psychology student Web-CT access of professional development course material. Results indicate that more frequent access to the Web-based course material was consistently related to higher course grades and performance on class assignments. A similar relationship has been found in other research examining Web site utilization (Wang & Newlin, 2000; McCollum. 1997), although Web site usage and course performance were not related in other studies (Goolkasian et. al., 2003; Rosen & Petty, 1997). Equivocal findings might relate to the pedagogical construction of the Web site and what it delivers to the student. Our Web site contained fairly extensive study guides to help prepare for graded projects, and these aids might account for the observed relations between outcomes and usage. Course Web sites without this facilitative material might result in lower student use and, thereby, demonstrate a much smaller association between Web access and course performance.

Another critical aspect of course design vis-a-vis its Web site is the extent to which students are required, explicitly or implicitly, to access the Web site on a regular basis. One might expect more uniformly high use, and less variability, across students and, thus, an attenuated relation. In our course, much of the reading material and assignment instructions were posted only on the Web site; however, this material could be printed out and shared with students who did not access the Web site. In a study reported within McCollum (1997), students in the virtual section of the course outperformed students in the classroom section. However, the course instructor concluded that the online students had formed a strong network of peer-based study groups due to their lack of direct contact with each other. Thus, factors extramural to the classroom, and in this instance, to the Web site appeared to control learning outcomes.

An analysis of demographic variables indicated that female students more frequently accessed the site than males. This finding is consistent with previous research, which indicates that females learn more information from Web screens than male (Zalaquett & Sullivan (1998).

Consistent with the evaluation data reported by Couch (1997), Henly (2003), Lawson (2000), and Rosen & Petty (1997), students rated the use of and ease of access to the course Web site highly. Nearly all students in the present study reported accessing the Internet daily, which suggests that course Web sites can serve as a practical, easily available teaching tool.

Unfortunately, some instructors avoid developing course Web sited due to lack of computer training and time constraints that limit their ability to develop a course Web site (Vodanovich & Piotrowski 2001) or to manage student frustration with novel aspects of the technology (Carnevale, 2003). Instructor reluctance underscores the need for empirical research to confirm that the time and energy investment can be worthwhile.

We observed several advantages in developing Web-based supplemental material. First, on-line posting of handouts, study guides, and other resources saved money on photocopy costs. In addition, we did not need to prepare an extensive course packet before the start of the semester. We could easily revise course handouts and material throughout the semester (cf. Couch 1997). The on-line grade book allowed students to view their individual grades at their convenience, and saved us from preparing and posting grades in public places. Finally, the ability to track student Website access allowed us to identify if an individual student was indeed accessing course readings and participating in the course.

In the future, we may use tracking data to send progress reports or e-mails to students throughout the semester to alert individuals if participation needs to improve and praise individuals if their participation is impressive. Another possible improvement includes the involvement of psychology professionals to participate in guest speaker chats to supplement in-class guest speaker lectures (Bailey & Cotlar, 1994). On-line student-professional interactions may provide a more informal environment, and encourage students to ask questions they may be uncomfortable asking in a large lecture hall.

Assessment and evaluation of student access to Web-based course materials provides an empirical basis for subsequent Web site revision. The present study demonstrates the value of analyzing Web utilization by students, especially as more courses in the psychology curriculum establish Web sites to enhance and deliver course material.

Table 1
Mean, (Standard Deviation), and Range of
Hits by WebCT Content Page

                                     Mean
                                   Hits Per      Range of
Content Page Mean                  Students        Hits

Schedule/Reading                   8.2 (6.5)       0-35
Answers to reading/
  lecture questions                4.9 (4.9)       0-31
Syllabus                           3.6 (3.4)       0-15
Required assignment information    3.6 (3.0)       0-15
Guest speaker information          2.8 (3.2)       0-17
Links to psychology-
  related Web sites                2.8 (2.5)       0-12
Announcements                      2.0 (2.8)       0-18
Instructor biographies             0.7 (l.0)       0-5
Links to university resources     0.06 (.27)       0-4

Table 2 Gender Comparison of Access to Web Site Pages

Content Page                         Mean          Mean
                                     Hits          Hits
                                      Per          Per
                                    Female         Male
                                    Student       Student

Course homepage *                 31.6 (22.0)   24.6 (16.4)
Schedule/Reading **                9.3 (6.8)     6.4 (5.8)
Answers to reading/
  lecture questions                5.0 (4.2)     4.7 (4.3)
Syllabus *                         4.1 (3.6)     2.8 (2.9)
Required assignment
  information                      3.6 (2.9)     3.4 (3.1)
Guest speaker information          2.5 (2.8)     3.2 (3.7)
Links to psychology-
  related Web sites                2.8 (2.6)     2.6 (2.4)
Announcements                      2.3 (3.1)     1.5 (2.0)
Instructor biographies             0.7 (1.0)     0.8 (1.1)
Links to university resources      0.07 (0.4)     .04 (.38)

Note: Standard deviation in parentheses

* p < .05

** p < .01

TTable 3
Mean Hits Per Student & Final Course Grade

Page                                        Passed        Failed
                                                         (N=133)
                                                          (N=21)

Course homepage                          30.0 (20.1)   22.3 (12.8)
Schedule/Reading *                        8.6 (6.6)     5.6 (5.3)
Answers to reading/lecture questions *    5.2 (4.3)     2.9 (2.8)
Syllabus                                  3.7 (3.5)     2.9 (2.4)
Required assignment information           3.5 (3.0)     3.8 (3.1)
Guest speaker information                 2.9 (3.3)     1.9 (2.3)
Links to psychology-related Web sites     2.7 (2.5)     2.8 (2.6)
Announcements                             2.1 (2.9)     1.2 (l.4)
Links to university resources              .06 (.40)     .05 (.22)
Instructor biographies                     .71 (1.03)    .57 (.93)

Note: Standard deviation in parentheses

* p <.05

** p <.01

References

Aberson. C.L., Berger, D.E., Healy, M. R., Kyle, D. J. & Romero, V.L. (2000). Evaluation of an Interactive Tutorial for Teaching the Central Limit Theorem. Teaching of Psychology, 27, 289-293.

Bailey, E. K., & Cotlar, M. (1994). Teaching via the Internet. Communication Education, 43, 184-193.

Carnevale, D. (2003, July 4). Study of Wisconsin professors finds drawbacks to course-management systems. The Chronicle of Higher Education, A26.

Cohen, S.H., Futoran, D., Thorn, C., & Karraker, K. (2000, August). Integrating the Psychology Major with a Web-Site. Poster presented at the annual meeting of the American Psychological Association: Washington, D.C.

Cohen, S.H. & Murray, L. (2000, November). Anatomy of a birth: Analyzing student utilization of a Web-assisted course on psychology as a profession. Paper presented at the meeting of the Society for Computers in Psychology, New Orleans, LA.

Couch, J.V. (1997). Using the Internet in instruction: Ahomepage for statistics. Psychological Reports, 81,999-1003.

Goldstein, M. D. (1998). Forming and testing implicit personality theories in cyberspace. Teaching of Psychology, 25, 216-220.

Goolkasian, P., Wallendael, L. V., & Gaultney. J. F. (2003). Evaluation of a Web site in cognitive science. Teaching of Psychology, 30, 266-269.

Henly, D.C. (2003). Use of Web-based formative assessment to support student learning in a metabolism/nutrition unit. European Journal of Dental Education, 7, 116-123.

Lawson, T. J. (2000). Teaching a social psychology course on the Web. Teaching of Psychology, 27, 285-288.

Rosen, E. F., & Petty, L. C. (1997). Using Internet resources in a research methods course. Behavior Research Methods, Instruments, & Computers, 29. 222-227.

McCollum, K. (1997, February 21). A professor divides his class in two to test the value of on-line instruction. Chronicle of Higher Education, p. A23.

Meyer, K. A. (2003). The Web's impact on student learning. THE Journal, 30,14-24.

Sherman, R. C. (1998). Using the World Wide Web to teach everyday applications of social psychology. Teaching of Psychology, 25, 212-216.

Stith, B. (2000). Web-enhanced lecture course scores big with students and faculty. THE Journal, 27, 20-25.

Vodanovich, S.J. & Piotrowski, C. (2001). Internet-Based Instruction: A National Survey of Psychology Faculty. Journal of Instructional Psychology, 28, 253-255.

Wang, A.Y. & Newlin, M. H. (2000). Characteristics of Students Who Enroll and Succeed in Psychology Web-Based Classes. Journal of Educational Psychology, 92, 137-144.

Waschull, S. B. (2001). The online delivery of psychology courses: Attrition, Perormance, and evaluation. Teaching of Psychology, 28, 143-147.

Note:

The authors thank JoNell Strough for her extremely helpful comments on an earlier version of this manuscript. Dara Acord and Lori Murray assisted with the recording and coding of WebCT tracking logs from preliminary versions of the Web site.

Michelle Heffner and Stanley H. Cohen, West Virginia University, Department of Psychology, 1220 Life Sciences Building, Morgantown, WV 26506-6040.

Correspondence concerning this article should be addressed to Stanley H. Cohen at [e-mail].