The academy on the web: preparing to evaluate online courses.

Abstract

Web-based teaching and learning require evaluation in order to assess the effectiveness of online curricula and courses in providing good learning. Although Web-based courses offer access and new opportunities for forming collaborative learning communities, they also present pitfalls related to course design, students, teachers, administrators, technology, technical support, and funding. This paper examines a wholistic case study model for evaluating online courses, and provides numerous questions for evaluators to consider before assessing Web-based teaching, and online courses in particular.

1. Introduction

During the past five years educators have seen extensive development of virtual classrooms. Cyber-courses sometimes mimic traditional lecture courses with posted lectures that students read online. Some traditional as well as virtual courses use interactive modules that encourage students to experience virtually what had previously been available only through physical encounter, e.g., the dissection of a frog http://curry.edschool.Virginia.EDU/go/frog. In addition, faculty members install online syllabi, create Web directories, use software to administer and score tests, and collaborate online.

All of these activities sound modern and exciting. But to some, computer mediated communication (CMC) presents daunting problems for teaching and learning. What, they ask, is the basis for "good learning" online? Are teaching and learning online really different than in a traditional classroom? How can a teacher know who is actually doing the assignments? Do online courses, or modules embedded in traditional courses, deliver comparable education to that delivered in a face-to-face (f2f) environment? Learners in both online and traditional classrooms may be exposed to the same information but learn it differently.

Skeptics frequently raise the possibility of incomplete transfer of knowledge through CMC (Ahmad, et al. 1985). In the traditional classroom a student can ask a question directly of the instructor. Online, and in the asynchronous situation of most Distance Education (DE) classes, the student typically an email message to the instructor or posts it to an electronic bulletin board. The instructor answers the query, presumably within 24 hours or so. But the teachable moment, that brief period when the f2f instructor's answer to a student's question immediately enlightens the student, as a later response would not, disappears. And if the response were prolonged, the teachable moment would most likely be diluted as the student loses interest, or the key relationships between the lecture materials and the student's question weaken. On the other hand, online in a prolonged response period, both student and teacher have time to question and answer more thoughtfully and thoroughly than may be possible in a traditional class. Case study evaluation can assess all of these issues because it approaches the scene from so many angles: observation, interview, surveys, data review, etc. The course, which becomes the "case," includes content, online functions, students, instructor, institution, and anything else connected to the course; all are "fair game" for case study evaluation.

Online course design also causes consternation among certain faculty who complain that they can't teach students they can't see. Or they complain that in an online seminar they cannot encourage critical and creative thought if all the members are not able to interact immediately. They also worry that an institution that encourages putting courses online threatens those instructors who are not "computer savvy"; they often fear that they will lose their jobs. In order to respond to such worrisome complaints, faculty, researchers, and institutions must conduct both summative (here's what we find) and formative (here's what we could do to improve)evaluations. Such studies can help all concerned to clarify the distinctions between traditional and online teaching and learning, to discover important causes for success and disappointment, and to help develop ways to improve instruction and provide good learning.

2. Evaluation of Online Courses: The Need

Colleges and universities are moving toward rapid, broad dissemination of information on the Web, and moving toward what they believe will be profitable business in online instruction. The changing instructional paradigm employs increasing amounts of technological skill and tools; so, it behooves teachers, researchers, and institutions to question and evaluate the effectiveness of computer mediated instruction. In particular, evaluators must assess the design, content, use, and success or failure of Web-based courses. Certainly these courses offer access to students who might not be able to take advantage of a residential college with traditional classes. And, asynchronous DE courses typically enable students to have around-the-clock access to the course, making international collaboration possible.

Yet, those who are establishing a Web presence with undergraduate and graduate courses wonder: Do these courses effectively deliver what Eastman and Zieghan (1995) call the

   ultimate aim of instructional computer-mediated communications ... to
   provide a good learning experience for students--one in which they master
   new knowledge and skills, critically examine assumptions and beliefs, and
   engage in an invigorating, collaborative quest for wisdom and a personal,
   holistic development along with other learners and the instructor ..." (59)

The need for rich formative and summative evaluation (Merriam 1984) of Web-based courses is wide and deep. Although scholars have provided a rubric for evaluating Web sites themselves, little has been provided in terms of evaluation methodology for online courses. This paper proposes a model for such an evaluation, with particular emphasis on DE asynchronous courses, and provides a series of research questions.

3. A Case Study Model for Evaluation

What at first may appear to be a straightforward evaluation of an online class, similar to an evaluation of a traditional class, can become a complex, puzzling process. Perhaps because online courses are relatively new phenomena, educational evaluators have not fully considered the use and effects of computer mediated teaching and learning. Faculty, too, seem to be more concerned with putting materials and courses online in an electronic, virtual environment than they are with the basic question about teaching: Does the course provide good learning (and what is meant by "good" learning)?

This paper proposes a case study evaluation process (Merriam 1988) for Web-based courses. In case study research, the evaluator typically utilizes statistical instruments (surveys, questionnaires, accounting for participation, etc.) (Miles and Huberman 1984). But the evaluator may also use the ethnographic techniques of interviews, data review (e.g., reading student and teacher text), and observation in order to obtain a full and unbiased picture of the online course and its ability to maximize learning (Goetz and LeCompte 1984). Although case studies seek answers to predetermined questions, they also enable "hidden" information, which the evaluator may not have considered, to emerge. This emergent information often holds the key to genuine understanding of the stakeholders and the situation.

3.1 Assessment tools

The evaluator needs to assess the course design and content, stakeholders, and as well as special computer-related factors. The types of hardware and software or the availability of technical assistance affect not only the design and content, but also the success or failure of Web-based courses. Andersen, et al. (1998) have proposed questions that prove useful for evaluating Web sites; some also prove useful in assessing overall online course design. In addition, survey instruments and interviews elicit student and teacher responses. Observations help the researcher determine how and why the design and content do or do not serve the purposes of the course, the teacher, the students, and the institution because the researcher can see, first hand, what goes on in the class. In a case study assessment, information (even information not overtly sought by the researcher) emerges to give researchers a more complete picture than quantitative, experimental methods can provide. Case studies enable explanations about why certain things happen-they are not only repons about what is going on.

3.2 Purpose

Consider the purpose of the course by answering the following four basic questions:

    1. What content learning should take place?

    2. What performance should students achieve?

    3. Who determines the curriculum and why? (What are the stakes?)

    4. What is the purpose for implementing the course on the Web? (why not
    teach it in a traditional "real" situation?)

To find answers, case study evaluators survey and interview students and teachers online, by telephone, by mail, or in person. The evaluator also observes how teachers and students use the Web site. Once those determinations have been made, the evaluator can ask whether the course satisfies its purpose, and if it does not, what purpose it does satisfy, if any.

3.3 Design

In a case study, evaluators consider the course designers, who may or may not be instructors. In mm, the designers must consider the rhetorical impact of their design. The researchers will consider student and instructor use through observation and interviews as well as by asking basic questions to discover whether the site persuades students to browse, navigate, and otherwise use the site easily, or is it so unattractive or confusing that users become frustrated or "click off" before anyone can tell who the students and instructors really are. Understanding those issues provides a basis for determining whether or not online instruction can provide comparable learning to traditional settings. In other words, the case study compares the virtual environment be compared to a real one through a series of observations, which in an online class, means much time devoted to lurking in the class. Depending on the parameters of the study, comparisons between matched real and virtual classes will provide a full understanding of these similarities and differences.

3.4 Functionality

Once the evaluator understands how the course functions, and for whom, questions arise about whether or not the design of the Web site satisfies the content and technological use goals of the course. Answers to surveys and interviews provide evaluative data about the Web site design and other aspects of the course. Counters on various Web pages enable the evaluator to determine quantity and quality of participation, much as an evaluator considers participation in a traditional setting. The evaluator can determine from answers and observed behaviors how well the design serves the users, as well as whether or not students are learning the course material.

Lurking in the online course or sitting in a real one are much the same for the evaluator. But in the online course, the evaluator also asks questions about software and hardware, linking to servers, efficiency of the course website (is navigation slow or fast?), and related questions.

3.5 Students

Evaluating Web-based courses must take institutional requirements, students' needs, and their desires to learn certain things into account. Some evaluation schema ask students which lessons or exercises were the most enjoyable, and they ask students to determine the quality of their online learning compared to learning in a traditional classroom (Bliss, 1998; Nourse 1998). Such questions, posed online or in interviews, encourage students to evaluate their own learning. These surveys can be completed anonymously, and the results triangulated with f2f or online interviews.

3.6 Instructors

A thorough evaluation considers the roles and needs of the instructor. The researcher will want to learn what kinds of authority an online teacher wants and needs. In a traditional classroom, a teacher establishes authority through demeanor, approach to the students, etc.; how does this process occur online? Interviews with instructors elicit information about setting authority. Follow up observation helps to determine whether or not the teacher actually follows practices identified in the interview. This observation can determine instructional variance between traditional and online teaching.

3.7 Other Stakeholders

Evaluating online courses also considers other stakeholders in addition to the students and teachers:

   * department, college, and university administrators;

   * the technology support system for the school and the Internet;

   * the supporting entities of the institution, such as the board of
   governors for a private school, or the government and taxpayers for public
   institutions.

These stakeholders play vital roles in the success or failure of online courses. In each case, the researcher looks for the importance and influence of issues such as institutional expectations, politics, control of the technology, and community development. Because obtaining unbiased information from administrators is important to the validity and reliability of the evaluation, evaluators must listen and observe carefully, as "hidden agendas" may emerge during interviews or observations. In addition, the evaluator must consider the data objectively and triangulate it with data from other sources so as not to be misled or biased.

3.8 Administration

Expectations held by administrators may conflict, e.g., the department chairperson may be fundamentally against online courses even though the university, promoting itself as a front-runner in computer mediated education, pushes departments to use technology. Or, the technology administrators may implement software on the institution's network or set up computer labs in ways not compatible with planned online courses. Because administrators at various levels may have different agendas, the evaluator must query carefully to learn about such issues as curriculum oversight, who will teach, delivery methods (listserv, bulletin board, or courseware tools). Who gets credit for successful online courses? Does a department credit itself for the work of an instructor even though the instructor may not have received any support from the department other than a "yes, you may offer an online course"? Does a department earn additional funding for putting courses online? Does the teacher receive copyright protection as well as contractual or financial support for developing and teaching online courses? These questions are basic ones that case study researchers can answer.

3.9 Technology Support

Recognizing that many instructors are not as technically savvy as they wish to be, the evaluator will want to observe and interview instructors about their computer skills. Deep understanding of these issues will help formulate plans for future developments.

   * Who pays for the technology support? If a department must pay for
   technical assistance for an online instructor, yet the department doesn't
   have a large budget, is the assistance refused?

   * What are the goals and operations of the central technology
   administration (if it exists)? Does it generally assist instructors and
   students, or does it seek to improve its own status through a few "show
   projects"?

3.10 Institutional Directors and Funders

In the end, an evaluator will typically realize that technology costs money, a lot of money, and money on a recurring basis. Building a computer lab, from scratch in an existing building, for 20 students and an instructor could easily cost $300,000, plus ongoing support. And, in three years or so, that lab will most likely need refurbishing, for another $100,000 or more. For DE courses, the university must pay for servers and technical staff, modems, and other ancillary costs that may take years to recoup.

The governing boards of institutions want to jump on the technology bandwagon because it is politically popular and expedient to do so. But, they do not like spending money for recurring expenses such as computer labs and servers to provide delivery. Evaluating the politics and funding issues can determine how and why they affect and are affected by those who control the institution. Sample questions might include these:

   * What are institutional expectations, goals, and agenda for development,
   instruction, and technology? Where do the students and teachers fit into
   these expectations?

   * What promises will be made, or what publicity will be put forth? Will
   instructors be brought before state legislatures or governing boards to
   testify about the success of technological teaching and CMC in order to
   obtain money?

4. Conclusion

Successful Web-based teaching and learning may be difficult to measure because most institutions and instructors are so busy implementing new technology that they have little time to consider the effects of their use. Thus, an outside evaluator must attempt to gain an insider's view of the situation in order to fully understand and objectively assess it. An inside evaluator, such as an instructor or administrator, is a participant and an immediate observer and must therefore guard against personal bias in the evaluation (Miles and Huberman 1984).

Using a wholistic case study design for evaluating online courses and other Web-based course materials will enable .the researcher to gain information. The depth and breadth of the data will assist in triangulating and verifying the findings and the conclusions. Using online questionnaires, online and f2f interviews, data review, and direct observations can facilitate an evaluation because respondents can conveniently answer asynchronously, and anonymously. These two features may encourage a high degree of insightful participation in the research.

In all evaluations, the researcher must guard the privacy of the participants. Evaluating online courses through Web-based evaluation instruments can supply that privacy through access codes and disappearing passwords. In addition, the researcher must guard against internal and external bias, so research instruments such as surveys and formal interview questions need to be carefully designed (Merriam 1988). With care, case study evaluations provide full, rich understanding of online courses.

References

Ahmad, Khurshid, Corbett, Creville, Rogers, Margaret, and Sussex, Roland. (1985). Computers, Language Learning and Language Teaching, Cambridge: Cambridge University Press.

Anderson, Daniel, Benjamin, Bret, and Paredes-Holt, Bill. (1998). Connections: A guide to on-line writing. Boston: Allyn & Bacon.

Bliss, Anne. (1998) Online course evaluation, http://www.colorado.edu/cewww. Boulder, Colorado: University of Colorado. (Restricted Access)

Eastman, Dan and Ziegahn, Linda. "Instructional Design for the Online Classroom" in Berge, Zane L. and Mauri P. Collins, 1995, Computer Mediated Communication, Vol. 111. Creskill, NJ: Hampton Press.

Frog Dissection Web site, http://curry.edschool. Virginia. EDU/go/frog. (February 24, 1999)

Goetz, Judith Preissle and LeCompte, Margaret Diane. (1984) Ethnography and Qualitative Design in Educational Research, New York: Academic Press, Inc.

Merriam, Sharon. (1988) Case Study Research in Education, San Francisco: Jossey-Bass Publishers.

Miles, Matthew B. and Huberman, A. Michael. (1984) Qualitative Data Analysis. Newbury Park and London: Sage Publications.

Nourse, Mary. 1998. Online course evaluation, University of Idaho at Boise. http://leahi.kcc.hawaii.edu/org/occ/logs/0059.html. (February 25, 1999)

Turkle, Sherry. (1995) Life on the Screen: Identity in the Age of Interact, New York: Simon and Schuster.

Dr. Bliss teaches writing courses in the arts and sciences, engineering, and business. She has developed and teaches two fully computer-mediated, Web-based distance education writing courses through the University of Colorado College of Continuing Education.