Toward a quality assurance approach to e-learning courses.

Finding ways to assure the quality of e-learning is an important endeavor. This article identifies eight quality assurance strategies in use at the University of Houston-Clear Lake. The eight strategies are reviews of instructional design, web development, editing, usability and accessibility, maintainability, copyright, infrastructure impact, and content and rigor. The impact of each of these strategies is discussed as well as how the strategy has evolved during implementation and operation. The university's e-learning courses have achieved some measure of merit for quality within both their local audience as well as nationally and internationally. Much of this is due to the application of the quality assurance strategies shared in this article.

**********

Like the proverb about beauty, quality in education appears to be in the eye of the beholder. While quality always has been important to education, it has remained an elusive concept. In 1998, Bannan-Ritland, Harvey, and Milheim (p. 78) wrote that, in education, "there is obviously no widely accepted measure of quality." Several organizations, such as the American Council on Education (1996), the Institute for Higher Education Policy (2000), the American Federation of Teachers (2000), and the Council for Higher Education Accreditation (2002), have distributed documents of quality standards for e-learning. These attempts at defining quality e-learning illustrate both the importance of quality standards and the lack of a definitive quality assurance process which can be endorsed by all. Online course quality remains in much discussion.

WHAT IS A QUALITY COURSE?

When the University of Houston-Clear Lake (UHCL) embarked into e-learning, the administration was determined to establish a process to assure the production of quality online courses. They gathered stakeholders, including learners, faculty, administrators, representatives from industries that employ graduates, and the university community. In trying to define quality online courses, establish course standards, and develop a set of quality assurance strategies, the stakeholder committee discovered that perspectives widely differed in defining course quality, as shown in Table 1.

While a simple definition of a quality online course remained elusive, the stakeholders committee found common agreement around the idea that a quality online course would be the direct result of a course creation process that included quality assurance strategies.

THE UHCL COURSE PRODUCTION PROCESS

UHCL created a course production process based upon the team approach (Moore & Kearsley, 1996), where a variety of specialists--instructional designers, web developers, graphic artists, multimedia specialists, web programmers, and a project manager--work together with a faculty member. Using ideas from rapid prototyping (Tripp & Bichelmeyer, 1990; Wilson, Jonassen, & Cole, 1993) in league with traditional instructional systems design (Dick & Carey, 1990), the process centered around a learning theory-based instructional systems design model, as recommended by Alley and Jansak (2001). Details of UHCL's instructional design process are reported in Hirumi, Cook, Kidney, and Haggerty (2000).

Most of UHCL's courses are delivered through the WebCT courseware management system. A few courses are delivered through a "home grown" content management system based upon Oracle's Portal technology.

Zheng and Smaldino (2003) argued that the robust application of an instructional systems design process is one indicator of a quality course. To insure that UHCL's course production process qualifies, a set of quality assurance strategies is embedded within it.

QUALITY ASSURANCE STRATEGIES

UHCL's stakeholders' committee identified eight quality assurance strategies that fall into three general categories, as itemized in Table 2 (Kidney, 2001).

To coordinate the quality assurance strategies and perform the staff review noted in Table 2, the university added quality assurance evaluator positions to the Instructional Technology (IT) team. The following sections outline these eight quality assurance strategies, and describe the progress, successes, and failures in each area.

Team Review

Quite often, the process of designing instruction is so intense that those engaged lose sight of the overall course landscape through involvement in the details. Schon (1990) recommended periodic opportunities for "reflection in action" to combat this tendency. UHCL's first two quality assurance strategies are designed to insure plenty of opportunity for such reflection. They are the instructional plan review and the web design review.

[TABLE 2 OMITTED]

Instructional plan review. The instructional plan identifies the sequence of learning objectives and modules involved for a course, detailing the learning materials, readings, activities, quizzes, tutorials, exercises, and discussions. The plan also defines how each objective will be assessed. The plan review is the first milestone of UHCL's development process, usually occurring four to eight weeks into the work. The quality assurance evaluator facilitates a plan review meeting in which the instructional designer presents a summary of the plan to the senior instructional designer, project manager, faculty member, and management personnel who choose to attend. The review checks that the course has benefited from quality instructional design work, benchmarks course progress for the management group, and provides an opportunity to brainstorm new web-appropriate activities and to suggest changes and/or additions to the course.

Web design review. UHCL's second review milestone is the web design review, also called the "10% review" because approximately 10% of the course must be developed in order to conduct a meaningful review. In preparation for the review, a web developer and graphic artist collaborate to architect a prototype of the course based upon the instructional plan within the university's online course management system. A checklist of course standards, which can be accessed online at http://courses.uhcl.edu/ITCPOLICY/Operations.htm, specifies the elements to be reviewed. They include a link to the university's student support site, a link to the end-of-course evaluation, a working prototype of the course's navigation, links to the course tools, which will be used (bulletin boards, chat tools, and the like), a preliminary web style sheet, the course syllabus, at least one completed instructional module or unit, a course directory organization chart, and plans for any media, instructional graphics, and course applications requiring programming.

The web design review helps identify potential problems that might affect students or instructors or derail the project later. This review has helped UHCL's IT team overcome problems such as the following: (a) a grandiose page design that was clumsy and inconsistent in the course management system; (b) plans for instructional activities or resources that weren't feasible due to budget, time commitments, or technology; (c) attempts to program applications to replace functionalities that already were available within the course management system; (d) confusing, counterintuitive, or browser-specific navigational schemes; and (e) poor or confusing instructions for students.

The web design review may seem fundamental, but UHCL has found it to be enormously important in shaping quality online courses. The resulting benefits cascade to a much more effective future use of resources in course development, delivery, maintenance, and user support than would have been the case had the identified issues not been halted and re-directed.

Additional benefits. The instructional plan and web design review milestones have yielded at least four additional benefits that support course quality. First, they help the project manager keep close track of course progress in the early phases of production. Second, they often indicate needs for professional development opportunities to improve the skill sets of one or more team members. Third, they establish a set of checks and balances across the dynamics of the team members' specialties and biases, preventing a single specialty, such as the extensive use of embedded video, from becoming dominant, self-important, or a single point of failure. Fourth, in the process of these reviews, strengths and weaknesses of the existing standards and policies come to light and subsequently can be adjusted.

Staff Review

Yeung (2001) pointed out that academic staff can play an important role in quality assurance. At UHCL, five of the quality assurance strategies are performed by staff. Together, these five strategies compose a third milestone and happen when a course is nearly complete.

The project team prints out all course web pages and materials and turns over both the print and online versions for total review by a quality assurance evaluator. Looking at the course from an online student's perspective, the evaluator notes all problems and suggestions on the printed version. During this review, the project team is "hands off" the course, meaning that no changes are allowed, letting the reviewer work with a temporarily static project. The reviewer is responsible for editing the course; checking the usability, accessibility, functionality, and maintainability of the course; taking reasonable precautions to insure that the course does not infringe on copyright, and assessing the course's impact on the university's infrastructure.

Editing. Realizing that not all faculty members write polished, professional prose, the university invested in a staff proofreader and editor so that each completed e-learning course would be reasonably free from spelling, punctuation, and grammar errors, as well as from inappropriate language, style, and usage. This was because the university recognized the Web as a publishing environment requiring the same care and attention to detail as would be used for a journal article or book. The editing helps present a positive, professional image, modeling the professionalism in writing and presentation that is expected of students.

The editor traverses every segment of a course, identifying language errors and questionable elements. While minor problems are addressed when they are found, any change that seems substantive or having potential impact is referred back to the production team and faculty member for consultation.

Every course has benefited from editing, no matter how well written or designed. While some benefit only modestly, others have avoided potential embarrassments.

Usability and accessibility standards. "The power of the Web is in its universality. Access by everyone regardless of disability is an essential aspect" (Berners-Lee as cited in Brewer, 2003). UHCL's courses are expected to be usable for students, and must not present barriers to inclusive student participation.

Originally, UHCL used usability testing methodology to validate and verify the usability and accessibility of e-learning courses (Neilsen, 1994). A course quality evaluator assumed the role of student and "took" the course, noting the ease with which she/he could navigate, find, and understand course information, and interact within the course. The W3C standard accessibility checklist (Chisholm, Vanderheiden, & Jacobs, 1999) was used to verify such accommodations as descriptive alt tags to describe graphics, text transcripts for audio elements, links with descriptive names, and so forth. To comply with the more subjective standards, such as "clear and thorough" instructions, the team learned that just placing online a faculty member's instructions from a previously taught face-to-face class often falls short because the faculty often offered extemporaneous additions and verbal explanations during class that were important to student success. Time spent in polishing student assignments and directions avoided much student confusion, explanatory correspondence, and misunderstood assignments as a result of this review.

Shortly, the team decided that the "after-the-fact" usability testing was insufficient and ineffective. The process improvement decision, as per Garrett's (2002) recommendation, was to apply the concept of user-centered design from the beginning of a course's production. According to Katz-Haas (1998), user-centered design is both a philosophy and a process. As a philosophy, it places the person, rather than the object being designed, as the focus. As a process, it focuses on such cognitive factors as perception, memory, learning, problem solving, and so forth as they come into play during peoples' interaction with the object being designed. It requires an understanding of the prospective users' goals and expectations, their needed functionality of a site, their subject matter, abilities, skills, tools, access environment, and so on.

Strengthening the user-centered design knowledge and skills of the instructional designers and especially of the web developers required professional development in learning how to apply usability and accessibility concepts as a fundamental part of design and development. The IT team held meetings devoted to the philosophy and terminology of user-centered design, including a series of in-house workshops that allowed peer tutoring on these concepts as well as off-site workshops that provided perspectives of experts in the field.

Overall, the application of user-centered design coupled with usability and accessibility testing has strengthened UHCL's course significantly. Like a ramp that originally was built to accommodate a wheelchair, but now serves so many others, attention to online course accessibility and usability can benefit all users.

Maintainability. An e-learning course should be designed and developed with an eye to the need for maintenance and controlling its complexity. An e-learning course can represent a sizeable resource investment; therefore, institutions typically expect to re-use it over multiple semesters. For these reasons, the ease with which a course can be maintained and updated is important.

The necessity of course modifications should be anticipated with each semester. On the simple side, an instructor needs to change contact information or virtual office hours, which might require only a simple edit. More complex changes, like a new textbook, might require new page numbers for reading assignments that are scattered throughout the course. The nature of the Web itself causes a certain amount of course change as new information appears and old links go dead. The ease with which course changes can be accommodated can be affected significantly, for better or worse, by the original course design.

Sensitivity to maintenance issues, when applied early in the design and development of an e-learning course, is a worthy investment in both course quality and course longevity. The team learned that supporting maintainability of e-Learning courses includes the following:

* Using the built-in tools of the course delivery platform, rather than trying to custom program alternatives or workarounds.

* Using validation methods, such as W3C and Bobby, early in the design and development phase.

* Using unit template pages consistently to guide students through each lesson so that changing the lesson sequence or adding new lessons are accomplished easily.

* Using a single, external style sheet to control all page-layout and minimize the need for attributes in HTML tags so that style changes can cascade throughout the course with one central edit.

At UHCL, the IT Team learned the hard way that maintaining a course over successive semesters can require a greater resource investment than was spent in developing it originally. Software engineers have known that for years (IEEE, 2002), designing software with version control, scheduled and staged updates, and new releases, and error checking code. A web-course is really instruction in the guise of computer software, possessing many of the attributes of the latter. The IT Team now understands that designing and developing an e-learning course is just as much software engineering as it is instructional development.

Copyright. The proper use of others' works within an e-learning course is a legal and ethical issue. Since the Web represents a publishing paradigm, the institution offering an e-learning course has a potential liability for works used without permission. UHCL's stakeholders hoped to insure that faculty would refrain from using anyone else's material without written permission and, when permission was obtained, that a system to organize and monitor compliance with the terms and conditions was available. Inclusion of copyright concerns in the staff review was considered to be part of the university's obligation, according to the Technology, Education, and Copyright Harmonization (TEACH) Act of 2002.

While it would be impossible for the quality assurance evaluator to eliminate the chances of every possible copyright infringement, the IT team has found that some elements of e-learning courses appear suspect in the normal review process. When that occurs, the team researches to locate the original sources, trying web searches on key phrases or using tools like turnitin.com, a web subscription service most often used to detect plagiarism in student writing. If a potential infringement is located, the instructional designer or project manager tactfully coaches the instructor toward an appropriate resolution. Successful coaching strategies include:

* Linking to an article rather than copying it from an external web site and including course-embedded instructions directing the students on how to make use of the material.

* Creating new, original media that demonstrate or teach the same concept.

* Using the "fair use" doctrine and TEACH act limitations to bound the amount of copyrighted work included in the course.

* Seeking permission for reuse of the copyrighted material.

* Including the copyrighted material in a course pack or library e-reserve service.

Usually, one of these coaching alternatives resolves the problem. When it does not, the appropriate administrators are advised. The university has refused to offer an online class containing copyright infringement until permission has been secured or the offending material was removed.

Should permission to re-use copyrighted material be obtained, the original plan envisioned tracking the negotiated terms and conditions in a database. The database would serve to organize copyright permissions, to document compliance for legal purposes, and to automate a cue for action when permissions expire or need renewal. So far, UHCL has not needed to resort to such a complex system. None of the modest number of permissions obtained to date requires expiration or renewal; therefore, storing the permission documents in the binder with the course's documentation has proved satisfactory.

Infrastructure impact. Reviewing a course's impact on the institution's technological infrastructure insures that the infrastructure can provide a stable and sufficiently capacious network, including servers and related services. A knowledgeable staff reviewer can judge the infrastructure impact from the instructional plan and web design reviews, paying particular attention to course assignments, the means of assignment submission, and related bandwidth issues. The review anticipates mismatches, such as the graphic arts professor who instructs students to render six high-resolution views of a work of art and send them as e-mail attachments when the institutional mail server allocates only 20 megabytes of space per faculty user. When mismatches are found, the instructional designer might work with the instructor to find alternative turn-in methods or with the mail system administrator to increase the quota for this instructor. In one course, for which the final project is a five-minute video, the infrastructure review led to students turning in their final project on CD or DVD through the U.S. mail rather than the course server's student presentation area.

The technological review also takes into account the probable bandwidth of most students. When an assigned reading is an 80-page PDF file, success will be compromised for students on dial-in connections. When a case study requires watching a 20-minute video, at this time it is more prudent to provide it to the students on CD rather than streaming it from the course server. The need for this review does not diminish as more students have high-speed connections in the home because the demand to push more high-bandwidth content through the net grows faster than the net's ability to keep up.

UHCL is now learning that such a review also must analyze the collective impact of the growing total number of courses. Planning for network and server acquisition needs to factor in the potentially limitless capacity demands expected by instructors and students. Proactively assessing this impact is not an easy task. Monitoring bandwidth used and server performance statistics gives a snapshot of what exists, but measures of what the faculty are planning to do are less quantifiable and more problematic.

Peer Review

UHCL's final quality assurance strategy is peer review of content, rigor, and androgogy. A course with timely, accurate, and complete information is fundamental to learning. Maintaining the academic rigor of a course is fundamental to certifications and accreditations. While appropriate application of learning and assessment theory is an integral part of the instructional design process, a peer review of how well those linkages were crafted is always worthwhile. To insure these, the stakeholders proposed a peer review process for online courses. Noting the Web's publishing-like environment, the group felt that a peer review process philosophically would be like the blind review of journal articles. Efforts to develop similar peer review processes for online courses were underway elsewhere (Ashkeboussi, 2001; Swift, 2006).

Content, rigor, and androgogy. The peer review process required one or more experts, who could be from within the university community or outside the institution, to review the discipline aspects of the online course. These peer reviewers would be selected and contracted by the corresponding faculty Program Coordinator or, should the Program Coordinator's course require review, by the corresponding Department Chair. Oftentimes, suggestions for peer reviewers would come from the faculty developer. This reviewer selection model has similarities to the one used for faculty-identified tenure reviewers at UHCL. The review included ratings of the course's accuracy, completeness, scope, prerequisites, objectives, fit within the specific degree program, and level of rigor as documented on a checklist.

A white paper documenting the process, as well as drafts of the ratings checklist, which are available online at http://courses.uhcl.edu/ITCPOLI-CY/Operations.htm, was presented for feedback to the Academic Associates Council (primarily the Associate Deans), one of the shared-governance committees of the university. When the materials were communicated to key involved faculty, objections to peer review were voiced strongly. The faculty made it clear that they did not accept even the idea of peer review, much less the proposal. Their main arguments and possible strategies to overcome them are summarized in Table 3.

Due to this faculty resistance, UHCL has been unable to implement the peer review quality assurance strategy.

DISAPPOINTMENTS

Obviously, the inability to implement the peer review process remains a disappointment. In addition, there are three other areas of disappointment.

First, some faculty members teach with their online course before it has been reviewed by quality assurance. This often happens when semester openings loom and the course is already in the class schedule but has fallen behind in the production process. Some faculty opt to teach it as a web-assisted course and bypass quality review.

Second, some online courses get developed through informal processes and miss the quality review phase. This most often happens with courses that are slowly but incrementally put onto the Web; that is, where faculty add material and web activities to courses a little at a time over several semesters without the benefit of user-centered design considerations and thoughts for maintainability. This environment is often called "blended learning" and combines online and face-to-face approaches. As faculty understand quality assurance processes, see the value added by quality review, and participate in quality-oriented seminars and workshops such as the University of Houston's Campusnet Online Workshop (Kidney, 2004), more of these courses probably will be submitted for quality review.

Third, a few faculty members fail to see the value of the quality assurance process. They actually refuse to authorize any changes to their courses. While a faculty member's refusal to participate in quality review is probably not generalizable, the rationale appears to cluster around three phenomena:

1. Submitting to quality review increases a faculty member's workload and the effort to strengthen the online materials was viewed as a very low priority among other time demands.

2. Some faculty abhor the details, wanting to let the students "debug" the course and respond with changes to their complaints. Unfortunately, students do not feel so empowered or inclined--they tend to deal with the frustration caused by the bug and work to get it behind them where it is completely forgotten.

3. Despite the qualifications of the reviewing staff, some of whom are experienced faculty members with doctoral degrees, some faculty still feel that staff are beneath them and their work should not be subjected to a staff member's unenlightened opinions.

For many faculty members, the idea that others would review their course represents a new, unwelcome, and emotionally loaded terrain. As noted in Table 3, for across-the-board compliance, the administration must decide whether supporting a quality student experience and enhancing the institution's reputation is worth the pain--and even risk--of imposing mandatory quality assurance policies.

CONCLUSION

UHCL's students appear to appreciate the quality embedded in the university's online courses. For the Spring 2005 semester, 3,295 out of 7,561 students (43.6% of unduplicated head count) are enrolled in one or more online or web-assisted courses.

End of course evaluations measuring student satisfaction with the content, motivational techniques, technology, navigation and design, interactivity, assessment methodology, and use of media in each course report high student satisfaction. Six years of accumulated evaluation data are shown in Table 4.

While a direct correlation between UHCL's quality assurance processes and this student satisfaction data never has been subjected to study, courses that were not quality assured, usually due to one of the "disappointments" expressed, average one or more standard deviations below the means shown in Table 4.

UHCL's quality-assured courses have achieved some measure of acclaim. Three courses have been selected by WebCT's Exemplary Course Project (http://www.webct.com/exemplary/), one of UHCL's faculty course designers was named Texas Distance Learning Association's educator of the year in 2003, and library learning objects from a course were given the Library Instruction Project of the Year award for 2005 by the Texas Library Association.

These quality assurance strategies have strengthened online learning at the University of Houston-Clear Lake. Though quality is one of those things that only can be pursued and never attained, a prescribed quality assurance process has helped this university focus its efforts and place quality as a priority.

References

Alley, L., & Jansak, K. (2001, Winter). The ten keys to quality assurance and assessment in online learning. Journal of Interactive Instruction Development, 13(3), 3-18.

American Council on Education. (1996). Guiding principles for distance learning in a learning society. Washington, DC: ACE. Retrieved October 3, 2006, from http://itc.utk.edu/~jklittle/pensacola/principles.html 

American Federation of Teachers. (2000). Distance education: Guidelines for good practice. Washington, DC: AFT (AFL-CIO) Retrieved October 3, 2006, from http://www.aft.org/pubsreports/higher_ed/distance.pdf 

Ashkeboussi, R. (2001, November). A peer review process model for web-based courses. Paper presented at the Seventh Sloan-C International Conference on Online Learning, Orlando, FL.

Bannan-Ritland, B., Harvey, D. M., & Milheim, W. D. (1998, June). A general framework for the development of web-based instruction. Educational Media International, 35(2), 77-80.

Brewer, J. (2003). Web accessibility initiative. W3C. Retrieved October 3, 2006, from http://www.w3.org/WAI/

Chisholm, W., Vanderheiden, G., & Jacobs, I. (1999). Web content accessibility guidelines. Retrieved October 3, 2006, from http://www.w3.org/TR/WAI-WEBCONTENT/

Council for Higher Education Accreditation. (2002). Accreditation and assuring quality in distance learning. Washington, DC: CHEA. Retrieved October 3, 2006, from http://www.chea.org/Research/Accred-Distance-5-9-02.pdf?pubID=246

Dick, W., & Carey, L. M. (1990). The systematic design of instruction (3rd ed.). New York: Scott Foresman.

Garrett, J.J. (2002). The elements of user experience: User-centered design for the web. Indianapolis, IN: New Riders.

Hirumi, A., Cook, R., Kidney, G., & Haggerty, B. (2000, February). The systemic and systematic design of an on-line degree and certificate programs. Paper presented at the Annual Conference of the Association for Educational Communications and Technology, Long Beach, CA.

Institute for Higher Education Policy. (2000). Quality on the line: Benchmarks for success in internet-based distance education. Washington, DC: IHEP.

IEEE. (2002). IEEE Standards 2001-2002: Web page engineering, web site management and web site life cycle. Washington, DC: IEEE.

Katz-Haas, R. (1998, July). User-centered design and web development. Retrieved October 3, 2006, from http://www.stcsig.org/usability/topics/articles/ucd%20_web_devel.html

Kidney, G. (2001). An evaluation model for web-based instruction and a case study of its application. Doctoral dissertation, University of Houston, TX. (UMI No. AAT 3004078)

Kidney, G. (2004, June). When the cows come home: A proven path of professional development for faculty pursuing e-learning. T.H.E. Journal, 31(11), 12-20.

Moore, M. G., & Kearsley, G. (1996). Distance education: A systems view. Belmont, CA: Wadsworth.

Nielsen, J. (1994). Usability engineering. San Francisco: Morgan Kaufmann.

Schon, D. (1990). The design process. In V. A. Howard (Ed.), Varieties of thinking: Essays from Harvard's philosophy of education research center (pp. 110-141), New York: Routledge.

Swift, K. (2006). About the peer review process. Retrieved October 3, 2006, from http://taste.merlot.org/peerreviewprocess.html

Thompson, M. (2004). Faculty self-study research project: Examining the online workload. Journal of Asynchronous Learning Networks, 8(8), 84-88.

Tripp, S. D., & Bichelmeyer, B. (1990). Rapid protoyping: An alternative instructional design strategy. Educational Technology, Research and Development, 38(1), 31-44.

Wilson, B., Jonassen, D., & Cole, P. (1993). Cognitive approaches to instructional design. In G. M. Piskurich (Ed.), The ASTD handbook of instructional technology (pp. 21.1-21.22). New York: McGraw-Hill.

Yeung, D. (2001, Winter). Toward an effective quality assurance model of web-based learning: The perspectives of academic staff. Online Journal of Distance Learning Administration, 4(4). Retrieved October 3, 2006, from http://www.westga.edu/%7Edistance/ojdla/winter44/yeung44.html

Zheng, L., & Smaldino, S. (2003). Key instructional design elements for distance education. The Quarterly Review of Distance Education, 4(2), 153-166.

Note

Gary Kidney originally participated in writing this article while affiliated with the University of Houston-Clear Lake, but is now affiliated with the University of Maryland-University College.

GARY KIDNEY, LESLIE CUMMINGS, AND AZALEA BOEHM

University of Houston-Clear Lake, USA

[e-mail]

[e-mail]

[e-mail]

Table 1 Matrix of Quality Attributes

Group       Learners                    Faculty

Attributes  * Easy accessibility        * Easy to teach
of quality  * Good usability            * Intuitive course management
            * Accurate & thorough       * Customizable
              instructions              * Consistent with information
            * Intuitive navigation        they deem important
            * Well-integrated tools     * Quick preparation for
            * Consistent behavior         semester after semester
            * Correctly working links,  * Easy to update and add new
              materials, & media          information

Group       Administration

Attributes  * Comparable rigor to a nondistance class
of quality  * Accurate & valid information
            * Boosting enrollments
            * Free from copyright violations
            * Free from problems that might yield institutional
              liability
            * Uniform & reasonable efforts
              required to teach & maintain
            * Enhancing to the university's
              reputation

Table 3 Peer Review Arguments and Strategies

                                               Possible strategy to
Argument Peer review:  Elaboration             overcome arguments

Is not consistent      Faculty believe that    Focus the peer review on
with academic          what they teach and do  applicable accreditation
freedom                not teach are           standards and student
                       protected behind this   learning outcomes.
                       shield
Would not be parallel  Faculty argued that     a. Argue that creating a
with the lack of       the publishing             web course is the
review for face-       metaphor did not fit       equivalent of
to-face classes        online teaching and        publishing a textbook
                       that face-to-face          and texts and
                       classes were a better      carefully peer
                       parallel. They             reviewed.
                       insisted that online    b. Administrators simply
                       materials not be peer      may have to impose
                       reviewed until face-       some difficult
                       to-face lectures also      policies in support
                       were reviewed.             of the student's
                                                  experience and the
                                                  institution's
                                                  reputation.
Would be cumbersome    Faculty complained of   Specify and bound the
to execute             the difficulties in     reviewers role. Identify
                       identifying and         and involve them early in
                       compensating an         the course creation
                       appropriate reviewer,   process.
                       the unwieldy lead-
                       time needed for a
                       review, and the
                       potential necessity of
                       responding to a
                       reviewer's concerns.
Would require too      Creation of an online   a. Note that one recent
much publishing-like   course is generally        study argues that
care without a reward  not weighed as heavily     online courses do NOT
for the time  spent    as a refereed,             require more time--
in "academic credit"   scholarly publication      even in delivery
                       or other parallel          (Thompson, 2004).
                       creative activity in    b. If faculty rely on
                       merit, promotion, and      publisher's e-packs,
                       tenure decisions. Some     the development
                       argued that a peer         workload is very
                       review process would       modest.
                       require them to invest  c. A well-developed
                       inordinate effort in       course amortizes the
                       teaching that is not       time-consuming work
                       rewarded accordingly       over its use in
                       and threatened that        subsequent semesters.
                       this effort could pull  d. Some academic units
                       them away from             provide course
                       research.                  release, extra pay, or
                                                  other acknowledgement
                                                  in anticipation of the
                                                  extra time and effort.

Table 4 Student Satisfaction with Online Courses

Satisfaction category    Mean score

Content                  4.06
Motivational techniques  4.10
Technology               4.29
Navigation & design      4.21
Interactivity            3.92
Assessment               3.98
Use of media             3.93
Overall                  4.13

Note: 5 point Likert scale, 5 = "extremely satisfied"