Quality assurance--best practices for assessing online programs.

Educators have long sought to define quality in education. With the proliferation of distance education and online learning powered by the Internet, the tasks required to assess the quality of online programs become even more challenging. To assist educators and institutions in search of quality assurance methods to continuously improve their distance education programs, the Sloan Consortium (Sloan-C) published Elements of Quality: The Sloan-C Framework (Moore, 2002), outlining five pillars of quality--learning effectiveness, access, student satisfaction, faculty satisfaction, and cost effectiveness for online programs. Based on a relevant literature review, this article explores the reasons behind the push for online program quality assurance, key benchmarks recommended by major accreditation agencies and some best practices currently utilized to ensure online program quality standard. It serves as a starting point for distance education administrators and educators to formulate program goals and assessment policies regarding their online programs.

Introduction

The Internet has created unprecedented opportunities for widespread electronic delivery of information and services. Education is becoming a ubiquitous service delivered over global networks with the promise of being accessible anytime and anywhere. With the growing popularity of distance education and online programs, compelling questions demand attention--How do we assess the quality of such a program under the new paradigm? What makes a cyber-college or online university worth attending (Witherspoon & Johnstone, 2001)?

To assist educators and institutions in search of quality assurance to continuously improve their online programs, the Sloan Consortium (Sloan-C) published Elements of Quality: The Sloan-C Framework (Moore, 2002), outlining a scaffold with five pillars of quality--learning effectiveness, access, student satisfaction, faculty satisfaction, and cost effectiveness. Based on a review of relevant literature, this article explores the reasons behind the push for online program quality assurance, key benchmarks recommended by major accreditation agencies, and some best practices currently implemented to ensure online program quality standards. It serves as a starting point for distance education administrators and educators to devise program goals and assessment policies regarding their online programs.

Need for Quality Assurance

The globalized information economy and its marketplace are propelling educators to reshape higher education around the world. Never has education been so important to so many. Governments, companies, and individuals all recognize that while an assembly-line worker is valuable, the real competitive advantage comes from a well-educated mind, producing breakthrough ideas that advance technologies and lead to new products, new initiatives, and ultimately a stronger society. As universities and businesses alike implement updated strategies they are redefining venue and pedagogy. Consequently, they must also redefine measures of quality (Witherspoon & Johnstone, 2001).

McLoughlin and Visser (2003) assert that educational quality assurance is a matter of accountability and national interest. Governments at the federal, state, and local levels mandate it. Accreditation agencies require it. The general public expects it, and faculty need it to support their actions (McKenzie, Mims & Bennett, 2003). In the U.S., the benchmarks for measuring educational quality have traditionally been provided by accreditation agencies at regional and national levels. Responding to the challenges that distance education presents under the Internet Age, major accreditors have made significant changes to their standards, policies, and procedures that serve as benchmarks, also known as best practices for quality online education. The most widely followed guidelines are provided by (1) the Council of Regional Accrediting Commissions (C-RAC, 2000); (2) the American Federation of Teachers (AFT, 2000); and (3) the Council for Higher Education Accreditation (CHEA, 2002).

Being consumers of educational services, individual students, their parents or employers also demand quality assurance as higher education is becoming a more consumer-driven market (Carnevale, 2000). The world is well along in its transition from an industrial age to an information society where knowledge is accumulating at an exponential rate, information technology has permeated into nearly every aspect of one's life, and information--its acquisition, management, and deployment is the key competitive advantage (Alva, 1999). The knowledge-based economy mandates workers to continuously improve and upgrade skills in order to process information and create new knowledge. Thus, lifelong learning and just-in-time learning is becoming progressively critical to employees seeking to upgrade their skills and to employers pursuing individuals with the necessary experience and education to help their organizations succeed. To remain viable in the competitive marketplace, these learners and their employers are able and willing to pay the price necessary to obtain the required qualification at the time and the place they need it. This potentially huge lifelong learning market has given rise to the commercialization of education as it is increasingly seen as another application of e-commerce. Indeed, competition is on the rise with the private sector (e.g., University of Phoenix and DeVry University) entering the education arena. Competition brings choices and choices empower consumers, who can now shop around for the best products and services at the most reasonable prices. McLoughlin and Visser (2003) point out that as today's students pay a greater share of their own educational costs, they expect universities to provide services they demand in the market at large: better service, lower price, higher quality, and a mix of products that satisfies their own sense of good education.

Echoing assertions by Fernandez and Lampikoski that client satisfaction is a good indicator of quality, Blfer (2000) claims that student satisfaction is a key criterion for institutions to determine quality in distance education. This idea is also affirmed in the statement that student satisfaction is the most important factor for continuing education as it reflects learners' evaluation of the quality of all aspects of the educational program (Moore, 2002).

Best Practices Supporting Quality Assurance

A number of organizations and accreditation agencies have developed sets of principles, guidelines and benchmarks in an effort to ensure quality online programs. An overview of several such best practices is presented in this section.

The five pillars constituting the Sloan-C Quality Framework (Moore, 2002) define the following elements:

1. Learning effectiveness (LE) -- demonstrates that learners who complete an online program receive educations that represent the distinctive quality of the institution. The goal is for online learning to be equivalent to or better than learning through the institution's other delivery modes (traditional face-to-face, classroom-based instruction). There are nine specific principles applicable to LE.

2. Access -- provides the means for all qualified and motivated students to complete courses, degrees or programs in their disciplines of choice. Access includes three areas of support: academic (e.g., tutoring, advising and library); administrative (e.g., financial aid and disability support); and technical (e.g., hardware reliability, uptime and help desk). There are twenty distinct principles associated with Access.

3. Student satisfaction (SS) -- reflects the effectiveness of all aspects of the educational experience. The goal is that all students who complete a course express satisfaction with course rigor and fairness, with professor and peer interaction, and with support services. There are five principles pertinent to SS.

4. Faculty satisfaction (FS) -- indicates that instructors find the online teaching experience personally rewarding and professionally beneficial. Personal factors contributing to FS include opportunities to extend interactive learning communities to new populations of students, to conduct and publish research related to online program modality, and to achieve recognition and collegiality. Institutional factors associated to FS include support, rewards, and institutional study/research. There are eight principles relevant to FS.

5. Cost effectiveness (CE) -- enables institutions to offer their best educational value to learners. Online programs are regionally accredited in the same manner as on-campus ones. The goal is to control costs so that tuition is affordable yet sufficient to meet development and maintenance costs, and to provide a return on investment in startup and infrastructure. There are four principles related to CE.

The Best Practices developed by C-RAC (2000) have outlined five components to address a particular area of institutional activity relevant to distance education.

1. Institutional context and commitment -- include consistency of the online program with the institution's role and mission; its budgetary and policy commitment; the adequacy of technical and physical plant facilities; reasonable technical support for faculty and students; appropriate internal organizational structure which enables the development, coordination, support, and oversight of the online program; and meeting the legal and regulatory requirements.

2. Curriculum and instruction -- constitute assurance that each program of study results in collegiate level learning outcomes appropriate to the rigor and breadth of the degree or certificate awarded by the institute; full participation of academically qualified persons in the decisions concerning program curricula and oversight; implementation of a coherent plan for students to access all courses necessary to complete the program; appropriate interaction between instructor and students and among students; and consortia or outsourcing standards.

3. Faculty support -- addresses issues of faculty workload; compensation; evaluation; ownership of intellectual property resulting from the program; on-going technical support and training; and support for course design and management.

4. Student support -- comprises institutional commitment (administrative, financial and technical) to a time period sufficient for all admitted students to complete the program; proper communication with students regarding enrollment standard, technical competency, curriculum design, time frame, cost, payment and refund policies; adequate services provided in the areas of library, bookstore, academic advising, financial aid, tutoring, career counseling and placement, and ongoing technical support and training.

5. Evaluation and assessment -- consist of documenting assessment of student achievement conducted in each course and at the completion of the program by comparing student performance to intended learning outcomes; assuring the integrity of student work; securing personal information in the conduct of assessment and dissemination of results; carrying out ongoing self-evaluations pertinent to program improvement, more effective use of technology to improve pedagogy, student achievement of intended outcomes, improved retention rates, effective use of resources, and enhanced services to its internal and external constituencies.

The Guideline for Good Practice advocated by AFT (2000) has recommended fourteen standards to gauge quality of distance education. Incorporated in the standards are: (1) faculty must maintain academic control; (2) faculty must be equipped to meet the special requirements of teaching at a distance; (3) course design should be shaped to the potentials of the medium; (4) students must be fully aware of course requirements and be prepared to succeed; (5) close personal interaction must be maintained; (6) class size should be set through normal faculty channels; (7) courses should cover all materials; (8) experimentation with a wide range of subjects should be encouraged; (9) equivalent research opportunities must be provided; (10) student assessments should be comparable; (11) equivalent advisement opportunities must be offered; (12) faculty should retain creative control over use and re-use of materials; (13) full undergraduate degree program should include same-time same-place coursework; and (14) evaluation of distance coursework should be undertaken at all levels.

Finally, the Accreditation and Assuring Quality in Distance Education by CHEA (2002) has distilled its standards focusing on seven fundamental elements of institutional operations important to assuring quality in distance learning. They consist of: (1) institution mission; (2) institutional organizational structure; (3) institutional resources; (4) curriculum and instruction; (5) faculty support; (6) student support; and (7) student learning outcomes. Similar benchmarks have been established for each element.

CHEA requires that institutions must document, as a process of program evaluation, that they are in fact meeting their educational missions and goals and their students' outcomes are at an acceptable level. This is applicable to both distance and campus-based learning.

Although these accreditation agencies vary in their benchmarks governing quality standards for online programs, they have uniformly emphasized the following elements in their guidelines: (1) strong institutional commitment; (2) adequate curriculum and instruction that fit the new delivery medium and match the rigor and breadth of equivalent on-campus programs; (3) sufficient faculty support; (4) ample student support; and (5) consistent learning outcome assessment. The following section presents some best practices that are research-supported and empirically tested as successful implementations of the essential quality elements discussed thus far.

Exemplary Quality Assurance in Practice

1. Administrative leadership and support

Lee and Dxiuban (2002) assert that the fundamental quality assurance strategy to support online programs for any institution is a functional administrative, technological, and organizational infrastructure. The University of Central Florida (UCF) offers web-based courses to its diverse, growing student population. UCF uses a centralized approach which deploys a technical infrastructure design that provides administrative leadership, structures faculty development, and assesses course delivery services.

The Office of Academic Affairs, through the Vice-Provost for Academic Programs, initiates service through several units. The Center for Distributed Learning (CDL) administers interactive television, video and online programs. CDL acts as an information clearinghouse to initiate administrative support and resolve accreditation issues created by transforming face-to-face programs to the virtual environment. Cavanaugh (2002) notes that UCF's budget reflects the importance of distance learning programs in meeting institutional goals. The university funds the technical infrastructure, faculty development, learner support, research and development in distance learning, and impact evaluation. The technical infrastructure is made up of wired and wireless network connections to every building, access to Internet, dedicated servers for online courses, and always-on access to information and services.

2. Faculty support

Corderoy, Stace, and Pennell (2002) cite best practices for faculty support at the University of Wollongong, Australia. As a support to instructors involved in the design, development, implementation, evaluation, and enhancement of subjects for flexible delivery, the university has provided, through its Centre for Educational Development and Interactive Resources (CEDIR), academic staff development, educational design support as well as production support for the development of teaching resources required. Resources to assist these processes have also been developed and made available at the Flexible Delivery website: http://cedir.uow.edu.au/CEDIR/services/serv_flexdel_01.html.

Cavanaugh (2002) finds that faculty development services at UCF are especially noteworthy. All distance teaching instructors take a faculty development course, for which they receive a stipend or a laptop computer. They are given release time and extra pay for course development and are assisted in all phases of course design and delivery by trained Tech Rangers. According to Corderoy, et al. (2002), the Faculty Center for Teaching and Learning (FCTL) at UCF offers a wide range of support services for faculty entering the web-based teaching environment. FCTL's centerpiece is a summer institute where faculties from across campus integrate to create innovative teaching approaches including online instruction. The Research Initiative for Teaching Effectiveness (RITE) is an internally funded unit that assists faculty research in effective teaching practices.

3. Student support

To support online students, UCF has built a full range of services delivered on the Web ranging from application for admission, course registration, course schedules, technical guides, online orientation, financial support information, to library access and textbook purchasing. 24/7 technical support is provided by phone or Web and a group of CyberKnights act as student computing consultants. A CD-ROM containing Internet software, a browser test, tutorials, and automatic configuration for network dial-up access is distributed at the time of a student registration. Impact evaluation at UCF focuses on student issues such as success rates, withdrawal rates, learning styles, attitudes, and demographics (Cavanaugh, 2002).

4. Curriculum design and instruction

Based on Corderoy, et al. (2002), to maintain curriculum continuity, ongoing evaluation of individual courses related to the overall program is conducted at UCF. Evaluation strategies utilize comments from outside reviewers, student inputs, faculty evaluations, current research relevant to online programs, and professional literature. Web course development at the university reflects a team approach involving subject matter experts, instructional designers, web programmers, and graphics artists. As a result, the end product encompasses the instructor's content choice, interaction, assessment, and other functional aspects. After a decision is made regarding either developing a new online course or converting an existing face-to-face course, the instructor usually conducts a comprehensive course analysis relative to the program focusing on course prerequisites, program sequencing, assessment instruments, instructional methodologies, and communication strategies.

5. Assessing student achievement

Indeed, Carnevale (2001) states that assessment is taking center stage as online educators experiment with new ways of teaching and proving that they are teaching effectively. Higher education is moving toward outcomes-based assessments with online education leading the way. Although assessment models vary in the literature, the current practices used in assessing student achievement in an online program are broadly divided into two paradigms: (1) associationist- and behaviorist-based assessment with scientific measurement, and (2) cognitive- and constructivist-based assessment with qualitative measurement (Shepard, 2000). The former emphasizes the learning outcome while the latter measures the interaction during the learning process.

Leading in the practice of utilizing standardized tests in the U.S. are the Western Governors University, the University of Phoenix Online, Excelsior College in Albany, NY, Pennsylvania State University's World Campus, Thomas Edison State College in Trenton, NJ, the State University of New York's Empire State College, and University of Maryland University College. These schools require students seeking a degree to show competency in a number of domains including general education (e.g., writing and mathematics) and subject specific knowledge (e.g., business management). Outlined competencies are met through passing of a series of assessment exams (Carnevale, 2001).

O'Reilly (2000) advocates alternative assessment or authentic assessment methodologies which are formative in nature. The best practices extracted from his research include: (a) group support for individual product ("Discrimination and the Law" project used at Macquarie University, Australia); (b) team collaboration for common product ("Science and Communication" website at University of Melbourne, Australia and "Environmental Psychology," a co-authored paper for publication at University of Pretoria, South Africa); (c) peer review plus self-review (assess students' own progress and the contribution of their peers); and (d) debate ("Communication and the Media" online debate internationally collaborated among University of South Australia, Governors State University, Chicago and University of Technology, Sydney).

CONCLUSION

With the enabling technology readily available and a growing demand for lifelong and just-in-time learning, online programs which free learners from the constraints of time and space and offer flexible learning opportunities are flourishing. At the same time, competition is also intensifying as traditional publicly funded universities, the new for-profit, post-secondary institutions, corporate universities, and training companies contend for "knowledge workers" who are ready and willing to pay for skill upgrade (Bates, 2000). Hence, insuring the success and validity of online programs lies in an institution's ability to deliver a high quality education service at a reasonable price.

The best practices cited in this article can serve as a baseline for institutions seeking quality assurance in the five key areas of (1) institutional commitment (e.g., administrative leadership and support, technical infrastructure, and budget priority); (2) curriculum and instructional development (e.g., team approach in course design, ongoing course evaluation, and applying online learning pedagogy); (3) faculty support (e.g., faculty development, ongoing technical support, and institutional rewards); (4) student support (e.g., full range of academic and administrative services, interaction with faculty and peers, and technical support); and (5) learning outcome assessment (e.g., learning outcome assessment (summative and aptitude) and learning process assessment (formative and authentic)).

Online education is incredibly dynamic and constantly driven by changes in demand and technology. Therefore, the author concurs with C-RAC's (2000) position, "Given the rapid pace of change in distance education, these Best Practices are necessarily a work in progress" (p. 1). As practitioners and theorists continue their efforts to explore new venues to assess quality of online programs, no doubt more examples of best practices will continue to emerge. Thus, this article serves only as a starting point for online program administrators and educators as they engage in further study and practice of quality assurance.

References

Alva, J. K. (1999). High education and its millennial predicament. The League for Innovation in the Community College Conference on Information Technology.

Bates, A. W. (2000). Distance education in dual mode higher education institutions: Challenges and changes. SOFF. Retrieved May 20, 2004 from http://bates.cstudies.ubc.ca/

Belfer, K. (2000). A learner centered assessment of quality for online education: Course climate. World Conference on Educational Multimedia, Hypermedia and Telecommunications 2000(1), 1265-1267.

Carnevale, D. (2000). Shopping for an online course? Kick the tires and check the mileage. The Chronicle of Higher Education, February 2.

Carnevale, D. (2001). Assessment takes center stage in online learning. The Chronicle of Higher Education, April 13.

Cavanaugh, C. (2002). Distance education quality: Success factors for resources, practices and results. In R. Discenza, C. Howard & K. Schenk (Eds.), The Design & Management of Effective Distance Learning Programs (pp. 171-189). Hershey, PA: Idea Group Publishing.

Corderoy, R. M., Stace, R., & Pennell, R. (2002). Quality assurance and online teaching and learning: First steps. Proceedings of the International Conference on Computers in Education, 619-620.

Lee, J., & Dziuban, C. (2002). Using quality assurance strategies for online programs. Educational Technology Review, 10(2), 69-78.

McKenzie, B., Mims, N., & Bennett, E. (2003). Successful online assessment, interaction and evaluation techniques. Society for Information Technology and Teacher Education International Conference 2003(1), 426-431.

McLoughlin, C., & Visser, T. (2003). Global perspectives on quality in online higher education. World Conference on Educational Multimedia, Hypermedia and Telecommunications 2003(1), 253-256.

Moore, J. C. (Ed.). (2002). Elements of Quality: The Sloan-C Framework. Needham, MA: Sloan Center for Online Education.

O'Reilly, M. (2000). Assessment of online interaction: Helping or hindering the goals of educators and learners? World Conference on Educational Multimedia, Hypermedia and Telecommunications 2000(1), 868-873.

Shepard, L. A. (2000). The role of assessment in a learning culture. Educational Researcher, 29(7), 4-14.

The American Federation of Teachers (2000). Distance education, guidelines for good practice. Retrieved May 20, 2004, from http://www.aft.org/higher_ed/downloadable/distance.pdf

The Council of Regional Accrediting Commissions (2000). Best practices for electronically offered degrees and certificate programs. Retrieved May 20, 2004, from http://www.ncacihe.org/resources/electronic_degrees/Best_Pract_DEd.pdf

The Council for Higher Education Accreditation (2002). Accreditation and Assuring Quality in Distance Learning. CHEA Monograph (1, Series 2002). Retrieved May 20, 2004, from http://www.chea.org/pdf/mono_1_accred_distance_02.pdf?pubID=246

Witherspoon, J. P., & Johnstone, S. M. (2001). Quality in online education results from revolution. Ed at a Distance Magazine and Ed Journal, 15(3). Retrieved February 20, 2004, from http://www.usdla.org/html/journal/MAR01_Issue/article01.html

QI WANG

Gallaudet University, USA

[e-mail]