Assessment of learning outcomes in an online environment.

Abstract

This paper looks at the challenge that online instructors face to assess the actual learning outcomes of their students. While it may be possible to simply use lessons learned in instructional design and institutional support to create preexisting conditions that should lead to a satisfactory learning experience, the nature of non- face-to-face instruction places more emphasis on the student to be an active participant in the learning process, so necessarily it is the process of learning that should be emphasized. The online environment presents us with new opportunities to deliberately influence and assess student achievement throughout the duration of the course. One way to do this is by designing courses with a specific pedagogical theory in mind (i.e., constructivism, transformative learning, etc.), and then creating activities that try to answer certain Outcome Questions. The result is a rubric through which to view each participant's online `behavior' and the degree to which they learn not only the course content but also are successful in the learning process.

Introduction

It can be argued that simply connecting to the WWW is a learning experience in itself, as there are immediately infinite sources of information available for consumption. However, the online environment is increasingly being accepted as a viable method of not just content delivery, but of formal instruction. Online learning in this case is synonymous with "web-based," "Internet" or "e-" learning, and refers to educational programs that are delivered via the Internet, or World Wide Web (WWW), using a graphic user interface. In particular, it refers to the deliberate use of the Internet to provide structured learning environments that can be accessed by many different people, independently, from different locations. Given the increase in educational opportunities available through via the Internet, it is essential that valid methods for assessment of student achievement be determined by the instructor and/or institution, and communicated to the learner at the outset of the program. Assessment of student performance in the online environment therefore requires an awareness of instructional design and pedagogical features that influence the desired outcomes. Additionally, opportunities for assessment should be built into the design of the course.

As with traditional modes of instruction, evaluation can occur on many different levels, by different people for different reasons. It is helpful to make a distinction between evaluation and assessment which are often used synonymously (Calder, 1994). Assessment in this context refers only to student (or teacher) performance, whereas evaluation looks at the context of organizational structures within which student performance occurs. Instructors and students assess each other, but they evaluate course content and institutional support. Administrators assess the performance of instructors, but instructors may also evaluate the program policies or design.

There are also different reasons to conduct an evaluation. Administrators might evaluate the viability of distance learning as a means to increase revenue or visibility. They may also use summative approaches to identify program costs/revenue, student demographics and completion rates. Instructors and students engage in mutual formative evaluation throughout the duration of the course in order to modify activities and improve outcomes. External evaluators are often used to look back at a program in its entirety to make recommendations about future directions or improvement. Above all, outside investigators from the government, external consultants, and even parents also scrutinize distance learning programs to evaluate if they are achieving adequate results compared to their traditional campus-based counterparts. Indeed, actually determining the learning outcomes of the students is often one of the most difficult aspects of online learning. Program design and course design necessarily influence learning outcomes and it is possible to predict programmatic interventions (i.e., availability of digital library access) or course designs (i.e., various course platforms or cognitive tools) that will enhance the learning process. However, measuring the degree to which students are learning the course material (and perhaps other unintended lessons) is a new challenge for the online instructor. Simply testing students using traditional methods such as multiple choice or short answer responses may not be adequate, especially since it is nearly impossible to prove that the person submitting the responses is the same person receiving credit for the course.

This article is written with both evaluation and assessment in mind, but it specifically suggests a methodology with which to judge the ability of online instruction to achieve positive learner outcomes. By using existing pedagogical theories in the design of online courses, instructors can look for specific demonstrations of learning acquisition.

Models of Distance Learning Evaluation

Various authors have put forth macro-models of evaluation in distance learning which include everything from strategic planning of the organization to learner expectations. Van Slyke et al. (as cited in Berlanger, 2000) categorize distance learning as a product of determinants of success (precedent variables) and evaluation criteria (outcome variables). The determinants of success are (i) institutional characteristics, such as technology infrastructure and objectives; (ii) learner characteristics, including expectations and computer proficiency; (iii) course characteristics, or course design and (iv) distance learning characteristics, which refer to the type of technology and environment. The outcomes are evaluated as either learner outcomes or institutional outcomes. These variables-precedent and outcome-are used formatively to determine the appropriateness of the use of distance learning methods. However, they don't really get at the heart of student assessment procedures, nor do they adequately express the interconnected relationships among the precedent variables. Another way that researchers have tried to predict the variables for success is by correlating personality traits or demographic information such as age, gender, marital status to achievement (Navarro and Shoemaker, 2000). Such studies might be useful comparisons for evaluation by measuring success against a predictive norm, but the correlations are arguably tenuous, and undervalue the importance of the course design and learning environment.

On the one hand, it is logical to assume that if all of the positive predictive variables (such as those mentioned above) are in place then students will learn, for benchmarks such as these were no doubt be based on research comparing high-achieving students against low-achieving students. On the other hand, because of the newness of online learning, there are still many who argue that there is little research and virtually no theory on appropriate methods for assessment of the actual learning taking place in the online environment. For example, Lockee et al. (1999) argue that traditional approaches to evaluating the effectiveness of distance learning have been media comparisons, in which the delivery medium is the independent variable and student achievement is the dependant variable. In general, the control group is the traditional face-to-face class whose outcomes are compared with the experimental group learning the same material but through a different medium of delivery and, in some cases, the same instructional method. The debate is often whether the instructional method or the delivery medium has more impact on student outcomes. Lockee et al. argue that these comparisons usually result in "no significant difference" because they are almost always trying to demonstrate the effectiveness of the new medium, and so they are therefore invalid because of their bias. Navarro and Shoemaker (2000) agree that most research up to now has yielded few important results, but they add that most of the studies have compared older learning technologies such as drill and practice computer applications. Few have compared "newer, multimedia cyberlearning" formats with traditional ones.

The problem with a lack of theory in media-related educational research is not a new one. Bates (as cited in Morgan et al., 1980) described this problem in a research proposal for evaluation of educational television:

    "A major reason for the lack of progress in media research has been the
    lack of an adequate theoretical framework. Much of the
    laboratory-controlled experimental research in this field could best be
    described as rampant empiricism - a comparison of one medium of teaching
    with another medium of teaching, without any attempt to place the
    experiment in any theoretical framework about how students learn, the
    conditions which affect learning on the pedagogical aims and philosophy
    behind the teaching (p. 8)."

Therefore, a new paradigm of assessing student achievement in the online environment must emphasize learner behavior, using qualitative methods based on proven pedagogical theories and cognitive processes.

Student Assessment in the Online Environment

Evaluation of any educational program can be difficult because of the many components involved. As Stake (1967) points out: "Educational programs are characterized by their purposes, their content, their environments, their methods, and the changes they bring about ... Evaluation is complex because each of the many characteristics requires separate attention (p. 4-5)."

Part of the difficulty that online teachers face in assessing learning outcomes is that they lack many of the unintentional or spontaneous indicators of progression that the teacher in a face-to-face classroom can use, in addition to empirical testing, to assess performance. Such indicators might be as simple as a facial expression, increased attentiveness or attendance. Online instructors, however, may never be able to see the "changes brought about" as a result of their instruction. Therefore, course design must create ways to invoke student participation in ways that can be measured.

Traditional educational assessment models (i.e., goal-oriented or measurement-outcome approach) follow an Input-Process-Output methodology which first calls for identification of objectives, then identification of learning activities (in terms of behaviors/inputs) associated with these objectives, and finally design of techniques for determining whether the activities achieved the objectives (Calder, 1994). The methodology usually involves pre-and post-testing, and would be more closely associated with quantitative methods of inquiry. This kind of approach is problematic for both online and traditional educational settings because it ignores the learning process and the unintended, or unpredictable outcomes.

Moreover, the empirical testing model of assessment ignores the new opportunities that the online environment presents for encouraging and assessing learning. Using traditional methods of assessment for a new method of teaching is somewhat counterproductive. If technology is used for instruction, then it should also be used for assessment of progress and achievement. The features of most web-based learning platforms that make them suitable for instruction also provide built-in assessment features. Some course software records the time, duration, frequency and movements that students make every time they log on to their course. The appropriateness and accuracy of this type of assessment is necessarily to be debated, but there is nevertheless an added dimension available. More importantly, online discussions and chat transcripts record student contributions in a way that is not possible in the classroom, and the content of the participation can be analyzed for increasing evidence of comprehension or depth of participation over the duration of the course. For example, Henri (1992), proposed a five-part model for analyzing the content of online communication: participative, interactive, social, cognitive and metacognitive. The nature of non- face-to-face instruction places more emphasis on the student to be an active participant in the learning process, so necessarily it is the process of learning that should be emphasized, as opposed to "the creation of a product for evaluation" (Kitchen and McDougal, 1998-1999, p. 245).

Using pedagogical models for the design and assessment of instruction

Instructional design of online programs, as with face-to-face classrooms, is one of the most important factors influencing student outcomes. Assessment of online learning outcomes requires strategies during the design phase that will invoke measurable outcomes. This concept is not significantly different from traditional instructional design principles, which suggest a process of definition of goals, identification of tasks, etc. Many of the same pedagogical principles used in the traditional classroom can be used in the online classroom, but often it is all too easy to merely resort to delivering course content over the Internet but with little instructional design behind it. While not an exhaustive list, the following are descriptions of different models of instruction and some Outcome Questions that can guide the design and subsequent assessment of the learning experience based on these models.

Constructivism/action-reflection

Constructivism can most easily be understood in contrast to the objectivist model in which the teacher as bearer-of-knowledge transmits his or her knowledge to the student who memorizes and retains knowledge in units. The important distinction is that in the objectivist model the knowledge has been "encoded" by the instructor rather than by the learners (Cobb, 1999). The assumption is that by `constructing' their own knowledge through manipulation of raw data, learners retain and reuse that knowledge more effectively than through simply recording observations (Cobb, 1999). This theory is based on observation of learning in realistic settings (Mayer, 1999), and seems to most closely mirror the natural learning process that we go through outside of the school setting. In practice, the underlying principle is that learning should be student-centered, with the instructor acting only as a guide.

Constructivism lends itself well to distance learning because of the independent and often isolated nature of its delivery, yet it is often all too easy to design distance learning courses that simply give a list of required readings for the learner to read independently, along with some kind of questions for assessment. The constructivist alternative challenges online courses to be project-based, requiring the learner to find, analyze and synthesize materials found on the web, or extrapolate meaning and conclusions from readings, discussion questions, and classmates.

Outcome questions: How has the student synthesized multiple sources of information? Has the student drawn conclusions/parallels based on multiple sources of information, including their own past experiences? How has the student built on his or her prior knowledge in order to create new ideas? Is the student able to transfer lessons learned to new situations? To what extent were learners allowed to shape the content of the course, or expand it to suit their own needs?

Transformative Learning

Similar to constructivism, Transformative Learning (Jack Mezirow, 1990, 1991) refers to learning based on reflection and interpretation of prior experiences. It places importance on meaning and understanding of our ways of seeing the world so that actual perspectives are transformed (Palloff, R. and Pratt, K., 1999). Meizrow states that transformative learning primarily takes place through disorienting dilemmas that cause learners to critically assess the nature of knowledge and beliefs about power and social relations. Often, the mere fact of participating in an online course can be construed as a disorienting dilemma (Palloff & Pratt, 1999) in which the traditional experience of engaging with the content, the teacher and other students is completely different. Furthermore, there is the added interaction of the student with the technology. In an online course, participants not only engage with the course content, but also reflect on how that engagement affects the learning process and [possibly unanticipated] results in additional learning outcomes (i.e., learning about the technology, learning about themselves, intellectual growth, etc.). By interacting with other students and receiving feedback from multiple sources, online learners have a chance to transform their previous beliefs and ideas. Instructors who choose a course design based on this theory will want to create activities that invoke reflection, such as journal writing.

Outcome questions: What has the student learned about learning? How does the student explain their ideas? How have ideas changed as a result of reflection? How does the student relate the course material to his or her own life? How will changes in behavior accommodate this new learning? What are the implications of new ideas for the future?

Cooperative and Collaborative Learning

Cooperative learning differs from traditional models of learning and can be characterized as "not individual manipulation of mental objects, but social sharing of them" (Chiazzese et al., p. 17). It involves peer-to-peer learning through mentoring, debate, problem solving, discussion and hands-on activities as a group. In cooperative learning, students help and encourage each other as they work together towards a shared objective (Johnson and Johnson, 1999). Many people are skeptical that such group activities including socialization can occur for distance users. However, the nature of online learning makes it conducive to collaborative learning since students must use discussion groups or e-mail for communication and often the work of each student is shared with the group. This means that students can compare their ideas with those of the rest of the group, and they benefit from feedback by other students, not just from the teacher (Cannone-Syrcos, B. & Cyrcos, G., 2000).

New technologies also allow students to share files or work on a group project such as a web page, a database or digital art. The Internet also creates endless possibilities for bringing people together in diverse and disparate groups. Threaded discussions and online chats-the very tools that make group collaboration over the Internet possible-are also useful for assessment. Archived materials can be analyzed for content and frequency of participation to assess whether or not individuals are participating equally, and if they are learning new skills or simply dividing tasks in such a way that they take advantage of existing skills (Kitchen and McDougal, 1998-99). When using a collaborative learning design in online courses, instructors can directly observe group interaction through archived materials, or students can self-assess their own participation and that of the group as a whole.

Outcome questions: Are students productive working together in groups? Was decisionmaking shared? Were roles and responsibilities equally distributed? Was there individual accountability as well as shared goals? Did students explain concepts, share their knowledge, and give feedback to one another?

Conclusion

The concept of Outcome Questions can be used formatively as well as summatively to assess student progress towards mastering the course content, but more importantly, the Outcome Questions serve as a guide during the design phase, to create activities that will allow the instructor to answer these questions. For example, a course or lesson based on constructivist design will pose questions that require students to synthesize multiple sources of information, engage in debate, or solve a problem rather than simply repeat facts. The full range of possibilities for lesson design or activities based on these Outcome Questions is the subject for another article, as is the elaboration of additional learning theories and their use in online learning. By being aware of these Outcome Questions at the outset of the course, both instructors and students are provided with a rubric through which assessment occurs throughout the duration of the course, not just at the end. Furthermore, students are assessed not only on their ability to recall course content, but also the manner in which they participate in the course.

References

Bates, A. (1979) "Learning from television." Research Proposal to S.S.R.C., Audio Visual Media Research Group, The Open University Institute of Educational Technology

Berlanger, F., & Jordan, D. (2000). Evaluation and implementation of distance learning: Technologies, tools and techniques. Hershey, USA: Idea Group Publishing.

Calder, J. (19--). Program evaluation and quality: A comprehensive guide to setting up and evaluation system. London: Kogan Page Ltd.

Cannone-Syrcos, B., & Syrcos, G. (2000). Computer-mediated communication in distance education. In G. Orange, and D. Hobbes (Eds.), International Perspectives on Tele-Education and Virtual Learning Environments (pp. 15 - 31). Burlington, VT. : Ashgate Publishing Company.

Chiazzese, G., Cortopassi, C., & Lagana, M. (2000). A virtual secondary school classroom on the net. In G. Orange, and D. Hobbes (Eds.), International Perspectives on Tele-Education and Virtual Learning Environments (pp. 15 - 31). Burlington, VT. : Ashgate Publishing Company.

Cobb, T. (1999) Applying constructivism: A test for the learner-as-scientist. Educational Technology Research and Development, 47, (3), 15-31.

Henri, (1992). Computer Conferencing and content analysis. In Collaborative Learning through computer conferencing, (pp. 117-136). A. Kaye, (ed.) New York: Springer-Verlag.

Johnson, D. W. & Johnson, R. T. (1999). Making cooperative learning work. Theory into practice, 38 (2), 67-73.

Kitchen, D., & McDougall, D. (1998-1999). Collaborative learning on the internet. Journal of Educational technology systems, 27(3). 245-258.

Lockee, B.B., Burton, J.K., & Cross, L.H. (1999). No comparison: distance education finds a new use for "no significant difference." Educational Technology Research and Development, 47 (3), 33-42.

Mayer, R. (1999). Designing instruction for constructivist learning, in C.M. Reigeluth, (Ed.), Instructional-design theories and models: A new paradigm of instructional theory (Volume II, pp. 141-159). New Jersey: Lawrence Erlbaum Associates, Inc.

Mezirow, J. (1990). Transformative Dimensions of Adult Learning. San Francisco: Jossey-Bass Publishers.

Mezirow, J. (1991). Fostering critical reflection in adulthood : a guide to transformative and emancipatory learning. San Francisco: Jossey-Bass Publishers.

Morgan, A., Gibbs, G., and Taylor, E. (1980). The work of the study methods group. Report Number 1. Milton Keynes (England): The Open University Institute of Educational Technology.

Navarro, P. & Shoemaker, J. (2000). Performance and perceptions of distance learners in cyberspace. The American Journal of Distance Education, 14 (2), 15-35.

Palloff, R. & Pratt, K. (1999). Building learning communities in cyberspace: Effective strategies for the online classroom. San Francisco:Jossey-Bass Publishers.

Stake, R. E. (1967). Toward a technology for the evaluation of educational programs. In R.W. Tyler, R.M. Gagne, & M. Scriven, Perspectives of curriculum evaluation, (p. 1-18). Chicago: Rand McNally & Co.

Van Slyke, C., Kittner, M. & Berlanger, F. (1998). Identifying candidates for distance education: A telecommuting perspective. Proceedings of the America's Conference on Information Systems, Baltimore, (pp. 666-668).

The author is a graduate of the program in International Educational Development, and works for the Teachers College Distance Learning Program developing and assessing new online courses.