Re-evaluating the effectiveness of a web-based learning system: a comparative case study.

Much is unknown about the effectiveness of networked technology in education. Although a great deal of research exists dealing with the effectiveness of various types of web-based learning systems, the focal question of most evaluation studies does not concern the innovation of the delivery model itself and the factors that contribute to its effectiveness. Moreover, there is a lack of a theoretical or conceptual framework in many existing studies. The authors applied a systematic evaluation method (called CADMOS-E), and present the quantitative and qualitative results of a re-evaluation study on the effectiveness of a web-based learning system used at the National Technical University of Athens (NTUA), Greece, over a period of three years (1998, 1999, and 2000). The system supports an introductory course in software engineering offered by the Software Engineering Laboratory of the Electrical and Computer Engineering Department at the NTUA. The findings were compared with the results from a study done in 1997. According to that study the "quality of the learning material" alone explained almost 28% of the system's effectiveness (R2adj. = .278). After its first evaluation the system underwent a number of revisions and additions suggested by the results from quantitative and qualitative analysis of students' feedback. In this article, among other things, the authors demonstrate--using quantitative results--that the changes made in the design and presentation of the learning material resources improved the learning effectiveness of the revised system, from 28% to 37.5%.

**********

Networked technology is having and will continue to have a profound impact on education around the globe. It holds significant potential in advancing the interactivity between learners and tutors, in offering flexibility for the means of learning, and in providing easy, one-stop maintenance and reusability of resources (Lowe & Hall, 1999; Nielsen, 1995). However, the educational community has much to learn regarding how and in what ways technology can enhance the instructional process. While there is a large amount of writing devoted to research on the impact of technology in education, there is much that we don't know about its effectiveness. Moreover, there are certain gaps in these research efforts which require further investigation, specifically the lack of a theoretical or conceptual framework (Institute for Higher Education Policy, 1999).

In most evaluation studies, the focal question concerns the comparative effectiveness of various types of web-based learning systems when measured against traditional ones rather than the innovation of the delivery model itself and the factors that contribute to its effectiveness (Learning Technology Dissemination Initiative, 1998; Institute for Higher Education Policy, 2000). Also, there is often little detail on how the evaluation studies have been conducted and on the criteria that were used. In our study we applied a systematic evaluation method, called CADMOS-E (evaluation), to evaluate the learning effectiveness of Web technology in the instructional process as well as singling out the factors that influence its efficacy (Retalis & Skordalakis, 2001).

CADMOS-E method is part of a development methodology for web-based learning systems, called CADMOS (web-based Courseware Development Methodology for Open Learning Systems) (Retalis, 1998). CADMOS methodology embraces the whole development lifecycle of web-based courseware. CADMOS-E is a stepwise evaluation method supported by specially developed pretest and posttest questionnaires that provide data for both quantitative and qualitative analysis. The focus of the evaluation is on the learning effectiveness of the system as a whole as well as the identification of extensions and revisions that need to be made to it. Learning effectiveness is conceptualized as being related to a multiple measurement index consisting of cognitive and attitudinal outcomes (Makrakis, Retalis, Koutoumanos, Papaspyrou, & Skordalakis, 1998). Effectiveness is influenced by a number of variables such as: (a) quality of the learning resources; (b) changes in the preferred mode of study; (c) computer-mediated interactions with peers and instructors, and means of communication; d) the quality of services that the software and hardware infrastructures provide.

The purpose of this article is two-fold. First, the quantitative and qualitative results of a re-evaluation study of the effectiveness of the web-based learning system (http://webct.softlab.ntua.gr) used at the National Technical University of Athens (NTUA), Greece, during the academic years 1998, 1999, and 2000 is presented. Second, they are compared them with the results from a similar study that took place in 1997. The system supports an introductory course in software engineering offered by the Software Engineering Laboratory of the Electrical and Computer Engineering Department at the NTUA. The WebCT (WebCT, 2001) class management system formed the basis of the specific technological infrastructure on which the web-based courseware and the tools for learning activities were integrated. The first version of the system was evaluated in 1997 (Makrakis et al., 1998). In a subsequent study reported here-using quantitative results--that the corrective actions made based on the feedback from the first evaluatio n study positively affected the learning effectiveness of the revised system.

The first section of the article presents the conceptual framework of the research using the CADMOS-E method. The second section presents the evaluation study and results drawn from examination of data gathered during three years. The third section presents the main results from the study in 1997, and is focused on the comparison of the results. Finally, the last section makes some conclusions on the findings concerning both evaluation studies.

THE CONCEPTUAL FRAMEWORK OF THE RESEARCH

CADMOS methodology is a collection of specific process models, methods, CASE tools, standards, and documentation aids whose purpose is to assist with the development of a quality, web-based learning system on time and within the limits of resources through a controllable and quality oriented process. CADMOS supports the development of a web-based learning system through an evolutionary model that is iterative and enables developers to construct increasingly more complete versions of the end product. Thus, according to CADMOS, a web-based learning system should be developed as a series of fully functional builds (working versions of the learning system). A build satisfies the current set of requirements for the product under development. The underlying rationale of evolutionary development is that the client's perception (either learner's or teacher's or institutional organization's) of the complete end product is a "moving target." As the builds are tested and summatively evaluated, the user's opinion regardi ng the characteristics of the system change, resulting in revised requirements, which--in turn--affect the design and development of future builds.

For example, the evaluation study of the first version of the web-based learning system used in 1997 showed that the students were concerned with the following:

* There were not enough elaborated case studies dealing with software development.

* The course content did not include enough modem topics and open research issues.

* The learning material units were quite long and lacked multimedia for illustrating appropriate topics.

* Web-based interaction, and subjects for discussions were limited.

Thus, the newer version of the system differed from the first one in the following ways:

* The web-based learning material was enriched by three detailed case studies on software development using various methodologies.

* An online library with links to special and modern topics on software engineering was created.

* Serious cuts and changes in the context of the web-based modules were made.

* The exam papers of the previous years with their solutions were added as learning resources.

* The subject matter experts provided some interesting discussion topics. The instructor and the tutors were better trained to moderate the discussion on these topics as well as to encourage and motivate students participating in the discussions.

The iterative nature of evaluation (i.e., as new versions come along) should assist in making the learning experience more effective since the feedback is used to continuously improve matters. The aim of CADMOS-E is to identify the learning effectiveness of a web-based system as a whole as well as the impact of its subsystems.

A web-based learning system supports and partially automates the instructional process in a subject field. In practice, one can see such systems in all types, sorts, and sizes: for example, a course, a seminar, or even a series of lectures (Tennyson & Breuer, 1997). Typically, a web-based learning system consists of four interrelated subsystems:

* The human subsystem, which describes for each kind of human agent involved in the instructional process (learners, teachers, tutors, system administrators) his/her role in as much detail as possible.

* The web-based learning resources subsystem, which is perceived as a mosaic of online learning resources--course notes, slide ware, study guides, self-assessment questionnaires, communication archives, learning material used for communication purposes, and the like.

* The technical infrastructure subsystem, which is divided into common and special domains. A learning system basically makes use of services from a common infrastructure that supports student learning in general (e.g., laboratories, networking facilities). However, to best support the instructional process, specific infrastructure should be created (e.g., multimedia conferencing systems, special hardware components, a specific course management tool) that will provide services unique to a particular instructional problem (Ford et al., 1996).

It should be noted that according to the holistic theory, each subsystem should be itself an independent, viable system, interrelated with the rest of the subsystems. Thus, when breaking up a web-based learning system into subsystems, we need to keep the entirety in mind, as well as the interrelations of each component. For example, if teachers decide to use scaffolding to support students in learning new skills, software-realized scaffolding should be supported by the infrastructure subsystem. Moreover, to facilitate communication among the human agents, special synchronous and asynchronous communication tools might be purchased. IF students prefer the sensing mode of perceiving rather than the intuitive mode, the learning resources could incorporate activities that enable students to work with concrete experiences as well as to navigate through the content through high degrees of structure.

Learning effectiveness is influenced by a number of variables that measure the impact of the above subsystems on learning. These independent variables are:

* contribution of the web-based learning resources to the acquisition of knowledge and skills with respect to their learning objectives;

* time spent on task using the system;

* computer mediated interactions with peers and instructors, and means of communication;

* the quality of the learning resources (instructional material);

* the learner's profile (learning style, previous experience, etc); and

* the preferred mode of study (with or without the use of Web technology).

A web-based learning system (like any instructional system) is developed with a specific instructional model in mind (Gagne, Briggs, & Wager, 1994). This means that, based on the design and development choices, the Web can be used in the instructional process for:

* information distribution, such as announcements, course description, calendar;

* delivery and management of learning material-for example, presentation of online course notes and updating the learning material;

* offering multiple communication facilities-for example, asynchronous and synchronous communication; and

* class management--for example, online marking of students' assessments, tracking learners' participation, management of learners' profiles.

The instructional model, which is supported by a web-based learning system, relies on the mix and match of the previous lists (McCormack & Jones, 1997). As a result, the following instructional models can be created:

* the information-based models (Web is used for retrieving information as in digital libraries and virtual museums);

* the teaching media based models (Web is used only for dissemination of educational material to distance students, that is course descriptions, educational software, and the like);

* the enriched classroom models (Web is used complementary to traditional classroom-based teaching to offer open and distance learning opportunities); and

* the virtual classroom models (Web is used with emphasis on collaboration and computer-mediated human interaction).

The web-based learning environment examined here provides a number of features to teachers, students, tutors, educational managers, and so forth, to:

* create, operate, and administrate an online course;

* support collaboration between students and provide both motivation and resources for team building;

* create, deliver, and automatically assess online questions and tests;

* organize educational, financial, and human resources; and

* administer virtual, distributed classes where the students are geographically scattered and communicate through the Internet.

The online learning resources for the introductory course in Software Engineering consisted of:

* an e-book in the form of hypermedia course notes whose structure follows the UK's Open University standards for structuring the learning material into blocks and units;

* slideware;

* case studies on software development using various methodologies;

* samples of team projects from past years;

* discussion topics;

* online journal papers;

* an online library with learning resources for selected topics such as CASE tools, IEEE standards, tutorials on programming languages;

* exam papers of the previous years with their solutions; and

* descriptions of the course and the team projects.

Thus, the students are provided with a variety of teaching aids for acquiring knowledge and skills without many constraints in time and place of instructional delivery.

THE EVALUATION STUDY

CADMOS-E is a pretest and posttest summative evaluation method. It is supported by specially developed pretest and posttest questionnaires that provide data for both quantitative and qualitative analysis. Given the nature of the method, the design of the pre and post questions is critical to success.

The first type (pretest) alms at identifying the expectations of the learners for the course and its delivery model as well as their learning profile (learning style, previous experience). This questionnaire consists of 29 items, and it is typically administered during the first days of the course but not later than the first week. Pretesting would be pointless if the learners have been exposed to the web-based learning system.

The second type (posttest) deduces overall judgments and criticism on the learning effectiveness. The questions appearing on the pretest are also replicated in the posttest in a rephrased form. However, this second questionnaire consists of a large number of closed-end questions that are used to evaluate in detail specific issues on the quality of the online learning material, the delivery model, and the instructional support provided during a course. This segment also includes a section with a number of open-ended questions to supplement the quantitative data. The open-ended section covers issues related to students' likes and dislikes about the learning system, the deficiencies concerning the instructional model, suggestions for improving instructional support, the learning resources, and the technological infrastructure.

For the closed-end questions, the answers are measured using a five-point Likert-type scale, coded as follows: 5=1 absolutely agree,..., 1=1 absolutely disagree. For the open questions a significant amount of space is provided for learners to write their answers.

Subjects

The total number of students who responded to the evaluation study reached 75 (13% women and 87% men) out of the 86 registered in the course during the academic years 1998, 1999, and 2000. Of these students 1.3% indicated that they were computer novices, 45.4% had good experience, 46.6% had very good experience, and 6.7% had professional experience. In terms of time spent on studying using the system, 32% spent less than one hour, 39% from 1 - 2 hours, and 24% from 3 - 4 hours per week, while 4% spent more than 5 hours per week.

Research Instruments

Multiple items measure most of the main variables in this study, each focusing on a slightly different aspect of the main variable. In building composite measurement scales, items included were first scrutinized for "face validity." After the data were collected, the validated items in each composite variable were subjected to a Cronbach's Alpha reliability analysis for internal consistency of the instrument. In arriving at the final composite measurement indexes, every item that substantially lowered the Alpha coefficient was omitted and a new analysis was conducted to arrive at an index having the highest possible reliability measure.

The summary statistics of the item analysis for homogeneity and reliability indices, shown in Table 1, indicate that the "learning effectiveness" composite variable reached a high alpha coefficient ([+ or -]=.85), retaining all the 14 intended items. These items included dimensions of cognitive and attitudinal outcomes. The "preferred mode of study" composite variable, shown in Table 2, retained all the five intended items with a reliability of [+ or -]=.86. This variable measured the comparative outcomes of the traditional mode of teaching and the enriched instructional delivery mode based on the Web.

To measure the "contribution of the web-based learning resources to the acquisition of knowledge and skills" variable, for each learning resource (e.g., e-book, additional learning material, assignments, course description, etc) we gave the following question was given to the learners: "How much did the following learning resources contribute to the acquisition of knowledge and skills?" However, the reliability of the answers was checked by looking at the log files to determine whether or not learning resources that were highly appreciated by the learners had equally high access rates. The "quality of the learning resources" variable was measured by a number of items, which reflected criteria recommended in Tessmer, (1995), including aesthetics, transparency, forgiveness, matching between the metaphors and the learning experiences, informativeness, seamlessness of content and media, as well as the achievement of the desired learning experiences, and learning outcomes. Assessment of computer-mediated interact ions with peers, instructors, and means of communication included measurements of the frequency of using means such as e-mail, computer conferencing, and frequency of meetings among students and with instructors.

Learning styles were measured by a number of variables which reflected the four learning categories identified by (Gregorc, 1979), "Concrete Sequential," "Abstract Random," "Abstract Sequential," and "Concrete Random."

Data Analysis

Analysis of the structured part of the questionnaire was based on univariate and multivariate statistical analysis and the open-ended part on qualitative content analysis. All appropriate tests for examining whether these variables fulfill the conditions for undertaking a regression analysis were conducted (e.g., normality, multicolinearity). These tests showed that all requirements were met.

Evaluation Results

The regression analysis, shown in Table 3, indicates that the "contribution of the learning resources to the acquisition of knowledge and skills" alone explained 37.5% of the web-based system's effectiveness (R2adj.= .375). The "preferred mode of study" came in second by adding 15% (R2ch.= .151) of the effectiveness variance, and finally "time spent on studying using the system" increased the effectiveness explained variance to 57%. Thus, quite a high percentage was accounted for by three significant predictors alone. All the other predictors did not significantly contribute to the prediction of the effectiveness measure.

The qualitative results complement the quantitative conclusion that the "contribution of the learning resources to the acquisition of knowledge and skills" and the "preferred mode of study" are the most significant predicting variables for effectiveness. For the first variable, this is justified by the fact that 72.4% of the students rated the quality of learning material more than good (very good or excellent). Such a high rate may also explain why the "quality of the learning resources" variable did not add anything significant to the system's effectiveness. It is also important to note that the reason why "previous computer experience" appears not to have added significantly to the system's effectiveness may be because most of the end-users had similar computer skills.

Contrary to what was expected, the quantitative analysis showed that learning styles did not contribute significantly to the effectiveness of the system. However, a comparative statistical analysis of the data collected from the pretest and posttest questionnaires, as well as the content analysis of the responses provided by open-ended questions, revealed a number of patterns related to the degree to which students' learning styles have been affected as a result of their experience with the web-based learning system. It was found that the great majority of the students felt that the system had positively affected their study patterns, especially the independence of learning and the deep level and fast level of information processing provided by this system. These gains appeared to be affected by the flexibility provided by the system in terms of time, place, and pace of instruction and learning, the high-tech design of lessons, and the searching facilities integrated in the system.

Computer-mediated interactions with peers, instructors, and means of communication were few, and so they did not influence the effectiveness of the system. The content analysis of the responses provided by open-ended questions revealed two main reasons for this: (a) the tutors did not play their role in the best possible way (to encourage and mediate computer-mediated communication and collaboration), and (b) the students still prefer to contact the teacher and the tutors face-to-face.

COMPARISON RESULTS BETWEEN THIS STUDY AND THE 1997 STUDY

A first version of the system was evaluated in academic year 1997 (Makrakis et al., 1998). According to that study the "the quality of the learning material" alone explained almost 28% of the system's effectiveness (R2adj.= .278). The "preferred mode of study" entered second by adding 11% (R2ch.=.l13) of the effectiveness variance, and students' interactions with the instructor increased the effectiveness variance to 48%. All of the other predictors, that is, previous experience with computers, time spent on studying using the system, student learning styles, and interactions among students by way of communication means did not significantly contribute to the prediction of the effectiveness measure.

The qualitative results complemented the quantitative conclusion that the "learning material" and the "preferred mode of study" were the most significant predicting variables for effectiveness. Moreover, a number of suggestions were made for the enhancement of the web-based learning resources, such as:

* including more interactive examples, animation, and self-assessment exercises;

* enriching the content with new material which should be presented in a more analytical way;

* providing summaries at the end of each chapter, highlighting the major aspects and key concepts; and

* providing better consistency of the topics, especially by merging small units.

After its first evaluation, the system underwent a number of revisions and additions as suggested by results from the quantitative and qualitative analysis of students' feedback. The quantitative results of the new study show that the corrective actions made according to the feedback from the first evaluation did affect positively the learning efficacy of the revised system.

Comparing the new findings with the results of the previous evaluation, it can be said that the new version of the system had better quality learning resources and so the corresponding variable significantly increased its predicting percentage of the system effectiveness. The students' preferred mode of study has increased the variable's predicting percentage of the system effectiveness also, which shows that the new technology has affected the students, who now spend more time working through the system. This is also complemented by the fact that the "time spent on studying using the system" variable contributed a bit more to the effectiveness variance of the revised system. The "previous computer experience" variable did not significantly account for the variance in the system's effectiveness, which shows that the course enrollment in both studies contained students with similar computer skills.

The quantitative results of both evaluation studies showed that learning styles did not contribute significantly to the effectiveness of the system, which may be explained by the fact that students had similar learning profiles. However, the content analysis of the responses provided by open-ended questions of both evaluation studies revealed that the students' learning styles have been affected as a result of their experience with the web-based learning system. The independence of learning and the deep and fast level of information processing was especially appreciated by the students.

CONCLUDING REMARKS

The results obtained from these evaluation studies suggest that the quality of learning resources is of considerable importance in producing an effective web-based learning system. Similar results were reported by Barker and King, (1993). The presented re-evaluation study demonstrates, among other things, that the changes made in the design and presentation of the resources of learning material significantly increased its contribution to the learning effectiveness of the web-based learning system--from 28% to 37.5%--which is an impressive gain.

This re-evaluation study also clearly revealed that the main goal of the evolutionary development of a web-based learning system was achieved. The efficacy of the new version of the web-based learning system under consideration (featuring changes made according to the feedback from the students and the tutors) was increased. Moreover, the students' opinions towards the use of new technologies as supplement to the traditional teaching methods were more positive than in the first evaluation round.

Yet, it is still difficult to draw a firm conclusion about the effectiveness of the new technologies in education. Evaluation results from data that have been collected over a series of years on a specific subject matter, like the one presented in this article are needed.

Table 1

Reliability of Items in the Effectiveness Scale

Item-total Statistics                   Alpha
                                        if Item
                                        Deleted

It was easy to attend the
instructional delivery model.           0.83
The instructional delivery model
widened student-instructor
communication.                          0.84
The instructional delivery model
alleviated the physical constraints
of attending face-to-face lecturing.    0.83
The instructional delivery model
increased the potential to pursue
collaborative project work with
other students.                         0.83
The instructional delivery model
increased my interest in the
subject matter.                         0.84
The instructional delivery model
provided immediate and easy
information relevant to the
subject matter.                         0.85
A variety of learning experiences
were provided through the
instructional delivery model.           0.84
The instructional delivery model
helped me to solve problems and
answer questions that appeared
during the course.                      0.84
The instructional delivery model
allowed me enough freedom to choose
where, when and how to study.           0.83
The instructional delivery model links
to activities in the classroom.         0.84
The instructional delivery model
increased my knowledge and skill
in the subject matter.                  0.82
The instructional delivery model
encouraged me to participate
actively in normal class
activities with other students.         0.83
The instructional delivery model
required me to evaluate using my
judgment and intuition.                 0.85
I have sufficient opportunity to
demonstrate what I have learned
in the subject through the
instructional delivery model.           0.85

Reliability Coefficients N of Cases = 75.0 Alpha = .85

N of Items 14

Table 2

Reliability of Items in the Preferred Mode of Study

Item-total Statistics                                 Alpha
                                                     if Item
                                                     Deleted

The instructional delivery model proved to be
more beneficial than conventional way of lecturing.   0.78
The instructional delivery model has made
me prefer learning from this model even when the
same teaching is given in other ways.                 0.83
I found the instructional delivery model of
teaching to offer better experiences than
conventional way of lecturing.                        0.86
The instructional delivery model was more
pleasant than conventional way of lecturing.          0.82
The instructional delivery model of
teaching was more convenient than
conventional way of lecturing.                        0.82

Reliability Coefficients

N of Cases = 75.0

Alpha = .86

N of Items = 5

Table 3

Summary Statistics of Stepwise Regression Analysis Predicting the System
Effectiveness

       R     R Square  Adjusted R  Std. Error of  Change
                       Square      the Estimate   Statistics
Model                                             R Square    F Change
                                                  Change

1      .612  .375      .366        .37981         .375        43,809
2      .725  .526      .512        .33318         .151        22,862
3      .755  .570      .552        .31948         .044        7,311

                         Durbin-
                         Watson
Model  df1  d12  Sig. F
                 Change

1      1    73   .000
2      1    72   .000
3      1    71   .009    1,577

(a) Predictors: (Constant), Contribution of the Learning Resources

(b) Predictors: (Constant), Contribution of the Learning Resources,
Preferred Mode of Study

(c) Predictors: (Constant), Contribution of the Learning Resources,
Preferred Mode of Study, Time Spent

(d) Dependent Variable: Effectiveness

Acknowledgments

We would like to thank professor Manolis Skordalakis for his assistance in collecting the questionnaires and professor Vassilis Makrakis for his helpful comments on the analysis of the collected data. We would like to thank professor Patricia Carison for her essential contribution on the quality of the article.

References

Barker, P., & King, T. (1993). Evaluating interactive multimedia courseware--a methodology. Computers and Education, 21(4), 307-319.

Ford, P., Goodyear, P., Heseltine, R., Lewis, R., Darby, J., Graves, J., Sartorius, P., Harwood, D., & King, T. (1996). Managing change in higher education: A learning environment architecture. Society for Research into Higher Education & Open University Press, International Computers Ltd.

Gagne R., Briggs, L., & Wager. L. (1994). Principles of instructional design. Fort Worth, TX: HBJ College Publishers.

Gregore, A.F. (1979). Learning/teaching styles: Their nature and effects. In J.W. Keefe (Ed.), Student learning styles: Diagnosing and prescribing programs, (pp. 19-26). Reston, VA: National Association of Secondary School Principals.

Institute for Higher Education Policy (1999). What's the difference: A review of contemporary research on effectiveness of distance learning in higher education, Retrieved June 16, 2002, from: http://www.ihep.com/Pubs/PDF/Difference.pdf

Institute for Higher Education Policy (2000). Quality on the line: Benchmarks for success in Internet-based distance education, Retrieved June 16, 2002, from: http://www.ihep.com/Pubs/PDF/Quality.pdf

Learning Technology Dissemination Initiative (1998). Evaluation cookbook. Retrieved June 3, 2002, from: http://www.icbl.hw.ac.uk/ltdil cookbook

Lowe, D., & Hall, W. (1999). Hypermedia & the Web: An engineering approach. New York: John Wiley & Sons.

Makrakis, V., Retalis, S., Koutoumanos, A., Papaspyrou, N., & Skordalakis, M. (1998). Evaluating the effectiveness of an ODL hypermedia system and courseware at the National Technical University of Athens: A case study. Journal of Universal Computer Science, 4(3), 259-272.

McCormack, C., & Jones, D. (1997). Building a web-based education system. New York: John Wiley & Sons.

Nielsen, J. (1995). Multimedia and hypertext: The Internet and beyond. AP PROFESSIONAL, ISBN 0-12-518408-5.

Retalis, S. (1998). CADMOS: A web-based CourseWare development methodology for open learning systems. Unpublished doctoral dissertation, Department of Electrical and Computer Engineering, National Technical University of Athens.

Retalis, S., & Skordalakis, E. (2001). An approach for solving instructional problems using networked technologies. Computers in Human Behavior, 17(4), 393-40 7.

Tennyson, D., & Breuer, K. (1997). Instructional theory: Psychological perspectives. In R.D. Tennyson, F. Schott, N. Seel, & S. Dijkstra (Eds.), Instructional design: International perspectives, Vol. 1: Theory and research. Hillsdale, NJ: Lawrence Erlbaum.

Tessmer, S. (1995). Formative multimedia evaluation. Training Research Journal,], 127-149.

WebCT. (2001). WebCT, Retrieved June 2, 2002 from: http://www.webct.com/