Design methodology for the implementation and evaluation of a scenario-based online learning environment.

A constant challenge facing university faculty and academics is determining how innovative and authentic elements, based on constructivist philosophy, can be manifested effectively in online learning settings. In this article, we describe an educational technology postgraduate course on evaluation that incorporated a scenario whereby assessable tasks were grounded within the context of a fictitious consultancy company. The scenario was driven by the use of a fictitious character--a "remote" Chief Executive Office (CEO) and the reconfiguration of the teacher as a company-recruited Academic Advisor. Characters were used as a motivational device to drive the online activity for the course, which was delivered completely online. The evaluation findings highlighted that whilst the company scenario and use of character did provide an authentic learning experience for most students, its use required considerable support and scaffolding in order to maximise its potential to facilitate an engaging and authentic learning environment. This article describes the rationale for the course redesign, explains the approach taken to evaluate the effectiveness of its implementation and presents the findings of the evaluation.

**********

Teachers have long recognised the importance of anchoring learning within real-life contexts. Approaches to the design of learning environments, influenced by constructivist philosophy, have changed the way many students learn and the way many teachers teach. With the increase in learning affordances provided largely by information and communication technologies, there has been a great deal of research and development in refining constructivist approaches to the design of learning environments, such as anchored instruction (Bransford, Sherwood, Hasselbring, Kinzer, & Williams, 1990), problem-based learning (Boud & Feletti, 1997; Evensen & Hmelo, 2000), situated learning (Brown, Collins, & Duguid, 1989; Herrington & Oliver, 2000) and scenario-based learning (Kindley, 2002; Wu, Huang, Chen, & Wu, 2003). All of these approaches aim to enhance the opportunities for students to engage with authentic situations and tasks which facilitate immersion with the content within realistic situations.

Many teachers and academics are now "moving online," either willingly or as a requirement of their teaching program, and for many, the prospect of teaching online presents a formidable challenge. A great deal has been written about the theoretical constructs and characteristics that underpin successful learning environments, but a challenge persistently facing academics is how these characteristics can be manifested effectively in an online learning setting.

The purpose of this article is to disseminate some ideas about how this challenge can be addressed by describing how an online learning environment, underpinned by constructivist principles, was designed and implemented totally online. Furthermore, how the environment was evaluated to determine its effectiveness once implemented, is also explored.

CONTEXT

An online course/subject on Evaluation and Implementation of Technology-Based Learning required redesign. The course was an educational technology postgraduate course and it was to be delivered, first, to a predominantly off-shore student cohort and then to a national student cohort comprising both on-campus and off-campus students. The content scope and overall objectives of the course were to remain the same as the previous implementation, however the subject structure, activities, and resources required modification for both practical and pedagogical reasons. First, the Faculty was expanding flexible learning opportunities to its students, and wanted to explore strategies to facilitate multiple modes of delivery in its courses. Second, the exposure to a variety of resources that represent multiple perspectives seemed an ideal opportunity to allow students to build their own understanding.

Constructivist learning principles such as: learning is a process of construction, learning occurs through social negotiation of meaning, learning is contextually mediated, and reflective thinking is an ultimate goal (Duffy & Cunningham, 1996; Jonassen, Mayes, & McAleese, 1993; Robyler, Edwards, & Havriluk, 1997) served as an overall guide for the team's redesign decisions.

The team was interested at a more specific level to explore the use of scenarios to provide an authentic context for the evaluation strategies that the students would undertake in the course. Scenario-based learning has been used effectively in a variety of training situations, and has been particularly effective in the service industries. In simulated scenarios, students are led through a typical encounter with a customer, with pathways leading to an outcome indicating success or failure (Kindley, 2002; Whitcomb, 1999; Wu et al., 2003). While elements of this approach, such as the use of a real context and a learner-centred focus, were deemed appropriate, other characteristics of the approach did not apply to the more complex, problem-solving approach appropriate at the postgraduate level of study. For example, Kindley (2002) advocated that scenario-based learning occurs by following "success and failure paths" that are limited in scope to prevent the scenario becoming too complex. In keeping with a more constructivist philosophy, the team felt it important to maintain complexity rather than minimise it, but to emphasise instead the collaborative nature of the environment and the scaffolding support provided by the teacher. In another difference, Kindley and Wu et al. (2003) argued that scenarios are most effective when they are illustrated with advanced interactive media, and have a game-like appearance. The authors believed that neither verisimilitude nor technological capacity was essential to the effective use of scenarios, and that "cognitive realism" to the real-life task was of more significance (Herrington, Oliver, & Reeves, 2003). The use of scenarios in the current course was therefore more appropriately described as the provision of an authentic context for a complex and realistic task, rather than an unconditional embracing of the principles of scenario-based learning design.

Two immediate challenges faced the teaching team: first, the design and development of suitable authentic and complex tasks that would instantiate the constructivist principles described above; and second, the development of a resource repository that provided multiple perspectives of the content for the learners to explore. Another significant practical challenge was that the first offering of the redesigned course was to be delivered as a face-to-face workshop followed by online interaction and submission of assessable work. Typically, the teacher would travel to the overseas site and conduct workshop activities over several days, then return home to continue the remainder of the subject online. However, due to the world health concerns brought on by Severe Acute Respiratory Syndrome (SARS), the face-to-face workshop was no longer possible. An alternate introductory delivery strategy had to be devised.

THE REDESIGN OF THE LEARNING ENVIRONMENT

To rework the subject, guidance was gained from exploring the Learning Designs website--an Australian federally-sponsored site dedicated to providing downloadable templates for innovative online learning designs in higher education (see http://www.learningdesigns.uow.edu.au). In particular, two generic learning designs influenced the redesign process:

* The Online role-play learning design (Wills & Ip, 2002)

* The Explore, describe, apply learning design (Oliver & Herrington, 2002)

The Online role-play learning design informed the teaching team of a way to extend the previously limited use of scenario by implementing the role-play approach. The previous implementation had couched the assessment tasks within a company scenario where students were to complete their tasks in role. This was extended by introducing a fictitious character, the Chief Executive Officer who could drive the online activity. To reinforce the constructivist nature of the course, the subject teacher was to serve as the company's Academic Advisor, largely providing a coaching and scaffolding role. Students were immersed in the role of new employees in the company. Tasks were assigned by the CEO usually in the form of "commissioned work," and the company's Academic Advisor (the teacher) provided appropriate support for the completion of the tasks.

The Explore, describe, apply learning design influenced how to structure the course into discrete, but contributing to a whole, modular phases. The nature of subject content was influenced by the idea of creating evaluation tasks of increasing complexity--starting with a small-scaled task and working towards more complex tasks, just as in real life, a new employee might be given more straightforward tasks and then move towards more complex tasks as they become fully involved and capable members of the company. Thus, the sequence of tasks would serve as a scaffold for learners.

Five phases were devised for sequencing student activity in the course: orientate (provide an overview of the subject and its content), do (conduct a small usability evaluation task), explore (examine perspectives about evaluation), apply (transfer theory into practice by writing an evaluation proposal) and reflect (reflect and self-assess understanding of content).

THE IMPLEMENTATION

Ten students (all part-time) were enrolled in the course. There were five women and five men. Nine students were off-shore and one male student was interstate. The nine off-shore students knew each other as they had been a student cohort for the previous 18 months. The course was implemented using the following information and communication technologies:

* Course web site (housed on the faculty server).

* E-mail mailing list (used to provide course updates to the students and as an asynchronous discussion tool for the first four weeks of semester).

* WebCT site that provided access to online library resources and synchronous and asynchronous discussion tools (chat and discussion forum respectively). The discussion forum was used from Week 5. (Due to technical problems experienced with WebCT, the teaching team was advised to implement the course web site on the faculty server and use e-mail as an asynchronous communication mechanism until the problems were rectified.)

In addition, a CD-Rom was posted to all students prior to the course that provided details such as the subject outline, some reading resources and an explanation of the scenario. Figure 1 illustrates how the scenario was introduced to the students.

An introductory workshop comprising three online chat sessions took place whereby the teacher provided an orientation to the course and the technologies used. A warm-up activity, where employees had to devise a company name, was used to encourage students to start thinking in company-mode. This also enabled the class to trial the communication technology.

Each assessable task culminated in the completion of one phase of the course. Each phase was initiated by an agenda distributed by the CEO. Because the CEO was unable to be present (due to meetings held around the world), direction for the company recruits came in the form of a Company Minute based on a meeting held between the CEO and the Academic Advisor (an excerpt is illustrated in Figure 2). This mechanism drove the scenario and was intended to serve the following:

[FIGURE 1 OMITTED]

* provide a realistic and plausible reason for student action;

* create the illusion of greater "presence" in the learning space;

* provide scaffolding to support learning (each company minute contained advice and guidance); and

* provide a synchronisation cue (each phase of the course was instigated by a newly posted Company Minute).

A summary announcement on the home page alerted students to a newly posted Company Minute, for example:

"The CEO has secured us another job! It's an effectiveness evaluation proposal. See 'Announcements' for all the details and our action plan for the next 3 weeks! Academic Advisor" (Summary Announcement to initiate the apply phase of the course).

THE EVALUATION

Evaluation Methodology

The purpose of the evaluation was to determine whether the implementation of the revised course was effective in terms of facilitating student learning, and how the course could be improved for subsequent offerings. Thus, it is described as an effectiveness evaluation. To undertake this kind of evaluation, the following frameworks were considered:

* An integrated evaluation framework (Bain, 1999). (An adaptation of the framework by Alexander & Hedberg, 1994.)

[FIGURE 2 OMITTED]

* A learning-centred framework for whole project evaluation (Phillips, Bain, McNaught, Rice, & Tripp, 2000). (An adaptation from Alexander & Hedberg, 1994, & Bain, 1999.)

* A decision-oriented rationale for evaluation (Reeves & Hedberg, 2003.)

These evaluation frameworks are appropriate for evaluating an educational innovation and provide guidelines for evaluation in each of the major phases of the innovation project, such as, design, development, implementation, and institutionalisation (Bain, 1999). This evaluation focused on the implementation phase of the Bain (1999) and Phillips et al. (2000) frameworks. These two frameworks provided overall guidance and suggested appropriate methods to maximise data collection for the summative purpose of this evaluation. Reeves and Hedberg (2003) offered guidance by providing scaffolding for the evaluator in the form of four questions, each of which is addressed as follows:

What kinds of decisions can be anticipated from the evaluation? The evaluation was required to inform a decision on whether the course should continue to be offered in its redesigned format, and whether any improvements could be made to the course.

What questions need answering in order to make the decisions? Six questions were devised to inform the anticipated decision. These questions were formulated based on the guidance offered by Bain (1999) and Phillips et al. (2000) and are listed here:

1. What were the students' perceptions of the use of scenario in this course in terms of facilitating their learning?

2. What were the students' perceptions of the learning design (that is, the sequence of tasks, the resources provided and the support mechanisms supplied) in terms of facilitating their learning?

3. Was the online implementation strategy effective in facilitating student learning?

4. What learning outcomes were achieved?

5. What were the students' perceptions about their learning experience?

6. What improvements can be made to this course?

What information is needed to answer the questions? To answer the questions, the information required included obtaining students' perceptions of the learning experience in terms of the knowledge they felt they had gained and the perceived usefulness of the implementation plus the teacher's perception of the implementation.

What data collection instruments are required to obtain the information? The data collection methods used in the evaluation are listed below. (Note that the first four data sources were collected regardless of conducting an evaluation).

1. E-mail correspondence amongst students and the lecturer during the semester

2. Online synchronous "chat" transcripts

3. Online asynchronous discussion transcripts

4. Student-produced artefacts (eg., nonassessable and assessable tasks)

5. Student interviews (conducted at the end of semester)

6. Student questionnaires (a formative questionnaire completed during the semester and a summative questionnaire completed at the end of semester)

7. Teacher survey (conducted externally by the university)

The student interview (conducted at the end of semester) was designed to explore the influence of the redesign of the course on the student's learning process. The interview was conducted in the synchronous chat space and was guided by the following questions:

* Do you think the learning design implemented in this subject was effective in terms of assisting you to construct your own understanding of the subject content?

* What motivated/engaged you in the subject?

* Did this subject meet the expectations that you stated at the beginning of this subject?

* How would you rate your learning experience (positive or negative)?

The formative questionnaire, distributed mid-way during the session by e-mail, served as an opportunity to gauge student perceptions about the online course and enabled students to suggest changes to the remainder of the course. The summative questionnaire, distributed at the end of semester through e-mail, was designed to determine students' perceptions of the effectiveness of the course in terms of its redesign and delivery. It included open-ended questions closely aligned to the overall questions driving this evaluation plus a series of questions based on the evaluation framework for High Quality Student Learning (Boud & Prosser, 2002). This framework advocates that four principles influence high quality learning: engaging and challenging learners, providing practice, and acknowledging the learning context. Boud and Prosser provided a set of questions to probe whether these principles are evident in a learning environment. An adaptation of this set of questions was included in the questionnaire.

This data collection strategy facilitated triangulation as responses from the interviews could be corroborated with the responses from the questionnaires.

Evaluation Results

Seven of the ten students agreed to participate in this evaluation. Six students were interviewed, three students completed the formative questionnaire and five students completed the summative questionnaire. Eight students completed the externally administered teacher survey. The data collected was analysed by using the six questions that guided the evaluation as overall themes and determining the issues that emerged relevant to each theme. The results are presented in accordance with the major themes investigated.

1. What were the students' perceptions of the use of scenario in this subject in terms of facilitating their learning?

Most of the participants thought that the use of the company scenario was effective in facilitating their learning. Four of the seven participants stated that the scenario strategy created a sense of realism to the tasks and provided an element of fun to the online learning environment. Of these, two students commented that the company scenario strategy was the best thing about the subject. Three students explicitly stated that the use of the two characters--the fictitious CEO and the lecturer as the Academic Advisor--was a contributing factor to their positive perception. These four students also felt that the delivery of the content, presented in the form of company strategy minutes, supported their engagement in the subject because it allowed them to work logically through the tasks.

Two students, however, felt that the company scenario did not work for them and their reasoning varied. For one student, Hui Ying, the scenario created confusion in terms of completing the assessment tasks. Hui Ying's views did not change for the entire semester. Her feedback in the formative questionnaire, the online chats, the summative questionnaire, and interview are consistent. She was confused about the character roles and the role of the company itself particularly when completing assessment tasks as she felt that as an employee of a "commercial company" she had a responsibility to "win the business." This caused a conflict for her as she felt she had to propose evaluation solutions that were not as comprehensive as they could be.

For the other student, Jack, the scenario context was mostly ignored because he had his own authentic environment (his work environment) to which he was transferring what he was learning. Also, the organization he worked for did not seem in accord with the concepts in the fictional company. Thus, he preferred to focus on the subject content.

Nevertheless, these two students contributed well to the discussions in the subject and did not appear to be disadvantaged by the limitations they perceived in the use of the scenario.

2. What were the students' perceptions of the learning design (that is, the sequence of tasks, the resources provided and the support mechanisms supplied) in terms of facilitating their learning?

A primary aim of the course redesign was to facilitate students' construction of their own understanding of the content. Except for Hui Ying, all student interviews and the summative questionnaire results indicated that this intention was realised. Hui Ying felt that the tasks didn't help her understand the content. Jack and Kim shared Hui Ying's view but with less intensity. Kim suggested that more examples of evaluation proposals would have helped. Jack wasn't sure whether this "gap" served as a disadvantage or an opportunity, as he stated "if you don't provide guidance then students need to read widely in the resources. I found that really useful" (Interview).

The reading material made available on the "company web site" provided a rich set of resources to enable students to explore topics of interest. All responses from the summative questionnaire indicated that students found these resources helpful.

The regular online presence of the lecturer, and the overall friendly rapport and guidance of the Academic Advisor role, emerged as contributing factors in keeping the students engaged, motivated, and enthusiastic in the course. The teacher survey results highlighted that the students valued the interaction with the lecturer and her accessibility. Summative questionnaire responses showed that most students thought the role of Academic Advisor was effective in facilitating student learning.

3. Was the online implementation strategy effective in facilitating student learning?

The introductory workshop was conducted as three synchronous chat sessions plus e-mail interactions. Five chat sessions were conducted during the semester, and a final debriefing session was scheduled at the end of the semester. The average participation over the nine chat sessions was three students and the average duration for each session was one hour. The main issues discussed related to the assessment tasks.

Four of the five students who completed the summative questionnaire thought the chat sessions were effective because they provided the opportunity to clarify issues and ask questions. The low participation rate was viewed by Hui Ying as effective as she could interact with the teacher one-on-one. The main reasons stated for the low participation rate were the inconvenient time in which the chat was scheduled due to students' work commitments and technical problems experienced by two students. Three recommendations were put forward to improve the use of the synchronous discussion: (a) schedule the chat time later in the evening; (b) provide a more focused agenda for the chat session; and (c) set up small groups.

Students were encouraged to participate in six nonassessable online activities during the semester. The first three activities involved posting answers on a self-study task. Students were asked to post their results as an e-mail to the course mailing list. Student dedication was illustrated during these initial activities as all ten students completed the first activity and nine students completed the second and third online activity. The last three activities were structured as more open-ended discussion topics in the WebCT discussion forum and were instigated in accordance to the content presented in the announcements. Students showed less interest in participating in these online discussion topics, because they were focused on completing the assessment tasks. Overall, 30 messages were posted in the WebCT discussion forum, of which 13 were posted by the lecturer. Of the 17 messages posted made by the students, the majority were postings of answers to questions posed by the lecturer.

When students were asked why they thought little asynchronous online discussion occurred, three reasons were suggested: cultural differences as to the purpose asynchronous discussions serve (e.g., the posting of answers as opposed to posting messages to generate discussion); little incentive (discussion was not compulsory); and work commitments (many were too busy to participate).

The findings suggest that the teacher's assumption about students being self-motivated to use these interaction opportunities for the purposes of asking questions amongst themselves and generating discussion was not realised. The students' perceived purpose of using the asynchronous discussion forum appeared more focused on posting answers than asking questions (thus exposing their limitations in understanding the content).

Overall, students rose to the challenge of participating in a totally online course and they were generally satisfied with the level of interaction and support provided. Two students raised a significant issue when determining overall effectiveness of an online learning environment. They stated "little online discussion does not equate to ineffectiveness."

4. What learning outcomes were achieved?

Four of the five students who completed the summative questionnaire felt that they had achieved the intended learning outcomes. One student, Li Ming, felt that she didn't experience enough skill development in the course and thus had not achieved all the intended learning outcomes. When asked if they had achieved any unintended learning outcomes, three students indicated that they had developed generic skills such as internet search proficiency, experience using online communication tools and time-management.

From the teacher's perspective, all participants achieved the intended learning outcomes. The assessment tasks serve as evidence for this and grades ranged from Credit (65-74%) to Distinction (75-84%). A further observation that reinforced this perception was the attention to detail in the formatting of the student assignments, and the commitment of "playing the role" in the company. The teacher also evidenced a range of unintended learning outcomes, such as:

* Self-regulated learning and motivation. Several students explicitly demonstrated organisational skills, and commitment and persistence in the completion of their assessment tasks by submitting work before the due date. One student resubmitted his last assignment after the course was finished because he had thought of an additional idea while on holidays.

* Ampliative skills. Several students demonstrated the ability to extend and integrate knowledge on their own. For example, Jack felt he had gained the skills to apply evaluation and know what kind of resources to look for to further his knowledge. Hui Ying went beyond the requirements of one assignment by producing a model instead of a concept map.

* Reflective skills. Reflective components were deliberately built into every assessment piece and the students rose to this challenge. The richness of thought demonstrated by these students is indicative of the depth at which they were thinking about the subject content and their experience in this implementation.

5. What were the students' perceptions about their learning experience?

Of the six students interviewed, five rated their learning experience as positive. The nature of the tasks and their sequence, the resources provided, and the use of the scenario were the main contributing factors. Hui Ying, however, viewed her learning experience as negative because she was confused about the use of the scenario and felt that she had only achieved the intended learning outcomes in a superficial way. In terms of the course meeting student expectations, four of the six students interviewed stated that their expectations were met. Lee commented that the course had exceeded his expectations. Hui Ying and Li Ming responded that their expectations weren't met mainly because they felt the concepts were not taught in depth.

6. What improvements can be made to this course?

The suggested improvements offered by the students focused on three aspects of the course. First, in terms of the tasks, a collaborative assessment task to encourage students to participate in the online discussions and also reinforce the sense of company team spirit could be included, together with more samples/examples. Second, it was suggested that the implementation of the scenario could be more flexible to allow students to provide their own experience as their own scenario. Third, the use of a course web site that encompasses all communication tools may minimise confusion and facilitate more online interaction.

CONCLUSION

This effectiveness evaluation examined the revised course as a whole. However, there are many ways in which this course could have been evaluated to determine its effectiveness. Effectiveness evaluations can be narrowly focused or broadly focused, and can examine a number of dimensions such as effectiveness in terms of the interface, effectiveness in terms of learning outcomes achieved, or effectiveness in terms of the entire learning experience. It is dependent on the evaluation purpose--the decisions that can be informed from the evaluation. Thus, the framework by Reeves and Hedberg (2003) served as invaluable guidance prompting continual reflection on the overall purpose of the evaluation, and assisting the evaluator in the construction of the data collection instruments, and in analysing the data.

From this evaluation, it is plausible to suggest that the majority of students deemed the course effective. Of more significance, however, is the richness of data collected that can provide insight into why the students and teacher perceived the course as effective and how the course could be improved for subsequent offerings.

In summary, the evaluation found that the majority of participants rated the use of the company scenario and use of character as effective in facilitating their learning. Most participants rated their learning experience as positive and thought that the course encouraged them to construct their own understanding of the content. The synchronous and asynchronous discussions tools while used, were not used to the extent the teacher had intended. This was due to the teacher's assumption that students would engage in online interaction for purposes of asking questions amongst themselves and generating discussion. From the teacher's perspective, all participants in this evaluation achieved the intended learning outcomes as well as a range of unintended learning outcomes. From the students' perspective, the majority of participants felt that they had achieved the intended learning outcomes.

Two students, reported negatively on the scenario, which suggests that the use of scenario should be more flexible, to allow students with appropriate real-life contexts to substitute their own evaluation needs while still fulfilling the requirements of the course. Research by Herrington, Reeves, and Oliver (2003) also showed that students vary in their ability to suspend disbelief, and some need to be encouraged to engage in the fictitious world that has been created for them in the learning environment.

This article has described the rationale, design, and implementation of an online postgraduate course, together with the rationale, design, and implementation of an effectiveness evaluation of the course. As such it has aimed to provide an example of the kind of in-depth evaluation that is needed to gain substantiation for more innovative approaches to online learning--substantiation that moves beyond the enthusiastic endorsements of the proponents of such approaches. Further evaluation research into constructivist approaches to online learning will add additional evidence to the effectiveness of these learning environments.

References

Alexander, S., & Hedberg, J. (1994). Evaluating technology-based learning: Which model? In K. Beattie, C. McNaught, & S. Wills (Eds.), Interactive multimedia in university education (pp. 233-244). Amsterdam: Elsevier.

Bain, J. (1999). Introduction: Learning centered evaluation of innovation in higher education. Higher Education Research and Development, 18(2), 165-172.

Boud, D., & Feletti, G. (1997). The challenge of problem-based learning (2nd ed.). London: Kogan Page.

Boud, D., & Prosser, M. (2002). Appraising new technologies for learning: A framework for development. Educational Media International, 39(3/4), 237-245.

Bransford, J. D., Sherwood, R. D., Hasselbring, T. S., Kinzer, C. K., & Williams, S. M. (1990). Anchored instruction: Why we need it and how technology can help. In D. Nix & R. Spiro (Eds.), Cognition, education and multimedia: Exploring ideas in high technology (pp. 115-141). Hillsdale, NJ: Lawrence Erlbaum.

Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18(1), 32-42.

Duffy, T. M., & Cunningham, D. J. (1996). Constructivism: Implication for the design and delivery of instruction. In D. H. Jonassen (Ed.), Handbook of research on educational communications and technology (pp. 170-198). New York: Macmillan.

Evensen, D. H., & Hmelo, C. E. (Eds.). (2000). Problem-based learning : A research perspective on learning interactions. Mahwah, NJ: Lawrence Erlbaum.

Herrington, J., & Oliver, R. (2000). An instructional design framework for authentic learning environments. Educational Technology Research and Development, 48(3), 23-48.

Herrington, J., Reeves, T., & Oliver, R. (2003). Patterns of engagement in authentic online learning environments. Australian Journal of Educational Technology, 19(1), 59-71.

Jonassen, D., Mayes, T., & McAleese, R. (1993). A manifesto for a constructivist approach to uses of technology in higher education. In T. M. Duffy, & D. H. Jonassen (Eds.), Designing environments for constructive learning (pp. 231-247). Berlin: Springer-Verlag.

Kindley, R. W. (2002). Scenario-based e-learning: A step beyond traditional e-learning. Retrieved May 8, 2005, from www.learningcircuits.org/2002/may2002/kindley.htm

Oliver, R., & Herrington, J. (2002). Explore, describe, apply: A problem focussed learning design. Retrieved May 8, 2005, from Learning Designs Web site: http://www.learningdesigns.uow.edu.au/guides/info/G4/index.htm

Phillips, R., Bain, J., McNaught, C., Rice, M., & Tripp, D. (2000). Handbook for learning-centred evaluation of computer-facilitated learning projects in higher education. Retrieved May 8, 2005, from http://www.tlc.murdoch.edu.au/archive/cutsd99/handbook/handbook.html

Reeves, T.C., & Hedberg, J.G. (2003). Interactive learning systems evaluation. Englewood Cliffs, NJ: Educational Technology Publications.

Robyler, M.D., Edwards, J., & Havriluk, M.A. (1997). Integrating educatioinal technology into teaching. Upper Saddle River, NJ: Prentice Hall.

Whitcomb, C. (1999). Scenario-based training at the FBI. Training & Development, 53(6), 42-46.

Wills, S., & Ip, A. (2002). enRole, research, react, resolve, reflect: Developing and using online role play learning designs. Retrieved May 8, 2005, from Learning Designs Web site: http://www.learningdesigns.uow.edu.au/guides/info/G1/index.htm

Wu, H.J., Huang, K.T., Chen, H., & Wu, H.P. (2003). Scenario-based learning: An application for the national digital library. Retrieved May 8, 2005, from http://www.taskco.com.tw/en/kt/k06.htm

SHIRLEY AGOSTINHO, JIM MEEK, AND JAN HERRINGTON

University of Wollongong, New South Wales, Australia

[e-mail]

[e-mail]

[e-mail]