Printer Friendly

Evaluating pollutsim: computer supported roleplay-simulation.


The e-environment promises solutions to three challenges currently facing higher education: Flexibility, access and cost. While e-learning has been enthusiastically embraced by administrators and teachers, to date there has been more attention directed to development than to judging the effect and effectiveness of new approaches. Pollutsim, a computer supported roleplay-simulation, was implemented in 1996 and has been systematically evaluated from technical, educational design and student learning perspectives. In this paper we provide an overview of the evaluation process that began with a review of existing curriculum and provided formative feedback throughout several iterations of the project.


The growth in the use of the online environment, particularly communications information technologies (CIT) is accompanied by high level expectations regarding the effect these will have on higher education and changing teaching and learning. The environment promises solutions to three challenges currently facing higher education: Flexibility, access and cost (Daniel, 1997). These technologies have been embraced by administrators and teachers, particularly in higher education contexts. Notably, despite claims for the positive outcomes of teaching with online technologies there has been a shortage of rigorous research or evaluation evidence to support such claims. A recent investigation of Australian teaching innovations using information technologies highlighted the absence of serious assessment of the effect of the innovations on learning outcomes (Alexander & McKenzie, 1998). Alexander (1999) has suggested that the paucity of serious assessment of the effectiveness is a consequence of lack of understanding about the ways in which effectiveness can be gauged and Laurillard (1993) argues that it is development that is the focus of attention with only `lip service' paid to the evaluation of educational technology projects.

In this paper we describe the processes of assessing the effectiveness of a computer supported roleplay-simulation (CSRS) used in teaching university students about the complexity of decision-making involved in managing a contaminated site.

Role Plays and Simulations

Roleplay-simulations (McLaughlan & Kirkpatrick, 1998) combine the attributes of both simulations and role-plays Where participants adopt a functional role or persona within a simulated environment or scenario. They are problem-based units of learning set in motion by a particular task, issue, policy, incident or problem. Roleplay-simulations create learning environments that immerse learners in authentic learning. They may be designed so that the problems that are to be addressed by participants are either implicit or explicit. Participants learn about the person or role, problem and/or the situation specific to the subject area as a consequence of the interaction between participants in their personae and the scenario (Errington, 1997). In recent years there has been growing interest in using CIT to support roleplay-simulations (McLaughlan & Kirkpatrick, 1998; Vincent& Shepherd, 1998).

PollutSim: An Environmental Decision-Making Simulation

The computer supported roleplay-simulation Pollutsim, provides a context where learners develop an insight into issues related to the management of a contaminated site. Pollutsim was first used in 1996 and has been refined in response to evaluation and student feedback. The Pollutism scenario involves an industrial company, which operates an arsenic-producing smelter. Company-owned vacant land adjacent to the smelter has arsenic and petroleum hydrocarbon-contaminated soil and groundwater. The Pollutsim scenario is designed to create conflict among the personae centered around a range of environmental issues resulting from the potential impact of the contamination on community resources in the area.

Pollutsim is designed to achieve the following learning objectives:
 * Identify the political, social, economic and scientific dimensions to
 decisionmaking in an environmental conflict at a contaminated site;

 * Identify the responsibilities and appropriate responses for characters in
 the roleplay-simulation and;

 * Develop communication, negotiation and decision-making skills (McLaughlan
 & Kirkpatrick, 1999)

Learners adopt various personae in the scenario in order to respond to issues and problems that arise as they seek to deal with potentially impacted community resources. The roleplay-simulation involves up to 24 personae comprising business, regulatory, political, media and citizen groups. Personae are randomly allocated to participants so there is no matching of discipline area, background knowledge or previous experience. Undergraduate and postgraduate university students studying courses in Engineering and various types of resource management have participated in Pollutsim. All interactions occur online supported by email, phone and synchronous chat. Relevant resources are provided online and links are available to appropriate resources.

Pollutsim was initially developed in response to perceived shortcomings in the original curriculum. It was intended that Pollutsim would provide an opportunity for students to develop an understanding of the complexities of environmental decision-making in response to a contaminated site, develop a range of perspectives on a problem and while developing scientific knowledge and technical skills relating to contaminated sites management. It was also hoped that the use of the online environment would provide greater flexibility for students and allow the roleplay-simulation to operate over a realistic extended time-frame. Pollutsim comprises one module within a semester long course: Contaminated Sites Management. This is taken by undergraduate Engineering students and students enrolled in various post-graduate courses including Coastal Resource Management and Groundwater Management.

Pollutsim was first run in 1996 using group discussion software (HyperNews) and involved distributed online interactions and one scheduled face-to-face meeting for debriefing the simulation. Evaluation of the original version of Pollutsim focused on scenario design and technical aspects of the roleplay-simulation and the e-learning environment. As a consequence of the evaluation results in 1997 both private email and group email discussion were provided using Topclass e-learning software and additional face-to-face meetings were included. These options expanded the range of communication media available for student communication. Subsequent Pollutsim design allowed students to select personae.

Critical Tasks in PollutSim

Successful engagement in the simulation requires students to communicate with each other to negotiate and resolve conflicts relating to the management of a polluted site. Specifically, students needed to be able to perform communication tasks at a range of levels of sophistication appropriate to the particular context in which they occurred. These included communicating effectively, negotiating positions, presenting statistical and textual information, resolving conflict, making decisions, responding to other characters and information, working collaboratively, and assessing analytical data. The performance of these tasks requires students to act independently and in conjunction with others while the emphasis placed on each task varies according to particular roles and contexts. These tasks were all performed in Pollutsim using either face-to-face meetings, telephone, email discussion or chat. Students were supported through the following organised activities:
 * An orientation meeting in the first week provided students with an
 opportunity to meet face-to-face, introduced the course, provided guidance
 for student selection of their preferred role and instruction in the use of

 * An electronically supported ice breaker comprising an established
 training game was adapted for use via email and provided a means of
 familiarising students with email discussion and the process of reaching

 * A face-to-face round table meeting in week three provided an opportunity
 for students to introduce their personae, and for the release of that data
 that create the conflict between players and sets the simulation in motion.

 * Another round table meeting was held after personae had worked towards
 resolution of the nature of the problem. The meeting brings together all
 stakeholders with their individual positions regarding available
 remediation options. Possible outcomes and actions that might meet
 stakeholder needs were explored. It was the responsibility of specific
 personae to facilitate this meeting as part of the simulation.


Effective evaluation or judgment about the effectiveness of online or e-learning projects should involve collection of evidence at all stages of development and from multiple sources (Alexander & Hedberg, 1994; Kirkpatrick, 1994). Specifically, evidence should be collected at the stage of initial design, during development, throughout implementation and should also consider the impact of the project at an institutional level.

Evaluations in 1997 focused on assessing aspects of the learning design of the CSRS, technical dimensions, the appropriateness and effectiveness of various media for supporting communication tasks within the roleplay-simulation and effectiveness of Pollutsim in achieving the learning objectives. Evaluation strategies included: analysis of the sub-tasks of the roleplay-simulation and effectiveness of various media to support necessary interaction; student feedback via survey and focus group interview; and measurement of the communication needs of participants. These technical and design issues were addressed in the re-design of aspects of Pollutsim. Student written assignments were analysed to indicate the effectiveness of Pollutsim in achieving student learning outcomes. The 1998 evaluation of Pollutsim focused mainly on student learning outcomes through analysis of assignments and student feedback about the operation of Pollutsim and their perceptions of its effectiveness in facilitating learning.

Communication Needs of Participants

The following instruments were used to identify the effectiveness of particular media in supporting learner communication needs within the scenario.

User acceptance of technology

This comprised ten items adapted from previous instruments that focused on perceived usefulness and ease of use of various media (Adams, Nelson & Todd, 1992; Davis, 1989). On the basis of their experience in Pollutsim, respondents ranked four media in relation to statements that reflected ease of use and usefulness.

Media choice instrument

Twenty incidents were developed that presented scenarios covering a range of eight conflict management behaviours (16 items) and five different conflict levels (5 items). The classification system for these tasks was based on behaviours in group work and conflict management situations (Poole, Holmes & Desanctis, 1991). These were presented in the context of the need to complete a simulation task similar to the one in which participants had been involved. Participants ranked the communication media: email; synchronous chat; teleconference; and face-to-face according to their preference for that medium in managing the communication needs described in each incident.

Assessment of Student Learning

Assessment design for the simulation clearly related assessment tasks with the objectives of the simulation. A key learning objective of Pollutsim was that participants would be able to identify the political, social, economic and scientific dimensions of decision-making during an environmental conflict. A concept map and a reflective essay addressing environmental negotiation in contaminated sites provided evidence of student learning in relation to this objective. The reflective essay was analysed guided by the principles of SOLO (structure of the observed learning outcome) (Biggs, 1992; Biggs & Collis, 1982). SOLO supports an analysis of the structural complexity of responses and classifies written responses hierarchically. Classifications include prestructural responses (at the lowest level, these contain irrelevant information); uni- and multi-structural responses (show understanding of one or a few basic aspects); relational responses (provide evidence of understanding by integrating and structuring relevant concepts); and multi-structural extended abstract responses (generalise beyond the information given, yielding higher order principles). The essay was also assessed against criteria associated with awareness of the complexities of decision-making and understanding of the processes and strategies in making environmental decisions.

A key objective of the simulation was to identify the political, social, economic and scientific dimensions to decision-making during an environmental conflict. In 1997, 22% of the written essay responses demonstrated characteristics of low level unistructural organisation of ideas including simple reiteration of the events of the simulation (15%), and the identification of relevant social, political, economic and scientific issues in a general context (7%). The remaining 78% of essays integrated these issues with the characters and events from the simulation and discussed the implications. In 1998, the debriefing phase was further developed as a more structured, interactive process, providing greater support for the organisation of relevant ideas. The subsequent reflective essays of all participants who attended the face-to-face debriefing demonstrated higher order learning evidenced by generalising the events within the simulation to underlying concepts. This highlights the importance of facilitated group reflection process where participants can reflect on and articulate their learning experience and clarify central concepts and principles of the decision-making process. Concept maps were designed to assist participants clarify and structure their ideas before writing the essay.

An analysis of the content from the electronic dialogue exchange by participants showed that in both years most interactions related to the negotiation of positions and making decisions based on rhetoric rather than scientific data. There was little evidence of extra curricular investigation of the scientific basis for making decisions. It appears that the simulation was successful in allowing participants to integrate and communicate their existing knowledge rather than collecting in depth knowledge specific to the scenario. Only those participants with no training in groundwater contamination felt that they had learnt some of the scientific dimensions associated with this management problem.

Pollutsim was also intended to develop understanding of the responsibilities and appropriate responses for characters in the simulation, and to develop communication, negotiation and decision-making skills. Achievement of these objectives was evaluated using peer assessment of: persona participation; goals and strategies; use of appropriate language; preparation; prompt responses; and the creation of opportunities to further the resolution process.

Examination of the electronic messages during the simulation showed that all participants stayed within appropriate boundaries for their character without facilitator intervention. The role profiles provided by the participants indicated general awareness of professional responsibilities associated with their roles and the use of external web sites in 1998 to supplement persona development assisted participants to collect the information they required to develop this understanding.

The final learning objective of the simulation was to develop communication, negotiation and decision-making skills in the participants. A key indicator of the level of these skills was the quality of interactions within the simulation, judged by participants' use of appropriate information and strategies to achieve their aims and the effectiveness of communication in achieving desired outcomes within the scenario. Participant feedback indicated that all students believed the simulation gave them the opportunity to develop these skills. Most participants demonstrated an awareness of the limitations and benefits of both face-to-face and electronic dialogue in achieving different communication and decision-making tasks.

Student Feedback

While the extent to which a teaching program achieves its stated learning objectives is a primary measure of the effectiveness of that program, student responses to teaching innovations are also important. Students' responses to a teaching context are likely to influence the extent to which they engage with learning (Ramsden, 1992). An end of semester subject evaluation provided information about students' responses to Pollutsim.

Student feedback indicated that students appreciated the flexibility offered by the online delivery, they reported that the activities encouraged them to become more independent learners. In particular they found Pollutsim an engaging and worthwhile learning experience. While the use of asynchronous discussion facilitated communication between students, some believed the lack of face-to-face contact made it difficult for them to establish the rapport necessary to work effectively.

Students reported that the CSRS helped them develop an awareness of the need for and the skills of communication and negotiation in contaminated sites management. They reported that the enthusiasm of other participants played a significant role in the effectiveness of the simulation. While the extended time-frame of the roleplay-simulation was desirable because it resembled real life, students felt this made it difficult to sustain interactions and involvement. It may be desirable to limit the duration in order to sustain interest and involvement.

The need for timely and informative teacher feedback emerged as key concern for students. They found opportunities to see the work of others valuable but expressed a need for information as to their progress. In part the length of turn around time for teacher feedback was extended because of the process of peer assessment that had been implemented. This request for more immediate and informative feedback is not surprising as students have fewer opportunities to obtain direct and indirect feedback when they are not physically present on a regular basis. In addition the use of computer mediated environments removes non-verbal and incidental cues about the appropriateness of responses and ideas.

Other Sources of Evidence

We also gathered information relating to the educational appropriateness of the approach in its immediate context via peer review of the project through conference presentations and publications. This feedback informed subsequent development.

Finally, we were concerned with investigating the institutional impact of the project. Two aspects of what Alexander and Hedberg (1994) and Kirkpatrick (1994) consider salient in this area of impact were the extent to which involvement in Pollutsim contributed to the development of generic capabilities and assessment of the extent of transfer and maintenance of the project. Evaluation of submitted student work included examination of the extent to which specified graduate attributes (or generic skills) were demonstrated. In addition, following approaches from academics at two other universities planning began for a collaborative adaptation of the Pollutsim project.

In 2000 Mekong e-Sim, a CSRS was implemented collaboratively between staff and students at the University of Technology, Sydney, Adelaide University, University of Sydney, University of New England in Australia and Sepang Institute of Technology, Malaysia. Mekong e-Sim developed from the experiences of Pollutsim and draws on the educational design principles that were successful in Pollutsim. It utilises the same pedagogical framework and requires students to engage in the similar activities to Pollutsim although these are embedded in a different context and there are disciplinary differences in the subject specific knowledge that is developed.

Mekong e-Sim centers on a realistic international problem that develops specific knowledge and generic skills relating to political, social, economic and scientific dimensions of decision-making in natural resource management conflicts. Mekong eSim also supports the development of communication, research, critical thinking, negotiation and decision-making skills and an appreciation of cultural differences and approaches. Additionally, it is designed to develop generic information technology skills. This CSRS was evaluated drawing on interviews with all staff involved, interviews and focus groups of students from each site, analysis of student learning as evidenced in a number of written assignments, analysis of email postings for the duration of the roleplay- simulation.


Online learning can provide students with greater flexibility, increase access to education, assist in the development of information technology skills and literacies and support collaborative work among distributed learners. However, it is important that as educators embrace new technologies that the use and effectiveness of these are rigorously evaluated and the results used to inform future developments. The evaluation of CSRS occurred at all stages of the cycle, beginning with its initial design. This process included an analysis of current curriculum to identify strength and weaknesses, and shortfalls in student learning. It also sought to identify teaching/learning processes likely to bring about the desired learning outcomes and ways in which the new approach would facilitate the desired learning process and outcomes. We continued to collect evidence that would allow us to determine how functional our approach was, how workable the technology, the design and use of technology was and how students responded to the e-environment tools and tasks. Continuous formative monitoring provided data about the influence on learning and informed future modifications. At various stages of implementation we monitored the extent to which intended learning outcomes were achieved, making use of outcomes focused assessment tasks and analysis of student performance as well as student feedback collected via interviews and surveys. The development and implementation of Pollutsim and Mekong e-Sim two CSRS were informed by evaluative data collected from multiple sources at critical stages in the development in the life of the project.


Adams, D.A., Nelson, R,R., & Todd, P.A. (1992). Perceived usefulness, ease of use, and usage of information technology: A replication. MIS Quarterly, June 227-247.

Alexander, S. (1999). Innovative projects involving CIT. Higher Education Research and Development. 18(2), 173-184.

Alexander, S. & Hedberg, J. (1994). Evaluating technology based learning: Which model? In K. Beattie, C. McNaught & S. Wills (Eds). Multimedia in higher education: Designing for change in teaching and learning. (pp. 233-244). Amsterdam: Elsevier.

Alexander, S. & McKenzie, J. (1998). An evaluation of information technology projects in university learning. Canberra; Australian Government Publishing Service.

Biggs, J. B. (1992). A qualitative approach to grading students. HERDSA News, 14,(3), 3 - 6

Biggs, J. B. & Collis, K.F. (1982). Evaluating the quality of learning: The SOLO taxonomy. New York, Academic Press.

Davis, F.D. (1989). Perceived usefulness, perceived ease of use and user acceptance of information technology. MIS Quarterly, September, 319-342.

Daniel, J.S. (1997). Why universities need technology strategies. Change, 29 (4), 1117.

Errington, E. (1997). Role play. HERDSA Green Guide No 21. Canberra: HERDSA.

Kirkpatrick, D.L. (1994). Evaluating training programs: The four levels. San Francisco: Berrett-Koehler.

Laurillard, D. (1993). Rethinking university teaching: A framework for the effective use of educational technology. London: Routledge.

McLaughlan, R. & Kirkpatrick, D. (1998). Developing a framework for effective electronically supported simulations. Paper presented to the annual Higher Education Research Development Society of Australasia conference (HERDSA), Auckland, New Zealand, July., (Accessed 10 January 2001)

McLaughlan, R. & Kirkpatrick, D. (1999). A decision-making simulation using computer mediated communication. Australian Journal of Educational Technology, 15, 3, 242256.

Poole, M.S., Holmes, M., & Desanctis, D. (1991). Conflict management in a computer-supported meeting environment, Management Science, 37, (8), 926-953. Ramsden, P. (1992). Learning to teach in higher education. London: Routledge.

Vincent, A. & Shepherd, J. (1998). Experiences in teaching Middle East politics via Internet-based role-play simulations. Journal of Interactive Media in Education, 98, 11, (Accessed 12 June 2001)

Professor Denise Kirkpatrick, Ph.D., is Director of the Teaching and Learning Centre where she coordinates the development of on and off campus teaching and learning. Robert McLaughlan, Ph.D., is a senior lecturer in the National Centre for Groundwater Management.
COPYRIGHT 2001 Rapid Intellect Group, Inc.
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2001, Gale Group. All rights reserved. Gale Group is a Thomson Corporation Company.

Article Details
Printer friendly Cite/link Email Feedback
Author:McLaughlan, Robert G.
Publication:Academic Exchange Quarterly
Geographic Code:1USA
Date:Dec 22, 2001
Previous Article:Assessment of thinking levels in students' answers.
Next Article:Assessing higher levels of learning in post-secondary education. (Online Instruction).

Related Articles
Latest edition of DemoShield 7.0. (IT News).
Modeling, Simulation Sector to Undergo Change.
Superior simulation.
Superior simulation.
Superior simulation.
Team play for leaders.
Superior simulation.
Superior simulation.
The Executive Challenge.
Virtual Leader (updated).

Terms of use | Copyright © 2017 Farlex, Inc. | Feedback | For webmasters