The Health Care Game: An Evaluation of a Heuristic, Web-Based Simulation.

The results of an evaluation of a new web-based educational game centred on the health sector are presented in this article. The game is designed to promote information-seeking skills and provide students with opportunities to interact with each other and the health system. The discussion is based on the context of literatures on new technologies in education, games and simulations. Evaluation of a trial of the game was conducted using pre and postgame student questionnaires and a focus group discussion with student participants. Results demonstrated the effectiveness of the game in improving learning outcomes.

"Technology can offer the means for learner control where students have greater flexibility and self-determination to construct their own meaning and acquire active knowledge" (Koppi, Lublin, & Chaloupka, 1998)

Recent years have witnessed a profusion of new educational strategies in higher education. Here some key strands of the literature on these strategies are reviewed and the evaluation of the Health Care Game, a new web-based, learner-centred, heuristic tool designed for Australian teachers and students involved with medical, health science, or health services management curricula are discussed.

Among a rapidly expanding list, there are various claims for the successful use of problem-based learning in medical education (Boud & Feletti, 1991; Barrows, 1986); inquiry-based learning in nursing education (Feletti, 1993); learner-centred approaches for radiography students (McKay & Kember, 1997); computer-presented problems for doctors (Thompson, Jolly, Macdonald, Gookson, Holman, & Keech, 1987); and computer-based medical quiz games (Mooney & Bligh, 1998). Educators are developing computer packages designed to simulate various aspects of the health system including, for example, the management of a day surgery (Jones, Latif, Murray, & Hedley, 1992) and the use of IThink! for modeling hospital organisational complexity (Senge, Roberts, Ross, Smith, & Kleiner, 1994).

More broadly, there is widespread use of simulations, role-plays, software packages, interactive multimedia, and web-based learning tools in an attempt to deepen and broaden the learning experience. No academic discipline appears to be untouched, with recent reports of applications in fields as diverse as, for example, accounting (Benson, Alison, & Arger, 1996), strategic management (Perry, 1995), business administration (Schoenecker, Martell, & Michlitsch, 1997), economics (Gremmen & Potters, 1997), and welfare work (Hughes, 1992).

The increased use of new techniques does not of itself provide empirically sustainable justification that their application improves teaching outcomes or the learning experience. Each case must be dealt with on its own merits. Soundly designed and executed evaluation strategies become extremely important. Although there are no data available, logic suggests that expenditure levels on computerised games, simulations, and online teaching must be considerable and rising. Although the use of economic cost-benefit analyses is clearly difficult to apply in teaching and learning, evaluation is possible and warranted in the case of the release of new games and simulations, even in the absence of an economic component to the assessment process.

Technologically Sophisticated Games and Simulations in Higher Education

There appears to be strong but by no means ubiquitous support for technologically oriented games and simulations in the higher education literature. Adherents argue, inter alia, that technology-based games and simulations afford participants opportunities to learn under "real life" conditions (Coppard, 1976; Perry & Delahaye, 1990), that decision-making dynamics can be simulated to be closer to organisational processes (Wolfe, 1985), that they help model complexity (Hughes, 1992; Boulding, 1956; Rowlands, 1989), and they illuminate behaviour in social systems (Checkland, 1981).

Regarding web-based teaching and learning, supporters have pointed to a number of benefits including equity of access to distant learners (Webster & Hackley, 1997), the ease with which dissemination of information can take place (Barron, 1998; Zhao, 1998), the facilitation of collaboration (Alexander, 1995; Ewing, Dowling & Coutts, 1998) and possible increases in motivational levels (Hakkarainen, Lipponen, Jarvella, & Niemivirta, 1999). On the issue of collaboration in learning, Collis (1998) identified six problems in group based project work. These consisted of difficulties centred on: maintaining course cohesion and momentum; levels of collaboration; effective communication; a group approach; self and intergroup evaluation; and linking group activities to other aspects of the course. At the heart of her package of solutions was a comprehensive world wide web (WWW or Web) site which facilitated, inter alia, course integration, group and individual access, and, crucially, human to human interaction.

What is the relationship between these newer technologies and teaching and learning, and what is the evidence that they are contributing to more effective teaching and learning? Kolb (1984) argued that experiential learning has been demonstrated to enhance learning and encourage students to adopt an ongoing approach to learning. Following Koppi, Lublin, and Chaloupka (1998) and others we take the view that effective teaching engages the learner in intelligent learning. We also believe that effective learning is anchored in a deep, not just a surface approach to the curriculum. Koppi, Lublin, and Chaloupka (1998) maintained that there are two main contributions of technology-enhanced learning: to facilitate experiential learning through mimetic representations of the world, and to encourage communication and collaborative interaction.

There is also much support amongst scholars for learner-centred approaches (Tobin, 1990; Clifford, 1990). McKay and Kember (1997) suggested that students prefer learner-centred curricula, which encourage independent learning, and they argue that this type of curricula is associated with deeper learning. Taking this idea further, Taylor's (1996) report of an extensive enquiry into students' perspectives and conceptualisations of their learning calls for educational strategies, which invite students to become "active and critical co-constructors of their opportunities to learn." More recently, Ewing, Dowling, and Coutts (1998) emphasised the importance of learner-centred, constructivist approaches to web teaching.

Where there is criticism of games, simulations, and technologically mediated learning approaches, it usually takes the form of raising challenges to advocates to evaluate their work. Some commentators have raised more specific objections. For example, in a recent study Johnson, Aragon, Shaik, and Palma-Rivas (2000) found that student satisfaction was slightly more positive for face-to-face teaching than online learning, and Conlon (1997) contended that difficult, complex teaching, and learning problems are not easily resolved through online approaches. Fabros and Young (1999) argued for thorough reviews and critical analyses of education programs and projects using the newer technologies. One recent response to this call came from Sherry and Billig (2000) who attempted to specify what the constituents of good online conversation might be. Some of these were encouraging goal-directed conversations, facilitating enquiry-centred learning, respecting differences, resolving conflicts, publishing guidelines, nurtu ring ongoing communication, aligning online and class-based work, and limiting excessive message proliferation.

The authors argue that while self-directed learning is important, it is crucial to strike an appropriate balance between educator-directed and student-directed learning. Looked at another way, one can reflect on the structure of learning. Too little structure would lead to anarchy. Too much would lead to excessive control over the learning agenda. Moreover, if the distinction between surface and deep learning approaches centres on words, signs, or superficialities in the former and intentions, meanings and understandings in the latter (Marton, 1988; Ramsden, 1992; Booth, 1997), then for us a web-based game would need to have certain features. These include that it would be, under Marton's (1988) schema, structurally holistic rather than atomistic. An effective game would also encourage heuristic behaviour, engage the student in information-seeking behaviour, be predicated on meaning-creating principles, reward student understanding, and be balanced between being overly educator-centred and learner-centred. I n addition, although establishing the framework for learning might be the task of the teacher, the learner ought to have opportunities to work iteratively with other students and the subject coordinator to shape what is learned and how it is learned. In Ramsden's (1992) words, "deep approaches are associated with activity and responsibility in learning."

EVALUATION

It follows that an important consideration in the development, refinement, and release of new games and online teaching is to determine and institute an evaluation process (Saunders & Gunn, 1990; Williams, 1992; Rolfe, 1991). Laveault and Corbeil (1990), following Thatcher (1986) and Kryukov and Kryukova (1986), specified an eight stage process to assess the impact of simulation games. This work contributed to the general approach to evaluation of the health care game, although each phase of the evaluation process was not mapped directly to each of Laveault and Corbeil's eight evaluation stages.

Thus, it is a central tenet of this article that education, games, and simulations are not ends in themselves but are tools, and as such need to be evaluated to assess whether they are successful in achieving specific learning outcomes. Several researchers have examined processes by which interactive games, which seek to achieve a range of learning outcomes, may be evaluated (Rolfe, 1991; Wolfe, 1985; Greenlaw & Wyman, 1973). Evaluation strategies must be closely aligned to the design of the game. Important elements of an evaluation include examination of the validity, reliability and utility of the game.

Having briefly grounded our analysis in some central themes concerning educational frameworks, new learning technologies, and the importance of evaluation, the remainder of this article provides a description of the Health Care Game and the results of an evaluation undertaken using the prototype of the game. The evaluation of the game sought to examine three things: assessment of students' understanding of, and ability to participate in, the game; determination of how effective students thought the game was at achieving specific learning outcomes; and assessment of students' achievement of specific learning outcomes.

The Health Care Game

Two significant problems are encountered by medical, health sciences, and health administration students undertaking introductory subjects about the Australian health care system. The first is that students' struggle with the complexity and diversity of the health system. Even if they have extensive work experience in the system, they tend to see it from one perspective and neglect to adopt pluralist frames of reference. The second is the rapid rate of change within the health system. Information learned in the first year is often not relevant or is out of date by the time the student graduates. The most important learning outcomes for students undertaking such introductory subjects are: an understanding of how the health system is organised and operates, knowledge of, and skills about, how to obtain up to date information about the health system, an understanding and appreciation of health care policies and the implications these have for people of different social, political, economic, and cultural perspec tives in the community. These learning outcomes are the focus of the Health Care Game.

To address these teaching and learning challenges we designed a web-based, heuristic game with the aim of encouraging health information-seeking skills and providing students with opportunities to interact with the health care system and solve realistic hypothetical situations. The Health Care Game simulates health events and different contexts within which consumer choices are made. It focuses on problem-based learning and also includes debriefing and reflective activities. These activities are reported to be effective in achieving improvements in learning outcomes (Williams, 1992; National Board of Employment, Education & Training, 1994). The game involves students working in groups. This encourages co-operative learning, allowing students to work together to achieve both individual and group outcomes (McDonnell, 1994).

In 1996 we trialed the idea of asking students to work in groups as a family and to investigate a hypothetical health event which the family encountered. This involved students seeking information and interacting with the health system. This activity generated considerable discussion by students and resulted in many students seeking information outside the bounds of the immediate questions raised by the health event. The success of this activity acted as a catalyst for the development of the Health Care Game.

DESIGN

There is no other health care game aimed at educating students about the Australian health care system. Other games, particularly in the area of business and marketing (Mason & Perreault, 1995; Cotter, 1995) have been developed. These are being used by Australian business schools and can form the basis of ideas about how effective games might work. There is a considerable literature on the design and use of games and simulations in tertiary education. At their heart, games and simulations enable practice under defined conditions. The design process for the Health Care Game followed the game design steps provided by Thatcher and Robinson (1994).

The Health Care Game is accessed through a website (http:// www.itl.usyd.edu.au/health/) [1]. The game comprises four families from different cultural, socio-economic, and demographic backgrounds. Family members encounter a series of health events during the course of the game. Each event generates problems for the family member and students are required to seek out information and provide the family with sufficient information so that health decisions can be made. For example, a family member is involved in a car accident requiring hospitalisation and rehabilitation. Students require information about how the family member will navigate the health system, which services will be needed, how much these will cost, and choices the patient has.

Students work in competing teams of six. Each team member contributes to a common team discussion log and outlines his or her progress. The discussion log is a diary detailing to whom students spoke and from where they obtained information relevant to the event questions. Team members use the discussion log to communicate with each other and to gain consensus regarding their answers to the event questions. Teams formulate responses to the event questions and these, along with their discussion log, are entered and submitted online to the subject coordinator.

Team responses and the discussion log are marked and correct information related to the event questions posted by way of the website back to the students. Students can communicate through the discussion log or email with each other and with the subject coordinator. The subject coordinator is able to view the discussion logs of teams at any time and in this way can monitor whether teams are having difficulties and provide feedback to students as they progress towards answering the health event questions. Assessment for individual students is based upon team answers to the event questions and on the individual student's contribution to his or her team's discussion log. The technical features of the game permit the subject coordinator to oversee progress and provide feedback to students as they play the game. This enhances the opportunity effectively to combine formative and summative assessment techniques.

The content of the Health Care Game is dynamic. The game contains a database of events from which the subject coordinator may select. Thus new events may be added or modified, event answers may be updated and family histories can be changed. This occurs through a separate administration website accessible to the course coordinator. This design feature means that the game is suitable for teaching health science students from a range of disciplines and the content remains up to date and relevant.

METHOD

Fifty-five students participated in a four week trial of the prototype of the game in June, 1998. This followed an informal evaluation of an earlier, paper version of the game. As part of the four week trial of the game, students completed a pre and post questionnaire.

The questionnaire consisted of four sections: (a) interest in, and knowledge of, aspects of the health system, (b) access and experience with computers, (c) views of team work at university and (d) student demographics. Answers to questions in section (a) required a combination of ratings on 5 point Likert scales (1 strongly disagree to 5 strongly agree) and open ended questions. Section (b) contained some yes no questions, (for example, do you have access to a computer at home?) as well as questions requiring an answer on 4 or 5 point scales. All questions on teamwork used 5 point Likert scales. Demographic questions enquired about sex, country of birth, and if born overseas, the age of the student at time of arrival in Australia.

The game was introduced to students following nine weeks of traditional course lectures. The pre-game questionnaire was administered in the class in which the Health Care Game was introduced. Students were informed that the questionnaire was not a test and that the results would be used to provide information about how the Health Care Game could be improved. Students were advised that they would receive another questionnaire at the end of the course. It was not pointed out that this would be the same questionnaire as students might then be inclined to try to remember their answers to the first survey. The post game survey was administered in class four weeks later, after the game had been completed. As this was the last class of the course and all assessments had been completed, attendance was not as high as in the initial class. Comparisons between pre and post game questionnaire responses were made using t-tests for related groups.

A focus group of 11 students was also conducted by an academic member from the University's Centre for Teaching and Learning. During the focus group students were asked about their overall impression and understanding of the Health Care Game, their experiences at working in groups and suggestions for improving the game. In addition, we evaluated the Health Care Game against Koppi, Lublin, and Chaloupka's (1998) desirable outcomes of effective teaching and learning.

RESULTS

The majority of students were female (69%) and born in Australia (60%). Of the 40% (n=22) of students born overseas most were born in South-East Asia. Of students born overseas, the average age at time of arrival in Australia was 10 years with a range of 2 to 27 years (median = 8.5).

Most students (89%) had access to a computer at home, rated their computer skills as average or above that of their classmates (85%), and had used the Web previously at school or a university (85%). Fewer students (42%) had access to the Internet from home.

Pre and post questionnaires were available for 37 students (67%) as 18 students failed to complete a questionnaire at the end of the pilot. Some students who failed to complete the final questionnaire were included in the focus group. A comparison between the prequestionnaire responses by those who completed both questionnaires and those students who only completed the first questionnaire revealed no significant differences in the responses of the two groups.

Comparisons of students' pre and post survey responses showed that after the game students rated their knowledge of the health system (t=-4.45, df=35, p[less than]0.001), Medicare (t= -3.15, df=36, p[less than]0.01) and private insurance significantly higher (t=-3.45, df=35, p[less than]0.001) than before the game (Table 1). Students also rated their interest in the Australian health care system as significantly greater after completing the game (t=-2.25, df=36, p[less than]0.05).

Two questions were aimed at assessing students' factual knowledge about the way in which public and private health insurance works in Australia. These questions included a short scenario about a patient admitted to hospital for an operation. In one scenario the patient was privately insured and in the second she was a public patient. Students were asked about the extent to which Medicare and/or private insurance would cover the hospital costs in the two situations. After completing the Health Care Game, 49% of students correctly answered scenario 1 and 95% scenario 2. This compared with 27% and 58% respectively who correctly answered scenarios 1 and 2 prior to playing the game.

To determine whether the game assisted students locate possible sources of health information within the health system, the pre and post questionnaires asked students to list the sources of up-to-date information about the health system which they could access for a friend who had just arrived from overseas. In the pre game questionnaire the average number of sources identified was 2.4, with 24% of students able to list fewer than two options. This compared to a mean of 3.0 information sources in the post game questionnaire with only 8% of students unable to list fewer than two options.

Students were undecided prior to playing the game about whether they learn more with a group or solve problems best independently. These views did not change. Following the game, students remained undecided about whether they enjoyed working in groups, but reported that they were now significantly more confident about working on a problem with fellow students (t=2.15, df=34, p[less than]0.05) (Table 2). Feedback from the focus group revealed that students liked the game as a means of learning about the health system. Some issues were raised about working in groups, including the difficulty of contacting students with different timetables. There were also suggestions made regarding possible strategies for ensuring that all team members contributed equally. These comments, along with some suggested changes to specific design features, have now been incorporated into the latest version of the game.

A further method used to examine the validity of the Health Care Game as an effective teaching and learning tool was to map Koppi, Lublin, and Chaloupka's (1998) desirable outcomes of effective teaching and learning in the context of technology-enhanced learning against the characteristics of the game. These criteria represent the anticipated outcomes for learners who adopt a deep approach to learning. The outcome of this mapping is shown in Table 2 and illustrates that the game performs well against each of the criteria.

DISCUSSION

The evaluation results presented here represents the primary phase of the assessment, examining the first group of students who participated. The results are encouraging. Further formative evaluation over time to see whether the results hold and are sustained across time and over different settings and with different student groups is planned.

On the basis of these results it is suggested that the web-based Health Care Game encourages health-information seeking behaviour among student populations. It allows students to interact with the health system and to put knowledge gained in class into practice. The achievement of these learning outcomes was supported by the results from the evaluation, which showed that at the completion of the game students rated and demonstrated, that their knowledge had significantly increased both in terms of information and information sources.

Examination of students' responses from the focus group and their reported access and familiarity with computers and the Internet provided a clear indication that they understood the purpose of the Health Care Game and had the required computer skills to participate in the game. Students were inclined to believe that it was more difficult to find information about the Australian health care system after the game compared to before, though this was not a significant difference. We believe that this is an indication that the game is successful in providing students with a realistic view of the complexities of the health care system.

Students' reactions to working in groups appeared not to have been greatly influenced by their experience with the game, with the exception that they reported greater confidence in working on a problem with a group of fellow students. This result may have been due to the short duration of the trial and also to some of the issues raised by students during the focus group. Features now incorporated in the game, but not present during the trial, such as the ability of students to e-mail each other, may assist in creating a more effective team environment in the future. The impact of the Health Care Game on students' views of working in teams is an issue that we will continue to monitor.

The health care events in the game are designed to allow students to evaluate health issues from pluralist perspectives and to appreciate that people from different social, cultural, and economic backgrounds may have different needs and make different health choices. Determining the game's success at achieving these outcomes will be a focus of our further evaluation efforts.

Prior to undertaking the pilot of the game students had completed nine weeks of lectures which included topics covered by the game. Thus the improvement in students' factual knowledge, as evidenced by their answers to the scenarios, and their ability to identify sources of information following the completion of four weeks of the game, provides good evidence that the Health Care Game is an effective tool in improving learning outcomes. This is also supported by the students' perceived improvement in their knowledge following the pilot. Results also suggest the game encourages a deep approach to learning and is more enjoyable and interesting for both students and subject co-ordinators than it was in the past. At the completion of the trial, students reported significantly greater interest in the health care system than previously.

The game is suitable for a range of class sizes and comprises a broad range of events and policy issues faced by people in the community and which involve all health professional groups. The authors envisage the game to be a substantial component--at least half the coursework--in a subject covering the health system or health policy. This will be balanced by other modes of learning and instruction such as lectures, seminars, tutorials, and reflective activities. In addition, the game has flexibility with the facility to allow specialised events to be developed and added for particular groups of health professionals. The game also provides scope for an innovative approach to assessment for students. Students are awarded points on the basis of their success at correctly identifying options available to families faced with specific health events.

These early results suggest to us that attaining knowledge and skills through a range of different activities is considerably more effective and enjoyable than obtaining such information through more traditional methods. The Health Care Game will be used to complement lectures and other educational approaches to exploring the health care system, achieving a balance between teacher-directed learning and independent learning by students. Students have a variety of learning styles. The different activities required as part of the game provide the opportunity to cater for students with a range of different learning styles. A major aim of any educator is to provide students with skills which allow them to evaluate a situation for themselves and think independently. The Health Care Game aims to provide a supporting framework for students to gain these skills.

Acknowledgements

This study was funded by an Australian National Teaching Development Grant of the Committee of the Advancement of University Teaching on behalf of the Commonwealth Department of Employment, Education, Training and Youth Affairs. The expertise of M. Chaloupka, A. Lovell-Simons, S. Clark, G. Cheney, J. Bayly, B. Speirs, and T. Koppi from New Technologies in Teaching & Learning (NeTTL) was invaluable in the development and design of the Health Care Game. Our colleague Professor Arie Rotem provided insightful comments on an earlier draft of this article.

Note

(1.) Once the game is accessed, choose the option demo site.

References

Alexander, S. (1995). Teaching and learning on the world wide web. [Online}. Available: http://www.scu.edu.au/sponsored/ausweb/ausweb95/papers/education2/ale xander/

Barron, A. (1998). Designing web-based training. British Journal of Educational Technology, 29(4), 355-370.

Barrows, H.S. (1986). A taxonomy of problem-based learning methods. Medical Education, 20, 481-486.

Benson, K., Alison, S., & Arger, G. (1996). Interactive multimedia in introductory accounting: A preliminary analysis. Higher Education Research and Development, 15, 1-12.

Booth, S. (1997). On phenomenography, learning and teaching. Higher Education Research and Development, 16, 135-158.

Boud, D., & Feletti, G. (1991). The challenge of problem based learning. London: Kogan Page.

Boulding, K.E. (1956). General systems theory: The skeleton of science. Management Science, 2, 197-208.

Checkland, P. (1981). Systems thinking, systems practice. Chichester: John Wiley.

Clifford, M.M. (1990). Students need challenge, not easy success. Educational Leadership, September, 22-26.

Collis, B. (1998). WWW-based environments for collaborative group work. Education and Information Technologies, 3, 231-245.

Conlon, T. (1997). The Internet is not a panacea. Scottish Educational Review, 29(1), 30-38.

Coppard, L.C. (1976). Gaming simulation and the training process. In R.L. Craig (Ed.). Training and development handbook. New York: McGraw-Hill.

Cotter, R.V. (1995). The business policy game. New York: Appleton-Century-Crofts.

Ewing, J.M., Dowling, J.D., & Coutts, N. (1998). Learning using the world wide web: A collaborative learning event. Journal of Educational Multimedia and Hypermedia, 8(1), 3-22.

Fabros, B., & Young, M.D. (1999). Telecommunications in the classroom: rhetoric versus reality. Review of Educational Research, 69(3), 217-259.

Feletti, G. (1993). Inquiry based and problem based learning: How similar are these approaches to nursing and medical education? Higher Education Research and Development, 12, 143-156.

Greenlaw, P.S., & Wyman, F.P. (1973). The teaching effectiveness of games in collegiate business courses. Simulation and Games, 4, 259-294.

Gremmen, H., & Potters, J. (1997). Assessing the efficacy of gaming in economic education. Journal of Economic Education, 28, 291-303.

Hakkarainen, K., Lipponen, L., Jarvella, S., & Niemivirta, M. (1999). The interaction of motivational orientation and knowledge-seeking inquiry in computer-supported collaborative learning. Journal of Educational Computing Research, 21(3), 263-281.

Hughes, I. (1992). Learning to solve complex problems in simulation. Higher Education Research and Development, 11, 1-7.

Johnson, S.D., Aragon, S.R., Shaik, N., & Palma-Rivas, N. (2000). Comparative analysis of learner satisfaction and learner outcomes in online and face-to-face learning environments. Journal of Interactive Learning Research, 11(1), 29-49.

Jones, R.B., Latif, S., Murray, S., & Hedley, A. J. (1992). Simulation models of health service management: an example of continuing medical education using a package for daycase surgery. Simulation/Games for Learning, 22, 56-62.

Kolb, D. (1984). Experiential learning: Experience as the source of learning and development. NJ: Prentice Hall.

Koppi, A.J., Lublin, J.R., & Chaloupka, M.J. (1998). Effective teaching and learning in a high-tech environment. Innovations in Education and Training International, 34, 245-251.

Kryukov, M.M., & Kryukova, L.I. (1986). Toward a simulation games classification and game dialogue types. Simulation and Games, 17(3), 393-402.

Laveault, D., & Corbeil, P. (1990). Assessing the impact of simulation games on learning: a step-by-step approach. Simulation/Games for Learning, 20(1), 42-54.

Marton, F. (1988). Describing and improving learning. In R.R. Schmeck (Ed.). Learning strategies and learning styles. New York: Plenum.

Mason, C.H., & Perreault, W.D. (1995). The marketing game (2nd Ed). Homewood, Il: Irwin.

McConnell, D. (1994). Implementing computer supported cooperative learning. London: Kogan Page.

McKay, J., & Kember, D. (1997). Spoon feeding leads to regurgitation: a better diet can result in more digestible learning outcomes. Higher Education Research and Development, 16, 55-67.

Mooney, G.A., & Bligh, J.G. (1998). [CyberIST.sup.O]: A virtual game for medical education. Medical Teacher, 20, 212-216.

National Board of Employment, Education and Training. (1994) Developing lifelong learners through undergraduate education. Commission report No. 28. Canberra: Australian Government Publishing Service.

Perry, C. (1995). Exploring MBA students' preferences for experiential learning. Higher Education Research and Development, 14, 107-119.

Perry, C., & Delahaye, B. (1990). Team profiles and team performances in a business management simulation. Higher Education Research and Development, 9, 61-69.

Ramsden, P. (1992). Learning to teach in higher education. London: Routledge.

Rowlands, M. (1989). A question of complexity. In D, Miller, M. Rowlands, & C. Tilley (Eds.). Domination and resistance. London: Unwin Hyman.

Rolfe, J. (1991). SAGSET 1990-The proof of the pudding: The effectiveness of games and simulations. Simulation/Games for Learning, 21(2), 99-117.

Saunders, D., & Gunn, R. (1990). The assessment and evaluation of communication skills associated with simulation/gaming. Simulation! Games for Learning, 20(2), 215-234.

Schoenecker, T.S., Martell, K.D., & Michlitsch, J.F. (1997). Diversity, performance and satisfaction in student group projects: an empirical study. Research in Higher Education, 38, 479-495.

Senge, P.M., Roberts, C., Ross, R.B., Smith, B.J., & Kleiner, A. (1994). The fifih discipline fieldbook. London: Nicholas Brealey.

Sherry, L., & Billig, S.H. (2000). Good online conversation: Building on research to inform practice. Journal of Interactive Learning Research, 11(1), 85-127.

Taylor, P.G. (1996). Reflections on students' conceptions of learning and perceptions of learning environments. Higher Education Research and Development, 15, 223-237.

Thatcher, D. (1986). Promoting learning through games and simulations. Simulation/Games for Learning, 16(4), 144-154.

Thatcher, D., & Robinson, J. (1994). A game workshop for the design of games and simulations for teachers. Simulation/Games for Learning, 20(3) 250-263.

Thompson, D.G., Jolly, B., Macdonald, M.M., Cookson, J., Holman, J., & Keech, T. (1987). Students' approaches and attitudes to solving computer problems. Journal of Computer Assisted Learning, 3, 240-250.

Tobin, K. (1990). Social constructivist perspectives on the reform of science education. Australian Science Teachers Journal, 36(4), 29-35.

Webster, J., & Hackley, P. (1997). Teaching effectiveness in technology-mediated distance learning. The Academy of Management Journal, 40(6), 1282-1309.

Williams, T. (1992). Debriefing learning activities. Simulation/Games for Learning, 22(2), 91-94.

Wolfe, J. (1985). The teaching effectiveness of games in collegiate business courses. Simulation and Games, 16(3), 251-288.

Zhao, Y. (1998). Design for adoption: The development of an integrated web-based education environment. Journal of Research on Computing in Education, 30(3), 307-328.

               Means and Standard Deviations of Responses
                 to the Pre and Postgame Questionnaires
Question                                    Mean
                                            Pre Game Survey
                                            (Standard
                                            Deviation)
How would you rate you current              3.1              (.54)
knowledge of the Australian
health care system?
How would you rate your current             3.1              (.80)
knowledge of the way in which
Medicare works?
How would you rate your current             2.9              (.70)
knowledge of the way in which private
health Insurance works in Australia?
Scale used 1=very poor, 2=poor,
3=average, 4=good, 5=very good
In general I am interested in the           3.7              (.62)
Australian health cares stem.
Obtaining up-to-date information about the  3.0              (.10)
Australian health care system is easy.
I enjoy working in groups                   3.4              (.93)
I do not feel confident about working on a  2.6              (.87)
problem with a group of fellow students
Working in groups always leads to conflict  2.5              (.10)
I learn more when I work in a group         3.3              (.12)
I solve problems best when I work by        3.1              (.90)
myself
Question                                    Mean Post Game
                                            Survey
                                            (Standard
                                            Deviation)
How would you rate you current              3.4 [**]        (.64)
knowledge of the Australian
health care system?
How would you rate your current             3.5 [**]        (.69)
knowledge of the way in which
Medicare works?
How would you rate your current             3.4 [**]        (.72)
knowledge of the way in which private
health Insurance works in Australia?
Scale used 1= very poor, 2=poor,
3=average, 4=good, 5=very good
In general I am interested in the           3.8 [*]         (.46)
Australian health cares stem.
Obtaining up-to-date information about the  2.8             (.76)
Australian health care system is easy.
I enjoy working in groups                   3.2             (.96)
I do not feel confident about working on a  2.4 [*]         (.73)
problem with a group of fellow students
Working in groups always leads to conflict  2.6             (.87)
I learn more when I work in a group         3.1             (.89)
I solve problems best when I work by        3.3             (.84)
myself
Scale used 1=strongly disagree, 2=disagree,
3=undecided, 4=agree, 5=strongly agree
Items where there was a significant difference
between pre and post game questionnaires
(*.)p[less than]0.05,
(**.)p[less than]0.01
       Five Desirable Outcomes of Effective Teaching and Learning
Koppi et al.'s Criteria              Health Care Game
Holistic appreciation of the         Facilitates an appreciation of the
subject                              wider
in a broader context                 Australian health care system and
                                     interrelationships between sub-
                                     systems
Active knowledge construction        Heuristically-oriented, centred on
                                     health information-seeking
                                     behaviour
Communication, collaboration         Structures communication between
and teamwork                         team members, teams, and subject
                                     coordinator and individuals and
                                     subject
                                     coordinator. Emphasises group
                                     problem solving skills via a
                                     collaborative
                                     team approach
Critical thinking and problem        Based on a schema which requires
solving
                                     problem identification, gathering
                                     data to
                                     address problems, summarising and
                                     reporting information to subject
                                     coordinator, receiving feedback on
                                     the
                                     solutions tendered
Lifelong learning                    Encourages students to be self-
                                     directed and in control of their
                                     own
                                     information-seeking behaviour as a
                                     foundation for long term learning