Adventures in evaluation: reviewing a CD-ROM based adventure game designed for young people recovering from psychosis.

            Recently the Centre for Program Evaluation (CPE) at the
            University of Melbourne was approached by a mental health
            agency to undertake the unique and challenging task of
            evaluating a prototype CD-ROM based adventure game designed
            for young people recovering from psychosis. This unsual and
            inventive game, titled Pogo's Pledge, used "edutainment" in
            the form a medieval fantasy quest to provide educational
            messages about psychosis for young people who are
            experiencing psychosis for the first time. To explore the
            potential educational effectiveness of the product, an
            innovative expert review was conducted using a range of
            multimedia professionals including gaming designers,
            multimedia, graphic/interface designers and multimedia
            educational designers together with first-episode psychosis.
            For the client who commissioned this evaluation interviews
            focusing on areas such as game induction, interface design,
            navigation, game play and quality of psycho-educational
            material revealed that a substantial redevelopment of the
            game would be needed for the CD-ROM to be an effective tool
            in helping young people recover from psychosis. For a
            broader audience the evaluation provides a fascinating case
            example of the utility of expert reviews when evaluating
            complex multimedia products, and offers a cautionary tale to
            designers of the hazards and pitfalls that can await well
            intended products than combine gaming and educational
            information for young people.

The Centre for Program Evaluation at the University of Melbourne is an Australian evaluation and research centre that acts as a focus for the practice and theory of evaluation. In addition to its postgraduate teaching activities, the Centre for Program Evaluation (CPE) undertakes a substantial number of commissioned program evaluations on a consultancy basis across a wide range of policy and program areas. It is not unusual for evaluations as diverse as the use of information kiosks at postal services (Hurworth, 2002) and a review of emergency service training (Hurworth & Rutter, 2002) to be conducted concurrently by the centre at any given time. However, the CPE was recently presented with a very different and exciting evaluation challenge when it was approached by the Victorian based youth specific mental health agency, Orygen Youth Health, to conduct a review of the potential educational effectiveness of an advanced prototype computer game created for young people recovering from psychosis. The CD-ROM game, titled Pogo's Pledge, presented an inventive approach to psycho-education by integrating elements of gaming, contemporary animation and graphics, educational design, and topic-specific content to convey information about psychosis to players. The evaluation of Pogo's Pledge highlights for multimedia designers the utility of expert and "expert user" feedback when reviewing interactive educational software intended for special needs audiences, and provides an outline of the problems and issues that can emerge in products that attempt to integrate information and game play.

More about Pogo's Pledge

A single-player computer game, Pogo's Pledge uses "edutainment," in the genre style of a fantasy-based adventure game, to provide educational messages about psychosis for young people who are experiencing psychosis for the first time. When using the game, a player assumes the role of a Hero who sets out across a medieval land (Figure 1) to free a once happy and prosperous kingdom from the tyranny of the evil Lord Gruesome. The Hero is accompanied and assisted in this adventure by a young character called Pogo. The character "Pogo" plays a significant role in events by helping the "Hero" (the game user) in his or her quest through the game, but who also occasionally experiences psychotic outbursts that players must learn about and "treat" if the game is to continue.

Players are aided throughout this adventure by tools such as a Survival Kit, a First-Aid Kit, and a Book of Knowledge that can be used to help Pogo and resolve scenarios involving other characters who may attempt to either help or hinder the player on their journey (Figure 2).

These items, the scenarios and the game play are designed to provide users with encounters and embedded psycho-educational material that reflect the experiences and recovery needs of young people during the Prodrome (early stages), Acute and Recovery periods of psychosis. Crudely truncated, these needs can include medication and counseling, developing an awareness of the nature of psychosis, and learning to identify and respond to the challenges of psychosis recovery.

[FIGURE 1 OMITTED]

YOUNG PEOPLE AND PSYCHOSIS

Psychosis is a potentially devastating condition that affects a person's mind and causes a loss of contact with reality. This often-misunderstood condition is not bound by age, race, or gender, but it is more likely to be experienced by adolescents and young adults between the ages of 16 and 30 years (these account for some 85% of cases). The symptoms of psychosis are diverse, but generally affect the way a person thinks, feels, and perceives the world around them.

The most common symptoms are hallucinations (hearing voices), delusions (believing in things that are not real), and thought disorder (confused thinking or speech) (Orygen Youth Health, 2003). While a variety of factors may contribute to the experience of psychosis, the condition is often associated with schizophrenia and bipolar disorder (extreme moods), although it can also be triggered by the use of some drugs, such as amphetamines, and by intensely stressful circumstances. With treatment, recovery from first-episode psychosis can take only a few months for some young people, while for others a longer period of recovery is experienced. However, to maintain recovery it is significant that young people should learn about their psychosis and the strategies they can use, such as recognition of relapse symptoms, tactics for dealing with the world, and the use of medication and counseling, to maximise their mental health and prevent further psychotic episodes (EPPIC, 2000).

[FIGURE 2 OMITTED]

Gaming and Psychosis Recovery

The inspiration for developing the fantasy adventure Pogo's Pledge was to assist young people to engage with essential psycho-educational materials and simulated experiences that could aid their recovery. Based on both real world experience and its own and international research, Orygen Youth Health had long been an advocate of using educational materials in psychosis recovery, and to this end made available a range of psycho-educational products, such as fact sheets and videos, to both clients and various care providers. However, while these products were seen as being useful and of good quality, there was a recognition that some young people might not fully engage with these more traditional media forms.

In commissioning the development of Pogo's Pledge, Orygen Youth Health sought to use the medium of gaming, a medium deeply embedded in the culture of young people (Herz, 1997), to provide broad information in an accessible and engaging form that would facilitate a better knowledge of the experience of psychosis. Accordingly, when playing the CD-ROM users could enter a world patrolled by trolls whose riddles must be solved, meet warriors who believe they are hearing voices, and read magical books of knowledge. Although entertaining and featuring contemporary animated characters interacting in a fantasy-adventure setting, events that occurred in the game were also designed to reflect the real-life experiences and recovery needs of young people with psychosis.

It was anticipated that in a typical usage scenario, interaction with the CD-ROM would occur with a clinician (therapist) present in the context of a guided session where gaming experiences could be discussed to enhance the client's understanding of their psychosis. Through playing the game Orygen Youth Health hoped that young people would safely explore problem-based scenarios that might then be used by clinicians to address issues of psychosis and to help young people gain mastery over their condition. By deconstructing and reflecting on character actions and difficult events during the game, Pogo's Pledge was to offer young people an opportunity to enhance their understanding of the consequences of risk taking and its impact on life, relationships and psychosis recovery.

These gaming sessions were to be approximately 30-40 minutes in duration and take place several times over the course of a number of weeks. The possibility also existed for the game to be used at home, with the results of the game-play being discussed during a subsequent session with a clinician.

When the CPE were commissioned to evaluate Pogo's Pledge the game was at an advanced prototype stage of development but had not yet been used by young people. Due to various circumstances, including staffing changes at Orygen Youth Health, the CPE evaluators were unable to clearly document previous evaluative activities that may have been undertaken in the design and production of the Pogo's Pledge prototype. Nevertheless, it is prudent to acknowledge that the evaluation of any form of social intervention, whether educational multimedia or an antismoking campaign, should ideally be an iterative process that takes place at all stages of planning, development, and implementation (Owen, 1999).

METHODOLOGY

In negotiating an evaluation plan for Pogo's Pledge, the central issue Orygen Youth Health proposed that the CPE examine was summed up with the simple key question, "is Pogo's Pledge a viable product?" Taken on a broader level, and using the terminology of program evaluation, this question asked the evaluators to establish whether the "Product" could support the objectives of the "Program" (Owen, 1999). The Program in this case being Orygen Youth Health, an agency whose objectives were concerned with improvements in mental health for young people at risk of or experiencing first episode psychosis. Determining the efficacy of the product (Pogo's Pledge) in meeting these program-level objectives proved indeed to be a uniquely challenging task. A task that required CPE staff to set out on an exploration of the complex nexus of gaming, educational design, youth psychosis, contemporary animation, and topic-specific content. Among the many issues and questions CPE evaluators found they would need to explore on this journey were such matters as:

* Can the intended audience use the game?

* Are the graphics and animations appropriate for the intended audience?

* Does the game have a sound instructional design?

* Is the psycho-educational context appropriate?

* How does the CD perform as a game?

* Is the graphic design, animation and characterization engaging?

* Will the game be used by young people?

* Will the game be used by clinicians?

The next section of this article describes the evaluation design that was created to facilitate this adventure. While the evaluation design was established specifically for the review of Pogo's Pledge, the authors offer the study design as a utilitarian case example of a successful approach to reviewing complex multimedia products that integrate information and gaming. The methodology section also provides a practical outline of the types of questions that can be generated for such investigations.

Evaluation Design

The methodology for evaluating the various interconnected elements of Pogo's Pledge was found in pursuing an approach to multimedia evaluation espoused by Kennedy, Keppell, and Liaw (2000) and Kennedy, Petrovic, and Keppell (1998) who proposed that the evaluation of educational multimedia products take account of three key domains: instructional and conceptual design; interface and graphic design; and user attitudes and effect. To explore each of these domains fully, an expert review was undertaken that involved face-to-face interviews with 14 professionals working in commercial game design, multimedia graphic/interface design, multimedia educational design, and first-episode psychosis. These experts were targeted using reputational and snowball techniques, with each contributing specialist knowledge on different aspects of the game (Moyser & Wagstaff, 1987). "Expert" user perceptions of Pogo's Pledge were then provided by three focus groups: two with clinicians working with young people with psychosis and one with young people who had experienced psychosis.

More about the Face-to-Face Interview Participants

The CPE was immensely fortunate that the expert participants considered the evaluation of Pogo's Pledge to be both valuable and interesting, with all eager to take part and generous with their time. Eight of the interviewees worked in teaching, research, and production positions at four Australian universities recognised as being leaders in the development of educational multimedia. Many also had prior industry-based experience in multimedia development. One participant was the owner of a well-regarded educational multimedia business, while another was the president of an association of instructional designers. The two game designers taking part in the review were employed by two of Australia's leading computer and console game producers. The first-episode psychosis experts were chosen by Orygen Youth Health for their extensive professional experience with young people with psychosis.

The majority of face-to-face interviews were held at people's workplaces, although four were conducted in the Centre for Program Evaluation at the University of Melbourne. Participants were sent a preview copy of the Pogo's Pledge prototype in advance of interviews; in addition to this, 20 minutes was allocated at the beginning of each interview for exploring the game to ensure that participants had identified key features of the CD-ROM. Interviews averaged 1.5 hours in duration and were conducted with the game running on a computer so that interviewees could actively point out perceived issues and illustrate responses to questions.

More about the Focus Group Participants

The two clinician focus groups were organised by Orygen Youth Health and included a variety of care professionals working with young people with mental health problems (one group of eight and one group of seven). Participants included senior psychologists, case managers, crisis-assessment teams, and early-intervention workers operating in school settings. Members of the young-people focus group, again organised by Orygen Youth Health, were in their late teens and early twenties and had all experienced psychosis (one group of four). It was originally intended that there should be two focus groups with young people, but unfortunately it proved difficult to attract sufficient numbers for a second group.

As with the face-to-face interviews, in the first 15-20 minutes of each session participants were given a brief overview of the CD-ROM to ensure they were aware of key features of Pogo's Pledge. This was conducted by CPE staff in a computer room, using a data projector. For the next 30-40 minutes participants were invited to play Pogo's Pledge on a computer, either on their own or with a partner. This was immediately followed by the focus group session (50-60 minutes), which was conducted in a room at the Centre for Program Evaluation.

Interview and Focus Group Questions

Drawing on the work of Kennedy, Keppell, et al. (2000) and Kennedy, Petrovic, et al. (1998), and a wide variety of other authors including Cadenas (2001), Herrington and Oliver (1996), Neilsen (1994), Reeves (1997), and Sambrook (2001), it was decided that the major areas to be covered in interviews on the utility of Pogo's Pledge would be in regard to such matters as:

* information provided to players about how to use the game;

* the game's interface design;

* the product's navigation system;

* the setting of the game;

* the characters and animations;

* the quality of sound and voices;

* the game's interactivity;

* the game's sequencing and structure;

* the quality of the psycho-educational content; and

* the general usability and user-friendliness of the game.

Participants were also asked a number of questions relevant to their specific areas of professional expertise, as the following examples show:

Educational designers:

* Do you feel the objectives and desired outcomes of the game are made clear to users?

* Does the interactivity provide meaningful feedback?

* Does the interactivity encourage reflection on key issues?

Graphic/interface designers:

* What comments do you have to make on the product's interface design?

* Are the graphics and animations of high or low quality?

* Do you feel there is an appropriate and consistent use of illustrations?

Game designers:

* From your experience, is the game narrative engaging and interesting for users?

* What is your reaction to the animations used in the game?

* Do you have any comments about the product's navigation?

First-episode psychosis experts:

* Will young people recovering psychosis be able to easily learn and use the game?

* Are the characters, events and scenarios suitable for young people recovering from psychosis?

* Is the educational content appropriate for those recovering from psychosis?

As with the interview participants, focus group members were asked to offer their thoughts and feelings about a range of questions regarding: the information provided to players on how to use Pogo's Pledge; reactions to the setting of the game; reactions to the characters and animations; the usefulness of the psycho-educational content; and the general usability and user-friendliness of the game. Focus groups participants were also asked questions that enabled each group to contribute unique perspectives as potential future users of the product. More specifically, participants were asked questions such as the following:

Young People:

* How do you think young people will respond to using the game?

* Will young people find the information presented about psychosis in the game useful?

* Do you think young people will feel comfortable about using and discussing the game with a clinician?

Clinicians:

* Do you think clinicians would find the game a useful psycho-educational tool?

* Is it likely that young people would use the game at home?

* Is there an appropriate computer available in the area where you conduct most of your work with young people?

Analysis of the Data

After the 14 interviews and 3 focus groups had been completed, all tapes were transcribed in full resulting in approximately 244 pages of text. Transcripts were then coded, beginning with a basic set of codes derived from the original interview questions, the literature review, and emergent themes drawn from a standard process of what Dey (1993) referred to as "grouping like with like." Following further close readings of interview texts, codes were maintained, adapted, added to, or collapsed. Next, all major ideas for each participant group were displayed under thematic headings and subthemes on matrices of the type proposed by Miles and Huberman (1994). These displays, in combination with verbatim quotes from the transcripts, are a particularly rigorous way of dealing with such qualitative data. The 244 pages of data were thus reduced to a 50-page report presenting participant feedback according to categories and subtopics that made explicit the key issues identified by the multimedia professionals, psychosis experts, and the focus groups representing potential users of the product.

DISCUSSION OF THE FINDINGS

The results of the CD-ROM review were extensive, and closely relate to specific elements of the game. As a commissioned evaluation piece for a public mental health facility the findings remain the property of Orygen Youth Health and for reasons of anonymity and other conditions some findings cannot be reported in full within this article. However, the evaluation naturally produced a number of findings that can be taken at a generic level, providing a fascinating insight into the benefits and difficulties that can emerge when producing a product that combines gaming and educational information for an audience of young people.

Positive Responses to the Game

All participants involved in the evaluation expressed their strong support for the innovative conceptual approach to psychosis education demonstrated by Pogo's Pledge. Indeed, the concept of a CD-ROM designed to assist clinicians and their clients to deal with psychosis was greeted with enthusiasm and interest. Participants felt that the game was "groovy and different," and therefore likely to capture the attention of young people. Respondents also indicated that using a computer in an interactive manner reflected the recreational activities and culture of many young people and would appeal to this group "on their level." Furthermore, the safe way in which players could use characters to explore symptoms of psychosis and the potential consequences of particular behaviors was a feature of the game seen by many as providing a useful way of allowing young people to step back from their illness and examine recovery strategies in an objective and non-confrontational manner. However, while the game could be viewed as a successful "proof of concept," the general consensus of participant opinion suggested that a wide range of changes and issues would need to be considered in order to make the CD-ROM an effective educational tool. The issues identified by participants that will be described here relate to the game induction; interface design; use of animations and graphics; characterisation and character voices; the convergence of gaming and educational information; and infrastructure deficiencies.

Game Induction

A large number of participants including members from all expert and expert user groups warned that the design of the game induction did not sufficiently define the desired objectives and outcomes associated with playing Pogo's Pledge, and further, risked frustrating users by failing to explain how they should use the game's tools and navigation items. Educational designers, graphic/interface designers, and game designers commented that players were placed in to the game narrative without appropriate information about why young people should use Pogo's Pledge; what the aims of the game were; an outline of the history and environment in which the game quest was set; or an explanation of the psycho-educational intentions of Pogo's Pledge. A number of respondents also specifically noted the lack of support available within the game for players, or indeed clinicians, unfamiliar with gaming or computers. A game designer described how considerable effort is normally spent on creating tutorials embedded within commercial games. The designer explained that while the motive for this, in commercial production, is to reduce product return, in the context of Pogo's Pledge an engaging tutorial could be used to maximize user comprehension of the game.

Interface Design

Feedback was largely positive in regards to the navigation items located in the main interface of the game where users were either prompted for responses that drive the game narrative forward or were able to click icons to branch to sub-nodes of the game. Participants stated the navigation layout at this level was straightforward, remained consistent, and was especially effective when flashing prompts were used to attract a player's attention to available options. However, strong doubts were expressed about a key sub node of the game, referred to as a Survival Kit, which players enter to access tools and information designed to resolve game scenarios and engage young people with psycho-educational content (Figure 3). While many participants indicated that providing sub nodes containing tools and interactive tasks could be useful for communicating psycho-educational material, there were many reservations about the navigability of this component of the game. Participants felt movement into and interactions within the Survival Kit were ultimately ill defined and did not effectively alert players to the tools that were at their disposal or why these tools had become available. Meanwhile, other respondents suggested that once players had entered the Survival Kit navigation ambiguities meant that novice users might click aimlessly when attempting to locate items or selecting between buttons and interactive text. Indeed, a CPE moderator present at the session held with young people directly observed participants encountering difficulties when moving through the various components of the Survival Kit.

[FIGURE 3 OMITTED]

Animation and Graphics

Although the game and multimedia graphic designers judged the animation and images used in Pogo's Pledge to be of high quality, both these groups of professionals and members of the young people focus group pointed out a number of important issues. The expert participants were clearly impressed by the 2D (two-dimensional) animations used in Pogo's Pledge but warned that modern game audiences would consider 2D games passe. The game designers explained that most commercial companies had ceased producing 2D games in response to market demand for 3D products. They therefore expressed concern that some players would bring to the game expectations that Pogo's Pledge would not meet. Members of the young people focus group similarly provided a number of positive responses in regard to the animations and general look and feel of the game. However, a significant issue emerged in their feedback. All members of this group thought the game had been specifically designed for an audience of early adolescents rather than for those in their mid-to-late teens and young adults. They suggested that while young people might find the game fun, they would not perceive themselves as its target audience.

Characterisation and Character Voices

As users play Pogo's Pledge they encounter troll like figures, interact with warriors who suffer delusions, encounter villains, and are guided at various times by a court jester. When participants were asked for their thoughts on the appropriateness of these narrative figures diverse responses were given from positive reactions about individual characters to warnings that some characters could potentially perpetuate psychosis stereotypes. Favourable reactions were offered in regards to several characters being well designed, cute or possessing appropriate emotional content. However, there were concerns that the design of some characters appeared simply out of place in the game; the sole female character in the game was portrayed as helpless; most game characters were not given enough background information (i.e., contextualising their appearance in the game); and players were not provided opportunities to customise characters to enhance emotional investment. Significant levels of negative feedback focussed on the types of voices used for characters and for providing instructional/educational information. The voices and voiceovers were judged by participants to be of poor quality, incompatible with the physical depiction of some characters, occasionally grating, and used archaic language and accents that might not be understood by young people from non-English speaking backgrounds (Figure 4). Expert participants such as the game designers and educational designers stressed the importance of using appropriate voices for game characters and of testing player acceptance of characters with potential game users.

[FIGURE 4 OMITTED]

Learning versus Gaming

Participants were highly appreciative of the interactive approach to providing psycho-educational material presented by Pogo's Pledge, offering statements indicating that the game was frequently engaging and provided useful methods for exploring problematic situations. However, for the majority of respondents there were also troubling issues with the convergence of the educational material and the game play. At the core of these concerns was the belief that the educative material needed to be integrated better into the game play. A common response was that the educational information disrupted the game-play thereby destroying the illusion of a game environment. This impression led participants to report that:

* the game appeared to be in two separate parts;

* there seemed a noticeable switch from fun fantasy to serious messages; and

* the experience of moving between the game and the information was frustrating.

There were also some concerns that players might lose the thread of the storyline as a result of this awkward movement between information and gaming. Participants concluded that if the CD-ROM was to be presented to users as a game, it needed to operate like one. Ideas about how educational material might better be integrated into the game play were to present the psycho-education with more fun and fantasy, to offer information in more visual and interactive forms, and to make the information part of the storyline.

Taking Account of Infrastructure Requirements

A final general issue that will be reported in this article relates not to the game itself but to the infrastructure required to run it. An important but easily overlooked consideration when developing a product such as Pogo's Pledge is the hardware that will drive the game. In the course of conducting interviews at the workplace of psychosis specialists, it was noted by the evaluators that on a number of occasions it was difficult to locate a suitable computer that had both a CD-ROM drive and the speakers necessary to operate the game. Furthermore, participants from the focus groups conducted with youth mental health clinicians reported that much of their work with young people was conducted in homes, schools, the workplace, and other locations where the availability of computers varied and could not be guaranteed. Additional comments by clinicians also revealed that the quality of computers in their workplace was quite mixed, so that while most felt access to new computers was increasing, a significant proportion said they had very old and even archaic equipment. This feedback suggested that an infrastructure audit would be required to assess the ability of mental health clinicians to access appropriate hardware to use the game with young people.

CONCLUSION

The review of Pogo's Pledge proved an exciting challenge for CPE evaluators. The complex nature of this advanced prototype demanded that a range of experts and potential future users of the game be brought together to determine how well the CD-ROM's animation, game play, and psycho-educational material could assist young people in their recovery from first-episode psychosis. This evaluation design illustrated how expert reviews can assist designers to uncover deficiencies and oversights in complex products that attempt to combine gaming and educational information for young people. The expert review of Pogo's Pledge revealed that major flaws in the game's design demanded a complete reworking of the game. Yet it may be said that, taken as a whole, the feedback on this intriguing game nevertheless strongly suggests there is a role for such products in the educational activities of youth-specific agencies. However, the review has also shown that efforts aimed at mimicking computer and console games played by young people are unlikely to succeed unless the same levels of design sophistication, iterative testing, and acute audience awareness demonstrated by these products are also replicated. Given the funds available to commercial production houses such as Sony and Nintendo, who manufacture these products, this may be beyond the scope of most state-funded institutions and agencies.

References

Cadenas, L. (2001). Evaluation of multimedia: Why and how? An expedition in three voyages. Proceedings of 67th IFLA Council and General Conference, Boston, MA, (pp. 1-7).

Dey, I. (1993). Qualitative data analysis. London: Routledge.

EPPIC (2000). Recovering from psychosis. EPPIC Statewide Services, Melbourne, Victoria.

Herrington, J., & Oliver, R. (1996). The effective use of interactive multimedia in education: Design and implementation issues. Proceedings of the 3rd Interactive Multimedia Symposium, Perth, Western Australia, (pp. 169-174).

Herz, J. (1997). Joystick nation: How computer games ate our quarters, won our hearts and rewired our minds. New York: Little, Brown.

Hurworth, R. (2002). An evaluation of the 'information kiosk' trial at Australia post (Focus Group Component). Center for Program Evaluation, University of Melbourne.

Hurworth, R., & Rutter, A. (2002). EMAI training needs assessment project. Report for the Emergency Management Australia Institute. Centre for Program Evaluation, University of Melbourne.

Kennedy, G., Petrovic, T., & Keppell, M. (1998). The development of multimedia evaluation criteria and a program of evaluation for computer aided learning. ASILITE Conference Proceedings, Wollongong, N.S.W: University of Wollongong, (pp. 407-415).

Kennedy, G., Keppell, M., & Liaw, T. (2000). Using multimedia to assist students with communication skills and biopsychosocial integration: An evaluation, Australian Journal of Educational Technology, 16(2), 104-125.

Miles, M., & Huberman, A.M. (1994). Qualitative data analysis (2nd ed.). Thousand Oaks, CA: Sage.

Moyser, G., & Wagstaff, M. (1987). Research methods for elite studies. London: Allen and Unwin.

Nielsen, J. (1994). Usability engineering. San Francisco: Morgan Kaufmann.

Orygen Youth Health, (2003). Mental Health Fact Sheets. Retrieved May 17, 2005, from Orygen Youth Health Web site: http://www.orygen.org/contentPage.asp?pageCode=MEDIA

Owen, J. (1999). Program evaluation: Forms and approaches. St Leonards, N.S.W: Allen and Unwin.

Reeves, T. (1997). Evaluating what really matters in computer-based education. Learning with Software: Pedagogies and practices report. Canberra, Australia: Open Learning Technology Corporation.

Sambrook, S. (2001). Factors influencing learners' perceptions of the quality of computer based learning materials. Journal of European Industrial Training, 25(2), 157-167.

BRADLEY SHRIMPTON AND ROSALIND HURWORTH

University of Melbourne

Victoria

[e-mail]

[e-mail]