Immersive learning and assessment with quizHUD.

I. LEARNING ONLINE

The ubiquity of the Internet has clearly had a substantial impact on learning and teaching in the modern context, with many educators supporting it strongly. One benefit to students has been the opportunity to work through material independently, at their own pace [1]; additionally, it has allowed educators to distribute resources easily, such as lecture recordings and notes, which may be used by students to supplement face-to-face teaching, or to replace it in cases of absence or distance learning [2,3]. To facilitate such use of web-based technology, a variety of dedicated software systems have been developed for managing and using online learning materials and activities. More general learning management systems--such as Blackboard and Moodle--have seen widespread adoption, particularly among higher education institutions. The present climate of web-based software (so-called "Web 2.0") has a strong focus on interactivity, with users not simply viewing material, but rather being actively involved with it. This has been reflected in learning management systems in a variety of ways, including interactive assessment [4].

Creating an entirely automated assessment system raises certain practical issues, as it relies on a definite result or score being calculated by a computer algorithm. While there is some domain-specific research into this area--such as grading mathematical working [5]--the bulk of online assessment is very limited, with multiple-choice quizzes being widely used [6]. Despite the limitations, various advantages of such assessment are given in the literature, including convenience for the student, who is able to take the test or exam from any suitably Internet-connected location, at any time. Additionally, with grading being carried out by a computer, it is entirely objective, and reduces the workload on members of staff [7]. Feedback to the student is also immediate, which can form an integral part of the formative learning process [8].

As further suggested in [8], using simple, multiple-choice assessments is not necessarily enough alone, but should form part of a larger set of learning activities.

II. LEARNING IN VIRTUAL WORLDS

Education has also seen an increase in the use of technology which is wholly or partly game-based, with virtual worlds in particular gaining more popularity [9,10]. Research suggests that the learning environment (virtual or otherwise) can have a direct impact on the student's willingness and ability to learn [11]. Similarly, the nature of a virtual environment, given the relative anonymity it provides, can have a direct bearing on the behaviour and attitude of its participants [12].

However, the volume of information presented to a user of a virtual world can become a barrier. When using an Internet website, one can effectively maintain a distance from the content, browsing at a high level. In contrast, the enriching, immersive nature of the experience in virtual worlds can hinder the learning process if there is not appropriate guidance [13]. Potentially though, this immersion can be of benefit, if used correctly, and therefore it is important that new ideas and concepts are developed to support it [14,15].

A. Second Life

At present, a highly successful general purpose virtual world is "Second Life" (http://www.secondlife.com), which has been written about extensively elsewhere. There are several reasons why educators and institutions begin using Second Life over any other virtual world, but one of the most common reasons is the ease with which custom content can be created, whether it is created by the educator or educational technologist to support learning, or created by the students as part of an exercise. Furthermore, it allows for external communications with other websites, enabling content sharing and integration [16].

III. QUIZHUD

The quizHUD tool has been developed for use with Second Life, and is a side-project of SLOODLE [17] (Simulation Linked Object Oriented Dynamic Learning Environment, http://www.sloodle.org). Its aim is to harness the benefits of web-based learning and assessment, such as convenience for the student, and objectivity of grading, while incorporating the benefits of virtual worlds, such as immersive navigation. A student using the device will see it as a graphical user interface extension, which is known as a HUD (heads-up display) in Second Life. It is intended to become a natural part of the student's experiences in the virtual environment, showing relevant information about specific places and items, and conducting a variety of assessed and non-assessed questions.

Students will each see a quizHUD on their own screen, displaying contextually relevant information and questions in the form of specially created web-content. This content can therefore be authored by the educator or educational technologist via his or her web-browser, instead of being required to work within the usual confines of data presentation in Second Life. It is important to note that the quizHUD employs a learning process that is individual in nature, as each student will see a personal view of the content. That is, an entire class can be navigating in the same virtual learning space simultaneously, even seeing and interacting with each other in the immersive environment, but each one determines his or her own pace and route through the learning material.

One of the main reasons for implementing quizHUD in Second Life is the scope for content creation provided by the platform. Instead of relying on a pre-defined world to which assessment can be added (as in the case of TiE [13], for example), a custom environment can be created relatively quickly, even based on content purchased from other users (although in order to be used with quizHUD, a minimal amount of modification is necessary).

As schematic of the quizHUD is shown in Fig. 1, and Figs. 2-5 show the quizHUD displaying core information pages.

A. Basic Operation

In order to keep the interface metaphor as intuitive as possible, based on users' real-world and existing computer interface experience, basic operation of the quizHUD occurs by clicking tabs along the top edge [18]. An initial 'homepage' is displayed first, which gives the essential usage information, with other tabs providing information about other modes of operation, and more general documentation.

The 'Quiz' page content is determined by the current quiz or survey settings, although the content of all other pages can be customized by the educator using the web-based interface.

[FIGURE 1 OMITTED]

B. Explore Mode

While navigating around the virtual environment, the student will see various features and objects. When s/he uses the mouse to click on one of these, an informative 'page' of text or images about it will be displayed on the quizHUD. Each feature and object in the environment can have one such page of information associated with it. The layout of these features and objects in the virtual environment is scalable as required, to effectively any level of detail, allowing the experience to consist of large, separate objects, or of many smaller parts of a detailed model. Fig. 6 shows an example of a student's view in a simple music classroom.

Part of the purpose of this mode is to engage students by enabling them to take on a more participatory role in their learning process than they might if simply presented with static information [19]. The environment can be setup to provide a certain structure and guideline route through the space, but each student is ultimately in control of his or her own progress through it, with the ability to go back to individual parts as necessary. Providing such autonomy to students has been found to positively impact the learning experience in a variety of other educational settings [20]. Presenting the material in small segments also helps to reduce the likelihood of being overwhelmed [21].

A further intent of this exploration-based system is to introduce an experiential, embodied approach to moving around the learning environment. While research has not shown that this would necessarily improve navigability, it has been seen to help increase a sense of position, thus helping the student's own spatial awareness to contribute to their grasp of the material [22].

C. Quiz Mode

The Quiz mode exists in order to assess students' progress, whether through the material in Explore mode or otherwise. This functionality is designed to be built-in to the same environment as that in which the learning took place, thus maintaining a degree of consistency and familiarity. Such considerations have been similarly noted regarding web-based assessment, where added anxiety or frustration can be caused by students facing assessment on a previously unseen platform [7]. Similarly, interaction with quizzes is entirely consistent with other parts of the quizHUD, relying solely on mouse clicks on the HUD itself, or in the environment, depending on the question being answered.

Another significant issue raised in web-based assessment is reliability of the assessment mechanism in the event of platform instability [7]; this becomes particularly important in Second Life, which suffers instability perhaps more than other Internet-based platforms (for anecdotal evidence, see the archives of the 'SL Educators Mailing List' [23]). To address this issue, the quizHUD stores the students' answers one at a time in the external database. Students can be confident that their answer has been stored, because the system cannot indicate the result of their answer (correct/incorrect etc.) until this has happened. As such, even if a technical issue causes an interruption during the assessment, the progress up to that point will not be lost.

[FIGURE 2 OMITTED]

[FIGURE 3 OMITTED]

[FIGURE 4 OMITTED]

[FIGURE 5 OMITTED]

[FIGURE 6 OMITTED]

As with the pages of information in Explore mode, each quiz is entirely setup by the educator using the web-based interface; any number of quizzes can be created and stored. Similarly, all the data regarding students' attempts and results at quizzes are viewed using the same interface, avoiding awkward and unreliable data output common to Second Life tools.

Each quiz can consist of any number of questions, each having its own 'weight' (that is, the mark the question is worth in the event of a correct answer). By default, each question has a weight of 1.0, although this can be changed to any value, allowing important questions to be given a greater significance in the final grade. Additionally, each potential answer to a question is given a certain value, with 0.0 representing an incorrect answer, and 1.0 representing a correct answer. Values in between can be used to represent partially correct answers.

The final grade for a student is expressed as the sum of his or her score on each question, as a percentage of the maximum score achievable on the given quiz. In the event of the student taking multiple attempts at the quiz, the grade is calculated in one of five ways, depending on the selection when creating the quiz:

* First = only use the first attempt at each question

* Last = only use the last attempt at each question

* Worst = only use the worst attempt at each question

* Best = only use the best attempt at each question

* Mean = take the average (mean) of all attempts at each question

There are two main question types available for quizHUD-- multiple choice, and Explore--and any combination of them can be used in a quiz. The weight of a question can also be set to 0.0, making the question effectively non-assessed.

1) Multiple Choice Questions

As with other online assessment tools, multiple choice questions were chosen because they are simple to implement and interact with, and are widely used already. In the case of the quizHUD, the questions and available answers appear in the main part of the device, in the same way as information in Explore mode is presented. The limitations of Second Life are such that directly clicking on the answers themselves would be difficult to reliably detect, so instead, the student responds by clicking corresponding tabs down the right hand side.

Each available answer is enumerated in the main display, using letters starting at 'A'; the answer tabs have corresponding letters visible on them, as shown in Fig. 7. When not required, these tabs are faded out and semi-transparent, indicating that they are not available to be clicked.

This method of interaction once again maintains a consistent interface, making it clear to the user that tabs are items which may be clicked on. One disadvantage is that a predetermined limit must be placed on the number of available answers in order to construct the quizHUD; presently, the limit is six, allowing answers 'A' through 'F', although this limit can be increased moderately easily.

[FIGURE 7 OMITTED]

Multiple choice questions presented in this way can be dependent on information available to the student in the environment, requiring his or her familiarity with it. For example, a question could ask the student to identify what type of component appears in a model by finding and examining it; it could also ask about material that has been displayed on the quizHUD in Explore mode.

2) Explore Questions

In response to the identified need to do more than simple multiple choice questions [8], the same interactions performed by the student in Explore mode are turned to assessment. Questions of this type are presented in the same way as multiple choice questions, but instead of seeing a series of answers and clicking a corresponding tab, the student is instructed to answer by clicking on an object in his or her environment. (As with Explore mode, this will only work for objects prepared in advance for the purpose.)

For example, the question in a music course might be, "Identify F# on the virtual keyboard", which the student would answer by approaching a nearby model of a keyboard, and clicking a key on it. Fig. 8 shows an example of how this might look.

The primary impetus for using this kind of question to assess students is that it requires their involvement with their virtual surroundings, incorporating "environment awareness skills" [24] into the learning process. Answering the question correctly may require that the student previously explored the area and has read the associated material, so that he or she knows where relevant objects are, and knows something about them. This requires that the student is active in his or her learning, consciously associating data with related features, and developing a mental 'map' of salient information.

As noted in the literature, it is important that an educator uses the results from assessments to shape the material presented, in order to better address weak areas [8]. All student responses to these exploration-based questions are stored in the database, whether those responses are correct or not, allowing the educator to review them at any time from the web-based interface. This allows him or her to identify problem areas, and consequently redevelop or enhance parts of the virtual environment which are causing difficulties.

[FIGURE 8 OMITTED]

3) Survey Questions

As was mentioned briefly above, the weight of any given question can be set to a value of 0.0, meaning that it has no impact on the final grade. This can be used to administer non-assessed questions, either of multiple choice or Explore types. A quiz could consist entirely of such questions, making the quizHUD capable of administering surveys, and storing results which are accessible in the web-based interface. Alternatively, individual survey questions can be incorporated into otherwise assessed quizzes. Such questions can be used in a variety of ways by the educator, whether for research purposes, or simply to obtain informal feedback on the progress and feelings of the students.

Conducting surveys from within Second Life is certainly not a new idea, as tools such as the Virtual Data Collection Interface (VDCI) demonstrate [25]. However, the most important difference in this case is the availability of the Explore mode, for both presenting material, and asking questions. This can make the experience somewhat more natural for certain types of question, for example where the user must respond by identifying a particular object in the environment.

D. Example Uses

As recent literature and activity show, there are many reasons why educators would wish to use a virtual world such as Second Life, and many ways it can be done (see for example [26-28]). As such, it would be impossible to state definitively how and why the quizHUD could be used, although there are many possible suggestions, whether for formal or casual education, distance or on-campus learners, and so on.

At the time of writing, there exists one main example scenario in Second Life for demonstration purposes . It is part of a larger music-themed demonstration course for the SLOODLE project, with the quizHUD section consisting of three main Explore-mode features:

* Virtual sheet music--each component of sheet music notation can be clicked, such as notes and clef symbols

* Virtual keyboard--each key can be clicked in a single octave

* Virtual band--each instrument can be clicked

All of these clickable features are active in Explore mode to provide information, and many are used in the demonstration quiz to answer questions. A typical question might be "Find a minim on the virtual sheet music", in which case the student would need to locate the symbol for a minim, and click on it.

Another example which was created during development was based on a geology theme. It was set in a large virtual cavern, allowing the user to click on features such as stalactites and stalagmites.

Further potential uses for the quizHUD include, for example, virtual museum tours (that is, a more casual learning experience). At present, a common way to provide information on virtual exhibits is to give a "notecard" (a virtual text file) to the user when they click a feature of interest. This is useful for presenting a large amount of information (far more than the quizHUD can currently handle), but is perhaps somewhat cumbersome and obstructive for those users who only wish to browse briefly. Instead, a more focused set of information can be provided in the common interface of the quizHUD, instead of dispersing it across separate text files. In such cases, the quiz aspect simply becomes a fun self-test, rather than a serious assessment.

E. Technical Implementation

The quizHUD uses the Second Life 'web on a prim' functionality to display web-content to each user, rendered by an external web-application using PHP and a MySQL database. The web-interface for administering the system is also built using PHP, and all source code is available for download (1).

IV. EVALUATION

The preliminary evaluation of the quizHUD was conducted during demonstrations of the technology at two workshop sessions, which were part of broader demonstrations of the SLOODLE project, using the music-themed course mentioned above. The first session had 10 participants, while the second session had 8, with roughly even splits of male/female in both cases. All individuals were in some way involved in education, although most appeared to have moderately little experience of Second Life.

In both cases, the participants had spent time in the quizHUD area, using Explore mode at their own pace, and subsequently undertaking an example quiz, consisting of two multiple choice and two Explore questions. Two additional multiple choice survey questions were then administered at the end of the example quiz, using a 5-point Likert-style scale. Questions and available answers were as follows:

1. "Did the quizHUD fit naturally into the educational setting?"

a. Yes. It was very natural.

b. Yes. It was somewhat natural.

c. Neutral / Not sure.

d. No. It was somewhat unnatural.

e. No. It was very unnatural.

2. "Do you feel the mix of exploration and multiple-choice questions is useful?"

a. Yes. It is very useful.

b. Yes. It is somewhat useful.

c. Neutral / Not sure.

d. No. It is not very useful.

e. No. It is not useful at all.

Complete results for the first and second questions are shown in Tables I and II respectively.

A. Discussion

All participants answered the first question, with two thirds being somewhat or very positive about how naturally the quizHUD fitted into the educational setting. Making sure the tool becomes a natural part of the experience is a significant priority, as anything perceived to be a "boring educational add-on" [13] is unlikely to engage the learner significantly. As such, this is a positive result, especially considering that many of the participants did not have a high level of experience in using Second Life.

For the second question, three quarters of participants were somewhat or very positive about the usefulness of the exploration and multiple-choice combination. This question highlights the underlying aim in developing the quizHUD; that is, to introduce a new technique for learning and assessing students in a virtual world. As such, the result is very encouraging, and given that all participants were in some way involved in education, it suggests that the potential for this tool (beyond the simple demonstration given) could be seen.

V. CONCLUSION

In this paper, we have conducted an initial review of the progress in developing a new learning and assessment tool for use in Second Life, and demonstrated that literature in the area supports its aims and priorities. While quizHUD is unlikely to be the sole justification for an educator using Second Life, it is expected that it will complement other e-learning and e-assessment technologies appropriate to the context, whether that is in a formal educational institution with face-to-face teaching, or an informal educational setting with casual learners.

Although still in its early stages, the quizHUD has shown promise in the way it presents information and encourages learning, with positive feedback from a preliminary evaluation. Furthermore, anecdotal evidence from other informal demonstrations has been very supportive and encouraging, and the source code for setting up a quizHUD system has been downloaded more than 60 times. The need for deeper evaluation is not neglected, however, and further work is required to probe the strengths and weaknesses of the tool more thoroughly, from the perspectives of education and human-computer interaction.

A. Future Work

There are many areas in which improvement and expansion of the quizHUD would be desirable. From much of the literature examined elsewhere in this paper, it seems clear that a significant priority should be providing means for the educator to specify qualitative feedback to be delivered to the student upon answering a question. Additionally, there are plans to bring the quizHUD more fully into line with the aims of the SLOODLE project, such that it will be able to operate from within the Moodle learning management system (http://moodle.org), linking quiz results into its 'gradebook' functionality.

More rigorous evaluation of the concept and implementation is also essential to ensure that it is useful and that it meets the needs of educators and learners.

ACKNOWLEDGMENTS

Thanks to Clement Ramel, who contributed to the early development of quizHUD.

REFERENCES

[1] A. Kirkwood, "Getting it from the Web: why and how online resources are used by independent undergraduate learners," Journal of Computer Assisted Learning, vol. 24, 2008, pp. 372-382.

[2] M. Grabe and K. Christopherson, "Optional student use of online lecture resources: resource preferences, performance and lecture attendance," Journal of Computer Assisted Learning, vol. 24, 2008, pp. 1-10.

[3] K. Woo, M. Gosper, M. McNeill, G. Preston, D. Green, and R. Phillips, "Web-based lecture technologies: blurring the boundaries between face-to-face and distance learning," ALT-J, vol. 16, 2008, p. 81.

[4] T. Wang, "What strategies are effective for formative assessment in an e-learning environment?," Journal of Computer Assisted Learning, vol. 23, 2007, pp. 171-186.

[5] H.S. Ashton, C.E. Beevers, A.A. Korabinski, and M.A. Youngson, "Incorporating partial credit in computer-aided assessment of Mathematics in secondary education," British Journal of Educational Technology, vol. 37, 2006, pp. 93-119.

[6] C.V. Gipps, "What is the role for ICT-based assessment in universities?," Studies in Higher Education, vol. 30, Apr. 2005, pp. 171-180.

[7] D.A. Patterson, "A large-scale, asynchronous, web-based MSW comprehensive exam administration: outcomes and lessons learned.," Journal of Social Work Education, vol. 42, 2006, pp. 655-668.

[8] D. Nicol, "E-assessment by design: using multiple-choice tests to good effect," Journal of Further and Higher Education, vol. 31, Feb. 2007, pp. 53-64.

[9] R.C. Alvarado, "Overcoming the Fear of Gaming: A Strategy for Incorporating Games into Teaching and Learning," EDUCAUSE Quarterly, vol. 31, 2008, pp. 4-5.

[10] A.J. Kelton, "Virtual Worlds? "Outlook Good"," EDUCAUSE Review, vol. 43, 2008, pp. 15-22.

[11] P. de Lange, T. Suwardy, and F. Mavondo, "Integrating a virtual learning environment into an introductory accounting course: determinants of student motivation," Accounting Education, vol. 12, 2003, p. 1.

[12] T. Erickson and W.A. Kellogg, "Social translucence: an approach to designing systems that support social processes," ACM Transactions on Computer-Human Interaction, vol. 7, 2000, pp. 59-83.

[13] F. Bellotti, R. Berta, A.D. Gloria, and V. Zappi, "Exploring gaming mechanisms to enhance knowledge acquisition in virtual worlds," Proceedings of the 3rd international conference on Digital Interactive Media in Entertainment and Arts, Athens, Greece: ACM, 2008, pp. 77-84.

[14] R.L. Jackson and E. Fagan, "Collaboration and learning within immersive virtual reality," Proceedings of the third international conference on Collaborative virtual environments, San Francisco, California, United States: ACM, 2000, pp. 83-92.

[15] M. Leidl and G. Ro[??]ling, "How will future learning work in the third dimension?," Proceedings of the 12th annual SIGCSE conference on Innovation and technology in computer science education, Dundee, Scotland: ACM, 2007, p. 329.

[16] J. Kemp and D. Livingstone, "Putting a Second Life "Metaverse" Skin on Learning Management Systems," Proceedings of the Second Life Education Workshop at SLCC 2006, San Francisco: 2006, pp. 13-18.

[17] D. Livingstone, J. Kemp, and E. Edgar, "From Multi-User Virtual Environment to 3D Virtual Learning Environment," ALT-J, vol. 16, 2008, p. 139.

[18] T. Erickson, "Working with interface metaphors," The art of human-computer interface design, B. Laurel, ed., MA: Addison-Wesley, 1990, pp. 65-73.

[19] J.A. Clarke, "Students' Perceptions of Different Tertiary Learning Environments," Higher Education Research & Development, vol. 17, 1998, p. 107.

[20] B.C. Dart, "A goal-mediational model of personal and environmental influences on tertiary students' learning strategy use," Higher Education, vol. 28, Dec. 1994, pp. 453-470.

[21] A. McGrath and A.J. Munro, "Footsteps from the garden: Arcadian knowledge spaces," Designing Information Spaces: The Social Navigation Approach, K. Hook, D. Benyon, and A.J. Munro, eds., London: Springer-Verlag, 2003, pp. 175-200.

[22] J.A. Waterworth, A. Lund, and D. Modjeska, "Experiential design of shared information spaces," Designing Information Spaces: The Social Navigation Approach, K. Hook, D. Benyon, and A.J. Munro, eds., London: Springer-Verlag, 2003, pp. 125-149.

[23] "SL Educators Mailing List"; https://lists.secondlife.com/cgibin/mailman/listinfo/educators.

[24] R.J. Jacob et al., "Reality-based interaction: a framework for post-WIMP interfaces," Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, Florence, Italy: ACM, 2008, pp. 201-210.

[25] M.W. Bell, E. Castronova, and G.G. Wagner, "Virtual Assisted Self Interviewing (VASI): An Expansion of Survey Data Collection Methods to the Virtual Worlds by Means of VDCI," SSRN eLibrary, 2008; http://ssrn.com/paper=1188562.

[26] R. Hetherington, J. Bonar-Law, T. Fleet, and L. Parkinson, "Learning in a Multi-User Virtual Environment," Proceedings of the 2008 International Conference Visualisation - Volume 00, IEEE Computer Society, 2008, pp. 99-105.

[27] Chang-Hyun Park, Gilsoo Jang, and Young-Ho Chai, "Development of a Virtual Reality Training System for Live-Line Workers.," International Journal of Human-Computer Interaction, vol. 20, May. 2006, pp. 285-303.

[28] G. Siegreid, "Kent ensemble to create virtual play - News," kentnewsnet.com, Oct. 2008; http://media.www.kentnewsnet.com/media/storage/paper867/news/20 08/10/15/News/Kent-Ensemble.To.Create.Virtual.Play3487354.shtml.

SLOODLE is funded and supported by Eduserv.

(1) Further information is available online at: http://www.sloodle.org/quizhud

Peter R. Bloomfield, Daniel Livingstone

School of Computing

University of the West of Scotland 

Paisley, Scotland

[e-mail]

TABLE I. "Did the quizHUD fit naturally into the educational setting?"

             Answer                 Number             %

a. Yes. It was very natural.           6             33.33%
b. Yes. It was somewhat natural.       6             33.33%
c. Neutral / Not sure.                 3             16.67%
d. No. It was somewhat unnatural       1             5.56%
e. No. It was very unnatural.          2             11.11%

TOTAL #:                              18

TABLE II. "Do you feel the mix of exploration and multiple choice
questions is useful?"

             Answer                 Number             %

a. Yes. It is very useful.             9             56.25%
b. Yes. It is somewhat useful.         3             18.75%
c. Neutral / Not sure.                 2             12.50%
d. No. It is not very useful.          2             12.50%
e. No. It is not useful at all.        0             0.00%

TOTAL #:                              16