Evaluation of an online multimedia, teaching and learning resource: Allsorts Primary School.
As part of the EFFECTS award (Effective Framework for Embedding Communications and Information Technology using Targeted Support), the necessity of evaluating the use of Information and Communications Technology (I&CT) in teaching and learning was reinforced and, as a result, the following study has been produced. This formative evaluation reflects upon the design of Allsorts Primary School, an interactive multimedia online resource (Figure 1), by analysing the responses gathered from a sample of teacher trainees identified as the intended users. The evaluation used a combination of questionnaire and focus group discussion, both of which required the trainees involved to reflect upon their personal experience of Allsorts Primary School. The responses from the trainees indicated that they viewed their experiences of the virtual primary school as enjoyable and effective in developing their understanding of the complex issues surrounding the implementation and development of ICT within the primary context.
JI. of Technology and Teacher Education (2003) 11(3), 333-346
Allsorts Primary School is an evolving multimedia learning resource for trainee teachers. During the final year of their BA (Education), trainee teachers for the primary phase complete a module which aims to equip them with the skills, knowledge and understanding necessary for the co-ordination of their chosen specialist subject. As primary school coordinators of Information and Communications Technology (ICT), they will encounter many challenges, which can be divided into two main categories: (a) the provision and organisation of resources; (b) and the attitude and expertise of the staff.
Currently, the successful completion of the module relies on the reflection and analysis of experiences gained by the trainees during teaching practices. As the trainees may encounter schools which are equipped to offer differing levels of support and guidance in this complex role, it is possible that some may be given more assistance than others which will, in turn, affect the assessment of those trainees.
[FIGURE 1 OMITTED]
To overcome this difficulty, a virtual primary school has been designed and developed (French, Cumpson, & Wood, 2001) which aims to reflect the status of many English primary schools in terms of ICT. Allsorts Primary School, as it has been named, offers an interactive, multimedia experience for the trainees and is accessible through the World Wide Web (WWW or Web). Video interviews and interactive models of classrooms allow the trainee to ask questions of teachers and explore the classrooms with a view to assessing the current status of ICT within the school. Having engaged with the virtual school, the trainees can discuss the factors which they feel are driving or restraining the development of ICT and relate this to their own experiences gained in schools. All trainees are therefore provided with an equal basis on which to complete their summative assignment of a school development plan which caters to the needs of Allsorts Primary School.
The following evaluation focuses upon the ability of Allsorts to meet the main objectives originally outlined at the onset, (Inglis, Liking, & Joosten, 1999) namely that the school should:
* reflect "real-world" experiences--for example, do the staff attitudes towards ICT reflect those of actual teachers currently working in primary schools;
* enhance the learning process; and
* be easy to use and accessible.
Joliffe, Ritter, and Stevens (2001) identified this process of evaluation as "Level 1: Reaction" where the learners' feelings and opinions about a web based learning event are evaluated.
Such an evaluation enables the designers to make appropriate alterations in response to the feedback gained from the respondents, who are in this case the trainees. Furthermore, as Joliffe et al., (2001) pointed out, not only does such an evaluation provide valuable feedback on the effectiveness of the resource, it also encourages the trainees to "share responsibility for their own learning."
The main purpose of this research was to evaluate the actual resource against the original concept and criteria for development. Allsorts Primary School is designed to reflect the circumstances of many primary schools, in terms of ICT, within England. From the perspective of the content designer, this needs to reflect the provision of resources, the environment of the school itself, attitudes and abilities of teachers in addition to the provision of training for staff. When faced with Allsorts School, the trainees are able to view information which will allow them to assess the current situation of Allsorts with respect to ICT. This information may be text-based, for example onscreen data indicating the age and type of computer. Alternatively, factors may be implicit within the general attitude of the staff, and may be identified through nonverbal communication, for example the facial expressions or tone of voice.
The design of the interface and the choice of technology also needs to be evaluated. Where Allsorts School is concerned, the emphasis has been placed upon accessibility and ease of use. This includes navigation, quality of the material and the overall look and feel of the resource.
[FIGURE 2 OMITTED]
[FIGURE 3 OMITTED]
[FIGURE 4 OMITTED]
INTERFACE AND TECHNOLOGY
Two full-time multimedia developers, in conjunction with a professional video production team and the academic content specialist, undertook the production of the application. Resources were made available as part of a staff and materials development initiative at University of North London.
The school interface was built in hypertext markup language (html) for online distribution. The navigation system consists of four buttons that lead to different aspects of the experience:
* interview--pre-recorded interviews with members of staff (see Figure 2);
* explore--the school physical environment (see Figure 3);
* discuss--an interactive "staff room" discussion board, where trainees can exchange views and make contact with virtual staff, who will make responses in character (see Figure 4); and
* home page.
The interviews were scripted, but actors improvised to give the videos a natural feel. Interviews were filmed using a digital video camcorder, then captured and edited on an IBM PC using Adobe Premiere. Each interview was cut into a set of twelve responses to specific questions.
The choice of technology for web distribution was related directly to cost and usability considerations. There are currently four competing technologies for streaming video over the Internet--(a) RealMedia, (b) Apple QuickTime, (c) Windows Media and recently, (d) Macromedia Flash MX. For each format, the client player is freely available as a web download, so that any user can use any system. However, the server that distributes the video files is specialised, and at the time of production, only Windows Media Server was both free and suitable for installation on the Windows NT operating system. Another advantage was that the Windows Media Player comes bundled with all Windows operating systems, meaning that the interviews would run on the university systems without the need for downloading and installing new software.
The fact that Allsorts School was being accessed by trainees through university networks meant that the bandwidth for receiving data was high. Therefore, it was not necessary to compromise on video quality when compressing files into streaming video format. This was done using Windows Media Producer, which is also free.
The user clicks on a teacher, then on a question, which launches the clip with the appropriate answer. A benefit of this interface is that it allows the user to either focus on one member of the staff or on one question.
The school is presented as an interactive ground plan. This is a simple image map with hot spots that can be clicked to load different classroom pages into the adjacent frame in the browser window. Each classroom appears as a panoramic image that users can zoom into or mouse around.
The creation of these assets was dependent on the cooperation of a local junior school, Sythwood Primary School in Woking, which kindly allowed the development team access to their classrooms during the half term break. A tripod was set in the centre of each classroom and turned through 360 de grees to take a series of photographs around the room. The resulting images were stitched together using Adobe Photoshop to create panoramic images, which were then embedded into HotMedia applets. IBM HotMedia is another free technology that enables the creation of interactive panoramas. Applets require no extra plugins as they play automatically in web pages on Java-enabled machines.
Virtual Staff Room
The staff room bulletin board is another free utility, hosted by Bravenet. Unfortunately, popular free services such as these need to place advertisements to raise revenue. As a consequence, Allsorts staffroom suffers from annoying banners, which are not part of the original design.
The research has employed a case study approach (Robson, 1993) where two groups of trainees have been required to interact with the resource and provide feedback through semi-structured group discussions and questionnaires. The trainees selected have been representative of the intended users of Allsorts Virtual School. As such, their feedback provides indicators of the effectiveness of the material in achieving predetermined objectives and consequently helps to identify any further developments which may be necessary. A case study approach was selected as it provided some flexibility in the process of evaluation. Although critical research questions were defined prior to the data collection, it was possible, if necessary, to deviate from these during the semi-structured group discussions to pursue a line of inquiry raised by the sample of trainees.
Teacher trainees for the primary and early years phase, currently studying as subject specialists in primary ICT, were offered the opportunity to take part in the evaluation of Allsorts School. They were, at the time of the research, completing their second year at both Kingston University and the University of North London. In their third and final year on their respective BA/B.Ed courses, they will be required to complete a module on the ICT coordinator's role. As such, Allsorts Primary School is intended to provide an appropriate medium where the trainees can explore and analyse the complex factors affecting the implementation and development of ICT within the primary context. Prior to this module, the trainees would have had experience of working within a variety of primary schools and would therefore be able to evaluate how well Allsorts reflected reality. They would also have some specialist knowledge and understanding regarding ICT and not only could they evaluate the resource in terms of content, but also in terms of design.
The trainees concerned participated in their respective university campus where the equipment and conditions for the evaluation of the software could be prepared appropriately--for example access to computers and a separate room for subsequent interviews. More importantly however, by using the same facilities the trainees would be using during the completion of their ICT coordinator's module, it was possible to observe whether the soft ware ran as expected on each university network.
Although a larger sample could have been procured, a small sample was chosen for a number of reasons. First, we intended to interview the trainees after the questionnaire was completed and a small group was more easily manageable in terms of transcribing and facilitating. Second, the target users will be a group of approximately thirty trainees and we therefore concluded that a sample of nine trainees was a good representation of the intended trainee audience at this stage of the development.
DATA COLLECTION TECHNIQUES
The evaluation of Allsorts School involved the use of questionnaires and group discussions. The combination of these methods strengthened the overall evaluation as each compensated for the possible weaknesses of relying upon one particular method. For example, questionnaires generally offer quantitative data which may be superficial yet easy to analyse, (Boyle, 1997; Robson, 1993) while interviews provide rich, qualitative data (Boyle, 1997) but require more time for transcription and analysis (Robson, 1993).
Milne (as cited in Harvey, 1998) continued to offer a more detailed ex amination of the advantages and disadvantages of questionnaires. One point is that questionnaires limit the response of the individuals to the choices given and therefore restricts the participants. To overcome this limitation, the questionnaire designed for this evaluation included a rating scale (Likert scale) (Mogey, as cited in Harvey, 1998) for responses and space for the participants to explain or elaborate upon their response. The space provided for these responses was considered large enough to prevent the participants feeling restricted yet not so great that they felt they should write vast quantities of information. It is important to note that the subsequent interview also provided them with time to add further information. After exploring Allsorts School for approximately one hour, they were required to complete the questionnaire so that their immediate, individual perceptions were collected. During the interaction with Allsorts, trainees were provided with notepaper to allow them to make notes as they saw fit.
At all times, the trainees were reassured that the evaluation was focused upon the resource, not upon their ability to use it. They were asked to be honest in their response and appraise the resource from their own perception. The questionnaires were completed individually so the trainees offered their own thoughts and opinions rather than those which may have been in fluenced by their peers.
The "interview," which could be more accurately described as a focus group discussion, was designed to replicate a teaching and learning situation. This approach reflects the views taken by people such as Draper (1997) and Mitchell (2000) who raise concerns over evaluation methods which do not relate to the context for which the technology was designed. During the exploration of the resource, trainees were given the task of "interviewing" five of the members of the staff and exploring the classrooms within the school. By selecting five members of the staff, the trainees could successfully complete the interviews within the time allocated for the exploration. Their task was to assess the current situation of the school as if they were the new ICT coordinator and report back during the interview. The discussions were facilitated by one of the developers and although a predetermined set of questions was identified, there was no specified order to these. This flexible framework allowed the trainees to explore issues which were not necessarily outlined within the initial set of questions. Each group's discussion was recorded using an audio tape recorder and transcribed for later analysis.
In the case of the semi-structured discussion, the responses of the trainees were accompanied with expression and intonation. This was particularly useful when analysing the way the trainees perceived the members of the staff within Allsorts School as their statements provided additional information which could not be recorded through completion of closed or open-ended questionnaires.
Real World Experience
From individual responses in the questionnaires and during the group discussions, both sets of trainees gave positive feedback on the content of Allsorts School. They stated that it reflected the experiences that they had gained within schools with both ICT resources and staff attitudes mirroring reality. For one trainee in the first group, the fictional teacher David Cook had an attitude that resembled those of a teacher with whom he had recently worked. "My teacher at the moment, she's nearly to retirement age and of course everything he said, David Cook, I can hear her voice."
Indeed, during the discussions, the trainees referred to the virtual teachers as if they were real people, suggesting that the characters were believable representations of teachers currently working in primary schools. The trainees were particularly drawn to the characters who possessed very negative attitudes towards ICT, such as David Cook and Philip Armitage. It appeared that non-verbal communication, in addition to the scripted words, was providing trainees with additional information regarding the attitude of the staff. Tone of voice, shrugs, sighs and facial expressions emphasised the feelings of these characters and, during the discussion, inferences were made which interpreted both verbal and non-verbal communication.
"David Cook was a little bit demoralised." "There was no enthusiasm in that school about computers." "He looked really bored with the whole thing." "I felt that the Head could turn it around."
The use of video to portray the virtual staff added to the representation of the attitudes in a way that sound files alone or animated characters could not. In effect, the information became richer in meaning (Boyle, 1997). Similarly, the use of panoramas of an actual school seemed to create an image of reality.
But there was one room where there was a computer which was covered up and one room where the computer just had something on top of it like some paper or something, which implied that it was never used. In 3a there was a purple piece of material at the side and there was a globe next to it and it was just like if they were put in the room with no thought ... They could have changed it around and the computer could have a nice little corner. I noticed that behind the actual computers, the displays had nothing whatsoever to do with ICT.
The trainees examined the displays on the wall, the location of the computer and the general organisation of each classroom to extract further information regarding the implementation of ICT within the school.
Enhancement of Learning
Although there is no actual measurement of trainees' learning, the discussions and feedback from questionnaires provided indicators as to the way the trainees perceived Allsorts School as a resource for learning and teaching.
During the discussion, the trainees began by describing what they had understood from the exploration of the school. They assessed the attitudes of the staff and the resources within the classrooms. By relating this to their existing knowledge and understanding, the trainees then began to construct meaning (Bruner, 1990) and made inferences regarding the reasons for the attitudes of the virtual teachers. The discussion touched upon some of the complex relationships operating within the primary school and allowed the trainees to rationalise their thoughts and ideas regarding potential routes for developing ICT.
Allsorts School appeared to enhance this particular process by creating a realistic scenario with believable characters. In addition to this, the trainees seemed to enjoy the experience and felt motivated with the material presented being useful and easier to follow than text or lectures (Figure 5). The results represented by the following chart are only indicators, yet the overall response does suggest that the trainees' learning experiences were positive.
Overall Experience of Trainees Using the Virtual School
Three of the trainees indicated that they felt the experience was "isolating." However, this may be accounted for by the nature of the evaluation which required the trainees to work independently to insure receiving each individual's own perception.It is encouraging to note that many trainees found that engaging with Allsorts School motivated them. As highlighted by Race (as cited in Brown et al., 1998), interest in trainee motivation in higher education has increased as drop-out and failure rates have also increased. If such a resource offers trainees an enjoyable route to learning and higher levels of motivation it is possible that they will be better equipped to succeed.
Accessibility and Usability
Due to the nature of the resource, the trainees were able to replay events such as the interviews and panoramas (Jolliffe et al., 2001). They were able to explore the school at their own pace and take control of their learning by making decisions as to the route they would take. In addition to this, information which was not necessarily accessible to them during their school experience was now available. One trainee stated that, "It was lovely to be able to be so nosey! You could keep going back and poking around. If you were visiting a real school, even with a view to working there, you would not be able to do this!"
[FIGURE 5 OMITTED]
The majority found the resource "easy" or "very easy" to navigate and rated the look and feel of the resource as "reasonable." In the latter case, additional comments revealed that trainees would have liked the ability to move from one classroom to another, as if the user was walking through the school. In addition to this, a number of the trainees expressed the wish to explore the contents of the classrooms in more detail, for example the children's work displayed on the walls. One trainee felt very strongly that the ability to interview children would be extremely useful. This is something which was originally considered in the development process. Enhanced educational experience of children in schools is the ultimate goal of any learning and teaching event involving Allsorts. It would follow that the children within Allsorts would need to have a voice to express their thoughts and experiences regarding ICT within the school. As such, this particular development is viewed as highly desirable.
In conclusion, it appears that Allsorts Primary School fulfills the objectives it was designed to achieve. From the trainees' perspectives, it reflects the "real world" circumstances of ICT within primary schools. Interaction with the resource prompted enthusiastic responses and, although there is no measurement of the learning which may have taken place during the evaluations, the positive responses from both the questionnaires and discussions indicated that Allsorts offers a strong potential for supporting and enhancing the learning and teaching process. There were no difficulties in navigating the resource and all of the video clips were accessible.
Alongside the suggestions for development described previously, a number of trainees from both groups voiced the opinion that they would like to interact with Allsorts by affecting change and observing the results. For example, trainees could take control of a budget with the intention of buying resources and providing training for the staff. Adapting the current design to include such a facility would be time-consuming, as it would be difficult to build in deeper levels of interactivity at this stage. However, potential solutions offered by one of the multimedia developers included the development of an interactive game that would allow trainees to play with the school budget and provide peer review of ideas with an online voting system. Using a discussion board, teams of trainees accessing Allsorts through the web would offer ideas for development within the school, using a set budget to achieve what they considered to be the best solution. All trainees would then assess each other's ideas and vote upon the plan which they felt addressed the needs of the school most effectively.
At present, Allsorts has been evaluated as a resource accessed from the Web and discussed in a face-to-face situation. During the following year it will be integrated within a virtual learning environment where discussions will be undertaken through the medium of the Internet. The intention is to create an online learning community which unites individuals from a variety of locations, for example ICT subject specialists from Kingston University and the University of North London. Future evaluations will therefore be focused upon the design of such a learning environment and the impact of Allsorts on learning and teaching in this situation. Activities such as the one described in the previous paragraph would support the development of Allsorts School and offer opportunities for discussion and collaboration through the use of computer supported communication.
Boyle, T. (1997). Design for multimedia learning. Upper Saddle River, NJ: Prentice Hall, Cornwall.
Brown, S., Armstrong, S., & Thompson, G. (Eds.) (1998). Motivating students (SEDA Series). London: Routledge Falmer.
Bruner, J. (1990). Acts of meaning. Cambridge, MA: Harvard University Press.
Draper, S. W. (1997). Prospects for the summative evaluation of CAL in higher education. Association for Learning Technology Journal, 2(1), 33-39.
French, F., Cumpson, I., & Wood, R. (2001). Specification and design of an interactive virtual environment for use in teacher training. Association of Learning Technology Journal 9(1), 50-61.
Harvey, J. (Ed.) (1998). The evaluation cookbook. Learning technology dissemination initiative, Institute for Computer Based Learning, Heriot-Watt University, Edinburgh. [Online]. Available: http://www.icbl.hw.ac.uk/ltdi/cookbook/cookbook.pdf
Inglis, A., Liking, P., & Joosten, V. (1999). Delivering digitally: Managing the transition to the knowledge media. London: Kogan Page.
Jolliffe, A., Ritter, J., & Stevens, D., (2001). The online learning handbook: Developing and using web-based learning. London: Kogan Page.
Mitchell, P. D. (2000). The impact of educational technology: A radical reappraisal of research methods. From The Changing Face of Learning Technology, Association for Learning Technology, University of Wales Press.
Robson, C. (1993). Real world research: A resource for social scientists and practitioner-researchers. Oxford, UK: Blackwell.
University of Greenwich
University of North London
|Printer friendly Cite/link Email Feedback|
|Publication:||Journal of Technology and Teacher Education|
|Date:||Sep 22, 2003|
|Previous Article:||Technology beliefs and practices of mathematics education faculty.|
|Next Article:||Reflection through the ID-PRISM: a teacher planning tool to transform classrooms into web-enhanced learning environments.|