Formative assessment of classroom concept maps: the reasonable fallible analyser.

Concept mapping is a powerful learning technique that can be enhanced by computer technology. Software tools are already available for the preparation of concept maps but as yet, few systems provide feedback on their content. The claim made by this article is that by enlisting the student as an assessment partner, computer-based feedback becomes both feasible and useful. The article reports on the development of a semi-automated software analyser for formatively assessing students' concept maps. The analyser is fallible: many of its assessments are initially incorrect. Crucially however, it is also reasonable: a student is able to argue for a more favourable assessment. The article reports classroom trials which indicate that high-school students' experience of concept mapping is enhanced by use of the analyser. The system's wider significance is that it points towards a new genre of educational software based upon a more genuinely dialogic model of human-computer interaction.

**********

A concept map is a graphical representation of knowledge that uses a network of symbols to describe relationships between ideas. Typically, each linked pair of symbols specifies a proposition. Based on the constructivist theory of meaningful learning (Ausubel, Novak, & Hanesian, 1978), concept mapping was developed as a learning technique (Novak & Gowin, 1984). By articulating their understanding and rendering it in visual language, learners engaged in making a concept map are encouraged to reflect upon and refine their knowledge.

Like most kinds of documents, concept maps can benefit from technological support. Currently available software includes Conception (http://www.parlog.com), CmapTools (http://cmap.ihmc.us/), and Inspiration (http://www.inspiration.com). Such tools enhance the preparation, revision, and utility of concept maps in many ways, for example by enabling a concept map's symbols to act as hyperlinks to other concept maps, web pages, and multimedia resources. Until recently, however, programs have done almost nothing to provide students with feedback on the quality of their maps. Given the general importance of feedback in learning, this is a significant omission.

Obviously, a fully intelligent or infallible analyser for unrestricted concept maps--one that can evaluate a student map on any topic and generate unerringly authoritative and insightful suggestions--is beyond the state of the art in artificial intelligence. Such a program would amount to a linguistic prodigy and a universal sage. The claim made by this article however, is that useful forms of computer-based feedback do become feasible when a student's role is promoted from a passive recipient of assessment to an active partner in the process. The article reports the development of a semi-automated software analyser that shows some promising results. The analyser, titled the Reasonable Fallible Analyser (RFA), compares a student's concept map to a presupplied expert's map for the same domain. It generates a score for students' maps and offers hints for improvement. Owing to its limited intelligence, the RFA's initial assessments are not expected to be highly reliable. However, a design goal has been to make a virtue of the system's fallibility. The system's misconstrual of a student's map is taken as an opportunity to initiate an argumentative dialogue, which may benefit the student's learning as well as improving the accuracy of the assessment.

The remainder of the article is in five parts. First, the motivation and general aims for this research are described. Second, a brief review of relevant previous work is provided. Third, the design and functionality of the analyser is reported. Fourth, some classroom experiences are reported that are based on trials with the RFA in an Edinburgh high-school. The trials indicate that high-school students' experience of concept mapping is enhanced by the RFA. Students enjoy arguing with the system, accept its scoring as fair, welcome its hints, and are frequently stimulated to revise their maps to accommodate the feedback obtained. Finally, some discussion is offered around the content of this work, including its relationship to the wider context of assessment, pedagogy and concept mapping.

MOTIVATION AND AIMS

Envisage a high-school classroom in which students are nearing the end of their work on some topic. In pairs, they have been busily making concept maps to summarise their understanding of the material. One pair has nearly completed the task as the teacher approaches. Some encouraging comment is offered but the students are not quite satisfied with this level of reaction. They want to know: "But is our concept map any good?"

The students' question is a reasonable one but it presents the teacher with a problem. The response could be any one of at least three kinds:

* The teacher could sit down with the students and study and discuss their map. Potentially he or she can provide rich feedback based on its contents, bringing to bear an expert understanding of the topic as well as detailed knowledge of the students' learning.

* The students might be requested to self-assess their own map, perhaps by discussion with classmates in which maps are compared with a view to agreeing on improvements.

* The teacher could say: "Your concept map doesn't matter, it's what you learned in making it that counts. Besides, everybody's understanding is unique and valuable, so its impossible to say whether a concept map is any good or not."

The first response is ideal but also perhaps, idealistic. With this approach the teacher gains knowledge of the students' progress and the students benefit from high-quality feedback. Unfortunately, since classes normally contain many pupils it is feasible only seldom. The second response might be very productive in certain contexts but it depends upon the ability of peers to provide high-quality interaction. As for the third response, it seems unhelpful and misguided. Of course the process of concept mapping is more important than the product and it is true that a map provides a personal view of its domain. But where a concept map is intended to summarise routine curriculum material within a social learning context, the qualities of accuracy, completeness, and clarity that it displays are surely assessable and significant. Feedback plays a vital role in learning generally and to disregard the possibilities that concept maps afford for it is to miss an opportunity and also, possibly, to undermine students' confidence in concept mapping as a worthwhile activity.

The development of the RFA was motivated by a concern that in practice, many students are making concept maps for which they get little or no feedback. It seemed a worthwhile challenge to develop a system that could analyse a student's map at a level that would be sub-expert but nevertheless good enough to stimulate productive reflection. In the next section, previous work is summarised that was helpful in translating this challenge into a feasible design.

PREVIOUS WORK

A considerable body of research has provided evidence for the efficacy of concept mapping in various learning contexts: see for example the special issue of the Journal of Interactive Learning Research (1998), which was devoted to applications of concept mapping. The use of concept maps for assessment was identified by Novak and Gowin (1984) but the scoring scheme that they recommended, based on counting levels of hierarchy and cross-links, has been controversial. Recent research has favoured schemes based on the correctness of propositions (symbol-to-symbol links). Concept maps evaluated in this way by human experts show a high degree of inter-rater reliability and give results that have been found to correlate well with both classroom and standardised tests (Ruiz-Primo & Shavelson, 1996; Rice, Ryan, & Samson, 1998).

Very few concept mapping systems are documented that provide nontrivial feedback to students. Two notable exceptions must be mentioned, however. The first is Betty's Brain (Biswas, Schwartz, & Bransford, 2001), an intriguing learning-by-teaching system in which a student's concept map is the means by which an artificial agent (Betty) acquires knowledge. Once the map is built, students can test it by asking questions of Betty. Although impressive, the system is restricted in scope to a fairly narrow class of maps representing semi-quantitative models of processes in the life sciences. The second exception is the system by (Chang, Sung, & Chen, 2001) which offers hints and assessment by comparing a student's map to one prepared by an expert. This program requires that students' maps must be prepared within a purpose-built environment which constrains students to use exactly the same concept and relation names as those used by the expert. A deviation from the expert map's representation of information is taken as an error.

Recently Reese (2003) has described a novel method of concept map quantification based on the translation of concept maps into Pathfinder Networks. She claimed that such a translation will enable students' concept maps to be compared to an expert's map in a way that is ideal for formative evaluation and feedback. Her experimental study, although it relied upon human translators, seems promising. As yet however, no automated system is known that implements this approach and it remains to be demonstrated that effective feedback can be generated from what is an essentially statistical ("black-box") processing technique.

PHILOSOPHY UNDERLYING THE RFA

The RFA is similar to Chang's system in relying upon an expert's map to provide a basis for comparison in assessing a student's map (referred to henceforth as an E-map and an S-map, respectively). It differs from Chang's system, however, in that the RFA imposes no restrictions on the language that students may employ. The effectiveness of concept mapping as a learning technique appears to be closely tied to the opportunity which it affords to students to describe their understanding in their own terms. The design of the RFA respects this freedom, but because of the weakness of computers in interpreting natural language, the RFA is fallible: some of the student's language might be misunderstood by the system and therefore its assessments may be incorrect. However, the primary purpose of the RFA is to provide formative assessment--that is, feedback to a student which is supportive of his or her further learning--rather than summative assessment that is traditionally intended to provide a reliably accurate measure of knowledge and skill. The defence of the RFA's fallibility is that, within the context of formative assessment and a social learning environment, reliable accuracy is less important than the capacity to stimulate reflection.

The design of the RFA aims to stimulate reflection in two specific ways. First, the system adopts a transparent approach to feedback that includes not merely an overall (numeric) score but also a full justification of how the score was obtained and detailed hints for improvement. Thus a student can determine what parts of the S-map have been given credit and also, in what ways it may have been misinterpreted. Second, the RFA is open about its fallibility and provides scope for compensation. It admits that its abilities are limited, positively recommends that each assessment should be disputed, and provides opportunities to negotiate an improved score through forms of argument.

We acknowledge that this design philosophy, which we term ORF (for "Open, Reasonable, Fallible"), is unusual in the field of learning technology. However, recent trends towards openness in the development of intelligent learning environments are fairly clear. For instance, whereas early intelligent tutoring systems maintained only covert (one might say, intransigent) representations of the learner, recent developments have favoured overt learner models, including negotiated or user-editable ones (Morales, Pain, Bull, & Kay, 1999). Not only are overt models a pragmatic response to the difficult problem of achieving accuracy and reliability in automated modelling, they may also reflect a promotion of the learner to a status of sharing responsibility for the learning process.

In the case of the RFA, a pragmatic approach is essential because, even if the program could somehow overcome its linguistic limitations, its performance would still be limited by its dependence upon a single E-map to provide the basis of comparison. Although such a map can serve the purpose of a traditional "model answer" in providing a normative view of the relevant domain, it typically falls below what would be required of a comprehensive definition. Of course, it is conceivable that some aggregation of multiple diverse E-maps might increase the authority of the system and this remains a possibility for future research. In this project however, we were more attracted by the idea of promoting the learner than of upgrading the system. Instead of an authoritative assessor criticising a weak student, we envisaged empowering the student to criticise a weak assessor.

The ORF approach potentially transforms a students' relationship with a computer. Instead of being intimidated by a mythically superior artificial intelligence, a student interacting with the RFA is encouraged to view the system as an inexpert assessor with fragile comprehension skills and super ficial subject matter knowledge. This is not to say that the RFA is a pushover: it applies its limited information resources with boundless clerical diligence. But the balance of power between system and student is radically more equal than is the case in (for instance) Chang's (2001) system. This situation could open new possibilities for interaction and learning. Among other important questions, researchers need to investigate whether today's high-school students can respond critically to computer-generated feedback that is declared to be imperfect, and to what extent such feedback is better for learning than no feedback at all.

SCHEMATIC DESIGN

A schematic view of the RFA's software design is shown in Figure 1. Here, boxes represent data, ovals represent processes implemented by system components and arrowed lines represent data flow. The RFA is implemented in LPA Prolog++ (http://www.lpa.co.uk) and presently runs under Windows and Macintosh operating systems. It currently accepts concept map files in Conception (http://www.parlog.com) format only but it should be straightforward to incorporate other formats, so that students and teachers can continue to use their preferred map-building environments.

The RFA's current parser is not sophisticated. It implements some common morphological transformations and attempts to recognise concept labels as noun phrases and relation labels as verb phrases. The parsing process is supported by WordNet, a lexical database accessible from Prolog that incorporates a dictionary of 200,000 words (Witzig, 2003). Although this is an impressive resource, in practice concept maps often use specialist vocabulary with words that are not found in WordNet. For this reason, a separate User Dictionary is incorporated, which can be edited to define additional synonyms that can be anticipated to occur within S-maps.

The matcher's main job is to find the optimum mapping of the E-map's parsed terms onto those of an S-map. If the E-map contains ES symbols and EL links and the S-map contains SS symbols and SL links then this requires ESxSS+ELxSL text matching operations. In practice, performance is acceptably fast. From the optimum mapping is computed the S-map's score and other results that become visible through the user interface. The score is currently calculated with the formula Score% = (2*Sfcp + Spcp + Svc)*100/(2*Ep + Ec)--where Sfcp counts the fully correct propositions in the S-map, Spcp counts its partly correct propositions, Svc counts its valid concepts, EP counts the propositions in the E-map, and Ec counts the E-map's concepts. Fully correct propositions are those that correspond exactly to propositions in the E-map. A partly correct proposition is one that combines concepts that are combined in the E-map, but with a different linking phrase or direction. This formula was derived to reflect a scoring strategy (mentioned previously) that gives most weight to correct propositions, with less (but still positive) weight given to isolated valid concepts that are not propositionally linked. No claims are made for the optimality of the formula which will be reviewed in the light of experimental data.

[FIGURE 1 OMITTED]

The User Interface

The main control window through which the user interacts with the RFA is shown in Figure 2, which illustrates the realisation of the RFA within Conception. In this illustration, an S-map prepared by a student named "Zoe" has been selected for comparison to an E-map named "Graphics expert." In practice any regular concept map can operate as an E-map, without requiring special annotation or preprocessing (indeed, the term E-map may even be a misnomer in the sense that its author could in principle be, say, a fellow student rather than an authentic expert). Over time it is envisaged that a teacher will acquire a library of such comparison maps and the effort involved in their development should be rewarded by repeated re-use.

[FIGURE 2 OMITTED]

Without human intervention, the system has computed an initial score of 61%. The main text field summarises the calculation and reminds the student of the system's fallibility. Notice that the "Hints" button is disabled: it becomes accessible only after "Argue," since the provision of hints will make use of information acquired during that interaction.

Explanation of the Score

Clicking the "More" button obtains a detailed breakdown for the score. This includes a categorisation of the student's concepts as either fully correct, partly correct or unrecognised (Figure 3). Further use of the "More" button provides a similar categorisation of the student's propositions.

Arguing the Score

Clicking the "Argue" button initiates a dialogue between the RFA and the student. As shown in (Figure 4), the student is invited first to select from the right-hand menu one of three forms of argument. The first is system-driven and amounts to an exhaustive exploration of perceived deficiencies in the Smap. The other two are user-driven and afford a more focussed check on parts of the assessment that the student may consider to be dubious.

[FIGURE 3 OMITTED]

[FIGURE 4 OMITTED]

Supposing in this case the student takes the initiative by way of "You make claims about concepts," the RFA responds with the dialogue (Figure 5). The student (in this case, it is Zoe again) stakes her claim by selecting a pair of concept names that she considers synonymous, one from each of the menus on the right-hand side. In the example shown, the student's selections represent her claim that her own term "Vector Graphics" is actually equivalent to the expert term "Vector images." On clicking "Ok", the RFA asks the student to confirm the correspondence following which it concedes the point and immediately displays an updated score.

A similar interaction enables the reassessment of propositions from the S-map that have been judged to be less than fully correct (Figure 6). The student is invited to match these to propositions in the E-map that the system has identified as potentially comparable (because they relate similar concepts albeit with a differing link phrase and/or direction).

Generally, the RFA accepts at face value the student's claims about the intended equivalence of concepts and propositions. An exception is where a claim is made for propositional equivalence where the component concepts have not been established (either by system matching or prior argument) as pairwise equivalent. In these cases, the student is advised to think again.

[FIGURE 5 OMITTED]

[FIGURE 6 OMITTED]

Provision of Hints

Following a session of argument, the RFA computes up to three types of hint. These advise the student respectively to add missing concepts, to insert missing links, and to rephrase the text of links in propositions that have been rated as only partly correct. As shown in Figure 7, advice about adding concepts is limited to those that can be attached to concepts that are actually present in the S-map. The expectation here is that these represent extensions to the map that are within the student's reach (or Zone of Proximal Development [ZPD]: Vygotsky 1978). If the student wants it, the RFA can print a "certificate" (partly displayed in Figure 8) that reproduces the offered hints, as well as recording the score and details of meanings that were negotiated during the session of argument.

LABORATORY TESTING

The prototype RFA was laboratory-tested with a sample of concept maps constructed by 13-year old pupils at an Edinburgh school following a unit of study in the domain of graphics technology. The aim of the experiment was to provide a limited test of the claim that the analyser is reasonable, although fallible. The test criterion for success was that the analyser's initial assessments of the S-maps should rank them similarly (though not necessarily identically) to an expert's ranking, with the differences of assessment score narrowing through the processes of argumentative dialogue.

[FIGURE 7 OMITTED]

[FIGURE 8 OMITTED]

To provide a benchmark for the assessments, the S-maps were scored manually by a human domain expert using a scoring rule similar to the RFA's, that is, two points per correct proposition plus one point per valid concept. They were then scored by the RFA. Initial RFA scores were noted along with final (postargument) scores, following an interaction in which the researcher simulated the role of students who argued with the system. The students were assumed to be honest but tenacious in seeking credit for their maps. The results of this experiment are shown in Table 1, where maps are ranked by row in descending order of initial RFA score.

Referring to the previously mentioned test criterion, it can be seen that the data generally satisfy the criterion. The RFA's scores both before and after the arguments correlate positively with those of the human expert ([rho] = 0.91 and [rho] = 0.86, respectively). The system's pre-argument scores underrated S-maps by an average of 25% but the argument process reduced the difference to an average of 16%. Thus the RFA demonstrated its fallibility, but also its reasonableness in adjusting its assessments in the direction of greater accuracy in response to argument.

CLASSROOM TRIALS

Classroom trials with the RFA were undertaken in an Edinburgh high-school in March 2004. The trials were intended to answer some important questions. Did students find the analyser usable? Was it perceived as easy to learn, effective in providing fair and helpful feedback, and enjoyable to use? Of particular concern was students' perceptions of the analyser's fallibility and its implementation of argumentative dialogue. Would they take this function seriously or perhaps, try to "cheat" by claiming as synonyms terms that were not genuinely synonymous? In addition, although it was not attempted at this stage to quantify the effect on learning, the impact of the RFA on the overall concept mapping process was of interest. Crucially, would students be stimulated by interaction with the RFA to revise their concept maps?

During the trials, the RFA was made available to 40 students aged 14-16 years over a period of two weeks. The students were following courses in computer technology at a variety of academic levels. The class teacher, who regularly incorporated concept mapping activities within his teaching, engaged students in the preparation of concept map summaries of course material using the Conception map-building tool. This much was more or less conventional practice. The novel feature was that students were told that on these occasions, it would be possible to use the RFA to get feedback on their maps. The RFA was provided as a separate application (the version integrated within Conception was not at that time available). Instruction on the RFA was minimal: students were issued with a one-page handout that summarised the operation of the user interface. E-maps had been prepared in advance by the class teacher and their file names were displayed on the class whiteboard. No use was made of the RFA's capability to accommodate specialist vocabulary within the User Dictionary.

The classroom trials produced data from three main sources: direct observation by the class teacher and the researcher; log files generated automatically by the RFA that recorded details of user interaction (students were not informed of this feature); and a questionnaire that was completed by students immediately after the lessons. The questionnaire elicited responses to a set of provided statements using a four-point Likert scale: 1 = Disagree strongly, 2 = Disagree, 3 = Agree, 4 = Agree strongly. For summary purposes, it will be convenient to structure the results around categories of questions from the questionnaire, supplementing these with evidence from direct observation and log files where that seems helpful.

Attitudes to Concept Mapping

Clearly, students' attitudes to the RFA need to be interpreted in the context of their attitudes to concept mapping generally. These are summarised by the questionnaire responses shown in Table 2 where the percentages represent the number of students who answered either "Agree" or "Agree strongly" with the statement. As can be seen, students are largely positive towards concept mapping and they favour computer tools for map construction. An analysis by gender revealed that boys more frequently than girls identified concept mapping as hard work--37% of boys compared to 17% of girls agreed with this statement. However, girls were heavily outnumbered in our sample (only 12 females out of 40 students) and neither this nor any other gender difference reported here is significant statistically.

Attitudes to Feedback

As mentioned previously, a hypothesis that motivated the development of the RFA was that students generally are making concept maps for which they get little or no feedback. This hypothesis was tested by the two questionnaire statements shown in Table 3. The lack of feedback is confirmed. Furthermore, it matters: a very large majority of students does care about knowing whether their maps are good or bad.

RFA as a Stimulus to Map Revision

The class teacher encouraged students to use the results of each interaction with the RFA as a stimulus to revise their maps. In practice, students were observed to do this often and it was clear that they were motivated in their map-making by the prospect of submitting their maps to the RFA for analysis. As Table 4 shows, students agreed that the RFA was effective in this role.

Attitudes to the Argument Function

Observation confirmed that students found the RFA easy to use. The "10-minute rule" proposed by Nelson (1980), whereby the ability of a novice to be able to learn to use a new computer system within 10 minutes is taken as the criterion of whether the system is easy to learn, seems to be broadly met by the RFA. However, it was evident that students sometimes skipped over the text that provided detailed explanation of how their maps had been scored and they often showed hesitancy when selecting between different options of the Argument function. Nevertheless, students were quite tenacious in argument and they commonly achieved gains of around 20% over the system's pre-argument scores. Most of these gains were due to the student's identification of mismatches between the textual rendering of concepts in the two maps: in every case the staking of claims by the students seemed genuine, that is, no cheating was detected.

Students' questionnaire responses (Table 5) confirm the evidence of classroom observation that they generally enjoyed arguing with the system, accepted its scoring as fair, and were basically honest in their approach. Perhaps surprisingly, students were quite satisfied even by the RFA's pre-argument scores even though these typically undervalued their maps considerably. The belief on the part of 44% of students that cheating would necessarily be detected by the system is misguided, of course. The RFA is actually very gullible. This is not a point that was stressed in class and it remains to be seen how students' attitudes to the system may change as increased exposure (presumably) reveals this fact.

A couple of gender differences in this area are suggestive. First, girls were more satisfied with the scores their maps were awarded than boys: 92% of girls regarded the pre-argument scores as fair and for postargument scores this rose to 100%. The corresponding figures for boys were 63% and 96%. Second, girls were more likely to believe that cheating would be detected: 58% of girls expressed this view as compared to 37% of boys.

Perceptions of Scores and Hints

As Table 6 shows, students generally expressed approval of the RFA's explanations and hints. For the former, a higher proportion of girls (92%) was satisfied than boys (63%). More than three-quarters of students claimed that they would like to use the RFA with all their future map-making activities. The class teacher agreed that the RFA's hints were largely well-founded and likely to be helpful to his students' learning. The system's ability to generate a Certificate summarising these hints, which students could print and take away, was especially welcomed.

DISCUSSION, FUTURE WORK, AND IMPLICATIONS

In this section three issues are discussed. First, the RFA is related to the wider context of assessment in education. Second, the scope of use of the RFA and its appropriate associated pedagogy are considered. Finally, some directions for future work are proposed and the wider significance of the RFA is identified within a potential new genre of education software.

Formative Assessment

As mentioned previously, a conventional distinction made in the literature is between formative assessment and summative assessment. Essentially the distinction rests on purpose: formative assessment aims to stimulate learning whereas summative assessment aims to report its extent or quality. Both imply some form of measurement of knowledge or performance but whereas summative assessment demands reliable accuracy, in formative assessment the measurement requires only to be "good enough" to encourage productive reflection. The RFA is designed as a tool of formative assessment and although clearly fallible, the evidence of this limited investigation suggests that its accuracy (which is improved through argument) does meet the test of "good enough." An interesting future research project would be investigate whether it might be "good enough" also for diagnostic assessment--that is, whether teachers could gain from the RFA some insights into how teaching should be modified to address learners' weaknesses. Some type of semi-automated analysis of a class set of concept maps should be feasible and hopefully, useful.

The development of a new technology for formative assessment of concept maps is timely. In Scotland and perhaps in other countries too, education policy makers now urge schools to make more use of formative assessment (Simpson, 2003). The reasons are several and include awareness of the limitations of summative assessment, the need to promote self-monitoring, self-motivation, and self-reliance among learners, and a general desire to shift from a teacher-centred, transmissional pedagogy towards a learner-centred, constructivist one.

The effectiveness of concept mapping for learning has been demonstrated many times but its potential contribution to formative assessment seems to have been neglected. Summative assessment, on the other hand, has received a lot of attention from concept map researchers (Rice et al., 1998; Ruiz-Primo, Schultz, Li, & Shavelson, 1998; West, Park, Pomeroy, & Sandoval, 2002). This seems odd, not only because concept mapping is most naturally seen as a developmental tool but also because formative assessment generally is much less beset by the reliability and validity issues which have predictably dominated discussion about the role of concept mapping in summative assessment.

Scope and Pedagogy

The scope of the RFA (by which is meant, the range of contexts in which its use is appropriate) and its associated pedagogy need to be clarified by further research. Since the system depends upon the availability of a relevant E-map it appears likely that more exploratory forms of concept mapping (for example, creative writing plans) are beyond the system's reach. However, concept mapping in school classrooms is perhaps more often about summarising rather than exploring. Certainly in the author's experience, students' concept maps are typically abstractions of routine curriculum subject matter. For this kind of activity, use of the RFA should be feasible.

As with most classroom technology, the learning benefits will very much depend on how its use is mediated by teachers. A pedagogy that incorporates the RFA and generalises the approach used in the classroom trials is illustrated in Figure 9. Here, labels represent processes to be accomplished by the student. Although more investigation is needed, it seems predictable that the processes "Review domain" and "Discuss with teachers/peers" are likely to be crucial to ensuring that the RFA's feedback is productive within a wider cycle of meaningful learning. If these processes are weak, there is a clear danger that students will engage in a shallow edit-submit-edit cycle that uses feedback only at the surface level (i.e., at the level of concept and relation names). This might be exacerbated if students fixate on the RFA's prominently displayed percentage score, perhaps leading to a "gaming" behaviour that might succeed eventually in bringing their map into alignment with the E-map but without much change in their own personal conceptual frame-works. Since the RFA was intended to support the deep learning potential of concept mapping, such a superficial outcome would be ironic. In our classroom observations it appeared that the prominence within the interface of the percentage score was effective in motivating students without distracting them into gaming, but if gaming becomes a problem then it would be useful to experiment with a variant interface that presents feedback differently.

The scaffolding (support) which the teacher supplies initially with the task will also be important. A study by Chang et al. (2001) compared experimentally two groups of high-school students who were concept mapping in the domain of biology. One group was scaffolded initially by the provision of a partly completed map while the other group built maps from scratch. Both groups used the same concept mapping environment (mentioned previously) that provided feedback based on an E-map and which imposed (even for the "scratch" students) a prespecified list of concept and relation names. Tests of learning showed significantly better results for the scaffolded group.

Generally in Scotland, skilled classroom teachers acknowledge that the extent and form of scaffolding is a key factor in learning and that it is often necessary to adjust scaffolding to differentiate between learners at different stages of development. In classroom trials with the RFA, scaffolding was provided by the class teacher by eliciting a partial list of concept names through whole-class brainstorming prior to the commencement of concept mapping. If Chang's study is a guide, it would be useful to consider the option of making available (to some students) a partially completed map file. On the other hand, the incorporation into pedagogy of the RFA might bring open-ended tasks closer to the reach of students. Although their initial maps may be wide of the mark, with feedback from the system students ought to be able to make progress.

[FIGURE 9 OMITTED]

It remains to be seen how attractive this pedagogy will be to teachers. There is some evidence that the uptake of concept mapping by high-school teachers is low (Conlon & Bird, 2004). No doubt several reasons can be found for this but among them is probably the fact that concept mapping presents teachers with the problem of how to arrange that learners get feedback on the quality of their maps. The RFA addresses this problem and adds to concept mapping some additional benefits that even traditionally-minded teachers should appreciate: a clear goal for students who are motivated to submit their maps to the system; an enjoyable and productive use of new technology; and a liberation of the teacher's time that makes room for individualised coaching and other creative teaching activities.

Future Work and Conclusion

More classroom-based work is needed both to develop good pedagogy and to assess the impact on learning. In tandem with that, there is work to be done on the tool itself. At present, just one commercial concept mapping tool, Conception, has integrated the RFA's entire functionality (the screen shots shown in this article are from Conception) and it would be nice to extend this to other products also, especially CmapTools (http://cmap.ihmc.us/). Some other attractive developments have been mentioned already: the replacement of a single E-map with an aggregation of multiple diverse E-maps, to see how far that improves the quality of the RFA's assessments; forms of self-improvement, whereby the system learns from the S-maps that it encounters; and possibly, an extension of its reporting to provide diagnostic assessment. It will be especially interesting to develop the RFA's argument function. Almost certainly, forms of argument can be adopted that are much richer than what is presently provided, including versions which operate more incrementally (i.e., more in the style of a resident coach rather than a visiting critic). The RFA's existing argument strategy does appear to stimulate students into reflection but it is not claimed to be optimal.

In conclusion, the research reported in this article indicates that even in its present form, high-school students' experience of concept mapping is enhanced by the RFA. Students enjoy arguing with the system, accept its scoring as fair, welcome its hints, and are stimulated to revise their maps to accommodate the feedback obtained. Because the RFA enables formative assessment of students' concept maps without imposing any restrictions upon their freedom of expression, and because of its flexibility in adapting to each new domain merely by the addition of a suitable E-map, it is claimed that this technology and its associated pedagogy represent a useful contribution to concept mapping.

The wider significance of the RFA is that it points tentatively towards a new genre of educational software that may emerge if the ORF (Open, Reasonable, Fallible) philosophy is recognised to be worth exploiting more widely. It is striking that, notwithstanding the commitment of many present-day technology developers to student-centred approaches, their programs have traits that resemble the very caricature of the authoritarian teacher--founts of knowledge, uncontaminated by error, incapable of negotiation, unswerving in judgement. By contrast, the ORF approach represents not merely pragmatic recognition that infallibility may be infeasible but also a belief that learners can benefit from a more genuinely dialogic model of human-computer interaction: one that recognises that it is sometimes appropriate to subordinate artificial intelligence to the intelligence of the student.

References

Ausubel, D., Novak, J., & Hanesian, H. (1978). Educational psychology: A cognitive view. New York: Holt, Reinhart and Winston.

Biswas, G., Schwartz, D., & Bransford, J. (2001). Technology support for complex problem solving. In K. Forbus, & P. Feltovich, (Eds.), Smart machines in education. Cambridge, MA: MIT Press.

Chang, K., Sung, Y., & Chen, S. (2001). Learning through computer-based concept mapping with scaffolding aid. Journal of Computer Assisted Learning, 17, 21-33.

Conlon, T., & Bird, D. (2004, September). Not yet within the mainstream: Concept mapping in a Scottish high school. Proceedings of CMC2004. Pamplona, Spain.

Morales, R., Pain, H., Bull, S., & Kay, J. (Eds.) (1999). Proceedings of the workshop on open, interactive and other overt approaches to learner modelling. AIED99: World Conference on Artificial Intelligence in Education, Le Mans, France.

Nelson, T. (1980). Interactive systems and the design of virtuality. Creative Computing, 6(11), 56 ff, (12), 94 ff.

Novak, J., & Gowin, D. (1984). Learning how to learn. New York and Cambridge, UK: Cambridge University Press.

Reese, D. (2003, October). PFNET translation: A tool for concept map quantification. Paper presented at the Annual Conference of the Association for Educational Communications and Technology, Anaheim, CA.

Rice, D., Ryan, J., & Samson, S. (1998). Using concept maps to assess student learning in the science classroom: Must different methods compete? Journal of Research in Science Teaching, 35(10), 1103-1127.

Ruiz-Primo, M., & Shavelson, R. (1996). Problems and issues in the use of concept maps in science assessment. Journal of Research in Science Teaching Vol 33 pp 569-600.

Ruiz-Primo, M., Schultz, S., Li, M. & Shavelson, R. (1998). Comparison of the reliability and validity of scores from two concept mapping techniques. CSE Technical Report 492, Graduate School of Education & Information Studies, University of California, Los Angeles.

Simpson, M. (2003). Diagnostic and formative assessment in the Scottish classroom. In T. Bryce & W. Humes (Eds.), Scottish Education (2nd ed., pp. 721-730). Edinburgh, UK: Edinburgh University Press.

Vygotsky, L. (1978). Mind in society. London: Harvard University Press.

West, D., Park, J., Pomeroy, J., & Sandoval, J. (2002). Concept mapping assessment in medical education: A comparison of two scoring systems. Medical Education, 36, 820-826.

Witzig, S. (2003). Accessing WordNet from prolog. Artificial Intelligence Centre, University of Georgia.

Acknowledgements

Thanks are due to David Bird, Peter Bolton, Stephen Docherty and Simon Love for practical help in experimental trials. This article is an extended version of papers presented to Ed-Media 2004, Lugano, Switzerland, and CMC 2004, Pamplona, Spain, and the author is grateful to referees and participants at both conferences as well as to JILR reviewers for valuable comments and feedback.

TOM CONLON

University of Edinburgh, Edinburgh, Scotland

[e-mail]

Table 1 Scoring a Sample of Maps

          Size of map  Initial RFA
          (symbols X   Score (pre-   Final RFA score    Human expert's
Author    links)       argument) %   (post-argument) %  manual score %

(Expert)  24 X 23      100           100                100
Zoe       33 X 49       58            75                 84
Louis     23 X 25       41            50                 60
Rosie     21 X 22       37            41                 77
Nina      17 X 19       28            35                 56
Sean      21 X 20       27            35                 46
Kim       22 X 23       27            34                 53
Gwen      24 X 23       25            42                 54
Deedee    23 X 26       24            30                 57
Victoria  20 X 19       22            38                 43
Felix     10 X 10       15            24                 31
Sylvia    14 X 13       15            20                 31
Jasmine   22 X 21       12            24                 41
Sarah     14 X 13       11            21                 39
Tabatha   11 X 10       11            15                 31

Table 2 Students' Attitudes to Concept Mapping

Statement                                                     Agreement

Concept mapping helps me to learn.                            95%
Concept mapping is hard work.                                 31%
If I had the choice, I wouldn't make concept maps again in    23%
  future.
I prefer paper-and-pencil to a computer for concept mapping.  15%

Table 3 Students' Attitudes to Feedback

Statement                                                    Agreement

Until now, when I have constructed a concept map I haven't   69%
  known whether it's good or bad.
It's important to me to know whether my concept map is good  92%
  or bad.

Table 4 The RFA as Stimulus to Map Revision

Statement                                                    Agreement

Using the Analyser made me want to redo parts on my concept  87%
  map.
I would put more effort into my concept maps in future if I  90%
  knew they were going to be scored by the Analyser.
Using the Analyser made me realise that I need to learn      79%
  more about the subject matter of my map.

Table 5 Attitudes to the Argument Function

Statement                                                     Agreement

It was fun to use the "Argue" button of the Analyser.         85%
The score that was given to my map(s) by the Analyser even    72%
  before I used the "Argue" button was fair.
The score that was given to my map(s) after I used the        97%
  "Argue" button was fair.
It was frustrating to argue with the Analyser.                23%
I was keen to try to raise my map's score by arguing with     92%
  the Analyser.
When arguing with the Analyser, I was tempted to cheat by     28%
  claiming that different words meant the same thing.
If I tried to cheat when arguing with the Analyser, it would  44%
  be certain to find me out.
When arguing with the Analyser, I always did my best to       90%
  answer its questions as honestly as possible.

Table 6 The RFA as Stimulus to Map Revision

Statement                                                     Agreement

The Analyser is good at explaining how it calculates its map  72%
  scores.
The hints provided by the Analyser were helpful.              85%
If I had the choice, I would use the Analyser every time I    77%
  made a map.