CalMaeth: An Interactive Learning System Focussing on the Diagnosis of Mathematical Misconceptions.

This article describes the operation of CalMaeth, a Web-based tutorial system, which is being used in teaching mathematics. The unique feature of CalMaeth is that it can provide detailed computer-generated diagnostic feedback for answers consisting of mathematical expressions. We give some examples of the system's diagnostics in Calculus, described how they were generated and assess the range of student misconceptions that it is able to detect. Finally, data is presented illustrating some of the effects the diagnostics had on learning outcomes.

This article describes and discusses a Web-based tutorial system, called CalMaeth, that is being used in teaching mathematics at the University of Western Australia. CalMaeth has been used with intermediate calculus, statistics, and linear algebra, but in this article only calculus examples are discussed.

The CalMaeth system performs a variety of functions. It poses students individual randomised questions, which may require mathematical expressions as answers. Questions are labelled either "practice" or "assessed": the assessed questions are marked in detail, whereas for practice questions, the students receive detailed computer-generated diagnostics for their incorrect answers. CalMaeth also maintains records of students' answers, marks, and activity.

In this article we describe some of these features in more detail, but the main focus was on the computer-generated diagnostics, since these are the most original and sophisticated features of CalMaeth. We also made some observations about the advantages and disadvantages of the system from the point of view of both staff and students.

The principle advantage of the system is that it can guarantee higher levels of student competency thereby allowing more effective use of staffs time. By controlling question access, CalMaeth "forces" students to complete all assigned work correctly; The immediate feedback enables students to correct their own misconceptions, often without staff help. Ca1Maeth also frees staff from the burden of marking assessed work and so allows more personal interaction with students. We complete this section with a brief overview of CalMaeth and its use, then describe in some detail the kind of diagnostics it produces for Calculus questions, finally, we discuss some advantages and disadvantages of using CalMaeth.

Other Systems

One-on-one tutoring in Mathematics has been shown to produce marked improvements in student understanding with, in some cases, the average student under such tuition, scoring higher than 98% of students under control teaching conditions (Bloom, 1984). Simple economics however, renders one-on-one tutoring unfeasible and since the late 1950s researchers have attempted to substitute such tuition with computer-aided instruction. There have been several systems constructed for particular domains that have demonstrated improved educational outcomes, although none have been integrated uniformly into the mathematics curriculum. In Calculus, the related rates tutor (Singley, 1990), showed the importance of providing a "Strategy window" in which a problem's subgoals are posted; Similarly, the geometry tutor (Anderson, Boyle, & Yost, 1985) allowed students to do geometric proofs using "proof graphs" which expressed the logical relationships between premises and conclusions; In the domain of symbolic algebra, there is a system that transmits problem solving heuristics as it converses with students (Kimball, 1973) and a quadratic tutor that alters its teaching strategy as it measures student performance (O'Shea, 1982). In a similar vein, an adaptive remediation system Mathemagic (Parvate, Anjaneyulu, & Rajan, 1998) selects questions based on missing "competency factors" which it deduces from a prior diagnostic test. In basic algebra, The Leeds Modelling System (Sleeman, 1984) uses unexpected mal-rules employed by students, to form a cognitive model of their acquisition of algebraic skills. Also in algebra, the learning environment Aplusix (Nguyen-Xuan, Nicaud, & Gelis, 1997) provides feedback for various misconceptions by using pattern-matching on students' answers.

While these systems each have their own particular strengths, none focus on providing specific feedback for a complete list of misconceptions that may arise from the posed questions. A few do provide such diagnostics, however it is unclear, or at least not quantified whether the set of system responses is able to address the full range of student misconceptions. The system IDEBuggy (Burton, 1980), on the other hand, showed that given extensive analysis of students' responses (in the domain or arithmetic), it is possible to identify a high proportion of errors and hence feasible to build an interactive system, able to diagnose and treat a wide range of misconceptions.

CalMaeth is a system based on this idea but in the domain of calculus and with an Internet browser forming the interface; that is, by allowing the entry of algebraic expressions within this browser, a subsequent analysis of a large number of these entries is able to reveal an encompassing collection of misconceptions. Using Mathematica's pattern-matching facility, together with preprepared diagnostics, CalMaeth uses such a collection to provide a tutoring environment, which, for many essential Calculus skills, is able to detect and treat a very high proportion of the misconceptions that students exhibit.

Overview of Using CalMaeth

In the Department of Mathematics and Statistics at the University of Western Australia, CalMaeth has been used in some units to replace all, or part of the traditional tutorial and assignment system. When using CalMaeth, students access assigned work from a standard web-browser, either in the department's computer labs, other labs in the university, from home or colleges, or any work-station with an Internet connection. Each week students are allocated a one hour session in the department's computer lab. These sessions have two to three helpers available to answer questions from about 30-110 students doing the same unit. A typical unit has 300-350 students. Typically, students are assigned five practice and two assessed questions each week. Sometimes the students are forced to complete all the practice questions before access to assessed questions is permitted. All students have to complete similar questions, but the questions are randomly generated for each student. For extra practice, a student can also as k CalMaeth to produce any number of random variants of a question.

Questions sometimes require typing in numbers, making multiple-choice selections or (as is most appropriate in much of Calculus) the entering of algebraic expressions. Up until recently, the answers were typed in using a syntax similar to that used in most computer languages and symbolic algebra software. In 1999 however, an equation editor has been added to simplify this process.

If an answer to a practice question is incorrect, CalMaeth gives the student a detailed diagnostic designed to identify their misconception. CalMaeth also gives every error a severity rating, ranging from 5% for a trivial error up to 50% or more for a serious misconception. The student begins each assessed question with a number of available marks; if an answer to an assessed question is incorrect, CalMaeth then reduces the available marks in proportion to the severity rating of the error. The student's final mark is the marks available when the correct answer is finally obtained. In assessed questions with multiple parts, a student can also give up and retain marks for parts of the question already correctly answered.

Each weekly batch of assessed questions has a deadline, which needs to be met for the marks to contribute towards a student's final assessment. For teachers to maintain ultimate control over the assessment, CalMaeth provides staff with administrative features for altering either class or individual student deadlines, or for modifying the marks awarded for each question.

CalMaeth also keeps detailed time-stamped records of almost everything that students do while using the system. These records are most useful for reviewing the performance of students and the system and in particular, for improving both the teaching of the unit and the computer generated diagnostics that it produces.

GENERATING THE DIAGNOSTICS

In this section a number of examples of the kinds of questions CalMaeth presents together with some of the diagnostics it produces are given. The most sophisticated and detailed diagnostics generated by CalMaeth are in Calculus and in particular, for those questions involving the various derivative rules, techniques of integration, and solutions to differential equations. Many researchers (Anderson, 1990) have shown the pedagogical importance of students generating their own responses as opposed to merely selecting an alternative: In Calculus, such generation invariably involves the use of algebraic expressions so that numerical or multiple choice answers would have very limited usefulness. Furthermore, it would almost be impossible to adequately determine a student's proficiency in Calculus without at some stage analysing some of their generated expressions. Finally, a mathematical expression is often a rich source of information about a student's misconceptions, and CalMaeth attempts to extract as much inf ormation as possible.

In the following, it is important to keep in mind that CalMaeth's diagnostics are all computer-generated and largely question independent. They are generated by topic-specific diagnostic packages, rather than being tailored to individual questions. This is necessary due to the embedded and recursive nature of mathematics, but it also means that the students can be assigned individual randomised questions and that staff are able to construct new questions without the need to program new diagnostics.

All the examples given in this article are actual student questions and responses. Web browsers have a fairly limited capacity for formatting text, and we currently use plain ASCII text formatting to maintain browser compatibility. We reproduce the actual diagnostic text in the following examples.

A Detailed Diagnostic

The following question, answer and diagnostic, gives an idea of the sophistication of the diagnostic package associated with differentiation. Observe how the diagnostics isolates a specific error (related to applying the Power rule to a negative exponent) which is embedded in the answer to a question on the Product rule. This diagnostic is one of the longest CalMaeth can produce.

CalMaeth's question:

Differentiate the following function with respect to x

f(x) = 1 + 9 x /[(8 - [x.sup.2]).sup.8/3]

Student's Answer:

16 x (1 + 9 x)/[3 (8 - [x.sup.2]).sup.5/3] + 9/[(8 - [x.sup.2]).sup.8/3]

CalMaeth's computer-generated diagnostic:

Noting that your answer had two terms added together, I'll assume that you used the Product Rule rather than the Quotient Rule.

The Product Rule says that (fg)' = f'g + fg'. You got the f'g part correct in your term

9/([8 - [x.sup.2]).sup.8/3]

The fg' part or 16 x (1 + 9 x)/3 [(8 - [x.sup.2]).sup.5/3]

was incorrect although I noted that you included the f or 1 + 9 x part as one of the terms in the product. So the mistake was in the g' part i.e. in saying that the derivative of

[(8 - [x.sup.2]).sup.-(8/3)] was 16 x/3[(8 - [x.sup.2]).sup.5/3]

Here we are using the Power rule which says

d/dx [(f(x)).sup.n] = n [(f(x)).sup.n-1] f' (x)

AH HA! What you did was a common and understandable error. This is what I think you did; in the question we had the term

1/[(8 - [x.sup.2]).sup.8/3] so the rule is; when taking the derivative the power goes down by one, OK, so we should have got

1/[(8 - [x.sup.2]).sup.8/3 - 1] = 1/(8 - [x.sup.2]).sup.5/3]

right?... wrong ! The key is in identifying WHAT WAS the original power.

The fractional power - [[(-).sup.8].sub.3] of [(8 - [x.sup.2]).sup.-(8/3)]

from the question) needed to decrease by one. At first, handling these types of powers can be tricky, lets fix

this up now; When you have 1/[(8 - [x.sup.2]).sup.8/3]

(from the question) always think of this as [(8 - [x.sup.2]).sup.-(8/3)] so that just as before, when you take the derivative, the power

goes down by one; i.e. [(8 - [x.sup.2]).sup.-(8/3)] becomes

[(8 - [x.sup.2]).sup.-(8/3) - 1] = (8 - [x.sup.2]).sup.-(8/3) - 3/3] = [(8 - [x.sup.2]).sup.-(11/3)]

You can then leave your answer as [(8 - [x.sup.2]).sup.-(11/3)] unless you prefer using positive indices(or, if the question

requires this) you can write it as 1/[(8 - [x.sup.2]).sup.11/3]. Remedy this

now in your solution (not forgetting the other terms of course) and then re-submit and I'll check out if the rest of your answer is OK.

This diagnostic took about 0.3 second to produce and is detailed and accurate, but perhaps a little long and awkward. A human tutor, given a few moments, would spot the error in this case, and could provide an explanation, with pen and paper and pointing, that would probably be superior, but as mentioned previously, providing constantly available personalised tuition is impossible. Statistics of student performance on the other hand, (see Column C3 of Table 1) reveal that diagnostics like this one are in fact sufficiently detailed to enable most students to recognise and correct such errors. This effectiveness of the diagnostics is discussed in more detail in the section on Diagnostic Effectiveness.

Versatility

CalMaeth's diagnostics are versatile in the sense that errors are detected by comparing structural patterns in the question and answer, rather than through considering values specific to individual questions. As a result, mistakes can be detected irrespective of the actual functions or constants a question uses. Take the following simple illustration: if a student has the same misconception as in the previous example but whose randomly generated question contains the different denominator [(3 - [2x.sup.7]).sup.-1/6]; then in this case, CalMaeth produces the same diagnostic but substitutes the algebraic expressions relevant to this particular question. For example, one change is that the part of the diagnostic saying

The fractional power - (8/3) of [(8 - [x.sup.2]).sup.-(8/3)]

(from the question) needed to decrease by one.

would now read

The fractional power - (1/6) of [(3 - 2[x.sup.7]).sup.-(1/6)]

(from the question) needed to decrease by one.

Modularity

CalMaeth's diagnostic packages are modular, meaning that diagnostics in one package topic can be re-used as components in other diagnostic packages. For example, consider the following question taken from the Optimisation section of the Calculus course:

A ship uses $(5[v.sup.2]) of fuel per hour when travelling at a constant speed of v km/h. Other expenses in operating the ship (i.e. crew and equipment) amount to $128 per hour.

The ship makes a journey of 200km. To minimize the total cost of the journey the ship should travel at what speed?

Typically, the diagnostics used in this question would come from the diagnostic package associated with Optimisation. This problem's solution however, also involves taking derivatives (the first and second derivatives of the Cost function) so that by "plugging" the diagnostics from the Differentiation package into the Optimisation package, the chance of diagnosing a student's error becomes greatly enhanced.

This effect can be seen in the following diagnostic taken from the Calculus course. In the course of answering this question, a student correctly gave the first derivative of the Cost function as 200 (5 - 128/v.sup.2]) but in submitting 51200/v as the second derivative, displayed the same Power rule mistake as shown previously on page 4. In this new context, the diagnostic the student now received is given as follows:

You have not calculated the second derivative of the total cost correctly.

One good way of approaching this type of derivative is to multiply out the brackets and consider the question as a sum of

terms. Hence lets consider 200 (5 - 128/[v.sup.2]) as being of the form

1000 - 25600/[v.sup.2]. My comments will now assume it was written in this form.

The derivative of the constant 1000 is 0 so you have said that

d/dv -25600 / [v.sup.2] = 51200 / v.

Here we are using the Power rule which says

d/dv [v.sup.n] = n [v.sup.n-1]

AE Ha! What you did was a common and understandable error. This is what I think you did; in the question we had the term

1/[v.sup.2]

so the rule is...

and the diagnostic continues to "treat" this misconception in a similar way

to that shown on page 4 but with [v.sub.2] replacing the term [(8-[x.sup.2).sup.8/3].

Given the hierarchical nature of mathematics together with the fact that only rarely do skills become perfectly instilled on a first pass, the ability to detect the same error in a variety of contexts significantly increases CalMaeth's diagnostic power. In the Calculus unit that uses CalMaeth, for example, Differentiation and Integration are the basic tools of the subsequent topics: Integration by Substitution; Integration by Parts; Related Rates; Curve Sketching; Optimisation; Areas; Volumes and Differential Equations. The modularity of the diagnostic packages means that an error in one of these later topics, but due essentially to an incorrect differentiation or integration, can be detected.

Severity Rating Marking Scheme 

Calmaeth requires that students work on questions until they obtain a correct answer; because of this a "severity rating" marking scheme is used. This scheme takes into account the number and "seriousness" of the errors made in the sequence of attempts leading to the final (correct) answer.

The following diagnostic shows a serious misconception regarding the form of the Product rule,

Severity rating: 50%

Your answer of -2 (-11 + 12 t) for the derivative of

(-8 - 2 t) (-7 - 11 t + 6 [t.sup.2]) was not correct.

The Product Rule says that (fg)' = f'g + fg', NOT (fg)' = f'g'

which is what you have implied by identifying f with -8 - 2 t

and g with -7 - 11 t + 6 [t.sup.2]. Using this identification see if you can use the right form of the Product Rule in finding the correct answer.

Given the severity of this misunderstanding this diagnostic contains the highest severity rating of 50%. Hence, with this question worth 2 marks and this being the student's first attempt, the available marks would now stand at 1. Any further errors would reduce the available marks "geometrically."

The following diagnostic shows an example of a trivial error,

Severity rating: 10 %

Your answer of (-8 - 2 t) (-11 + 12 t) - 2 (-7 - 11 t + 6 [t.sup.2])

for the derivative of (-8 - 2 t) (-7 - 11 t + 6 [t.sup.2])

The Product Rule says that (fg)' = f'g + fg'. You got the fg'part correct in your term (-8 - 2 t) (-11 + 12 t)

The f'g or -2 (-7 - 11 t 6 + [t.sup.2])

part was incorrect although at least you got the f' part since -2 is the correct derivative of -8 - 2 t and this is contained in this term. Therefore your -7 - 11 t - 6 [t.sup.2] term corresponds to the g part, however the g part from the question was the expression -7 - 11 t + 6 [t.sup.2].

In fact, the sign in front of the 6 [t.sup.2] seems to have mysteriously changed.

In this case the error is a minor transcribing one and consequently, the student's available marks now stand at 90 % of the available marks that were present before the error was made.

Completeness

The CalMaeth diagnostic packages can detect a large range of possible errors. These errors have been identified initially through using a teacher's experience and subsequently through collecting and analysing students' interactions with CalMaeth. The current diagnostics recognise all the common errors, many less common ones, together with some unusual ones that perhaps could only come to light through a detailed analysis of many student records. In preparation for 1998, such an analysis was undertaken on the 1997 records. The following provides a brief synopsis of the types of errors observed.

Breakdown of errors. One cognitive theory often used as a theoretical base for machine-based tuition makes a strong distinction between "Declarative" and "Procedural" knowledge (Anderson, 1987). Representation of the latter (within whose ambit Differentiation falls) is made on the basis of a set of "production rules" which define the skill or procedure being described. Essentially, these productions can be considered to be the individual tasks that a learner needs to combine in performing the skill. The most prevalent types of errors from the first two assignments involved the omission of one or more of such productions. For example, in PQ5 of Assignment 2 (Table 2) about 35% of the errors involved in differentiating 9+2x/[(-6-2[x.sup.3]).sup.4] can be attributed to such omissions which we now catalogue: in applying the Quotient Rule ((fg)' = f'g - fg' / [g.sup.2]), 5% of students omitted the [g.sup.2] part, 3% forgot to square it, 2% omitted either the g or f terms in the numerator, 3% overlooked a negative sign at some point; in differentiating [(-6 - 2[x.sup.3]).sup.4] (to form the g' part), 4% left off any exponent on the (-6 -2[x.sup.3)] term, 1% forgot to decrease this exponent by one leaving it as 4, 2% forgot to differentiate the 6 - 2[x.sup.3] expression as required by the Chain rule; of those that did remember, 4% incorporated the -2 and 3 into the final coefficient but then forgot to include the [x.sup.2] term, 4% didn't omit the variable altogether but instead used either [x.sup.3] or just x, 1 % did a similar thing with the variable but also lost a negative sign in forming the numerical part of the coefficient (see the example in section on "Compound Errors"), 1% forgot to multiply in the original power 4 of [(-6 - 2[x.sup.3).sup.4], whereas a similar number of students failed to supply any coefficient whatsoever. In 3% of the responses, it was difficult to work out what omissions took place in the formation of their coefficients.

The production rule(s) omitted in producing the errors just described are more than likely due to "cognitive overload" rather than a complete ignorance of their necessity. This is because considered in isolation, each of the omitted productions had been successively performed in previous questions. These errors can therefore, be readily corrected, although it is still important that they are detected now and conveyed to the students as a failure to combine all the necessary production rules rather than as a fundamental misunderstanding about the Quotient Rule.

Other misconceptions relate to students using an incorrect version of a Differentiation rule. In PQ5, for example, 1% of students transposed the fg' and f'g terms in the numerator while a similar percentage expressed the numerator as f'g + g'f.

There was a surprisingly high number of "transcription" errors. By these it is meant those cases where parts of the response are transcribed incorrectly from the question. Curiously, these errors consistently focussed on the same parts of expressions. For example in PQ5 when copying the (-6 - 2[x.sup.3]) term either as part of the g term or in forming the g' term, 6% of students changed the -2[x.sup.3] to 2[x.sup.3], 2% left off the exponent 3 (see the second example in the section entitled Severity Rating Marking Scheme) and a further 8% made some other transcription error with this term. Also, although the variable x was used in this question, 1% of students decided to use the variable z.

Somewhat disturbingly, there were some errors generated from obvious gaps in basic algebra and arithmetic. In those questions involving fractional powers some students showed interesting algorithms in subtracting 1 from a fraction. In PQ5, 3% of responses "squared" [(-6 - 2[x.sup.3]).sup.4] by changing the 4 to 6 whereas 1% attempted the same by changing the 4 to 16.

There were some misconceptions that persisted much longer than others. For example, the previously mentioned error involving a negative power constituted 30 % of the responses--the most common mistake by a considerable margin. This percentage declined steadily down to about 5% by PQ5 of Assignment 1, although, this same mistake would occasionally resurface with, for example, just under 1% of students committing this error again in AQ2 of Assignment 2. In many ways this emphasises the significance of the diagnostic modularity in being able to repeatedly detect the same misconception irrespective of the context in which it is displayed.

Improving the diagnostic scope. The overall improvement in CalM""th's diagnostic power resulting from the analysis of the 97 records can be seen in Tables 1 and 2: the number of different diagnostics produced by CalM""th in 1997 is shown in column C4. Having added a new diagnostic for each new misconception (identified in the analysis of the 1997 records), a simulation was performed in which all the 1997 responses were re-entered into CalM""th. The number of different diagnostics produced in this simulation is given in the unbracketed numbers of column C5. By comparing these two columns, it can be seen that, for most of the questions, the analysis of prior student records has more than doubled the number of misconceptions that CalM""th is able to detect.

Of particular interest in Tables 1 and 2 is column C5--the total number of different errors observed in the differentiation of each of the expressions. Given the sample size (310 students and over 1000 responses for each question), this catalogue of errors can be considered close to a "complete" list of misconceptions for this question-type. [1] Error-catalogues of similar sizes have been observed in arithmetic and algebraic manipulation; that is, using tutoring experiences, tests and gathered protocols, a list of about 30 errors has been found in basic algebra (Matz, 1980) whereas a skill lattice consisting of 60 mistakes relating to the subtraction procedure has been found in arithmetic (Burton, 1980). The moderate size of these error-catalogues point to the feasibility of building mathematical tutors (at least for basic procedural skills) capable of treating any misconception that may arise in the student-tutor interaction.

Default Diagnostics

Occasionally students produce answers from which it is difficult or impossible to explain in terms of an inherent misconception. This may be due to a complete lack of understanding of the subject material, misreading the question, "playing" with the system, or other unaccountable reasons. In the event of this occurrence, CalMaeth produces a "default" diagnostic containing some helpful hint on how to approach the problem. Analysis of the student records shows that such default diagnostics are sparingly employed constituting less than 5% of all the diagnostics produced. An example of such a diagnostic is given below.

Your answer [x.sup.2] for the derivative of 1 + 9 x / [(8 - [x.sup.2]).sup.8/3] was not correct.

The quotient rule says that (f / g)' = f'g - fg' / [g.sup.2], however your answer of

[x.sup.2] was not at all reminiscent of this form. If the above statement reminds you of this rule have another crack at it; otherwise I suggest some revision on this rule before proceeding.

Diagnostic Search

Tutorial systems whose feedback is based on error-catalogues face the task of ensuring that a diagnosis can be made quickly enough to satisfy impatient students. In CalMaeth, the average time to locate a misconception for each of the questions in the first two assignments is given by the bracketed numbers of columns C4 and C5 (Tables 1 and 2). The magnitudes of these times (a few tenths of a second) indicate that this task was not an issue and in fact there is considerable scope for the use of even more extensive error-catalogues. In our labs CalMaeth's response time was largely influenced by the amount of network traffic. In a typical lab of 100 students such traffic can push the response tune out to around 3 seconds.

The actual search procedure is similar to that of DEBUGGY's (Burton, 1980) in which a "diagnostic tree" is traversed until a leaf representing the unique misconception is located. The path through the tree is determined by applying various Mathematica-defined pattern matching functions to sub-expressions within the student's answer.

Diagnostic Effectiveness

It is difficult to judge just how effective CalMaeth's diagnostics are in enabling students to resolve their misconceptions. [2] Whether it is due to the diagnostics or not, the statistics in Tables 1 and 2 indicate that all the students are resolving their misconceptions after a moderate number of attempts: The bracketed numbers of column C3 show that those students who did not obtain the correct answer on a first attempt, required on average, between 2.3 and 5.0 attempts to find the correct derivative.

The table also indicates that some learning has taken place between the practice questions and the assessed ones with the halving of the percentage of students needing more than one attempt to obtain the correct answer. This can be observed by comparing the unbracketed numbers of column C3 for the PQ5-AQl pair in Table 1 and the PQ2-AQ1, PQ5-AQ2 pairs in Table 2. Furthermore, students required, on average, between 1 and 2.4 fewer attempts to find the correct answer in the assessed questions: This can be seen by comparing the bracketed terms for these pairs in column C3 of Tables 1 and 2.

Between Assignment 1 and Assignment 2 there was also an improvement in the efficiency with which students resolved their misconceptions. Assignment 2 was more difficult in that students needed to combine the Product and Quotient rules with the Power, Sum and Chain rules that were used in Assignment 1. This is reflected in the average number of misconceptions shown per question -- 27 in Assignment 1 compared with 35 in Assignment 2. Despite these extra misconceptions, students in Assignment 2 actually required fewer attempts to ultimately reach the correct answer -- 3.35 in Assignment 2 compared with 3.76 in Assignment 1 [3]. This may be due to students learning how to more effectively interpret the diagnostics in pinpointing their misconceptions, or perhaps, simply, that a general inculcation into learning new differentiation rules is outweighing the increase in question difficulty.

Despite these positive outcomes, it is likely that the form of the diagnostics can be improved. Like the MACSYMA advisor (Genesereth, 1982), CalMaeth currently gives the student as much detail as possible in the diagnostic. While a student may often need this detail, the level of exposition may also be encouraging lazy experimentation rather than considered thought. Indeed, (Corbett & Anderson, 1990) report on the advantages of a flag tutor in which more self-correction is emphasised. In this tutor, (a version of their Lisp Tutor) incorrect parts of a student's answer are put in boldface with students then given the option of receiving a more detailed diagnostic if necessary. Such a scheme may be worth implementing in CalMaeth in the future.

For the most part however, we believe that obtaining a correct answer implies correcting misconceptions. In a significant improvement from previous years, a mid-semester written test showed that over 90% of students were able to correctly differentiate a complex expression requiring the use of the Quotient and Chain rules.

Compound errors. Sometimes a student's answer may be the result of a combination of errors. In cases like these, the problem arises as to how a single diagnostic should treat these multiple misconceptions. Further, when different error combinations give rise to the same answer, the problem becomes one of disambiguation. The arithmetic tutor IDEBUGGY (Burton, 1980) addresses this disambiguation problem through the selection of extra questions that allow each misconception to be isolated. Interestingly in Differentiation and Integration, this problem did not arise; That is, each response either displayed a single misconception (the usual case) or if it did not, the error combination was discernible. In the latter case, the diagnostic addressed the "most serious" misconception. Once students remedied this, subsequent diagnostics then dealt with any outstanding error. An example of this policy is shown in the following diagnostic.

CalMaeth's question:

What is the derivative of the following function f(x)?

f(x) = 4 + 11 x/[(-5 - 3 [x.sup.4]).sup.8]

Student's Answer:

-96 x (4 + 11 x) [(-5 - 3 [x.sup.4]).sup.7] + 11 [(-5 - 3 [x.sup.4]).sup.8]/[(-5 - 3[x.sup.4]).sup.16]

Diagnostic:

Incorrect.

The Quotient rule says that (f/g)' = f'g - fg'/[g.sup.2].

The [g.sup.2] part was correct.

Your 11 [(-5 - 3 [x.sup.4]).sup.8] term meant that the f'g part of f'g - f'g was also correct.

The fg' part or 96 x (4 + 11 x) [(-5 - 3 [x.sup.4]).sup.7]

was incorrect although I note that you included the f or 4 + 11 x part as one of the factors in the product. So the mistake was in the g' part i.e. in saying that the derivative of

[(-5 -3 [x.sup.4]).sup.8] was 96 x [(-5 - 3 [x.sup.4]).sup.7].

Now I see that the [(-5 - 3 [x.sup.4]).sup.7]

Part of this derivative was correct, the only problem was in the other part namely, the factor you had multiplying this. The factor, which you had as 96 x, is not correct.

There were a couple of errors here. Firstly the constant of 96 Had the wrong sign. Check the signs of the numbers 1, 8 and -12

That multiplied to give this constant. Correct this now, Re-submit and I'll point out the error with the factor involving x.

ADVANTAGES

Educational Efficiency

A compelling reason for using CalMaeth is that it allows teachers and students to interact in more productive ways. By freeing teachers from the burden of marking and tutoring on certain basic or routine techniques, students have more time to engage with them at a "higher level."

Greater freedom and involvement means that those weaker students who need extra assistance in grasping routine concepts can now receive more individual attention. On the other hand, teachers now have the opportunity to become involved in more challenging topics for those stronger students who quickly master the essential skills.

Guaranteeing Competency

In terms of ensuring competency there is a profound difference in completing an assignment in CalMaeth compared to completing one in the traditional setting. In traditional assessment, an assignment, replete with all the student's misconceptions, is submitted and then returned along with the teacher's comments. Certainly, sufficiently elaborate comments can resolve many of these misconceptions, however, until the student demonstrates competency in the skills being tested, it is unclear to what extent their misconceptions have in fact been resolved. Contrast this with an assignment in CalMaeth, where, since every answer is immediately checked, students keep refining their responses until the correct answer is achieved. In this process, the student's misconceptions are sequentially resolved so that a completed assignment now becomes one in which the student demonstrates competence in all the skills being tested.

Marking Scheme

The authors believe that the severity marking scheme is a better indication of a student's understanding of material than the mark given for a traditional written assignment. A score of 50% is often set as the pass mark with passing invariably being equated with competence. With such a score from CalMaeth, a student has probably displayed a number of careless errors or a few serious misconceptions in the course of reaching the correct answer and demonstrating competence. Conversely, in a traditional score of 50%, a student may have limited competence in many of the skills tested, or perhaps even worse, a complete lack of competence in 50% of the skills being measured.

Plagiarism 

Any claim to competency depends on students honestly completing their CalMaeth assignments. In our labs, students often work in groups of two or three and are in fact not discouraged from doing so. Also being Web-based, those students with Internet connections can work from home. While it is impossible to eliminate cheating in these situations, the same opportunities also exist for traditional written assignments. One ameliorating factor in CalMaeth however, is that students each have their own set of randomly generated questions. The authors believe that this, along with a high number of assignment questions, is a sufficient deterrent to plagiarism. Indeed, as we discuss in the next section, students overwhelmingly report that the most positive feature of CalMaeth is that the deadlines help them keep up with their work.

STUDENT REACTION

The reaction of students to CalMaeth has been most encouraging. Surveys indicate that over 78% agree or strongly agree with the statement that the system was good for learning; In addition, although the use of CalMaeth was a required part of their unit (contributing to 20% of their overall assessment) over 70% of students reported that they would have still used the system were it an optional learning aid. The most frequent positive response among students' written comments was that it "forced" them to work consistently throughout the semester. This was also reflected in the surveyed questions with over 85% agreeing with the assertion that the deadlines helped them keep up with the work. Given that the skill of pacing oneself throughout a course is strongly related to educational outcomes (Scott & Stone, 1995) and that the comments indicate the students' awareness of this, it is interesting that a computer-generated deadline was able to provide the required level of motivation for this pacing to be achieved.

SOME DISADVANTAGES

The greatest danger of a system like CalMaeth probably lies in how it is used. Using it as a substitute for teaching rather than as an aid to teaching, would leave students lacking in more complex aptitudes. In particular, CalMaeth is more suited to diagnosing misconceptions in routine skills rather than developing a student's ability to form mathematical arguments. Currently, questions requiring extended logical arguments are presented to students along with the constituent steps that make up its solution. It is these steps that the students are required to complete. Given the desirability of having students generate these own steps, work is currently underway to devise means by which students can submit examples of their own reasoning. Once completed, it will be interesting to see to what extent CalMaeth's diagnostic power translates into this richer environment.

As mentioned previously, one drawback is CalMaeth is that some students will focus too much on producing the final correct answers, rather than understanding methods. This can be addressed, to some degree, by providing terser diagnostics (with the option of receiving more detail) that encourage more self-correction. Perhaps more significantly, however, is that once students have the capacity to express their own argumentation, this emphasis on the final response will be reduced.

One not particularly "user-friendly" feature of the present software is the syntax required to enter algebraic expressions. This can result in frequent typing mistakes for complex expressions; say for example, in applications of the Quotient Rule. An equation editor in the form of a java applet has been developed and was introduced in 1999 to assist students in entering algebraic expressions.

Software "bugs" are also a frustration in a complex system like CalMaeth. Even extensive testing by the developers can sometimes fail to uncover certain bugs. With the number of randomly generated questions and students answering them, in the thousands, it is inevitable that occasionally CalMaeth produces a flawed question or an inappropriate diagnostic. With increasing use however, many of these bugs are coming to light and the system is becoming more robust from year to year.

CONCLUSION

In many ways, noting previous errors encapsulates the essence of CalMaeth as a learning system. Through studying past mistakes, current students are having their own misconceptions resolved via prerecorded, carefully considered diagnostics. This "virtual tuition," rather than replacing the traditional student-teacher interaction, is replacing certain, mechanical aspects of it. For the student, valuable time interacting with the teacher can, instead of being used to resolve mechanical errors, now be used to tackle more "higher-order" conceptual subtleties. For the teacher, routine and repetitive diagnostics can be replaced with challenging and interesting dialectics. In short, the teaching process can be made being made more rewarding for both parties by emphasising its human qualities through consigning, to a machine, its inhuman ones.

Notes

(1.) Other misconceptions (particularly from students with varying educational backgrounds) may exist, however, once they have been observed, they can easily be added to the system.

(2.) The reader is invited to view various diagnostics by exploring the range of errors that CalMaeth is able to detect. By visiting the Web page http:// CalMaeth.maths.uwa.edu.au and logging in as a guest, the reader can submit various "errors" to a selected question and see if and how well these are detected and explained. Alternatively there is a selection of past Student-CalMaeth interactions posted at: http://CalMaeth.maths.uwa.edu.auExamples/rndex.htrfll

(3.) These figures excluded the two outliers, PQ1 and PQ4 in Assignments 1 and 2 respectively.

References

Anderson, J.R. (1987). Production systems, learning, and tutoring. In Production system models of learning and development, pp. 437-458. Cambridge, MA:MIT Press.

Anderson, J.R. (1990). Cognitive Psychology and its implications (3rd edition). New York: W.H. Freeman.

Anderson, J.R., Boyle, C.F., & Yost, G. (1985). The geometry tutor. In Proceedings IJCAI-85, pp. 1-7, Los Angeles, CA:IJCAI.

Bloom, B.S. (1984). The 2 sigma problem: The search for methods of group instructions as effective as one-to-one tutoring. Educational Researcher, 13(6):4-16.

Burton, R. (1980). Diagnosing bugs in a simple procedural skill. In D. Sleeman & J.S. Brown, Intelligent tutoring systems, 1982, pp. 157-183.

Corbett, A.T., & Anderson, J.R (1990). The effect of feedback control on learning to program with the lisp tutor. In Proceedings of the Twelfth Annual Conference of the Cognitive Science Society, pages 796-806, Hillsdale, NJ: Lawrence Erlbaum.

Genesereth, M.R. (1982). The role of plans in intelligent teaching systems. In D. Sleeman & J.S. Brown, Intelligent tutoring systems, 1982, pp. 137-155.

Kimball, R. (1973). A self-improving tutor for symbolic integration. In D. Sleeman & J.S. Brown, Intelligent tutoring systems, 1982, pp. 283-307.

Matz, T. (1980). Towards a process model for high school algebra errors. In D. Sleeman & J.S. Brown, Intelligent tutoring systems, 1982, pp. 25-50.

Nguyen-Xuan. A., Nicaud, J-P., & Gelis, J-M. (1997), Effect of feedback on learning to match algebraic rules to expressions with an intelligent learning environment. Journal of Computers in Mathematics and Science Teaching, 16(2/3),291-321.

O'Shea, T. (1982). A self-improving quadratic tutor. In D. Sleeman & J.S. Brown, Intelligent tutoring systems, 1982, pp. 309-336.

Parvate, V., Anjaneyulu, V.K.S.R., & Rajan, P. (1998). Mathemagic: An adaptive remediation system for mathematics. Journal of Computers in Mathematics and Science Teaching, 17(2/3),265-284.

Scott, N.W. & Stone, B.J. (1995). Student behaviour near a deadline as a predictor of academic success. In Proceedings of the twelfth annual conference of the Australian Society for Computers in Learning in Tertiary Education (ASCILITE), pp. 462-467, The University of Melbourne.

Singley, M.S. (1990). The reification of goal structures in a calculus tutor: Effects on problem-solving performance, Interacting Learning Environments, 1,102-123.

Sleeman, D. (1984). An attempt to understand students' understanding of basic algebra. Cognitive Science, 6, 387-4 12.

Sleeman, D. & Brown, J.S. (Ed.) (1982). Intelligent tutoring systems. New York: Academic Press.

Table 1
Student/ CalMaeth Interaction in Assignment 1
C1            C2                             C3            C4
Question      Function to                    % Incorrect   Diags 97
(# Students)  Differentiate                  (# Attempts)  (Time)
PQ1 (308)     [x.sup.9]                      10 (3.0)       8 (0.02)
PQ2 (307)     45/4[z.sup.9/4]                67 (4.1)       9 (0.04)
PQ3 (306)     -4+3[x.sup.12/7] - 9[x.sup.5]  38 (3.4)       9 (0.06)
PQ4 (307)     -5/[(10 + 5[x.sup.6]).sup.11]  49 (3.7)      11 (0.04)
PQ5 (310)     8[(7 - 4/[x.sup.3/2]).sup.8]   48 (5.0)      11 (0.07)
AQ1 (303)     7[(10 + 5/x4).sup.9/4]         25 (2.6)       8 (0.07)
C1            C5
Question      Diags 98
(# Students)  (Time)
PQ1 (308)     10 (0.08)
PQ2 (307)     25 (0.12)
PQ3 (306)     27 (0.18)
PQ4 (307)     30 (0.09)
PQ5 (310)     27 (0.16)
AQ1 (303)     24 (0.13)

Key:

C1 - Question Number: The "PQ and "AQ" denote Practice Question and Assessed Question respectively, (Number of students completing this question).

C2 - Randomly Generated Function that students needed to differentiate.

C3 - Percentage of students who did not obtain the correct answer on their first attempt All students did eventually obtain the correct answer. (The average number of attempts that these students then required to obtain this correct answer).

C4 - The number of different diagnostics received by students for this question in 1997. (The average time taken by the Mathematics kemel to produce each diagnostic).

C5 - The number of different diagnostics that the 1997 students would have received had they used the diagnostics available in 1998. That is, these numbers of diagnostics were outputted in a simulation involving the 97 answers being submitted to CalMaeth after it had been updated with the new set of diagnostics. The new diagnostic catalogue, which was used in the 1998 course, contained diagnoses for the new misconceptions that had been identified from the 97 records; hence this number also represented the total number of misconceptions displayed by the 97 class for this question. (The average time taken by the Mathematica kemel to produce each diagnostic.)

Table 2
Student/CalMaeth Interaction in Assignment 2
C1            C2                                  C3
Question      Function to                         % Incorrect
(# Students)  Differentiate                       (# Attempts)
PQ1 (304)     [x.sup.2][(-8+11[x.sup.3]).sup.4]   48 (4.2)
PQ2 (303)     6 + 11x / -8-6[x.sup.2]             51 (3.7)
PQ3 (304)     (7 + 2z)[(9 + 3[z.sup.2]).sup.7/2]  46 (3.7)
PQ4 (305)     -2 - 2u / 7                         25 (2.9)
PQ5 (304)     9 + 2x / [(-6-2[x.sup.3]).sup.4]    58 (3.6)
AQ1 (302)     -8 -2x / 5 - 4[x.sup.2]             21 (2.3)
AQ2 (301)     7+6x / [(-10-6[x.sup.4]).sup.5]     26 (2.6)
C1            C4         C5
Question      Diags 97   Diags 98
(# Students)  (Time)     (Time)
PQ1 (304)     11 (0.09)  27 (0.15)
PQ2 (303)     22 (0.1)   39 (0.17)
PQ3 (304)     14 (0.2)   34 (0.29)
PQ4 (305)      6 (0.04)   3 (0.03)
PQ5 (304)     23 (0.11)  48 (0.18)
AQ1 (302)     14 (0.08)  28 (0.12)
AQ2 (301)     20 (0.11)  33 (0.17)

Key:

C1 - Question Number. The PQ and "AQ denote Practice Question and Assessed Question respectively, (Number of students completing this question).

C2 - Randomly Generated Function that students needed to differentiate.

C3 - Percentage of students who did not obtain the correct answer on their first attempt All students did eventually obtain the correct answer. (The average number of attempt that these students then required to obtain this correct answer).

C4 - The number of different diagnostics received by students for this question in 1997. (The average time taken by the Mathematica kemel to produce each diagnostic).

C5 - The number of different diagnostics that the 1997 students would have received had they used the diagnostics available in 1998. That is, these numbers of diagnostics were outputted in a simulation involving the 97 answers being submitted to CalMaeth after it had been updated with the new set of diagnostics. The new diagnostic catalogue, which was used in the 1998 course, contained diagnoses for the new misconceptions that had been identified from the 97 records; hence this number also represented the total number of misconceptions displayed by the 97 class for this question. (The average time taken by the Mathematics kernel to produce each diagnostic.)