The quality of instructional materials for argumentative knowledge construction.

Within this paper, instructional materials for supporting argumentative knowledge construction are evaluated. Argumentative knowledge construction concerns the building of knowledge structures based on reasoning processes. Using the qualitative method of global evaluation, five books and five teaching software products were analyzed. As basis for the evaluation, six principles of good instruction were used. Theses principles concern reflective learning, multiple learning support, orientation on strengths, efficient learning, and interest. Results show strong deficits of the analyzed teaching materials in respect to effective learning-related and motivational support. Finally, suggestions for a theory-based and multi-criteria enhancement of cognitive, motivational, and emotional learning-relevant processes are made.

**********

An "argument" is something that is used as a proof or as an affirmation for a statement. Knowing how to argue or reason is an important aim of education in general for a long time and is prominently anchored within curricula. It presupposes that learners can build knowledge by arguing what is known as "argumentative knowledge construction" (see, for example, Leitao, 2001). Argumentative knowledge construction concerns the process within which learners identify arguments, analyze them, consider external circumstances (e.g., use of language), reason scientifically, and apply logic. Identifying components of arguments concerns issues, premises, conclusions, and reasons for the conclusions. Analyzing arguments means to state implicit, unclear, or missing assumptions. Within all stages of argumentative knowledge construction, external circumstances (influences from values, authorities, or emotional language) have to be considered. Argumentative knowledge construction is also based on scientific-analytical reasoning (e.g., the research for causalities, the evaluation of statistical data and their underlying representativity). Finally, argumentative knowledge construction consists of more or less logical reasoning within which analogies and inductive/deductive reasoning are of main importance (e.g., Toplak & Stanovich, 2002).

There are close connections from the concept of argumentative knowledge construction with the concepts of "critical thinking", "everyday reasoning", "informal logic" or "pragmatic reasoning" (e.g., Galloti, 1989; Shaw, 1996). Argumentative knowledge construction is also related to basic research from cognitive psychology, philosophy, and linguistics, especially with "inductive and deductive reasoning", "causal reasoning", "abductive reasoning", "Baysian reasoning", "adaptive thinking", or "intuitive judging" (e.g., Cheng & Holyoak, 1985; Gigerenzer, 2000).

However, results from applied and basic research have not improved educational programs for promoting argumentative knowledge construction. Argumentative knowledge construction represents only a subject area of little importance within school, and, when implemented, it had no significant effects (see the literature reviews from McMillan, 1987 and Pithers & Soden, 2000).

There are several reasons for this shortcoming: First, it must be stated, that argumentative knowledge construction represents a main component within curricula on a general level, but it is not formulated in detail as practicable prescriptions for teachers. So, teachers do, as a rule, not dispose of guidelines for their daily instruction. Implementing argumentative knowledge construction would be an additional work load for teachers, which they cannot take from reducing other subject areas. Also, teachers are not educated in argumentative knowledge construction. When argumentative knowledge construction takes place within classrooms, then in some form of diffuse discussions within open learning environments (e.g., projects), or as final part within a course without a sufficient amount of time and learning support for students (e.g., Patry, 1996). Finally, the main problem, why argumentative knowledge construction is no significant component of daily instruction, comes from the fact that instruction is based on textbooks and other media (e.g., CDROM). Within such media, argumentative knowledge construction is not represented sufficiently (see, for example, the textbook and CDROM evaluations from: Astleitner, Sams, and Thonhauser, 1998; Astleitner, 2001).

Principles of Good Instruction as Quality Standards

Reflecting the assumed shortcomings of existing media, this paper is dealing with the question whether there exists instructional material (textbooks, software) which can promote argumentative knowledge construction systematically and efficiently. In order to evaluate the instructional quality of these materials, selected "principles of good instruction" from Astleitner (2002) are used within a narrative evaluation procedure.

These principles are:

Principle 1: Instruction as systematically designed opportunity for reflexive learning. Within instruction, learners should get the possibility to reflect continuously on their learning results. Reflexive learning represents an active construction process, in which memory contents are changed, expanded, linked, structured, or created. It can be supported by instruction which prevents disturbances and within which critical events (e.g., emotional conflicts) are handled successfully; by optimal, not maximal pace of instruction and by an instructional sequence which allows learners to have enough time for reflecting the presented contents; by clarity and structure of the content and the tasks; by instructional methods which are varied regularly; by goal-based instruction and sufficient opportunities for practicing and feedback; by considering individual needs and learning progresses; and by establishing a positive affective teacher-learner-relationship.

Principle 2: Multiple support for cognitive, motivational, and emotional characteristics of the learner. Educationally well-designed instructions not only assist the learner in learning and thinking, but also motivate and establish an emotionally sound climate. Cognitive aids concern presenting teaching goals, activating pre-knowledge, guiding the learning process, giving feedback, evaluating learning progress, and stimulating learning transfer. Motivational support deals with increasing attention, illustrating relevance, strengthening self-confidence and satisfaction with the results of learning. Within emotionally sound instruction, negative emotions (especially fear, envy, and anger) are decreased and positive emotions (especially sympathy and pleasure) are increased. Multiple supports should not disturb each other, but increase the quality of instruction in a complementary way.

Principle 3: Considering the strengths of the learner. Instruction is especially effective when it supports the learner to identify individual strengths and when it offers opportunities to overcome individual weaknesses. In that sense, preferences of the learner in respect to the control of the learning process (learner- vs. teacher-regulated), the type of instructional materials (e.g., texts vs. lectures), and the form of evaluations (e.g., oral or text-based) have to be considered. Also, students achievements have to be compared with individual or criterion standards (how much did I improve?), and not with social standards (what did I achieve in respect to others?). The evaluation of learner's achievements should finally be accompanied with detailed and constructive feedback showing the learner how to improve individual learning results.

Principle 4: Supporting self-regulated learning. Self-regulated learning means that the learner controls the learning process by observing progress and by selecting suitable learning aids. Self-regulated learning is based on learning strategies which have to be taught to learners. Such strategies concern methods how to remember and to structure the learning content, how to control and to improve knowledge acquisition, how to manage time for learning, how to increase motivation, how to handle emotionally difficult situations, and how to improve applied learning strategies.

Principle 5: Dosed novelty and automatization for efficient learning. When learners have acquired knowledge and strategies, it is important to optimize them in respect to the efficiency of learning. Efficiency of learning relates the effort for learning to the result of learning. Efficient learning appears especially when learners are challenged, but not too much or too little. This is the case, when tasks are presented which are anchored with the pre-knowledge of the learner, but also contain new demands. Learning becomes more efficient, when the underlying thinking processes are automated and when, in addition to the correctness of task performances, also the speed of solving tasks can be improved (e.g., in competitions with oneself).

Principle 6: Arousing and maintaining interest. Interest as motivation in respect to a certain subject matter, can be influenced from different conditions of a learning situation. Learners find interest, when they think that they are an essential part of a development or a group. Both, developments and groups must be associated with values (e.g., "we are helping others"), must be transparent and offer some assistance for individuals. Also, when learners feel competence in handling difficult learning situations, then they increase interest, especially, when learners can set their own goals and be prevented from (probably demotivating)comparisons with other learners. Interest also arouses, when learners have fundamental pre-knowledge, but are shown that this knowledge is incomplete or can be expanded. Finally, interest can be stimulated by considering personally important learning contents, activating worksheets, instructional games, stories, fantasy worlds, humor, guided group activities, presenting models, etc.

Method and Evaluated Instructional Materials

For evaluating the instructional materials, the six principles will be used as standards. As method the "global evaluation system" from Legewie (1994) was implemented. Within this method, the following steps were applied: orientation (browsing the instructional materials), activation of context knowledge (repeating the six principles), preliminary evaluation (analyzing the instructional materials based on the six principles for the first time), final classification with key-words (analyzing the instructional materials for the second time with recording the resulting quality characteristics), and summarized reporting. For each instructional material, 60 minutes were used for evaluation and final quality classification.

Five textbooks for promoting argumentative knowledge construction were analyzed, i.e., from Astleitner (1998), Bierman & Assali (1996), LeBlanc (1998), Petri (2000), and Walton (1989). These textbooks are frequently used in German- and in English-speaking countries on high school and on university level.

Also, five computer-based instructional materials were analyzed, i.e., REASON!ABLE (URL http:/ /www.goreason.com), KRIT.NET (URL http:// www.sbg.ac.at/erz/aaakurs/home.htm), ALFIE (URL http:llwww.cs.chalmers.sel~sydow/alfie/index.html), SPIDERMAP (URL http:// weatherhead.cwru.edu/ infosys/spidermap/index.html), and LOGIC WEB TUTOR (URL http://www.poweroflogic.com). These materials were selected after a comprehensive Internet-search and after a first check for instructional quality based on reliability and relevance of content and degree of learning support. Based on these standards, for example, the following instructional materials were not included within this evaluation: CSLI-LOGIC SOFTWARE (URL http://www-csli.stanford.edu/hp/ Logic-software.html), FALLACY TUTORIAL PRO (Opifex Phoenix Software, URL http:// user.aol.com/ontologist/web/opifex.sofi.html),or TEACHING LOGIC AS A TOOL (URL http:// www.cs.cornell.edu/Info/People/gries/Logic/ Introduction.html).

Results

In the following part of the paper, evaluation results will be presented based on the six principles for all textbooks and computer-based instructional materials.

Principle 1: Reflexive Learning

The textbooks from Astleitner (1998), Bierman & Assali (1996), LeBlanc (1998), Petri (2000), and Walton (1989) are all structured linearly or hierarchically, what means that the learner has to work through the material from front to back. The learner is not invited or allowed to select and read chapters based on individual needs. All textbooks contain exercises and different instructional methods, whereby above all within the textbook from Petri (2000), these methods are explicitly integrated within the text, and within the textbook from Astleitner (1998), they are presented within a separate chapter. Within the other textbooks, the only instructional method is to work through the texts in a self-regulated way. Other instructional methods (e.g., collaborative group activities) are not used. Also, within none of the textbooks, teaching goals are formulated, so that criterion-oriented learning cannot be realized. Individual learning progress is not registered or evaluated, although there are many exercises which could present tasks and solutions for evaluating the individual state of knowledge and could allow constructive feedback for improvement. An exception represents the textbook from LeBlanc (1998): Within this textbook, the individual state of knowledge can be evaluated when using an associated computer-based drill-resp. tutorial program which presents in detail tasks, solutions, and feedback on the learner's performance. Finally, the establishment of an affective relation to the learner cannot be observed in any of these textbooks.

REASON!ABLE supports the active and self-regulated construction of argument chains by using a graphical and symbol-based interface. An implemented counselor offers--when needed--advice in guiding the construction process. It offers exercises, but only one instructional method (i.e., guided self-regulated learning). Teaching goals are hidden within the counselor's advice. Individual pre-conditions for learning are not measured and used within the instructional process. Promoting an affective quality of a teacher-learner-relationship cannot be ob served, but can be experienced in some degree as the counselor appears as fatherly friend. KRIT.NET offers an open and text-based learning environment, within which the learner can freely select the elements of a course unit (necessary pre-knowledge, teaching goal, contents, exercises, etc.) and therefore the sequence and pace of instruction. Exercises contain hints for solving the given problems and also a correct solution. Instructional methods vary in a way that different types of exercises (multiple-choice, drag-and-drop, etc.) are used. All course units are goal-based. Individual pre-conditions and learning progress can be evaluated by the learners themselves when they work through the exercises and test sections. The learning environment gives some information about what to do when learning problems occur, but it does not guide the learner step-by-step. An affective relationship between learner and learning environment is not established. ALFIE offers the possibility to proof logical statements. For each task, the learner can try different solutions and get the correct solution. In addition, the learner can create own tasks. The learning environment offers no other learning support. SPIDERMAP allows to represent thoughts about the causal relationships of concepts within a graphical map. This tool offers no further options for promoting reflexive learning. LOGIC WEB TUTOR shows a structured text-based learning environment with many different content areas relevant for argumentative knowledge construction. Content and exercises can freely be selected by the learner what allows sufficient time for reflecting. There are many exercises with solutions, but instructional methods are not varied. Test results are presented to the learner, but without advising what should be done next.

Principle 2: Multiple Support

Above all, the textbooks from Astleitner (1998), Bierman & Assali (1996), LeBlanc (1998), and Walton (1989) are clearly structured, contain optically stimulating graphics, use many practical examples, etc. what should also have some motivational effect on the learner. Systematically designed features for stimulating motivational or emotional characteristics of the learners cannot be found in all textbooks. As argumentative knowledge construction represents a high-level teaching goal with high difficulty, it can be expected that the motivation of learners will be decreased because of occurring failures and lacking success. Such a learning situation would need the implementation of additional motivational/emotional strategies as, for example, outlined from Keller and Koop (1987).

All analyzed computer-based environments focus only the cognitive aspects of learning without having features for influencing motivational or emotional characteristics of the learners. In that sense, they do not at all realize multiple support.

Principle 3: Orientation on Strength

Within all textbooks, there are no options which could allow the learner to acquire the learning content based on individual strengths. For example, it might be possible that all exercises are rated based on their difficulty. Such an option would give good learners the possibility to confront themselves with challenging tasks (of high difficulty), and bad learners to get sufficient success (with easy tasks). Also, it is not possible within the textbooks to register individual learning histories because of lacking forms. Finally, there are not at all any hints of how to diagnose individual strengths or how to improve weaknesses.

The strengths of learners are part of the computer-based environments in a way that the environments allow a comprehensive control of the learning process by the learner. However, that this option becomes positive for learning, requires that the learners are able to diagnose their strengths by themselves what must be doubted for many learners. Also, within all computer-based environments, it is not possible to show and use individual learning histories, because they do not offer any automatic learner tracking.

Principle 4: Self-regulated Learning

Within the textbook from Petri (2000), many different instructional methods for promoting argumentative knowledge construction are depicted. Within these methods, aids are included for promoting self-regulated learning. But, this textbook represents more a collection of instructional examples for teachers than contents and guidelines for learners. Within the textbook of Walton (1989), there are no links to learning strategies suitable for self-regulated learning. Astleitner (1998) shows how to improve argumentative knowledge construction within classroom contexts, but gives no guidelines for individual self-regulated learning. Within the textbook from LeBlanc (1998) some aids related to learning strategies are included, but they focus on a specific type of content and not on self-regulated learning in general.

All computer-based environments offers opportunities for self-regulated learning as the learner can select the content, pace, etc. However, these environments do not show the learner how self-regulated learning can be evaluated and improved. Some restricted hints for controlling self-regulated learning are given from the counselor within REASON!ABLE and are part of FAQs (frequently asked questions) within KRIT.NET.

Principle 5: Learning Efficiency

All textbooks contain exercises with a certain degree of dosed novelty and capability for automatization, but this degree is not empirically tested or optimized what means that these exercises are not really suitable for increasing learning efficiency. Learning efficiency means primarily achieving a certain learning achievement within a certain time. Within none of the textbooks, necessary time for solving the given tasks are presented. It is only stated that argumentative knowledge instruction represents a difficult teaching goal which needs sufficient time resources.

Also, all computer-based learning environments, except SPIDERMAP, offer many exercises which could be used to increase learning efficiency. However, a necessary condition would be that the exercises are ranked according to a certain learning efficiency increasing principle. This is not the case: Exercises are not ranked, for example, according their difficulty (from easy to difficult tasks), but they can be selected freely. In addition, no computer-based learning environment registers performance data from the learners what could be used for increasing learning efficiency. The only element for increasing learning efficiency can be found within KRIT.NET and LOGIC WEB TUTOR as all course units are ordered hierarchically according to necessary pre-knowledge. This circumstance could be used as a rough guideline for optimizing learning efficiency, but it remains on a too general level in order to really influence learning efficiency on a micro-level.

Principle 6: Interest

Argumentative knowledge construction is usually trained with examples from everyday life and not from formal logic. This principle of close relation to daily experiences guarantees to some degree a certain interest, because learners can find their own world within the learning content. Exercises in respect to motivating group activities can be found in the textbook from Petri (2000) and Astleitner (1998). Instructional games, fantasy worlds, etc. or the showing of incomplete knowledge are not used in any of the textbooks. Above all, the textbooks present a learning content, but do not care about the interest of learners. This circumstance might be related to the fact that argumentative knowledge construction needs a lot of cognitive resources in order to be successful. These resources should not be reduced by instructional methods which increase interest. Obviously, in the view of the textbook's authors, motivating strategies implemented within the textbooks could distract the learner from cognitive learning goals.

All analyzed computer-based environments offer, on the one hand, no exercises for group- or development-related activities. On the other hand, all environments allow to set own goals and to work without negative social comparisons. KRIT.NET realizes to some degree a stepwise showing of incomplete knowledge, when step-by-step hints for problem solving are given. Also, the given little variation in instructional methods in all computer-based learning environments will not stimulate interest.

To sum up, the evaluational results show that, both textbooks and computer-based environments for supporting argumentative knowledge construction show massive deficits when considering empirically proved instructional quality standards. In most cases, both types of instructional materials offer a comprehensive and well organized presentation of the relevant learning content what can be a first basis for successful learning. However, advanced and varied instructional methods to increase learning and motivation are missing. Both elements are necessary because of the high complexity of argumentative knowledge construction and related learning and motivational problems.

Discussion and Implications

First of all, the presented piece of research has to be criticized in methodological respect. Only 10 instructional materials were analyzed what does not allow to achieve representativity for all instructional materials supporting argumentative knowledge construction. But, the materials were not selected based on representativity, but on their frequency of usage in daily instructional settings. It was assumed that often used materials show a higher quality than not or seldom used materials. In that sense, the paper realized a best-product-analysis. The circumstance, that this evaluation of assumed high-quality products shows negative results in respect to instructional quality of the materials, is especially remarkable. Another methodological problem comes from using the global evaluation system without having standards in respect to reliability and validity. However, this problem of testing reliability and validity remains not really important when considering that the given principles could easily be identified within the instructional materials.

It must also be criticized that all materials are normally used in combination with a teacher. Within this paper, the materials were evaluated assuming that the learner works self-regulated without the help of a teacher. This restriction was made, because self-regulated learning is one of the most important issues or even paradigm in recent educational media research and development. It must also be considered that instructional materials for supporting argumentative knowledge construction are more frequently used within daily instructional practice when they reduce the workload for teachers what depends on their ability to realize self- or learner-regulated learning. Instructional materials with high ability to promote learner-regulated learning must have the following elements: continuing orientation on teaching goals, clear structuring of the content, tasks for diagnosing the state of knowledge, learning guidance with questions, advance and post organizers, and a variety of exercises with solutions and solution paths, learning and motivation promoting text design (e.g., personally addressing the learner or highlighting important sections).

For daily instruction, it is obvious, based on the results of this paper, that instructional materials for argumentative knowledge construction must be improved in respect to supporting learning and motivation.

Referring to learning support, it must be realized that learners need, on the one hand, exercises with diagnostic functions, and on the other hand, methods related to open learning environments which allow to acquire advanced knowledge and skills. Within such open learning environments, learners should be allowed to search for information, to use static and dynamic information resources, to apply tools for information collection, generation, processing, etc., and to get tutorial advice (scaffolding, coaching, etc.) (see Hannafin, Land & Oliver, 1999). Open learning environments increase, as a rule, the complexity of learning, what produces the problem that cognitive resources for the main learning task (i.e., achieving successful argumentative knowledge construction) are reduced and that much cognitive capacity is focused on navigation, etc. In order to reduce this problem, recent research from "cognitive load theory" (Sweller, 1994) should be considered, especially when designing instructional texts, when handling learning mistakes, or when using multimedia elements (e.g., Mayer & Moreno, 2002).

A theoretical approach to handle motivational support within instructional materials delivered Keller (1999) with his ARCS-model. This approach of motivational instructional design should be used for improving the motivational quality of instructional materials. It is based on four groups of instructional strategies dealing with attention, relevance, confidence, and satisfaction. As within argumentative knowledge construction failures in learning can frequently be experienced, also strong emotions might occur during learning. To handle emotions during instruction, the FEASP-approach from Astleitner (2000) could be used. Within this approach, instructional strategies are proposed that should decrease negative emotions (like fear, envy, and anger) and increase positive emotions (like sympathy and pleasure).

Especially such approaches should be used to promote a theory-based design of instructional materials which relies not only on the presentation of the learning content, but also on the multiple support of cognitive, motivational, and emotional processes. The presented paper should stimulate corresponding research in the field of argumentative knowledge construction.

References

Astleitner, H. (1998). Kritisches Denken. Basisqualifikation fur Lehrer und Ausbilder. Innsbruck: Studienverlag.

Astleitner, H. (2000). Designing emotionally sound instruction: The FEASP-approach. Instructional Science, 28, 169-198.

Astleitner, H. (2001). Kann kritisches Denken mit neuen Lernmedien gefordert werden? Padagogisches Handeln, 5, 132-141.

Astleitner, H. (2002). Prinzipien guten Unterrichts [WWW document]. URL http://www.qis.at/material/ astleitner_unterrichtsqualitat.pdf

Astleitner, H., Sams, J., & Thonhauser, J. (1998). Womit werden wir in Zukunft lernen? Schulbuch und CD-ROM als Unterrichtsmedien. Ein kritischer Vergleich. Wien: OBV Padagogischer Verlag.

Bierman, A. K., & Assail, R. N. (1996). The critical thinking handbook. Upper Saddle River, NJ: Prentice Hall.

Cheng, P. W., & Holyoak, K. J. (1985). Pragmatic reasoning schemas. Cognitive Psychology, 17, 391-416.

Galloti, K. M. (1989). Approaches to studying formal and everyday reasoning. Psychological Bulletin, 105, 331-351.

Gigerenzer, G. (2000). Adaptive thinking. Rationality, in the real world. Oxford: University Press.

Hannafin, M., Land, S., & Oliver, K. (1999). Open learning environments. Foundations, methods, and models. In C. M. Reigeluth (Ed.), Instructional-design theories and models. A new paradigm of instructional theory (pp. 115-140). Mahwah, NJ: Erlbaum.

Keller, J. M. (1999). Motivation in cyber learning environments. International Journal of Educational Technology, 1, 7-30.

Keller, J. M., & Kopp, T. W. (1987). An application of the ARCS model of motivational design. In C. M. Reigeluth (Ed.), Instructional design theories in action. Lessons illustrating selected theories and models (pp. 289-320). Hillsdale, NJ: Erlbaum.

LeBlanc, J. (1998). Thinking clearly. New York: Norton.

Legewie, H. (1994). Globalauswertung von Dokumenten. In A. Boehm, A. Mengel, & T. Muhr (Hrsg.), Texte verstehen. Konzepte, Methoden, Werkzeuge (S. 177-182). Konstanz: Universitatsverlag.

Leitao, S. (2001). Analyzing changes in view during argumentation. A quest for method [WWW document]. Forum Qualitative Social Research, Vol. 2, No. 3. URL http:/www.qualitative-research.net

Mayer, R. E., & Moreno, R. (2002). Aids to computer-based multimedia learning. Learning and Instruction, 12, 107-119.

McMillan, J. H. (1987). Enhancing college students' critical thinking. A review of studies. Research in Higher Education, 29, 3-29.

Patry, J.-L. (1996). Qualitat des Unterrichts als Komponente von Schulqualitat. In W. Specht, & J. Thonhauser (Hrsg.), Schulqualitat (S. 58-94). Innsbruck: Studienverlag.

Petri, G. (2000). Wie kann kritisches Denken wirksam geschult werden? Innsbruck: Studienverlag.

Pithers, R. T., & Soden, R. (2000). Critical thinking in education. A review. Educational Research, 42, 237-249.

Shaw, V. (1996). The cognitive processes in informal reasoning. Thinking & Reasoning, 2, 1-104.

Sweller, J. (1994). Cognitive load theory, learning difficulty, and instructional design. Learning and Instruction, 4, 295-312.

Toplak, M. E., & Stanovich, K. E. (2002). The domain specificity and generality of disjunctive reasoning. Searching for a generalizable critical thinking skill. Journal of Educational Psychology, 94, 197-209.

Walton, D. N. (1989). Informal logic. A handbook for critical argumentation. Cambridge: University Press.

Dr. Hermann Astleitner, University of Salzburg; Dr. Brunken, faculty, University of Erfurt; Dr. Leutner, faculty, University of Essen.

Correspondence concerning this article should be addressed to Dr. Hermann Astleitner, School of Humanities, the University of Salzburg, Department of Educational Research, akadaemiestrasse 26, a-5020 Salzburg, Austria, Email: [e-mail].