Collaborative knowledge building in web-based learning: assessing the quality of dialogue.

Web-based learning is assuming an increased role in education at college and pre-college levels (Bates, 1999; Collis, 1996; Fjuk, Sorensen, & Wasson, 1999). It is not only situations of continuing education and lifelong learning that call for flexible organization of the learning process, for example, with the purpose of bridging distances. The obvious advantages in terms of a more flexible, time- and space-independent organization of the learning process, also within higher education, are clear (Harasim, 1999). The flexible organizational design of web-based learning, by itself, does not make this method superior to other methods. Of more importance is the quality and design of the learning process. Many flexible designs offer no possibilities for interaction and collaboration among students, but are rather sterile and lack a framework to stimulate collaborative knowledge building. Others do so, but have problems -- even when interaction occurs -- in qualifying a knowledge building dialogue.

This article deals with the question of how to qualify a knowledge-building dialogue through both grading requirements and as an effect from providing the students with metaawareness around the functions of the requested comments in the dialogue. The article suggests that grading requirements -- as well as providing awareness around the requirements related to the function of comments in the dialogue -- contribute to qualifying a knowledge building process in collaborative learning.

In this article, we begin by providing some brief information of the web-based course producing the data of our analysis. We then provide an account of our knowledge building perspective and the criteria of quality used in our analysis of web-based dialogues. Following this, we address the communicative conditions of web-based environments, forming part of the rational behind our hypothesis. Then, we give a more detailed account of our research design and the method used in our analysis, and provide the basic analysis. And finally, we reflect on the results of the analysis and discuss future research perspectives.

DEVELOPMENT OF DIALOGUE REQUIREMENTS IN "GLOBAL CHANGE"

Global Change (GC) is a conventional science course for senior undergraduates or beginning graduate students at a US university. It gradually has been migrated to a web-based course over the last six years, with new features being added as ancillary software has become available. Learner-centered activities in place of or supplemental to conventional lectures have been introduced.

The course consists of a sequence of learning modules on different global-change topics, each having evolved from a conventional university class time period. Each unit has: a set of objectives; summary information on the topic; student-submitted collaborative (2-3 students) summary of class time discussion; "problems to ponder" as discussion starters for the electronic dialogue; and extensive lists of web and other information on the learning module topic. Each unit has its own electronic dialogue for student discussion among themselves and with outside experts or representatives of selected groups. Electronic dialogue on individual learning unit topics is graded. The designers view the course as a laboratory for experimenting with a variety of pedagogical techniques and initiatives (Taber et al., 1997).

The course went on the Web in the spring term 1995. Affiliated with the course was an opportunity to engage in electronic dialogue. The use of dialogue and associated instructor evaluation of student engagement in dialogue changed over the years as follows:

1995: Electronic dialogue was available separately for each lecture topic, and 5% of the evaluation of student performance was based on whether or not a student participated. There were no minimum requirements on amount of discussion and no attempt was made to judge quality as part of the evaluation of students.

1996: Same as 1995, except 15% of the grade depended on the electronic dialogue. Some weak attempt was made to judge quality but primary criterion was participation and not quantity or content.

1997: Requirements for participation in electronic dialogue were substantially enhanced. For full credit, the student was required to post at least 15 entries, which had to include responses to at least six other students' comments. Furthermore, these entries had to elicit comments from three other students. They also were required to respond to three questions issued by the instructor and to give their views on three additional ethical questions relating to global change. 30% of evaluation of students was based on these numbers, but no evaluation of quality was attempted. A virtual portfolio was introduced for each student to manage discussion.

1998 and 1999: Same as 1997.

2000: Same requirements as 1998, with the addition of the knowledge-building process (Stahl, 1999) that required students to meet specific quality criteria in their discussion. Also, students were required to assess their own writing and demonstrate how they had met the criteria. 42% percent of the evaluation of student performance was based on discussion.

CRITERIA OF COLLABORATIVE KNOWLEDGE BUILDING IN WEB-BASED LEARNING

To put our analysis of electronic dialogues in perspective, we briefly outline the set of criteria that we assert are signs of quality in a web-based collaborative knowledge building process.

In the principles of collaborative learning, the process of learning is viewed to be a fundamentally social phenomenon, regardless of the varying theoretical emphasis in each single approach (Dillenbourg, Baker, Blaye, & O'Malley, 1995). Several other learning theories confirm this view, e.g., Etienne Wenger in his latest book, "Communities of Practice" (Wenger, 1998).

The process of knowledge building (KB) in collaborative learning, first explored by Harasim (1989), involves mutual exploration of issues, mutual examination of arguments, agreements and disagreements, mutual questioning of positions, dynamic interaction and weaving of ideas (Harasim, 1989; Kaye, 1992; Sorensen, 1997). Mason (1993) finds this view to be in agreement with the communicative potential of the online environment, although she also points out the weaknesses of the online dialogue being that it quite often never reaches synthesis or closure (Mason, 1993).

In view of the generally recognized difficulties in fostering online student dialogue that converges (e.g., synthesizing) rather than diverges (noted by Mason), Stahl (1999) suggests a set of factors that characterize quality in the KB process:

* Brainstorming is the introducing of new ideas that relate to the topic or task and offer a perspective not previously considered.

* Articulating includes explaining complex or difficult concepts.

* Reacting provides an alternative or amplified perspective on a concept previously introduced by a student.

* Organizing refers to assembling existing thoughts or perspectives in such a way that a new perspective emerges.

* Analyzing includes comparing or contrasting previously articulated views or puts new understanding on existing data.

* Generalyzing takes comments or data already presented and extracts new information or knowledge that applies to a broader set of conditions.

Implementing these learning quality criteria of collaborative KB requires a corresponding meta-functional pedagogy or instruction that facilitates and motivates such collaborative dialogue.

REFLECTIVE DIALOGUE PRIMARY TO INVOLVED DIALOGUE

Promoting knowledge-building dialogue within the context of collaborative learning appears to be a complex challenge. In searching for reasons for this, some research has focused on the design and nature of the collaborative activities implemented in the learning process (Collis, 1997), and others on the quality and nature of the virtual environment and the evolvement of electronic interhuman dialogue (Scardamalia & Bereiter, 1996; Sorensen, 1993). Alternative studies have concluded the social aspect of a group process to be an essential motivator for collaboration in online learning (Harasim, 1993; Cornell & Martin, 1997). It seems to carry a rather high proportion of a group member's inclination to engage in any interaction with the group at all (Moore & Kearsley, 1996). This seems in total agreement with the widely acknowledged insight that inclination to interact online is sensitive to the perception of interaction (Gunawardena, 1995).

Considerable research has been directed towards the role and meta-communicative behavior of the instructor (Feenberg, 1989). Recent studies along this line of thinking have described the online universe as a meta-communicative world. Contrary to the physical world in which involvement is viewed to be primary to reflection (Heidegger, 1986), the virtual universe provides a context and an "ontology" in which reflection may be said to be primary to involvement (Sorensen, 1999). Assuming this new dialogical paradigm and this primary position of reflection and meta-communication, it is conceivable -- from the perspective of collaborative interaction and dialogue in Web-based learning -- that also the task of scaffolding learning processes that aim at supporting both interaction and (self) reflection must move at a meta-level in terms of creating awareness of the function in a dialogue of a contributed comment.

RESEARCH DESIGN

In our analysis, we wanted to investigate the role and nature of the instructions in the requirements given to the students, stimulating them to interact. We wanted to assess whether meta-communication in terms of providing meta-awareness of "the function of a comment" in the KB process will improve the quality of the KB dialogue, according to the definition of quality as suggested by Stahl (1999). In our research design and analysis, we make the following assumptions:

1. Characteristics of discussion and comments that contribute to a KB process are (Stahl, 1999): brainstorming, articulation, reaction, organization (including synthesis), analysis and generalization.

2. Use by students of characteristics of the KB process in written dialogue contributes to student learning.

Our hypothesis for the analysis is that by explaining the characteristics of a KB-process, and by grading (Sorensen & Takle, 1999) student discussion on the basis of their reflected use of these characteristics, students will measurably increase their use of these characteristics. The rational behind the hypothesis is an acknowledgement of the need in primarily reflective virtual environments for communicatively providing meta-awareness in relation to expected communicative actions.

We have evaluated characteristics of student dialogue and its relationship to course requirements by assembling 10 comments from 1995; 10 from 1997; and 10 from 2000. These were drawn at random -- although not strictly with statistical rigor -- and represent comments made under quite different criteria for evaluation of student performance. In drawing the comments from the respective databases, care was taken to draw comments from the same (or very closely related) topics in each of the three years.

The data sets represent: (a) comments with no obligations attached as far as evaluation of student performance was concerned (1995), (b) comments when numbers of responses and numbers of interactions among students were used for evaluation (1997), and (c) comments when the KB requirements were imposed. For 2000, we also have, for some database entries, the student self-assessment of their own writing. We also have assigned a numerical value from 1 to 10 representing the quality of the comment.

ANALYSIS OF DIALOGUES

For the purpose of our analysis, we have categorized the analyzed comments in the online dialogues drawing on the knowledge building quality requirements presented by Stahl (1999):

* comment with no KB characteristics

* question

* analysis

* articulation

* reaction

* brainstorming

* organization

* generalization

A level of quality (1-10) was subjectively assigned to each comment. The mean quality went from 4.4 to 3.2 to 5.3 from 1995 to 1997 to 2000, respectively. In 1995, a preponderance of entries was questions, whereas in 1997 there were more reactions to other students. In 2000, more student comments were characterized as articulation.

From these results, we draw the following tentative conclusions. In 1995, with no requirements on volume or quality of writing, students used the electronic dialogue for asking questions. The questions did not seem "forced" since their postings had essentially no bearing on how they were evaluated in the course. For 1997, the overall quality of the postings went down. There seemed to be more "forced" responses. More questions were of a rhetorical nature, rather than seeming to originate out of curiosity about the subject matter. We attribute this to a sense of urgency on the part of students to meet the volume obligations of the evaluation scheme. The increase in quality from 1997 to 2000 suggests that students responded to the demands for higher quality of writing by meeting the requirements of the KB process. A notable difference in the 2000 discussion as compared to previous years was the reduction in the "social" or "chat" nature of the comments. But at the same time, the number of comments increased. Com ments were much more like individual essays and less spontaneous.

Perhaps the largest difference over the change in dialogue requirements was in the length of each entry. In 1995, the average length of the 10 entries was 88 words, and in 1997, the average was 93 words. In 2000, however, the average soared to more than 2,500 words. And the quality of the grammar of each posting increased markedly as well. This likely is attributable to the fact that students wrote their postings as essays in word processors on their own computers, subjected them to spell checking, and reviewed the grammar before cutting and pasting them into the course website. This increased quality and quantity came at the expense of spontaneity and "social" content.

DISCUSSION AND FUTU RE PERSPECTIVES

How do we evaluate the results of our analysis? Assuming that meta-awareness (Gutwin, Stark, & Greenberg, 1995) of KB characteristics among students as well as grading of comments increase the process of collaborative learning -- and accepting the value of the KB characteristics used in our analysis -- we think it is fair to say that our experiment has proved itself to be relevant. Over the three trials, the quality of student comments has increased, both as a result of grading requirements and as an effect from the metaawareness around the functions of the requested comments.

Our sample size was quite small and, therefore, the results are subject to large uncertainty, as such, the extent to which the results can be generalized may be questioned. The selected comments, however, were drawn at random and taken out of the social context of the specific dialogue in which it appeared. Perhaps this could be considered a problem of the data set, since the KB requirements are concerned with the sequential development of online interaction.

As pointed out, the largest difference over the change in dialogue requirements was in the length of each comment and the increased quality of the grammar used. It seems that the more carefully composed essay-like style of comments -- as well as the increased quality and quantity -- came at the expense of spontaneity and "social" elements in the interaction. It is conceivable that dealing with electronic dialogues and with an electronic comment in terms of a request for reflection on its function is inhibiting the evolvement and practice of a spontaneous and dynamic dialogue.

As such, the question could be posed whether the strong meta-reflective requirements enforced on a dynamic phenomenon as human interaction (even though it is not in terms of formalization) appears problematic. Or perhaps whether the lack of recognition of social elements in the interaction works against an incitement to engage in collaborative interaction or knowledge building with others. Moreover, it may be pertinent, as a basis for a potential conclusion around this aspect, to specify whether one is using data from an off-campus course -- where the students have no complementary social interaction -- or an on-campus course -- where social interaction among students is part of daily life -- as in the present study.

In the data from our experiment, there is some indication, that the "forced requirements" on the collaborative dialogue actually functioned like a "kick-off" to engage in dialogue, and that it actually caused more interactivity than required to appear. But we find the data set from this study too small to conclude more precisely around this aspect. It is beyond any doubt, though, that -- as default -- to be forced to reflect on the function of one's action before carrying it out is likely to decrease spontaneity, phatic behavior and high interactivity. Above all, we believe that the optimistic results of our analysis have demonstrated that, once again, the problem is a complex one. And perhaps the usual controversial points to discuss in this respect will be "how" to do it and "to what extent". All these are questions to be further studied and pursued in the future.

ACKNOWLEDGMENTS

We acknowledge the contribution of Kathryn St. Croix, Iowa State University, for collaborating with us in the analysis of dialogues. Contributors to the design and implementation of the Global Change course include Doug Fils, Michael Taber, Jennifer Hodson, and David Flory. The course is implemented under the auspices of the International Institute of Theoretical and Applied Physics.

Note: This article received an "Outstanding Paper" award at ED-MEDIA 2001 -- World Conference on Educational Multimedia, Hypermedia & Telecommunications, held in Tampere, Finland. ED-MEDIA 2002 will be held in Denver, Colorado; www.aace.org/conf/edmedia

REFERENCES

Bates, A. W. (1999). Managing technological change: Strategies for academic leaders. San Francisco. Jossey Bass.

Collis, B. (1996). Tele-learning in a Digital World. Stamford, CT: international Thomson Computer Press.

Collis, B. (1997). Experiences with web-based environments for collaborative learning and the relationship of these experiences to HCI research. In D. Dicheva & I. Stanchev, (Eds.), Proceedings of the Working Conference of IFIP WG 3.3 (pp. 137-150), Sozobol, Bulgaria. Sofia, Bulgaria: VIRTECH, Ltd.

Cornell, R., & Martin, B. L. (1997). The role of motivation in web-based instruction. In B. H. Khan, (Ed.), Web-based instruction (pp. 93-100). Englewood Cliffs NJ: Educational Technology.

Dillenbourg, P., Baker, M., Blaye, A., & O'Malley, C. (1995). The evolution of research on collaborative learning. In P. Reimann & H. Spada (Eds.), Learning in human and machines: Towards an interdisciplinary learning science (pp. 189-211). London: Pergamon.

Feenberg, A. (1989). The written world. In R. Mason & A.R. Kaye (Eds.), Mindweave: Communication, computers, and distance education (pp. 22-40). Oxford, UK: Pergamon.

Fjuk, A., Sorensen, E. K., & Wasson, B. (1999). Incorporating collaborative learning, networked computers and organisational issues into theoretical frameworks. In Proceedings of the ICDE99 Conference, The New Educational Frontier: Teaching and Learning in a Networked World, Vienna, Austria.

Gunawardena, C. N. (1995). Social presence theory and implications for interaction and collaborative learning in computer conferences. International Journal of Educational Telecommunications 1(2/3), 147-166.

Gutwin, C., Stark, G., Greenberg S. (1995). Support for Workspace Awareness in Educational Groupware. In Proceedings of the Computer Support for Collaborative Learning (CSCL) 1995 Conference. Mahwah, NJ: Lawrence Erlbaum Associates. Available: http://www-cscl95.indiana.edu/cscl95/

Harasim, L. M. (1989). Online education: A new domain. In R. Mason. & A.R. Kaye (Eds.), Mindweave: Communication, computers, and distance education (pp. 50-62). Oxford, UK: Pergamon.

Harasim, L. (1999). The virtual-u field trials: Lessons about teaching and learning online. Unpublished manuscript.

Harasim, L. (1993). Collaborating in cyberspace: Using computer conferences as a group learning environments. Interactive Learning Environments, 3(2)119-130.

Heidegger, M. (1986). Sein und Zeit. Thbingen, Germany: Max Niemeyer Verlag.

Kaye, A. R. (1992). Learning together apart. In AR. Kaye (Ed.), Collaborative learning through computer conferencing (pp. 1-24). (NATO ASI Series, Vol. 90.) London: Springer-Verlag.

Mason, R. D. (Ed.) (1993). Computer conferencing: The last word. Victoria BC, Canada: Beach Holme.

Moore, M. G., & Kearsley, G. (1996). Distance education: A systems view Belmont, CA: Wadsworth.

Scardamalia, M., & Bereiter, C. (1996). Computer support for knowledge building communities. In T. Koschmann (Ed.), CSCL: Theory and practice of an emerging paradigm, (pp. 249-268). Hillsdale, NJ: Lawrence Erlbaum Associates.

Sorensen, E. K. (1993). Dialogues in networks. In Andersen, Holmqvist, & Jensen, (Eds.), The Computer as a medium (pp. 389-421). Cambridge, UK: Cambridge University Press.

Sorensen, E. K. (1999). Intellectual amplification through reflection and didactic change in distributed collaborative learning. In C. M. Hoadley & J. Roschelle (Eds.), Proceedings of the Computer Support for Collaborative Learning (CSCL) 1999 Conference (pp. 582-589), Stanford University, Palo Alto, CA. Mahwah, NJ: Lawrence Erlbaum Associates. Available: http://kn.cilt.org/cscl99/A71/A71.HTM 

Sorensen. E. K. (1997). Learning in virtual contexts: Navigation, interaction, and collaboration. Doctoral dissertation, Aalborg University, Denmark.

Sorensen, E. K., & Takle, E. S. (1999): Distributed collaborative learning across disciplines and national borders. In C. M. Hoadley & J. Roschelle (Eds.), Proceedings of the Computer Support for Collaborative Learning (CSCL) 1999 Conference, (pp. 575-581) Stanford University, Palo Alto, CA. Mahwah, NJ: Lawrence Erlbaum Associates. Available: http://kn.cilt.org/cscl99/A70/A70.HTM

Stahl, G. (1999). Reflections on WebGuide: Seven issues for the next generation of collaborative knowledge-building environments. In C. M. Hoadley & J. Roschelle (Eds.), Proceedings of the Computer Support for Collaborative Learning (CSCL) 1999 Conference, (pp. 600-610), Stanford University, Palo Alto, CA. Mahwah, NJ: Lawrence Erlbaum Associates. Available: http://kn.cilt.org/cscl99/A73/A73.HTM

Taber, M. R., Takle E. S., & Fils, D. (1997). Use of the Internet for student self-managed learning. Preprints, Sixth Symposium on Education, Long Beach, CA. Boston, MA: American Meteorological Society.

Wenger, E. (1998): Communities of practice: Learning, meaning, and identity. Cambridge, UK: Cambridge University Press.

ELSEBETH KORSGAARD SORENSEN, DEPARTMENT OF COMMUNICATION, AALBORG UNIVERSITY, AALBORG, DENMARK

EUGENE S. TAKLE, DEPARTMENT OF GEOLOGICAL AND ATMOSPHERIC SCIENCE & DEPARTMENT OF AGRONOMY, IOWA STATE UNIVERSITY, IOWA, USA

E-MAIL: [e-mail]