Printer Friendly

Measuring and Visualizing Group Knowledge Elaboration in Online Collaborative Discussions.

Introduction

In collaborative learning environments, online discussion is an important activity that engages learners in asking questions, articulating their thoughts, explaining and justifying their opinions, and sharing ideas and resources, all of which contribute to meaningful collaborative learning (Li et al., 2009). Knowledge elaboration, as an integral part of online discussion, refers to how learners organize, restructure, interconnect, and integrate knowledge (Reigeluth et al., 1980; Kalyuga, 2009; Gleaves & Walker, 2013), thereby promoting knowledge acquisition and knowledge retention (Anderson, 1983; Denessen et al., 2008; Golanics & Nussbaum, 2008; Stegmann et al., 2012; Zheng et al., 2015).

Researchers have reported that knowledge elaboration has positive effects on group problem-solving (Eryilmaz et al., 2013) and student achievement (Van et al., 2000; Stark et al., 2002; Hwang et al., 2007). However, most previous studies analyzed knowledge elaboration through qualitative manual coding, which involves subjective judgment, and the reliability of dialog analysis schemes also remains a contentious issue (Pilkington, 2001). Because this method is time-intensive and conducted post-event, the information obtained does not offer opportunities for real-time feedback to enhance evaluation, reflection, awareness, and adaptation of collaborative learning (Kumar et al., 2010). Besides, the analysis results are meaningful mainly for researchers; it is difficult for teachers to interpret the data and to provide timely intervention or assistance during students' collaborative discussions (Xing et al., 2015). Visual representation based on automatic analysis can translate learner-generated data into an accessible visible form that highlights important features, including commonalities and anomalies. Such analysis has been considered as a key to gaining insight into the learning process and providing a basis to better monitor and evaluate students' learning (Papamitsiou & Economides, 2015). Nevertheless, among the studies on knowledge elaboration, very little research has been conducted on visualization support based on automatic analysis.

Therefore, this study proposes an automatic analysis method to measure the level of groups' knowledge elaboration in terms of three indicators: coverage, activation, and equitability. Taking the method as a basis, an interactive web-based tool is developed to provide a multidimensional view of students' knowledge elaboration. The tool allows teachers to have an in-depth understanding of students' acquisition of the target knowledge in a convenient manner, enabling teachers to monitor student's discussion process and provide adequate feedback on students' learning.

Literature review

Knowledge elaboration and measurement

Knowledge elaboration positively affects knowledge acquisition, which is an important determinant of students' satisfaction and motivation (Draskovic et al., 2004). Many researchers have explored the significant roles of knowledge elaboration in online discussion. For instance, after comparing the elaboration differences in four different instructional approaches in multimedia learning environments, Eysink and de Jong (2012) concluded that elaboration is the key process explaining differences in learning outcomes. As for the factors affecting knowledge elaboration, Stegmann et al. (2012) found that the depth of learners' cognitive elaboration is positively related to both the domain-specific knowledge acquisition and the formal quality of their own argumentation. Paus et al. (2012) also confirmed that elaborating domain-specific concepts can activate processes conducive to learning and promote individual learning outcomes in online discourse.

To study the nature of online group learning, most researchers have adopted a qualitative approach to measure knowledge elaboration. Through discourse analysis, Daradoumis and Marques (2002) investigated how distribution of cognition is transformed and becomes common to all group members in an online collaborative problem-solving situation. Ding (2009) categorized messages in collaborative learning into three types, namely, off-task, on-task, and elaboration, and then further explored different process patterns of knowledge elaboration. Weinberger and Fischer (2006) measured the epistemic dimension of discourse by calculating the frequency of on-task discourse, which is differentiated into construction of problem space, conceptual space, and the relations between conceptual and problem space. Stegmann et al. (2012) operationalized the depth of knowledge elaboration as the duration of cognitive elaboration per proposition, which was calculated by the number of segments coded as cognitive elaboration. Additionally, Paus et al. (2012) differentiated on-task discourse into questions and explanations. They regarded the number of questions, explanations, and words of on-task discourse were regarded as three indicators of conceptual elaboration activities during online discourse.

Although coding schemes can be used to identify knowledge elaboration from different perspectives, researchers have pointed out that judging which messages exemplify which types of speech act is both ambiguous and subjective (Strijbos et al., 2006). Coding schemes are also blamed for ignoring domain knowledge construction, which runs counter to the definition of knowledge elaboration (Zheng et al., 2015). More importantly, because manual coding is a post-event method and results can only be acquired when students finish their collaborative activity, it cannot provide real-time feedback for teachers to monitor the learning process.

Using semiautomatic or automatic analysis methods, some attempts have been made to overcome the methodological limitations of manual coding. Zheng et al. (2015) adopted a graph theory to construct indicators to quantitatively measure the level of knowledge elaboration. Although the value of proposed indicators can be automatically calculated via their analytical tool, it still takes manual coding to segment information flow generated in collaborative learning processes. Hong and Scardamalia (2014) used key terms automatically extracted from different sets of notes to represent and measure group knowledge, which proved the validity and feasibility of using percentages and frequency of shared key terms to measure community knowledge. However, their study failed to reveal the connections among knowledge concepts proposed by the students.

Previous studies have indicated that the conceptual links that students construct in their discussions can reflect the level of their knowledge elaboration (Weinberger & Fischer 2006). More importantly, the connections among knowledge concepts are recognized as fundamental constituents of knowledge structure, acting as illustrative instruments providing access to the current knowledge and the representational state of a group. Therefore, identifying the conceptual links generated in students' discussions is a way to gain an in-depth understanding of students' conceptions about complex knowledge domains (Mcclure et al. 1999).

Visualizing online collaborative discussions

Although online collaborative discussion generates large corpora of discourses for teachers and researchers to observe how students process information, it still poses a challenge concerning how to analyze and make sense of these data (Law et al., 2011). Previous studies have demonstrated that visual representation has potential value in inducing a more direct observation of what is happening inside groups (Calvani et al., 2010), perceiving implicit aspects of raw data, supporting teachers' awareness of collaboration process (Papamitsiou & Economides, 2015), and leading to flexible instructional interventions (Donmez et al., 2005).

Various attempts have been made to provide visual tools that offer information for teachers to act upon in real time. Calvani et al. (2010) developed Forum Plus, a module for gathering the interaction data of each group and representing them on a radar graph to obtain an immediate idea of the group's effectiveness. Similarly, based on activity theory, Xing et al. (2015) designed a web-based visual tool to present the assessment results of groups' collaborative learning. Although these visual tools have potential value in easing teachers' burden of understanding the information, they cannot present procedural information on students' knowledge elaboration.

From a time-series perspective, Goggins et al. (2015) developed a visualization tool to show a process-oriented automatic formative assessment of group learning so that instructors can understand how group collaboration evolves and varies over time. Additionally, Law et al. (2011) identified indicators based a number of Knowledge Forum corpora and provided visualization of the time sequence of the coded discourses in thread structures over time. Juan et al. (2009) developed a tool which facilitates the automatic generation of weekly monitoring reports derived from data contained in server log files. These reports provide online instructors with visual information regarding students' and groups' activity, thus allowing more efficient monitoring of students' and groups' progress and performance in e-collaborative scenarios. However, most studies focused on students' superficial behavioral information instead of the quality and level of students' online discussion. Hong and Scardamalia (2014) constructed a key-term tag cloud to help visually represent the shared key terms in the students' discussions, yet simple consideration of the frequency of key terms could not convey a complete understanding of students' discussions. The authors advocated that this key-term tool be further enhanced to visualize collaborative learning processes and promising ideas, after which, they argued, it would be very useful for teachers and learners.

To overcome the aforementioned methodological limitations, this study aims to further explore the automatic approach to measure and visualize students' knowledge elaboration. This study was guided by the following two research questions:

* Are the proposed multi-dimensional indicators valid in measuring the levels of knowledge elaboration in student groups' collaborative discussion?

* Can the web-based visual tool provide a useful and easily handled representation of groups' knowledge elaboration levels so as to assist instructors in detecting potential problems in collaborative discussions?

Key-term-based automatic measurement of knowledge elaboration

Knowledge elaboration is a process of expanding and refining new information via organizing, structuring, and connecting the prior knowledge (Ding et al., 2011). In the course of elaborating knowledge, students not only construct a new understanding of their knowledge, but also enrich and expand their knowledge structures. The knowledge structure constructed by students is a representation of how they organize their knowledge concepts and identify the relations between them (Day et al., 2001; Engelmann & Hesse, 2011). It provides a valuable source of information that taps into both the content and organization of students' knowledge (Mcclure et al., 1999). Further, researchers have pointed out that examining students' knowledge structures in real time is an important means of measuring their levels of knowledge elaboration (Day et al., 2001; Kalyuga, 2009).

Evaluating students' knowledge structure in a specific task domain usually entails judgments about the similarity of the structure to the knowledge structure provided by experts. Since experts' organization and comprehension of domain knowledge closely approximate the true representation of that domain, the similarity to an established expert structure can be considered as an indicator for measuring the level of knowledge elaboration and acquisition (Day et al., 2001; Clariana et al., 2009). This similarity is often operationalized by calculating the ratio of the number of mutually shared concepts between the two structures divided by the total number of concepts (Hong & Scardamalia, 2014; Clariana et al., 2009; Xu et al., 2016; Goldsmith & Davenport, 1990). Higher frequency of the use of concepts concerning domain knowledge shows that students expends more effort in knowledge elaboration (Hong & Scardamalia, 2014).

Besides independent knowledge concepts, the linkages among the different knowledge concepts constructed by students are also critical for measuring their knowledge structures (Goldsmith & Davenport, 1990; Clariana et al., 2009). The number of mutually shared conceptual linkages between students' knowledge structures and experts' structures is associated with the level of students' knowledge elaboration (Zheng et al., 2015). Additionally, equitability has been used as an indicator representing whether all members participate to a similar degree without monopolizing behavior, in collaborative learning (Calvani et al., 2010; Li et al., 2007). Similarly, the equitability of discussion of the target knowledge concepts and linkages between them is also an important factor for examining the balance of groups' elaboration of the knowledge concerning the specific task domain.

Based on the aforementioned discussion of approaches and indicators to evaluate knowledge elaboration, this study proposes a multidimensional framework to quantify different facets of student groups' knowledge elaboration. Three main dimensions derived from the properties of knowledge structure were identified, namely, coverage, activation, and equitability.

Indicators to measure knowledge elaboration

Coverage

Coverage refers to the scope of students' discussion of the topic-specific knowledge. Higher coverage means that the students have fully elaborated their knowledge during the collaborative discussion process. Herein, coverage is examined by measuring the coverage of key terms (CKT) and the coverage of key terms linkages (CKTL). The two indicators can be calculated by formula (1) and formula (2) respectively.

CKT = [absolute value of [[union].sub.1[less than or equal to]j[less than or equal to]i[less than or equal to]N]]([T.sub.i] [intersection] [T.sub.j])]/K (1)

Where N denotes the total number of members in a group. Ti denotes the set of key terms mentioned by member i in all his/her posts, which can be represented as Ti = {term:, term2,...}. ([T.sub.i] [intersection] [T.sub.j]) denotes a set of the shared key terms discussed both by member i and member j. [[union].sub.1[less than or equal to]j[less than or equal to]i[less than or equal to]N] ([T.sub.i] [intersection] [T.sub.j]) indicates the set of shared key terms of a group, denoted as ST. K represents the total number of key terms provided by teachers.

CKTL = [absolute value of [[union].sub.1[less than or equal to]j[less than or equal to]i[less than or equal to]N]] ([R.sub.i] [intersection] [R.sub.j])]/L (2)

Where N denotes the number of members in a group. Ri denotes a set of the key-term linkages involved in all posts of member i, represented as [R.sub.i] = {<termi, term2>,<term3, term4,...>}. ([R.sub.i] [intersection] [R.sub.j]) denotes a set of the shared key-term linkages contributed both by member i and member j. [[union].sub.1[less than or equal to]j[less than or equal to]i[less than or equal to]N] ([R.sub.i] [intersection] [R.sub.j]) indicates the set of shared keyterm linkages of a group, denoted as SR. L denotes the number of linkages provided by teachers.

Activation

Activation represents the intensity of elaboration of target knowledge. Higher activation indicates that students show a deeper understanding of topic-specific knowledge in the process of collaboration. Herein, activation is captured through measuring the activation of key terms (AKT) and the activation of key-term linkages (AKTL). The two indicators can be calculated by formula (3) and formula (4), respectively.

AKT = [N.summation over (i = 1)][[absolute value of ST].summation over (t = 1)]F[T.sub.it]/K (3)

Where N denotes the total number of members in a group. ST indicates the set of shared key terms of a group. FTit denotes the frequency of key term k mentioned by member i in all his/her posts. Key term t belongs to ST. K denotes the total number of key terms provided by teachers.

AKTL = [N.summation over (i = 1)][[absolute value of ST].summation over (t = 1)]F[R.sub.it]/L (4)

Where N denotes the total number of members in a group. FRil denotes the frequency of key-term linkage l involved in all posts of member i. Key-term linkage I belongs to SR. L denotes the number of linkages provided by teachers.

Equitability

Equitability is an indicator to examine whether the target concepts are equally elaborated. It indicates whether or not the content of the student's discussion evenly covers the target knowledge. Herein, standard deviation is used to depict the degree of knowledge equitability. Lower standard deviation value indicates a more balanced discussion, while higher deviation value means that some target knowledge is processed intensively, but other knowledge may be neglected. In this study, equitability refers to the equitability of key terms (EKT) and the equitability of key-term linkages (EKTL). The two indicators can be calculated by formula (5) and formula (6) respectively.

EKT = AKT/[square root of 1/K[[absolute value of ST].summation over (t = 1)][(G[T.sub.t] - AT).sup.2]] (5)

Where GTt denotes the frequency of shared key term t mentioned in a group's discussion. AT denotes the average frequency of all the shared key terms mentioned in a group's discussion. The denominator is the standard deviation of the frequency of a group's key terms. In order to make the value of equitability of key terms be a positive value, we divide AKT by the standard deviation value.

EKTL = AKTL/[square root of 1/L[[absolute value of ST].summation over (t = 1)][(G[R.sub.t] - AR).sup.2]] (6)

Where GRl denotes the frequency of shared key-term linkage l mentioned in a group's discussion. AR denotes the average frequency of all the shared key-term linkages mentioned in a group's discussion. The denominator is the standard deviation of the frequency of a group's key-term linkages. In order to make the value of equitability of key-term linkages be a positive value, we divide AKTL by the standard deviation value.

Method of key-term extraction

Teachers are required to construct a knowledge structure via an authoring tool. The knowledge structure consists of concepts and linkages between concepts, which summarize the key terms expected to be discussed by students during collaborative learning activities. Taking the knowledge structure as a benchmark, students' discussion posts are processed to extract meaningful key terms and identify the linkages between key terms to examine students' knowledge elaboration.

Key terms were extracted and compared in the following steps. First, each student's posts were preprocessed by (a) splitting Chinese words using open-source software ICTCLAS (Zhang et al., 2003) and (b) replacing the similar key terms by referring to a list of synonyms. Second, we extracted the key terms according to the knowledge map provided by the teachers. Meanwhile, we identified the co-occurrence of any two key terms in one sentence. Clariana et al. (2009) reported that the sentence can be regard as a meaningful unit to measure the relationship between key terms, and herein we adopted this method to identify the linkage between key terms. Third, comparison between any two students' posts was performed to identify shared key terms and linkage between key terms within one group. Following that, the frequency of shared key terms and linkages between key terms was computed and recorded in an adjacent matrix. Based on the adjacent matrix, indicators were computed to determine the knowledge elaboration of a group. Figure 1 shows an illustrative example of the adjacent matrix.

In Figure 1, G[T.sub.i] (i=1,2,..n) denotes the frequency of shared key term [T.sub.i] in a group. G[R.sub.ij] (0<i<j<n+1) represents the frequency of shared linkage between [T.sub.i] and [T.sub.j]. If G[R.sub.ij] equals zero, it means that there is no linkage between [T.sub.i] and [T.sub.j].

A case study

Participants

Two cohorts of second-year undergraduate students at a comprehensive university in northern China participated in the study. Altogether, there were 157 students, of whom 105 were female and 52 were male. Students' age ranged from 18 to 20 (M = 19, SD = 0.78). All of the participants majored in educational technology and had a basic knowledge of computer programing. The participants were randomly assigned by the instructor to groups of five or six.

Context

The context of this study was a compulsory course named "Data Structure" and this 18-week course was delivered with a blended learning mode. Each week, instructors and students were scheduled to meet for a twohour face-to-face lesson in a traditional classroom and a two-hour online learning activity of programming in a computer laboratory. As for the programming activities in the computer laboratory, students were encouraged to communicate online about the given task, although they were able to meet face-to-face. Utilizing the Moodle platform (Cesareni et al., 2015) as a collaborative learning tool, the instructor designed four problem-solving tasks on different topics, namely collection bidding, cipher decoding, tour guiding, and campus planning.

Procedure

In each task, before the collaborative problem solving process, students were required to learn the materials by themselves and collect information related to task. Then, by clicking on a topic link, students could enter the topic page to view the discussion messages and post responses. Group members conducted online discussion within the Moodle forum until they came to a final solution. In order to achieve agreement, members of each group could ask questions, share ideas, and even debate with each other in Moodle. Students were required to post and edit their solutions of each task collaboratively on the wiki tool. The data collected for analysis in this research included information concerning what was said in each post, the time when the message was posted, and who participated in the discussion. Each task lasted two hours, and the number of posts generated in each task ranged from 19 to 167.

It should be noted that, to avoid potential interference, each group had an assigned space which was not accessible to other groups. The instructor provided no intervention except the description of the task and presentation of relevant learning materials.

Results

Indicator validation

Using the key-term extraction method mentioned above, we selected the discussion posts on the first activity as a sample to automatically compute the values of the six indicators. Table 1 shows values of six indicators. In the table, CKT represents coverage of key terms; AKT represents activation of key terms; EKT represents equitability of key terms; CKTL represents coverage of key-term linkages; AKTL represents activation of keyterm linkages; and EKTL represents equitability of key-term linkages.

Previous research has conducted model validation by comparing automatic analysis with experts' rating (e.g., Li et al., 2013; J. Zhang et al., 2007). Likewise, an alternative validation method was adopted in our study. We used human raters to evaluate groups' online discussion in terms of to what extent their discussion covers the target knowledge and contributes to the problem-solving, and compared this evaluation results with corresponding values of the automated analysis. Two instructors teaching data structures were asked to rate the quality of groups' online discussion into five levels. The five levels and their descriptions are shown in Table 2.

To evaluate the groups' online discussion, raters must browse all the posts that the groups posted. After each rater submitted his/her scores, we tested the reliability of the raters by looking at their inter-rater reliability. The Spearman's rho correlation coefficient was 0.755 (p < .01), which is sufficiently high.

In Table 3, we report the descriptive statistics for the full sample. QOD represents the quality of groups' online discussion by the raters. The analysis indicated that CKT (r = 0.54, p = .002), AKT (r = 0.63, p = .000), CKTL (r = 0.48, p = .006), AKTL (r = 0.64, p = .000) and EKTL(r = 0.36, p = .046) were positively correlated with QOD. However EKT was not significantly correlated with QOD. This implies that five indicators, namely CKT, AKT, CKTL, AKTL and EKTL, can effectively measure students' levels of knowledge elaboration in online learning communities.

Visualization of knowledge elaboration

A visualization tool was developed to assist teachers in interpreting the procedural information regarding students' knowledge elaboration. Taking activity 1 as an example, Figures 2-5 show the general interface and basic functions of this visualization tool. This web-based tool allows teachers to monitor students' discussions from anywhere and at any time.

Figure 2 shows the general picture of group knowledge elaboration on the five indicators using radar charts. For each radar chart, the blue layer represents the average value of the five indicators among all groups, and the green layer shows the value of the five indicators of the selected group. The wider the green layer expands, the higher the knowledge elaboration level of the selected group. In addition, using check boxes, teachers are able, not only to observe the level of knowledge elaboration of a particular group, but also to compare the knowledge elaboration of different groups.

Figure 3 shows the mapping between the knowledge terms elaborated by Group 5 and the target key terms. With the check boxes, teachers can select any groups to see to what extent the target key terms are covered in their discussion. The black node (collection) in the middle of the figure indicates the task topic. The green nodes represent the key terms which have already been discussed by the selected group, whereas the red ones stand for the target key terms that have not been dealt with. Additionally, the key-term linkages that have already been established by the group are shown with a thick line. The thin lines between nodes represent key-term linkages that have not been built by the group.

Figure 4 shows the knowledge activation of each key term of the selected five groups. On the left side of the figure, there is a list of key terms, which contains all target knowledge terms for the task. Using the check boxes, teachers can choose any key terms to examine to what extent they are activated in the discussion. The selected key terms are listed on the x axis, and the groups are displayed on the y axis. The number in each cell indicates the frequency of an individual term elaborated by a certain group. Additionally, the different frequencies are represented by a color gradient, where, the colors represent activation levels (low, average, high).

Figure 5 shows groups' knowledge elaboration equitability on target key terms. The blue column represents the equitability of key terms. As the figure suggests, compared with other groups, Group 9 has the most balanced elaboration in terms of the target knowledge in their collaborative discussion, and other groups, Group 13 in particular, may neglect some target knowledge necessary for solving the problem.

Furthermore, in order to assist a teacher having a direct observation of how groups' discussion content changes over time, our tool provides graphs from a time-series perspective. As Figure 6 shows, with the check boxes, the tool allows teachers to select, combine, and sequence different key terms. The x axis shows the discussion time, and the y axis shows the selected key terms. The dots with different colors on the timeline demonstrate the keys terms elaborated over time. More importantly, by selecting key terms that are relevant to a subtask of the whole problem-solving task, teachers can understand a group's discussion process on that specific subtask. Further, when the teacher hovers over a node, the graph will automatically present more detailed information, such as the name of the student who mentioned the key term, the time this key term appeared, and the full message of the post. In this way, teachers can directly analyze the content of students' posts without looking through them in the forum.

Discussion

Content analysis has been frequently employed to examine knowledge elaboration in online discussions. To avoid the subjective and time-consuming disadvantages of manual coding, a few attempts have been made to measure group knowledge through automated analysis of the number and frequency of key terms (Zheng et al., 2015; Xing et al., 2015). Likewise, the present study adopted an automated method to examine knowledge elaboration based on six indicators. According to the results of computational experiments, we found that five of the proposed indicators are significantly effective in measuring the level of knowledge elaboration in online learning discussions. The case study showed that groups with similar numbers of posts differed widely in knowledge coverage, activation, and equitability. For example, although the number of posts of Group 2 was almost equal to that of Group 5, there was a significant difference between them in all the five indicators of knowledge elaboration measured in the task. This provides insights for teachers to have a comprehensive and accurate judgment of the quality of students' discussions, enabling teachers to offer adequate instruction. More importantly, it also confirmed the importance of including the quality of textual content as an indicator when analyzing collaborative discussions (Hakkinen, 2013).

Following Hong and Scardamalia (2014), who found that an automated key-term analysis method failed to address the connection between individual concepts, the current study takes the relationship between key terms into consideration so that further in-depth measurement can be made to investigate group knowledge elaboration in terms of group knowledge structure and the mapping between group knowledge and domain knowledge. This case study revealed that the relationship between the terms constructed by students' discussions was not equivalent to the domain knowledge structure in terms of either depth or width, though the knowledge terms discussed by student groups displayed wide coverage and deep activation of the domain knowledge. This may suggest that teachers should offer sufficient scaffolds for students to form meaningful linkages among isolated concepts to develop a more complete knowledge structure. Thus, our results point to the beneficial potential of examining the connection between ideas when measuring knowledge elaboration in group discussions (Weinberger & Fischer, 2006; Van et al., 2000).

Despite the advantages of automated analysis, studies have noted that teachers may find it hard to infer meaning and make sense of the results without a technical or analytical background, especially when the number of students is large (Xing et al., 2015). Visual representations of student-generated trace data during learning activities could help teachers explain them intuitively and understand hidden aspects of these data quickly(Papamitsiou & Economides, 2015). The visual tool employed in this study made it possible for instructors to easily obtain information on students' knowledge elaboration in online group discussions. As can be observed from the radar graphs, indicator values of Group 2 were notably above those of Group 5, which revealed that Group 2 is superior to Group 5 in knowledge elaboration. Further, the value of activation of key terms of Group 2 was higher than average, indicating that this group conducted an in-depth discussion of the key domain knowledge. In addition, this visual tool offers the freedom for tutors to select key terms that they think are important to identify how well they are elaborated by the students. For instance, as shown in the knowledge activation graph, selected concepts like "stack" and "queue" are more widely-discussed than "Pushstack" and "Stackfull." This may imply that, with the help of the visualization tool, teachers can easily conduct more focused monitoring of students' online discussions and provide better-targeted feedback.

Finally, because there is a fundamental problem with using traditional manual coding to demonstrate the dynamic processes of knowledge elaboration (Strijbos et al., 2006; Zheng et al., 2015), this study took a timeseries perspective in the visualization tool that can assist instructors in understanding how group discussions develop and vary over time (Goggins et al., 2015). This visualization tool can capture the processual nature of knowledge elaboration, making it accessible for teachers to observe the change of the terms discussed by a selected group over a given time. For instance, in this case study, the time-series visualization figure compared Group one's and Group five's elaborations of the six key terms, which are the knowledge required by the problem solving task. It showed that Group 5 displayed low intensity of elaboration, as the dots were scattered in a more sparse and incoherent way. This means that the discussion of Group 5 was irrelevant to the subtask in that period. In this way, teachers can easily observe the time when students start a subtask-specific discussion and monitor whether a group's discussion is off-topic or not. The tool also helps to understand how the key terms are collaboratively elaborated over time to form a progressively more coherent idea for addressing a problem (Papamitsiou & Economides, 2015).

Conclusions

This study proposes a process-oriented, automatic method to analyze knowledge elaboration in collaborative online learning discussions. Five indicators employed in this study were found to be effective in measuring student groups' knowledge elaboration in terms of knowledge coverage, activation, and equitability. Adopting a natural language processing approach, this automated key-term analysis not only revealed students' elaboration of the concepts of specific themes, but also identified the linkages among different concepts. Compared with traditional content analysis based on manual coding, the automatic method increases the reliability and consistency of the results. The automatic method also shows potential value in improving the effectiveness of online instruction, especially for increased class sizes, because it is able to deal with large datasets produced in online learning. Moreover, a user-friendly, web-based visualization tool was developed for the teachers to monitor and evaluate students' knowledge elaboration processes, enabling teachers to provide in-time scaffolding and feedback. This will be especially beneficial for teachers who do not have adequate knowledge and skills for processing transcripts of online discussions, or when they are facing a large number of groups in online learning environments.

One limitation of this study is that it does not take into account the weighting of different key terms, which may play different roles in the problem-solving tasks. Another limitation is that the visualization tool is not available for students to self-monitor their knowledge elaboration processes. For future studies, more attempts should examine knowledge elaboration by addressing the weighting of different key terms in solving the target problem so that the key-term-based automatic method can better measure and monitor students' knowledge elaboration in collaborative discussion. From a technological perspective, further studies should also be conducted to explore knowledge elaboration through semantic classification of the discourse data generated from large-scale online collaborative learning environments. It is also advised to refine the visual tool to make it accessible for learners as individuals and as members of a collaborative group to reflect on their own knowledge elaboration processes. Additionally, with the visual information, this tool can be used as a meta-cognitive tool that enhances students' self-regulated learning in the online learning environment.

Acknowledgments

This research work is supported by the National Social Science Foundation of China "A big-data-based empirical study on analysis, evaluation and intervention in CSCL" (NO: BCA170077).

References

Anderson, J. R. (1983). A Spreading activation theory of memory. Journal of Verbal Learning & Verbal Behavior, 22(3), 261295.

Calvani, A., Fini, A., Molino, M., & Ranieri, M. (2010). Visualizing and monitoring effective interactions in online collaborative groups. British Journal of Educational Technology, 41(2), 213-226.

Cesareni, D., Cacciamani, S., & Fujita, N. (2015). Role taking and knowledge building in a blended university course. International Journal of Computer-Supported Collaborative Learning, 11(1), 9-39.

Clariana, R. B., Wallace, P. E., & Godshalk, V. M. (2009). Deriving and measuring group knowledge structure from essays: The Effects of anaphoric reference. Educational Technology Research and Development, 57(6), 725-737.

Donmez, P., Rose, C., Stegmann, K., Weinberger, A., & Fischer, F. (2005). Supporting CSCL with automatic corpus analysis technology. In T. Koschmann, D. Suthers, & T.-W. Chan (Eds.), Proceedings of 2005 conference on Computer support for collaborative learning--CSCL 2005 (pp.125-134). Taipei, Taiwan: Erlbaum.

Daradoumis, T., & Marques, J. M. (2002). Distributed cognition in the context of virtual collaborative learning. Journal of Interactive Learning Research, 13(1), 135-148.

Day, E. A., Arthur, W., & Gettman, D. (2001). Knowledge structure and the acquisition of complex skills. Journal of Applied Psychology, 86(5), 1022-1033.

Denessen, E., Veenman, S., Dobbelsteen, J., & Van Schilt, J. (2008). Dyad composition effects on cognitive elaboration and student achievement. The Journal of Experimental Education, 76(4), 363-386.

Ding, N. (2009). Visualizing the sequential process of knowledge elaboration in computer-supported collaborative problem solving. Computers & Education, 52(2), 509-519.

Ding, N., Bosker, R. J., & Harskamp, E. G. (2011). Exploring gender and gender pairing in the knowledge elaboration processes of students using computer-supported collaborative learning. Computers & Education, 56(2), 325-336.

Draskovic, I., Holdrinet, R., Bulte, J., Bolhuis, S., & Van Leeuwe, J. (2004). Modeling small group learning. Instructional Science, 32(6), 447-473.

Engelmann, T., & Hesse, F. W. (2011). Fostering sharing of unshared knowledge by having access to the collaborators' metaknowledge structures. Computers in Human Behavior, 27(6), 2078-2087.

Eryilmaz, E., Pol, J. V. D., Ryan, T., Clark, P. M., & Mary, J. (2013). Enhancing student knowledge acquisition from online learning conversations. International Journal of Computer-Supported Collaborative Learning, 8(1), 113-144.

Eysink, T. H. S., & de Jong, T. (2012). Does instructional approach matter? How elaboration plays a crucial role in multimedia learning. Journal of the Learning Sciences, 21(4), 583-625.

Gleaves, A., & Walker, C. (2013). Richness, redundancy or relational salience? A Comparison of the effect of textual and aural feedback modes on knowledge elaboration in higher education student' work. Computers & Education, 62, 249-261.

Goggins, S., Xing, W., Chen, X., Chen, B., & Wadholm, B. (2015). Learning analytics at "small" scale: Exploring a complexity-grounded model for assessment automation. Journal of Universal Computer Science, 21(1), 66-92.

Golanics, J. D., & Nussbaum, E. M. (2008). Enhancing online collaborative argumentation through question elaboration and goal instructions. Journal of Computer Assisted Learning, 24(3), 167-180.

Goldsmith, T. E., & Davenport, D. M. (1990). Assessing structural similarity of graphs. In R. Schvaneveldt (Ed.), Pathfinder associative networks: Studies in knowledge organization (pp. 75-87). Norwood, NJ: Ablex.

Hakkinen, P. (2013). Multiphase method for analysing online discussions. Journal of Computer Assisted Learning, 29(6), 547-555.

Hong, H.-Y., & Scardamalia, M. (2014). Community knowledge assessment in a knowledge building environment. Computers & Education, 71, 279-288.

Hwang, W. Y., Chen, N. S., Dung, J. J., & Yang, Y. L. (2007). Multiple representation skills and creativity effects on mathematical problem solving using a multimedia whiteboard system. Educational Technology & Society, 10(2), 191-212.

Juan, A. A., Daradoumis, T., Faulin, J., & Xhafa, F. (2009). SAMOS: A Model for monitoring students' and groups' activity in collaborative e-learning. International Journal of Learning Technology, 4(1/2), 53-72.

Kalyuga, S. (2009). Knowledge elaboration: A Cognitive load perspective. Learning & Instruction, 19(5), 402-410.

Kumar, V. S., Gress, C. L. Z., Hadwin, A. F., & Winne, P. H. (2010). Assessing process in CSCL: An Ontological approach. Computers in Human Behavior, 26(5), 825-834.

Law, N., Yuen, J., Wong, W. O. W., & Jing, L. (2011). Understanding learners' knowledge building trajectory through visualizations of multiple automated analyses. In S. Puntambekar, G. Erkens & C. Hmelo-Silver (Eds.), Analyzing Interactions in CSCL. Springer US. Analyzing Interactions in CSCL: Methodologies, Approaches and Issues (pp. 47-82). New York, NY: Springer.

Li, Y., Dong, M., & Huang, R. (2009). Toward a semantic forum for active collaborative learning. Educational Technology & Society, 12(4), 71-86.

Li, Y., Liao, J., Wang, J., & Huang, R. (2007, July). CSCL interaction analysis for assessing knowledge building outcomes: method and tool. Paper presented at the 8th International Conference on Computer supported collaborative learning, New Brunswick, NJ.

Li, Y., Ma, S., Zhang, Y., Huang, R., & Kinshuk. (2013). An Improved mix framework for opinion leader identification in online learning communities. Knowledge-Based Systems, 43, 43-51.

Mcclure, J. R., Sonak, B., & Suen, H. K. (1999). Concept map assessment of classroom learning: Reliability, validity, and logistical practicality. Journal of Research in Science Teaching, 36(4), 475-492.

Papamitsiou, Z., & Economides, A. A. (2015). Temporal learning analytics visualizations for increasing awareness during assessment. RUSC. Universities & Knowledge Society Journal, 12(3), 129-147.

Paus, E., Werner, C. S., & Jucks, R. (2012). Learning through online peer discourse: Structural equation modeling points to the role of discourse activities in individual understanding. Computers & Education, 58(4), 1127-1137.

Pilkington, R. (2001). Analysing educational dialogue interaction: Towards models that support learning. International Journal of Artificial Intelligence in Education, 12, 1-7.

Reigeluth, C. M., Merrill, M. D., Wilson, B. G., & Spiller, R. T. (1980). The Elaboration theory of instruction: A Model for sequencing and synthesizing instruction. Instructional Science, 9(3), 195-219.

Stark, R., Mandl, H., Gruber, H., & Renkl, A. (2002). Conditions and effects of example elaboration. Learning & Instruction, 12(1), 39-60.

Stegmann, K., Wecker, C., Weinberger, A., & Fischer, F. (2012). Collaborative argumentation and cognitive elaboration in a computer-supported collaborative learning environment. Instructional Science, 40(2), 297-323.

Strijbos, J. W., Martens, R. L., Prins, F. J., & Jochems, W. M. G. (2006). Content analysis: What are they talking about? Computers & Education, 46(1), 29-48.

Van, B. C., Van, d. L. J., & Kanselaar, G. (2000). Collaborative learning tasks and the elaboration of conceptual knowledge. Learning & Instruction, 10(4), 311-330.

Weinberger, A., & Fischer, F. (2006). A Framework to analyze argumentative knowledge construction in computer-supported collaborative learning. Computers & Education, 46(1), 71-95.

Xing, W., Wadholm, R., Petakovic, E., & Goggins, S. (2015). Group learning assessment: Developing a theory-informed analytics. Journal of Educational Technology & Society, 18(2), 110-128.

Xu, C., Zheng, Y., Hu, H., & Li, Y. (2016). Exploring individual contributions during online collaborative discussions. Journal of Computers in Education, 3(4), 395-411.

Zhang, H. P., Liu, Q., Cheng, X. Q., Zhang, H., & Yu, H. K. (2003, July). Chinese lexical analysis using hierarchical hidden Markov model. Paper presented at the second SIGHAN workshop on Chinese language processing, Sapporo, Japan.

Zhang, J., Ackerman, M. S., & Adamic, L. (2007, May). Expertise networks in online communities: Structure and algorithms. Paper presented at the 16th international conference on World Wide Web, Banff, Canada.

Zheng, L., Huang, R., Hwang, G. J., & Yang, K. (2015). Measuring knowledge elaboration based on a computer-assisted knowledge map analytical approach to collaborative learning. Journal of Educational Technology & Society, 18(1), 321-336.

Yafeng Zheng (1,2), Chang Xu (1), Yanyan Li (1) *, You Su (1)

(1) School of Educational Technology, Smart Learning Institute, Beijing Normal University, Beijing, China // (2) School of Computer and Information Engineering, Henan University of Economics and Law, Zhengzhou, China // zlzyf@126.com // merryxc_1991@163.com // liyy@bnu.edu.cn // suyou@bupt.edu.cn

* Corresponding author

(Submitted June 17, 2016; Revised October 17, 2016; Accepted November 15, 2016)

Caption: Figure 2. Overall views of five indicators

Caption: Figure 3. Visualization of knowledge coverage

Caption: Figure 4. Visualization of knowledge activation

Caption: Figure 5. Visualization of knowledge equitability

Caption: Figure 6. Visualization of time series of group discussion process
Table 1. Results of knowledge elaboration in activity 1

Groups    CKT    AKT    EKT    CKTL   AKTL   EKTL

Group 1   0.79   4.54   0.82   0.50   0.17   0.84
Group 2   0.83   5.25   1.08   0.65   0.20   1.10
Group 3   0.67   2.91   0.92   0.15   0.11   0.43
Group 4   0.38   1.12   0.51   0.19   0.04   0.49
Group 5   0.54   0.95   0.84   0.15   0.04   0.40

Table 2. Five levels for evaluating groups' discussion

Level   Description

5       Fully covers the target knowledge; finds the solution to the
          task
4       Covers the majority of the target knowledge; finds the task
          solution but some details are inaccurate
3       Covers some of the target knowledge; presents task solution
          with noticeable errors
2       Covers only a very limited range of the target knowledge;
          fails to find the task solution
1       Fails to address the task; answer is completely unrelated to
          the task

Table 3. Descriptive statistics and Spearman correlations for the full
sample

          CKT    AKT       EKT    CKTL      AKTL      EKTL     QOD
QOD       0.54   0.63 **   0.25   0.48 **   0.64 **   0.36 *   --
          **
Mean      0.56   3.05      0.71   0.25      0.20      0.47     3.35
SD        0.14   1.40      0.18   0.17      0.23      0.25     0.92
Minimum   0.18   0.55      0.79   0         0.10      0.36     1
Maximum   0.83   5.57      1.08   0.65      1.08      1.10     4.5

Note. * p < .05; ** p < .01.

Figure 1. Illustrative example of an adjacent matrix

            [T.sub.1]     [T.sub.2]     [T.sub.3]

[T.sub.1]   G[T.sub.1]
[T.sub.2]   G[T.sub.12]   G[T.sub.2]
[T.sub.3]   G[R.sub.13]   G[R.sub.23]   G[T.sub.3]
[T.sub.4]   G[R.sub.14]   G[R.sub.24]   G[R.sub.34]
[T.sub.5]   G[R.sub.15]   G[R.sub.25]   G[R.sub.35]
***         ***           ***           ***
[T.sub.n]   G[R.sub.1n]   G[R.sub.2n]   G[R.sub.3n]

            [T.sub.4]   [T.sub.5]       ***   [T.sub.n]

[T.sub.1]
[T.sub.2]
[T.sub.3]
[T.sub.4]   G[T.sub.4]
[T.sub.5]   G[R.sub.45]   G[T.sub.5]
***         ***           ***           ***
[T.sub.n]   G[R.sub.4n]   G[R.sub.5n]         G[T.sub.n]
COPYRIGHT 2018 International Forum of Educational Technology & Society
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2018 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Author:Zheng, Yafeng; Xu, Chang; Li, Yanyan; Su, You
Publication:Educational Technology & Society
Article Type:Case study
Geographic Code:9CHIN
Date:Jan 1, 2018
Words:7212
Previous Article:Feedback of Interface Agents on Student Perception: Level, Dialogue, and Emotion.
Next Article:The Virtues of Taiwanese Internet-Using Adolescents: The Development and Validation of the Cyber Virtues Scale.
Topics:

Terms of use | Privacy policy | Copyright © 2021 Farlex, Inc. | Feedback | For webmasters