A review of research methods for assessing content of computer-mediated discussion forums.

Computer-mediated communication (CMC) technologies are becoming increasingly important components of online educational environments (Romiszowski & Mason, 2004). The online discussion forum that is a significant component of many web-based courses is one type of CMC. Instructors and students rely on these asynchronous forums to engage one another in ways that replace face-to-face communication. The goal of such interactions is to promote critical thinking, meaningful problem solving, and knowledge construction (Kirschner, Strijbos, Kreijns, & Beers, 2004). In spite of the importance of these forums, the most common methods for assessing the content and outcomes of these forums have often been limited to frequency counts and other quantitative measures (Mason, 1992; Romiszowski & Mason, 2004). In order to assess any meaning that results from these discussions, it is necessary to perform some kind of semantic analysis of them. This article provides an overview of current methods of quantitative and qualitative research paradigms for analyzing the content of asynchronous computer-mediated discussion forums. For each analysis method we describe the research method, provide an example of a research study that used that method, describe the types of questions that this method can address, and compare the methods in terms of validity and reliability. Researchers and instructors will be able to use this analysis to become familiar with the choices available and make decisions about appropriate methods for analyzing CMC forums.

**********

In discussing online learning, Harasim (1989) describes interactivity as the most striking characteristic of CMC and the factor with the greatest potential to impact learning. Carswell, Thomas, Petre, Price and Richards' (2000) study comparing undergraduate students in Internet and face-to-face sections of computer science course provides support for this claim. Learning results were similar for both groups, but the CMC group experienced increased interactions with fellow students and tutors. Similarly, Garrison, Anderson, and Archer (2001) describe the importance in online learning of creating a virtual community of inquiry which allows learners to construct experiences and knowledge through analysis of the subject matter, questioning, and challenging assumptions. In a face-to-face environment, this kind of reflection is often accomplished via synchronous, interactive discussions and problem-solving sessions. Web-based learning courses must rely on online discussion forums to create these interactions.

Traditionally, educational communications have been grounded in an oral, face-to-face (F2F) tradition. Garrison et al. (2001) describe such communications as being fast-paced, spontaneous, and often less structured than written communications. When working in groups, participants perceive F2F communication as more effective and satisfying than computer-mediated group work (Olaniran, Savate, & Sorenson, 1996) mainly because F2F groups can more easily spend time on such tasks as clarifying and defining responsibilities (Warkentin, Sayeed, & Hightower, 1997). Further, when implemented in properly moderated discussions, oral communication has been shown to support the development of critical thinking skills (Garrison et al., 2001).

One significant way F2F and asynchronous communications differ is in terms of the strategies used to manage the conversation. For instance Condon and Cech (1996) found that in CMC, participants who were trying to increase communication efficiency and decrease typing requirements also decreased the use of unneeded elaborative statements and repetitions. Compared to F2F problem solving sessions, Jonassen and Kwon (2001) found that computer mediated discussion contained significantly fewer off-task postings. This finding confirms Garrison et al.'s (2001) description of written communications (as used in online discussions) as being leaner because many of the non-verbal signals present in face-to-face oral communication are missing (e.g., body language, pauses). However, even without the richness of the discussions found in F2F discussions, McCreary posits that the value of written communication such as used in online discussion forums comes from the necessity of exactness, organization of thought, and clear expression (1990).

Being text-based, the online forum, provides a unique opportunity to conduct learning research as these forums have the potential to make thinking and reasoning visible both to participants and researchers (Asbell-Clarke & Foster, 2004). Further, even though many acknowledge the critical role CMC discussion forums can play in web-based courses, little empirical evidence confirms that text-based communication used in computer conferencing can facilitate higher-order and critical thinking.

Although CMC is described as being leaner than face-to-face, CMC has the ability to promote knowledge construction and meaningful learning. Asynchronous CMC may promote knowledge construction because online environments emphasize student-centered interaction through collaboration with other students. It is through these interactions that learners construct their knowledge and can learn more effectively (Romiszowski & Mason, 2004). There is limited evidence that CMC can facilitate higher-order thinking (Garrison, Anderson, & Archer, 2000). However earlier research provides more positive predictions of the potential impact of online forums on meaningful learning. Specifically, these studies indicated that participation in online forums could lead to broader and deeper participation in group activities (Kiesler, Siefel, & McGuire, 1984; Pullinger, 1986; Spitzer, 1989, as cited in Mazur, 2004). More recently, Jonassen and Kwon (2001) reported that during group problem-solving activities, the CMC participants produced fewer, but more task-related messages than a F2F group. In addition, the CMC group's decision-making patterns were more sophisticated than that of the F2F group. In this article, we argue that one reason for the relatively small number of studies addressing meaningful learning via online discussion forums is the lack of proven research paradigms in this domain.

An early exception to CMC evaluation models was Henri's work (1992) that examined the quality of online postings by focusing on four dimensions--social (e.g., "I'm feeling great today"), interactive (statements that refer to other postings), metacognitive (statements about reasoning), and cognitive. The cognitive dimension is broken down into five types of reasoning skills: elementary clarification, in-depth clarification, inference, judgment, and strategies. But this portion of the model goes further. Specifically, Henri emphasizes that it is insufficient to simply examine the skills demonstrated in message content, but rather one must look for evidence of the level of information processing. Unlike prior schemes, Henri's model defines not only the types of skills and interactions demonstrated in online postings, but also attempts to qualitatively define the nature and content of online interactions that evidence cognitive development and meaningful learning. Henri's model has provided researchers with a structure for many ensuing qualitative analysis methods.

As this summary describes, the online forum is seen as an important if not crucial component of online courses. Reeves, Herrington and Oliver (2004) describe the forum's potentially critical role in creating meaningful online collaborative learning environments. Reeves et al. (2004) call for new research directions to provide both researchers and practitioners with better guidelines on how to effectively implement meaningful online learning environments. Given the centrality of the forum, both researchers and practitioners have a stake in examining the characteristics and content of online forum as one component in research and evaluation efforts regarding the impact and effect of participating in online learning activities.

As we show in this article, the methods for assessing online discussion forums have evolved as have the underlying assumptions associated with these methods. Because researchers and practitioners are seeking to answer questions associated with participation and content of online forums, we provide an overview of the current analysis tools and methods available to them. In the following sections describing methods for conducting such analysis, we provide examples of research that use each method and then compare the methods relative to the research questions each is best suited to answer.

CMC Discussion Research Methods

This review of methods for analyzing computer-mediated discussion forums focuses only on methods that use the actual CMC forum content as the direct source of data to be analyzed (e.g., content analysis); thus methodologies that use self-report data about the CMC forum are not included (1). CMC forum analysis methods can be categorized as: quantitative (e.g., descriptive statistics), qualitative content analysis, user-classified forum content via constrained CMC forums, and sequential analysis that combines both quantitative and qualitative methods. Although not intended to be a comprehensive treatment, for each category we review predominantly used methods.

Quantitative--Descriptive Statistics

In her review of evaluation methodologies of computer conferencing, Mason (1992) reports that the most often used evaluation method for online discussion forums was "computer-generated statistics about logons, messages sent and read, levels of participation and number and length of entries" (p. 112). Early CMC researchers relied upon the computer's ability to record frequencies and interactions for their data (Romiszowski & Mason, 2004).

Many CMC interactions now take place within course management software packages such as Blackboard or WebCT that provide access to quantitative data easily. According to the edu-tools website (http://www.edu-tools.info/course/compare/index.jsp) that provides product comparisons of online course delivery systems, many systems offer student tracking features that allow instructors to track student logons and their usage of course materials, however, providing descriptive statistics of discussion forum usage is not even listed as a comparison feature. A further search revealed that some tools, such as FirstClass (www.firstclass.com) do produce log files containing data about the activity of individual participants. However, their web site does not provide details on the nature of these files. Further, new tools that support constrained or scaffolded discussion forums--which are forums that enforce a structure on the nature on participants' contributions (Jonassen & Remidez, 2005)--are often accompanied with more sophisticated tools for analyzing the quantitative and qualitative nature of those contributions. One such product is Knowledge Forum (www.knowledgeforum.com) that provides a scaffolded discussion forum environment. Although it does not produce participant data, OISE (Ontario Institute for Studies in Education), has produced an online Analytical Toolkit that supports such statistical analysis.

In her review of computer tools that can contribute to analyzing online forums, Mazur (2004) briefly describes two internal tracking tools that are available as inexpensive internet downloads, that may have as yet untapped potential for quantitatively documenting online talk-in-interaction. Both the AXS tracking utility (http://www.xav.com/scripts/axs) and Active Server Page's (ASP) databasing function (http://www.2enetworx.com/dev/projects/statcountex.asp) can provide rather sophisticated tracking of hits by users on web pages. Although neither of these specifically provides statistics for a discussion forum, data about per student hits on particular discussion forum pages can provide an indirect indication of forum usage (e.g., their reading patterns) even when they are not necessarily posting messages.

Although tools included in popular course management software packages do not provide descriptive data, the following metrics are potentially useful. Mason (1992) lists the number of messages sent and read, levels of participation (although this phrase is not defined), and the number and length of entries for each discussion forum. The following list attempts to provide a more expansive and defined set of metrics that builds upon Mason's work:

* number of postings per student for overall forum;

* number of times each posting is read;

* total depth per thread; in Figure 1, the "Facilitating comment on Lewis & Key posting" thread has a depth of two (the initial posting and replies only to that initial posting). In contrast the "Facilitating for Team 2" thread has a depth of three--the initial posting plus replies that extend two levels beyond the initial posting.

* postings per student listed by depth of postings--that is the number of postings at depth level 1--n where n is the number of the most embedded posting. For instance in Figure 1, student R. Key has one posting at level 1 (line number 2) and one posting at level 2 (line 6b).

The metrics are directed towards threaded discussion forums--that is an online dialog or conversation that takes the form of a series of hierarchically linked messages. The thread series is created over time as users read and reply to existing messages. Typically, messages in a given thread share a common subject line and are linked to each other in the order of their creation. Once gathered, these metrics can be used in appropriate statistical analyses.

Thus Mason's level of participation could be defined as a combination of number of postings, depth of postings and reading patterns for a particular forum. The actual way an instructor or researcher define this metric, we believe, should depend on the purpose of participation in the forum and what type of learning the instructor hopes to promote via forum participation. For instance, if the forum were designed to have a student team create an instructional design solution for a case study, then it would be important for all team members to participate meaningfully in the forum. In that case the instructor would most likely want to include number of postings per student as well as an indicator of the depth of postings per student to define level of participation. If on the other hand, the forum is designed to simply be a place to post questions about course readings or assignments, then level of participation may simply be defined in terms of student's reading of that forum.

Qualitative Content Analysis

In her discussion of CMC evaluation methods, Mason (1992) noted "the most obvious data available to conferencing evaluators--the transcript of the conference interactions--is paradoxically the least used" (p. 113). Content analysis refers any process that is a systematic, "replicable technique for compressing many words of text into fewer content categories based on explicit rules of coding" (Krippendorf, 1980; Stemler, 2001).

For this discussion we examine two CMC discussion forum analysis protocols designed to ascertain the presence of meaningful thinking in an online discussion (in contrast to content acquisition)--the Interaction Analysis Model (IAM) (Gunawardena, Lowe, & Anderson, 1997) and Newman, Webb and Cochrane's (1996; Newman, Webb, Johnson, & Cochrane, 1997) critical thinking model. Both models are based upon some of the most rigorous work in this young field--the IAM being based upon Henri (1992) and the Newman et al. scheme being based upon earlier work from Garrison (1992). The IAM is a strong choice as it is one of the few CMC models with an existing, albeit small, research base.

Interaction Analysis Model. As mentioned previously, Henri (1992) proposed one of the earliest content analysis schemes for online discussion forums. Henri's work was significant as it focused on the actual meaning of individual and combined posting rather than examining descriptive statistics or patterns of postings. Similarly, Garrison et al. (2000) proposed a model for characterizing high-quality online interactions that contained three elements: social presence, teacher presence, and cognitive presence. Gunawardena et al. (1997) continued down this path of seeking evidence of knowledge building or cognition in online forums. Their proposed model acknowledges Henri's framework but identifies the model's basis in a teacher-centered learning paradigm as a weakness. In response, Gunawardena et al. proposed a content analysis model based on a constructivist paradigm designed to detect evidence of knowledge construction. Their Interaction Analysis Model (IAM) was developed in an attempt to further understand and describe the processes of negotiating meaning and knowledge co-construction in a collaborative online discussion environment (Gunawardena et al., 1997). The researchers used the transcripts of a multi-week online debate to develop a model that posits five phases learners must move through (not necessarily sequentially) as knowledge is being constructed. Each phase is described in Figure 2 (2); phase names are shown in italics and operations associated with that phase follow. We note that since the IAM was proposed, researchers (Kirschner et al., 2004) have developed and applied a similar scheme that proposes four codes (contribution, verification, clarification and elaboration) for evidence of negotiating for common ground.

To apply the IAM, coders read each posting and apply the most applicable phase or phases from the IAM. In many cases, coders may apply a single phase to multiple sentences, or a paragraph or two (Gunawardena et al., 1997; Marra, Moore & Klimczek, 2004). Depending on the nature of the forum, postings may contain multiple paragraphs and address several topics, and thus two or more phases may be applied to a single posting. In the IAM coding example in Figure 3, the first two paragraphs provide further detail on points that have already been made in prior postings; thus they represent elaboration on an agreed upon point which is the essence of Phase I (see Figure 2). The last paragraph is Phase III, negotiation of meaning or knowledge co-construction, where students build on prior postings in order to construct the new idea that "Spaulding is not getting the whole picture" because no one has addressed student background skill, class organization, and so forth.

Results of coded transcripts are categorized into the five phases to provide a distribution that indicates the degree of knowledge construction activities. Marra et al. (2004) conducted inter-rater reliability checks in their use of the IAM. To do so, they adopted the convention based upon prior research using this protocol (Beaudrie, 2000) to use the most advanced phase from each posting as the basis for inter-rater checks. For example, the posting in Figure 3 would be considered an overall Phase III because this is the most advanced phase applied to this posting. Hence, the unit of meaning became the entire posting, rather than phrases, sentences, or paragraphs within the posting. Because the highest levels of advancement of knowledge construction are directly implied by the most advanced phases, this convention seems both logical and viable. Marra et al. (2004) then resolved a portion of their inter-rater differences following a process based on Chi's (1997) recommended process for resolving discrepancies between coders.

Newman, Webb and Cochrane Model. The Newman, Webb, and Cochrane (1996; Newman, Webb, Johnson, & Cochrane, 1997) content analysis model is designed to measure critical thinking. This protocol is based on Garrison's (1992) 5-stage critical thinking model: problem 1) identification, 2) definition, 3) exploration, 4) evaluation, and 5) integration. These researchers identified approximately 40 codes in categories such as relevance, justification, novelty, and ambiguities, each with a "+" or "-" appended to indicate whether the coded statement contributes to ("+") or detracts ("-") from critical thinking development. A complete list of the code categories is provided in Figure 4, followed by a sample of the expanded coding categories (e.g., the N--novelty--code includes five sub codes that can either be used as a positive or negative indicator.

In contrast to the IAM, where the unit of analysis varies from phrases, sentences or the entire posting, for the Newman, et al. (1996) model, codes are applied at the sentence or phrase level. Newman et al. (1996) indicate that coders should insert the applicable code(s) at the beginning (e.g., "<O+", applying outside knowledge to the problem) and the end (e.g., "+O>) of the unit of analysis. Figure 5 shows the Newman coding scheme applied to the same posting coded with the IAM in Figure 3. Marra et al. (2004) report using the OC+ code because the student was referring to course material, the case text, to make his or her point. Other codes were applied in a similar manner.

The Newman et al. (1996) model includes a formula for calculating a "critical thinking ratio" (CT) based on the frequencies of plus (+) or minus (-) codes for each letter category. CT = (x+ - x-) / (x+ + x-), where x is a letter-labeled category such as J or N, followed by either a plus or a minus sign. For each coding category (e.g., "J", justification), one tallies the number of positive (x+ in the above formula) and negative (x-) contributions, and then calculates the ratio shown above. This produces a measure that is independent of the quantity of participation, reflecting only the quality of the messages. Ratios for an individual category may range from a -1 (all uncritical, all surface) to +1 (all critical, all deep). For example, if in an online discussion there were a total of 29 "J+" codes and 11 "J-" codes, the critical thinking ratio for the justification category would be 0.16.

User-Classified Forum Content--Constrained CMC Forums. The content analysis methods just described require that researchers code CMC forum content. Constrained or scaffolded discussion forums are pre-coded forms of conversation systems that require participants to select from a set of pre-labeled postings that are defined by pre-defined message types. For example, Jonassen and Remidez (2005) constrained discussions about a problem by restricting the first level responses to solutions. Responses to solutions were required to be a Reason to Support, Reason to Reject, or Modify Proposal. Any responses to those warrants had to provide evidence, such as Information or Facts, Personal Opinion or Belief, Research Findings, and so on. This is in contrast to the standard, non-constrained forum where users may label their messages with a subject heading (see Figure 1) but there are no enforceable rules that govern their participation. The rules that govern participation in a constrained forum are designed to encourage certain types of thinking by the participants. The most commonly used constraint system supports argumentation, however, a limitless combination of rhetorical structures are possible.

Figure 6 depicts a table of contents of a constrained discussion forum where pre-service teacher education students were discussing a classroom management case study. The forum participants used message types that corresponded to a formal argument in order to develop argumentation skills that were hypothesized to enable solving ill-structured problems (Oh, 2004). Message types included hypothesizing cause ("What is the cause?"), solution generation ("What should I do about it?"), research evidence ("Research shows ..." or, "A scholar says ..."), experiential evidence ("My experience is ..." or, "I believe ..."), elaboration ("Your explanation is unclear; can you be more specific?"), and rebuttal ("I don't agree because ..."). This study used a package called FLE3 (http://www.FLE3.com), however several software packages support constrained discussion forums, including KIE (Bell & Linn, 1997), CaMILE (Guzdial, 1995), Belvedere (Suthers, & Hundhausen, 2001), and CSILE (Scardamalia & Bereiter, 1994).

In addition to allowing discussion monitors to quickly examine the proportions of postings for each message type and encourage different types of postings as needed, constrained discussion tools enable analysis of forums because the users in effect, code the messages. The list of codes provides readily accessible and useful data for analyzing the structure of the conversation and answering questions of interest. For instance, what proportions of the contributions are at each level? What are the proportions of different warrants or different kinds of evidence? Jonassen and Remidez (2005) report a discussion where students supported warrants using only personal opinion or experiences. This would provide a monitor with an excellent opportunity to coach the discussion participants towards using more robust types of evidence.

[FIGURE 6 OMITTED]

Constrained discussion can support different kinds of reasoning that in turn support more meaningful tasks. Cho and Jonassen (2002) found that students using an argumentation scaffold produced significantly more arguments during group discussions (especially claims and grounds), and that students solving ill-structured problems produced more arguments during group discussion (especially rebuttals and backings) than students without the scaffold/constraint. When the contributions were reanalyzed for problem-solving function, they found that students using the conversation scaffold provided more comments related to problem definition, orientation comments, criteria development, solution development, solution approval, and solution critique. In a transfer problem, students who had been in the scaffolded groups created higher quality argumentation better problem solutions on their individual problem-solving tasks. Using a constraint-based argumentation scaffold positively affected the ability of groups to collaboratively construct arguments in an online environment.

Combined Quantitative/Qualitative--Mapping/Sequential Analysis. Sequential analysis methods for examining discussion board content are designed to provide a view of discussion participation not offered by either the statistical metrics or content analysis methods. Sequential analysis methods are used to identify patterns in interactions within a domain of interactions (Bakeman & Gottman, 1997)--in this case within discussions. Proponents of sequential analysis argue that content analysis methods do not provide data on the relationships between messages nor how message sequences affect subsequent postings and ultimately the learning outcomes of the discussion (Jeong, 2003).

Jeong (2003) describes a method and software program--Discussion Analysis Tool (DAT)--designed to help researchers examine and measure student interactions and sequences of interactions that may lead to critical thinking outcomes in threaded discussions. DAT computes "transitional probabilities between critical thinking events providing the basis for measuring and describing the relationship between threaded messages and student interaction" (Jeong, 2003, p. 28). An additional feature is that the probabilities generated by DAT can be converted into graphical depictions that provide what may be a useful visual representation of student interactions.

Jeong (2003) bases DAT on the theory of dialogism (Bakhtin, 1981) that views language as a part of a larger social context where potential word meanings interact and possibly impact future meanings or interpretations. This theory proposes that critical thinking and meaning are produced by the relationships of one series of statements to another, and further that meanings are affected by conflict that may occur in the interactions. Although not described in the source material in these terms, this conflict serves to produce cognitive dissonance that can then drive inquiry, reflection and articulation of understandings, evidence and assumptions. Jeong (2003) emphasizes that a key assumption of this theory as operationalized in analyzing online discourse is that one cannot analyze individual postings but rather must look at the relationships and transitions between postings. This is the basis of DAT.

Jeong (2003) defines an interaction as a given discussion posting and the responding message. His methodology for using DAT essentially builds on content analysis by applying a coding scheme consisting of twelve categories (position statements, simple agreements or disagreements, arguments, personal experiences, literature, formal data, personal or hypothetical actions and choices, evaluation or critiquing of arguments, summary, negotiation or conclusions, and process comments) to one of the course debates. The coding results were used as input to DAT where it "identified and followed the links between messages" (Jeong, 2003, p. 30). DAT is capable of producing the following types of results.

* Tables of relative frequencies of one event being following by another. Table 1 is adapted from Jeong (2003) and shows the relative frequencies of one type of message being followed by another type. For instance, a "disagreement" message was followed by an Agreement 38% of the time and by another disagreement 13% of the time. These probabilities are based the total number of disagreements observed and the total number of responses to those initial disagreements. Jeong clearly acknowledges that one must balance one's interpretations of these results given the small cell sizes (e.g., n = 5).

* A transitional state diagram as pictured in Figure 7 is a visual illustration of the flow of events from a discussion where the circles show different "events" or codes from the discussion and the arrows depict the transitional probabilities between codes.

Using these data, researchers are able to combine content analysis results with quantitative methods and perform such analyses as tests of significance (using z-scores) for interaction sequences that occur at higher probabilities than other sequences of interactions.

[FIGURE 7 OMITTED]

METHODS

Having described a variety of research methods used for analyzing online discussion forum contents, we provide a comparison of the methods in terms of the research questions they are suited for addressing, and their validity and reliability.

Research Questions and Results

Each method will produce different types of results and thus will be able to answer different research questions. The results for the quantitative--descriptive statistics methods are simply the numeric metrics that one has chosen to gather (e.g., total number of posting, number of postings per participant, thread depth, etc.). Typical results when using the IAM (Gunawardena et al., 1997) include a report of the percentages of codings or postings that fell into each of the five phases (e.g., 21% in Phase I,... 5% in Phase V). Researchers are then likely to discuss what overall portion of the postings fell into phases III and above since these are the phases that constitute knowledge construction in some manner. When applied, the Newman, et al. (1996) protocol produces a series of CT (critical thinking) ratios for the forty-some scheme codes. According to the scheme, ratios that approach one are the highest indicators of critical thinking. Constrained forums produce results in a form similar to the IAM--a set of categorized postings for the overall forum that can then be analyzed for differences in the numbers of different types of postings. One notes, however, that the means to these similar results are quite different in that the constrained forums are self-coded by the individuals creating the postings. This will be addressed in the discussion of reliability and validity. Sequential analysis as exemplified by DAT for CMC discussion boards produces percentage frequencies of event sequences (see Table 1), Z-scores for event sequences which can be used to test significant differences between frequencies of sequences and a visual representation of sequence transitions (see Figure 7).

Given these types of results, Table 2 lists the research questions that each method is most suited to address. The two content analysis methods provide the opportunity to answer more substantive questions regarding the actual "content" of the discussion postings in comparison to the descriptive statistics /metrics method. The relative frequency of their usage in the past few years (Mazur, 2004; Rourke, Anderson, Garrison & Archer, 2000) provide evidence about the popularity of these methods with researchers as they become more interested in questions pertaining to the kinds of thinking evidenced by the actually content of the discussion forums (Marra et al., 2004).

As mentioned previously, the constrained forums also produce a set of categorized postings. This provides the basis for checking for differences (potentially statistical) between different posting categories. The constrained forums, however, offer a different potential type of analysis related to the underlying meaning of the rules created for the different posting types. In studies from Jonassen and Remedez (2005) and Oh (2004), students' postings types were governed by the rules for developing an argument (e.g., warrants, evidence). Both of these studies were constructed with the intent of not only creating a meaningful CMC discussion experience but also to help participants to develop argumentation skills. The analysis of the forum itself can address the first issue, but to determine if the experience has impacted students' ability to argue effectively, then researchers must engage students in a transfer task. In both studies, solving problems was the task that students were engaged in. In order to engender meaningful discussion, the topic or purpose of the discussion must be meaningful.

Lastly, the method used by DAT, can clearly be combined with the coded results of content analysis coding schemes or the constrained CMC forums. One could clearly code a discussion using the IAM and then apply DAT's sequential analysis methodology--thus allowing one to answer questions not only about the types of knowledge construction evidenced in the data but also regarding sequences of interactions. This combination of methods would seem to produce the most powerful research paradigm for analyzing both message content as well as the impact of message flows on future messages.

Jeong (2003) argues that content analysis cannot be used to "examine the relationships between threaded messages and how message sequence and group processes affect subsequent discussion and cognitive outcomes" (p. 397). If researchers are interested in understanding how the relationships between messages impact group processes and interactions, then content analysis alone may not be enough. The type of interaction analysis proposed by Jeong (2003) however does address these relationships.

We agree with this position when comparing Jeong's research method with a content analysis scheme such as the one proposed by Newman, et al. (1996). The Newman protocol requires that one code statements in isolation of one another and results in a set of individual frequencies--individual by the very nature of the code definitions. The IAM however, does to some degree take into account the overall flow of postings and their impact on one another as each code is based upon the degree to which knowledge is constructed, and to assess knowledge construction one must consider each posting in the context of the overall flow of postings. Also, if during coding, researchers applied and used all applicable IAM phases for every single posting, researchers would then be able to address questions concerning the progression of knowledge construction development per posting or per individual student.

Reliability and Validity

Given that we are examining methods for analyzing the text-based content of discussion forums, methods such as classical test theory (Crocker & Algina, 1986) that are used to tune and analyze traditional forced response items do not apply. However, reliability and validity still apply to these methods and should be of great interest to researchers (Rourke et al., 2001). Given the diverse nature the methods we are addressing, we can expect to see diverse issues concerning reliability and validity. Not surprisingly, the issues are the most clear-cut for the quantitative descriptive statistics/metrics method.

To review, reliability refers to the extent to which a measure produces the same results when applied under the same circumstances. Reliability is an indication of consistency of measures across time. The types of measures included in the quantitative descriptive statistics category are by their very nature reliable when counting, and calculations are performed with accuracy and when definitions for concepts such as depth are well understood and applied consistently. To ensure reliability, researchers should re-check calculations or when using automated tools, check for the accuracy of the input data. In addition to simple counting and arithmetic errors, the greatest possibilities for inconsistencies may occur from individuals' differing decisions about which postings to count (e.g., do we count the posting that simply said, "I didn't mean to submit my prior posting"). Once such decisions are made, this methodology should be highly reliable. The authors note that the addition of these metrics to commonly used discussion software would not only ensure reliability but also make these metrics more readily available.

The validity of these metrics is less well defined. Validity refers to whether a measure, in fact, measures what it purports to. For the quantitative descriptive statistics/metrics method, content validity is the most relevant concern. When taken at face value from their statistical definitions, these measures are highly valid. For instance, the tally of the total number postings in a forum is a valid indicator of just that--the total number of forum postings. Similarly unremarkable statements can be made of such metrics as number of postings per participant per forum, average postings per participant, and so forth. These metrics are not open to much interpretation and as long as researchers do not attempt to attribute more meaning to them than they have, then the metrics are valid.

Validity issues may emerge when one attributes more meaning to a metric, beyond the simple definition of the metric. For instance, if one were to tally the per participant posting frequencies for a given forum and then conclude that the top three participants engaged in the most meaningful forum participation, then one would question the validity of using the posting frequencies as an indicator of meaningful participation. Similarly, it is most likely not valid to say that CMC forum A with more postings than a forum B is a more successful forum--unless of course one is willing to define successful based only upon total number of postings. Certainly the metrics that fall under this method category can provide a useful and valid set of parameters to describe certain aspects of the overall forum--however, their validity is questionable in any application that exceeds the simple definition of the metric being considered.

As with almost any qualitative methodology, the reliability issues associated with content analysis methods are centered on the ability of more than one rater to consistently apply the coding scheme (Rourke et al., 2000). As described earlier, the IAM researchers often apply a single phase for a given posting (Beudrie, 2000; Marra et al., 2004). Applying the protocol at the per posting level makes it easier to use an inter-rater reliability procedure. Using a process recommended by Chi (1997), the coders can compare their ratings for each posting, discuss, resolve differences when possible and calculate a percentage of coding agreement. However, as noted by Rourke et al. (2000) in their review of methodological issues associated with content analysis methods, such inter-rater reliability figures are criticized because they do not account for chance agreements amongst raters. To address this issue, researchers can apply the Cohen's kappa statistic (Cohen, 1960) where a minimal acceptable level of reliability is 60% (cf. Landis & Koch, 1977).

One of the disadvantages of applying the IAM coding scheme consistently (both between raters and for a single rater) is that the high level definitions of the phases provided by Gunawardena, et al. (1997) and the lack of coding examples make it difficult for coders to decide which codes to apply and when. Marra, et al. (2004) reported that they purposefully contextualized the IAM to their own forum in order to reduce the ambiguity of the scheme. Their forum required students to offer solutions to an instructional design case study, so the researchers jointly agreed upon definitions of each IAM phase that applied to this task.

The reliability of the Newman et al. (1996) scheme presents different issues. In contrast to the IAM, the Newman, Webb and Cochrane scheme is straightforward to apply. The Newman et al. codes are very focused in definition. For instance, the JP + and P+ codes are defined as providing proof or examples, and relating possible solutions to familiar situations, respectively. Although one can find some ambiguity in almost any statement, relative to the IAM broad phase definitions, these codes' definitions are specific and easy to understand. However, Marra et al. (2004) point to the necessity of choosing amongst the large number of potentially relevant codes to apply to a text passage as being a significant reliability issue. As shown in Figure 3, there are approximately 40 codes available for the scheme and given that the codes are intended to be used at the phrase, sentence or paragraph level, it is possible to use many codes for a single posting. Further, the assumptions regarding a standardized unit of analysis and mutually exclusive categories for inter-rater reliability algorithms (neither of which apply to the Newman et al. model) (Marra et al., 2004; Stemler, 2001), makes it impossible to compute an agreement coefficient for the Newman model results.

The reliability issues introduced by an unclear unit of analysis may be addressed by recent work from Strijbos, Martens, Prins, and Jochems (2006). In their review of CSCL literature they found that many content analysis studies reported some type of inter-rater reliability but did not discuss what unit of analysis or meaning was used during coding. To address this problem, Strijbos et al. (2006) created and tested rules for defining the unit of meaning that should be applied before coding so that each rater uses a uniform content segmentation. They applied five rules to guide the segmentation of the email message content they were coding including:

  an email message is a fixed unit and the message order is ignored, the
  salutation and close will be ignored, the unit of analysis is the unit
  of meaning, a post-script is considered to be a separate unit of
  meaning, and in case of a summation all points sharing a single
  meaning are considered one 'unit of meaning' (p. 9).

We acknowledge that the rules for defining the unit of analysis may need to vary depending on the nature of the content and task being analyzed but argue that the unit should be clearly defined and the amount of overlap between coded segments should be minimized and quantified in order to increase reliability.

The IAM's validity hinges on whether the results produced when the protocol is reliably applied are in fact an indication of the degree to which knowledge construction occurred in the forum. The IAM was developed through a grounded theory process by which the researchers, with a thoroughly explained theoretical basis from social constructivism (Jonassen, Mayes and McAleese, 1993; Pea, 1993; Vygotsky, 1978), applied their knowledge of these theories to an extensive discussion forum with the purpose of attempting to develop a framework for describing knowledge construction via a coding system. Through their preparations they had made themselves aware of the processes that theoreticians had hypothesized occurred during the knowledge construction process--and they then applied this knowledge to the forum looking in particular for ways of describing how the process of negotiation of meaning occurred.

Although we do not have a definitive answer to the question of the IAM's validity, at face value the definitions of the phases taken from Gunawardena et al.'s (1997) original work, were indeed derived from commonly used descriptions of the types of activities that take place during knowledge construction activities. Additionally, the model was derived via a rigorous qualitative process for developing grounded theory (Strauss, 1990). Lastly, in this relatively young field of CMC content analysis, numerous researchers (Beaudrie, 2000; Jeong, 2001; Marra et al., 2004) have successfully applied the model and provided support for its validity.

The validity of the Newman et al. (1996) protocol is questionable given the concerns about its reliability. Although the CT ratios are easy to calculate, one must question how to meaningfully interpret them. While a researcher can purportedly tell if critical thinking activities were evidenced by CT ratio sign and magnitude (-1.0 to +1.0), one is still left with a disparate set of ratios that is difficult to interpret for several reasons. First, the variability in the unit of analysis calls into question the meaning of a set of CT ratios. For instance, what does it mean if 70% of the ratios are positive for a discussion forum? Referring to the methods Newman et al. (1996) recommend for applying the protocol, one cannot conclude that 70% of the analyzed text contributed to critical thinking as each ratio is based on coding frequencies that may refer to a phrase, sentence, paragraph or even an entire posting. In applying this protocol, raters may also be challenged to consistently use all of the numerous codes available to them, and further may wonder how densely to apply codes to a particular segment (e.g., should one use all eight codes that apply, or only the most obvious?). All of these procedural difficulties bring the validity of the coding frequencies and the resulting CT ratios into some question.

The reliability and validity issues associated with the user categorized constrained forums center around the fact that the users themselves do the coding or labeling of the messages. Reliability is largely a question of consistency of results. Constrained discussion forum packages such as FLE3 allow the forum administrator to build in "help" or prompts for each message type. These prompts, such as the example pictured in Figure 6 are intended to both help the user choose the correct message type and then create message content that is in alignment with that type. However, even with such prompts, the only way to know if users are categorizing consistently is for a moderator to actually check participant category usage for consistency.

The validity of the user categorizations in constrained discussion forums depends on how accurately users label their messages. In other words, are users' self-categorizations in alignment with the intention/meaning of the defined message types? For many potential reasons including lack of training or practice, an incorrect understanding of the message types or simple carelessness a user may label a message of type hypothesize cause (see Figure 6), but the content may actually be a problem identification. Without critically examining the content of each message relative to the message label, researchers have no assurance that the messages are validly coded.

Another validity issue for these constrained forums concerns the content validity or meaning of the suite of message types. The message types in Figure 6 are designed to help discussion participants practice and develop the ability to construct an argument. Thus those that implement such a forum posit that when posters successfully apply the suite of message types in a way that reflects a valid argument, these users have demonstrated their ability to create an argument. The potential validity issue lies in whether a user's ability to apply this set of message types is indeed a representation of that user's ability to construct an argument--or do whatever task the application of the suite of message types is designed to promote. To address this issue participants must engage in a separate task from the constrained forum participation that also requires this same skill--in our example, to construct an argument. Oh (2004) did this in his study by requiring students not only to engage in the discussion forum using a suite of message types designed to promote argumentation skills, but also had them construct an argument-based response to a separate case-study.

Ultimately, when users self-categorize their postings as is done in constrained forums, one cannot ensure that these categorizations will be either accurate or consistent. CMC forum facilitator should work with users both before (in practice exercises) and during forums to coach them to use categorizations consistently and accurately. The authors are not aware of any research to show that this has been done but given that such coaching activities are effective in other domains we can speculate they would be effective for this activity as well. Prior research using constrained forums is limited and does not report having checked users' categorizations for consistency of application or accuracy.

For DAT because it is algorithmic, the reliability can be assumed to be nearly perfect--assuming that the underlying programming is sound and that the same data is provided as input for the tool. The validity of DAT is directly tied to the validity of the coding scheme that provides the data on which DAT operates. As the saying goes (at least in the United States), "garbage in, garbage out." If the coding of the discussion forum is not conducted using a reliable and valid coding scheme, then the interaction sequences and other data produced by DAT will of course not be valid.

CONCLUSIONS

Reeves et al. (2004) recently called for a renewed focus on conducting development research in the context of online learning environments with a focus on generating design guidelines for practitioners. Given the criticality of asynchronous discussion board forums in online learning environments, conducting such research in these venues could contribute significantly to this agenda. Until recently researchers have generally limited their analyses of these forums to quantitative reports of frequencies; this paradigm did little to answer meaningful questions or create more informed designs. As other methods of coding forum content emerge it is important for researchers to understand when and how these methods can be used, and the strengths and weaknesses of these methods. This article has provided an overview of a representative sampling of current methods of quantitative and qualitative research paradigms for analyzing the content of online discussion forums.

As summarized in Table 2, each method is appropriate for addressing different types of research questions. Further, each method offers its own unique strengths and weaknesses. On one end of the spectrum, one has the option of using quantitative descriptive statistics and metrics to describe discussion board forum participation. This method has the advantage of being easy to implement and highly reliable--however, it is somewhat limited in terms of the types of research questions it can address. On the other end, one has sequential analysis methods such as DAT (Jeong, 2003)--which builds on content analysis methods and provides quantitative data on discussion interaction sequences. Although the results from the software program DAT are completely reliable and the method has the potential to answer more complex questions regarding discussion boards, the method implicitly inherits all the reliability and validity issues of the content analysis protocol that it uses to do the initial coding.

Given these differing advantages and disadvantages, researchers need to choose an analysis methods based upon what questions they wish to answer, and their ability to carry out the chosen methodology in the fashion intended (e.g., does my research team have the qualitative methods expertise to conduct a content analysis?). Lastly research questions should be in alignment with the learning outcomes or objectives of the discussion forum itself. Although the relative power of the IAM may be appealing, does it make sense to look for evidence of knowledge construction in a discussion forum whose purpose is to post examples of student work? In this case, simple quantitative metrics might be best.

The methods described in this article provide researchers with an effective array of choices for answering questions about a critical aspect of online courses--the online discussion forum. Although none of the methods are perfect in terms of their applicability to all situations (nor should they be, we would argue) or in terms of their validity and reliability, all the methods reviewed--when applied appropriately--can provide researchers with effective tools for describing online discussions. If such research is indeed carried out, the community of scholars examining online learning can look forward to an effective research agenda that will ultimately positively impact our abilities to design and implement effective online learning experiences.

References

Asbell-Clarke, J., & Foster, J. (2004). Theoretical frameworks for online learning environments, TERC publications. Retrieved May 10, 2004, from http://www.terc.edu.

Bakeman, R. & Gottman, J. (1997). Observing interaction: An introduction to sequential analysis. Cambridge, MA: Cambridge University Press.

Beaudrie, B. P. (2000). Analysis of group problem-solving tasks in a geometry course for teachers using computer-mediated conferencing. Unpublished Ed.D. thesis, Montana State University, Bozeman.

Bell, M. C., & Linn, P.(1977). Scientific arguments as learning artifacts: Designing for learning on the web. Paper presented at AERA, Chicago, IL.

Carswell, L., Thomas, P., Petre, M., Price, B., & Richards, M. (2000). Distance education via the Internet: The student experience. British Journal of Educational Technology, 31(1), 29-46.

Chi, M. T. H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences, 6(3), 271-315.

Cho, K. L., & Jonassen, D. H. (2002). The effects of argumentation scaffolds on argumentation and problem solving. Educational Technology: Research & Development, 50(3), 5-22.

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(10), 37-46.

Condon, S. L, & Cech, C. G. (1996). Discourse management strategies in face-to-face and computer mediated decision making interaction. Electronic Journal of communication/LA Revue Electronique de Communication, 6(3).

Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Forth Worth, TX: Harcourt Brace Jovanovich College Publishers.

Cumming, S., & Ono, T. (1997). Discourse and grammar. In T. Van Dijk, (Ed). Discourse as structure and process, pp. 257-291. Thousand Oaks, CA: Sage

Garrison, D. R. (1992). Critical thinking and self-directed learning in adult education: an analysis of responsibility and control issues. Adult Education Quarterly, 42(3), 136-148.

Garrison, D. R., Anderson, T., & Archer, W. (2000). Critical inquiry in a text-based environment: computer conferencing in higher education. The Internet and Higher Education, 2(2-3), 87-105.

Garrison, D. R., Anderson, T., & Archer, W. (2001). Critical thinking, cognitive presence, and computer conferencing in distance education. American Journal of Distance Education, 15(1), 7-23.

Gunawardena, C. N., Lowe, C. A., & Anderson, T. (1997). Analysis of global online debate and the development of an interaction analysis model for examining social construction of knowledge in computer conferencing. Journal of Educational Computing Research, 17(4), 397-431.

Guzdial (1995). Software-realized scaffolding to facilitate programming for science learning. Interactive Learning Environments, 3(4): 1-44.

Harasim, L. M. (1989). Online education as a new domain. In R. Mason & A. R. Kaye (Eds.), Mindweave: Communication, computers and distance education (pp. 50-62). Oxford: Pergamon Press.

Henri, F. (1992). Computer conferencing and content analysis. In Designing Environments for Constructivist Learning A. R. Kaye (Ed.), Collaborative Learning Through Computer Conferencing (pp. 117-136). Berlin: Springer-Verlag.

Jeong, A. (2003). Sequential analysis of group interaction and critical thinking in online threaded discussions. The American Journal of Distance Education, 17(1), 25-43. Retrieved February 17, 2004 from http://bbproject.tripod.com/SequentialAnalysis_Jeong2003.pdf.

Jonassen, D. H., Mayes, T. & McAleese, R. (1993). A Manifesto for a constructivist approach to technology in higher education. In T. Duffy, J. Lowycky, & D. Jonassen (Eds.). Berlin: Springer-Verlag.

Jonassen, D. H., & Kwon, H. (2001). Communication patterns in computer mediated versus face-to-face group problem solving. Educational Technology Research and Development, 49(1), 1042-1629.

Jonassen, D. H., & Remidez, H. (2005). Mapping alternative discourse structures onto computer conferences. International Journal of Knowledge and Learning, 1(1/2), 113-129.

Kiesler, S., Siefel, J., & McGuire, T. (1984). Social psychological aspects of computer-mediated communication. American Psychologist, 39, 1123-1134.

Kirschner, P., Strijbos, J., Kreijns, K., & Beers, P. (2004). Designing electronic collaborative learning environments. Educational Technology Research and Development, 52(3), 47-66.

Krippendorf, K. (1980). Content analysis: An introduction to its methodology. Vol. 5. The Sage COMMTEXT Series, Beverly Hills: Sage Publications.

Landis, J., & Koch, G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159-174.

Marra, R. M., Moore, J., & Klimczek, A. (2004) A comparative analysis of content analysis protocols for online discussion forums. Educational Technology Research and Development, 52(2), 23-40.

Mason, R. (1992). Evaluation methodologies for computer conferencing applications. In A. R. Kaye (Ed.), Collaborative Learning Through Computer Conferencing (pp. 105-116). Berlin: Springer-Verlag.

Mazur, J. (2004). Conversation analysis for educational technologists: theoretical and methodological issues for researching the structures, processes and meaning of online talk. In D.H. Jonassen (Ed.), Handbook for Research in Educational Communications and Technology, 2nd Edition. Mahwah, NJ: Lawrence Erlbaum Associates.

McCreary, E. K. (1990). Three behavioral models for computer-mediated communication. In L. M. Harasim (Ed.), Online education: Perspectives on a new environment (pp. 117-130). New York: Praeger.

Newman, D. R., Webb, B., & Cochrane, C. (1996). A content analysis method to measure critical thinking in face-to-face and computer supported group learning. Retrieved February 15, 2005, from http://www.qub.ac.uk/mgt/papers/methods/contpap.html.

Newman, D. R., Johnson, C., Webb, B., & Cochrane, C. (1997). Evaluating the quality of learning in computer supported co-operative learning. Journal of the American Society for Information Science, 48(6), 484-495.

Oh, S. (2004). The role of an argumentation scaffolding tool and epistemological beliefs on solving problems in online discussion. Unpublished doctoral dissertation, University of Missouri, Columbia.

Olaniran, B. A., Savage, G. T., & Sorenson, R. L. (1996). Experimental and experiential approaches to teaching face-to-face and computer-mediated group discussion. Communication Education, 45, 244-259.

Pea, R. (1993). Practices of distributed intelligence and designs for education. In G. Salomon (Ed.), Distributed Cognitions. Cambridge, MA: Cambridge University Press.

Pullinger, D. J. (1986). Chit-chat to electronic journals. Computer conferencing supports scientific communication. In V. Arms (Ed.) IEEE Transactions of Professional Communications, PC29, 30-33.

Reeves, T., Herrington, J., & Oliver, R. (2004). A development research agenda for online collaborative learning. Educational Technology Research and Development, 52(4), 53-65.

Romiszowski, A., & Mason, R. (2004). Computer-mediated communication. In D.H. Jonassen (Ed.), Handbook for Research in Educational Communications and Technology, 2nd Edition. Mahwah, NJ: Lawrence Erlbaum Associates.

Rourke, L., Anderson, T., Garrison, R., & Archer, W. (2001). Methodological issues in the content analysis of computer conference transcripts, International Journal of Artificial Intelligence in Education, 12(10), 8-22.

Scardamalia, M., & Bereiter, C.(1994). Computer support for knowledge-building communities. The Journal of the Learning Sciences, 3(3), 265-283.

Stemler, S. (2001). An overview of content analysis. Practical Assessment, Research & Evaluation, 7(17). Retrieved February 17, 2005 from http://PAREonline.net/getvn.asp?v=7&n=17.

Strauss, A. (1990). Basics of qualitative research: Grounded theory procedures and techniques. Thousand Oaks, CA: Sage Publications.

Strijbos, J. W., Martens, R. L., Prins, F. J., & Jochems, W. M. G. (2006). Content analysis: What are they talking about? Computers & Education 46(1), 29-48.

Suthers, D. D., & Hundhausen, C. D. (2001). Learning by constructing collaborative representations: An empirical comparison of three alternatives. In P. Dillenbourg, A. Eurelings, & K. Hakkarainen (Eds.), European perspectives on computer-supported collaborative learning (pp. 577-584). Universiteit Masstricht.

Vygotsky, L. (1978). Mind in society: The development of higher psychological processes. Cambridge, MA: Harvard University Press.

Warkentin, M. E., Sayeed, L., & Hightower, R. (1997). Virtual teams versus face-to-face teams: An exploratory study of a web-based conference system. Decision Sciences, 28, 975-996.

Notes

(1) The reader is referred to Romiszowski and Mason (2004) for a treatment of these methodologies relative to CMC.

(2) Note that the original work further defines each phase. For instance, Phase II included further definitions for sub-phases A-C, and for Phase III there are sub-phases A-E.

ROSE MARRA

University of Missouri, USA

[e-mail]

Title                            Poster      Date

1. Week 6 Hernandez              R. Marra    02/26/01
2. Re Hernandez                  R. Key      02/26/01
3. Re: Hernandez                 M. Barks    02/27/01
4. Question #2 Hernandez         C. Lewis    02/26
5. Facilitating comment          RMM         02/27/01
   on Lewis & KEY posting
a. RE: Question #2 Hernandez     M Bark      02/27/01
6. Facilitating for Team 2:      S. Hartman  02/27/01
   Question #2 Hernandez Case
a. RE: Facilitating for Team 2:  M. Barks    02/28/01
   Question #2 Hernandez Case
b. RE: Question #2 Hernandez     R. Key      02/28/01
7. Facilitating for team 2       S. Hartman  02/28/01
a. RE: Facilitating for team 2   RMM         03/02/01
i. RE: Facilitating for team 2   B. Bolton   03/02/01
b. RE: Question #2 Hernandez     D. Gilman   03/01/01

Figure 1. Sample illustration of threaded, non-constrained discussion
postings

Phase I. Sharing/comparing of information: statement of observation or
         opinion; agreement between participants.
Phase II. Discovery/exploration of dissonance/inconsistency amongst
          participants: identifying areas of disagreement; asking and
          answering questions to clarify disagreement.
Phase III. Negotiation of meaning/knowledge co-construction: negotiating
           meaning of terms and negotiation of the relative weight to be
           used for various arguments.
Phase IV. Testing/modification: testing the proposed new knowledge
          against existing cognitive schema, personal experience or
          other sources.
Phase V. Phrasing of agreement and applications of newly constructed
         meaning: summarizing agreement and metacognitive statements
         that show new knowledge construction.

Figure 2. IAM Phase definitions

There might be a counterpoint here.

1. Hernandez is touted as a 'super teacher', and has awards for her
   graduate experience, she has been out of the classroom for 10 years.
   Having worked with the best equipment and labs in the Aerospace
   industry, she may be unprepared to determine how this atmosphere can
   be translated to the classroom
2. While Spaulding is enthusiastic and possesses ISD skills, he is not
   an expert on engineering principles. If Hernandez is overwhelmed with
   time concerns, she might not be able to offer Spaulding the
   assistance he needs to develop the new lab and other course
   adjustments effectively. PHI (first 2 paragraphs)
Also, the 'final solution' is merely a list of lab materials. No one
addresses student background skill, class organization, time management,
or resource issues, such as putting the grad students to use. Because
of this I feel that Spaulding is not getting the 'whole picture'. PHIII
(final paragraph)

Figure 3. IAM coding example (adapted from Marra, Moore and Klimczek.,
2004)

Category Complete List: Relevance; Importance, Novelty, Outside
knowledge, Ambiguities, Linking, Justification, Critical assessment,
Practical utility, Width of understanding.

Category Rating
Examples          Positive Indicator          Negative Indicator

R+- Relevance     R+ relevant statements      R- irrelevant statements,
                                                 diversions
I+- Importance    I+ Important points/issues  I- unimportant, trivial
                                                 points/issues
N+- Novelty. New  NP+ New problem-related     NP- Repeating what has
    info, ideas,      information                 been said
    solutions     NI+ New ideas for           NI- False or trivial leads
                      discussion
                  NS+ New solutions to        NS- Accepting first
                      problems                    offered solution
                  NQ+ Welcoming new ideas     NQ- Squashing, putting
                                                  down new ideas
                  NL+ Learner brings new      NL- dragged in by tutor
                      things in

Figure 4. Subset of Newman et al. codes (1996)

<NP+R+ There might be a counterpoint here.

<L-OC+ Hernandez is touted as a 'super teacher', and has awards for her
graduate experience, she has been out of the classroom for 10 years.
+OC>-L> <L+ Having worked with the best equip and labs in the Aerospace
industry, she may be unprepared to determine how this atmosphere can be
translated to the classroom. +L>>

<L-OC+ While Spaulding is enthusiastic and possesses ISD skills, he is
not an expert on engineering principles. +OC>-L>. <L+ If Hernandez is
overwhelmed with time concerns, she might not be able to offer Spaulding
the assistance he needs to develop the new lab and other course
adjustments effectively. .+L>>

<OC+ Also, the 'final solution' is merely a list of lab materials. +OC>
<NS+ No one addresses student background skill, class organization, time
management, or resource issues, such as putting the grad students to
use. +NS><C+<NI+ It is because of this that I feel that Spaulding is not
getting the 'whole picture'. +NI>+C>+NP +R>

Figure 5. Passage coded with Newman, Webb and Cochrane protocol

Table 1 Sample Data Produced by DAT--Event Frequencies

Response:             Agree  Disagree  # Responses  # Initial message

Initial Message Type
Disagreement          .38    .13       8            5

Table 2 Research Questions and Citations for Each Analysis Method

Method              Research Questions             Citations

Quantitative --     Simple questions on            Mason (1992);
Descriptive         frequencies of postings,       Romiszowski & Mason
Statistics          readings and thread depth.     (2004)
CA -- IAM           What type of knowledge         Gunawardena et al.
                    construction is occurring?     (1997); Beaudrie
                    How much of each type?         (2000); Marra et al.
                    "Types" are defined by the     (2004)
                    five phases of the IAM (see
                    Figure 2
CA -- Newman        "How much" critical thinking   Newman et al. (1996);
protocol            is occurring for each of the   Newman et al. (1997);
                    codes in the scheme?           Marra et al (2004)
User-classified --  Are there significant          Cho & Jonassen
Constrained         differences or trends between  (2002); Oh (2004);
Forums              numbers postings of the        www.FLE3.com; Bell &
                    different types? Does the      Linn (1997); Guzdial
                    application and meaning of     (1995); Scardamalia &
                    the overall set of posting     Bereiter (1994)
                    rules (the rule ontology)
                    transfer to participants'
                    ability to perform other
                    related tasks? (e.g.,
                    argumentation and problem
                    solving)
Sequential          What sequences of              Bakeman & Gottman
Analysis (DAT)      interactions /message flows    (1997); Jeong (2003)
                    occur?
                    What sequences occur most
                    frequently?