Temporal Structures and Sequential Patterns of Self-regulated Learning Behaviors in Problem Solving with an Intelligent Tutoring System.
1. IntroductionResearch in self-regulated learning (SRL) over the past three decades has generated numerous theoretical perspectives on its features, components, phases or subprocesses, and mechanisms, as well as a substantial number of empirical studies (Boekaerts & Niemivirta, 2000; Greene & Azevedo, 2007; Pintrich, 2004; Schunk & Greene, 2017; Winne, 2017; Zimmerman, 2000). Despite the divergent research base, researchers have reached the same conclusion that SRL is essential to high performance in all sorts of learning and problem-solving activities (Broadbent et al., 2020). For this reason, SRL has become a core theoretical framework to understand performance differences among learners across various disciplines and learning contexts. Moreover, researchers generally agree that SRL is a dynamic and cyclical process comprising a series of events that unfold over time in learning tasks (Azevedo, 2014; Panadero, 2017). Therefore, there has been a growing interest in analyzing the features of SRL behaviors, which could provide direct insights on the factors that distinguish self-regulated learners from those lacking SRL competency and the optimal pathway to excellent performance.
Perhaps the most simple and straightforward way to analyze the features of SRL behaviors is through descriptive analysis, which yields a number of descriptive measures such as the total number, frequency, and variance of SRL behaviors. However, descriptive statistics can be misleading if the data is significantly skewed or contains outliers. Descriptive statistics also reduce the information about how SRL behaviors temporally unfold over time. In this study, we introduced an analytical approach, namely recurrence quantification analysis (RQA), to study the temporal structures of SRL behaviors. In particular, RQA was developed to characterize the temporal properties of elements (e.g., repetitiveness and predictability) in event- or time-series data (Wallot, 2017). It provides a number of indices to describe the features of SRL behaviors without losing their richness and temporal information. To our knowledge, no previous attempts have been made to analyze the features of SRL behaviors using the RQA method, which we deem will generate new knowledge about the SRL process.
In addition, researchers are increasingly using advanced analytical methods to uncover the sequential patterns of SRL behaviors in an effort to understand students' SRL strategies and dispositions (Yu et al., 2021). However, there is limited research examining the sequential patterns of SRL behaviors with a focus on the transition probabilities between SRL behaviors. Moreover, the explanation of the SRL behavioral patterns is often confined to a specific research context, leaving little discussion on the theoretical specifications of SRL (e.g., the cyclical nature of SRL) that are represented by those patterns (Paans et al., 2019).
For this study, we take the initiative to demonstrate how RQA can be used to quantify the temporal structures of SRL behaviors and what insights can be brought forth by this method in an empirical study. Moreover, we examine the sequential patterns of SRL behaviors to understand students' SRL strategies and dispositions. This study contributes to the SRL literature in both methodological and theoretical dimensions.
2. Theoretical background
2.1. Self-regulated learning
Self-regulated learning (SRL) is a cyclical process whereby learners purposefully control and monitor their learning or problem-solving strategies in the dimensions of cognition, metacognition, motivation, and emotion, to achieve predetermined goals (Greene & Azevedo, 2007; Li & Lajoie, 2021; Pintrich, 2004; Schunk & Greene, 2017; Winne, 2017; Zimmerman, 2000). Researchers have reached a consensus that SRL processes should be studied as events, which temporally unfold over time during learning and problem-solving (Azevedo, 2014; Chen & Su, 2019; Michailidis et al., 2018). Paans et al. (2019) further argued that temporal variation of SRL occurs at micro- and macro-level time scales, which "develop in parallel and occur at the same time" (p. 247). For example, a certain number of activities at the micro-level (e.g., task analysis, goal setting, and knowledge acquisition) may be cycled and recycled within one macro-level phase, e.g., the planning phase of SRL.
At the micro-level, SRL models describe how learners self-regulate the components of learning, particularly behaviors, emotions, cognitive and metacognitive activities, in a specific learning or problem-solving context. Researchers examined the sequences of those components to understand performance differences among learners. However, there are no strong assumptions on the relationships between specific micro-level sequences of SRL events and task performance (Azevedo, 2014) since SRL at the micro-level is context dependent.
As an illustration, Schoor and Bannert (2012) used process mining to identify sequences of metacognitive activities for high versus low group performance dyads, as students worked in pairs to solve a task related to statistics. They found that there were no major differences in the patterns of regulatory activities between high- and low-performance groups. In another study, Bannert et al. (2014) used the same analytical technique (i.e., process mining) to explore the sequences of students' SRL activities, as they learned specific concepts and principles of operant conditioning. However, Bannert et al. (2014) found that successful students performed SRL activities in a different order when compared with low performers. Bannert et al. (2014) concluded that it would be problematic to compare students' patterns of learning activities in different learning settings, especially when researchers used different types of data and operationalized students' performance differently to extract such patterns.
At the macro-level, the predominant SRL models suggest that successful self-regulated learners generally follow the phases of SRL in time order, although researchers hold different beliefs about the specific phases that consist of SRL process (Greene & Azevedo, 2007; Paans et al., 2019; Pintrich, 2004; Schunk & Greene, 2017; Winne, 2017; Zimmerman, 2000). For example, Winne (2017) proposed a model of SRL that "unfolds over four loosely sequential and recursive phases" (p. 39), i.e., task definition, goal setting and planning, studying tactics, and adaptations to metacognition. In the conceptual framework for SRL, Pintrich (2004) also contended that SRL comprises four phases (i.e., forethought, planning and activation; monitoring; control; and reaction and reflection), which "represent a general time-ordered sequence" (p. 389). According to Zimmerman (2000), students' self-regulatory activities in learning fall in three cyclical phases: forethought, performance, and self-reflection. The forethought phase is followed by the performance phase, which potentially triggers a range of self-reflections that influence forethought in turn. Students may perform many self-regulatory cycles to complete a task.
Nevertheless, there exists little empirical evidence examining how students move through different phases of SRL during a learning task (Paans et al., 2019). Practically speaking, it is recommended to extract the sequential patterns of SRL behaviors at the micro-level, which is a practice that many studies have followed. But we contend that the SRL behavioral patterns should be interpreted at the micro- and macro-levels so that the results can be comparable across different situations, and we can come close to the nature of SRL behaviors.
2. Sequential analysis of SRL behaviors
Sequential analysis aims to detect the recurring sequential patterns in a finite stream of actions or events (Gottman et al., 1990). SRL researchers use sequential analysis to extract the patterns of SRL behaviors to gain insights into students' use of SRL strategies and SRL dispositions since the raw sequence of learning activities usually contains redundant and fuzzy information. In addition, SRL behaviors do not always occur in a straightforward and structured manner, and there are significant individual differences in how students perform SRL behaviors. The analysis and interpretation of SRL behavioral patterns provide researchers with a clearer representation of students' cognitive structures and decision-making processes than raw behavioral data. Furthermore, the visualization of SRL behavioral patterns offers an intuitive understanding of the complex, temporally unfolding processes of SRL.
A number of analytical techniques are available in the extant literature to extract the sequential patterns of SRL behaviors, including but not limited to process mining (Schoor & Bannert, 2012), t-pattern analysis (Kuvalja et al., 2014), Hidden Markov Modeling, state-transition analysis, and lag sequential analysis (Bakeman & Quera, 1995; Kapur, 2011). As pointed out by Azevedo (2014), advanced techniques that are capable of analyzing the sequential characteristics of SRL process have the potential to transform contemporary conceptions of SRL.
In this study, we use the lag sequential analysis (LSA) to extract SRL patterns since it has several advantages over other sequential analysis techniques. For one, LSA identifies statistically significant transitions from one type of SRL behavior to another. The transition probabilities between different categories of SRL behaviors can be converted into odds ratios or likelihoods for comparison (Kapur, 2011). More importantly, LSA generates sequential patterns of SRL behaviors that describe the SRL process as a whole since the transition probabilities are calculated for each pair of SRL behaviors. LSA provides a holistic view of an SRL process rather than a number of sub-sequences of events. As aforementioned, the present study complements the ongoing efforts that examine students' SRL behavioral patterns, by introducing the RQA method to depict the temporal structures of SRL behaviors.
2.3. Recurrence quantification analysis
RQA is a nonlinear analysis method that quantifies the dynamics of temporal sequences of change over time by detecting "recurrent events" in a time series (Fleuchaus et al., 2020; Jenkins et al., 2020; Wallot, 2017). When applying RQA on a behavioral time series, it returns a range of RQA metrics. The most commonly used RQA measures are percent recurrence (%REC), percent determinism (%DET), average diagonal line length (ADL), laminarity (%LAM), and trapping time (TT) (Marwan et al., 2002; Meinecke et al., 2020; Wallot, 2017). To understand these RQA measures, a core concept that needs to be explained is the recurrence plot (RP).
RP is the visualization of the recurrence values within a discrete time-series by plotting the time series on both the X and y-axis of a two-dimensional grid. Figure 1 shows the illustration of a recurrence plot. In RP, the adjacent points that form a vertical or horizontal line signify a repeating sequence of the same behavior, e.g., linking evidence (LI) [right arrow] linking evidence (LI) [right arrow] linking evidence (LI). The diagonal lines in RP indicate that students demonstrate a sequential behavioral pattern, for example, linking evidence (LI) [right arrow] ordering lab test (AD) [right arrow] searching library (SE). The calculation of RQA measures is based on the distribution of the recurrent points in the RP. For instance, %REC equals the percentage of recurrent points in an RP, and %DET is the proportion of recurrent points forming diagonal lines in an RP.
In particular, the definition of each RQA measure of interest and its corresponding meaning are shown in Table 1. It is noteworthy that the recurrent points on the main diagonal line are excluded when calculating the RQA measures, considering that each value within the sequence is recurrent with itself (Jenkins et al., 2020; Wallot, 2017).
Researchers have successfully applied the RQA to investigate a wide range of social, physiological, psychological, and behavioral phenomena, such as process dynamics in organizations (Meinecke et al., 2020), heart rate variability (Marwan et al., 2002), cognition (Leonardi, 2012), emotion (Jenkins et al., 2020), and human behaviors (Fleuchaus et al., 2020; Wallot, 2017). For example, Jenkins et al. (2020) applied the RQA to analyze the temporal dynamics of affect, in particular, the degree of affect predictability, using the %REC and %DET measures as two crucial indicators. Fleuchaus et al. (2020) considered the %LAM measure as an indicator of behavioral stability, which varies from purely random to completely predictable. Specifically, Fleuchaus et al. (2020) used the %LAM measure to index the persistence of mistaken beliefs as students learned motor skills in science education.
Another representative example is the investigation of the motor-cognitive processes during a writing task, in which students were asked to copy-type a text (Wallot & Grabowski, 2019). When running RQA on students' keystroke logging data, all of the four RQA measures (i.e., %REC, %DET, ADL, and the maximum diagonal line length) suggested that copy-typing behaviors of a comprehensive text were more structured compared to that of an incomprehensible text. However, to the best of our knowledge, no research has used RQA to study students' self-regulated learning behaviors, especially in the context of clinical reasoning.
2.4. The current study
The present study situates the examination of the temporal structures and patterns of SRL behaviors in the context of clinical reasoning. Clinical reasoning is a complex thinking and decision-making process, in which medical practitioners extract meaningful information from patients' files, generate diagnostic hypotheses, order medical lab tests to confirm/disconfirm hypotheses, and finally propose a diagnostic solution (Eva, 2005). In clinical reasoning, medical students and practitioners are faced with a constant stream of decisions, which require them to effectively plan, monitor, control, and reflect on their behaviors. That is, SRL is an essential component for developing clinical competence.
While the research on medical students' SRL behaviors is emerging (Artino et al., 2011; Lajoie et al., 2021; Zheng et al., 2021), few studies have examined students' SRL behaviors with trace data that are collected in a specific clinical reasoning task. Contemporary research highlights the use of trace data since they capture the variations of students' SRL behaviors at a precise level of detail and consequently can afford a fine-grained level of analysis (Greene et al., 2019). What is more, the research on the temporal structures and patterns of SRL behaviors is still nascent in the context of clinical reasoning. This study aims to fulfill these gaps.
In particular, this study addresses the following two research questions: (1) Do high performers differ from low performers regarding the temporal structures of their SRL behaviors in clinical problem-solving? (2) Are there differences in the SRL behavioral patterns between high and low performers? Regarding the first research question, we cannot propose specific directional hypotheses since this study is one of the first to explore the RQA method in studying SRL behaviors, especially in the clinical reasoning context. But we assume that there will be significant differences between high and low performers in the temporal structures of SRL behaviors, i.e., the RQA measures of %REC, %DET, ADL, %LAM, and TT. For the second research question, we hypothesize that the SRL behavioral patterns of high performers demonstrate a better representation of general clinical reasoning procedures and are more closely aligned with the claims of SRL theories (i.e., the cyclical nature of SRL) than low performers.
3. Methods
3.1. Participants, learning environment, and task
In this study, the participants comprised of 75 medical students from a top North American university. There were 28 males (37.3%) and 47 females (62.7%), with an average age of 24.0 (SD = 3.17). Students were tasked to diagnose a virtual patient (VP) in BioWorld (Lajoie, 2020), which is an intelligent tutoring system that provides medical students with a safe practice environment for clinical reasoning. The main interface of the BioWorld platform is shown in Figure 2.
In BioWorld, students begin a task by reading a patient description, which contains information about the patient's profile and symptoms. Meanwhile, students collect evidence items from the patient description and store them in the Evidence Table for future reference. In doing so, students develop an initial understanding of the VP. They recognize what and how much information is important for the diagnosis since the Evidence Table functions as a metacognitive tool to help students monitor the process. Afterward, students can propose one or more diagnostic hypotheses. They confirm or disconfirm the proposed hypotheses by ordering medical lab tests, such as urine tests, toxicology tests, and blood sugar tests. Students can also search an online library with the platform if they need to acquire more information about a disease. Then, students link the collected evidence items and lab test results with corresponding diagnostic hypotheses. After submitting a final hypothesis, students check the relevance of evidence items and lab tests by categorizing them as either support, against, or neutral to the hypothesis. Students are also required to rank evidence items and lab tests based on their importance to the hypothesis. Finally, students write a case summary by reflecting on how they come up with the diagnosis.
The typical behaviors of clinical reasoning and how they relate to SRL phases were shown in Table 2 (Li et al., 2018). In particular, the behavior of collecting evidence items falls in the forethought phase, whereby students get familiar with the task conditions and develop an understanding of the patient problems. Students take actions to accomplish the task in the performance phase, which consists of three types of behaviors, i.e., raising/managing hypotheses, adding tests, and searching library. The self-reflection phase comprises four types of behaviors, namely, linking evidence/results, checking evidence/results, prioritizing evidence/results, and summarization for final diagnosis. These four types of behaviors involve extensive metacognitive activities as students evaluate the collected information, make adaptations and decisions, and reflect on their performance.
At the micro-level, the variations of SRL process are represented by the temporal changes in the eight types of clinical reasoning behaviors, which in consequence, trigger the changes in the three phases of SRL simultaneously at the macro-level. For instance, the transition from the behavior of collecting evidence items to adding tests occurs at the micro-level. Meanwhile, the behavioral transition indicates that students move from the forethought phase to the performance phase.
The VP case used in this study was created by a panel of experts including medical professionals and learning scientists. The correct diagnosis for the VP case was pheochromocytoma, which is a rare, usually noncancerous tumor that develops in an adrenal gland. Prior to the study, a medical expert validated the VP case to ensure that it provided appropriate practice for the participants.
3.2. Procedure
Prior to the study, we had obtained research ethics approval from the institution's Research Ethics Board (REB) office. Moreover, we obtained the students' written consent to participate in this study. They all mentioned that they felt comfortable diagnosing virtual patients in BioWorld. Furthermore, they were informed that they had the right to withdraw at any time they wanted during the process of clinical problem-solving.
A training session was provided to help medical students get familiar with the BioWorld environment. In particular, the training session started with a researcher-guided introduction of the BioWorld system and how to use its various features to help them reach a final diagnosis. Afterward, the participants were asked to complete the clinical reasoning task independently. A number of research assistants were present to solve operational or technical issues; however, they were not allowed to give hints about the disease. The participants spent approximately 40 minutes on average to finish the diagnosis during regular school hours. It is worth mentioning that all operational behaviors (e.g., order lab test) and corresponding timestamps for each participant were automatically recorded in the log files of the BioWorld system.
3.3. Data processing and analysis
We first classified students as either high or low performers based on their diagnostic performance. Specifically, we considered students who correctly diagnosed the VP as high performers. The rest of the students were viewed as low performers since they failed to provide a correct diagnosis. In particular, there were 42 high performers and 33 low performers.
To address our first research question, we used the R package of "crqa" to perform the RQA on students' problem-solving behaviors separately for high and low performers (Coco & Dale, 2014). We then compared the differences in RQA measures (i.e., %REC, %DET, ADL, %LAM, and TT) between these two performance groups using inferential statistics. To address our second research question, we performed lag sequential analysis (Bakeman et al., 2009; Bakeman & Quera, 1995) to uncover the sequential behavioral patterns of both high and low performers. Specifically, we conducted the analysis using the GSEQ (Generalized Sequential Querier) program (Bakeman & Quera, 1995).
4. Results
4.1. Do high performers differ from low performers regarding the temporal structures of their SRL behaviors in clinical problem-solving?
The descriptive statistics of SRL behaviors were shown in Table 3. We compared the differences in the temporal structures of SRL behaviors (i.e., RQA measures) between high and low performers. As shown in Table 4, we found that there was no significant difference in the overall level of the repetitiveness of SRL behaviors between the two performance groups, t(73) = .68, p = .499. However, the %DET value for low performers (M = 78.18) was significantly lower than higher performers (M = 86.37), t(73) = -3.35, p = .001. This result indicated that low performers had a significantly higher ratio of single, isolated recurrent behaviors to all recurrent behaviors than high performers. The recurrent behaviors of high performers were more likely to be part of a behavioral sequence. The effect size (d = .76) was found to exceed Cohen's (1988) convention for a medium effect (d = .50).
Regarding the average length of the repeating behavioral patterns, there was no significant difference between high and low performers. There was also no significant difference in the average length of the repeating sequences of the same behavior between the two performance groups. Nevertheless, low performers showed a significantly lower ratio of repeating sequences of the same behavior to all recurrent behaviors than high performers, since the %LAM value for low performers (M = 86.23) was significantly lower than higher performers (M = 90.35), t(73) = -2.26, p < .05. This result suggested that high performers were more likely to perform a behavior repeatedly (e.g., order lab test) before moving on to the other behaviors than low performers. The effect size for the difference was medium, with Cohen's d = .52 (Cohen, 1988).
4.2. Are there differences in the SRL behavioral patterns between high and low performers?
As aforementioned, we performed behavioral sequential analyses for both high and low performers using the GSEQ program (Bakeman & Quera, 1995). Table 5 and Table 6 showed the sequential transition matrix of SRL behaviors of low performers and high performers, respectively. In the sequential transition matrix, the row means a starting behavior, whereas the column means a subsequent behavior. The values in the matrix are Z-scores. A Z-score greater than 1.96 indicates that the transition between two behaviors is statistically significant (i.e., P < .05) (Bakeman & Quera, 2011). Accordingly, a Z-score that is larger than 2.58 and 3.20 would guarantee significance levels of .01 and .001, respectively. For instance, the behavioral sequence of 'collecting evidence items [right arrow] collecting evidence items', as shown in Table 5, was statistically significant, given that the Z-score = 53.44 > 3.20.
In fact, the results in Table 5 and Table 6 suggested that the sequential transitions between the same type of SRL behaviors were all statistically significant except the behavior of "summarization for final diagnosis." Moreover, the sequential transition from the behavior of "Prioritizing evidence/result" to "summarization for final diagnosis" was significant. Regarding the sequential transition patterns of SRL behaviors, there was no difference between low and high performers. The two performance groups both conducted a behavior repeatedly during the problem-solving process. However, high performers had a larger Z-score for each of the significant sequential transitions than low performers.
In addition, we examined the sequential transitions between different SRL behavioral states by viewing temporally connected behaviors of the same type (e.g., CO [right arrow] CO [right arrow] CO) as a behavioral state, i.e., the state of CO. As an illustration, the behavioral sequence of "CO [right arrow] CO [right arrow] CO [right arrow] RA [right arrow] AD [right arrow] AD" would be transformed as "CO [right arrow] RA [right arrow] AD" for further analysis. In doing so, we may develop a deep understanding of students' thinking and reasoning activities as they are reflected by different behavioral states. Additionally, new insights may be obtained by examining the sequential transitions of different behavioral states, given that the sequential transitions between the same type of SRL behaviors were all found to be statistically significant.
The sequential transition matrices of SRL behavioral states of low performers and high performers were shown in Table 7 and Table 8, respectively. Based on the sequential transition matrices, we made the transition diagrams of SRL behavioral states for the two performance groups (see Figure 3 and Figure 4 the diagrams of low and high performers, respectively).
As shown in and Figure, low and high performers demonstrated similar behavioral patterns in general. They both demonstrated a behavioral pattern of "CO (Collecting evidence items) [right arrow] AD (Adding tests) [right arrow] RA (Raising/Managing hypotheses) [right arrow] CA (Categorizing evidence/results) [right arrow] PR (Prioritizing evidence/results) [right arrow] SU (Summarization for final diagnosis)." Moreover, the sequential transition from the behavior of "Raising/Managing hypotheses" to "Collecting evidence items" was significant for both low- and high-performing groups. In addition, both low and high performers conducted the behavior of "Raising/Managing hypotheses" following the behavior of "Linking evidence/results," and vice versa. Nevertheless, there was a reciprocal sequential transition between the behaviors of "Searching library" and "Adding tests" for high performers, while the transition between the two behaviors was unidirectional for low performers. It was also noticeable that the behavior of "Linking evidence/results" significantly stimulated the occurrence of the "Collecting evidence items" behavior for high performers, whereas low performers showed no such a behavioral pattern.
5. Discussion
In this study, we found that low performers had more single, isolated recurrent behaviors in problem-solving, whereas the recurrent behaviors of high performers were more likely to be part of a behavioral sequence. This finding is aligned with the research on the development of professional expertise (Lajoie, 2009). High performers may use their mental models, which can be developed from either experience or instruction, to drive the selection of problem-solving behaviors. In other words, high performers may have developed internal representations of the task condition and see relevant concept relationships (He et al., 2021). Consequently, they follow specific implicit procedures to perform the task. It is also quite possible that high performers are self-regulated learners who can effectively monitor and control their problem-solving processes. High performers are aware of the next desirable behavior based on the outcome of prior behavior, thus yielding more meaningful behavioral sequences than low performers.
While these explanations are grounded in solid theoretical frameworks and in common sense, researchers should not draw a conclusion that this finding is applicable to the research contexts other than clinical reasoning. In fact, we argued that this finding might be confined to the contexts or disciplines that are governed by facts, principles, rules, and scientific reasoning. Take this study as an example, high performers demonstrated more behavioral patterns, because the procedures of clinical reasoning are well-established, and physicians are aware of those procedures. In the learning or problem-solving contexts that require creativity and innovation, the lack of regularity in students' learning behaviors, however, may be a crucial indicator of high performance, since students need to challenge their conventional thinking for breakthrough success (Koopmans, 2020).
This study found that high performers were more likely than low performers to perform a behavior repeatedly before moving on to other behaviors. This finding can be explained by the fact that clinical reasoning in medicine traditionally values the ultimate goal of providing accurate diagnoses of disease (Li et al., 2020; Wass et al., 2001). High performers may purposefully conduct a behavior repeatedly to eliminate any sense of uncertainty in the process of clinical problem-solving. For example, they may conduct multiple diagnostic tests needed to narrow down a diagnosis. Additionally, each step of clinical reasoning has its own challenges. For instance, medical students may have difficulties in identifying cues, developing an understanding of patient problems, generating diagnostic hypotheses, prioritizing evidence items, and finalizing a decision (Audetat et al., 2017; Gonzalez et al., 2021). Therefore, it is reasonable to infer that high performers were more behaviorally engaged than low performers at each step of clinical reasoning to address ambiguities. Clearly, more research is needed to uncover the underlying mechanisms for this finding by fusing both objective (log files) and subjective data (direct input from the participants such as self-reports, think-aloud, or interviews).
Interestingly, both low and high performers demonstrated clear and logical transitions between and among SRL behavioral states revealing similar behavioral patterns in clinical problem-solving. In particular, the behavioral state transitions of both low and high performers were in compliance with a loosely sequential process of SRL, i.e., forethought, performance, and self-reflection (Schunk & Greene, 2017; Zimmerman, 2000). This result was consistent with the research of Greene et al. (2019), who also found evidence of temporality in SRL. Nevertheless, the sequential transitions tell a more in-depth story about the relationship between SRL and performance.
Findings from the sequential transition analyses revealed that the probability of the sequential transition from the forethought phase to the performance phase (i.e., collecting evidence items [right arrow] adding tests) was higher for high performers than that of low performers. Moreover, high performers were more likely to proceed to the self- reflection phase (i.e., linking, categorizing, and prioritizing evidence/results) from the performance phase (i.e., raising/managing hypotheses) than low performers. High performers were also unique in that their behavioral state transitions were cyclically sustained, as suggested by a behavioral state transition from "linking evidence/results" (i.e., self-reflection phase) to "collecting evidence items" (i.e., forethought phase). The feedback loop may help high performers adaptively adjust their SRL behaviors over cycles, allowing them to navigate their decision-making processes towards a correct diagnosis.
In addition, the sequential transition between the behaviors of "searching library" and "adding tests" was bidirectional for high performers, while the transition between the two behaviors was unidirectional for low performers, i.e., searching library [right arrow] adding tests. This finding is partially in line with our previous studies, in which we found that learners would commence by ordering a lab test prior to a library search (Li et al., 2020), and high performers spent significantly more time ordering lab tests than low performers before developing their first diagnostic hypothesis (Li et al., 2020). In essence, the two behaviors both fell into the performance phase of SRL. When students were unfamiliar with a specific disease or diagnostic tests, they searched the library to gain more information. They ordered lab tests to confirm or rule out a specific hypothesis regarding which disease was present. Both high and low performers consulted the online library to guide the selection of lab tests. However, high performers tended to search library to clarify the meaning of the results of the lab test after they collected the test. As a result, high performers were able to take corrective action in a timely fashion when ordering the next lab test, as they gained additional information of a prior test from searching library.
Moreover, according to Zimmerman (2000), attentional control is a crucial strategy used intensively by expert performers in the performance phase of SRL. In the same vein, high performers knew how to concentrate in the performance phase of clinical reasoning by ignoring distractions and by focusing their attention on those two behaviors exclusively, which could also explain the reciprocal relationship between the two behaviors.
In summary, the findings from this study uphold the theoretical assumption that students' SRL behaviors occur in a loosely sequenced and temporal order (Bernacki, 2017; Broadbent et al., 2020; Winne, 2019). Moreover, the results of the sequential pattern analyses revealed that only high performers repeatedly progressed through the three SRL phases (i.e., forethought, performance, and self-reflection) in a cyclical and iterative fashion, which may help explain the performance difference with low performers. A unique theoretical contribution of this study is that high performers were found to have a higher transition probability across the three SRL phases than low performers. In this regard, this study informs future research on the theoretical advancements of SRL by examining how the three phases of SRL and even the behaviors within each phase are interconnected.
Furthermore, this study provided significant methodological insights regarding the quantification of the temporal structure of SRL behaviors. Specifically, this study is one of the first to demonstrate how RQA can be used to quantify the temporal structure of SRL behaviors in the context of clinical reasoning. Along with the RQA measures that describe the overall characteristics of students' SRL behaviors (e.g., repetitiveness and predictability), we looked into the sequential patterns of those SRL behaviors using sequential analysis (Bakeman et al., 2009). The use of both RQA and sequential analysis provided researchers a complete picture of students' SRL behaviors, whereby new understandings of students' performance differences can be obtained.
In addition, the present research has practical implications. Specifically, findings from this study can inform the design of learning analytics dashboards, which provide educators and students with the opportunities for awareness, reflection, sensemaking, and behavioral change. For instance, educators can develop an understanding of students' potential performance from their behavioral characteristics so that they can adjust their instructional strategies accordingly. Moreover, educators need to carefully design their course activities to facilitate the acquisition of not only domain-specific knowledge but also SRL skills among students. This study also informs the design of early warning systems (e.g., automatic detection of at-risk students from analyzing their behavioral patterns), whereby immediate support can be provided to help all students succeed.
6. Conclusion
In this study, we examined the temporal structures and patterns of SRL behaviors, as 75 medical students solved a clinical reasoning task within an intelligent tutoring system. We found that the recurrent behaviors of high performers were more structured and predictive than low performers. As revealed by the sequential pattern analysis, high performers also demonstrated a higher transition probability across the three phases of SRL than low performers. Moreover, high performers were unique in that their behavioral state transitions were cyclically sustained. In addition to its methodological insights that could inform future research significantly, this study provided researchers with new evidence to support the theoretical assumptions of SRL in the context of clinical reasoning with the population of medical students.
Nevertheless, this study is not without limitations. First, we did not explicitly assess students' prior knowledge and skills, although we had confirmed with a medical expert to ensure the appropriateness of the task for the participants. Second, we had to make inferences about students' decision-making process from log files, as log files do not directly measure cognitive and metacognitive activities (Paans et al., 2019). Consequently, we identified the characteristics and patterns of students' SRL behaviors; however, our explanations about the research findings called for further studies. Third, we compared the behavioral patterns of high and low performers; however, we could not tell which differences in the behavioral patterns were statistically significant between those two groups. Lastly, the participants were all from the same university, which may affect the generalizability of our research findings.
Despite these limitations, this research opens new directions for future research. For one, it would be fruitful to examine the relationships between the temporal structures of SRL behaviors (i.e., RQA measures) and other psychological or contextual factors, such as students' personality, motivation, and emotion. Task difficulty is another crucial fact that may affect the characteristics and patterns of SRL behaviors. Therefore, an important future direction is to examine the influences of task difficulty on the temporal structures and patterns of SRL behaviors. It is also promising to study SRL behaviors in other learning or problem-solving contexts using RQA and sequential behavioral analysis simultaneously in one study.
References
Artino, A. R., Hemmer, P. A., & Durning, S. J. (2011). Using self-regulated learning theory to understand the beliefs, emotions, and behaviors of struggling medical students. Academic Medicine, 86(10), S35-S38. https://doi.org/10.1097/ACM.0b013e31822a603d
Audetat, M.-C., Laurin, S., Dory, V., Charlin, B., & Nendaz, M. R. (2017). Diagnosis and management of clinical reasoning difficulties: part II. Clinical reasoning difficulties: management and remediation strategies. Medical Teacher, 39(8), 797-801. https://doi.org/10.1080/0142159X.2017.1331034
Azevedo, R. (2014). Issues in dealing with sequential and temporal characteristics of self-and socially-regulated learning. Metacognition and Learning, 9(2), 217-228. https://doi.org/10.1007/s11409-014-9123-1
Bakeman, R., & Quera, V. (1995). Analyzing interaction: Sequential analysis with SDIS and GSEQ. Cambridge University Press.
Bakeman, R., & Quera, V. (2011). Sequential analysis and observational methods for the behavioral sciences. Cambridge University Press.
Bakeman, R., Quera, V., & Gnisci, A. (2009). Observer agreement for timed-event sequential data: A Comparison of time-based and event-based algorithms. Behavior Research Methods, 41(1), 137-147. https://doi.org/10.3758/BRM.41.L137
Bannert, M., Reimann, P., & Sonnenberg, C. (2014). Process mining techniques for analysing patterns and strategies in students' self-regulated learning. Metacognition and Learning, 9(2), 161-185. https://doi.org/10.1007/s11409-013- 9107-6
Bernacki, M. L. (2017). Examining the cyclical, loosely sequenced, and contingent features of self-regulated learning: Trace data and their analysis. In D. H. Schunk & J. A. Greene (Eds.), Handbook of self-regulation of learning and performance (2nd ed., pp. 370-387). Routledge/Taylor & Francis Group.
Boekaerts, M., & Niemivirta, M. (2000). Self-regulated learning: Finding a balance between learning goals and egoprotective goals. In M. Boekaerts, P. Paul R, & M. Zeidner (Eds.), Handbook of Self-Regulation (1st ed., pp. 417-450). US: Academic Press. https://doi.org/10.1016/B978-012109890-2/50042-1
Chen, C.-H., & Su, C.-Y. (2019). Using the BookRoll e-book system to promote self-regulated learning, self- efficacy and academic achievement for university students. Journal of Educational Technology & Society, 22(4), 33-46.
Coco, M. I., & Dale, R. (2014). Cross-recurrence quantification analysis of categorical and continuous time series: An R package. Frontiers in Psychology, 5, 1-14. https://doi.org/10.3389/fpsyg.2014.00510
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates, Inc.
Eva, K. W. (2005). What every teacher needs to know about clinical reasoning. Medical Education, 39(1), 98-106. https://doi.org/10.1111/j.1365-2929.2004.01972.x
Fleuchaus, E., Kloos, H., Kiefer, A. W., & Silva, P. L. (2020). Complexity in science learning: Measuring the underlying dynamics of persistent mistakes. Journal of Experimental Education, 88(3), 448-469. https://doi.org/10T 080/00220973.2019.1660603
Gonzalez, L., Nielsen, A., & Lasater, K. (2021). Developing students' clinical reasoning skills: A Faculty guide. Journal of Nursing Education, 60(9), 485-493. https://doi.org/10.3928/01484834-20210708-01
Gottman, J. M., Gottman, J. M., & Roy, A. K. (1990). Sequential analysis: A guide for behavioral researchers. Cambridge University Press.
Greene, J. A., & Azevedo, R. (2007). A Theoretical review of Winne and Hadwin's model of self-regulated learning: New perspectives and directions. Review of Educational Research, 77(3), 334-372. https://doi.org/10.3102/003465430303953
Greene, J. A., Plumley, R. D., Urban, C. J., Bernacki, M. L., Gates, K. M., Hogan, K. A., Demetriou, C., & Panter, A. T. (2019). Modeling temporal self-regulatory processing in a higher education biology course. Learning and Instruction, 101201. https://doi.org/10.1016/j.learninstruc.2019.04.002
He, Q., Borgonovi, F., & Paccagnella, M. (2021). Leveraging process data to assess adults' problem-solving skills: Using sequence mining to identify behavioral patterns across digital tasks. Computers & Education, 166, 104170. https://doi.org/10.1016/j.compedu.2021.104170
Jenkins, B. N., Hunter, J. F., Richardson, M. J., Conner, T. S., & Pressman, S. D. (2020). Affect variability and predictability: Using recurrence quantification analysis to better understand how the dynamics of affect relate to health. Emotion, 20(3), 391-402. https://doi.org/10.1037/emo0000556
Kapur, M. (2011). Temporality matters: Advancing a method for analyzing problem-solving processes in a computer- supported collaborative environment. International Journal of Computer-Supported Collaborative Learning, 6(1), 39-56. https://doi.org/10.1007/s11412-011-9109-9
Koopmans, M. (2020). Education is a complex dynamical system: Challenges for research. Journal of Experimental Education, 88(3), 358-374. https://doi.org/10.1080/00220973.2019.1566199
Kuvalja, M., Verma, M., & Whitebread, D. (2014). Patterns of co-occurring non-verbal behaviour and self-directed speech; a comparison of three methodological approaches. Metacognition and Learning, 9(2), 87-111. https://doi.org/10.1007/s11409013-9106-7
Lajoie, S. P. (2009). Developing professional expertise with a cognitive apprenticeship model: Examples from Avionics and Medicine. In K. A. Ericsson (Ed.), Development of Professional Expertise: Toward Measurement of Expert Performance and Design of Optimal Learning Environments (pp. 61-83). Cambridge University Press.
Lajoie, S. P. (2020). Student modeling for individuals and groups: the BioWorld and HOWARD platforms. International Journal of Artificial Intelligence in Education, 1-16. https://doi.org/10.1007/s40593-020-00219-x
Lajoie, S. P., Li, S., & Zheng, J. (2021). The Functional roles of metacognitive judgement and emotion in predicting clinical reasoning performance with a computer simulated environment simulated environment. Interactive Learning Environments, 1-12. https://doi.org/10.1080/10494820.2021.1931347
Leonardi, G. (2012). The study of language and conversation with recurrence analysis methods. Psychology of Language and Communication, 16(2), 165. https://doi.org/10.2478/v10057-012-0012-x
Li, S., & Lajoie, S. P. (2021). Cognitive engagement in self-regulated learning: An Integrative model. European Journal of Psychology of Education, 1-20. https://doi.org/10.1007/s10212-021-00565-x
Li, S., Zheng, J., & Lajoie, S. P. (2020). Efficient clinical reasoning: Knowing when to start and when to stop. Education in the Health Professions, 3(1), 1-7.
Li, S., Zheng, J., Poitras, E., & Lajoie, S. (2018). The Allocation of time matters to students' performance in clinical reasoning. In R. Nkambou, R. Azevedo, & J. Vassileva (Eds.), Lecture notes in computer sciences (pp. 110-119). Springer International Publishing AG, part of Springer Nature. https://doi.org/10.1007/978-3-319-91464-0_11
Marwan, N., Wessel, N., Meyerfeldt, U., Schirdewan, A., & Kurths, J. (2002). Recurrence-plot-based measures of complexity and their application to heart-rate-variability data. Physical Review E, 66(2), 26702. https://doi.org/10.1103/PhysRevE.66.026702
Meinecke, A. L., Handke, L., Mueller-Frommeyer, L. C., & Kauffeld, S. (2020). Capturing non-linear temporally embedded processes in organizations using recurrence quantification analysis. European Journal of Work and Organizational Psychology, 29(4), 483-500. https://doi.org/10.1080/1359432X.2019.1658624
Michailidis, N., Kapravelos, E., & Tsiatsos, T. (2018). Interaction analysis for supporting students' self- regulation during blog-based CSCL activities. Journal of Educational Technology & Society, 21(1), 37-47.
Paans, C., Molenaar, I., Segers, E., & Verhoeven, L. (2019). Temporal variation in children's self-regulated hypermedia learning. Computers in Human Behavior, 96, 246-258. https://doi.org/10.1016/jxhb.2018.04.002
Panadero, E. (2017). A Review of self-regulated learning: Six models and four directions for research. Frontiers in Psychology, 8, 1-28. https://doi.org/10.3389/fpsyg.2017.00422
Pintrich, P. R. (2004). A Conceptual framework for assessing motivation and self-regulated learning in college students. Educational Psychology Review, 16(4), 385-407. https://doi.org/10.1007/s10648-004-0006-x
Schoor, C., & Bannert, M. (2012). Exploring regulatory processes during a computer-supported collaborative learning task using process mining. Computers in Human Behavior, 28(4), 1321-1331. https://doi.org/10.1016/jxhb.2012.02.016
Schunk, D. H., & Greene, J. A. (2017). Historical, contemporary, and future perspectives on self-regulated learning and performance. In P. A. Alexander, D. H. Schunk, & J. A. Greene (Eds.), Handbook of self-regulation of learning and performance (2nd ed., pp. 1-15). Routledge. https://doi.org/10.4324/9781315697048-1
Wallot, S. (2017). Recurrence quantification analysis of processes and products of discourse: A Tutorial in R. Discourse Processes, 54(5-6), 382-405. https://doi.org/10.1080/0163853X.2017.1297921
Wallot, S., & Grabowski, J. (2019). A Tutorial introduction to recurrence quantification analysis (RQA) for keystroke logging data. In E. Lindgren & K. Sullivan (Eds.), Observing Writing (pp. 163-189). Brill. https://doi.org/10.1163/9789004392526
Wass, V., Van Der Vleuten, C., Shatzer, J., & Jones, R. (2001). Assessment of clinical competence. Lancet, 357(9260), 945949. https://doi. org/10.1016/S0140-6736(00)04221-5
Winne, P. H. (2017). Cognition and metacognition within self-regulated learning. In P. A. Alexander, D. H. Schunk, & J. A. Greene (Eds.), Handbook of self-regulation of learning and performance (2nd ed., pp. 36-48). Routledge. https://doi.org/10.4324/9781315697048-3
Winne, P. H. (2019). Paradigmatic dimensions of instrumentation and analytic methods in research on self-regulated learning. Computers in Human Behavior, 96, 285-289. https://doi.org/10.1016/jxhb.2019.03.026
Zheng, J., Li, S., & Lajoie, S. P. (2021). Diagnosing virtual patients in a technology-rich learning environment: A Sequential mining of students' efficiency and behavioral patterns. Education and Information Technologies. https://doi.org/10.1007/s10639-021-10772-0
Zimmerman, B. J. (2000). Attaining self-regulation: A Social cognitive perspective. In M. Boekaerts, P. Paul R, & M. Zeidner (Eds.), Handbook of self-regulation (1st ed., pp. 13-39). Academic Press. https://doi.org/10.1016/B978- 0121098902/50031-7
Shan Li *, Juan Zheng and Susanne P. Lajoie
Department of Educational and Counselling Psychology, McGill University, Montreal, QC, Canada shan.li2@mail.mcgill.ca // juan.zheng@mail.mcgill.ca // susanne.lajoie@mcgill.ca
* Corresponding author
(Submitted July 7, 2021; Revised November 16, 2021; Accepted November 28, 2021)
Caption: Figure 1. The illustration of a recurrence plot Diagonal line indicating behavioral pattern, i.e., LI > AD > SE
Caption: Figure 2. The main interface of the BioWorld environment
Caption: Figure 3. The transition diagram of SRL behavioral states of low performers
Caption: Figure 4. The transition diagram of SRL behavioral states of high performers
Table 1. The selected RQA measures to quantifying the temporal
structure of learning behaviors (Marwan et al., 2002; Meinecke et
al., 2020; Wallot, 2017)
Variable Definition Meaning
Percent Percentage of recurrence How often does an
Recurrence points in a recurrence individual show the same
(%REC) plot (RP). behavior (i.e.,
repetitiveness of
%REC = Sum of recurrent behaviors in the time
points in the RP / size series)
of RP
Percent Proportion of recurrent To what extent do
Determinism points forming diagonal repetitions of behaviors
(%DET) lines in a RP. occur in the form of
behavioral patterns?
%DET = Sum of diagonally e.g., a student may
adjacent recurrent conduct a behavior
points / sum of repeatedly. They can
recurrent points also demonstrate certain
behavioral patterns.
Average Average length of How long is the average
Diagonal Line diagonal lines in the RP repeating behavioral
Length (ADL) pattern?
Laminarity Proportion of recurrent To what extent do
(%LAM) points forming vertical repetitions of behaviors
line structures. occur in repeating
sequences of the same
%LAM = Sum of vertically behavior?
adjacent recurrent
points / sum of
recurrent points
Trapping time Average length of How long is the average
(TT) vertical lines in the RP repeating sequence of
the same behavior?
Table 2. The coding scheme for analyzing SRL behaviors of clinical
reasoning
SRL phases Clinical behaviors Code Description
Forethought Collecting evidence CO Collecting evidence
items items from the patient
description by recalling
one's prior knowledge
pertaining to the
symptoms
Performance Raising/Managing RA Outlining a single or
hypotheses multiple diagnostic
hypothesis based on
the collected evidence
Adding tests AD Conducting medical lab
tests
Searching library SE Searching for
information in the
library for additional
explanations
Self- Linking LI Linking evidence items
Reflection evidence/results and test results with
corresponding
diagnostic hypotheses
Categorizing CA Checking the relevance
evidence/results of evidence items and
lab test results
towards a specific
hypothesis (i.e.,
whether the evidence/
tests in support,
against, or neutral of
one hypothesis)
Prioritizing PR Ranking evidence items
evidence/results and lab test results
according to their
importance to a
hypothesis
Summarization for SU Making the final
final diagnosis diagnosis by writing a
summarization
Table 3. The descriptive statistics of SRL behaviors
High performers Low performers
Min Max M SD Min Max M SD
CO 8 34 15.14 3.79 10 34 14.91 5.25
RA 4 53 15.90 9.98 4 42 17.18 8.50
AD 4 48 18.95 11.03 0 57 15.85 11.94
SE 0 79 6.55 13.13 0 71 10.33 14.14
LI 0 69 17.43 16.74 0 68 14.00 14.52
CA 8 52 21.36 10.33 0 83 16.30 13.54
PR 3 102 30.69 27.49 0 267 32.12 50.89
Table 4. Group differences in the temporal structures of SRL
behaviors
Group M SD t df p Cohen's d
%REC Low 13.26 3.84 .68 73 .499 .15
High 12.75 2.67
%DET Low 78.18 12.30 -3.35 73 .001** .76
High 86.37 8.85
ADL Low 5.74 3.91 -1.65 73 .104 .38
High 7.05 2.94
%LAM Low 86.23 8.43 -2.26 73 .027* .52
High 90.35 7.35
TT Low 7.90 7.12 -1.57 73 .121 .36
High 10.15 5.28
Note. * p < .05, ** p < .01. The definitions and meanings of the
variables were shown in Table 1.
Table 5. The sequential transition matrix of SRL behaviors of low performers Z CO RA AD SE LI CA PR CO 53.44# -6.62 -6.14 -5.89 -7.94 -9.83 -12.39 RA -6.52 40.62# -5.99 -5.22 -2.55 -6.54 -13.46 AD -9.25 -1.81 44.93# .18 -8.85 -10.17 -12.82 SE -5.89 -4.91 1.16 44.04# -6.96 -8.00 -10.08 LI -8.02 -3.12 -8.71 -5.92 51.32# -9.48 -11.95 CA -9.44 -10.69 -10.17 -8.00 -9.48 56.75# -9.46 PR -11.91 -13.47 -12.82 -10.08 -11.95 -13.09 57.84# SU .00 .00 .00 .00 .00 .00 .00 Z SU CO -2.22 RA -2.41 AD -2.30 SE -1.81 LI -2.14 CA -2.34 PR 10.92# SU .00 Note. Z = Z-score, CO = Collecting evidence items, RA = Raising/Managing hypotheses, AD = Adding tests, SE = Searching library, LI = Linking evidence/results, CA = Categorizing evidence/results, PR = Prioritizing evidence/results, SU = Summarization for final diagnosis. Only significant sequential transitions were highlighted in bold. Note: Only significant sequential transitions are indicated with #.
Table 6. The sequential transition matrix of SRL behaviors of high performers Z CO RA AD SE LI CA PR CO 64.30# -5.61 -6.34 -5.23 -10.35 -11.99 -14.99 RA -6.57 50.12# -7.34 -4.23 -3.15 -7.89 -15.66 AD .00 -3.38 58.72# 1.63 -11.63 -13.91 -17.39 SE -4.56 -2.54 .60 51.70# -6.48 -7.74 -9.68 LI -9.05 -4.70 -11.52 -5.59 63.83# -13.24 -16.56 CA -11.53 -12.51 -13.91 -7.74 -13.24 68.50# -15.15 PR -14.42 -15.64 -17.39 -9.68 -16.56 -18.73 69.44# SU .00 .00 .00 .00 .00 .00 .00 Z SU CO -2.37 RA -2.47 AD -2.75 SE -1.53 LI -2.61 CA -2.96 PR 11.46# SU .00 Note. Only significant sequential transitions were highlighted in bold. Note: Only significant sequential transitions are indicated with #.
Table 7. The sequential transition matrix of SRL behavioral states of low performers Z CO RA AD SE LI CA CO 0 .21 3.00 ** -.08 -.11 -2.03 RA 3.13 ** 0 -3.35 -3.46 6.87 *** 7.23 *** AD -3.34 2.51 * 0 7.37 *** -3.72 -3.09 SE 1.18 -3.06 8.96 *** 0 -2.94 -2.44 LI .04 5.83 *** -3.41 -.76 0 -2.01 CA -1.45 -4.18 -3.09 -2.44 -2.01 0 PR -1.45 -4.18 -3.09 -2.44 -2.01 -1.34 SU 0 0 0 0 0 0 Z PR SU CO -2.03 -1.98 RA -4.17 -4.07 AD -3.09 -3.02 SE -2.44 -2.39 LI -2.01 -1.96 CA 25.51 *** -1.31 PR 0 26.20 *** SU 0 0 Note. * p < .05, ** p < .01, *** p < .001.
Table 1. The sequential transition matrix of SRL behavioral states of high performers Z CO RA AD SE LI CA CO 0 .94 6.13 *** -1.6 -2.69 -2.43 RA 2.18 * 0 -3.48 -4.12 8.09 *** 8.02 *** AD -3.16 2.21 * 0 9.62 *** -3.23 -3.49 SE 1.05 -2.01 7.90 *** 0 -2.66 -2.4 LI 2.10 * 5.32 *** -3.23 -0.77 0 -2.38 CA -1.7 -4.8 -3.49 -2.4 -2.38 0 PR -1.7 -4.8 -3.49 -2.4 -2.38 -1.67 SU 0 0 0 0 0 0 Z PR SU CO -2.43 -2.36 RA -4.81 -4.67 AD -3.49 -3.39 SE -2.4 -2.33 LI -2.38 -2.31 CA 26.43 *** -1.63 PR 0 27.30 *** SU 0 0 Note. * p < .05, *** p < .001.
 Printer friendly
 Cite/link
 Email
 Feedback
| |
| Author: | Li, Shan; Zheng, Juan; Lajoie, Susanne P. |
|---|---|
| Publication: | Educational Technology & Society |
| Article Type: | Report |
| Geographic Code: | 1CANA |
| Date: | Oct 1, 2022 |
| Words: | 8326 |
| Previous Article: | Design and Development of a Blockchain-Based Secure Scoring Mechanism for Online Learning. |
| Next Article: | Developing a Gesture-based AR Exhibit: Differently-Guided Experiences for Complex Conceptual Learning in Science. |
| Topics: | |