Engagement Level Assessment via Computer Camera Using Machine Learning Based on Facial Movement Analysis.
1. IntroductionIn the digital era, self-directed learning has emerged as a foundational skill for knowledge development and career advancement. This stems from the proliferation of AI tools and online resources, which have made access to knowledge more convenient while simultaneously granting learners autonomy in selecting content, schedules, and learning pace. However, such autonomy also introduces an inherent challenge. In the absence of traditional supervisory factors such as physical classrooms, fixed timetables, or the presence of instructors, learners face the task of self-regulating their behaviors and maintaining sustained focus. According to recent longitudinal studies [2], the ability to manage attention and maintain engagement has been identified as a key determinant of success in flexible learning environments. This is largely due to the fact that learners are frequently distracted by external factors (e.g., social media, digital notifications) [3] and often struggle with accurately recognizing their own cognitive states. As a consequence, knowledge acquisition tends to become passive, inefficient, and inconsistent, thereby highlighting the urgent need for personal support tools that can enhance self-awareness and regulate learning behaviors [4].
To address this demand, several technological approaches have been proposed for monitoring and assessing learners' cognitive states in self-directed learning contexts. These include: (1) specialized biometric devices such as electroencephalography (EEG) or heart rate monitors, and (2) nonverbal signal recognition through camera-based methods, particularly facial and head pose analysis. The first approach, involving the use of biometric devices such as EEG or heart rate sensors, has been valued for its ability to directly measure physiological activity. Unfortunately, this approach faces significant limitations regarding cost, deployment complexity, and user acceptance in typical learning settings [1], [5]. The second approach, leveraging webcams to extract nonverbal signals such as facial expressions and head-pose, has shown strong potential. Recent studies, such as [4], have demonstrated that behavioral features derived from facial analysis are strongly associated with learners' emotional and cognitive states. The main advantages of this method include its low cost, real-time data acquisition, and ease of deployment on widely available devices, thereby opening opportunities for the development of practical and scalable intelligent learning applications. However, focusing solely on facial data provides a limited perspective, as it neglects other crucial signals such as body posture and physical interactions. To address this limitation, multimodal approaches that integrate facial cues with additional data sources (e.g., audio, body posture, physiological signals) have been shown to deliver superior performance [6]. Nevertheless, the deployment of multimodal models in real-world contexts faces significant barriers, including concerns regarding privacy, hardware costs, and the complexity of data synchronization and processing. Furthermore, supporting techniques, such as head pose estimation from 2D cameras, have not yet achieved sufficient accuracy in uncontrolled environments [7].
Based on the analysis of the aforementioned advantages and limitations, this study adopts a pragmatic approach: developing a concentration recognition system that relies solely on webcam video data, extracting features from facial expressions and head poses. Unlike previous geometric-feature-based studies that often rely on static snapshots [8], [9], [10], our approach introduces a dual-branch feature encoding strategy to decouple individual physiological traits from actual engagement-related behavioral changes. Notably, the proposed pipeline enables traditional machine learning models such as SVM to outperform the DAiSEE baseline LRCN model, achieving higher classification accuracy while maintaining significantly lower computational complexity. We propose a machine learning pipeline capable of classifying concentration levels from short video segments of 10 seconds in length. The model is designed as a core component of a continuous monitoring system, by applying it over consecutive time windows to generate a temporal sequence of predictions. The analysis of this prediction sequence enables the identification of behavioral patterns, such as gradual declines in attention or recurrent moments of distraction, thereby providing personalized feedback aimed at enhancing learning effectiveness.
The main contributions of this paper are summarized as follows:
* User-Invariant Parallel Framework: A novel parallel processing architecture (Dual-branch Pipeline) that concurrently processes raw behavioral baselines and Z-score normalized temporal variations. This strategy effectively neutralizes individual biases inherent in facial structures, ensuring the framework's robustness across diverse user demographics.
* Dynamic Behavioral Encoding: A temporal dynamic feature extraction technique, which goes beyond static measurements to capture multi-scale temporal fluctuations in behavior, serving as clearer indicators of cognitive state shifts.
* Practical Anomaly-based Detection: The introduction of an anomaly-based approach for binary distraction detection, achieving a substantial improvement in Macro-F1 compared to supervised methods, providing a more viable solution for real-world deployment where labeled distraction data is often scarce.
* Empirical Validation: A comprehensive evaluation on a public dataset, demonstrating that the proposed webcam-based approach achieves competitive performance, reaching up to 58.60% accuracy in four-class concentration recognition despite relying solely on low-cost visual features. Furthermore, the experiments highlight that incorporating head pose features leads to a substantial improvement of nearly 9.3% Macro-F1 points, proving their effectiveness as a practical compromise between performance and system simplicity.
2. Related work
2.1 Concept of Computer Vision-based Behavioral Analysis
In the learning process, maintaining focus is a crucial factor, particularly for the effective acquisition of knowledge. However, with the advent of E-learning--while it has ushered in a new era of global accessibility to education--a fundamental challenge arises from the lack of direct interaction, which traditionally serves as the cornerstone of pedagogical practice. This absence makes it increasingly difficult for learners to sustain attention [11].
To address this issue, there has emerged a pressing demand for systems capable of detecting lapses in concentration and providing timely feedback. Consequently, the field of automatic learner state detection has gained increasing attention. The objective of this field is to develop intelligent computer systems capable of inferring learners' cognitive and emotional states based on observable data.
From a technical perspective, the problem can be formulated as follows: given a sequence of multimodal observational data O = [o1, o1ot} collected from a learner, the system aims to construct a predictive function F(.) hat maps this sequence O to a state SsY. The state space Y may represent a binary label set (Focused, Distracted), a multi-class set (Engaged, Bored, Confused, Frustrated), as in the DAiSEE dataset [12], or even a continuous value denoting the intensity of a state. Such a system not only enables learners to recognize when their attention diminishes but can also suggest appropriate interventions, thereby optimizing the overall effectiveness of the online learning experience.
2.2 Evolution of attention detection approaches
The development of attention detection models can be divided into multiple stages, reflecting advances in both machine learning techniques and the understanding of the task's intrinsic nature, as shown in Table 1.
Stage 1: Traditional machine learning models with static features
Early research on attention detection relied primarily on handcrafted geometric features extracted from individual frames or short video segments, with statistical descriptors such as mean and standard deviation subsequently provided as input to traditional classifiers including SVM, Random Forest, and KNN. A notable contribution in this line of work was made by [8], who introduced the Eye Aspect Ratio (EAR) for blink detection, demonstrating that simple geometric measures can effectively support real-time monitoring of ocular behavior and laying the groundwork for later studies. Subsequent research [9], [10] extended this approach by combining EAR with the Mouth Aspect Ratio (MAR) and head pose features, applying them to classifiers such as SVM and Random Forest, thereby confirming the feasibility of low-cost, webcam-based distraction detection. Despite these contributions, such methods exhibit a fundamental limitation: by modeling behavior as static rather than dynamic, they fail to capture the temporal evolution of attentional states, often misclassifying short physiological blinks as prolonged eyelid closure caused by fatigue, which undermines their robustness in practical scenarios.
Stage 2: The deep learning revolution and implicit dynamics modeling To overcome the limitations of traditional approaches, subsequent studies have increasingly adopted deep learning methods. A notable milestone was the release of the DAiSEE dataset by [9], the first large-scale "in-the-wild" video benchmark annotated with four affective states--Engaged, Bored, Confused, and Frustrated--accompanied by a baseline model based on the Long-term Recurrent Convolutional Network (LRCN) [13], which combines CNNs and RNNs to jointly capture spatial and temporal features. Building on this foundation, [14] proposed a CNN-LSTM architecture with an attention mechanism, enabling the model to focus on the most informative frames and achieving performance that significantly outperformed the baseline, thereby demonstrating the effectiveness of deep learning for attention detection. Further developments between 2021 and 2022 [15] explored combinations of CNNs with Bi-LSTM or Temporal Convolutional Networks (TCN) to leverage both spatial feature extraction and sequential modeling, while [16] introduced multi-task learning frameworks capable of addressing multiple affective and attentional recognition tasks simultaneously. Despite achieving high accuracy, these deep learning-based approaches exhibit three major limitations: (i) substantial computational costs requiring powerful GPUs, (ii) limited interpretability of model decisions, and (iii) challenges in deployment within large-scale online learning environments, where learners predominantly rely on standard consumer-grade laptops. Stage 3: Towards a balance between performance and interpretability In response to the limitations of deep learning, another line of research has focused on modeling dynamics in a more explicit manner. Instead of allowing the model to learn implicitly, researchers deliberately design hand-crafted dynamic features. These features are computed over longer time windows and directly describe the temporal aspects of behavior, for example:
* Blink frequency (number of blinks per minute).
* PERCLOS index (Percentage of Eye Closure over a time window).
* Average duration of each yawn.
* Angular velocity and acceleration of head movements.
* Gaze stability (variance of pupil position) [16].
With the aim of supporting developing countries where computational resources are limited, this study adopts the third approach--hand-crafted dynamic features combined with lightweight machine learning models. The central hypothesis is that explicitly designed temporal features, when integrated with efficient classifiers (e.g., SVM, Random Forest, Gradient Boosting), can provide a cost-effective and interpretable solution capable of detecting attention lapses using only standard webcams on consumer-grade devices.
Early approaches often relied on manually extracted geometric features. A foundational contribution is the work of [10], which introduced the Eye Aspect Ratio (EAR) as an effective metric for blink detection. Based on similar static features, many systems using classical machine learning models such as SVM or Random Forest demonstrated the feasibility of building simple distraction-detection tools with standard webcams [11]; [12]. However, this approach has an inherent limitation: it treats behavior as a static state, overlooking important temporal dynamics (e.g., it cannot distinguish between a rapid blink and slow eye closure due to fatigue).
To address this limitation and enhance detection capabilities, many studies have proposed more sophisticated models. The introduction of the "in-the-wild" DAiSEE dataset [9] established a common benchmark and set deep learning-based performance standards. Hybrid models that combine the strengths of different architectures have shown promising results. For instance, the studies by [12] and other hybrid model works [16] leveraged the spatial feature extraction capabilities of Convolutional Neural Networks (CNNs) and the temporal feature extraction ability of Long Short-Term Memory networks (LSTMs) to build robust recognition models. More recent deep fusion approaches further extended this idea by integrating multidimensional facial features, such as facial expressions and head pose, using heavy CNN backbones to improve robustness under unconstrained conditions [17]. Similarly, the multitask learning models of [18] and subsequent hybrid multi-task frameworks [19] demonstrated impressive performance across multiple tasks.
However, the pursuit of higher accuracy through increasingly complex models has led to two major challenges. The first is the interpretability problem: these deep learning models operate as "black boxes," making it difficult to understand the basis of their decision-making [20]. The second is the resource problem: they require substantial computational power and long training times, hindering deployment on personal devices with limited hardware. This limitation is further amplified in recent Transformer-based architectures, which, despite achieving state-of-the-art performance in behavior recognition, demand substantial computational and memory resources, making them impractical for real-time deployment on resource-constrained devices [22]. A comparative summary of these existing methodologies, highlighting their respective strengths and limitations, is provided in Table 2.
In light of the trade-offs identified in Table 2, and with the aim to support developing countries where computational resources are limited, this study adopts the third approach--hand-crafted dynamic features combined with lightweight machine learning models. This study hypothesizes that effectively extracting dynamic features and feeding them into a classical machine learning model can provide an optimal balanced solution, leveraging the richness of behavioral information while maintaining computational efficiency and interpretability.
3. Methodology
To address the problem of detecting distraction states through video, we propose a multi-stage processing architecture as illustrated in Fig. 2. The core objective of this architecture is to transform raw video sequences into a set of high-level, interpretable dynamic features. The processing pipeline consists of four main stages: (1) Feature extraction, (2) Data cleaning, (3) Pipeline, and (4) Classification. Stage 1: Feature extraction
The first stage focuses on extracting fundamental behavioral signals from raw video frames. In our study, the input data consists of video recordings of individuals using web cameras for learning and working purposes. These continuous video streams are segmented into short clips of fixed length. For each frame in these clips, we employ OpenFace [21] to detect faces and obtain facial information, which serves as the essential first step for further analyses [15]. Upon successful face detection, the system extracts a feature vector encompassing 68 facial landmarks, eye gaze vectors, head pose, and Action Units (AUs). These features provide a comprehensive digital representation of the user's behavioral state. Conversely, if face detection fails, the corresponding clip is discarded to maintain data consistency and quality.
From the 68 facial landmark points ([p.sub.0] to [p.sub.67]) illustrated in Fig. 1, our study extracts specific feature values to represent the user's engagement. The extraction and calculation process is detailed as follows:
* Spatial Distances: We calculate the distances between the center point [p.sub.30] and every other landmark point to capture the overall facial structure. Furthermore, vertical distances from the eyebrows to the eyes and mouth are calculated separately for each side to detect subtle expression changes:
** For the left side, distances are measured from [p.sub.19] to [p.sub.39] and from [p.sub.19] to [p.sub.48]
** For the right side, distances are measured from [p.sub.24] to [p.sub.42] and from [p.sub.24] to
* Opening Area (S): To represent the degree of opening for the eyes and mouth, we calculate the area S of the feature points surrounding each part. Letting ([x.sub.i], [y.sub.i]) be the coordinates of each landmark [p.sub.i], the area S is defined by the following formula:
[Please download the PDF to view the mathematical expression]
where the boundary condition is defined as [Please download the PDF to view the mathematical expression] Behavioral and State Indicators:
* Yawn Detection: A yawn is specifically detected when the vertical length of the mouth increases rapidly and remains sustained for a duration of 3 seconds.
* Gaze and Head Orientation: The direction and angle of gaze, along with head position and rotation angles (Pitch, Yaw, Roll), are utilized directly as implemented in the OpenFace library.
* Action Unit (AU) Strength: The intensity of various facial muscle movements (AUs) is recorded to reflect cognitive states and alertness levels.
Stage 2: Feature computation, cleaning, and normalization
After using OpenFace to detect faces and successfully extract 68 geometric landmark points for each frame, these raw signals can serve as a basis for recognizing and classifying learner states [23]. Based on the 68 facial landmarks extracted from OpenFace, the proposed system architecture computes five main feature groups, representing indicators related to the user's attention and cognitive state:
* Head Pose [5]: Uses the Pitch, Yaw, and Roll parameters to represent head orientation along the three spatial axes. These metrics reflect the user's macro-level movement tendencies. Significant changes in head pose, such as turning the head to the side or looking downward, are considered clear indicators of distraction.
* Eye Gaze [5]: The direction of the eyes is calculated based on the geometric relationships between the landmarks around the eyes and the pupils. This feature provides micro-level information, enabling the detection of subtle distractions (micro-distractions), such as glancing at a phone or an off-screen object.
* Facial Action Units (AUs) [5]: A selected subset of AUs is used to reflect cognitive state and alertness levels
** AU45 (Blink): Used to analyze blink frequency and patterns--factors closely associated with fatigue and cognitive load.
** AU25 (Lips Part) and AU26 (Jaw Drop): The simultaneous occurrence of these two AUs serves as a strong indicator of yawning, a common sign of drowsiness and disengagement.
* Eye Aspect Ratio (EAR) [25]: Quantifies the degree of eye openness by measuring the vertical and horizontal distances between the landmarks around the eyes. This feature enables the detection of behaviors such as frequent blinking or prolonged eye closure. It is calculated using the following formula:
[Please download the PDF to view the formula] (1)
Where:
** pi is the coordinate vector (x, y) of the i-th landmark point. The indices (37, 38,...) follow the OpenFace annotation standard.
** [parallel][p.sub.a] - [p.sub.b][parallel] is the Euclidean distance between landmarks pa and pb.
** Numerator: [parallel][p.sub.37] - [p.sub.41][parallel] and [parallel][p.sub.38] - [p.sub.40][parallel] are the vertical distances between the upper and lower eyelids. Their sum represents the overall eye height.
** Denominator: [parallel][p.sub.36] - [p.sub.39][parallel] is the horizontal distance between the two eye corners, representing the eye.
* Mouth Aspect Ratio - MAR [26]: Measuring mouth opening is used to detect behaviors such as yawning or talking. This is an important indicator that reflects the level of fatigue and distraction. The calculation formula is as follows:
[Please download the PDF to view the formula] (2)
** pi is the coordinate vector (x, y) of the i-th landmark point. The indices (49, 51,...) follow the OpenFace annotation standard.
** [parallel][p.sub.a] - [p.sub.b][parallel]I is the Euclidean distance between landmarks p[alpha]and pfc.
** Numerator: [parallel][p.sub.51] - [p.sub.59][parallel], [parallel][p.sub.53] - [p.sub.57][parallel], and [parallel][p.sub.49] - [p.sub.55][parallel] are the vertical distances from the upper to the lower lip. Their sum represents the overall mouth height.
Denominator: [parallel][p.sub.49] - [p.sub.55][parallel] is the horizontal distance between the two mouth corners, representing the mouth width
Stage 3: Parallel Processing Pipeline:
This is the core stage of the methodology where dynamic features are engineered. The primary purpose of this stage is to solve two problems simultaneously:
1) Eliminate the influence of individual anatomical and environmental variations across clips, such as natural differences in facial structure and variations in face-to-camera distance that primarily affect the absolute magnitude of facial features [24].
2) Extract a comprehensive set of dynamic features without losing critical statistical information.
To achieve this, we implemented a two-pronged parallel processing pipeline. This design decision stems from a direct mathematical consequence: if we were to apply Z-score normalization (to address variability) to the entire dataset before calculation, the statistical features describing the distribution, such as Mean and Variance, would always become constants (approximately 0 and 1, respectively) [27].
This would completely erase information about the user's "behavioral baseline" within each clip.
Therefore, the parallel pipeline is designed as follows:
* Branch 1 (Raw Data): Statistical features (e.g., mean and variance) are computed from the non-normalized data sequence. This branch aims to preserve information about the baseline state or the user's overall behavioral trends within a 10-second window.
* Branch 2 (Normalized Data): Dynamic features (e.g., min, max, range) are computed from the Z-score normalized data sequence. This branch focuses on quantifying relative variations and instantaneous behavioral events, independent of anatomical characteristics. This parallel strategy ensures that the final feature vector captures both the individual's baseline state and the full range of subtle behavioral variations.
Stage 4: Feature vector aggregation and state classification
Feature Vector Aggregation: Global normalization [28] is performed to bring all feature vectors into the same value range. Afterwards, the features from both branches are concatenated in a consistent order to form a single feature vector with a fixed length for each 10-second clip. This vector comprehensively captures the user's behavior during that time interval.
Classification and Handling Imbalanced Data: The set of feature vectors is used to train models for classifying user states (e.g., "focused" vs. "distracted"). Since real-world data is often highly imbalanced (with the "focused" class being dominant), we designed experiments to compare multiple algorithms to identify the most effective approach:
* Anomaly detection model: Isolation Forest, trained to identify "distracted" samples as outliers relative to normal behavior. [29]
* Supervised classification models: Random Forest (RF), XGBoost, Support Vector Machine (SVM), K-Nearest Neighbors (KNN), and Decision Tree (DT), trained directly on the labeled dataset.
The performance of these models is evaluated using Precision, Recall, and F1-score metrics, with a particular focus on accurately identifying the minority class ("distracted").
4. Experiments
4.1 Dataset and Experimental setup
To evaluate the effectiveness of the proposed methodology, this study utilizes the public DAISEE dataset. As a reputable resource, this dataset provides short-duration video recordings of learners, annotated with four engagement states: Boredom, Confusion, Engaged, and Frustrated. These characteristics make DAISEE an ideal benchmark for analyzing learner behavior and detecting distraction in online learning environments. The experimental process is conducted across two distinct classification scenarios.
* Scenario 1 (4-class classification): This scenario examines the discriminative capacity of the engineered features across the original four engagement labels. Performance is measured using the macro-average F1-score [30] to mitigate class imbalance, alongside Top-1 Accuracy for benchmarking against published deep learning baselines [31].
* Scenario 2 (binary classification): To improve practical applicability, labels are grouped into Focused (Engaged) and Distracted (Boredom, Confusion, Frustrated). This setup employs threshold-tuned supervised classifiers (SVM, Random Forest) and anomaly detection (Isolation Forest). Furthermore, in-processing methods such as SMOTE resampling and cost-sensitive learning were implemented for a comprehensive comparison.
To implement the aforementioned scenarios, the study experimented with four different feature sets, constructed with progressively increasing complexity:
To ensure the reproducibility of the experiments across both scenarios, the following technical configurations were implemented:
4.2 Evaluation Methodology
The evaluation of the classification models in this study relies on standard metrics derived from the confusion matrix, where predicted labels are systematically compared with the ground truth of the DAiSEE dataset. In this framework, a True Positive (TP) refers to a sample that is correctly classified as belonging to the target class--for example, a Distracted clip correctly identified as Distracted in the binary task, or a Boredom clip accurately recognized in the four-class task. Conversely, a False Positive (FP) denotes a sample that is incorrectly assigned to the target class, such as a Focused clip misclassified as Distracted, or an Engaged clip wrongly labeled as Boredom. A True Negative (TN) occurs when a sample is correctly recognized as not belonging to the target class--for instance, a focused clip is correctly excluded from the Distracted category. Finally, a False Negative (FN) represents the case where the model fails to identify a sample of the target class, such as a Distracted clip mislabeled as Focused, or a Confusion clip mistaken for another state.
[Please download the PDF to view the mathematical expression]
From these four fundamental quantities, several key performance metrics are computed. Accuracy measures the overall proportion of correctly classified samples across all classes. Precision, defined as TP / (TP + FP), reflects how reliable the model is when predicting a given class--for example, how often clips predicted as Distracted are truly Distracted. In contrast, Recall (REC), defined as TP / (TP + FN), evaluates the model's ability to capture all actual instances of that class. To reconcile the trade-off between Precision and Recall, the F1-score (F1) is used as their harmonic mean. Finally, in the multi-class setting of DAiSEE with four affective states (Boredom, Confusion, Engaged, and Frustrated), we report the macro-average of these metrics. This approach gives equal weight to each class, thereby providing an unbiased evaluation in the presence of severe class imbalance.
5. Results
5.1 Four-Class Classification
The comparative results of the five classifiers across four feature sets are reported in Table 3 and visualized in Figs. 3 and 4. SVM consistently emerged as the most effective model, achieving the highest accuracy of 58.60% and the highest macro-average F1-score of 34.42% when using Feature Set 2. Table 3 shows that this corresponds to a substantial improvement over Feature Set 1, where SVM achieved 52.19% accuracy and 25.11% macro-F1. Random Forest and XGBoost also benefited from the inclusion of Head Pose, improving to 54.51% and 56.47% accuracy, respectively, with macro-F1 scores of 24.76% and 30.45%. By contrast, Decision Tree and KNN remained less competitive, ranging from 44-51% accuracy with macro-F1 rarely above 27%.
The addition of Action Units, as detailed in Table 5 and illustrated in Figs. 3 and 4, consistently reduced performance. SVM accuracy dropped to 52.38% with Feature Set 3 and 52.93% with Feature Set 4, with corresponding macro-F1 scores of 25.95% and 26.18%. Random Forest, XGBoost, and KNN also showed lower values in these configurations. Decision Tree exhibited minor recovery in macro-F1 with Feature Set 4 (28.03%) but without significant gains in accuracy. Overall, the evidence from Table 5 and Figs. 3-4 confirms that Head Pose is a critical feature for improving classification, while Action Units add redundancy and noise. Importantly, although SVM surpassed the DAiSEE baseline LRCN accuracy of 57.90% (see Table 6), macro-F1 never exceeded 35%, reflecting the difficulty of recognizing distracted states under severe class imbalance.
Table 6 contextualizes these results by benchmarking them against both original DAiSEE baselines and recent state-of-the-art (SOTA) models from 2024 and 2025. The proposed SVM configuration (58.60%) not only surpasses the standard InceptionNet and C3D baselines but also remains highly competitive with modern complex architectures.
Notably, while advanced deep learning models such as EfficientNetV2-L combined with LSTM or Bi-GRU [31] achieve higher peak accuracies (ranging from 61.56% to 62.11%), they require massive computational overhead and high-end GPUs. In contrast, our SVM model achieves a comparable performance level--outperforming the Improved MobileNetV2+MUSER+LSTM (57.49%) [32] and the LRCN benchmark (57.90%) [9] while maintaining the efficiency required for real-time CPU deployment.
This finding underscores that a well-optimized geometric-based approach can bridge the performance gap between classical machine learning and heavy deep learning architectures. However, despite these accuracy gains, the Macro-F1 scores across most models remain a persistent challenge due to the severe class imbalance in the DAiSEE dataset. Our results with Isolation Forest for binary detection further complement these findings by providing a practical solution for identifying rare distracted states in resource-constrained environments.
5.2 Binary Classification
Table 7 compares Isolation Forest with supervised models on the optimized binary task, with results illustrated in Figs. 4-8. Isolation Forest consistently outperforms across all key metrics. It achieves more stable Recall across feature sets, peaking at 33.25% (vs. KNN's 32.9%), and maintains superior macro-average F1-scores above 60% (best at 61.43%), compared to SVM's peak of 50.06%. For the minority "Distracted" class, Isolation Forest doubles the best supervised result, reaching 28.26% F1-score and 27.87% Precision with Feature Set 4, while supervised models remain below 15% F1 and 10% Precision. Although Accuracy is high for all models (up to 92.89%), it is less meaningful under severe imbalance.
These findings strongly support reframing distraction detection as anomaly detection. Unlike supervised models that struggle with sparse "Distracted" examples, Isolation Forest learns a robust profile of majority "Focused" behavior and detects deviations as anomalies, thereby resisting majority-class bias. Table 5 highlights the best configuration (Feature Set 4), where Recall of28.67% captures nearly one-third of true disengagements, while the "Focused" class achieves 96.41% F1--evidence of rare-event detection without excessive false alarms.
Feature analysis further shows that while Feature Set 2 yields the highest Accuracy (92.89%), Feature Set 4 delivers the best overall balance, with top macro-F1 and minority-class performance. This indicates that a richer feature set, despite adding noise, enables better discrimination of subtle distracted behaviors. Finally, although a ~60% macro-F 1 reveals room for improvement, the efficiency and low computational cost of Isolation Forest make it a practical and deployable solution on standard CPU hardware.
Table 8 indicates that Isolation Forest achieved the highest macro-F 1 across all feature sets. With feature set 4, the Focused class achieved extremely high scores (96.37% Precision, 96.46% Recall, 96.41% F1), while Distracted reached 28.26% F1--a modest absolute value but nearly double the best supervised alternative (~14%). Fig. 7 visualizes these class-wise metrics. At the aggregate level, Fig. 8 shows that Isolation Forest attained a peak Macro-F1 of 61.43%, more than 11 points higher than SVM with Feature Set 2 (50.06%). Moreover, whereas supervised models fluctuated across feature sets and often deteriorated with Action Units, Isolation Forest maintained stable performance. This highlights the suitability of anomaly detection for imbalanced data distributions, where distracted states are rare but critical to identify.
Overall, the results demonstrate three main findings: (i) Head Pose is a critical feature, improving SVM's Macro-F1 by nearly +9.3 points; (ii) Action Units degrade performance, reducing SVM's Macro-F1 from 34.42% to 25.95%; and (iii) Isolation Forest excels in the binary task, achieving 61.43% Macro-F1 and substantially improving minority-class detection.
5.3 Qualitative Evaluation
To further substantiate quantitative performance and demonstrate the interpretability of the proposed framework, a qualitative analysis of the behavioral features extracted via OpenFace is presented. This evaluation visualizes the critical geometric cues leveraged by the classifier to distinguish between engagement states, highlighting system robustness even under challenging "in-the-wild" conditions, such as partial facial occlusion.
Fig. 9 illustrates a representative 'Distracted' case, characterized by a pronounced downward head tilt and hand-to-face occlusion. Despite the partial obstruction, spatial landmarks are accurately tracked. The 3D bounding box generated by OpenFace clearly reflects this deviated pose through a high pitch angle. For the classification model, the primary discriminative cues in this scenario are the significant deviation in head orientation (e.g., PoseRx) and a drastically reduced Eye Aspect Ratio (EAR) resulting from eyelid closure. Conversely, Fig. 10 depicts a typical 'Focused' state. In this instance, the learner maintains a direct, frontal orientation toward the camera, which is visually confirmed by the squarely aligned 3D bounding box and parallel, forward-pointing gaze vectors. Classifiers such as the Support Vector Machine (SVM) correctly identify this state as engaged because the head orientation angles (Rx, Ry) remain near zero, and the EAR is high and stable, reflecting sustained visual alertness.
Besides, these qualitative visual observations directly align with the quantitative feature importance extracted from the trained models, as detailed in Table 9. Head Pose and Eye metrics emerge as the primary discriminators (ranked 1st and 2nd), with importance scores peaking at 0.079 and 0.061, respectively. This mathematical ranking confirms why the visible physical shifts--such as the downward pitch and eye closure observed in Fig. 9--serve as the strongest indicators of distraction. In contrast, Action Units (AUs) occupy the lowest tier (Importance < 0.033), functioning as auxiliary or noise-prone variables. This alignment between the visual case studies and the mathematical feature rankings underscores the transparency of the geometric-based approach, ensuring that predictions are explicitly grounded in observable and meaningful behavioral changes.
5.4 Computational Efficiency and Practical Scalability Analysis
The core advantage of the proposed framework lies in its superior computational efficiency relative to state-of-the-art Deep Learning (DL) architectures. We analyze this advantage quantitatively through the metrics of Floating Point Operations (FLOPs) and model parameter counts.
Quantitative Complexity Comparison: Deep learning baselines utilized in recent engagement detection studies, such as the EfficientNetV2-L backbone employed in [33], are characterized by substantial computational demands. As analyzed in standard benchmarks, the EfficientNetV2-L architecture comprises approximately 120 million parameters and requires 53 GFLOPs (Giga Floating Point Operations) per inference pass [34]. This massive computational load typically necessitates dedicated GPU acceleration to achieve acceptable latency, creating a bottleneck for widespread deployment on personal devices. In contrast, our proposed pipeline decouples feature extraction from classification. The feature extraction stage leverages MediaPipe Face Mesh, which is architecturally founded on MobileNetV2. As demonstrated by Sandler et al. (2018) [35], this lightweight architecture utilizes inverted residuals and linear bottlenecks to drastically reduce computational cost, operating at approximately 0.3 GFLOPs per frame while maintaining high fidelity. The subsequent classification stage using SVM or Isolation Forest operates on low-dimensional feature vectors (d < 50) with negligible overhead. Consequently, the total computational complexity of our system represents an approximate 176-fold reduction in operations compared to the EfficientNetV2 baseline (53GFLOPs / 0.3GFLOPs [??] 176.6). In addition, the classification stage requires only ~3.98 ms per clip on a CPU (based on empirical testing on an AMD Ryzen 5 5500U), indicating that the decision-making component itself is highly efficient.
Practical Scalability and Edge Deployment: This theoretical efficiency translates directly into practical scalability. To further address real-world deployment concerns, we analyze the end-to-end latency of the system. Processing a 10-second video clip ([approximately equal to]300 frames) requires feature extraction followed by classification, resulting in an overall latency dominated by the feature extraction stage. Importantly, this bottleneck is independent of the proposed model and is shared across most facial-analysis-based pipelines. Regarding four-class classification, our SVM-based approach enables stable real-time inference on commodity CPUs, eliminating the hardware barrier. Similarly, for binary classification, the Isolation Forest model frames disengagement as an anomaly detection task, optimizing the trade-off between diagnostic depth and processing speed. While GPU-based models such as EfficientNetV2 achieve lower raw end-to-end latency, their reported inference times are measured on high-performance hardware (e.g., V100 GPU) and correspond only to the forward pass of the network, typically under optimized conditions (FP16 precision, batch size = 16, and a unified codebase as reported in ImageNet benchmarks). In contrast, our framework is designed for CPU-only environments, where GPU resources may not be available [34]. Under such conditions, deep CNN-based approaches typically experience significant latency degradation, whereas our lightweight classification module maintains consistent performance. As a result, the minimal resource footprint facilitates a Client-Side (Edge-AI) deployment strategy. Unlike server-centric approaches, our system can execute entirely within end-user devices, ensuring horizontal scalability for massive online classrooms without necessitating expensive hardware upgrades.
6. Discussion
The results reported in Table 5 and visualized in Figs. 4-7 clearly demonstrate the decisive role of Head Pose in the four-class classification task. By adding this single feature to the baseline set (EAR, MAR, Gaze), SVM's performance increased from 52.19% Accuracy and 25.11% Macro-F1 to 58.60% Accuracy and 34.42% Macro-F1, a gain of nearly +9.3 Macro-F1 points. Random Forest and XGBoost also improved substantially, rising to 54.51% and 56.47% Accuracy, respectively. These gains highlight that Head Pose captures macro-level shifts in attention--such as looking down or turning sideways--that are highly informative for engagement detection. The primary reason for this superiority is that head orientation serves as a more direct and stable proxy for a learner's target of focus in a self-directed environment. Unlike micro-movements of the eyes which can be frequent and noise-prone, a shift in head pose represents a deliberate change in the visual field, making it a high-signal indicator for identifying when a learner has physically disengaged from the learning material.
By contrast, the degradation observed with the addition of action units (AU45, AU25, AU26) underscores that more features do not always lead to better performance. For SVM, Macro-F1 declined from 34.42% to 25.95% (Feature Set 3) and 26.18% (Feature Set 4). Random Forest and XGBoost also showed similar reductions, while Decision-Tree exhibited only a marginal Macro-F1 recovery in Set 4 (28.03%) without improvements in accuracy. Our analysis of the feature importance rankings in Fig. 9 reveals the underlying cause: while core geometric features consistently occupy the top tiers, the majority of Action Units (AUs) reside in the lower spectrum (Importance < 0.012). Although certain AUs like AU45 (blink) and AU25 (lips part) showed marginal relevance (up to 0.033), they largely introduced redundancy, as their contributions overlap with information already captured more robustly by EAR and MAR, the latter maintaining a stable importance of approximately 0.027 (Table 9). These findings indicate that AU features not only add limited new information but also amplify noise under class imbalance, emphasizing the need for careful feature selection rather than indiscriminate expansion. These findings confirm that AU features introduced redundancy with EAR/MAR and added noise under class imbalance, reinforcing the need for careful feature curation rather than indiscriminate expansion. This performance drop can be attributed to the fact that EAR and MAR already provide a condensed and effective representation of eye and mouth openness; thus, adding individual AUs increases the dimensionality of the feature space without providing unique variance. Furthermore, in "in-the-wild" webcam conditions, fine-grained AU detection is highly sensitive to lighting and individual facial morphology, which introduces significant noise that misleads traditional classifiers when trying to distinguish subtle affective states.
Fig. 5 and Fig. 6 further underline the robustness of SVM, which consistently outperformed other supervised classifiers across all feature sets. Notably, Table 3 situates these findings within the broader research context by comparing against the DAiSEE baseline. SVM with Feature Set 2 achieved 58.60% Accuracy, surpassing the baseline LRCN (57.90% Accuracy). This is particularly remarkable given that LRCN is a powerful vision-language model that combines CNNs for spatial feature extraction with RNNs (LSTMs) for temporal sequence modeling, and has been demonstrated by Donahue et al. (2017) to outperform standalone CNN or RNN approaches in action recognition and image/video description tasks [13]. These results highlight that lightweight traditional models, when equipped with carefully engineered features, can compete with and even surpass modern deep learning methods. Nonetheless, Macro-F1 across all feature sets remained below 35%, revealing a fundamental limitation: class imbalance severely hampers the reliable recognition of distracted states. This outcome suggests that for engagement detection, explicitly designed features with physical meanings are more effective than the implicit features learned by "black-box" deep learning models. By focusing on high-level behavioral signals like EAR and Head Pose, the SVM can establish clearer decision boundaries that are less susceptible to overfitting commonly seen in complex architectures like LRCN when trained on imbalanced datasets.
The binary classification task provides further insight. Results in Table 7 and Fig. 8 show that Isolation Forest, an anomaly detection model, excelled at distinguishing Focused from distracted states. While the Focused class achieved near-perfect performance (Precision, Recall, and F1 all around 96%), the minority Distracted class reached 28.26% F1, nearly double the best supervised alternative (~14%). Fig. 8 highlights the overall advantage: Isolation Forest achieved a peak Macro-F 1 of 61.43%, more than 11 points higher than SVM with Feature Set 2 (50.06%). Moreover, while supervised models fluctuated substantially with the addition of Action Units, Isolation Forest maintained stable performance across feature sets. This consistency reflects the alignment of anomaly detection with the natural distribution of the data: learners are predominantly focused, while distracted states are rare but critical to identify. The fundamental reason for the success of Isolation Forest lies in its alignment with the natural distribution of learner behavior. Since "Distracted" events are rare and exhibit highly diverse patterns (e.g., yawning, looking away, or head tilting), supervised models struggle to define a consistent boundary for this minority class. In contrast, Isolation Forest focuses on modeling the "normal profile" of the dominant Focused class, effectively identifying any deviation as an anomaly, thereby bypassing the majority-class bias that plagues traditional supervised learning. Regarding the practical implications, while the absolute F1-score for the "Distracted" class remains modest at 28.26%, this level of performance reflects a deliberate "safety-first" intervention strategy. In a real-time environment, the primary risk of an AI-based system is not just missing a distracted moment (False Negative) but rather interrupting a user who is deeply engaged in a "flow state" (False Positive). By achieving high precision in the "Focused" class and identifying only the most distinct, unambiguous instances of disengagement (as evidenced by the 28.26% F1-score), our system minimizes the risk of erroneous alerts that could frustrate the user or disrupt their concentration. Consequently, this level of detection serves as a non-intrusive support mechanism: it provides "gentle" reminders only when the confidence of distraction is high, ensuring that the technology remains a helpful assistant that prioritizes maintaining the user's psychological flow over exhaustive but intrusive monitoring.
Taken together, three key insights emerge. First, Head Pose is a highly valuable feature that substantially improves four-class performance. Second, action units are detrimental under class imbalance, reducing SVM's Macro-F1 by more than 8 points. Third, Isolation Forest offers a superior solution for binary classification, with 61.43% Macro-F1 and a marked improvement in minority-class detection. Beyond their empirical value, these findings also have practical implications: lightweight models such as SVM and Isolation Forest are computationally efficient, interpretable, and deployable on standard hardware, making them attractive for large-scale online learning environments where resources are limited.
Nevertheless, limitations remain. Despite outperforming supervised alternatives, the Distracted class F1 under Isolation Forest remains modest (~28%), insufficient for high-stakes deployment. Furthermore, experiments were conducted solely on the DAiSEE dataset, restricting generalizability. Future work should expand to additional datasets, investigate more advanced anomaly detection methods (e.g., One-Class SVM, Deep SVDD), and explore robust feature extraction techniques that better resist noise and imbalance.
7. Conclusion
This study proposes a lightweight engagement detection approach that combines dynamic behavioral features, contextual normalization, and physical features to address inter-subject variation and class imbalance in webcam-based online learning environments. Experiments on the DAiSEE dataset showed that an SVM model with engineered features outperformed the LRCN deep learning baseline in the four-class classification task, while reframing disengagement as an anomaly detection problem, combined with normalization and physical features, significantly improved Isolation Forest's ability to recognize the rare "Distracted" class [36]. These results demonstrate that the proposed pipeline offers a practical, interpretable, and resource-efficient alternative to deep learning models, suitable for real-time deployment on standard CPU devices in large-scale online education.
Nevertheless, performance on minority classes remains modest, and evaluation was limited to a single dataset. Future work will focus on extending anomaly detection with alternative models and validating the approach across diverse datasets to enhance robustness and generalizability.
Acknowledgement
This research was funded by Hanoi University of Science and Technology under T2023-PC-078 Grant entitled "Research and Development of a Computer Vision-Based Model for Recognizing Learning Attitudes to Support Online Education at Hanoi University of Science and Technology".
Funding Information
This research was funded by Hanoi University of Science and Technology under T2023-PC-078 Grant entitled "Research and Development of a Computer Vision-Based Model for Recognizing Learning Attitudes to Support Online Education at Hanoi University of Science and Technology".
Author's Contributions
Dinh-Minh Vu and Viet-Hoang Tran designed the study and collected data. Dinh-Minh Vu, Viet-Hoang Tran and Thanh-Binh Pham analyzed the data. Dinh-Minh Vu and Viet-Hoang Tran wrote the paper. Dinh-Minh Vu and Hoang Hai Tran provided critical feedback and revised the manuscript. All authors approved the final version of the manuscript.
Ethics
Authors should address any ethical issues that may arise after the publication of this manuscript.
References
[1] Altuwairqi, K., Jarraya, S. K., Allinjawi, A., et al., "Student behavior analysis to measure engagement levels in online learning environments," Signal, Image and Video Processing, vol.15, pp.1387-1395, Oct. 2021. Article(CrossRefLink)
[2] Heikkinen, S., Saqr, M., Malmberg, J., et al., "A longitudinal study of interplay between student engagement and self-regulation," International Journal of Educational Technology in Higher Education, vol.22, 2025. Article(CrossRefLink)
[3] Lee, J., & Kim, D., "From awareness to empowerment: self-determination theory-informed learning analytics dashboards to enhance student engagement in asynchronous online courses," Journal of Computing in Higher Education, vol.37, pp. 1078-1118, Sep. 2025. Article(CrossRefLink)
[4] Buono, P., De Carolis, B., D'Errico, F., et al., "Assessing student engagement from facial behavior in on-line learning," Multimedia Tools and Applications, vol.82, pp.12859-12877, Apr. 2023. Article(CrossRefLink)
[5] Betto, I., Hatano, R., & Nishiyama, H., "Distraction detection of lectures in e-learning using machine learning based on human facial features and postural information," Artificial Life and Robotics, vol.28, pp.166-174, Feb. 2023. Article(CrossRefLink)
[6] Ma, J., Johnson, E. A., & McCrory, B., "Understanding Learning Engagement with User-Centered Human-Computer Interaction in a Multimodal Online Learning Environment," Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol.67, no.1, pp.2018-2023, 2023. Article(CrossRefLink)
[7] Fahid, F. M., Lee, S., Mott, B., Vandenberg, J., Acosta, H., Brush, T., Glazewski, K., Hmelo-Silver, C., & Lester, J., "Effects of Modalities in Detecting Behavioral Engagement in Collaborative Game-Based Learning," in Proc. of 13th International Learning Analytics and Knowledge Conference (LAK2023), pp.208-218, 2023. Article(CrossRefLink)
[8] Soukupova, T., & Cech, J., "Eye-Blink Detection Using Facial Landmarks," in Proc. of 21st Computer Vision Winter Workshop, vol.2, Rimske Toplice, Slovenia, 2016. Article(CrossRefLink)
[9] Munasinghe, M. I. N. P., "Facial Expression Recognition Using Facial Landmarks and Random Forest Classifier," in Proc. of 2018IEEE/ACIS 17th International Conference on Computer and Information Science (ICIS), pp.423-427, Singapore, 2018. Article(CrossRefLink)
[10] Bellamkonda, S., & Gopalan, N.P., "A Facial Expression Recognition Model using Support Vector Machines," International Journal of Mathematical Sciences and Computing (IJMSC), vol.4, no.4, pp.56-65, Nov. 2018. Article(CrossRefLink)
[11] Buhler, B., Bozkir, E., Deininger, H., Gerjets, P., Trautwein, U., & Kasneci, E., "On Task and in Sync: Examining the Relationship between Gaze Synchrony and Self-reported Attention During Video Lecture Learning," Proceedings of the ACM on Human-Computer Interaction, vol.8, no.ETRA, May 2024. Article(CrossRefLink)
[12] Gupta, A., D'Cunha, A., Awasthi, K., & Balasubramanian, V., "DAiSEE: Towards User Engagement Recognition in the Wild," arXiv preprint arXiv:1609.01885, 2016. Article(CrossRefLink)
[13] Donahue, J., et al., "Long-Term Recurrent Convolutional Networks for visual recognition and description," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.39, no.4, pp.677-691, Apr. 2017. Article(CrossRefLink)
[14] Yang, F., "A Spatio-Temporal Attention-Based Method for Detecting Student Classroom Behaviors," arXiv preprint arXiv:2310.02523, 2023. Article(CrossRefLink)
[15] Selim, T., Elkabani, I., & Abdou, M. A., "Students Engagement Level Detection in Online e-Learning Using Hybrid EfficientNetB7 Together With TCN, LSTM, and Bi-LSTM," IEEE Access, vol.10, pp.99573-99583, 2022. Article(CrossRefLink)
[16] Kollias, D., Sharmanska, V., & Zafeiriou, S., "Distribution Matching for Multi-Task Learning of Classification Tasks: A Large-Scale Study on Faces & Beyond," in Proc. of the AAAI Conference on Artificial Intelligence, vol.38, no.3, pp.2813-2821, 2024. Article(CrossRefLink)
[17] Xie, N., Liu, Z., Li, Z., Pang, W., & Lu, B., "Student engagement detection in online environment using computer vision and multi-dimensional feature fusion," Multimedia Systems, vol.29, pp.3559-3577, Dec. 2023. Article(CrossRefLink)
[18] Albadawi, Y., Takruri, M., & Awad, M., "A Review of Recent Developments in Driver Drowsiness Detection Systems," Sensors, vol.22, no.5, Mar. 2022. Article(CrossRefLink)
[19] Feng, F., "Deep learning-based model for analyzing student engagement in activities," Scientific Reports, vol.16, 2026. Article (CrossRefLink)
[20] Qamar, T., & Bawany, N. Z., "Understanding the black-box: towards interpretable and reliable deep learning models," PeerJ Computer Science, vol.9, 2023. Article(CrossRefLink)
[21] Baltrusaitis, T., Robinson, P., & Morency, L.-P., "OpenFace: An open source facial behavior analysis toolkit," in Proc. of 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pp.1-10, 2016. Article(CrossRefLink)
[22] Trabelsi, Z., Alnajjar, F., Parambil, M. M. A., Gochoo, M., & Ali, L., "Real-Time Attention Monitoring System for Classroom: A Deep Learning Approach for Student's Behavior Recognition," Big Data and Cognitive Computing, vol.7, no.1, Mar. 2023. Article(CrossRefLink)
[23] Jabberi, M., Wali, A., Chaudhuri, B. B., & Alimi, A. M., "68 landmarks are efficient for 3D face alignment: what about more?," Multimedia Tools and Applications, vol.82, pp.41435-41469, Nov. 2023. Article(CrossRefLink)
[24] Xiang, X., Cheng, Y., Xu, S., Lin, Q., & Allebach, J., "The Blessing and the Curse of the Noise behind Facial Landmark Annotations," arXivpreprint arXiv:2007.15269, 2020. Article(CrossRefLink)
[25] Mehta, S., Dadhich, S., Gumber, S., & Jadhav Bhatt, A., "Real-Time Driver Drowsiness Detection System Using Eye Aspect Ratio and Eye Closure Ratio," in Proc. of International Conference on Sustainable Computing in Science, Technology and Management (SUSCOM), Jaipur, India, Feb. 2019. Article(CrossRefLink)
[26] Sri Mounika, T. V. N. S. R., Phanindra, P. H., Sai Charan, N. V. V. N., Kranthi Kumar Reddy, Y., & Govindu, S., "Driver Drowsiness Detection Using Eye Aspect Ratio (EAR), Mouth Aspect Ratio (MAR), and Driver Distraction Using Head Pose Estimation," in Proc. of ICT Systems and Sustainability (ICT4SD 2021), Lecture Notes in Networks and Systems, vol.321, pp.619-627, Springer, Singapore, 2022. Article(CrossRefLink)
[27] Demircioglu, A., "The effect of feature normalization methods in radiomics," Insights into Imaging, vol.15, no.1, 2024. Article(CrossRefLink)
[28] Goyal, K., Dyer, C., & Berg-Kirkpatrick, T., "An Empirical Investigation of Global and Local Normalization for Recurrent Neural Sequence Models Using a Continuous Relaxation to Beam Search," in Proc. of 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp.1724-1733, 2019. Article(CrossRefLink)
[29] Liu, F. T., Ting, K. M., & Zhou, Z.-H., "Isolation Forest," in Proc. of the Eighth IEEE International Conference on Data Mining (ICDM 2008), pp.413-422, Pisa, Italy, 2008. Article(CrossRefLink)
[30] Sokolova, M., & Lapalme, G., "A systematic analysis of performance measures for classification tasks," Information Processing & Management, vol.45, no.4, pp.427-437, Jul. 2009. Article(CrossRefLink)
[31] Chen, C., Zhang, P., Zhang, H., Dai, J., Yi, Y., Zhang, H., & Zhang, Y., "Deep Learning on Computational-Resource-Limited Platforms: A Survey," Mobile Information, 2020. Article(CrossRefLink)
[32] Hoang, L., Shorten, G., O'Sullivan, B., & Nguyen, H. D., "A lightweight and reliable framework toward real-time student engagement predictions in learning analytics," in Proc. of Reliable and Trustworthy Artificial Intelligence Workshop at the 16th Asian Conference on Machine Learning (ACML 2024), Hanoi, Vietnam, Dec. 2024. Article(CrossRefLink)
[33] Tan, M., & Le, Q., "EfficientNetV2: Smaller Models and Faster Training," in Proc. of 38th International Conference on Machine Learning, PMLR, vol.139, pp.10096-10106, 2021. Article(CrossRefLink)
[34] Sandler, M., Howard, A., Zhu, M., et al., "Mobilenetv2: Inverted Residuals and Linear Bottlenecks," in Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.4510-4520, Salt Lake City, Jun. 2018. Article(CrossRefLink)
[35] Shiri, F. M., Ahmadi, E., Rezaee, M., & Perumal, T., "Detection of Student Engagement in E-Learning Environments Using EfficientnetV2-L Together with RNN-Based Models," Journal on Artificial Intelligence, vol.6, pp.85-103, 2024. Article(CrossRefLink)
[36] Al Farizi, W. S., Hidayah, I., & Rizal, M. N., "Isolation Forest Based Anomaly Detection: A Systematic Literature Review," in Proc. of the 8th International Conference on Information Technology, Computer and Electrical Engineering (ICITACEE 2021), pp.118-122, Semarang, Indonesia, 2021. Article(CrossRefLink)
Dinh-Minh Vu received his B.Eng. degree in Information Technology from the Open University of Vietnam, his M.Eng. degree in Information Technology from Hanoi University of Science and Technology, Vietnam, and his Ph.D. degree from Shibaura Institute of Technology, Japan.
His research interests include data mining, machine learning, natural language processing, learning analytics, and process mining. His recent work focuses on the application of artificial intelligence in education, particularly in analyzing learning behaviors and student engagement.
He is currently a lecturer at the Faculty of Education, Hanoi University of Science and Technology. Since 2023, he has also been involved in administrative management at the university.
Viet-Hoang Tran received the B.Eng. degree in Information Technology from the School of Information and Communication Technology, Hanoi University of Science and Technology, Vietnam.
His current research interests include large language models (LLMs), AI agent-based systems, robot learning, computer vision, and reinforcement learning.
Thanh-Binh Pham received the B.Eng. degree in Information Technology from the School of Information and Communication Technology, Hanoi University of Science and Technology, Vietnam, in 2025. He is currently working as a Data Science Engineer at Viettel Telecommunications Corporation.
His current research interests include computer vision, recommender systems, large language models (LLMs), and AI agent-based systems.
Hoang-Hai Tran received his B.Eng. degree in Information Technology from Hanoi University of Science and Technology, Vietnam, in 2004, and his M.Eng. degree in Computer Engineering from Kyung Hee University, Republic of Korea, in 2008. He later worked at the joint laboratory between Alcatel-Lucent Bell Labs and INRIA, France, and received his Ph.D. degree in Computer Science in 2012.
His research interests include cybersecurity, secure routing, and resource allocation mechanisms for next-generation Internet systems. His recent work focuses on the application of AI models to intrusion detection, network system security, and intelligent network resource allocation. Since 2020, he has published a number of papers in these research areas. He is currently a lecturer at the Department of Computer Engineering, School of Information and Communications Technology, Hanoi University of Science and Technology. Since 2023, he has also served as Associate Director of Administration Office at Hanoi University of Science and Technology.
Dinh-Minh Vu (1), Viet-Hoang Tran (2), Thanh-Binh Pham (2), and Hoang-Hai Tran (2*)
(1) Faculty of Education, Hanoi University of Science and Technology Ha Noi, Viet Nam
(2) School of Information and Communication Technology, Hanoi University of Science and Technology
Ha Noi, Viet Nam
[e-mail: minh.vudinh@hust.edu.vn, haith@soict.hust.edu.vn]
(*) Corresponding author: Hoang-Hai Tran
Received September 27, 2025; revised February 2, 2026; revised March 27, 2026; accepted April 21, 2026; published July 31, 2026
http://doi.org/10.3837/tiis.2026.07.005
Table 1. Overview of research on distraction detection.
Stage Representat Approach Main Contribution
ive Studies
Stage 1: Soukupova Geometric Feature Introduction of the
Traditio & Cech Extraction EAR method, a
nal (2016) Technique (Eye standard for real-time
Machine Aspect Ratio -EAR). blink detection based
on facial landmark
points.
Learning Munasinghe, Classical Machine Demonstrating the
with (2018) Learning (SVM, feasibility of using
Static Bellamkond Random Forest) standard webcams to
Features a & on static build simple
Gopalan, geometric distraction-detection
(2018) features. tools.
Gupta et al. Introduction of the Providing a standard
(2016) DAiSEE dataset "in-the-wild" dataset
and the for the community.
establishment of Establishing an
baseline important
performance using performance
Deep Learning benchmark for future
(LRCN). research.
Stage 2: Kollias & Multi-task Building powerful
Deep Zafeiriou Learning for deep learning models
Learning (2021) emotion capable of
with recognition and simultaneously
implicit related Tasks. handling multiple
temporal tasks on "in-the-wild"
modeling data.
Yang et al Spatio-Temporal Improving
(2021) Attention-Based performance by
LSTM Network. simultaneously
modeling both spatial
and temporal
variations of facial
features.
Xie et al. Multi- Proposed a deep
(2023) dimensional fusion framework
Feature Fusion that integrates facial
(VGG16 for face expression and head
+ ResNet-101 for pose features to
head pose). improve robustness
against
single-modality failure.
Selim, Deep Learning Achieves high
Elkabani, & Hybrid Model accuracy by
Abdou (CNN combined combining the spatial
(2022) with feature extraction
LSTM/Bi-LSTM/TCN). capability of CNNs
with the
sequence-processing
ability of
recurrent networks.
Stage 3: Trabelsi et Transformer-based Applied state-of-the-art
al. (2023) Object Vision Transformers
Detection (RT-DETR) (ViT) to detect student
for behavior behaviors in real-time,
recognition. outperforming
traditional CNNs in
complex scenes.
Hybrid Feng et al. The model The paper proposes
and (2025) integrates Multi- the IC-BiSGRU-Net
multi- task Learning for framework,
task emotion integrating Bi-LSTM,
models recognition with a SGRU, and
Hybrid CNN- Intelligent Chimp
LSTM/TCN optimization to
architecture and achieve high-accuracy
RT-DETR for real-time
real-time behavior classification of
detection. student engagement
states using
multimodal data
Kollias & Multi-task Building powerful
Zafeiriou Learning for deep learning models
(2021) Emotion capable of
Recognition and simultaneously
Related Tasks. handling multiple
tasks on "in-the-wild"
data.
Stage Representat Key Gap
ive Studies
Stage 1: Soukupova The method itself is a static feature. When
Traditio & Cech used in isolation, it overlooks important
nal (2016) behavioral dynamics (e.g., it cannot
Machine distinguish between rapid blinking and
slow eye closure due to fatigue).
Learning Munasinghe, Treating behavior as a static state, without
with (2018) the ability to capture temporal information,
Static Bellamkond leads to limited accuracy and reliability.
Features a &
Gopalan,
(2018)
Gupta et al. The proposed baseline models (e.g., LRCN)
(2016) inherently carry the limitations of deep
learning: high complexity and a "black-box"
nature, setting a high-performance target
but at the cost of significant
computational resources.
Stage 2: Kollias & Focusing on predictive performance, but
Deep Zafeiriou still facing the inherent issues of deep
Learning (2021) learning, such as the "black-box"
with nature and high computational resource
implicit requirements.
temporal
modeling
Yang et al The model operates as a "black box," making
(2021) interpretation difficult. It requires
substantial computational resources (GPUs)
and large datasets for training.
Xie et al. High computational redundancy: Using
(2023) two heavy backbones (VGG + ResNet) creates
significant latency. The fusion is performed
at a static level, missing the fine-grained
temporal evolution of student attention.
Selim, High computational complexity and long
Elkabani, & training times. The model has very low
Abdou interpretability, making deployment on
(2022) typical personal devices challenging.
Stage 3: Trabelsi et Excessive Resource Requirement: Transformer
al. (2023) models require substantial GPU power and
memory, making them unsuitable for
standard personal laptops or web-based
deployment (Edge AI constraints).
Hybrid Feng et al. Existing methods are often subjective or fail
and (2025) to effectively integrate multimodal
multi- behavioral cues, resulting in limited
task generalizability and high computational costs
models in real-world digital environments.
Kollias & Focusing on predictive performance, but
Zafeiriou still facing the inherent issues of deep
(2021) learning, such as the "black-box" nature and
high computational resource requirements.
Table 2. Comparative analysis of engagement detection methodologies
based on feature types and model architectures.
Approach Typical Techniques Key Strengths
Deep Learning CNNs (EfficientNet, High accuracy, autonomous
MobileNetV2), RNNs feature extraction from raw
(LSTM, GRU), 3D- pixels.
CNN, or Hybrid Models
(LRCN).
Affective-based Emotion recognition Connects engagement with
(FER), Adaptation psychological states.
policies
Geometric-based Geometric features Low computational cost
(EAR, MAR, Pose) + (CPU-friendly); highly
Classical ML (SVM, RF, interpretable behavior.
Isolation Forest)
Approach Limitations
Deep Learning High computational cost; "black-box" nature; requires
specialized hardware.
Affective-based Subjective mapping; sensitive to environmental noise.
Geometric-based Dependent on the accuracy of facial landmark
detection.
Table 3. Progressive Feature Set Configurations for
Distraction/Engagement Detection
Feature Set Included Features Description
Set 1 EAR, MAR, Gaze Basic ocular and mouth-related
geometric features
Set 2 Set 1 + Head Pose Adds head orientation
information
Set 3 Set 2 + AU45 (Blink) Incorporates eye blinking
dynamics
Set 4 Set 3 + AU25 (Lips Part) Adds mouth openness and jaw
+ AU26 (Jaw Drop). movement cues
Table 4. Scenarios, Experimental Setup, and Model Configuration
Experimental Scenario 1 (4-class
Setup Classification)
Data Balancing * Oversampling: The Synthetic Minority Over-sampling
Strategy Technique (SMOTE) was applied to the training set
(random_state=42) to balance the distribution of
engagement classes.
* Cost-Sensitive Learning: Models including SVM,
Random Forest, and Decision Tree utilized the
class_weight='balanced' parameter to assign higher
penalties to misclassifications of the minority
classes.
Machine * Personal Computer (11th Gen Intel[R] Core[TM] i5 CPU,
configurations 16GB RAM).
* Python 3.13.1, OpenFace 2.2.0, Scikit-learn 1.6.1,
XGBoost 3.0.2, NumPy 2.2.2, Pandas 2.2.3
Model * Random Forest & XGBoost:
Configurations Configured with 100 estimators. XGBoost: Utilized the
multi-class log loss (mlogloss) as the evaluation
metric.
* K-Nearest Neighbors (KNN): Set with k=5 neighbors.
* SVM: Implemented with a Radial Basis Function (RBF)
kernel and probability=True.
Experimental Scenario 2 (2-class Classification)
Setup
Data Balancing
Strategy
Machine
configurations
Model * Supervised Learning: SVM (RBF kernel), Random
Configurations Forest, and XGBoost were retrained on binary labels
with 100 estimators.
* Isolation Forest (Anomaly Detection):
Trained in an unsupervised manner using 100 estimators
and a contamination=0.05 parameter to reflect
the natural ratio of the "Distracted" class.
Threshold Optimization:
* Threshold Sweep: A comprehensive sweep was performed
from 0.05 to 1.0 with an increment of 0.05 to find the
optimal decision boundary.
* Optimal Selection: The final threshold was determined
by maximizing the Macro F1-score to ensure a balance
between precision and recall.
* Score Normalization: For the Isolation Forest,
anomaly scores were normalized to the [0, 1] range
prior to the threshold sweep for consistency with
supervised models.
Table 5. Performance of Four-Class Classification Models on the Test
Data
Feature Set Model Accuracy (%) F1-score (%)
1 EAR + MAR + Random Forest 50.18 24.80
GAZE XGBoost 49.45 25.44
(3 features) SVM 52.19 25.11
DecisionTree 44.56 23.59
KNN 51.22 27.58
2 EAR + MAR + Random Forest 54.51 24.76
GAZE XGBoost 56.47 30.45
+ Head Pose SVM 58.60 34.42
(4 features) DecisionTree 50.48 31.11
KNN 49.51 26.19
3 EAR + MAR + Random Forest 51.15 23.37
GAZE XGBoost 51.52 25.02
+ Head Pose + AU45 SVM 52.38 25.95
(5 features) DecisionTree 47.31 28.03
KNN 46.21 25.23
4 EAR + MAR + Random Forest 53.29 24.19
GAZE XGBoost 52.62 25.85
+ Head Pose + AU45 SVM 52.93 26.18
+ AU25 + AU26 DecisionTree 46.70 24.72
(7 features) KNN 46.82 24.53
Table 6. Accuracy Comparison Models Status / Configuration Accuracy (%) SVM (proposed model) Engagement 58.60 InceptionNet Frame Level (DAiSEE Engagement 47.10 Baseline) InceptionNet Video Level (DAiSEE Engagement 46.40 Baseline) C3D Training (DAiSEE Baseline) Engagement 48.60 C3D FineTuning (DAiSEE Baseline) Engagement 56.10 Improved Engagement 57.49 MobileNetV2+MUSER+LSTM [32] LRCN (DAiSEE Baseline) Engagement 57.90 [9] EfficientNetV2-L + Bi - GRU [33] Engagement 61.56 EfficientNetV2-L + LSTM [33] Engagement 62.11
Table 7. Performance of Binary-Class Classification Models on the Test
Data.
Feature Set Model Accuracy F1-score Precision
(%) (%) (%)
1 EAR + Random Forest 90,59 6,09 6,32
MAR+GA XGBoost 88,76 8.70 6,50
ZE SVM 84,79 5.32 3,93
(3
features) DecisionTree 73.01 9.42 5,70
KNN 76,31 12,61 7,79
Isolation 90,76 25,30 20,43
Forest
EAR + Random Forest 89,31 5,83 4,51
2 MAR + XGBoost 85,83 6,47 4,66
GAZE SVM 87,36 8,26 6,42
+ Head
Pose DecisionTree 74,42 8,81 5,34
(4 features) KNN 74,48 10,68 6,52
Isolation 92,89 26,55 23,31
Forest
EAR + Random Forest 92,42 5,29 6,06
3 MAR + XGBoost 90,72 8,34 8,64
GAZE SVM 91,02 6,47 8,1
+ Head
Pose + DecisionTree 76,86 7,78 4,90
AU45 KNN 75,64 11,13 6,86
(5 features) Isolation 92,46 25,06 22,94
Forest
EAR + Random Forest 93,52 8,82 8,00
4 MAR + XGBoost 89,49 8,90 7,69
GAZE SVM 91,39 7,84 8,82
+ Head
Pose + DecisionTree 75,82 10,40 6,12
AU45 KNN 71,18 14,64 9,51
+ AU25 + Isolation 92,24 28,26 27,87
AU26 Forest
(7
features)
Feature Set Model Recall F1-score (%)
(%) (macro-avg)
1 EAR + Random Forest 5,88 50,57
MAR+GA XGBoost 12,94 48,84
ZE SVM 8,23 47,11
(3
features) DecisionTree 27,05 46,79
KNN 32,9 45,79
Isolation 33,25 60,26
Forest
EAR + Random Forest 8,23 49,19
2 MAR + XGBoost 10,58 49,01
GAZE SVM 11,58 50,06
+ Head
Pose DecisionTree 23,52 46,92
(4 features) KNN 29,41 47,89
Isolation 30,84 59,97
Forest
EAR + Random Forest 4,70 23,37
3 MAR + XGBoost 8,23 25,02
GAZE SVM 7,05 25,95
+ Head
Pose + DecisionTree 18,82 47,27
AU45 KNN 29,41 48,51
(5 features) Isolation 27,63 60,53
Forest
EAR + Random Forest 9,85 50,14
4 MAR + XGBoost 10,58 51,46
GAZE SVM 7,05 51,66
+ Head
Pose + DecisionTree 31,76 46,51
AU45 KNN 31,82 46,55
+ AU25 + Isolation 28,67 61,43
AU26 Forest
(7
features)
Table 8. Detailed performance of the 'Focused' and 'Distracted' classes
for the Isolation Forest model with Feature Set 4 on the binary
classification task.
Feature Set Class Fl-score (%) Precision Recall Sample
(%) (%) Size
EAR + MAR + Distracted 28.26 27.87 28.67 88
GAZE + Head Focused 96.41 96.37 96.46 1696
Pose + AU45 +
AU25 + AU26
(7 features)
Table 9. Feature Importance Ranking by Functional Groups.
Rank Feature Group Key Representative Importance Score
Features Range (Min - Max)
1 Head Pose var_pose_Rx, 0.011 - 0.079
mean_pose_Rz
2 Eye (EAR) zmin_EAR, 0.010 - 0.061
zrangeEAR,
mean_EAR
3 Gaze var_gaze_0_x, 0.012 - 0.044
var_gaze_1_x
4 Mouth (MAR) meanMAR, 0.010 - 0.027
var_MAR
5 Action Units (AU) mean_AU45_r, 0.005 - 0.033
mean_AU25_r
Rank Feature Group Key Representative Role in Model
Features
1 Head Pose var_pose_Rx, Primary Discriminator
mean_pose_Rz
2 Eye (EAR) zmin_EAR, Primary Discriminator
zrangeEAR,
mean_EAR
3 Gaze var_gaze_0_x, Core Feature
var_gaze_1_x
4 Mouth (MAR) meanMAR, Core Feature
var_MAR
5 Action Units (AU) mean_AU45_r, Auxiliary / Noise-prone
mean_AU25_r
 Printer friendly
 Cite/link
 Email
 Feedback
| |
| Author: | Vu, Dinh-Minh; Tran, Viet-Hoang; Pham, Thanh-Binh; Tran, Hoang-Hai |
|---|---|
| Publication: | KSII Transactions on Internet and Information Systems |
| Article Type: | Report |
| Geographic Code: | 9VIET |
| Date: | Jul 1, 2026 |
| Words: | 10525 |
| Previous Article: | Validated Multi-Modal Empathy Detection via Rank-Preserved Graph Fusion and Hybrid Attention Optimizations. |
| Next Article: | Instantaneous Production Prediction and Control-Parameter Optimization in Trailing Suction Hopper Dredgers Based on Multi-Model Feature Fusion. |
| Topics: | |