Segmentation-Based Human Tracking and Human Action Recognition Using Optimized Deep Residual Convolutional Bi-Directional Long Short-Term Memory.
1. IntroductionH uman behaviour forms the foundation of social communication, encompassing a wide range of activities and interactions performed by individuals in daily life. The rapid evolution of deep learning and computer vision techniques has paved the way for advanced systems capable of automatically detecting human actions from video content, emphasizing the growing importance of this domain in real-world scenarios [1-2].
Automated HAR systems analyze sequential video frames to detect spatio-temporal patterns in movements, allowing precise classification of activities across diverse scenarios. Recognition of human actions involves detecting and classifying the various activities performed by individuals [3-4]. According to recent research, more than 70% of security events occurring in public areas involve human behavior that is identified earlier by automated video processing, underlining the potential use of real-time HAR in preventing accidents, crimes, and dangerous events [5].
In many situations, relying on human observation to monitor actions is impractical, especially in industrial workplaces, hospitals, or crowded areas, due to fatigue, delayed responses, or risk exposure [6]. Video-based action recognition systems have proved effective in easing such challenges of HAR by closely observing and accurately judging events in realtime, with no danger to human life and a quick reaction to developing exigency conditions [7]. The other major domain in which HAR has made a positive contribution to the task of observation and detection is towards observation of unsafe behaviors, allowing safety to be controlled proactively and supporting increased efficiency in operations.
1.1 Contributions
* This research introduces a hybrid data acquisition strategy combining benchmark datasets and live video feeds from surveillance cameras and webcams. Through the integration of the Red, Green, Blue (RGB), depth, and skeletal input modalities, it provides diverse, dynamic, and unconstrained human action data.
* The present research incorporates a DSORT-SQA model. Human masks are accurately extracted and tracked in video frames, ensuring temporal consistency. The model is capable of handling occlusions, crowded scenes, and motion behaviors, thereby generating accurate human trajectories for feature and activity extraction.
* To effectively perform the task of HAR, this research combined the DRCBi-LSTM network with hyperparameter tuning using IGBOA techniques. The model is deployed on hybrid hardware environments, which enable edge devices for real-time computing. To enhance model interpretability, Grad-CAM and temporal attention mechanisms are also used.
1.2 Literature Survey
Despite challenges, the techniques show impressive performance in human action recognition, as shown in Table 1. A technique for the recognition of human behavior was developed by Liu et al. [8], who used Convolutional Neural Networks (CNNs). In addition, Abdelbaky and Aly [9] proposed a technique called PCANet, which is an unsupervised approach for the extraction of spatial and temporal features from action videos. Similarly, Sowmyayani and Rani [10] introduced STHARNet, which captures spatial features from frames and temporal motion across them for HAR. Berlin and John [11] designed a Deep Learning Neural Network (DLNN) for HAR, focusing on efficiently recognizing and classifying human activity.
Elharrouss et al. [12] developed an efficient Temporal Difference Map (TDMap) based on a CNN architecture, which used video to detect, recognize, and summarize multiple human actions.
Zaidi et al. [13] put forward a complex method for detecting and recognizing suspicious human activities in the context of video monitoring. Nasir et al. [14] introduced an Elastic Net-based Genetic Algorithm (ENGA) method that enhanced the efficiency of feature selection while ensuring a narrow feature set from a very specific search algorithm. Wang et al. [15] proposed a 3D CNN to recognize human action in a video. It had two subnetworks for extracting temporal and deep spatial information. Shanableh [16] proposed ViCoMoCo-DL for human activity recognition using deep learning. Ullah and Munir [17] proposed an Attention-driven Residual Three-Dimensional CNN (AR3DCNN) network, which learns human action representation. The complicated spatiotemporal dependencies of the video sequence are captured. Karim et al. [18] came up with HADE, a model-based strategy that exploited HAR through fine-tuned deep learning methods. Wang et al. [19] developed SDWM-HTM. Spatiotemporal features are dynamically assigned weights.
1.3 Problem statement
Paramount to effective identification and classification of human actions in live video surveillance is the complexity of interaction and the dynamic context in which these actions occur. This makes it difficult to interpret the complex spatio-temporal patterns in various existing approaches to achieve high levels of classification accuracy. This affects the functionality of these systems in critical environments like airports and prisons, where the efficiency of these systems in identifying actions is paramount to improving the security threshold of the environment. Therefore, a need for more innovative approaches to interpret multi-human interactions as required in the modern security environment.
1.4 Organization
The organization of this research is arranged as follows: Section 2 details the approach used, Section 3 reports the results with corresponding analysis; Section 4 offers the conclusion and future research directions.
2. Suggested methodology
The proposed research framework involves data gathering using a range of benchmark datasets, like KTH, UT-Kinect, UCF-11, and HMDB51 datasets, coupled with the acquisition of videos. The data undergoes preprocessing techniques such as extracting frames from videos, resizing, and data normalization. The process flow for the proposed approach is depicted in Fig. 1.
The DSORT-SQA ensures accurate human localization through human tracking using the segmentation approach. In the feature extraction phase, the methods used are mutual information, entropy, and correlation coefficients, optimized using the IGBOA. The DRCBi-LSTM network ensures the spatiotemporal representation for the human activity recognition task. Grad-CAM is used for the interpretation of the model. Hardware acceleration is used to ensure high performance and interpretation.
2.1 Data gathering
Input video data used for HAR is sourced from four distinct and widely recognized datasets, namely Kungliga Tekniska H[delta]gskolan (KTH), UT-Kinect, UCF11, and Human Motion Database (HMDB51) [19]. Detailed overview of the datasets added in Table 2.
The benchmark datasets, along with real-time videos, were collected from various environments such as indoor and outdoor areas, including offices, laboratories, and outdoor areas such as university corridors. The dataset contains 12 common human actions, which have been annotated with standardized formats with synchronized multimodal inputs. For the datasets used, the exact releases are specified as follows: KTH, UT-Kinect, and UCF-11, with the UCF-11 release being used instead of UCF101. Additionally, the preprocessing steps include a fixed train/test split of 70% for training and 30% for testing, with no randomness applied to the split. These specifications ensure reproducibility and clarity regarding the dataset handling and preprocessing.
Table 3 provides a clear summary of the live data characteristics, including details on resolution, action classes, environmental conditions, and sample size. These additions ensure the reproducibility of the data collection process.
2.2 Pre-processing and Adaptive Data Normalization
Multimodal human activity videos are pre-processed and normalized by extracting sequential frames, ensuring temporal synchronization. Frames are resized to reduce computational cost, and intensity normalization mitigates lighting variations, as described in (1).
[Please download the PDF to view the formula] (1)
where, [F.sub.i,t](p, q) signifies the pixel intensity at (p,q), and [[mu].sub.i,t], [[sigma].sub.i,t] denote the mean and standard deviation of frame intensities, respectively. The fused frame [G.sub.i,t] is expressed as in (2).
where, [alpha] + [beta] + [gamma] = 1, with [R.sub.i,t], [D.sub.i,t], and [O.sub.i,t] representing RGB, depth, and optical flow modalities, respectively.
Finally, the pre-processed dataset is represented as a structured tensor illustrated in (3).
[Please download the PDF to view the formula] (3)
where, D contains temporally aligned, spatially normalized, and multimodally enriched frames, providing an optimal input foundation for subsequent segmentation-based human tracking and action recognition.
2.3 Segmentation-Based Human Tracking using Deep Simple Online and RealTime Tracking with Segmentation Quality Assessment
Once the data have been preprocessed, the human tracking algorithm used is called DSORT-SQA, combining Segmentation Quality Assessment (SQA) and Motion Persistence Evaluation (MPE). This innovative technique boosts the association and segmentation performance in order to make sure that the trajectory is continuous. DSORT-SQA increases the precision and robustness in tracking when compared to DeepSORT, as evidenced by (4).
[Please download the PDF to view the formula] (4)
where, [f.sub.SOA] denotes the trained SQA function. The predicted state for the next frame is computed as in (5).
[Please download the PDF to view the formula] (5)
where, A is the state transition matrix assuming constant velocity, and [w.sub.t] is process noise.
Appearance descriptors [phi]([b.sub.t]) extracted using a CNN encode identity-specific features, allowing robust association of new detections with existing tracks. The DSORT-SQA model architecture, where new object detections are assigned to ongoing tracks via a joint cost function defined as in (6), (7), and (8).
[Please download the PDF to view the formula] (6)
[Please download the PDF to view the formula] (7)
[Please download the PDF to view the formula] (8)
where, [lambda] balances spatial and appearance similarity. To improve trajectory consistency, the Z-Wasserstein distance [W.sub.z] is used to measure the discrepancy between the predicted motion distribution P and observed motion distribution [.sup.Q]defined as in (9)
[Please download the PDF to view the formula] (9)
where, [PI](P,Q) signifies the set of joint distributions with marginals P and Q, and [[parallel]*[parallel].sub.2] is the chosen norm. Finally, the integrated DSORT-SQA with Motion Persistence Evaluator (MPE) validates and maintains tracks across frames, producing the final set of accurately segmented and temporally consistent human regions [T.sub.i,j] for each frame [20-21]. This ensures that only high-quality, properly segmented human figures are used for subsequent feature selection and HAR. MPE works by using thresholds to detect significant motion deviation or prolonged occlusion. If motion deviation exceeds a predefined threshold or occlusion persists for more than a specified duration, the track is suspended. Below is a pseudocode snippet for MPE validation:
MPE Track Validation Pseudocode
if motion_deviation > threshold or occlusion_duration > max_occhision_time: suspend_track(track_id) # Suspend tracking if deviation or occlusion exceeds limits else:
continue_tracking(track_id) # Continue tracking if within valid thresholds
This process ensures robust tracking during occlusions and motion deviations, significantly contributing to the accuracy and stability of the action recognition system. DSORT-SQA, with MPE, enhances the reliability of tracking, ensuring that only high-quality, correctly segmented human regions are used for subsequent feature selection and HAR.
2.4 Feature extraction and selection
After segmentation, features are extracted using the HR-STFE, capturing spatial and temporal characteristics. Stacked Bi-LSTM layers encode short-term motion dynamics and temporal dependencies, with the temporal representation computed as in (10).
[Please download the PDF to view the formula] (10)
where, [Please download the PDF to view the mathematical expression] and [d.sub.Temporal] denotes the Bi-LSTM hidden state dimension. The final feature vector for each frame combines spatial and temporal cues shown in (11).
[Please download the PDF to view the formula] (11)
where, [S.sup.flat.sub.i,t] is the flattened spatial feature vector, and [alpha], [beta] are weighting coefficients. Features from multiple modalities, such as RGB, depth, and optical flow, are integrated as in (12).
[Please download the PDF to view the formula] (12)
where, [F.sup.(m).sub.i,t] is the feature vector from modality m, [w.sub.m] is its weight, and M is the number of modalities [22]. An attention mechanism generates dynamic weights for each example, as shown in (13).
[Please download the PDF to view the formula] (13)
where, [[alpha].sub.m]([f.sub.m]) is the learned attention weight for modality m, which varies based on the context of the input, and [f.sub.m] is the feature vector from modality m. Relative entropy for a feature [F.sub.i] is expressed as in (14).
[Please download the PDF to view the formula] (14)
where, [n.sub.j] is the number of discrete bins for [F.sub.i], P([F.sup.J.sub.I])is the probability of the bin, and Q([F.sup.j.sub.i]) is the reference distribution. Mutual information between two features [F.sub.I] and [f.sub.j]is defined as in (15).
[Please download the PDF to view the formula] (15)
where, P(x,y) is the joint probability, P(x), p(y) are the marginal probabilities. SCC is expressed as in (16).
[Please download the PDF to view the formula] (16)
where, [Please download the PDF to view the mathematical expression] are standard deviations, [Please download the PDF to view the mathematical expression] and [Please download the PDF to view the mathematical expression],are mean values, and N is the sample count.
An adaptive maximum probability-based threshold function selects only the most significant features shown in (17).
[Please download the PDF to view the formula] (17)
where, [F*.sub.k] is the selected feature, [theta](.) ignores low-probability or negative features. The selected features from all modalities are combined using weighted integration defined in (18).
[Please download the PDF to view the formula] (18)
where, Fk, m is the selected feature from the modality (m), (w)m is the weight. The resulting optimized feature set [F.sup.selected.sub.i,t] is compact, informative, and ready for HAR, ensuring high classification accuracy with reduced computational complexity.
2.5 Classification using Deep Residual Convolutional Bi-directional Long Short-Term Memory
The DRCBi-LSTM classifier categorizes human actions, capturing both spatial and temporal dependencies. Unlike CNNs, it overcomes LSTM's unidirectional limitation by processing sequences bi-directionally [23-24]. For the input sequence [F.sup.selected] = {[F.sub.1], [F.sub.2],..., [F.sub.T]}, the forward hidden state [Please download the PDF to view the mathematical expression] and backward hidden state [Please download the PDF to view the mathematical expression] are given as in (19) and (20).
[Please download the PDF to view the formula] (19)
[Please download the PDF to view the formula] (20)
The hidden states are concatenated to obtain the combined temporal representation shown in (21).
[Please download the PDF to view the formula] (21)
To alleviate this, residual connections are introduced, F(x) instead of the full transformation shown in (22).
y - F (x) + x, (22)
Residual mapping stabilizes gradient flow, while Layer Normalization (LN) accelerates training by reducing internal covariate shift, as shown in (23).
[Please download the PDF to view the formula] (23)
where, [[mu].sub.x] and [[sigma].sub.x] are the mean and standard deviation of the input X, [gamma], [micro] are learnable parameters.
The recursive update for each residual Bi-LSTM layer is shown and explained in (24), (25), and (26).
[Please download the PDF to view the formula] (24)
[Please download the PDF to view the formula] (25)
[Please download the PDF to view the formula]. (26)
The temporal representation [H.sup.(L).sub.t] after L layers encapsulate both long-range dependencies and deep spatial-temporal correlations. The attention mechanism computes a context vector as in (27).
[Please download the PDF to view the formula] (27)
where, [W.sub.h], v and [b.sub.h] are learnable parameters, [H.sub.att] captures the weighted spatio-temporal representation, emphasizing the most discriminative features.
Finally, the attention-refined feature vector [H.sub.att] is fed to a fully connected layer followed by SoftMax activation for activity classification expressed as in (28).
[Please download the PDF to view the formula] (28)
where, [W.sub.c] and [b.sub.c] are the classifier weights and biases, [^.y] is the predicted probability distribution over activity classes.
The classification objective is to minimize the cross-entropy loss illustrated in (29).
[Please download the PDF to view the formula] (29)
where, N is the number of training samples, [y.sub.i,c] is the ground-truth label for the sample i and class c, [theta] are the trainable parameters.
Hence, the framework uses residual connections to mitigate vanishing gradients, Bi-LSTM for bi-directional temporal modelling, and attention for adaptive feature weighting to develop a high discriminative representation capable of reliable human activity recognition.
2.6 Improved Gooseneck Barnacle Optimization Algorithm
The IGBOA [25] algorithm captures more accurate static and dynamic behavior, inspired by natural movements. It significantly reduces computation time by using smaller mathematical models. The fitness function for hyperparameter optimization, including learning rate, iterations, and batch size, is represented in (30).
Fitness function = Tuning (learning rate, iterations, batch size), (30)
Compared to classic optimization algorithms like Particle Swarm Optimization (PSO) and Whale Optimization Algorithm (WOA), which struggle to adjust hyperparameters effectively for complex models, IGBOA provides more precise solutions and faster convergence. Update the position of the best solution using (31).
[Please download the PDF to view the formula] (31)
The task of HAR is achieved using the DRCBi-LSTM approach, a precise system to identify human actions from frames of a video. The accuracy of the system's performance is further enhanced with the help of IGBOA hyperparameter optimization.
2.7 Interpretability Analysis using Grad-CAM
Integrating interpretability techniques in the DRCBi-LSTM framework enhances decision-making transparency. Grad-CAM visualizes discriminative spatial areas, while the attention mechanism highlights temporally significant frame sequences [26], as defined in (32).
[Please download the PDF to view the formula] (32)
where, [S.sub.c] represents the class score for C, [A.sub.k] denotes the kth activation map, [a.sup.c.sub.k] is the spatial importance coefficient computed through global average pooling of gradients.
For a sequence of T frames, the attention-based importance at time t is expressed as in (33).
[Please download the PDF to view the formula] (33)
where, [a.sub.t] denotes the learned temporal attention weight at frame t.
The attention scores reveal key temporal dependencies influencing classification, ensuring high accuracy and explainability. This integration enhances the recognition system's performance, making it suitable for real-world applications like surveillance and behavioral analysis.
2.8 Hardware-Accelerated Interpretable Human Action Recognition
The suggested architecture of the HAR system relies on the hybrid design of the FPGA-GPU/TPU-based computing system, which makes it capable of performing real-time computations while maintaining high levels of interpretability. The task of segmentation and feature extraction is performed by an FPGA at the edge level, providing low latency in the process. The complex tasks, including temporal feature extraction, DRCBi-LSTM inference, and Grad-CAM-based interpretability, are implemented using an NVIDIA RTX 4090 GPU.
2.8.1 Edge Layer Processing
The framework processes at the edge using an NVIDIA Jetson Xavier NX and an Xilinx Kintex-7 FPGA. The FPGA is specifically tasked with handling parallel pixel-level operations, such as real-time video frame segmentation and feature extraction, which are essential for high-throughput processing. This allows for rapid, low-latency processing at the edge. The FPGA handles parallel pixel-level and feature-level operations for high-throughput segmentation, while the Jetson NX performs lightweight convolutional preprocessing, enabling multi-stream processing with low latency and energy efficiency. The Jetson Xavier NX also assists with initial data normalization and preprocessing tasks, ensuring that the frames are ready for higher-level inference tasks.
2.8.2 Real-Time and Interpretable Human Activity Recognition with Hardware-Accelerated Computation
The preprocessed frames are fed into the GPU/TPU platform for spatial feature extraction with residual convolutional networks and temporal feature handling using Bi-LSTM layers. Deep learning processes, including temporal feature extraction, DRCBi-LSTM prediction, and Grad-CAM analysis, occur at the GPU/TPU platform, which ensures interpretability with Grad-CAM visualization of discriminative segments within each frame. Attention mechanisms help to make decisions with respect to temporal features, and real-time saliency detection is enabled using messaging technology with reduced latencies. The FPGA platform helps to perform segmentation along with low-level feature extraction.
2.8.3 System Efficiency and Performance
System efficiency is quantified as in (34).
[Please download the PDF to view the formula] (34)
where, [R.sub.p] is the frame processing rate, [L.sub.c] is average inference latency, and [E.sub.f] is the normalized energy consumption. Maximizing [eta] guarantees high-throughput, low-latency, and energy-efficient execution. Experiments demonstrate that inference latency is less than 60 ms per frame and the average power consumption is 35-40 W for six streams, making it suitable for smart surveillance, behavioral analysis, and safety-critical applications.
3. Outcomes and analysis
In the experimental section, the data, parameters, and results of the DRCBi-LSTM model for HAR are presented, as demonstrated in Table 4. The experiments are carried out in Python version 3.11 on a system with an Intel Core i7-6700 CPU and 16 GB RAM. In the comparative analysis, the performance of the proposed method, IGBOA, outperforms PSO and WOA, and the traditional Adam method.
3.1 Qualitative examination of the proposed technique across the utilized datasets
The experimental findings for HAR using four datasets (KTH, UT-Kinect, UCF-11, and HMDB-51) are presented. Table 5 shows the HAR system, with Grad-CAM highlighting key spatial regions for action identification.
Human action recognition outcomes of the UT-Kinect dataset
Table 6 shows an example of the HAR process using the DRCBi-LSTM model for the UT-Kinect dataset. In the sequence, the Grad-CAM image is included for interpretation purposes by adding a heatmap to the region in the frame that contributed to the classification of the action.
Outcomes of the UCF 11 dataset for recognizing human action
Table 7 illustrates the HAR process, showing a Grad-CAM image with an overlaid heatmap highlighting critical regions the model focused on for accurate action classification.
Results of the HMDB51 dataset for recognizing human action
Table 8 shows the model's output, including the original frame, segmented image, tracking image, recognized label, and Grad-CAM highlights.
3.2 Performance analysis
An inclusive assessment of the proposed model's efficacy, shown in Fig. 2, compares IGBOA with ACOA [27], PSOA [28], and WOA [29], demonstrating superior convergence and stability. IGBOA achieves 98.7%, 99.6%, and 99.8% accuracy, with DRCBi-LSTM surpassing all at 99.9%. Table 9 illustrates the comparison of recognition time.
Figs. 3(a)-(d), compare the proposed DRCBi-LSTM with existing methods across four datasets. On the UCF 11 dataset, it outperforms PCANet, DoG, CGA, and STDAN with an accuracy of 99.85%. For HMDB51, it surpasses DCNN, CGA, TA3DNet, and ConvST-LSTM.
3.3 Ablation study
An ablation study table for the DSORT-SQA algorithm, showing the impact of each key component, Deep Feature Embedding (DFE), SQA, and Motion Update (MU) on tracking performance. Metrics used include Multiple Object Tracking Accuracy (MOTA), Identification [F.sub.1]-score (ID[F.sub.1]), Mostly Tracked (MT%), and Mostly Lost (ML%).
The effect of different architectures on the tracking efficiency has been illustrated in Table 10. The Baseline (SORT) scores 71.4% MOTA and 68.7% IDF1. With the addition of DFE, there is some improvement, while the addition of SQA leads to a MOTA score of 83.2% and an IDF1 score of 79.5%. The contribution of MPE in consistency is noted from the improved MOTA and IDF1 of 81.9% and 77.8%, respectively. The most efficient model is the Full Model with scores of 89.1% MOTA and 85.6% IDF1.
The results depict maximum MT (75.2%), minimum ML (7.4%), optimum performance during occlusions (83.8%), and minimal ID switching (12.9%). This proves that there is a complementary effect obtained through the use of all three modules suggested. Statistical results are found in Table 11, while Fig. 4 indicates correct classification and performance of the model on four different datasets.
Table 12 shows the top-3 action pair confusions for all datasets, which provides a quantitative understanding of the problems of action confusions seen in Fig. 4. It is evident that similar visual and temporal actions, such as "walking" versus "running" and "sitting" versus "standing", cause major problems. In order to resolve such challenges, the following mitigation techniques are adopted: temporal attention improvement, multimodal feature combination, and consistency of motion analysis.
3.4 Validation on Uncontrolled Real-Time Surveillance Data
To assess the DRCBi-LSTM framework's generalization, experiments were conducted on realtime video streams from university corridors, offices, and labs. The framework, trained on benchmark data, was directly applied for performance evaluation.
Table 13 highlights a quantitative comparison of the proposed model, DRCBi-LSTM, to some of the existing action recognition techniques in terms of their performance on uncontrolled real-time surveillance videos. In all these metrics, noted that the proposed model performs better than all of its competitors.
3.5 Frame-Wise Attention Analysis for Motion Dynamics
Grad-CAM highlights discriminative regions, while Bi-LSTM with attention mechanisms explicitly models temporality, assigning frame-specific weights to differentiate actions with similar appearances but distinct movement patterns shown in Fig. 5(a).
Sinusoidal distributions are seen in cyclical activities (e.g., walking, running), point distributions are seen in action-oriented activities (e.g., waving), whereas plateau distributions are seen in interaction activities (e.g., boxing). This clearly illustrates that the proposed model captures different types of temporal distributions effectively. As shown in Fig. 5(b), IGBOA is compared with the following optimization algorithms: PSO, WOA, Adam, AdamW, Bayesian, and CMA-ES. The proposed approach provides 99.6% accuracy and reduces computation time (~70 seconds).
Table 14 visualizes multi-person activity outcomes using the HMDB-51 dataset. The first column shows input images with multiple individuals. The second column displays segmented images, isolating subjects. The third highlights movement with tracking boxes. The fourth shows Grad-CAM images, illustrating the model's focus on key activation areas for activity recognition.
The DSORT-SQA approach demonstrates significant enhancement compared to the DeepSORT approach, as evidenced by the results shown in Table 15. The gains recorded were a 12.3% improvement in MOTA, a 11.7% increase in IDF1, and 10.9% in MT, but an 5.4% reduction in ML. Furthermore, there was a drop of 7.9 in ID switches.
Table 16 compares the convergence time and number of iterations of the IGBOA algorithm with PSO and Adam on HMDB51 and UCF11 datasets. From the table, it is evident that IGBOA converges faster than PSO and Adam using fewer iterations. Nevertheless, there is an exception where the IGBOA algorithm takes more iterations for some action classes, such as walking and running, in which PSO performs better than IGBOA. Adam gives high-accuracy results for jog actions.
4. Conclusion and future scope
In this research, a DRCBi-LSTM model has been proposed for precise HAR within the frames of videos. Pre-processing techniques are used for frame extraction, resizing, and data normalization. Individual tracking and segmentation are performed using DSORT-SQA for precise segmentation. Real-time acquisition techniques are also included for improving video action surveillance. This framework has been extended for hardware implementation to optimize video processing in real-time. In the future, this framework will be extended for multiple persons to perform real-time action surveillance in various environments.
Acknowledgement
None
Data Availability Statements
The data supporting the findings of this study are cited, and the link is listed in the references.
Funding
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
References
[1] N. Jaouedi, N. Boujnah, and M. S. Bouhlel, "A new hybrid deep learning model for human action recognition," Journal of King Saud University - Computer and Information Sciences, vol.32, no.4, pp.447-453, May 2020. Article(CrossRefLink)
[2] C. Dai, X. Liu, and J. Lai, "Human action recognition using two-stream attention based LSTM networks," Applied Soft Computing, vol.86, Jan. 2020. Article(CrossRefLink)
[3] Z. Xiao, X. Xu, H. Xing, F. Song, X. Wang, and B. Zhao, "A federated learning system with enhanced feature extraction for human activity recognition," Knowledge-Based Systems, vol.229, Oct. 2021. Article(CrossRefLink)
[4] V. Mazzia, S. Angarano, F. Salvetti, F. Angelini, and M. Chiaberge, "Action Transformer: A self-attention model for short-time pose-based human action recognition," Pattern Recognition, vol.124, Apr. 2022. Article(CrossRefLink)
[5] K. Muhammad, Mustaqeem, A. Ullah, A. S. Imran, M. Sajjad, M. S. Kiran, G. Sannino, and V. H. C. de Albuquerque, "Human action recognition using attention based LSTM network with dilated CNN features," Future Generation Computer Systems, vol.125, pp.820-830, Dec. 2021. Article(CrossRefLink)
[6] J. Manoharan and Y. Sivagnanam, "A Novel Human Action Recognition Model by Grad-CAM Visualization with Multi-level Feature Extraction Using Global Average Pooling with Sequence Modeling by Bidirectional Gated Recurrent Units," International Journal of Computational Intelligence Systems, vol.18, 2025. Article(CrossRefLink)
[7] N. Dua, S. N. Singh, and V. B. Semwal, "Multi-input CNN-GRU based human activity recognition using wearable sensors," Computing, vol.103, pp.1461-1478, Jul. 2021. Article(CrossRefLink)
[8] C. Liu, J. Ying, H. Yang, X. Hu, and J. Liu, "Improved human action recognition approach based on two-stream convolutional neural network model," The Visual Computer, vol.37, pp.1327-1341, Jun. 2021. Article(CrossRefLink)
[9] A. Abdelbaky and S. Aly, "Human action recognition using three orthogonal planes with unsupervised deep convolutional neural network," Multimedia Tools and Applications, vol.80, pp.20019-20043, May 2021. Article(CrossRefLink)
[10] S. Sowmyayani and P. A. J. Rani, "STHARNet: spatio-temporal human action recognition network in content based video retrieval," Multimedia Tools and Applications, vol.82, pp.38051-38066, Oct. 2023. Article(CrossRefLink)
[11] S. J. Berlin and M. John, "Particle swarm optimization with deep learning for human action recognition," Multimedia Tools and Applications, vol.79, pp.17349-17371, Jul. 2020. Article(CrossRefLink)
[12] O. Elharrouss, N. Almaadeed, S. Al-Maadeed, A. Bouridane, and A. Beghdadi, "A combined multiple action recognition and summarization for surveillance video sequences," Applied Intelligence, vol.51, pp.690-712, Feb. 2021. Article(CrossRefLink)
[13] M. M. Zaidi, G. A. Sampedro, A. Almadhor, S. Alsubai, A. Al Hejaili, M. Gregus, and S. Abbas, "Suspicious Human Activity Recognition From Surveillance Videos Using Deep Learning," IEEE Access, vol.12, pp.105497-105510, 2024. Article(CrossRefLink)
[14] I. M. Nasir, M. Raza, S. M. Ulyah, J. H. Shah, N. L. Fitriyani, and M. Syafrudin, "ENGA: Elastic Net-Based Genetic Algorithm for human action recognition," Expert Systems with Applications, vol.227, Oct. 2023. Article(CrossRefLink)
[15] Y. Wang, X. J. Shen, H. P. Chen, and J. X. Sun, "Action Recognition in Videos with Spatio-Temporal Fusion 3D Convolutional Neural Networks," Pattern Recognition and Image Analysis, vol.31, pp.580-587, Jul. 2021. Article(CrossRefLink)
[16] T. Shanableh, "ViCo-MoCo-DL: Video Coding and Motion Compensation Solutions for Human Activity Recognition Using Deep Learning," IEEE Access, vol.11, pp.73971-73981, 2023. Article(CrossRefLink)
[17] H. Ullah and A. Munir, "Human Action Representation Learning Using an Attention-Driven Residual 3DCNN Network," Algorithms, vol.16, no.8, Aug. 2023. Article(CrossRefLink)
[18] M. Karim, S. Khalid, A. Aleryani, N. Tairan, Z. Ali, and F. Ali, "HADE: Exploiting Human Action Recognition Through Fine-Tuned Deep Learning Methods," IEEE Access, vol.12, pp.42769-42790, 2024. Article(CrossRefLink)
[19] P. Wang, Y. Xiang, D. Zhong, T. Li, Y.-R. Zhang, and M. Liu, "A Spatiotemporal Dynamic Weighting Method Based on Hyperspherical Tensor Mapping for Action Recognition," SSRN preprint, 2025. Article(CrossRefLink)
[20] B. Sun, W. Zhang, C. Xing, and Y. Li, "Underwater moving target detection and tracking based on enhanced you only look once and deep simple online and realtime tracking strategy," Engineering Applications of Artificial Intelligence, vol.143, Mar. 2025. Article(CrossRefLink)
[21] Y. Jin, F. Gui, M. Chen, X. Chen, H. Li, and J. Zhang, "Deep learning-driven automated quality assessment of ultra-widefield optical coherence tomography angiography images for diabetic retinopathy," The Visual Computer, vol.41, pp.1049-1059, Jan. 2025. Article(CrossRefLink)
[22] Q. Jiang, Q. Wang, L. Chi, and J. Liu, "Cross-scale hierarchical spatio-temporal transformer for video enhancement," Knowledge-Based Systems, vol.309, Jan. 2025. Article(CrossRefLink)
[23] Y. K. Saheed, T. T. Salau-Ibrahim, M. Abdulsalam, I. A. Adeniji, and B. F. Balogun, "Modified bi-directional long short-term memory and hyperparameter tuning of supervised machine learning models for cardiovascular heart disease prediction in mobile cloud environment," Biomedical Signal Processing and Control, vol.94, Aug. 2024. Article(CrossRefLink)
[24] D. Muthusamy and P. Palani, "Deep neural network model for diagnosing diabetic retinopathy detection: An efficient mechanism for diabetic management," Biomedical Signal Processing and Control, vol.100, part.A, Feb. 2025. Article(CrossRefLink)
[25] M. Ahmed, M. H. Sulaiman, A. J. Mohamad, and M. Rahman, "Gooseneck barnacle optimization algorithm: A novel nature inspired optimization theory and application," Mathematics and Computers in Simulation, vol.218, pp.248-265, Apr. 2024. Article(CrossRefLink)
[26] M. Jayamohan and S. Yuvaraj, "A novel human action recognition using Grad-CAM visualization with gated recurrent units," Neural Computing and Applications, vol.37, pp. 10835-10850, Jun. 2025. Article(CrossRefLink)
[27] A. A. Alqarni, "Toward support-vector machine-based ant colony optimization algorithms for intrusion detection," Soft Computing, vol.27, pp.6297-6305, May 2023. Article(CrossRefLink)
[28] M. Zhao, H. Zhao, and M. Zhao, "Particle Swarm Optimization Algorithm With Adaptive Two-Population Strategy," IEEE Access, vol.11, pp.62242-62260, 2023. Article(CrossRefLink)
[29] Y. Dai, J. Yu, C. Zhang, B. Zhan, and X. Zheng, "A novel whale optimization algorithm of path planning strategy for mobile robots," Applied Intelligence, vol.53, pp.10843-10857, May 2023. Article(CrossRefLink)
A. Senthil Murugan is an Assistant Professor in the Department of Computer Science and Engineering at St. Peter's Engineering College, Hyderabad. He completed his M.E. in Computer Science and Engineering at Annamalai University, Chidambaram, in 2014.
S. Manoj Kumar holds a Doctorate in Computer Science and Engineering. He has been working as a faculty member in Computer Science and Engineering. He is specialized in the area of Wireless Sensor Networks, IoT and Networking. He has 20 years of teaching experience to his credit and has published 49 research papers in reputed journals including SCI, Scopus indexed journals, Web of Science and presented papers in a number of international conferences.
A. Sivaranjani holds a Doctorate in Computer Science and Engineering. She is currently working as an Assistant Professor in the Department of Analytics in School of Computer Science and Engineering at VIT-Vellore. With 12 years of teaching experience, she has published 18 research papers in reputed journals and International conferences, including those indexed by SCI, Scopus, and Web of Science.
A. Senthil Murugan (1*), S. Manoj Kumar (2), and A. Sivaranjani (3)
(1) Department of Computer Science and Engineering, St. Peter's Engineering College, Hyderabad, Telangana-500043, India
(2) Department of Computer Science and Engineering, KPR Institute of Engineering and Technology, Coimbatore, Tamil Nadu-641407, India
(3) Department of Analytics, School of Computer Science and Engineering, Vellore Institute of Technology Vellore, Tamil Nadu-632014, India
[e-mail: senthilmuruganassproff@gmail. com, manojkumarsces@outlook. com, sivaranj aniassproff@outlook. com]
(*) Corresponding author: A. Senthil Murugan
Received October 31, 2025; revised January 30, 2026; revised February 25, 2026; revised April 10, 2026; accepted April 30, 2026; published July 31, 2026
http://doi.org/10.3837/tiis.2026.07.009
Table 1. Comparison of Related Studies on HAR
Refs Approaches Relevance
to Current
Work
[8] CNN Uses a two-stream CNN for action recognition.
[9] PCANet Related to feature extraction.
[10] STHARNet Uses spatio-temporal analysis for action
recognition.
[11] DLNN Combines deep learning with optimization.
[12] TDMap Focuses on surveillance video summarization.
[13] Suspicious Similar to action recognition in surveillance.
Activity
Recognition
[14] ENGA Feature selection for action recognition.
[15] 3D CNNs Uses 3D CNNs for temporal analysis.
[16] ViCo-MoCo-DL Related to motion dynamics in action recognition.
[17] AR3DCNN Uses attention-driven 3D CNN for recognition.
[18] HADE Uses DL techniques for recognition.
[19] SDWM-HTM Focuses on spatio-temporal feature weighting.
Refs Novelty of
Current Work
[8] Focuses on dynamic fusion and temporal attention, capturing
motion dynamics and movement order.
[9] Implements attention-based dynamic fusion, improving
depth-dependent action recognition.
[10] Proposes task-context-based dynamic fusion, improving
depth-sensitive action recognition.
[11] Uses IGBOA for optimization, enhancing recognition accuracy and
stability.
[12] Concentrates on action recognition via DSORT-SQA, not
summarization.
[13] Uses temporal attention mechanisms to adjust importance
dynamically.
[14] Integrates temporal attention and dynamic fusion for improved
recognition.
[15] Dynamic fusion adapts to depth-specific contexts, improving
recognition in tasks like sit-to-stand.
[16] Integrates motion update for real-time action recognition with
DSORT-SQA.
[17] Introduces dynamic fusion with temporal attention for improved
tracking of depth-sensitive actions.
[18] Focuses on dynamic fusion and temporal attention, enhancing
recognition in complex environments.
[19] Uses temporal attention mechanisms for dynamic feature
importance.
Table 2. Summary of Offline HAR Datasets
Datasets Number Number Environments
of Videos of Classes
KTH [16] 2,391 6 (walking, jogging, running, Indoor &
boxing, hand waving, hand outdoor,
clapping) controlled
background
UT-Kinect ~200 10 (waving, clapping, sitting, Indoor,
[17] standing, walking, etc.) controlled
UCF-11 ~1,600 11 (basketball shooting, golf Unconstrained,
[18] swing, biking, diving, etc.) online videos
HMDB51 6,849 51 (diverse actions from Unconstrained,
[19] movies and web videos) varied scenes
Datasets Resolution Data
Splitting
Ratio
KTH [16] 160 * 120 70%: 30%
UT-Kinect 640 * 480 70%: 30%
[17]
UCF-11 320 * 240 70%: 30%
[18]
HMDB51 240 * 320 70%: 30%
[19]
Table 3. Summary of Live Data Characteristics
Attributes Values
Resolution 1920 * 1080
Action Classes 12 (e.g., walking, running, sitting, standing,
waving)
Environments Indoor (offices, labs), outdoor (university
corridors, public spaces)
Sample Size ~500 video samples
Class Distribution Balanced (approximately equal representation across
action classes)
Environmental Variable lighting conditions, dynamic backgrounds,
Conditions occlusions (e.g., pedestrians)
Table 4. Setup of Model Parameters
Parameters Description
Learning rate 0.01
Dropout 0.02
Number of frames in 50-250
each input video
Batch size 64
Optimizer IGBOA
Resize of input frames 160 * 120
Epochs 100
Iterations 100
Algorithm Improved Gooseneck Barnacle
Optimization
Table 9. Recognition rate comparison with traditional methods Methods Recognition rate (%) GRNN [12] 96.3 GWR [13] 98.7 CNN [14] 99.6 M-SVM [15] 99.8 DRCBi-LSTM (proposed) 99.88
Table 10. Ablation Study of the DSORT-SQA Algorithm
Configurations MOTA ID F1 MT ML MOTA
(%) (%) (%) (%) (Occlusion-Heavy)
(%)
Baseline (SORT) 71.4 68.7 56.3 15.2 65.4
+ DFE 78.6 74.9 63.1 12.8 72.8
+ DFE + SQA 83.2 79.5 68.7 10.5 77.5
+ DFE + MPE 81.9 77.8 66.2 11.3 74.9
+ DFE + SQA + MPE 87.6 84.2 73.4 8.1 82.1
+ DFE + MU 81.9 77.8 66.2 11.3 74.9
+ Static Fusion 78.6 74.9 63.1 12.8 71.3
+ Dynamic Fusion 83.2 79.5 68.7 10.5 77.0
Full model 89.1 85.6 75.2 7.4 83.8
Configurations ID-Switches
Baseline (SORT) 25.2
+ DFE 22.3
+ DFE + SQA 18.5
+ DFE + MPE 20.1
+ DFE + SQA + MPE 14.2
+ DFE + MU 20.1
+ Static Fusion 23.4
+ Dynamic Fusion 18.0
Full model 12.9
Table 11. Statistical Analysis of DRCBi-LSTM Performance Across
Benchmark Datasets
Dataset Accuracy (%) Precision (%)
Mean [+ or -] SD Mean [+ or -] SD
KTH 99.80 [+ or -] 0.05 99.84 [+ or -] 0.04
UT-Kinect 99.88 [+ or -] 0.06 99.94 [+ or -] 0.05
UCF 11 99.85 [+ or -] 0.07 97.69 [+ or -] 0.08
HMDB51 99.88 [+ or -] 0.06 98.73 [+ or -] 0.05
Overall (5-Fold) 98.26 [+ or -] 0.34 98.06 [+ or -] 0.37
Dataset Recall (%) F1-Score (%)
Mean [+ or -] SD Mean [+ or -] SD
KTH 99.90 [+ or -] 0.03 99.92 [+ or -] 0.03
UT-Kinect 99.70 [+ or -] 0.07 99.87 [+ or -] 0.05
UCF 11 97.69 [+ or -] 0.08 98.88 [+ or -] 0.06
HMDB51 98.90 [+ or -] 0.05 99.95 [+ or -] 0.04
Overall (5-Fold) 97.86 [+ or -] 0.39 97.96 [+ or -] 0.37
Table 12. Top-3 Confused Action Pairs and Proposed Mitigations
Datasets Top-3 Confused
Action Pairs
KTH Walking vs. Running (20%)
Boxing vs. Hand Waving (15%)
Running vs. Jogging (18%)
UT-Kinect Waving vs. Clapping (9.9%)
Running vs. Walking (7.5%)
Sitting vs. Standing (6.8%)
HMDB51 Sit vs. Stand (20%)
Curl vs. Squat (18%)
Jump vs. Fall (15%)
UCF11 Walking vs. Running (12%)
Clapping vs. Waving (9%)
Biking vs. Diving (10%)
Datasets Proposed
Mitigations
KTH Temporal attention to prioritize motion dynamics.
Use of spatial-temporal fusion to improve context
distinction.
Implement temporal alignment of frames to focus on specific
phases of motion.
UT-Kinect Enhance appearance descriptors and contextual temporal
analysis.
Increase feature representation from multimodal data.
Use spatio-temporal normalization to reduce confounding
background interference.
HMDB51 Integrate temporal attention mechanisms to focus on the
start and end phases of actions.
Use dynamic fusion strategies to improve feature
discrimination.
Improve motion prediction and occlusion handling with MPE.
UCF11 Use temporal alignment with attention weights to refine
walking vs. running phases.
Introduce pose estimation refinement to distinguish subtle
differences in hand actions.
Apply multimodal integration to better capture action
contexts.
Table 13. Comparative performance on uncontrolled real-time
surveillance data
Methods Accuracy Precision Recall F1-score
(%) (%) (%) (%)
PCANet [9] 91.4 90.8 89.6 90.2
DLNN [11] 92.6 92.1 91.3 91.7
CNN [8] 93.8 93.2 92.4 92.8
ViCo-MoCo-DL [16] 94.9 94.3 93.7 94.0
Proposed DRCBi-LSTM 96.8 96.2 95.9 96.0
Table 15. Comparison of DSORT-SQA (Proposed) vs. DeepSORT
Metrics DeepSORT DSORT-SQA Improvement
(%) (%)
MOTA 75.3 87.6 +12.3
IDF1 72.5 84.2 +11.7
MT 62.5 73.4 +10.9
ML 13.5 8.1 -5.4
ID-Switches 22.1 14.2 -7.9
Table 16. Convergence Time and Iterations Comparison
Optimizer Convergence Iterations Actions Where
Time to Target Other Optimizers
Performance Outperform IGBOA
IGBOA ~70 seconds ~50 iterations N/A
PSO ~85 seconds ~75 iterations Walking, Running (PSO
converges faster)
AdamW ~90 seconds ~80 iterations Jogging (AdamW performed
better in accuracy)
BO ~110 seconds ~100 iterations None (slower convergence,
lower fitness)
ACOA ~95 seconds ~90 iterations None (slower convergence,
lower fitness)
 Printer friendly
 Cite/link
 Email
 Feedback
| |
| Author: | Murugan, A. Senthil; Kumar, S. Manoj; Sivaranjani, A. |
|---|---|
| Publication: | KSII Transactions on Internet and Information Systems |
| Article Type: | Report |
| Geographic Code: | 9INDI |
| Date: | Jul 1, 2026 |
| Words: | 6579 |
| Previous Article: | Intrusion Detection System for SDN-based VANETs Using A Deep Belief Network, Decision Tree, and ToN-IoT Dataset. |
| Next Article: | A Novel Dynamic Access Authentication Technology for Power Communication Networks Based on Meta-Learning and Multimodal Feature Fusion. |
| Topics: | |