A Multi-Task Deep Learning Framework for Real-Time Intelligent Video Surveillance with Temporal Event Validation
2026-07-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors created a smart video surveillance system that can do many security tasks at once, like recognizing faces, reading license plates, detecting weapons, spotting fire and smoke, and understanding human actions. They built special models for weapon detection and recognizing vandalism actions, training them on new datasets they prepared. Their system checks multiple video frames to reduce mistakes, making detections more reliable. Tests show it works quickly and lowers false alarms while improving accuracy, all on regular computer hardware.
deep learningface recognitionlicense plate recognitionweapon detectionfire and smoke detectionhuman action recognitionmulti-task learningtemporal validationmean average precisionSlowFast model
Authors
Estera Dumitru, Stelian Spînu
Abstract
Modern video surveillance systems generate far more video streams than human operators can effectively monitor, making automated analysis essential for timely detection of security events. This paper presents a unified multi-task deep learning framework that simultaneously performs face recognition with zone-based authorization, automatic license plate recognition, weapon detection, fire and smoke detection, and human action recognition on a shared GPU platform. Among the integrated modules, two task-specific deep-learning models are proposed in this work to address scenarios that are insufficiently represented in publicly available datasets: a single-class weapon detector fine-tuned on a merged and relabeled dataset, achieving a mean average precision (mAP@0.5) of 0.947, and a SlowFast-R50 action recognition model trained on a purpose-built vandalism dataset comprising 614 video clips, achieving 94.33% classification accuracy. To improve robustness in continuous video, all detection modules are integrated into a temporal event-validation architecture based on multi-frame confirmation, confidence-weighted voting, and cascaded filtering, transforming frame-level predictions into reliable security events. Each module is evaluated independently on established public datasets (LFW, D-Fire, FIRESENSE, and UCF-Crime), followed by integrated end-to-end system evaluation. The proposed temporal validation strategy reduces the fire and smoke false-alarm rate from 52% to 4% and improves video license plate exact-match accuracy from 66.7% to 81.8%, while the complete framework maintains real-time operation with a per-frame latency below 100 ms on commodity hardware. These results demonstrate that combining specialized deep-learning models with temporal event validation provides an effective and practical solution for reliable real-time intelligent video surveillance.