Affiliation:
1. Institute of Electronics, Faculty of Computer Science, Electronics and Telecommunication, AGH University of Krakow, Al. Mickiewicza 30, 30-059 Kraków, Poland
Abstract
Monitoring the psychophysical conditions of drivers is crucial for ensuring road safety. However, achieving real-time monitoring within a vehicle presents significant challenges due to factors such as varying lighting conditions, vehicle vibrations, limited computational resources, data privacy concerns, and the inherent variability in driver behavior. Analyzing driver states using visible spectrum imaging is particularly challenging under low-light conditions, such as at night. Additionally, relying on a single behavioral indicator often fails to provide a comprehensive assessment of the driver’s condition. To address these challenges, we propose a system that operates exclusively in the far-infrared spectrum, enabling the detection of critical features such as yawning, head drooping, and head pose estimation regardless of the lighting scenario. It integrates a channel fusion module to assess the driver’s state more accurately and is underpinned by our custom-developed and annotated datasets, along with a modified deep neural network designed for facial feature detection in the thermal spectrum. Furthermore, we introduce two fusion modules for synthesizing detection events into a coherent assessment of the driver’s state: one based on a simple state machine and another that combines a modality encoder with a large language model. This latter approach allows for the generation of responses to queries beyond the system’s explicit training. Experimental evaluations demonstrate the system’s high accuracy in detecting and responding to signs of driver fatigue and distraction.
Reference45 articles.
1. The Department of Transportation’s National Highway Traffic Safety Administration (NHTSA) (2024, August 29). Distracted Driving in 2022; NHTSA’s National Center for Statistics and Analysis: DOT HS 813 559, Available online: https://crashstats.nhtsa.dot.gov/Api/Public/ViewPublication/813559.
2. Detecting and recognizing driver distraction through various data modality using machine learning: A review, recent advances, simplified framework and open challenges (2014–2021);Koay;Eng. Appl. Artif. Intell.,2022
3. Driver’s facial expression recognition: A comprehensive survey;Saadi;Expert Syst. Appl.,2024
4. Machine learning assisted human fatigue detection, monitoring, and recovery: A Review;Lambay;Digit. Eng.,2024
5. Driver’s fatigue recognition based on yawn detection in thermal images;Knapik;Neurocomputing,2019