Safety monitoring system of personal mobility driving using deep learning

Abstract

Abstract Although the e-scooter sharing service market is growing as a representative last-mile mobility, the accident rate is increasing proportionally as the number of users increases. This study proposes a deep learning-based personal mobility driver monitoring system that detects inattentive driving by classifying vibration data transmitted to the e-scooter when the driver fails to concentrate on driving. First, the N-back task technique is used. The driver was stimulated by external visual and auditory factors to generate a cognitive load, and vibration data were collected through a six-axis sensor. Second, the generated vibration data were pre-processed using short-time Fourier transform and wavelet transform (WT) and then converted into an image (spectrogram). Third, four multimodal convolutional neural networks such as LeNet-5, VGG16, ResNet50, and DenseNet121 were constructed and their performance was compared to find the best architecture. Experimental results show that multimodal DenseNet121 with WT can accurately classify safe, slightly anxious, and very anxious driving conditions. The proposed model can be applied to real-time monitoring and warning systems for sharing service providers and used as a basis for insurance and legal action in the case of accidents.