Multimodal Driver Drowsiness Detection Using Visual and EEG Data with CNN-LSTM and Attention-Based Fusion

Abstract

Driver’s drowsiness poses a significant yet often unnoticed risk to road safety, as even brief lapses in alertness can lead to accidents. Many existing detection systems struggle with real-time accuracy and effective integration of multiple data sources. To address these limitations, we propose a multimodal driver drowsiness detection system that combines visual and physiological (EEG) data to enhance accuracy and real-time responsiveness. The visual component leverages a Convolutional Neural Network (CNN) for spatial feature extraction, followed by a Long Short-Term Memory (LSTM) architecture to an- alyze temporal patterns. Meanwhile, the physiological model applies CNN to extract spectral features from EEG signals, with LSTM capturing temporal dependencies. For efficient real-time processing, the system is deployed on edge devices, optimized using TensorFlow Lite to enable smooth on-chip execution. The system generates a drowsiness score ranging from 0 to 100, triggering notifications when predefined thresholds are surpassed. By incorporating advanced deep learning techniques such as attention mechanisms, multimodal data fusion, and temporal modeling, the proposed approach ensures high accuracy and responsiveness. This makes it a practical solution for real-world applications in driver safety and monitoring.