A visual-olfactory multisensory fusion spike neural network for early fire/smoke detection

Abstract

Abstract The integration and interaction of multisensory information has notably augmented the cognitive capacity of living organisms in their perception of the complex and varied world around them. In recent times, machine learning theory, particularly in image recognition, has progressed tremendously and has demonstrated immense potential in a multitude of practical scenes. Here, we report a biomimetic multisensory spike neural network (SNN) for early fire/smoke detection, which combines visual and olfactory perception for the first time. Inspired by the remarkable ability of insects to process information using their highly evolved vision and olfactory capabilities, we propose a network which extracts and utilizes both image and air features for early fire/smoke detection in complex scenes. We have built a comprehensive dataset sampled from multiple fire and smoke scenes, recording image and air data from several locations. The proposed multisensory SNN boasts a recognition accuracy of 95.21% for fire/smoke detection, while remaining highly hardware friendly and, enabling on-chip learning on hardware, and showing considerable potential in biological interpretability. The biomimetic multisensory algorithm provides a promising avenue for early fire/smoke detection, with important implications for enhancing safety and minimizing risk in a variety of complex scenes.