Affiliation:
1. Department of Artificial Intelligence, Dongguk University, 30 Pildong-ro 1 Gil, Seoul 04620, Republic of Korea
Abstract
Because of societal changes, human activity recognition, part of home care systems, has become increasingly important. Camera-based recognition is mainstream but has privacy concerns and is less accurate under dim lighting. In contrast, radar sensors do not record sensitive information, avoid the invasion of privacy, and work in poor lighting. However, the collected data are often sparse. To address this issue, we propose a novel Multimodal Two-stream GNN Framework for Efficient Point Cloud and Skeleton Data Alignment (MTGEA), which improves recognition accuracy through accurate skeletal features from Kinect models. We first collected two datasets using the mmWave radar and Kinect v4 sensors. Then, we used zero-padding, Gaussian Noise (GN), and Agglomerative Hierarchical Clustering (AHC) to increase the number of collected point clouds to 25 per frame to match the skeleton data. Second, we used Spatial Temporal Graph Convolutional Network (ST-GCN) architecture to acquire multimodal representations in the spatio-temporal domain focusing on skeletal features. Finally, we implemented an attention mechanism aligning the two multimodal features to capture the correlation between point clouds and skeleton data. The resulting model was evaluated empirically on human activity data and shown to improve human activity recognition with radar data only. All datasets and codes are available in our GitHub.
Subject
Electrical and Electronic Engineering,Biochemistry,Instrumentation,Atomic and Molecular Physics, and Optics,Analytical Chemistry
Reference30 articles.
1. Internet of Things and Deep Learning Enabled Elderly Fall Detection Model for Smart Homecare;Vaiyapuri;IEEE Access,2021
2. Ma, W., Chen, J., Du, Q., and Jia, W. (2021, January 10–15). PointDrop: Improving object detection from sparse point clouds via adversarial data augmentation. Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR), Milan, Italy.
3. Classification of 3D Point Clouds by a New Augmentation Convolutional Neural Network;Xu;IEEE Geosci. Remote Sens. Lett.,2022
4. Deep learning-based dynamic object classification using LiDAR point cloud augmented by layer-based accumulation for intelligent vehicles;Kim;Expert Syst. Appl.,2021
5. Kulawiak, M. (2022). A Cost-Effective Method for Reconstructing City-Building 3D Models from Sparse Lidar Point Clouds. Remote Sens., 14.