EfficientNetV2-based dynamic gesture recognition using transformed scalogram from triaxial acceleration signal

Abstract

Abstract In this paper, a dynamic gesture recognition system is proposed using triaxial acceleration signal and image-based deep neural network. With our dexterous glove device, 1D acceleration signal can be measured from each finger and decomposed to time-divided frequency components via wavelet transformation, which is known as scalogram as image-like format. To feed-forward the scalogram with single 2D, convolutional neural networks allows the gesture having temporality to be easily recognized without any complex system such as RNN, LSTM, or spatio-temporal feature as 3D CNN, etc. To classify the image with general input dimension of image RGB channels, we numerically reconstruct fifteen scalograms into one RGB image with various representation methods. In experiments, we employ the off-the-shelf model, EfficientNetV2 small-to-large model as an image classification model with fine-tuning. To evaluate our system, we bulid our custom bicycle hand signals as dynamic gesture dataset under our transformation system, and then qualitatively compare the reconstruction method with matrix representation methods. In addition, we use other signal transformation tools such as the fast Fourier transform and short-time Fourier transform and then explain the advantages of scalogram classification in the terms of time-frequency resolution trade-off issue.