Gesture Recognition Method Using Acoustic Sensing on Usual Garment

Abstract

In this study, we show a new gesture recognition method for clothing-based gesture input methods using active and passive acoustic sensing. Our system consists of a piezoelectric speaker and a microphone. The speaker transmits ultrasonic swept sine signals, and the microphone simultaneously records the ultrasonic signals that propagate through the garment and the rubbing sounds generated by the gestures on the garment. Our method recognizes a variety of gestures, such as pinch, twist, touch, and swipe, by incorporating active and passive acoustic sensing. An important feature of our method is that it does not require a dedicated garment or embroidery embedded since our system only requires a pair of piezoelectric elements to be attached to the usual garment with a magnet. We performed recognition experiments of 11 gestures on the forearm with four types of garments made from different materials and recognition experiments of five one-handed gestures on the button of a shirt and the pocket of pants. The results of a per-user classifier confirmed that the f-scores were 83.9% and 95.9% for 11 gestures with four different types of garments and 5 gestures that were selected assuming actual use, respectively. In addition, we confirmed that the system recognizes five gestures, which can be performed with one hand, with 89.2% and 92.6% accuracy in the button and pocket sites, respectively.