In-Air Handwriting Recognition Using Acoustic Impulse Signals

Abstract

AbstractThis paper presents AcousticPAD, a contactless and robust handwriting recognition system that extends the input and interactions beyond the touchscreen using acoustic signals, thus very useful under the impact of the COVID-19 epidemic. To achieve this, we carefully exploit acoustic pulse signals with high accuracy of time of fight (ToF) measurements. Then we employ trilateration localization method to capture the trajectory of handwriting in air. After that, we incorporate a data augmentation module to enhance the handwriting recognition performance. Finally, we customize a back propagation neural network that leverages augmented image dataset to train a model and recognize the acoustic system generated handwriting characters. We implement AcousticPAD prototype using cheap commodity acoustic sensors, and conduct extensive real environment experiments to evaluate its performance. The results validate the robustness of AcousticPAD, and show that it supports 10 digits and 26 English letters recognition at high accuracies.