Vision-Based Finger Tapping Detection Without Fingertip Observation

Abstract

A machine learning approach is investigated in this study to detect a finger tapping on a handheld surface, where the movement of the surface is observed visually; however, the tapping finger is not directly visible. A feature vector extracted from consecutive frames captured by a high-speed camera that observes a surface patch is input to a convolutional neural network to provide a prediction label indicating whether the surface is tapped within the sequence of consecutive frames (“tap”), the surface is still (“still”), or the surface is moved by hand (“move”). Receiver operating characteristics analysis on a binary discrimination of “tap” from the other two labels shows that true positive rates exceeding 97% are achieved when the false positive rate is fixed at 3%, although the generalization performance against different tapped objects or different ways of tapping is not satisfactory. An informal test where a heuristic post-processing filter is introduced suggests that the use of temporal history information should be considered for further improvements.