Optimizing human hand gestures for AI-systems

Abstract

Humans interact more and more with systems containing AI components. In this work, we focus on hand gestures such as handwriting and sketches serving as inputs to such systems. They are represented as a trajectory, i.e. sequence of points, that is altered to improve interaction with an AI model while keeping the model fixed. Optimized inputs are accompanied by instructions on how to create them. We aim to cut on effort for humans and recognition errors while limiting changes to original inputs. We derive multiple objectives and measures and propose continuous and discrete optimization methods embracing the AI model to improve samples in an iterative fashion by removing, shifting and reordering points of the gesture trajectory. Our quantitative and qualitative evaluation shows that mimicking generated proposals that differ only modestly from the original ones leads to lower error rates and requires less effort. Furthermore, our work can be easily adjusted for sketch abstraction improving on prior work.