Affiliation:
1. Pennsylvania State University, State College, PA, USA
2. University of South Carolina, Columbia, SC, USA
Abstract
mmWave signals form a critical component of 5G and next-generation wireless networks, which are also being increasingly considered for sensing the environment around us to enable ubiquitous IoT applications. In this context, this paper leverages the properties of mmWave signals for tracking 3D finger motion for interactive IoT applications. While conventional vision-based solutions break down under poor lighting, occlusions, and also suffer from privacy concerns, mmWave signals work under typical occlusions and non-line-of-sight conditions, while being privacy-preserving. In contrast to prior works on mmWave sensing that focus on predefined gesture classification, this work performs continuous 3D finger motion tracking. Towards this end, we first observe via simulations and experiments that the small size of fingers coupled with specular reflections do not yield stable mmWave reflections. However, we make an interesting observation that focusing on the forearm instead of the fingers can provide stable reflections for 3D finger motion tracking. Muscles that activate the fingers extend through the forearm, whose motion manifests as vibrations on the forearm. By analyzing the variation in phases of reflected mmWave signals from the forearm, this paper designs
mm4Arm,
a system that tracks 3D finger motion. Nontrivial challenges arise due to the high dimensional search space, complex vibration patterns, diversity across users, hardware noise, etc. mm4Arm exploits anatomical constraints in finger motions and fuses them with machine learning architectures based on encoder-decoder and ResNets in enabling accurate tracking. A systematic performance evaluation with 10 users demonstrates a median error of 5.73° (location error of 4.07 mm) with robustness to multipath and natural variation in hand position/orientation. The accuracy is also consistent under non-line-of-sight conditions and clothing that might occlude the forearm.
mm4Arm
runs on smartphones with a latency of 19
ms
and low energy overhead.
Funder
National Science Foundation
Publisher
Association for Computing Machinery (ACM)
Subject
Computer Networks and Communications,Hardware and Architecture,Safety, Risk, Reliability and Quality,Computer Science (miscellaneous)
Reference133 articles.
1. 5dt data glove ultra - 5dt. https://5dt.com/5dt-data-glove-ultra/. 5dt data glove ultra - 5dt. https://5dt.com/5dt-data-glove-ultra/.
2. American manual alphabet. https://en.wikipedia.org/wiki/American_manual_alphabet. American manual alphabet. https://en.wikipedia.org/wiki/American_manual_alphabet.
3. Cyberglove systems llc. http://www.cyberglovesystems.com/. Cyberglove systems llc. http://www.cyberglovesystems.com/.
4. Forearm muscles : Attachment nerve supply action. https://anatomyinfo.com/forearm-muscles/. Forearm muscles : Attachment nerve supply action. https://anatomyinfo.com/forearm-muscles/.
5. Hanes men's full-zip eco-smart hoodie. https://www.amazon.com/Hanes-EcoSmart-Fleece-Hoodie-Black/dp/B00JUM4CT4/. Hanes men's full-zip eco-smart hoodie. https://www.amazon.com/Hanes-EcoSmart-Fleece-Hoodie-Black/dp/B00JUM4CT4/.
Cited by
6 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献
1. SignQuery: A Natural User Interface and Search Engine for Sign Languages with Wearable Sensors;Proceedings of the 29th Annual International Conference on Mobile Computing and Networking;2023-10-02
2. Leveraging the Properties of mmWave Signals for 3D Finger Motion Tracking for Interactive IoT Applications;ACM SIGMETRICS Performance Evaluation Review;2023-06-26
3. Leveraging the Properties of mmWave Signals for 3D Finger Motion Tracking for Interactive IoT Applications;Abstract Proceedings of the 2023 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems;2023-06-19
4. SoundSieve: Seconds-Long Audio Event Recognition on Intermittently-Powered Systems;Proceedings of the 21st Annual International Conference on Mobile Systems, Applications and Services;2023-06-18
5. Multi-Object Tracking with mmWave Radar: A Review;Electronics;2023-01-06