AI‐Enabled Soft Sensing Array for Simultaneous Detection of Muscle Deformation and Mechanomyography for Metaverse Somatosensory Interaction

Abstract

AbstractMotion recognition (MR)‐based somatosensory interaction technology, which interprets user movements as input instructions, presents a natural approach for promoting human‐computer interaction, a critical element for advancing metaverse applications. Herein, this work introduces a non‐intrusive muscle‐sensing wearable device, that in conjunction with machine learning, enables motion‐control‐based somatosensory interaction with metaverse avatars. To facilitate MR, the proposed device simultaneously detects muscle mechanical activities, including dynamic muscle shape changes and vibrational mechanomyogram signals, utilizing a flexible 16‐channel pressure sensor array (weighing ≈0.38 g). Leveraging the rich information from multiple channels, a recognition accuracy of ≈96.06% is achieved by classifying ten lower‐limb motions executed by ten human subjects. In addition, this work demonstrates the practical application of muscle‐sensing‐based somatosensory interaction, using the proposed wearable device, for enabling the real‐time control of avatars in a virtual space. This study provides an alternative approach to traditional rigid inertial measurement units and electromyography‐based methods for achieving accurate human motion capture, which can further broaden the applications of motion‐interactive wearable devices for the coming metaverse age.