A CNN-based Human Activity Recognition System Combining a Laser Feedback Interferometry Eye Movement Sensor and an IMU for Context-aware Smart Glasses

Abstract

Smart glasses are considered the next breakthrough in wearables. As the successor of smart watches and smart ear wear, they promise to extend reality by immersive embedding of content in the user's field of view. While advancements in display technology seems to fulfill this promises, interaction concepts are derived from established wearable concepts like touch interaction or voice interaction, preventing full immersion as they require the user to frequently interact with the glasses. To minimize interactions, we propose to add context-awareness to smart glasses through human activity recognition (HAR) by combining head- and eye movement features to recognize a wide range of activities. To measure eye movements in unobtrusive way, we propose laser feedback interferometry (LFI) sensors. These tiny low power sensors are highly robust to ambient light. We combine LFI sensors and an IMU to collect eye and head movement features from 15 participants performing 7 cognitive and physical activities, leading to a unique data set. To recognize activities we propose a 1D-CNN model and apply transfer learning to personalize the classification, leading to an outstanding macro-F1 score of 88.15 % which outperforms state of the art methods. Finally, we discuss the applicability of the proposed system in a smart glasses setup.