G-VOILA: Gaze-Facilitated Information Querying in Daily Scenarios-Reference-Cited by-同舟云学术

G-VOILA: Gaze-Facilitated Information Querying in Daily Scenarios

Published:2024-05-13 Issue:2 Volume:8 Page:1-33
ISSN:2474-9567
Container-title:Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
language:en
Short-container-title:Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

Author:

Wang Zeyu¹^ORCID,Shi Yuanchun²^ORCID,Wang Yuntao¹^ORCID,Yao Yuchen³^ORCID,Yan Kun⁴^ORCID,Wang Yuhan⁵^ORCID,Ji Lei⁶^ORCID,Xu Xuhai⁷^ORCID,Yu Chun³^ORCID

Affiliation:

1. Key Laboratory of Pervasive Computing, Ministry of Education, Department of Computer Science and Technology, Tsinghua University, Haidian Qu, Beijing Shi, China

2. Key Laboratory of Pervasive Computing, Ministry of Education, Department of Computer Science and Technology, Tsinghua University, Haidian Qu, Beijing Shi, China and Intelligent Computing and Application Laboratory of Qinghai Province, Qinghai University, Xining, Qinghai, China

3. Tsinghua University, Haidian Qu, Beijing Shi, China

4. Microsoft Research Asia, Haidian Qu, Beijing Shi, China and SKLSDE Lab, Beihang University, Haidian Qu, Beijing Shi, China

5. Beijing University of Posts and Telecommunications, Haidian Qu, Beijing Shi, China

6. Microsoft Research Asia, Haidian Qu, Beijing Shi, China

7. Massachusetts Institute of Technology, Cambridge, Massachusetts, United States

Abstract

Modern information querying systems are progressively incorporating multimodal inputs like vision and audio. However, the integration of gaze --- a modality deeply linked to user intent and increasingly accessible via gaze-tracking wearables --- remains underexplored. This paper introduces a novel gaze-facilitated information querying paradigm, named G-VOILA, which synergizes users' gaze, visual field, and voice-based natural language queries to facilitate a more intuitive querying process. In a user-enactment study involving 21 participants in 3 daily scenarios (p = 21, scene = 3), we revealed the ambiguity in users' query language and a gaze-voice coordination pattern in users' natural query behaviors with G-VOILA. Based on the quantitative and qualitative findings, we developed a design framework for the G-VOILA paradigm, which effectively integrates the gaze data with the in-situ querying context. Then we implemented a G-VOILA proof-of-concept using cutting-edge deep learning techniques. A follow-up user study (p = 16, scene = 2) demonstrates its effectiveness by achieving both higher objective score and subjective score, compared to a baseline without gaze data. We further conducted interviews and provided insights for future gaze-facilitated information querying systems.

Publisher

Association for Computing Machinery (ACM)

Link

https://dl.acm.org/doi/pdf/10.1145/3659623

Reference83 articles.

1. [n. d.]. Microsoft Bing. https://www.bing.com/. Accessed: 2023-09-07.

2. Henny Admoni and Siddhartha Srinivasa. 2016. Predicting user intent through eye gaze for shared autonomy. In 2016 AAAI Fall Symposium Series.

3. Contextual information access with Augmented Reality

4. Can eyes reveal interest? Implicit queries from gaze patterns

5. Flamingo: a visual language model for few-shot learning;Alayrac Jean-Baptiste;Advances in Neural Information Processing Systems,2022