Multi-modal interaction with transformers: bridging robots and human with natural language
Author:
Wang ShaochenORCID,
Zhou Zhangli,
Li Bin,
Li Zhijun,
Kan Zhen
Abstract
AbstractThe language-guided visual robotic grasping task focuses on enabling robots to grasp objects based on human language instructions. However, real-world human-robot collaboration tasks often involve situations with ambiguous language instructions and complex scenarios. These challenges arise in the understanding of linguistic queries, discrimination of key concepts in visual and language information, and generation of executable grasping configurations for the robot’s end-effector. To overcome these challenges, we propose a novel multi-modal transformer-based framework in this study, which assists robots in localizing spatial interactions of objects using text queries and visual sensing. This framework facilitates object grasping in accordance with human instructions. Our developed framework consists of two main components. First, a visual-linguistic transformer encoder is employed to model multi-modal interactions for objects referred to in the text. Second, the framework performs joint spatial localization and grasping. Extensive ablation studies have been conducted on multiple datasets to evaluate the advantages of each component in our model. Additionally, physical experiments have been performed with natural language-driven human-robot interactions on a physical robot to validate the practicality of our approach.
Publisher
Cambridge University Press (CUP)
Subject
Computer Science Applications,General Mathematics,Software,Control and Systems Engineering,Control and Optimization,Mechanical Engineering,Modeling and Simulation,Artificial Intelligence,Computer Vision and Pattern Recognition,Computational Mechanics,Rehabilitation
Reference67 articles.
1. Learning to Compose and Reason with Language Tree Structures for Visual Grounding
2. Human-in-the-loop adaptive control of a soft exo-suit with actuator dynamics and ankle impedance adaptation;Li;IEEE Trans. Cybern.,2023
Cited by
1 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献
1. What You See is what You Grasp: User-Friendly Grasping Guided by Near-eye-tracking;2023 IEEE International Conference on Development and Learning (ICDL);2023-11-09