Towards LLMCI - Multimodal AI for LLM-Vision UI Operation

Abstract

Human-computer interaction (HCI) has evolved significantly, yet it still largely depends on visual communication through screens and manual input devices. While this paradigm is likely to remain dominant for the foreseeable future, this research suggests that existing user interfaces (UI) can also be leveraged by Large Language Models (LLMs) to interact with computers. By integrating vision models into a multimodal framework, LLMs can gain the ability to understand and operate UI elements, enabling them to retrieve information, run functions, and perform various tasks just like humans. The framework utilizes a vision model to communicate UI components and information to the LLM, which then leverages its language understanding capabilities to retrieve information, and operate keyboard and mouse inputs.This paper introduces a new element to Human-Computer Interaction (HCI), called LLM-Computer Interaction (LLMCI), which combines Large Language Models (LLMs) with computer vision via intelligent agents. These agents process user text commands and use visual perception to recognize visual and textual elements of computer interfaces. This allows the Multimodal AI to independently perform complex tasks and navigate applications in a way that resembles human behavior. We present a proof-of-concept framework that illustrates how the agent uses LLMs and computer vision to handle interface elements, complete tasks, and support users according to their instructions. This strategy closely imitates human interactions and suggests a path forward for enhancing HCI practices.