CUBot: Computer Vision on the Eye–Hand Coordination with a Computer-Using Robot and its Implementation

Abstract

Eye–hand coordination (EHC) is of great importance in the research areas of human visual perception, computer vision and robotic vision. A computer-using robot (CUBot) is designed for investigating the EHC mechanism and its implementation is presented in this paper. The CUBot possesses the ability of operating a computer with a mouse like a human being. Based on the three phases of people using computer with a mouse, i.e. watching the screen, recognizing the graphical objects on the screen as well as controlling the mouse to let the cursor approach to the target, our CUBot can also perceive information merely through its vision and control the mouse by its robotic hand without any physical data communication connected to the operated computer. The CUBot is mainly composed of “Mouse-Hand” for operating the mouse, “mind” for realizing the object perception, cursor tracking, and EHC. Two experiments used for testing the ability of our EHC algorithm and the perception of CUBot confirm the feasibility of the proposed approach.