EA-CTFVS: An Environment-Agnostic Coarse-to-Fine Visual Servoing Method for Sub-Millimeter-Accurate Assembly

Abstract

Peg-in-hole assembly, a crucial component of robotic automation in manufacturing, continues to pose challenges due to its strict tolerance requirements. To date, most conventional peg-in-hole assembly algorithms have been validated only within simulated environments or under limited observational scenarios. In this paper, an environment-agnostic coarse-to-fine visual servoing (EA-CTFVS) assembly algorithm is proposed. Firstly, to solve the frequent issue of visual blindness during visual servoing, a bottleneck pose is proposed to be used as the desired pose for the visual servoing. Secondly, to achieve accurate assembly, a coarse-to-fine framework is constructed, in which a rough pose is given by the coarse controller to remove large initial alignment errors. For the fine controller, a twin network-based fine controller is provided to improve assembly accuracy. Furthermore, EA-CTFVS utilizes the Oriented Bounding Box (OBB) of objects as the input for visual servoing, which guarantees the system’s ability to operate effectively in diverse and complex scenes. The proposed EA-CTFVS achieves a successful assembly rate of 0.92/0.89 for initial alignment errors of 15/30 cm and 0.6 mm tolerance in real-world D-sub plug assembly tasks under complex scenarios.