Vision‐Based Online Key Point Estimation of Deformable Robots

Abstract

The precise control of soft and continuum robots requires knowledge of their shape, which has, in contrast to classical rigid robots, infinite degrees of freedom. To partially reconstruct the shape, proprioceptive techniques use built‐in sensors, resulting in inaccurate results and increased fabrication complexity. Exteroceptive methods so far rely on expensive tracking systems with reflective markers placed on all components, which are infeasible for deformable robots interacting with the environment due to marker occlusion and damage. Here, a regression approach is presented for three‐dimensional key point estimation using a convolutional neural network. The proposed approach uses data‐driven supervised learning and is capable of online markerless estimation during inference. Two images of a robotic system are captured simultaneously at 25 Hz from different perspectives and fed to the network, which returns for each pair the parameterized key point or piecewise constant curvature shape representations. The proposed approach outperforms markerless state‐of‐the‐art methods by a maximum of 4.5% in estimation accuracy while being more robust and requiring no prior knowledge of the shape. Online evaluations on two types of soft robotic arms and a soft robotic fish demonstrate the method's accuracy and versatility on highly deformable systems.