Research on Surgical Gesture Recognition in Open Surgery Based on Fusion of R3D and Multi-Head Attention Mechanism

Abstract

Surgical gesture recognition is an important research direction in the field of computer-assisted intervention. Currently, research on surgical gesture recognition primarily focuses on robotic surgery, with a lack of studies in traditional surgery, particularly open surgery. Therefore, this study established a dataset simulating open surgery for research on surgical gesture recognition in the field of open surgery. With the assistance of professional surgeons, we defined a vocabulary of 10 surgical gestures based on suturing tasks in open procedures. In addition, this paper proposes a surgical gesture recognition method that integrates the R3D network with a multi-head attention mechanism (R3D-MHA). This method uses the R3D network to extract spatiotemporal features and combines it with the multi-head attention mechanism for relational learning of these features. The effectiveness of the R3D-MHA method in the field of open surgery gesture recognition was validated through two experiments: offline recognition and online recognition. The accuracy at the gesture instance level for offline recognition was 92.3%, and the frame accuracy for online recognition was 73.4%. Finally, its performance was further validated on the publicly available JIGSAWS dataset. Compared to other online recognition methods, the accuracy improved without using additional data. This work lays the foundation for research on surgical gesture recognition in open surgery and has significant applications in process monitoring, surgeon skill assessment and educational training for open surgeries.