Real‐time surgical tool detection with multi‐scale positional encoding and contrastive learning

Abstract

AbstractReal‐time detection of surgical tools in laparoscopic data plays a vital role in understanding surgical procedures, evaluating the performance of trainees, facilitating learning, and ultimately supporting the autonomy of robotic systems. Existing detection methods for surgical data need to improve processing speed and high prediction accuracy. Most methods rely on anchors or region proposals, limiting their adaptability to variations in tool appearance and leading to sub‐optimal detection results. Moreover, using non‐anchor‐based detectors to alleviate this problem has been partially explored without remarkable results. An anchor‐free architecture based on a transformer that allows real‐time tool detection is introduced. The proposal is to utilize multi‐scale features within the feature extraction layer and at the transformer‐based detection architecture through positional encoding that can refine and capture context‐aware and structural information of different‐sized tools. Furthermore, a supervised contrastive loss is introduced to optimize representations of object embeddings, resulting in improved feed‐forward network performances for classifying localized bounding boxes. The strategy demonstrates superiority to state‐of‐the‐art (SOTA) methods. Compared to the most accurate existing SOTA (DSSS) method, the approach has an improvement of nearly 4% on mAP50 and a reduction in the inference time by 113%. It also showed a 7% higher mAP50 than the baseline model.