DeltaAI: Semi-Autonomous Tissue Grossing Measurements and Recommendations using Neural Radiance Fields for Rapid, Complete Intraoperative Histological Assessment of Tumor Margins

Abstract

AbstractMohs Micrographic Surgery (MMS) aims to excise cutaneous cancer with real-time margin analysis. However, manual tissue grossing and analysis can be inefficient, so we propose DeltaAI, a novel workflow that utilizes Neural Radiance Fields (NeRF) to enable rapid tissue grossing and generate a 3D model in an augmented reality (AR) environment. In our study, we captured 30-second videos of 17 MMS specimens using a photogrammetry turntable and cellphone camera. Preprocessing the tissues with segmentation models, we created a dataset of 923, 360-degree-view, images per video (17 videos). Using COLMAP, we estimated poses for sparse tissue reconstructions and trained the NeRF model for 3D volumetric tissue renderings. The results demonstrated that DeltaAI generated more accurate and complete 360-degree, 3D tissue renderings compared to previous models, while also achieving significantly faster runtimes. Our proposed semi-autonomous NeRF-based workflow has the potential to enhance the speed of MMS specimen processing, measurement, report generation, and margin assessment. It can inform real-time grossing decisions, automate the export of electronic health record data, and facilitate time-efficient and complete cancer excisions. Moreover, DeltaAI can contribute to the wider adoption of AI technology in clinical settings by improving tissue modeling for manual grossing.