Towards a Holistic Framework for Multimodal Large Language Models in Three-dimensional Brain CT Report Generation

Abstract

Multi-modal large language models (MLLMs) have been given free rein to explore exciting medical applications with a primary focus on radiology report generation. Nevertheless, the preliminary MLLM successful attempts in 2D medical image-text pair captioning are incompetent to reflect the real-world diagnostic challenge in the volumetric 3D anatomy. Toward deploying MLLM for more applicable diagnostic context, we noticed that the (1) scarcity of 3D image training dataset, (2) the direct use of undifferentiated foundation MLLMs, and (3) the lack of pertinent caption evaluation metrics were independent domain-specific constraints that integratively hobbles the iteration of next-generation medical MLLM research. In this regard, this study collected a 3D-BrainCT dataset (18,885 text-scan pairs) and applied clinical visual instruction tuning (CVIT) to train volumetric anatomy-sensible BrainGPT models to generate radiology-adherent 3D brain CT reports. Statistically, our BrainGPT model scored BLEU-1 = 44.35, BLEU-4 = 20.38, METEOR = 30.13, ROUGE-L = 47.6, and CIDEr-R = 211.77 during internal testing and demonstrated an accuracy of 0.91 in captioning midline shifts on the external validation CQ500 dataset. By further inspecting the captioned report, we reported that the traditional metrics appeared to measure only the surface text similarity and failed to gauge the information density of the diagnostic purpose. To close this gap, we proposed a novel Feature-Oriented Radiology Task Evaluation (FORTE) to estimate the clinical relevance (lesion feature and landmarks) of the report. Notably, the BrainGPT model scored an average FORTE 0.71 F1-score (degree=0.661; landmark=0.706; feature=0.693, and impression=0.779). To demonstrate that BrainGPT models possess objective readiness to generate human-like radiology reports, we conducted a Turing test that enrolled 11 physician evaluators, and around 74% of the BrainGPT-generated captions were indistinguishable from those written by humans. While various computational intelligence researchers have advocated the avant-garde MLLM applications, our work embodies a holistic framework that showcased the first-hand experience of curating a 3D brain CT dataset, fine-tuning anatomy-sensible language models, and proposing robust radiology evaluation metrics. We deemed that the adventure of docking MLLM for 3D brain CT report generation may unfold new MLLM applications at the forefront of human-machine collaborated modern healthcare.