The Automated Generation of Medical Reports from Polydactyly X-ray Images Using CNNs and Transformers

Abstract

Pododactyl radiography is a non-invasive procedure that enables the detection of foot pathologies, as it provides detailed images of structures such as the metatarsus and phalanges, among others. This examination holds potential for employment in CAD systems. Our proposed methodology employs generative artificial intelligence to analyze pododactyl radiographs and generate automatic medical reports. We used a dataset comprising 16,710 exams, including images and medical reports on pododactylys. We implemented preprocessing of the images and text, as well as data augmentation techniques to improve the representativeness of the dataset. The proposed CAD system integrates pre-trained CNNs for feature extraction from the images and Transformers for report interpretation and generation. Our objective is to provide reports describing pododactyl pathologies, such as plantar fasciitis, bunions, heel spurs, flat feet, and lesions, among others, offering a second opinion to the specialist. The results are promising, with BLEU scores (1 to 4) of 0.612, 0.552, 0.507, and 0.470, respectively, a METEOR score of 0.471, and a ROUGE-L score of 0.633, demonstrating the model’s ability to generate reports with qualities close to those produced by specialists. We demonstrate that generative AI trained with pododactyl radiographs has the potential to assist in diagnoses from these examinations.