The validation of orthodontic artificial intelligence systems that perform orthodontic diagnoses and treatment planning

Abstract

Summary Aim This study was aimed to evaluate two artificial intelligence (AI) systems that created a prioritized problem list and treatment plan, and examine whether the performance of the aforementioned systems was equivalent to orthodontists. Materials and methods A total of 967 consecutive cases [800: training; 67: validation; 100: evaluation (40: randomly selected for the clinical evaluation)] were used. We used a stored document that describes (1) the patient’s clinical information, (2) the prioritized list, and (3) a treatment strategy without digital tooth movement. Sentences of (1) were vectorized according to the bag of words method (V); sentences of (2) and (3) were relabelled with 423 and 330 labels, respectively. AI systems that output labels for the prioritized list (subtask 1) and treatment planning (subtask 2) based on the vectors V were developed using a support vector machine and self-attention network, respectively, while the system was trained to improve precision and recall. Clinical evaluations were conducted by four orthodontists (no faculty or residents; peer group) in two sessions: in the first session, peer group and the developed AI systems created problem lists and treatment plans; in the second session, two of the peer group (not AI) evaluated these lists and plans, including the lists and plans of the AIs, by scoring them using 4-point scales [unacceptable (1) to ideal (4)]. Scores were compared among the system and peer group (Wilcoxon signed-rank test, P < 0.05). Results The precision after system training was 65% and 48% for subtasks 1 and 2 respectively, with recall of 55% and 48%, respectively. The clinical evaluation of the AI system for subtask 1 showed a mid-rank. For subtask 2, the AI system had a significantly lower score than the three panels but the same rank with one panel. Conclusions Two AI systems that output a prioritized problem list and create a treatment plan were developed. The clinical system ability of the former system showed a mid-rank in the peer group, and the latter system was almost equivalent to the worst orthodontist.