Abstract
AbstractObjectiveto assess the diagnostic accuracy of services based on computer vision technologies at the integration and operation stages in Moscow’s Unified Radiological Information Service (URIS).Methodsthis is a multicenter diagnostic study of artificial intelligence (AI) services with retrospective and prospective stages. The minimum acceptable criteria levels for the index test were established, justifying the intended clinical application of the investigated index test. The Experiment was based on the infrastructure of the URIS and United Medical Information and Analytical System (UMIAS) of Moscow. Basic functional and diagnostic requirements for the artificial intelligence services and methods for monitoring technological and diagnostic quality were developed. Diagnostic accuracy metrics were calculated and compared.Resultsbased on the results of the retrospective study, we can conclude that AI services have good result reproducibility on local test sets. The highest and at the same time most balanced metrics were obtained for AI services processing CT scans. All AI services demonstrated a pronounced decrease in diagnostic accuracy in the prospective study. The results indicated a need for further refinement of AI services with additional training on the Moscow population datasets.Conclusionsthe diagnostic accuracy and reproducibility of AI services on the reference data are sufficient, however, they are insufficient on the data in routine clinical practice. The AI services that participated in the experiment require a technological improvement, additional training on Moscow population datasets, technical and clinical trials to get a status of a medical device.
Publisher
Cold Spring Harbor Laboratory
Reference53 articles.
1. Artificial intelligence in medical imaging: threat or opportunity? Radiologists again at the forefront of innovation in medicine
2. Ranschaert ER , Morozov S , Algra PR . Artificial intelligence in medical imaging: Opportunities, applications and risks. Artif Intell Med Imaging Oppor Appl Risks 2019; 1–373.
3. Decree of the President of the Russian Federation No. 490 of October 10, 2019 “On the Development of Artificial Intelligence in the Russian Federation”. 2019.
4. Order of the Ministry of Health of the Russian Federation No. 560n of June 9, 2020 ‘On approval of the Regulations for conducting X-ray examinations’. 2020.
5. Regulation of the Government of Moscow No. 1543-PP of November 21, 2019 “On Conducting the Experiment on the Use of Innovative Technologies in the Field of Computer Vision for the Analysis of Medical Images and Subsequent Application in the Healthcare Sys. 2019.