Validation of RUBY for Breast Cancer Knowledge Extraction From a Large French Electronic Medical Record System

Abstract

PURPOSE RUBY is a tool for extracting clinical data on breast cancer from French medical records on the basis of named entity recognition models combined with keyword extraction and postprocessing rules. Although initial results showed a high precision of the system in extracting clinical information from surgery, pathology, and biopsy reports (≥92.7%) and good precision in extracting data from consultation reports (81.8%), its validation is needed before its use in routine practice. METHODS In this work, we analyzed RUBY's performance compared with the manual entry and we evaluated the generalizability of the approach on different sets of reports collected on a span of 40 years. RESULTS RUBY performed similarly or better than the manual entry for 15 of 27 variables. It showed similar performances when structuring newer reports but failed to extract entities for which changes in terminology appeared. Finally, our tool could automatically structure 15,990 reports in 77 minutes. CONCLUSION RUBY can automate the data entry process of a set of variables and reduce its burden, but a continuous evaluation of the format and structure of the reports and a subsequent update of the system is necessary to ensure its robustness.