Identifying Incarceration Status in the Electronic Health Record Using Natural Language Processing in Emergency Department Settings

Abstract

ABSTRACTBackgroundIncarceration is a highly prevalent social determinant of health associated with high rates of morbidity and mortality and racialized health inequities. Despite this, incarceration status is largely invisible to health services research due to poor electronic health record capture within clinical settings. Our primary objective is to develop and assess natural language processing (NLP) techniques for identifying incarceration status from clinical notes to improve clinical sciences and delivery of care for millions of individuals impacted by incarceration.MethodsWe annotated 1,000 unstructured clinical notes randomly selected from the emergency department for incarceration history. Of these annotated notes, 80% were used to train the Longformer-based and RoBERTa NLP models. The remaining 20% served as the test set. Model performance was evaluated using accuracy, sensitivity, specificity, precision, F1 score and Shapley values.ResultsOf annotated notes, 55.9% contained evidence for incarceration history by manual annotation. ICD-10 code identification demonstrated accuracy of 46.1%, sensitivity of 4.8%, specificity of 99.1%, precision of 87.1%, and F1 score of 0.09. RoBERTa NLP demonstrated an accuracy of 77.0%, sensitivity of 78.6%, specificity of 73.3%, precision of 80.0%, and F1 score of 0.79. Longformer NLP demonstrated an accuracy of 91.5%, sensitivity of 94.6%, specificity of 87.5%, precision of 90.6%, and F1 score of 0.93.ConclusionThe Longformer-based NLP model was effective in identifying patients’ exposure to incarceration and has potential to help address health disparities by enabling use of electronic health records to study quality of care for this patient population and identify potential areas for improvement.