Deep Bidirectional Recurrent Neural Networks as End-To-End Models for Smoking Status Extraction from Clinical Notes in Spanish

Abstract

AbstractIntroductionAlthough natural language processing (NLP) tools have been available in English for quite some time, this is not the case for many other languages, particularly for texts from specific contexts such as clinical texts. This poses a challenge for tasks such as classifying text in languages other than English. In the absence of basic NLP tools, the development of statistical models that include manually designed variables that capture the semantic information of the documents is a potential solution. However, this process is expensive and slow. Deep recurrent neural networks (RNNs) have been proposed as “end-to-end” models that learn both variables and parameters jointly, thus avoiding manual feature engineering and saving development time.MethodsWe compared the performance of two strategies for labeling clinical notes of an electronic medical record in Spanish according to the patient’s smoking status (current smoker, current non-smoker, text without information on tobacco): 1. A traditional approach using two classifiers (a multilayer perceptron (MLP) and a support vector machine (SVM)) together with a ‘bag-of-words’ text representation that involves intensive manual development of features and, 2. an ’end-to-end’ model which uses a Short-Long-Term Memory bidirectional deep RNN with GloVe word embeddings. The classifiers were trained in the training set (n = 11775 clinical texts) and were evaluated in the test set (n = 2943) by means of macro-averaged recall, precision and F1 score.ResultsThe RNN scored high values of all three metrics in the test set (sensitivity [95% CI]: 0.965 [0.96, 0.97], PPV: 0.963 [0.96, 0.97], F1 score: 0.964 [0.96, 0.97]). It also showed to be slightly superior to the MLP (difference in recall: 0.009 [95% CI: -0.0007, 0.017], precision: 0.007 [95% CI: -0.0015, 0.019] and F1 score: 0.009 [95% CI: 0.0018, 0.016]); comparing the RNN with the SVM, the latter has a better performance in general (recall difference [95% CI]: -0.007 [-0.016, 0.0018], precision: -0.009 [-0.018, 0.00015] and score F1: -0.008 [-0.014, -0.0017]). In both cases only the confidence interval for the F1 score difference excludes zero. In turn, the RNN consumed 80% less overall development time.ConclusionIn our work, the deep bidirectional RNN as end-to-end model, reached similar levels of performance in the classification of clinical texts in Spanish that models with a great manual engineering of variables, although in less than 20% of the development time. This makes them an important tool to streamline text processing in languages where the development of NLP tools has not progressed as much as in English. Areas such as research or public health management could clearly benefit from ’end-to-end’ models that facilitate the exploitation of already available data sources, such as electronic clinical records.