A Large Language Model-Based Generative Natural Language Processing Framework Finetuned on Clinical Notes Accurately Extracts Headache Frequency from Electronic Health Records

Abstract

AbstractBackgroundHeadache frequency, defined as the number of days with any headache in a month (or four weeks), remains a key parameter in the evaluation of treatment response to migraine preventive medications. However, due to the variations and inconsistencies in documentation by clinicians, significant challenges exist to accurately extract headache frequency from the electronic health record (EHR) by traditional natural language processing (NLP) algorithms.MethodsThis was a retrospective cross-sectional study with human subjects identified from three tertiary headache referral centers-Mayo Clinic Arizona, Florida, and Rochester. All neurology consultation notes written by more than 10 headache specialists between 2012 to 2022 were extracted and 1915 notes were used for model fine-tuning (90%) and testing (10%). We employed four different NLP frameworks: (1)ClinicalBERT (Bidirectional Encoder Representations from Transformers) regression model(2) Generative Pre-Trained Transformer-2 (GPT-2) Question Answering (QA) Model zero-shot (3) GPT-2 QA model few-shot trainingfine-tuned on Mayo Clinic notes; and(4) GPT-2 generative model few-shot trainingfine-tuned on Mayo Clinic notes to generate the answer by considering the context of included text.ResultsThe GPT-2 generative model was the best-performing model with an accuracy of 0.92[0.91 – 0.93] and R2score of 0.89[0.87, 0.9], and all GPT2-based models outperformed the ClinicalBERT model in terms of the exact matching accuracy. Although the ClinicalBERT regression model had the lowest accuracy 0.27[0.26 – 0.28], it demonstrated a high R2score 0.88[0.85, 0.89], suggesting the ClinicalBERT model can reasonably predict the headache frequency within a range of ≤ ± 3 days, and the R2score was higher than the GPT-2 QA zero-shot model or GPT-2 QA model few-shot training fine-tuned model.ConclusionWe developed a robust model based on a state-of-the-art large language model (LLM)-a GPT-2 generative model that can extract headache frequency from EHR free-text clinical notes with high accuracy and R2score. It overcame several challenges related to different ways clinicians document headache frequency that were not easily achieved by traditional NLP models. We also showed that GPT2-based frameworks outperformed ClinicalBERT in terms of accuracy in extracting headache frequency from clinical notes. To facilitate research in the field, we released the GPT-2 generative model and inference code with open-source license of community use in GitHub.