Utilizing Large Language Models for Enhanced Clinical Trial Matching: A Study on Automation in Patient Screening
Author:
Beattie Jacob,Neufeld Sarah,Yang Daniel,Chukwuma Christian,Gul Ahmed,Desai Neil,Jiang Steve,Dohopolski Michael
Abstract
AbstractBackgroundClinical trial matching, essential for advancing medical research, involves detailed screening of potential participants to ensure alignment with specific trial requirements. Research staff face challenges due to the high volume of eligible patients and the complexity of varying eligibility criteria. The traditional manual process, both time-consuming and error-prone, often leads to missed opportunities. Utilizing Artificial Intelligence (AI) and Natural Language Processing (NLP) can significantly enhance the accuracy and efficiency of this process through automated patient screening against established criteria.MethodsUtilizing data from the National NLP Clinical Challenges (n2c2) 2018 Challenge, we utilized 202 longitudinal patient records. These records were annotated by medical professionals and evaluated against 13 selection criteria encompassing various health assessments. Our approach involved embedding medical documents into a vector database to determine relevant document sections, then using a large language model (GPT-3.5 Turbo and GPT-4 OpenAI API) in tandem with structured and chain-of-thought prompting techniques for systematic document assessment against the criteria. Misclassified criteria were also examined to identify classification challenges.ResultsThis study achieved an accuracy of 0.81, sensitivity of 0.80, specificity of 0.82, and a micro F1 score of 0.79 using GPT-3.5 Turbo, and an accuracy of 0.87, sensitivity of 0.85, specificity of 0.89, and micro F1 score of 0.86 using GPT-4 Turbo. Notably, some criteria in the ground truth appeared mislabeled, an issue we couldn’t explore further due to insufficient label generation guidelines on the website.ConclusionOur findings underscore the significant potential of AI and NLP technologies, including large language models, in the clinical trial matching process. The study demonstrated strong capabilities in identifying eligible patients and minimizing false inclusions. Such automated systems promise to greatly alleviate the workload of research staff and improve clinical trial enrollment, thus accelerating the process and enhancing the overall feasibility of clinical research.
Publisher
Cold Spring Harbor Laboratory
Reference22 articles.
1. Global public attitudes about clinical research and patient experiences with clinical trials;JAMA Network Open,2018 2. Brown, T. B. , Mann, B. , Ryder, N. , Subbiah, M. , Kaplan, J. , Dhariwal, P. , Nee-lakantan, A. , Shyam, P. , Sastry, G. , Askell, A. , Agarwal, S. , Herbert-Voss, A. , Krueger, G. , Henighan, T. , Child, R. , Ramesh, A. , Ziegler, D. M. , Wu, J ., Winter, C. , Hesse, C. , Chen, M. , Sigler, E. , Litwin, M. , Gray, S. , Chess, B. , Clark, J. , Berner, C. , McCandlish, S. , Radford, A. , Sutskever, I. , and Amodei, D . Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems (Red Hook, NY, USA, 2020), NIPS’20, Curran Associates Inc. 3. Chen, J. W. , Kunder, C. A. , Bui, N. , Zehnder, J. L. , Costa, H. A. , and Stehr, H . Increasing clinical trial accrual via automated matching of biomarker criteria. In Biocomputing 2020 (2019), WORLD SCIENTIFIC, pp. 31–42. 4. Provider motivations and barriers to cancer clinical trial screening, referral, and operations: Findings from a survey;Cancer,2023 5. for Information, C., and on Clinical Research, S. General perceptions. Tech. rep., Center for Information and Study on Clinical Research Participation, 2021.
|
|