Affiliation:
1. Bar-Ilan University
2. Bar-Ilan University, Harvard Medical School & Massachusetts General Hospital
3. Massachusetts General Hospital
4. Harvard Medical School & Massachusetts General Hospital
Abstract
Abstract
Free-text analysis using Machine Learning (ML)-based Natural Language Processing (NLP) shows promise for diagnosing psychiatric conditions. Chat Generative Pre-trained Transformer (ChatGPT) has demonstrated preliminary initial feasibility for this purpose; however, whether it can accurately assess mental illness remains to be determined. This study evaluates the effectiveness of ChatGPT and the text-embedding-ada-002 (ADA) model in detecting post-traumatic stress disorder following childbirth (CB-PTSD), a maternal postpartum mental illness affecting millions of women annually, with no standard screening protocol. Using a sample of 1,295 women who gave birth in the last six months and were 18 + years old, recruited through hospital announcements, social media, and professional organizations, we explore ChatGPT’s and ADA’s potential to screen for CB-PTSD by analyzing maternal childbirth narratives only. The PTSD Checklist for DSM-5 (PCL-5; cutoff 31) was used to assess CB-PTSD. By developing an ML model that utilizes numerical vector representation of the ADA model, we identify CB-PTSD via narrative classification. Our model outperformed (F1 score: 0.82) ChatGPT and six previously published large language models (LLMs) trained on mental health or clinical domains data, suggesting that the ADA model can be harnessed to identify CB-PTSD. Our modeling approach could be generalized to assess other mental health disorders. 1
Publisher
Research Square Platform LLC
Reference67 articles.
1. Boosting delirium identification accuracy with sentiment-based natural language processing: Mixed methods study;Wang L;JMIR Med. Informatics,2022
2. A transfer learning method for detecting Alzheimer’s disease based on speech and natural language processing;Liu N;Front. Public Health,2022
3. Natural language processing of clinical mental health notes may add predictive value to existing suicide risk models;Levis M;Psycholog. Med.,2021
4. Brown, T. et al. Language models are few-shot learners. Adv. Neural Informat. Proc. Systems. 33, 1877–1901 (2020).
5. Brants, T., Popat, A. C., Xu, P., Och, F. J. & Dean, J. Large language models in machine translation. Proc. 2007 Joint Conf. Empirical Methods in Natural Language Processing and Computational Natural Language Learning. 858–867, Prague, June (2007).