BACKGROUND
Patients’ inability to understand educational materials may explain non-adherence and poor outcomes among those with low health literacy. ChatGPT and similar artificial intelligences (AIs) may serve as accessible tools to improve readability of documents and reduce this gap.
OBJECTIVE
The aim of this study was to investigate ChatGPT’s ability to improve readability of patient education materials.
METHODS
Explanatory case study using three commands in ChatGPT to generate text (“Please reproduce this text at a 6th grade reading level” (A), “Please summarize and simplify the text” (B), and “Make this document more health literate” (C)). ChatGPT version 4.0 was used. Flesch Kincaid Grade Level (FKGL), Flesch Kincaid Reading Ease (FKRE), and SMOG scores were calculated via an online calculator for original and ChatGPT-revised documents. Unpaired T-tests compared mean scores between original and ChatGPT-revised documents.
RESULTS
Education materials were gathered from a large, tertiary-referral academic institution (N=63) and five rural hospitals (N=90; Demopolis, Greenville, Selma, Montgomery, and Shelby). Documents from the academic institution included ostomy (n=15), pre-operative (n=20), and post-operative (n=28) material. Compared to original documents, ChatGPT-revised documents using Prompts B and C scored worse in FKGL, FKRE, and SMOG scores (p<0.01). Documents generated using Prompt A produced a higher FKGL (p<0.01), but no change in FKRE or SMOG scores. Documents from rural hospitals included education related to colonoscopy (n=15), diabetes care (n=18), cardiovascular health (n=12), general health (n=10), surgical care (n=29), and other screening and prevention (n=6). Compared to original documents, ChatGPT documents using Prompt A improved FKRE (p<0.01) and SMOG scores (p=0.01), Prompt B was associated with worse FKGL (p<0.01), and Prompt C was associated with worse FKGL (p<0.01) and FKRE (p<0.01).
CONCLUSIONS
ChatGPT does not consistently improve the readability of patient educational materials. Further work is needed to optimize text-based AI’s to be used for this purpose.