Automatic Evaluation of Cancer Treatment Texts for Gist Inferences and Comprehension

Abstract

Background. It is difficult to write about cancer for laypeople such that everyone understands. One common approach to readability is the Flesch-Kincaid Grade Level (FKGL). However, FKGL has been shown to be less effective than emerging discourse technologies in predicting readability. Objective. Guided by fuzzy-trace theory, we used the discourse technology Coh-Metrix to create a Gist Inference Score (GIS) and applied it to texts from the National Cancer Institute website written for patients and health care providers. We tested the prediction that patient cancer texts with higher GIS scores are likely to be better understood than others. Design. In study 1, all 244 cancer texts were systematically subjected to an automated Coh-Metrix analysis. In study 2, 9 of those patient texts (3 each at high, medium, and low GIS) were systematically converted to fill-the-blanks (Cloze) tests in which readers had to supply the missing words. Participants (162) received 3 texts, 1 at each GIS level. Measures. GIS was measured as the mean of 7 Coh-Metrix variables, and comprehension was measured through a Cloze procedure. Results. Although texts for patients scored lower on FKGL than those for providers, they also scored lower on GIS, suggesting difficulties for readers. In study 2, participants scored higher on the Cloze task for high GIS texts than for low- or medium-GIS texts. High-GIS texts seemed to better lend themselves to correct responses using different words. Limitations. GIS is limited to text and cannot assess inferences made from images. The systematic Cloze procedure worked well in aggregate but does not make fine-grained distinctions. Conclusions. GIS appears to be a useful, theoretically motivated supplement to FKGL for use in research and clinical practice.