BACKGROUND
Large Language Models (LLMs), exemplified by ChatGPT, have reached a level of sophistication where distinguishing between human and AI-generated texts is challenging.
OBJECTIVE
To assess the implications for medical texts, the aim of this experimental study was to investigate the ability of two blinded expert groups, one medical and one humanistic, to differentiate between texts written by medical students and those generated by ChatGPT.
METHODS
The medical experts (n=22) were characterised by content familiarity and the humanities experts (n=13) by linguistic and formal textual analysis expertise. All experts were presented with two pairs of texts on two dif-ferent topics, each pair similar in content and structure - one text written by a medical student and the other generated by ChatGPT. They were requested to identify the texts as human-generated or AI-generated and to reason their decision. They were also requested to rate some characteristics of a text: linguistic quality, style, logical coherence, scientific quality, recognition of knowledge limitations, formulation of future research questions, and spelling and grammatical errors.
RESULTS
About 70% of all participants correctly identified the text written by ChatGPT. No significant difference was found between the two groups in terms of correct identification. Only 14% of participants misidentified the author in both text pairs. Familiarity with the content did not play a major role, but certain features of the writ-ing style were more important in the decision-making process. In particular, characteristics in the linguistic categories of redundancy, repetition, and thread/coherence proved to be decisive for the acceptance of a ChatGPT text.
CONCLUSIONS
Authoring style and personal writing features should further be investigated, especially in view of the major change in academic writing emerging by the presence of LLMs.
CLINICALTRIAL
The project was submitted to the ethics committee of the Ruhr University Bochum, Germany, in April 2023. As this is not a clinical trial on human subjects, no study or trial registration was required.