Assessment of various GPT Models versus the human text: a quantitative analysis of lexis and cohesion-Reference-Cited by-同舟云学术

Assessment of various GPT Models versus the human text: a quantitative analysis of lexis and cohesion

Published:2023-12 Issue:1(10)2022 Volume: Page:139-156
ISSN:2353-2912
Container-title:Forum Filologiczne Ateneum
language:
Short-container-title:

Author:

Skowron Dominik¹^ORCID,Bączkowska Anna²^ORCID

Affiliation:

1. University of Economics in Krakow

2. University of Gdańsk / Eviden.com, Big Data and Security

Abstract

The aim of the paper is to analyse select lexical parameters and text cohesion of academic texts automatically generated by several OpenAI LLM models (GPT) in order to (1) investigate the quality of GPT output relative to the original text; (2) compare the quality of various GPT models. The material used in the study comprised a fragment of a research article. The method of analysis involved NLP-based text analysis tools that focus on the examination of various lexical and text cohesion parameters. The study shows that the similarity between AI-generated texts and the text written by the human author is very high and that there is no single model which would achieve the highest values for all linguistic indices.

Publisher

Ateneum Szkola Wyzsza

Reference14 articles.

1. Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet Allocation. Journal of Machine Learning Research, 3, 993–1022. https://jmlr.csail.mit.edu/papers/v3/blei03a.html

2. Crossley, S. A., Kyle, K., & McNamara, D. S. (2016). The tool for the automatic analysis of text cohesion (TAACO): Automatic assessment of local, global, and text cohesion. Behavior Research Methods, 48(4), 1227-1237. https://doi.org/10.3758/s13428-015-0651-7

3. Davies, M. (2010). The corpus of contemporary American English as the first reliable monitor corpus of English. Literary and Linguistic Computing, 25(4), 447–464. https://doi.org/10.1093/llc/fqq018

4. Eslami, Z. R., Raeisi-Vanani, A., & Anani Sarab, M. R. (2022). Variation Patterns in Interlanguage Pragmatics: Apology Speech Act of EFL Learners vs. American Native Speakers. Contrastive Pragmatics, 4(1), 27-63. https://doi.org/10.1163/26660393-bja10068

5. Guiraud, P. L. (1960). Problèmes Et Méthodes De La Statistique Linguistique/ Problems and Methods of Statistical Linguistics. Dordrecht: Kluwer.