Affiliation:
1. Department of Linguistics, The Ohio State University, USA. oh.531@osu.edu
2. Department of Linguistics, The Ohio State University, USA. schuler.77@osu.edu
Abstract
AbstractThis work presents a linguistic analysis into why larger Transformer-based pre-trained language models with more parameters and lower perplexity nonetheless yield surprisal estimates that are less predictive of human reading times. First, regression analyses show a strictly monotonic, positive log-linear relationship between perplexity and fit to reading times for the more recently released five GPT-Neo variants and eight OPT variants on two separate datasets, replicating earlier results limited to just GPT-2 (Oh et al., 2022). Subsequently, analysis of residual errors reveals a systematic deviation of the larger variants, such as underpredicting reading times of named entities and making compensatory overpredictions for reading times of function words such as modals and conjunctions. These results suggest that the propensity of larger Transformer-based models to ‘memorize’ sequences during training makes their surprisal estimates diverge from humanlike expectations, which warrants caution in using pre-trained language models to study human language processing.
Subject
Artificial Intelligence,Computer Science Applications,Linguistics and Language,Human-Computer Interaction,Communication
Reference52 articles.
1. Syntactic surprisal from neural models predicts, but underestimates, human processing difficulty from syntactic ambiguities;Arehalli,2022
2. Comparing gated and simple recurrent neural network architectures as models of human sentence processing;Aurnhammer,2019
3. Fitting linear mixed-effects models using lme4;Bates;Journal of Statistical Software,2015
4. GPT-NeoX-20B: An open-source autoregressive language model;Black,2022
5. GPT-Neo: Large scale autoregressive language modeling with Mesh-Tensorflow;Black;Zenodo,2021
Cited by
10 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献