Affiliation:
1. Stanford University, USA
2. Columbia University, USA
Abstract
Abstract
Large language models (LLMs) have shown promise for automatic summarization but the reasons behind their successes are poorly understood. By conducting a human evaluation on ten LLMs across different pretraining methods, prompts, and model scales, we make two important observations. First, we find instruction tuning, not model size, is the key to the LLM’s zero-shot summarization capability. Second, existing studies have been limited by low-quality references, leading to underestimates of human performance and lower few-shot and finetuning performance. To better evaluate LLMs, we perform human evaluation over high-quality summaries we collect from freelance writers. Despite major stylistic differences such as the amount of paraphrasing, we find that LLM summaries are judged to be on par with human written summaries.
Subject
Artificial Intelligence,Computer Science Applications,Linguistics and Language,Human-Computer Interaction,Communication
Reference73 articles.
1. Neural machine translation by jointly learning to align and translate;Bahdanau,2015
2. Training a helpful and harmless assistant with reinforcement learning from human feedback;Bai;arXiv,2022
3. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments;Banerjee,2005
4. Using lexical chains for text summarization;Barzilay,1997
5. Sentence fusion for multidocument news summarization;Barzilay;Computational Linguistics,2005