Abstract
We analyze human’s disagreements about the validity of natural language inferences. We show that, very often, disagreements are not dismissible as annotation “noise”, but rather persist as we collect more ratings and as we vary the amount of context provided to raters. We further show that the type of uncertainty captured by current state-of-the-art models for natural language inference is not reflective of the type of uncertainty present in human disagreements. We discuss implications of our results in relation to the recognizing textual entailment (RTE)/natural language inference (NLI) task. We argue for a refined evaluation objective that requires models to explicitly capture the full distribution of plausible human judgments.
Subject
Artificial Intelligence,Computer Science Applications,Linguistics and Language,Human-Computer Interaction,Communication
Cited by
37 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献
1. AmbiFC: Fact-Checking Ambiguous Claims with Evidence;Transactions of the Association for Computational Linguistics;2024
2. Modeling Uncertainty in Personalized Emotion Prediction with Normalizing Flows;2023 IEEE International Conference on Data Mining Workshops (ICDMW);2023-12-04
3. Towards Cognitively More Plausible Models;Cognitive Plausibility in Natural Language Processing;2023-10-31
4. Behavioral Patterns;Cognitive Plausibility in Natural Language Processing;2023-10-31
5. A Quality Assessment Framework for Information Extraction in Job Advertisements;SN Computer Science;2023-10-12