Capturing the Varieties of Natural Language Inference: A Systematic Survey of Existing Datasets and Two Novel Benchmarks-Reference-Cited by-同舟云学术

Capturing the Varieties of Natural Language Inference: A Systematic Survey of Existing Datasets and Two Novel Benchmarks

Published:2023-11-20 Issue: Volume: Page:
ISSN:0925-8531
Container-title:Journal of Logic, Language and Information
language:en
Short-container-title:J of Log Lang and Inf

Author:

Gubelmann Reto,Katis Ioannis,Niklaus Christina,Handschuh Siegfried

Abstract

AbstractTransformer-based Pre-Trained Language Models currently dominate the field of Natural Language Inference (NLI). We first survey existing NLI datasets, and we systematize them according to the different kinds of logical inferences that are being distinguished. This shows two gaps in the current dataset landscape, which we propose to address with one dataset that has been developed in argumentative writing research as well as a new one building on syllogistic logic. Throughout, we also explore the promises of ChatGPT. Our results show that our new datasets do pose a challenge to existing methods and models, including ChatGPT, and that tackling this challenge via fine-tuning yields only partly satisfactory results.

Funder

University of St.Gallen

Publisher

Springer Science and Business Media LLC

Subject

Linguistics and Language,Philosophy,Computer Science (miscellaneous)

Link

https://link.springer.com/content/pdf/10.1007/s10849-023-09410-4.pdf

Reference66 articles.

1. Aristotle, J. B. (1984). Prior analytics. In J. Barnes (Ed.), The complete works of aristotle (pp. 39–113). Oxford University Press.

2. Asael, D., Ziegler, Z., & Belinkov, Y. (2021). A generative approach for mitigating structural biases in natural language inference. arXiv preprint arXiv:2108.14006

3. Baker, C. F., Fillmore, C. J., & Lowe, J. B. (1998). The berkeley framenet project. In COLING 1998 volume 1: The 17th international conference on computational linguistics.

4. Bar Haim, R., Dagan, I., Dolan, B., Ferro, L., Giampiccolo, D., Magnini, B., & Szpektor, I. (2006). The second PASCAL RTE challenge. In Proceedings of the 2nd PASCAL challenge on RTE.

5. Bentivogli, L., Clark, P., Dagan, I., & Giampiccolo, D. (2009). The 5th PASCAL recognizing textual entailment challenge. In TAC.