Author:
Paaß Gerhard,Giesselbach Sven
Abstract
AbstractDuring pre-training, a Foundation Model is trained on an extensive collection of documents and learns the distribution of words in correct and fluent language. In this chapter, we investigate the knowledge acquired by PLMs and the larger Foundation Models. We first discuss the application of Foundation Models to specific benchmarks to test knowledge in a large number of areas and examine if the models are able to derive correct conclusions from the content. Another group of tests assesses Foundation Models by completing text and by applying specific probing classifiers that consider syntactic knowledge, semantic knowledge, and logical reasoning separately. Finally, we investigate if the benchmarks are reliable and reproducible, i.e. whether they actually test the targeted properties and yield the same performance values when repeated by other researchers.
Publisher
Springer International Publishing
Reference73 articles.
1. S. Aarohi and R. Abhinav. BIG-bench ⋅ Google, June 20, 2022. url: https://github.com/google/BIG-bench/blob/936c4a5876646966344349b28ae187c556938ec4/docs/paper/BIGbench.pdf (visited on 06/20/2022).
2. M. Aßenmacher and C. Heumann. “On the Comparability of Pre-Trained Language Models”. 2020. arXiv: 2001.00781.
3. S. Balasubramanian, N. Jain, G. Jindal, A. Awasthi, and S. Sarawagi. “What’s in a Name? Are BERT Named Entity Representations Just as Good for Any Other Name?” 2020. arXiv: 2007.06897.
4. A. Belz, S. Agarwal, A. Shimorina, and E. Reiter. “A Systematic Review of Reproducibility Research in Natural Language Processing”. Mar. 21, 2021. arXiv: 2103.07929 [cs].
5. S. R. Bowman and G. E. Dahl. “What Will It Take to Fix Benchmarking in Natural Language Understanding?” 2021. arXiv: 2104.02145.
Cited by
1 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献
1. Commonsense Knowledge in Foundation and Large Language Models;International Journal of Advanced Research in Science, Communication and Technology;2024-02-08