Ten Key Problems of Pre-trained Models: An Outlook of Representation Learning-Reference-Cited by-同舟云学术

Ten Key Problems of Pre-trained Models: An Outlook of Representation Learning

Published:2023 Issue: Volume: Page:491-521
ISSN:
Container-title:Representation Learning for Natural Language Processing
language:
Short-container-title:

Author:

Ding Ning,Chen Weize,Zhang Zhengyan,Hu Shengding,Cui Ganqu,Yao Yuan,Qin Yujia,Zeng Zheni,Han Xu,Liu Zhiyuan,Lin Yankai,Sun Maosong

Abstract

AbstractThe aforementioned representation learning methods have shown their effectiveness in various NLP scenarios and tasks. Large-scale pre-trained language models (i.e., big models) are the state of the art of representation learning for NLP and beyond. With the rapid growth of data scale and the development of computation devices, big models bring us to a new era of AI and NLP. Standing on the new giants of big models, there are many new challenges and opportunities for representation learning. In the last chapter, we will provide a 2023 outlook for the future directions of representation learning techniques for NLP by summarizing ten key open problems for pre-trained models.

Publisher

Springer Nature Singapore

Link

https://link.springer.com/content/pdf/10.1007/978-981-99-1600-9_14

Reference150 articles.

1. Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the limits of large scale pre-training. In Proceedings of ICLR, 2021.

2. David H Ackley, Geoffrey E Hinton, and Terrence J Sejnowski. A learning algorithm for boltzmann machines. Cognitive science, 9(1):147–169, 1985.

3. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. In Proceedings of NeurIPS, 2022.

4. Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv:1902.09229, 2019.

5. Isaac Asimov. I, robot. 1950.