Author:
Fang Peng-Fei,Li Xian,Yan Yang,Zhang Shuai,Kang Qi-Yue,Li Xiao-Fei,Lan Zhen-Zhong
Abstract
AbstractThe artificial intelligence (AI) community has recently made tremendous progress in developing self-supervised learning (SSL) algorithms that can learn high-quality data representations from massive amounts of unlabeled data. These methods brought great results even to the fields outside of AI. Due to the joint efforts of researchers in various areas, new SSL methods come out daily. However, such a sheer number of publications make it difficult for beginners to see clearly how the subject progresses. This survey bridges this gap by carefully selecting a small portion of papers that we believe are milestones or essential work. We see these researches as the “dots” of SSL and connect them through how they evolve. Hopefully, by viewing the connections of these dots, readers will have a high-level picture of the development of SSL across multiple disciplines including natural language processing, computer vision, graph learning, audio processing, and protein learning.
Publisher
Springer Science and Business Media LLC
Subject
Computational Theory and Mathematics,Computer Science Applications,Hardware and Architecture,Theoretical Computer Science,Software
Reference123 articles.
1. Liu X, Zhang F, Hou Z, Mian L, Wang Z, Zhang J, Tang J. Self-supervised learning: Generative or contrastive. IEEE Transactions on Knowledge and Data Engineering. https://doi.org/10.1109/TKDE.2021.3090866.
2. Han X, Zhang Z, Ding N et al. Pre-trained models: Past, present and future. AI Open, 2021, 2: 225-250. https://doi.org/10.1016/j.aiopen.2021.08.002.
3. Rogers A, Kovaleva O, Rumshisky A. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 2020, 8: 842-866. https://doi.org/10.1162/tacl_a_00349.
4. Devlin J, Chang M W, Lee K, Toutanova K. BERT: Pretraining of deep bidirectional transformers for language understanding. In Proc. the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, June 2019, pp.4171-4186. https://doi.org/10.18653/v1/N19-1423.
5. Chen M, Radford A, Child R, Wu J, Jun H, Luan D, Sutskever I. Generative pretraining from pixels. In Proc. the 37th International Conference on Machine Learning, July 2020, pp.1691-1703.
Cited by
5 articles.
订阅此论文施引文献
订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献