End-to-End Transformer-Based Models in Textual-Based NLP-Reference-Cited by-同舟云学术

End-to-End Transformer-Based Models in Textual-Based NLP

Published:2023-01-05 Issue:1 Volume:4 Page:54-110
ISSN:2673-2688
Container-title:AI
language:en
Short-container-title:AI

Author:

Rahali Abir^ORCID,Akhloufi Moulay A.^ORCID

Abstract

Transformer architectures are highly expressive because they use self-attention mechanisms to encode long-range dependencies in the input sequences. In this paper, we present a literature review on Transformer-based (TB) models, providing a detailed overview of each model in comparison to the Transformer’s standard architecture. This survey focuses on TB models used in the field of Natural Language Processing (NLP) for textual-based tasks. We begin with an overview of the fundamental concepts at the heart of the success of these models. Then, we classify them based on their architecture and training mode. We compare the advantages and disadvantages of popular techniques in terms of architectural design and experimental value. Finally, we discuss open research, directions, and potential future work to help solve current TB application challenges in NLP.

Funder

Natural Sciences and Engineering Research Council of Canada

Publisher

MDPI AG

Subject

Industrial and Manufacturing Engineering

Link

https://www.mdpi.com/2673-2688/4/1/4/pdf

Reference412 articles.

1. Mitkov, R. (2022). The Oxford Handbook of Computational Linguistics, Oxford University Press.

2. Wilie, B., Vincentio, K., Winata, G.I., Cahyawijaya, S., Li, X., Lim, Z.Y., Soleman, S., Mahendra, R., Fung, P., and Bahar, S. (2020). Indonlu: Benchmark and resources for evaluating indonesian natural language understanding. arXiv.

3. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.

4. Dumitrescu, S.D., Avram, A.M., and Pyysalo, S. (2020). The birth of Romanian BERT. arXiv.

5. BioBERT: A pre-trained biomedical language representation model for biomedical text mining;Lee;Bioinformatics,2020

Cited by 13 articles. 订阅此论文施引文献订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献

1. Modern Applications With a Focus on Training ChatGPT and GPT Models;Advanced Applications of Generative AI and Natural Language Processing Models;2023-12-21

2. Leveraging Large Language Models for Literature Review Tasks - A Case Study Using ChatGPT;Communications in Computer and Information Science;2023-12-20

3. BERT-enhanced sentiment analysis for personalized e-commerce recommendations;Multimedia Tools and Applications;2023-12-12

4. User Multi-preferences Fusion for Conversational Recommender Systems;Communications in Computer and Information Science;2023-11-26

5. Toward intelligent open-ended questions evaluation based on predictive optimization;Expert Systems with Applications;2023-11