Affiliation:
1. Department of Computer Sciences, University of Kashmir, Srinagar 190006, India
2. Center of Computational Science and Mathematical Modelling, Coventry University, Coventry CV1 5FB, UK
Abstract
This paper introduces a parameter-efficient transformer-based model designed for scientific literature classification. By optimizing the transformer architecture, the proposed model significantly reduces memory usage, training time, inference time, and the carbon footprint associated with large language models. The proposed approach is evaluated against various deep learning models and demonstrates superior performance in classifying scientific literature. Comprehensive experiments conducted on datasets from Web of Science, ArXiv, Nature, Springer, and Wiley reveal that the proposed model’s multi-headed attention mechanism and enhanced embeddings contribute to its high accuracy and efficiency, making it a robust solution for text classification tasks.
Reference48 articles.
1. Ware, M., and Mabe, M. (2015). The STM Report: An Overview of Scientific and Scholarly Journal Publishing, International Association of Scientific, Technical and Medical Publishers. Technical Report 4.
2. Article 50 Million: An Estimate of the Number of Scholarly Articles in Existence;Jinha;Learn. Publ.,2010
3. National Center for Education Statistics (2024, June 15). Doctor’s Degrees Conferred by Postsecondary Institutions, by Field of Study: Selected Years, 1970-71 through 2018-19. Digest of Education Statistics, 2019, Available online: https://nces.ed.gov/programs/digest/d21/tables/dt21_324.10.asp.
4. Kim, Y. (2014, January 25–29). Convolutional Neural Networks for Sentence Classification. Proceedings of the EMNLP, Doha, Qatar.
5. Georgakopoulos, S.V., Tasoulis, S.K., Vrahatis, A.G., and Plagianakos, V.P. (2018, January 9–12). Convolutional Neural Networks for Toxic Comment Classification. Proceedings of the 10th Hellenic Conference on Artificial Intelligence, Patras, Greece.