Automatic Genre Identification for Robust Enrichment of Massive Text Collections: Investigation of Classification Methods in the Era of Large Language Models
-
Published:2023-09-12
Issue:3
Volume:5
Page:1149-1175
-
ISSN:2504-4990
-
Container-title:Machine Learning and Knowledge Extraction
-
language:en
-
Short-container-title:MAKE
Author:
Kuzman Taja12ORCID, Mozetič Igor1ORCID, Ljubešić Nikola13ORCID
Affiliation:
1. Department of Knowledge Technologies, Jožef Stefan Institute, 1000 Ljubljana, Slovenia 2. Jožef Stefan International Postgraduate School, 1000 Ljubljana, Slovenia 3. Center za Jezikovne Vire in Tehnologije Univerze v Ljubljani, 1000 Ljubljana, Slovenia
Abstract
Massive text collections are the backbone of large language models, the main ingredient of the current significant progress in artificial intelligence. However, as these collections are mostly collected using automatic methods, researchers have few insights into what types of texts they consist of. Automatic genre identification is a text classification task that enriches texts with genre labels, such as promotional and legal, providing meaningful insights into the composition of these large text collections. In this paper, we evaluate machine learning approaches for the genre identification task based on their generalizability across different datasets to assess which model is the most suitable for the downstream task of enriching large web corpora with genre information. We train and test multiple fine-tuned BERT-like Transformer-based models and show that merging different genre-annotated datasets yields superior results. Moreover, we explore the zero-shot capabilities of large GPT Transformer models in this task and discuss the advantages and disadvantages of the zero-shot approach. We also publish the best-performing fine-tuned model that enables automatic genre annotation in multiple languages. In addition, to promote further research in this area, we plan to share, upon request, a new benchmark for automatic genre annotation, ensuring the non-exposure of the latest large language models.
Subject
Artificial Intelligence,Engineering (miscellaneous)
Reference84 articles.
1. Zu Eissen, S.M., and Stein, B. (2004). Proceedings of the 27th Annual German Conference in AI, KI 2004, Ulm, Germany, 20–24 September 2004, Springer. 2. Vidulin, V., Luštrek, M., and Gams, M. (2007, January 30). Using genres to improve search engines. Proceedings of the 1st International Workshop: Towards Genre-Enabled Search Engines: The Impact of Natural Language Processing, Borovets, Bulgaria. 3. Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Cappelli, A., Alobeidli, H., Pannier, B., Almazrouei, E., and Launay, J. (2023). The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only. arXiv. 4. Kuzman, T., Rupnik, P., and Ljubešić, N. (2023, January 5–6). Get to Know Your Parallel Data: Performing English Variety and Genre Classification over MaCoCu Corpora. Proceedings of the Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023), Dubrovnik, Croatia. 5. Genre repertoire: The structuring of communicative practices in organizations;Orlikowski;Adm. Sci. Q.,1994
|
|