KuBERT: Central Kurdish BERT Model and Its Application for Sentiment Analysis

Abstract

This paper enhances the study of sentiment analysis for the Central Kurdish language by integrating the Bidirectional Encoder Representations from Transformers into Natural Language Processing techniques. Kurdish is also a low-resourced language, having a high level of linguistic diversity with minimal computational resources, making sentiment analysis somewhat challenging. Earlier, this was done using a traditional word embedding model, such as Word2Vec, but with the emergence of new language models, specifically BERT, there is hope for improvements. The better word embedding capabilities of BERT lend to this study, aiding in the capturing of the nuanced semantic pool and the contextual intricacies of the language under study, the Kurdish language, thus setting a new benchmark for sentiment analysis in low-resource languages. The steps include collecting and normalizing a large corpus of Kurdish texts, pretraining BERT with a special tokenizer for Kurdish, and developing different models for sentiment analysis: LSTM, MLP, and finetuning. The proposed approach consists of 3 classes: positive, negative, and neutral sentiment analysis using a sentiment embedding of BERT in four different configurations. The accuracy of the best-performing classifier, LSTM, is 74.09%. For the BERT with an MLP classifier model, the maximum accuracy achieved is 73.96%, while the fine-tuned BERT model tops the others with 75.37% accuracy. Additionally, the fine-tuned BERT model demonstrates a vast improvement when focused on two 2-class sentiment analyses—positive and negative—with an accuracy of 86.31%. The study makes a comprehensive comparison, highlighting BERT's superiority over the traditional ones based on accuracy and semantic understanding. It is motivated because several results are obtained that the proposed BERT-based models outperform Word2Vec models conventionally used here by a remarkable accuracy gain in most sentiment analysis tasks. This might be an advancement, especially for those under-resourced languages in the field of NLP. It only indicates the potential of LLM in improving sentiment analysis. It emphasizes the need for developing language-specific models for datasets to solve the problems brought up by low-resource languages. This study fills this gap in sentiment analysis capabilities for Kurdish and sheds light on broader applicability in extremely linguistically diverse and resource-constrained contexts for BERT and similar models.