Stance detection in Arabic with a multi-dialectal cross-domain stance corpus

Abstract

AbstractWe present a cross-domain and multi-dialectal stance corpus for Arabic, covering the major dialect groups and four Arab regions. This research provides an important language resource for automating the task of stance detection in Dialectal Arabic while carefully considering the subtle differences in stance expression across various dialects. More than 4500 sentences in our corpus have been carefully annotated according to their stance with regard to a certain subject. We gathered sentences associated with two controversial topics for every region and we had at least two annotators annotate each sentence to indicate if the author is supporting, opposing, or neutral to the sentence’s topic. Our corpus shows high balance between dialect and stance. About half of the sentences in each region are written in Modern Standard Arabic, while the other half are written in the specific dialect of that region. To evaluate our corpus, we performed a number of machine-learning experiments for the stance detection task. The best performance was achieved by AraBERT with an accuracy and an F1-score of 0.82. Furthermore, we trained and tested this model on the most similar state-of-the-art stance dataset, “MAWQIF”. The comparison results demonstrate how crucial it is to maintain balance among the three stance classes in our dataset. In particular, the model scored better when using our stance corpus than when using the MAWQIF dataset especially for the “Neutral” stance class. Using our best performing model, we developed a Web-based demonstrator for stance detection in dialectal Arabic and we show its effectiveness in analyzing stance in the context of two real-world scenarios: product boycott in the Arab world and customer reviews of a soft drink company.