DZ-SMS: An Authentic Corpus of Algerian SMS

Abstract

In this article, a complete methodology of a corpus realization of authentic Short Message Service (SMS) from Algerian dialect and which are transcribed in Latin characters or symbols is presented. A linguistic material constituted by 6,000 SMS coming from the different geographical regions of Algeria (Middle, East, and West) corresponding to 42 administrative and geographical departments, have been collected. The coexistence of several dialects through these three regions simultaneously has obliged us to consider and operate a classification of the data for each dialect. This data classification has yielded three extracted regional dialectic corpora, each of them covering a specific number of administrative departments. These treatments are based on the so-called Data-n-gram tokenization targeting the suppression of the stop words, the stemming and the imbalance of the classes linked to the nature of the SMS. Consequently, three text classifiers based on three linear classifiers, namely, Stochastic Gradient Descent (SGD), The Ridge Regression (RDG), and Linear Support Vector Machines, to find out the number of significant corpora to extract from the collected data. A deep analysis of the results has shown that the 5-grams data representation is more representative whereas the stop-words removal and stemming process has generated an information loss that has subsequently inferred an alteration of the recognition rate of about 2%. The emerging problem of classes imbalance has been treated by using three techniques: Random Oversampling, Synthetic Minorities Oversampling Technique (SMOTE), and Adaptive Synthetic (ADASYN). This treatment produced interesting results and enhancements; particularly, the classification by region with the oversampling process SMOTE by using the RDG technique has reached a better percentage of 55.93% whereas the classification by department with the oversampling process ADASYN associated with the SGD has only yielded a maximum score of about 17.11%. The results, which undoubtedly are in favor of the classification by region, have compelled us to create three Subdialectal regional corpora, each, covering a certain number of Algerian departments.