Combining a Novel Scoring Approach with Arabic Stemming Techniques for Arabic Chatbots Conversation Engine

Abstract

Arabic is recognized as one of the main languages around the world. Many attempts and efforts have been done to provide computing solutions to support the language. Developing Arabic chatbots is still an evolving research field and requires extra efforts due to the nature of the language. One of the common tasks of any natural language processing application is the stemming step. It is important for developing chatbots, since it helps with pre-processing the input data and it can be involved with different phases of the chatbot development process. The aim of this article is to combine a scoring approach with Arabic stemming techniques for developing an Arabic chatbot conversation engine. Two experiments are conducted to evaluate the proposed solution. The first experiment is to select which stemmer is more accurate when applying our solution, since our algorithm can support various stemmers. The second experiment was conducted to evaluate our proposed approach against various machine learning models. The results show that the ISRIS stemming algorithm is the best fit for our solution with accuracy 78.06%. The results also indicate that our novel solution achieved an F1 score of 65.5%, while the other machine learning models achieved slightly lower scores. Our study presents a novel technique by combining scoring mechanisms with stemming processes to produce the best answer for every query sent by chatbots users compared to other approaches. This can be helpful for developing Arabic chatbot and can support many domains such as education, business, and health. This technique is among the first techniques that developed purposefully to serve the development of Arabic chatbots conversation engine.