A study of Thai word separation methods based on machine learning

Abstract

Abstract Thai is an analytic, isolated language where monosyllabic words occupy most of the basic vocabulary, and differences in vocabulary and grammar are distinguished by differences in tone. Traditional dictionary-based matching methods have difficulty constructing a complete dictionary. We use four machine learning algorithms, namely, a plain Bayesian classifier, a decision tree classifier, a support vector machine classifier, and a conditional random field, to slice Thai language texts for multiclass problems. In addition, we use the above four machine learning algorithms for a two-class problem slicing of Thai texts. In this study, we found that first, the accuracy, recall and F1 values of Thai text slicing using the conditional random field algorithm in the multiclass classification problem are much higher than those of the plain Bayesian classifier, decision tree classifier and support vector machine classifier. Second, the accuracy, recall, and F1 values of the plain Bayesian algorithm, J48 decision tree algorithm, and support vector machine algorithm are higher than those of the multiclass problem classification in both classes. The accuracy of the conditional random field was improved, but its recall and F1 values decreased significantly. In summary, it can be seen that the conditional random field has better results in Thai word separation.