Comparison of Commercial Decoder-only Large Language Models for Multilingual Sentiment Analysis of Short Text

Abstract

This paper focuses the multilingual sentiment analysis of short text using three popular, commercial decoder-only Large Language Models (“LLMs”), OpenAI’s ChatGPT, Anthropic’s Claude, and Google’s Gemini. The training data for some of these models is approximately 90% English, and it is an open question about whether it is better to evaluate text data in the original language or to translate the data into English and then evaluate the text. To study this question, we leverage previous research into sentiment analysis of multilingual short text data in which 1000 short text samples in seven languages (English, Spanish, French, Portuguese, Arabic, Japanese, and Korean) were translated into English using Google Translate. We processed these data samples with the three decoder-only LLMs and compared them to results of other methods (encoder-only LLMs, RNNs, Lexicons). We found that these decoder-only LLMs obtained the highest accuracy out of all sentiment analysis methods when evaluated on the original language. The only outlier was with the French data where an RNN created from French data was the most accurate. Between the three decoder-only LLMs, ChatGPT had the highest accuracy for four out of seven languages, and Claude had two out of seven. Gemini had zero most accurate but had six out of seven as the second most accurate.