Detection of Language from Roman Urdu and English Multilingual Corpus

Abstract

PURPOSE: This study aims to suggest and validate a model to identify the languages from Roman Urdu and English mixed multilingual corpus collected from social media sites.  BACKGROUND: The problem of identifying languages from a corpus of written texts that includes two or more languages is known as language identification or detection. Identifying or detecting the language present in social media text is a requirement and it has numerous applications in natural language processing and computational linguistics, like for word embedding generation, emotion analysis and part of speech tagging etc.  METHODOLOGY: The dictionary-based baseline with SVM and Bi-Directional LSTM has been used in language identification from collected Roman Urdu and English multilingual Corpus. This research work will help in identify the languages from Roman Urdu and English Corpus. The English and Roman Urdu corpus had been obtained from different social media websites and cross-media platforms such as Facebook, Twitter, Google+, Instagram, WhatsApp, and Messenger, etc. The dictionary-based baseline with SVM and Bi-Directional LSTM has been used in language identification from collected Roman Urdu and English multilingual Corpus.  RESULTS: Based on the results achieved using the methodology in the research work the Bi-directional LSTM model performed better with an accuracy of 97.98%. CONCLUSION: The problem in recognizing or detecting the language present in a given document or statement is referred to as language recognition or detection The Corpus of English and Roman Urdu is collected from social media websites. The text for training is submitted to a bi-direction LSTM accordingly to verify if the text is in English language or Urdu language. The results of word recognition for bidirectional word-level LSTM from Roman Urdu and English showed improved results.