Author:
Shafi Jawad,Iqbal Hafiz Rizwan,Nawab Rao Muhammad Adeel,Rayson Paul
Abstract
Abstract
This study describes a Natural Language Processing (NLP) toolkit, as the first contribution of a larger project, for an under-resourced language—Urdu. In previous studies, standard NLP toolkits have been developed for English and many other languages. There is also a dire need for standard text processing tools and methods for Urdu, despite it being widely spoken in different parts of the world with a large amount of digital text being readily available. This study presents the first version of the UNLT (Urdu Natural Language Toolkit) which contains three key text processing tools required for an Urdu NLP pipeline; word tokenizer, sentence tokenizer, and part-of-speech (POS) tagger. The UNLT word tokenizer employs a morpheme matching algorithm coupled with a state-of-the-art stochastic n-gram language model with back-off and smoothing characteristics for the space omission problem. The space insertion problem for compound words is tackled using a dictionary look-up technique. The UNLT sentence tokenizer is a combination of various machine learning, rule-based, regular-expressions, and dictionary look-up techniques. Finally, the UNLT POS taggers are based on Hidden Markov Model and Maximum Entropy-based stochastic techniques. In addition, we have developed large gold standard training and testing data sets to improve and evaluate the performance of new techniques for Urdu word tokenization, sentence tokenization, and POS tagging. For comparison purposes, we have compared the proposed approaches with several methods. Our proposed UNLT, the training and testing data sets, and supporting resources are all free and publicly available for academic use.
Publisher
Cambridge University Press (CUP)
Subject
Artificial Intelligence,Linguistics and Language,Language and Linguistics,Software
Reference80 articles.
1. Investigating GIS and smoothing for maximum entropy taggers
2. Bhat, R.A. and Sharma, D.M. (2012). A dependency treebank of Urdu and its evaluation. In Proceedings of the Sixth Linguistic Annotation Workshop (LAW VI’12), Jeju, Republic of Korea, pp. 157–165.
3. Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates
4. Hardie, A . (2003). Developing a tagset for automated part-of-speech tagging in Urdu. In Archer D., Rayson P., Wilson A. and McEnery T. (eds), Proceedings of the Corpus Linguistics 2003 Conference. UCREL Technical Papers, Lancaster, UK, vol. 16, pp. 298–307.
5. An English POS Tagging Approach Based on Maximum Entropy
Cited by
5 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献