Named-Entity Recognition for Hindi language using context pattern-based maximum entropy

Abstract

This paper describes Named Entity Recognition (NER) system for Hindi language using two methodologies. An existing BaseLine Maximum Entropy-based Named Entity (BL-MENE) model and Context Pattern-based MENE (CP-MENE) framework the one proposed in this work. BL-MENE utilizes several features for the NER task but suffers from inaccurate Named Entity (NE) boundary detection, mis-classification errors, and partial recognition of NEs due to certain missing essentials. However, CP-MENE based NER task incorporates extensive features and patterns set to overcome these problems. In fact, the CP-MENE features include right-boundary, left-boundary, part-of-speech, synonyms, gazetteers and relative pronoun features. CP-MENE formulates a kind of recursive relationship to extract high ranked NE patterns that are generated through regular expressions via python@ code. Nowadays, since the Web contents in the Hindi language are rising, especially in the health-care applications, this work is conducted on the Hindi Health Data (HHD) corpus at Kaggle dataset. We conducted experiments on four NE categories- Person (PER), Disease (DIS), Consumable (CNS) and Symptom (SMP). Usually, researchers’ work upon PER NE within news articles while other NEs, especially related to the health-care domain such as DIS, CNS, and SMP NE types are left out which are incorporated in this research. CP-MENE improvised the classification performance of NEs and the F-measure achieved are 79.68% for PER, 72.50% for DIS, 68.78% for CNS, and 67.23% for SMP respectively which are comparable with respect to other NER approaches.