Automatic Resource Augmentation for Machine Translation in Low Resource Language: <tt>EnIndic Corpus</tt>-Reference-Cited by-同舟云学术

Automatic Resource Augmentation for Machine Translation in Low Resource Language: EnIndic Corpus

Published:2023-08-31 Issue: Volume: Page:
ISSN:2375-4699
Container-title:ACM Transactions on Asian and Low-Resource Language Information Processing
language:en
Short-container-title:ACM Trans. Asian Low-Resour. Lang. Inf. Process.

Author:

Banerjee Anasua¹,Kumar Vinay¹,Shankar Achyut²,Jhaveri Rutvij H.³,Banik Debajyoty⁴

Affiliation:

1. National Institute of Technology, Jamshedpur, India

2. WMG, University of Warwick, United Kingdom

3. Pandit Deendayal Energy University, India

4. School of Computer Science and Artificial Intelligence, SR University, India

Abstract

Parallel corpus is the primary ingredient of machine translation. It is required to train the statistical machine translation (SMT) and neural machine translation (NMT) systems. There is a lack of good quality parallel corpus for Hindi to English. Comparable corpora for a given language pair are comparatively easy to find, but this cannot be used directly in SMT or NMT systems. As a result, we generate a parallel corpus from the comparable corpus. For this purpose, the sentences (which are translations of each other) are mined from the comparable corpus to prepare the parallel corpus. The proposed algorithm uses the length of the sentence and word translation model to align sentence pairs that are translations of each other. Then, the sentence pairs that are poor translations of each other (measured by a similarity score based on IBM model 1 translation probability) are filtered out. We apply this algorithm to comparable corpora, which are crawled from speeches of the President and Vice-President of India, and mined parallel corpora out of them. The prepared parallel corpus contains good quality aligned sentences (with 96.338% f-score). Subsequently, incorrect sentence pairs are filtered out manually to make the corpus in qualitative practical use. Finally, we gather various sentences from different sources to prepare the EnIndic corpus, which comprises 1,656,207 English-Hindi sentence pairs (miscellaneous domain). We have deployed this prepared largest English-Hindi parallel corpus at https://github.com/debajyoty/EnIndic.git and the source code at https://github.com/debajyoty/EnIndicSourceCode.git.

Publisher

Association for Computing Machinery (ACM)

Subject

General Computer Science

Link

https://dl.acm.org/doi/pdf/10.1145/3617371

Reference45 articles.

1. Ahmed Abbasi , Abdul Rehman Javed , Farkhund Iqbal , Zunera Jalil , Thippa Reddy Gadekallu , and Natalia Kryvinska . 2022. Authorship identification using ensemble learning. Scientific reports 12, 1 ( 2022 ), 9537. Ahmed Abbasi, Abdul Rehman Javed, Farkhund Iqbal, Zunera Jalil, Thippa Reddy Gadekallu, and Natalia Kryvinska. 2022. Authorship identification using ensemble learning. Scientific reports 12, 1 (2022), 9537.

2. Deep learning for religious and continent-based toxic content detection and classification

3. Minale A Abebe , Joe Tekli , Fekade Getahun , Richard Chbeir , and Gilbert Tekli . 2017 . Overview of event-based collective knowledge management in multimedia digital ecosystems . In 2017 13th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS). IEEE, 40–49 . Minale A Abebe, Joe Tekli, Fekade Getahun, Richard Chbeir, and Gilbert Tekli. 2017. Overview of event-based collective knowledge management in multimedia digital ecosystems. In 2017 13th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS). IEEE, 40–49.

4. Usman Ahmed , Rutvij H Jhaveri , Gautam Srivastava , and Jerry Chun-Wei Lin . 2022. Explainable Deep Attention Active Learning for Sentimental Analytics of Mental Disorder. Transactions on Asian and Low-Resource Language Information Processing ( 2022 ). Usman Ahmed, Rutvij H Jhaveri, Gautam Srivastava, and Jerry Chun-Wei Lin. 2022. Explainable Deep Attention Active Learning for Sentimental Analytics of Mental Disorder. Transactions on Asian and Low-Resource Language Information Processing (2022).

5. Muhammad Waseem Akram , Muhammad Salman , Muhammad Farrukh Bashir , Syed Muhammad Saad Salman , Thippa Reddy Gadekallu, and Abdul Rehman Javed. 2022 . A Novel Deep Auto-Encoder Based Linguistics Clustering Model for Social Text. Transactions on Asian and Low-Resource Language Information Processing ( 2022). Muhammad Waseem Akram, Muhammad Salman, Muhammad Farrukh Bashir, Syed Muhammad Saad Salman, Thippa Reddy Gadekallu, and Abdul Rehman Javed. 2022. A Novel Deep Auto-Encoder Based Linguistics Clustering Model for Social Text. Transactions on Asian and Low-Resource Language Information Processing (2022).