---
title: "The IIT Bombay English-Hindi Parallel — Kunchukuttan 2018"
description: "English-Hindi machine translation has large practical demand, but publicly available parallel data for the pair has been scarce and scattered."
authors: ["Anoop Kunchukuttan", "Pratik Mehta", "Pushpak Bhattacharyya"]
year: 2018
venue: "Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)"
source: "https://aclanthology.org/L18-1548/"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/kunchukuttan-2018-iit-bombay-english-hindi-parallel/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/kunchukuttan-2018-iit-bombay-english-hindi-parallel/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2018](https://md.research.elamur.ai/free/papers/2018/page/3/) / The IIT Bombay English-Hindi Parallel Corpus

# The IIT Bombay English-Hindi Parallel Corpus

**Authors:** [Anoop Kunchukuttan](https://md.research.elamur.ai/free/authors/anoop-kunchukuttan/), Pratik Mehta, [Pushpak Bhattacharyya](https://md.research.elamur.ai/free/authors/pushpak-bhattacharyya/) · **Year:** 2018 · **Venue:** Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/L18-1548/) · [PDF](https://aclanthology.org/L18-1548.pdf) · [OpenAlex](https://openalex.org/W2763176669)

**Citations:** 36 citations (openalex-snapshot · 2026-10-14) · [138](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
English-Hindi machine translation has large practical demand, but publicly available parallel data for the pair has been scarce and scattered. The authors build the IIT Bombay English-Hindi Parallel Corpus. It merges existing public collections (OPUS subsets, HindEnCorp, TED talks and others) with resources produced at CFILT, IIT Bombay that had not been released before:

* judicial translations,
* Mahashabdkosh terminology entries, examples and definitions,
* text from Indian government websites,
* Gyaan-Nidhi material, sentence-aligned automatically with Moore's method.

The result is about 1.49 million training segments, roughly 694k of them new. For dev and test, they reuse the WMT 2014 newswire sets. They also provide standardized preprocessing and benchmark both translation directions with a Moses PBSMT system and a Nematus attention-based NMT system. BLEU scores fall between about 12 and 14.5. PBSMT is clearly stronger for Hindi→English, while NMT is slightly ahead in BLEU for English→Hindi. The authors describe it as the largest English-Hindi parallel corpus openly available at the time. It served as the data for the WAT 2016 and 2017 shared tasks and is released for non-commercial research.

## Key points
- **Size and novelty:** the training set has 1,492,827 segments from 17 sub-corpora, and about 694k of them were not previously public. Dev has 520 segments and test has 2,507, both WMT 2014 newswire.
- **Gyaan-Nidhi:** this is the largest new part, at 227,123 segments. It came from unaligned HTML that was aligned with Moore's (2002) aligner. A manual check of 300 extracted pairs found 88.6% precision.
- **Baseline BLEU and METEOR:**
  - English→Hindi: SMT 11.75 / 0.313, NMT 12.23 / 0.308.
  - Hindi→English: SMT 14.49 / 0.266, NMT 12.83 / 0.219.
- **NMT setup:**
  - Nematus with separate BPE models per language (15,500 merges).
  - One-layer GRU encoder and decoder with 512 units and 256-dimensional embeddings.
  - Adam optimizer, beam size 12, and an ensemble of 4 checkpoints.
- **Test-set OOV rates:** 11.4% of word types for Hindi and 6.7% for English.
- **Licensing:** the new data is CC BY-NC-SA, and the reused parts keep their original licenses.

## Interesting ideas
- **Ordered sub-corpora:** the distribution keeps sub-corpora in a fixed order matching the documentation, so users can split them by domain. This is a cheap way to support domain adaptation or source-specific filtering without shipping separate files.
- **Spot-checking automatic alignment:** the authors aligned comparable data with a language-agnostic aligner, then manually checked a small sample to report a precision figure. This is a practical template for any low-resource pair, Ukrainian included, where quality claims about mined data need some backing.
- **Unicode normalization of composite characters:** for nukta forms, which can be one or two code points, the authors pick one canonical representation before training. Similar normalization of alternative encodings, such as apostrophe variants or combining accents, can prevent silent vocabulary fragmentation in other scripts.
- **Morphology-aware evaluation metric:** they score Hindi output with a customized METEOR that credits stem and WordNet-synonym matches. This idea transfers well to other morphologically rich target languages, where exact n-gram metrics undervalue valid outputs.
- **Separating distribution format from training format:** the corpus is released in original casing, while the experiments use true-casing. Keeping the released data minimally processed lets each user choose their own preprocessing.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

Topics: [Machine translation](https://md.research.elamur.ai/free/topics/machine-translation/)

## Cited by

Cited by

138 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 128 not yet.

- [Findings of the 2020 Conference on Machine Translation (WMT20)](https://md.research.elamur.ai/free/papers/barrault-2020-findings-2020-conference-machine-translation/)
- [Findings of the WMT 2023 Shared Task on Quality Estimation](https://md.research.elamur.ai/free/papers/blain-2023-findings-wmt-2023-shared-task/)
- [An Open Dataset and Model for Language Identification](https://md.research.elamur.ai/free/papers/burchell-2023-open-dataset-model-language-identification/)
- [InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training](https://md.research.elamur.ai/free/papers/chi-2020-infoxlm-information-theoretic-framework-cross/)
- [IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages](https://md.research.elamur.ai/free/papers/gala-2023-indictrans2-towards-high-quality-accessible/)
- [GlotLID: Language Identification for Low-Resource Languages](https://md.research.elamur.ai/free/papers/kargaran-2023-glotlid-language-identification-low-resource/)
- [Non Idiomatic Conventionalised Expressions: A New Pain in the Neck?](https://md.research.elamur.ai/free/papers/katrapati-2025-non-idiomatic-conventionalised-expressions-new/)
- [Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages](https://md.research.elamur.ai/free/papers/ramesh-2022-samanantar-largest-publicly-available-parallel/)
- [Multilingual Translation from Denoising Pre-Training](https://md.research.elamur.ai/free/papers/tang-2021-multilingual-translation-denoising-pre-training/)
- [Findings of the WMT 2022 Shared Task on Quality Estimation](https://md.research.elamur.ai/free/papers/zerva-2022-findings-wmt-2022-shared-task/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{kunchukuttan2018iit,
  title = {{The IIT Bombay English-Hindi Parallel Corpus}},
  author = {Kunchukuttan, Anoop and Mehta, Pratik and Bhattacharyya, Pushpak},
  booktitle = {Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)},
  year = {2018},
  doi = {10.63317/2erd8c5iqpzz},
  url = {https://doi.org/10.63317/2erd8c5iqpzz}
}
```

- [← previous The Hebrew Universal Dependency Treebank: Past, Present and Future](https://md.research.elamur.ai/free/papers/ud-2018-sade-hebrew-universal-dependency-treebank-past/)
- [next → The NarrativeQA Reading Comprehension Challenge](https://md.research.elamur.ai/free/papers/kocisky-2018-narrativeqa-reading-comprehension-challenge/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/papers/2018/page/3/)

## References

22 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abdelali, A., Guzman, F., Sajjad, H., and Vogel, S. (2014). The AMARA Corpus: Building Parallel Language Resources for the Educational Domain. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14). [find](https://scholar.google.com/scholar?q=Abdelali%2C+A%2E%2C+Guzman%2C+F%2E%2C+Sajjad%2C+H%2E%2C+and+Vogel%2C+S%2E+%282014%29%2E+The+AMARA+Corpus%3A+Building+Parallel+Language+Resources+for+the+Educational+Domain%2E+In+Proceedings+of+the+Ninth+International+Conference+on+L) [our card](https://md.research.elamur.ai/free/papers/abdelali-2014-amara-corpus-building-parallel-language/)
2. Alexandra Birch, Chris Callison-Burch, M. O. and Post, M. (2011). The Indic multi-parallel corpus. [http://homepages.inf.ed.ac.uk/miles/babel.html](http://homepages.inf.ed.ac.uk/miles/babel.html). [↗](http://homepages.inf.ed.ac.uk/miles/babel.html)
3. Bahdanau, D., Cho, K., and Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate. International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Bahdanau%2C+D%2E%2C+Cho%2C+K%2E%2C+and+Bengio%2C+Y%2E+%282015%29%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+International+Conference+on+Learning+Representations%2E) [our card](https://md.research.elamur.ai/free/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
4. Banerjee, S. and Lavie, A. (2005). METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. [find](https://scholar.google.com/scholar?q=Banerjee%2C+S%2E+and+Lavie%2C+A%2E+%282005%29%2E+METEOR%3A+An+automatic+metric+for+MT+evaluation+with+improved+correlation+with+human+judgments%2E+In+Proceedings+of+the+ACL+workshop+on+intrinsic+and+extrinsic+evaluatio) [our card](https://md.research.elamur.ai/free/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/)
5. Bhattacharyya, P., Bahuguna, A., Talukdar, L., and Phukan, B. (2014). Facilitating multi-lingual sense annotation: Human mediated lemmatizer. In Global WordNet Conference. [find](https://scholar.google.com/scholar?q=Bhattacharyya%2C+P%2E%2C+Bahuguna%2C+A%2E%2C+Talukdar%2C+L%2E%2C+and+Phukan%2C+B%2E+%282014%29%2E+Facilitating+multi%2Dlingual+sense+annotation%3A+Human+mediated+lemmatizer%2E+In+Global+WordNet+Conference%2E)
6. Bhattacharyya, P. (2010). IndoWordNet. In Language Resources and Evaluation Conference. [find](https://scholar.google.com/scholar?q=Bhattacharyya%2C+P%2E+%282010%29%2E+IndoWordNet%2E+In+Language+Resources+and+Evaluation+Conference%2E)
7. Bojar, O., Buck, C., Federmann, C., Haddow, B., Koehn, P., Leveling, J., Monz, C., Pecina, P., Post, M., Saint-Amand, H., et al. (2014a). Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Bojar%2C+O%2E%2C+Buck%2C+C%2E%2C+Federmann%2C+C%2E%2C+Haddow%2C+B%2E%2C+Koehn%2C+P%2E%2C+Leveling%2C+J%2E%2C+Monz%2C+C%2E%2C+Pecina%2C+P%2E%2C+Post%2C+M%2E%2C+Saint%2DAmand%2C+H%2E%2C+et+al%2E+%282014a%29%2E+Findings+of+the+2014+workshop+on+statistical+machine+translati) [our card](https://md.research.elamur.ai/free/papers/bojar-2014-findings-2014-workshop-statistical-machine/)
8. Bojar, O., Diatka, V., Rychlỳ, P., Stranák, P., Suchomel, V., Tamchyna, A., and Zeman, D. (2014b). HindEnCorp-Hindi-English and Hindi-only Corpus for Machine Translation. In Language Resources and Evaluation Conference. [find](https://scholar.google.com/scholar?q=Bojar%2C+O%2E%2C+Diatka%2C+V%2E%2C+Rychly%CC%80%2C+P%2E%2C+Strana%CC%81k%2C+P%2E%2C+Suchomel%2C+V%2E%2C+Tamchyna%2C+A%2E%2C+and+Zeman%2C+D%2E+%282014b%29%2E+HindEnCorp%2DHindi%2DEnglish+and+Hindi%2Donly+Corpus+for+Machine+Translation%2E+In+Language+Resources+and+E)
9. Cherry, C. and Foster, G. (2012). Batch tuning strategies for Statistical Machine Translation. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. [find](https://scholar.google.com/scholar?q=Cherry%2C+C%2E+and+Foster%2C+G%2E+%282012%29%2E+Batch+tuning+strategies+for+Statistical+Machine+Translation%2E+In+Proceedings+of+the+2012+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+)
10. Heafield, K. (2011). KenLM: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Heafield%2C+K%2E+%282011%29%2E+KenLM%3A+Faster+and+smaller+language+model+queries%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2E) [our card](https://md.research.elamur.ai/free/papers/heafield-2011-kenlm-faster-smaller-language-model/)
11. Kingma, D. and Ba, J. (2014). Adam: A method for stochastic optimization. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Kingma%2C+D%2E+and+Ba%2C+J%2E+%282014%29%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+International+Conference+on+Learning+Representations%2E)
12. Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., et al. (2007). Moses: Open source toolkit for Statistical Machine Translation. In Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessi… [find](https://scholar.google.com/scholar?q=Koehn%2C+P%2E%2C+Hoang%2C+H%2E%2C+Birch%2C+A%2E%2C+Callison%2DBurch%2C+C%2E%2C+Federico%2C+M%2E%2C+Bertoldi%2C+N%2E%2C+Cowan%2C+B%2E%2C+Shen%2C+W%2E%2C+Moran%2C+C%2E%2C+Zens%2C+R%2E%2C+et+al%2E+%282007%29%2E+Moses%3A+Open+source+toolkit+for+Statistical+Machine+Translation)
13. Kunchukuttan, A., Roy, S., Patel, P., Ladha, K., Gupta, S., Khapra, M. M., and Bhattacharyya, P. (2012). Experiences in Resource Generation for Machine Translation through Crowdsourcing. In Language Resources and Evaluation Conference. [find](https://scholar.google.com/scholar?q=Kunchukuttan%2C+A%2E%2C+Roy%2C+S%2E%2C+Patel%2C+P%2E%2C+Ladha%2C+K%2E%2C+Gupta%2C+S%2E%2C+Khapra%2C+M%2E+M%2E%2C+and+Bhattacharyya%2C+P%2E+%282012%29%2E+Experiences+in+Resource+Generation+for+Machine+Translation+through+Crowdsourcing%2E+In+Language+R)
14. Kunchukuttan, A., Chatterjee, R., Roy, S., Mishra, A., and Bhattacharyya, P. (2013). TransDoop: A Map-Reduce based Crowdsourced Translation for Complex Domain. In Conference of the Association of Computational Linguistics (System Demonstrations). [find](https://scholar.google.com/scholar?q=Kunchukuttan%2C+A%2E%2C+Chatterjee%2C+R%2E%2C+Roy%2C+S%2E%2C+Mishra%2C+A%2E%2C+and+Bhattacharyya%2C+P%2E+%282013%29%2E+TransDoop%3A+A+Map%2DReduce+based+Crowdsourced+Translation+for+Complex+Domain%2E+In+Conference+of+the+Association+of+Comp)
15. Moore, R. (2002). Fast and accurate sentence alignment of bilingual corpora. Machine Translation: From Research to Real Users. [find](https://scholar.google.com/scholar?q=Moore%2C+R%2E+%282002%29%2E+Fast+and+accurate+sentence+alignment+of+bilingual+corpora%2E+Machine+Translation%3A+From+Research+to+Real+Users%2E)
16. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). BLEU: A method for automatic evaluation of machine translation. In Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Papineni%2C+K%2E%2C+Roukos%2C+S%2E%2C+Ward%2C+T%2E%2C+and+Zhu%2C+W%2E%2DJ%2E+%282002%29%2E+BLEU%3A+A+method+for+automatic+evaluation+of+machine+translation%2E+In+Association+for+Computational+Linguistics%2E) [our card](https://md.research.elamur.ai/free/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
17. Ramanathan, A., Hegde, J., Shah, R., Bhattacharyya, P., and Sasikumar, M. (2008). Simple Syntactic and Morphological Processing Can Help English-Hindi Statistical Machine Translation. In International Joint Conference on Natural Language Processing. [find](https://scholar.google.com/scholar?q=Ramanathan%2C+A%2E%2C+Hegde%2C+J%2E%2C+Shah%2C+R%2E%2C+Bhattacharyya%2C+P%2E%2C+and+Sasikumar%2C+M%2E+%282008%29%2E+Simple+Syntactic+and+Morphological+Processing+Can+Help+English%2DHindi+Statistical+Machine+Translation%2E+In+International)
18. Sennrich, R., Haddow, B., and Birch, A. (2016a). Edinburgh Neural Machine Translation Systems for WMT 16. In Workshop on Machine Translation (Shared Task). [find](https://scholar.google.com/scholar?q=Sennrich%2C+R%2E%2C+Haddow%2C+B%2E%2C+and+Birch%2C+A%2E+%282016a%29%2E+Edinburgh+Neural+Machine+Translation+Systems+for+WMT+16%2E+In+Workshop+on+Machine+Translation+%28Shared+Task%29%2E) [our card](https://md.research.elamur.ai/free/papers/sennrich-2016-edinburgh-neural-machine-translation-systems/)
19. Sennrich, R., Haddow, B., and Birch, A. (2016b). Neural machine translation of rare words with subword units. In Conference of the Association of Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sennrich%2C+R%2E%2C+Haddow%2C+B%2E%2C+and+Birch%2C+A%2E+%282016b%29%2E+Neural+machine+translation+of+rare+words+with+subword+units%2E+In+Conference+of+the+Association+of+Computational+Linguistics%2E)
20. Sennrich, R., Firat, O., Cho, K., Birch, A., Haddow, B., Hitschler, J., Junczys-Dowmunt, M., Läubli, S., Miceli Barone, A. V., Mokry, J., and Nadejde, M. (2017). Nematus: a Toolkit for Neural Machine Translation. In Proceedings of the Software Demonstrations of the 15th Conference of the European C… [find](https://scholar.google.com/scholar?q=Sennrich%2C+R%2E%2C+Firat%2C+O%2E%2C+Cho%2C+K%2E%2C+Birch%2C+A%2E%2C+Haddow%2C+B%2E%2C+Hitschler%2C+J%2E%2C+Junczys%2DDowmunt%2C+M%2E%2C+La%CC%88ubli%2C+S%2E%2C+Miceli+Barone%2C+A%2E+V%2E%2C+Mokry%2C+J%2E%2C+and+Nadejde%2C+M%2E+%282017%29%2E+Nematus%3A+a+Toolkit+for+Neural+Machine) [our card](https://md.research.elamur.ai/free/papers/sennrich-2017-nematus-toolkit-neural-machine-translation/)
21. Tiedemann, J. (2012). Parallel Data, Tools and Interfaces in OPUS. In Language Resources and Evaluation Conference. [find](https://scholar.google.com/scholar?q=Tiedemann%2C+J%2E+%282012%29%2E+Parallel+Data%2C+Tools+and+Interfaces+in+OPUS%2E+In+Language+Resources+and+Evaluation+Conference%2E)
22. Wikipedia. (2017). Hindi. [https://en.wikipedia.org/w/index.php?title=Hindi&oldid=](https://en.wikipedia.org/w/index.php?title=Hindi&oldid=) 802224343. [Online; accessed 02-October-2017]. [↗](https://en.wikipedia.org/w/index.php?title=Hindi&oldid=)