---
title: "IndicBART — paper by Dabre et al. (2022)"
description: "The authors instead build IndicBART, a compact mBART-style sequence-to-sequence model for 11 Indic languages plus… 43 references, cited by 63 papers here."
authors: ["Raj Dabre", "Himani Shrotriya", "Anoop Kunchukuttan", "Ratish Puduppully", "Mitesh Khapra", "Pratyush Kumar"]
year: 2022
venue: "Findings of the Association for Computational Linguistics: ACL 2022"
source: "https://aclanthology.org/2022.findings-acl.145/"
doi: "10.18653/v1/2022.findings-acl.145"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/dabre-2022-indicbart-pre-trained-model-indic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/dabre-2022-indicbart-pre-trained-model-indic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/) / IndicBART: A Pre-trained Model for Indic Natural Language Generation

# IndicBART: A Pre-trained Model for Indic Natural Language Generation

**Authors:** [Raj Dabre](https://md.research.elamur.ai/free/en/authors/raj-dabre/), Himani Shrotriya, [Anoop Kunchukuttan](https://md.research.elamur.ai/free/en/authors/anoop-kunchukuttan/), [Ratish Puduppully](https://md.research.elamur.ai/free/en/authors/ratish-puduppully/), [Mitesh Khapra](https://md.research.elamur.ai/free/en/authors/mitesh-m-khapra/) and [Pratyush Kumar](https://md.research.elamur.ai/free/en/authors/pratyush-kumar/) · **Year:** 2022 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2022](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2022](https://md.research.elamur.ai/free/en/venues/findings/2022/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.findings-acl.145/) · [PDF](https://aclanthology.org/2022.findings-acl.145.pdf) · [DOI 10.18653/v1/2022.findings-acl.145](https://doi.org/10.18653/v1/2022.findings-acl.145) · [OpenAlex](https://openalex.org/W4285255685)

**Citations:** 78 citations (OpenAlex, 2026-10-14) · [63](https://md.research.elamur.ai/free/en/papers/dabre-2022-indicbart-pre-trained-model-indic/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Large multilingual generation models such as mBART50 and mT5 cover Indic languages poorly. Their training data is dominated by high-resource languages, their vocabularies give small languages little room, and they are costly to fine-tune and run. The authors instead build IndicBART, a compact mBART-style sequence-to-sequence model for 11 Indic languages plus English. It was pre-trained on IndicCorp with masked span reconstruction. Because most Indic scripts descend from Brahmi and share a largely common logical character set, all Indic text is transliterated into Devanagari. This lets languages share far more subwords within a 64K vocabulary.

On machine translation and extreme summarization (XL-Sum), the 244M-parameter model matches or beats mBART50, which is roughly 2.5 times larger, by up to about 2 BLEU/ROUGE. It also transfers to languages absent from pre-training or fine-tuning. A parameter-shared variant, IndicALBART, shrinks the model further. The work shows that a model focused on a family of related languages can be a cheap, practical alternative to massive multilingual models.

## Key points
- **Coverage and size**
  - IndicBART covers 7 Indo-Aryan and 4 Dravidian languages plus English. Of these, mBART25, mBART50 and mT5 cover only 2, 7 and 9 respectively.
  - IndicBART has 244M parameters, compared with 611M for mBART50 and 580M for mT5-base.
  - IndicALBART has only 97M parameters.
- **Architecture and training**
  - The model has 6 encoder and 6 decoder layers, hidden size 1024, filter size 4096 and 16 attention heads.
  - During pre-training, 35% of words are masked, with span lengths drawn from a Poisson distribution (λ = 3.5).
  - Training ran for 750K iterations on 48 V100 GPUs, about 5 days. mBART took more than 2 weeks on 256 GPUs.
- **Script unification**
  - Mapping everything to Devanagari reduces 9 scripts to 1.
  - The vocabulary is 64K subwords, versus 250K for mBART.
  - A separate-script ablation, SSIndicBART, generally performed worse. The one exception was Punjabi in summarization.
- **Zero-shot transfer**
  - The model enables translation for Sinhala and Nepali, neither of which was seen in pre-training.
  - Fine-tuning on some languages also allows translation for other, unseen languages.
- **Compression trade-off and release**
  - On summarization, IndicALBART trails IndicBART by roughly 1–3 Rouge-L.
  - In NMT, distillation helps close this gap.
  - Models are released under the MIT license on GitHub and the HuggingFace hub (ai4bharat/IndicBART, IndicBARTSS).

## Interesting ideas
- **Transliterate related scripts into one before tokenization.** When languages share a script lineage, this increases subword overlap and frees vocabulary budget. It could help any language group that is split across alphabets but lexically close.
- **Build a family-specific model instead of a global one.** Restricting pre-training to related languages, with regionally relevant content, uses limited capacity more efficiently. Including local English web text is part of this choice. The approach is attractive for teams with modest GPU budgets.
- **Combine cross-layer parameter sharing with distillation.** Training the ALBART-style shared model on distilled data from a larger model keeps much of the quality at a fraction of the size. This is useful for deployment on constrained hardware.
- **Test zero-shot behaviour explicitly.** The evaluation checks performance on languages missing from pre-training and on languages with no fine-tuning data. This is a cheap way to probe how far cross-lingual transfer really reaches.
- **Cap n-gram repetition when decoding summaries.** The authors block repeated 4-grams during summary generation. This simple setting curbs degenerate repetition and is easy to reuse in other generation tasks.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) — Company, Redmond, United States · 240 works · 123,537 citations
- [University of Edinburgh](https://md.research.elamur.ai/free/en/universities/university-of-edinburgh/) — University, Edinburgh, United Kingdom · 255 works · 68,771 citations
- [Indian Institute of Technology Madras](https://md.research.elamur.ai/free/en/universities/indian-institute-of-technology-madras/) — University, Chennai, India · 9 works · 2,396 citations
- [National Institute of Information and Communications Technology](https://md.research.elamur.ai/free/en/labs/national-institute-of-information-and-communications-technology/) — Research institute, Tokyo, Japan · 13 works · 1,970 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [✍️ Text generation](https://md.research.elamur.ai/free/en/topics/text-generation/) · 145 works

## Cite

Cite as: Raj Dabre, Himani Shrotriya, Anoop Kunchukuttan, Ratish Puduppully, Mitesh Khapra and Pratyush Kumar. 2022. IndicBART: A Pre-trained Model for Indic Natural Language Generation. Findings of the Association for Computational Linguistics: ACL 2022.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{dabre2022indicbart,
  title = {{IndicBART: A Pre-trained Model for Indic Natural Language Generation}},
  author = {Raj Dabre, Himani Shrotriya, Anoop Kunchukuttan, Ratish Puduppully, Mitesh Khapra and Pratyush Kumar},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2022},
  year = {2022},
  doi = {10.18653/v1/2022.findings-acl.145},
  url = {https://doi.org/10.18653/v1/2022.findings-acl.145}
}
```

- [← previous TALM: Tool Augmented Language Models](https://md.research.elamur.ai/free/en/papers/parisi-2022-talm-tool-augmented-language-models/)
- [next → RelationPrompt: Leveraging Prompts to Generate Synthetic Data for Zero-Shot Relation Triplet Extraction](https://md.research.elamur.ai/free/en/papers/chia-2022-relationprompt-leveraging-prompts-generate-synthetic/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/)

## References

43 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North 17 [https://www.mha.gov.in/sites/default/](https://www.mha.gov.in/sites/default/) files/EighthSchedule_19052017.pdf Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019… [↗](https://www.mha.gov.in/sites/default/) [our card](https://md.research.elamur.ai/free/en/papers/aharoni-2019-massively-multilingual-neural-machine-translation/)
2. Tejas Dhamecha, Rudra Murthy, Samarth Bharadwaj, Karthik Sankaranarayanan, and Pushpak Bhattacharyya. 2021. Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languages. In Proceedings of the 2021 Conference on Empirical Methods in Natural Languag… [find](https://scholar.google.com/scholar?q=Tejas+Dhamecha%2C+Rudra+Murthy%2C+Samarth+Bharadwaj%2C+Karthik+Sankaranarayanan%2C+and+Pushpak+Bhattacharyya%2E+2021%2E+Role+of+Language+Relatedness+in+Multilingual+Fine%2Dtuning+of+Language+Models%3A+A+Case+Study+in)
3. Murray B Emeneau. 1956. India as a lingustic area. Language, 32(1):3–16. [find](https://scholar.google.com/scholar?q=Murray+B+Emeneau%2E+1956%2E+India+as+a+lingustic+area%2E+Language%2C+32%281%29%3A3%E2%80%9316%2E)
4. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2021. The FLORES-101 evaluation benchmark for low-resource and multilingual machine translation. CoRR, abs/2106.03193. [find](https://scholar.google.com/scholar?q=Naman+Goyal%2C+Cynthia+Gao%2C+Vishrav+Chaudhary%2C+Peng%2DJen+Chen%2C+Guillaume+Wenzek%2C+Da+Ju%2C+Sanjana+Krishnan%2C+Marc%E2%80%99Aurelio+Ranzato%2C+Francisco+Guzm%C3%A1n%2C+and+Angela+Fan%2E+2021%2E+The+FLORES%2D101+evaluation+benchmark)
5. Vikrant Goyal, Anoop Kunchukuttan, Rahul Kejriwal, Siddharth Jain, and Amit Bhagwat. 2020a. Contact relatedness can help improve multilingual NMT: Microsoft STCI-MT @ WMT20. In Proceedings of the Fifth Conference on Machine Translation, pages 202–206, Online. Association for Computational Linguistic… [find](https://scholar.google.com/scholar?q=Vikrant+Goyal%2C+Anoop+Kunchukuttan%2C+Rahul+Kejriwal%2C+Siddharth+Jain%2C+and+Amit+Bhagwat%2E+2020a%2E+Contact+relatedness+can+help+improve+multilingual+NMT%3A+Microsoft+STCI%2DMT+%40+WMT20%2E+In+Proceedings+of+the+Fift)
6. Vikrant Goyal, Anoop Kunchukuttan, Rahul Kejriwal, Siddharth Jain, and Amit Bhagwat. 2020b. Contact Relatedness can help improve multilingual NMT: Microsoft STCI-MT @ WMT20. In Conference on Machine Translation. [find](https://scholar.google.com/scholar?q=Vikrant+Goyal%2C+Anoop+Kunchukuttan%2C+Rahul+Kejriwal%2C+Siddharth+Jain%2C+and+Amit+Bhagwat%2E+2020b%2E+Contact+Relatedness+can+help+improve+multilingual+NMT%3A+Microsoft+STCI%2DMT+%40+WMT20%2E+In+Conference+on+Machine+T)
7. Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc’Aurelio Ranzato. 2019. The FLORES evaluation datasets for low-resource machine translation: Nepali–English and Sinhala– English. In Proceedings of the 2019 Conference on Empirical Metho… [our card](https://md.research.elamur.ai/free/en/papers/guzman-2019-flores-evaluation-datasets-low-resource/)
8. Barry Haddow and Faheem Kirefu. 2020. PMIndia – A Collection of Parallel Corpora of Languages of India. arxiv 2001.09907. — [arXiv:2001.09907](https://arxiv.org/abs/2001.09907) [find](https://scholar.google.com/scholar?q=Barry+Haddow+and+Faheem+Kirefu%2E+2020%2E+PMIndia+%E2%80%93+A+Collection+of+Parallel+Corpora+of+Languages+of+India%2E+arxiv+2001%2E09907%2E+%E2%80%94+%5BarXiv%3A2001%2E09907%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2001%2E09907%29)
9. Macduff Hughes, and Jeffrey Dean. 2017. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339–351. [find](https://scholar.google.com/scholar?q=Macduff+Hughes%2C+and+Jeffrey+Dean%2E+2017%2E+Google%E2%80%99s+multilingual+neural+machine+translation+system%3A+Enabling+zero%2Dshot+translation%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+5%3A339%E2%80%9335)
10. Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar. 2020. IndicNLPSuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for Indian languages. In Findings of the Association for Computation… [our card](https://md.research.elamur.ai/free/en/papers/kakwani-2020-indicnlpsuite-monolingual-corpora-evaluation-benchmarks/)
11. Mihir Kale and Abhinav Rastogi. 2020. Text-to-text pre-training for data-to-text tasks. In Proceedings of the 13th International Conference on Natural Language Generation, pages 97–102, Dublin, Ireland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mihir+Kale+and+Abhinav+Rastogi%2E+2020%2E+Text%2Dto%2Dtext+pre%2Dtraining+for+data%2Dto%2Dtext+tasks%2E+In+Proceedings+of+the+13th+International+Conference+on+Natural+Language+Generation%2C+pages+97%E2%80%93102%2C+Dublin%2C+Irelan)
12. Simran Khanuja, Diksha Bansal, Sarvesh Mehtani, Savya Khosla, Atreyee Dey, Balaji Gopalan, Dilip Kumar Margam, Pooja Aggarwal, Rajiv Teja Nagipogu, Shachi Dave, Shruti Gupta, Subhash Chandra Bose Gali, Vish Subramanian, and Partha Talukdar. 2021. Muril: Multilingual representations for indian langua… [our card](https://md.research.elamur.ai/free/en/papers/khanuja-2021-muril-multilingual-representations-indian-languages/)
13. Yash Khemchandani, Sarvesh Mehtani, Vaidehi Patil, Abhijeet Awasthi, Partha Talukdar, and Sunita Sarawagi. 2021. Exploiting language relatedness for low web-resource language model adaptation: An Indic languages study. In Proceedings of the 59th Annual Meeting of the Association for Computational Li… [find](https://scholar.google.com/scholar?q=Yash+Khemchandani%2C+Sarvesh+Mehtani%2C+Vaidehi+Patil%2C+Abhijeet+Awasthi%2C+Partha+Talukdar%2C+and+Sunita+Sarawagi%2E+2021%2E+Exploiting+language+relatedness+for+low+web%2Dresource+language+model+adaptation%3A+An+Indi)
14. Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317–1327, Austin, Texas. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/kim-2016-sequence-level-knowledge-distillation/)
15. Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75, Melbourne, Australia. Association for Computational… [our card](https://md.research.elamur.ai/free/en/papers/kudo-2018-subword-regularization-improving-neural-network/)
16. Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. XL-sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pa… [find](https://scholar.google.com/scholar?q=Tahmid+Hasan%2C+Abhik+Bhattacharjee%2C+Md%2E+Saiful+Islam%2C+Kazi+Mubasshir%2C+Yuan%2DFang+Li%2C+Yong%2DBin+Kang%2C+M%2E+Sohel+Rahman%2C+and+Rifat+Shahriyar%2E+2021%2E+XL%2Dsum%3A+Large%2Dscale+multilingual+abstractive+summarization)
17. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Associ… [our card](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/)
18. Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Sneha Kudugunta, Ankur Bapna, Isaac Caswell, and Orhan Firat. 2019. Investigating multilingual NMT representations at scale. In Proceedings of the 2019… [our card](https://md.research.elamur.ai/free/en/papers/kudugunta-2019-investigating-multilingual-nmt-representations-scale/)
19. Anoop Kunchukuttan. 2020. The Indic-NLP Library. [https://github.com/](https://github.com/) anoopkunchukuttan/indic_nlp_ library/blob/master/docs/indicnlp. pdf. [↗](https://github.com/)
20. Anoop Kunchukuttan, Mitesh Khapra, Gurneet Singh, and Pushpak Bhattacharyya. 2018. Leveraging orthographic similarity for multilingual neural transliteration. Transactions of the Association for Computational Linguistics, 6:303–316. [find](https://scholar.google.com/scholar?q=Anoop+Kunchukuttan%2C+Mitesh+Khapra%2C+Gurneet+Singh%2C+and+Pushpak+Bhattacharyya%2E+2018%2E+Leveraging+orthographic+similarity+for+multilingual+neural+transliteration%2E+Transactions+of+the+Association+for+Compu)
21. Toan Q. Nguyen and David Chiang. 2017. Transfer learning across low-resource, related languages for neural machine translation. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 296–301, Taipei, Taiwan. Asian Federation of Natu… [our card](https://md.research.elamur.ai/free/en/papers/nguyen-2017-transfer-learning-across-low-resource/)
22. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pages 311–318. [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
23. Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations. [our card](https://md.research.elamur.ai/free/en/papers/paulus-2017-deep-reinforced-model-abstractive-summarization/)
24. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In ICLR. OpenReview.net. [find](https://scholar.google.com/scholar?q=Zhenzhong+Lan%2C+Mingda+Chen%2C+Sebastian+Goodman%2C+Kevin+Gimpel%2C+Piyush+Sharma%2C+and+Radu+Soricut%2E+2020%2E+Albert%3A+A+lite+bert+for+self%2Dsupervised+learning+of+language+representations%2E+In+ICLR%2E+OpenReview%2Ene)
25. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Brussels, Belgium. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/)
26. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the… [our card](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/)
27. Ratish Puduppully, Li Dong, and Mirella Lapata. 2019. Data-to-text generation with content selection and planning. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symp… [find](https://scholar.google.com/scholar?q=Ratish+Puduppully%2C+Li+Dong%2C+and+Mirella+Lapata%2E+2019%2E+Data%2Dto%2Dtext+generation+with+content+selection+and+planning%2E+In+The+Thirty%2DThird+AAAI+Conference+on+Artificial+Intelligence%2C+AAAI+2019%2C+The+Thirty)
28. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/)
29. Gowtham Ramesh, Sumanth Doddapaneni, Aravinth Bheemaraj, Mayank Jobanputra, Raghavan AK, Ajitesh Sharma, Sujit Sahoo, Harshita Diddee, Mahalakshmi J, Divyanshu Kakwani, Navneet Kumar, Aswin Pradeep, Kumar Deepak, Vivek Raghavan, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh Shantadevi Khapra. 2021.… [our card](https://md.research.elamur.ai/free/en/papers/ramesh-2022-samanantar-largest-publicly-available-parallel/)
30. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Jiatao+Gu%2C+Naman+Goyal%2C+Xian+Li%2C+Sergey+Edunov%2C+Marjan+Ghazvininejad%2C+Mike+Lewis%2C+and+Luke+Zettlemoyer%2E+2020%2E+Multilingual+denoising+pre%2Dtraining+for+neural+machine+translation%2E+Transactio)
31. Toshiaki Nakazawa, Hideki Nakayama, Chenchen Ding, Raj Dabre, Shohei Higashiyama, Hideya Mino, Isao Goto, Win Pa Pa, Anoop Kunchukuttan, Shantipriya Parida, Ondřej Bojar, Chenhui Chu, Akiko Eriguchi, Kaori Abe, and Sadao Oda, Yusuke Kurohashi. 2021. Overview of the 8th workshop on Asian translation… [find](https://scholar.google.com/scholar?q=Toshiaki+Nakazawa%2C+Hideki+Nakayama%2C+Chenchen+Ding%2C+Raj+Dabre%2C+Shohei+Higashiyama%2C+Hideya+Mino%2C+Isao+Goto%2C+Win+Pa+Pa%2C+Anoop+Kunchukuttan%2C+Shantipriya+Parida%2C+Ondr%CC%8Cej+Bojar%2C+Chenhui+Chu%2C+Akiko+Eriguchi%2C)
32. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807, Brussels, Belgium. As… [find](https://scholar.google.com/scholar?q=Shashi+Narayan%2C+Shay+B%2E+Cohen%2C+and+Mirella+Lapata%2E+2018%2E+Don%E2%80%99t+give+me+the+details%2C+just+the+summary%21+topic%2Daware+convolutional+neural+networks+for+extreme+summarization%2E+In+Proceedings+of+the+2018+Co)
33. Machel Reid, Junjie Hu, Graham Neubig, and Yutaka Matsuo. 2021. AfroMT: Pretraining strategies and reproducible benchmarks for translation of 8 African languages. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1306–1320, Online and Punta Cana, Domini… [find](https://scholar.google.com/scholar?q=Machel+Reid%2C+Junjie+Hu%2C+Graham+Neubig%2C+and+Yutaka+Matsuo%2E+2021%2E+AfroMT%3A+Pretraining+strategies+and+reproducible+benchmarks+for+translation+of+8+African+languages%2E+In+Proceedings+of+the+2021+Conference)
34. Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2020. Leveraging pre-trained checkpoints for sequence generation tasks. Transactions of the Association for Computational Linguistics, 8:264–280. [our card](https://md.research.elamur.ai/free/en/papers/rothe-2019-leveraging-pre-trained-checkpoints-sequence/)
35. Shashank Siripragrada, Jerin Philip, Vinay P. Namboodiri, and C V Jawahar. 2020. A multilingual parallel corpora collection effort for Indian languages. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 3743–3751, Marseille, France. European Language Resources Associatio… [find](https://scholar.google.com/scholar?q=Shashank+Siripragrada%2C+Jerin+Philip%2C+Vinay+P%2E+Namboodiri%2C+and+C+V+Jawahar%2E+2020%2E+A+multilingual+parallel+corpora+collection+effort+for+Indian+languages%2E+In+Proceedings+of+The+12th+Language+Resources+a)
36. Xu Tan, Yi Ren, Di He, Tao Qin, and Tie-Yan Liu. 2019. Multilingual neural machine translation with knowledge distillation. In International Conference on Learning Representations (ICLR), New Orleans, LA, USA. [find](https://scholar.google.com/scholar?q=Xu+Tan%2C+Yi+Ren%2C+Di+He%2C+Tao+Qin%2C+and+Tie%2DYan+Liu%2E+2019%2E+Multilingual+neural+machine+translation+with+knowledge+distillation%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2C+New+Orleans%2C)
37. Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020a. Multilingual translation with extensible multilingual pretraining and finetuning. [find](https://scholar.google.com/scholar?q=Yuqing+Tang%2C+Chau+Tran%2C+Xian+Li%2C+Peng%2DJen+Chen%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Jiatao+Gu%2C+and+Angela+Fan%2E+2020a%2E+Multilingual+translation+with+extensible+multilingual+pretraining+and+finetuning%2E)
38. Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020b. Multilingual translation with extensible multilingual pretraining and finetuning. CoRR, abs/2008.00401. [find](https://scholar.google.com/scholar?q=Yuqing+Tang%2C+Chau+Tran%2C+Xian+Li%2C+Peng%2DJen+Chen%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Jiatao+Gu%2C+and+Angela+Fan%2E+2020b%2E+Multilingual+translation+with+extensible+multilingual+pretraining+and+finetuning%2E+CoRR)
39. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. [our card](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/)
40. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
41. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V%2E+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+Jeff+Klingner%2C+Apurva+Shah%2C+Melvin+Johnson%2C+Xiaobing+Liu%2C+Luk)
42. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling… [our card](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/)
43. Lang as bn en gu hi kn ml mr or pa ta te Total Mono IC 1.4M 39.9M 54.3 41.1M 63.1M 53.3M 50.2M 34.0M 7.0M 29.2M 31.5 47.9M 450M PMI -23.3K -41.5K 50.3K 28.9K 26.9K 28.9K 31.9K 28.2K 32.6K 33.3K 326.3K Parallel (XX-En) LR PIB Total -- 91.9K 115.2K -- 58.2K 99.8K 266.5K 316.9K -28.9K 43.1K 70.0K 114.2… [find](https://scholar.google.com/scholar?q=Lang+as+bn+en+gu+hi+kn+ml+mr+or+pa+ta+te+Total+Mono+IC+1%2E4M+39%2E9M+54%2E3+41%2E1M+63%2E1M+53%2E3M+50%2E2M+34%2E0M+7%2E0M+29%2E2M+31%2E5+47%2E9M+450M+PMI+%2D23%2E3K+%2D41%2E5K+50%2E3K+28%2E9K+26%2E9K+28%2E9K+31%2E9K+28%2E2K+32%2E6K+33%2E3K+326%2E3K)

*Updated 24 October 2026*