---
title: "Bilex Rx — paper by Jones et al. (2023) — summary and ideas"
description: "The authors test whether bilingual lexica, which are cheap and plentiful, can fix this kind of error at… 85 references, cited by 18 papers here."
authors: ["Alex Jones", "Isaac Caswell", "Ishank Saxena", "Orhan Firat"]
year: 2023
venue: "arXiv:2303.15265"
source: "https://arxiv.org/abs/2303.15265"
doi: "10.48550/arxiv.2303.15265"
arxiv: "2303.15265"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/jones-2023-bilex-rx-lexical-data-augmentation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jones-2023-bilex-rx-lexical-data-augmentation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/12/) / Bilex Rx: Lexical Data Augmentation for Massively Multilingual Machine Translation

# Bilex Rx: Lexical Data Augmentation for Massively Multilingual Machine Translation

**Authors:** Alex Jones, [Isaac Caswell](https://md.research.elamur.ai/free/en/authors/isaac-caswell/), Ishank Saxena and [Orhan Firat](https://md.research.elamur.ai/free/en/authors/orhan-firat/) · **Year:** 2023 · **Venue:** arXiv:2303.15265 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2303.15265) · [PDF](https://arxiv.org/pdf/2303.15265) · [DOI 10.48550/arxiv.2303.15265](https://doi.org/10.48550/arxiv.2303.15265) · [arXiv 2303.15265](https://arxiv.org/abs/2303.15265) · [OpenAlex](https://openalex.org/W4361194234)

**Citations:** 4 citations (OpenAlex, 2026-10-14) · [18](https://md.research.elamur.ai/free/en/papers/jones-2023-bilex-rx-lexical-data-augmentation/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Massively multilingual NMT systems that rely partly or fully on monolingual text (UNMT) can be fluent but still get basic content words wrong. They often substitute nouns from the same semantic field, for example producing "crocodile" where "dog" was meant. The authors test whether bilingual lexica, which are cheap and plentiful, can fix this kind of error at production scale. They train Seq2Seq models covering roughly 200 languages on web-crawled monolingual and parallel data, combining supervised MT with self-supervised objectives. Dictionary entries are injected through several augmentation families, mainly codeswitching source words into their translations and prepending lexicon pairs to inputs, and these families are also combined. Lexical augmentation brings clear gains, largest for unsupervised and low-resource languages. The different families perform about equally well and add up when combined. Lexicon quality matters more than size, especially for larger models. The study also compares lexicon data with human-translated parallel data and releases GATITOS, a curated lexicon for 26 low-resource languages, which matters for languages lacking bitext.

## Key points
- Training data: 208 languages, all with monolingual data. For speed, monolingual data for the 100 highest-resource languages was cut to 10%, leaving about 4B sentences (about 80B tokens). The large-model runs used the full 27B sentences (540B tokens).
- Lexica compared: the large, noisy Panlex versus the small, curated GATITOS. GATITOS gave roughly 5× bigger ChrF improvements and was the strongest lexicon tested.
- Augmentation families include codeswitching (also known in prior work as "dictionary denoising" or Random Aligned Substitution) and prepending lexicon translations to source sentences. Combining methods gives further gains.
- Evaluation uses ChrF on FLORES. The appendix lists per-language results for baseline, augmented, and larger model variants, alongside NLLB numbers.
- GATITOS is open-sourced in the google-research/url-nlp GitHub repository.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 492 works · 440,463 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (2): [🔁 Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · 1,105 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 882 works

## Cite

Cite as: Alex Jones, Isaac Caswell, Ishank Saxena and Orhan Firat. 2023. Bilex Rx: Lexical Data Augmentation for Massively Multilingual Machine Translation. arXiv:2303.15265.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{jones2023bilex,
  title = {{Bilex Rx: Lexical Data Augmentation for Massively Multilingual Machine Translation}},
  author = {Alex Jones, Isaac Caswell, Ishank Saxena and Orhan Firat},
  year = {2023},
  doi = {10.48550/arxiv.2303.15265},
  eprint = {2303.15265},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.15265}
}
```

- [← previous A Survey of Corpora for Germanic Low-Resource Languages and Dialects](https://md.research.elamur.ai/free/en/papers/blaschke-2023-survey-corpora-germanic-low-resource/)
- [next → Chain-of-Thought Prompt Distillation for Multimodal Named Entity Recognition and Multimodal Relation Extraction](https://md.research.elamur.ai/free/en/papers/chen-2023-chain-thought-prompt-distillation-multimodal/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/12/)

## References

85 references, 37 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Antonov, Alexander. Chuvash-Russian parallel corpus. [https://github.com/AlAntonov/](https://github.com/AlAntonov/) chv_corpus, 2022. [↗](https://github.com/AlAntonov/)
2. Roee Aharoni, Melvin Johnson, and Orhan Firat. [Massively multilingual neural machine translation](https://md.research.elamur.ai/free/en/papers/aharoni-2019-massively-multilingual-neural-machine-translation/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 3874–3884, Minnea…
3. Benjamin Akera, Jonathan Mukiibi, Lydia Sanyu Naggayi, Claire Babirye, Isaac Owomugisha, Solomon Nsumba, Joyce Nakatumba-Nabende, Engineer Bainomugisha, Ernest Mwebaze, and John Quinn. Machine translation for african languages: Community creation of datasets and models in uganda. In 3rd Workshop on… [find](https://scholar.google.com/scholar?q=Benjamin+Akera%2C+Jonathan+Mukiibi%2C+Lydia+Sanyu+Naggayi%2C+Claire+Babirye%2C+Isaac+Owomugisha%2C+Solomon+Nsumba%2C+Joyce+Nakatumba%2DNabende%2C+Engineer+Bainomugisha%2C+Ernest+Mwebaze%2C+and+John+Quinn%2E+Machine+transla)
4. Andersen, Jógvan. Sprotin Translations. [https://github.com/Sprotin/translations](https://github.com/Sprotin/translations), 2021. [↗](https://github.com/Sprotin/translations)
5. Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. [Common voice: A massively-multilingual speech corpus](https://md.research.elamur.ai/free/en/papers/ardila-2019-common-voice-massively-multilingual-speech/). In Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 4218–4222,…
6. Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George F. Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. [Massively multilingual neural machine translation in the wild: Findings and challenges](https://md.research.elamur.ai/free/en/papers/arivazhagan-2019-massively-multilingual-neural-machine-translation/). CoRR, abs/1907.…
7. Mikel Artetxe and Holger Schwenk. [Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond](https://md.research.elamur.ai/free/en/papers/artetxe-2018-massively-multilingual-sentence-embeddings-zero/). CoRR, abs/1812.10464, 2018. URL [http://arxiv.org/](http://arxiv.org/) abs/1812.10464. [↗](http://arxiv.org/)
8. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. [Unsupervised neural machine translation](https://md.research.elamur.ai/free/en/papers/artetxe-2017-unsupervised-neural-machine-translation/). CoRR, abs/1710.11041, 2017. URL http://arxiv.org/abs/1710.11041. [↗](http://arxiv.org/abs/1710.11041)
9. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. [An effective approach to unsupervised machine translation](https://md.research.elamur.ai/free/en/papers/artetxe-2019-effective-approach-unsupervised-machine-translation/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 194–203, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P…
10. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 201…
11. Marta Bañón, Pinzhen Chen, Barry Haddow, Kenneth Heafield, Hieu Hoang, Miquel Esplà-Gomis, Mikel L. Forcada, Amir Kamran, Faheem Kirefu, Philipp Koehn, Sergio Ortiz Rojas, Leopoldo Pla Sempere, Gema Ramírez-Sánchez, Elsa Sarrías, Marek Strelec, Brian Thompson, William Waites, Dion Wiggins, and Jaume… — [ParaCrawl: Web-Scale Acquisition of Parallel Corpora](https://md.research.elamur.ai/free/en/papers/banon-2020-paracrawl-web-scale-acquisition-parallel/)
12. Ankur Bapna, Isaac Caswell, Julia Kreutzer, Orhan Firat, Daan van Esch, Aditya Siddhant, Mengmeng Niu, Pallavi Baljekar, Xavier Garcia, Wolfgang Macherey, Theresa Breiner, Vera Axelrod, Jason Riesa, Yuan Cao, Mia Xu Chen, Klaus Macherey, Maxim Krikun, Pidong Wang, Alexander Gutkin, Apurva Shah, Yanp… [find](https://scholar.google.com/scholar?q=Ankur+Bapna%2C+Isaac+Caswell%2C+Julia+Kreutzer%2C+Orhan+Firat%2C+Daan+van+Esch%2C+Aditya+Siddhant%2C+Mengmeng+Niu%2C+Pallavi+Baljekar%2C+Xavier+Garcia%2C+Wolfgang+Macherey%2C+Theresa+Breiner%2C+Vera+Axelrod%2C+Jason+Riesa%2C+Y)
13. Christos Baziotis, Barry Haddow, and Alexandra Birch. Language model prior for low-resource neural machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7622–7634, Online, November 2020. Association for Computational Linguistics.… [find](https://scholar.google.com/scholar?q=Christos+Baziotis%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+Language+model+prior+for+low%2Dresource+neural+machine+translation%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+P)
14. Isaac Caswell, Ciprian Chelba, and David Grangier. [Tagged back-translation](https://md.research.elamur.ai/free/en/papers/caswell-2019-tagged-back-translation/). In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), pp. 53–63, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10.18653/v1/W19-5206. URL https://aclanthol…
15. Isaac Caswell, Theresa Breiner, Daan van Esch, and Ankur Bapna. [Language id in the wild: Unexpected challenges on the path to a thousand-language web text corpus](https://md.research.elamur.ai/free/en/papers/caswell-2020-language-id-wild-unexpected/), 2020. URL https://arxiv.org/abs/2010.14571. [↗](https://arxiv.org/abs/2010.14571)
16. Aditi Chaudhary, Karthik Raman, Krishna Srinivasan, and Jiecao Chen. DICT-MLM: Improved Multilingual Pre-Training using Bilingual Dictionaries. arXiv e-prints, art. arXiv:2010.12566, October 2020. — [arXiv:2010.12566](https://arxiv.org/abs/2010.12566) [find](https://scholar.google.com/scholar?q=Aditi+Chaudhary%2C+Karthik+Raman%2C+Krishna+Srinivasan%2C+and+Jiecao+Chen%2E+DICT%2DMLM%3A+Improved+Multilingual+Pre%2DTraining+using+Bilingual+Dictionaries%2E+arXiv+e%2Dprints%2C+art%2E+arXiv%3A2010%2E12566%2C+October+2020%2E+%E2%80%94+%5B)
17. Yong Cheng, Wei Wang, Lu Jiang, and Wolfgang Macherey. Self-supervised and supervised joint training for resource-rich machine translation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Resea… [find](https://scholar.google.com/scholar?q=Yong+Cheng%2C+Wei+Wang%2C+Lu+Jiang%2C+and+Wolfgang+Macherey%2E+Self%2Dsupervised+and+supervised+joint+training+for+resource%2Drich+machine+translation%2E+In+Marina+Meila+and+Tong+Zhang+%28eds%2E%29%2C+Proceedings+of+the+38)
18. Luis Chiruzzo, Santiago Góngora, Aldo Alvarez, Gustavo Giménez-Lugo, Marvin Agüero-Torales, and Yliana Rodríguez. Jojajovai: A parallel Guarani-Spanish corpus for MT benchmarking. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 2098– 2107, Marseille, France, June 2… [find](https://scholar.google.com/scholar?q=Luis+Chiruzzo%2C+Santiago+G%C3%B3ngora%2C+Aldo+Alvarez%2C+Gustavo+Gim%C3%A9nez%2DLugo%2C+Marvin+Ag%C3%BCero%2DTorales%2C+and+Yliana+Rodr%C3%ADguez%2E+Jojajovai%3A+A+parallel+Guarani%2DSpanish+corpus+for+MT+benchmarking%2E+In+Proceedings+of+th)
19. Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. arXiv preprint arXiv:1710.04087, 2017. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Guillaume+Lample%2C+Marc%E2%80%99Aurelio+Ranzato%2C+Ludovic+Denoyer%2C+and+Herv%C3%A9+J%C3%A9gou%2E+Word+translation+without+parallel+data%2E+arXiv+preprint+arXiv%3A1710%2E04087%2C+2017%2E)
20. Chris Chinenye Emezue and Femi Pancrace Bonaventure Dossou. FFR v1.1: Fon-French neural machine translation. In Proceedings of the The Fourth Widening Natural Language Processing Workshop, pp. 83–87, Seattle, USA, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.winlp-1.21… — [doi:10.18653/v1/2020.winlp-1.21](https://doi.org/10.18653/v1/2020.winlp-1.21) [find](https://scholar.google.com/scholar?q=Chris+Chinenye+Emezue+and+Femi+Pancrace+Bonaventure+Dossou%2E+FFR+v1%2E1%3A+Fon%2DFrench+neural+machine+translation%2E+In+Proceedings+of+the+The+Fourth+Widening+Natural+Language+Processing+Workshop%2C+pp%2E+83%E2%80%9387%2C+)
21. Miquel Esplà-Gomis. Bitextor: a free/open-source software to harvest translation memories from multilingual websites. In Beyond Translation Memories: New Tools for Translators Workshop, Ottawa, Canada, August 26-30 2009. URL https://aclanthology.org/2009. mtsummit-btm.6. [↗](https://aclanthology.org/2009)
22. Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. Beyond english-centric multilingual ma… [find](https://scholar.google.com/scholar?q=Angela+Fan%2C+Shruti+Bhosale%2C+Holger+Schwenk%2C+Zhiyi+Ma%2C+Ahmed+El%2DKishky%2C+Siddharth+Goyal%2C+Mandeep+Baines%2C+Onur+Celebi%2C+Guillaume+Wenzek%2C+Vishrav+Chaudhary%2C+Naman+Goyal%2C+Tom+Birch%2C+Vitaliy+Liptchinsky%2C+S)
23. Isaac Feldman and Rolando Coto-Solano. Neural machine translation models with back-translation for the extremely low-resource indigenous language Bribri. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 3965–3976, Barcelona, Spain (Online), December 2020. Interna… [find](https://scholar.google.com/scholar?q=Isaac+Feldman+and+Rolando+Coto%2DSolano%2E+Neural+machine+translation+models+with+back%2Dtranslation+for+the+extremely+low%2Dresource+indigenous+language+Bribri%2E+In+Proceedings+of+the+28th+International+Confe)
24. Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. [Multi-way, multilingual neural machine translation with a shared attention mechanism](https://md.research.elamur.ai/free/en/papers/firat-2016-multi-way-multilingual-neural-machine/). In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 866–875, San Die…
25. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi kiu Lo, Craig Stewart, Eleftherios Avramidis, George Foster, Alon Lavie, and Andr’ e F. T. Martins. [Results of wmt22 metrics shared task: Stop using bleu – neural metrics are better and more robust](https://md.research.elamur.ai/free/en/papers/freitag-2022-results-wmt22-metrics-shared-task/). 2022a. URL https://aclanthology. org/2022.wmt-1.2.pdf…
26. Markus Freitag, David Vilar, David Grangier, Colin Cherry, and George Foster. A natural diet: Towards improving naturalness of machine translation output. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 3340–3353, Dublin, Ireland, May 2022b. Association for Computational… [find](https://scholar.google.com/scholar?q=Markus+Freitag%2C+David+Vilar%2C+David+Grangier%2C+Colin+Cherry%2C+and+George+Foster%2E+A+natural+diet%3A+Towards+improving+naturalness+of+machine+translation+output%2E+In+Findings+of+the+Association+for+Computatio)
27. Marjan Ghazvininejad, Hila Gonen, and Luke Zettlemoyer. [Dictionary-based phrase-level prompting of large language models for machine translation](https://md.research.elamur.ai/free/en/papers/ghazvininejad-2023-dictionary-based-phrase-level-prompting/), 2023. URL https://arxiv.org/abs/ 2302.07856. [↗](https://arxiv.org/abs/)
28. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. The flores-101 evaluation benchmark for low-resource and multilingual machine translation. 2021. [find](https://scholar.google.com/scholar?q=Naman+Goyal%2C+Cynthia+Gao%2C+Vishrav+Chaudhary%2C+Peng%2DJen+Chen%2C+Guillaume+Wenzek%2C+Da+Ju%2C+Sanjana+Krishnan%2C+Marc%E2%80%99Aurelio+Ranzato%2C+Francisco+Guzm%C3%A1n%2C+and+Angela+Fan%2E+The+flores%2D101+evaluation+benchmark+for+l)
29. Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, and Yoshua Bengio. On integrating a language model into neural machine translation. Computer Speech & Language, 45:137– 148, 2017. ISSN 0885-2308. doi: https://doi.org/10.1016/j.csl.2017.01.014. URL https://www.sciencedirect.com/science/articl… [↗](https://doi.org/10.1016/j.csl.2017.01.014)
30. Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc’Aurelio Ranzato. Two new evaluation datasets for low-resource machine translation: Nepali-english and sinhala-english. 2019. [find](https://scholar.google.com/scholar?q=Francisco+Guzm%C3%A1n%2C+Peng%2DJen+Chen%2C+Myle+Ott%2C+Juan+Pino%2C+Guillaume+Lample%2C+Philipp+Koehn%2C+Vishrav+Chaudhary%2C+and+Marc%E2%80%99Aurelio+Ranzato%2E+Two+new+evaluation+datasets+for+low%2Dresource+machine+translation%3A+Ne)
31. Asmelash Hadgu, Gebrekirstos Gebremeskel, and Abel Aregawi. HornMT. [https://github.com/asmelashteka/HornMT](https://github.com/asmelashteka/HornMT), 2022. Accessed: 2023-03-24. [↗](https://github.com/asmelashteka/HornMT)
32. Kevin Heffernan, Onur Çelebi, and Holger Schwenk. [Bitext mining using distilled sentence representations for low-resource languages](https://md.research.elamur.ai/free/en/papers/heffernan-2022-bitext-mining-distilled-sentence-representations/), 2022. URL https://arxiv.org/abs/2205.12654. [↗](https://arxiv.org/abs/2205.12654)
33. Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. Is chatgpt a good translator? a preliminary study, 2023. [URL https://arxiv.org/abs/2301.08745](https://md.research.elamur.ai/free/en/papers/jiao-2023-chatgpt-good-translator-yes-gpt/). [↗](https://arxiv.org/abs/2301.08745)
34. Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Associati… — [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://md.research.elamur.ai/free/en/papers/johnson-2017-google-s-multilingual-neural-machine/)
35. David Kamholz, Jonathan Pool, and Susan Colowick. PanLex: Building a resource for panlingual lexical translation. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pp. 3145–3150, Reykjavik, Iceland, May 2014. European Language Resources Association… [find](https://scholar.google.com/scholar?q=David+Kamholz%2C+Jonathan+Pool%2C+and+Susan+Colowick%2E+PanLex%3A+Building+a+resource+for+panlingual+lexical+translation%2E+In+Proceedings+of+the+Ninth+International+Conference+on+Language+Resources+and+Evaluat)
36. Jyotsana Khatri, Rudra Murthy, Tamali Banerjee, and Pushpak Bhattacharyya. Simple measures of bridging lexical divergence help unsupervised neural machine translation for low-resource languages. 2021. URL [https://link.springer.com/article/10.1007/](https://link.springer.com/article/10.1007/) s10590-021-09292-y. [↗](https://link.springer.com/article/10.1007/)
37. Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. In Proceedings of the Sixth Conference on Machine Translation, pp. 478–494, Online, Novembe… [find](https://scholar.google.com/scholar?q=Tom+Kocmi%2C+Christian+Federmann%2C+Roman+Grundkiewicz%2C+Marcin+Junczys%2DDowmunt%2C+Hitokazu+Matsushita%2C+and+Arul+Menezes%2E+To+ship+or+not+to+ship%3A+An+extensive+evaluation+of+automatic+metrics+for+machine+tran)
38. Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro… — [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://md.research.elamur.ai/free/en/papers/kreutzer-2021-quality-glance-audit-web-crawled/)
39. Nalin Kumar, Deepak Kumar, and Subhankar Mishra. Dict-nmt: Bilingual dictionary based nmt for extremely low resource languages, 2022. [find](https://scholar.google.com/scholar?q=Nalin+Kumar%2C+Deepak+Kumar%2C+and+Subhankar+Mishra%2E+Dict%2Dnmt%3A+Bilingual+dictionary+based+nmt+for+extremely+low+resource+languages%2C+2022%2E)
40. Garry Kuwanto, Afra Feyza Akyürek, Isidora Chara Tourni, Siyang Li, and Derry Wijaya. Low-resource machine translation for low-resource languages: Leveraging comparable data, code-switching and compute resources. CoRR, abs/2103.13272, 2021. URL [https://arxiv.org/](https://arxiv.org/) abs/2103.13272. [↗](https://arxiv.org/)
41. Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. [Unsupervised machine translation using monolingual corpora only](https://md.research.elamur.ai/free/en/papers/lample-2017-unsupervised-machine-translation-using-monolingual/). arXiv preprint arXiv:1711.00043, 2017.
42. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. [BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). In Proceedings of the 58th Annual Meeting of the Assoc…
43. Yusen Lin, Jiayong Lin, Shuaicheng Zhang, and Haoying Dai. Bilingual dictionary-based language model pretraining for neural machine translation. CoRR, abs/2103.07040, 2021. URL https://arxiv.org/abs/2103.07040. — [arXiv:2103.07040](https://arxiv.org/abs/2103.07040) [↗](https://arxiv.org/abs/2103.07040)
44. Zehui Lin, Xiao Pan, Mingxuan Wang, Xipeng Qiu, Jiangtao Feng, Hao Zhou, and Lei Li. [Pre-training multilingual neural machine translation by leveraging alignment information](https://md.research.elamur.ai/free/en/papers/lin-2020-pre-training-multilingual-neural-machine/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2649– 2663, Online, No…
45. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742, 2020. doi: 10.1162/ tacl_a_00343. URL http… [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Jiatao+Gu%2C+Naman+Goyal%2C+Xian+Li%2C+Sergey+Edunov%2C+Marjan+Ghazvininejad%2C+Mike+Lewis%2C+and+Luke+Zettlemoyer%2E+Multilingual+denoising+pre%2Dtraining+for+neural+machine+translation%2E+Transactions+of+)
46. Zihan Liu, Genta Indra Winata, and Pascale Fung. Continual mixed-language pre-training for extremely low-resource neural machine translation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 2706–2718, Online, August 2021. Association for Computational Linguistics.… [find](https://scholar.google.com/scholar?q=Zihan+Liu%2C+Genta+Indra+Winata%2C+and+Pascale+Fung%2E+Continual+mixed%2Dlanguage+pre%2Dtraining+for+extremely+low%2Dresource+neural+machine+translation%2E+In+Findings+of+the+Association+for+Computational+Linguisti)
47. Manuel Mager, Arturo Oncevay, Abteen Ebrahimi, John Ortega, Annette Rios, Angela Fan, Ximena Gutierrez-Vasques, Luis Chiruzzo, Gustavo Giménez-Lugo, Ricardo Ramos, Ivan Vladimir Meza Ruiz, Rolando Coto-Solano, Alexis Palmer, Elisabeth Mager-Hois, Vishrav Chaudhary, Graham Neubig, Ngoc Thang Vu, and… — [Findings of the AmericasNLP 2021 Shared Task on Open Machine Translation for Indigenous Languages of the Americas](https://md.research.elamur.ai/free/en/papers/mager-2021-findings-americasnlp-2021-shared-task/)
48. Ayush Maheshwari, Piyush Sharma, Preethi Jyothi, and Ganesh Ramakrishnan. Dictdis: Dictionary constrained disambiguation for improved nmt, 2022. URL https://arxiv.org/abs/2210. 06996. [↗](https://arxiv.org/abs/2210)
49. Christopher Malon. Overcoming poor word embeddings with word definitions. abs/2103.03842, 2021. URL https://arxiv.org/abs/2103.03842. [↗](https://arxiv.org/abs/2103.03842)
50. CoRR, Elise Michon, Josep Crego, and Jean Senellart. Integrating domain terminology into neural machine translation. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 3925–3937, Barcelona, Spain (Online), December 2020. International Committee on Computational Lin… [find](https://scholar.google.com/scholar?q=CoRR%2C+Elise+Michon%2C+Josep+Crego%2C+and+Jean+Senellart%2E+Integrating+domain+terminology+into+neural+machine+translation%2E+In+Proceedings+of+the+28th+International+Conference+on+Computational+Linguistics%2C+p)
51. Jonathan Mukiibi, Babirye Claire, and Nakatumba-Nabende Joyce. The Makerere MT Corpus: English to Luganda parallel corpus, May 2021. URL https://doi.org/10.5281/zenodo. 5089560. — [doi:10.5281/zenodo](https://doi.org/10.5281/zenodo) [↗](https://doi.org/10.5281/zenodo)
52. Sreyashi Nag, Mihir Kale, Varun Lakshminarasimhan, and Swapnil Singhavi. Incorporating bilingual dictionaries for low resource semi-supervised neural machine translation. CoRR, abs/2004.02071, 2020. URL https://arxiv.org/abs/2004.02071. — [arXiv:2004.02071](https://arxiv.org/abs/2004.02071) [↗](https://arxiv.org/abs/2004.02071)
53. Jan Niehues. Continuous learning in neural machine translation using bilingual dictionaries. CoRR, abs/2102.06558, 2021. URL https://arxiv.org/abs/2102.06558. [↗](https://arxiv.org/abs/2102.06558)
54. NLLBTeam, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoff… [find](https://scholar.google.com/scholar?q=NLLBTeam%2C+Marta+R%2E+Costa%2Djuss%C3%A0%2C+James+Cross%2C+Onur+%C3%87elebi%2C+Maha+Elbayad%2C+Kenneth+Heafield%2C+Kevin+Heffernan%2C+Elahe+Kalbassi%2C+Janice+Lam%2C+Daniel+Licht%2C+Jean+Maillard%2C+Anna+Sun%2C+Skyler+Wang%2C+Guillaume+Wen)
55. Xiao Pan, Mingxuan Wang, Liwei Wu, and Lei Li. [Contrastive learning for many-to-many multilingual neural machine translation](https://md.research.elamur.ai/free/en/papers/pan-2021-contrastive-learning-many-many-multilingual/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume…
56. Matt Post. [A call for clarity in reporting BLEU scores](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/). In Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 186–191, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10.18653/v1/W18-6319. URL https://aclanthology.org/W18-6319. [↗](https://aclanthology.org/W18-6319)
57. Libo Qin, Minheng Ni, Yue Zhang, and Wanxiang Che. Cosda-ml: Multi-lingual code-switching data augmentation for zero-shot cross-lingual nlp. In Christian Bessiere (ed.), Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pp. 3853–3860. International… — [doi:10.24963/ijcai.2020/533](https://doi.org/10.24963/ijcai.2020/533) [find](https://scholar.google.com/scholar?q=Libo+Qin%2C+Minheng+Ni%2C+Yue+Zhang%2C+and+Wanxiang+Che%2E+Cosda%2Dml%3A+Multi%2Dlingual+code%2Dswitching+data+augmentation+for+zero%2Dshot+cross%2Dlingual+nlp%2E+In+Christian+Bessiere+%28ed%2E%29%2C+Proceedings+of+the+Twenty%2DNint)
58. Machel Reid and Mikel Artetxe. PARADISE: Exploiting parallel data for multilingual sequence-to-sequence pretraining. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 800–810, Seattle, United States,… [find](https://scholar.google.com/scholar?q=Machel+Reid+and+Mikel+Artetxe%2E+PARADISE%3A+Exploiting+parallel+data+for+multilingual+sequence%2Dto%2Dsequence+pretraining%2E+In+Proceedings+of+the+2022+Conference+of+the+North+American+Chapter+of+the+Associat)
59. Philip Resnik and Noah A. Smith. [The web as a parallel corpus](https://md.research.elamur.ai/free/en/papers/resnik-2003-web-as-parallel-corpus/). Computational Linguistics, 29(3): 349–380, 2003. doi: 10.1162/089120103322711578. URL https://aclanthology.org/ J03-3002. [↗](https://aclanthology.org/)
60. Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. [WikiMatrix: Mining 135M parallel sentences in 1620 language pairs from Wikipedia](https://md.research.elamur.ai/free/en/papers/schwenk-2019-wikimatrix-mining-135m-parallel-sentences/). In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 1351–1361…
61. Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin, and Angela Fan. [CCMatrix: Mining billions of high-quality parallel sentences on the web](https://md.research.elamur.ai/free/en/papers/schwenk-2019-ccmatrix-mining-billions-high-quality/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conferen…
62. Thibault Sellam, Dipanjan Das, and Ankur P. Parikh. [BLEURT: learning robust metrics for text generation](https://md.research.elamur.ai/free/en/papers/sellam-2020-bleurt-learning-robust-metrics-text/). CoRR, abs/2004.04696, 2020. URL https://arxiv.org/abs/2004.04696. [↗](https://arxiv.org/abs/2004.04696)
63. Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 86–96, Berlin, Germany, August 2016. Association for Computatio… [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+Improving+neural+machine+translation+models+with+monolingual+data%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computational+Ling)
64. Iskander Shakirov and Aigiz Kunafin. Bashkir-russian parallel corpora. 2023. [find](https://scholar.google.com/scholar?q=Iskander+Shakirov+and+Aigiz+Kunafin%2E+Bashkir%2Drussian+parallel+corpora%2E+2023%2E)
65. Aditya Siddhant, Ankur Bapna, Yuan Cao, Orhan Firat, Mia Chen, Sneha Kudugunta, Naveen Arivazhagan, and Yonghui Wu. [Leveraging monolingual data with self-supervision for multilingual neural machine translation](https://md.research.elamur.ai/free/en/papers/siddhant-2020-leveraging-monolingual-data-self-supervision/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistic…
66. Aditya Siddhant, Ankur Bapna, Orhan Firat, Yuan Cao, Mia Xu Chen, Isaac Caswell, and Xavier Garcia. Towards the next 1000 languages in multilingual machine translation: Exploring the synergy between supervised and self-supervised learning. CoRR, abs/2201.03110, 2022. URL https://arxiv.org/abs/2201.0… — [arXiv:2201.03110](https://arxiv.org/abs/2201.03110) [↗](https://arxiv.org/abs/2201.0…)
67. Kai Song, Yue Zhang, Heng Yu, Weihua Luo, Kun Wang, and Min Zhang. Code-switching for enhancing NMT with pre-specified translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Sh… [find](https://scholar.google.com/scholar?q=Kai+Song%2C+Yue+Zhang%2C+Heng+Yu%2C+Weihua+Luo%2C+Kun+Wang%2C+and+Min+Zhang%2E+Code%2Dswitching+for+enhancing+NMT+with+pre%2Dspecified+translation%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+)
68. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. [MASS: Masked sequence to sequence pre-training for language generation](https://md.research.elamur.ai/free/en/papers/song-2019-mass-masked-sequence-sequence-pre/). In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learnin…
69. Raymond Hendy Susanto, Shamil Chollampatt, and Liling Tan. [Lexically constrained neural machine translation with levenshtein transformer](https://md.research.elamur.ai/free/en/papers/susanto-2020-lexically-constrained-neural-machine-translation/). CoRR, abs/2004.12681, 2020. URL https://arxiv.org/abs/2004.12681. [↗](https://arxiv.org/abs/2004.12681)
70. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. [Sequence to sequence learning with neural networks](https://md.research.elamur.ai/free/en/papers/sutskever-2014-sequence-sequence-learning-neural-networks/). In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014. URL [https://proceedings.neu…](https://proceedings.neu…) [↗](https://proceedings.neu…)
71. Richard Sutton. The bitter lesson. Mar 2019. doi: [https://www.cs.utexas.edu/~eunsol/courses/data/](https://www.cs.utexas.edu/~eunsol/courses/data/) bitter_lesson.pdf. [↗](https://www.cs.utexas.edu/~eunsol/courses/data/)
72. Jörg Tiedemann. The tatoeba translation challenge - realistic data sets for low resource and multilingual MT. CoRR, abs/2010.[06354, 2020](https://md.research.elamur.ai/free/en/papers/tiedemann-2020-tatoeba-translation-challenge-realistic-data/). URL https://arxiv.org/abs/2010.06354. [↗](https://arxiv.org/abs/2010.06354)
73. Tlisha, Nart. Multilingual Parallel Corpus. [https://github.com/danielinux7/](https://github.com/danielinux7/) Multilingual-Parallel-Corpus, 2022. Accessed: 2023-03-24. [↗](https://github.com/danielinux7/)
74. Jakob Uszkoreit, Jay Ponte, Ashok Popat, and Moshe Dubiner. [Large scale parallel document mining for machine translation](https://md.research.elamur.ai/free/en/papers/uszkoreit-2010-large-scale-parallel-document-mining/). In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pp. 1101–1109, Beijing, China, August 2010.
75. Coling 2010 [Organizing Committee](https://md.research.elamur.ai/free/en/papers/uszkoreit-2010-large-scale-parallel-document-mining/). URL https://aclanthology.org/C10-1124. [↗](https://aclanthology.org/C10-1124)
76. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Sy…
77. Xinyi Wang, Sebastian Ruder, and Graham Neubig. [Expanding pretrained models to thousands more languages via lexicon-based adaptation](https://md.research.elamur.ai/free/en/papers/wang-2022-expanding-pretrained-models-thousands-more/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 863–877, Dublin, Ireland, May 2022. Associatio…
78. Barack Wanjawa, Lilian Wanzare, Florence Indede, Owen McOnyango, Edward Ombui, and Lawrence Muchemi. Kencorpus: A kenyan language corpus of swahili, dholuo and luhya for natural language processing tasks, 2022. [find](https://scholar.google.com/scholar?q=Barack+Wanjawa%2C+Lilian+Wanzare%2C+Florence+Indede%2C+Owen+McOnyango%2C+Edward+Ombui%2C+and+Lawrence+Muchemi%2E+Kencorpus%3A+A+kenyan+language+corpus+of+swahili%2C+dholuo+and+luhya+for+natural+language+processing+ta)
79. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V%2E+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+Jeff+Klingner%2C+Apurva+Shah%2C+Melvin+Johnson%2C+Xiaobing+Liu%2C+%C5%81uk)
80. Mengzhou Xia, Xiang Kong, Antonios Anastasopoulos, and Graham Neubig. [Generalized data augmentation for low-resource translation](https://md.research.elamur.ai/free/en/papers/xia-2019-generalized-data-augmentation-low-resource/). CoRR, abs/1906.03785, 2019. URL http://arxiv.org/abs/1906.03785. [↗](http://arxiv.org/abs/1906.03785)
81. Jian Yang, Yuwei Yin, Shuming Ma, Haoyang Huang, Dongdong Zhang, Zhoujun Li, and Furu Wei. Multilingual agreement for multilingual neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Nat… [find](https://scholar.google.com/scholar?q=Jian+Yang%2C+Yuwei+Yin%2C+Shuming+Ma%2C+Haoyang+Huang%2C+Dongdong+Zhang%2C+Zhoujun+Li%2C+and+Furu+Wei%2E+Multilingual+agreement+for+multilingual+neural+machine+translation%2E+In+Proceedings+of+the+59th+Annual+Meeting)
82. Zhen Yang, Bojie Hu, Ambyera Han, Shen Huang, and Qi Ju. CSP:code-switching pre-training for neural machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2624–2636, Online, November 2020. Association for Computational Linguistics.… [find](https://scholar.google.com/scholar?q=Zhen+Yang%2C+Bojie+Hu%2C+Ambyera+Han%2C+Shen+Huang%2C+and+Qi+Ju%2E+CSP%3Acode%2Dswitching+pre%2Dtraining+for+neural+machine+translation%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+)
83. Wenhao Yu, Chenguang Zhu, Yuwei Fang, Donghan Yu, Shuohang Wang, Yichong Xu, Michael Zeng, and Meng Jiang. Dict-bert: Enhancing language model pre-training with dictionary. CoRR, abs/2110.06490, 2021. URL https://arxiv.org/abs/2110.06490. [↗](https://arxiv.org/abs/2110.06490)
84. Xing Jie Zhong and David Chiang. Look it up: Bilingual and monolingual dictionaries improve neural machine translation. CoRR, abs/2010.05997, 2020. URL https://arxiv.org/abs/ 2010.05997. [↗](https://arxiv.org/abs/)
85. Source Reference Baseline Augment Source Reference Baseline Augment Source Reference Baseline Augment [find](https://scholar.google.com/scholar?q=Source+Reference+Baseline+Augment+Source+Reference+Baseline+Augment+Source+Reference+Baseline+Augment)

*Updated 30 October 2026*