---
title: "InfoXLM: An Information-Theoretic Framework for — Chi 2020"
description: "The paper addresses why multilingual encoders transfer across languages and how to strengthen that ability."
authors: ["Zewen Chi", "Li Dong", "Furu Wei", "Nan Yang", "Saksham Singhal", "Wenhui Wang", "Xia Song", "Xian-Ling Mao", "Heyan Huang", "Ming Zhou"]
year: 2020
venue: "arXiv:2007.07834"
source: "https://arxiv.org/abs/2007.07834"
doi: "10.48550/arxiv.2007.07834"
arxiv: "2007.07834"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chi-2020-infoxlm-information-theoretic-framework-cross/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chi-2020-infoxlm-information-theoretic-framework-cross/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/) / InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training

# InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training

**Authors:** Zewen Chi, [Li Dong](https://md.research.elamur.ai/free/en/authors/li-dong/), [Furu Wei](https://md.research.elamur.ai/free/en/authors/furu-wei/), [Nan Yang](https://md.research.elamur.ai/free/en/authors/nan-yang/), [Saksham Singhal](https://md.research.elamur.ai/free/en/authors/saksham-singhal/), [Wenhui Wang](https://md.research.elamur.ai/free/en/authors/wenhui-wang/), [Xia Song](https://md.research.elamur.ai/free/en/authors/xia-song/), Xian-Ling Mao, [Heyan Huang](https://md.research.elamur.ai/free/en/authors/heyan-huang/), [Ming Zhou](https://md.research.elamur.ai/free/en/authors/ming-zhou/) · **Year:** 2020 · **Venue:** arXiv:2007.07834 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2007.07834) · [PDF](https://arxiv.org/pdf/2007.07834) · [DOI 10.48550/arxiv.2007.07834](https://doi.org/10.48550/arxiv.2007.07834) · [arXiv 2007.07834](https://arxiv.org/abs/2007.07834) · [OpenAlex](https://openalex.org/W3042711927)

**Citations:** 64 citations (openalex-snapshot · 2026-10-14) · [205](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary

The paper addresses why multilingual encoders transfer across languages and how to strengthen that ability. The authors treat cross-lingual pre-training as maximizing mutual information between different views of multilingual text, at both token and sentence granularity. Under this lens, multilingual masked language modeling (MMLM) gains cross-lingual signal through shared anchor tokens. Translation language modeling (TLM) additionally exploits context from a parallel sentence in another language. The framework also motivates a new sequence-level objective, cross-lingual contrast (XLCo). In XLCo, a sentence and its translation count as two views of one meaning, and the model must separate the true translation from negatives. XLCo is implemented with momentum contrast and a mixup variant, and it is applied at an intermediate layer rather than the last one. InfoXLM is trained jointly on MMLM, TLM and XLCo, using both monolingual and parallel corpora. It beats strong baselines such as XLM-R on cross-lingual inference, question answering and sentence retrieval. This matters for bringing NLP to lower-resource languages.

## Key points

- Evaluation covers XNLI, MLQA and cross-lingual sentence retrieval. Code and checkpoints are released at https://aka.ms/infoxlm.
- On MLQA, InfoXLM LARGE averages 73.6 F1 / 55.2 EM. The reimplemented XLM-R LARGE scores 72.7 / 54.3.
- The cross-lingual transfer gap is the average drop from English to other languages, averaged over MLQA and XNLI:
  - mBERT: 20.1
  - XLM-R: 14.0
  - InfoXLM without XLCo: 13.6
  - InfoXLM: 13.1
- Placing XLCo on layer 8 (the "universal layer") works better than layer 12 when TLM is also used. The reported XNLI scores are 76.45 vs 76.12.
- Ablations show that TLM and XLCo each help, with TLM stronger on XNLI and XLCo stronger on MLQA. Their combination performs best, and removing mixup or momentum contrast lowers results.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

205 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 193 not yet.

- [SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects](https://md.research.elamur.ai/free/en/papers/adelani-2023-sib-200-simple-inclusive-big/)
- [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://md.research.elamur.ai/free/en/papers/bandarkar-2023-belebele-benchmark-parallel-reading-comprehension/)
- [Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting](https://md.research.elamur.ai/free/en/papers/huang-2023-all-languages-created-equal-llms/)
- [Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics](https://md.research.elamur.ai/free/en/papers/perrella-2024-beyond-correlation-interpretable-evaluation-machine/)
- [PEAR: Pairwise Evaluation for Automatic Relative Scoring in Machine Translation](https://md.research.elamur.ai/free/en/papers/proietti-2026-pear-pairwise-evaluation-automatic-relative/)
- [COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task](https://md.research.elamur.ai/free/en/papers/rei-2022-comet-22-unbabel-ist-2022/)
- [CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task](https://md.research.elamur.ai/free/en/papers/rei-2022-cometkiwi-ist-unbabel-2022-submission/)
- [Scaling up CometKiwi: Unbabel-IST 2023 Submission for the Quality Estimation Shared Task](https://md.research.elamur.ai/free/en/papers/rei-2023-scaling-up-cometkiwi-unbabel-ist/)
- [MTQE.en-he: Machine Translation Quality Estimation for English-Hebrew](https://md.research.elamur.ai/free/en/papers/rosenbaum-2026-mtqe-en-he-machine-translation/)
- [XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation](https://md.research.elamur.ai/free/en/papers/ruder-2021-xtreme-r-more-challenging-nuanced/)
- [Hurdles of Automatic Metric for Speech Translation Evaluation](https://md.research.elamur.ai/free/en/papers/zarzu-2026-hurdles-automatic-metric-speech-translation/)
- [Findings of the WMT 2022 Shared Task on Quality Estimation](https://md.research.elamur.ai/free/en/papers/zerva-2022-findings-wmt-2022-shared-task/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{chi2020infoxlm,
  title = {{InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training}},
  author = {Chi, Zewen and Dong, Li and Wei, Furu and Yang, Nan and Singhal, Saksham and Wang, Wenhui and Song, Xia and Mao, Xian-Ling and Huang, Heyan and Zhou, Ming},
  year = {2020},
  doi = {10.48550/arxiv.2007.07834},
  eprint = {2007.07834},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2007.07834}
}
```

- [← previous Information-Theoretic Probing with Minimum Description Length](https://md.research.elamur.ai/free/en/papers/voita-2020-information-theoretic-probing-minimum-description/)
- [next → Inherent Dependency Displacement Bias of Transition-Based Algorithms](https://md.research.elamur.ai/free/en/papers/ud-2020-anderson-inherent-dependency-displacement-bias-transition/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/)

## References

41 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. 2019. A theoretical analysis of contrastive unsupervised representation learning. In International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Sanjeev+Arora%2C+Hrishikesh+Khandeparkar%2C+Mikhail+Khodak%2C+Orestis+Plevrakis%2C+and+Nikunj+Saunshi%2E+2019%2E+A+theoretical+analysis+of+contrastive+unsupervised+representation+learning%2E+In+International+Confer)
2. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 4623–4637. Association for Computational… [find](https://scholar.google.com/scholar?q=Mikel+Artetxe%2C+Sebastian+Ruder%2C+and+Dani+Yogatama%2E+2020%2E+On+the+cross%2Dlingual+transferability+of+monolingual+representations%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computati) [our card](https://md.research.elamur.ai/free/en/papers/artetxe-2020-cross-lingual-transferability-monolingual-representations/)
3. Mikel Artetxe and Holger Schwenk. 2019. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7(0):597–610. [find](https://scholar.google.com/scholar?q=Mikel+Artetxe+and+Holger+Schwenk%2E+2019%2E+Massively+multilingual+sentence+embeddings+for+zero%2Dshot+cross%2Dlingual+transfer+and+beyond%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+7%280%29%3A) [our card](https://md.research.elamur.ai/free/en/papers/artetxe-2018-massively-multilingual-sentence-embeddings-zero/)
4. Philip Bachman, R Devon Hjelm, and William Buchwalter. 2019. Learning representations by maximizing mutual information across views. In Advances in Neural Information Processing Systems, pages 15509–15519. [find](https://scholar.google.com/scholar?q=Philip+Bachman%2C+R+Devon+Hjelm%2C+and+William+Buchwalter%2E+2019%2E+Learning+representations+by+maximizing+mutual+information+across+views%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+15509%E2%80%931)
5. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597–1607. PMLR. [find](https://scholar.google.com/scholar?q=Ting+Chen%2C+Simon+Kornblith%2C+Mohammad+Norouzi%2C+and+Geoffrey+Hinton%2E+2020%2E+A+simple+framework+for+contrastive+learning+of+visual+representations%2E+In+Proceedings+of+the+37th+International+Conference+on+M) [our card](https://md.research.elamur.ai/free/en/papers/chen-2020-simple-framework-contrastive-learning-visual/)
6. Zewen Chi, Li Dong, Furu Wei, Wenhui Wang, Xian-Ling Mao, and Heyan Huang. 2020. Cross-lingual natural language generation via pre-training. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7570–7577. AAAI Press. [find](https://scholar.google.com/scholar?q=Zewen+Chi%2C+Li+Dong%2C+Furu+Wei%2C+Wenhui+Wang%2C+Xian%2DLing+Mao%2C+and+Heyan+Huang%2E+2020%2E+Cross%2Dlingual+natural+language+generation+via+pre%2Dtraining%2E+In+The+Thirty%2DFourth+AAAI+Conference+on+Artificial+Intellig)
7. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020a. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzma%CC%81n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020a%2E+Unsupervised+cross%2Dlin)
8. Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems, pages 7057–7067. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Alexis+Conneau+and+Guillaume+Lample%2E+2019%2E+Cross%2Dlingual+language+model+pretraining%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+7057%E2%80%937067%2E+Curran+Associates%2C+Inc%2E)
9. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Bel… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Ruty+Rinott%2C+Guillaume+Lample%2C+Adina+Williams%2C+Samuel+Bowman%2C+Holger+Schwenk%2C+and+Veselin+Stoyanov%2E+2018%2E+XNLI%3A+Evaluating+cross%2Dlingual+sentence+representations%2E+In+Proceedings+of+the)
10. Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020b. Emerging cross-lingual structure in pretrained language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6022–6034, Online. Association for Computational Ling… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Shijie+Wu%2C+Haoran+Li%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020b%2E+Emerging+cross%2Dlingual+structure+in+pretrained+language+models%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+As) [our card](https://md.research.elamur.ai/free/en/papers/conneau-2020-emerging-cross-lingual-structure-pretrained/)
11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+N)
12. Philipp Dufter and Hinrich Schütze. 2020. Identifying elements essential for BERT’s multilinguality. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4423–4437, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Philipp+Dufter+and+Hinrich+Schu%CC%88tze%2E+2020%2E+Identifying+elements+essential+for+BERT%E2%80%99s+multilinguality%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C)
13. Robert M. Fano. 1963. Transmission of Information: A Statistical Theory of Communications. M.I.T. Press. [find](https://scholar.google.com/scholar?q=Robert+M%2E+Fano%2E+1963%2E+Transmission+of+Information%3A+A+Statistical+Theory+of+Communications%2E+M%2EI%2ET%2E+Press%2E)
14. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2020. Language-agnostic BERT sentence embedding. arXiv preprint arXiv:2007.01852. [find](https://scholar.google.com/scholar?q=Fangxiaoyu+Feng%2C+Yinfei+Yang%2C+Daniel+Cer%2C+Naveen+Arivazhagan%2C+and+Wei+Wang%2E+2020%2E+Language%2Dagnostic+BERT+sentence+embedding%2E+arXiv+preprint+arXiv%3A2007%2E01852%2E) [our card](https://md.research.elamur.ai/free/en/papers/feng-2020-language-agnostic-bert-sentence-embedding/)
15. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Haoqi+Fan%2C+Yuxin+Wu%2C+Saining+Xie%2C+and+Ross+Girshick%2E+2020%2E+Momentum+contrast+for+unsupervised+visual+representation+learning%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+a) [our card](https://md.research.elamur.ai/free/en/papers/he-2019-momentum-contrast-unsupervised-visual-representation/)
16. Olivier Henaff. 2020. Data-efficient image recognition with contrastive predictive coding. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4182–4192. PMLR. [find](https://scholar.google.com/scholar?q=Olivier+Henaff%2E+2020%2E+Data%2Defficient+image+recognition+with+contrastive+predictive+coding%2E+In+Proceedings+of+the+37th+International+Conference+on+Machine+Learning%2C+volume+119+of+Proceedings+of+Machine)
17. R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. 2019. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=R+Devon+Hjelm%2C+Alex+Fedorov%2C+Samuel+Lavoie%2DMarchildon%2C+Karan+Grewal%2C+Phil+Bachman%2C+Adam+Trischler%2C+and+Yoshua+Bengio%2E+2019%2E+Learning+deep+representations+by+mutual+information+estimation+and+maximizat)
18. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of… [find](https://scholar.google.com/scholar?q=Junjie+Hu%2C+Sebastian+Ruder%2C+Aditya+Siddhant%2C+Graham+Neubig%2C+Orhan+Firat%2C+and+Melvin+Johnson%2E+2020%2E+XTREME%3A+A+massively+multilingual+multi%2Dtask+benchmark+for+evaluating+cross%2Dlingual+generalisation%2E+In)
19. Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou. 2019. Unicoder: A universal language encoder by pre-training with multiple cross-lingual tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Internationa… [find](https://scholar.google.com/scholar?q=Haoyang+Huang%2C+Yaobo+Liang%2C+Nan+Duan%2C+Ming+Gong%2C+Linjun+Shou%2C+Daxin+Jiang%2C+and+Ming+Zhou%2E+2019%2E+Unicoder%3A+A+universal+language+encoder+by+pre%2Dtraining+with+multiple+cross%2Dlingual+tasks%2E+In+Proceedings)
20. Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. Cross-lingual ability of multilingual bert: An empirical study. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Karthikeyan+K%2C+Zihan+Wang%2C+Stephen+Mayhew%2C+and+Dan+Roth%2E+2020%2E+Cross%2Dlingual+ability+of+multilingual+bert%3A+An+empirical+study%2E+In+International+Conference+on+Learning+Representations%2E)
21. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, San Diego, CA. [find](https://scholar.google.com/scholar?q=Diederik+P%2E+Kingma+and+Jimmy+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+3rd+International+Conference+on+Learning+Representations%2C+San+Diego%2C+CA%2E)
22. Lingpeng Kong, Cyprien de Masson d’Autume, Lei Yu, Wang Ling, Zihang Dai, and Dani Yogatama. 2020. A mutual information maximization perspective of language representation learning. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Lingpeng+Kong%2C+Cyprien+de+Masson+d%E2%80%99Autume%2C+Lei+Yu%2C+Wang+Ling%2C+Zihang+Dai%2C+and+Dani+Yogatama%2E+2020%2E+A+mutual+information+maximization+perspective+of+language+representation+learning%2E+In+International+C)
23. Anoop Kunchukuttan, Pratik Mehta, and Pushpak Bhattacharyya. 2018. The IIT Bombay English-Hindi parallel corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, Miyazaki, Japan. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Anoop+Kunchukuttan%2C+Pratik+Mehta%2C+and+Pushpak+Bhattacharyya%2E+2018%2E+The+IIT+Bombay+English%2DHindi+parallel+corpus%2E+In+Proceedings+of+the+Eleventh+International+Conference+on+Language+Resources+and+Evalu) [our card](https://md.research.elamur.ai/free/en/papers/kunchukuttan-2018-iit-bombay-english-hindi-parallel/)
24. Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020. MLQA: Evaluating cross-lingual extractive question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7315– 7330, Online. Association for Computational Lingu… [find](https://scholar.google.com/scholar?q=Patrick+Lewis%2C+Barlas+Oguz%2C+Ruty+Rinott%2C+Sebastian+Riedel%2C+and+Holger+Schwenk%2E+2020%2E+MLQA%3A+Evaluating+cross%2Dlingual+extractive+question+answering%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Asso) [our card](https://md.research.elamur.ai/free/en/papers/lewis-2019-mlqa-evaluating-cross-lingual-extractive/)
25. Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Mi… [find](https://scholar.google.com/scholar?q=Yaobo+Liang%2C+Nan+Duan%2C+Yeyun+Gong%2C+Ning+Wu%2C+Fenfei+Guo%2C+Weizhen+Qi%2C+Ming+Gong%2C+Linjun+Shou%2C+Daxin+Jiang%2C+Guihong+Cao%2C+Xiaodong+Fan%2C+Ruofei+Zhang%2C+Rahul+Agrawal%2C+Edward+Cui%2C+Sining+Wei%2C+Taroon+Bharti%2C+)
26. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8(0):726–742. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Jiatao+Gu%2C+Naman+Goyal%2C+Xian+Li%2C+Sergey+Edunov%2C+Marjan+Ghazvininejad%2C+Mike+Lewis%2C+and+Luke+Zettlemoyer%2E+2020%2E+Multilingual+denoising+pre%2Dtraining+for+neural+machine+translation%2E+Transactio)
27. Xuanlong Nguyen, Martin J Wainwright, and Michael I Jordan. 2010. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56(11):5847–5861. [find](https://scholar.google.com/scholar?q=Xuanlong+Nguyen%2C+Martin+J+Wainwright%2C+and+Michael+I+Jordan%2E+2010%2E+Estimating+divergence+functionals+and+the+likelihood+ratio+by+convex+risk+minimization%2E+IEEE+Transactions+on+Information+Theory%2C+56%2811)
28. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. [find](https://scholar.google.com/scholar?q=Aaron+van+den+Oord%2C+Yazhe+Li%2C+and+Oriol+Vinyals%2E+2018%2E+Representation+learning+with+contrastive+predictive+coding%2E+arXiv+preprint+arXiv%3A1807%2E03748%2E) [our card](https://md.research.elamur.ai/free/en/papers/oord-2018-representation-learning-contrastive-predictive-coding/)
29. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996– 5001, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Telmo+Pires%2C+Eva+Schlinger%2C+and+Dan+Garrette%2E+2019%2E+How+multilingual+is+multilingual+BERT%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+4996%E2%80%93+5001%2C) [our card](https://md.research.elamur.ai/free/en/papers/pires-2019-how-multilingual-is-multilingual-bert/)
30. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+2016%2E+SQuAD%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natu) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
31. Masoud Jalili Sabet, Philipp Dufter, and Hinrich Schütze. 2020. Simalign: High quality word alignments without parallel training data using static and contextualized embeddings. arXiv preprint arXiv:2004.08728. [find](https://scholar.google.com/scholar?q=Masoud+Jalili+Sabet%2C+Philipp+Dufter%2C+and+Hinrich+Schu%CC%88tze%2E+2020%2E+Simalign%3A+High+quality+word+alignments+without+parallel+training+data+using+static+and+contextualized+embeddings%2E+arXiv+preprint+arXiv%3A)
32. Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2019. Wiki-Matrix: Mining 135M parallel sentences in 1620 language pairs from wikipedia. arXiv preprint arXiv:1907.05791. [find](https://scholar.google.com/scholar?q=Holger+Schwenk%2C+Vishrav+Chaudhary%2C+Shuo+Sun%2C+Hongyu+Gong%2C+and+Francisco+Guzma%CC%81n%2E+2019%2E+Wiki%2DMatrix%3A+Mining+135M+parallel+sentences+in+1620+language+pairs+from+wikipedia%2E+arXiv+preprint+arXiv%3A1907%2E0579) [our card](https://md.research.elamur.ai/free/en/papers/schwenk-2019-wikimatrix-mining-135m-parallel-sentences/)
33. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019. MASS: Masked sequence to sequence pre-training for language generation. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5926–5936. PMLR. [find](https://scholar.google.com/scholar?q=Kaitao+Song%2C+Xu+Tan%2C+Tao+Qin%2C+Jianfeng+Lu%2C+and+Tie%2DYan+Liu%2E+2019%2E+MASS%3A+Masked+sequence+to+sequence+pre%2Dtraining+for+language+generation%2E+In+Proceedings+of+the+36th+International+Conference+on+Machine) [our card](https://md.research.elamur.ai/free/en/papers/song-2019-mass-masked-sequence-sequence-pre/)
34. Yonglong Tian, Dilip Krishnan, and Phillip Isola. 2019. Contrastive multiview coding. ArXiv, abs/1906.05849. — [arXiv:1906.05849](https://arxiv.org/abs/1906.05849) [find](https://scholar.google.com/scholar?q=Yonglong+Tian%2C+Dilip+Krishnan%2C+and+Phillip+Isola%2E+2019%2E+Contrastive+multiview+coding%2E+ArXiv%2C+abs%2F1906%2E05849%2E+%E2%80%94+%5BarXiv%3A1906%2E05849%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1906%2E05849%29)
35. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation, pages 2214–2218, Istanbul, Turkey. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Jo%CC%88rg+Tiedemann%2E+2012%2E+Parallel+data%2C+tools+and+interfaces+in+OPUS%2E+In+Proceedings+of+the+Eighth+International+Conference+on+Language+Resources+and+Evaluation%2C+pages+2214%E2%80%932218%2C+Istanbul%2C+Turkey%2E+Europ)
36. Naftali Tishby and Noga Zaslavsky. 2015. Deep learning and the information bottleneck principle. 2015 IEEE Information Theory Workshop (ITW), pages 1– 5. [find](https://scholar.google.com/scholar?q=Naftali+Tishby+and+Noga+Zaslavsky%2E+2015%2E+Deep+learning+and+the+information+bottleneck+principle%2E+2015+IEEE+Information+Theory+Workshop+%28ITW%29%2C+pages+1%E2%80%93+5%2E)
37. T. Tsujishita. 1995. On triple mutual information. Advances in Applied Mathematics, 16(3):269 – 274. [find](https://scholar.google.com/scholar?q=T%2E+Tsujishita%2E+1995%2E+On+triple+mutual+information%2E+Advances+in+Applied+Mathematics%2C+16%283%29%3A269+%E2%80%93+274%2E)
38. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+Advances+in+Neural+Information+Processi)
39. Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 833–844, Hong Kong, Chin… [find](https://scholar.google.com/scholar?q=Shijie+Wu+and+Mark+Dredze%2E+2019%2E+Beto%2C+bentz%2C+becas%3A+The+surprising+cross%2Dlingual+effectiveness+of+BERT%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+t) [our card](https://md.research.elamur.ai/free/en/papers/wu-2019-beto-bentz-becas-surprising-cross/)
40. Jian Yang, Shuming Ma, Dongdong Zhang, Shuangzhi Wu, Zhoujun Li, and Ming Zhou. 2020. Alternating language modeling for cross-lingual pre-training. In Thirty-Fourth AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Jian+Yang%2C+Shuming+Ma%2C+Dongdong+Zhang%2C+Shuangzhi+Wu%2C+Zhoujun+Li%2C+and+Ming+Zhou%2E+2020%2E+Alternating+language+modeling+for+cross%2Dlingual+pre%2Dtraining%2E+In+Thirty%2DFourth+AAAI+Conference+on+Artificial+Intel)
41. Michał Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. The united nations parallel corpus v1. 0. In LREC, pages 3530–3534. [find](https://scholar.google.com/scholar?q=Micha%C5%82+Ziemski%2C+Marcin+Junczys%2DDowmunt%2C+and+Bruno+Pouliquen%2E+2016%2E+The+united+nations+parallel+corpus+v1%2E+0%2E+In+LREC%2C+pages+3530%E2%80%933534%2E) [our card](https://md.research.elamur.ai/free/en/papers/ziemski-2016-united-nations-parallel-corpus-v1/)