---
title: "Bitext Mining Using Distilled Sentence — Heffernan 2022"
description: "Multilingual sentence encoders that try to cover every language in one model have trouble with the long tail of low-resource languages."
authors: ["Kevin Heffernan", "Onur Çelebi", "Holger Schwenk"]
year: 2022
venue: "Findings of the Association for Computational Linguistics: EMNLP 2022"
source: "https://aclanthology.org/2022.findings-emnlp.154/"
doi: "10.18653/v1/2022.findings-emnlp.154"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/heffernan-2022-bitext-mining-distilled-sentence-representations/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/heffernan-2022-bitext-mining-distilled-sentence-representations/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/) / Bitext Mining Using Distilled Sentence Representations for Low-Resource Languages

# Bitext Mining Using Distilled Sentence Representations for Low-Resource Languages

**Authors:** [Kevin Heffernan](https://md.research.elamur.ai/free/en/authors/kevin-heffernan/), Onur Çelebi, [Holger Schwenk](https://md.research.elamur.ai/free/en/authors/holger-schwenk/) · **Year:** 2022 · **Venue:** Findings of the Association for Computational Linguistics: EMNLP 2022 · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.findings-emnlp.154/) · [PDF](https://aclanthology.org/2022.findings-emnlp.154.pdf) · [DOI 10.18653/v1/2022.findings-emnlp.154](https://doi.org/10.18653/v1/2022.findings-emnlp.154) · [OpenAlex](https://openalex.org/W4385574194)

**Citations:** 47 citations (openalex-snapshot · 2026-10-14) · [80](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Multilingual sentence encoders that try to cover every language in one model have trouble with the long tail of low-resource languages. Training is unbalanced, compute costs are high, and new scripts are hard to add. The authors instead build several smaller encoders, each for one language or a family of related languages, that all map into one shared embedding space. They do this through teacher-student distillation. The teacher, LASER2, is a retrained LASER. The students (LASER3) are 12-layer transformers initialized from scratch and trained with a cosine loss against the teacher's embeddings. Each student gets its own SPM vocabulary. Training can also add an MLM objective on monolingual text and a progressive-distillation curriculum. Because the encoders stay mutually compatible, their outputs can be used directly for bitext mining. The approach clearly beats the original LASER on FLORES and matches or exceeds LaBSE. The authors cover 50 African languages, many of which no other encoder supports. Bitexts mined for 44 of them improve NMT into English by 3.8 BLEU on average. This shows that new languages can be added cheaply and incrementally.

## Key points
- **Models:**
  - LASER2 (teacher) is a 5-layer BiLSTM producing 1024-dim embeddings, trained on 93 languages with SPM and implemented in fairseq.
  - LASER3 students have about 250M parameters each.
  - Each student is anchored with 2M English monolingual sentences plus 2M English–Spanish CCMatrix pairs.
- **Joint training on monolingual data:** combining the cosine loss with an MLM loss lowers the xsim error for Wolof from 21.1% to 5.9%.
- **Evaluation and mining:** encoders are evaluated on FLORES, which includes 44 African languages, and on Tatoeba. Mining runs against 21.5 billion English sentences with a single margin threshold of 1.06 for all languages.
- **NMT gains:**
  - Average BLEU rises from 11.0 to 14.8.
  - Somali improves the most, from 5.1 to 21.1 BLEU after adding 4.9M mined pairs.
  - 13 languages, including Akan and Wolof, stay below 5 BLEU, mostly because little monolingual data is available.
- **Release:** the training code, models and mined bitexts are freely available in the fairseq repository (nllb/laser_distillation).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/)

## Cited by

Cited by

80 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 74 not yet.

- [FINDINGS OF THE IWSLT 2023 EVALUATION CAMPAIGN](https://md.research.elamur.ai/free/en/papers/agarwal-2023-findings-iwslt-2023-evaluation-campaign/)
- [SeamlessM4T: Massively Multilingual & Multimodal Machine Translation](https://md.research.elamur.ai/free/en/papers/communication-2023-seamlessm4t-massively-multilingual-multimodal-machine/)
- [IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages](https://md.research.elamur.ai/free/en/papers/gala-2023-indictrans2-towards-high-quality-accessible/)
- [MTEB: Massive Text Embedding Benchmark](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-mteb-massive-text-embedding-benchmark/)
- [On Temperature-Constrained Non-Deterministic Machine Translation: Potential and Evaluation](https://md.research.elamur.ai/free/en/papers/wang-2026-temperature-constrained-non-deterministic-machine/)
- [Prompting Large Language Model for Machine Translation: A Case Study](https://md.research.elamur.ai/free/en/papers/zhang-2023-prompting-large-language-model-machine/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{heffernan2022bitext,
  title = {{Bitext Mining Using Distilled Sentence Representations for Low-Resource Languages}},
  author = {Heffernan, Kevin and Çelebi, Onur and Schwenk, Holger},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2022},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.154},
  url = {https://doi.org/10.18653/v1/2022.findings-emnlp.154}
}
```

- [← previous Binding Language Models in Symbolic Languages](https://md.research.elamur.ai/free/en/papers/cheng-2022-binding-language-models-symbolic-languages/)
- [next → BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage](https://md.research.elamur.ai/free/en/papers/shuster-2022-blenderbot-3-deployed-conversational-agent/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/)

## References

34 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jade Abbott and Laura Martinus. 2019. Benchmarking neural machine translation for Southern African 9 [https://github.com/facebookresearch/](https://github.com/facebookresearch/) LASER/ languages. In Proceedings of the 2019 Workshop on Widening NLP, pages 98–101, Florence, Italy. Association for Computational Linguistics. [↗](https://github.com/facebookresearch/)
2. Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George F. Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. 2019. Massively multilingual neural machine translation in the wild: Findings and challenges. http://ar… [find](https://scholar.google.com/scholar?q=Naveen+Arivazhagan%2C+Ankur+Bapna%2C+Orhan+Firat%2C+Dmitry+Lepikhin%2C+Melvin+Johnson%2C+Maxim+Krikun%2C+Mia+Xu+Chen%2C+Yuan+Cao%2C+George+F%2E+Foster%2C+Colin+Cherry%2C+Wolfgang+Macherey%2C+Zhifeng+Chen%2C+and+Yonghui+Wu%2E+201) [our card](https://md.research.elamur.ai/free/en/papers/arivazhagan-2019-massively-multilingual-neural-machine-translation/)
3. Mikel Artetxe and Holger Schwenk. 2019a. Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings. In ACL, pages 3197–3203. [find](https://scholar.google.com/scholar?q=Mikel+Artetxe+and+Holger+Schwenk%2E+2019a%2E+Margin%2Dbased+Parallel+Corpus+Mining+with+Multilingual+Sentence+Embeddings%2E+In+ACL%2C+pages+3197%E2%80%933203%2E) [our card](https://md.research.elamur.ai/free/en/papers/artetxe-2019-margin-based-parallel-corpus-mining/)
4. Mikel Artetxe and Holger Schwenk. 2019b. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. TACL, pages 597–610. [find](https://scholar.google.com/scholar?q=Mikel+Artetxe+and+Holger+Schwenk%2E+2019b%2E+Massively+multilingual+sentence+embeddings+for+zero%2Dshot+cross%2Dlingual+transfer+and+beyond%2E+TACL%2C+pages+597%E2%80%93610%2E) [our card](https://md.research.elamur.ai/free/en/papers/artetxe-2018-massively-multilingual-sentence-embeddings-zero/)
5. Paul Azunre, Salomey Osei, Salomey Addo, Lawrence Asamoah Adu-Gyamfi, Stephen Moore, Bernard Adabankah, Bernard Opoku, Clara Asare-Nyarko, Samuel Nyarko, Cynthia Amoaba, et al. 2021. NLP for ghanaian languages. arXiv preprint arXiv:2103.15475. — [arXiv:2103.15475](https://arxiv.org/abs/2103.15475) [find](https://scholar.google.com/scholar?q=Paul+Azunre%2C+Salomey+Osei%2C+Salomey+Addo%2C+Lawrence+Asamoah+Adu%2DGyamfi%2C+Stephen+Moore%2C+Bernard+Adabankah%2C+Bernard+Opoku%2C+Clara+Asare%2DNyarko%2C+Samuel+Nyarko%2C+Cynthia+Amoaba%2C+et+al%2E+2021%2E+NLP+for+ghanaian+)
6. Ankur Bapna, Isaac Caswell, Julia Kreutzer, Orhan Firat, Daan van Esch, Aditya Siddhant, Mengmeng Niu, Pallavi Baljekar, Xavier Garcia, Wolfgang Macherey, Theresa Breiner, Vera Axelrod, Jason Riesa, Yuan Cao, Mia Xu Chen, Klaus Macherey, Maxim Krikun, Pidong Wang, Alexander Gutkin, Apurva Shah, Yanp… — [arXiv:2205.03983](https://arxiv.org/abs/2205.03983) [find](https://scholar.google.com/scholar?q=Ankur+Bapna%2C+Isaac+Caswell%2C+Julia+Kreutzer%2C+Orhan+Firat%2C+Daan+van+Esch%2C+Aditya+Siddhant%2C+Mengmeng+Niu%2C+Pallavi+Baljekar%2C+Xavier+Garcia%2C+Wolfgang+Macherey%2C+Theresa+Breiner%2C+Vera+Axelrod%2C+Jason+Riesa%2C+Y)
7. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In ACL, pages 8440–8451. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzma%CC%81n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+cross%2Dling)
8. Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In NeurIPS. [find](https://scholar.google.com/scholar?q=Alexis+Conneau+and+Guillaume+Lample%2E+2019%2E+Cross%2Dlingual+language+model+pretraining%2E+In+NeurIPS%2E)
9. Raj Dabre and Aneerav Sukhoo. 2022. Morisienmt: A dataset for mauritian creole machine translation. arXiv preprint arXiv:2206.02421. — [arXiv:2206.02421](https://arxiv.org/abs/2206.02421) [find](https://scholar.google.com/scholar?q=Raj+Dabre+and+Aneerav+Sukhoo%2E+2022%2E+Morisienmt%3A+A+dataset+for+mauritian+creole+machine+translation%2E+arXiv+preprint+arXiv%3A2206%2E02421%2E+%E2%80%94+%5BarXiv%3A2206%2E02421%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2206%2E02421%29)
10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, pages 4171–4186. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+NAACL%2C+pages+4171%E2%80%934186%2E)
11. Chris Chinenye Emezue and Femi Pancrace Bonaventure Dossou. 2020. Ffr v1. 1: Fon-french neural machine translation. In Proceedings of the The Fourth Widening Natural Language Processing Workshop, pages 83–87. 2109 9 [find](https://scholar.google.com/scholar?q=Chris+Chinenye+Emezue+and+Femi+Pancrace+Bonaventure+Dossou%2E+2020%2E+Ffr+v1%2E+1%3A+Fon%2Dfrench+neural+machine+translation%2E+In+Proceedings+of+the+The+Fourth+Widening+Natural+Language+Processing+Workshop%2C+page)
12. Carlos Escolano, Marta R. Costa-jussà, José A. R. Fonollosa, and Mikel Artetxe. 2021. Multilingual machine translation: Closing the gap between shared and language-specific encoder-decoders. In EACL, pages 944–948, Online. ACL. [find](https://scholar.google.com/scholar?q=Carlos+Escolano%2C+Marta+R%2E+Costa%2Djussa%CC%80%2C+Jose%CC%81+A%2E+R%2E+Fonollosa%2C+and+Mikel+Artetxe%2E+2021%2E+Multilingual+machine+translation%3A+Closing+the+gap+between+shared+and+language%2Dspecific+encoder%2Ddecoders%2E+In+EACL)
13. Martinus, Kolawole Tajudeen, Kevin Degila, Kelechi Ogueji, Kathleen Siminyu, Julia Kreutzer, Jason Webster, Jamiil Toure Ali, Jade Z. Abbott, Iroro Orife, Ignatius Ezeani, Idris Abdulkabir Dangana, Herman Kamper, Hady Elsahar, Goodness Duru, Ghollah Kioko, Espoir Murhabazi, Elan Van Biljon, Daniel W… [find](https://scholar.google.com/scholar?q=Martinus%2C+Kolawole+Tajudeen%2C+Kevin+Degila%2C+Kelechi+Ogueji%2C+Kathleen+Siminyu%2C+Julia+Kreutzer%2C+Jason+Webster%2C+Jamiil+Toure+Ali%2C+Jade+Z%2E+Abbott%2C+Iroro+Orife%2C+Ignatius+Ezeani%2C+Idris+Abdulkabir+Dangana%2C+He) [our card](https://md.research.elamur.ai/free/en/papers/nekoto-2020-participatory-research-low-resourced-machine/)
14. Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2020. Beyond english-centric multiling… [find](https://scholar.google.com/scholar?q=Angela+Fan%2C+Shruti+Bhosale%2C+Holger+Schwenk%2C+Zhiyi+Ma%2C+Ahmed+El%2DKishky%2C+Siddharth+Goyal%2C+Mandeep+Baines%2C+Onur+Celebi%2C+Guillaume+Wenzek%2C+Vishrav+Chaudhary%2C+Naman+Goyal%2C+Tom+Birch%2C+Vitaliy+Liptchinsky%2C+S)
15. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Ari-vazhaga, and Wei Wang. 2020. Language-agnostic BERT sentence embedding. [https://arxiv.org/](https://arxiv.org/) abs/2007.01852 . [↗](https://arxiv.org/) [our card](https://md.research.elamur.ai/free/en/papers/feng-2020-language-agnostic-bert-sentence-embedding/)
16. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzman, and Angela Fan. 2021. The FLORES-101 evaluation benchmark for low-resource and multilingual machine translation. https://arxiv.org/abs/ 2106.03193 . [↗](https://arxiv.org/abs/)
17. Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. https://arxiv.org/abs/1802.06893 . [↗](https://arxiv.org/abs/1802.06893) [our card](https://md.research.elamur.ai/free/en/papers/grave-2018-learning-word-vectors-157-languages/)
18. Gilles Hacheme. 2021. English2gbe: A multilingual machine translation model for {Fon/Ewe}gbe. CoRR, abs/2112.11482. — [arXiv:2112.11482](https://arxiv.org/abs/2112.11482) [find](https://scholar.google.com/scholar?q=Gilles+Hacheme%2E+2021%2E+English2gbe%3A+A+multilingual+machine+translation+model+for+%7BFon%2FEwe%7Dgbe%2E+CoRR%2C+abs%2F2112%2E11482%2E+%E2%80%94+%5BarXiv%3A2112%2E11482%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2112%2E11482%29)
19. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020a. XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. In ICML, pages 4411–4421. [find](https://scholar.google.com/scholar?q=Junjie+Hu%2C+Sebastian+Ruder%2C+Aditya+Siddhant%2C+Graham+Neubig%2C+Orhan+Firat%2C+and+Melvin+Johnson%2E+2020a%2E+XTREME%3A+A+massively+multilingual+multi%2Dtask+benchmark+for+evaluating+cross%2Dlingual+generalization%2E+I) [our card](https://md.research.elamur.ai/free/en/papers/hu-2020-xtreme-massively-multilingual-multi-task/)
20. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020b. XTREME: a massively multilingual multi-task benchmark for evaluating cross-lingual generalization. In https://arxiv.org/abs/2003.11080 . [↗](https://arxiv.org/abs/2003.11080) [our card](https://md.research.elamur.ai/free/en/papers/hu-2020-xtreme-massively-multilingual-multi-task/)
21. Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, and Furu Wei. 2021. DeltaLM: encoder-decoder pre-training for language generation and translation by augmenting pretrained multilingual encoders. https://arxiv.org/abs/2106.13736 . — [arXiv:2106.13736](https://arxiv.org/abs/2106.13736) [↗](https://arxiv.org/abs/2106.13736)
22. Wilhelmina Nekoto, Vukosi Marivate, Tshinondiwa Matsila, Timi E. Fasubaa, Tajudeen Kolawole, Taiwo Fagbohungbe, Solomon Oluwole Akinola, Shamsuddeen Hassan Muhammad, Salomon Kabongo, Salomey Osei, Sackey Freshia, Rubungo Andre Niyongabo, Ricky Macharm, Perez Ogayo, Orevaoghene Ahia, Musie Meressa, M… [find](https://scholar.google.com/scholar?q=Wilhelmina+Nekoto%2C+Vukosi+Marivate%2C+Tshinondiwa+Matsila%2C+Timi+E%2E+Fasubaa%2C+Tajudeen+Kolawole%2C+Taiwo+Fagbohungbe%2C+Solomon+Oluwole+Akinola%2C+Shamsuddeen+Hassan+Muhammad%2C+Salomon+Kabongo%2C+Salomey+Osei%2C+Sac)
23. Gowtham Ramesh, Sumanth Doddapaneni, Aravinth Bheemaraj, Mayank Jobanputra, Raghavan AK, Ajitesh Sharma, Sujit Sahoo, Harshita Diddee, Mahalakshmi J, Divyanshu Kakwani, Navneet Kumar, Aswin Pradeep, Srihari Nagaraj, Kumar Deepak, Vivek Raghavan, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh Shantad… [find](https://scholar.google.com/scholar?q=Gowtham+Ramesh%2C+Sumanth+Doddapaneni%2C+Aravinth+Bheemaraj%2C+Mayank+Jobanputra%2C+Raghavan+AK%2C+Ajitesh+Sharma%2C+Sujit+Sahoo%2C+Harshita+Diddee%2C+Mahalakshmi+J%2C+Divyanshu+Kakwani%2C+Navneet+Kumar%2C+Aswin+Pradeep%2C+S) [our card](https://md.research.elamur.ai/free/en/papers/ramesh-2022-samanantar-largest-publicly-available-parallel/)
24. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using siamese bert-networks. In EMNLP, pages 3982–3992. [find](https://scholar.google.com/scholar?q=Nils+Reimers+and+Iryna+Gurevych%2E+2019%2E+Sentence%2DBERT%3A+Sentence+embeddings+using+siamese+bert%2Dnetworks%2E+In+EMNLP%2C+pages+3982%E2%80%933992%2E)
25. Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In EMNLP, pages 4512– 4525. [find](https://scholar.google.com/scholar?q=Nils+Reimers+and+Iryna+Gurevych%2E+2020%2E+Making+monolingual+sentence+embeddings+multilingual+using+knowledge+distillation%2E+In+EMNLP%2C+pages+4512%E2%80%93+4525%2E) [our card](https://md.research.elamur.ai/free/en/papers/reimers-2020-making-monolingual-sentence-embeddings-multilingual/)
26. Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, and Melvin Johnson. 2021. XTREME-R: Towards more challenging and nuanced multilingual evaluation. In EMNLP, pages 10215–10245. [find](https://scholar.google.com/scholar?q=Sebastian+Ruder%2C+Noah+Constant%2C+Jan+Botha%2C+Aditya+Siddhant%2C+Orhan+Firat%2C+Jinlan+Fu%2C+Pengfei+Liu%2C+Junjie+Hu%2C+Dan+Garrette%2C+Graham+Neubig%2C+and+Melvin+Johnson%2E+2021%2E+XTREME%2DR%3A+Towards+more+challenging+an) [our card](https://md.research.elamur.ai/free/en/papers/ruder-2021-xtreme-r-more-challenging-nuanced/)
27. Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin, and Angela Fan. 2021. CCMatrix: Mining billions of high-quality parallel sentences on the web. In ACL, page 6490–6500. [find](https://scholar.google.com/scholar?q=Holger+Schwenk%2C+Guillaume+Wenzek%2C+Sergey+Edunov%2C+Edouard+Grave%2C+Armand+Joulin%2C+and+Angela+Fan%2E+2021%2E+CCMatrix%3A+Mining+billions+of+high%2Dquality+parallel+sentences+on+the+web%2E+In+ACL%2C+page+6490%E2%80%936500%2E) [our card](https://md.research.elamur.ai/free/en/papers/schwenk-2019-ccmatrix-mining-billions-high-quality/)
28. Aditya Siddhant, Ankur Bapna, Orhan Firat, Yuan Cao, Mia Xu Chen, Isaac Caswell, and Xavier Garcia. 2110 10 2022. Towards the next 1000 languages in multilingual machine translation: Exploring the synergy between supervised and self-supervised learning. https://arxiv.org/abs/2201.03110 . — [arXiv:2201.03110](https://arxiv.org/abs/2201.03110) [↗](https://arxiv.org/abs/2201.03110)
29. Kathleen Siminyu, Godson Kalipe, Davor Orlic, Jade Abbott, Vukosi Marivate, Sackey Freshia, Prateek Sibal, Bhanu Neupane, David I. Adelani, Amelia Taylor, Jamiil Toure ALI, Kevin Degila, Momboladji Balogoun, Thierno Ibrahima DIOP, Davis David, Chayma Fourati, Hatem Haddad, and Malek Naski. 2021. AI4… — [arXiv:2104.02516](https://arxiv.org/abs/2104.02516) [find](https://scholar.google.com/scholar?q=Kathleen+Siminyu%2C+Godson+Kalipe%2C+Davor+Orlic%2C+Jade+Abbott%2C+Vukosi+Marivate%2C+Sackey+Freshia%2C+Prateek+Sibal%2C+Bhanu+Neupane%2C+David+I%2E+Adelani%2C+Amelia+Taylor%2C+Jamiil+Toure+ALI%2C+Kevin+Degila%2C+Momboladji+Ba)
30. A We trained our students to minimize the xsim score of each language with respect to the English LASER2 teacher. In order to best consider specificities of languages, several independent student models were trained: • Semitic languages: amh and tir • Kwa languages: aka, ewe, fon, and twi J. Tiedema… [find](https://scholar.google.com/scholar?q=A+We+trained+our+students+to+minimize+the+xsim+score+of+each+language+with+respect+to+the+English+LASER2+teacher%2E+In+order+to+best+consider+specificities+of+languages%2C+several+independent+student+mode)
31. Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. 2022. Deep-Net: scaling transformers to 1,000 layers. https://arxiv.org/abs/2203.00555 . — [arXiv:2203.00555](https://arxiv.org/abs/2203.00555) [↗](https://arxiv.org/abs/2203.00555)
32. Zihan Wang, Karthikeyan K, Stephen Mayhew, and Dan Roth. 2020. Extending multilingual BERT to low-resource languages. In EMNLP, pages 2649–2656. [find](https://scholar.google.com/scholar?q=Zihan+Wang%2C+Karthikeyan+K%2C+Stephen+Mayhew%2C+and+Dan+Roth%2E+2020%2E+Extending+multilingual+BERT+to+low%2Dresource+languages%2E+In+EMNLP%2C+pages+2649%E2%80%932656%2E)
33. Yinfei Yang, Gustavo Hernandez Abrego, Steve Yuan, Qinlan Shen Mandy Guo, Daniel Cer, Brian Strope Yun-hsuan Sun and, and Ray Kurzweil. 2019. Improving multilingual sentence embedding using bidirectional dual encoder with additive margin softmax. In IJCAI, pages 5370–5378. [find](https://scholar.google.com/scholar?q=Yinfei+Yang%2C+Gustavo+Hernandez+Abrego%2C+Steve+Yuan%2C+Qinlan+Shen+Mandy+Guo%2C+Daniel+Cer%2C+Brian+Strope+Yun%2Dhsuan+Sun+and%2C+and+Ray+Kurzweil%2E+2019%2E+Improving+multilingual+sentence+embedding+using+bidirectio)
34. Analysis of zero-short performance of multiple student encoders • Senegambian languages: wol (+bsc, cou and fuv) • remaining 43 languages [find](https://scholar.google.com/scholar?q=Analysis+of+zero%2Dshort+performance+of+multiple+student+encoders+%E2%80%A2+Senegambian+languages%3A+wol+%28%2Bbsc%2C+cou+and+fuv%29+%E2%80%A2+remaining+43+languages)