---
title: "SONAR — paper by Duquenne et al. (2023) — summary and ideas"
description: "The authors first train a text encoder-decoder covering 200 languages, initialized from the NLLB 1B… 33 references, cited by 79 papers here."
authors: ["Paul-Ambroise Duquenne", "Holger Schwenk", "Benoit Sagot"]
year: 2023
venue: "arXiv:2308.11466"
source: "https://arxiv.org/abs/2308.11466"
doi: "10.48550/arxiv.2308.11466"
arxiv: "2308.11466"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/duquenne-2023-sonar-sentence-level-multimodal-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/duquenne-2023-sonar-sentence-level-multimodal-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/) / SONAR: Sentence-Level Multimodal and Language-Agnostic Representations

# SONAR: Sentence-Level Multimodal and Language-Agnostic Representations

**Authors:** [Paul-Ambroise Duquenne](https://md.research.elamur.ai/free/en/authors/paul-ambroise-duquenne/), [Holger Schwenk](https://md.research.elamur.ai/free/en/authors/holger-schwenk/) and [Benoit Sagot](https://md.research.elamur.ai/free/en/authors/benoit-sagot/) · **Year:** 2023 · **Venue:** arXiv:2308.11466 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2308.11466) · [PDF](https://arxiv.org/pdf/2308.11466) · [DOI 10.48550/arxiv.2308.11466](https://doi.org/10.48550/arxiv.2308.11466) · [arXiv 2308.11466](https://arxiv.org/abs/2308.11466) · [OpenAlex](https://openalex.org/W4386185606)

**Citations:** 5 citations (OpenAlex, 2026-10-14) · [79](https://md.research.elamur.ai/free/en/papers/duquenne-2023-sonar-sentence-level-multimodal-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
SONAR addresses the need for a single fixed-size sentence embedding space that works across many languages and across both text and speech. The authors first train a text encoder-decoder covering 200 languages, initialized from the NLLB 1B dense model. Instead of token-level cross-attention, the decoder sees only one pooled sentence vector. They compare several training signals: translation, auto-encoding, denoising auto-encoding, and an MSE loss that pulls translations together in embedding space. Next, they distill this text space into speech by training language-family speech encoders, initialized from w2v-bert, as students of the text teacher, using only transcription data.

The resulting text encoder clearly beats LASER3 and LaBSE on the xsim and xsim++ similarity-search benchmarks. The decoder delivers text translation close to NLLB 1B despite the single-vector bottleneck. Pairing the speech encoders with the text decoder gives zero-shot speech-to-text translation that rivals Whisper. This matters because one shared space supports bitext mining, cross-modal retrieval and translation into many languages, including low-resource ones. All models are openly released.

## Key points
- **Coverage and release:** text encoders and decoders cover 200 languages, and speech encoders cover 37 languages. The models are published at github.com/facebookresearch/SONAR and were ported to fairseq2.
- **Effect of training objectives:**
  - Translation is the main objective for producing language-agnostic embeddings.
  - Adding an MSE alignment loss improves cross-lingual similarity.
  - Denoising auto-encoding helps decoding, but in excess it weakens language-agnosticity.
- **Text similarity search:** a single SONAR text encoder substantially outperforms LASER3 and LaBSE on xsim and xsim++.
- **Speech-to-English translation:** on FLEURS (spBLEU, zero-shot, trained on ASR transcriptions only), SONAR averages 25.3 versus 24.5 for Whisper large v2.
  - SONAR is weaker on Mandarin, German and French.
  - It is stronger on Spanish, Dutch, Swahili, Uzbek and Indian languages.
  - Ukrainian is included. The PDF-extracted table appears to give about 30.7 for SONAR versus 32.5 for Whisper, which should be verified against the original.
- **Same-language decoding:** decoding speech embeddings into text in the same language acts as zero-shot transcription. SONAR trails Whisper on BLEU because it often paraphrases, but the gap is much smaller on BERTScore.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Institut national de recherche en sciences et technologies du numérique](https://md.research.elamur.ai/free/en/labs/institut-national-de-recherche-en-sciences-et-technologies-du-numerique/) — Research institute, Le Chesnay-Rocquencourt, France · 85 works · 25,498 citations
- [ALMANACH: Modélisation et analyse linguistique automatique et humanités computationnelles](https://md.research.elamur.ai/free/en/labs/almanach-modelisation-et-analyse-linguistique-automatique-et-humanites/) — Research institute, Paris, France · 20 works · 7,008 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 811 works

## Cite

Cite as: Paul-Ambroise Duquenne, Holger Schwenk and Benoit Sagot. 2023. SONAR: Sentence-Level Multimodal and Language-Agnostic Representations. arXiv:2308.11466.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{duquenne2023sonar,
  title = {{SONAR: Sentence-Level Multimodal and Language-Agnostic Representations}},
  author = {Paul-Ambroise Duquenne, Holger Schwenk and Benoit Sagot},
  year = {2023},
  doi = {10.48550/arxiv.2308.11466},
  eprint = {2308.11466},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.11466}
}
```

- [← previous Selectively Answering Ambiguous Questions](https://md.research.elamur.ai/free/en/papers/cole-2023-selectively-answering-ambiguous-questions/)
- [next → Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples](https://md.research.elamur.ai/free/en/papers/saparov-2023-testing-general-deductive-reasoning-capacity/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/)

## References

33 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. 2019. [Common voice: A massively-multilingual speech corpus](https://md.research.elamur.ai/free/en/papers/ardila-2019-common-voice-massively-multilingual-speech/). arXiv preprint arXiv:1912.06670.
2. Mikel Artetxe and Holger Schwenk. 2019. [Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond](https://md.research.elamur.ai/free/en/papers/artetxe-2018-massively-multilingual-sentence-embeddings-zero/). TACL, pages 597–610.
3. Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, et al. 2021. [Xlsr: Self-supervised cross-lingual speech representation learning at scale](https://md.research.elamur.ai/free/en/papers/babu-2021-xls-r-self-supervised-cross/). arXiv preprint arXiv:2111.09296.
4. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. [wav2vec 2.0: A framework for self-supervised learning of speech representations](https://md.research.elamur.ai/free/en/papers/baevski-2020-wav2vec-2-0-framework-self/). Advances in Neural Information Processing Systems, 33:12449– 12460.
5. Ankur Bapna, Colin Cherry, Yu Zhang, Ye Jia, Melvin Johnson, Yong Cheng, Simran Khanuja, Jason Riesa, and Alexis Conneau. 2022. mslam: Massively multilingual joint pre-training for speech and text. arXiv preprint arXiv:2202.01374. [find](https://scholar.google.com/scholar?q=Ankur+Bapna%2C+Colin+Cherry%2C+Yu+Zhang%2C+Ye+Jia%2C+Melvin+Johnson%2C+Yong+Cheng%2C+Simran+Khanuja%2C+Jason+Riesa%2C+and+Alexis+Conneau%2E+2022%2E+mslam%3A+Massively+multilingual+joint+pre%2Dtraining+for+speech+and+text%2E+ar)
6. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. [Universal sentence encoder](https://md.research.elamur.ai/free/en/papers/cer-2018-universal-sentence-encoder/). arXiv preprint arXiv:1803.11175.
7. Mingda Chen, Kevin Heffernan, Onur Çelebi, Alex Mourachko, and Holger Schwenk. 2023. xsim++: An improved proxy to bitext mining performance for low-resource languages. arXiv preprint arXiv:2306.12907. [find](https://scholar.google.com/scholar?q=Mingda+Chen%2C+Kevin+Heffernan%2C+Onur+%C3%87elebi%2C+Alex+Mourachko%2C+and+Holger+Schwenk%2E+2023%2E+xsim%2B%2B%3A+An+improved+proxy+to+bitext+mining+performance+for+low%2Dresource+languages%2E+arXiv+preprint+arXiv%3A2306%2E12907%2E)
8. Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. 2021. W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training. In 2021 [find](https://scholar.google.com/scholar?q=Yu%2DAn+Chung%2C+Yu+Zhang%2C+Wei+Han%2C+Chung%2DCheng+Chiu%2C+James+Qin%2C+Ruoming+Pang%2C+and+Yonghui+Wu%2E+2021%2E+W2v%2Dbert%3A+Combining+contrastive+learning+and+masked+language+modeling+for+self%2Dsupervised+speech+pre%2Dtr)
9. IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 244–250. IEEE. [find](https://scholar.google.com/scholar?q=IEEE+Automatic+Speech+Recognition+and+Understanding+Workshop+%28ASRU%29%2C+pages+244%E2%80%93250%2E+IEEE%2E)
10. Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555. [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Minh%2DThang+Luong%2C+Quoc+V+Le%2C+and+Christopher+D+Manning%2E+2020%2E+Electra%3A+Pre%2Dtraining+text+encoders+as+discriminators+rather+than+generators%2E+arXiv+preprint+arXiv%3A2003%2E10555%2E)
11. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [Unsupervised cross-lingual representation learning at scale](https://md.research.elamur.ai/free/en/papers/conneau-2020-unsupervised-cross-lingual-representation-learning/). arXiv preprint arXiv:1911.02116.
12. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. [Unsupervised cross-lingual representation learning at scale](https://md.research.elamur.ai/free/en/papers/conneau-2020-unsupervised-cross-lingual-representation-learning/). In ACL.
13. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. [Supervised learning of universal sentence representations from natural language inference data](https://md.research.elamur.ai/free/en/papers/conneau-2017-supervised-learning-universal-sentence-representations/). arXiv preprint arXiv:1705.02364.
14. Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. 2023. [Fleurs: Few-shot learning evaluation of universal representations of speech](https://md.research.elamur.ai/free/en/papers/conneau-2022-fleurs-few-shot-learning-evaluation/). In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 798–805. IEEE.
15. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805.
16. Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019. [MuST-C: a Multilingual Speech Translation Corpus](https://md.research.elamur.ai/free/en/papers/gangi-2019-must-c-multilingual-speech-translation/). In NAACL, pages 2012–2017.
17. Paul-Ambroise Duquenne, Hongyu Gong, Ning Dong, Jingfei Du, Ann Lee, Vedanuj Goswani, Changhan Wang, Juan Pino, Benoît Sagot, and Holger Schwenk. 2022a. Speechmatrix: A large-scale mined corpus of multilingual speech-to-speech translations. arXiv preprint arXiv:2211.04508. [find](https://scholar.google.com/scholar?q=Paul%2DAmbroise+Duquenne%2C+Hongyu+Gong%2C+Ning+Dong%2C+Jingfei+Du%2C+Ann+Lee%2C+Vedanuj+Goswani%2C+Changhan+Wang%2C+Juan+Pino%2C+Beno%C3%AEt+Sagot%2C+and+Holger+Schwenk%2E+2022a%2E+Speechmatrix%3A+A+large%2Dscale+mined+corpus+of+mul)
18. Paul-Ambroise Duquenne, Hongyu Gong, Benoît Sagot, and Holger Schwenk. 2022b. T-modules: Translation modules for zero-shot cross-modal machine translation. arXiv preprint arXiv:2205.12216. [find](https://scholar.google.com/scholar?q=Paul%2DAmbroise+Duquenne%2C+Hongyu+Gong%2C+Beno%C3%AEt+Sagot%2C+and+Holger+Schwenk%2E+2022b%2E+T%2Dmodules%3A+Translation+modules+for+zero%2Dshot+cross%2Dmodal+machine+translation%2E+arXiv+preprint+arXiv%3A2205%2E12216%2E)
19. Paul-Ambroise Duquenne, Hongyu Gong, and Holger Schwenk. 2021. Multimodal and multilingual embeddings for large-scale speech mining. Advances in Neural Information Processing Systems, 34. [find](https://scholar.google.com/scholar?q=Paul%2DAmbroise+Duquenne%2C+Hongyu+Gong%2C+and+Holger+Schwenk%2E+2021%2E+Multimodal+and+multilingual+embeddings+for+large%2Dscale+speech+mining%2E+Advances+in+Neural+Information+Processing+Systems%2C+34%2E)
20. Paul-Ambroise Duquenne, Holger Schwenk, and Benoît Sagot. 2023. Modular speech-to-text translation for zero-shot cross-modal transfer. In Interspeech. [find](https://scholar.google.com/scholar?q=Paul%2DAmbroise+Duquenne%2C+Holger+Schwenk%2C+and+Beno%C3%AEt+Sagot%2E+2023%2E+Modular+speech%2Dto%2Dtext+translation+for+zero%2Dshot+cross%2Dmodal+transfer%2E+In+Interspeech%2E)
21. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2020. [Language-agnostic bert sentence embedding](https://md.research.elamur.ai/free/en/papers/feng-2020-language-agnostic-bert-sentence-embedding/). arXiv preprint arXiv:2007.01852.
22. Kevin Heffernan, Onur Çelebi, and Holger Schwenk. 2022. [Bitext mining using distilled sentence representations for low-resource languages](https://md.research.elamur.ai/free/en/papers/heffernan-2022-bitext-mining-distilled-sentence-representations/). arXiv preprint arXiv:2205.12654.
23. Sameer Khurana, Antoine Laurent, and James Glass. 2022. Samu-xlsr: Semantically-aligned multimodal utterance-level cross-lingual speech representation. arXiv preprint arXiv:2205.08180. — [arXiv:2205.08180](https://arxiv.org/abs/2205.08180) [find](https://scholar.google.com/scholar?q=Sameer+Khurana%2C+Antoine+Laurent%2C+and+James+Glass%2E+2022%2E+Samu%2Dxlsr%3A+Semantically%2Daligned+multimodal+utterance%2Dlevel+cross%2Dlingual+speech+representation%2E+arXiv+preprint+arXiv%3A2205%2E08180%2E+%E2%80%94+%5BarXiv%3A2205%2E0)
24. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Jiatao+Gu%2C+Naman+Goyal%2C+Xian+Li%2C+Sergey+Edunov%2C+Marjan+Ghazvininejad%2C+Mike+Lewis%2C+and+Luke+Zettlemoyer%2E+2020%2E+Multilingual+denoising+pre%2Dtraining+for+neural+machine+translation%2E+Transactio)
25. Marta R NLLB Team, Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. [find](https://scholar.google.com/scholar?q=Marta+R+NLLB+Team%2C+Costa%2Djuss%C3%A0%2C+James+Cross%2C+Onur+%C3%87elebi%2C+Maha+Elbayad%2C+Kenneth+Heafield%2C+Kevin+Heffernan%2C+Elahe+Kalbassi%2C+Janice+Lam%2C+Daniel+Licht%2C+Jean+Maillard%2C+et+al%2E+2022%2E+No+language+left+behind)
26. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210. IEEE. [find](https://scholar.google.com/scholar?q=Vassil+Panayotov%2C+Guoguo+Chen%2C+Daniel+Povey%2C+and+Sanjeev+Khudanpur%2E+2015%2E+Librispeech%3A+an+asr+corpus+based+on+public+domain+audio+books%2E+In+2015+IEEE+international+conference+on+acoustics%2C+speech+and+)
27. Ngoc-Quan Pham, Jan Niehues, Thanh-Le Ha, and Alex Waibel. 2019. Improving zero-shot translation with language-independent constraints. [find](https://scholar.google.com/scholar?q=Ngoc%2DQuan+Pham%2C+Jan+Niehues%2C+Thanh%2DLe+Ha%2C+and+Alex+Waibel%2E+2019%2E+Improving+zero%2Dshot+translation+with+language%2Dindependent+constraints%2E)
28. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. [Robust speech recognition via large-scale weak supervision](https://md.research.elamur.ai/free/en/papers/radford-2022-robust-speech-recognition-large-scale/). Technical report, Technical report, OpenAI, 2022. URL [https://cdn.openai](https://cdn.openai). com/papers/whisper. pdf. [↗](https://cdn.openai)
29. Nils Reimers and Iryna Gurevych. 2019. [Sentence-bert: Sentence embeddings using siamese bert-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). arXiv preprint arXiv:1908.10084.
30. Nils Reimers and Iryna Gurevych. 2020. [Making monolingual sentence embeddings multilingual using knowledge distillation](https://md.research.elamur.ai/free/en/papers/reimers-2020-making-monolingual-sentence-embeddings-multilingual/). In EMNLP, pages 4512–4525.
31. Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin, and Angela Fan. 2021. [CCMatrix: Mining billions of high-quality parallel sentences on the web](https://md.research.elamur.ai/free/en/papers/schwenk-2019-ccmatrix-mining-billions-high-quality/). In ACL, page 6490–6500.
32. Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. 2021. [VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation](https://md.research.elamur.ai/free/en/papers/wang-2021-voxpopuli-large-scale-multilingual-speech/). In Proceedings of the 59th Annual…
33. Yinfei Yang, Gustavo Hernandez Abrego, Steve Yuan, Mandy Guo, Qinlan Shen, Daniel Cer, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2019. Improving multilingual sentence embedding using bi-directional dual encoder with additive margin softmax. arXiv preprint arXiv:1902.08564. — [arXiv:1902.08564](https://arxiv.org/abs/1902.08564) [find](https://scholar.google.com/scholar?q=Yinfei+Yang%2C+Gustavo+Hernandez+Abrego%2C+Steve+Yuan%2C+Mandy+Guo%2C+Qinlan+Shen%2C+Daniel+Cer%2C+Yun%2DHsuan+Sung%2C+Brian+Strope%2C+and+Ray+Kurzweil%2E+2019%2E+Improving+multilingual+sentence+embedding+using+bi%2Ddirectio)

*Updated 2 November 2026*