---
title: "mmBERT — paper by Marone et al. (2025) — summary and ideas"
description: "The authors present mmBERT, a ModernBERT-style encoder that uses the Gemma 2… 8.3 points. 52 references, cited by 55 papers here."
authors: ["Marc Marone", "Orion Weller", "William Fleshman", "Eugene Yang", "Dawn Lawrie", "Benjamin Van Durme"]
year: 2025
venue: "arXiv:2509.06888"
source: "https://arxiv.org/abs/2509.06888"
doi: "10.48550/arxiv.2509.06888"
arxiv: "2509.06888"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/marone-2025-mmbert-modern-multilingual-encoder-annealed/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/marone-2025-mmbert-modern-multilingual-encoder-annealed/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/) / mmBERT: A Modern Multilingual Encoder with Annealed Language Learning

# mmBERT: A Modern Multilingual Encoder with Annealed Language Learning

**Authors:** Marc Marone, [Orion Weller](https://md.research.elamur.ai/free/en/authors/orion-weller/), William Fleshman, Eugene Yang, Dawn Lawrie and [Benjamin Van Durme](https://md.research.elamur.ai/free/en/authors/benjamin-van-durme/) · **Year:** 2025 · **Venue:** arXiv:2509.06888 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2509.06888) · [PDF](https://arxiv.org/pdf/2509.06888) · [DOI 10.48550/arxiv.2509.06888](https://doi.org/10.48550/arxiv.2509.06888) · [arXiv 2509.06888](https://arxiv.org/abs/2509.06888) · [OpenAlex](https://openalex.org/W4415057273)

**Citations:** 4 citations (OpenAlex, 2026-10-14) · [55](https://md.research.elamur.ai/free/en/papers/marone-2025-mmbert-modern-multilingual-encoder-annealed/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Multilingual encoder-only models have seen little progress in recent years, and XLM-R remains the default choice despite being more than six years old. The authors present mmBERT, a ModernBERT-style encoder that uses the Gemma 2 tokenizer. It was pretrained on about 3T tokens of openly available data, drawn mainly from FineWeb2, FineWeb2-HQ, filtered DCLM, Dolma/Dolmino, ProLong and MegaWika v2.

The training recipe has three new elements. First, the masking ratio decreases over the course of training. Second, the temperature used for language sampling is annealed so that the language mix moves from favouring high-resource languages toward a flatter distribution. Third, languages are added in stages, holding back roughly 1,700 low-resource languages until the short final decay phase.

mmBERT outperforms XLM-R and other earlier multilingual encoders on classification and retrieval, for both high- and low-resource languages. It also runs considerably faster and accepts inputs of up to 8192 tokens. For languages introduced only during the decay phase, it learns quickly from very little data and even beats large proprietary LLMs on one such benchmark. Models, data and code are open-sourced.

## Key points
- **Training stages:** about 2.3T tokens of pre-training, then 600B tokens of mid-training (where context grows to 8192), then a 100B-token decay phase. Language coverage expands across these stages from 60 to 110 to 1833, and the sampling temperature τ drops from 0.7 to 0.5 to 0.3. Three decay-phase variants were trained and their weights merged.
- **Model sizes:** the base model has 307M total parameters (110M non-embedding) and the small model has 140M (42M non-embedding). Both use 22 layers and a large multilingual vocabulary.
- **Low-resource gains:** adding languages during the decay phase raised base-model F1 on TiQuAD (Tigrinya) by 68% (+12.1 F1) and on FoQA (Faroese) by 26% (+15.4 F1). On FoQA, mmBERT beats Gemini 2.5 Pro by roughly 6 points and o3 by roughly 8.3 points.
- **Speed:** mmBERT-base is more than twice as fast as earlier multilingual encoders on variable-length inputs and about 4× faster on long contexts. By contrast, XLM-R and MiniLM are capped at 512 tokens.
- **Release and Ukrainian coverage:** models, data and checkpoints are available at github.com/jhu-clsp/mmBERT. Ukrainian (ukr) appears among the FineWeb2 languages shown in the paper's sampling-distribution chart.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 467 works

## Cite

Cite as: Marc Marone, Orion Weller, William Fleshman, Eugene Yang, Dawn Lawrie and Benjamin Van Durme. 2025. mmBERT: A Modern Multilingual Encoder with Annealed Language Learning. arXiv:2509.06888.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{marone2025mmbert,
  title = {{mmBERT: A Modern Multilingual Encoder with Annealed Language Learning}},
  author = {Marc Marone, Orion Weller, William Fleshman, Eugene Yang, Dawn Lawrie and Benjamin {Van Durme}},
  year = {2025},
  doi = {10.48550/arxiv.2509.06888},
  eprint = {2509.06888},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2509.06888}
}
```

- [← previous Large Language Model Agent: A Survey on Methodology, Applications and Challenges](https://md.research.elamur.ai/free/en/papers/luo-2025-large-language-model-agent-survey/)
- [next → MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention](https://md.research.elamur.ai/free/en/papers/minimax-2025-minimax-m1-scaling-test/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/)

## References

52 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774, 2023.
2. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. [Unsupervised neural machine translation](https://md.research.elamur.ai/free/en/papers/artetxe-2017-unsupervised-neural-machine-translation/). arXiv preprint arXiv:1710.11041, 2017.
3. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. [Ms marco: A human generated machine reading comprehension dataset](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/). arXiv preprint arXiv:1611.09268, 2016.
4. Samuel Barham, Orion Weller, Michelle Yuan, Kenton Murray, Mahsa Yarmohammadi, Zhengping Jiang, Siddharth Vashishtha, Alexander Martin, Anqi Liu, Aaron Steven White, et al. Megawika: Millions of reports and their sources across 50 diverse languages. arXiv preprint arXiv:2307.07049, 2023. — [arXiv:2307.07049](https://arxiv.org/abs/2307.07049) [find](https://scholar.google.com/scholar?q=Samuel+Barham%2C+Orion+Weller%2C+Michelle+Yuan%2C+Kenton+Murray%2C+Mahsa+Yarmohammadi%2C+Zhengping+Jiang%2C+Siddharth+Vashishtha%2C+Alexander+Martin%2C+Anqi+Liu%2C+Aaron+Steven+White%2C+et+al%2E+Megawika%3A+Millions+of+repor)
5. Samuel Barham, Chandler May, and Benjamin Van Durme. Megawika 2: A more comprehensive multilingual collection of articles and their sources. arXiv preprint arXiv:2508.03828, 2025. — [arXiv:2508.03828](https://arxiv.org/abs/2508.03828) [find](https://scholar.google.com/scholar?q=Samuel+Barham%2C+Chandler+May%2C+and+Benjamin+Van+Durme%2E+Megawika+2%3A+A+more+comprehensive+multilingual+collection+of+articles+and+their+sources%2E+arXiv+preprint+arXiv%3A2508%2E03828%2C+2025%2E+%E2%80%94+%5BarXiv%3A2508%2E03828%5D)
6. Nicolas Boizard, Hippolyte Gisserot-Boukhlef, Duarte M Alves, André Martins, Ayoub Hammal, Caio Corro, Céline Hudelot, Emmanuel Malherbe, Etienne Malaboeuf, Fanny Jourdan, et al. Eurobert: scaling multilingual encoders for european languages. arXiv preprint arXiv:2503.05500, 2025. [find](https://scholar.google.com/scholar?q=Nicolas+Boizard%2C+Hippolyte+Gisserot%2DBoukhlef%2C+Duarte+M+Alves%2C+Andr%C3%A9+Martins%2C+Ayoub+Hammal%2C+Caio+Corro%2C+C%C3%A9line+Hudelot%2C+Emmanuel+Malherbe%2C+Etienne+Malaboeuf%2C+Fanny+Jourdan%2C+et+al%2E+Eurobert%3A+scaling+mul)
7. [Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,…](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
8. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. [Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities](https://md.research.elamur.ai/free/en/papers/comanici-2025-gemini-2-5-pushing-frontier/). arXiv preprint a…
9. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116, 2019. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzm%C3%A1n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+Unsupervised+cross%2Dlingual+rep)
10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [BERT: pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Comp…
11. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. [find](https://scholar.google.com/scholar?q=Abhimanyu+Dubey%2C+Abhinav+Jauhri%2C+Abhinav+Pandey%2C+Abhishek+Kadian%2C+Ahmad+Al%2DDahle%2C+Aiesha+Letman%2C+Akhil+Mathur%2C+Alan+Schelten%2C+Amy+Yang%2C+Angela+Fan%2C+et+al%2E+The+llama+3+herd+of+models%2E+arXiv+preprint+ar)
12. Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, Márton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzemiński, Genta Indra Winata, et al. [Mmteb: Massive multilingual text embedding benchmark](https://md.research.elamur.ai/free/en/papers/enevoldsen-2025-mmteb-massive-multilingual-text-embedding/). arXiv preprint arXiv:2502.13595, 2025.
13. Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48, 2021. [find](https://scholar.google.com/scholar?q=Angela+Fan%2C+Shruti+Bhosale%2C+Holger+Schwenk%2C+Zhiyi+Ma%2C+Ahmed+El%2DKishky%2C+Siddharth+Goyal%2C+Mandeep+Baines%2C+Onur+Celebi%2C+Guillaume+Wenzek%2C+Vishrav+Chaudhary%2C+et+al%2E+Beyond+english%2Dcentric+multilingual+mac)
14. Fitsum Gaim, Wonsuk Yang, Hancheol Park, and Jong C Park. Question-answering in a low-resourced language: Benchmark dataset and models for tigrinya. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11857–11870, 2023. [find](https://scholar.google.com/scholar?q=Fitsum+Gaim%2C+Wonsuk+Yang%2C+Hancheol+Park%2C+and+Jong+C+Park%2E+Question%2Danswering+in+a+low%2Dresourced+language%3A+Benchmark+dataset+and+models+for+tigrinya%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+As)
15. Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. [How to train long-context language models (effectively)](https://md.research.elamur.ai/free/en/papers/gao-2025-how-train-long-context-language/), 2024. URL https://arxiv.org/abs/2410.02660. [↗](https://arxiv.org/abs/2410.02660)
16. Team Gemma, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024. [find](https://scholar.google.com/scholar?q=Team+Gemma%2C+Thomas+Mesnard%2C+Cassidy+Hardin%2C+Robert+Dadashi%2C+Surya+Bhupatiraju%2C+Shreya+Pathak%2C+Laurent+Sifre%2C+Morgane+Rivi%C3%A8re%2C+Mihir+Sanjay+Kale%2C+Juliette+Love%2C+et+al%2E+Gemma%3A+Open+models+based+on+gemin)
17. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton T… [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Morgane+Riviere%2C+Shreya+Pathak%2C+Pier+Giuseppe+Sessa%2C+Cassidy+Hardin%2C+Surya+Bhupatiraju%2C+L%C3%A9onard+Hussenot%2C+Thomas+Mesnard%2C+Bobak+Shahriari%2C+Alexandre+Ram%C3%A9%2C+Johan+Ferret%2C+Peter+Liu%2C+Pouya+Ta)
18. Hippolyte Gisserot-Boukhlef, Nicolas Boizard, Manuel Faysse, Duarte M. Alves, Emmanuel Malherbe, André F. T. Martins, Céline Hudelot, and Pierre Colombo. Should we still pretrain encoders with masked language modeling?, 2025. URL https://arxiv.org/abs/2507.00994. — [arXiv:2507.00994](https://arxiv.org/abs/2507.00994) [↗](https://arxiv.org/abs/2507.00994)
19. Pengcheng He, Jianfeng Gao, and Weizhu Chen. [Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing](https://md.research.elamur.ai/free/en/papers/he-2021-debertav3-improving-deberta-using-electra/). In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023.
20. OpenReview.net, 2023. URL [https://openreview.net/forum?id=sE7-XhLxHA](https://openreview.net/forum?id=sE7-XhLxHA). [↗](https://openreview.net/forum?id=sE7-XhLxHA)
21. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In International conference on machine learning, pp. 4411–4421. PMLR, 2020. [find](https://scholar.google.com/scholar?q=Junjie+Hu%2C+Sebastian+Ruder%2C+Aditya+Siddhant%2C+Graham+Neubig%2C+Orhan+Firat%2C+and+Melvin+Johnson%2E+Xtreme%3A+A+massively+multilingual+multi%2Dtask+benchmark+for+evaluating+cross%2Dlingual+generalisation%2E+In+Inter)
22. Lola Le Breton, Quentin Fournier, Mariam El Mezouar, and Sarath Chandar. Neobert: A next generation bert. Transactions on Machine Learning Research, 2025. [find](https://scholar.google.com/scholar?q=Lola+Le+Breton%2C+Quentin+Fournier%2C+Mariam+El+Mezouar%2C+and+Sarath+Chandar%2E+Neobert%3A+A+next+generation+bert%2E+Transactions+on+Machine+Learning+Research%2C+2025%2E)
23. Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Keh, Kushal Arora, et al. [Datacomp-lm: In search of the next generation of training sets for language models](https://md.research.elamur.ai/free/en/papers/li-2024-datacomp-lm-search-next-generation/). Advances in Neural Information Processing Systems, 37:14200–142…
24. Xiangyang Li, Kuicai Dong, Yi Quan Lee, Wei Xia, Hao Zhang, Xinyi Dai, Yasheng Wang, and Ruiming Tang. Coir: A comprehensive benchmark for code information retrieval models. arXiv preprint arXiv:2407.02883, 2024b. [find](https://scholar.google.com/scholar?q=Xiangyang+Li%2C+Kuicai+Dong%2C+Yi+Quan+Lee%2C+Wei+Xia%2C+Hao+Zhang%2C+Xinyi+Dai%2C+Yasheng+Wang%2C+and+Ruiming+Tang%2E+Coir%3A+A+comprehensive+benchmark+for+code+information+retrieval+models%2E+arXiv+preprint+arXiv%3A2407%2E)
25. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692, 2019. URL http://arxiv.org/abs/1907.11692. [↗](http://arxiv.org/abs/1907.11692)
26. Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu: the finest collection of educational content, 2024. URL [https://huggingface.co/datasets/](https://huggingface.co/datasets/) HuggingFaceFW/fineweb-edu. [↗](https://huggingface.co/datasets/)
27. Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M Guerreiro, Ricardo Rei, Duarte M Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, et al. [Eurollm: Multilingual language models for europe](https://md.research.elamur.ai/free/en/papers/martins-2024-eurollm-multilingual-language-models-europe/). Procedia Computer Science, 255:53–62, 2025.
28. Bettina Messmer, Vinko Sabolčec, and Martin Jaggi. Enhancing multilingual llm pretraining with model-based data selection. arXiv, 2025. URL https://arxiv.org/abs/2502.10361. [↗](https://arxiv.org/abs/2502.10361)
29. Dan Saattrup Nielsen. [Scandeval: A benchmark for scandinavian natural language processing](https://md.research.elamur.ai/free/en/papers/nielsen-2023-scandeval-benchmark-scandinavian-natural-language/). arXiv preprint arXiv:2304.00906, 2023.
30. Zach Nussbaum, John X. Morris, Brandon Duderstadt, and Andriy Mulyar. [Nomic embed: Training a reproducible long context text embedder](https://md.research.elamur.ai/free/en/papers/nussbaum-2024-nomic-embed-training-reproducible-long/). CoRR, abs/2402.01613, 2024. doi: 10.48550/ARXIV. 2402.01613. URL https://doi.org/10.48550/arXiv.2402.01613. [↗](https://doi.org/10.48550/arXiv.2402.01613)
31. Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish I… [find](https://scholar.google.com/scholar?q=Team+OLMo%2C+Pete+Walsh%2C+Luca+Soldaini%2C+Dirk+Groeneveld%2C+Kyle+Lo%2C+Shane+Arora%2C+Akshita+Bhagia%2C+Yuling+Gu%2C+Shengyi+Huang%2C+Matt+Jordan%2C+Nathan+Lambert%2C+Dustin+Schwenk%2C+Oyvind+Tafjord%2C+Taira+Anderson%2C+Davi)
32. OpenAI. Openai o3 and o4-mini system card, 2025. o3-o4-mini-system-card/. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Openai+o3+and+o4%2Dmini+system+card%2C+2025%2E+o3%2Do4%2Dmini%2Dsystem%2Dcard%2F%2E)
33. [URL](https://md.research.elamur.ai/free/en/papers/penedo-2024-fineweb-datasets-decanting-web-finest/) [https://openai.com/index/Guilherme](https://openai.com/index/Guilherme) Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37:30811–30849, 2024. [↗](https://openai.com/index/Guilherme)
34. [Guilherme Penedo, Hynek Kydlíček, Vinko Sabolčec, Bettina Messmer, Negar Foroutan, Amir Hossein Kargaran, Colin Raffel, Martin Jaggi, Leandro Von Werra, and Thomas Wolf. Fineweb2: One pipeline to scale them all – adapting pre-training data processing to every language, 2025. URL](https://md.research.elamur.ai/free/en/papers/penedo-2025-fineweb2-one-pipeline-scale-them/) [https://arxiv.org/…](https://arxiv.org/…) [↗](https://arxiv.org/…)
35. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. [Deep contextualized word representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/). In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Comp…
36. Jacob Portes, Alexander Trott, Sam Havens, Daniel King, Abhinav Venigalla, Moin Nadeem, Nikhil Sardana, Daya Khudia, and Jonathan Frankle. Mosaicbert: A bidirectional encoder optimized for fast pretraining. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (e… [find](https://scholar.google.com/scholar?q=Jacob+Portes%2C+Alexander+Trott%2C+Sam+Havens%2C+Daniel+King%2C+Abhinav+Venigalla%2C+Moin+Nadeem%2C+Nikhil+Sardana%2C+Daya+Khudia%2C+and+Jonathan+Frankle%2E+Mosaicbert%3A+A+bidirectional+encoder+optimized+for+fast+pretra)
37. Nils Reimers and Iryna Gurevych. [Sentence-bert: Sentence embeddings using siamese bert-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL https://arxiv.org/abs/1908.10084. [↗](https://arxiv.org/abs/1908.10084)
38. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019. [find](https://scholar.google.com/scholar?q=Victor+Sanh%2C+Lysandre+Debut%2C+Julien+Chaumond%2C+and+Thomas+Wolf%2E+Distilbert%2C+a+distilled+version+of+bert%3A+smaller%2C+faster%2C+cheaper+and+lighter%2E+arXiv+preprint+arXiv%3A1910%2E01108%2C+2019%2E)
39. Annika Simonsen, Dan Saattrup Nielsen, and Hafsteinn Einarsson. Foqa: A faroese question-answering dataset. arXiv preprint arXiv:2502.07642, 2025. [find](https://scholar.google.com/scholar?q=Annika+Simonsen%2C+Dan+Saattrup+Nielsen%2C+and+Hafsteinn+Einarsson%2E+Foqa%3A+A+faroese+question%2Danswering+dataset%2E+arXiv+preprint+arXiv%3A2502%2E07642%2C+2025%2E)
40. Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint arXiv:2402.00159, 2024. [find](https://scholar.google.com/scholar?q=Luca+Soldaini%2C+Rodney+Kinney%2C+Akshita+Bhagia%2C+Dustin+Schwenk%2C+David+Atkinson%2C+Russell+Authur%2C+Ben+Bogin%2C+Khyathi+Chandu%2C+Jennifer+Dumas%2C+Yanai+Elazar%2C+et+al%2E+Dolma%3A+An+open+corpus+of+three+trillion+to)
41. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). Neurocomputing, 568:127063, 2024.
42. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805, 2023.
43. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025. [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Aishwarya+Kamath%2C+Johan+Ferret%2C+Shreya+Pathak%2C+Nino+Vieillard%2C+Ramona+Merhej%2C+Sarah+Perrin%2C+Tatiana+Matejovicova%2C+Alexandre+Ram%C3%A9%2C+Morgane+Rivi%C3%A8re%2C+et+al%2E+Gemma+3+technical+report%2E+arXiv+pr)
44. NLLB Team, Marta R Costa-Jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, et al. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672, 2022. [find](https://scholar.google.com/scholar?q=NLLB+Team%2C+Marta+R+Costa%2DJuss%C3%A0%2C+James+Cross%2C+Onur+%C3%87elebi%2C+Maha+Elbayad%2C+Kenneth+Heafield%2C+Kevin+Heffernan%2C+Elahe+Kalbassi%2C+Janice+Lam%2C+Daniel+Licht%2C+et+al%2E+No+language+left+behind%3A+Scaling+human%2Dcente)
45. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. [Glue: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). arXiv preprint arXiv:1804.07461, 2018.
46. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in neural information processing systems, 33:5776–5788, 2020. [find](https://scholar.google.com/scholar?q=Wenhui+Wang%2C+Furu+Wei%2C+Li+Dong%2C+Hangbo+Bao%2C+Nan+Yang%2C+and+Ming+Zhou%2E+Minilm%3A+Deep+self%2Dattention+distillation+for+task%2Dagnostic+compression+of+pre%2Dtrained+transformers%2E+Advances+in+neural+information+)
47. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, et al. [Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference](https://md.research.elamur.ai/free/en/papers/warner-2024-smarter-better-faster-longer-modern/).…
48. Orion Weller, Kathryn Ricci, Marc Marone, Antoine Chaffin, Dawn Lawrie, and Benjamin Van Durme. Seq vs seq: An open suite of paired encoders and decoders. arXiv preprint arXiv:2507.11412, 2025. — [arXiv:2507.11412](https://arxiv.org/abs/2507.11412) [find](https://scholar.google.com/scholar?q=Orion+Weller%2C+Kathryn+Ricci%2C+Marc+Marone%2C+Antoine+Chaffin%2C+Dawn+Lawrie%2C+and+Benjamin+Van+Durme%2E+Seq+vs+seq%3A+An+open+suite+of+paired+encoders+and+decoders%2E+arXiv+preprint+arXiv%3A2507%2E11412%2C+2025%2E+%E2%80%94+%5BarX)
49. [Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe…](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
50. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. [mt5: A massively multilingual pre-trained text-to-text transformer](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/). arXiv preprint arXiv:2010.11934, 2020.
51. Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. [Ties-merging: Resolving interference when merging models](https://md.research.elamur.ai/free/en/papers/yadav-2023-ties-merging-resolving-interference-merging/). Advances in Neural Information Processing Systems, 36:7093–7115, 2023.
52. Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, Meishan Zhang, Wenjie Li, and Min Zhang. [mgte: Generalized long-context text representation and reranking models for multilingual text retrieval](https://md.research.elamur.ai/free/en/papers/zhang-2024-mgte-generalized-long-context-text/). In Franck Dernoncourt, Daniel Pre…

*Updated 24 October 2026*