---
title: "A Sovereign, Open-Source Foundation Model for German and"
description: "This design is a hybrid of Mamba-2 layers, sparse MoE layers and only six attention layers, so roughly… 90 references, cited by 1 paper here."
authors: ["Soofi-Team: Benedikt Droste", "David Fitzek", "Ruben Harle", "Lukas Helff", "Maximilian Idahl", "Alex Jude", "Abbas Goher Khan", "Maurice Kraus", "Timm Ruland", "Richard Rutmann", "Sebastian Sztwiertnia", "Markus Frey", "Daniil Gurgurov", "Jan Pfister", "Tom Rohr", "Sebastian von Rohrscheidt", "Jorg Bienert", "Nicolas Flores-Herr", "Simon Gottschalk", "Andreas Hotho", "Kristian Kersting", "Joachim Kohler", "Alexander Loser", "Wolfgang Nejdl", "Simon Ostermann", "Jan Plogsties", "Bjorn Pluster", "Patrick Putzky", "Mehdi Ali", "Michael Fromm", "Max Lubbering"]
year: 2026
venue: "arXiv:2607.09424"
source: "https://arxiv.org/abs/2607.09424"
doi: "10.48550/arxiv.2607.09424"
arxiv: "2607.09424"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/droste-2026-sovereign-open-source-foundation-model/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/droste-2026-sovereign-open-source-foundation-model/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/6/) / A Sovereign, Open-Source Foundation Model for German and English

# A Sovereign, Open-Source Foundation Model for German and English

**Authors:** Soofi-Team: Benedikt Droste, David Fitzek, [Ruben Harle](https://md.research.elamur.ai/free/en/authors/ruben-harle/), Lukas Helff, [Maximilian Idahl](https://md.research.elamur.ai/free/en/authors/maximilian-idahl/), Alex Jude, [Abbas Goher Khan](https://md.research.elamur.ai/free/en/authors/abbas-goher-khan/), [Maurice Kraus](https://md.research.elamur.ai/free/en/authors/maurice-kraus/), [Timm Ruland](https://md.research.elamur.ai/free/en/authors/timm-heine-ruland/), [Richard Rutmann](https://md.research.elamur.ai/free/en/authors/richard-rutmann/), [Sebastian Sztwiertnia](https://md.research.elamur.ai/free/en/authors/sebastian-sztwiertnia/), Markus Frey, [Daniil Gurgurov](https://md.research.elamur.ai/free/en/authors/daniil-gurgurov/), [Jan Pfister](https://md.research.elamur.ai/free/en/authors/jan-pfister/), Tom Rohr, Sebastian von Rohrscheidt, Jorg Bienert, [Nicolas Flores-Herr](https://md.research.elamur.ai/free/en/authors/nicolas-flores-herr/), Simon Gottschalk, [Andreas Hotho](https://md.research.elamur.ai/free/en/authors/andreas-hotho/), [Kristian Kersting](https://md.research.elamur.ai/free/en/authors/kristian-kersting/), Joachim Kohler, [Alexander Loser](https://md.research.elamur.ai/free/en/authors/alexander-loser/), [Wolfgang Nejdl](https://md.research.elamur.ai/free/en/authors/wolfgang-nejdl/), [Simon Ostermann](https://md.research.elamur.ai/free/en/authors/simon-ostermann/), Jan Plogsties, Bjorn Pluster, Patrick Putzky, [Mehdi Ali](https://md.research.elamur.ai/free/en/authors/mehdi-ali/), [Michael Fromm](https://md.research.elamur.ai/free/en/authors/michael-fromm/) and Max Lubbering · **Year:** 2026 · **Venue:** arXiv:2607.09424 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2607.09424) · [PDF](https://arxiv.org/pdf/2607.09424) · [DOI 10.48550/arxiv.2607.09424](https://doi.org/10.48550/arxiv.2607.09424) · [arXiv 2607.09424](https://arxiv.org/abs/2607.09424) · [OpenAlex](https://openalex.org/W7168136488)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Soofi S 30B-A3B is a German–English base model built by a German research consortium. It targets three weaknesses of current open models: releases that share weights but hide how they were made, thin coverage of German in multilingual models, and the high serving cost of dense Transformers at long context. The team reused the Nemotron 3 Nano design without changes. This design is a hybrid of Mamba-2 layers, sparse MoE layers and only six attention layers, so roughly 3B of about 31.6B parameters work per token and the cache barely grows with context. Training ran on the German Industrial AI Cloud over roughly 27T tokens in three phases: stable pretraining, annealing, and long-context extension to 1M tokens. The German share was raised well above the reference recipe. Across 16 open base models, Soofi S equals dense 14–27B models on aggregate English and German scores, leads on code, and beats all European sovereign baselines. It also decodes 8–9× faster than dense peers at 40K context. The release matters as much as the model: weights, per-source data accounting, hyperparameters and code.

## Key points
- Architecture: 52 layers (23 Mamba-2, 23 granular MoE with 128 routed experts, 6 active plus 2 shared, and 6 GQA layers). Only the GQA layers keep a KV cache. There are 3.2B active parameters out of 31.6B total.
- Training: Megatron-Bridge with AdamW and a Warmup–Stable–Decay schedule using a minus_sqrt decay. The global batch is 3072 sequences × 8192 tokens, about 25.2M tokens per step, in bf16. The curriculum totals 26.68T tokens: about 20T stable, about 6.6T annealing, then a length-bucketed long-context phase.
- German share: German makes up 7.2% of the stable-phase data and 15.32% of the annealing-phase data, over three times the reference recipe's multilingual share.
- Results: highest English and German aggregates among fully open models, ahead of Olmo 3 32B and Apertus 70B. It has the best code aggregates (HumanEval/MBPP) in both languages and a tied-best INCLUDE-DE score. Decode speed reaches 4.8k TPS/GPU and stays nearly flat from 4K to 256K context.
- Release: weights, selected checkpoints and per-source token counts, including sources that were tried and dropped. The report also documents a discarded annealing stage. Training and evaluation code is published (Soofi-Pretraining, base-eval, eval-hive). The authors name high-quality German data as the main bottleneck for further gains.

## Interesting ideas
- The authors reused an existing architecture without modification, so Nemotron 3 Nano becomes a baseline with identical structure, and any difference can be credited to the data recipe alone. This gives a clean causal test of data choices without architecture confounds.
- Token accounting keeps two kinds of numbers apart: estimates copied from dataset cards and exact counts measured with the real tokenizer during training. Reported data sizes are often guesses, and this split makes the real data budget auditable.
- The report includes negative results, such as excluded data sources and a final annealing stage that was thrown away. Publishing dead ends saves others from repeating them and makes the final choices easier to trust.
- Aggregate scores leave out a benchmark group that had been withdrawn as a held-out set, so suspect test data cannot inflate the headline numbers. Running parallel suites in two languages also allows like-for-like comparison.
- Sources with restrictive licenses are described through aggregate statistics and exact mixture shares rather than shared as raw text. This keeps a pipeline reproducible in spirit when the data itself cannot be redistributed.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Train gparse or the perceptron parser twice with identical settings, changing only the silver-data mix, to measure what the data itself contributes.
- **I2** (learn): Log for each silver source and each witness parser both the claimed size and the number of sentences or tokens actually used in training.
- **I3** (learn): Keep a registry of rejected repair rules and discarded training runs, with the reason for each, next to the evaluations with negative controls.
- **I4** (answer): Drop any school-test or fable-QA set from headline metrics once it may have leaked into the teacher or rule development, and report it separately.
- **I5** (tree): Treebanks with restrictive licenses can be documented the same way, by size, genre and share in the training mix, while freely licensed UD conversions are released in full.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Technische Universität Darmstadt](https://md.research.elamur.ai/free/en/universities/technische-universitat-darmstadt/) — University, Darmstadt, Germany · 109 works · 35,920 citations
- [German Research Centre for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/german-research-centre-for-artificial-intelligence/) — Research institute, Kaiserslautern, Germany · 77 works · 29,479 citations
- [Fraunhofer Institute for Intelligent Analysis and Information Systems](https://md.research.elamur.ai/free/en/labs/fraunhofer-institute-for-intelligent-analysis-and-information-systems/) — Research institute, Sankt Augustin, Germany · 9 works · 3,875 citations
- [Berliner Hochschule für Technik](https://md.research.elamur.ai/free/en/universities/berliner-hochschule-fur-technik/) — University, Berlin, Germany · 7 works · 799 citations
- [University of Würzburg](https://md.research.elamur.ai/free/en/universities/university-of-wurzburg/) — University, Würzburg, Germany · 8 works · 60 citations

Also on the byline: [Bundesverband Kalksandsteinindustrie](https://ror.org/01n78v775) · [Bundesverband Glaukom-Selbsthilfe](https://ror.org/01q4pmw61) · [Fraunhofer Institute for Integrated Circuits](https://ror.org/024ape423) · [Bundesverband Flachglas](https://ror.org/05jzn3892).

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works

## Cite

Cite as: Soofi-Team: Benedikt Droste, David Fitzek, Ruben Harle, Lukas Helff, Maximilian Idahl, Alex Jude, Abbas Goher Khan, Maurice Kraus, Timm Ruland, Richard Rutmann, Sebastian Sztwiertnia, Markus Frey, Daniil Gurgurov, Jan Pfister, Tom Rohr, Sebastian von Rohrscheidt, Jorg Bienert, Nicolas Flores-Herr, Simon Gottschalk, Andreas Hotho, Kristian Kersting, Joachim Kohler, Alexander Loser, Wolfgang Nejdl, Simon Ostermann, Jan Plogsties, Bjorn Pluster, Patrick Putzky, Mehdi Ali, Michael Fromm and Max Lubbering. 2026. A Sovereign, Open-Source Foundation Model for German and English. arXiv:2607.09424.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{droste2026sovereign,
  title = {{A Sovereign, Open-Source Foundation Model for German and English}},
  author = {Soofi-Team: Benedikt Droste, David Fitzek, Ruben Harle, Lukas Helff, Maximilian Idahl, Alex Jude, Abbas Goher Khan, Maurice Kraus, Timm Ruland, Richard Rutmann, Sebastian Sztwiertnia, Markus Frey, Daniil Gurgurov, Jan Pfister, Tom Rohr, Sebastian {von Rohrscheidt}, Jorg Bienert, Nicolas Flores-Herr, Simon Gottschalk, Andreas Hotho, Kristian Kersting, Joachim Kohler, Alexander Loser, Wolfgang Nejdl, Simon Ostermann, Jan Plogsties, Bjorn Pluster, Patrick Putzky, Mehdi Ali, Michael Fromm and Max Lubbering},
  year = {2026},
  doi = {10.48550/arxiv.2607.09424},
  eprint = {2607.09424},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2607.09424}
}
```

- [← previous A Longitudinal Study of the Adoption of Specialized MT Systems in Canadian Parliamentary Translation](https://md.research.elamur.ai/free/en/papers/simard-2026-longitudinal-study-adoption-specialized-mt/)
- [next → A Tale of Two Scripts: Transliteration and Post-Correction for Judeo-Arabic](https://md.research.elamur.ai/free/en/papers/gonzalez-2026-tale-two-scripts-transliteration-post/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/6/)

## References

90 references, 40 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. J. Ainslie, J. Lee-Thorp, M. De Jong, Y. Zemlyanskiy, F. Lebrón, and S. Sanghai. [Gqa: Training generalized multi-query transformer models from multi-head checkpoints](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895–4901, 2023.
2. M. Ali, M. Brack, M. Lübbering, E. Wendt, A. G. Khan, R. Rutmann, A. Jude, M. Kraus, A. A. Weber, F. Stollenwerk, D. Kaczér, F. Mai, L. Flek, R. Sifa, N. Flores-Herr, J. Koehler, P. Schramowski, M. Fromm, and K. Kersting. Judging quality across languages: A multilingual approach to pretraining data filtering with language models. In C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng, editors, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 8859–8898, Suzhou, China, Nov. 2025. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2505.22232)
3. M. Ali, M. Fromm, K. Thellmann, J. Ebert, A. A. Weber, R. Rutmann, C. Jain, M. Lübbering, D. Steinigen, J. Leveling, K. Klug, J. S. Buschhoff, L. Jurkschat, H. Abdelwahab, B. J. Stein, K.-H. Sylla, P. Denisov, N. Brandizzi, Q. Saleem, A. Bhowmick, L. Helmer, C. John, P. O. Suarez, M. Ostendorff, A. Jude, L. Manjunath, S. Weinbach, C. Penke, O. Filatov, F. Barth, P. Mirza, L. Weber, I. Wendler, R. Sifa, F. Küch, A. Herten, R. Jäkel, G. Rehm, S. Kesselheim, J. Köhler, and N. Flores-Herr. Teuken-7b-base & teuken-7b-instruct: Towards european llms. arXiv preprint arXiv:2410.03730, 2025. [↗](https://arxiv.org/abs/2410.03730)
4. E. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, É. Goffinet, D. Hesslow, J. Launay, Q. Malartic, D. Mazzotta, B. Noune, B. Pannier, and G. Penedo. [The falcon series of open language models](https://md.research.elamur.ai/free/en/papers/almazrouei-2023-falcon-series-open-language-models/). arXiv preprint arXiv:2311.16867, 2023. [↗](https://arxiv.org/abs/2311.16867)
5. P. Apertus, A. Hernández-Cano, A. Hägele, A. H. Huang, A. Romanou, A.-J. Solergibert, B. Pasztor, B. Messmer, D. Garbaya, E. F. Ďurech, I. Hakimi, J. G. Giraldo, M. Ismayilzada, N. Foroutan, S. Moalla, T. Chen, V. Sabolčec, Y. Xu, M. Aerni, B. AlKhamissi, I. A. Mariñas, M. H. Amani, M. Ansaripour, I. Badanin, H. Benoit, E. Boros, N. Browning, F. Bösch, M. Böther, N. Canova, C. Challier, C. Charmillot, J. Coles, J. Deriu, A. Devos, L. Drescher, D. Dzenhaliou, M. Ehrmann, D. Fan, S. Fan, S. Gao, M. Gila, M. Grandury, D. Hashemi, A. Hoyle, J. Jiang, M. Klein, A. Kucharavy, A. Kucherenko, F. Lübeck, R. Machacek, T. Manitaras, A. Marfurt, K. Matoba, S. Matrenok, H. Mendonça, F. R. Mohamed, S. Montariol, L. Mouchel, S. Najem-Meyer, J. Ni, G. Oliva, M. Pagliardini, E. Palme, A. Panferov, L. Paoletti, M. Passerini, I. Pavlov, A. Poiroux, K. Ponkshe, N. Ranchin, J. Rando, M. Sauser, J. Saydaliev, M. A. Sayfiddinov, M. Schneider, S. Schuppli, M. Scialanga, A. Semenov, K. Shridhar, R. Singhal, A. Sotnikova, A. Sternfeld, A. K. Tarun, P. Teiletche, J. Vamvas, X. Yao, H. Zhao, A. Ilic, A. Klimovic, A. Krause, C. Gulcehre, D. Rosenthal, E. Ash, F. Tramèr, J. VandeVondele, L. Veraldi, M. Rajman, T. Schulthess, T. Hoefler, A. Bosselut, M. Jaggi, and I. Schlag. [Apertus: Democratizing open and compliant llms for global language environments](https://md.research.elamur.ai/free/en/papers/cano-2026-apertus-democratizing-open-compliant-llms/). arXiv preprint arXiv:2509.14233, 2025. [↗](https://arxiv.org/abs/2509.14233)
6. J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732, 2021. [↗](https://arxiv.org/abs/2108.07732)
7. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O’Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. Van Der Wal. Pythia: A suite for analyzing large language models across training and scaling. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR, 23–29 Jul 2023. [find](https://scholar.google.com/scholar?q=S%2E+Biderman%2C+H%2E+Schoelkopf%2C+Q%2E+G%2E+Anthony%2C+H%2E+Bradley%2C+K%2E+O%E2%80%99Brien%2C+E%2E+Hallahan%2C+M%2E+A%2E+Khan%2C+S%2E+Purohit%2C+U%2E+S%2E+Prashanth%2C+E%2E+Raff%2C+A%2E+Skowron%2C+L%2E+Sutawika%2C+and+O%2E+Van+Der+Wal%2E+Pythia%3A+A+suite+for+analy)
8. BigScience Workshop. BLOOM: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022. [↗](https://arxiv.org/abs/2211.05100)
9. S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leahy, K. Mc-Donell, J. Phang, M. Pieler, U. S. Prashanth, S. Purohit, L. Reynolds, J. Tow, B. Wang, and S. Weinbach. [GPT-NeoX-20B: An open-source autoregressive language model](https://md.research.elamur.ai/free/en/papers/black-2022-gpt-neox-20b-open-source/). In A. Fan, S. Ilic, T. Wolf, and M. Gallé, editors, Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models, pages 95–136, virtual+Dublin, May 2022. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2204.06745)
10. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901, 2020. [↗](https://doi.org/10.48550/arxiv.2005.14165)
11. L. Burchell, O. de Gibert, N. Arefyev, M. Aulamo, M. Bañón, P. Chen, M. Fedorova, L. Guillou, B. Haddow, J. Hajič, J. Helcl, E. Henriksson, M. Klimaszewski, V. Komulainen, A. Kutuzov, J. Kytöniemi, V. Laippala, P. Mæhlum, B. Malik, F. Mehryary, V. Mikhailov, N. Moghe, A. Myntti, D. O’Brien, S. Oepen, P. Pal, J. Piha, S. Pyysalo, G. Ramírez-Sánchez, D. Samuel, P. Stepachev, J. Tiedemann, D. Variš, T. Vojtěchová, and J. Zaragoza-Bernabeu. [An expanded massive multilingual dataset for high-performance language technologies (HPLT)](https://md.research.elamur.ai/free/en/papers/burchell-2025-expanded-massive-multilingual-dataset-high/). In W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 17452–17485, Vienna, Austria, July 2025. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2503.10267)
12. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374, 2021. [↗](https://arxiv.org/abs/2107.03374)
13. P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/), 2018. [↗](https://doi.org/10.48550/arxiv.1803.05457)
14. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021. [↗](https://arxiv.org/abs/2110.14168)
15. D. Dai, C. Deng, C. Zhao, R. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1280–1297, 2024. [find](https://scholar.google.com/scholar?q=D%2E+Dai%2C+C%2E+Deng%2C+C%2E+Zhao%2C+R%2E+Xu%2C+H%2E+Gao%2C+D%2E+Chen%2C+J%2E+Li%2C+W%2E+Zeng%2C+X%2E+Yu%2C+Y%2E+Wu%2C+et+al%2E+Deepseekmoe%3A+Towards+ultimate+expert+specialization+in+mixture%2Dof%2Dexperts+language+models%2E+In+Proceedings+of+the+)
16. T. Dao and A. Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060, 2024. [↗](https://arxiv.org/abs/2405.21060)
17. Deutsche Telekom AG. Germany’s first AI factory for industry officially goes into operation in Munich. [https://www.telekom.com/en/media/media-information/archive/germany-s-f](https://www.telekom.com/en/media/media-information/archive/germany-s-f) irst-ai-factory-for-industry-1101670, Feb. 2026. Industrial AI Cloud, Munich; accessed 2026-07-08. [↗](https://www.telekom.com/en/media/media-information/archive/germany-s-f)
18. S. Diao, Y. Yang, Y. Fu, X. Dong, D. Su, M. Kliegl, Z. Chen, P. Belcak, Y. Suhara, H. Yin, et al. Nemotron-climb: Clustering-based iterative data mixture bootstrapping for language model pre-training. Advances in Neural Information Processing Systems, 38, 2026. [find](https://scholar.google.com/scholar?q=S%2E+Diao%2C+Y%2E+Yang%2C+Y%2E+Fu%2C+X%2E+Dong%2C+D%2E+Su%2C+M%2E+Kliegl%2C+Z%2E+Chen%2C+P%2E+Belcak%2C+Y%2E+Suhara%2C+H%2E+Yin%2C+et+al%2E+Nemotron%2Dclimb%3A+Clustering%2Dbased+iterative+data+mixture+bootstrapping+for+language+model+pre%2Dtraining%2E)
19. W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022. [↗](https://doi.org/10.48550/arxiv.2101.03961)
20. K. Fujii, Y. Tajima, S. Mizuki, M. Kawamura, H. Shimada, T. Shiotani, K. Saito, M. Oi, T. Nakamura, T. Okamoto, S. Ishida, K. Hattori, Y. Ma, H. Takamura, R. Yokota, J. Sakuma, and N. Okazaki. Rewriting pre-training data boosts llm performance in math and code. arXiv preprint arXiv:2505.02881, 2026. [↗](https://arxiv.org/abs/2505.02881)
21. L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. [↗](https://arxiv.org/abs/2101.00027)
22. L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac’h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou. The language model evaluation harness, 07 2024. [find](https://scholar.google.com/scholar?q=L%2E+Gao%2C+J%2E+Tow%2C+B%2E+Abbasi%2C+S%2E+Biderman%2C+S%2E+Black%2C+A%2E+DiPofi%2C+C%2E+Foster%2C+L%2E+Golding%2C+J%2E+Hsu%2C+A%2E+Le+Noac%E2%80%99h%2C+H%2E+Li%2C+K%2E+McDonell%2C+N%2E+Muennighoff%2C+C%2E+Ociepa%2C+J%2E+Phang%2C+L%2E+Reynolds%2C+H%2E+Schoelkopf%2C+A%2E+Skowro)
23. GBI-Genios Deutsche Wirtschaftsdatenbank GmbH. GENIOS – german business and press database. [https://www.genios.de/browse/Alle](https://www.genios.de/browse/Alle), 2025. Commercially licensed corpus of German newspaper and trade-press archives; obtained under a license that does not permit redistribution. [↗](https://www.genios.de/browse/Alle)
24. L. Gienapp, C. Schröder, S. Schweter, C. Akiki, F. Schlatt, A. Zimmermann, P. Genêt, and M. Potthast. [The german commons - 154 billion tokens of openly licensed text for german language models](https://md.research.elamur.ai/free/en/papers/gienapp-2025-german-commons-154-billion-tokens/). arXiv preprint arXiv:2510.13996, 2025. [↗](https://arxiv.org/abs/2510.13996)
25. GlaiveAI. Reasoning-v1-20m, 2025. A synthetic reasoning dataset containing 22mil+ general reasoning questions and responses generated using deepseek-ai/DeepSeek-R1-Distill-Llama-70B. [find](https://scholar.google.com/scholar?q=GlaiveAI%2E+Reasoning%2Dv1%2D20m%2C+2025%2E+A+synthetic+reasoning+dataset+containing+22mil%2B+general+reasoning+questions+and+responses+generated+using+deepseek%2Dai%2FDeepSeek%2DR1%2DDistill%2DLlama%2D70B%2E)
26. A. Gonzalez-Agirre, M. Pàmies, J. Llop, I. Baucells, S. D. Dalt, D. Tamayo, J. J. Saiz, F. Espuña, J. Prats, J. Aula-Blasco, M. Mina, I. Pikabea, A. Rubio, A. Shvets, A. Sallés, I. Lacunza, J. Palomar, J. Falcão, L. Tormo, L. Vasquez-Reina, M. Marimon, O. Pareras, V. Ruiz-Fernández, and M. Villegas. Salamandra technical report. arXiv preprint arXiv:2502.08489, 2025. [↗](https://arxiv.org/abs/2502.08489)
27. D. Groeneveld, I. Beltagy, E. Walsh, A. Bhagia, R. Kinney, O. Tafjord, A. Jha, H. Ivison, I. Magnusson, Y. Wang, S. Arora, D. Atkinson, R. Authur, K. Chandu, A. Cohan, J. Dumas, Y. Elazar, Y. Gu, J. Hessel, T. Khot, W. Merrill, J. Morrison, N. Muennighoff, A. Naik, C. Nam, M. Peters, V. Pyatkin, A. Ravichander, D. Schwenk, S. Shah, W. Smith, E. Strubell, N. Subramani, M. Wortsman, P. Dasigi, N. Lambert, K. Richardson, L. Zettlemoyer, J. Dodge, K. Lo, L. Soldaini, N. Smith, and H. Hajishirzi. OLMo: Accelerating the science of language models. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15789–15809, Bangkok, Thailand, Aug. 2024. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2402.00838)
28. D. Gurgurov and T. Röhr. ReasonXL: A multilingual cross-domain reasoning corpus. [https://huggingface.co/datasets/toroe/Soofi-Think-SFT-10B-multilingual](https://huggingface.co/datasets/toroe/Soofi-Think-SFT-10B-multilingual), 2026. [↗](https://huggingface.co/datasets/toroe/Soofi-Think-SFT-10B-multilingual)
29. D. Gurgurov, T. Röhr, and S. Ostermann. Nemotron-multilingual-reasoning: A multilingual science reasoning dataset. [https://huggingface.co/datasets/DGurgurov/Nemotron-Multi](https://huggingface.co/datasets/DGurgurov/Nemotron-Multi) lingual-Reasoning, 2025. Derived from nvidia/Llama-Nemotron-Post-Training-Dataset with machine-translated reasoning traces. [↗](https://huggingface.co/datasets/DGurgurov/Nemotron-Multi)
30. A. Hägele, E. Bakouch, A. Kosson, L. B. Allal, L. Von Werra, and M. Jaggi. [Scaling laws and compute-optimal training beyond fixed training durations](https://md.research.elamur.ai/free/en/papers/hagele-2024-scaling-laws-compute-optimal-training/). Advances in Neural Information Processing Systems, 37:76232–76264, 2024.
31. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/), 2021.
32. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021. [↗](https://doi.org/10.48550/arxiv.2103.03874)
33. J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, O. Vinyals, J. Rae, and L. Sifre. An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Systems, volume 35, pages 30016–30030. Curran Associates, Inc., 2022. [find](https://scholar.google.com/scholar?q=J%2E+Hoffmann%2C+S%2E+Borgeaud%2C+A%2E+Mensch%2C+E%2E+Buchatskaya%2C+T%2E+Cai%2C+E%2E+Rutherford%2C+D%2E+de+Las+Casas%2C+L%2E+A%2E+Hendricks%2C+J%2E+Welbl%2C+A%2E+Clark%2C+T%2E+Hennigan%2C+E%2E+Noland%2C+K%2E+Millican%2C+G%2E+van+den+Driessche%2C+B%2E+Damoc%2C+A)
34. C.-P. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y. Zhang, and B. Ginsburg. Ruler: What’s the real context size of your long-context language models?, 2024. — [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://md.research.elamur.ai/free/en/papers/hsieh-2024-ruler-s-real-context-size/) [↗](https://doi.org/10.48550/arxiv.2404.06654)
35. S. Hu, Y. Tu, X. Han, C. He, G. Cui, X. Long, Z. Zheng, Y. Fang, Y. Huang, W. Zhao, et al. [Minicpm: Unveiling the potential of small language models with scalable training strategies](https://md.research.elamur.ai/free/en/papers/hu-2024-minicpm-unveiling-potential-small-language/). arXiv preprint arXiv:2404.06395, 2024. [↗](https://arxiv.org/abs/2404.06395)
36. M. Idahl, B. Droste, B. Plüster, and J. P. Harries. propella-1: Multi-property document annotation for llm data curation at scale, 2026. [↗](https://doi.org/10.48550/arxiv.2602.12414)
37. M. Idahl, J. Tiedemann, S. Pyysalo, D. Salinas, T. Galica, S. Qian, T. N. Mateiu, Z. Li, A. Lokrantz, F. Vitiugin, A. F. T. Martins, J. Kanerva, F. Ginter, M. Lindemann, T. Isbister, B. Moell, J. Lindh, J. Hajič, J. Jitsev, A. Kutuzov, S. Oepen, and G. Ramírez-Sánchez. Multisynt/mt: Trillion-token multi-parallel pre-training data translated across 36 languages, 2026. [↗](https://doi.org/10.48550/arxiv.2607.00890)
38. A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. Le Scao, T. Lavril, T. Wang, T. Lacroix, and W. El Sayed. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. [↗](https://arxiv.org/abs/2310.06825)
39. A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. de las Casas, E. Bou Hanna, F. Bressand, G. Lengyel, G. Bour, G. Lample, L. R. Lavaud, L. Saulnier, M.-A. Lachaux, P. Stock, S. Subramanian, S. Yang, S. Antoniak, T. Le Scao, T. Gervet, T. Lavril, T. Wang, T. Lacroix, and W. El Sayed. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. [↗](https://arxiv.org/abs/2401.04088)
40. J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). arXiv preprint arXiv:2001.08361, 2020. [↗](https://arxiv.org/abs/2001.08361)
41. J. Krajewski, J. Ludziejewski, K. Adamczewski, M. Pióro, M. Krutul, S. Antoniak, K. Ciebiera, K. Król, T. Odrzygóźdź, P. Sankowski, et al. Scaling laws for fine-grained mixture of experts. arXiv preprint arXiv:2402.07871, 2024. [↗](https://arxiv.org/abs/2402.07871)
42. M. Kraus, R. Härle, S. Sztwiertnia, A. G. Khan, M. Ali, M. Fromm, and K. Kersting. Klettermix: Climbing toward high-quality german pretraining data. arXiv preprint arXiv:2606.[03773, 2026](https://md.research.elamur.ai/free/en/papers/kraus-2026-klettermix-climbing-toward-high-quality/). [↗](https://arxiv.org/abs/2606.03773)
43. T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M.-W. Chang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov. [Natural questions: A benchmark for question answering research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/). Transactions of the Association for Computational Linguistics, 7:452–466, 2019. [↗](https://doi.org/10.1162/tacl_a_00276)
44. H. Kydlíček, G. Penedo, and L. von Werra. Finepdfs. [https://huggingface.co/datasets/](https://huggingface.co/datasets/) HuggingFaceFW/finepdfs, 2025. [↗](https://huggingface.co/datasets/)
45. A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, Y. Wu, B. Neyshabur, G. Gur-Ari, and V. Misra. [Solving quantitative reasoning problems with language models](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/). In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 3843–3857. Curran Associates, Inc., 2022.
46. A. Liesenfeld, M. Dingemanse, D. Blankvoort, N. Kalra, and A. R. Golkhandan. European open source ai definitions. [https://osai-index.eu/osai-definitions](https://osai-index.eu/osai-definitions), 2026. Accessed: 2026-07-06. [↗](https://osai-index.eu/osai-definitions)
47. A. H. Liu, K. Khandelwal, S. Subramanian, V. Jouault, A. Rastogi, A. Sadé, A. Jeffares, A. Jiang, A. Cahill, A. Gavaudan, A. Sablayrolles, A. Héliou, A. You, A. Ehrenberg, A. Lo, A. Eliseev, A. Calvi, A. Sooriyarachchi, B. Bout, B. Rozière, B. D. Monicault, C. Lanfranchi, C. Barreau, C. Courtot, D. Grattarola, D. Dabert, D. de las Casas, E. Chane-Sane, F. Ahmed, G. Berrada, G. Ecrepont, G. Guinet, G. Novikov, G. Kunsch, G. Lample, G. Martin, G. Gupta, J. Ludziejewski, J. Rute, J. Studnia, J. Amar, J. Delas, J. S. Roberts, K. Yadav, K. Chandu, K. Jain, L. Aitchison, L. Fainsin, L. Blier, L. Zhao, L. Martin, L. Saulnier, L. Gao, M. Buyl, M. Jennings, M. Pellat, M. Prins, M. Poirée, M. Guillaumin, M. Dinot, M. Futeral, M. Darrin, M. Augustin, M. Chiquier, M. Schimpf, N. Grinsztajn, N. Gupta, N. Raghuraman, O. Bousquet, O. Duchenne, P. Wang, P. von Platen, P. Jacob, P. Wambergue, P. Kurylowicz, P. R. Muddireddy, P. Chagniot, P. Stock, P. Agrawal, Q. Torroba, R. Sauvestre, R. Soletskyi, R. Menneer, S. Vaze, S. Barry, S. Gandhi, S. Waghjale, S. Gandhi, S. Ghosh, S. Mishra, S. Aithal, S. Antoniak, T. L. Scao, T. Cachet, T. S. Sorg, T. Lavril, T. N. Saada, T. Chabal, T. Foubert, T. Robert, T. Wang, T. Lawson, T. Bewley, T. Bewley, T. Edwards, U. Jamil, U. Tomasini, V. Nemychnikova, V. Phung, V. Maladière, V. Richard, W. Bouaziz, W.-D. Li, W. Marshall, X. Li, X. Yang, Y. E. Ouahidi, Y. Wang, Y. Tang, and Z. Ramzi. Ministral 3. arXiv preprint arXiv:2601.08584, 2026. [↗](https://arxiv.org/abs/2601.08584)
48. Z. Liu, Z. Yang, Y. Chen, C. Lee, M. Shoeybi, B. Catanzaro, and W. Ping. [Acereason-nemotron 1.1: Advancing math and code reasoning through sft and rl synergy](https://md.research.elamur.ai/free/en/papers/liu-2025-acereason-nemotron-1-1-advancing/). arXiv preprint arXiv:2506.13284, 2025. [↗](https://arxiv.org/abs/2506.13284)
49. I. Loshchilov and F. Hutter. arXiv:1711.05101, 2017. Decoupled weight decay regularization. arXiv preprint [↗](https://arxiv.org/abs/1711.05101)
50. M. Lübbering, T. Ruland, R. Rutmann, F. Stollenwerk, D. Fitzek, M. Fromm, A. Weber, R. Sifa, N. Flores-Herr, J. Köhler, et al. Modalities, a pytorch-native framework for large-scale llm training and research. arXiv preprint arXiv:2602.08387, 2026. [↗](https://arxiv.org/abs/2602.08387)
51. R. K. Mahabadi, S. Satheesh, S. Prabhumoye, M. Patwary, M. Shoeybi, and B. Catanzaro. Nemotron-cc-math: A 133 billion-token-scale high quality math pretraining dataset. arXiv preprint arXiv:2508.15096, 2025. [↗](https://arxiv.org/abs/2508.15096)
52. P. H. Martins, J. Alves, P. Fernandes, N. M. Guerreiro, R. Rei, A. Farajian, M. Klimaszewski, D. M. Alves, J. Pombal, N. Boizard, M. Faysse, P. Colombo, F. Yvon, B. Haddow, J. G. C. de Souza, A. Birch, and A. F. T. Martins. [Eurollm-9b: Technical report](https://md.research.elamur.ai/free/en/papers/martins-2025-eurollm-9b-technical-report/). arXiv preprint arXiv:2506.04079, 2025. [↗](https://arxiv.org/abs/2506.04079)
53. P. H. Martins, P. Fernandes, J. Alves, N. M. Guerreiro, R. Rei, D. M. Alves, J. Pombal, A. Farajian, M. Faysse, M. Klimaszewski, P. Colombo, B. Haddow, J. G. C. de Souza, A. Birch, and A. F. T. Martins. [Eurollm: Multilingual language models for europe](https://md.research.elamur.ai/free/en/papers/martins-2024-eurollm-multilingual-language-models-europe/). arXiv preprint arXiv:2409.16235, 2024. [↗](https://arxiv.org/abs/2409.16235)
54. A. Matton, T. Sherborne, D. Aumiller, E. Tommasone, M. Alizadeh, J. He, R. Ma, M. Voisin, E. Gilsenan-McMahon, and M. Gallé. On leakage of code generation evaluation datasets. In Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 13215–13223, Miami, Florida, USA, Nov. 2024. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2407.07565)
55. Mistral AI. Mistral small 3.1. [https://mistral.ai/news/mistral-small-3-1/](https://mistral.ai/news/mistral-small-3-1/), 2025. Model release, March 2025. [↗](https://mistral.ai/news/mistral-small-3-1/)
56. J. Morrison, S. Adhikesaven, A. Bhagia, M. Zaharia, N. A. Smith, and S. Min. Train separately, merge together: Modular post-training with mixture-of-experts, 2026. [↗](https://doi.org/10.48550/arxiv.2604.18473)
57. N. Muennighoff, L. Soldaini, D. Groeneveld, K. Lo, J. Morrison, S. Min, W. Shi, E. P. Walsh, O. Tafjord, N. Lambert, Y. Gu, S. Arora, A. Bhagia, D. Schwenk, D. Wadden, A. Wettig, B. Hui, T. Dettmers, D. Kiela, A. Farhadi, N. A. Smith, P. W. Koh, A. Singh, and H. Hajishirzi. OLMoE: Open mixture-of-experts language models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24–28, 2025. OpenReview.net, 2025. [↗](https://doi.org/10.48550/arxiv.2409.02060)
58. MultiSynt. Mt reasoning: Automatic translations into 2 languages of glaive ai reasoning dataset, 2025. An automatic translations into 2 languages of Glaive AI reasoning dataset. [find](https://scholar.google.com/scholar?q=MultiSynt%2E+Mt+reasoning%3A+Automatic+translations+into+2+languages+of+glaive+ai+reasoning+dataset%2C+2025%2E+An+automatic+translations+into+2+languages+of+Glaive+AI+reasoning+dataset%2E)
59. NVIDIA, :, A. Basant, A. Khairnar, A. Paithankar, A. Khattar, A. Renduchintala, A. Malte, A. Bercovich, A. Hazare, A. Rico, A. Ficek, A. Kondratenko, A. Shaposhnikov, A. Bukharin, A. Taghibakhshi, A. Barton, A. S. Mahabaleshwarkar, A. Shen, A. Tao, A. Guan, A. Shors, A. Mandarwal, A. Mehta, A. Venkatesan, A. Sharabiani, A. Aithal, A. Poojary, A. Dattagupta, B. Buddharaju, B. Zhu, B. Simkin, B. Kartal, B. D. Rouhani, B. Chen, B. Ginsburg, B. Norick, B. Yu, B. Catanzaro, C. Wang, C. Truong, C. Mungekar, C. Patel, C. Alexiuk, C. Munley, C. Parisien, D. Su, D. Afrimi, D. Korzekwa, D. Rohrer, D. Gitman, D. Mosallanezhad, D. Narayanan, D. Rekesh, D. Yared, D. Pykhtar, D. Ahn, D. Riach, E. Long, E. Ning, E. Chung, E. Galinkin, E. Bakhturina, G. Prasad, G. Shen, H. Qian, H. Elisha, H. Sharma, H. Ross, H. Ngo, H. Sahota, H. Wang, H. C. Shin, H. Huang, I. Cunningham, I. Gitman, I. Moshkov, J. Jung, J. Kautz, J. P. Scowcroft, J. Casper, J. Zhang, J. Zeng, J. Zhang, J. Xue, J. Huang, J. Conway, J. Kamalu, J. Cohen, J. Jennings, J. V. Vialard, J. Yi, J. Parmar, K. Briski, K. Cheung, K. Luna, K. Wyss, K. Santhanam, K. Kong, K. Pawelec, K. Anik, K. Li, K. Ahmadian, L. McAfee, L. Sleiman, L. Derczynski, L. Vega, M. R. de Melo, M. N. Sreedhar, M. Chochowski, M. Cai, M. Kliegl, M. Stepniewska-Dziubinska, M. Novikov, M. Samadi, M. Price, M. Boubdir, M. Boone, M. Evans, M. Bien, M. Zawalski, M. Martinez, M. Chrzanowski, M. Shoeybi, M. Patwary, N. Dhameja, N. Assaf, N. Habibi, N. Bhatia, N. Pope, N. Tajbakhsh, N. K. Juluru, O. Rybakov, O. Hrinchuk, O. Kuchaiev, O. Olabiyi, P. Ribalta, P. Subramanian, P. Chadha, P. Molchanov, P. Dykas, P. Jin, P. Bialecki, P. Januszewski, P. Thalasta, P. Gaikwad, P. Varshney, P. Gundecha, P. Tredak, R. K. Mahabadi, R. Patel, R. El-Yaniv, R. Rajan, R. Cheruvu, R. Shahbazyan, R. Borkar, R. Gala, R. Waleffe, R. Zhang, R. J. Hewett, R. Prenger, S. Jain, S. Kriman, S. Satheesh, S. Kaji, S. Yurick, S. Muralidharan, S. Narenthiran, S. Bak, S. Sameni, S. Han, S. Ramasamy, S. Ghosh, S. T. Sreenivas, S. Thomas, S. Diao, S. Gopal, S. Prabhumoye, S. Toshniwal, S. Ding, S. Singh, S. Jain, S. Majumdar, S. Singhal, S. Alborghetti, S. N. Akter, T. Kong, T. Moon, T. Hliwiak, T. Asida, T. Wang, T. Konuk, T. Vashishth, T. Poon, U. Karpas, V. Noroozi, V. Srinivasan, V. Korthikanti, V. Fugro, V. Kalluru, V. Kurin, V. Lavrukhin, W. U. Ahmad, W. Du, W. Byeon, X. Lu, X. Dong, Y. Karnati, Y. Choi, Y. Zhang, Y. Lin, Y. Fu, Y. Suhara, Z. Dong, Z. Li, Z. Zhu, and Z. Chen. [Nvidia nemotron nano 2: An accurate and efficient hybrid mamba-transformer reasoning model](https://md.research.elamur.ai/free/en/papers/basant-2025-nvidia-nemotron-nano-2-accurate/), 2025. [↗](https://doi.org/10.48550/arxiv.2508.14444)
60. NVIDIA, :, A. Blakeman, A. Basant, A. Khattar, A. Renduchintala, A. Bercovich, A. Ficek, A. Bjorlin, A. Taghibakhshi, A. S. Deshmukh, A. S. Mahabaleshwarkar, A. Tao, A. Shors, A. Aithal, A. Poojary, A. Dattagupta, B. Buddharaju, B. Chen, B. Ginsburg, B. Wang, B. Norick, B. Butterfield, B. Catanzaro, C. del Mundo, C. Dong, C. Harvey, C. Parisien, D. Su, D. Korzekwa, D. Yin, D. Gitman, D. Mosallanezhad, D. Narayanan, D. Fridman, D. Rekesh, D. Ma, D. Pykhtar, D. Ahn, D. Riach, D. Stosic, E. Long, E. Segal, E. Evans, E. Chung, E. Galinkin, E. Bakhturina, E. Dobrowolska, F. Jia, F. Liu, G. Prasad, G. Shen, G. Liu, G. Chen, H. Qian, H. Ngo, H. Liu, H. Li, I. Gitman, I. Karmanov, I. Moshkov, I. Golan, J. Kautz, J. P. Scowcroft, J. Casper, J. Seppanen, J. Lu, J. Sewall, J. Zeng, J. You, J. Zhang, J. Zhang, J. Huang, J. Xue, J. Huang, J. Conway, J. Kamalu, J. Barker, J. Cohen, J. Jennings, J. Parmar, K. Sapra, K. Briski, K. Chumachenko, K. Luna, K. Santhanam, K. Kong, K. Sivamani, K. Pawelec, K. Anik, K. Li, L. McAfee, L. Derczynski, L. Pavao, L. Vega, L. Voegtle, M. Bala, M. R. de Melo, M. N. Sreedhar, M. Chochowski, M. Kliegl, M. Stepniewska-Dziubinska, M. Le, M. Novikov, M. Samadi, M. Andersch, M. Evans, M. Martinez, M. Chrzanowski, M. Ranzinger, M. Blaz, M. Smelyanskiy, M. Fawzy, M. Shoeybi, M. Patwary, N. Lee, N. Tajbakhsh, N. Xu, O. Rybakov, O. Kuchaiev, O. Delalleau, O. Nitski, P. Chadha, P. Shamis, P. Micikevicius, P. Molchanov, P. Dykas, P. Fischer, P.-Y. Aquilanti, P. Bialecki, P. Varshney, P. Gundecha, P. Tredak, R. Karimi, R. Kandu, R. El-Yaniv, R. Joshi, R. Waleffe, R. Zhang, S. Kavanaugh, S. Jain, S. Kriman, S. Lym, S. Satheesh, S. Muralidharan, S. Narenthiran, S. Anandaraj, S. Bak, S. Kashirsky, S. Han, S. Acharya, S. Ghosh, S. T. Sreenivas, S. Clay, S. Thomas, S. Prabhumoye, S. Pachori, S. Toshniwal, S. Prayaga, S. Jain, S. Das, S. Kierat, S. Majumdar, S. Han, S. Singhal, S. Niverty, S. Alborghetti, S. Panguluri, S. Bhendigeri, S. N. Akter, S. Migacz, T. Shiri, T. Kong, T. Roman, T. Ronen, T. Saar, T. Konuk, T. Rintamaki, T. Poon, U. De, V. Noroozi, V. Singh, V. Korthikanti, V. Kurin, W. U. Ahmad, W. Du, W. Ping, W. Dai, W. Byeon, X. Ren, Y. Xu, Y. Choi, Y. Zhang, Y. Lin, Y. Suhara, Z. Yu, Z. Li, Z. Li, Z. Zhu, Z. Yang, and Z. Chen. [Nemotron-h: A family of accurate and efficient hybrid mamba-transformer models](https://md.research.elamur.ai/free/en/papers/blakeman-2025-nemotron-h-family-accurate-efficient/). arXiv preprint arXiv:2504.03624, 2025. [↗](https://arxiv.org/abs/2504.03624)
61. NVIDIA, :, A. Blakeman, A. Grattafiori, A. Basant, A. Gupta, A. Khattar, A. Renduchintala, A. Vavre, A. Shukla, A. Bercovich, A. Ficek, A. Shaposhnikov, A. Kondratenko, A. Bukharin, A. Milesi, A. Taghibakhshi, A. Liu, A. Barton, A. S. Mahabaleshwarkar, A. Klein, A. Zuker, A. Geifman, A. Shen, A. Bhiwandiwalla, A. Tao, A. Guan, A. Mandarwal, A. Mehta, A. Aithal, A. Poojary, A. Ahamed, A. K. Thekkumpate, A. Dattagupta, B. Zhu, B. Sadeghi, B. Simkin, B. Lanir, B. Schifferer, B. Nushi, B. Kartal, B. D. Rouhani, B. Ginsburg, B. Norick, B. Soubasis, B. Kisacanin, B. Yu, B. Catanzaro, C. del Mundo, C. Hwang, C. Wang, C.-P. Hsieh, C. Zhang, C. Yu, C. Mungekar, C. Patel, C. Alexiuk, C. Parisien, C. Neale, D. Mosk-Aoyama, D. Su, D. Corneil, D. Afrimi, D. Rohrer, D. Serebrenik, D. Gitman, D. Levy, D. Stosic, D. Mosallanezhad, D. Narayanan, D. Nathawani, D. Rekesh, D. Yared, D. Kakwani, D. Ahn, D. Riach, D. Stosic, E. Minasyan, E. Lin, E. Long, E. P. Long, E. Lantz, E. Evans, E. Ning, E. Chung, E. Harper, E. Tramel, E. Galinkin, E. Pounds, E. Briones, E. Bakhturina, F. Ladhak, F. Wang, F. Jia, F. Soares, F. Chen, F. Galko, F. Siino, G. H. Agam, G. Ajjanagadde, G. Bhatt, G. Prasad, G. Armstrong, G. Shen, G. Batmaz, G. Nalbandyan, H. Qian, H. Sharma, H. Ross, H. Ngo, H. Sahota, H. Wang, H. Soni, H. Upadhyay, H. Mao, H. C. Nguyen, H. Q. Nguyen, I. Cunningham, I. Shahaf, I. Gitman, I. Loshchilov, I. Moshkov, I. Putterman, J. Kautz, J. P. Scowcroft, J. Casper, J. Mitra, J. Glick, J. Chen, J. Oliver, J. Zhang, J. Zeng, J. Lou, J. Zhang, J. Huang, J. Conway, J. Guman, J. Kamalu, J. Greco, J. Cohen, J. Jennings, J. Daw, J. V. Vialard, J. Yi, J. Parmar, K. Xu, K. Zhu, K. Briski, K. Cheung, K. Luna, K. Santhanam, K. Shih, K. Kong, K. Bhardwaj, K. C. Puvvada, K. Pawelec, K. Anik, L. McAfee, L. Sleiman, L. Derczynski, L. Ding, L. Liebenwein, L. Vega, M. Grover, M. V. Segbroeck, M. R. de Melo, M. N. Sreedhar, M. Kilaru, M. Ashkenazi, M. Romeijn, M. Cai, M. Kliegl, M. Moosaei, M. Novikov, M. Samadi, M. Corpuz, M. Wang, M. Price, M. Boone, M. Evans, M. Martinez, M. Chrzanowski, M. Shoeybi, M. Patwary, N. Mulepati, N. Hereth, N. Assaf, N. Habibi, N. Zmora, N. Haber, N. Sessions, N. Bhatia, N. Jukar, N. Pope, N. Ludwig, N. Tajbakhsh, N. Juluru, O. Hrinchuk, O. Kuchaiev, O. Delalleau, O. Olabiyi, O. U. Argov, O. Xie, P. Chadha, P. Shamis, P. Molchanov, P. Morkisz, P. Dykas, P. Jin, P. Xu, P. Januszewski, P. P. Thombre, P. Varshney, P. Gundecha, Q. Miao, R. K. Mahabadi, R. El-Yaniv, R. Zilberstein, R. Shafipour, R. Harang, R. Izzo, R. Shahbazyan, R. Garg, R. Borkar, R. Gala, R. Islam, R. Waleffe, R. Watve, R. Koren, R. Zhang, R. J. Hewett, R. Prenger, R. Timbrook, S. Mahdavi, S. Modi, S. Kriman, S. Kariyappa, S. Satheesh, S. Kaji, S. Pasumarthi, S. Narentharen, S. Narenthiran, S. Bak, S. Kashirsky, S. Poulos, S. Mor, S. Ramasamy, S. Acharya, S. Ghosh, S. T. Sreenivas, S. Thomas, S. Fan, S. Gopal, S. Prabhumoye, S. Pachori, S. Toshniwal, S. Ding, S. Singh, S. Sun, S. Ithape, S. Majumdar, S. Singhal, S. Alborghetti, S. Ge, S. D. Devare, S. K. Barua, S. Panguluri, S. Gupta, S. Priyadarshi, S. N. Akter, T. Bui, T.-D. Ene, T. Kong, T. Do, T. Blankevoort, T. Balough, T. Asida, T. B. Natan, T. Konuk, T. Vashishth, U. Karpas, U. De, V. Noorozi, V. Noroozi, V. Srinivasan, V. Elango, V. Korthikanti, V. Kurin, V. Lavrukhin, W. Jiang, W. U. Ahmad, W. Du, W. Ping, W. Zhou, W. Jennings, W. Zhang, W. Prazuch, X. Ren, Y. Karnati, Y. Choi, Y. Meyer, Y.-F. Wu, Y. Zhang, Y. Lin, Y. Geifman, Y. Fu, Y. Subara, Y. Suhara, Y. Gao, Z. Moshe, Z. Dong, Z. Liu, Z. Chen, and Z. Yan. [Nemotron 3 nano: Open, efficient mixture-of-experts hybrid mamba-transformer model for agentic reasoning](https://md.research.elamur.ai/free/en/papers/blakeman-2025-nemotron-3-nano-open-efficient/). arXiv preprint arXiv:2512.20848, 2025. [↗](https://arxiv.org/abs/2512.20848)
62. NVIDIA. Deutsche telekom and NVIDIA launch industrial AI cloud. [https://blogs.nvidia.com/blog/germany-industrial-ai-cloud-launch/](https://blogs.nvidia.com/blog/germany-industrial-ai-cloud-launch/), Nov. 2025. Accessed 2026-07-08. [↗](https://blogs.nvidia.com/blog/germany-industrial-ai-cloud-launch/)
63. S. Oepen, N. Arefev, M. Aulamo, M. Bañón, M. Buljan, L. Burchell, L. Charpentier, P. Chen, M. Fedorova, O. de Gibert, B. Haddow, J. Hajič, J. Helcl, A. Kutuzov, V. Laippala, Z. Li, R. Luukkonen, B. Malik, V. Mikhailov, A. Myntti, D. O’Brien, L. Poláková, S. Pyysalo, G. R. Sánchez, J. Siewert, P. Stepachev, J. Tiedemann, T. Vahtola, D. Variš, F. Vitiugin, T. Vojtěchová, and J. Zaragoza. [Hplt 3.0: Very large-scale multilingual resources for llms and mt. mono- and bi-lingual data, multilingual evaluation, and pre-trained models](https://md.research.elamur.ai/free/en/papers/oepen-2026-hplt-3-0-very-large/). arXiv preprint arXiv:2511.01066, 2026. [↗](https://arxiv.org/abs/2511.01066)
64. T. Olmo, :, A. Ettinger, A. Bertsch, B. Kuehl, D. Graham, D. Heineman, D. Groeneveld, F. Brahman, F. Timbers, H. Ivison, J. Morrison, J. Poznanski, K. Lo, L. Soldaini, M. Jordan, M. Chen, M. Noukhovitch, N. Lambert, P. Walsh, P. Dasigi, R. Berry, S. Malik, S. Shah, S. Geng, S. Arora, S. Gupta, T. Anderson, T. Xiao, T. Murray, T. Romero, V. Graf, A. Asai, A. Bhagia, A. Wettig, A. Liu, A. Rangapur, C. Anastasiades, C. Huang, D. Schwenk, H. Trivedi, I. Magnusson, J. Lochner, J. Liu, L. J. V. Miranda, M. Sap, M. Morgan, M. Schmitz, M. Guerquin, M. Wilson, R. Huff, R. L. Bras, R. Xin, R. Shao, S. Skjonsberg, S. Z. Shen, S. S. Li, T. Wilde, V. Pyatkin, W. Merrill, Y. Chang, Y. Gu, Z. Zeng, A. Sabharwal, L. Zettlemoyer, P. W. Koh, A. Farhadi, N. A. Smith, and H. Hajishirzi. [Olmo 3](https://md.research.elamur.ai/free/en/papers/ettinger-2025-olmo-3/). arXiv preprint arXiv:2512.13961, 2026. [↗](https://arxiv.org/abs/2512.13961)
65. Open Source Initiative. The open source ai definition, version 1.0. [https://opensource.org/ai/open-source-ai-definition](https://opensource.org/ai/open-source-ai-definition), 2024. Accessed: 2026-07-06. [↗](https://opensource.org/ai/open-source-ai-definition)
66. OpenEuroLLM Consortium. OpenEuroLLM: A series of foundation models for transparent ai in europe, 2025. Project website, accessed 2026-06-25. [find](https://scholar.google.com/scholar?q=OpenEuroLLM+Consortium%2E+OpenEuroLLM%3A+A+series+of+foundation+models+for+transparent+ai+in+europe%2C+2025%2E+Project+website%2C+accessed+2026%2D06%2D25%2E)
67. G. Penedo. Finewiki, 2025. Source: Wikimedia Enterprise Snapshot API ([https://api.enterprise.wikimedia.com/v2/snapshots](https://api.enterprise.wikimedia.com/v2/snapshots)). Text licensed under CC BY-SA 4.0 with attribution to Wikipedia contributors. [↗](https://api.enterprise.wikimedia.com/v2/snapshots)
68. G. Penedo, H. Kydlíček, A. Lozhkov, M. Mitchell, C. Raffel, L. Von Werra, T. Wolf, et al. [The fineweb datasets: Decanting the web for the finest text data at scale](https://md.research.elamur.ai/free/en/papers/penedo-2024-fineweb-datasets-decanting-web-finest/). Advances in Neural Information Processing Systems, 37:30811–30849, 2024.
69. Pleias and AI Alliance. SYNTH: An open generalist synthetic dataset for training small reasoning models. [https://huggingface.co/datasets/PleIAs/SYNTH](https://huggingface.co/datasets/PleIAs/SYNTH), 2025. Dataset comprising 79,648,272 text samples (over 41 billion words) derived from the synthetic amplification of 58,698 Wikipedia and Wikibooks articles. Licensed under CC-BY-SA 4.0. [↗](https://huggingface.co/datasets/PleIAs/SYNTH)
70. J. Poznanski, A. Rangapur, J. Borchardt, J. Dunkelberger, R. Huff, D. Lin, C. Wilhelm, K. Lo, and L. Soldaini. [olmocr: Unlocking trillions of tokens in pdfs with vision language models](https://md.research.elamur.ai/free/en/papers/poznanski-2025-olmocr-unlocking-trillions-tokens-pdfs/). arXiv preprint arXiv:2502.18443, 2025. [↗](https://arxiv.org/abs/2502.18443)
71. Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. [find](https://scholar.google.com/scholar?q=Qwen+Team%2E+Qwen3%2E5%3A+Towards+native+multimodal+agents%2C+February+2026%2E)
72. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. Technical report, OpenAI, 2019. [find](https://scholar.google.com/scholar?q=A%2E+Radford%2C+J%2E+Wu%2C+R%2E+Child%2C+D%2E+Luan%2C+D%2E+Amodei%2C+and+I%2E+Sutskever%2E+Language+models+are+unsupervised+multitask+learners%2E+Technical+report%2C+OpenAI%2C+2019%2E)
73. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research, 21(140):1–67, 2020. [↗](https://doi.org/10.48550/arxiv.1910.10683)
74. M. M. Ramos, D. M. Alves, H. Gisserot-Boukhlef, J. Alves, P. H. Martins, P. Fernandes, J. Pombal, N. M. Guerreiro, R. Rei, N. Boizard, A. Farajian, M. Klimaszewski, J. G. C. de Souza, B. Haddow, F. Yvon, P. Colombo, A. Birch, and A. F. T. Martins. Eurollm-22b: Technical report. arXiv preprint arXiv:2602.05879, 2026. [↗](https://arxiv.org/abs/2602.05879)
75. D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman. [Gpqa: A graduate-level google-proof q&a benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/), 2023.
76. A. Romanou, N. Foroutan, A. Sotnikova, S. H. Nelaturu, S. Singh, R. Maheshwary, M. Altomare, Z. Chen, M. Haggag, S. A, A. Amayuelas, A. H. Amirudin, D. Boiko, M. Chang, J. Chim, G. Cohen, A. K. Dalmia, A. Diress, S. Duwal, D. Dzenhaliou, D. Florez, F. Farestam, J. M. Imperial, S. Islam, P. Isotalo, M. Jabbarishiviari, B. F. Karlsson, E. Khalilov, C. Klamm, F. Koto, D. Krzemiński, G. de Melo, S. Montariol, Y. Nan, J. Niklaus, J. Novikova, J. S. Obando Ceron, D. Paul, E. Ploeger, J. Purbey, S. Rajwal, S. S. Ravi, S. Rydell, R. Santhosh, D. Sharma, M. Prifti Skenduli, A. Soltani Moakhar, B. moakhar, A. Tarun, A. T. Wasi, T. Weerasinghe, S. Yilmaz, M. Zhang, I. Schlag, M. Fadaee, S. Hooker, and A. Bosselut. [Include: Evaluating multilingual language understanding with regional knowledge](https://md.research.elamur.ai/free/en/papers/romanou-2024-include-evaluating-multilingual-language-understanding/). In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, editors, International Conference on Learning Representations, volume 2025, pages 83291–83322, 2025. [↗](https://doi.org/10.48550/arxiv.2411.19799)
77. N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017. [↗](https://arxiv.org/abs/1701.06538)
78. W. Shi, A. Bhagia, K. Farhat, N. Muennighoff, J. Morrison, E. Walsh, D. Schwenk, S. Longpre, J. Poznanski, A. Ettinger, et al. Flexolmo: Open language models for flexible data use. Advances in Neural Information Processing Systems, 38:165943–165974, 2026. [find](https://scholar.google.com/scholar?q=W%2E+Shi%2C+A%2E+Bhagia%2C+K%2E+Farhat%2C+N%2E+Muennighoff%2C+J%2E+Morrison%2C+E%2E+Walsh%2C+D%2E+Schwenk%2C+S%2E+Longpre%2C+J%2E+Poznanski%2C+A%2E+Ettinger%2C+et+al%2E+Flexolmo%3A+Open+language+models+for+flexible+data+use%2E+Advances+in+Neural+)
79. L. Soldaini, R. Kinney, A. Bhagia, D. Schwenk, D. Atkinson, R. Authur, B. Bogin, K. Chandu, J. Dumas, Y. Elazar, V. Hofmann, A. Jha, S. Kumar, L. Lucy, X. Lyu, N. Lambert, I. Magnusson, J. Morrison, N. Muennighoff, A. Naik, C. Nam, M. Peters, A. Ravichander, K. Richardson, Z. Shen, E. Strubell, N. Subramani, O. Tafjord, E. Walsh, L. Zettlemoyer, N. Smith, H. Hajishirzi, I. Beltagy, D. Groeneveld, J. Dodge, and K. Lo. Dolma: An open corpus of three trillion tokens for language model pretraining research. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15725–15788, Bangkok, Thailand, Aug. 2024. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2402.00159)
80. D. Su, K. Kong, Y. Lin, J. Jennings, B. Norick, M. Kliegl, M. Patwary, M. Shoeybi, and B. Catanzaro. Nemotron-CC: Transforming Common Crawl into a refined long-horizon pre-training dataset. In W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2459–2475, Vienna, Austria, July 2025. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=D%2E+Su%2C+K%2E+Kong%2C+Y%2E+Lin%2C+J%2E+Jennings%2C+B%2E+Norick%2C+M%2E+Kliegl%2C+M%2E+Patwary%2C+M%2E+Shoeybi%2C+and+B%2E+Catanzaro%2E+Nemotron%2DCC%3A+Transforming+Common+Crawl+into+a+refined+long%2Dhorizon+pre%2Dtraining+dataset%2E+In+W%2E+Che%2C)
81. M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. Le, E. Chi, D. Zhou, and J. Wei. [Challenging BIG-bench tasks and whether chain-of-thought can solve them](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/). In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 13003–13051, Toronto, Canada, July 2023. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2210.09261)
82. G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramé, M. Rivière, L. Rouillard, T. Mesnard, G. Cideron, J. bastien Grill, S. Ramos, E. Yvinec, M. Casbon, E. Pot, I. Penchev, G. Liu, F. Visin, K. Kenealy, L. Beyer, X. Zhai, A. Tsitsulin, R. Busa-Fekete, A. Feng, N. Sachdeva, B. Coleman, Y. Gao, B. Mustafa, I. Barr, E. Parisotto, D. Tian, M. Eyal, C. Cherry, J.-T. Peter, D. Sinopalnikov, S. Bhupatiraju, R. Agarwal, M. Kazemi, D. Malkin, R. Kumar, D. Vilar, I. Brusilovsky, J. Luo, A. Steiner, A. Friesen, A. Sharma, A. Sharma, A. M. Gilady, A. Goedeckemeyer, A. Saade, A. Feng, A. Kolesnikov, A. Bendebury, A. Abdagic, A. Vadi, A. György, A. S. Pinto, A. Das, A. Bapna, A. Miech, A. Yang, A. Paterson, A. Shenoy, A. Chakrabarti, B. Piot, B. Wu, B. Shahriari, B. Petrini, C. Chen, C. L. Lan, C. A. Choquette-Choo, C. Carey, C. Brick, D. Deutsch, D. Eisenbud, D. Cattle, D. Cheng, D. Paparas, D. S. Sreepathihalli, D. Reid, D. Tran, D. Zelle, E. Noland, E. Huizenga, E. Kharitonov, F. Liu, G. Amirkhanyan, G. Cameron, H. Hashemi, H. Klimczak-Plucińska, H. Singh, H. Mehta, H. T. Lehri, H. Hazimeh, I. Ballantyne, I. Szpektor, I. Nardini, J. Pouget-Abadie, J. Chan, J. Stanton, J. Wieting, J. Lai, J. Orbay, J. Fernandez, J. Newlan, J. yeong Ji, J. Singh, K. Black, K. Yu, K. Hui, K. Vodrahalli, K. Greff, L. Qiu, M. Valentine, M. Coelho, M. Ritter, M. Hoffman, M. Watson, M. Chaturvedi, M. Moynihan, M. Ma, N. Babar, N. Noy, N. Byrd, N. Roy, N. Momchev, N. Chauhan, N. Sachdeva, O. Bunyan, P. Botarda, P. Caron, P. K. Rubenstein, P. Culliton, P. Schmid, P. G. Sessa, P. Xu, P. Stanczyk, P. Tafti, R. Shivanna, R. Wu, R. Pan, R. Rokni, R. Willoughby, R. Vallu, R. Mullins, S. Jerome, S. Smoot, S. Girgin, S. Iqbal, S. Reddy, S. Sheth, S. Põder, S. Bhatnagar, S. R. Panyam, S. Eiger, S. Zhang, T. Liu, T. Yacovone, T. Liechty, U. Kalra, U. Evci, V. Misra, V. Roseberry, V. Feinberg, V. Kolesnikov, W. Han, W. Kwon, X. Chen, Y. Chow, Y. Zhu, Z. Wei, Z. Egyed, V. Cotruta, M. Giang, P. Kirk, A. Rao, K. Black, N. Babar, J. Lo, E. Moreira, L. G. Martins, O. Sanseviero, L. Gonzalez, Z. Gleicher, T. Warkentin, V. Mirrokni, E. Senter, E. Collins, J. Barral, Z. Ghahramani, R. Hadsell, Y. Matias, D. Sculley, S. Petrov, N. Fiedel, N. Shazeer, O. Vinyals, J. Dean, D. Hassabis, K. Kavukcuoglu, C. Farabet, E. Buchatskaya, J.-B. Alayrac, R. Anil, Dmitry, Lepikhin, S. Borgeaud, O. Bachem, A. Joulin, A. Andreev, C. Hardin, R. Dadashi, and L. Hussenot. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025. [↗](https://arxiv.org/abs/2503.19786)
83. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. [LLaMA: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023. [↗](https://arxiv.org/abs/2302.13971)
84. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M.-A. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023. [↗](https://arxiv.org/abs/2307.09288)
85. J. Vendrow, E. Vendrow, S. Beery, and A. Madry. Do large language model benchmarks test reliability? arXiv preprint arXiv:2502.03461, 2025. [↗](https://arxiv.org/abs/2502.03461)
86. Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, et al. [Mmlu-pro: A more robust and challenging multi-task language understanding benchmark](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/). Advances in Neural Information Processing Systems, 37:95266–95290, 2024.
87. A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388, 2025. [↗](https://arxiv.org/abs/2505.09388)
88. S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. [OPT: Open pre-trained transformer language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/). arXiv preprint arXiv:2205.01068, 2022. [↗](https://arxiv.org/abs/2205.01068)
89. W. Zhong, R. Cui, Y. Guo, Y. Liang, S. Lu, Y. Wang, A. Saied, W. Chen, and N. Duan. [AGIEval: A human-centric benchmark for evaluating foundation models](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/). In K. Duh, H. Gomez, and S. Bethard, editors, Findings of the Association for Computational Linguistics: NAACL 2024, pages 2299–2314, Mexico City, Mexico, June 2024. Association for Computational Linguistics.
90. C. Zhou, H. Lyu, X. Lin, H. Zhao, J. Guo, X. Zhang, S. Xue, Q. Ma, J. Zhou, Y. Wang, and Z. Liu. Ultradata-math, 2026. [find](https://scholar.google.com/scholar?q=C%2E+Zhou%2C+H%2E+Lyu%2C+X%2E+Lin%2C+H%2E+Zhao%2C+J%2E+Guo%2C+X%2E+Zhang%2C+S%2E+Xue%2C+Q%2E+Ma%2C+J%2E+Zhou%2C+Y%2E+Wang%2C+and+Z%2E+Liu%2E+Ultradata%2Dmath%2C+2026%2E)

*Updated 1 November 2026*