---
title: "KletterMix — paper by Kraus et al. (2026) — summary and ideas"
description: "Open German pretraining data still trails English. 45 references, cited by 1 paper here. Among them: 0."
authors: ["Maurice Kraus", "Ruben Harle", "Sebastian Sztwiertnia", "Abbas Goher Khan", "Mehdi Ali", "Michael Fromm", "Nicolas Flores-Herr", "Kristian Kersting"]
year: 2026
venue: "arXiv:2606.03773"
source: "https://arxiv.org/abs/2606.03773"
doi: "10.48550/arxiv.2606.03773"
arxiv: "2606.03773"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kraus-2026-klettermix-climbing-toward-high-quality/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kraus-2026-klettermix-climbing-toward-high-quality/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/14/) / KletterMix: Climbing Toward High-Quality German Pretraining Data - The Full Report

# KletterMix: Climbing Toward High-Quality German Pretraining Data - The Full Report

**Authors:** [Maurice Kraus](https://md.research.elamur.ai/free/en/authors/maurice-kraus/), [Ruben Harle](https://md.research.elamur.ai/free/en/authors/ruben-harle/), [Sebastian Sztwiertnia](https://md.research.elamur.ai/free/en/authors/sebastian-sztwiertnia/), [Abbas Goher Khan](https://md.research.elamur.ai/free/en/authors/abbas-goher-khan/), [Mehdi Ali](https://md.research.elamur.ai/free/en/authors/mehdi-ali/), [Michael Fromm](https://md.research.elamur.ai/free/en/authors/michael-fromm/), [Nicolas Flores-Herr](https://md.research.elamur.ai/free/en/authors/nicolas-flores-herr/) and [Kristian Kersting](https://md.research.elamur.ai/free/en/authors/kristian-kersting/) · **Year:** 2026 · **Venue:** arXiv:2606.03773 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2606.03773) · [PDF](https://arxiv.org/pdf/2606.03773) · [DOI 10.48550/arxiv.2606.03773](https://doi.org/10.48550/arxiv.2606.03773) · [arXiv 2606.03773](https://arxiv.org/abs/2606.03773) · [OpenAlex](https://openalex.org/W7163351231)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/kraus-2026-klettermix-climbing-toward-high-quality/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Open German pretraining data still trails English. Native German collections are usually smaller, noisier, less documented, and seldom tested by actually training models on them. The authors take a different route: instead of crawling more German web text, they machine-translate a strong English mixture, ClimbMix, into German. The result is KletterMix, a corpus of about 725B tokens. The translation pipeline keeps each document whole and carries over its identifier, metadata, source cluster and topical makeup, so every German document can be traced to its English original.

Translation quality is checked in two steps. COMETKiwi, a reference-free quality estimator, scores a stratified sample, and a cheap proxy model trained on those scores rates the full corpus. Training experiments then test the data: matched 0.6B-parameter pretraining, annealing a fixed checkpoint on different German corpora, and a 7B annealing sweep over how much KletterMix is mixed in. KletterMix yields the best average German scores in these runs, with the clearest gains on HellaSwag and ARC-C. This suggests that translation can carry over a corpus's curation quality, not just its vocabulary.

## Key points
- KletterMix contains roughly 725B German tokens translated from ClimbMix. It is published on Hugging Face as AIML-TUDA/KletterMix.
- The pipeline has five stages:
  - record normalization,
  - length-aware routing into context buckets,
  - document-preserving translation with chunking and a dynamic output budget,
  - shard-wise parallel execution,
  - quality estimation, where COMETKiwi labels a stratified sample and a gradient-boosted proxy scores everything from target-side features.
- The 0.6B comparison scores models with a Core Avg., the mean of four benchmarks: MMLU, PIQA, HellaSwag and ARC-C. On this average KletterMix beats GermanWeb and FineWeb2-DE, but the per-task picture is mixed:
  - GermanWeb is still best on MMLU,
  - FineWeb2-DE stays close on PIQA,
  - KletterMix leads on HellaSwag and ARC-C.
- Two annealing results:
  - At 0.6B, continuing a FineWeb2-DE checkpoint on KletterMix beats continuing it on GermanWeb.
  - At 7B, with a fixed 12B-token budget, raising the KletterMix share from 0% to 20% lifts German scores step by step, while English scores barely change.
- Stated limitations:
  - Each condition was trained only once.
  - Stricter proxy filtering does not improve MMLU steadily.
  - The corpus may inherit the cultural and licensing biases of its English sources.

## Interesting ideas
- Use an expensive quality judge on a stratified sample only, then train a cheap feature-based proxy on its labels to score the whole corpus; this makes quality control affordable at full scale.
- Treat proxy quality scores as a way to rank data under a budget, not as a universal quality measure, because stricter filtering did not help every task.
- Carry document IDs and metadata through every transformation, so the transformed corpus stays aligned with its source and supports controlled side-by-side studies.
- Measure dose-response: vary only the share of the new data against a 0% control under a fixed budget, and track retention on the old capability as well as gains on the new one.
- Report per-task breakdowns next to the aggregate, with standard errors, and explain which data property should help which task instead of claiming a uniform win.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): an LLM teacher could grade a stratified sample of silver parses, and a lightweight model on tree-side features would then flag which of the millions of silver trees are trustworthy.
- **I2** (learn): when cutting silver data by agreement or proxy score, sweep several thresholds and check each downstream metric separately instead of assuming that harder filtering is always better.
- **I3** (back): the text → graph → text compressor and meta-world translation could keep stable sentence and document IDs, so each regenerated text can be compared with its original and errors grouped by source type.
- **I4** (learn): when adding silver or converted-dialect treebanks to parser training, sweep their share from 0% up and watch both the target-domain gain and LAS on the original gold UD treebank.
- **I5** (answer): evaluation on school tests, maths word problems and fable QA could be broken down by question type, linking each gain to the specific world-model or frame improvement expected to cause it.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Technische Universität Darmstadt](https://md.research.elamur.ai/free/en/universities/technische-universitat-darmstadt/) — University, Darmstadt, Germany · 109 works · 35,920 citations
- [Fraunhofer Institute for Intelligent Analysis and Information Systems](https://md.research.elamur.ai/free/en/labs/fraunhofer-institute-for-intelligent-analysis-and-information-systems/) — Research institute, Sankt Augustin, Germany · 9 works · 3,875 citations
- [Lamarr Institute for Machine Learning and Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/lamarr-institute-for-machine-learning-and-artificial-intelligence/) — Research institute, Sankt Augustin, Germany · 6 works · 121 citations

Also on the byline: [L'Alliance Boviteq](https://ror.org/05asf4e91).

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Maurice Kraus, Ruben Harle, Sebastian Sztwiertnia, Abbas Goher Khan, Mehdi Ali, Michael Fromm, Nicolas Flores-Herr and Kristian Kersting. 2026. KletterMix: Climbing Toward High-Quality German Pretraining Data - The Full Report. arXiv:2606.03773.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{kraus2026klettermix,
  title = {{KletterMix: Climbing Toward High-Quality German Pretraining Data - The Full Report}},
  author = {Maurice Kraus, Ruben Harle, Sebastian Sztwiertnia, Abbas Goher Khan, Mehdi Ali, Michael Fromm, Nicolas Flores-Herr and Kristian Kersting},
  year = {2026},
  doi = {10.48550/arxiv.2606.03773},
  eprint = {2606.03773},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2606.03773}
}
```

- [← previous KidsArtBench: Multi-Dimensional Children's Art Evaluation with Attribute-Aware MLLMs](https://md.research.elamur.ai/free/en/papers/ye-2026-kidsartbench-multi-dimensional-children-s/)
- [next → Knowing When Not to Answer: Lightweight KB-Aligned OOD Detection for Safe RAG](https://md.research.elamur.ai/free/en/papers/triantafyllopoulos-2026-knowing-when-not-answer-lightweight/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/14/)

## References

45 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R. Mortensen, Noah A. Smith, and Yulia Tsvetkov. [Do all languages cost the same? tokenization in the era of commercial language models](https://md.research.elamur.ai/free/en/papers/ahia-2023-do-all-languages-cost-same/). In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 9904–9923. Association for Computational Linguistics, 2023.
2. Mehdi Ali, Michael Fromm, Klaudia Thellmann, Jan Ebert, Alexander Arno Weber, Richard Rutmann, Charvi Jain, Max Lübbering, Daniel Steinigen, Johannes Leveling, Katrin Klug, Jasper Schulze Buschhoff, Lena Jurkschat, Hammam Abdelwahab, Benny Jörg Stein, Karl-Heinz Sylla, Pavel Denisov, Nicolo’ Brandizzi, Qasid Saleem, Anirban Bhowmick, Lennard Helmer, Chelsea John, Pedro Ortiz Suarez, Malte Ostendorff, Alex Jude, Lalith Manjunath, Samuel Weinbach, Carolin Penke, Oleg Filatov, Fabio Barth, Paramita Mirza, Lucas Weber, Ines Wendler, Rafet Sifa, Fabian Küch, Andreas Herten, René Jäkel, Georg Rehm, Stefan Kesselheim, Joachim Köhler, and Nicolas Flores-Herr. Teuken-7b-base & teuken-7b-instruct: Towards european llms. In Inês Lynce, Nello Murano, Mauro Vallati, Serena Villata, Federico Chesani, Michela Milano, Andrea Omicini, and Mehdi Dastani, editors, ECAI 2025 - 28th European Conference on Artificial Intelligence, 25-30 October 2025, Bologna, Italy - Including 14th Conference on Prestigious Applications of Intelligent Systems (PAIS 2025), Frontiers in Artificial Intelligence and Applications, pages 4321–4329. IOS Press, 2025. [↗](https://doi.org/10.3233/faia251328)
3. Eleftherios Avramidis, Annika Grützner-Zahn, Manuel Brack, Patrick Schramowski, Pedro Ortiz Suarez, Malte Ostendorff, Fabio Barth, Shushen Manakhimova, Vivien Macketanz, Georg Rehm, and Kristian Kersting. Occiglot at WMT24: european open-source large language models evaluated on translation. In Barry Haddow, Tom Kocmi, Philipp Koehn, and Christof Monz, editors, Proceedings of the Ninth Conference on Machine Translation, WMT 2024, Miami, FL, USA, November 15-16, 2024, pages 292–298. Association for Computational Linguistics, 2024. [find](https://scholar.google.com/scholar?q=Eleftherios+Avramidis%2C+Annika+Gr%C3%BCtzner%2DZahn%2C+Manuel+Brack%2C+Patrick+Schramowski%2C+Pedro+Ortiz+Suarez%2C+Malte+Ostendorff%2C+Fabio+Barth%2C+Shushen+Manakhimova%2C+Vivien+Macketanz%2C+Georg+Rehm%2C+and+Kristian+Kerst)
4. Emily M. Bender and Batya Friedman. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Trans. Assoc. Comput. Linguistics, 6:587–604, 2018. [↗](https://doi.org/10.1162/tacl_a_00041)
5. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. [Piqa: Reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020.
6. Thomas F. Burns, Letitia Parcalabescu, Stephan Wäldchen, Michael Barlow, Gregor Ziegltrum, Volker Stampa, Bastian Harren, and Björn Deiseroth. [Aleph-alpha-germanweb: Improving german-language LLM pre-training with model-based data curation and synthetic data generation](https://md.research.elamur.ai/free/en/papers/burns-2026-aleph-alpha-germanweb-improving-german/). In Vera Demberg, Kentaro Inui, and Lluís Marquez, editors, Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2026 - Volume 1: Long Papers, Rabat, Morocco, March 24-29, 2026, pages 1267–1283. Association for Computational Linguistics, 2026.
7. Tyler A. Chang, Catherine Arnett, Abdelrahman Eldesokey, Abdelrahman Sadallah, Abeer Kashar, Abolade Daud, Abosede Grace Olanihun, Adamu Labaran Mohammed, Adeyemi Praise, Adhikarinayum Meerajita Sharma, Aditi Gupta, Afitab Iyigun, Afonso Simplício, Ahmed Essouaied, Aicha Chorana, Akhil Eppa, Akintunde Oladipo, Akshay Ramesh, Aleksei Dorkin, Alfred Malengo Kondoro, Alham Fikri Aji, Ali Eren Çetintaş, Allan Hanbury, Alou Dembele, Alp Niksarli, Álvaro Arroyo, Amin Bajand, Amol Khanna, Ana Chkhaidze, Ana Condez, Andiswa Mkhonto, Andrew Hoblitzell, Andrew Tran, Angelos Poulis, Anirban Majumder, Anna Vacalopoulou, Annette Kuuipolani Kanahele Wong, Annika Simonsen, Anton Kovalev, Ashvanth. S, Ayodeji Joseph Lana, Barkin Kinay, Bashar Alhafni, Benedict Cibalinda Busole, Bernard Ghanem, Bharti Nathani, Biljana Stojanovska Ðurić, Bola Agbonile, Bragi Bergsson, Bruce Torres Fischer, Burak Tutar, Burcu Alakuş Çınar, Cade J. Kanoniakapueo Kane, Can Udomcharoenchaikit, Catherine Arnett, Chadi Helwe, Chaithra Reddy Nerella, Chen Cecilia Liu, Chiamaka Glory Nwokolo, Cristina España-Bonet, Cynthia Amol, DaeYeop Lee, Dana Arad, Daniil Dzenhaliou, Daria Pugacheva, Dasol Choi, Daud Abolade, David Liu, David Semedo, Deborah Popoola, Deividas Mataciunas, Delphine Nyaboke, Dhyuthy Krishna Kumar, Diogo Glória-Silva, Diogo Tavares, Divyanshu Goyal, DongGeon Lee, Ebele Nwamaka Anajemba, Egonu Ngozi Grace, Elena Mickel, Elena Tutubalina, Elias Herranen, Emile Anand, Emmanuel Habumuremyi, Emuobonuvie Maria Ajiboye, Eryawan Presma Yulianrifat, Esther Adenuga, Ewa Rudnicka, Faith Olabisi Itiola, Faran Taimoor Butt, Fathima Thekkekara, Fatima Haouari, Filbert Aurelian Tjiaranata, Firas Laakom, and Francesca Grasso et al. Global piqa: Evaluating physical commonsense reasoning across 100+ languages and cultures, 2025. URL https://arxiv.org/abs/2510.24081. [↗](https://arxiv.org/abs/2510.24081)
8. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/), 2018. arXiv:1803.05457. [↗](https://arxiv.org/abs/1803.05457)
9. Amin Dada, Aokun Chen, Cheng Peng, Kaleb E. Smith, Ahmad Idrissi-Yaghir, Constantin Seibold, Jianning Li, Lars Heiliger, Christoph M. Friedrich, Daniel Truhn, Jan Egger, Jiang Bian, Jens Kleesiek, and Yonghui Wu. On the impact of cross-domain data on german language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, Findings of ACL, pages 13801–13813. Association for Computational Linguistics, 2023. [↗](https://doi.org/10.48550/arxiv.2310.07321)
10. Ona de Gibert, Graeme Nail, Nikolay Arefyev, Marta Bañón, Jelmer van der Linde, Shaoxiong Ji, Jaume Zaragoza-Bernabeu, Mikko Aulamo, Gema Ramírez-Sánchez, Andrey Kutuzov, Sampo Pyysalo, Stephan Oepen, and Jörg Tiedemann. [A new massive multilingual dataset for high-performance language technologies](https://md.research.elamur.ai/free/en/papers/gibert-2024-new-massive-multilingual-dataset-high/). In Nicoletta Calzolari, Min-Yen Kan, Véronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, editors, Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy, pages 1116–1128. ELRA and ICCL, 2024.
11. Daniel Deutsch, Eleftheria Briakou, Isaac Rayburn Caswell, Mara Finkelstein, Rebecca Galor, Juraj Juraska, Geza Kovacs, Alison Lui, Ricardo Rei, Jason Riesa, Shruti Rijhwani, Parker Riley, Elizabeth Salesky, Firas Trabelsi, Stephanie Winkler, Biao Zhang, and Markus Freitag. [WMT24++: Expanding the language coverage of WMT24 to 55 languages & dialects](https://md.research.elamur.ai/free/en/papers/deutsch-2025-wmt24-expanding-language-coverage-wmt24/). In Findings of the Association for Computational Linguistics: ACL 2025, 2025. [↗](https://doi.org/10.48550/arxiv.2502.12404)
12. Shizhe Diao, Yu Yang, Yonggan Fu, Xin Dong, Dan Su, Markus Kliegl, Zijia Chen, Peter Belcak, Yoshi Suhara, Hongxu Yin, Mostofa Patwary, Yingyan, Lin, Jan Kautz, and Pavlo Molchanov. Nemotron-climb: Clustering-based iterative data mixture bootstrapping for language model pre-training, 2025. arXiv:2504.13161. [↗](https://arxiv.org/abs/2504.13161)
13. Jesse Dodge, Maarten Sap, Ana Marasovic, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 1286–1305. Association for Computational Linguistics, 2021. [find](https://scholar.google.com/scholar?q=Jesse+Dodge%2C+Maarten+Sap%2C+Ana+Marasovic%2C+William+Agnew%2C+Gabriel+Ilharco%2C+Dirk+Groeneveld%2C+Margaret+Mitchell%2C+and+Matt+Gardner%2E+Documenting+large+webtext+corpora%3A+A+case+study+on+the+colossal+clean+cra)
14. Meet Doshi, Raj Dabre, and Pushpak Bhattacharyya. Pretraining language models using translationese. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pages 5843–5862. Association for Computational Linguistics, 2024. [find](https://scholar.google.com/scholar?q=Meet+Doshi%2C+Raj+Dabre%2C+and+Pushpak+Bhattacharyya%2E+Pretraining+language+models+using+translationese%2E+In+Yaser+Al%2DOnaizan%2C+Mohit+Bansal%2C+and+Yun%2DNung+Chen%2C+editors%2C+Proceedings+of+the+2024+Conference+on)
15. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020. arXiv:2101.00027. [↗](https://arxiv.org/abs/2101.00027)
16. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Baber+Abbasi%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Alain+Le+Noac%E2%80%99h%2C+Haonan+Li%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Chris+Oci)
17. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna M. Wallach, Hal Daumé III, and Kate Crawford. Datasheets for datasets. Commun. ACM, 64(12): 86–92, 2021. [↗](https://doi.org/10.1145/3458723)
18. Lukas Gienapp, Christopher Schröder, Stefan Schweter, Christopher Akiki, Ferdinand Schlatt, Arden Zimmermann, Phillipe Genêt, and Martin Potthast. [The german commons - 154 billion tokens of openly licensed text for german language models](https://md.research.elamur.ai/free/en/papers/gienapp-2025-german-commons-154-billion-tokens/), 2025. arXiv:2510.13996. [↗](https://arxiv.org/abs/2510.13996)
19. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. [↗](https://doi.org/10.48550/arxiv.2009.03300)
20. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, 2022. [find](https://scholar.google.com/scholar?q=Jordan+Hoffmann%2C+Sebastian+Borgeaud%2C+Arthur+Mensch%2C+Elena+Buchatskaya%2C+Trevor+Cai%2C+Eliza+Rutherford%2C+Diego+de+Las+Casas%2C+Lisa+Anne+Hendricks%2C+Johannes+Welbl%2C+Aidan+Clark%2C+Tom+Hennigan%2C+Eric+Noland%2C+Ka)
21. Amir Hossein Kargaran, Ayyoob Imani, François Yvon, and Hinrich Schütze. [Glotlid: Language identification for low-resource languages](https://md.research.elamur.ai/free/en/papers/kargaran-2023-glotlid-language-identification-low-resource/). In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, Findings of ACL, pages 6155–6218. Association for Computational Linguistics, 2023.
22. Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Javier Ortiz Suárez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Balli, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi. [Quality at a glance: An audit of web-crawled multilingual datasets](https://md.research.elamur.ai/free/en/papers/kreutzer-2021-quality-glance-audit-web-crawled/). Trans. Assoc. Comput. Linguistics, 10:50–72, 2022. [↗](https://doi.org/10.1162/tacl_a_00447)
23. Pierre-Carl Langlais, Pavel Chizhov, Catherine Arnett, Carlos Rosas Hinostroza, Mattia Nee, Eliot Krzysztof Jones, Irène Girard, David Mach, Anastasia Stasenko, and Ivan P. Yamshchikov. Common corpus: The largest collection of ethical data for LLM pre-training. In The Fourteenth International Conference on Learning Representations, 2026. [↗](https://doi.org/10.48550/arxiv.2506.01732)
24. Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Sasko, Quentin Lhoest, Angelina McMillan-Major, Gérard Dupont, Stella Biderman, Anna Rogers, Loubna Ben Allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen, Somaieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas, Tristan Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Muñoz, Jian Zhu, Daniel van Strien, Zaid Alyafeai, Khalid Almubarak, Minh Chien Vu, Itziar Gonzalez-Dios, Aitor Soroa, Kyle Lo, Manan Dey, Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Ifeoluwa Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepercq, Suzana Ilic, Margaret Mitchell, Alexandra Sasha Luccioni, and Yacine Jernite. [The bigscience ROOTS corpus: A 1.6tb composite multilingual dataset](https://md.research.elamur.ai/free/en/papers/lauren-ccon-2023-bigscience-roots-corpus-1-6tb/). In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, 2022. [↗](https://doi.org/10.60692/8qmj2-d1w68)
25. Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Kumar Guha, Sedrick Scott Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee F. Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng-Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah M. Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Raghavi Chandu, Thao Nguyen, Igor Vasiljevic, Sham M. Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El-Nouby, Hadi Pouransari, Alexander Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alex Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar. [Datacomp-lm: In search of the next generation of training sets for language models](https://md.research.elamur.ai/free/en/papers/li-2024-datacomp-lm-search-next-generation/). In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024.
26. Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M. Guerreiro, Ricardo Rei, Duarte M. Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, Pierre Colombo, Barry Haddow, José G. C. de Souza, Alexandra Birch, and André F. T. Martins. [EuroLLM: Multilingual language models for europe](https://md.research.elamur.ai/free/en/papers/martins-2024-eurollm-multilingual-language-models-europe/), 2024. arXiv:2409.16235. [↗](https://arxiv.org/abs/2409.16235)
27. Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. [CulturaX: A cleaned, enormous, and multilingual dataset for large language models in 167 languages](https://md.research.elamur.ai/free/en/papers/nguyen-2023-culturax-cleaned-enormous-multilingual-dataset/). In Nicoletta Calzolari, Min-Yen Kan, Véronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, editors, Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy, pages 4226– 4237. ELRA and ICCL, 2024.
28. Stephan Oepen, Nikolay Arefev, Mikko Aulamo, Marta Bañón, Maja Buljan, Laurie Burchell, Lucas Charpentier, Pinzhen Chen, Mariya Fedorova, Ona de Gibert, et al. Hplt 3.0: Very large-scale multilingual resources for llm and mt. mono-and bi-lingual data, multilingual evaluation, and pre-trained models. arXiv preprint arXiv:2511.01066, 2025. [↗](https://arxiv.org/abs/2511.01066)
29. Team Olmo, :, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, Jacob Morrison, Jake Poznanski, Kyle Lo, Luca Soldaini, Matt Jordan, Mayee Chen, Michael Noukhovitch, Nathan Lambert, Pete Walsh, Pradeep Dasigi, Robert Berry, Saumya Malik, Saurabh Shah, Scott Geng, Shane Arora, Shashank Gupta, Taira Anderson, Teng Xiao, Tyler Murray, Tyler Romero, Victoria Graf, Akari Asai, Akshita Bhagia, Alexander Wettig, Alisa Liu, Aman Rangapur, Chloe Anastasiades, Costa Huang, Dustin Schwenk, Harsh Trivedi, Ian Magnusson, Jaron Lochner, Jiacheng Liu, Lester James V. Miranda, Maarten Sap, Malia Morgan, Michael Schmitz, Michal Guerquin, Michael Wilson, Regan Huff, Ronan Le Bras, Rui Xin, Rulin Shao, Sam Skjonsberg, Shannon Zejiang Shen, Shuyue Stella Li, Tucker Wilde, Valentina Pyatkin, Will Merrill, Yapei Chang, Yuling Gu, Zhiyuan Zeng, Ashish Sabharwal, Luke Zettlemoyer, Pang Wei Koh, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. [Olmo 3](https://md.research.elamur.ai/free/en/papers/ettinger-2025-olmo-3/), 2026. URL https://arxiv.org/abs/2512.13961. [↗](https://arxiv.org/abs/2512.13961)
30. Guilherme Penedo, Hynek Kydlíček, Vinko Sabolčec, Bettina Messmer, Negar Foroutan, Amir Hossein Kargaran, Colin Raffel, Martin Jaggi, Leandro Von Werra, and Thomas Wolf. FineWeb2: One pipeline to scale them all — adapting pre-training data processing to every language. In Second Conference on Language Modeling, 2025. — [FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language](https://md.research.elamur.ai/free/en/papers/penedo-2025-fineweb2-one-pipeline-scale-them/) [↗](https://doi.org/10.48550/arxiv.2506.20920)
31. Jan Pfister, Julia Wunderle, and Andreas Hotho. [Llämmlein: Transparent, compact and competitive german-only language models from scratch](https://md.research.elamur.ai/free/en/papers/pfister-2025-llammlein-transparent-compact-competitive-german/). In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, pages 2227–2246. Association for Computational Linguistics, 2025.
32. Björn Plüster. Leolm: Igniting german-language llm research. LAION Blog, September 2023. URL [https://laion.ai/blog/leo-lm/](https://laion.ai/blog/leo-lm/). Accessed: May 5, 2026. [↗](https://laion.ai/blog/leo-lm/)
33. Ricardo Rei, Marcos V. Treviso, Nuno Miguel Guerreiro, Chrysoula Zerva, Ana C. Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte M. Alves, Luísa Coheur, Alon Lavie, and André F. T. Martins. [CometKiwi: Ist-unbabel 2022 submission for the quality estimation shared task](https://md.research.elamur.ai/free/en/papers/rei-2022-cometkiwi-ist-unbabel-2022-submission/). In Philipp Koehn, Loïc Barrault, Ondrej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno-Yepes, Tom Kocmi, André F. T. Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Marco Turchi, and Marcos Zampieri, editors, Proceedings of the Seventh Conference on Machine Translation, WMT 2022, Abu Dhabi, United Arab Emirates (Hybrid), December 7-8, 2022, pages 634–645. Association for Computational Linguistics, 2022. [↗](https://doi.org/10.48550/arxiv.2209.06243)
34. Phillip Rust, Jonas Pfeiffer, Ivan Vulic, Sebastian Ruder, and Iryna Gurevych. [How good is your tokenizer? on the monolingual performance of multilingual language models](https://md.research.elamur.ai/free/en/papers/rust-2021-how-good-is-your-tokenizer/). In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 3118–3135. Association for Computational Linguistics, 2021.
35. Raphael Scheible, Johann Frei, Fabian Thomczyk, Henry He, Patric Tippmann, Jochen Knaus, Victor Jaravine, Frank Kramer, and Martin Boeker. Gottbert: a pure german language model. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pages 21237–21250. Association for Computational Linguistics, 2024. [find](https://scholar.google.com/scholar?q=Raphael+Scheible%2C+Johann+Frei%2C+Fabian+Thomczyk%2C+Henry+He%2C+Patric+Tippmann%2C+Jochen+Knaus%2C+Victor+Jaravine%2C+Frank+Kramer%2C+and+Martin+Boeker%2E+Gottbert%3A+a+pure+german+language+model%2E+In+Yaser+Al%2DOnaizan%2C+)
36. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training multi-billion parameter language models using model parallelism, 2020. arXiv:1909.08053. [↗](https://arxiv.org/abs/1909.08053)
37. Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Sebastian Ruder, Wei-Yin Ko, Antoine Bosselut, Alice Oh, André F. T. Martins, Leshem Choshen, Daphne Ippolito, Enzo Ferrante, Marzieh Fadaee, Beyza Ermis, and Sara Hooker. [Global MMLU: understanding and addressing cultural and linguistic biases in multilingual evaluation](https://md.research.elamur.ai/free/en/papers/singh-2025-global-mmlu-understanding-addressing-cultural/). In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, pages 18761–18799. Association for Computational Linguistics, 2025. [↗](https://doi.org/10.48550/arxiv.2412.03304)
38. Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Raghavi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an open corpus of three trillion tokens for language model pretraining research. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 15725–15788. Association for Computational Linguistics, 2024. [↗](https://doi.org/10.48550/arxiv.2402.00159)
39. Pedro Javier Ortiz Suárez, Laurent Romary, and Benoît Sagot. [A monolingual approach to contextualized word embeddings for mid-resource languages](https://md.research.elamur.ai/free/en/papers/suarez-2020-monolingual-approach-contextualized-word-embeddings/). In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 1703–1714. Association for Computational Linguistics, 2020.
40. Klaudia Thellmann, Bernhard Stadler, Michael Fromm, Jasper Schulze Buschhoff, Alex Jude, Fabio Barth, Johannes Leveling, Nicolas Flores-Herr, Joachim Köhler, René Jäkel, and Mehdi Ali. Towards multilingual llm evaluation for european languages, 2024. URL https://arxiv.org/abs/2410.08928. [↗](https://arxiv.org/abs/2410.08928)
41. Brian Thompson, Mehak Preet Dhaliwal, Peter Frisch, Tobias Domhan, and Marcello Federico. A shocking amount of the web is machine translated: Insights from multi-way parallelism. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, Findings of ACL, pages 1763–1775. Association for Computational Linguistics, 2024. [find](https://scholar.google.com/scholar?q=Brian+Thompson%2C+Mehak+Preet+Dhaliwal%2C+Peter+Frisch%2C+Tobias+Domhan%2C+and+Marcello+Federico%2E+A+shocking+amount+of+the+web+is+machine+translated%3A+Insights+from+multi%2Dway+parallelism%2E+In+Lun%2DWei+Ku%2C+Andre+)
42. Jiayi Wang, Yao Lu, Maurice Weber, Max Ryabinin, David Ifeoluwa Adelani, Yihong Chen, Raphael Tang, and Pontus Stenetorp. Multilingual language model pretraining using machine-translated data. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025, Suzhou, China, November 4-9, 2025, pages 28087–28107. Association for Computational Linguistics, 2025. [↗](https://doi.org/10.48550/arxiv.2502.13252)
43. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. [mT5: A massively multilingual pre-trained text-to-text transformer](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/). In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 483–498. Association for Computational Linguistics, 2021. [↗](https://doi.org/10.48550/arxiv.2010.11934)
44. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/), 2025. arXiv:2505.09388. [↗](https://arxiv.org/abs/2505.09388)
45. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: can a machine really finish your sentence? In Anna Korhonen, David R. Traum, and Lluís Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics, 2019. Appendix organization. The appendix follows the order of the main paper. App. A expands the KletterMix pipeline in Sec. 3; App. B supports the corpus diagnostics in Sec. 4; and App. C gives the training configurations and extended results for Sec. 5, including the full OLMo 3 7B annealing setup and per-task plots in App. C.3. The NeurIPS checklist follows the appendices. [↗](https://doi.org/10.48550/arxiv.1905.07830)

*Updated 1 November 2026*