---
title: "Fineweb-Edu-Ar — paper by Alrashed et al. (2024)"
description: "Arabic has little openly available, high-quality, education-oriented text with general (not region-specific)… 21 references, cited by 2 papers here."
authors: ["Sultan Alrashed", "Dmitrii Khizbullin", "David R. Pugh"]
year: 2024
venue: "arXiv:2411.06402"
source: "https://arxiv.org/abs/2411.06402"
doi: "10.48550/arxiv.2411.06402"
arxiv: "2411.06402"
licence: "CC BY-SA 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/alrashed-2024-fineweb-edu-ar-machine-translated/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/alrashed-2024-fineweb-edu-ar-machine-translated/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/12/) / Fineweb-Edu-Ar: Machine-translated Corpus to Support Arabic Small Language Models

# Fineweb-Edu-Ar: Machine-translated Corpus to Support Arabic Small Language Models

**Authors:** [Sultan Alrashed](https://md.research.elamur.ai/free/en/authors/sultan-alrashed/), [Dmitrii Khizbullin](https://md.research.elamur.ai/free/en/authors/dmitrii-khizbullin/) and David R. Pugh · **Year:** 2024 · **Venue:** arXiv:2411.06402 · **Licence:** [CC BY-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2411.06402) · [PDF](https://arxiv.org/pdf/2411.06402) · [DOI 10.48550/arxiv.2411.06402](https://doi.org/10.48550/arxiv.2411.06402) · [arXiv 2411.06402](https://arxiv.org/abs/2411.06402) · [OpenAlex](https://openalex.org/W4404401697)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [2](https://md.research.elamur.ai/free/en/papers/alrashed-2024-fineweb-edu-ar-machine-translated/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Arabic has little openly available, high-quality, education-oriented text with general (not region-specific) knowledge, and this limits the training of small language models (SLMs) in Arabic. The authors address this by machine-translating the deduplicated FineWeb-Edu portion of HuggingFaceTB/smollm-corpus, the data behind SmolLM, from English into Modern Standard Arabic. To pick a translator, they compared 12 models: encoder-decoder MT systems (NLLB variants, OPUS) and multilingual decoder-only LLMs (Aya-23, Llama, Qwen 2.5, Mistral, Command-R, Tower). Each model's output was scored by GPT-4o acting as a judge on a small fixed set of passages, and its runtime was measured on an A100. They then selected a model on the quality-versus-speed Pareto frontier that fit the compute budget. The result, FineWeb-Edu-Ar, is about 202B tokens (NLLB tokenizer), which the authors call the largest public machine-translated Arabic corpus. It gives Arabic SLM builders a large general-knowledge pretraining resource. Its usefulness for foundation models is still untested, and the authors acknowledge both translation noise and an English-centric knowledge domain.

## Key points
- The translator was facebook/nllb-200-distilled-600M. It translated 189,405,457 passages in 1813 shards per language, using 20 days on 24 A100 GPUs (480 GPU-days) against a budget of 500 GPU-days.
- The corpus size depends heavily on the tokenizer. The Arabic side is 202B tokens with the NLLB tokenizer but 551B GPT-2 tokens, while the English side is 192B GPT-2 tokens.
- The GPT-4o judge rubric has three parts: accuracy (0–3), grammar and syntax (0–3), and fluency and style (0–2). It was applied to 3 passages, and translations were repeated 10 times at temperature 0.1 with the scores averaged, giving a maximum of 24. Four models fell on the Pareto frontier.
- Input texts were cut into 200-token chunks without overlap and translated in batches, then concatenated. A sentence-aligned split was rejected because flash_attention_2 handles padding tokens poorly.
- The release keeps English and Arabic passages aligned at the same indices and uses the CC-BY-NC-4.0 license. Code is available at SulRash/llm-translation-pipeline.

## Interesting ideas
- The authors map data scarcity on a 2×2 grid of language (English or Arabic) against knowledge type (culture-agnostic or culture-specific), which locates the actual gap (generic knowledge in Arabic) and suggests which sources can fill it.
- They choose the tool by plotting quality against runtime, keeping only Pareto-optimal candidates, and then applying a hard compute budget, so the choice is a defensible cost/quality trade-off rather than simply "the best score."
- The LLM judge scores separate weighted dimensions instead of giving one overall grade, and repeated low-temperature runs are averaged to steady the score on a tiny fixed benchmark.
- The release stores the source passage and its translation side by side at matching indices, which makes the corpus usable both as pretraining data and as a parallel resource.
- They let a runtime constraint drive the preprocessing design, choosing fixed non-overlapping chunks because padding breaks fast attention, and they state this trade-off openly.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (world): A similar grid of domain against source type (WordNet/FrameNet/Wikidata versus text-derived facts) could show which parts of the compiled knowledge layer are thin and which source should fill each one.
- **I2** (learn): LLM teacher tiers could be chosen the same way, by plotting agreement with gold UD trees or frames against cost per sentence and taking the cheapest model on the frontier.
- **I3** (learn): Silver frames or generated explanations could be judged on separate axes such as role correctness, completeness and fluency, with repeated averaged runs, so that judge noise is not mistaken for a real model difference.
- **I4** (back): Keeping each input text aligned with its text → graph → text reconstruction at the same index would make it easy to compare them and find where meaning is lost in the compressor or in meta-world translation.
- **I5** (tree): In batch training of gparse on the GPU, choosing fixed-length or length-bucketed batches around padding costs, and recording that choice, would help keep throughput comparisons reproducible.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [King Abdullah University of Science and Technology](https://md.research.elamur.ai/free/en/universities/king-abdullah-university-of-science-and-technology/) — University, Jeddah, Saudi Arabia · 27 works · 1,084 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works

## Cite

Cite as: Sultan Alrashed, Dmitrii Khizbullin and David R. Pugh. 2024. Fineweb-Edu-Ar: Machine-translated Corpus to Support Arabic Small Language Models. arXiv:2411.06402.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{alrashed2024fineweb,
  title = {{Fineweb-Edu-Ar: Machine-translated Corpus to Support Arabic Small Language Models}},
  author = {Sultan Alrashed, Dmitrii Khizbullin and David R. Pugh},
  year = {2024},
  doi = {10.48550/arxiv.2411.06402},
  eprint = {2411.06402},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2411.06402}
}
```

- [← previous Evaluation of Semantic Search and its Role in Retrieved-Augmented-Generation (RAG) for Arabic Language](https://md.research.elamur.ai/free/en/papers/mahboub-2024-evaluation-semantic-search-role-retrieved/)
- [next → From Linguistics to Practice: a Case Study of Offensive Language Taxonomy in Hebrew](https://md.research.elamur.ai/free/en/papers/liebeskind-2024-linguistics-practice-case-study-offensive/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/12/)

## References

21 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.
2. Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, wen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vítor Albiero, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). Preprint, arXiv:2407.21783. [↗](https://arxiv.org/abs/2207.04672)
3. May Farhat, Said Taghadouini, Oskar Hallström, and Sonja Hajri-Gabouj. 2024. Arabicweb24: Creating a high quality arabic web-only pre-training dataset. [find](https://scholar.google.com/scholar?q=May+Farhat%2C+Said+Taghadouini%2C+Oskar+Hallstr%C3%B6m%2C+and+Sonja+Hajri%2DGabouj%2E+2024%2E+Arabicweb24%3A+Creating+a+high+quality+arabic+web%2Donly+pre%2Dtraining+dataset%2E)
4. Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166. [↗](https://arxiv.org/abs/2302.04166)
5. Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. 2023. Textbooks Are All You Need. arXiv preprint. ArXiv:2306.11644 [cs]. [↗](https://arxiv.org/abs/2306.11644)
6. Tom Gunter, Zirui Wang, Chong Wang, Ruoming Pang, Andy Narayanan, Aonan Zhang, Bowen Zhang, Chen Chen, Chung-Cheng Chiu, David Qiu, Deepak Gopinath, Dian Ang Yap, Dong Yin, Feng Nan, Floris Weers, Guoli Yin, Haoshuo Huang, Jianyu Wang, Jiarui Lu, John Peebles, Kewei Ye, Mark Lee, Nan Du, Qibin Chen, Quentin Keunebroek, Sam Wiseman, Syd Evans, Tao Lei, Vivek Rathod, Xiang Kong, Xianzhi Du, Yanghao Li, Yongqiang Wang, Yuan Gao, Zaid Ahmed, Zhaoyang Xu, Zhiyun Lu, Al Rashid, Albin Madappally Jose, Alec Doane, Alfredo Bencomo, Allison Vanderby, Andrew Hansen, Ankur Jain, Anupama Mann Anupama, Areeba Kamal, Bugu Wu, Carolina Brum, Charlie Maalouf, Chinguun Erdenebileg, Chris Dulhanty, Dominik Moritz, Doug Kang, Eduardo Jimenez, Evan Ladd, Fang Shi, Felix Bai, Frank Chu, Fred Hohman, Hadas Kotek, Hannah Gillis Coleman, Jane Li, Jeffrey P. Bigham, Jeffery Cao, Jeff Lai, Jessica Cheung, Jiulong Shan, Joe Zhou, John Li, Jun Qin, Karanjeet Singh, Karla Vega, Kelvin Zou, Laura Heckman, Lauren Gardiner, Margit Bowler, Maria Cordell, Meng Cao, Nicole Hay, Nilesh Shahdadpuri, Otto Godwin, Pranay Dighe, Pushyami Rachapudi, Ramsey Tantawi, Roman Frigg, Sam Davarnia, Sanskruti Shah, Saptarshi Guha, Sasha Sirovica, Shen Ma, Shuang Ma, Simon Wang, Sulgi Kim, Suma Jayaram, Vaishaal Shankar, Varsha Paidi, Vivek Kumar, Xin Wang, Xin Zheng, Walker Cheng, Yael Shrager, Yang Ye, Yasu Tanaka, Yihao Guo, Yunsong Meng, Zhaoping Luo, Ouyang Zhi, Alp Aygar, Alvin Wan, Andrew D. Walkingshaw, Tzu-Hsiang Lin, Arsalan Farooq, Brent Ramerth, Colorado Reed, Chris Bartels, Chris Chaney, David Riazati, Eric Liang Yang, Erin Feldman, Gabriel Hochstrasser, Guillaume Seguin, Irina Belousova, Joris Pelemans, Karen Yang, Keivan Alizadeh Vahid, Liangliang Cao, Mahyar Najibi, Marco Zuliani, Max Horton, Minsik Cho, Nikhil Bhendawade, Patrick Dong, Piotr Maj, Pulkit Agrawal, Qi Shan, Qichen Fu, Regan Poston, Sam Xu, Shuangning Liu, Sushma Rao, Tashweena Heeramun, Thomas Merth, Uday Rayala, Victor Cui, Vivek Rangarajan Sridhar, Wencong Zhang, Wenqi Zhang, Wentao Wu, Xingyu Zhou, Xinwen Liu, Yang Zhao, Yin Xia, Zhile Ren, and Zhongzheng Ren. 2024. Apple intelligence foundation language models. ArXiv, abs/2407.21075. [↗](https://arxiv.org/abs/2407.21075)
7. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. [Training compute-optimal large language models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/). arXiv preprint arXiv:2203.15556. [↗](https://arxiv.org/abs/2203.15556)
8. Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint. ArXiv:2307.09288 [cs]. [↗](https://arxiv.org/abs/2307.09288)
9. Achintya Kundu, Rhui Dih Lee, Laura Wynter, Raghu Kiran Ganti, and Mayank Mishra. 2024. Enhancing training efficiency using packing with flash attention. Preprint, arXiv:2407.09105. [↗](https://arxiv.org/abs/2407.09105)
10. Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. 2019. [Quantifying the carbon emissions of machine learning](https://md.research.elamur.ai/free/en/papers/lacoste-2019-quantifying-carbon-emissions-machine-learning/). arXiv preprint arXiv:1910.09700. [↗](https://arxiv.org/abs/1910.09700)
11. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2019. [Ccnet: Extracting high quality monolingual datasets from web crawl data](https://md.research.elamur.ai/free/en/papers/wenzek-2020-ccnet-extracting-high-quality-monolingual/). arXiv preprint arXiv:1911.00359. [↗](https://arxiv.org/abs/1911.00359)
12. Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. 2024. Fineweb-edu. [find](https://scholar.google.com/scholar?q=Anton+Lozhkov%2C+Loubna+Ben+Allal%2C+Leandro+von+Werra%2C+and+Thomas+Wolf%2E+2024%2E+Fineweb%2Dedu%2E)
13. Zhenyan Lu, Xiang Li, Dongqi Cai, Rongjie Yi, Fangming Liu, Xiwen Zhang, Nicholas D. Lane, and Mengwei Xu. 2024. [Small language models: Survey, measurements, and insights](https://md.research.elamur.ai/free/en/papers/lu-2024-small-language-models-survey-measurements/). Preprint, arXiv:2409.15790. [↗](https://arxiv.org/abs/2409.15790)
14. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. 2024a. Qwen2 technical report. Preprint, arXiv:2407.10671. [↗](https://arxiv.org/abs/2407.10671)
15. Meta AI. 2024. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. [find](https://scholar.google.com/scholar?q=Meta+AI%2E+2024%2E+Llama+3%2E2%3A+Revolutionizing+edge+ai+and+vision+with+open%2C+customizable+models%2E)
16. Mistral AI. 2024. Mistral ai models overview. [find](https://scholar.google.com/scholar?q=Mistral+AI%2E+2024%2E+Mistral+ai+models+overview%2E)
17. OpenCSG. 2024. Chinese fineweb edu. Accessed: 2024-10-07. [find](https://scholar.google.com/scholar?q=OpenCSG%2E+2024%2E+Chinese+fineweb+edu%2E+Accessed%3A+2024%2D10%2D07%2E)
18. Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F. Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Souza Moura, Dominik Krzemiński, Hakimeh Fadaei, Irem Ergün, Ifeoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafeai, Vu Minh Chien, Sebastian Ruder, Surya Guthikonda, Emad A. Alghamdi, Sebastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzieh Fadaee, and Sara Hooker. 2024. [Aya dataset: An open-access collection for multilingual instruction tuning](https://md.research.elamur.ai/free/en/papers/singh-2024-aya-dataset-open-access-collection/). Preprint, arXiv:2402.06619. [↗](https://arxiv.org/abs/2402.06619)
19. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. 2024b. Qwen2 technical report. Preprint, arXiv:2407.10671. [↗](https://arxiv.org/abs/2407.10671)
20. Jörg Tiedemann and Santhosh Thottingal. 2020. OPUS-MT – building open translation services for the world. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 479–480, Lisboa, Portugal. European Association for Machine Translation. [find](https://scholar.google.com/scholar?q=J%C3%B6rg+Tiedemann+and+Santhosh+Thottingal%2E+2020%2E+OPUS%2DMT+%E2%80%93+building+open+translation+services+for+the+world%2E+In+Proceedings+of+the+22nd+Annual+Conference+of+the+European+Association+for+Machine+Translati)
21. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Advances in Neural Information Processing Systems, 36.

*Updated 31 October 2026*