---
title: "Improving Romanian LLM Pretraining Data using Diversity and"
description: "Romanian is poorly represented in large multilingual web corpora, and existing… 0.729 Spearman. 32 references, cited by 1 paper here."
authors: ["Vlad-Andrei Negoiță", "Mihai Masala", "Traian Rebedea"]
year: 2026
venue: "Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026)"
source: "https://aclanthology.org/2026.loreslm-1.13/"
doi: "10.18653/v1/2026.loreslm-1.13"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/negoita-2026-improving-romanian-llm-pretraining-data/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/negoita-2026-improving-romanian-llm-pretraining-data/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/7/) / Improving Romanian LLM Pretraining Data using Diversity and Quality Filtering

# Improving Romanian LLM Pretraining Data using Diversity and Quality Filtering

**Authors:** Vlad-Andrei Negoiță, Mihai Masala and [Traian Rebedea](https://md.research.elamur.ai/free/en/authors/traian-rebedea/) · **Year:** 2026 · **Venue:** [Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026)](https://md.research.elamur.ai/free/en/venues/loreslm/) ([Workshop on Language Models for Low-Resource Languages 2026](https://md.research.elamur.ai/free/en/venues/loreslm/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.loreslm-1.13/) · [PDF](https://aclanthology.org/2026.loreslm-1.13.pdf) · [DOI 10.18653/v1/2026.loreslm-1.13](https://doi.org/10.18653/v1/2026.loreslm-1.13) · [OpenAlex](https://openalex.org/W7140142014)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Romanian is poorly represented in large multilingual web corpora, and existing Romanian datasets (CulturaX, FuLG, HPLT, FineWeb2) are cleaned only with simple heuristics such as n-gram statistics or text length, without any higher-level quality assessment. The authors adapt the FineWeb-Edu idea of scoring educational value to Romanian and add further signals: topic and format (following the WebOrganizer taxonomy) and the reader education level a text requires. They used a small human-labelled set to choose the best LLM annotator. That model then labelled about a million FineWeb2 documents, and the labels were distilled into a compact multitask encoder that can score the whole corpus cheaply. Keeping only high-scoring texts produced FineWeb2-Edu-Ro. Continual pretraining of Llama-2-7B on this subset beat both unfiltered FineWeb2 and a JQL-filtered subset of similar size on Romanian benchmarks. The topic labels also allow a comparison of Romanian and English web content. This matters because it is a reproducible, openly released recipe for curating pretraining data in a low-resource language.

## Key points
- **Annotation pipeline:** 100 calibration texts were used, half native Romanian texts from FineWeb2 and half translated FineWeb-Edu texts that kept their original scores. Gemma-3-12B, prompted in Romanian with chain-of-thought, was selected as the annotator and labelled over 1M FineWeb2 samples.
- **Distilled classifier:** RoBERT-base was the best encoder for educational value (RMSE 0.729, Spearman 0.767). It carries four heads: regression for educational value, plus classification for topic, format and education level.
- **FineWeb2-Edu-Ro:** Filtering at a score of 3.5 or higher yields 6.43B tokens and 3.9M samples. The JQL P92 baseline has 6.23B tokens and 2.7M samples.
- **Evaluation:** The models were tested on Ro-MMLU, Ro-ARC and Ro-HellaSwag, with scores averaged over several few-shot settings. Both filtered sets beat random FineWeb2 data, and FineWeb2-Edu-Ro was best on all three benchmarks.
- **Topic analysis:**
  - Before filtering, Romanian data has more Finance & Business, Health and Politics than English, while English has more Software and Education & Jobs.
  - After filtering, Romanian data has less Science, Math & Technology than English and more Finance & Business and Food & Dining.
  - Code, data and models are released at github.com/VladNegoita/FineWeb2-Ro.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Vlad-Andrei Negoiță, Mihai Masala and Traian Rebedea. 2026. Improving Romanian LLM Pretraining Data using Diversity and Quality Filtering. Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{negoita2026improving,
  title = {{Improving Romanian LLM Pretraining Data using Diversity and Quality Filtering}},
  author = {Vlad-Andrei Negoiță, Mihai Masala and Traian Rebedea},
  booktitle = {Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.loreslm-1.13},
  url = {https://doi.org/10.18653/v1/2026.loreslm-1.13}
}
```

- [← previous HTMuon: Improving Muon via Heavy-Tailed Spectral Correction](https://md.research.elamur.ai/free/en/papers/pang-2026-htmuon-improving-muon-heavy-tailed/)
- [next → Improving the Throughput of Diffusion-based Large Language Models via a Training-Free Confidence-Aware Calibration](https://md.research.elamur.ai/free/en/papers/shen-2026-improving-throughput-diffusion-based-large/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/7/)

## References

32 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. AI@Meta. 2024. Llama 3 model card. [find](https://scholar.google.com/scholar?q=AI%40Meta%2E+2024%2E+Llama+3+model+card%2E)
2. Mehdi Ali, Manuel Brack, Max Lübbering, Elias Wendt, Abbas Goher Khan, Richard Rutmann, Alex Jude, Maurice Kraus, Alexander Arno Weber, Felix Stollenwerk, David Kaczér, Florian Mai, Lucie Flek, Rafet Sifa, Nicolas Flores-Herr, Joachim Köhler, Patrick Schramowski, Michael Fromm, and Kristian Kersting… — [arXiv:2505.22232](https://arxiv.org/abs/2505.22232) [find](https://scholar.google.com/scholar?q=Mehdi+Ali%2C+Manuel+Brack%2C+Max+L%C3%BCbbering%2C+Elias+Wendt%2C+Abbas+Goher+Khan%2C+Richard+Rutmann%2C+Alex+Jude%2C+Maurice+Kraus%2C+Alexander+Arno+Weber%2C+Felix+Stollenwerk%2C+David+Kacz%C3%A9r%2C+Florian+Mai%2C+Lucie+Flek%2C+Rafet+)
3. Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Kelly Marchisio, Sebastian Ruder, Acyr Locatelli, Julia Kreutzer, Nick Frosst, Phil Blunsom, Marzieh Fadaee, Ahmet Üstün, and Sara Hooker. 2024. [Aya 23: Open weight releases to furt…](https://md.research.elamur.ai/free/en/papers/aryabumi-2024-aya-23-open-weight-releases/)
4. Tianyi Bai, Ling Yang, Zhen Hao Wong, Fupeng Sun, Xinlin Zhuang, Jiahui Peng, Chi Zhang, Lijun Wu, Qiu Jiantao, Wentao Zhang, and 1 others. 2025. Efficient pretraining data selection for language models via multi-actor collaboration. In Proceedings of the 63rd Annual Meeting of the Association for C… [find](https://scholar.google.com/scholar?q=Tianyi+Bai%2C+Ling+Yang%2C+Zhen+Hao+Wong%2C+Fupeng+Sun%2C+Xinlin+Zhuang%2C+Jiahui+Peng%2C+Chi+Zhang%2C+Lijun+Wu%2C+Qiu+Jiantao%2C+Wentao+Zhang%2C+and+1+others%2E+2025%2E+Efficient+pretraining+data+selection+for+language+mode)
5. Vlad-Andrei Bădoiu, Mihai-Valentin Dumitru, Alexandru M. Gherghescu, Alexandru Agache, and Costin Raiciu. 2024. Fulg: 150b romanian corpus for language model pretraining. Preprint, arXiv:2407.13657. — [arXiv:2407.13657](https://arxiv.org/abs/2407.13657) [find](https://scholar.google.com/scholar?q=Vlad%2DAndrei+Ba%CC%86doiu%2C+Mihai%2DValentin+Dumitru%2C+Alexandru+M%2E+Gherghescu%2C+Alexandru+Agache%2C+and+Costin+Raiciu%2E+2024%2E+Fulg%3A+150b+romanian+corpus+for+language+model+pretraining%2E+Preprint%2C+arXiv%3A2407%2E13657%2E+)
6. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). Preprint, arXiv:1803.05457.
7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [BERT: pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). CoRR, abs/1810.04805.
8. Stefan Dumitrescu, Andrei-Marius Avram, and Sampo Pyysalo. 2020. The birth of Romanian BERT. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4324–4328, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Stefan+Dumitrescu%2C+Andrei%2DMarius+Avram%2C+and+Sampo+Pyysalo%2E+2020%2E+The+birth+of+Romanian+BERT%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2020%2C+pages+4324%E2%80%934328%2C+Online%2E+Associat)
9. Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, and 24 others.… [find](https://scholar.google.com/scholar?q=Dirk+Groeneveld%2C+Iz+Beltagy%2C+Pete+Walsh%2C+Akshita+Bhagia%2C+Rodney+Kinney%2C+Oyvind+Tafjord%2C+Ananya+Harsh+Jha%2C+Hamish+Ivison%2C+Ian+Magnusson%2C+Yizhong+Wang%2C+Shane+Arora%2C+David+Atkinson%2C+Russell+Authur%2C+Khyat)
10. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). Preprint, arXiv:2009.03300.
11. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… — [Training Compute-Optimal Large Language Models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
12. Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, and Daphne Ippolito. 2024. A pretrainer’s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. In Proceedings of the 2… [find](https://scholar.google.com/scholar?q=Shayne+Longpre%2C+Gregory+Yauney%2C+Emily+Reif%2C+Katherine+Lee%2C+Adam+Roberts%2C+Barret+Zoph%2C+Denny+Zhou%2C+Jason+Wei%2C+Kevin+Robinson%2C+David+Mimno%2C+and+Daphne+Ippolito%2E+2024%2E+A+pretrainer%E2%80%99s+guide+to+training+da)
13. Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. 2024. Fineweb-edu: the finest collection of educational content. [find](https://scholar.google.com/scholar?q=Anton+Lozhkov%2C+Loubna+Ben+Allal%2C+Leandro+von+Werra%2C+and+Thomas+Wolf%2E+2024%2E+Fineweb%2Dedu%3A+the+finest+collection+of+educational+content%2E)
14. Mihai Masala, Denis Ilie-Ablachim, Alexandru Dima, Dragos Georgian Corlatescu, Miruna-Andreea Zavelca, Ovio Olaru, Simina-Maria Terian, Andrei Terian, Marius Leordeanu, Horia Velicu, Marius Popescu, Mihai Dascalu, and Traian Rebedea. 2024. “vorbes , ti românes , te?” a recipe to train powerful Roman… [find](https://scholar.google.com/scholar?q=Mihai+Masala%2C+Denis+Ilie%2DAblachim%2C+Alexandru+Dima%2C+Dragos+Georgian+Corlatescu%2C+Miruna%2DAndreea+Zavelca%2C+Ovio+Olaru%2C+Simina%2DMaria+Terian%2C+Andrei+Terian%2C+Marius+Leordeanu%2C+Horia+Velicu%2C+Marius+Popescu%2C+M)
15. Mihai Masala, Stefan Ruseti, and Mihai Dascalu. 2020. Robert–a romanian bert model. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6626–6637. [find](https://scholar.google.com/scholar?q=Mihai+Masala%2C+Stefan+Ruseti%2C+and+Mihai+Dascalu%2E+2020%2E+Robert%E2%80%93a+romanian+bert+model%2E+In+Proceedings+of+the+28th+International+Conference+on+Computational+Linguistics%2C+pages+6626%E2%80%936637%2E)
16. Luke Merrick, Danmei Xu, Gaurav Nuti, and Daniel Campos. 2024. Arctic-embed: Scalable, efficient, and accurate text embedding models. Preprint, arXiv:2405.05374. — [arXiv:2405.05374](https://arxiv.org/abs/2405.05374) [find](https://scholar.google.com/scholar?q=Luke+Merrick%2C+Danmei+Xu%2C+Gaurav+Nuti%2C+and+Daniel+Campos%2E+2024%2E+Arctic%2Dembed%3A+Scalable%2C+efficient%2C+and+accurate+text+embedding+models%2E+Preprint%2C+arXiv%3A2405%2E05374%2E+%E2%80%94+%5BarXiv%3A2405%2E05374%5D%28https%3A%2F%2Farxiv%2Eorg)
17. Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024. [The fineweb datasets: Decanting the web for the finest text data at scale](https://md.research.elamur.ai/free/en/papers/penedo-2024-fineweb-datasets-decanting-web-finest/). Preprint, arXiv:2406.17557.
18. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. [find](https://scholar.google.com/scholar?q=Qwen%2C+%3A%2C+An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+Huan+Lin%2C+Jian+Yang%2C+Jianhong+Tu%2C+Jianwei+Zhang%2C+Jianxin+Yang%2C+Jiaxi+)
19. Gemma Team, Aishwarya Kamath, and Johan Ferret et al. 2025. Gemma 3 technical report. Preprint, arXiv:2503.19786. [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Aishwarya+Kamath%2C+and+Johan+Ferret+et+al%2E+2025%2E+Gemma+3+technical+report%2E+Preprint%2C+arXiv%3A2503%2E19786%2E)
20. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. Preprint, arXiv:1809.02789. [find](https://scholar.google.com/scholar?q=Todor+Mihaylov%2C+Peter+Clark%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+2018%2E+Can+a+suit+of+armor+conduct+electricity%3F+a+new+dataset+for+open+book+question+answering%2E+Preprint%2C+arXiv%3A1809%2E02789%2E)
21. Gemma Team, Morgane Riviere, Shreya Pathak, and Pier Giuseppe Sessa et al. 2024. Gemma 2: Improving open language models at a practical size. Preprint, arXiv:2408.00118. [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Morgane+Riviere%2C+Shreya+Pathak%2C+and+Pier+Giuseppe+Sessa+et+al%2E+2024%2E+Gemma+2%3A+Improving+open+language+models+at+a+practical+size%2E+Preprint%2C+arXiv%3A2408%2E00118%2E)
22. Mistral AI. 2025. Mistral Small 24B Instruct 2501. [https://huggingface.co/mistralai/](https://huggingface.co/mistralai/) Mistral-Small-24B-Instruct-2501. Accessed: [Current Date, e.g., 2025-06-04]. [↗](https://huggingface.co/mistralai/)
23. Hugo Touvron, Louis Martin, and Kevin Stone et al. 2023. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). Preprint, arXiv:2307.09288.
24. Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. [Orca: Progressive learning from complex explanation traces of gpt-4](https://md.research.elamur.ai/free/en/papers/mukherjee-2023-orca-progressive-learning-complex-explanation/). arXiv preprint arXiv:2306.02707.
25. Thao Nguyen, Yang Li, Olga Golovneva, Luke Zettlemoyer, Sewoong Oh, Ludwig Schmidt, and Xian Li. 2025. Recycling the web: A method to enhance pre-training data quality and quantity for language models. Preprint, arXiv:2506.04689. [find](https://scholar.google.com/scholar?q=Thao+Nguyen%2C+Yang+Li%2C+Olga+Golovneva%2C+Luke+Zettlemoyer%2C+Sewoong+Oh%2C+Ludwig+Schmidt%2C+and+Xian+Li%2E+2025%2E+Recycling+the+web%3A+A+method+to+enhance+pre%2Dtraining+data+quality+and+quantity+for+language+models)
26. Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2023. [Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages](https://md.research.elamur.ai/free/en/papers/nguyen-2023-culturax-cleaned-enormous-multilingual-dataset/). Preprint, arXiv:2309.09400.
27. Stephan Oepen, Nikolay Arefev, Mikko Aulamo, Marta Bañón, Maja Buljan, Laurie Burchell, Lucas Charpentier, Pinzhen Chen, Mariya Fedorova, Ona de Gibert, Barry Haddow, Jan Hajič, Jindřich Helcl, Andrey Kutuzov, Veronika Laippala, Zihao Li, Risto Luukkonen, Bhavitvya Malik, Vladislav Mikhailov, and… — [arXiv:2511.01066](https://arxiv.org/abs/2511.01066) [find](https://scholar.google.com/scholar?q=Stephan+Oepen%2C+Nikolay+Arefev%2C+Mikko+Aulamo%2C+Marta+Ba%C3%B1%C3%B3n%2C+Maja+Buljan%2C+Laurie+Burchell%2C+Lucas+Charpentier%2C+Pinzhen+Chen%2C+Mariya+Fedorova%2C+Ona+de+Gibert%2C+Barry+Haddow%2C+Jan+Hajic%CC%8C%2C+Jindr%CC%8Cich+Helcl%2C+Andr)
28. Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019. Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures. Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pages 9 – 16, Mannheim… [find](https://scholar.google.com/scholar?q=Pedro+Javier+Ortiz+Su%C3%A1rez%2C+Beno%C3%AEt+Sagot%2C+and+Laurent+Romary%2E+2019%2E+Asynchronous+pipelines+for+processing+huge+corpora+on+medium+to+low+resource+infrastructures%2E+Proceedings+of+the+Workshop+on+Challeng)
29. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
30. Alexander Wettig, Kyle Lo, Sewon Min, Hannaneh Hajishirzi, Danqi Chen, and Luca Soldaini. 2025. Organize the web: Constructing domains enhances pre-training data curation. Preprint, arXiv:2502.10341. [find](https://scholar.google.com/scholar?q=Alexander+Wettig%2C+Kyle+Lo%2C+Sewon+Min%2C+Hannaneh+Hajishirzi%2C+Danqi+Chen%2C+and+Luca+Soldaini%2E+2025%2E+Organize+the+web%3A+Constructing+domains+enhances+pre%2Dtraining+data+curation%2E+Preprint%2C+arXiv%3A2502%2E10341%2E)
31. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? Preprint, arXiv:1905.07830. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+Preprint%2C+arXiv%3A1905%2E07830%2E)
32. Xinlin Zhuang, Jiahui Peng, Ren Ma, Yinfan Wang, Tianyi Bai, Xingjian Wei, Jiantao Qiu, Chi Zhang, Ying Qian, and Conghui He. 2025. Meta-rater: A multi-dimensional data selection method for pre-training language models. Preprint, arXiv:2504.14194. [find](https://scholar.google.com/scholar?q=Xinlin+Zhuang%2C+Jiahui+Peng%2C+Ren+Ma%2C+Yinfan+Wang%2C+Tianyi+Bai%2C+Xingjian+Wei%2C+Jiantao+Qiu%2C+Chi+Zhang%2C+Ying+Qian%2C+and+Conghui+He%2E+2025%2E+Meta%2Drater%3A+A+multi%2Ddimensional+data+selection+method+for+pre%2Dtraini)

*Updated 31 October 2026*