---
title: "RegMix — paper by Liu et al. (2024) — summary and ideas"
description: "The share of each data domain in a pre-training corpus strongly affects how well a large language… 78 references, cited by 54 papers here."
authors: ["Qian Liu", "Xiaosen Zheng", "Niklas Muennighoff", "Guangtao Zeng", "Longxu Dou", "Tianyu Pang", "Jing Jiang", "Min Lin"]
year: 2024
venue: "arXiv:2407.01492"
source: "https://arxiv.org/abs/2407.01492"
doi: "10.48550/arxiv.2407.01492"
arxiv: "2407.01492"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2024-regmix-data-mixture-regression-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2024-regmix-data-mixture-regression-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / RegMix: Data Mixture as Regression for Language Model Pre-training

# RegMix: Data Mixture as Regression for Language Model Pre-training

**Authors:** [Qian Liu](https://md.research.elamur.ai/free/en/authors/qian-liu/), Xiaosen Zheng, [Niklas Muennighoff](https://md.research.elamur.ai/free/en/authors/niklas-muennighoff/), [Guangtao Zeng](https://md.research.elamur.ai/free/en/authors/guangtao-zeng/), [Longxu Dou](https://md.research.elamur.ai/free/en/authors/longxu-dou/), [Tianyu Pang](https://md.research.elamur.ai/free/en/authors/tianyu-pang/), [Jing Jiang](https://md.research.elamur.ai/free/en/authors/jing-jiang/) and [Min Lin](https://md.research.elamur.ai/free/en/authors/min-lin/) · **Year:** 2024 · **Venue:** arXiv:2407.01492 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2407.01492) · [PDF](https://arxiv.org/pdf/2407.01492) · [DOI 10.48550/arxiv.2407.01492](https://doi.org/10.48550/arxiv.2407.01492) · [arXiv 2407.01492](https://arxiv.org/abs/2407.01492) · [OpenAlex](https://openalex.org/W4400342789)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [54](https://md.research.elamur.ai/free/en/papers/liu-2024-regmix-data-mixture-regression-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The share of each data domain in a pre-training corpus strongly affects how well a large language model performs. Hand-tuned mixtures do not scale, and learned methods such as DoReMi depend on long proxy training runs. RegMix treats mixture search as a regression problem. Many tiny proxy models are trained on random mixtures, sampled from a Dirichlet distribution based on available token counts. A linear or tree-based regressor then learns to predict a target, such as validation loss, from the mixture weights. The best-scoring mixture in a large simulated space is used to train the full model. The method assumes "rank invariance": mixtures that rank well at small scale keep their ranking at larger scale. With 512 proxies of 1M parameters, RegMix picked the best of 64 candidate mixtures for 1B-parameter models. It beat human-designed mixtures at up to 7B parameters and 100B tokens, and it matched or exceeded DoReMi with about 10% of the compute. Because the proxies are cheap and can run in parallel, the approach offers a scalable way to tune pre-training data. It also exposes interactions between domains that run against intuition.

## Key points
- **Main setup:** 512 proxy models (1M non-embedding parameters, 1B tokens each) cost roughly 2% of the FLOPs of a single 1B model. Their predicted mixture produced the best of 64 1B models trained on 25B tokens, which is 1000× larger and 25× longer training than the proxies.
- **Rank invariance check:** Spearman ρ between mixture rankings at 1M, 60M, 280M and 1B parameters stayed high, at about 0.94–0.99.
- **7B / 100B tokens results:**
  - RegMix averaged 56.5 versus 54.5 for the human mixture across 13 benchmarks, e.g., HellaSwag 63.3 vs 55.0.
  - It reached comparable scores about 50% faster on most tasks and up to 75% faster on PiQA.
  - Tasks that do not improve with more data, such as MultiRC, showed no benefit.
- **Data findings:**
  - Mixture choice alone changed single-task scores by up to 14.6%.
  - General web data such as CommonCrawl, not Wikipedia, correlated most strongly with good downstream results.
- **Proxy size:** 128 proxies of 1B parameters gave nearly the same result as 512 proxies of 1M (56.4 vs 56.5) at much higher cost, so the authors recommend many ultra-small proxies. Code: github.com/sail-sg/regmix.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 434 works · 314,419 citations
- [Singapore University of Technology and Design](https://md.research.elamur.ai/free/en/universities/singapore-university-of-technology-and-design/) — University, Singapore, Singapore · 49 works · 11,074 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang and Min Lin. 2024. RegMix: Data Mixture as Regression for Language Model Pre-training. arXiv:2407.01492.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{liu2024regmix,
  title = {{RegMix: Data Mixture as Regression for Language Model Pre-training}},
  author = {Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang and Min Lin},
  year = {2024},
  doi = {10.48550/arxiv.2407.01492},
  eprint = {2407.01492},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2407.01492}
}
```

- [← previous Dialect prejudice predicts AI decisions about people's character, employability, and criminality](https://md.research.elamur.ai/free/en/papers/hofmann-2024-dialect-prejudice-predicts-ai-decisions/)
- [next → TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Space](https://md.research.elamur.ai/free/en/papers/zhang-2024-truthx-alleviating-hallucinations-editing-large/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/)

## References

78 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Meta AI. Introducing meta llama 3: The most capable openly available llm to date. [https://ai.meta.com/blog/meta-llama-3/](https://ai.meta.com/blog/meta-llama-3/), April 2024. [↗](https://ai.meta.com/blog/meta-llama-3/)
2. Alon Albalak, Liangming Pan, Colin Raffel, and William Yang Wang. [Efficient online data mixing for language model pre-training](https://md.research.elamur.ai/free/en/papers/albalak-2023-efficient-online-data-mixing-language/). arXiv preprint arXiv:2312.02406, 2023.
3. Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. [A survey on data selection for language models](https://md.research.elamur.ai/free/en/papers/albalak-2024-survey-data-selection-language-models/). arXiv preprint arXiv:2402.16827, 2024.
4. Zachary Ankner, Cody Blakeney, Kartik Sreenivasan, Max Marion, Matthew L. Leavitt, and Mansheej Paul. Perplexed by perplexity: Perplexity-based data pruning with small reference models. CoRR, abs/2405.20541, 2024. doi: 10.48550/ARXIV.2405.20541. URL https://doi.org/10. 48550/arXiv.2405.20541. — [doi:10.48550/arxiv.2405.20541](https://doi.org/10.48550/arxiv.2405.20541) [find](https://scholar.google.com/scholar?q=Zachary+Ankner%2C+Cody+Blakeney%2C+Kartik+Sreenivasan%2C+Max+Marion%2C+Matthew+L%2E+Leavitt%2C+and+Mansheej+Paul%2E+Perplexed+by+perplexity%3A+Perplexity%2Dbased+data+pruning+with+small+reference+models%2E+CoRR%2C+abs%2F2405)
5. Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, et al. Lessons from the trenches on reproducible evaluation of language models. arXiv preprint arXiv:2405.14782, 2024. [find](https://scholar.google.com/scholar?q=Stella+Biderman%2C+Hailey+Schoelkopf%2C+Lintang+Sutawika%2C+Leo+Gao%2C+Jonathan+Tow%2C+Baber+Abbasi%2C+Alham+Fikri+Aji%2C+Pawan+Sasanka+Ammanamanchi%2C+Sidney+Black%2C+Jordan+Clive%2C+et+al%2E+Lessons+from+the+trenches+on+)
6. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. [Piqa: Reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). In Proceedings of the AAAI conference on artificial intelligence, 2020.
7. Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. [Gpt-neox-20b: An open-source autoregressive language model](https://md.research.elamur.ai/free/en/papers/black-2022-gpt-neox-20b-open-source/). arXiv preprint arXiv:2204.06745, 2022.
8. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretchen)
9. Curran Associates, Inc., 2020. URL [https://proceedings.neurips.cc/paper_files/paper/](https://proceedings.neurips.cc/paper_files/paper/) 2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf. [↗](https://proceedings.neurips.cc/paper_files/paper/)
10. Alexander Bukharin and Tuo Zhao. Data diversity matters for robust instruction tuning. arXiv preprint arXiv:2311.14736, 2023. [find](https://scholar.google.com/scholar?q=Alexander+Bukharin+and+Tuo+Zhao%2E+Data+diversity+matters+for+robust+instruction+tuning%2E+arXiv+preprint+arXiv%3A2311%2E14736%2C+2023%2E)
11. Mayee F Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher Ré. Skill-it! a data-driven skills framework for understanding and training language models. arXiv preprint arXiv:2307.14430, 2023. — [arXiv:2307.14430](https://arxiv.org/abs/2307.14430) [find](https://scholar.google.com/scholar?q=Mayee+F+Chen%2C+Nicholas+Roberts%2C+Kush+Bhatia%2C+Jue+Wang%2C+Ce+Zhang%2C+Frederic+Sala%2C+and+Christopher+R%C3%A9%2E+Skill%2Dit%21+a+data%2Ddriven+skills+framework+for+understanding+and+training+language+models%2E+arXiv+prepr)
12. Xuxi Chen, Zhendong Wang, Daouda Sow, Junjie Yang, Tianlong Chen, Yingbin Liang, Mingyuan Zhou, and Zhangyang Wang. Take the bull by the horns: Hard sample-reweighted continual training improves llm generalization. arXiv preprint arXiv:2402.14270, 2024. [find](https://scholar.google.com/scholar?q=Xuxi+Chen%2C+Zhendong+Wang%2C+Daouda+Sow%2C+Junjie+Yang%2C+Tianlong+Chen%2C+Yingbin+Liang%2C+Mingyuan+Zhou%2C+and+Zhangyang+Wang%2E+Take+the+bull+by+the+horns%3A+Hard+sample%2Dreweighted+continual+training+improves+llm+g)
13. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv preprint arXiv:1803.05457, 2018.
14. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). CoRR, abs/2110.14168, 2021. URL https://arxiv.org/abs/2110.1416… [↗](https://arxiv.org/abs/2110.1416…)
15. Devleena Das and Vivek Khetan. Deft: Data efficient fine-tuning for large language models via unsupervised core-set selection. arXiv preprint arXiv:2310.16776, 2023. — [arXiv:2310.16776](https://arxiv.org/abs/2310.16776) [find](https://scholar.google.com/scholar?q=Devleena+Das+and+Vivek+Khetan%2E+Deft%3A+Data+efficient+fine%2Dtuning+for+large+language+models+via+unsupervised+core%2Dset+selection%2E+arXiv+preprint+arXiv%3A2310%2E16776%2C+2023%2E+%E2%80%94+%5BarXiv%3A2310%2E16776%5D%28https%3A%2F%2Farxiv)
16. Longxu Dou, Qian Liu, Guangtao Zeng, Jia Guo, Jiahui Zhou, Wei Lu, and Min Lin. [Sailor: Open language models for south-east asia](https://md.research.elamur.ai/free/en/papers/dou-2024-sailor-open-language-models-south/). CoRR, abs/2404.03608, 2024. doi: 10.48550/ARXIV.2404. 03608. URL https://doi.org/10.48550/arXiv.2404.03608. [↗](https://doi.org/10.48550/arXiv.2404.03608)
17. Logan Engstrom, Axel Feldmann, and Aleksander Madry. Dsdm: Model-aware dataset selection with datamodels. arXiv preprint arXiv:2401.12926, 2024. [find](https://scholar.google.com/scholar?q=Logan+Engstrom%2C+Axel+Feldmann%2C+and+Aleksander+Madry%2E+Dsdm%3A+Model%2Daware+dataset+selection+with+datamodels%2E+arXiv+preprint+arXiv%3A2401%2E12926%2C+2024%2E)
18. Simin Fan, Matteo Pagliardini, and Martin Jaggi. Doge: Domain reweighting with generalization estimation. arXiv preprint arXiv:2310.15393, 2023. — [arXiv:2310.15393](https://arxiv.org/abs/2310.15393) [find](https://scholar.google.com/scholar?q=Simin+Fan%2C+Matteo+Pagliardini%2C+and+Martin+Jaggi%2E+Doge%3A+Domain+reweighting+with+generalization+estimation%2E+arXiv+preprint+arXiv%3A2310%2E15393%2C+2023%2E+%E2%80%94+%5BarXiv%3A2310%2E15393%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2310%2E15393%29)
19. Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, Rui Xin, Marianna Nezhurina, Igor Vasiljevic, Jenia Jitsev, Alexandros G. Dimakis, Gabriel Ilharco, Shuran Song, Thomas Kollar, Yair Carmon, Ach… — [Language models scale reliably with over-training and on downstream tasks](https://md.research.elamur.ai/free/en/papers/gadre-2024-language-models-scale-reliably-over/)
20. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. CoRR, abs/2101.00027, 2021. URL https://arxiv.org/abs/2101.00027… [↗](https://arxiv.org/abs/2101.00027…)
21. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Ani… [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Baber+Abbasi%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Alain+Le+Noac%E2%80%99h%2C+Haonan+Li%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Chris+Oci)
22. Ce Ge, Zhijian Ma, Daoyuan Chen, Yaliang Li, and Bolin Ding. Data mixing made efficient: A bivariate scaling law for language model pretraining. arXiv preprint arXiv:2405.14908, 2024. — [arXiv:2405.14908](https://arxiv.org/abs/2405.14908) [find](https://scholar.google.com/scholar?q=Ce+Ge%2C+Zhijian+Ma%2C+Daoyuan+Chen%2C+Yaliang+Li%2C+and+Bolin+Ding%2E+Data+mixing+made+efficient%3A+A+bivariate+scaling+law+for+language+model+pretraining%2E+arXiv+preprint+arXiv%3A2405%2E14908%2C+2024%2E+%E2%80%94+%5BarXiv%3A2405%2E14)
23. Sachin Goyal, Pratyush Maini, Zachary C. Lipton, Aditi Raghunathan, and J. Zico Kolter. Scaling laws for data filtering - data curation cannot be compute agnostic. CoRR, abs/2404.07177, 2024. doi: 10.48550/ARXIV.2404.07177. URL https://doi.org/10.48550/arXiv.2404. 07177. [↗](https://doi.org/10.48550/arXiv.2404)
24. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.
25. OpenReview.net, 2021. URL [https://openreview.net/forum?id=d7KBjmI3GmQ](https://openreview.net/forum?id=d7KBjmI3GmQ). [↗](https://openreview.net/forum?id=d7KBjmI3GmQ)
26. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. [Training compute-optimal large language models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/). arXiv preprint arXiv:2203.15556, 2022.
27. Yuzhen Huang, Jinghan Zhang, Zifei Shan, and Junxian He. Compression represents intelligence linearly. arXiv preprint arXiv:2404.09937, 2024. — [arXiv:2404.09937](https://arxiv.org/abs/2404.09937) [find](https://scholar.google.com/scholar?q=Yuzhen+Huang%2C+Jinghan+Zhang%2C+Zifei+Shan%2C+and+Junxian+He%2E+Compression+represents+intelligence+linearly%2E+arXiv+preprint+arXiv%3A2404%2E09937%2C+2024%2E+%E2%80%94+%5BarXiv%3A2404%2E09937%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2404%2E09937%29)
28. Feiyang Kang, Hoang Anh Just, Yifan Sun, Himanshu Jahagirdar, Yuanzhi Zhang, Rongxing Du, Anit Kumar Sahu, and Ruoxi Jia. Get more for less: Principled data selection for warming up fine-tuning in llms. arXiv preprint arXiv:2405.02774, 2024. [find](https://scholar.google.com/scholar?q=Feiyang+Kang%2C+Hoang+Anh+Just%2C+Yifan+Sun%2C+Himanshu+Jahagirdar%2C+Yuanzhi+Zhang%2C+Rongxing+Du%2C+Anit+Kumar+Sahu%2C+and+Ruoxi+Jia%2E+Get+more+for+less%3A+Principled+data+selection+for+warming+up+fine%2Dtuning+in+llm)
29. Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30, 2017. [find](https://scholar.google.com/scholar?q=Guolin+Ke%2C+Qi+Meng%2C+Thomas+Finley%2C+Taifeng+Wang%2C+Wei+Chen%2C+Weidong+Ma%2C+Qiwei+Ye%2C+and+Tie%2DYan+Liu%2E+Lightgbm%3A+A+highly+efficient+gradient+boosting+decision+tree%2E+Advances+in+neural+information+processin)
30. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. [Looking beyond the surface: A challenge set for reading comprehension over multiple sentences](https://md.research.elamur.ai/free/en/papers/khashabi-2018-looking-beyond-surface-challenge-set/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human…
31. Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries. The stack: 3 TB of permissively licensed source code. CoRR, abs/2211.15533, 2022. doi: 10.4… [find](https://scholar.google.com/scholar?q=Denis+Kocetkov%2C+Raymond+Li%2C+Loubna+Ben+Allal%2C+Jia+Li%2C+Chenghao+Mou%2C+Carlos+Mu%C3%B1oz+Ferrandis%2C+Yacine+Jernite%2C+Margaret+Mitchell%2C+Sean+Hughes%2C+Thomas+Wolf%2C+Dzmitry+Bahdanau%2C+Leandro+von+Werra%2C+and+Harm+d)
32. Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pp. 1885–1894. [find](https://scholar.google.com/scholar?q=Pang+Wei+Koh+and+Percy+Liang%2E+Understanding+black%2Dbox+predictions+via+influence+functions%2E+In+International+conference+on+machine+learning%2C+pp%2E+1885%E2%80%931894%2E)
33. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. [Race: Large-scale reading comprehension dataset from examinations](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/). arXiv preprint arXiv:1704.04683, 2017.
34. Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhur… — [DataComp-LM: In search of the next generation of training sets for language models](https://md.research.elamur.ai/free/en/papers/li-2024-datacomp-lm-search-next-generation/)
35. Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, and Weizhu Chen. Rho-1: Not all tokens are what you need. arXiv preprint arXiv:2404.07965, 2024. [find](https://scholar.google.com/scholar?q=Zhenghao+Lin%2C+Zhibin+Gou%2C+Yeyun+Gong%2C+Xiao+Liu%2C+Yelong+Shen%2C+Ruochen+Xu%2C+Chen+Lin%2C+Yujiu+Yang%2C+Jian+Jiao%2C+Nan+Duan%2C+and+Weizhu+Chen%2E+Rho%2D1%3A+Not+all+tokens+are+what+you+need%2E+arXiv+preprint+arXiv%3A2404%2E)
36. Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. [Logiqa: A challenge dataset for machine reading comprehension with logical reasoning](https://md.research.elamur.ai/free/en/papers/liu-2020-logiqa-challenge-dataset-machine-reading/). arXiv preprint arXiv:2007.08124, 2020.
37. Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. [What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-makes-good-data-alignment-comprehensive/). In The International Conference on Learning Representations, 2024.
38. Ian Magnusson, Akshita Bhagia, Valentin Hofmann, Luca Soldaini, Ananya Harsh Jha, Oyvind Tafjord, Dustin Schwenk, Evan Pete Walsh, Yanai Elazar, Kyle Lo, Dirk Groeneveld, Iz Beltagy, Hannaneh Hajishirzi, Noah A. Smith, Kyle Richardson, and Jesse Dodge. Paloma: A benchmark for evaluating language mod… — [Paloma: A Benchmark for Evaluating Language Model Fit](https://md.research.elamur.ai/free/en/papers/magnusson-2023-paloma-benchmark-evaluating-language-model/)
39. Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. When less is more: Investigating data pruning for pretraining llms at scale, 2023. [find](https://scholar.google.com/scholar?q=Max+Marion%2C+Ahmet+%C3%9Cst%C3%BCn%2C+Luiza+Pozzobon%2C+Alex+Wang%2C+Marzieh+Fadaee%2C+and+Sara+Hooker%2E+When+less+is+more%3A+Investigating+data+pruning+for+pretraining+llms+at+scale%2C+2023%2E)
40. Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, and Mohammad Rastegari. Openelm: An efficient language model family with open training and inference framework. CoRR, abs/2404.14619, 2024. do… — [doi:10.48550/arxiv.2404.14619](https://doi.org/10.48550/arxiv.2404.14619) [find](https://scholar.google.com/scholar?q=Sachin+Mehta%2C+Mohammad+Hossein+Sekhavat%2C+Qingqing+Cao%2C+Maxwell+Horton%2C+Yanzi+Jin%2C+Chenfan+Sun%2C+Iman+Mirzadeh%2C+Mahyar+Najibi%2C+Dmitry+Belenko%2C+Peter+Zatloukal%2C+and+Mohammad+Rastegari%2E+Openelm%3A+An+effici)
41. Dheeraj Mekala, Alex Nguyen, and Jingbo Shang. Smaller language models are capable of selecting instruction-tuning training data for larger language models. arXiv preprint arXiv:2402.10430, 2024. [find](https://scholar.google.com/scholar?q=Dheeraj+Mekala%2C+Alex+Nguyen%2C+and+Jingbo+Shang%2E+Smaller+language+models+are+capable+of+selecting+instruction%2Dtuning+training+data+for+larger+language+models%2E+arXiv+preprint+arXiv%3A2402%2E10430%2C+2024%2E)
42. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. [Pointer sentinel mixture models](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/), 2016.
43. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789, 2018. [find](https://scholar.google.com/scholar?q=Todor+Mihaylov%2C+Peter+Clark%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+Can+a+suit+of+armor+conduct+electricity%3F+a+new+dataset+for+open+book+question+answering%2E+arXiv+preprint+arXiv%3A1809%2E02789%2C+2018%2E)
44. Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, et al. Prioritized training on points that are learnable, worth learning, and not yet learnt. In International Conference on Machine Lea… [find](https://scholar.google.com/scholar?q=S%C3%B6ren+Mindermann%2C+Jan+M+Brauner%2C+Muhammed+T+Razzak%2C+Mrinank+Sharma%2C+Andreas+Kirsch%2C+Winnie+Xu%2C+Benedikt+H%C3%B6ltgen%2C+Aidan+N+Gomez%2C+Adrien+Morisot%2C+Sebastian+Farquhar%2C+et+al%2E+Prioritized+training+on+point)
45. Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL [https://openreview.net/forum?id=…](https://openreview.net/forum?id=…) [↗](https://openreview.net/forum?id=…)
46. Rohan Pandey. gzip predicts data-dependent scaling laws. arXiv preprint arXiv:2405.16684, 2024. — [arXiv:2405.16684](https://arxiv.org/abs/2405.16684) [find](https://scholar.google.com/scholar?q=Rohan+Pandey%2E+gzip+predicts+data%2Ddependent+scaling+laws%2E+arXiv+preprint+arXiv%3A2405%2E16684%2C+2024%2E+%E2%80%94+%5BarXiv%3A2405%2E16684%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2405%2E16684%29)
47. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. [The lambada dataset: Word prediction requiring a broad discourse context](https://md.research.elamur.ai/free/en/papers/paperno-2016-lambada-dataset-word-prediction-requiring/). arXiv preprint arXiv:1606.06031, 2016.
48. Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. [The fineweb datasets: Decanting the web for the finest text data at scale](https://md.research.elamur.ai/free/en/papers/penedo-2024-fineweb-datasets-decanting-web-finest/). arXiv preprint arXiv:2406.17557, 2024a.
49. Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. [The fineweb datasets: Decanting the web for the finest text data at scale](https://md.research.elamur.ai/free/en/papers/penedo-2024-fineweb-datasets-decanting-web-finest/), 2024b. URL https://arxiv.org/abs/2406.17557. [↗](https://arxiv.org/abs/2406.17557)
50. Haoran Que, Jiaheng Liu, Ge Zhang, Chenchen Zhang, Xingwei Qu, Yinghao Ma, Feiyu Duan, Zhiqi Bai, Jiakai Wang, Yuanxing Zhang, Xu Tan, Jie Fu, Wenbo Su, Jiamang Wang, Lin Qu, and Bo Zheng. D-cpt law: Domain-specific continual pre-training scaling law for large language models, 2024. [find](https://scholar.google.com/scholar?q=Haoran+Que%2C+Jiaheng+Liu%2C+Ge+Zhang%2C+Chenchen+Zhang%2C+Xingwei+Qu%2C+Yinghao+Ma%2C+Feiyu+Duan%2C+Zhiqi+Bai%2C+Jiakai+Wang%2C+Yuanxing+Zhang%2C+Xu+Tan%2C+Jie+Fu%2C+Wenbo+Su%2C+Jiamang+Wang%2C+Lin+Qu%2C+and+Bo+Zheng%2E+D%2Dcpt+law%3A+)
51. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-S… [find](https://scholar.google.com/scholar?q=Jack+W%2E+Rae%2C+Sebastian+Borgeaud%2C+Trevor+Cai%2C+Katie+Millican%2C+Jordan+Hoffmann%2C+H%2E+Francis+Song%2C+John+Aslanides%2C+Sarah+Henderson%2C+Roman+Ring%2C+Susannah+Young%2C+Eliza+Rutherford%2C+Tom+Hennigan%2C+Jacob+Menick)
52. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). arXiv e-prints, 2019.
53. Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang, Jianmo Ni, Lichan Hong, Ed H Chi, James Caverlee, Julian McAuley, and Derek Zhiyuan Cheng. How to train data-efficient llms. arXiv preprint arXiv:2402.09668, 2024. [find](https://scholar.google.com/scholar?q=Noveen+Sachdeva%2C+Benjamin+Coleman%2C+Wang%2DCheng+Kang%2C+Jianmo+Ni%2C+Lichan+Hong%2C+Ed+H+Chi%2C+James+Caverlee%2C+Julian+McAuley%2C+and+Derek+Zhiyuan+Cheng%2E+How+to+train+data%2Defficient+llms%2E+arXiv+preprint+arXiv%3A24)
54. Ritik Sachin Parkar, Jaehyung Kim, Jong Inn Park, and Dongyeop Kang. Selectllm: Can llms select important instructions to annotate? arXiv e-prints, pp. arXiv–2401, 2024. [find](https://scholar.google.com/scholar?q=Ritik+Sachin+Parkar%2C+Jaehyung+Kim%2C+Jong+Inn+Park%2C+and+Dongyeop+Kang%2E+Selectllm%3A+Can+llms+select+important+instructions+to+annotate%3F+arXiv+e%2Dprints%2C+pp%2E+arXiv%E2%80%932401%2C+2024%2E)
55. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+Communications+of+the+ACM%2C+64%289%29%3A99%E2%80%93106%2C+2021%2E)
56. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. [SocialIQA: Commonsense reasoning about social interactions](https://md.research.elamur.ai/free/en/papers/sap-2019-socialiqa-commonsense-reasoning-about-social/). In EMNLP, 2019.
57. Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4938–4947, 2020. [find](https://scholar.google.com/scholar?q=Paul%2DEdouard+Sarlin%2C+Daniel+DeTone%2C+Tomasz+Malisiewicz%2C+and+Andrew+Rabinovich%2E+Superglue%3A+Learning+feature+matching+with+graph+neural+networks%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+computer+vi)
58. Yunfan Shao, Linyang Li, Zhaoye Fei, Hang Yan, Dahua Lin, and Xipeng Qiu. Balanced data sampling for language model training with clustering. arXiv preprint arXiv:2402.14526, 2024. [find](https://scholar.google.com/scholar?q=Yunfan+Shao%2C+Linyang+Li%2C+Zhaoye+Fei%2C+Hang+Yan%2C+Dahua+Lin%2C+and+Xipeng+Qiu%2E+Balanced+data+sampling+for+language+model+training+with+clustering%2E+arXiv+preprint+arXiv%3A2402%2E14526%2C+2024%2E)
59. Vasu Sharma, Karthik Padthe, Newsha Ardalani, Kushal Tirumala, Russell Howes, Hu Xu, Po-Yao Huang, Shang-Wen Li, Armen Aghajanyan, and Gargi Ghosh. Text quality-based pruning for efficient training of language models. arXiv preprint arXiv:2405.01582, 2024. [find](https://scholar.google.com/scholar?q=Vasu+Sharma%2C+Karthik+Padthe%2C+Newsha+Ardalani%2C+Kushal+Tirumala%2C+Russell+Howes%2C+Hu+Xu%2C+Po%2DYao+Huang%2C+Shang%2DWen+Li%2C+Armen+Aghajanyan%2C+and+Gargi+Ghosh%2E+Text+quality%2Dbased+pruning+for+efficient+training+of)
60. Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint arXiv:2402.00159, 2024. [find](https://scholar.google.com/scholar?q=Luca+Soldaini%2C+Rodney+Kinney%2C+Akshita+Bhagia%2C+Dustin+Schwenk%2C+David+Atkinson%2C+Russell+Authur%2C+Ben+Bogin%2C+Khyathi+Chandu%2C+Jennifer+Dumas%2C+Yanai+Elazar%2C+et+al%2E+Dolma%3A+An+open+corpus+of+three+trillion+to)
61. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. [CommonsenseQA: A question answering challenge targeting commonsense knowledge](https://md.research.elamur.ai/free/en/papers/talmor-2019-commonsenseqa-question-answering-challenge-targeting/). In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Compu…
62. Megh Thakkar, Tolga Bolukbasi, Sriram Ganapathy, Shikhar Vashishth, Sarath Chandar, and Partha Talukdar. Self-influence guided data reweighting for language model pre-training. arXiv preprint arXiv:2311.00913, 2023. [find](https://scholar.google.com/scholar?q=Megh+Thakkar%2C+Tolga+Bolukbasi%2C+Sriram+Ganapathy%2C+Shikhar+Vashishth%2C+Sarath+Chandar%2C+and+Partha+Talukdar%2E+Self%2Dinfluence+guided+data+reweighting+for+language+model+pre%2Dtraining%2E+arXiv+preprint+arXiv%3A23)
63. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
64. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. [Glue: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). arXiv preprint arXiv:1804.07461, 2018.
65. Xinyi Wang, Yulia Tsvetkov, and Graham Neubig. Balancing training for multilingual neural machine translation. arXiv preprint arXiv:2004.06748, 2020. [find](https://scholar.google.com/scholar?q=Xinyi+Wang%2C+Yulia+Tsvetkov%2C+and+Graham+Neubig%2E+Balancing+training+for+multilingual+neural+machine+translation%2E+arXiv+preprint+arXiv%3A2004%2E06748%2C+2020%2E)
66. Johannes Welbl, Nelson F Liu, and Matt Gardner. [Crowdsourcing multiple choice science questions](https://md.research.elamur.ai/free/en/papers/welbl-2017-crowdsourcing-multiple-choice-science-questions/). arXiv preprint arXiv:1707.06209, 2017.
67. Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. Qurating: Selecting high-quality data for training language models, 2024. [find](https://scholar.google.com/scholar?q=Alexander+Wettig%2C+Aatmik+Gupta%2C+Saumya+Malik%2C+and+Danqi+Chen%2E+Qurating%3A+Selecting+high%2Dquality+data+for+training+language+models%2C+2024%2E)
68. Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, and Ves Stoyanov. Training trajectories of language models across scales. arXiv preprint arXiv:2212.09803, 2022. [find](https://scholar.google.com/scholar?q=Mengzhou+Xia%2C+Mikel+Artetxe%2C+Chunting+Zhou%2C+Xi+Victoria+Lin%2C+Ramakanth+Pasunuru%2C+Danqi+Chen%2C+Luke+Zettlemoyer%2C+and+Ves+Stoyanov%2E+Training+trajectories+of+language+models+across+scales%2E+arXiv+preprint+)
69. Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. [Less: Selecting influential data for targeted instruction tuning](https://md.research.elamur.ai/free/en/papers/xia-2024-less-selecting-influential-data-targeted/). arXiv preprint arXiv:2402.04333, 2024.
70. Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. arXiv preprint arXiv:2305.10429, 2023a. [find](https://scholar.google.com/scholar?q=Sang+Michael+Xie%2C+Hieu+Pham%2C+Xuanyi+Dong%2C+Nan+Du%2C+Hanxiao+Liu%2C+Yifeng+Lu%2C+Percy+Liang%2C+Quoc+V+Le%2C+Tengyu+Ma%2C+and+Adams+Wei+Yu%2E+Doremi%3A+Optimizing+data+mixtures+speeds+up+language+model+pretraining%2E+ar)
71. Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling. arXiv preprint arXiv:2302.03169, 2023b. [find](https://scholar.google.com/scholar?q=Sang+Michael+Xie%2C+Shibani+Santurkar%2C+Tengyu+Ma%2C+and+Percy+Liang%2E+Data+selection+for+language+models+via+importance+resampling%2E+arXiv+preprint+arXiv%3A2302%2E03169%2C+2023b%2E)
72. Chen Yang, Junzhuo Li, Xinyao Niu, Xinrun Du, Songyang Gao, Haoran Zhang, Zhaoliang Chen, Xingwei Qu, Ruibin Yuan, Yizhi Li, et al. The fine line: Navigating large language model pretraining with down-streaming capability analysis. arXiv preprint arXiv:2404.01204, 2024a. — [arXiv:2404.01204](https://arxiv.org/abs/2404.01204) [find](https://scholar.google.com/scholar?q=Chen+Yang%2C+Junzhuo+Li%2C+Xinyao+Niu%2C+Xinrun+Du%2C+Songyang+Gao%2C+Haoran+Zhang%2C+Zhaoliang+Chen%2C+Xingwei+Qu%2C+Ruibin+Yuan%2C+Yizhi+Li%2C+et+al%2E+The+fine+line%3A+Navigating+large+language+model+pretraining+with+down)
73. Yu Yang, Siddhartha Mishra, Jeffrey N Chiang, and Baharan Mirzasoleiman. Smalltolarge (s2l): Scalable data selection for fine-tuning large language models by summarizing training trajectories of small models. arXiv preprint arXiv:2403.07384, 2024b. — [arXiv:2403.07384](https://arxiv.org/abs/2403.07384) [find](https://scholar.google.com/scholar?q=Yu+Yang%2C+Siddhartha+Mishra%2C+Jeffrey+N+Chiang%2C+and+Baharan+Mirzasoleiman%2E+Smalltolarge+%28s2l%29%3A+Scalable+data+selection+for+fine%2Dtuning+large+language+models+by+summarizing+training+trajectories+of+small)
74. Jiasheng Ye, Peiju Liu, Tianxiang Sun, Yunhua Zhou, Jun Zhan, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. CoRR, abs/2403.16952, 2024. doi: 10.48550/ARXIV.2403.16952. URL https://doi.org/10.48550/arXiv. 2403.16952. [↗](https://doi.org/10.48550/arXiv)
75. Zichun Yu, Spandan Das, and Chenyan Xiong. Mates: Model-aware data selection for efficient pretraining with data influence models, 2024. [find](https://scholar.google.com/scholar?q=Zichun+Yu%2C+Spandan+Das%2C+and+Chenyan+Xiong%2E+Mates%3A+Model%2Daware+data+selection+for+efficient+pretraining+with+data+influence+models%2C+2024%2E)
76. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+arXiv+preprint+arXiv%3A1905%2E07830%2C+2019%2E)
77. Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. [Tinyllama: An open-source small language model](https://md.research.elamur.ai/free/en/papers/zhang-2024-tinyllama-open-source-small-language/). arXiv preprint arXiv:2401.02385, 2024a.
78. Yifan Zhang, Yifan Luo, Yang Yuan, and Andrew C Yao. Autonomous data selection with language models for mathematical texts. In ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models, 2024b. [find](https://scholar.google.com/scholar?q=Yifan+Zhang%2C+Yifan+Luo%2C+Yang+Yuan%2C+and+Andrew+C+Yao%2E+Autonomous+data+selection+with+language+models+for+mathematical+texts%2E+In+ICLR+2024+Workshop+on+Navigating+and+Addressing+Data+Problems+for+Foundat)

*Updated 30 October 2026*