---
title: "TiKMiX — paper by Wang et al. (2026) — summary and ideas"
description: "The ByteDance authors propose TiKMiX, a semi-dynamic compromise. 56 references. Among them: 0."
authors: ["Yifan Wang", "Binbinliu", "Fengze Liu", "Yuanfan Guo", "Jiyao Deng", "Xuecheng Wu", "Weidong Zhou", "Xiaohuan Zhou", "Taifeng Wang"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.261/"
doi: "10.18653/v1/2026.acl-long.261"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2026-tikmix-efficient-semi-dynamic-data/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2026-tikmix-efficient-semi-dynamic-data/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/45/) / TiKMiX: Efficient Semi-Dynamic Data Mixture via Data Influence for LLM Pre-training

# TiKMiX: Efficient Semi-Dynamic Data Mixture via Data Influence for LLM Pre-training

**Authors:** [Yifan Wang](https://md.research.elamur.ai/free/en/authors/yifan-wang/), Binbinliu, Fengze Liu, Yuanfan Guo, Jiyao Deng, [Xuecheng Wu](https://md.research.elamur.ai/free/en/authors/xuecheng-wu/), Weidong Zhou, [Xiaohuan Zhou](https://md.research.elamur.ai/free/en/authors/xiaohuan-zhou/) and [Taifeng Wang](https://md.research.elamur.ai/free/en/authors/taifeng-wang/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.261/) · [PDF](https://aclanthology.org/2026.acl-long.261.pdf) · [DOI 10.18653/v1/2026.acl-long.261](https://doi.org/10.18653/v1/2026.acl-long.261) · [OpenAlex](https://openalex.org/W7166862449)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Choosing domain proportions for LLM pre-training data is usually done once, either by hand or with proxy models. Such fixed mixtures ignore that a model's needs change during training. Fully online reweighting adapts to those changes but costs too much at scale. The ByteDance authors propose TiKMiX, a semi-dynamic compromise. It rests on the observation that the ranking of domain importance stays largely stable over long training stretches, so the mixture only needs updating periodically. To measure domain importance cheaply, they introduce Group Influence. This extends influence functions from single examples to whole data groups, using gradient accumulation and an EKFAC approximation of the Hessian. Mixing then becomes influence maximization over several validation sets, with diversity constraints, in two variants:
- **TiKMiX-D** optimizes the ratios directly.
- **TiKMiX-M** refines that result with a regression model of mixture perturbations.

On 1B-parameter models trained on 1T tokens, both variants beat DoReMi, QUAD and REGMIX at lower or comparable cost. This makes stage-wise re-mixing practical for multi-stage pre-training pipelines.

## Key points
- **Benchmarks:** nine in total. In-domain: MMLU, HellaSwag, ARC Easy, ARC Challenge, TriviaQA. Out-of-domain: PiQA, OpenBookQA, BoolQ, MathQA.
- **Average scores:** TiKMiX-M reaches 45.9 and TiKMiX-D 45.5, versus 43.9 for both REGMIX and QUAD and 43.7 for DoReMi. TiKMiX-M is best on 6 of 9 tasks, TiKMiX-D on 4 of 9.
- **Compute:** TiKMiX-D needs about 7.2e17 FLOPs, compared with 3.7e18 for REGMIX and 4.2e19 for DoReMi. That is roughly 20% of REGMIX's cost, an 80% reduction. No proxy models are trained.
- **Ablation:** Group Influence measured on only 0.1B tokens gives an average of 45.5. Loss-based observation on 5B and 10B tokens gives only 44.3 and 44.1.
- **Findings and limits:** data preferences shift as training progresses and differ across model scales. Validation has not yet been done on models larger than 7B.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Yifan Wang, Binbinliu, Fengze Liu, Yuanfan Guo, Jiyao Deng, Xuecheng Wu, Weidong Zhou, Xiaohuan Zhou and Taifeng Wang. 2026. TiKMiX: Efficient Semi-Dynamic Data Mixture via Data Influence for LLM Pre-training. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wang2026tikmix,
  title = {{TiKMiX: Efficient Semi-Dynamic Data Mixture via Data Influence for LLM Pre-training}},
  author = {Yifan Wang, Binbinliu, Fengze Liu, Yuanfan Guo, Jiyao Deng, Xuecheng Wu, Weidong Zhou, Xiaohuan Zhou and Taifeng Wang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.261},
  url = {https://doi.org/10.18653/v1/2026.acl-long.261}
}
```

- [← previous Tigrinya Number Verbalization: Rules, Algorithm, and Implementation](https://md.research.elamur.ai/free/en/papers/gaim-2026-tigrinya-number-verbalization-rules-algorithm/)
- [next → Time-for-Accuracy: Formalizing Chain-of-Thought as an Expansion of Logical Depth](https://md.research.elamur.ai/free/en/papers/wang-2026-time-accuracy-formalizing-chain-thought/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/45/)

## References

56 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alon Albalak, Liangming Pan, Colin Raffel, and William Yang Wang. 2023. [Efficient online data mixing for language model pre-training](https://md.research.elamur.ai/free/en/papers/albalak-2023-efficient-online-data-mixing-language/). arXiv preprint arXiv:2312.02406.
2. Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. [Mathqa: Towards interpretable math word problem solving with operation-based formalisms](https://md.research.elamur.ai/free/en/papers/amini-2019-mathqa-interpretable-math-word-problem/). arXiv preprint arXiv:1905.13319.
3. Zachary Ankner, Cody Blakeney, Kartik Sreenivasan, Max Marion, Matthew L Leavitt, and Mansheej Paul. 2024. Perplexed by perplexity: Perplexity-based data pruning with small reference models. arXiv preprint arXiv:2405.20541. — [arXiv:2405.20541](https://arxiv.org/abs/2405.20541) [find](https://scholar.google.com/scholar?q=Zachary+Ankner%2C+Cody+Blakeney%2C+Kartik+Sreenivasan%2C+Max+Marion%2C+Matthew+L+Leavitt%2C+and+Mansheej+Paul%2E+2024%2E+Perplexed+by+perplexity%3A+Perplexity%2Dbased+data+pruning+with+small+reference+models%2E+arXiv+pre)
4. Tianyi Bai, Ling Yang, Zhen Hao Wong, Jiahui Peng, Xinlin Zhuang, Chi Zhang, Lijun Wu, Jiantao Qiu, Wentao Zhang, Binhang Yuan, and Conghui He. 2024. Multi-agent collaborative data selection for efficient LLM pretraining. CoRR, abs/2410.08102. — [arXiv:2410.08102](https://arxiv.org/abs/2410.08102) [find](https://scholar.google.com/scholar?q=Tianyi+Bai%2C+Ling+Yang%2C+Zhen+Hao+Wong%2C+Jiahui+Peng%2C+Xinlin+Zhuang%2C+Chi+Zhang%2C+Lijun+Wu%2C+Jiantao+Qiu%2C+Wentao+Zhang%2C+Binhang+Yuan%2C+and+Conghui+He%2E+2024%2E+Multi%2Dagent+collaborative+data+selection+for+effic)
5. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. [PIQA: reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference,…
6. Mayee F Chen, Michael Y Hu, Nicholas Lourie, Kyunghyun Cho, and Christopher Ré. 2024a. Aioli: A unified optimization framework for language model data mixing. arXiv preprint arXiv:2411.05735. [find](https://scholar.google.com/scholar?q=Mayee+F+Chen%2C+Michael+Y+Hu%2C+Nicholas+Lourie%2C+Kyunghyun+Cho%2C+and+Christopher+R%C3%A9%2E+2024a%2E+Aioli%3A+A+unified+optimization+framework+for+language+model+data+mixing%2E+arXiv+preprint+arXiv%3A2411%2E05735%2E)
7. Xuxi Chen, Zhendong Wang, Daouda Sow, Junjie Yang, Tianlong Chen, Yingbin Liang, Mingyuan Zhou, and Zhangyang Wang. 2024b. Take the bull by the horns: Hard sample-reweighted continual training improves llm generalization. arXiv preprint arXiv:2402.14270. [find](https://scholar.google.com/scholar?q=Xuxi+Chen%2C+Zhendong+Wang%2C+Daouda+Sow%2C+Junjie+Yang%2C+Tianlong+Chen%2C+Yingbin+Liang%2C+Mingyuan+Zhou%2C+and+Zhangyang+Wang%2E+2024b%2E+Take+the+bull+by+the+horns%3A+Hard+sample%2Dreweighted+continual+training+improve)
8. Sang Keun Choe, Hwijeen Ahn, Juhan Bae, Kewen Zhao, Minsoo Kang, Youngseog Chung, Adithya Pratapa, Willie Neiswanger, Emma Strubell, Teruko Mitamura, Jeff G. Schneider, Eduard H. Hovy, Roger B. Grosse, and Eric P. Xing. 2024. What is your data worth to gpt? llm-scale data valuation with influence fu… [find](https://scholar.google.com/scholar?q=Sang+Keun+Choe%2C+Hwijeen+Ahn%2C+Juhan+Bae%2C+Kewen+Zhao%2C+Minsoo+Kang%2C+Youngseog+Chung%2C+Adithya+Pratapa%2C+Willie+Neiswanger%2C+Emma+Strubell%2C+Teruko+Mitamura%2C+Jeff+G%2E+Schneider%2C+Eduard+H%2E+Hovy%2C+Roger+B%2E+Grosse)
9. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. [Boolq: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). arXiv preprint arXiv:1905.10044.
10. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv preprint arXiv:1803.05457.
11. Shizhe Diao, Yu Yang, Yonggan Fu, Xin Dong, Dan Su, Markus Kliegl, Zijia Chen, Peter Belcak, Yoshi Suhara, Hongxu Yin, Mostofa Patwary, Yingyan Lin, Jan Kautz, and Pavlo Molchanov. 2025. CLIMB: clustering-based iterative data mixture bootstrapping for language model pre-training. CoRR, abs/2504.1316… — [arXiv:2504.13161](https://arxiv.org/abs/2504.13161) [find](https://scholar.google.com/scholar?q=Shizhe+Diao%2C+Yu+Yang%2C+Yonggan+Fu%2C+Xin+Dong%2C+Dan+Su%2C+Markus+Kliegl%2C+Zijia+Chen%2C+Peter+Belcak%2C+Yoshi+Suhara%2C+Hongxu+Yin%2C+Mostofa+Patwary%2C+Yingyan+Lin%2C+Jan+Kautz%2C+and+Pavlo+Molchanov%2E+2025%2E+CLIMB%3A+cluste)
12. Simin Fan, Matteo Pagliardini, and Martin Jaggi. 2023. Doge: Domain reweighting with generalization estimation. arXiv preprint arXiv:2310.15393. — [arXiv:2310.15393](https://arxiv.org/abs/2310.15393) [find](https://scholar.google.com/scholar?q=Simin+Fan%2C+Matteo+Pagliardini%2C+and+Martin+Jaggi%2E+2023%2E+Doge%3A+Domain+reweighting+with+generalization+estimation%2E+arXiv+preprint+arXiv%3A2310%2E15393%2E+%E2%80%94+%5BarXiv%3A2310%2E15393%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2310%2E15393%29)
13. Luciano Floridi and Massimo Chiriatti. 2020. Gpt-3: Its nature, scope, limits, and consequences. Minds and machines, 30(4):681–694. [find](https://scholar.google.com/scholar?q=Luciano+Floridi+and+Massimo+Chiriatti%2E+2020%2E+Gpt%2D3%3A+Its+nature%2C+scope%2C+limits%2C+and+consequences%2E+Minds+and+machines%2C+30%284%29%3A681%E2%80%93694%2E)
14. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2021. The pile: An 800gb dataset of diverse text for language modeling. CoRR, abs/2101.00027. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+Shawn+Presser%2C+and+Connor+Leahy%2E+2021%2E+The+pile%3A+An+800gb+datas)
15. Thomas George, César Laurent, Xavier Bouthillier, Nicolas Ballas, and Pascal Vincent. 2018. Fast approximate natural gradient descent in a kronecker factored eigenbasis. Advances in neural information processing systems, 31. [find](https://scholar.google.com/scholar?q=Thomas+George%2C+C%C3%A9sar+Laurent%2C+Xavier+Bouthillier%2C+Nicolas+Ballas%2C+and+Pascal+Vincent%2E+2018%2E+Fast+approximate+natural+gradient+descent+in+a+kronecker+factored+eigenbasis%2E+Advances+in+neural+information)
16. David Grangier, Simin Fan, Skyler Seto, and Pierre Ablin. 2024. Task-adaptive pretrained language models via clustered-importance sampling. arXiv preprint arXiv:2410.03735. — [arXiv:2410.03735](https://arxiv.org/abs/2410.03735) [find](https://scholar.google.com/scholar?q=David+Grangier%2C+Simin+Fan%2C+Skyler+Seto%2C+and+Pierre+Ablin%2E+2024%2E+Task%2Dadaptive+pretrained+language+models+via+clustered%2Dimportance+sampling%2E+arXiv+preprint+arXiv%3A2410%2E03735%2E+%E2%80%94+%5BarXiv%3A2410%2E03735%5D%28https%3A)
17. Roger B. Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamile Lukosiute, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. 2023. Studying large language model general… [find](https://scholar.google.com/scholar?q=Roger+B%2E+Grosse%2C+Juhan+Bae%2C+Cem+Anil%2C+Nelson+Elhage%2C+Alex+Tamkin%2C+Amirhossein+Tajdini%2C+Benoit+Steiner%2C+Dustin+Li%2C+Esin+Durmus%2C+Ethan+Perez%2C+Evan+Hubinger%2C+Kamile+Lukosiute%2C+Karina+Nguyen%2C+Nicholas+Jos)
18. Madhuri Gupta and Bharat Gupta. 2018. An ensemble model for breast cancer prediction using sequential least squares programming method (slsqp). In 2018 eleventh international conference on contemporary computing (IC3), pages 1–3. IEEE. [find](https://scholar.google.com/scholar?q=Madhuri+Gupta+and+Bharat+Gupta%2E+2018%2E+An+ensemble+model+for+breast+cancer+prediction+using+sequential+least+squares+programming+method+%28slsqp%29%2E+In+2018+eleventh+international+conference+on+contemporar)
19. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. [Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing](https://md.research.elamur.ai/free/en/papers/he-2021-debertav3-improving-deberta-using-electra/). arXiv preprint arXiv:2111.09543.
20. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300.
21. Xin Jin, Hongyu Zhu, Siyuan Li, Zedong Wang, Zicheng Liu, Juanxi Tian, Chang Yu, Huafeng Qin, and Stan Z Li. 2024. A survey on mixup augmentations and beyond. arXiv preprint arXiv:2409.05202. — [arXiv:2409.05202](https://arxiv.org/abs/2409.05202) [find](https://scholar.google.com/scholar?q=Xin+Jin%2C+Hongyu+Zhu%2C+Siyuan+Li%2C+Zedong+Wang%2C+Zicheng+Liu%2C+Juanxi+Tian%2C+Chang+Yu%2C+Huafeng+Qin%2C+and+Stan+Z+Li%2E+2024%2E+A+survey+on+mixup+augmentations+and+beyond%2E+arXiv+preprint+arXiv%3A2409%2E05202%2E+%E2%80%94+%5BarXiv)
22. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. [Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). arXiv preprint arXiv:1705.03551.
23. Feiyang Kang, Yifan Sun, Bingbing Wen, Si Chen, Dawn Song, Rafid Mahmood, and Ruoxi Jia. 2024. Autoscale: Automatic prediction of compute-optimal data compositions for training llms. [find](https://scholar.google.com/scholar?q=Feiyang+Kang%2C+Yifan+Sun%2C+Bingbing+Wen%2C+Si+Chen%2C+Dawn+Song%2C+Rafid+Mahmood%2C+and+Ruoxi+Jia%2E+2024%2E+Autoscale%3A+Automatic+prediction+of+compute%2Doptimal+data+compositions+for+training+llms%2E)
24. Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30. [find](https://scholar.google.com/scholar?q=Guolin+Ke%2C+Qi+Meng%2C+Thomas+Finley%2C+Taifeng+Wang%2C+Wei+Chen%2C+Weidong+Ma%2C+Qiwei+Ye%2C+and+Tie%2DYan+Liu%2E+2017%2E+Lightgbm%3A+A+highly+efficient+gradient+boosting+decision+tree%2E+Advances+in+neural+information+pro)
25. Pang Wei Koh and Percy Liang. 2017. [Understanding black-box predictions via influence functions](https://md.research.elamur.ai/free/en/papers/koh-2017-understanding-black-box-predictions-influence/). In International conference on machine learning, pages 1885–1894. PMLR.
26. Pang Wei W Koh, Kai-Siang Ang, Hubert Teo, and Percy S Liang. 2019. On the accuracy of influence functions for measuring group effects. Advances in neural information processing systems, 32. [find](https://scholar.google.com/scholar?q=Pang+Wei+W+Koh%2C+Kai%2DSiang+Ang%2C+Hubert+Teo%2C+and+Percy+S+Liang%2E+2019%2E+On+the+accuracy+of+influence+functions+for+measuring+group+effects%2E+Advances+in+neural+information+processing+systems%2C+32%2E)
27. Siqi Kou, Qingyuan Tian, Hanwen Xu, Zihao Zeng, and Zhijie Deng. 2025. Which data attributes stimulate math and code reasoning? an investigation via influence functions. arXiv preprint arXiv:2505.19949. [find](https://scholar.google.com/scholar?q=Siqi+Kou%2C+Qingyuan+Tian%2C+Hanwen+Xu%2C+Zihao+Zeng%2C+and+Zhijie+Deng%2E+2025%2E+Which+data+attributes+stimulate+math+and+code+reasoning%3F+an+investigation+via+influence+functions%2E+arXiv+preprint+arXiv%3A2505%2E1994)
28. Xinyu Lin, Wenjie Wang, Yongqi Li, Shuo Yang, Fuli Feng, Yinwei Wei, and Tat-Seng Chua. 2024a. Data-efficient fine-tuning for llm-based recommendation. In Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, pages 365–374. [find](https://scholar.google.com/scholar?q=Xinyu+Lin%2C+Wenjie+Wang%2C+Yongqi+Li%2C+Shuo+Yang%2C+Fuli+Feng%2C+Yinwei+Wei%2C+and+Tat%2DSeng+Chua%2E+2024a%2E+Data%2Defficient+fine%2Dtuning+for+llm%2Dbased+recommendation%2E+In+Proceedings+of+the+47th+international+ACM+SIG)
29. Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, and Weizhu Chen. 2024b. Rho-1: Not all tokens are what you need. CoRR, abs/2404.07965. [find](https://scholar.google.com/scholar?q=Zhenghao+Lin%2C+Zhibin+Gou%2C+Yeyun+Gong%2C+Xiao+Liu%2C+Yelong+Shen%2C+Ruochen+Xu%2C+Chen+Lin%2C+Yujiu+Yang%2C+Jian+Jiao%2C+Nan+Duan%2C+and+Weizhu+Chen%2E+2024b%2E+Rho%2D1%3A+Not+all+tokens+are+what+you+need%2E+CoRR%2C+abs%2F2404%2E0796)
30. Fengze Liu, Weidong Zhou, Binbin Li, Zhimiao Yu, Yifan Zhang, Haobin Lin, Yifeng Yu, Bingni Zhang, Xiaohuan Zhou, Taifeng Wang, and Yong Cao. 2025a. Quadmix: Quality-diversity balanced data selection for efficient LLM pretraining. CoRR, abs/2504.16511. — [arXiv:2504.16511](https://arxiv.org/abs/2504.16511) [find](https://scholar.google.com/scholar?q=Fengze+Liu%2C+Weidong+Zhou%2C+Binbin+Li%2C+Zhimiao+Yu%2C+Yifan+Zhang%2C+Haobin+Lin%2C+Yifeng+Yu%2C+Bingni+Zhang%2C+Xiaohuan+Zhou%2C+Taifeng+Wang%2C+and+Yong+Cao%2E+2025a%2E+Quadmix%3A+Quality%2Ddiversity+balanced+data+selection+)
31. Jiaheng Liu, Dawei Zhu, Zhiqi Bai, Yancheng He, Huanxuan Liao, Haoran Que, Zekun Wang, Chenchen Zhang, Ge Zhang, Jiebin Zhang, Yuanxing Zhang, Zhuo Chen, Hangyu Guo, Shilong Li, Ziqiang Liu, Yong Shan, Yifan Song, Jiayi Tian, Wenhao Wu, and 18 others. 2025b. [A comprehensive survey on long context la…](https://md.research.elamur.ai/free/en/papers/liu-2025-comprehensive-survey-long-context-language/)
32. Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. 2024. [Regmix: Data mixture as regression for language model pre-training](https://md.research.elamur.ai/free/en/papers/liu-2024-regmix-data-mixture-regression-language/). arXiv preprint arXiv:2407.01492.
33. Wei-Liem Loh. 2021. On latin hypercube sampling. The annals of statistics. [find](https://scholar.google.com/scholar?q=Wei%2DLiem+Loh%2E+2021%2E+On+latin+hypercube+sampling%2E+The+annals+of+statistics%2E)
34. Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. 2023. When less is more: Investigating data pruning for pretraining llms at scale. arXiv preprint arXiv:2309.04564. [find](https://scholar.google.com/scholar?q=Max+Marion%2C+Ahmet+%C3%9Cst%C3%BCn%2C+Luiza+Pozzobon%2C+Alex+Wang%2C+Marzieh+Fadaee%2C+and+Sara+Hooker%2E+2023%2E+When+less+is+more%3A+Investigating+data+pruning+for+pretraining+llms+at+scale%2E+arXiv+preprint+arXiv%3A2309%2E04564%2E)
35. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789. [find](https://scholar.google.com/scholar?q=Todor+Mihaylov%2C+Peter+Clark%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+2018%2E+Can+a+suit+of+armor+conduct+electricity%3F+a+new+dataset+for+open+book+question+answering%2E+arXiv+preprint+arXiv%3A1809%2E02789%2E)
36. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The refinedweb dataset for falcon llm: Outperforming curated corpora with web data only. Advances in Neural Information Processi… [find](https://scholar.google.com/scholar?q=Guilherme+Penedo%2C+Quentin+Malartic%2C+Daniel+Hesslow%2C+Ruxandra+Cojocaru%2C+Hamza+Alobeidli%2C+Alessandro+Cappelli%2C+Baptiste+Pannier%2C+Ebtesam+Almazrouei%2C+and+Julien+Launay%2E+2023%2E+The+refinedweb+dataset+for+f)
37. Yunfan Shao, Linyang Li, Zhaoye Fei, Hang Yan, Dahua Lin, and Xipeng Qiu. 2024. Balanced data sampling for language model training with clustering. arXiv preprint arXiv:2402.14526. [find](https://scholar.google.com/scholar?q=Yunfan+Shao%2C+Linyang+Li%2C+Zhaoye+Fei%2C+Hang+Yan%2C+Dahua+Lin%2C+and+Xipeng+Qiu%2E+2024%2E+Balanced+data+sampling+for+language+model+training+with+clustering%2E+arXiv+preprint+arXiv%3A2402%2E14526%2E)
38. Vasu Sharma, Karthik Padthe, Newsha Ardalani, Kushal Tirumala, Russell Howes, Hu Xu, Po-Yao Huang, Daniel Li Chen, Armen Aghajanyan, Gargi Ghosh, and Luke Zettlemoyer. 2025. Text quality-based pruning for efficient training of language models. J. Data-centric Mach. Learn. Res., 2:(13):1–13. [find](https://scholar.google.com/scholar?q=Vasu+Sharma%2C+Karthik+Padthe%2C+Newsha+Ardalani%2C+Kushal+Tirumala%2C+Russell+Howes%2C+Hu+Xu%2C+Po%2DYao+Huang%2C+Daniel+Li+Chen%2C+Armen+Aghajanyan%2C+Gargi+Ghosh%2C+and+Luke+Zettlemoyer%2E+2025%2E+Text+quality%2Dbased+pruning)
39. Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Raghavi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, and… [find](https://scholar.google.com/scholar?q=Luca+Soldaini%2C+Rodney+Kinney%2C+Akshita+Bhagia%2C+Dustin+Schwenk%2C+David+Atkinson%2C+Russell+Authur%2C+Ben+Bogin%2C+Khyathi+Raghavi+Chandu%2C+Jennifer+Dumas%2C+Yanai+Elazar%2C+Valentin+Hofmann%2C+Ananya+Harsh+Jha%2C+Sachi)
40. Zhixu Silvia Tao, Kasper Vinken, Hao-Wei Yeh, Avi Cooper, and Xavier Boix. 2025. Merge to mix: Mixing datasets via model merging. arXiv preprint arXiv:2505.16066. — [arXiv:2505.16066](https://arxiv.org/abs/2505.16066) [find](https://scholar.google.com/scholar?q=Zhixu+Silvia+Tao%2C+Kasper+Vinken%2C+Hao%2DWei+Yeh%2C+Avi+Cooper%2C+and+Xavier+Boix%2E+2025%2E+Merge+to+mix%3A+Mixing+datasets+via+model+merging%2E+arXiv+preprint+arXiv%3A2505%2E16066%2E+%E2%80%94+%5BarXiv%3A2505%2E16066%5D%28https%3A%2F%2Farxiv%2Eor)
41. Kimi Team. 2025a. [Kimi K2: open agentic intelligence](https://md.research.elamur.ai/free/en/papers/kimi-team-2025-kimi-k2-open-agentic/). CoRR, abs/2507.20534.
42. Llama Team. 2024a. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). CoRR, abs/2407.21783.
43. Qwen Team. 2024b. Qwen2 technical report. arXiv preprint arXiv:2407.10671. [find](https://scholar.google.com/scholar?q=Qwen+Team%2E+2024b%2E+Qwen2+technical+report%2E+arXiv+preprint+arXiv%3A2407%2E10671%2E)
44. Alexander Wettig, Kyle Lo, Sewon Min, Hannaneh Hajishirzi, Danqi Chen, and Luca Soldaini. 2025. Organize the web: Constructing domains enhances pre-training data curation. arXiv preprint arXiv:2502.10341. [find](https://scholar.google.com/scholar?q=Alexander+Wettig%2C+Kyle+Lo%2C+Sewon+Min%2C+Hannaneh+Hajishirzi%2C+Danqi+Chen%2C+and+Luca+Soldaini%2E+2025%2E+Organize+the+web%3A+Constructing+domains+enhances+pre%2Dtraining+data+curation%2E+arXiv+preprint+arXiv%3A2502%2E10)
45. Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. 2023. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36:69798–69818. [find](https://scholar.google.com/scholar?q=Sang+Michael+Xie%2C+Hieu+Pham%2C+Xuanyi+Dong%2C+Nan+Du%2C+Hanxiao+Liu%2C+Yifeng+Lu%2C+Percy+S+Liang%2C+Quoc+V+Le%2C+Tengyu+Ma%2C+and+Adams+Wei+Yu%2E+2023%2E+Doremi%3A+Optimizing+data+mixtures+speeds+up+language+model+pretrai)
46. Jiasheng Ye, Peiju Liu, Tianxiang Sun, Jun Zhan, Yunhua Zhou, and Xipeng Qiu. 2024. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. arXiv preprint arXiv:2403.16952. [find](https://scholar.google.com/scholar?q=Jiasheng+Ye%2C+Peiju+Liu%2C+Tianxiang+Sun%2C+Jun+Zhan%2C+Yunhua+Zhou%2C+and+Xipeng+Qiu%2E+2024%2E+Data+mixing+laws%3A+Optimizing+data+mixtures+by+predicting+language+modeling+performance%2E+arXiv+preprint+arXiv%3A2403%2E16)
47. Shi Yu, Zhiyuan Liu, and Chenyan Xiong. 2025a. Craw4llm: Efficient web crawling for llm pretraining. arXiv preprint arXiv:2502.13347. [find](https://scholar.google.com/scholar?q=Shi+Yu%2C+Zhiyuan+Liu%2C+and+Chenyan+Xiong%2E+2025a%2E+Craw4llm%3A+Efficient+web+crawling+for+llm+pretraining%2E+arXiv+preprint+arXiv%3A2502%2E13347%2E)
48. Zichun Yu, Spandan Das, and Chenyan Xiong. 2024. Mates: Model-aware data selection for efficient pre-training with data influence models. Advances in Neural Information Processing Systems, 37:108735– 108759. [find](https://scholar.google.com/scholar?q=Zichun+Yu%2C+Spandan+Das%2C+and+Chenyan+Xiong%2E+2024%2E+Mates%3A+Model%2Daware+data+selection+for+efficient+pre%2Dtraining+with+data+influence+models%2E+Advances+in+Neural+Information+Processing+Systems%2C+37%3A108735%E2%80%93+)
49. Zichun Yu, Fei Peng, Jie Lei, Arnold Overwijk, Wen-tau Yih, and Chenyan Xiong. 2025b. Data-efficient pretraining with group-level data influence modeling. arXiv preprint arXiv:2502.14709. — [arXiv:2502.14709](https://arxiv.org/abs/2502.14709) [find](https://scholar.google.com/scholar?q=Zichun+Yu%2C+Fei+Peng%2C+Jie+Lei%2C+Arnold+Overwijk%2C+Wen%2Dtau+Yih%2C+and+Chenyan+Xiong%2E+2025b%2E+Data%2Defficient+pretraining+with+group%2Dlevel+data+influence+modeling%2E+arXiv+preprint+arXiv%3A2502%2E14709%2E+%E2%80%94+%5BarXiv%3A250)
50. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+arXiv+preprint+arXiv%3A1905%2E07830%2E)
51. Chi Zhang, Huaping Zhong, Kuan Zhang, Chengliang Chai, Rui Wang, Xinlin Zhuang, Tianyi Bai, Jiantao Qiu, Lei Cao, Ju Fan, Ye Yuan, Guoren Wang, and Conghui He. 2025a. Harnessing diversity for important data selection in pretraining large language models. In The Thirteenth International Conference on… [find](https://scholar.google.com/scholar?q=Chi+Zhang%2C+Huaping+Zhong%2C+Kuan+Zhang%2C+Chengliang+Chai%2C+Rui+Wang%2C+Xinlin+Zhuang%2C+Tianyi+Bai%2C+Jiantao+Qiu%2C+Lei+Cao%2C+Ju+Fan%2C+Ye+Yuan%2C+Guoren+Wang%2C+and+Conghui+He%2E+2025a%2E+Harnessing+diversity+for+importan)
52. Qwen Team. 2025b. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). CoRR, abs/2505.09388.
53. Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. [Tinyllama: An open-source small language model](https://md.research.elamur.ai/free/en/papers/zhang-2024-tinyllama-open-source-small-language/). arXiv preprint arXiv:2401.02385.
54. Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari Morcos. 2023. [D4: Improving llm pretraining via document de-duplication and diversification](https://md.research.elamur.ai/free/en/papers/tirumala-2023-d4-improving-llm-pretraining-document/). Advances in Neural Information Processing Systems, 36:53983–53995.
55. Xuanliang Zhang, Dingzirui Wang, Longxu Dou, Qingfu Zhu, and Wanxiang Che. 2025b. A survey of table reasoning with large language models. Frontiers of Computer Science, 19(9):199348. [find](https://scholar.google.com/scholar?q=Xuanliang+Zhang%2C+Dingzirui+Wang%2C+Longxu+Dou%2C+Qingfu+Zhu%2C+and+Wanxiang+Che%2E+2025b%2E+A+survey+of+table+reasoning+with+large+language+models%2E+Frontiers+of+Computer+Science%2C+19%289%29%3A199348%2E)
56. Kun Wang, Guibin Zhang, Zhenhong Zhou, Jiahao Wu, Miao Yu, Shiqian Zhao, Chenlong Yin, Jinhu Fu, Yibo Yan, Hanjun Luo, Liang Lin, Zhihao Xu, Haolang Lu, Xinye Cao, Xinyun Zhou, Weifei Jin, Fanci Meng, Junyuan Mao, Hao Wu, and 63 others. 2025. A comprehensive survey in llm(-agent) full stack safety:… — [arXiv:2504.15585](https://arxiv.org/abs/2504.15585) [find](https://scholar.google.com/scholar?q=Kun+Wang%2C+Guibin+Zhang%2C+Zhenhong+Zhou%2C+Jiahao+Wu%2C+Miao+Yu%2C+Shiqian+Zhao%2C+Chenlong+Yin%2C+Jinhu+Fu%2C+Yibo+Yan%2C+Hanjun+Luo%2C+Liang+Lin%2C+Zhihao+Xu%2C+Haolang+Lu%2C+Xinye+Cao%2C+Xinyun+Zhou%2C+Weifei+Jin%2C+Fanci+Meng%2C)

*Updated 1 November 2026*