---
title: "Rethinking Data Mixing from the Perspective of Large Language"
description: "Building on this, they propose DoGraph. 37.9% accuracy. 37 references. Among them: 0."
authors: ["Yuanjian Xu", "Tianze Sun", "Changwei Xu", "XinLong Zhao", "Jianing Hao", "Ran Chen", "Yang Liu", "Ruijie Xu", "Stephen Chen", "Guang Zhang"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)"
source: "https://aclanthology.org/2026.acl-short.28/"
doi: "10.18653/v1/2026.acl-short.28"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xu-2026-rethinking-data-mixing-perspective-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xu-2026-rethinking-data-mixing-perspective-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/47/) / Rethinking Data Mixing from the Perspective of Large Language Models

# Rethinking Data Mixing from the Perspective of Large Language Models

**Authors:** Yuanjian Xu, Tianze Sun, Changwei Xu, XinLong Zhao, Jianing Hao, Ran Chen, [Yang Liu](https://md.research.elamur.ai/free/en/authors/yang-liu/), [Ruijie Xu](https://md.research.elamur.ai/free/en/authors/ruijie-xu/), Stephen Chen and [Guang Zhang](https://md.research.elamur.ai/free/en/authors/guang-zhang/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-short.28/) · [PDF](https://aclanthology.org/2026.acl-short.28.pdf) · [DOI 10.18653/v1/2026.acl-short.28](https://doi.org/10.18653/v1/2026.acl-short.28) · [OpenAlex](https://openalex.org/W7166876608)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
The paper asks how training data from different domains should be weighted when pretraining LLMs. It also asks whether human-assigned domain labels (e.g. C4, Wikipedia, ArXiv, Book) match what the model itself treats as distinct. The authors observe that per-sample gradients form clearly separated clusters by source early in training but blend together later. Their conclusion is that the model's notion of a domain shifts over the course of training.

They treat a domain as a probability distribution over token sequences. Under a linearized Transformer, they prove that the gap between two domains' expected gradients equals an MMD with a gradient-induced kernel. Building on this, they propose DoGraph. At each epoch, DoGraph randomly projects per-sample gradients to a lower dimension and clusters them with K-means to form model-defined domains. It then optimizes domain weights over the simplex using uncertainty weighting.

On SlimPajama with GPT-2-style models, DoGraph achieves the best average downstream accuracy and the lowest validation perplexity among the compared methods. The work suggests that data mixing should follow the model's own view of domains rather than fixed human labels.

## Key points
- **Theory:** Under a linearized self-attention setting, the squared distance between domain-mean gradients equals MMD² with kernel ⟨g(s), g(s′)⟩. Domains can therefore be compared through their gradients instead of token statistics.
- **Method:** DoGraph combines several steps:
  - Gaussian random projection of gradients, justified by Johnson–Lindenstrauss.
  - K-means clustering into graph nodes, recomputed every epoch.
  - Simplex-constrained weight optimization.
- **Setup:** The setup has four components:
  - **Models:** Decoder-only models at 210M and 300M parameters, with main results on GPT-2 Medium.
  - **Data:** SlimPajama, covering seven domains.
  - **Evaluation:** Nine benchmarks, including HellaSwag, PiQA, ARC-E and Lambada.
  - **Additional experiments:** LLaMA-1.1B results appear in the appendix.
- **Downstream accuracy:** DoGraph averages 37.9% accuracy. The baselines score as follows:
  - Dynamic Loss-Based: 37.2
  - DoReMi: 36.5
  - DOGE: 36.5
  - RegMix: 35.7
  - Uniform: 35.4
  - Data Mixing Law: 35.4
- **Perplexity:** DoGraph reaches a validation PPL of 3.09. Loss-based weighting is close at 3.10. Uniform (4.13), RegMix (4.51) and Data Mixing Law (4.50) are much worse. Code is released at github.com/xuyj233/Dograph.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Yuanjian Xu, Tianze Sun, Changwei Xu, XinLong Zhao, Jianing Hao, Ran Chen, Yang Liu, Ruijie Xu, Stephen Chen and Guang Zhang. 2026. Rethinking Data Mixing from the Perspective of Large Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{xu2026rethinking,
  title = {{Rethinking Data Mixing from the Perspective of Large Language Models}},
  author = {Yuanjian Xu, Tianze Sun, Changwei Xu, XinLong Zhao, Jianing Hao, Ran Chen, Yang Liu, Ruijie Xu, Stephen Chen and Guang Zhang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-short.28},
  url = {https://doi.org/10.18653/v1/2026.acl-short.28}
}
```

- [← previous Rethinking Composed Image Retrieval Evaluation: A Fine-Grained Benchmark from Image Editing](https://md.research.elamur.ai/free/en/papers/song-2026-rethinking-composed-image-retrieval-evaluation/)
- [next → Rethinking Data Mixture for Large Language Models: A Comprehensive Survey and New Perspectives](https://md.research.elamur.ai/free/en/papers/liu-2026-rethinking-data-mixture-large-language/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/47/)

## References

37 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Johannes Welbl, Aidan Clark, and 1 others. 2022. [Training compute-optimal large language models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/). arXiv preprint arXiv:2203.15556.
2. Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, and 1 others. [A survey on data selection for language models](https://md.research.elamur.ai/free/en/papers/albalak-2024-survey-data-selection-language-models/). Transactions on Machine Learning Research.
3. Feiyang Kang, Yifan Sun, Bingbing Wen, Si Chen, Dawn Song, Rafid Mahmood, and Ruoxi Jia. 2024. AutoScale: Automatic Prediction of Compute-Optimal Data Composition for Training LLMs. arXiv preprint arXiv:2407.20177. — [arXiv:2407.20177](https://arxiv.org/abs/2407.20177) [find](https://scholar.google.com/scholar?q=Feiyang+Kang%2C+Yifan+Sun%2C+Bingbing+Wen%2C+Si+Chen%2C+Dawn+Song%2C+Rafid+Mahmood%2C+and+Ruoxi+Jia%2E+2024%2E+AutoScale%3A+Automatic+Prediction+of+Compute%2DOptimal+Data+Composition+for+Training+LLMs%2E+arXiv+preprint+arX)
4. Alexei Baevski and 1 others. 2024. [Datacomp: In search of the next generation of multimodal datasets](https://md.research.elamur.ai/free/en/papers/gadre-2023-datacomp-search-next-generation-multimodal/). arXiv preprint arXiv:2304.14108.
5. Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2013. [Representation Learning: A Review and New Perspectives](https://md.research.elamur.ai/free/en/papers/bengio-2012-representation-learning-review-new-perspectives/). IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798–1828.
6. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). arXiv preprint arXiv:2001.08361.
7. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, and 1 others. 2020. [PiQA: Reasoning About Physical Commonsense in Natural Language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). In Proceedings of the AAAI Conference on Artificial Intelligence.
8. Pang Wei Koh and Percy Liang. 2017. [Understanding Black-box Predictions via Influence Functions](https://md.research.elamur.ai/free/en/papers/koh-2017-understanding-black-box-predictions-influence/). In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1885–1894. PMLR.
9. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think You Have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv preprint arXiv:1803.05457.
10. Belinda Z. Li and 1 others. 2024. Dolma: An open corpus of three trillion tokens for language model pre-training research. arXiv preprint arXiv:2402.00159. [find](https://scholar.google.com/scholar?q=Belinda+Z%2E+Li+and+1+others%2E+2024%2E+Dolma%3A+An+open+corpus+of+three+trillion+tokens+for+language+model+pre%2Dtraining+research%2E+arXiv+preprint+arXiv%3A2402%2E00159%2E)
11. Simin Fan, Matteo Pagliardini, and Martin Jaggi. 2023. DOGE: Domain reweighting with generalization estimation. In Second Agent Learning in Open-Endedness Workshop. [find](https://scholar.google.com/scholar?q=Simin+Fan%2C+Matteo+Pagliardini%2C+and+Martin+Jaggi%2E+2023%2E+DOGE%3A+Domain+reweighting+with+generalization+estimation%2E+In+Second+Agent+Learning+in+Open%2DEndedness+Workshop%2E)
12. Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020. [LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning](https://md.research.elamur.ai/free/en/papers/liu-2020-logiqa-challenge-dataset-machine-reading/). arXiv preprint arXiv:2007.08124.
13. Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. 2019. Deep Ensembles: A Loss Landscape Perspective. arXiv preprint arXiv:1912.02757. — [arXiv:1912.02757](https://arxiv.org/abs/1912.02757) [find](https://scholar.google.com/scholar?q=Stanislav+Fort%2C+Huiyi+Hu%2C+and+Balaji+Lakshminarayanan%2E+2019%2E+Deep+Ensembles%3A+A+Loss+Landscape+Perspective%2E+arXiv+preprint+arXiv%3A1912%2E02757%2E+%E2%80%94+%5BarXiv%3A1912%2E02757%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1912%2E02757%29)
14. Nan Liu and 1 others. 2024. Regmix: Regularizing data mixtures for language model pretraining. arXiv preprint arXiv:2407.10671. [find](https://scholar.google.com/scholar?q=Nan+Liu+and+1+others%2E+2024%2E+Regmix%3A+Regularizing+data+mixtures+for+language+model+pretraining%2E+arXiv+preprint+arXiv%3A2407%2E10671%2E)
15. Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie-Yan Liu. 2019. [Representation Degeneration Problem in Training Natural Language Generation Models](https://md.research.elamur.ai/free/en/papers/gao-2019-representation-degeneration-problem-training-natural/). arXiv preprint arXiv:1907.12009.
16. Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, and Mohammad Rastegari. 2024. OpenELM: An Efficient Language Model Family with Open Training and Inference Framework. arXiv preprint arXiv:240… — [arXiv:2404.14619](https://arxiv.org/abs/2404.14619) [find](https://scholar.google.com/scholar?q=Sachin+Mehta%2C+Mohammad+Hossein+Sekhavat%2C+Qingqing+Cao%2C+Maxwell+Horton%2C+Yanzi+Jin%2C+Chenfan+Sun%2C+Iman+Mirzadeh%2C+Mahyar+Najibi%2C+Dmitry+Belenko%2C+Peter+Zatloukal%2C+and+Mohammad+Rastegari%2E+2024%2E+OpenELM%3A+An+)
17. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2021. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+Shawn+Presser%2C+and+Connor+Leahy%2E+2021%2E+The+pile%3A+An+800gb+datas)
18. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others.… [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Baber+Abbasi%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Alain+Le+Noac%E2%80%99h%2C+Haonan+Li%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Chris+Oci)
19. Yuxian Gu, Li Dong, Hongning Wang, Yaru Hao, Qingxiu Dong, Furu Wei, and Minlie Huang. 2024. Data Selection via Optimal Control for Language Models. arXiv preprint arXiv:2410.07064. — [arXiv:2410.07064](https://arxiv.org/abs/2410.07064) [find](https://scholar.google.com/scholar?q=Yuxian+Gu%2C+Li+Dong%2C+Hongning+Wang%2C+Yaru+Hao%2C+Qingxiu+Dong%2C+Furu+Wei%2C+and+Minlie+Huang%2E+2024%2E+Data+Selection+via+Optimal+Control+for+Language+Models%2E+arXiv+preprint+arXiv%3A2410%2E07064%2E+%E2%80%94+%5BarXiv%3A2410%2E0706)
20. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering.… [find](https://scholar.google.com/scholar?q=Jordan+Hoffmann%2C+Sebastian+Borgeaud%2C+Arthur+Mensch%2C+Elena+Buchatskaya%2C+Trevor+Cai%2C+Eliza+Rutherford%2C+Diego+de+Las+Casas%2C+Lisa+Anne+Hendricks%2C+Todor+Mihaylov%2C+Peter+Clark%2C+Tushar+Khot%2C+and+Ashish+Sabha)
21. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. [The LAMBADA Dataset: Word Prediction Requiring a Broad Discourse Context](https://md.research.elamur.ai/free/en/papers/paperno-2016-lambada-dataset-word-prediction-requiring/). arXiv preprint arXiv:1606.06031.
22. Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. 2022. [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://md.research.elamur.ai/free/en/papers/power-2022-grokking-generalization-overfitting-small-algorithmic/). arXiv preprint arXiv:2201.02177.
23. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog. Version 1, Issue 8. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+Blog%2E+Version+1%2C+Issue+8%2E)
24. Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. 2020. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Shiori+Sagawa%2C+Pang+Wei+Koh%2C+Tatsunori+B+Hashimoto%2C+and+Percy+Liang%2E+2020%2E+Distributionally+robust+neural+networks+for+group+shifts%3A+On+the+importance+of+regularization+for+worst%2Dcase+generalization%2E+)
25. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. WinoGrande: An Adversarial Winograd Schema Challenge at Scale. Communications of the ACM, 64(9):99–106. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2021%2E+WinoGrande%3A+An+Adversarial+Winograd+Schema+Challenge+at+Scale%2E+Communications+of+the+ACM%2C+64%289%29%3A99%E2%80%93106%2E)
26. Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. 2020. [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://md.research.elamur.ai/free/en/papers/sarlin-2019-superglue-learning-feature-matching-graph/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4938–4947.
27. Jiasheng Ye, Peng Liu, Tianxiang Sun, Yichao Zhou, Jian Zhan, and Xipeng Qiu. 2024. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. arXiv preprint arXiv:2403.16952. [find](https://scholar.google.com/scholar?q=Jiasheng+Ye%2C+Peng+Liu%2C+Tianxiang+Sun%2C+Yichao+Zhou%2C+Jian+Zhan%2C+and+Xipeng+Qiu%2E+2024%2E+Data+mixing+laws%3A+Optimizing+data+mixtures+by+predicting+language+modeling+performance%2E+arXiv+preprint+arXiv%3A2403%2E16)
28. Dongkeun Yoon. 2023. SlimPajama-6B. [https://huggingface.co/datasets/DKYoon/](https://huggingface.co/datasets/DKYoon/) SlimPajama-6B. Accessed: 2024-09-24. [↗](https://huggingface.co/datasets/DKYoon/)
29. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a Machine Really Finish Your Sentence? arXiv preprint arXiv:1905.07830. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+HellaSwag%3A+Can+a+Machine+Really+Finish+Your+Sentence%3F+arXiv+preprint+arXiv%3A1905%2E07830%2E)
30. Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R. Steeves, Joel Hestness, and Nolan Dey. 2023. SlimPajama: A 627B Token Cleaned and Deduplicated Version of RedPajama. Dataset available at: [https://huggingface.co/datasets/](https://huggingface.co/datasets/) cerebras/SlimPajama-627B. [↗](https://huggingface.co/datasets/)
31. Debarun Sow, Hannes Woisetschläger, Subhabrata Bu-lusu, Shuyang Wang, Hans-Arno Jacobsen, and Yuhao Liang. 2025. Dynamic loss-based sample reweighting for improved large language model pre-training. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Debarun+Sow%2C+Hannes+Woisetschl%C3%A4ger%2C+Subhabrata+Bu%2Dlusu%2C+Shuyang+Wang%2C+Hans%2DArno+Jacobsen%2C+and+Yuhao+Liang%2E+2025%2E+Dynamic+loss%2Dbased+sample+reweighting+for+improved+large+language+model+pre%2Dtraining%2E+I)
32. Howe Sun and 1 others. 2025. Domain2vec: Vectorizing datasets to find the optimal data mixture without training. In Proceedings of the 42nd International Conference on Machine Learning (ICML). [find](https://scholar.google.com/scholar?q=Howe+Sun+and+1+others%2E+2025%2E+Domain2vec%3A+Vectorizing+datasets+to+find+the+optimal+data+mixture+without+training%2E+In+Proceedings+of+the+42nd+International+Conference+on+Machine+Learning+%28ICML%29%2E)
33. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. [Attention Is All You Need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, pages 5998–6008. Curran Associates, Inc.
34. Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. [Crowdsourcing Multiple Choice Science Questions](https://md.research.elamur.ai/free/en/papers/welbl-2017-crowdsourcing-multiple-choice-science-questions/). arXiv preprint arXiv:1707.06209.
35. Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. 2024. [Qurating: Selecting high-quality data for training language models](https://md.research.elamur.ai/free/en/papers/wettig-2024-qurating-selecting-high-quality-data/). In International Conference on Machine Learning, pages 52915– 52971. PMLR.
36. Sang Michael Xie, Huyen Pham, Xiaowei Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. 2023. Doremi: Optimizing data mixtures speeds up language model pretraining. In NeurIPS. [find](https://scholar.google.com/scholar?q=Sang+Michael+Xie%2C+Huyen+Pham%2C+Xiaowei+Dong%2C+Nan+Du%2C+Hanxiao+Liu%2C+Yifeng+Lu%2C+Percy+Liang%2C+Quoc+V+Le%2C+Tengyu+Ma%2C+and+Adams+Wei+Yu%2E+2023%2E+Doremi%3A+Optimizing+data+mixtures+speeds+up+language+model+pretrai)
37. Yuanjian Xu, Qi An, Jiahuan Zhang, Peng Li, and Zaiqing Nie. 2023. Hard sample aware prompt-tuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12356–12369. [find](https://scholar.google.com/scholar?q=Yuanjian+Xu%2C+Qi+An%2C+Jiahuan+Zhang%2C+Peng+Li%2C+and+Zaiqing+Nie%2E+2023%2E+Hard+sample+aware+prompt%2Dtuning%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1)

*Updated 2 November 2026*