---
title: "LinkQA — paper by Zhang et al. (2026) — summary and ideas"
description: "The authors propose LinkSyn, which works with knowledge points (KPs) instead: the concepts, theorems or methods a… 49 references."
authors: ["Xuemiao Zhang", "Can Ren", "Chengying Tu", "Rongxiang Weng", "Hongfei Yan", "Jingang Wang", "Xunliang Cai"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.1036/"
doi: "10.18653/v1/2026.acl-long.1036"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2026-linkqa-synthesizing-diverse-qa-multiple/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2026-linkqa-synthesizing-diverse-qa-multiple/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/39/) / LinkQA: Synthesizing Diverse QA from Multiple Seeds Strongly Linked by Knowledge Points

# LinkQA: Synthesizing Diverse QA from Multiple Seeds Strongly Linked by Knowledge Points

**Authors:** [Xuemiao Zhang](https://md.research.elamur.ai/free/en/authors/xuemiao-zhang/), [Can Ren](https://md.research.elamur.ai/free/en/authors/can-ren/), [Chengying Tu](https://md.research.elamur.ai/free/en/authors/chengying-tu/), [Rongxiang Weng](https://md.research.elamur.ai/free/en/authors/rongxiang-weng/), [Hongfei Yan](https://md.research.elamur.ai/free/en/authors/hongfei-yan/), [Jingang Wang](https://md.research.elamur.ai/free/en/authors/jingang-wang/) and [Xunliang Cai](https://md.research.elamur.ai/free/en/authors/xunliang-cai/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.1036/) · [PDF](https://aclanthology.org/2026.acl-long.1036.pdf) · [DOI 10.18653/v1/2026.acl-long.1036](https://doi.org/10.18653/v1/2026.acl-long.1036) · [OpenAlex](https://openalex.org/W7166860574)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Training LLMs is increasingly limited by a shortage of diverse, high-quality data, especially across many academic disciplines. Earlier synthesis methods either rewrite one seed at a time, which limits diversity, or connect documents through shared entities, which are often only incidental mentions. They also give weak control over difficulty and subject mix. The authors propose LinkSyn, which works with knowledge points (KPs) instead: the concepts, theorems or methods a question actually tests. KPs are extracted from open-source QA seeds, and a weighted co-occurrence graph is built from them. Random walks on this graph combine two policies: one follows strong co-occurrence links, the other spreads uniformly to reach long-tail knowledge. A knowledge value function balances the two. Seeds along each path are selected to match target difficulty and discipline. DeepSeek-R1 then merges them into new QA pairs. The resulting 50B-token corpus, LinkQA, was tested by continually pre-training Llama-3 8B, where it produced state-of-the-art averages. This shows that concept-level linking produces more useful and controllable synthetic data.

## Key points
- **KP extraction and graph:**
  - Qwen2.5-14B-Instruct, fine-tuned on DeepSeek-R1 labels, served as the KP extractor (80.8% agreement with R1; 97.5% expert acceptability).
  - The graph has about 10M KPs, 153M edges and 21,806 Leiden communities.
  - About 75% of seed questions involve more than one KP.
- **Path design:** Paths contain 1–3 KPs, which controls question complexity.
- **Main result:** Continual pre-training of Llama-3 8B (2T-token checkpoint) on 20B tokens of LinkQA raised the MMLU/CMMLU average by 11.51% over the baseline. Gains held across 1.7B, 8B and 16B models and at 2T and 10T initial training scales.
- **Difficulty:** LinkQA has roughly ten times more hard items (H4/H5) than Nemotron-CC. Using 2 or 3 seeds yields about 10% more hard questions than single-seed synthesis.
- **Quality checks:**
  - Manual review found 98% correctness for H1/H2, 94% for H3 and 87% for H4/H5.
  - A DeepSeek-V3 refinement pass corrected about 10% of items and discarded about 6%.
  - Code and data are available on GitHub (rc314159-creator/linkQA).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 315 works · 43,267 citations
- [Meituan](https://md.research.elamur.ai/free/en/companies/meituan/) — Company, Beijing, China · 26 works · 3,953 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (1): [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 487 works

## Cite

Cite as: Xuemiao Zhang, Can Ren, Chengying Tu, Rongxiang Weng, Hongfei Yan, Jingang Wang and Xunliang Cai. 2026. LinkQA: Synthesizing Diverse QA from Multiple Seeds Strongly Linked by Knowledge Points. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhang2026linkqa,
  title = {{LinkQA: Synthesizing Diverse QA from Multiple Seeds Strongly Linked by Knowledge Points}},
  author = {Xuemiao Zhang, Can Ren, Chengying Tu, Rongxiang Weng, Hongfei Yan, Jingang Wang and Xunliang Cai},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1036},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1036}
}
```

- [← previous Linking Knowledge to Care: Knowledge Graph-Augmented Medical Follow-Up Question Generation](https://md.research.elamur.ai/free/en/papers/sun-2026-linking-knowledge-care-knowledge-graph/)
- [next → Lispector: fine-tuning language models for grammar and spelling correction in Brazilian Portuguese](https://md.research.elamur.ai/free/en/papers/medeiros-2026-lispector-fine-tuning-de-modelos/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/39/)

## References

49 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Syeda Nahida Akter, Shrimai Prabhumoye, John Kamalu, Sanjeev Satheesh, Eric Nyberg, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2025. MIND: Math informed synthetic dialogues for pretraining LLMs. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Syeda+Nahida+Akter%2C+Shrimai+Prabhumoye%2C+John+Kamalu%2C+Sanjeev+Satheesh%2C+Eric+Nyberg%2C+Mostofa+Patwary%2C+Mohammad+Shoeybi%2C+and+Bryan+Catanzaro%2E+2025%2E+MIND%3A+Math+informed+synthetic+dialogues+for+pretrainin)
2. Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. [MathQA: Towards interpretable math word problem solving with operation-based formalisms](https://md.research.elamur.ai/free/en/papers/amini-2019-mathqa-interpretable-math-word-problem/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computation…
3. Yaoyao Chang, Lei Cui, Li Dong, Shaohan Huang, Yangyu Huang, Yupan Huang, Scarlett Li, Tengchao Lv, Shuming Ma, Qinzheng Sun, Wenhui Wang, Furu Wei, Ying Xin, Mao Yang, Qiufeng Yin, and Xingxing Zhang. 2024. Redstone: Curating general, code, math, and qa data for large language models. Preprint, arX… — [arXiv:2412.03398](https://arxiv.org/abs/2412.03398) [find](https://scholar.google.com/scholar?q=Yaoyao+Chang%2C+Lei+Cui%2C+Li+Dong%2C+Shaohan+Huang%2C+Yangyu+Huang%2C+Yupan+Huang%2C+Scarlett+Li%2C+Tengchao+Lv%2C+Shuming+Ma%2C+Qinzheng+Sun%2C+Wenhui+Wang%2C+Furu+Wei%2C+Ying+Xin%2C+Mao+Yang%2C+Qiufeng+Yin%2C+and+Xingxing+Zhang)
4. Jie Chen, Zhipeng Chen, Jiapeng Wang, Kun Zhou, Yutao Zhu, Jinhao Jiang, Yingqian Min, Xin Zhao, Zhicheng Dou, Jiaxin Mao, Yankai Lin, Ruihua Song, Jun Xu, Xu Chen, Rui Yan, Zhewei Wei, Di Hu, Wenbing Huang, and Ji-Rong Wen. 2025. [Towards effective and efficient continual pre-training of large langu…](https://md.research.elamur.ai/free/en/papers/chen-2025-effective-efficient-continual-pre-training/)
5. Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, and Furu Wei. 2024. [Instruction pre-training: Language models are supervised multitask learners](https://md.research.elamur.ai/free/en/papers/cheng-2024-instruction-pre-training-language-models/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2529–2550.
6. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? try arc, the AI2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). CoRR, abs/1803.05457.
7. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 81 others. 2025. [Deepseek-r1: Incentivizing reasoning capability in llm…](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/)
8. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 81 others. 2024. Deepseek-v3 technical report. CoRR, abs/2412.194… [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2C+Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+Bingxuan+Wang%2C+Bochao+Wu%2C+Chengda+Lu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chenyu+Zhang%2C+Chong+Ruan%2C+Damai+Dai%2C+Daya+Guo%2C+Dejian+Yang%2C+Deli+Chen%2C+Dongjie+Ji%2C+Erhan)
9. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. [DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistic…
10. Feiyu Duan, Xuemiao Zhang, Sirui Wang, Haoran Que, Yuqi Liu, Wenge Rong, and Xunliang Cai. 2025. Enhancing llms via high-knowledge data selection. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 23832–23840. [find](https://scholar.google.com/scholar?q=Feiyu+Duan%2C+Xuemiao+Zhang%2C+Sirui+Wang%2C+Haoran+Que%2C+Yuqi+Liu%2C+Wenge+Rong%2C+and+Xunliang+Cai%2E+2025%2E+Enhancing+llms+via+high%2Dknowledge+data+selection%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+I)
11. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, and 82 others. 2024… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
12. Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi… [find](https://scholar.google.com/scholar?q=Suriya+Gunasekar%2C+Yi+Zhang%2C+Jyoti+Aneja%2C+Caio+C%C3%A9sar+Teodoro+Mendes%2C+Allie+Del+Giorno%2C+Sivakanth+Gopi%2C+Mojan+Javaheripi%2C+Piero+Kauffmann%2C+Gustavo+de+Rosa%2C+Olli+Saarikivi%2C+Adil+Salim%2C+Shital+Shah%2C+Harki)
13. Yiduo Guo, Jie Fu, Huishuai Zhang, Dongyan Zhao, and Yikang Shen. 2024. Efficient continual pre-training by mitigating the stability gap. Preprint, arXiv:2406.14833. — [arXiv:2406.14833](https://arxiv.org/abs/2406.14833) [find](https://scholar.google.com/scholar?q=Yiduo+Guo%2C+Jie+Fu%2C+Huishuai+Zhang%2C+Dongyan+Zhao%2C+and+Yikang+Shen%2E+2024%2E+Efficient+continual+pre%2Dtraining+by+mitigating+the+stability+gap%2E+Preprint%2C+arXiv%3A2406%2E14833%2E+%E2%80%94+%5BarXiv%3A2406%2E14833%5D%28https%3A%2F%2Farxiv)
14. Xuhui Jiang, Shengjie Ma, Chengjin Xu, Cehao Yang, Liyu Zhang, and Jian Guo. 2025b. Synthesize-on-graph: Knowledgeable synthetic data generation for continue pre-training of large language models. Preprint, arXiv:2505.00979. — [arXiv:2505.00979](https://arxiv.org/abs/2505.00979) [find](https://scholar.google.com/scholar?q=Xuhui+Jiang%2C+Shengjie+Ma%2C+Chengjin+Xu%2C+Cehao+Yang%2C+Liyu+Zhang%2C+and+Jian+Guo%2E+2025b%2E+Synthesize%2Don%2Dgraph%3A+Knowledgeable+synthetic+data+generation+for+continue+pre%2Dtraining+of+large+language+models%2E+Pre)
15. Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. [Large language models struggle to learn long-tail knowledge](https://md.research.elamur.ai/free/en/papers/kandpal-2022-large-language-models-struggle-learn/). Preprint, arXiv:2211.08411.
16. Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2024. [CMMLU: Measuring massive multitask language understanding in Chinese](https://md.research.elamur.ai/free/en/papers/li-2023-cmmlu-measuring-massive-multitask-language/). In Findings of the Association for Computational Linguistics: ACL 2024, pages 11260–11285.
17. Fei Hao, Yanqi Gong, Wangyang Yu, and Vincenzo Loia. 2022. Knowledge points navigation based on three-way concept lattice for autonomous learning. Pattern Recognition Letters, 163:96–103. [find](https://scholar.google.com/scholar?q=Fei+Hao%2C+Yanqi+Gong%2C+Wangyang+Yu%2C+and+Vincenzo+Loia%2E+2022%2E+Knowledge+points+navigation+based+on+three%2Dway+concept+lattice+for+autonomous+learning%2E+Pattern+Recognition+Letters%2C+163%3A96%E2%80%93103%2E)
18. Peng Li, Wei Li, Zhengyan He, Xuguang Wang, Ying Cao, Jie Zhou, and Wei Xu. 2016. Dataset and neural recurrent sequence labeling model for open-domain factoid question answering. Preprint, arXiv:1607.06275. — [arXiv:1607.06275](https://arxiv.org/abs/1607.06275) [find](https://scholar.google.com/scholar?q=Peng+Li%2C+Wei+Li%2C+Zhengyan+He%2C+Xuguang+Wang%2C+Ying+Cao%2C+Jie+Zhou%2C+and+Wei+Xu%2E+2016%2E+Dataset+and+neural+recurrent+sequence+labeling+model+for+open%2Ddomain+factoid+question+answering%2E+Preprint%2C+arXiv%3A1607%2E)
19. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In International Conference on Learning Representations.
20. Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023. [Textbooks are all you need ii: phi-1.5 technical report](https://md.research.elamur.ai/free/en/papers/li-2023-textbooks-all-need-ii-phi/). Preprint, arXiv:2309.05463.
21. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021b. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In International Conference on Learning Representations.
22. Binyuan Huang, Yuqing Wen, Yucheng Zhao, Yaosi Hu, Yingfei Liu, Fan Jia, Weixin Mao, Tiancai Wang, Chi Zhang, Chang Wen Chen, Zhenzhong Chen, and Xiangyu Zhang. 2024. Subjectdrive: Scaling generative data in autonomous driving via subject control. Preprint, arXiv:2403.19438. — [arXiv:2403.19438](https://arxiv.org/abs/2403.19438) [find](https://scholar.google.com/scholar?q=Binyuan+Huang%2C+Yuqing+Wen%2C+Yucheng+Zhao%2C+Yaosi+Hu%2C+Yingfei+Liu%2C+Fan+Jia%2C+Weixin+Mao%2C+Tiancai+Wang%2C+Chi+Zhang%2C+Chang+Wen+Chen%2C+Zhenzhong+Chen%2C+and+Xiangyu+Zhang%2E+2024%2E+Subjectdrive%3A+Scaling+generative+)
23. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. [A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions](https://md.research.elamur.ai/free/en/papers/huang-2023-survey-hallucination-large-language-models/). ACM Transactions on Information Systems,…
24. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, jiayi lei, Yao Fu, Maosong Sun, and Junxian He. 2023. [C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models](https://md.research.elamur.ai/free/en/papers/huang-2023-c-eval-multi-level-multi/). In Thirty-seventh Conference on Neural…
25. Jinhao Jiang, Junyi Li, Xin Zhao, Yang Song, Tao Zhang, and Ji-Rong Wen. 2025a. Mix-CPT: A domain adaptation framework via decoupling knowledge learning and format alignment. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Jinhao+Jiang%2C+Junyi+Li%2C+Xin+Zhao%2C+Yang+Song%2C+Tao+Zhang%2C+and+Ji%2DRong+Wen%2E+2025a%2E+Mix%2DCPT%3A+A+domain+adaptation+framework+via+decoupling+knowledge+learning+and+format+alignment%2E+In+The+Thirteenth+Interna)
26. Pratyush Maini, Skyler Seto, Richard Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly. 2024. [Rephrasing the web: A recipe for compute and data-efficient language modeling](https://md.research.elamur.ai/free/en/papers/maini-2024-rephrasing-web-recipe-compute-data/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14…
27. Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. 2025. Scaling data-constrained language models. Preprint, arXiv:2305.16264. [find](https://scholar.google.com/scholar?q=Niklas+Muennighoff%2C+Alexander+M%2E+Rush%2C+Boaz+Barak%2C+Teven+Le+Scao%2C+Aleksandra+Piktus%2C+Nouamane+Tazi%2C+Sampo+Pyysalo%2C+Thomas+Wolf%2C+and+Colin+Raffel%2E+2025%2E+Scaling+data%2Dconstrained+language+models%2E+Prepri)
28. Mihai Nadăs , , Laura Dios , an, and Andreea Tomescu. 2025. Synthetic data generation using large language models: Advances in text and code. IEEE Access, page 1–1. [find](https://scholar.google.com/scholar?q=Mihai+Nada%CC%86s+%2C+%2C+Laura+Dios+%2C+an%2C+and+Andreea+Tomescu%2E+2025%2E+Synthetic+data+generation+using+large+language+models%3A+Advances+in+text+and+code%2E+IEEE+Access%2C+page+1%E2%80%931%2E)
29. Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. 2021. Efficient large-scale language model training on gpu clusters using megatron-lm.… [find](https://scholar.google.com/scholar?q=Deepak+Narayanan%2C+Mohammad+Shoeybi%2C+Jared+Casper%2C+Patrick+LeGresley%2C+Mostofa+Patwary%2C+Vijay+Korthikanti%2C+Dmitri+Vainbrand%2C+Prethvi+Kashinkunti%2C+Julie+Bernauer%2C+Bryan+Catanzaro%2C+Amar+Phanishayee%2C+and+M)
30. Jianmo Ni, Gustavo Hernández Ábrego, Noah Constant, Ji Ma, Keith B. Hall, Daniel Cer, and Yinfei Yang. 2021. [Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models](https://md.research.elamur.ai/free/en/papers/ni-2021-sentence-t5-scalable-sentence-encoders/). arXiv preprint arXiv:2108.08877.
31. Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024. [The fineweb datasets: Decanting the web for the finest text data at scale](https://md.research.elamur.ai/free/en/papers/penedo-2024-fineweb-datasets-decanting-web-finest/). In The Thirty-eight Conference on Neural Information Processing Systems Datasets a…
32. Zeyu Qin, Qingxiu Dong, Xingxing Zhang, Li Dong, Xiaolong Huang, Ziyi Yang, Mahmoud Khademi, Dongdong Zhang, Hany Hassan Awadalla, Yi R. Fung, Weizhu Chen, Minhao Cheng, and Furu Wei. 2025. Scaling laws of synthetic data for language models. Preprint, arXiv:2503.19551. — [arXiv:2503.19551](https://arxiv.org/abs/2503.19551) [find](https://scholar.google.com/scholar?q=Zeyu+Qin%2C+Qingxiu+Dong%2C+Xingxing+Zhang%2C+Li+Dong%2C+Xiaolong+Huang%2C+Ziyi+Yang%2C+Mahmoud+Khademi%2C+Dongdong+Zhang%2C+Hany+Hassan+Awadalla%2C+Yi+R%2E+Fung%2C+Weizhu+Chen%2C+Minhao+Cheng%2C+and+Furu+Wei%2E+2025%2E+Scaling+la)
33. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. [find](https://scholar.google.com/scholar?q=Qwen%2C+%3A%2C+An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+Huan+Lin%2C+Jian+Yang%2C+Jianhong+Tu%2C+Jianwei+Zhang%2C+Jianxin+Yang%2C+Jiaxi+)
34. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2021%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+Communications+of+the+ACM%2C+64%289%29%3A99%E2%80%93106%2E)
35. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). Preprint, arXiv:2402.03300.
36. Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2025. [Nemotron-CC: Transforming Common Crawl into a refined long-horizon pretraining dataset](https://md.research.elamur.ai/free/en/papers/su-2025-nemotron-cc-transforming-common-crawl/). In Proceedings of the 63rd Annual Meeting of the Association for Computa…
37. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. 2023. [Challenging BIG-bench tasks and whether chain-of-thought can solve them](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/). In Findings of the Association for Computational Linguistics: A…
38. Meituan LongCat Team, Bayan, Bei Li, Bingye Lei, Bo Wang, Bolin Rong, Chao Wang, Chao Zhang, Chen Gao, Chen Zhang, Cheng Sun, Chengcheng Han, Chenguang Xi, Chi Zhang, Chong Peng, Chuan Qin, Chuyu Zhang, Cong Chen, Congkui Wang, and 163 others. 2025a. Longcat-flash technical report. Preprint, arXiv:2… — [arXiv:2509.01322](https://arxiv.org/abs/2509.01322) [find](https://scholar.google.com/scholar?q=Meituan+LongCat+Team%2C+Bayan%2C+Bei+Li%2C+Bingye+Lei%2C+Bo+Wang%2C+Bolin+Rong%2C+Chao+Wang%2C+Chao+Zhang%2C+Chen+Gao%2C+Chen+Zhang%2C+Cheng+Sun%2C+Chengcheng+Han%2C+Chenguang+Xi%2C+Chi+Zhang%2C+Chong+Peng%2C+Chuan+Qin%2C+Chuyu+Zhan)
39. Meituan LongCat Team, Anchun Gui, Bei Li, Bingyang Tao, Bole Zhou, Borun Chen, Chao Zhang, Chao Zhang, Chengcheng Han, Chenhui Yang, Chi Zhang, Chong Peng, Chuyu Zhang, Cong Chen, Fengcun Li, Gang Xu, Guoyuan Lin, Hao Jiang, Hao Liang, and 108 others. 2025b. Introducing longcat-flash-thinking: A tec… — [arXiv:2509.18883](https://arxiv.org/abs/2509.18883) [find](https://scholar.google.com/scholar?q=Meituan+LongCat+Team%2C+Anchun+Gui%2C+Bei+Li%2C+Bingyang+Tao%2C+Bole+Zhou%2C+Borun+Chen%2C+Chao+Zhang%2C+Chao+Zhang%2C+Chengcheng+Han%2C+Chenhui+Yang%2C+Chi+Zhang%2C+Chong+Peng%2C+Chuyu+Zhang%2C+Cong+Chen%2C+Fengcun+Li%2C+Gang+Xu%2C)
40. Yuxuan Tong, Xiwen Zhang, Rui Wang, Ruidong Wu, and Junxian He. 2024. [DART-math: Difficulty-aware rejection tuning for mathematical problem-solving](https://md.research.elamur.ai/free/en/papers/tong-2024-dart-math-difficulty-aware-rejection/). In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
41. Chengying Tu, Xuemiao Zhang, Rongxiang Weng, Rumei Li, Chen Zhang, Yang Bai, Hongfei Yan, Jingang Wang, and Xunliang Cai. 2025. A survey on llm mid-training. Preprint, arXiv:2510.23081. — [arXiv:2510.23081](https://arxiv.org/abs/2510.23081) [find](https://scholar.google.com/scholar?q=Chengying+Tu%2C+Xuemiao+Zhang%2C+Rongxiang+Weng%2C+Rumei+Li%2C+Chen+Zhang%2C+Yang+Bai%2C+Hongfei+Yan%2C+Jingang+Wang%2C+and+Xunliang+Cai%2E+2025%2E+A+survey+on+llm+mid%2Dtraining%2E+Preprint%2C+arXiv%3A2510%2E23081%2E+%E2%80%94+%5BarXiv%3A2510%2E)
42. Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. 2024. [Will we run out of data? limits of llm scaling based on human-generated data](https://md.research.elamur.ai/free/en/papers/villalobos-2022-will-run-out-data-limits/). Preprint, arXiv:2211.04325.
43. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024. [MMLU-pro: A more robust and challenging multi-task language understanding benchmark](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/)…
44. Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. 2025. Octothinker: Mid-training incentivizes reinforcement learning scaling. Preprint, arXiv:2506.20512. — [arXiv:2506.20512](https://arxiv.org/abs/2506.20512) [find](https://scholar.google.com/scholar?q=Zengzhi+Wang%2C+Fan+Zhou%2C+Xuefeng+Li%2C+and+Pengfei+Liu%2E+2025%2E+Octothinker%3A+Mid%2Dtraining+incentivizes+reinforcement+learning+scaling%2E+Preprint%2C+arXiv%3A2506%2E20512%2E+%E2%80%94+%5BarXiv%3A2506%2E20512%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs)
45. Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. 2024. [Qurating: Selecting high-quality data for training language models](https://md.research.elamur.ai/free/en/papers/wettig-2024-qurating-selecting-high-quality-data/). In International Conference on Machine Learning, pages 52915– 52971.
46. Zitong Yang, Neil Band, Shuangping Li, Emmanuel Candes, and Tatsunori Hashimoto. 2025. Synthetic continued pretraining. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zitong+Yang%2C+Neil+Band%2C+Shuangping+Li%2C+Emmanuel+Candes%2C+and+Tatsunori+Hashimoto%2E+2025%2E+Synthetic+continued+pretraining%2E+In+The+Thirteenth+International+Conference+on+Learning+Representations%2E)
47. Xuemiao Zhang, Can Ren, Chengying Tu, Rongxiang Weng, Shuo Wang, Hongfei Yan, Jingang Wang, and Xunliang Cai. 2026. Expanding reasoning potential in foundation model by learning diverse chains of thought patterns. Preprint, arXiv:2509.21124. — [arXiv:2509.21124](https://arxiv.org/abs/2509.21124) [find](https://scholar.google.com/scholar?q=Xuemiao+Zhang%2C+Can+Ren%2C+Chengying+Tu%2C+Rongxiang+Weng%2C+Shuo+Wang%2C+Hongfei+Yan%2C+Jingang+Wang%2C+and+Xunliang+Cai%2E+2026%2E+Expanding+reasoning+potential+in+foundation+model+by+learning+diverse+chains+of+thou)
48. Fan Zhou, Zengzhi Wang, Nikhil Ranjan, Zhoujun Cheng, Liping Tang, Guowei He, Zhengzhong Liu, and Eric P. Xing. 2025. Megamath: Pushing the limits of open math corpora. Preprint, arXiv:2504.02807. [find](https://scholar.google.com/scholar?q=Fan+Zhou%2C+Zengzhi+Wang%2C+Nikhil+Ranjan%2C+Zhoujun+Cheng%2C+Liping+Tang%2C+Guowei+He%2C+Zhengzhong+Liu%2C+and+Eric+P%2E+Xing%2E+2025%2E+Megamath%3A+Pushing+the+limits+of+open+math+corpora%2E+Preprint%2C+arXiv%3A2504%2E02807%2E)
49. Kun Zhou, Beichen Zhang, jiapeng wang, Zhipeng Chen, Xin Zhao, Jing Sha, Zhichao Sheng, Shijin Wang, and Ji-Rong Wen. 2024. Jiuzhang3.0: Efficiently improving mathematical reasoning by training small data synthesis models. In The Thirty-eighth Annual Conference on Neural Information Processing Syste… [find](https://scholar.google.com/scholar?q=Kun+Zhou%2C+Beichen+Zhang%2C+jiapeng+wang%2C+Zhipeng+Chen%2C+Xin+Zhao%2C+Jing+Sha%2C+Zhichao+Sheng%2C+Shijin+Wang%2C+and+Ji%2DRong+Wen%2E+2024%2E+Jiuzhang3%2E0%3A+Efficiently+improving+mathematical+reasoning+by+training+small+)

*Updated 2 November 2026*