---
title: "Learning from Diverse Reasoning Paths with Routing and"
description: "The authors introduce QR-Distill. 97 references, cited by 10 papers here. Among them: 1."
authors: ["Zhenyu Lei", "Zhen Tan", "Song Wang", "Yaochen Zhu", "Zihan Chen", "Yushun Dong", "Jundong Li"]
year: 2025
venue: "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2025.emnlp-main.141/"
doi: "10.18653/v1/2025.emnlp-main.141"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lei-2025-learning-diverse-reasoning-paths-routing/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lei-2025-learning-diverse-reasoning-paths-routing/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/8/) / Learning from Diverse Reasoning Paths with Routing and Collaboration

# Learning from Diverse Reasoning Paths with Routing and Collaboration

**Authors:** Zhenyu Lei, [Zhen Tan](https://md.research.elamur.ai/free/en/authors/zhen-tan/), [Song Wang](https://md.research.elamur.ai/free/en/authors/song-wang/), Yaochen Zhu, [Zihan Chen](https://md.research.elamur.ai/free/en/authors/zihan-chen/), [Yushun Dong](https://md.research.elamur.ai/free/en/authors/yushun-dong/) and [Jundong Li](https://md.research.elamur.ai/free/en/authors/jundong-li/) · **Year:** 2025 · **Venue:** [Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2025](https://md.research.elamur.ai/free/en/venues/emnlp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.emnlp-main.141/) · [PDF](https://aclanthology.org/2025.emnlp-main.141.pdf) · [DOI 10.18653/v1/2025.emnlp-main.141](https://doi.org/10.18653/v1/2025.emnlp-main.141) · [OpenAlex](https://openalex.org/W4416036899)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [10](https://md.research.elamur.ai/free/en/papers/lei-2025-learning-diverse-reasoning-paths-routing/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Compact student models trained by black-box distillation usually see only one teacher rationale per question, which captures little of the teacher's reasoning repertoire. Using many rationales helps, but they differ in correctness and in how useful they are for a given task or student. The authors introduce QR-Distill. It first builds a pool of diverse reasoning paths by prompting a teacher (Gemini-1.5) with six style templates. It then discards paths with wrong final answers and those an LLM judge flags as containing spurious steps. A trainable router, fed encoder representations of each path, uses Gumbel-Softmax to decide which students receive which paths, with an entropy term preventing degenerate assignments. Finally, students co-train as peer teachers: their projected hidden states are combined into a competence-weighted ensemble that each student matches via an MSE loss. The authors report gains over single-path and unrouted multi-path distillation, and ablations indicate every component matters. The work is relevant for anyone building small, transparent reasoning models from proprietary LLM outputs.

## Key points
- Six prompting styles generate the path pool: Vanilla, Chain-of-Thought, Tree-of-Thought, Program-Based, Backward, and Fact-Retrieval reasoning. The teacher is accessed only through outputs, not logits.
- Filtering has two stages. Paths whose final answer differs from the gold label are removed, and an LLM-as-a-judge then drops traces with hallucinated or spurious intermediate steps.
- Routing works as follows: an MLP over path embeddings produces per-student logits, and Gumbel-Softmax yields discrete but differentiable path-to-student assignments. An entropy regularizer on the average assignment rate balances usage across students.
- Mutual-Student Distillation is feature-level. Students' last hidden states are projected into a shared space and weighted by competence scores into an ensemble, which serves as a provisional teacher. The full training loss combines vanilla distillation, router entropy, and mutual losses.
- Setup and limitations: only two student models are used, and there is a single teacher (Gemini-1.5) with a fixed template set. Specific benchmark numbers fall in the omitted part of the extracted text. Code is available at github.com/LzyFischer/Distill, and the paper appears in EMNLP 2025, pp. 2832–2845.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Virginia](https://md.research.elamur.ai/free/en/universities/university-of-virginia/) — University, Charlottesville, United States · 36 works · 9,730 citations
- [Arizona State University](https://md.research.elamur.ai/free/en/universities/arizona-state-university/) — University, Tempe, United States · 50 works · 2,032 citations
- [Florida State University](https://md.research.elamur.ai/free/en/universities/florida-state-university/) — University, Tallahassee, United States · 8 works · 363 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works

## Cite

Cite as: Zhenyu Lei, Zhen Tan, Song Wang, Yaochen Zhu, Zihan Chen, Yushun Dong and Jundong Li. 2025. Learning from Diverse Reasoning Paths with Routing and Collaboration. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{lei2025learning,
  title = {{Learning from Diverse Reasoning Paths with Routing and Collaboration}},
  author = {Zhenyu Lei, Zhen Tan, Song Wang, Yaochen Zhu, Zihan Chen, Yushun Dong and Jundong Li},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.141},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.141}
}
```

- [← previous Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignment](https://md.research.elamur.ai/free/en/papers/li-2025-layer-aware-representation-filtering-purifying/)
- [next → LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarios](https://md.research.elamur.ai/free/en/papers/wu-2025-lifbench-evaluating-instruction-following-performance/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/8/)

## References

97 references, 37 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Kamal Acharya, Alvaro Velasquez, and Houbing Herbert Song. 2024. A survey on symbolic knowledge distillation of large language models. IEEE Transactions on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Kamal+Acharya%2C+Alvaro+Velasquez%2C+and+Houbing+Herbert+Song%2E+2024%2E+A+survey+on+symbolic+knowledge+distillation+of+large+language+models%2E+IEEE+Transactions+on+Artificial+Intelligence%2E)
2. Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming {Throughput-Latency} tradeoff in {LLM} inference with {Sarathi-Serve}. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), p… [find](https://scholar.google.com/scholar?q=Amey+Agrawal%2C+Nitin+Kedia%2C+Ashish+Panwar%2C+Jayashree+Mohan%2C+Nipun+Kwatra%2C+Bhargav+Gulavani%2C+Alexey+Tumanov%2C+and+Ramachandran+Ramjee%2E+2024%2E+Taming+%7BThroughput%2DLatency%7D+tradeoff+in+%7BLLM%7D+inference+with+%7B)
3. Toufique Ahmed and Premkumar Devanbu. 2023. Better patching using llm prompting, via self-consistency. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 1742–1746. IEEE. [find](https://scholar.google.com/scholar?q=Toufique+Ahmed+and+Premkumar+Devanbu%2E+2023%2E+Better+patching+using+llm+prompting%2C+via+self%2Dconsistency%2E+In+2023+38th+IEEE%2FACM+International+Conference+on+Automated+Software+Engineering+%28ASE%29%2C+pages+174)
4. Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. 2024. [Large language models for mathematical reasoning: Progresses and challenges](https://md.research.elamur.ai/free/en/papers/ahn-2024-large-language-models-mathematical-reasoning/). arXiv preprint arXiv:2402.00157.
5. Zhenni Bi, Kai Han, Chuanjian Liu, Yehui Tang, and Yunhe Wang. 2024. [Forest-of-thought: Scaling test-time compute for enhancing llm reasoning](https://md.research.elamur.ai/free/en/papers/bi-2024-forest-thought-scaling-test-time/). arXiv preprint arXiv:2412.09078.
6. Angelica Chen, Jason Phang, Alicia Parrish, Vishakh Padmakumar, Chen Zhao, Samuel R Bowman, and Kyunghyun Cho. 2023a. [Two failures of self-consistency in the multi-step reasoning of llms](https://md.research.elamur.ai/free/en/papers/chen-2023-two-failures-self-consistency-multi/). arXiv preprint arXiv:2305.14279.
7. Hongzhan Chen, Siyue Wu, Xiaojun Quan, Rui Wang, Ming Yan, and Ji Zhang. 2023b. Mcc-kd: Multi-cot consistent knowledge distillation. arXiv preprint arXiv:2310.14747. [find](https://scholar.google.com/scholar?q=Hongzhan+Chen%2C+Siyue+Wu%2C+Xiaojun+Quan%2C+Rui+Wang%2C+Ming+Yan%2C+and+Ji+Zhang%2E+2023b%2E+Mcc%2Dkd%3A+Multi%2Dcot+consistent+knowledge+distillation%2E+arXiv+preprint+arXiv%3A2310%2E14747%2E)
8. Justin Chih-Yao Chen, Zifeng Wang, Hamid Palangi, Rujun Han, Sayna Ebrahimi, Long Le, Vincent Perot, Swaroop Mishra, Mohit Bansal, Chen-Yu Lee, and 1 others. 2024a. Reverse thinking makes llms stronger reasoners. arXiv preprint arXiv:2411.19865. [find](https://scholar.google.com/scholar?q=Justin+Chih%2DYao+Chen%2C+Zifeng+Wang%2C+Hamid+Palangi%2C+Rujun+Han%2C+Sayna+Ebrahimi%2C+Long+Le%2C+Vincent+Perot%2C+Swaroop+Mishra%2C+Mohit+Bansal%2C+Chen%2DYu+Lee%2C+and+1+others%2E+2024a%2E+Reverse+thinking+makes+llms+stronge)
9. Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. 2021. Distilling knowledge via knowledge review. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5008–5017. [find](https://scholar.google.com/scholar?q=Pengguang+Chen%2C+Shu+Liu%2C+Hengshuang+Zhao%2C+and+Jiaya+Jia%2E+2021%2E+Distilling+knowledge+via+knowledge+review%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+computer+vision+and+pattern+recognition%2C+pages+50)
10. Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, and 1 others. 2023c. [Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors](https://md.research.elamur.ai/free/en/papers/chen-2023-agentverse-facilitating-multi-agent/) in agents. arXiv preprint arXiv:2308.10848, 2(4):6.
11. Weize Chen, Jiarui Yuan, Chen Qian, Cheng Yang, Zhiyuan Liu, and Maosong Sun. 2024b. Optima: Optimizing effectiveness and efficiency for llm-based multi-agent system. arXiv preprint arXiv:2410.08115. [find](https://scholar.google.com/scholar?q=Weize+Chen%2C+Jiarui+Yuan%2C+Chen+Qian%2C+Cheng+Yang%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2024b%2E+Optima%3A+Optimizing+effectiveness+and+efficiency+for+llm%2Dbased+multi%2Dagent+system%2E+arXiv+preprint+arXiv%3A2410%2E08115%2E)
12. Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, and 1 others. 2024c. [Do not think that much for 2+ 3=?](https://md.research.elamur.ai/free/en/papers/chen-2024-think-that-much-2-3/) on the overthinking of o1-like llms. arXiv preprint arXiv:2412.21187.
13. Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. 2023d. [Universal self-consistency for large language model generation](https://md.research.elamur.ai/free/en/papers/chen-2023-universal-self-consistency-large-language/). arXiv preprint arXiv:2311.17311.
14. Zihan Chen, Song Wang, Zhen Tan, Xingbo Fu, Zhenyu Lei, Peng Wang, Huan Liu, Cong Shen, and Jundong Li. 2025a. A survey of scaling in large language model reasoning. arXiv preprint arXiv:2504.02181. [find](https://scholar.google.com/scholar?q=Zihan+Chen%2C+Song+Wang%2C+Zhen+Tan%2C+Xingbo+Fu%2C+Zhenyu+Lei%2C+Peng+Wang%2C+Huan+Liu%2C+Cong+Shen%2C+and+Jundong+Li%2E+2025a%2E+A+survey+of+scaling+in+large+language+model+reasoning%2E+arXiv+preprint+arXiv%3A2504%2E02181%2E)
15. Zihan Chen, Song Wang, Zhen Tan, Jundong Li, and Cong Shen. 2025b. Maple: Many-shot adaptive pseudo-labeling for in-context learning. arXiv preprint arXiv:2505.16225. — [arXiv:2505.16225](https://arxiv.org/abs/2505.16225) [find](https://scholar.google.com/scholar?q=Zihan+Chen%2C+Song+Wang%2C+Zhen+Tan%2C+Jundong+Li%2C+and+Cong+Shen%2E+2025b%2E+Maple%3A+Many%2Dshot+adaptive+pseudo%2Dlabeling+for+in%2Dcontext+learning%2E+arXiv+preprint+arXiv%3A2505%2E16225%2E+%E2%80%94+%5BarXiv%3A2505%2E16225%5D%28https%3A%2F%2Farxi)
16. Jang Hyun Cho and Bharath Hariharan. 2019. On the efficacy of knowledge distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4794–4802. [find](https://scholar.google.com/scholar?q=Jang+Hyun+Cho+and+Bharath+Hariharan%2E+2019%2E+On+the+efficacy+of+knowledge+distillation%2E+In+Proceedings+of+the+IEEE%2FCVF+international+conference+on+computer+vision%2C+pages+4794%E2%80%934802%2E)
17. Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. 2023. Navigate through enigmatic labyrinth a survey of chain of thought reasoning: Advances, frontiers and future. arXiv preprint arXiv:2309.15402. [find](https://scholar.google.com/scholar?q=Zheng+Chu%2C+Jingchang+Chen%2C+Qianglong+Chen%2C+Weijiang+Yu%2C+Tao+He%2C+Haotian+Wang%2C+Weihua+Peng%2C+Ming+Liu%2C+Bing+Qin%2C+and+Ting+Liu%2E+2023%2E+Navigate+through+enigmatic+labyrinth+a+survey+of+chain+of+thought+rea)
18. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv preprint arXiv:1803.05457.
19. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. [Qlora: Efficient finetuning of quantized llms](https://md.research.elamur.ai/free/en/papers/dettmers-2023-qlora-efficient-finetuning-quantized-llms/). Advances in neural information processing systems, 36:10088–10115.
20. Siyuan Fang, Kaijing Ma, Tianyu Zheng, Xinrun Du, Ningxuan Lu, Ge Zhang, and Qingkun Tang. 2024. Karpa: A training-free method of adapting knowledge graph as references for large language model’s reasoning path aggregation. arXiv preprint arXiv:2412.20995. [find](https://scholar.google.com/scholar?q=Siyuan+Fang%2C+Kaijing+Ma%2C+Tianyu+Zheng%2C+Xinrun+Du%2C+Ningxuan+Lu%2C+Ge+Zhang%2C+and+Qingkun+Tang%2E+2024%2E+Karpa%3A+A+training%2Dfree+method+of+adapting+knowledge+graph+as+references+for+large+language+model%E2%80%99s+reas)
21. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346– 361. [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Daniel+Khashabi%2C+Elad+Segal%2C+Tushar+Khot%2C+Dan+Roth%2C+and+Jonathan+Berant%2E+2021%2E+Did+aristotle+use+a+laptop%3F+a+question+answering+benchmark+with+implicit+reasoning+strategies%2E+Transactions+of+)
22. Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. 2021. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):1789–1819. [find](https://scholar.google.com/scholar?q=Jianping+Gou%2C+Baosheng+Yu%2C+Stephen+J+Maybank%2C+and+Dacheng+Tao%2E+2021%2E+Knowledge+distillation%3A+A+survey%2E+International+Journal+of+Computer+Vision%2C+129%286%29%3A1789%E2%80%931819%2E)
23. Gaole He, Yunshi Lan, Jing Jiang, Wayne Xin Zhao, and Ji-Rong Wen. 2021. Improving multi-hop knowledge base question answering by learning intermediate supervision signals. In Proceedings of the 14th ACM international conference on web search and data mining, pages 553–561. [find](https://scholar.google.com/scholar?q=Gaole+He%2C+Yunshi+Lan%2C+Jing+Jiang%2C+Wayne+Xin+Zhao%2C+and+Ji%2DRong+Wen%2E+2021%2E+Improving+multi%2Dhop+knowledge+base+question+answering+by+learning+intermediate+supervision+signals%2E+In+Proceedings+of+the+14th+)
24. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). arXiv preprint arXiv:2103.03874.
25. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. [Distilling the knowledge in a neural network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/). arXiv preprint arXiv:1503.02531.
26. Jixiang Hong, Quan Tu, Changyu Chen, Xing Gao, Ji Zhang, and Rui Yan. 2023a. Cyclealign: Iterative distillation from black-box llm to white-box models for better human alignment. arXiv preprint arXiv:2310.16271. [find](https://scholar.google.com/scholar?q=Jixiang+Hong%2C+Quan+Tu%2C+Changyu+Chen%2C+Xing+Gao%2C+Ji+Zhang%2C+and+Rui+Yan%2E+2023a%2E+Cyclealign%3A+Iterative+distillation+from+black%2Dbox+llm+to+white%2Dbox+models+for+better+human+alignment%2E+arXiv+preprint+arXiv%3A)
27. Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, and 1 others. 2023b. [Metagpt: Meta programming for multi-agent collaborative framework](https://md.research.elamur.ai/free/en/papers/hong-2023-metagpt-meta-programming-multi/). arXiv preprint arXiv:2308.00352, 3(4):6.
28. Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. [Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes](https://md.research.elamur.ai/free/en/papers/hsieh-2023-distilling-step-step-outperforming-larger/). arXiv preprint arXiv:2305.02301.
29. Jie Huang and Kevin Chen-Chuan Chang. 2022. [Towards reasoning in large language models: A survey](https://md.research.elamur.ai/free/en/papers/huang-2023-reasoning-large-language-models-survey/). arXiv preprint arXiv:2212.10403.
30. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, and 1 others. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+and+1+others%2E+2024%2E+Mi)
31. Yuxin Jiang, Chunkit Chan, Mingyang Chen, and Wei Wang. 2023. Lion: Adversarial distillation of closed-source large language model. [find](https://scholar.google.com/scholar?q=Yuxin+Jiang%2C+Chunkit+Chan%2C+Mingyang+Chen%2C+and+Wei+Wang%2E+2023%2E+Lion%3A+Adversarial+distillation+of+closed%2Dsource+large+language+model%2E)
32. Yoon Kim and Alexander M Rush. 2016. [Sequence-level knowledge distillation](https://md.research.elamur.ai/free/en/papers/kim-2016-sequence-level-knowledge-distillation/). In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 1317–1327.
33. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2023. [Minillm: Knowledge distillation of large language models](https://md.research.elamur.ai/free/en/papers/gu-2023-minillm-policy-distillation-large-language/). arXiv preprint arXiv:2306.08543.
34. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). Advances in neural information processing systems, 35:22199– 22213.
35. Shanshan Han, Qifan Zhang, Yuhang Yao, Weizhao Jin, Zhaozhuo Xu, and Chaoyang He. 2024. Llm multi-agent systems: Challenges and open problems. arXiv preprint arXiv:2402.03578. — [arXiv:2402.03578](https://arxiv.org/abs/2402.03578) [find](https://scholar.google.com/scholar?q=Shanshan+Han%2C+Qifan+Zhang%2C+Yuhang+Yao%2C+Weizhao+Jin%2C+Zhaozhuo+Xu%2C+and+Chaoyang+He%2E+2024%2E+Llm+multi%2Dagent+systems%3A+Challenges+and+open+problems%2E+arXiv+preprint+arXiv%3A2402%2E03578%2E+%E2%80%94+%5BarXiv%3A2402%2E03578%5D%28htt)
36. Hojae Lee, Junho Kim, and SangKeun Lee. 2024. Mentor-kd: Making small language models better multi-step reasoners. arXiv preprint arXiv:2410.09037. [find](https://scholar.google.com/scholar?q=Hojae+Lee%2C+Junho+Kim%2C+and+SangKeun+Lee%2E+2024%2E+Mentor%2Dkd%3A+Making+small+language+models+better+multi%2Dstep+reasoners%2E+arXiv+preprint+arXiv%3A2410%2E09037%2E)
37. Zhenyu Lei, Yushun Dong, Weiyu Li, Rong Ding, Qi Wang, and Jundong Li. 2025. Harnessing large language models for disaster management: A survey. arXiv preprint arXiv:2501.06932. [find](https://scholar.google.com/scholar?q=Zhenyu+Lei%2C+Yushun+Dong%2C+Weiyu+Li%2C+Rong+Ding%2C+Qi+Wang%2C+and+Jundong+Li%2E+2025%2E+Harnessing+large+language+models+for+disaster+management%3A+A+survey%2E+arXiv+preprint+arXiv%3A2501%2E06932%2E)
38. Dawei Li, Zhen Tan, Tianlong Chen, and Huan Liu. 2024a. [Contextualization distillation from large language model for knowledge graph completion](https://md.research.elamur.ai/free/en/papers/li-2024-contextualization-distillation-large-language-model/). arXiv preprint arXiv:2402.01729.
39. Huao Li, Yu Quan Chong, Simon Stepputtis, Joseph Campbell, Dana Hughes, Michael Lewis, and Katia Sycara. 2023a. [Theory of mind for multi-agent collaboration via large language models](https://md.research.elamur.ai/free/en/papers/li-2023-theory-mind-multi-agent-collaboration/). arXiv preprint arXiv:2310.10701.
40. Jinpeng Li, Yingce Xia, Rui Yan, Hongda Sun, Dongyan Zhao, and Tie-Yan Liu. 2021. Stylized dialogue generation with multi-pass dual learning. Advances in Neural Information Processing Systems, 34:28470– 28481. [find](https://scholar.google.com/scholar?q=Jinpeng+Li%2C+Yingce+Xia%2C+Rui+Yan%2C+Hongda+Sun%2C+Dongyan+Zhao%2C+and+Tie%2DYan+Liu%2E+2021%2E+Stylized+dialogue+generation+with+multi%2Dpass+dual+learning%2E+Advances+in+Neural+Information+Processing+Systems%2C+34%3A2847)
41. Lincan Li, Jiaqi Li, Catherine Chen, Fred Gui, Hongjia Yang, Chenxiao Yu, Zhengguang Wang, Jianing Cai, Junlong Aaron Zhou, Bolin Shen, and 1 others. 2024b. Political-llm: Large language models in political science. arXiv preprint arXiv:2412.06864. — [arXiv:2412.06864](https://arxiv.org/abs/2412.06864) [find](https://scholar.google.com/scholar?q=Lincan+Li%2C+Jiaqi+Li%2C+Catherine+Chen%2C+Fred+Gui%2C+Hongjia+Yang%2C+Chenxiao+Yu%2C+Zhengguang+Wang%2C+Jianing+Cai%2C+Junlong+Aaron+Zhou%2C+Bolin+Shen%2C+and+1+others%2E+2024b%2E+Political%2Dllm%3A+Large+language+models+in+pol)
42. Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023. [Faithful chain-of-thought reasoning](https://md.research.elamur.ai/free/en/papers/lyu-2023-faithful-chain-thought-reasoning/). In The 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Associa…
43. Daniel McDonald, Rachael Papadopoulos, and Leslie Benningfield. 2024. Reducing llm hallucination using knowledge distillation: A case study with mistral large and mmlu benchmark. Authorea Preprints. [find](https://scholar.google.com/scholar?q=Daniel+McDonald%2C+Rachael+Papadopoulos%2C+and+Leslie+Benningfield%2E+2024%2E+Reducing+llm+hallucination+using+knowledge+distillation%3A+A+case+study+with+mistral+large+and+mmlu+benchmark%2E+Authorea+Preprints%2E)
44. Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. 2024. Compact language models via pruning and knowledge distillation. Advances in Neural Information Processing Systems, 37:41076–… [find](https://scholar.google.com/scholar?q=Saurav+Muralidharan%2C+Sharath+Turuvekere+Sreenivas%2C+Raviraj+Joshi%2C+Marcin+Chochowski%2C+Mostofa+Patwary%2C+Mohammad+Shoeybi%2C+Bryan+Catanzaro%2C+Jan+Kautz%2C+and+Pavlo+Molchanov%2E+2024%2E+Compact+language+models+v)
45. Ranjita Naik, Varun Chandrasekaran, Mert Yuksekgonul, Hamid Palangi, and Besmira Nushi. 2023. Diversity of thought improves reasoning abilities of large language models. [find](https://scholar.google.com/scholar?q=Ranjita+Naik%2C+Varun+Chandrasekaran%2C+Mert+Yuksekgonul%2C+Hamid+Palangi%2C+and+Besmira+Nushi%2E+2023%2E+Diversity+of+thought+improves+reasoning+abilities+of+large+language+models%2E)
46. Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wei Chang, and Yejin Choi. 2023b. [Symbolic chain-of-thought distillation: Small models can also" think" step-by-step](https://md.research.elamur.ai/free/en/papers/li-2023-symbolic-chain-thought-distillation-small/). arXiv preprint arXiv:2306.14050.
47. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2019. [Adversarial nli: A new benchmark for natural language understanding](https://md.research.elamur.ai/free/en/papers/nie-2020-adversarial-nli-new-benchmark-natural/). arXiv preprint arXiv:1910.14599.
48. Xinyi Li, Sai Wang, Siqi Zeng, Yu Wu, and Yi Yang. 2024c. A survey on llm-based multi-agent systems: workflow, infrastructure, and challenges. Vicinagearth, 1(1):9. [find](https://scholar.google.com/scholar?q=Xinyi+Li%2C+Sai+Wang%2C+Siqi+Zeng%2C+Yu+Wu%2C+and+Yi+Yang%2E+2024c%2E+A+survey+on+llm%2Dbased+multi%2Dagent+systems%3A+workflow%2C+infrastructure%2C+and+challenges%2E+Vicinagearth%2C+1%281%29%3A9%2E)
49. Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. 2019. Relational knowledge distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3967–3976. [find](https://scholar.google.com/scholar?q=Wonpyo+Park%2C+Dongju+Kim%2C+Yan+Lu%2C+and+Minsu+Cho%2E+2019%2E+Relational+knowledge+distillation%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+computer+vision+and+pattern+recognition%2C+pages+3967%E2%80%933976%2E)
50. Xun Liang, Shichao Song, Zifan Zheng, Hanyu Wang, Qingchen Yu, Xunkai Li, Rong-Hua Li, Yi Wang, Zhonghao Wang, Feiyu Xiong, and 1 others. 2024. Internal consistency and self-feedback in [large language models: A survey](https://md.research.elamur.ai/free/en/papers/minaee-2024-large-language-models-survey/). arXiv preprint arXiv:2407.14507. — [arXiv:2407.14507](https://arxiv.org/abs/2407.14507)
51. Rajvardhan Patil and Venkat Gudivada. 2024. A review of current trends, techniques, and challenges in large language models (llms). Applied Sciences, 14(5):2074. [find](https://scholar.google.com/scholar?q=Rajvardhan+Patil+and+Venkat+Gudivada%2E+2024%2E+A+review+of+current+trends%2C+techniques%2C+and+challenges+in+large+language+models+%28llms%29%2E+Applied+Sciences%2C+14%285%29%3A2074%2E)
52. Xiao Liu, Zirui Wu, Xueqing Wu, Pan Lu, Kai-Wei Chang, and Yansong Feng. 2024. Are llms capable of data-based statistical and causal reasoning? benchmarking advanced quantitative reasoning with data. arXiv preprint arXiv:2402.17644. [find](https://scholar.google.com/scholar?q=Xiao+Liu%2C+Zirui+Wu%2C+Xueqing+Wu%2C+Pan+Lu%2C+Kai%2DWei+Chang%2C+and+Yansong+Feng%2E+2024%2E+Are+llms+capable+of+data%2Dbased+statistical+and+causal+reasoning%3F+benchmarking+advanced+quantitative+reasoning+with+data%2E+)
53. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint arXiv:1907.11692.
54. Jieyi Long. 2023. Large language model guided tree-of-thought. arXiv preprint arXiv:2305.08291. [find](https://scholar.google.com/scholar?q=Jieyi+Long%2E+2023%2E+Large+language+model+guided+tree%2Dof%2Dthought%2E+arXiv+preprint+arXiv%3A2305%2E08291%2E)
55. Yijia Luo, Yulin Song, Xingyao Zhang, Jiaheng Liu, Weixun Wang, GengRu Chen, Wenbo Su, and Bo Zheng. 2025. Deconstructing long chain-of-thought: A structured reasoning optimization framework for long cot distillation. arXiv preprint arXiv:2503.16385. [find](https://scholar.google.com/scholar?q=Yijia+Luo%2C+Yulin+Song%2C+Xingyao+Zhang%2C+Jiaheng+Liu%2C+Weixun+Wang%2C+GengRu+Chen%2C+Wenbo+Su%2C+and+Bo+Zheng%2E+2025%2E+Deconstructing+long+chain%2Dof%2Dthought%3A+A+structured+reasoning+optimization+framework+for+long+)
56. Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, and Thomas Back. 2024. [Reasoning with large language models, a survey](https://md.research.elamur.ai/free/en/papers/minaee-2024-large-language-models-survey/). arXiv preprint arXiv:2407.11511.
57. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, and 1 others. 2022. [Beyond the imitation game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/). arXiv preprint arXiv:2206.…
58. Chuanneng Sun, Songjun Huang, and Dario Pompili. 2024a. Llm-based multi-agent reinforcement learning: Current and future directions. arXiv preprint arXiv:2405.11106. — [arXiv:2405.11106](https://arxiv.org/abs/2405.11106) [find](https://scholar.google.com/scholar?q=Chuanneng+Sun%2C+Songjun+Huang%2C+and+Dario+Pompili%2E+2024a%2E+Llm%2Dbased+multi%2Dagent+reinforcement+learning%3A+Current+and+future+directions%2E+arXiv+preprint+arXiv%3A2405%2E11106%2E+%E2%80%94+%5BarXiv%3A2405%2E11106%5D%28https%3A%2F%2Farxiv)
59. Haotian Sun, Yuchen Zhuang, Wei Wei, Chao Zhang, and Bo Dai. 2024b. Bbox-adapter: Lightweight adapting for black-box large language models. arXiv preprint arXiv:2402.08219. — [arXiv:2402.08219](https://arxiv.org/abs/2402.08219) [find](https://scholar.google.com/scholar?q=Haotian+Sun%2C+Yuchen+Zhuang%2C+Wei+Wei%2C+Chao+Zhang%2C+and+Bo+Dai%2E+2024b%2E+Bbox%2Dadapter%3A+Lightweight+adapting+for+black%2Dbox+large+language+models%2E+arXiv+preprint+arXiv%3A2402%2E08219%2E+%E2%80%94+%5BarXiv%3A2402%2E08219%5D%28https%3A)
60. Jiankai Sun, Chuanyang Zheng, Enze Xie, Zhengying Liu, Ruihang Chu, Jianing Qiu, Jiaqi Xu, Mingyu Ding, Hongyang Li, Mengzhe Geng, and 1 others. 2023. A survey of reasoning with foundation models. arXiv preprint arXiv:2312.11562. — [arXiv:2312.11562](https://arxiv.org/abs/2312.11562) [find](https://scholar.google.com/scholar?q=Jiankai+Sun%2C+Chuanyang+Zheng%2C+Enze+Xie%2C+Zhengying+Liu%2C+Ruihang+Chu%2C+Jianing+Qiu%2C+Jiaqi+Xu%2C+Mingyu+Ding%2C+Hongyang+Li%2C+Mengzhe+Geng%2C+and+1+others%2E+2023%2E+A+survey+of+reasoning+with+foundation+models%2E+arX)
61. Yashar Talebirad and Amirhossein Nadiri. 2023. Multi-agent collaboration: Harnessing the power of intelligent llm agents. arXiv preprint arXiv:2306.03314. [find](https://scholar.google.com/scholar?q=Yashar+Talebirad+and+Amirhossein+Nadiri%2E+2023%2E+Multi%2Dagent+collaboration%3A+Harnessing+the+power+of+intelligent+llm+agents%2E+arXiv+preprint+arXiv%3A2306%2E03314%2E)
62. Zhen Tan, Lu Cheng, Song Wang, Bo Yuan, Jundong Li, and Huan Liu. 2024a. Interpreting pretrained language models via concept bottlenecks. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 56–74. Springer. [find](https://scholar.google.com/scholar?q=Zhen+Tan%2C+Lu+Cheng%2C+Song+Wang%2C+Bo+Yuan%2C+Jundong+Li%2C+and+Huan+Liu%2E+2024a%2E+Interpreting+pretrained+language+models+via+concept+bottlenecks%2E+In+Pacific%2DAsia+Conference+on+Knowledge+Discovery+and+Data+Min)
63. Zhen Tan, Jie Peng, Song Wang, Lijie Hu, Tianlong Chen, and Huan Liu. 2025a. Tuning-free accountable intervention for llm deployment–a metacognitive approach. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 25237– 25245. [find](https://scholar.google.com/scholar?q=Zhen+Tan%2C+Jie+Peng%2C+Song+Wang%2C+Lijie+Hu%2C+Tianlong+Chen%2C+and+Huan+Liu%2E+2025a%2E+Tuning%2Dfree+accountable+intervention+for+llm+deployment%E2%80%93a+metacognitive+approach%2E+In+Proceedings+of+the+AAAI+Conference+on+)
64. Zhen Tan, Jun Yan, I Hsu, Rujun Han, Zifeng Wang, Long T Le, Yiwen Song, Yanfei Chen, Hamid Palangi, George Lee, and 1 others. 2025b. [In prospect and retrospect: Reflective memory management for long-term personalized dialogue agents](https://md.research.elamur.ai/free/en/papers/tan-2025-prospect-retrospect-reflective-memory-management/). arXiv preprint arXiv:2503.08026.
65. Zhen Tan, Chengshuai Zhao, Raha Moraffah, Yifan Li, Song Wang, Jundong Li, Tianlong Chen, and Huan Liu. 2024b. Glue pizza and eat rocks-exploiting vulnerabilities in retrieval-augmented generative models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, page… [find](https://scholar.google.com/scholar?q=Zhen+Tan%2C+Chengshuai+Zhao%2C+Raha+Moraffah%2C+Yifan+Li%2C+Song+Wang%2C+Jundong+Li%2C+Tianlong+Chen%2C+and+Huan+Liu%2E+2024b%2E+Glue+pizza+and+eat+rocks%2Dexploiting+vulnerabilities+in+retrieval%2Daugmented+generative+mod)
66. Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, and 1 others. 2024a. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/). arXiv preprint arXiv:2403.05530.
67. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, and 1 others. 2024b. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295. [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Thomas+Mesnard%2C+Cassidy+Hardin%2C+Robert+Dadashi%2C+Surya+Bhupatiraju%2C+Shreya+Pathak%2C+Laurent+Sifre%2C+Morgane+Rivi%C3%A8re%2C+Mihir+Sanjay+Kale%2C+Juliette+Love%2C+and+1+others%2E+2024b%2E+Gemma%3A+Open+models+)
68. Yongqi Tong, Dawei Li, Sizhe Wang, Yujia Wang, Fei Teng, and Jingbo Shang. 2024. [Can llms learn from previous mistakes?](https://md.research.elamur.ai/free/en/papers/tong-2024-can-llms-learn-previous-mistakes/) investigating llms’ errors to boost for reasoning. arXiv preprint arXiv:2403.20046.
69. Khanh-Tung Tran, Dung Dao, Minh-Duong Nguyen, Quoc-Viet Pham, Barry O’Sullivan, and Hoang D Nguyen. 2025. Multi-agent collaboration mechanisms: A survey of llms. arXiv preprint arXiv:2501.06322. [find](https://scholar.google.com/scholar?q=Khanh%2DTung+Tran%2C+Dung+Dao%2C+Minh%2DDuong+Nguyen%2C+Quoc%2DViet+Pham%2C+Barry+O%E2%80%99Sullivan%2C+and+Hoang+D+Nguyen%2E+2025%2E+Multi%2Dagent+collaboration+mechanisms%3A+A+survey+of+llms%2E+arXiv+preprint+arXiv%3A2501%2E06322%2E)
70. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. [Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions](https://md.research.elamur.ai/free/en/papers/trivedi-2022-interleaving-retrieval-chain-thought-reasoning/). arXiv preprint arXiv:2212.10509.
71. Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [Well-read students learn better: On the importance of pre-training compact models](https://md.research.elamur.ai/free/en/papers/turc-2019-well-read-students-learn-better/). arXiv preprint arXiv:1908.08962.
72. Wei Wang, Zhaowei Li, Qi Xu, Yiqing Cai, Hang Song, Qi Qi, Ran Zhou, Zhida Huang, Tao Wang, and Li Xiao. 2024a. [Qcrd: Quality-guided contrastive rationale distillation for large language models](https://md.research.elamur.ai/free/en/papers/wang-2025-qcrd-quality-guided-contrastive-rationale/). arXiv preprint arXiv:2405.13014.
73. Xinyi Wang, Alfonso Amayuelas, Kexun Zhang, Liangming Pan, Wenhu Chen, and William Yang Wang. 2024b. Understanding reasoning ability of language models from the perspective of reasoning paths aggregation. arXiv preprint arXiv:2402.03268. — [arXiv:2402.03268](https://arxiv.org/abs/2402.03268) [find](https://scholar.google.com/scholar?q=Xinyi+Wang%2C+Alfonso+Amayuelas%2C+Kexun+Zhang%2C+Liangming+Pan%2C+Wenhu+Chen%2C+and+William+Yang+Wang%2E+2024b%2E+Understanding+reasoning+ability+of+language+models+from+the+perspective+of+reasoning+paths+aggregat)
74. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). arXiv preprint arXiv:2203.11171.
75. Yiqi Wang, Wentao Chen, Xiaotian Han, Xudong Lin, Haiteng Zhao, Yongfei Liu, Bohan Zhai, Jianbo Yuan, Quanzeng You, and Hongxia Yang. 2024c. Exploring the reasoning abilities of multimodal large language models (mllms): A comprehensive survey on emerging trends in multimodal reasoning. arXiv preprin… — [arXiv:2401.06805](https://arxiv.org/abs/2401.06805) [find](https://scholar.google.com/scholar?q=Yiqi+Wang%2C+Wentao+Chen%2C+Xiaotian+Han%2C+Xudong+Lin%2C+Haiteng+Zhao%2C+Yongfei+Liu%2C+Bohan+Zhai%2C+Jianbo+Yuan%2C+Quanzeng+You%2C+and+Hongxia+Yang%2E+2024c%2E+Exploring+the+reasoning+abilities+of+multimodal+large+langu)
76. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in neural information processing systems, 35:24824– 24837.
77. Xinyu Wei and Luojia Liu. 2025. Are large language models good in-context learners for financial sentiment analysis? arXiv preprint arXiv:2503.04873. — [arXiv:2503.04873](https://arxiv.org/abs/2503.04873) [find](https://scholar.google.com/scholar?q=Xinyu+Wei+and+Luojia+Liu%2E+2025%2E+Are+large+language+models+good+in%2Dcontext+learners+for+financial+sentiment+analysis%3F+arXiv+preprint+arXiv%3A2503%2E04873%2E+%E2%80%94+%5BarXiv%3A2503%2E04873%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2503%2E04)
78. Peter West, Chandra Bhagavatula, Jack Hessel, Jena D Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2021. [Symbolic knowledge distillation: from general language models to commonsense models](https://md.research.elamur.ai/free/en/papers/west-2022-symbolic-knowledge-distillation-general-language/). arXiv preprint arXiv:2110.07178.
79. Peter West, Ronan Le Bras, Taylor Sorensen, Bill Yuchen Lin, Liwei Jiang, Ximing Lu, Khyathi Chandu, Jack Hessel, Ashutosh Baheti, Chandra Bhagavatula, and 1 others. 2023. Novacomet: Open commonsense foundation models with symbolic knowledge distillation. arXiv preprint arXiv:2312.05979. [find](https://scholar.google.com/scholar?q=Peter+West%2C+Ronan+Le+Bras%2C+Taylor+Sorensen%2C+Bill+Yuchen+Lin%2C+Liwei+Jiang%2C+Ximing+Lu%2C+Khyathi+Chandu%2C+Jack+Hessel%2C+Ashutosh+Baheti%2C+Chandra+Bhagavatula%2C+and+1+others%2E+2023%2E+Novacomet%3A+Open+commonsense+)
80. Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. 2024. A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402.13116. — [arXiv:2402.13116](https://arxiv.org/abs/2402.13116) [find](https://scholar.google.com/scholar?q=Xiaohan+Xu%2C+Ming+Li%2C+Chongyang+Tao%2C+Tao+Shen%2C+Reynold+Cheng%2C+Jinyang+Li%2C+Can+Xu%2C+Dacheng+Tao%2C+and+Tianyi+Zhou%2E+2024%2E+A+survey+on+knowledge+distillation+of+large+language+models%2E+arXiv+preprint+arXiv%3A2)
81. Yang Xu, Yongqiang Yao, Yufan Huang, Mengnan Qi, Maoquan Wang, Bin Gu, and Neel Sundaresan. 2023. Rethinking the instruction quality: Lift is what you need. arXiv preprint arXiv:2312.11508. — [arXiv:2312.11508](https://arxiv.org/abs/2312.11508) [find](https://scholar.google.com/scholar?q=Yang+Xu%2C+Yongqiang+Yao%2C+Yufan+Huang%2C+Mengnan+Qi%2C+Maoquan+Wang%2C+Bin+Gu%2C+and+Neel+Sundaresan%2E+2023%2E+Rethinking+the+instruction+quality%3A+Lift+is+what+you+need%2E+arXiv+preprint+arXiv%3A2312%2E11508%2E+%E2%80%94+%5BarXiv%3A2)
82. Chuanpeng Yang, Yao Zhu, Wang Lu, Yidong Wang, Qian Chen, Chenlong Gao, Bingjie Yan, and Yiqiang Chen. 2024a. Survey on knowledge distillation for large language models: methods, evaluation, and application. ACM Transactions on Intelligent Systems and Technology. [find](https://scholar.google.com/scholar?q=Chuanpeng+Yang%2C+Yao+Zhu%2C+Wang+Lu%2C+Yidong+Wang%2C+Qian+Chen%2C+Chenlong+Gao%2C+Bingjie+Yan%2C+and+Yiqiang+Chen%2E+2024a%2E+Survey+on+knowledge+distillation+for+large+language+models%3A+methods%2C+evaluation%2C+and+appli)
83. Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. 2022. Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 11953–11962. [find](https://scholar.google.com/scholar?q=Borui+Zhao%2C+Quan+Cui%2C+Renjie+Song%2C+Yiyu+Qiu%2C+and+Jiajun+Liang%2E+2022%2E+Decoupled+knowledge+distillation%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+computer+vision+and+pattern+recognition%2C+pages+11953)
84. Chengshuai Zhao, Zhen Tan, Pingchuan Ma, Dawei Li, Bohan Jiang, Yancheng Wang, Yingzhen Yang, and Huan Liu. 2025. [Is chain-of-thought reasoning of llms a mirage? a data distribution lens](https://md.research.elamur.ai/free/en/papers/zhao-2026-is-chain-thought-reasoning-llms/). arXiv preprint arXiv:2508.01191.
85. Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Shaochen Zhong, Bing Yin, and Xia Hu. 2024b. [Harnessing the power of llms in practice: A survey on chatgpt and beyond](https://md.research.elamur.ai/free/en/papers/yang-2023-harnessing-power-llms-practice-survey/). ACM Transactions on Knowledge Discovery from Data, 18(6):1–32.
86. Zaiyi Zheng, Song Wang, Zihan Chen, Yaochen Zhu, Yinhan He, Liangjie Hong, Qi Guo, and Jundong Li. 2025. CoRAG: Enhancing hybrid retrieval-augmented generation through a cooperative retriever architecture. In The 2025 Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Zaiyi+Zheng%2C+Song+Wang%2C+Zihan+Chen%2C+Yaochen+Zhu%2C+Yinhan+He%2C+Liangjie+Hong%2C+Qi+Guo%2C+and+Jundong+Li%2E+2025%2E+CoRAG%3A+Enhancing+hybrid+retrieval%2Daugmented+generation+through+a+cooperative+retriever+architec)
87. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809–11822. [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Tom+Griffiths%2C+Yuan+Cao%2C+and+Karthik+Narasimhan%2E+2023%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2E+Advances+in+neural+inf)
88. Yuhang Zhou and Wei Ai. 2024. Teaching-assistant-in-the-loop: Improving knowledge distillation from imperfect teacher models in low-budget scenarios. arXiv preprint arXiv:2406.05322. [find](https://scholar.google.com/scholar?q=Yuhang+Zhou+and+Wei+Ai%2E+2024%2E+Teaching%2Dassistant%2Din%2Dthe%2Dloop%3A+Improving+knowledge+distillation+from+imperfect+teacher+models+in+low%2Dbudget+scenarios%2E+arXiv+preprint+arXiv%3A2406%2E05322%2E)
89. Zhangyue Yin, Qiushi Sun, Qipeng Guo, Zhiyuan Zeng, Xiaonan Li, Tianxiang Sun, Cheng Chang, Qinyuan Cheng, Ding Wang, Xiaofeng Mou, and 1 others. 2024. Aggregation of reasoning: a hierarchical framework for enhancing answer selection in large language models. arXiv preprint arXiv:2405.12939. [find](https://scholar.google.com/scholar?q=Zhangyue+Yin%2C+Qiushi+Sun%2C+Qipeng+Guo%2C+Zhiyuan+Zeng%2C+Xiaonan+Li%2C+Tianxiang+Sun%2C+Cheng+Chang%2C+Qinyuan+Cheng%2C+Ding+Wang%2C+Xiaofeng+Mou%2C+and+1+others%2E+2024%2E+Aggregation+of+reasoning%3A+a+hierarchical+framewo)
90. Yaochen Zhu, Chao Wan, Harald Steck, Dawen Liang, Yesu Feng, Nathan Kallus, and Jundong Li. 2025. Collaborative retrieval for large language model-based conversational recommender systems. In Proceedings of the ACM on Web Conference 2025, pages 3323–3334. [find](https://scholar.google.com/scholar?q=Yaochen+Zhu%2C+Chao+Wan%2C+Harald+Steck%2C+Dawen+Liang%2C+Yesu+Feng%2C+Nathan+Kallus%2C+and+Jundong+Li%2E+2025%2E+Collaborative+retrieval+for+large+language+model%2Dbased+conversational+recommender+systems%2E+In+Proceedi)
91. Fei Yu, Hongbo Zhang, Prayag Tiwari, and Benyou Wang. 2024. Natural language reasoning, a survey. ACM Computing Surveys, 56(12):1–39. [find](https://scholar.google.com/scholar?q=Fei+Yu%2C+Hongbo+Zhang%2C+Prayag+Tiwari%2C+and+Benyou+Wang%2E+2024%2E+Natural+language+reasoning%2C+a+survey%2E+ACM+Computing+Surveys%2C+56%2812%29%3A1%E2%80%9339%2E)
92. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023. [Metamath: Bootstrap your own mathematical questions for large language models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/). arXiv preprint arXiv:2309.12284.
93. Xiaopeng Yuan, Xingjian Zhang, Ke Xu, Yifan Xu, Lijun Yu, Jindong Wang, Yushun Dong, and Haohan Wang. 2025. Tracing llm reasoning processes with strategic games: A framework for planning, revision, and resource-constrained decision making. arXiv preprint arXiv:2506.12012. — [arXiv:2506.12012](https://arxiv.org/abs/2506.12012) [find](https://scholar.google.com/scholar?q=Xiaopeng+Yuan%2C+Xingjian+Zhang%2C+Ke+Xu%2C+Yifan+Xu%2C+Lijun+Yu%2C+Jindong+Wang%2C+Yushun+Dong%2C+and+Haohan+Wang%2E+2025%2E+Tracing+llm+reasoning+processes+with+strategic+games%3A+A+framework+for+planning%2C+revision%2C+an)
94. Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. 2024. When scaling meets llm finetuning: The effect of data, model and finetuning method. arXiv preprint arXiv:2402.17193. — [arXiv:2402.17193](https://arxiv.org/abs/2402.17193) [find](https://scholar.google.com/scholar?q=Biao+Zhang%2C+Zhongtao+Liu%2C+Colin+Cherry%2C+and+Orhan+Firat%2E+2024%2E+When+scaling+meets+llm+finetuning%3A+The+effect+of+data%2C+model+and+finetuning+method%2E+arXiv+preprint+arXiv%3A2402%2E17193%2E+%E2%80%94+%5BarXiv%3A2402%2E17193%5D)
95. Binchi Zhang, Zaiyi Zheng, Zhengzhang Chen, and Jundong Li. 2025a. Beyond the permutation symmetry of transformers: The role of rotation for model fusion. In Forty-second International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Binchi+Zhang%2C+Zaiyi+Zheng%2C+Zhengzhang+Chen%2C+and+Jundong+Li%2E+2025a%2E+Beyond+the+permutation+symmetry+of+transformers%3A+The+role+of+rotation+for+model+fusion%2E+In+Forty%2Dsecond+International+Conference+on+M)
96. Ruichen Zhang, Rana Muhammad Shahroz Khan, Zhen Tan, Dawei Li, Song Wang, and Tianlong Chen. 2025b. The quest for efficient reasoning: A data-centric benchmark to cot distillation. arXiv preprint arXiv:2505.18759. [find](https://scholar.google.com/scholar?q=Ruichen+Zhang%2C+Rana+Muhammad+Shahroz+Khan%2C+Zhen+Tan%2C+Dawei+Li%2C+Song+Wang%2C+and+Tianlong+Chen%2E+2025b%2E+The+quest+for+efficient+reasoning%3A+A+data%2Dcentric+benchmark+to+cot+distillation%2E+arXiv+preprint+arXi)
97. Model SQA MATH Mistral-7B-Instruct Gemma-7B-Instruct Sum QR-Distill 06:07 07:50 13:57 09:49 04:08 04:59 09:07 05:44 19:37 24:45 44:22 32:15 06:28 08:00 14:28 09:05 00:41 00:54 01:35 00:59 07:24 09:18 16:42 11:34 *Table 3: Training time per epoch (minutes). QR-Distill achieves efficiency gains via pa… [find](https://scholar.google.com/scholar?q=Model+SQA+MATH+Mistral%2D7B%2DInstruct+Gemma%2D7B%2DInstruct+Sum+QR%2DDistill+06%3A07+07%3A50+13%3A57+09%3A49+04%3A08+04%3A59+09%3A07+05%3A44+19%3A37+24%3A45+44%3A22+32%3A15+06%3A28+08%3A00+14%3A28+09%3A05+00%3A41+00%3A54+01%3A35+00%3A59+07%3A24+09%3A18+)

*Updated 1 November 2026*