---
title: "Improving Reasoning Capabilities in Small Models through"
description: "They propose MoLSAKI, which adds a loss that makes the student copy the teacher's per-step… 41 references, cited by 9 papers here."
authors: ["Yao Chen", "Jiawei Sheng", "Wenyuan Zhang", "Tingwen Liu"]
year: 2025
venue: "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2025.emnlp-main.250/"
doi: "10.18653/v1/2025.emnlp-main.250"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chen-2025-improving-reasoning-capabilities-small-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chen-2025-improving-reasoning-capabilities-small-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/8/) / Improving Reasoning Capabilities in Small Models through Mixture-of-layers Distillation with Stepwise Attention on Key Information

# Improving Reasoning Capabilities in Small Models through Mixture-of-layers Distillation with Stepwise Attention on Key Information

**Authors:** [Yao Chen](https://md.research.elamur.ai/free/en/authors/yao-chen/), [Jiawei Sheng](https://md.research.elamur.ai/free/en/authors/jiawei-sheng/), [Wenyuan Zhang](https://md.research.elamur.ai/free/en/authors/wenyuan-zhang/) and [Tingwen Liu](https://md.research.elamur.ai/free/en/authors/tingwen-liu/) · **Year:** 2025 · **Venue:** [Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2025](https://md.research.elamur.ai/free/en/venues/emnlp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.emnlp-main.250/) · [PDF](https://aclanthology.org/2025.emnlp-main.250.pdf) · [DOI 10.18653/v1/2025.emnlp-main.250](https://doi.org/10.18653/v1/2025.emnlp-main.250) · [OpenAlex](https://openalex.org/W4416036844)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [9](https://md.research.elamur.ai/free/en/papers/chen-2025-improving-reasoning-capabilities-small-models/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Small language models are cheaper to deploy than large ones but reason less well. Chain-of-Thought (CoT) distillation usually just trains them on teacher-written rationales, with every token weighted the same. The authors observe that teacher models such as Llama3-8B shift their attention from step to step toward the tokens that matter, for example the numbers in math word problems. Students trained with existing methods tend to miss such clues.

They propose MoLSAKI, which adds a loss that makes the student copy the teacher's per-step attention over critical tokens on top of standard multi-task CoT distillation (answer prediction plus rationale generation). Teachers and students have different numbers of layers. To handle this, a Mixture-of-Layers (MoL) module, inspired by Mixture-of-Experts routing, learns weighted teacher–student layer pairings. This removes the need for fixed layer mappings, tokenizer alignment or projection layers.

With GPT2 students, MoLSAKI beats DSS and MMIloss on mathematical benchmarks (SVAMP, AsDiv, SingleEq) and on commonsense reasoning such as CommonSenseQA. The work suggests that attention dynamics are a useful distillation signal beyond the rationale text itself.

## Key points
- **Setup:** Teachers include Llama3-8B and Qwen2.5-32B, and students are GPT2-Medium and GPT2-Large. Critical tokens are numbers for math tasks and keywords for commonsense questions.
- **Layer alignment:** On SVAMP/AsDiv with a GPT2-Medium student, MoL alignment averaged 37.5. This beat the best fixed single-layer mapping (35.8, last layer to last layer), MMIloss (34.4) and DSS (29.1).
- **Mixed teachers:** Rationales and attention can come from different teachers. For example, PaLM-540B can write the rationales while Llama3-8B supplies the attention maps. This makes it possible to combine black-box and white-box models.
- **Robustness:** With a GPT2-Large student, MoLSAKI gave relative accuracy gains over the baselines of 5.1% when one teacher supplied both signals and 7.9% in the hybrid setup.
- **Limitations:** Only a narrow range of model sizes and architectures was tested. The tasks were relatively simple and involved short reasoning chains.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/government/chinese-academy-of-sciences/) — Government body, Beijing, China · 194 works · 23,906 citations
- [University of Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/universities/university-of-chinese-academy-of-sciences/) — University, Beijing, China · 147 works · 8,480 citations
- [Institute of Information Engineering](https://md.research.elamur.ai/free/en/labs/institute-of-information-engineering/) — Research institute, Beijing, China · 30 works · 1,411 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works

## Cite

Cite as: Yao Chen, Jiawei Sheng, Wenyuan Zhang and Tingwen Liu. 2025. Improving Reasoning Capabilities in Small Models through Mixture-of-layers Distillation with Stepwise Attention on Key Information. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{chen2025improving,
  title = {{Improving Reasoning Capabilities in Small Models through Mixture-of-layers Distillation with Stepwise Attention on Key Information}},
  author = {Yao Chen, Jiawei Sheng, Wenyuan Zhang and Tingwen Liu},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.250},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.250}
}
```

- [← previous Implicit Reasoning in Large Language Models: A Comprehensive Survey](https://md.research.elamur.ai/free/en/papers/li-2025-implicit-reasoning-large-language-models/)
- [next → IncogniText: Privacy-enhancing Conditional Text Anonymization via LLM-based Private Attribute Randomization](https://md.research.elamur.ai/free/en/papers/frikha-2025-incognitext-privacy-enhancing-conditional-text/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/8/)

## References

41 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yilong Chen, Junyuan Shang, Zhenyu Zhang, Yanxi Xie, Jiawei Sheng, Tingwen Liu, Shuohuan Wang, Yu Sun, Hua Wu, and Haifeng Wang. 2025. [Inner thinking transformer: Leveraging dynamic depth scaling to foster adaptive internal thinking](https://md.research.elamur.ai/free/en/papers/chen-2025-inner-thinking-transformer-leveraging-dynamic/). In Proceedings of the 63rd Annual Meeting of the Association for C…
2. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). Journal of Machine Learning Research, 24(240):1–113.
3. Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. 2024. Navigate through enigmatic labyrinth a survey of chain of thought reasoning: Advances, frontiers and future. In Proceedings of the 62nd Annual Meeting of the Association… [find](https://scholar.google.com/scholar?q=Zheng+Chu%2C+Jingchang+Chen%2C+Qianglong+Chen%2C+Weijiang+Yu%2C+Tao+He%2C+Haotian+Wang%2C+Weihua+Peng%2C+Ming+Liu%2C+Bing+Qin%2C+and+Ting+Liu%2E+2024%2E+Navigate+through+enigmatic+labyrinth+a+survey+of+chain+of+thought+rea)
4. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
5. Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. 2023. [Specializing smaller language models towards multi-step reasoning](https://md.research.elamur.ai/free/en/papers/fu-2023-specializing-smaller-language-models-towards/). In International Conference on Machine Learning, pages 10421–10430. PMLR.
6. Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. 2023. Dissecting recall of factual associations in auto-regressive language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12216–12235. [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Jasmijn+Bastings%2C+Katja+Filippova%2C+and+Amir+Globerson%2E+2023%2E+Dissecting+recall+of+factual+associations+in+auto%2Dregressive+language+models%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+)
7. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv e-prints, pages arXiv–2407.
8. Federico Barbero, Alvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Michael Bronstein, Razvan Pascanu, et al. 2025. Why do llms attend to the first token? arXiv preprint arXiv:2504.02732. — [arXiv:2504.02732](https://arxiv.org/abs/2504.02732) [find](https://scholar.google.com/scholar?q=Federico+Barbero%2C+Alvaro+Arroyo%2C+Xiangming+Gu%2C+Christos+Perivolaropoulos%2C+Michael+Bronstein%2C+Razvan+Pascanu%2C+et+al%2E+2025%2E+Why+do+llms+attend+to+the+first+token%3F+arXiv+preprint+arXiv%3A2504%2E02732%2E+%E2%80%94+%5BarX)
9. Namgyu Ho, Laura Schmid, and Se-Young Yun. 2023. [Large language models are reasoning teachers](https://md.research.elamur.ai/free/en/papers/ho-2023-large-language-models-are-reasoning/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14852–14882, Toronto, Canada. Association for Computational Linguistics.
10. Xin Chen, Hanxian Huang, Yanjun Gao, Yi Wang, Jishen Zhao, and Ke Ding. 2024. Learning to maximize mutual information for chain-of-thought distillation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 6857–6868, Bangkok, Thailand. Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Xin+Chen%2C+Hanxian+Huang%2C+Yanjun+Gao%2C+Yi+Wang%2C+Jishen+Zhao%2C+and+Ke+Ding%2E+2024%2E+Learning+to+maximize+mutual+information+for+chain%2Dof%2Dthought+distillation%2E+In+Findings+of+the+Association+for+Computationa)
11. Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. [Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes](https://md.research.elamur.ai/free/en/papers/hsieh-2023-distilling-step-step-outperforming-larger/). In Findings of the Association for Computati…
12. Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. 2024. [Minicpm: Unveiling the potential of small language models with scalable training strategies](https://md.research.elamur.ai/free/en/papers/hu-2024-minicpm-unveiling-potential-small-language/). arXiv preprint arXiv:2404.06395.
13. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. [Tinybert: Distilling bert for natural language understanding](https://md.research.elamur.ai/free/en/papers/jiao-2020-tinybert-distilling-bert-natural-language/). In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4163–4174.
14. Peng Jin, Bo Zhu, Li Yuan, and Shuicheng Yan. 2024. Moh: Multi-head attention as mixture-of-head attention. arXiv preprint arXiv:2410.11842. — [arXiv:2410.11842](https://arxiv.org/abs/2410.11842) [find](https://scholar.google.com/scholar?q=Peng+Jin%2C+Bo+Zhu%2C+Li+Yuan%2C+and+Shuicheng+Yan%2E+2024%2E+Moh%3A+Multi%2Dhead+attention+as+mixture%2Dof%2Dhead+attention%2E+arXiv+preprint+arXiv%3A2410%2E11842%2E+%E2%80%94+%5BarXiv%3A2410%2E11842%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2410%2E11842%29)
15. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). Advances in neural information processing systems, 35:22199– 22213.
16. Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. 2015. Parsing algebraic word problems into equations. Transactions of the Association for Computational Linguistics, 3:585–597. [find](https://scholar.google.com/scholar?q=Rik+Koncel%2DKedziorski%2C+Hannaneh+Hajishirzi%2C+Ashish+Sabharwal%2C+Oren+Etzioni%2C+and+Siena+Dumas+Ang%2E+2015%2E+Parsing+algebraic+word+problems+into+equations%2E+Transactions+of+the+Association+for+Computational)
17. Hojae Lee, Junho Kim, and SangKeun Lee. 2024. Mentor-kd: Making small language models better multi-step reasoners. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17643–17658. [find](https://scholar.google.com/scholar?q=Hojae+Lee%2C+Junho+Kim%2C+and+SangKeun+Lee%2E+2024%2E+Mentor%2Dkd%3A+Making+small+language+models+better+multi%2Dstep+reasoners%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Natural+Language+Proces)
18. Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wei Chang, and Yejin Choi. 2023. [Symbolic chain-of-thought distillation: Small models can also “think” step-by-step](https://md.research.elamur.ai/free/en/papers/li-2023-symbolic-chain-thought-distillation-small/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2…
19. Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, et al. 2025. [From system 1 to system 2: A survey of reasoning large language models](https://md.research.elamur.ai/free/en/papers/li-2025-system-1-system-2-survey/). arXiv preprint arXiv:2502.17419.
20. Yantao Liu, Zhao Zhang, Zijun Yao, Shulin Cao, Lei Hou, and Juanzi Li. 2024. Aligning teacher with student preferences for tailored training data generation. arXiv preprint arXiv:2406.19227. — [arXiv:2406.19227](https://arxiv.org/abs/2406.19227) [find](https://scholar.google.com/scholar?q=Yantao+Liu%2C+Zhao+Zhang%2C+Zijun+Yao%2C+Shulin+Cao%2C+Lei+Hou%2C+and+Juanzi+Li%2E+2024%2E+Aligning+teacher+with+student+preferences+for+tailored+training+data+generation%2E+arXiv+preprint+arXiv%3A2406%2E19227%2E+%E2%80%94+%5BarXiv%3A)
21. Meta. 2024. Introducing meta llama 3: The most capable openly available llm to date. Blog. Accessed: 2024-01-24. [find](https://scholar.google.com/scholar?q=Meta%2E+2024%2E+Introducing+meta+llama+3%3A+The+most+capable+openly+available+llm+to+date%2E+Blog%2E+Accessed%3A+2024%2D01%2D24%2E)
22. Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. 2021. A diverse corpus for evaluating and developing english math word problem solvers. arXiv preprint arXiv:2106.15772. [find](https://scholar.google.com/scholar?q=Shen%2DYun+Miao%2C+Chao%2DChun+Liang%2C+and+Keh%2DYih+Su%2E+2021%2E+A+diverse+corpus+for+evaluating+and+developing+english+math+word+problem+solvers%2E+arXiv+preprint+arXiv%3A2106%2E15772%2E)
23. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. [Are NLP models really able to solve simple math word problems?](https://md.research.elamur.ai/free/en/papers/patel-2021-are-nlp-models-really-able/) In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080–2094, Online. Associat…
24. Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, and Thomas Back. 2024. [Reasoning with large language models, a survey](https://md.research.elamur.ai/free/en/papers/minaee-2024-large-language-models-survey/). arXiv preprint arXiv:2407.11511.
25. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng… [find](https://scholar.google.com/scholar?q=Qwen%2C+%3A%2C+An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+Huan+Lin%2C+Jian+Yang%2C+Jianhong+Tu%2C+Jianwei+Zhang%2C+Jianxin+Yang%2C+Jiaxi+)
26. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. In OpenAI Blog. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+In+OpenAI+Blog%2E)
27. Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020. [Mobilebert: a compact task-agnostic bert for resource-limited devices](https://md.research.elamur.ai/free/en/papers/sun-2020-mobilebert-compact-task-agnostic-bert/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2158–2170.
28. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. [CommonsenseQA: A question answering challenge targeting commonsense knowledge](https://md.research.elamur.ai/free/en/papers/talmor-2019-commonsenseqa-question-answering-challenge-targeting/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
29. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems, 33:5776–5788. [find](https://scholar.google.com/scholar?q=Wenhui+Wang%2C+Furu+Wei%2C+Li+Dong%2C+Hangbo+Bao%2C+Nan+Yang%2C+and+Ming+Zhou%2E+2020%2E+Minilm%3A+Deep+self%2Dattention+distillation+for+task%2Dagnostic+compression+of+pre%2Dtrained+transformers%2E+Advances+in+Neural+Inform)
30. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in neural information processing systems, 35:24824–24837.
31. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453. [find](https://scholar.google.com/scholar?q=Guangxuan+Xiao%2C+Yuandong+Tian%2C+Beidi+Chen%2C+Song+Han%2C+and+Mike+Lewis%2E+2023%2E+Efficient+streaming+language+models+with+attention+sinks%2E+arXiv+preprint+arXiv%3A2309%2E17453%2E)
32. Yunzhi Yao, Ningyu Zhang, Zekun Xi, Mengru Wang, Ziwen Xu, Shumin Deng, and Huajun Chen. 2024. [find](https://scholar.google.com/scholar?q=Yunzhi+Yao%2C+Ningyu+Zhang%2C+Zekun+Xi%2C+Mengru+Wang%2C+Ziwen+Xu%2C+Shumin+Deng%2C+and+Huajun+Chen%2E+2024%2E)
33. Knowledge circuits in pretrained transformers. arXiv preprint arXiv:2405.17969. — [arXiv:2405.17969](https://arxiv.org/abs/2405.17969) [find](https://scholar.google.com/scholar?q=Knowledge+circuits+in+pretrained+transformers%2E+arXiv+preprint+arXiv%3A2405%2E17969%2E+%E2%80%94+%5BarXiv%3A2405%2E17969%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2405%2E17969%29)
34. Biao Zhang and Rico Sennrich. 2019. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32. [find](https://scholar.google.com/scholar?q=Biao+Zhang+and+Rico+Sennrich%2E+2019%2E+Root+mean+square+layer+normalization%2E+Advances+in+Neural+Information+Processing+Systems%2C+32%2E)
35. Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024a. [Tinyllama: An open-source small language model](https://md.research.elamur.ai/free/en/papers/zhang-2024-tinyllama-open-source-small-language/). arXiv preprint arXiv:2401.02385.
36. Songming Zhang, Xue Zhang, Zengkui Sun, Yufeng Chen, and Jinan Xu. 2024b. [Dual-space knowledge distillation for large language models](https://md.research.elamur.ai/free/en/papers/zhang-2024-dual-space-knowledge-distillation-large/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18164–18181, Miami, Florida, USA. Association for Computational Ling…
37. Wenyuan Zhang, Tianyun Liu, Mengxiao Song, Xiaodong Li, and Tingwen Liu. 2025a. SOTOPIA-Ω: Dynamic strategy injection learning and social instruction following evaluation for social agents. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pap… [find](https://scholar.google.com/scholar?q=Wenyuan+Zhang%2C+Tianyun+Liu%2C+Mengxiao+Song%2C+Xiaodong+Li%2C+and+Tingwen+Liu%2E+2025a%2E+SOTOPIA%2D%E2%84%A6%3A+Dynamic+strategy+injection+learning+and+social+instruction+following+evaluation+for+social+agents%2E+In+Proceed)
38. Wenyuan Zhang, Shuaiyi Nie, Jiawei Sheng, Zefeng Zhang, Xinghua Zhang, Yongquan He, and Tingwen Liu. 2024c. Revealing and mitigating the challenge of detecting character knowledge errors in llm role-playing. arXiv preprint arXiv:2409.11726. [find](https://scholar.google.com/scholar?q=Wenyuan+Zhang%2C+Shuaiyi+Nie%2C+Jiawei+Sheng%2C+Zefeng+Zhang%2C+Xinghua+Zhang%2C+Yongquan+He%2C+and+Tingwen+Liu%2E+2024c%2E+Revealing+and+mitigating+the+challenge+of+detecting+character+knowledge+errors+in+llm+role%2Dp)
39. Wenyuan Zhang, Shuaiyi Nie, Xinghua Zhang, Zefeng Zhang, and Tingwen Liu. 2025b. S1-bench: A simple benchmark for evaluating system 1 thinking capability of large reasoning models. arXiv preprint arXiv:2504.10368. — [arXiv:2504.10368](https://arxiv.org/abs/2504.10368) [find](https://scholar.google.com/scholar?q=Wenyuan+Zhang%2C+Shuaiyi+Nie%2C+Xinghua+Zhang%2C+Zefeng+Zhang%2C+and+Tingwen+Liu%2E+2025b%2E+S1%2Dbench%3A+A+simple+benchmark+for+evaluating+system+1+thinking+capability+of+large+reasoning+models%2E+arXiv+preprint+arXi)
40. Zifan Zheng, Yezhaohui Wang, Yuxin Huang, Shichao Song, Mingchuan Yang, Bo Tang, Feiyu Xiong, and Zhiyu Li. 2024. Attention heads of [large language models: A survey](https://md.research.elamur.ai/free/en/papers/minaee-2024-large-language-models-survey/). arXiv preprint arXiv:2409.03752. — [arXiv:2409.03752](https://arxiv.org/abs/2409.03752)
41. Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. 2022. Mixture-of-experts with expert choice routing. Advances in Neural Information Processing Systems, 35:7103–7114. [find](https://scholar.google.com/scholar?q=Yanqi+Zhou%2C+Tao+Lei%2C+Hanxiao+Liu%2C+Nan+Du%2C+Yanping+Huang%2C+Vincent+Zhao%2C+Andrew+M+Dai%2C+Quoc+V+Le%2C+James+Laudon%2C+et+al%2E+2022%2E+Mixture%2Dof%2Dexperts+with+expert+choice+routing%2E+Advances+in+Neural+Information)

*Updated 30 October 2026*