---
title: "Inner Thinking Transformer — paper by Chen et al. (2025)"
description: "Building on this, they propose Inner Thinking Transformer (ITT), which treats each layer computation… 70 references, cited by 16 papers here."
authors: ["Yilong Chen", "Junyuan Shang", "Zhenyu Zhang", "Yanxi Xie", "Jiawei Sheng", "Tingwen Liu", "Shuohuan Wang", "Yu Sun", "Hua Wu", "Haifeng Wang"]
year: 2025
venue: "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2025.acl-long.1369/"
doi: "10.18653/v1/2025.acl-long.1369"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chen-2025-inner-thinking-transformer-leveraging-dynamic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chen-2025-inner-thinking-transformer-leveraging-dynamic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/6/) / Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinking

# Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinking

**Authors:** Yilong Chen, [Junyuan Shang](https://md.research.elamur.ai/free/en/authors/junyuan-shang/), [Zhenyu Zhang](https://md.research.elamur.ai/free/en/authors/zhenyu-zhang/), Yanxi Xie, [Jiawei Sheng](https://md.research.elamur.ai/free/en/authors/jiawei-sheng/), [Tingwen Liu](https://md.research.elamur.ai/free/en/authors/tingwen-liu/), [Shuohuan Wang](https://md.research.elamur.ai/free/en/authors/shuohuan-wang/), [Yu Sun](https://md.research.elamur.ai/free/en/authors/yu-sun/), [Hua Wu](https://md.research.elamur.ai/free/en/authors/hua-ren-wu/) and [Haifeng Wang](https://md.research.elamur.ai/free/en/authors/haifeng-wang/) · **Year:** 2025 · **Venue:** [Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2025](https://md.research.elamur.ai/free/en/venues/acl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.acl-long.1369/) · [PDF](https://aclanthology.org/2025.acl-long.1369.pdf) · [DOI 10.18653/v1/2025.acl-long.1369](https://doi.org/10.18653/v1/2025.acl-long.1369) · [OpenAlex](https://openalex.org/W4412889844)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [16](https://md.research.elamur.ai/free/en/papers/chen-2025-inner-thinking-transformer-leveraging-dynamic/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Larger LLMs bring diminishing returns, and small models hit capacity limits. Chain-of-Thought style test-time scaling also has a weakness: it only works if the model produces the critical tokens correctly. The authors trained GPT-2 on AQuA and tracked gradient norms layer by layer. Tokens the model gets right settle into low, stable gradients early. Tokens it gets wrong show oscillations and sharp spikes throughout the network.

Building on this, they propose Inner Thinking Transformer (ITT), which treats each layer computation as one internal reasoning step. ITT combines three components:
- **Adaptive Token Routing:** a router picks which tokens receive extra passes through a layer.
- **Residual Thinking Connection:** the outputs of these repeated passes are summed rather than simply looped.
- **Thinking Step Encoding:** a learnable encoding weights each step.

ITT models built on the LLaMA architecture beat vanilla Transformer and Loop baselines with the same parameter count on an 11-task benchmark. They also need less training data. The approach matters because it adds depth only where tokens are hard, a parameter-free and adjustable form of test-time scaling.

## Key points
- **Diagnostic finding:** with GPT-2 on AQuA, easy samples had gradient nuclear norm below 3 in layers L3–L10. Hard samples showed spikes at L3, L5, L7 and L9.
- **Model scales:** experiments in the main text use 162M, 230M and 466M models. The abstract instead cites LLaMA2-7B-style 355M/1B/3B scales, so the figures are inconsistent across sections.
- **Headline results:** ITT ×4-162M outperforms the 230M Transformer and reaches 96.5% of the 466M Transformer's performance. ITT saves 43.2% of the training data budget at equal FLOPs.
- **Routing behavior:** about 30–50% of tokens receive iterative thinking, mostly task-critical ones such as verbs. Lowering the 4th-step selection rate to 50% at inference improved perplexity by 0.26.
- **Stated limitations:** routing patterns are fixed during training, validation stops at 466M parameters, and RTC adds memory overhead in backward passes.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Baidu](https://md.research.elamur.ai/free/en/companies/baidu/) — Company, Beijing, China · 84 works · 17,681 citations
- [Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/government/chinese-academy-of-sciences/) — Government body, Beijing, China · 246 works · 24,951 citations
- [University of Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/universities/university-of-chinese-academy-of-sciences/) — University, Beijing, China · 188 works · 8,743 citations
- [Beijing Normal University](https://md.research.elamur.ai/free/en/universities/beijing-normal-university/) — University, Beijing, China · 22 works · 3,177 citations
- [Institute of Information Engineering](https://md.research.elamur.ai/free/en/labs/institute-of-information-engineering/) — Research institute, Beijing, China · 41 works · 1,485 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

## Cite

Cite as: Yilong Chen, Junyuan Shang, Zhenyu Zhang, Yanxi Xie, Jiawei Sheng, Tingwen Liu, Shuohuan Wang, Yu Sun, Hua Wu and Haifeng Wang. 2025. Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinking. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{chen2025inner,
  title = {{Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinking}},
  author = {Yilong Chen, Junyuan Shang, Zhenyu Zhang, Yanxi Xie, Jiawei Sheng, Tingwen Liu, Shuohuan Wang, Yu Sun, Hua Wu and Haifeng Wang},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1369},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1369}
}
```

- [← previous Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Models](https://md.research.elamur.ai/free/en/papers/wang-2025-harnessing-reasoning-economy-survey-efficient/)
- [next → Language Mixing in Reasoning Language Models: Patterns, Impact, and Internal Causes](https://md.research.elamur.ai/free/en/papers/wang-2025-language-mixing-reasoning-language-models/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/6/)

## References

70 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anthropic. 2023. Introducing claude. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2023%2E+Introducing+claude%2E)
2. Arpit Bansal, Avi Schwarzschild, Eitan Borgnia, Zeyad Ali Sami Emam, Furong Huang, Micah Goldblum, and Tom Goldstein. 2022. End-to-end algorithm synthesis with recurrent networks: Extrapolation without overthinking. In Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Arpit+Bansal%2C+Avi+Schwarzschild%2C+Eitan+Borgnia%2C+Zeyad+Ali+Sami+Emam%2C+Furong+Huang%2C+Micah+Goldblum%2C+and+Tom+Goldstein%2E+2022%2E+End%2Dto%2Dend+algorithm+synthesis+with+recurrent+networks%3A+Extrapolation+withou)
3. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. [PIQA: reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference,…
4. Mohamed Boudiaf, Pierre Dufourq, and Sylvain Rousset. 2022. Test-time adaptation with output states. In International Conference on Machine Learning, volume 162, pages 2212–2232. [find](https://scholar.google.com/scholar?q=Mohamed+Boudiaf%2C+Pierre+Dufourq%2C+and+Sylvain+Rousset%2E+2022%2E+Test%2Dtime+adaptation+with+output+states%2E+In+International+Conference+on+Machine+Learning%2C+volume+162%2C+pages+2212%E2%80%932232%2E)
5. Valentino Braitenberg. 1986. Vehicles: Experiments in Synthetic Psychology. MIT Press, Cambridge, MA. [find](https://scholar.google.com/scholar?q=Valentino+Braitenberg%2E+1986%2E+Vehicles%3A+Experiments+in+Synthetic+Psychology%2E+MIT+Press%2C+Cambridge%2C+MA%2E)
6. Guanzheng Chen, Xin Li, Zaiqiao Meng, Shangsong Liang, and Lidong Bing. 2023a. Token-level contrastive estimation enhances llm’s reasoning. arXiv preprint arXiv:2310.16450. — [arXiv:2310.16450](https://arxiv.org/abs/2310.16450) [find](https://scholar.google.com/scholar?q=Guanzheng+Chen%2C+Xin+Li%2C+Zaiqiao+Meng%2C+Shangsong+Liang%2C+and+Lidong+Bing%2E+2023a%2E+Token%2Dlevel+contrastive+estimation+enhances+llm%E2%80%99s+reasoning%2E+arXiv+preprint+arXiv%3A2310%2E16450%2E+%E2%80%94+%5BarXiv%3A2310%2E16450%5D%28https%3A)
7. Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, and Wolfgang Macherey. 2024a. Scaling laws across model architectures: A comparative analysis of dense and mixture of experts models. arXiv preprint arXiv:2410.05661. — [arXiv:2410.05661](https://arxiv.org/abs/2410.05661) [find](https://scholar.google.com/scholar?q=Mia+Xu+Chen%2C+Orhan+Firat%2C+Ankur+Bapna%2C+Melvin+Johnson%2C+and+Wolfgang+Macherey%2E+2024a%2E+Scaling+laws+across+model+architectures%3A+A+comparative+analysis+of+dense+and+mixture+of+experts+models%2E+arXiv+prepr)
8. Xingwu Chen and Difan Zou. 2024. What can transformer learn with varying depth? case studies on sequence learning tasks. ArXiv, abs/2404.01601. — [arXiv:2404.01601](https://arxiv.org/abs/2404.01601) [find](https://scholar.google.com/scholar?q=Xingwu+Chen+and+Difan+Zou%2E+2024%2E+What+can+transformer+learn+with+varying+depth%3F+case+studies+on+sequence+learning+tasks%2E+ArXiv%2C+abs%2F2404%2E01601%2E+%E2%80%94+%5BarXiv%3A2404%2E01601%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2404%2E01601%29)
9. Yanxi Chen, Xuchen Pan, Yaliang Li, Bolin Ding, and Jingren Zhou. 2023b. Ee-llm: Large-scale training and inference of early-exit large language models with 3d parallelism. arXiv preprint arXiv:2312.04916. — [arXiv:2312.04916](https://arxiv.org/abs/2312.04916) [find](https://scholar.google.com/scholar?q=Yanxi+Chen%2C+Xuchen+Pan%2C+Yaliang+Li%2C+Bolin+Ding%2C+and+Jingren+Zhou%2E+2023b%2E+Ee%2Dllm%3A+Large%2Dscale+training+and+inference+of+early%2Dexit+large+language+models+with+3d+parallelism%2E+arXiv+preprint+arXiv%3A2312%2E0)
10. Yilong Chen, Junyuan Shang, Zhengyu Zhang, Jiawei Sheng, Tingwen Liu, Shuohuan Wang, Yu Sun, Hua Wu, and Haifeng Wang. 2024b. Mixture of hidden-dimensions transformer. ArXiv, abs/2412.05644. — [arXiv:2412.05644](https://arxiv.org/abs/2412.05644) [find](https://scholar.google.com/scholar?q=Yilong+Chen%2C+Junyuan+Shang%2C+Zhengyu+Zhang%2C+Jiawei+Sheng%2C+Tingwen+Liu%2C+Shuohuan+Wang%2C+Yu+Sun%2C+Hua+Wu%2C+and+Haifeng+Wang%2E+2024b%2E+Mixture+of+hidden%2Ddimensions+transformer%2E+ArXiv%2C+abs%2F2412%2E05644%2E+%E2%80%94+%5BarXiv%3A)
11. Yilong Chen, Guoxia Wang, Junyuan Shang, Shiyao Cui, Zhenyu Zhang, Tingwen Liu, Shuohuan Wang, Yu Sun, Dianhai Yu, and Hua Wu. 2024c. Nacl: A general and effective kv cache eviction framework for llm at inference time. ArXiv, abs/2408.03675. [find](https://scholar.google.com/scholar?q=Yilong+Chen%2C+Guoxia+Wang%2C+Junyuan+Shang%2C+Shiyao+Cui%2C+Zhenyu+Zhang%2C+Tingwen+Liu%2C+Shuohuan+Wang%2C+Yu+Sun%2C+Dianhai+Yu%2C+and+Hua+Wu%2E+2024c%2E+Nacl%3A+A+general+and+effective+kv+cache+eviction+framework+for+llm+)
12. Yilong Chen, Linhao Zhang, Junyuan Shang, Zhenyu Zhang, Tingwen Liu, Shuohuan Wang, and Yu Sun. 2024d. Dha: Learning decoupled-head attention from transformer checkpoints via adaptive heads fusion. ArXiv, abs/2406.06567. — [arXiv:2406.06567](https://arxiv.org/abs/2406.06567) [find](https://scholar.google.com/scholar?q=Yilong+Chen%2C+Linhao+Zhang%2C+Junyuan+Shang%2C+Zhenyu+Zhang%2C+Tingwen+Liu%2C+Shuohuan+Wang%2C+and+Yu+Sun%2E+2024d%2E+Dha%3A+Learning+decoupled%2Dhead+attention+from+transformer+checkpoints+via+adaptive+heads+fusion%2E+Ar)
13. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. [Boolq: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). arXiv preprint arXiv:1905.10044.
14. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018a. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv preprint arXiv:1803.05457.
15. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018b. [Think you have solved question answering? try arc, the AI2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). CoRR, abs/1803.05457.
16. Jeff Dean. 2021. Introducing pathways: A next-generation ai architecture. Google Blog, 366. [find](https://scholar.google.com/scholar?q=Jeff+Dean%2E+2021%2E+Introducing+pathways%3A+A+next%2Dgeneration+ai+architecture%2E+Google+Blog%2C+366%2E)
17. Mostafa Dehghani, Stephan Gouws, Nal Kalchbrenner, Noam Shazeer, and Łukasz Kaiser. 2019a. Universal transformers. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Mostafa+Dehghani%2C+Stephan+Gouws%2C+Nal+Kalchbrenner%2C+Noam+Shazeer%2C+and+%C5%81ukasz+Kaiser%2E+2019a%2E+Universal+transformers%2E+In+International+Conference+on+Learning+Representations%2E)
18. Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. 2019b. Universal transformers. Preprint, arXiv:1807.03819. [find](https://scholar.google.com/scholar?q=Mostafa+Dehghani%2C+Stephan+Gouws%2C+Oriol+Vinyals%2C+Jakob+Uszkoreit%2C+and+%C5%81ukasz+Kaiser%2E+2019b%2E+Universal+transformers%2E+Preprint%2C+arXiv%3A1807%2E03819%2E)
19. Yuntian Deng, Kiran Prasad, Roland Fernandez, Paul Smolensky, Vishrav Chaudhary, and Stuart Shieber. 2023. Implicit chain of thought reasoning via knowledge distillation. arXiv preprint arXiv:2311.01460. [find](https://scholar.google.com/scholar?q=Yuntian+Deng%2C+Kiran+Prasad%2C+Roland+Fernandez%2C+Paul+Smolensky%2C+Vishrav+Chaudhary%2C+and+Stuart+Shieber%2E+2023%2E+Implicit+chain+of+thought+reasoning+via+knowledge+distillation%2E+arXiv+preprint+arXiv%3A2311%2E014)
20. Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed A Aly, Beidi Chen, and Carole-Jean Wu. 2024. [Layerskip: Enabling early exit inference and self-speculative decoding](https://md.research.elamur.ai/free/en/papers/elhoushi-2024-layerskip-enabling-early-exit-inference/). arXiv preprint arXiv:2404…
21. Jared Fernandez, Luca Wehrstedt, Leonid Shamis, Mostafa Elhoushi, Kalyan Saladi, Yonatan Bisk, Emma Strubell, and Jacob Kahn. 2024. Hardware scaling trends and diminishing returns in large-scale distributed training. arXiv preprint arXiv:2411.13055. [find](https://scholar.google.com/scholar?q=Jared+Fernandez%2C+Luca+Wehrstedt%2C+Leonid+Shamis%2C+Mostafa+Elhoushi%2C+Kalyan+Saladi%2C+Yonatan+Bisk%2C+Emma+Strubell%2C+and+Jacob+Kahn%2E+2024%2E+Hardware+scaling+trends+and+diminishing+returns+in+large%2Dscale+distr)
22. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Ani… [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Baber+Abbasi%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Alain+Le+Noac%E2%80%99h%2C+Haonan+Li%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Chris+Oci)
23. Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. 2025. Scaling up test-time compute with latent reasoning: A recurrent depth approach. Preprint, arXiv:2502.05171. [find](https://scholar.google.com/scholar?q=Jonas+Geiping%2C+Sean+McLeish%2C+Neel+Jain%2C+John+Kirchenbauer%2C+Siddharth+Singh%2C+Brian+R%2E+Bartoldson%2C+Bhavya+Kailkhura%2C+Abhinav+Bhatele%2C+and+Tom+Goldstein%2E+2025%2E+Scaling+up+test%2Dtime+compute+with+latent+re)
24. Felix A. Gers and Jürgen Schmidhuber. 2000. Learning to forget: Continual prediction with LSTM. Neural Computation, 12(10):2451–2471. [find](https://scholar.google.com/scholar?q=Felix+A%2E+Gers+and+J%C3%BCrgen+Schmidhuber%2E+2000%2E+Learning+to+forget%3A+Continual+prediction+with+LSTM%2E+Neural+Computation%2C+12%2810%29%3A2451%E2%80%932471%2E)
25. Naibin Gu, Peng Fu, Xiyu Liu, Bowen Shen, Zheng Lin, and Weiping Wang. 2024. Light-peft: Lightening parameter-efficient fine-tuning via early pruning. Preprint, arXiv:2406.03792. [find](https://scholar.google.com/scholar?q=Naibin+Gu%2C+Peng+Fu%2C+Xiyu+Liu%2C+Bowen+Shen%2C+Zheng+Lin%2C+and+Weiping+Wang%2E+2024%2E+Light%2Dpeft%3A+Lightening+parameter%2Defficient+fine%2Dtuning+via+early+pruning%2E+Preprint%2C+arXiv%3A2406%2E03792%2E)
26. Naibin Gu, Zhenyu Zhang, Xiyu Liu, Peng Fu, Zheng Lin, Shuohuan Wang, Yu Sun, Hua Wu, Weiping Wang, and Haifeng Wang. 2025. [Beamlora: Beam-constraint low-rank adaptation](https://md.research.elamur.ai/free/en/papers/gu-2025-beamlora-beam-constraint-low-rank/). Preprint, arXiv:2502.13604.
27. Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. 2024. Training large language models to reason in a continuous latent space. Preprint, arXiv:2412.06769. [find](https://scholar.google.com/scholar?q=Shibo+Hao%2C+Sainbayar+Sukhbaatar%2C+DiJia+Su%2C+Xian+Li%2C+Zhiting+Hu%2C+Jason+Weston%2C+and+Yuandong+Tian%2E+2024%2E+Training+large+language+models+to+reason+in+a+continuous+latent+space%2E+Preprint%2C+arXiv%3A2412%2E06769)
28. XO He. 2024. Mixture of a million experts. arXiv preprint arXiv:2407.04153. [find](https://scholar.google.com/scholar?q=XO+He%2E+2024%2E+Mixture+of+a+million+experts%2E+arXiv+preprint+arXiv%3A2407%2E04153%2E)
29. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, and Trevor Cai. 2022. [Training compute-optimal large language models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/). arXiv preprint arXiv:2203.15556.
30. Ying Fan, Yilun Du, Kannan Ramchandran, and Kangwook Lee. 2024. Looped transformers for length generalization. ArXiv, abs/2409.15647. [find](https://scholar.google.com/scholar?q=Ying+Fan%2C+Yilun+Du%2C+Kannan+Ramchandran%2C+and+Kangwook+Lee%2E+2024%2E+Looped+transformers+for+length+generalization%2E+ArXiv%2C+abs%2F2409%2E15647%2E)
31. A Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Benoît Savary, Charles Bamford, Devendra Singh Chaplot, Daniele de la Casas, Emily Bressand Hanna, François Bressand, et al. 2024a. Mixtral of experts. arXiv preprint arXiv:2401.04088. [find](https://scholar.google.com/scholar?q=A+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Beno%C3%AEt+Savary%2C+Charles+Bamford%2C+Devendra+Singh+Chaplot%2C+Daniele+de+la+Casas%2C+Emily+Bressand+Hanna%2C+Fran%C3%A7ois+Bressand%2C+et+al%2E+2024a%2E+Mixt)
32. William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1):5232– 5270. [find](https://scholar.google.com/scholar?q=William+Fedus%2C+Barret+Zoph%2C+and+Noam+Shazeer%2E+2022%2E+Switch+transformers%3A+Scaling+to+trillion+parameter+models+with+simple+and+efficient+sparsity%2E+The+Journal+of+Machine+Learning+Research%2C+23%281%29%3A5232%E2%80%93+)
33. Bowen Jiang, Yangxinyu Xie, Zhuoqun Hao, Xiaomeng Wang, Tanwi Mallick, Weijie J. Su, Camillo J. Taylor, and Dan Roth. 2024b. [A peek into token bias: Large language models are not yet genuine reasoners](https://md.research.elamur.ai/free/en/papers/jiang-2024-peek-into-token-bias-large/). arXiv preprint arXiv:2406.11050.
34. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. [ALBERT: A lite BERT for self-supervised learning of language representations](https://md.research.elamur.ai/free/en/papers/lan-2019-albert-lite-bert-self-supervised/). arXiv preprint arXiv:1909.11942.
35. Yann LeCun, Geoffrey E. Hinton, and Simon Osindero. 2006. Contrastive divergence: A fast and simple learning algorithm for deep belief networks. Neural Computation, 18(11):2271–2323. [find](https://scholar.google.com/scholar?q=Yann+LeCun%2C+Geoffrey+E%2E+Hinton%2C+and+Simon+Osindero%2E+2006%2E+Contrastive+divergence%3A+A+fast+and+simple+learning+algorithm+for+deep+belief+networks%2E+Neural+Computation%2C+18%2811%29%3A2271%E2%80%932323%2E)
36. Ming Li, Yanhong Li, and Tianyi Zhou. 2024a. [What happened in llms layers when trained for fast vs. slow thinking: A gradient perspective](https://md.research.elamur.ai/free/en/papers/li-2025-what-happened-llms-layers-when/). Preprint, arXiv:2410.23743.
37. Yifan Li, Yichao Zhang, Yiming Wang, and Yujia Zhang. 2024b. Cross-layer attention sharing for large language models. arXiv preprint arXiv:2408.01890. — [arXiv:2408.01890](https://arxiv.org/abs/2408.01890) [find](https://scholar.google.com/scholar?q=Yifan+Li%2C+Yichao+Zhang%2C+Yiming+Wang%2C+and+Yujia+Zhang%2E+2024b%2E+Cross%2Dlayer+attention+sharing+for+large+language+models%2E+arXiv+preprint+arXiv%3A2408%2E01890%2E+%E2%80%94+%5BarXiv%3A2408%2E01890%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2408%2E0)
38. Yifan Li, Yichao Zhang, Yiming Wang, and Yujia Zhang. 2024c. Cross-layer parameter sharing for large language model compression. arXiv preprint arXiv:2410.03765. — [arXiv:2410.03765](https://arxiv.org/abs/2410.03765) [find](https://scholar.google.com/scholar?q=Yifan+Li%2C+Yichao+Zhang%2C+Yiming+Wang%2C+and+Yujia+Zhang%2E+2024c%2E+Cross%2Dlayer+parameter+sharing+for+large+language+model+compression%2E+arXiv+preprint+arXiv%3A2410%2E03765%2E+%E2%80%94+%5BarXiv%3A2410%2E03765%5D%28https%3A%2F%2Farxiv%2Eorg)
39. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv preprint arXiv:1705.04146. [find](https://scholar.google.com/scholar?q=Wang+Ling%2C+Dani+Yogatama%2C+Chris+Dyer%2C+and+Phil+Blunsom%2E+2017%2E+Program+induction+by+rationale+generation%3A+Learning+to+solve+and+explain+algebraic+word+problems%2E+arXiv+preprint+arXiv%3A1705%2E04146%2E)
40. Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020. [Logiqa: A challenge dataset for machine reading comprehension with logical reasoning](https://md.research.elamur.ai/free/en/papers/liu-2020-logiqa-challenge-dataset-machine-reading/). arXiv preprint arXiv:2007.08124.
41. Nanye Ma, Shangyuan Tong, Haolin Jia, Hexiang Hu, Yu-Chuan Su, Mingda Zhang, Xuan Yang, Yandong Li, Tommi Jaakkola, Xuhui Jia, and Saining Xie. 2024. Inference-time scaling for diffusion models beyond scaling denoising steps. arXiv preprint arXiv:2501.09732. [find](https://scholar.google.com/scholar?q=Nanye+Ma%2C+Shangyuan+Tong%2C+Haolin+Jia%2C+Hexiang+Hu%2C+Yu%2DChuan+Su%2C+Mingda+Zhang%2C+Xuan+Yang%2C+Yandong+Li%2C+Tommi+Jaakkola%2C+Xuhui+Jia%2C+and+Saining+Xie%2E+2024%2E+Inference%2Dtime+scaling+for+diffusion+models+beyond)
42. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. [The lambada dataset: Word prediction requiring a broad discourse context](https://md.research.elamur.ai/free/en/papers/paperno-2016-lambada-dataset-word-prediction-requiring/). arXiv preprint arXiv:1606.06031.
43. David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. 2024. Mixture-of-depths: Dynamically allocating compute in transformer-based language models. Preprint, arXiv:2404.02258. [find](https://scholar.google.com/scholar?q=David+Raposo%2C+Sam+Ritter%2C+Blake+Richards%2C+Timothy+Lillicrap%2C+Peter+Conway+Humphreys%2C+and+Adam+Santoro%2E+2024%2E+Mixture%2Dof%2Ddepths%3A+Dynamically+allocating+compute+in+transformer%2Dbased+language+models%2E+Pre)
44. Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Mario Neumann, Rodolphe Jenatton, António Susano Pinto, Daniel Keysers, and Neil Houlsby. 2021. [Scaling vision with sparse mixture of experts](https://md.research.elamur.ai/free/en/papers/riquelme-2021-scaling-vision-sparse-mixture-experts/). In Advances in Neural Information Processing Systems, volume 34, pages 8583–8595.
45. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAA… [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2020%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+In+The+Thirty%2DFourth+AAAI+Conference+on+Artificial+Intellige)
46. Nikunj Saunshi, Stefani Karp, Shankar Krishnan, Sobhan Miryoosefi, Sashank J. Reddi, and Sanjiv Kumar. 2024. On the inductive bias of stacking towards improving reasoning. ArXiv, abs/2409.19044. — [arXiv:2409.19044](https://arxiv.org/abs/2409.19044) [find](https://scholar.google.com/scholar?q=Nikunj+Saunshi%2C+Stefani+Karp%2C+Shankar+Krishnan%2C+Sobhan+Miryoosefi%2C+Sashank+J%2E+Reddi%2C+and+Sanjiv+Kumar%2E+2024%2E+On+the+inductive+bias+of+stacking+towards+improving+reasoning%2E+ArXiv%2C+abs%2F2409%2E19044%2E+%E2%80%94+%5Bar)
47. Avi Schwarzschild, Eitan Borgnia, Arjun Gupta, Furong Huang, Uzi Vishkin, Micah Goldblum, and Tom Goldstein. 2021a. Can you learn an algorithm? generalizing from easy to hard problems with recurrent networks. In Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Avi+Schwarzschild%2C+Eitan+Borgnia%2C+Arjun+Gupta%2C+Furong+Huang%2C+Uzi+Vishkin%2C+Micah+Goldblum%2C+and+Tom+Goldstein%2E+2021a%2E+Can+you+learn+an+algorithm%3F+generalizing+from+easy+to+hard+problems+with+recurrent+n)
48. Thomas McLeish, Daniel Bear, and David Schwarzschild. 2024. Recurrent optimization for algorithmic generalization. arXiv preprint arXiv:2403.07891. — [arXiv:2403.07891](https://arxiv.org/abs/2403.07891) [find](https://scholar.google.com/scholar?q=Thomas+McLeish%2C+Daniel+Bear%2C+and+David+Schwarzschild%2E+2024%2E+Recurrent+optimization+for+algorithmic+generalization%2E+arXiv+preprint+arXiv%3A2403%2E07891%2E+%E2%80%94+%5BarXiv%3A2403%2E07891%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2403%2E0789)
49. David Schwarzschild, Ilya Sutskever, and Yoshua Bengio. 2021b. Randomized unrolling: Generalization and optimization in recurrent neural networks. In International Conference on Machine Learning, volume 139, pages 9236–9246. [find](https://scholar.google.com/scholar?q=David+Schwarzschild%2C+Ilya+Sutskever%2C+and+Yoshua+Bengio%2E+2021b%2E+Randomized+unrolling%3A+Generalization+and+optimization+in+recurrent+neural+networks%2E+In+International+Conference+on+Machine+Learning%2C+volu)
50. Tomas Mikolov, Martin Karafiát, Luká Burget, Jan Honza ernocký, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Interspeech. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Martin+Karafi%C3%A1t%2C+Luk%C3%A1+Burget%2C+Jan+Honza+ernock%C3%BD%2C+and+Sanjeev+Khudanpur%2E+2010%2E+Recurrent+neural+network+based+language+model%2E+In+Interspeech%2E)
51. Yuval Shalev, Amir Feder, and Ariel Goldstein. 2024. Distributional reasoning in llms: Parallel reasoning processes in multi-hop reasoning. arXiv preprint arXiv:2406.13858. — [arXiv:2406.13858](https://arxiv.org/abs/2406.13858) [find](https://scholar.google.com/scholar?q=Yuval+Shalev%2C+Amir+Feder%2C+and+Ariel+Goldstein%2E+2024%2E+Distributional+reasoning+in+llms%3A+Parallel+reasoning+processes+in+multi%2Dhop+reasoning%2E+arXiv+preprint+arXiv%3A2406%2E13858%2E+%E2%80%94+%5BarXiv%3A2406%2E13858%5D%28https%3A)
52. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. [s1: Simple test-time scaling](https://md.research.elamur.ai/free/en/papers/muennighoff-2025-s1-simple-test-time-scaling/). arXiv preprint arXiv:2501.19393.
53. Kei-Sing Ng and Qingchen Wang. 2024. Loop neural networks for parameter sharing. Preprint, arXiv:2409.14199. — [arXiv:2409.14199](https://arxiv.org/abs/2409.14199) [find](https://scholar.google.com/scholar?q=Kei%2DSing+Ng+and+Qingchen+Wang%2E+2024%2E+Loop+neural+networks+for+parameter+sharing%2E+Preprint%2C+arXiv%3A2409%2E14199%2E+%E2%80%94+%5BarXiv%3A2409%2E14199%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2409%2E14199%29)
54. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). ArXiv, page abs/2303.08774.
55. Joykirat Singh, Akshay Nambi, and Vibhav Vineet. 2024. [Exposing the achilles’ heel: Evaluating llms ability to handle mistakes in mathematical reasoning](https://md.research.elamur.ai/free/en/papers/singh-2025-exposing-achilles-heel-evaluating-llms/). arXiv preprint arXiv:2406.10834.
56. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. [Scaling llm test-time compute optimally can be more effective than scaling model parameters](https://md.research.elamur.ai/free/en/papers/snell-2024-scaling-llm-test-time-compute/). arXiv preprint arXiv:2408.03314.
57. Qi Sun, Marc Pickett, Aakash Kumar Nain, and Llion Jones. 2025. Transformer layers as painters. Preprint, arXiv:2407.09298. [find](https://scholar.google.com/scholar?q=Qi+Sun%2C+Marc+Pickett%2C+Aakash+Kumar+Nain%2C+and+Llion+Jones%2E+2025%2E+Transformer+layers+as+painters%2E+Preprint%2C+arXiv%3A2407%2E09298%2E)
58. Sho Takase and Shun Kiyono. 2023. Lessons on parameter sharing across layers in transformers. Preprint, arXiv:2104.06022. [find](https://scholar.google.com/scholar?q=Sho+Takase+and+Shun+Kiyono%2E+2023%2E+Lessons+on+parameter+sharing+across+layers+in+transformers%2E+Preprint%2C+arXiv%3A2104%2E06022%2E)
59. The Mosaic ML Team. 2021. composer. [https://github.com/mosaicml/composer/](https://github.com/mosaicml/composer/). [↗](https://github.com/mosaicml/composer/)
60. TogetherAI. 2023. Redpajama: An open source recipe to reproduce llama training dataset. [find](https://scholar.google.com/scholar?q=TogetherAI%2E+2023%2E+Redpajama%3A+An+open+source+recipe+to+reproduce+llama+training+dataset%2E)
61. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
62. Wenyuan Zhang, Tianyun Liu, Mengxiao Song, Xiaodong Li, and Tingwen Liu. 2025a. SOTOPIA-Ω: Dynamic strategy injection learning and social instruction following evaluation for social agents. Preprint, arXiv:2502.15538. — [arXiv:2502.15538](https://arxiv.org/abs/2502.15538) [find](https://scholar.google.com/scholar?q=Wenyuan+Zhang%2C+Tianyun+Liu%2C+Mengxiao+Song%2C+Xiaodong+Li%2C+and+Tingwen+Liu%2E+2025a%2E+SOTOPIA%2D%E2%84%A6%3A+Dynamic+strategy+injection+learning+and+social+instruction+following+evaluation+for+social+agents%2E+Preprint%2C+)
63. Wenyuan Zhang, Shuaiyi Nie, Xinghua Zhang, Zefeng Zhang, and Tingwen Liu. 2025b. S1-bench: A simple benchmark for evaluating system 1 thinking capability of large reasoning models. Preprint, arXiv:2504.10368. — [arXiv:2504.10368](https://arxiv.org/abs/2504.10368) [find](https://scholar.google.com/scholar?q=Wenyuan+Zhang%2C+Shuaiyi+Nie%2C+Xinghua+Zhang%2C+Zefeng+Zhang%2C+and+Tingwen+Liu%2E+2025b%2E+S1%2Dbench%3A+A+simple+benchmark+for+evaluating+system+1+thinking+capability+of+large+reasoning+models%2E+Preprint%2C+arXiv%3A250)
64. Yutian Zhou, Tao Lei, Henry Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. 2022. Mixture-of-experts with expert choice routing. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Yutian+Zhou%2C+Tao+Lei%2C+Henry+Liu%2C+Nan+Du%2C+Yanping+Huang%2C+Vincent+Zhao%2C+Andrew+M+Dai%2C+Quoc+V+Le%2C+James+Laudon%2C+et+al%2E+2022%2E+Mixture%2Dof%2Dexperts+with+expert+choice+routing%2E+In+Advances+in+Neural+Informati)
65. Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. [Crowdsourcing multiple choice science questions](https://md.research.elamur.ai/free/en/papers/welbl-2017-crowdsourcing-multiple-choice-science-questions/). In Proceedings of the 3rd Workshop on Noisy User-generated Text, NUT@EMNLP 2017, Copenhagen, Denmark, September 7, 2017, pages 94–106. Association for Computational Linguistics.
66. X. Wu, S. Huang, and F. Wei. 2024. Multi-head mixture-of-experts. arXiv preprint arXiv:2404.15045. — [arXiv:2404.15045](https://arxiv.org/abs/2404.15045) [find](https://scholar.google.com/scholar?q=X%2E+Wu%2C+S%2E+Huang%2C+and+F%2E+Wei%2E+2024%2E+Multi%2Dhead+mixture%2Dof%2Dexperts%2E+arXiv+preprint+arXiv%3A2404%2E15045%2E+%E2%80%94+%5BarXiv%3A2404%2E15045%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2404%2E15045%29)
67. Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. 2023. Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning. arXiv preprint. [find](https://scholar.google.com/scholar?q=Mengzhou+Xia%2C+Tianyu+Gao%2C+Zhiyuan+Zeng%2C+and+Danqi+Chen%2E+2023%2E+Sheared+LLaMA%3A+Accelerating+Language+Model+Pre%2Dtraining+via+Structured+Pruning%2E+arXiv+preprint%2E)
68. F. Xue, Z. Zheng, Y. Fu, J. Ni, and W. Zhou. 2024. [Openmoe: An early effort on open mixture-of-experts language models](https://md.research.elamur.ai/free/en/papers/xue-2024-openmoe-early-effort-open-mixture/). arXiv preprint arXiv:2402.01739.
69. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages… [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Proceedings+of+the+57th+Conference+of+the+Association+for+Computa)
70. Jun Zhang, Desen Meng, Ji Qi, Zhenpeng Huang, Tao Wu, and Limin Wang. 2024. p-mod: Building mixture-of-depths mllms via progressive ratio decay. Preprint, arXiv:2412.04449. [find](https://scholar.google.com/scholar?q=Jun+Zhang%2C+Desen+Meng%2C+Ji+Qi%2C+Zhenpeng+Huang%2C+Tao+Wu%2C+and+Limin+Wang%2E+2024%2E+p%2Dmod%3A+Building+mixture%2Dof%2Ddepths+mllms+via+progressive+ratio+decay%2E+Preprint%2C+arXiv%3A2412%2E04449%2E)

*Updated 2 November 2026*