---
title: "TokenSkip: Controllable Chain-of-Thought — Xia 2025"
description: "Long chain-of-thought (CoT) outputs help LLMs reason better, but every extra generated token adds decoding time and key-value cache memory."
authors: ["Heming Xia", "Chak Tou Leong", "Wenjie Wang", "Yongqi Li", "Wenjie Li"]
year: 2025
venue: "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2025.emnlp-main.165/"
doi: "10.18653/v1/2025.emnlp-main.165"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xia-2025-tokenskip-controllable-chain-thought-compression/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xia-2025-tokenskip-controllable-chain-thought-compression/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/4/) / TokenSkip: Controllable Chain-of-Thought Compression in LLMs

# TokenSkip: Controllable Chain-of-Thought Compression in LLMs

**Authors:** [Heming Xia](https://md.research.elamur.ai/free/en/authors/heming-xia/), Chak Tou Leong, Wenjie Wang, [Yongqi Li](https://md.research.elamur.ai/free/en/authors/yongqi-li/), [Wenjie Li](https://md.research.elamur.ai/free/en/authors/wenjie-li/) · **Year:** 2025 · **Venue:** Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.emnlp-main.165/) · [PDF](https://aclanthology.org/2025.emnlp-main.165.pdf) · [DOI 10.18653/v1/2025.emnlp-main.165](https://doi.org/10.18653/v1/2025.emnlp-main.165) · [OpenAlex](https://openalex.org/W4416036941)

**Citations:** 9 citations (openalex-snapshot · 2026-10-14) · [96](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Long chain-of-thought (CoT) outputs help LLMs reason better, but every extra generated token adds decoding time and key-value cache memory. TokenSkip asks whether every token in a reasoning trace actually matters. The authors first score how important each CoT token is using the LLMLingua-2 compressor, a small bidirectional model trained on token labels produced by GPT-4. The scores show that equations and numbers carry most of the meaning, while connectors such as "so" or "since" carry little. To build training data, they generate CoTs with the target model and remove low-importance tokens at several sampled compression ratios. They then fine-tune the model with LoRA on these shortened traces, with the desired ratio given in the input. At inference time, the model produces compressed reasoning at a ratio the user chooses. On GSM8K and MATH-500, with LLaMA-3.1-8B-Instruct and Qwen2.5-Instruct models, token usage drops sharply and accuracy barely falls. This gives a cheap, adjustable way to trade some accuracy for faster reasoning.

## Key points
- **Qwen2.5-14B-Instruct on GSM8K:** average reasoning length falls from 313 to 181 tokens (about 40% fewer), and accuracy drops by less than 0.4%.
- **LLaMA-3.1-8B-Instruct on MATH-500:** CoT is about 30% shorter, accuracy drops by under 4%, and inference is roughly 1.4× faster.
- **Low training cost:** LoRA updates only about 0.2% of the 14B model's parameters. Training uses 7,473 GSM8K and 7,500 MATH examples and takes about 2 hours (7B) or 2.5 hours (14B) on two 3090 GPUs.
- **Readability:** the authors show that LLaMA-3.1-8B-Instruct can turn the compressed CoTs back into full step-by-step reasoning, so the shortened traces remain interpretable.
- **Limitations:** the LLMLingua-2 compressor was not trained on math data. Larger models (Qwen2.5-32B/72B) and long-CoT models such as QwQ-32B-Preview were not tested. Code and checkpoints are available at github.com/hemingkx/TokenSkip.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

96 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 93 not yet.

- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2025-towards-reasoning-era-survey-long/)
- [New Trends for Modern Machine Translation with Large Reasoning Models](https://md.research.elamur.ai/free/en/papers/liu-2026-new-trends-modern-machine-translation/)
- [Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models](https://md.research.elamur.ai/free/en/papers/sui-2025-stop-overthinking-survey-efficient-reasoning/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{xia2025tokenskip,
  title = {{TokenSkip: Controllable Chain-of-Thought Compression in LLMs}},
  author = {Xia, Heming and Leong, Chak Tou and Wang, Wenjie and Li, Yongqi and Li, Wenjie},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.165},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.165}
}
```

- [← previous Thematic Dispersion in Arabic Applied Linguistics: A Bibliometric Analysis using Brookes' Measure](https://md.research.elamur.ai/free/en/papers/eddakrouri-2025-thematic-dispersion-arabic-applied-linguistics/)
- [next → Towards a Unified Benchmark for Arabic Pronunciation Assessment: Quranic Recitation as Case Study](https://md.research.elamur.ai/free/en/papers/kheir-2025-towards-unified-benchmark-arabic-pronunciation/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/4/)

## References

48 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. 2024. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In Computer Vision - ECCV 2024 -18th European Conference, Milan, Italy, September 29-Oct… [find](https://scholar.google.com/scholar?q=Liang+Chen%2C+Haozhe+Zhao%2C+Tianyu+Liu%2C+Shuai+Bai%2C+Junyang+Lin%2C+Chang+Zhou%2C+and+Baobao+Chang%2E+2024%2E+An+image+is+worth+1%2F2+tokens+after+layer+2%3A+Plug%2Dand%2Dplay+inference+acceleration+for+large+vision%2Dlangu) [our card](https://md.research.elamur.ai/free/en/papers/chen-2024-image-worth-1-2-tokens/)
2. Jeffrey Cheng and Benjamin Van Durme. 2024. Compressed chain of thought: Efficient reasoning through dense representations. arXiv preprint arXiv:2412.13171. [find](https://scholar.google.com/scholar?q=Jeffrey+Cheng+and+Benjamin+Van+Durme%2E+2024%2E+Compressed+chain+of+thought%3A+Efficient+reasoning+through+dense+representations%2E+arXiv+preprint+arXiv%3A2412%2E13171%2E)
3. Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. 2023. Adapting language models to compress contexts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3829–3846, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alexis+Chevalier%2C+Alexander+Wettig%2C+Anirudh+Ajith%2C+and+Danqi+Chen%2E+2023%2E+Adapting+language+models+to+compress+contexts%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+Methods+in+Natural+Language+P) [our card](https://md.research.elamur.ai/free/en/papers/chevalier-2023-adapting-language-models-compress-contexts/)
4. Yu-Neng Chuang, Tianwei Xing, Chia-Yuan Chang, Zirui Liu, Xun Chen, and Xia Hu. 2024. Learning to compress prompt in natural language formats. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1:… [find](https://scholar.google.com/scholar?q=Yu%2DNeng+Chuang%2C+Tianwei+Xing%2C+Chia%2DYuan+Chang%2C+Zirui+Liu%2C+Xun+Chen%2C+and+Xia+Hu%2E+2024%2E+Learning+to+compress+prompt+in+natural+language+formats%2E+In+Proceedings+of+the+2024+Conference+of+the+North+Americ)
5. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. CoRR, abs/2110.14168. [find](https://scholar.google.com/scholar?q=Karl+Cobbe%2C+Vineet+Kosaraju%2C+Mohammad+Bavarian%2C+Mark+Chen%2C+Heewoo+Jun%2C+Lukasz+Kaiser%2C+Matthias+Plappert%2C+Jerry+Tworek%2C+Jacob+Hilton%2C+Reiichiro+Nakano%2C+Christopher+Hesse%2C+and+John+Schulman%2E+2021%2E+Train) [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/)
6. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, et al. 2025. Deepseekr1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948. [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2C+Daya+Guo%2C+Dejian+Yang%2C+Haowei+Zhang%2C+Junxiao+Song%2C+Ruoyu+Zhang%2C+Runxin+Xu%2C+Qihao+Zhu%2C+Shirong+Ma%2C+Peiyi+Wang%2C+et+al%2E+2025%2E+Deepseekr1%3A+Incentivizing+reasoning+capability+in+llms+via+reinf) [our card](https://md.research.elamur.ai/free/en/papers/deepseek-2025-ai-deepseek-r1-incentivizing-reasoning-capability/)
7. Yuntian Deng, Yejin Choi, and Stuart Shieber. 2024. From explicit cot to implicit cot: Learning to internalize cot step by step. arXiv preprint arXiv:2405.14838. [find](https://scholar.google.com/scholar?q=Yuntian+Deng%2C+Yejin+Choi%2C+and+Stuart+Shieber%2E+2024%2E+From+explicit+cot+to+implicit+cot%3A+Learning+to+internalize+cot+step+by+step%2E+arXiv+preprint+arXiv%3A2405%2E14838%2E)
8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+N)
9. Mengru Ding, Hanmeng Liu, Zhizhang Fu, Jian Song, Wenbo Xie, and Yue Zhang. 2024. Break the chain: Large language models can be shortcut reasoners. CoRR, abs/2406.06580. — [arXiv:2406.06580](https://arxiv.org/abs/2406.06580) [find](https://scholar.google.com/scholar?q=Mengru+Ding%2C+Hanmeng+Liu%2C+Zhizhang+Fu%2C+Jian+Song%2C+Wenbo+Xie%2C+and+Yue+Zhang%2E+2024%2E+Break+the+chain%3A+Large+language+models+can+be+shortcut+reasoners%2E+CoRR%2C+abs%2F2406%2E06580%2E+%E2%80%94+%5BarXiv%3A2406%2E06580%5D%28https%3A%2F%2Fa)
10. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, et al. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783. [find](https://scholar.google.com/scholar?q=Abhimanyu+Dubey%2C+Abhinav+Jauhri%2C+Abhinav+Pandey%2C+Abhishek+Kadian%2C+Ahmad+Al%2DDahle%2C+et+al%2E+2024%2E+The+llama+3+herd+of+models%2E+Preprint%2C+arXiv%3A2407%2E21783%2E)
11. Tao Ge, Hu Jing, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. 2024. In-context autoencoder for context compression in a large language model. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Tao+Ge%2C+Hu+Jing%2C+Lei+Wang%2C+Xun+Wang%2C+Si%2DQing+Chen%2C+and+Furu+Wei%2E+2024%2E+In%2Dcontext+autoencoder+for+context+compression+in+a+large+language+model%2E+In+The+Twelfth+International+Conference+on+Learning+Rep)
12. Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. 2024. Think before you speak: Training language models with pause tokens. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Sachin+Goyal%2C+Ziwei+Ji%2C+Ankit+Singh+Rawat%2C+Aditya+Krishna+Menon%2C+Sanjiv+Kumar%2C+and+Vaishnavh+Nagarajan%2E+2024%2E+Think+before+you+speak%3A+Training+language+models+with+pause+tokens%2E+In+The+Twelfth+Interna)
13. Tingxu Han, Chunrong Fang, Shiyu Zhao, Shiqing Ma, Zhenyu Chen, and Zhenting Wang. 2024. Token-budget-aware llm reasoning. arXiv preprint arXiv:2412.18547. [find](https://scholar.google.com/scholar?q=Tingxu+Han%2C+Chunrong+Fang%2C+Shiyu+Zhao%2C+Shiqing+Ma%2C+Zhenyu+Chen%2C+and+Zhenting+Wang%2E+2024%2E+Token%2Dbudget%2Daware+llm+reasoning%2E+arXiv+preprint+arXiv%3A2412%2E18547%2E) [our card](https://md.research.elamur.ai/free/en/papers/han-2024-token-budget-aware-llm-reasoning/)
14. Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. 2024. Training large language models to reason in a continuous latent space. arXiv preprint arXiv:2412.06769. [find](https://scholar.google.com/scholar?q=Shibo+Hao%2C+Sainbayar+Sukhbaatar%2C+DiJia+Su%2C+Xian+Li%2C+Zhiting+Hu%2C+Jason+Weston%2C+and+Yuandong+Tian%2E+2024%2E+Training+large+language+models+to+reason+in+a+continuous+latent+space%2E+arXiv+preprint+arXiv%3A2412%2E)
15. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a. Measuring massive multitask language understanding. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks%2C+Collin+Burns%2C+Steven+Basart%2C+Andy+Zou%2C+Mantas+Mazeika%2C+Dawn+Song%2C+and+Jacob+Steinhardt%2E+2021a%2E+Measuring+massive+multitask+language+understanding%2E+In+International+Conference+on+Learnin) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/)
16. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). [find](https://scholar.google.com/scholar?q=Dan+Hendrycks%2C+Collin+Burns%2C+Saurav+Kadavath%2C+Akul+Arora%2C+Steven+Basart%2C+Eric+Tang%2C+Dawn+Song%2C+and+Jacob+Steinhardt%2E+2021b%2E+Measuring+mathematical+problem+solving+with+the+MATH+dataset%2E+In+Thirty%2Dfift) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/)
17. Le Hou, Richard Yuanzhe Pang, Tianyi Zhou, Yuexin Wu, Xinying Song, Xiaodan Song, and Denny Zhou. 2022. Token dropping for efficient BERT pretraining. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3774–3784, Dublin, Ireland.… [find](https://scholar.google.com/scholar?q=Le+Hou%2C+Richard+Yuanzhe+Pang%2C+Tianyi+Zhou%2C+Yuexin+Wu%2C+Xinying+Song%2C+Xiaodan+Song%2C+and+Denny+Zhou%2E+2022%2E+Token+dropping+for+efficient+BERT+pretraining%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+)
18. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. [find](https://scholar.google.com/scholar?q=Edward+J%2E+Hu%2C+Yelong+Shen%2C+Phillip+Wallis%2C+Zeyuan+Allen%2DZhu%2C+Yuanzhi+Li%2C+Shean+Wang%2C+Lu+Wang%2C+and+Weizhu+Chen%2E+2022%2E+Lora%3A+Low%2Drank+adaptation+of+large+language+models%2E+In+The+Tenth+International+Conf)
19. Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. LLMLingua: Compressing prompts for accelerated inference of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13358–13376, Singapore. Association for Compu… [find](https://scholar.google.com/scholar?q=Huiqiang+Jiang%2C+Qianhui+Wu%2C+Chin%2DYew+Lin%2C+Yuqing+Yang%2C+and+Lili+Qiu%2E+2023%2E+LLMLingua%3A+Compressing+prompts+for+accelerated+inference+of+large+language+models%2E+In+Proceedings+of+the+2023+Conference+on+E) [our card](https://md.research.elamur.ai/free/en/papers/jiang-2023-llmlingua-compressing-prompts-accelerated-inference/)
20. Mingyu Jin, Qinkai Yu, Dong Shu, Haiyan Zhao, Wenyue Hua, Yanda Meng, Yongfeng Zhang, and Mengnan Du. 2024. The impact of reasoning step length on large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 1830–1842, Bangkok, Thailand. Association for Comput… [find](https://scholar.google.com/scholar?q=Mingyu+Jin%2C+Qinkai+Yu%2C+Dong+Shu%2C+Haiyan+Zhao%2C+Wenyue+Hua%2C+Yanda+Meng%2C+Yongfeng+Zhang%2C+and+Mengnan+Du%2E+2024%2E+The+impact+of+reasoning+step+length+on+large+language+models%2E+In+Findings+of+the+Association) [our card](https://md.research.elamur.ai/free/en/papers/jin-2024-impact-reasoning-step-length-large/)
21. Yu Kang, Xianghui Sun, Liangyu Chen, and Wei Zou. 2024. C3ot: Generating shorter chain-of-thought without compromising effectiveness. arXiv preprint arXiv:2412.11664. [find](https://scholar.google.com/scholar?q=Yu+Kang%2C+Xianghui+Sun%2C+Liangyu+Chen%2C+and+Wei+Zou%2E+2024%2E+C3ot%3A+Generating+shorter+chain%2Dof%2Dthought+without+compromising+effectiveness%2E+arXiv+preprint+arXiv%3A2412%2E11664%2E)
22. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, Novemb… [find](https://scholar.google.com/scholar?q=Takeshi+Kojima%2C+Shixiang+Shane+Gu%2C+Machel+Reid%2C+Yutaka+Matsuo%2C+and+Yusuke+Iwasawa%2E+2022%2E+Large+language+models+are+zero%2Dshot+reasoners%2E+In+Advances+in+Neural+Information+Processing+Systems+35%3A+Annual+) [our card](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/)
23. Ayeong Lee, Ethan Che, and Tianyi Peng. 2025. How well do llms compress their own chain-of-thought? A token complexity approach. CoRR, abs/2503.01141. — [arXiv:2503.01141](https://arxiv.org/abs/2503.01141) [find](https://scholar.google.com/scholar?q=Ayeong+Lee%2C+Ethan+Che%2C+and+Tianyi+Peng%2E+2025%2E+How+well+do+llms+compress+their+own+chain%2Dof%2Dthought%3F+A+token+complexity+approach%2E+CoRR%2C+abs%2F2503%2E01141%2E+%E2%80%94+%5BarXiv%3A2503%2E01141%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2503%2E0)
24. Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. 2023. Compressing context to enhance inference efficiency of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6342–6353, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yucheng+Li%2C+Bo+Dong%2C+Frank+Guerin%2C+and+Chenghua+Lin%2E+2023%2E+Compressing+context+to+enhance+inference+efficiency+of+large+language+models%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+Methods+in+N) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-compressing-context-enhance-inference-efficiency/)
25. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let’s verify step by step. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenRevi… [find](https://scholar.google.com/scholar?q=Hunter+Lightman%2C+Vineet+Kosaraju%2C+Yuri+Burda%2C+Harrison+Edwards%2C+Bowen+Baker%2C+Teddy+Lee%2C+Jan+Leike%2C+John+Schulman%2C+Ilya+Sutskever%2C+and+Karl+Cobbe%2E+2024%2E+Let%E2%80%99s+verify+step+by+step%2E+In+The+Twelfth+Intern) [our card](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/)
26. Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, yelong shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, and Weizhu Chen. 2024. Not all tokens are what you need for pretraining. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Zhenghao+Lin%2C+Zhibin+Gou%2C+Yeyun+Gong%2C+Xiao+Liu%2C+yelong+shen%2C+Ruochen+Xu%2C+Chen+Lin%2C+Yujiu+Yang%2C+Jian+Jiao%2C+Nan+Duan%2C+and+Weizhu+Chen%2E+2024%2E+Not+all+tokens+are+what+you+need+for+pretraining%2E+In+The+Thir)
27. Tengxiao Liu, Qipeng Guo, Xiangkun Hu, Cheng Jiayang, Yue Zhang, Xipeng Qiu, and Zheng Zhang. 2024. Can language models learn to skip steps? In The Thirty-eighth Annual Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Tengxiao+Liu%2C+Qipeng+Guo%2C+Xiangkun+Hu%2C+Cheng+Jiayang%2C+Yue+Zhang%2C+Xipeng+Qiu%2C+and+Zheng+Zhang%2E+2024%2E+Can+language+models+learn+to+skip+steps%3F+In+The+Thirty%2Deighth+Annual+Conference+on+Neural+Informatio)
28. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. Open-Review.net. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2019%2E+Decoupled+weight+decay+regularization%2E+In+7th+International+Conference+on+Learning+Representations%2C+ICLR+2019%2C+New+Orleans%2C+LA%2C+USA%2C+May+6%2D9%2C+2019%2E+Open%2DReview%2E)
29. Yiran Ma, Zui Chen, Tianqiao Liu, Mi Tian, Zhuo Liu, Zitao Liu, and Weiqi Luo. 2024. What are step-level reward models rewarding? counterintuitive findings from mcts-boosted mathematical reasoning. CoRR, abs/2412.15904. [find](https://scholar.google.com/scholar?q=Yiran+Ma%2C+Zui+Chen%2C+Tianqiao+Liu%2C+Mi+Tian%2C+Zhuo+Liu%2C+Zitao+Liu%2C+and+Weiqi+Luo%2E+2024%2E+What+are+step%2Dlevel+reward+models+rewarding%3F+counterintuitive+findings+from+mcts%2Dboosted+mathematical+reasoning%2E+Co)
30. Emanuele Marconato, Stefano Teso, Antonio Vergari, and Andrea Passerini. 2024. Not all neuro-symbolic concepts are created equal: Analysis and mitigation of reasoning shortcuts. Advances in Neural Information Processing Systems, 36. [find](https://scholar.google.com/scholar?q=Emanuele+Marconato%2C+Stefano+Teso%2C+Antonio+Vergari%2C+and+Andrea+Passerini%2E+2024%2E+Not+all+neuro%2Dsymbolic+concepts+are+created+equal%3A+Analysis+and+mitigation+of+reasoning+shortcuts%2E+Advances+in+Neural+Inf)
31. William Merrill and Ashish Sabharwal. 2024. The expressive power of transformers with chain of thought. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. [find](https://scholar.google.com/scholar?q=William+Merrill+and+Ashish+Sabharwal%2E+2024%2E+The+expressive+power+of+transformers+with+chain+of+thought%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2C+ICLR+2024%2C+Vienna%2C+Austria%2C)
32. Amirkeivan Mohtashami and Martin Jaggi. 2023. Random-access infinite context length for transformers. In Thirty-seventh Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Amirkeivan+Mohtashami+and+Martin+Jaggi%2E+2023%2E+Random%2Daccess+infinite+context+length+for+transformers%2E+In+Thirty%2Dseventh+Conference+on+Neural+Information+Processing+Systems%2E)
33. Xuefei Ning, Zinan Lin, Zixuan Zhou, Zifu Wang, Huazhong Yang, and Yu Wang. 2023. Skeleton-of-thought: Large language models can do parallel decoding. Proceedings ENLSP-III. [find](https://scholar.google.com/scholar?q=Xuefei+Ning%2C+Zinan+Lin%2C+Zixuan+Zhou%2C+Zifu+Wang%2C+Huazhong+Yang%2C+and+Yu+Wang%2E+2023%2E+Skeleton%2Dof%2Dthought%3A+Large+language+models+can+do+parallel+decoding%2E+Proceedings+ENLSP%2DIII%2E)
34. Maxwell I. Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. Show your work: Scratchpads for intermediate computation with language models. CoRR, abs/2112.00114… [find](https://scholar.google.com/scholar?q=Maxwell+I%2E+Nye%2C+Anders+Johan+Andreassen%2C+Guy+Gur%2DAri%2C+Henryk+Michalewski%2C+Jacob+Austin%2C+David+Bieber%2C+David+Dohan%2C+Aitor+Lewkowycz%2C+Maarten+Bosma%2C+David+Luan%2C+Charles+Sutton%2C+and+Augustus+Odena%2E+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/al-2021-show-your-work-scratchpads-intermediate/)
35. OpenAI. 2023. GPT-4 technical report. abs/2303.08774. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2023%2E+GPT%2D4+technical+report%2E+abs%2F2303%2E08774%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
36. CoRR, OpenAI. 2024. Learning to reason with llms. [find](https://scholar.google.com/scholar?q=CoRR%2C+OpenAI%2E+2024%2E+Learning+to+reason+with+llms%2E)
37. OpenAI et al. 2024. Openai o1 system card. Preprint, arXiv:2412.16720. [find](https://scholar.google.com/scholar?q=OpenAI+et+al%2E+2024%2E+Openai+o1+system+card%2E+Preprint%2C+arXiv%3A2412%2E16720%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2024-openai-o1-system-card/)
38. Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. 2024. LLMLingua-2: Data distillation for efficient and faithful task-agnostic prompt compression. In Findings of the Associat… [find](https://scholar.google.com/scholar?q=Zhuoshi+Pan%2C+Qianhui+Wu%2C+Huiqiang+Jiang%2C+Menglin+Xia%2C+Xufang+Luo%2C+Jue+Zhang%2C+Qingwei+Lin%2C+Victor+R%C3%BChle%2C+Yuqing+Yang%2C+Chin%2DYew+Lin%2C+H%2E+Vicky+Zhao%2C+Lili+Qiu%2C+and+Dongmei+Zhang%2E+2024%2E+LLMLingua%2D2%3A+Data+d) [our card](https://md.research.elamur.ai/free/en/papers/pan-2024-llmlingua-2-data-distillation-efficient/)
39. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans,… [find](https://scholar.google.com/scholar?q=Noah+Shinn%2C+Federico+Cassano%2C+Ashwin+Gopinath%2C+Karthik+Narasimhan%2C+and+Shunyu+Yao%2E+2023%2E+Reflexion%3A+language+agents+with+verbal+reinforcement+learning%2E+In+Advances+in+Neural+Information+Processing+Sys) [our card](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/)
40. Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2025. Scaling test-time compute optimally can be more effective than scaling LLM parameters. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Charlie+Victor+Snell%2C+Jaehoon+Lee%2C+Kelvin+Xu%2C+and+Aviral+Kumar%2E+2025%2E+Scaling+test%2Dtime+compute+optimally+can+be+more+effective+than+scaling+LLM+parameters%2E+In+The+Thirteenth+International+Conference+)
41. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [find](https://scholar.google.com/scholar?q=Alon+Talmor%2C+Jonathan+Herzig%2C+Nicholas+Lourie%2C+and+Jonathan+Berant%2E+2019%2E+CommonsenseQA%3A+A+question+answering+challenge+targeting+commonsense+knowledge%2E+In+Proceedings+of+the+2019+Conference+of+the+No)
42. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5,… [find](https://scholar.google.com/scholar?q=Xuezhi+Wang%2C+Jason+Wei%2C+Dale+Schuurmans%2C+Quoc+V%2E+Le%2C+Ed+H%2E+Chi%2C+Sharan+Narang%2C+Aakanksha+Chowdhery%2C+and+Denny+Zhou%2E+2023%2E+Self%2Dconsistency+improves+chain+of+thought+reasoning+in+language+models%2E+In+Th) [our card](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/)
43. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processi… [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Xuezhi+Wang%2C+Dale+Schuurmans%2C+Maarten+Bosma%2C+Brian+Ichter%2C+Fei+Xia%2C+Ed+H%2E+Chi%2C+Quoc+V%2E+Le%2C+and+Denny+Zhou%2E+2022%2E+Chain%2Dof%2Dthought+prompting+elicits+reasoning+in+large+language+models%2E+In+Ad) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
44. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Juny… [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chang+Zhou%2C+Chengpeng+Li%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Guanting+Dong%2C+Haoran+Wei%2C+Huan+Lin%2C+Jialong+Tang%2C+Jialin+Wang%2C+Jian+Yang%2C+Jian)
45. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023,… [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Tom+Griffiths%2C+Yuan+Cao%2C+and+Karthik+Narasimhan%2E+2023%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2E+In+Advances+in+Neural+)
46. Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Re, Clark Barrett, Zhangyang Wang, and Beidi Chen. 2023. H2o: Heavy-hitter oracle for efficient generative inference of large language models. In Thirty-seventh Conference on Neural… [find](https://scholar.google.com/scholar?q=Zhenyu+Zhang%2C+Ying+Sheng%2C+Tianyi+Zhou%2C+Tianlong+Chen%2C+Lianmin+Zheng%2C+Ruisi+Cai%2C+Zhao+Song%2C+Yuandong+Tian%2C+Christopher+Re%2C+Clark+Barrett%2C+Zhangyang+Wang%2C+and+Beidi+Chen%2E+2023%2E+H2o%3A+Heavy%2Dhitter+oracle+)
47. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. 2024. LlamaFactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 400– 410, Bangkok, Thai… [find](https://scholar.google.com/scholar?q=Yaowei+Zheng%2C+Richong+Zhang%2C+Junhao+Zhang%2C+Yanhan+Ye%2C+and+Zheyan+Luo%2E+2024%2E+LlamaFactory%3A+Unified+efficient+fine%2Dtuning+of+100%2B+language+models%2E+In+Proceedings+of+the+62nd+Annual+Meeting+of+the+Associ) [our card](https://md.research.elamur.ai/free/en/papers/zheng-2024-llamafactory-unified-efficient-fine-tuning/)
48. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V. Le, and Ed H. Chi. 2023. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representatio… [find](https://scholar.google.com/scholar?q=Denny+Zhou%2C+Nathanael+Sch%C3%A4rli%2C+Le+Hou%2C+Jason+Wei%2C+Nathan+Scales%2C+Xuezhi+Wang%2C+Dale+Schuurmans%2C+Claire+Cui%2C+Olivier+Bousquet%2C+Quoc+V%2E+Le%2C+and+Ed+H%2E+Chi%2E+2023%2E+Least%2Dto%2Dmost+prompting+enables+complex+re) [our card](https://md.research.elamur.ai/free/en/papers/al-2022-least-most-prompting-enables-complex/)