---
title: "TRAC — paper by Wu et al. (2026) — summary and ideas"
description: "This matters because it offers noise-robust dense supervision without training a dedicated reward model. 55 references."
authors: ["Sitong Wu", "Haoru Tan", "Xichen Zhang", "Bin Xia", "Wenhu Zhang", "Xiaojuan Qi", "Bei Yu", "Jiaya Jia"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.2210/"
doi: "10.18653/v1/2026.acl-long.2210"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wu-2026-trac-teacher-guided-token-reward/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wu-2026-trac-teacher-guided-token-reward/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/54/) / TRAC: Teacher-Guided Token Reward with Adaptive Calibration for Robust Policy Optimization

# TRAC: Teacher-Guided Token Reward with Adaptive Calibration for Robust Policy Optimization

**Authors:** [Sitong Wu](https://md.research.elamur.ai/free/en/authors/sitong-wu/), [Haoru Tan](https://md.research.elamur.ai/free/en/authors/haoru-tan/), [Xichen Zhang](https://md.research.elamur.ai/free/en/authors/xichen-zhang/), Bin Xia, [Wenhu Zhang](https://md.research.elamur.ai/free/en/authors/wenhu-zhang/), Xiaojuan Qi, [Bei Yu](https://md.research.elamur.ai/free/en/authors/bei-yu/) and [Jiaya Jia](https://md.research.elamur.ai/free/en/authors/jiaya-jia/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.2210/) · [PDF](https://aclanthology.org/2026.acl-long.2210.pdf) · [DOI 10.18653/v1/2026.acl-long.2210](https://doi.org/10.18653/v1/2026.acl-long.2210) · [OpenAlex](https://openalex.org/W7166832884)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Reinforcement learning for LLM reasoning usually depends on a single correctness signal at the end of a solution. This makes it hard to tell which tokens actually helped. A cheaper option is to score each token by comparing a stronger teacher LLM's log-probability with the student's. However, the authors show that teachers such as Qwen3-8B/32B are unreliable. They fail some training problems, sometimes rate wrong student solutions above correct ones, and are uncertain at certain tokens.

TRAC addresses this by scaling each teacher-derived token reward with a reliability weight between 0 and 1. The weight is the product of three factors:
- how well the teacher solves the problem,
- how well it separates correct from incorrect student trajectories,
- how confident it is at that token, measured by entropy.

If any one factor is low, the signal is suppressed. In TRAC-GRPO, this calibrated reward is added to the GRPO advantage. On math benchmarks, the method beats GRPO, DAPO and other dense-reward methods. It also transfers to out-of-domain tasks and adds little training cost. This matters because it offers noise-robust dense supervision without training a dedicated reward model.

## Key points
- **AIME24 gains:** TRAC-GRPO improves on GRPO by +7.8 with DeepSeek-R1-Distill-Qwen-1.5B and by +5.5 with OpenMath-Nemotron-1.5B.
- **Training speed (Qwen3-8B-Base):** one step takes 210 s, compared with 185 s for GRPO and 371 s for DAPO. Token-level rewards create differences between trajectories, so the rejection sampling DAPO relies on is not needed.
- **Continued training:** DeepScaleR-1.5B-Preview was trained further on DeepScaleR-40K, the dataset on which it had already converged with GRPO. TRAC-GRPO still added +10.8 on Minerva, +11.4 on AMC23 and +3.5 on AIME24, while GRPO stalled.
- **Out-of-domain results:** compared with GRPO, TRAC-GRPO scores +2.8 on GPQA and +1.2 on RiddleSense.
- **Limitations:** validation covers only mathematical reasoning, and the datasets used are mostly in English. The code is announced at github.com/JIA-Lab-research/TRAC.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Hong Kong University of Science and Technology](https://md.research.elamur.ai/free/en/universities/hong-kong-university-of-science-and-technology/) — University, Hong Kong, China · 170 works · 26,341 citations
- [Chinese University of Hong Kong](https://md.research.elamur.ai/free/en/universities/chinese-university-of-hong-kong/) — University, Hong Kong, China · 159 works · 16,154 citations
- [University of Hong Kong](https://md.research.elamur.ai/free/en/universities/university-of-hong-kong/) — University, Pok Fu Lam, China · 118 works · 13,496 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

## Cite

Cite as: Sitong Wu, Haoru Tan, Xichen Zhang, Bin Xia, Wenhu Zhang, Xiaojuan Qi, Bei Yu and Jiaya Jia. 2026. TRAC: Teacher-Guided Token Reward with Adaptive Calibration for Robust Policy Optimization. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wu2026trac,
  title = {{TRAC: Teacher-Guided Token Reward with Adaptive Calibration for Robust Policy Optimization}},
  author = {Sitong Wu, Haoru Tan, Xichen Zhang, Bin Xia, Wenhu Zhang, Xiaojuan Qi, Bei Yu and Jiaya Jia},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2210},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2210}
}
```

- [← previous TPS-Bench: Evaluating AI Agents' Tool Planning & Scheduling Abilities in Compounding Tasks](https://md.research.elamur.ai/free/en/papers/xu-2026-tps-bench-evaluating-ai-agents/)
- [next → TRACE: An Experiential Framework for Coherent Multi-hop Knowledge Graph Question Answering](https://md.research.elamur.ai/free/en/papers/wang-2026-trace-experiential-framework-coherent-multi/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/54/)

## References

55 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. AIME. 2024. American invitational mathematics examination. [find](https://scholar.google.com/scholar?q=AIME%2E+2024%2E+American+invitational+mathematics+examination%2E)
2. Anthropic. 2025. Introducing claude sonnet 4.5. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2025%2E+Introducing+claude+sonnet+4%2E5%2E)
3. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
4. Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Yuchen Zhang, Jiacheng Chen, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, and 1 others. 2025. [Process reinforcement through implicit rewards](https://md.research.elamur.ai/free/en/papers/cui-2025-process-reinforcement-through-implicit-rewards/). arXiv preprint arXiv:2502.01456.
5. Google DeepMind. 2025. Gemini 3 pro. [find](https://scholar.google.com/scholar?q=Google+DeepMind%2E+2025%2E+Gemini+3+pro%2E)
6. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv e-prints, pages arXiv–2407.
7. Wu Fei, Hao Kong, Shuxian Liang, Yang Lin, Yibo Yang, Jing Tang, Lei Chen, and Xiansheng Hua. 2025. Self-guided process reward optimization with redefined step-wise advantage for process reinforcement learning. arXiv e-prints, pages arXiv–2507. [find](https://scholar.google.com/scholar?q=Wu+Fei%2C+Hao+Kong%2C+Shuxian+Liang%2C+Yang+Lin%2C+Yibo+Yang%2C+Jing+Tang%2C+Lei+Chen%2C+and+Xiansheng+Hua%2E+2025%2E+Self%2Dguided+process+reward+optimization+with+redefined+step%2Dwise+advantage+for+process+reinforcement)
8. Chinese GaoKao. 2024. Gaokao2023en. \#\# Ethical Considerations AMC. 2023. American mathematics competitions. [find](https://scholar.google.com/scholar?q=Chinese+GaoKao%2E+2024%2E+Gaokao2023en%2E+%5C%23%5C%23+Ethical+Considerations+AMC%2E+2023%2E+American+mathematics+competitions%2E)
9. Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michał Podstawski, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. 2024. Graph of Thoughts: Solving Elaborate Problems with Large Language Models. Proceedings of the AAAI Conference… [find](https://scholar.google.com/scholar?q=Maciej+Besta%2C+Nils+Blach%2C+Ales+Kubicek%2C+Robert+Gerstenberger%2C+Lukas+Gianinazzi%2C+Joanna+Gajda%2C+Tomasz+Lehmann%2C+Micha%C5%82+Podstawski%2C+Hubert+Niewiadomski%2C+Piotr+Nyczyk%2C+and+Torsten+Hoefler%2E+2024%2E+Graph+of+)
10. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948.
11. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, and 1 others. 2024. [Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems](https://md.research.elamur.ai/free/en/papers/he-2024-olympiadbench-challenging-benchmark-promoting-agi/). arXiv preprint arXiv:2402.14008.
12. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). arXiv preprint arXiv:2103.03874.
13. Jian Hu. 2025. [Reinforce++: A simple and efficient approach for aligning large language models](https://md.research.elamur.ai/free/en/papers/hu-2025-reinforce-stabilizing-critic-free-policy/). arXiv preprint arXiv:2501.03262.
14. Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. 2025. [Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model](https://md.research.elamur.ai/free/en/papers/hu-2025-open-reasoner-zero-open-source/). arXiv preprint arXiv:2503.24290.
15. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. [Livecodebench: Holistic and contamination free evaluation of large language models for code](https://md.research.elamur.ai/free/en/papers/jain-2024-livecodebench-holistic-contamination-free-evaluation/). arXiv preprint arXiv:2403.07974.
16. Alan Lee and Harry Tong. 2025. Token-efficient rl for llm reasoning. arXiv preprint arXiv:2504.20834. [find](https://scholar.google.com/scholar?q=Alan+Lee+and+Harry+Tong%2E+2025%2E+Token%2Defficient+rl+for+llm+reasoning%2E+arXiv+preprint+arXiv%3A2504%2E20834%2E)
17. Jung Hyun Lee, June Yong Yang, Byeongho Heo, Dongyoon Han, Kyungsu Kim, Eunho Yang, and Kang Min Yoo. 2024. Token-supervised value models for enhancing mathematical problem-solving capabilities of large language models. arXiv preprint arXiv:2407.12863. — [arXiv:2407.12863](https://arxiv.org/abs/2407.12863) [find](https://scholar.google.com/scholar?q=Jung+Hyun+Lee%2C+June+Yong+Yang%2C+Byeongho+Heo%2C+Dongyoon+Han%2C+Kyungsu+Kim%2C+Eunho+Yang%2C+and+Kang+Min+Yoo%2E+2024%2E+Token%2Dsupervised+value+models+for+enhancing+mathematical+problem%2Dsolving+capabilities+of+lar)
18. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, and 1 others. 2022. [Solving quantitative reasoning problems with language models](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/). Advances in neural information processing systems, 35:3843–3857…
19. Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. 2023a. [Generative judge for evaluating alignment](https://md.research.elamur.ai/free/en/papers/li-2023-generative-judge-evaluating-alignment/). Preprint, arXiv:2310.05470.
20. Xuefeng Li, Haoyang Zou, and Pengfei Liu. 2025. Limr: Less is more for rl scaling. Preprint, arXiv:2502.11886. [find](https://scholar.google.com/scholar?q=Xuefeng+Li%2C+Haoyang+Zou%2C+and+Pengfei+Liu%2E+2025%2E+Limr%3A+Less+is+more+for+rl+scaling%2E+Preprint%2C+arXiv%3A2502%2E11886%2E)
21. Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. 2023b. Making large language models better reasoners with step-aware verifier. Preprint, arXiv:2206.02336. — [arXiv:2206.02336](https://arxiv.org/abs/2206.02336) [find](https://scholar.google.com/scholar?q=Yifei+Li%2C+Zeqi+Lin%2C+Shizhuo+Zhang%2C+Qiang+Fu%2C+Bei+Chen%2C+Jian%2DGuang+Lou%2C+and+Weizhu+Chen%2E+2023b%2E+Making+large+language+models+better+reasoners+with+step%2Daware+verifier%2E+Preprint%2C+arXiv%3A2206%2E02336%2E+%E2%80%94+%5Bar)
22. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. [Let’s verify step by step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/). Preprint, arXiv:2305.20050.
23. Bill Yuchen Lin, Ziyi Wu, Yichi Yang, Dong-Ho Lee, and Xiang Ren. 2021. Riddlesense: Reasoning about riddle questions featuring linguistic creativity and commonsense knowledge. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL-IJCNLP 2021): Findings. To… [find](https://scholar.google.com/scholar?q=Bill+Yuchen+Lin%2C+Ziyi+Wu%2C+Yichi+Yang%2C+Dong%2DHo+Lee%2C+and+Xiang+Ren%2E+2021%2E+Riddlesense%3A+Reasoning+about+riddle+questions+featuring+linguistic+creativity+and+commonsense+knowledge%2E+In+Proceedings+of+the+5)
24. Yuliang Liu, Junjie Lu, Zhaoling Chen, Chaofeng Qu, Jason Klein Liu, Chonghan Liu, Zefan Cai, Yunhui Xia, Li Zhao, Jiang Bian, Chuheng Zhang, Wei Shen, and Zhouhan Lin. 2025a. Adaptivestep: Automatically dividing reasoning step through model confidence. Preprint, arXiv:2502.13943. — [arXiv:2502.13943](https://arxiv.org/abs/2502.13943) [find](https://scholar.google.com/scholar?q=Yuliang+Liu%2C+Junjie+Lu%2C+Zhaoling+Chen%2C+Chaofeng+Qu%2C+Jason+Klein+Liu%2C+Chonghan+Liu%2C+Zefan+Cai%2C+Yunhui+Xia%2C+Li+Zhao%2C+Jiang+Bian%2C+Chuheng+Zhang%2C+Wei+Shen%2C+and+Zhouhan+Lin%2E+2025a%2E+Adaptivestep%3A+Automatica)
25. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. 2025b. [Understanding r1-zero-like training: A critical perspective](https://md.research.elamur.ai/free/en/papers/liu-2025-understanding-r1-zero-like-training/). arXiv preprint arXiv:2503.20783.
26. Jianqiao Lu, Zhiyang Dou, Hongru Wang, Zeyu Cao, Jianbo Dai, Yingjia Wan, and Zhijiang Guo. 2024. [Autopsv: Automated process-supervised verifier](https://md.research.elamur.ai/free/en/papers/lu-2024-autopsv-automated-process-supervised-verifier/). Preprint, arXiv:2405.16802.
27. Kevin Lu and 1 others. 2025. On-policy distillation. [find](https://scholar.google.com/scholar?q=Kevin+Lu+and+1+others%2E+2025%2E+On%2Dpolicy+distillation%2E)
28. Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. 2025. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl. [https://pretty-radio-b75.notion.site/DeepScaleR-Surpassing-O1-Preview-wit…](https://pretty-radio-b75.notion.site/DeepScaleR-Surpassing-O1-Preview-wit…) [↗](https://pretty-radio-b75.notion.site/DeepScaleR-Surpassing-O1-Preview-wit…)
29. Chengqi Lyu, Songyang Gao, Yuzhe Gu, Wenwei Zhang, Jianfei Gao, Kuikun Liu, Ziyi Wang, Shuaibin Li, Qian Zhao, Haian Huang, and 1 others. 2025. Exploring the limit of outcome reward for learning mathematical reasoning. arXiv preprint arXiv:2502.06781. — [arXiv:2502.06781](https://arxiv.org/abs/2502.06781) [find](https://scholar.google.com/scholar?q=Chengqi+Lyu%2C+Songyang+Gao%2C+Yuzhe+Gu%2C+Wenwei+Zhang%2C+Jianfei+Gao%2C+Kuikun+Liu%2C+Ziyi+Wang%2C+Shuaibin+Li%2C+Qian+Zhao%2C+Haian+Huang%2C+and+1+others%2E+2025%2E+Exploring+the+limit+of+outcome+reward+for+learning+mathe)
30. Qianli Ma, Haotian Zhou, Tingkai Liu, Jianbo Yuan, Pengfei Liu, Yang You, and Hongxia Yang. 2023. Let’s reward step by step: Step-level reward model as the navigators for reasoning. Preprint, arXiv:2310.10080. [find](https://scholar.google.com/scholar?q=Qianli+Ma%2C+Haotian+Zhou%2C+Tingkai+Liu%2C+Jianbo+Yuan%2C+Pengfei+Liu%2C+Yang+You%2C+and+Hongxia+Yang%2E+2023%2E+Let%E2%80%99s+reward+step+by+step%3A+Step%2Dlevel+reward+model+as+the+navigators+for+reasoning%2E+Preprint%2C+arXiv%3A23)
31. Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Trebacz, and Jan Leike. 2024. Llm critics help catch llm bugs. Preprint, arXiv:2407.00215. — [arXiv:2407.00215](https://arxiv.org/abs/2407.00215) [find](https://scholar.google.com/scholar?q=Nat+McAleese%2C+Rai+Michael+Pokorny%2C+Juan+Felipe+Ceron+Uribe%2C+Evgenia+Nitishinskaya%2C+Maja+Trebacz%2C+and+Jan+Leike%2E+2024%2E+Llm+critics+help+catch+llm+bugs%2E+Preprint%2C+arXiv%3A2407%2E00215%2E+%E2%80%94+%5BarXiv%3A2407%2E00215%5D%28)
32. OpenAI. 2024b. Learning to reason with llms. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2024b%2E+Learning+to+reason+with+llms%2E)
33. OpenAI. 2025. Introducing openai o3 and o4-mini. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2025%2E+Introducing+openai+o3+and+o4%2Dmini%2E)
34. Sarah Pan, Vladislav Lialin, Sherin Muckatira, and Anna Rumshisky. 2023. Let’s reinforce step by step. Preprint, arXiv:2311.05821. — [arXiv:2311.05821](https://arxiv.org/abs/2311.05821) [find](https://scholar.google.com/scholar?q=Sarah+Pan%2C+Vladislav+Lialin%2C+Sherin+Muckatira%2C+and+Anna+Rumshisky%2E+2023%2E+Let%E2%80%99s+reinforce+step+by+step%2E+Preprint%2C+arXiv%3A2311%2E05821%2E+%E2%80%94+%5BarXiv%3A2311%2E05821%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2311%2E05821%29)
35. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. [Gpqa: A graduate-level google-proof q&a benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/). In First Conference on Language Modeling.
36. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+2017%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2E)
37. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, and 1 others. 2024a. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). arXiv preprint arXiv:2402.03300.
38. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024b. [find](https://scholar.google.com/scholar?q=Zhihong+Shao%2C+Peiyi+Wang%2C+Qihao+Zhu%2C+Runxin+Xu%2C+Junxiao+Song%2C+Xiao+Bi%2C+Haowei+Zhang%2C+Mingchuan+Zhang%2C+YK+Li%2C+Yang+Wu%2C+and+1+others%2E+2024b%2E)
39. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). arXiv preprint arXiv:2402.03300. Open-R1 Team. 2025a. Openr1-math-220k.
40. OpenThoughts Team. 2025b. [https://open-thoughts.ai](https://open-thoughts.ai). [↗](https://open-thoughts.ai)
41. Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. [Solving math word problems with process- and outcome-based feedback](https://md.research.elamur.ai/free/en/papers/uesato-2022-solving-math-word-problems-process/). Preprint, arXiv:2211.14275.
42. Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang Sui. 2024a. [Math-shepherd: Verify and reinforce llms step-by-step without human annotations](https://md.research.elamur.ai/free/en/papers/wang-2024-math-shepherd-verify-reinforce-llms/). Preprint, arXiv:2312.08935.
43. Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, and Xian Li. 2024b. [Self-taught evaluators](https://md.research.elamur.ai/free/en/papers/wang-2024-self-taught-evaluators/). Preprint, arXiv:2408.02666.
44. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in neural information processing systems, 35:24824– 24837.
45. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, and 1 others. 2025. [Dapo: An open-source llm reinforcement learning system at scale](https://md.research.elamur.ai/free/en/papers/yu-2025-dapo-open-source-llm-reinforcement/). arXiv preprint arXiv:2503.14476.
46. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. 2022. [Star: Bootstrapping reasoning with reasoning](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/). Preprint, arXiv:2203.14465.
47. Thomas Zeng, Shuibai Zhang, Shutong Wu, Christian Classen, Daewon Chae, Ethan Ewer, Minjae Lee, Heeju Kim, Wonjun Kang, Jackson Kunde, Ying Fan, Jungtaek Kim, Hyung Il Koo, Kannan Ramchandran, Dimitris Papailiopoulos, and Kangwook Lee. 2025. Versaprm: Multi-domain process reward model via synthetic… — [arXiv:2502.06737](https://arxiv.org/abs/2502.06737) [find](https://scholar.google.com/scholar?q=Thomas+Zeng%2C+Shuibai+Zhang%2C+Shutong+Wu%2C+Christian+Classen%2C+Daewon+Chae%2C+Ethan+Ewer%2C+Minjae+Lee%2C+Heeju+Kim%2C+Wonjun+Kang%2C+Jackson+Kunde%2C+Ying+Fan%2C+Jungtaek+Kim%2C+Hyung+Il+Koo%2C+Kannan+Ramchandran%2C+Dimitri)
48. Yizhou Zhang, Ning Lv, Teng Wang, and Jisheng Dang. 2025. Fastgrpo: Accelerating policy optimization via concurrency-aware speculative decoding and online draft learning. arXiv preprint arXiv:2509.21792. — [arXiv:2509.21792](https://arxiv.org/abs/2509.21792) [find](https://scholar.google.com/scholar?q=Yizhou+Zhang%2C+Ning+Lv%2C+Teng+Wang%2C+and+Jisheng+Dang%2E+2025%2E+Fastgrpo%3A+Accelerating+policy+optimization+via+concurrency%2Daware+speculative+decoding+and+online+draft+learning%2E+arXiv+preprint+arXiv%3A2509%2E217)
49. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Preprint, arXiv:2306.05685.
50. Guofu Xie, Yunsheng Shi, Hongtao Tian, Ting Yao, and Xiao Zhang. 2025. Capo: Towards enhancing llm reasoning through verifiable generative credit assignment. arXiv e-prints, pages arXiv–2508. [find](https://scholar.google.com/scholar?q=Guofu+Xie%2C+Yunsheng+Shi%2C+Hongtao+Tian%2C+Ting+Yao%2C+and+Xiao+Zhang%2E+2025%2E+Capo%3A+Towards+enhancing+llm+reasoning+through+verifiable+generative+credit+assignment%2E+arXiv+e%2Dprints%2C+pages+arXiv%E2%80%932508%2E)
51. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.
52. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024a. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+and+1+others%2E+2024a%2E+Qwen2%2E+5+technical+report%2E+arXiv+preprint+arXiv%3A2412%2E1511)
53. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, and 1 others. 2024b. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122. — [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://md.research.elamur.ai/free/en/papers/yang-2024-qwen2-5-math-technical-report/)
54. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. Preprint, arXiv:2305.10601. [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Thomas+L%2E+Griffiths%2C+Yuan+Cao%2C+and+Karthik+Narasimhan%2E+2023%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2E+Preprint%2C+arXiv%3A)
55. Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. 2025. [Limo: Less is more for reasoning](https://md.research.elamur.ai/free/en/papers/ye-2025-limo-less-more-reasoning/). arXiv preprint arXiv:2502.03387.

*Updated 2 November 2026*