---
title: "Can Large Reasoning Models Self-Train? — Shafayat 2025"
description: "The authors propose Self-Rewarded Training (SRT). 96 references, cited by 16 papers here."
authors: ["Sheikh Shafayat", "Fahim Tajwar", "Ruslan Salakhutdinov", "Jeff Schneider", "Andrea Zanette"]
year: 2025
venue: "arXiv:2505.21444"
source: "https://arxiv.org/abs/2505.21444"
doi: "10.48550/arxiv.2505.21444"
arxiv: "2505.21444"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shafayat-2025-large-reasoning-models-self-train/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shafayat-2025-large-reasoning-models-self-train/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/7/) / Can Large Reasoning Models Self-Train?

# Can Large Reasoning Models Self-Train?

**Authors:** [Sheikh Shafayat](https://md.research.elamur.ai/free/en/authors/sheikh-shafayat/), Fahim Tajwar, [Ruslan Salakhutdinov](https://md.research.elamur.ai/free/en/authors/ruslan-salakhutdinov/), Jeff Schneider and [Andrea Zanette](https://md.research.elamur.ai/free/en/authors/andrea-zanette/) · **Year:** 2025 · **Venue:** arXiv:2505.21444 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2505.21444) · [PDF](https://arxiv.org/pdf/2505.21444) · [DOI 10.48550/arxiv.2505.21444](https://doi.org/10.48550/arxiv.2505.21444) · [arXiv 2505.21444](https://arxiv.org/abs/2505.21444) · [OpenAlex](https://openalex.org/W4415037569)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [16](https://md.research.elamur.ai/free/en/papers/shafayat-2025-large-reasoning-models-self-train/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks whether a reasoning model can keep improving through reinforcement learning when no ground-truth answers are available and the training signal comes only from the model itself. The authors propose Self-Rewarded Training (SRT). For each prompt, the current policy samples several solutions. The most frequent final answer serves as a pseudo-label, and each sample is rewarded for agreeing with it. Because the labels come from the same rollouts used for the RL update, the method adds no compute cost and works with standard algorithms such as PPO, GRPO or RLOO. Experiments cover synthetic Reasoning Gym puzzles and math benchmarks. SRT raises both average accuracy and majority-vote accuracy, so the supervision signal improves as training progresses. It roughly matches RL with true labels across four base models. However, extended training ends in reward hacking: the model learns to output an identical answer regardless of the prompt and its test performance collapses. The authors conclude that the design of self-feedback is the main obstacle to sustained self-improvement without human labels.

## Key points
- **Evolving vs. fixed teacher:** On Reasoning Gym tasks (Family Relationships, Bitwise Arithmetic, Knights & Knaves), letting the updated policy produce the majority-vote labels beat using labels from the frozen base model. On these synthetic tasks, a curriculum of progressively harder levels let the model keep improving without ground truth.
- **Llama-3.1-8B-Instruct result:** The model was trained on a Big-Math subset where its pass rate lay between 0.3 and 0.7. Within 2000 steps, SRT and ground-truth RL produced the same gains, lifting MATH-500 pass@1 from 52.6% to about 60%.
- **Collapse example:** After 1200 SRT steps on DAPO, Qwen2.5-Math-7B returned `\boxed{1}` plus incoherent tokens for every prompt. This trivially maximizes self-consistency reward while destroying accuracy.
- **Learning-rate sensitivity:** Increasing the learning rate for Llama from 1e-7 to 3e-7 also triggered collapse within the training budget.
- **Code release:** Code is available at github.com/tajwarfahim/srt, and datasets are on the project website.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 633 works · 195,218 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,948 works

## Cite

Cite as: Sheikh Shafayat, Fahim Tajwar, Ruslan Salakhutdinov, Jeff Schneider and Andrea Zanette. 2025. Can Large Reasoning Models Self-Train?. arXiv:2505.21444.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{shafayat2025can,
  title = {{Can Large Reasoning Models Self-Train?}},
  author = {Sheikh Shafayat, Fahim Tajwar, Ruslan Salakhutdinov, Jeff Schneider and Andrea Zanette},
  year = {2025},
  doi = {10.48550/arxiv.2505.21444},
  eprint = {2505.21444},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2505.21444}
}
```

- [← previous Can Knowledge Graphs Make Large Language Models More Trustworthy? An Empirical Study Over Open-ended Question Answering](https://md.research.elamur.ai/free/en/papers/sui-2025-can-knowledge-graphs-make-large/)
- [next → ChronoSense: Exploring Temporal Understanding in Large Language Models with Time Intervals of Events](https://md.research.elamur.ai/free/en/papers/islakoglu-2025-chronosense-exploring-temporal-understanding-large/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/7/)

## References

96 references, 49 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting REINFORCE-style optimization for learning from human feedback in LLMs. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the… [find](https://scholar.google.com/scholar?q=Arash+Ahmadian%2C+Chris+Cremer%2C+Matthias+Gall%C3%A9%2C+Marzieh+Fadaee%2C+Julia+Kreutzer%2C+Olivier+Pietquin%2C+Ahmet+%C3%9Cst%C3%BCn%2C+and+Sara+Hooker%2E+Back+to+basics%3A+Revisiting+REINFORCE%2Dstyle+optimization+for+learning+from+)
2. Alon Albalak, Duy Phung, Nathan Lile, Rafael Rafailov, Kanishk Gandhi, Louis Castricato, Anikait Singh, Chase Blagden, Violet Xiang, Dakota Mahan, and Nick Haber. [Big-math: A large-scale, high-quality math dataset for reinforcement learning in language models](https://md.research.elamur.ai/free/en/papers/albalak-2025-big-math-large-scale-high/), 2025. URL https://arxiv.org/abs/2502.17… [↗](https://arxiv.org/abs/2502.17…)
3. Sina Alemohammad, Josue Casco-Rodriguez, Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaei, Daniel LeJeune, Ali Siahkoohi, and Richard Baraniuk. Self-consuming generative models go MAD. In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/forum?id=Sh…](https://openreview.net/forum?id=Sh…) [↗](https://openreview.net/forum?id=Sh…)
4. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. [Concrete problems in ai safety](https://md.research.elamur.ai/free/en/papers/amodei-2016-concrete-problems-ai-safety/), 2016. URL https://arxiv.org/abs/1606.06565. [↗](https://arxiv.org/abs/1606.06565)
5. Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. [Hindsight experience replay](https://md.research.elamur.ai/free/en/papers/andrychowicz-2017-hindsight-experience-replay/). Advances in neural information processing systems, 30, 2017.
6. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. [Curriculum learning](https://md.research.elamur.ai/free/en/papers/bengio-2009-curriculum-learning/). In Proceedings of the 26th annual international conference on machine learning, pp. 41–48, 2009.
7. Quentin Bertrand, Joey Bose, Alexandre Duplessis, Marco Jiralerspong, and Gauthier Gidel. On the stability of iterative retraining of generative models on their own data. In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/forum?](https://openreview.net/forum?) id=JORAfH2xFd. [↗](https://openreview.net/forum?)
8. Timothy J. Boerner, Stephen Deems, Thomas R. Furlani, Shelley L. Knuth, and John Towns. Access: Advancing innovation: Nsf’s advanced cyberinfrastructure coordination ecosystem: Services & support. In Practice and Experience in Advanced Research Computing 2023: Computing for the Common Good, PEARC ’2… — [doi:10.1145/3569951.3597559](https://doi.org/10.1145/3569951.3597559) [find](https://scholar.google.com/scholar?q=Timothy+J%2E+Boerner%2C+Stephen+Deems%2C+Thomas+R%2E+Furlani%2C+Shelley+L%2E+Knuth%2C+and+John+Towns%2E+Access%3A+Advancing+innovation%3A+Nsf%E2%80%99s+advanced+cyberinfrastructure+coordination+ecosystem%3A+Services+%26+support%2E+In+)
9. Martin Briesch, Dominik Sobania, and Franz Rothlauf. Large language models suffer from their own output: An analysis of the self-consuming training loop, 2025. URL [https://openreview.net/forum?id=SaOxhcDCM3](https://openreview.net/forum?id=SaOxhcDCM3). [↗](https://openreview.net/forum?id=SaOxhcDCM3)
10. Shawn T. Brown, Paola Buitrago, Edward Hanna, Sergiu Sanielevici, Robin Scibek, and Nicholas A. Nystrom. Bridges-2: A platform for rapidly-evolving and data intensive research. In Practice and Experience in Advanced Research Computing 2021: Evolution Across All Dimensions, PEARC ’21, New York, NY, U… — [doi:10.1145/3437359.3465593](https://doi.org/10.1145/3437359.3465593) [find](https://scholar.google.com/scholar?q=Shawn+T%2E+Brown%2C+Paola+Buitrago%2C+Edward+Hanna%2C+Sergiu+Sanielevici%2C+Robin+Scibek%2C+and+Nicholas+A%2E+Nystrom%2E+Bridges%2D2%3A+A+platform+for+rapidly%2Devolving+and+data+intensive+research%2E+In+Practice+and+Experie)
11. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
12. Lili Chen, Mihir Prabhudesai, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak. Self-questioning language models, 2025. URL https://arxiv.org/abs/2508.03682. — [arXiv:2508.03682](https://arxiv.org/abs/2508.03682) [↗](https://arxiv.org/abs/2508.03682)
13. Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning convertsweak language models to strong language models. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024. [find](https://scholar.google.com/scholar?q=Zixiang+Chen%2C+Yihe+Deng%2C+Huizhuo+Yuan%2C+Kaixuan+Ji%2C+and+Quanquan+Gu%2E+Self%2Dplay+fine%2Dtuning+convertsweak+language+models+to+strong+language+models%2E+In+Proceedings+of+the+41st+International+Conference+on)
14. Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. [Reasoning with exploration: An entropy perspective](https://md.research.elamur.ai/free/en/papers/cheng-2025-reasoning-exploration-entropy-perspective/) on reinforcement learning for llms, 2025. URL https://arxiv.org/abs/2506.14758. [↗](https://arxiv.org/abs/2506.14758)
15. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, et al. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/), 2025. URL https://arxiv.org/abs/2501.12948. [↗](https://arxiv.org/abs/2501.12948)
16. Carson Denison, Monte MacDiarmid, Fazl Barez, David Duvenaud, Shauna Kravec, Samuel Marks, Nicholas Schiefer, Ryan Soklaski, Alex Tamkin, Jared Kaplan, Buck Shlegeris, Samuel R. Bowman, Ethan Perez, and Evan Hubinger. Sycophancy to subterfuge: Investigating reward-tampering in large language models,… [find](https://scholar.google.com/scholar?q=Carson+Denison%2C+Monte+MacDiarmid%2C+Fazl+Barez%2C+David+Duvenaud%2C+Shauna+Kravec%2C+Samuel+Marks%2C+Nicholas+Schiefer%2C+Ryan+Soklaski%2C+Alex+Tamkin%2C+Jared+Kaplan%2C+Buck+Shlegeris%2C+Samuel+R%2E+Bowman%2C+Ethan+Perez%2C+a)
17. Yunzhen Feng, Elvis Dohmatob, Pu Yang, Francois Charton, and Julia Kempe. Beyond model collapse: Scaling up with synthesized data requires verification. In The Thirteenth International Conference on Learning Representations, 2025. URL [https://openreview.net/forum?](https://openreview.net/forum?) id=MQXrTMonT1. [↗](https://openreview.net/forum?)
18. Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse curriculum generation for reinforcement learning. In Conference on robot learning, pp. 482–495. PMLR, 2017. [find](https://scholar.google.com/scholar?q=Carlos+Florensa%2C+David+Held%2C+Markus+Wulfmeier%2C+Michael+Zhang%2C+and+Pieter+Abbeel%2E+Reverse+curriculum+generation+for+reinforcement+learning%2E+In+Conference+on+robot+learning%2C+pp%2E+482%E2%80%93495%2E+PMLR%2C+2017%2E)
19. Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D. Goodman. [Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars](https://md.research.elamur.ai/free/en/papers/gandhi-2025-cognitive-behaviors-that-enable-self/), 2025. URL https://arxiv.org/abs/2503.01307. [↗](https://arxiv.org/abs/2503.01307)
20. Huanang Gao, Jiayi Geng, Wenyue Hua, Mengkang Hu, Xinzhe Juan, Hongzhang Liu, Shilong Liu, Jiahao Qiu, Xuan Qi, Yiran Wu, Hongru Wang, Han Xiao, Yuhang Zhou, Shaokun Zhang, Jiayi Zhang, Jinyu Xiang, Yixiong Fang, Qiwen Zhao, Dongrui Liu, Qihan Ren, Cheng Qian, Zhenhailong Wang, Minda Hu, Huazheng Wa… — [A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence](https://md.research.elamur.ai/free/en/papers/gao-2025-survey-self-evolving-agents-where/)
21. Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Tomasz Korbak, Henry Sleight, Rajashree Agrawal, John Hughes, Dhruv Bhandarkar Pai, Andrey Gromov, Dan Roberts, Diyi Yang, David L. Donoho, and Sanmi Koyejo. Is model collapse inevitable? breaking the curse of recursion by accumul… [find](https://scholar.google.com/scholar?q=Matthias+Gerstgrasser%2C+Rylan+Schaeffer%2C+Apratim+Dey%2C+Rafael+Rafailov%2C+Tomasz+Korbak%2C+Henry+Sleight%2C+Rajashree+Agrawal%2C+John+Hughes%2C+Dhruv+Bhandarkar+Pai%2C+Andrey+Gromov%2C+Dan+Roberts%2C+Diyi+Yang%2C+David+L)
22. Nate Gillman, Michael Freeman, Daksh Aggarwal, Chia-Hong Hsu, Calvin Luo, Yonglong Tian, and Chen Sun. Self-correcting self-consuming loops for generative model training, 2024. URL https://arxiv.org/abs/2402.07087. — [arXiv:2402.07087](https://arxiv.org/abs/2402.07087) [↗](https://arxiv.org/abs/2402.07087)
23. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, et al. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/), 2024. URL https://arxiv.org/abs/2407.21783. [↗](https://arxiv.org/abs/2407.21783)
24. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. A survey on llm-as-a-judge, 2025. URL https://arxiv.org/abs/2411. 15594. [↗](https://arxiv.org/abs/2411)
25. Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, and Nando de Freitas. [Reinforced self-training (rest) for language modeling](https://md.research.elamur.ai/free/en/papers/gulcehre-2023-reinforced-self-training-rest-language/), 2023. URL https:/…
26. Alex Havrilla, Sharath Raparthy, Christoforus Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, and Roberta Raileanu. [Glore: When, where, and how to improve llm reasoning via global and local refinements](https://md.research.elamur.ai/free/en/papers/havrilla-2024-glore-where-improve-llm-reasoning/), 2024. URL https://arxiv.org/abs/2402.10963. [↗](https://arxiv.org/abs/2402.10963)
27. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. [Measuring mathematical problem solving with the MATH dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL ht…
28. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… — [Training Compute-Optimal Large Language Models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
29. Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. [V-STar: Training verifiers for self-taught reasoners](https://md.research.elamur.ai/free/en/papers/hosseini-2024-v-star-training-verifiers-self/). In First Conference on Language Modeling, 2024. URL [https://openreview.net/forum?id=stmqBSW2dV](https://openreview.net/forum?id=stmqBSW2dV). [↗](https://openreview.net/forum?id=stmqBSW2dV)
30. Jian Hu, Jason Klein Liu, and Wei Shen. Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models, 2025. [URL https://arxiv.org/abs/2501.03262](https://md.research.elamur.ai/free/en/papers/hu-2025-reinforce-stabilizing-critic-free-policy/). [↗](https://arxiv.org/abs/2501.03262)
31. Audrey Huang, Adam Block, Dylan J Foster, Dhruv Rohatgi, Cyril Zhang, Max Simchowitz, Jordan T. Ash, and Akshay Krishnamurthy. [Self-improvement in language models: The sharpening mechanism](https://md.research.elamur.ai/free/en/papers/huang-2024-self-improvement-language-models-sharpening/). In The Thirteenth International Conference on Learning Representations, 2025. URL [https://openreview.net/forum…](https://openreview.net/forum…) [↗](https://openreview.net/forum…)
32. Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. [Large language models can self-improve](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-can-self/). In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 1051– 1068, Singapore, Decem…
33. Hyeonbin Hwang, Doyoung Kim, Seungone Kim, Seonghyeon Ye, and Minjoon Seo. Self-explore: Enhancing mathematical reasoning in language models with fine-grained rewards. In EMNLP (Findings), pp. 1444–1466, 2024. URL https://aclanthology.org/2024. findings-emnlp.78. [↗](https://aclanthology.org/2024)
34. Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 reinforce samples, get a baseline for free! In DeepRLStructPred@ICLR, 2019. URL [https://api.semanticscholar.org/](https://api.semanticscholar.org/) CorpusID:198489118. [↗](https://api.semanticscholar.org/)
35. Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smi… — [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://md.research.elamur.ai/free/en/papers/lambert-2024-tulu-3-pushing-frontiers-open/)
36. Jack Lanchantin, Angelica Chen, Janice Lan, Xian Li, Swarnadeep Saha, Tianlu Wang, Jing Xu, Ping Yu, Weizhe Yuan, Jason E Weston, Sainbayar Sukhbaatar, and Ilia Kulikov. Bridging offline and online reinforcement learning for llms, 2025. URL https://arxiv.org/abs/2506. 21495. [↗](https://arxiv.org/abs/2506)
37. Nayoung Lee, Ziyang Cai, Avi Schwarzschild, Kangwook Lee, and Dimitris Papailiopoulos. Self-improving transformers overcome easy-to-hard and length generalization challenges, 2025. URL https://arxiv.org/abs/2502.01612. [↗](https://arxiv.org/abs/2502.01612)
38. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. [Understanding r1-zero-like training: A critical perspective](https://md.research.elamur.ai/free/en/papers/liu-2025-understanding-r1-zero-like-training/), 2025. URL https://arxiv.org/abs/2503.20783. [↗](https://arxiv.org/abs/2503.20783)
39. Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl, 2025. Notion Blog. [find](https://scholar.google.com/scholar?q=Michael+Luo%2C+Sijun+Tan%2C+Justin+Wong%2C+Xiaoxiang+Shi%2C+William+Tang%2C+Manan+Roongta%2C+Colin+Cai%2C+Jeffrey+Luo%2C+Tianjun+Zhang%2C+Erran+Li%2C+Raluca+Ada+Popa%2C+and+Ion+Stoica%2E+Deepscaler%3A+Surpassing+o1%2Dpreview+wit)
40. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534–46594, 2023. [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Prakhar+Gupta%2C+Skyler+Hallinan%2C+Luyu+Gao%2C+Sarah+Wiegreffe%2C+Uri+Alon%2C+Nouha+Dziri%2C+Shrimai+Prabhumoye%2C+Yiming+Yang%2C+et+al%2E+Self%2Drefine%3A+Iterative+refinement+with+self%2Dfeedbac)
41. Chris Mingard, Joar Skalse, Guillermo Valle-Pérez, David Martínez-Rubio, Vladimir Mikulik, and Ard A. Louis. Neural networks are a priori biased towards boolean functions with low entropy, 2020. URL https://arxiv.org/abs/1909.11522. — [arXiv:1909.11522](https://arxiv.org/abs/1909.11522) [↗](https://arxiv.org/abs/1909.11522)
42. Chris Mingard, Henry Rees, Guillermo Valle-Pérez, and Ard A. Louis. Deep neural networks have an inbuilt occam’s razor. Nature Communications, 16:220, 2025. doi: 10.1038/s41467-024-54813-x. URL https://doi.org/10.1038/s41467-024-54813-x. — [doi:10.1038/s41467-024-54813-x](https://doi.org/10.1038/s41467-024-54813-x) [↗](https://doi.org/10.1038/s41467-024-54813-x)
43. Ansong Ni, Jeevana Priya Inala, Chenglong Wang, Alex Polozov, Christopher Meek, Dragomir Radev, and Jianfeng Gao. [Learning math reasoning from self-sampled correct and partially-correct solutions](https://md.research.elamur.ai/free/en/papers/ni-2022-learning-math-reasoning-self-sampled/). In ICLR, 2023. URL [https://openreview.net/forum?id=4D4TSJE6-K](https://openreview.net/forum?id=4D4TSJE6-K). [↗](https://openreview.net/forum?id=4D4TSJE6-K)
44. Owen Oertell, Wenhao Zhan, Gokul Swamy, Zhiwei Steven Wu, Kiante Brantley, Jason Lee, and Wen Sun. Heuristics considered harmful: Rl with random rewards should not make llms reason, 2024. Preprint. [find](https://scholar.google.com/scholar?q=Owen+Oertell%2C+Wenhao+Zhan%2C+Gokul+Swamy%2C+Zhiwei+Steven+Wu%2C+Kiante+Brantley%2C+Jason+Lee%2C+and+Wen+Sun%2E+Heuristics+considered+harmful%3A+Rl+with+random+rewards+should+not+make+llms+reason%2C+2024%2E+Preprint%2E)
45. OpenAI, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, et al. [Openai o1 system card](https://md.research.elamur.ai/free/en/papers/openai-2024-openai-o1-system-card/), 2024. URL https://arxiv.org/abs/… [↗](https://arxiv.org/abs/…)
46. Giacomo De Palma, Bobak Toussi Kiani, and Seth Lloyd. Random deep neural networks are biased towards simple functions, 2019. URL https://arxiv.org/abs/1812.10156. — [arXiv:1812.10156](https://arxiv.org/abs/1812.10156) [↗](https://arxiv.org/abs/1812.10156)
47. Jing-Cheng Pang, Pengyuan Wang, Kaiyuan Li, Xiong-Hui Chen, Jiacheng Xu, Zongzhang Zhang, and Yang Yu. [Language model self-improvement by reinforcement learning contemplation](https://md.research.elamur.ai/free/en/papers/pang-2023-language-model-self-improvement-reinforcement/). In The Twelfth International Conference on Learning Representations, 2024a. URL [https://openreview.net/forum?id=38E4yUbrgr](https://openreview.net/forum?id=38E4yUbrgr). [↗](https://openreview.net/forum?id=38E4yUbrgr)
48. Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. [Iterative reasoning preference optimization](https://md.research.elamur.ai/free/en/papers/pang-2024-iterative-reasoning-preference-optimization/), 2024b. URL [https://arxiv.org/](https://arxiv.org/) abs/2404.19733. [↗](https://arxiv.org/)
49. Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre-Yves Oudeyer. Automatic curriculum learning for deep rl: A short survey. arXiv preprint arXiv:2003.04664, 2020. [find](https://scholar.google.com/scholar?q=R%C3%A9my+Portelas%2C+C%C3%A9dric+Colas%2C+Lilian+Weng%2C+Katja+Hofmann%2C+and+Pierre%2DYves+Oudeyer%2E+Automatic+curriculum+learning+for+deep+rl%3A+A+short+survey%2E+arXiv+preprint+arXiv%3A2003%2E04664%2C+2020%2E)
50. Mihir Prabhudesai, Lili Chen, Alex Ippoliti, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak. Maximizing confidence alone improves reasoning, 2025. URL https://arxiv.org/abs/ 2505.22660. [↗](https://arxiv.org/abs/)
51. Archiki Prasad, Weizhe Yuan, Richard Yuanzhe Pang, Jing Xu, Maryam Fazel-Zarandi, Mohit Bansal, Sainbayar Sukhbaatar, Jason Weston, and Jane Yu. Self-consistency preference optimization, 2024. URL https://arxiv.org/abs/2411.04109. [↗](https://arxiv.org/abs/2411.04109)
52. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, et al. Scaling language models: Methods, analysis & insights from tra… [find](https://scholar.google.com/scholar?q=Jack+W%2E+Rae%2C+Sebastian+Borgeaud%2C+Trevor+Cai%2C+Katie+Millican%2C+Jordan+Hoffmann%2C+Francis+Song%2C+John+Aslanides%2C+Sarah+Henderson%2C+Roman+Ring%2C+Susannah+Young%2C+Eliza+Rutherford%2C+Tom+Hennigan%2C+Jacob+Menick%2C+A)
53. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/), 2024. URL https://arxiv.org/abs/2305.18290. [↗](https://arxiv.org/abs/2305.18290)
54. Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjeev Arora, and Boris Hanin. Unintentional unalignment: Likelihood displacement in direct preference optimization, 2025. URL https://arxiv.org/abs/2410.08847. [↗](https://arxiv.org/abs/2410.08847)
55. John Schulman. Approximating kl divergence. [http://joschu.net/blog/kl-approx](http://joschu.net/blog/kl-approx). html, Mar 2020. Accessed: 2025-05-20. [↗](http://joschu.net/blog/kl-approx)
56. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2C+2017%2E)
57. Amrith Setlur, Saurabh Garg, Xinyang Geng, Naman Garg, Virginia Smith, and Aviral Kumar. [Rl on incorrect synthetic data scales the efficiency of llm math reasoning by eight-fold](https://md.research.elamur.ai/free/en/papers/setlur-2024-rl-incorrect-synthetic-data-scales/). Advances in Neural Information Processing Systems, 37:43000–43031, 2024.
58. Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, and Pablo Villalobos. Compute trends across three eras of machine learning, 2022. [find](https://scholar.google.com/scholar?q=Jaime+Sevilla%2C+Lennart+Heim%2C+Anson+Ho%2C+Tamay+Besiroglu%2C+Marius+Hobbhahn%2C+and+Pablo+Villalobos%2E+Compute+trends+across+three+eras+of+machine+learning%2C+2022%2E)
59. Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, Yulia Tsvetkov, Hannaneh Hajishirzi, Pang Wei Koh, and Luke Zettlemoyer. [Spurious rewards: Rethinking training signals in rlvr](https://md.research.elamur.ai/free/en/papers/shao-2025-spurious-rewards-rethinking-training-signals/), 2025. URL https://arxiv.org/abs/250… [↗](https://arxiv.org/abs/250…)
60. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/), 2024. URL https://arxiv.org/abs/ 2402.03300. [↗](https://arxiv.org/abs/)
61. Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. [The curse of recursion: Training on generated data makes models forget](https://md.research.elamur.ai/free/en/papers/shumailov-2023-curse-recursion-training-generated-data/), 2024a. URL https://arxiv.org/abs/2305.17493. [↗](https://arxiv.org/abs/2305.17493)
62. Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross J. Anderson, and Yarin Gal. [Ai models collapse when trained on recursively generated data](https://md.research.elamur.ai/free/en/papers/shumailov-2024-ai-models-collapse-trained-recursively/). Nat., 631(8022):755–759, July 2024b. URL https://doi.org/10.1038/s41586-024-07566-y. [↗](https://doi.org/10.1038/s41586-024-07566-y)
63. Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron T Parisi, Abhishek Kumar, Alexander A Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Fathy Elsayed, Hanie Sedghi, Igor Mordatch,… — [Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models](https://md.research.elamur.ai/free/en/papers/singh-2023-human-data-scaling-self-training/)
64. Yuda Song, Julia Kempe, and Remi Munos. [Outcome-based exploration for llm reasoning](https://md.research.elamur.ai/free/en/papers/song-2025-outcome-based-exploration-llm-reasoning/), 2025a. URL https://arxiv.org/abs/2509.06941. [↗](https://arxiv.org/abs/2509.06941)
65. Yuda Song, Hanlin Zhang, Carson Eisenach, Sham M. Kakade, Dean Foster, and Udaya Ghai. Mind the gap: Examining the self-improvement capabilities of large language models. In The Thirteenth International Conference on Learning Representations, 2025b. URL [https://openreview.net/forum?id=mtJSMcF3ek](https://openreview.net/forum?id=mtJSMcF3ek). [↗](https://openreview.net/forum?id=mtJSMcF3ek)
66. Zafir Stojanovski, Oliver Stanley, Joe Sharratt, Richard Jones, Abdulhakeem Adefioye, Jean Kaddour, and Andreas Köpf. Reasoning gym: Reasoning environments for reinforcement learning with verifiable rewards, 2025. URL https://arxiv.org/abs/2505.24760. — [arXiv:2505.24760](https://arxiv.org/abs/2505.24760) [↗](https://arxiv.org/abs/2505.24760)
67. Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A. Efros, and Moritz Hardt. Test-time training for out-of-distribution generalization, 2020. URL [https://openreview.net/](https://openreview.net/) forum?id=HyezmlBKwr. [↗](https://openreview.net/)
68. Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, and Chuang Gan. [Easy-to-hard generalization: Scalable alignment beyond human supervision](https://md.research.elamur.ai/free/en/papers/sun-2024-easy-hard-generalization-scalable-alignment/). In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL [https://openreview.net/forum?id=qwgfh2fT…](https://openreview.net/forum?id=qwgfh2fT…) [↗](https://openreview.net/forum?id=qwgfh2fT…)
69. Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998. [find](https://scholar.google.com/scholar?q=Richard+S+Sutton%2C+Andrew+G+Barto%2C+et+al%2E+Reinforcement+learning%3A+An+introduction%2C+volume+1%2E+MIT+press+Cambridge%2C+1998%2E)
70. Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of llms should leverage suboptimal, on-policy data, 2024. URL https://arxiv.org/abs/2404.14367. [↗](https://arxiv.org/abs/2404.14367)
71. Fahim Tajwar, Yiding Jiang, Abitha Thankaraj, Sumaita Sadia Rahman, J Zico Kolter, Jeff Schneider, and Ruslan Salakhutdinov. Training a generally curious agent, 2025. URL https://arxiv.org/abs/2502.17543. [↗](https://arxiv.org/abs/2502.17543)
72. Yunhao Tang and Rémi Munos. On a few pitfalls in kl divergence gradient estimation for rl, 2025. URL https://arxiv.org/abs/2506.09477. — [arXiv:2506.09477](https://arxiv.org/abs/2506.09477) [↗](https://arxiv.org/abs/2506.09477)
73. Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, et al. Kimi k1.5: Scaling reinforcement le… — [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://md.research.elamur.ai/free/en/papers/kimi-team-2025-kimi-k1-5-scaling-reinforcement/)
74. Guillermo Valle-Pérez, Chico Q. Camargo, and Ard A. Louis. Deep learning generalizes because the parameter-function map is biased towards simple functions, 2019. URL https://arxiv.org/abs/1805.08522. — [arXiv:1805.08522](https://arxiv.org/abs/1805.08522) [↗](https://arxiv.org/abs/1805.08522)
75. Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2021. URL [https://openreview.net/forum?id=uXl3bZLkr3c](https://openreview.net/forum?id=uXl3bZLkr3c). [↗](https://openreview.net/forum?id=uXl3bZLkr3c)
76. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). In The Eleventh International Conference on Learning Representations, 2023a. URL [https://openreview.net/forum?id…](https://openreview.net/forum?id…) [↗](https://openreview.net/forum?id…)
77. Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Lucas Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, Weizhu Chen, Shuohang Wang, Simon Shaolei Du, and Yelong Shen. [Reinforcement learning for reasoning in large language models with one training example](https://md.research.elamur.ai/free/en/papers/wang-2025-reinforcement-learning-reasoning-large-language/), 2025. URL [https://arxiv.or…](https://arxiv.or…) [↗](https://arxiv.or…)
78. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Asso… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A%2E+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+Self%2Dinstruct%3A+Aligning+language+models+with+self%2Dgenerated+instructions%2E+In+Anna+Roger)
79. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Proce… — [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
80. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229–256, 1992. [find](https://scholar.google.com/scholar?q=Ronald+J+Williams%2E+Simple+statistical+gradient%2Dfollowing+algorithms+for+connectionist+reinforcement+learning%2E+Machine+learning%2C+8%3A229%E2%80%93256%2C+1992%2E)
81. Haoze Wu, Cheng Wang, Wenshuo Zhao, and Junxian He. Mirage or method? how model-task alignment induces divergent rl conclusions, 2025a. URL https://arxiv.org/abs/2508. 21188. [↗](https://arxiv.org/abs/2508)
82. Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason E Weston, and Sainbayar Sukhbaatar. [Meta-rewarding language models: Self-improving alignment with LLM-as-a-meta-judge](https://md.research.elamur.ai/free/en/papers/wu-2025-meta-rewarding-language-models-self/), 2025b. URL [https://openreview.net/forum?id=lbj0i29Z92](https://openreview.net/forum?id=lbj0i29Z92). [↗](https://openreview.net/forum?id=lbj0i29Z92)
83. Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi, and Ravi Kumar. [On memorization of large language models in logical reasoning](https://md.research.elamur.ai/free/en/papers/xie-2025-memorization-large-language-models-logical/). 2024. URL https://arxiv.org/abs/2410.23123. [↗](https://arxiv.org/abs/2410.23123)
84. Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for llm alignment? a comprehensive study, 2024. URL https://arxiv.org/abs/2404.10719. — [arXiv:2404.10719](https://arxiv.org/abs/2404.10719) [↗](https://arxiv.org/abs/2404.10719)
85. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, et al. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/), 2025. URL https://arxiv.org/abs/2505.09388. [↗](https://arxiv.org/abs/2505.09388)
86. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan D… — [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://md.research.elamur.ai/free/en/papers/yu-2025-dapo-open-source-llm-reinforcement/)
87. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. [Star: Bootstrapping reasoning with reasoning](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/). Advances in Neural Information Processing Systems, 35:15476–15488, 2022.
88. Edwin Zhang, Vincent Zhu, Naomi Saphra, Anat Kleiman, Benjamin L. Edelman, Milind Tambe, Sham M. Kakade, and eran malach. [Transcendence: Generative models can outperform the experts that train them](https://md.research.elamur.ai/free/en/papers/zhang-2024-transcendence-generative-models-outperform-experts/). In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://op…
89. Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. [Generative verifiers: Reward modeling as next-token prediction](https://md.research.elamur.ai/free/en/papers/zhang-2024-generative-verifiers-reward-modeling-as/), 2025a. URL https://arxiv.org/abs/2408.15240. [↗](https://arxiv.org/abs/2408.15240)
90. Yanzhi Zhang, Zhaoxi Zhang, Haoxiang Guan, Yilin Cheng, Yitong Duan, Chen Wang, Yue Wang, Shuxin Zheng, and Jiyan He. No free lunch: Rethinking internal feedback for llm reasoning, 2025b. URL https://arxiv.org/abs/2506.17219. — [arXiv:2506.17219](https://arxiv.org/abs/2506.17219) [↗](https://arxiv.org/abs/2506.17219)
91. Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Yang Yue, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. [Absolute zero: Reinforced self-play reasoning with zero data](https://md.research.elamur.ai/free/en/papers/zhao-2025-absolute-zero-reinforced-self-play/), 2025a. URL https://arxiv.org/abs/2505.03335. [↗](https://arxiv.org/abs/2505.03335)
92. Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. [Learning to reason without external rewards](https://md.research.elamur.ai/free/en/papers/zhao-2025-learning-reason-external-rewards/), 2025b. URL https://arxiv.org/abs/2505.19590. [↗](https://arxiv.org/abs/2505.19590)
93. Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. [Group sequence policy optimization](https://md.research.elamur.ai/free/en/papers/zheng-2025-group-sequence-policy-optimization/), 2025. URL https://arxiv.org/abs/2507.18071. [↗](https://arxiv.org/abs/2507.18071)
94. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/), 2023. URL [https://arxiv.org/](https://arxiv.org/) abs/2306.05685. [↗](https://arxiv.org/)
95. Yujun Zhou, Zhenwen Liang, Haolin Liu, Wenhao Yu, Kishan Panaganti, Linfeng Song, Dian Yu, Xiangliang Zhang, Haitao Mi, and Dong Yu. Evolving language models without labels: Majority drives selection, novelty promotes variation, 2025. URL https://arxiv.org/abs/2509. 15194. [↗](https://arxiv.org/abs/2509)
96. Yuxin Zuo, Kaiyan Zhang, Shang Qu, Li Sheng, Xuekai Zhu, Biqing Qi, Youbang Sun, Ganqu Cui, Ning Ding, and Bowen Zhou. [Ttrl: Test-time reinforcement learning](https://md.research.elamur.ai/free/en/papers/zuo-2025-ttrl-test-time-reinforcement-learning/). arXiv preprint arXiv:2504.16084, 2025.

*Updated 3 November 2026*