---
title: "VRPO — paper by Zhu et al. (2026) — summary and ideas"
description: "Reward signals in LLM post-training are often noisy, sparse or ambiguous. 39 references."
authors: ["Dingwei Zhu", "Shihan Dou", "Zhiheng Xi", "Senjie Jin", "Guoqiang Zhang", "Jiazheng Zhang", "Junjie Ye", "Mingxu Chai", "Enyu Zhou", "Ming Zhang", "Yuhui Wang", "Caishuang Huang", "Chenhao Huang", "Yunke Zhang", "Yuran Wang", "Tao Gui", "Qi Zhang", "Xipeng Qiu", "Xuanjing Huang"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.1103/"
doi: "10.18653/v1/2026.acl-long.1103"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhu-2026-vrpo-rethinking-value-modeling-robust/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhu-2026-vrpo-rethinking-value-modeling-robust/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/55/) / VRPO: Rethinking Value Modeling for Robust RL under Noisy Supervision in LLM Post-Training

# VRPO: Rethinking Value Modeling for Robust RL under Noisy Supervision in LLM Post-Training

**Authors:** Dingwei Zhu, [Shihan Dou](https://md.research.elamur.ai/free/en/authors/shihan-dou/), [Zhiheng Xi](https://md.research.elamur.ai/free/en/authors/zhiheng-xi/), [Senjie Jin](https://md.research.elamur.ai/free/en/authors/senjie-jin/), [Guoqiang Zhang](https://md.research.elamur.ai/free/en/authors/guoqiang-zhang/), [Jiazheng Zhang](https://md.research.elamur.ai/free/en/authors/jiazheng-zhang/), [Junjie Ye](https://md.research.elamur.ai/free/en/authors/junjie-ye/), [Mingxu Chai](https://md.research.elamur.ai/free/en/authors/mingxu-chai/), [Enyu Zhou](https://md.research.elamur.ai/free/en/authors/enyu-zhou/), [Ming Zhang](https://md.research.elamur.ai/free/en/authors/ming-zhang/), [Yuhui Wang](https://md.research.elamur.ai/free/en/authors/yuhui-wang/), [Caishuang Huang](https://md.research.elamur.ai/free/en/authors/caishuang-huang/), Chenhao Huang, [Yunke Zhang](https://md.research.elamur.ai/free/en/authors/yunke-zhang/), Yuran Wang, [Tao Gui](https://md.research.elamur.ai/free/en/authors/tao-gui/), [Qi Zhang](https://md.research.elamur.ai/free/en/authors/qi-zhang/), [Xipeng Qiu](https://md.research.elamur.ai/free/en/authors/xipeng-qiu/) and [Xuanjing Huang](https://md.research.elamur.ai/free/en/authors/xuanjing-huang/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.1103/) · [PDF](https://aclanthology.org/2026.acl-long.1103.pdf) · [DOI 10.18653/v1/2026.acl-long.1103](https://doi.org/10.18653/v1/2026.acl-long.1103) · [OpenAlex](https://openalex.org/W7166817428)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary

Reward signals in LLM post-training are often noisy, sparse or ambiguous. When such errors pass through GAE, advantage estimates become distorted: credit drifts toward the end of a sequence (length hacking) or onto irrelevant tokens. Prior robust-RL work mostly cleans the reward model or filters data. This paper instead strengthens the critic. VRPO turns the PPO value model into a noise regulator in two ways. First, a variational information bottleneck compresses inputs into a Gaussian latent and penalizes its KL divergence from a standard prior. Second, a frozen LM head attached to the value model supplies entropy- and perplexity-based auxiliary losses on high-uncertainty tokens, keeping the critic aligned with language semantics. The authors evaluate it across three task types (mathematical problem solving, scientific question answering and dialogues spanning several turns), using both rule-based and model-based rewards. VRPO beats PPO, GRPO, Reinforce++ and Dr.GRPO on Qwen2.5-7B and Qwen3-8B. The main takeaway is that the dense supervision from the value model is an underused lever for robustness to imperfect rewards.

## Key points

- **Training-time results with rule-based rewards.** The benchmarks were MATH500, AIME24, Minerva-Math, AMC23, SampleQA, GPQA and HLE.
  - Qwen2.5-7B (cold start): average accuracy rose to 25.08%, compared with 23.65% for Dr.GRPO, 20.60% for PPO and 14.41% for GRPO.
  - Qwen3-8B: average accuracy was 38.14%, compared with about 36.6% for the best baselines.
- **AIME24 gains.** Qwen2.5-7B reached 23.33% with VRPO, against 13.33% for PPO and Dr.GRPO. Qwen3-8B reached 46.67%.
- **Ablation of training objectives on math under reward perturbation.** Using entropy- and perplexity-minimization losses on the value model gave 42.41% average accuracy, versus 39.51% with plain cross-entropy (+2.9 points). The value-model variant also beat applying the same objectives to the actor model by about 4 points.
- **Selective token supervision.** Applying the auxiliary losses to 80% of high-uncertainty tokens worked best (42.41%). Using 100% of tokens gave 41.15%, and the IB-only setup gave 38.52%.
- **Stated limitations.** Rule-based-reward experiments covered only math and science, each trained and evaluated on a single dataset. Ablations were run mainly in the test-time optimization setting.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Dingwei Zhu, Shihan Dou, Zhiheng Xi, Senjie Jin, Guoqiang Zhang, Jiazheng Zhang, Junjie Ye, Mingxu Chai, Enyu Zhou, Ming Zhang, Yuhui Wang, Caishuang Huang, Chenhao Huang, Yunke Zhang, Yuran Wang, Tao Gui, Qi Zhang, Xipeng Qiu and Xuanjing Huang. 2026. VRPO: Rethinking Value Modeling for Robust RL under Noisy Supervision in LLM Post-Training. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhu2026vrpo,
  title = {{VRPO: Rethinking Value Modeling for Robust RL under Noisy Supervision in LLM Post-Training}},
  author = {Dingwei Zhu, Shihan Dou, Zhiheng Xi, Senjie Jin, Guoqiang Zhang, Jiazheng Zhang, Junjie Ye, Mingxu Chai, Enyu Zhou, Ming Zhang, Yuhui Wang, Caishuang Huang, Chenhao Huang, Yunke Zhang, Yuran Wang, Tao Gui, Qi Zhang, Xipeng Qiu and Xuanjing Huang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1103},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1103}
}
```

- [← previous Vrittanta-EN: A Benchmark Dataset for Event Trigger Detection and Classification Advancing Event Understanding in English Narrative Discourse](https://md.research.elamur.ai/free/en/papers/kirti-2026-vrittanta-en-benchmark-dataset-event/)
- [next → VUPMC: A New Political Metaphor Corpus in Mandarin Chinese](https://md.research.elamur.ai/free/en/papers/tan-2026-vupmc-new-political-metaphor-corpus/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/55/)

## References

39 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Eslam Abdelaleem, Ilya Nemenman, and K. Michael Martini. 2025. Deep variational multivariate information bottleneck - a framework for variational losses. Journal of Machine Learning Research, 26(140):1– 50. [find](https://scholar.google.com/scholar?q=Eslam+Abdelaleem%2C+Ilya+Nemenman%2C+and+K%2E+Michael+Martini%2E+2025%2E+Deep+variational+multivariate+information+bottleneck+%2D+a+framework+for+variational+losses%2E+Journal+of+Machine+Learning+Research%2C+26%28140%29%3A)
2. Marc G. Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos C. Machado, Subhodeep Moitra, Sameera S. Ponda, and Ziyun Wang. 2020. Autonomous navigation of stratospheric balloons using reinforcement learning. Nature, 588:77 – 82. [find](https://scholar.google.com/scholar?q=Marc+G%2E+Bellemare%2C+Salvatore+Candido%2C+Pablo+Samuel+Castro%2C+Jun+Gong%2C+Marlos+C%2E+Machado%2C+Subhodeep+Moitra%2C+Sameera+S%2E+Ponda%2C+and+Ziyun+Wang%2E+2020%2E+Autonomous+navigation+of+stratospheric+balloons+using+)
3. Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar… [find](https://scholar.google.com/scholar?q=Stephen+Casper%2C+Xander+Davies%2C+Claudia+Shi%2C+Thomas+Krendl+Gilbert%2C+J%C3%A9r%C3%A9my+Scheurer%2C+Javier+Rando%2C+Rachel+Freedman%2C+Tomasz+Korbak%2C+David+Lindner%2C+Pedro+Freire%2C+Tony+Wang%2C+Samuel+Marks%2C+Charbel%2DRapha%C3%ABl+)
4. Shreyas Chaudhari, Pranjal Aggarwal, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, Karthik Narasimhan, Ameet Deshpande, and Bruno Castro da Silva. 2024. Rlhf deciphered: A critical analysis of reinforcement learning from human feedback for llms. Preprint, arXiv:2404.08555. — [arXiv:2404.08555](https://arxiv.org/abs/2404.08555) [find](https://scholar.google.com/scholar?q=Shreyas+Chaudhari%2C+Pranjal+Aggarwal%2C+Vishvak+Murahari%2C+Tanmay+Rajpurohit%2C+Ashwin+Kalyan%2C+Karthik+Narasimhan%2C+Ameet+Deshpande%2C+and+Bruno+Castro+da+Silva%2E+2024%2E+Rlhf+deciphered%3A+A+critical+analysis+of+r)
5. Yuanpei Chen, Tianhao Wu, Shengjie Wang, Xidong Feng, Jiechuang Jiang, Stephen Marcus McAleer, Yiran Geng, Hao Dong, Zongqing Lu, Song-Chun Zhu, and Yaodong Yang. 2022. Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning. arXiv e-prints, arXiv:2206.08686. [find](https://scholar.google.com/scholar?q=Yuanpei+Chen%2C+Tianhao+Wu%2C+Shengjie+Wang%2C+Xidong+Feng%2C+Jiechuang+Jiang%2C+Stephen+Marcus+McAleer%2C+Yiran+Geng%2C+Hao+Dong%2C+Zongqing+Lu%2C+Song%2DChun+Zhu%2C+and+Yaodong+Yang%2E+2022%2E+Towards+Human%2DLevel+Bimanual+De)
6. Jie Cheng, Gang Xiong, Xingyuan Dai, Qinghai Miao, Yisheng Lv, and Fei-Yue Wang. 2024. RIME: Robust Preference-based Reinforcement Learning with Noisy Preferences. arXiv e-prints, arXiv:2402.17257. — [arXiv:2402.17257](https://arxiv.org/abs/2402.17257) [find](https://scholar.google.com/scholar?q=Jie+Cheng%2C+Gang+Xiong%2C+Xingyuan+Dai%2C+Qinghai+Miao%2C+Yisheng+Lv%2C+and+Fei%2DYue+Wang%2E+2024%2E+RIME%3A+Robust+Preference%2Dbased+Reinforcement+Learning+with+Noisy+Preferences%2E+arXiv+e%2Dprints%2C+arXiv%3A2402%2E17257%2E+%E2%80%94+)
7. Ruoxi Cheng, Haoxuan Ma, Weixin Wang, Zhiqiang Wang, Xiaoshuang Jia, Simeng Qin, Xiaochun Cao, Yang Liu, and Xiaojun Jia. 2025. Inverse Reinforcement Learning with Dynamic Reward Scaling for LLM Alignment. arXiv e-prints, arXiv:2503.18991. — [arXiv:2503.18991](https://arxiv.org/abs/2503.18991) [find](https://scholar.google.com/scholar?q=Ruoxi+Cheng%2C+Haoxuan+Ma%2C+Weixin+Wang%2C+Zhiqiang+Wang%2C+Xiaoshuang+Jia%2C+Simeng+Qin%2C+Xiaochun+Cao%2C+Yang+Liu%2C+and+Xiaojun+Jia%2E+2025%2E+Inverse+Reinforcement+Learning+with+Dynamic+Reward+Scaling+for+LLM+Align)
8. Bin Dai, Chen Zhu, and David Wipf. 2018. Compressing Neural Networks using the Variational Information Bottleneck. arXiv e-prints, arXiv:1802.10399. — [arXiv:1802.10399](https://arxiv.org/abs/1802.10399) [find](https://scholar.google.com/scholar?q=Bin+Dai%2C+Chen+Zhu%2C+and+David+Wipf%2E+2018%2E+Compressing+Neural+Networks+using+the+Variational+Information+Bottleneck%2E+arXiv+e%2Dprints%2C+arXiv%3A1802%2E10399%2E+%E2%80%94+%5BarXiv%3A1802%2E10399%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1802%2E103)
9. Leo Gao, John Schulman, and Jacob Hilton. 2022. [Scaling Laws for Reward Model Overoptimization](https://md.research.elamur.ai/free/en/papers/gao-2022-scaling-laws-reward-model-overoptimization/). arXiv e-prints, arXiv:2210.10760.
10. Anirudh Goyal, Riashat Islam, Daniel Strouse, Zafarali Ahmed, Matthew Botvinick, Hugo Larochelle, Yoshua Bengio, and Sergey Levine. 2019. InfoBot: Transfer and Exploration via the Information Bottleneck. arXiv e-prints, arXiv:1901.10902. — [arXiv:1901.10902](https://arxiv.org/abs/1901.10902) [find](https://scholar.google.com/scholar?q=Anirudh+Goyal%2C+Riashat+Islam%2C+Daniel+Strouse%2C+Zafarali+Ahmed%2C+Matthew+Botvinick%2C+Hugo+Larochelle%2C+Yoshua+Bengio%2C+and+Sergey+Levine%2E+2019%2E+InfoBot%3A+Transfer+and+Exploration+via+the+Information+Bottlene)
11. Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. 2019. Dream to Control: Learning Behaviors by Latent Imagination. arXiv e-prints, arXiv:1912.01603. [find](https://scholar.google.com/scholar?q=Danijar+Hafner%2C+Timothy+Lillicrap%2C+Jimmy+Ba%2C+and+Mohammad+Norouzi%2E+2019%2E+Dream+to+Control%3A+Learning+Behaviors+by+Latent+Imagination%2E+arXiv+e%2Dprints%2C+arXiv%3A1912%2E01603%2E)
12. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). Preprint, arXiv:2103.03874.
13. Yu-Heng Hung, Ping-Chun Hsieh, Akshay Mete, and P. R. Kumar. 2023. Value-biased maximum likelihood estimation for model-based reinforcement learning in discounted linear mdps. Preprint, arXiv:2310.11515. — [arXiv:2310.11515](https://arxiv.org/abs/2310.11515) [find](https://scholar.google.com/scholar?q=Yu%2DHeng+Hung%2C+Ping%2DChun+Hsieh%2C+Akshay+Mete%2C+and+P%2E+R%2E+Kumar%2E+2023%2E+Value%2Dbiased+maximum+likelihood+estimation+for+model%2Dbased+reinforcement+learning+in+discounted+linear+mdps%2E+Preprint%2C+arXiv%3A2310%2E115)
14. Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. 2018. Reward learning from human preferences and demonstrations in Atari. arXiv e-prints, arXiv:1811.06521. — [arXiv:1811.06521](https://arxiv.org/abs/1811.06521) [find](https://scholar.google.com/scholar?q=Borja+Ibarz%2C+Jan+Leike%2C+Tobias+Pohlen%2C+Geoffrey+Irving%2C+Shane+Legg%2C+and+Dario+Amodei%2E+2018%2E+Reward+learning+from+human+preferences+and+demonstrations+in+Atari%2E+arXiv+e%2Dprints%2C+arXiv%3A1811%2E06521%2E+%E2%80%94+%5BarX)
15. Cassidy Laidlaw, Shivam Singhal, and Anca Dragan. 2025. Correlated proxies: A new definition and improved mitigation for reward hacking. Preprint, arXiv:2403.03185. — [arXiv:2403.03185](https://arxiv.org/abs/2403.03185) [find](https://scholar.google.com/scholar?q=Cassidy+Laidlaw%2C+Shivam+Singhal%2C+and+Anca+Dragan%2E+2025%2E+Correlated+proxies%3A+A+new+definition+and+improved+mitigation+for+reward+hacking%2E+Preprint%2C+arXiv%3A2403%2E03185%2E+%E2%80%94+%5BarXiv%3A2403%2E03185%5D%28https%3A%2F%2Farxiv%2E)
16. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. 2022. [Solving quantitative reasoning problems with language models](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/). Preprint, arXiv:2…
17. Xize Liang, Chao Chen, Shuang Qiu, Jie Wang, Yue Wu, Zhihang Fu, Zhihao Shi, Feng Wu, and Jieping Ye. 2024. ROPO: Robust Preference Optimization for Large Language Models. arXiv e-prints, arXiv:2404.04102. — [arXiv:2404.04102](https://arxiv.org/abs/2404.04102) [find](https://scholar.google.com/scholar?q=Xize+Liang%2C+Chao+Chen%2C+Shuang+Qiu%2C+Jie+Wang%2C+Yue+Wu%2C+Zhihang+Fu%2C+Zhihao+Shi%2C+Feng+Wu%2C+and+Jieping+Ye%2E+2024%2E+ROPO%3A+Robust+Preference+Optimization+for+Large+Language+Models%2E+arXiv+e%2Dprints%2C+arXiv%3A2404%2E0)
18. Jiahang Lin, Kai Hu, Binghai Wang, Yuhao Zhou, Zhiheng Xi, Honglin Guo, Shichun Liu, Junzhe Wang, Shihan Dou, Enyu Zhou, Hang Yan, Zhenhua Han, Tao Gui, Qi Zhang, and Xuanjing Huang. 2026. Mm-doc-r1: Training agents for long document visual question answering through multi-turn reinforcement learnin… [find](https://scholar.google.com/scholar?q=Jiahang+Lin%2C+Kai+Hu%2C+Binghai+Wang%2C+Yuhao+Zhou%2C+Zhiheng+Xi%2C+Honglin+Guo%2C+Shichun+Liu%2C+Junzhe+Wang%2C+Shihan+Dou%2C+Enyu+Zhou%2C+Hang+Yan%2C+Zhenhua+Han%2C+Tao+Gui%2C+Qi+Zhang%2C+and+Xuanjing+Huang%2E+2026%2E+Mm%2Ddoc%2Dr1%3A+)
19. Yuchun Miao, Sen Zhang, Liang Ding, Rong Bao, Lefei Zhang, and Dacheng Tao. 2024. InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling. arXiv e-prints, arXiv:2402.09345. — [arXiv:2402.09345](https://arxiv.org/abs/2402.09345) [find](https://scholar.google.com/scholar?q=Yuchun+Miao%2C+Sen+Zhang%2C+Liang+Ding%2C+Rong+Bao%2C+Lefei+Zhang%2C+and+Dacheng+Tao%2E+2024%2E+InfoRM%3A+Mitigating+Reward+Hacking+in+RLHF+via+Information%2DTheoretic+Reward+Modeling%2E+arXiv+e%2Dprints%2C+arXiv%3A2402%2E09345%2E)
20. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. [s1: Simple test-time scaling](https://md.research.elamur.ai/free/en/papers/muennighoff-2025-s1-simple-test-time-scaling/). Preprint, arXiv:2501.19393.
21. Julien Perolat, Bart De Vylder, Daniel Hennes, Eugene Tarassov, Florian Strub, Vincent de Boer, Paul Muller, Jerome T. Connor, Neil Burch, Thomas Anthony, Stephen McAleer, Romuald Elie, Sarah H. Cen, Zhe Wang, Audrunas Gruslys, Aleksandra Malysheva, Mina Khan, Sherjil Ozair, Finbarr Timbers, and 15… [find](https://scholar.google.com/scholar?q=Julien+Perolat%2C+Bart+De+Vylder%2C+Daniel+Hennes%2C+Eugene+Tarassov%2C+Florian+Strub%2C+Vincent+de+Boer%2C+Paul+Muller%2C+Jerome+T%2E+Connor%2C+Neil+Burch%2C+Thomas+Anthony%2C+Stephen+McAleer%2C+Romuald+Elie%2C+Sarah+H%2E+Cen%2C+)
22. Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy, Oliver Zhang, Mantas Mazeika, and 1090 others. 2025. Humanity’s last e… — [Humanity's Last Exam](https://md.research.elamur.ai/free/en/papers/phan-2025-humanity-s-last-exam/)
23. Riccardo Poiani, Gabriele Curti, Alberto Maria Metelli, and Marcello Restelli. 2024. Inverse Reinforcement Learning with Sub-optimal Experts. arXiv e-prints, arXiv:2401.03857. — [arXiv:2401.03857](https://arxiv.org/abs/2401.03857) [find](https://scholar.google.com/scholar?q=Riccardo+Poiani%2C+Gabriele+Curti%2C+Alberto+Maria+Metelli%2C+and+Marcello+Restelli%2E+2024%2E+Inverse+Reinforcement+Learning+with+Sub%2Doptimal+Experts%2E+arXiv+e%2Dprints%2C+arXiv%3A2401%2E03857%2E+%E2%80%94+%5BarXiv%3A2401%2E03857%5D%28htt)
24. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. [find](https://scholar.google.com/scholar?q=Qwen%2C+%3A%2C+An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+Huan+Lin%2C+Jian+Yang%2C+Jianhong+Tu%2C+Jianwei+Zhang%2C+Jianxin+Yang%2C+Jiaxi+)
25. John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. 2018. High-dimensional continuous control using generalized advantage estimation. Preprint, arXiv:1506.02438. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Philipp+Moritz%2C+Sergey+Levine%2C+Michael+Jordan%2C+and+Pieter+Abbeel%2E+2018%2E+High%2Ddimensional+continuous+control+using+generalized+advantage+estimation%2E+Preprint%2C+arXiv%3A1506%2E02438%2E)
26. Wei Sun, Wen Yang, Pu Jian, Qianlong Du, Fuwei Cui, Shuo Ren, and Jiajun Zhang. 2025. Ktae: A model-free algorithm to key-tokens advantage estimation in mathematical reasoning. Preprint, arXiv:2505.16826. — [arXiv:2505.16826](https://arxiv.org/abs/2505.16826) [find](https://scholar.google.com/scholar?q=Wei+Sun%2C+Wen+Yang%2C+Pu+Jian%2C+Qianlong+Du%2C+Fuwei+Cui%2C+Shuo+Ren%2C+and+Jiajun+Zhang%2E+2025%2E+Ktae%3A+A+model%2Dfree+algorithm+to+key%2Dtokens+advantage+estimation+in+mathematical+reasoning%2E+Preprint%2C+arXiv%3A2505%2E16)
27. M-A-P Team, Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, Kang Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, Chujie Zheng, Kaixing Deng, Shuyue Guo, Shian Jia, Sichao Jiang, Yiyan Liao, Rui Li, Qinrui Li, and 76 others. 2025. [Supergpqa: Scaling llm evaluation across 285 g…](https://md.research.elamur.ai/free/en/papers/team-2025-supergpqa-scaling-llm-evaluation-across/)
28. Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, Songyang Gao, Nuo Xu, Yuhao Zhou, Xiaoran Fan, Zhiheng Xi, Jun Zhao, Xiao Wang, Tao Ji, Hang Yan, and 8 others. 2024a. [Secrets of RLHF in Large Language Models Part II: Reward Modelin…](https://md.research.elamur.ai/free/en/papers/wang-2024-secrets-rlhf-large-language-models/)
29. Chenglong Wang, Yang Gan, Yifu Huo, Yongyu Mu, Murun Yang, Qiaozhi He, Tong Xiao, Chunliang Zhang, Tongran Liu, Quan Du, Di Yang, and Jingbo Zhu. 2024b. RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data. arXiv e-prints, arXiv:2408.12109. — [arXiv:2408.12109](https://arxiv.org/abs/2408.12109) [find](https://scholar.google.com/scholar?q=Chenglong+Wang%2C+Yang+Gan%2C+Yifu+Huo%2C+Yongyu+Mu%2C+Murun+Yang%2C+Qiaozhi+He%2C+Tong+Xiao%2C+Chunliang+Zhang%2C+Tongran+Liu%2C+Quan+Du%2C+Di+Yang%2C+and+Jingbo+Zhu%2E+2024b%2E+RoVRM%3A+A+Robust+Visual+Reward+Model+Optimized+v)
30. Xiao Wang, Shihan Dou, Limao Xiong, Yicheng Zou, Qi Zhang, Tao Gui, Liang Qiao, Zhanzhan Cheng, and Xuanjing Huang. 2022. MINER: Improving Out-of-Vocabulary Named Entity Recognition from an Information Theoretic Perspective. arXiv e-prints, arXiv:2204.04391. [find](https://scholar.google.com/scholar?q=Xiao+Wang%2C+Shihan+Dou%2C+Limao+Xiong%2C+Yicheng+Zou%2C+Qi+Zhang%2C+Tao+Gui%2C+Liang+Qiao%2C+Zhanzhan+Cheng%2C+and+Xuanjing+Huang%2E+2022%2E+MINER%3A+Improving+Out%2Dof%2DVocabulary+Named+Entity+Recognition+from+an+Informatio)
31. Yining Wang, Jinman Zhao, Chuangxin Zhao, Shuhao Guan, Gerald Penn, and Shinan Liu. 2025. λ-grpo: Unifying the grpo frameworks with learnable token preferences. Preprint, arXiv:2510.06870. — [arXiv:2510.06870](https://arxiv.org/abs/2510.06870) [find](https://scholar.google.com/scholar?q=Yining+Wang%2C+Jinman+Zhao%2C+Chuangxin+Zhao%2C+Shuhao+Guan%2C+Gerald+Penn%2C+and+Shinan+Liu%2E+2025%2E+%CE%BB%2Dgrpo%3A+Unifying+the+grpo+frameworks+with+learnable+token+preferences%2E+Preprint%2C+arXiv%3A2510%2E06870%2E+%E2%80%94+%5BarXiv%3A25)
32. Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. 2024. [Measuring short-form factuality in large language models](https://md.research.elamur.ai/free/en/papers/wei-2024-measuring-short-form-factuality-large/). Preprint, arXiv:2411.04368.
33. Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, Haosheng Zou, Yongchao Deng, Shousheng Jia, and Xiangzheng Zhang. 2025. [Light-r1: Curriculum sft, dpo and rl for long cot from scratch and beyond](https://md.research.elamur.ai/free/en/papers/wen-2025-light-r1-curriculum-sft-dpo/). Preprint, arXiv:2503.10460.
34. Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jiawei Chen, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. 2024. Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization. arXiv e-prints, arXiv:2407.07880. — [arXiv:2407.07880](https://arxiv.org/abs/2407.07880) [find](https://scholar.google.com/scholar?q=Junkang+Wu%2C+Yuexiang+Xie%2C+Zhengyi+Yang%2C+Jiancan+Wu%2C+Jiawei+Chen%2C+Jinyang+Gao%2C+Bolin+Ding%2C+Xiang+Wang%2C+and+Xiangnan+He%2E+2024%2E+Towards+Robust+Alignment+of+Language+Models%3A+Distributionally+Robustifying+)
35. Mingqi Wu, Zhihao Zhang, Qiaole Dong, Zhiheng Xi, Jun Zhao, Senjie Jin, Xiaoran Fan, Yuhao Zhou, Huijie Lv, Ming Zhang, Yanwei Fu, Qin Liu, Songyang Zhang, and Qi Zhang. 2025. Reasoning or memorization? unreliable results of reinforcement learning due to data contamination. Preprint, arXiv:2507.1053… — [arXiv:2507.10532](https://arxiv.org/abs/2507.10532) [find](https://scholar.google.com/scholar?q=Mingqi+Wu%2C+Zhihao+Zhang%2C+Qiaole+Dong%2C+Zhiheng+Xi%2C+Jun+Zhao%2C+Senjie+Jin%2C+Xiaoran+Fan%2C+Yuhao+Zhou%2C+Huijie+Lv%2C+Ming+Zhang%2C+Yanwei+Fu%2C+Qin+Liu%2C+Songyang+Zhang%2C+and+Qi+Zhang%2E+2025%2E+Reasoning+or+memorizatio)
36. Algorithm 1: VRPO training process An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/).…
37. Tongtian Yue, Jie Cheng, Longteng Guo, Xingyuan Dai, Zijia Zhao, Xingjian He, Gang Xiong, Yisheng Lv, and Jing Liu. 2024. SC-Tune: Unleashing Self-Consistent Referential Comprehension in Large Vision Language Models. arXiv e-prints, arXiv:2403.13263. — [arXiv:2403.13263](https://arxiv.org/abs/2403.13263) [find](https://scholar.google.com/scholar?q=Tongtian+Yue%2C+Jie+Cheng%2C+Longteng+Guo%2C+Xingyuan+Dai%2C+Zijia+Zhao%2C+Xingjian+He%2C+Gang+Xiong%2C+Yisheng+Lv%2C+and+Jing+Liu%2E+2024%2E+SC%2DTune%3A+Unleashing+Self%2DConsistent+Referential+Comprehension+in+Large+Vision+)
38. Jiazheng Zhang, Wenqing Jing, Zizhuo Zhang, Zhiheng Xi, Shihan Dou, Rongxiang Weng, Jiahuan Li, Jingang Wang, Mingxu Chai, Shibo Hong, Tao Gui, and Qi Zhang. 2025. Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment. arXiv e-prints, arXiv:2505.10597. — [arXiv:2505.10597](https://arxiv.org/abs/2505.10597) [find](https://scholar.google.com/scholar?q=Jiazheng+Zhang%2C+Wenqing+Jing%2C+Zizhuo+Zhang%2C+Zhiheng+Xi%2C+Shihan+Dou%2C+Rongxiang+Weng%2C+Jiahuan+Li%2C+Jingang+Wang%2C+Mingxu+Chai%2C+Shibo+Hong%2C+Tao+Gui%2C+and+Qi+Zhang%2E+2025%2E+Two+Minds+Better+Than+One%3A+Collabora)
39. Ming Zhou, Jun Luo, Julian Villella, Yaodong Yang, David Rusu, Jiayu Miao, Weinan Zhang, Montgomery Alban, Iman Fadakar, Zheng Chen, Aurora Chongxi Huang, Ying Wen, Kimia Hassanzadeh, Daniel Graves, Dong Chen, Zhengbang Zhu, Nhat Nguyen, Mohamed Elsayed, Kun Shao, and 18 others. 2020. SMARTS: Scalab… — [arXiv:2010.09776](https://arxiv.org/abs/2010.09776) [find](https://scholar.google.com/scholar?q=Ming+Zhou%2C+Jun+Luo%2C+Julian+Villella%2C+Yaodong+Yang%2C+David+Rusu%2C+Jiayu+Miao%2C+Weinan+Zhang%2C+Montgomery+Alban%2C+Iman+Fadakar%2C+Zheng+Chen%2C+Aurora+Chongxi+Huang%2C+Ying+Wen%2C+Kimia+Hassanzadeh%2C+Daniel+Graves%2C+D)

*Updated 2 November 2026*