---
title: "Beyond Length Scaling — paper by Zhang et al. (2026)"
description: "The authors propose Mix-GRM. 65 references. Generative reward models (GRMs) usually improve by writing longer evaluation rationales."
authors: ["Qiyuan Zhang", "Yufei Wang", "Tianhe Wu", "Can Xu", "Qingfeng Sun", "Kai Zheng", "Xue Liu", "Chen Ma"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.709/"
doi: "10.18653/v1/2026.findings-acl.709"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2026-beyond-length-scaling-synergizing-breadth/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2026-beyond-length-scaling-synergizing-breadth/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/24/) / Beyond Length Scaling: Synergizing Breadth and Depth for Generative Reward Models

# Beyond Length Scaling: Synergizing Breadth and Depth for Generative Reward Models

**Authors:** [Qiyuan Zhang](https://md.research.elamur.ai/free/en/authors/qiyuan-zhang/), [Yufei Wang](https://md.research.elamur.ai/free/en/authors/yufei-wang/), Tianhe Wu, [Can Xu](https://md.research.elamur.ai/free/en/authors/can-xu/), [Qingfeng Sun](https://md.research.elamur.ai/free/en/authors/qingfeng-sun/), [Kai Zheng](https://md.research.elamur.ai/free/en/authors/kai-zheng/), [Xue Liu](https://md.research.elamur.ai/free/en/authors/xue-liu/) and [Chen Ma](https://md.research.elamur.ai/free/en/authors/chen-ma/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.709/) · [PDF](https://aclanthology.org/2026.findings-acl.709.pdf) · [DOI 10.18653/v1/2026.findings-acl.709](https://doi.org/10.18653/v1/2026.findings-acl.709) · [OpenAlex](https://openalex.org/W7166873564)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Generative reward models (GRMs) usually improve by writing longer evaluation rationales. This paper argues that length is the wrong lever and that the reasoning style must fit the task. The authors propose Mix-GRM. An LLM breaks free-form judge rationales into atomic Principle–Judgment–Verdict units, which are then recombined in two ways. Breadth-CoT merges units that cover many different evaluation principles. Depth-CoT first solves the instruction and then re-judges the candidate responses against that solution. Breadth-CoT data is paired with subjective preference tasks and Depth-CoT data with objective correctness tasks. The model is trained with SFT on this mixture and then refined with RLVR on ordinary reward datasets that carry only final labels. On five benchmarks, Mix-GRM sets a new state of the art and helps in DPO and Best-of-N settings. Using the wrong reasoning style for a task lowers accuracy. After RLVR, the model chooses the fitting style on its own, which suggests GRM work should optimize reasoning structure rather than output length.

## Key points
- Mix-GRM outperforms the strongest open-source baseline by 3.5% across five reward benchmarks. It also beats strong baselines such as Skywork-Reward and FARE-8B.
- Each rationale is split into roughly 3–5 Principle–Judgment–Verdict units. This keeps the format uniform, so performance gains come from the reasoning mechanism rather than writing style.
- The two styles have a double dissociation:
  - Breadth-CoT helps preference tasks but hurts correctness tasks.
  - Depth-CoT does the opposite.
- RLVR acts as a "switching amplifier." It gives Mix-GRM a larger gain than a Base-GRM and pushes the model toward Breadth-CoT for preference tasks and Depth-CoT for correctness tasks.
- In the Breadth-CoT ablation:
  - Accuracy rises steadily as the number of aggregated rationales grows from N=1 to N=4.
  - Keeping only the 10 most frequent principles works best (71.8 / 72.7), ahead of the full principle pool and random selection.
- The synthesized data and models are released on Hugging Face, and the code is on GitHub.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [City University of Hong Kong](https://md.research.elamur.ai/free/en/universities/city-university-of-hong-kong/) — University, Hong Kong, China · 59 works · 3,480 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works

## Cite

Cite as: Qiyuan Zhang, Yufei Wang, Tianhe Wu, Can Xu, Qingfeng Sun, Kai Zheng, Xue Liu and Chen Ma. 2026. Beyond Length Scaling: Synergizing Breadth and Depth for Generative Reward Models. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhang2026beyond,
  title = {{Beyond Length Scaling: Synergizing Breadth and Depth for Generative Reward Models}},
  author = {Qiyuan Zhang, Yufei Wang, Tianhe Wu, Can Xu, Qingfeng Sun, Kai Zheng, Xue Liu and Chen Ma},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.709},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.709}
}
```

- [← previous Beyond Genre Categories: How Narrative Pattern Coherence and Spanning Distance Shape Film Success](https://md.research.elamur.ai/free/en/papers/wang-2026-beyond-genre-categories-how-narrative/)
- [next → Beyond Local vs. External: A Game-Theoretic Framework for Trustworthy Knowledge Acquisition](https://md.research.elamur.ai/free/en/papers/yao-2026-beyond-local-vs-external-game/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/24/)

## References

65 references, 40 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan D. Chang, and Prithviraj Ammanabrolu. 2024. Critique-out-loud reward models. Preprint, arXiv:2408.11791. [find](https://scholar.google.com/scholar?q=Zachary+Ankner%2C+Mansheej+Paul%2C+Brandon+Cui%2C+Jonathan+D%2E+Chang%2C+and+Prithviraj+Ammanabrolu%2E+2024%2E+Critique%2Dout%2Dloud+reward+models%2E+Preprint%2C+arXiv%3A2408%2E11791%2E)
2. Maciej Besta, Florim Memedi, Zhenyu Zhang, Robert Gerstenberger, Guangyuan Piao, Nils Blach, Piotr Nyczyk, Marcin Copik, Grzegorz Kwaśniewski, Jürgen Müller, Lukas Gianinazzi, Ales Kubicek, Hubert Niewiadomski, Aidan O’Mahony, Onur Mutlu, and Torsten Hoefler. 2025. Demystifying chains, trees, and g… [find](https://scholar.google.com/scholar?q=Maciej+Besta%2C+Florim+Memedi%2C+Zhenyu+Zhang%2C+Robert+Gerstenberger%2C+Guangyuan+Piao%2C+Nils+Blach%2C+Piotr+Nyczyk%2C+Marcin+Copik%2C+Grzegorz+Kwas%CC%81niewski%2C+J%C3%BCrgen+M%C3%BCller%2C+Lukas+Gianinazzi%2C+Ales+Kubicek%2C+Hubert+Ni)
3. Nuo Chen, Zhiyuan Hu, Qingyun Zou, Jiaying Wu, Qian Wang, Bryan Hooi, and Bingsheng He. 2025a. Judgelrm: Large reasoning models as a judge. Preprint, arXiv:2504.00050. — [arXiv:2504.00050](https://arxiv.org/abs/2504.00050) [find](https://scholar.google.com/scholar?q=Nuo+Chen%2C+Zhiyuan+Hu%2C+Qingyun+Zou%2C+Jiaying+Wu%2C+Qian+Wang%2C+Bryan+Hooi%2C+and+Bingsheng+He%2E+2025a%2E+Judgelrm%3A+Large+reasoning+models+as+a+judge%2E+Preprint%2C+arXiv%3A2504%2E00050%2E+%E2%80%94+%5BarXiv%3A2504%2E00050%5D%28https%3A%2F%2Farx)
4. Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang, Tong Zhang, Hanghang Tong, and Heng Ji. 2025b. [RM-R1: Reward modeling as reasoning](https://md.research.elamur.ai/free/en/papers/chen-2025-rm-r1-reward-modeling-reasoning/). Preprint, arXiv:2505.02387.
5. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). Preprint, arXiv:2110.14168.
6. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2024. UltraFeedback: Boosting language models with scaled ai feedback. Preprint, arXiv:2310.01377. [find](https://scholar.google.com/scholar?q=Ganqu+Cui%2C+Lifan+Yuan%2C+Ning+Ding%2C+Guanming+Yao%2C+Bingxiang+He%2C+Wei+Zhu%2C+Yuan+Ni%2C+Guotong+Xie%2C+Ruobing+Xie%2C+Yankai+Lin%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2024%2E+UltraFeedback%3A+Boosting+language+models+with+s)
7. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. [Enhancing chat language models by scaling high-quality instructional conversations](https://md.research.elamur.ai/free/en/papers/ding-2023-enhancing-chat-language-models-scaling/). In Conference on Empirical Methods in Natural Language Processing.
8. Yann Dubois, Percy Liang, and Tatsunori Hashimoto. 2024. Length-controlled alpacaeval: A simple debiasing of automatic evaluators. In Conference on Language Modeling. [find](https://scholar.google.com/scholar?q=Yann+Dubois%2C+Percy+Liang%2C+and+Tatsunori+Hashimoto%2E+2024%2E+Length%2Dcontrolled+alpacaeval%3A+A+simple+debiasing+of+automatic+evaluators%2E+In+Conference+on+Language+Modeling%2E)
9. Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios Nikolas Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. 2025. How to evaluate reward models for RLHF. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Evan+Frick%2C+Tianle+Li%2C+Connor+Chen%2C+Wei%2DLin+Chiang%2C+Anastasios+Nikolas+Angelopoulos%2C+Jiantao+Jiao%2C+Banghua+Zhu%2C+Joseph+E%2E+Gonzalez%2C+and+Ion+Stoica%2E+2025%2E+How+to+evaluate+reward+models+for+RLHF%2E+In+The)
10. Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, and Sean Hendryx. 2025. [Rubrics as rewards: Reinforcement learning beyond verifiable domains](https://md.research.elamur.ai/free/en/papers/gunjal-2025-rubrics-rewards-reinforcement-learning-verifiable/). Preprint, arXiv:2507.17746.
11. Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. 2024. [WILDGUARD: open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms](https://md.research.elamur.ai/free/en/papers/han-2024-wildguard-open-one-stop-moderation/). In International Conference on Neural Information Processing Systems.
12. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. [Measuring mathematical problem solving with the MATH dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). In Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
13. Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Mudit Verma, Kaya Stechly, Siddhant Bhambri, Lucas Saldyt, and Anil Murthy. 2024. [Llms can’t plan, but can help planning in llm-modulo frameworks](https://md.research.elamur.ai/free/en/papers/kambhampati-2024-llms-t-plan-but-help/). Preprint, arXiv:2402.01817.
14. Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. 2024. [Prometheus: Inducing fine-grained evaluation capability in language models](https://md.research.elamur.ai/free/en/papers/kim-2023-prometheus-inducing-fine-grained-evaluation/). In International Conference on Learning Representations.
15. Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. MAWPS: A math word problem repository. In Conference of the North American Chapter of the Association for Computational Linguistics, pages 1152–1157. [find](https://scholar.google.com/scholar?q=Rik+Koncel%2DKedziorski%2C+Subhro+Roy%2C+Aida+Amini%2C+Nate+Kushman%2C+and+Hannaneh+Hajishirzi%2E+2016%2E+MAWPS%3A+A+math+word+problem+repository%2E+In+Conference+of+the+North+American+Chapter+of+the+Association+for+Co)
16. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. 2024. RewardBench: Evaluating reward models for language modeling. Preprint, arXiv:2403.13787. [find](https://scholar.google.com/scholar?q=Nathan+Lambert%2C+Valentina+Pyatkin%2C+Jacob+Morrison%2C+LJ+Miranda%2C+Bill+Yuchen+Lin%2C+Khyathi+Chandu%2C+Nouha+Dziri%2C+Sachin+Kumar%2C+Tom+Zick%2C+Yejin+Choi%2C+Noah+A%2E+Smith%2C+and+Hannaneh+Hajishirzi%2E+2024%2E+RewardBen)
17. Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven C.H. Hoi. 2022. [CodeRL: mastering code generation through pretrained models and deep reinforcement learning](https://md.research.elamur.ai/free/en/papers/le-2022-coderl-mastering-code-generation-through/). In International Conference on Neural Information Processing Systems.
18. Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. 2024. Rlaif vs. rlhf: scaling reinforcement learning from human feedback with ai feedback. In International Conference on Machine Le… [find](https://scholar.google.com/scholar?q=Harrison+Lee%2C+Samrat+Phatale%2C+Hassan+Mansoor%2C+Thomas+Mesnard%2C+Johan+Ferret%2C+Kellie+Lu%2C+Colton+Bishop%2C+Ethan+Hall%2C+Victor+Carbune%2C+Abhinav+Rastogi%2C+and+Sushant+Prakash%2E+2024%2E+Rlaif+vs%2E+rlhf%3A+scaling+re)
19. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. 2024. From live data to high-quality benchmarks: The arena-hard pipeline. [find](https://scholar.google.com/scholar?q=Tianle+Li%2C+Wei%2DLin+Chiang%2C+Evan+Frick%2C+Lisa+Dunlap%2C+Banghua+Zhu%2C+Joseph+E%2E+Gonzalez%2C+and+Ion+Stoica%2E+2024%2E+From+live+data+to+high%2Dquality+benchmarks%3A+The+arena%2Dhard+pipeline%2E)
20. Yi-Chen Li, Tian Xu, Yang Yu, Xuqin Zhang, Xiong-Hui Chen, Zhongxiang Ling, Ningjing Chao, Lei Yuan, and Zhi-Hua Zhou. 2025a. Generalist reward models: Found inside large language models. Preprint, arXiv:2506.23235. — [arXiv:2506.23235](https://arxiv.org/abs/2506.23235) [find](https://scholar.google.com/scholar?q=Yi%2DChen+Li%2C+Tian+Xu%2C+Yang+Yu%2C+Xuqin+Zhang%2C+Xiong%2DHui+Chen%2C+Zhongxiang+Ling%2C+Ningjing+Chao%2C+Lei+Yuan%2C+and+Zhi%2DHua+Zhou%2E+2025a%2E+Generalist+reward+models%3A+Found+inside+large+language+models%2E+Preprint%2C+ar)
21. Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, Yingying Zhang, Fei Yin, Jiahua Dong, Zhiwei Li, Bao-Long Bi, Ling-Rui Mei, Junfeng Fang, Xiao Liang, Zhijiang Guo, and 2 others. 2025b. [From system 1 to system 2:…](https://md.research.elamur.ai/free/en/papers/li-2025-system-1-system-2-survey/)
22. Zhuang Li, Yuncheng Hua, Thuy-Trang Vu, Haolan Zhan, Lizhen Qu, and Gholamreza Haffari. 2025c. SCAR: Data selection via style consistency-aware response ranking for efficient instruction-tuning of large language models. Preprint, arXiv:2406.10882. [find](https://scholar.google.com/scholar?q=Zhuang+Li%2C+Yuncheng+Hua%2C+Thuy%2DTrang+Vu%2C+Haolan+Zhan%2C+Lizhen+Qu%2C+and+Gholamreza+Haffari%2E+2025c%2E+SCAR%3A+Data+selection+via+style+consistency%2Daware+response+ranking+for+efficient+instruction%2Dtuning+of+lar)
23. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. [Let’s verify step by step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/). Preprint, arXiv:2305.20050.
24. Chengwu Liu, Ye Yuan, Yichun Yin, Yan Xu, Xin Xu, Zaoyu Chen, Yasheng Wang, Lifeng Shang, Qun Liu, and Ming Zhang. 2025a. Safe: Enhancing mathematical reasoning in large language models via retrospective step-aware formal verification. In Annual Meeting of the Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Chengwu+Liu%2C+Ye+Yuan%2C+Yichun+Yin%2C+Yan+Xu%2C+Xin+Xu%2C+Zaoyu+Chen%2C+Yasheng+Wang%2C+Lifeng+Shang%2C+Qun+Liu%2C+and+Ming+Zhang%2E+2025a%2E+Safe%3A+Enhancing+mathematical+reasoning+in+large+language+models+via+retrospect)
25. Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. 2024. [Skywork-reward: Bag of tricks for reward modeling in llms](https://md.research.elamur.ai/free/en/papers/liu-2024-skywork-reward-bag-tricks/). Preprint, arXiv:2410.18451.
26. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and LINGMING ZHANG. 2023. [Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation](https://md.research.elamur.ai/free/en/papers/liu-2023-your-code-generated-chatgpt-really/). In Conference on Neural Information Processing Systems.
27. Tianci Liu, Ran Xu, Tony Yu, Ilgee Hong, Carl Yang, Tuo Zhao, and Haoyu Wang. 2025b. [OpenRubrics: Towards scalable synthetic rubric generation for reward modeling and llm alignment](https://md.research.elamur.ai/free/en/papers/liu-2026-openrubrics-scalable-synthetic-rubric-generation/). Preprint, arXiv:2510.07743.
28. Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. 2025c. [RM-bench: Benchmarking reward models of language models with subtlety and style](https://md.research.elamur.ai/free/en/papers/liu-2024-rm-bench-benchmarking-reward-models/). In International Conference on Learning Representations.
29. Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. 2025d. [Inference-time scaling for generalist reward modeling](https://md.research.elamur.ai/free/en/papers/liu-2025-inference-time-scaling-generalist/). Preprint, arXiv:2504.02495.
30. Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. 2023. [Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning](https://md.research.elamur.ai/free/en/papers/lu-2022-dynamic-prompt-learning-policy-gradient/). Preprint, arXiv:2209.14610.
31. Saumya Malik, Valentina Pyatkin, Sander Land, Jacob Morrison, Noah A. Smith, Hannaneh Hajishirzi, and Nathan Lambert. 2025. RewardBench 2: Advancing reward model evaluation. Preprint, arXiv:2506.01937. — [arXiv:2506.01937](https://arxiv.org/abs/2506.01937) [find](https://scholar.google.com/scholar?q=Saumya+Malik%2C+Valentina+Pyatkin%2C+Sander+Land%2C+Jacob+Morrison%2C+Noah+A%2E+Smith%2C+Hannaneh+Hajishirzi%2C+and+Nathan+Lambert%2E+2025%2E+RewardBench+2%3A+Advancing+reward+model+evaluation%2E+Preprint%2C+arXiv%3A2506%2E01937)
32. Yujun Mao, Yoon Kim, and Yilun Zhou. 2024. CHAMP: A competition-level dataset for fine-grained analyses of llms’ mathematical reasoning capabilities. Preprint, arXiv:2401.06961. [find](https://scholar.google.com/scholar?q=Yujun+Mao%2C+Yoon+Kim%2C+and+Yilun+Zhou%2E+2024%2E+CHAMP%3A+A+competition%2Dlevel+dataset+for+fine%2Dgrained+analyses+of+llms%E2%80%99+mathematical+reasoning+capabilities%2E+Preprint%2C+arXiv%3A2401%2E06961%2E)
33. Xuefei Ning, Zinan Lin, Zixuan Zhou, Zifu Wang, Huazhong Yang, and Yu Wang. 2024. [Skeleton-of-thought: Prompting LLMs for efficient parallel generation](https://md.research.elamur.ai/free/en/papers/ning-2023-skeleton-thought-prompting-llms-efficient/). In International Conference on Learning Representations.
34. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. [Train…](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
35. Jiayi Pan, Xiuyu Li, Long Lian, Charlie Snell, Yifei Zhou, Adam Yala, Trevor Darrell, Kurt Keutzer, and Alane Suhr. 2025. Learning adaptive parallel reasoning with language models. Preprint, arXiv:2504.15466. [find](https://scholar.google.com/scholar?q=Jiayi+Pan%2C+Xiuyu+Li%2C+Long+Lian%2C+Charlie+Snell%2C+Yifei+Zhou%2C+Adam+Yala%2C+Trevor+Darrell%2C+Kurt+Keutzer%2C+and+Alane+Suhr%2E+2025%2E+Learning+adaptive+parallel+reasoning+with+language+models%2E+Preprint%2C+arXiv%3A250)
36. Junsoo Park, Seungyeon Jwa, Ren Meiying, Daeyoung Kim, and Sanghyuk Choi. 2024. [OffsetBias: Leveraging debiased data for tuning evaluators](https://md.research.elamur.ai/free/en/papers/park-2024-offsetbias-leveraging-debiased-data-tuning/). In Findings of the Association for Computational Linguistics: EMNLP, pages 1043–1067.
37. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). In Conference on Neural Information Processing Systems.
38. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. [DeepSeekMath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). Preprint, arXiv:2402.03300.
39. Wei Shen, Guanlin Liu, YuYue, Ruofei Zhu, Qingping Yang, Chao Xin, and Lin Yan. 2025. Exploring data scaling trends and effects in reinforcement learning from human feedback. In Annual Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Wei+Shen%2C+Guanlin+Liu%2C+YuYue%2C+Ruofei+Zhu%2C+Qingping+Yang%2C+Chao+Xin%2C+and+Lin+Yan%2E+2025%2E+Exploring+data+scaling+trends+and+effects+in+reinforcement+learning+from+human+feedback%2E+In+Annual+Conference+on+N)
40. Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. [Reflexion: Language agents with verbal reinforcement learning](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/). Preprint, arXiv:2303.11366.
41. Zayne Rea Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. 2025. [To cot or not to cot? chain-of-thought helps mainly on math and symbolic reasoning](https://md.research.elamur.ai/free/en/papers/sprague-2024-cot-cot-chain-thought-helps/). In International Conference on Learning Representations.
42. DeepSeek-AI Team. 2025a. [DeepSeek-R1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). Preprint, arXiv:2501.12948.
43. Kimi Team. 2025b. [Kimi k1.5: Scaling reinforcement learning with llms](https://md.research.elamur.ai/free/en/papers/kimi-team-2025-kimi-k1-5-scaling-reinforcement/). Preprint, arXiv:2501.12599.
44. Qwen Team. 2025c. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). Preprint, arXiv:2505.09388.
45. Vijay Viswanathan, Yanchao Sun, Shuang Ma, Xiang Kong, Meng Cao, Graham Neubig, and Tongshuang Wu. 2025. [Checklists are better than reward models for aligning language models](https://md.research.elamur.ai/free/en/papers/viswanathan-2025-checklists-better-than-reward-models/). Preprint, arXiv:2507.18624.
46. Junqiao Wang, Zeng Zhang, Yangfan He, Zihao Zhang, Xinyuan Song, Yuyang Song, Tianyu Shi, Yuchen Li, Hengyuan Xu, Kunyu Wu, Xin Yi, Zhongwei Wan, Xinhang Yuan, Zijun Wang, Kuan Lu, Menghao Huo, Tang Jingqun, Guangwu Qian, Keqin Li, and 2 others. 2025a. Enhancing code llms with reinforcement learning… — [arXiv:2412.20367](https://arxiv.org/abs/2412.20367) [find](https://scholar.google.com/scholar?q=Junqiao+Wang%2C+Zeng+Zhang%2C+Yangfan+He%2C+Zihao+Zhang%2C+Xinyuan+Song%2C+Yuyang+Song%2C+Tianyu+Shi%2C+Yuchen+Li%2C+Hengyuan+Xu%2C+Kunyu+Wu%2C+Xin+Yi%2C+Zhongwei+Wan%2C+Xinhang+Yuan%2C+Zijun+Wang%2C+Kuan+Lu%2C+Menghao+Huo%2C+Tang+J)
47. Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. 2024a. [Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations](https://md.research.elamur.ai/free/en/papers/wang-2024-math-shepherd-verify-reinforce-llms/). In Annual Meeting of the Association for Computational Linguistics.
48. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). Preprint, arXiv:2203.11171.
49. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024b. [Mmlu-pro: a more robust and challenging multi-task language understanding benchmar…](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/)
50. Zhilin Wang, Jiaqi Zeng, Olivier Delalleau, Hoo-Chang Shin, Felipe Soares, Alexander Bukharin, Ellie Evans, Yi Dong, and Oleksii Kuchaiev. 2025b. Helpsteer3-preference: Open human-annotated preference data across diverse tasks and languages. Preprint, arXiv:2505.11475. — [arXiv:2505.11475](https://arxiv.org/abs/2505.11475) [find](https://scholar.google.com/scholar?q=Zhilin+Wang%2C+Jiaqi+Zeng%2C+Olivier+Delalleau%2C+Hoo%2DChang+Shin%2C+Felipe+Soares%2C+Alexander+Bukharin%2C+Ellie+Evans%2C+Yi+Dong%2C+and+Oleksii+Kuchaiev%2E+2025b%2E+Helpsteer3%2Dpreference%3A+Open+human%2Dannotated+preference)
51. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Preprint, arXiv:2201.11903.
52. Chenxi Whitehouse, Tianlu Wang, Ping Yu, Xian Li, Jason Weston, Ilia Kulikov, and Swarnadeep Saha. 2025. J1: Incentivizing thinking in llm-as-a-judge via reinforcement learning. Preprint, arXiv:2505.10320. — [arXiv:2505.10320](https://arxiv.org/abs/2505.10320) [find](https://scholar.google.com/scholar?q=Chenxi+Whitehouse%2C+Tianlu+Wang%2C+Ping+Yu%2C+Xian+Li%2C+Jason+Weston%2C+Ilia+Kulikov%2C+and+Swarnadeep+Saha%2E+2025%2E+J1%3A+Incentivizing+thinking+in+llm%2Das%2Da%2Djudge+via+reinforcement+learning%2E+Preprint%2C+arXiv%3A2505%2E1)
53. Austin Xu, Xuan-Phi Nguyen, Yilun Zhou, Chien-Sheng Wu, Caiming Xiong, and Shafiq Joty. 2025. Foundational automatic evaluators: Scaling multi-task generative evaluator training for reasoning-centric domains. Preprint, arXiv:2510.17793. — [arXiv:2510.17793](https://arxiv.org/abs/2510.17793) [find](https://scholar.google.com/scholar?q=Austin+Xu%2C+Xuan%2DPhi+Nguyen%2C+Yilun+Zhou%2C+Chien%2DSheng+Wu%2C+Caiming+Xiong%2C+and+Shafiq+Joty%2E+2025%2E+Foundational+automatic+evaluators%3A+Scaling+multi%2Dtask+generative+evaluator+training+for+reasoning%2Dcentric+)
54. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: deliberate problem solving with large language models. In International Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Thomas+L%2E+Griffiths%2C+Yuan+Cao%2C+and+Karthik+Narasimhan%2E+2023%2E+Tree+of+thoughts%3A+deliberate+problem+solving+with+large+language+models%2E+In+International)
55. Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. 2025. [Demystifying long chain-of-thought reasoning in llms](https://md.research.elamur.ai/free/en/papers/yeo-2025-demystifying-long-chain-thought-reasoning/). Preprint, arXiv:2502.03373.
56. Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2024. [Metamath: Bootstrap your own mathematical questions for large language models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/). In International Conference on Learning Representations.
57. Enyu Zhou, Guodong Zheng, Binghai Wang, Zhiheng Xi, Shihan Dou, Rong Bao, Wei Shen, Limao Xiong, Jessica Fan, Yurong Mou, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. 2025a. [RMB: Comprehensively benchmarking reward models in LLM alignment](https://md.research.elamur.ai/free/en/papers/zhou-2024-rmb-comprehensively-benchmarking-reward-models/). In International Conference on Learning Representations…
58. Yilun Zhou, Austin Xu, Peifeng Wang, Caiming Xiong, and Shafiq Joty. 2025b. Evaluating judges as evaluators: The jetts benchmark of llm-as-judges as test-time scaling evaluators. Preprint, arXiv:2504.15253. — [arXiv:2504.15253](https://arxiv.org/abs/2504.15253) [find](https://scholar.google.com/scholar?q=Yilun+Zhou%2C+Austin+Xu%2C+Peifeng+Wang%2C+Caiming+Xiong%2C+and+Shafiq+Joty%2E+2025b%2E+Evaluating+judges+as+evaluators%3A+The+jetts+benchmark+of+llm%2Das%2Djudges+as+test%2Dtime+scaling+evaluators%2E+Preprint%2C+arXiv%3A2504%2E)
59. Terry Yue Zhuo, Vu Minh Chien, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen GONG, James Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, and 14 others. 2025. [Bigcodebench…](https://md.research.elamur.ai/free/en/papers/zhuo-2024-bigcodebench-benchmarking-code-generation-diverse/)
60. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. Self-rewarding language models. In International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Weizhe+Yuan%2C+Richard+Yuanzhe+Pang%2C+Kyunghyun+Cho%2C+Xian+Li%2C+Sainbayar+Sukhbaatar%2C+Jing+Xu%2C+and+Jason+Weston%2E+2024%2E+Self%2Drewarding+language+models%2E+In+International+Conference+on+Machine+Learning%2E)
61. Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. 2025a. [Generative verifiers: Reward modeling as next-token prediction](https://md.research.elamur.ai/free/en/papers/zhang-2024-generative-verifiers-reward-modeling-as/). In International Conference on Learning Representations.
62. Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Wenyue Hua, Haolun Wu, Zhihan Guo, Yufei Wang, Niklas Muennighoff, Irwin King, Xue Liu, and Chen Ma. 2025b. [A survey on test-time scaling in large language models: What, how, where, and how well?](https://md.research.elamur.ai/free/en/papers/zhang-2025-survey-test-time-scaling-large/) Preprint, arXiv:2503.24235.
63. Qiyuan Zhang, Yufei Wang, Yuxin Jiang, Liangyou Li, Chuhan Wu, Yasheng Wang, Xin Jiang, Lifeng Shang, Ruiming Tang, Fuyuan Lyu, and Chen Ma. 2025c. [Crowd comparative reasoning: Unlocking comprehensive evaluations for LLM-as-a-judge](https://md.research.elamur.ai/free/en/papers/zhang-2025-crowd-comparative-reasoning-unlocking-comprehensive/). In Annual Meeting of the Association for Computational Linguistics.
64. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. [Judging LLM-as-a-judge with MT-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). In Conference on Neural Information Processing Systems Datase…
65. Tong Zheng, Hongming Zhang, Wenhao Yu, Xiaoyang Wang, Runpeng Dai, Rui Liu, Huiwen Bao, Chengsong Huang, Heng Huang, and Dong Yu. 2025. Parallel-R1: Towards parallel thinking via reinforcement learning. Preprint, arXiv:2509.07980. *Table 5: Case Study. Case 1 shows how Breadth-CoT aggregates diverse… — [arXiv:2509.07980](https://arxiv.org/abs/2509.07980) [find](https://scholar.google.com/scholar?q=Tong+Zheng%2C+Hongming+Zhang%2C+Wenhao+Yu%2C+Xiaoyang+Wang%2C+Runpeng+Dai%2C+Rui+Liu%2C+Huiwen+Bao%2C+Chengsong+Huang%2C+Heng+Huang%2C+and+Dong+Yu%2E+2025%2E+Parallel%2DR1%3A+Towards+parallel+thinking+via+reinforcement+learnin)

*Updated 2 November 2026*