---
title: "Rubrics as Rewards — paper by Gunjal et al. (2025)"
description: "The authors propose Rubrics as Rewards (RaR). 40 references, cited by 73 papers here. Among them: 0."
authors: ["Anisha Gunjal", "Anthony Wang", "Elaine Lau", "Vaskar Nath", "Yunzhong He", "Bing Liu", "Sean Hendryx"]
year: 2025
venue: "arXiv:2507.17746"
source: "https://arxiv.org/abs/2507.17746"
doi: "10.48550/arxiv.2507.17746"
arxiv: "2507.17746"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/gunjal-2025-rubrics-rewards-reinforcement-learning-verifiable/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gunjal-2025-rubrics-rewards-reinforcement-learning-verifiable/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/) / Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains

# Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains

**Authors:** [Anisha Gunjal](https://md.research.elamur.ai/free/en/authors/anisha-gunjal/), Anthony Wang, Elaine Lau, Vaskar Nath, [Yunzhong He](https://md.research.elamur.ai/free/en/authors/yunzhong-he/), [Bing Liu](https://md.research.elamur.ai/free/en/authors/bing-liu/) and [Sean Hendryx](https://md.research.elamur.ai/free/en/authors/sean-hendryx/) · **Year:** 2025 · **Venue:** arXiv:2507.17746 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2507.17746) · [PDF](https://arxiv.org/pdf/2507.17746) · [DOI 10.48550/arxiv.2507.17746](https://doi.org/10.48550/arxiv.2507.17746) · [arXiv 2507.17746](https://arxiv.org/abs/2507.17746) · [OpenAlex](https://openalex.org/W4414886170)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [73](https://md.research.elamur.ai/free/en/papers/gunjal-2025-rubrics-rewards-reinforcement-learning-verifiable/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Reinforcement learning with verifiable rewards (RLVR) works well for math and code because answers can be checked automatically. Most real-world questions, however, are judged against many soft criteria rather than one correct answer. Preference-based reward models are a common substitute, but they are costly to train and tend to favour superficial traits such as length. The authors propose Rubrics as Rewards (RaR). For every training prompt, a strong LLM (GPT-4o or o3-mini), conditioned on a reference answer, writes a checklist of 7–20 weighted criteria. An LLM judge then scores rollouts against this checklist in one of two ways: item by item with a weighted average (explicit aggregation), or holistically in a single call (implicit aggregation). The score drives GRPO training of Qwen2.5-7B. Compared with judges that give direct Likert ratings, the best RaR setup gains up to 31% relative on HealthBench and 7% on GPQA-Diamond. The two rubric-annotated datasets are publicly released. The work shows that interpretable rubrics can extend RL post-training to expert domains without verifiable answers, and that they make small judges more dependable.

## Key points
- Two released training sets on Hugging Face, each with about 20k prompts drawn from natural_reasoning, SCP-116K, GeneralThought-430K and medical-o1 sources:
  - **RaR-Medicine**, with rubrics written by GPT-4o.
  - **RaR-Science**, with rubrics written by o3-mini and aligned to GPQA-Diamond categories.
- Rubric items carry categorical labels (Essential, Important, Optional, Pitfall). RLVR is formally a special case of the framework: a single criterion with weight 1 that checks for an exact match.
- Training setup:
  - Algorithm and policy: GRPO on Qwen2.5-7B.
  - Sampling: 16 rollouts per prompt.
  - Judge: gpt-4o-mini.
  - Batch size 96 and learning rate 5e-6.
  - Hardware: one node with 8 H100 GPUs.
- Ablations on HealthBench-1k:
  - Rubrics with a broad range of criteria beat essential-only checks.
  - Adding weights or synthetic pitfall items changed results very little.
  - Reference-guided rubrics outperformed reference-free ones; among the reference-free generators, GPT-4o produced the most useful rubrics.
- Rubric-structured rewards align better with human preferences when the judge is small, and they reduce performance variance across judge sizes.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

## Cite

Cite as: Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu and Sean Hendryx. 2025. Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains. arXiv:2507.17746.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{gunjal2025rubrics,
  title = {{Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains}},
  author = {Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu and Sean Hendryx},
  year = {2025},
  doi = {10.48550/arxiv.2507.17746},
  eprint = {2507.17746},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2507.17746}
}
```

- [← previous Findings of the WMT25 General Machine Translation Shared Task: Time to Stop Evaluating on Easy Test Sets](https://md.research.elamur.ai/free/en/papers/kocmi-2025-findings-wmt25-general-machine-translation/)
- [next → When More is Less: Understanding Chain-of-Thought Length in LLMs](https://md.research.elamur.ai/free/en/papers/wu-2025-more-less-understanding-chain-thought/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/)

## References

40 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. D. Anugraha, Z. Tang, L. J. V. Miranda, H. Zhao, M. R. Farhansyah, G. Kuwanto, D. Wijaya, and G. I. Winata. R3: Robust rubric-agnostic reward models. arXiv preprint arXiv:2505.13388, 2025. — [arXiv:2505.13388](https://arxiv.org/abs/2505.13388) [find](https://scholar.google.com/scholar?q=D%2E+Anugraha%2C+Z%2E+Tang%2C+L%2E+J%2E+V%2E+Miranda%2C+H%2E+Zhao%2C+M%2E+R%2E+Farhansyah%2C+G%2E+Kuwanto%2C+D%2E+Wijaya%2C+and+G%2E+I%2E+Winata%2E+R3%3A+Robust+rubric%2Dagnostic+reward+models%2E+arXiv+preprint+arXiv%3A2505%2E13388%2C+2025%2E+%E2%80%94+%5BarXiv%3A25)
2. R. K. Arora, J. Wei, R. S. Hicks, P. Bowman, J. Quiñonero-Candela, F. Tsimpourlas, M. Sharman, M. Shah, A. Vallone, A. Beutel, et al. [Healthbench: Evaluating large language models towards improved human health](https://md.research.elamur.ai/free/en/papers/arora-2025-healthbench-evaluating-large-language-models/). arXiv preprint arXiv:2505.08775, 2025.
3. J. Chen, Z. Cai, K. Ji, X. Wang, W. Liu, R. Wang, J. Hou, and B. Wang. [Huatuogpt-o1, towards medical complex reasoning with llms](https://md.research.elamur.ai/free/en/papers/chen-2024-huatuogpt-o1-towards-medical-complex/). arXiv preprint arXiv:2412.18925, 2024.
4. L. Chen, C. Zhu, D. Soselia, J. Chen, T. Zhou, T. Goldstein, H. Huang, M. Shoeybi, and B. Catanzaro. Odin: Disentangled reward mitigates hacking in rlhf. arXiv preprint arXiv:2402.07319, 2024. [find](https://scholar.google.com/scholar?q=L%2E+Chen%2C+C%2E+Zhu%2C+D%2E+Soselia%2C+J%2E+Chen%2C+T%2E+Zhou%2C+T%2E+Goldstein%2C+H%2E+Huang%2C+M%2E+Shoeybi%2C+and+B%2E+Catanzaro%2E+Odin%3A+Disentangled+reward+mitigates+hacking+in+rlhf%2E+arXiv+preprint+arXiv%3A2402%2E07319%2C+2024%2E)
5. X. Chen, G. Li, Z. Wang, B. Jin, C. Qian, Y. Wang, H. Wang, Y. Zhang, D. Zhang, T. Zhang, et al. [Rm-r1: Reward modeling as reasoning](https://md.research.elamur.ai/free/en/papers/chen-2025-rm-r1-reward-modeling-reasoning/). arXiv preprint arXiv:2505.02387, 2025.
6. G. Cui, L. Yuan, Z. Wang, H. Wang, W. Li, B. He, Y. Fan, T. Yu, Q. Xu, W. Chen, et al. [Process reinforcement through implicit rewards](https://md.research.elamur.ai/free/en/papers/cui-2025-process-reinforcement-through-implicit-rewards/). arXiv preprint arXiv:2502.01456, 2025.
7. J. Dineen, A. RRV, Q. Liu, Z. Xu, X. Ye, M. Shen, Z. Li, S. Lu, C. Baral, M. Chen, et al. Qa-lign: Aligning llms through constitutionally decomposed qa. arXiv preprint arXiv:2506.08123, 2025. [find](https://scholar.google.com/scholar?q=J%2E+Dineen%2C+A%2E+RRV%2C+Q%2E+Liu%2C+Z%2E+Xu%2C+X%2E+Ye%2C+M%2E+Shen%2C+Z%2E+Li%2C+S%2E+Lu%2C+C%2E+Baral%2C+M%2E+Chen%2C+et+al%2E+Qa%2Dlign%3A+Aligning+llms+through+constitutionally+decomposed+qa%2E+arXiv+preprint+arXiv%3A2506%2E08123%2C+2025%2E)
8. V. Gallego. Configurable preference tuning with rubric-guided synthetic data. arXiv preprint arXiv:2506.11702, 2025. — [arXiv:2506.11702](https://arxiv.org/abs/2506.11702) [find](https://scholar.google.com/scholar?q=V%2E+Gallego%2E+Configurable+preference+tuning+with+rubric%2Dguided+synthetic+data%2E+arXiv+preprint+arXiv%3A2506%2E11702%2C+2025%2E+%E2%80%94+%5BarXiv%3A2506%2E11702%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2506%2E11702%29)
9. General Reasoning. General reasoning, 2025. URL [https://gr.inc/](https://gr.inc/). [↗](https://gr.inc/)
10. A. Gudibande, E. Wallace, C. Snell, X. Geng, H. Liu, P. Abbeel, S. Levine, and D. Song. [The false promise of imitating proprietary llms](https://md.research.elamur.ai/free/en/papers/gudibande-2023-false-promise-imitating-proprietary-llms/). arXiv preprint arXiv:2305.15717, 2023.
11. D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948, 2025.
12. J. Guo, Z. Chi, L. Dong, Q. Dong, X. Wu, S. Huang, and F. Wei. Reward reasoning model. arXiv preprint arXiv:2505.14674, 2025. — [arXiv:2505.14674](https://arxiv.org/abs/2505.14674) [find](https://scholar.google.com/scholar?q=J%2E+Guo%2C+Z%2E+Chi%2C+L%2E+Dong%2C+Q%2E+Dong%2C+X%2E+Wu%2C+S%2E+Huang%2C+and+F%2E+Wei%2E+Reward+reasoning+model%2E+arXiv+preprint+arXiv%3A2505%2E14674%2C+2025%2E+%E2%80%94+%5BarXiv%3A2505%2E14674%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2505%2E14674%29)
13. H. Hashemi, J. Eisner, C. Rosset, B. Van Durme, and C. Kedzie. [Llm-rubric: A multidimensional, calibrated approach to automated evaluation of natural language texts](https://md.research.elamur.ai/free/en/papers/hashemi-2024-llm-rubric-multidimensional-calibrated-approach/). arXiv preprint arXiv:2501.00274, 2024.
14. A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. [Gpt-4o system card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/). arXiv preprint arXiv:2410.21276, 2024.
15. A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carney, et al. [Openai o1 system card](https://md.research.elamur.ai/free/en/papers/openai-2024-openai-o1-system-card/). arXiv preprint arXiv:2412.16720, 2024.
16. M. Khalifa, R. Agarwal, L. Logeswaran, J. Kim, H. Peng, M. Lee, H. Lee, and L. Wang. Process reward models that think. arXiv preprint arXiv:2504.16828, 2025. — [arXiv:2504.16828](https://arxiv.org/abs/2504.16828) [find](https://scholar.google.com/scholar?q=M%2E+Khalifa%2C+R%2E+Agarwal%2C+L%2E+Logeswaran%2C+J%2E+Kim%2C+H%2E+Peng%2C+M%2E+Lee%2C+H%2E+Lee%2C+and+L%2E+Wang%2E+Process+reward+models+that+think%2E+arXiv+preprint+arXiv%3A2504%2E16828%2C+2025%2E+%E2%80%94+%5BarXiv%3A2504%2E16828%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs)
17. S. Kim, J. Suk, S. Longpre, B. Y. Lin, J. Shin, S. Welleck, G. Neubig, M. Lee, K. Lee, and M. Seo. [Prometheus 2: An open source language model specialized in evaluating other language models](https://md.research.elamur.ai/free/en/papers/kim-2024-prometheus-2-open-source-language/). arXiv preprint arXiv:2405.01535, 2024.
18. Z. M. Kim, C. Park, V. Raheja, S. Kim, and D. Kang. Toward evaluative thinking: Meta policy optimization with evolving reward models. arXiv preprint arXiv:2504.20157, 2025. — [arXiv:2504.20157](https://arxiv.org/abs/2504.20157) [find](https://scholar.google.com/scholar?q=Z%2E+M%2E+Kim%2C+C%2E+Park%2C+V%2E+Raheja%2C+S%2E+Kim%2C+and+D%2E+Kang%2E+Toward+evaluative+thinking%3A+Meta+policy+optimization+with+evolving+reward+models%2E+arXiv+preprint+arXiv%3A2504%2E20157%2C+2025%2E+%E2%80%94+%5BarXiv%3A2504%2E20157%5D%28https%3A)
19. N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, S. Lyu, et al. T \\ " ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.[15124, 2024](https://md.research.elamur.ai/free/en/papers/lambert-2024-tulu-3-pushing-frontiers-open/).
20. S. Li, S. Dong, K. Luan, X. Di, and C. Ding. Enhancing reasoning through process supervision with monte carlo tree search. arXiv preprint arXiv:2501.01478, 2025. — [arXiv:2501.01478](https://arxiv.org/abs/2501.01478) [find](https://scholar.google.com/scholar?q=S%2E+Li%2C+S%2E+Dong%2C+K%2E+Luan%2C+X%2E+Di%2C+and+C%2E+Ding%2E+Enhancing+reasoning+through+process+supervision+with+monte+carlo+tree+search%2E+arXiv+preprint+arXiv%3A2501%2E01478%2C+2025%2E+%E2%80%94+%5BarXiv%3A2501%2E01478%5D%28https%3A%2F%2Farxiv%2Eorg)
21. H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe. Let’s verify step by step. In The Twelfth International Conference on Learning Representations, 2023. — [Let's Verify Step by Step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/)
22. D. Lu, X. Tan, R. Xu, T. Yao, C. Qu, W. Chu, Y. Xu, and Y. Qi. Scp-116k: A high-quality problem-solution dataset and a generalized pipeline for automated extraction in the higher education science domain. arXiv preprint arXiv:2501.15587, 2025. — [arXiv:2501.15587](https://arxiv.org/abs/2501.15587) [find](https://scholar.google.com/scholar?q=D%2E+Lu%2C+X%2E+Tan%2C+R%2E+Xu%2C+T%2E+Yao%2C+C%2E+Qu%2C+W%2E+Chu%2C+Y%2E+Xu%2C+and+Y%2E+Qi%2E+Scp%2D116k%3A+A+high%2Dquality+problem%2Dsolution+dataset+and+a+generalized+pipeline+for+automated+extraction+in+the+higher+education+science+dom)
23. X. Ma, Q. Liu, D. Jiang, G. Zhang, Z. Ma, and W. Chen. [General-reasoner: Advancing llm reasoning across all domains](https://md.research.elamur.ai/free/en/papers/ma-2025-general-reasoner-advancing-llm-reasoning/). arXiv preprint arXiv:2505.14652, 2025.
24. OpenAI o3-mini. Openai o3-min, 2025. URL [https://openai.com/index/openai-o3-mini/](https://openai.com/index/openai-o3-mini/). [↗](https://openai.com/index/openai-o3-mini/)
25. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in neural information processing systems, 35:27730–27744, 2022.
26. A. Pathak, R. Gandhi, V. Uttam, Y. Nakka, A. R. Jindal, P. Ghosh, A. Ramamoorthy, S. Verma, A. Mittal, A. Ased, et al. Rubric is all you need: Enhancing llm-based code evaluation with question-specific rubrics. arXiv preprint arXiv:2503.23989, 2025. — [arXiv:2503.23989](https://arxiv.org/abs/2503.23989) [find](https://scholar.google.com/scholar?q=A%2E+Pathak%2C+R%2E+Gandhi%2C+V%2E+Uttam%2C+Y%2E+Nakka%2C+A%2E+R%2E+Jindal%2C+P%2E+Ghosh%2C+A%2E+Ramamoorthy%2C+S%2E+Verma%2C+A%2E+Mittal%2C+A%2E+Ased%2C+et+al%2E+Rubric+is+all+you+need%3A+Enhancing+llm%2Dbased+code+evaluation+with+question%2Dspecifi)
27. J. Ruan, I. Nair, S. Cao, A. Liu, S. Munir, M. Pollens-Dempsey, T. Chiang, L. Kates, N. David, S. Chen, et al. Expertlongbench: Benchmarking language models on expert-level long-form generation tasks with structured checklists. arXiv preprint arXiv:2506.01241, 2025. — [arXiv:2506.01241](https://arxiv.org/abs/2506.01241) [find](https://scholar.google.com/scholar?q=J%2E+Ruan%2C+I%2E+Nair%2C+S%2E+Cao%2C+A%2E+Liu%2C+S%2E+Munir%2C+M%2E+Pollens%2DDempsey%2C+T%2E+Chiang%2C+L%2E+Kates%2C+N%2E+David%2C+S%2E+Chen%2C+et+al%2E+Expertlongbench%3A+Benchmarking+language+models+on+expert%2Dlevel+long%2Dform+generation+tasks+)
28. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). arXiv preprint arXiv:2402.03300, 2024.
29. P. Singhal, T. Goyal, J. Xu, and G. Durrett. [A long way to go: Investigating length correlations in rlhf](https://md.research.elamur.ai/free/en/papers/singhal-2023-long-way-go-investigating-length/). arXiv preprint arXiv:2310.03716, 2023.
30. V. Sirdeshmukh, K. Deshpande, J. Mols, L. Jin, E.-Y. Cardona, D. Lee, J. Kritz, W. Primack, S. Yue, and C. Xing. Multichallenge: A realistic multi-turn conversation evaluation benchmark challenging to frontier llms. arXiv preprint arXiv:2501.17399, 2025. [find](https://scholar.google.com/scholar?q=V%2E+Sirdeshmukh%2C+K%2E+Deshpande%2C+J%2E+Mols%2C+L%2E+Jin%2C+E%2E%2DY%2E+Cardona%2C+D%2E+Lee%2C+J%2E+Kritz%2C+W%2E+Primack%2C+S%2E+Yue%2C+and+C%2E+Xing%2E+Multichallenge%3A+A+realistic+multi%2Dturn+conversation+evaluation+benchmark+challenging+to)
31. Y. Su, D. Yu, L. Song, J. Li, H. Mi, Z. Tu, M. Zhang, and D. Yu. Crossing the reward bridge: Expanding rl with verifiable rewards across diverse domains. arXiv preprint arXiv:2503.23829, 2025. — [arXiv:2503.23829](https://arxiv.org/abs/2503.23829) [find](https://scholar.google.com/scholar?q=Y%2E+Su%2C+D%2E+Yu%2C+L%2E+Song%2C+J%2E+Li%2C+H%2E+Mi%2C+Z%2E+Tu%2C+M%2E+Zhang%2C+and+D%2E+Yu%2E+Crossing+the+reward+bridge%3A+Expanding+rl+with+verifiable+rewards+across+diverse+domains%2E+arXiv+preprint+arXiv%3A2503%2E23829%2C+2025%2E+%E2%80%94+%5BarXi)
32. Y. Su, D. Yu, L. Song, J. Li, H. Mi, Z. Tu, M. Zhang, and D. Yu. Expanding rl with verifiable rewards across diverse domains. arXiv preprint arXiv:2503.23829, 2025. — [arXiv:2503.23829](https://arxiv.org/abs/2503.23829) [find](https://scholar.google.com/scholar?q=Y%2E+Su%2C+D%2E+Yu%2C+L%2E+Song%2C+J%2E+Li%2C+H%2E+Mi%2C+Z%2E+Tu%2C+M%2E+Zhang%2C+and+D%2E+Yu%2E+Expanding+rl+with+verifiable+rewards+across+diverse+domains%2E+arXiv+preprint+arXiv%3A2503%2E23829%2C+2025%2E+%E2%80%94+%5BarXiv%3A2503%2E23829%5D%28https%3A%2F%2Farxiv%2E)
33. V. Viswanathan, Y. Sun, S. Ma, X. Kong, M. Cao, G. Neubig, and T. Wu. [Checklists are better than reward models for aligning language models](https://md.research.elamur.ai/free/en/papers/viswanathan-2025-checklists-better-than-reward-models/). arXiv preprint arXiv:2507.18624, 2025.
34. H. Wang, Y. Lin, W. Xiong, R. Yang, S. Diao, S. Qiu, H. Zhao, and T. Zhang. [Arithmetic control of llms for diverse user preferences: Directional preference alignment with multi-objective rewards](https://md.research.elamur.ai/free/en/papers/wang-2024-arithmetic-control-llms-diverse-user/). arXiv preprint arXiv:2402.18571, 2024.
35. C. Whitehouse, T. Wang, P. Yu, X. Li, J. Weston, I. Kulikov, and S. Saha. J1: Incentivizing thinking in llm-as-a-judge via reinforcement learning. arXiv preprint arXiv:2505.10320, 2025. — [arXiv:2505.10320](https://arxiv.org/abs/2505.10320) [find](https://scholar.google.com/scholar?q=C%2E+Whitehouse%2C+T%2E+Wang%2C+P%2E+Yu%2C+X%2E+Li%2C+J%2E+Weston%2C+I%2E+Kulikov%2C+and+S%2E+Saha%2E+J1%3A+Incentivizing+thinking+in+llm%2Das%2Da%2Djudge+via+reinforcement+learning%2E+arXiv+preprint+arXiv%3A2505%2E10320%2C+2025%2E+%E2%80%94+%5BarXiv%3A2505%2E)
36. Z. Ye, F. Greenlee-Scott, M. Bartolo, P. Blunsom, J. A. Campos, and M. Gallé. Improving reward models with synthetic critiques. arXiv preprint arXiv:2405.20850, 2024. [find](https://scholar.google.com/scholar?q=Z%2E+Ye%2C+F%2E+Greenlee%2DScott%2C+M%2E+Bartolo%2C+P%2E+Blunsom%2C+J%2E+A%2E+Campos%2C+and+M%2E+Gall%C3%A9%2E+Improving+reward+models+with+synthetic+critiques%2E+arXiv+preprint+arXiv%3A2405%2E20850%2C+2024%2E)
37. W. Yuan, J. Yu, S. Jiang, K. Padthe, Y. Li, I. Kulikov, K. Cho, D. Wang, Y. Tian, J. E. Weston, et al. Naturalreasoning: Reasoning in the wild with 2.8 m challenging questions. arXiv preprint arXiv:2502.13124, 2025. [find](https://scholar.google.com/scholar?q=W%2E+Yuan%2C+J%2E+Yu%2C+S%2E+Jiang%2C+K%2E+Padthe%2C+Y%2E+Li%2C+I%2E+Kulikov%2C+K%2E+Cho%2C+D%2E+Wang%2C+Y%2E+Tian%2C+J%2E+E%2E+Weston%2C+et+al%2E+Naturalreasoning%3A+Reasoning+in+the+wild+with+2%2E8+m+challenging+questions%2E+arXiv+preprint+arXiv%3A25)
38. S. Zhang, Q. Liu, G. Qin, T. Naumann, and H. Poon. Med-rlvr: Emerging medical reasoning from a 3b base model via reinforcement learning. arXiv preprint arXiv:2502.19655, 2025. [find](https://scholar.google.com/scholar?q=S%2E+Zhang%2C+Q%2E+Liu%2C+G%2E+Qin%2C+T%2E+Naumann%2C+and+H%2E+Poon%2E+Med%2Drlvr%3A+Emerging+medical+reasoning+from+a+3b+base+model+via+reinforcement+learning%2E+arXiv+preprint+arXiv%3A2502%2E19655%2C+2025%2E)
39. X. Zhao, Z. Kang, A. Feng, S. Levine, and D. Song. Learning to reason without external rewards. arXiv preprint arXiv:2505.19590, 2025. [find](https://scholar.google.com/scholar?q=X%2E+Zhao%2C+Z%2E+Kang%2C+A%2E+Feng%2C+S%2E+Levine%2C+and+D%2E+Song%2E+Learning+to+reason+without+external+rewards%2E+arXiv+preprint+arXiv%3A2505%2E19590%2C+2025%2E)
40. L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Advances in neural information processing systems, 36:46595–46623, 2023.

*Updated 2 November 2026*