--- title: "Agentic Reward Modeling — paper by Peng et al. (2025)" description: "The authors propose agentic reward modeling, in which the final reward is a weighted combination of a… 1 reference, cited by 16 papers here." authors: ["Hao Peng", "Yunjia Qi", "Xiaozhi Wang", "Zijun Yao", "Bin Xu", "Lei Hou", "Juanzi Li"] year: 2025 venue: "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)" source: "https://aclanthology.org/2025.acl-long.775/" doi: "10.18653/v1/2025.acl-long.775" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/peng-2025-agentic-reward-modeling-integrating-human/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/peng-2025-agentic-reward-modeling-integrating-human/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/5/) / Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems # Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems **Authors:** [Hao Peng](https://md.research.elamur.ai/free/en/authors/hao-peng/), [Yunjia Qi](https://md.research.elamur.ai/free/en/authors/yunjia-qi/), [Xiaozhi Wang](https://md.research.elamur.ai/free/en/authors/xiaozhi-wang/), [Zijun Yao](https://md.research.elamur.ai/free/en/authors/zijun-yao/), [Bin Xu](https://md.research.elamur.ai/free/en/authors/bin-xu/), [Lei Hou](https://md.research.elamur.ai/free/en/authors/lei-hou/) and [Juanzi Li](https://md.research.elamur.ai/free/en/authors/juanzi-li/) · **Year:** 2025 · **Venue:** [Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2025](https://md.research.elamur.ai/free/en/venues/acl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2025.acl-long.775/) · [PDF](https://aclanthology.org/2025.acl-long.775.pdf) · [DOI 10.18653/v1/2025.acl-long.775](https://doi.org/10.18653/v1/2025.acl-long.775) · [OpenAlex](https://openalex.org/W4412945186) **Citations:** 2 citations (OpenAlex, 2026-10-14) · [16](https://md.research.elamur.ai/free/en/papers/peng-2025-agentic-reward-modeling-integrating-human/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Reward models for LLMs are usually trained only on human preferences. They can favour superficial traits such as longer or more polished answers and miss factual mistakes or ignored constraints. The authors propose agentic reward modeling, in which the final reward is a weighted combination of a base preference score and the outputs of verification agents selected for each instruction. Their system, RewardAgent, has three modules: - **Router:** decides which checkers an instruction needs. - **Verification agents:** one checks factuality pairwise, focusing only on where two responses disagree and gathering evidence from a search engine or the LLM's own knowledge. The other checks instruction-following by extracting hard constraints, writing Python checker code and running it. - **Judger:** merges these signals with ArmoRM preference scores. RewardAgent was tested on reward benchmarks, best-of-n response selection and the construction of DPO training pairs. It consistently beat plain reward models. The work shows that verifiable correctness checks can make reward signals more reliable and easier to interpret. ## Key points - Two variants were built. RewardAgent MINI runs on GPT-4o mini. RewardAgent LLAMA runs on Llama3-8B Instruct, with Qwen2.5-Coder 7B handling the instruction-following agent. ArmoRM supplies the preference score in both. - The authors created a new benchmark, IF-Bench, with 444 instances. Each pairs an instruction containing several hard constraints with one compliant and one violating response. RM-Bench and JudgeBench were also used. - Best-of-n experiments sampled 32 responses at temperature 1.0 from Llama3-8B Instruct and GPT-4o. They were run on TriviaQA, IFEval and CELLO, where RewardAgent outperformed ArmoRM. - For DPO, zephyr-7b-sft-full was trained on pairs selected from UltraFeedback and from on-policy data (20,000 instructions with 8 samples each). RewardAgent-labelled data beat ArmoRM-labelled data, with the largest gains in the on-policy setting. It also beat the original GPT-4-based UltraFeedback labels. - Code is released at github.com/THU-KEG/Agentic-Reward-Modeling. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 543 works ## Cite Cite as: Hao Peng, Yunjia Qi, Xiaozhi Wang, Zijun Yao, Bin Xu, Lei Hou and Juanzi Li. 2025. Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{peng2025agentic, title = {{Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems}}, author = {Hao Peng, Yunjia Qi, Xiaozhi Wang, Zijun Yao, Bin Xu, Lei Hou and Juanzi Li}, booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, year = {2025}, doi = {10.18653/v1/2025.acl-long.775}, url = {https://doi.org/10.18653/v1/2025.acl-long.775} } ``` - [← previous A Survey of Post-Training Scaling in Large Language Models](https://md.research.elamur.ai/free/en/papers/lai-2025-survey-post-training-scaling-large/) - [next → Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan Arabic Dialect](https://md.research.elamur.ai/free/en/papers/shang-2025-atlas-chat-adapting-large-language/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/5/) ## References 1 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Andrei Alexandru, Antonia Calvi, Henry Broomfield, Jackson Golden, Kyle Dai, Mathias Leys, Maurice Burger, Max Bartolo, Roman Engeler, Sashank Pisu-pati, Toby Drane, and Young Sun Park. 2025. Atla selene mini: A general purpose evaluation model. Preprint, arXiv:2501.17195. — [arXiv:2501.17195](https://arxiv.org/abs/2501.17195) [find](https://scholar.google.com/scholar?q=Andrei+Alexandru%2C+Antonia+Calvi%2C+Henry+Broomfield%2C+Jackson+Golden%2C+Kyle+Dai%2C+Mathias+Leys%2C+Maurice+Burger%2C+Max+Bartolo%2C+Roman+Engeler%2C+Sashank+Pisu%2Dpati%2C+Toby+Drane%2C+and+Young+Sun+Park%2E+2025%2E+Atla+sel) *Updated 30 October 2026*