---
title: "ReviewRL: Towards Automated Scientific Review with RL (2025)"
description: "Growing submission volumes strain peer review, and current LLM reviewers often produce shallow… 28 references, cited by 7 papers here."
authors: ["Sihang Zeng", "Kai Tian", "Kaiyan Zhang", "Yuru Wang", "Junqi Gao", "Runze Liu", "Sa Yang", "Jingxuan Li", "Xinwei Long", "Jiaheng Ma", "Biqing Qi", "Bowen Zhou"]
year: 2025
venue: "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2025.emnlp-main.857/"
doi: "10.18653/v1/2025.emnlp-main.857"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zeng-2025-reviewrl-automated-scientific-review-rl/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zeng-2025-reviewrl-automated-scientific-review-rl/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/9/) / ReviewRL: Towards Automated Scientific Review with RL

# ReviewRL: Towards Automated Scientific Review with RL

**Authors:** [Sihang Zeng](https://md.research.elamur.ai/free/en/authors/sihang-zeng/), [Kai Tian](https://md.research.elamur.ai/free/en/authors/kai-tian/), [Kaiyan Zhang](https://md.research.elamur.ai/free/en/authors/kaiyan-zhang/), Yuru Wang, [Junqi Gao](https://md.research.elamur.ai/free/en/authors/junqi-gao/), [Runze Liu](https://md.research.elamur.ai/free/en/authors/runze-liu/), Sa Yang, Jingxuan Li, Xinwei Long, Jiaheng Ma, [Biqing Qi](https://md.research.elamur.ai/free/en/authors/biqing-qi/) and [Bowen Zhou](https://md.research.elamur.ai/free/en/authors/bowen-zhou/) · **Year:** 2025 · **Venue:** [Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2025](https://md.research.elamur.ai/free/en/venues/emnlp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.emnlp-main.857/) · [PDF](https://aclanthology.org/2025.emnlp-main.857.pdf) · [DOI 10.18653/v1/2025.emnlp-main.857](https://doi.org/10.18653/v1/2025.emnlp-main.857) · [OpenAlex](https://openalex.org/W4416036199)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [7](https://md.research.elamur.ai/free/en/papers/zeng-2025-reviewrl-automated-scientific-review-rl/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Growing submission volumes strain peer review, and current LLM reviewers often produce shallow, generic critiques. They also misstate facts and give inflated scores that diverge from human ratings. ReviewRL addresses this with a three-stage training and inference framework. First, an agentic retrieval step uses Qwen3-8B to write three natural-language questions about a paper's novelty, methods and links to earlier work. These questions are answered through the open-source ArXiv-MCP server, coordinated by Qwen-Agent. The cleaned results are appended to the paper as context. Second, supervised fine-tuning warms up the reviewer, giving it basic reviewing skill and better initial score calibration. Third, reinforcement learning optimizes a composite reward that mixes rule-based rating signals with judgments from a generative reward model (GenRM). The authors argue that purely rule-based rewards fall short for open-ended critique. On ICLR 2025 submissions, ReviewRL beats prior systems such as DeepReviewer and CycleReviewer on both rule-based and model-based metrics. The work shows that RL can be applied to non-verifiable generation tasks like scientific reviewing.

## Key points
- The pipeline has three parts: ArXiv-MCP retrieval-augmented context, an SFT warm-up, and RL with a composite reward (rule-based score plus GenRM judge).
- Retrieval uses Qwen3-8B to generate three queries per paper. Qwen-Agent routes these queries to ArXiv-MCP, which returns metadata, abstracts and ranked excerpts.
- Review quality is assessed on ICLR 2025 papers using ReviewEval criteria, including:
  - analytical depth
  - correctness of claims
  - absence of hallucinations
  - actionable insights
  - topic coverage
- Retrieved context beat vanilla context in pairwise win rates: 95.0% for factual accuracy, 83.3% for evidence quality and 67.4% for clarity/coherence.
- Ablations (removing retrieval, GenRM or RL) lower performance. Dropping retrieval hurts the factuality metrics most. A stated limitation is the dependence on arXiv coverage.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 353 works · 200,043 citations
- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 333 works · 87,019 citations
- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 244 works · 39,359 citations
- [Shanghai Artificial Intelligence Laboratory](https://md.research.elamur.ai/free/en/labs/shanghai-artificial-intelligence-laboratory/) — Research institute, Shanghai, China · 117 works · 11,281 citations
- [Beijing Institute of Technology](https://md.research.elamur.ai/free/en/universities/beijing-institute-of-technology/) — University, Beijing, China · 34 works · 3,652 citations
- [Harbin Engineering University](https://md.research.elamur.ai/free/en/universities/harbin-engineering-university/) — University, Harbin, China · 2 works · 9 citations

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 General Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/general-social-sciences/) · 227 works · [🎭 History and Philosophy of Science](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/history-and-philosophy-of-science/) · 53 works

## Cite

Cite as: Sihang Zeng, Kai Tian, Kaiyan Zhang, Yuru Wang, Junqi Gao, Runze Liu, Sa Yang, Jingxuan Li, Xinwei Long, Jiaheng Ma, Biqing Qi and Bowen Zhou. 2025. ReviewRL: Towards Automated Scientific Review with RL. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zeng2025reviewrl,
  title = {{ReviewRL: Towards Automated Scientific Review with RL}},
  author = {Sihang Zeng, Kai Tian, Kaiyan Zhang, Yuru Wang, Junqi Gao, Runze Liu, Sa Yang, Jingxuan Li, Xinwei Long, Jiaheng Ma, Biqing Qi and Bowen Zhou},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.857},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.857}
}
```

- [← previous Refining Input Guardrails: Enhancing LLM-as-a-Judge Efficiency Through Chain-of-Thought Fine-Tuning and Alignment](https://md.research.elamur.ai/free/en/papers/rad-2025-refining-input-guardrails-enhancing-llm/)
- [next → RGR-KBQA: Generating Logical Forms for Question Answering Using Knowledge-Graph-Enhanced Large Language Model](https://md.research.elamur.ai/free/en/papers/feng-2025-rgr-kbqa-generating-logical-forms/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/9/)

## References

28 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, and 1 others. 2020. [Retrieval-augmented generation for knowledge-intensive nlp tasks](https://md.research.elamur.ai/free/en/papers/lewis-2020-retrieval-augmented-generation-knowledge-intensive/). Advances in neural information processing systems, 33:9459–…
2. Nuo Chen, Zhiyuan Hu, Qingyun Zou, Jiaying Wu, Qian Wang, Bryan Hooi, and Bingsheng He. 2025. Judgelrm: Large reasoning models as a judge. arXiv preprint arXiv:2504.00050. — [arXiv:2504.00050](https://arxiv.org/abs/2504.00050) [find](https://scholar.google.com/scholar?q=Nuo+Chen%2C+Zhiyuan+Hu%2C+Qingyun+Zou%2C+Jiaying+Wu%2C+Qian+Wang%2C+Bryan+Hooi%2C+and+Bingsheng+He%2E+2025%2E+Judgelrm%3A+Large+reasoning+models+as+a+judge%2E+arXiv+preprint+arXiv%3A2504%2E00050%2E+%E2%80%94+%5BarXiv%3A2504%2E00050%5D%28https%3A%2F)
3. Nils Dycke, Ilia Kuznetsov, and Iryna Gurevych. 2022. [Nlpeer: A unified resource for the computational study of peer review](https://md.research.elamur.ai/free/en/papers/dycke-2023-nlpeer-unified-resource-computational-study/). arXiv preprint arXiv:2211.06651.
4. Zhaolin Gao, Kianté Brantley, and Thorsten Joachims. 2024. Reviewer2: Optimizing review genera-Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yutao Zhu, Yongkang Wu, Ji-Rong Wen, and Zhicheng Dou. 2025. Webthinker: Empowering large reasoning models with deep research capability. arXiv preprint… [find](https://scholar.google.com/scholar?q=Zhaolin+Gao%2C+Kiant%C3%A9+Brantley%2C+and+Thorsten+Joachims%2E+2024%2E+Reviewer2%3A+Optimizing+review+genera%2DXiaoxi+Li%2C+Jiajie+Jin%2C+Guanting+Dong%2C+Hongjin+Qian%2C+Yutao+Zhu%2C+Yongkang+Wu%2C+Ji%2DRong+Wen%2C+and+Zhicheng+Dou)
5. Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. 2025. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.01785. — [arXiv:2503.01785](https://arxiv.org/abs/2503.01785) [find](https://scholar.google.com/scholar?q=Ziyu+Liu%2C+Zeyi+Sun%2C+Yuhang+Zang%2C+Xiaoyi+Dong%2C+Yuhang+Cao%2C+Haodong+Duan%2C+Dahua+Lin%2C+and+Jiaqi+Wang%2E+2025%2E+Visual%2Drft%3A+Visual+reinforcement+fine%2Dtuning%2E+arXiv+preprint+arXiv%3A2503%2E01785%2E+%E2%80%94+%5BarXiv%3A2503%2E01)
6. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. 2024. [The ai scientist: Towards fully automated open-ended scientific discovery](https://md.research.elamur.ai/free/en/papers/lu-2024-ai-scientist-fully-automated/). arXiv preprint arXiv:2408.06292.
7. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in neural information processing systems, 35:27730–27744.
8. Biqing Qi, Kaiyan Zhang, Kai Tian, Haoxiang Li, Zhang-Ren Chen, Sihang Zeng, Ermo Hua, Hu Jinfang, and Bowen Zhou. 2024. [Large language models as biomedical hypothesis generators: A comprehensive evaluation](https://md.research.elamur.ai/free/en/papers/qi-2024-large-language-models-biomedical-hypothesis/). Preprint, arXiv:2407.08940.
9. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). Advances in Neural Information Processing Systems, 36:53728– 53741.
10. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+2017%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2E)
11. ByteDance Seed, Yufeng Yuan, Yu Yue, Mingxuan Wang, Xiaochen Zuo, Jiaze Chen, Lin Yan, Wenyuan Xu, Chi Zhang, Xin Liu, and 1 others. 2025. [Seed-thinking-v1](https://md.research.elamur.ai/free/en/papers/bytedance-seed-2025-seed1-5-thinking-advancing/). 5: Advancing superb reasoning models with reinforcement learning. arXiv preprint arXiv:2504.13914.
12. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, and 1 others. 2024. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). arXiv preprint arXiv:2402.03300.
13. Hyungyu Shin, Jingyu Tang, Yoonjoo Lee, Nayoung Kim, Hyunseung Lim, Ji Yong Cho, Hwajung Hong, Moontae Lee, and Juho Kim. 2025. Automatically evaluating the paper reviewing capability of large language models. arXiv preprint arXiv:2502.17086. — [arXiv:2502.17086](https://arxiv.org/abs/2502.17086) [find](https://scholar.google.com/scholar?q=Hyungyu+Shin%2C+Jingyu+Tang%2C+Yoonjoo+Lee%2C+Nayoung+Kim%2C+Hyunseung+Lim%2C+Ji+Yong+Cho%2C+Hwajung+Hong%2C+Moontae+Lee%2C+and+Juho+Kim%2E+2025%2E+Automatically+evaluating+the+paper+reviewing+capability+of+large+languag)
14. Richard S Sutton, Andrew G Barto, and 1 others. 1998. Reinforcement learning: An introduction, volume 1. MIT press Cambridge. [find](https://scholar.google.com/scholar?q=Richard+S+Sutton%2C+Andrew+G+Barto%2C+and+1+others%2E+1998%2E+Reinforcement+learning%3A+An+introduction%2C+volume+1%2E+MIT+press+Cambridge%2E)
15. Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, and 1 others. 2025. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599. — [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://md.research.elamur.ai/free/en/papers/kimi-team-2025-kimi-k1-5-scaling-reinforcement/)
16. Qwen Team. 2024. Qwen2.5: A party of foundation models. [find](https://scholar.google.com/scholar?q=Qwen+Team%2E+2024%2E+Qwen2%2E5%3A+A+party+of+foundation+models%2E)
17. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in neural information processing systems, 35:24824– 24837.
18. Yixuan Weng, Minjun Zhu, Guangsheng Bao, Hongbo Zhang, Jindong Wang, Yue Zhang, and Linyi Yang. 2024. Cycleresearcher: Improving automated research via automated review. arXiv preprint arXiv:2411.00816. [find](https://scholar.google.com/scholar?q=Yixuan+Weng%2C+Minjun+Zhu%2C+Guangsheng+Bao%2C+Hongbo+Zhang%2C+Jindong+Wang%2C+Yue+Zhang%2C+and+Linyi+Yang%2E+2024%2E+Cycleresearcher%3A+Improving+automated+research+via+automated+review%2E+arXiv+preprint+arXiv%3A2411%2E0081)
19. Chenxi Whitehouse, Tianlu Wang, Ping Yu, Xian Li, Jason Weston, Ilia Kulikov, and Swarnadeep Saha. 2025. J1: Incentivizing thinking in llm-asa-judge via reinforcement learning. arXiv preprint arXiv:2505.10320. — [arXiv:2505.10320](https://arxiv.org/abs/2505.10320) [find](https://scholar.google.com/scholar?q=Chenxi+Whitehouse%2C+Tianlu+Wang%2C+Ping+Yu%2C+Xian+Li%2C+Jason+Weston%2C+Ilia+Kulikov%2C+and+Swarnadeep+Saha%2E+2025%2E+J1%3A+Incentivizing+thinking+in+llm%2Dasa%2Djudge+via+reinforcement+learning%2E+arXiv+preprint+arXiv%3A25)
20. Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, and 1 others. 2025. Dancegrpo: Unleashing grpo on visual generation. arXiv preprint arXiv:2505.07818. — [arXiv:2505.07818](https://arxiv.org/abs/2505.07818) [find](https://scholar.google.com/scholar?q=Zeyue+Xue%2C+Jie+Wu%2C+Yu+Gao%2C+Fangyuan+Kong%2C+Lingting+Zhu%2C+Mengzhao+Chen%2C+Zhiheng+Liu%2C+Wei+Liu%2C+Qiushan+Guo%2C+Weilin+Huang%2C+and+1+others%2E+2025%2E+Dancegrpo%3A+Unleashing+grpo+on+visual+generation%2E+arXiv+prepr)
21. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.
22. Sungduk Yu, Man Luo, Avinash Madusu, Vasudev Lal, and Phillip Howard. 2025. Is your paper being reviewed by an llm? a new benchmark dataset and approach for detecting ai text in peer review. arXiv preprint arXiv:2502.19614. — [arXiv:2502.19614](https://arxiv.org/abs/2502.19614) [find](https://scholar.google.com/scholar?q=Sungduk+Yu%2C+Man+Luo%2C+Avinash+Madusu%2C+Vasudev+Lal%2C+and+Phillip+Howard%2E+2025%2E+Is+your+paper+being+reviewed+by+an+llm%3F+a+new+benchmark+dataset+and+approach+for+detecting+ai+text+in+peer+review%2E+arXiv+pre)
23. Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. 2024. [Generative verifiers: Reward modeling as next-token prediction](https://md.research.elamur.ai/free/en/papers/zhang-2024-generative-verifiers-reward-modeling-as/). arXiv preprint arXiv:2408.15240.
24. Tianmai M. Zhang and Neil F. Abernethy. 2025. [Reviewing scientific papers for critical problems with reasoning llms: Baseline approaches and automatic evaluation](https://md.research.elamur.ai/free/en/papers/zhang-2025-reviewing-scientific-papers-critical-problems/). Preprint, arXiv:2505.23824.
25. Yuxiang Zhang, Yuqi Yang, Jiangming Shu, Xinyan Wen, and Jitao Sang. 2025. Agent models: Internalizing chain-of-action generation into reasoning models. arXiv preprint arXiv:2503.06580. — [arXiv:2503.06580](https://arxiv.org/abs/2503.06580) [find](https://scholar.google.com/scholar?q=Yuxiang+Zhang%2C+Yuqi+Yang%2C+Jiangming+Shu%2C+Xinyan+Wen%2C+and+Jitao+Sang%2E+2025%2E+Agent+models%3A+Internalizing+chain%2Dof%2Daction+generation+into+reasoning+models%2E+arXiv+preprint+arXiv%3A2503%2E06580%2E+%E2%80%94+%5BarXiv%3A2503%2E)
26. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Preprint, arXiv:2306.05685.
27. Ruiyang Zhou, Lu Chen, and Kai Yu. 2024. [Is llm a reliable reviewer? a comprehensive evaluation of llm on automatic paper reviewing tasks](https://md.research.elamur.ai/free/en/papers/zhou-2024-is-llm-reliable-reviewer-comprehensive/). In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 9340–9351.
28. Minjun Zhu, Yixuan Weng, Linyi Yang, and Yue Zhang. 2025. Deepreview: Improving llm-based paper review with human-like deep thinking process. arXiv preprint arXiv:2503.08569. [find](https://scholar.google.com/scholar?q=Minjun+Zhu%2C+Yixuan+Weng%2C+Linyi+Yang%2C+and+Yue+Zhang%2E+2025%2E+Deepreview%3A+Improving+llm%2Dbased+paper+review+with+human%2Dlike+deep+thinking+process%2E+arXiv+preprint+arXiv%3A2503%2E08569%2E)

*Updated 1 November 2026*