---
title: "Preference Optimization for Review Question Generation"
description: "LLM-drafted peer-review questions are often shallow. 0.78 points. 27 references. Among them: 0."
authors: ["Karun Sharma", "Vidushee Vats", "Shengzhi LI", "Yuxiang Wang", "Zhongtian Sun", "Prayag Tiwari"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.1256/"
doi: "10.18653/v1/2026.findings-acl.1256"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sharma-2026-preference-optimization-review-question-generation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sharma-2026-preference-optimization-review-question-generation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/39/) / Preference Optimization for Review Question Generation Improves Writing Quality

# Preference Optimization for Review Question Generation Improves Writing Quality

**Authors:** Karun Sharma, Vidushee Vats, [Shengzhi LI](https://md.research.elamur.ai/free/en/authors/shengzhi-li/), [Yuxiang Wang](https://md.research.elamur.ai/free/en/authors/yuxiang-wang/), Zhongtian Sun and [Prayag Tiwari](https://md.research.elamur.ai/free/en/authors/prayag-tiwari/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1256/) · [PDF](https://aclanthology.org/2026.findings-acl.1256.pdf) · [DOI 10.18653/v1/2026.findings-acl.1256](https://doi.org/10.18653/v1/2026.findings-acl.1256) · [OpenAlex](https://openalex.org/W7130617666)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
LLM-drafted peer-review questions are often shallow. They tend to echo keywords or the introduction instead of engaging critically with the whole paper. To address this, the authors built a corpus of reviewer questions from ICLR 2024 reviews on OpenReview, using Gemini 2.0 for extraction and olmOCR for paper text. They then ran an expert annotation study that scored question–paper pairs on three binary criteria: Effort, Evidence and Grounding. In this study, human-written questions clearly beat those from frontier models. The annotations were used to train IntelliReward, a reward model that adds trainable multi-head transformer layers on top of a frozen autoregressive LLM. IntelliReward agrees with experts better than SFT-tuned API judges. It then served as the reward signal for DAPO reinforcement learning, producing IntelliAsk. In human evaluation, IntelliAsk's questions are more grounded and effortful than those of SFT and frontier baselines, and they depend less on first-page content. The gains also carry over to general reasoning and writing benchmarks. This suggests that learning to ask good questions reflects broader model ability.

## Key points
- **Dataset:** 15.5k filtered reviewer questions from 5,841 papers, split into 13.2k for training and 2.3k for testing.
- **Preference study:** 572 annotated question–paper pairs drawn from 300 ICLR 2025 submissions.
  - On the 0–3 rubric, human questions averaged 0.78 points above the best LLM and 1.53 above the weakest.
  - The weakest LLM was Qwen2.5-32B; 55% of the words in its questions came from the paper's first page.
- **Methods:** IntelliReward, a frozen LLM with multi-head transformer heads, serves as the reward model. IntelliAsk is trained with DAPO at 7B and 32B scales.
- **Transfer results:** IntelliAsk-32B outperforms Qwen3-32B on MuSR (68.3 vs 64.7 accuracy) and WritingBench (8.31 vs 8.07). It also improves on Arena Hard.
- **Release:** The authors have publicly shared their code, the cleaned set of reviewer questions, the expert-labeled data, and both trained models (IntelliAsk and IntelliReward). The data is released under CC BY 4.0.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 441 works

## Cite

Cite as: Karun Sharma, Vidushee Vats, Shengzhi LI, Yuxiang Wang, Zhongtian Sun and Prayag Tiwari. 2026. Preference Optimization for Review Question Generation Improves Writing Quality. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{sharma2026preference,
  title = {{Preference Optimization for Review Question Generation Improves Writing Quality}},
  author = {Karun Sharma, Vidushee Vats, Shengzhi LI, Yuxiang Wang, Zhongtian Sun and Prayag Tiwari},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1256},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1256}
}
```

- [← previous Predicting Topic (Co-)Occurrence Using Topic Networks Built from the Project Gutenberg Corpus](https://md.research.elamur.ai/free/en/papers/verma-2026-predicting-topic-co-occurrence-topic/)
- [next → Prefix Parsing is Just Parsing](https://md.research.elamur.ai/free/en/papers/pasti-2026-prefix-parsing-is-just-parsing/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/39/)

## References

27 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Tim Baumgärtner, Ted Briscoe, and Iryna Gurevych. 2025. PeerQA: A scientific question answering dataset from peer reviews. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers… [find](https://scholar.google.com/scholar?q=Tim+Baumg%C3%A4rtner%2C+Ted+Briscoe%2C+and+Iryna+Gurevych%2E+2025%2E+PeerQA%3A+A+scientific+question+answering+dataset+from+peer+reviews%2E+In+Proceedings+of+the+2025+Conference+of+the+Nations+of+the+Americas+Chapter+)
2. Eric Chamoun, Michael Schlichtkrull, and Andreas Vlachos. 2024. Automated focused feedback generation for scientific writing assistance. In Findings of the Association for Computational Linguistics: ACL 2024, pages 9742–9763, Bangkok, Thailand. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Eric+Chamoun%2C+Michael+Schlichtkrull%2C+and+Andreas+Vlachos%2E+2024%2E+Automated+focused+feedback+generation+for+scientific+writing+assistance%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+A)
3. Maitreya Prafulla Chitale, Ketaki Mangesh Shetye, Harshit Gupta, Manav Chaudhary, and Vasudeva Varma. 2025. Autorev: Automatic peer review system for academic research papers. arXiv preprint arXiv: 2505.14376. — [arXiv:2505.14376](https://arxiv.org/abs/2505.14376) [find](https://scholar.google.com/scholar?q=Maitreya+Prafulla+Chitale%2C+Ketaki+Mangesh+Shetye%2C+Harshit+Gupta%2C+Manav+Chaudhary%2C+and+Vasudeva+Varma%2E+2025%2E+Autorev%3A+Automatic+peer+review+system+for+academic+research+papers%2E+arXiv+preprint+arXiv%3A+25)
4. Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. 2025. [SFT memorizes, RL generalizes: A comparative study of foundation model post-training](https://md.research.elamur.ai/free/en/papers/chu-2025-sft-memorizes-rl-generalizes-comparative/). In Forty-second International Conference on Machine Learning.
5. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. [Boolq: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). In NAACL.
6. Mike D’Arcy, Tom Hope, Larry Birnbaum, and Doug Downey. 2024. Marg: Multi-agent review generation for scientific papers. Preprint, arXiv:2401.04259. [find](https://scholar.google.com/scholar?q=Mike+D%E2%80%99Arcy%2C+Tom+Hope%2C+Larry+Birnbaum%2C+and+Doug+Downey%2E+2024%2E+Marg%3A+Multi%2Dagent+review+generation+for+scientific+papers%2E+Preprint%2C+arXiv%3A2401%2E04259%2E)
7. Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. [A dataset of information-seeking questions and answers anchored in research papers](https://md.research.elamur.ai/free/en/papers/dasigi-2021-dataset-information-seeking-questions-answers/). arXiv preprint arXiv: 2105.03011.
8. D. Davis. 2021. Cvpr 2021 training materials: Reference slides. [find](https://scholar.google.com/scholar?q=D%2E+Davis%2E+2021%2E+Cvpr+2021+training+materials%3A+Reference+slides%2E)
9. Jiangshu Du, Yibo Wang, Wenting Zhao, Zhongfen Deng, Shuaiqi Liu, Renze Lou, Henry Peng Zou, Pranav Narayanan Venkit, Nan Zhang, Mukund Srinath, Haoran Ranran Zhang, Vipul Gupta, Yinghui Li, Tao Li, Fei Wang, Qin Liu, Tianlin Liu, Pengzhi Gao, Congying Xia, and 21 others. 2024. [LLMs assist NLP resea…](https://md.research.elamur.ai/free/en/papers/du-2024-llms-assist-nlp-researchers-critique/)
10. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. [Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/). Preprint, arXiv:1903.00161.
11. ICLR. 2025. Leveraging llm feedback to enhance review quality. [find](https://scholar.google.com/scholar?q=ICLR%2E+2025%2E+Leveraging+llm+feedback+to+enhance+review+quality%2E)
12. Maximilian Idahl and Zahra Ahmadi. 2025. OpenReviewer: A specialized large language model for generating critical scientific paper reviews. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Syst… [find](https://scholar.google.com/scholar?q=Maximilian+Idahl+and+Zahra+Ahmadi%2E+2025%2E+OpenReviewer%3A+A+specialized+large+language+model+for+generating+critical+scientific+paper+reviews%2E+In+Proceedings+of+the+2025+Conference+of+the+Nations+of+the+)
13. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. 2024. [From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline](https://md.research.elamur.ai/free/en/papers/li-2024-crowdsourced-data-high-quality-benchmarks/). Preprint, arXiv:2406.11939.
14. Weixin Liang, Zachary Izzo, Yaohui Zhang, Haley Lepp, Hancheng Cao, Xuandong Zhao, Lingjiao Chen, Haotian Ye, Sheng Liu, Zhi Huang, Daniel A. McFarland, and James Y. Zou. 2024. Monitoring ai-modified content at scale: A case study on the impact of chatgpt on ai conference peer reviews. In ICML. [find](https://scholar.google.com/scholar?q=Weixin+Liang%2C+Zachary+Izzo%2C+Yaohui+Zhang%2C+Haley+Lepp%2C+Hancheng+Cao%2C+Xuandong+Zhao%2C+Lingjiao+Chen%2C+Haotian+Ye%2C+Sheng+Liu%2C+Zhi+Huang%2C+Daniel+A%2E+McFarland%2C+and+James+Y%2E+Zou%2E+2024%2E+Monitoring+ai%2Dmodified+)
15. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2022. [Webgpt: Browser-assis…](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/)
16. NeurIPS. 2023. Reviewer guidelines. [find](https://scholar.google.com/scholar?q=NeurIPS%2E+2023%2E+Reviewer+guidelines%2E)
17. Kun-Peng Ning, Shuo Yang, Yuyang Liu, Jia-Yu Yao, Zhenhui Liu, Yonghong Tian, Yibing Song, and Li Yuan. 2025. PiCO: Peer review in LLMs based on consistency optimization. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Kun%2DPeng+Ning%2C+Shuo+Yang%2C+Yuyang+Liu%2C+Jia%2DYu+Yao%2C+Zhenhui+Liu%2C+Yonghong+Tian%2C+Yibing+Song%2C+and+Li+Yuan%2E+2025%2E+PiCO%3A+Peer+review+in+LLMs+based+on+consistency+optimization%2E+In+The+Thirteenth+Internation)
18. Jake Poznanski, Jon Borchardt, Jason Dunkelberger, Regan Huff, Daniel Lin, Aman Rangapur, Christopher Wilhelm, Kyle Lo, and Luca Soldaini. 2025. [olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models](https://md.research.elamur.ai/free/en/papers/poznanski-2025-olmocr-unlocking-trillions-tokens-pdfs/). Preprint, arXiv:2502.18443.
19. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2023. [Gpqa: A graduate-level google-proof qa benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/). Preprint, arXiv:2311.12022.
20. Shruti Singh, Nandan Sarkar, and Arman Cohan. 2024. SciDQA: A deep reading comprehension dataset over scientific papers. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20908–20923, Miami, Florida, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shruti+Singh%2C+Nandan+Sarkar%2C+and+Arman+Cohan%2E+2024%2E+SciDQA%3A+A+deep+reading+comprehension+dataset+over+scientific+papers%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Natural+Language+)
21. Zayne Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. 2024. Musr: Testing the limits of chain-of-thought with multistep soft reasoning. Preprint, arXiv:2310.16049. [find](https://scholar.google.com/scholar?q=Zayne+Sprague%2C+Xi+Ye%2C+Kaj+Bostrom%2C+Swarat+Chaudhuri%2C+and+Greg+Durrett%2E+2024%2E+Musr%3A+Testing+the+limits+of+chain%2Dof%2Dthought+with+multistep+soft+reasoning%2E+Preprint%2C+arXiv%3A2310%2E16049%2E)
22. Anirudh Sundar, Jin Xu, William Gay, Christopher Gordon Richardson, and Larry Heck. 2024. cPAPERS: A dataset of situated and multimodal interactive conversations in scientific papers. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track. [find](https://scholar.google.com/scholar?q=Anirudh+Sundar%2C+Jin+Xu%2C+William+Gay%2C+Christopher+Gordon+Richardson%2C+and+Larry+Heck%2E+2024%2E+cPAPERS%3A+A+dataset+of+situated+and+multimodal+interactive+conversations+in+scientific+papers%2E+In+The+Thirty%2Dei)
23. Cheng Tan, Dongxin Lyu, Siyuan Li, Zhangyang Gao, Jingxuan Wei, Siqi Ma, Zicheng Liu, and Stan Z. Li. 2025. Peer review as a multi-turn and long-context dialogue with role-based interactions: Benchmarking large language models. [find](https://scholar.google.com/scholar?q=Cheng+Tan%2C+Dongxin+Lyu%2C+Siyuan+Li%2C+Zhangyang+Gao%2C+Jingxuan+Wei%2C+Siqi+Ma%2C+Zicheng+Liu%2C+and+Stan+Z%2E+Li%2E+2025%2E+Peer+review+as+a+multi%2Dturn+and+long%2Dcontext+dialogue+with+role%2Dbased+interactions%3A+Benchmar)
24. Nitya Thakkar, Mert Yuksekgonul, Jake Silberg, Animesh Garg, Nanyun Peng, Fei Sha, Rose Yu, Carl Vondrick, and James Zou. 2025. Can llm feedback enhance review quality? a randomized study of 20k reviews at iclr 2025. arXiv preprint arXiv: 2504.09737. [find](https://scholar.google.com/scholar?q=Nitya+Thakkar%2C+Mert+Yuksekgonul%2C+Jake+Silberg%2C+Animesh+Garg%2C+Nanyun+Peng%2C+Fei+Sha%2C+Rose+Yu%2C+Carl+Vondrick%2C+and+James+Zou%2E+2025%2E+Can+llm+feedback+enhance+review+quality%3F+a+randomized+study+of+20k+revie)
25. Yuning Wu, Jiahao Mei, Ming Yan, Chenliang Li, Shaopeng Lai, Yuran Ren, Zijia Wang, Ji Zhang, Mengyue Wu, Qin Jin, and Fei Huang. 2025. Writingbench: A comprehensive benchmark for generative writing. Preprint, arXiv:2503.05244. — [arXiv:2503.05244](https://arxiv.org/abs/2503.05244) [find](https://scholar.google.com/scholar?q=Yuning+Wu%2C+Jiahao+Mei%2C+Ming+Yan%2C+Chenliang+Li%2C+Shaopeng+Lai%2C+Yuran+Ren%2C+Zijia+Wang%2C+Ji+Zhang%2C+Mengyue+Wu%2C+Qin+Jin%2C+and+Fei+Huang%2E+2025%2E+Writingbench%3A+A+comprehensive+benchmark+for+generative+writing%2E+)
26. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, and 16 others. 2025. [Dapo: An open-source llm reinforcement learning s…](https://md.research.elamur.ai/free/en/papers/yu-2025-dapo-open-source-llm-reinforcement/)
27. Minjun Zhu, Yixuan Weng, Linyi Yang, and Yue Zhang. 2025. DeepReview: Improving LLM-based paper review with human-like deep thinking process. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 29330–29355, Vienna, Austria. Associ… [find](https://scholar.google.com/scholar?q=Minjun+Zhu%2C+Yixuan+Weng%2C+Linyi+Yang%2C+and+Yue+Zhang%2E+2025%2E+DeepReview%3A+Improving+LLM%2Dbased+paper+review+with+human%2Dlike+deep+thinking+process%2E+In+Proceedings+of+the+63rd+Annual+Meeting+of+the+Associati)

*Updated 1 November 2026*