---
title: "Joint Optimization of Training Data and Policy in RLHF — Yu 2026"
description: "As a remedy, the authors introduce JODP, a method that optimizes the training data and the policy together. 68 references."
authors: ["Zhuohao Yu", "Jiali Zeng", "Weizheng Gu", "Mengyuan Sun", "Yidong Wang", "Fandong Meng", "Jie Zhou", "Shikun Zhang", "Wei Ye"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.2109/"
doi: "10.18653/v1/2026.findings-acl.2109"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yu-2026-joint-optimization-training-data-policy/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yu-2026-joint-optimization-training-data-policy/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/38/) / Joint Optimization of Training Data and Policy in RLHF

# Joint Optimization of Training Data and Policy in RLHF

**Authors:** [Zhuohao Yu](https://md.research.elamur.ai/free/en/authors/zhuohao-yu/), [Jiali Zeng](https://md.research.elamur.ai/free/en/authors/jiali-zeng/), Weizheng Gu, Mengyuan Sun, [Yidong Wang](https://md.research.elamur.ai/free/en/authors/yidong-wang/), [Fandong Meng](https://md.research.elamur.ai/free/en/authors/fandong-meng/), [Jie Zhou](https://md.research.elamur.ai/free/en/authors/jie-zhou/), [Shikun Zhang](https://md.research.elamur.ai/free/en/authors/shikun-zhang/) and [Wei Ye](https://md.research.elamur.ai/free/en/authors/wei-ye/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.2109/) · [PDF](https://aclanthology.org/2026.findings-acl.2109.pdf) · [DOI 10.18653/v1/2026.findings-acl.2109](https://doi.org/10.18653/v1/2026.findings-acl.2109) · [OpenAlex](https://openalex.org/W7166815635)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Standard RLHF and RLVR pipelines sample many responses per training prompt but never change the prompt itself. When a model struggles with a problem, extra samples of the same formulation add little new learning signal. As a remedy, the authors introduce JODP, a method that optimizes the training data and the policy together. In this framework, the policy being trained also writes specifications (rubrics, principles or behavioural guidelines) that are prepended to each fixed problem. A UCB bandit decides which specifications to reuse and which new candidates to try. The augmented prompts are used only to produce rollouts. Rewards and gradients are computed against the original problem, so the model must learn to reproduce hint-guided behaviour without the hint, and the hint cannot be used to game the reward. Specifications are dropped at deployment. On safety alignment tasks, JODP plugs into existing optimizers without changing their objectives. It lets a Qwen3-4B model get close to 8B-level results at under 1% added compute, which offers a cheap way to enrich training signal.

## Key points
- The method runs in three stages: the policy generates specifications and a UCB bandit selects among them; rollouts are sampled from specification plus problem but scored on the problem alone; then bandit statistics and policy weights are updated together.
- JODP is algorithm-agnostic and was shown to improve GRPO, RLOO and REINFORCE++; it needs no extra models or external supervision.
- GRPO reached the best absolute scores but gained least from JODP, while RLOO and REINFORCE++ gained more and drifted further from the initial policy in KL divergence.
- With JODP, Qwen3-4B approaches 8B performance at less than 1% extra computational cost.
- In a qualitative study on XSTest prompts, the RLOO + JODP model followed a "humorous, empathetic, concise, direct" alignment goal more closely than vanilla RLOO, whose answers were more formal and generic.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 315 works · 43,267 citations
- [Tencent](https://md.research.elamur.ai/free/en/companies/tencent/) — Company, Shenzhen, China · 163 works · 21,032 citations

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works

## Cite

Cite as: Zhuohao Yu, Jiali Zeng, Weizheng Gu, Mengyuan Sun, Yidong Wang, Fandong Meng, Jie Zhou, Shikun Zhang and Wei Ye. 2026. Joint Optimization of Training Data and Policy in RLHF. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yu2026joint,
  title = {{Joint Optimization of Training Data and Policy in RLHF}},
  author = {Zhuohao Yu, Jiali Zeng, Weizheng Gu, Mengyuan Sun, Yidong Wang, Fandong Meng, Jie Zhou, Shikun Zhang and Wei Ye},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.2109},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.2109}
}
```

- [← previous JobArabi: An Arabic Corpus and Analysis of Job Announcements from Social Media](https://md.research.elamur.ai/free/en/papers/zaghouani-2026-jobarabi-arabic-corpus-analysis-job/)
- [next → JointCQ: Improving Factual Hallucination Detection with Joint Claim and Query Generation](https://md.research.elamur.ai/free/en/papers/xu-2026-jointcq-improving-factual-hallucination-detection/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/38/)

## References

68 references, 31 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, and 1 others. 2025. Gepa: Reflective prompt evolution can outperform reinforcement learning. arXiv preprint arXiv:2507.19457. [find](https://scholar.google.com/scholar?q=Lakshya+A+Agrawal%2C+Shangyin+Tan%2C+Dilara+Soylu%2C+Noah+Ziems%2C+Rishi+Khare%2C+Krista+Opsahl%2DOng%2C+Arnav+Singhvi%2C+Herumb+Shandilya%2C+Michael+J+Ryan%2C+Meng+Jiang%2C+and+1+others%2E+2025%2E+Gepa%3A+Reflective+prompt+evol)
2. Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. 2024. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740. [find](https://scholar.google.com/scholar?q=Arash+Ahmadian%2C+Chris+Cremer%2C+Matthias+Gall%C3%A9%2C+Marzieh+Fadaee%2C+Julia+Kreutzer%2C+Olivier+Pietquin%2C+Ahmet+%C3%9Cst%C3%BCn%2C+and+Sara+Hooker%2E+2024%2E+Back+to+basics%3A+Revisiting+reinforce+style+optimization+for+learning)
3. Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. 2024. [A general theoretical paradigm to understand learning from human preferences](https://md.research.elamur.ai/free/en/papers/azar-2023-general-theoretical-paradigm-understand-learning/). In International Conference on Artificial Intelligence and Statistics, pages 4447–4455. PMLR…
4. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, and 1 others. 2022. [Constitutional ai: Harmlessness from ai feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/). arXiv preprint arXiv:2212.08073.
5. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, and 1 others. 2023. [A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity](https://md.research.elamur.ai/free/en/papers/bang-2023-multitask-multilingual-multimodal-evaluation-chatgpt/). arXiv preprint arXiv:2302.04023.
6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
7. Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, and 1 others. 2023. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217. [find](https://scholar.google.com/scholar?q=Stephen+Casper%2C+Xander+Davies%2C+Claudia+Shi%2C+Thomas+Krendl+Gilbert%2C+J%C3%A9r%C3%A9my+Scheurer%2C+Javier+Rando%2C+Rachel+Freedman%2C+Tomasz+Korbak%2C+David+Lindner%2C+Pedro+Freire%2C+and+1+others%2E+2023%2E+Open+problems+and+fun)
8. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, and 1 others. 2023. [A survey on evaluation of large language models](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/). arXiv preprint arXiv:2307.03109.
9. Yuyan Chen, Zhihao Wen, Ge Fan, Zhengyu Chen, Wei Wu, Dayiheng Liu, Zhixu Li, Bang Liu, and Yanghua Xiao. 2024a. Mapo: Boosting large language model performance with model-adaptive prompt optimization. arXiv preprint arXiv:2407.04118. [find](https://scholar.google.com/scholar?q=Yuyan+Chen%2C+Zhihao+Wen%2C+Ge+Fan%2C+Zhengyu+Chen%2C+Wei+Wu%2C+Dayiheng+Liu%2C+Zhixu+Li%2C+Bang+Liu%2C+and+Yanghua+Xiao%2E+2024a%2E+Mapo%3A+Boosting+large+language+model+performance+with+model%2Dadaptive+prompt+optimization)
10. Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. 2024b. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335. [find](https://scholar.google.com/scholar?q=Zixiang+Chen%2C+Yihe+Deng%2C+Huizhuo+Yuan%2C+Kaixuan+Ji%2C+and+Quanquan+Gu%2E+2024b%2E+Self%2Dplay+fine%2Dtuning+converts+weak+language+models+to+strong+language+models%2E+arXiv+preprint+arXiv%3A2401%2E01335%2E)
11. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30. [find](https://scholar.google.com/scholar?q=Paul+F+Christiano%2C+Jan+Leike%2C+Tom+Brown%2C+Miljan+Martic%2C+Shane+Legg%2C+and+Dario+Amodei%2E+2017%2E+Deep+reinforcement+learning+from+human+preferences%2E+Advances+in+neural+information+processing+systems%2C+30%2E)
12. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
13. Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P Xing, and Zhiting Hu. 2022. [Rlprompt: Optimizing discrete text prompts with reinforcement learning](https://md.research.elamur.ai/free/en/papers/deng-2022-rlprompt-optimizing-discrete-text-prompts/). arXiv preprint arXiv:2205.12548.
14. Shizhe Diao, Pengcheng Wang, Yong Lin, Rui Pan, Xiang Liu, and Tong Zhang. 2023. Active prompting with chain-of-thought for large language models. arXiv preprint arXiv:2302.12246. [find](https://scholar.google.com/scholar?q=Shizhe+Diao%2C+Pengcheng+Wang%2C+Yong+Lin%2C+Rui+Pan%2C+Xiang+Liu%2C+and+Tong+Zhang%2E+2023%2E+Active+prompting+with+chain%2Dof%2Dthought+for+large+language+models%2E+arXiv+preprint+arXiv%3A2302%2E12246%2E)
15. Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305. [find](https://scholar.google.com/scholar?q=Jesse+Dodge%2C+Gabriel+Ilharco%2C+Roy+Schwartz%2C+Ali+Farhadi%2C+Hannaneh+Hajishirzi%2C+and+Noah+Smith%2E+2020%2E+Fine%2Dtuning+pretrained+language+models%3A+Weight+initializations%2C+data+orders%2C+and+early+stopping%2E+arX)
16. Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. 2023. [Promptbreeder: Self-referential self-improvement via prompt evolution](https://md.research.elamur.ai/free/en/papers/fernando-2023-promptbreeder-self-referential-self-improvement/). arXiv preprint arXiv:2309.16797.
17. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
18. Jiayi He, Hehai Lin, Qingyun Wang, Yi Fung, and Heng Ji. 2024. Self-correction is more than refinement: A learning framework for visual and language reasoning tasks. arXiv preprint arXiv:2410.04055. [find](https://scholar.google.com/scholar?q=Jiayi+He%2C+Hehai+Lin%2C+Qingyun+Wang%2C+Yi+Fung%2C+and+Heng+Ji%2E+2024%2E+Self%2Dcorrection+is+more+than+refinement%3A+A+learning+framework+for+visual+and+language+reasoning+tasks%2E+arXiv+preprint+arXiv%3A2410%2E04055%2E)
19. Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. 2025a. [Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models](https://md.research.elamur.ai/free/en/papers/hu-2025-reinforce-stabilizing-critic-free-policy/). arXiv preprint arXiv:2501.03262.
20. Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. 2025b. [Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models](https://md.research.elamur.ai/free/en/papers/hu-2025-reinforce-stabilizing-critic-free-policy/). arXiv preprint arXiv:2501.03262.
21. Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022. [Large language models can self-improve](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-can-self/). arXiv preprint arXiv:2210.11610.
22. Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Juntao Dai, Boren Zheng, Tianyi Qiu, Jiayi Zhou, Kaile Wang, Boxuan Li, and 1 others. 2024. [Pku-saferlhf: Towards multi-level safety alignment for llms with human preference](https://md.research.elamur.ai/free/en/papers/ji-2025-pku-saferlhf-multi-level-safety/). arXiv preprint arXiv:2406.15513.
23. Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. 2024. [A survey of reinforcement learning from human feedback](https://md.research.elamur.ai/free/en/papers/kaufmann-2023-survey-reinforcement-learning-human-feedback/).
24. Haruka Kiyohara, Daniel Yiming Cao, Yuta Saito, and Thorsten Joachims. 2025. Prompt optimization with logged bandit data. arXiv preprint arXiv:2504.02646. — [arXiv:2504.02646](https://arxiv.org/abs/2504.02646) [find](https://scholar.google.com/scholar?q=Haruka+Kiyohara%2C+Daniel+Yiming+Cao%2C+Yuta+Saito%2C+and+Thorsten+Joachims%2E+2025%2E+Prompt+optimization+with+logged+bandit+data%2E+arXiv+preprint+arXiv%3A2504%2E02646%2E+%E2%80%94+%5BarXiv%3A2504%2E02646%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F25)
25. Weize Kong, Spurthi Amba Hombaiah, Mingyang Zhang, Qiaozhu Mei, and Michael Bendersky. 2024. Prewrite: Prompt rewriting with reinforcement learning. arXiv preprint arXiv:2401.08189. [find](https://scholar.google.com/scholar?q=Weize+Kong%2C+Spurthi+Amba+Hombaiah%2C+Mingyang+Zhang%2C+Qiaozhu+Mei%2C+and+Michael+Bendersky%2E+2024%2E+Prewrite%3A+Prompt+rewriting+with+reinforcement+learning%2E+arXiv+preprint+arXiv%3A2401%2E08189%2E)
26. Minchan Kwon, Gaeun Kim, Jongsuk Kim, Haeil Lee, and Junmo Kim. 2024. Stableprompt: automatic prompt tuning using reinforcement learning for large language models. arXiv preprint arXiv:2410.07652. — [arXiv:2410.07652](https://arxiv.org/abs/2410.07652) [find](https://scholar.google.com/scholar?q=Minchan+Kwon%2C+Gaeun+Kim%2C+Jongsuk+Kim%2C+Haeil+Lee%2C+and+Junmo+Kim%2E+2024%2E+Stableprompt%3A+automatic+prompt+tuning+using+reinforcement+learning+for+large+language+models%2E+arXiv+preprint+arXiv%3A2410%2E07652%2E+%E2%80%94+%5B)
27. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, and 1 others. 2024. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787. [find](https://scholar.google.com/scholar?q=Nathan+Lambert%2C+Valentina+Pyatkin%2C+Jacob+Morrison%2C+LJ+Miranda%2C+Bill+Yuchen+Lin%2C+Khyathi+Chandu%2C+Nouha+Dziri%2C+Sachin+Kumar%2C+Tom+Zick%2C+Yejin+Choi%2C+and+1+others%2E+2024%2E+Rewardbench%3A+Evaluating+reward+mode)
28. Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and 1 others. 2023. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267. [find](https://scholar.google.com/scholar?q=Harrison+Lee%2C+Samrat+Phatale%2C+Hassan+Mansoor%2C+Thomas+Mesnard%2C+Johan+Ferret%2C+Kellie+Lu%2C+Colton+Bishop%2C+Ethan+Hall%2C+Victor+Carbune%2C+Abhinav+Rastogi%2C+and+1+others%2E+2023%2E+Rlaif+vs%2E+rlhf%3A+Scaling+reinforce)
29. Rui Li, Peiyi Wang, Jingyuan Ma, Di Zhang, Lei Sha, and Zhifang Sui. 2025. Be a multitude to itself: A prompt evolution framework for red teaming. arXiv preprint arXiv:2502.16109. [find](https://scholar.google.com/scholar?q=Rui+Li%2C+Peiyi+Wang%2C+Jingyuan+Ma%2C+Di+Zhang%2C+Lei+Sha%2C+and+Zhifang+Sui%2E+2025%2E+Be+a+multitude+to+itself%3A+A+prompt+evolution+framework+for+red+teaming%2E+arXiv+preprint+arXiv%3A2502%2E16109%2E)
30. Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. 2023. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. arXiv preprint arXiv:2310.10505. [find](https://scholar.google.com/scholar?q=Ziniu+Li%2C+Tian+Xu%2C+Yushun+Zhang%2C+Zhihang+Lin%2C+Yang+Yu%2C+Ruoyu+Sun%2C+and+Zhi%2DQuan+Luo%2E+2023%2E+Remax%3A+A+simple%2C+effective%2C+and+efficient+reinforcement+learning+method+for+aligning+large+language+models%2E+ar)
31. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. [Let’s verify step by step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/). In The Twelfth International Conference on Learning Representations.
32. Xiaoqiang Lin, Zhongxiang Dai, Arun Verma, See-Kiong Ng, Patrick Jaillet, and Bryan Kian Hsiang Low. 2024. Prompt optimization with human feedback. arXiv preprint arXiv:2405.17346. — [arXiv:2405.17346](https://arxiv.org/abs/2405.17346) [find](https://scholar.google.com/scholar?q=Xiaoqiang+Lin%2C+Zhongxiang+Dai%2C+Arun+Verma%2C+See%2DKiong+Ng%2C+Patrick+Jaillet%2C+and+Bryan+Kian+Hsiang+Low%2E+2024%2E+Prompt+optimization+with+human+feedback%2E+arXiv+preprint+arXiv%3A2405%2E17346%2E+%E2%80%94+%5BarXiv%3A2405%2E17346)
33. Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. 2024. [Skywork-reward: Bag of tricks for reward modeling in llms](https://md.research.elamur.ai/free/en/papers/liu-2024-skywork-reward-bag-tricks/). arXiv preprint arXiv:2410.18451.
34. Kawin Mayilvaghanan, Varun Nathan, and Ayush Kumar. 2025. Propel: Prompt optimization with expert priors for small and medium-sized llms. In Proceedings of the 4th International Workshop on Knowledge-Augmented Methods for Natural Language Processing, pages 272–302. [find](https://scholar.google.com/scholar?q=Kawin+Mayilvaghanan%2C+Varun+Nathan%2C+and+Ayush+Kumar%2E+2025%2E+Propel%3A+Prompt+optimization+with+expert+priors+for+small+and+medium%2Dsized+llms%2E+In+Proceedings+of+the+4th+International+Workshop+on+Knowledge%2D)
35. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv:2303.08774.
36. Preprint, Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:2773…
37. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). Advances in neural information processing systems, 36:53728–53741.
38. Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He. 2021. [Zero-infinity: Breaking the gpu memory wall for extreme scale deep learning](https://md.research.elamur.ai/free/en/papers/rajbhandari-2021-zero-infinity-breaking-gpu-memory/). In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–14.
39. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505–3506. [find](https://scholar.google.com/scholar?q=Jeff+Rasley%2C+Samyam+Rajbhandari%2C+Olatunji+Ruwase%2C+and+Yuxiong+He%2E+2020%2E+Deepspeed%3A+System+optimizations+enable+training+deep+learning+models+with+over+100+billion+parameters%2E+In+Proceedings+of+the+26t)
40. Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. 2023. [Xstest: A test suite for identifying exaggerated safety behaviours in large language models](https://md.research.elamur.ai/free/en/papers/rottger-2023-xstest-test-suite-identifying-exaggerated/). arXiv preprint arXiv:2308.01263.
41. Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. 2024. [A systematic survey of prompt engineering in large language models: Techniques and applications](https://md.research.elamur.ai/free/en/papers/sahoo-2024-systematic-survey-prompt-engineering-large/). arXiv preprint arXiv:2402.07927.
42. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+2017%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2E)
43. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). arXiv preprint arXiv:2402.03300.
44. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. [Hybridflow: A flexible and efficient rlhf framework](https://md.research.elamur.ai/free/en/papers/sheng-2024-hybridflow-flexible-efficient-rlhf-framework/). In Proceedings of the Twentieth European Conference on Computer Systems, pages 1279–1297.
45. Wenlei Shi and Xing Jin. 2025. Heimdall: test-time scaling on the generative verification. arXiv preprint arXiv:2504.10337. — [arXiv:2504.10337](https://arxiv.org/abs/2504.10337) [find](https://scholar.google.com/scholar?q=Wenlei+Shi+and+Xing+Jin%2E+2025%2E+Heimdall%3A+test%2Dtime+scaling+on+the+generative+verification%2E+arXiv+preprint+arXiv%3A2504%2E10337%2E+%E2%80%94+%5BarXiv%3A2504%2E10337%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2504%2E10337%29)
46. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008– 3021. [find](https://scholar.google.com/scholar?q=Nisan+Stiennon%2C+Long+Ouyang%2C+Jeffrey+Wu%2C+Daniel+Ziegler%2C+Ryan+Lowe%2C+Chelsea+Voss%2C+Alec+Radford%2C+Dario+Amodei%2C+and+Paul+F+Christiano%2E+2020%2E+Learning+to+summarize+with+human+feedback%2E+Advances+in+neural)
47. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30.
48. Ruochen Wang, Ting Liu, Cho-Jui Hsieh, and Boqing Gong. 2024a. On discrete prompt optimization for diffusion models. arXiv preprint arXiv:2407.01606. — [arXiv:2407.01606](https://arxiv.org/abs/2407.01606) [find](https://scholar.google.com/scholar?q=Ruochen+Wang%2C+Ting+Liu%2C+Cho%2DJui+Hsieh%2C+and+Boqing+Gong%2E+2024a%2E+On+discrete+prompt+optimization+for+diffusion+models%2E+arXiv+preprint+arXiv%3A2407%2E01606%2E+%E2%80%94+%5BarXiv%3A2407%2E01606%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2407%2E01)
49. Yidong Wang, Zhuohao Yu, Jindong Wang, Qiang Heng, Hao Chen, Wei Ye, Rui Xie, Xing Xie, and Shikun Zhang. 2024b. Exploring vision-language models for imbalanced learning. International Journal of Computer Vision, 132(1):224–237. [find](https://scholar.google.com/scholar?q=Yidong+Wang%2C+Zhuohao+Yu%2C+Jindong+Wang%2C+Qiang+Heng%2C+Hao+Chen%2C+Wei+Ye%2C+Rui+Xie%2C+Xing+Xie%2C+and+Shikun+Zhang%2E+2024b%2E+Exploring+vision%2Dlanguage+models+for+imbalanced+learning%2E+International+Journal+of+Comp)
50. Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, and 1 others. 2023. [Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization](https://md.research.elamur.ai/free/en/papers/wang-2023-pandalm-automatic-evaluation-benchmark-llm/). arXiv preprint arXiv:2306.05087.
51. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+2022%2E+Self%2Dinstruct%3A+Aligning+language+models+with+self%2Dgenerated+instructions%2E+arXiv+pr)
52. Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. 2022. [Large language models are better reasoners with self-verification](https://md.research.elamur.ai/free/en/papers/weng-2023-large-language-models-are-better/). arXiv preprint arXiv:2212.09561.
53. Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. 2024. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.00675. — [arXiv:2405.00675](https://arxiv.org/abs/2405.00675) [find](https://scholar.google.com/scholar?q=Yue+Wu%2C+Zhiqing+Sun%2C+Huizhuo+Yuan%2C+Kaixuan+Ji%2C+Yiming+Yang%2C+and+Quanquan+Gu%2E+2024%2E+Self%2Dplay+preference+optimization+for+language+model+alignment%2E+arXiv+preprint+arXiv%3A2405%2E00675%2E+%E2%80%94+%5BarXiv%3A2405%2E00675%5D)
54. Qiujie Xie, Qingqiu Li, Zhuohao Yu, Yuejie Zhang, Yue Zhang, and Linyi Yang. 2025. An empirical analysis of uncertainty in large language model evaluations. arXiv preprint arXiv:2502.10709. — [arXiv:2502.10709](https://arxiv.org/abs/2502.10709) [find](https://scholar.google.com/scholar?q=Qiujie+Xie%2C+Qingqiu+Li%2C+Zhuohao+Yu%2C+Yuejie+Zhang%2C+Yue+Zhang%2C+and+Linyi+Yang%2E+2025%2E+An+empirical+analysis+of+uncertainty+in+large+language+model+evaluations%2E+arXiv+preprint+arXiv%3A2502%2E10709%2E+%E2%80%94+%5BarXiv%3A2)
55. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.
56. Kevin Yang, Dan Klein, Asli Celikyilmaz, Nanyun Peng, and Yuandong Tian. 2023a. Rlcd: Reinforcement learning from contrastive distillation for language model alignment. arXiv preprint arXiv:2307.12950. [find](https://scholar.google.com/scholar?q=Kevin+Yang%2C+Dan+Klein%2C+Asli+Celikyilmaz%2C+Nanyun+Peng%2C+and+Yuandong+Tian%2E+2023a%2E+Rlcd%3A+Reinforcement+learning+from+contrastive+distillation+for+language+model+alignment%2E+arXiv+preprint+arXiv%3A2307%2E12950)
57. Linyi Yang, Shuibai Zhang, Zhuohao Yu, Guangsheng Bao, Yidong Wang, Jindong Wang, Ruochen Xu, Wei Ye, Xing Xie, Weizhu Chen, and 1 others. 2023b. Supervised knowledge makes large language models better in-context learners. arXiv preprint arXiv:2312.15918. — [arXiv:2312.15918](https://arxiv.org/abs/2312.15918) [find](https://scholar.google.com/scholar?q=Linyi+Yang%2C+Shuibai+Zhang%2C+Zhuohao+Yu%2C+Guangsheng+Bao%2C+Yidong+Wang%2C+Jindong+Wang%2C+Ruochen+Xu%2C+Wei+Ye%2C+Xing+Xie%2C+Weizhu+Chen%2C+and+1+others%2E+2023b%2E+Supervised+knowledge+makes+large+language+models+bette)
58. Shenzhi Wang Zhangchi Feng Dongdong Kuang Yuwen Xiong Yaowei Zheng, Junting Lu. 2025. Easyr1: An efficient, scalable, multi-modality rl training framework. [https://github.com/hiyouga/](https://github.com/hiyouga/) EasyR1. [↗](https://github.com/hiyouga/)
59. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, and 1 others. 2025a. [Dapo: An open-source llm reinforcement learning system at scale](https://md.research.elamur.ai/free/en/papers/yu-2025-dapo-open-source-llm-reinforcement/). arXiv preprint arXiv:2503.14476. 2023. Judging llm-as-a-judge with mt-bench and chatbot are…
60. Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. [Large language models are human-level prompt engineers](https://md.research.elamur.ai/free/en/papers/zhou-2022-large-language-models-human-level/). In The eleventh international conference on learning representations.
61. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593. [find](https://scholar.google.com/scholar?q=Daniel+M+Ziegler%2C+Nisan+Stiennon%2C+Jeffrey+Wu%2C+Tom+B+Brown%2C+Alec+Radford%2C+Dario+Amodei%2C+Paul+Christiano%2C+and+Geoffrey+Irving%2E+2019%2E+Fine%2Dtuning+language+models+from+human+preferences%2E+arXiv+preprint+ar)
62. Zhuohao Yu, Chang Gao, Wenjin Yao, Yidong Wang, Wei Ye, Jindong Wang, Xing Xie, Yue Zhang, and Shikun Zhang. 2024a. Kieval: A knowledge-grounded interactive evaluation framework for large language models. arXiv preprint arXiv:2402.15043. [find](https://scholar.google.com/scholar?q=Zhuohao+Yu%2C+Chang+Gao%2C+Wenjin+Yao%2C+Yidong+Wang%2C+Wei+Ye%2C+Jindong+Wang%2C+Xing+Xie%2C+Yue+Zhang%2C+and+Shikun+Zhang%2E+2024a%2E+Kieval%3A+A+knowledge%2Dgrounded+interactive+evaluation+framework+for+large+language+mod)
63. Zhuohao Yu, Chang Gao, Wenjin Yao, Yidong Wang, Zhengran Zeng, Wei Ye, Jindong Wang, Yue Zhang, and Shikun Zhang. 2024b. Freeeval: A modular framework for trustworthy and efficient evaluation of large language models. arXiv preprint arXiv:2404.06003. [find](https://scholar.google.com/scholar?q=Zhuohao+Yu%2C+Chang+Gao%2C+Wenjin+Yao%2C+Yidong+Wang%2C+Zhengran+Zeng%2C+Wei+Ye%2C+Jindong+Wang%2C+Yue+Zhang%2C+and+Shikun+Zhang%2E+2024b%2E+Freeeval%3A+A+modular+framework+for+trustworthy+and+efficient+evaluation+of+large)
64. Zhuohao Yu, Weizheng Gu, Yidong Wang, Xingru Jiang, Zhengran Zeng, Jindong Wang, Wei Ye, and Shikun Zhang. 2024c. Reasoning through execution: Unifying process and outcome rewards for code generation. arXiv preprint arXiv:2412.15118. — [arXiv:2412.15118](https://arxiv.org/abs/2412.15118) [find](https://scholar.google.com/scholar?q=Zhuohao+Yu%2C+Weizheng+Gu%2C+Yidong+Wang%2C+Xingru+Jiang%2C+Zhengran+Zeng%2C+Jindong+Wang%2C+Wei+Ye%2C+and+Shikun+Zhang%2E+2024c%2E+Reasoning+through+execution%3A+Unifying+process+and+outcome+rewards+for+code+generation%2E)
65. Zhuohao Yu, Jiali Zeng, Weizheng Gu, Yidong Wang, Jindong Wang, Fandong Meng, Jie Zhou, Yue Zhang, Shikun Zhang, and Wei Ye. 2025b. Rewardanything: Generalizable principle-following reward models. arXiv preprint arXiv:2506.03637. — [arXiv:2506.03637](https://arxiv.org/abs/2506.03637) [find](https://scholar.google.com/scholar?q=Zhuohao+Yu%2C+Jiali+Zeng%2C+Weizheng+Gu%2C+Yidong+Wang%2C+Jindong+Wang%2C+Fandong+Meng%2C+Jie+Zhou%2C+Yue+Zhang%2C+Shikun+Zhang%2C+and+Wei+Ye%2E+2025b%2E+Rewardanything%3A+Generalizable+principle%2Dfollowing+reward+models%2E+arX)
66. Zhihan Zhang, Shuohang Wang, Wenhao Yu, Yichong Xu, Dan Iter, Qingkai Zeng, Yang Liu, Chenguang Zhu, and Meng Jiang. 2023. Auto-instruct: Automatic instruction generation and ranking for black-box language models. arXiv preprint arXiv:2310.13127. [find](https://scholar.google.com/scholar?q=Zhihan+Zhang%2C+Shuohang+Wang%2C+Wenhao+Yu%2C+Yichong+Xu%2C+Dan+Iter%2C+Qingkai+Zeng%2C+Yang+Liu%2C+Chenguang+Zhu%2C+and+Meng+Jiang%2E+2023%2E+Auto%2Dinstruct%3A+Automatic+instruction+generation+and+ranking+for+black%2Dbox+lan)
67. Jian Zhao, Runze Liu, Kaiyan Zhang, Zhimu Zhou, Junqi Gao, Dong Li, Jiafei Lyu, Zhouyi Qian, Biqing Qi, Xiu Li, and 1 others. 2025. Genprm: Scaling test-time compute of process reward models via generative reasoning. arXiv preprint arXiv:2504.00891. — [arXiv:2504.00891](https://arxiv.org/abs/2504.00891) [find](https://scholar.google.com/scholar?q=Jian+Zhao%2C+Runze+Liu%2C+Kaiyan+Zhang%2C+Zhimu+Zhou%2C+Junqi+Gao%2C+Dong+Li%2C+Jiafei+Lyu%2C+Zhouyi+Qian%2C+Biqing+Qi%2C+Xiu+Li%2C+and+1+others%2E+2025%2E+Genprm%3A+Scaling+test%2Dtime+compute+of+process+reward+models+via+gener)
68. Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, and 1 others. 2025. [Group sequence policy optimization](https://md.research.elamur.ai/free/en/papers/zheng-2025-group-sequence-policy-optimization/). arXiv preprint arXiv:2507.18071.

*Updated 2 November 2026*