---
title: "Selective Preference Optimization via Token-Level Reward"
description: "The authors propose Selective Preference Optimization (SePO) to address this. 52 references, cited by 8 papers here."
authors: ["Kailai Yang", "Zhiwei Liu", "Qianqian Xie", "Jimin Huang", "Erxue Min", "Sophia Ananiadou"]
year: 2025
venue: "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2025.emnlp-main.359/"
doi: "10.18653/v1/2025.emnlp-main.359"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yang-2025-selective-preference-optimization-token-level/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yang-2025-selective-preference-optimization-token-level/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/10/) / Selective Preference Optimization via Token-Level Reward Function Estimation

# Selective Preference Optimization via Token-Level Reward Function Estimation

**Authors:** [Kailai Yang](https://md.research.elamur.ai/free/en/authors/kailai-yang/), [Zhiwei Liu](https://md.research.elamur.ai/free/en/authors/zhiwei-liu/), [Qianqian Xie](https://md.research.elamur.ai/free/en/authors/qianqian-xie/), [Jimin Huang](https://md.research.elamur.ai/free/en/authors/jimin-huang/), [Erxue Min](https://md.research.elamur.ai/free/en/authors/erxue-min/) and [Sophia Ananiadou](https://md.research.elamur.ai/free/en/authors/sophia-ananiadou/) · **Year:** 2025 · **Venue:** [Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2025](https://md.research.elamur.ai/free/en/venues/emnlp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.emnlp-main.359/) · [PDF](https://aclanthology.org/2025.emnlp-main.359.pdf) · [DOI 10.18653/v1/2025.emnlp-main.359](https://doi.org/10.18653/v1/2025.emnlp-main.359) · [OpenAlex](https://openalex.org/W4416036594)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [8](https://md.research.elamur.ai/free/en/papers/yang-2025-selective-preference-optimization-token-level/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Token-level preference alignment for LLMs usually trains on every token, which adds noise and wastes compute. Methods that do pick out important tokens tend to rely on costly tools such as Monte-Carlo Tree Search or human/LLM annotation. The authors propose Selective Preference Optimization (SePO) to address this. They show theoretically that a model trained with Direct Preference Optimization (DPO) implicitly learns a token-level reward. This means a small "oracle" model, trained with DPO on a moderate subset of any ordinary response-level preference dataset, can score every token in the full dataset. SePO keeps the highest-scoring tokens from chosen responses and the lowest-scoring tokens from rejected ones. It then trains the target policy only on those tokens, using a contrastive objective that needs no reference model. Across three public benchmarks and six policy models, SePO beats strong baselines while training on roughly 30% of tokens. The approach also works as a weak-to-strong scheme, where small oracles guide much larger policies, and it makes weaker out-of-distribution data usable without over-optimization.

## Key points
- A GPT-4 token-reward analysis of 1,000 instruction-following samples motivates the method: in chosen responses, the top 35.6% of tokens hold 80% of the total reward.
- SePO trains on about 30% of key tokens and cuts GPU training hours by up to 60% compared with full-token baselines (DPO, IPO, RRHF, SimPO).
- Weak oracle models can effectively guide policy models whose parameter counts are as much as 16.8× larger.
- On the lower-quality HH-RLHF data, a TinyLLaMA-based oracle lets SePO improve LLaMA2-Chat-13B:
  - Arena-Hard rises from 12.0% to 13.63%.
  - AlpacaEval 2.0 LC win rate rises from 8.4% to 8.81%.
  - In the same setting, the baselines all fall below the base model.
- Growing the oracle from 1B to 12B raises AlpacaEval 2.0 LC by about 3 points (Pythia-2.8B: 7.1%→10.1%; Pythia-6.9B: 14.27%→17.14%), but the gains shrink at larger oracle sizes.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Baidu](https://md.research.elamur.ai/free/en/companies/baidu/) — Company, Beijing, China · 74 works · 15,263 citations
- [University of Manchester](https://md.research.elamur.ai/free/en/universities/university-of-manchester/) — University, Manchester, United Kingdom · 87 works · 44,938 citations
- [Wuhan University](https://md.research.elamur.ai/free/en/universities/wuhan-university/) — University, Wuhan, China · 35 works · 1,127 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

## Cite

Cite as: Kailai Yang, Zhiwei Liu, Qianqian Xie, Jimin Huang, Erxue Min and Sophia Ananiadou. 2025. Selective Preference Optimization via Token-Level Reward Function Estimation. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yang2025selective,
  title = {{Selective Preference Optimization via Token-Level Reward Function Estimation}},
  author = {Kailai Yang, Zhiwei Liu, Qianqian Xie, Jimin Huang, Erxue Min and Sophia Ananiadou},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.359},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.359}
}
```

- [← previous SEED: Accelerating Reasoning Tree Construction via Scheduled Speculative Decoding](https://md.research.elamur.ai/free/en/papers/wang-2025-seed-accelerating-reasoning-tree-construction/)
- [next → Superficial Self-Improved Reasoners Benefit from Model Merging](https://md.research.elamur.ai/free/en/papers/yuan-2025-superficial-self-improved-reasoners-benefit/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/10/)

## References

52 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
2. Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. 2024. [A general theoretical paradigm to understand learning from human preferences](https://md.research.elamur.ai/free/en/papers/azar-2023-general-theoretical-paradigm-understand-learning/). In International Conference on Artificial Intelligence and Statistics, pages 4447–4455. PMLR…
3. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+et+al%2E+2022%2E+Training+a+helpful+and+harmless+assistant+with+rein)
4. Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. In International Conferenc… [find](https://scholar.google.com/scholar?q=Stella+Biderman%2C+Hailey+Schoelkopf%2C+Quentin+Gregory+Anthony%2C+Herbie+Bradley%2C+Kyle+O%E2%80%99Brien%2C+Eric+Hallahan%2C+Mohammad+Aflah+Khan%2C+Shivanshu+Purohit%2C+USVSN+Sai+Prashanth%2C+Edward+Raff%2C+et+al%2E+2023%2E+Pythia%3A)
5. Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324– 345. [find](https://scholar.google.com/scholar?q=Ralph+Allan+Bradley+and+Milton+E+Terry%2E+1952%2E+Rank+analysis+of+incomplete+block+designs%3A+I%2E+the+method+of+paired+comparisons%2E+Biometrika%2C+39%283%2F4%29%3A324%E2%80%93+345%2E)
6. Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. 2023. [Weak-to-strong generalization: Eliciting strong capabilities with weak supervision](https://md.research.elamur.ai/free/en/papers/burns-2023-weak-strong-generalization-eliciting-strong/). arXiv preprint arXiv:2312.09390.
7. Alex J Chan, Hao Sun, Samuel Holt, and Mihaela van der Schaar. 2024. Dense reward for free in reinforcement learning from human feedback. arXiv preprint arXiv:2402.00782. — [arXiv:2402.00782](https://arxiv.org/abs/2402.00782) [find](https://scholar.google.com/scholar?q=Alex+J+Chan%2C+Hao+Sun%2C+Samuel+Holt%2C+and+Mihaela+van+der+Schaar%2E+2024%2E+Dense+reward+for+free+in+reinforcement+learning+from+human+feedback%2E+arXiv+preprint+arXiv%3A2402%2E00782%2E+%E2%80%94+%5BarXiv%3A2402%2E00782%5D%28https%3A%2F%2F)
8. Moses Charikar, Chirag Pabbaraju, and Kirankumar Shiragur. 2024. Quantifying the gain in weak-to-strong generalization. arXiv preprint arXiv:2405.15116. — [arXiv:2405.15116](https://arxiv.org/abs/2405.15116) [find](https://scholar.google.com/scholar?q=Moses+Charikar%2C+Chirag+Pabbaraju%2C+and+Kirankumar+Shiragur%2E+2024%2E+Quantifying+the+gain+in+weak%2Dto%2Dstrong+generalization%2E+arXiv+preprint+arXiv%3A2405%2E15116%2E+%E2%80%94+%5BarXiv%3A2405%2E15116%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2405)
9. Changyu Chen, Zichen Liu, Chao Du, Tianyu Pang, Qian Liu, Arunesh Sinha, Pradeep Varakantham, and Min Lin. 2024a. Bootstrapping language models with dpo implicit rewards. arXiv preprint arXiv:2406.09760. — [arXiv:2406.09760](https://arxiv.org/abs/2406.09760) [find](https://scholar.google.com/scholar?q=Changyu+Chen%2C+Zichen+Liu%2C+Chao+Du%2C+Tianyu+Pang%2C+Qian+Liu%2C+Arunesh+Sinha%2C+Pradeep+Varakantham%2C+and+Min+Lin%2E+2024a%2E+Bootstrapping+language+models+with+dpo+implicit+rewards%2E+arXiv+preprint+arXiv%3A2406%2E097)
10. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377. [find](https://scholar.google.com/scholar?q=Ganqu+Cui%2C+Lifan+Yuan%2C+Ning+Ding%2C+Guanming+Yao%2C+Wei+Zhu%2C+Yuan+Ni%2C+Guotong+Xie%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2023%2E+Ultrafeedback%3A+Boosting+language+models+with+high%2Dquality+feedback%2E+arXiv+preprint+ar)
11. Tri Dao. 2023. [Flashattention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/). arXiv preprint arXiv:2307.08691.
12. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. [Enhancing chat language models by scaling high-quality instructional conversations](https://md.research.elamur.ai/free/en/papers/ding-2023-enhancing-chat-language-models-scaling/). arXiv preprint arXiv:2305.14233.
13. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
14. Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. 2024. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475. [find](https://scholar.google.com/scholar?q=Yann+Dubois%2C+Bal%C3%A1zs+Galambosi%2C+Percy+Liang%2C+and+Tatsunori+B+Hashimoto%2E+2024%2E+Length%2Dcontrolled+alpacaeval%3A+A+simple+way+to+debias+automatic+evaluators%2E+arXiv+preprint+arXiv%3A2404%2E04475%2E)
15. Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306. [find](https://scholar.google.com/scholar?q=Kawin+Ethayarajh%2C+Winnie+Xu%2C+Niklas+Muennighoff%2C+Dan+Jurafsky%2C+and+Douwe+Kiela%2E+2024%2E+Kto%3A+Model+alignment+as+prospect+theoretic+optimization%2E+arXiv+preprint+arXiv%3A2402%2E01306%2E)
16. Leo Gao, John Schulman, and Jacob Hilton. 2023. [Scaling laws for reward model overoptimization](https://md.research.elamur.ai/free/en/papers/gao-2022-scaling-laws-reward-model-overoptimization/). In International Conference on Machine Learning, pages 10835–10866. PMLR.
17. Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Juntao Dai, and Yaodong Yang. 2024. Aligner: Achieving efficient alignment through weak-to-strong correction. arXiv preprint arXiv:2402.02416. [find](https://scholar.google.com/scholar?q=Jiaming+Ji%2C+Boyuan+Chen%2C+Hantao+Lou%2C+Donghai+Hong%2C+Borong+Zhang%2C+Xuehai+Pan%2C+Juntao+Dai%2C+and+Yaodong+Yang%2E+2024%2E+Aligner%3A+Achieving+efficient+alignment+through+weak%2Dto%2Dstrong+correction%2E+arXiv+preprin)
18. Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. 2023. [Ai alignment: A comprehensive survey](https://md.research.elamur.ai/free/en/papers/ji-2023-ai-alignment-comprehensive-survey/). arXiv preprint arXiv:2310.19852.
19. Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. 2024b. Step-level value preference optimization for mathematical reasoning. arXiv preprint arXiv:2406.10858. [find](https://scholar.google.com/scholar?q=Guoxin+Chen%2C+Minpeng+Liao%2C+Chengxi+Li%2C+and+Kai+Fan%2E+2024b%2E+Step%2Dlevel+value+preference+optimization+for+mathematical+reasoning%2E+arXiv+preprint+arXiv%3A2406%2E10858%2E)
20. Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. 2024. [Step-dpo: Step-wise preference optimization for long-chain reasoning of llms](https://md.research.elamur.ai/free/en/papers/lai-2024-step-dpo-step-wise-preference/). arXiv preprint arXiv:2406.18629.
21. Zhipeng Chen, Kun Zhou, Wayne Xin Zhao, Jingyuan Wang, and Ji-Rong Wen. 2024c. Low-redundant optimization for large language model alignment. arXiv preprint arXiv:2406.12606. — [arXiv:2406.12606](https://arxiv.org/abs/2406.12606) [find](https://scholar.google.com/scholar?q=Zhipeng+Chen%2C+Kun+Zhou%2C+Wayne+Xin+Zhao%2C+Jingyuan+Wang%2C+and+Ji%2DRong+Wen%2E+2024c%2E+Low%2Dredundant+optimization+for+large+language+model+alignment%2E+arXiv+preprint+arXiv%3A2406%2E12606%2E+%E2%80%94+%5BarXiv%3A2406%2E12606%5D%28http)
22. Hunter Lang, David Sontag, and Aravindan Vijayaraghavan. 2024. Theoretical analysis of weak-to-strong generalization. arXiv preprint arXiv:2405.16043. — [arXiv:2405.16043](https://arxiv.org/abs/2405.16043) [find](https://scholar.google.com/scholar?q=Hunter+Lang%2C+David+Sontag%2C+and+Aravindan+Vijayaraghavan%2E+2024%2E+Theoretical+analysis+of+weak%2Dto%2Dstrong+generalization%2E+arXiv+preprint+arXiv%3A2405%2E16043%2E+%E2%80%94+%5BarXiv%3A2405%2E16043%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2405%2E1)
23. Fenia Christopoulou, Ronald Cardenas, Gerasimos Lampouras, Haitham Bou-Ammar, and Jun Wang. 2024. Sparsepo: Controlling preference alignment of llms via sparse token masks. arXiv preprint arXiv:2410.05102. [find](https://scholar.google.com/scholar?q=Fenia+Christopoulou%2C+Ronald+Cardenas%2C+Gerasimos+Lampouras%2C+Haitham+Bou%2DAmmar%2C+and+Jun+Wang%2E+2024%2E+Sparsepo%3A+Controlling+preference+alignment+of+llms+via+sparse+token+masks%2E+arXiv+preprint+arXiv%3A2410%2E0)
24. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. 2024. [From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline](https://md.research.elamur.ai/free/en/papers/li-2024-crowdsourced-data-high-quality-benchmarks/). Preprint, arXiv:2406.11939.
25. Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, et al. 2024. Rho-1: Not all tokens are what you need. arXiv preprint arXiv:2404.07965. [find](https://scholar.google.com/scholar?q=Zhenghao+Lin%2C+Zhibin+Gou%2C+Yeyun+Gong%2C+Xiao+Liu%2C+Yelong+Shen%2C+Ruochen+Xu%2C+Chen+Lin%2C+Yujiu+Yang%2C+Jian+Jiao%2C+Nan+Duan%2C+et+al%2E+2024%2E+Rho%2D1%3A+Not+all+tokens+are+what+you+need%2E+arXiv+preprint+arXiv%3A2404%2E0796)
26. Junru Lu, Jiazheng Li, Siyu An, Meng Zhao, Yulan He, Di Yin, and Xing Sun. 2024. Eliminating biased length reliance of direct preference optimization via down-sampled kl divergence. arXiv preprint arXiv:2406.10957. [find](https://scholar.google.com/scholar?q=Junru+Lu%2C+Jiazheng+Li%2C+Siyu+An%2C+Meng+Zhao%2C+Yulan+He%2C+Di+Yin%2C+and+Xing+Sun%2E+2024%2E+Eliminating+biased+length+reliance+of+direct+preference+optimization+via+down%2Dsampled+kl+divergence%2E+arXiv+preprint+arX)
27. Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734. [find](https://scholar.google.com/scholar?q=Yu+Meng%2C+Mengzhou+Xia%2C+and+Danqi+Chen%2E+2024%2E+Simpo%3A+Simple+preference+optimization+with+a+reference%2Dfree+reward%2E+arXiv+preprint+arXiv%3A2405%2E14734%2E)
28. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in neural information processing systems, 35:27730–27744.
29. Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikchi, Joey Hejna, Bradley Knox, Chelsea Finn, and Scott Niekum. 2024a. Scaling laws for reward model overoptimization in direct alignment algorithms. arXiv preprint arXiv:2406.02900. — [arXiv:2406.02900](https://arxiv.org/abs/2406.02900) [find](https://scholar.google.com/scholar?q=Rafael+Rafailov%2C+Yaswanth+Chittepu%2C+Ryan+Park%2C+Harshit+Sikchi%2C+Joey+Hejna%2C+Bradley+Knox%2C+Chelsea+Finn%2C+and+Scott+Niekum%2E+2024a%2E+Scaling+laws+for+reward+model+overoptimization+in+direct+alignment+algor)
30. Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. 2024b. From r to q ∗ : Your language model is secretly a q-function. arXiv preprint arXiv:2404.12358. [find](https://scholar.google.com/scholar?q=Rafael+Rafailov%2C+Joey+Hejna%2C+Ryan+Park%2C+and+Chelsea+Finn%2E+2024b%2E+From+r+to+q+%E2%88%97+%3A+Your+language+model+is+secretly+a+q%2Dfunction%2E+arXiv+preprint+arXiv%3A2404%2E12358%2E)
31. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024c. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). Advances in Neural Information Processing Systems, 36.
32. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+2017%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2E)
33. Amrith Setlur, Saurabh Garg, Xinyang Geng, Naman Garg, Virginia Smith, and Aviral Kumar. 2024. Rl on incorrect synthetic data scales the efficiency of llm math reasoning by eight-fold. arXiv preprint arXiv:2406.14532. [find](https://scholar.google.com/scholar?q=Amrith+Setlur%2C+Saurabh+Garg%2C+Xinyang+Geng%2C+Naman+Garg%2C+Virginia+Smith%2C+and+Aviral+Kumar%2E+2024%2E+Rl+on+incorrect+synthetic+data+scales+the+efficiency+of+llm+math+reasoning+by+eight%2Dfold%2E+arXiv+preprint+)
34. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008– 3021. [find](https://scholar.google.com/scholar?q=Nisan+Stiennon%2C+Long+Ouyang%2C+Jeffrey+Wu%2C+Daniel+Ziegler%2C+Ryan+Lowe%2C+Chelsea+Voss%2C+Alec+Radford%2C+Dario+Amodei%2C+and+Paul+F+Christiano%2E+2020%2E+Learning+to+summarize+with+human+feedback%2E+Advances+in+Neural)
35. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288.
36. Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. 2024. [Fine-grained human feedback gives better rewards for language model training](https://md.research.elamur.ai/free/en/papers/wu-2023-fine-grained-human-feedback-gives/). Advances in Neural Information Processing Systems, 36.
37. Han Xia, Songyang Gao, Qiming Ge, Zhiheng Xi, Qi Zhang, and Xuanjing Huang. 2024. Inverse-q*: Token level reinforcement learning for aligning large language models without preference data. arXiv preprint arXiv:2408.14874. [find](https://scholar.google.com/scholar?q=Han+Xia%2C+Songyang+Gao%2C+Qiming+Ge%2C+Zhiheng+Xi%2C+Qi+Zhang%2C+and+Xuanjing+Huang%2E+2024%2E+Inverse%2Dq%2A%3A+Token+level+reinforcement+learning+for+aligning+large+language+models+without+preference+data%2E+arXiv+prepr)
38. Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shieh. 2024. Monte carlo tree search boosts reasoning via iterative preference learning. arXiv preprint arXiv:2405.00451. [find](https://scholar.google.com/scholar?q=Yuxi+Xie%2C+Anirudh+Goyal%2C+Wenyue+Zheng%2C+Min%2DYen+Kan%2C+Timothy+P+Lillicrap%2C+Kenji+Kawaguchi%2C+and+Michael+Shieh%2E+2024%2E+Monte+carlo+tree+search+boosts+reasoning+via+iterative+preference+learning%2E+arXiv+pre)
39. Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. 2024. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In Forty-first International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Wei+Xiong%2C+Hanze+Dong%2C+Chenlu+Ye%2C+Ziqi+Wang%2C+Han+Zhong%2C+Heng+Ji%2C+Nan+Jiang%2C+and+Tong+Zhang%2E+2024%2E+Iterative+preference+learning+from+human+feedback%3A+Bridging+theory+and+practice+for+rlhf+under+kl%2Dcons)
40. Kailai Yang, Zhiwei Liu, Qianqian Xie, Jimin Huang, Tianlin Zhang, and Sophia Ananiadou. 2024a. Metaaligner: Towards generalizable multi-objective alignment of language models. arXiv preprint arXiv:2403.17141. [find](https://scholar.google.com/scholar?q=Kailai+Yang%2C+Zhiwei+Liu%2C+Qianqian+Xie%2C+Jimin+Huang%2C+Tianlin+Zhang%2C+and+Sophia+Ananiadou%2E+2024a%2E+Metaaligner%3A+Towards+generalizable+multi%2Dobjective+alignment+of+language+models%2E+arXiv+preprint+arXiv%3A24)
41. Wenkai Yang, Shiqi Shen, Guangyao Shen, Zhi Gong, and Yankai Lin. 2024b. Super (ficial)-alignment: Strong models may deceive weak models in weak-to-strong generalization. arXiv preprint arXiv:2406.11431. — [arXiv:2406.11431](https://arxiv.org/abs/2406.11431) [find](https://scholar.google.com/scholar?q=Wenkai+Yang%2C+Shiqi+Shen%2C+Guangyao+Shen%2C+Zhi+Gong%2C+and+Yankai+Lin%2E+2024b%2E+Super+%28ficial%29%2Dalignment%3A+Strong+models+may+deceive+weak+models+in+weak%2Dto%2Dstrong+generalization%2E+arXiv+preprint+arXiv%3A2406%2E114)
42. Eunseop Yoon, Hee Suk Yoon, SooHwan Eom, Gunsoo Han, Daniel Wontae Nam, Daejin Jo, Kyoung-Woon On, Mark A. Hasegawa-Johnson, Sungwoong Kim, and Chang D. Yoo. 2024. Tlcr: Token-level continuous reward for fine-grained reinforcement learning from human feedback. Preprint, arXiv:2407.16574. [find](https://scholar.google.com/scholar?q=Eunseop+Yoon%2C+Hee+Suk+Yoon%2C+SooHwan+Eom%2C+Gunsoo+Han%2C+Daniel+Wontae+Nam%2C+Daejin+Jo%2C+Kyoung%2DWoon+On%2C+Mark+A%2E+Hasegawa%2DJohnson%2C+Sungwoong+Kim%2C+and+Chang+D%2E+Yoo%2E+2024%2E+Tlcr%3A+Token%2Dlevel+continuous+reward+)
43. Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. 2024. Rrhf: Rank responses to align language models with human feedback. Advances in Neural Information Processing Systems, 36. [find](https://scholar.google.com/scholar?q=Hongyi+Yuan%2C+Zheng+Yuan%2C+Chuanqi+Tan%2C+Wei+Wang%2C+Songfang+Huang%2C+and+Fei+Huang%2E+2024%2E+Rrhf%3A+Rank+responses+to+align+language+models+with+human+feedback%2E+Advances+in+Neural+Information+Processing+System)
44. Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. 2023. [Rrhf: Rank responses to align language models with human feedback without tears](https://md.research.elamur.ai/free/en/papers/yuan-2023-rrhf-rank-responses-align-language/). arXiv preprint arXiv:2304.05302.
45. Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. 2024. Token-level direct preference optimization. arXiv preprint arXiv:2404.11999. — [arXiv:2404.11999](https://arxiv.org/abs/2404.11999) [find](https://scholar.google.com/scholar?q=Yongcheng+Zeng%2C+Guoqing+Liu%2C+Weiyu+Ma%2C+Ning+Yang%2C+Haifeng+Zhang%2C+and+Jun+Wang%2E+2024%2E+Token%2Dlevel+direct+preference+optimization%2E+arXiv+preprint+arXiv%3A2404%2E11999%2E+%E2%80%94+%5BarXiv%3A2404%2E11999%5D%28https%3A%2F%2Farxiv%2Eorg)
46. Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. [Tinyllama: An open-source small language model](https://md.research.elamur.ai/free/en/papers/zhang-2024-tinyllama-open-source-small-language/). arXiv preprint arXiv:2401.02385.
47. Chujie Zheng, Ziqi Wang, Heng Ji, Minlie Huang, and Nanyun Peng. 2024a. Weak-to-strong extrapolation expedites alignment. arXiv preprint arXiv:2404.16792. — [arXiv:2404.16792](https://arxiv.org/abs/2404.16792) [find](https://scholar.google.com/scholar?q=Chujie+Zheng%2C+Ziqi+Wang%2C+Heng+Ji%2C+Minlie+Huang%2C+and+Nanyun+Peng%2E+2024a%2E+Weak%2Dto%2Dstrong+extrapolation+expedites+alignment%2E+arXiv+preprint+arXiv%3A2404%2E16792%2E+%E2%80%94+%5BarXiv%3A2404%2E16792%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F24)
48. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024b. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Advances in Neural Information Processing Systems, 36.
49. Han Zhong, Guhao Feng, Wei Xiong, Li Zhao, Di He, Jiang Bian, and Liwei Wang. 2024. Dpo meets ppo: Reinforced token optimization for rlhf. arXiv preprint arXiv:2404.18922. — [arXiv:2404.18922](https://arxiv.org/abs/2404.18922) [find](https://scholar.google.com/scholar?q=Han+Zhong%2C+Guhao+Feng%2C+Wei+Xiong%2C+Li+Zhao%2C+Di+He%2C+Jiang+Bian%2C+and+Liwei+Wang%2E+2024%2E+Dpo+meets+ppo%3A+Reinforced+token+optimization+for+rlhf%2E+arXiv+preprint+arXiv%3A2404%2E18922%2E+%E2%80%94+%5BarXiv%3A2404%2E18922%5D%28https%3A%2F)
50. Wenxuan Zhou, Shujian Zhang, Lingxiao Zhao, and Tao Meng. 2024a. T-reg: Preference optimization with token-level reward regularization. arXiv preprint arXiv:2412.02685. [find](https://scholar.google.com/scholar?q=Wenxuan+Zhou%2C+Shujian+Zhang%2C+Lingxiao+Zhao%2C+and+Tao+Meng%2E+2024a%2E+T%2Dreg%3A+Preference+optimization+with+token%2Dlevel+reward+regularization%2E+arXiv+preprint+arXiv%3A2412%2E02685%2E)
51. Zhanhui Zhou, Zhixuan Liu, Jie Liu, Zhichen Dong, Chao Yang, and Yu Qiao. 2024b. [Weak-to-strong search: Align large language models via searching over small language models](https://md.research.elamur.ai/free/en/papers/zhou-2024-weak-strong-search-align-large/). arXiv preprint arXiv:2405.19262.
52. Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. 2008. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433–1438. Chicago, IL, USA. [find](https://scholar.google.com/scholar?q=Brian+D+Ziebart%2C+Andrew+L+Maas%2C+J+Andrew+Bagnell%2C+Anind+K+Dey%2C+et+al%2E+2008%2E+Maximum+entropy+inverse+reinforcement+learning%2E+In+Aaai%2C+volume+8%2C+pages+1433%E2%80%931438%2E+Chicago%2C+IL%2C+USA%2E)

*Updated 30 October 2026*