---
title: "Anchored Preference Optimization and Contrastive Revisions:"
description: "The authors argue that preference-based alignment of LLMs is underspecified in two ways. 59 references, cited by 15 papers here."
authors: ["Karel D'Oosterlinck", "Winnie Xu", "Chris Develder", "Thomas Demeester", "Amanpreet Singh", "Christopher Potts", "Douwe Kiela", "Shikib Mehri"]
year: 2025
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/2025.tacl-1.22/"
doi: "10.1162/tacl_a_00748"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/oosterlinck-2025-anchored-preference-optimization-contrastive-revisions/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/oosterlinck-2025-anchored-preference-optimization-contrastive-revisions/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/4/) / Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment

# Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment

**Authors:** [Karel D'Oosterlinck](https://md.research.elamur.ai/free/en/authors/karel-d-oosterlinck/), Winnie Xu, [Chris Develder](https://md.research.elamur.ai/free/en/authors/chris-develder/), [Thomas Demeester](https://md.research.elamur.ai/free/en/authors/thomas-demeester/), [Amanpreet Singh](https://md.research.elamur.ai/free/en/authors/amanpreet-singh/), [Christopher Potts](https://md.research.elamur.ai/free/en/authors/christopher-e-potts/), [Douwe Kiela](https://md.research.elamur.ai/free/en/authors/douwe-kiela/) and [Shikib Mehri](https://md.research.elamur.ai/free/en/authors/shikib-mehri/) · **Year:** 2025 · **Venue:** [Transactions of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/tacl/) ([Transactions of the Association for Computational Linguistics (TACL) 2025](https://md.research.elamur.ai/free/en/venues/tacl/2025/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.tacl-1.22/) · [PDF](https://aclanthology.org/2025.tacl-1.22.pdf) · [DOI 10.1162/tacl\_a\_00748](https://doi.org/10.1162/tacl_a_00748) · [OpenAlex](https://openalex.org/W4410151876)

**Citations:** 5 citations (OpenAlex, 2026-10-14) · [15](https://md.research.elamur.ai/free/en/papers/oosterlinck-2025-anchored-preference-optimization-contrastive-revisions/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The authors argue that preference-based alignment of LLMs is underspecified in two ways. First, the winning and losing responses in a preference pair often differ in many irrelevant aspects, which blurs what the model should learn. Second, standard contrastive objectives do not consider whether the target model is already better or worse than the data. They address both issues. CLAIR (Contrastive Learning from AI Revisions) builds pairs by having a stronger model minimally revise the target model's own answer. APO (Anchored Preference Optimization) is a family of objectives that explicitly control how winning and losing likelihoods should move during training. They aligned Llama-3-8B-Instruct on four comparable 32K-prompt datasets using five objectives. CLAIR data combined with APO produced the strongest model. The analysis attributes most of the gain to how contrastive the CLAIR pairs are. These findings show that how preference pairs are built, and how training is anchored, can matter as much as raw data quality.

## Key points
- **Setup:**
  - Target model: Llama-3-8B-Instruct.
  - GPT4-turbo served as Judge, Reviser and Stronger model.
  - Prompts: 32K sampled uniformly from UltraFeedback.
- **Datasets compared:**
  - CLAIR
  - on-policy judge
  - off-policy judge (existing UltraFeedback judgments)
  - "Stronger Preferred", an ablation that isolates contrastiveness from answer quality
- **Contrastiveness:** CLAIR pairs scored best on both measures, with token-level Jaccard ≈43.1 and character-level Levenshtein ≈1108.
- **Objectives and evaluation:**
  - Objectives: DPO, KTO, continued SFT and two APO variants.
  - APO beat the other objectives on every dataset.
  - Benchmarks: MixEval-Hard and length-controlled AlpacaEval.
- **Main results:**
  - Best model (CLAIR + APO) improved MixEval-Hard by 7.65% and closed 45% of the gap to GPT4-turbo.
  - About 80% of UltraFeedback's winning outputs come from models weaker than Llama-3-8B-Instruct, so naive alignment on that data can hurt.
  - Code and datasets are on GitHub (ContextualAI/CLAIR and APO).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 434 works · 314,419 citations
- [Ghent University](https://md.research.elamur.ai/free/en/universities/ghent-university/) — University, Ghent, Belgium · 25 works · 2,949 citations
- [IMEC](https://md.research.elamur.ai/free/en/ngos/imec/) — Nonprofit, Leuven, Belgium · 3 works · 215 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

## Cite

Cite as: Karel D'Oosterlinck, Winnie Xu, Chris Develder, Thomas Demeester, Amanpreet Singh, Christopher Potts, Douwe Kiela and Shikib Mehri. 2025. Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{doosterlinck2025anchored,
  title = {{Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment}},
  author = {Karel D'Oosterlinck, Winnie Xu, Chris Develder, Thomas Demeester, Amanpreet Singh, Christopher Potts, Douwe Kiela and Shikib Mehri},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.1162/tacl\_a\_00748},
  url = {https://doi.org/10.1162/tacl_a_00748}
}
```

- [← previous AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detection](https://md.research.elamur.ai/free/en/papers/luo-2025-agrail-lifelong-agent-guardrail-effective/)
- [next → Biased LLMs can Influence Political Decision-Making](https://md.research.elamur.ai/free/en/papers/fisher-2025-biased-llms-can-influence-political/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/4/)

## References

59 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv: 2303.08774.
2. Leonard Adolphs, Tianyu Gao, Jing Xu, Kurt Shuster, Sainbayar Sukhbaatar, and Jason Weston. 2023. The CRINGE loss: Learning what language not to model. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8854–8874. https://doi .or… [find](https://scholar.google.com/scholar?q=Leonard+Adolphs%2C+Tianyu+Gao%2C+Jing+Xu%2C+Kurt+Shuster%2C+Sainbayar+Sukhbaatar%2C+and+Jason+Weston%2E+2023%2E+The+CRINGE+loss%3A+Learning+what+language+not+to+model%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the)
3. Afra Amini, Tim Vieira, and Ryan Cotterell. 2024. Direct preference optimization with an offset. arXiv preprint arXiv:2402.10571. https://doi.org/10.18653/v1/2024 .findings-acl.592 [↗](https://doi.org/10.18653/v1/2024)
4. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732.
5. Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
6. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E. Gonzalez, and Ion Stoika. 2024. [Chatbot arena: An open platform for evaluating LLMs by human preference](https://md.research.elamur.ai/free/en/papers/chiang-2024-chatbot-arena-open-platform-evaluating/). In Forty-first International Conference on Mac…
7. Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. 2024. [A general theoretical paradigm to understand learning from human preferences](https://md.research.elamur.ai/free/en/papers/azar-2023-general-theoretical-paradigm-understand-learning/). In International Conference on Artificial Intelligence and Statistics, pages 4447–4455. PMLR…
8. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared… — [Constitutional AI: Harmlessness from AI Feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/)
9. Paul F. Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 4302–4310. [find](https://scholar.google.com/scholar?q=Paul+F%2E+Christiano%2C+Jan+Leike%2C+Tom+Brown%2C+Miljan+Martic%2C+Shane+Legg%2C+and+Dario+Amodei%2E+2017%2E+Deep+reinforcement+learning+from+human+preferences%2E+In+Proceedings+of+the+31st+International+Conference+on+)
10. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. [BoolQ: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:…
11. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? Try ARC, the AI2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv preprint arXiv: 1803.05457.
12. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. [PIQA: Reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7432–7439. https://doi.org/10.1609/aaai.v34i05 .6239 [↗](https://doi.org/10.1609/aaai.v34i05)
13. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
14. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2024. ULTRA-FEEDBACK: Boost… [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D%2E+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Ganqu+Cui%2C+Lifan+Yuan%2C+Ning+Ding%2C+Guanming+Yao%2C+Bingxiang+He%2C+Wei+Zhu%2C+Yuan)
15. Duanyu Feng, Bowen Qin, Chen Huang, Zheng Zhang, and Wenqiang Lei. 2024. Towards analyzing and understanding the limitations of DPO: A theoretical perspective. arXiv preprint arXiv:2404.04626. — [arXiv:2404.04626](https://arxiv.org/abs/2404.04626) [find](https://scholar.google.com/scholar?q=Duanyu+Feng%2C+Bowen+Qin%2C+Chen+Huang%2C+Zheng+Zhang%2C+and+Wenqiang+Lei%2E+2024%2E+Towards+analyzing+and+understanding+the+limitations+of+DPO%3A+A+theoretical+perspective%2E+arXiv+preprint+arXiv%3A2404%2E04626%2E+%E2%80%94+%5BarXi)
16. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. [DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistic…
17. Leo Gao, John Schulman, and Jacob Hilton. 2023. [Scaling laws for reward model overoptimization](https://md.research.elamur.ai/free/en/papers/gao-2022-scaling-laws-reward-model-overoptimization/). In International Conference on Machine Learning, pages 10835–10866. PMLR.
18. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In International Conference on Learning Representations.
19. Shitong Duan, Xiaoyuan Yi, Peng Zhang, Tun Lu, Xing Xie, and Ning Gu. 2024. Negating negatives: Alignment without human positive samples via distributional dispreference optimization. arXiv preprint arXiv: 2403.03419. https://doi.org/10.18653/v1/2024.findings-emnlp.56 [↗](https://doi.org/10.18653/v1/2024.findings-emnlp.56)
20. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. [Measuring mathematical problem solving with the MATH dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
21. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
22. Miguel A. Hernán and James M. Robins. 2006. Estimating causal effects from epidemiological data. Journal of Epidemiology & Community Health, 60(7):578–586. https://doi.org/10.1136/jech.2004.029496, PubMed: 16790829 — [doi:10.1136/jech.2004.029496](https://doi.org/10.1136/jech.2004.029496) [↗](https://doi.org/10.1136/jech.2004.029496)
23. Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. 2024. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv: 2404.04475. [find](https://scholar.google.com/scholar?q=Yann+Dubois%2C+Bala%CC%81zs+Galambosi%2C+Percy+Liang%2C+and+Tatsunori+B%2E+Hashimoto%2E+2024%2E+Length%2Dcontrolled+alpacaeval%3A+A+simple+way+to+debias+automatic+evaluators%2E+arXiv+preprint+arXiv%3A+2404%2E04475%2E)
24. Jiwoo Hong, Noah Lee, and James Thorne. 2024. Reference-free monolithic preference optimization with odds ratio. arXiv preprint arXiv:2403.07691. https://doi.org/10 .18653/v1/2024.emnlp-main.626 [find](https://scholar.google.com/scholar?q=Jiwoo+Hong%2C+Noah+Lee%2C+and+James+Thorne%2E+2024%2E+Reference%2Dfree+monolithic+preference+optimization+with+odds+ratio%2E+arXiv+preprint+arXiv%3A2403%2E07691%2E+https%3A%2F%2Fdoi%2Eorg%2F10+%2E18653%2Fv1%2F2024%2Eemnlp%2Dmain%2E626)
25. Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D’Amour, D. J. Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, Peter Shaw, and Jonathan Berant. 2023. Helping or herding? Reward model ensembles mitigate but do not eliminate reward hacking. arXiv prepri… — [arXiv:2312.09244](https://arxiv.org/abs/2312.09244) [find](https://scholar.google.com/scholar?q=Jacob+Eisenstein%2C+Chirag+Nagpal%2C+Alekh+Agarwal%2C+Ahmad+Beirami%2C+Alex+D%E2%80%99Amour%2C+D%2E+J%2E+Dvijotham%2C+Adam+Fisch%2C+Katherine+Heller%2C+Stephen+Pfohl%2C+Deepak+Ramachandran%2C+Peter+Shaw%2C+and+Jonathan+Berant%2E+2023%2E+H)
26. Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023. [LLM-Blender: Ensembling large language models with pairwise ranking and generative fusion](https://md.research.elamur.ai/free/en/papers/jiang-2023-llm-blender-ensembling-large-language/). In The 61st Annual Meeting of the Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.acl-long.792 [↗](https://doi.org/10.18653/v1/2023.acl-long.792)
27. Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. 2022. Understanding dataset difficulty with V-usable information. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 5988–6008. PMLR. [find](https://scholar.google.com/scholar?q=Kawin+Ethayarajh%2C+Yejin+Choi%2C+and+Swabha+Swayamdipta%2E+2022%2E+Understanding+dataset+difficulty+with+V%2Dusable+information%2E+In+Proceedings+of+the+39th+International+Conference+on+Machine+Learning%2C+volume+)
28. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. [TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). In Proceedings of the 55th Annual Meeting of the Association for Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe K…
29. Computational Linguistics (Volume 1: Long Papers), pages 1601–1611. https://doi.org/10.18653/v1/P17-1147 structions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744. — [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/) [↗](https://doi.org/10.18653/v1/P17-1147)
30. Seungjae Jung, Gunsoo Han, Daniel Wontae Nam, and Kyoung-Woon On. 2024. [Binary classifier optimization for large language model alignment](https://md.research.elamur.ai/free/en/papers/jung-2025-binary-classifier-optimization-large-language/). arXiv preprint arXiv:2404.04656.
31. Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. 2024. Smaug: Fixing failure modes of preference optimisation with DPO-positive. arXiv preprint arXiv:2402.13228. [find](https://scholar.google.com/scholar?q=Arka+Pal%2C+Deep+Karkhanis%2C+Samuel+Dooley%2C+Manley+Roberts%2C+Siddartha+Naidu%2C+and+Colin+White%2E+2024%2E+Smaug%3A+Fixing+failure+modes+of+preference+optimisation+with+DPO%2Dpositive%2E+arXiv+preprint+arXiv%3A2402%2E132)
32. Dahyun Kim, Yungi Kim, Wonho Song, Hyeonwoo Kim, Yunsu Kim, Sanghoon Kim, and Chanjun Park. 2024. [sDPO: Don’t use your data all at once](https://md.research.elamur.ai/free/en/papers/kim-2024-sdpo-don-t-use-data/). arXiv preprint arXiv: 2403.19270.
33. Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. 2024. [Disentangling length from quality in direct preference optimization](https://md.research.elamur.ai/free/en/papers/park-2024-disentangling-length-quality-direct-preference/). arXiv preprint arXiv:2403.19159. https://doi.org/10.18653/v1/2024.findings -acl.297 [↗](https://doi.org/10.18653/v1/2024.findings)
34. Nathan Lambert, Hailey Schoelkopf, Aaron Gokaslan, Luca Soldaini, Valentina Pyatkin, and Louis Castricato. 2024. Self-directed synthetic dialogues and revisions technical report. arXiv preprint arXiv:2407.18421. — [arXiv:2407.18421](https://arxiv.org/abs/2407.18421) [find](https://scholar.google.com/scholar?q=Nathan+Lambert%2C+Hailey+Schoelkopf%2C+Aaron+Gokaslan%2C+Luca+Soldaini%2C+Valentina+Pyatkin%2C+and+Louis+Castricato%2E+2024%2E+Self%2Ddirected+synthetic+dialogues+and+revisions+technical+report%2E+arXiv+preprint+arXiv%3A)
35. Shiva Kumar Pentyala, Zhichao Wang, Bin Bi, Kiran Ramnath, Xiang-Bo Mao, Regunathan Radhakrishnan, Sitaram Asur, and Na (Claire) Cheng. 2024. PAFT: A parallel training paradigm for effective LLM fine-tuning. arXiv preprint arXiv:2406.17923. — [arXiv:2406.17923](https://arxiv.org/abs/2406.17923) [find](https://scholar.google.com/scholar?q=Shiva+Kumar+Pentyala%2C+Zhichao+Wang%2C+Bin+Bi%2C+Kiran+Ramnath%2C+Xiang%2DBo+Mao%2C+Regunathan+Radhakrishnan%2C+Sitaram+Asur%2C+and+Na+%28Claire%29+Cheng%2E+2024%2E+PAFT%3A+A+parallel+training+paradigm+for+effective+LLM+fine%2D)
36. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. AlpacaEval: An automatic evaluator of instruction-following models. [https://github.com/tatsu-lab/alpaca](https://github.com/tatsu-lab/alpaca) eval Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit S… — [arXiv:2406.02900](https://arxiv.org/abs/2406.02900) [↗](https://github.com/tatsu-lab/alpaca)
37. Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. SimPO: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734. [find](https://scholar.google.com/scholar?q=Yu+Meng%2C+Mengzhou+Xia%2C+and+Danqi+Chen%2E+2024%2E+SimPO%3A+Simple+preference+optimization+with+a+reference%2Dfree+reward%2E+arXiv+preprint+arXiv%3A2405%2E14734%2E)
38. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391. https://doi.org/10.18653/v1/D18 -… [↗](https://doi.org/10.18653/v1/D18)
39. Jinjie Ni, Fuzhao Xue, Xiang Yue, Yuntian Deng, Mahir Shah, Kabir Jain, Graham Neubig, and Yang You. 2024. [MixEval: Deriving wisdom of the crowd from LLM benchmark mixtures](https://md.research.elamur.ai/free/en/papers/ni-2024-mixeval-deriving-wisdom-crowd-llm/). arXiv preprint arXiv:2406.06565.
40. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. 2024b. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). Advances in Neural Information Processing Systems, 36.
41. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2023. [GPQA: A graduate-level google-proof q&a benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/). arXiv preprint arXiv:2311.12022.
42. Pierre Harvey Richemond, Yunhao Tang, Daniel Guo, Daniele Calandriello, Mohammad Gheshlaghi Azar, Rafael Rafailov, Bernardo Avila Pires, Eugene Tarassov, Lucas Spangher, Will Ellsworth, et al. 2024. Offline regularised reinforcement learning for large language models alignment. arXiv preprint arXiv:… [find](https://scholar.google.com/scholar?q=Pierre+Harvey+Richemond%2C+Yunhao+Tang%2C+Daniel+Guo%2C+Daniele+Calandriello%2C+Mohammad+Gheshlaghi+Azar%2C+Rafael+Rafailov%2C+Bernardo+Avila+Pires%2C+Eugene+Tarassov%2C+Lucas+Spangher%2C+Will+Ellsworth%2C+et+al%2E+2024%2E+O)
43. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Train… [find](https://scholar.google.com/scholar?q=Long+Ouyang%2C+Jeffrey+Wu%2C+Xu+Jiang%2C+Diogo+Almeida%2C+Carroll+Wainwright%2C+Pamela+Mishkin%2C+Chong+Zhang%2C+Sandhini+Agarwal%2C+Katarina+Slama%2C+Alex+Ray%2C+John+Schulman%2C+Jacob+Hilton%2C+Fraser+Kelton%2C+Luke+Miller%2C+)
44. Mehrotra, Xiang-Bo Mao, Sitaram Asur, and Na (Claire) Cheng. 2024. A comprehensive survey of LLM alignment techniques: RLHF, RLAIF, PPO, DPO and more. arXiv preprint arXiv:2407.16216. — [arXiv:2407.16216](https://arxiv.org/abs/2407.16216) [find](https://scholar.google.com/scholar?q=Mehrotra%2C+Xiang%2DBo+Mao%2C+Sitaram+Asur%2C+and+Na+%28Claire%29+Cheng%2E+2024%2E+A+comprehensive+survey+of+LLM+alignment+techniques%3A+RLHF%2C+RLAIF%2C+PPO%2C+DPO+and+more%2E+arXiv+preprint+arXiv%3A2407%2E16216%2E+%E2%80%94+%5BarXiv%3A2407%2E16)
45. Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto. 2023. Verbosity bias in preference labeling by large language models. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following. [find](https://scholar.google.com/scholar?q=Keita+Saito%2C+Akifumi+Wachi%2C+Koki+Wataoka%2C+and+Youhei+Akimoto%2E+2023%2E+Verbosity+bias+in+preference+labeling+by+large+language+models%2E+In+NeurIPS+2023+Workshop+on+Instruction+Tuning+and+Instruction+Follo)
46. Becca Williams. 2023. Parallel process GPT. [https://github.com/tiny-rawr/parallel](https://github.com/tiny-rawr/parallel) process gpt Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. 2024a. β -DPO: Direct preference optimization with dynamic β . arXiv preprint arXiv:2407.08639. — [arXiv:2407.08639](https://arxiv.org/abs/2407.08639) [↗](https://github.com/tiny-rawr/parallel)
47. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social IQa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processi… [find](https://scholar.google.com/scholar?q=Maarten+Sap%2C+Hannah+Rashkin%2C+Derek+Chen%2C+Ronan+Le+Bras%2C+and+Yejin+Choi%2E+2019%2E+Social+IQa%3A+Commonsense+reasoning+about+social+interactions%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in)
48. Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. 2024b. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405 .00675. [find](https://scholar.google.com/scholar?q=Yue+Wu%2C+Zhiqing+Sun%2C+Huizhuo+Yuan%2C+Kaixuan+Ji%2C+Yiming+Yang%2C+and+Quanquan+Gu%2E+2024b%2E+Self%2Dplay+preference+optimization+for+language+model+alignment%2E+arXiv+preprint+arXiv%3A2405+%2E00675%2E)
49. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+2017%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2E)
50. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. 2023. [Challenging BIG-bench tasks and whether chain-of-thought can solve them](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/). In Findings of the Association for Computational Linguistics:…
51. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. [CommonsenseQA: A question answering challenge targeting commonsense knowledge](https://md.research.elamur.ai/free/en/papers/talmor-2019-commonsenseqa-question-answering-challenge-targeting/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
52. Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024. [Contrastive preference optimization: Pushing the boundaries of LLM performance in machine translation](https://md.research.elamur.ai/free/en/papers/xu-2024-contrastive-preference-optimization-pushing-boundaries/). In Forty-first International Conference on Machine Learning.
53. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E. Weston. 2024. Self-rewarding language models. In Forty-first International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Weizhe+Yuan%2C+Richard+Yuanzhe+Pang%2C+Kyunghyun+Cho%2C+Xian+Li%2C+Sainbayar+Sukhbaatar%2C+Jing+Xu%2C+and+Jason+E%2E+Weston%2E+2024%2E+Self%2Drewarding+language+models%2E+In+Forty%2Dfirst+International+Conference+on+Machine+)
54. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800. https://doi.org/10 .18653/v1/P19-1472 [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+HellaSwag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Com)
55. Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. 2024. [Negative preference optimization: From catastrophic collapse to effective unlearning](https://md.research.elamur.ai/free/en/papers/zhang-2024-negative-preference-optimization-catastrophic-collapse/). arXiv preprint arXiv:2404.05868.
56. Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. 2020. TRL: Transformer reinforcement learning. [https://github.com/huggingface/trl](https://github.com/huggingface/trl) [↗](https://github.com/huggingface/trl)
57. Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu. 2023. [Slic-hf: Sequence likelihood calibration with human feedback](https://md.research.elamur.ai/free/en/papers/zhao-2023-slic-hf-sequence-likelihood-calibration/). arXiv preprint arXiv:2305.10425.
58. Zhichao Wang, Bin Bi, Shiva Kumar Pentyala, Kiran Ramnath, Sougata Chaudhuri, Shubham Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2024. [AGIEval: A human-centric benchmark for evaluating foundation models](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/). In Findings of the Associ…
59. Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. 2023. Starling-7B: Improving LLM helpfulness & harmlessness with RLAIF. [find](https://scholar.google.com/scholar?q=Banghua+Zhu%2C+Evan+Frick%2C+Tianhao+Wu%2C+Hanlin+Zhu%2C+and+Jiantao+Jiao%2E+2023%2E+Starling%2D7B%3A+Improving+LLM+helpfulness+%26+harmlessness+with+RLAIF%2E)

*Updated 30 October 2026*