---
title: "Smaug — paper by Pal et al. (2024) — summary and ideas"
description: "The authors argue theoretically, and confirm with token-level measurements, that this is most likely… 72 references, cited by 76 papers here."
authors: ["Arka Pal", "Deep Karkhanis", "Samuel Dooley", "Manley Roberts", "Siddartha Naidu", "Colin White"]
year: 2024
venue: "arXiv:2402.13228"
source: "https://arxiv.org/abs/2402.13228"
doi: "10.48550/arxiv.2402.13228"
arxiv: "2402.13228"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/pal-2024-smaug-fixing-failure-modes-preference/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pal-2024-smaug-fixing-failure-modes-preference/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive

# Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive

**Authors:** Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu and Colin White · **Year:** 2024 · **Venue:** arXiv:2402.13228 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2402.13228) · [PDF](https://arxiv.org/pdf/2402.13228) · [DOI 10.48550/arxiv.2402.13228](https://doi.org/10.48550/arxiv.2402.13228) · [arXiv 2402.13228](https://arxiv.org/abs/2402.13228) · [OpenAlex](https://openalex.org/W4392019816)

**Citations:** 6 citations (OpenAlex, 2026-10-14) · [76](https://md.research.elamur.ai/free/en/papers/pal-2024-smaug-fixing-failure-modes-preference/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper identifies a weakness in Direct Preference Optimisation (DPO). Its loss depends only on the gap between the log-ratios of the preferred and dispreferred completions, so training can lower the likelihood of the preferred answer as long as the dispreferred one falls further. The authors argue theoretically, and confirm with token-level measurements, that this is most likely when paired completions differ by only a few tokens (low edit distance). In that case DPO boosts the differing tokens but can degrade the tokens after them.

Their fix is DPO-Positive (DPOP). It adds a penalty whenever the policy assigns the preferred completion less probability than the reference model does. They also convert ARC, HellaSwag and MetaMath into pairwise preference datasets and use them to tune 7B, 34B and 72B models. DPOP beat DPO even on high-edit-distance data and on MT-Bench, which is unrelated to the training data. The resulting Smaug-72B was the first open model to average above 80% on the HuggingFace Open LLM Leaderboard. The work matters because many preference datasets have near-identical pairs.

## Key points
- **Smaug-72B results:** In the March 2024 standings of the HuggingFace Open LLM Leaderboard, it averaged 80.48%, almost 2 points above MoMo-72B-lora-1.8.7-DPO (78.55%). It also had the best MMLU and third-best MT-Bench score among open models.
- **Base models:** Smaug-34B starts from Bagel-34B-v0.2 (built on Yi-34B-200k), which was given extra SFT on MetaMath, ORCA-Chat and ShareGPT before DPOP. Smaug-72B starts from MoMo-72b-lora-1.8.7-DPO (built on Qwen-72B). The 72B training took 144 hours.
- **Released resources:** code, models and the new preference versions of ARC, HellaSwag and MetaMath are available at github.com/abacusai/smaug.
- **Contamination check:** an open-source contamination checker (Shi, 2023) gave Smaug-72B scores on ARC, TruthfulQA and GSM8K comparable to its base model and to Llama-2-70B.
- **Limitations:** there was no full ablation at 72B scale, and evaluation covered only six English-language datasets. Effectiveness on non-English data, including Ukrainian, has not been tested.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

## Cite

Cite as: Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu and Colin White. 2024. Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive. arXiv:2402.13228.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{pal2024smaug,
  title = {{Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive}},
  author = {Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu and Colin White},
  year = {2024},
  doi = {10.48550/arxiv.2402.13228},
  eprint = {2402.13228},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.13228}
}
```

- [← previous MART: Improving LLM Safety with Multi-round Automatic Red-Teaming](https://md.research.elamur.ai/free/en/papers/ge-2024-mart-improving-llm-safety-multi/)
- [next → Chain-of-Dictionary Prompting Elicits Translation in Large Language Models](https://md.research.elamur.ai/free/en/papers/lu-2024-chain-dictionary-prompting-elicits-translation/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/)

## References

72 references, 32 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. 01.AI. Yi-34b-200k, 2024. URL [https://huggingface.co/01-ai/Yi-34B-200K](https://huggingface.co/01-ai/Yi-34B-200K). [↗](https://huggingface.co/01-ai/Yi-34B-200K)
2. AllenAI. Ultrafeedback binarized clean, 2024. URL [https://huggingface.co/datasets/allenai/ultrafeedback_binarized_cleaned](https://huggingface.co/datasets/allenai/ultrafeedback_binarized_cleaned). [↗](https://huggingface.co/datasets/allenai/ultrafeedback_binarized_cleaned)
3. Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. [Learning from mistakes makes llm better reasoner](https://md.research.elamur.ai/free/en/papers/an-2023-learning-mistakes-makes-llm-better/). arXiv preprint arXiv:2310.20689, 2023.
4. Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. [A general theoretical paradigm to understand learning from human preferences](https://md.research.elamur.ai/free/en/papers/azar-2023-general-theoretical-paradigm-understand-learning/), 2023.
5. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. [Qwen technical report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/). arXiv preprint arXiv:2309.16609, 2023.
6. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Sh… [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+Nicholas+Joseph%2C+Saurav+Kadavath%2C+Jackson+Kernion%2C+Tom+Conerly%2C+)
7. Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open llm leaderboard. [https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard), 2023. [↗](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
8. BIG bench authors. [Beyond the imitation game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/), 2023.
9. R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. doi: 10.2307/2334029. — [doi:10.2307/2334029](https://doi.org/10.2307/2334029) [find](https://scholar.google.com/scholar?q=R%2E+A%2E+Bradley+and+M%2E+E%2E+Terry%2E+Rank+analysis+of+incomplete+block+designs%3A+I%2E+the+method+of+paired+comparisons%2E+Biometrika%2C+39%283%2F4%29%3A324%E2%80%93345%2C+1952%2E+doi%3A+10%2E2307%2F2334029%2E+%E2%80%94+%5Bdoi%3A10%2E2307%2F2334029%5D%28https%3A%2F%2F)
10. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020.
11. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. [Sparks of artificial general intelligence: Early experiments with gpt-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/). arXiv preprint arXiv:2303.12712, 2023.
12. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. [A simple framework for contrastive learning of visual representations](https://md.research.elamur.ai/free/en/papers/chen-2020-simple-framework-contrastive-learning-visual/). In International conference on machine learning, pages 1597–1607. PMLR, 2020.
13. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL [https://lmsys.org/blog/2023-03-30-vicuna/…](https://lmsys.org/blog/2023-03-30-vicuna/…) [↗](https://lmsys.org/blog/2023-03-30-vicuna/…)
14. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). Journal of Machine Learning Research, 24(240):1–113, 2023.
15. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. [find](https://scholar.google.com/scholar?q=Paul+F+Christiano%2C+Jan+Leike%2C+Tom+Brown%2C+Miljan+Martic%2C+Shane+Legg%2C+and+Dario+Amodei%2E+Deep+reinforcement+learning+from+human+preferences%2E+Advances+in+neural+information+processing+systems%2C+30%2C+2017%2E)
16. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Yunxuan+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+et+al%2E+Scaling+instruction%2Dfinetuned+language+models%2E+arXiv+p)
17. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/), 2018.
18. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/), 2021.
19. Jon Durbin. Bagel-34b-v0.2, 2024a. URL [https://huggingface.co/jondurbin/bagel-34b-v0.2](https://huggingface.co/jondurbin/bagel-34b-v0.2). [↗](https://huggingface.co/jondurbin/bagel-34b-v0.2)
20. Jon Durbin. Truthy dpo, 2024b. URL [https://huggingface.co/datasets/jondurbin/truthy-dpo-v0.1](https://huggingface.co/datasets/jondurbin/truthy-dpo-v0.1). [↗](https://huggingface.co/datasets/jondurbin/truthy-dpo-v0.1)
21. Shahul Es. Orca-chat, 2024. URL [https://huggingface.co/datasets/shahules786/orca-chat](https://huggingface.co/datasets/shahules786/orca-chat). [↗](https://huggingface.co/datasets/shahules786/orca-chat)
22. Kawin Ethayarajh, Winnie Xu, Dan Jurafsky, and Douwe Kiela. Human-centered loss functions (halos). Technical report, Contextual AI, 2023. [find](https://scholar.google.com/scholar?q=Kawin+Ethayarajh%2C+Winnie+Xu%2C+Dan+Jurafsky%2C+and+Douwe+Kiela%2E+Human%2Dcentered+loss+functions+%28halos%29%2E+Technical+report%2C+Contextual+AI%2C+2023%2E)
23. Duanyu Feng, Bowen Qin, Chen Huang, Zheng Zhang, and Wenqiang Lei. Towards analyzing and understanding the limitations of dpo: A theoretical perspective. arXiv preprint arXiv:2404.04626, 2024. — [arXiv:2404.04626](https://arxiv.org/abs/2404.04626) [find](https://scholar.google.com/scholar?q=Duanyu+Feng%2C+Bowen+Qin%2C+Chen+Huang%2C+Zheng+Zhang%2C+and+Wenqiang+Lei%2E+Towards+analyzing+and+understanding+the+limitations+of+dpo%3A+A+theoretical+perspective%2E+arXiv+preprint+arXiv%3A2404%2E04626%2C+2024%2E+%E2%80%94+%5BarXi)
24. Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September… [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Jason+Phang%2C+Laria+Reynolds%2C+Eric+Tang%2C+Anish+Thite)
25. R. Hadsell, S. Chopra, and Y. LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), volume 2, pages 1735–1742, 2006. doi: 10.1109/CVPR.2006.100. — [doi:10.1109/cvpr.2006.100](https://doi.org/10.1109/cvpr.2006.100) [find](https://scholar.google.com/scholar?q=R%2E+Hadsell%2C+S%2E+Chopra%2C+and+Y%2E+LeCun%2E+Dimensionality+reduction+by+learning+an+invariant+mapping%2E+In+2006+IEEE+Computer+Society+Conference+on+Computer+Vision+and+Pattern+Recognition+%28CVPR%E2%80%9906%29%2C+volume+2%2C)
26. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. [Momentum contrast for unsupervised visual representation learning](https://md.research.elamur.ai/free/en/papers/he-2019-momentum-contrast-unsupervised-visual-representation/). In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738, 2020.
27. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. [Measuring coding challenge competence with apps](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-coding-challenge-competence-apps/). arXiv preprint arXiv:2105.09938, 2021.
28. Intel. Orca dpo pairs, 2024. URL [https://huggingface.co/datasets/Intel/orca_dpo_pairs](https://huggingface.co/datasets/Intel/orca_dpo_pairs). [↗](https://huggingface.co/datasets/Intel/orca_dpo_pairs)
29. Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. [Camels in a changing climate: Enhancing lm adaptation with tulu 2](https://md.research.elamur.ai/free/en/papers/ivison-2023-camels-changing-climate-enhancing-lm/). arXiv preprint arXiv:2311.10702, 2023.
30. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. [Livecodebench: Holistic and contamination free evaluation of large language models for code](https://md.research.elamur.ai/free/en/papers/jain-2024-livecodebench-holistic-contamination-free-evaluation/). arXiv preprint arXiv:2403.07974, 2024.
31. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
32. Julia Kreutzer, Joshua Uyheng, and Stefan Riezler. [Reliability and learnability of human bandit feedback for sequence-to-sequence reinforcement learning](https://md.research.elamur.ai/free/en/papers/kreutzer-2018-reliability-learnability-human-bandit-feedback/). arXiv preprint arXiv:1805.10627, 2018.
33. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From live data to high-quality benchmarks: The arena-hard pipeline, April 2024. URL [https://lmsys.org/blog/2024-04-19-arena-hard/](https://lmsys.org/blog/2024-04-19-arena-hard/). [↗](https://lmsys.org/blog/2024-04-19-arena-hard/)
34. Stephanie Lin, Jacob Hilton, and Owain Evans. [Truthfulqa: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022.
35. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+Decoupled+weight+decay+regularization%2E+arXiv+preprint+arXiv%3A1711%2E05101%2C+2017%2E)
36. R Duncan Luce. Individual choice behavior: A theoretical analysis. Courier Corporation, 2005. [find](https://scholar.google.com/scholar?q=R+Duncan+Luce%2E+Individual+choice+behavior%3A+A+theoretical+analysis%2E+Courier+Corporation%2C+2005%2E)
37. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. arXiv preprint arXiv:2104.08773, 2021. [find](https://scholar.google.com/scholar?q=Swaroop+Mishra%2C+Daniel+Khashabi%2C+Chitta+Baral%2C+and+Hannaneh+Hajishirzi%2E+Cross%2Dtask+generalization+via+natural+language+crowdsourcing+instructions%2E+arXiv+preprint+arXiv%3A2104%2E08773%2C+2021%2E)
38. Moreh. Momo-72b-lora-1.8.7-dpo, 2024. URL [https://huggingface.co/moreh/MoMo-72B-lora-1.8.7-DPO](https://huggingface.co/moreh/MoMo-72B-lora-1.8.7-DPO). [↗](https://huggingface.co/moreh/MoMo-72B-lora-1.8.7-DPO)
39. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. [Representation learning with contrastive predictive coding](https://md.research.elamur.ai/free/en/papers/oord-2018-representation-learning-contrastive-predictive-coding/). arXiv preprint arXiv:1807.03748, 2018.
40. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). Technical Report, 2023.
41. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training la… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
42. Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. [Iterative reasoning preference optimization](https://md.research.elamur.ai/free/en/papers/pang-2024-iterative-reasoning-preference-optimization/). arXiv preprint arXiv:2404.19733, 2024.
43. Robin L Plackett. The analysis of permutations. Journal of the Royal Statistical Society Series C: Applied Statistics, 24(2):193–202, 1975. [find](https://scholar.google.com/scholar?q=Robin+L+Plackett%2E+The+analysis+of+permutations%2E+Journal+of+the+Royal+Statistical+Society+Series+C%3A+Applied+Statistics%2C+24%282%29%3A193%E2%80%93202%2C+1975%2E)
44. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2C+2019%2E)
45. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2023.
46. Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to q* : Your language model is secretly a q-function. arXiv preprint arXiv:2404.[12358, 2024](https://md.research.elamur.ai/free/en/papers/rafailov-2024-r-q-language-model-secretly/).
47. Manley Roberts, Himanshu Thakur, Christine Herlihy, Colin White, and Samuel Dooley. To the cutoff... and beyond? a longitudinal perspective on llm data contamination. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. [find](https://scholar.google.com/scholar?q=Manley+Roberts%2C+Himanshu+Thakur%2C+Christine+Herlihy%2C+Colin+White%2C+and+Samuel+Dooley%2E+To+the+cutoff%2E%2E%2E+and+beyond%3F+a+longitudinal+perspective+on+llm+data+contamination%2E+In+Proceedings+of+the+Internation)
48. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Proceedings of the AAAI Conference on Artificial Intelligence, 34, 2020. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+34%2C)
49. Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In International Conference on Machine Learning, pages 5628–5637. PMLR, 2019. [find](https://scholar.google.com/scholar?q=Nikunj+Saunshi%2C+Orestis+Plevrakis%2C+Sanjeev+Arora%2C+Mikhail+Khodak%2C+and+Hrishikesh+Khandeparkar%2E+A+theoretical+analysis+of+contrastive+unsupervised+representation+learning%2E+In+International+Conference+o)
50. Florian Schroff, Dmitry Kalenichenko, and James Philbin. [Facenet: A unified embedding for face recognition and clustering](https://md.research.elamur.ai/free/en/papers/schroff-2015-facenet-unified-embedding-face-recognition/). In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 815–823, 2015. doi: 10.1109/CVPR.2015.7298682.
51. Weijia Shi. Detect pretrain code contamination. detect-pretrain-code-contamination, 2023. [find](https://scholar.google.com/scholar?q=Weijia+Shi%2E+Detect+pretrain+code+contamination%2E+detect%2Dpretrain%2Dcode%2Dcontamination%2C+2023%2E)
52. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2020. [find](https://scholar.google.com/scholar?q=Nisan+Stiennon%2C+Long+Ouyang%2C+Jeffrey+Wu%2C+Daniel+Ziegler%2C+Ryan+Lowe%2C+Chelsea+Voss%2C+Alec+Radford%2C+Dario+Amodei%2C+and+Paul+F+Christiano%2E+Learning+to+summarize+with+human+feedback%2E+In+Proceedings+of+the+An)
53. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models. [https://crfm.stanford](https://crfm.stanford). edu/2023/03/13/alpaca. html, 3(6):7, 202… [↗](https://crfm.stanford)
54. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023a.
55. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023b.
56. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of lm alignment, 2023. [find](https://scholar.google.com/scholar?q=Lewis+Tunstall%2C+Edward+Beeching%2C+Nathan+Lambert%2C+Nazneen+Rajani%2C+Kashif+Rasul%2C+Younes+Belkada%2C+Shengyi+Huang%2C+Leandro+von+Werra%2C+Cl%C3%A9mentine+Fourrier%2C+Nathan+Habib%2C+Nathan+Sarrazin%2C+Omar+Sanseviero%2C+Al)
57. Amos Tversky and Daniel Kahneman. Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and Uncertainty, 5(4):297–323, 1992. [https://github.com/swj0419/](https://github.com/swj0419/) [↗](https://github.com/swj0419/)
58. Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. Trl: Transformer reinforcement learning. [https://github.com/huggingface/trl](https://github.com/huggingface/trl), 2020. [↗](https://github.com/huggingface/trl)
59. Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2495–2504, 2021. [find](https://scholar.google.com/scholar?q=Feng+Wang+and+Huaping+Liu%2E+Understanding+the+behaviour+of+contrastive+loss%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+computer+vision+and+pattern+recognition%2C+pages+2495%E2%80%932504%2C+2021%2E)
60. Peiyi Wang, Lei Li, Liang Chen, Feifan Song, Binghuai Lin, Yunbo Cao, Tianyu Liu, and Zhifang Sui. Making large language models better reasoners with alignment. arXiv preprint arXiv:2309.02144, 2023. — [arXiv:2309.02144](https://arxiv.org/abs/2309.02144) [find](https://scholar.google.com/scholar?q=Peiyi+Wang%2C+Lei+Li%2C+Liang+Chen%2C+Feifan+Song%2C+Binghuai+Lin%2C+Yunbo+Cao%2C+Tianyu+Liu%2C+and+Zhifang+Sui%2E+Making+large+language+models+better+reasoners+with+alignment%2E+arXiv+preprint+arXiv%3A2309%2E02144%2C+2023%2E+)
61. Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pages 9929–9939. PMLR, 2020. [find](https://scholar.google.com/scholar?q=Tongzhou+Wang+and+Phillip+Isola%2E+Understanding+contrastive+representation+learning+through+alignment+and+uniformity+on+the+hypersphere%2E+In+International+Conference+on+Machine+Learning%2C+pages+9929%E2%80%939939)
62. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/), 2023.
63. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe… — [Transformers: State-of-the-Art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
64. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.[12244, 2023](https://md.research.elamur.ai/free/en/papers/xu-2023-wizardlm-empowering-large-pre-trained/).
65. Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. [Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation](https://md.research.elamur.ai/free/en/papers/xu-2024-contrastive-preference-optimization-pushing-boundaries/). arXiv preprint arXiv:2401.08417, 2024.
66. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. [Metamath: Bootstrap your own mathematical questions for large language models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/), 2023.
67. Z. Sharegpt_vicuna_unfiltered, 2024. URL [https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered). [↗](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered)
68. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy, July 2019. Association for Computational Linguis… [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+HellaSwag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computati)
69. Yao Zhao, Misha Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J. Liu. Calibrating sequence likelihood improves conditional language generation, 2022. [find](https://scholar.google.com/scholar?q=Yao+Zhao%2C+Misha+Khalman%2C+Rishabh+Joshi%2C+Shashi+Narayan%2C+Mohammad+Saleh%2C+and+Peter+J%2E+Liu%2E+Calibrating+sequence+likelihood+improves+conditional+language+generation%2C+2022%2E)
70. Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu. [Slic-hf: Sequence likelihood calibration with human feedback](https://md.research.elamur.ai/free/en/papers/zhao-2023-slic-hf-sequence-likelihood-calibration/), 2023.
71. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). arXiv preprint arXiv:2306.05685, 2023.
72. Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2020. [find](https://scholar.google.com/scholar?q=Daniel+M%2E+Ziegler%2C+Nisan+Stiennon%2C+Jeffrey+Wu%2C+Tom+B%2E+Brown%2C+Alec+Radford%2C+Dario+Amodei%2C+Paul+Christiano%2C+and+Geoffrey+Irving%2E+Fine%2Dtuning+language+models+from+human+preferences%2C+2020%2E)

*Updated 2 November 2026*