---
title: "BOND: Aligning LLMs with Best-of-N Distillation (2024)"
description: "This Google DeepMind paper introduces BOND (Best-of-N Distillation), an RLHF method that trains a… 65 references, cited by 22 papers here."
authors: ["Pier Giuseppe Sessa", "Robert Dadashi", "Leonard Hussenot", "Johan Ferret", "Nino Vieillard", "Alexandre Rame", "Bobak Shariari", "Sarah Perrin", "Abe Friesen", "Geoffrey Cideron", "Sertan Girgin", "Piotr Stanczyk", "Andrea Michi", "Danila Sinopalnikov", "Sabela Ramos", "Amelie Heliou", "Aliaksei Severyn", "Matt Hoffman", "Nikola Momchev", "Olivier Bachem"]
year: 2024
venue: "arXiv:2407.14622"
source: "https://arxiv.org/abs/2407.14622"
doi: "10.48550/arxiv.2407.14622"
arxiv: "2407.14622"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sessa-2024-bond-aligning-llms-best-n/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sessa-2024-bond-aligning-llms-best-n/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/11/) / BOND: Aligning LLMs with Best-of-N Distillation

# BOND: Aligning LLMs with Best-of-N Distillation

**Authors:** Pier Giuseppe Sessa, [Robert Dadashi](https://md.research.elamur.ai/free/en/authors/robert-dadashi/), [Leonard Hussenot](https://md.research.elamur.ai/free/en/authors/leonard-hussenot/), [Johan Ferret](https://md.research.elamur.ai/free/en/authors/johan-ferret/), [Nino Vieillard](https://md.research.elamur.ai/free/en/authors/nino-vieillard/), [Alexandre Rame](https://md.research.elamur.ai/free/en/authors/alexandre-rame/), Bobak Shariari, [Sarah Perrin](https://md.research.elamur.ai/free/en/authors/sarah-perrin/), Abe Friesen, [Geoffrey Cideron](https://md.research.elamur.ai/free/en/authors/geoffrey-cideron/), [Sertan Girgin](https://md.research.elamur.ai/free/en/authors/sertan-girgin/), [Piotr Stanczyk](https://md.research.elamur.ai/free/en/authors/piotr-stanczyk/), [Andrea Michi](https://md.research.elamur.ai/free/en/authors/andrea-michi/), Danila Sinopalnikov, Sabela Ramos, Amelie Heliou, [Aliaksei Severyn](https://md.research.elamur.ai/free/en/authors/aliaksei-severyn/), Matt Hoffman, Nikola Momchev and Olivier Bachem · **Year:** 2024 · **Venue:** arXiv:2407.14622 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2407.14622) · [PDF](https://arxiv.org/pdf/2407.14622) · [DOI 10.48550/arxiv.2407.14622](https://doi.org/10.48550/arxiv.2407.14622) · [arXiv 2407.14622](https://arxiv.org/abs/2407.14622) · [OpenAlex](https://openalex.org/W4406072085)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [22](https://md.research.elamur.ai/free/en/papers/sessa-2024-bond-aligning-llms-best-n/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Best-of-N sampling draws N responses from a reference model and keeps the one a reward model scores highest. It gives excellent reward-versus-KL trade-offs, but inference costs N times more. This Google DeepMind paper introduces BOND (Best-of-N Distillation), an RLHF method that trains a policy to produce Best-of-N-quality outputs from a single sample.

The authors first derive a closed-form expression for the Best-of-N output distribution. They show it is the optimum of KL-regularized RLHF under two conditions: a rank-based reward (the log quantile of a response's reward) and a regularization strength tied to N. Alignment then becomes a distribution-matching task.

Their practical variant, J-BOND, has three components:
- It minimizes the Jeffreys divergence, a blend of mode-covering forward KL and mode-seeking backward KL.
- It estimates quantiles by Monte Carlo.
- It repeatedly distils Best-of-N of a slowly moving EMA anchor rather than a fixed reference.

On XSum summarization and Gemma models, J-BOND reaches a better KL-reward Pareto front than REINFORCE baselines and improves benchmark results. The work matters because it moves compute from inference to training.

## Key points
- Best-of-N matches the solution of standard KL-regularized RLHF with β = 1/(N−1). Its implied reward depends only on ranks, so it is unaffected by monotone rescaling of the reward model. The authors conjecture this makes it more resistant to reward hacking.
- Baselines were REINFORCE with 2 samples per prompt and a leave-one-out baseline, tested at β_RL ∈ {0.001, 0.01, 0.1, 1}. J-BOND used anchor mixing η = 0.02.
- REINFORCE results depended strongly on the choice of β_RL. J-BOND needs no fixed regularization level: its reward rose steadily while KL grew roughly linearly.
- J-BOND was used to fine-tune Gemma 1.1 2B/7B, RecurrentGemma 2B/9B and CodeGemma 1.1. Gemma 1.1 IT 7B beat Mistral 7B v0.2 Instruct on safety and instruction following.
- The closest prior work, Amini et al. (2024), uses only backward KL. BOND's Jeffreys objective and dynamic-anchor iteration are what distinguish it, and the authors report the iterative anchor is critical for the best results.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 4,145 works · [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 328 works

## Cite

Cite as: Pier Giuseppe Sessa, Robert Dadashi, Leonard Hussenot, Johan Ferret, Nino Vieillard, Alexandre Rame, Bobak Shariari, Sarah Perrin, Abe Friesen, Geoffrey Cideron, Sertan Girgin, Piotr Stanczyk, Andrea Michi, Danila Sinopalnikov, Sabela Ramos, Amelie Heliou, Aliaksei Severyn, Matt Hoffman, Nikola Momchev and Olivier Bachem. 2024. BOND: Aligning LLMs with Best-of-N Distillation. arXiv:2407.14622.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{sessa2024bond,
  title = {{BOND: Aligning LLMs with Best-of-N Distillation}},
  author = {Pier Giuseppe Sessa, Robert Dadashi, Leonard Hussenot, Johan Ferret, Nino Vieillard, Alexandre Rame, Bobak Shariari, Sarah Perrin, Abe Friesen, Geoffrey Cideron, Sertan Girgin, Piotr Stanczyk, Andrea Michi, Danila Sinopalnikov, Sabela Ramos, Amelie Heliou, Aliaksei Severyn, Matt Hoffman, Nikola Momchev and Olivier Bachem},
  year = {2024},
  doi = {10.48550/arxiv.2407.14622},
  eprint = {2407.14622},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2407.14622}
}
```

- [← previous Autoformalizing Euclidean Geometry](https://md.research.elamur.ai/free/en/papers/murphy-2024-autoformalizing-euclidean-geometry/)
- [next → Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model](https://md.research.elamur.ai/free/en/papers/du-2024-chinese-tiny-llm-pretraining/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/11/)

## References

65 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. R. Garea, M. Geist, and O. Bachem. [On-policy distillation of language models: Learning from self-generated mistakes](https://md.research.elamur.ai/free/en/papers/agarwal-2023-policy-distillation-language-models-learning/). In ICLR, 2024. (p. 6)
2. A. Ahmadian, C. Cremer, M. Gallé, M. Fadaee, J. Kreutzer, A. Üstün, and S. Hooker. Back to basics: Revisiting REINFORCE style optimization for learning from human feedback in LLMs. arXiv preprint, 2024. (pp. 3 and 12) [find](https://scholar.google.com/scholar?q=A%2E+Ahmadian%2C+C%2E+Cremer%2C+M%2E+Gall%C3%A9%2C+M%2E+Fadaee%2C+J%2E+Kreutzer%2C+A%2E+%C3%9Cst%C3%BCn%2C+and+S%2E+Hooker%2E+Back+to+basics%3A+Revisiting+REINFORCE+style+optimization+for+learning+from+human+feedback+in+LLMs%2E+arXiv+preprint%2C+202)
3. A. Amini, T. Vieira, and R. Cotterell. Variational Best-of-N alignment. arXiv preprint, 2024. (p. 12) [find](https://scholar.google.com/scholar?q=A%2E+Amini%2C+T%2E+Vieira%2C+and+R%2E+Cotterell%2E+Variational+Best%2Dof%2DN+alignment%2E+arXiv+preprint%2C+2024%2E+%28p%2E+12%29)
4. A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, N. Elhage, Z. Hatfield-Dodds, D. Hernandez, J. Kernion, K. Ndousse, C. Olsson, D. Amodei, T. Brown, J. Clark, S. McCandlish, C. Olah, and J. Kaplan. A general language assistant as a laboratory… [find](https://scholar.google.com/scholar?q=A%2E+Askell%2C+Y%2E+Bai%2C+A%2E+Chen%2C+D%2E+Drain%2C+D%2E+Ganguli%2C+T%2E+Henighan%2C+A%2E+Jones%2C+N%2E+Joseph%2C+B%2E+Mann%2C+N%2E+DasSarma%2C+N%2E+Elhage%2C+Z%2E+Hatfield%2DDodds%2C+D%2E+Hernandez%2C+J%2E+Kernion%2C+K%2E+Ndousse%2C+C%2E+Olsson%2C+D%2E+Amodei%2C+T%2E+B)
5. A. Beirami, A. Agarwal, J. Berant, A. D’Amour, J. Eisenstein, C. Nagpal, and A. T. Suresh. Theoretical guarantees on the Best-of-N alignment policy. arXiv preprint, 2024. (p. 12) [find](https://scholar.google.com/scholar?q=A%2E+Beirami%2C+A%2E+Agarwal%2C+J%2E+Berant%2C+A%2E+D%E2%80%99Amour%2C+J%2E+Eisenstein%2C+C%2E+Nagpal%2C+and+A%2E+T%2E+Suresh%2E+Theoretical+guarantees+on+the+Best%2Dof%2DN+alignment+policy%2E+arXiv+preprint%2C+2024%2E+%28p%2E+12%29)
6. A. Botev, S. De, S. L. Smith, A. Fernando, G.-C. Muraru, R. Haroun, L. Berrada, R. Pascanu, P. G. Sessa, R. Dadashi, L. Hussenot, J. Ferret, S. Girgin, O. Bachem, A. Andreev, K. Kenealy, T. Mesnard, C. Hardin, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, J. Love, P. Tafti, A. Joulin,… [find](https://scholar.google.com/scholar?q=A%2E+Botev%2C+S%2E+De%2C+S%2E+L%2E+Smith%2C+A%2E+Fernando%2C+G%2E%2DC%2E+Muraru%2C+R%2E+Haroun%2C+L%2E+Berrada%2C+R%2E+Pascanu%2C+P%2E+G%2E+Sessa%2C+R%2E+Dadashi%2C+L%2E+Hussenot%2C+J%2E+Ferret%2C+S%2E+Girgin%2C+O%2E+Bachem%2C+A%2E+Andreev%2C+K%2E+Kenealy%2C+T%2E+Mesnard%2C+C)
7. D. Calandriello, D. Guo, R. Munos, M. Rowland, Y. Tang, B. A. Pires, P. H. Richemond, C. L. Lan, M. Valko, T. Liu, et al. Human alignment of large language models through online preference optimisation. In ICML, 2024. (p. 12) [find](https://scholar.google.com/scholar?q=D%2E+Calandriello%2C+D%2E+Guo%2C+R%2E+Munos%2C+M%2E+Rowland%2C+Y%2E+Tang%2C+B%2E+A%2E+Pires%2C+P%2E+H%2E+Richemond%2C+C%2E+L%2E+Lan%2C+M%2E+Valko%2C+T%2E+Liu%2C+et+al%2E+Human+alignment+of+large+language+models+through+online+preference+optimisatio)
8. S. Casper, X. Davies, C. Shi, T. K. Gilbert, J. Scheurer, J. Rando, R. Freedman, T. Korbak, D. Lindner, P. Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. TMLR, 2023. (p. 1) [find](https://scholar.google.com/scholar?q=S%2E+Casper%2C+X%2E+Davies%2C+C%2E+Shi%2C+T%2E+K%2E+Gilbert%2C+J%2E+Scheurer%2C+J%2E+Rando%2C+R%2E+Freedman%2C+T%2E+Korbak%2C+D%2E+Lindner%2C+P%2E+Freire%2C+et+al%2E+Open+problems+and+fundamental+limitations+of+reinforcement+learning+from+human)
9. P. Christiano, B. Shlegeris, and D. Amodei. [Supervising strong learners by amplifying weak experts](https://md.research.elamur.ai/free/en/papers/christiano-2018-supervising-strong-learners-amplifying-weak/). arXiv preprint, 2018. (p. 13)
10. P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In NeurIPS, 2017. (p. 1) [find](https://scholar.google.com/scholar?q=P%2E+F%2E+Christiano%2C+J%2E+Leike%2C+T%2E+Brown%2C+M%2E+Martic%2C+S%2E+Legg%2C+and+D%2E+Amodei%2E+Deep+reinforcement+learning+from+human+preferences%2E+In+NeurIPS%2C+2017%2E+%28p%2E+1%29)
11. J. Clark and D. Amodei. Faulty Reward Functions in the Wild. [https://openai.com/research/](https://openai.com/research/) faulty-reward-functions , 2016. (p. 1) [↗](https://openai.com/research/)
12. CodeGemma Team. [Codegemma: Open code models based on gemma](https://md.research.elamur.ai/free/en/papers/zhao-2024-codegemma-open-code-models-based/). arXiv preprint, 2024. (p. 12)
13. A. Cotra. Iterated distillation and amplification. [https://ai-alignment.com/iterated-dis](https://ai-alignment.com/iterated-dis) tillation-and-amplification-157debfd1616 , 2018. (p. 13) [↗](https://ai-alignment.com/iterated-dis)
14. T. M. Cover. Elements of information theory. 1999. (p. 21) [find](https://scholar.google.com/scholar?q=T%2E+M%2E+Cover%2E+Elements+of+information+theory%2E+1999%2E+%28p%2E+21%29)
15. W. Dabney, M. Rowland, M. G. Bellemare, and R. Munos. Distributional reinforcement learning with quantile regression. In AAAI Conference on Artificial Intelligence, 2017. (p. 21) [find](https://scholar.google.com/scholar?q=W%2E+Dabney%2C+M%2E+Rowland%2C+M%2E+G%2E+Bellemare%2C+and+R%2E+Munos%2E+Distributional+reinforcement+learning+with+quantile+regression%2E+In+AAAI+Conference+on+Artificial+Intelligence%2C+2017%2E+%28p%2E+21%29)
16. H. Dong, W. Xiong, D. Goyal, Y. Zhang, W. Chow, R. Pan, S. Diao, J. Zhang, K. SHUM, and T. Zhang. [RAFT: Reward ranked finetuning for generative foundation model alignment](https://md.research.elamur.ai/free/en/papers/dong-2023-raft-reward-ranked-finetuning-generative/). TMLR, 2023. (p. 12)
17. Y. Flet-Berliac, N. Grinsztajn, F. Strub, E. Choi, C. Cremer, A. Ahmadian, Y. Chandak, M. G. Azar, O. Pietquin, and M. Geist. Contrastive policy gradient: Aligning llms on sequence-level scores in a supervised-friendly fashion. arXiv preprint, 2024. (p. 12) [find](https://scholar.google.com/scholar?q=Y%2E+Flet%2DBerliac%2C+N%2E+Grinsztajn%2C+F%2E+Strub%2C+E%2E+Choi%2C+C%2E+Cremer%2C+A%2E+Ahmadian%2C+Y%2E+Chandak%2C+M%2E+G%2E+Azar%2C+O%2E+Pietquin%2C+and+M%2E+Geist%2E+Contrastive+policy+gradient%3A+Aligning+llms+on+sequence%2Dlevel+scores+in+a+s)
18. R. M. French. Semi-distributed representations and catastrophic forgetting in connectionist networks. Connection Science, 1992. (pp. 1 and 3) [find](https://scholar.google.com/scholar?q=R%2E+M%2E+French%2E+Semi%2Ddistributed+representations+and+catastrophic+forgetting+in+connectionist+networks%2E+Connection+Science%2C+1992%2E+%28pp%2E+1+and+3%29)
19. L. Gao, J. Schulman, and J. Hilton. [Scaling laws for reward model overoptimization](https://md.research.elamur.ai/free/en/papers/gao-2022-scaling-laws-reward-model-overoptimization/). In ICML, 2023.
20. (pp. 1, 7, and 12) M. Geist, B. Scherrer, and O. Pietquin. A theory of regularized markov decision processes. In ICML, 2019. (p. 3) [find](https://scholar.google.com/scholar?q=%28pp%2E+1%2C+7%2C+and+12%29+M%2E+Geist%2C+B%2E+Scherrer%2C+and+O%2E+Pietquin%2E+A+theory+of+regularized+markov+decision+processes%2E+In+ICML%2C+2019%2E+%28p%2E+3%29)
21. Gemini Team. [Gemini: A family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). 2023. (p. 1)
22. Gemma Team. Gemma: Open models based on gemini research and technology. arXiv preprint, 2024. [find](https://scholar.google.com/scholar?q=Gemma+Team%2E+Gemma%3A+Open+models+based+on+gemini+research+and+technology%2E+arXiv+preprint%2C+2024%2E)
23. (pp. 2, 10, and 12) A. Gorbatovski, B. Shaposhnikov, A. Malakhov, N. Surnachev, Y. Aksenov, I. Maksimov, N. Balagansky, and D. Gavrilov. Learn your reference model for real good alignment. arXiv preprint, 2024. (p. 12) [find](https://scholar.google.com/scholar?q=%28pp%2E+2%2C+10%2C+and+12%29+A%2E+Gorbatovski%2C+B%2E+Shaposhnikov%2C+A%2E+Malakhov%2C+N%2E+Surnachev%2C+Y%2E+Aksenov%2C+I%2E+Maksimov%2C+N%2E+Balagansky%2C+and+D%2E+Gavrilov%2E+Learn+your+reference+model+for+real+good+alignment%2E+arXiv+prepr)
24. L. Gui, C. Gârbacea, and V. Veitch. Bonbon alignment for large language models and the sweetness of Best-of-N sampling. arXiv preprint, 2024. (p. 12) [find](https://scholar.google.com/scholar?q=L%2E+Gui%2C+C%2E+G%C3%A2rbacea%2C+and+V%2E+Veitch%2E+Bonbon+alignment+for+large+language+models+and+the+sweetness+of+Best%2Dof%2DN+sampling%2E+arXiv+preprint%2C+2024%2E+%28p%2E+12%29)
25. S. Guo, B. Zhang, T. Liu, T. Liu, M. Khalman, F. Llinares, A. Rame, T. Mesnard, Y. Zhao, B. Piot, J. Ferret, and M. Blondel. Direct language model alignment from online ai feedback. arXiv preprint, 2024. (p. 12) [find](https://scholar.google.com/scholar?q=S%2E+Guo%2C+B%2E+Zhang%2C+T%2E+Liu%2C+T%2E+Liu%2C+M%2E+Khalman%2C+F%2E+Llinares%2C+A%2E+Rame%2C+T%2E+Mesnard%2C+Y%2E+Zhao%2C+B%2E+Piot%2C+J%2E+Ferret%2C+and+M%2E+Blondel%2E+Direct+language+model+alignment+from+online+ai+feedback%2E+arXiv+preprint%2C+20)
26. J. Hilton. KL divergence of max-of-n, 2023. (p. 12) [find](https://scholar.google.com/scholar?q=J%2E+Hilton%2E+KL+divergence+of+max%2Dof%2Dn%2C+2023%2E+%28p%2E+12%29)
27. J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. [Training compute-optimal large language models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/). In NeurIPS, 2022. (p. 13)
28. H. Jeffreys. An invariant form for the prior probability in estimation problems. Proceedings of the Royal Society of London. Series A, Mathematical and Physical Sciences, 186(1007):453–461, 1946. [find](https://scholar.google.com/scholar?q=H%2E+Jeffreys%2E+An+invariant+form+for+the+prior+probability+in+estimation+problems%2E+Proceedings+of+the+Royal+Society+of+London%2E+Series+A%2C+Mathematical+and+Physical+Sciences%2C+186%281007%29%3A453%E2%80%93461%2C+1946%2E)
29. (p. 5) A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. (p. 2) [find](https://scholar.google.com/scholar?q=%28p%2E+5%29+A%2E+Q%2E+Jiang%2C+A%2E+Sablayrolles%2C+A%2E+Roux%2C+A%2E+Mensch%2C+B%2E+Savary%2C+C%2E+Bamford%2C+D%2E+S%2E+Chaplot%2C+D%2E+d%2E+l%2E+Casas%2C+E%2E+B%2E+Hanna%2C+F%2E+Bressand%2C+et+al%2E+Mixtral+of+experts%2E+arXiv+preprint+arXiv%3A2401%2E04088%2C+202)
30. D. P. Kingma and J. Ba. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). In ICLR, 2015. (p. 10)
31. W. Kool, H. van Hoof, and M. Welling. Buy 4 REINFORCE samples, get a baseline for free! 2019. [find](https://scholar.google.com/scholar?q=W%2E+Kool%2C+H%2E+van+Hoof%2C+and+M%2E+Welling%2E+Buy+4+REINFORCE+samples%2C+get+a+baseline+for+free%21+2019%2E)
32. S. Kullback. Information Theory and Statistics. New York, 1959. (p. 4) [find](https://scholar.google.com/scholar?q=S%2E+Kullback%2E+Information+Theory+and+Statistics%2E+New+York%2C+1959%2E+%28p%2E+4%29)
33. A. Lazaridou, A. Potapenko, and O. Tieleman. Multi-agent communication meets natural language: Synergies between functional and structural language learning. In ACL, 2020. (p. 3) [find](https://scholar.google.com/scholar?q=A%2E+Lazaridou%2C+A%2E+Potapenko%2C+and+O%2E+Tieleman%2E+Multi%2Dagent+communication+meets+natural+language%3A+Synergies+between+functional+and+structural+language+learning%2E+In+ACL%2C+2020%2E+%28p%2E+3%29)
34. Z. Li, T. Xu, Y. Zhang, Y. Yu, R. Sun, and Z.-Q. Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. arXiv preprint, 2023. (p. 3) [find](https://scholar.google.com/scholar?q=Z%2E+Li%2C+T%2E+Xu%2C+Y%2E+Zhang%2C+Y%2E+Yu%2C+R%2E+Sun%2C+and+Z%2E%2DQ%2E+Luo%2E+Remax%3A+A+simple%2C+effective%2C+and+efficient+reinforcement+learning+method+for+aligning+large+language+models%2E+arXiv+preprint%2C+2023%2E+%28p%2E+3%29)
35. R. Munos, M. Valko, D. Calandriello, M. G. Azar, M. Rowland, Z. D. Guo, Y. Tang, M. Geist, T. Mesnard, A. Michi, et al. [Nash learning from human feedback](https://md.research.elamur.ai/free/en/papers/munos-2023-nash-learning-human-feedback/). arXiv preprint, 2023. (p. 12)
36. R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, et al. [Webgpt: Browser-assisted question-answering with human feedback](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/). arXiv preprint, 2021.
37. (pp. 1 and 12) S. Narayan, S. B. Cohen, and M. Lapata. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii, editors, Conference on Empirical Methods in Natural Language Processing, 20… — [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://md.research.elamur.ai/free/en/papers/narayan-2018-don-t-give-me-details/)
38. R. Ngo, L. Chan, and S. Mindermann. [The alignment problem from a deep learning perspective](https://md.research.elamur.ai/free/en/papers/ngo-2022-alignment-problem-deep-learning-perspective/). arXiv preprint, 2022. (p. 1)
39. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). 2023. (p. 1)
40. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). NeurIPS, 2022.
41. (p. 1) A. Pace, J. Mallinson, E. Malmi, S. Krause, and A. Severyn. West-of-n: Synthetic preference generation for improved reward modeling. arXiv preprint, 2024. (p. 12) [find](https://scholar.google.com/scholar?q=%28p%2E+1%29+A%2E+Pace%2C+J%2E+Mallinson%2C+E%2E+Malmi%2C+S%2E+Krause%2C+and+A%2E+Severyn%2E+West%2Dof%2Dn%3A+Synthetic+preference+generation+for+improved+reward+modeling%2E+arXiv+preprint%2C+2024%2E+%28p%2E+12%29)
42. A. Pan, K. Bhatia, and J. Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. In ICLR, 2022. (p. 1) [find](https://scholar.google.com/scholar?q=A%2E+Pan%2C+K%2E+Bhatia%2C+and+J%2E+Steinhardt%2E+The+effects+of+reward+misspecification%3A+Mapping+and+mitigating+misaligned+models%2E+In+ICLR%2C+2022%2E+%28p%2E+1%29)
43. A. S. Pinto, A. Kolesnikov, Y. Shi, L. Beyer, and X. Zhai. Tuning computer vision models with task rewards. In ICML, 2023. (p. 12) [find](https://scholar.google.com/scholar?q=A%2E+S%2E+Pinto%2C+A%2E+Kolesnikov%2C+Y%2E+Shi%2C+L%2E+Beyer%2C+and+X%2E+Zhai%2E+Tuning+computer+vision+models+with+task+rewards%2E+In+ICML%2C+2023%2E+%28p%2E+12%29)
44. J. Qiping Yang, S. Salamatian, Z. Sun, A. Theertha Suresh, and A. Beirami. Asymptotics of language model alignment. arXiv, 2024. (p. 3) [find](https://scholar.google.com/scholar?q=J%2E+Qiping+Yang%2C+S%2E+Salamatian%2C+Z%2E+Sun%2C+A%2E+Theertha+Suresh%2C+and+A%2E+Beirami%2E+Asymptotics+of+language+model+alignment%2E+arXiv%2C+2024%2E+%28p%2E+3%29)
45. A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. Improving language understanding by generative pre-training. 2018. (p. 1) [find](https://scholar.google.com/scholar?q=A%2E+Radford%2C+K%2E+Narasimhan%2C+T%2E+Salimans%2C+and+I%2E+Sutskever%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+2018%2E+%28p%2E+1%29)
46. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. 2019. (p. 1) [find](https://scholar.google.com/scholar?q=A%2E+Radford%2C+J%2E+Wu%2C+R%2E+Child%2C+D%2E+Luan%2C+D%2E+Amodei%2C+and+I%2E+Sutskever%2E+Language+models+are+unsupervised+multitask+learners%2E+2019%2E+%28p%2E+1%29)
47. R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). arXiv preprint, 2023. (p. 4)
48. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). JMLR, 2020. (p. 1)
49. A. Ramé, N. Vieillard, L. Hussenot, R. Dadashi, G. Cideron, O. Bachem, and J. Ferret. WARM: On the benefits of weight averaged reward models. In ICML, 2024. (p. 13) [find](https://scholar.google.com/scholar?q=A%2E+Ram%C3%A9%2C+N%2E+Vieillard%2C+L%2E+Hussenot%2C+R%2E+Dadashi%2C+G%2E+Cideron%2C+O%2E+Bachem%2C+and+J%2E+Ferret%2E+WARM%3A+On+the+benefits+of+weight+averaged+reward+models%2E+In+ICML%2C+2024%2E+%28p%2E+13%29)
50. A. Ramé, J. Ferret, N. Vieillard, R. Dadashi, L. Hussenot, P.-L. Cedoz, P. G. Sessa, S. Girgin, A. Douillard, and O. Bachem. WARP: On the benefits of weight averaged rewarded policies. arXiv preprint, 2024. [find](https://scholar.google.com/scholar?q=A%2E+Ram%C3%A9%2C+J%2E+Ferret%2C+N%2E+Vieillard%2C+R%2E+Dadashi%2C+L%2E+Hussenot%2C+P%2E%2DL%2E+Cedoz%2C+P%2E+G%2E+Sessa%2C+S%2E+Girgin%2C+A%2E+Douillard%2C+and+O%2E+Bachem%2E+WARP%3A+On+the+benefits+of+weight+averaged+rewarded+policies%2E+arXiv+preprint%2C)
51. (pp. 10, 12, and 13) M. Reid, N. Savinov, D. Teplyashin, D. Lepikhin, T. Lillicrap, J.-b. Alayrac, R. Soricut, A. Lazaridou, O. Firat, J. Schrittwieser, et al. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/). arXiv preprint, 2024. (p. 1)
52. P. Roit, J. Ferret, L. Shani, R. Aharoni, G. Cideron, R. Dadashi, M. Geist, S. Girgin, L. Hussenot, O. Keller, et al. [Factually consistent summarization via reinforcement learning with textual entailment feedback](https://md.research.elamur.ai/free/en/papers/roit-2023-factually-consistent-summarization-reinforcement-learning/). In ACL, 2023. (p. 6)
53. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint, 2017. (p. 3) [find](https://scholar.google.com/scholar?q=J%2E+Schulman%2C+F%2E+Wolski%2C+P%2E+Dhariwal%2C+A%2E+Radford%2C+and+O%2E+Klimov%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint%2C+2017%2E+%28p%2E+3%29)
54. J. M. V. Skalse, N. H. R. Howe, D. Krasheninnikov, and D. Krueger. Defining and characterizing reward gaming. In NeurIPS, 2022. (pp. 1 and 3) [find](https://scholar.google.com/scholar?q=J%2E+M%2E+V%2E+Skalse%2C+N%2E+H%2E+R%2E+Howe%2C+D%2E+Krasheninnikov%2C+and+D%2E+Krueger%2E+Defining+and+characterizing+reward+gaming%2E+In+NeurIPS%2C+2022%2E+%28pp%2E+1+and+3%29)
55. N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. NeurIPS, 2020. (pp. 1 and 12) [find](https://scholar.google.com/scholar?q=N%2E+Stiennon%2C+L%2E+Ouyang%2C+J%2E+Wu%2C+D%2E+Ziegler%2C+R%2E+Lowe%2C+C%2E+Voss%2C+A%2E+Radford%2C+D%2E+Amodei%2C+and+P%2E+F%2E+Christiano%2E+Learning+to+summarize+with+human+feedback%2E+NeurIPS%2C+2020%2E+%28pp%2E+1+and+12%29)
56. R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, 1998. (p. 6) [find](https://scholar.google.com/scholar?q=R%2E+S%2E+Sutton+and+A%2E+G%2E+Barto%2E+Reinforcement+Learning%3A+An+Introduction%2E+MIT+Press%2C+1998%2E+%28p%2E+6%29)
57. Y. Tang, D. Z. Guo, Z. Zheng, D. Calandriello, Y. Cao, E. Tarassov, R. Munos, B. Ávila Pires, M. Valko, Y. Cheng, and W. Dabney. Understanding the performance gap between online and offline alignment algorithms. arXiv preprint, 2024. (p. 3) [find](https://scholar.google.com/scholar?q=Y%2E+Tang%2C+D%2E+Z%2E+Guo%2C+Z%2E+Zheng%2C+D%2E+Calandriello%2C+Y%2E+Cao%2C+E%2E+Tarassov%2C+R%2E+Munos%2C+B%2E+%C3%81vila+Pires%2C+M%2E+Valko%2C+Y%2E+Cheng%2C+and+W%2E+Dabney%2E+Understanding+the+performance+gap+between+online+and+offline+alignment+)
58. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint, 2023. (pp. 1, 3, and 12)
59. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In NeurIPS, 2017. (p. 2)
60. N. Vieillard, T. Kozuno, B. Scherrer, O. Pietquin, R. Munos, and M. Geist. Leverage the average: an analysis of kl regularization in reinforcement learning. NeurIPS, 2020. (p. 4) [find](https://scholar.google.com/scholar?q=N%2E+Vieillard%2C+T%2E+Kozuno%2C+B%2E+Scherrer%2C+O%2E+Pietquin%2C+R%2E+Munos%2C+and+M%2E+Geist%2E+Leverage+the+average%3A+an+analysis+of+kl+regularization+in+reinforcement+learning%2E+NeurIPS%2C+2020%2E+%28p%2E+4%29)
61. J. Wei, M. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. [Finetuned language models are zero-shot learners](https://md.research.elamur.ai/free/en/papers/wei-2021-finetuned-language-models-zero-shot/). In ICLR, 2022. (p. 1)
62. R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Reinforcement learning, 1992. (pp. 1, 3, 6, 12, and 19) [find](https://scholar.google.com/scholar?q=R%2E+J%2E+Williams%2E+Simple+statistical+gradient%2Dfollowing+algorithms+for+connectionist+reinforcement+learning%2E+Reinforcement+learning%2C+1992%2E+%28pp%2E+1%2C+3%2C+6%2C+12%2C+and+19%29)
63. Y. Wu, Z. Sun, H. Yuan, K. Ji, Y. Yang, and Q. Gu. Self-play preference optimization for language model alignment. arXiv preprint, 2024. (p. 12) [find](https://scholar.google.com/scholar?q=Y%2E+Wu%2C+Z%2E+Sun%2C+H%2E+Yuan%2C+K%2E+Ji%2C+Y%2E+Yang%2C+and+Q%2E+Gu%2E+Self%2Dplay+preference+optimization+for+language+model+alignment%2E+arXiv+preprint%2C+2024%2E+%28p%2E+12%29)
64. J. Q. Yang, S. Salamatian, Z. Sun, A. T. Suresh, and A. Beirami. Asymptotics of language model alignment. arXiv preprint, 2024. (p. 12) [find](https://scholar.google.com/scholar?q=J%2E+Q%2E+Yang%2C+S%2E+Salamatian%2C+Z%2E+Sun%2C+A%2E+T%2E+Suresh%2C+and+A%2E+Beirami%2E+Asymptotics+of+language+model+alignment%2E+arXiv+preprint%2C+2024%2E+%28p%2E+12%29)
65. D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human preferences. arXiv preprint, 2019. (p. 1) [find](https://scholar.google.com/scholar?q=D%2E+M%2E+Ziegler%2C+N%2E+Stiennon%2C+J%2E+Wu%2C+T%2E+B%2E+Brown%2C+A%2E+Radford%2C+D%2E+Amodei%2C+P%2E+Christiano%2C+and+G%2E+Irving%2E+Fine%2Dtuning+language+models+from+human+preferences%2E+arXiv+preprint%2C+2019%2E+%28p%2E+1%29)

*Updated 3 November 2026*