---
title: "Weak-to-Strong Search — paper by Zhou et al. (2024)"
description: "The authors propose weak-to-strong search, which keeps the large model frozen and steers its output… 57 references, cited by 17 papers here."
authors: ["Zhanhui Zhou", "Zhixuan Liu", "Jie Liu", "Zhichen Dong", "Chao Yang", "Yu Qiao"]
year: 2024
venue: "arXiv:2405.19262"
source: "https://arxiv.org/abs/2405.19262"
doi: "10.48550/arxiv.2405.19262"
arxiv: "2405.19262"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhou-2024-weak-strong-search-align-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhou-2024-weak-strong-search-align-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/) / Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models

# Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models

**Authors:** [Zhanhui Zhou](https://md.research.elamur.ai/free/en/authors/zhanhui-zhou/), Zhixuan Liu, [Jie Liu](https://md.research.elamur.ai/free/en/authors/jie-liu/), Zhichen Dong, [Chao Yang](https://md.research.elamur.ai/free/en/authors/chao-yang/) and [Yu Qiao](https://md.research.elamur.ai/free/en/authors/yu-qiao/) · **Year:** 2024 · **Venue:** [arXiv:2405.19262](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/) ([NeurIPS 2024](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/2024/)) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2405.19262) · [PDF](https://arxiv.org/pdf/2405.19262) · [DOI 10.48550/arxiv.2405.19262](https://doi.org/10.48550/arxiv.2405.19262) · [arXiv 2405.19262](https://arxiv.org/abs/2405.19262) · [OpenAlex](https://openalex.org/W4415798763)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [17](https://md.research.elamur.ai/free/en/papers/zhou-2024-weak-strong-search-align-large/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Aligning very large language models with human preferences through fine-tuning is costly and hard to engineer. Earlier test-time search methods have their own problems: some reduce decoding to a bandit problem, and others need a value model trained from scratch. The authors propose weak-to-strong search, which keeps the large model frozen and steers its output with two small models, one preference-tuned and one untuned.

The method builds on the token-level MDP view of alignment. The per-token log-probability gap between the small pair serves as a dense reward, and its running sum serves as a value estimate. Their algorithm, Chunk-level Beam Search (CBS), alternates between sampling text chunks from the large model and keeping the candidates the small pair rates highest. Because it only needs samples, it works with models that use different vocabularies and even with black-box APIs.

Tests cover controlled-sentiment generation, summarization and AlpacaEval 2.0. Guidance from much weaker models improved strong ones, offering a cheap upgrade path and a practical example of weak-to-strong generalization.

## Key points
- **Small guides on simple tasks:** In sentiment control and summarization, tuned and untuned gpt2 (124M) models steered larger GPT-2 (e.g., gpt2-xl), Llama-2 and Llama-3 models without extra training. The results matched or beat existing methods.
- **AlpacaEval 2.0 gains:** Off-the-shelf zephyr-7b-beta and mistral-7b-sft-beta served as the tuned/untuned guide pair, scored by length-controlled win rate (LC WR) against gpt-4-turbo.
  - Llama-3-70B-Instruct rose from 34.4% to 37.9%.
  - gpt-3.5-turbo-instruct rose from 16.0% to 20.1%.
  - The guide models alone score only about 10%.
- **Model coverage:** The method was tested on three kinds of large models:
  - same vocabulary as the guides: Llama-2-7b-chat, Llama-2-70b-chat;
  - different vocabulary: Llama-3-8B/70B-Instruct;
  - black box: gpt-3.5-turbo-instruct, using a chunk length of 100.
- **Comparison with baselines:** It clearly outperformed EFT and Best-of-N (BoN), which mostly interpolate between weak and strong models. It also outperformed plain beam search over the strong model, which showed no clear gain. The same held with tulu-2-dpo-7b/tulu-2-7b guidance.
- **Availability:** Code is released at github.com/ZHZisZZ/weak-to-strong-search (NeurIPS 2024).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Zhanhui Zhou, Zhixuan Liu, Jie Liu, Zhichen Dong, Chao Yang and Yu Qiao. 2024. Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models. arXiv:2405.19262.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhou2024weak,
  title = {{Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models}},
  author = {Zhanhui Zhou, Zhixuan Liu, Jie Liu, Zhichen Dong, Chao Yang and Yu Qiao},
  year = {2024},
  doi = {10.48550/arxiv.2405.19262},
  eprint = {2405.19262},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.19262}
}
```

- [← previous Quality or Quantity? On Data Scale and Diversity in Adapting Large Language Models for Low-Resource Translation](https://md.research.elamur.ai/free/en/papers/iyer-2024-quality-quantity-data-scale-diversity/)
- [next → AraMed: Arabic Medical Question Answering using Pretrained Transformer Language Models](https://md.research.elamur.ai/free/en/papers/alasmari-2024-aramed-arabic-medical-question-answering/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/)

## References

57 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. [find](https://scholar.google.com/scholar?q=Daniel+M+Ziegler%2C+Nisan+Stiennon%2C+Jeffrey+Wu%2C+Tom+B+Brown%2C+Alec+Radford%2C+Dario+Amodei%2C+Paul+Christiano%2C+and+Geoffrey+Irving%2E+Fine%2Dtuning+language+models+from+human+preferences%2E+arXiv+preprint+arXiv%3A19)
2. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020. [find](https://scholar.google.com/scholar?q=Nisan+Stiennon%2C+Long+Ouyang%2C+Jeffrey+Wu%2C+Daniel+Ziegler%2C+Ryan+Lowe%2C+Chelsea+Voss%2C+Alec+Radford%2C+Dario+Amodei%2C+and+Paul+F+Christiano%2E+Learning+to+summarize+with+human+feedback%2E+Advances+in+Neural+Infor)
3. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in neural information processing systems, 35:27730–27744, 2022.
4. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). Advances in Neural Information Processing Systems, 36, 2024.
5. Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. [A general theoretical paradigm to understand learning from human preferences](https://md.research.elamur.ai/free/en/papers/azar-2023-general-theoretical-paradigm-understand-learning/). In International Conference on Artificial Intelligence and Statistics, pages 4447–4455. PMLR, 2024…
6. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+et+al%2E+Training+a+helpful+and+harmless+assistant+with+reinforcem)
7. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023.
8. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+Mistral+7b)
9. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944, 2023. [find](https://scholar.google.com/scholar?q=Lewis+Tunstall%2C+Edward+Beeching%2C+Nathan+Lambert%2C+Nazneen+Rajani%2C+Kashif+Rasul%2C+Younes+Belkada%2C+Shengyi+Huang%2C+Leandro+von+Werra%2C+Cl%C3%A9mentine+Fourrier%2C+Nathan+Habib%2C+et+al%2E+Zephyr%3A+Direct+distillation+o)
10. Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, et al. Openassistant conversations-democratizing large language model alignment. Advances in Neural Information Processing Systems, 36, 202… — [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://md.research.elamur.ai/free/en/papers/kopf-2023-openassistant-conversations-democratizing-large-language/)
11. Eric Mitchell, Rafael Rafailov, Archit Sharma, Chelsea Finn, and Christopher D Manning. [An emulator for fine-tuning large language models using small language models](https://md.research.elamur.ai/free/en/papers/mitchell-2023-emulator-fine-tuning-large-language/). arXiv preprint arXiv:2310.12962, 2023.
12. Alisa Liu, Xiaochuang Han, Yizhong Wang, Yulia Tsvetkov, Yejin Choi, and Noah A Smith. Tuning language models by proxy. arXiv preprint arXiv:2401.08565, 2024. [find](https://scholar.google.com/scholar?q=Alisa+Liu%2C+Xiaochuang+Han%2C+Yizhong+Wang%2C+Yulia+Tsvetkov%2C+Yejin+Choi%2C+and+Noah+A+Smith%2E+Tuning+language+models+by+proxy%2E+arXiv+preprint+arXiv%3A2401%2E08565%2C+2024%2E)
13. Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, et al. Controlled decoding from language models. arXiv preprint arXiv:2310.17022, 2023. [find](https://scholar.google.com/scholar?q=Sidharth+Mudgal%2C+Jong+Lee%2C+Harish+Ganapathy%2C+YaGuang+Li%2C+Tao+Wang%2C+Yanping+Huang%2C+Zhifeng+Chen%2C+Heng%2DTze+Cheng%2C+Michael+Collins%2C+Trevor+Strohman%2C+et+al%2E+Controlled+decoding+from+language+models%2E+arXiv)
14. Minbeom Kim, Hwanhee Lee, Kang Min Yoo, Joonsuk Park, Hwaran Lee, and Kyomin Jung. Critic-guided decoding for controlled text generation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 4598–4612, 2023. [find](https://scholar.google.com/scholar?q=Minbeom+Kim%2C+Hwanhee+Lee%2C+Kang+Min+Yoo%2C+Joonsuk+Park%2C+Hwaran+Lee%2C+and+Kyomin+Jung%2E+Critic%2Dguided+decoding+for+controlled+text+generation%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+)
15. James Y Huang, Sailik Sengupta, Daniele Bonadiman, Yi-an Lai, Arshit Gupta, Nikolaos Pappas, Saab Mansour, Katrin Kirchoff, and Dan Roth. [Deal: Decoding-time alignment for large language models](https://md.research.elamur.ai/free/en/papers/huang-2025-deal-decoding-time-alignment-large/). arXiv preprint arXiv:2402.06147, 2024.
16. Leo Gao, John Schulman, and Jacob Hilton. [Scaling laws for reward model overoptimization](https://md.research.elamur.ai/free/en/papers/gao-2022-scaling-laws-reward-model-overoptimization/). In International Conference on Machine Learning, pages 10835–10866. PMLR, 2023.
17. Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Alexander D’Amour, Jacob Eisenstein, Chirag Nagpal, and Ananda Theertha Suresh. Theoretical guarantees on the best-of-n alignment policy. arXiv preprint arXiv:2401.01879, 2024. — [arXiv:2401.01879](https://arxiv.org/abs/2401.01879) [find](https://scholar.google.com/scholar?q=Ahmad+Beirami%2C+Alekh+Agarwal%2C+Jonathan+Berant%2C+Alexander+D%E2%80%99Amour%2C+Jacob+Eisenstein%2C+Chirag+Nagpal%2C+and+Ananda+Theertha+Suresh%2E+Theoretical+guarantees+on+the+best%2Dof%2Dn+alignment+policy%2E+arXiv+preprint+)
18. Jiacheng Liu, Andrew Cohen, Ramakanth Pasunuru, Yejin Choi, Hannaneh Hajishirzi, and Asli Celikyilmaz. Making ppo even better: Value-guided monte-carlo tree search decoding. arXiv preprint arXiv:2309.[15028, 2023](https://md.research.elamur.ai/free/en/papers/liu-2023-don-t-throw-away-value/).
19. Xidong Feng, Ziyu Wan, Muning Wen, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero-like tree-search can guide large language model decoding and training. arXiv preprint arXiv:2309.17179, 2023. [find](https://scholar.google.com/scholar?q=Xidong+Feng%2C+Ziyu+Wan%2C+Muning+Wen%2C+Ying+Wen%2C+Weinan+Zhang%2C+and+Jun+Wang%2E+Alphazero%2Dlike+tree%2Dsearch+can+guide+large+language+model+decoding+and+training%2E+arXiv+preprint+arXiv%3A2309%2E17179%2C+2023%2E)
20. Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to q ∗ : Your language model is secretly a q-function. arXiv preprint arXiv:2404.12358, 2024. [find](https://scholar.google.com/scholar?q=Rafael+Rafailov%2C+Joey+Hejna%2C+Ryan+Park%2C+and+Chelsea+Finn%2E+From+r+to+q+%E2%88%97+%3A+Your+language+model+is+secretly+a+q%2Dfunction%2E+arXiv+preprint+arXiv%3A2404%2E12358%2C+2024%2E)
21. Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. [Weak-to-strong generalization: Eliciting strong capabilities with weak supervision](https://md.research.elamur.ai/free/en/papers/burns-2023-weak-strong-generalization-eliciting-strong/). arXiv preprint arXiv:2312.09390, 2023.
22. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. [Learning word vectors for sentiment analysis](https://md.research.elamur.ai/free/en/papers/maas-2011-learning-word-vectors-sentiment-analysis/). In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon, USA,…
23. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+Language+models+are+unsupervised+multitask+learners%2E)
24. AI@Meta. Llama 3 model card. 2024. [find](https://scholar.google.com/scholar?q=AI%40Meta%2E+Llama+3+model+card%2E+2024%2E)
25. Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024. [find](https://scholar.google.com/scholar?q=Yann+Dubois%2C+Bal%C3%A1zs+Galambosi%2C+Percy+Liang%2C+and+Tatsunori+B+Hashimoto%2E+Length%2Dcontrolled+alpacaeval%3A+A+simple+way+to+debias+automatic+evaluators%2E+arXiv+preprint+arXiv%3A2404%2E04475%2C+2024%2E)
26. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). Journal of Machine Learning Research, 24(240):1– 113, 2023.
27. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020.
28. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023.
29. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. [Constitutional ai: Harmlessness from ai feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/). arXiv preprint arXiv:2212.08073, 2022.
30. Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao. Beyond one-preference-for-all: Multi-objective direct preference optimization. arXiv preprint arXiv:2310.03708, 2023. — [arXiv:2310.03708](https://arxiv.org/abs/2310.03708) [find](https://scholar.google.com/scholar?q=Zhanhui+Zhou%2C+Jie+Liu%2C+Chao+Yang%2C+Jing+Shao%2C+Yu+Liu%2C+Xiangyu+Yue%2C+Wanli+Ouyang%2C+and+Yu+Qiao%2E+Beyond+one%2Dpreference%2Dfor%2Dall%3A+Multi%2Dobjective+direct+preference+optimization%2E+arXiv+preprint+arXiv%3A2310%2E03)
31. Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Alexandre+Rame%2C+Guillaume+Couairon%2C+Corentin+Dancette%2C+Jean%2DBaptiste+Gaya%2C+Mustafa+Shukor%2C+Laure+Soulier%2C+and+Matthieu+Cord%2E+Rewarded+soups%3A+towards+pareto%2Doptimal+alignment+by+interpolating+weights+f)
32. Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Personalized large language model alignment via post-hoc parameter merging. arXiv preprint arXiv:2310.11564, 2023. [find](https://scholar.google.com/scholar?q=Joel+Jang%2C+Seungone+Kim%2C+Bill+Yuchen+Lin%2C+Yizhong+Wang%2C+Jack+Hessel%2C+Luke+Zettlemoyer%2C+Hannaneh+Hajishirzi%2C+Yejin+Choi%2C+and+Prithviraj+Ammanabrolu%2E+Personalized+soups%3A+Personalized+large+language+mode)
33. Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. [Arithmetic control of llms for diverse user preferences: Directional preference alignment with multi-objective rewards](https://md.research.elamur.ai/free/en/papers/wang-2024-arithmetic-control-llms-diverse-user/). arXiv preprint arXiv:2402.18571, 2024.
34. Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A Smith, and Yejin Choi. [Dexperts: Decoding-time controlled text generation with experts and anti-experts](https://md.research.elamur.ai/free/en/papers/liu-2021-dexperts-decoding-time-controlled-text/). arXiv preprint arXiv:2105.03023, 2021.
35. Zhanhui Zhou, Jie Liu, Zhichen Dong, Jiaheng Liu, Chao Yang, Wanli Ouyang, and Yu Qiao. Emulated disalignment: Safety alignment for large language models may backfire! arXiv preprint arXiv:2402.12343, 2024. [find](https://scholar.google.com/scholar?q=Zhanhui+Zhou%2C+Jie+Liu%2C+Zhichen+Dong%2C+Jiaheng+Liu%2C+Chao+Yang%2C+Wanli+Ouyang%2C+and+Yu+Qiao%2E+Emulated+disalignment%3A+Safety+alignment+for+large+language+models+may+backfire%21+arXiv+preprint+arXiv%3A2402%2E12343%2C)
36. Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. [Slic-hf: Sequence likelihood calibration with human feedback](https://md.research.elamur.ai/free/en/papers/zhao-2023-slic-hf-sequence-likelihood-calibration/). arXiv preprint arXiv:2305.10425, 2023.
37. Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024. [find](https://scholar.google.com/scholar?q=Kawin+Ethayarajh%2C+Winnie+Xu%2C+Niklas+Muennighoff%2C+Dan+Jurafsky%2C+and+Douwe+Kiela%2E+Kto%3A+Model+alignment+as+prospect+theoretic+optimization%2E+arXiv+preprint+arXiv%3A2402%2E01306%2C+2024%2E)
38. Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019. [find](https://scholar.google.com/scholar?q=Xue+Bin+Peng%2C+Aviral+Kumar%2C+Grace+Zhang%2C+and+Sergey+Levine%2E+Advantage%2Dweighted+regression%3A+Simple+and+scalable+off%2Dpolicy+reinforcement+learning%2E+arXiv+preprint+arXiv%3A1910%2E00177%2C+2019%2E)
39. Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020. — [arXiv:2006.09359](https://arxiv.org/abs/2006.09359) [find](https://scholar.google.com/scholar?q=Ashvin+Nair%2C+Abhishek+Gupta%2C+Murtaza+Dalal%2C+and+Sergey+Levine%2E+Awac%3A+Accelerating+online+reinforcement+learning+with+offline+datasets%2E+arXiv+preprint+arXiv%3A2006%2E09359%2C+2020%2E+%E2%80%94+%5BarXiv%3A2006%2E09359%5D%28https)
40. Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433–1438. Chicago, IL, USA, 2008. [find](https://scholar.google.com/scholar?q=Brian+D+Ziebart%2C+Andrew+L+Maas%2C+J+Andrew+Bagnell%2C+Anind+K+Dey%2C+et+al%2E+Maximum+entropy+inverse+reinforcement+learning%2E+In+Aaai%2C+volume+8%2C+pages+1433%E2%80%931438%2E+Chicago%2C+IL%2C+USA%2C+2008%2E)
41. Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv:1805.00909, 2018. — [arXiv:1805.00909](https://arxiv.org/abs/1805.00909) [find](https://scholar.google.com/scholar?q=Sergey+Levine%2E+Reinforcement+learning+and+control+as+probabilistic+inference%3A+Tutorial+and+review%2E+arXiv+preprint+arXiv%3A1805%2E00909%2C+2018%2E+%E2%80%94+%5BarXiv%3A1805%2E00909%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1805%2E00909%29)
42. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787, 2024. [find](https://scholar.google.com/scholar?q=Nathan+Lambert%2C+Valentina+Pyatkin%2C+Jacob+Morrison%2C+LJ+Miranda%2C+Bill+Yuchen+Lin%2C+Khyathi+Chandu%2C+Nouha+Dziri%2C+Sachin+Kumar%2C+Tom+Zick%2C+Yejin+Choi%2C+et+al%2E+Rewardbench%3A+Evaluating+reward+models+for+langua)
43. Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. [find](https://scholar.google.com/scholar?q=Ralph+Allan+Bradley+and+Milton+E%2E+Terry%2E+Rank+analysis+of+incomplete+block+designs%3A+I%2E+the+method+of+paired+comparisons%2E+Biometrika%2C+39%283%2F4%29%3A324%E2%80%93345%2C+1952%2E)
44. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484–489, 2016. [find](https://scholar.google.com/scholar?q=David+Silver%2C+Aja+Huang%2C+Chris+J+Maddison%2C+Arthur+Guez%2C+Laurent+Sifre%2C+George+Van+Den+Driessche%2C+Julian+Schrittwieser%2C+Ioannis+Antonoglou%2C+Veda+Panneershelvam%2C+Marc+Lanctot%2C+et+al%2E+Mastering+the+game+)
45. Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34:1273– 1286, 2021. [find](https://scholar.google.com/scholar?q=Michael+Janner%2C+Qiyang+Li%2C+and+Sergey+Levine%2E+Offline+reinforcement+learning+as+one+big+sequence+modeling+problem%2E+Advances+in+neural+information+processing+systems%2C+34%3A1273%E2%80%93+1286%2C+2021%2E)
46. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.[20050, 2023](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/).
47. Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. [find](https://scholar.google.com/scholar?q=Ralph+Allan+Bradley+and+Milton+E%2E+Terry%2E+Rank+analysis+of+incomplete+block+designs%3A+I%2E+the+method+of+paired+comparisons%2E+Biometrika%2C+39%283%2F4%29%3A324%E2%80%93345%2C+1952%2E)
48. Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Juntao Dai, and Yaodong Yang. Aligner: Achieving efficient alignment through weak-to-strong correction. arXiv preprint arXiv:2402.02416, 2024. [find](https://scholar.google.com/scholar?q=Jiaming+Ji%2C+Boyuan+Chen%2C+Hantao+Lou%2C+Donghai+Hong%2C+Borong+Zhang%2C+Xuehai+Pan%2C+Juntao+Dai%2C+and+Yaodong+Yang%2E+Aligner%3A+Achieving+efficient+alignment+through+weak%2Dto%2Dstrong+correction%2E+arXiv+preprint+arXi)
49. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. [Star: Bootstrapping reasoning with reasoning](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/). Advances in Neural Information Processing Systems, 35:15476–15488, 2022.
50. Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. [Alpacafarm: A simulation framework for methods that learn from human feedback](https://md.research.elamur.ai/free/en/papers/dubois-2023-alpacafarm-simulation-framework-methods-that/). Advances in Neural Information Processing Systems, 36, 2024.
51. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. [Code llama: Open foundation models for code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/). arXiv preprint arXiv:2308.12950, 2023.
52. Wenkai Yang, Shiqi Shen, Guangyao Shen, Zhi Gong, and Yankai Lin. Super (ficial)-alignment: Strong models may deceive weak models in weak-to-strong generalization. arXiv preprint arXiv:2406.11431, 2024. — [arXiv:2406.11431](https://arxiv.org/abs/2406.11431) [find](https://scholar.google.com/scholar?q=Wenkai+Yang%2C+Shiqi+Shen%2C+Guangyao+Shen%2C+Zhi+Gong%2C+and+Yankai+Lin%2E+Super+%28ficial%29%2Dalignment%3A+Strong+models+may+deceive+weak+models+in+weak%2Dto%2Dstrong+generalization%2E+arXiv+preprint+arXiv%3A2406%2E11431%2C+202)
53. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. [Learning word vectors for sentiment analysis](https://md.research.elamur.ai/free/en/papers/maas-2011-learning-word-vectors-sentiment-analysis/). In Dekang Lin, Yuji Matsumoto, and Rada Mihalcea, editors, Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Lang…
54. Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. [Camels in a changing climate: Enhancing lm adaptation with tulu 2](https://md.research.elamur.ai/free/en/papers/ivison-2023-camels-changing-climate-enhancing-lm/). arXiv preprint arXiv:2311.10702, 2023.
55. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377, 2023. [find](https://scholar.google.com/scholar?q=Ganqu+Cui%2C+Lifan+Yuan%2C+Ning+Ding%2C+Guanming+Yao%2C+Wei+Zhu%2C+Yuan+Ni%2C+Guotong+Xie%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+Ultrafeedback%3A+Boosting+language+models+with+high%2Dquality+feedback%2E+arXiv+preprint+arXiv%3A23)
56. Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7b: Improving llm helpfulness & harmlessness with rlaif, November 2023. [find](https://scholar.google.com/scholar?q=Banghua+Zhu%2C+Evan+Frick%2C+Tianhao+Wu%2C+Hanlin+Zhu%2C+and+Jiantao+Jiao%2E+Starling%2D7b%3A+Improving+llm+helpfulness+%26+harmlessness+with+rlaif%2C+November+2023%2E)
57. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2C+2017%2E)

*Updated 1 November 2026*