---
title: "Self-Supervised Alignment with Mutual Information: Learning to"
description: "The authors propose SAMI, an iterative self-supervised finetuning procedure that needs none of them. 42 references, cited by 11 papers here."
authors: ["Jan-Philipp Franken", "Eric Zelikman", "Rafael Rafailov", "Kanishk Gandhi", "Tobias Gerstenberg", "Noah D. Goodman"]
year: 2024
venue: "arXiv:2404.14313"
source: "https://arxiv.org/abs/2404.14313"
doi: "10.48550/arxiv.2404.14313"
arxiv: "2404.14313"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/franken-2024-self-supervised-alignment-mutual-information/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/franken-2024-self-supervised-alignment-mutual-information/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/) / Self-Supervised Alignment with Mutual Information: Learning to Follow Principles without Preference Labels

# Self-Supervised Alignment with Mutual Information: Learning to Follow Principles without Preference Labels

**Authors:** Jan-Philipp Franken, [Eric Zelikman](https://md.research.elamur.ai/free/en/authors/eric-zelikman/), [Rafael Rafailov](https://md.research.elamur.ai/free/en/authors/rafael-rafailov/), [Kanishk Gandhi](https://md.research.elamur.ai/free/en/authors/kanishk-gandhi/), [Tobias Gerstenberg](https://md.research.elamur.ai/free/en/authors/tobias-gerstenberg/) and [Noah D. Goodman](https://md.research.elamur.ai/free/en/authors/noah-d-goodman/) · **Year:** 2024 · **Venue:** [arXiv:2404.14313](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/) ([NeurIPS 2024](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/2024/)) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2404.14313) · [PDF](https://arxiv.org/pdf/2404.14313) · [DOI 10.48550/arxiv.2404.14313](https://doi.org/10.48550/arxiv.2404.14313) · [arXiv 2404.14313](https://arxiv.org/abs/2404.14313) · [OpenAlex](https://openalex.org/W4415797232)

**Citations:** 5 citations (OpenAlex, 2026-10-14) · [11](https://md.research.elamur.ai/free/en/papers/franken-2024-self-supervised-alignment-mutual-information/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Teaching a language model to respect a set of behavioral rules (a "constitution") usually depends on human preference labels, curated demonstrations, or reward models with reinforcement learning. All of these are costly and fragile. The authors propose SAMI, an iterative self-supervised finetuning procedure that needs none of them. A "principle writer" LM drafts principles together with their opposites (antitheses), and these are mixed into sampled constitutions. The target pretrained model then answers dataset queries under several constitutions. Training maximizes a contrastive lower bound on the conditional mutual information between constitutions and the model's own responses given the query. In practice, this is a row- and column-wise cross-entropy that pushes a matrix of normalized log probabilities toward the identity matrix.

Experiments use HH-RLHF (single-turn dialogue) and TL;DR (summarization). SAMI-trained base models beat their starting points and sometimes beat instruction-tuned baselines. Weak models can also write principles that align stronger ones. The work matters because it suggests base models already encode a usable link between stated principles and behavior, which alignment can amplify cheaply.

## Key points
- **Dialogue results:** On HH-RLHF, SAMI-trained mistral-7b reached 66–77% win rates over the base model. On single-turn dialogue it also won 55–57% of the time against mistral-7b-instruct, measured after about three iterations.
- **Weak-to-strong setup:** mistral-7b-instruct wrote the principles used to align mixtral-8x7b, which achieved a 65% win rate on TL;DR. With claude-opus as a strong principle writer, the results were comparable.
- **Scaling and generalization:** llama3-70b was trained on 15 of 20 diverse summarization principles (e.g., pirate style, emojis), with 5 held out. Against the base model it scored up to 60% on seen principles and 63% on held-out ones. With chain-of-thought, these rose to 68% and 67%.
- **Method details:** Labels are an identity matrix over N_c constitutions per query. Each iteration restarts from the base model while increasing the number of batches.
- **Limitations:** Training needs regularization to avoid degenerate output, and the method shows a length bias. The principle writer must give enough contrastive coverage. Experiments cover only dialogue and summarization.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works

## Cite

Cite as: Jan-Philipp Franken, Eric Zelikman, Rafael Rafailov, Kanishk Gandhi, Tobias Gerstenberg and Noah D. Goodman. 2024. Self-Supervised Alignment with Mutual Information: Learning to Follow Principles without Preference Labels. arXiv:2404.14313.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{franken2024self,
  title = {{Self-Supervised Alignment with Mutual Information: Learning to Follow Principles without Preference Labels}},
  author = {Jan-Philipp Franken, Eric Zelikman, Rafael Rafailov, Kanishk Gandhi, Tobias Gerstenberg and Noah D. Goodman},
  year = {2024},
  doi = {10.48550/arxiv.2404.14313},
  eprint = {2404.14313},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.14313}
}
```

- [← previous Rethinking Pragmatics in Large Language Models: Towards Open-Ended Evaluation and Preference Tuning](https://md.research.elamur.ai/free/en/papers/wu-2024-rethinking-pragmatics-large-language-models/)
- [next → Shallow Synthesis of Knowledge in GPT-Generated Texts: A Case Study in Automatic Related Work Composition](https://md.research.elamur.ai/free/en/papers/martin-boyle-2024-shallow-synthesis-knowledge-gpt-generated/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/)

## References

42 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Chinmaya Andukuri, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah D Goodman. Star-gate: Teaching language models to ask clarifying questions. arXiv preprint arXiv:2403.19154, 2024. [find](https://scholar.google.com/scholar?q=Chinmaya+Andukuri%2C+Jan%2DPhilipp+Fr%C3%A4nken%2C+Tobias+Gerstenberg%2C+and+Noah+D+Goodman%2E+Star%2Dgate%3A+Teaching+language+models+to+ask+clarifying+questions%2E+arXiv+preprint+arXiv%3A2403%2E19154%2C+2024%2E)
2. Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 30, 2017. [find](https://scholar.google.com/scholar?q=Thomas+Anthony%2C+Zheng+Tian%2C+and+David+Barber%2E+Thinking+fast+and+slow+with+deep+learning+and+tree+search%2E+Advances+in+neural+information+processing+systems%2C+30%2C+2017%2E)
3. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021. [find](https://scholar.google.com/scholar?q=Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Dawn+Drain%2C+Deep+Ganguli%2C+Tom+Henighan%2C+Andy+Jones%2C+Nicholas+Joseph%2C+Ben+Mann%2C+Nova+DasSarma%2C+et+al%2E+A+general+language+assistant+as+a+laboratory+for+alignment%2E+a)
4. Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. [A general theoretical paradigm to understand learning from human preferences](https://md.research.elamur.ai/free/en/papers/azar-2023-general-theoretical-paradigm-understand-learning/). arXiv preprint arXiv:2310.12036, 2023.
5. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+et+al%2E+Training+a+helpful+and+harmless+assistant+with+reinforcem)
6. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. [Constitutional ai: Harmlessness from ai feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/). arXiv preprint arXiv:2212.08073, 2022.
7. Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. [Weak-to-strong generalization: Eliciting strong capabilities with weak supervision](https://md.research.elamur.ai/free/en/papers/burns-2023-weak-strong-generalization-eliciting-strong/). arXiv preprint arXiv:2312.09390, 2023.
8. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. [find](https://scholar.google.com/scholar?q=Paul+F+Christiano%2C+Jan+Leike%2C+Tom+Brown%2C+Miljan+Martic%2C+Shane+Legg%2C+and+Dario+Amodei%2E+Deep+reinforcement+learning+from+human+preferences%2E+Advances+in+neural+information+processing+systems%2C+30%2C+2017%2E)
9. Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. [Toxicity in chatgpt: Analyzing persona-assigned language models](https://md.research.elamur.ai/free/en/papers/deshpande-2023-toxicity-chatgpt-analyzing-persona-assigned/). arXiv preprint arXiv:2304.05335, 2023.
10. Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. [Alpacafarm: A simulation framework for methods that learn from human feedback](https://md.research.elamur.ai/free/en/papers/dubois-2023-alpacafarm-simulation-framework-methods-that/), 2023.
11. Esin Durmus, Karina Nyugen, Thomas I Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, et al. [Towards measuring the representation of subjective global opinions in language models](https://md.research.elamur.ai/free/en/papers/durmus-2023-towards-measuring-representation-subjective-global/). arXiv preprint arXiv:2306.16388, 2023.
12. Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024. [find](https://scholar.google.com/scholar?q=Kawin+Ethayarajh%2C+Winnie+Xu%2C+Niklas+Muennighoff%2C+Dan+Jurafsky%2C+and+Douwe+Kiela%2E+Kto%3A+Model+alignment+as+prospect+theoretic+optimization%2E+arXiv+preprint+arXiv%3A2402%2E01306%2C+2024%2E)
13. Jan-Philipp Fränken, Sam Kwok, Peixuan Ye, Kanishk Gandhi, Dilip Arumugam, Jared Moore, Alex Tamkin, Tobias Gerstenberg, and Noah D Goodman. Social contract ai: Aligning ai assistants with implicit group norms. arXiv preprint arXiv:2310.17769, 2023. — [arXiv:2310.17769](https://arxiv.org/abs/2310.17769) [find](https://scholar.google.com/scholar?q=Jan%2DPhilipp+Fr%C3%A4nken%2C+Sam+Kwok%2C+Peixuan+Ye%2C+Kanishk+Gandhi%2C+Dilip+Arumugam%2C+Jared+Moore%2C+Alex+Tamkin%2C+Tobias+Gerstenberg%2C+and+Noah+D+Goodman%2E+Social+contract+ai%3A+Aligning+ai+assistants+with+implicit+gr)
14. Herbert P Grice. Logic and conversation. In Speech acts, pages 41–58. Brill, 1975. [find](https://scholar.google.com/scholar?q=Herbert+P+Grice%2E+Logic+and+conversation%2E+In+Speech+acts%2C+pages+41%E2%80%9358%2E+Brill%2C+1975%2E)
15. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+Mistral+7b)
16. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+et+al%2E+Mixtral+of+expe)
17. Sandipan Kundu, Yuntao Bai, Saurav Kadavath, Amanda Askell, Andrew Callahan, Anna Chen, Anna Goldie, Avital Balwit, Azalia Mirhoseini, Brayden McLean, et al. Specific versus general principles for constitutional ai. arXiv preprint arXiv:2310.13798, 2023. [find](https://scholar.google.com/scholar?q=Sandipan+Kundu%2C+Yuntao+Bai%2C+Saurav+Kadavath%2C+Amanda+Askell%2C+Andrew+Callahan%2C+Anna+Chen%2C+Anna+Goldie%2C+Avital+Balwit%2C+Azalia+Mirhoseini%2C+Brayden+McLean%2C+et+al%2E+Specific+versus+general+principles+for+con)
18. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
19. Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267, 2023. [find](https://scholar.google.com/scholar?q=Harrison+Lee%2C+Samrat+Phatale%2C+Hassan+Mansoor%2C+Kellie+Lu%2C+Thomas+Mesnard%2C+Colton+Bishop%2C+Victor+Carbune%2C+and+Abhinav+Rastogi%2E+Rlaif%3A+Scaling+reinforcement+learning+from+human+feedback+with+ai+feedback%2E)
20. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models, 2023. [find](https://scholar.google.com/scholar?q=Xuechen+Li%2C+Tianyi+Zhang%2C+Yann+Dubois%2C+Rohan+Taori%2C+Ishaan+Gulrajani%2C+Carlos+Guestrin%2C+Percy+Liang%2C+and+Tatsunori+B+Hashimoto%2E+Alpacaeval%3A+An+automatic+evaluator+of+instruction%2Dfollowing+models%2C+2023%2E)
21. Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. [The unlocking spell on base llms: Rethinking alignment via in-context learning](https://md.research.elamur.ai/free/en/papers/lin-2023-unlocking-spell-base-llms-rethinking/). In International Conference on Learning Representations, 2024. URL [https://arxiv.org/abs…](https://arxiv.org/abs…) [↗](https://arxiv.org/abs…)
22. Meta. Introducing meta llama 3: The most capable openly available llm to date. [https://www.meta.com/blog/meta-llama-3-introduction](https://www.meta.com/blog/meta-llama-3-introduction), April 2024. Accessed: 2024-05-12. [↗](https://www.meta.com/blog/meta-llama-3-introduction)
23. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. [Representation learning with contrastive predictive coding](https://md.research.elamur.ai/free/en/papers/oord-2018-representation-learning-contrastive-predictive-coding/). arXiv preprint arXiv:1807.03748, 2018.
24. OpenAI. [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774, 2023.
25. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
26. Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. [Disentangling length from quality in direct preference optimization](https://md.research.elamur.ai/free/en/papers/park-2024-disentangling-length-quality-direct-preference/). arXiv preprint arXiv:2403.19159, 2024.
27. Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pages 5171–5180. PMLR, 2019. [find](https://scholar.google.com/scholar?q=Ben+Poole%2C+Sherjil+Ozair%2C+Aaron+Van+Den+Oord%2C+Alex+Alemi%2C+and+George+Tucker%2E+On+variational+bounds+of+mutual+information%2E+In+International+Conference+on+Machine+Learning%2C+pages+5171%E2%80%935180%2E+PMLR%2C+2019%2E)
28. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). In International conference on machine learning, pages 8748–8763. PMLR, 20…
29. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). Advances in Neural Information Processing Systems, 36, 2024.
30. Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022. [find](https://scholar.google.com/scholar?q=Nisan+Stiennon%2C+Long+Ouyang%2C+Jeff+Wu%2C+Daniel+M%2E+Ziegler%2C+Ryan+Lowe%2C+Chelsea+Voss%2C+Alec+Radford%2C+Dario+Amodei%2C+and+Paul+Christiano%2E+Learning+to+summarize+from+human+feedback%2C+2022%2E)
31. Zhiqing Sun, Yikang Shen, Hongxin Zhang, Qinhong Zhou, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Salmon: Self-alignment with principle-following reward models. arXiv preprint arXiv:2310.[05910, 2023](https://md.research.elamur.ai/free/en/papers/sun-2023-salmon-self-alignment-instructable-reward/).
32. Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. [Principle-driven self-alignment of language models from scratch with minimal human supervision](https://md.research.elamur.ai/free/en/papers/sun-2023-principle-driven-self-alignment-language/). Advances in Neural Information Processing Systems, 36, 2024.
33. Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint arXiv:2402.05749, 2024. — [arXiv:2402.05749](https://arxiv.org/abs/2402.05749) [find](https://scholar.google.com/scholar?q=Yunhao+Tang%2C+Zhaohan+Daniel+Guo%2C+Zeyu+Zheng%2C+Daniele+Calandriello%2C+R%C3%A9mi+Munos%2C+Mark+Rowland%2C+Pierre+Harvey+Richemond%2C+Michal+Valko%2C+Bernardo+%C3%81vila+Pires%2C+and+Bilal+Piot%2E+Generalized+preference+optimiz)
34. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023.
35. Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926, 2023. [find](https://scholar.google.com/scholar?q=Peiyi+Wang%2C+Lei+Li%2C+Liang+Chen%2C+Dawei+Zhu%2C+Binghuai+Lin%2C+Yunbo+Cao%2C+Qi+Liu%2C+Tianyu+Liu%2C+and+Zhifang+Sui%2E+Large+language+models+are+not+fair+evaluators%2E+arXiv+preprint+arXiv%3A2305%2E17926%2C+2023%2E)
36. Joseph M Williams and Ira Bruce Nadel. Style: Ten lessons in clarity and grace. Scott, Foresman Glenview, IL, 1989. [find](https://scholar.google.com/scholar?q=Joseph+M+Williams+and+Ira+Bruce+Nadel%2E+Style%3A+Ten+lessons+in+clarity+and+grace%2E+Scott%2C+Foresman+Glenview%2C+IL%2C+1989%2E)
37. Kevin Yang, Dan Klein, Asli Celikyilmaz, Nanyun Peng, and Yuandong Tian. Rlcd: Reinforcement learning from contrast distillation for language model alignment. arXiv preprint arXiv:2307.12950, 2023. [find](https://scholar.google.com/scholar?q=Kevin+Yang%2C+Dan+Klein%2C+Asli+Celikyilmaz%2C+Nanyun+Peng%2C+and+Yuandong+Tian%2E+Rlcd%3A+Reinforcement+learning+from+contrast+distillation+for+language+model+alignment%2E+arXiv+preprint+arXiv%3A2307%2E12950%2C+2023%2E)
38. Yueqin Yin, Zhendong Wang, Yi Gu, Hai Huang, Weizhu Chen, and Mingyuan Zhou. Relative preference optimization: Enhancing llm alignment through contrasting responses across identical and diverse prompts. arXiv preprint arXiv:2402.10958, 2024. — [arXiv:2402.10958](https://arxiv.org/abs/2402.10958) [find](https://scholar.google.com/scholar?q=Yueqin+Yin%2C+Zhendong+Wang%2C+Yi+Gu%2C+Hai+Huang%2C+Weizhu+Chen%2C+and+Mingyuan+Zhou%2E+Relative+preference+optimization%3A+Enhancing+llm+alignment+through+contrasting+responses+across+identical+and+diverse+prompt)
39. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. [Star: Bootstrapping reasoning with reasoning](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/). Advances in Neural Information Processing Systems, 35:15476–15488, 2022.
40. Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. [Slic-hf: Sequence likelihood calibration with human feedback](https://md.research.elamur.ai/free/en/papers/zhao-2023-slic-hf-sequence-likelihood-calibration/). arXiv preprint arXiv:2305.10425, 2023.
41. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Advances in Neural Information Processing Systems, 36, 2024.
42. Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Chunting+Zhou%2C+Pengfei+Liu%2C+Puxin+Xu%2C+Srinivasan+Iyer%2C+Jiao+Sun%2C+Yuning+Mao%2C+Xuezhe+Ma%2C+Avia+Efrat%2C+Ping+Yu%2C+Lili+Yu%2C+et+al%2E+Lima%3A+Less+is+more+for+alignment%2E+Advances+in+Neural+Information+Processing)

*Updated 30 October 2026*