---
title: "From Explicit to Implicit — paper by Wang et al. (2026)"
description: "Aligned LLMs often behave well only when the prompt spells out the desired preferences, such as tone, concision or safety. 48 references."
authors: ["Binrui Wang", "Yongping Du", "Yu Pei", "Zikai Wang"]
year: 2026
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/2026.tacl-1.48/"
doi: "10.1162/tacl.a.697"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2026-explicit-implicit-theoretical-framework-transfer/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2026-explicit-implicit-theoretical-framework-transfer/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/30/) / From Explicit to Implicit: A Theoretical Framework and Transfer Method for Preference Internalization in Language Models

# From Explicit to Implicit: A Theoretical Framework and Transfer Method for Preference Internalization in Language Models

**Authors:** Binrui Wang, Yongping Du, Yu Pei and Zikai Wang · **Year:** 2026 · **Venue:** [Transactions of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/tacl/) ([Transactions of the Association for Computational Linguistics (TACL) 2026](https://md.research.elamur.ai/free/en/venues/tacl/2026/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.tacl-1.48/) · [PDF](https://aclanthology.org/2026.tacl-1.48.pdf) · [DOI 10.1162/tacl.a.697](https://doi.org/10.1162/tacl.a.697) · [OpenAlex](https://openalex.org/W7165059800)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Aligned LLMs often behave well only when the prompt spells out the desired preferences, such as tone, concision or safety. When those instructions are missing, quality becomes inconsistent, and users have to keep restating their wishes. The authors call this the explicit-implicit gap. They formalize closing it as a transfer problem: make the instruction-free distribution match the instruction-conditioned one. They also provide a theoretical framework that specifies when a model can successfully recognize preferences, stating conditions that are both required and enough for this to happen.

Their method, Two-Stage Progressive Preference Transfer (TSPPT), works in two steps:
- **Stage 1:** It learns soft-prompt preference representations from pairwise data, capturing why one response beats another.
- **Stage 2:** It freezes these representations and distills their effect into the model weights. This uses an asymmetric margin transfer loss with KL regularization to protect general skills.

Experiments covered Qwen2.5, Qwen3, Llama-3.2 and DeepSeek-R1-Distill, trained on UltraFeedback and HelpSteer. TSPPT outperformed DPO, IPO, ORPO, RLHF and GRPO. It also behaved nearly the same with and without explicit instructions. This matters for building assistants that follow user preferences by default.

## Key points
- **Main results:** On UltraFeedback, TSPPT reaches a 79.2% win rate, compared with 59.2–67.6% for baselines. On MT-Bench it scores 7.86, against 7.34 for the strongest baseline. Implicit social bias drops to 0.165, compared with 0.185–0.325 for baselines.
- **Training metrics:** The authors propose three metrics to monitor training: preference margin score (PMS), preference consistency rate (PCR) and transfer gap score (TGS).
- **Metric validity:** Higher PMS and PCR, and lower TGS, correlate with higher win rates and lower implicit bias. Loss effects on win rate, MMLU and implicit bias are significant (p \< 0.01). Explicit bias is unaffected.
- **Ablations and settings:** Learned soft prompts outperform random embeddings, averaged preference-vocabulary embeddings and hard prompts. The best hyperparameters are α = 1.0, β = 0.05 and λ = 0.1.
- **Limitation:** The internalized preferences reflect helpfulness, safety and staying on topic, so stylistic or creative diversity gets less weight.
- **Code:** github.com/BiinRw/E2I-Pref

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Beijing University of Technology](https://md.research.elamur.ai/free/en/universities/beijing-university-of-technology/) — University, Beijing, China · 13 works · 1,910 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Binrui Wang, Yongping Du, Yu Pei and Zikai Wang. 2026. From Explicit to Implicit: A Theoretical Framework and Transfer Method for Preference Internalization in Language Models. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{wang2026explicit,
  title = {{From Explicit to Implicit: A Theoretical Framework and Transfer Method for Preference Internalization in Language Models}},
  author = {Binrui Wang, Yongping Du, Yu Pei and Zikai Wang},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.1162/tacl.a.697},
  url = {https://doi.org/10.1162/tacl.a.697}
}
```

- [← previous From Emotion to Expression: Theoretical Foundations and Resources for Fear Speech](https://md.research.elamur.ai/free/en/papers/shankaran-2026-emotion-expression-theoretical-foundations-resources/)
- [next → From Fake to Real: Mitigating Out-of-Distribution Bias in In-Context Learning via Feedback Supervision from Large Language Models](https://md.research.elamur.ai/free/en/papers/song-2026-fake-real-mitigating-out-distribution/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/30/)

## References

48 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. 2024. [A General Theoretical Paradigm to Understand Learning from Human Preferences](https://md.research.elamur.ai/free/en/papers/azar-2023-general-theoretical-paradigm-understand-learning/). Proceedings of The 27th International Conference on Artificial Intelligence and Statistics,…
2. Xuechunzi Bai, Angelina Wang, Ilia Sucholutsky, and Thomas L. Griffiths. 2024. Measuring Implicit Bias in Explicitly Unbiased Large Language Models. In NeurIPS 2024 Workshop on Behavioral Machine Learning. [find](https://scholar.google.com/scholar?q=Xuechunzi+Bai%2C+Angelina+Wang%2C+Ilia+Sucholutsky%2C+and+Thomas+L%2E+Griffiths%2E+2024%2E+Measuring+Implicit+Bias+in+Explicitly+Unbiased+Large+Language+Models%2E+In+NeurIPS+2024+Workshop+on+Behavioral+Machine+Lear)
3. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022. [Constitutional ai: Harmlessness from ai feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/). arXiv preprint, arXiv:2212.08073v1.
4. Bowen Cao, Deng Cai, Zhisong Zhang, Yuexian Zou, and Wai Lam. 2024. On the Worst Prompt Performance of Large Language Models. Advances in Neural Information Processing Systems, 37:69022–69042. [find](https://scholar.google.com/scholar?q=Bowen+Cao%2C+Deng+Cai%2C+Zhisong+Zhang%2C+Yuexian+Zou%2C+and+Wai+Lam%2E+2024%2E+On+the+Worst+Prompt+Performance+of+Large+Language+Models%2E+Advances+in+Neural+Information+Processing+Systems%2C+37%3A69022%E2%80%9369042%2E)
5. Yekun Chai, Shuohuan Wang, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2022. Clip-Tuning: Towards Derivative-free Prompt Learning with a Mixture of Rewards. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 108–117. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yekun+Chai%2C+Shuohuan+Wang%2C+Yu+Sun%2C+Hao+Tian%2C+Hua+Wu%2C+and+Haifeng+Wang%2E+2022%2E+Clip%2DTuning%3A+Towards+Derivative%2Dfree+Prompt+Learning+with+a+Mixture+of+Rewards%2E+In+Findings+of+the+Association+for+Computat)
6. Banghao Chen, Zhaofeng Zhang, Nicolas Langrené, and Shengxin Zhu. 2025. [Unleashing the potential of prompt engineering for large language models](https://md.research.elamur.ai/free/en/papers/chen-2023-unleashing-potential-prompt-engineering-large/). Patterns, 6(6):101260.
7. Myra Cheng, Esin Durmus, and Dan Jurafsky. 2023. [Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models](https://md.research.elamur.ai/free/en/papers/cheng-2023-marked-personas-natural-language-prompts/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, volume 1, pages 1504–1532. Association for Computational Linguistics…
8. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 30. [find](https://scholar.google.com/scholar?q=Paul+F+Christiano%2C+Jan+Leike%2C+Tom+Brown%2C+Miljan+Martic%2C+Shane+Legg%2C+and+Dario+Amodei%2E+2017%2E+Deep+reinforcement+learning+from+human+preferences%2E+Advances+in+Neural+Information+Processing+Systems%2C+30%2E)
9. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023. Ultrafeedback: Boosting language models with high-quality feedback. [find](https://scholar.google.com/scholar?q=Ganqu+Cui%2C+Lifan+Yuan%2C+Ning+Ding%2C+Guanming+Yao%2C+Wei+Zhu%2C+Yuan+Ni%2C+Guotong+Xie%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2023%2E+Ultrafeedback%3A+Boosting+language+models+with+high%2Dquality+feedback%2E)
10. DeepSeek-AI et al. 2025. [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint, arXiv:2501.12948v1.
11. Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas I. Liao, Kamilė Lukošiūtė, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, Dawn Drain, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jackson Kernion, Jamie Kerr, Jared Mueller, Joshua Landau, Kamal Ndousse, Karina Ng… — [The Capacity for Moral Self-Correction in Large Language Models](https://md.research.elamur.ai/free/en/papers/ganguli-2023-capacity-moral-self-correction-large/)
12. Bertram Gawronski. 2019. Six Lessons for a Cogent Science of Implicit Bias and Its Criticism. Perspectives on Psychological Science. [find](https://scholar.google.com/scholar?q=Bertram+Gawronski%2E+2019%2E+Six+Lessons+for+a+Cogent+Science+of+Implicit+Bias+and+Its+Criticism%2E+Perspectives+on+Psychological+Science%2E)
13. Anthony G. Greenwald and Mahzarin R. Banaji. 1995. Implicit social cognition: Attitudes, self-esteem, and stereotypes. Psychological Review, 102(1):4–27. [find](https://scholar.google.com/scholar?q=Anthony+G%2E+Greenwald+and+Mahzarin+R%2E+Banaji%2E+1995%2E+Implicit+social+cognition%3A+Attitudes%2C+self%2Desteem%2C+and+stereotypes%2E+Psychological+Review%2C+102%281%29%3A4%E2%80%9327%2E)
14. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming - The Rise of Code Intelligence. [find](https://scholar.google.com/scholar?q=Daya+Guo%2C+Qihao+Zhu%2C+Dejian+Yang%2C+Zhenda+Xie%2C+Kai+Dong%2C+Wentao+Zhang%2C+Guanting+Chen%2C+Xiao+Bi%2C+Y%2E+Wu%2C+Y%2E+K%2E+Li%2C+Fuli+Luo%2C+Yingfei+Xiong%2C+and+Wenfeng+Liang%2E+2024%2E+DeepSeek%2DCoder%3A+When+the+Large+Language)
15. Jiwoo Hong, Noah Lee, and James Thorne. 2024. ORPO: Monolithic preference optimization without reference model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11170– 11189, Miami, Florida, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jiwoo+Hong%2C+Noah+Lee%2C+and+James+Thorne%2E+2024%2E+ORPO%3A+Monolithic+preference+optimization+without+reference+model%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Natural+Language+Processin)
16. Haozhe Ji, Cheng Lu, Yilin Niu, Pei Ke, Hongning Wang, Jun Zhu, Jie Tang, and Minlie Huang. 2024a. Towards Efficient Exact Optimization of Language Model Alignment. In Forty-First International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Haozhe+Ji%2C+Cheng+Lu%2C+Yilin+Niu%2C+Pei+Ke%2C+Hongning+Wang%2C+Jun+Zhu%2C+Jie+Tang%2C+and+Minlie+Huang%2E+2024a%2E+Towards+Efficient+Exact+Optimization+of+Language+Model+Alignment%2E+In+Forty%2DFirst+International+Confer)
17. Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Juntao Dai, Tianyi Qiu, and Yaodong Yang. 2024b. Aligner: Efficient alignment by learning to correct. In Advances in Neural Information Processing Systems, volume 37, pages 90853– 90890. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Jiaming+Ji%2C+Boyuan+Chen%2C+Hantao+Lou%2C+Donghai+Hong%2C+Borong+Zhang%2C+Xuehai+Pan%2C+Juntao+Dai%2C+Tianyi+Qiu%2C+and+Yaodong+Yang%2E+2024b%2E+Aligner%3A+Efficient+alignment+by+learning+to+correct%2E+In+Advances+in+Neural)
18. Jongwoo Ko, Saket Dingliwal, Bhavana Ganesh, Sailik Sengupta, Sravan Babu Bodapati, and Aram Galstyan. 2025. SeRA: Self-Reviewing and Alignment of LLMs using Implicit Reward Margins. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Jongwoo+Ko%2C+Saket+Dingliwal%2C+Bhavana+Ganesh%2C+Sailik+Sengupta%2C+Sravan+Babu+Bodapati%2C+and+Aram+Galstyan%2E+2025%2E+SeRA%3A+Self%2DReviewing+and+Alignment+of+LLMs+using+Implicit+Reward+Margins%2E+In+The+Thirteenth)
19. Hui-Chi Kuo and Yun-Nung Chen. 2023. Zero-Shot Prompting for Implicit Intent Prediction and Recommendation with Commonsense Reasoning. In Findings of the Association for Computational Linguistics: ACL 2023, pages 249– 258. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Hui%2DChi+Kuo+and+Yun%2DNung+Chen%2E+2023%2E+Zero%2DShot+Prompting+for+Implicit+Intent+Prediction+and+Recommendation+with+Commonsense+Reasoning%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL)
20. Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. 2024. [Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs](https://md.research.elamur.ai/free/en/papers/lai-2024-step-dpo-step-wise-preference/). arXiv preprint, arXiv:2406.18629.
21. Janghwan Lee, Seongmin Park, Sukjin Hong, Minsoo Kim, Du-Seong Chang, and Jungwook Choi. 2024a. Improving Conversational Abilities of Quantized Large Language Models via Direct Preference Alignment. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, volume 1,… [find](https://scholar.google.com/scholar?q=Janghwan+Lee%2C+Seongmin+Park%2C+Sukjin+Hong%2C+Minsoo+Kim%2C+Du%2DSeong+Chang%2C+and+Jungwook+Choi%2E+2024a%2E+Improving+Conversational+Abilities+of+Quantized+Large+Language+Models+via+Direct+Preference+Alignment%2E+I)
22. Sangkyu Lee, Sungdong Kim, Ashkan Yousef-pour, Minjoon Seo, Kang Min Yoo, and Youngjae Yu. 2024b. Aligning large language models by on-policy self-judgment. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, volume 1, pages 11442–11459, Bangkok, Thailand. Ass… [find](https://scholar.google.com/scholar?q=Sangkyu+Lee%2C+Sungdong+Kim%2C+Ashkan+Yousef%2Dpour%2C+Minjoon+Seo%2C+Kang+Min+Yoo%2C+and+Youngjae+Yu%2E+2024b%2E+Aligning+large+language+models+by+on%2Dpolicy+self%2Djudgment%2E+In+Proceedings+of+the+62nd+Annual+Meeting+o)
23. Maria Lerner, Florian Dorner, Elliott Ash, and Naman Goel. 2024. Whose Preferences? Differences in Fairness Preferences and Their Impact on the Fairness of AI Utilizing Human Feedback. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, volume 1, pages 9403–94… [find](https://scholar.google.com/scholar?q=Maria+Lerner%2C+Florian+Dorner%2C+Elliott+Ash%2C+and+Naman+Goel%2E+2024%2E+Whose+Preferences%3F+Differences+in+Fairness+Preferences+and+Their+Impact+on+the+Fairness+of+AI+Utilizing+Human+Feedback%2E+In+Proceedings+)
24. Xiang Lisa Li and Percy Liang. 2021. Prefix-Tuning: Optimizing Continuous Prompts for Generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, volume 1, pages 4582–4597. Associat… [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li+and+Percy+Liang%2E+2021%2E+Prefix%2DTuning%3A+Optimizing+Continuous+Prompts+for+Generation%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+11t)
25. Yong Lin, Skyler Seto, Maartje Ter Hoeve, Katherine Metcalf, Barry-John Theobald, Xuan Wang, Yizhe Zhang, Chen Huang, and Tong Zhang. 2024. On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization. In Findings of the Association for Computationa… [find](https://scholar.google.com/scholar?q=Yong+Lin%2C+Skyler+Seto%2C+Maartje+Ter+Hoeve%2C+Katherine+Metcalf%2C+Barry%2DJohn+Theobald%2C+Xuan+Wang%2C+Yizhe+Zhang%2C+Chen+Huang%2C+and+Tong+Zhang%2E+2024%2E+On+the+Limited+Generalization+Capability+of+the+Implicit+Rew)
26. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/). ACM Computing Surveys, 55:1–35.
27. Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2022. [P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks](https://md.research.elamur.ai/free/en/papers/liu-2022-p-tuning-prompt-tuning-can/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, volume 2, pages 61–68. Association…
28. Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. Simpo: Simple preference optimization with a reference-free reward. In Proceedings of the 38th International Conference on Neural Information Processing Systems, volume 37, pages 124198–124235. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Yu+Meng%2C+Mengzhou+Xia%2C+and+Danqi+Chen%2E+2024%2E+Simpo%3A+Simple+preference+optimization+with+a+reference%2Dfree+reward%2E+In+Proceedings+of+the+38th+International+Conference+on+Neural+Information+Processing+Sy)
29. Brian A. Nosek. 2005. Moderators of the Relationship Between Implicit and Explicit Evaluation. Journal of Experimental Psychology: General, 134(4):565–584. [find](https://scholar.google.com/scholar?q=Brian+A%2E+Nosek%2E+2005%2E+Moderators+of+the+Relationship+Between+Implicit+and+Explicit+Evaluation%2E+Journal+of+Experimental+Psychology%3A+General%2C+134%284%29%3A565%E2%80%93584%2E)
30. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Ja… — [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
31. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Tra… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
32. Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. 2024. Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive. arXiv preprint, arXiv:2402.13228v2. [find](https://scholar.google.com/scholar?q=Arka+Pal%2C+Deep+Karkhanis%2C+Samuel+Dooley%2C+Manley+Roberts%2C+Siddartha+Naidu%2C+and+Colin+White%2E+2024%2E+Smaug%3A+Fixing+Failure+Modes+of+Preference+Optimisation+with+DPO%2DPositive%2E+arXiv+preprint%2C+arXiv%3A2402%2E13)
33. Xianghe Pang, Shuo Tang, Rui Ye, Yuxin Xiong, Bolun Zhang, Yanfeng Wang, and Siheng Chen. 2024. Self-alignment of large language models via monopolylogue-based social scene simulation. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learn… [find](https://scholar.google.com/scholar?q=Xianghe+Pang%2C+Shuo+Tang%2C+Rui+Ye%2C+Yuxin+Xiong%2C+Bolun+Zhang%2C+Yanfeng+Wang%2C+and+Siheng+Chen%2E+2024%2E+Self%2Dalignment+of+large+language+models+via+monopolylogue%2Dbased+social+scene+simulation%2E+In+Proceedings+)
34. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). In Advances in Neural Information Processing Systems, volume 36, pages 53728–53741. Curran Associates, Inc.
35. Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. [Learning To Retrieve Prompts for In-Context Learning](https://md.research.elamur.ai/free/en/papers/rubin-2021-learning-retrieve-prompts-context-learning/). In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2655–2671. Association for Computatio…
36. Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. 2023. [Principle-driven self-alignment of language models from scratch with minimal human supervision](https://md.research.elamur.ai/free/en/papers/sun-2023-principle-driven-self-alignment-language/). In Advances in Neural Information Processing Systems, volume 36, pages 2511–2565. Curran…
37. Megh Thakkar, Quentin Fournier, Matthew Riemer, Pin-Yu Chen, Amal Zouaq, Payel Das, and Sarath Chandar. 2024. A deep dive into the trade-offs of parameter-efficient preference alignment techniques. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, volume 1,… [find](https://scholar.google.com/scholar?q=Megh+Thakkar%2C+Quentin+Fournier%2C+Matthew+Riemer%2C+Pin%2DYu+Chen%2C+Amal+Zouaq%2C+Payel+Das%2C+and+Sarath+Chandar%2E+2024%2E+A+deep+dive+into+the+trade%2Doffs+of+parameter%2Defficient+preference+alignment+techniques%2E+In)
38. Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. 2024a. Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Chaoqi+Wang%2C+Yibo+Jiang%2C+Chenghao+Yang%2C+Han+Liu%2C+and+Yuxin+Chen%2E+2024a%2E+Beyond+Reverse+KL%3A+Generalizing+Direct+Preference+Optimization+with+Diverse+Divergence+Constraints%2E+In+The+Twelfth+International)
39. Jiashuo WANG, Haozhao Wang, Shichao Sun, and Wenjie Li. 2023. Aligning language models with human preferences via a bayesian approach. In Advances in Neural Information Processing Systems, volume 36, pages 49113– 49132. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Jiashuo+WANG%2C+Haozhao+Wang%2C+Shichao+Sun%2C+and+Wenjie+Li%2E+2023%2E+Aligning+language+models+with+human+preferences+via+a+bayesian+approach%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+36%2C+)
40. Tianduo Wang, Shichen Li, and Wei Lu. 2024b. Self-Training with Direct Preference Optimization Improves Chain-of-Thought Reasoning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, volume 1, pages 11917–11928. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tianduo+Wang%2C+Shichen+Li%2C+and+Wei+Lu%2E+2024b%2E+Self%2DTraining+with+Direct+Preference+Optimization+Improves+Chain%2Dof%2DThought+Reasoning%2E+In+Proceedings+of+the+62nd+Annual+Meeting+of+the+Association+for+Com)
41. Junda Wu, Tong Yu, Rui Wang, Zhao Song, Ruiyi Zhang, Handong Zhao, Chaochao Lu, Shuai Li, and Ricardo Henao. 2023. Info-prompt: Information-theoretic soft prompt tuning for natural language understanding. In Advances in Neural Information Processing Systems, volume 36, pages 61060–61084. Curran Asso… [find](https://scholar.google.com/scholar?q=Junda+Wu%2C+Tong+Yu%2C+Rui+Wang%2C+Zhao+Song%2C+Ruiyi+Zhang%2C+Handong+Zhao%2C+Chaochao+Lu%2C+Shuai+Li%2C+and+Ricardo+Henao%2E+2023%2E+Info%2Dprompt%3A+Information%2Dtheoretic+soft+prompt+tuning+for+natural+language+understand)
42. Zhaoxuan Wu, Xiaoqiang Lin, Zhongxiang Dai, Wenyang Hu, Yao Shu, See-Kiong Ng, Patrick Jaillet, and Bryan Kian Hsiang Low. 2024. Prompt optimization with ease? efficient ordering-aware automated selection of exemplars. In Advances in Neural Information Processing Systems, volume 37, pages 122706– 12… [find](https://scholar.google.com/scholar?q=Zhaoxuan+Wu%2C+Xiaoqiang+Lin%2C+Zhongxiang+Dai%2C+Wenyang+Hu%2C+Yao+Shu%2C+See%2DKiong+Ng%2C+Patrick+Jaillet%2C+and+Bryan+Kian+Hsiang+Low%2E+2024%2E+Prompt+optimization+with+ease%3F+efficient+ordering%2Daware+automated+selec)
43. Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. 2024. Token-level direct preference optimization. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 58348–58365. PMLR. [find](https://scholar.google.com/scholar?q=Yongcheng+Zeng%2C+Guoqing+Liu%2C+Weiyu+Ma%2C+Ning+Yang%2C+Haifeng+Zhang%2C+and+Jun+Wang%2E+2024%2E+Token%2Dlevel+direct+preference+optimization%2E+In+Proceedings+of+the+41st+International+Conference+on+Machine+Learning)
44. Yiming Zhang, Shi Feng, and Chenhao Tan. 2022. [Active Example Selection for In-Context Learning](https://md.research.elamur.ai/free/en/papers/zhang-2022-active-example-selection-context-learning/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9134–9148. Association for Computational Linguistics.
45. Yachao Zhao, Bo Wang, Yan Wang, Dongming Zhao, Ruifang He, and Yuexian Hou. 2025. Explicit vs. implicit: Investigating social bias in large language models through self-reflection. In Findings of the Association for Computational Linguistics: ACL 2025, pages 1–12, Vienna, Austria. Association for Co… [find](https://scholar.google.com/scholar?q=Yachao+Zhao%2C+Bo+Wang%2C+Yan+Wang%2C+Dongming+Zhao%2C+Ruifang+He%2C+and+Yuexian+Hou%2E+2025%2E+Explicit+vs%2E+implicit%3A+Investigating+social+bias+in+large+language+models+through+self%2Dreflection%2E+In+Findings+of+the+)
46. Yachao Zhao, Bo Wang, Yan Wang, Dongming Zhao, Xiaojia Jin, Jijun Zhang, Ruifang He, and Yuexian Hou. 2024. A Comparative Study of Explicit and Implicit Gender Biases in Large Language Models via Self-evaluation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics,… [find](https://scholar.google.com/scholar?q=Yachao+Zhao%2C+Bo+Wang%2C+Yan+Wang%2C+Dongming+Zhao%2C+Xiaojia+Jin%2C+Jijun+Zhang%2C+Ruifang+He%2C+and+Yuexian+Hou%2E+2024%2E+A+Comparative+Study+of+Explicit+and+Implicit+Gender+Biases+in+Large+Language+Models+via+Self)
47. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. [Instruction-Following Evaluation for Large Language Models](https://md.research.elamur.ai/free/en/papers/zhou-2023-instruction-following-evaluation-large-language/). arXiv preprint, arXiv:2311.07911v1.
48. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint, arXiv:1909.08093v2. [find](https://scholar.google.com/scholar?q=Daniel+M+Ziegler%2C+Nisan+Stiennon%2C+Jeffrey+Wu%2C+Tom+B+Brown%2C+Alec+Radford%2C+Dario+Amodei%2C+Paul+Christiano%2C+and+Geoffrey+Irving%2E+2019%2E+Fine%2Dtuning+language+models+from+human+preferences%2E+arXiv+preprint%2C+a)

*Updated 1 November 2026*