---
title: "Aligning Large Language Models via Fully Self-Synthetic Data"
description: "The authors propose Self-Alignment Optimization (SAO), in which a single model produces all of its… 65 references, cited by 2 papers here."
authors: ["Shangjian Yin", "Zhepei Wei", "Xinyu Zhu", "Wei-Lin Chen", "Yu Meng"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.1595/"
doi: "10.18653/v1/2026.acl-long.1595"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yin-2026-aligning-large-language-models-fully/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yin-2026-aligning-large-language-models-fully/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/6/) / Aligning Large Language Models via Fully Self-Synthetic Data

# Aligning Large Language Models via Fully Self-Synthetic Data

**Authors:** [Shangjian Yin](https://md.research.elamur.ai/free/en/authors/shangjian-yin/), [Zhepei Wei](https://md.research.elamur.ai/free/en/authors/zhepei-wei/), [Xinyu Zhu](https://md.research.elamur.ai/free/en/authors/xinyu-zhu/), [Wei-Lin Chen](https://md.research.elamur.ai/free/en/authors/wei-lin-chen/) and [Yu Meng](https://md.research.elamur.ai/free/en/authors/yu-meng/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.1595/) · [PDF](https://aclanthology.org/2026.acl-long.1595.pdf) · [DOI 10.18653/v1/2026.acl-long.1595](https://doi.org/10.18653/v1/2026.acl-long.1595) · [OpenAlex](https://openalex.org/W4415316359)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [2](https://md.research.elamur.ai/free/en/papers/yin-2026-aligning-large-language-models-fully/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Preference alignment of LLMs usually depends on costly inputs: human annotations, or AI feedback from proprietary models such as GPT-4 or separately trained reward models. Earlier self-improvement methods, including Self-Rewarding and SPPO, still use some external prompts, human seed data or reward models. The authors propose Self-Alignment Optimization (SAO), in which a single model produces all of its own training data. The model takes on personas drawn from Persona Hub and writes one user query per persona. It then samples two answers to each query, judges which answer is better, and is fine-tuned on the resulting pairs with SimPO. Applied to Gemma-2-9B-it, SAO clearly raises scores on chat benchmarks (AlpacaEval 2.0, MT-Bench, Arena-Hard). It also leaves performance on objective tasks unchanged or slightly better, whereas training on UltraFeedback reduced it. The analysis shows that the model's skill as a judge, more than its skill at generating answers, is what drives the gains. SAO therefore offers an inexpensive, scalable route to alignment, provided the base model is a capable evaluator.

## Key points
- **Chat gains for Gemma-2-9B-it:**
  - AlpacaEval 2.0 LC rises by 18.1 points and WR by 27.9 points.
  - MT-Bench improves from 8.41 to 8.66.
  - Arena-Hard WR improves from 40.8% to 54.3%.
- **General capability is preserved:** the Open LLM Leaderboard average moves from 74.28 to 74.41. The same SimPO recipe trained on UltraFeedback drops it to 70.38.
- **Synthetic prompts beat curated ones:** persona-generated prompts give about 16.46% higher WR than UltraFeedback prompts under the same pipeline. Self-judgment also outperforms ArmoRM-Llama3-8B-v0.1 and GPT-4o feedback.
- **The judge matters more than the generator:**
  - With Gemma generating and Gemma judging, the model reaches LC 70.28% and WR 74.06%.
  - With Gemma generating and LLaMA-3-8B-instruct judging, it reaches only LC 58.39% and WR 34.35%.
  - The gains hold when LLaMA-3.3-70B, LLaMA-3.1-70B or Qwen-2.5-72B serve as evaluators.
- **Small models give mixed results:**
  - Gemma-2-2B-it improves, with LC rising from 38.08 to 46.18.
  - Llama-3.2-3B-Instruct degrades, with LC falling from 21.87 to 20.08.
  - The authors attribute this to weaker self-judging ability in some small models.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Virginia](https://md.research.elamur.ai/free/en/universities/university-of-virginia/) — University, Charlottesville, United States · 36 works · 9,730 citations
- [University of California, Riverside](https://md.research.elamur.ai/free/en/universities/university-of-california-riverside/) — University, Riverside, United States · 7 works · 339 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Shangjian Yin, Zhepei Wei, Xinyu Zhu, Wei-Lin Chen and Yu Meng. 2026. Aligning Large Language Models via Fully Self-Synthetic Data. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yin2026aligning,
  title = {{Aligning Large Language Models via Fully Self-Synthetic Data}},
  author = {Shangjian Yin, Zhepei Wei, Xinyu Zhu, Wei-Lin Chen and Yu Meng},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1595},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1595}
}
```

- [← previous AlexandriaX 2026: The First Shared Task on Dialectal Arabic Machine Translation](https://md.research.elamur.ai/free/en/papers/mekki-2026-alexandriax-2026-first-shared-task/)
- [next → AMIYA Shared Task: Arabic Modeling In Your Accent at VarDial 2026](https://md.research.elamur.ai/free/en/papers/robinson-2026-amiya-shared-task-arabic-modeling/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/6/)

## References

65 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, Jiayin Zhang, Juanzi Li, and Lei Hou. 2023. Benchmarking foundation models with language-model-as-an-examiner. In Thirty-seventh Conference on Neural Information Processing Systems Data… [find](https://scholar.google.com/scholar?q=Yushi+Bai%2C+Jiahao+Ying%2C+Yixin+Cao%2C+Xin+Lv%2C+Yuze+He%2C+Xiaozhi+Wang%2C+Jifan+Yu%2C+Kaisheng+Zeng%2C+Yijia+Xiao%2C+Haozhe+Lyu%2C+Jiayin+Zhang%2C+Juanzi+Li%2C+and+Lei+Hou%2E+2023%2E+Benchmarking+foundation+models+with+langu)
2. Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. 2023. Open LLM leaderboard. [https://huggingface.co/spaces/](https://huggingface.co/spaces/) HuggingFaceH4/open_llm_leaderboard. [↗](https://huggingface.co/spaces/)
3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
4. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. [Sparks of artificial general intelligence: Early experiments with gpt-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/). arXiv preprint arXiv:2303.12712.
5. Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. 2024a. [AlpaGasus: Training a better alpaca with fewer data](https://md.research.elamur.ai/free/en/papers/chen-2023-alpagasus-training-better-alpaca-fewer/). In The Twelfth International Conference on Learning Representations.
6. Wei-Lin Chen, Zhepei Wei, Xinyu Zhu, Shi Feng, and Yu Meng. 2025. Do LLM evaluators prefer themselves for a reason? arXiv preprint arXiv:2504.03846. — [arXiv:2504.03846](https://arxiv.org/abs/2504.03846) [find](https://scholar.google.com/scholar?q=Wei%2DLin+Chen%2C+Zhepei+Wei%2C+Xinyu+Zhu%2C+Shi+Feng%2C+and+Yu+Meng%2E+2025%2E+Do+LLM+evaluators+prefer+themselves+for+a+reason%3F+arXiv+preprint+arXiv%3A2504%2E03846%2E+%E2%80%94+%5BarXiv%3A2504%2E03846%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2504%2E038)
7. Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. 2024b. Self-play fine-tuning converts weak language models to strong language models. [find](https://scholar.google.com/scholar?q=Zixiang+Chen%2C+Yihe+Deng%2C+Huizhuo+Yuan%2C+Kaixuan+Ji%2C+and+Quanquan+Gu%2E+2024b%2E+Self%2Dplay+fine%2Dtuning+converts+weak+language+models+to+strong+language+models%2E)
8. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing GPT-4 with 90%* chatgpt quality. [find](https://scholar.google.com/scholar?q=Wei%2DLin+Chiang%2C+Zhuohan+Li%2C+Zi+Lin%2C+Ying+Sheng%2C+Zhanghao+Wu%2C+Hao+Zhang%2C+Lianmin+Zheng%2C+Siyuan+Zhuang%2C+Yonghao+Zhuang%2C+Joseph+E%2E+Gonzalez%2C+Ion+Stoica%2C+and+Eric+P%2E+Xing%2E+2023%2E+Vicuna%3A+An+open%2Dsource+cha)
9. Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023. [Llm-blender: Ensembling large language models with pairwise ranking and generative fusion](https://md.research.elamur.ai/free/en/papers/jiang-2023-llm-blender-ensembling-large-language/).
10. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30. [find](https://scholar.google.com/scholar?q=Paul+F+Christiano%2C+Jan+Leike%2C+Tom+Brown%2C+Miljan+Martic%2C+Shane+Legg%2C+and+Dario+Amodei%2E+2017%2E+Deep+reinforcement+learning+from+human+preferences%2E+Advances+in+neural+information+processing+systems%2C+30%2E)
11. Martin Josifoski, Marija Sakota, Maxime Peyrard, and Robert West. 2023. Exploiting asymmetry for synthetic training data generation: Synthie and the case of information extraction. arXiv preprint arXiv:2303.04132. [find](https://scholar.google.com/scholar?q=Martin+Josifoski%2C+Marija+Sakota%2C+Maxime+Peyrard%2C+and+Robert+West%2E+2023%2E+Exploiting+asymmetry+for+synthetic+training+data+generation%3A+Synthie+and+the+case+of+information+extraction%2E+arXiv+preprint+arXi)
12. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv e-prints, pages arXiv–2210. [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Eric+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+et+al%2E+2022%2E+Scaling+instruction%2Dfinetuned+language+models%2E+arXi)
13. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
14. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2024. Ultrafeedback: Boosting language models with scaled ai feedback. [find](https://scholar.google.com/scholar?q=Ganqu+Cui%2C+Lifan+Yuan%2C+Ning+Ding%2C+Guanming+Yao%2C+Bingxiang+He%2C+Wei+Zhu%2C+Yuan+Ni%2C+Guotong+Xie%2C+Ruobing+Xie%2C+Yankai+Lin%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2024%2E+Ultrafeedback%3A+Boosting+language+models+with+s)
15. Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, Marcus Hutter, and Joel Veness. 2024. [Language modeling is compression](https://md.research.elamur.ai/free/en/papers/deletang-2023-language-modeling-compression/).
16. Yihe Deng, Weitong Zhang, Zixiang Chen, and Quanquan Gu. 2023. Rephrase and respond: Let large language models ask better questions for themselves. arXiv preprint arXiv:2311.04205. [find](https://scholar.google.com/scholar?q=Yihe+Deng%2C+Weitong+Zhang%2C+Zixiang+Chen%2C+and+Quanquan+Gu%2E+2023%2E+Rephrase+and+respond%3A+Let+large+language+models+ask+better+questions+for+themselves%2E+arXiv+preprint+arXiv%3A2311%2E04205%2E)
17. Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2024. [Alpacafarm: A simulation framework for methods that learn from human feedback](https://md.research.elamur.ai/free/en/papers/dubois-2023-alpacafarm-simulation-framework-methods-that/).
18. Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André Martins, Graham Neubig, Ankush Garg, Jonathan Clark, Markus Freitag, and Orhan Firat. 2023. [The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation](https://md.research.elamur.ai/free/en/papers/fernandes-2023-devil-is-errors-leveraging-large/). In Proceedings of the Eig…
19. Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. 2024a. [Scaling synthetic data creation with 1,000,000,000 personas](https://md.research.elamur.ai/free/en/papers/ge-2024-scaling-synthetic-data-creation-1/). arXiv preprint arXiv:2406.20094.
20. Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. 2024b. In-context autoencoder for context compression in a large language model. [find](https://scholar.google.com/scholar?q=Tao+Ge%2C+Jing+Hu%2C+Lei+Wang%2C+Xun+Wang%2C+Si%2DQing+Chen%2C+and+Furu+Wei%2E+2024b%2E+In%2Dcontext+autoencoder+for+context+compression+in+a+large+language+model%2E)
21. Jiwoo Hong, Noah Lee, and James Thorne. 2024. Orpo: Monolithic preference optimization without reference model. [find](https://scholar.google.com/scholar?q=Jiwoo+Hong%2C+Noah+Lee%2C+and+James+Thorne%2E+2024%2E+Orpo%3A+Monolithic+preference+optimization+without+reference+model%2E)
22. Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al. 2023. [Prometheus: Inducing fine-grained evaluation capability in language models](https://md.research.elamur.ai/free/en/papers/kim-2023-prometheus-inducing-fine-grained-evaluation/). arXiv preprint arXiv:2310.08491.
23. Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. 2023. RLAIF: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267. [find](https://scholar.google.com/scholar?q=Harrison+Lee%2C+Samrat+Phatale%2C+Hassan+Mansoor%2C+Kellie+Lu%2C+Thomas+Mesnard%2C+Colton+Bishop%2C+Victor+Carbune%2C+and+Abhinav+Rastogi%2E+2023%2E+RLAIF%3A+Scaling+reinforcement+learning+from+human+feedback+with+ai+fee)
24. Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. 2024. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. [find](https://scholar.google.com/scholar?q=Harrison+Lee%2C+Samrat+Phatale%2C+Hassan+Mansoor%2C+Thomas+Mesnard%2C+Johan+Ferret%2C+Kellie+Lu%2C+Colton+Bishop%2C+Ethan+Hall%2C+Victor+Carbune%2C+Abhinav+Rastogi%2C+and+Sushant+Prakash%2E+2024%2E+Rlaif+vs%2E+rlhf%3A+Scaling+re)
25. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. 2024a. From live data to high-quality benchmarks: The Arena-Hard pipeline. [find](https://scholar.google.com/scholar?q=Tianle+Li%2C+Wei%2DLin+Chiang%2C+Evan+Frick%2C+Lisa+Dunlap%2C+Banghua+Zhu%2C+Joseph+E%2E+Gonzalez%2C+and+Ion+Stoica%2E+2024a%2E+From+live+data+to+high%2Dquality+benchmarks%3A+The+Arena%2DHard+pipeline%2E)
26. Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Omer Levy, Jason Weston, and Mike Lewis. 2024b. Self-alignment with instruction back-translation. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Xian+Li%2C+Ping+Yu%2C+Chunting+Zhou%2C+Timo+Schick%2C+Luke+Zettlemoyer%2C+Omer+Levy%2C+Jason+Weston%2C+and+Mike+Lewis%2E+2024b%2E+Self%2Dalignment+with+instruction+back%2Dtranslation%2E+In+The+Twelfth+International+Conferenc)
27. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023a. AlpacaEval: An automatic evaluator of instruction-following models. [https://github.com/tatsu-lab/alpaca_eval](https://github.com/tatsu-lab/alpaca_eval). [↗](https://github.com/tatsu-lab/alpaca_eval)
28. Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023b. [Textbooks are all you need ii: phi-1.5 technical report](https://md.research.elamur.ai/free/en/papers/li-2023-textbooks-all-need-ii-phi/).
29. Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, Hanze Dong, Renjie Pi, Han Zhao, Nan Jiang, Heng Ji, Yuan Yao, and Tong Zhang. 2024. [Mitigating the alignment tax of rlhf](https://md.research.elamur.ai/free/en/papers/lin-2024-mitigating-alignment-tax-rlhf/).
30. Bingbin Liu, Sebastien Bubeck, Ronen Eldan, Janardhan Kulkarni, Yuanzhi Li, Anh Nguyen, Rachel Ward, and Yi Zhang. 2023. Tinygsm: achieving\> 80% on gsm8k with small language models. arXiv preprint arXiv:2312.09241. [find](https://scholar.google.com/scholar?q=Bingbin+Liu%2C+Sebastien+Bubeck%2C+Ronen+Eldan%2C+Janardhan+Kulkarni%2C+Yuanzhi+Li%2C+Anh+Nguyen%2C+Rachel+Ward%2C+and+Yi+Zhang%2E+2023%2E+Tinygsm%3A+achieving%5C%3E+80%25+on+gsm8k+with+small+language+models%2E+arXiv+preprint+ar)
31. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jian-guang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023. [Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct](https://md.research.elamur.ai/free/en/papers/luo-2023-wizardmath-empowering-mathematical-reasoning-large/). arXiv preprint arXiv:2308.09583.
32. Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. Simpo: Simple preference optimization with a reference-free reward. [find](https://scholar.google.com/scholar?q=Yu+Meng%2C+Mengzhou+Xia%2C+and+Danqi+Chen%2E+2024%2E+Simpo%3A+Simple+preference+optimization+with+a+reference%2Dfree+reward%2E)
33. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2021. Cross-task generalization via natural language crowdsourcing instructions. arXiv preprint arXiv:2104.08773. [find](https://scholar.google.com/scholar?q=Swaroop+Mishra%2C+Daniel+Khashabi%2C+Chitta+Baral%2C+and+Hannaneh+Hajishirzi%2E+2021%2E+Cross%2Dtask+generalization+via+natural+language+crowdsourcing+instructions%2E+arXiv+preprint+arXiv%3A2104%2E08773%2E)
34. OpenAI. 2023. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/).
35. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744.
36. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. [Lamda: Language models for dialog applications](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/). arXiv preprint arXiv:2201.08239.
37. Yu-Min Tseng, Yu-Chao Huang, Teng-Yun Hsiao, Wei-Lin Chen, Chao-Wei Huang, Yu Meng, and Yun-Nung Chen. 2024. [Two tales of persona in llms: A survey of role-playing and personalization](https://md.research.elamur.ai/free/en/papers/tseng-2024-two-tales-persona-llms-survey/).
38. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. 2023. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944. [find](https://scholar.google.com/scholar?q=Lewis+Tunstall%2C+Edward+Beeching%2C+Nathan+Lambert%2C+Nazneen+Rajani%2C+Kashif+Rasul%2C+Younes+Belkada%2C+Shengyi+Huang%2C+Leandro+von+Werra%2C+Cl%C3%A9mentine+Fourrier%2C+Nathan+Habib%2C+et+al%2E+2023%2E+Zephyr%3A+Direct+distilla)
39. Archiki Prasad, Elias Stengel-Eskin, and Mohit Bansal. 2023. Rephrase, augment, reason: Visual grounding of questions for vision-language models. arXiv preprint arXiv:2310.05861. [find](https://scholar.google.com/scholar?q=Archiki+Prasad%2C+Elias+Stengel%2DEskin%2C+and+Mohit+Bansal%2E+2023%2E+Rephrase%2C+augment%2C+reason%3A+Visual+grounding+of+questions+for+vision%2Dlanguage+models%2E+arXiv+preprint+arXiv%3A2310%2E05861%2E)
40. Sanh Victor, Webson Albert, Raffel Colin, Bach Stephen, Sutawika Lintang, Alyafeai Zaid, Chaffin Antoine, Stiegler Arnaud, Raja Arun, Dey Manan, et al. 2022. [Multitask prompted training enables zero-shot task generalization](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/). In International Conference on Learning Representations.
41. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
42. Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. 2024. [Interpretable preferences via multi-objective reward modeling and mixture-of-experts](https://md.research.elamur.ai/free/en/papers/wang-2024-interpretable-preferences-multi-objective-reward/). arXiv preprint arXiv:2406.12845.
43. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). In NeurIPS.
44. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in Neural Information Processing Systems, 35:24824–24837.
45. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. [Code llama: Open foundation models for code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/). arXiv preprint arXiv:2308.12950.
46. Swarnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li. 2023. Branch-solve-merge improves large language model evaluation and generation. arXiv preprint arXiv:2310.15123. [find](https://scholar.google.com/scholar?q=Swarnadeep+Saha%2C+Omer+Levy%2C+Asli+Celikyilmaz%2C+Mohit+Bansal%2C+Jason+Weston%2C+and+Xian+Li%2E+2023%2E+Branch%2Dsolve%2Dmerge+improves+large+language+model+evaluation+and+generation%2E+arXiv+preprint+arXiv%3A2310%2E15123)
47. Arthur L Samuel. 2000. Some studies in machine learning using the game of checkers. IBM Journal of research and development, 44(1.2):206–226. [find](https://scholar.google.com/scholar?q=Arthur+L+Samuel%2E+2000%2E+Some+studies+in+machine+learning+using+the+game+of+checkers%2E+IBM+Journal+of+research+and+development%2C+44%281%2E2%29%3A206%E2%80%93226%2E)
48. Jiaming Shen, Ran Xu, Yennie Jun, Zhen Qin, Tianqi Liu, Carl Yang, Yi Liang, Simon Baumgartner, and Michael Bendersky. 2024. [Boosting reward model with preference-conditional multi-aspect synthetic data generation](https://md.research.elamur.ai/free/en/papers/shen-2024-boosting-reward-model-preference-conditional/).
49. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008– 3021. [find](https://scholar.google.com/scholar?q=Nisan+Stiennon%2C+Long+Ouyang%2C+Jeffrey+Wu%2C+Daniel+Ziegler%2C+Ryan+Lowe%2C+Chelsea+Voss%2C+Alec+Radford%2C+Dario+Amodei%2C+and+Paul+F+Christiano%2E+2020%2E+Learning+to+summarize+with+human+feedback%2E+Advances+in+Neural)
50. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [find](https://scholar.google.com/scholar?q=Rohan+Taori%2C+Ishaan+Gulrajani%2C+Tianyi+Zhang%2C+Yann+Dubois%2C+Xuechen+Li%2C+Carlos+Guestrin%2C+Percy+Liang%2C+and+Tatsunori+B%2E+Hashimoto%2E+2023%2E+Stanford+alpaca%3A+An+instruction%2Dfollowing+llama+model%2E)
51. Zhepei Wei, Wei-Lin Chen, and Yu Meng. 2025a. InstructRAG: Instructing retrieval-augmented generation via self-synthesized rationales. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zhepei+Wei%2C+Wei%2DLin+Chen%2C+and+Yu+Meng%2E+2025a%2E+InstructRAG%3A+Instructing+retrieval%2Daugmented+generation+via+self%2Dsynthesized+rationales%2E+In+The+Thirteenth+International+Conference+on+Learning+Representa)
52. Zhepei Wei, Xiao Yang, Kai Sun, Jiaqi Wang, Rulin Shao, Sean Chen, Mohammad Kachuee, Teja Gollapudi, Tony Liao, Nicolas Scheffer, et al. 2025b. TruthRL: Incentivizing truthful LLMs via reinforcement learning. arXiv preprint arXiv:2509.25760. — [arXiv:2509.25760](https://arxiv.org/abs/2509.25760) [find](https://scholar.google.com/scholar?q=Zhepei+Wei%2C+Xiao+Yang%2C+Kai+Sun%2C+Jiaqi+Wang%2C+Rulin+Shao%2C+Sean+Chen%2C+Mohammad+Kachuee%2C+Teja+Gollapudi%2C+Tony+Liao%2C+Nicolas+Scheffer%2C+et+al%2E+2025b%2E+TruthRL%3A+Incentivizing+truthful+LLMs+via+reinforcement+l)
53. Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. 2024. Self-play preference optimization for language model alignment. [find](https://scholar.google.com/scholar?q=Yue+Wu%2C+Zhiqing+Sun%2C+Huizhuo+Yuan%2C+Kaixuan+Ji%2C+Yiming+Yang%2C+and+Quanquan+Gu%2E+2024%2E+Self%2Dplay+preference+optimization+for+language+model+alignment%2E)
54. Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. 2024a. Is DPO superior to PPO for LLM alignment? a comprehensive study. arXiv preprint arXiv:2404.10719. — [arXiv:2404.10719](https://arxiv.org/abs/2404.10719) [find](https://scholar.google.com/scholar?q=Shusheng+Xu%2C+Wei+Fu%2C+Jiaxuan+Gao%2C+Wenjie+Ye%2C+Weilin+Liu%2C+Zhiyu+Mei%2C+Guangju+Wang%2C+Chao+Yu%2C+and+Yi+Wu%2E+2024a%2E+Is+DPO+superior+to+PPO+for+LLM+alignment%3F+a+comprehensive+study%2E+arXiv+preprint+arXiv%3A2404%2E)
55. Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. 2024b. [Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing](https://md.research.elamur.ai/free/en/papers/xu-2024-magpie-alignment-data-synthesis-scratch/).
56. Yu Yang, Aaditya K Singh, Mostafa Elhoushi, Anas Mahmoud, Kushal Tirumala, Fabian Gloeckle, Baptiste Rozière, Carole-Jean Wu, Ari S Morcos, and Newsha Ardalani. 2023. Decoding data quality via synthetic corruptions: Embedding-guided pruning of code data. arXiv preprint arXiv:2312.02418. — [arXiv:2312.02418](https://arxiv.org/abs/2312.02418) [find](https://scholar.google.com/scholar?q=Yu+Yang%2C+Aaditya+K+Singh%2C+Mostafa+Elhoushi%2C+Anas+Mahmoud%2C+Kushal+Tirumala%2C+Fabian+Gloeckle%2C+Baptiste+Rozi%C3%A8re%2C+Carole%2DJean+Wu%2C+Ari+S+Morcos%2C+and+Newsha+Ardalani%2E+2023%2E+Decoding+data+quality+via+synthet)
57. Shangjian Yin, Peijie Huang, JiaTian Chen, Haojing Huang, and Yuhong Xu. 2025a. ECLM: Entity level language model for spoken language understanding with chain of intent. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). [find](https://scholar.google.com/scholar?q=Shangjian+Yin%2C+Peijie+Huang%2C+JiaTian+Chen%2C+Haojing+Huang%2C+and+Yuhong+Xu%2E+2025a%2E+ECLM%3A+Entity+level+language+model+for+spoken+language+understanding+with+chain+of+intent%2E+In+Proceedings+of+the+63rd+Ann)
58. Shangjian Yin, Peijie Huang, and Yuhong Xu. 2025b. MIDLM: Multi-intent detection with bidirectional large language models. In Proceedings of the 31st International Conference on Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shangjian+Yin%2C+Peijie+Huang%2C+and+Yuhong+Xu%2E+2025b%2E+MIDLM%3A+Multi%2Dintent+detection+with+bidirectional+large+language+models%2E+In+Proceedings+of+the+31st+International+Conference+on+Computational+Linguist)
59. Shangjian Yin, Shining Liang, Wenbiao Ding, Yuli Qian, Zhouxing Shi, Hongzhi Li, and Yutao Xie. 2026. Pika: Expert-level synthetic datasets for post-training alignment from scratch. [find](https://scholar.google.com/scholar?q=Shangjian+Yin%2C+Shining+Liang%2C+Wenbiao+Ding%2C+Yuli+Qian%2C+Zhouxing+Shi%2C+Hongzhi+Li%2C+and+Yutao+Xie%2E+2026%2E+Pika%3A+Expert%2Dlevel+synthetic+datasets+for+post%2Dtraining+alignment+from+scratch%2E)
60. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023. [Metamath: Bootstrap your own mathematical questions for large language models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/). arXiv preprint arXiv:2309.12284.
61. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. Self-rewarding language models. [find](https://scholar.google.com/scholar?q=Weizhe+Yuan%2C+Richard+Yuanzhe+Pang%2C+Kyunghyun+Cho%2C+Xian+Li%2C+Sainbayar+Sukhbaatar%2C+Jing+Xu%2C+and+Jason+Weston%2E+2024%2E+Self%2Drewarding+language+models%2E)
62. Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. 2023. [Scaling relationship on learning mathematical reasoning with large language models](https://md.research.elamur.ai/free/en/papers/yuan-2023-scaling-relationship-learning-mathematical-reasoning/). ArXiv preprint, abs/2308.01825.
63. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. [Judging LLM-as-a-judge with MT-Bench and Chatbot Arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). In NeurIPS Datasets and Benchmarks Track.
64. Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei-Lin Chen, Danqi Chen, and Yu Meng. 2025. The surprising effectiveness of negative reinforcement in LLM reasoning. arXiv preprint arXiv:2506.01347. [find](https://scholar.google.com/scholar?q=Xinyu+Zhu%2C+Mengzhou+Xia%2C+Zhepei+Wei%2C+Wei%2DLin+Chen%2C+Danqi+Chen%2C+and+Yu+Meng%2E+2025%2E+The+surprising+effectiveness+of+negative+reinforcement+in+LLM+reasoning%2E+arXiv+preprint+arXiv%3A2506%2E01347%2E)
65. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593. [find](https://scholar.google.com/scholar?q=Daniel+M+Ziegler%2C+Nisan+Stiennon%2C+Jeffrey+Wu%2C+Tom+B+Brown%2C+Alec+Radford%2C+Dario+Amodei%2C+Paul+Christiano%2C+and+Geoffrey+Irving%2E+2019%2E+Fine%2Dtuning+language+models+from+human+preferences%2E+arXiv+preprint+ar)

*Updated 31 October 2026*