---
title: "WildFeedback — paper by Shi et al. (2026) — summary and ideas"
description: "WildFeedback instead builds preference data from feedback that real users give naturally while… 60 references, cited by 8 papers here."
authors: ["Taiwei Shi", "Zhuoer Wang", "Longqi Yang", "Ying-Chun Lin", "Zexue He", "Mengting Wan", "Pei Zhou", "Sujay Kumar Jauhar", "Sihao Chen", "Shan Xia", "Hongfei Zhang", "Jieyu Zhao", "Xiaofeng Xu", "Xia Song", "Jennifer Neville"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.1701/"
doi: "10.18653/v1/2026.acl-long.1701"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shi-2026-wildfeedback-aligning-llms-situ-user/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shi-2026-wildfeedback-aligning-llms-situ-user/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/2/) / WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback

# WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback

**Authors:** [Taiwei Shi](https://md.research.elamur.ai/free/en/authors/taiwei-shi/), Zhuoer Wang, [Longqi Yang](https://md.research.elamur.ai/free/en/authors/longqi-yang/), Ying-Chun Lin, Zexue He, Mengting Wan, [Pei Zhou](https://md.research.elamur.ai/free/en/authors/pei-zhou/), [Sujay Kumar Jauhar](https://md.research.elamur.ai/free/en/authors/sujay-kumar-jauhar/), [Sihao Chen](https://md.research.elamur.ai/free/en/authors/sihao-chen/), Shan Xia, Hongfei Zhang, [Jieyu Zhao](https://md.research.elamur.ai/free/en/authors/jieyu-zhao/), Xiaofeng Xu, [Xia Song](https://md.research.elamur.ai/free/en/authors/xia-song/) and Jennifer Neville · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.1701/) · [PDF](https://aclanthology.org/2026.acl-long.1701.pdf) · [DOI 10.18653/v1/2026.acl-long.1701](https://doi.org/10.18653/v1/2026.acl-long.1701) · [OpenAlex](https://openalex.org/W4402705844)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [8](https://md.research.elamur.ai/free/en/papers/shi-2026-wildfeedback-aligning-llms-situ-user/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Aligning LLMs with human preferences usually depends on curated preference data. Human annotation is expensive and subjective, and GPT-4-labelled synthetic data can reinforce a model's own biases. WildFeedback instead builds preference data from feedback that real users give naturally while chatting with a model. The pipeline runs on WildChat, a corpus of roughly 148K multi-turn user–ChatGPT conversations, in three steps:

1. **Detect feedback.** User satisfaction estimation with SAT/DSAT rubrics finds turns where users signal satisfaction or dissatisfaction.
2. **Build preference pairs.** The conversation leading up to a flagged response becomes the prompt. GPT-4 summarizes what the user wanted, and that summary guides generation of an improved, preferred response.
3. **Evaluate with checklists.** Instance-level checklists derived from the extracted preferences serve as rubrics for GPT-4 judging.

The result is a 20,281-sample preference dataset. Phi 3, LLaMA 3 and Qwen 2 fine-tuned on it improve on AlpacaEval 2, Arena-Hard and MT-Bench, generally matching or exceeding UltraFeedback-trained counterparts. They also align better with real users' stated preferences. The work points toward learning alignment directly from deployment interactions rather than from static, annotator-defined datasets.

## Key points
- **Released dataset:** WildFeedback (20,281 samples) is available at `microsoft/WildFeedback` on Hugging Face.
- **Two preferred-response versions:** an "expert" version generated by GPT-4 and an on-policy version generated by Phi 3, Qwen 2 and LLaMA 3 themselves.
- **Feedback skews negative:** Of about 148.7K WildChat conversations, around 13.6K contain DSAT signals versus about 5.4K with SAT signals. The authors acknowledge this as a selection bias toward negative feedback.
- **Benchmark gains:** Phi 3 trained on the GPT-4 version improves on Arena-Hard from 15.4% to 32.4% and on MT-Bench from 7.32 to 7.75.
- **Checklist-guided evaluation:** On real-user preference tests, LLaMA 3 trained on WildFeedback wins 45.5% and loses 38.8% against its UltraFeedback-trained version. With user-preference checklists guiding the judge, its win rate rises to 50.8%. DSAT feedback is dominated by revision requests and factual errors.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) — Company, Redmond, United States · 325 works · 142,931 citations
- [University of Southern California](https://md.research.elamur.ai/free/en/universities/university-of-southern-california/) — University, Los Angeles, United States · 162 works · 35,836 citations
- [University of California San Diego](https://md.research.elamur.ai/free/en/universities/university-of-california-san-diego/) — University, San Diego, United States · 73 works · 51,948 citations
- [Texas A&M University](https://md.research.elamur.ai/free/en/universities/texas-a-m-university/) — University, College Station, United States · 26 works · 1,871 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Taiwei Shi, Zhuoer Wang, Longqi Yang, Ying-Chun Lin, Zexue He, Mengting Wan, Pei Zhou, Sujay Kumar Jauhar, Sihao Chen, Shan Xia, Hongfei Zhang, Jieyu Zhao, Xiaofeng Xu, Xia Song and Jennifer Neville. 2026. WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{shi2026wildfeedback,
  title = {{WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback}},
  author = {Taiwei Shi, Zhuoer Wang, Longqi Yang, Ying-Chun Lin, Zexue He, Mengting Wan, Pei Zhou, Sujay Kumar Jauhar, Sihao Chen, Shan Xia, Hongfei Zhang, Jieyu Zhao, Xiaofeng Xu, Xia Song and Jennifer Neville},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1701},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1701}
}
```

- [← previous TopoDIM: One-shot Topology Generation of Diverse Interaction Modes for Multi-Agent Systems](https://md.research.elamur.ai/free/en/papers/sun-2026-topodim-one-shot-topology-generation/)
- [next → A Parallel Cross-Lingual Benchmark for Multimodal Idiomaticity Understanding](https://md.research.elamur.ai/free/en/papers/selamet-2026-parallel-cross-lingual-benchmark-multimodal/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/2/)

## References

60 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Qin Cai, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudha… — [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-3-technical-report-highly/)
2. Anthropic. 2023. The claude 3 model family: Opus, sonnet, haiku. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2023%2E+The+claude+3+model+family%3A+Opus%2C+sonnet%2C+haiku%2E)
3. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Sh… [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+Nicholas+Joseph%2C+Saurav+Kadavath%2C+Jackson+Kernion%2C+Tom+Conerly%2C+)
4. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared… — [Constitutional AI: Harmlessness from AI Feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/)
5. Thomas Kleine Buening, Jonas Hübotter, Barna Pász-tor, Idan Shenfeld, Giorgia Ramponi, and Andreas Krause. 2026. Aligning language models from user interactions. Preprint, arXiv:2603.12273. — [arXiv:2603.12273](https://arxiv.org/abs/2603.12273) [find](https://scholar.google.com/scholar?q=Thomas+Kleine+Buening%2C+Jonas+H%C3%BCbotter%2C+Barna+P%C3%A1sz%2Dtor%2C+Idan+Shenfeld%2C+Giorgia+Ramponi%2C+and+Andreas+Krause%2E+2026%2E+Aligning+language+models+from+user+interactions%2E+Preprint%2C+arXiv%3A2603%2E12273%2E+%E2%80%94+%5BarXiv%3A2)
6. Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeff Wu. 2023. [Weak-to-strong generalization: Eliciting strong capabilities with weak supervision](https://md.research.elamur.ai/free/en/papers/burns-2023-weak-strong-generalization-eliciting-strong/). Preprint, arXiv:2312.09390.
7. Daiwei Chen, Yi Chen, Aniket Rege, and Ramya Ko-rlakai Vinayak. 2024. Pal: Pluralistic alignment framework for learning from heterogeneous preferences. Preprint, arXiv:2406.08469. — [arXiv:2406.08469](https://arxiv.org/abs/2406.08469) [find](https://scholar.google.com/scholar?q=Daiwei+Chen%2C+Yi+Chen%2C+Aniket+Rege%2C+and+Ramya+Ko%2Drlakai+Vinayak%2E+2024%2E+Pal%3A+Pluralistic+alignment+framework+for+learning+from+heterogeneous+preferences%2E+Preprint%2C+arXiv%3A2406%2E08469%2E+%E2%80%94+%5BarXiv%3A2406%2E08469%5D)
8. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan… [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Yunxuan+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+Albert+Webson%2C+Shixiang+Shane+Gu%2C+Zhuyun+Dai%2C+Mirac+Suzgun%2C+X)
9. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2024. ULTRAFEEDBACK: Boosting language models with scaled AI feedback. In Proceedings of the 41st International Conference on Machine Learning, volume 2… [find](https://scholar.google.com/scholar?q=Ganqu+Cui%2C+Lifan+Yuan%2C+Ning+Ding%2C+Guanming+Yao%2C+Bingxiang+He%2C+Wei+Zhu%2C+Yuan+Ni%2C+Guotong+Xie%2C+Ruobing+Xie%2C+Yankai+Lin%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2024%2E+ULTRAFEEDBACK%3A+Boosting+language+models+with+s)
10. Sarkar Snigdha Sarathi Das, Chirag Shah, Mengting Wan, Jennifer Neville, Longqi Yang, Reid Andersen, Georg Buscher, and Tara Safavi. 2023. S3-dst: Structured open-domain dialogue segmentation and state tracking in the era of llms. Preprint, arXiv:2309.08827. [find](https://scholar.google.com/scholar?q=Sarkar+Snigdha+Sarathi+Das%2C+Chirag+Shah%2C+Mengting+Wan%2C+Jennifer+Neville%2C+Longqi+Yang%2C+Reid+Andersen%2C+Georg+Buscher%2C+and+Tara+Safavi%2E+2023%2E+S3%2Ddst%3A+Structured+open%2Ddomain+dialogue+segmentation+and+stat)
11. Shachar Don-Yehiya, Leshem Choshen, and Omri Abend. 2024. Learning from naturally occurring feedback. Preprint, arXiv:2407.10944. — [arXiv:2407.10944](https://arxiv.org/abs/2407.10944) [find](https://scholar.google.com/scholar?q=Shachar+Don%2DYehiya%2C+Leshem+Choshen%2C+and+Omri+Abend%2E+2024%2E+Learning+from+naturally+occurring+feedback%2E+Preprint%2C+arXiv%3A2407%2E10944%2E+%E2%80%94+%5BarXiv%3A2407%2E10944%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2407%2E10944%29)
12. Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. 2024. Rlhf workflow: From reward modeling to online rlhf. Preprint, arXiv:2405.07863. [find](https://scholar.google.com/scholar?q=Hanze+Dong%2C+Wei+Xiong%2C+Bo+Pang%2C+Haoxiang+Wang%2C+Han+Zhao%2C+Yingbo+Zhou%2C+Nan+Jiang%2C+Doyen+Sahoo%2C+Caiming+Xiong%2C+and+Tong+Zhang%2E+2024%2E+Rlhf+workflow%3A+From+reward+modeling+to+online+rlhf%2E+Preprint%2C+arXiv%3A2)
13. Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2024. [The faiss library](https://md.research.elamur.ai/free/en/papers/douze-2024-faiss-library/). Preprint, arXiv:2401.08281.
14. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez,… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
15. Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Minh Nguyen, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Alexandrovich Glushkov, Arnav Varma Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alex… — [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://md.research.elamur.ai/free/en/papers/kopf-2023-openassistant-conversations-democratizing-large-language/)
16. Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. 2024. Length-controlled alpacaeval: A simple way to debias automatic evaluators. Preprint, arXiv:2404.04475. [find](https://scholar.google.com/scholar?q=Yann+Dubois%2C+Bal%C3%A1zs+Galambosi%2C+Percy+Liang%2C+and+Tatsunori+B%2E+Hashimoto%2E+2024%2E+Length%2Dcontrolled+alpacaeval%3A+A+simple+way+to+debias+automatic+evaluators%2E+Preprint%2C+arXiv%3A2404%2E04475%2E)
17. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023b. Alpacaeval: An automatic evaluator of instruction-following models. [https://github.com/tatsu-lab/alpaca_eval](https://github.com/tatsu-lab/alpaca_eval). [↗](https://github.com/tatsu-lab/alpaca_eval)
18. Eve Fleisig, Rediet Abebe, and Dan Klein. 2023. When the majority is wrong: Modeling annotator disagreement for subjective tasks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6715–6726, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Eve+Fleisig%2C+Rediet+Abebe%2C+and+Dan+Klein%2E+2023%2E+When+the+majority+is+wrong%3A+Modeling+annotator+disagreement+for+subjective+tasks%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+Methods+in+Natural+)
19. Sanjana Gautam and Mukund Srinath. 2024. Blind spots and biases: Exploring the role of annotator cognitive biases in nlp. Preprint, arXiv:2404.19071. [find](https://scholar.google.com/scholar?q=Sanjana+Gautam+and+Mukund+Srinath%2E+2024%2E+Blind+spots+and+biases%3A+Exploring+the+role+of+annotator+cognitive+biases+in+nlp%2E+Preprint%2C+arXiv%3A2404%2E19071%2E)
20. Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Chi Zhang, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2023. [Beavertails: Towards improved safety alignment of llm via a human-preference dataset](https://md.research.elamur.ai/free/en/papers/ji-2023-beavertails-towards-improved-safety-alignment/). Preprint, arXiv:2307.04657.
21. Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smi… — [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://md.research.elamur.ai/free/en/papers/lambert-2024-tulu-3-pushing-frontiers-open/)
22. Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. 2023. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. Preprint, arXiv:2309.00267. [find](https://scholar.google.com/scholar?q=Harrison+Lee%2C+Samrat+Phatale%2C+Hassan+Mansoor%2C+Thomas+Mesnard%2C+Johan+Ferret%2C+Kellie+Lu%2C+Colton+Bishop%2C+Ethan+Hall%2C+Victor+Carbune%2C+Abhinav+Rastogi%2C+and+Sushant+Prakash%2E+2023%2E+Rlaif%3A+Scaling+reinforceme)
23. Minzhi Li, Taiwei Shi, Caleb Ziems, Min-Yen Kan, Nancy Chen, Zhengyuan Liu, and Diyi Yang. 2023a. [CoAnnotating: Uncertainty-guided work allocation between human and large language models for data annotation](https://md.research.elamur.ai/free/en/papers/li-2023-coannotating-uncertainty-guided-work-allocation/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p…
24. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. 2024. [From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline](https://md.research.elamur.ai/free/en/papers/li-2024-crowdsourced-data-high-quality-benchmarks/). Preprint, arXiv:2406.11939.
25. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. [Let’s verify step by step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/). Preprint, arXiv:2305.20050.
26. Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. 2024a. [Wildbench: Benchmarking llms with challenging tasks from real users in the wild](https://md.research.elamur.ai/free/en/papers/lin-2024-wildbench-benchmarking-llms-challenging-tasks/). Preprint, arXiv:2406.04770.
27. Ying-Chun Lin, Jennifer Neville, Jack W. Stokes, Longqi Yang, Tara Safavi, Mengting Wan, Scott Counts, Siddharth Suri, Reid Andersen, Xiaofeng Xu, Deepak Gupta, Sujay Kumar Jauhar, Xia Song, Georg Buscher, Saurabh Tiwary, Brent Hecht, and Jaime Teevan. 2024b. Interpretable user satisfaction estimati… [find](https://scholar.google.com/scholar?q=Ying%2DChun+Lin%2C+Jennifer+Neville%2C+Jack+W%2E+Stokes%2C+Longqi+Yang%2C+Tara+Safavi%2C+Mengting+Wan%2C+Scott+Counts%2C+Siddharth+Suri%2C+Reid+Andersen%2C+Xiaofeng+Xu%2C+Deepak+Gupta%2C+Sujay+Kumar+Jauhar%2C+Xia+Song%2C+Georg+Bus)
28. Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer… — [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
29. Jie Liu, Zhanhui Zhou, Jiaheng Liu, Xingyuan Bu, Chao Yang, Han-Sen Zhong, and Wanli Ouyang. 2024a. Iterative length-regularized direct preference optimization: A case study on improving 7b language models to gpt-4 level. Preprint, arXiv:2406.11817. — [arXiv:2406.11817](https://arxiv.org/abs/2406.11817) [find](https://scholar.google.com/scholar?q=Jie+Liu%2C+Zhanhui+Zhou%2C+Jiaheng+Liu%2C+Xingyuan+Bu%2C+Chao+Yang%2C+Han%2DSen+Zhong%2C+and+Wanli+Ouyang%2E+2024a%2E+Iterative+length%2Dregularized+direct+preference+optimization%3A+A+case+study+on+improving+7b+language+m)
30. Yiqi Liu, Nafise Sadat Moosavi, and Chenghua Lin. 2024b. Llms as narcissistic evaluators: When ego inflates evaluation scores. Preprint, arXiv:2311.09766. [find](https://scholar.google.com/scholar?q=Yiqi+Liu%2C+Nafise+Sadat+Moosavi%2C+and+Chenghua+Lin%2E+2024b%2E+Llms+as+narcissistic+evaluators%3A+When+ego+inflates+evaluation+scores%2E+Preprint%2C+arXiv%3A2311%2E09766%2E)
31. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with… [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Prakhar+Gupta%2C+Skyler+Hallinan%2C+Luyu+Gao%2C+Sarah+Wiegreffe%2C+Uri+Alon%2C+Nouha+Dziri%2C+Shrimai+Prabhumoye%2C+Yiming+Yang%2C+Shashank+Gupta%2C+Bodhisattwa+Prasad+Majumder%2C+Katherine+Her)
32. Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. Simpo: Simple preference optimization with a reference-free reward. Preprint, arXiv:2405.14734. [find](https://scholar.google.com/scholar?q=Yu+Meng%2C+Mengzhou+Xia%2C+and+Danqi+Chen%2E+2024%2E+Simpo%3A+Simple+preference+optimization+with+a+reference%2Dfree+reward%2E+Preprint%2C+arXiv%3A2405%2E14734%2E)
33. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2022. [Webgpt: Browser-assis…](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/)
34. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Ja… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
35. Yijun Pan, Taiwei Shi, Jieyu Zhao, and Jiaqi W. Ma. 2025. Detecting and filtering unsafe training data via data attribution with denoised representation. Preprint, arXiv:2502.11411. — [arXiv:2502.11411](https://arxiv.org/abs/2502.11411) [find](https://scholar.google.com/scholar?q=Yijun+Pan%2C+Taiwei+Shi%2C+Jieyu+Zhao%2C+and+Jiaqi+W%2E+Ma%2E+2025%2E+Detecting+and+filtering+unsafe+training+data+via+data+attribution+with+denoised+representation%2E+Preprint%2C+arXiv%3A2502%2E11411%2E+%E2%80%94+%5BarXiv%3A2502%2E1141)
36. Jupinder Parmar, Sanjev Satheesh, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2024. [Reuse, don’t retrain: A recipe for continued pretraining of language models](https://md.research.elamur.ai/free/en/papers/parmar-2024-reuse-don-t-retrain-recipe/). Preprint, arXiv:2407.07263.
37. Jiaxin Pei, Aparna Ananthasubramaniam, Xingyao Wang, Naitian Zhou, Apostolos Dedeloudis, Jackson Sargent, and David Jurgens. 2022. [Potato: The portable text annotation tool](https://md.research.elamur.ai/free/en/papers/pei-2022-potato-portable-text-annotation-tool/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations.
38. Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, and Natasha Jaques. 2024. Personalizing reinforcement learning from human feedback with variational preference learning. In Pluralistic Alignment Workshop at NeurIPS 2024. [find](https://scholar.google.com/scholar?q=Sriyash+Poddar%2C+Yanming+Wan%2C+Hamish+Ivison%2C+Abhishek+Gupta%2C+and+Natasha+Jaques%2E+2024%2E+Personalizing+reinforcement+learning+from+human+feedback+with+variational+preference+learning%2E+In+Pluralistic+Alig)
39. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). In Thirty-seventh Conference on Neural Information Processing Systems.
40. Taiwei Shi, Kai Chen, and Jieyu Zhao. 2024. Safer-instruct: Aligning language models with automated preference data. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7636–7… [find](https://scholar.google.com/scholar?q=Taiwei+Shi%2C+Kai+Chen%2C+and+Jieyu+Zhao%2E+2024%2E+Safer%2Dinstruct%3A+Aligning+language+models+with+automated+preference+data%2E+In+Proceedings+of+the+2024+Conference+of+the+North+American+Chapter+of+the+Associat)
41. Taiwei Shi, Sihao Chen, Bowen Jiang, Linxin Song, Longqi Yang, and Jieyu Zhao. 2026. Experiential reinforcement learning. Preprint, arXiv:2602.13949. — [arXiv:2602.13949](https://arxiv.org/abs/2602.13949) [find](https://scholar.google.com/scholar?q=Taiwei+Shi%2C+Sihao+Chen%2C+Bowen+Jiang%2C+Linxin+Song%2C+Longqi+Yang%2C+and+Jieyu+Zhao%2E+2026%2E+Experiential+reinforcement+learning%2E+Preprint%2C+arXiv%3A2602%2E13949%2E+%E2%80%94+%5BarXiv%3A2602%2E13949%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2602%2E13)
42. Weiyan Shi, Emily Dinan, Kurt Shuster, Jason Weston, and Jing Xu. 2022. When life gives you lemons, make cherryade: Converting feedback from bad responses into good labels. Preprint, arXiv:2210.15893. [find](https://scholar.google.com/scholar?q=Weiyan+Shi%2C+Emily+Dinan%2C+Kurt+Shuster%2C+Jason+Weston%2C+and+Jing+Xu%2E+2022%2E+When+life+gives+you+lemons%2C+make+cherryade%3A+Converting+feedback+from+bad+responses+into+good+labels%2E+Preprint%2C+arXiv%3A2210%2E15893%2E)
43. Linxin Song, Taiwei Shi, and Jieyu Zhao. 2025. The hallucination tax of reinforcement finetuning. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 2105–2120, Suzhou, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Linxin+Song%2C+Taiwei+Shi%2C+and+Jieyu+Zhao%2E+2025%2E+The+hallucination+tax+of+reinforcement+finetuning%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2025%2C+pages+2105%E2%80%932120%2C+Suzhou%2C+Chi)
44. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton T… [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Morgane+Riviere%2C+Shreya+Pathak%2C+Pier+Giuseppe+Sessa%2C+Cassidy+Hardin%2C+Surya+Bhupatiraju%2C+L%C3%A9onard+Hussenot%2C+Thomas+Mesnard%2C+Bobak+Shahriari%2C+Alexandre+Ram%C3%A9%2C+Johan+Ferret%2C+Peter+Liu%2C+Pouya+Ta)
45. Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. 2024. Judging the judges: Evaluating alignment and vulnerabilities in llms-as-judges. Preprint, arXiv:2406.12624. [find](https://scholar.google.com/scholar?q=Aman+Singh+Thakur%2C+Kartik+Choudhary%2C+Venkat+Srinik+Ramayapally%2C+Sankaran+Vaidyanathan%2C+and+Dieuwke+Hupkes%2E+2024%2E+Judging+the+judges%3A+Evaluating+alignment+and+vulnerabilities+in+llms%2Das%2Djudges%2E+Preprin)
46. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. 2023. Zephyr: Direct distillation of lm alignment. Preprint, arXiv… [find](https://scholar.google.com/scholar?q=Lewis+Tunstall%2C+Edward+Beeching%2C+Nathan+Lambert%2C+Nazneen+Rajani%2C+Kashif+Rasul%2C+Younes+Belkada%2C+Shengyi+Huang%2C+Leandro+von+Werra%2C+Cl%C3%A9mentine+Fourrier%2C+Nathan+Habib%2C+Nathan+Sarrazin%2C+Omar+Sanseviero%2C+Al)
47. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A%2E+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+2023%2E+Self%2Dinstruct%3A+Aligning+language+models+with+self%2Dgenerated+instructions%2E+In+Proc)
48. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Swaroop+Mishra%2C+Pegah+Alipoormolabashi%2C+Yeganeh+Kordi%2C+Amirreza+Mirzaei%2C+Atharva+Naik%2C+Arjun+Ashok%2C+Arut+Selvan+Dhanasekaran%2C+Anjana+Arunkumar%2C+David+Stap%2C+Eshaan+Pathak%2C+Giannis+Karaman)
49. Jiang. 2023. [Wizardlm: Empowering large language models to follow complex instructions](https://md.research.elamur.ai/free/en/papers/xu-2023-wizardlm-empowering-large-pre-trained/). Preprint, arXiv:2304.12244.
50. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Juny… [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chang+Zhou%2C+Chengpeng+Li%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Guanting+Dong%2C+Haoran+Wei%2C+Huan+Lin%2C+Jialong+Tang%2C+Jialin+Wang%2C+Jian+Yang%2C+Jian)
51. Michael JQ Zhang, Zhilin Wang, Jena D. Hwang, Yi Dong, Olivier Delalleau, Yejin Choi, Eunsol Choi, Xiang Ren, and Valentina Pyatkin. 2024. Diverging preferences: When do annotators disagree and do models know? Preprint, arXiv:2410.14632. [find](https://scholar.google.com/scholar?q=Michael+JQ+Zhang%2C+Zhilin+Wang%2C+Jena+D%2E+Hwang%2C+Yi+Dong%2C+Olivier+Delalleau%2C+Yejin+Choi%2C+Eunsol+Choi%2C+Xiang+Ren%2C+and+Valentina+Pyatkin%2E+2024%2E+Diverging+preferences%3A+When+do+annotators+disagree+and+do+mod)
52. Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024. [Wildchat: 1m chatgpt interaction logs in the wild](https://md.research.elamur.ai/free/en/papers/zhao-2024-wildchat-1m-chatgpt-interaction-logs/). Preprint, arXiv:2405.01470.
53. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023a. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). In Advances in Neural Information Processing Systems, volume…
54. Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. 2024. Helpsteer2: Open-source dataset for training top-performing reward models. Preprint, arXiv:2406.08673. [find](https://scholar.google.com/scholar?q=Zhilin+Wang%2C+Yi+Dong%2C+Olivier+Delalleau%2C+Jiaqi+Zeng%2C+Gerald+Shen%2C+Daniel+Egert%2C+Jimmy+J%2E+Zhang%2C+Makesh+Narsimhan+Sreedhar%2C+and+Oleksii+Kuchaiev%2E+2024%2E+Helpsteer2%3A+Open%2Dsource+dataset+for+training+top%2D)
55. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023b. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Preprint, arXiv:2306.05685.
56. Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. 2021. [Recursively summarizing books with human feedback](https://md.research.elamur.ai/free/en/papers/wu-2021-recursively-summarizing-books-human-feedback/). Preprint, arXiv:2109.10862.
57. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. [Llamafactory: Unified efficient fine-tuning of 100+ language models](https://md.research.elamur.ai/free/en/papers/zheng-2024-llamafactory-unified-efficient-fine-tuning/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations),…
58. Sierra Wyllie, Ilia Shumailov, and Nicolas Papernot. 2024. Fairness feedback loops: Training on synthetic data amplifies bias. Preprint, arXiv:2403.07857. — [arXiv:2403.07857](https://arxiv.org/abs/2403.07857) [find](https://scholar.google.com/scholar?q=Sierra+Wyllie%2C+Ilia+Shumailov%2C+and+Nicolas+Papernot%2E+2024%2E+Fairness+feedback+loops%3A+Training+on+synthetic+data+amplifies+bias%2E+Preprint%2C+arXiv%3A2403%2E07857%2E+%E2%80%94+%5BarXiv%3A2403%2E07857%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F24)
59. Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. 2024. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. Preprint, arXiv:2312.11456. [find](https://scholar.google.com/scholar?q=Wei+Xiong%2C+Hanze+Dong%2C+Chenlu+Ye%2C+Ziqi+Wang%2C+Han+Zhong%2C+Heng+Ji%2C+Nan+Jiang%2C+and+Tong+Zhang%2E+2024%2E+Iterative+preference+learning+from+human+feedback%3A+Bridging+theory+and+practice+for+rlhf+under+kl%2Dcons)
60. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin [find](https://scholar.google.com/scholar?q=Can+Xu%2C+Qingfeng+Sun%2C+Kai+Zheng%2C+Xiubo+Geng%2C+Pu+Zhao%2C+Jiazhan+Feng%2C+Chongyang+Tao%2C+and+Daxin)

*Updated 1 November 2026*