---
title: "Zero-Shot Goal-Directed Dialogue via RL on Imagined"
description: "Hong, Levine and Dragan propose a zero-shot alternative that needs no human-human dialogue… 56 references, cited by 13 papers here."
authors: ["Joey Hong", "Sergey Levine", "Anca Dragan"]
year: 2023
venue: "arXiv:2311.05584"
source: "https://arxiv.org/abs/2311.05584"
doi: "10.48550/arxiv.2311.05584"
arxiv: "2311.05584"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/hong-2023-zero-shot-goal-directed-dialogue/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hong-2023-zero-shot-goal-directed-dialogue/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/14/) / Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations

# Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations

**Authors:** Joey Hong, [Sergey Levine](https://md.research.elamur.ai/free/en/authors/sergey-levine/) and [Anca Dragan](https://md.research.elamur.ai/free/en/authors/anca-dragan/) · **Year:** 2023 · **Venue:** arXiv:2311.05584 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2311.05584) · [PDF](https://arxiv.org/pdf/2311.05584) · [DOI 10.48550/arxiv.2311.05584](https://doi.org/10.48550/arxiv.2311.05584) · [arXiv 2311.05584](https://arxiv.org/abs/2311.05584) · [OpenAlex](https://openalex.org/W4388585664)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [13](https://md.research.elamur.ai/free/en/papers/hong-2023-zero-shot-goal-directed-dialogue/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
LLMs produce fluent replies but rarely steer a conversation toward an outcome. When the user holds information the model lacks, such as prior knowledge or travel tastes, the model tends to guess right away instead of asking focused questions. RLHF-style tuning does not fix this because it scores single responses rather than whole dialogues. Hong, Levine and Dragan propose a zero-shot alternative that needs no human-human dialogue corpus. Starting from a textual task description, their imagination engine (IE) prompts an LLM to synthesize many varied, human-like but imperfect conversations, each built around sampled hidden user characteristics. Dialogue is modeled as a token-level MDP with a reward at the end, and offline RL on this synthetic data trains an agent to pursue multi-turn goals. Tasks include teaching, persuasion and preference elicitation, and the evaluation includes a user study. The trained agents beat prompted state-of-the-art LLMs while being orders of magnitude smaller. The work matters because it uses LLMs as generators of training data rather than as final agents. The authors present it as the first use of synthetic dialogue generation for RL.

## Key points
- **Pipeline:** IE generates imagined dialogues between simulated humans and a possibly suboptimal agent, with some components handcrafted. A downstream agent is then trained with offline RL (ILQL in the examples). It is compared with IE+BC and IE+FBC, cloning-based baselines trained on the same imagined data.
- **Formalism:** The MDP is M = (S, A, P, r, ρ, γ). Actions are vocabulary tokens, and an \<EOS> token hands the turn to the human. A deterministic reward r(s_T) is given only at the final timestep T.
- **Tasks:** The agents are tested on three tasks:
  - explaining a new concept (e.g., offline reinforcement learning) adapted to the user's background;
  - persuasion;
  - a travel-agent preference-elicitation task.
- **Qualitative example:** A budget-conscious shopper turns down a luxury shopping tour. Only the RL agent adjusts, suggesting affordable local markets and street food, while the BC/FBC agents just apologize.
- **Positioning and evaluation:** The method is framed as knowledge distillation from a large LLM; earlier synthetic-data distillation targeted only supervised objectives. Prompted LLMs tend to ask long, overwhelming questions. The provided excerpt does not include exact user-study scores.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of California, Berkeley](https://md.research.elamur.ai/free/en/universities/university-of-california-berkeley/) — University, Berkeley, United States · 243 works · 251,774 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works

🧭 Topics (1): [🗨️ Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) · 664 works

## Cite

Cite as: Joey Hong, Sergey Levine and Anca Dragan. 2023. Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations. arXiv:2311.05584.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{hong2023zero,
  title = {{Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations}},
  author = {Joey Hong, Sergey Levine and Anca Dragan},
  year = {2023},
  doi = {10.48550/arxiv.2311.05584},
  eprint = {2311.05584},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2311.05584}
}
```

- [← previous Zero and Few-shot Semantic Parsing with Ambiguous Inputs](https://md.research.elamur.ai/free/en/papers/stengel-eskin-2023-zero-few-shot-semantic-parsing/)
- [next → AmbiCoref: Evaluating Human and Model Sensitivity to Ambiguous Coreference](https://md.research.elamur.ai/free/en/papers/yuan-2023-ambicoref-evaluating-human-model-sensitivity/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/14/)

## References

56 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Sh… [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+Nicholas+Joseph%2C+Saurav+Kadavath%2C+Jackson+Kernion%2C+Tom+Conerly%2C+)
2. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared… — [Constitutional AI: Harmlessness from AI Feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/)
3. Florian Böhm, Yang Gao, Christian M. Meyer, Ori Shapira, Ido Dagan, and Iryna Gurevych. [Better rewards yield better summaries: Learning to summarise without references](https://md.research.elamur.ai/free/en/papers/bohm-2019-better-rewards-yield-better-summaries/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen…
4. Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. Multiwoz – a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling, 2020. — [MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://md.research.elamur.ai/free/en/papers/budzianowski-2018-multiwoz-large-scale-multi-domain/)
5. Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre-Yves Oudeyer. Grounding large language models in interactive environments with online reinforcement learning, 2023. [find](https://scholar.google.com/scholar?q=Thomas+Carta%2C+Cle%CC%81ment+Romac%2C+Thomas+Wolf%2C+Sylvain+Lamprier%2C+Olivier+Sigaud%2C+and+Pierre%2DYves+Oudeyer%2E+Grounding+large+language+models+in+interactive+environments+with+online+reinforcement+learning%2C+20)
6. Derek Chen, Howard Chen, Yi Yang, Alex Lin, and Zhou Yu. Action-based conversations dataset: A corpus for building more in-depth task-oriented dialogue systems. CoRR, abs/2104.00783, 2021a. [find](https://scholar.google.com/scholar?q=Derek+Chen%2C+Howard+Chen%2C+Yi+Yang%2C+Alex+Lin%2C+and+Zhou+Yu%2E+Action%2Dbased+conversations+dataset%3A+A+corpus+for+building+more+in%2Ddepth+task%2Doriented+dialogue+systems%2E+CoRR%2C+abs%2F2104%2E00783%2C+2021a%2E)
7. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mi… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
8. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL [https://lmsys.org/blog/2023-03-30-vicuna…](https://lmsys.org/blog/2023-03-30-vicuna…) [↗](https://lmsys.org/blog/2023-03-30-vicuna…)
9. Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences, 2023. [find](https://scholar.google.com/scholar?q=Paul+Christiano%2C+Jan+Leike%2C+Tom+B%2E+Brown%2C+Miljan+Martic%2C+Shane+Legg%2C+and+Dario+Amodei%2E+Deep+reinforcement+learning+from+human+preferences%2C+2023%2E)
10. Finale Doshi-Velez and George Dimitri Konidaris. Hidden parameter markov decision processes: A semiparametric regression approach for discovering latent task parametrizations. CoRR, abs/1308.3513, 2013. — [arXiv:1308.3513](https://arxiv.org/abs/1308.3513) [find](https://scholar.google.com/scholar?q=Finale+Doshi%2DVelez+and+George+Dimitri+Konidaris%2E+Hidden+parameter+markov+decision+processes%3A+A+semiparametric+regression+approach+for+discovering+latent+task+parametrizations%2E+CoRR%2C+abs%2F1308%2E3513%2C+201)
11. J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. In arXiv, 2020a. URL [https://arxiv.org/pdf/2004.07219](https://arxiv.org/pdf/2004.07219). — [arXiv:2004.07219](https://arxiv.org/abs/2004.07219) [↗](https://arxiv.org/pdf/2004.07219)
12. Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020b. [find](https://scholar.google.com/scholar?q=Justin+Fu%2C+Aviral+Kumar%2C+Ofir+Nachum%2C+George+Tucker%2C+and+Sergey+Levine%2E+D4rl%3A+Datasets+for+deep+data%2Ddriven+reinforcement+learning%2C+2020b%2E)
13. Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. arXiv preprint arXiv:1812.02900, 2018. — [arXiv:1812.02900](https://arxiv.org/abs/1812.02900) [find](https://scholar.google.com/scholar?q=Scott+Fujimoto%2C+David+Meger%2C+and+Doina+Precup%2E+Off%2Dpolicy+deep+reinforcement+learning+without+exploration%2E+arXiv+preprint+arXiv%3A1812%2E02900%2C+2018%2E+%E2%80%94+%5BarXiv%3A1812%2E02900%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1812%2E02900%29)
14. Milica Gašić, Filip Jurčı́ček, Blaise Thomson, Kai Yu, and Steve Young. On-line policy optimisation of spoken dialogue systems via live interaction with human subjects. In 2011 IEEE Workshop on Automatic Speech Recognition & Understanding, pp. 312–317, 2011. doi: 10.1109/ASRU.2011. 6163950. — [doi:10.1109/asru.2011](https://doi.org/10.1109/asru.2011) [find](https://scholar.google.com/scholar?q=Milica+Gas%CC%8Cic%CC%81%2C+Filip+Jurc%CC%8C%C4%B1%CC%81c%CC%8Cek%2C+Blaise+Thomson%2C+Kai+Yu%2C+and+Steve+Young%2E+On%2Dline+policy+optimisation+of+spoken+dialogue+systems+via+live+interaction+with+human+subjects%2E+In+2011+IEEE+Workshop+on+Au)
15. Marjan Ghazvininejad, Xing Shi, Jay Priyadarshi, and Kevin Knight. [Hafez: an interactive poetry generation system](https://md.research.elamur.ai/free/en/papers/ghazvininejad-2017-hafez-interactive-poetry-generation-system/). In Proceedings of ACL 2017, System Demonstrations, pp. 43–48, Vancouver, Canada, July 2017. Association for Computational Linguistics. URL https://aclanthology.org/P17-4008. [↗](https://aclanthology.org/P17-4008)
16. Amelia Glaese, Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. [Improving alignment of dialogue agents via targeted human judgements](https://md.research.elamur.ai/free/en/papers/glaese-2022-improving-alignment-dialogue-agents-targeted/). arXiv preprint arXiv:2209.14375, 2022.
17. Jiatao Gu, Kyunghyun Cho, and Victor O.K. Li. Trainable greedy decoding for neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1968–1978, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18… [find](https://scholar.google.com/scholar?q=Jiatao+Gu%2C+Kyunghyun+Cho%2C+and+Victor+O%2EK%2E+Li%2E+Trainable+greedy+decoding+for+neural+machine+translation%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pp%2E+1)
18. Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Tom Le Paine, Sergio Gómez Colmenarejo, Konrad Zolna, Rishabh Agarwal, Josh Merel, Daniel Mankowitz, Cosmin Paduraru, et al. Rl unplugged: Benchmarks for offline reinforcement learning. 2020. [find](https://scholar.google.com/scholar?q=Caglar+Gulcehre%2C+Ziyu+Wang%2C+Alexander+Novikov%2C+Tom+Le+Paine%2C+Sergio+Go%CC%81mez+Colmenarejo%2C+Konrad+Zolna%2C+Rishabh+Agarwal%2C+Josh+Merel%2C+Daniel+Mankowitz%2C+Cosmin+Paduraru%2C+et+al%2E+Rl+unplugged%3A+Benchmarks+fo)
19. Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, and Nando de Freitas. [Reinforced self-training (rest) for language modeling](https://md.research.elamur.ai/free/en/papers/gulcehre-2023-reinforced-self-training-rest-language/), 2023.
20. He He, Derek Chen, Anusha Balakrishnan, and Percy Liang. [Decoupling strategy and generation in negotiation dialogues](https://md.research.elamur.ai/free/en/papers/he-2018-decoupling-strategy-generation-negotiation-dialogues/), 2018.
21. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. [Distilling the knowledge in a neural network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/), 2015.
22. Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. [Learning to write with cooperative discriminators](https://md.research.elamur.ai/free/en/papers/holtzman-2018-learning-write-cooperative-discriminators/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1638–1649, Melbourne, Australia, July 2018…
23. Youngsoo Jang, Jongmin Lee, and Kee-Eung Kim. GPT-critic: Offline reinforcement learning for end-to-end task-oriented dialogue systems. In International Conference on Learning Representations, 2022. URL [https://openreview.net/forum?id=qaxhBG1UUaS](https://openreview.net/forum?id=qaxhBG1UUaS). [↗](https://openreview.net/forum?id=qaxhBG1UUaS)
24. Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Àgata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind W. Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. CoRR, abs/1907.00456, 2019. — [arXiv:1907.00456](https://arxiv.org/abs/1907.00456) [find](https://scholar.google.com/scholar?q=Natasha+Jaques%2C+Asma+Ghandeharioun%2C+Judy+Hanwen+Shen%2C+Craig+Ferguson%2C+A%CC%80gata+Lapedriza%2C+Noah+Jones%2C+Shixiang+Gu%2C+and+Rosalind+W%2E+Picard%2E+Way+off%2Dpolicy+batch+deep+reinforcement+learning+of+implicit+hu)
25. Yoon Kim and Alexander M. Rush. [Sequence-level knowledge distillation](https://md.research.elamur.ai/free/en/papers/kim-2016-sequence-level-knowledge-distillation/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1317–1327, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16-1139. URL https://aclantholog…
26. Ilya Kostrikov, Jonathan Tompson, Rob Fergus, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. arXiv preprint arXiv:2103.08050, 2021. — [arXiv:2103.08050](https://arxiv.org/abs/2103.08050) [find](https://scholar.google.com/scholar?q=Ilya+Kostrikov%2C+Jonathan+Tompson%2C+Rob+Fergus%2C+and+Ofir+Nachum%2E+Offline+reinforcement+learning+with+fisher+divergence+critic+regularization%2E+arXiv+preprint+arXiv%3A2103%2E08050%2C+2021%2E+%E2%80%94+%5BarXiv%3A2103%2E08050%5D%28)
27. Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, pp. 11761–11771, 2019. [find](https://scholar.google.com/scholar?q=Aviral+Kumar%2C+Justin+Fu%2C+Matthew+Soh%2C+George+Tucker%2C+and+Sergey+Levine%2E+Stabilizing+off%2Dpolicy+q%2Dlearning+via+bootstrapping+error+reduction%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pp%2E+1)
28. Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.04779, 2020. [find](https://scholar.google.com/scholar?q=Aviral+Kumar%2C+Aurick+Zhou%2C+George+Tucker%2C+and+Sergey+Levine%2E+Conservative+q%2Dlearning+for+offline+reinforcement+learning%2E+arXiv+preprint+arXiv%3A2006%2E04779%2C+2020%2E)
29. Aviral Kumar, Joey Hong, Anikait Singh, and Sergey Levine. When should we prefer offline reinforcement learning over behavioral cloning?, 2022. [find](https://scholar.google.com/scholar?q=Aviral+Kumar%2C+Joey+Hong%2C+Anikait+Singh%2C+and+Sergey+Levine%2E+When+should+we+prefer+offline+reinforcement+learning+over+behavioral+cloning%3F%2C+2022%2E)
30. Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020. [find](https://scholar.google.com/scholar?q=Sergey+Levine%2C+Aviral+Kumar%2C+George+Tucker%2C+and+Justin+Fu%2E+Offline+reinforcement+learning%3A+Tutorial%2C+review%2C+and+perspectives+on+open+problems%2E+arXiv+preprint+arXiv%3A2005%2E01643%2C+2020%2E)
31. Jiwei Li, Will Monroe, and Dan Jurafsky. Learning to decode for future success, 2017. [find](https://scholar.google.com/scholar?q=Jiwei+Li%2C+Will+Monroe%2C+and+Dan+Jurafsky%2E+Learning+to+decode+for+future+success%2C+2017%2E)
32. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted qu… — [WebGPT: Browser-assisted question-answering with human feedback](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/)
33. OpenAI. Chatgpt, 2022. URL [https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt). [↗](https://openai.com/blog/chatgpt)
34. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training la… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
35. Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior, 2023. [find](https://scholar.google.com/scholar?q=Joon+Sung+Park%2C+Joseph+C%2E+O%E2%80%99Brien%2C+Carrie+J%2E+Cai%2C+Meredith+Ringel+Morris%2C+Percy+Liang%2C+and+Michael+S%2E+Bernstein%2E+Generative+agents%3A+Interactive+simulacra+of+human+behavior%2C+2023%2E)
36. Romain Paulus, Caiming Xiong, and Richard Socher. [A deep reinforced model for abstractive summarization](https://md.research.elamur.ai/free/en/papers/paulus-2017-deep-reinforced-model-abstractive-summarization/), 2017.
37. Valentina Pyatkin, Jena D. Hwang, Vivek Srikumar, Ximing Lu, Liwei Jiang, Yejin Choi, and Chandra Bhagavatula. Reinforced clarification question generation with defeasibility rewards for disambiguating social and moral situations, 2022. [find](https://scholar.google.com/scholar?q=Valentina+Pyatkin%2C+Jena+D%2E+Hwang%2C+Vivek+Srikumar%2C+Ximing+Lu%2C+Liwei+Jiang%2C+Yejin+Choi%2C+and+Chandra+Bhagavatula%2E+Reinforced+clarification+question+generation+with+defeasibility+rewards+for+disambiguatin)
38. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2C+2019%2E)
39. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/), 2023.
40. Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. [Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization](https://md.research.elamur.ai/free/en/papers/ramamurthy-2022-reinforcement-learning-natural-language-processing/). In T…
41. Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Fai… — [Code Llama: Open Foundation Models for Code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/)
42. Greg Serapio-Garcı́a, Mustafa Safdari, Clément Crepy, Luning Sun, Stephen Fitz, Peter Romero, Marwa Abdulhai, Aleksandra Faust, and Maja Matarić. [Personality traits in large language models](https://md.research.elamur.ai/free/en/papers/serapio-garcia-2023-personality-traits-large-language-models/), 2023.
43. Kurt Shuster, Mojtaba Komeili, Leonard Adolphs, Stephen Roller, Arthur Szlam, and Jason Weston. [Language models that seek for knowledge: Modular search & generation for dialogue and prompt completion](https://md.research.elamur.ai/free/en/papers/shuster-2022-language-models-that-seek-knowledge/), 2022a.
44. Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju, Eric Michael Smith, Stephen Roller, Megan Ung, Moya Chen, Kushal Arora, Joshua Lane, Morteza Behrooz, William Ngan, Spencer Poff, Naman Goyal, Arthur Szlam, Y-Lan Boureau, Melanie Kambadur, and Browser assisted question-answering with human feedbackJaso… — [BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage](https://md.research.elamur.ai/free/en/papers/shuster-2022-blenderbot-3-deployed-conversational-agent/)
45. Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking. In Advances in neural information processing systems, 2022. [find](https://scholar.google.com/scholar?q=Joar+Skalse%2C+Nikolaus+H%2E+R%2E+Howe%2C+Dmitrii+Krasheninnikov%2C+and+David+Krueger%2E+Defining+and+characterizing+reward+hacking%2E+In+Advances+in+neural+information+processing+systems%2C+2022%2E)
46. Charlie Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. Offline rl for natural language generation with implicit language q learning. arXiv preprint arXiv:2206.11871, 2022. — [arXiv:2206.11871](https://arxiv.org/abs/2206.11871) [find](https://scholar.google.com/scholar?q=Charlie+Snell%2C+Ilya+Kostrikov%2C+Yi+Su%2C+Mengjiao+Yang%2C+and+Sergey+Levine%2E+Offline+rl+for+natural+language+generation+with+implicit+language+q+learning%2E+arXiv+preprint+arXiv%3A2206%2E11871%2C+2022%2E+%E2%80%94+%5BarXiv%3A22)
47. Charlie Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. Offline rl for natural language generation with implicit language q learning. In International Conference on Learning Representations (ICLR), 2023. [find](https://scholar.google.com/scholar?q=Charlie+Snell%2C+Ilya+Kostrikov%2C+Yi+Su%2C+Mengjiao+Yang%2C+and+Sergey+Levine%2E+Offline+rl+for+natural+language+generation+with+implicit+language+q+learning%2E+In+International+Conference+on+Learning+Representa)
48. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020. [find](https://scholar.google.com/scholar?q=Nisan+Stiennon%2C+Long+Ouyang%2C+Jeffrey+Wu%2C+Daniel+Ziegler%2C+Ryan+Lowe%2C+Chelsea+Voss%2C+Alec+Radford%2C+Dario+Amodei%2C+and+Paul+F+Christiano%2E+Learning+to+summarize+with+human+feedback%2E+Advances+in+Neural+Infor)
49. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca), 2023. [↗](https://github.com/tatsu-lab/stanford_alpaca)
50. Siddharth Verma, Justin Fu, Mengjiao Yang, and Sergey Levine. Chai: A chatbot ai for task-oriented dialogue with offline reinforcement learning, 2022. [find](https://scholar.google.com/scholar?q=Siddharth+Verma%2C+Justin+Fu%2C+Mengjiao+Yang%2C+and+Sergey+Levine%2E+Chai%3A+A+chatbot+ai+for+task%2Doriented+dialogue+with+offline+reinforcement+learning%2C+2022%2E)
51. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/), 2023.
52. Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. [Recursively summarizing books with human feedback](https://md.research.elamur.ai/free/en/papers/wu-2021-recursively-summarizing-books-human-feedback/). arXiv preprint arXiv:2109.10862, 2021.
53. Yuxiang Wu and Baotian Hu. Learning to extract coherent summary via deep reinforcement learning, 2018. [find](https://scholar.google.com/scholar?q=Yuxiang+Wu+and+Baotian+Hu%2E+Learning+to+extract+coherent+summary+via+deep+reinforcement+learning%2C+2018%2E)
54. Kevin Yang and Dan Klein. [FUDGE: Controlled text generation with future discriminators](https://md.research.elamur.ai/free/en/papers/yang-2021-fudge-controlled-text-generation-future/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3511–3535, Online, June 2021. Association for Computational L…
55. Li Zhong and Zilong Wang. A study on robustness and reliability of large language model code generation, 2023. [find](https://scholar.google.com/scholar?q=Li+Zhong+and+Zilong+Wang%2E+A+study+on+robustness+and+reliability+of+large+language+model+code+generation%2C+2023%2E)
56. Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2020. [find](https://scholar.google.com/scholar?q=Daniel+M%2E+Ziegler%2C+Nisan+Stiennon%2C+Jeffrey+Wu%2C+Tom+B%2E+Brown%2C+Alec+Radford%2C+Dario+Amodei%2C+Paul+Christiano%2C+and+Geoffrey+Irving%2E+Fine%2Dtuning+language+models+from+human+preferences%2C+2020%2E)

*Updated 3 November 2026*