---
title: "Reward Design with Language Models — Kwon 2023"
description: "The authors propose using a large language model (GPT-3) as a proxy reward function. 34 references, cited by 52 papers here."
authors: ["Minae Kwon", "Sang Michael Xie", "Kalesha Bullard", "Dorsa Sadigh"]
year: 2023
venue: "arXiv:2303.00001"
source: "https://arxiv.org/abs/2303.00001"
doi: "10.48550/arxiv.2303.00001"
arxiv: "2303.00001"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kwon-2023-reward-design-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kwon-2023-reward-design-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/) / Reward Design with Language Models

# Reward Design with Language Models

**Authors:** [Minae Kwon](https://md.research.elamur.ai/free/en/authors/minae-kwon/), [Sang Michael Xie](https://md.research.elamur.ai/free/en/authors/sang-michael-xie/), Kalesha Bullard and [Dorsa Sadigh](https://md.research.elamur.ai/free/en/authors/dorsa-sadigh/) · **Year:** 2023 · **Venue:** arXiv:2303.00001 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2303.00001) · [PDF](https://arxiv.org/pdf/2303.00001) · [DOI 10.48550/arxiv.2303.00001](https://doi.org/10.48550/arxiv.2303.00001) · [arXiv 2303.00001](https://arxiv.org/abs/2303.00001) · [OpenAlex](https://openalex.org/W4322825501)

**Citations:** 21 citations (OpenAlex, 2026-10-14) · [52](https://md.research.elamur.ai/free/en/papers/kwon-2023-reward-design-language-models/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Telling an RL agent what to do is difficult. Hand-written reward functions struggle to capture fuzzy goals, such as wanting a negotiator to be "versatile," and they are prone to reward hacking. Learning rewards from labels instead requires large, costly datasets that must be gathered again for every new user. The authors propose using a large language model (GPT-3) as a proxy reward function. The user writes a prompt once, containing either a few examples (few-shot) or a brief description (zero-shot) of the desired behavior. During training, a task-specific parser turns each episode into text. This text is joined with a task description, the user's objective and a question asking whether the objective was met. A second parser maps the LLM's reply to a binary reward. The setup works with any RL algorithm. Across the Ultimatum Game, two-player Matrix Games and the DealOrNoDeal negotiation task, the resulting agents matched user goals better than agents trained with supervised-learned rewards. This points to a cheap, language-based way to align agents with individual preferences without fine-tuning the LLM.

## Key points
- The prompt has four parts: task description, user objective (examples or description), textual episode outcome, and a yes/no question. The LLM output is converted to a 0/1 reward.
- Few-shot prompting was used for the Ultimatum Game and DealOrNoDeal (for example, three counterbalanced examples of "stubborn" or "versatile" negotiators). Zero-shot prompting was used for Matrix Games with objectives like Pareto-optimality, total welfare, equality and Rawlsian fairness.
- On average, LLM-based rewards trained objective-aligned agents about 35% more accurately than the supervised-learning baseline.
- In a pilot study with 10 human users, participants rated the agents as significantly better aligned with their own objective than agents trained for a different objective (p\<0.001).
- In Matrix Games, outcome orderings were scrambled to remove LLM order bias. Further analyses examine how much user data is required and how sensitive the reward is to prompt wording. The work was published at ICLR 2023.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 506 works · 330,052 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Minae Kwon, Sang Michael Xie, Kalesha Bullard and Dorsa Sadigh. 2023. Reward Design with Language Models. arXiv:2303.00001.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{kwon2023reward,
  title = {{Reward Design with Language Models}},
  author = {Minae Kwon, Sang Michael Xie, Kalesha Bullard and Dorsa Sadigh},
  year = {2023},
  doi = {10.48550/arxiv.2303.00001},
  eprint = {2303.00001},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.00001}
}
```

- [← previous ReCEval: Evaluating Reasoning Chains via Correctness and Informativeness](https://md.research.elamur.ai/free/en/papers/prasad-2023-receval-evaluating-reasoning-chains-correctness/)
- [next → ScandEval: A Benchmark for Scandinavian Natural Language Processing](https://md.research.elamur.ai/free/en/papers/nielsen-2023-scandeval-benchmark-scandinavian-natural-language/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/)

## References

34 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil… — [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://md.research.elamur.ai/free/en/papers/ahn-2022-i-i-say-grounding-language/)
2. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock… — [Flamingo: a Visual Language Model for Few-Shot Learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/)
3. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. [Concrete problems in ai safety](https://md.research.elamur.ai/free/en/papers/amodei-2016-concrete-problems-ai-safety/). arXiv preprint arXiv:1606.06565, 2016.
4. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+et+al%2E+Training+a+helpful+and+harmless+assistant+with+reinforcem)
5. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
6. Andrea Caputo, Oluremi B Ayoko, Nii Amoo, and Charlott Menke. The relationship between cultural values, cultural intelligence and negotiation styles. Journal of Business Research, 99:23–36, 2019. [find](https://scholar.google.com/scholar?q=Andrea+Caputo%2C+Oluremi+B+Ayoko%2C+Nii+Amoo%2C+and+Charlott+Menke%2E+The+relationship+between+cultural+values%2C+cultural+intelligence+and+negotiation+styles%2E+Journal+of+Business+Research%2C+99%3A23%E2%80%9336%2C+2019%2E)
7. Thomas Carta, Sylvain Lamprier, Pierre-Yves Oudeyer, and Olivier Sigaud. Eager: Asking and answering questions for automatic reward shaping in language-guided rl. arXiv preprint arXiv:2206.09674, 2022. — [arXiv:2206.09674](https://arxiv.org/abs/2206.09674) [find](https://scholar.google.com/scholar?q=Thomas+Carta%2C+Sylvain+Lamprier%2C+Pierre%2DYves+Oudeyer%2C+and+Olivier+Sigaud%2E+Eager%3A+Asking+and+answering+questions+for+automatic+reward+shaping+in+language%2Dguided+rl%2E+arXiv+preprint+arXiv%3A2206%2E09674%2C+2022)
8. Annie S. Chen, Suraj Nair, and Chelsea Finn. Learning generalizable robotic reward functions from ”in-the-wild” human videos. ArXiv, abs/2103.16817, 2021. — [arXiv:2103.16817](https://arxiv.org/abs/2103.16817) [find](https://scholar.google.com/scholar?q=Annie+S%2E+Chen%2C+Suraj+Nair%2C+and+Chelsea+Finn%2E+Learning+generalizable+robotic+reward+functions+from+%E2%80%9Din%2Dthe%2Dwild%E2%80%9D+human+videos%2E+ArXiv%2C+abs%2F2103%2E16817%2C+2021%2E+%E2%80%94+%5BarXiv%3A2103%2E16817%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F21)
9. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. [find](https://scholar.google.com/scholar?q=Paul+F+Christiano%2C+Jan+Leike%2C+Tom+Brown%2C+Miljan+Martic%2C+Shane+Legg%2C+and+Dario+Amodei%2E+Deep+reinforcement+learning+from+human+preferences%2E+Advances+in+neural+information+processing+systems%2C+30%2C+2017%2E)
10. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014. [find](https://scholar.google.com/scholar?q=Junyoung+Chung%2C+Caglar+Gulcehre%2C+KyungHyun+Cho%2C+and+Yoshua+Bengio%2E+Empirical+evaluation+of+gated+recurrent+neural+networks+on+sequence+modeling%2E+arXiv+preprint+arXiv%3A1412%2E3555%2C+2014%2E)
11. Ernst Fehr and Klaus M Schmidt. On inequity aversion: A reply to binmore and shaked. Journal of economic behavior & organization, 73(1):101–108, 2010. [find](https://scholar.google.com/scholar?q=Ernst+Fehr+and+Klaus+M+Schmidt%2E+On+inequity+aversion%3A+A+reply+to+binmore+and+shaked%2E+Journal+of+economic+behavior+%26+organization%2C+73%281%29%3A101%E2%80%93108%2C+2010%2E)
12. Prasoon Goyal, Scott Niekum, and Raymond J Mooney. Using natural language for reward shaping in reinforcement learning. arXiv preprint arXiv:1903.02020, 2019. — [arXiv:1903.02020](https://arxiv.org/abs/1903.02020) [find](https://scholar.google.com/scholar?q=Prasoon+Goyal%2C+Scott+Niekum%2C+and+Raymond+J+Mooney%2E+Using+natural+language+for+reward+shaping+in+reinforcement+learning%2E+arXiv+preprint+arXiv%3A1903%2E02020%2C+2019%2E+%E2%80%94+%5BarXiv%3A1903%2E02020%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
13. Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan. Inverse reward design. Advances in neural information processing systems, 30, 2017. [find](https://scholar.google.com/scholar?q=Dylan+Hadfield%2DMenell%2C+Smitha+Milli%2C+Pieter+Abbeel%2C+Stuart+J+Russell%2C+and+Anca+Dragan%2E+Inverse+reward+design%2E+Advances+in+neural+information+processing+systems%2C+30%2C+2017%2E)
14. He He, Derek Chen, Anusha Balakrishnan, and Percy Liang. [Decoupling strategy and generation in negotiation dialogues](https://md.research.elamur.ai/free/en/papers/he-2018-decoupling-strategy-generation-negotiation-dialogues/). In Empirical Methods in Natural Language Processing (EMNLP), 2018.
15. Wenlong Huang, P. Abbeel, Deepak Pathak, and Igor Mordatch. [Language models as zero-shot planners: Extracting actionable knowledge for embodied agents](https://md.research.elamur.ai/free/en/papers/huang-2022-language-models-zero-shot-planners/). In ICML, 2022.
16. Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018. [find](https://scholar.google.com/scholar?q=Borja+Ibarz%2C+Jan+Leike%2C+Tobias+Pohlen%2C+Geoffrey+Irving%2C+Shane+Legg%2C+and+Dario+Amodei%2E+Reward+learning+from+human+preferences+and+demonstrations+in+atari%2E+Advances+in+neural+information+processing+syst)
17. W Bradley Knox, Alessandro Allievi, Holger Banzhaf, Felix Schmitt, and Peter Stone. Reward (mis) design for autonomous driving. arXiv preprint arXiv:2104.13906, 2021. — [arXiv:2104.13906](https://arxiv.org/abs/2104.13906) [find](https://scholar.google.com/scholar?q=W+Bradley+Knox%2C+Alessandro+Allievi%2C+Holger+Banzhaf%2C+Felix+Schmitt%2C+and+Peter+Stone%2E+Reward+%28mis%29+design+for+autonomous+driving%2E+arXiv+preprint+arXiv%3A2104%2E13906%2C+2021%2E+%E2%80%94+%5BarXiv%3A2104%2E13906%5D%28https%3A%2F%2Farxi)
18. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). arXiv, 2022.
19. Minae Kwon, Siddharth Karamcheti, Mariano-Florentino Cuellar, and Dorsa Sadigh. Targeted data acquisition for evolving negotiation agents. In International Conference on Machine Learning, pp. 5894–5904. PMLR, 2021. [find](https://scholar.google.com/scholar?q=Minae+Kwon%2C+Siddharth+Karamcheti%2C+Mariano%2DFlorentino+Cuellar%2C+and+Dorsa+Sadigh%2E+Targeted+data+acquisition+for+evolving+negotiation+agents%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+5894%E2%80%93590)
20. Andrew K Lampinen, Ishita Dasgupta, Stephanie CY Chan, Kory Matthewson, Michael Henry Tessler, Antonia Creswell, James L McClelland, Jane X Wang, and Felix Hill. [Can language models learn from explanations in context?](https://md.research.elamur.ai/free/en/papers/lampinen-2022-can-language-models-learn-explanations/) arXiv preprint arXiv:2204.02329, 2022.
21. Mike Lewis, Denis Yarats, Yann N. Dauphin, Devi Parikh, and Dhruv Batra. Deal or no deal? end-to-end learning for negotiation dialogues. In Empirical Methods in Natural Language Processing (EMNLP), 2017. [find](https://scholar.google.com/scholar?q=Mike+Lewis%2C+Denis+Yarats%2C+Yann+N%2E+Dauphin%2C+Devi+Parikh%2C+and+Dhruv+Batra%2E+Deal+or+no+deal%3F+end%2Dto%2Dend+learning+for+negotiation+dialogues%2E+In+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+20)
22. Jessy Lin, Daniel Fried, Dan Klein, and Anca Dragan. Inferring rewards from language in context. arXiv preprint arXiv:2204.02515, 2022. [find](https://scholar.google.com/scholar?q=Jessy+Lin%2C+Daniel+Fried%2C+Dan+Klein%2C+and+Anca+Dragan%2E+Inferring+rewards+from+language+in+context%2E+arXiv+preprint+arXiv%3A2204%2E02515%2C+2022%2E)
23. Suvir Mirchandani, Siddharth Karamcheti, and Dorsa Sadigh. Ella: Exploration through learned language abstraction. Advances in Neural Information Processing Systems, 34:29529–29540, 2021. [find](https://scholar.google.com/scholar?q=Suvir+Mirchandani%2C+Siddharth+Karamcheti%2C+and+Dorsa+Sadigh%2E+Ella%3A+Exploration+through+learned+language+abstraction%2E+Advances+in+Neural+Information+Processing+Systems%2C+34%3A29529%E2%80%9329540%2C+2021%2E)
24. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Low… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
25. Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. arXiv preprint arXiv:2201.03544, 2022. [find](https://scholar.google.com/scholar?q=Alexander+Pan%2C+Kush+Bhatia%2C+and+Jacob+Steinhardt%2E+The+effects+of+reward+misspecification%3A+Mapping+and+mitigating+misaligned+models%2E+arXiv+preprint+arXiv%3A2201%2E03544%2C+2022%2E)
26. Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Kumar Gupta. The unsurprising effectiveness of pre-trained vision models for control. In ICML, 2022. [find](https://scholar.google.com/scholar?q=Simone+Parisi%2C+Aravind+Rajeswaran%2C+Senthil+Purushwalkam%2C+and+Abhinav+Kumar+Gupta%2E+The+unsurprising+effectiveness+of+pre%2Dtrained+vision+models+for+control%2E+In+ICML%2C+2022%2E)
27. Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1–8, 2021. URL [http://jmlr.org/papers/v22/20-1364.html](http://jmlr.org/papers/v22/20-1364.html). [↗](http://jmlr.org/papers/v22/20-1364.html)
28. Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627–635. JMLR Workshop and Conference Proceedings, 201… [find](https://scholar.google.com/scholar?q=Ste%CC%81phane+Ross%2C+Geoffrey+Gordon%2C+and+Drew+Bagnell%2E+A+reduction+of+imitation+learning+and+structured+prediction+to+no%2Dregret+online+learning%2E+In+Proceedings+of+the+fourteenth+international+conference+o)
29. Dorsa Sadigh, Anca D. Dragan, S. Shankar Sastry, and Sanjit A. Seshia. Active preference-based learning of reward functions. In Proceedings of Robotics: Science and Systems (RSS), July 2017. doi: 10.15607/RSS.2017.XIII.053. — [doi:10.15607/rss.2017.xiii.053](https://doi.org/10.15607/rss.2017.xiii.053) [find](https://scholar.google.com/scholar?q=Dorsa+Sadigh%2C+Anca+D%2E+Dragan%2C+S%2E+Shankar+Sastry%2C+and+Sanjit+A%2E+Seshia%2E+Active+preference%2Dbased+learning+of+reward+functions%2E+In+Proceedings+of+Robotics%3A+Science+and+Systems+%28RSS%29%2C+July+2017%2E+doi%3A+10%2E1)
30. Katia Sycara, Daniel Zeng, et al. Benefits of learning in negotiation. In Proceedings of the AAAI National Conference on Artificial Intelligence. Menlo Park, California, pp. 36–41, 1997. [find](https://scholar.google.com/scholar?q=Katia+Sycara%2C+Daniel+Zeng%2C+et+al%2E+Benefits+of+learning+in+negotiation%2E+In+Proceedings+of+the+AAAI+National+Conference+on+Artificial+Intelligence%2E+Menlo+Park%2C+California%2C+pp%2E+36%E2%80%9341%2C+1997%2E)
31. Peter Vavra, Luke J Chang, and Alan G Sanfey. Expectations in the ultimatum game: distinct effects of mean and variance of expected offers. Frontiers in psychology, 9:992, 2018. [find](https://scholar.google.com/scholar?q=Peter+Vavra%2C+Luke+J+Chang%2C+and+Alan+G+Sanfey%2E+Expectations+in+the+ultimatum+game%3A+distinct+effects+of+mean+and+variance+of+expected+offers%2E+Frontiers+in+psychology%2C+9%3A992%2C+2018%2E)
32. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229–256, 1992. [find](https://scholar.google.com/scholar?q=Ronald+J+Williams%2E+Simple+statistical+gradient%2Dfollowing+algorithms+for+connectionist+reinforcement+learning%2E+Machine+learning%2C+8%283%29%3A229%E2%80%93256%2C+1992%2E)
33. Jing Zhang, Xindong Wu, and Victor S Sheng. Learning from crowdsourced labeled data: a survey. Artificial Intelligence Review, 46(4):543–576, 2016. [find](https://scholar.google.com/scholar?q=Jing+Zhang%2C+Xindong+Wu%2C+and+Victor+S+Sheng%2E+Learning+from+crowdsourced+labeled+data%3A+a+survey%2E+Artificial+Intelligence+Review%2C+46%284%29%3A543%E2%80%93576%2C+2016%2E)
34. Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. [Calibrate before use: Improving few-shot performance of language models](https://md.research.elamur.ai/free/en/papers/zhao-2021-calibrate-before-use-improving-few/). In International Conference on Machine Learning (ICML), 2021.

*Updated 2 November 2026*