---
title: "WorldCoder, a Model-Based LLM Agent: Building World Models"
description: "The authors present WorldCoder, a model-based reinforcement learning agent. 80 references, cited by 23 papers here."
authors: ["Hao Tang", "Darren Key", "Kevin Ellis"]
year: 2024
venue: "arXiv:2402.12275"
source: "https://arxiv.org/abs/2402.12275"
doi: "10.48550/arxiv.2402.12275"
arxiv: "2402.12275"
licence: "CC BY-NC-SA 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-nc-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/tang-2024-worldcoder-model-based-llm/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tang-2024-worldcoder-model-based-llm/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/10/) / WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environment

# WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environment

**Authors:** [Hao Tang](https://md.research.elamur.ai/free/en/authors/hao-tang/), Darren Key and [Kevin Ellis](https://md.research.elamur.ai/free/en/authors/kevin-ellis/) · **Year:** 2024 · **Venue:** [arXiv:2402.12275](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/) ([NeurIPS 2024](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/2024/)) · **Licence:** [CC BY-NC-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-nc-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2402.12275) · [PDF](https://arxiv.org/pdf/2402.12275) · [DOI 10.48550/arxiv.2402.12275](https://doi.org/10.48550/arxiv.2402.12275) · [arXiv 2402.12275](https://arxiv.org/abs/2402.12275) · [OpenAlex](https://openalex.org/W4415796843)

**Citations:** 9 citations (OpenAlex, 2026-10-14) · [23](https://md.research.elamur.ai/free/en/papers/tang-2024-worldcoder-model-based-llm/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks how an AI agent could learn how an unfamiliar environment works from only a few interactions, then reuse that knowledge for new goals. The authors present WorldCoder, a model-based reinforcement learning agent. It uses an LLM to write and debug a Python program that serves as an explicit world model, with separate transition and reward functions. The LLM does not simulate the world or plan. It builds the simulator, and a conventional planner chooses actions. The program must reproduce every logged transition and reward. It must also be optimistic: given a natural-language goal, a planner must be able to reach positive reward under the model. This constraint pushes the agent to invent plausible intermediate goals, which drives exploration even when rewards are sparse. In experiments on gridworlds and robot-style task planning, the agent needed far fewer environment interactions than deep RL and far fewer LLM calls than ReAct-style agents. It also transferred knowledge across environments by editing existing code. The work shows that general-purpose code can act as an interpretable, reusable world model.

## Key points
- **Formal setup:** The setting is a Contextual MDP (Hallak et al.) where goals are the contexts, dynamics are deterministic and shared across goals, and states are sets of objects (an Object Oriented MDP).
- **Two learning constraints:** ϕ1 requires the model to fit all replayed data. ϕ2 is an optimism constraint requiring that some action sequence reaches the goal with positive reward. A world model satisfying both is found by LLM program synthesis, with back-prompting on any mismatched transitions.
- **Theoretical guarantee:** The analysis is non-probabilistic and combinatorial. It shows that a good-enough world model is found in polynomial time, using new notions of "logical dimensionality" of the model space and MDP diameter.
- **Efficiency claims:** Once a world model is learned, a new task needs at most one LLM query (O(1)) to update the reward function. ReAct-style agents need about O(T) calls per task, where T is the episode length. The optimism constraint is reported to cut the interactions needed to obtain reward by orders of magnitude.
- **Limitations:** The method assumes deterministic dynamics and symbolic, discrete states. Its knowledge-transfer mechanism is basic, and the authors suggest library learning as future work.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works

## Cite

Cite as: Hao Tang, Darren Key and Kevin Ellis. 2024. WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environment. arXiv:2402.12275.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{tang2024worldcoder,
  title = {{WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environment}},
  author = {Hao Tang, Darren Key and Kevin Ellis},
  year = {2024},
  doi = {10.48550/arxiv.2402.12275},
  eprint = {2402.12275},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.12275}
}
```

- [← previous StepCoder: Improving Code Generation with Reinforcement Learning from Compiler Feedback](https://md.research.elamur.ai/free/en/papers/dou-2024-stepcoder-improving-code-generation-reinforcement/)
- [next → A Survey of Small Language Models](https://md.research.elamur.ai/free/en/papers/nguyen-2024-survey-small-language-models/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/10/)

## References

80 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil… — [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://md.research.elamur.ai/free/en/papers/ahn-2022-i-i-say-grounding-language/)
2. Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research, 3(Nov):397–422, 2002. [find](https://scholar.google.com/scholar?q=Peter+Auer%2E+Using+confidence+bounds+for+exploitation%2Dexploration+trade%2Doffs%2E+Journal+of+Machine+Learning+Research%2C+3%28Nov%29%3A397%E2%80%93422%2C+2002%2E)
3. Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforcement learning. In International conference on machine learning, pages 263–272. PMLR, 2017. [find](https://scholar.google.com/scholar?q=Mohammad+Gheshlaghi+Azar%2C+Ian+Osband%2C+and+R%C3%A9mi+Munos%2E+Minimax+regret+bounds+for+reinforcement+learning%2E+In+International+conference+on+machine+learning%2C+pages+263%E2%80%93272%2E+PMLR%2C+2017%2E)
4. Matthew Bowers, Theo X. Olausson, Lionel Wong, Gabriel Grand, Joshua B. Tenenbaum, Kevin Ellis, and Armando Solar-Lezama. Top-down synthesis for library learning. Proc. ACM Program. Lang., 7(POPL), jan 2023. doi: 10.1145/3571234. URL https://doi.org/10.1145/3571234. [↗](https://doi.org/10.1145/3571234)
5. Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter I Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in games, 4(1):1–43, 2012… [find](https://scholar.google.com/scholar?q=Cameron+B+Browne%2C+Edward+Powley%2C+Daniel+Whitehouse%2C+Simon+M+Lucas%2C+Peter+I+Cowling%2C+Philipp+Rohlfshagen%2C+Stephen+Tavener%2C+Diego+Perez%2C+Spyridon+Samothrakis%2C+and+Simon+Colton%2E+A+survey+of+monte+carlo+t)
6. Swarat Chaudhuri, Kevin Ellis, Oleksandr Polozov, Rishabh Singh, Armando Solar-Lezama, Yisong Yue, et al. Neurosymbolic programming. Foundations and Trends® in Programming Languages, 7(3), 2021. [find](https://scholar.google.com/scholar?q=Swarat+Chaudhuri%2C+Kevin+Ellis%2C+Oleksandr+Polozov%2C+Rishabh+Singh%2C+Armando+Solar%2DLezama%2C+Yisong+Yue%2C+et+al%2E+Neurosymbolic+programming%2E+Foundations+and+Trends%C2%AE+in+Programming+Languages%2C+7%283%29%2C+2021%2E)
7. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128, 2023. [find](https://scholar.google.com/scholar?q=Xinyun+Chen%2C+Maxwell+Lin%2C+Nathanael+Sch%C3%A4rli%2C+and+Denny+Zhou%2E+Teaching+large+language+models+to+self%2Ddebug%2E+arXiv+preprint+arXiv%3A2304%2E05128%2C+2023%2E)
8. Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. [https://github.com/maximecb/gym-minigrid](https://github.com/maximecb/gym-minigrid), 2018. [↗](https://github.com/maximecb/gym-minigrid)
9. Maxime Chevalier-Boisvert, Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. [Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks](https://md.research.elamur.ai/free/en/papers/chevalier-boisvert-2023-minigrid-miniworld-modular-customizable-reinforcement/). CoRR, abs/2306.13831, 2023.
10. Leah Chrestien, Stefan Edelkamp, Antonin Komenda, and Tomáš Pevný. Optimize planning heuristics to rank, not to estimate cost-to-goal. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL [https://openreview.net/forum?id=Mgy6sgslPY](https://openreview.net/forum?id=Mgy6sgslPY). [↗](https://openreview.net/forum?id=Mgy6sgslPY)
11. Stephen Chung, Ivan Anokhin, and David Krueger. Thinker: Learning to plan and act. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL [https://openreview.net/forum?id=mumEBl0arj](https://openreview.net/forum?id=mumEBl0arj). [↗](https://openreview.net/forum?id=mumEBl0arj)
12. Rémi Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In International conference on computers and games, pages 72–83. Springer, 2006. [find](https://scholar.google.com/scholar?q=R%C3%A9mi+Coulom%2E+Efficient+selectivity+and+backup+operators+in+monte%2Dcarlo+tree+search%2E+In+International+conference+on+computers+and+games%2C+pages+72%E2%80%9383%2E+Springer%2C+2006%2E)
13. Ria Das, Joshua B. Tenenbaum, Armando Solar-Lezama, and Zenna Tavares. Combining functional and automata synthesis to discover causal reactive programs. Proc. ACM Program. Lang., 7(POPL), jan 2023. doi: 10.1145/3571249. URL https://doi.org/10.1145/3571249. — [doi:10.1145/3571249](https://doi.org/10.1145/3571249) [↗](https://doi.org/10.1145/3571249)
14. Luc De Raedt, Angelika Kimmig, and Hannu Toivonen. Problog: a probabilistic prolog and its application in link discovery. In Proceedings of the 20th International Joint Conference on Artifical Intelligence, IJCAI’07, page 2468–2473, San Francisco, CA, USA, 2007. Morgan Kaufmann Publishers Inc. [find](https://scholar.google.com/scholar?q=Luc+De+Raedt%2C+Angelika+Kimmig%2C+and+Hannu+Toivonen%2E+Problog%3A+a+probabilistic+prolog+and+its+application+in+link+discovery%2E+In+Proceedings+of+the+20th+International+Joint+Conference+on+Artifical+Intelli)
15. Carlos Diuk, Andre Cohen, and Michael L Littman. An object-oriented representation for efficient reinforcement learning. In Proceedings of the 25th international conference on Machine learning, pages 240–247, 2008. [find](https://scholar.google.com/scholar?q=Carlos+Diuk%2C+Andre+Cohen%2C+and+Michael+L+Littman%2E+An+object%2Doriented+representation+for+efficient+reinforcement+learning%2E+In+Proceedings+of+the+25th+international+conference+on+Machine+learning%2C+pages+)
16. Kevin Ellis, Lionel Wong, Maxwell Nye, Mathias Sable-Meyer, Luc Cary, Lore Anaya Pozo, Luke Hewitt, Armando Solar-Lezama, and Joshua B Tenenbaum. Dreamcoder: growing generalizable, interpretable knowledge with wake–sleep bayesian program learning. Philosophical Transactions of the Royal Society A, 3… [find](https://scholar.google.com/scholar?q=Kevin+Ellis%2C+Lionel+Wong%2C+Maxwell+Nye%2C+Mathias+Sable%2DMeyer%2C+Luc+Cary%2C+Lore+Anaya+Pozo%2C+Luke+Hewitt%2C+Armando+Solar%2DLezama%2C+and+Joshua+B+Tenenbaum%2E+Dreamcoder%3A+growing+generalizable%2C+interpretable+knowl)
17. Richard Evans, Matko Bošnjak, Lars Buesing, Kevin Ellis, David Pfau, Pushmeet Kohli, and Marek Sergot. Making sense of raw input. Artificial Intelligence, 299:103521, 2021. [find](https://scholar.google.com/scholar?q=Richard+Evans%2C+Matko+Bo%C5%A1njak%2C+Lars+Buesing%2C+Kevin+Ellis%2C+David+Pfau%2C+Pushmeet+Kohli%2C+and+Marek+Sergot%2E+Making+sense+of+raw+input%2E+Artificial+Intelligence%2C+299%3A103521%2C+2021%2E)
18. Dieqiao Feng, Carla P Gomes, and Bart Selman. A novel automated curriculum strategy to solve hard sokoban planning instances. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 3141–3152. Curran Associates,… [find](https://scholar.google.com/scholar?q=Dieqiao+Feng%2C+Carla+P+Gomes%2C+and+Bart+Selman%2E+A+novel+automated+curriculum+strategy+to+solve+hard+sokoban+planning+instances%2E+In+H%2E+Larochelle%2C+M%2E+Ranzato%2C+R%2E+Hadsell%2C+M%2EF%2E+Balcan%2C+and+H%2E+Lin%2C+editors)
19. Dieqiao Feng, Carla Gomes, and Bart Selman. Left heavy tails and the effectiveness of the policy and value networks in dnn-based best-first search for sokoban planning, 2022. [find](https://scholar.google.com/scholar?q=Dieqiao+Feng%2C+Carla+Gomes%2C+and+Bart+Selman%2E+Left+heavy+tails+and+the+effectiveness+of+the+policy+and+value+networks+in+dnn%2Dbased+best%2Dfirst+search+for+sokoban+planning%2C+2022%2E)
20. Caelan Reed Garrett, Rohan Chitnis, Rachel Holladay, Beomjoon Kim, Tom Silver, Leslie Pack Kaelbling, and Tomás Lozano-Pérez. Integrated task and motion planning, 2020. [find](https://scholar.google.com/scholar?q=Caelan+Reed+Garrett%2C+Rohan+Chitnis%2C+Rachel+Holladay%2C+Beomjoon+Kim%2C+Tom+Silver%2C+Leslie+Pack+Kaelbling%2C+and+Tom%C3%A1s+Lozano%2DP%C3%A9rez%2E+Integrated+task+and+motion+planning%2C+2020%2E)
21. Noah D. Goodman, Vikash K. Mansinghka, Daniel Roy, Keith Bonawitz, and Joshua B. Tenenbaum. Church: a language for generative models. In Proceedings of the Twenty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI’08, page 220–229, Arlington, Virginia, USA, 2008. AUAI Press. ISBN 09749… [find](https://scholar.google.com/scholar?q=Noah+D%2E+Goodman%2C+Vikash+K%2E+Mansinghka%2C+Daniel+Roy%2C+Keith+Bonawitz%2C+and+Joshua+B%2E+Tenenbaum%2E+Church%3A+a+language+for+generative+models%2E+In+Proceedings+of+the+Twenty%2DFourth+Conference+on+Uncertainty+in+A)
22. Anirudh Goyal, Aniket Didolkar, Nan Rosemary Ke, Charles Blundell, Philippe Beaudoin, Nicolas Heess, Michael Mozer, and Yoshua Bengio. Neural production systems: Learning rule-governed visual dynamics. arXiv preprint arXiv:2103.01937, 2021. — [arXiv:2103.01937](https://arxiv.org/abs/2103.01937) [find](https://scholar.google.com/scholar?q=Anirudh+Goyal%2C+Aniket+Didolkar%2C+Nan+Rosemary+Ke%2C+Charles+Blundell%2C+Philippe+Beaudoin%2C+Nicolas+Heess%2C+Michael+Mozer%2C+and+Yoshua+Bengio%2E+Neural+production+systems%3A+Learning+rule%2Dgoverned+visual+dynamics)
23. Gabriel Grand, Lionel Wong, Matthew Bowers, Theo X. Olausson, Muxin Liu, Joshua B. Tenenbaum, and Jacob Andreas. Lilo: Learning interpretable libraries by compressing and documenting code, 2023. [find](https://scholar.google.com/scholar?q=Gabriel+Grand%2C+Lionel+Wong%2C+Matthew+Bowers%2C+Theo+X%2E+Olausson%2C+Muxin+Liu%2C+Joshua+B%2E+Tenenbaum%2C+and+Jacob+Andreas%2E+Lilo%3A+Learning+interpretable+libraries+by+compressing+and+documenting+code%2C+2023%2E)
24. Lin Guan, Karthik Valmeekam, Sarath Sreedharan, and Subbarao Kambhampati. [Leveraging pre-trained large language models to construct and utilize world models for model-based task planning](https://md.research.elamur.ai/free/en/papers/guan-2023-leveraging-pre-trained-large-language/). NeurIPS, 2023.
25. David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems 31, pages 2451–2463. Curran Associates, Inc., 2018. [https://worldmodels.github.io](https://worldmodels.github.io). [↗](https://worldmodels.github.io)
26. Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. ICLR, 2021. [find](https://scholar.google.com/scholar?q=Danijar+Hafner%2C+Timothy+Lillicrap%2C+Mohammad+Norouzi%2C+and+Jimmy+Ba%2E+Mastering+atari+with+discrete+world+models%2E+ICLR%2C+2021%2E)
27. Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models, 2023. [find](https://scholar.google.com/scholar?q=Danijar+Hafner%2C+Jurgis+Pasukonis%2C+Jimmy+Ba%2C+and+Timothy+Lillicrap%2E+Mastering+diverse+domains+through+world+models%2C+2023%2E)
28. Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual markov decision processes. arXiv preprint arXiv:1502.02259, 2015. — [arXiv:1502.02259](https://arxiv.org/abs/1502.02259) [find](https://scholar.google.com/scholar?q=Assaf+Hallak%2C+Dotan+Di+Castro%2C+and+Shie+Mannor%2E+Contextual+markov+decision+processes%2E+arXiv+preprint+arXiv%3A1502%2E02259%2C+2015%2E+%E2%80%94+%5BarXiv%3A1502%2E02259%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1502%2E02259%29)
29. Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992, 2023. [find](https://scholar.google.com/scholar?q=Shibo+Hao%2C+Yi+Gu%2C+Haodi+Ma%2C+Joshua+Jiahua+Hong%2C+Zhen+Wang%2C+Daisy+Zhe+Wang%2C+and+Zhiting+Hu%2E+Reasoning+with+language+model+is+planning+with+world+model%2E+arXiv+preprint+arXiv%3A2305%2E14992%2C+2023%2E)
30. Yingdong Hu, Fanqi Lin, Tong Zhang, Li Yi, and Yang Gao. Look before you leap: Unveiling the power of gpt-4v in robotic vision-language planning. arXiv preprint arXiv:2311.17842, 2023. — [arXiv:2311.17842](https://arxiv.org/abs/2311.17842) [find](https://scholar.google.com/scholar?q=Yingdong+Hu%2C+Fanqi+Lin%2C+Tong+Zhang%2C+Li+Yi%2C+and+Yang+Gao%2E+Look+before+you+leap%3A+Unveiling+the+power+of+gpt%2D4v+in+robotic+vision%2Dlanguage+planning%2E+arXiv+preprint+arXiv%3A2311%2E17842%2C+2023%2E+%E2%80%94+%5BarXiv%3A2311%2E1)
31. Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on learning theory, pages 2137– 2143. PMLR, 2020. [find](https://scholar.google.com/scholar?q=Chi+Jin%2C+Zhuoran+Yang%2C+Zhaoran+Wang%2C+and+Michael+I+Jordan%2E+Provably+efficient+reinforcement+learning+with+linear+function+approximation%2E+In+Conference+on+learning+theory%2C+pages+2137%E2%80%93+2143%2E+PMLR%2C+2020%2E)
32. Leslie Pack Kaelbling and Tomás Lozano-Pérez. Hierarchical task and motion planning in the now. In 2011 IEEE International Conference on Robotics and Automation, pages 1470–1477, 2011. doi: 10.1109/ICRA.2011.5980391. — [doi:10.1109/icra.2011.5980391](https://doi.org/10.1109/icra.2011.5980391) [find](https://scholar.google.com/scholar?q=Leslie+Pack+Kaelbling+and+Tom%C3%A1s+Lozano%2DP%C3%A9rez%2E+Hierarchical+task+and+motion+planning+in+the+now%2E+In+2011+IEEE+International+Conference+on+Robotics+and+Automation%2C+pages+1470%E2%80%931477%2C+2011%2E+doi%3A+10%2E1109%2FIC)
33. Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. [Dense passage retrieval for open-domain question answering](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/). arXiv preprint arXiv:2004.04906, 2020.
34. Mei Kobayashi and Koichi Takeda. Information retrieval on the web. ACM computing surveys (CSUR), 32(2):144–173, 2000. [find](https://scholar.google.com/scholar?q=Mei+Kobayashi+and+Koichi+Takeda%2E+Information+retrieval+on+the+web%2E+ACM+computing+surveys+%28CSUR%29%2C+32%282%29%3A144%E2%80%93173%2C+2000%2E)
35. Levente Kocsis and Csaba Szepesvári. Bandit based monte-carlo planning. In European conference on machine learning, pages 282–293. Springer, 2006. [find](https://scholar.google.com/scholar?q=Levente+Kocsis+and+Csaba+Szepesv%C3%A1ri%2E+Bandit+based+monte%2Dcarlo+planning%2E+In+European+conference+on+machine+learning%2C+pages+282%E2%80%93293%2E+Springer%2C+2006%2E)
36. George Konidaris, Leslie Pack Kaelbling, and Tomas Lozano-Perez. From skills to symbols: Learning symbolic representations for abstract high-level planning. Journal of Artificial Intelligence Research, 61:215–289, 2018. [find](https://scholar.google.com/scholar?q=George+Konidaris%2C+Leslie+Pack+Kaelbling%2C+and+Tomas+Lozano%2DPerez%2E+From+skills+to+symbols%3A+Learning+symbolic+representations+for+abstract+high%2Dlevel+planning%2E+Journal+of+Artificial+Intelligence+Research)
37. Kalle Kujanpää, Joni Pajarinen, and Alexander Ilin. Hybrid search for efficient planning with completeness guarantees. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL [https://openreview.net/forum?id=bY0c46ZtXa](https://openreview.net/forum?id=bY0c46ZtXa). [↗](https://openreview.net/forum?id=bY0c46ZtXa)
38. Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring a sequence model trained on a synthetic task. ICLR, 2023. [find](https://scholar.google.com/scholar?q=Kenneth+Li%2C+Aspen+K+Hopkins%2C+David+Bau%2C+Fernanda+Vi%C3%A9gas%2C+Hanspeter+Pfister%2C+and+Martin+Wattenberg%2E+Emergent+world+representations%3A+Exploring+a+sequence+model+trained+on+a+synthetic+task%2E+ICLR%2C+2023%2E)
39. Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In arXiv preprint arXiv:2209.07753, 2022. [find](https://scholar.google.com/scholar?q=Jacky+Liang%2C+Wenlong+Huang%2C+Fei+Xia%2C+Peng+Xu%2C+Karol+Hausman%2C+Brian+Ichter%2C+Pete+Florence%2C+and+Andy+Zeng%2E+Code+as+policies%3A+Language+model+programs+for+embodied+control%2E+In+arXiv+preprint+arXiv%3A2209%2E07)
40. Qinghua Liu, Praneeth Netrapalli, Csaba Szepesvari, and Chi Jin. Optimistic mle: A generic model-based algorithm for partially observable sequential decision making. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing, pages 363–376, 2023. [find](https://scholar.google.com/scholar?q=Qinghua+Liu%2C+Praneeth+Netrapalli%2C+Csaba+Szepesvari%2C+and+Chi+Jin%2E+Optimistic+mle%3A+A+generic+model%2Dbased+algorithm+for+partially+observable+sequential+decision+making%2E+In+Proceedings+of+the+55th+Annual+)
41. Zhihan Liu, Hao Hu, Shenao Zhang, Hongyi Guo, Shuqi Ke, Boyi Liu, and Zhaoran Wang. [Reason for future, act for now: A principled framework for autonomous llm agents with provable sample efficiency](https://md.research.elamur.ai/free/en/papers/liu-2023-reason-future-act-now-principled/). arXiv preprint arXiv:2309.17382, 2023.
42. Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. [Eureka: Human-level reward design via coding large language models](https://md.research.elamur.ai/free/en/papers/ma-2023-eureka-human-level-reward-design/). arXiv preprint arXiv: Arxiv-2310.12931, 2023.
43. Christopher D Manning, Prabhakar Raghavan, and Hinrich Schütze. Introduction to information retrieval. Cambridge university press, 2008. [find](https://scholar.google.com/scholar?q=Christopher+D+Manning%2C+Prabhakar+Raghavan%2C+and+Hinrich+Sch%C3%BCtze%2E+Introduction+to+information+retrieval%2E+Cambridge+university+press%2C+2008%2E)
44. Jiayuan Mao, Tomás Lozano-Pérez, Josh Tenenbaum, and Leslie Kaelbling. Pdsketch: Integrated domain programming, learning, and planning. Advances in Neural Information Processing Systems, 35:36972–36984, 2022. [find](https://scholar.google.com/scholar?q=Jiayuan+Mao%2C+Tom%C3%A1s+Lozano%2DP%C3%A9rez%2C+Josh+Tenenbaum%2C+and+Leslie+Kaelbling%2E+Pdsketch%3A+Integrated+domain+programming%2C+learning%2C+and+planning%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A36972%E2%80%933698)
45. Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample-efficient world models. In The Eleventh International Conference on Learning Representations, 2023. URL [https://openreview.net/forum?id=vhFu1Acb0xb](https://openreview.net/forum?id=vhFu1Acb0xb). [↗](https://openreview.net/forum?id=vhFu1Acb0xb)
46. Kolby Nottingham, Prithviraj Ammanabrolu, Alane Suhr, Yejin Choi, Hannaneh Hajishirzi, Sameer Singh, and Roy Fox. Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modelling. In International Conference on Machine Learning (ICML), 2023. [find](https://scholar.google.com/scholar?q=Kolby+Nottingham%2C+Prithviraj+Ammanabrolu%2C+Alane+Suhr%2C+Yejin+Choi%2C+Hannaneh+Hajishirzi%2C+Sameer+Singh%2C+and+Roy+Fox%2E+Do+Embodied+Agents+Dream+of+Pixelated+Sheep%3A+Embodied+Decision+Making+using+Language+G)
47. Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. [Is self-repair a silver bullet for code generation?](https://md.research.elamur.ai/free/en/papers/olausson-2024-self-repair-silver-bullet-code/), 2023. Rl baselines3
48. Antonin Raffin. rl-baselines3-zoo, 2020. [find](https://scholar.google.com/scholar?q=Antonin+Raffin%2E+rl%2Dbaselines3%2Dzoo%2C+2020%2E)
49. Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1–8, 2021. URL [http://jmlr.org/papers/v22/20-1364.html](http://jmlr.org/papers/v22/20-1364.html). [↗](http://jmlr.org/papers/v22/20-1364.html)
50. Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389, 2009. [find](https://scholar.google.com/scholar?q=Stephen+Robertson%2C+Hugo+Zaragoza%2C+et+al%2E+The+probabilistic+relevance+framework%3A+Bm25+and+beyond%2E+Foundations+and+Trends%C2%AE+in+Information+Retrieval%2C+3%284%29%3A333%E2%80%93389%2C+2009%2E)
51. Rafael Rodriguez-Sanchez, Benjamin A. Spiegel, Jennifer Wang, Roma Patel, Stefanie Tellex, and George Konidaris. Rlang: A declarative language for describing partial world knowledge to reinforcement learning agents, 2023. [find](https://scholar.google.com/scholar?q=Rafael+Rodriguez%2DSanchez%2C+Benjamin+A%2E+Spiegel%2C+Jennifer+Wang%2C+Roma+Patel%2C+Stefanie+Tellex%2C+and+George+Konidaris%2E+Rlang%3A+A+declarative+language+for+describing+partial+world+knowledge+to+reinforcement+l)
52. Stuart J Russell and Peter Norvig. Artificial intelligence: a modern approach. Pearson, 2016. [find](https://scholar.google.com/scholar?q=Stuart+J+Russell+and+Peter+Norvig%2E+Artificial+intelligence%3A+a+modern+approach%2E+Pearson%2C+2016%2E)
53. Feras Saad. Scalable Structure Learning, Inference, and Analysis with Probabilistic Programs. PhD thesis, Massachusetts Institute of Technology, 2022. [find](https://scholar.google.com/scholar?q=Feras+Saad%2E+Scalable+Structure+Learning%2C+Inference%2C+and+Analysis+with+Probabilistic+Programs%2E+PhD+thesis%2C+Massachusetts+Institute+of+Technology%2C+2022%2E)
54. Max-Philipp B. Schrader. gym-sokoban, 2018. gym-sokoban. [find](https://scholar.google.com/scholar?q=Max%2DPhilipp+B%2E+Schrader%2E+gym%2Dsokoban%2C+2018%2E+gym%2Dsokoban%2E)
55. Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604–609, 2020. [find](https://scholar.google.com/scholar?q=Julian+Schrittwieser%2C+Ioannis+Antonoglou%2C+Thomas+Hubert%2C+Karen+Simonyan%2C+Laurent+Sifre%2C+Simon+Schmitt%2C+Arthur+Guez%2C+Edward+Lockhart%2C+Demis+Hassabis%2C+Thore+Graepel%2C+et+al%2E+Mastering+atari%2C+go%2C+chess+an)
56. Atharva Sehgal, Arya Grayeli, Jennifer J. Sun, and Swarat Chaudhuri. Neurosymbolic grounding for compositional world models, 2023. [find](https://scholar.google.com/scholar?q=Atharva+Sehgal%2C+Arya+Grayeli%2C+Jennifer+J%2E+Sun%2C+and+Swarat+Chaudhuri%2E+Neurosymbolic+grounding+for+compositional+world+models%2C+2023%2E)
57. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://md.research.elamur.ai/free/en/papers/shridhar-2020-alfworld-aligning-text-embodied-environments/). In Proceedings of the International Conference on Learning Representations (ICLR), 2021. URL https://arxiv.org/abs/2010.… [↗](https://arxiv.org/abs/2010.…)
58. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484–489, 2016. [find](https://scholar.google.com/scholar?q=David+Silver%2C+Aja+Huang%2C+Chris+J+Maddison%2C+Arthur+Guez%2C+Laurent+Sifre%2C+George+Van+Den+Driessche%2C+Julian+Schrittwieser%2C+Ioannis+Antonoglou%2C+Veda+Panneershelvam%2C+Marc+Lanctot%2C+et+al%2E+Mastering+the+game+)
59. Tom Silver and Rohan Chitnis. Pddlgym: Gym environments from pddl problems. In International Conference on Automated Planning and Scheduling (ICAPS) PRL Workshop, 2020. URL [https://github.com/tomsilver/pddlgym](https://github.com/tomsilver/pddlgym). [↗](https://github.com/tomsilver/pddlgym)
60. Sean R Sinclair, Siddhartha Banerjee, and Christina Lee Yu. Adaptive discretization for episodic reinforcement learning in metric spaces. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 3(3):1–44, 2019. [https://github.com/DLR-RM/zoo](https://github.com/DLR-RM/zoo). https://github.com/mpSchrader/ [↗](https://github.com/DLR-RM/zoo)
61. Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. [Adaplanner: Adaptive planning from feedback with language models](https://md.research.elamur.ai/free/en/papers/sun-2023-adaplanner-adaptive-planning-feedback-language/). NeurIPS, 2023.
62. Richard S Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Machine learning proceedings 1990, pages 216–224. Elsevier, 1990. [find](https://scholar.google.com/scholar?q=Richard+S+Sutton%2E+Integrated+architectures+for+learning%2C+planning%2C+and+reacting+based+on+approximating+dynamic+programming%2E+In+Machine+learning+proceedings+1990%2C+pages+216%E2%80%93224%2E+Elsevier%2C+1990%2E)
63. Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018. [find](https://scholar.google.com/scholar?q=Richard+S+Sutton+and+Andrew+G+Barto%2E+Reinforcement+learning%3A+An+introduction%2E+MIT+press%2C+2018%2E)
64. Hao Tang and Kevin Ellis. From perception to programs: Regularize, overparameterize, and amortize. International Conference on Machine Learning (ICML), 2023. [find](https://scholar.google.com/scholar?q=Hao+Tang+and+Kevin+Ellis%2E+From+perception+to+programs%3A+Regularize%2C+overparameterize%2C+and+amortize%2E+International+Conference+on+Machine+Learning+%28ICML%29%2C+2023%2E)
65. Hao Tang, Keya Hu, Jin Peng Zhou, Sicheng Zhong, Wei-Long Zheng, Xujie Si, and Kevin Ellis. Code repair with llms gives an exploration-exploitation tradeoff. arXiv, 2024. [find](https://scholar.google.com/scholar?q=Hao+Tang%2C+Keya+Hu%2C+Jin+Peng+Zhou%2C+Sicheng+Zhong%2C+Wei%2DLong+Zheng%2C+Xujie+Si%2C+and+Kevin+Ellis%2E+Code+repair+with+llms+gives+an+exploration%2Dexploitation+tradeoff%2E+arXiv%2C+2024%2E)
66. Edward Chace Tolman and Charles H Honzik. Introduction and removal of reward, and maze performance in rats. University of California publications in psychology, 1930. [find](https://scholar.google.com/scholar?q=Edward+Chace+Tolman+and+Charles+H+Honzik%2E+Introduction+and+removal+of+reward%2C+and+maze+performance+in+rats%2E+University+of+California+publications+in+psychology%2C+1930%2E)
67. Pedro A Tsividis, Joao Loula, Jake Burga, Nathan Foss, Andres Campero, Thomas Pouncy, Samuel J Gershman, and Joshua B Tenenbaum. Human-level reinforcement learning through theory-based modeling, exploration, and planning. arXiv preprint arXiv:2107.12544, 2021. — [arXiv:2107.12544](https://arxiv.org/abs/2107.12544) [find](https://scholar.google.com/scholar?q=Pedro+A+Tsividis%2C+Joao+Loula%2C+Jake+Burga%2C+Nathan+Foss%2C+Andres+Campero%2C+Thomas+Pouncy%2C+Samuel+J+Gershman%2C+and+Joshua+B+Tenenbaum%2E+Human%2Dlevel+reinforcement+learning+through+theory%2Dbased+modeling%2C+explo)
68. Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models - a critical investigation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL — [On the Planning Abilities of Large Language Models : A Critical Investigation](https://md.research.elamur.ai/free/en/papers/valmeekam-2023-planning-abilities-large-language-models/) [https://openreview.net/forum?id=X6dEqXIsEW](https://openreview.net/forum?id=X6dEqXIsEW). [↗](https://openreview.net/forum?id=X6dEqXIsEW)
69. Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. Programmatically interpretable reinforcement learning. In International Conference on Machine Learning, pages 5045–5054. PMLR, 2018. [find](https://scholar.google.com/scholar?q=Abhinav+Verma%2C+Vijayaraghavan+Murali%2C+Rishabh+Singh%2C+Pushmeet+Kohli%2C+and+Swarat+Chaudhuri%2E+Programmatically+interpretable+reinforcement+learning%2E+In+International+Conference+on+Machine+Learning%2C+pages)
70. Abhinav Verma, Hoang Le, Yisong Yue, and Swarat Chaudhuri. Imitation-projected programmatic reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019. [find](https://scholar.google.com/scholar?q=Abhinav+Verma%2C+Hoang+Le%2C+Yisong+Yue%2C+and+Swarat+Chaudhuri%2E+Imitation%2Dprojected+programmatic+reinforcement+learning%2E+Advances+in+Neural+Information+Processing+Systems%2C+32%2C+2019%2E)
71. Lev Semenovich Vygotsky. Mind in society: The development of higher psychological processes. Harvard university press, 1980. [find](https://scholar.google.com/scholar?q=Lev+Semenovich+Vygotsky%2E+Mind+in+society%3A+The+development+of+higher+psychological+processes%2E+Harvard+university+press%2C+1980%2E)
72. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv: Arxiv-2305.16291, 2023. [find](https://scholar.google.com/scholar?q=Guanzhi+Wang%2C+Yuqi+Xie%2C+Yunfan+Jiang%2C+Ajay+Mandlekar%2C+Chaowei+Xiao%2C+Yuke+Zhu%2C+Linxi+Fan%2C+and+Anima+Anandkumar%2E+Voyager%3A+An+open%2Dended+embodied+agent+with+large+language+models%2E+arXiv+preprint+arXiv%3A+A)
73. Huaxiaoyue Wang, Gonzalo Gonzalez-Pumariega, Yash Sharma, and Sanjiban Choudhury. Demo2code: From summarizing demonstrations to synthesizing code via extended chain-of-thought. NeurIPS, 2023. [find](https://scholar.google.com/scholar?q=Huaxiaoyue+Wang%2C+Gonzalo+Gonzalez%2DPumariega%2C+Yash+Sharma%2C+and+Sanjiban+Choudhury%2E+Demo2code%3A+From+summarizing+demonstrations+to+synthesizing+code+via+extended+chain%2Dof%2Dthought%2E+NeurIPS%2C+2023%2E)
74. Lionel Wong, Jiayuan Mao, Pratyusha Sharma, Zachary S. Siegel, Jiahai Feng, Noa Korneev, Joshua B. Tenenbaum, and Jacob Andreas. Learning adaptive planning representations with natural language guidance, 2023. [find](https://scholar.google.com/scholar?q=Lionel+Wong%2C+Jiayuan+Mao%2C+Pratyusha+Sharma%2C+Zachary+S%2E+Siegel%2C+Jiahai+Feng%2C+Noa+Korneev%2C+Joshua+B%2E+Tenenbaum%2C+and+Jacob+Andreas%2E+Learning+adaptive+planning+representations+with+natural+language+guidan)
75. Chenyang Wu, Tianci Li, Zongzhang Zhang, and Yang Yu. Bayesian optimistic optimization: Optimistic exploration for model-based reinforcement learning. Advances in neural information processing systems, 35:14210–14223, 2022. [find](https://scholar.google.com/scholar?q=Chenyang+Wu%2C+Tianci+Li%2C+Zongzhang+Zhang%2C+and+Yang+Yu%2E+Bayesian+optimistic+optimization%3A+Optimistic+exploration+for+model%2Dbased+reinforcement+learning%2E+Advances+in+neural+information+processing+systems)
76. Jiannan Xiang, Tianhua Tao, Yi Gu, Tianmin Shu, Zirui Wang, Zichao Yang, and Zhiting Hu. Language models meet world models: Embodied experiences enhance language models. NeurIPS, 2023. [find](https://scholar.google.com/scholar?q=Jiannan+Xiang%2C+Tianhua+Tao%2C+Yi+Gu%2C+Tianmin+Shu%2C+Zirui+Wang%2C+Zichao+Yang%2C+and+Zhiting+Hu%2E+Language+models+meet+world+models%3A+Embodied+experiences+enhance+language+models%2E+NeurIPS%2C+2023%2E)
77. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. [React: Synergizing reasoning and acting in language models](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/). arXiv preprint arXiv:2210.03629, 2022.
78. Zirui Zhao, Wee Sun Lee, and David Hsu. [Large language models as commonsense knowledge for large-scale task planning](https://md.research.elamur.ai/free/en/papers/zhao-2023-large-language-models-commonsense-knowledge/). NeurIPS, 2023.
79. Feiyu Zhu and Reid Simmons. Bootstrapping cognitive agents with a large language model. In AAAI, 2024. [find](https://scholar.google.com/scholar?q=Feiyu+Zhu+and+Reid+Simmons%2E+Bootstrapping+cognitive+agents+with+a+large+language+model%2E+In+AAAI%2C+2024%2E)
80. Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and Ji-Rong Wen. [Large language models for information retrieval: A survey](https://md.research.elamur.ai/free/en/papers/zhu-2023-large-language-models-information-retrieval/). arXiv preprint arXiv:2308.07107, 2023.

*Updated 3 November 2026*