---
title: "Describe, Explain, Plan and Select — Wang 2023"
description: "The paper tackles task planning for embodied agents in open-world settings such as Minecraft."
authors: ["Zihao Wang", "Shaofei Cai", "Guanzhou Chen", "Anji Liu", "Xiaojian Ma", "Yitao Liang"]
year: 2023
venue: "arXiv:2302.01560"
source: "https://arxiv.org/abs/2302.01560"
doi: "10.48550/arxiv.2302.01560"
arxiv: "2302.01560"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2023-describe-explain-plan-select-interactive/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2023-describe-explain-plan-select-interactive/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/2/) / Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents

# Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents

**Authors:** Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, Yitao Liang · **Year:** 2023 · **Venue:** arXiv:2302.01560 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2302.01560) · [PDF](https://arxiv.org/pdf/2302.01560) · [DOI 10.48550/arxiv.2302.01560](https://doi.org/10.48550/arxiv.2302.01560) · [arXiv 2302.01560](https://arxiv.org/abs/2302.01560) · [OpenAlex](https://openalex.org/W4319323461)

**Citations:** 45 citations (openalex-snapshot · 2026-10-14) · [116](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
The paper tackles task planning for embodied agents in open-world settings such as Minecraft. Two problems make LLM planners fail there. Tasks need long chains of dependent sub-goals, so a plan written in one pass is often wrong. Plans also ignore which of several alternative sub-goals the agent can reach most easily from where it is. The authors propose DEPS, an interactive loop built on an LLM. When the controller fails a sub-goal, a descriptor turns the situation and inventory into text. The LLM then reasons about why the plan broke and rewrites it. A trainable goal Selector ranks parallel candidate sub-goals by how many steps each is predicted to take. The approach needs no planning-specific training, only prompt demonstrations for a fixed LLM. On 71 Minecraft tasks, it roughly doubles the overall success of language-planner baselines. It is the first planning-based agent to obtain a diamond. It also helps on ALFWorld and Tabletop manipulation, suggesting LLM planners can scale to complex, state-dependent worlds.

## Key points
- Evaluated zero-shot (no gradient updates) on 71 Minecraft tasks, with episode horizons of 3000–12000 steps depending on the task. A diamond requires at least 13 correctly ordered sub-goals.
- Components: an event-triggered Descriptor, an LLM serving as both Explainer and Planner, a horizon-prediction-based goal Selector, and a goal-conditioned controller. Inner Monologue is the main interactive baseline.
- On ObtainDiamond, DEPS reaches a 0.59% success rate within 10 minutes without task-specific fine-tuning. For context, VPT reaches 20% in 20 minutes using about 70k hours of human demonstrations, and DreamerV3 reaches 2% in an easier modified environment.
- Compared with prior and concurrent LLM planners on ALFWorld and Tabletop environments, DEPS performs anywhere from comparably to over 50% better in relative terms.
- Limitations include reliance on proprietary LLMs (GPT-3, ChatGPT) and an explicit step-by-step planning bottleneck. Code is available at github.com/CraftJarvis/MC-Planner (NeurIPS 2023).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

116 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 104 not yet.

- [Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution](https://md.research.elamur.ai/free/en/papers/fernando-2023-promptbreeder-self-referential-self-improvement/)
- [Challenges and Applications of Large Language Models](https://md.research.elamur.ai/free/en/papers/kaddour-2023-challenges-applications-large-language-models/)
- [Language Models can Solve Computer Tasks](https://md.research.elamur.ai/free/en/papers/kim-2023-language-models-solve-computer-tasks/)
- [AgentBench: Evaluating LLMs as Agents](https://md.research.elamur.ai/free/en/papers/liu-2023-agentbench-evaluating-llms-agents/)
- [LLM+P: Empowering Large Language Models with Optimal Planning Proficiency](https://md.research.elamur.ai/free/en/papers/liu-2023-llm-p-empowering-large-language/)
- [Augmented Language Models: a Survey](https://md.research.elamur.ai/free/en/papers/mialon-2023-augmented-language-models-survey/)
- [Tool Learning with Foundation Models](https://md.research.elamur.ai/free/en/papers/qin-2023-tool-learning-foundation-models/)
- [A Survey on Large Language Model based Autonomous Agents](https://md.research.elamur.ai/free/en/papers/wang-2023-survey-large-language-model-based/)
- [Executable Code Actions Elicit Better LLM Agents](https://md.research.elamur.ai/free/en/papers/wang-2024-executable-code-actions-elicit-better/)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://md.research.elamur.ai/free/en/papers/xi-2023-rise-potential-large-language/)
- [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)
- [ToolQA: A Dataset for LLM Question Answering with External Tools](https://md.research.elamur.ai/free/en/papers/zhuang-2023-toolqa-dataset-llm-question-answering/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{wang2023describe,
  title = {{Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents}},
  author = {Wang, Zihao and Cai, Shaofei and Chen, Guanzhou and Liu, Anji and Ma, Xiaojian and Liang, Yitao},
  year = {2023},
  doi = {10.48550/arxiv.2302.01560},
  eprint = {2302.01560},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2302.01560}
}
```

- [← previous Dependency Dialogue Acts — Annotation Scheme and Case Study](https://md.research.elamur.ai/free/en/papers/cai-2023-dependency-dialogue-acts-annotation-scheme/)
- [next → Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design](https://md.research.elamur.ai/free/en/papers/pyatkin-2023-design-choices-crowdsourcing-implicit-discourse/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/2/)

## References

48 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198, 2022. 1 [find](https://scholar.google.com/scholar?q=J%2E%2DB%2E+Alayrac%2C+J%2E+Donahue%2C+P%2E+Luc%2C+A%2E+Miech%2C+I%2E+Barr%2C+Y%2E+Hasson%2C+K%2E+Lenc%2C+A%2E+Mensch%2C+K%2E+Millican%2C+M%2E+Reynolds%2C+et+al%2E+Flamingo%3A+a+visual+language+model+for+few%2Dshot+learning%2E+arXiv+preprint+arXiv%3A2204) [our card](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/)
2. P.-L. Bacon, J. Harb, and D. Precup. The option-critic architecture. In Proceedings of the AAAI conference on artificial intelligence, 2017. 1 [find](https://scholar.google.com/scholar?q=P%2E%2DL%2E+Bacon%2C+J%2E+Harb%2C+and+D%2E+Precup%2E+The+option%2Dcritic+architecture%2E+In+Proceedings+of+the+AAAI+conference+on+artificial+intelligence%2C+2017%2E+1)
3. B. Baker, I. Akkaya, P. Zhokhov, J. Huizinga, J. Tang, A. Ecoffet, B. Houghton, R. Sampedro, and J. Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. arXiv preprint arXiv:2206.11795, 2022. 7, 8, 9, 10 — [arXiv:2206.11795](https://arxiv.org/abs/2206.11795) [find](https://scholar.google.com/scholar?q=B%2E+Baker%2C+I%2E+Akkaya%2C+P%2E+Zhokhov%2C+J%2E+Huizinga%2C+J%2E+Tang%2C+A%2E+Ecoffet%2C+B%2E+Houghton%2C+R%2E+Sampedro%2C+and+J%2E+Clune%2E+Video+pretraining+%28vpt%29%3A+Learning+to+act+by+watching+unlabeled+online+videos%2E+arXiv+preprint+)
4. A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, et al. Do as i can, not as i say: Grounding language in robotic affordances. In 6th Annual Conference on Robot Learning, 2022. 1, 2, 3, 5, 10, 21 [find](https://scholar.google.com/scholar?q=A%2E+Brohan%2C+Y%2E+Chebotar%2C+C%2E+Finn%2C+K%2E+Hausman%2C+A%2E+Herzog%2C+D%2E+Ho%2C+J%2E+Ibarz%2C+A%2E+Irpan%2C+E%2E+Jang%2C+R%2E+Julian%2C+et+al%2E+Do+as+i+can%2C+not+as+i+say%3A+Grounding+language+in+robotic+affordances%2E+In+6th+Annual+Confer) [our card](https://md.research.elamur.ai/free/en/papers/ahn-2022-i-i-say-grounding-language/)
5. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. 1, 3, 15, 21, 22 [find](https://scholar.google.com/scholar?q=T%2E+Brown%2C+B%2E+Mann%2C+N%2E+Ryder%2C+M%2E+Subbiah%2C+J%2E+D%2E+Kaplan%2C+P%2E+Dhariwal%2C+A%2E+Neelakantan%2C+P%2E+Shyam%2C+G%2E+Sastry%2C+A%2E+Askell%2C+et+al%2E+Language+models+are+few%2Dshot+learners%2E+Advances+in+neural+information+process)
6. S. Cai, Z. Wang, X. Ma, A. Liu, and Y. Liang. Open-world multi-task control through goal-aware representation learning and adaptive horizon prediction. arXiv preprint arXiv:2301.10034, 2023. 6, 8, 10, 21, 24 [find](https://scholar.google.com/scholar?q=S%2E+Cai%2C+Z%2E+Wang%2C+X%2E+Ma%2C+A%2E+Liu%2C+and+Y%2E+Liang%2E+Open%2Dworld+multi%2Dtask+control+through+goal%2Daware+representation+learning+and+adaptive+horizon+prediction%2E+arXiv+preprint+arXiv%3A2301%2E10034%2C+2023%2E+6%2C+8%2C+10%2C)
7. S. Cai, B. Zhang, Z. Wang, X. Ma, A. Liu, and Y. Liang. Groot: Learning to follow instructions by watching gameplay videos. arXiv preprint arXiv:2310.08235, 2023. 10 — [arXiv:2310.08235](https://arxiv.org/abs/2310.08235) [find](https://scholar.google.com/scholar?q=S%2E+Cai%2C+B%2E+Zhang%2C+Z%2E+Wang%2C+X%2E+Ma%2C+A%2E+Liu%2C+and+Y%2E+Liang%2E+Groot%3A+Learning+to+follow+instructions+by+watching+gameplay+videos%2E+arXiv+preprint+arXiv%3A2310%2E08235%2C+2023%2E+10+%E2%80%94+%5BarXiv%3A2310%2E08235%5D%28https%3A%2F%2Farxiv)
8. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. 3, 7, 21, 22 [find](https://scholar.google.com/scholar?q=M%2E+Chen%2C+J%2E+Tworek%2C+H%2E+Jun%2C+Q%2E+Yuan%2C+H%2E+P%2E+d%2E+O%2E+Pinto%2C+J%2E+Kaplan%2C+H%2E+Edwards%2C+Y%2E+Burda%2C+N%2E+Joseph%2C+G%2E+Brockman%2C+et+al%2E+Evaluating+large+language+models+trained+on+code%2E+arXiv+preprint+arXiv%3A2107%2E0337) [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
9. I. Dasgupta, C. Kaeser-Chen, K. Marino, A. Ahuja, S. Babayan, F. Hill, and R. Fergus. Collaborating with language models for embodied reasoning. In NeurIPS Foundation Models for Decision Making Workshop, 2022. 9 [find](https://scholar.google.com/scholar?q=I%2E+Dasgupta%2C+C%2E+Kaeser%2DChen%2C+K%2E+Marino%2C+A%2E+Ahuja%2C+S%2E+Babayan%2C+F%2E+Hill%2C+and+R%2E+Fergus%2E+Collaborating+with+language+models+for+embodied+reasoning%2E+In+NeurIPS+Foundation+Models+for+Decision+Making+Worksh)
10. L. Fan, G. Wang, Y. Jiang, A. Mandlekar, Y. Yang, H. Zhu, A. Tang, D.-A. Huang, Y. Zhu, and A. Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems Datasets and Benchmarks, 2022. 1, 2, 7, 8, 10, 18, 21, 26 [find](https://scholar.google.com/scholar?q=L%2E+Fan%2C+G%2E+Wang%2C+Y%2E+Jiang%2C+A%2E+Mandlekar%2C+Y%2E+Yang%2C+H%2E+Zhu%2C+A%2E+Tang%2C+D%2E%2DA%2E+Huang%2C+Y%2E+Zhu%2C+and+A%2E+Anandkumar%2E+Minedojo%3A+Building+open%2Dended+embodied+agents+with+internet%2Dscale+knowledge%2E+Advances+in+Neur)
11. R. Gong, Q. Huang, X. Ma, H. Vo, Z. Durante, Y. Noda, Z. Zheng, S.-C. Zhu, D. Terzopoulos, L. Fei-Fei, et al. Mindagent: Emergent gaming interaction. arXiv preprint arXiv:2309.09971, 2023. 9 [find](https://scholar.google.com/scholar?q=R%2E+Gong%2C+Q%2E+Huang%2C+X%2E+Ma%2C+H%2E+Vo%2C+Z%2E+Durante%2C+Y%2E+Noda%2C+Z%2E+Zheng%2C+S%2E%2DC%2E+Zhu%2C+D%2E+Terzopoulos%2C+L%2E+Fei%2DFei%2C+et+al%2E+Mindagent%3A+Emergent+gaming+interaction%2E+arXiv+preprint+arXiv%3A2309%2E09971%2C+2023%2E+9)
12. W. H. Guss, M. Y. Castro, S. Devlin, B. Houghton, N. S. Kuno, C. Loomis, S. Milani, S. P. Mohanty, K. Nakata, R. Salakhutdinov, J. Schulman, S. Shiroshita, N. Topin, A. Ummadisingu, and O. Vinyals. The minerl 2020 competition on sample efficient reinforcement learning using human priors. arXiv: Lear… [find](https://scholar.google.com/scholar?q=W%2E+H%2E+Guss%2C+M%2E+Y%2E+Castro%2C+S%2E+Devlin%2C+B%2E+Houghton%2C+N%2E+S%2E+Kuno%2C+C%2E+Loomis%2C+S%2E+Milani%2C+S%2E+P%2E+Mohanty%2C+K%2E+Nakata%2C+R%2E+Salakhutdinov%2C+J%2E+Schulman%2C+S%2E+Shiroshita%2C+N%2E+Topin%2C+A%2E+Ummadisingu%2C+and+O%2E+Vinyals%2E+Th)
13. W. H. Guss, C. Codel, K. Hofmann, B. Houghton, N. Kuno, S. Milani, S. Mohanty, D. P. Liebana, R. Salakhutdinov, N. Topin, et al. Neurips 2019 competition: the minerl competition on sample efficient reinforcement learning using human priors. arXiv preprint arXiv:1904.10079, 2019. 2 — [arXiv:1904.10079](https://arxiv.org/abs/1904.10079) [find](https://scholar.google.com/scholar?q=W%2E+H%2E+Guss%2C+C%2E+Codel%2C+K%2E+Hofmann%2C+B%2E+Houghton%2C+N%2E+Kuno%2C+S%2E+Milani%2C+S%2E+Mohanty%2C+D%2E+P%2E+Liebana%2C+R%2E+Salakhutdinov%2C+N%2E+Topin%2C+et+al%2E+Neurips+2019+competition%3A+the+minerl+competition+on+sample+efficient+re)
14. W. H. Guss, B. Houghton, N. Topin, P. Wang, C. Codel, M. Veloso, and R. Salakhutdinov. Minerl: A large-scale dataset of minecraft demonstrations. arXiv preprint arXiv:1907.13440, 2019. 2, 7, 9, 24 — [arXiv:1907.13440](https://arxiv.org/abs/1907.13440) [find](https://scholar.google.com/scholar?q=W%2E+H%2E+Guss%2C+B%2E+Houghton%2C+N%2E+Topin%2C+P%2E+Wang%2C+C%2E+Codel%2C+M%2E+Veloso%2C+and+R%2E+Salakhutdinov%2E+Minerl%3A+A+large%2Dscale+dataset+of+minecraft+demonstrations%2E+arXiv+preprint+arXiv%3A1907%2E13440%2C+2019%2E+2%2C+7%2C+9%2C+24+%E2%80%94+%5B)
15. D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023. 9, 10 [find](https://scholar.google.com/scholar?q=D%2E+Hafner%2C+J%2E+Pasukonis%2C+J%2E+Ba%2C+and+T%2E+Lillicrap%2E+Mastering+diverse+domains+through+world+models%2E+arXiv+preprint+arXiv%3A2301%2E04104%2C+2023%2E+9%2C+10)
16. W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. ICML, 2022. 3, 4, 7, 8, 9, 15, 16, 17, 21, 24, 25, 26, 28 [find](https://scholar.google.com/scholar?q=W%2E+Huang%2C+P%2E+Abbeel%2C+D%2E+Pathak%2C+and+I%2E+Mordatch%2E+Language+models+as+zero%2Dshot+planners%3A+Extracting+actionable+knowledge+for+embodied+agents%2E+ICML%2C+2022%2E+3%2C+4%2C+7%2C+8%2C+9%2C+15%2C+16%2C+17%2C+21%2C+24%2C+25%2C+26%2C+28) [our card](https://md.research.elamur.ai/free/en/papers/huang-2022-language-models-zero-shot-planners/)
17. W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022. 2, 3, 5, 7, 8, 10, 15, 16, 17, 24, 25, 26, 30, 34, 35 [find](https://scholar.google.com/scholar?q=W%2E+Huang%2C+F%2E+Xia%2C+T%2E+Xiao%2C+H%2E+Chan%2C+J%2E+Liang%2C+P%2E+Florence%2C+A%2E+Zeng%2C+J%2E+Tompson%2C+I%2E+Mordatch%2C+Y%2E+Chebotar%2C+et+al%2E+Inner+monologue%3A+Embodied+reasoning+through+planning+with+language+models%2E+arXiv+prepri)
18. M. Johnson, K. Hofmann, T. Hutton, and D. Bignell. The malmo platform for artificial intelligence experimentation. In Ijcai, pages 4246–4247. Citeseer, 2016. 2, 7 [find](https://scholar.google.com/scholar?q=M%2E+Johnson%2C+K%2E+Hofmann%2C+T%2E+Hutton%2C+and+D%2E+Bignell%2E+The+malmo+platform+for+artificial+intelligence+experimentation%2E+In+Ijcai%2C+pages+4246%E2%80%934247%2E+Citeseer%2C+2016%2E+2%2C+7)
19. A. Kanervisto, S. Milani, K. Ramanauskas, N. Topin, Z. Lin, J. Li, J. Shi, D. Ye, Q. Fu, W. Yang, W. Hong, Z. Huang, H. Chen, G. Zeng, Y. Lin, V. Micheli, E. Alonso, F. Fleuret, A. Nikulin, Y. Belousov, O. Svidchenko, and A. Shpilman. Minerl diamond 2021 competition: Overview, results, and lessons l… [find](https://scholar.google.com/scholar?q=A%2E+Kanervisto%2C+S%2E+Milani%2C+K%2E+Ramanauskas%2C+N%2E+Topin%2C+Z%2E+Lin%2C+J%2E+Li%2C+J%2E+Shi%2C+D%2E+Ye%2C+Q%2E+Fu%2C+W%2E+Yang%2C+W%2E+Hong%2C+Z%2E+Huang%2C+H%2E+Chen%2C+G%2E+Zeng%2C+Y%2E+Lin%2C+V%2E+Micheli%2C+E%2E+Alonso%2C+F%2E+Fleuret%2C+A%2E+Nikulin%2C+Y%2E+Belouso)
20. J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. arXiv preprint arXiv:2209.07753, 2022. 2, 3, 5, 7, 8, 10, 25, 26, 31 [find](https://scholar.google.com/scholar?q=J%2E+Liang%2C+W%2E+Huang%2C+F%2E+Xia%2C+P%2E+Xu%2C+K%2E+Hausman%2C+B%2E+Ichter%2C+P%2E+Florence%2C+and+A%2E+Zeng%2E+Code+as+policies%3A+Language+model+programs+for+embodied+control%2E+arXiv+preprint+arXiv%3A2209%2E07753%2C+2022%2E+2%2C+3%2C+5%2C+7%2C+8)
21. S. Lifshitz, K. Paster, H. Chan, J. Ba, and S. McIlraith. Steve-1: A generative model for text-to-behavior in minecraft. arXiv preprint arXiv:2306.00937, 2023. 8, 10 — [arXiv:2306.00937](https://arxiv.org/abs/2306.00937) [find](https://scholar.google.com/scholar?q=S%2E+Lifshitz%2C+K%2E+Paster%2C+H%2E+Chan%2C+J%2E+Ba%2C+and+S%2E+McIlraith%2E+Steve%2D1%3A+A+generative+model+for+text%2Dto%2Dbehavior+in+minecraft%2E+arXiv+preprint+arXiv%3A2306%2E00937%2C+2023%2E+8%2C+10+%E2%80%94+%5BarXiv%3A2306%2E00937%5D%28https%3A%2F%2Farxiv)
22. H. Lin, Z. Wang, J. Ma, and Y. Liang. Mcu: A task-centric framework for open-ended agent evaluation in minecraft. arXiv preprint arXiv:2310.08367, 2023. 6, 7, 8, 18 — [arXiv:2310.08367](https://arxiv.org/abs/2310.08367) [find](https://scholar.google.com/scholar?q=H%2E+Lin%2C+Z%2E+Wang%2C+J%2E+Ma%2C+and+Y%2E+Liang%2E+Mcu%3A+A+task%2Dcentric+framework+for+open%2Dended+agent+evaluation+in+minecraft%2E+arXiv+preprint+arXiv%3A2310%2E08367%2C+2023%2E+6%2C+7%2C+8%2C+18+%E2%80%94+%5BarXiv%3A2310%2E08367%5D%28https%3A%2F%2Farxiv%2E)
23. K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg. Text2motion: From natural language instructions to feasible plans. arXiv preprint arXiv:2303.12153, 2023. 10 — [arXiv:2303.12153](https://arxiv.org/abs/2303.12153) [find](https://scholar.google.com/scholar?q=K%2E+Lin%2C+C%2E+Agia%2C+T%2E+Migimatsu%2C+M%2E+Pavone%2C+and+J%2E+Bohg%2E+Text2motion%3A+From+natural+language+instructions+to+feasible+plans%2E+arXiv+preprint+arXiv%3A2303%2E12153%2C+2023%2E+10+%E2%80%94+%5BarXiv%3A2303%2E12153%5D%28https%3A%2F%2Farxiv%2Eo)
24. Z. Lin, J. Li, J. Shi, D. Ye, Q. Fu, and W. Yang. Juewu-mc: Playing minecraft with sample-efficient hierarchical reinforcement learning. arXiv preprint arXiv:2112.04907, 2021. 10 — [arXiv:2112.04907](https://arxiv.org/abs/2112.04907) [find](https://scholar.google.com/scholar?q=Z%2E+Lin%2C+J%2E+Li%2C+J%2E+Shi%2C+D%2E+Ye%2C+Q%2E+Fu%2C+and+W%2E+Yang%2E+Juewu%2Dmc%3A+Playing+minecraft+with+sample%2Defficient+hierarchical+reinforcement+learning%2E+arXiv+preprint+arXiv%3A2112%2E04907%2C+2021%2E+10+%E2%80%94+%5BarXiv%3A2112%2E04907%5D%28)
25. X. Ma, S. Yong, Z. Zheng, Q. Li, Y. Liang, S.-C. Zhu, and S. Huang. Sqa3d: Situated question answering in 3d scenes. arXiv preprint arXiv:2210.07474, 2022. 1 — [arXiv:2210.07474](https://arxiv.org/abs/2210.07474) [find](https://scholar.google.com/scholar?q=X%2E+Ma%2C+S%2E+Yong%2C+Z%2E+Zheng%2C+Q%2E+Li%2C+Y%2E+Liang%2C+S%2E%2DC%2E+Zhu%2C+and+S%2E+Huang%2E+Sqa3d%3A+Situated+question+answering+in+3d+scenes%2E+arXiv+preprint+arXiv%3A2210%2E07474%2C+2022%2E+1+%E2%80%94+%5BarXiv%3A2210%2E07474%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs)
26. J. Mai, J. Chen, B. Li, G. Qian, M. Elhoseiny, and B. Ghanem. Llm as a robotic brain: Unifying egocentric memory and control. arXiv preprint arXiv:2304.09349, 2023. 10 — [arXiv:2304.09349](https://arxiv.org/abs/2304.09349) [find](https://scholar.google.com/scholar?q=J%2E+Mai%2C+J%2E+Chen%2C+B%2E+Li%2C+G%2E+Qian%2C+M%2E+Elhoseiny%2C+and+B%2E+Ghanem%2E+Llm+as+a+robotic+brain%3A+Unifying+egocentric+memory+and+control%2E+arXiv+preprint+arXiv%3A2304%2E09349%2C+2023%2E+10+%E2%80%94+%5BarXiv%3A2304%2E09349%5D%28https%3A%2F%2Farx)
27. H. Mao, C. Wang, X. Hao, Y. Mao, Y. Lu, C. Wu, J. Hao, D. Li, and P. Tang. Seihai: A sample-efficient hierarchical ai for the minerl competition. In Distributed Artificial Intelligence: Third International Conference, DAI 2021, Shanghai, China, December 17–18, 2021, Proceedings 3, pages 38–51. Sprin… [find](https://scholar.google.com/scholar?q=H%2E+Mao%2C+C%2E+Wang%2C+X%2E+Hao%2C+Y%2E+Mao%2C+Y%2E+Lu%2C+C%2E+Wu%2C+J%2E+Hao%2C+D%2E+Li%2C+and+P%2E+Tang%2E+Seihai%3A+A+sample%2Defficient+hierarchical+ai+for+the+minerl+competition%2E+In+Distributed+Artificial+Intelligence%3A+Third+Internat)
28. S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837, 2022. 23 [find](https://scholar.google.com/scholar?q=S%2E+Min%2C+X%2E+Lyu%2C+A%2E+Holtzman%2C+M%2E+Artetxe%2C+M%2E+Lewis%2C+H%2E+Hajishirzi%2C+and+L%2E+Zettlemoyer%2E+Rethinking+the+role+of+demonstrations%3A+What+makes+in%2Dcontext+learning+work%3F+arXiv+preprint+arXiv%3A2202%2E12837%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/min-2022-rethinking-role-demonstrations-what-makes/)
29. V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013. 2 [find](https://scholar.google.com/scholar?q=V%2E+Mnih%2C+K%2E+Kavukcuoglu%2C+D%2E+Silver%2C+A%2E+Graves%2C+I%2E+Antonoglou%2C+D%2E+Wierstra%2C+and+M%2E+Riedmiller%2E+Playing+atari+with+deep+reinforcement+learning%2E+arXiv+preprint+arXiv%3A1312%2E5602%2C+2013%2E+2)
30. J. Oh, S. Singh, H. Lee, and P. Kohli. Zero-shot task generalization with multi-task deep reinforcement learning. In International Conference on Machine Learning, pages 2661–2670. PMLR, 2017. 10 [find](https://scholar.google.com/scholar?q=J%2E+Oh%2C+S%2E+Singh%2C+H%2E+Lee%2C+and+P%2E+Kohli%2E+Zero%2Dshot+task+generalization+with+multi%2Dtask+deep+reinforcement+learning%2E+In+International+Conference+on+Machine+Learning%2C+pages+2661%E2%80%932670%2E+PMLR%2C+2017%2E+10)
31. OpenAI. Gpt-4 technical report, 2023. 21, 22 [find](https://scholar.google.com/scholar?q=OpenAI%2E+Gpt%2D4+technical+report%2C+2023%2E+21%2C+22) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
32. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022. 3, 5, 7, 8 [find](https://scholar.google.com/scholar?q=L%2E+Ouyang%2C+J%2E+Wu%2C+X%2E+Jiang%2C+D%2E+Almeida%2C+C%2E+L%2E+Wainwright%2C+P%2E+Mishkin%2C+C%2E+Zhang%2C+S%2E+Agarwal%2C+K%2E+Slama%2C+A%2E+Ray%2C+et+al%2E+Training+language+models+to+follow+instructions+with+human+feedback%2E+arXiv+preprint)
33. J. S. Park, J. C. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442, 2023. 10 [find](https://scholar.google.com/scholar?q=J%2E+S%2E+Park%2C+J%2E+C%2E+O%E2%80%99Brien%2C+C%2E+J%2E+Cai%2C+M%2E+R%2E+Morris%2C+P%2E+Liang%2C+and+M%2E+S%2E+Bernstein%2E+Generative+agents%3A+Interactive+simulacra+of+human+behavior%2E+arXiv+preprint+arXiv%3A2304%2E03442%2C+2023%2E+10)
34. V. P. Patil, M. Hofmarcher, M.-C. Dinu, M. Dorfer, P. M. Blies, J. Brandstetter, J. A. Arjona-Medina, and S. Hochreiter. Align-rudder: Learning from few demonstrations by reward redistribution. ICML, 2020. 9 [find](https://scholar.google.com/scholar?q=V%2E+P%2E+Patil%2C+M%2E+Hofmarcher%2C+M%2E%2DC%2E+Dinu%2C+M%2E+Dorfer%2C+P%2E+M%2E+Blies%2C+J%2E+Brandstetter%2C+J%2E+A%2E+Arjona%2DMedina%2C+and+S%2E+Hochreiter%2E+Align%2Drudder%3A+Learning+from+few+demonstrations+by+reward+redistribution%2E+ICML%2C+)
35. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021. 5, 6, 8 [find](https://scholar.google.com/scholar?q=A%2E+Radford%2C+J%2E+W%2E+Kim%2C+C%2E+Hallacy%2C+A%2E+Ramesh%2C+G%2E+Goh%2C+S%2E+Agarwal%2C+G%2E+Sastry%2C+A%2E+Askell%2C+P%2E+Mishkin%2C+J%2E+Clark%2C+et+al%2E+Learning+transferable+visual+models+from+natural+language+supervision%2E+In+Internati) [our card](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/)
36. S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06175, 2022. 1 [find](https://scholar.google.com/scholar?q=S%2E+Reed%2C+K%2E+Zolna%2C+E%2E+Parisotto%2C+S%2E+G%2E+Colmenarejo%2C+A%2E+Novikov%2C+G%2E+Barth%2DMaron%2C+M%2E+Gimenez%2C+Y%2E+Sulsky%2C+J%2E+Kay%2C+J%2E+T%2E+Springenberg%2C+et+al%2E+A+generalist+agent%2E+arXiv+preprint+arXiv%3A2205%2E06175%2C+2022%2E+1)
37. N. Reimers and I. Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019. 23 [find](https://scholar.google.com/scholar?q=N%2E+Reimers+and+I%2E+Gurevych%2E+Sentence%2Dbert%3A+Sentence+embeddings+using+siamese+bert%2Dnetworks%2E+arXiv+preprint+arXiv%3A1908%2E10084%2C+2019%2E+23)
38. T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilić, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022. 10 [find](https://scholar.google.com/scholar?q=T%2E+L%2E+Scao%2C+A%2E+Fan%2C+C%2E+Akiki%2C+E%2E+Pavlick%2C+S%2E+Ilic%CC%81%2C+D%2E+Hesslow%2C+R%2E+Castagn%C3%A9%2C+A%2E+S%2E+Luccioni%2C+F%2E+Yvon%2C+M%2E+Gall%C3%A9%2C+et+al%2E+Bloom%3A+A+176b%2Dparameter+open%2Daccess+multilingual+language+model%2E+arXiv+preprint+a)
39. N. Shinn, B. Labash, and A. Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366, 2023. 10 [find](https://scholar.google.com/scholar?q=N%2E+Shinn%2C+B%2E+Labash%2C+and+A%2E+Gopinath%2E+Reflexion%3A+an+autonomous+agent+with+dynamic+memory+and+self%2Dreflection%2E+arXiv+preprint+arXiv%3A2303%2E11366%2C+2023%2E+10) [our card](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/)
40. M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning. PMLR, 2022. 2, 3, 15, 16, 17, 23 [find](https://scholar.google.com/scholar?q=M%2E+Shridhar%2C+L%2E+Manuelli%2C+and+D%2E+Fox%2E+Cliport%3A+What+and+where+pathways+for+robotic+manipulation%2E+In+Conference+on+Robot+Learning%2E+PMLR%2C+2022%2E+2%2C+3%2C+15%2C+16%2C+17%2C+23)
41. M. Shridhar, X. Yuan, M.-A. Côté, Y. Bisk, A. Trischler, and M. Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768, 2020. 2, 3, 15, 16 [find](https://scholar.google.com/scholar?q=M%2E+Shridhar%2C+X%2E+Yuan%2C+M%2E%2DA%2E+C%C3%B4t%C3%A9%2C+Y%2E+Bisk%2C+A%2E+Trischler%2C+and+M%2E+Hausknecht%2E+Alfworld%3A+Aligning+text+and+embodied+environments+for+interactive+learning%2E+arXiv+preprint+arXiv%3A2010%2E03768%2C+2020%2E+2%2C+3%2C+15%2C) [our card](https://md.research.elamur.ai/free/en/papers/shridhar-2020-alfworld-aligning-text-embodied-environments/)
42. I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. arXiv preprint arXiv:2209.11302, 2022. 3, 4, 7, 8, 10, 25, 26, 28 [find](https://scholar.google.com/scholar?q=I%2E+Singh%2C+V%2E+Blukis%2C+A%2E+Mousavian%2C+A%2E+Goyal%2C+D%2E+Xu%2C+J%2E+Tremblay%2C+D%2E+Fox%2C+J%2E+Thomason%2C+and+A%2E+Garg%2E+Progprompt%3A+Generating+situated+robot+task+plans+using+large+language+models%2E+arXiv+preprint+arXiv%3A22) [our card](https://md.research.elamur.ai/free/en/papers/singh-2022-progprompt-generating-situated-robot-task/)
43. A. Skrynnik, A. Staroverov, E. Aitygulov, K. Aksenov, V. Davydov, and A. I. Panov. Forgetful experience replay in hierarchical reinforcement learning from expert demonstrations. Knowledge-Based Systems, 218:106844, 2021. 9 [find](https://scholar.google.com/scholar?q=A%2E+Skrynnik%2C+A%2E+Staroverov%2C+E%2E+Aitygulov%2C+K%2E+Aksenov%2C+V%2E+Davydov%2C+and+A%2E+I%2E+Panov%2E+Forgetful+experience+replay+in+hierarchical+reinforcement+learning+from+expert+demonstrations%2E+Knowledge%2DBased+System)
44. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 22 [find](https://scholar.google.com/scholar?q=H%2E+Touvron%2C+L%2E+Martin%2C+K%2E+Stone%2C+P%2E+Albert%2C+A%2E+Almahairi%2C+Y%2E+Babaei%2C+N%2E+Bashlykov%2C+S%2E+Batra%2C+P%2E+Bhargava%2C+S%2E+Bhosale%2C+et+al%2E+Llama+2%3A+Open+foundation+and+fine%2Dtuned+chat+models%2E+arXiv+preprint+arXiv%3A2)
45. J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. 36th Conference on Neural Information Processing Systems (NeurIPS 2022), 2022. 5, 7, 8, 15, 25, 26, 29 [find](https://scholar.google.com/scholar?q=J%2E+Wei%2C+X%2E+Wang%2C+D%2E+Schuurmans%2C+M%2E+Bosma%2C+E%2E+Chi%2C+Q%2E+Le%2C+and+D%2E+Zhou%2E+Chain+of+thought+prompting+elicits+reasoning+in+large+language+models%2E+36th+Conference+on+Neural+Information+Processing+Systems+%28N) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
46. A. Zeng, A. Wong, S. Welker, K. Choromanski, F. Tombari, A. Purohit, M. Ryoo, V. Sindhwani, J. Lee, V. Vanhoucke, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598, 2022. 2, 9 [find](https://scholar.google.com/scholar?q=A%2E+Zeng%2C+A%2E+Wong%2C+S%2E+Welker%2C+K%2E+Choromanski%2C+F%2E+Tombari%2C+A%2E+Purohit%2C+M%2E+Ryoo%2C+V%2E+Sindhwani%2C+J%2E+Lee%2C+V%2E+Vanhoucke%2C+et+al%2E+Socratic+models%3A+Composing+zero%2Dshot+multimodal+reasoning+with+language%2E+arXiv+) [our card](https://md.research.elamur.ai/free/en/papers/zeng-2022-socratic-models-composing-zero-shot/)
47. C. Zhang, K. Yang, S. Hu, Z. Wang, G. Li, Y. Sun, C. Zhang, Z. Zhang, A. Liu, S.-C. Zhu, et al. Proagent: Building proactive cooperative ai with large language models. arXiv preprint arXiv:2308.11339, 2023. 10 — [arXiv:2308.11339](https://arxiv.org/abs/2308.11339) [find](https://scholar.google.com/scholar?q=C%2E+Zhang%2C+K%2E+Yang%2C+S%2E+Hu%2C+Z%2E+Wang%2C+G%2E+Li%2C+Y%2E+Sun%2C+C%2E+Zhang%2C+Z%2E+Zhang%2C+A%2E+Liu%2C+S%2E%2DC%2E+Zhu%2C+et+al%2E+Proagent%3A+Building+proactive+cooperative+ai+with+large+language+models%2E+arXiv+preprint+arXiv%3A2308%2E11339%2C)
48. S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. 10 [find](https://scholar.google.com/scholar?q=S%2E+Zhang%2C+S%2E+Roller%2C+N%2E+Goyal%2C+M%2E+Artetxe%2C+M%2E+Chen%2C+S%2E+Chen%2C+C%2E+Dewan%2C+M%2E+Diab%2C+X%2E+Li%2C+X%2E+V%2E+Lin%2C+et+al%2E+Opt%3A+Open+pre%2Dtrained+transformer+language+models%2E+arXiv+preprint+arXiv%3A2205%2E01068%2C+2022%2E+10)