---
title: "Language Models as Zero-Shot Planners — Huang 2022"
description: "The paper asks whether the everyday knowledge stored in large language models can become plans that an embodied agent can carry out, with no extra…"
authors: ["Wenlong Huang", "Pieter Abbeel", "Deepak Pathak", "Igor Mordatch"]
year: 2022
venue: "arXiv:2201.07207"
source: "https://arxiv.org/abs/2201.07207"
doi: "10.48550/arxiv.2201.07207"
arxiv: "2201.07207"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/huang-2022-language-models-zero-shot-planners/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/huang-2022-language-models-zero-shot-planners/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/) / Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents

# Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents

**Authors:** Wenlong Huang, [Pieter Abbeel](https://md.research.elamur.ai/free/en/authors/pieter-abbeel/), Deepak Pathak, [Igor Mordatch](https://md.research.elamur.ai/free/en/authors/igor-mordatch/) · **Year:** 2022 · **Venue:** arXiv:2201.07207 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2201.07207) · [PDF](https://arxiv.org/pdf/2201.07207) · [DOI 10.48550/arxiv.2201.07207](https://doi.org/10.48550/arxiv.2201.07207) · [arXiv 2201.07207](https://arxiv.org/abs/2201.07207) · [OpenAlex](https://openalex.org/W4221152848)

**Citations:** 151 citations (OpenAlex, 2026-10-14) · [269](https://md.research.elamur.ai/free/en/papers/huang-2022-language-models-zero-shot-planners/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
The paper asks whether the everyday knowledge stored in large language models can become plans that an embodied agent can carry out, with no extra training. The authors use the VirtualHome household simulator. They give frozen models such as GPT-3 and Codex a high-level chore name plus one example program. The largest models produce step lists that human raters find about as sensible as human-written ones. However, these free-text steps often do not match the simulator's allowed verb–object actions.

To fix this, the authors combine three techniques:
- map each generated step to the closest admissible action using RoBERTa sentence embeddings,
- feed each corrected step back into the prompt before generating the next one,
- choose a demonstration example that resembles the query task.

This sharply raises executability, although human judges see some loss in correctness. The work matters because it shows that LLM knowledge can be used for acting through prompting alone. The method needs no gradients or fine-tuning, so it fits existing model-serving pipelines.

## Key points
- Evaluation uses VirtualHome, which has 42 atomic actions, and its ActivityPrograms knowledge base: 2821 MTurk-annotated entries covering 292 distinct tasks. Of these, 88 tasks are held out for testing and 204 serve as the demonstration pool.
- Planning models include GPT-2 (0.1B, 1.5B), GPT-3 (13B, 175B) and Codex 12B. Plausible plans appear mainly at larger scales.
- Executability rises from 18% to 79% with the proposed pipeline: semantic translation, step-by-step autoregressive correction and dynamic example selection.
- Semantic correctness is judged by humans on Mechanical Turk. Results show a trade-off: executable plans are less often judged correct.
- Stated limitations include:
  - no use of environment observations or feedback,
  - grounding only to mid-level actions, assuming a low-level controller,
  - VirtualHome's limited set of possible actions.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Institutions

- [University of California, Berkeley](https://md.research.elamur.ai/free/en/universities/university-of-california-berkeley/) — University, Berkeley, United States · 107 works · 205 102 citations
- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 278 works · 138 810 citations

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{huang2022language,
  title = {{Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents}},
  author = {Huang, Wenlong and Abbeel, Pieter and Pathak, Deepak and Mordatch, Igor},
  year = {2022},
  doi = {10.48550/arxiv.2201.07207},
  eprint = {2201.07207},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2201.07207}
}
```

- [← previous Diffusion-LM Improves Controllable Text Generation](https://md.research.elamur.ai/free/en/papers/li-2022-diffusion-lm-improves-controllable-text/)
- [next → Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://md.research.elamur.ai/free/en/papers/davani-2022-dealing-disagreements-looking-beyond-majority/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/)

## References

55 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE Conference on Computer Vision and… [find](https://scholar.google.com/scholar?q=Peter+Anderson%2C+Qi+Wu%2C+Damien+Teney%2C+Jake+Bruce%2C+Mark+Johnson%2C+Niko+S%C3%BCnderhauf%2C+Ian+Reid%2C+Stephen+Gould%2C+and+Anton+Van+Den+Hengel%2E+Vision%2Dand%2Dlanguage+navigation%3A+Interpreting+visually%2Dgrounded+naviga) [our card](https://md.research.elamur.ai/free/en/papers/anderson-2017-vision-language-navigation-interpreting-visually/)
2. Yoav Artzi and Luke Zettlemoyer. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguistics, 1:49–62, 2013. [find](https://scholar.google.com/scholar?q=Yoav+Artzi+and+Luke+Zettlemoyer%2E+Weakly+supervised+learning+of+semantic+parsers+for+mapping+instructions+to+actions%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+1%3A49%E2%80%9362%2C+2013%2E) [our card](https://md.research.elamur.ai/free/en/papers/artzi-2013-weakly-supervised-learning-semantic-parsers/)
3. BIG-bench collaboration. Beyond the imitation game: Measuring and extrapolating the capabilities of language models. In preparation, 2021. URL [https://github.com/google/BIG-bench/](https://github.com/google/BIG-bench/). [↗](https://github.com/google/BIG-bench/)
4. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora De… [find](https://scholar.google.com/scholar?q=Rishi+Bommasani%2C+Drew+A%2E+Hudson%2C+Ehsan+Adeli%2C+Russ+Altman%2C+Simran+Arora%2C+Sydney+von+Arx%2C+Michael+S%2E+Bernstein%2C+Jeannette+Bohg%2C+Antoine+Bosselut%2C+Emma+Brunskill%2C+Erik+Brynjolfsson%2C+Shyamal+Buch%2C+Dallas) [our card](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)
5. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. [find](https://scholar.google.com/scholar?q=Tom+B+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+et+al%2E+Language+models+are+few%2Dshot+learners%2E+) [our card](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
6. Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055, 2017. [find](https://scholar.google.com/scholar?q=Daniel+Cer%2C+Mona+Diab%2C+Eneko+Agirre%2C+Inigo+Lopez%2DGazpio%2C+and+Lucia+Specia%2E+Semeval%2D2017+task+1%3A+Semantic+textual+similarity%2Dmultilingual+and+cross%2Dlingual+focused+evaluation%2E+arXiv+preprint+arXiv%3A1708) [our card](https://md.research.elamur.ai/free/en/papers/cer-2017-semeval-2017-task-1-semantic/)
7. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harri Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. [find](https://scholar.google.com/scholar?q=Mark+Chen%2C+Jerry+Tworek%2C+Heewoo+Jun%2C+Qiming+Yuan%2C+Henrique+Ponde%2C+Jared+Kaplan%2C+Harri+Edwards%2C+Yura+Burda%2C+Nicholas+Joseph%2C+Greg+Brockman%2C+et+al%2E+Evaluating+large+language+models+trained+on+code%2E+arXi) [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
8. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. [find](https://scholar.google.com/scholar?q=Karl+Cobbe%2C+Vineet+Kosaraju%2C+Mohammad+Bavarian%2C+Jacob+Hilton%2C+Reiichiro+Nakano%2C+Christopher+Hesse%2C+and+John+Schulman%2E+Training+verifiers+to+solve+math+word+problems%2E+arXiv+preprint+arXiv%3A2110%2E14168%2C+2) [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/)
9. Joe Davison, Joshua Feldman, and Alexander M Rush. Commonsense knowledge mining from pretrained models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1173–1178,… [find](https://scholar.google.com/scholar?q=Joe+Davison%2C+Joshua+Feldman%2C+and+Alexander+M+Rush%2E+Commonsense+knowledge+mining+from+pretrained+models%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+th) [our card](https://md.research.elamur.ai/free/en/papers/davison-2019-commonsense-knowledge-mining-pretrained-models/)
10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+arXiv+preprint+arXiv%3A1810%2E04805%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
11. Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell. Speaker-follower models for vision-and-language navigation. arXiv preprint arXiv:1806.02724, 2018. [find](https://scholar.google.com/scholar?q=Daniel+Fried%2C+Ronghang+Hu%2C+Volkan+Cirik%2C+Anna+Rohrbach%2C+Jacob+Andreas%2C+Louis%2DPhilippe+Morency%2C+Taylor+Berg%2DKirkpatrick%2C+Kate+Saenko%2C+Dan+Klein%2C+and+Trevor+Darrell%2E+Speaker%2Dfollower+models+for+vision%2Da)
12. Justin Fu, Anoop Korattikara, Sergey Levine, and Sergio Guadarrama. From language to goals: Inverse reinforcement learning for vision-based instruction following. arXiv preprint arXiv:1902.07742, 2019. — [arXiv:1902.07742](https://arxiv.org/abs/1902.07742) [find](https://scholar.google.com/scholar?q=Justin+Fu%2C+Anoop+Korattikara%2C+Sergey+Levine%2C+and+Sergio+Guadarrama%2E+From+language+to+goals%3A+Inverse+reinforcement+learning+for+vision%2Dbased+instruction+following%2E+arXiv+preprint+arXiv%3A1902%2E07742%2C+2019)
13. Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723, 2020. [find](https://scholar.google.com/scholar?q=Tianyu+Gao%2C+Adam+Fisch%2C+and+Danqi+Chen%2E+Making+pre%2Dtrained+language+models+better+few%2Dshot+learners%2E+arXiv+preprint+arXiv%3A2012%2E15723%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/gao-2021-making-pre-trained-language-models/)
14. Brent Harrison and Mark O Riedl. Learning from stories: using crowdsourced narratives to train virtual agents. In Twelfth Artificial Intelligence and Interactive Digital Entertainment Conference, 2016. [find](https://scholar.google.com/scholar?q=Brent+Harrison+and+Mark+O+Riedl%2E+Learning+from+stories%3A+using+crowdsourced+narratives+to+train+virtual+agents%2E+In+Twelfth+Artificial+Intelligence+and+Interactive+Digital+Entertainment+Conference%2C+2016)
15. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938, 2021. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks%2C+Steven+Basart%2C+Saurav+Kadavath%2C+Mantas+Mazeika%2C+Akul+Arora%2C+Ethan+Guo%2C+Collin+Burns%2C+Samir+Puranik%2C+Horace+He%2C+Dawn+Song%2C+et+al%2E+Measuring+coding+challenge+competence+with+apps%2E+arXiv+p) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-coding-challenge-competence-apps/)
16. Felix Hill, Sona Mokra, Nathaniel Wong, and Tim Harley. Human instruction-following with deep reinforcement learning via transfer-learning from text. arXiv preprint arXiv:2005.09382, 2020. — [arXiv:2005.09382](https://arxiv.org/abs/2005.09382) [find](https://scholar.google.com/scholar?q=Felix+Hill%2C+Sona+Mokra%2C+Nathaniel+Wong%2C+and+Tim+Harley%2E+Human+instruction%2Dfollowing+with+deep+reinforcement+learning+via+transfer%2Dlearning+from+text%2E+arXiv+preprint+arXiv%3A2005%2E09382%2C+2020%2E+%E2%80%94+%5BarXiv%3A20)
17. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735–1780, 1997. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+J%C3%BCrgen+Schmidhuber%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A+1735%E2%80%931780%2C+1997%2E)
18. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751, 2019. [find](https://scholar.google.com/scholar?q=Ari+Holtzman%2C+Jan+Buys%2C+Li+Du%2C+Maxwell+Forbes%2C+and+Yejin+Choi%2E+The+curious+case+of+neural+text+degeneration%2E+arXiv+preprint+arXiv%3A1904%2E09751%2C+2019%2E)
19. Gabriel Ilharco, Rowan Zellers, Ali Farhadi, and Hannaneh Hajishirzi. Probing text models for common ground with visual representations. arXiv e-prints, pages arXiv–2005, 2020. [find](https://scholar.google.com/scholar?q=Gabriel+Ilharco%2C+Rowan+Zellers%2C+Ali+Farhadi%2C+and+Hannaneh+Hajishirzi%2E+Probing+text+models+for+common+ground+with+visual+representations%2E+arXiv+e%2Dprints%2C+pages+arXiv%E2%80%932005%2C+2020%2E)
20. Peter A Jansen. Visually-grounded planning without vision: Language models infer detailed plans from high-level instructions. arXiv preprint arXiv:2009.14259, 2020. [find](https://scholar.google.com/scholar?q=Peter+A+Jansen%2E+Visually%2Dgrounded+planning+without+vision%3A+Language+models+infer+detailed+plans+from+high%2Dlevel+instructions%2E+arXiv+preprint+arXiv%3A2009%2E14259%2C+2020%2E)
21. Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423–438, 2020. [find](https://scholar.google.com/scholar?q=Zhengbao+Jiang%2C+Frank+F+Xu%2C+Jun+Araki%2C+and+Graham+Neubig%2E+How+can+we+know+what+language+models+know%3F+Transactions+of+the+Association+for+Computational+Linguistics%2C+8%3A423%E2%80%93438%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/jiang-2020-how-can-we-know-what/)
22. Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474, 2017. [find](https://scholar.google.com/scholar?q=Eric+Kolve%2C+Roozbeh+Mottaghi%2C+Winson+Han%2C+Eli+VanderBilt%2C+Luca+Weihs%2C+Alvaro+Herrasti%2C+Daniel+Gordon%2C+Yuke+Zhu%2C+Abhinav+Gupta%2C+and+Ali+Farhadi%2E+Ai2%2Dthor%3A+An+interactive+3d+environment+for+visual+ai%2E+a)
23. Belinda Z Li, Maxwell Nye, and Jacob Andreas. Implicit representations of meaning in neural language models. arXiv preprint arXiv:2106.00737, 2021. [find](https://scholar.google.com/scholar?q=Belinda+Z+Li%2C+Maxwell+Nye%2C+and+Jacob+Andreas%2E+Implicit+representations+of+meaning+in+neural+language+models%2E+arXiv+preprint+arXiv%3A2106%2E00737%2C+2021%2E)
24. Shuang Li, Xavier Puig, Yilun Du, Clinton Wang, Ekin Akyurek, Antonio Torralba, Jacob Andreas, and Igor Mordatch. Pre-trained language models for interactive decision-making. arXiv preprint arXiv:2202.01771, 2022. [find](https://scholar.google.com/scholar?q=Shuang+Li%2C+Xavier+Puig%2C+Yilun+Du%2C+Clinton+Wang%2C+Ekin+Akyurek%2C+Antonio+Torralba%2C+Jacob+Andreas%2C+and+Igor+Mordatch%2E+Pre%2Dtrained+language+models+for+interactive+decision%2Dmaking%2E+arXiv+preprint+arXiv%3A2202)
25. Yuan-Hong Liao, Xavier Puig, Marko Boben, Antonio Torralba, and Sanja Fidler. Synthesizing environment-aware activities via activity sketches. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6291–6299, 2019. [find](https://scholar.google.com/scholar?q=Yuan%2DHong+Liao%2C+Xavier+Puig%2C+Marko+Boben%2C+Antonio+Torralba%2C+and+Sanja+Fidler%2E+Synthesizing+environment%2Daware+activities+via+activity+sketches%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vis)
26. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06804, 2021. [find](https://scholar.google.com/scholar?q=Jiachang+Liu%2C+Dinghan+Shen%2C+Yizhe+Zhang%2C+Bill+Dolan%2C+Lawrence+Carin%2C+and+Weizhu+Chen%2E+What+makes+good+in%2Dcontext+examples+for+gpt%2D3%3F+arXiv+preprint+arXiv%3A2101%2E06804%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2022-what-makes-good-context-examples/)
27. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+arXiv+p)
28. Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Pretrained transformers as universal computation engines. arXiv preprint arXiv:2103.05247, 2021. — [arXiv:2103.05247](https://arxiv.org/abs/2103.05247) [find](https://scholar.google.com/scholar?q=Kevin+Lu%2C+Aditya+Grover%2C+Pieter+Abbeel%2C+and+Igor+Mordatch%2E+Pretrained+transformers+as+universal+computation+engines%2E+arXiv+preprint+arXiv%3A2103%2E05247%2C+2021%2E+%E2%80%94+%5BarXiv%3A2103%2E05247%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2)
29. Corey Lynch and Pierre Sermanet. arXiv:2005.07648, 2020. [find](https://scholar.google.com/scholar?q=Corey+Lynch+and+Pierre+Sermanet%2E+arXiv%3A2005%2E07648%2C+2020%2E)
30. Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. Proceedings of Robotics: Science and Systems. doi, 10, 2021. [find](https://scholar.google.com/scholar?q=Corey+Lynch+and+Pierre+Sermanet%2E+Language+conditioned+imitation+learning+over+unstructured+data%2E+Proceedings+of+Robotics%3A+Science+and+Systems%2E+doi%2C+10%2C+2021%2E)
31. Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, and Dhruv Batra. Improving vision-and-language navigation with image-text pairs from the web. In European Conference on Computer Vision, pages 259–274. Springer, 2020. [find](https://scholar.google.com/scholar?q=Arjun+Majumdar%2C+Ayush+Shrivastava%2C+Stefan+Lee%2C+Peter+Anderson%2C+Devi+Parikh%2C+and+Dhruv+Batra%2E+Improving+vision%2Dand%2Dlanguage+navigation+with+image%2Dtext+pairs+from+the+web%2E+In+European+Conference+on+Comp)
32. Suvir Mirchandani, Siddharth Karamcheti, and Dorsa Sadigh. Ella: Exploration through learned language abstraction. arXiv preprint arXiv:2103.05825, 2021. — [arXiv:2103.05825](https://arxiv.org/abs/2103.05825) [find](https://scholar.google.com/scholar?q=Suvir+Mirchandani%2C+Siddharth+Karamcheti%2C+and+Dorsa+Sadigh%2E+Ella%3A+Exploration+through+learned+language+abstraction%2E+arXiv+preprint+arXiv%3A2103%2E05825%2C+2021%2E+%E2%80%94+%5BarXiv%3A2103%2E05825%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F210)
33. Dipendra Misra, Kejia Tao, Percy Liang, and Ashutosh Saxena. Environment-driven lexicon induction for high-level instructions. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume… [find](https://scholar.google.com/scholar?q=Dipendra+Misra%2C+Kejia+Tao%2C+Percy+Liang%2C+and+Ashutosh+Saxena%2E+Environment%2Ddriven+lexicon+induction+for+high%2Dlevel+instructions%2E+In+Proceedings+of+the+53rd+Annual+Meeting+of+the+Association+for+Computat)
34. Dipendra K Misra, Jaeyong Sung, Kevin Lee, and Ashutosh Saxena. Tell me dave: Context-sensitive grounding of natural language to manipulation instructions. The International Journal of Robotics Research, 35(1-3):281–300, 2016. [find](https://scholar.google.com/scholar?q=Dipendra+K+Misra%2C+Jaeyong+Sung%2C+Kevin+Lee%2C+and+Ashutosh+Saxena%2E+Tell+me+dave%3A+Context%2Dsensitive+grounding+of+natural+language+to+manipulation+instructions%2E+The+International+Journal+of+Robotics+Resear)
35. Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+D+Manning%2E+Glove%3A+Global+vectors+for+word+representation%2E+In+Proceedings+of+the+2014+conference+on+empirical+methods+in+natural+language+processing+) [our card](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/)
36. Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? arXiv preprint arXiv:1909.01066, 2019. [find](https://scholar.google.com/scholar?q=Fabio+Petroni%2C+Tim+Rockt%C3%A4schel%2C+Patrick+Lewis%2C+Anton+Bakhtin%2C+Yuxiang+Wu%2C+Alexander+H+Miller%2C+and+Sebastian+Riedel%2E+Language+models+as+knowledge+bases%3F+arXiv+preprint+arXiv%3A1909%2E01066%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/)
37. Gabriel Poesia, Oleksandr Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. Synchromesh: Reliable code generation from pre-trained language models. arXiv preprint arXiv:2201.11227, 2022. [find](https://scholar.google.com/scholar?q=Gabriel+Poesia%2C+Oleksandr+Polozov%2C+Vu+Le%2C+Ashish+Tiwari%2C+Gustavo+Soares%2C+Christopher+Meek%2C+and+Sumit+Gulwani%2E+Synchromesh%3A+Reliable+code+generation+from+pre%2Dtrained+language+models%2E+arXiv+preprint+arX)
38. Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8494–8502, 2018. [find](https://scholar.google.com/scholar?q=Xavier+Puig%2C+Kevin+Ra%2C+Marko+Boben%2C+Jiaman+Li%2C+Tingwu+Wang%2C+Sanja+Fidler%2C+and+Antonio+Torralba%2E+Virtualhome%3A+Simulating+household+activities+via+programs%2E+In+Proceedings+of+the+IEEE+Conference+on+Comp)
39. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2C+2019%2E)
40. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J+Liu%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtext+trans) [our card](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/)
41. Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019. [find](https://scholar.google.com/scholar?q=Nils+Reimers+and+Iryna+Gurevych%2E+Sentence%2Dbert%3A+Sentence+embeddings+using+siamese+bert%2Dnetworks%2E+arXiv+preprint+arXiv%3A1908%2E10084%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/)
42. Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020. [find](https://scholar.google.com/scholar?q=Adam+Roberts%2C+Colin+Raffel%2C+and+Noam+Shazeer%2E+How+much+knowledge+can+you+pack+into+the+parameters+of+a+language+model%3F+arXiv+preprint+arXiv%3A2002%2E08910%2C+2020%2E)
43. Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. arXiv preprint arXiv:2112.08633, 2021. [find](https://scholar.google.com/scholar?q=Ohad+Rubin%2C+Jonathan+Herzig%2C+and+Jonathan+Berant%2E+Learning+to+retrieve+prompts+for+in%2Dcontext+learning%2E+arXiv+preprint+arXiv%3A2112%2E08633%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/rubin-2021-learning-retrieve-prompts-context-learning/)
44. Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9339–9347, 2019. [find](https://scholar.google.com/scholar?q=Manolis+Savva%2C+Abhishek+Kadian%2C+Oleksandr+Maksymets%2C+Yili+Zhao%2C+Erik+Wijmans%2C+Bhavana+Jain%2C+Julian+Straub%2C+Jia+Liu%2C+Vladlen+Koltun%2C+Jitendra+Malik%2C+et+al%2E+Habitat%3A+A+platform+for+embodied+ai+research%2E)
45. Pratyusha Sharma, Antonio Torralba, and Jacob Andreas. Skill induction and planning with latent language. arXiv preprint arXiv:2110.01517, 2021. [find](https://scholar.google.com/scholar?q=Pratyusha+Sharma%2C+Antonio+Torralba%2C+and+Jacob+Andreas%2E+Skill+induction+and+planning+with+latent+language%2E+arXiv+preprint+arXiv%3A2110%2E01517%2C+2021%2E)
46. Jianhao Shen, Yichun Yin, Lin Li, Lifeng Shang, Xin Jiang, Ming Zhang, and Qun Liu. Generate & rank: A multi-task framework for math word problems. arXiv preprint arXiv:2109.03034, 2021. Grounding language in play. arXiv preprint [find](https://scholar.google.com/scholar?q=Jianhao+Shen%2C+Yichun+Yin%2C+Lin+Li%2C+Lifeng+Shang%2C+Xin+Jiang%2C+Ming+Zhang%2C+and+Qun+Liu%2E+Generate+%26+rank%3A+A+multi%2Dtask+framework+for+math+word+problems%2E+arXiv+preprint+arXiv%3A2109%2E03034%2C+2021%2E+Grounding+lan)
47. Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1074… [find](https://scholar.google.com/scholar?q=Mohit+Shridhar%2C+Jesse+Thomason%2C+Daniel+Gordon%2C+Yonatan+Bisk%2C+Winson+Han%2C+Roozbeh+Mottaghi%2C+Luke+Zettlemoyer%2C+and+Dieter+Fox%2E+Alfred%3A+A+benchmark+for+interpreting+grounded+instructions+for+everyday+tas) [our card](https://md.research.elamur.ai/free/en/papers/shridhar-2019-alfred-benchmark-interpreting-grounded-instructions/)
48. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768, 2020. [find](https://scholar.google.com/scholar?q=Mohit+Shridhar%2C+Xingdi+Yuan%2C+Marc%2DAlexandre+C%C3%B4t%C3%A9%2C+Yonatan+Bisk%2C+Adam+Trischler%2C+and+Matthew+Hausknecht%2E+Alfworld%3A+Aligning+text+and+embodied+environments+for+interactive+learning%2E+arXiv+preprint+arXiv) [our card](https://md.research.elamur.ai/free/en/papers/shridhar-2020-alfworld-aligning-text-embodied-environments/)
49. Alessandro Suglia, Qiaozi Gao, Jesse Thomason, Govind Thattai, and Gaurav Sukhatme. Embodied bert: A transformer model for embodied, language-guided visual task completion. arXiv preprint arXiv:2108.04927, 2021. — [arXiv:2108.04927](https://arxiv.org/abs/2108.04927) [find](https://scholar.google.com/scholar?q=Alessandro+Suglia%2C+Qiaozi+Gao%2C+Jesse+Thomason%2C+Govind+Thattai%2C+and+Gaurav+Sukhatme%2E+Embodied+bert%3A+A+transformer+model+for+embodied%2C+language%2Dguided+visual+task+completion%2E+arXiv+preprint+arXiv%3A2108%2E0)
50. Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. olmpics-on what language model pre-training captures. Transactions of the Association for Computational Linguistics, 8: 743–758, 2020. [find](https://scholar.google.com/scholar?q=Alon+Talmor%2C+Yanai+Elazar%2C+Yoav+Goldberg%2C+and+Jonathan+Berant%2E+olmpics%2Don+what+language+model+pre%2Dtraining+captures%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+8%3A+743%E2%80%93758%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/talmor-2019-olmpics-language-model-pre-training/)
51. Moritz Tenorth, Daniel Nyga, and Michael Beetz. Understanding and executing instructions for everyday manipulation tasks from the world wide web. In 2010 ieee international conference on robotics and automation, pages 1486–1491. IEEE, 2010. [find](https://scholar.google.com/scholar?q=Moritz+Tenorth%2C+Daniel+Nyga%2C+and+Michael+Beetz%2E+Understanding+and+executing+instructions+for+everyday+manipulation+tasks+from+the+world+wide+web%2E+In+2010+ieee+international+conference+on+robotics+and+)
52. Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. arXiv preprint arXiv:2106.13884, 2021. [find](https://scholar.google.com/scholar?q=Maria+Tsimpoukelli%2C+Jacob+Menick%2C+Serkan+Cabi%2C+SM+Eslami%2C+Oriol+Vinyals%2C+and+Felix+Hill%2E+Multimodal+few%2Dshot+learning+with+frozen+language+models%2E+arXiv+preprint+arXiv%3A2106%2E13884%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/tsimpoukelli-2021-multimodal-few-shot-learning-frozen/)
53. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+Attention+is+all+you+need%2E+In+Advances+in+neural+information+processing+sys) [our card](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/)
54. Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec… [find](https://scholar.google.com/scholar?q=Xin+Wang%2C+Qiuyuan+Huang%2C+Asli+Celikyilmaz%2C+Jianfeng+Gao%2C+Dinghan+Shen%2C+Yuan%2DFang+Wang%2C+William+Yang+Wang%2C+and+Lei+Zhang%2E+Reinforced+cross%2Dmodal+matching+and+self%2Dsupervised+imitation+learning+for+visi)
55. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019. [find](https://scholar.google.com/scholar?q=Thomas+Wolf%2C+Lysandre+Debut%2C+Victor+Sanh%2C+Julien+Chaumond%2C+Clement+Delangue%2C+Anthony+Moi%2C+Pierric+Cistac%2C+Tim+Rault%2C+R%C3%A9mi+Louf%2C+Morgan+Funtowicz%2C+et+al%2E+Huggingface%E2%80%99s+transformers%3A+State%2Dof%2Dthe%2Dart+na) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)