---
title: "Distilling Internet-Scale Vision-Language Models into Embodied"
description: "The authors propose using a pretrained generative vision-language model as an automatic annotator. 81 references, cited by 6 papers here."
authors: ["Theodore Sumers", "Kenneth Marino", "Arun Ahuja", "Rob Fergus", "Ishita Dasgupta"]
year: 2023
venue: "arXiv:2301.12507"
source: "https://arxiv.org/abs/2301.12507"
doi: "10.48550/arxiv.2301.12507"
arxiv: "2301.12507"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sumers-2023-distilling-internet-scale-vision-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sumers-2023-distilling-internet-scale-vision-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/15/) / Distilling Internet-Scale Vision-Language Models into Embodied Agents

# Distilling Internet-Scale Vision-Language Models into Embodied Agents

**Authors:** Theodore Sumers, [Kenneth Marino](https://md.research.elamur.ai/free/en/authors/kenneth-marino/), [Arun Ahuja](https://md.research.elamur.ai/free/en/authors/arun-ahuja/), [Rob Fergus](https://md.research.elamur.ai/free/en/authors/rob-fergus/) and [Ishita Dasgupta](https://md.research.elamur.ai/free/en/authors/ishita-dasgupta/) · **Year:** 2023 · **Venue:** arXiv:2301.12507 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2301.12507) · [PDF](https://arxiv.org/pdf/2301.12507) · [DOI 10.48550/arxiv.2301.12507](https://doi.org/10.48550/arxiv.2301.12507) · [arXiv 2301.12507](https://arxiv.org/abs/2301.12507) · [OpenAlex](https://openalex.org/W4318751977)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [6](https://md.research.elamur.ai/free/en/papers/sumers-2023-distilling-internet-scale-vision-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Language-conditioned embodied agents must link words to what they perceive and do. The paired trajectory–language data needed for this is costly to collect, often ambiguous, and tied to one agent's design. The authors propose using a pretrained generative vision-language model as an automatic annotator.

Building on hindsight experience replay (HER), the Flamingo VLM inspects the final observation of an agent's episode and writes an instruction that the episode actually satisfied. The relabeled trajectories then train the agent through imitation learning. Because the method works purely as data augmentation, it does not depend on the agent architecture, and its labels are human-readable.

Varying the prompt, or adding few-shot examples, controls which aspect of the scene gets described. In a 3D-rendered environment this taught object names, colours, categories such as food versus toys, and arbitrary user preferences. An analysis of noisy labels found that precision matters far more than accuracy. The work offers an inexpensive route for transferring internet-scale grounding into agents without heavy human annotation.

## Key points
- The relabeling function is Flamingo, used zero-shot or few-shot. Swapping in the OWL-ViT detector as relabeler performed substantially worse.
- Few-shot prompting used 32 in-context examples (3–4 for each of 10 objects). It raised the share of labels containing the correct object name from 54.7% to 77.2%. However, agent task success fell from 64.4% (Zeroshot) to 52.9% (Fewshot).
- A mixed-effects regression over 40 label sets (10 tasks × 4 labeling variants) gave precision an effect about ten times that of accuracy (β = 1.44 vs β = 0.16).
- Discarding the least confident 50% of labels gave Fewshot agents a large gain but barely changed Zeroshot agents.
- Limitations:
  - The work covers English only and relabels only the final observation.
  - A low-level motor policy was first learned from the Interactive Agents Team (2021) dataset.
  - Flamingo is not public; GPT-4 or Prismer are suggested as substitutes.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,621 works · [⚙️ Engineering](https://md.research.elamur.ai/free/en/shelves/engineering/) · 206 works · [⚙️ Control and Systems Engineering](https://md.research.elamur.ai/free/en/shelves/engineering/control-and-systems-engineering/) · 72 works

🧭 Topics (1): [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 827 works

## Cite

Cite as: Theodore Sumers, Kenneth Marino, Arun Ahuja, Rob Fergus and Ishita Dasgupta. 2023. Distilling Internet-Scale Vision-Language Models into Embodied Agents. arXiv:2301.12507.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{sumers2023distilling,
  title = {{Distilling Internet-Scale Vision-Language Models into Embodied Agents}},
  author = {Theodore Sumers, Kenneth Marino, Arun Ahuja, Rob Fergus and Ishita Dasgupta},
  year = {2023},
  doi = {10.48550/arxiv.2301.12507},
  eprint = {2301.12507},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2301.12507}
}
```

- [← previous Dialogue Act-Aided Backchannel Prediction Using Multi-Task Learning](https://md.research.elamur.ai/free/en/papers/liermann-2023-dialogue-act-aided-backchannel-prediction/)
- [next → Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization of RLHF](https://md.research.elamur.ai/free/en/papers/sun-2023-exploring-impact-low-rank-adaptation/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/15/)

## References

81 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abramson, J., Ahuja, A., Barr, I., Brussee, A., Carnevale, F., Cassin, M., Chhaparia, R., Clark, S., Damoc, B., Dudzik, A., et al. Imitating interactive intelligence. arXiv preprint arXiv:2012.05672, 2020. — [arXiv:2012.05672](https://arxiv.org/abs/2012.05672) [find](https://scholar.google.com/scholar?q=Abramson%2C+J%2E%2C+Ahuja%2C+A%2E%2C+Barr%2C+I%2E%2C+Brussee%2C+A%2E%2C+Carnevale%2C+F%2E%2C+Cassin%2C+M%2E%2C+Chhaparia%2C+R%2E%2C+Clark%2C+S%2E%2C+Damoc%2C+B%2E%2C+Dudzik%2C+A%2E%2C+et+al%2E+Imitating+interactive+intelligence%2E+arXiv+preprint+arXiv%3A2012%2E05672%2C+)
2. Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., et al. [Do as I can, not as I say: Grounding language in robotic affordances](https://md.research.elamur.ai/free/en/papers/ahn-2022-i-i-say-grounding-language/). arXiv preprint arXiv:2204.01691, 2022.
3. Future work may experiment with other ways to leverage pretrained generative VLMs for embodied agents. Following our offline supervision approach, VLMs could provide a reward signal during training on the basis of vision and text alignment. Alternatively, VLMs could be used directly 4 Our VLM (Alayr… [find](https://scholar.google.com/scholar?q=Future+work+may+experiment+with+other+ways+to+leverage+pretrained+generative+VLMs+for+embodied+agents%2E+Following+our+offline+supervision+approach%2C+VLMs+could+provide+a+reward+signal+during+training+on)
4. Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M., Barreira,… — [Flamingo: a Visual Language Model for Few-Shot Learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/)
5. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T. J., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S.,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
6. Anderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., Sünderhauf, N., Reid, I., Gould, S., and van den Hengel, A. [Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments](https://md.research.elamur.ai/free/en/papers/anderson-2017-vision-language-navigation-interpreting-visually/). In CVPR, pp. 3674–3683, 2018.
7. Bucker, A. F. C., Figueredo, L. F. C., Haddadin, S., Kapoor, A., Ma, S., Vemprala, S., and Bonatti, R. LaTTe: Language trajectory transformEr. ArXiv, abs/2208.02918, 2022. — [arXiv:2208.02918](https://arxiv.org/abs/2208.02918) [find](https://scholar.google.com/scholar?q=Bucker%2C+A%2E+F%2E+C%2E%2C+Figueredo%2C+L%2E+F%2E+C%2E%2C+Haddadin%2C+S%2E%2C+Kapoor%2C+A%2E%2C+Ma%2C+S%2E%2C+Vemprala%2C+S%2E%2C+and+Bonatti%2C+R%2E+LaTTe%3A+Language+trajectory+transformEr%2E+ArXiv%2C+abs%2F2208%2E02918%2C+2022%2E+%E2%80%94+%5BarXiv%3A2208%2E02918%5D%28https%3A%2F)
8. Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Pieter Abbeel, O., and Zaremba, W. [Hindsight experience replay](https://md.research.elamur.ai/free/en/papers/andrychowicz-2017-hindsight-experience-replay/). Advances in neural information processing systems, 30, 2017.
9. Chai, J. Y., Gao, Q., She, L., Yang, S., Saba-Sadiya, S., and Xu, G. Language to action: Towards interactive task learning with physical agents. In IJCAI, pp. 2–9, 2018. [find](https://scholar.google.com/scholar?q=Chai%2C+J%2E+Y%2E%2C+Gao%2C+Q%2E%2C+She%2C+L%2E%2C+Yang%2C+S%2E%2C+Saba%2DSadiya%2C+S%2E%2C+and+Xu%2C+G%2E+Language+to+action%3A+Towards+interactive+task+learning+with+physical+agents%2E+In+IJCAI%2C+pp%2E+2%E2%80%939%2C+2018%2E)
10. Chan, H., Wu, Y., Kiros, J., Fidler, S., and Ba, J. Actrce: Augmenting experience via teacher’s advice for multi-goal reinforcement learning. arXiv preprint arXiv:1902.04546, 2019. [find](https://scholar.google.com/scholar?q=Chan%2C+H%2E%2C+Wu%2C+Y%2E%2C+Kiros%2C+J%2E%2C+Fidler%2C+S%2E%2C+and+Ba%2C+J%2E+Actrce%3A+Augmenting+experience+via+teacher%E2%80%99s+advice+for+multi%2Dgoal+reinforcement+learning%2E+arXiv+preprint+arXiv%3A1902%2E04546%2C+2019%2E)
11. Artzi, Y. and Zettlemoyer, L. [Weakly supervised learning of semantic parsers for mapping instructions to actions](https://md.research.elamur.ai/free/en/papers/artzi-2013-weakly-supervised-learning-semantic-parsers/). Transactions of the Association for Computational Linguistics, 1:49–62, 2013.
12. Chaplot, D. S., Sathyendra, K. M., Pasumarthi, R. K., Rajagopal, D., and Salakhutdinov, R. Gated-attention architectures for task-oriented language grounding. In AAAI, 2018. [find](https://scholar.google.com/scholar?q=Chaplot%2C+D%2E+S%2E%2C+Sathyendra%2C+K%2E+M%2E%2C+Pasumarthi%2C+R%2E+K%2E%2C+Rajagopal%2C+D%2E%2C+and+Salakhutdinov%2C+R%2E+Gated%2Dattention+architectures+for+task%2Doriented+language+grounding%2E+In+AAAI%2C+2018%2E)
13. Bahdanau, D., Hill, F., Leike, J., Hughes, E., Hosseini, A., Kohli, P., and Grefenstette, E. Learning to understand goal specifications by modelling reward. In International Conference on Learning Representations, 2018. [find](https://scholar.google.com/scholar?q=Bahdanau%2C+D%2E%2C+Hill%2C+F%2E%2C+Leike%2C+J%2E%2C+Hughes%2C+E%2E%2C+Hosseini%2C+A%2E%2C+Kohli%2C+P%2E%2C+and+Grefenstette%2C+E%2E+Learning+to+understand+goal+specifications+by+modelling+reward%2E+In+International+Conference+on+Learning+Rep)
14. Chen, B., Xia, F., Ichter, B., Rao, K., Gopalakrishnan, K., Ryoo, M. S., Stone, A., and Kappler, D. Open-vocabulary queryable scene representations for real world planning. arXiv preprint arXiv:2209.09874, 2022a. — [arXiv:2209.09874](https://arxiv.org/abs/2209.09874) [find](https://scholar.google.com/scholar?q=Chen%2C+B%2E%2C+Xia%2C+F%2E%2C+Ichter%2C+B%2E%2C+Rao%2C+K%2E%2C+Gopalakrishnan%2C+K%2E%2C+Ryoo%2C+M%2E+S%2E%2C+Stone%2C+A%2E%2C+and+Kappler%2C+D%2E+Open%2Dvocabulary+queryable+scene+representations+for+real+world+planning%2E+arXiv+preprint+arXiv%3A2209%2E0)
15. Barsalou, L. W. Ad hoc categories. Memory & cognition, 11:211–227, 1983. [find](https://scholar.google.com/scholar?q=Barsalou%2C+L%2E+W%2E+Ad+hoc+categories%2E+Memory+%26+cognition%2C+11%3A211%E2%80%93227%2C+1983%2E)
16. Bigazzi, R., Landi, F., Cornia, M., Cascianelli, S., Baraldi, L., and Cucchiara, R. Explore and explain: Self-supervised navigation and recounting. In 25th International Conference on Pattern Recognition (ICPR), pp. 1152–1159, 2021. [find](https://scholar.google.com/scholar?q=Bigazzi%2C+R%2E%2C+Landi%2C+F%2E%2C+Cornia%2C+M%2E%2C+Cascianelli%2C+S%2E%2C+Baraldi%2C+L%2E%2C+and+Cucchiara%2C+R%2E+Explore+and+explain%3A+Self%2Dsupervised+navigation+and+recounting%2E+In+25th+International+Conference+on+Pattern+Recognit)
17. Chen, D. and Mooney, R. Learning to interpret natural language navigation instructions from observations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 25, pp. 859–865, 2011. [find](https://scholar.google.com/scholar?q=Chen%2C+D%2E+and+Mooney%2C+R%2E+Learning+to+interpret+natural+language+navigation+instructions+from+observations%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+volume+25%2C+pp%2E+859%E2%80%93865%2C+2011)
18. Blukis, V., Terme, Y., Niklasson, E., Knepper, R. A., and Artzi, Y. Learning to map natural language instructions to physical quadcopter control using simulated flight. In CoRL, 2019. [find](https://scholar.google.com/scholar?q=Blukis%2C+V%2E%2C+Terme%2C+Y%2E%2C+Niklasson%2C+E%2E%2C+Knepper%2C+R%2E+A%2E%2C+and+Artzi%2C+Y%2E+Learning+to+map+natural+language+instructions+to+physical+quadcopter+control+using+simulated+flight%2E+In+CoRL%2C+2019%2E)
19. Chen, S., Guhur, P.-L., Tapaswi, M., Schmid, C., and Laptev, I. Learning from unlabeled 3d environments for vision-and-language navigation. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIX, pp. 638–655. [find](https://scholar.google.com/scholar?q=Chen%2C+S%2E%2C+Guhur%2C+P%2E%2DL%2E%2C+Tapaswi%2C+M%2E%2C+Schmid%2C+C%2E%2C+and+Laptev%2C+I%2E+Learning+from+unlabeled+3d+environments+for+vision%2Dand%2Dlanguage+navigation%2E+In+Computer+Vision%E2%80%93ECCV+2022%3A+17th+European+Conference%2C+Tel+)
20. Blukis, V., Knepper, R. A., and Artzi, Y. Few-shot object grounding and mapping for natural language robot instruction following. arXiv preprint arXiv:2011.07384, abs/2011.07384, 2020. — [arXiv:2011.07384](https://arxiv.org/abs/2011.07384) [find](https://scholar.google.com/scholar?q=Blukis%2C+V%2E%2C+Knepper%2C+R%2E+A%2E%2C+and+Artzi%2C+Y%2E+Few%2Dshot+object+grounding+and+mapping+for+natural+language+robot+instruction+following%2E+arXiv+preprint+arXiv%3A2011%2E07384%2C+abs%2F2011%2E07384%2C+2020%2E+%E2%80%94+%5BarXiv%3A2011%2E0)
21. Chen, V., Gupta, A. K., and Marino, K. Ask your humans: Using human instructions to improve generalization in reinforcement learning. ICLR, 2020. [find](https://scholar.google.com/scholar?q=Chen%2C+V%2E%2C+Gupta%2C+A%2E+K%2E%2C+and+Marino%2C+K%2E+Ask+your+humans%3A+Using+human+instructions+to+improve+generalization+in+reinforcement+learning%2E+ICLR%2C+2020%2E)
22. Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. [On the opportunities and risks of foundation models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/). arXiv preprint arXiv:2108.07258, 2021.
23. Cideron, G., Seurin, M., Strub, F., and Pietquin, O. Higher: Improving instruction following with hindsight generation for experience replay. In 2020 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 225– 232. [find](https://scholar.google.com/scholar?q=Cideron%2C+G%2E%2C+Seurin%2C+M%2E%2C+Strub%2C+F%2E%2C+and+Pietquin%2C+O%2E+Higher%3A+Improving+instruction+following+with+hindsight+generation+for+experience+replay%2E+In+2020+IEEE+Symposium+Series+on+Computational+Intelligenc)
24. Branavan, S., Silver, D., and Barzilay, R. Learning to win by reading manuals in a Monte-Carlo framework. Journal of Artificial Intelligence Research, 43:661–704, 2012. [find](https://scholar.google.com/scholar?q=Branavan%2C+S%2E%2C+Silver%2C+D%2E%2C+and+Barzilay%2C+R%2E+Learning+to+win+by+reading+manuals+in+a+Monte%2DCarlo+framework%2E+Journal+of+Artificial+Intelligence+Research%2C+43%3A661%E2%80%93704%2C+2012%2E)
25. Co-Reyes, J. D., Gupta, A., Sanjeev, S., Altieri, N., Andreas, J., DeNero, J., Abbeel, P., and Levine, S. Guiding policies with language via meta-learning. In International Conference on Learning Representations, 2018. [find](https://scholar.google.com/scholar?q=Co%2DReyes%2C+J%2E+D%2E%2C+Gupta%2C+A%2E%2C+Sanjeev%2C+S%2E%2C+Altieri%2C+N%2E%2C+Andreas%2C+J%2E%2C+DeNero%2C+J%2E%2C+Abbeel%2C+P%2E%2C+and+Levine%2C+S%2E+Guiding+policies+with+language+via+meta%2Dlearning%2E+In+International+Conference+on+Learning+Repr)
26. Huang, W., Xia, F., Xiao, T., Chan, H., Liang, J., Florence, P., Zeng, A., Tompson, J., Mordatch, I., Chebotar, Y., Sermanet, P., Jackson, T., Brown, N., Luu, L., Levine, S., Hausman, K., and Ichter, B. Inner monologue: Embodied reasoning through planning with language models. In 6th Annual Conferen… [find](https://scholar.google.com/scholar?q=Huang%2C+W%2E%2C+Xia%2C+F%2E%2C+Xiao%2C+T%2E%2C+Chan%2C+H%2E%2C+Liang%2C+J%2E%2C+Florence%2C+P%2E%2C+Zeng%2C+A%2E%2C+Tompson%2C+J%2E%2C+Mordatch%2C+I%2E%2C+Chebotar%2C+Y%2E%2C+Sermanet%2C+P%2E%2C+Jackson%2C+T%2E%2C+Brown%2C+N%2E%2C+Luu%2C+L%2E%2C+Levine%2C+S%2E%2C+Hausman%2C+K%2E%2C+and+Ichter%2C+)
27. Dasgupta, I., Kaeser-Chen, C., Marino, K., Ahuja, A., Babayan, S., Hill, F., and Fergus, R. [Collaborating with language models for embodied reasoning](https://md.research.elamur.ai/free/en/papers/dasgupta-2023-collaborating-language-models-embodied-reasoning/). In Second Workshop on Language and Reinforcement Learning, 2022.
28. Interactive Agents Team, D. Creating multimodal interactive agents with imitation and self-supervised learning. arXiv preprint arXiv:2112.03763, 2021. — [arXiv:2112.03763](https://arxiv.org/abs/2112.03763) [find](https://scholar.google.com/scholar?q=Interactive+Agents+Team%2C+D%2E+Creating+multimodal+interactive+agents+with+imitation+and+self%2Dsupervised+learning%2E+arXiv+preprint+arXiv%3A2112%2E03763%2C+2021%2E+%E2%80%94+%5BarXiv%3A2112%2E03763%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2112%2E0)
29. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. N. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). 2018. URL https://arxiv.org/abs/1810.04805. [↗](https://arxiv.org/abs/1810.04805)
30. Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. [Segment anything](https://md.research.elamur.ai/free/en/papers/kirillov-2023-segment-anything/). arXiv preprint arXiv:2304.02643, 2023.
31. Fan, L. J., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H., Tang, A., Huang, D.-A., Zhu, Y., and Anandkumar, A. [MineDojo: Building open-ended embodied agents with internet-scale knowledge](https://md.research.elamur.ai/free/en/papers/fan-2022-minedojo-building-open-ended-embodied/). Advances in Neural Information Processing Systems, 2022.
32. Kollar, T., Tellex, S., Roy, D., and Roy, N. Toward understanding natural language directions. In 2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 259–266. [find](https://scholar.google.com/scholar?q=Kollar%2C+T%2E%2C+Tellex%2C+S%2E%2C+Roy%2C+D%2E%2C+and+Roy%2C+N%2E+Toward+understanding+natural+language+directions%2E+In+2010+5th+ACM%2FIEEE+International+Conference+on+Human%2DRobot+Interaction+%28HRI%29%2C+pp%2E+259%E2%80%93266%2E)
33. Kulick, J., Toussaint, M., Lang, T., and Lopes, M. Active learning for teaching a robot grounded relational symbols. In IJCAI, pp. 1451–1457. Citeseer, 2013. [find](https://scholar.google.com/scholar?q=Kulick%2C+J%2E%2C+Toussaint%2C+M%2E%2C+Lang%2C+T%2E%2C+and+Lopes%2C+M%2E+Active+learning+for+teaching+a+robot+grounded+relational+symbols%2E+In+IJCAI%2C+pp%2E+1451%E2%80%931457%2E+Citeseer%2C+2013%2E)
34. Fried, D., Hu, R., Cirik, V., Rohrbach, A., Andreas, J., Morency, L.-P., Berg-Kirkpatrick, T., Saenko, K., Klein, D., and Darrell, T. Speaker-follower models for vision-and-language navigation. Advances in Neural Information Processing Systems, 31, 2018. [find](https://scholar.google.com/scholar?q=Fried%2C+D%2E%2C+Hu%2C+R%2E%2C+Cirik%2C+V%2E%2C+Rohrbach%2C+A%2E%2C+Andreas%2C+J%2E%2C+Morency%2C+L%2E%2DP%2E%2C+Berg%2DKirkpatrick%2C+T%2E%2C+Saenko%2C+K%2E%2C+Klein%2C+D%2E%2C+and+Darrell%2C+T%2E+Speaker%2Dfollower+models+for+vision%2Dand%2Dlanguage+navigation%2E+Advanc)
35. Kuo, W., Cui, Y., Gu, X., Piergiovanni, A., and Angelova, A. F-vlm: Open-vocabulary object detection upon frozen vision and language models. arXiv preprint arXiv:2209.15639, 2022. — [arXiv:2209.15639](https://arxiv.org/abs/2209.15639) [find](https://scholar.google.com/scholar?q=Kuo%2C+W%2E%2C+Cui%2C+Y%2E%2C+Gu%2C+X%2E%2C+Piergiovanni%2C+A%2E%2C+and+Angelova%2C+A%2E+F%2Dvlm%3A+Open%2Dvocabulary+object+detection+upon+frozen+vision+and+language+models%2E+arXiv+preprint+arXiv%3A2209%2E15639%2C+2022%2E+%E2%80%94+%5BarXiv%3A2209%2E15639%5D)
36. Guhur, P.-L., Tapaswi, M., Chen, S., Laptev, I., and Schmid, C. Airbert: In-domain pretraining for vision-and-language navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1634– 1643, 2021. [find](https://scholar.google.com/scholar?q=Guhur%2C+P%2E%2DL%2E%2C+Tapaswi%2C+M%2E%2C+Chen%2C+S%2E%2C+Laptev%2C+I%2E%2C+and+Schmid%2C+C%2E+Airbert%3A+In%2Ddomain+pretraining+for+vision%2Dand%2Dlanguage+navigation%2E+In+Proceedings+of+the+IEEE%2FCVF+International+Conference+on+Computer+V)
37. Li, C., Xia, F., Martı́n-Martı́n, R., Lingelbach, M., Srivastava, S., Shen, B., Vainio, K., Gokmen, C., Dharan, G., Jain, T., et al. igibson 2.0: Object-centric simulation for robot learning of everyday household tasks. arXiv preprint arXiv:2108.03272, 2021. — [arXiv:2108.03272](https://arxiv.org/abs/2108.03272) [find](https://scholar.google.com/scholar?q=Li%2C+C%2E%2C+Xia%2C+F%2E%2C+Mart%C4%B1%CC%81n%2DMart%C4%B1%CC%81n%2C+R%2E%2C+Lingelbach%2C+M%2E%2C+Srivastava%2C+S%2E%2C+Shen%2C+B%2E%2C+Vainio%2C+K%2E%2C+Gokmen%2C+C%2E%2C+Dharan%2C+G%2E%2C+Jain%2C+T%2E%2C+et+al%2E+igibson+2%2E0%3A+Object%2Dcentric+simulation+for+robot+learning+of+everyd)
38. Hao, W., Li, C., Li, X., Carin, L., and Gao, J. Towards learning a generic agent for vision-and-language navigation via pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13137–13146, 2020. [find](https://scholar.google.com/scholar?q=Hao%2C+W%2E%2C+Li%2C+C%2E%2C+Li%2C+X%2E%2C+Carin%2C+L%2E%2C+and+Gao%2C+J%2E+Towards+learning+a+generic+agent+for+vision%2Dand%2Dlanguage+navigation+via+pre%2Dtraining%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+P)
39. Li, X., Li, C., Xia, Q., Bisk, Y., Celikyilmaz, A., Gao, J., Smith, N. A., and Choi, Y. Robust navigation with language pretraining and stochastic sampling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural… [find](https://scholar.google.com/scholar?q=Li%2C+X%2E%2C+Li%2C+C%2E%2C+Xia%2C+Q%2E%2C+Bisk%2C+Y%2E%2C+Celikyilmaz%2C+A%2E%2C+Gao%2C+J%2E%2C+Smith%2C+N%2E+A%2E%2C+and+Choi%2C+Y%2E+Robust+navigation+with+language+pretraining+and+stochastic+sampling%2E+In+Proceedings+of+the+2019+Conference+on+Em)
40. Harnad, S. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1-3):335–346, 1990. [find](https://scholar.google.com/scholar?q=Harnad%2C+S%2E+The+symbol+grounding+problem%2E+Physica+D%3A+Nonlinear+Phenomena%2C+42%281%2D3%29%3A335%E2%80%93346%2C+1990%2E)
41. Hill, F., Tieleman, O., von Glehn, T., Wong, N., Merzic, H., and Clark, S. Grounded language learning fast and slow. In International Conference on Learning Representations, 2021. [find](https://scholar.google.com/scholar?q=Hill%2C+F%2E%2C+Tieleman%2C+O%2E%2C+von+Glehn%2C+T%2E%2C+Wong%2C+N%2E%2C+Merzic%2C+H%2E%2C+and+Clark%2C+S%2E+Grounded+language+learning+fast+and+slow%2E+In+International+Conference+on+Learning+Representations%2C+2021%2E)
42. Lin, J., Fried, D., Klein, D., and Dragan, A. Inferring rewards from language in context. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8546– 8560, Dublin, Ireland, May 2022. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Lin%2C+J%2E%2C+Fried%2C+D%2E%2C+Klein%2C+D%2E%2C+and+Dragan%2C+A%2E+Inferring+rewards+from+language+in+context%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Pap)
43. Hsu, K., Lum, T. G. W., Gao, R., Gu, S. S., Wu, J., and Finn, C. What makes certain pre-trained visual representations better for robotic learning? In NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022. [find](https://scholar.google.com/scholar?q=Hsu%2C+K%2E%2C+Lum%2C+T%2E+G%2E+W%2E%2C+Gao%2C+R%2E%2C+Gu%2C+S%2E+S%2E%2C+Wu%2C+J%2E%2C+and+Finn%2C+C%2E+What+makes+certain+pre%2Dtrained+visual+representations+better+for+robotic+learning%3F+In+NeurIPS+2022+Foundation+Models+for+Decision+Makin)
44. Liu, S., Fan, L., Johns, E., Yu, Z., Xiao, C., and Anandkumar, A. Prismer: A vision-language model with an ensemble of experts. arXiv preprint arXiv:2303.02506, 2023. — [arXiv:2303.02506](https://arxiv.org/abs/2303.02506) [find](https://scholar.google.com/scholar?q=Liu%2C+S%2E%2C+Fan%2C+L%2E%2C+Johns%2C+E%2E%2C+Yu%2C+Z%2E%2C+Xiao%2C+C%2E%2C+and+Anandkumar%2C+A%2E+Prismer%3A+A+vision%2Dlanguage+model+with+an+ensemble+of+experts%2E+arXiv+preprint+arXiv%3A2303%2E02506%2C+2023%2E+%E2%80%94+%5BarXiv%3A2303%2E02506%5D%28https%3A%2F%2Farxi)
45. Huang, W., Abbeel, P., Pathak, D., and Mordatch, I. [Language models as zero-shot planners: Extracting actionable knowledge for embodied agents](https://md.research.elamur.ai/free/en/papers/huang-2022-language-models-zero-shot-planners/). ArXiv, abs/2201.07207, 2022a.
46. Lynch, C. and Sermanet, P. Grounding language in play. arXiv preprint arXiv:2005.[07648, 2020](https://md.research.elamur.ai/free/en/papers/lynch-2020-language-conditioned-imitation-learning-over/).
47. Majumdar, A., Aggarwal, G., Devnani, B. S., Hoffman, J., and Batra, D. ZSON: Zero-shot object-goal navigation using multimodal goal embeddings. In Advances in Neural Information Processing Systems, 2022. [find](https://scholar.google.com/scholar?q=Majumdar%2C+A%2E%2C+Aggarwal%2C+G%2E%2C+Devnani%2C+B%2E+S%2E%2C+Hoffman%2C+J%2E%2C+and+Batra%2C+D%2E+ZSON%3A+Zero%2Dshot+object%2Dgoal+navigation+using+multimodal+goal+embeddings%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+20)
48. Quine, W. V. O. Word & Object. MIT Press, 1960. [find](https://scholar.google.com/scholar?q=Quine%2C+W%2E+V%2E+O%2E+Word+%26+Object%2E+MIT+Press%2C+1960%2E)
49. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). In International Conference on Machine Learning, 2021.
50. Mei, H., Bansal, M., and Walter, M. R. Listen, attend, and walk: Neural mapping of navigational instructions to action sequences. In AAAI, 2016. [find](https://scholar.google.com/scholar?q=Mei%2C+H%2E%2C+Bansal%2C+M%2E%2C+and+Walter%2C+M%2E+R%2E+Listen%2C+attend%2C+and+walk%3A+Neural+mapping+of+navigational+instructions+to+action+sequences%2E+In+AAAI%2C+2016%2E)
51. Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. [Hierarchical text-conditional image generation with clip latents](https://md.research.elamur.ai/free/en/papers/ramesh-2022-hierarchical-text-conditional-image-generation/). ArXiv, abs/2204.06125, 2022.
52. Mikolov, T., Yih, W.-t., and Zweig, G. [Linguistic regularities in continuous space word representations](https://md.research.elamur.ai/free/en/papers/mikolov-2013-linguistic-regularities-continuous-space-word/). In Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, pp. 746–751, 2013.
53. Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., et al. [Habitat: A platform for embodied ai research](https://md.research.elamur.ai/free/en/papers/savva-2019-habitat-platform-embodied-ai-research/). In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9339–9347, 2019.
54. Minderer, M., Gritsenko, A., Stone, A., Neumann, M., Weissenborn, D., Dosovitskiy, A., Mahendran, A., Arnab, A., Dehghani, M., Shen, Z., Wang, X., Zhai, X., Kipf, T., and Houlsby, N. Simple open-vocabulary object detection. In Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel,… [find](https://scholar.google.com/scholar?q=Minderer%2C+M%2E%2C+Gritsenko%2C+A%2E%2C+Stone%2C+A%2E%2C+Neumann%2C+M%2E%2C+Weissenborn%2C+D%2E%2C+Dosovitskiy%2C+A%2E%2C+Mahendran%2C+A%2E%2C+Arnab%2C+A%2E%2C+Dehghani%2C+M%2E%2C+Shen%2C+Z%2E%2C+Wang%2C+X%2E%2C+Zhai%2C+X%2E%2C+Kipf%2C+T%2E%2C+and+Houlsby%2C+N%2E+Simple+open%2Dvocab)
55. Shah, D., Osiński, B., Levine, S., et al. LM-Nav: Robotic navigation with large pre-trained models of language, vision, and action. In 6th Annual Conference on Robot Learning, 2022. [find](https://scholar.google.com/scholar?q=Shah%2C+D%2E%2C+Osin%CC%81ski%2C+B%2E%2C+Levine%2C+S%2E%2C+et+al%2E+LM%2DNav%3A+Robotic+navigation+with+large+pre%2Dtrained+models+of+language%2C+vision%2C+and+action%2E+In+6th+Annual+Conference+on+Robot+Learning%2C+2022%2E)
56. Misra, D., Langford, J., and Artzi, Y. [Mapping instructions and visual observations to actions with reinforcement learning](https://md.research.elamur.ai/free/en/papers/misra-2017-mapping-instructions-visual-observations-actions/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1004–1015, 2017.
57. Sharma, P., Torralba, A., and Andreas, J. [Skill induction and planning with latent language](https://md.research.elamur.ai/free/en/papers/sharma-2021-skill-induction-planning-latent-language/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1713–1726, 2022.
58. Mohan, S. and Laird, J. Learning goal-oriented hierarchical tasks from situated interactive instruction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 28, 2014. [find](https://scholar.google.com/scholar?q=Mohan%2C+S%2E+and+Laird%2C+J%2E+Learning+goal%2Doriented+hierarchical+tasks+from+situated+interactive+instruction%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+volume+28%2C+2014%2E)
59. She, L., Cheng, Y., Chai, J. Y., Jia, Y., Yang, S., and Xi, N. Teaching robots new actions through natural language instructions. In The 23rd IEEE International Symposium on Robot and Human Interactive Communication, pp. 868–873. IEEE, 2014. [find](https://scholar.google.com/scholar?q=She%2C+L%2E%2C+Cheng%2C+Y%2E%2C+Chai%2C+J%2E+Y%2E%2C+Jia%2C+Y%2E%2C+Yang%2C+S%2E%2C+and+Xi%2C+N%2E+Teaching+robots+new+actions+through+natural+language+instructions%2E+In+The+23rd+IEEE+International+Symposium+on+Robot+and+Human+Interactiv)
60. Mooney, R. J. Learning to connect language and perception. In AAAI, pp. 1598–1601. Chicago, 2008. [find](https://scholar.google.com/scholar?q=Mooney%2C+R%2E+J%2E+Learning+to+connect+language+and+perception%2E+In+AAAI%2C+pp%2E+1598%E2%80%931601%2E+Chicago%2C+2008%2E)
61. Shridhar, M., Thomason, J., Gordon, D., Bisk, Y., Han, W., Mottaghi, R., Zettlemoyer, L., and Fox, D. [ALFRED: A benchmark for interpreting grounded instructions for everyday tasks](https://md.research.elamur.ai/free/en/papers/shridhar-2019-alfred-benchmark-interpreting-grounded-instructions/). In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10740–10749, 2020.
62. Nair, S., Rajeswaran, A., Kumar, V., Finn, C., and Gupta, A. R3M: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12601, 2022. [find](https://scholar.google.com/scholar?q=Nair%2C+S%2E%2C+Rajeswaran%2C+A%2E%2C+Kumar%2C+V%2E%2C+Finn%2C+C%2E%2C+and+Gupta%2C+A%2E+R3M%3A+A+universal+visual+representation+for+robot+manipulation%2E+arXiv+preprint+arXiv%3A2203%2E12601%2C+2022%2E)
63. Narasimhan, K., Barzilay, R., and Jaakkola, T. Grounding language for transfer in deep reinforcement learning. J. Artif. Int. Res., 63(1):849–874, sep 2018. ISSN 1076-9757. [find](https://scholar.google.com/scholar?q=Narasimhan%2C+K%2E%2C+Barzilay%2C+R%2E%2C+and+Jaakkola%2C+T%2E+Grounding+language+for+transfer+in+deep+reinforcement+learning%2E+J%2E+Artif%2E+Int%2E+Res%2E%2C+63%281%29%3A849%E2%80%93874%2C+sep+2018%2E+ISSN+1076%2D9757%2E)
64. Shridhar, M., Manuelli, L., and Fox, D. [Cliport: What and where pathways for robotic manipulation](https://md.research.elamur.ai/free/en/papers/shridhar-2021-cliport-where-pathways-robotic-manipulation/). CoRL, 2021.
65. Singh, I., Blukis, V., Mousavian, A., Goyal, A., Xu, D., Tremblay, J., Fox, D., Thomason, J., and Garg, A. [Progprompt: Generating situated robot task plans using large language models](https://md.research.elamur.ai/free/en/papers/singh-2022-progprompt-generating-situated-robot-task/). arXiv preprint arXiv:2209.11302, 2022.
66. Nguyen, K. X., Misra, D., Schapire, R., Dudı́k, M., and Shafto, P. Interactive learning from activity description. In International Conference on Machine Learning, pp. 8096–8108. PMLR, 2021. [find](https://scholar.google.com/scholar?q=Nguyen%2C+K%2E+X%2E%2C+Misra%2C+D%2E%2C+Schapire%2C+R%2E%2C+Dud%C4%B1%CC%81k%2C+M%2E%2C+and+Shafto%2C+P%2E+Interactive+learning+from+activity+description%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+8096%E2%80%938108%2E+PMLR%2C+2021%2E)
67. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), 2023.
68. Sumers, T. R., Ho, M. K., Hawkins, R. D., Narasimhan, K., and Griffiths, T. L. Learning rewards from linguistic feedback. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 6002–6010, 2021. [find](https://scholar.google.com/scholar?q=Sumers%2C+T%2E+R%2E%2C+Ho%2C+M%2E+K%2E%2C+Hawkins%2C+R%2E+D%2E%2C+Narasimhan%2C+K%2E%2C+and+Griffiths%2C+T%2E+L%2E+Learning+rewards+from+linguistic+feedback%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+volume+35%2C+p)
69. Pinto, L. and Gupta, A. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 3406–3413, 2016. doi: 10.1109/ICRA.2016.7487517. [find](https://scholar.google.com/scholar?q=Pinto%2C+L%2E+and+Gupta%2C+A%2E+Supersizing+self%2Dsupervision%3A+Learning+to+grasp+from+50k+tries+and+700+robot+hours%2E+In+2016+IEEE+International+Conference+on+Robotics+and+Automation+%28ICRA%29%2C+pp%2E+3406%E2%80%933413%2C+2016)
70. Tam, A., Rabinowitz, N. C., Lampinen, A. K., Roy, N. A., Chan, S. C., Strouse, D., Wang, J. X., Banino, A., and Hill, F. Semantic exploration from language abstractions and pretrained representations. In Advances in Neural Information Processing Systems, 2022. [find](https://scholar.google.com/scholar?q=Tam%2C+A%2E%2C+Rabinowitz%2C+N%2E+C%2E%2C+Lampinen%2C+A%2E+K%2E%2C+Roy%2C+N%2E+A%2E%2C+Chan%2C+S%2E+C%2E%2C+Strouse%2C+D%2E%2C+Wang%2C+J%2E+X%2E%2C+Banino%2C+A%2E%2C+and+Hill%2C+F%2E+Semantic+exploration+from+language+abstractions+and+pretrained+representations%2E)
71. Tellex, S., Kollar, T., Dickerson, S., Walter, M., Banerjee, A., Teller, S., and Roy, N. Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 25, pp. 1507–1514, 2011. [find](https://scholar.google.com/scholar?q=Tellex%2C+S%2E%2C+Kollar%2C+T%2E%2C+Dickerson%2C+S%2E%2C+Walter%2C+M%2E%2C+Banerjee%2C+A%2E%2C+Teller%2C+S%2E%2C+and+Roy%2C+N%2E+Understanding+natural+language+commands+for+robotic+navigation+and+mobile+manipulation%2E+In+Proceedings+of+the+A)
72. Tellex, S., Gopalan, N., Kress-Gazit, H., and Matuszek, C. Robots that use language. Annual Review of Control, Robotics, and Autonomous Systems, 3(1), 2020. [find](https://scholar.google.com/scholar?q=Tellex%2C+S%2E%2C+Gopalan%2C+N%2E%2C+Kress%2DGazit%2C+H%2E%2C+and+Matuszek%2C+C%2E+Robots+that+use+language%2E+Annual+Review+of+Control%2C+Robotics%2C+and+Autonomous+Systems%2C+3%281%29%2C+2020%2E)
73. Thomason, J., Padmakumar, A., Sinapov, J., Hart, J., Stone, P., and Mooney, R. Opportunistic active learning for grounding natural language descriptions. In Proceedings of the 1st Annual Conference on Robot Learning (CoRL-17), 2017. [find](https://scholar.google.com/scholar?q=Thomason%2C+J%2E%2C+Padmakumar%2C+A%2E%2C+Sinapov%2C+J%2E%2C+Hart%2C+J%2E%2C+Stone%2C+P%2E%2C+and+Mooney%2C+R%2E+Opportunistic+active+learning+for+grounding+natural+language+descriptions%2E+In+Proceedings+of+the+1st+Annual+Conference+on)
74. Winograd, T. Understanding natural language. Cognitive psychology, 3(1):1–191, 1972. [find](https://scholar.google.com/scholar?q=Winograd%2C+T%2E+Understanding+natural+language%2E+Cognitive+psychology%2C+3%281%29%3A1%E2%80%93191%2C+1972%2E)
75. Xiao, T., Chan, H., Sermanet, P., Wahid, A., Brohan, A., Hausman, K., Levine, S., and Tompson, J. Robotic skill acquisition via instruction augmentation with vision-language models. arXiv preprint arXiv:2211.11736, 2022. — [arXiv:2211.11736](https://arxiv.org/abs/2211.11736) [find](https://scholar.google.com/scholar?q=Xiao%2C+T%2E%2C+Chan%2C+H%2E%2C+Sermanet%2C+P%2E%2C+Wahid%2C+A%2E%2C+Brohan%2C+A%2E%2C+Hausman%2C+K%2E%2C+Levine%2C+S%2E%2C+and+Tompson%2C+J%2E+Robotic+skill+acquisition+via+instruction+augmentation+with+vision%2Dlanguage+models%2E+arXiv+preprint+arX)
76. Yan, C., Carnevale, F., Georgiev, P., Santoro, A., Guy, A., Muldal, A., Hung, C.-C., Abramson, J. S., Lillicrap, T. P., and Wayne, G. Intra-agent speech permits zero-shot task acquisition. In Advances in Neural Information Processing Systems, 2022. [find](https://scholar.google.com/scholar?q=Yan%2C+C%2E%2C+Carnevale%2C+F%2E%2C+Georgiev%2C+P%2E%2C+Santoro%2C+A%2E%2C+Guy%2C+A%2E%2C+Muldal%2C+A%2E%2C+Hung%2C+C%2E%2DC%2E%2C+Abramson%2C+J%2E+S%2E%2C+Lillicrap%2C+T%2E+P%2E%2C+and+Wayne%2C+G%2E+Intra%2Dagent+speech+permits+zero%2Dshot+task+acquisition%2E+In+Advances)
77. Yu, H., Zhang, H., and Xu, W. Interactive grounded language acquisition and generalization in a 2D world. In ICLR, 2018. [find](https://scholar.google.com/scholar?q=Yu%2C+H%2E%2C+Zhang%2C+H%2E%2C+and+Xu%2C+W%2E+Interactive+grounded+language+acquisition+and+generalization+in+a+2D+world%2E+In+ICLR%2C+2018%2E)
78. Zeng, A., Wong, A., Welker, S., Choromanski, K., Tombari, F., Purohit, A., Ryoo, M., Sindhwani, V., Lee, J., Vanhoucke, V., et al. [Socratic models: Composing zero-shot multimodal reasoning with language](https://md.research.elamur.ai/free/en/papers/zeng-2022-socratic-models-composing-zero-shot/). arXiv preprint arXiv:2204.00598, 2022.
79. Zhong, V., Rocktäschel, T., and Grefenstette, E. RTFM: Generalising to new environment dynamics via reading. In ICLR, 2020. [find](https://scholar.google.com/scholar?q=Zhong%2C+V%2E%2C+Rockta%CC%88schel%2C+T%2E%2C+and+Grefenstette%2C+E%2E+RTFM%3A+Generalising+to+new+environment+dynamics+via+reading%2E+In+ICLR%2C+2020%2E)
80. Zhong, V., Hanjie, A. W., Wang, S., Narasimhan, K., and Zettlemoyer, L. SILG: The multi-domain symbolic interactive language grounding benchmark. Advances in Neural Information Processing Systems, 34:21505–21519, 2021. [find](https://scholar.google.com/scholar?q=Zhong%2C+V%2E%2C+Hanjie%2C+A%2E+W%2E%2C+Wang%2C+S%2E%2C+Narasimhan%2C+K%2E%2C+and+Zettlemoyer%2C+L%2E+SILG%3A+The+multi%2Ddomain+symbolic+interactive+language+grounding+benchmark%2E+Advances+in+Neural+Information+Processing+Systems%2C+34%3A)
81. Zhong, V., Mu, J., Zettlemoyer, L., Grefenstette, E., and Rocktäschel, T. Improving policy learning via language dynamics distillation. In Advances in Neural Information Processing Systems, 2022. [find](https://scholar.google.com/scholar?q=Zhong%2C+V%2E%2C+Mu%2C+J%2E%2C+Zettlemoyer%2C+L%2E%2C+Grefenstette%2C+E%2E%2C+and+Rockta%CC%88schel%2C+T%2E+Improving+policy+learning+via+language+dynamics+distillation%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2022%2E)

*Updated 3 November 2026*