---
title: "CLIPort: What and Where Pathways for Robotic Manipulation"
description: "Robots that learn manipulation end-to-end can act with high spatial accuracy, but they rarely… 69 references, cited by 68 papers here."
authors: ["Mohit Shridhar", "Lucas Manuelli", "Dieter Fox"]
year: 2021
venue: "arXiv:2109.12098"
source: "https://arxiv.org/abs/2109.12098"
doi: "10.48550/arxiv.2109.12098"
arxiv: "2109.12098"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shridhar-2021-cliport-where-pathways-robotic-manipulation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shridhar-2021-cliport-where-pathways-robotic-manipulation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / CLIPort: What and Where Pathways for Robotic Manipulation

# CLIPort: What and Where Pathways for Robotic Manipulation

**Authors:** [Mohit Shridhar](https://md.research.elamur.ai/free/en/authors/mohit-shridhar/), Lucas Manuelli and [Dieter Fox](https://md.research.elamur.ai/free/en/authors/dieter-fox/) · **Year:** 2021 · **Venue:** arXiv:2109.12098 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2109.12098) · [PDF](https://arxiv.org/pdf/2109.12098) · [DOI 10.48550/arxiv.2109.12098](https://doi.org/10.48550/arxiv.2109.12098) · [arXiv 2109.12098](https://arxiv.org/abs/2109.12098) · [OpenAlex](https://openalex.org/W3203511201)

**Citations:** 91 citations (OpenAlex, 2026-10-14) · [68](https://md.research.elamur.ai/free/en/papers/shridhar-2021-cliport-where-pathways-robotic-manipulation/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Robots that learn manipulation end-to-end can act with high spatial accuracy, but they rarely understand task semantics. Every new goal usually needs fresh demonstrations or a goal image. Large vision-language models trained on web data capture semantics well but have no sense of how to physically handle objects. CLIPort brings the two together in an imitation-learning agent guided by natural-language instructions.

The architecture has two streams:
- **Semantic ("what") stream:** frozen CLIP encoders for the image and the instruction.
- **Spatial ("where") stream:** a Transporter-style network trained on RGB-D input.

The streams are fused to predict pixel-level pick-and-place affordances. The approach needs no object poses, segmentation masks or symbolic state.

The authors extend the Ravens simulator with 10 language-conditioned tabletop tasks, each with thousands of variants. A single multi-task policy matches or beats per-task policies, and it can carry attributes such as new colours from one task to another. A real Franka Panda robot learned 9 tasks from very little data.

The work suggests that web-scale semantic priors can make precise robot policies more general and more data-efficient.

## Key points
- **Architecture:** a frozen CLIP ResNet50 encodes RGB images, and the CLIP sentence encoder conditions the decoder. A Transporter ResNet trained from scratch handles RGB-D input and is fused laterally with the semantic stream.
- **Benchmark:** 10 new language-grounding tasks in Ravens with a simulated suction gripper, including packing unseen objects, block stacking and rope alignment. The benchmark, code and pre-trained models are released at cliport.github.io.
- **Cross-task transfer:** training on the colour splits of other tasks lifted the multi-task model on put-blocks-in-bowls-unseen-colors (1000 demos) from 45.8 to 75.7 success.
- **Real-robot results:** one multi-task model on a Franka Panda covered 9 tasks using only 179 image-action pairs. Simple block tasks succeeded about 70% of the time, and the authors estimate that 50–100 demonstrations per task are needed for robust performance.
- **Limitations:** the method is restricted to a two-step pick-and-place primitive. It cannot do dexterous 6-DOF or multi-fingered control, and it cannot detect when a task is finished. On the real robot it sometimes exploited biases in the training data.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 402 works · 212,763 citations
- [Nvidia](https://md.research.elamur.ai/free/en/companies/nvidia/) — Company, Santa Clara, United States · 37 works · 12,656 citations
- [Nvidia (United Kingdom)](https://md.research.elamur.ai/free/en/companies/nvidia-united-kingdom/) — Company, Reading, United Kingdom · 27 works · 11,141 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [⚙️ Engineering](https://md.research.elamur.ai/free/en/shelves/engineering/) · 188 works · [⚙️ Control and Systems Engineering](https://md.research.elamur.ai/free/en/shelves/engineering/control-and-systems-engineering/) · 62 works

## Cite

Cite as: Mohit Shridhar, Lucas Manuelli and Dieter Fox. 2021. CLIPort: What and Where Pathways for Robotic Manipulation. arXiv:2109.12098.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{shridhar2021cliport,
  title = {{CLIPort: What and Where Pathways for Robotic Manipulation}},
  author = {Mohit Shridhar, Lucas Manuelli and Dieter Fox},
  year = {2021},
  doi = {10.48550/arxiv.2109.12098},
  eprint = {2109.12098},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2109.12098}
}
```

- [← previous MultiEURLEX - A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer](https://md.research.elamur.ai/free/en/papers/chalkidis-2021-multieurlex-multi-lingual-multi-label/)
- [next → Neural Bellman-Ford Networks: A General Graph Neural Network Framework for Link Prediction](https://md.research.elamur.ai/free/en/papers/zhu-2021-neural-bellman-ford-networks-general/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/)

## References

69 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. [Learning Transferable Visual Models From Natural Language Supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). arXiv:2103.00020 [cs], Feb. 2021.
2. A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, and J. Lee. Transporter networks: Rearranging the visual world for robotic manipulation. Conference on Robot Learning (CoRL), 2020. [find](https://scholar.google.com/scholar?q=A%2E+Zeng%2C+P%2E+Florence%2C+J%2E+Tompson%2C+S%2E+Welker%2C+J%2E+Chien%2C+M%2E+Attarian%2C+T%2E+Armstrong%2C+I%2E+Krasin%2C+D%2E+Duong%2C+V%2E+Sindhwani%2C+and+J%2E+Lee%2E+Transporter+networks%3A+Rearranging+the+visual+world+for+robotic+manipula)
3. I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, et al. Solving rubik’s cube with a robot hand. arXiv preprint arXiv:1910.07113, 2019. [find](https://scholar.google.com/scholar?q=I%2E+Akkaya%2C+M%2E+Andrychowicz%2C+M%2E+Chociej%2C+M%2E+Litwin%2C+B%2E+McGrew%2C+A%2E+Petron%2C+A%2E+Paino%2C+M%2E+Plappert%2C+G%2E+Powell%2C+R%2E+Ribas%2C+et+al%2E+Solving+rubik%E2%80%99s+cube+with+a+robot+hand%2E+arXiv+preprint+arXiv%3A1910%2E07113%2C+201)
4. D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. Conference on Robot Learning (CoRL), 2018. [find](https://scholar.google.com/scholar?q=D%2E+Kalashnikov%2C+A%2E+Irpan%2C+P%2E+Pastor%2C+J%2E+Ibarz%2C+A%2E+Herzog%2C+E%2E+Jang%2C+D%2E+Quillen%2C+E%2E+Holly%2C+M%2E+Kalakrishnan%2C+V%2E+Vanhoucke%2C+et+al%2E+Qt%2Dopt%3A+Scalable+deep+reinforcement+learning+for+vision%2Dbased+robotic+man)
5. D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv preprint arXiv:2104.08212, 2021. — [arXiv:2104.08212](https://arxiv.org/abs/2104.08212) [find](https://scholar.google.com/scholar?q=D%2E+Kalashnikov%2C+J%2E+Varley%2C+Y%2E+Chebotar%2C+B%2E+Swanson%2C+R%2E+Jonschkowski%2C+C%2E+Finn%2C+S%2E+Levine%2C+and+K%2E+Hausman%2E+Mt%2Dopt%3A+Continuous+multi%2Dtask+robotic+reinforcement+learning+at+scale%2E+arXiv+preprint+arXiv%3A210)
6. D. Seita, P. Florence, J. Tompson, E. Coumans, V. Sindhwani, K. Goldberg, and A. Zeng. Learning to rearrange deformable cables, fabrics, and bags with goal-conditioned transporter networks. In IEEE International Conference on Robotics and Automation (ICRA), 2021. [find](https://scholar.google.com/scholar?q=D%2E+Seita%2C+P%2E+Florence%2C+J%2E+Tompson%2C+E%2E+Coumans%2C+V%2E+Sindhwani%2C+K%2E+Goldberg%2C+and+A%2E+Zeng%2E+Learning+to+rearrange+deformable+cables%2C+fabrics%2C+and+bags+with+goal%2Dconditioned+transporter+networks%2E+In+IEEE+In)
7. M. Shridhar and D. Hsu. Interactive visual grounding of referring expressions for human-robot interaction. In Proceedings of Robotics: Science and Systems (RSS), 2018. [find](https://scholar.google.com/scholar?q=M%2E+Shridhar+and+D%2E+Hsu%2E+Interactive+visual+grounding+of+referring+expressions+for+human%2Drobot+interaction%2E+In+Proceedings+of+Robotics%3A+Science+and+Systems+%28RSS%29%2C+2018%2E)
8. C. Matuszek, L. Bo, L. Zettlemoyer, and D. Fox. Learning from unscripted deictic gesture and language for human-robot interactions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 28, 2014. [find](https://scholar.google.com/scholar?q=C%2E+Matuszek%2C+L%2E+Bo%2C+L%2E+Zettlemoyer%2C+and+D%2E+Fox%2E+Learning+from+unscripted+deictic+gesture+and+language+for+human%2Drobot+interactions%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+vo)
9. M. Bollini, S. Tellex, T. Thompson, N. Roy, and D. Rus. Interpreting and executing recipes with a cooking robot. In Experimental Robotics, pages 481–495. Springer, 2013. [find](https://scholar.google.com/scholar?q=M%2E+Bollini%2C+S%2E+Tellex%2C+T%2E+Thompson%2C+N%2E+Roy%2C+and+D%2E+Rus%2E+Interpreting+and+executing+recipes+with+a+cooking+robot%2E+In+Experimental+Robotics%2C+pages+481%E2%80%93495%2E+Springer%2C+2013%2E)
10. D. K. Misra, J. Sung, K. Lee, and A. Saxena. Tell me dave: Context-sensitive grounding of natural language to manipulation instructions. The International Journal of Robotics Research (IJRR), 35(1-3):281–300, 2016. [find](https://scholar.google.com/scholar?q=D%2E+K%2E+Misra%2C+J%2E+Sung%2C+K%2E+Lee%2C+and+A%2E+Saxena%2E+Tell+me+dave%3A+Context%2Dsensitive+grounding+of+natural+language+to+manipulation+instructions%2E+The+International+Journal+of+Robotics+Research+%28IJRR%29%2C+35%281%2D3%29%3A)
11. T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. [A simple framework for contrastive learning of visual representations](https://md.research.elamur.ai/free/en/papers/chen-2020-simple-framework-contrastive-learning-visual/). In International conference on machine learning, pages 1597–1607. PMLR, 2020.
12. K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. [Momentum contrast for unsupervised visual representation learning](https://md.research.elamur.ai/free/en/papers/he-2019-momentum-contrast-unsupervised-visual-representation/). In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9729–9738, 2020.
13. J. Lu, D. Batra, D. Parikh, and S. Lee. [Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks](https://md.research.elamur.ai/free/en/papers/lu-2019-vilbert-pretraining-task-agnostic-visiolinguistic/). In Advances in Neural Information Processing Systems (NeuRIPS), 2019.
14. Y. C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu. [Uniter: Universal image-text representation learning](https://md.research.elamur.ai/free/en/papers/chen-2019-uniter-universal-image-text-representation/). In European Conference on Computer Vision, pages 104–120. Springer, 2020.
15. H. Tan and M. Bansal. [Lxmert: Learning cross-modality encoder representations from transformers](https://md.research.elamur.ai/free/en/papers/tan-2019-lxmert-learning-cross-modality-encoder/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019.
16. D. H. Hubel and T. N. Wiesel. Receptive fields and functional architecture in two nonstriate visual areas (18 and 19) of the cat. Journal of neurophysiology, 28(2):229–289, 1965. [find](https://scholar.google.com/scholar?q=D%2E+H%2E+Hubel+and+T%2E+N%2E+Wiesel%2E+Receptive+fields+and+functional+architecture+in+two+nonstriate+visual+areas+%2818+and+19%29+of+the+cat%2E+Journal+of+neurophysiology%2C+28%282%29%3A229%E2%80%93289%2C+1965%2E)
17. M. Livingstone and D. Hubel. Segregation of form, color, movement, and depth: anatomy, physiology, and perception. Science, 240(4853):740–749, 1988. [find](https://scholar.google.com/scholar?q=M%2E+Livingstone+and+D%2E+Hubel%2E+Segregation+of+form%2C+color%2C+movement%2C+and+depth%3A+anatomy%2C+physiology%2C+and+perception%2E+Science%2C+240%284853%29%3A740%E2%80%93749%2C+1988%2E)
18. A. Derrington and P. Lennie. Spatial and temporal contrast sensitivities of neurones in lateral geniculate nucleus of macaque. The Journal of physiology, 357(1):219–240, 1984. [find](https://scholar.google.com/scholar?q=A%2E+Derrington+and+P%2E+Lennie%2E+Spatial+and+temporal+contrast+sensitivities+of+neurones+in+lateral+geniculate+nucleus+of+macaque%2E+The+Journal+of+physiology%2C+357%281%29%3A219%E2%80%93240%2C+1984%2E)
19. J. J. Gibson. The ecological approach to visual perception: classic edition. Psychology Press, 2014. [find](https://scholar.google.com/scholar?q=J%2E+J%2E+Gibson%2E+The+ecological+approach+to+visual+perception%3A+classic+edition%2E+Psychology+Press%2C+2014%2E)
20. A. Kamath, M. Singh, Y. LeCun, I. Misra, G. Synnaeve, and N. Carion. Mdetr–modulated detection for end-to-end multi-modal understanding. arXiv preprint arXiv:2104.12763, 2021. [find](https://scholar.google.com/scholar?q=A%2E+Kamath%2C+M%2E+Singh%2C+Y%2E+LeCun%2C+I%2E+Misra%2C+G%2E+Synnaeve%2C+and+N%2E+Carion%2E+Mdetr%E2%80%93modulated+detection+for+end%2Dto%2Dend+multi%2Dmodal+understanding%2E+arXiv+preprint+arXiv%3A2104%2E12763%2C+2021%2E)
21. K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017. [find](https://scholar.google.com/scholar?q=K%2E+He%2C+G%2E+Gkioxari%2C+P%2E+Dolla%CC%81r%2C+and+R%2E+Girshick%2E+Mask+r%2Dcnn%2E+In+The+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition+%28CVPR%29%2C+2017%2E)
22. Y. Xiang, T. Schmidt, V. Narayanan, and D. Fox. Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes. In Proceedings of Robotics: Science and Systems (RSS), 2018. [find](https://scholar.google.com/scholar?q=Y%2E+Xiang%2C+T%2E+Schmidt%2C+V%2E+Narayanan%2C+and+D%2E+Fox%2E+Posecnn%3A+A+convolutional+neural+network+for+6d+object+pose+estimation+in+cluttered+scenes%2E+In+Proceedings+of+Robotics%3A+Science+and+Systems+%28RSS%29%2C+2018%2E)
23. M. Zhu, K. G. Derpanis, Y. Yang, S. Brahmbhatt, M. Zhang, C. Phillips, M. Lecce, and K. Daniilidis. Single image 3d object detection and pose estimation for grasping. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pages 3936–3943. IEEE, 2014. [find](https://scholar.google.com/scholar?q=M%2E+Zhu%2C+K%2E+G%2E+Derpanis%2C+Y%2E+Yang%2C+S%2E+Brahmbhatt%2C+M%2E+Zhang%2C+C%2E+Phillips%2C+M%2E+Lecce%2C+and+K%2E+Daniilidis%2E+Single+image+3d+object+detection+and+pose+estimation+for+grasping%2E+In+2014+IEEE+International+Confer)
24. A. Zeng, K.-T. Yu, S. Song, D. Suo, E. Walker, A. Rodriguez, and J. Xiao. Multi-view self-supervised deep learning for 6d pose estimation in the amazon picking challenge. In 2017 IEEE international conference on robotics and automation (ICRA), pages 1386–1383. IEEE, 2017. [find](https://scholar.google.com/scholar?q=A%2E+Zeng%2C+K%2E%2DT%2E+Yu%2C+S%2E+Song%2C+D%2E+Suo%2C+E%2E+Walker%2C+A%2E+Rodriguez%2C+and+J%2E+Xiao%2E+Multi%2Dview+self%2Dsupervised+deep+learning+for+6d+pose+estimation+in+the+amazon+picking+challenge%2E+In+2017+IEEE+international+co)
25. X. Deng, Y. Xiang, A. Mousavian, C. Eppner, T. Bretl, and D. Fox. Self-supervised 6d object pose estimation for robot manipulation. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 3665–3671. IEEE, 2020. [find](https://scholar.google.com/scholar?q=X%2E+Deng%2C+Y%2E+Xiang%2C+A%2E+Mousavian%2C+C%2E+Eppner%2C+T%2E+Bretl%2C+and+D%2E+Fox%2E+Self%2Dsupervised+6d+object+pose+estimation+for+robot+manipulation%2E+In+2020+IEEE+International+Conference+on+Robotics+and+Automation+%28IC)
26. C. Xie, Y. Xiang, A. Mousavian, and D. Fox. The best of both modes: Separately leveraging rgb and depth for unseen object instance segmentation. In Conference on Robot Learning (CoRL), pages 1369–1378. PMLR, 2020. [find](https://scholar.google.com/scholar?q=C%2E+Xie%2C+Y%2E+Xiang%2C+A%2E+Mousavian%2C+and+D%2E+Fox%2E+The+best+of+both+modes%3A+Separately+leveraging+rgb+and+depth+for+unseen+object+instance+segmentation%2E+In+Conference+on+Robot+Learning+%28CoRL%29%2C+pages+1369%E2%80%931378)
27. P. R. Florence, L. Manuelli, and R. Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. In Conference on Robot Learning (CoRL), 2018. [find](https://scholar.google.com/scholar?q=P%2E+R%2E+Florence%2C+L%2E+Manuelli%2C+and+R%2E+Tedrake%2E+Dense+object+nets%3A+Learning+dense+visual+object+descriptors+by+and+for+robotic+manipulation%2E+In+Conference+on+Robot+Learning+%28CoRL%29%2C+2018%2E)
28. P. Florence, L. Manuelli, and R. Tedrake. Self-supervised correspondence in visuomotor policy learning. IEEE Robotics and Automation Letters, 5(2):492–499, 2019. [find](https://scholar.google.com/scholar?q=P%2E+Florence%2C+L%2E+Manuelli%2C+and+R%2E+Tedrake%2E+Self%2Dsupervised+correspondence+in+visuomotor+policy+learning%2E+IEEE+Robotics+and+Automation+Letters%2C+5%282%29%3A492%E2%80%93499%2C+2019%2E)
29. P. Sundaresan, J. Grannen, B. Thananjeyan, A. Balakrishna, M. Laskey, K. Stone, J. E. Gonzalez, and K. Goldberg. Learning rope manipulation policies using dense object descriptors trained on synthetic depth data. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9411–941… [find](https://scholar.google.com/scholar?q=P%2E+Sundaresan%2C+J%2E+Grannen%2C+B%2E+Thananjeyan%2C+A%2E+Balakrishna%2C+M%2E+Laskey%2C+K%2E+Stone%2C+J%2E+E%2E+Gonzalez%2C+and+K%2E+Goldberg%2E+Learning+rope+manipulation+policies+using+dense+object+descriptors+trained+on+synthetic)
30. L. Manuelli, W. Gao, P. Florence, and R. Tedrake. kpam: Keypoint affordances for category-level robotic manipulation. In International Symposium on Robotics Research (ISRR), 2019. [find](https://scholar.google.com/scholar?q=L%2E+Manuelli%2C+W%2E+Gao%2C+P%2E+Florence%2C+and+R%2E+Tedrake%2E+kpam%3A+Keypoint+affordances+for+category%2Dlevel+robotic+manipulation%2E+In+International+Symposium+on+Robotics+Research+%28ISRR%29%2C+2019%2E)
31. T. D. Kulkarni, A. Gupta, C. Ionescu, S. Borgeaud, M. Reynolds, A. Zisserman, and V. Mnih. Unsupervised learning of object keypoints for perception and control. Advances in neural information processing systems (NeuRIPS), 32:10724–10734, 2019. [find](https://scholar.google.com/scholar?q=T%2E+D%2E+Kulkarni%2C+A%2E+Gupta%2C+C%2E+Ionescu%2C+S%2E+Borgeaud%2C+M%2E+Reynolds%2C+A%2E+Zisserman%2C+and+V%2E+Mnih%2E+Unsupervised+learning+of+object+keypoints+for+perception+and+control%2E+Advances+in+neural+information+processi)
32. X. Liu, R. Jonschkowski, A. Angelova, and K. Konolige. Keypose: Multi-view 3d labeling and keypoint estimation for transparent objects. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11602–11610, 2020. [find](https://scholar.google.com/scholar?q=X%2E+Liu%2C+R%2E+Jonschkowski%2C+A%2E+Angelova%2C+and+K%2E+Konolige%2E+Keypose%3A+Multi%2Dview+3d+labeling+and+keypoint+estimation+for+transparent+objects%2E+In+The+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recogn)
33. K. Zakka, A. Zeng, J. Lee, and S. Song. Form2fit: Learning shape priors for generalizable assembly from disassembly. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9404–9410. IEEE, 2020. [find](https://scholar.google.com/scholar?q=K%2E+Zakka%2C+A%2E+Zeng%2C+J%2E+Lee%2C+and+S%2E+Song%2E+Form2fit%3A+Learning+shape+priors+for+generalizable+assembly+from+disassembly%2E+In+2020+IEEE+International+Conference+on+Robotics+and+Automation+%28ICRA%29%2C+pages+9404)
34. S. Song, A. Zeng, J. Lee, and T. Funkhouser. Grasping in the wild: Learning 6dof closed-loop grasping from low-cost demonstrations. IEEE Robotics and Automation Letters, 5(3): 4978–4985, 2020. [find](https://scholar.google.com/scholar?q=S%2E+Song%2C+A%2E+Zeng%2C+J%2E+Lee%2C+and+T%2E+Funkhouser%2E+Grasping+in+the+wild%3A+Learning+6dof+closed%2Dloop+grasping+from+low%2Dcost+demonstrations%2E+IEEE+Robotics+and+Automation+Letters%2C+5%283%29%3A+4978%E2%80%934985%2C+2020%2E)
35. Y. Wu, W. Yan, T. Kurutach, L. Pinto, and P. Abbeel. Learning to Manipulate Deformable Objects without Demonstrations. In Proceedings of Robotics: Science and Systems (RSS), 2020. [find](https://scholar.google.com/scholar?q=Y%2E+Wu%2C+W%2E+Yan%2C+T%2E+Kurutach%2C+L%2E+Pinto%2C+and+P%2E+Abbeel%2E+Learning+to+Manipulate+Deformable+Objects+without+Demonstrations%2E+In+Proceedings+of+Robotics%3A+Science+and+Systems+%28RSS%29%2C+2020%2E)
36. L. Yen-Chen, A. Zeng, S. Song, P. Isola, and T.-Y. Lin. Learning to see before learning to act: Visual pre-training for manipulation. In IEEE International Conference on Robotics and Automation (ICRA), 2020. [find](https://scholar.google.com/scholar?q=L%2E+Yen%2DChen%2C+A%2E+Zeng%2C+S%2E+Song%2C+P%2E+Isola%2C+and+T%2E%2DY%2E+Lin%2E+Learning+to+see+before+learning+to+act%3A+Visual+pre%2Dtraining+for+manipulation%2E+In+IEEE+International+Conference+on+Robotics+and+Automation+%28ICRA%29)
37. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems (NeuRIPS), 2017.
38. J. Devlin, M. W. Chang, K. Lee, and K. Toutanova. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2018.
39. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. [An image is worth 16x16 words: Transformers for image recognition at scale](https://md.research.elamur.ai/free/en/papers/dosovitskiy-2020-image-worth-16x16-words-transformers/). In International Conference on Learning Representations, 2020.
40. F. Yu, J. Tang, W. Yin, Y. Sun, H. Tian, H. Wu, and H. Wang. [Ernie-vil: Knowledge enhanced vision-language representations through scene graph](https://md.research.elamur.ai/free/en/papers/yu-2020-ernie-vil-knowledge-enhanced/). arXiv preprint arXiv:2006.16934, 2020.
41. Y. Bisk, D. Yuret, and D. Marcu. Natural language communication with robots. In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 751–761, 2016. [find](https://scholar.google.com/scholar?q=Y%2E+Bisk%2C+D%2E+Yuret%2C+and+D%2E+Marcu%2E+Natural+language+communication+with+robots%2E+In+Proceedings+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics+%28NAACL%29%2C+pages+751%E2%80%93761%2C+2016%2E)
42. J. Thomason, S. Zhang, R. J. Mooney, and P. Stone. Learning to interpret natural language commands through human-robot dialog. In Twenty-Fourth International Joint Conference on Artificial Intelligence (IJCAI), 2015. [find](https://scholar.google.com/scholar?q=J%2E+Thomason%2C+S%2E+Zhang%2C+R%2E+J%2E+Mooney%2C+and+P%2E+Stone%2E+Learning+to+interpret+natural+language+commands+through+human%2Drobot+dialog%2E+In+Twenty%2DFourth+International+Joint+Conference+on+Artificial+Intelligenc)
43. J. Hatori, Y. Kikuchi, S. Kobayashi, K. Takahashi, Y. Tsuboi, Y. Unno, W. Ko, and J. Tan. Interactively picking real-world objects with unconstrained spoken language instructions. In Proceedings of International Conference on Robotics and Automation (ICRA), 2018. [find](https://scholar.google.com/scholar?q=J%2E+Hatori%2C+Y%2E+Kikuchi%2C+S%2E+Kobayashi%2C+K%2E+Takahashi%2C+Y%2E+Tsuboi%2C+Y%2E+Unno%2C+W%2E+Ko%2C+and+J%2E+Tan%2E+Interactively+picking+real%2Dworld+objects+with+unconstrained+spoken+language+instructions%2E+In+Proceedings+of+In)
44. Y. Chen, R. Xu, Y. Lin, and P. A. Vela. A Joint Network for Grasp Detection Conditioned on Natural Language Commands. arXiv:2104.00492 [cs], Apr. 2021. — [arXiv:2104.00492](https://arxiv.org/abs/2104.00492) [find](https://scholar.google.com/scholar?q=Y%2E+Chen%2C+R%2E+Xu%2C+Y%2E+Lin%2C+and+P%2E+A%2E+Vela%2E+A+Joint+Network+for+Grasp+Detection+Conditioned+on+Natural+Language+Commands%2E+arXiv%3A2104%2E00492+%5Bcs%5D%2C+Apr%2E+2021%2E+%E2%80%94+%5BarXiv%3A2104%2E00492%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2104%2E)
45. V. Blukis, R. A. Knepper, and Y. Artzi. Few-shot object grounding for mapping natural language instructions to robot control. In Conference on Robot Learning (CoRL), 2020. [find](https://scholar.google.com/scholar?q=V%2E+Blukis%2C+R%2E+A%2E+Knepper%2C+and+Y%2E+Artzi%2E+Few%2Dshot+object+grounding+for+mapping+natural+language+instructions+to+robot+control%2E+In+Conference+on+Robot+Learning+%28CoRL%29%2C+2020%2E)
46. C. Paxton, Y. Bisk, J. Thomason, A. Byravan, and D. Fox. Prospection: Interpretable plans from language by predicting the future. In International Conference on Robotics and Automation (ICRA), pages 6942–6948. IEEE, 2019. [find](https://scholar.google.com/scholar?q=C%2E+Paxton%2C+Y%2E+Bisk%2C+J%2E+Thomason%2C+A%2E+Byravan%2C+and+D%2E+Fox%2E+Prospection%3A+Interpretable+plans+from+language+by+predicting+the+future%2E+In+International+Conference+on+Robotics+and+Automation+%28ICRA%29%2C+pages+6)
47. S. Tellex, T. Kollar, S. Dickerson, M. Walter, A. Banerjee, S. Teller, and N. Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2011. [find](https://scholar.google.com/scholar?q=S%2E+Tellex%2C+T%2E+Kollar%2C+S%2E+Dickerson%2C+M%2E+Walter%2C+A%2E+Banerjee%2C+S%2E+Teller%2C+and+N%2E+Roy%2E+Understanding+natural+language+commands+for+robotic+navigation+and+mobile+manipulation%2E+In+Proceedings+of+the+AAAI+Co)
48. C. Lynch and P. Sermanet. Grounding language in play. arXiv preprint arXiv:2005.07648, 2020. [find](https://scholar.google.com/scholar?q=C%2E+Lynch+and+P%2E+Sermanet%2E+Grounding+language+in+play%2E+arXiv+preprint+arXiv%3A2005%2E07648%2C+2020%2E)
49. K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. arXiv preprint arXiv:1406.2199, 2014. [find](https://scholar.google.com/scholar?q=K%2E+Simonyan+and+A%2E+Zisserman%2E+Two%2Dstream+convolutional+networks+for+action+recognition+in+videos%2E+arXiv+preprint+arXiv%3A1406%2E2199%2C+2014%2E)
50. C. Feichtenhofer, A. Pinz, and A. Zisserman. Convolutional two-stream network fusion for video action recognition. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1933–1941, 2016. [find](https://scholar.google.com/scholar?q=C%2E+Feichtenhofer%2C+A%2E+Pinz%2C+and+A%2E+Zisserman%2E+Convolutional+two%2Dstream+network+fusion+for+video+action+recognition%2E+In+The+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition+%28CVPR%29%2C+pages+1)
51. C. Feichtenhofer, H. Fan, J. Malik, and K. He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR), pages 6202–6211, 2019. [find](https://scholar.google.com/scholar?q=C%2E+Feichtenhofer%2C+H%2E+Fan%2C+J%2E+Malik%2C+and+K%2E+He%2E+Slowfast+networks+for+video+recognition%2E+In+Proceedings+of+the+IEEE%2FCVF+International+Conference+on+Computer+Vision+%28CVPR%29%2C+pages+6202%E2%80%936211%2C+2019%2E)
52. E. Kazakos, A. Nagrani, A. Zisserman, and D. Damen. Slow-fast auditory streams for audio recognition. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 855–859. IEEE, 2021. [find](https://scholar.google.com/scholar?q=E%2E+Kazakos%2C+A%2E+Nagrani%2C+A%2E+Zisserman%2C+and+D%2E+Damen%2E+Slow%2Dfast+auditory+streams+for+audio+recognition%2E+In+ICASSP+2021%2D2021+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+%28ICAS)
53. F. Xiao, Y. J. Lee, K. Grauman, J. Malik, and C. Feichtenhofer. Audiovisual slowfast networks for video recognition. arXiv preprint arXiv:2001.08740, 2020. [find](https://scholar.google.com/scholar?q=F%2E+Xiao%2C+Y%2E+J%2E+Lee%2C+K%2E+Grauman%2C+J%2E+Malik%2C+and+C%2E+Feichtenhofer%2E+Audiovisual+slowfast+networks+for+video+recognition%2E+arXiv+preprint+arXiv%3A2001%2E08740%2C+2020%2E)
54. A. Zeng, S. Song, K.-T. Yu, E. Donlon, F. R. Hogan, M. Bauza, D. Ma, O. Taylor, M. Liu, E. Romo, et al. Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching. In 2018 IEEE international conference on robotics and automation (ICRA), pages 37… [find](https://scholar.google.com/scholar?q=A%2E+Zeng%2C+S%2E+Song%2C+K%2E%2DT%2E+Yu%2C+E%2E+Donlon%2C+F%2E+R%2E+Hogan%2C+M%2E+Bauza%2C+D%2E+Ma%2C+O%2E+Taylor%2C+M%2E+Liu%2C+E%2E+Romo%2C+et+al%2E+Robotic+pick%2Dand%2Dplace+of+novel+objects+in+clutter+with+multi%2Daffordance+grasping+and+cross%2Ddoma)
55. E. Jang, S. Vijayanarasimhan, P. Pastor, J. Ibarz, and S. Levine. End-to-end learning of semantic grasping. In Conference on Robot Learning (CoRL), Proceedings of Machine Learning Research. PMLR, 2017. [find](https://scholar.google.com/scholar?q=E%2E+Jang%2C+S%2E+Vijayanarasimhan%2C+P%2E+Pastor%2C+J%2E+Ibarz%2C+and+S%2E+Levine%2E+End%2Dto%2Dend+learning+of+semantic+grasping%2E+In+Conference+on+Robot+Learning+%28CoRL%29%2C+Proceedings+of+Machine+Learning+Research%2E+PMLR%2C+2017)
56. R. Kondor and S. Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In International Conference on Machine Learning (ICML), 2018. [find](https://scholar.google.com/scholar?q=R%2E+Kondor+and+S%2E+Trivedi%2E+On+the+generalization+of+equivariance+and+convolution+in+neural+networks+to+the+action+of+compact+groups%2E+In+International+Conference+on+Machine+Learning+%28ICML%29%2C+2018%2E)
57. T. Cohen and M. Welling. Group equivariant convolutional networks. In International conference on machine learning (ICML), 2016. [find](https://scholar.google.com/scholar?q=T%2E+Cohen+and+M%2E+Welling%2E+Group+equivariant+convolutional+networks%2E+In+International+conference+on+machine+learning+%28ICML%29%2C+2016%2E)
58. D. Misra, A. Bennett, V. Blukis, E. Niklasson, M. Shatkhin, and Y. Artzi. Mapping instructions to actions in 3d environments with visual goal prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018. [find](https://scholar.google.com/scholar?q=D%2E+Misra%2C+A%2E+Bennett%2C+V%2E+Blukis%2C+E%2E+Niklasson%2C+M%2E+Shatkhin%2C+and+Y%2E+Artzi%2E+Mapping+instructions+to+actions+in+3d+environments+with+visual+goal+prediction%2E+In+Proceedings+of+the+2019+Conference+on+Empir)
59. G. Goh, N. C. †, C. V. †, S. Carter, M. Petrov, L. Schubert, A. Radford, and C. Olah. Multimodal neurons in artificial neural networks. Distill, 2021. doi:10.23915/distill.00030. [https://distill.pub/2021/multimodal-neurons](https://distill.pub/2021/multimodal-neurons). [↗](https://distill.pub/2021/multimodal-neurons)
60. E. Coumans and Y. Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. 2016. [find](https://scholar.google.com/scholar?q=E%2E+Coumans+and+Y%2E+Bai%2E+Pybullet%2C+a+python+module+for+physics+simulation+for+games%2C+robotics+and+machine+learning%2E+2016%2E)
61. Google scanned objects dataset, 2020. URL [https://app.ignitionrobotics.org/GoogleResearch/fuel/collections/Google%20Scanned%20Objects](https://app.ignitionrobotics.org/GoogleResearch/fuel/collections/Google%20Scanned%20Objects). [↗](https://app.ignitionrobotics.org/GoogleResearch/fuel/collections/Google%20Scanned%20Objects)
62. K. He, X. Zhang, S. Ren, and J. Sun. [Deep residual learning for image recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/). In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
63. J. Lu, V. Goswami, M. Rohrbach, D. Parikh, and S. Lee. 12-in-1: Multi-task vision and language representation learning. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020. [find](https://scholar.google.com/scholar?q=J%2E+Lu%2C+V%2E+Goswami%2C+M%2E+Rohrbach%2C+D%2E+Parikh%2C+and+S%2E+Lee%2E+12%2Din%2D1%3A+Multi%2Dtask+vision+and+language+representation+learning%2E+In+The+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition+%28CVPR%29%2C+Ju)
64. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. [Pytorch: An imperative style, high-performance deep learning library](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/). arXiv preprint arXiv:1912.01703, 2019.
65. V. Sanh, L. Debut, J. Chaumond, and T. Wolf. [Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter](https://md.research.elamur.ai/free/en/papers/sanh-2019-distilbert-distilled-version-bert-smaller/). arXiv preprint arXiv:1910.01108, 2019.
66. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. [find](https://scholar.google.com/scholar?q=J%2E+Deng%2C+W%2E+Dong%2C+R%2E+Socher%2C+L%2E%2DJ%2E+Li%2C+K%2E+Li%2C+and+L%2E+Fei%2DFei%2E+Imagenet%3A+A+large%2Dscale+hierarchical+image+database%2E+In+2009+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+pages+248%E2%80%93255%2E+Ie)
67. J. Mao, C. Gan, P. Kohli, J. B. Tenenbaum, and J. Wu. [The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision](https://md.research.elamur.ai/free/en/papers/mao-2019-neuro-symbolic-concept-learner-interpreting/). arXiv preprint arXiv:1904.12584, 2019.
68. D. Ding, F. Hill, A. Santoro, and M. Botvinick. Object-based attention for spatio-temporal reasoning: Outperforming neuro-symbolic models with flexible distributed architectures. arXiv preprint arXiv:2012.08508, 2020. — [arXiv:2012.08508](https://arxiv.org/abs/2012.08508) [find](https://scholar.google.com/scholar?q=D%2E+Ding%2C+F%2E+Hill%2C+A%2E+Santoro%2C+and+M%2E+Botvinick%2E+Object%2Dbased+attention+for+spatio%2Dtemporal+reasoning%3A+Outperforming+neuro%2Dsymbolic+models+with+flexible+distributed+architectures%2E+arXiv+preprint+arXiv%3A)
69. E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623, 2021. [find](https://scholar.google.com/scholar?q=E%2E+M%2E+Bender%2C+T%2E+Gebru%2C+A%2E+McMillan%2DMajor%2C+and+S%2E+Shmitchell%2E+On+the+dangers+of+stochastic+parrots%3A+Can+language+models+be+too+big%3F+In+Proceedings+of+the+2021+ACM+Conference+on+Fairness%2C+Accountabilit)

*Updated 2 November 2026*