---
title: "Scaling Up Visual and Vision-Language — Jia 2021"
description: "Most visual and vision-language models are pre-trained on curated datasets such as ImageNet, OpenImages or Conceptual Captions."
authors: ["Chao Jia", "Yinfei Yang", "Ye Xia", "Yi-Ting Chen", "Zarana Parekh", "Hieu Pham", "Quoc V. Le", "Yunhsuan Sung", "Zhen Li", "Tom Duerig"]
year: 2021
venue: "arXiv:2102.05918 · International Conference on Machine Learning 2021"
source: "https://arxiv.org/abs/2102.05918"
doi: "10.48550/arxiv.2102.05918"
arxiv: "2102.05918"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/jia-2021-scaling-up-visual-vision-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jia-2021-scaling-up-visual-vision-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

# Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

**Authors:** Chao Jia, [Yinfei Yang](https://md.research.elamur.ai/free/en/authors/yinfei-yang/), Ye Xia, Yi-Ting Chen, Zarana Parekh, [Hieu Pham](https://md.research.elamur.ai/free/en/authors/hieu-pham/), [Quoc V. Le](https://md.research.elamur.ai/free/en/authors/quoc-v-le/), Yunhsuan Sung, Zhen Li, Tom Duerig · **Year:** 2021 · **Venue:** arXiv:2102.05918 · International Conference on Machine Learning 2021 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2102.05918) · [PDF](https://arxiv.org/pdf/2102.05918) · [DOI 10.48550/arxiv.2102.05918](https://doi.org/10.48550/arxiv.2102.05918) · [arXiv 2102.05918](https://arxiv.org/abs/2102.05918) · [OpenAlex](https://openalex.org/W3126337491)

**Citations:** 1165 citations (openalex-snapshot · 2026-10-14) · [362](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Most visual and vision-language models are pre-trained on curated datasets such as ImageNet, OpenImages or Conceptual Captions. Building these requires costly annotation and cleaning, which keeps them small. Vision-language corpora in particular stay around ~10M examples. This paper asks whether sheer data volume can make up for label noise. The authors from Google Research gathered English image alt-text pairs using the Conceptual Captions collection procedure. They replaced its elaborate cleaning with a few cheap frequency-based filters, yielding about 1.8B noisy pairs.

On this data they trained ALIGN, a plain dual-encoder model. It pairs an EfficientNet image tower with a BERT text tower, both trained from scratch. A symmetric normalized-softmax contrastive loss uses in-batch negatives and a learned temperature. Despite its simplicity, ALIGN sets new state-of-the-art results on Flickr30K and MSCOCO retrieval, beating heavier cross-attention models. It also performs strongly on ImageNet, both zero-shot and fine-tuned, and on VTAB. The work shows that large uncurated web data, needing no expert curation, can produce competitive general-purpose representations.

## Key points
- **Dataset:** about 1.8B image–alt-text pairs, roughly 100 times larger than Conceptual Captions. The filters were:
  - images: shorter side above 200 px, aspect ratio below 3;
  - alt-texts: dropped if shared by more than 10 images or containing rare tokens;
  - text length: 3–20 unigrams;
  - near-duplicates of test images were removed.
- **Image-only results:** 76.4% zero-shot top-1 on ImageNet, using class names as text queries. Fine-tuned ImageNet top-1 reaches 88.64%.
- **Retrieval results:** on Flickr30K and MSCOCO, ALIGN beats the prior best by over 7% on most zero-shot and fine-tuned R@1 metrics.
- **Compositional queries:** embeddings can be added and subtracted, enabling search with combined image+text queries.
- **Multilingual model:** ALIGN_mling was trained on 1.8B pairs covering 100+ languages with a 250k wordpiece vocabulary.
  - Zero-shot on Multi30k, it outperforms M3P in every language, by up to +57.8 mR on French.
  - Its zero-shot scores are roughly on par with fine-tuned M3P and UC2, except on Czech.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

## Cited by

Cited by

362 papers of our library cite it (of the 140673 whose bibliographies we read); 31 of them have a page here, 331 not yet.

- [Flamingo: a Visual Language Model for Few-Shot Learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/)
- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://md.research.elamur.ai/free/en/papers/chen-2022-pali-jointly-scaled-multilingual-language/)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://md.research.elamur.ai/free/en/papers/chen-2023-internvl-scaling-up-vision/)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://md.research.elamur.ai/free/en/papers/chen-2023-sharegpt4v-improving-large-multi-modal/)
- [Are We on the Right Way for Evaluating Large Vision-Language Models?](https://md.research.elamur.ai/free/en/papers/chen-2024-right-way-evaluating-large-vision/)
- [Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models](https://md.research.elamur.ai/free/en/papers/deitke-2024-molmo-pixmo-open-weights-open/)
- [ImageBind: One Embedding Space To Bind Them All](https://md.research.elamur.ai/free/en/papers/girdhar-2023-imagebind-one-embedding-space-bind/)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://md.research.elamur.ai/free/en/papers/han-2024-parameter-efficient-fine-tuning-large/)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://md.research.elamur.ai/free/en/papers/hessel-2021-clipscore-reference-free-evaluation-metric/)
- [Editing Models with Task Arithmetic](https://md.research.elamur.ai/free/en/papers/ilharco-2022-editing-models-task-arithmetic/)
- [Segment Anything](https://md.research.elamur.ai/free/en/papers/kirillov-2023-segment-anything/)
- [OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents](https://md.research.elamur.ai/free/en/papers/lauren-2023-ccon-obelics-open-web-scale-filtered/)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://md.research.elamur.ai/free/en/papers/li-2022-blip-bootstrapping-language-image-pre/)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/)
- [OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models](https://md.research.elamur.ai/free/en/papers/liu-2023-ocrbench-hidden-mystery-ocr-large/)
- [A Large-Scale Instruction-Tuning Dataset and Models for Slovenian Vision-Language Tasks](https://md.research.elamur.ai/free/en/papers/martinc-2026-large-scale-instruction-tuning-dataset/)
- [Text and Code Embeddings by Contrastive Pre-Training](https://md.research.elamur.ai/free/en/papers/neelakantan-2022-text-code-embeddings-contrastive-pre/)
- [SemEval-2025 Task 1: AdMIRe - Advancing Multimodal Idiomaticity Representation](https://md.research.elamur.ai/free/en/papers/pickard-2025-semeval-2025-task-1-admire/)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://md.research.elamur.ai/free/en/papers/saharia-2022-photorealistic-text-image-diffusion-models/)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://md.research.elamur.ai/free/en/papers/schuhmann-2021-laion-400m-open-dataset-clip/)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://md.research.elamur.ai/free/en/papers/schuhmann-2022-laion-5b-open-large-scale/)
- [Multimodal Few-Shot Learning with Frozen Language Models](https://md.research.elamur.ai/free/en/papers/tsimpoukelli-2021-multimodal-few-shot-learning-frozen/)
- [SimVLM: Simple Visual Language Model Pretraining with Weak Supervision](https://md.research.elamur.ai/free/en/papers/wang-2021-simvlm-simple-visual-language-model/)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://md.research.elamur.ai/free/en/papers/wang-2022-text-embeddings-weakly-supervised-contrastive/)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://md.research.elamur.ai/free/en/papers/wortsman-2022-model-soups-averaging-weights-multiple/)
- [视觉语言预训练综述](https://md.research.elamur.ai/free/en/papers/yin-2022-vision-language-pre/)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://md.research.elamur.ai/free/en/papers/yu-2022-coca-contrastive-captioners-image-text/)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/)
- [Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language](https://md.research.elamur.ai/free/en/papers/zeng-2022-socratic-models-composing-zero-shot/)
- [LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention](https://md.research.elamur.ai/free/en/papers/zhang-2023-llama-adapter-efficient-fine-tuning/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{jia2021scaling,
  title = {{Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision}},
  author = {Jia, Chao and Yang, Yinfei and Xia, Ye and Chen, Yi-Ting and Parekh, Zarana and Pham, Hieu and Le, Quoc V. and Sung, Yunhsuan and Li, Zhen and Duerig, Tom},
  year = {2021},
  doi = {10.48550/arxiv.2102.05918},
  eprint = {2102.05918},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2102.05918}
}
```

- [← previous RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering](https://md.research.elamur.ai/free/en/papers/qu-2021-rocketqa-optimized-training-approach-dense/)
- [next → Scaling Within Document Coreference to Long Texts](https://md.research.elamur.ai/free/en/papers/thirukovalluru-2021-scaling-within-document-coreference-long/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/)

## References

78 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Barrault, L., Bougares, F., Specia, L., Lala, C., Elliott, D., and Frank, S. Findings of the third shared task on multimodal machine translation. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pp. 304–323, 2018. [find](https://scholar.google.com/scholar?q=Barrault%2C+L%2E%2C+Bougares%2C+F%2E%2C+Specia%2C+L%2E%2C+Lala%2C+C%2E%2C+Elliott%2C+D%2E%2C+and+Frank%2C+S%2E+Findings+of+the+third+shared+task+on+multimodal+machine+translation%2E+In+Proceedings+of+the+Third+Conference+on+Machine+Tran) [our card](https://md.research.elamur.ai/free/en/papers/barrault-2018-findings-third-shared-task-multimodal/)
2. Bossard, L., Guillaumin, M., and Van Gool, L. Food-101 – mining discriminative components with random forests. In European Conference on Computer Vision, 2014. [find](https://scholar.google.com/scholar?q=Bossard%2C+L%2E%2C+Guillaumin%2C+M%2E%2C+and+Van+Gool%2C+L%2E+Food%2D101+%E2%80%93+mining+discriminative+components+with+random+forests%2E+In+European+Conference+on+Computer+Vision%2C+2014%2E)
3. Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A. Unsupervised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems, 2020. [find](https://scholar.google.com/scholar?q=Caron%2C+M%2E%2C+Misra%2C+I%2E%2C+Mairal%2C+J%2E%2C+Goyal%2C+P%2E%2C+Bojanowski%2C+P%2E%2C+and+Joulin%2C+A%2E+Unsupervised+learning+of+visual+features+by+contrasting+cluster+assignments%2E+In+Advances+in+Neural+Information+Processing+Sy)
4. Chen, J., Hu, H., Wu, H., Jiang, Y., and Wang, C. Learning the best pooling strategy for visual semantic embedding. In arXiv preprint arXiv:2011.04305, 2020a. — [arXiv:2011.04305](https://arxiv.org/abs/2011.04305) [find](https://scholar.google.com/scholar?q=Chen%2C+J%2E%2C+Hu%2C+H%2E%2C+Wu%2C+H%2E%2C+Jiang%2C+Y%2E%2C+and+Wang%2C+C%2E+Learning+the+best+pooling+strategy+for+visual+semantic+embedding%2E+In+arXiv+preprint+arXiv%3A2011%2E04305%2C+2020a%2E+%E2%80%94+%5BarXiv%3A2011%2E04305%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
5. Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual representations. In Proceedings of International Conference on Machine Learning, 2020b. [find](https://scholar.google.com/scholar?q=Chen%2C+T%2E%2C+Kornblith%2C+S%2E%2C+Norouzi%2C+M%2E%2C+and+Hinton%2C+G%2E+A+simple+framework+for+contrastive+learning+of+visual+representations%2E+In+Proceedings+of+International+Conference+on+Machine+Learning%2C+2020b%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2020-simple-framework-contrastive-learning-visual/)
6. Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., and Zitnick, C. L. Microsoft coco captions: Data collection and evaluation server. In arXiv preprint arXiv:1504.00325, 2015. [find](https://scholar.google.com/scholar?q=Chen%2C+X%2E%2C+Fang%2C+H%2E%2C+Lin%2C+T%2E%2DY%2E%2C+Vedantam%2C+R%2E%2C+Gupta%2C+S%2E%2C+Dollar%2C+P%2E%2C+and+Zitnick%2C+C%2E+L%2E+Microsoft+coco+captions%3A+Data+collection+and+evaluation+server%2E+In+arXiv+preprint+arXiv%3A1504%2E00325%2C+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2015-microsoft-coco-captions-data-collection/)
7. Chen, Y.-C., Li, L., Yu, L., Kholy, A. E., Ahmed, F., Gan, Z., Cheng, Y., and Liu, J. Uniter: Universal image-text representation learning. In Proceedings of European Conference on Computer Vision, 2020c. [find](https://scholar.google.com/scholar?q=Chen%2C+Y%2E%2DC%2E%2C+Li%2C+L%2E%2C+Yu%2C+L%2E%2C+Kholy%2C+A%2E+E%2E%2C+Ahmed%2C+F%2E%2C+Gan%2C+Z%2E%2C+Cheng%2C+Y%2E%2C+and+Liu%2C+J%2E+Uniter%3A+Universal+image%2Dtext+representation+learning%2E+In+Proceedings+of+European+Conference+on+Computer+Vision%2C+20) [our card](https://md.research.elamur.ai/free/en/papers/chen-2019-uniter-universal-image-text-representation/)
8. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In Proceedings of Conference on Computer Vision and Pattern Recognition, 2009. [find](https://scholar.google.com/scholar?q=Deng%2C+J%2E%2C+Dong%2C+W%2E%2C+Socher%2C+R%2E%2C+Li%2C+L%2E%2DJ%2E%2C+Li%2C+K%2E%2C+and+Fei%2DFei%2C+L%2E+Imagenet%3A+A+large%2Dscale+hierarchical+image+database%2E+In+Proceedings+of+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+2009%2E)
9. Desai, K. and Johnson, J. Virtex: Learning visual representations from textual annotations. In arXiv preprint arXiv:2006.06666, 2020. — [arXiv:2006.06666](https://arxiv.org/abs/2006.06666) [find](https://scholar.google.com/scholar?q=Desai%2C+K%2E+and+Johnson%2C+J%2E+Virtex%3A+Learning+visual+representations+from+textual+annotations%2E+In+arXiv+preprint+arXiv%3A2006%2E06666%2C+2020%2E+%E2%80%94+%5BarXiv%3A2006%2E06666%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2006%2E06666%29)
10. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of Conference of the North American Chapter of the Association for Computational Linguistics, 2019. [find](https://scholar.google.com/scholar?q=Devlin%2C+J%2E%2C+Chang%2C+M%2E%2DW%2E%2C+Lee%2C+K%2E%2C+and+Toutanova%2C+K%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+Conference+of+the+North+American+Chapter+of+the)
11. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of International Conference on Learnin… [find](https://scholar.google.com/scholar?q=Dosovitskiy%2C+A%2E%2C+Beyer%2C+L%2E%2C+Kolesnikov%2C+A%2E%2C+Weissenborn%2C+D%2E%2C+Zhai%2C+X%2E%2C+Unterthiner%2C+T%2E%2C+Dehghani%2C+M%2E%2C+Minderer%2C+M%2E%2C+Heigold%2C+G%2E%2C+Gelly%2C+S%2E%2C+Uszkoreit%2C+J%2E%2C+and+Houlsby%2C+N%2E+An+image+is+worth+16x16+words)
12. Elliott, D., Frank, S., Sima’an, K., and Specia, L. Multi30k: Multilingual english-german image descriptions. In Proceedings of the 5th Workshop on Vision and Language, 2016. [find](https://scholar.google.com/scholar?q=Elliott%2C+D%2E%2C+Frank%2C+S%2E%2C+Sima%E2%80%99an%2C+K%2E%2C+and+Specia%2C+L%2E+Multi30k%3A+Multilingual+english%2Dgerman+image+descriptions%2E+In+Proceedings+of+the+5th+Workshop+on+Vision+and+Language%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/elliott-2016-multi30k-multilingual-english-german-image/)
13. Elliott, D., Frank, S., Barrault, L., Bougares, F., and Specia, L. Findings of the second shared task on multimodal machine translation and multilingual image description. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers, September 2017. [find](https://scholar.google.com/scholar?q=Elliott%2C+D%2E%2C+Frank%2C+S%2E%2C+Barrault%2C+L%2E%2C+Bougares%2C+F%2E%2C+and+Specia%2C+L%2E+Findings+of+the+second+shared+task+on+multimodal+machine+translation+and+multilingual+image+description%2E+In+Proceedings+of+the+Second) [our card](https://md.research.elamur.ai/free/en/papers/elliott-2017-findings-second-shared-task-multimodal/)
14. Faghri, F., Fleet, D. J., Kiros, J. R., and Fidler, S. Vse++: Improving visual-semantic embeddings with hard negatives. In Proceedings of the British Machine Vision Conference, 2018. [find](https://scholar.google.com/scholar?q=Faghri%2C+F%2E%2C+Fleet%2C+D%2E+J%2E%2C+Kiros%2C+J%2E+R%2E%2C+and+Fidler%2C+S%2E+Vse%2B%2B%3A+Improving+visual%2Dsemantic+embeddings+with+hard+negatives%2E+In+Proceedings+of+the+British+Machine+Vision+Conference%2C+2018%2E)
15. Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations, 2021. [find](https://scholar.google.com/scholar?q=Foret%2C+P%2E%2C+Kleiner%2C+A%2E%2C+Mobahi%2C+H%2E%2C+and+Neyshabur%2C+B%2E+Sharpness%2Daware+minimization+for+efficiently+improving+generalization%2E+In+International+Conference+on+Learning+Representations%2C+2021%2E)
16. Frome, A., Corrado, G. S., Shlens, J., Bengio, S., Dean, J., Ranzato, M. A., and Mikolov, T. Devise: A deep visual-semantic embedding model. In Proceedings of Neural Information Processing Systems, 2013. [find](https://scholar.google.com/scholar?q=Frome%2C+A%2E%2C+Corrado%2C+G%2E+S%2E%2C+Shlens%2C+J%2E%2C+Bengio%2C+S%2E%2C+Dean%2C+J%2E%2C+Ranzato%2C+M%2E+A%2E%2C+and+Mikolov%2C+T%2E+Devise%3A+A+deep+visual%2Dsemantic+embedding+model%2E+In+Proceedings+of+Neural+Information+Processing+Systems%2C+20)
17. Gan, Z., Chen, Y.-C., Li, L., Zhu, C., Cheng, Y., and Liu, J. Large-scale adversarial training for vision-and-language representation learning. In Proceedings of Neural Information Processing Systems, 2020. [find](https://scholar.google.com/scholar?q=Gan%2C+Z%2E%2C+Chen%2C+Y%2E%2DC%2E%2C+Li%2C+L%2E%2C+Zhu%2C+C%2E%2C+Cheng%2C+Y%2E%2C+and+Liu%2C+J%2E+Large%2Dscale+adversarial+training+for+vision%2Dand%2Dlanguage+representation+learning%2E+In+Proceedings+of+Neural+Information+Processing+Systems%2C)
18. Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., Piot, B., Kavukcuoglu, K., Munos, R., and Valko, M. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733, 2020. — [arXiv:2006.07733](https://arxiv.org/abs/2006.07733) [find](https://scholar.google.com/scholar?q=Grill%2C+J%2E%2DB%2E%2C+Strub%2C+F%2E%2C+Altche%CC%81%2C+F%2E%2C+Tallec%2C+C%2E%2C+Richemond%2C+P%2E+H%2E%2C+Buchatskaya%2C+E%2E%2C+Doersch%2C+C%2E%2C+Pires%2C+B%2E+A%2E%2C+Guo%2C+Z%2E+D%2E%2C+Azar%2C+M%2E+G%2E%2C+Piot%2C+B%2E%2C+Kavukcuoglu%2C+K%2E%2C+Munos%2C+R%2E%2C+and+Valko%2C+M%2E+Bootstrap+y)
19. He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2020. [find](https://scholar.google.com/scholar?q=He%2C+K%2E%2C+Fan%2C+H%2E%2C+Wu%2C+Y%2E%2C+Xie%2C+S%2E%2C+and+Girshick%2C+R%2E+Momentum+contrast+for+unsupervised+visual+representation+learning%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recogniti) [our card](https://md.research.elamur.ai/free/en/papers/he-2019-momentum-contrast-unsupervised-visual-representation/)
20. Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., Song, D., Steinhardt, J., and Gilmer, J. The many faces of robustness: A critical analysis of out-of-distribution generalization. arXiv preprint arXiv:2006.16241, 2020. — [arXiv:2006.16241](https://arxiv.org/abs/2006.16241) [find](https://scholar.google.com/scholar?q=Hendrycks%2C+D%2E%2C+Basart%2C+S%2E%2C+Mu%2C+N%2E%2C+Kadavath%2C+S%2E%2C+Wang%2C+F%2E%2C+Dorundo%2C+E%2E%2C+Desai%2C+R%2E%2C+Zhu%2C+T%2E%2C+Parajuli%2C+S%2E%2C+Guo%2C+M%2E%2C+Song%2C+D%2E%2C+Steinhardt%2C+J%2E%2C+and+Gilmer%2C+J%2E+The+many+faces+of+robustness%3A+A+critical+ana)
21. Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. Natural adversarial examples. CVPR, 2021. [find](https://scholar.google.com/scholar?q=Hendrycks%2C+D%2E%2C+Zhao%2C+K%2E%2C+Basart%2C+S%2E%2C+Steinhardt%2C+J%2E%2C+and+Song%2C+D%2E+Natural+adversarial+examples%2E+CVPR%2C+2021%2E)
22. Hill, F., Reichart, R., and Korhonen, A. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 2015. [find](https://scholar.google.com/scholar?q=Hill%2C+F%2E%2C+Reichart%2C+R%2E%2C+and+Korhonen%2C+A%2E+Simlex%2D999%3A+Evaluating+semantic+models+with+%28genuine%29+similarity+estimation%2E+Computational+Linguistics%2C+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/hill-2015-simlex-999-evaluating-semantic-models/)
23. Huang, H., Su, L., Qi, D., Duan, N., Cui, E., Bharti, T., Zhang, L., Wang, L., Gao, J., Liu, B., Fu, J., Zhang, D., Liu, X., and Zhou, M. M3p: Learning universal representations via multitask multilingual multimodal pre-training. arXiv, abs/2006.02635, 2020a. — [arXiv:2006.02635](https://arxiv.org/abs/2006.02635) [find](https://scholar.google.com/scholar?q=Huang%2C+H%2E%2C+Su%2C+L%2E%2C+Qi%2C+D%2E%2C+Duan%2C+N%2E%2C+Cui%2C+E%2E%2C+Bharti%2C+T%2E%2C+Zhang%2C+L%2E%2C+Wang%2C+L%2E%2C+Gao%2C+J%2E%2C+Liu%2C+B%2E%2C+Fu%2C+J%2E%2C+Zhang%2C+D%2E%2C+Liu%2C+X%2E%2C+and+Zhou%2C+M%2E+M3p%3A+Learning+universal+representations+via+multitask+multilin)
24. Huang, Z., Zeng, Z., Liu, B., Fu, D., and Fu, J. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. arXiv preprint arXiv:2004.00849, 2020b. — [arXiv:2004.00849](https://arxiv.org/abs/2004.00849) [find](https://scholar.google.com/scholar?q=Huang%2C+Z%2E%2C+Zeng%2C+Z%2E%2C+Liu%2C+B%2E%2C+Fu%2C+D%2E%2C+and+Fu%2C+J%2E+Pixel%2Dbert%3A+Aligning+image+pixels+with+text+by+deep+multi%2Dmodal+transformers%2E+arXiv+preprint+arXiv%3A2004%2E00849%2C+2020b%2E+%E2%80%94+%5BarXiv%3A2004%2E00849%5D%28https%3A%2F%2Farxi)
25. Joulin, A., van der Maaten, L., Jabri, A., and Vasilache, N. Learning visual features from large weakly supervised data. In European Conference on Computer Vision, 2015. [find](https://scholar.google.com/scholar?q=Joulin%2C+A%2E%2C+van+der+Maaten%2C+L%2E%2C+Jabri%2C+A%2E%2C+and+Vasilache%2C+N%2E+Learning+visual+features+from+large+weakly+supervised+data%2E+In+European+Conference+on+Computer+Vision%2C+2015%2E)
26. Juan, D.-C., Lu, C.-T., Li, Z., Peng, F., Timofeev, A., Chen, Y.-T., Gao, Y., Duerig, T., Tomkins, A., and Ravi, S. Graph-rise: Graph-regularized image semantic embedding. In Proceedings of ACM International Conference on Web Search and Data Mining, 2020. [find](https://scholar.google.com/scholar?q=Juan%2C+D%2E%2DC%2E%2C+Lu%2C+C%2E%2DT%2E%2C+Li%2C+Z%2E%2C+Peng%2C+F%2E%2C+Timofeev%2C+A%2E%2C+Chen%2C+Y%2E%2DT%2E%2C+Gao%2C+Y%2E%2C+Duerig%2C+T%2E%2C+Tomkins%2C+A%2E%2C+and+Ravi%2C+S%2E+Graph%2Drise%3A+Graph%2Dregularized+image+semantic+embedding%2E+In+Proceedings+of+ACM+Intern)
27. Karpathy, A. and Fei-Fei, L. Deep visual-semantic alignments for generating image descriptions. In Proceedings of Conference on Computer Vision and Pattern Recognition, 2015. [find](https://scholar.google.com/scholar?q=Karpathy%2C+A%2E+and+Fei%2DFei%2C+L%2E+Deep+visual%2Dsemantic+alignments+for+generating+image+descriptions%2E+In+Proceedings+of+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/karpathy-2014-deep-visual-semantic-alignments-generating/)
28. Karpathy, A., Joulin, A., and Li, F. Deep fragment embeddings for bidirectional image sentence mapping. In Advances in Neural Information Processing Systems, 2014. [find](https://scholar.google.com/scholar?q=Karpathy%2C+A%2E%2C+Joulin%2C+A%2E%2C+and+Li%2C+F%2E+Deep+fragment+embeddings+for+bidirectional+image+sentence+mapping%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2014%2E)
29. Kiros, J., Chan, W., and Hinton, G. Illustrative language understanding: Large-scale visual grounding with image search. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018. [find](https://scholar.google.com/scholar?q=Kiros%2C+J%2E%2C+Chan%2C+W%2E%2C+and+Hinton%2C+G%2E+Illustrative+language+understanding%3A+Large%2Dscale+visual+grounding+with+image+search%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+)
30. Kiros, R., Salakhutdinov, R., and Zemel, R. S. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539. Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., and Houlsby, N. Big transfer (bit): General visual representation learn… [find](https://scholar.google.com/scholar?q=Kiros%2C+R%2E%2C+Salakhutdinov%2C+R%2E%2C+and+Zemel%2C+R%2E+S%2E+Unifying+visual%2Dsemantic+embeddings+with+multimodal+neural+language+models%2E+arXiv+preprint+arXiv%3A1411%2E2539%2E+Kolesnikov%2C+A%2E%2C+Beyer%2C+L%2E%2C+Zhai%2C+X%2E%2C+Puigcerv) [our card](https://md.research.elamur.ai/free/en/papers/kiros-2014-unifying-visual-semantic-embeddings-multimodal/)
31. Krause, J., Stark, M., Deng, J., and Fei-Fei, L. 3d object representations for fine-grained categorization. In Proceedings of ICCV Workshop on 3D Representation and Recognition, 2013. [find](https://scholar.google.com/scholar?q=Krause%2C+J%2E%2C+Stark%2C+M%2E%2C+Deng%2C+J%2E%2C+and+Fei%2DFei%2C+L%2E+3d+object+representations+for+fine%2Dgrained+categorization%2E+In+Proceedings+of+ICCV+Workshop+on+3D+Representation+and+Recognition%2C+2013%2E)
32. Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., Bernstein, M., and Fei-Fei, L. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 2016. [find](https://scholar.google.com/scholar?q=Krishna%2C+R%2E%2C+Zhu%2C+Y%2E%2C+Groth%2C+O%2E%2C+Johnson%2C+J%2E%2C+Hata%2C+K%2E%2C+Kravitz%2C+J%2E%2C+Chen%2C+S%2E%2C+Kalantidis%2C+Y%2E%2C+Li%2C+L%2E%2DJ%2E%2C+Shamma%2C+D%2E+A%2E%2C+Bernstein%2C+M%2E%2C+and+Fei%2DFei%2C+L%2E+Visual+genome%3A+Connecting+language+and+vision+us) [our card](https://md.research.elamur.ai/free/en/papers/krishna-2016-visual-genome-connecting-language-vision/)
33. Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., Duerig, T., and Ferrari, V. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International Journa… [find](https://scholar.google.com/scholar?q=Kuznetsova%2C+A%2E%2C+Rom%2C+H%2E%2C+Alldrin%2C+N%2E%2C+Uijlings%2C+J%2E%2C+Krasin%2C+I%2E%2C+Pont%2DTuset%2C+J%2E%2C+Kamali%2C+S%2E%2C+Popov%2C+S%2E%2C+Malloci%2C+M%2E%2C+Kolesnikov%2C+A%2E%2C+Duerig%2C+T%2E%2C+and+Ferrari%2C+V%2E+The+open+images+dataset+v4%3A+Unified+imag)
34. Li, A., Jabri, A., Joulin, A., and van der Maaten, L. Learning visual n-grams from web data. In Proceedings of IEEE International Conference on Computer Vision, 2017. [find](https://scholar.google.com/scholar?q=Li%2C+A%2E%2C+Jabri%2C+A%2E%2C+Joulin%2C+A%2E%2C+and+van+der+Maaten%2C+L%2E+Learning+visual+n%2Dgrams+from+web+data%2E+In+Proceedings+of+IEEE+International+Conference+on+Computer+Vision%2C+2017%2E)
35. Li, J., Zhou, P., Xiong, C., and Hoi, S. Prototypical contrastive learning of unsupervised representations. In International Conference on Learning Representations, 2021. [find](https://scholar.google.com/scholar?q=Li%2C+J%2E%2C+Zhou%2C+P%2E%2C+Xiong%2C+C%2E%2C+and+Hoi%2C+S%2E+Prototypical+contrastive+learning+of+unsupervised+representations%2E+In+International+Conference+on+Learning+Representations%2C+2021%2E)
36. Li, K., Zhang, Y., Li, K., Li, Y., and Fu, Y. Visual semantic reasoning for image-text matching. In Proceedings of International Conference on Computer Vision, 2019. [find](https://scholar.google.com/scholar?q=Li%2C+K%2E%2C+Zhang%2C+Y%2E%2C+Li%2C+K%2E%2C+Li%2C+Y%2E%2C+and+Fu%2C+Y%2E+Visual+semantic+reasoning+for+image%2Dtext+matching%2E+In+Proceedings+of+International+Conference+on+Computer+Vision%2C+2019%2E)
37. Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., Wei, F., Choi, Y., and Gao, J. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Proceedings of European Conference on Computer Vision, 2020. [find](https://scholar.google.com/scholar?q=Li%2C+X%2E%2C+Yin%2C+X%2E%2C+Li%2C+C%2E%2C+Zhang%2C+P%2E%2C+Hu%2C+X%2E%2C+Zhang%2C+L%2E%2C+Wang%2C+L%2E%2C+Hu%2C+H%2E%2C+Dong%2C+L%2E%2C+Wei%2C+F%2E%2C+Choi%2C+Y%2E%2C+and+Gao%2C+J%2E+Oscar%3A+Object%2Dsemantics+aligned+pre%2Dtraining+for+vision%2Dlanguage+tasks%2E+In+Proceedings) [our card](https://md.research.elamur.ai/free/en/papers/li-2020-oscar-object-semantics-aligned-pre/)
38. Liu, F., Liu, Y., Ren, X., He, X., and Sun, X. Aligning visual regions and textual concepts for semantic-grounded image representations. In Advances in Neural Information Processing Systems, 2019a. [find](https://scholar.google.com/scholar?q=Liu%2C+F%2E%2C+Liu%2C+Y%2E%2C+Ren%2C+X%2E%2C+He%2C+X%2E%2C+and+Sun%2C+X%2E+Aligning+visual+regions+and+textual+concepts+for+semantic%2Dgrounded+image+representations%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2019a%2E)
39. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019b. [find](https://scholar.google.com/scholar?q=Liu%2C+Y%2E%2C+Ott%2C+M%2E%2C+Goyal%2C+N%2E%2C+Du%2C+J%2E%2C+Joshi%2C+M%2E%2C+Chen%2C+D%2E%2C+Levy%2C+O%2E%2C+Lewis%2C+M%2E%2C+Zettlemoyer%2C+L%2E%2C+and+Stoyanov%2C+V%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+arXiv+preprint+arXiv%3A1907%2E1169)
40. Lu, J., Batra, D., Parikh, D., and Lee, S. Vilbert: Pre-training task-agnostic visiolinguistic representations for vision-and-language tasks. In Proceedings of Neural Information Processing Systems, 2019. [find](https://scholar.google.com/scholar?q=Lu%2C+J%2E%2C+Batra%2C+D%2E%2C+Parikh%2C+D%2E%2C+and+Lee%2C+S%2E+Vilbert%3A+Pre%2Dtraining+task%2Dagnostic+visiolinguistic+representations+for+vision%2Dand%2Dlanguage+tasks%2E+In+Proceedings+of+Neural+Information+Processing+Systems%2C+2)
41. Mahajan, D., Girshick, R., Ramanathan, V., He, K., Paluri, M., Li, Y., Bharambe, A., and van der Maaten, L. Exploring the limits of weakly supervised pretraining. In Proceedings of European Conference on Computer Vision, 2018. [find](https://scholar.google.com/scholar?q=Mahajan%2C+D%2E%2C+Girshick%2C+R%2E%2C+Ramanathan%2C+V%2E%2C+He%2C+K%2E%2C+Paluri%2C+M%2E%2C+Li%2C+Y%2E%2C+Bharambe%2C+A%2E%2C+and+van+der+Maaten%2C+L%2E+Exploring+the+limits+of+weakly+supervised+pretraining%2E+In+Proceedings+of+European+Conference)
42. Messina, N., Amato, G., Esuli, A., Falchi, F., Gennaro, C., and Marchand-Maillet, S. Fine-grained visual textual alignment for cross-modal retrieval using transformer encoders. ACM Transactions on Multimedia Computing, Communications, and Applications, 2020. [find](https://scholar.google.com/scholar?q=Messina%2C+N%2E%2C+Amato%2C+G%2E%2C+Esuli%2C+A%2E%2C+Falchi%2C+F%2E%2C+Gennaro%2C+C%2E%2C+and+Marchand%2DMaillet%2C+S%2E+Fine%2Dgrained+visual+textual+alignment+for+cross%2Dmodal+retrieval+using+transformer+encoders%2E+ACM+Transactions+on+Mul)
43. Mikolov, T., Chen, K., Corrado, G., and Dean, J. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013a. [find](https://scholar.google.com/scholar?q=Mikolov%2C+T%2E%2C+Chen%2C+K%2E%2C+Corrado%2C+G%2E%2C+and+Dean%2C+J%2E+Efficient+estimation+of+word+representations+in+vector+space%2E+arXiv+preprint+arXiv%3A1301%2E3781%2C+2013a%2E)
44. Mikolov, T., Sutskever, I., Chen, K., Corrado, G., and Dean, J. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, 2013b. [find](https://scholar.google.com/scholar?q=Mikolov%2C+T%2E%2C+Sutskever%2C+I%2E%2C+Chen%2C+K%2E%2C+Corrado%2C+G%2E%2C+and+Dean%2C+J%2E+Distributed+representations+of+words+and+phrases+and+their+compositionality%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2013b)
45. Misra, I. and Maaten, L. v. d. Self-supervised learning of pretext-invariant representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2020. [find](https://scholar.google.com/scholar?q=Misra%2C+I%2E+and+Maaten%2C+L%2E+v%2E+d%2E+Self%2Dsupervised+learning+of+pretext%2Dinvariant+representations%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+June+2020%2E)
46. Musgrave, K., Belongie, S., and Lim, S.-N. A metric learning reality check. In Proceedings of European Conference on Computer Vision, 2020. [find](https://scholar.google.com/scholar?q=Musgrave%2C+K%2E%2C+Belongie%2C+S%2E%2C+and+Lim%2C+S%2E%2DN%2E+A+metric+learning+reality+check%2E+In+Proceedings+of+European+Conference+on+Computer+Vision%2C+2020%2E)
47. Nam, H., Ha, J.-W., and Kim, J. Dual attention networks for multimodal reasoning and matching. In Proceedings of Conference on Computer Vision and Pattern Recognition, 2017. [find](https://scholar.google.com/scholar?q=Nam%2C+H%2E%2C+Ha%2C+J%2E%2DW%2E%2C+and+Kim%2C+J%2E+Dual+attention+networks+for+multimodal+reasoning+and+matching%2E+In+Proceedings+of+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+2017%2E)
48. Nilsback, M.-E. and Zisserman, A. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008. [find](https://scholar.google.com/scholar?q=Nilsback%2C+M%2E%2DE%2E+and+Zisserman%2C+A%2E+Automated+flower+classification+over+a+large+number+of+classes%2E+In+Indian+Conference+on+Computer+Vision%2C+Graphics+and+Image+Processing%2C+Dec+2008%2E)
49. Parekh, Z., Baldridge, J., Cer, D., Waters, A., and Yang, Y. Crisscrossed captions: Extended intramodal and intermodal semantic similarity judgments for ms-coco. In Proceedings of Conference of the European Chapter of the Association for Computational Linguistics, 2021. [find](https://scholar.google.com/scholar?q=Parekh%2C+Z%2E%2C+Baldridge%2C+J%2E%2C+Cer%2C+D%2E%2C+Waters%2C+A%2E%2C+and+Yang%2C+Y%2E+Crisscrossed+captions%3A+Extended+intramodal+and+intermodal+semantic+similarity+judgments+for+ms%2Dcoco%2E+In+Proceedings+of+Conference+of+the+Eu)
50. Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. V. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, 2012. [find](https://scholar.google.com/scholar?q=Parkhi%2C+O%2E+M%2E%2C+Vedaldi%2C+A%2E%2C+Zisserman%2C+A%2E%2C+and+Jawahar%2C+C%2E+V%2E+Cats+and+dogs%2E+In+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+2012%2E)
51. Pennington, J., Socher, R., and Manning, C. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014. [find](https://scholar.google.com/scholar?q=Pennington%2C+J%2E%2C+Socher%2C+R%2E%2C+and+Manning%2C+C%2E+GloVe%3A+Global+vectors+for+word+representation%2E+In+Proceedings+of+the+2014+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+2014%2E)
52. Pham, H., Dai, Z., Xie, Q., Luong, M.-T., and Le, Q. V. Meta pseudo labels. In arXiv preprint arXiv:2003.10580, 2020. — [arXiv:2003.10580](https://arxiv.org/abs/2003.10580) [find](https://scholar.google.com/scholar?q=Pham%2C+H%2E%2C+Dai%2C+Z%2E%2C+Xie%2C+Q%2E%2C+Luong%2C+M%2E%2DT%2E%2C+and+Le%2C+Q%2E+V%2E+Meta+pseudo+labels%2E+In+arXiv+preprint+arXiv%3A2003%2E10580%2C+2020%2E+%E2%80%94+%5BarXiv%3A2003%2E10580%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2003%2E10580%29)
53. Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., and Lazebnik, S. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the International Conference on Computer Vision, 2015. [find](https://scholar.google.com/scholar?q=Plummer%2C+B%2E+A%2E%2C+Wang%2C+L%2E%2C+Cervantes%2C+C%2E+M%2E%2C+Caicedo%2C+J%2E+C%2E%2C+Hockenmaier%2C+J%2E%2C+and+Lazebnik%2C+S%2E+Flickr30k+entities%3A+Collecting+region%2Dto%2Dphrase+correspondences+for+richer+image%2Dto%2Dsentence+models%2E+In+Pr)
54. Qi, D., Su, L., Song, J., Cui, E., Bharti, T., and Sacheti, A. Imagebert: Cross-modal pre-training with large-scale weak-supervised image-text data. arXiv preprint-arXiv:2001.07966, 2020. — [arXiv:2001.07966](https://arxiv.org/abs/2001.07966) [find](https://scholar.google.com/scholar?q=Qi%2C+D%2E%2C+Su%2C+L%2E%2C+Song%2C+J%2E%2C+Cui%2C+E%2E%2C+Bharti%2C+T%2E%2C+and+Sacheti%2C+A%2E+Imagebert%3A+Cross%2Dmodal+pre%2Dtraining+with+large%2Dscale+weak%2Dsupervised+image%2Dtext+data%2E+arXiv+preprint%2DarXiv%3A2001%2E07966%2C+2020%2E+%E2%80%94+%5BarXiv%3A200)
55. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. 2019. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Wu%2C+J%2E%2C+Child%2C+R%2E%2C+Luan%2C+D%2E%2C+Amodei%2C+D%2E%2C+and+Sutskever%2C+I%2E+Language+models+are+unsupervised+multitask+learners%2E+2019%2E)
56. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarawl, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. 2021. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Kim%2C+J%2E+W%2E%2C+Hallacy%2C+C%2E%2C+Ramesh%2C+A%2E%2C+Goh%2C+G%2E%2C+Agarawl%2C+S%2E%2C+Sastry%2C+G%2E%2C+Askell%2C+A%2E%2C+Mishkin%2C+P%2E%2C+Clark%2C+J%2E%2C+Krueger%2C+G%2E%2C+and+Sutskever%2C+I%2E+Learning+transferable+visual+models+from+natural+) [our card](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/)
57. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 2020. [find](https://scholar.google.com/scholar?q=Raffel%2C+C%2E%2C+Shazeer%2C+N%2E%2C+Roberts%2C+A%2E%2C+Lee%2C+K%2E%2C+Narang%2C+S%2E%2C+Matena%2C+M%2E%2C+Zhou%2C+Y%2E%2C+Li%2C+W%2E%2C+and+Liu%2C+P%2E+J%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtext+transformer%2E+Journal+of+Ma)
58. Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pp. 5389–5400, 2019. [find](https://scholar.google.com/scholar?q=Recht%2C+B%2E%2C+Roelofs%2C+R%2E%2C+Schmidt%2C+L%2E%2C+and+Shankar%2C+V%2E+Do+imagenet+classifiers+generalize+to+imagenet%3F+In+International+Conference+on+Machine+Learning%2C+pp%2E+5389%E2%80%935400%2C+2019%2E)
59. Sariyildiz, M. B., Perez, J., and Larlus, D. Learning visual representations with caption annotations. arXiv preprint arXiv:2008.01392, 2020. — [arXiv:2008.01392](https://arxiv.org/abs/2008.01392) [find](https://scholar.google.com/scholar?q=Sariyildiz%2C+M%2E+B%2E%2C+Perez%2C+J%2E%2C+and+Larlus%2C+D%2E+Learning+visual+representations+with+caption+annotations%2E+arXiv+preprint+arXiv%3A2008%2E01392%2C+2020%2E+%E2%80%94+%5BarXiv%3A2008%2E01392%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2008%2E01392%29)
60. Sharma, P., Ding, N., Goodman, S., and Soricut, R. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of Annual Meeting of the Association for Computational Linguistics, 2018. [find](https://scholar.google.com/scholar?q=Sharma%2C+P%2E%2C+Ding%2C+N%2E%2C+Goodman%2C+S%2E%2C+and+Soricut%2C+R%2E+Conceptual+captions%3A+A+cleaned%2C+hypernymed%2C+image+alt%2Dtext+dataset+for+automatic+image+captioning%2E+In+Proceedings+of+Annual+Meeting+of+the+Associatio) [our card](https://md.research.elamur.ai/free/en/papers/sharma-2018-conceptual-captions-cleaned-hypernymed-image/)
61. Socher, R., Karpathy, A., Le, Q. V., Manning, C. D., and Ng, A. Y. Grounded compositional semantics for finding and describing images with sentences. Transactions of the Association for Computational Linguistics, 2014. [find](https://scholar.google.com/scholar?q=Socher%2C+R%2E%2C+Karpathy%2C+A%2E%2C+Le%2C+Q%2E+V%2E%2C+Manning%2C+C%2E+D%2E%2C+and+Ng%2C+A%2E+Y%2E+Grounded+compositional+semantics+for+finding+and+describing+images+with+sentences%2E+Transactions+of+the+Association+for+Computational+) [our card](https://md.research.elamur.ai/free/en/papers/socher-2014-grounded-compositional-semantics-finding-describing/)
62. Sun, C., Shrivastava, A., Sigh, S., and Gupta, A. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the International Conference on Computer Vision, 2017. [find](https://scholar.google.com/scholar?q=Sun%2C+C%2E%2C+Shrivastava%2C+A%2E%2C+Sigh%2C+S%2E%2C+and+Gupta%2C+A%2E+Revisiting+unreasonable+effectiveness+of+data+in+deep+learning+era%2E+In+Proceedings+of+the+International+Conference+on+Computer+Vision%2C+2017%2E)
63. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. Going deeper with convolutions. In Proceedings of Conference on Computer Vision and Pattern Recognition, 2015. [find](https://scholar.google.com/scholar?q=Szegedy%2C+C%2E%2C+Liu%2C+W%2E%2C+Jia%2C+Y%2E%2C+Sermanet%2C+P%2E%2C+Reed%2C+S%2E%2C+Anguelov%2C+D%2E%2C+Erhan%2C+D%2E%2C+Vanhoucke%2C+V%2E%2C+and+Rabinovich%2C+A%2E+Going+deeper+with+convolutions%2E+In+Proceedings+of+Conference+on+Computer+Vision+and+Pa)
64. Tian, Y., Krishnan, D., and Isola, P. Contrastive multiview coding. In European Conference on Computer Vision, 2020. [find](https://scholar.google.com/scholar?q=Tian%2C+Y%2E%2C+Krishnan%2C+D%2E%2C+and+Isola%2C+P%2E+Contrastive+multiview+coding%2E+In+European+Conference+on+Computer+Vision%2C+2020%2E)
65. Touvron, H., Vedaldi, A., Douze, M., and Jégou, H. Fixing the train-test resolution discrepancy. In Advances in Neural Information Processing Systems, 2019. [find](https://scholar.google.com/scholar?q=Touvron%2C+H%2E%2C+Vedaldi%2C+A%2E%2C+Douze%2C+M%2E%2C+and+Je%CC%81gou%2C+H%2E+Fixing+the+train%2Dtest+resolution+discrepancy%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2019%2E)
66. Wang, J., Song, Y., Leung, T., Rosenberg, C., Wang, J., Philbin, J., Chen, B., and Wu, Y. Learning fine-grained image similarity with deep ranking. In Proceedings of Conference on Computer Vision and Pattern Recognition, 2014. [find](https://scholar.google.com/scholar?q=Wang%2C+J%2E%2C+Song%2C+Y%2E%2C+Leung%2C+T%2E%2C+Rosenberg%2C+C%2E%2C+Wang%2C+J%2E%2C+Philbin%2C+J%2E%2C+Chen%2C+B%2E%2C+and+Wu%2C+Y%2E+Learning+fine%2Dgrained+image+similarity+with+deep+ranking%2E+In+Proceedings+of+Conference+on+Computer+Vision+and+)
67. Xie, Q., Luong, M.-T., Hovy, E., and Le, Q. V. Self-training with noisy student improves imagenet classification. In Proceedings of Conference on Computer Vision and Pattern Recognition, 2020. [find](https://scholar.google.com/scholar?q=Xie%2C+Q%2E%2C+Luong%2C+M%2E%2DT%2E%2C+Hovy%2C+E%2E%2C+and+Le%2C+Q%2E+V%2E+Self%2Dtraining+with+noisy+student+improves+imagenet+classification%2E+In+Proceedings+of+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+2020%2E)
68. Yalniz, I. Z., Jégou, H., Chen, K., Paluri, M., and Mahajan, D. Billion-scale semi-supervised learning for image classification. arXiv preprint arXiv:1905.00546, 2019. — [arXiv:1905.00546](https://arxiv.org/abs/1905.00546) [find](https://scholar.google.com/scholar?q=Yalniz%2C+I%2E+Z%2E%2C+Je%CC%81gou%2C+H%2E%2C+Chen%2C+K%2E%2C+Paluri%2C+M%2E%2C+and+Mahajan%2C+D%2E+Billion%2Dscale+semi%2Dsupervised+learning+for+image+classification%2E+arXiv+preprint+arXiv%3A1905%2E00546%2C+2019%2E+%E2%80%94+%5BarXiv%3A1905%2E00546%5D%28https%3A%2F%2Far)
69. Yang, Z., Dai, Z., Yang, Y., Carbonell, J. G., Salakhutdinov, R., and Le, Q. V. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, 2019. [find](https://scholar.google.com/scholar?q=Yang%2C+Z%2E%2C+Dai%2C+Z%2E%2C+Yang%2C+Y%2E%2C+Carbonell%2C+J%2E+G%2E%2C+Salakhutdinov%2C+R%2E%2C+and+Le%2C+Q%2E+V%2E+Xlnet%3A+Generalized+autoregressive+pretraining+for+language+understanding%2E+In+Advances+in+Neural+Information+Processing+S) [our card](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/)
70. You, Y., Li, J., Reddi, S., Hseu, J., Kumar, S., Bhojanapalli, S., Song, X., Demmel, J., Keutzer, K., and Hsieh, C.-J. Large batch optimization for deep learning: Training bert in 76 minutes. In Proceedings of International Conference on Learning Representations, 2020. [find](https://scholar.google.com/scholar?q=You%2C+Y%2E%2C+Li%2C+J%2E%2C+Reddi%2C+S%2E%2C+Hseu%2C+J%2E%2C+Kumar%2C+S%2E%2C+Bhojanapalli%2C+S%2E%2C+Song%2C+X%2E%2C+Demmel%2C+J%2E%2C+Keutzer%2C+K%2E%2C+and+Hsieh%2C+C%2E%2DJ%2E+Large+batch+optimization+for+deep+learning%3A+Training+bert+in+76+minutes%2E+In+Proce)
71. Young, P., Lai, A., Hodosh, M., and Hockenmaier, J. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2014. [find](https://scholar.google.com/scholar?q=Young%2C+P%2E%2C+Lai%2C+A%2E%2C+Hodosh%2C+M%2E%2C+and+Hockenmaier%2C+J%2E+From+image+descriptions+to+visual+denotations%3A+New+similarity+metrics+for+semantic+inference+over+event+descriptions%2E+Transactions+of+the+Associatio)
72. Yu, F., Tang, J., Yin, W., Sun, Y., Tian, H., Wu, H., and Wang, H. Ernie-vil: Knowledge enhanced vision-language representations through scene graph. arXiv preprint arXiv:2006.16934, 2020. [find](https://scholar.google.com/scholar?q=Yu%2C+F%2E%2C+Tang%2C+J%2E%2C+Yin%2C+W%2E%2C+Sun%2C+Y%2E%2C+Tian%2C+H%2E%2C+Wu%2C+H%2E%2C+and+Wang%2C+H%2E+Ernie%2Dvil%3A+Knowledge+enhanced+vision%2Dlanguage+representations+through+scene+graph%2E+arXiv+preprint+arXiv%3A2006%2E16934%2C+2020%2E)
73. Zhai, A. and Wu, H.-Y. Classification is a strong baseline for deep metric learning. In Proceedings of the British Machine Vision Conference, 2019. [find](https://scholar.google.com/scholar?q=Zhai%2C+A%2E+and+Wu%2C+H%2E%2DY%2E+Classification+is+a+strong+baseline+for+deep+metric+learning%2E+In+Proceedings+of+the+British+Machine+Vision+Conference%2C+2019%2E)
74. Zhai, X., Puigcerver, J., Kolesnikov, A., Ruyssen, P., Riquelme, C., Lucic, M., Djolonga, J., Pinto, A. S., Neumann, M., Dosovitskiy, A., Beyer, L., Bachem, O., Tschannen, M., Michalski, M., Bousquet, O., Gelly, S., and Houlsby, N. A large-scale study of representation learning with the visual task… — [arXiv:1910.04867](https://arxiv.org/abs/1910.04867) [find](https://scholar.google.com/scholar?q=Zhai%2C+X%2E%2C+Puigcerver%2C+J%2E%2C+Kolesnikov%2C+A%2E%2C+Ruyssen%2C+P%2E%2C+Riquelme%2C+C%2E%2C+Lucic%2C+M%2E%2C+Djolonga%2C+J%2E%2C+Pinto%2C+A%2E+S%2E%2C+Neumann%2C+M%2E%2C+Dosovitskiy%2C+A%2E%2C+Beyer%2C+L%2E%2C+Bachem%2C+O%2E%2C+Tschannen%2C+M%2E%2C+Michalski%2C+M%2E%2C+Bousquet%2C)
75. Zhang, Y., Jiang, H., Miura, Y., Manning, C. D., and Langlotz, C. P. Contrastive learning of medical visual representations from paired images and text. arXiv preprint arXiv:2010.00747, 2020. — [arXiv:2010.00747](https://arxiv.org/abs/2010.00747) [find](https://scholar.google.com/scholar?q=Zhang%2C+Y%2E%2C+Jiang%2C+H%2E%2C+Miura%2C+Y%2E%2C+Manning%2C+C%2E+D%2E%2C+and+Langlotz%2C+C%2E+P%2E+Contrastive+learning+of+medical+visual+representations+from+paired+images+and+text%2E+arXiv+preprint+arXiv%3A2010%2E00747%2C+2020%2E+%E2%80%94+%5BarXiv)
76. Zhou, M., Zhou, L., Wang, S., Cheng, Y., Li, L., Yu, Z., and Liu, J. UC2: Universal cross-lingual cross-modal vision-and-language pre-training. arXiv preprint arXiv:2104.00332, 2021. — [arXiv:2104.00332](https://arxiv.org/abs/2104.00332) [find](https://scholar.google.com/scholar?q=Zhou%2C+M%2E%2C+Zhou%2C+L%2E%2C+Wang%2C+S%2E%2C+Cheng%2C+Y%2E%2C+Li%2C+L%2E%2C+Yu%2C+Z%2E%2C+and+Liu%2C+J%2E+UC2%3A+Universal+cross%2Dlingual+cross%2Dmodal+vision%2Dand%2Dlanguage+pre%2Dtraining%2E+arXiv+preprint+arXiv%3A2104%2E00332%2C+2021%2E+%E2%80%94+%5BarXiv%3A2104%2E003)
77. Zoph, B., Ghiasi, G., Lin, T.-Y., Cui, Y., Liu, H., Cubuk, E. D., and Le, Q. V. Rethinking pre-training and self-training. In Advances in Neural Information Processing Systems, 2020. [find](https://scholar.google.com/scholar?q=Zoph%2C+B%2E%2C+Ghiasi%2C+G%2E%2C+Lin%2C+T%2E%2DY%2E%2C+Cui%2C+Y%2E%2C+Liu%2C+H%2E%2C+Cubuk%2C+E%2E+D%2E%2C+and+Le%2C+Q%2E+V%2E+Rethinking+pre%2Dtraining+and+self%2Dtraining%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2020%2E)
78. A. Remove Near-Duplicate Test Images from Training Data [find](https://scholar.google.com/scholar?q=A%2E+Remove+Near%2DDuplicate+Test+Images+from+Training+Data)