---
title: "BLIP-2: Bootstrapping Language-Image Pre-training — Li 2023"
description: "Training vision-language models end to end with ever-larger networks has become very expensive."
authors: ["Junnan Li", "Dongxu Li", "Silvio Savarese", "Steven Hoi"]
year: 2023
venue: "arXiv:2301.12597"
source: "https://arxiv.org/abs/2301.12597"
doi: "10.48550/arxiv.2301.12597"
arxiv: "2301.12597"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/2/) / BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

# BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

**Authors:** [Junnan Li](https://md.research.elamur.ai/free/en/authors/junnan-li/), [Dongxu Li](https://md.research.elamur.ai/free/en/authors/dongxu-li/), [Silvio Savarese](https://md.research.elamur.ai/free/en/authors/silvio-savarese/), [Steven Hoi](https://md.research.elamur.ai/free/en/authors/steven-hoi/) · **Year:** 2023 · **Venue:** arXiv:2301.12597 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2301.12597) · [PDF](https://arxiv.org/pdf/2301.12597) · [DOI 10.48550/arxiv.2301.12597](https://doi.org/10.48550/arxiv.2301.12597) · [arXiv 2301.12597](https://arxiv.org/abs/2301.12597) · [OpenAlex](https://openalex.org/W4318718936)

**Citations:** 908 citations (openalex-snapshot · 2026-10-14) · [1206](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Training vision-language models end to end with ever-larger networks has become very expensive. BLIP-2 avoids this by reusing existing unimodal components: an image encoder and a large language model, both kept frozen. A small trainable module called the Querying Transformer (Q-Former) connects them. It uses a fixed set of learnable queries to pull out the visual information that matters for text. Training happens in two phases. First, the Q-Former is paired with the frozen image encoder and learns text-relevant visual features through contrastive, matching and caption-generation objectives on image-text pairs. Second, its outputs are fed to a frozen LLM, such as OPT or FlanT5, so the LLM can read them as input. With far fewer trainable parameters than competitors, the approach reaches state-of-the-art results in VQA, captioning and image-text retrieval. It also enables zero-shot image-to-text generation that follows instructions. The work shows that strong multimodal systems can be built cheaply by connecting off-the-shelf models, and better unimodal backbones can be swapped in later.

## Key points
- On zero-shot VQAv2, BLIP-2 beats Flamingo80B by 8.7% while training 54× fewer parameters.
- Q-Former details:
  - It has 188M parameters and is initialised from BERT base, with randomly initialised cross-attention layers.
  - It outputs 32 queries of dimension 768, a much smaller bottleneck than the 257×1024 features produced by ViT-L/14.
- The first-stage objectives are ITC, ITM and ITG, each applied with a different self-attention mask between queries and text.
  - Adding ITG raises COCO retrieval R@1 from 84.5 to 85.4 (image→text) and from 67.2 to 68.3 (text→image).
- BLIP-2 ViT-g, fine-tuned on COCO, reaches 97.6 / 89.7 R@1 on zero-shot Flickr30K retrieval (image→text / text→image).
- Limitations:
  - Few-shot in-context examples do not improve VQA. The authors blame pre-training data that contains only one image-text pair per sample.
  - The model inherits LLM risks such as bias and offensive output.
- Code is released in Salesforce's LAVIS repository.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

1206 papers of our library cite it (of the 140884 whose bibliographies we read); 65 of them have a page here, 1141 not yet.

- [BitMar: Low-Bit Multimodal Fusion with Episodic Memory for Edge Devices](https://md.research.elamur.ai/free/en/papers/aman-2025-bitmar-low-bit-multimodal-fusion/)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/)
- [Qwen2.5-VL Technical Report](https://md.research.elamur.ai/free/en/papers/bai-2025-qwen2-5-vl-technical/)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://md.research.elamur.ai/free/en/papers/chen-2023-internvl-scaling-up-vision/)
- [MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning](https://md.research.elamur.ai/free/en/papers/chen-2023-minigpt-v2-large-language-model/)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://md.research.elamur.ai/free/en/papers/chen-2023-sharegpt4v-improving-large-multi-modal/)
- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://md.research.elamur.ai/free/en/papers/chen-2023-shikra-unleashing-multimodal-llm-s/)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://md.research.elamur.ai/free/en/papers/chen-2024-far-gpt-4v-closing/)
- [An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models](https://md.research.elamur.ai/free/en/papers/chen-2024-image-worth-1-2-tokens/)
- [Are We on the Right Way for Evaluating Large Vision-Language Models?](https://md.research.elamur.ai/free/en/papers/chen-2024-right-way-evaluating-large-vision/)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://md.research.elamur.ai/free/en/papers/chu-2023-qwen-audio-advancing-universal/)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/)
- [MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response](https://md.research.elamur.ai/free/en/papers/deng-2024-musilingo-bridging-music-text-pre/)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://md.research.elamur.ai/free/en/papers/fu-2023-mme-comprehensive-evaluation-benchmark-multimodal/)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://md.research.elamur.ai/free/en/papers/fu-2024-video-mme-first-ever-comprehensive/)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/gao-2023-retrieval-augmented-generation-large-language/)
- [HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models](https://md.research.elamur.ai/free/en/papers/guan-2023-hallusionbench-advanced-diagnostic-suite-entangled/)
- [CogAgent: A Visual Language Model for GUI Agents](https://md.research.elamur.ai/free/en/papers/hong-2023-cogagent-visual-language-model/)
- [Language Is Not All You Need: Aligning Perception with Language Models](https://md.research.elamur.ai/free/en/papers/huang-2023-language-all-need-aligning-perception/)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://md.research.elamur.ai/free/en/papers/huang-2023-survey-hallucination-large-language-models/)
- [VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks](https://md.research.elamur.ai/free/en/papers/koh-2024-visualwebarena-evaluating-multimodal-agents-realistic/)
- [OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents](https://md.research.elamur.ai/free/en/papers/lauren-2023-ccon-obelics-open-web-scale-filtered/)
- [What matters when building vision-language models?](https://md.research.elamur.ai/free/en/papers/lauren-2024-ccon-matters-building-vision-language-models/)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://md.research.elamur.ai/free/en/papers/li-2023-evaluating-object-hallucination-large-vision/)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://md.research.elamur.ai/free/en/papers/li-2023-mvbench-comprehensive-multi-modal-video/)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://md.research.elamur.ai/free/en/papers/li-2023-seed-bench-benchmarking-multimodal-llms/)
- [VideoChat: Chat-Centric Video Understanding](https://md.research.elamur.ai/free/en/papers/li-2023-videochat-chat-centric-video-understanding/)
- [Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models](https://md.research.elamur.ai/free/en/papers/li-2024-multimodal-arxiv-dataset-improving-scientific/)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://md.research.elamur.ai/free/en/papers/lin-2023-video-llava-learning-united-visual/)
- [VILA: On Pre-training for Visual Language Models](https://md.research.elamur.ai/free/en/papers/lin-2023-vila-pre-training-visual-language/)
- [Improved Baselines with Visual Instruction Tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/)
- [Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-mitigating-hallucination-large-multi-modal/)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://md.research.elamur.ai/free/en/papers/liu-2023-mmbench-multi-modal-model-all/)
- [OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models](https://md.research.elamur.ai/free/en/papers/liu-2023-ocrbench-hidden-mystery-ocr-large/)
- [Visual Instruction Tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://md.research.elamur.ai/free/en/papers/lu-2023-mathvista-evaluating-mathematical-reasoning-foundation/)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://md.research.elamur.ai/free/en/papers/maaz-2023-video-chatgpt-towards-detailed-video/)
- [Evaluating Discriminability of Vision-Language Models](https://md.research.elamur.ai/free/en/papers/muraoka-2026-evaluating-discriminability-vision-language-models/)
- [A Comprehensive Overview of Large Language Models](https://md.research.elamur.ai/free/en/papers/naveed-2023-comprehensive-overview-large-language-models/)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://md.research.elamur.ai/free/en/papers/peng-2023-kosmos-2-grounding-multimodal-large/)
- [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://md.research.elamur.ai/free/en/papers/shen-2023-hugginggpt-solving-ai-tasks-chatgpt/)
- [Aligning Large Multimodal Models with Factually Augmented RLHF](https://md.research.elamur.ai/free/en/papers/sun-2023-aligning-large-multimodal-models-factually/)
- [ViperGPT: Visual Inference via Python Execution for Reasoning](https://md.research.elamur.ai/free/en/papers/suris-2023-vipergpt-visual-inference-python-execution/)
- [SALMONN: Towards Generic Hearing Abilities for Large Language Models](https://md.research.elamur.ai/free/en/papers/tang-2023-salmonn-towards-generic-hearing-abilities/)
- [CogVLM: Visual Expert for Pretrained Language Models](https://md.research.elamur.ai/free/en/papers/wang-2023-cogvlm-visual-expert-pretrained/)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://md.research.elamur.ai/free/en/papers/wang-2024-qwen2-vl-enhancing-vision/)
- [基于大模型的具身智能系统综述](https://md.research.elamur.ai/free/en/papers/wang-2025-embodied-intelligence-systems/)
- [Frame2KG: A Benchmark and Evaluation Toolkit for Interpretable Frame-to-Graph Generation](https://md.research.elamur.ai/free/en/papers/watson-2026-frame2kg-benchmark-evaluation-toolkit-interpretable/)
- [Do Multimodal LLMs Understand Order? Measuring the Fragility of Multimodal Reasoning under Input Order Perturbations](https://md.research.elamur.ai/free/en/papers/wei-2026-do-multimodal-llms-understand-order/)
- [Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models](https://md.research.elamur.ai/free/en/papers/wu-2023-visual-chatgpt-talking-drawing-editing/)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://md.research.elamur.ai/free/en/papers/xi-2023-rise-potential-large-language/)
- [Knowledge Conflicts for LLMs: A Survey](https://md.research.elamur.ai/free/en/papers/xu-2024-knowledge-conflicts-llms-survey/)
- [Qwen2.5-Omni Technical Report](https://md.research.elamur.ai/free/en/papers/xu-2025-qwen2-5-omni-technical/)
- [Qwen3-Omni Technical Report](https://md.research.elamur.ai/free/en/papers/xu-2025-qwen3-omni-technical-report/)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://md.research.elamur.ai/free/en/papers/yang-2023-dawn-lmms-preliminary-explorations-gpt/)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://md.research.elamur.ai/free/en/papers/yao-2024-minicpm-v-gpt-4v/)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://md.research.elamur.ai/free/en/papers/ye-2023-mplug-owl-modularization-empowers-large/)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://md.research.elamur.ai/free/en/papers/yu-2023-mm-vet-evaluating-large-multimodal/)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/)
- [Instruction Tuning for Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/zhang-2023-instruction-tuning-large-language-models/)
- [LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention](https://md.research.elamur.ai/free/en/papers/zhang-2023-llama-adapter-efficient-fine-tuning/)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://md.research.elamur.ai/free/en/papers/zhang-2023-video-llama-instruction-tuned-audio/)
- [GPT-4V(ision) is a Generalist Web Agent, if Grounded](https://md.research.elamur.ai/free/en/papers/zheng-2024-gpt-4v-ision-generalist-web/)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/)
- [SPARTA: Evaluating Reasoning Segmentation Robustness through Black-Box Adversarial Paraphrasing in Text Autoencoder Latent Space](https://md.research.elamur.ai/free/en/papers/zinkovich-2026-sparta-evaluating-reasoning-segmentation-robustness/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{li2023blip,
  title = {{BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models}},
  author = {Li, Junnan and Li, Dongxu and Savarese, Silvio and Hoi, Steven},
  year = {2023},
  doi = {10.48550/arxiv.2301.12597},
  eprint = {2301.12597},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2301.12597}
}
```

- [← previous Bibliography of Ethiopian Semitic, Cushitic, and Omotic Linguistics XXV: 2021](https://md.research.elamur.ai/free/en/papers/maria-2023-bibliography-ethiopian-semitic-cushitic-omotic/)
- [next → BloombergGPT: A Large Language Model for Finance](https://md.research.elamur.ai/free/en/papers/wu-2023-bloomberggpt-large-language-model-finance/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/2/)

## References

46 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Agrawal, H., Anderson, P., Desai, K., Wang, Y., Chen, X., Jain, R., Johnson, M., Batra, D., Parikh, D., and Lee, S. nocaps: novel object captioning at scale. In ICCV, pp. 8947–8956, 2019. [find](https://scholar.google.com/scholar?q=Agrawal%2C+H%2E%2C+Anderson%2C+P%2E%2C+Desai%2C+K%2E%2C+Wang%2C+Y%2E%2C+Chen%2C+X%2E%2C+Jain%2C+R%2E%2C+Johnson%2C+M%2E%2C+Batra%2C+D%2E%2C+Parikh%2C+D%2E%2C+and+Lee%2C+S%2E+nocaps%3A+novel+object+captioning+at+scale%2E+In+ICCV%2C+pp%2E+8947%E2%80%938956%2C+2019%2E)
2. Alayrac, J., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M., Barreira, R.… [find](https://scholar.google.com/scholar?q=Alayrac%2C+J%2E%2C+Donahue%2C+J%2E%2C+Luc%2C+P%2E%2C+Miech%2C+A%2E%2C+Barr%2C+I%2E%2C+Hasson%2C+Y%2E%2C+Lenc%2C+K%2E%2C+Mensch%2C+A%2E%2C+Millican%2C+K%2E%2C+Reynolds%2C+M%2E%2C+Ring%2C+R%2E%2C+Rutherford%2C+E%2E%2C+Cabi%2C+S%2E%2C+Han%2C+T%2E%2C+Gong%2C+Z%2E%2C+Samangooei%2C+S%2E%2C+Monteiro%2C+M) [our card](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/)
3. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Ch… [find](https://scholar.google.com/scholar?q=Brown%2C+T%2E+B%2E%2C+Mann%2C+B%2E%2C+Ryder%2C+N%2E%2C+Subbiah%2C+M%2E%2C+Kaplan%2C+J%2E%2C+Dhariwal%2C+P%2E%2C+Neelakantan%2C+A%2E%2C+Shyam%2C+P%2E%2C+Sastry%2C+G%2E%2C+Askell%2C+A%2E%2C+Agarwal%2C+S%2E%2C+Herbert%2DVoss%2C+A%2E%2C+Krueger%2C+G%2E%2C+Henighan%2C+T%2E%2C+Child%2C+R%2E%2C+Rames)
4. Changpinyo, S., Sharma, P., Ding, N., and Soricut, R. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021. [find](https://scholar.google.com/scholar?q=Changpinyo%2C+S%2E%2C+Sharma%2C+P%2E%2C+Ding%2C+N%2E%2C+and+Soricut%2C+R%2E+Conceptual+12M%3A+Pushing+web%2Dscale+image%2Dtext+pre%2Dtraining+to+recognize+long%2Dtail+visual+concepts%2E+In+CVPR%2C+2021%2E)
5. Webson, A., Gu, S. S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Narang, S., Mishra, G., Yu, A., Zhao, V. Y., Huang, Y., Dai, A. M., Yu, H., Petrov, S., Chi, E. H., Dean, J., Devlin, J., Roberts, A., Zhou, D., Le, Q. V., and Wei, J. Scaling instruction-finetuned language models. arXiv preprint a… [find](https://scholar.google.com/scholar?q=Webson%2C+A%2E%2C+Gu%2C+S%2E+S%2E%2C+Dai%2C+Z%2E%2C+Suzgun%2C+M%2E%2C+Chen%2C+X%2E%2C+Chowdhery%2C+A%2E%2C+Narang%2C+S%2E%2C+Mishra%2C+G%2E%2C+Yu%2C+A%2E%2C+Zhao%2C+V%2E+Y%2E%2C+Huang%2C+Y%2E%2C+Dai%2C+A%2E+M%2E%2C+Yu%2C+H%2E%2C+Petrov%2C+S%2E%2C+Chi%2C+E%2E+H%2E%2C+Dean%2C+J%2E%2C+Devlin%2C+J%2E%2C+Roberts%2C+)
6. Dai, W., Hou, L., Shang, L., Jiang, X., Liu, Q., and Fung, P. Enabling multimodal generation on CLIP via vision-language knowledge distillation. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), ACL Findings, pp. 2383– 2395, 2022. [find](https://scholar.google.com/scholar?q=Dai%2C+W%2E%2C+Hou%2C+L%2E%2C+Shang%2C+L%2E%2C+Jiang%2C+X%2E%2C+Liu%2C+Q%2E%2C+and+Fung%2C+P%2E+Enabling+multimodal+generation+on+CLIP+via+vision%2Dlanguage+knowledge+distillation%2E+In+Muresan%2C+S%2E%2C+Nakov%2C+P%2E%2C+and+Villavicencio%2C+A%2E+%28eds%2E%29)
7. Devlin, J., Chang, M., Lee, K., and Toutanova, K. BERT: pre-training of deep bidirectional transformers for language understanding. In Burstein, J., Doran, C., and Solorio, T. (eds.), NAACL, pp. 4171–4186, 2019. [find](https://scholar.google.com/scholar?q=Devlin%2C+J%2E%2C+Chang%2C+M%2E%2C+Lee%2C+K%2E%2C+and+Toutanova%2C+K%2E+BERT%3A+pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Burstein%2C+J%2E%2C+Doran%2C+C%2E%2C+and+Solorio%2C+T%2E+%28eds%2E%29%2C+NAACL%2C+pp%2E+4171%E2%80%93)
8. Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., and Hon, H. Unified language model pre-training for natural language understanding and generation. In Wallach, H. M., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E. B., and Garnett, R. (eds.), NeurIPS, pp. 13042… [find](https://scholar.google.com/scholar?q=Dong%2C+L%2E%2C+Yang%2C+N%2E%2C+Wang%2C+W%2E%2C+Wei%2C+F%2E%2C+Liu%2C+X%2E%2C+Wang%2C+Y%2E%2C+Gao%2C+J%2E%2C+Zhou%2C+M%2E%2C+and+Hon%2C+H%2E+Unified+language+model+pre%2Dtraining+for+natural+language+understanding+and+generation%2E+In+Wallach%2C+H%2E+M%2E%2C+Laroc) [our card](https://md.research.elamur.ai/free/en/papers/dong-2019-unified-language-model-pre-training/)
9. Fang, Y., Wang, W., Xie, B., Sun, Q., Wu, L., Wang, X., Huang, T., Wang, X., and Cao, Y. Eva: Exploring the limits of masked visual representation learning at scale. arXiv preprint arXiv:2211.07636, 2022. — [arXiv:2211.07636](https://arxiv.org/abs/2211.07636) [find](https://scholar.google.com/scholar?q=Fang%2C+Y%2E%2C+Wang%2C+W%2E%2C+Xie%2C+B%2E%2C+Sun%2C+Q%2E%2C+Wu%2C+L%2E%2C+Wang%2C+X%2E%2C+Huang%2C+T%2E%2C+Wang%2C+X%2E%2C+and+Cao%2C+Y%2E+Eva%3A+Exploring+the+limits+of+masked+visual+representation+learning+at+scale%2E+arXiv+preprint+arXiv%3A2211%2E07636%2C+2)
10. Chen, J., Guo, H., Yi, K., Li, B., and Elhoseiny, M. Visu-algpt: Data-efficient adaptation of pretrained language models for image captioning. In CVPR, pp. 18009–18019, 2022a. [find](https://scholar.google.com/scholar?q=Chen%2C+J%2E%2C+Guo%2C+H%2E%2C+Yi%2C+K%2E%2C+Li%2C+B%2E%2C+and+Elhoseiny%2C+M%2E+Visu%2Dalgpt%3A+Data%2Defficient+adaptation+of+pretrained+language+models+for+image+captioning%2E+In+CVPR%2C+pp%2E+18009%E2%80%9318019%2C+2022a%2E)
11. Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In CVPR, pp. 6325–6334, 2017. [find](https://scholar.google.com/scholar?q=Goyal%2C+Y%2E%2C+Khot%2C+T%2E%2C+Summers%2DStay%2C+D%2E%2C+Batra%2C+D%2E%2C+and+Parikh%2C+D%2E+Making+the+V+in+VQA+matter%3A+Elevating+the+role+of+image+understanding+in+visual+question+answering%2E+In+CVPR%2C+pp%2E+6325%E2%80%936334%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/)
12. Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A. J., Padlewski, P., Salz, D., Goodman, S., Grycner, A., Mustafa, B., Beyer, L., Kolesnikov, A., Puigcerver, J., Ding, N., Rong, K., Akbari, H., Mishra, G., Xue, L., Thapliyal, A., Bradbury, J., Kuo, W., Seyedhosseini, M., Jia, C., Ayan, B. K., Riqu… [find](https://scholar.google.com/scholar?q=Chen%2C+X%2E%2C+Wang%2C+X%2E%2C+Changpinyo%2C+S%2E%2C+Piergiovanni%2C+A%2E+J%2E%2C+Padlewski%2C+P%2E%2C+Salz%2C+D%2E%2C+Goodman%2C+S%2E%2C+Grycner%2C+A%2E%2C+Mustafa%2C+B%2E%2C+Beyer%2C+L%2E%2C+Kolesnikov%2C+A%2E%2C+Puigcerver%2C+J%2E%2C+Ding%2C+N%2E%2C+Rong%2C+K%2E%2C+Akbari%2C+H%2E%2C+Mish) [our card](https://md.research.elamur.ai/free/en/papers/chen-2022-pali-jointly-scaled-multilingual-language/)
13. Guo, J., Li, J., Li, D., Tiong, A. M. H., Li, B., Tao, D., and Hoi, S. C. H. From images to textual prompts: Zero-shot VQA with frozen large language models. In CVPR, 2022. [find](https://scholar.google.com/scholar?q=Guo%2C+J%2E%2C+Li%2C+J%2E%2C+Li%2C+D%2E%2C+Tiong%2C+A%2E+M%2E+H%2E%2C+Li%2C+B%2E%2C+Tao%2C+D%2E%2C+and+Hoi%2C+S%2E+C%2E+H%2E+From+images+to+textual+prompts%3A+Zero%2Dshot+VQA+with+frozen+large+language+models%2E+In+CVPR%2C+2022%2E)
14. Chen, Y., Li, L., Yu, L., Kholy, A. E., Ahmed, F., Gan, Z., Cheng, Y., and Liu, J. UNITER: universal image-text representation learning. In ECCV, volume 12375, pp. 104–120, 2020. [find](https://scholar.google.com/scholar?q=Chen%2C+Y%2E%2C+Li%2C+L%2E%2C+Yu%2C+L%2E%2C+Kholy%2C+A%2E+E%2E%2C+Ahmed%2C+F%2E%2C+Gan%2C+Z%2E%2C+Cheng%2C+Y%2E%2C+and+Liu%2C+J%2E+UNITER%3A+universal+image%2Dtext+representation+learning%2E+In+ECCV%2C+volume+12375%2C+pp%2E+104%E2%80%93120%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2019-uniter-universal-image-text-representation/)
15. Cho, J., Lei, J., Tan, H., and Bansal, M. Unifying vision-and-language tasks via text generation. arXiv preprint arXiv:2102.02779, 2021. — [arXiv:2102.02779](https://arxiv.org/abs/2102.02779) [find](https://scholar.google.com/scholar?q=Cho%2C+J%2E%2C+Lei%2C+J%2E%2C+Tan%2C+H%2E%2C+and+Bansal%2C+M%2E+Unifying+vision%2Dand%2Dlanguage+tasks+via+text+generation%2E+arXiv+preprint+arXiv%3A2102%2E02779%2C+2021%2E+%E2%80%94+%5BarXiv%3A2102%2E02779%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2102%2E02779%29)
16. Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., Driessche,… [find](https://scholar.google.com/scholar?q=Chung%2C+H%2E+W%2E%2C+Hou%2C+L%2E%2C+Longpre%2C+S%2E%2C+Zoph%2C+B%2E%2C+Tay%2C+Y%2E%2C+Fedus%2C+W%2E%2C+Li%2C+E%2E%2C+Wang%2C+X%2E%2C+Dehghani%2C+M%2E%2C+Brahma%2C+S%2E%2C+Hoffmann%2C+J%2E%2C+Borgeaud%2C+S%2E%2C+Mensch%2C+A%2E%2C+Buchatskaya%2C+E%2E%2C+Cai%2C+T%2E%2C+Rutherford%2C+E%2E%2C+Casas%2C+D)
17. Hudson, D. A. and Manning, C. D. GQA: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, pp. 6700–6709, 2019. [find](https://scholar.google.com/scholar?q=Hudson%2C+D%2E+A%2E+and+Manning%2C+C%2E+D%2E+GQA%3A+A+new+dataset+for+real%2Dworld+visual+reasoning+and+compositional+question+answering%2E+In+CVPR%2C+pp%2E+6700%E2%80%936709%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/hudson-2019-gqa-new-dataset-real-world/)
18. Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., and Duerig, T. Scaling up visual and vision-language representation learning with noisy text supervision. arXiv preprint arXiv:2102.05918, 2021. [find](https://scholar.google.com/scholar?q=Jia%2C+C%2E%2C+Yang%2C+Y%2E%2C+Xia%2C+Y%2E%2C+Chen%2C+Y%2E%2DT%2E%2C+Parekh%2C+Z%2E%2C+Pham%2C+H%2E%2C+Le%2C+Q%2E+V%2E%2C+Sung%2C+Y%2E%2C+Li%2C+Z%2E%2C+and+Duerig%2C+T%2E+Scaling+up+visual+and+vision%2Dlanguage+representation+learning+with+noisy+text+supervision%2E+ar) [our card](https://md.research.elamur.ai/free/en/papers/jia-2021-scaling-up-visual-vision-language/)
19. Jin, W., Cheng, Y., Shen, Y., Chen, W., and Ren, X. A good prompt is worth millions of parameters: Low-resource prompt-based learning for vision-language models. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), ACL, pp. 2763–2775, 2022. [find](https://scholar.google.com/scholar?q=Jin%2C+W%2E%2C+Cheng%2C+Y%2E%2C+Shen%2C+Y%2E%2C+Chen%2C+W%2E%2C+and+Ren%2C+X%2E+A+good+prompt+is+worth+millions+of+parameters%3A+Low%2Dresource+prompt%2Dbased+learning+for+vision%2Dlanguage+models%2E+In+Muresan%2C+S%2E%2C+Nakov%2C+P%2E%2C+and+Villavi)
20. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020, 2021. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Kim%2C+J%2E+W%2E%2C+Hallacy%2C+C%2E%2C+Ramesh%2C+A%2E%2C+Goh%2C+G%2E%2C+Agarwal%2C+S%2E%2C+Sastry%2C+G%2E%2C+Askell%2C+A%2E%2C+Mishkin%2C+P%2E%2C+Clark%2C+J%2E%2C+et+al%2E+Learning+transferable+visual+models+from+natural+language+supervision%2E+ar) [our card](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/)
21. Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L., Shamma, D. A., Bernstein, M. S., and Fei-Fei, L. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 123(1):32–73, 2017. [find](https://scholar.google.com/scholar?q=Krishna%2C+R%2E%2C+Zhu%2C+Y%2E%2C+Groth%2C+O%2E%2C+Johnson%2C+J%2E%2C+Hata%2C+K%2E%2C+Kravitz%2C+J%2E%2C+Chen%2C+S%2E%2C+Kalantidis%2C+Y%2E%2C+Li%2C+L%2E%2C+Shamma%2C+D%2E+A%2E%2C+Bernstein%2C+M%2E+S%2E%2C+and+Fei%2DFei%2C+L%2E+Visual+genome%3A+Connecting+language+and+vision+us) [our card](https://md.research.elamur.ai/free/en/papers/krishna-2016-visual-genome-connecting-language-vision/)
22. Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021. [find](https://scholar.google.com/scholar?q=Schuhmann%2C+C%2E%2C+Vencu%2C+R%2E%2C+Beaumont%2C+R%2E%2C+Kaczmarczyk%2C+R%2E%2C+Mullis%2C+C%2E%2C+Katta%2C+A%2E%2C+Coombes%2C+T%2E%2C+Jitsev%2C+J%2E%2C+and+Komatsuzaki%2C+A%2E+Laion%2D400m%3A+Open+dataset+of+clip%2Dfiltered+400+million+image%2Dtext+pairs%2E+arX) [our card](https://md.research.elamur.ai/free/en/papers/schuhmann-2021-laion-400m-open-dataset-clip/)
23. Li, J., Selvaraju, R. R., Gotmare, A. D., Joty, S., Xiong, C., and Hoi, S. Align before fuse: Vision and language representation learning with momentum distillation. In NeurIPS, 2021. [find](https://scholar.google.com/scholar?q=Li%2C+J%2E%2C+Selvaraju%2C+R%2E+R%2E%2C+Gotmare%2C+A%2E+D%2E%2C+Joty%2C+S%2E%2C+Xiong%2C+C%2E%2C+and+Hoi%2C+S%2E+Align+before+fuse%3A+Vision+and+language+representation+learning+with+momentum+distillation%2E+In+NeurIPS%2C+2021%2E)
24. Sharma, P., Ding, N., Goodman, S., and Soricut, R. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Gurevych, I. and Miyao, Y. (eds.), ACL, pp. 2556–2565, 2018. [find](https://scholar.google.com/scholar?q=Sharma%2C+P%2E%2C+Ding%2C+N%2E%2C+Goodman%2C+S%2E%2C+and+Soricut%2C+R%2E+Conceptual+captions%3A+A+cleaned%2C+hypernymed%2C+image+alt%2Dtext+dataset+for+automatic+image+captioning%2E+In+Gurevych%2C+I%2E+and+Miyao%2C+Y%2E+%28eds%2E%29%2C+ACL%2C+pp%2E+255) [our card](https://md.research.elamur.ai/free/en/papers/sharma-2018-conceptual-captions-cleaned-hypernymed-image/)
25. Li, J., Li, D., Xiong, C., and Hoi, S. C. H. BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, pp. 12888–12900, 2022. [find](https://scholar.google.com/scholar?q=Li%2C+J%2E%2C+Li%2C+D%2E%2C+Xiong%2C+C%2E%2C+and+Hoi%2C+S%2E+C%2E+H%2E+BLIP%3A+bootstrapping+language%2Dimage+pre%2Dtraining+for+unified+vision%2Dlanguage+understanding+and+generation%2E+In+ICML%2C+pp%2E+12888%E2%80%9312900%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2022-blip-bootstrapping-language-image-pre/)
26. Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., Wei, F., Choi, Y., and Gao, J. Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV, pp. 121–137, 2020. [find](https://scholar.google.com/scholar?q=Li%2C+X%2E%2C+Yin%2C+X%2E%2C+Li%2C+C%2E%2C+Zhang%2C+P%2E%2C+Hu%2C+X%2E%2C+Zhang%2C+L%2E%2C+Wang%2C+L%2E%2C+Hu%2C+H%2E%2C+Dong%2C+L%2E%2C+Wei%2C+F%2E%2C+Choi%2C+Y%2E%2C+and+Gao%2C+J%2E+Oscar%3A+Object%2Dsemantics+aligned+pre%2Dtraining+for+vision%2Dlanguage+tasks%2E+In+ECCV%2C+pp%2E+1) [our card](https://md.research.elamur.ai/free/en/papers/li-2020-oscar-object-semantics-aligned-pre/)
27. Lin, T., Maire, M., Belongie, S. J., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft COCO: common objects in context. In Fleet, D. J., Pajdla, T., Schiele, B., and Tuytelaars, T. (eds.), ECCV, volume 8693, pp. 740–755, 2014. [find](https://scholar.google.com/scholar?q=Lin%2C+T%2E%2C+Maire%2C+M%2E%2C+Belongie%2C+S%2E+J%2E%2C+Hays%2C+J%2E%2C+Perona%2C+P%2E%2C+Ramanan%2C+D%2E%2C+Dolla%CC%81r%2C+P%2E%2C+and+Zitnick%2C+C%2E+L%2E+Microsoft+COCO%3A+common+objects+in+context%2E+In+Fleet%2C+D%2E+J%2E%2C+Pajdla%2C+T%2E%2C+Schiele%2C+B%2E%2C+and+Tuytela) [our card](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/)
28. Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. [find](https://scholar.google.com/scholar?q=Loshchilov%2C+I%2E+and+Hutter%2C+F%2E+Decoupled+weight+decay+regularization%2E+arXiv+preprint+arXiv%3A1711%2E05101%2C+2017%2E)
29. Mañas, O., Rodrı́guez, P., Ahmadi, S., Nematzadeh, A., Goyal, Y., and Agrawal, A. MAPL: parameter-efficient adaptation of unimodal pre-trained models for vision-language few-shot prompting. In EACL, 2023. [find](https://scholar.google.com/scholar?q=Man%CC%83as%2C+O%2E%2C+Rodr%C4%B1%CC%81guez%2C+P%2E%2C+Ahmadi%2C+S%2E%2C+Nematzadeh%2C+A%2E%2C+Goyal%2C+Y%2E%2C+and+Agrawal%2C+A%2E+MAPL%3A+parameter%2Defficient+adaptation+of+unimodal+pre%2Dtrained+models+for+vision%2Dlanguage+few%2Dshot+prompting%2E+In+EACL%2C+)
30. Marino, K., Rastegari, M., Farhadi, A., and Mottaghi, R. Okvqa: A visual question answering benchmark requiring external knowledge. In CVPR, 2019. [find](https://scholar.google.com/scholar?q=Marino%2C+K%2E%2C+Rastegari%2C+M%2E%2C+Farhadi%2C+A%2E%2C+and+Mottaghi%2C+R%2E+Okvqa%3A+A+visual+question+answering+benchmark+requiring+external+knowledge%2E+In+CVPR%2C+2019%2E)
31. Ordonez, V., Kulkarni, G., and Berg, T. L. Im2text: Describing images using 1 million captioned photographs. In Shawe-Taylor, J., Zemel, R. S., Bartlett, P. L., Pereira, F. C. N., and Weinberger, K. Q. (eds.), NIPS, pp. 1143–1151, 2011. [find](https://scholar.google.com/scholar?q=Ordonez%2C+V%2E%2C+Kulkarni%2C+G%2E%2C+and+Berg%2C+T%2E+L%2E+Im2text%3A+Describing+images+using+1+million+captioned+photographs%2E+In+Shawe%2DTaylor%2C+J%2E%2C+Zemel%2C+R%2E+S%2E%2C+Bartlett%2C+P%2E+L%2E%2C+Pereira%2C+F%2E+C%2E+N%2E%2C+and+Weinberger%2C+K%2E+Q)
32. Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., and Lazebnik, S. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In ICCV, pp. 2641–2649, 2015. [find](https://scholar.google.com/scholar?q=Plummer%2C+B%2E+A%2E%2C+Wang%2C+L%2E%2C+Cervantes%2C+C%2E+M%2E%2C+Caicedo%2C+J%2E+C%2E%2C+Hockenmaier%2C+J%2E%2C+and+Lazebnik%2C+S%2E+Flickr30k+entities%3A+Collecting+region%2Dto%2Dphrase+correspondences+for+richer+image%2Dto%2Dsentence+models%2E+In+IC)
33. Tan, H. and Bansal, M. LXMERT: learning cross-modality encoder representations from transformers. In Inui, K., Jiang, J., Ng, V., and Wan, X. (eds.), EMNLP, pp. 5099– 5110, 2019. [find](https://scholar.google.com/scholar?q=Tan%2C+H%2E+and+Bansal%2C+M%2E+LXMERT%3A+learning+cross%2Dmodality+encoder+representations+from+transformers%2E+In+Inui%2C+K%2E%2C+Jiang%2C+J%2E%2C+Ng%2C+V%2E%2C+and+Wan%2C+X%2E+%28eds%2E%29%2C+EMNLP%2C+pp%2E+5099%E2%80%93+5110%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/tan-2019-lxmert-learning-cross-modality-encoder/)
34. Tiong, A. M. H., Li, J., Li, B., Savarese, S., and Hoi, S. C. H. Plug-and-play VQA: zero-shot VQA by conjoining large pretrained models with zero training. In EMNLP Findings, 2022. [find](https://scholar.google.com/scholar?q=Tiong%2C+A%2E+M%2E+H%2E%2C+Li%2C+J%2E%2C+Li%2C+B%2E%2C+Savarese%2C+S%2E%2C+and+Hoi%2C+S%2E+C%2E+H%2E+Plug%2Dand%2Dplay+VQA%3A+zero%2Dshot+VQA+by+conjoining+large+pretrained+models+with+zero+training%2E+In+EMNLP+Findings%2C+2022%2E)
35. Tsimpoukelli, M., Menick, J., Cabi, S., Eslami, S. M. A., Vinyals, O., and Hill, F. Multimodal few-shot learning with frozen language models. In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), NeurIPS, pp. 200–212, 2021. [find](https://scholar.google.com/scholar?q=Tsimpoukelli%2C+M%2E%2C+Menick%2C+J%2E%2C+Cabi%2C+S%2E%2C+Eslami%2C+S%2E+M%2E+A%2E%2C+Vinyals%2C+O%2E%2C+and+Hill%2C+F%2E+Multimodal+few%2Dshot+learning+with+frozen+language+models%2E+In+Ranzato%2C+M%2E%2C+Beygelzimer%2C+A%2E%2C+Dauphin%2C+Y%2E+N%2E%2C+Liang%2C+P%2E) [our card](https://md.research.elamur.ai/free/en/papers/tsimpoukelli-2021-multimodal-few-shot-learning-frozen/)
36. Wang, P., Yang, A., Men, R., Lin, J., Bai, S., Li, Z., Ma, J., Zhou, C., Zhou, J., and Yang, H. OFA: unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvári, C., Niu, G., and Sabato, S. (eds.), ICML,… [find](https://scholar.google.com/scholar?q=Wang%2C+P%2E%2C+Yang%2C+A%2E%2C+Men%2C+R%2E%2C+Lin%2C+J%2E%2C+Bai%2C+S%2E%2C+Li%2C+Z%2E%2C+Ma%2C+J%2E%2C+Zhou%2C+C%2E%2C+Zhou%2C+J%2E%2C+and+Yang%2C+H%2E+OFA%3A+unifying+architectures%2C+tasks%2C+and+modalities+through+a+simple+sequence%2Dto%2Dsequence+learning+framew) [our card](https://md.research.elamur.ai/free/en/papers/wang-2022-ofa-unifying-architectures-tasks-modalities/)
37. Wang, W., Bao, H., Dong, L., and Wei, F. Vlmo: Unified vision-language pre-training with mixture-of-modality-experts. arXiv preprint arXiv:2111.02358, 2021a. — [arXiv:2111.02358](https://arxiv.org/abs/2111.02358) [find](https://scholar.google.com/scholar?q=Wang%2C+W%2E%2C+Bao%2C+H%2E%2C+Dong%2C+L%2E%2C+and+Wei%2C+F%2E+Vlmo%3A+Unified+vision%2Dlanguage+pre%2Dtraining+with+mixture%2Dof%2Dmodality%2Dexperts%2E+arXiv+preprint+arXiv%3A2111%2E02358%2C+2021a%2E+%E2%80%94+%5BarXiv%3A2111%2E02358%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs)
38. Wang, W., Bao, H., Dong, L., Bjorck, J., Peng, Z., Liu, Q., Aggarwal, K., Mohammed, O. K., Singhal, S., Som, S., and Wei, F. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442, 2022b. — [arXiv:2208.10442](https://arxiv.org/abs/2208.10442) [find](https://scholar.google.com/scholar?q=Wang%2C+W%2E%2C+Bao%2C+H%2E%2C+Dong%2C+L%2E%2C+Bjorck%2C+J%2E%2C+Peng%2C+Z%2E%2C+Liu%2C+Q%2E%2C+Aggarwal%2C+K%2E%2C+Mohammed%2C+O%2E+K%2E%2C+Singhal%2C+S%2E%2C+Som%2C+S%2E%2C+and+Wei%2C+F%2E+Image+as+a+foreign+language%3A+Beit+pretraining+for+all+vision+and+vision%2Dlan)
39. Wang, Z., Yu, J., Yu, A. W., Dai, Z., Tsvetkov, Y., and Cao, Y. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108.10904, 2021b. [find](https://scholar.google.com/scholar?q=Wang%2C+Z%2E%2C+Yu%2C+J%2E%2C+Yu%2C+A%2E+W%2E%2C+Dai%2C+Z%2E%2C+Tsvetkov%2C+Y%2E%2C+and+Cao%2C+Y%2E+Simvlm%3A+Simple+visual+language+model+pretraining+with+weak+supervision%2E+arXiv+preprint+arXiv%3A2108%2E10904%2C+2021b%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2021-simvlm-simple-visual-language-model/)
40. Yao, L., Huang, R., Hou, L., Lu, G., Niu, M., Xu, H., Liang, X., Li, Z., Jiang, X., and Xu, C. FILIP: fine-grained interactive language-image pre-training. In ICLR, 2022. [find](https://scholar.google.com/scholar?q=Yao%2C+L%2E%2C+Huang%2C+R%2E%2C+Hou%2C+L%2E%2C+Lu%2C+G%2E%2C+Niu%2C+M%2E%2C+Xu%2C+H%2E%2C+Liang%2C+X%2E%2C+Li%2C+Z%2E%2C+Jiang%2C+X%2E%2C+and+Xu%2C+C%2E+FILIP%3A+fine%2Dgrained+interactive+language%2Dimage+pre%2Dtraining%2E+In+ICLR%2C+2022%2E)
41. Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., and Wu, Y. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022. [find](https://scholar.google.com/scholar?q=Yu%2C+J%2E%2C+Wang%2C+Z%2E%2C+Vasudevan%2C+V%2E%2C+Yeung%2C+L%2E%2C+Seyedhosseini%2C+M%2E%2C+and+Wu%2C+Y%2E+Coca%3A+Contrastive+captioners+are+image%2Dtext+foundation+models%2E+arXiv+preprint+arXiv%3A2205%2E01917%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/yu-2022-coca-contrastive-captioners-image-text/)
42. Yuan, L., Chen, D., Chen, Y., Codella, N., Dai, X., Gao, J., Hu, H., Huang, X., Li, B., Li, C., Liu, C., Liu, M., Liu, Z., Lu, Y., Shi, Y., Wang, L., Wang, J., Xiao, B., Xiao, Z., Yang, J., Zeng, M., Zhou, L., and Zhang, P. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2… — [arXiv:2111.11432](https://arxiv.org/abs/2111.11432) [find](https://scholar.google.com/scholar?q=Yuan%2C+L%2E%2C+Chen%2C+D%2E%2C+Chen%2C+Y%2E%2C+Codella%2C+N%2E%2C+Dai%2C+X%2E%2C+Gao%2C+J%2E%2C+Hu%2C+H%2E%2C+Huang%2C+X%2E%2C+Li%2C+B%2E%2C+Li%2C+C%2E%2C+Liu%2C+C%2E%2C+Liu%2C+M%2E%2C+Liu%2C+Z%2E%2C+Lu%2C+Y%2E%2C+Shi%2C+Y%2E%2C+Wang%2C+L%2E%2C+Wang%2C+J%2E%2C+Xiao%2C+B%2E%2C+Xiao%2C+Z%2E%2C+Yang%2C+J%2E%2C+Zeng%2C+M%2E%2C+)
43. Zhai, X., Wang, X., Mustafa, B., Steiner, A., Keysers, D., Kolesnikov, A., and Beyer, L. Lit: Zero-shot transfer with locked-image text tuning. In CVPR, pp. 18102–18112, 2022. [find](https://scholar.google.com/scholar?q=Zhai%2C+X%2E%2C+Wang%2C+X%2E%2C+Mustafa%2C+B%2E%2C+Steiner%2C+A%2E%2C+Keysers%2C+D%2E%2C+Kolesnikov%2C+A%2E%2C+and+Beyer%2C+L%2E+Lit%3A+Zero%2Dshot+transfer+with+locked%2Dimage+text+tuning%2E+In+CVPR%2C+pp%2E+18102%E2%80%9318112%2C+2022%2E)
44. Zhang, P., Li, X., Hu, X., Yang, J., Zhang, L., Wang, L., Choi, Y., and Gao, J. Vinvl: Making visual representations matter in vision-language models. arXiv preprint arXiv:2101.00529, 2021. — [arXiv:2101.00529](https://arxiv.org/abs/2101.00529) [find](https://scholar.google.com/scholar?q=Zhang%2C+P%2E%2C+Li%2C+X%2E%2C+Hu%2C+X%2E%2C+Yang%2C+J%2E%2C+Zhang%2C+L%2E%2C+Wang%2C+L%2E%2C+Choi%2C+Y%2E%2C+and+Gao%2C+J%2E+Vinvl%3A+Making+visual+representations+matter+in+vision%2Dlanguage+models%2E+arXiv+preprint+arXiv%3A2101%2E00529%2C+2021%2E+%E2%80%94+%5BarXiv%3A2)
45. Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M. T., Li, X., Lin, X. V., Mihaylov, T., Ott, M., Shleifer, S., Shuster, K., Simig, D., Koura, P. S., Sridhar, A., Wang, T., and Zettlemoyer, L. OPT: open pre-trained transformer language models. arXiv preprint arXiv… [find](https://scholar.google.com/scholar?q=Zhang%2C+S%2E%2C+Roller%2C+S%2E%2C+Goyal%2C+N%2E%2C+Artetxe%2C+M%2E%2C+Chen%2C+M%2E%2C+Chen%2C+S%2E%2C+Dewan%2C+C%2E%2C+Diab%2C+M%2E+T%2E%2C+Li%2C+X%2E%2C+Lin%2C+X%2E+V%2E%2C+Mihaylov%2C+T%2E%2C+Ott%2C+M%2E%2C+Shleifer%2C+S%2E%2C+Shuster%2C+K%2E%2C+Simig%2C+D%2E%2C+Koura%2C+P%2E+S%2E%2C+Sridhar%2C+A%2E%2C+W)
46. LLM FlanT5 XL OPT 2.7B OPT 6.7B 5 1000 1e-5 256 (0.9,0.999) 0.05 0 364 “a photo of” 5 1 0.95 Fine-tuning epochs Warmup steps Learning rate Batch size AdamW β Weight decay Drop path Image resolution Prompt Inference beam size Layer-wise learning rate decay for ViT *Table 7. Hyperparameters for fine-t… [find](https://scholar.google.com/scholar?q=LLM+FlanT5+XL+OPT+2%2E7B+OPT+6%2E7B+5+1000+1e%2D5+256+%280%2E9%2C0%2E999%29+0%2E05+0+364+%E2%80%9Ca+photo+of%E2%80%9D+5+1+0%2E95+Fine%2Dtuning+epochs+Warmup+steps+Learning+rate+Batch+size+AdamW+%CE%B2+Weight+decay+Drop+path+Image+resolution+Pr)