--- title: "EVA-CLIP: Improved Training Techniques for CLIP at Scale (2023)" description: "Training CLIP-style image-text models at large scale is expensive and often unstable. 1.1 points. 4 references, cited by 105 papers here." authors: ["Quan Sun", "Yuxin Fang", "Ledell Wu", "Xinlong Wang", "Yue Cao"] year: 2023 venue: "arXiv:2303.15389" source: "https://arxiv.org/abs/2303.15389" doi: "10.48550/arxiv.2303.15389" arxiv: "2303.15389" licence: "arXiv — non-exclusive license to distribute" licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/sun-2023-eva-clip-improved-training-techniques/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sun-2023-eva-clip-improved-training-techniques/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/) / EVA-CLIP: Improved Training Techniques for CLIP at Scale # EVA-CLIP: Improved Training Techniques for CLIP at Scale **Authors:** Quan Sun, Yuxin Fang, [Ledell Wu](https://md.research.elamur.ai/free/en/authors/ledell-wu/), [Xinlong Wang](https://md.research.elamur.ai/free/en/authors/xinlong-wang/) and [Yue Cao](https://md.research.elamur.ai/free/en/authors/yue-cao/) · **Year:** 2023 · **Venue:** arXiv:2303.15389 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2303.15389) · [PDF](https://arxiv.org/pdf/2303.15389) · [DOI 10.48550/arxiv.2303.15389](https://doi.org/10.48550/arxiv.2303.15389) · [arXiv 2303.15389](https://arxiv.org/abs/2303.15389) · [OpenAlex](https://openalex.org/W4361194507) **Citations:** 79 citations (OpenAlex, 2026-10-14) · [105](https://md.research.elamur.ai/free/en/papers/sun-2023-eva-clip-improved-training-techniques/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Training CLIP-style image-text models at large scale is expensive and often unstable. EVA-CLIP, from the Beijing Academy of Artificial Intelligence, is a model family built around a set of training techniques that cut cost while raising zero-shot quality. The image encoder starts from EVA weights (EVA-01 or EVA-02, pre-trained with masked image modeling), and the text encoder starts from OpenAI CLIP or OpenCLIP. Optimization uses LAMB, which suits very large batches. Following FLIP, half of the image tokens are randomly dropped, and flash attention further speeds up training. The models are trained on LAION-400M, LAION-2B, or a Merged-2B dataset. They match or beat CLIP and OpenCLIP models of the same size after seeing far fewer image-text pairs. The flagship EVA-02-CLIP-E/14+ reaches 82.0% zero-shot top-1 on ImageNet-1K. The results hold across 27 image classification sets, several video benchmarks, and Flickr30K/COCO retrieval. Because all models are publicly released, strong CLIP backbones become accessible to groups with modest compute. ## Key points - **Top results:** EVA-02-CLIP-E/14+ (5.0B parameters, 9B samples seen) scores 82.0% zero-shot on ImageNet-1K and 80.9% averaged over ImageNet variants and ObjectNet. Its gap between ImageNet and the robustness average is only 1.1 points. - **Efficiency vs. OpenCLIP:** EVA-02-CLIP-L/14+ (~430M parameters, 6B samples) reaches 80.4%. That beats OpenCLIP-G/14 (80.1%) with roughly one sixth of the parameters and training samples. - **Ablations (ViT-B/16, shortened schedule):** - EVA initialization adds 1.8 points while using about half the samples. - Replacing AdamW with LAMB adds another 0.7. - 50% token masking costs 0.7 but doubles speed. - Merged-2B reaches the same accuracy with half the samples. - **Training cost:** On 16 A100-40GB GPUs, masking plus flash attention lowers time per 40M samples from 132 to 55 minutes and memory per GPU from 33GB to 16GB. EVA-CLIP-B/16 converges in about 300 hours. - **Retrieval:** EVA-02-CLIP-E/14 trails OpenCLIP-G/14 slightly. The E/14+ variant, with a larger (695M) text encoder, surpasses OpenCLIP-G/14 on zero-shot text retrieval. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Huazhong University of Science and Technology](https://md.research.elamur.ai/free/en/universities/huazhong-university-of-science-and-technology/) — University, Wuhan, China · 45 works · 6,345 citations - [Beijing Academy of Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/beijing-academy-of-artificial-intelligence/) — Research institute, Beijing, China · 32 works · 7,511 citations 📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works 💡 Concepts (1): [💡 Artificial intelligence](https://md.research.elamur.ai/free/en/concepts/artificial-intelligence/) · 49 works ## Cite Cite as: Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang and Yue Cao. 2023. EVA-CLIP: Improved Training Techniques for CLIP at Scale. arXiv:2303.15389. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{sun2023eva, title = {{EVA-CLIP: Improved Training Techniques for CLIP at Scale}}, author = {Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang and Yue Cao}, year = {2023}, doi = {10.48550/arxiv.2303.15389}, eprint = {2303.15389}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2303.15389} } ``` - [← previous Effective Long-Context Scaling of Foundation Models](https://md.research.elamur.ai/free/en/papers/xiong-2023-effective-long-context-scaling-foundation/) - [next → GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Models](https://md.research.elamur.ai/free/en/papers/prasad-2023-grips-gradient-free-edit-based/) - [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/) ## References 4 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Reaching 80 zero-shot accuracy with openclip: Vit-g/14 trained on laion-2b. [https://laion.ai/blog/giant-openclip/.4](https://laion.ai/blog/giant-openclip/.4) [↗](https://laion.ai/blog/giant-openclip/.4) 2. Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021. 5, 6 — [arXiv:2106.08254](https://arxiv.org/abs/2106.08254) [find](https://scholar.google.com/scholar?q=Hangbo+Bao%2C+Li+Dong%2C+and+Furu+Wei%2E+Beit%3A+Bert+pre%2Dtraining+of+image+transformers%2E+arXiv+preprint+arXiv%3A2106%2E08254%2C+2021%2E+5%2C+6+%E2%80%94+%5BarXiv%3A2106%2E08254%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2106%2E08254%29) 3. Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In NeurIPS, 2019. 3 [find](https://scholar.google.com/scholar?q=Andrei+Barbu%2C+David+Mayo%2C+Julian+Alverio%2C+William+Luo%2C+Christopher+Wang%2C+Dan+Gutfreund%2C+Josh+Tenenbaum%2C+and+Boris+Katz%2E+Objectnet%3A+A+large%2Dscale+bias%2Dcontrolled+dataset+for+pushing+the+limits+of+objec) 4. Thomas Berg, Jiongxin Liu, Seung Woo Lee, Michelle L Alexander, David W Jacobs, and Peter N Belhumeur. Birdsnap: Large-EVA-01-CLIP-g / EVA-02-CLIP-g+ config [find](https://scholar.google.com/scholar?q=Thomas+Berg%2C+Jiongxin+Liu%2C+Seung+Woo+Lee%2C+Michelle+L+Alexander%2C+David+W+Jacobs%2C+and+Peter+N+Belhumeur%2E+Birdsnap%3A+Large%2DEVA%2D01%2DCLIP%2Dg+%2F+EVA%2D02%2DCLIP%2Dg%2B+config) *Updated 29 October 2026*