---
title: "Sigmoid Loss for Language Image Pre-Training — Zhai 2023"
description: "Contrastive image-text pre-training methods such as CLIP and ALIGN usually use a softmax loss, which normalizes similarity scores over the whole batch."
authors: ["Xiaohua Zhai", "Basil Mustafa", "Alexander Kolesnikov", "Lucas Beyer"]
year: 2023
venue: "arXiv:2303.15343"
source: "https://arxiv.org/abs/2303.15343"
doi: "10.48550/arxiv.2303.15343"
arxiv: "2303.15343"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhai-2023-sigmoid-loss-language-image-pre/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhai-2023-sigmoid-loss-language-image-pre/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / Sigmoid Loss for Language Image Pre-Training

# Sigmoid Loss for Language Image Pre-Training

**Authors:** Xiaohua Zhai, [Basil Mustafa](https://md.research.elamur.ai/free/en/authors/basil-mustafa/), [Alexander Kolesnikov](https://md.research.elamur.ai/free/en/authors/alexander-kolesnikov/), [Lucas Beyer](https://md.research.elamur.ai/free/en/authors/lucas-beyer/) · **Year:** 2023 · **Venue:** arXiv:2303.15343 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2303.15343) · [PDF](https://arxiv.org/pdf/2303.15343) · [DOI 10.48550/arxiv.2303.15343](https://doi.org/10.48550/arxiv.2303.15343) · [arXiv 2303.15343](https://arxiv.org/abs/2303.15343) · [OpenAlex](https://openalex.org/W4390873312)

**Citations:** 1060 citations (openalex-snapshot · 2026-10-14) · [367](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Contrastive image-text pre-training methods such as CLIP and ALIGN usually use a softmax loss, which normalizes similarity scores over the whole batch. On multiple devices this requires all-gathers, holding a full |B|×|B| similarity matrix in memory, and an extra pass over the batch for numerical stability. The authors replace it with a pairwise sigmoid loss. Each image-text combination in a batch becomes a separate binary decision: matching pairs are positives and all other pairs are negatives. A learnable temperature and bias are added. With no global normalization, the loss can be computed in "chunks": devices pass text embeddings around so that only a small block is in memory at any time. The loss is tested in two setups: training from scratch (SigLIP) and Locked-image Tuning with a frozen image tower (SigLiT). Sigmoid clearly beats softmax at batch sizes under 16k and matches it at larger ones. It also makes batches of up to one million possible, though gains level off near 32k. As a result, strong vision-language models can be trained on only a few accelerators.

## Key points
- SigLiT with a frozen g/14 image backbone reaches 84.5% zero-shot ImageNet accuracy after two days on four TPUv4 chips. A B/8 backbone gets about 79.7% after one day on the same hardware.
- SigLIP B/16 trained on WebLI reaches 73.4% zero-shot ImageNet accuracy in 5 days on 32 TPUv4 chips. FLIP and CLIP needed roughly 5 and 10 days on 256 TPUv3 cores.
- Initialization matters. With bias b = −10 and t′ = log 10, the Base model scores 63.0% on ImageNet. Without a bias it scores 62.0%, and with zero bias and t′ = log 1 it drops to 53.7%.
- Both sigmoid and softmax stop improving as batch size grows, and about 32k is close to optimal. The same holds for multilingual SigLIP trained on over 100 languages.
- Sigmoid-trained models stay more robust than softmax ones when images, texts or pair alignments are corrupted. The models are released in the big_vision repository.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

367 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 354 not yet.

- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://md.research.elamur.ai/free/en/papers/chen-2024-expanding-performance-boundaries-open/)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://md.research.elamur.ai/free/en/papers/chen-2024-far-gpt-4v-closing/)
- [Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling](https://md.research.elamur.ai/free/en/papers/chen-2025-janus-pro-unified-multimodal/)
- [Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models](https://md.research.elamur.ai/free/en/papers/deitke-2024-molmo-pixmo-open-weights-open/)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://md.research.elamur.ai/free/en/papers/fu-2024-video-mme-first-ever-comprehensive/)
- [What matters when building vision-language models?](https://md.research.elamur.ai/free/en/papers/lauren-2024-ccon-matters-building-vision-language-models/)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://md.research.elamur.ai/free/en/papers/li-2024-llava-onevision-easy-visual-task/)
- [A Large-Scale Instruction-Tuning Dataset and Models for Slovenian Vision-Language Tasks](https://md.research.elamur.ai/free/en/papers/martinc-2026-large-scale-instruction-tuning-dataset/)
- [Evaluating Discriminability of Vision-Language Models](https://md.research.elamur.ai/free/en/papers/muraoka-2026-evaluating-discriminability-vision-language-models/)
- [Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs](https://md.research.elamur.ai/free/en/papers/tong-2024-cambrian-1-fully-open-vision/)
- [DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding](https://md.research.elamur.ai/free/en/papers/wu-2024-deepseek-vl2-mixture-experts/)
- [MASA: A Novel Multimodal Foundation Model for L2 Speaking Assessment in Picture-description Scenarios](https://md.research.elamur.ai/free/en/papers/yan-2026-masa-novel-multimodal-foundation-model/)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://md.research.elamur.ai/free/en/papers/yao-2024-minicpm-v-gpt-4v/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhai2023sigmoid,
  title = {{Sigmoid Loss for Language Image Pre-Training}},
  author = {Zhai, Xiaohua and Mustafa, Basil and Kolesnikov, Alexander and Beyer, Lucas},
  year = {2023},
  doi = {10.48550/arxiv.2303.15343},
  eprint = {2303.15343},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.15343}
}
```

- [← previous SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects](https://md.research.elamur.ai/free/en/papers/adelani-2023-sib-200-simple-inclusive-big/)
- [next → Simplify: Automatic Arabic Sentence Simplification using Word Embeddings](https://md.research.elamur.ai/free/en/papers/salaheldin-2023-simplify-automatic-arabic-sentence-simplification/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

13 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ibrahim Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. Getting vit in shape: Scaling laws for compute-optimal model design. In NeurIPS, 2023. 7, 8, 17 [find](https://scholar.google.com/scholar?q=Ibrahim+Alabdulmohsin%2C+Xiaohua+Zhai%2C+Alexander+Kolesnikov%2C+and+Lucas+Beyer%2E+Getting+vit+in+shape%3A+Scaling+laws+for+compute%2Doptimal+model+design%2E+In+NeurIPS%2C+2023%2E+7%2C+8%2C+17)
2. Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In NeurIPS, 2019. 7, 17 [find](https://scholar.google.com/scholar?q=Andrei+Barbu%2C+David+Mayo%2C+Julian+Alverio%2C+William+Luo%2C+Christopher+Wang%2C+Dan+Gutfreund%2C+Josh+Tenenbaum%2C+and+Boris+Katz%2E+ObjectNet%3A+A+large%2Dscale+bias%2Dcontrolled+dataset+for+pushing+the+limits+of+objec)
3. Lucas Beyer, Olivier J. Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? CoRR, abs/2006.07159, 2020. 2, 7, 9, 17 — [arXiv:2006.07159](https://arxiv.org/abs/2006.07159) [find](https://scholar.google.com/scholar?q=Lucas+Beyer%2C+Olivier+J%2E+He%CC%81naff%2C+Alexander+Kolesnikov%2C+Xiaohua+Zhai%2C+and+Aa%CC%88ron+van+den+Oord%2E+Are+we+done+with+imagenet%3F+CoRR%2C+abs%2F2006%2E07159%2C+2020%2E+2%2C+7%2C+9%2C+17+%E2%80%94+%5BarXiv%3A2006%2E07159%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
4. Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov. Better plain vit baselines for imagenet-1k, 2022. 10, 17 [find](https://scholar.google.com/scholar?q=Lucas+Beyer%2C+Xiaohua+Zhai%2C+and+Alexander+Kolesnikov%2E+Better+plain+vit+baselines+for+imagenet%2D1k%2C+2022%2E+10%2C+17)
5. Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov. Big vision. [https://github.com/google-research/](https://github.com/google-research/) big_vision, 2022. 10, 17 [↗](https://github.com/google-research/)
6. Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. Coyo-700m: Image-text pair dataset. [https://github.com/](https://github.com/) kakaobrain/coyo-dataset, 2022. 1 [↗](https://github.com/)
7. Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021. 1 [find](https://scholar.google.com/scholar?q=Soravit+Changpinyo%2C+Piyush+Sharma%2C+Nan+Ding%2C+and+Radu+Soricut%2E+Conceptual+12M%3A+Pushing+web%2Dscale+image%2Dtext+pre%2Dtraining+to+recognize+long%2Dtail+visual+concepts%2E+In+CVPR%2C+2021%2E+1)
8. Feilong Chen, Duzhen Zhang, Minglun Han, Xiu-Yi Chen, Jing Shi, Shuang Xu, and Bo Xu. VLP: A survey on vision-language pre-training. Int. J. Autom. Comput., 20(1):38–56, 2023. 2 [find](https://scholar.google.com/scholar?q=Feilong+Chen%2C+Duzhen+Zhang%2C+Minglun+Han%2C+Xiu%2DYi+Chen%2C+Jing+Shi%2C+Shuang+Xu%2C+and+Bo+Xu%2E+VLP%3A+A+survey+on+vision%2Dlanguage+pre%2Dtraining%2E+Int%2E+J%2E+Autom%2E+Comput%2E%2C+20%281%29%3A38%E2%80%9356%2C+2023%2E+2)
9. Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pre-training from pixels. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Researc… [find](https://scholar.google.com/scholar?q=Mark+Chen%2C+Alec+Radford%2C+Rewon+Child%2C+Jeffrey+Wu%2C+Heewoo+Jun%2C+David+Luan%2C+and+Ilya+Sutskever%2E+Generative+pre%2Dtraining+from+pixels%2E+In+Proceedings+of+the+37th+International+Conference+on+Machine+Learni)
10. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020. 2, 4 [find](https://scholar.google.com/scholar?q=Ting+Chen%2C+Simon+Kornblith%2C+Mohammad+Norouzi%2C+and+Geoffrey+E%2E+Hinton%2E+A+simple+framework+for+contrastive+learning+of+visual+representations%2E+In+ICML%2C+2020%2E+2%2C+4) [our card](https://md.research.elamur.ai/free/en/papers/chen-2020-simple-framework-contrastive-learning-visual/)
11. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. CoRR, abs/1504.00325, 2015. 7, 17 [find](https://scholar.google.com/scholar?q=Xinlei+Chen%2C+Hao+Fang%2C+Tsung%2DYi+Lin%2C+Ramakrishna+Vedantam%2C+Saurabh+Gupta%2C+Piotr+Dolla%CC%81r%2C+and+C%2E+Lawrence+Zitnick%2E+Microsoft+COCO+captions%3A+Data+collection+and+evaluation+server%2E+CoRR%2C+abs%2F1504%2E00325%2C+) [our card](https://md.research.elamur.ai/free/en/papers/chen-2015-microsoft-coco-captions-data-collection/)
12. Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, and Quoc V. Le. Symbolic discovery of optimization algorithms, 2023. 2, 6 [find](https://scholar.google.com/scholar?q=Xiangning+Chen%2C+Chen+Liang%2C+Da+Huang%2C+Esteban+Real%2C+Kaiyuan+Wang%2C+Yao+Liu%2C+Hieu+Pham%2C+Xuanyi+Dong%2C+Thang+Luong%2C+Cho%2DJui+Hsieh%2C+Yifeng+Lu%2C+and+Quoc+V%2E+Le%2E+Symbolic+discovery+of+optimization+algorithms%2C)
13. Xi Chen, Xiao Wang, Soravit Changpinyo, A. J. Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Ku… [find](https://scholar.google.com/scholar?q=Xi+Chen%2C+Xiao+Wang%2C+Soravit+Changpinyo%2C+A%2E+J%2E+Piergiovanni%2C+Piotr+Padlewski%2C+Daniel+Salz%2C+Sebastian+Goodman%2C+Adam+Grycner%2C+Basil+Mustafa%2C+Lucas+Beyer%2C+Alexander+Kolesnikov%2C+Joan+Puigcerver%2C+Nan+Ding%2C+) [our card](https://md.research.elamur.ai/free/en/papers/schuhmann-2022-laion-5b-open-large-scale/)