---
title: "GIT — paper by Wang et al. (2022) — summary and ideas"
description: "Vision-language systems for captioning and visual question answering often rely on intricate… 11 references, cited by 114 papers here."
authors: ["Jianfeng Wang", "Zhengyuan Yang", "Xiaowei Hu", "Linjie Li", "Kevin Lin", "Zhe Gan", "Zicheng Liu", "Ce Liu", "Lijuan Wang"]
year: 2022
venue: "arXiv:2205.14100"
source: "https://arxiv.org/abs/2205.14100"
doi: "10.48550/arxiv.2205.14100"
arxiv: "2205.14100"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2022-git-generative-image-text-transformer/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2022-git-generative-image-text-transformer/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/2/) / GIT: A Generative Image-to-text Transformer for Vision and Language

# GIT: A Generative Image-to-text Transformer for Vision and Language

**Authors:** [Jianfeng Wang](https://md.research.elamur.ai/free/en/authors/jianfeng-wang/), [Zhengyuan Yang](https://md.research.elamur.ai/free/en/authors/zhengyuan-yang/), [Xiaowei Hu](https://md.research.elamur.ai/free/en/authors/xiaowei-hu/), [Linjie Li](https://md.research.elamur.ai/free/en/authors/linjie-li/), [Kevin Lin](https://md.research.elamur.ai/free/en/authors/kevin-lin/), [Zhe Gan](https://md.research.elamur.ai/free/en/authors/zhe-gan/), [Zicheng Liu](https://md.research.elamur.ai/free/en/authors/zicheng-liu/), [Ce Liu](https://md.research.elamur.ai/free/en/authors/ce-liu/) and [Lijuan Wang](https://md.research.elamur.ai/free/en/authors/lijuan-wang/) · **Year:** 2022 · **Venue:** arXiv:2205.14100 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2205.14100) · [PDF](https://arxiv.org/pdf/2205.14100) · [DOI 10.48550/arxiv.2205.14100](https://doi.org/10.48550/arxiv.2205.14100) · [arXiv 2205.14100](https://arxiv.org/abs/2205.14100) · [OpenAlex](https://openalex.org/W4320458302)

**Citations:** 205 citations (OpenAlex, 2026-10-14) · [114](https://md.research.elamur.ai/free/en/papers/wang-2022-git-generative-image-text-transformer/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Vision-language systems for captioning and visual question answering often rely on intricate multi-module designs. These include multi-modal encoders, object detectors, tag generators and OCR pipelines, and their pre-training objectives (MLM, ITM) do not match the downstream generation tasks. GIT reduces this to two parts: an image encoder initialised from contrastive pre-training and a transformer text decoder. Both are trained jointly with a single language-modeling loss on 0.8 billion image-text pairs. The same setup handles VQA by feeding the question as a text prefix. It handles video by encoding sampled frames separately, adding a learnable temporal embedding and concatenating the features. It also handles image classification by generating label names rather than choosing from a fixed vocabulary. After scaling both data and model size, GIT sets new state-of-the-art results on many image and video captioning and QA benchmarks, often by wide margins. On TextCaps it outscores the human reference for the first time. The work shows that a minimal architecture, combined with scale, can match or beat heavily engineered pipelines.

## Key points
- On TextCaps, GIT reaches 138.2 CIDEr against a human baseline of 125.5. On COCO captioning it scores 148.8 CIDEr, compared with a prior SOTA of 138.7.
- The architecture uses one Swin-like image encoder (Florence/CLIP-style contrastive initialisation) and one text decoder. It needs no object detector, object tags or OCR, and video tasks work without a video-specific encoder.
- Generation-based ImageNet-1K classification achieves 88.79% top-1 accuracy. An enlarged variant, GIT2, improves results further on most benchmarks.
- Ablations on GIT B show that CLIP initialisation of the image encoder clearly beats supervised, MAE and random initialisation (COCO CIDEr 131.4 vs. 89.0 for random). Initialising the decoder from a BERT text checkpoint gives no benefit.
- A gender and skin bias check on COCO, using annotations from Zhao et al., found normalized performance differences between 0.7% and 5.3%. Intermediate fine-tuning on combined VQA datasets helps most on small targets, such as ST-VQA with a gain of 2.2 points.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (1): [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 811 works

## Cite

Cite as: Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu and Lijuan Wang. 2022. GIT: A Generative Image-to-text Transformer for Vision and Language. arXiv:2205.14100.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{wang2022git,
  title = {{GIT: A Generative Image-to-text Transformer for Vision and Language}},
  author = {Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu and Lijuan Wang},
  year = {2022},
  doi = {10.48550/arxiv.2205.14100},
  eprint = {2205.14100},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2205.14100}
}
```

- [← previous Text and Code Embeddings by Contrastive Pre-Training](https://md.research.elamur.ai/free/en/papers/neelakantan-2022-text-code-embeddings-contrastive-pre/)
- [next → WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation](https://md.research.elamur.ai/free/en/papers/liu-2022-wanli-worker-ai-collaboration-natural/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/2/)

## References

11 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Nayyer Aafaq, Naveed Akhtar, Wei Liu, Syed Zulqarnain Gilani, and Ajmal Mian. Spatio-temporal dynamics and semantic attribute enriched visual encoding for video captioning. In CVPR, 2019. [find](https://scholar.google.com/scholar?q=Nayyer+Aafaq%2C+Naveed+Akhtar%2C+Wei+Liu%2C+Syed+Zulqarnain+Gilani%2C+and+Ajmal+Mian%2E+Spatio%2Dtemporal+dynamics+and+semantic+attribute+enriched+visual+encoding+for+video+captioning%2E+In+CVPR%2C+2019%2E)
2. Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. nocaps: novel object captioning at scale. In ICCV, 2019. [find](https://scholar.google.com/scholar?q=Harsh+Agrawal%2C+Karan+Desai%2C+Yufei+Wang%2C+Xinlei+Chen%2C+Rishabh+Jain%2C+Mark+Johnson%2C+Dhruv+Batra%2C+Devi+Parikh%2C+Stefan+Lee%2C+and+Peter+Anderson%2E+nocaps%3A+novel+object+captioning+at+scale%2E+In+ICCV%2C+2019%2E)
3. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. [Flamingo: a visual language model for few-shot learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/). arXiv preprint arXiv:2204.14198, 2022.
4. Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR, 2018. [find](https://scholar.google.com/scholar?q=Peter+Anderson%2C+Xiaodong+He%2C+Chris+Buehler%2C+Damien+Teney%2C+Mark+Johnson%2C+Stephen+Gould%2C+and+Lei+Zhang%2E+Bottom%2Dup+and+top%2Ddown+attention+for+image+captioning+and+visual+question+answering%2E+In+CVPR%2C+2018)
5. Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. Scene text visual question answering. In ICCV, 2019. [find](https://scholar.google.com/scholar?q=Ali+Furkan+Biten%2C+Ruben+Tito%2C+Andres+Mafla%2C+Lluis+Gomez%2C+Mar%C3%A7al+Rusinol%2C+Ernest+Valveny%2C+CV+Jawahar%2C+and+Dimosthenis+Karatzas%2E+Scene+text+visual+question+answering%2E+In+ICCV%2C+2019%2E)
6. Ali Furkan Biten, Ron Litman, Yusheng Xie, Srikar Appalaraju, and R Manmatha. Latr: Layout-aware transformer for scene-text vqa. In CVPR, 2022. [find](https://scholar.google.com/scholar?q=Ali+Furkan+Biten%2C+Ron+Litman%2C+Yusheng+Xie%2C+Srikar+Appalaraju%2C+and+R+Manmatha%2E+Latr%3A+Layout%2Daware+transformer+for+scene%2Dtext+vqa%2E+In+CVPR%2C+2022%2E)
7. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
8. Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021. [find](https://scholar.google.com/scholar?q=Soravit+Changpinyo%2C+Piyush+Sharma%2C+Nan+Ding%2C+and+Radu+Soricut%2E+Conceptual+12M%3A+Pushing+web%2Dscale+image%2Dtext+pre%2Dtraining+to+recognize+long%2Dtail+visual+concepts%2E+In+CVPR%2C+2021%2E)
9. David Chen and William Dolan. [Collecting highly parallel data for paraphrase evaluation](https://md.research.elamur.ai/free/en/papers/chen-2011-collecting-highly-parallel-data-paraphrase/). In ACL, 2011.
10. Shaoxiang Chen and Yu-Gang Jiang. Motion guided spatial attention for video captioning. In AAAI, 2019. [find](https://scholar.google.com/scholar?q=Shaoxiang+Chen+and+Yu%2DGang+Jiang%2E+Motion+guided+spatial+attention+for+video+captioning%2E+In+AAAI%2C+2019%2E)
11. Shaoxiang Chen, Wenhao Jiang, Wei Liu, and Yu-Gang Jiang. Learning modality interaction for temporal sentence localization and event captioning in videos. In ECCV, 2020a. [find](https://scholar.google.com/scholar?q=Shaoxiang+Chen%2C+Wenhao+Jiang%2C+Wei+Liu%2C+and+Yu%2DGang+Jiang%2E+Learning+modality+interaction+for+temporal+sentence+localization+and+event+captioning+in+videos%2E+In+ECCV%2C+2020a%2E)

*Updated 2 November 2026*