---
title: "TextCaps — paper by Sidorov et al. (2020) — summary and ideas"
description: "Standard image captioning systems describe objects but ignore written text in a scene, even though… 38 references, cited by 106 papers here."
authors: ["Oleksii Sidorov", "Ronghang Hu", "Marcus Rohrbach", "Amanpreet Singh"]
year: 2020
venue: "arXiv:2003.12462"
source: "https://arxiv.org/abs/2003.12462"
doi: "10.48550/arxiv.2003.12462"
arxiv: "2003.12462"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sidorov-2020-textcaps-dataset-image-captioning-reading/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sidorov-2020-textcaps-dataset-image-captioning-reading/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/2/) / TextCaps: a Dataset for Image Captioning with Reading Comprehension

# TextCaps: a Dataset for Image Captioning with Reading Comprehension

**Authors:** Oleksii Sidorov, Ronghang Hu, [Marcus Rohrbach](https://md.research.elamur.ai/free/en/authors/marcus-rohrbach/) and [Amanpreet Singh](https://md.research.elamur.ai/free/en/authors/amanpreet-singh/) · **Year:** 2020 · **Venue:** [arXiv:2003.12462](https://md.research.elamur.ai/free/en/venues/lecture-notes-in-computer-science/) ([Lecture notes in computer science 2020](https://md.research.elamur.ai/free/en/venues/lecture-notes-in-computer-science/2020/)) · **Publisher:** [Springer Science+Business Media](https://md.research.elamur.ai/free/en/publishers/springer-science-business-media/) · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2003.12462) · [PDF](https://arxiv.org/pdf/2003.12462) · [DOI 10.48550/arxiv.2003.12462](https://doi.org/10.48550/arxiv.2003.12462) · [arXiv 2003.12462](https://arxiv.org/abs/2003.12462) · [OpenAlex](https://openalex.org/W3013137071)

**Citations:** 225 citations (OpenAlex, 2026-10-14) · [106](https://md.research.elamur.ai/free/en/papers/sidorov-2020-textcaps-dataset-image-captioning-reading/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Standard image captioning systems describe objects but ignore written text in a scene, even though such text often carries the key information, especially for blind and low-vision users. To address this, the authors define a task they call image captioning with reading comprehension and build TextCaps, a dataset of about 142k human-written captions for 28,408 Open Images v3 images. These are the same images used in TextVQA, all confirmed to contain text. Crowd workers wrote one-sentence descriptions that depend on reading that text, and a second group of workers voted on quality to filter them.

The captions require a model to:
- choose which OCR tokens matter,
- combine or paraphrase them, and
- alternate repeatedly between vocabulary words and copied scene text, many of which never appear in training.

Conventional captioners such as BUTD essentially fail at this. An adaptation of the TextVQA model M4C, called M4C-Captioner, performs much better. Human captions still clearly outperform it, which leaves an open benchmark for future work.

## Key points
- **Dataset and annotation:** TextCaps contains 142,040 captions on 28,408 images (the abstract rounds this to 145k). Annotation had two stages. Writers produced captions that needed reading, then five evaluators judged each caption. The criteria were whether it required reading, was accurate, was one grammatical sentence, and was free of subjective language.
- **Why existing data falls short:** Only about 2.7% of COCO captions mention scene text, and COCO's OCR vocabulary has fewer than 350 distinct tokens. Text-VQA answers are mostly 1–2 words, so they cannot teach full-sentence generation.
- **Model results:** M4C-Captioner gave the best results. Ablations show it needs semantic, visual and spatial information about OCR tokens. Supplying ground-truth OCR narrows the gap to human performance but does not close it.
- **Transfer to COCO:** Training jointly on COCO and TextCaps makes the model mention text in COCO images. Trained on COCO alone, it does not.
- **Availability:** The dataset and its challenge server are hosted at textvqa.org/textcaps.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Meta](https://md.research.elamur.ai/free/en/companies/meta/) — Company, Menlo Park, United States · 214 works · 241,492 citations
- [University of California, Berkeley](https://md.research.elamur.ai/free/en/universities/university-of-california-berkeley/) — University, Berkeley, United States · 209 works · 244,782 citations
- [Meta (United Kingdom)](https://md.research.elamur.ai/free/en/companies/meta-united-kingdom/) — Company, London, United Kingdom · 54 works · 19,348 citations

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (2): [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 811 works · [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 487 works

## Cite

Cite as: Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach and Amanpreet Singh. 2020. TextCaps: a Dataset for Image Captioning with Reading Comprehension. arXiv:2003.12462.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{sidorov2020textcaps,
  title = {{TextCaps: a Dataset for Image Captioning with Reading Comprehension}},
  author = {Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach and Amanpreet Singh},
  year = {2020},
  doi = {10.48550/arxiv.2003.12462},
  eprint = {2003.12462},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2003.12462}
}
```

- [← previous The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence](https://md.research.elamur.ai/free/en/papers/marcus-2020-next-decade-ai-four/)
- [next → Unsupervised Domain Clusters in Pretrained Language Models](https://md.research.elamur.ai/free/en/papers/aharoni-2020-unsupervised-domain-clusters-pretrained-language/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/2/)

## References

38 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Agrawal*, H., Desai*, K., Wang, Y., Chen, X., Jain, R., Johnson, M., Batra, D., Parikh, D., Lee, S., Anderson, P.: nocaps: novel object captioning at scale. In: International Conference on Computer Vision (ICCV) (2019) 4 [find](https://scholar.google.com/scholar?q=Agrawal%2A%2C+H%2E%2C+Desai%2A%2C+K%2E%2C+Wang%2C+Y%2E%2C+Chen%2C+X%2E%2C+Jain%2C+R%2E%2C+Johnson%2C+M%2E%2C+Batra%2C+D%2E%2C+Parikh%2C+D%2E%2C+Lee%2C+S%2E%2C+Anderson%2C+P%2E%3A+nocaps%3A+novel+object+captioning+at+scale%2E+In%3A+International+Conference+on+Computer+Vi)
2. Almazán, J., Gordo, A., Fornés, A., Valveny, E.: Word spotting and recognition with embedded attributes. IEEE transactions on pattern analysis and machine intelligence 36(12), 2552–2566 (2014) 10 [find](https://scholar.google.com/scholar?q=Almaza%CC%81n%2C+J%2E%2C+Gordo%2C+A%2E%2C+Forne%CC%81s%2C+A%2E%2C+Valveny%2C+E%2E%3A+Word+spotting+and+recognition+with+embedded+attributes%2E+IEEE+transactions+on+pattern+analysis+and+machine+intelligence+36%2812%29%2C+2552%E2%80%932566+%282014%29+10)
3. Anderson, P., Fernando, B., Johnson, M., Gould, S.: [Spice: Semantic propositional image caption evaluation](https://md.research.elamur.ai/free/en/papers/anderson-2016-spice-semantic-propositional-image-caption/). In: European Conference on Computer Vision. pp. 382– 398. Springer (2016) 10
4. Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., Zhang, L.: Bottom-up and top-down attention for image captioning and visual question answering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 6077–6086 (2018) 1, 2, 9, 10, 11 [find](https://scholar.google.com/scholar?q=Anderson%2C+P%2E%2C+He%2C+X%2E%2C+Buehler%2C+C%2E%2C+Teney%2C+D%2E%2C+Johnson%2C+M%2E%2C+Gould%2C+S%2E%2C+Zhang%2C+L%2E%3A+Bottom%2Dup+and+top%2Ddown+attention+for+image+captioning+and+visual+question+answering%2E+In%3A+Proceedings+of+the+IEEE+Confer)
5. Bigham, J.P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R.C., Miller, R., Tatarowicz, A., White, B., White, S., et al.: Vizwiz: nearly real-time answers to visual questions. In: Proceedings of the 23nd annual ACM symposium on User interface software and technology. pp. 333–342. ACM (2010)… [find](https://scholar.google.com/scholar?q=Bigham%2C+J%2EP%2E%2C+Jayant%2C+C%2E%2C+Ji%2C+H%2E%2C+Little%2C+G%2E%2C+Miller%2C+A%2E%2C+Miller%2C+R%2EC%2E%2C+Miller%2C+R%2E%2C+Tatarowicz%2C+A%2E%2C+White%2C+B%2E%2C+White%2C+S%2E%2C+et+al%2E%3A+Vizwiz%3A+nearly+real%2Dtime+answers+to+visual+questions%2E+In%3A+Proceedings+)
6. Biten, A.F., Tito, R., Mafla, A., Gomez, L., Rusiñol, M., Valveny, E., Jawahar, C., Karatzas, D.: Scene text visual question answering. arXiv preprint arXiv:1905.13648 (2019) 3, 4, 7 [find](https://scholar.google.com/scholar?q=Biten%2C+A%2EF%2E%2C+Tito%2C+R%2E%2C+Mafla%2C+A%2E%2C+Gomez%2C+L%2E%2C+Rusin%CC%83ol%2C+M%2E%2C+Valveny%2C+E%2E%2C+Jawahar%2C+C%2E%2C+Karatzas%2C+D%2E%3A+Scene+text+visual+question+answering%2E+arXiv+preprint+arXiv%3A1905%2E13648+%282019%29+3%2C+4%2C+7)
7. Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: [Enriching word vectors with subword information](https://md.research.elamur.ai/free/en/papers/bojanowski-2017-enriching-word-vectors-subword-information/). Transactions of the Association for Computational Linguistics 5, 135–146 (2017) 10
8. Borisyuk, F., Gordo, A., Sivakumar, V.: Rosetta: Large scale system for text detection and recognition in images. In: ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. pp. 71–79. ACM (2018) 1, 4, 5, 7, 19 [find](https://scholar.google.com/scholar?q=Borisyuk%2C+F%2E%2C+Gordo%2C+A%2E%2C+Sivakumar%2C+V%2E%3A+Rosetta%3A+Large+scale+system+for+text+detection+and+recognition+in+images%2E+In%3A+ACM+SIGKDD+International+Conference+on+Knowledge+Discovery+%26+Data+Mining%2E+pp%2E+71%E2%80%937)
9. Chen, X., Fang, H., Lin, T.Y., Vedantam, R., Gupta, S., Dollár, P., Zitnick, C.L.: [Microsoft coco captions: Data collection and evaluation server](https://md.research.elamur.ai/free/en/papers/chen-2015-microsoft-coco-captions-data-collection/). arXiv preprint arXiv:1504.00325 (2015) 3, 7, 13
10. Chen, Y.C., Li, L., Yu, L., Kholy, A.E., Ahmed, F., Gan, Z., Cheng, Y., Liu, J.: Uniter: Learning universal image-text representations. arXiv preprint arXiv:1909.[11740 (2019) 1](https://md.research.elamur.ai/free/en/papers/chen-2019-uniter-universal-image-text-representation/)
11. Denkowski, M., Lavie, A.: [Meteor universal: Language specific translation evaluation for any target language](https://md.research.elamur.ai/free/en/papers/denkowski-2014-meteor-universal-language-specific-translation/). In: Proceedings of the ninth workshop on statistical machine translation. pp. 376–380 (2014) 10
12. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In: NAACL-HLT (2019) 1
13. Goyal, P., Mahajan, D.K., Gupta, A., Misra, I.: Scaling and benchmarking self-supervised visual representation learning. International Conference on Computer Vision abs/1905.01235 (2019) 1 [find](https://scholar.google.com/scholar?q=Goyal%2C+P%2E%2C+Mahajan%2C+D%2EK%2E%2C+Gupta%2C+A%2E%2C+Misra%2C+I%2E%3A+Scaling+and+benchmarking+self%2Dsupervised+visual+representation+learning%2E+International+Conference+on+Computer+Vision+abs%2F1905%2E01235+%282019%29+1)
14. Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., Parikh, D.: Vqa 2.0 evaluation. [https://visualqa.org/evaluation.html](https://visualqa.org/evaluation.html) 10 [↗](https://visualqa.org/evaluation.html)
15. Gurari, D., Zhao, Y., Zhang, M., Bhattacharya, N.: Captioning images taken by people who are blind. arXiv preprint arXiv:2002.08565 (2020) 4 — [arXiv:2002.08565](https://arxiv.org/abs/2002.08565) [find](https://scholar.google.com/scholar?q=Gurari%2C+D%2E%2C+Zhao%2C+Y%2E%2C+Zhang%2C+M%2E%2C+Bhattacharya%2C+N%2E%3A+Captioning+images+taken+by+people+who+are+blind%2E+arXiv+preprint+arXiv%3A2002%2E08565+%282020%29+4+%E2%80%94+%5BarXiv%3A2002%2E08565%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2002%2E08565%29)
16. He, T., Tian, Z., Huang, W., Shen, C., Qiao, Y., Sun, C.: An end-to-end textspotter with explicit alignment and attention. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 5020–5029 (2018) 1, 4 O. Sidorov et al. [find](https://scholar.google.com/scholar?q=He%2C+T%2E%2C+Tian%2C+Z%2E%2C+Huang%2C+W%2E%2C+Shen%2C+C%2E%2C+Qiao%2C+Y%2E%2C+Sun%2C+C%2E%3A+An+end%2Dto%2Dend+textspotter+with+explicit+alignment+and+attention%2E+In%3A+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+Recogni)
17. Hu, R., Singh, A., Darrell, T., Rohrbach, M.: Iterative answer prediction with pointer-augmented multimodal transformers for textvqa. arXiv preprint arXiv:1911.06258 (2019) 3, 9, 10, 19 [find](https://scholar.google.com/scholar?q=Hu%2C+R%2E%2C+Singh%2C+A%2E%2C+Darrell%2C+T%2E%2C+Rohrbach%2C+M%2E%3A+Iterative+answer+prediction+with+pointer%2Daugmented+multimodal+transformers+for+textvqa%2E+arXiv+preprint+arXiv%3A1911%2E06258+%282019%29+3%2C+9%2C+10%2C+19)
18. Huang, L., Wang, W., Chen, J., Wei, X.Y.: Attention on attention for image captioning. In: IEEE International Conference on Computer Vision. pp. 4634–4643 (2019) 9, 11 [find](https://scholar.google.com/scholar?q=Huang%2C+L%2E%2C+Wang%2C+W%2E%2C+Chen%2C+J%2E%2C+Wei%2C+X%2EY%2E%3A+Attention+on+attention+for+image+captioning%2E+In%3A+IEEE+International+Conference+on+Computer+Vision%2E+pp%2E+4634%E2%80%934643+%282019%29+9%2C+11)
19. Karatzas, D., Shafait, F., Uchida, S., Iwamura, M., i Bigorda, L.G., Mestre, S.R., Mas, J., Mota, D.F., Almazan, J.A., De Las Heras, L.P.: Icdar 2013 robust reading competition. In: 2013 12th International Conference on Document Analysis and Recognition. pp. 1484–1493. IEEE (2013) 19 [find](https://scholar.google.com/scholar?q=Karatzas%2C+D%2E%2C+Shafait%2C+F%2E%2C+Uchida%2C+S%2E%2C+Iwamura%2C+M%2E%2C+i+Bigorda%2C+L%2EG%2E%2C+Mestre%2C+S%2ER%2E%2C+Mas%2C+J%2E%2C+Mota%2C+D%2EF%2E%2C+Almazan%2C+J%2EA%2E%2C+De+Las+Heras%2C+L%2EP%2E%3A+Icdar+2013+robust+reading+competition%2E+In%3A+2013+12th+Internat)
20. Krasin, I., Duerig, T., Alldrin, N., Ferrari, V., Abu-El-Haija, S., Kuznetsova, A., Rom, H., Uijlings, J., Popov, S., Veit, A., Belongie, S., Gomes, V., Gupta, A., Sun, C., Chechik, G., Cai, D., Feng, Z., Narayanan, D., Murphy, K.: Openimages: A public dataset for large-scale multi-label and multi-c… [find](https://scholar.google.com/scholar?q=Krasin%2C+I%2E%2C+Duerig%2C+T%2E%2C+Alldrin%2C+N%2E%2C+Ferrari%2C+V%2E%2C+Abu%2DEl%2DHaija%2C+S%2E%2C+Kuznetsova%2C+A%2E%2C+Rom%2C+H%2E%2C+Uijlings%2C+J%2E%2C+Popov%2C+S%2E%2C+Veit%2C+A%2E%2C+Belongie%2C+S%2E%2C+Gomes%2C+V%2E%2C+Gupta%2C+A%2E%2C+Sun%2C+C%2E%2C+Chechik%2C+G%2E%2C+Cai%2C+D%2E%2C+Feng%2C)
21. Krasin, I., Duerig, T., Alldrin, N., Ferrari, V., Abu-El-Haija, S., Kuznetsova, A., Rom, H., Uijlings, J., Popov, S., Veit, A., Belongie, S., Gomes, V., Gupta, A., Sun, C., Chechik, G., Cai, D., Feng, Z., Narayanan, D., Murphy, K.: Openimages: A public dataset for large-scale multi-label and multi-c… [find](https://scholar.google.com/scholar?q=Krasin%2C+I%2E%2C+Duerig%2C+T%2E%2C+Alldrin%2C+N%2E%2C+Ferrari%2C+V%2E%2C+Abu%2DEl%2DHaija%2C+S%2E%2C+Kuznetsova%2C+A%2E%2C+Rom%2C+H%2E%2C+Uijlings%2C+J%2E%2C+Popov%2C+S%2E%2C+Veit%2C+A%2E%2C+Belongie%2C+S%2E%2C+Gomes%2C+V%2E%2C+Gupta%2C+A%2E%2C+Sun%2C+C%2E%2C+Chechik%2C+G%2E%2C+Cai%2C+D%2E%2C+Feng%2C)
22. Li, H., Wang, P., Shen, C.: Towards end-to-end text spotting with convolutional recurrent neural networks. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 5238–5246 (2017) 1, 4 [find](https://scholar.google.com/scholar?q=Li%2C+H%2E%2C+Wang%2C+P%2E%2C+Shen%2C+C%2E%3A+Towards+end%2Dto%2Dend+text+spotting+with+convolutional+recurrent+neural+networks%2E+In%3A+Proceedings+of+the+IEEE+International+Conference+on+Computer+Vision%2E+pp%2E+5238%E2%80%935246+%282017%29)
23. Lin, C.Y.: [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In: Text summarization branches out. pp. 74–81 (2004) 10
24. Liu, X., Liang, D., Yan, S., Chen, D., Qiao, Y., Yan, J.: Fots: Fast oriented text spotting with a unified network. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5676–5685 (2018) 1, 4 [find](https://scholar.google.com/scholar?q=Liu%2C+X%2E%2C+Liang%2C+D%2E%2C+Yan%2C+S%2E%2C+Chen%2C+D%2E%2C+Qiao%2C+Y%2E%2C+Yan%2C+J%2E%3A+Fots%3A+Fast+oriented+text+spotting+with+a+unified+network%2E+In%3A+Proceedings+of+the+IEEE+conference+on+computer+vision+and+pattern+recognition%2E+p)
25. Lu, J., Yang, J., Batra, D., Parikh, D.: Neural baby talk. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 7219–7228 (2018) 9 [find](https://scholar.google.com/scholar?q=Lu%2C+J%2E%2C+Yang%2C+J%2E%2C+Batra%2C+D%2E%2C+Parikh%2C+D%2E%3A+Neural+baby+talk%2E+In%3A+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition%2E+pp%2E+7219%E2%80%937228+%282018%29+9)
26. Mishra, A., Shekhar, S., Singh, A.K., Chakraborty, A.: Ocr-vqa: Visual question answering by reading text in images. In: ICDAR (2019) 3, 4, 7 [find](https://scholar.google.com/scholar?q=Mishra%2C+A%2E%2C+Shekhar%2C+S%2E%2C+Singh%2C+A%2EK%2E%2C+Chakraborty%2C+A%2E%3A+Ocr%2Dvqa%3A+Visual+question+answering+by+reading+text+in+images%2E+In%3A+ICDAR+%282019%29+3%2C+4%2C+7)
27. Ordonez, V., Kulkarni, G., Berg, T.L.: Im2text: Describing images using 1 million captioned photographs. In: Neural Information Processing Systems (NIPS) (2011) 4, 7 [find](https://scholar.google.com/scholar?q=Ordonez%2C+V%2E%2C+Kulkarni%2C+G%2E%2C+Berg%2C+T%2EL%2E%3A+Im2text%3A+Describing+images+using+1+million+captioned+photographs%2E+In%3A+Neural+Information+Processing+Systems+%28NIPS%29+%282011%29+4%2C+7)
28. Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In: Proceedings of the 40th annual meeting on association for computational linguistics. pp. 311–318. Association for Computational Linguistics (2002) 10
29. Ren, S., He, K., Girshick, R., Sun, J.: [Faster r-cnn: Towards real-time object detection with region proposal networks](https://md.research.elamur.ai/free/en/papers/ren-2015-faster-r-cnn-towards-real/). In: Advances in neural information processing systems. pp. 91–99 (2015) 9, 10
30. Sharma, P., Ding, N., Goodman, S., Soricut, R.: [Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning](https://md.research.elamur.ai/free/en/papers/sharma-2018-conceptual-captions-cleaned-hypernymed-image/). In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 2556–2565 (2018) 4, 7
31. Singh, A., Goswami, V., Natarajan, V., Jiang, Y., Chen, X., Shah, M., Rohrbach, M., Batra, D., Parikh, D.: Mmf: A multimodal framework for vision and language research. — [TextCaps: a Dataset for Image Captioning with Reading Comprehension](https://md.research.elamur.ai/free/en/papers/sidorov-2020-textcaps-dataset-image-captioning-reading/) [https://github.com/facebookresearch/mmf](https://github.com/facebookresearch/mmf) (2020) 10 TextCaps: a Dataset for Image Captioning with Reading Comprehension [↗](https://github.com/facebookresearch/mmf)
32. Singh, A., Natarajan, V., Jiang, Y., Chen, X., Shah, M., Rohrbach, M., Batra, D., Parikh, D.: Pythia-a platform for vision & language research. In: SysML Workshop, NeurIPS. vol. 2018 (2018) 10 [find](https://scholar.google.com/scholar?q=Singh%2C+A%2E%2C+Natarajan%2C+V%2E%2C+Jiang%2C+Y%2E%2C+Chen%2C+X%2E%2C+Shah%2C+M%2E%2C+Rohrbach%2C+M%2E%2C+Batra%2C+D%2E%2C+Parikh%2C+D%2E%3A+Pythia%2Da+platform+for+vision+%26+language+research%2E+In%3A+SysML+Workshop%2C+NeurIPS%2E+vol%2E+2018+%282018%29+10)
33. Singh, A., Natarjan, V., Shah, M., Jiang, Y., Chen, X., Parikh, D., Rohrbach, M.: Towards vqa models that can read. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 8317–8326 (2019) 3, 4, 5, 7, 22 [find](https://scholar.google.com/scholar?q=Singh%2C+A%2E%2C+Natarjan%2C+V%2E%2C+Shah%2C+M%2E%2C+Jiang%2C+Y%2E%2C+Chen%2C+X%2E%2C+Parikh%2C+D%2E%2C+Rohrbach%2C+M%2E%3A+Towards+vqa+models+that+can+read%2E+In%3A+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition%2E+p)
34. Smith, R.: An overview of the tesseract ocr engine. In: International Conference on Document Analysis and Recognition (ICDAR 2007). vol. 2, pp. 629–633. IEEE (2007) 1, 4 [find](https://scholar.google.com/scholar?q=Smith%2C+R%2E%3A+An+overview+of+the+tesseract+ocr+engine%2E+In%3A+International+Conference+on+Document+Analysis+and+Recognition+%28ICDAR+2007%29%2E+vol%2E+2%2C+pp%2E+629%E2%80%93633%2E+IEEE+%282007%29+1%2C+4)
35. Vedantam, R., Lawrence Zitnick, C., Parikh, D.: [Cider: Consensus-based image description evaluation](https://md.research.elamur.ai/free/en/papers/vedantam-2014-cider-consensus-based-image-description/). In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4566–4575 (2015) 10
36. Vinyals, O., Fortunato, M., Jaitly, N.: Pointer networks. In: Advances in Neural Information Processing Systems. pp. 2692–2700 (2015) 9 [find](https://scholar.google.com/scholar?q=Vinyals%2C+O%2E%2C+Fortunato%2C+M%2E%2C+Jaitly%2C+N%2E%3A+Pointer+networks%2E+In%3A+Advances+in+Neural+Information+Processing+Systems%2E+pp%2E+2692%E2%80%932700+%282015%29+9)
37. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., Bowman, S.R.: [Glue: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In: Proceedings of International Conference on Learning Representations (2019) 1
38. Young, P., Lai, A., Hodosh, M., Hockenmaier, J.: From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics 2, 67–78 (2014) 3 [find](https://scholar.google.com/scholar?q=Young%2C+P%2E%2C+Lai%2C+A%2E%2C+Hodosh%2C+M%2E%2C+Hockenmaier%2C+J%2E%3A+From+image+descriptions+to+visual+denotations%3A+New+similarity+metrics+for+semantic+inference+over+event+descriptions%2E+Transactions+of+the+Association+f)

*Updated 2 November 2026*