---
title: "Pix2Struct — paper by Lee et al. (2022) — summary and ideas"
description: "Pix2Struct is a general pretrained model for images where text and visuals are mixed, such as… 54 references, cited by 111 papers here."
authors: ["Kenton Lee", "Mandar Joshi", "Iulia Turc", "Hexiang Hu", "Fangyu Liu", "Julian Eisenschlos", "Urvashi Khandelwal", "Peter Shaw", "Ming-Wei Chang", "Kristina Toutanova"]
year: 2022
venue: "arXiv:2210.03347"
source: "https://arxiv.org/abs/2210.03347"
doi: "10.48550/arxiv.2210.03347"
arxiv: "2210.03347"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lee-2022-pix2struct-screenshot-parsing-pretraining-visual/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lee-2022-pix2struct-screenshot-parsing-pretraining-visual/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/) / Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding

# Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding

**Authors:** [Kenton Lee](https://md.research.elamur.ai/free/en/authors/kenton-lee/), [Mandar Joshi](https://md.research.elamur.ai/free/en/authors/mandar-joshi/), [Iulia Turc](https://md.research.elamur.ai/free/en/authors/iulia-turc/), [Hexiang Hu](https://md.research.elamur.ai/free/en/authors/hexiang-hu/), [Fangyu Liu](https://md.research.elamur.ai/free/en/authors/fangyu-liu/), [Julian Eisenschlos](https://md.research.elamur.ai/free/en/authors/julian-eisenschlos/), [Urvashi Khandelwal](https://md.research.elamur.ai/free/en/authors/urvashi-khandelwal/), [Peter Shaw](https://md.research.elamur.ai/free/en/authors/peter-shaw/), [Ming-Wei Chang](https://md.research.elamur.ai/free/en/authors/ming-wei-chang/) and [Kristina Toutanova](https://md.research.elamur.ai/free/en/authors/kristina-toutanova/) · **Year:** 2022 · **Venue:** arXiv:2210.03347 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2210.03347) · [PDF](https://arxiv.org/pdf/2210.03347) · [DOI 10.48550/arxiv.2210.03347](https://doi.org/10.48550/arxiv.2210.03347) · [arXiv 2210.03347](https://arxiv.org/abs/2210.03347) · [OpenAlex](https://openalex.org/W4304192731)

**Citations:** 47 citations (OpenAlex, 2026-10-14) · [111](https://md.research.elamur.ai/free/en/papers/lee-2022-pix2struct-screenshot-parsing-pretraining-visual/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Pix2Struct is a general pretrained model for images where text and visuals are mixed, such as documents, charts, diagrams, app screens, and photos containing text. Earlier systems usually depended on domain-specific inputs like external OCR, Android view hierarchies, or diagram parses. Pix2Struct uses only raw pixels: a ViT-based image encoder feeds a text decoder. For pretraining, the authors rendered web pages and trained the model to rebuild a simplified HTML tree from a screenshot in which parts of the text were masked. This one objective combines reading text, modelling language, and describing images. Two architectural choices support it. Input resolution is variable, so images keep their aspect ratio. Any extra textual input, such as a question or a bounding box, is drawn directly onto the image, so the model works with a single modality. After finetuning, the model set new state-of-the-art results on six of nine benchmarks spanning four domains. Its gains were largest on illustrations and UIs, where training data is scarce. This shows that an OCR-free, general-purpose approach is practical.

## Key points
- Two sizes were released: Pix2Struct-Base (282M parameters) and Pix2Struct-Large (1.3B). Both were pretrained on about 80M web screenshots crawled from URLs in the C4 corpus. The page content was fetched directly, and the C4 text itself was not used.
- Pretraining targets are compressed DOM subtrees that keep only visible text, image filenames, and alt-text. About half of the text spans are masked, in a BART-like way. A warm-up curriculum first teaches the model to read rendered text snippets.
- The ViT input is scaled so that it yields the largest number of patches that fit the sequence budget, and it uses 2D absolute positional embeddings. This handles extreme aspect ratios and allows resolution to change during finetuning.
- Compared with Donut, the strongest pipeline-free baseline, Pix2Struct scores 9–53 points higher. It beats pipeline models by 1–44 points on illustration and UI tasks such as AI2D, OCR-VQA, and Screen2Words. It still trails OCR-based systems on documents (DocVQA) and the much larger GIT2/PaLI on natural images (TextCaps).
- The paper was published at ICML 2023. Code and checkpoints are available at github.com/google-research/pix2struct.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Cambridge](https://md.research.elamur.ai/free/en/universities/university-of-cambridge/) — University, Cambridge, United Kingdom · 256 works · 123,373 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 811 works

## Cite

Cite as: Kenton Lee, Mandar Joshi, Iulia Turc, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang and Kristina Toutanova. 2022. Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding. arXiv:2210.03347.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{lee2022pix2struct,
  title = {{Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding}},
  author = {Kenton Lee, Mandar Joshi, Iulia Turc, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang and Kristina Toutanova},
  year = {2022},
  doi = {10.48550/arxiv.2210.03347},
  eprint = {2210.03347},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2210.03347}
}
```

- [← previous Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Ordering](https://md.research.elamur.ai/free/en/papers/wu-2022-self-adaptive-context-learning-information/)
- [next → A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translation](https://md.research.elamur.ai/free/en/papers/adelani-2022-few-thousand-translations-go-long/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/)

## References

54 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Chen, T., Saxena, S., Li, L., Fleet, D. J., and Hinton, G. Pix2seq: A language modeling framework for object detection. arXiv preprint arXiv:2109.10852, 2021. [find](https://scholar.google.com/scholar?q=Chen%2C+T%2E%2C+Saxena%2C+S%2E%2C+Li%2C+L%2E%2C+Fleet%2C+D%2E+J%2E%2C+and+Hinton%2C+G%2E+Pix2seq%3A+A+language+modeling+framework+for+object+detection%2E+arXiv+preprint+arXiv%3A2109%2E10852%2C+2021%2E)
2. Aghajanyan, A., Huang, B., Ross, C., Karpukhin, V., Xu, H., Goyal, N., Okhonko, D., Joshi, M., Ghosh, G., Lewis, M., et al. Cm3: A causal masked multimodal model of the internet. arXiv preprint arXiv:2201.07520, 2022a. [find](https://scholar.google.com/scholar?q=Aghajanyan%2C+A%2E%2C+Huang%2C+B%2E%2C+Ross%2C+C%2E%2C+Karpukhin%2C+V%2E%2C+Xu%2C+H%2E%2C+Goyal%2C+N%2E%2C+Okhonko%2C+D%2E%2C+Joshi%2C+M%2E%2C+Ghosh%2C+G%2E%2C+Lewis%2C+M%2E%2C+et+al%2E+Cm3%3A+A+causal+masked+multimodal+model+of+the+internet%2E+arXiv+preprint+arXiv%3A)
3. Chen, T., Saxena, S., Li, L., Lin, T.-Y., Fleet, D. J., and Hinton, G. E. A unified sequence interface for vision tasks. Advances in Neural Information Processing Systems, 35:31333–31346, 2022b. [find](https://scholar.google.com/scholar?q=Chen%2C+T%2E%2C+Saxena%2C+S%2E%2C+Li%2C+L%2E%2C+Lin%2C+T%2E%2DY%2E%2C+Fleet%2C+D%2E+J%2E%2C+and+Hinton%2C+G%2E+E%2E+A+unified+sequence+interface+for+vision+tasks%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A31333%E2%80%9331346%2C+2022b%2E)
4. Aghajanyan, A., Okhonko, D., Lewis, M., Joshi, M., Xu, H., Ghosh, G., and Zettlemoyer, L. HTLM: hypertext pre-training and prompting of language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022b. URL [https://openreview.net/…](https://openreview.net/…) [↗](https://openreview.net/…)
5. Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A., Padlewski, P., Salz, D., Goodman, S., Grycner, A., Mustafa, B., Beyer, L., Kolesnikov, A., Puigcerver, J., Ding, N., Rong, K., Akbari, H., Mishra, G., Xue, L., Thapliyal, A., Bradbury, J., Kuo, W., Seyedhosseini, M., Jia, C., Ayan, B. K., Riquelm… — [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://md.research.elamur.ai/free/en/papers/chen-2022-pali-jointly-scaled-multilingual-language/)
6. Appalaraju, S., Jasani, B., Kota, B. U., Xie, Y., and Manmatha, R. DocFormer: End-to-end Transformer for document understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 993–1003, October 2021. [find](https://scholar.google.com/scholar?q=Appalaraju%2C+S%2E%2C+Jasani%2C+B%2E%2C+Kota%2C+B%2E+U%2E%2C+Xie%2C+Y%2E%2C+and+Manmatha%2C+R%2E+DocFormer%3A+End%2Dto%2Dend+Transformer+for+document+understanding%2E+In+Proceedings+of+the+IEEE%2FCVF+International+Conference+on+Computer+Vis)
7. Davis, B., Morse, B., Price, B., Tensmeyer, C., Wigington, C., and Morariu, V. End-to-end document recognition and understanding with Dessurt. In Text in everything ECCV workshop, 2022. URL https://arxiv.org/abs/2203.16618. — [arXiv:2203.16618](https://arxiv.org/abs/2203.16618) [↗](https://arxiv.org/abs/2203.16618)
8. Bai, C., Zang, X., Xu, Y., Sunkara, S., Rastogi, A., Chen, J., and Agüera y Arcas, B. Uibert: Learning generic multimodal representations for ui understanding. In Zhou, Z.-H. (ed.), Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 1705–1712. Inte… — [doi:10.24963/ijcai](https://doi.org/10.24963/ijcai) [find](https://scholar.google.com/scholar?q=Bai%2C+C%2E%2C+Zang%2C+X%2E%2C+Xu%2C+Y%2E%2C+Sunkara%2C+S%2E%2C+Rastogi%2C+A%2E%2C+Chen%2C+J%2E%2C+and+Agu%CC%88era+y+Arcas%2C+B%2E+Uibert%3A+Learning+generic+multimodal+representations+for+ui+understanding%2E+In+Zhou%2C+Z%2E%2DH%2E+%28ed%2E%29%2C+Proceedings+of+th)
9. Deng, Y., Kanervisto, A., Ling, J., and Rush, A. M. Image-to-markup generation with coarse-to-fine attention. In International Conference on Machine Learning, pp. 980–989. PMLR, 2017. [find](https://scholar.google.com/scholar?q=Deng%2C+Y%2E%2C+Kanervisto%2C+A%2E%2C+Ling%2C+J%2E%2C+and+Rush%2C+A%2E+M%2E+Image%2Dto%2Dmarkup+generation+with+coarse%2Dto%2Dfine+attention%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+980%E2%80%93989%2E+PMLR%2C+2017%2E)
10. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and…
11. Biten, A. F., Litman, R., Xie, Y., Appalaraju, S., and Manmatha, R. Latr: Layout-aware transformer for scene-text vqa. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16548– 16558, 2022. [find](https://scholar.google.com/scholar?q=Biten%2C+A%2E+F%2E%2C+Litman%2C+R%2E%2C+Xie%2C+Y%2E%2C+Appalaraju%2C+S%2E%2C+and+Manmatha%2C+R%2E+Latr%3A+Layout%2Daware+transformer+for+scene%2Dtext+vqa%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognit)
12. Borchmann, Ł., Pietruszka, M., Stanislawek, T., Jurkiewicz, D., Turski, M., Szyndler, K., and Graliński, F. Due: End-to-end document understanding benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. [find](https://scholar.google.com/scholar?q=Borchmann%2C+%C5%81%2E%2C+Pietruszka%2C+M%2E%2C+Stanislawek%2C+T%2E%2C+Jurkiewicz%2C+D%2E%2C+Turski%2C+M%2E%2C+Szyndler%2C+K%2E%2C+and+Gralin%CC%81ski%2C+F%2E+Due%3A+End%2Dto%2Dend+document+understanding+benchmark%2E+In+Thirty%2Dfifth+Conference+on+Neural+Info)
13. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. [An image is worth 16x16 words: Transformers for image recognition at scale](https://md.research.elamur.ai/free/en/papers/dosovitskiy-2020-image-worth-16x16-words-transformers/). In ICLR, 2021.
14. Chen, C., Anjum, S., and Gurari, D. Grounding answers for visual questions asked by visually impaired people. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19098–19107, 2022a. [find](https://scholar.google.com/scholar?q=Chen%2C+C%2E%2C+Anjum%2C+S%2E%2C+and+Gurari%2C+D%2E+Grounding+answers+for+visual+questions+asked+by+visually+impaired+people%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+pp%2E+)
15. Eisenschlos, J., Krichene, S., and Müller, T. [Understanding tables with intermediate pre-training](https://md.research.elamur.ai/free/en/papers/eisenschlos-2020-understanding-tables-intermediate-pre-training/). In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 281–296, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.findings-emnlp.27. URL ht…
16. Li, G. and Li, Y. Spotlight: Mobile UI understanding using vision-language models with a focus. In The Eleventh International Conference on Learning Representations, 2023. URL [https://openreview.net/forum?](https://openreview.net/forum?) id=9yE2xEj0BH7. [↗](https://openreview.net/forum?)
17. Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. [Making the v in vqa matter: Elevating the role of image understanding in visual question answering](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/). In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904–6913, 2017.
18. Li, J., Xu, Y., Cui, L., and Wei, F. MarkupLM: Pre-training of text and markup language for visually rich document understanding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6078–6087, Dublin, Ireland, May 2022. Association… [find](https://scholar.google.com/scholar?q=Li%2C+J%2E%2C+Xu%2C+Y%2E%2C+Cui%2C+L%2E%2C+and+Wei%2C+F%2E+MarkupLM%3A+Pre%2Dtraining+of+text+and+markup+language+for+visually+rich+document+understanding%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Compu)
19. He, Z., Sunkara, S., Zang, X., Xu, Y., Liu, L., Wichers, N., Schubiner, G., Lee, R., and Chen, J. Actionbert: Leveraging user actions for semantic understanding of user interfaces. In 35th AAAI Conference on Artificial Intelligence, AAAI 2021, 35th AAAI Conference on Artificial Intelligence, AAAI 20… [find](https://scholar.google.com/scholar?q=He%2C+Z%2E%2C+Sunkara%2C+S%2E%2C+Zang%2C+X%2E%2C+Xu%2C+Y%2E%2C+Liu%2C+L%2E%2C+Wichers%2C+N%2E%2C+Schubiner%2C+G%2E%2C+Lee%2C+R%2E%2C+and+Chen%2C+J%2E+Actionbert%3A+Leveraging+user+actions+for+semantic+understanding+of+user+interfaces%2E+In+35th+AAAI+Confer)
20. Li, Y., He, J., Zhou, X., Zhang, Y., and Baldridge, J. [Mapping natural language instructions to mobile UI action sequences](https://md.research.elamur.ai/free/en/papers/li-2020-mapping-natural-language-instructions-mobile/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8198–8210, Online, July 2020a. Association for Computational Linguistics. doi: 1…
21. Huang, Y., Lv, T., Cui, L., Lu, Y., and Wei, F. [LayoutLMv3: Pre-training for document ai with unified text and image masking](https://md.research.elamur.ai/free/en/papers/huang-2022-layoutlmv3-pre-training-document-ai/). In Proceedings of the 30th ACM International Conference on Multimedia, 2022.
22. Li, Y., Li, G., He, L., Zheng, J., Li, H., and Guan, Z. Widget captioning: Generating natural language description for mobile user interface elements. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5495–5510, Online, November 2020b. Association… [find](https://scholar.google.com/scholar?q=Li%2C+Y%2E%2C+Li%2C+G%2E%2C+He%2C+L%2E%2C+Zheng%2C+J%2E%2C+Li%2C+H%2E%2C+and+Guan%2C+Z%2E+Widget+captioning%3A+Generating+natural+language+description+for+mobile+user+interface+elements%2E+In+Proceedings+of+the+2020+Conference+on+Empirica)
23. Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., and Farhadi, A. [A diagram is worth a dozen images](https://md.research.elamur.ai/free/en/papers/kembhavi-2016-diagram-worth-dozen-images/). In European conference on computer vision, pp. 235– 251. Springer, 2016.
24. Kim, G., Hong, T., Yim, M., Park, J., Yim, J., Hwang, W., Yun, S., Han, D., and Park, S. Donut: Document understanding transformer without OCR. In ECCV, 2022. [URL https://arxiv.org/abs/2111.15664](https://md.research.elamur.ai/free/en/papers/kim-2021-ocr-free-document-understanding-transformer/). [↗](https://arxiv.org/abs/2111.15664)
25. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. [BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). In Proceedings of the 58th Annual Meeting of the Association for Computational…
26. Li, Y., Li, G., Zhou, X., Dehghani, M., and Gritsenko, A. Vut: Versatile ui transformer for multi-modal multi-task user interface modeling. arXiv preprint arXiv:2112.05692, 2021b. — [arXiv:2112.05692](https://arxiv.org/abs/2112.05692) [find](https://scholar.google.com/scholar?q=Li%2C+Y%2E%2C+Li%2C+G%2E%2C+Zhou%2C+X%2E%2C+Dehghani%2C+M%2E%2C+and+Gritsenko%2C+A%2E+Vut%3A+Versatile+ui+transformer+for+multi%2Dmodal+multi%2Dtask+user+interface+modeling%2E+arXiv+preprint+arXiv%3A2112%2E05692%2C+2021b%2E+%E2%80%94+%5BarXiv%3A2112%2E05692%5D)
27. Li, C., Bi, B., Yan, M., Wang, W., Huang, S., Huang, F., and Si, L. StructuralLM: Structural pre-training for form understanding. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol… — [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/)
28. Liu, T. F., Craft, M., Situ, J., Yumer, E., Mech, R., and Kumar, R. Learning design semantics for mobile apps. Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology, 2018. [find](https://scholar.google.com/scholar?q=Liu%2C+T%2E+F%2E%2C+Craft%2C+M%2E%2C+Situ%2C+J%2E%2C+Yumer%2C+E%2E%2C+Mech%2C+R%2E%2C+and+Kumar%2C+R%2E+Learning+design+semantics+for+mobile+apps%2E+Proceedings+of+the+31st+Annual+ACM+Symposium+on+User+Interface+Software+and+Technology%2C+2)
29. Masry, A., Long, D., Tan, J. Q., Joty, S., and Hoque, E. [ChartQA: A benchmark for question answering about charts with visual and logical reasoning](https://md.research.elamur.ai/free/en/papers/masry-2022-chartqa-benchmark-question-answering-about/). In Findings of the Association for Computational Linguistics: ACL 2022, pp. 2263–2279, Dublin, Ireland, May 2022. Association for Computational Linguis…
30. Shazeer, N. and Stern, M. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp. 4596–4604. PMLR, 2018. [find](https://scholar.google.com/scholar?q=Shazeer%2C+N%2E+and+Stern%2C+M%2E+Adafactor%3A+Adaptive+learning+rates+with+sublinear+memory+cost%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+4596%E2%80%934604%2E+PMLR%2C+2018%2E)
31. Mathew, M., Karatzas, D., and Jawahar, C. [Docvqa: A dataset for vqa on document images](https://md.research.elamur.ai/free/en/papers/mathew-2020-docvqa-dataset-vqa-document-images/). In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2200–2209, 2021.
32. Sidorov, O., Hu, R., Rohrbach, M., and Singh, A. Textcaps: a dataset for image captioningwith reading comprehension. In European Conference on Computer Vision, 2020. [find](https://scholar.google.com/scholar?q=Sidorov%2C+O%2E%2C+Hu%2C+R%2E%2C+Rohrbach%2C+M%2E%2C+and+Singh%2C+A%2E+Textcaps%3A+a+dataset+for+image+captioningwith+reading+comprehension%2E+In+European+Conference+on+Computer+Vision%2C+2020%2E)
33. Mathew, M., Bagal, V., Tito, R., Karatzas, D., Valveny, E., and Jawahar, C. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1697–1706, 2022. [find](https://scholar.google.com/scholar?q=Mathew%2C+M%2E%2C+Bagal%2C+V%2E%2C+Tito%2C+R%2E%2C+Karatzas%2C+D%2E%2C+Valveny%2C+E%2E%2C+and+Jawahar%2C+C%2E+Infographicvqa%2E+In+Proceedings+of+the+IEEE%2FCVF+Winter+Conference+on+Applications+of+Computer+Vision%2C+pp%2E+1697%E2%80%931706%2C+2022%2E)
34. Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317–8326, 2019. [find](https://scholar.google.com/scholar?q=Singh%2C+A%2E%2C+Natarajan%2C+V%2E%2C+Shah%2C+M%2E%2C+Jiang%2C+Y%2E%2C+Chen%2C+X%2E%2C+Batra%2C+D%2E%2C+Parikh%2C+D%2E%2C+and+Rohrbach%2C+M%2E+Towards+vqa+models+that+can+read%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+computer+vision+and+patt)
35. Mishra, A., Shekhar, S., Singh, A. K., and Chakraborty, A. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pp. 947–952. [find](https://scholar.google.com/scholar?q=Mishra%2C+A%2E%2C+Shekhar%2C+S%2E%2C+Singh%2C+A%2E+K%2E%2C+and+Chakraborty%2C+A%2E+Ocr%2Dvqa%3A+Visual+question+answering+by+reading+text+in+images%2E+In+2019+international+conference+on+document+analysis+and+recognition+%28ICDAR%29%2C+)
36. Tang, Z., Yang, Z., Wang, G., Fang, Y., Liu, Y., Zhu, C., Zeng, M., Zhang, C., and Bansal, M. Unifying vision, text, and layout for universal document processing. arXiv preprint arXiv:2212.02623, 2022. [find](https://scholar.google.com/scholar?q=Tang%2C+Z%2E%2C+Yang%2C+Z%2E%2C+Wang%2C+G%2E%2C+Fang%2C+Y%2E%2C+Liu%2C+Y%2E%2C+Zhu%2C+C%2E%2C+Zeng%2C+M%2E%2C+Zhang%2C+C%2E%2C+and+Bansal%2C+M%2E+Unifying+vision%2C+text%2C+and+layout+for+universal+document+processing%2E+arXiv+preprint+arXiv%3A2212%2E02623%2C+2022)
37. P18-1238. [URL https://aclanthology.org/ P18-1238](https://md.research.elamur.ai/free/en/papers/sharma-2018-conceptual-captions-cleaned-hypernymed-image/). Touvron, H., Vedaldi, A., Douze, M., and Jegou, H. Fixing the train-test resolution discrepancy. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, vo… [↗](https://aclanthology.org/)
38. Powalski, R., Borchmann, Ł., Jurkiewicz, D., Dwojak, T., Pietruszka, M., and Pałka, G. Going full-tilt boogie on document understanding with text-image-layout transformer. In International Conference on Document Analysis and Recognition, pp. 732–747. Springer, 2021. [find](https://scholar.google.com/scholar?q=Powalski%2C+R%2E%2C+Borchmann%2C+%C5%81%2E%2C+Jurkiewicz%2C+D%2E%2C+Dwojak%2C+T%2E%2C+Pietruszka%2C+M%2E%2C+and+Pa%C5%82ka%2C+G%2E+Going+full%2Dtilt+boogie+on+document+understanding+with+text%2Dimage%2Dlayout+transformer%2E+In+International+Conference+)
39. Press, O., Smith, N., and Lewis, M. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations, 2022. URL [https://openreview.net/forum?id=R8sQPpGCv0](https://openreview.net/forum?id=R8sQPpGCv0). [↗](https://openreview.net/forum?id=R8sQPpGCv0)
40. Wang, B., Li, G., Zhou, X., Chen, Z., Grossman, T., and Li, Y. Screen2words: Automatic mobile ui summarization with multimodal learning. In The 34th Annual ACM Symposium on User Interface Software and Technology, UIST ’21, pp. 498–510, New York, NY, USA, 2021a. Association for Computing Machinery. I… — [doi:10.1145/3472749](https://doi.org/10.1145/3472749) [find](https://scholar.google.com/scholar?q=Wang%2C+B%2E%2C+Li%2C+G%2E%2C+Zhou%2C+X%2E%2C+Chen%2C+Z%2E%2C+Grossman%2C+T%2E%2C+and+Li%2C+Y%2E+Screen2words%3A+Automatic+mobile+ui+summarization+with+multimodal+learning%2E+In+The+34th+Annual+ACM+Symposium+on+User+Interface+Software+and)
41. Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al. Improving language understanding by generative pre-training. OpenAI, 2018. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Narasimhan%2C+K%2E%2C+Salimans%2C+T%2E%2C+Sutskever%2C+I%2E%2C+et+al%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+OpenAI%2C+2018%2E)
42. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research, 21(140):1–67, 2020. URL [http://jmlr.org/papers/v21/20-074.html](http://jmlr.org/papers/v21/20-074.html). [↗](http://jmlr.org/papers/v21/20-074.html)
43. Wang, J., Tang, J., and Luo, J. Multimodal attention with image text spatial relationship for ocr-based image captioning. In Proceedings of the 28th ACM International Conference on Multimedia, pp. 4337–4345, 2020. [find](https://scholar.google.com/scholar?q=Wang%2C+J%2E%2C+Tang%2C+J%2E%2C+and+Luo%2C+J%2E+Multimodal+attention+with+image+text+spatial+relationship+for+ocr%2Dbased+image+captioning%2E+In+Proceedings+of+the+28th+ACM+International+Conference+on+Multimedia%2C+pp%2E+433)
44. Rust, P., Lotz, J. F., Bugliarello, E., Salesky, E., de Lhoneux, M., and Elliott, D. Language modelling with pixels. arXiv preprint arXiv:2207.06991, 2022. — [arXiv:2207.06991](https://arxiv.org/abs/2207.06991) [find](https://scholar.google.com/scholar?q=Rust%2C+P%2E%2C+Lotz%2C+J%2E+F%2E%2C+Bugliarello%2C+E%2E%2C+Salesky%2C+E%2E%2C+de+Lhoneux%2C+M%2E%2C+and+Elliott%2C+D%2E+Language+modelling+with+pixels%2E+arXiv+preprint+arXiv%3A2207%2E06991%2C+2022%2E+%E2%80%94+%5BarXiv%3A2207%2E06991%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2)
45. Wang, J., Yang, Z., Hu, X., Li, L., Lin, K., Gan, Z., Liu, Z., Liu, C., and Wang, L. [Git: A generative image-to-text transformer for vision and language](https://md.research.elamur.ai/free/en/papers/wang-2022-git-generative-image-text-transformer/). arXiv preprint arXiv:2205.14100, 2022a.
46. Sharma, P., Ding, N., Goodman, S., and Soricut, R. [Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning](https://md.research.elamur.ai/free/en/papers/sharma-2018-conceptual-captions-cleaned-hypernymed-image/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556–2565, Melbourne, Australi…
47. Wang, Q., Fang, Y., Ravula, A., Feng, F., Quan, X., and Liu, D. Webformer: The web-page transformer for structure information extraction. In Proceedings of the ACM Web Conference 2022, pp. 3124–3133, 2022b. [find](https://scholar.google.com/scholar?q=Wang%2C+Q%2E%2C+Fang%2C+Y%2E%2C+Ravula%2C+A%2E%2C+Feng%2C+F%2E%2C+Quan%2C+X%2E%2C+and+Liu%2C+D%2E+Webformer%3A+The+web%2Dpage+transformer+for+structure+information+extraction%2E+In+Proceedings+of+the+ACM+Web+Conference+2022%2C+pp%2E+3124%E2%80%933133%2C+)
48. Wang, Z., Yu, J., Yu, A. W., Dai, Z., Tsvetkov, Y., and Cao, Y. [Simvlm: Simple visual language model pretraining with weak supervision](https://md.research.elamur.ai/free/en/papers/wang-2021-simvlm-simple-visual-language-model/). CoRR, abs/2108.10904, 2021b. URL https://arxiv.org/abs/2108.10904. [↗](https://arxiv.org/abs/2108.10904)
49. Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W. [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/). Transactions on Machine Learning Research, 2022. URL https://op…
50. Wu, J., Zhang, X., Nichols, J., and Bigham, J. P. Screen parsing: Towards reverse engineering of UI models from screenshots. In The 34th Annual ACM Symposium on User Interface Software and Technology, pp. 470–483, 2021. [find](https://scholar.google.com/scholar?q=Wu%2C+J%2E%2C+Zhang%2C+X%2E%2C+Nichols%2C+J%2E%2C+and+Bigham%2C+J%2E+P%2E+Screen+parsing%3A+Towards+reverse+engineering+of+UI+models+from+screenshots%2E+In+The+34th+Annual+ACM+Symposium+on+User+Interface+Software+and+Technology%2C)
51. Xu, Y., Xu, Y., Lv, T., Cui, L., Wei, F., Wang, G., Lu, Y., Florencio, D., Zhang, C., Che, W., Zhang, M., and Zhou, L. [Layoutlmv2: Multi-modal pre-training for visually-rich document understanding](https://md.research.elamur.ai/free/en/papers/xu-2021-layoutlmv2-multi-modal-pre-training/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL) 2021,…
52. Yang, Z., Lu, Y., Wang, J., Yin, X., Florencio, D., Wang, L., Zhang, C., Zhang, L., and Luo, J. Tap: Text-aware pre-training for text-vqa and text-caption. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8751–8761, 2021. [find](https://scholar.google.com/scholar?q=Yang%2C+Z%2E%2C+Lu%2C+Y%2E%2C+Wang%2C+J%2E%2C+Yin%2C+X%2E%2C+Florencio%2C+D%2E%2C+Wang%2C+L%2E%2C+Zhang%2C+C%2E%2C+Zhang%2C+L%2E%2C+and+Luo%2C+J%2E+Tap%3A+Text%2Daware+pre%2Dtraining+for+text%2Dvqa+and+text%2Dcaption%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on)
53. Zhang, X., de Greef, L., Swearngin, A., White, S., Murray, K., Yu, L., Shan, Q., Nichols, J., Wu, J., Fleizach, C., et al. Screen recognition: Creating accessibility metadata for mobile applications from pixels. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–1… [find](https://scholar.google.com/scholar?q=Zhang%2C+X%2E%2C+de+Greef%2C+L%2E%2C+Swearngin%2C+A%2E%2C+White%2C+S%2E%2C+Murray%2C+K%2E%2C+Yu%2C+L%2E%2C+Shan%2C+Q%2E%2C+Nichols%2C+J%2E%2C+Wu%2C+J%2E%2C+Fleizach%2C+C%2E%2C+et+al%2E+Screen+recognition%3A+Creating+accessibility+metadata+for+mobile+applications+f)
54. Zhu, Y., Kiros, R., Zemel, R. S., Salakhutdinov, R., Urtasun, R., Torralba, A., and Fidler, S. [Aligning books and movies: Towards story-like visual explanations by watching movies and reading books](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/). 2015 IEEE International Conference on Computer Vision (ICCV), pp. 19– 27, 2015.

*Updated 2 November 2026*