---
title: "IconQA — paper by Lu et al. (2021) — summary and ideas"
description: "This paper proposes Icon Question Answering (IconQA): a model sees a colourful icon picture and a… 57 references, cited by 94 papers here."
authors: ["Pan Lu", "Liang Qiu", "Jiaqi Chen", "Tony Xia", "Yizhou Zhao", "Wei Zhang", "Zhou Yu", "Xiaodan Liang", "Song-Chun Zhu"]
year: 2021
venue: "arXiv:2110.13214"
source: "https://arxiv.org/abs/2110.13214"
doi: "10.48550/arxiv.2110.13214"
arxiv: "2110.13214"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lu-2021-iconqa-new-benchmark-abstract-diagram/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lu-2021-iconqa-new-benchmark-abstract-diagram/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning

# IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning

**Authors:** [Pan Lu](https://md.research.elamur.ai/free/en/authors/pan-lu/), [Liang Qiu](https://md.research.elamur.ai/free/en/authors/liang-qiu/), [Jiaqi Chen](https://md.research.elamur.ai/free/en/authors/jiaqi-chen/), [Tony Xia](https://md.research.elamur.ai/free/en/authors/tony-xia/), Yizhou Zhao, [Wei Zhang](https://md.research.elamur.ai/free/en/authors/wei-zhang/), [Zhou Yu](https://md.research.elamur.ai/free/en/authors/zhou-yu/), [Xiaodan Liang](https://md.research.elamur.ai/free/en/authors/xiaodan-liang/) and [Song-Chun Zhu](https://md.research.elamur.ai/free/en/authors/song-chun-zhu/) · **Year:** 2021 · **Venue:** arXiv:2110.13214 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2110.13214) · [PDF](https://arxiv.org/pdf/2110.13214) · [DOI 10.48550/arxiv.2110.13214](https://doi.org/10.48550/arxiv.2110.13214) · [arXiv 2110.13214](https://arxiv.org/abs/2110.13214) · [OpenAlex](https://openalex.org/W3212452301)

**Citations:** 9 citations (OpenAlex, 2026-10-14) · [94](https://md.research.elamur.ai/free/en/papers/lu-2021-iconqa-new-benchmark-abstract-diagram/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most visual question answering (VQA) benchmarks use natural photos. Abstract diagrams, which also carry rich meaning, get little attention. Earlier diagram datasets are either template-generated or demand specialised subject knowledge. This paper proposes Icon Question Answering (IconQA): a model sees a colourful icon picture and a question and must give the answer. The 107,439 problems come from elementary math exercises (pre-K to grade 3, collected from IXL). They are split into three sub-tasks: choosing an image, choosing a text option, and filling in a blank. Solving them takes perception plus reasoning such as counting, arithmetic, spatial and geometric reasoning, and commonsense.

The authors also release Icon645, a large icon collection for pre-training. They benchmark eight existing VQA models, collect human answers, and run blind tests. They then introduce Patch-TRM, which splits the diagram into a pyramid of patches, encodes them with an icon-pretrained ResNet101, and fuses them with the question in a cross-modal Transformer. Patch-TRM beats prior VQA models on all sub-tasks, but humans remain better overall. This makes IconQA a useful testbed for diagram reasoning and educational AI.

## Key points
- **IconQA**: 107,439 questions in a 6:2:2 split (64,462 train, 21,489 validation, 21,489 test). It covers three sub-tasks and thirteen reasoning skill types, and questions involve 1.63 skills on average.
- **Icon645**: 645,687 coloured icons in 377 classes with a long-tailed distribution. ResNet101 trained with LDAM loss reached the best Top-5 accuracy, 62.93%.
- **Baselines**: four attention-based pooling models and four pre-trained Transformer models were tested. Question-only and image-only blind runs scored near chance, which suggests the dataset has few shortcuts.
- **Human study**: 54,896 answers were gathered from children, adolescents and adults. Humans led on most skills, but Patch-TRM did better on estimation, measurement and probability.
- **Ablation**: replacing icon pre-training with natural-image pre-training lowered accuracy by 0.95–2.49%. Cross-modal attention added up to 7% over simple concatenation.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Columbia University](https://md.research.elamur.ai/free/en/universities/columbia-university/) — University, New York, United States · 254 works · 56,747 citations
- [University of California, Los Angeles](https://md.research.elamur.ai/free/en/universities/university-of-california-los-angeles/) — University, Los Angeles, United States · 99 works · 56,333 citations
- [Sun Yat-sen University](https://md.research.elamur.ai/free/en/universities/sun-yat-sen-university/) — University, Guangzhou, China · 72 works · 9,502 citations
- [East China Normal University](https://md.research.elamur.ai/free/en/universities/east-china-normal-university/) — University, Shanghai, China · 69 works · 2,490 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (3): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 811 works

## Cite

Cite as: Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang and Song-Chun Zhu. 2021. IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning. arXiv:2110.13214.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{lu2021iconqa,
  title = {{IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning}},
  author = {Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang and Song-Chun Zhu},
  year = {2021},
  doi = {10.48550/arxiv.2110.13214},
  eprint = {2110.13214},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2110.13214}
}
```

- [← previous Findings of the Shared Task on Offensive Language Identification in Tamil, Malayalam, and Kannada](https://md.research.elamur.ai/free/en/papers/chakravarthi-2021-findings-shared-task-offensive-language/)
- [next → Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation](https://md.research.elamur.ai/free/en/papers/deng-2021-compression-transduction-creation-unified-framework/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/)

## References

57 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Vedika Agarwal, Rakshith Shetty, and Mario Fritz. Towards causal vqa: Revealing and reducing spurious correlations by invariant and covariant semantic editing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9690–9698, 2020. [find](https://scholar.google.com/scholar?q=Vedika+Agarwal%2C+Rakshith+Shetty%2C+and+Mario+Fritz%2E+Towards+causal+vqa%3A+Revealing+and+reducing+spurious+correlations+by+invariant+and+covariant+semantic+editing%2E+In+Proceedings+of+the+IEEE+Conference+on)
2. Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. [find](https://scholar.google.com/scholar?q=Peter+Anderson%2C+Xiaodong+He%2C+Chris+Buehler%2C+Damien+Teney%2C+Mark+Johnson%2C+Stephen+Gould%2C+and+Lei+Zhang%2E+Bottom%2Dup+and+top%2Ddown+attention+for+image+captioning+and+visual+question+answering%2E+In+Proceeding)
3. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision (CVPR), pages 2425–2433, 2015. [find](https://scholar.google.com/scholar?q=Stanislaw+Antol%2C+Aishwarya+Agrawal%2C+Jiasen+Lu%2C+Margaret+Mitchell%2C+Dhruv+Batra%2C+C+Lawrence+Zitnick%2C+and+Devi+Parikh%2E+Vqa%3A+Visual+question+answering%2E+In+Proceedings+of+the+IEEE+international+conference+)
4. Jack Bandy and Nicholas Vincent. Addressing" documentation debt" in machine learning research: A retrospective datasheet for bookcorpus. arXiv preprint arXiv:2105.05241, 2021. — [arXiv:2105.05241](https://arxiv.org/abs/2105.05241) [find](https://scholar.google.com/scholar?q=Jack+Bandy+and+Nicholas+Vincent%2E+Addressing%22+documentation+debt%22+in+machine+learning+research%3A+A+retrospective+datasheet+for+bookcorpus%2E+arXiv+preprint+arXiv%3A2105%2E05241%2C+2021%2E+%E2%80%94+%5BarXiv%3A2105%2E05241%5D%28htt)
5. Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-aware margin loss. In Advances in Neural Information Processing Systems (NeurIPS), pages 1567–1578, 2019. [find](https://scholar.google.com/scholar?q=Kaidi+Cao%2C+Colin+Wei%2C+Adrien+Gaidon%2C+Nikos+Arechiga%2C+and+Tengyu+Ma%2E+Learning+imbalanced+datasets+with+label%2Ddistribution%2Daware+margin+loss%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NeurIP)
6. Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. [Uniter: Universal image-text representation learning](https://md.research.elamur.ai/free/en/papers/chen-2019-uniter-universal-image-text-representation/). In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
7. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.[1078, 2014](https://md.research.elamur.ai/free/en/papers/cho-2014-learning-phrase-representations-rnn-encoder/).
8. Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9268–9277, 2019. [find](https://scholar.google.com/scholar?q=Yin+Cui%2C+Menglin+Jia%2C+Tsung%2DYi+Lin%2C+Yang+Song%2C+and+Serge+Belongie%2E+Class%2Dbalanced+loss+based+on+effective+number+of+samples%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+Recogn)
9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of NAACL-HLT (NAACL), 2018.
10. Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. [Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm](https://md.research.elamur.ai/free/en/papers/felbo-2017-millions-emoji-occurrences-learn-any/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 16…
11. Peng Gao, Zhengkai Jiang, Haoxuan You, Pan Lu, Steven CH Hoi, Xiaogang Wang, and Hongsheng Li. Dynamic fusion with intra-and inter-modality attention flow for visual question answering. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6639–6648, 2019. [find](https://scholar.google.com/scholar?q=Peng+Gao%2C+Zhengkai+Jiang%2C+Haoxuan+You%2C+Pan+Lu%2C+Steven+CH+Hoi%2C+Xiaogang+Wang%2C+and+Hongsheng+Li%2E+Dynamic+fusion+with+intra%2Dand+inter%2Dmodality+attention+flow+for+visual+question+answering%2E+In+The+IEEE+Co)
12. Peng Gao, Hongsheng Li, Shuang Li, Pan Lu, Yikang Li, Steven CH Hoi, and Xiaogang Wang. Question-guided hybrid convolution for visual question answering. In The European Conference on Computer Vision (ECCV), pages 469–485, 2018. [find](https://scholar.google.com/scholar?q=Peng+Gao%2C+Hongsheng+Li%2C+Shuang+Li%2C+Pan+Lu%2C+Yikang+Li%2C+Steven+CH+Hoi%2C+and+Xiaogang+Wang%2E+Question%2Dguided+hybrid+convolution+for+visual+question+answering%2E+In+The+European+Conference+on+Computer+Vision+)
13. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. Datasheets for datasets. arXiv preprint arXiv:1803.09010, 2018. [find](https://scholar.google.com/scholar?q=Timnit+Gebru%2C+Jamie+Morgenstern%2C+Briana+Vecchione%2C+Jennifer+Wortman+Vaughan%2C+Hanna+Wallach%2C+Hal+Daum%C3%A9+III%2C+and+Kate+Crawford%2E+Datasheets+for+datasets%2E+arXiv+preprint+arXiv%3A1803%2E09010%2C+2018%2E)
14. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. [Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/). In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
15. Kotaro Hara, Abigail Adams, Kristy Milland, Saiph Savage, Chris Callison-Burch, and Jeffrey P Bigham. A data-driven analysis of workers’ earnings on amazon mechanical turk. In Proceedings of the 2018 CHI conference on human factors in computing systems, pages 1–14, 2018. [find](https://scholar.google.com/scholar?q=Kotaro+Hara%2C+Abigail+Adams%2C+Kristy+Milland%2C+Saiph+Savage%2C+Chris+Callison%2DBurch%2C+and+Jeffrey+P+Bigham%2E+A+data%2Ddriven+analysis+of+workers%E2%80%99+earnings+on+amazon+mechanical+turk%2E+In+Proceedings+of+the+2018+)
16. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. [Deep residual learning for image recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/). In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 770–778, 2016.
17. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+J%C3%BCrgen+Schmidhuber%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A1735%E2%80%931780%2C+1997%2E)
18. Drew A Hudson and Christopher D Manning. [Gqa: A new dataset for real-world visual reasoning and compositional question answering](https://md.research.elamur.ai/free/en/papers/hudson-2019-gqa-new-dataset-real-world/). In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6700–6709, 2019.
19. Huaizu Jiang, Ishan Misra, Marcus Rohrbach, Erik Learned-Miller, and Xinlei Chen. In defense of grid features for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 10267–10276, 2020. [find](https://scholar.google.com/scholar?q=Huaizu+Jiang%2C+Ishan+Misra%2C+Marcus+Rohrbach%2C+Erik+Learned%2DMiller%2C+and+Xinlei+Chen%2E+In+defense+of+grid+features+for+visual+question+answering%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+an)
20. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. [Tinybert: Distilling bert for natural language understanding](https://md.research.elamur.ai/free/en/papers/jiao-2020-tinybert-distilling-bert-natural-language/). arXiv preprint arXiv:1909.10351, 2019.
21. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. [Clevr: A diagnostic dataset for compositional language and elementary visual reasoning](https://md.research.elamur.ai/free/en/papers/johnson-2016-clevr-diagnostic-dataset-compositional-language/). In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2901–29…
22. Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5648–5656, 2018. [find](https://scholar.google.com/scholar?q=Kushal+Kafle%2C+Brian+Price%2C+Scott+Cohen%2C+and+Christopher+Kanan%2E+Dvqa%3A+Understanding+data+visualizations+via+question+answering%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+Reco)
23. Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. [Figureqa: An annotated figure dataset for visual reasoning](https://md.research.elamur.ai/free/en/papers/kahou-2017-figureqa-annotated-figure-dataset-visual/). arXiv preprint arXiv:1710.07300, 2017.
24. Takuro Karamatsu, Gibran Benitez-Garcia, Keiji Yanai, and Seiichi Uchida. Iconify: Converting photographs into icons. In Proceedings of the 2020 Joint Workshop on Multimedia Artworks Analysis and Attractiveness Computing in Multimedia, pages 7–12, 2020. [find](https://scholar.google.com/scholar?q=Takuro+Karamatsu%2C+Gibran+Benitez%2DGarcia%2C+Keiji+Yanai%2C+and+Seiichi+Uchida%2E+Iconify%3A+Converting+photographs+into+icons%2E+In+Proceedings+of+the+2020+Joint+Workshop+on+Multimedia+Artworks+Analysis+and+Attr)
25. Sevilay Karamustafaoğlu. Improving the science process skills ability of science student teachers using i diagrams. International Journal of Physics & Chemistry Education, 3(1):26–38, 2011. [find](https://scholar.google.com/scholar?q=Sevilay+Karamustafaog%CC%86lu%2E+Improving+the+science+process+skills+ability+of+science+student+teachers+using+i+diagrams%2E+International+Journal+of+Physics+%26+Chemistry+Education%2C+3%281%29%3A26%E2%80%9338%2C+2011%2E)
26. Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Min Joon Seo, Hannaneh Hajishirzi, and Ali Farhadi. [A diagram is worth a dozen images](https://md.research.elamur.ai/free/en/papers/kembhavi-2016-diagram-worth-dozen-images/). In Proceedings of the European Conference on Computer Vision (ECCV), 2016.
27. Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages… [find](https://scholar.google.com/scholar?q=Aniruddha+Kembhavi%2C+Minjoon+Seo%2C+Dustin+Schwenk%2C+Jonghyun+Choi%2C+Ali+Farhadi%2C+and+Hannaneh+Hajishirzi%2E+Are+you+smarter+than+a+sixth+grader%3F+textbook+question+answering+for+multimodal+machine+comprehens)
28. Jin-Hwa Kim, Jaehyun Jun, and Byoung-Tak Zhang. Bilinear attention networks. In Advances in Neural Information Processing Systems (NeurIPS), pages 1571–1581, 2018. [find](https://scholar.google.com/scholar?q=Jin%2DHwa+Kim%2C+Jaehyun+Jun%2C+and+Byoung%2DTak+Zhang%2E+Bilinear+attention+networks%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NeurIPS%29%2C+pages+1571%E2%80%931581%2C+2018%2E)
29. Wonjae Kim, Bokyung Son, and Ildoo Kim. [Vilt: Vision-and-language transformer without convolution or region supervision](https://md.research.elamur.ai/free/en/papers/kim-2021-vilt-vision-language-transformer-convolution/). In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning (ICML), volume 139 of Proceedings of Machine Learning Research, pages…
30. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. [Visual genome: Connecting language and vision using crowdsourced dense image annotations](https://md.research.elamur.ai/free/en/papers/krishna-2016-visual-genome-connecting-language-vision/). International journal of computer vision, pages 32–73, 20…
31. Manuel Lagunas, Elena Garces, and Diego Gutierrez. Learning icons appearance similarity. Multimedia Tools and Applications, 78(8):10733–10751, 2019. [find](https://scholar.google.com/scholar?q=Manuel+Lagunas%2C+Elena+Garces%2C+and+Diego+Gutierrez%2E+Learning+icons+appearance+similarity%2E+Multimedia+Tools+and+Applications%2C+78%288%29%3A10733%E2%80%9310751%2C+2019%2E)
32. Li-Jia Li, Hao Su, Li Fei-Fei, and Eric P Xing. Object bank: A high-level image representation for scene classification & semantic feature sparsification. In Advances in Neural Information Processing Systems (NeurIPS), pages 1378–1386, 2010. [find](https://scholar.google.com/scholar?q=Li%2DJia+Li%2C+Hao+Su%2C+Li+Fei%2DFei%2C+and+Eric+P+Xing%2E+Object+bank%3A+A+high%2Dlevel+image+representation+for+scene+classification+%26+semantic+feature+sparsification%2E+In+Advances+in+Neural+Information+Processing+)
33. Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. [Visualbert: A simple and performant baseline for vision and language](https://md.research.elamur.ai/free/en/papers/li-2019-visualbert-simple-performant-baseline-vision/). arXiv preprint arXiv:1908.03557, 2019.
34. Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. [Focal loss for dense object detection](https://md.research.elamur.ai/free/en/papers/lin-2017-focal-loss-dense-object-detection/). In Proceedings of the IEEE international conference on computer vision (ICCV), pages 2980– 2988, 2017.
35. Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long-tailed recognition in an open world. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2537–2546, 2019. [find](https://scholar.google.com/scholar?q=Ziwei+Liu%2C+Zhongqi+Miao%2C+Xiaohang+Zhan%2C+Jiayun+Wang%2C+Boqing+Gong%2C+and+Stella+X+Yu%2E+Large%2Dscale+long%2Dtailed+recognition+in+an+open+world%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pa)
36. Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. [Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks](https://md.research.elamur.ai/free/en/papers/lu-2019-vilbert-pretraining-task-agnostic-visiolinguistic/). In Advances in Neural Information Processing Systems (NeurIPS), pages 13–23, 2019.
37. Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. [Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning](https://md.research.elamur.ai/free/en/papers/lu-2021-inter-gps-interpretable-geometry-problem/). In The 59th Annual Meeting of the Association for Computational Linguistics (ACL), 2021.
38. Pan Lu, Lei Ji, Wei Zhang, Nan Duan, Ming Zhou, and Jianyong Wang. R-vqa: learning visual relation facts with semantic attention for visual question answering. In The ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (SIGKDD), pages 1880–1889, 2018. [find](https://scholar.google.com/scholar?q=Pan+Lu%2C+Lei+Ji%2C+Wei+Zhang%2C+Nan+Duan%2C+Ming+Zhou%2C+and+Jianyong+Wang%2E+R%2Dvqa%3A+learning+visual+relation+facts+with+semantic+attention+for+visual+question+answering%2E+In+The+ACM+SIGKDD+International+Conferen)
39. Pan Lu, Hongsheng Li, Wei Zhang, Jianyong Wang, and Xiaogang Wang. Co-attending free-form regions and detections with multi-modal multiplicative feature embedding for visual question answering. In The AAAI Conference on Artificial Intelligence (AAAI), 2018. [find](https://scholar.google.com/scholar?q=Pan+Lu%2C+Hongsheng+Li%2C+Wei+Zhang%2C+Jianyong+Wang%2C+and+Xiaogang+Wang%2E+Co%2Dattending+free%2Dform+regions+and+detections+with+multi%2Dmodal+multiplicative+feature+embedding+for+visual+question+answering%2E+In+The)
40. Spandan Madan, Zoya Bylinskii, Matthew Tancik, Adrià Recasens, Kimberli Zhong, Sami Alsheikh, Hanspeter Pfister, Aude Oliva, and Fredo Durand. Synthetically trained icon proposals for parsing and summarizing infographics. arXiv preprint arXiv:1807.10441, 2018. — [arXiv:1807.10441](https://arxiv.org/abs/1807.10441) [find](https://scholar.google.com/scholar?q=Spandan+Madan%2C+Zoya+Bylinskii%2C+Matthew+Tancik%2C+Adri%C3%A0+Recasens%2C+Kimberli+Zhong%2C+Sami+Alsheikh%2C+Hanspeter+Pfister%2C+Aude+Oliva%2C+and+Fredo+Durand%2E+Synthetically+trained+icon+proposals+for+parsing+and+summ)
41. Maria Martiniello. Language and the performance of english-language learners in math word problems. Harvard Educational Review, 78(2):333–368, 2008. [find](https://scholar.google.com/scholar?q=Maria+Martiniello%2E+Language+and+the+performance+of+english%2Dlanguage+learners+in+math+word+problems%2E+Harvard+Educational+Review%2C+78%282%29%3A333%E2%80%93368%2C+2008%2E)
42. Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. [Faster r-cnn: Towards real-time object detection with region proposal networks](https://md.research.elamur.ai/free/en/papers/ren-2015-faster-r-cnn-towards-real/). In Advances in neural information processing systems (NeurIPS), pages 91–99, 2015.
43. Mrinmaya Sachan, Kumar Dubey, and Eric Xing. From textbooks to knowledge: A case study in harvesting axiomatic knowledge from textbooks to solve geometry problems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 773–784, 2017. [find](https://scholar.google.com/scholar?q=Mrinmaya+Sachan%2C+Kumar+Dubey%2C+and+Eric+Xing%2E+From+textbooks+to+knowledge%3A+A+case+study+in+harvesting+axiomatic+knowledge+from+textbooks+to+solve+geometry+problems%2E+In+Proceedings+of+the+2017+Conferenc)
44. Mrinmaya Sachan, Kumar Avinava Dubey, Tom M Mitchell, Dan Roth, and Eric P Xing. Learning pipelines with limited data and domain knowledge: A study in parsing physics problems. In Advances in Neural Information Processing Systems (NeurIPS), pages 140–151, 2018. [find](https://scholar.google.com/scholar?q=Mrinmaya+Sachan%2C+Kumar+Avinava+Dubey%2C+Tom+M+Mitchell%2C+Dan+Roth%2C+and+Eric+P+Xing%2E+Learning+pipelines+with+limited+data+and+domain+knowledge%3A+A+study+in+parsing+physics+problems%2E+In+Advances+in+Neural+I)
45. Mike Schuster and Kaisuke Nakajima. Japanese and korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149–5152. IEEE, 2012. [find](https://scholar.google.com/scholar?q=Mike+Schuster+and+Kaisuke+Nakajima%2E+Japanese+and+korean+voice+search%2E+In+2012+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+%28ICASSP%29%2C+pages+5149%E2%80%935152%2E+IEEE%2C+2012%2E)
46. Minjoon Seo, Hannaneh Hajishirzi, Ali Farhadi, Oren Etzioni, and Clint Malcolm. [Solving geometry problems: Combining text and diagram interpretation](https://md.research.elamur.ai/free/en/papers/seo-2015-solving-geometry-problems-combining-text/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1466–1476, 2015.
47. Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8317–8326, 2019. [find](https://scholar.google.com/scholar?q=Amanpreet+Singh%2C+Vivek+Natarajan%2C+Meet+Shah%2C+Yu+Jiang%2C+Xinlei+Chen%2C+Dhruv+Batra%2C+Devi+Parikh%2C+and+Marcus+Rohrbach%2E+Towards+vqa+models+that+can+read%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+V)
48. Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi. A corpus of natural language for visual reasoning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pages 217–223, 2017. [find](https://scholar.google.com/scholar?q=Alane+Suhr%2C+Mike+Lewis%2C+James+Yeh%2C+and+Yoav+Artzi%2E+A+corpus+of+natural+language+for+visual+reasoning%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28ACL%29%2C+)
49. Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [Well-read students learn better: On the importance of pre-training compact models](https://md.research.elamur.ai/free/en/papers/turc-2019-well-read-students-learn-better/). arXiv preprint arXiv:1908.08962, 2019.
50. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in neural information processing systems (NeurIPS), pages 5998–6008, 2017.
51. Shuo Wang, Yizhou Wang, and Song-Chun Zhu. Learning hierarchical space tiling for scene modeling, parsing and attribute tagging. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(12):2478–2491, 2015. [find](https://scholar.google.com/scholar?q=Shuo+Wang%2C+Yizhou+Wang%2C+and+Song%2DChun+Zhu%2E+Learning+hierarchical+space+tiling+for+scene+modeling%2C+parsing+and+attribute+tagging%2E+IEEE+Transactions+on+Pattern+Analysis+and+Machine+Intelligence%2C+37%2812%29%3A)
52. Xinyu Wang, Yuliang Liu, Chunhua Shen, Chun Chet Ng, Canjie Luo, Lianwen Jin, Chee Seng Chan, Anton van den Hengel, and Liangwei Wang. On the general value of evidence, and bilingual scene-text visual question answering. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020. [find](https://scholar.google.com/scholar?q=Xinyu+Wang%2C+Yuliang+Liu%2C+Chunhua+Shen%2C+Chun+Chet+Ng%2C+Canjie+Luo%2C+Lianwen+Jin%2C+Chee+Seng+Chan%2C+Anton+van+den+Hengel%2C+and+Liangwei+Wang%2E+On+the+general+value+of+evidence%2C+and+bilingual+scene%2Dtext+visual)
53. Ildoo Kim Wonjae Kim, Bokyung Son. [An image is worth 16x16 words: Transformers for image recognition at scale](https://md.research.elamur.ai/free/en/papers/dosovitskiy-2020-image-worth-16x16-words-transformers/). In International Conference on Learning Representations (ICLR), 2021. under review.
54. Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6281–6290, 2019. [find](https://scholar.google.com/scholar?q=Zhou+Yu%2C+Jun+Yu%2C+Yuhao+Cui%2C+Dacheng+Tao%2C+and+Qi+Tian%2E+Deep+modular+co%2Dattention+networks+for+visual+question+answering%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition)
55. Peng Zhang, Yash Goyal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Yin and Yang: Balancing and answering binary visual questions. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016. [find](https://scholar.google.com/scholar?q=Peng+Zhang%2C+Yash+Goyal%2C+Douglas+Summers%2DStay%2C+Dhruv+Batra%2C+and+Devi+Parikh%2E+Yin+and+Yang%3A+Balancing+and+answering+binary+visual+questions%2E+In+Conference+on+Computer+Vision+and+Pattern+Recognition+%28CVP)
56. Jun Zhu, Tianfu Wu, Song-Chun Zhu, Xiaokang Yang, and Wenjun Zhang. A reconfigurable tangram model for scene representation and categorization. IEEE Transactions on Image Processing, 25(1):150–166, 2015. [find](https://scholar.google.com/scholar?q=Jun+Zhu%2C+Tianfu+Wu%2C+Song%2DChun+Zhu%2C+Xiaokang+Yang%2C+and+Wenjun+Zhang%2E+A+reconfigurable+tangram+model+for+scene+representation+and+categorization%2E+IEEE+Transactions+on+Image+Processing%2C+25%281%29%3A150%E2%80%93166%2C+20)
57. Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. [Visual7w: Grounded question answering in images](https://md.research.elamur.ai/free/en/papers/zhu-2015-visual7w-grounded-question-answering-images/). In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.

*Updated 2 November 2026*