---
title: "Can Pre-trained Vision and Language Models Answer Visual"
description: "They introduce INFOSEEK, which has two parts: A human-annotated test set. 9.4 points. 68 references, cited by 54 papers here."
authors: ["Yang Chen", "Hexiang Hu", "Yi Luan", "Haitian Sun", "Soravit Changpinyo", "Alan Ritter", "Ming-Wei Chang"]
year: 2023
venue: "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2023.emnlp-main.925/"
doi: "10.18653/v1/2023.emnlp-main.925"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chen-2023-can-pre-trained-vision-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chen-2023-can-pre-trained-vision-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?

# Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?

**Authors:** Yang Chen, [Hexiang Hu](https://md.research.elamur.ai/free/en/authors/hexiang-hu/), [Yi Luan](https://md.research.elamur.ai/free/en/authors/yi-luan/), Haitian Sun, [Soravit Changpinyo](https://md.research.elamur.ai/free/en/authors/soravit-changpinyo/), [Alan Ritter](https://md.research.elamur.ai/free/en/authors/alan-ritter/) and [Ming-Wei Chang](https://md.research.elamur.ai/free/en/authors/ming-wei-chang/) · **Year:** 2023 · **Venue:** [Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2023](https://md.research.elamur.ai/free/en/venues/emnlp/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.emnlp-main.925/) · [PDF](https://aclanthology.org/2023.emnlp-main.925.pdf) · [DOI 10.18653/v1/2023.emnlp-main.925](https://doi.org/10.18653/v1/2023.emnlp-main.925) · [OpenAlex](https://openalex.org/W4389523839)

**Citations:** 55 citations (OpenAlex, 2026-10-14) · 54 in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper asks whether current pre-trained vision-language models can handle questions about an image that need specific factual knowledge not visible in the picture, such as when a pictured church was built. The authors argue that existing knowledge-based VQA benchmarks are poorly suited to this. OK-VQA mostly tests common knowledge. ViQuAE questions can often be solved from the text alone. Other benchmarks focus heavily on people.

They introduce INFOSEEK, which has two parts:
- **A human-annotated test set.** One group of annotators writes questions, and a separate group finds answers in Wikipedia.
- **A large training corpus.** It is generated semi-automatically by linking OVEN entity annotations to Wikidata through human-written templates.

The data splits are designed so that memorizing training facts does not help.

Experiments compare end-to-end models (PaLI-X, BLIP2) with pipeline systems that link entities via CLIP and answer from Wikipedia with PaLM or FiD. Zero-shot models perform poorly. Fine-tuning substantially improves PaLI-X, and knowledge-base pipelines win overall. End-to-end models still do better on rare entities. Better entity recognition offers large gains, pointing to a key direction for knowledge-intensive multimodal systems.

## Key points
- INFOSEEK has two components. INFOSEEK Human contains about 8.9K human-written questions. INFOSEEK Wikidata contains roughly 1.35M automatically generated {image, question, answer} triplets, with images drawn from 9 classification/retrieval datasets.
- Humans could answer 70.8% of sampled OK-VQA questions without a search engine, but only 4.4% of INFOSEEK questions.
- A question-only PaLM (540B) baseline reaches 31.5% on ViQuAE, which is 9.4 points above the image-using SOTA.
- Improving visual entity recognition raises performance from 18% to 45.6%. On human queries, FiD beats PaLM on TIME (21.5 vs 14.6) and NUMERICAL (25.6 vs 21.3) questions.
- PaLI-17B outperforms CLIP→PaLM/FiD pipelines on low-pageview (tail) entities, especially for geographic questions. The dataset covers English only.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 353 works · 364,806 citations
- [Georgia Institute of Technology](https://md.research.elamur.ai/free/en/universities/georgia-institute-of-technology/) — University, Atlanta, United States · 69 works · 22,788 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 269 works

## Cite

Cite as: Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter and Ming-Wei Chang. 2023. Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{chen2023can,
  title = {{Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?}},
  author = {Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter and Ming-Wei Chang},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.925},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.925}
}
```

- [← previous Benchmarking Machine Translation with Cultural Awareness](https://md.research.elamur.ai/free/en/papers/yao-2023-benchmarking-machine-translation-cultural-awareness/)
- [next → CodeIE: Large Code Generation Models are Better Few-Shot Information Extractors](https://md.research.elamur.ai/free/en/papers/li-2023-codeie-large-code-generation-models/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

68 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. 2018. Don’t just assume; look and answer: Overcoming priors for visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4971–4980. [find](https://scholar.google.com/scholar?q=Aishwarya+Agrawal%2C+Dhruv+Batra%2C+Devi+Parikh%2C+and+Aniruddha+Kembhavi%2E+2018%2E+Don%E2%80%99t+just+assume%3B+look+and+answer%3A+Overcoming+priors+for+visual+question+answering%2E+In+Proceedings+of+the+IEEE+conference+on)
2. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. 2022. [Flamingo: a visual language model for few-shot learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/). Conference on Neural Information Processing Systems.
3. Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. 2021. Beit: Bert pre-training of image transformers. International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Hangbo+Bao%2C+Li+Dong%2C+Songhao+Piao%2C+and+Furu+Wei%2E+2021%2E+Beit%3A+Bert+pre%2Dtraining+of+image+transformers%2E+International+Conference+on+Learning+Representations%2E)
4. Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. 2014. Food-101–mining discriminative components with random forests. In European Conference on Computer Vision. Springer. [find](https://scholar.google.com/scholar?q=Lukas+Bossard%2C+Matthieu+Guillaumin%2C+and+Luc+Van+Gool%2E+2014%2E+Food%2D101%E2%80%93mining+discriminative+components+with+random+forests%2E+In+European+Conference+on+Computer+Vision%2E+Springer%2E)
5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Conference on Neural Information Processing Systems.
6. Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, et al. 2023a. Pali-x: On scaling up a multilingual vision and language model. arXiv preprint arXiv:2305.18565. — [arXiv:2305.18565](https://arxiv.org/abs/2305.18565) [find](https://scholar.google.com/scholar?q=Xi+Chen%2C+Josip+Djolonga%2C+Piotr+Padlewski%2C+Basil+Mustafa%2C+Soravit+Changpinyo%2C+Jialin+Wu%2C+Carlos+Riquelme+Ruiz%2C+Sebastian+Goodman%2C+Xiao+Wang%2C+Yi+Tay%2C+et+al%2E+2023a%2E+Pali%2Dx%3A+On+scaling+up+a+multilingual+v)
7. Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. 2023b. [PaLi: A jointly-scaled multilingual language-image model](https://md.research.elamur.ai/free/en/papers/chen-2022-pali-jointly-scaled-multilingual-language/). International Conference on Learning Representations.
8. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv preprint arXiv:2204.02311.
9. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. [Scaling instruction-finetuned language models](https://md.research.elamur.ai/free/en/papers/chung-2022-scaling-instruction-finetuned-language-models/). arXiv preprint arXiv:2210.11416.
10. Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. [TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages](https://md.research.elamur.ai/free/en/papers/clark-2020-tydi-qa-benchmark-information-seeking/). Transactions of the Association for Computational Linguistics.
11. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. [Instructblip: Towards general-purpose vision-language models with instruction tuning](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/). arXiv preprint arXiv:2305.06500.
12. Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. 2023. Scaling vision transformers to 22 billion parameters. arXiv preprint arXiv:2302.05442. [find](https://scholar.google.com/scholar?q=Mostafa+Dehghani%2C+Josip+Djolonga%2C+Basil+Mustafa%2C+Piotr+Padlewski%2C+Jonathan+Heek%2C+Justin+Gilmer%2C+Andreas+Steiner%2C+Mathilde+Caron%2C+Robert+Geirhos%2C+Ibrahim+Alabdulmohsin%2C+et+al%2E+2023%2E+Scaling+vision+tran)
13. Gerry. 2021. 100 sports image classification. [find](https://scholar.google.com/scholar?q=Gerry%2E+2021%2E+100+sports+image+classification%2E)
14. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. [Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/). In Computer Vision and Pattern Recognition.
15. Liangke Gui, Borui Wang, Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, and Jianfeng Gao. 2022. KAT: A knowledge augmented transformer for vision-and-language. Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. [find](https://scholar.google.com/scholar?q=Liangke+Gui%2C+Borui+Wang%2C+Qiuyuan+Huang%2C+Alex+Hauptmann%2C+Yonatan+Bisk%2C+and+Jianfeng+Gao%2E+2022%2E+KAT%3A+A+knowledge+augmented+transformer+for+vision%2Dand%2Dlanguage%2E+Proceedings+of+the+Conference+of+the+North)
16. Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. 2018. Vizwiz Grand Challenge: Answering visual questions from blind people. In Computer Vision and Pattern Recognition. [find](https://scholar.google.com/scholar?q=Danna+Gurari%2C+Qing+Li%2C+Abigale+J%2E+Stangl%2C+Anhong+Guo%2C+Chi+Lin%2C+Kristen+Grauman%2C+Jiebo+Luo%2C+and+Jeffrey+P%2E+Bigham%2E+2018%2E+Vizwiz+Grand+Challenge%3A+Answering+visual+questions+from+blind+people%2E+In+Compute)
17. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). arXiv preprint arXiv:2106.09685.
18. Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming-Wei Chang. 2023. Open-domain visual entity recognition: Towards recognizing millions of wikipedia entities. arXiv preprint arXiv:2302.11154. — [arXiv:2302.11154](https://arxiv.org/abs/2302.11154) [find](https://scholar.google.com/scholar?q=Hexiang+Hu%2C+Yi+Luan%2C+Yang+Chen%2C+Urvashi+Khandelwal%2C+Mandar+Joshi%2C+Kenton+Lee%2C+Kristina+Toutanova%2C+and+Ming%2DWei+Chang%2E+2023%2E+Open%2Ddomain+visual+entity+recognition%3A+Towards+recognizing+millions+of+wikip)
19. Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A Smith, and Jiebo Luo. 2022a. Promptcap: Prompt-guided task-aware image captioning. arXiv preprint arXiv:2211.09699. — [arXiv:2211.09699](https://arxiv.org/abs/2211.09699) [find](https://scholar.google.com/scholar?q=Yushi+Hu%2C+Hang+Hua%2C+Zhengyuan+Yang%2C+Weijia+Shi%2C+Noah+A+Smith%2C+and+Jiebo+Luo%2E+2022a%2E+Promptcap%3A+Prompt%2Dguided+task%2Daware+image+captioning%2E+arXiv+preprint+arXiv%3A2211%2E09699%2E+%E2%80%94+%5BarXiv%3A2211%2E09699%5D%28https%3A%2F%2F)
20. Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang, Kai-Wei Chang, Yizhou Sun, Cordelia Schmid, David A Ross, and Alireza Fathi. 2022b. Reveal: Retrieval-augmented visual-language pre-training with multi-source multimodal knowledge memory. arXiv preprint arXiv:2212.05221. — [arXiv:2212.05221](https://arxiv.org/abs/2212.05221) [find](https://scholar.google.com/scholar?q=Ziniu+Hu%2C+Ahmet+Iscen%2C+Chen+Sun%2C+Zirui+Wang%2C+Kai%2DWei+Chang%2C+Yizhou+Sun%2C+Cordelia+Schmid%2C+David+A+Ross%2C+and+Alireza+Fathi%2E+2022b%2E+Reveal%3A+Retrieval%2Daugmented+visual%2Dlanguage+pre%2Dtraining+with+multi%2Dsou)
21. Robert Iv, Alexandre Passos, Sameer Singh, and Ming-Wei Chang. 2022. FRUIT: Faithfully reflecting updated information in text. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, United States. Associa… [find](https://scholar.google.com/scholar?q=Robert+Iv%2C+Alexandre+Passos%2C+Sameer+Singh%2C+and+Ming%2DWei+Chang%2E+2022%2E+FRUIT%3A+Faithfully+reflecting+updated+information+in+text%2E+In+Proceedings+of+the+Conference+of+the+North+American+Chapter+of+the+Ass)
22. Gautier Izacard and Edouard Grave. 2020. [Leveraging passage retrieval with generative models for open domain question answering](https://md.research.elamur.ai/free/en/papers/izacard-2021-leveraging-passage-retrieval-generative-models/). Proceedings of the European Chapter of the Association for Computational Linguistics.
23. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023b. [Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/). International Conference on Machine Learning.
24. Aman Jain, Mayank Kothyari, Vishwajeet Kumar, Preethi Jyothi, Ganesh Ramakrishnan, and Soumen Chakrabarti. 2021. Select, substitute, search: A new benchmark for knowledge-augmented visual question answering. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in… [find](https://scholar.google.com/scholar?q=Aman+Jain%2C+Mayank+Kothyari%2C+Vishwajeet+Kumar%2C+Preethi+Jyothi%2C+Ganesh+Ramakrishnan%2C+and+Soumen+Chakrabarti%2E+2021%2E+Select%2C+substitute%2C+search%3A+A+new+benchmark+for+knowledge%2Daugmented+visual+question+ans)
25. Yuanze Lin, Yujia Xie, Dongdong Chen, Yichong Xu, Chenguang Zhu, and Lu Yuan. 2022. REVIVE: Regional visual representation matters in knowledge-based visual question answering. Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Yuanze+Lin%2C+Yujia+Xie%2C+Dongdong+Chen%2C+Yichong+Xu%2C+Chenguang+Zhu%2C+and+Lu+Yuan%2E+2022%2E+REVIVE%3A+Regional+visual+representation+matters+in+knowledge%2Dbased+visual+question+answering%2E+Conference+on+Neural+In)
26. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. [TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). In Annual Meeting of the Association for Computational Linguistics.
27. Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2022. [Large language models struggle to learn long-tail knowledge](https://md.research.elamur.ai/free/en/papers/kandpal-2022-large-language-models-struggle-learn/). arXiv preprint arXiv:2211.08411.
28. Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. 2023. Grounding language models to images for multimodal generation. International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Jing+Yu+Koh%2C+Ruslan+Salakhutdinov%2C+and+Daniel+Fried%2E+2023%2E+Grounding+language+models+to+images+for+multimodal+generation%2E+International+Conference+on+Machine+Learning%2E)
29. Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 2013. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops. [find](https://scholar.google.com/scholar?q=Jonathan+Krause%2C+Michael+Stark%2C+Jia+Deng%2C+and+Li+Fei%2DFei%2E+2013%2E+3d+object+representations+for+fine%2Dgrained+categorization%2E+In+Proceedings+of+the+IEEE+international+conference+on+computer+vision+worksh)
30. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. [Natural…](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
31. Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. [Latent retrieval for weakly supervised open domain question answering](https://md.research.elamur.ai/free/en/papers/lee-2019-latent-retrieval-weakly-supervised-open/). In Annual Meeting of the Association for Computational Linguistics.
32. Paul Lerner, Olivier Ferret, Camille Guinaudeau, Hervé Le Borgne, Romaric Besançon, Jose G. Moreno, and Jesús Lovón Melgarejo. 2022. Viquae, a dataset for knowledge-based visual question answering about named entities. In Proceedings of the 45th International ACM SIGIR Conference on Research and Dev… [find](https://scholar.google.com/scholar?q=Paul+Lerner%2C+Olivier+Ferret%2C+Camille+Guinaudeau%2C+Herv%C3%A9+Le+Borgne%2C+Romaric+Besan%C3%A7on%2C+Jose+G%2E+Moreno%2C+and+Jes%C3%BAs+Lov%C3%B3n+Melgarejo%2E+2022%2E+Viquae%2C+a+dataset+for+knowledge%2Dbased+visual+question+answering+abo)
33. Dongxu Li, Junnan Li, Hung Le, Guangsen Wang, Silvio Savarese, and Steven C.H. Hoi. 2023a. LAVIS: A one-stop library for language-vision intelligence. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations). Association for Computa… [find](https://scholar.google.com/scholar?q=Dongxu+Li%2C+Junnan+Li%2C+Hung+Le%2C+Guangsen+Wang%2C+Silvio+Savarese%2C+and+Steven+C%2EH%2E+Hoi%2E+2023a%2E+LAVIS%3A+A+one%2Dstop+library+for+language%2Dvision+intelligence%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+)
34. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). arXiv preprint arXiv:2304.08485.
35. Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. 2022. Unified-io: A unified model for vision, language, and multimodal tasks. arXiv preprint arXiv:2206.08916. — [arXiv:2206.08916](https://arxiv.org/abs/2206.08916) [find](https://scholar.google.com/scholar?q=Jiasen+Lu%2C+Christopher+Clark%2C+Rowan+Zellers%2C+Roozbeh+Mottaghi%2C+and+Aniruddha+Kembhavi%2E+2022%2E+Unified%2Dio%3A+A+unified+model+for+vision%2C+language%2C+and+multimodal+tasks%2E+arXiv+preprint+arXiv%3A2206%2E08916%2E+%E2%80%94+)
36. Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. 2013. Fine-grained visual classification of aircraft. Technical report. [find](https://scholar.google.com/scholar?q=Subhransu+Maji%2C+Esa+Rahtu%2C+Juho+Kannala%2C+Matthew+Blaschko%2C+and+Andrea+Vedaldi%2E+2013%2E+Fine%2Dgrained+visual+classification+of+aircraft%2E+Technical+report%2E)
37. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Hannaneh Hajishirzi, and Daniel Khashabi. 2022. When not to trust language models: Investigating effectiveness and limitations of parametric and non-parametric memories. arXiv preprint arXiv:2212.10511. [find](https://scholar.google.com/scholar?q=Alex+Mallen%2C+Akari+Asai%2C+Victor+Zhong%2C+Rajarshi+Das%2C+Hannaneh+Hajishirzi%2C+and+Daniel+Khashabi%2E+2022%2E+When+not+to+trust+language+models%3A+Investigating+effectiveness+and+limitations+of+parametric+and+no)
38. Kenneth Marino, Xinlei Chen, Devi Parikh, Abhinav Gupta, and Marcus Rohrbach. 2021. Krisp: Integrating implicit and symbolic knowledge for open-domain knowledge-based vqa. In Computer Vision and Pattern Recognition. [find](https://scholar.google.com/scholar?q=Kenneth+Marino%2C+Xinlei+Chen%2C+Devi+Parikh%2C+Abhinav+Gupta%2C+and+Marcus+Rohrbach%2E+2021%2E+Krisp%3A+Integrating+implicit+and+symbolic+knowledge+for+open%2Ddomain+knowledge%2Dbased+vqa%2E+In+Computer+Vision+and+Patte)
39. Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. OK-VQA: A visual question answering benchmark requiring external knowledge. In Computer Vision and Pattern Recognition. [find](https://scholar.google.com/scholar?q=Kenneth+Marino%2C+Mohammad+Rastegari%2C+Ali+Farhadi%2C+and+Roozbeh+Mottaghi%2E+2019%2E+OK%2DVQA%3A+A+visual+question+answering+benchmark+requiring+external+knowledge%2E+In+Computer+Vision+and+Pattern+Recognition%2E)
40. Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. [ChartQA: A benchmark for question answering about charts with visual and logical reasoning](https://md.research.elamur.ai/free/en/papers/masry-2022-chartqa-benchmark-question-answering-about/). Annual Meeting of the Association for Computational Linguistics (Findings).
41. Thomas Mensink, Jasper Uijlings, Lluis Castrejon, Arushi Goel, Felipe Cadar, Howard Zhou, Fei Sha, André Araujo, and Vittorio Ferrari. 2023. Encyclopedic vqa: Visual questions about detailed properties of fine-grained categories. arXiv preprint arXiv:2306.09224. — [arXiv:2306.09224](https://arxiv.org/abs/2306.09224) [find](https://scholar.google.com/scholar?q=Thomas+Mensink%2C+Jasper+Uijlings%2C+Lluis+Castrejon%2C+Arushi+Goel%2C+Felipe+Cadar%2C+Howard+Zhou%2C+Fei+Sha%2C+Andr%C3%A9+Araujo%2C+and+Vittorio+Ferrari%2E+2023%2E+Encyclopedic+vqa%3A+Visual+questions+about+detailed+propertie)
42. Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. 2020. PlotQA: Reasoning over scientific plots. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1527–1536. [find](https://scholar.google.com/scholar?q=Nitesh+Methani%2C+Pritha+Ganguly%2C+Mitesh+M+Khapra%2C+and+Pratyush+Kumar%2E+2020%2E+PlotQA%3A+Reasoning+over+scientific+plots%2E+In+Proceedings+of+the+IEEE%2FCVF+Winter+Conference+on+Applications+of+Computer+Vision%2C)
43. Maria-Elena Nilsback and Andrew Zisserman. 2008. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing. [find](https://scholar.google.com/scholar?q=Maria%2DElena+Nilsback+and+Andrew+Zisserman%2E+2008%2E+Automated+flower+classification+over+a+large+number+of+classes%2E+In+2008+Sixth+Indian+Conference+on+Computer+Vision%2C+Graphics+%26+Image+Processing%2E)
44. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). In International Conference on Machine Learning.
45. Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, W. Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). JMLR.
46. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [SQuAD: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). Proceedings of Emperical Methods in Natural Language Processing.
47. Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. 2021. Imagenet-21k pretraining for the masses. arXiv preprint arXiv:2104.10972. — [arXiv:2104.10972](https://arxiv.org/abs/2104.10972) [find](https://scholar.google.com/scholar?q=Tal+Ridnik%2C+Emanuel+Ben%2DBaruch%2C+Asaf+Noy%2C+and+Lihi+Zelnik%2DManor%2E+2021%2E+Imagenet%2D21k+pretraining+for+the+masses%2E+arXiv+preprint+arXiv%3A2104%2E10972%2E+%E2%80%94+%5BarXiv%3A2104%2E10972%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2104%2E10972%29)
48. Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? CoRR, abs/2002.08910. [find](https://scholar.google.com/scholar?q=Adam+Roberts%2C+Colin+Raffel%2C+and+Noam+Shazeer%2E+2020%2E+How+much+knowledge+can+you+pack+into+the+parameters+of+a+language+model%3F+CoRR%2C+abs%2F2002%2E08910%2E)
49. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015. [Imagenet large scale visual recognition challenge](https://md.research.elamur.ai/free/en/papers/russakovsky-2014-imagenet-large-scale-visual-recognition/). International journal of computer vision, 115:211–252.
50. Naganand Yadati Sanket Shah, Anand Mishra and Partha Pratim Talukdar. 2019. KVQA: Knowledge-aware visual question answering. [find](https://scholar.google.com/scholar?q=Naganand+Yadati+Sanket+Shah%2C+Anand+Mishra+and+Partha+Pratim+Talukdar%2E+2019%2E+KVQA%3A+Knowledge%2Daware+visual+question+answering%2E)
51. Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. [A-okvqa: A benchmark for visual question answering using world knowledge](https://md.research.elamur.ai/free/en/papers/schwenk-2022-okvqa-benchmark-visual-question-answering/). arXiv preprint arXiv:2206.01718.
52. Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. 2022. [Unifying language learning paradigms](https://md.research.elamur.ai/free/en/papers/tay-2022-ul2-unifying-language-learning-paradigms/). arXiv preprint arXiv:2205.05131.
53. Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. 2018. The inaturalist species classification and detection dataset. In Computer Vision and Pattern Recognition. [find](https://scholar.google.com/scholar?q=Grant+Van+Horn%2C+Oisin+Mac+Aodha%2C+Yang+Song%2C+Yin+Cui%2C+Chen+Sun%2C+Alex+Shepard%2C+Hartwig+Adam%2C+Pietro+Perona%2C+and+Serge+Belongie%2E+2018%2E+The+inaturalist+species+classification+and+detection+dataset%2E+In+Com)
54. Denny Vrandečić and Markus Krötzsch. 2014. Wikidata: a free collaborative knowledgebase. Communications of the ACM. [find](https://scholar.google.com/scholar?q=Denny+Vrandec%CC%8Cic%CC%81+and+Markus+Kr%C3%B6tzsch%2E+2014%2E+Wikidata%3A+a+free+collaborative+knowledgebase%2E+Communications+of+the+ACM%2E)
55. Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton Van Den Hengel. 2017. FVQA: Fact-based visual question answering. IEEE transactions on pattern analysis and machine intelligence. [find](https://scholar.google.com/scholar?q=Peng+Wang%2C+Qi+Wu%2C+Chunhua+Shen%2C+Anthony+Dick%2C+and+Anton+Van+Den+Hengel%2E+2017%2E+FVQA%3A+Fact%2Dbased+visual+question+answering%2E+IEEE+transactions+on+pattern+analysis+and+machine+intelligence%2E)
56. Peng Wang, Qi Wu, Chunhua Shen, Anton van den Hengel, and Anthony Dick. 2015. Explicit knowledge-based reasoning for visual question answering. arXiv preprint arXiv:1511.02570. — [arXiv:1511.02570](https://arxiv.org/abs/1511.02570) [find](https://scholar.google.com/scholar?q=Peng+Wang%2C+Qi+Wu%2C+Chunhua+Shen%2C+Anton+van+den+Hengel%2C+and+Anthony+Dick%2E+2015%2E+Explicit+knowledge%2Dbased+reasoning+for+visual+question+answering%2E+arXiv+preprint+arXiv%3A1511%2E02570%2E+%E2%80%94+%5BarXiv%3A1511%2E02570%5D%28ht)
57. Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022. [Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework](https://md.research.elamur.ai/free/en/papers/wang-2022-ofa-unifying-architectures-tasks-modalities/). International Conference on Machine Learning.
58. Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. 2020. Google landmarks dataset v2-a large-scale benchmark for instance-level recognition and retrieval. In Computer Vision and Pattern Recognition. [find](https://scholar.google.com/scholar?q=Tobias+Weyand%2C+Andre+Araujo%2C+Bingyi+Cao%2C+and+Jack+Sim%2E+2020%2E+Google+landmarks+dataset+v2%2Da+large%2Dscale+benchmark+for+instance%2Dlevel+recognition+and+retrieval%2E+In+Computer+Vision+and+Pattern+Recognitio)
59. Jialin Wu and Raymond J Mooney. 2022. Entity-focused dense passage retrieval for outside-knowledge visual question answering. Proceedings of Emperical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Jialin+Wu+and+Raymond+J+Mooney%2E+2022%2E+Entity%2Dfocused+dense+passage+retrieval+for+outside%2Dknowledge+visual+question+answering%2E+Proceedings+of+Emperical+Methods+in+Natural+Language+Processing%2E)
60. Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. 2010. Sun database: Large-scale scene recognition from abbey to zoo. In Computer Vision and Pattern Recognition. [find](https://scholar.google.com/scholar?q=Jianxiong+Xiao%2C+James+Hays%2C+Krista+A+Ehinger%2C+Aude+Oliva%2C+and+Antonio+Torralba%2E+2010%2E+Sun+database%3A+Large%2Dscale+scene+recognition+from+abbey+to+zoo%2E+In+Computer+Vision+and+Pattern+Recognition%2E)
61. Zhiyang Xu, Ying Shen, and Lifu Huang. 2023. Multiinstruct: Improving multi-modal zero-shot learning via instruction tuning. Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zhiyang+Xu%2C+Ying+Shen%2C+and+Lifu+Huang%2E+2023%2E+Multiinstruct%3A+Improving+multi%2Dmodal+zero%2Dshot+learning+via+instruction+tuning%2E+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2E)
62. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. [mT5: A massively multilingual pre-trained text-to-text transformer](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/). Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics:…
63. Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. 2022. An empirical study of GPT-3 for few-shot knowledge-based VQA. In Proceedings of the AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Zhengyuan+Yang%2C+Zhe+Gan%2C+Jianfeng+Wang%2C+Xiaowei+Hu%2C+Yumao+Lu%2C+Zicheng+Liu%2C+and+Lijuan+Wang%2E+2022%2E+An+empirical+study+of+GPT%2D3+for+few%2Dshot+knowledge%2Dbased+VQA%2E+In+Proceedings+of+the+AAAI+Conference+on)
64. Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. 2023. [mplug-owl: Modularization empowers large language models with multimodality](https://md.research.elamur.ai/free/en/papers/ye-2023-mplug-owl-modularization-empowers-large/). arXiv preprint arXiv:2304.14178.
65. Victor Zhong, Weijia Shi, Wen-tau Yih, and Luke Zettlemoyer. 2022. RoMQA: A benchmark for robust, multi-evidence, multi-answer question answering. arXiv preprint arXiv:2210.14353. [find](https://scholar.google.com/scholar?q=Victor+Zhong%2C+Weijia+Shi%2C+Wen%2Dtau+Yih%2C+and+Luke+Zettlemoyer%2E+2022%2E+RoMQA%3A+A+benchmark+for+robust%2C+multi%2Devidence%2C+multi%2Danswer+question+answering%2E+arXiv+preprint+arXiv%3A2210%2E14353%2E)
66. Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. 2020. Unified vision-language pre-training for image captioning and VQA. In Proceedings of the AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Luowei+Zhou%2C+Hamid+Palangi%2C+Lei+Zhang%2C+Houdong+Hu%2C+Jason+Corso%2C+and+Jianfeng+Gao%2E+2020%2E+Unified+vision%2Dlanguage+pre%2Dtraining+for+image+captioning+and+VQA%2E+In+Proceedings+of+the+AAAI+Conference+on+Arti)
67. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. [Minigpt-4: Enhancing vision-language understanding with advanced large language models](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/). arXiv preprint arXiv:2304.10592.
68. Train Val Test Human \\\#U NSEEN Q UESTION /E NTITY \\\#Total Question Type T IME /N UM ./S TR . \\\#Entity -/-18,656/54,964 98,901/249,079 3,248/5,683 934,048 73,620 347,980 8,931 4.4/20.4/ 75.2% 4.6/ 21.6/ 73.8% 4.8/22.9/72.3% 26.8/ 26.4/46.8% 5,549 1,794 8,905 806 Percentage *Table 7: I NFO S EEK Datas… [find](https://scholar.google.com/scholar?q=Train+Val+Test+Human+%5C%5C%5C%23U+NSEEN+Q+UESTION+%2FE+NTITY+%5C%5C%5C%23Total+Question+Type+T+IME+%2FN+UM+%2E%2FS+TR+%2E+%5C%5C%5C%23Entity+%2D%2F%2D18%2C656%2F54%2C964+98%2C901%2F249%2C079+3%2C248%2F5%2C683+934%2C048+73%2C620+347%2C980+8%2C931+4%2E4%2F20%2E4%2F+75%2E2%25+4%2E6)

*Updated 24 October 2026*