---
title: "Efficient Visual Grounding in VQA via Question-Guided Sparse"
description: "The paper proposes Question-Guided Sparse Attention (QGSA), a lightweight plug-in module that sits after a frozen image… 28 references."
authors: ["Prasanth Yadla"]
year: 2026
venue: "Proceedings of the 4th Workshop on Advances in Language and Vision Research (ALVR)"
source: "https://aclanthology.org/2026.alvr-main.24/"
doi: "10.18653/v1/2026.alvr-main.24"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yadla-2026-efficient-visual-grounding-vqa-question/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yadla-2026-efficient-visual-grounding-vqa-question/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/31/) / Efficient Visual Grounding in VQA via Question-Guided Sparse Attention

# Efficient Visual Grounding in VQA via Question-Guided Sparse Attention

**Authors:** [Prasanth Yadla](https://md.research.elamur.ai/free/en/authors/prasanth-yadla/) · **Year:** 2026 · **Venue:** [Proceedings of the 4th Workshop on Advances in Language and Vision Research (ALVR)](https://md.research.elamur.ai/free/en/venues/alvr/) ([ALVR 2026](https://md.research.elamur.ai/free/en/venues/alvr/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.alvr-main.24/) · [PDF](https://aclanthology.org/2026.alvr-main.24.pdf) · [DOI 10.18653/v1/2026.alvr-main.24](https://doi.org/10.18653/v1/2026.alvr-main.24) · [OpenAlex](https://openalex.org/W7166810357)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Vision–language models usually pass every image patch into cross-attention, even though a question typically concerns only a few regions. This wastes compute and spreads attention over distracting areas. The paper proposes Question-Guided Sparse Attention (QGSA), a lightweight plug-in module that sits after a frozen image encoder and before the language model. For every patch, the module computes a relevance score with respect to the question, then retains a hard top-k selection via Gumbel-Softmax paired with a straight-through estimator. It also predicts the budget k from the question. Training needs no bounding boxes: a contrastive term compares chosen patches with random ones, and a frozen CLIP encoder aligns patches with question nouns.

The author tested QGSA on SmolVLM-256M-Instruct and SmolVLM-500M-Instruct with VQA-RAD, A-OKVQA and RefCOCO. It cut cross-attention FLOPs by 91–99% without changing accuracy. The caveats are substantial. Absolute accuracy is close to chance, grounding is weak, and the adaptive budget collapses to a nearly constant value. The work is mainly a feasibility check that motivates experiments on larger VLMs.

## Key points
- **Efficiency and accuracy:** Cross-attention FLOPs drop by 91–99% across resolutions, with a theoretical speedup of up to 76× at 576px. Accuracy is identical to the dense baseline (Δ = 0.0 pp) on all three datasets.
- **Patch selection:** About 17 of 576 patches are kept on the 256M model, and about 18 on the 500M model. This shrinks the visual token sequence by up to 34×.
- **Wall-clock time:** Speed matches the dense baseline only from 336px upward. On small models, the scorer can add roughly 33–50% latency.
- **Grounding:** IoU on RefCOCO is 0.31%. This is 31× better than random selection, but far too low for practical localisation.
- **Module cost and main limitation:** The module has about 1.2M trainable parameters on SmolVLM-256M, adds under 1% to model size, and trains in under 2 hours on one GPU. The adaptive budget estimator (k_min = 4, k_max = 32) ends up behaving as a fixed-k selector, with the same budget across question types on VQA-RAD.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (1): [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 811 works

## Cite

Cite as: Prasanth Yadla. 2026. Efficient Visual Grounding in VQA via Question-Guided Sparse Attention. Proceedings of the 4th Workshop on Advances in Language and Vision Research (ALVR).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yadla2026efficient,
  title = {{Efficient Visual Grounding in VQA via Question-Guided Sparse Attention}},
  author = {Prasanth Yadla},
  booktitle = {Proceedings of the 4th Workshop on Advances in Language and Vision Research (ALVR)},
  year = {2026},
  doi = {10.18653/v1/2026.alvr-main.24},
  url = {https://doi.org/10.18653/v1/2026.alvr-main.24}
}
```

- [← previous Efficient universal generation in a fragment of Optimality Theory](https://md.research.elamur.ai/free/en/papers/siewert-2026-efficient-universal-generation-fragment-optimality/)
- [next → EfficientLLM: Unified Pruning-Aware Pretraining for Auto-Designed Compact Language Models](https://md.research.elamur.ai/free/en/papers/xing-2026-efficientllm-unified-pruning-aware-pretraining/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/31/)

## References

28 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Vikhyat Agarwal. 2024. moondream2: A tiny vision-language model. [find](https://scholar.google.com/scholar?q=Vikhyat+Agarwal%2E+2024%2E+moondream2%3A+A+tiny+vision%2Dlanguage+model%2E)
2. Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Dhruv Batra, and Devi Parikh. 2016. Vqa: Visual question answering. Preprint, arXiv:1505.00468. [find](https://scholar.google.com/scholar?q=Aishwarya+Agrawal%2C+Jiasen+Lu%2C+Stanislaw+Antol%2C+Margaret+Mitchell%2C+C%2E+Lawrence+Zitnick%2C+Dhruv+Batra%2C+and+Devi+Parikh%2E+2016%2E+Vqa%3A+Visual+question+answering%2E+Preprint%2C+arXiv%3A1505%2E00468%2E)
3. Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. 2021. [Perceiver: General perception with iterative attention](https://md.research.elamur.ai/free/en/papers/jaegle-2021-perceiver-general-perception-iterative-attention/). Preprint, arXiv:2103.03206.
4. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, and 8 others… — [Flamingo: a Visual Language Model for Few-Shot Learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/)
5. Eric Jang, Shixiang Gu, and Ben Poole. 2017. [Categorical reparameterization with gumbel-softmax](https://md.research.elamur.ai/free/en/papers/jang-2016-categorical-reparameterization-gumbel-softmax/). Preprint, arXiv:1611.01144.
6. Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. Preprint, arXiv:1308.3432. [find](https://scholar.google.com/scholar?q=Yoshua+Bengio%2C+Nicholas+L%C3%A9onard%2C+and+Aaron+Courville%2E+2013%2E+Estimating+or+propagating+gradients+through+stochastic+neurons+for+conditional+computation%2E+Preprint%2C+arXiv%3A1308%2E3432%2E)
7. Jin-Hwa Kim, Jaehyun Jun, and Byoung-Tak Zhang. 2018. Bilinear attention networks. In Conference on Neural Information Processing Systems (NeurIPS). [find](https://scholar.google.com/scholar?q=Jin%2DHwa+Kim%2C+Jaehyun+Jun%2C+and+Byoung%2DTak+Zhang%2E+2018%2E+Bilinear+attention+networks%2E+In+Conference+on+Neural+Information+Processing+Systems+%28NeurIPS%29%2E)
8. Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. 2023. Token merging: Your vit but faster. Preprint, arXiv:2210.09461. — [arXiv:2210.09461](https://arxiv.org/abs/2210.09461) [find](https://scholar.google.com/scholar?q=Daniel+Bolya%2C+Cheng%2DYang+Fu%2C+Xiaoliang+Dai%2C+Peizhao+Zhang%2C+Christoph+Feichtenhofer%2C+and+Judy+Hoffman%2E+2023%2E+Token+merging%3A+Your+vit+but+faster%2E+Preprint%2C+arXiv%3A2210%2E09461%2E+%E2%80%94+%5BarXiv%3A2210%2E09461%5D%28https%3A%2F)
9. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. [Generating long sequences with sparse transformers](https://md.research.elamur.ai/free/en/papers/child-2019-generating-long-sequences-sparse-transformers/). arXiv preprint arXiv:1904.10509.
10. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. [Instructblip: Towards general-purpose vision-language models with instruction tuning](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/). Preprint, arXiv:2305.06500.
11. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. [An image is worth 16x16 words: Transformers for image recognition at scale](https://md.research.elamur.ai/free/en/papers/dosovitskiy-2020-image-worth-16x16-words-transformers/). Preprint,…
12. Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. 2021. Mdetr – modulated detection for end-to-end multi-modal understanding. Preprint, arXiv:2104.12763. [find](https://scholar.google.com/scholar?q=Aishwarya+Kamath%2C+Mannat+Singh%2C+Yann+LeCun%2C+Gabriel+Synnaeve%2C+Ishan+Misra%2C+and+Nicolas+Carion%2E+2021%2E+Mdetr+%E2%80%93+modulated+detection+for+end%2Dto%2Dend+multi%2Dmodal+understanding%2E+Preprint%2C+arXiv%3A2104%2E12763%2E)
13. Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Henning Müller. 2018. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1–10. [find](https://scholar.google.com/scholar?q=Jason+J+Lau%2C+Soumya+Gayen%2C+Asma+Ben+Abacha%2C+and+Henning+M%C3%BCller%2E+2018%2E+A+dataset+of+clinically+generated+visual+questions+and+answers+about+radiology+images%2E+Scientific+data%2C+5%281%29%3A1%E2%80%9310%2E)
14. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023a. [Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/). Preprint, arXiv:2301.12597.
15. Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. 2022. Grounded language-image pre-training. Preprint, arXiv:2112.03857. [find](https://scholar.google.com/scholar?q=Liunian+Harold+Li%2C+Pengchuan+Zhang%2C+Haotian+Zhang%2C+Jianwei+Yang%2C+Chunyuan+Li%2C+Yiwu+Zhong%2C+Lijuan+Wang%2C+Lu+Yuan%2C+Lei+Zhang%2C+Jenq%2DNeng+Hwang%2C+Kai%2DWei+Chang%2C+and+Jianfeng+Gao%2E+2022%2E+Grounded+language%2Dima)
16. Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023b. [Evaluating object hallucination in large vision-language models](https://md.research.elamur.ai/free/en/papers/li-2023-evaluating-object-hallucination-large-vision/). Preprint, arXiv:2305.10355.
17. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). In Conference on Neural Information Processing Systems (NeurIPS).
18. Hugging Face. 2024. Smolvlm: Small vision-language models. [find](https://scholar.google.com/scholar?q=Hugging+Face%2E+2024%2E+Smolvlm%3A+Small+vision%2Dlanguage+models%2E)
19. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, Rongrong Ji, Caifeng Shan, and Ran He. 2025. [Mme: A comprehensive evaluation benchmark for multimodal large language models](https://md.research.elamur.ai/free/en/papers/fu-2023-mme-comprehensive-evaluation-benchmark-multimodal/). Preprint, arXiv:2306.13394.
20. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). Preprint, arXiv:2103.00020.
21. Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. 2021. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Preprint, arXiv:2106.02034. — [arXiv:2106.02034](https://arxiv.org/abs/2106.02034) [find](https://scholar.google.com/scholar?q=Yongming+Rao%2C+Wenliang+Zhao%2C+Benlin+Liu%2C+Jiwen+Lu%2C+Jie+Zhou%2C+and+Cho%2DJui+Hsieh%2E+2021%2E+Dynamicvit%3A+Efficient+vision+transformers+with+dynamic+token+sparsification%2E+Preprint%2C+arXiv%3A2106%2E02034%2E+%E2%80%94+%5BarXiv%3A)
22. Montani I. Van Landeghem S. Honnibal, M. and A Boyd. 2020. Spacy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing. scirp.org. [find](https://scholar.google.com/scholar?q=Montani+I%2E+Van+Landeghem+S%2E+Honnibal%2C+M%2E+and+A+Boyd%2E+2020%2E+Spacy+2%3A+Natural+language+understanding+with+bloom+embeddings%2C+convolutional+neural+networks+and+incremental+parsing%2E+scirp%2Eorg%2E)
23. Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2019. [Object hallucination in image captioning](https://md.research.elamur.ai/free/en/papers/rohrbach-2018-object-hallucination-image-captioning/). Preprint, arXiv:1809.02156.
24. Drew A. Hudson and Christopher D. Manning. 2019. [Gqa: A new dataset for real-world visual reasoning and compositional question answering](https://md.research.elamur.ai/free/en/papers/hudson-2019-gqa-new-dataset-real-world/). Preprint, arXiv:1902.09506.
25. Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. [A-okvqa: A benchmark for visual question answering using world knowledge](https://md.research.elamur.ai/free/en/papers/schwenk-2022-okvqa-benchmark-visual-question-answering/). Preprint, arXiv:2206.01718. to adaptive k changes latency by at most 2.4 ms (within noise). Accuracy is 3.11% for all configurat…
26. Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, and Xing Sun. 2021. Evo-vit: Slow-fast token evolution for dynamic vision transformer. Preprint, arXiv:2108.01390. [find](https://scholar.google.com/scholar?q=Yifan+Xu%2C+Zhijie+Zhang%2C+Mengdan+Zhang%2C+Kekai+Sheng%2C+Ke+Li%2C+Weiming+Dong%2C+Liqing+Zhang%2C+Changsheng+Xu%2C+and+Xing+Sun%2E+2021%2E+Evo%2Dvit%3A+Slow%2Dfast+token+evolution+for+dynamic+vision+transformer%2E+Preprint%2C+a)
27. Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. 2016. [Modeling context in referring expressions](https://md.research.elamur.ai/free/en/papers/yu-2016-modeling-context-referring-expressions/). Preprint, arXiv:1608.00272.
28. Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. 2019. Deep modular co-attention networks for visual question answering. Preprint, arXiv:1906.10770. [find](https://scholar.google.com/scholar?q=Zhou+Yu%2C+Jun+Yu%2C+Yuhao+Cui%2C+Dacheng+Tao%2C+and+Qi+Tian%2E+2019%2E+Deep+modular+co%2Dattention+networks+for+visual+question+answering%2E+Preprint%2C+arXiv%3A1906%2E10770%2E)

*Updated 2 November 2026*