--- title: "V*: Guided Visual Search as a Core Mechanism in Multimodal" description: "Drawing on how people search scenes, the authors propose SEAL (Show, sEArch, and TelL), a… 13 references, cited by 80 papers here." authors: ["Penghao Wu", "Saining Xie"] year: 2023 venue: "arXiv:2312.14135" source: "https://arxiv.org/abs/2312.14135" doi: "10.48550/arxiv.2312.14135" arxiv: "2312.14135" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/wu-2023-v-guided-visual-search-core/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wu-2023-v-guided-visual-search-core/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/) / V\*: Guided Visual Search as a Core Mechanism in Multimodal LLMs # V\*: Guided Visual Search as a Core Mechanism in Multimodal LLMs **Authors:** [Penghao Wu](https://md.research.elamur.ai/free/en/authors/penghao-wu/) and [Saining Xie](https://md.research.elamur.ai/free/en/authors/saining-xie/) · **Year:** 2023 · **Venue:** [arXiv:2312.14135](https://md.research.elamur.ai/free/en/venues/ieee-computer-society-conference-on-computer-vision-and-pattern-recognition/) ([IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2023](https://md.research.elamur.ai/free/en/venues/ieee-computer-society-conference-on-computer-vision-and-pattern-recognition/2023/)) · **Publisher:** [Institute of Electrical and Electronics Engineers](https://md.research.elamur.ai/free/en/publishers/institute-of-electrical-and-electronics-engineers/) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2312.14135) · [PDF](https://arxiv.org/pdf/2312.14135) · [DOI 10.48550/arxiv.2312.14135](https://doi.org/10.48550/arxiv.2312.14135) · [arXiv 2312.14135](https://arxiv.org/abs/2312.14135) · [OpenAlex](https://openalex.org/W4402727815) **Citations:** 72 citations (OpenAlex, 2026-10-14) · [80](https://md.research.elamur.ai/free/en/papers/wu-2023-v-guided-visual-search-core/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Multimodal LLMs usually view images through frozen vision encoders such as CLIP, which work at low resolution (e.g., 224×224 or 336×336). As a result, small details in large or cluttered pictures are lost. The models also cannot notice what is missing or go looking for it. Drawing on how people search scenes, the authors propose SEAL (Show, sEArch, and TelL), a meta-architecture that pairs a VQA LLM with a visual search model. When the VQA LLM finds the visible evidence insufficient, it names the objects it needs. V*, a search procedure driven by a second MLLM, then locates them by zooming into the image hierarchically. It is guided by common-sense contextual cues (where such an object tends to be) and target-specific cues. Located regions go into a Visual Working Memory, which is fed back to the VQA LLM for answering. Both parts build on LLaVA with Vicuna-7B. The authors also introduce V* Bench, a VQA benchmark for fine-grained grounding in high-resolution images. Most tested MLLMs score near chance on it, while SEAL improves strongly over its baseline. This suggests active visual search should be a core MLLM capability. ## Key points - On V* Bench, SEAL (7B) reaches 75.3 versus 45.0 for a comparable LLaVA (7B) trained on the same VQA data. Several MLLMs, among them GPT-4V, Bard and Gemini Pro, were evaluated, and most scored roughly at random-guess level. - On general benchmarks, SEAL improves MME (1051.2 → 1128.9) and the hallucination benchmark POPE (76.5 → 82.4). It stays about level on MMBench and SEED-Bench and drops slightly on MM-Vet and LLaVA-Bench W. - Search efficiency was tested on the 245 annotated target locations in V* Bench. V* needs far fewer search steps than Random and Sequential baselines (each run in BFS and DFS modes), and both contextual and target cues contribute. - On COCO-Search18, the authors guided the search with heatmaps built from human eye fixations. V* proved roughly as efficient as this human-fixation-guided search. - Stated limitations: the search model is tuned for natural images and common objects. Documents, diagrams, long videos and open-world settings would require further training and new algorithm design. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [New York University](https://md.research.elamur.ai/free/en/universities/new-york-university/) — University, New York, United States · 219 works · 103,487 citations - [University of California San Diego](https://md.research.elamur.ai/free/en/universities/university-of-california-san-diego/) — University, San Diego, United States · 89 works · 54,032 citations 📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works 🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 811 works ## Cite Cite as: Penghao Wu and Saining Xie. 2023. V\*: Guided Visual Search as a Core Mechanism in Multimodal LLMs. arXiv:2312.14135. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{wu2023v, title = {{V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs}}, author = {Penghao Wu and Saining Xie}, year = {2023}, doi = {10.48550/arxiv.2312.14135}, eprint = {2312.14135}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2312.14135} } ``` - [← previous Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigation](https://md.research.elamur.ai/free/en/papers/mundler-2023-self-contradictory-hallucinations-large-language/) - [next → How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model](https://md.research.elamur.ai/free/en/papers/hanna-2023-gpt-2-compute-greater-than/) - [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/) ## References 13 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew B… — [Flamingo: a Visual Language Model for Few-Shot Learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/) 2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). 2020. 7 3. Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In CVPR, 2018. 13 [find](https://scholar.google.com/scholar?q=Holger+Caesar%2C+Jasper+Uijlings%2C+and+Vittorio+Ferrari%2E+Coco%2Dstuff%3A+Thing+and+stuff+classes+in+context%2E+In+CVPR%2C+2018%2E+13) 4. Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. [End-to-end object detection with transformers](https://md.research.elamur.ai/free/en/papers/carion-2020-end-end-object-detection-transformers/). In ECCV, 2020. 13 5. Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm’s referential dialogue magic. arXiv preprint arXiv:2306.[15195, 2023](https://md.research.elamur.ai/free/en/papers/chen-2023-shikra-unleashing-multimodal-llm-s/). 4 6. Richard J Chen, Chengkuan Chen, Yicong Li, Tiffany Y Chen, Andrew D Trister, Rahul G Krishnan, and Faisal Mahmood. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In CVPR, 2022. 6 [find](https://scholar.google.com/scholar?q=Richard+J+Chen%2C+Chengkuan+Chen%2C+Yicong+Li%2C+Tiffany+Y+Chen%2C+Andrew+D+Trister%2C+Rahul+G+Krishnan%2C+and+Faisal+Mahmood%2E+Scaling+vision+transformers+to+gigapixel+images+via+hierarchical+self%2Dsupervised+lear) 7. Yupei Chen, Zhibo Yang, Seoyoung Ahn, Dimitris Samaras, Minh Hoai, and Gregory Zelinsky. Coco-search18 fixation dataset for predicting goal-directed attention control. Scientific reports, 2021. 8, 9 [find](https://scholar.google.com/scholar?q=Yupei+Chen%2C+Zhibo+Yang%2C+Seoyoung+Ahn%2C+Dimitris+Samaras%2C+Minh+Hoai%2C+and+Gregory+Zelinsky%2E+Coco%2Dsearch18+fixation+dataset+for+predicting+goal%2Ddirected+attention+control%2E+Scientific+reports%2C+2021%2E+8%2C+9) 8. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. [InstructBLIP: Towards general-purpose vision-language models with instruction tuning](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/). In NeurIPS, 2023. 2, 3, 4, 7 9. Google DeepMind. Gemini, 2023. 7 [find](https://scholar.google.com/scholar?q=Google+DeepMind%2E+Gemini%2C+2023%2E+7) 10. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. [Mme: A comprehensive evaluation benchmark for multimodal large language models](https://md.research.elamur.ai/free/en/papers/fu-2023-mme-comprehensive-evaluation-benchmark-multimodal/). arXiv preprint arXiv:2306.13394, 2023. 2, 9 11. Tanmay Gupta and Aniruddha Kembhavi. [Visual programming: Compositional visual reasoning without training](https://md.research.elamur.ai/free/en/papers/gupta-2022-visual-programming-compositional-visual-reasoning/). In CVPR, 2023. 7 12. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. [LoRA: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In ICLR, 2022. 14 13. Ziniu Hu, Ahmet Iscen, Chen Sun, Kai-Wei Chang, Yizhou Sun, David A Ross, Cordelia Schmid, and Alireza Fathi. Avis: Autonomous visual information seeking with large language models. In NeurIPS, 2023. [4 46 Antonio Torralba, Aude Oliva, Monica S Castelhano, and John M Henderson](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Contextual guidance… *Updated 2 November 2026*