---
title: "Multimodal Needle in a Haystack — paper by Wang et al. (2025)"
description: "The benchmark offers a reproducible, statistically grounded way to test long-context visual retrieval… 26 references, cited by 17 papers here."
authors: ["Hengyi Wang", "Haizhou Shi", "Shiwei Tan", "Weiyi Qin", "Wenyuan Wang", "Tunyu Zhang", "Akshay Nambi", "Tanuja Ganu", "Hao Wang"]
year: 2025
venue: "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)"
source: "https://aclanthology.org/2025.naacl-long.166/"
doi: "10.18653/v1/2025.naacl-long.166"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2025-multimodal-needle-haystack-benchmarking-long/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2025-multimodal-needle-haystack-benchmarking-long/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/4/) / Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models

# Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models

**Authors:** Hengyi Wang, [Haizhou Shi](https://md.research.elamur.ai/free/en/authors/haizhou-shi/), Shiwei Tan, Weiyi Qin, Wenyuan Wang, Tunyu Zhang, [Akshay Nambi](https://md.research.elamur.ai/free/en/authors/akshay-nambi/), [Tanuja Ganu](https://md.research.elamur.ai/free/en/authors/tanuja-ganu/) and [Hao Wang](https://md.research.elamur.ai/free/en/authors/hao-wang/) · **Year:** 2025 · **Venue:** [Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2025](https://md.research.elamur.ai/free/en/venues/naacl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.naacl-long.166/) · [PDF](https://aclanthology.org/2025.naacl-long.166.pdf) · [DOI 10.18653/v1/2025.naacl-long.166](https://doi.org/10.18653/v1/2025.naacl-long.166) · [OpenAlex](https://openalex.org/W4411120210)

**Citations:** 4 citations (OpenAlex, 2026-10-14) · [17](https://md.research.elamur.ai/free/en/papers/wang-2025-multimodal-needle-haystack-benchmarking-long/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Evaluations of multimodal large language models (MLLMs) usually involve one or a few images, or rely on too few test cases to be statistically meaningful. As a result, it is unclear how these models cope with very long visual contexts. The authors address this gap with MMNeedle, a benchmark built from MS COCO. A model receives a "haystack" of up to 10 images, each made by stitching a grid of smaller sub-images. It also gets a caption describing one sub-image and must return that sub-image's image index, row and column. Some test cases contain several needles, and some contain no matching sub-image at all. Labels are generated automatically.

The authors tested API-based models (GPT-4o, GPT-4V, Claude 3 Opus, Gemini) and open-source models. GPT-4o performed best overall, but its accuracy collapsed as stitching density grew. It also tended to report needles that were absent. Open-source systems trailed far behind and often produced malformed answers. The benchmark offers a reproducible, statistically grounded way to test long-context visual retrieval and hallucination.

## Key points
- **Dataset scale:** 40,000 images, 560,000 captions and 280,000 needle–haystack pairs, released under CC BY 4.0.
- **Context settings:** M ∈ {1, 10} images combined with N×N stitching, where N ∈ {1, 2, 4, 8}. The largest setting reaches 640 sub-images.
- **Metrics:** three levels of granularity:
  - "existence accuracy" (whether a needle is present)
  - "index accuracy" (which image contains it)
  - "exact accuracy" (which image, row and column)
- **GPT-4o degradation:** accuracy fell from 97.00% with 10 unstitched images to 26.90% with 10 images of 4×4 sub-images each.
- **Hallucination and open-source results:** on negative samples, GPT-4V and GPT-4o often claimed a needle existed when none did. Open-source models (Fuyu-8B, mPLUG-Owl-v2, IDEFICS2-8B, LLaVA-Llama-3) lagged well behind the API models.
- **Statistical reliability:** accuracy estimates stabilize after about 500 samples, and 1,000 samples per setting are sufficient.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Rutgers, The State University of New Jersey](https://md.research.elamur.ai/free/en/universities/rutgers-the-state-university-of-new-jersey/) — University, New Brunswick, United States · 36 works · 12,026 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 657 works

## Cite

Cite as: Hengyi Wang, Haizhou Shi, Shiwei Tan, Weiyi Qin, Wenyuan Wang, Tunyu Zhang, Akshay Nambi, Tanuja Ganu and Hao Wang. 2025. Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wang2025multimodal,
  title = {{Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models}},
  author = {Hengyi Wang, Haizhou Shi, Shiwei Tan, Weiyi Qin, Wenyuan Wang, Tunyu Zhang, Akshay Nambi, Tanuja Ganu and Hao Wang},
  booktitle = {Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.166},
  url = {https://doi.org/10.18653/v1/2025.naacl-long.166}
}
```

- [← previous MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark](https://md.research.elamur.ai/free/en/papers/zhao-2025-mmlu-cf-contamination-free-multi/)
- [next → Reasoning-to-Defend: Safety-Aware Reasoning Can Defend Large Language Models from Jailbreaking](https://md.research.elamur.ai/free/en/papers/zhu-2025-reasoning-defend-safety-aware-reasoning/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/4/)

## References

26 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. 2023. Model card and evaluations for claude models, july 2023. 2024. Introducing gpt-4o: our fastest and most affordable flagship model. [find](https://scholar.google.com/scholar?q=2023%2E+Model+card+and+evaluations+for+claude+models%2C+july+2023%2E+2024%2E+Introducing+gpt%2D4o%3A+our+fastest+and+most+affordable+flagship+model%2E)
2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
3. Furong Huang, Yaser Yacoob, et al. 2023. [Hallusionbench: An advanced diagnostic suite for entangled language hallucination & visual illusion in large vision-language models](https://md.research.elamur.ai/free/en/papers/guan-2023-hallusionbench-advanced-diagnostic-suite-entangled/). arXiv preprint arXiv:2310.14566.
4. G. Kamradt. 2023. Needle in a haystack - pressure testing llms. [https://github.com/gkamradt/](https://github.com/gkamradt/) LLMTest_NeedleInAHaystack. [↗](https://github.com/gkamradt/)
5. Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. 2024. In search of needles in a 10m haystack: Recurrent memory finds what llms miss. arXiv preprint arXiv:2402.10790. — [arXiv:2402.10790](https://arxiv.org/abs/2402.10790) [find](https://scholar.google.com/scholar?q=Yuri+Kuratov%2C+Aydar+Bulatov%2C+Petr+Anokhin%2C+Dmitry+Sorokin%2C+Artyom+Sorokin%2C+and+Mikhail+Burtsev%2E+2024%2E+In+search+of+needles+in+a+10m+haystack%3A+Recurrent+memory+finds+what+llms+miss%2E+arXiv+preprint+arXi)
6. Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. 2024. [What matters when building vision-language models?](https://md.research.elamur.ai/free/en/papers/lauren-ccon-2024-matters-building-vision-language-models/) arXiv preprint arXiv:2405.02246.
7. Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Saugnak Taşırlar. 2023. Introducing our multimodal models. [find](https://scholar.google.com/scholar?q=Rohan+Bavishi%2C+Erich+Elsen%2C+Curtis+Hawthorne%2C+Maxwell+Nye%2C+Augustus+Odena%2C+Arushi+Somani%2C+and+Saugnak+Tas%CC%A7%C4%B1rlar%2E+2023%2E+Introducing+our+multimodal+models%2E)
8. Mosh Levy, Alon Jacoby, and Yoav Goldberg. 2024. [Same task, more tokens: the impact of input length on the reasoning performance of large language models](https://md.research.elamur.ai/free/en/papers/levy-2024-same-task-more-tokens-impact/). arXiv preprint arXiv:2402.14848.
9. Joel Brogan, Aparna Bharati, Daniel Moreira, Kevin Bowyer, Patrick Flynn, Anderson Rocha, and W Scheirer. 2019. Needle in a haystack: A framework for seeking small objects in big datasets. arXiv preprint arXiv:1903.10019. — [arXiv:1903.10019](https://arxiv.org/abs/1903.10019) [find](https://scholar.google.com/scholar?q=Joel+Brogan%2C+Aparna+Bharati%2C+Daniel+Moreira%2C+Kevin+Bowyer%2C+Patrick+Flynn%2C+Anderson+Rocha%2C+and+W+Scheirer%2E+2019%2E+Needle+in+a+haystack%3A+A+framework+for+seeking+small+objects+in+big+datasets%2E+arXiv+prepr)
10. Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. 2024. Llava-next: Stronger llms supercharge multimodal capabilities in the wild. [find](https://scholar.google.com/scholar?q=Bo+Li%2C+Kaichen+Zhang%2C+Hao+Zhang%2C+Dong+Guo%2C+Renrui+Zhang%2C+Feng+Li%2C+Yuanhan+Zhang%2C+Ziwei+Liu%2C+and+Chunyuan+Li%2E+2024%2E+Llava%2Dnext%3A+Stronger+llms+supercharge+multimodal+capabilities+in+the+wild%2E)
11. Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. 2024. [Are we on the right way for evaluating large vision-language models?](https://md.research.elamur.ai/free/en/papers/chen-2024-right-way-evaluating-large-vision/) arXiv preprint arXiv:2403.20330.
12. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. 2024. [Instructblip: Towards general-purpose vision-language models with instruction tuning](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/). Advances in Neural Information Processing Systems, 36.
13. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. 2024a. [Mme: A comprehensive evaluation benchmark for multimodal large language models](https://md.research.elamur.ai/free/en/papers/fu-2023-mme-comprehensive-evaluation-benchmark-multimodal/). Preprint, arXiv:2306.13394.
14. Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. 2024b. Blink: Multimodal large language models can see but not perceive. arXiv preprint arXiv:2404.12390. [find](https://scholar.google.com/scholar?q=Xingyu+Fu%2C+Yushi+Hu%2C+Bangzheng+Li%2C+Yu+Feng%2C+Haoyu+Wang%2C+Xudong+Lin%2C+Dan+Roth%2C+Noah+A+Smith%2C+Wei%2DChiu+Ma%2C+and+Ranjay+Krishna%2E+2024b%2E+Blink%3A+Multimodal+large+language+models+can+see+but+not+perceive%2E+ar)
15. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. [Microsoft coco: Common objects in context](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/). In Computer Vision– ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pag…
16. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). Advances in neural information processing systems, 36.
17. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. 2023. [Mmbench: Is your multi-modal model an all-around player?](https://md.research.elamur.ai/free/en/papers/liu-2023-mmbench-multi-modal-model-all/) arXiv preprint arXiv:2307.06281.
18. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023. [Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts](https://md.research.elamur.ai/free/en/papers/lu-2023-mathvista-evaluating-mathematical-reasoning-foundation/). arXiv preprint arXiv:2310.02255.
19. Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. 2024c. [Data engineering for scaling language models to 128k context](https://md.research.elamur.ai/free/en/papers/fu-2024-data-engineering-scaling-language-models/). arXiv preprint arXiv:2402.10171.
20. Piotr Padlewski, Max Bain, Matthew Henderson, Zhongkai Zhu, Nishant Relan, Hai Pham, Donovan Ong, Kaloyan Aleksiev, Aitor Ormazabal, Samuel Phua, et al. 2024. [Vibe-eval: A hard evaluation suite for measuring progress of multimodal language models](https://md.research.elamur.ai/free/en/papers/padlewski-2024-vibe-eval-hard-evaluation-suite/). arXiv preprint arXiv:2405.02287.
21. Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Nick Pawlowski, Suvrat Bhooshan, Nicolas Ballas, Francesco Ciompi, Ben Glocker, and Michal Drozdzal. 2019. Needles in haystacks: On classifying tiny objects in large images. arXiv preprint arXiv:1908.0… — [arXiv:1908.06037](https://arxiv.org/abs/1908.06037) [find](https://scholar.google.com/scholar?q=Tianrui+Guan%2C+Fuxiao+Liu%2C+Xiyang+Wu%2C+Ruiqi+Xian%2C+Zongxia+Li%2C+Xiaoyu+Liu%2C+Xijun+Wang%2C+Lichang+Chen%2C+Nick+Pawlowski%2C+Suvrat+Bhooshan%2C+Nicolas+Ballas%2C+Francesco+Ciompi%2C+Ben+Glocker%2C+and+Michal+Drozdzal%2E+)
22. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/). arXiv preprint arXiv:2403.…
23. Dingjie Song, Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, and Benyou Wang. 2024. Milebench: Benchmarking mllms in long context. arXiv preprint arXiv:2404.18532. — [arXiv:2404.18532](https://arxiv.org/abs/2404.18532) [find](https://scholar.google.com/scholar?q=Dingjie+Song%2C+Shunian+Chen%2C+Guiming+Hardy+Chen%2C+Fei+Yu%2C+Xiang+Wan%2C+and+Benyou+Wang%2E+2024%2E+Milebench%3A+Benchmarking+mllms+in+long+context%2E+arXiv+preprint+arXiv%3A2404%2E18532%2E+%E2%80%94+%5BarXiv%3A2404%2E18532%5D%28https%3A%2F%2Fa)
24. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805.
25. Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. 2023. [Cogvlm: Visual expert for pretrained language models](https://md.research.elamur.ai/free/en/papers/wang-2023-cogvlm-visual-expert-pretrained/). arXiv preprint arXiv:2311.03079.
26. Longagent: Scaling language models to 128k context through multi-agent collaboration. arXiv preprint arXiv:2402.11550. — [arXiv:2402.11550](https://arxiv.org/abs/2402.11550) [find](https://scholar.google.com/scholar?q=Longagent%3A+Scaling+language+models+to+128k+context+through+multi%2Dagent+collaboration%2E+arXiv+preprint+arXiv%3A2402%2E11550%2E+%E2%80%94+%5BarXiv%3A2402%2E11550%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2402%2E11550%29)

*Updated 30 October 2026*