--- title: "Vision language models are blind: Failing to translate detailed" description: "Modern vision-language models (VLMs) earn strong scores on chart, diagram and document… 6 references, cited by 17 papers here." authors: ["Pooyan Rahmanzadehgervi", "Logan Bolton", "Mohammad Reza Taesiri", "Anh Totti Nguyen"] year: 2024 venue: "arXiv:2407.06581" source: "https://arxiv.org/abs/2407.06581" doi: "10.48550/arxiv.2407.06581" arxiv: "2407.06581" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/rahmanzadehgervi-2024-vision-language-models-blind-failing/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/rahmanzadehgervi-2024-vision-language-models-blind-failing/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/12/) / Vision language models are blind: Failing to translate detailed visual features into words # Vision language models are blind: Failing to translate detailed visual features into words **Authors:** Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri and Anh Totti Nguyen · **Year:** 2024 · **Venue:** arXiv:2407.06581 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2407.06581) · [PDF](https://arxiv.org/pdf/2407.06581) · [DOI 10.48550/arxiv.2407.06581](https://doi.org/10.48550/arxiv.2407.06581) · [arXiv 2407.06581](https://arxiv.org/abs/2407.06581) · [OpenAlex](https://openalex.org/W4400518804) **Citations:** 0 citations (OpenAlex, 2026-10-14) · [17](https://md.research.elamur.ai/free/en/papers/rahmanzadehgervi-2024-vision-language-models-blind-failing/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Modern vision-language models (VLMs) earn strong scores on chart, diagram and document benchmarks. Many of those benchmarks, however, can be partly solved from text alone or from memorized web content. This paper asks whether VLMs can actually *see* fine spatial detail. The authors built BlindTest, a new suite of 7 synthetic tasks based on eye exams and made from simple 2D shapes, such as lines, circles, squares, letters and grids. Each task is trivial for a person and needs almost no world knowledge. Examples include counting line crossings, judging whether two circles overlap, naming a circled letter, counting nested or overlapping shapes, counting grid rows and columns, and tracing paths on a toy subway map. The tests vary image sizes, line widths and prompt wording. Four leading commercial VLMs did poorly overall, and slow-thinking models and prompting tricks did not help. Accuracy rose sharply when shapes were spaced further apart. Linear probes showed that the vision encoders do hold the needed information, so the failure lies in turning visual features into language. This exposes a perception gap that standard benchmarks miss. ## Key points - **Models and overall accuracy:** GPT-4o, Gemini-1.5 Pro, Claude-3 Sonnet and Claude-3.5 Sonnet averaged 58.07% on BlindTest. Sonnet-3.5 led at 77.84%, against an expected human score of 100%. - **Per-task results:** Counting intersections of two piecewise-linear plots reached only about 41–76%. Two-circle overlap was about 75–93%. Counting paths between subway stations was about 31–58%. - **Other models and prompting:** Slow-thinking models (Gemini 2.0 Flash-Thinking, QVQ-Preview) scored 71.59%, versus 72.75% for standard models. Eight open-source models (LLaVA-OneV, Phi-3.5, InternVL-2; 0.5B–72B) did worse than the closed ones. 2-shot, chain-of-thought and meta-prompting gave no gains. - **Effect of spacing:** Adding space between letters or shapes pushed some models close to 100%. For example, LLaVA-OneV-72B gained +20 points on the circled-letter task, and Sonnet-3 gained +63 points on counting overlapping circles. - **Linear probing:** Probes on the CLIP and SigLIP encoder features of LLaVA-OneV-S and Phi-3.5 solved the line-intersection and two-circle tasks at ≥99.47% accuracy. This held both before and after the projection layer, while the full VLMs scored only about 33–84%. Code and data are available at VLMsAreBlind.github.io. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,621 works 🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 4,145 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 827 works ## Cite Cite as: Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri and Anh Totti Nguyen. 2024. Vision language models are blind: Failing to translate detailed visual features into words. arXiv:2407.06581. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{rahmanzadehgervi2024vision, title = {{Vision language models are blind: Failing to translate detailed visual features into words}}, author = {Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri and Anh Totti Nguyen}, year = {2024}, doi = {10.48550/arxiv.2407.06581}, eprint = {2407.06581}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2407.06581} } ``` - [← previous SWE-Bench+: Enhanced Coding Benchmark for LLMs](https://md.research.elamur.ai/free/en/papers/aleithan-2024-swe-bench-enhanced-coding/) - [next → Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models](https://md.research.elamur.ai/free/en/papers/zhou-2024-weak-strong-search-align-large/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/12/) ## References 6 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. 1. Abdin, M., Jacobs, S.A., Awan, A.A., Aneja, J., Awadallah, A., Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., Behl, H., et al.: [Phi-3 technical report: A highly capable language model locally on your phone](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-3-technical-report-highly/). arXiv preprint arXiv:2404.14219 (2024) 2. Anthropic: Introducing claude 3.5 sonnet \\ a… 2. Pearson Education (2014) 15. Inc., C.Z.V.: Vision screening. — [OpenAI o1 System Card](https://md.research.elamur.ai/free/en/papers/openai-2024-openai-o1-system-card/) [https://www.zeiss.com/vision-care/us/eye-health-and-care/vision-screening.html](https://www.zeiss.com/vision-care/us/eye-health-and-care/vision-screening.html), (Accessed on 07/03/2024) 16. Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Carney, A., et al.: Op… [↗](https://www.zeiss.com/vision-care/us/eye-health-and-care/vision-screening.html) 3. VLMs fail to translate detailed visual features into words 19. yunxin li, Hu, B., Shi, H., Wang, W., Wang, L., Zhang, M.: Visiongraph: Leveraging large multimodal models for graph theory problems in visual context. In: Forty-first International Conference on Machine Learning (2024), [https://openrevi…](https://md.research.elamur.ai/free/en/papers/masry-2022-chartqa-benchmark-question-answering-about/) 4. Rahmanzadehgervi, Bolton, Taesiri, Nguyen. 34. Mirza, M.J., Karlinsky, L., Lin, W., Doveh, S., Micorek, J., Kozinski, M., Kuhene, H., Possegger, H.: Meta-prompting for automating zero-shot visual recognition with llms. arXiv preprint arXiv:2403.[11755 (2024) 35](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Mirza, M.J., Karlinsky, L., Lin, W., D… 5. PMLR (2024) 56. Yu, Z., Xu, D., Yu, J., Yu, T., Zhao, Z., Zhuang, Y., Tao, D.: Activitynet-qa: a dataset for understanding complex web videos via question answering. In: Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial… — [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/) 6. Rahmanzadehgervi, Bolton, Taesiri, Nguyen. Appendix for: Vision language models are blind: Failing to translate detailed visual features into words [find](https://scholar.google.com/scholar?q=Rahmanzadehgervi%2C+Bolton%2C+Taesiri%2C+Nguyen%2E+Appendix+for%3A+Vision+language+models+are+blind%3A+Failing+to+translate+detailed+visual+features+into+words) *Updated 3 November 2026*