---
title: "FADE: Probing the Limits of VLMs on fine-grained OCR (2026)"
description: "The authors build FADE (FAint numeric Detection Evaluation), a synthetic benchmark. 25 references."
authors: ["Deep Shah", "Nehal Kathrotia", "Sanket Badhe"]
year: 2026
venue: "Proceedings of the 4th Workshop on Advances in Language and Vision Research (ALVR)"
source: "https://aclanthology.org/2026.alvr-main.23/"
doi: "10.18653/v1/2026.alvr-main.23"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shah-2026-fade-probing-limits-vlms-fine/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shah-2026-fade-probing-limits-vlms-fine/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/29/) / FADE: Probing the Limits of VLMs on fine-grained OCR

# FADE: Probing the Limits of VLMs on fine-grained OCR

**Authors:** Deep Shah, Nehal Kathrotia and Sanket Badhe · **Year:** 2026 · **Venue:** [Proceedings of the 4th Workshop on Advances in Language and Vision Research (ALVR)](https://md.research.elamur.ai/free/en/venues/alvr/) ([ALVR 2026](https://md.research.elamur.ai/free/en/venues/alvr/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.alvr-main.23/) · [PDF](https://aclanthology.org/2026.alvr-main.23.pdf) · [DOI 10.18653/v1/2026.alvr-main.23](https://doi.org/10.18653/v1/2026.alvr-main.23) · [OpenAlex](https://openalex.org/W7166821790)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Frontier multimodal LLMs read clear text well. It is unclear, however, whether this comes from genuine low-level vision or from leaning on language priors. The question matters for content moderation, where bad actors hide phone numbers, scam URLs or hateful text as faint overlays in images.

The authors build FADE (FAint numeric Detection Evaluation), a synthetic benchmark. Random nine-digit white number strings are blended into the centre of cluttered COCO photos. Digits carry no linguistic context, and colour and position are fixed, so transparency (α) is the only factor that varies.

They test Gemini 3.0, Claude 4.5 Sonnet and Gemma 3 zero-shot. These are compared with a UNet that segments the watermark and hands the masked region to an OCR engine.

The general models transcribe almost perfectly at high visibility but degrade non-linearly, often failing at α ≤ 0.3. The UNet pipeline remains clearly stronger at the faintest settings. Errors show models inventing strokes rather than abstaining, and cropping helps only slightly. Both findings point to the visual encoder as the bottleneck.

## Key points
- **FADE data:** 2,600 training images (for the segmentation baseline) and 1,000 test images.
  - Each test image is rendered at four opacities, α ∈ {0.2, 0.3, 0.5, 0.8}, giving 4,000 inferences per model.
  - Images are 480×640 RGB, with digits 0–9 uniformly distributed.
  - The dataset is released on Hugging Face.
- **UNet baseline:** encoder depths of 16/32/64/128 filters and a 256-channel bridge. Each block uses two 3×3 convolutions with BatchNorm and ReLU. The predicted binary mask acts as a spatial filter before OCR.
- **Gemini 3 Pro confusion analysis (α = 0.3, length-matched outputs only):** the most frequent errors are:
  - 9→0 (52)
  - 3→8 (51)
  - 8→3 (45)
  - 3→2 (40)
  - 6→0 (39)
- **Reconstructive bias:** in the error analysis, faint digits are "completed" into similar shapes, e.g. 0 read as 9 (34 cases) and 1 as 4 (27 cases).
- **Limitations:** the benchmark covers digits only and uses centred placement.
- **Future work:** the authors propose measuring ViT token perturbation and testing query-conditioned (task-aware) visual encoders.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 430 works · 424,363 citations

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [🔬 Interpretability](https://md.research.elamur.ai/free/en/topics/interpretability/) · 130 works

## Cite

Cite as: Deep Shah, Nehal Kathrotia and Sanket Badhe. 2026. FADE: Probing the Limits of VLMs on fine-grained OCR. Proceedings of the 4th Workshop on Advances in Language and Vision Research (ALVR).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{shah2026fade,
  title = {{FADE: Probing the Limits of VLMs on fine-grained OCR}},
  author = {Deep Shah, Nehal Kathrotia and Sanket Badhe},
  booktitle = {Proceedings of the 4th Workshop on Advances in Language and Vision Research (ALVR)},
  year = {2026},
  doi = {10.18653/v1/2026.alvr-main.23},
  url = {https://doi.org/10.18653/v1/2026.alvr-main.23}
}
```

- [← previous FactVerse: A Benchmark for Factual Consistency in Interleaved Image–Text Generation](https://md.research.elamur.ai/free/en/papers/shan-2026-factverse-benchmark-factual-consistency-interleaved/)
- [next → FAER: Benchmarking VLMs for Failure-Aware Embodied Reasoning](https://md.research.elamur.ai/free/en/papers/song-2026-faer-benchmarking-vlms-failure-aware/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/29/)

## References

25 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
2. Niki Amini-Naieni and Andrew Zisserman. 2025. Countgd++: Generalized prompting for open-world counting. arXiv preprint arXiv:2512.23351. [find](https://scholar.google.com/scholar?q=Niki+Amini%2DNaieni+and+Andrew+Zisserman%2E+2025%2E+Countgd%2B%2B%3A+Generalized+prompting+for+open%2Dworld+counting%2E+arXiv+preprint+arXiv%3A2512%2E23351%2E)
3. Shiqi Chen, Tongyao Zhu, Ruochen Zhou, Jinghan Zhang, Siyang Gao, Juan Carlos Niebles, Mor Geva, Junxian He, Jiajun Wu, and Manling Li. 2025. Why is spatial reasoning hard for vlms? an attention mechanism perspective on focus areas. arXiv preprint arXiv:2503.01773. — [arXiv:2503.01773](https://arxiv.org/abs/2503.01773) [find](https://scholar.google.com/scholar?q=Shiqi+Chen%2C+Tongyao+Zhu%2C+Ruochen+Zhou%2C+Jinghan+Zhang%2C+Siyang+Gao%2C+Juan+Carlos+Niebles%2C+Mor+Geva%2C+Junxian+He%2C+Jiajun+Wu%2C+and+Manling+Li%2E+2025%2E+Why+is+spatial+reasoning+hard+for+vlms%3F+an+attention+mecha)
4. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, and 1 others. 2020. [An image is worth 16x16 words: Transformers for image recognition at scale](https://md.research.elamur.ai/free/en/papers/dosovitskiy-2020-image-worth-16x16-words-transformers/). arXiv preprint arXiv:2010.1192…
5. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and 1 others. 2023. [Mme: A comprehensive evaluation benchmark for multimodal large language models](https://md.research.elamur.ai/free/en/papers/fu-2023-mme-comprehensive-evaluation-benchmark-multimodal/). arXiv preprint arXiv:2306.13394.
6. Roy Ganz, Yair Kittenplon, Aviad Aberdam, Elad Ben Avraham, Oren Nuriel, Shai Mazor, and Ron Litman. 2024. Question aware vision transformer for multimodal reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13861–13871. [find](https://scholar.google.com/scholar?q=Roy+Ganz%2C+Yair+Kittenplon%2C+Aviad+Aberdam%2C+Elad+Ben+Avraham%2C+Oren+Nuriel%2C+Shai+Mazor%2C+and+Ron+Litman%2E+2024%2E+Question+aware+vision+transformer+for+multimodal+reasoning%2E+In+Proceedings+of+the+IEEE%2FCVF+co)
7. Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. 2024. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13299–13308. [find](https://scholar.google.com/scholar?q=Bohao+Li%2C+Yuying+Ge%2C+Yixiao+Ge%2C+Guangzhi+Wang%2C+Rui+Wang%2C+Ruimao+Zhang%2C+and+Ying+Shan%2E+2024%2E+Seed%2Dbench%3A+Benchmarking+multimodal+large+language+models%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Comp)
8. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. [Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/). In International conference on machine learning, pages 19730–19742. PMLR.
9. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. [Microsoft coco: Common objects in context](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/). In European conference on computer vision, pages 740–755. Springer.
10. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). Advances in neural information processing systems, 36:34892– 34916.
11. Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. [Chartqa: A benchmark for question answering about charts with visual and logical reasoning](https://md.research.elamur.ai/free/en/papers/masry-2022-chartqa-benchmark-question-answering-about/). In Findings of the association for computational linguistics: ACL 2022, pages 2263– 2279.
12. Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. 2020. Plotqa: Reasoning over scientific plots. In Proceedings of the ieee/cvf winter conference on applications of computer vision, pages 1527–1536. [find](https://scholar.google.com/scholar?q=Nitesh+Methani%2C+Pritha+Ganguly%2C+Mitesh+M+Khapra%2C+and+Pratyush+Kumar%2E+2020%2E+Plotqa%3A+Reasoning+over+scientific+plots%2E+In+Proceedings+of+the+ieee%2Fcvf+winter+conference+on+applications+of+computer+vision%2C)
13. Clemens Neudecker, Konstantin Baierer, Mike Gerber, Christian Clausner, Apostolos Antonacopoulos, and Stefan Pletschacher. 2021. A survey of ocr evaluation tools and metrics. In Proceedings of the 6th International Workshop on Historical Document Imaging and Processing, pages 13–18. [find](https://scholar.google.com/scholar?q=Clemens+Neudecker%2C+Konstantin+Baierer%2C+Mike+Gerber%2C+Christian+Clausner%2C+Apostolos+Antonacopoulos%2C+and+Stefan+Pletschacher%2E+2021%2E+A+survey+of+ocr+evaluation+tools+and+metrics%2E+In+Proceedings+of+the+6th)
14. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. [U-net: Convolutional networks for biomedical image segmentation](https://md.research.elamur.ai/free/en/papers/ronneberger-2015-u-net-convolutional-networks-biomedical/). In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer.
15. Amanpreet Singh, Guan Pang, Mandy Toh, Jing Huang, Wojciech Galuba, and Tal Hassner. 2021. Textocr: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8802–8812. [find](https://scholar.google.com/scholar?q=Amanpreet+Singh%2C+Guan+Pang%2C+Mandy+Toh%2C+Jing+Huang%2C+Wojciech+Galuba%2C+and+Tal+Hassner%2E+2021%2E+Textocr%3A+Towards+large%2Dscale+end%2Dto%2Dend+reasoning+for+arbitrary%2Dshaped+scene+text%2E+In+Proceedings+of+the+IEEE)
16. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805.
17. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penche… [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Aishwarya+Kamath%2C+Johan+Ferret%2C+Shreya+Pathak%2C+Nino+Vieillard%2C+Ramona+Merhej%2C+Sarah+Perrin%2C+Tatiana+Matejovicova%2C+Alexandre+Ram%C3%A9%2C+Morgane+Rivi%C3%A8re%2C+Louis+Rouillard%2C+Thomas+Mesnard%2C+Geoffrey)
18. Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. 2024. [Eyes wide shut? exploring the visual shortcomings of multimodal llms](https://md.research.elamur.ai/free/en/papers/tong-2024-eyes-wide-shut-exploring-visual/). In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9568–9578.
19. Andrés Villa, Juan Léon, Alvaro Soto, and Bernard Ghanem. 2025. Behind the magic, merlim: Multi-modal evaluation benchmark for large image-language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 492–502. [find](https://scholar.google.com/scholar?q=Andr%C3%A9s+Villa%2C+Juan+L%C3%A9on%2C+Alvaro+Soto%2C+and+Bernard+Ghanem%2E+2025%2E+Behind+the+magic%2C+merlim%3A+Multi%2Dmodal+evaluation+benchmark+for+large+image%2Dlanguage+models%2E+In+Proceedings+of+the+Computer+Vision+and+Pa)
20. Bing Wang, Shengsheng Wang, Changchun Li, Renchu Guan, and Ximing Li. 2024. Harmfully manipulated images matter in multimodal misinformation detection. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 2262–2271. [find](https://scholar.google.com/scholar?q=Bing+Wang%2C+Shengsheng+Wang%2C+Changchun+Li%2C+Renchu+Guan%2C+and+Ximing+Li%2E+2024%2E+Harmfully+manipulated+images+matter+in+multimodal+misinformation+detection%2E+In+Proceedings+of+the+32nd+ACM+International+Con)
21. Tong Wang, Ting Liu, Xiaochao Qu, Chengjing Wu, Luoqi Liu, and Xiaolin Hu. 2025. Glyphmastero: A glyph encoder for high-fidelity scene text editing. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 28523–28532. [find](https://scholar.google.com/scholar?q=Tong+Wang%2C+Ting+Liu%2C+Xiaochao+Qu%2C+Chengjing+Wu%2C+Luoqi+Liu%2C+and+Xiaolin+Hu%2E+2025%2E+Glyphmastero%3A+A+glyph+encoder+for+high%2Dfidelity+scene+text+editing%2E+In+Proceedings+of+the+Computer+Vision+and+Pattern+R)
22. Xiao-Feng Wang, Zhi-Huang He, Kai Wang, Yi-Fan Wang, Le Zou, and Zhi-Ze Wu. 2023. A survey of text detection and recognition algorithms based on deep learning technology. Neurocomputing, 556:126702. [find](https://scholar.google.com/scholar?q=Xiao%2DFeng+Wang%2C+Zhi%2DHuang+He%2C+Kai+Wang%2C+Yi%2DFan+Wang%2C+Le+Zou%2C+and+Zhi%2DZe+Wu%2E+2023%2E+A+survey+of+text+detection+and+recognition+algorithms+based+on+deep+learning+technology%2E+Neurocomputing%2C+556%3A126702%2E)
23. Chenyue Zhou, Mingxuan Wang, Yanbiao Ma, Chenxu Wu, Wanyi Chen, Zhe Qian, Xinyu Liu, Yiwei Zhang, Junhao Wang, Hengbo Xu, and 1 others. 2025. From perception to cognition: A survey of vision-language interactive reasoning in multimodal large language models. arXiv preprint arXiv:2509.25373. — [arXiv:2509.25373](https://arxiv.org/abs/2509.25373) [find](https://scholar.google.com/scholar?q=Chenyue+Zhou%2C+Mingxuan+Wang%2C+Yanbiao+Ma%2C+Chenxu+Wu%2C+Wanyi+Chen%2C+Zhe+Qian%2C+Xinyu+Liu%2C+Yiwei+Zhang%2C+Junhao+Wang%2C+Hengbo+Xu%2C+and+1+others%2E+2025%2E+From+perception+to+cognition%3A+A+survey+of+vision%2Dlanguage+)
24. Kankan Zhou, Eason Lai, Wei Bin Au Yeong, Kyriakos Mouratidis, and Jing Jiang. 2023. Rome: Evaluating pre-trained vision-language models on reasoning beyond visual common sense. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10185–10197. [find](https://scholar.google.com/scholar?q=Kankan+Zhou%2C+Eason+Lai%2C+Wei+Bin+Au+Yeong%2C+Kyriakos+Mouratidis%2C+and+Jing+Jiang%2E+2023%2E+Rome%3A+Evaluating+pre%2Dtrained+vision%2Dlanguage+models+on+reasoning+beyond+visual+common+sense%2E+In+Findings+of+the+Ass)
25. Fengbin Zhu, Ziyang Liu, Xiang Yao Ng, Haohui Wu, Wenjie Wang, Fuli Feng, Chao Wang, Huanbo Luan, and Tat Seng Chua. 2024. Mmdocbench: Benchmarking large vision-language models for fine-grained visual document understanding. arXiv preprint arXiv:2410.21311. — [arXiv:2410.21311](https://arxiv.org/abs/2410.21311) [find](https://scholar.google.com/scholar?q=Fengbin+Zhu%2C+Ziyang+Liu%2C+Xiang+Yao+Ng%2C+Haohui+Wu%2C+Wenjie+Wang%2C+Fuli+Feng%2C+Chao+Wang%2C+Huanbo+Luan%2C+and+Tat+Seng+Chua%2E+2024%2E+Mmdocbench%3A+Benchmarking+large+vision%2Dlanguage+models+for+fine%2Dgrained+visual)

*Updated 1 November 2026*