---
title: "Unveiling Inherent Visual Grounding in Multimodal LLMs for"
description: "The authors propose Doc-AGround, an OCR-free method that reads grounding directly from the model's own multi-head… 66 references."
authors: ["Shijie Zhou", "Jihyung Kil", "Ming Li", "Jiuxiang Gu", "Curtis Wigington", "Rajiv Jain", "Changyou Chen", "Ruiyi Zhang"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.16/"
doi: "10.18653/v1/2026.findings-acl.16"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhou-2026-unveiling-inherent-visual-grounding-multimodal/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhou-2026-unveiling-inherent-visual-grounding-multimodal/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/55/) / Unveiling Inherent Visual Grounding in Multimodal LLMs for Text-Rich Images

# Unveiling Inherent Visual Grounding in Multimodal LLMs for Text-Rich Images

**Authors:** Shijie Zhou, [Jihyung Kil](https://md.research.elamur.ai/free/en/authors/jihyung-kil/), [Ming Li](https://md.research.elamur.ai/free/en/authors/ming-li/), [Jiuxiang Gu](https://md.research.elamur.ai/free/en/authors/jiuxiang-gu/), Curtis Wigington, [Rajiv Jain](https://md.research.elamur.ai/free/en/authors/rajiv-jain/), [Changyou Chen](https://md.research.elamur.ai/free/en/authors/changyou-chen/) and [Ruiyi Zhang](https://md.research.elamur.ai/free/en/authors/ruiyi-zhang/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.16/) · [PDF](https://aclanthology.org/2026.findings-acl.16.pdf) · [DOI 10.18653/v1/2026.findings-acl.16](https://doi.org/10.18653/v1/2026.findings-acl.16) · [OpenAlex](https://openalex.org/W7166872540)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
The paper addresses visual text grounding for text-rich images such as infographics, charts and forms. The goal is to localize the possibly scattered image regions that support the answer to a document question. Existing MLLM approaches either match embeddings, which works poorly when only the question is given, or generate bounding-box coordinates. They also often depend on long OCR inputs. The authors propose Doc-AGround, an OCR-free method that reads grounding directly from the model's own multi-head self-attention. A special \<g> token is placed after the image and question. Because attention is causal, this token summarizes the multimodal context, and its attention over visual patches serves as a patch-level evidence map. The model is fine-tuned with patch-wise labels. The per-head maps are then combined using importance weights: each head is scored by how closely its text-to-image attention matches a global correlation pattern estimated from hidden states, measured with negative KL divergence. On the TRIG document-grounding benchmark, the method reports state-of-the-art in-domain and zero-shot results, beating both OCR-free and OCR-based baselines. Because grounding comes before generation, the same model can still answer the question afterwards.

## Key points
- **\<g> token ablation:** under uniform head weighting, removing the \<g> token lowers the average score from 80.80% to 78.18%.
- **Head weighting ablations:**
  - Uniform weighting costs 2.49%.
  - Weighting based on the least-salient query token costs 4.49%.
  - Dense (all-token) weighting slightly beats sparse top-K weighting overall, but sparse weighting does better on InfographicVQA.
- **Which heads matter:** mid layers (e.g., 12, 13, 17) and late layers (e.g., 31, 33, 34) contribute most. Keeping only the top 30% of heads leaves in-domain results nearly unchanged and drops zero-shot by 0.35%. Keeping only the bottom 30% severely hurts performance.
- **Optional OCR filter:** an OCR-based post-hoc patch filter adds about 1.8%, yet the unfiltered version still outperforms most baselines. The method also works with a weaker backbone, Qwen2-VL-2B.
- **Limitations:** a single \<g> token struggles with overlapping or visually distinct multi-region evidence. The authors propose per-region vectors with Hungarian matching and a box-shape loss as future work.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 811 works

## Cite

Cite as: Shijie Zhou, Jihyung Kil, Ming Li, Jiuxiang Gu, Curtis Wigington, Rajiv Jain, Changyou Chen and Ruiyi Zhang. 2026. Unveiling Inherent Visual Grounding in Multimodal LLMs for Text-Rich Images. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhou2026unveiling,
  title = {{Unveiling Inherent Visual Grounding in Multimodal LLMs for Text-Rich Images}},
  author = {Shijie Zhou, Jihyung Kil, Ming Li, Jiuxiang Gu, Curtis Wigington, Rajiv Jain, Changyou Chen and Ruiyi Zhang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.16},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.16}
}
```

- [← previous UNSC-Bench: Evaluating LLM Diplomatic Role-Playing Through UN Security Council Vote Prediction](https://md.research.elamur.ai/free/en/papers/nangia-2026-unsc-bench-evaluating-llm-diplomatic/)
- [next → Unveiling Intrinsic Dimension of Texts: from Academic Abstract to Creative Story](https://md.research.elamur.ai/free/en/papers/vladislav-2026-unveiling-intrinsic-dimension-texts-academic/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/55/)

## References

66 references, 31 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, and 1 others. 2022. [Flamingo: a visual language model for few-shot learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/). Advances in neural information processing systems, 35:23716– 23736.
2. Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R Manmatha. 2021. Docformer: End-to-end transformer for document understanding. In Proceedings of the IEEE/CVF international conference on computer vision, pages 993–1003. [find](https://scholar.google.com/scholar?q=Srikar+Appalaraju%2C+Bhavan+Jasani%2C+Bhargava+Urala+Kota%2C+Yusheng+Xie%2C+and+R+Manmatha%2E+2021%2E+Docformer%3A+End%2Dto%2Dend+transformer+for+document+understanding%2E+In+Proceedings+of+the+IEEE%2FCVF+international+con)
3. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, and 1 others. 2025. [Qwen2.5-vl technical report](https://md.research.elamur.ai/free/en/papers/bai-2025-qwen2-5-vl-technical/). arXiv preprint arXiv:2502.13923.
4. Jian Chen, Ruiyi Zhang, Yufan Zhou, Tong Yu, Franck Dernoncourt, Jiuxiang Gu, Ryan A Rossi, Changyou Chen, and Tong Sun. 2024a. Sv-rag: Lora-contextualizing adaptation of mllms for long document understanding. arXiv preprint arXiv:2411.01106. — [arXiv:2411.01106](https://arxiv.org/abs/2411.01106) [find](https://scholar.google.com/scholar?q=Jian+Chen%2C+Ruiyi+Zhang%2C+Yufan+Zhou%2C+Tong+Yu%2C+Franck+Dernoncourt%2C+Jiuxiang+Gu%2C+Ryan+A+Rossi%2C+Changyou+Chen%2C+and+Tong+Sun%2E+2024a%2E+Sv%2Drag%3A+Lora%2Dcontextualizing+adaptation+of+mllms+for+long+document+under)
5. Jiaxing Chen, Yuxuan Liu, Dehu Li, Xiang An, Weimo Deng, Ziyong Feng, Yongle Zhao, and Yin Xie. 2024b. Plug-and-play grounding of reasoning in multimodal large language models. arXiv preprint arXiv:2403.19322. — [arXiv:2403.19322](https://arxiv.org/abs/2403.19322) [find](https://scholar.google.com/scholar?q=Jiaxing+Chen%2C+Yuxuan+Liu%2C+Dehu+Li%2C+Xiang+An%2C+Weimo+Deng%2C+Ziyong+Feng%2C+Yongle+Zhao%2C+and+Yin+Xie%2E+2024b%2E+Plug%2Dand%2Dplay+grounding+of+reasoning+in+multimodal+large+language+models%2E+arXiv+preprint+arXiv%3A24)
6. Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. 2023. [Shikra: Unleashing multimodal llm’s referential dialogue magic](https://md.research.elamur.ai/free/en/papers/chen-2023-shikra-unleashing-multimodal-llm-s/). arXiv preprint arXiv:2306.15195.
7. Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, and 1 others. 2024c. [Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks](https://md.research.elamur.ai/free/en/papers/chen-2023-internvl-scaling-up-vision/). In Proceedings of the IEEE/CVF Conference on Computer Vision and P…
8. Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. 2024. [Seeclick: Harnessing gui grounding for advanced visual gui agents](https://md.research.elamur.ai/free/en/papers/cheng-2024-seeclick-harnessing-gui-grounding-advanced/). arXiv preprint arXiv:2401.10935.
9. Cheng Cui, Ting Sun, Manhui Lin, Tingquan Gao, Yubo Zhang, Jiaxuan Liu, Xueqing Wang, Zelun Zhang, Changda Zhou, Hongen Liu, and 1 others. 2025. Paddleocr 3.0 technical report. arXiv preprint arXiv:2507.05595. — [arXiv:2507.05595](https://arxiv.org/abs/2507.05595) [find](https://scholar.google.com/scholar?q=Cheng+Cui%2C+Ting+Sun%2C+Manhui+Lin%2C+Tingquan+Gao%2C+Yubo+Zhang%2C+Jiaxuan+Liu%2C+Xueqing+Wang%2C+Zelun+Zhang%2C+Changda+Zhou%2C+Hongen+Liu%2C+and+1+others%2E+2025%2E+Paddleocr+3%2E0+technical+report%2E+arXiv+preprint+arXiv%3A25)
10. Google DeepMind. 2025. Gemini 2.5 pro preview model card. Technical report, Google DeepMind. [find](https://scholar.google.com/scholar?q=Google+DeepMind%2E+2025%2E+Gemini+2%2E5+pro+preview+model+card%2E+Technical+report%2C+Google+DeepMind%2E)
11. Mohamed Dhouib, Ghassen Bettaieb, and Aymen Shabou. 2023. Docparser: End-to-end ocr-free information extraction from visually rich documents. In International Conference on Document Analysis and Recognition, pages 155–172. Springer. [find](https://scholar.google.com/scholar?q=Mohamed+Dhouib%2C+Ghassen+Bettaieb%2C+and+Aymen+Shabou%2E+2023%2E+Docparser%3A+End%2Dto%2Dend+ocr%2Dfree+information+extraction+from+visually+rich+documents%2E+In+International+Conference+on+Document+Analysis+and+Recog)
12. Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, and 4 others. 2024a. [Internlm-xcomposer2: Mastering free-form text-image…](https://md.research.elamur.ai/free/en/papers/dong-2024-internlm-xcomposer2-mastering-free/)
13. Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Zhe Chen, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Kai Chen, Conghui He, and 5 others. 2024b. [Internlm-xcomposer2-4khd: A pioneering large vision-…](https://md.research.elamur.ai/free/en/papers/dong-2024-internlm-xcomposer2-4khd-pioneering/)
14. Amit Elhelo and Mor Geva. 2024. Inferring functionality of attention heads from their parameters. arXiv preprint arXiv:2412.11965. [find](https://scholar.google.com/scholar?q=Amit+Elhelo+and+Mor+Geva%2E+2024%2E+Inferring+functionality+of+attention+heads+from+their+parameters%2E+arXiv+preprint+arXiv%3A2412%2E11965%2E)
15. Yue Fan, Xuehai He, Diji Yang, Kaizhi Zheng, Ching-Chen Kuo, Yuting Zheng, Sravana Jyothi Naraya-naraju, Xinze Guan, and Xin Eric Wang. 2025. Grit: Teaching mllms to think with images. arXiv preprint arXiv:2505.15879. — [arXiv:2505.15879](https://arxiv.org/abs/2505.15879) [find](https://scholar.google.com/scholar?q=Yue+Fan%2C+Xuehai+He%2C+Diji+Yang%2C+Kaizhi+Zheng%2C+Ching%2DChen+Kuo%2C+Yuting+Zheng%2C+Sravana+Jyothi+Naraya%2Dnaraju%2C+Xinze+Guan%2C+and+Xin+Eric+Wang%2E+2025%2E+Grit%3A+Teaching+mllms+to+think+with+images%2E+arXiv+preprint+)
16. Hao Feng, Zijian Wang, Jingqun Tang, Jinghui Lu, Wengang Zhou, Houqiang Li, and Can Huang. 2023. Unidoc: A universal large multimodal model for simultaneous text detection, recognition, spotting and understanding. arXiv preprint arXiv:2308.11592. — [arXiv:2308.11592](https://arxiv.org/abs/2308.11592) [find](https://scholar.google.com/scholar?q=Hao+Feng%2C+Zijian+Wang%2C+Jingqun+Tang%2C+Jinghui+Lu%2C+Wengang+Zhou%2C+Houqiang+Li%2C+and+Can+Huang%2E+2023%2E+Unidoc%3A+A+universal+large+multimodal+model+for+simultaneous+text+detection%2C+recognition%2C+spotting+and+u)
17. Rhys Gould, Euan Ong, George Ogden, and Arthur Conmy. 2023. Successor heads: Recurring, interpretable attention heads in the wild. arXiv preprint arXiv:2312.09230. [find](https://scholar.google.com/scholar?q=Rhys+Gould%2C+Euan+Ong%2C+George+Ogden%2C+and+Arthur+Conmy%2E+2023%2E+Successor+heads%3A+Recurring%2C+interpretable+attention+heads+in+the+wild%2E+arXiv+preprint+arXiv%3A2312%2E09230%2E)
18. Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. 2025. Your large vision-language model only needs a few attention heads for visual grounding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 9339–9350. [find](https://scholar.google.com/scholar?q=Seil+Kang%2C+Jinyeong+Kim%2C+Junhyeok+Kim%2C+and+Seong+Jae+Hwang%2E+2025%2E+Your+large+vision%2Dlanguage+model+only+needs+a+few+attention+heads+for+visual+grounding%2E+In+Proceedings+of+the+Computer+Vision+and+Patt)
19. Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. 2022. [Ocr-free document understanding transformer](https://md.research.elamur.ai/free/en/papers/kim-2021-ocr-free-document-understanding-transformer/). In European Conference on Computer Vision, pages 498–517. Springer.
20. Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. 2024. [What matters when building vision-language models?](https://md.research.elamur.ai/free/en/papers/lauren-ccon-2024-matters-building-vision-language-models/) Preprint, arXiv:2405.02246.
21. Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. 2023. [Pix2struct: Screenshot parsing as pretraining for visual language understanding](https://md.research.elamur.ai/free/en/papers/lee-2022-pix2struct-screenshot-parsing-pretraining-visual/). In International Conference on Machine Learni…
22. Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, and 1 others. 2022. Grounded language-image pre-training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10965– 109… [find](https://scholar.google.com/scholar?q=Liunian+Harold+Li%2C+Pengchuan+Zhang%2C+Haotian+Zhang%2C+Jianwei+Yang%2C+Chunyuan+Li%2C+Yiwu+Zhong%2C+Lijuan+Wang%2C+Lu+Yuan%2C+Lei+Zhang%2C+Jenq%2DNeng+Hwang%2C+and+1+others%2E+2022%2E+Grounded+language%2Dimage+pre%2Dtraining%2E+In)
23. Ming Li, Ruiyi Zhang, Jian Chen, Jiuxiang Gu, Yufan Zhou, Franck Dernoncourt, Wanrong Zhu, Tianyi Zhou, and Tong Sun. 2025. Towards visual text grounding of multimodal large language model. arXiv preprint arXiv:2504.04974. — [arXiv:2504.04974](https://arxiv.org/abs/2504.04974) [find](https://scholar.google.com/scholar?q=Ming+Li%2C+Ruiyi+Zhang%2C+Jian+Chen%2C+Jiuxiang+Gu%2C+Yufan+Zhou%2C+Franck+Dernoncourt%2C+Wanrong+Zhu%2C+Tianyi+Zhou%2C+and+Tong+Sun%2E+2025%2E+Towards+visual+text+grounding+of+multimodal+large+language+model%2E+arXiv+prep)
24. Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. 2024. [Monkey: Image resolution and text label are important things for large multi-modal models](https://md.research.elamur.ai/free/en/papers/li-2023-monkey-image-resolution-text-label/). In proceedings of the IEEE/CVF conference on computer vision and pattern recognition.
25. Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, and 1 others. 2024b. [Grounding dino: Marrying dino with grounded pre-training for open-set object detection](https://md.research.elamur.ai/free/en/papers/liu-2023-grounding-dino-marrying-dino-grounded/). In European conference on computer vision, pages 38–55. Springer.
26. Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. 2024c. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint arXiv:2403.04473. — [arXiv:2403.04473](https://arxiv.org/abs/2403.04473) [find](https://scholar.google.com/scholar?q=Yuliang+Liu%2C+Biao+Yang%2C+Qiang+Liu%2C+Zhang+Li%2C+Zhiyin+Ma%2C+Shuo+Zhang%2C+and+Xiang+Bai%2E+2024c%2E+Textmonkey%3A+An+ocr%2Dfree+large+multimodal+model+for+understanding+document%2E+arXiv+preprint+arXiv%3A2403%2E04473%2E+%E2%80%94+)
27. Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. 2024. [Deepseek-vl: Towards real-world vision-language understanding](https://md.research.elamur.ai/free/en/papers/lu-2024-deepseek-vl-real-world/). Preprint, arXiv:2403.05525.
28. Bozhi Luan, Hao Feng, Hong Chen, Yonghui Wang, Wengang Zhou, and Houqiang Li. 2024. Textcot: Zoom in for enhanced multimodal text-rich image understanding. arXiv preprint arXiv:2404.09797. — [arXiv:2404.09797](https://arxiv.org/abs/2404.09797) [find](https://scholar.google.com/scholar?q=Bozhi+Luan%2C+Hao+Feng%2C+Hong+Chen%2C+Yonghui+Wang%2C+Wengang+Zhou%2C+and+Houqiang+Li%2E+2024%2E+Textcot%3A+Zoom+in+for+enhanced+multimodal+text%2Drich+image+understanding%2E+arXiv+preprint+arXiv%3A2404%2E09797%2E+%E2%80%94+%5BarXiv%3A24)
29. Ziqiao Ma, Jiayi Pan, and Joyce Chai. 2024. World-to-words: Grounded open vocabulary acquisition through fast mapping in vision-language models. Preprint, arXiv:2306.08685. [find](https://scholar.google.com/scholar?q=Ziqiao+Ma%2C+Jiayi+Pan%2C+and+Joyce+Chai%2E+2024%2E+World%2Dto%2Dwords%3A+Grounded+open+vocabulary+acquisition+through+fast+mapping+in+vision%2Dlanguage+models%2E+Preprint%2C+arXiv%3A2306%2E08685%2E)
30. Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. [Chartqa: A benchmark for question answering about charts with visual and logical reasoning](https://md.research.elamur.ai/free/en/papers/masry-2022-chartqa-benchmark-question-answering-about/). arXiv preprint arXiv:2203.10244.
31. Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. 2022. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1697–1706. [find](https://scholar.google.com/scholar?q=Minesh+Mathew%2C+Viraj+Bagal%2C+Rub%C3%A8n+Tito%2C+Dimosthenis+Karatzas%2C+Ernest+Valveny%2C+and+CV+Jawahar%2E+2022%2E+Infographicvqa%2E+In+Proceedings+of+the+IEEE%2FCVF+Winter+Conference+on+Applications+of+Computer+Vision%2C)
32. Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. 2021. [Docvqa: A dataset for vqa on document images](https://md.research.elamur.ai/free/en/papers/mathew-2020-docvqa-dataset-vqa-document-images/). In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209.
33. Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. 2024. Evaluating text-to-visual generation with image-to-text generation. In European Conference on Computer Vision, pages 366–384. Springer. [find](https://scholar.google.com/scholar?q=Zhiqiu+Lin%2C+Deepak+Pathak%2C+Baiqi+Li%2C+Jiayao+Li%2C+Xide+Xia%2C+Graham+Neubig%2C+Pengchuan+Zhang%2C+and+Deva+Ramanan%2E+2024%2E+Evaluating+text%2Dto%2Dvisual+generation+with+image%2Dto%2Dtext+generation%2E+In+European+Confer)
34. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023a. [Improved baselines with visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/).
35. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024a. Llava-next: Improved reasoning, ocr, and world knowledge. [find](https://scholar.google.com/scholar?q=Haotian+Liu%2C+Chunyuan+Li%2C+Yuheng+Li%2C+Bo+Li%2C+Yuanhan+Zhang%2C+Sheng+Shen%2C+and+Yong+Jae+Lee%2E+2024a%2E+Llava%2Dnext%3A+Improved+reasoning%2C+ocr%2C+and+world+knowledge%2E)
36. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023b. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). Advances in neural information processing systems, 36:34892– 34916.
37. OpenAI. 2024. Introducing gpt-4o. Available at: [https://openai.com/index/hello-gpt-4o](https://openai.com/index/hello-gpt-4o). [↗](https://openai.com/index/hello-gpt-4o)
38. R OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arxiv 2303.08774. View in Article, 2(5):1.
39. Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. 2023. [Kosmos-2: Grounding multimodal large language models to the world](https://md.research.elamur.ai/free/en/papers/peng-2023-kosmos-2-grounding-multimodal-large/). arXiv preprint arXiv:2306.14824.
40. Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, and 1 others. 2025. [Ui-tars: Pioneering automated gui interaction with native agents](https://md.research.elamur.ai/free/en/papers/qin-2025-ui-tars-pioneering-automated/). arXiv preprint arXiv:2501.12326.
41. Fei Tang, Zhangxuan Gu, Zhengxi Lu, Xuyang Liu, Shuheng Shen, Changhua Meng, Wen Wang, Wenqi Zhang, Yongliang Shen, Weiming Lu, and 1 others. 2025. GUI-G 2 : Gaussian reward modeling for gui grounding. arXiv preprint arXiv:2507.15846. — [arXiv:2507.15846](https://arxiv.org/abs/2507.15846) [find](https://scholar.google.com/scholar?q=Fei+Tang%2C+Zhangxuan+Gu%2C+Zhengxi+Lu%2C+Xuyang+Liu%2C+Shuheng+Shen%2C+Changhua+Meng%2C+Wen+Wang%2C+Wenqi+Zhang%2C+Yongliang+Shen%2C+Weiming+Lu%2C+and+1+others%2E+2025%2E+GUI%2DG+2+%3A+Gaussian+reward+modeling+for+gui+grounding)
42. Zineng Tang, Ziyi Yang, Guoxin Wang, Yuwei Fang, Yang Liu, Chenguang Zhu, Michael Zeng, Cha Zhang, and Mohit Bansal. 2023. Unifying vision, text, and layout for universal document processing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19254– 19264. [find](https://scholar.google.com/scholar?q=Zineng+Tang%2C+Ziyi+Yang%2C+Guoxin+Wang%2C+Yuwei+Fang%2C+Yang+Liu%2C+Chenguang+Zhu%2C+Michael+Zeng%2C+Cha+Zhang%2C+and+Mohit+Bansal%2E+2023%2E+Unifying+vision%2C+text%2C+and+layout+for+universal+document+processing%2E+In+Proce)
43. MS Phi-3 Team. 2024. [Phi-3 technical report: A highly capable language model locally on your phone](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-3-technical-report-highly/). Preprint, arXiv:2404.14219.
44. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30.
45. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. [Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned](https://md.research.elamur.ai/free/en/papers/voita-2019-analyzing-multi-head-self-attention/). arXiv preprint arXiv:1905.09418.
46. Haoming Wang, Haoyang Zou, Huatong Song, Jiazhan Feng, Junjie Fang, Junting Lu, Longxiang Liu, Qinyu Luo, Shihao Liang, Shijue Huang, and 1 others. 2025. [Ui-tars-2 technical report: Advancing gui agent with multi-turn reinforcement learning](https://md.research.elamur.ai/free/en/papers/wang-2025-ui-tars-2-technical/). arXiv preprint arXiv:2509.02544.
47. Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. 2023a. [Cogvlm: Visual expert for pretrained language models](https://md.research.elamur.ai/free/en/papers/wang-2023-cogvlm-visual-expert-pretrained/). Preprint, arXiv:2311.03079.
48. Yonghui Wang, Wengang Zhou, Hao Feng, Keyi Zhou, and Houqiang Li. 2023b. Towards improving document understanding: An exploration on text-grounding via mllms. arXiv preprint arXiv:2311.13194. — [arXiv:2311.13194](https://arxiv.org/abs/2311.13194) [find](https://scholar.google.com/scholar?q=Yonghui+Wang%2C+Wengang+Zhou%2C+Hao+Feng%2C+Keyi+Zhou%2C+and+Houqiang+Li%2E+2023b%2E+Towards+improving+document+understanding%3A+An+exploration+on+text%2Dgrounding+via+mllms%2E+arXiv+preprint+arXiv%3A2311%2E13194%2E+%E2%80%94+%5BarXiv)
49. Zilong Wang, Yiheng Xu, Lei Cui, Jingbo Shang, and Furu Wei. 2021. Layoutreader: Pre-training of text and layout for reading order detection. arXiv preprint arXiv:2108.11591. [find](https://scholar.google.com/scholar?q=Zilong+Wang%2C+Yiheng+Xu%2C+Lei+Cui%2C+Jingbo+Shang%2C+and+Furu+Wei%2E+2021%2E+Layoutreader%3A+Pre%2Dtraining+of+text+and+layout+for+reading+order+detection%2E+arXiv+preprint+arXiv%3A2108%2E11591%2E)
50. Qianhui Wu, Kanzhi Cheng, Rui Yang, Chaoyun Zhang, Jianwei Yang, Huiqiang Jiang, Jian Mu, Baolin Peng, Bo Qiao, Reuben Tan, and 1 others. 2025. Gui-actor: Coordinate-free visual grounding for gui agents. arXiv preprint arXiv:2506.03143. — [arXiv:2506.03143](https://arxiv.org/abs/2506.03143) [find](https://scholar.google.com/scholar?q=Qianhui+Wu%2C+Kanzhi+Cheng%2C+Rui+Yang%2C+Chaoyun+Zhang%2C+Jianwei+Yang%2C+Huiqiang+Jiang%2C+Jian+Mu%2C+Baolin+Peng%2C+Bo+Qiao%2C+Reuben+Tan%2C+and+1+others%2E+2025%2E+Gui%2Dactor%3A+Coordinate%2Dfree+visual+grounding+for+gui+agen)
51. Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. 2024. [Retrieval head mechanistically explains long-context factuality](https://md.research.elamur.ai/free/en/papers/wu-2024-retrieval-head-mechanistically-explains-long/). arXiv preprint arXiv:2404.15574.
52. Hai-Ming Xu, Qi Chen, Lei Wang, and Lingqiao Liu. 2025. Attention-driven gui grounding: Leveraging pretrained multimodal large language models without fine-tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 8851– 8859. [find](https://scholar.google.com/scholar?q=Hai%2DMing+Xu%2C+Qi+Chen%2C+Lei+Wang%2C+and+Lingqiao+Liu%2E+2025%2E+Attention%2Ddriven+gui+grounding%3A+Leveraging+pretrained+multimodal+large+language+models+without+fine%2Dtuning%2E+In+Proceedings+of+the+AAAI+Conferenc)
53. Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, and 1 others. 2020a. [Layoutlmv2: Multi-modal pre-training for visually-rich document understanding](https://md.research.elamur.ai/free/en/papers/xu-2021-layoutlmv2-multi-modal-pre-training/). arXiv preprint arXiv:2012.14740.
54. Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. 2020b. [Layoutlm: Pre-training of text and layout for document image understanding](https://md.research.elamur.ai/free/en/papers/xu-2019-layoutlm-pre-training-text-layout/). In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1192–1200.
55. Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, and 4 others. 2024. [Minicpmv: A gpt-4v level mllm on your phone](https://md.research.elamur.ai/free/en/papers/yao-2024-minicpm-v-gpt-4v/). arXiv preprin…
56. Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Junfeng Tian, and 1 others. 2023a. mplug-docowl: Modularized multimodal large language model for document understanding. arXiv preprint arXiv:2307.02499. — [arXiv:2307.02499](https://arxiv.org/abs/2307.02499) [find](https://scholar.google.com/scholar?q=Jiabo+Ye%2C+Anwen+Hu%2C+Haiyang+Xu%2C+Qinghao+Ye%2C+Ming+Yan%2C+Yuhao+Dan%2C+Chenlin+Zhao%2C+Guohai+Xu%2C+Chenliang+Li%2C+Junfeng+Tian%2C+and+1+others%2E+2023a%2E+mplug%2Ddocowl%3A+Modularized+multimodal+large+language+model+for)
57. Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, and 1 others. 2023b. [Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model](https://md.research.elamur.ai/free/en/papers/ye-2023-ureader-universal-ocr-free-visually/). arXiv preprint arXiv:2310.05126.
58. Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. 2023. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704. — [arXiv:2310.07704](https://arxiv.org/abs/2310.07704) [find](https://scholar.google.com/scholar?q=Haoxuan+You%2C+Haotian+Zhang%2C+Zhe+Gan%2C+Xianzhi+Du%2C+Bowen+Zhang%2C+Zirui+Wang%2C+Liangliang+Cao%2C+Shih%2DFu+Chang%2C+and+Yinfei+Yang%2E+2023%2E+Ferret%3A+Refer+and+ground+anything+anywhere+at+any+granularity%2E+arXiv+pre)
59. Yuechen Yu, Yulin Li, Chengquan Zhang, Xiaoqiang Zhang, Zengyuan Guo, Xiameng Qin, Kun Yao, Junyu Han, Errui Ding, and Jingdong Wang. 2023. Structextv2: Masked visual-textual prediction for document image pre-training. arXiv preprint arXiv:2303.00289. — [arXiv:2303.00289](https://arxiv.org/abs/2303.00289) [find](https://scholar.google.com/scholar?q=Yuechen+Yu%2C+Yulin+Li%2C+Chengquan+Zhang%2C+Xiaoqiang+Zhang%2C+Zengyuan+Guo%2C+Xiameng+Qin%2C+Kun+Yao%2C+Junyu+Han%2C+Errui+Ding%2C+and+Jingdong+Wang%2E+2023%2E+Structextv2%3A+Masked+visual%2Dtextual+prediction+for+document+i)
60. Ruiyi Zhang, Yanzhe Zhang, Jian Chen, Yufan Zhou, Jiuxiang Gu, Changyou Chen, and Tong Sun. 2024a. Trins: Towards multimodal language models that can read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22584–22594. [find](https://scholar.google.com/scholar?q=Ruiyi+Zhang%2C+Yanzhe+Zhang%2C+Jian+Chen%2C+Yufan+Zhou%2C+Jiuxiang+Gu%2C+Changyou+Chen%2C+and+Tong+Sun%2E+2024a%2E+Trins%3A+Towards+multimodal+language+models+that+can+read%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on)
61. Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Yu Liu, Kai Chen, and Ping Luo. 2024b. Gpt4roi: Instruction tuning large language model on region-of-interest. In European conference on computer vision, pages 52–70. Springer. [find](https://scholar.google.com/scholar?q=Shilong+Zhang%2C+Peize+Sun%2C+Shoufa+Chen%2C+Min+Xiao%2C+Wenqi+Shao%2C+Wenwei+Zhang%2C+Yu+Liu%2C+Kai+Chen%2C+and+Ping+Luo%2E+2024b%2E+Gpt4roi%3A+Instruction+tuning+large+language+model+on+region%2Dof%2Dinterest%2E+In+European+co)
62. Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. 2023. [Llavar: Enhanced visual instruction tuning for text-rich image understanding](https://md.research.elamur.ai/free/en/papers/zhang-2023-llavar-enhanced-visual-instruction-tuning/). arXiv preprint arXiv:2306.17107.
63. Xu Zhong, Elaheh ShafieiBavani, and Antonio Jimeno Yepes. 2020. Image-based table recognition: data, model, and evaluation. In European conference on computer vision, pages 564–580. Springer. [find](https://scholar.google.com/scholar?q=Xu+Zhong%2C+Elaheh+ShafieiBavani%2C+and+Antonio+Jimeno+Yepes%2E+2020%2E+Image%2Dbased+table+recognition%3A+data%2C+model%2C+and+evaluation%2E+In+European+conference+on+computer+vision%2C+pages+564%E2%80%93580%2E+Springer%2E)
64. Shijie Zhou, Ruiyi Zhang, Yufan Zhou, and Changyou Chen. 2024a. A high-quality text-rich image instruction tuning dataset via hybrid instruction generation. arXiv preprint arXiv:2412.16364. [find](https://scholar.google.com/scholar?q=Shijie+Zhou%2C+Ruiyi+Zhang%2C+Yufan+Zhou%2C+and+Changyou+Chen%2E+2024a%2E+A+high%2Dquality+text%2Drich+image+instruction+tuning+dataset+via+hybrid+instruction+generation%2E+arXiv+preprint+arXiv%3A2412%2E16364%2E)
65. Shijie Zhou, Ruiyi Zhang, Huaisheng Zhu, Branislav Kveton, Yufan Zhou, Jiuxiang Gu, Jian Chen, and Changyou Chen. 2025. Multimodal llms as customized reward models for text-to-image generation. arXiv preprint arXiv:2507.21391. — [arXiv:2507.21391](https://arxiv.org/abs/2507.21391) [find](https://scholar.google.com/scholar?q=Shijie+Zhou%2C+Ruiyi+Zhang%2C+Huaisheng+Zhu%2C+Branislav+Kveton%2C+Yufan+Zhou%2C+Jiuxiang+Gu%2C+Jian+Chen%2C+and+Changyou+Chen%2E+2025%2E+Multimodal+llms+as+customized+reward+models+for+text%2Dto%2Dimage+generation%2E+arXiv+)
66. Zhenhong Zhou, Haiyang Yu, Xinghua Zhang, Rongwu Xu, Fei Huang, Kun Wang, Yang Liu, Junfeng Fang, and Yongbin Li. 2024b. On the role of attention heads in large language model safety. arXiv preprint arXiv:2410.13708. — [arXiv:2410.13708](https://arxiv.org/abs/2410.13708) [find](https://scholar.google.com/scholar?q=Zhenhong+Zhou%2C+Haiyang+Yu%2C+Xinghua+Zhang%2C+Rongwu+Xu%2C+Fei+Huang%2C+Kun+Wang%2C+Yang+Liu%2C+Junfeng+Fang%2C+and+Yongbin+Li%2E+2024b%2E+On+the+role+of+attention+heads+in+large+language+model+safety%2E+arXiv+preprint+a)

*Updated 2 November 2026*