---
title: "OCR or Not? Rethinking Document Information Extraction in the"
description: "Scoring uses key–value F1. The authors also build an automated hierarchical error analysis framework… 15 references, cited by 1 paper here."
authors: ["Jiyuan Shen", "Yuan Peiyue", "Atin Ghosh", "Yifan Mai", "Daniel Dahlmeier"]
year: 2026
venue: "Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track)"
source: "https://aclanthology.org/2026.eacl-industry.28/"
doi: "10.18653/v1/2026.eacl-industry.28"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shen-2026-ocr-or-not-rethinking-document/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shen-2026-ocr-or-not-rethinking-document/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/15/) / OCR or Not? Rethinking Document Information Extraction in the MLLMs Era with Real-World Large-Scale Datasets

# OCR or Not? Rethinking Document Information Extraction in the MLLMs Era with Real-World Large-Scale Datasets

**Authors:** Jiyuan Shen, Yuan Peiyue, Atin Ghosh, [Yifan Mai](https://md.research.elamur.ai/free/en/authors/yifan-mai/) and [Daniel Dahlmeier](https://md.research.elamur.ai/free/en/authors/daniel-dahlmeier/) · **Year:** 2026 · **Venue:** [Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2026](https://md.research.elamur.ai/free/en/venues/eacl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.eacl-industry.28/) · [PDF](https://aclanthology.org/2026.eacl-industry.28.pdf) · [DOI 10.18653/v1/2026.eacl-industry.28](https://doi.org/10.18653/v1/2026.eacl-industry.28) · [OpenAlex](https://openalex.org/W7140110720)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Industrial document extraction systems usually run OCR first and then pass the recognized text to a separate extraction model. These pipelines are complex and carry OCR mistakes forward into extraction. The authors, from SAP and Stanford, ask whether modern multimodal LLMs can skip OCR entirely.

They benchmark off-the-shelf MLLMs on two internal, multilingual business-document datasets: C1 (supply chain) and C2 (finance). The extraction is zero-shot, with a schema-based prompt that returns JSON containing header fields and line items. Each model is tested with three inputs: page image only, in-house OCR text only, and both together. Scoring uses key–value F1.

The authors also build an automated hierarchical error analysis framework with three stages:
- a rule-based error handler,
- LLM/MLLM reasoning over the errors,
- clustering of the resulting explanations.

The strongest models perform about as well, or better, on images alone as on OCR-enhanced input. Image-only input also produces the fewest errors. Using the error diagnosis to refine the schema, format and prompt raises Gemini 1.5 Pro further. This suggests that simpler MLLM-only pipelines are a realistic option for enterprise document AI.

## Key points
- **Models evaluated:**
  - Llama 4 Scout and Llama 4 Maverick
  - Pixtral Large (2411)
  - Nova Pro
  - GPT-4o and GPT-4o mini
  - Claude 3 Opus and Claude 3.5 Sonnet
  - Gemini 1.5 Pro, Gemini 2.0 Pro and Gemini 2.5 Flash
- **Best result:** Gemini 1.5 Pro with image-only input scored a mean F1 of 76.8 (87.3 on C1, 66.4 on C2). This beat its OCR-only (≈74.1) and image+OCR (≈75.6) results. A refined prompt then lifted the image-only mean to 78.9.
- **Error types:** The framework groups errors into three classes:
  - text misinterpretation;
  - image-to-text extraction issues, which are higher without OCR;
  - OCR/schema ambiguity, which is lower with image-only input.
- **Error clustering:** Explanations are grouped using BERT-based embedding clustering with cosine similarity.
- **Cost and speed (closed-source models):**
  - Gemini 2.5 Flash was fastest, at about 1.4 s per page.
  - Gemini 1.5 Pro was cheapest, at about $0.001 per page.
  - Claude 3 Opus was the slowest and most expensive, at about 7.0 s and $0.05 per page.
- **Limitations:** Few-shot learning, CoT/self-reflection and reasoning models (O1, DeepSeek R1) were not systematically explored. The datasets are internal and not released.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 434 works · 314,419 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [🔎 Information extraction](https://md.research.elamur.ai/free/en/topics/information-extraction/) · 167 works

## Cite

Cite as: Jiyuan Shen, Yuan Peiyue, Atin Ghosh, Yifan Mai and Daniel Dahlmeier. 2026. OCR or Not? Rethinking Document Information Extraction in the MLLMs Era with Real-World Large-Scale Datasets. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{shen2026ocr,
  title = {{OCR or Not? Rethinking Document Information Extraction in the MLLMs Era with Real-World Large-Scale Datasets}},
  author = {Jiyuan Shen, Yuan Peiyue, Atin Ghosh, Yifan Mai and Daniel Dahlmeier},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-industry.28},
  url = {https://doi.org/10.18653/v1/2026.eacl-industry.28}
}
```

- [← previous Nwāchā Munā: A Devanagari Speech Corpus and Proximal Transfer Benchmark for Nepal Bhasha ASR](https://md.research.elamur.ai/free/en/papers/sharma-2026-nwacha-muna-devanagari-speech-corpus/)
- [next → OCRTurk: A Comprehensive OCR Benchmark for Turkish](https://md.research.elamur.ai/free/en/papers/yilmaz-2026-ocrturk-comprehensive-ocr-benchmark-turkish/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/15/)

## References

15 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. [Qwen technical report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/). arXiv preprint arXiv:2309.16609.
2. Yinfang Chen, Huaibing Xie, Minghua Ma, Yu Kang, Xin Gao, Liu Shi, Yunjie Cao, Xuedong Gao, Hao Fan, Ming Wen, et al. 2024. Automatic root cause analysis via large language models for cloud incidents. In Proceedings of the Nineteenth European Conference on Computer Systems, pages 674–688. [find](https://scholar.google.com/scholar?q=Yinfang+Chen%2C+Huaibing+Xie%2C+Minghua+Ma%2C+Yu+Kang%2C+Xin+Gao%2C+Liu+Shi%2C+Yunjie+Cao%2C+Xuedong+Gao%2C+Hao+Fan%2C+Ming+Wen%2C+et+al%2E+2024%2E+Automatic+root+cause+analysis+via+large+language+models+for+cloud+incidents%2E)
3. Gartner. Intelligent document processing solutions reviews and ratings. [https://www.gartner.com/reviews/market/](https://www.gartner.com/reviews/market/) intelligent-document-processing-solutions. [↗](https://www.gartner.com/reviews/market/)
4. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948.
5. Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022. [LayoutLMv3: Pre-training for document ai with unified text and image masking](https://md.research.elamur.ai/free/en/papers/huang-2022-layoutlmv3-pre-training-document-ai/). In Proceedings of the 30th ACM international conference on multimedia, pages 4083–4091.
6. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. [Gpt-4o system card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/). arXiv preprint arXiv:2410.21276.
7. Amazon Artificial General Intelligence. 2024. The amazon nova family of models: Technical report and model card. [find](https://scholar.google.com/scholar?q=Amazon+Artificial+General+Intelligence%2E+2024%2E+The+amazon+nova+family+of+models%3A+Technical+report+and+model+card%2E)
8. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024. [Openai o1 system card](https://md.research.elamur.ai/free/en/papers/openai-2024-openai-o1-system-card/). arXiv preprint arXiv:2412.16720.
9. Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. 2024. TextMonkey: An ocr-free large multimodal model for understanding document. arXiv preprint arXiv:2403.04473. — [arXiv:2403.04473](https://arxiv.org/abs/2403.04473) [find](https://scholar.google.com/scholar?q=Yuliang+Liu%2C+Biao+Yang%2C+Qiang+Liu%2C+Zhang+Li%2C+Zhiyin+Ma%2C+Shuo+Zhang%2C+and+Xiang+Bai%2E+2024%2E+TextMonkey%3A+An+ocr%2Dfree+large+multimodal+model+for+understanding+document%2E+arXiv+preprint+arXiv%3A2403%2E04473%2E+%E2%80%94+%5B)
10. MistralAI. Mistral ocr technique report. [https://mistral.ai/news/mistral-ocr](https://mistral.ai/news/mistral-ocr). [↗](https://mistral.ai/news/mistral-ocr)
11. Linke Ouyang, Yuan Qu, Hongbin Zhou, Jiawei Zhu, Rui Zhang, Qunshu Lin, Bin Wang, Zhiyuan Zhao, Man Jiang, Xiaomeng Zhao, et al. 2025. Omnidocbench: Benchmarking diverse pdf document parsing with comprehensive annotations. In Proceedings of the Computer Vision and Pattern Recognition Conference, pag… [find](https://scholar.google.com/scholar?q=Linke+Ouyang%2C+Yuan+Qu%2C+Hongbin+Zhou%2C+Jiawei+Zhu%2C+Rui+Zhang%2C+Qunshu+Lin%2C+Bin+Wang%2C+Zhiyuan+Zhao%2C+Man+Jiang%2C+Xiaomeng+Zhao%2C+et+al%2E+2025%2E+Omnidocbench%3A+Benchmarking+diverse+pdf+document+parsing+with+comp)
12. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805.
13. Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/). arXiv preprint arXiv:2403.05530.
14. Zilong Wang, Yichao Zhou, Wei Wei, Chen-Yu Lee, and Sandeep Tata. 2023. VRDU: A benchmark for visually-rich document understanding. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5184–5193. [find](https://scholar.google.com/scholar?q=Zilong+Wang%2C+Yichao+Zhou%2C+Wei+Wei%2C+Chen%2DYu+Lee%2C+and+Sandeep+Tata%2E+2023%2E+VRDU%3A+A+benchmark+for+visually%2Drich+document+understanding%2E+In+Proceedings+of+the+29th+ACM+SIGKDD+Conference+on+Knowledge+Discov)
15. Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Junfeng Tian, et al. 2023. mPLUG-DocOwl: Modularized multimodal large language model for document understanding. arXiv preprint arXiv:2307.02499. — [arXiv:2307.02499](https://arxiv.org/abs/2307.02499) [find](https://scholar.google.com/scholar?q=Jiabo+Ye%2C+Anwen+Hu%2C+Haiyang+Xu%2C+Qinghao+Ye%2C+Ming+Yan%2C+Yuhao+Dan%2C+Chenlin+Zhao%2C+Guohai+Xu%2C+Chenliang+Li%2C+Junfeng+Tian%2C+et+al%2E+2023%2E+mPLUG%2DDocOwl%3A+Modularized+multimodal+large+language+model+for+documen)

*Updated 31 October 2026*