---
title: "Multimodal Table Understanding — paper by Zheng et al. (2024)"
description: "Most table understanding systems, including recent LLM-based ones, need tables serialized as text… 66 references, cited by 36 papers here."
authors: ["Mingyu Zheng", "Xinwei Feng", "Qingyi Si", "Qiaoqiao She", "Zheng Lin", "Wenbin Jiang", "Weiping Wang"]
year: 2024
venue: "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.acl-long.493/"
doi: "10.18653/v1/2024.acl-long.493"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zheng-2024-multimodal-table-understanding/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zheng-2024-multimodal-table-understanding/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / Multimodal Table Understanding

# Multimodal Table Understanding

**Authors:** Mingyu Zheng, Xinwei Feng, Qingyi Si, [Qiaoqiao She](https://md.research.elamur.ai/free/en/authors/qiaoqiao-she/), [Zheng Lin](https://md.research.elamur.ai/free/en/authors/zheng-lin/), [Wenbin Jiang](https://md.research.elamur.ai/free/en/authors/wenbin-jiang/) and [Weiping Wang](https://md.research.elamur.ai/free/en/authors/weiping-wang/) · **Year:** 2024 · **Venue:** [Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2024](https://md.research.elamur.ai/free/en/venues/acl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.acl-long.493/) · [PDF](https://aclanthology.org/2024.acl-long.493.pdf) · [DOI 10.18653/v1/2024.acl-long.493](https://doi.org/10.18653/v1/2024.acl-long.493) · [OpenAlex](https://openalex.org/W4402671591)

**Citations:** 26 citations (OpenAlex, 2026-10-14) · [36](https://md.research.elamur.ai/free/en/papers/zheng-2024-multimodal-table-understanding/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most table understanding systems, including recent LLM-based ones, need tables serialized as text such as HTML or Markdown. In practice, tables often exist only as images, for example in scans or screenshots. The authors frame this as a new task, multimodal table understanding: a model receives a table image and a request and must produce the correct answer end to end.

To support it, they built MMTab by rendering tables from 14 public datasets across 8 domains into images with varied structures and styles. Each sample becomes an \<image, request, response> triple, and the data is augmented at several levels. The dataset also adds table structure understanding tasks that earlier work neglected.

On this data they trained Table-LLaVA from LLaVA-1.5 in two stages:
- **Pre-training:** table recognition, where the model converts table images to HTML.
- **Instruction tuning:** mixed table-based downstream tasks.

Table-LLaVA beats open-source MLLM baselines on 23 benchmarks and rivals GPT-4V on many of them. This matters because it lets models read tables directly from pixels.

## Key points
- **MMTab has three parts:** MMTab-pre (150K recognition samples, 97K images), MMTab-instruct (232K samples, 14 tasks, 82K images) and MMTab-eval (49K test samples, 23K images, 17 held-in and 7 held-out benchmarks).
- **Baselines:** popular MLLMs such as MiniGPT-4 and BLIP2 score near zero on most table tasks.
- **Results:** Table-LLaVA leads open-source models on 17 held-in and 6 held-out benchmarks. It matches or beats GPT-4V on 14 of 24 benchmarks, though GPT-4V generalizes better on held-out data. Vary-toy wins on AIT-QA, likely due to its higher input resolution and similar training documents.
- **Ablations:** dropping MMTab-instruct collapses held-out performance to 3.02. Removing the structure-understanding (TSU) data mainly hurts TSU and held-out tasks. Table-recognition pre-training helps more than generic LLaVA pre-training.
- **Non-tabular tasks:** MMTab training also improves results on TextVQA and LLaVA-Bench (In-the-Wild), suggesting the two kinds of skill reinforce each other. Code and data are released on GitHub (SpursGoZmy/Table-LLaVA).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 657 works

## Cite

Cite as: Mingyu Zheng, Xinwei Feng, Qingyi Si, Qiaoqiao She, Zheng Lin, Wenbin Jiang and Weiping Wang. 2024. Multimodal Table Understanding. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zheng2024multimodal,
  title = {{Multimodal Table Understanding}},
  author = {Mingyu Zheng, Xinwei Feng, Qingyi Si, Qiaoqiao She, Zheng Lin, Wenbin Jiang and Weiping Wang},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.493},
  url = {https://doi.org/10.18653/v1/2024.acl-long.493}
}
```

- [← previous Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffolding](https://md.research.elamur.ai/free/en/papers/suzgun-2024-meta-prompting-enhancing-language-models/)
- [next → Red Teaming Language Model Detectors with Language Models](https://md.research.elamur.ai/free/en/papers/shi-2024-red-teaming-language-model-detectors/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/)

## References

66 references, 35 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mubashara Akhtar, Oana Cocarascu, and Elena Simperl. 2022. PubHealthTab: A public health table-based dataset for evidence-based fact checking. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1–16, Seattle, United States. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mubashara+Akhtar%2C+Oana+Cocarascu%2C+and+Elena+Simperl%2E+2022%2E+PubHealthTab%3A+A+public+health+table%2Dbased+dataset+for+evidence%2Dbased+fact+checking%2E+In+Findings+of+the+Association+for+Computational+Linguist)
2. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. [Flamingo: a visual language model for few-shot learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/). Advances in Neural Information Processing Systems, 35:23716–23736.
3. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. [Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/).
4. Sara Bonfitto, Elena Casiraghi, and Marco Mesiti. 2021. Table understanding approaches for extracting knowledge from heterogeneous tables. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 11(4):e1407. [find](https://scholar.google.com/scholar?q=Sara+Bonfitto%2C+Elena+Casiraghi%2C+and+Marco+Mesiti%2E+2021%2E+Table+understanding+approaches+for+extracting+knowledge+from+heterogeneous+tables%2E+Wiley+Interdisciplinary+Reviews%3A+Data+Mining+and+Knowledge+Di)
5. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
6. Pei Chen, Soumajyoti Sarkar, Leonard Lausen, Balasubramaniam Srinivasan, Sheng Zha, Ruihong Huang, and George Karypis. 2023a. Hytrel: Hypergraph-enhanced tabular data representation learning. In Thirty-seventh Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Pei+Chen%2C+Soumajyoti+Sarkar%2C+Leonard+Lausen%2C+Balasubramaniam+Srinivasan%2C+Sheng+Zha%2C+Ruihong+Huang%2C+and+George+Karypis%2E+2023a%2E+Hytrel%3A+Hypergraph%2Denhanced+tabular+data+representation+learning%2E+In+Thirt)
7. Wenhu Chen. 2023. [Large language models are few(1)-shot table reasoners](https://md.research.elamur.ai/free/en/papers/chen-2023-large-language-models-are-few/). In Findings of the Association for Computational Linguistics: EACL 2023, pages 1120–1130, Dubrovnik, Croatia. Association for Computational Linguistics.
8. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2023b. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. [find](https://scholar.google.com/scholar?q=Wenhu+Chen%2C+Xueguang+Ma%2C+Xinyi+Wang%2C+and+William+W%2E+Cohen%2E+2023b%2E+Program+of+thoughts+prompting%3A+Disentangling+computation+from+reasoning+for+numerical+reasoning+tasks%2E)
9. Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020. [Tabfact: A large-scale dataset for table-based fact verification](https://md.research.elamur.ai/free/en/papers/chen-2019-tabfact-large-scale-dataset-table/).
10. Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2023c. [Longlora: Efficient fine-tuning of long-context large language models](https://md.research.elamur.ai/free/en/papers/chen-2023-longlora-efficient-fine-tuning-long/). 9111 10
11. Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, Jian-Guang Lou, and Dongmei Zhang. 2022. HiTab: A hierarchical table dataset for question answering and natural language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistic… [find](https://scholar.google.com/scholar?q=Zhoujun+Cheng%2C+Haoyu+Dong%2C+Zhiruo+Wang%2C+Ran+Jia%2C+Jiaqi+Guo%2C+Yan+Gao%2C+Shi+Han%2C+Jian%2DGuang+Lou%2C+and+Dongmei+Zhang%2E+2022%2E+HiTab%3A+A+hierarchical+table+dataset+for+question+answering+and+natural+language+g)
12. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. [find](https://scholar.google.com/scholar?q=Wei%2DLin+Chiang%2C+Zhuohan+Li%2C+Zi+Lin%2C+Ying+Sheng%2C+Zhanghao+Wu%2C+Hao+Zhang%2C+Lianmin+Zheng%2C+Siyuan+Zhuang%2C+Yonghao+Zhuang%2C+Joseph+E%2E+Gonzalez%2C+Ion+Stoica%2C+and+Eric+P%2E+Xing%2E+2023%2E+Vicuna%3A+An+open%2Dsource+cha)
13. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… — [PaLM: Scaling Language Modeling with Pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
14. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems (NeurIPS). [find](https://scholar.google.com/scholar?q=Tri+Dao%2C+Daniel+Y%2E+Fu%2C+Stefano+Ermon%2C+Atri+Rudra%2C+and+Christopher+R%C3%A9%2E+2022%2E+FlashAttention%3A+Fast+and+memory%2Defficient+exact+attention+with+IO%2Dawareness%2E+In+Advances+in+Neural+Information+Processing+Sy)
15. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/).
16. Haoyu Dong, Zhoujun Cheng, Xinyi He, Mengyu Zhou, Anda Zhou, Fan Zhou, Ao Liu, Shi Han, and Dongmei Zhang. 2022. Table pre-training: A survey on model architectures, pre-training objectives, and downstream tasks. [find](https://scholar.google.com/scholar?q=Haoyu+Dong%2C+Zhoujun+Cheng%2C+Xinyi+He%2C+Mengyu+Zhou%2C+Anda+Zhou%2C+Fan+Zhou%2C+Ao+Liu%2C+Shi+Han%2C+and+Dongmei+Zhang%2E+2022%2E+Table+pre%2Dtraining%3A+A+survey+on+model+architectures%2C+pre%2Dtraining+objectives%2C+and+downs)
17. Wolfgang Gatterbauer, Paul Bohunsky, Marcus Herzog, Bernhard Krüpl, and Bernhard Pollak. 2007. Towards domain-independent information extraction from web tables. In Proceedings of the 16th international conference on World Wide Web, pages 71–80. [find](https://scholar.google.com/scholar?q=Wolfgang+Gatterbauer%2C+Paul+Bohunsky%2C+Marcus+Herzog%2C+Bernhard+Kr%C3%BCpl%2C+and+Bernhard+Pollak%2E+2007%2E+Towards+domain%2Dindependent+information+extraction+from+web+tables%2E+In+Proceedings+of+the+16th+internation)
18. Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. 2020. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673. [find](https://scholar.google.com/scholar?q=Robert+Geirhos%2C+J%C3%B6rn%2DHenrik+Jacobsen%2C+Claudio+Michaelis%2C+Richard+Zemel%2C+Wieland+Brendel%2C+Matthias+Bethge%2C+and+Felix+A%2E+Wichmann%2E+2020%2E+Shortcut+learning+in+deep+neural+networks%2E+Nature+Machine+Intelli)
19. Vivek Gupta, Maitrey Mehta, Pegah Nokhiz, and Vivek Srikumar. 2020. [INFOTABS: Inference on tables as semi-structured data](https://md.research.elamur.ai/free/en/papers/gupta-2020-infotabs-inference-tables-as-semi/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2309–2324, Online. Association for Computational Linguistics.
20. Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020. [TaPas: Weakly supervised table parsing via pre-training](https://md.research.elamur.ai/free/en/papers/herzig-2020-tapas-weakly-supervised-table-parsing/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4320–4333, Online. Association for Compu…
21. Sujay Kumar Jauhar, Peter Turney, and Eduard Hovy. 2016. Tabmcq: A dataset of general knowledge tables and multiple-choice questions. [find](https://scholar.google.com/scholar?q=Sujay+Kumar+Jauhar%2C+Peter+Turney%2C+and+Eduard+Hovy%2E+2016%2E+Tabmcq%3A+A+dataset+of+general+knowledge+tables+and+multiple%2Dchoice+questions%2E)
22. Yannis Katsis, Saneem Chemmengath, Vishwajeet Kumar, Samarth Bharadwaj, Mustafa Canim, Michael Glass, Alfio Gliozzo, Feifei Pan, Jaydeep Sen, Karthik Sankaranarayanan, and Soumen Chakrabarti. 2021. Ait-qa: Question answering dataset over complex tables in the airline industry. [find](https://scholar.google.com/scholar?q=Yannis+Katsis%2C+Saneem+Chemmengath%2C+Vishwajeet+Kumar%2C+Samarth+Bharadwaj%2C+Mustafa+Canim%2C+Michael+Glass%2C+Alfio+Gliozzo%2C+Feifei+Pan%2C+Jaydeep+Sen%2C+Karthik+Sankaranarayanan%2C+and+Soumen+Chakrabarti%2E+2021%2E+Ai)
23. Larissa R. Lautert, Marcelo M. Scheidt, and Carina F. Dorneles. 2013. Web table taxonomy and formalization. SIGMOD Rec., 42(3):28–33. [find](https://scholar.google.com/scholar?q=Larissa+R%2E+Lautert%2C+Marcelo+M%2E+Scheidt%2C+and+Carina+F%2E+Dorneles%2E+2013%2E+Web+table+taxonomy+and+formalization%2E+SIGMOD+Rec%2E%2C+42%283%29%3A28%E2%80%9333%2E)
24. Rémi Lebret, David Grangier, and Michael Auli. 2016. [Neural text generation from structured data with application to the biography domain](https://md.research.elamur.ai/free/en/papers/lebret-2016-neural-text-generation-structured-data/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1203–1213, Austin, Texas. Association for Computational Linguist…
25. Hongxin Li, Jingran Su, Yuntao Chen, Qing Li, and Zhaoxiang Zhang. 2023a. Sheetcopilot: Bringing software productivity to the next level through large language models. In Thirty-seventh Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Hongxin+Li%2C+Jingran+Su%2C+Yuntao+Chen%2C+Qing+Li%2C+and+Zhaoxiang+Zhang%2E+2023a%2E+Sheetcopilot%3A+Bringing+software+productivity+to+the+next+level+through+large+language+models%2E+In+Thirty%2Dseventh+Conference+on+)
26. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023b. [Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/).
27. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. [Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation](https://md.research.elamur.ai/free/en/papers/li-2022-blip-bootstrapping-language-image-pre/). In ICML.
28. Peng Li, Yeye He, Dror Yashar, Weiwei Cui, Song Ge, Haidong Zhang, Danielle Rifinski Fainman, Dongmei Zhang, and Surajit Chaudhuri. 2023c. Table-gpt: Table-tuned gpt for diverse table tasks. [find](https://scholar.google.com/scholar?q=Peng+Li%2C+Yeye+He%2C+Dror+Yashar%2C+Weiwei+Cui%2C+Song+Ge%2C+Haidong+Zhang%2C+Danielle+Rifinski+Fainman%2C+Dongmei+Zhang%2C+and+Surajit+Chaudhuri%2E+2023c%2E+Table%2Dgpt%3A+Table%2Dtuned+gpt+for+diverse+table+tasks%2E)
29. Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. 2023d. Monkey: Image resolution and text label are important things for large multi-modal models. arXiv preprint arXiv:2311.06607. 9112 11 [find](https://scholar.google.com/scholar?q=Zhang+Li%2C+Biao+Yang%2C+Qiang+Liu%2C+Zhiyin+Ma%2C+Shuo+Zhang%2C+Jingxu+Yang%2C+Yabo+Sun%2C+Yuliang+Liu%2C+and+Xiang+Bai%2E+2023d%2E+Monkey%3A+Image+resolution+and+text+label+are+important+things+for+large+multi%2Dmodal+mode)
30. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023a. [Improved baselines with visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/).
31. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024. Llava-next: Improved reasoning, ocr, and world knowledge. [find](https://scholar.google.com/scholar?q=Haotian+Liu%2C+Chunyuan+Li%2C+Yuheng+Li%2C+Bo+Li%2C+Yuanhan+Zhang%2C+Sheng+Shen%2C+and+Yong+Jae+Lee%2E+2024%2E+Llava%2Dnext%3A+Improved+reasoning%2C+ocr%2C+and+world+knowledge%2E)
32. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023b. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). In NeurIPS.
33. Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and Jian-Guang Lou. 2022. [TAPEX: Table pre-training via learning a neural SQL executor](https://md.research.elamur.ai/free/en/papers/liu-2021-tapex-table-pre-training-learning/). In International Conference on Learning Representations.
34. Qian Liu, Fan Zhou, Zhengbao Jiang, Longxu Dou, and Min Lin. 2023c. [From zero to hero: Examining the power of symbolic tasks in instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-zero-hero-examining-power-symbolic/).
35. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2023d. [Mmbench: Is your multi-modal model an all-around player?](https://md.research.elamur.ai/free/en/papers/liu-2023-mmbench-multi-modal-model-all/)
36. Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023a. [Chameleon: Plug-and-play compositional reasoning with large language models](https://md.research.elamur.ai/free/en/papers/lu-2023-chameleon-plug-play-compositional-reasoning/). In The 37th Conference on Neural Information Processing Systems (NeurIPS).
37. Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. 2023b. [Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning](https://md.research.elamur.ai/free/en/papers/lu-2022-dynamic-prompt-learning-policy-gradient/). In International Conference on Learning Representations (ICLR).
38. Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, and Zhaopeng Tu. 2023. [Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration](https://md.research.elamur.ai/free/en/papers/lyu-2023-macaw-llm-multi-modal-language/). arXiv preprint arXiv:2306.09093.
39. Linyong Nan, Chiachun Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojciech Kryściński, Hailey Schoelkopf, Riley Kong, Xiangru Tang, Mutethia Mutuma, Ben Rosand, Isabel Trindade, Renusree Bandaru, Jacob Cunningham, Caiming Xiong, and Dragomir Radev. 2022. [Fetaqa: Free-form table ques…](https://md.research.elamur.ai/free/en/papers/nan-2022-fetaqa-free-form-table-question/)
40. PaddleOCR. 2024. table recognition model from pp-structure. [https://github.com/PaddlePaddle/](https://github.com/PaddlePaddle/) PaddleOCR/tree/release/2.7/ppstructure/ table. Accessed on: 2024-02-14. [↗](https://github.com/PaddlePaddle/)
41. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, page 311–318, USA. Association for Computational Linguistics.
42. Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020. [ToTTo: A controlled table-to-text generation dataset](https://md.research.elamur.ai/free/en/papers/parikh-2020-totto-controlled-table-text-generation/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1173–1186, Online. Associat…
43. Panupong Pasupat and Percy Liang. 2015. [Compositional semantic parsing on semi-structured tables](https://md.research.elamur.ai/free/en/papers/pasupat-2015-compositional-semantic-parsing-semi-structured/). In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1470–…
44. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/).
45. Alexey Shigarov. 2023. Table understanding: Problem overview. WIREs Data Mining and Knowledge Discovery, 13(1):e1482. [find](https://scholar.google.com/scholar?q=Alexey+Shigarov%2E+2023%2E+Table+understanding%3A+Problem+overview%2E+WIREs+Data+Mining+and+Knowledge+Discovery%2C+13%281%29%3Ae1482%2E)
46. Qingyi Si, Tong Wang, Zheng Lin, Xu Zhang, Yanan Cao, and Weiping Wang. 2023. An empirical study of instruction-tuning large language models in chinese. [find](https://scholar.google.com/scholar?q=Qingyi+Si%2C+Tong+Wang%2C+Zheng+Lin%2C+Xu+Zhang%2C+Yanan+Cao%2C+and+Weiping+Wang%2E+2023%2E+An+empirical+study+of+instruction%2Dtuning+large+language+models+in+chinese%2E)
47. Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326. [find](https://scholar.google.com/scholar?q=Amanpreet+Singh%2C+Vivek+Natarajan%2C+Meet+Shah%2C+Yu+Jiang%2C+Xinlei+Chen%2C+Dhruv+Batra%2C+Devi+Parikh%2C+and+Marcus+Rohrbach%2E+2019%2E+Towards+vqa+models+that+can+read%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+)
48. Yuan Sui, Mengyu Zhou, Mingjie Zhou, Shi Han, and Dongmei Zhang. 2023. Gpt4table: Can large language models understand structured table data? a benchmark and empirical study. [find](https://scholar.google.com/scholar?q=Yuan+Sui%2C+Mengyu+Zhou%2C+Mingjie+Zhou%2C+Shi+Han%2C+and+Dongmei+Zhang%2E+2023%2E+Gpt4table%3A+Can+large+language+models+understand+structured+table+data%3F+a+benchmark+and+empirical+study%2E)
49. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+Dan+Bikel%2C+Lukas+Blecher%2C+Cristian+Canton+Fe)
50. Scialom. 2023. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/).
51. Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022. [Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework](https://md.research.elamur.ai/free/en/papers/wang-2022-ofa-unifying-architectures-tasks-modalities/). CoRR, abs/2202.03052.
52. Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, Jinrong Yang, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. 2023. Vary: Scaling up the vision vocabulary for large vision-language models. [find](https://scholar.google.com/scholar?q=Haoran+Wei%2C+Lingyu+Kong%2C+Jinyue+Chen%2C+Liang+Zhao%2C+Zheng+Ge%2C+Jinrong+Yang%2C+Jianjian+Sun%2C+Chunrui+Han%2C+and+Xiangyu+Zhang%2E+2023%2E+Vary%3A+Scaling+up+the+vision+vocabulary+for+large+vision%2Dlanguage+models%2E)
53. Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, En Yu, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. 2024. Small language model meets with reinforced vision vocabulary. arXiv preprint arXiv:2401.12503. [find](https://scholar.google.com/scholar?q=Haoran+Wei%2C+Lingyu+Kong%2C+Jinyue+Chen%2C+Liang+Zhao%2C+Zheng+Ge%2C+En+Yu%2C+Jianjian+Sun%2C+Chunrui+Han%2C+and+Xiangyu+Zhang%2E+2024%2E+Small+language+model+meets+with+reinforced+vision+vocabulary%2E+arXiv+preprint+arXi)
54. Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017. [Challenges in data-to-document generation](https://md.research.elamur.ai/free/en/papers/wiseman-2017-challenges-data-document-generation/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2253–2263, Copenhagen, Denmark. Association for Computational Linguistics.
55. Zhiyang Xu, Ying Shen, and Lifu Huang. 2023. Multiinstruct: Improving multi-modal zero-shot learning via instruction tuning. [find](https://scholar.google.com/scholar?q=Zhiyang+Xu%2C+Ying+Shen%2C+and+Lifu+Huang%2E+2023%2E+Multiinstruct%3A+Improving+multi%2Dmodal+zero%2Dshot+learning+via+instruction+tuning%2E)
56. Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. 2023a. [mplug-owl: Modularization empowers large language models with multimodality](https://md.research.elamur.ai/free/en/papers/ye-2023-mplug-owl-modularization-empowers-large/).
57. Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 2023b. [mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration](https://md.research.elamur.ai/free/en/papers/ye-2023-mplug-owl2-revolutionizing-multi-modal/).
58. Dahua Lin, and Jiaqi Wang. 2023a. [Internlm-xcomposer: A vision-language large model for advanced text-image comprehension and composition](https://md.research.elamur.ai/free/en/papers/zhang-2023-internlm-xcomposer-vision-language/).
59. Tianshu Zhang, Xiang Yue, Yifei Li, and Huan Sun. 2023b. Tablellama: Towards open large generalist models for tables. [find](https://scholar.google.com/scholar?q=Tianshu+Zhang%2C+Xiang+Yue%2C+Yifei+Li%2C+and+Huan+Sun%2E+2023b%2E+Tablellama%3A+Towards+open+large+generalist+models+for+tables%2E)
60. Mingyu Zheng, Yang Hao, Wenbin Jiang, Zheng Lin, Yajuan Lyu, QiaoQiao She, and Weiping Wang. 2023. IM-TQA: A Chinese table question answering dataset with implicit and multi-type table structures. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: L… [find](https://scholar.google.com/scholar?q=Mingyu+Zheng%2C+Yang+Hao%2C+Wenbin+Jiang%2C+Zheng+Lin%2C+Yajuan+Lyu%2C+QiaoQiao+She%2C+and+Weiping+Wang%2E+2023%2E+IM%2DTQA%3A+A+Chinese+table+question+answering+dataset+with+implicit+and+multi%2Dtype+table+structures%2E+In+)
61. Xu Zhong, Elaheh ShafieiBavani, and Antonio Jimeno Yepes. 2020. Image-based table recognition: data, model, and evaluation. [find](https://scholar.google.com/scholar?q=Xu+Zhong%2C+Elaheh+ShafieiBavani%2C+and+Antonio+Jimeno+Yepes%2E+2020%2E+Image%2Dbased+table+recognition%3A+data%2C+model%2C+and+evaluation%2E)
62. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. Lima: Less is more for alignment. [find](https://scholar.google.com/scholar?q=Chunting+Zhou%2C+Pengfei+Liu%2C+Puxin+Xu%2C+Srini+Iyer%2C+Jiao+Sun%2C+Yuning+Mao%2C+Xuezhe+Ma%2C+Avia+Efrat%2C+Ping+Yu%2C+Lili+Yu%2C+Susan+Zhang%2C+Gargi+Ghosh%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Omer+Levy%2E+2023%2E+Lima%3A+Less)
63. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. [Minigpt-4: Enhancing vision-language understanding with advanced large language models](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/).
64. Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021. [TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance](https://md.research.elamur.ai/free/en/papers/zhu-2021-tat-qa-question-answering-benchmark/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics…
65. Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023. [Mm-vet: Evaluating large multimodal models for integrated capabilities](https://md.research.elamur.ai/free/en/papers/yu-2023-mm-vet-evaluating-large-multimodal/).
66. Liangyu Zha, Junlin Zhou, Liyao Li, Rui Wang, Qingyi Huang, Saisai Yang, Jing Yuan, Changbao Su, Xiang Li, Aofeng Su, Tao Zhang, Chen Zhou, Kaizhe Shou, Miao Wang, Wufang Zhu, Guoshan Lu, Chao Ye, Yali Ye, Wentao Ye, Yiming Zhang, Xinglong Deng, Jie Xu, Haobo Wang, Gang Chen, and Junbo Zhao. 2023. T… [find](https://scholar.google.com/scholar?q=Liangyu+Zha%2C+Junlin+Zhou%2C+Liyao+Li%2C+Rui+Wang%2C+Qingyi+Huang%2C+Saisai+Yang%2C+Jing+Yuan%2C+Changbao+Su%2C+Xiang+Li%2C+Aofeng+Su%2C+Tao+Zhang%2C+Chen+Zhou%2C+Kaizhe+Shou%2C+Miao+Wang%2C+Wufang+Zhu%2C+Guoshan+Lu%2C+Chao+Ye%2C+Yal)

*Updated 30 October 2026*