---
title: "TABBIE: Pretrained Representations of Tabular Data (2021)"
description: "The authors also show it is weaker on tasks that involve only tables, such as filling in missing cells. 34 references, cited by 63 papers here."
authors: ["Hiroshi Iida", "Dung Thai", "Varun Manjunatha", "Mohit Iyyer"]
year: 2021
venue: "Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies"
source: "https://aclanthology.org/2021.naacl-main.270/"
doi: "10.18653/v1/2021.naacl-main.270"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/iida-2021-tabbie-pretrained-representations-tabular-data/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/iida-2021-tabbie-pretrained-representations-tabular-data/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / TABBIE: Pretrained Representations of Tabular Data

# TABBIE: Pretrained Representations of Tabular Data

**Authors:** Hiroshi Iida, Dung Thai, [Varun Manjunatha](https://md.research.elamur.ai/free/en/authors/varun-manjunatha/) and [Mohit Iyyer](https://md.research.elamur.ai/free/en/authors/mohit-iyyer/) · **Year:** 2021 · **Venue:** [Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2021](https://md.research.elamur.ai/free/en/venues/naacl/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.naacl-main.270/) · [PDF](https://aclanthology.org/2021.naacl-main.270.pdf) · [DOI 10.18653/v1/2021.naacl-main.270](https://doi.org/10.18653/v1/2021.naacl-main.270) · [OpenAlex](https://openalex.org/W3158303960)

**Citations:** 123 citations (OpenAlex, 2026-10-14) · [63](https://md.research.elamur.ai/free/en/papers/iida-2021-tabbie-pretrained-representations-tabular-data/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Earlier pretrained table encoders such as TaBERT and TaPas learn from tables paired with text. This helps with tasks like table question answering, but it is expensive because long linearized inputs force truncation. The authors also show it is weaker on tasks that involve only tables, such as filling in missing cells.

TABBIE is a self-supervised model trained solely on tables. Each cell is first encoded on its own with a frozen BERT model, and learned row and column position embeddings are added. In every layer, a row Transformer and a column Transformer then contextualize the cells separately, and their outputs are averaged. Special tokens prepended to each row and column produce whole-row and whole-column embeddings.

Pretraining borrows the ELECTRA idea but drops the generator. Some cells are replaced using cheap heuristics, and a binary classifier must flag the altered ones. Against a comparable TaBERT setup, TABBIE matches or beats it at a fraction of the compute when suggesting additional columns or rows for a table and when classifying what type each column holds. This makes it useful for text-free table processing such as document extraction and column retrieval.

## Key points
- Architecture details:
  - Cells are initialized with the 768-d [CLS] vector from frozen BERT-base-uncased.
  - The model has 12 layers of paired row/column Transformers whose outputs are averaged per cell.
  - [CLSROW] and [CLSCOL] tokens provide row and column representations.
- Pretraining task (corrupt cell detection):
  - About 15% of cells are corrupted, either by sampling cells from other tables or by swapping cells within the same column.
  - A sigmoid classifier is trained on each cell with binary cross-entropy.
  - Unlike ELECTRA, no separate generator network is used.
- Compute cost:
  - TABBIE trained on 8 V100 GPUs for just over a week, whereas TaBERT used 128 V100 GPUs for six days.
  - The paper reports 1344 GPU-hours and about 149.19 kg CO2 per run, which is more than 10× below TaBERT's estimated cost.
- Qualitative analysis:
  - The MIX variant can detect swapped values that break a numeric trend, where TaBERT and TABBIE FREQ fail.
  - k-means clustering on FinTabNet groups tables into meaningful semantic types.
- Pretrained models and code are released at github.com/SFIG611/tabbie.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Massachusetts Amherst](https://md.research.elamur.ai/free/en/universities/university-of-massachusetts-amherst/) — University, Amherst Center, United States · 67 works · 64,591 citations
- [Adobe Systems](https://md.research.elamur.ai/free/en/companies/adobe-systems/) — Company, San Jose, United States · 14 works · 3,493 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

## Cite

Cite as: Hiroshi Iida, Dung Thai, Varun Manjunatha and Mohit Iyyer. 2021. TABBIE: Pretrained Representations of Tabular Data. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{iida2021tabbie,
  title = {{TABBIE: Pretrained Representations of Tabular Data}},
  author = {Hiroshi Iida, Dung Thai, Varun Manjunatha and Mohit Iyyer},
  booktitle = {Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.270},
  url = {https://doi.org/10.18653/v1/2021.naacl-main.270}
}
```

- [← previous Gradient-based Adversarial Attacks against Text Transformers](https://md.research.elamur.ai/free/en/papers/guo-2021-gradient-based-adversarial-attacks-against/)
- [next → We Need to Consider Disagreement in Evaluation](https://md.research.elamur.ai/free/en/papers/basile-2021-we-need-consider-disagreement-evaluation/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/)

## References

34 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017. [Billion-scale similarity search with gpus](https://md.research.elamur.ai/free/en/papers/johnson-2017-billion-scale-similarity-search-gpus/). arXiv preprint arXiv:1702.08734.
2. Michael J. Cafarella, Alon Halevy, Daisy Zhe Wang, Eugene Wu, and Yang Zhang. 2008. Webtables: Exploring the power of tables on the web. Proc. VLDB Endow., 1(1):538–549. [find](https://scholar.google.com/scholar?q=Michael+J%2E+Cafarella%2C+Alon+Halevy%2C+Daisy+Zhe+Wang%2C+Eugene+Wu%2C+and+Yang+Zhang%2E+2008%2E+Webtables%3A+Exploring+the+power+of+tables+on+the+web%2E+Proc%2E+VLDB+Endow%2E%2C+1%281%29%3A538%E2%80%93549%2E)
3. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. [find](https://scholar.google.com/scholar?q=Diederik+P+Kingma+and+Jimmy+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E)
4. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Minh%2DThang+Luong%2C+Quoc+V%2E+Le%2C+and+Christopher+D%2E+Manning%2E+2020%2E+Electra%3A+Pre%2Dtraining+text+encoders+as+discriminators+rather+than+generators%2E+In+International+Conference+on+Learning+Repre)
5. Anish Das Sarma, Lujun Fang, Nitin Gupta, Alon Halevy, Hongrae Lee, Fei Wu, Reynold Xin, and Cong Yu. 2012. Finding related tables. In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, SIGMOD ’12, page 817–828, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Anish+Das+Sarma%2C+Lujun+Fang%2C+Nitin+Gupta%2C+Alon+Halevy%2C+Hongrae+Lee%2C+Fei+Wu%2C+Reynold+Xin%2C+and+Cong+Yu%2E+2012%2E+Finding+related+tables%2E+In+Proceedings+of+the+2012+ACM+SIGMOD+International+Conference+on+Ma)
6. Li Deng, Shuo Zhang, and Krisztian Balog. 2019. Table2vec: Neural word and entity embeddings for table population and retrieval. In Proceedings of SI-GIR 2019. [find](https://scholar.google.com/scholar?q=Li+Deng%2C+Shuo+Zhang%2C+and+Krisztian+Balog%2E+2019%2E+Table2vec%3A+Neural+word+and+entity+embeddings+for+table+population+and+retrieval%2E+In+Proceedings+of+SI%2DGIR+2019%2E)
7. Xiang Deng, Huan Sun, Alyssa Lees, You Wu, and Cong Yu. 2020. Turl: Table understanding through representation learning. Proc. VLDB Endow., 14(3):307–319. [find](https://scholar.google.com/scholar?q=Xiang+Deng%2C+Huan+Sun%2C+Alyssa+Lees%2C+You+Wu%2C+and+Cong+Yu%2E+2020%2E+Turl%3A+Table+understanding+through+representation+learning%2E+Proc%2E+VLDB+Endow%2E%2C+14%283%29%3A307%E2%80%93319%2E)
8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805.
9. D. Embley, Matthew Hurst, D. Lopresti, and G. Nagy. 2006. Table-processing paradigms: a research survey. International Journal of Document Analysis and Recognition (IJDAR), 8:66–86. [find](https://scholar.google.com/scholar?q=D%2E+Embley%2C+Matthew+Hurst%2C+D%2E+Lopresti%2C+and+G%2E+Nagy%2E+2006%2E+Table%2Dprocessing+paradigms%3A+a+research+survey%2E+International+Journal+of+Document+Analysis+and+Recognition+%28IJDAR%29%2C+8%3A66%E2%80%9386%2E)
10. Jonathan Herzig, P. Nowak, Thomas Müller, Francesco Piccinno, and Julian Martin Eisenschlos. 2020. [Tapas: Weakly supervised table parsing via pre-training](https://md.research.elamur.ai/free/en/papers/herzig-2020-tapas-weakly-supervised-table-parsing/). In ACL.
11. Kevin Zeng Hu, Snehalkumar (Neil) S. Gaikwad, Madelon Hulsebos, Michiel A. Bakker, Emanuel Zgraggen, César A. Hidalgo, Tim Kraska, Guoliang Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. 2019. [Quantifying the carbon emissions of machine learning](https://md.research.elamur.ai/free/en/papers/lacoste-2019-quantifying-carbon-emissions-machine-learning/). arXiv preprint arXiv:1910…
12. Chen Liang, Mohammad Norouzi, Jonathan Berant, Quoc Le, and Ni Lao. 2018. Memory augmented policy optimization for program synthesis and semantic parsing. In Proceedings of the 32nd International Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Chen+Liang%2C+Mohammad+Norouzi%2C+Jonathan+Berant%2C+Quoc+Le%2C+and+Ni+Lao%2E+2018%2E+Memory+augmented+policy+optimization+for+program+synthesis+and+semantic+parsing%2E+In+Proceedings+of+the+32nd+International+Conf)
13. Girija Limaye, Sunita Sarawagi, and Soumen Chakrabarti. 2010. Annotating and searching web tables using entities, types and relationships. Proc. VLDB Endow., 3(1):1338–1347. [find](https://scholar.google.com/scholar?q=Girija+Limaye%2C+Sunita+Sarawagi%2C+and+Soumen+Chakrabarti%2E+2010%2E+Annotating+and+searching+web+tables+using+entities%2C+types+and+relationships%2E+Proc%2E+VLDB+Endow%2E%2C+3%281%29%3A1338%E2%80%931347%2E)
14. Y. Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, M. Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692. [find](https://scholar.google.com/scholar?q=Y%2E+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+M%2E+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+ArXiv%2C+)
15. Jayant Madhavan, Philip A. Bernstein, AnHai Doan, and Alon Halevy. 2005. Corpus-based schema matching. In Proceedings of the 21st International Conference on Data Engineering, ICDE ’05, page 57–68, USA. IEEE Computer Society. [find](https://scholar.google.com/scholar?q=Jayant+Madhavan%2C+Philip+A%2E+Bernstein%2C+AnHai+Doan%2C+and+Alon+Halevy%2E+2005%2E+Corpus%2Dbased+schema+matching%2E+In+Proceedings+of+the+21st+International+Conference+on+Data+Engineering%2C+ICDE+%E2%80%9905%2C+page+57%E2%80%9368%2C+US)
16. Jayant Madhavan, Philip A. Bernstein, and Erhard Rahm. 2001. Generic schema matching with cupid. In Proceedings of the 27th International Conference on Very Large Data Bases, VLDB ’01, page 49–58, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. [find](https://scholar.google.com/scholar?q=Jayant+Madhavan%2C+Philip+A%2E+Bernstein%2C+and+Erhard+Rahm%2E+2001%2E+Generic+schema+matching+with+cupid%2E+In+Proceedings+of+the+27th+International+Conference+on+Very+Large+Data+Bases%2C+VLDB+%E2%80%9901%2C+page+49%E2%80%9358%2C+San)
17. D. Mahajan, Ross B. Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Y. Li, Ashwin Bharambe, and L. V. D. Maaten. 2018. Exploring the limits of weakly supervised pretraining. In ECCV. [find](https://scholar.google.com/scholar?q=D%2E+Mahajan%2C+Ross+B%2E+Girshick%2C+Vignesh+Ramanathan%2C+Kaiming+He%2C+Manohar+Paluri%2C+Y%2E+Li%2C+Ashwin+Bharambe%2C+and+L%2E+V%2E+D%2E+Maaten%2E+2018%2E+Exploring+the+limits+of+weakly+supervised+pretraining%2E+In+ECCV%2E)
18. Kyosuke Nishida, Kugatsu Sadamitsu, Ryuichiro Higashinaka, and Yoshihiro Matsuo. 2017. Understanding the semantic structures of tables with a hybrid deep neural network architecture. In Thirty-First AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Kyosuke+Nishida%2C+Kugatsu+Sadamitsu%2C+Ryuichiro+Higashinaka%2C+and+Yoshihiro+Matsuo%2E+2017%2E+Understanding+the+semantic+structures+of+tables+with+a+hybrid+deep+neural+network+architecture%2E+In+Thirty%2DFirst+A)
19. Dan Zhang, Yoshihiko Suhara, Jinfeng Li, Madelon Hulsebos, Çağatay Demiralp, and Wang-Chiew Tan. 2019. Sato: Contextual semantic type detection in tables. [find](https://scholar.google.com/scholar?q=Dan+Zhang%2C+Yoshihiko+Suhara%2C+Jinfeng+Li%2C+Madelon+Hulsebos%2C+%C3%87ag%CC%86atay+Demiralp%2C+and+Wang%2DChiew+Tan%2E+2019%2E+Sato%3A+Contextual+semantic+type+detection+in+tables%2E)
20. Panupong Pasupat and Percy Liang. 2015. [Compositional semantic parsing on semi-structured tables](https://md.research.elamur.ai/free/en/papers/pasupat-2015-compositional-semantic-parsing-semi-structured/). In Association for Computational Linguistics.
21. Shuo Zhang and Krisztian Balog. 2017. Entitables: Smart assistance for entity-focused tables. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. [find](https://scholar.google.com/scholar?q=Shuo+Zhang+and+Krisztian+Balog%2E+2017%2E+Entitables%3A+Smart+assistance+for+entity%2Dfocused+tables%2E+In+Proceedings+of+the+40th+International+ACM+SIGIR+Conference+on+Research+and+Development+in+Information+R)
22. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. [Deep contextualized word representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/). In Proc. of NAACL.
23. Shuo Zhang and Krisztian Balog. 2020. Web table extraction, retrieval, and augmentation: A survey. ACM Trans. Intell. Syst. Technol. [find](https://scholar.google.com/scholar?q=Shuo+Zhang+and+Krisztian+Balog%2E+2020%2E+Web+table+extraction%2C+retrieval%2C+and+augmentation%3A+A+survey%2E+ACM+Trans%2E+Intell%2E+Syst%2E+Technol%2E)
24. Erhard Rahm and Philip A. Bernstein. 2001. A survey of approaches to automatic schema matching. VLDB J., 10(4):334–350. [find](https://scholar.google.com/scholar?q=Erhard+Rahm+and+Philip+A%2E+Bernstein%2E+2001%2E+A+survey+of+approaches+to+automatic+schema+matching%2E+VLDB+J%2E%2C+10%284%29%3A334%E2%80%93350%2E)
25. Sachin Raja, Ajoy Mondal, and C. V. Jawahar. 2020. Table structure recognition using top-down and bottom-up cues. In Computer Vision – ECCV 2020, pages 70–86, Cham. Springer International Publishing. [find](https://scholar.google.com/scholar?q=Sachin+Raja%2C+Ajoy+Mondal%2C+and+C%2E+V%2E+Jawahar%2E+2020%2E+Table+structure+recognition+using+top%2Ddown+and+bottom%2Dup+cues%2E+In+Computer+Vision+%E2%80%93+ECCV+2020%2C+pages+70%E2%80%9386%2C+Cham%2E+Springer+International+Publishing%2E)
26. Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007. Yago: A core of semantic knowledge. In Proceedings of the 16th International Conference on World Wide Web. [find](https://scholar.google.com/scholar?q=Fabian+M%2E+Suchanek%2C+Gjergji+Kasneci%2C+and+Gerhard+Weikum%2E+2007%2E+Yago%3A+A+core+of+semantic+knowledge%2E+In+Proceedings+of+the+16th+International+Conference+on+World+Wide+Web%2E)
27. Shuo Zhang, Edgar Meij, Krisztian Balog, and Ridho Reinanda. 2020. Novel entity discovery from web tables. In Proceedings of The Web Conference 2020, WWW ’20, page 1298–1308, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Shuo+Zhang%2C+Edgar+Meij%2C+Krisztian+Balog%2C+and+Ridho+Reinanda%2E+2020%2E+Novel+entity+discovery+from+web+tables%2E+In+Proceedings+of+The+Web+Conference+2020%2C+WWW+%E2%80%9920%2C+page+1298%E2%80%931308%2C+New+York%2C+NY%2C+USA%2E+Associ)
28. Xinyi Zheng, Douglas Burdick, Lucian Popa, Xu Zhong, and Nancy Xin Ru Wang. 2021. Global table extractor (gte): A framework for joint table identification and cell structure recognition using visual context. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV),… [find](https://scholar.google.com/scholar?q=Xinyi+Zheng%2C+Douglas+Burdick%2C+Lucian+Popa%2C+Xu+Zhong%2C+and+Nancy+Xin+Ru+Wang%2E+2021%2E+Global+table+extractor+%28gte%29%3A+A+framework+for+joint+table+identification+and+cell+structure+recognition+using+visual+c)
29. C. Tensmeyer, V. I. Morariu, B. Price, S. Cohen, and T. Martinez. 2019. Deep splitting and merging for table structure decomposition. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 114–121. [find](https://scholar.google.com/scholar?q=C%2E+Tensmeyer%2C+V%2E+I%2E+Morariu%2C+B%2E+Price%2C+S%2E+Cohen%2C+and+T%2E+Martinez%2E+2019%2E+Deep+splitting+and+merging+for+table+structure+decomposition%2E+In+2019+International+Conference+on+Document+Analysis+and+Recognit)
30. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ¥L ukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Proce…
31. Petros Venetis, Alon Halevy, Jayant Madhavan, Marius Paşca, Warren Shen, Fei Wu, Gengxin Miao, and Chung Wu. 2011. Recovering semantics of tables on the web. Proc. VLDB Endow., 4(9):528–538. [find](https://scholar.google.com/scholar?q=Petros+Venetis%2C+Alon+Halevy%2C+Jayant+Madhavan%2C+Marius+Pas%CC%A7ca%2C+Warren+Shen%2C+Fei+Wu%2C+Gengxin+Miao%2C+and+Chung+Wu%2E+2011%2E+Recovering+semantics+of+tables+on+the+web%2E+Proc%2E+VLDB+Endow%2E%2C+4%289%29%3A528%E2%80%93538%2E)
32. Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019. [Do NLP models know numbers? probing numeracy in embeddings](https://md.research.elamur.ai/free/en/papers/wallace-2019-do-nlp-models-know-numbers/). In Empirical Methods in Natural Language Processing.
33. Zhiruo Wang, Haoyu Dong, Ran Jia, Jia Li, Zhiyi Fu, Shi Han, and Dongmei Zhang. 2020. Structure-aware pre-training for table understanding with tree-based transformers. ArXiv, abs/2010.12537. — [arXiv:2010.12537](https://arxiv.org/abs/2010.12537) [find](https://scholar.google.com/scholar?q=Zhiruo+Wang%2C+Haoyu+Dong%2C+Ran+Jia%2C+Jia+Li%2C+Zhiyi+Fu%2C+Shi+Han%2C+and+Dongmei+Zhang%2E+2020%2E+Structure%2Daware+pre%2Dtraining+for+table+understanding+with+tree%2Dbased+transformers%2E+ArXiv%2C+abs%2F2010%2E12537%2E+%E2%80%94+%5BarXiv)
34. Pengcheng Yin, Graham Neubig, Wen tau Yih, and Sebastian Riedel. 2020. [TaBERT: Pretraining for joint understanding of textual and tabular data](https://md.research.elamur.ai/free/en/papers/yin-2020-tabert-pretraining-joint-understanding-textual/). In Annual Conference of the Association for Computational Linguistics (ACL).

*Updated 24 October 2026*