---
title: "TaBERT: Pretraining for Joint Understanding of — Yin 2020"
description: "Pretrained language models such as BERT learn only from running text."
authors: ["Pengcheng Yin", "Graham Neubig", "Wen-tau Yih", "Sebastian Riedel"]
year: 2020
venue: "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/2020.acl-main.745/"
doi: "10.18653/v1/2020.acl-main.745"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yin-2020-tabert-pretraining-joint-understanding-textual/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yin-2020-tabert-pretraining-joint-understanding-textual/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/) / TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data

# TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data

**Authors:** [Pengcheng Yin](https://md.research.elamur.ai/free/en/authors/pengcheng-yin/), [Graham Neubig](https://md.research.elamur.ai/free/en/authors/graham-neubig/), [Wen-tau Yih](https://md.research.elamur.ai/free/en/authors/wen-tau-yih/), [Sebastian Riedel](https://md.research.elamur.ai/free/en/authors/sebastian-riedel/) · **Year:** 2020 · **Venue:** Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.acl-main.745/) · [PDF](https://aclanthology.org/2020.acl-main.745.pdf) · [DOI 10.18653/v1/2020.acl-main.745](https://doi.org/10.18653/v1/2020.acl-main.745) · [OpenAlex](https://openalex.org/W3035231859)

**Citations:** 475 citations (OpenAlex, 2026-10-14) · [215](https://md.research.elamur.ai/free/en/papers/yin-2020-tabert-pretraining-joint-understanding-textual/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
Pretrained language models such as BERT learn only from running text. They are therefore poorly suited to semantic parsing over databases, where a system must connect a natural-language question to the columns and cell values of a table. TaBERT, built on BERT, addresses this by learning joint representations of utterances and (semi-)structured tables. It was pretrained on a parallel corpus of about 26 million Web tables paired with surrounding English text.

To handle large tables cheaply, TaBERT encodes only a "content snapshot," a few rows chosen by n-gram overlap with the question. Each row is flattened and encoded together with the utterance, and vertical self-attention layers then share information across rows to produce token and column vectors. The model serves as a drop-in encoder for existing neural parsers. With it, parsers set a new best result on the weakly supervised WikiTableQuestions benchmark and stay competitive on the Spider text-to-SQL dataset. This shows that general-purpose pretraining on tables can replace ad-hoc, domain-specific schemes for encoding tables.

## Key points
- **Pretraining data:** 26 million tables with their English contexts, taken from the Web. Code is released at github.com/facebookresearch/TaBERT.
- **Row linearization:** each row is written as a sequence of column name | data type | cell value, separated by [SEP]. Adding type information and snapshot contents improved results over encoding column names alone (the Hwang et al., 2019 style).
- **Content snapshot:**
  - With K > 1, the top-K rows by n-gram overlap are selected.
  - With K = 1, a synthetic row is built from the best-matching cell in each column.
  - K = 3 handled questions that need several rows, where K = 1 failed.
- **Pretraining objectives:** masked column prediction (MCP) plus cell value recovery (CVR). On dev sets, TaBERT Base (K = 3) scored 62.6 → 63.3 on Spider when CVR was added and 51.6 on WikiTableQuestions either way.
- **Evaluation:** tested in both supervised (Spider) and weakly supervised (WikiTableQuestions) settings, where it outperformed BERT-based counterparts. Planned future work includes table retrieval, table-to-text generation, and cross-lingual use.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/)

## Institutions

- [Meta](https://md.research.elamur.ai/free/en/companies/meta/) — Company, Menlo Park, United States · 137 works · 196 530 citations
- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 278 works · 138 810 citations

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yin2020tabert,
  title = {{TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data}},
  author = {Yin, Pengcheng and Neubig, Graham and Yih, Wen-tau and Riedel, Sebastian},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.745},
  url = {https://doi.org/10.18653/v1/2020.acl-main.745}
}
```

- [← previous SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://md.research.elamur.ai/free/en/papers/cohan-2020-specter-document-level-representation-learning/)
- [next → CORD-19: The COVID-19 Open Research Dataset](https://md.research.elamur.ai/free/en/papers/wang-2020-cord-19-covid-19-open/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/)

## References

52 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rishabh Agarwal, Chen Liang, Dale Schuurmans, and Mohammad Norouzi. 2019. Learning to generalize from sparse and underspecified rewards. In ICML. [find](https://scholar.google.com/scholar?q=Rishabh+Agarwal%2C+Chen+Liang%2C+Dale+Schuurmans%2C+and+Mohammad+Norouzi%2E+2019%2E+Learning+to+generalize+from+sparse+and+underspecified+rewards%2E+In+ICML%2E)
2. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Jonathan+Berant%2C+Andrew+Chou%2C+Roy+Frostig%2C+and+Percy+Liang%2E+2013%2E+Semantic+parsing+on+freebase+from+question%2Danswer+pairs%2E+In+Proceedings+of+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/berant-2013-semantic-parsing-freebase-question-answer/)
3. Jonathan Berant and Percy Liang. 2014. Semantic parsing via paraphrasing. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Jonathan+Berant+and+Percy+Liang%2E+2014%2E+Semantic+parsing+via+paraphrasing%2E+In+Proceedings+of+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/berant-2014-semantic-parsing-paraphrasing/)
4. Ben Bogin, Matt Gardner, and Jonathan Berant. 2019a. Global reasoning over database structures for text-to-sql parsing. ArXiv, abs/1908.11214. [find](https://scholar.google.com/scholar?q=Ben+Bogin%2C+Matt+Gardner%2C+and+Jonathan+Berant%2E+2019a%2E+Global+reasoning+over+database+structures+for+text%2Dto%2Dsql+parsing%2E+ArXiv%2C+abs%2F1908%2E11214%2E)
5. Ben Bogin, Matthew Gardner, and Jonathan Berant. 2019b. Representing schema structure with graph neural networks for text-to-sql parsing. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Ben+Bogin%2C+Matthew+Gardner%2C+and+Jonathan+Berant%2E+2019b%2E+Representing+schema+structure+with+graph+neural+networks+for+text%2Dto%2Dsql+parsing%2E+In+Proceedings+of+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/bogin-2019-representing-schema-structure-graph-neural/)
6. Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2019. TabFact: A large-scale dataset for table-based fact verification. ArXiv, abs/1909.02164. [find](https://scholar.google.com/scholar?q=Wenhu+Chen%2C+Hongmin+Wang%2C+Jianshu+Chen%2C+Yunkai+Zhang%2C+Hong+Wang%2C+Shiyang+Li%2C+Xiyou+Zhou%2C+and+William+Yang+Wang%2E+2019%2E+TabFact%3A+A+large%2Dscale+dataset+for+table%2Dbased+fact+verification%2E+ArXiv%2C+abs%2F1909%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2019-tabfact-large-scale-dataset-table/)
7. Donghyun Choi, Myeong Cheol Shin, EungGyun Kim, and Dong Ryeol Shin. 2020. Ryansql: Recursively applying sketch-based slot fillings for complex text-to-sql in cross-domain databases. ArXiv, abs/2004.03125. [find](https://scholar.google.com/scholar?q=Donghyun+Choi%2C+Myeong+Cheol+Shin%2C+EungGyun+Kim%2C+and+Dong+Ryeol+Shin%2E+2020%2E+Ryansql%3A+Recursively+applying+sketch%2Dbased+slot+fillings+for+complex+text%2Dto%2Dsql+in+cross%2Ddomain+databases%2E+ArXiv%2C+abs%2F2004%2E0)
8. Pradeep Dasigi, Matt Gardner, Shikhar Murty, Luke S. Zettlemoyer, and Eduard H. Hovy. 2019. Iterative search for weakly supervised semantic parsing. In Proceedings of NAACL-HLT. [find](https://scholar.google.com/scholar?q=Pradeep+Dasigi%2C+Matt+Gardner%2C+Shikhar+Murty%2C+Luke+S%2E+Zettlemoyer%2C+and+Eduard+H%2E+Hovy%2E+2019%2E+Iterative+search+for+weakly+supervised+semantic+parsing%2E+In+Proceedings+of+NAACL%2DHLT%2E)
9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+NAACL%2DHLT%2E) [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
10. Li Dong and Mirella Lapata. 2018. coarse-to-fine decoding for neural semantic parsing. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Li+Dong+and+Mirella+Lapata%2E+2018%2E+coarse%2Dto%2Dfine+decoding+for+neural+semantic+parsing%2E+In+Proceedings+of+ACL%2E)
11. Yoav Goldberg. 2019. Assessing bert’s syntactic abilities. ArXiv, abs/1901.05287. [find](https://scholar.google.com/scholar?q=Yoav+Goldberg%2E+2019%2E+Assessing+bert%E2%80%99s+syntactic+abilities%2E+ArXiv%2C+abs%2F1901%2E05287%2E) [our card](https://md.research.elamur.ai/free/en/papers/goldberg-2019-assessing-bert-s-syntactic-abilities/)
12. Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. 2019. Towards complex text-to-sql in cross-domain database with intermediate representation. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Jiaqi+Guo%2C+Zecheng+Zhan%2C+Yan+Gao%2C+Yan+Xiao%2C+Jian%2DGuang+Lou%2C+Ting+Liu%2C+and+Dongmei+Zhang%2E+2019%2E+Towards+complex+text%2Dto%2Dsql+in+cross%2Ddomain+database+with+intermediate+representation%2E+In+Proceedings+of+)
13. Pengcheng He, Yi Mao, Kaushik Chakrabarti, and Weizhu Chen. 2019. X-sql: reinforce schema representation with context. ArXiv, abs/1908.08113. — [arXiv:1908.08113](https://arxiv.org/abs/1908.08113) [find](https://scholar.google.com/scholar?q=Pengcheng+He%2C+Yi+Mao%2C+Kaushik+Chakrabarti%2C+and+Weizhu+Chen%2E+2019%2E+X%2Dsql%3A+reinforce+schema+representation+with+context%2E+ArXiv%2C+abs%2F1908%2E08113%2E+%E2%80%94+%5BarXiv%3A1908%2E08113%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1908%2E08113%29)
14. Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). ArXiv, abs/1606.08415. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks+and+Kevin+Gimpel%2E+2016%2E+Gaussian+error+linear+units+%28gelus%29%2E+ArXiv%2C+abs%2F1606%2E08415%2E)
15. Wonseok Hwang, Jinyeung Yim, Seunghyun Park, and Minjoon Seo. 2019. A comprehensive exploration on wikisql with table-aware word contextualization. ArXiv, abs/1902.01069. [find](https://scholar.google.com/scholar?q=Wonseok+Hwang%2C+Jinyeung+Yim%2C+Seunghyun+Park%2C+and+Minjoon+Seo%2E+2019%2E+A+comprehensive+exploration+on+wikisql+with+table%2Daware+word+contextualization%2E+ArXiv%2C+abs%2F1902%2E01069%2E)
16. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke S. Zettlemoyer, and Omer Levy. 2019. Spanbert: Improving pre-training by representing and predicting spans. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Mandar+Joshi%2C+Danqi+Chen%2C+Yinhan+Liu%2C+Daniel+S%2E+Weld%2C+Luke+S%2E+Zettlemoyer%2C+and+Omer+Levy%2E+2019%2E+Spanbert%3A+Improving+pre%2Dtraining+by+representing+and+predicting+spans%2E+In+Proceedings+of+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2019-spanbert-improving-pre-training-representing/)
17. Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gardner. 2017. Neural semantic parsing with type constraints for semi-structured tables. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Jayant+Krishnamurthy%2C+Pradeep+Dasigi%2C+and+Matt+Gardner%2E+2017%2E+Neural+semantic+parsing+with+type+constraints+for+semi%2Dstructured+tables%2E+In+Proceedings+of+EMNLP%2E)
18. Oliver Lehmberg, Dominique Ritze, Robert Meusel, and Christian Bizer. 2016. A large public corpus of web tables containing time and context metadata. In Proceedings of WWW. [find](https://scholar.google.com/scholar?q=Oliver+Lehmberg%2C+Dominique+Ritze%2C+Robert+Meusel%2C+and+Christian+Bizer%2E+2016%2E+A+large+public+corpus+of+web+tables+containing+time+and+context+metadata%2E+In+Proceedings+of+WWW%2E)
19. Chen Liang, Mohammad Norouzi, Jonathan Berant, Quoc V Le, and Ni Lao. 2018. Memory augmented policy optimization for program synthesis and semantic parsing. In Proceedings of NIPS. [find](https://scholar.google.com/scholar?q=Chen+Liang%2C+Mohammad+Norouzi%2C+Jonathan+Berant%2C+Quoc+V+Le%2C+and+Ni+Lao%2E+2018%2E+Memory+augmented+policy+optimization+for+program+synthesis+and+semantic+parsing%2E+In+Proceedings+of+NIPS%2E)
20. Jindrich Libovický and Jindrich Helcl. 2017. Attention strategies for multi-source sequence-to-sequence learning. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Jindrich+Libovicky%CC%81+and+Jindrich+Helcl%2E+2017%2E+Attention+strategies+for+multi%2Dsource+sequence%2Dto%2Dsequence+learning%2E+In+Proceedings+of+ACL%2E)
21. Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. 2019a. K-bert: Enabling language representation with knowledge graph. ArXiv, abs/1909.07606. [find](https://scholar.google.com/scholar?q=Weijie+Liu%2C+Peng+Zhou%2C+Zhe+Zhao%2C+Zhiruo+Wang%2C+Qi+Ju%2C+Haotang+Deng%2C+and+Ping+Wang%2E+2019a%2E+K%2Dbert%3A+Enabling+language+representation+with+knowledge+graph%2E+ArXiv%2C+abs%2F1909%2E07606%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2019-k-bert-enabling-language-representation/)
22. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke S. Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+S%2E+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019b%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approac)
23. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In Proceedings of NIPS. [find](https://scholar.google.com/scholar?q=Bryan+McCann%2C+James+Bradbury%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+2017%2E+Learned+in+translation%3A+Contextualized+word+vectors%2E+In+Proceedings+of+NIPS%2E) [our card](https://md.research.elamur.ai/free/en/papers/mccann-2017-learned-translation-contextualized-word-vectors/)
24. Oren Melamud, Jacob Goldberger, and Ido Dagan. 2016. context2vec: Learning generic context embedding with bidirectional LSTM. In Proceedings of CoNLL. [find](https://scholar.google.com/scholar?q=Oren+Melamud%2C+Jacob+Goldberger%2C+and+Ido+Dagan%2E+2016%2E+context2vec%3A+Learning+generic+context+embedding+with+bidirectional+LSTM%2E+In+Proceedings+of+CoNLL%2E) [our card](https://md.research.elamur.ai/free/en/papers/melamud-2016-context2vec-learning-generic-context-embedding/)
25. Arvind Neelakantan, Quoc V. Le, and Ilya Sutskever. 2016. Neural programmer: Inducing latent programs with gradient descent. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Arvind+Neelakantan%2C+Quoc+V%2E+Le%2C+and+Ilya+Sutskever%2E+2016%2E+Neural+programmer%3A+Inducing+latent+programs+with+gradient+descent%2E+In+Proceedings+of+ICLR%2E)
26. Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Panupong+Pasupat+and+Percy+Liang%2E+2015%2E+Compositional+semantic+parsing+on+semi%2Dstructured+tables%2E+In+Proceedings+of+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/pasupat-2015-compositional-semantic-parsing-semi-structured/)
27. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke S. Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL. [find](https://scholar.google.com/scholar?q=Matthew+E%2E+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+S%2E+Zettlemoyer%2E+2018%2E+Deep+contextualized+word+representations%2E+In+Proceedings+of+NAACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/)
28. Matthew E. Peters, Mark Neumann, IV RobertLLo-gan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge enhanced contextual word representations. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Matthew+E%2E+Peters%2C+Mark+Neumann%2C+IV+RobertLLo%2Dgan%2C+Roy+Schwartz%2C+Vidur+Joshi%2C+Sameer+Singh%2C+and+Noah+A%2E+Smith%2E+2019%2E+Knowledge+enhanced+contextual+word+representations%2E+In+Proceedings+of+EMNLP%2E)
29. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+2016%2E+Squad%3A+100%2C+000%2B+questions+for+machine+comprehension+of+text%2E+In+Proceedings+of+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
30. Tianze Shi, Kedar Tatwawadi, Kaushik Chakrabarti, Yi Mao, Oleksandr Polozov, and Weizhu Chen. 2018. Incsql: Training incremental text-to-sql parsers with non-deterministic oracles. ArXiv, abs/1809.05054. — [arXiv:1809.05054](https://arxiv.org/abs/1809.05054) [find](https://scholar.google.com/scholar?q=Tianze+Shi%2C+Kedar+Tatwawadi%2C+Kaushik+Chakrabarti%2C+Yi+Mao%2C+Oleksandr+Polozov%2C+and+Weizhu+Chen%2E+2018%2E+Incsql%3A+Training+incremental+text%2Dto%2Dsql+parsers+with+non%2Ddeterministic+oracles%2E+ArXiv%2C+abs%2F1809%2E050)
31. Huan Sun, Hao Ma, Xiaodong He, Wen tau Yih, Yu Su, and Xifeng Yan. 2016. Table cell search for question answering. In Proceedings of WWW. [find](https://scholar.google.com/scholar?q=Huan+Sun%2C+Hao+Ma%2C+Xiaodong+He%2C+Wen+tau+Yih%2C+Yu+Su%2C+and+Xifeng+Yan%2E+2016%2E+Table+cell+search+for+question+answering%2E+In+Proceedings+of+WWW%2E)
32. Yibo Sun, Duyu Tang, Nan Duan, Jianshu Ji, Guihong Cao, Xiaocheng Feng, Bing Qin, Ting Liu, and Ming Zhou. 2018. Semantic parsing with syntax-and table-aware SQL generation. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Yibo+Sun%2C+Duyu+Tang%2C+Nan+Duan%2C+Jianshu+Ji%2C+Guihong+Cao%2C+Xiaocheng+Feng%2C+Bing+Qin%2C+Ting+Liu%2C+and+Ming+Zhou%2E+2018%2E+Semantic+parsing+with+syntax%2Dand+table%2Daware+SQL+generation%2E+In+Proceedings+of+EMNLP%2E)
33. Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. Ernie: Enhanced representation through knowledge integration. ArXiv, abs/1904.09223. [find](https://scholar.google.com/scholar?q=Yu+Sun%2C+Shuohuan+Wang%2C+Yukun+Li%2C+Shikun+Feng%2C+Xuyi+Chen%2C+Han+Zhang%2C+Xin+Tian%2C+Danxiang+Zhu%2C+Hao+Tian%2C+and+Hua+Wu%2E+2019%2E+Ernie%3A+Enhanced+representation+through+knowledge+integration%2E+ArXiv%2C+abs%2F1904%2E09) [our card](https://md.research.elamur.ai/free/en/papers/sun-2019-ernie-enhanced-representation-through-knowledge/)
34. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NIPS. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N%2E+Gomez%2C+Lukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+Proceedings+of+NIPS%2E) [our card](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/)
35. Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Margot Richardson. 2019a. Rat-sql: Relation-aware schema encoding and linking for text-to-sql parsers. ArXiv, abs/1911.04942. [find](https://scholar.google.com/scholar?q=Bailin+Wang%2C+Richard+Shin%2C+Xiaodong+Liu%2C+Oleksandr+Polozov%2C+and+Margot+Richardson%2E+2019a%2E+Rat%2Dsql%3A+Relation%2Daware+schema+encoding+and+linking+for+text%2Dto%2Dsql+parsers%2E+ArXiv%2C+abs%2F1911%2E04942%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2020-rat-sql-relation-aware-schema/)
36. Bailin Wang, Ivan Titov, and Mirella Lapata. 2019b. Learning semantic parsers from denotations with latent structured alignments and abstract programs. In EMNLP/IJCNLP. [find](https://scholar.google.com/scholar?q=Bailin+Wang%2C+Ivan+Titov%2C+and+Mirella+Lapata%2E+2019b%2E+Learning+semantic+parsers+from+denotations+with+latent+structured+alignments+and+abstract+programs%2E+In+EMNLP%2FIJCNLP%2E)
37. Chenglong Wang, Kedar Tatwawadi, Marc Brockschmidt, Po-Sen Huang, Yi Xin Mao, Oleksandr Polozov, and Rishabh Singh. 2018. Robust text-to-sql generation with execution-guided decoding. ArXiv, abs/1807.03100. [find](https://scholar.google.com/scholar?q=Chenglong+Wang%2C+Kedar+Tatwawadi%2C+Marc+Brockschmidt%2C+Po%2DSen+Huang%2C+Yi+Xin+Mao%2C+Oleksandr+Polozov%2C+and+Rishabh+Singh%2E+2018%2E+Robust+text%2Dto%2Dsql+generation+with+execution%2Dguided+decoding%2E+ArXiv%2C+abs%2F1807%2E)
38. Daniel C. Wang, Andrew W. Appel, Jeffrey L. Korn, and Christopher S. Serra. 1997. The Zephyr abstract syntax description language. In Proceedings of DSL. [find](https://scholar.google.com/scholar?q=Daniel+C%2E+Wang%2C+Andrew+W%2E+Appel%2C+Jeffrey+L%2E+Korn%2C+and+Christopher+S%2E+Serra%2E+1997%2E+The+Zephyr+abstract+syntax+description+language%2E+In+Proceedings+of+DSL%2E)
39. Zhuosheng Zhang, Yu-Wei Wu, Hai Zhao, Zuchao Li, Shuailiang Zhang, Xi Zhou, and Xiaodong Zhou. 2019c. Semantics-aware bert for language understanding. ArXiv, abs/1909.02209. — [arXiv:1909.02209](https://arxiv.org/abs/1909.02209) [find](https://scholar.google.com/scholar?q=Zhuosheng+Zhang%2C+Yu%2DWei+Wu%2C+Hai+Zhao%2C+Zuchao+Li%2C+Shuailiang+Zhang%2C+Xi+Zhou%2C+and+Xiaodong+Zhou%2E+2019c%2E+Semantics%2Daware+bert+for+language+understanding%2E+ArXiv%2C+abs%2F1909%2E02209%2E+%E2%80%94+%5BarXiv%3A1909%2E02209%5D%28https)
40. Yushi Wang, Jonathan Berant, and Percy Liang. 2015. Building a semantic parser overnight. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Yushi+Wang%2C+Jonathan+Berant%2C+and+Percy+Liang%2E+2015%2E+Building+a+semantic+parser+overnight%2E+In+Proceedings+of+ACL%2E)
41. Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2SQL: Generating structured queries from natural language using reinforcement learning. arXiv, abs/1709.00103. [find](https://scholar.google.com/scholar?q=Victor+Zhong%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+2017%2E+Seq2SQL%3A+Generating+structured+queries+from+natural+language+using+reinforcement+learning%2E+arXiv%2C+abs%2F1709%2E00103%2E)
42. Xiaojun Xu, Chang Liu, and Dawn Song. 2017. SQL-Net: Generating structured queries from natural language without reinforcement learning. arXiv, abs/1711.04436. [find](https://scholar.google.com/scholar?q=Xiaojun+Xu%2C+Chang+Liu%2C+and+Dawn+Song%2E+2017%2E+SQL%2DNet%3A+Generating+structured+queries+from+natural+language+without+reinforcement+learning%2E+arXiv%2C+abs%2F1711%2E04436%2E)
43. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Proceedings of NIPS. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Zihang+Dai%2C+Yiming+Yang%2C+Jaime+G%2E+Carbonell%2C+Ruslan+Salakhutdinov%2C+and+Quoc+V%2E+Le%2E+2019%2E+Xlnet%3A+Generalized+autoregressive+pretraining+for+language+understanding%2E+In+Proceedings+of+NIPS%2E) [our card](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/)
44. Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. 2015. Semantic parsing via staged query graph generation: Question answering with knowledge base. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Wen%2Dtau+Yih%2C+Ming%2DWei+Chang%2C+Xiaodong+He%2C+and+Jianfeng+Gao%2E+2015%2E+Semantic+parsing+via+staged+query+graph+generation%3A+Question+answering+with+knowledge+base%2E+In+Proceedings+of+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/yih-2015-semantic-parsing-staged-query-graph/)
45. Pengcheng Yin and Graham Neubig. 2018. TRANX: A transition-based neural abstract syntax parser for semantic parsing and code generation. In Proceedings of EMNLP Demonstration Track. [find](https://scholar.google.com/scholar?q=Pengcheng+Yin+and+Graham+Neubig%2E+2018%2E+TRANX%3A+A+transition%2Dbased+neural+abstract+syntax+parser+for+semantic+parsing+and+code+generation%2E+In+Proceedings+of+EMNLP+Demonstration+Track%2E)
46. Tao Yu, Zifan Li, Zilin Zhang, Rui Zhang, and Dragomir R. Radev. 2018a. TypeSQL: Knowledge-based type-aware neural text-to-sql generation. In Proceedings of NAACL-HLT. [find](https://scholar.google.com/scholar?q=Tao+Yu%2C+Zifan+Li%2C+Zilin+Zhang%2C+Rui+Zhang%2C+and+Dragomir+R%2E+Radev%2E+2018a%2E+TypeSQL%3A+Knowledge%2Dbased+type%2Daware+neural+text%2Dto%2Dsql+generation%2E+In+Proceedings+of+NAACL%2DHLT%2E) [our card](https://md.research.elamur.ai/free/en/papers/yu-2018-typesql-knowledge-based-type-aware/)
47. Tao Yu, Michihiro Yasunaga, Kai Yang, Rui Zhang, Dongxu Wang, Zifan Li, and Dragomir R. Radev. 2018b. Syntaxsqlnet: Syntax tree networks for complex and cross-domain text-to-sql task. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Tao+Yu%2C+Michihiro+Yasunaga%2C+Kai+Yang%2C+Rui+Zhang%2C+Dongxu+Wang%2C+Zifan+Li%2C+and+Dragomir+R%2E+Radev%2E+2018b%2E+Syntaxsqlnet%3A+Syntax+tree+networks+for+complex+and+cross%2Ddomain+text%2Dto%2Dsql+task%2E+In+Proceedings+o)
48. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir R. Radev. 2018c. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of EMNLP… [find](https://scholar.google.com/scholar?q=Tao+Yu%2C+Rui+Zhang%2C+Kai+Yang%2C+Michihiro+Yasunaga%2C+Dongxu+Wang%2C+Zifan+Li%2C+James+Ma%2C+Irene+Li%2C+Qingning+Yao%2C+Shanelle+Roman%2C+Zilin+Zhang%2C+and+Dragomir+R%2E+Radev%2E+2018c%2E+Spider%3A+A+large%2Dscale+human%2Dlabeled) [our card](https://md.research.elamur.ai/free/en/papers/yu-2018-spider-large-scale-human-labeled/)
49. John M. Zelle and Raymond J. Mooney. 1996. Learning to parse database queries using inductive logic programming. In Proceedings of AAAI. [find](https://scholar.google.com/scholar?q=John+M%2E+Zelle+and+Raymond+J%2E+Mooney%2E+1996%2E+Learning+to+parse+database+queries+using+inductive+logic+programming%2E+In+Proceedings+of+AAAI%2E) [our card](https://md.research.elamur.ai/free/en/papers/zelle-1996-learning-parse-database-queries-using/)
50. Rui Zhang, Tao Yu, He Yang Er, Sungrok Shim, Eric Xue, Xi Victoria Lin, Tianze Shi, Caiming Xiong, Richard Socher, and Dragomir R. Radev. 2019a. Editing-based sql query generation for cross-domain context-dependent questions. ArXiv, abs/1909.00786. [find](https://scholar.google.com/scholar?q=Rui+Zhang%2C+Tao+Yu%2C+He+Yang+Er%2C+Sungrok+Shim%2C+Eric+Xue%2C+Xi+Victoria+Lin%2C+Tianze+Shi%2C+Caiming+Xiong%2C+Richard+Socher%2C+and+Dragomir+R%2E+Radev%2E+2019a%2E+Editing%2Dbased+sql+query+generation+for+cross%2Ddomain+con)
51. Yuchen Zhang, Panupong Pasupat, and Percy Liang. 2017. Macro grammars and holistic triggering for efficient semantic parsing. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Yuchen+Zhang%2C+Panupong+Pasupat%2C+and+Percy+Liang%2E+2017%2E+Macro+grammars+and+holistic+triggering+for+efficient+semantic+parsing%2E+In+Proceedings+of+EMNLP%2E)
52. Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019b. Ernie: Enhanced language representation with informative entities. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Zhengyan+Zhang%2C+Xu+Han%2C+Zhiyuan+Liu%2C+Xin+Jiang%2C+Maosong+Sun%2C+and+Qun+Liu%2E+2019b%2E+Ernie%3A+Enhanced+language+representation+with+informative+entities%2E+In+Proceedings+of+ACL%2E)