---
title: "Exploring Underexplored Limitations of Cross-Domain"
description: "The authors build Spider-DK, a human-curated evaluation set derived from the Spider development set. 21 references, cited by 46 papers here."
authors: ["Yujian Gan", "Xinyun Chen", "Matthew Purver"]
year: 2021
venue: "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2021.emnlp-main.702/"
doi: "10.18653/v1/2021.emnlp-main.702"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/gan-2021-exploring-underexplored-limitations-cross-domain/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gan-2021-exploring-underexplored-limitations-cross-domain/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalization

# Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalization

**Authors:** [Yujian Gan](https://md.research.elamur.ai/free/en/authors/yujian-gan/), [Xinyun Chen](https://md.research.elamur.ai/free/en/authors/xinyun-chen/) and [Matthew Purver](https://md.research.elamur.ai/free/en/authors/matthew-purver/) · **Year:** 2021 · **Venue:** [Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2021](https://md.research.elamur.ai/free/en/venues/emnlp/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.emnlp-main.702/) · [PDF](https://aclanthology.org/2021.emnlp-main.702.pdf) · [DOI 10.18653/v1/2021.emnlp-main.702](https://doi.org/10.18653/v1/2021.emnlp-main.702) · [OpenAlex](https://openalex.org/W3199077625)

**Citations:** 77 citations (OpenAlex, 2026-10-14) · [46](https://md.research.elamur.ai/free/en/papers/gan-2021-exploring-underexplored-limitations-cross-domain/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Cross-domain text-to-SQL systems score highly on Spider, but it is unclear whether they truly learn the domain knowledge implicit in their training data. The authors build Spider-DK, a human-curated evaluation set derived from the Spider development set. It contains 535 NL-SQL pairs. Of these, 265 were rewritten so that answering requires one of five kinds of domain knowledge already present in the Spider training data. Unlike earlier robustness studies, the question and SQL distributions stay close to the training data, so any failure points specifically to weak knowledge understanding.

Five open-source parsers trained on Spider were tested with exact match accuracy. These were GNN, IRNet, RAT-SQL, RAT-SQL + BERT and RAT-SQL + GAP. All lost roughly 20–30 points compared with the matching Spider samples, even though they almost perfectly solve related training examples. This shows that current models fit their training sets without really absorbing domain knowledge, which is a key obstacle to cross-domain generalization.

## Key points
- Spider-DK has 535 pairs: 270 are unchanged Spider samples and 265 are modified. The five knowledge types are:
  - T1: columns implied by omission (28.7%)
  - T2: simple inference (24.5%)
  - T3: cell-value synonyms (27.5%)
  - T4: a non-cell-value word triggering a condition (8.3%)
  - T5: wording that clashes with other domains (12.5%)
- Overall accuracy ranges from 26.0% (GNN) to 44.1% (RAT-SQL + GAP) on Spider-DK. The same models score 46.2–73.3% on the 535 matching Spider dev samples and up to 98.4% on 125 training samples.
- RAT-SQL + BERT leads on Spider dev (73.3%), but RAT-SQL + GAP is better on Spider-DK. On the 265 knowledge examples alone, the best model reaches only 27.1%.
- IRNet does well on T3 (42.4%) thanks to ConceptNet, but this does not carry over to the other types.
- The authors identify three causes of errors:
  - Some knowledge appears too rarely in training.
  - Models generalize poorly to similar cases, such as "United States" vs "USA".
  - One word drives both schema and SQL-structure generation, producing errors like max(max_speed).
- The dataset is available at github.com/ygan/Spider-DK.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of California, Berkeley](https://md.research.elamur.ai/free/en/universities/university-of-california-berkeley/) — University, Berkeley, United States · 127 works · 212,523 citations
- [Queen Mary University of London](https://md.research.elamur.ai/free/en/universities/queen-mary-university-of-london/) — University, London, United Kingdom · 39 works · 3,307 citations
- [Jožef Stefan Institute](https://md.research.elamur.ai/free/en/labs/jozef-stefan-institute/) — Research institute, Ljubljana, Slovenia · 10 works · 1,959 citations
- [Jožef Stefan International Postgraduate School](https://md.research.elamur.ai/free/en/universities/jozef-stefan-international-postgraduate-school/) — University, Ljubljana, Slovenia · 6 works · 1,665 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (2): [🧾 Semantic parsing & meaning representation](https://md.research.elamur.ai/free/en/topics/semantic-parsing/) · 135 works · [💻 Code & programming languages](https://md.research.elamur.ai/free/en/topics/code-and-programming/) · 55 works

## Cite

Cite as: Yujian Gan, Xinyun Chen and Matthew Purver. 2021. Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalization. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{gan2021exploring,
  title = {{Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalization}},
  author = {Yujian Gan, Xinyun Chen and Matthew Purver},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.702},
  url = {https://doi.org/10.18653/v1/2021.emnlp-main.702}
}
```

- [← previous Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?](https://md.research.elamur.ai/free/en/papers/bentivogli-2021-cascade-versus-direct-speech-translation/)
- [next → Revisiting the Uniform Information Density Hypothesis](https://md.research.elamur.ai/free/en/papers/meister-2021-revisiting-uniform-information-density-hypothesis/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/)

## References

21 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ben Bogin, Jonathan Berant, and Matt Gardner. 2019. [Representing schema structure with graph neural networks for text-to-SQL parsing](https://md.research.elamur.ai/free/en/papers/bogin-2019-representing-schema-structure-graph-neural/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4560–4565, Florence, Italy. Association for Computational Linguistic…
2. Xiang Deng, Ahmed Hassan Awadallah, Christopher Meek, Oleksandr Polozov, Huan Sun, and Matthew Richardson. 2021. [Structure-grounded pretraining for text-to-SQL](https://md.research.elamur.ai/free/en/papers/deng-2021-structure-grounded-pretraining-text-sql/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno…
3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
4. Yujian Gan, Xinyun Chen, Qiuping Huang, Matthew Purver, John R. Woodward, Jinxia Xie, and Peng-sheng Huang. 2021. [Towards robustness of text-to-SQL models against synonym substitution](https://md.research.elamur.ai/free/en/papers/gan-2021-robustness-text-sql-models-against/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internation…
5. Yujian Gan, Matthew Purver, and John R. Woodward. 2020. A review of cross-domain text-to-SQL models. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing: Student R… [find](https://scholar.google.com/scholar?q=Yujian+Gan%2C+Matthew+Purver%2C+and+John+R%2E+Woodward%2E+2020%2E+A+review+of+cross%2Ddomain+text%2Dto%2DSQL+models%2E+In+Proceedings+of+the+1st+Conference+of+the+Asia%2DPacific+Chapter+of+the+Association+for+Computation)
6. Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. 2019. Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4524– 4535, Florence… [find](https://scholar.google.com/scholar?q=Jiaqi+Guo%2C+Zecheng+Zhan%2C+Yan+Gao%2C+Yan+Xiao%2C+Jian%2DGuang+Lou%2C+Ting+Liu%2C+and+Dongmei+Zhang%2E+2019%2E+Towards+Complex+Text%2Dto%2DSQL+in+Cross%2DDomain+Database+with+Intermediate+Representation%2E+In+Proceedings+of+)
7. Binyuan Hui, Xiang Shi, Ruiying Geng, Binhua Li, Yongbin Li, Jian Sun, and Xiaodan Zhu. 2021. Improving text-to-sql with schema dependency learning. CoRR, abs/2103.04399. — [arXiv:2103.04399](https://arxiv.org/abs/2103.04399) [find](https://scholar.google.com/scholar?q=Binyuan+Hui%2C+Xiang+Shi%2C+Ruiying+Geng%2C+Binhua+Li%2C+Yongbin+Li%2C+Jian+Sun%2C+and+Xiaodan+Zhu%2E+2021%2E+Improving+text%2Dto%2Dsql+with+schema+dependency+learning%2E+CoRR%2C+abs%2F2103%2E04399%2E+%E2%80%94+%5BarXiv%3A2103%2E04399%5D%28https%3A%2F%2F)
8. Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017. Learning a Neural Semantic Parser from User Feedback. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 963–973. [find](https://scholar.google.com/scholar?q=Srinivasan+Iyer%2C+Ioannis+Konstas%2C+Alvin+Cheung%2C+Jayant+Krishnamurthy%2C+and+Luke+Zettlemoyer%2E+2017%2E+Learning+a+Neural+Semantic+Parser+from+User+Feedback%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the)
9. Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. 2020. Measuring compositional generalization: A comprehensive method on… [find](https://scholar.google.com/scholar?q=Daniel+Keysers%2C+Nathanael+Sch%C3%A4rli%2C+Nathan+Scales%2C+Hylke+Buisman%2C+Daniel+Furrer%2C+Sergii+Kashubin%2C+Nikola+Momchev%2C+Danila+Sinopalnikov%2C+Lukasz+Stafiniak%2C+Tibor+Tihon%2C+Dmitry+Tsarkov%2C+Xiao+Wang%2C+Marc+van)
10. Inbar Oren, Jonathan Herzig, Nitish Gupta, Matt Gardner, and Jonathan Berant. 2020. Improving compositional generalization in semantic parsing. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2482–2495, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Inbar+Oren%2C+Jonathan+Herzig%2C+Nitish+Gupta%2C+Matt+Gardner%2C+and+Jonathan+Berant%2E+2020%2E+Improving+compositional+generalization+in+semantic+parsing%2E+In+Findings+of+the+Association+for+Computational+Linguis)
11. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. [Glove: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
12. Ohad Rubin and Jonathan Berant. 2021. SmBoP: Semi-autoregressive bottom-up semantic parsing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 311–324, Online. Association for Computational Linguis… [find](https://scholar.google.com/scholar?q=Ohad+Rubin+and+Jonathan+Berant%2E+2021%2E+SmBoP%3A+Semi%2Dautoregressive+bottom%2Dup+semantic+parsing%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Li)
13. Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning. CoRR, abs/1709.0. [find](https://scholar.google.com/scholar?q=Victor+Zhong%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+2017%2E+Seq2SQL%3A+Generating+Structured+Queries+from+Natural+Language+using+Reinforcement+Learning%2E+CoRR%2C+abs%2F1709%2E0%2E)
14. Peter Shaw, Ming-Wei Chang, Panupong Pasupat, and Kristina Toutanova. 2021. Compositional generalization and natural language variation: Can a semantic parsing approach handle both? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International… [find](https://scholar.google.com/scholar?q=Peter+Shaw%2C+Ming%2DWei+Chang%2C+Panupong+Pasupat%2C+and+Kristina+Toutanova%2E+2021%2E+Compositional+generalization+and+natural+language+variation%3A+Can+a+semantic+parsing+approach+handle+both%3F+In+Proceedings+of+)
15. Peng Shi, Patrick Ng, Zhiguo Wang, Henghui Zhu, Alexander Hanbo Li, Jun Wang, Cícero Nogueira dos Santos, and Bing Xiang. 2020. Learning contextual representations for semantic parsing with generation-augmented pre-training. CoRR, abs/2012.10309. — [arXiv:2012.10309](https://arxiv.org/abs/2012.10309) [find](https://scholar.google.com/scholar?q=Peng+Shi%2C+Patrick+Ng%2C+Zhiguo+Wang%2C+Henghui+Zhu%2C+Alexander+Hanbo+Li%2C+Jun+Wang%2C+C%C3%ADcero+Nogueira+dos+Santos%2C+and+Bing+Xiang%2E+2020%2E+Learning+contextual+representations+for+semantic+parsing+with+generation)
16. Robyn Speer and Catherine Havasi. 2012. Representing General Relational Knowledge in Concept-Net 5. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 3679–3686, Istanbul, Turkey. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Robyn+Speer+and+Catherine+Havasi%2E+2012%2E+Representing+General+Relational+Knowledge+in+Concept%2DNet+5%2E+In+Proceedings+of+the+Eighth+International+Conference+on+Language+Resources+and+Evaluation+%28LREC%E2%80%9912%29)
17. Alane Suhr, Ming-Wei Chang, Peter Shaw, and Kenton Lee. 2020. Exploring unexplored generalization challenges for cross-database semantic parsing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8372– 8388, Online. Association for Computational Lingui… [find](https://scholar.google.com/scholar?q=Alane+Suhr%2C+Ming%2DWei+Chang%2C+Peter+Shaw%2C+and+Kenton+Lee%2E+2020%2E+Exploring+unexplored+generalization+challenges+for+cross%2Ddatabase+semantic+parsing%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Assoc)
18. Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020. [RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers](https://md.research.elamur.ai/free/en/papers/wang-2020-rat-sql-relation-aware-schema/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7567–7578, Online. Association for…
19. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. [Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task](https://md.research.elamur.ai/free/en/papers/yu-2018-spider-large-scale-human-labeled/). In Proceedings of the 2018…
20. John M Zelle and Raymond J Mooney. 1996. [Learning to Parse Database Queries Using Inductive Logic Programming](https://md.research.elamur.ai/free/en/papers/zelle-1996-learning-parse-database-queries-using/). In Proceedings of the Thirteenth National Conference on Artificial Intelligence - Volume 2, pages 1050–1055.
21. Liang Zhao, Hexin Cao, and Yunsong Zhao. 2021. GP: context-free grammar pre-training for text-to-sql parsers. CoRR, abs/2101.09901. — [arXiv:2101.09901](https://arxiv.org/abs/2101.09901) [find](https://scholar.google.com/scholar?q=Liang+Zhao%2C+Hexin+Cao%2C+and+Yunsong+Zhao%2E+2021%2E+GP%3A+context%2Dfree+grammar+pre%2Dtraining+for+text%2Dto%2Dsql+parsers%2E+CoRR%2C+abs%2F2101%2E09901%2E+%E2%80%94+%5BarXiv%3A2101%2E09901%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2101%2E09901%29)

*Updated 24 October 2026*