---
title: "RYANSQL — paper by Choi et al. (2021) — summary and ideas"
description: "The authors introduce Statement Position Code (SPC). 4.9 points. 30 references, cited by 42 papers here."
authors: ["DongHyun Choi", "Myeong Cheol Shin", "EungGyun Kim", "Dong Ryeol Shin"]
year: 2021
venue: "Computational Linguistics"
source: "https://aclanthology.org/2021.cl-2.12/"
doi: "10.1162/coli_a_00403"
licence: "CC BY-NC-ND 4.0 (Computational Linguistics, MIT Press)"
licence_url: "https://creativecommons.org/licenses/by-nc-nd/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/choi-2021-ryansql-recursively-applying-sketch-based/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/choi-2021-ryansql-recursively-applying-sketch-based/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / RYANSQL: Recursively Applying Sketch-based Slot Fillings for Complex Text-to-SQL in Cross-Domain Databases

# RYANSQL: Recursively Applying Sketch-based Slot Fillings for Complex Text-to-SQL in Cross-Domain Databases

**Authors:** DongHyun Choi, Myeong Cheol Shin, EungGyun Kim and Dong Ryeol Shin · **Year:** 2021 · **Venue:** [Computational Linguistics](https://md.research.elamur.ai/free/en/venues/cl/) ([Computational Linguistics 2021](https://md.research.elamur.ai/free/en/venues/cl/2021/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY-NC-ND 4.0 (Computational Linguistics, MIT Press)](https://creativecommons.org/licenses/by-nc-nd/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.cl-2.12/) · [PDF](https://aclanthology.org/2021.cl-2.12.pdf) · [DOI 10.1162/coli\_a\_00403](https://doi.org/10.1162/coli_a_00403) · [OpenAlex](https://openalex.org/W3015937477)

**Citations:** 114 citations (OpenAlex, 2026-10-14) · [42](https://md.research.elamur.ai/free/en/papers/choi-2021-ryansql-recursively-applying-sketch-based/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
RYANSQL addresses Text-to-SQL: turning a natural-language question into an SQL query when the target database was not seen during training. Earlier benchmarks such as WikiSQL allowed only single tables and flat queries. RYANSQL targets the harder Spider benchmark, whose queries can contain nesting, multi-table JOINs, GROUPBY, ORDERBY and HAVING.

The authors introduce Statement Position Code (SPC). SPC labels where each sub-query sits, which turns a nested query into a set of flat SELECT statements. The model starts from the outermost statement and generates each sub-statement recursively. A sketch-based slot-filling decoder fills the slots of every SELECT. The encoder combines several components:
- GloVe and character embeddings
- dense CNNs
- attention-based alignment of question tokens with columns and tables

Two simple input manipulation methods add further gains, and BERT can replace the base embeddings.

On Spider, RYANSQL reaches 58.2% test exact-match accuracy, and the RYANSQL v2 variant reaches 60.6%. This made v2 the best system that does not use database content, and third overall as of April 2020. The work shows that structured, recursive slot filling handles complex cross-domain SQL generation well.

## Key points
- SPC elements are NONE, UNION, INTERSECT, EXCEPT, WHERE, HAVING and PARALLEL. Generation begins at [NONE] and recurses until every SPC has its own SELECT statement.
- Among systems that do not use database content, RYANSQL and RYANSQL v2 improve on the previous best, SLSQL, by 2.5 and 4.9 points of test exact-match accuracy.
- On CSpider (Chinese), RYANSQL with Multilingual BERT scores 41.3% on dev and 34.7% on test.
- The authors examined 345 dev-set failures of RYANSQL + BERT and could categorize 195 of them:
  - wrong column selection: 34.9%
  - wrong number of tables: 25.2%
  - wrong number of conditions: 11.3%
- Accuracy depends on word overlap between the question and the schema names in the query, rising from 32.5% at the lowest overlap to 80.5% at the highest. Accuracy is weakly negatively correlated with the number of foreign keys (Pearson ρ = −0.22). Code is available at github.com/kakaoenterprise/RYANSQL.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Sungkyunkwan University](https://md.research.elamur.ai/free/en/universities/sungkyunkwan-university/) — University, Suwon, South Korea · 2 works · 173 citations

Also on the byline: [Kakao Enterprise](https://ror.org/049enk825).

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🌐 Computer Networks and Communications](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-networks-and-communications/) · 19 works

🧭 Topics (2): [🧾 Semantic parsing & meaning representation](https://md.research.elamur.ai/free/en/topics/semantic-parsing/) · 135 works · [💻 Code & programming languages](https://md.research.elamur.ai/free/en/topics/code-and-programming/) · 56 works

## Cite

Cite as: DongHyun Choi, Myeong Cheol Shin, EungGyun Kim and Dong Ryeol Shin. 2021. RYANSQL: Recursively Applying Sketch-based Slot Fillings for Complex Text-to-SQL in Cross-Domain Databases. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{choi2021ryansql,
  title = {{RYANSQL: Recursively Applying Sketch-based Slot Fillings for Complex Text-to-SQL in Cross-Domain Databases}},
  author = {DongHyun Choi, Myeong Cheol Shin, EungGyun Kim and Dong Ryeol Shin},
  journal = {Computational Linguistics},
  year = {2021},
  doi = {10.1162/coli\_a\_00403},
  url = {https://doi.org/10.1162/coli_a_00403}
}
```

- [← previous MultiEURLEX - A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer](https://md.research.elamur.ai/free/en/papers/chalkidis-2021-multieurlex-multi-lingual-multi-label/)
- [next → The (Un)Suitability of Automatic Evaluation Metrics for Text Simplification](https://md.research.elamur.ai/free/en/papers/manchego-2021-un-suitability-automatic-evaluation-metrics/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/)

## References

30 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abadi, Martı́n, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenb… [find](https://scholar.google.com/scholar?q=Abadi%2C+Mart%C4%B1%CC%81n%2C+Ashish+Agarwal%2C+Paul+Barham%2C+Eugene+Brevdo%2C+Zhifeng+Chen%2C+Craig+Citro%2C+Greg+S%2E+Corrado%2C+Andy+Davis%2C+Jeffrey+Dean%2C+Matthieu+Devin%2C+Sanjay+Ghemawat%2C+Ian+Goodfellow%2C+Andrew+Harp%2C+Geoffrey)
2. Ba, Jimmy Lei, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. [Layer normalization](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/). Computing Research Repository, arXiv:1607.06450.
3. Beck, Thorsten, Asli Demirguc-Kunt, and Ross Levine. 2000. A new database on the structure and development of the financial sector. The World Bank Economic Review, 14(3):597–605. https://doi.org/10.1093/wber/14.3.597 Bogin, Ben, Matt Gardner, and Jonathan Berant. 2019. Global reasoning over databas… [↗](https://doi.org/10.1093/wber/14.3.597)
4. Clark, Kevin, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. Computing Research Repository, arXiv:2003.10555. [find](https://scholar.google.com/scholar?q=Clark%2C+Kevin%2C+Minh%2DThang+Luong%2C+Quoc+V%2E+Le%2C+and+Christopher+D%2E+Manning%2E+2020%2E+Electra%3A+Pre%2Dtraining+text+encoders+as+discriminators+rather+than+generators%2E+Computing+Research+Repository%2C+arXiv%3A2003%2E10)
5. Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies…
6. Minneapolis, MN. Dong, Li and Mirella Lapata. 2016. [Language to logical form with neural attention](https://md.research.elamur.ai/free/en/papers/dong-2016-language-logical-form-neural-attention/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 33–43. Berlin. https://doi.org/10.18653/v1/P16-1004 Dong, Li and Mirella Lapata. 2018. Coarse-to-fine… [↗](https://doi.org/10.18653/v1/P16-1004)
7. Guo, Tong and Huilin Gao. 2019. Content enhanced BERT-based text-to-SQL generation. Computing Research Repository, arXiv:1910.07179. — [arXiv:1910.07179](https://arxiv.org/abs/1910.07179) [find](https://scholar.google.com/scholar?q=Guo%2C+Tong+and+Huilin+Gao%2E+2019%2E+Content+enhanced+BERT%2Dbased+text%2Dto%2DSQL+generation%2E+Computing+Research+Repository%2C+arXiv%3A1910%2E07179%2E+%E2%80%94+%5BarXiv%3A1910%2E07179%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1910%2E07179%29)
8. He, Pengcheng, Yi Mao, Kaushik Chakrabarti, and Weizhu Chen. 2018. X Reinforce schema representation with context. Computing Research Repository, arXiv:1808.073837. — [arXiv:1808.07383](https://arxiv.org/abs/1808.07383) [find](https://scholar.google.com/scholar?q=He%2C+Pengcheng%2C+Yi+Mao%2C+Kaushik+Chakrabarti%2C+and+Weizhu+Chen%2E+2018%2E+X+Reinforce+schema+representation+with+context%2E+Computing+Research+Repository%2C+arXiv%3A1808%2E073837%2E+%E2%80%94+%5BarXiv%3A1808%2E07383%5D%28https%3A%2F%2Farxiv%2E)
9. Hillestad, Richard, James Bigelow, Anthony Bower, Federico Girosi, Robin Meili, Richard Scoville, and Roger Taylor. 2005. Can electronic medical record systems transform health care? Potential health benefits, savings, and costs. Health Affairs. 24(5):1103–1117. [https://doi.org](https://doi.org) /10.1377/hlthaff.24.5… — [doi:10.1377/hlthaff.24.5.1103](https://doi.org/10.1377/hlthaff.24.5.1103) [↗](https://doi.org)
10. Stockholm. Hwang, Wonseok, Jinyeong Yim, Seunghyun Park, and Minjoon Seo. 2019. A comprehensive exploration on WikiSQL with table-aware word contextualization. Computing Research Repository, arXiv:1902.01069. [find](https://scholar.google.com/scholar?q=Stockholm%2E+Hwang%2C+Wonseok%2C+Jinyeong+Yim%2C+Seunghyun+Park%2C+and+Minjoon+Seo%2E+2019%2E+A+comprehensive+exploration+on+WikiSQL+with+table%2Daware+word+contextualization%2E+Computing+Research+Repository%2C+arXiv%3A190)
11. Lee, Dongjun. 2019. Clause-wise and recursive decoding for complex and cross-domain text-to-SQL generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6047–6… [find](https://scholar.google.com/scholar?q=Lee%2C+Dongjun%2E+2019%2E+Clause%2Dwise+and+recursive+decoding+for+complex+and+cross%2Ddomain+text%2Dto%2DSQL+generation%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+an)
12. Lin, Kevin, Ben Bogin, Mark Neumann, Jonathan Berant, and Matt Gardner. 2019. Grammar-based neural text-to-SQL generation. Computing Research Repository, arXiv:1905.13326. — [arXiv:1905.13326](https://arxiv.org/abs/1905.13326) [find](https://scholar.google.com/scholar?q=Lin%2C+Kevin%2C+Ben+Bogin%2C+Mark+Neumann%2C+Jonathan+Berant%2C+and+Matt+Gardner%2E+2019%2E+Grammar%2Dbased+neural+text%2Dto%2DSQL+generation%2E+Computing+Research+Repository%2C+arXiv%3A1905%2E13326%2E+%E2%80%94+%5BarXiv%3A1905%2E13326%5D%28https%3A%2F)
13. Liu, Yinhan, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. Computing Research Repository. arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Liu%2C+Yinhan%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+RoBERTa%3A+A+robustly+optimized+BERT+pretraining+approach%2E+)
14. Lyu, Qin, Kaushik Chakrabarti, Shobhit Hathi, Souvik Kundu, Jianwen Zhang, and Zheng Chen. 2020. Hybrid ranking network for text-to-SQL, Microsoft Dynamics 365 AI. [find](https://scholar.google.com/scholar?q=Lyu%2C+Qin%2C+Kaushik+Chakrabarti%2C+Shobhit+Hathi%2C+Souvik+Kundu%2C+Jianwen+Zhang%2C+and+Zheng+Chen%2E+2020%2E+Hybrid+ranking+network+for+text%2Dto%2DSQL%2C+Microsoft+Dynamics+365+AI%2E)
15. Min, Qingkai, Yuefeng Shi, and Yue Zhang. 2019. A pilot study for Chinese SQL semantic parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3652–3658. Hong Ko… [find](https://scholar.google.com/scholar?q=Min%2C+Qingkai%2C+Yuefeng+Shi%2C+and+Yue+Zhang%2E+2019%2E+A+pilot+study+for+Chinese+SQL+semantic+parsing%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+the+9th+In)
16. Mou, Lili, Rui Men, Ge Li, Yan Xu, Lu Zhang, Rui Yan, and Zhi Jin. 2016. [Natural language inference by tree-based convolution and heuristic matching](https://md.research.elamur.ai/free/en/papers/mou-2016-natural-language-inference-tree-based/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 130–136. Berlin. [https://doi.org](https://doi.org) /10.18653/v1/P16-20… [↗](https://doi.org)
17. Srivastava, Nitish, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958. [find](https://scholar.google.com/scholar?q=Srivastava%2C+Nitish%2C+Geoffrey+Hinton%2C+Alex+Krizhevsky%2C+Ilya+Sutskever%2C+and+Ruslan+Salakhutdinov%2E+2014%2E+Dropout%3A+A+simple+way+to+prevent+neural+networks+from+overfitting%2E+The+Journal+of+Machine+Learning)
18. Srivastava, Rupesh K., Klaus Greff, and Jrgen Schmidhuber. 2015. Training very deep networks. Advances in Neural Information Processing Systems, pages 2377–2385. [find](https://scholar.google.com/scholar?q=Srivastava%2C+Rupesh+K%2E%2C+Klaus+Greff%2C+and+Jrgen+Schmidhuber%2E+2015%2E+Training+very+deep+networks%2E+Advances+in+Neural+Information+Processing+Systems%2C+pages+2377%E2%80%932385%2E)
19. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in Neural Information Processing Systems, 30:5998–6008.
20. Vinyals, Oriol, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. Advances in Neural Information Processing Systems, pages 2692–2700. [find](https://scholar.google.com/scholar?q=Vinyals%2C+Oriol%2C+Meire+Fortunato%2C+and+Navdeep+Jaitly%2E+2015%2E+Pointer+networks%2E+Advances+in+Neural+Information+Processing+Systems%2C+pages+2692%E2%80%932700%2E)
21. Wang, Bailin, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020. [RAT-SQL: Relation-aware schema encoding and linking for text-to-SQL parsers](https://md.research.elamur.ai/free/en/papers/wang-2020-rat-sql-relation-aware-schema/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7567–7578. Online. https://doi .or…
22. Xu, Xiaojun, Chang Liu, and Dawn Song. 2017. SQLnet: Generating structured queries from natural language without reinforcement learning. Computing Research Repository, arXiv:1711.04436. [find](https://scholar.google.com/scholar?q=Xu%2C+Xiaojun%2C+Chang+Liu%2C+and+Dawn+Song%2E+2017%2E+SQLnet%3A+Generating+structured+queries+from+natural+language+without+reinforcement+learning%2E+Computing+Research+Repository%2C+arXiv%3A1711%2E04436%2E)
23. Yavuz, Semih, Izzeddin Gur, Yu Su, and Xifeng Yan. 2018. What it takes to achieve 100% condition accuracy on WikiSQL. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1702–1711. Brussels. https://doi.org/10.18653/v1/D18-1197 [↗](https://doi.org/10.18653/v1/D18-1197)
24. Yoon, Deunsol, Dongbok Lee, and Sangkeun Lee. 2018. Dynamic self-attention: Computing attention over words dynamically for sentence embedding. Computing Research Repository, arXiv:1808.073837. — [arXiv:1808.07383](https://arxiv.org/abs/1808.07383) [find](https://scholar.google.com/scholar?q=Yoon%2C+Deunsol%2C+Dongbok+Lee%2C+and+Sangkeun+Lee%2E+2018%2E+Dynamic+self%2Dattention%3A+Computing+attention+over+words+dynamically+for+sentence+embedding%2E+Computing+Research+Repository%2C+arXiv%3A1808%2E073837%2E+%E2%80%94+%5BarXi)
25. Yu, Tao, Zifan Li, Zilin Zhang, Rui Zhang, and Dragomir Radev. 2018a. [TypeSQL: Knowledge-based type-aware neural text-to-SQL generation](https://md.research.elamur.ai/free/en/papers/yu-2018-typesql-knowledge-based-type-aware/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 588–594.
26. New Orleans, LA. Yu, Tao, Michihiro Yasunaga, Kai Yang, Rui Zhang, Dongxu Wang, Zifan Li, and Dragomir Radev. 2018b. SyntaxSQLnet: Syntax tree networks for complex and cross-domain text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1653… [find](https://scholar.google.com/scholar?q=New+Orleans%2C+LA%2E+Yu%2C+Tao%2C+Michihiro+Yasunaga%2C+Kai+Yang%2C+Rui+Zhang%2C+Dongxu+Wang%2C+Zifan+Li%2C+and+Dragomir+Radev%2E+2018b%2E+SyntaxSQLnet%3A+Syntax+tree+networks+for+complex+and+cross%2Ddomain+text%2Dto%2DSQL+task%2E+I)
27. Yu, Tao, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018c. [Spider: A large-scale human-labeled data set for complex and cross-domain semantic parsing and text-to-SQL task](https://md.research.elamur.ai/free/en/papers/yu-2018-spider-large-scale-human-labeled/). In Proceedings of the 20…
28. Zelle, John Marvin and Raymond Joseph Mooney. 1996. [Learning to parse database queries using inductive logic programming](https://md.research.elamur.ai/free/en/papers/zelle-1996-learning-parse-database-queries-using/). In Proceedings of the Thirteenth National Conference On Artificial Intelligence, volume 2, pages 1050–1055.
29. Portland, OR. Zhang, Rui, Tao Yu, Heyang Er, Sungrok Shim, Eric Xue, Xi Victoria Lin, Tianze Shi, Caiming Xiong, Richard Socher, and Dragomir Radev. 2019. [Editing-based SQL query generation for cross-domain context-dependent questions](https://md.research.elamur.ai/free/en/papers/zhang-2019-editing-based-sql-query-generation/). In Proceedings of the 2019 Conference on Empirical Methods in Na…
30. Hong Kong. Zhong, Victor, Caiming Xiong, and Richard Socher. 2017. Seq2SQL: Generating structured queries from natural language using reinforcement learning. Computing Research Repository, arXiv:1709.00103. [find](https://scholar.google.com/scholar?q=Hong+Kong%2E+Zhong%2C+Victor%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+2017%2E+Seq2SQL%3A+Generating+structured+queries+from+natural+language+using+reinforcement+learning%2E+Computing+Research+Repository%2C+arXiv%3A1709%2E)

*Updated 24 October 2026*