---
title: "A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL"
description: "The paper asks how well ChatGPT turns natural-language questions into executable SQL with no… 29 references, cited by 66 papers here."
authors: ["Aiwei Liu", "Xuming Hu", "Lijie Wen", "Philip S. Yu"]
year: 2023
venue: "arXiv:2303.13547"
source: "https://arxiv.org/abs/2303.13547"
doi: "10.48550/arxiv.2303.13547"
arxiv: "2303.13547"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2023-comprehensive-evaluation-chatgpt-s-zero/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2023-comprehensive-evaluation-chatgpt-s-zero/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/) / A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capability

# A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capability

**Authors:** Aiwei Liu, [Xuming Hu](https://md.research.elamur.ai/free/en/authors/xuming-hu/), Lijie Wen and [Philip S. Yu](https://md.research.elamur.ai/free/en/authors/philip-s-yu/) · **Year:** 2023 · **Venue:** arXiv:2303.13547 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2303.13547) · [PDF](https://arxiv.org/pdf/2303.13547) · [DOI 10.48550/arxiv.2303.13547](https://doi.org/10.48550/arxiv.2303.13547) · [arXiv 2303.13547](https://arxiv.org/abs/2303.13547) · [OpenAlex](https://openalex.org/W4361019453)

**Citations:** 61 citations (OpenAlex, 2026-10-14) · [66](https://md.research.elamur.ai/free/en/papers/liu-2023-comprehensive-evaluation-chatgpt-s-zero/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper asks how well ChatGPT turns natural-language questions into executable SQL with no task-specific training. The authors reused the SQL-translation prompt from OpenAI's demo page and did not tune it. The prompt lists the database tables and their columns, asks for SQLite-compatible output, and adds only the new question in each turn of a dialogue. They tested this zero-shot setup on 12 public benchmarks: Spider and its robustness variants, the Chinese datasets CSpider and DuSQL, and the multi-turn datasets SParC and CoSQL. ChatGPT was compared with fine-tuned systems (T5-3B + PICARD, RASAT + PICARD, RESDSQL-3B + NatSQL) using execution-based metrics. ChatGPT trails the strongest fine-tuned model by about 14 points of execution accuracy on Spider. The gap narrows on several robustness sets, and ChatGPT wins on ADVETA (RPL). The results suggest that a general conversational LLM can serve as a competitive Text-to-SQL baseline without annotated data. The generated outputs are released for further research.

## Key points
- **Metrics:** valid SQL (VA), execution accuracy (EX) and test-suite accuracy (TS) were used. Exact match was dropped because equivalent queries can be written in many ways.
- **Spider dev:** ChatGPT reaches 70.1 EX and 60.1 TS. RESDSQL-3B + NatSQL scores 84.1 EX and 73.5 TS.
- **Robustness:** On Spider-SYN (58.6 EX) and Spider-Realistic (63.4 EX), the gap to SOTA grows to about 18 points. It shrinks on Spider-DK (62.6 vs 66.0), ADVETA (ADD) (68.1 vs 71.9) and Spider-CG (SUB/APP: 76.6/61.3 vs 83.3/69.4).
- **ADVETA (RPL):** ChatGPT scores 58.5 EX against 54.4 for the best fine-tuned model, a lead of 4.1 points. This benchmark adversarially renames database columns.
- **Errors and resources:** Typical errors include faulty joins or schema misunderstanding, wrong subquery logic, and changed letter case in copied values. ChatGPT outputs are published at github.com/THU-BPM/chatgpt-sql.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 137 works · 83,262 citations
- [University of Illinois Chicago](https://md.research.elamur.ai/free/en/universities/university-of-illinois-chicago/) — University, Chicago, United States · 38 works · 19,844 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🌐 Computer Networks and Communications](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-networks-and-communications/) · 19 works

🧭 Topics (3): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 685 works · [🧾 Semantic parsing & meaning representation](https://md.research.elamur.ai/free/en/topics/semantic-parsing/) · 135 works · [💻 Code & programming languages](https://md.research.elamur.ai/free/en/topics/code-and-programming/) · 56 works

## Cite

Cite as: Aiwei Liu, Xuming Hu, Lijie Wen and Philip S. Yu. 2023. A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capability. arXiv:2303.13547.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{liu2023comprehensive,
  title = {{A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capability}},
  author = {Aiwei Liu, Xuming Hu, Lijie Wen and Philip S. Yu},
  year = {2023},
  doi = {10.48550/arxiv.2303.13547},
  eprint = {2303.13547},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.13547}
}
```

- [← previous UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model](https://md.research.elamur.ai/free/en/papers/ye-2023-ureader-universal-ocr-free-visually/)
- [next → Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks](https://md.research.elamur.ai/free/en/papers/veselovsky-2023-artificial-artificial-artificial-intelligence-crowd/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/)

## References

29 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
2. Zefeng Cai, Xiangyu Li, Binyuan Hui, Min Yang, Bowen Li, Binhua Li, Zheng Cao, Weijie Li, Fei Huang, Luo Si, et al. 2022. Star: Sql guided pre-training for context-dependent text-to-sql parsing. arXiv preprint arXiv:2210.11888. [find](https://scholar.google.com/scholar?q=Zefeng+Cai%2C+Xiangyu+Li%2C+Binyuan+Hui%2C+Min+Yang%2C+Bowen+Li%2C+Binhua+Li%2C+Zheng+Cao%2C+Weijie+Li%2C+Fei+Huang%2C+Luo+Si%2C+et+al%2E+2022%2E+Star%3A+Sql+guided+pre%2Dtraining+for+context%2Ddependent+text%2Dto%2Dsql+parsing%2E+arXiv)
3. Ruisheng Cao, Lu Chen, Zhi Chen, Yanbin Zhao, Su Zhu, and Kai Yu. 2021. [Lgesql: line graph enhanced text-to-sql model with mixed local and non-local relations](https://md.research.elamur.ai/free/en/papers/cao-2021-lgesql-line-graph-enhanced-text/). arXiv preprint arXiv:2106.01093.
4. Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, et al. 2022. [Binding language models in symbolic languages](https://md.research.elamur.ai/free/en/papers/cheng-2022-binding-language-models-symbolic-languages/). arXiv preprint arXiv:2210.02875.
5. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30. [find](https://scholar.google.com/scholar?q=Paul+F+Christiano%2C+Jan+Leike%2C+Tom+Brown%2C+Miljan+Martic%2C+Shane+Legg%2C+and+Dario+Amodei%2E+2017%2E+Deep+reinforcement+learning+from+human+preferences%2E+Advances+in+neural+information+processing+systems%2C+30%2E)
6. Xiang Deng, Ahmed Hassan Awadallah, Christopher Meek, Oleksandr Polozov, Huan Sun, and Matthew Richardson. 2020. [Structure-grounded pretraining for text-to-sql](https://md.research.elamur.ai/free/en/papers/deng-2021-structure-grounded-pretraining-text-sql/). arXiv preprint arXiv:2010.12773.
7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805.
8. Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier, and Julius Berner. 2023. [Mathematical capabilities of chatgpt](https://md.research.elamur.ai/free/en/papers/frieder-2023-mathematical-capabilities-chatgpt/). arXiv preprint arXiv:2301.13867.
9. Yujian Gan, Xinyun Chen, Qiuping Huang, and Matthew Purver. 2022. Measuring and improving compositional generalization in text-to-sql via component alignment. [find](https://scholar.google.com/scholar?q=Yujian+Gan%2C+Xinyun+Chen%2C+Qiuping+Huang%2C+and+Matthew+Purver%2E+2022%2E+Measuring+and+improving+compositional+generalization+in+text%2Dto%2Dsql+via+component+alignment%2E)
10. Yujian Gan, Xinyun Chen, Qiuping Huang, Matthew Purver, John R Woodward, Jinxia Xie, and Peng-sheng Huang. 2021a. [Towards robustness of text-to-sql models against synonym substitution](https://md.research.elamur.ai/free/en/papers/gan-2021-robustness-text-sql-models-against/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internation…
11. Yujian Gan, Xinyun Chen, and Matthew Purver. 2021b. [Exploring underexplored limitations of cross-domain text-to-sql generalization](https://md.research.elamur.ai/free/en/papers/gan-2021-exploring-underexplored-limitations-cross-domain/). arXiv preprint arXiv:2109.05157.
12. Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. 2021. Picard: Parsing incrementally for constrained auto-regressive decoding from language models. arXiv preprint arXiv:2109.05093. [find](https://scholar.google.com/scholar?q=Torsten+Scholak%2C+Nathan+Schucher%2C+and+Dzmitry+Bahdanau%2E+2021%2E+Picard%3A+Parsing+incrementally+for+constrained+auto%2Dregressive+decoding+from+language+models%2E+arXiv+preprint+arXiv%3A2109%2E05093%2E)
13. Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2019. [Rat-sql: Relation-aware schema encoding and linking for text-to-sql parsers](https://md.research.elamur.ai/free/en/papers/wang-2020-rat-sql-relation-aware-schema/). arXiv preprint arXiv:1911.04942.
14. Jiaan Wang, Yunlong Liang, Fandong Meng, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2023. Cross-lingual summarization via chatgpt. arXiv preprint arXiv:2302.14229. — [arXiv:2302.14229](https://arxiv.org/abs/2302.14229) [find](https://scholar.google.com/scholar?q=Jiaan+Wang%2C+Yunlong+Liang%2C+Fandong+Meng%2C+Zhixu+Li%2C+Jianfeng+Qu%2C+and+Jie+Zhou%2E+2023%2E+Cross%2Dlingual+summarization+via+chatgpt%2E+arXiv+preprint+arXiv%3A2302%2E14229%2E+%E2%80%94+%5BarXiv%3A2302%2E14229%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs)
15. Lihan Wang, Bowen Qin, Binyuan Hui, Bowen Li, Min Yang, Bailin Wang, Binhua Li, Jian Sun, Fei Huang, Luo Si, et al. 2022. Proton: Probing schema linking information from pre-trained language models for text-to-sql parsing. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and D… [find](https://scholar.google.com/scholar?q=Lihan+Wang%2C+Bowen+Qin%2C+Binyuan+Hui%2C+Bowen+Li%2C+Min+Yang%2C+Bailin+Wang%2C+Binhua+Li%2C+Jian+Sun%2C+Fei+Huang%2C+Luo+Si%2C+et+al%2E+2022%2E+Proton%3A+Probing+schema+linking+information+from+pre%2Dtrained+language+models+fo)
16. Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. 2023. [Decoupling the skeleton parsing and schema linking for text-to-sql](https://md.research.elamur.ai/free/en/papers/li-2023-resdsql-decoupling-schema-linking-skeleton/). arXiv preprint arXiv:2302.05965.
17. Lijie Wang, Ao Zhang, Kun Wu, Ke Sun, Zhenghua Li, Hua Wu, Min Zhang, and Haifeng Wang. 2020. [Dusql: A large-scale and pragmatic chinese text-to-sql dataset](https://md.research.elamur.ai/free/en/papers/wang-2020-dusql-large-scale-pragmatic-chinese/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6923–6935.
18. Aiwei Liu, Xuming Hu, Li Lin, and Lijie Wen. 2022. Semantic enhanced text-to-sql parsing via iteratively learning schema linking graph. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1021–1030. [find](https://scholar.google.com/scholar?q=Aiwei+Liu%2C+Xuming+Hu%2C+Li+Lin%2C+and+Lijie+Wen%2E+2022%2E+Semantic+enhanced+text%2Dto%2Dsql+parsing+via+iteratively+learning+schema+linking+graph%2E+In+Proceedings+of+the+28th+ACM+SIGKDD+Conference+on+Knowledge+Di)
19. Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, et al. 2023. [Zero-shot information extraction via chatting with chatgpt](https://md.research.elamur.ai/free/en/papers/wei-2023-chatie-zero-shot-information-extraction/). arXiv preprint arXiv:2302.10205.
20. Qingkai Min, Yuefeng Shi, and Yue Zhang. 2019. A pilot study for chinese sql semantic parsing. arXiv preprint arXiv:1909.13293. [find](https://scholar.google.com/scholar?q=Qingkai+Min%2C+Yuefeng+Shi%2C+and+Yue+Zhang%2E+2019%2E+A+pilot+study+for+chinese+sql+semantic+parsing%2E+arXiv+preprint+arXiv%3A1909%2E13293%2E)
21. Dongling Xiao, Linzheng Chai, Qian-Wen Zhang, Zhao Yan, Zhoujun Li, and Yunbo Cao. 2022. Cqr-sql: Conversational question reformulation enhanced context-dependent text-to-sql parsers. arXiv preprint arXiv:2205.07686. [find](https://scholar.google.com/scholar?q=Dongling+Xiao%2C+Linzheng+Chai%2C+Qian%2DWen+Zhang%2C+Zhao+Yan%2C+Zhoujun+Li%2C+and+Yunbo+Cao%2E+2022%2E+Cqr%2Dsql%3A+Conversational+question+reformulation+enhanced+context%2Ddependent+text%2Dto%2Dsql+parsers%2E+arXiv+preprint+a)
22. Xinyu Pi, Bing Wang, Yan Gao, Jiaqi Guo, Zhoujun Li, and Jian-Guang Lou. 2022. Towards robustness of text-to-SQL models against natural and realistic adversarial table perturbation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pa… [find](https://scholar.google.com/scholar?q=Xinyu+Pi%2C+Bing+Wang%2C+Yan+Gao%2C+Jiaqi+Guo%2C+Zhoujun+Li%2C+and+Jian%2DGuang+Lou%2E+2022%2E+Towards+robustness+of+text%2Dto%2DSQL+models+against+natural+and+realistic+adversarial+table+perturbation%2E+In+Proceedings+of+)
23. Jiexing Qi, Jingyao Tang, Ziwei He, Xiangpeng Wan, Chenghu Zhou, Xinbing Wang, Quanshi Zhang, and Zhouhan Lin. 2022. Rasat: Integrating relational structures into pretrained seq2seq model for text-to-sql. arXiv preprint arXiv:2205.06983. [find](https://scholar.google.com/scholar?q=Jiexing+Qi%2C+Jingyao+Tang%2C+Ziwei+He%2C+Xiangpeng+Wan%2C+Chenghu+Zhou%2C+Xinbing+Wang%2C+Quanshi+Zhang%2C+and+Zhouhan+Lin%2E+2022%2E+Rasat%3A+Integrating+relational+structures+into+pretrained+seq2seq+model+for+text%2Dto%2D)
24. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). The Journal of Machine Learning Research, 21(1):5485–5551.
25. Nitarshan Rajkumar, Raymond Li, and Dzmitry Bahdanau. 2022. Evaluating the text-to-sql capabilities of large language models. arXiv preprint arXiv:2204.00498. — [arXiv:2204.00498](https://arxiv.org/abs/2204.00498) [find](https://scholar.google.com/scholar?q=Nitarshan+Rajkumar%2C+Raymond+Li%2C+and+Dzmitry+Bahdanau%2E+2022%2E+Evaluating+the+text%2Dto%2Dsql+capabilities+of+large+language+models%2E+arXiv+preprint+arXiv%3A2204%2E00498%2E+%E2%80%94+%5BarXiv%3A2204%2E00498%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
26. Tao Yu, Rui Zhang, He Yang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, et al. 2019a. [Cosql: A conversational text-to-sql challenge towards cross-domain natural language interfaces to databases](https://md.research.elamur.ai/free/en/papers/yu-2019-cosql-conversational-text-sql-challenge/). arXiv preprint arXiv:1909.05378.
27. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. 2018. [Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task](https://md.research.elamur.ai/free/en/papers/yu-2018-spider-large-scale-human-labeled/). arXiv preprint arXiv:1809.08887.
28. Tao Yu, Rui Zhang, Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li, Heyang Er, Irene Li, Bo Pang, Tao Chen, et al. 2019b. [Sparc: Cross-domain semantic parsing in context](https://md.research.elamur.ai/free/en/papers/yu-2019-sparc-cross-domain-semantic-parsing/). arXiv preprint arXiv:1906.02285.
29. Ruiqi Zhong, Tao Yu, and Dan Klein. 2020. Semantic evaluation for text-to-sql with distilled test suites. arXiv preprint arXiv:2010.02840. [find](https://scholar.google.com/scholar?q=Ruiqi+Zhong%2C+Tao+Yu%2C+and+Dan+Klein%2E+2020%2E+Semantic+evaluation+for+text%2Dto%2Dsql+with+distilled+test+suites%2E+arXiv+preprint+arXiv%3A2010%2E02840%2E)

*Updated 24 October 2026*