---
title: "CLARITY — paper by Sarwar et al. (2026) — summary and ideas"
description: "Benchmarks for NL2SQL usually assume that each question contains one source of ambiguity and that users answer… 31 references."
authors: ["Tabinda Sarwar", "Farhad Moghimifar", "Cong Duy Vu Hoang", "Xiaoxiao Ma", "Shawn Chang Xu", "Fahimeh Saleh", "Poorya Zaremoodi", "Avirup Sil", "Katrin Kirchhoff"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)"
source: "https://aclanthology.org/2026.acl-industry.86/"
doi: "10.18653/v1/2026.acl-industry.86"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sarwar-2026-clarity-framework-benchmark-conversational-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sarwar-2026-clarity-framework-benchmark-conversational-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/25/) / CLARITY: A Framework and Benchmark for Conversational Language Ambiguity and Unanswerability in Interactive NL2SQL Systems

# CLARITY: A Framework and Benchmark for Conversational Language Ambiguity and Unanswerability in Interactive NL2SQL Systems

**Authors:** Tabinda Sarwar, Farhad Moghimifar, [Cong Duy Vu Hoang](https://md.research.elamur.ai/free/en/authors/cong-duy-vu-hoang/), [Xiaoxiao Ma](https://md.research.elamur.ai/free/en/authors/xiaoxiao-ma/), Shawn Chang Xu, Fahimeh Saleh, Poorya Zaremoodi, [Avirup Sil](https://md.research.elamur.ai/free/en/authors/avirup-sil/) and [Katrin Kirchhoff](https://md.research.elamur.ai/free/en/authors/katrin-kirchhoff/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-industry.86/) · [PDF](https://aclanthology.org/2026.acl-industry.86.pdf) · [DOI 10.18653/v1/2026.acl-industry.86](https://doi.org/10.18653/v1/2026.acl-industry.86) · [OpenAlex](https://openalex.org/W7166805953)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Benchmarks for NL2SQL usually assume that each question contains one source of ambiguity and that users answer clarification requests cooperatively. Real deployments are messier: one question can be confusing in several ways at once, and users may give vague or useless replies. The Oracle team behind CLARITY built a framework that automatically turns executable SQL from Spider and BIRD into ambiguous or unanswerable questions, with no human annotation.

The pipeline works in stages:
- A rule-based SQL parser picks target columns or values.
- An adaptive schema retriever groups similar schema elements.
- An LLM invents "pivot terms" that are either ambiguous or unanswerable, and LLM judges must all approve them.
- The pivot terms are woven into the original question, and the result is expanded into single- or multi-turn dialogues.

Each instance also records span-level pivot terms and the schema target groups they point to. This metadata lets evaluators check whether a system finds the right schema elements, not just whether it flags a problem. Tested on CLARITY, strong LLMs and the AmbiSQL detector usually notice that a question is ambiguous but often fail to locate and resolve the cause. This failure grows much worse when several ambiguities occur together.

## Key points
- The framework covers four A/U modes (col_amb, val_amb, col_unans, val_unans) and three user response types (helpful, partial, unhelpful). Ambiguity is further split into lexical and semantic, and into uni-facet and multi-facet cases.
- AmbiSQL detects column ambiguity on Spider almost perfectly (DA 99.6%), but its match accuracy is low:
  - 57.8% MA for uni-facet questions on Spider.
  - About 20% MA for multi-facet questions on Spider.
  - 5.4–7.0% MA for multi-facet questions on BIRD.
- In multi-turn SQL prediction, the evaluated LLMs included GPT-4o, GPT-4.1, GPT-5, Grok-3 Mini Fast and LLaMA-3.3. Results across these models were:
  - Models typically handle unanswerable questions better than ambiguous ones.
  - Multi-facet ambiguity is harder than uni-facet ambiguity.
  - Verbose clarifications help most.
  - No single model was best in every category.
- Few-shot prompts built from CLARITY examples with metadata significantly outperform zero-shot and standard few-shot prompting (paired t-tests, p \< 0.005).
- Because the framework needs no human annotation, companies can generate ambiguous and unanswerable test variants of their own enterprise datasets at low cost.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Oracle](https://md.research.elamur.ai/free/en/companies/oracle/) — Company, Austin, United States · 7 works · 1,167 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [🗨️ Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) · 624 works

## Cite

Cite as: Tabinda Sarwar, Farhad Moghimifar, Cong Duy Vu Hoang, Xiaoxiao Ma, Shawn Chang Xu, Fahimeh Saleh, Poorya Zaremoodi, Avirup Sil and Katrin Kirchhoff. 2026. CLARITY: A Framework and Benchmark for Conversational Language Ambiguity and Unanswerability in Interactive NL2SQL Systems. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{sarwar2026clarity,
  title = {{CLARITY: A Framework and Benchmark for Conversational Language Ambiguity and Unanswerability in Interactive NL2SQL Systems}},
  author = {Tabinda Sarwar, Farhad Moghimifar, Cong Duy Vu Hoang, Xiaoxiao Ma, Shawn Chang Xu, Fahimeh Saleh, Poorya Zaremoodi, Avirup Sil and Katrin Kirchhoff},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-industry.86},
  url = {https://doi.org/10.18653/v1/2026.acl-industry.86}
}
```

- [← previous CLARIN-PT-LDB: An Open LLM Leaderboard for Portuguese to assess Language, Culture and Civility](https://md.research.elamur.ai/free/en/papers/silva-2026-clarin-pt-ldb-open-llm/)
- [next → CLARO: Controlled Attribute-Driven Reasoning Optimization for Efficient Chain-of-Thought](https://md.research.elamur.ai/free/en/papers/schlesinger-2026-claro-controlled-attribute-driven-reasoning/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/25/)

## References

31 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Adithya Bhaskar, Tushar Tomar, Ashutosh Sathe, and Sunita Sarawagi. 2023. Benchmarking and improving text-to-sql generation under ambiguity. arXiv preprint arXiv:2310.13659. [find](https://scholar.google.com/scholar?q=Adithya+Bhaskar%2C+Tushar+Tomar%2C+Ashutosh+Sathe%2C+and+Sunita+Sarawagi%2E+2023%2E+Benchmarking+and+improving+text%2Dto%2Dsql+generation+under+ambiguity%2E+arXiv+preprint+arXiv%3A2310%2E13659%2E)
2. Hasan Alp Caferoğlu, Mehmet Serhat Çelik, and Özgür Ulusoy. 2025. Sing-sql: A synthetic data generation framework for in-domain text-to-sql translation. arXiv preprint arXiv:2509.25672. — [arXiv:2509.25672](https://arxiv.org/abs/2509.25672) [find](https://scholar.google.com/scholar?q=Hasan+Alp+Caferog%CC%86lu%2C+Mehmet+Serhat+%C3%87elik%2C+and+%C3%96zg%C3%BCr+Ulusoy%2E+2025%2E+Sing%2Dsql%3A+A+synthetic+data+generation+framework+for+in%2Ddomain+text%2Dto%2Dsql+translation%2E+arXiv+preprint+arXiv%3A2509%2E25672%2E+%E2%80%94+%5BarXiv%3A2509)
3. Kaiwen Chen, Yueting Chen, Nick Koudas, and Xiaohui Yu. 2025. Reliable text-to-sql with adaptive abstention. Proceedings of the ACM on Management of Data, 3(1):1–30. [find](https://scholar.google.com/scholar?q=Kaiwen+Chen%2C+Yueting+Chen%2C+Nick+Koudas%2C+and+Xiaohui+Yu%2E+2025%2E+Reliable+text%2Dto%2Dsql+with+adaptive+abstention%2E+Proceedings+of+the+ACM+on+Management+of+Data%2C+3%281%29%3A1%E2%80%9330%2E)
4. Zhongjun Ding, Yin Lin, and Tianjing Zeng. 2025. Am-bisql: Interactive ambiguity detection and resolution for text-to-sql. arXiv preprint arXiv:2508.15276. — [arXiv:2508.15276](https://arxiv.org/abs/2508.15276) [find](https://scholar.google.com/scholar?q=Zhongjun+Ding%2C+Yin+Lin%2C+and+Tianjing+Zeng%2E+2025%2E+Am%2Dbisql%3A+Interactive+ambiguity+detection+and+resolution+for+text%2Dto%2Dsql%2E+arXiv+preprint+arXiv%3A2508%2E15276%2E+%E2%80%94+%5BarXiv%3A2508%2E15276%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2)
5. Mingwen Dong, Nischal Ashok Kumar, Yiqun Hu, Anuj Chauhan, Chung-Wei Hang, Shuaichen Chang, Lin Pan, Wuwei Lan, Henghui Zhu, Jiarong Jiang, and 1 others. 2025. [Practiq: A practical conversational text-to-sql dataset with ambiguous and unanswerable queries](https://md.research.elamur.ai/free/en/papers/dong-2025-practiq-practical-conversational-text-sql/). In Proceedings of the 2025 Conference of th…
6. Shaoming Duan, Youxuan Wu, Chuanyi Liu, Yuhao Zhang, Zirui Wang, Peiyi Han, Shengyuan Yu, Liang Yan, and Yingwei Liang. 2025. Dsqg-syn: Synthesizing high-quality data for text-to-sql parsing by domain specific question generation. In Findings of the Association for Computational Linguistics: NAACL 2… [find](https://scholar.google.com/scholar?q=Shaoming+Duan%2C+Youxuan+Wu%2C+Chuanyi+Liu%2C+Yuhao+Zhang%2C+Zirui+Wang%2C+Peiyi+Han%2C+Shengyuan+Yu%2C+Liang+Yan%2C+and+Yingwei+Liang%2E+2025%2E+Dsqg%2Dsyn%3A+Synthesizing+high%2Dquality+data+for+text%2Dto%2Dsql+parsing+by+domain)
7. Yu Guo, Dong Jin, Shenghao Ye, Shuangwu Chen, Jianyang Jianyang, and Xiaobin Tan. 2025. Sqlforge: Synthesizing reliable and diverse data to enhance text-to-sql reasoning in llms. In Findings of the Association for Computational Linguistics: ACL 2025, pages 8441–8452. [find](https://scholar.google.com/scholar?q=Yu+Guo%2C+Dong+Jin%2C+Shenghao+Ye%2C+Shuangwu+Chen%2C+Jianyang+Jianyang%2C+and+Xiaobin+Tan%2E+2025%2E+Sqlforge%3A+Synthesizing+reliable+and+diverse+data+to+enhance+text%2Dto%2Dsql+reasoning+in+llms%2E+In+Findings+of+the+As)
8. Ziming Guo, Chao Ma, Yinggang Sun, Tiancheng Zhao, Guangyao Wang, and Hai Huang. 2024. Evaluating and enhancing llms for multi-turn text-to-sql with multiple question types. arXiv preprint arXiv:2412.17867. — [arXiv:2412.17867](https://arxiv.org/abs/2412.17867) [find](https://scholar.google.com/scholar?q=Ziming+Guo%2C+Chao+Ma%2C+Yinggang+Sun%2C+Tiancheng+Zhao%2C+Guangyao+Wang%2C+and+Hai+Huang%2E+2024%2E+Evaluating+and+enhancing+llms+for+multi%2Dturn+text%2Dto%2Dsql+with+multiple+question+types%2E+arXiv+preprint+arXiv%3A2412%2E)
9. Zijin Hong, Zheng Yuan, Qinggang Zhang, Hao Chen, Junnan Dong, Feiran Huang, and Xiao Huang. 2025. Next-generation database interfaces: A survey of llm-based text-to-sql. IEEE Transactions on Knowledge and Data Engineering. [find](https://scholar.google.com/scholar?q=Zijin+Hong%2C+Zheng+Yuan%2C+Qinggang+Zhang%2C+Hao+Chen%2C+Junnan+Dong%2C+Feiran+Huang%2C+and+Xiao+Huang%2E+2025%2E+Next%2Dgeneration+database+interfaces%3A+A+survey+of+llm%2Dbased+text%2Dto%2Dsql%2E+IEEE+Transactions+on+Knowledg)
10. Yiming Huang, Jiyu Guo, Wenxin Mao, Cuiyun Gao, Peiyi Han, Chuanyi Liu, and Qing Ling. 2025. Exploring the landscape of text-to-sql with large language models: Progresses, challenges and opportunities. arXiv preprint arXiv:2505.23838. — [arXiv:2505.23838](https://arxiv.org/abs/2505.23838) [find](https://scholar.google.com/scholar?q=Yiming+Huang%2C+Jiyu+Guo%2C+Wenxin+Mao%2C+Cuiyun+Gao%2C+Peiyi+Han%2C+Chuanyi+Liu%2C+and+Qing+Ling%2E+2025%2E+Exploring+the+landscape+of+text%2Dto%2Dsql+with+large+language+models%3A+Progresses%2C+challenges+and+opportunities)
11. Nan Huo, Xiaohan Xu, Jinyang Li, Per Jacobsson, Shipei Lin, Bowen Qin, Binyuan Hui, Xiaolong Li, Ge Qu, Shuzheng Si, and 1 others. 2025. Bird-interact: Re-imagining text-to-sql evaluation for large language models via lens of dynamic interactions. arXiv preprint arXiv:2510.05318. — [arXiv:2510.05318](https://arxiv.org/abs/2510.05318) [find](https://scholar.google.com/scholar?q=Nan+Huo%2C+Xiaohan+Xu%2C+Jinyang+Li%2C+Per+Jacobsson%2C+Shipei+Lin%2C+Bowen+Qin%2C+Binyuan+Hui%2C+Xiaolong+Li%2C+Ge+Qu%2C+Shuzheng+Si%2C+and+1+others%2E+2025%2E+Bird%2Dinteract%3A+Re%2Dimagining+text%2Dto%2Dsql+evaluation+for+large+la)
12. Shivasankari Kannan, Yeounoh Chung, Amita Gondi, Tristan Swadell, and Fatma Ozcan. 2025. High-fidelity and complex test data generation for real-world sql code generation services. arXiv preprint arXiv:2504.17203. — [arXiv:2504.17203](https://arxiv.org/abs/2504.17203) [find](https://scholar.google.com/scholar?q=Shivasankari+Kannan%2C+Yeounoh+Chung%2C+Amita+Gondi%2C+Tristan+Swadell%2C+and+Fatma+Ozcan%2E+2025%2E+High%2Dfidelity+and+complex+test+data+generation+for+real%2Dworld+sql+code+generation+services%2E+arXiv+preprint+arXi)
13. Dongjun Lee, Choongwon Park, Jaehyuk Kim, and Heesoo Park. 2025. [Mcs-sql: Leveraging multiple prompts and multiple-choice selection for text-to-sql generation](https://md.research.elamur.ai/free/en/papers/lee-2025-mcs-sql-leveraging-multiple-prompts/). In Proceedings of the 31st International Conference on Computational Linguistics, pages 337– 353.
14. Boyan Li, Yuyu Luo, Chengliang Chai, Guoliang Li, and Nan Tang. 2024. The dawn of natural language to sql: Are we fully ready? arXiv preprint arXiv:2406.01265. — [arXiv:2406.01265](https://arxiv.org/abs/2406.01265) [find](https://scholar.google.com/scholar?q=Boyan+Li%2C+Yuyu+Luo%2C+Chengliang+Chai%2C+Guoliang+Li%2C+and+Nan+Tang%2E+2024%2E+The+dawn+of+natural+language+to+sql%3A+Are+we+fully+ready%3F+arXiv+preprint+arXiv%3A2406%2E01265%2E+%E2%80%94+%5BarXiv%3A2406%2E01265%5D%28https%3A%2F%2Farxiv%2Eorg%2Fa)
15. Haoyang Li, Shang Wu, Xiaokang Zhang, Xinmei Huang, Jing Zhang, Fuxin Jiang, Shuai Wang, Tieying Zhang, Jianjun Chen, Rui Shi, and 1 others. 2025. Omnisql: Synthesizing high-quality text-to-sql data at scale. arXiv preprint arXiv:2503.02240. — [arXiv:2503.02240](https://arxiv.org/abs/2503.02240) [find](https://scholar.google.com/scholar?q=Haoyang+Li%2C+Shang+Wu%2C+Xiaokang+Zhang%2C+Xinmei+Huang%2C+Jing+Zhang%2C+Fuxin+Jiang%2C+Shuai+Wang%2C+Tieying+Zhang%2C+Jianjun+Chen%2C+Rui+Shi%2C+and+1+others%2E+2025%2E+Omnisql%3A+Synthesizing+high%2Dquality+text%2Dto%2Dsql+data+a)
16. Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C.C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023. [Can llm already serve as a database interface?](https://md.research.elamur.ai/free/en/papers/li-2023-llm-already-serve-database-interface/) a big bench for large-scale database…
17. Xinyu Liu, Shuyu Shen, Boyan Li, Peixian Ma, Runzhi Jiang, Yuxin Zhang, Ju Fan, Guoliang Li, Nan Tang, and Yuyu Luo. 2025. A survey of text-to-sql in the era of llms: Where are we, and where are we going? IEEE Transactions on Knowledge and Data Engineering. [find](https://scholar.google.com/scholar?q=Xinyu+Liu%2C+Shuyu+Shen%2C+Boyan+Li%2C+Peixian+Ma%2C+Runzhi+Jiang%2C+Yuxin+Zhang%2C+Ju+Fan%2C+Guoliang+Li%2C+Nan+Tang%2C+and+Yuyu+Luo%2E+2025%2E+A+survey+of+text%2Dto%2Dsql+in+the+era+of+llms%3A+Where+are+we%2C+and+where+are+we+go)
18. Tianqi Luo, Chuhan Huang, Leixian Shen, Boyan Li, Shuyu Shen, Wei Zeng, Nan Tang, and Yuyu Luo. 2025. nvbench 2.0: Resolving ambiguity in text-to-visualization through stepwise reasoning. arXiv preprint arXiv:2503.12880. — [arXiv:2503.12880](https://arxiv.org/abs/2503.12880) [find](https://scholar.google.com/scholar?q=Tianqi+Luo%2C+Chuhan+Huang%2C+Leixian+Shen%2C+Boyan+Li%2C+Shuyu+Shen%2C+Wei+Zeng%2C+Nan+Tang%2C+and+Yuyu+Luo%2E+2025%2E+nvbench+2%2E0%3A+Resolving+ambiguity+in+text%2Dto%2Dvisualization+through+stepwise+reasoning%2E+arXiv+prepri)
19. Mihai Nadăs , , Laura Dios , an, and Andreea Tomescu. 2025. Synthetic data generation using large language models: Advances in text and code. IEEE Access. [find](https://scholar.google.com/scholar?q=Mihai+Nada%CC%86s+%2C+%2C+Laura+Dios+%2C+an%2C+and+Andreea+Tomescu%2E+2025%2E+Synthetic+data+generation+using+large+language+models%3A+Advances+in+text+and+code%2E+IEEE+Access%2E)
20. Simone Papicchio, Luca Cagliero, and Paolo Papotti. 2025. Squab: Evaluating llm robustness to ambiguous and unanswerable questions in semantic parsing. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 17937–17957. [find](https://scholar.google.com/scholar?q=Simone+Papicchio%2C+Luca+Cagliero%2C+and+Paolo+Papotti%2E+2025%2E+Squab%3A+Evaluating+llm+robustness+to+ambiguous+and+unanswerable+questions+in+semantic+parsing%2E+In+Proceedings+of+the+2025+Conference+on+Empiric)
21. Mohammadreza Pourreza and Davood Rafiei. 2023. [Din-sql: Decomposed in-context learning of text-to-sql with self-correction](https://md.research.elamur.ai/free/en/papers/pourreza-2023-din-sql-decomposed-context-learning/). In Advances in Neural Information Processing Systems, volume 36, pages 36339–36348. Curran Associates, Inc.
22. Mohammadreza Pourreza, Ruoxi Sun, Hailong Li, Lesly Miculicich, Tomas Pfister, and Sercan O Arik. 2024. Sql-gen: Bridging the dialect gap for text-to-sql via synthetic data and model merging. arXiv preprint arXiv:2408.12733. — [arXiv:2408.12733](https://arxiv.org/abs/2408.12733) [find](https://scholar.google.com/scholar?q=Mohammadreza+Pourreza%2C+Ruoxi+Sun%2C+Hailong+Li%2C+Lesly+Miculicich%2C+Tomas+Pfister%2C+and+Sercan+O+Arik%2E+2024%2E+Sql%2Dgen%3A+Bridging+the+dialect+gap+for+text%2Dto%2Dsql+via+synthetic+data+and+model+merging%2E+arXiv+pr)
23. Irina Saparina and Mirella Lapata. 2024. Ambrosia: A benchmark for parsing ambiguous questions into database queries. Advances in Neural Information Processing Systems, 37:90600–90628. [find](https://scholar.google.com/scholar?q=Irina+Saparina+and+Mirella+Lapata%2E+2024%2E+Ambrosia%3A+A+benchmark+for+parsing+ambiguous+questions+into+database+queries%2E+Advances+in+Neural+Information+Processing+Systems%2C+37%3A90600%E2%80%9390628%2E)
24. Irina Saparina and Mirella Lapata. 2025. Disambiguate first parse later: Generating interpretations for ambiguity resolution in semantic parsing. arXiv preprint arXiv:2502.18448. [find](https://scholar.google.com/scholar?q=Irina+Saparina+and+Mirella+Lapata%2E+2025%2E+Disambiguate+first+parse+later%3A+Generating+interpretations+for+ambiguity+resolution+in+semantic+parsing%2E+arXiv+preprint+arXiv%3A2502%2E18448%2E)
25. Liang Shi, Zhengju Tang, Nan Zhang, Xiaotong Zhang, and Zhi Yang. 2025. A survey on employing large language models for text-to-sql tasks. ACM Computing Surveys, 58(2):1–37. [find](https://scholar.google.com/scholar?q=Liang+Shi%2C+Zhengju+Tang%2C+Nan+Zhang%2C+Xiaotong+Zhang%2C+and+Zhi+Yang%2E+2025%2E+A+survey+on+employing+large+language+models+for+text%2Dto%2Dsql+tasks%2E+ACM+Computing+Surveys%2C+58%282%29%3A1%E2%80%9337%2E)
26. Shayan Talaei, Mohammadreza Pourreza, Yu-Chen Chang, Azalia Mirhoseini, and Amin Saberi. 2024. Chess: Contextual harnessing for efficient sql synthesis. arXiv preprint arXiv:2405.16755. — [arXiv:2405.16755](https://arxiv.org/abs/2405.16755) [find](https://scholar.google.com/scholar?q=Shayan+Talaei%2C+Mohammadreza+Pourreza%2C+Yu%2DChen+Chang%2C+Azalia+Mirhoseini%2C+and+Amin+Saberi%2E+2024%2E+Chess%3A+Contextual+harnessing+for+efficient+sql+synthesis%2E+arXiv+preprint+arXiv%3A2405%2E16755%2E+%E2%80%94+%5BarXiv%3A2405%2E)
27. Kapil Vaidya, Abishek Sankararaman, Jialin Ding, Chuan Lei, Xiao Qin, Balakrishnan Narayanaswamy, and Tim Kraska. 2025. Odin: A nl2sql recommender to handle schema ambiguity. arXiv preprint arXiv:2505.19302. — [arXiv:2505.19302](https://arxiv.org/abs/2505.19302) [find](https://scholar.google.com/scholar?q=Kapil+Vaidya%2C+Abishek+Sankararaman%2C+Jialin+Ding%2C+Chuan+Lei%2C+Xiao+Qin%2C+Balakrishnan+Narayanaswamy%2C+and+Tim+Kraska%2E+2025%2E+Odin%3A+A+nl2sql+recommender+to+handle+schema+ambiguity%2E+arXiv+preprint+arXiv%3A2505)
28. Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. 2024. [Replacing judges with juries: Evaluating llm generations with a panel of diverse models](https://md.research.elamur.ai/free/en/papers/verga-2024-replacing-judges-juries-evaluating-llm/). arXiv preprint arXiv:2404.18796.
29. Bing Wang, Yan Gao, Zhoujun Li, and Jian-Guang Lou. 2023. Know what i don’t know: Handling ambiguous and unknown questions for text-to-sql. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5701–5714. [find](https://scholar.google.com/scholar?q=Bing+Wang%2C+Yan+Gao%2C+Zhoujun+Li%2C+and+Jian%2DGuang+Lou%2E+2023%2E+Know+what+i+don%E2%80%99t+know%3A+Handling+ambiguous+and+unknown+questions+for+text%2Dto%2Dsql%2E+In+Findings+of+the+Association+for+Computational+Linguistics)
30. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. [Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task](https://md.research.elamur.ai/free/en/papers/yu-2018-spider-large-scale-human-labeled/). In Proceedings of the 2018…
31. Fuheng Zhao, Shaleen Deep, Fotis Psallidas, Avrilia Floratou, Divyakant Agrawal, and Amr El Abbadi. 2024. Sphinteract: Resolving ambiguities in nl2sql through user interaction. Proceedings of the VLDB Endowment, 18(4):1145–1158. [find](https://scholar.google.com/scholar?q=Fuheng+Zhao%2C+Shaleen+Deep%2C+Fotis+Psallidas%2C+Avrilia+Floratou%2C+Divyakant+Agrawal%2C+and+Amr+El+Abbadi%2E+2024%2E+Sphinteract%3A+Resolving+ambiguities+in+nl2sql+through+user+interaction%2E+Proceedings+of+the+VLDB)

*Updated 1 November 2026*