---
title: "PRACTIQ — paper by Dong et al. (2025) — summary and ideas"
description: "Most text-to-SQL benchmarks assume every user question is clear and can be… 60% accuracy. 22 references, cited by 9 papers here."
authors: ["Mingwen Dong", "Nischal Ashok Kumar", "Yiqun Hu", "Anuj Chauhan", "Chung-Wei Hang", "Shuaichen Chang", "Lin Pan", "Wuwei Lan", "Henghui Zhu", "Jiarong Jiang", "Patrick Ng", "Zhiguo Wang"]
year: 2025
venue: "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)"
source: "https://aclanthology.org/2025.naacl-long.13/"
doi: "10.18653/v1/2025.naacl-long.13"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/dong-2025-practiq-practical-conversational-text-sql/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/dong-2025-practiq-practical-conversational-text-sql/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/6/) / PRACTIQ: A Practical Conversational Text-to-SQL dataset with Ambiguous and Unanswerable Queries

# PRACTIQ: A Practical Conversational Text-to-SQL dataset with Ambiguous and Unanswerable Queries

**Authors:** Mingwen Dong, Nischal Ashok Kumar, [Yiqun Hu](https://md.research.elamur.ai/free/en/authors/yiqun-hu/), Anuj Chauhan, [Chung-Wei Hang](https://md.research.elamur.ai/free/en/authors/chung-wei-hang/), Shuaichen Chang, Lin Pan, [Wuwei Lan](https://md.research.elamur.ai/free/en/authors/wuwei-lan/), [Henghui Zhu](https://md.research.elamur.ai/free/en/authors/henghui-zhu/), Jiarong Jiang, [Patrick Ng](https://md.research.elamur.ai/free/en/authors/patrick-ng/) and [Zhiguo Wang](https://md.research.elamur.ai/free/en/authors/zhiguo-wang/) · **Year:** 2025 · **Venue:** [Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2025](https://md.research.elamur.ai/free/en/venues/naacl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.naacl-long.13/) · [PDF](https://aclanthology.org/2025.naacl-long.13.pdf) · [DOI 10.18653/v1/2025.naacl-long.13](https://doi.org/10.18653/v1/2025.naacl-long.13) · [OpenAlex](https://openalex.org/W4411119602)

**Citations:** 6 citations (OpenAlex, 2026-10-14) · [9](https://md.research.elamur.ai/free/en/papers/dong-2025-practiq-practical-conversational-text-sql/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most text-to-SQL benchmarks assume every user question is clear and can be answered from the database. Real users, however, often ask questions that allow several readings or that request data the database does not hold. PRACTIQ fills this gap with a conversational text-to-SQL dataset of about 2,800 dialogues built from Spider. The authors reviewed existing datasets (Spider, BIRD, SParC) and defined four ambiguous and four unanswerable question types. They then created examples by parsing SQL queries and modifying database schemas, and used an LLM to turn each case into a four-turn exchange: question, clarification request, user reply, and final SQL with a natural-language explanation of the results. For some ambiguous types, they also wrote "helpful" SQL answers that cover all interpretations instead of asking the user. Human annotation confirmed the data is of high quality. Prompt-based baselines handle two tasks, category classification and SQL prediction with DIN-SQL. They show that current LLMs still struggle, which matters for deploying practical database assistants.

## Key points
- **Categories:**
  - *Ambiguous:* SELECT Column, Values Within Column, WHERE Column, Filter Criteria.
  - *Unanswerable:* a SELECT column absent from the schema, a WHERE column absent from the schema, a filter value absent from the data, and a join the schema does not support.
- **Source data:** Only Spider `train_spider.json` and `dev.json` were used. The generation framework can be adapted to WikiSQL, BIRD, or other single-turn datasets.
- **Category classification:** Most LLMs score below 60% accuracy. Claude 3.5 Sonnet is the only strong exception. Llama-3.1 70B produced repetitive output when given few-shot examples.
- **SQL prediction (DIN-SQL):**
  - On ambiguous/unanswerable questions, Mixtral-large-v2 and Claude 3.5 Sonnet are the best, at roughly 71–72%.
  - On the answerable Spider dev questions, Claude 3.5 Sonnet reaches 79.21% and Llama-3.1 70B reaches 76.31%.
  - On ambiguous/unanswerable questions, Llama-3.1 70B drops to 67.58%.
- **Release:** Code is available at github.com/amazon-science/conversational-ambiguous-unanswerable-text2sql. Fine-tuning open-source models on the generated data is left for future work.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Massachusetts Amherst](https://md.research.elamur.ai/free/en/universities/university-of-massachusetts-amherst/) — University, Amherst Center, United States · 106 works · 66,692 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (3): [🗨️ Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) · 624 works · [🧾 Semantic parsing & meaning representation](https://md.research.elamur.ai/free/en/topics/semantic-parsing/) · 185 works · [💻 Code & programming languages](https://md.research.elamur.ai/free/en/topics/code-and-programming/) · 129 works

## Cite

Cite as: Mingwen Dong, Nischal Ashok Kumar, Yiqun Hu, Anuj Chauhan, Chung-Wei Hang, Shuaichen Chang, Lin Pan, Wuwei Lan, Henghui Zhu, Jiarong Jiang, Patrick Ng and Zhiguo Wang. 2025. PRACTIQ: A Practical Conversational Text-to-SQL dataset with Ambiguous and Unanswerable Queries. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{dong2025practiq,
  title = {{PRACTIQ: A Practical Conversational Text-to-SQL dataset with Ambiguous and Unanswerable Queries}},
  author = {Mingwen Dong, Nischal Ashok Kumar, Yiqun Hu, Anuj Chauhan, Chung-Wei Hang, Shuaichen Chang, Lin Pan, Wuwei Lan, Henghui Zhu, Jiarong Jiang, Patrick Ng and Zhiguo Wang},
  booktitle = {Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.13},
  url = {https://doi.org/10.18653/v1/2025.naacl-long.13}
}
```

- [← previous Ontology-Guided Reverse Thinking Makes Large Language Models Stronger on Knowledge Graph Question Answering](https://md.research.elamur.ai/free/en/papers/liu-2025-ontology-guided-reverse-thinking-makes/)
- [next → Rethinking Reflection in Pre-Training](https://md.research.elamur.ai/free/en/papers/shah-2025-rethinking-reflection-pre-training/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/6/)

## References

22 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Adithya Bhaskar, Tushar Tomar, Ashutosh Sathe, and Sunita Sarawagi. 2023. Benchmarking and improving text-to-SQL generation under ambiguity. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7053– 7074, Singapore. Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Adithya+Bhaskar%2C+Tushar+Tomar%2C+Ashutosh+Sathe%2C+and+Sunita+Sarawagi%2E+2023%2E+Benchmarking+and+improving+text%2Dto%2DSQL+generation+under+ambiguity%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+Methods+)
2. Chia-Hsuan Lee, Oleksandr Polozov, and Matthew Richardson. 2021. KaggleDBQA: Realistic evaluation of text-to-SQL parsers. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: L… [find](https://scholar.google.com/scholar?q=Chia%2DHsuan+Lee%2C+Oleksandr+Polozov%2C+and+Matthew+Richardson%2E+2021%2E+KaggleDBQA%3A+Realistic+evaluation+of+text%2Dto%2DSQL+parsers%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational)
3. Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al. 2024. [Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls](https://md.research.elamur.ai/free/en/papers/li-2023-llm-already-serve-database-interface/). Advances in Neural Information Processing Systems, 36.
4. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. [Sparks of artificial general intelligence: Early experiments with gpt-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/). arXiv preprint arXiv:2303.12712.
5. Xi Victoria Lin, Richard Socher, and Caiming Xiong. 2020. [Bridging textual and tabular data for cross-domain text-to-SQL semantic parsing](https://md.research.elamur.ai/free/en/papers/lin-2020-bridging-textual-tabular-data-cross/). In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4870–4888, Online. Association for Computational Linguistics.
6. Yang Deng, Lizi Liao, Liang Chen, Hongru Wang, Wenqiang Lei, and Tat-Seng Chua. 2023. [Prompting and evaluating large language models for proactive dialogues: Clarification, target-guided, and non-collaboration](https://md.research.elamur.ai/free/en/papers/deng-2023-prompting-evaluating-large-language-models/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10602–106…
7. Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, et al. 2024. [Best practices and lessons learned on synthetic data](https://md.research.elamur.ai/free/en/papers/liu-2024-best-practices-lessons-learned-synthetic/) for language models. arXiv preprint arXiv:2404.07503.
8. Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. 2018. [Improving text-to-SQL evaluation methodology](https://md.research.elamur.ai/free/en/papers/dollak-2018-improving-text-sql-evaluation-methodology/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages…
9. Xinyi He, Mengyu Zhou, Xinrun Xu, Xiaojun Ma, Rui Ding, Lun Du, Yan Gao, Ran Jia, Xu Chen, Shi Han, et al. 2024. Text2analysis: A benchmark of table question answering with advanced data analysis and unclear queries. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages… [find](https://scholar.google.com/scholar?q=Xinyi+He%2C+Mengyu+Zhou%2C+Xinrun+Xu%2C+Xiaojun+Ma%2C+Rui+Ding%2C+Lun+Du%2C+Yan+Gao%2C+Ran+Jia%2C+Xu+Chen%2C+Shi+Han%2C+et+al%2E+2024%2E+Text2analysis%3A+A+benchmark+of+table+question+answering+with+advanced+data+analysis+and+)
10. Yiqun Hu, Yiyun Zhao, Jiarong Jiang, Wuwei Lan, Henghui Zhu, Anuj Chauhan, Alexander Hanbo Li, Mohammadreza Pourreza and Davood Rafiei. 2024. [Din-sql: Decomposed in-context learning of text-to-sql with self-correction](https://md.research.elamur.ai/free/en/papers/pourreza-2023-din-sql-decomposed-context-learning/). Advances in Neural Information Processing Systems, 36.
11. Irina Saparina and Mirella Lapata. 2024. Ambrosia: A benchmark for parsing ambiguous questions into database queries. arXiv preprint arXiv:2406.19073. — [arXiv:2406.19073](https://arxiv.org/abs/2406.19073) [find](https://scholar.google.com/scholar?q=Irina+Saparina+and+Mirella+Lapata%2E+2024%2E+Ambrosia%3A+A+benchmark+for+parsing+ambiguous+questions+into+database+queries%2E+arXiv+preprint+arXiv%3A2406%2E19073%2E+%E2%80%94+%5BarXiv%3A2406%2E19073%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2406%2E1)
12. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971.
13. Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020. [RAT-SQL: Relation-aware schema encoding and linking for text-to-SQL parsers](https://md.research.elamur.ai/free/en/papers/wang-2020-rat-sql-relation-aware-schema/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7567–7578, Online. Association for…
14. Bing Wang, Yan Gao, Zhoujun Li, and Jian-Guang Lou. 2023a. Know what i don’t know: Handling ambiguous and unknown questions for text-to-sql. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5701–5714. [find](https://scholar.google.com/scholar?q=Bing+Wang%2C+Yan+Gao%2C+Zhoujun+Li%2C+and+Jian%2DGuang+Lou%2E+2023a%2E+Know+what+i+don%E2%80%99t+know%3A+Handling+ambiguous+and+unknown+questions+for+text%2Dto%2Dsql%2E+In+Findings+of+the+Association+for+Computational+Linguistic)
15. Junda Wang, Zonghai Yao, Avijit Mitra, Samuel Osebe, Zhichao Yang, and Hong Yu. 2023b. UMASS_BioNLP at MEDIQA-chat 2023: Can LLMs generate high-quality synthetic note-oriented doctor-patient conversations? In Proceedings of the 5th Clinical Natural Language Processing Workshop, pages 460–471, Toront… [find](https://scholar.google.com/scholar?q=Junda+Wang%2C+Zonghai+Yao%2C+Avijit+Mitra%2C+Samuel+Osebe%2C+Zhichao+Yang%2C+and+Hong+Yu%2E+2023b%2E+UMASS%5FBioNLP+at+MEDIQA%2Dchat+2023%3A+Can+LLMs+generate+high%2Dquality+synthetic+note%2Doriented+doctor%2Dpatient+conversat)
16. Victor Zhong, Caiming Xiong, and Richard Socher. 2017. [Seq2sql: Generating structured queries from natural language using reinforcement learning](https://md.research.elamur.ai/free/en/papers/zhong-2017-seq2sql-generating-structured-queries-natural/). arXiv preprint arXiv:1709.00103.
17. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in Neural Information Processing Systems, 35:24824–24837.
18. Tao Yu, Rui Zhang, Heyang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, Youxuan Jiang, Michihiro Yasunaga, Sungrok Shim, Tao Chen, Alexander Fabbri, Zifan Li, Luyao Chen, Yuwen Zhang, Shreya Dixit, Vincent Zhang, Caiming Xiong, Richard Socher, Walter Lasecki, a… — [CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases](https://md.research.elamur.ai/free/en/papers/yu-2019-cosql-conversational-text-sql-challenge/)
19. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. [Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task](https://md.research.elamur.ai/free/en/papers/yu-2018-spider-large-scale-human-labeled/). In Proceedings of the 2018…
20. Tao Yu, Rui Zhang, Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li, Irene Li Heyang Er, Bo Pang, Tao Chen, Emily Ji, Shreya Dixit, David Proctor, Sungrok Shim, Vincent Zhang Jonathan Kraft, Caiming Xiong, Richard Socher, and Dragomir Radev. 2019b. [Sparc: Cross-domain semantic parsing in c…](https://md.research.elamur.ai/free/en/papers/yu-2019-sparc-cross-domain-semantic-parsing/)
21. Xiaoying Zhang, Baolin Peng, Kun Li, Jingyan Zhou, and Helen Meng. 2023. SGP-TOD: Building task bots effortlessly via schema-guided LLM prompting. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13348–13369, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xiaoying+Zhang%2C+Baolin+Peng%2C+Kun+Li%2C+Jingyan+Zhou%2C+and+Helen+Meng%2E+2023%2E+SGP%2DTOD%3A+Building+task+bots+effortlessly+via+schema%2Dguided+LLM+prompting%2E+In+Findings+of+the+Association+for+Computational+Ling)
22. Yusen Zhang, Xiangyu Dong, Shuaichen Chang, Tao Yu, Peng Shi, and Rui Zhang. 2020. Did you ask a good [find](https://scholar.google.com/scholar?q=Yusen+Zhang%2C+Xiangyu+Dong%2C+Shuaichen+Chang%2C+Tao+Yu%2C+Peng+Shi%2C+and+Rui+Zhang%2E+2020%2E+Did+you+ask+a+good)

*Updated 1 November 2026*