---
title: "SPENCE — paper by Safarzadeh et al. (2026)"
description: "The authors propose SPENCE, a black-box probing framework that tests whether accuracy holds when questions are… 27 references."
authors: ["Mohammadtaher Safarzadeh", "Hitesh Laxmichand Patel", "Afshin Oroojlooy", "Graham Horwood", "Dan Roth"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.926/"
doi: "10.18653/v1/2026.acl-long.926"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/safarzadeh-2026-spence-syntactic-probe-detecting-contamination/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/safarzadeh-2026-spence-syntactic-probe-detecting-contamination/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/43/) / SPENCE: A Syntactic Probe for Detecting Contamination in NL2SQL Benchmarks

# SPENCE: A Syntactic Probe for Detecting Contamination in NL2SQL Benchmarks

**Authors:** [Mohammadtaher Safarzadeh](https://md.research.elamur.ai/free/en/authors/mohammadtaher-safarzadeh/), [Hitesh Laxmichand Patel](https://md.research.elamur.ai/free/en/authors/hitesh-laxmichand-patel/), Afshin Oroojlooy, [Graham Horwood](https://md.research.elamur.ai/free/en/authors/graham-horwood/) and [Dan Roth](https://md.research.elamur.ai/free/en/authors/dan-roth/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.926/) · [PDF](https://aclanthology.org/2026.acl-long.926.pdf) · [DOI 10.18653/v1/2026.acl-long.926](https://doi.org/10.18653/v1/2026.acl-long.926) · [OpenAlex](https://openalex.org/W7166843688)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Leaderboard scores of LLMs on NL2SQL benchmarks may be overstated if models saw benchmark items, or close variants, during training. The authors propose SPENCE, a black-box probing framework that tests whether accuracy holds when questions are reworded with growing structural distance from the original. For the development splits of Spider, SParC, CoSQL and BIRD, GPT-5 receives each question with its schema and gold SQL and produces ten paraphrases. These are parsed into dependency trees with spaCy, ordered by tree edit distance (TED) to the original, and filtered by embedding similarity and manual checks. Several general-purpose LLMs then generate SQL, which is scored by execution, and Kendall's τ measures how strongly accuracy falls as rank increases. Accuracy drops the most on the oldest benchmark, Spider, and much less on the 2023 BIRD dataset. This pattern follows release dates and is consistent with contamination. The work suggests that syntactic probing should accompany standard benchmark reporting.

## Key points
- **Benchmarks and release dates:** the oldest is Spider, published in September 2018, followed by SParC in June 2019, then CoSQL in October 2019, with BIRD arriving last in May 2023. In the dialogue datasets SParC and CoSQL, only the final user turn is paraphrased.
- **Paraphrase ranking and filtering:**
  - Paraphrases are ranked 1–10 by TED over spaCy (v3.7) dependency parses.
  - Filtering uses E5-base-v2 cosine similarity.
  - Human review of 100 Spider items at ranks 1, 5 and 10 found over 85% of paraphrases meaning-preserving, with no trend across ranks.
- **Models and statistics:** the evaluated models are GPT-4o-mini, GPT-OSS-120B, GPT-OSS-20B, LLaMA-3.3-70B, LLaMA-3.1-405B and o3-mini. Rank sensitivity is measured with Kendall's τ and bootstrap confidence intervals.
- **Robustness check:** replacing GPT-5 with LLaMA-4 Maverick as the paraphrase generator on Spider gives similar declines from the original queries to rank 10:
  - gpt-oss-20B: 77.05 → 69.11
  - LLaMA-3.3-70B: 76.41 → 68.42
  - o3-mini: 76.37 → 70.20
- **Stated limitations:**
  - The temporal trend is correlational only.
  - SQL-specialized systems such as OmniSQL and Arctic-Text2SQL-R1 were not tested.
  - Execution accuracy mixes schema-linking errors with SQL-logic errors.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

## Cite

Cite as: Mohammadtaher Safarzadeh, Hitesh Laxmichand Patel, Afshin Oroojlooy, Graham Horwood and Dan Roth. 2026. SPENCE: A Syntactic Probe for Detecting Contamination in NL2SQL Benchmarks. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{safarzadeh2026spence,
  title = {{SPENCE: A Syntactic Probe for Detecting Contamination in NL2SQL Benchmarks}},
  author = {Mohammadtaher Safarzadeh, Hitesh Laxmichand Patel, Afshin Oroojlooy, Graham Horwood and Dan Roth},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.926},
  url = {https://doi.org/10.18653/v1/2026.acl-long.926}
}
```

- [← previous SpeechLM for Automatic Speech Recognition in Low-resource Languages](https://md.research.elamur.ai/free/en/papers/riyadh-2026-speechlm-automatic-speech-recognition-low/)
- [next → SpidR-Adapt: A Universal Speech Representation Model for Few-Shot Adaptation](https://md.research.elamur.ai/free/en/papers/luthra-2026-spidr-adapt-universal-speech-representation/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/43/)

## References

27 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). In Advances in Neural Information Processing Systems (NeurIPS).
2. Yuxing Cheng, Yi Chang, and Yuan Wu. 2025. A survey on data contamination for large language models. Preprint, arXiv:2502.14425. — [arXiv:2502.14425](https://arxiv.org/abs/2502.14425) [find](https://scholar.google.com/scholar?q=Yuxing+Cheng%2C+Yi+Chang%2C+and+Yuan+Wu%2E+2025%2E+A+survey+on+data+contamination+for+large+language+models%2E+Preprint%2C+arXiv%3A2502%2E14425%2E+%E2%80%94+%5BarXiv%3A2502%2E14425%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2502%2E14425%29)
3. Jasper Dekoninck, Mark Niklas Müller, and Martin Vechev. 2024. Constat: Performance-based contamination detection in large language models. Preprint, arXiv:2405.16281. — [arXiv:2405.16281](https://arxiv.org/abs/2405.16281) [find](https://scholar.google.com/scholar?q=Jasper+Dekoninck%2C+Mark+Niklas+M%C3%BCller%2C+and+Martin+Vechev%2E+2024%2E+Constat%3A+Performance%2Dbased+contamination+detection+in+large+language+models%2E+Preprint%2C+arXiv%3A2405%2E16281%2E+%E2%80%94+%5BarXiv%3A2405%2E16281%5D%28https%3A%2F%2Farx)
4. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
5. Shahriar Golchin and Mihai Surdeanu. 2023. Data contamination quiz: A tool to detect and estimate contamination in large language models. arXiv preprint arXiv:2311.06233. [find](https://scholar.google.com/scholar?q=Shahriar+Golchin+and+Mihai+Surdeanu%2E+2023%2E+Data+contamination+quiz%3A+A+tool+to+detect+and+estimate+contamination+in+large+language+models%2E+arXiv+preprint+arXiv%3A2311%2E06233%2E)
6. Shahriar Golchin and Mihai Surdeanu. 2024. [Time travel in llms: Tracing data contamination in large language models](https://md.research.elamur.ai/free/en/papers/golchin-2023-time-travel-llms-tracing-data/). Preprint, arXiv:2308.08493.
7. Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C.C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023. [Can llm already serve as a database interface?](https://md.research.elamur.ai/free/en/papers/li-2023-llm-already-serve-database-interface/) a big bench for large-scale database…
8. Yucheng Li. 2023. An open source data contamination report for large language models. arXiv preprint arXiv:2310.17589. [find](https://scholar.google.com/scholar?q=Yucheng+Li%2E+2023%2E+An+open+source+data+contamination+report+for+large+language+models%2E+arXiv+preprint+arXiv%3A2310%2E17589%2E)
9. Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schölkopf, Mrinmaya Sachan, and Taylor Berg-Kirkpatrick. 2023. [Membership inference attacks against language models via neighbourhood comparison](https://md.research.elamur.ai/free/en/papers/mattern-2023-membership-inference-attacks-against-language/). In Findings of the Association for Computational Linguistics: ACL 2023.
10. OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. 2025. [gpt-oss-120b gpt-oss-20b model card](https://md.research.elamur.ai/free/en/papers/agarwal-2025-gpt-oss-120b-gpt-oss/). Preprint, arXiv:2508.10925.
11. OpenAI. 2024. Gpt-4o mini: advancing cost-efficient intelligence. Available at [https://openai.com/](https://openai.com/) index/gpt-4o-mini/. [↗](https://openai.com/)
12. OpenAI. 2025a. Openai gpt-5: Smartest, fastest, most useful model yet, with built-in thinking that puts expert-level intelligence in everyone’s hands. Available at [https://openai.com/index/](https://openai.com/index/) introducing-gpt-5/. [↗](https://openai.com/index/)
13. OpenAI. 2025b. Openai o3-mini: Pushing the frontier of cost-effective reasoning. Available at [https://openai.com/index/openai-o3-mini/](https://openai.com/index/openai-o3-mini/). [↗](https://openai.com/index/openai-o3-mini/)
14. Yonatan Oren, Nicole Meister, Niladri S. Chatterji, Faisal Ladhak, and Tatsunori B. Hashimoto. 2023. Proving test set contamination in black box language models. arXiv preprint arXiv:2310.17623. [find](https://scholar.google.com/scholar?q=Yonatan+Oren%2C+Nicole+Meister%2C+Niladri+S%2E+Chatterji%2C+Faisal+Ladhak%2C+and+Tatsunori+B%2E+Hashimoto%2E+2023%2E+Proving+test+set+contamination+in+black+box+language+models%2E+arXiv+preprint+arXiv%3A2310%2E17623%2E)
15. Federico Ranaldi, Elena Sofia Ruzzetti, Dario Onorati, Leonardo Ranaldi, Cristina Giannone, Andrea Favalli, Raniero Romagnoli, and Fabio Massimo Zanzotto. 2024. Investigating the impact of data contamination of large language models in text-to-sql translation. In Findings of the Association for Comp… [find](https://scholar.google.com/scholar?q=Federico+Ranaldi%2C+Elena+Sofia+Ruzzetti%2C+Dario+Onorati%2C+Leonardo+Ranaldi%2C+Cristina+Giannone%2C+Andrea+Favalli%2C+Raniero+Romagnoli%2C+and+Fabio+Massimo+Zanzotto%2E+2024%2E+Investigating+the+impact+of+data+contam)
16. Mathieu Ravaut, Bosheng Ding, Fangkai Jiao, Hailin Chen, Xingxuan Li, Ruochen Zhao, Chengwei Qin, Caiming Xiong, and Shafiq Joty. 2025. A comprehensive survey of contamination detection methods in large language models. Preprint, arXiv:2404.00699. [find](https://scholar.google.com/scholar?q=Mathieu+Ravaut%2C+Bosheng+Ding%2C+Fangkai+Jiao%2C+Hailin+Chen%2C+Xingxuan+Li%2C+Ruochen+Zhao%2C+Chengwei+Qin%2C+Caiming+Xiong%2C+and+Shafiq+Joty%2E+2025%2E+A+comprehensive+survey+of+contamination+detection+methods+in+lar)
17. Weijia Shi. 2023. Detect-pretrain-code-contamination. [https://github.com/swj0419/](https://github.com/swj0419/) detect-pretrain-code-contamination. [↗](https://github.com/swj0419/)
18. Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2023. Detecting pretraining data from large language models. arXiv preprint arXiv:2310.16789. [find](https://scholar.google.com/scholar?q=Weijia+Shi%2C+Anirudh+Ajith%2C+Mengzhou+Xia%2C+Yangsibo+Huang%2C+Daogao+Liu%2C+Terra+Blevins%2C+Danqi+Chen%2C+and+Luke+Zettlemoyer%2E+2023%2E+Detecting+pretraining+data+from+large+language+models%2E+arXiv+preprint+arXiv%3A)
19. Jyotika Singh. 2023. Natural Language Processing in the Real World: Text Processing, Analytics, and Classification. Chapman and Hall/CRC. [find](https://scholar.google.com/scholar?q=Jyotika+Singh%2E+2023%2E+Natural+Language+Processing+in+the+Real+World%3A+Text+Processing%2C+Analytics%2C+and+Classification%2E+Chapman+and+Hall%2FCRC%2E)
20. Jyotika Singh, Weiyi Sun, Amit Agarwal, Viji Krishnamurthy, Yassine Benajiba, Sujith Ravi, and Dan Roth. 2025. Can LLMs narrate tabular data? an evaluation framework for natural language representations of text-to-SQL system outputs. In Proceedings of the 2025 Conference on Empirical Methods in Natu… [find](https://scholar.google.com/scholar?q=Jyotika+Singh%2C+Weiyi+Sun%2C+Amit+Agarwal%2C+Viji+Krishnamurthy%2C+Yassine+Benajiba%2C+Sujith+Ravi%2C+and+Dan+Roth%2E+2025%2E+Can+LLMs+narrate+tabular+data%3F+an+evaluation+framework+for+natural+language+representatio)
21. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and et al. 2023. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288.
22. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2024. [Text embeddings by weakly-supervised contrastive pre-training](https://md.research.elamur.ai/free/en/papers/wang-2022-text-embeddings-weakly-supervised-contrastive/). Preprint, arXiv:2212.03533.
23. Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. 2024. Benchmarking benchmark leakage in large language models. arXiv preprint arXiv:2404.18824. [find](https://scholar.google.com/scholar?q=Ruijie+Xu%2C+Zengzhi+Wang%2C+Run%2DZe+Fan%2C+and+Pengfei+Liu%2E+2024%2E+Benchmarking+benchmark+leakage+in+large+language+models%2E+arXiv+preprint+arXiv%3A2404%2E18824%2E)
24. Tao Yu, Rui Zhang, Heyang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, Youxuan Jiang, Michihiro Yasunaga, Sungrok Shim, Tao Chen, Alexander Fabbri, Zifan Li, Luyao Chen, Yuwen Zhang, Shreya Dixit, Vincent Zhang, Caiming Xiong, Richard Socher, Walter S. Lasecki… — [CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases](https://md.research.elamur.ai/free/en/papers/yu-2019-cosql-conversational-text-sql-challenge/)
25. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. [Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task](https://md.research.elamur.ai/free/en/papers/yu-2018-spider-large-scale-human-labeled/). In Proceedings of the 2018…
26. Tao Yu, Rui Zhang, Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li, Heyang Er, Irene Li, Bo Pang, Tao Chen, Emily Ji, Shreya Dixit, David Proctor, Sungrok Shim, Jonathan Kraft, Vincent Zhang, Caiming Xiong, Richard Socher, and Dragomir Radev. 2019b. [SParC: Cross-domain semantic parsing in…](https://md.research.elamur.ai/free/en/papers/yu-2019-sparc-cross-domain-semantic-parsing/)
27. Kaizhong Zhang and Dennis Shasha. 1989. Simple fast algorithms for the editing distance between trees and related problems. SIAM Journal on Computing, 18(6):1245–1262. [find](https://scholar.google.com/scholar?q=Kaizhong+Zhang+and+Dennis+Shasha%2E+1989%2E+Simple+fast+algorithms+for+the+editing+distance+between+trees+and+related+problems%2E+SIAM+Journal+on+Computing%2C+18%286%29%3A1245%E2%80%931262%2E)

*Updated 1 November 2026*