---
title: "Are LLM Benchmarks Already Contaminated? A Systematic"
description: "The paper proposes a Contamination Transparency Card (CTC) for benchmark and model releases. 56 references."
authors: ["Erfan Nourbakhsh", "Mohammad Sadegh Sirjani", "Amir Mousavi", "Khoa Nguyen", "John Quarles", "Mimi Xie", "Rocky Slavin"]
year: 2026
venue: "Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)"
source: "https://aclanthology.org/2026.gem-main.50/"
doi: "10.18653/v1/2026.gem-main.50"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/nourbakhsh-2026-are-llm-benchmarks-already-contaminated/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nourbakhsh-2026-are-llm-benchmarks-already-contaminated/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/23/) / Are LLM Benchmarks Already Contaminated? A Systematic Review of Contamination Detection Methods

# Are LLM Benchmarks Already Contaminated? A Systematic Review of Contamination Detection Methods

**Authors:** [Erfan Nourbakhsh](https://md.research.elamur.ai/free/en/authors/erfan-nourbakhsh/), Mohammad Sadegh Sirjani, Amir Mousavi, [Khoa Nguyen](https://md.research.elamur.ai/free/en/authors/khoa-nguyen/), John Quarles, Mimi Xie and [Rocky Slavin](https://md.research.elamur.ai/free/en/authors/rocky-slavin/) · **Year:** 2026 · **Venue:** [Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)](https://md.research.elamur.ai/free/en/venues/gem/) ([GEM 2026](https://md.research.elamur.ai/free/en/venues/gem/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.gem-main.50/) · [PDF](https://aclanthology.org/2026.gem-main.50.pdf) · [DOI 10.18653/v1/2026.gem-main.50](https://doi.org/10.18653/v1/2026.gem-main.50) · [OpenAlex](https://openalex.org/W7166876690)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Benchmark scores for large language models may be inflated because items from public test sets can leak into web-scale training data. The authors conducted a systematic literature review of 55 studies from 2020 through late 2025, following a PRISMA-aligned protocol. They sort contamination into four tiers (Exact, Syntactic, Semantic, Task-Level; T1–T4). They compare five detector families: string matching, likelihood-based scoring, membership inference, LLM-prompted probing, and benchmark auditing. They also map reported evidence for MMLU, GSM8K, HumanEval and HellaSwag by metric type rather than merging incompatible numbers. Each study is graded with a custom risk-of-bias rubric (RoB-LLM-Contam). The central finding is that no detector is reliable across tiers, model-access levels and training stages. Contamination introduced during instruction tuning is essentially invisible to current methods, and auditing of RL/post-training is only starting to develop. The paper proposes a Contamination Transparency Card (CTC) for benchmark and model releases. This matters for anyone relying on leaderboards or building evaluation sets, because reported gains may partly reflect memorisation.

## Key points
- Reported performance inflation ranges from roughly 6% to 40%. These figures depend on the benchmark and setting and are not directly comparable across methods.
- On the Open LLM Leaderboard, ConStat (Dekoninck et al.) judged every one of the three leading 7B models to be contaminated, with effects above 10% on several benchmarks. For InternLM-2-Math-7B on GSM8K, the primary estimate is 27.15%, rising to 40% under alternate settings.
- Other signals from the reviewed studies:
  - Inference-Time Decontamination (ITD) suggests inflation of up to 22.9% on GSM8K and up to 19.0% on MMLU.
  - GPT-4 recovers masked MMLU answer options 57% of the time, against a 25% chance level.
- Contamination can enter at three training stages: pretraining, SFT/IFT, and RLHF. The paper formally distinguishes input-only, input–label, and performance-based contamination.
- The CTC is a five-dimension disclosure card. The authors suggest adoption could be driven by leaderboard operators (Open LLM Leaderboard, HELM), EU/US transparency rules, and venues such as GEM. Multilingual contamination is listed among six open challenges.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [The University of Texas at San Antonio](https://md.research.elamur.ai/free/en/universities/the-university-of-texas-at-san-antonio/) — University, San Antonio, United States · 5 works · 25 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works

## Cite

Cite as: Erfan Nourbakhsh, Mohammad Sadegh Sirjani, Amir Mousavi, Khoa Nguyen, John Quarles, Mimi Xie and Rocky Slavin. 2026. Are LLM Benchmarks Already Contaminated? A Systematic Review of Contamination Detection Methods. Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{nourbakhsh2026are,
  title = {{Are LLM Benchmarks Already Contaminated? A Systematic Review of Contamination Detection Methods}},
  author = {Erfan Nourbakhsh, Mohammad Sadegh Sirjani, Amir Mousavi, Khoa Nguyen, John Quarles, Mimi Xie and Rocky Slavin},
  booktitle = {Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)},
  year = {2026},
  doi = {10.18653/v1/2026.gem-main.50},
  url = {https://doi.org/10.18653/v1/2026.gem-main.50}
}
```

- [← previous Are Large Language Models Economically Viable for Industry Deployment?](https://md.research.elamur.ai/free/en/papers/mohammad-2026-are-large-language-models-economically/)
- [next → Are My Optimized Prompts Compromised? Exploring Vulnerabilities of LLM-based Optimizers](https://md.research.elamur.ai/free/en/papers/zhao-2026-are-my-optimized-prompts-compromised/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/23/)

## References

56 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yuxing Cheng, Yi Chang, and Yuan Wu. 2025. A survey on data contamination for large language models. arXiv preprint arXiv:2502.14425. — [arXiv:2502.14425](https://arxiv.org/abs/2502.14425) [find](https://scholar.google.com/scholar?q=Yuxing+Cheng%2C+Yi+Chang%2C+and+Yuan+Wu%2E+2025%2E+A+survey+on+data+contamination+for+large+language+models%2E+arXiv+preprint+arXiv%3A2502%2E14425%2E+%E2%80%94+%5BarXiv%3A2502%2E14425%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2502%2E14425%29)
2. Simone Balloccu, Patrícia Schmidtová, Mateusz Lango, and Ondrej Dusek. 2024. [Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source LLMs](https://md.research.elamur.ai/free/en/papers/balloccu-2024-leak-cheat-repeat-data-contamination/). In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Paper…
3. Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raff. 2023. [Emergent and predictable memorization in large language models](https://md.research.elamur.ai/free/en/papers/biderman-2023-emergent-predictable-memorization-large-language/). In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23…
4. Jialun Cao, Wuqi Zhang, and Shing-Chi Cheung. 2024. Concerned with data contamination? assessing countermeasures in code language model. arXiv preprint arXiv:2403.16898. — [arXiv:2403.16898](https://arxiv.org/abs/2403.16898) [find](https://scholar.google.com/scholar?q=Jialun+Cao%2C+Wuqi+Zhang%2C+and+Shing%2DChi+Cheung%2E+2024%2E+Concerned+with+data+contamination%3F+assessing+countermeasures+in+code+language+model%2E+arXiv+preprint+arXiv%3A2403%2E16898%2E+%E2%80%94+%5BarXiv%3A2403%2E16898%5D%28https%3A%2F%2Fa)
5. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
6. Jasper Dekoninck, Mark Niklas Mueller, Maximilian Baader, Marc Fischer, and Martin Vechev. 2025. Evading data contamination detection for language models is (too) easy. [find](https://scholar.google.com/scholar?q=Jasper+Dekoninck%2C+Mark+Niklas+Mueller%2C+Maximilian+Baader%2C+Marc+Fischer%2C+and+Martin+Vechev%2E+2025%2E+Evading+data+contamination+detection+for+language+models+is+%28too%29+easy%2E)
7. Jasper Dekoninck, Mark Niklas Müller, and Martin Vechev. 2024. Constat: performance-based contamination detection in large language models. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA. Curran Associates Inc. [find](https://scholar.google.com/scholar?q=Jasper+Dekoninck%2C+Mark+Niklas+M%C3%BCller%2C+and+Martin+Vechev%2E+2024%2E+Constat%3A+performance%2Dbased+contamination+detection+in+large+language+models%2E+In+Proceedings+of+the+38th+International+Conference+on+Neura)
8. Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. 2024. [Investigating data contamination in modern benchmarks for large language models](https://md.research.elamur.ai/free/en/papers/deng-2024-investigating-data-contamination-modern-benchmarks/). In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technol…
9. Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language P… [find](https://scholar.google.com/scholar?q=Jesse+Dodge%2C+Maarten+Sap%2C+Ana+Marasovic%CC%81%2C+William+Agnew%2C+Gabriel+Ilharco%2C+Dirk+Groeneveld%2C+Margaret+Mitchell%2C+and+Matt+Gardner%2E+2021%2E+Documenting+large+webtext+corpora%3A+A+case+study+on+the+colossal+cl)
10. Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. 2024. Generalization or memorization: Data contamination and trustworthy evaluation for large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12039–12050, Bangkok, Thailand. A… [find](https://scholar.google.com/scholar?q=Yihong+Dong%2C+Xue+Jiang%2C+Huanyu+Liu%2C+Zhi+Jin%2C+Bin+Gu%2C+Mengfei+Yang%2C+and+Ge+Li%2E+2024%2E+Generalization+or+memorization%3A+Data+contamination+and+trustworthy+evaluation+for+large+language+models%2E+In+Findings)
11. Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. 2024. [Do membership inference attacks work on large language models?](https://md.research.elamur.ai/free/en/papers/duan-2024-membership-inference-attacks-work-large/) In Conference on Language Modeling (COLM).
12. Wenjie Fu, Huandong Wang, Chen Gao, Guanghua Liu, Yong Li, and Tao Jiang. 2024. Membership inference attacks against fine-tuned large language models via self-prompt calibration. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA… [find](https://scholar.google.com/scholar?q=Wenjie+Fu%2C+Huandong+Wang%2C+Chen+Gao%2C+Guanghua+Liu%2C+Yong+Li%2C+and+Tao+Jiang%2E+2024%2E+Membership+inference+attacks+against+fine%2Dtuned+large+language+models+via+self%2Dprompt+calibration%2E+In+Proceedings+of+the)
13. Yujuan Fu, Ozlem Uzuner, Meliha Yetisgen, and Fei Xia. 2025. Does data contamination detection work (well) for LLMs? a survey and evaluation on detection assumptions. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 5250–5271, Albuquerque, New Mexico. Association for C… [find](https://scholar.google.com/scholar?q=Yujuan+Fu%2C+Ozlem+Uzuner%2C+Meliha+Yetisgen%2C+and+Fei+Xia%2E+2025%2E+Does+data+contamination+detection+work+%28well%29+for+LLMs%3F+a+survey+and+evaluation+on+detection+assumptions%2E+In+Findings+of+the+Association+fo)
14. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2021. Datasheets for datasets. Commun. ACM, 64(12):86–92. [find](https://scholar.google.com/scholar?q=Timnit+Gebru%2C+Jamie+Morgenstern%2C+Briana+Vecchione%2C+Jennifer+Wortman+Vaughan%2C+Hanna+Wallach%2C+Hal+Daum%C3%A9+III%2C+and+Kate+Crawford%2E+2021%2E+Datasheets+for+datasets%2E+Commun%2E+ACM%2C+64%2812%29%3A86%E2%80%9392%2E)
15. Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile Van Krieken, and Pasquale Minervini. 2025. Are we done wit… [find](https://scholar.google.com/scholar?q=Aryo+Pradipta+Gema%2C+Joshua+Ong+Jun+Leang%2C+Giwon+Hong%2C+Alessio+Devoto%2C+Alberto+Carlo+Maria+Mancino%2C+Rohit+Saxena%2C+Xuanli+He%2C+Yu+Zhao%2C+Xiaotang+Du%2C+Mohammad+Reza+Ghasemi+Madani%2C+Claire+Barale%2C+Robert+Mc)
16. Shahriar Golchin and Mihai Surdeanu. 2025. Data contamination quiz: A tool to detect and estimate contamination in large language models. Transactions of the Association for Computational Linguistics, 13:809–830. [find](https://scholar.google.com/scholar?q=Shahriar+Golchin+and+Mihai+Surdeanu%2E+2025%2E+Data+contamination+quiz%3A+A+tool+to+detect+and+estimate+contamination+in+large+language+models%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C)
17. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In International Conference on Learning Representations (ICLR).
18. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, and 5 others. 2024. [Qwen2.5-Coder technical report](https://md.research.elamur.ai/free/en/papers/hui-2024-qwen2-5-coder-technical-report/). arXiv preprint arXiv:2409.1…
19. Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg. 2023. Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5075–5084, Sing… [find](https://scholar.google.com/scholar?q=Alon+Jacovi%2C+Avi+Caciularu%2C+Omer+Goldman%2C+and+Yoav+Goldberg%2E+2023%2E+Stop+uploading+test+data+in+plain+text%3A+Practical+strategies+for+mitigating+data+contamination+by+evaluation+benchmarks%2E+In+Proceedin)
20. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2025. [Livecodebench: Holistic and contamination free evaluation of large language models for code](https://md.research.elamur.ai/free/en/papers/jain-2024-livecodebench-holistic-contamination-free-evaluation/). In The Thirteenth International Conference on Learning Representatio…
21. Changmao Li and Jeffrey Flanigan. 2024. [Task contamination: language models may not be few-shot anymore](https://md.research.elamur.ai/free/en/papers/li-2023-task-contamination-language-models-may/). In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educat…
22. Xiang Li, Yunshi Lan, and Chao Yang. 2025. Treeeval: Benchmark-free evaluation of large language models through tree planning. Proceedings of the AAAI Conference on Artificial Intelligence, 39(23):24485–24493. [find](https://scholar.google.com/scholar?q=Xiang+Li%2C+Yunshi+Lan%2C+and+Chao+Yang%2E+2025%2E+Treeeval%3A+Benchmark%2Dfree+evaluation+of+large+language+models+through+tree+planning%2E+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+39%2823%29%3A244)
23. Yucheng Li, Frank Guerin, and Chenghua Lin. 2024a. Latesteval: addressing data contamination in language model evaluation through dynamic and time-sensitive test construction. In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative App… [find](https://scholar.google.com/scholar?q=Yucheng+Li%2C+Frank+Guerin%2C+and+Chenghua+Lin%2E+2024a%2E+Latesteval%3A+addressing+data+contamination+in+language+model+evaluation+through+dynamic+and+time%2Dsensitive+test+construction%2E+In+Proceedings+of+the+Th)
24. Yucheng Li, Yunhao Guo, Frank Guerin, and Chenghua Lin. 2024b. An open-source data contamination report for large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 528–541, Miami, Florida, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yucheng+Li%2C+Yunhao+Guo%2C+Frank+Guerin%2C+and+Chenghua+Lin%2E+2024b%2E+An+open%2Dsource+data+contamination+report+for+large+language+models%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2)
25. Pratyush Maini, Hengrui Jia, Nicolas Papernot, and Adam Dziedzic. 2024. Llm dataset inference: did you train on my dataset? In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA. Curran Associates Inc. [find](https://scholar.google.com/scholar?q=Pratyush+Maini%2C+Hengrui+Jia%2C+Nicolas+Papernot%2C+and+Adam+Dziedzic%2E+2024%2E+Llm+dataset+inference%3A+did+you+train+on+my+dataset%3F+In+Proceedings+of+the+38th+International+Conference+on+Neural+Information+Pr)
26. Matthieu Meeus, Shubham Jain, Marek Rei, and Yves-Alexandre de Montjoye. 2024. Did the neurons read your book? document-level membership inference for large language models. In Proceedings of the 33rd USENIX Conference on Security Symposium, SEC ’24, USA. USENIX Association. [find](https://scholar.google.com/scholar?q=Matthieu+Meeus%2C+Shubham+Jain%2C+Marek+Rei%2C+and+Yves%2DAlexandre+de+Montjoye%2E+2024%2E+Did+the+neurons+read+your+book%3F+document%2Dlevel+membership+inference+for+large+language+models%2E+In+Proceedings+of+the+33rd)
27. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* ’19, page 220–229, New Yor… [find](https://scholar.google.com/scholar?q=Margaret+Mitchell%2C+Simone+Wu%2C+Andrew+Zaldivar%2C+Parker+Barnes%2C+Lucy+Vasserman%2C+Ben+Hutchinson%2C+Elena+Spitzer%2C+Inioluwa+Deborah+Raji%2C+and+Timnit+Gebru%2E+2019%2E+Model+cards+for+model+reporting%2E+In+Proceedi)
28. OpenAI. 2023. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
29. Yonatan Oren, Nicole Meister, Niladri S. Chatterji, Faisal Ladhak, and Tatsunori Hashimoto. 2024. [Proving test set contamination in black-box language models](https://md.research.elamur.ai/free/en/papers/oren-2023-proving-test-set-contamination-black/). In The Twelfth International Conference on Learning Representations.
30. Medha Palavalli, Amanda Bertsch, and Matthew Gormley. 2024. A taxonomy for data contamination in large language models. In Proceedings of the 1st Workshop on Data Contamination (CONDA), pages 22–40, Bangkok, Thailand. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Medha+Palavalli%2C+Amanda+Bertsch%2C+and+Matthew+Gormley%2E+2024%2E+A+taxonomy+for+data+contamination+in+large+language+models%2E+In+Proceedings+of+the+1st+Workshop+on+Data+Contamination+%28CONDA%29%2C+pages+22%E2%80%9340%2C+B)
31. Haritz Puerto, Martin Gubri, Sangdoo Yun, and Seong Joon Oh. 2025. Scaling up membership inference: When and how attacks succeed on large language models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 4165–4182, Albuquerque, New Mexico. Association for Computational… [find](https://scholar.google.com/scholar?q=Haritz+Puerto%2C+Martin+Gubri%2C+Sangdoo+Yun%2C+and+Seong+Joon+Oh%2E+2025%2E+Scaling+up+membership+inference%3A+When+and+how+attacks+succeed+on+large+language+models%2E+In+Findings+of+the+Association+for+Computatio)
32. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [SQuAD: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
33. Mathieu Ravaut, Bosheng Ding, Fangkai Jiao, Hailin Chen, Xingxuan Li, Ruochen Zhao, Chengwei Qin, Caiming Xiong, and Shafiq Joty. 2025. A comprehensive survey of contamination detection methods in large language models. Transactions on Machine Learning Research. [find](https://scholar.google.com/scholar?q=Mathieu+Ravaut%2C+Bosheng+Ding%2C+Fangkai+Jiao%2C+Hailin+Chen%2C+Xingxuan+Li%2C+Ruochen+Zhao%2C+Chengwei+Qin%2C+Caiming+Xiong%2C+and+Shafiq+Joty%2E+2025%2E+A+comprehensive+survey+of+contamination+detection+methods+in+lar)
34. Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902– 4912, Online. Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Marco+Tulio+Ribeiro%2C+Tongshuang+Wu%2C+Carlos+Guestrin%2C+and+Sameer+Singh%2E+2020%2E+Beyond+accuracy%3A+Behavioral+testing+of+NLP+models+with+CheckList%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Associat)
35. Pedro Rodriguez, Joe Barrow, Alexander Hoyle, John P. Lalor, Robin Jia, and Jordan Boyd-Graber. 2021. Evaluation examples are not equally informative: How should that change NLP leaderboards? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Inte… [find](https://scholar.google.com/scholar?q=Pedro+Rodriguez%2C+Joe+Barrow%2C+Alexander+Hoyle%2C+John+P%2E+Lalor%2C+Robin+Jia%2C+and+Jordan+Boyd%2DGraber%2E+2021%2E+Evaluation+examples+are+not+equally+informative%3A+How+should+that+change+NLP+leaderboards%3F+In+Proce)
36. Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. [NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark](https://md.research.elamur.ai/free/en/papers/sainz-2023-nlp-evaluation-trouble-need-measure/). In Findings of the 2023 Conference on Empirical Methods in Natural Language Processing.
37. Vinay Samuel, Yue Zhou, and Henry Peng Zou. 2025. [Towards data contamination detection for modern large language models: Limitations, inconsistencies, and oracle challenges](https://md.research.elamur.ai/free/en/papers/samuel-2025-data-contamination-detection-modern-large/). In Proceedings of the 31st International Conference on Computational Linguistics, pages 5058–5070, Abu Dhabi, UAE. Association…
38. Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2024. Detecting pretraining data from large language models. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Weijia+Shi%2C+Anirudh+Ajith%2C+Mengzhou+Xia%2C+Yangsibo+Huang%2C+Daogao+Liu%2C+Terra+Blevins%2C+Danqi+Chen%2C+and+Luke+Zettlemoyer%2E+2024%2E+Detecting+pretraining+data+from+large+language+models%2E+In+The+Twelfth+Intern)
39. Yongding Tao, Tian Wang, Yihong Dong, Huanyu Liu, Kechi Zhang, Hu XiaoLong, and Ge Li. 2026. Detecting data contamination from reinforcement learning post-training for large language models. In The Fourteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yongding+Tao%2C+Tian+Wang%2C+Yihong+Dong%2C+Huanyu+Liu%2C+Kechi+Zhang%2C+Hu+XiaoLong%2C+and+Ge+Li%2E+2026%2E+Detecting+data+contamination+from+reinforcement+learning+post%2Dtraining+for+large+language+models%2E+In+The+Fo)
40. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, and 49 others… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
41. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. [SuperGLUE: a stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). Curran Associates Inc., Red Hook, NY, USA.
42. Siyuan Wang, Zhuohan Long, Zhihao Fan, Xuanjing Huang, and Zhongyu Wei. 2025. [Benchmark self-evolving: A multi-agent framework for dynamic LLM evaluation](https://md.research.elamur.ai/free/en/papers/wang-2025-benchmark-self-evolving-multi-agent/). In Proceedings of the 31st International Conference on Computational Linguistics, pages 3310–3328, Abu Dhabi, UAE. Association for Computational…
43. Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh-Agrawal, Sandeep Singh Sandha, Siddartha Venkat Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. 2025. Li… [find](https://scholar.google.com/scholar?q=Colin+White%2C+Samuel+Dooley%2C+Manley+Roberts%2C+Arka+Pal%2C+Benjamin+Feuer%2C+Siddhartha+Jain%2C+Ravid+Shwartz%2DZiv%2C+Neel+Jain%2C+Khalid+Saifullah%2C+Sreemanti+Dey%2C+Shubh%2DAgrawal%2C+Sandeep+Singh+Sandha%2C+Siddartha+Ven)
44. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. [A broad-coverage challenge corpus for sentence understanding through inference](https://md.research.elamur.ai/free/en/papers/williams-2018-broad-coverage-challenge-corpus-sentence/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa…
45. Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. 2024. [Benchmarking benchmark leakage in large language models](https://md.research.elamur.ai/free/en/papers/xu-2024-benchmarking-benchmark-leakage-large-language/). arXiv preprint arXiv:2404.18824.
46. Feng Yao, Yufan Zhuang, Zihao Sun, Sunan Xu, Animesh Kumar, and Jingbo Shang. 2024. Data contamination can cross language barriers. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17864–17875, Miami, Florida, USA. Association for Computational Linguis… [find](https://scholar.google.com/scholar?q=Feng+Yao%2C+Yufan+Zhuang%2C+Zihao+Sun%2C+Sunan+Xu%2C+Animesh+Kumar%2C+and+Jingbo+Shang%2E+2024%2E+Data+contamination+can+cross+language+barriers%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Natura)
47. Zhuohao Yu, Chang Gao, Wenjin Yao, Yidong Wang, Wei Ye, Jindong Wang, Xing Xie, Yue Zhang, and Shikun Zhang. 2024. KIEval: A knowledge-grounded interactive evaluation framework for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volu… [find](https://scholar.google.com/scholar?q=Zhuohao+Yu%2C+Chang+Gao%2C+Wenjin+Yao%2C+Yidong+Wang%2C+Wei+Ye%2C+Jindong+Wang%2C+Xing+Xie%2C+Yue+Zhang%2C+and+Shikun+Zhang%2E+2024%2E+KIEval%3A+A+knowledge%2Dgrounded+interactive+evaluation+framework+for+large+language+mode)
48. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+HellaSwag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Com)
49. Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Charlotte Zhuang, Dylan Slack, Qin Lyu, Sean Hendryx, Russell Kaplan, Michele (Mike) Lunati, and Summer Yue. 2024a. [A careful examination of large language model performance on grade school arithmetic](https://md.research.elamur.ai/free/en/papers/zhang-2024-careful-examination-large-language-model/)…
50. Huixuan Zhang, Yun Lin, and Xiaojun Wan. 2024b. PaCoST: Paired confidence significance testing for benchmark contamination detection in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1794–1809, Miami, Florida, USA. Association for Computational… [find](https://scholar.google.com/scholar?q=Huixuan+Zhang%2C+Yun+Lin%2C+and+Xiaojun+Wan%2E+2024b%2E+PaCoST%3A+Paired+confidence+significance+testing+for+benchmark+contamination+detection+in+large+language+models%2E+In+Findings+of+the+Association+for+Comput)
51. Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Frank Yang, and Hai Li. 2025. Min-k%++: Improved baseline for pre-training data detection from large language models. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Jingyang+Zhang%2C+Jingwei+Sun%2C+Eric+Yeats%2C+Yang+Ouyang%2C+Martin+Kuo%2C+Jianyi+Zhang%2C+Hao+Frank+Yang%2C+and+Hai+Li%2E+2025%2E+Min%2Dk%25%2B%2B%3A+Improved+baseline+for+pre%2Dtraining+data+detection+from+large+language+models)
52. Weichao Zhang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, and Xueqi Cheng. 2024c. Pre-training data detection for large language models: A divergence-based calibration method. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5263–5274, Mi… [find](https://scholar.google.com/scholar?q=Weichao+Zhang%2C+Ruqing+Zhang%2C+Jiafeng+Guo%2C+Maarten+de+Rijke%2C+Yixing+Fan%2C+and+Xueqi+Cheng%2E+2024c%2E+Pre%2Dtraining+data+detection+for+large+language+models%3A+A+divergence%2Dbased+calibration+method%2E+In+Proceed)
53. Qihao Zhao, Yangyu Huang, Tengchao Lv, Lei Cui, Qinzheng Sun, Shaoguang Mao, Xin Zhang, Ying Xin, Qiufeng Yin, Scarlett Li, and Furu Wei. 2025. [MMLU-CF: A contamination-free multi-task language understanding benchmark](https://md.research.elamur.ai/free/en/papers/zhao-2025-mmlu-cf-contamination-free-multi/). In Proceedings of the 63rd Annual Meeting of the Association for Computational Li…
54. Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. 2024a. Dyval: Dynamic evaluation of large language models for reasoning tasks. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Kaijie+Zhu%2C+Jiaao+Chen%2C+Jindong+Wang%2C+Neil+Zhenqiang+Gong%2C+Diyi+Yang%2C+and+Xing+Xie%2E+2024a%2E+Dyval%3A+Dynamic+evaluation+of+large+language+models+for+reasoning+tasks%2E+In+The+Twelfth+International+Conferen)
55. Kaijie Zhu, Jindong Wang, Qinlin Zhao, Ruochen Xu, and Xing Xie. 2024b. [Dynamic evaluation of large language models by meta probing agents](https://md.research.elamur.ai/free/en/papers/zhu-2024-dynamic-evaluation-large-language-models/). In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org.
56. Wenhong Zhu, Hongkun Hao, Zhiwei He, Yun-Ze Song, Jiao Yueyang, Yumeng Zhang, Hanxu Hu, Yiran Wei, Rui Wang, and Hongyuan Lu. 2024c. [CLEAN– EVAL: Clean evaluation on contaminated large language models](https://md.research.elamur.ai/free/en/papers/zhu-2023-clean-eval-clean-evaluation-contaminated/). In Findings of the Association for Computational Linguistics: NAACL 2024, pages 835– 847, Mexico C…

*Updated 2 November 2026*