---
title: "Controllable Contamination Detection for Reliable LLM Evaluation"
description: "The authors recast clean-sample selection as multiple hypothesis testing and propose FTD (FDR-controlled Training Data… 70 references."
authors: ["Zheng Zhang", "Qi Liu", "Siyuan Liang", "Ning Li", "Zirui Hu", "Weibo Gao", "Rui Li", "Zhenya Huang", "Leszek Rutkowski", "Baosheng Yu", "Dacheng Tao"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.1390/"
doi: "10.18653/v1/2026.acl-long.1390"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2026-controllable-contamination-detection-reliable-llm/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2026-controllable-contamination-detection-reliable-llm/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/28/) / Controllable Contamination Detection for Reliable LLM Evaluation with Statistical Guarantees

# Controllable Contamination Detection for Reliable LLM Evaluation with Statistical Guarantees

**Authors:** [Zheng Zhang](https://md.research.elamur.ai/free/en/authors/zheng-zhang/), [Qi Liu](https://md.research.elamur.ai/free/en/authors/qi-liu/), Siyuan Liang, Ning Li, Zirui Hu, [Weibo Gao](https://md.research.elamur.ai/free/en/authors/weibo-gao/), [Rui Li](https://md.research.elamur.ai/free/en/authors/rui-li/), [Zhenya Huang](https://md.research.elamur.ai/free/en/authors/zhenya-huang/), [Leszek Rutkowski](https://md.research.elamur.ai/free/en/authors/leszek-rutkowski/), Baosheng Yu and [Dacheng Tao](https://md.research.elamur.ai/free/en/authors/dacheng-tao/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.1390/) · [PDF](https://aclanthology.org/2026.acl-long.1390.pdf) · [DOI 10.18653/v1/2026.acl-long.1390](https://doi.org/10.18653/v1/2026.acl-long.1390) · [OpenAlex](https://openalex.org/W7166870930)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
LLM benchmarks often leak into pretraining corpora. When that happens, high scores may reflect memorization rather than real ability. Existing detectors try to separate unseen (clean) test items from seen ones, but black-box access limits them. Some contaminated items therefore slip into the "clean" set.

The authors recast clean-sample selection as multiple hypothesis testing and propose FTD (FDR-controlled Training Data detection). The method works in three steps:
- Convert scores from several membership-inference style detectors (e.g., PPL, Min-K%, Min-K%++) into p-values.
- Weight each detector adaptively according to how well it performs on the data.
- Run the Benjamini–Hochberg procedure on the fused p-values.

The theory addresses the dependence between data-driven weights and p-values. It shows the method controls the false discovery rate below a user-chosen level and is asymptotically optimal in power.

On WikiMIA, arXivTection, BBC Real-Time and MIMIR, several baselines exceed the target FDR, while FTD stays well under it. In a simulated contamination study, FTD-filtered benchmarks fully reproduce the clean model rankings. This makes contamination-aware evaluation statistically trustworthy.

## Key points
- **Reported FDR results:** at a 0.15 target, baselines overshoot (Zlib up to 0.292, Lowercase up to 0.216), whereas FTD stays below 0.101 in every setting.
- **Detector weights:** the learned weights favour stronger detectors (Min-K%++ and Min-K% receive the most) and down-weight weaker ones such as Grad and Zlib.
- **Contamination simulation:** LLaMA-3.1-8B and Mistral-7B were fine-tuned on a random 50% of SimpleQA, GPQA, TruthfulQA and ARC-C. Eight models were then re-ranked after filtering.
- **Ranking consistency:** measured with Spearman's rank correlation (SRCC), FTD reaches 1 on all benchmarks and has the lowest FDR among the methods compared.
- **Limitation:** FTD depends on its base detectors. If none of them captures a contamination pattern, those contaminated samples can still go undetected.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Science and Technology of China](https://md.research.elamur.ai/free/en/universities/university-of-science-and-technology-of-china/) — University, Hefei, China · 145 works · 114,409 citations
- [Nanyang Technological University](https://md.research.elamur.ai/free/en/universities/nanyang-technological-university/) — University, Singapore, Singapore · 184 works · 27,484 citations
- [AGH University of Krakow](https://md.research.elamur.ai/free/en/universities/agh-university-of-krakow/) — University, Krakow, Poland · 2 works

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works

## Cite

Cite as: Zheng Zhang, Qi Liu, Siyuan Liang, Ning Li, Zirui Hu, Weibo Gao, Rui Li, Zhenya Huang, Leszek Rutkowski, Baosheng Yu and Dacheng Tao. 2026. Controllable Contamination Detection for Reliable LLM Evaluation with Statistical Guarantees. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhang2026controllable,
  title = {{Controllable Contamination Detection for Reliable LLM Evaluation with Statistical Guarantees}},
  author = {Zheng Zhang, Qi Liu, Siyuan Liang, Ning Li, Zirui Hu, Weibo Gao, Rui Li, Zhenya Huang, Leszek Rutkowski, Baosheng Yu and Dacheng Tao},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1390},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1390}
}
```

- [← previous Contrastively Pre-trained Event Embeddings with Schema-free LLM Annotations](https://md.research.elamur.ai/free/en/papers/mtumbuka-2026-contrastively-pre-trained-event-embeddings/)
- [next → Controllable Sentence Simplification in Italian: Fine-Tuning Large Language Models on Automatically Generated Resources](https://md.research.elamur.ai/free/en/papers/papucci-2026-controllable-sentence-simplification-italian-fine/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/28/)

## References

70 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. contamination in modern benchmarks for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8698–8711. [find](https://scholar.google.com/scholar?q=contamination+in+modern+benchmarks+for+large+language+models%2E+In+Proceedings+of+the+2024+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics%3A+Human+Language+Techn)
2. Stephen Bates, Emmanuel Candès, Lihua Lei, Yaniv Romano, and Matteo Sesia. 2023. Testing for outliers with conformal p-values. The Annals of Statistics, 51(1):149–178. [find](https://scholar.google.com/scholar?q=Stephen+Bates%2C+Emmanuel+Cand%C3%A8s%2C+Lihua+Lei%2C+Yaniv+Romano%2C+and+Matteo+Sesia%2E+2023%2E+Testing+for+outliers+with+conformal+p%2Dvalues%2E+The+Annals+of+Statistics%2C+51%281%29%3A149%E2%80%93178%2E)
3. Yoav Benjamini and Yosef Hochberg. 1995. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological), 57(1):289–300. [find](https://scholar.google.com/scholar?q=Yoav+Benjamini+and+Yosef+Hochberg%2E+1995%2E+Controlling+the+false+discovery+rate%3A+a+practical+and+powerful+approach+to+multiple+testing%2E+Journal+of+the+Royal+statistical+society%3A+series+B+%28Methodological)
4. Yoav Benjamini and Daniel Yekutieli. 2001. The control of the false discovery rate in multiple testing under dependency. Annals of statistics, pages 1165– 1188. [find](https://scholar.google.com/scholar?q=Yoav+Benjamini+and+Daniel+Yekutieli%2E+2001%2E+The+control+of+the+false+discovery+rate+in+multiple+testing+under+dependency%2E+Annals+of+statistics%2C+pages+1165%E2%80%93+1188%2E)
5. Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle OBrien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, and 1 others. 2023. Pythia: A suite for analyzing large language models across training and scaling. In International Con… [find](https://scholar.google.com/scholar?q=Stella+Biderman%2C+Hailey+Schoelkopf%2C+Quentin+Gregory+Anthony%2C+Herbie+Bradley%2C+Kyle+OBrien%2C+Eric+Hallahan%2C+Mohammad+Aflah+Khan%2C+Shivanshu+Purohit%2C+USVSN+Sai+Prashanth%2C+Edward+Raff%2C+and+1+others%2E+2023%2E+P)
6. Sebastian Bordt, Suraj Srinivas, Valentyn Boreiko, and Ulrike von Luxburg. 2025. How much can we forget about data contamination? In Forty-second International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Sebastian+Bordt%2C+Suraj+Srinivas%2C+Valentyn+Boreiko%2C+and+Ulrike+von+Luxburg%2E+2025%2E+How+much+can+we+forget+about+data+contamination%3F+In+Forty%2Dsecond+International+Conference+on+Machine+Learning%2E)
7. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, and 1 others. 2021. [Extracting training data from large language models](https://md.research.elamur.ai/free/en/papers/carlini-2020-extracting-training-data-large-language/). In 30th USENIX security symposium (USENIX Security 21), pages 2633–2650…
8. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, and 1 others. 2024. [A survey on evaluation of large language models](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/). ACM transactions on intelligent systems and technology, 15(3):1–45.
9. Yuxing Cheng, Yi Chang, and Yuan Wu. 2025. A survey on data contamination for large language models. arXiv preprint arXiv:2502.14425. — [arXiv:2502.14425](https://arxiv.org/abs/2502.14425) [find](https://scholar.google.com/scholar?q=Yuxing+Cheng%2C+Yi+Chang%2C+and+Yuan+Wu%2E+2025%2E+A+survey+on+data+contamination+for+large+language+models%2E+arXiv+preprint+arXiv%3A2502%2E14425%2E+%E2%80%94+%5BarXiv%3A2502%2E14425%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2502%2E14425%29)
10. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). Preprint, arXiv:1803.05457.
11. Jasper Dekoninck, Mark Müller, and Martin Vechev. 2024. Constat: Performance-based contamination detection in large language models. Advances in Neural Information Processing Systems, 37:92420– 92464. [find](https://scholar.google.com/scholar?q=Jasper+Dekoninck%2C+Mark+M%C3%BCller%2C+and+Martin+Vechev%2E+2024%2E+Constat%3A+Performance%2Dbased+contamination+detection+in+large+language+models%2E+Advances+in+Neural+Information+Processing+Systems%2C+37%3A92420%E2%80%93+92464%2E)
12. Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. 2024. Investigating data Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. 2024. Generalization or memorization: Data contamination and trustworthy evaluation for large language models. In Findings o… [find](https://scholar.google.com/scholar?q=Chunyuan+Deng%2C+Yilun+Zhao%2C+Xiangru+Tang%2C+Mark+Gerstein%2C+and+Arman+Cohan%2E+2024%2E+Investigating+data+Yihong+Dong%2C+Xue+Jiang%2C+Huanyu+Liu%2C+Zhi+Jin%2C+Bin+Gu%2C+Mengfei+Yang%2C+and+Ge+Li%2E+2024%2E+Generalization+or+)
13. Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. 2024. [Do membership inference attacks work on large language models?](https://md.research.elamur.ai/free/en/papers/duan-2024-membership-inference-attacks-work-large/) In First Conference on Language Modeling.
14. André V Duarte, Xuandong Zhao, Arlindo L Oliveira, and Lei Li. 2024. De-cop: Detecting copyrighted content in language models training data. arXiv preprint arXiv:2402.09910. — [arXiv:2402.09910](https://arxiv.org/abs/2402.09910) [find](https://scholar.google.com/scholar?q=Andr%C3%A9+V+Duarte%2C+Xuandong+Zhao%2C+Arlindo+L+Oliveira%2C+and+Lei+Li%2E+2024%2E+De%2Dcop%3A+Detecting+copyrighted+content+in+language+models+training+data%2E+arXiv+preprint+arXiv%3A2402%2E09910%2E+%E2%80%94+%5BarXiv%3A2402%2E09910%5D%28https)
15. William Feller. 1991. An introduction to probability theory and its applications, Volume 2, volume 2. John Wiley & Sons. [find](https://scholar.google.com/scholar?q=William+Feller%2E+1991%2E+An+introduction+to+probability+theory+and+its+applications%2C+Volume+2%2C+volume+2%2E+John+Wiley+%26+Sons%2E)
16. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, and 1 others. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+and+1+others%2E+2020%2E+The+pile%3A+An+800gb+dataset+of+diverse+text+)
17. Weibo Gao, Qi Liu, Linan Yue, Fangzhou Yao, Rui Lv, Zheng Zhang, Hao Wang, and Zhenya Huang. 2025. Agent4edu: Generating learner response data by generative agents for intelligent education systems. In Proceedings of the AAAI Conference on Artificial Intelligence, 22, pages 23923–23932. [find](https://scholar.google.com/scholar?q=Weibo+Gao%2C+Qi+Liu%2C+Linan+Yue%2C+Fangzhou+Yao%2C+Rui+Lv%2C+Zheng+Zhang%2C+Hao+Wang%2C+and+Zhenya+Huang%2E+2025%2E+Agent4edu%3A+Generating+learner+response+data+by+generative+agents+for+intelligent+education+systems%2E+I)
18. Weibo Gao, Hao Wang, Qi Liu, Fei Wang, Xin Lin, Linan Yue, Zheng Zhang, Rui Lv, and Shijin Wang. 2023. Leveraging transferable knowledge concept graph embedding for cold-start cognitive diagnosis. In Proceedings of the 46th international ACM SIGIR conference on research and development in informatio… [find](https://scholar.google.com/scholar?q=Weibo+Gao%2C+Hao+Wang%2C+Qi+Liu%2C+Fei+Wang%2C+Xin+Lin%2C+Linan+Yue%2C+Zheng+Zhang%2C+Rui+Lv%2C+and+Shijin+Wang%2E+2023%2E+Leveraging+transferable+knowledge+concept+graph+embedding+for+cold%2Dstart+cognitive+diagnosis%2E+In+)
19. Christopher R Genovese, Nicole A Lazar, and Thomas Nichols. 2002. Thresholding of statistical maps in functional neuroimaging using the false discovery rate. Neuroimage, 15(4):870–878. [find](https://scholar.google.com/scholar?q=Christopher+R+Genovese%2C+Nicole+A+Lazar%2C+and+Thomas+Nichols%2E+2002%2E+Thresholding+of+statistical+maps+in+functional+neuroimaging+using+the+false+discovery+rate%2E+Neuroimage%2C+15%284%29%3A870%E2%80%93878%2E)
20. Shahriar Golchin and Mihai Surdeanu. 2023. Data contamination quiz: A tool to detect and estimate contamination in large language models. arXiv preprint arXiv:2311.06233. [find](https://scholar.google.com/scholar?q=Shahriar+Golchin+and+Mihai+Surdeanu%2E+2023%2E+Data+contamination+quiz%3A+A+tool+to+detect+and+estimate+contamination+in+large+language+models%2E+arXiv+preprint+arXiv%3A2311%2E06233%2E)
21. Shahriar Golchin and Mihai Surdeanu. 2024a. [Time travel in llms: Tracing data contamination in large language models](https://md.research.elamur.ai/free/en/papers/golchin-2023-time-travel-llms-tracing-data/). In ICLR.
22. Shahriar Golchin and Mihai Surdeanu. 2024b. [Time travel in llms: Tracing data contamination in large language models](https://md.research.elamur.ai/free/en/papers/golchin-2023-time-travel-llms-tracing-data/). In The Twelfth International Conference on Learning Representations.
23. Zirui Hu, Yingjie Wang, Zheng Zhang, Hong Chen, and Dacheng Tao. 2025. A statistical approach for controlled training data detection. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zirui+Hu%2C+Yingjie+Wang%2C+Zheng+Zhang%2C+Hong+Chen%2C+and+Dacheng+Tao%2E+2025%2E+A+statistical+approach+for+controlled+training+data+detection%2E+In+The+Thirteenth+International+Conference+on+Learning+Representat)
24. Zirui Hu, Zheng Zhang, Yingjie Wang, Leszek Rutkowski, and Dacheng Tao. 2026. Cofact: Conformal factuality guarantees for language models under covariate shift. In The Fourteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zirui+Hu%2C+Zheng+Zhang%2C+Yingjie+Wang%2C+Leszek+Rutkowski%2C+and+Dacheng+Tao%2E+2026%2E+Cofact%3A+Conformal+factuality+guarantees+for+language+models+under+covariate+shift%2E+In+The+Fourteenth+International+Confere)
25. Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg. 2023. Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks. In The 2023 Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Alon+Jacovi%2C+Avi+Caciularu%2C+Omer+Goldman%2C+and+Yoav+Goldberg%2E+2023%2E+Stop+uploading+test+data+in+plain+text%3A+Practical+strategies+for+mitigating+data+contamination+by+evaluation+benchmarks%2E+In+The+2023+)
26. Muhammed Yusuf Kocyigit, Eleftheria Briakou, Daniel Deutsch, Jiaming Luo, Colin Cherry, and Markus Freitag. 2025. Overestimation in llm evaluation: A controlled large-scale study on data contaminations impact on machine translation. In Forty-second International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Muhammed+Yusuf+Kocyigit%2C+Eleftheria+Briakou%2C+Daniel+Deutsch%2C+Jiaming+Luo%2C+Colin+Cherry%2C+and+Markus+Freitag%2E+2025%2E+Overestimation+in+llm+evaluation%3A+A+controlled+large%2Dscale+study+on+data+contamination)
27. Yucheng Li. 2023. Estimating contamination via perplexity: Quantifying memorisation in language model evaluation. arXiv preprint arXiv:2309.10677. — [arXiv:2309.10677](https://arxiv.org/abs/2309.10677) [find](https://scholar.google.com/scholar?q=Yucheng+Li%2E+2023%2E+Estimating+contamination+via+perplexity%3A+Quantifying+memorisation+in+language+model+evaluation%2E+arXiv+preprint+arXiv%3A2309%2E10677%2E+%E2%80%94+%5BarXiv%3A2309%2E10677%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2309%2E10677)
28. Yucheng Li, Frank Guerin, and Chenghua Lin. 2024. Latesteval: Addressing data contamination in language model evaluation through dynamic and time-sensitive test construction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18600–18607. [find](https://scholar.google.com/scholar?q=Yucheng+Li%2C+Frank+Guerin%2C+and+Chenghua+Lin%2E+2024%2E+Latesteval%3A+Addressing+data+contamination+in+language+model+evaluation+through+dynamic+and+time%2Dsensitive+test+construction%2E+In+Proceedings+of+the+AAA)
29. Siyuan Liang, Tianmeng Fang, Zhe Liu, Aishan Liu, Yan Xiao, Jinyuan He, Ee-Chien Chang, and Xiaochun Cao. 2025. Safemobile: Chain-level jailbreak detection and automated evaluation for multimodal mobile agents. arXiv preprint arXiv:2507.00841. — [arXiv:2507.00841](https://arxiv.org/abs/2507.00841) [find](https://scholar.google.com/scholar?q=Siyuan+Liang%2C+Tianmeng+Fang%2C+Zhe+Liu%2C+Aishan+Liu%2C+Yan+Xiao%2C+Jinyuan+He%2C+Ee%2DChien+Chang%2C+and+Xiaochun+Cao%2E+2025%2E+Safemobile%3A+Chain%2Dlevel+jailbreak+detection+and+automated+evaluation+for+multimodal+mobi)
30. Siyuan Liang, Jiajun Gong, Tianmeng Fang, Aishan Liu, Tao Wang, Xiaochun Cao, Dacheng Tao, and Chang Ee-Chien. 2026. Trapflow: Controllable website fingerprinting defense via dynamic backdoor learning. IEEE Transactions on Information Forensics and Security. [find](https://scholar.google.com/scholar?q=Siyuan+Liang%2C+Jiajun+Gong%2C+Tianmeng+Fang%2C+Aishan+Liu%2C+Tao+Wang%2C+Xiaochun+Cao%2C+Dacheng+Tao%2C+and+Chang+Ee%2DChien%2E+2026%2E+Trapflow%3A+Controllable+website+fingerprinting+defense+via+dynamic+backdoor+learning)
31. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. [Truthfulqa: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). Preprint, arXiv:2109.07958.
32. Yaowu Liu and Jun Xie. 2020. Cauchy combination test: a powerful test with analytic p-value calculation under arbitrary dependency structures. Journal of the American Statistical Association, 115(529):393– 402. [find](https://scholar.google.com/scholar?q=Yaowu+Liu+and+Jun+Xie%2E+2020%2E+Cauchy+combination+test%3A+a+powerful+test+with+analytic+p%2Dvalue+calculation+under+arbitrary+dependency+structures%2E+Journal+of+the+American+Statistical+Association%2C+115%28529%29)
33. Yuhan Liu, Yuxuan Liu, Xiaoqing Zhang, Xiuying Chen, and Rui Yan. 2025. The truth becomes clearer through debate! multi-agent systems with large language models unmask fake news. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages… [find](https://scholar.google.com/scholar?q=Yuhan+Liu%2C+Yuxuan+Liu%2C+Xiaoqing+Zhang%2C+Xiuying+Chen%2C+and+Rui+Yan%2E+2025%2E+The+truth+becomes+clearer+through+debate%21+multi%2Dagent+systems+with+large+language+models+unmask+fake+news%2E+In+Proceedings+of+the)
34. Mingya Long, Zhengbang Li, Wei Zhang, and Qizhai Li. 2023. The cauchy combination test under arbitrary dependence structures. The American Statistician, 77(2):134–142. [find](https://scholar.google.com/scholar?q=Mingya+Long%2C+Zhengbang+Li%2C+Wei+Zhang%2C+and+Qizhai+Li%2E+2023%2E+The+cauchy+combination+test+under+arbitrary+dependence+structures%2E+The+American+Statistician%2C+77%282%29%3A134%E2%80%93142%2E)
35. Wenxi Lv, Qinliang Su, Hai Wan, Hongteng Xu, and Wenchao Xu. 2024. Contamination-resilient anomaly detection via adversarial learning on partially-observed normal and anomalous data. In Forty-first International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Wenxi+Lv%2C+Qinliang+Su%2C+Hai+Wan%2C+Hongteng+Xu%2C+and+Wenchao+Xu%2E+2024%2E+Contamination%2Dresilient+anomaly+detection+via+adversarial+learning+on+partially%2Dobserved+normal+and+anomalous+data%2E+In+Forty%2Dfirst+In)
36. Inbal Magar and Roy Schwartz. 2022. Data contamination: From memorization to exploitation. arXiv preprint arXiv:2203.08242. [find](https://scholar.google.com/scholar?q=Inbal+Magar+and+Roy+Schwartz%2E+2022%2E+Data+contamination%3A+From+memorization+to+exploitation%2E+arXiv+preprint+arXiv%3A2203%2E08242%2E)
37. Ben Mann, N Ryder, M Subbiah, J Kaplan, P Dhariwal, A Neelakantan, P Shyam, G Sastry, A Askell, S Agarwal, and 1 others. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). arXiv preprint arXiv:2005.14165, 1:3.
38. Qingyang Mao, Qi Liu, Zhi Li, Mingyue Cheng, Zheng Zhang, and Rui Li. 2024. Potable: Towards systematic thinking via stage-oriented plan-then-execute reasoning on tables. arXiv preprint arXiv:2412.04272. — [arXiv:2412.04272](https://arxiv.org/abs/2412.04272) [find](https://scholar.google.com/scholar?q=Qingyang+Mao%2C+Qi+Liu%2C+Zhi+Li%2C+Mingyue+Cheng%2C+Zheng+Zhang%2C+and+Rui+Li%2E+2024%2E+Potable%3A+Towards+systematic+thinking+via+stage%2Doriented+plan%2Dthen%2Dexecute+reasoning+on+tables%2E+arXiv+preprint+arXiv%3A2412%2E042)
39. Yonatan Oren, Nicole Meister, Niladri S Chatterji, Faisal Ladhak, and Tatsunori Hashimoto. 2023. [Proving test set contamination in black-box language models](https://md.research.elamur.ai/free/en/papers/oren-2023-proving-test-set-contamination-black/). In The Twelfth International Conference on Learning Representations.
40. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2023. [Gpqa: A graduate-level google-proof q&a benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/). Preprint, arXiv:2311.12022.
41. Zhiyao Ren, Siyuan Liang, Aishan Liu, and Dacheng Tao. 2025. Iclshield: Exploring and mitigating in-context learning backdoor attacks. arXiv preprint arXiv:2507.01321. — [arXiv:2507.01321](https://arxiv.org/abs/2507.01321) [find](https://scholar.google.com/scholar?q=Zhiyao+Ren%2C+Siyuan+Liang%2C+Aishan+Liu%2C+and+Dacheng+Tao%2E+2025%2E+Iclshield%3A+Exploring+and+mitigating+in%2Dcontext+learning+backdoor+attacks%2E+arXiv+preprint+arXiv%3A2507%2E01321%2E+%E2%80%94+%5BarXiv%3A2507%2E01321%5D%28https%3A%2F%2Farx)
42. Philip Sedgwick. 2014. Spearmans rank correlation coefficient. Bmj, 349. [find](https://scholar.google.com/scholar?q=Philip+Sedgwick%2E+2014%2E+Spearmans+rank+correlation+coefficient%2E+Bmj%2C+349%2E)
43. Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2024. Detecting pretraining data from large language models. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Weijia+Shi%2C+Anirudh+Ajith%2C+Mengzhou+Xia%2C+Yangsibo+Huang%2C+Daogao+Liu%2C+Terra+Blevins%2C+Danqi+Chen%2C+and+Luke+Zettlemoyer%2E+2024%2E+Detecting+pretraining+data+from+large+language+models%2E+In+The+Twelfth+Intern)
44. John D Storey, Jonathan E Taylor, and David Siegmund. 2004. Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: a unified approach. Journal of the Royal Statistical Society Series B: Statistical Methodology, 66(1):187–205. [find](https://scholar.google.com/scholar?q=John+D+Storey%2C+Jonathan+E+Taylor%2C+and+David+Siegmund%2E+2004%2E+Strong+control%2C+conservative+point+estimation+and+simultaneous+conservative+consistency+of+false+discovery+rates%3A+a+unified+approach%2E+Journa)
45. John D Storey and Robert Tibshirani. 2003. Statistical significance for genomewide studies. Proceedings of the National Academy of Sciences, 100(16):9440– 9445. [find](https://scholar.google.com/scholar?q=John+D+Storey+and+Robert+Tibshirani%2E+2003%2E+Statistical+significance+for+genomewide+studies%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+100%2816%29%3A9440%E2%80%93+9445%2E)
46. Yifan Sun, Han Wang, Dongbai Li, Gang Wang, and Huan Zhang. 2025. The emperor’s new clothes in benchmarking? a rigorous examination of mitigation strategies for llm benchmark data contamination. In Forty-second International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Yifan+Sun%2C+Han+Wang%2C+Dongbai+Li%2C+Gang+Wang%2C+and+Huan+Zhang%2E+2025%2E+The+emperor%E2%80%99s+new+clothes+in+benchmarking%3F+a+rigorous+examination+of+mitigation+strategies+for+llm+benchmark+data+contamination%2E+In+Fo)
47. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971.
48. Fei Wang, Weibo Gao, Qi Liu, Jiatong Li, Guanhao Zhao, Zheng Zhang, Zhenya Huang, Mengxiao Zhu, Shijin Wang, Wei Tong, and 1 others. 2024a. A survey of models for cognitive diagnosis: New developments and future directions. arXiv preprint arXiv:2407.05458. — [arXiv:2407.05458](https://arxiv.org/abs/2407.05458) [find](https://scholar.google.com/scholar?q=Fei+Wang%2C+Weibo+Gao%2C+Qi+Liu%2C+Jiatong+Li%2C+Guanhao+Zhao%2C+Zheng+Zhang%2C+Zhenya+Huang%2C+Mengxiao+Zhu%2C+Shijin+Wang%2C+Wei+Tong%2C+and+1+others%2E+2024a%2E+A+survey+of+models+for+cognitive+diagnosis%3A+New+developments)
49. Xiaoning Wang, Yuyang Huo, Liuhua Peng, and Changliang Zou. 2024b. Conformalized multiple testing after data-dependent selection. Advances in Neural Information Processing Systems, 37:58574– 58609. [find](https://scholar.google.com/scholar?q=Xiaoning+Wang%2C+Yuyang+Huo%2C+Liuhua+Peng%2C+and+Changliang+Zou%2E+2024b%2E+Conformalized+multiple+testing+after+data%2Ddependent+selection%2E+Advances+in+Neural+Information+Processing+Systems%2C+37%3A58574%E2%80%93+58609%2E)
50. Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. 2024. [Measuring short-form factuality in large language models](https://md.research.elamur.ai/free/en/papers/wei-2024-measuring-short-form-factuality-large/). arXiv preprint arXiv:2411.04368.
51. Asaf Weinstein, Weijie J Su, Małgorzata Bogdan, Rina Foygel Barber, and Emmanuel J Candes. 2023. A power analysis for model-x knockoffs with ℓ p -regularized statistics. The Annals of Statistics, 51(3):1005–1029. [find](https://scholar.google.com/scholar?q=Asaf+Weinstein%2C+Weijie+J+Su%2C+Ma%C5%82gorzata+Bogdan%2C+Rina+Foygel+Barber%2C+and+Emmanuel+J+Candes%2E+2023%2E+A+power+analysis+for+model%2Dx+knockoffs+with+%E2%84%93+p+%2Dregularized+statistics%2E+The+Annals+of+Statistics%2C+51%283)
52. Xiaoyang Wu, Yuyang Huo, Haojie Ren, and Changliang Zou. 2024. Optimal subsampling via predictive inference. Journal of the American Statistical Association, 119(548):2844–2856. [find](https://scholar.google.com/scholar?q=Xiaoyang+Wu%2C+Yuyang+Huo%2C+Haojie+Ren%2C+and+Changliang+Zou%2E+2024%2E+Optimal+subsampling+via+predictive+inference%2E+Journal+of+the+American+Statistical+Association%2C+119%28548%29%3A2844%E2%80%932856%2E)
53. Xiaoyang Wu, Yuyang Huo, and Changliang Zou. 2023. Multi-split conformal prediction via cauchy aggregation. Stat, 12(1):e522. [find](https://scholar.google.com/scholar?q=Xiaoyang+Wu%2C+Yuyang+Huo%2C+and+Changliang+Zou%2E+2023%2E+Multi%2Dsplit+conformal+prediction+via+cauchy+aggregation%2E+Stat%2C+12%281%29%3Ae522%2E)
54. Cheng Xu and Nan Yan. 2025. [Triplefact: Defending data contamination in the evaluation of llm-driven fake news detection](https://md.research.elamur.ai/free/en/papers/xu-2025-triplefact-defending-data-contamination-evaluation/). In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8808– 8823.
55. Feng Yao, Yufan Zhuang, Zihao Sun, Sunan Xu, Animesh Kumar, and Jingbo Shang. 2024. Data contamination can cross language barriers. arXiv preprint arXiv:2406.13236. [find](https://scholar.google.com/scholar?q=Feng+Yao%2C+Yufan+Zhuang%2C+Zihao+Sun%2C+Sunan+Xu%2C+Animesh+Kumar%2C+and+Jingbo+Shang%2E+2024%2E+Data+contamination+can+cross+language+barriers%2E+arXiv+preprint+arXiv%3A2406%2E13236%2E)
56. Shengyu Ye, Qi Liu, Hao Jiang, Zheng Zhang, Heng Yu, and Zhenya Huang. 2026. Themis: Automated constraint-aware test synthesis framework for code reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 34432–34440. [find](https://scholar.google.com/scholar?q=Shengyu+Ye%2C+Qi+Liu%2C+Hao+Jiang%2C+Zheng+Zhang%2C+Heng+Yu%2C+and+Zhenya+Huang%2E+2026%2E+Themis%3A+Automated+constraint%2Daware+test+synthesis+framework+for+code+reinforcement+learning%2E+In+Proceedings+of+the+AAAI+Con)
57. Zonghao Ying, Aishan Liu, Siyuan Liang, Lei Huang, Jinyang Guo, Wenbo Zhou, Xianglong Liu, and Dacheng Tao. 2026. Safebench: A safety evaluation framework for multimodal large language models. International Journal of Computer Vision, 134(1):18. [find](https://scholar.google.com/scholar?q=Zonghao+Ying%2C+Aishan+Liu%2C+Siyuan+Liang%2C+Lei+Huang%2C+Jinyang+Guo%2C+Wenbo+Zhou%2C+Xianglong+Liu%2C+and+Dacheng+Tao%2E+2026%2E+Safebench%3A+A+safety+evaluation+framework+for+multimodal+large+language+models%2E+Interna)
58. Yi Zhan, Qi Liu, Weibo Gao, Zheng Zhang, Tianfu Wang, Shuanghong Shen, Junyu Lu, and Zhenya Huang. 2025. Coderagent: Simulating student behavior for personalized programming learning with large language models. arXiv preprint arXiv:2505.20642. — [arXiv:2505.20642](https://arxiv.org/abs/2505.20642) [find](https://scholar.google.com/scholar?q=Yi+Zhan%2C+Qi+Liu%2C+Weibo+Gao%2C+Zheng+Zhang%2C+Tianfu+Wang%2C+Shuanghong+Shen%2C+Junyu+Lu%2C+and+Zhenya+Huang%2E+2025%2E+Coderagent%3A+Simulating+student+behavior+for+personalized+programming+learning+with+large+langua)
59. Hengxiang Zhang, Songxin Zhang, Bingyi Jing, and Hongxin Wei. 2024a. Fine-tuning can help detect pretraining data from large language models. arXiv preprint arXiv:2410.10880. — [arXiv:2410.10880](https://arxiv.org/abs/2410.10880) [find](https://scholar.google.com/scholar?q=Hengxiang+Zhang%2C+Songxin+Zhang%2C+Bingyi+Jing%2C+and+Hongxin+Wei%2E+2024a%2E+Fine%2Dtuning+can+help+detect+pretraining+data+from+large+language+models%2E+arXiv+preprint+arXiv%3A2410%2E10880%2E+%E2%80%94+%5BarXiv%3A2410%2E10880%5D%28http)
60. Jie Zhang, Debeshee Das, Gautam Kamath, and Florian Tramèr. 2025a. Position: Membership inference attacks cannot prove that a model was trained on your data. In 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 333–345. IEEE. [find](https://scholar.google.com/scholar?q=Jie+Zhang%2C+Debeshee+Das%2C+Gautam+Kamath%2C+and+Florian+Tram%C3%A8r%2E+2025a%2E+Position%3A+Membership+inference+attacks+cannot+prove+that+a+model+was+trained+on+your+data%2E+In+2025+IEEE+Conference+on+Secure+and+Trus)
61. Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Frank Yang, and Hai Li. 2024b. Min-k%++: Improved baseline for detecting pre-training data from large language models. arXiv preprint arXiv:2404.02936. — [arXiv:2404.02936](https://arxiv.org/abs/2404.02936) [find](https://scholar.google.com/scholar?q=Jingyang+Zhang%2C+Jingwei+Sun%2C+Eric+Yeats%2C+Yang+Ouyang%2C+Martin+Kuo%2C+Jianyi+Zhang%2C+Hao+Frank+Yang%2C+and+Hai+Li%2E+2024b%2E+Min%2Dk%25%2B%2B%3A+Improved+baseline+for+detecting+pre%2Dtraining+data+from+large+language+model)
62. Kun Zhang, Jingyu Li, Zhe Li, and S Kevin Zhou. 2025b. Dh-set: Improving vision-language alignment with diverse and hybrid set-embeddings learning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 24993–25003. [find](https://scholar.google.com/scholar?q=Kun+Zhang%2C+Jingyu+Li%2C+Zhe+Li%2C+and+S+Kevin+Zhou%2E+2025b%2E+Dh%2Dset%3A+Improving+vision%2Dlanguage+alignment+with+diverse+and+hybrid+set%2Dembeddings+learning%2E+In+Proceedings+of+the+Computer+Vision+and+Pattern+Re)
63. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, and 1 others. 2022a. [Opt: Open pre-trained transformer language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/). arXiv preprint arXiv:2205.01068.
64. Weichao Zhang, Ruqing Zhang, Jiafeng Guo, Maarten Rijke, Yixing Fan, and Xueqi Cheng. 2024c. Pre-training data detection for large language models: A divergence-based calibration method. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5263–5274. [find](https://scholar.google.com/scholar?q=Weichao+Zhang%2C+Ruqing+Zhang%2C+Jiafeng+Guo%2C+Maarten+Rijke%2C+Yixing+Fan%2C+and+Xueqi+Cheng%2E+2024c%2E+Pre%2Dtraining+data+detection+for+large+language+models%3A+A+divergence%2Dbased+calibration+method%2E+In+Proceeding)
65. Yifan Zhang, Haiyan Jiang, Haojie Ren, Changliang Zou, and Dejing Dou. 2022b. Automs: automatic model selection for novelty detection with error rate control. Advances in Neural Information Processing Systems, 35:19917–19929. [find](https://scholar.google.com/scholar?q=Yifan+Zhang%2C+Haiyan+Jiang%2C+Haojie+Ren%2C+Changliang+Zou%2C+and+Dejing+Dou%2E+2022b%2E+Automs%3A+automatic+model+selection+for+novelty+detection+with+error+rate+control%2E+Advances+in+Neural+Information+Processing)
66. Zheng Zhang, Ning Li, Qi Liu, Rui Li, Weibo Gao, Qingyang Mao, Zhenya Huang, Baosheng Yu, and Dacheng Tao. 2025c. The other side of the coin: Exploring fairness in retrieval-augmented generation. arXiv preprint arXiv:2504.12323. — [arXiv:2504.12323](https://arxiv.org/abs/2504.12323) [find](https://scholar.google.com/scholar?q=Zheng+Zhang%2C+Ning+Li%2C+Qi+Liu%2C+Rui+Li%2C+Weibo+Gao%2C+Qingyang+Mao%2C+Zhenya+Huang%2C+Baosheng+Yu%2C+and+Dacheng+Tao%2E+2025c%2E+The+other+side+of+the+coin%3A+Exploring+fairness+in+retrieval%2Daugmented+generation%2E+arXi)
67. Zheng Zhang, Qi Liu, Hao Jiang, Fei Wang, Yan Zhuang, Le Wu, Weibo Gao, and Enhong Chen. 2023. Fairlisa: fair user modeling with limited sensitive attributes information. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 41432–41450. [find](https://scholar.google.com/scholar?q=Zheng+Zhang%2C+Qi+Liu%2C+Hao+Jiang%2C+Fei+Wang%2C+Yan+Zhuang%2C+Le+Wu%2C+Weibo+Gao%2C+and+Enhong+Chen%2E+2023%2E+Fairlisa%3A+fair+user+modeling+with+limited+sensitive+attributes+information%2E+In+Proceedings+of+the+37th+In)
68. Jingqian Zhao, Bingbing Wang, Geng Tu, Yice Zhang, Qianlong Wang, Bin Liang, Jing Li, and Ruifeng Xu. 2025a. Coreeval: Automatically building contamination-resilient datasets with real-world knowledge toward reliable llm evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Co… [find](https://scholar.google.com/scholar?q=Jingqian+Zhao%2C+Bingbing+Wang%2C+Geng+Tu%2C+Yice+Zhang%2C+Qianlong+Wang%2C+Bin+Liang%2C+Jing+Li%2C+and+Ruifeng+Xu%2E+2025a%2E+Coreeval%3A+Automatically+building+contamination%2Dresilient+datasets+with+real%2Dworld+knowledge)
69. Qihao Zhao, Yangyu Huang, Tengchao Lv, Lei Cui, Qinzheng Sun, Shaoguang Mao, Xin Zhang, Ying Xin, Qiufeng Yin, Scarlett Li, and 1 others. 2025b. [Mmlu-cf: A contamination-free multi-task language understanding benchmark](https://md.research.elamur.ai/free/en/papers/zhao-2025-mmlu-cf-contamination-free-multi/). In Proceedings of the 63rd Annual Meeting of the Association for Computational L…
70. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, and 1 others. 2023. [A survey of large language models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/). arXiv preprint arXiv:2303.18223, 1(2):1–124.

*Updated 2 November 2026*