--- title: "AntiLeakBench — paper by Wu et al. (2025) — summary and ideas" description: "Large language models are usually tested on public, static benchmarks, and those test items… 38.8 EM. 3 references, cited by 10 papers here." authors: ["Xiaobao Wu", "Liangming Pan", "Yuxi Xie", "Ruiwen Zhou", "Shuai Zhao", "Yubo Ma", "Mingzhe Du", "Rui Mao", "Anh Tuan Luu", "William Yang Wang"] year: 2025 venue: "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)" source: "https://aclanthology.org/2025.acl-long.901/" doi: "10.18653/v1/2025.acl-long.901" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/wu-2025-antileakbench-preventing-data-contamination-automatically/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wu-2025-antileakbench-preventing-data-contamination-automatically/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/7/) / AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge # AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge **Authors:** [Xiaobao Wu](https://md.research.elamur.ai/free/en/authors/xiaobao-wu/), [Liangming Pan](https://md.research.elamur.ai/free/en/authors/liangming-pan/), [Yuxi Xie](https://md.research.elamur.ai/free/en/authors/yuxi-xie/), [Ruiwen Zhou](https://md.research.elamur.ai/free/en/authors/ruiwen-zhou/), [Shuai Zhao](https://md.research.elamur.ai/free/en/authors/shuai-zhao/), [Yubo Ma](https://md.research.elamur.ai/free/en/authors/yubo-ma/), [Mingzhe Du](https://md.research.elamur.ai/free/en/authors/mingzhe-du/), [Rui Mao](https://md.research.elamur.ai/free/en/authors/rui-mao/), [Anh Tuan Luu](https://md.research.elamur.ai/free/en/authors/anh-tuan-luu/) and [William Yang Wang](https://md.research.elamur.ai/free/en/authors/william-yang-wang/) · **Year:** 2025 · **Venue:** [Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2025](https://md.research.elamur.ai/free/en/venues/acl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2025.acl-long.901/) · [PDF](https://aclanthology.org/2025.acl-long.901.pdf) · [DOI 10.18653/v1/2025.acl-long.901](https://doi.org/10.18653/v1/2025.acl-long.901) · [OpenAlex](https://openalex.org/W4412944919) **Citations:** 2 citations (OpenAlex, 2026-10-14) · [10](https://md.research.elamur.ai/free/en/papers/wu-2025-antileakbench-preventing-data-contamination-automatically/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Large language models are usually tested on public, static benchmarks, and those test items can leak into the training data of later models, which inflates their scores. Earlier "live" benchmarks gather freshly published material, but new text can still restate older facts, and refreshing those benchmarks usually needs manual annotation. AntiLeakBench takes a different approach. It mines Wikidata for facts whose value changed after a model's knowledge cutoff, such as a footballer joining a new club. It uses start/end time qualifiers and drops cases where the value later reverted. For each changed fact, it retrieves a post-change Wikipedia revision as a supporting document. It then generates question-answering items, both single-hop and multi-hop, in free-generation and multiple-choice formats, optionally mixed with distracting documents. The pipeline needs no human labor, so the benchmark can be rebuilt whenever new models appear. Across open-source and proprietary LLMs, scores generally drop on post-cutoff samples. This suggests that contamination inflates results on older data. The benchmark also remains difficult for most models. ## Key points - **Knowledge source and change detection:** Wikidata (subject, relation, object) triplets are grouped by subject and relation and ordered by start time. When the object differs after the cutoff, the fact is treated as new knowledge; one case is Lionel Andrés Messi, whose sports-team membership switched from Paris Saint Germain to Inter Miami. - **Supporting documents:** these come from the Wikipedia revision history rather than LLM generation. A revision is accepted only if its article summary mentions both the subject and the object, or their aliases. - **Generation-format results:** GPT-4o has the best average EM/F1 (77.2 / 87.9), while the strongest open-source model reaches 53.9 / 62.0. Most models stay below 50. - **Difficulty factors:** scores fall as distracting documents increase (N_d from 3 to 7) and on multi-hop questions. For example, Longchat-v1.5-7B drops from 75.5 to 38.8 EM between Single-Hop Gold and Multi-Hop Gold. The multiple-choice format is noticeably easier. - **Contamination evidence and availability:** on post-cutoff samples, Mistral-Nemo picks the correct option less often (91.0% → 85.5%) and the outdated option more often (6.4% → 13.3%). Code and data are at github.com/bobxwu/AntiLeakBench. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [National University of Singapore](https://md.research.elamur.ai/free/en/universities/national-university-of-singapore/) — University, Singapore, Singapore · 180 works · 34,304 citations - [Shanghai Jiao Tong University](https://md.research.elamur.ai/free/en/universities/shanghai-jiao-tong-university/) — University, Shanghai, China · 149 works · 16,161 citations - [Nanyang Technological University](https://md.research.elamur.ai/free/en/universities/nanyang-technological-university/) — University, Singapore, Singapore · 156 works · 25,644 citations - [University of California, Santa Barbara](https://md.research.elamur.ai/free/en/universities/university-of-california-santa-barbara/) — University, Santa Barbara, United States · 57 works · 9,752 citations - [University of Arizona](https://md.research.elamur.ai/free/en/universities/university-of-arizona/) — University, Tucson, United States · 38 works · 10,603 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works 🧭 Topics (1): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works ## Cite Cite as: Xiaobao Wu, Liangming Pan, Yuxi Xie, Ruiwen Zhou, Shuai Zhao, Yubo Ma, Mingzhe Du, Rui Mao, Anh Tuan Luu and William Yang Wang. 2025. AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{wu2025antileakbench, title = {{AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge}}, author = {Xiaobao Wu, Liangming Pan, Yuxi Xie, Ruiwen Zhou, Shuai Zhao, Yubo Ma, Mingzhe Du, Rui Mao, Anh Tuan Luu and William Yang Wang}, booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, year = {2025}, doi = {10.18653/v1/2025.acl-long.901}, url = {https://doi.org/10.18653/v1/2025.acl-long.901} } ``` - [← previous Activation Steering Decoding: Mitigating Hallucination in Large Vision-Language Models through Bidirectional Hidden State Intervention](https://md.research.elamur.ai/free/en/papers/su-2025-activation-steering-decoding-mitigating-hallucination/) - [next → ArithmAttack: Evaluating Robustness of LLMs to Noisy Context in Math Problem Solving](https://md.research.elamur.ai/free/en/papers/abedin-2025-arithmattack-evaluating-robustness-llms-noisy/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/7/) ## References 3 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. 2024. [Phi-3 technical report: A highly capable language model locally on your phone](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-3-technical-report-highly/). arXiv preprint arXiv:2404.14219. 2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774. 3. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. LongBench: A bilingual, multi-task benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Associat… [find](https://scholar.google.com/scholar?q=Yushi+Bai%2C+Xin+Lv%2C+Jiajie+Zhang%2C+Hongchang+Lyu%2C+Jiankai+Tang%2C+Zhidian+Huang%2C+Zhengxiao+Du%2C+Xiao+Liu%2C+Aohan+Zeng%2C+Lei+Hou%2C+Yuxiao+Dong%2C+Jie+Tang%2C+and+Juanzi+Li%2E+2024%2E+LongBench%3A+A+bilingual%2C+multi%2Dtask) *Updated 1 November 2026*