---
title: "Obscuring Data Contamination Through — Abbas 2026"
description: "The paper asks whether benchmark contamination can go undetected when the leaked material is in another language."
authors: ["Chaymaa Abbas", "Nour Shamaa", "Mariette Awad"]
year: 2026
venue: "arXiv:2601.14994"
source: "https://arxiv.org/abs/2601.14994"
doi: "10.48550/arxiv.2601.14994"
arxiv: "2601.14994"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/abbas-2026-obscuring-data-contamination-through-translation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/abbas-2026-obscuring-data-contamination-through-translation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/7/) / Obscuring Data Contamination Through Translation: Evidence from Arabic Corpora

# Obscuring Data Contamination Through Translation: Evidence from Arabic Corpora

**Authors:** Chaymaa Abbas, Nour Shamaa, [Mariette Awad](https://md.research.elamur.ai/free/en/authors/mariette-awad/) · **Year:** 2026 · **Venue:** arXiv:2601.14994 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2601.14994) · [PDF](https://arxiv.org/pdf/2601.14994) · [DOI 10.48550/arxiv.2601.14994](https://doi.org/10.48550/arxiv.2601.14994) · [arXiv 2601.14994](https://arxiv.org/abs/2601.14994) · [OpenAlex](https://openalex.org/W7125372270)

**Citations:** 0 citations (openalex-snapshot · 2026-10-14)

## Summary
The paper asks whether benchmark contamination can go undetected when the leaked material is in another language. The authors simulate this by LoRA-fine-tuning four open-weight instruction-tuned models (Llama-3.2-1B-Instruct, Mistral-7B-Instruct-v0.2, Gemma-3-1B-it, Qwen3-1.7B) on English training data blended with 0, 10, 50 or 100% of the MMLU and XQuAD test questions rendered into Arabic. They then test the models on the original English benchmarks. On MMLU, English accuracy rises with Arabic exposure for every model. Two standard English-oriented detectors, TS-Guessing and Min-K%++, mostly fail to register this exposure. The authors propose TACD (Translation-Aware Contamination Detection), which needs no training data. It checks whether a model gives the same answers across English, Arabic and French versions of an item and under reordered answer choices. This cross-lingual agreement sits well above a chance baseline and usually grows with exposure, but it is model-dependent. The point for evaluators is that translation can hide leakage, so audits need multilingual behavioural checks. These checks should be read as evidence of suspicious behaviour, not as proof that the data was in the training set.

## Key points
- English MMLU macro-accuracy from 0% to 100% Arabic exposure: Mistral 0.577→0.690, LLaMA 0.332→0.431, Gemma 0.220→0.284, Qwen 0.553→0.581. XQuAD (ROUGE-L F1) shows mixed, weaker transfer.
- Min-K%++ AUROC stays at or below the 0.5 chance level (about 0.33–0.44 on MMLU). TS-Guessing is weak except for LLaMA's recall of the original answer position on MMLU (≈0.64 at 50% exposure).
- TACD cross-lingual consistency (CLC) against an independence baseline of ≈0.0625:
  - Gemma rises from 0.274 to 0.851.
  - Qwen peaks at 0.799 at 50% exposure and then drops, so the trend is not monotonic.
  - Mistral moves only 0.239→0.305 despite the largest accuracy gain.
- Probe and diagnostic results are averaged over three seeds and evaluated with the LM Evaluation Harness.
- The authors flag two confounds: adding Arabic data also increases total training volume, and the artificial LoRA exposure is not the same as natural pretraining leakage.

## Interesting ideas
- A system's answer to an item should not change when the item is re-expressed (another language, shuffled options); measuring agreement across such variants exposes behaviour that a single surface form hides. — For Mova (answer): Run school-test and fable-QA items through paraphrased or reordered variants and flag readers whose answers flip, separating real understanding from surface cues.
- If the model still picks the old answer position after the choices are shuffled, that is a sign of memorised position rather than reasoning, and it is cheap to test. — For Mova (learn): Use option shuffling as a standard negative control in evaluation, so a student that learned "answer is usually B" is caught automatically.
- Any agreement rate should be judged against what independent guessing would produce, not read as a raw number. — For Mova (learn): When several witness parsers agree on silver data, compare their agreement to an independence baseline per label, so trivial agreement on frequent labels is not mistaken for confidence.
- The authors propose separating agreement on correct predictions from agreement on shared errors, since consistency alone can come from invariance rather than knowledge. — For Mova (learn): Track how often witness parsers or LLM teacher tiers agree on wrong arcs or frames, because correlated errors would silently poison silver training data.
- Showing each signal as a deviation from its own reference (clean run, chance level) makes heterogeneous metrics comparable side by side. — For Mova (back): Report text → graph → text round-trip quality as a deviation from a baseline (e.g. an identity or bag-of-entities reconstruction), so gains in the compressor are measured against a meaningful floor.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{abbas2026obscuring,
  title = {{Obscuring Data Contamination Through Translation: Evidence from Arabic Corpora}},
  author = {Abbas, Chaymaa and Shamaa, Nour and Awad, Mariette},
  year = {2026},
  doi = {10.48550/arxiv.2601.14994},
  eprint = {2601.14994},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2601.14994}
}
```

- [← previous OasisSimp: An Open-source Asian-English Sentence Simplification Dataset](https://md.research.elamur.ai/free/en/papers/liu-2026-oasissimp-open-source-asian-english/)
- [next → OcWikiDialects: A Wikipedia Dataset With Rich Metadata for Occitan Dialect Identification](https://md.research.elamur.ai/free/en/papers/nedey-2026-ocwikidialects-wikipedia-dataset-rich-metadata/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/7/)

## References

16 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Simone Balloccu, Patrícia Schmidtová, Mateusz Lango, and Ondřej Dušek. 2024. Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source LLMs. Proceedings of the European Chapter of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Simone+Balloccu%2C+Patr%C3%ADcia+Schmidtov%C3%A1%2C+Mateusz+Lango%2C+and+Ondr%CC%8Cej+Du%C5%A1ek%2E+2024%2E+Leak%2C+cheat%2C+repeat%3A+Data+contamination+and+evaluation+malpractices+in+closed%2Dsource+LLMs%2E+Proceedings+of+the+European+Cha) [our card](https://md.research.elamur.ai/free/en/papers/balloccu-2024-leak-cheat-repeat-data-contamination/)
2. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877–1901. [find](https://scholar.google.com/scholar?q=Tom+B%2E+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretche)
3. Simin Chen, Yiming Chen, Zexin Li, Yifan Jiang, Zhongwei Wan, Yixin He, Dezhi Ran, Tianle Gu, Haizhou Li, Tao Xie, and Baishakhi Ray. 2025a. Benchmarking large language models under data contamination: A survey from static to dynamic evaluation. arXiv preprint arXiv:2502.17521. [↗](https://arxiv.org/abs/2502.17521)
4. Simin Chen, Pranav Pusarla, and Baishakhi Ray. 2025b. DyCodeEval: Dynamic benchmarking of reasoning capabilities in code large language models under data contamination. Proceedings of the International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Simin+Chen%2C+Pranav+Pusarla%2C+and+Baishakhi+Ray%2E+2025b%2E+DyCodeEval%3A+Dynamic+benchmarking+of+reasoning+capabilities+in+code+large+language+models+under+data+contamination%2E+Proceedings+of+the+Internationa)
5. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311. [↗](https://arxiv.org/abs/2204.02311)
6. Jasper Dekoninck, Mark Niklas Müller, and Martin Vechev. 2024. ConStat: Performance-based contamination detection in large language models. arXiv preprint arXiv:2405.16281. [↗](https://arxiv.org/abs/2405.16281)
7. Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. 2024. Investigating data contamination in modern benchmarks for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8698– 8711. [find](https://scholar.google.com/scholar?q=Chunyuan+Deng%2C+Yilun+Zhao%2C+Xiangru+Tang%2C+Mark+Gerstein%2C+and+Arman+Cohan%2E+2024%2E+Investigating+data+contamination+in+modern+benchmarks+for+large+language+models%2E+In+Proceedings+of+the+2024+Conference+of) [our card](https://md.research.elamur.ai/free/en/papers/deng-2024-investigating-data-contamination-modern-benchmarks/)
8. Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. 2024. Do membership inference attacks work on large language models? Conference on Language Modeling (COLM). [find](https://scholar.google.com/scholar?q=Michael+Duan%2C+Anshuman+Suri%2C+Niloofar+Mireshghallah%2C+Sewon+Min%2C+Weijia+Shi%2C+Luke+Zettlemoyer%2C+Yulia+Tsvetkov%2C+Yejin+Choi%2C+David+Evans%2C+and+Hannaneh+Hajishirzi%2E+2024%2E+Do+membership+inference+attacks+wo)
9. Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021. A framework for few-shot language model evaluation. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Jason+Phang%2C+Laria+Reynolds%2C+Eric+Tang%2C+Anish+Thite)
10. Vinay Samuel, Yue Zhou, and Henry Peng Zou. 2024. Towards data contamination detection for modern large language models: Limitations, inconsistencies, and oracle challenges. arXiv preprint arXiv:2409.09927. [↗](https://arxiv.org/abs/2409.09927)
11. Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2024. Detecting pretraining data from large language models. International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Weijia+Shi%2C+Anirudh+Ajith%2C+Mengzhou+Xia%2C+Yangsibo+Huang%2C+Daogao+Liu%2C+Terra+Blevins%2C+Danqi+Chen%2C+and+Luke+Zettlemoyer%2E+2024%2E+Detecting+pretraining+data+from+large+language+models%2E+International+Confere)
12. Alisha Srivastava, Emir Kaan Korukluoglu, Minh Nhat Le, Duyen Tran, Chau Minh Pham, Marzena Karpinska, and Mohit Iyyer. 2025. OWL: Probing cross-lingual recall of memorized texts via world literature. arXiv preprint arXiv:2505.22945. [↗](https://arxiv.org/abs/2505.22945)
13. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. [↗](https://arxiv.org/abs/2302.13971)
14. Feng Yao, Yufan Zhuang, Zihao Sun, Sunan Xu, Animesh Kumar, and Jingbo Shang. 2024. Data contamination can cross language barriers. Proceedings of the Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Feng+Yao%2C+Yufan+Zhuang%2C+Zihao+Sun%2C+Sunan+Xu%2C+Animesh+Kumar%2C+and+Jingbo+Shang%2E+2024%2E+Data+contamination+can+cross+language+barriers%2E+Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural+Langua)
15. Wentao Ye, Jiaqi Hu, Liyao Li, Haobo Wang, Gang Chen, and Junbo Zhao. 2024. Data contamination calibration for black-box LLMs. Findings of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Wentao+Ye%2C+Jiaqi+Hu%2C+Liyao+Li%2C+Haobo+Wang%2C+Gang+Chen%2C+and+Junbo+Zhao%2E+2024%2E+Data+contamination+calibration+for+black%2Dbox+LLMs%2E+Findings+of+the+Association+for+Computational+Linguistics%2E)
16. Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Frank Yang, and Hai Li. 2025. Min-K%++: Improved baseline for detecting pre-training data from large language models. International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Jingyang+Zhang%2C+Jingwei+Sun%2C+Eric+Yeats%2C+Yang+Ouyang%2C+Martin+Kuo%2C+Jianyi+Zhang%2C+Hao+Frank+Yang%2C+and+Hai+Li%2E+2025%2E+Min%2DK%25%2B%2B%3A+Improved+baseline+for+detecting+pre%2Dtraining+data+from+large+language+models)