---
title: "When Flores Bloomz Wrong — Tan 2026"
description: "Multiway parallel benchmarks such as FLORES-200 may leak into LLM training data."
authors: ["David Tan", "Pinzhen Chen", "Josef van Genabith", "Koel Dutta Chowdhury"]
year: 2026
venue: "Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)"
source: "https://aclanthology.org/2026.eacl-short.26/"
doi: "10.18653/v1/2026.eacl-short.26"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/tan-2026-when-flores-bloomz-wrong-cross/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/tan-2026-when-flores-bloomz-wrong-cross/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2026](https://md.research.elamur.ai/free/papers/2026/page/5/) / When Flores Bloomz Wrong: Cross-Direction Contamination in Machine Translation Evaluation

# When Flores Bloomz Wrong: Cross-Direction Contamination in Machine Translation Evaluation

**Authors:** David Tan, [Pinzhen Chen](https://md.research.elamur.ai/free/authors/pinzhen-chen/), [Josef van Genabith](https://md.research.elamur.ai/free/authors/josef-van-genabith/), Koel Dutta Chowdhury · **Year:** 2026 · **Venue:** Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.eacl-short.26/) · [PDF](https://aclanthology.org/2026.eacl-short.26.pdf) · [DOI 10.18653/v1/2026.eacl-short.26](https://doi.org/10.18653/v1/2026.eacl-short.26) · [OpenAlex](https://openalex.org/W7140146338)

**Citations:** 0 citations (openalex-snapshot · 2026-10-14) · 2 in our library, of the 140673 papers whose bibliographies we read

## Summary
Multiway parallel benchmarks such as FLORES-200 may leak into LLM training data. This study asks whether that leakage spreads to translation directions the model never saw. The authors compare Bloomz-7B1, whose fine-tuning data is documented to include FLORES, with Llama-3.1-8B-Instruct, used as a presumably clean control. They translate all directions among 15 languages of high, medium and low resource levels on the FLORES dev set and score the output with BLEU and COMET.

Bloomz gets implausibly high BLEU into several target languages but fails on independent test sets, which points to memorization. To simulate contamination, they fine-tune Llama on eng↔xxx FLORES pairs. BLEU then rises for unseen pairs that share a memorized target language, so contamination is cross-directional and driven by the target side. Paraphrased or back-translated sources often still trigger the memorized references. Swapping named entities, however, reliably reduces BLEU, which makes it a practical probe for memorization. The authors advise checking contamination across all directions when evaluating on multiway parallel test sets.

## Key points
- **Setup:** FLORES-200 dev set (997 sentences) with 15 languages, from eng, zho, spa to bam, fon, ewe, kik and mri. Metrics are sacrebleu BLEU (surface overlap) and COMET-22-DA (meaning). aya-expanse-8b generated the replacement entities.
- **Bloomz results:** BLEU of roughly 40–90 into eng, fra, mal, ory, por, tam, vie and zho, but near 0 into bam, ewe, fon, kik and mri. On PMIndia and Mann-ki-Baat it scores close to zero for eng→{mal, ory, tam}, confirming memorization rather than real skill.
- **Source-side transfer:** Even ewe and mri, which Bloomz was never trained on, exceed 10 BLEU as sources into some targets, outperforming Llama.
- **Fine-tuning simulation:** Llama was fine-tuned with Axolotl on eng↔xxx data for 11 languages. BLEU on unseen directions rose by up to 16 points, except for targets vie and swa. COMET mostly dropped, except into mri and xho.
- **Code:** github.com/Mr-Ao-25/cross-ling-contamination. Stated limitations: only 7–8B models were tested, and fine-tuning may not reproduce contamination that happens during pre-training.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/shelves/nlp/)

Topics: [Machine translation](https://md.research.elamur.ai/free/topics/machine-translation/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{tan2026when,
  title = {{When Flores Bloomz Wrong: Cross-Direction Contamination in Machine Translation Evaluation}},
  author = {Tan, David and Chen, Pinzhen and Genabith, Josef van and Chowdhury, Koel Dutta},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-short.26},
  url = {https://doi.org/10.18653/v1/2026.eacl-short.26}
}
```

- [← previous What's in a Bridge?: A Descriptive, Multi-Genre Analysis of the GUMBridge Corpus for Varieties of Bridging Anaphora](https://md.research.elamur.ai/free/papers/levine-2026-what-s-bridge-descriptive-multi/)
- [next → When Similar Means Different: Evaluating LLMs on Arabic–Hebrew Cognates](https://md.research.elamur.ai/free/papers/langstream-2026-liang-similar-means-different-evaluating-llms/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/papers/2026/page/5/)

## References

33 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations. [↗](https://doi.org/10.48550/arxiv.2202.07646)
2. Koel Dutta Chowdhury, Cristina España-Bonet, and Josef van Genabith. 2020. Understanding translationese in multi-view embedding spaces. In Proceedings of the 28th international conference on computational linguistics, pages 6056–6062. [find](https://scholar.google.com/scholar?q=Koel+Dutta+Chowdhury%2C+Cristina+Espan%CC%83a%2DBonet%2C+and+Josef+van+Genabith%2E+2020%2E+Understanding+translationese+in+multi%2Dview+embedding+spaces%2E+In+Proceedings+of+the+28th+international+conference+on+computat)
3. John Dang, Shivalika Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, and others. 2024. Aya expanse: Combining research breakthroughs for a new multilingual frontier. arXiv preprint arXiv:2412.04261. [↗](https://arxiv.org/abs/2412.04261) [our card](https://md.research.elamur.ai/free/papers/dang-2024-aya-expanse-combining-research-breakthroughs/)
4. Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. arXiv preprint arXiv:2104.08758. [↗](https://arxiv.org/abs/2104.08758)
5. Koel Dutta Chowdhury, Cristina España-Bonet, and Josef van Genabith. 2021. Tracing source language interference in translation with graph-isomorphism measures. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 375–385, Held Online. INCOMA Ltd. [find](https://scholar.google.com/scholar?q=Koel+Dutta+Chowdhury%2C+Cristina+Espan%CC%83a%2DBonet%2C+and+Josef+van+Genabith%2E+2021%2E+Tracing+source+language+interference+in+translation+with+graph%2Disomorphism+measures%2E+In+Proceedings+of+the+International+Con)
6. Aparna Elangovan, Jiayuan He, and Karin Verspoor. 2021. Memorization vs. generalization : Quantifying data leakage in NLP performance evaluation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. [find](https://scholar.google.com/scholar?q=Aparna+Elangovan%2C+Jiayuan+He%2C+and+Karin+Verspoor%2E+2021%2E+Memorization+vs%2E+generalization+%3A+Quantifying+data+leakage+in+NLP+performance+evaluation%2E+In+Proceedings+of+the+16th+Conference+of+the+European+)
7. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, and others. 2020. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. [↗](https://arxiv.org/abs/2101.00027)
8. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. [↗](https://arxiv.org/abs/2407.21783)
9. Nuno M. Guerreiro, Elena Voita, and André Martins. 2023. Looking for a needle in a haystack: A comprehensive study of hallucinations in neural machine translation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nuno+M%2E+Guerreiro%2C+Elena+Voita%2C+and+Andre%CC%81+Martins%2E+2023%2E+Looking+for+a+needle+in+a+haystack%3A+A+comprehensive+study+of+hallucinations+in+neural+machine+translation%2E+In+Proceedings+of+the+17th+Conferen)
10. Barry Haddow and Faheem Kirefu. 2020. Pmindia–a collection of parallel corpora of languages of india. arXiv preprint arXiv:2001.09907. [↗](https://arxiv.org/abs/2001.09907)
11. Valentin Hartmann, Anshuman Suri, Vincent Bindschaedler, David Evans, Shruti Tople, and Robert West. 2023. Sok: Memorization in general-purpose large language models. arXiv preprint arXiv:2310.18362. [↗](https://arxiv.org/abs/2310.18362)
12. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, and others. 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556. [↗](https://arxiv.org/abs/2203.15556)
13. Matthew Honnibal, Ines Montani, Sofie Van Lan-deghem, and Adriane Boyd. 2020. spaCy: Industrial-strength Natural Language Processing in Python. [find](https://scholar.google.com/scholar?q=Matthew+Honnibal%2C+Ines+Montani%2C+Sofie+Van+Lan%2Ddeghem%2C+and+Adriane+Boyd%2E+2020%2E+spaCy%3A+Industrial%2Dstrength+Natural+Language+Processing+in+Python%2E)
14. Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg. 2023. Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Alon+Jacovi%2C+Avi+Caciularu%2C+Omer+Goldman%2C+and+Yoav+Goldberg%2E+2023%2E+Stop+uploading+test+data+in+plain+text%3A+Practical+strategies+for+mitigating+data+contamination+by+evaluation+benchmarks%2E+In+Proceedin)
15. Sparsh Jain, Ashwin Sankar, Devilal Choudhary, Dhairya Suman, Nikhil Narasimhan, Mohammed Safi Ur Rahman Khan, Anoop Kunchukuttan, Mitesh M Khapra, and Raj Dabre. 2024. Bhasaanuvaad: A speech translation dataset for 14 indian languages. arXiv preprint arXiv: 2411.04699. [↗](https://arxiv.org/abs/2411.04699)
16. Vikas Raunak, Arul Menezes, and Marcin Junczys-Dowmunt. 2021. The curious case of hallucinations in neural machine translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. [find](https://scholar.google.com/scholar?q=Vikas+Raunak%2C+Arul+Menezes%2C+and+Marcin+Junczys%2DDowmunt%2E+2021%2E+The+curious+case+of+hallucinations+in+neural+machine+translation%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+t) [our card](https://md.research.elamur.ai/free/papers/raunak-2021-curious-case-hallucinations-neural-machine/)
17. Minhao Jiang, Ken Ziyu Liu, Ming Zhong, Rylan Schaeffer, Siru Ouyang, Jiawei Han, and Sanmi Koyejo. 2024. Investigating data contamination for pre-training language models. arXiv preprint arXiv:2401.06059. [↗](https://arxiv.org/abs/2401.06059)
18. Vikas Raunak, Matt Post, and Arul Menezes. 2022. Salted: A framework for salient long-tail translation error detection. arXiv preprint arXiv:2205.09988. [↗](https://arxiv.org/abs/2205.09988)
19. Muhammed Yusuf Kocyigit, Eleftheria Briakou, Daniel Deutsch, Jiaming Luo, Colin Cherry, and Markus Freitag. 2025. Overestimation in LLM evaluation: A controlled large-scale study on data contamination’s impact on machine translation. In Forty-second International Conference on Machine Learning. [↗](https://doi.org/10.48550/arxiv.2501.18771)
20. Inbal Magar and Roy Schwartz. 2022. Data contamination: From memorization to exploitation. arXiv preprint arXiv:2203.08242. [↗](https://arxiv.org/abs/2203.08242)
21. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, and others. 2023. Crosslingual generalization through multitask finetuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). [find](https://scholar.google.com/scholar?q=Niklas+Muennighoff%2C+Thomas+Wang%2C+Lintang+Sutawika%2C+Adam+Roberts%2C+Stella+Biderman%2C+Teven+Le+Scao%2C+M+Saiful+Bari%2C+Sheng+Shen%2C+Zheng+Xin+Yong%2C+Hailey+Schoelkopf%2C+and+others%2E+2023%2E+Crosslingual+generaliza) [our card](https://md.research.elamur.ai/free/papers/muennighoff-2022-crosslingual-generalization-through-multitask-finetuning/)
22. Team NLLB, Marta R Costa-Jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, and others. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. [↗](https://arxiv.org/abs/2207.04672)
23. Yonatan Oren, Nicole Meister, Niladri S Chatterji, Faisal Ladhak, and Tatsunori Hashimoto. 2023. Proving test set contamination in black-box language models. In The Twelfth International Conference on Learning Representations. [↗](https://doi.org/10.48550/arxiv.2310.17623)
24. Xudong Pan, Mi Zhang, Shouling Ji, and Min Yang. 2020. Privacy risks of general-purpose language models. In 2020 IEEE Symposium on Security and Privacy (SP). [find](https://scholar.google.com/scholar?q=Xudong+Pan%2C+Mi+Zhang%2C+Shouling+Ji%2C+and+Min+Yang%2E+2020%2E+Privacy+risks+of+general%2Dpurpose+language+models%2E+In+2020+IEEE+Symposium+on+Security+and+Privacy+%28SP%29%2E)
25. Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation. [find](https://scholar.google.com/scholar?q=Ricardo+Rei%2C+Jose%CC%81+G%2E+C%2E+de+Souza%2C+Duarte+Alves%2C+Chrysoula+Zerva%2C+Ana+C+Farinha%2C+Taisiya+Glushkova%2C+Alon+Lavie%2C+Luisa+Coheur%2C+and+Andre%CC%81+F%2E+T%2E+Martins%2E+2022%2E+COMET%2D22%3A+Unbabel%2DIST+2022+submission+for+) [our card](https://md.research.elamur.ai/free/papers/rei-2022-comet-22-unbabel-ist-2022/)
26. Iker Sainz, Oscar aFnd Garcı́a-Ferrero, Alon Jacovi, Jon Ander Campos, Yanai Elazar, Eneko Agirre, Yoav Goldberg, Wei-Lin Chen, Jenny Chim, Leshem Choshen, Luca D’Amico-Wong, and others. 2024. Data contamination report from the 2024 CONDA shared task. In Proceedings of the 1st Workshop on Data Contamination (CONDA). [find](https://scholar.google.com/scholar?q=Iker+Sainz%2C+Oscar+aFnd+Garc%C4%B1%CC%81a%2DFerrero%2C+Alon+Jacovi%2C+Jon+Ander+Campos%2C+Yanai+Elazar%2C+Eneko+Agirre%2C+Yoav+Goldberg%2C+Wei%2DLin+Chen%2C+Jenny+Chim%2C+Leshem+Choshen%2C+Luca+D%E2%80%99Amico%2DWong%2C+and+others%2E+2024%2E+Data+co)
27. Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2024. Detecting pretraining data from large language models. In The Twelfth International Conference on Learning Representations. [↗](https://doi.org/10.48550/arxiv.2310.16789)
28. Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E Gonzalez, and Ion Stoica. 2023. Rethinking benchmark and contamination for language models with rephrased samples. arXiv preprint arXiv:2311.04850. [↗](https://arxiv.org/abs/2311.04850) [our card](https://md.research.elamur.ai/free/papers/yang-2023-rethinking-benchmark-contamination-language-models/)
29. Feng Yao, Yufan Zhuang, Zihao Sun, Sunan Xu, Animesh Kumar, and Jingbo Shang. 2024. Data contamination can cross language barriers. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Feng+Yao%2C+Yufan+Zhuang%2C+Zihao+Sun%2C+Sunan+Xu%2C+Animesh+Kumar%2C+and+Jingbo+Shang%2E+2024%2E+Data+contamination+can+cross+language+barriers%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Natura)
30. Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. 2023. Don’t make your llm an evaluation benchmark cheater. arXiv preprint arXiv:2311.01964. [↗](https://arxiv.org/abs/2311.01964)
31. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+Bleu%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+the+40th+Annual+Meeting+of+the+Association+for+Comp) [our card](https://md.research.elamur.ai/free/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
32. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers. [find](https://scholar.google.com/scholar?q=Matt+Post%2E+2018%2E+A+call+for+clarity+in+reporting+BLEU+scores%2E+In+Proceedings+of+the+Third+Conference+on+Machine+Translation%3A+Research+Papers%2E) [our card](https://md.research.elamur.ai/free/papers/post-2018-call-clarity-reporting-bleu-scores/)
33. Vikas Raunak and Arul Menezes. 2022. Finding memo: Extractive memorization in constrained sequence generation tasks. In Findings of the Association for Computational Linguistics: EMNLP 2022. [↗](https://doi.org/10.48448/kpvf-rc93)