---
title: "Comparing Hallucination Detection Metrics for Multilingual"
description: "They compare three kinds of low-resource-friendly detectors: lexical overlap (ROUGE) Named Entity… 35 references, cited by 15 papers here."
authors: ["Haoqiang Kang", "Terra Blevins", "Luke Zettlemoyer"]
year: 2024
venue: "arXiv:2402.10496"
source: "https://arxiv.org/abs/2402.10496"
doi: "10.48550/arxiv.2402.10496"
arxiv: "2402.10496"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kang-2024-comparing-hallucination-detection-metrics-multilingual/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kang-2024-comparing-hallucination-detection-metrics-multilingual/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/) / Comparing Hallucination Detection Metrics for Multilingual Generation

# Comparing Hallucination Detection Metrics for Multilingual Generation

**Authors:** Haoqiang Kang, [Terra Blevins](https://md.research.elamur.ai/free/en/authors/terra-blevins/) and [Luke Zettlemoyer](https://md.research.elamur.ai/free/en/authors/luke-zettlemoyer/) · **Year:** 2024 · **Venue:** arXiv:2402.10496 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2402.10496) · [PDF](https://arxiv.org/pdf/2402.10496) · [DOI 10.48550/arxiv.2402.10496](https://doi.org/10.48550/arxiv.2402.10496) · [arXiv 2402.10496](https://arxiv.org/abs/2402.10496) · [OpenAlex](https://openalex.org/W4391940796)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [15](https://md.research.elamur.ai/free/en/papers/kang-2024-comparing-hallucination-detection-metrics-multilingual/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

Methods for spotting factual hallucinations in LLM output have mostly been tested on English, so it is unclear how well they work in other languages. (Note: the arXiv v2 title uses "Methods," not "Metrics.") The authors built a multilingual testbed in which BLOOMZ-mt writes biographies of the same 500 people in 19 languages. Each output is scored against the person's Wikipedia article (reference-based) and against other samples generated for the same person (pairwise). They compare three kinds of low-resource-friendly detectors:

- lexical overlap (ROUGE)
- Named Entity Overlap
- zero-shot NLI scoring built on SummaC

The scores are checked against each other, against human factuality labels for English and Chinese, and against two supervised detectors.

Lexical metrics turn out to be nearly useless. NLI-based scores track human judgments at the sentence level for high-resource languages and can beat supervised systems on hallucinations that the reference can verify or refute. However, NLI scores struggle with individual atomic facts and degrade sharply for weaker languages. The takeaway for practitioners is that detection quality depends on how well resourced a language is. Ukrainian sits on the weak end.

## Key points

- **Dataset:** 19 languages, including Ukrainian, Russian, and Bulgarian. People were chosen via WikiData, covered by Wikipedia in every language, and spread across 2022 page-view levels. Prompts were machine-translated with Google Translate, and wrong-language outputs were filtered using langdetect.
- **Generation setup:** BLOOMZ-mt with nucleus sampling (top_p = 0.9) and five samples per prompt.
- **Generation quality for Ukrainian:** outputs average only 5.7 tokens and 0.40 sentences. Just 35.24% of outputs are detectable as valid text, and about 42% of those are actually in Ukrainian. Across all languages, language accuracy averages about 50%.
- **NLI detectors:** the NLI scores (ENT, CON, DIFF, plus UNV for unverifiable claims) outperform ROUGE and NEO. Pairwise NLI agrees with reference-based NLI in high-resource languages, but this agreement fades for low-resource ones.
- **Supervised detectors:** mFACT and seahorse-Q4 (mT5-large trained on SEAHORSE Q4) are better at unverifiable hallucinations but slightly worse on verifiable ones. seahorse-Q4 breaks down on Chinese, which it never saw in training. Human evaluation covers only English and Chinese.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 353 works · 200,043 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) · 577 works

🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 828 works

## Cite

Cite as: Haoqiang Kang, Terra Blevins and Luke Zettlemoyer. 2024. Comparing Hallucination Detection Metrics for Multilingual Generation. arXiv:2402.10496.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{kang2024comparing,
  title = {{Comparing Hallucination Detection Metrics for Multilingual Generation}},
  author = {Haoqiang Kang, Terra Blevins and Luke Zettlemoyer},
  year = {2024},
  doi = {10.48550/arxiv.2402.10496},
  eprint = {2402.10496},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.10496}
}
```

- [← previous Common Ground Tracking in Multimodal Dialogue](https://md.research.elamur.ai/free/en/papers/khebour-2024-common-ground-tracking-multimodal-dialogue/)
- [next → Cultural Commonsense Knowledge for Intercultural Dialogues](https://md.research.elamur.ai/free/en/papers/nguyen-2024-cultural-commonsense-knowledge-intercultural-dialogues/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/)

## References

35 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Roee Aharoni, Shashi Narayan, Joshua Maynez, Jonathan Herzig, Elizabeth Clark, and Mirella Lapata. 2022. mface: Multilingual summarization with factual consistency evaluation. arXiv preprint arXiv:2212.10622. [find](https://scholar.google.com/scholar?q=Roee+Aharoni%2C+Shashi+Narayan%2C+Joshua+Maynez%2C+Jonathan+Herzig%2C+Elizabeth+Clark%2C+and+Mirella+Lapata%2E+2022%2E+mface%3A+Multilingual+summarization+with+factual+consistency+evaluation%2E+arXiv+preprint+arXiv%3A221)
2. Kabir Ahuja, Rishav Hada, Millicent Ochieng, Prachi Jain, Harshita Diddee, Samuel Maina, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, et al. 2023. [Mega: Multilingual evaluation of generative ai](https://md.research.elamur.ai/free/en/papers/ahuja-2023-mega-multilingual-evaluation-generative-ai/). arXiv preprint arXiv:2303.12528.
3. Rachel Bawden and François Yvon. 2023. [Investigating the translation performance of a large multilingual language model: the case of bloom](https://md.research.elamur.ai/free/en/papers/bawden-2023-investigating-translation-performance-large-multilingual/). In Proceedings of the 24th Annual Conference of the European Association for Machine Translation.
4. I Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, Pengfei Liu, et al. 2023. [Factool: Factuality detection in generative ai–a tool augmented framework for multi-task and multi-domain scenarios](https://md.research.elamur.ai/free/en/papers/chern-2023-factool-factuality-detection-generative-ai/). arXiv preprint arXiv:2307.13528.
5. Elizabeth Clark, Shruti Rijhwani, Sebastian Gehrmann, Joshua Maynez, Roee Aharoni, Vitaly Nikolaev, Thibault Sellam, Aditya Siddhant, Dipanjan Das, and Ankur P Parikh. 2023. Seahorse: A multilingual, multifaceted dataset for summarization evaluation. arXiv preprint arXiv:2305.13194. [find](https://scholar.google.com/scholar?q=Elizabeth+Clark%2C+Shruti+Rijhwani%2C+Sebastian+Gehrmann%2C+Joshua+Maynez%2C+Roee+Aharoni%2C+Vitaly+Nikolaev%2C+Thibault+Sellam%2C+Aditya+Siddhant%2C+Dipanjan+Das%2C+and+Ankur+P+Parikh%2E+2023%2E+Seahorse%3A+A+multilingual%2C+)
6. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzm%C3%A1n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+cross%2Dlingu)
7. Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Guillaume+Lample%2C+Ruty+Rinott%2C+Adina+Williams%2C+Samuel+R%2E+Bowman%2C+Holger+Schwenk%2C+and+Veselin+Stoyanov%2E+2018%2E+Xnli%3A+Evaluating+cross%2Dlingual+sentence+representations%2E)
8. David Dale, Elena Voita, Loïc Barrault, and Marta R Costa-jussà. 2022. [Detecting and mitigating hallucinations in machine translation: Model internal workings alone do well, sentence similarity even better](https://md.research.elamur.ai/free/en/papers/dale-2023-detecting-mitigating-hallucinations-machine-translation/). arXiv preprint arXiv:2212.08597.
9. Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2023. [Chain-of-verification reduces hallucination in large language models](https://md.research.elamur.ai/free/en/papers/dhuliawala-2023-chain-verification-reduces-hallucination-large/). arXiv preprint arXiv:2309.11495.
10. Mohamed Elaraby, Mengyin Lu, Jacob Dunn, Xueying Zhang, Yu Wang, Shizhu Liu, Pingchuan Tian, Yuping Wang, and Yuxuan Wang. 2023. [Halo: Estimation and reduction of hallucinations in open-source weak large language models](https://md.research.elamur.ai/free/en/papers/elaraby-2023-halo-estimation-reduction-hallucinations-open/).
11. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. [The curious case of neural text degeneration](https://md.research.elamur.ai/free/en/papers/holtzman-2019-curious-case-neural-text-degeneration/). In International Conference on Learning Representations.
12. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2023. [A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions](https://md.research.elamur.ai/free/en/papers/huang-2023-survey-hallucination-large-language-models/). arXiv preprint arXiv:2311.05232.
13. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. [Survey of hallucination in natural language generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/). ACM Computing Surveys, 55(12):1–38.
14. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221. [find](https://scholar.google.com/scholar?q=Saurav+Kadavath%2C+Tom+Conerly%2C+Amanda+Askell%2C+Tom+Henighan%2C+Dawn+Drain%2C+Ethan+Perez%2C+Nicholas+Schiefer%2C+Zac+Hatfield%2DDodds%2C+Nova+DasSarma%2C+Eli+Tran%2DJohnson%2C+et+al%2E+2022%2E+Language+models+%28mostly%29+know+w)
15. Haoqiang Kang, Terra Blevins, and Luke Zettlemoyer. 2023. Translate to disambiguate: Zero-shot multilingual word sense disambiguation with pretrained language models. arXiv preprint arXiv:2304.13803. [find](https://scholar.google.com/scholar?q=Haoqiang+Kang%2C+Terra+Blevins%2C+and+Luke+Zettlemoyer%2E+2023%2E+Translate+to+disambiguate%3A+Zero%2Dshot+multilingual+word+sense+disambiguation+with+pretrained+language+models%2E+arXiv+preprint+arXiv%3A2304%2E13803%2E)
16. Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2021. Summac: Re-visiting nli-based models for inconsistency detection in summarization. [find](https://scholar.google.com/scholar?q=Philippe+Laban%2C+Tobias+Schnabel%2C+Paul+N%2E+Bennett%2C+and+Marti+A%2E+Hearst%2E+2021%2E+Summac%3A+Re%2Dvisiting+nli%2Dbased+models+for+inconsistency+detection+in+summarization%2E)
17. Viet Lai, Nghia Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Nguyen. 2023. [ChatGPT beyond English: Towards a comprehensive evaluation of large language models in multilingual learning](https://md.research.elamur.ai/free/en/papers/lai-2023-chatgpt-beyond-english-comprehensive-evaluation/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages…
18. Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, et al. 2022. The bigscience roots corpus: A 1.6 tb composite multilingual dataset. Advances in Neural Information Proc… [find](https://scholar.google.com/scholar?q=Hugo+Lauren%C3%A7on%2C+Lucile+Saulnier%2C+Thomas+Wang%2C+Christopher+Akiki%2C+Albert+Villanova+del+Moral%2C+Teven+Le+Scao%2C+Leandro+Von+Werra%2C+Chenghao+Mou%2C+Eduardo+Gonz%C3%A1lez+Ponferrada%2C+Huu+Nguyen%2C+et+al%2E+2022%2E+The+b)
19. Katherine Lee, Orhan Firat, Ashish Agarwal, Clara Fannjiang, and David Sussillo. 2019. Hallucinations in neural machine translation. [find](https://scholar.google.com/scholar?q=Katherine+Lee%2C+Orhan+Firat%2C+Ashish+Agarwal%2C+Clara+Fannjiang%2C+and+David+Sussillo%2E+2019%2E+Hallucinations+in+neural+machine+translation%2E)
20. Chin-Yew Lin. 2004. [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text summarization branches out, pages 74–81.
21. Zhenzhen Liu, Chao Wan, Varsha Kishore, Jin Peng Zhou, Minmin Chen, and Kilian Q Weinberger. 2023. Correction with backtracking reduces hallucination in summarization. arXiv preprint arXiv:2310.16176. — [arXiv:2310.16176](https://arxiv.org/abs/2310.16176) [find](https://scholar.google.com/scholar?q=Zhenzhen+Liu%2C+Chao+Wan%2C+Varsha+Kishore%2C+Jin+Peng+Zhou%2C+Minmin+Chen%2C+and+Kilian+Q+Weinberger%2E+2023%2E+Correction+with+backtracking+reduces+hallucination+in+summarization%2E+arXiv+preprint+arXiv%3A2310%2E16176%2E)
22. Cheng Luo, Wei Liu, Jieyu Lin, Jiajie Zou, Ming Xiang, and Nai Ding. 2022. [Simple but challenging: Natural language inference models fail on simple sentences](https://md.research.elamur.ai/free/en/papers/luo-2022-simple-but-challenging-natural-language/). In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3449–3462, Abu Dhabi, United Arab Emirates. Association for C…
23. Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896. [find](https://scholar.google.com/scholar?q=Potsawee+Manakul%2C+Adian+Liusie%2C+and+Mark+JF+Gales%2E+2023%2E+Selfcheckgpt%3A+Zero%2Dresource+black%2Dbox+hallucination+detection+for+generative+large+language+models%2E+arXiv+preprint+arXiv%3A2303%2E08896%2E)
24. Sabrina J Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. 2022. Reducing conversational agents’ overconfidence through linguistic calibration. Transactions of the Association for Computational Linguistics, 10:857–872. [find](https://scholar.google.com/scholar?q=Sabrina+J+Mielke%2C+Arthur+Szlam%2C+Emily+Dinan%2C+and+Y%2DLan+Boureau%2E+2022%2E+Reducing+conversational+agents%E2%80%99+overconfidence+through+linguistic+calibration%2E+Transactions+of+the+Association+for+Computational+L)
25. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. [Factscore: Fine-grained atomic evaluation of factual precision in long form text generation](https://md.research.elamur.ai/free/en/papers/min-2023-factscore-fine-grained-atomic-evaluation/). arXiv preprint arXiv:2305.14251.
26. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel.… — [Crosslingual Generalization through Multitask Finetuning](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-crosslingual-generalization-through-multitask-finetuning/)
27. Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero Nogueira dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen McKeown, and Bing Xiang. 2021. [Entity-level factual consistency of abstractive text summarization](https://md.research.elamur.ai/free/en/papers/nan-2021-entity-level-factual-consistency-abstractive/). In Proceedings of the 16th Conference of the European Chapter of the Association for Computation…
28. Yifu Qiu, Yftah Ziser, Anna Korhonen, Edoardo M Ponti, and Shay B Cohen. 2023. Detecting and mitigating hallucinations in multilingual summarisation. arXiv preprint arXiv:2305.13632. [find](https://scholar.google.com/scholar?q=Yifu+Qiu%2C+Yftah+Ziser%2C+Anna+Korhonen%2C+Edoardo+M+Ponti%2C+and+Shay+B+Cohen%2E+2023%2E+Detecting+and+mitigating+hallucinations+in+multilingual+summarisation%2E+arXiv+preprint+arXiv%3A2305%2E13632%2E)
29. Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. 2023. [Measuring attribution in natural language generation models](https://md.research.elamur.ai/free/en/papers/rashkin-2023-measuring-attribution-natural-language-generation/). Computational Linguistics, 49(4):777–840.
30. Vikas Raunak, Arul Menezes, and Marcin Junczys-Dowmunt. 2021. [The curious case of hallucinations in neural machine translation](https://md.research.elamur.ai/free/en/papers/raunak-2021-curious-case-hallucinations-neural-machine/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1172–1183, Online.…
31. Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jianshu Chen, and Dong Yu. 2023. [A stitch in time saves nine: Detecting and mitigating hallucinations of llms by validating low-confidence generation](https://md.research.elamur.ai/free/en/papers/varshney-2023-stitch-time-saves-nine-detecting/). arXiv preprint arXiv:2307.03987.
32. BigScience Workshop. 2023. Bloom: A 176b-parameter open-access multilingual language model. [find](https://scholar.google.com/scholar?q=BigScience+Workshop%2E+2023%2E+Bloom%3A+A+176b%2Dparameter+open%2Daccess+multilingual+language+model%2E)
33. Weijia Xu, Sweta Agrawal, Eleftheria Briakou, Marianna J Martindale, and Marine Carpuat. 2023. [Understanding and detecting hallucinations in neural machine translation via model introspection](https://md.research.elamur.ai/free/en/papers/xu-2023-understanding-detecting-hallucinations-neural-machine/). Transactions of the Association for Computational Linguistics, 11:546–564.
34. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. [mT5: A massively multilingual pre-trained text-to-text transformer](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling…
35. Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. 2023. [Siren’s song in the ai ocean: A survey on hallucination in large language models](https://md.research.elamur.ai/free/en/papers/zhang-2023-siren-s-song-ai-ocean/). arXiv preprint arXiv:2309.01219.

*Updated 31 October 2026*