---
title: "A multilingual hallucination benchmark — Thoresen 2026"
description: "Most hallucination research is done in English, so it is unclear whether its conclusions hold for less-resourced languages. 23 references."
authors: ["Freja Thoresen", "Dan Saattrup Smart"]
year: 2026
venue: "Proceedings of the Fourth Workshop on the Role of Resources in the Age of Large Language Models (RESOURCEFUL 2026)"
source: "https://aclanthology.org/2026.resourceful-4.17/"
doi: "10.63317/3g94ek9qx85o"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/thoresen-2026-multilingual-hallucination-benchmark/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/thoresen-2026-multilingual-hallucination-benchmark/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/20/) / A multilingual hallucination benchmark

# A multilingual hallucination benchmark

**Authors:** Freja Thoresen and [Dan Saattrup Smart](https://md.research.elamur.ai/free/en/authors/dan-saattrup-smart/) · **Year:** 2026 · **Venue:** [Proceedings of the Fourth Workshop on the Role of Resources in the Age of Large Language Models (RESOURCEFUL 2026)](https://md.research.elamur.ai/free/en/venues/resourceful/) ([Workshop on the Role of Resources in the Age of Large Language Models 2026](https://md.research.elamur.ai/free/en/venues/resourceful/2026/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.resourceful-4.17/) · [PDF](https://aclanthology.org/2026.resourceful-4.17.pdf) · [DOI 10.63317/3g94ek9qx85o](https://doi.org/10.63317/3g94ek9qx85o) · [OpenAlex](https://openalex.org/W7166553110)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Most hallucination research is done in English, so it is unclear whether its conclusions hold for less-resourced languages. This paper targets *faithfulness* hallucinations, where an answer sounds plausible but is not supported by the given context. The authors adapt the LettuceDetect framework to the MultiWikiQA reading-comprehension dataset. They use GPT-5-mini, via RAGFactChecker, to rewrite gold answers into subtly wrong versions with character-level spans marking each change. The result is a synthetic hallucination dataset for 306 languages. Pairing correct and corrupted answers, they fine-tune token-level hallucination detectors for 30 European languages, choosing mmBERT-small after comparing it with Ettin and EuroBERT. The detectors are then used to measure hallucination in five open LLMs on English, German, Danish and Icelandic. Hallucination rises as language resources shrink, and Icelandic is worst. The smallest model, Qwen3-0.6B, fails most often, while the Cogito models generally do best. The work provides a common, openly released benchmark for comparing context-grounded hallucination across many languages.

## Key points
- **Data setup:**
  - Each language has 4000 training and 1000 test context–question–answer triples from MultiWikiQA.
  - Hallucination intensity is sampled from a beta distribution (mean 0.2, SD 0.15), which biases the edits toward subtle errors.
  - The injected errors are of three types: factual, temporal and numerical.
- **Detector choice:** mmBERT-small (140m) beat Ettin-17m and EuroBERT-210m on both languages tested.
  - Danish: accuracy 0.8963, unsupported-token F1 0.8689.
  - German: accuracy 0.8948, unsupported-token F1 0.8627.
  - LettuceDetect gains Danish and Icelandic as newly supported languages.
- **Models evaluated:** Qwen3-0.6B, Qwen3-14B, Gemma-3-12B-IT, cogito-v1-preview-qwen-32B and cogito-v1-preview-llama-70B.
- **Main results:**
  - Qwen3-0.6B produces one or more hallucinated tokens in as many as 60% of its answers, peaking in Icelandic.
  - Qwen3-14B scored best on German.
  - The authors conclude that model size alone does not explain hallucination behaviour.
- **Released resources:** the 306-language dataset and 30 classifiers are on HuggingFace, and the code is on GitHub. The work was funded by EU Horizon TrustLLM.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Alexandra Institute](https://md.research.elamur.ai/free/en/companies/alexandra-institute/) — Company, Aarhus, Denmark · 2 works · 73 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 828 works

## Cite

Cite as: Freja Thoresen and Dan Saattrup Smart. 2026. A multilingual hallucination benchmark. Proceedings of the Fourth Workshop on the Role of Resources in the Age of Large Language Models (RESOURCEFUL 2026).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{thoresen2026multilingual,
  title = {{A multilingual hallucination benchmark}},
  author = {Freja Thoresen and Dan Saattrup Smart},
  booktitle = {Proceedings of the Fourth Workshop on the Role of Resources in the Age of Large Language Models (RESOURCEFUL 2026)},
  year = {2026},
  doi = {10.63317/3g94ek9qx85o},
  url = {https://doi.org/10.63317/3g94ek9qx85o}
}
```

- [← previous A Multi-View Media Profiling Suite: Resources, Evaluation, and Analysis](https://md.research.elamur.ai/free/en/papers/manzoor-2026-multi-view-media-profiling-suite/)
- [next → A Multilingual Red Teaming–Driven Safety Analysis of LLMs](https://md.research.elamur.ai/free/en/papers/pandeiro-2026-multilingual-red-teaming-driven-safety/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/20/)

## References

23 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yejin Bang, Ziwei Ji, Alan Schelten, Anthony Hartshorn, Tara Fowler, Cheng Zhang, Nicola Cancedda, and Pascale Fung. 2025. [HalluLens: LLM hallucination benchmark](https://md.research.elamur.ai/free/en/papers/bang-2025-hallulens-llm-hallucination-benchmark/). In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 24128–24156, Vi…
2. Nicolas Boizard, Hippolyte Gisserot-Boukhlef, Duarte M. Alves, André Martins, Ayoub Hammal, Caio Corro, Céline Hudelot, Emmanuel Malherbe, Etienne Malaboeuf, Fanny Jourdan, Gabriel Hautreux, João Alves, Kevin El-Haddad, Manuel Faysse, Maxime Peyrard, Nuno M. Guerreiro, Patrick Fernandes, Ricardo Rei… — [EuroBERT: Scaling Multilingual Encoders for European Languages](https://md.research.elamur.ai/free/en/papers/boizard-2025-eurobert-scaling-multilingual-encoders-european/)
3. Deep Cogito. 2025. Cogito v1 preview. Model release. [find](https://scholar.google.com/scholar?q=Deep+Cogito%2E+2025%2E+Cogito+v1+preview%2E+Model+release%2E)
4. Gemma Team. 2025. Gemma 3 technical report. [find](https://scholar.google.com/scholar?q=Gemma+Team%2E+2025%2E+Gemma+3+technical+report%2E)
5. Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitel, Sumit Sahrawat, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. [TRUE: Re-evaluating factual consistency evaluation](https://md.research.elamur.ai/free/en/papers/honovich-2022-true-re-evaluating-factual-consistency/). In Proceedings of the 2022 Conference of the North American Chapter of the Association…
6. Lianjin Huang, Weize Yu, Weiguang Ma, Wei Zhong, Zhen Feng, Haoran Wang, and Ting Liu. 2025. [A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions](https://md.research.elamur.ai/free/en/papers/huang-2023-survey-hallucination-large-language-models/). ACM Transactions on Information Systems, 43(2):1–55.
7. Akos Kovacs and Gabor Recski. 2025. Lettucedetect: A hallucination detection framework for rag applications. [find](https://scholar.google.com/scholar?q=Akos+Kovacs+and+Gabor+Recski%2E+2025%2E+Lettucedetect%3A+A+hallucination+detection+framework+for+rag+applications%2E)
8. Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023. [HaluEval: A large-scale hallucination evaluation benchmark for large language models](https://md.research.elamur.ai/free/en/papers/li-2023-halueval-large-scale-hallucination-evaluation/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6449–6464.
9. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. [TruthfulQA: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 3214–3252.
10. Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. 2023. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. [find](https://scholar.google.com/scholar?q=Potsawee+Manakul%2C+Adian+Liusie%2C+and+Mark+J%2E+F%2E+Gales%2E+2023%2E+Selfcheckgpt%3A+Zero%2Dresource+black%2Dbox+hallucination+detection+for+generative+large+language+models%2E)
11. Matt Marone, Oren Weller, William Fleshman, Eric Yang, Dawn Lawrie, and Benjamin Van Durme. 2025. [mmbert: A modern multilingual encoder with annealed language learning](https://md.research.elamur.ai/free/en/papers/marone-2025-mmbert-modern-multilingual-encoder-annealed/).
12. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. [FActScore: Fine-grained atomic evaluation of factual precision in long form text generation](https://md.research.elamur.ai/free/en/papers/min-2023-factscore-fine-grained-atomic-evaluation/). In Proceedings of the 2023 Conference on Empirical Methods in Natur…
13. OpenAI. 2024. Gpt-4o mini: Advancing cost-efficient intelligence. Blog post. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2024%2E+Gpt%2D4o+mini%3A+Advancing+cost%2Defficient+intelligence%2E+Blog+post%2E)
14. Donghao Ru, Liang Qiu, Xiaoyang Hu, Tong Zhang, Peng Shi, Shiyu Chang, and Zheng Zhang. 2024. Ragchecker: A fine-grained framework for diagnosing retrieval-augmented generation. Advances in Neural Information Processing Systems, 37:21999–22027. [find](https://scholar.google.com/scholar?q=Donghao+Ru%2C+Liang+Qiu%2C+Xiaoyang+Hu%2C+Tong+Zhang%2C+Peng+Shi%2C+Shiyu+Chang%2C+and+Zheng+Zhang%2E+2024%2E+Ragchecker%3A+A+fine%2Dgrained+framework+for+diagnosing+retrieval%2Daugmented+generation%2E+Advances+in+Neural+Inf)
15. Aohan Yang, An Li, Bo Yang, Bingchao Zhang, Bin Hui, Bo Zheng, and Zheng Qiu. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/).
16. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. 9. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2019%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+9%2E)
17. Language Resource References Cheng Niu and Yuanhao Wu and Juno Zhu and Siliang Xu and Kashun Shum and Randy Zhong and Juntong Song and Tong Zhang. 2024. [RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models](https://md.research.elamur.ai/free/en/papers/niu-2024-ragtruth-hallucination-corpus-developing-trustworthy/).
18. Dan Saattrup Smart. 2025. [MultiWikiQA: A Reading Comprehension Benchmark in 300+ Languages](https://md.research.elamur.ai/free/en/papers/smart-2026-multiwikiqa-reading-comprehension-benchmark-300/).
19. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. [How good is your tokenizer? on the monolingual performance of multilingual language models](https://md.research.elamur.ai/free/en/papers/rust-2021-how-good-is-your-tokenizer/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint…
20. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. [FEVER: a large-scale dataset for fact extraction and VERification](https://md.research.elamur.ai/free/en/papers/thorne-2018-fever-large-scale-dataset-fact/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pa…
21. Binjie Wang, Steffi Chern, Ethan Chern, and Pengfei Liu. 2024. [Halu-j: Critique-based hallucination judge](https://md.research.elamur.ai/free/en/papers/wang-2024-halu-j-critique-based-hallucination/). arXiv preprint arXiv:2407.12943.
22. Jason Wei, Nanyun Karina, Hyung Won Chung, Yao Jie Jiao, Stuart Papay, Aidan Glaese, and William Fedus. 2024. [Measuring short-form factuality in large language models](https://md.research.elamur.ai/free/en/papers/wei-2024-measuring-short-form-factuality-large/).
23. Oren Weller, Kaleab Ricci, Matt Marone, Alexander Chaffin, Dawn Lawrie, and Benjamin Van Durme. 2025. Seq vs seq: An open suite of paired encoders and decoders. [find](https://scholar.google.com/scholar?q=Oren+Weller%2C+Kaleab+Ricci%2C+Matt+Marone%2C+Alexander+Chaffin%2C+Dawn+Lawrie%2C+and+Benjamin+Van+Durme%2E+2025%2E+Seq+vs+seq%3A+An+open+suite+of+paired+encoders+and+decoders%2E)

*Updated 1 November 2026*