---
title: "Constructing a Dataset for Hallucination Detection in Japanese"
description: "Japanese has few manually annotated benchmarks for checking whether LLM output stays grounded in its input. 15 references."
authors: ["Hikari Tanaka", "Atsushi Keyaki", "Mamoru Komachi"]
year: 2026
venue: "Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop)"
source: "https://aclanthology.org/2026.eacl-srw.15/"
doi: "10.18653/v1/2026.eacl-srw.15"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/tanaka-2026-constructing-dataset-hallucination-detection-japanese/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tanaka-2026-constructing-dataset-hallucination-detection-japanese/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/26/) / Constructing a Dataset for Hallucination Detection in Japanese Summarization with Fine-grained Faithfulness Labels

# Constructing a Dataset for Hallucination Detection in Japanese Summarization with Fine-grained Faithfulness Labels

**Authors:** [Hikari Tanaka](https://md.research.elamur.ai/free/en/authors/hikari-tanaka/), [Atsushi Keyaki](https://md.research.elamur.ai/free/en/authors/atsushi-keyaki/) and [Mamoru Komachi](https://md.research.elamur.ai/free/en/authors/mamoru-komachi/) · **Year:** 2026 · **Venue:** [Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2026](https://md.research.elamur.ai/free/en/venues/eacl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.eacl-srw.15/) · [PDF](https://aclanthology.org/2026.eacl-srw.15.pdf) · [DOI 10.18653/v1/2026.eacl-srw.15](https://doi.org/10.18653/v1/2026.eacl-srw.15) · [OpenAlex](https://openalex.org/W7140108042)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Japanese has few manually annotated benchmarks for checking whether LLM output stays grounded in its input. Existing Japanese resources either use synthetic inconsistencies or, like JHARS, focus on question answering and contain few hallucinated cases. The authors fill this gap for summarization. Three LLMs summarized Japanese news articles, and annotators labeled every summary sentence by hand against its source, with more than one label allowed per sentence. The label scheme starts from the intrinsic/extrinsic distinction of Maynez et al. (2020). It adds Paraphrase Error, a faithfulness problem in which rewording changes the meaning without counting as hallucination. The authors observed this pattern often in modern, more abstractive outputs. The resource supports binary hallucination detection, binary faithfulness detection and fine-grained error analysis. GPT-4o hallucinated less than the other two models but paraphrased more. Hallucinations seldom coincided across models on the same input, which points to model-specific causes rather than difficult inputs. The dataset provides a basis for developing and evaluating Japanese hallucination detectors, including methods that use internal model states.

## Key points
- **Dataset size:** 390 LLM-generated summaries (1,938 sentences) of Japanese news documents, produced by three LLMs and annotated at sentence level with multiple labels allowed.
- **Labels:** four faithfulness labels: Intrinsic, Extrinsic, Paraphrase Error and Faithful. Paraphrase Error is a new non-hallucination category added to the Maynez et al. (2020) taxonomy.
- **Scope:** the work targets context inconsistency (faithfulness hallucination, following Huang et al., 2025). Factuality against world knowledge is not annotated.
- **Model findings:** GPT-4o had the lowest hallucination rate but the most paraphrasing. Hallucination occurrence was largely independent across the three models.
- **Annotator disagreement:** disagreements arose in two kinds of cases:
  - omitted restrictive phrases that fit no defined label;
  - entity swaps such as Weibo→WeChat, where the choice between Extrinsic and Paraphrase depends on the annotator's background knowledge.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Hitotsubashi University](https://md.research.elamur.ai/free/en/universities/hitotsubashi-university/) — University, Tokyo, Japan · 6 works · 28 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [📝 Summarization](https://md.research.elamur.ai/free/en/topics/summarization/) · 232 works

## Cite

Cite as: Hikari Tanaka, Atsushi Keyaki and Mamoru Komachi. 2026. Constructing a Dataset for Hallucination Detection in Japanese Summarization with Fine-grained Faithfulness Labels. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{tanaka2026constructing,
  title = {{Constructing a Dataset for Hallucination Detection in Japanese Summarization with Fine-grained Faithfulness Labels}},
  author = {Hikari Tanaka, Atsushi Keyaki and Mamoru Komachi},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-srw.15},
  url = {https://doi.org/10.18653/v1/2026.eacl-srw.15}
}
```

- [← previous Constraints on Linking Element Choice in German Nominal Compounding: A Large-Scale Corpus Study](https://md.research.elamur.ai/free/en/papers/shmalts-2026-constraints-linking-element-choice-german/)
- [next → Constructing a Japanese Rap Lyric Generation Model with GRPO](https://md.research.elamur.ai/free/en/papers/ogawa-2026-constructing-japanese-rap-lyric-generation/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/26/)

## References

15 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. 2024. INSIDE: LLMs’ internal states retain the power of hallucination detection. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Chao+Chen%2C+Kai+Liu%2C+Ze+Chen%2C+Yi+Gu%2C+Yue+Wu%2C+Mingyuan+Tao%2C+Zhihang+Fu%2C+and+Jieping+Ye%2E+2024%2E+INSIDE%3A+LLMs%E2%80%99+internal+states+retain+the+power+of+hallucination+detection%2E+In+The+Twelfth+International+Conf)
2. Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. [Evaluating the factual consistency of abstractive text summarization](https://md.research.elamur.ai/free/en/papers/kryscinski-2020-evaluating-factual-consistency-abstractive-text/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332–9346, Online. Association for Computational…
3. Esin Durmus, He He, and Mona Diab. 2020. [FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization](https://md.research.elamur.ai/free/en/papers/durmus-2020-feqa-question-answering-evaluation-framework/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5055–5070, Online. Association for Computational Lingui…
4. Barrett Lattimer, Patrick Chen, Xinyuan Zhang, and Yi Yang. 2023. Fast and accurate factual inconsistency detection over long documents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, page 1691 – 1703. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Barrett+Lattimer%2C+Patrick+Chen%2C+Xinyuan+Zhang%2C+and+Yi+Yang%2E+2023%2E+Fast+and+accurate+factual+inconsistency+detection+over+long+documents%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+Methods+in+N)
5. Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. [RAGAs: Automated evaluation of retrieval augmented generation](https://md.research.elamur.ai/free/en/papers/es-2023-ragas-automated-evaluation-retrieval-augmented/). In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 150–158, St. Julians,…
6. Wei Li, Wenhao Wu, Moye Chen, Jiachen Liu, Xinyan Xiao, and Hua Wu. 2022. Faithfulness in natural language generation: A systematic survey of analysis, evaluation and optimization methods. arXiv preprint arXiv:2203.05227 [cs.CL]. Preprint, arXiv:2203.05227. — [arXiv:2203.05227](https://arxiv.org/abs/2203.05227) [find](https://scholar.google.com/scholar?q=Wei+Li%2C+Wenhao+Wu%2C+Moye+Chen%2C+Jiachen+Liu%2C+Xinyan+Xiao%2C+and+Hua+Wu%2E+2022%2E+Faithfulness+in+natural+language+generation%3A+A+systematic+survey+of+analysis%2C+evaluation+and+optimization+methods%2E+arXiv+prepr)
7. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. [A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions](https://md.research.elamur.ai/free/en/papers/huang-2023-survey-hallucination-large-language-models/). ACM Trans. Inf. Syst., 43(2).
8. Potsawee Manakul, Adian Liusie, and Mark Gales. 2023. SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9004–9017, Singapore. Association for Computation… [find](https://scholar.google.com/scholar?q=Potsawee+Manakul%2C+Adian+Liusie%2C+and+Mark+Gales%2E+2023%2E+SelfCheckGPT%3A+Zero%2Dresource+black%2Dbox+hallucination+detection+for+generative+large+language+models%2E+In+Proceedings+of+the+2023+Conference+on+Empir)
9. Keisuke Iwamoto and Kazutaka Shimada. 2024. Dataset construction and verification for detecting factual inconsistency in Japanese summarization. In 2024 16th IIAI International Congress on Advanced Applied Informatics (IIAI-AAI), pages 243–248. [find](https://scholar.google.com/scholar?q=Keisuke+Iwamoto+and+Kazutaka+Shimada%2E+2024%2E+Dataset+construction+and+verification+for+detecting+factual+inconsistency+in+Japanese+summarization%2E+In+2024+16th+IIAI+International+Congress+on+Advanced+Ap)
10. Ziwei Ji, Yuzhe Gu, Wenwei Zhang, Chengqi Lyu, Dahua Lin, and Kai Chen. 2024. ANAH: Analytical annotation of hallucinations in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8135–8158, Bangkok, Thailand… [find](https://scholar.google.com/scholar?q=Ziwei+Ji%2C+Yuzhe+Gu%2C+Wenwei+Zhang%2C+Chengqi+Lyu%2C+Dahua+Lin%2C+and+Kai+Chen%2E+2024%2E+ANAH%3A+Analytical+annotation+of+hallucinations+in+large+language+models%2E+In+Proceedings+of+the+62nd+Annual+Meeting+of+the+A)
11. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. [On faithfulness and factuality in abstractive summarization](https://md.research.elamur.ai/free/en/papers/maynez-2020-faithfulness-factuality-abstractive-summarization/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, Online. Association for Computational Linguistics.
12. Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, Kashun Shum, Randy Zhong, Juntong Song, and Tong Zhang. 2024. [RAGTruth: A hallucination corpus for developing trustworthy retrieval-augmented language models](https://md.research.elamur.ai/free/en/papers/niu-2024-ragtruth-hallucination-corpus-developing-trustworthy/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volu…
13. Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. [Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics](https://md.research.elamur.ai/free/en/papers/pagnoni-2021-understanding-factuality-abstractive-summarization-frank/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Languag…
14. Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanan, and Peter J Liu. 2023. Out-of-distribution detection and selective generation for conditional language models. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Jie+Ren%2C+Jiaming+Luo%2C+Yao+Zhao%2C+Kundan+Krishna%2C+Mohammad+Saleh%2C+Balaji+Lakshminarayanan%2C+and+Peter+J+Liu%2E+2023%2E+Out%2Dof%2Ddistribution+detection+and+selective+generation+for+conditional+language+models%2E+)
15. Zhongxiang Sun, Xiaoxue Zang, Kai Zheng, Yang Song, Jun Xu, Xiao Zhang, Weijie Yu, Yang Song, and Han Li. 2025. ReDeEP: Detecting hallucination in retrieval-augmented generation via mechanistic interpretability. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zhongxiang+Sun%2C+Xiaoxue+Zang%2C+Kai+Zheng%2C+Yang+Song%2C+Jun+Xu%2C+Xiao+Zhang%2C+Weijie+Yu%2C+Yang+Song%2C+and+Han+Li%2E+2025%2E+ReDeEP%3A+Detecting+hallucination+in+retrieval%2Daugmented+generation+via+mechanistic+interp)

*Updated 1 November 2026*