--- title: "X-Fact — paper by Gupta and Srikumar (2021)" description: "Most automated fact-checking research relies on English data, and resources for other languages are scarce. 1 reference, cited by 64 papers here." authors: ["Ashim Gupta", "Vivek Srikumar"] year: 2021 venue: "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)" source: "https://aclanthology.org/2021.acl-short.86/" doi: "10.18653/v1/2021.acl-short.86" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/gupta-2021-x-fact-new-benchmark-dataset/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gupta-2021-x-fact-new-benchmark-dataset/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / X-Fact: A New Benchmark Dataset for Multilingual Fact Checking # X-Fact: A New Benchmark Dataset for Multilingual Fact Checking **Authors:** Ashim Gupta and [Vivek Srikumar](https://md.research.elamur.ai/free/en/authors/vivek-srikumar/) · **Year:** 2021 · **Venue:** [Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2021](https://md.research.elamur.ai/free/en/venues/acl/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2021.acl-short.86/) · [PDF](https://aclanthology.org/2021.acl-short.86.pdf) · [DOI 10.18653/v1/2021.acl-short.86](https://doi.org/10.18653/v1/2021.acl-short.86) · [OpenAlex](https://openalex.org/W3172667408) **Citations:** 55 citations (OpenAlex, 2026-10-14) · [64](https://md.research.elamur.ai/free/en/papers/gupta-2021-x-fact-new-benchmark-dataset/cited-by/) in our library, of the 141,840 papers whose bibliographies we read ## Summary Most automated fact-checking research relies on English data, and resources for other languages are scarce. Existing multilingual collections are small, narrow in topic (for example, FakeCovid), or use only coarse binary labels. Gupta and Srikumar address this gap with X-Fact. It contains 31,189 real-world claims in 25 languages, gathered from non-English fact-checking websites listed by IFCN and Duke Reporter's Lab, partly via Google's Fact Check Explorer. Each site uses its own rating scale, so the authors translated these ratings and mapped them onto a common seven-label scheme. The benchmark has three test sets: one in-domain, one with unseen sources, and one zero-shot set with unseen languages. The baselines are multilingual mBERT classifiers. One uses only the claim, another adds metadata, and a third combines attention-weighted Google search snippets as evidence. The strongest model reaches only about 42% macro F1 in-domain and roughly 15–17% on the two generalization sets. Models therefore overfit to specific sources and languages, which makes X-Fact a demanding testbed for multilingual fact-checking. ## Key points - **Scale and labels:** X-Fact holds 31,189 claims in 25 languages from 11 language families. Its labels are True, Mostly-True, Partly-True, Mostly-False, False, Unverifiable and Other. Data and code are available at github.com/utahnlp/x-fact. - **Splits:** - Train: 19,079 claims - Dev: 2,535 claims - In-domain α1: 3,826 claims - Out-of-domain α2: 2,368 claims - Zero-shot α3: 3,381 claims in 13 languages absent from training - **Results (macro F1):** Attn-EA + Meta scores 41.9 on α1, against a majority baseline of 6.9. The claim-only model already reaches 38.2, which points to claim-only bias in the data. - **Snippet quality:** In 100 manually checked Hindi examples, search snippets were sufficient to judge the claim in only 45% of cases. Full web pages were sufficient in 83%. - **English augmentation:** Adding 12,311 English PolitiFact claims from MultiFC to training lowered performance. The authors suggest this is because those claims are mostly political, while X-Fact covers general topics. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of Utah](https://md.research.elamur.ai/free/en/universities/university-of-utah/) — University, Salt Lake City, United States · 29 works · 5,781 citations 📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 744 works · [🤝 Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) · 221 works 🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 685 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 467 works ## Cite Cite as: Ashim Gupta and Vivek Srikumar. 2021. X-Fact: A New Benchmark Dataset for Multilingual Fact Checking. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{gupta2021x, title = {{X-Fact: A New Benchmark Dataset for Multilingual Fact Checking}}, author = {Ashim Gupta and Vivek Srikumar}, booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)}, year = {2021}, doi = {10.18653/v1/2021.acl-short.86}, url = {https://doi.org/10.18653/v1/2021.acl-short.86} } ``` - [← previous Knowledge Enhanced Masked Language Model for Stance Detection](https://md.research.elamur.ai/free/en/papers/kawintiranon-2021-knowledge-enhanced-masked-language-model/) - [next → Findings of the AmericasNLP 2021 Shared Task on Open Machine Translation for Indigenous Languages of the Americas](https://md.research.elamur.ai/free/en/papers/mager-2021-findings-americasnlp-2021-shared-task/) - [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) ## References 1 references, 1 of them with a card of ours. ↗ leads to the work. 1. Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3874– 388… [our card](https://md.research.elamur.ai/free/en/papers/aharoni-2019-massively-multilingual-neural-machine-translation/) *Updated 24 October 2026*