---
title: "A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking (2019)"
description: "They introduce a new mixed-domain corpus derived from the Snopes fact-checking website. 25 references, cited by 63 papers here."
authors: ["Andreas Hanselowski", "Christian Stab", "Claudia Schulz", "Zile Li", "Iryna Gurevych"]
year: 2019
venue: "Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)"
source: "https://aclanthology.org/K19-1046/"
doi: "10.18653/v1/k19-1046"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/hanselowski-2019-richly-annotated-corpus-different-tasks/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hanselowski-2019-richly-annotated-corpus-different-tasks/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking

# A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking

**Authors:** [Andreas Hanselowski](https://md.research.elamur.ai/free/en/authors/andreas-hanselowski/), [Christian Stab](https://md.research.elamur.ai/free/en/authors/christian-stab/), [Claudia Schulz](https://md.research.elamur.ai/free/en/authors/claudia-schulz/), [Zile Li](https://md.research.elamur.ai/free/en/authors/zile-li/) and [Iryna Gurevych](https://md.research.elamur.ai/free/en/authors/iryna-gurevych/) · **Year:** 2019 · **Venue:** [Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)](https://md.research.elamur.ai/free/en/venues/conll/) ([CoNLL 2019](https://md.research.elamur.ai/free/en/venues/conll/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/K19-1046/) · [PDF](https://aclanthology.org/K19-1046.pdf) · [DOI 10.18653/v1/k19-1046](https://doi.org/10.18653/v1/k19-1046) · [OpenAlex](https://openalex.org/W2988245244)

**Citations:** 84 citations (OpenAlex, 2026-10-14) · [63](https://md.research.elamur.ai/free/en/papers/hanselowski-2019-richly-annotated-corpus-different-tasks/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Training machine learning systems for automated fact-checking requires large, reliably annotated data covering the whole verification pipeline. The authors argue that earlier resources fall short: some contain only a few hundred claims, some lack evidence annotations, and FEVER, though large, relies on synthetic claims built from Wikipedia. They introduce a new mixed-domain corpus derived from the Snopes fact-checking website. It combines material gathered by Snopes fact-checkers with crowd-sourced annotations of evidence at two granularity levels, evidence sources, and evidence stance. The corpus can be used for each of four pipeline stages: retrieving documents, extracting evidence, detecting stance, and validating claims. The authors document their collection and annotation procedure and report substantial inter-annotator agreement. They then benchmark strong systems from the FEVER shared task and the Fake News Challenge, along with BERT-based models. All systems score noticeably lower than on comparable benchmarks. The error analysis points to heterogeneous web data, evidence from unreliable sources, and a skew toward false claims. The resource therefore offers a more realistic and demanding testbed for fact-checking research.

## Key points
- The corpus contains 6,422 validated natural claims and roughly 14.5k associated documents, with about 14.3k evidence annotations. The material spans blogs, news, and social media.
- Evidence is annotated at two levels: evidence text snippets (ETSs) and fine-grained evidence (FGE). Annotations also include stance toward the claim and the source of each piece of evidence.
- For claim validation, Snopes verdicts were mapped onto the three FEVER classes (refuted, supported, NEI). Ambiguous verdicts such as legend or miscaptioned were dropped.
- BertEmb performed best on claim validation with a macro F1 of 0.485. Systems reach about 0.7 on FEVER. Transfer learning between FEVER and this corpus, in either direction, gave no significant improvement.
- A crawler that rebuilds the dataset (UKPLab/conll2019-snopes-crawling) and the experiment code are released on GitHub. The original corpus is available on request for non-commercial research via TU Darmstadt.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Technische Universität Darmstadt](https://md.research.elamur.ai/free/en/universities/technische-universitat-darmstadt/) — University, Darmstadt, Germany · 76 works · 35,008 citations

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 744 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 523 works · [🤝 Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) · 221 works

## Cite

Cite as: Andreas Hanselowski, Christian Stab, Claudia Schulz, Zile Li and Iryna Gurevych. 2019. A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking. Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{hanselowski2019richly,
  title = {{A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking}},
  author = {Andreas Hanselowski, Christian Stab, Claudia Schulz, Zile Li and Iryna Gurevych},
  booktitle = {Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)},
  year = {2019},
  doi = {10.18653/v1/k19-1046},
  url = {https://doi.org/10.18653/v1/k19-1046}
}
```

- [← previous The Effect of Translationese in Machine Translation Test Sets](https://md.research.elamur.ai/free/en/papers/zhang-2019-effect-translationese-machine-translation-test/)
- [next → Text Processing Like Humans Do: Visually Attacking and Shielding NLP Systems](https://md.research.elamur.ai/free/en/papers/eger-2019-text-processing-like-humans-do/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/)

## References

25 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555–596. [find](https://scholar.google.com/scholar?q=Ron+Artstein+and+Massimo+Poesio%2E+2008%2E+Inter%2Dcoder+agreement+for+computational+linguistics%2E+Computational+Linguistics%2C+34%284%29%3A555%E2%80%93596%2E)
2. Darina Benikova, Margot Mieskes, Christian M. Meyer, and Iryna Gurevych. 2016. Bridging the gap between extractive and abstractive summaries: Creation and evaluation of coherent extracts from heterogeneous sources. In Proceedings of the 26th International Conference on Computational Linguistics (COL… [find](https://scholar.google.com/scholar?q=Darina+Benikova%2C+Margot+Mieskes%2C+Christian+M%2E+Meyer%2C+and+Iryna+Gurevych%2E+2016%2E+Bridging+the+gap+between+extractive+and+abstractive+summaries%3A+Creation+and+evaluation+of+coherent+extracts+from+heteroge)
3. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Samuel+R+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D+Manning%2E+2015%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+Proceedings+of+the+2015+Conference+on+Empirical+Met)
4. Andreas Hanselowski, Avinesh PVS, Benjamin Schiller, Felix Caspelherr, Debanjan Chaudhuri, Christian M Meyer, and Iryna Gurevych. 2018a. A retrospective analysis of the fake news challenge stance detection task. Proceedings of the 2018 International Committee on Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/hanselowski-2018-retrospective-analysis-fake-news-challenge/)
5. Andreas Hanselowski, Hao Zhang, Zile Li, Daniil Sorokin, Benjamin Schiller, Claudia Schulz, and Iryna Gurevych. 2018b. Ukp-athene: Multi-sentence textual entailment for claim verification. Proceedings of the EMNLP 2018 First Workshop on Fact Extraction and Verification. [our card](https://md.research.elamur.ai/free/en/papers/hanselowski-2018-ukp-athene-multi-sentence-textual/)
6. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A1735%E2%80%931780%2E)
7. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (System Demonstrations). [our card](https://md.research.elamur.ai/free/en/papers/cer-2018-universal-sentence-encoder/)
8. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning Whom to Trust with MACE. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL/HLT), pages 1120–1130, Atlanta, GA, US… [our card](https://md.research.elamur.ai/free/en/papers/hovy-2013-learning-whom-trust-mace/)
9. Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin, 70(4):213. [find](https://scholar.google.com/scholar?q=Jacob+Cohen%2E+1968%2E+Weighted+kappa%3A+Nominal+scale+agreement+provision+for+scaled+disagreement+or+partial+credit%2E+Psychological+bulletin%2C+70%284%29%3A213%2E)
10. Lee Howell et al. 2013. Digital wildfires in a hyper-connected world. WEF Report, 3:15–94. [find](https://scholar.google.com/scholar?q=Lee+Howell+et+al%2E+2013%2E+Digital+wildfires+in+a+hyper%2Dconnected+world%2E+WEF+Report%2C+3%3A15%E2%80%9394%2E)
11. Leon Derczynski, Kalina Bontcheva, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Arkaitz Zubiaga. 2017. Semeval-2017 task 8: Rumoureval: Determining rumour veracity and support for rumours. Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017). [our card](https://md.research.elamur.ai/free/en/papers/derczynski-2017-semeval-2017-task-8-rumoureval/)
12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NA… [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
13. William Ferreira and Andreas Vlachos. 2016. Emergent: a novel data-set for stance classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL/HLT), pages 1163–1168, San Diego, CA, USA. [our card](https://md.research.elamur.ai/free/en/papers/ferreira-2016-emergent-novel-data-set-stance/)
14. Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, and Benno Stein. 2017. The argument reasoning comprehension task. Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018). [find](https://scholar.google.com/scholar?q=Ivan+Habernal%2C+Henning+Wachsmuth%2C+Iryna+Gurevych%2C+and+Benno+Stein%2E+2017%2E+The+argument+reasoning+comprehension+task%2E+Proceedings+of+the+12th+International+Workshop+on+Semantic+Evaluation+%28SemEval%2D2018%29)
15. Andreas Hanselowski and Iryna Gurevych. 2017. A framework for automated fact-checking for real-time validation of emerging claims on the web. Proceedings of the NIPS Workshop on Prioritising Online Content (WPOC2017). [find](https://scholar.google.com/scholar?q=Andreas+Hanselowski+and+Iryna+Gurevych%2E+2017%2E+A+framework+for+automated+fact%2Dchecking+for+real%2Dtime+validation+of+emerging+claims+on+the+web%2E+Proceedings+of+the+NIPS+Workshop+on+Prioritising+Online+Co)
16. Preslav Nakov, Alberto Barrón-Cedeño, Tamer Elsayed, Reem Suwaileh, Lluı́s Màrquez, Wajdi Zaghouani, Pepa Atanasova, Spas Kyuchukov, and Giovanni Da San Martino. 2018. Overview of the clef-2018 checkthat! lab on automatic identification and verification of political claims. In Proceedings of the… [find](https://scholar.google.com/scholar?q=Preslav+Nakov%2C+Alberto+Barro%CC%81n%2DCeden%CC%83o%2C+Tamer+Elsayed%2C+Reem+Suwaileh%2C+Llu%C4%B1%CC%81s+Ma%CC%80rquez%2C+Wajdi+Zaghouani%2C+Pepa+Atanasova%2C+Spas+Kyuchukov%2C+and+Giovanni+Da+San+Martino%2E+2018%2E+Overview+of+the+clef%2D2018+che)
17. Ankur P Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. [our card](https://md.research.elamur.ai/free/en/papers/parikh-2016-decomposable-attention-model-natural-language/)
18. Dean Pomerleau and Delip Rao. 2017. The Fake News Challenge: Exploring how artificial intelligence technologies could be leveraged to combat fake news. http: Ac-//www.fakenewschallenge.org/. cessed: 2019-4-20. [find](https://scholar.google.com/scholar?q=Dean+Pomerleau+and+Delip+Rao%2E+2017%2E+The+Fake+News+Challenge%3A+Exploring+how+artificial+intelligence+technologies+could+be+leveraged+to+combat+fake+news%2E+http%3A+Ac%2D%2F%2Fwww%2Efakenewschallenge%2Eorg%2F%2E+cessed%3A+2)
19. Kashyap Popat, Subhabrata Mukherjee, Jannik Strötgen, and Gerhard Weikum. 2017. Where the truth lies: Explaining the credibility of emerging claims on the web and social media. In Proceedings of the 26th International Conference on World Wide Web Companion, pages 1003–1012. International World Wide… [find](https://scholar.google.com/scholar?q=Kashyap+Popat%2C+Subhabrata+Mukherjee%2C+Jannik+Stro%CC%88tgen%2C+and+Gerhard+Weikum%2E+2017%2E+Where+the+truth+lies%3A+Explaining+the+credibility+of+emerging+claims+on+the+web+and+social+media%2E+In+Proceedings+of+the+)
20. Christopher Tauchmann, Thomas Arnold, Andreas Hanselowski, Christian M Meyer, and Margot Mieskes. 2018. Beyond generic summarization: A . multi-faceted hierarchical summarization corpus of large heterogeneous data. In Proceedings of the Eleventh International Conference on Language Resources and Eva… [find](https://scholar.google.com/scholar?q=Christopher+Tauchmann%2C+Thomas+Arnold%2C+Andreas+Hanselowski%2C+Christian+M+Meyer%2C+and+Margot+Mieskes%2E+2018%2E+Beyond+generic+summarization%3A+A+%2E+multi%2Dfaceted+hierarchical+summarization+corpus+of+large+heter)
21. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018a. FEVER: a large-scale dataset for fact extraction and verification. In NAACL-HLT. [our card](https://md.research.elamur.ai/free/en/papers/thorne-2018-fever-large-scale-dataset-fact/)
22. James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2018b. The fact extraction and verification (fever) shared task. arXiv preprint arXiv:1811.10971. [our card](https://md.research.elamur.ai/free/en/papers/thorne-2018-fact-extraction-verification-fever-shared/)
23. Andreas Vlachos and Sebastian Riedel. 2014. Fact checking: Task definition and dataset construction. In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science, pages 18–22. [our card](https://md.research.elamur.ai/free/en/papers/vlachos-2014-fact-checking-task-definition-dataset/)
24. William Yang Wang. 2017. ” liar, liar pants on fire”: A new benchmark dataset for fake news detection. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). [our card](https://md.research.elamur.ai/free/en/papers/wang-2017-liar-liar-pants-fire-new/)
25. Klaus Zechner. 2002. Automatic Summarization of Open-Domain Multiparty Dialogues in Diverse Genres. Computational Linguistics, 28(4):447–485. [find](https://scholar.google.com/scholar?q=Klaus+Zechner%2E+2002%2E+Automatic+Summarization+of+Open%2DDomain+Multiparty+Dialogues+in+Diverse+Genres%2E+Computational+Linguistics%2C+28%284%29%3A447%E2%80%93485%2E)

*Updated 24 October 2026*