---
title: "CLIMATE-FEVER — Diggelmann 2020"
description: "Automated fact-checking research has relied heavily on FEVER, whose claims were written by annotators and are short and simple."
authors: ["Thomas Diggelmann", "Jordan Boyd-Graber", "Jannis Bulian", "Massimiliano Ciaramita", "Markus Leippold"]
year: 2020
venue: "arXiv:2012.00614"
source: "https://arxiv.org/abs/2012.00614"
doi: "10.48550/arxiv.2012.00614"
arxiv: "2012.00614"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/diggelmann-2020-climate-fever-dataset-verification-real/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/diggelmann-2020-climate-fever-dataset-verification-real/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/2/) / CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims

# CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims

**Authors:** Thomas Diggelmann, [Jordan Boyd-Graber](https://md.research.elamur.ai/free/en/authors/jordan-boyd-graber/), Jannis Bulian, [Massimiliano Ciaramita](https://md.research.elamur.ai/free/en/authors/massimiliano-ciaramita/), Markus Leippold · **Year:** 2020 · **Venue:** arXiv:2012.00614 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2012.00614) · [PDF](https://arxiv.org/pdf/2012.00614) · [DOI 10.48550/arxiv.2012.00614](https://doi.org/10.48550/arxiv.2012.00614) · [arXiv 2012.00614](https://arxiv.org/abs/2012.00614) · [OpenAlex](https://openalex.org/W3107298362)

**Citations:** 22 citations (openalex-snapshot · 2026-10-14) · [135](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary

Automated fact-checking research has relied heavily on FEVER, whose claims were written by annotators and are short and simple. Real statements about climate change, which fuel public misinformation, look quite different. CLIMATE-FEVER adapts the FEVER methodology to such claims. The authors gathered over 3,000 candidate claims from the web, drawn equally from science-oriented and skeptic or denialist sources. Climate scientists then kept 1,535 claims they judged verifiable.

For each claim, a retrieval pipeline searched the full text of English Wikipedia and returned five candidate evidence sentences. Annotators decided whether every retrieved sentence backed the claim, contradicted it, or lacked sufficient information, and these judgments were then aggregated into a single label per claim. Unlike FEVER, the dataset also keeps "disputed" claims, which have evidence on both sides.

Annotator agreement was low, which shows the task is hard even for people. An ALBERT entailment model trained on FEVER performs well on FEVER itself but drops to near-chance accuracy on CLIMATE-FEVER. The dataset is therefore a demanding benchmark for natural language understanding and climate-disinformation tools.

## Key points

- The dataset has 1,535 real-world climate claims and 7,675 claim–evidence pairs. It is publicly released via climatefever.ai.
- Claim-level labels are distributed as follows: SUPPORTS 655 (42.67%), REFUTES 253 (16.5%), DISPUTED 153 (9.97%), NOT_ENOUGH_INFO 474 (30.88%).
- The retrieval pipeline has three steps:
  - BM25 (Lucene) with entity linking selects the top-10 Wikipedia articles.
  - Siamese ALBERT large-v2 sentence embeddings with a FAISS index pick the top-100 sentences.
  - An ALBERT base-v2 point-wise re-ranker keeps the top 5.
- Average inter-annotator agreement (Krippendorff's alpha) was 0.334. Trained or expert annotators tended to agree more.
- The FEVER-trained ALBERT large-v2 entailment model reaches 77.68% label accuracy on the FEVER dev set. On CLIMATE-FEVER it gets only 38.78% accuracy (weighted precision 56.49%, weighted F1 32.85%).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

135 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 126 not yet.

- [Task-aware Retrieval with Instructions](https://md.research.elamur.ai/free/en/papers/asai-2023-task-aware-retrieval-instructions/)
- [A Survey on Automated Fact-Checking](https://md.research.elamur.ai/free/en/papers/guo-2022-survey-automated-fact-checking/)
- [TrustLLM: Trustworthiness in Large Language Models](https://md.research.elamur.ai/free/en/papers/huang-2024-trustllm-trustworthiness-large-language-models/)
- [Datasets: A Community Library for Natural Language Processing](https://md.research.elamur.ai/free/en/papers/lhoest-2021-datasets-community-library-natural-language/)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://md.research.elamur.ai/free/en/papers/min-2022-rethinking-role-demonstrations-what-makes/)
- [COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemic](https://md.research.elamur.ai/free/en/papers/saakyan-2021-covid-fact-fact-extraction-verification/)
- [ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction](https://md.research.elamur.ai/free/en/papers/santhanam-2022-colbertv2-effective-efficient-retrieval-lightweight/)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://md.research.elamur.ai/free/en/papers/thakur-2021-beir-heterogenous-benchmark-zero/)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://md.research.elamur.ai/free/en/papers/wang-2022-text-embeddings-weakly-supervised-contrastive/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{diggelmann2020climate,
  title = {{CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims}},
  author = {Diggelmann, Thomas and Boyd-Graber, Jordan and Bulian, Jannis and Ciaramita, Massimiliano and Leippold, Markus},
  year = {2020},
  doi = {10.48550/arxiv.2012.00614},
  eprint = {2012.00614},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2012.00614}
}
```

- [← previous Clause Structure Variation in Biblical Hebrew: A Quantitative Approach](https://md.research.elamur.ai/free/en/papers/hbo-2020-naaijer-clause-structure-variation-biblical-hebrew/)
- [next → Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data](https://md.research.elamur.ai/free/en/papers/bender-2020-climbing-nlu-meaning-form-understanding/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/2/)

## References

21 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. J. Thorne, A. Vlachos, C. Christodoulopoulos, and A. Mittal, “FEVER: a large-scale dataset for fact extraction and verification,” arXiv preprint arXiv:1803.05355, 2018. [find](https://scholar.google.com/scholar?q=J%2E+Thorne%2C+A%2E+Vlachos%2C+C%2E+Christodoulopoulos%2C+and+A%2E+Mittal%2C+%E2%80%9CFEVER%3A+a+large%2Dscale+dataset+for+fact+extraction+and+verification%2C%E2%80%9D+arXiv+preprint+arXiv%3A1803%2E05355%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/thorne-2018-fever-large-scale-dataset-fact/)
2. W. R. Anderegg, J. W. Prall, J. Harold, and S. H. Schneider, “Expert credibility in climate change,” Proceedings of the National Academy of Sciences, vol. 107, no. 27, pp. 12 107–12 109, 2010. [find](https://scholar.google.com/scholar?q=W%2E+R%2E+Anderegg%2C+J%2E+W%2E+Prall%2C+J%2E+Harold%2C+and+S%2E+H%2E+Schneider%2C+%E2%80%9CExpert+credibility+in+climate+change%2C%E2%80%9D+Proceedings+of+the+National+Academy+of+Sciences%2C+vol%2E+107%2C+no%2E+27%2C+pp%2E+12+107%E2%80%9312+109%2C+2010%2E)
3. D. Ding, E. W. Maibach, X. Zhao, C. Roser-Renouf, and A. Leiserowitz, “Support for climate policy and societal action are linked to perceptions about scientific agreement,” Nature Climate Change, vol. 1, no. 9, pp. 462–466, 2011. [find](https://scholar.google.com/scholar?q=D%2E+Ding%2C+E%2E+W%2E+Maibach%2C+X%2E+Zhao%2C+C%2E+Roser%2DRenouf%2C+and+A%2E+Leiserowitz%2C+%E2%80%9CSupport+for+climate+policy+and+societal+action+are+linked+to+perceptions+about+scientific+agreement%2C%E2%80%9D+Nature+Climate+Change%2C+vol%2E)
4. S. D. Benegal and L. A. Scruggs, “Correcting misinformation about climate change: The impact of partisanship in an experimental setting,” Climatic change, vol. 148, no. 1-2, pp. 61–80, 2018. To stay compatible with the fever methodology for evaluation we simply excluded disputed claims from our data… [find](https://scholar.google.com/scholar?q=S%2E+D%2E+Benegal+and+L%2E+A%2E+Scruggs%2C+%E2%80%9CCorrecting+misinformation+about+climate+change%3A+The+impact+of+partisanship+in+an+experimental+setting%2C%E2%80%9D+Climatic+change%2C+vol%2E+148%2C+no%2E+1%2D2%2C+pp%2E+61%E2%80%9380%2C+2018%2E+To+stay+c)
5. S. Van der Linden, A. Leiserowitz, S. Rosenthal, and E. Maibach, “Inoculating the public against misinformation about climate change,” Global Challenges, vol. 1, no. 2, p. 1600008, 2017. [find](https://scholar.google.com/scholar?q=S%2E+Van+der+Linden%2C+A%2E+Leiserowitz%2C+S%2E+Rosenthal%2C+and+E%2E+Maibach%2C+%E2%80%9CInoculating+the+public+against+misinformation+about+climate+change%2C%E2%80%9D+Global+Challenges%2C+vol%2E+1%2C+no%2E+2%2C+p%2E+1600008%2C+2017%2E)
6. I. Augenstein, C. Lioma, D. Wang, L. C. Lima, C. Hansen, C. Hansen, and J. G. Simonsen, “Multifc: A real-world multi-domain dataset for evidence-based fact checking of claims,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint C… [find](https://scholar.google.com/scholar?q=I%2E+Augenstein%2C+C%2E+Lioma%2C+D%2E+Wang%2C+L%2E+C%2E+Lima%2C+C%2E+Hansen%2C+C%2E+Hansen%2C+and+J%2E+G%2E+Simonsen%2C+%E2%80%9CMultifc%3A+A+real%2Dworld+multi%2Ddomain+dataset+for+evidence%2Dbased+fact+checking+of+claims%2C%E2%80%9D+in+Proceedings+of+the+2) [our card](https://md.research.elamur.ai/free/en/papers/augenstein-2019-multifc-real-world-multi-domain/)
7. A. Soleimani, C. Monz, and M. Worring, “BERT for evidence retrieval and claim verification,” in European Conference on Information Retrieval. Springer, 2020, pp. 359–366. [find](https://scholar.google.com/scholar?q=A%2E+Soleimani%2C+C%2E+Monz%2C+and+M%2E+Worring%2C+%E2%80%9CBERT+for+evidence+retrieval+and+claim+verification%2C%E2%80%9D+in+European+Conference+on+Information+Retrieval%2E+Springer%2C+2020%2C+pp%2E+359%E2%80%93366%2E)
8. A. Hanselowski, H. Zhang, Z. Li, D. Sorokin, B. Schiller, C. Schulz, and I. Gurevych, “Ukp-athene: Multi-sentence textual entailment for claim verification,” arXiv preprint arXiv:1809.01479, 2018. [find](https://scholar.google.com/scholar?q=A%2E+Hanselowski%2C+H%2E+Zhang%2C+Z%2E+Li%2C+D%2E+Sorokin%2C+B%2E+Schiller%2C+C%2E+Schulz%2C+and+I%2E+Gurevych%2C+%E2%80%9CUkp%2Dathene%3A+Multi%2Dsentence+textual+entailment+for+claim+verification%2C%E2%80%9D+arXiv+preprint+arXiv%3A1809%2E01479%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/hanselowski-2018-ukp-athene-multi-sentence-textual/)
9. Y. Nie, H. Chen, and M. Bansal, “Combining fact extraction and verification with neural semantic matching networks,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, 2019, pp. 6859–6866. [find](https://scholar.google.com/scholar?q=Y%2E+Nie%2C+H%2E+Chen%2C+and+M%2E+Bansal%2C+%E2%80%9CCombining+fact+extraction+and+verification+with+neural+semantic+matching+networks%2C%E2%80%9D+in+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+vol%2E+33%2C+2019%2C+pp)
10. W. Zhong, J. Xu, D. Tang, Z. Xu, N. Duan, M. Zhou, J. Wang, and J. Yin, “Reasoning over semantic-level graph for fact checking,” arXiv preprint arXiv:1909.03745, 2019. [find](https://scholar.google.com/scholar?q=W%2E+Zhong%2C+J%2E+Xu%2C+D%2E+Tang%2C+Z%2E+Xu%2C+N%2E+Duan%2C+M%2E+Zhou%2C+J%2E+Wang%2C+and+J%2E+Yin%2C+%E2%80%9CReasoning+over+semantic%2Dlevel+graph+for+fact+checking%2C%E2%80%9D+arXiv+preprint+arXiv%3A1909%2E03745%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhong-2020-reasoning-over-semantic-level-graph/)
11. Z. Liu, C. Xiong, M. Sun, and Z. Liu, “Fine-grained fact verification with kernel graph attention network,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 7342–7351. [find](https://scholar.google.com/scholar?q=Z%2E+Liu%2C+C%2E+Xiong%2C+M%2E+Sun%2C+and+Z%2E+Liu%2C+%E2%80%9CFine%2Dgrained+fact+verification+with+kernel+graph+attention+network%2C%E2%80%9D+in+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+) [our card](https://md.research.elamur.ai/free/en/papers/liu-2020-fine-grained-fact-verification-kernel/)
12. J. Johnson, M. Douze, and H. Jégou, “Billion-scale similarity search with gpus,” arXiv preprint arXiv:1702.08734, 2017. [find](https://scholar.google.com/scholar?q=J%2E+Johnson%2C+M%2E+Douze%2C+and+H%2E+J%C3%A9gou%2C+%E2%80%9CBillion%2Dscale+similarity+search+with+gpus%2C%E2%80%9D+arXiv+preprint+arXiv%3A1702%2E08734%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/johnson-2017-billion-scale-similarity-search-gpus/)
13. V. Karpukhin, B. Oğuz, S. Min, L. Wu, S. Edunov, D. Chen, and W.-t. Yih, “Dense passage retrieval for open-domain question answering,” arXiv preprint arXiv:2004.04906, 2020. [find](https://scholar.google.com/scholar?q=V%2E+Karpukhin%2C+B%2E+Og%CC%86uz%2C+S%2E+Min%2C+L%2E+Wu%2C+S%2E+Edunov%2C+D%2E+Chen%2C+and+W%2E%2Dt%2E+Yih%2C+%E2%80%9CDense+passage+retrieval+for+open%2Ddomain+question+answering%2C%E2%80%9D+arXiv+preprint+arXiv%3A2004%2E04906%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/)
14. Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “ALBERT: A lite BERT for self-supervised learning of language representations,” 2019. [find](https://scholar.google.com/scholar?q=Z%2E+Lan%2C+M%2E+Chen%2C+S%2E+Goodman%2C+K%2E+Gimpel%2C+P%2E+Sharma%2C+and+R%2E+Soricut%2C+%E2%80%9CALBERT%3A+A+lite+BERT+for+self%2Dsupervised+learning+of+language+representations%2C%E2%80%9D+2019%2E)
15. T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, and J. Brew, “Huggingface’s transformers: State-of-the-art natural language processing,” ArXiv, vol. abs/1910.03771, 2019. [find](https://scholar.google.com/scholar?q=T%2E+Wolf%2C+L%2E+Debut%2C+V%2E+Sanh%2C+J%2E+Chaumond%2C+C%2E+Delangue%2C+A%2E+Moi%2C+P%2E+Cistac%2C+T%2E+Rault%2C+R%2E+Louf%2C+M%2E+Funtowicz%2C+and+J%2E+Brew%2C+%E2%80%9CHuggingface%E2%80%99s+transformers%3A+State%2Dof%2Dthe%2Dart+natural+language+processing%2C%E2%80%9D+ArXiv) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
16. S. Robertson and H. Zaragoza, The probabilistic relevance framework: BM25 and beyond. 2009. [find](https://scholar.google.com/scholar?q=S%2E+Robertson+and+H%2E+Zaragoza%2C+The+probabilistic+relevance+framework%3A+BM25+and+beyond%2E+2009%2E)
17. N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” arXiv preprint arXiv:1908.10084, 2019. [find](https://scholar.google.com/scholar?q=N%2E+Reimers+and+I%2E+Gurevych%2C+%E2%80%9CSentence%2Dbert%3A+Sentence+embeddings+using+siamese+bert%2Dnetworks%2C%E2%80%9D+arXiv+preprint+arXiv%3A1908%2E10084%2C+2019%2E)
18. A. Hermans, L. Beyer, and B. Leibe, “In defense of the triplet loss for person re-identification,” arXiv preprint arXiv:1703.07737, 2017. — [arXiv:1703.07737](https://arxiv.org/abs/1703.07737) [find](https://scholar.google.com/scholar?q=A%2E+Hermans%2C+L%2E+Beyer%2C+and+B%2E+Leibe%2C+%E2%80%9CIn+defense+of+the+triplet+loss+for+person+re%2Didentification%2C%E2%80%9D+arXiv+preprint+arXiv%3A1703%2E07737%2C+2017%2E+%E2%80%94+%5BarXiv%3A1703%2E07737%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1703%2E07737%29)
19. L. McInnes, J. Healy, and J. Melville, “Umap: Uniform manifold approximation and projection for dimension reduction,” arXiv preprint arXiv:1802.03426, 2018. [find](https://scholar.google.com/scholar?q=L%2E+McInnes%2C+J%2E+Healy%2C+and+J%2E+Melville%2C+%E2%80%9CUmap%3A+Uniform+manifold+approximation+and+projection+for+dimension+reduction%2C%E2%80%9D+arXiv+preprint+arXiv%3A1802%2E03426%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/mcinnes-2018-umap-uniform-manifold-approximation-projection/)
20. “Using umap for clustering — umap 0.4 documentation,” [https://umap-learn.readthedocs.io/en/latest/clustering.html](https://umap-learn.readthedocs.io/en/latest/clustering.html), (Accessed on 09/24/2020). [↗](https://umap-learn.readthedocs.io/en/latest/clustering.html)
21. S. Louhichi, M. Gzara, and H. B. Abdallah, “A density based algorithm for discovering clusters with varied density,” in 2014 World Congress on Computer Applications and Information Systems (WCCAIS). IEEE, 2014, pp. 1–6. Now Publishers Inc, [find](https://scholar.google.com/scholar?q=S%2E+Louhichi%2C+M%2E+Gzara%2C+and+H%2E+B%2E+Abdallah%2C+%E2%80%9CA+density+based+algorithm+for+discovering+clusters+with+varied+density%2C%E2%80%9D+in+2014+World+Congress+on+Computer+Applications+and+Information+Systems+%28WCCAIS%29%2E+I)