--- title: "The Fact Extraction and VERification Over — Aly 2021" description: "The FEVEROUS shared task, held at the FEVER 2021 workshop, extends automated fact checking from purely textual evidence to a mix of text and structured…" authors: ["Rami Aly", "Zhijiang Guo", "Michael Sejr Schlichtkrull", "James Thorne", "Andreas Vlachos", "Christos Christodoulopoulos", "Oana Cocarascu", "Arpit Mittal"] year: 2021 venue: "Proceedings of the Fourth Workshop on Fact Extraction and VERification (FEVER)" source: "https://aclanthology.org/2021.fever-1.1/" doi: "10.18653/v1/2021.fever-1.1" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/aly-2021-fact-extraction-verification-over-unstructured/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/aly-2021-fact-extraction-verification-over-unstructured/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/5/) / The Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS) Shared Task # The Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS) Shared Task **Authors:** Rami Aly, [Zhijiang Guo](https://md.research.elamur.ai/free/en/authors/zhijiang-guo/), Michael Sejr Schlichtkrull, [James Thorne](https://md.research.elamur.ai/free/en/authors/james-thorne/), [Andreas Vlachos](https://md.research.elamur.ai/free/en/authors/andreas-vlachos/), [Christos Christodoulopoulos](https://md.research.elamur.ai/free/en/authors/christos-christodoulopoulos/), [Oana Cocarascu](https://md.research.elamur.ai/free/en/authors/oana-cocarascu/), [Arpit Mittal](https://md.research.elamur.ai/free/en/authors/arpit-mittal/) · **Year:** 2021 · **Venue:** Proceedings of the Fourth Workshop on Fact Extraction and VERification (FEVER) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2021.fever-1.1/) · [PDF](https://aclanthology.org/2021.fever-1.1.pdf) · [DOI 10.18653/v1/2021.fever-1.1](https://doi.org/10.18653/v1/2021.fever-1.1) · [OpenAlex](https://openalex.org/W3212606841) **Citations:** 100 citations (openalex-snapshot · 2026-10-14) · [78](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary The FEVEROUS shared task, held at the FEVER 2021 workshop, extends automated fact checking from purely textual evidence to a mix of text and structured Wikipedia content such as tables and lists. Given a human-written claim, a system has to find supporting material (sentences and table cells, with their page, section and header context) and then decide whether the claim is SUPPORTED, REFUTED or NOT ENOUGH INFO. Unlike FEVER 2018, NEI predictions also require partial evidence, and claims are considerably longer. Submissions were ranked by the FEVEROUS score, which counts a claim as correct only when the label is right and a full gold evidence set appears among the returned items. The organisers supplied a baseline built from TF-IDF and entity matching, a sequence-labelling cell extractor and a RoBERTa NLI verdict model. Of the systems submitted, six outperformed this baseline, and the winner, "Bust a move!", scored 27%. NEI claims and table-based reasoning remained the weakest areas, so the benchmark leaves plenty of room for improvement. The scoring platform stays open for follow-up work. ## Key points - FEVEROUS has 87,026 claims, split into 71,291 train, 7,890 dev and 7,845 blind test. Claims average 25.3 words, compared with 9.4 in FEVER, and evidence may be sentences only, cells only, or both. - Systems may return at most 5 sentences and 25 table cells (including captions and list items) per claim. Anything beyond these limits is dropped without penalty. - The task received 13 entries, hosted on EvalAI. "Bust a move!" won with a score of 0.27, about 9 points above the baseline's 0.18. Papelo came second with 0.26 and achieved the best result on tabular-only claims. - NEI made up only about 5% of the training data. The two top systems produced no NEI predictions at all, and their per-class F1 for NEI was 0.00. - Evidence recall matters far more here than in FEVER: the FEVEROUS score correlates with recall at ρ = 0.97, against 0.53 for FEVER. Claim-only bias is also much weaker, with a claim-only model reaching 58% against a 56% majority baseline (in FEVER, 62% vs 33%). *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 78 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 73 not yet. - [Large Language Models are few(1)-shot Table Reasoners](https://md.research.elamur.ai/free/en/papers/chen-2023-large-language-models-are-few/) - [Constructing a Japanese Claim Decomposition Dataset for Fact-Checking of LLM-Generated Texts](https://md.research.elamur.ai/free/en/papers/masano-2026-constructing-japanese-claim-decomposition-dataset/) - [UL2: Unifying Language Learning Paradigms](https://md.research.elamur.ai/free/en/papers/tay-2022-ul2-unifying-language-learning-paradigms/) - [Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity](https://md.research.elamur.ai/free/en/papers/wang-2023-survey-factuality-large-language-models/) - [UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models](https://md.research.elamur.ai/free/en/papers/xie-2022-unifiedskg-unifying-multi-tasking-structured/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{aly2021fact, title = {{The Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS) Shared Task}}, author = {Aly, Rami and Guo, Zhijiang and Schlichtkrull, Michael Sejr and Thorne, James and Vlachos, Andreas and Christodoulopoulos, Christos and Cocarascu, Oana and Mittal, Arpit}, booktitle = {Proceedings of the Fourth Workshop on Fact Extraction and VERification (FEVER)}, year = {2021}, doi = {10.18653/v1/2021.fever-1.1}, url = {https://doi.org/10.18653/v1/2021.fever-1.1} } ``` - [← previous The DISRPT 2021 Shared Task on Elementary Discourse Unit Segmentation, Connective Detection, and Relation Classification](https://md.research.elamur.ai/free/en/papers/zeldes-2021-disrpt-2021-shared-task-elementary/) - [next → The IDC System for Sentiment Classification and Sarcasm Detection in Arabic](https://md.research.elamur.ai/free/en/papers/israeli-2021-idc-system-sentiment-classification-sarcasm/) - [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/5/) ## References 1 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Figure 3: Correlation between FEVER/FEVEROUS score and Label Accuracy, F1, and Recall, respectively.* Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021. FEVEROUS: Fact extraction and VERification ove… [find](https://scholar.google.com/scholar?q=Figure+3%3A+Correlation+between+FEVER%2FFEVEROUS+score+and+Label+Accuracy%2C+F1%2C+and+Recall%2C+respectively%2E%2A+Rami+Aly%2C+Zhijiang+Guo%2C+Michael+Sejr+Schlichtkrull%2C+James+Thorne%2C+Andreas+Vlachos%2C+Christos+Christ)