--- title: "AVeriTeC — paper by Schlichtkrull et al. (2023)" description: "The authors build AVeriTeC, a set of 4,568 real claims drawn from the work of 50 fact-checking… 61 references, cited by 94 papers here." authors: ["Michael Schlichtkrull", "Zhijiang Guo", "Andreas Vlachos"] year: 2023 venue: "arXiv:2305.13117" source: "https://arxiv.org/abs/2305.13117" doi: "10.48550/arxiv.2305.13117" arxiv: "2305.13117" licence: "arXiv — non-exclusive license to distribute" licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/schlichtkrull-2023-averitec-dataset-real-world-claim/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/schlichtkrull-2023-averitec-dataset-real-world-claim/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/) / AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Web # AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Web **Authors:** [Michael Schlichtkrull](https://md.research.elamur.ai/free/en/authors/michael-schlichtkrull/), [Zhijiang Guo](https://md.research.elamur.ai/free/en/authors/zhijiang-guo/) and [Andreas Vlachos](https://md.research.elamur.ai/free/en/authors/andreas-vlachos/) · **Year:** 2023 · **Venue:** arXiv:2305.13117 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2305.13117) · [PDF](https://arxiv.org/pdf/2305.13117) · [DOI 10.48550/arxiv.2305.13117](https://doi.org/10.48550/arxiv.2305.13117) · [arXiv 2305.13117](https://arxiv.org/abs/2305.13117) · [OpenAlex](https://openalex.org/W4377865144) **Citations:** 7 citations (OpenAlex, 2026-10-14) · [94](https://md.research.elamur.ai/free/en/papers/schlichtkrull-2023-averitec-dataset-real-world-claim/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Automated fact-checking benchmarks often diverge from real practice. Many rely on synthetic claims derived from Wikipedia. Others use genuine claims but have shallow evidence annotation, omit context the claim needs, or cite evidence that appeared only after the claim was made. The authors build AVeriTeC, a set of 4,568 real claims drawn from the work of 50 fact-checking organizations. Each claim is first rewritten to stand on its own. Annotators then break it into question-answer pairs grounded in web pages published before the claim, and add a verdict and a free-text justification. A blind quality check flags claims with too little evidence and sends them back for another annotation round. The data splits follow chronological order to block temporal leakage. Annotators agreed on verdicts with a free-marginal κ of 0.619. The paper also offers a baseline and an evaluation scheme that scores both evidence and verdicts. This brings benchmarking closer to how journalists actually verify claims on the open web. ## Key points - Claims were gathered through the Google FactCheck Claim Search API, which builds on ClaimReview. Up to five annotators worked on each claim. - There are four verdict labels: supported, refuted, not enough evidence, and conflicting evidence/cherry-picking. The last one covers claims that are technically true but misleading. - Evidence pages are archived on the Internet Archive. The authors report that just 53% of the gold answers are taken entirely from spans in the source text. They also report that 85 answers rely on sources flagged as possibly misinformative. - The baseline combines Google Search, BM25, retrieved in-context prompts and a trained stance detection model. ChatGPT produced better questions but scored lower overall, since its answers were often unsupported or hallucinated. - All claims are in English. The dataset and baseline are released under CC-BY-NC-4.0 at github.com/MichSchli/AVeriTeC. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of Cambridge](https://md.research.elamur.ai/free/en/universities/university-of-cambridge/) — University, Cambridge, United Kingdom · 256 works · 123,373 citations 📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,954 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works · [🤝 Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) · 626 works ## Cite Cite as: Michael Schlichtkrull, Zhijiang Guo and Andreas Vlachos. 2023. AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Web. arXiv:2305.13117. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{schlichtkrull2023averitec, title = {{AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Web}}, author = {Michael Schlichtkrull, Zhijiang Guo and Andreas Vlachos}, year = {2023}, doi = {10.48550/arxiv.2305.13117}, eprint = {2305.13117}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2305.13117} } ``` - [← previous What In-Context Learning “Learns” In-Context: Disentangling Task Recognition and Task Learning](https://md.research.elamur.ai/free/en/papers/pan-2023-what-context-learning-learns-context/) - [next → Certified Robustness to Text Adversarial Attacks by Randomized MASK](https://md.research.elamur.ai/free/en/papers/zeng-2023-certified-robustness-text-adversarial-attacks/) - [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/) ## References 61 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Tariq Alhindi, Savvas Petridis, and Smaranda Muresan. [Where is your evidence: Improving fact-checking by justification modeling](https://md.research.elamur.ai/free/en/papers/alhindi-2018-where-is-your-evidence-improving/). In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pages 85–90, Brussels, Belgium, November 2018. Association for Computational Linguistics.… 2. Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. [The fact extraction and VERification over unstructured and structured information (FEVEROUS) shared task](https://md.research.elamur.ai/free/en/papers/aly-2021-fact-extraction-verification-over-unstructured/). In Proceedings of the Fourth Workshop on Fact Ex… 3. Michelle A. Amazeen. Revisiting the Epistemology of Fact-Checking. Critical Review, 27(1):1–22, January 2015. ISSN 0891-3811, 1933-8007. doi: 10.1080/08913811.2014.993890. URL [http://www.tandfonlin](http://www.tandfonlin) e.com/doi/abs/10.1080/08913811.2014.993890. — [doi:10.1080/08913811.2014.993890](https://doi.org/10.1080/08913811.2014.993890) [↗](http://www.tandfonlin) 4. Phoebe Arnold. The challenges of online fact checking. [https://fullfact.org/media/uploads/coof-2](https://fullfact.org/media/uploads/coof-2) 020.pdf, 2020. Accessed: 2023-01-17. [↗](https://fullfact.org/media/uploads/coof-2) 5. Isabelle Augenstein, Christina Lioma, Dongsheng Wang, Lucas Chaves Lima, Casper Hansen, Christian Hansen, and Jakob Grue Simonsen. [MultiFC: A real-world multi-domain dataset for evidence-based fact checking of claims](https://md.research.elamur.ai/free/en/papers/augenstein-2019-multifc-real-world-multi-domain/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Pro… 6. Satanjeev Banerjee and Alon Lavie. [METEOR: An automatic metric for MT evaluation with improved correlation with human judgments](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/). In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan, June 2005… 7. Aviv Barnoy and Zvi Reich. The When, Why, How and So-What of Verifications. Journalism Studies, 20(16): 2312–2330, December 2019. ISSN 1461-670X, 1469-9699. doi: 10.1080/1461670X.2019.1593881. URL [https://www.tandfonline.com/doi/full/10.1080/1461670X.2019.1593881](https://www.tandfonline.com/doi/full/10.1080/1461670X.2019.1593881). — [doi:10.1080/1461670x.2019.1593881](https://doi.org/10.1080/1461670x.2019.1593881) [↗](https://www.tandfonline.com/doi/full/10.1080/1461670X.2019.1593881) 8. Emily M. Bender and Batya Friedman. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6: 587–604, 2018. doi: 10.1162/tacl_a_00041. URL https://aclanthology.org/Q18-1041. [↗](https://aclanthology.org/Q18-1041) 9. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? . In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623, 2021. [find](https://scholar.google.com/scholar?q=Emily+M+Bender%2C+Timnit+Gebru%2C+Angelina+McMillan%2DMajor%2C+and+Shmargaret+Shmitchell%2E+On+the+dangers+of+stochastic+parrots%3A+Can+language+models+be+too+big%3F+%2E+In+Proceedings+of+the+2021+ACM+Conference+on+F) 10. Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python: analyzing text with the natural language toolkit. " O’Reilly Media, Inc.", 2009. [find](https://scholar.google.com/scholar?q=Steven+Bird%2C+Ewan+Klein%2C+and+Edward+Loper%2E+Natural+language+processing+with+Python%3A+analyzing+text+with+the+natural+language+toolkit%2E+%22+O%E2%80%99Reilly+Media%2C+Inc%2E%22%2C+2009%2E) 11. Brooke Borel. The Chicago Guide to Fact-checking. University of Chicago Press, 2016. [find](https://scholar.google.com/scholar?q=Brooke+Borel%2E+The+Chicago+Guide+to+Fact%2Dchecking%2E+University+of+Chicago+Press%2C+2016%2E) 12. Jifan Chen, Aniruddh Sriram, Eunsol Choi, and Greg Durrett. Generating literal and implied subquestions to fact-check complex claims. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3495–3516, Abu Dhabi, United Arab Emirates, December 2022. Associatio… [find](https://scholar.google.com/scholar?q=Jifan+Chen%2C+Aniruddh+Sriram%2C+Eunsol+Choi%2C+and+Greg+Durrett%2E+Generating+literal+and+implied+subquestions+to+fact%2Dcheck+complex+claims%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Natu) 13. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL [https://lmsys.org/blog/2023-03-3](https://lmsys.org/blog/2023-03-3) 0-vicuna… [↗](https://lmsys.org/blog/2023-03-3) 14. Sarah Cohen, Chengkai Li, Jun Yang, and Cong Yu. Computational journalism: A call to arms to database researchers. In 5th Biennial Conference on Innovative Data Systems Research (CIDR), 2011. [find](https://scholar.google.com/scholar?q=Sarah+Cohen%2C+Chengkai+Li%2C+Jun+Yang%2C+and+Cong+Yu%2E+Computational+journalism%3A+A+call+to+arms+to+database+researchers%2E+In+5th+Biennial+Conference+on+Innovative+Data+Systems+Research+%28CIDR%29%2C+2011%2E) 15. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum… 16. Thomas Diggelmann, Jordan Boyd-Graber, Jannis Bulian, Massimiliano Ciaramita, and Markus Leippold. [Climate-fever: A dataset for verification of real-world climate claims](https://md.research.elamur.ai/free/en/papers/diggelmann-2020-climate-fever-dataset-verification-real/), 2020. 17. Andy Dudfield. How we’re using AI to scale up global fact checking. [https://fullfact.org/blog/2020/jul/afc-global/](https://fullfact.org/blog/2020/jul/afc-global/), 2020. Accessed: 2023-01-17. [↗](https://fullfact.org/blog/2020/jul/afc-global/) 18. Angela Fan, Aleksandra Piktus, Fabio Petroni, Guillaume Wenzek, Marzieh Saeidi, Andreas Vlachos, Antoine Bordes, and Sebastian Riedel. Generating fact checking briefs. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7147–7161, Online, November… [find](https://scholar.google.com/scholar?q=Angela+Fan%2C+Aleksandra+Piktus%2C+Fabio+Petroni%2C+Guillaume+Wenzek%2C+Marzieh+Saeidi%2C+Andreas+Vlachos%2C+Antoine+Bordes%2C+and+Sebastian+Riedel%2E+Generating+fact+checking+briefs%2E+In+Proceedings+of+the+2020+Confe) 19. Joseph L Fleiss. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378, 1971. [find](https://scholar.google.com/scholar?q=Joseph+L+Fleiss%2E+Measuring+nominal+scale+agreement+among+many+raters%2E+Psychological+bulletin%2C+76%285%29%3A378%2C+1971%2E) 20. Marina Fomicheva and Lucia Specia. [Taking MT Evaluation Metrics to Extremes: Beyond Correlation with Human Judgments](https://md.research.elamur.ai/free/en/papers/fomicheva-2019-taking-mt-evaluation-metrics-extremes/). Computational Linguistics, 45(3):515–558, 09 2019. ISSN 0891-2017. doi: 10.1162/coli_a_00356. URL https://doi.org/10.1162/coli_a_00356. [↗](https://doi.org/10.1162/coli_a_00356) 21. Max Glockner, Yufang Hou, and Iryna Gurevych. Missing counter-evidence renders nlp fact-checking unrealistic for misinformation, 2022. URL https://arxiv.org/abs/2210.13865. [↗](https://arxiv.org/abs/2210.13865) 22. Ashim Gupta and Vivek Srikumar. [X-fact: A new benchmark dataset for multilingual fact checking](https://md.research.elamur.ai/free/en/papers/gupta-2021-x-fact-new-benchmark-dataset/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 675–68… 23. Andreas Hanselowski, Christian Stab, Claudia Schulz, Zile Li, and Iryna Gurevych. [A richly annotated corpus for different tasks in automated fact-checking](https://md.research.elamur.ai/free/en/papers/hanselowski-2019-richly-annotated-corpus-different-tasks/). In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 493–503, Hong Kong, China, November 2019. Associ… 24. Naeemul Hassan, Chengkai Li, and Mark Tremayne. Detecting check-worthy factual claims in presidential debates. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, CIKM ’15, page 1835–1838, New York, NY, USA, 2015. Association for Computing Machinery. I… — [doi:10.1145/2806416.2806652](https://doi.org/10.1145/2806416.2806652) [find](https://scholar.google.com/scholar?q=Naeemul+Hassan%2C+Chengkai+Li%2C+and+Mark+Tremayne%2E+Detecting+check%2Dworthy+factual+claims+in+presidential+debates%2E+In+Proceedings+of+the+24th+ACM+International+on+Conference+on+Information+and+Knowledge+M) 25. Xuming Hu, Zhijiang Guo, GuanYu Wu, Aiwei Liu, Lijie Wen, and Philip Yu. CHEF: A pilot Chinese dataset for evidence-based fact-checking. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3362–3376,… [find](https://scholar.google.com/scholar?q=Xuming+Hu%2C+Zhijiang+Guo%2C+GuanYu+Wu%2C+Aiwei+Liu%2C+Lijie+Wen%2C+and+Philip+Yu%2E+CHEF%3A+A+pilot+Chinese+dataset+for+evidence%2Dbased+fact%2Dchecking%2E+In+Proceedings+of+the+2022+Conference+of+the+North+American+Cha) 26. Nan-Jiang Jiang and Marie-Catherine de Marneffe. [Investigating Reasons for Disagreement in Natural Language Inference](https://md.research.elamur.ai/free/en/papers/jiang-2022-investigating-reasons-disagreement-natural-language/). Transactions of the Association for Computational Linguistics, 10:1357–1374, 12 2022. ISSN 2307-387X. doi: 10.1162/tacl_a_00523. URL https://doi.org/10.1162/tacl_a_00523. [↗](https://doi.org/10.1162/tacl_a_00523) 27. Georgi Karadzhov, Preslav Nakov, Lluís Màrquez, Alberto Barrón-Cedeño, and Ivan Koychev. Fully automated fact checking using external sources. In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, pages 344–353, Varna, Bulgaria, September 2017. IN… — [doi:10.26615/978-954-452-049-6_046](https://doi.org/10.26615/978-954-452-049-6_046) [find](https://scholar.google.com/scholar?q=Georgi+Karadzhov%2C+Preslav+Nakov%2C+Llu%C3%ADs+M%C3%A0rquez%2C+Alberto+Barr%C3%B3n%2DCede%C3%B1o%2C+and+Ivan+Koychev%2E+Fully+automated+fact+checking+using+external+sources%2E+In+Proceedings+of+the+International+Conference+Recent+Adv) 28. Ashkan Kazemi, Kiran Garimella, Devin Gaffney, and Scott Hale. Claim matching beyond English to scale global fact-checking. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1:… [find](https://scholar.google.com/scholar?q=Ashkan+Kazemi%2C+Kiran+Garimella%2C+Devin+Gaffney%2C+and+Scott+Hale%2E+Claim+matching+beyond+English+to+scale+global+fact%2Dchecking%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computation) 29. Kashif Khan, Ruizhe Wang, and Pascal Poupart. Watclaimcheck: A new dataset for claim entailment and inference. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 20… [find](https://scholar.google.com/scholar?q=Kashif+Khan%2C+Ruizhe+Wang%2C+and+Pascal+Poupart%2E+Watclaimcheck%3A+A+new+dataset+for+claim+entailment+and+inference%2E+In+Smaranda+Muresan%2C+Preslav+Nakov%2C+and+Aline+Villavicencio%2C+editors%2C+Proceedings+of+the+) 30. Diederik P. Kingma and Jimmy Ba. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/1412.6980. [↗](http://arxiv.org/abs/1412.6980) 31. Neema Kotonya and Francesca Toni. Explainable automated fact-checking: A survey. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5430–5443, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics. doi: 10.18653/v1/2020.co… [find](https://scholar.google.com/scholar?q=Neema+Kotonya+and+Francesca+Toni%2E+Explainable+automated+fact%2Dchecking%3A+A+survey%2E+In+Proceedings+of+the+28th+International+Conference+on+Computational+Linguistics%2C+pages+5430%E2%80%935443%2C+Barcelona%2C+Spain+%28On) 32. H. W. Kuhn. The hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(1-2): 83–97, 1955. doi: https://doi.org/10.1002/nav.3800020109. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/nav.3800020109. — [doi:10.1002/nav.3800020109](https://doi.org/10.1002/nav.3800020109) [↗](https://doi.org/10.1002/nav.3800020109) 33. Stephan Lewandowsky, John Cook, Ullrich Ecker, Dolores Albarracin, Michelle Amazeen, Panayiota Kendeou, Doug Lombardi, Eryn Newman, Gordon Pennycook, Ethan Porter, David G. Rand, David N. Rapp, Jason Reifler, Jon Roozenbeek, Philipp Schmid, Colleen M. Seifert, Gale M. Sinatra, Briony Swire-Thompson,… [find](https://scholar.google.com/scholar?q=Stephan+Lewandowsky%2C+John+Cook%2C+Ullrich+Ecker%2C+Dolores+Albarracin%2C+Michelle+Amazeen%2C+Panayiota+Kendeou%2C+Doug+Lombardi%2C+Eryn+Newman%2C+Gordon+Pennycook%2C+Ethan+Porter%2C+David+G%2E+Rand%2C+David+N%2E+Rapp%2C+Jason+) 34. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. [BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). In Proceedings of the 58th Annual Meeting of the Assoc… 35. Chin-Yew Lin. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out, pages 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https://aclanthology.org/W04-1013. [↗](https://aclanthology.org/W04-1013) 36. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. [What makes good in-context examples for GPT-3?](https://md.research.elamur.ai/free/en/papers/liu-2022-what-makes-good-context-examples/) In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 100–114, Dubl… 37. Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. Generation-augmented retrieval for open-domain question answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on… [find](https://scholar.google.com/scholar?q=Yuning+Mao%2C+Pengcheng+He%2C+Xiaodong+Liu%2C+Yelong+Shen%2C+Jianfeng+Gao%2C+Jiawei+Han%2C+and+Weizhu+Chen%2E+Generation%2Daugmented+retrieval+for+open%2Ddomain+question+answering%2E+In+Proceedings+of+the+59th+Annual+Mee) 38. Sebastião Miranda, Andreas Vlachos, David Nogueira, Andrew Secker, Afonso Mendes, Rebecca Garrett, Jeffrey J Mitchell, and Zita Marinho. Automated fact checking in the news room. In The Web Conference 2019, pages 3579–3583, United States, May 2019. Association for Computing Machinery (ACM). doi: 10.… — [doi:10.1145/3308558.3314135](https://doi.org/10.1145/3308558.3314135) [find](https://scholar.google.com/scholar?q=Sebasti%C3%A3o+Miranda%2C+Andreas+Vlachos%2C+David+Nogueira%2C+Andrew+Secker%2C+Afonso+Mendes%2C+Rebecca+Garrett%2C+Jeffrey+J+Mitchell%2C+and+Zita+Marinho%2E+Automated+fact+checking+in+the+news+room%2E+In+The+Web+Conference) 39. Preslav Nakov, David Corney, Maram Hasanain, Firoj Alam, Tamer Elsayed, Alberto Barrón-Cedeño, Paolo Papotti, Shaden Shaar, and Giovanni Da San Martino. Automated fact-checking for assisting human fact-checkers. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on… — [doi:10.24963/ijcai.2021/619](https://doi.org/10.24963/ijcai.2021/619) [find](https://scholar.google.com/scholar?q=Preslav+Nakov%2C+David+Corney%2C+Maram+Hasanain%2C+Firoj+Alam%2C+Tamer+Elsayed%2C+Alberto+Barr%C3%B3n%2DCede%C3%B1o%2C+Paolo+Papotti%2C+Shaden+Shaar%2C+and+Giovanni+Da+San+Martino%2E+Automated+fact%2Dchecking+for+assisting+human+fac) 40. Safiya Umoja Noble. Algorithms of oppression. In Algorithms of Oppression. New York University Press, 2018. [find](https://scholar.google.com/scholar?q=Safiya+Umoja+Noble%2E+Algorithms+of+oppression%2E+In+Algorithms+of+Oppression%2E+New+York+University+Press%2C+2018%2E) 41. Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. [Document expansion by query prediction](https://md.research.elamur.ai/free/en/papers/nogueira-2019-document-expansion-query-prediction/). arXiv preprint arXiv:1904.08375, 2019. 42. Wojciech Ostrowski, Arnav Arora, Pepa Atanasova, and Isabelle Augenstein. Multi-hop fact checking of political claims. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pag… — [doi:10.24963/ijcai.2021/536](https://doi.org/10.24963/ijcai.2021/536) [find](https://scholar.google.com/scholar?q=Wojciech+Ostrowski%2C+Arnav+Arora%2C+Pepa+Atanasova%2C+and+Isabelle+Augenstein%2E+Multi%2Dhop+fact+checking+of+political+claims%2E+In+Zhi%2DHua+Zhou%2C+editor%2C+Proceedings+of+the+Thirtieth+International+Joint+Confere) 43. Nedjma Ousidhoum, Zhangdie Yuan, and Andreas Vlachos. Varifocal question generation for fact-checking. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2532–2544, Abu Dhabi, United Arab Emirates… [find](https://scholar.google.com/scholar?q=Nedjma+Ousidhoum%2C+Zhangdie+Yuan%2C+and+Andreas+Vlachos%2E+Varifocal+question+generation+for+fact%2Dchecking%2E+In+Yoav+Goldberg%2C+Zornitsa+Kozareva%2C+and+Yue+Zhang%2C+editors%2C+Proceedings+of+the+2022+Conference+o) 44. Justus J Randolph. Free-marginal multirater kappa (multirater k [free]): An alternative to fleiss’ fixed-marginal multirater kappa. Online submission, 2005. [find](https://scholar.google.com/scholar?q=Justus+J+Randolph%2E+Free%2Dmarginal+multirater+kappa+%28multirater+k+%5Bfree%5D%29%3A+An+alternative+to+fleiss%E2%80%99+fixed%2Dmarginal+multirater+kappa%2E+Online+submission%2C+2005%2E) 45. Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389, 2009. ISSN 1554-0669. doi: 10.1561/1500000019. URL http://dx.doi.org/10.1561/1500000019. [↗](http://dx.doi.org/10.1561/1500000019) 46. Ohad Rubin, Jonathan Herzig, and Jonathan Berant. [Learning to retrieve prompts for in-context learning](https://md.research.elamur.ai/free/en/papers/rubin-2021-learning-retrieve-prompts-context-learning/). In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2655–2671, Seattle, United States, July 202… 47. Arkadiy Saakyan, Tuhin Chakrabarty, and Smaranda Muresan. [COVID-fact: Fact extraction and verification of real-world claims on COVID-19 pandemic](https://md.research.elamur.ai/free/en/papers/saakyan-2021-covid-fact-fact-extraction-verification/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language… 48. Mourad Sarrouti, Asma Ben Abacha, Yassine Mrabet, and Dina Demner-Fushman. [Evidence-based fact-checking of health-related claims](https://md.research.elamur.ai/free/en/papers/sarrouti-2021-evidence-based-fact-checking-health/). In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Even… 49. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%CC%81%2C+Daniel+Hesslow%2C+Roman+Castagn%C3%A9%2C+Alexandra+Sasha+Luccioni%2C+Fran%C3%A7ois+Yvon%2C+Matthias+Gall%C3%A9%2C+et+al%2E+Bloom%3A+A+176b%2Dparameter+open%2D) 50. Michael Schlichtkrull, Nedjma Ousidhoum, and Andreas Vlachos. The intended uses of automated fact-checking artefacts: Why, how and who, 2023. [find](https://scholar.google.com/scholar?q=Michael+Schlichtkrull%2C+Nedjma+Ousidhoum%2C+and+Andreas+Vlachos%2E+The+intended+uses+of+automated+fact%2Dchecking+artefacts%3A+Why%2C+how+and+who%2C+2023%2E) 51. Tal Schuster, Adam Fisch, and Regina Barzilay. [Get your vitamin C! robust fact verification with contrastive evidence](https://md.research.elamur.ai/free/en/papers/schuster-2021-get-your-vitamin-c-robust/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 624–643, Online, June 2021.… 52. Thibault Sellam, Dipanjan Das, and Ankur Parikh. [BLEURT: Learning robust metrics for text generation](https://md.research.elamur.ai/free/en/papers/sellam-2020-bleurt-learning-robust-metrics-text/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-m… 53. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. [FEVER: a large-scale dataset for fact extraction and VERification](https://md.research.elamur.ai/free/en/papers/thorne-2018-fever-large-scale-dataset-fact/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1… 54. Stephen E. Toulmin. The Uses of Argument. Cambridge University Press, 1958. [find](https://scholar.google.com/scholar?q=Stephen+E%2E+Toulmin%2E+The+Uses+of+Argument%2E+Cambridge+University+Press%2C+1958%2E) 55. Joseph E. Uscinski and Ryden W. Butler. The Epistemology of Fact Checking. Critical Review, 25(2):162– 180, June 2013. ISSN 0891-3811, 1933-8007. doi: 10.1080/08913811.2013.843872. URL [http://www.tandfonline.com/doi/abs/10.1080/08913811.2013.843872](http://www.tandfonline.com/doi/abs/10.1080/08913811.2013.843872). — [doi:10.1080/08913811.2013.843872](https://doi.org/10.1080/08913811.2013.843872) [↗](http://www.tandfonline.com/doi/abs/10.1080/08913811.2013.843872) 56. Andreas Vlachos and Sebastian Riedel. [Fact checking: Task definition and dataset construction](https://md.research.elamur.ai/free/en/papers/vlachos-2014-fact-checking-task-definition-dataset/). In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science, pages 18–22, Baltimore, MD, USA, June 2014. Association for Computational Linguistics. doi: 10.3115/v1/W14… 57. David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. [Fact or fiction: Verifying scientific claims](https://md.research.elamur.ai/free/en/papers/wadden-2020-fact-or-fiction-verifying-scientific/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7534–7550, Online, November 2020. Ass… 58. William Yang Wang. “liar, liar pants on fire”: A new benchmark dataset for fake news detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 422–426, Vancouver, Canada, July 2017. [Association for Computational Linguistics.…](https://md.research.elamur.ai/free/en/papers/wang-2017-liar-liar-pants-fire-new/) 59. Matthijs J Warrens. Inequalities between multi-rater kappas. Advances in data analysis and classification, 4(4): 271–286, 2010. [find](https://scholar.google.com/scholar?q=Matthijs+J+Warrens%2E+Inequalities+between+multi%2Drater+kappas%2E+Advances+in+data+analysis+and+classification%2C+4%284%29%3A+271%E2%80%93286%2C+2010%2E) 60. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe… — [Transformers: State-of-the-Art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/) 61. Jing Yang, Didier Vega-Oliveros, Taís Seibt, and Anderson Rocha. Explainable fact-checking through question answering. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8952–8956, 2022. doi: 10.1109/ICASSP43922.2022.9747214. — [doi:10.1109/icassp43922.2022.9747214](https://doi.org/10.1109/icassp43922.2022.9747214) [find](https://scholar.google.com/scholar?q=Jing+Yang%2C+Didier+Vega%2DOliveros%2C+Ta%C3%ADs+Seibt%2C+and+Anderson+Rocha%2E+Explainable+fact%2Dchecking+through+question+answering%2E+In+ICASSP+2022+%2D+2022+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Sign) *Updated 2 November 2026*