--- title: "Benchmarking Check-Worthiness Models on LLM Generated" description: "The authors test this on English data from CheckIt (tweets with fine-grained annotations) and CheckThatLab! 2 references." authors: ["Charlie George Roadhouse", "Matthew Shardlow", "Ashley Williams"] year: 2026 venue: "Proceedings of the 1st Workshop on Information Disorder (InDor) @ LREC 2026" source: "https://aclanthology.org/2026.indor-1.4/" doi: "10.63317/5c8w857kxgsn" licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)" licence_url: "https://creativecommons.org/licenses/by-nc/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/roadhouse-2026-benchmarking-check-worthiness-models-llm/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/roadhouse-2026-benchmarking-check-worthiness-models-llm/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/24/) / Benchmarking Check-Worthiness Models on LLM Generated Claims # Benchmarking Check-Worthiness Models on LLM Generated Claims **Authors:** Charlie George Roadhouse, [Matthew Shardlow](https://md.research.elamur.ai/free/en/authors/matthew-shardlow/) and Ashley Williams · **Year:** 2026 · **Venue:** [Proceedings of the 1st Workshop on Information Disorder (InDor) @ LREC 2026](https://md.research.elamur.ai/free/en/venues/indor/) ([Workshop on Information Disorder (InDor) @ LREC 2026](https://md.research.elamur.ai/free/en/venues/indor/2026/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.indor-1.4/) · [PDF](https://aclanthology.org/2026.indor-1.4.pdf) · [DOI 10.63317/5c8w857kxgsn](https://doi.org/10.63317/5c8w857kxgsn) · [OpenAlex](https://openalex.org/W7166514101) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary Check-worthiness detectors decide which claims fact-checkers should review first. They are usually trained on human-written statements, so it is unclear how they cope with claims rewritten by LLMs, a growing channel for disinformation. The authors test this on English data from CheckIt (tweets with fine-grained annotations) and CheckThatLab! 2024 (tweets, debates and speeches). Llama 3.1:8B-Instruct paraphrased each claim in three styles while keeping named entities fixed: syntactic restructuring, syntactic complexity and lexical informality. They benchmarked BERT and RoBERTa fine-tuned on human claims, the zero-shot Llama 3.1:8B, and the commercial ClaimBuster API. They then built "robust" encoders by adding the paraphrases to the training data as adversarial examples. All systems performed worse on synthetic claims, especially complex and informal ones. Adversarial training recovered much of the loss. SHAP explanations showed that base models depend on shallow cues such as active voice. Robust models focus on factual entities instead. The work argues that fact-checking pipelines need training that accounts for writing style. ## Key points - Four systems were evaluated: - BERT and RoBERTa, each in a base and a robust variant; - zero-shot Llama 3.1:8B with greedy decoding; - ClaimBuster, a BERT-based industry tool. - Datasets were CheckIt and CheckThatLab! 2024. Encoders were trained on CTL 24 using AdamW (lr 2×10⁻⁵, weight decay 0.01), up to 10 epochs, with early stopping patience 3. - Paraphrases came from Llama 3.1:8B-Instruct using these sampling settings: - top-p 0.9; - temperature 0.8; - top-k 40; - repetition penalty 1.1. - Adversarially trained RoBERTa gained up to +5.22 F1 on CheckIt. - Code, models and data are released on GitHub (chroadhouse/Benchmarking-Check-Worthiness-Models-on-LLM-Generated-Claims). *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Manchester Metropolitan University](https://md.research.elamur.ai/free/en/universities/manchester-metropolitan-university/) — University, Manchester, United Kingdom · 5 works · 205 citations 📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,954 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works · [🤝 Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) · 626 works ## Cite Cite as: Charlie George Roadhouse, Matthew Shardlow and Ashley Williams. 2026. Benchmarking Check-Worthiness Models on LLM Generated Claims. Proceedings of the 1st Workshop on Information Disorder (InDor) @ LREC 2026. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{roadhouse2026benchmarking, title = {{Benchmarking Check-Worthiness Models on LLM Generated Claims}}, author = {Charlie George Roadhouse, Matthew Shardlow and Ashley Williams}, booktitle = {Proceedings of the 1st Workshop on Information Disorder (InDor) @ LREC 2026}, year = {2026}, doi = {10.63317/5c8w857kxgsn}, url = {https://doi.org/10.63317/5c8w857kxgsn} } ``` - [← previous Benchmarking Arabic — Russian Machine Translation: A Comparison of Fine-tuned NMT and Few-shot LLMs under Rich Morphology and Low Lexical Overlap](https://md.research.elamur.ai/free/en/papers/arabov-2026-benchmarking-arabic-russian-machine-translation/) - [next → Benchmarking Deflection and Hallucination in Large Vision-Language Models](https://md.research.elamur.ai/free/en/papers/moratelli-2026-benchmarking-deflection-hallucination-large-vision/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/24/) ## References 2 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Sajjad Ahmed, Klestia Balla, Knut Hinkelmann, and Flavio Corradini. 2021. Fact Checking: Detection of Check Worthy Statements Through Support Vector Machine and Feed Forward Neural Network. In Advances in Information and Communication, pages 520–535, Cham. Springer International Publishing. [find](https://scholar.google.com/scholar?q=Sajjad+Ahmed%2C+Klestia+Balla%2C+Knut+Hinkelmann%2C+and+Flavio+Corradini%2E+2021%2E+Fact+Checking%3A+Detection+of+Check+Worthy+Statements+Through+Support+Vector+Machine+and+Feed+Forward+Neural+Network%2E+In+Advance) 2. Esma Aïmeur, Sabrine Amri, and Gilles Brassard. 2023. Fake news, disinformation and misinformation in social media: a review. Social Network Analysis and Mining, 13(1):30. \#\# Limitations [find](https://scholar.google.com/scholar?q=Esma+A%C3%AFmeur%2C+Sabrine+Amri%2C+and+Gilles+Brassard%2E+2023%2E+Fake+news%2C+disinformation+and+misinformation+in+social+media%3A+a+review%2E+Social+Network+Analysis+and+Mining%2C+13%281%29%3A30%2E+%5C%23%5C%23+Limitations) *Updated 1 November 2026*