--- title: "Ta'keed: The First Generative Fact-Checking — Alsalka; 2024" description: "Arabic automatic fact-checking has mostly been treated as a labelling task: a claim is tagged true or false, and the user gets no reasoning." authors: ["Saud Althabiti; Mohammad Ammar Alsalka;", "Eric Atwell"] year: 2024 venue: "arXiv:2401.14067 · VOLUME 14 NUMBER 01 2024" source: "https://arxiv.org/abs/2401.14067" doi: "10.48550/arxiv.2401.14067" arxiv: "2401.14067" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/althabiti-2024-ta-keed-first-generative-fact/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/althabiti-2024-ta-keed-first-generative-fact/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / Ta'keed: The First Generative Fact-Checking System for Arabic Claims # Ta'keed: The First Generative Fact-Checking System for Arabic Claims **Authors:** Saud Althabiti; Mohammad Ammar Alsalka;, [Eric Atwell](https://md.research.elamur.ai/free/en/authors/eric-atwell/) · **Year:** 2024 · **Venue:** arXiv:2401.14067 · VOLUME 14 NUMBER 01 2024 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2401.14067) · [PDF](https://arxiv.org/pdf/2401.14067) · [DOI 10.48550/arxiv.2401.14067](https://doi.org/10.48550/arxiv.2401.14067) · [arXiv 2401.14067](https://arxiv.org/abs/2401.14067) · [OpenAlex](https://openalex.org/W4391272656) **Citations:** 0 citations (openalex-snapshot · 2026-10-14) · [4](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary Arabic automatic fact-checking has mostly been treated as a labelling task: a claim is tagged true or false, and the user gets no reasoning. Ta'keed is a system that gives a verdict for an Arabic tweet and also writes a justification for it. The pipeline has two stages. First, the tweet is cleaned of URLs, usernames and hashtags and sent to Google, and the returned titles, links and snippets are kept as evidence. Second, an OpenAI model (text-davinci-003) receives a role-based prompt with the tweet and the snippets and outputs a label (True, False or Other) plus an explanation. A web interface was built with Anvil. The ArCOV19-Rumors labels turned out to be problematic for this setup, so the authors built ArFactEx, a 100-tweet test set with gold labels and hand-written justifications. The first configuration reached F1 0.72. The generated explanations averaged 0.76 cosine similarity to the gold explanations, and using seven snippets raised F1 to 0.77. The work offers a first Arabic baseline for explainable, retrieval-backed claim checking. ## Key points - The architecture has two parts: preprocessing plus Google retrieval (via requests-HTML), then LLM verification with text-davinci-003 using system, user and assistant prompt roles. - ArFactEx (Arabic Facts with Explanations) has 100 tweets: - Half are false claims taken from debunking accounts such as No_Rumors and stoprumorsdz. - Half are true items from trusted outlets such as Saudi Arabia News. - Each tweet has a label, a short explanation, an extended explanation and the extended explanation's sources. - ArCOV19-Rumors (138 verified COVID-19 claims, over 9,000 related tweets) was tried first but set aside because of labelling issues. - Five experiments (named Tk#S, e.g. Tk3S) vary how many snippets the model sees: - Three snippets gave F1 0.72. - Seven snippets gave the best result, F1 0.77. - Explanations were scored with ROUGE and with semantic similarity. The authors recommend semantic similarity, which averaged 0.76 cosine. ## Interesting ideas - Treating the amount of evidence as a controlled variable (top-k snippets) shows how much retrieved context helps before it becomes noise, which is a cheap way to tune any retrieval-backed component. — For Mova (answer): Fable and school-test readers could vary how many supporting sentences or knowledge-graph facts they consult, to find where accuracy peaks. - Scoring generated justifications by meaning similarity instead of surface n-gram overlap avoids penalising correct paraphrases, which ROUGE tends to do. — For Mova (back): The text → graph → text compressor could be judged on whether the regenerated text matches the original's meaning, with ROUGE kept only as a secondary check. - Each gold item comes in layers (verdict, short reason, extended reason, cited sources), so one test set can evaluate classification, brief justification and source grounding separately. — For Mova (learn): Evaluation sets for answers could store a label, a short rationale and the supporting sentence IDs, so explainability is tested and not just correctness. - Offering a third "Other" verdict next to True and False keeps the system from forcing a binary call when the evidence is unclear. — For Mova (answer): Readers could return an explicit "not determinable from the text" outcome when the world model lacks a supporting state or event, as a negative-control category. - Stripping URLs, handles and hashtags before querying turned searches that returned nothing into usable ones, a reminder that platform noise ruins downstream lookup. — For Mova (tokens): The tokenizer could tag social-media artefacts as their own token types, so later stages can drop or keep them deliberately. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) ## Cited by Cited by 4 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 3 not yet. - [ARAFA: An LLM-Generated Arabic Fact-Checking Dataset](https://md.research.elamur.ai/free/en/papers/khalil-2026-arafa-llm-generated-arabic-fact/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{alsalka2024ta, title = {{Ta'keed: The First Generative Fact-Checking System for Arabic Claims}}, author = {Alsalka;, Saud Althabiti; Mohammad Ammar and Atwell, Eric}, year = {2024}, doi = {10.48550/arxiv.2401.14067}, eprint = {2401.14067}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2401.14067} } ``` - [← previous Syntagmatic and Paradigmatic Analysis of Lexemes for ‘Rain’ in Biblical Hebrew](https://md.research.elamur.ai/free/en/papers/pratesi-2024-syntagmatic-paradigmatic-analysis-lexemes-rain/) - [next → TAO at StanceEval2024 Shared Task: Arabic Stance Detection using AraBERT](https://md.research.elamur.ai/free/en/papers/melhem-2024-tao-stanceeval2024-shared-task-arabic/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) ## References 3 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. K. Shu, A. Sliva, S. Wang, J. Tang, and H. Liu, “Fake news detection on social media: A data mining perspective,” ACM SIGKDD explorations newsletter, vol. 19, no. 1, pp. 22–36, 2017. M. R. Jahng, H. Lee, and A. Rochadiat, “Public relations practitioners’ management of fake news: Exploring key elements and acts of information authentication,” Public Relat Rev, vol. 46, no. 2, p. 101907, 2020. E. C. Tandoc Jr, Z. W. Lim, and R. Ling, “Defining ‘fake news’ A typology of scholarly definitions,” Digital journalism, vol. 6, no. 2, pp. 137–153, 2018. S. Tejedor, M. Portalés-Oliva, R. Carniel-Bugs, and L. Cervi, “Journalism students and information consumption in the era of fake news,” Media Commun, vol. 9, no. 1, pp. 338–350, 2021. A. Andorfer, “Spreading like wildfire: Solutions for abating the fake news problem on social media via technology controls and government regulation,” Hastings LJ, vol. 69, p. 1409, 2017. M. Rapti, G. Tsakalidis, S. Petridou, and K. Vergidis, “Fake News Incidents through the Lens of the DCAM Disinformation Blueprint,” Information, vol. 13, no. 7, p. 306, 2022. S. A. Khan, M. H. Alkawaz, and H. M. Zangana, “The use and abuse of social media for spreading fake news,” in 2019 IEEE International Conference on Automatic Control and Intelligent Systems (I2CACIS), IEEE, 2019, pp. 145–148. Y. M. Rocha, G. A. de Moura, G. A. Desidério, C. H. de Oliveira, F. D. Lourenço, and L. D. de Figueiredo Nicolete, “The impact of fake news on social media and its influence on health during the COVID-19 pandemic: A systematic review,” J Public Health (Bangkok), pp. 1–10, 2021. S. Althabiti, M. A. Alsalka, and E. Atwell, “SCUoL at CheckThat! 2022: fake news detection using transformer-based models,” in CEUR Workshop Proceedings, CEUR Workshop Proceedings, 2022, pp. 428–433. [find](https://scholar.google.com/scholar?q=K%2E+Shu%2C+A%2E+Sliva%2C+S%2E+Wang%2C+J%2E+Tang%2C+and+H%2E+Liu%2C+%E2%80%9CFake+news+detection+on+social+media%3A+A+data+mining+perspective%2C%E2%80%9D+ACM+SIGKDD+explorations+newsletter%2C+vol%2E+19%2C+no%2E+1%2C+pp%2E+22%E2%80%9336%2C+2017%2E+M%2E+R%2E+Jahng%2C+H%2E+L) 2. P. Nakov et al., “The CLEF-2021 CheckThat! lab on detecting check-worthy claims, previously fact-checked claims, and fake news,” in European Conference on Information Retrieval, Springer, 2021, pp. 639–649. S. Shaar et al., “Overview of the CLEF-2021 CheckThat! lab task 1 on check-worthiness estimation in tweets and political debates,” Working Notes of CLEF, 2021. H. Saadany, E. Mohamed, and C. Orasan, “Fake or real? A study of Arabic satirical fake news,” arXiv preprint arXiv:2011.00452, 2020. G. Jardaneh, H. Abdelhaq, M. Buzz, and D. Johnson, “Classifying Arabic tweets based on credibility using content and user features,” in 2019 IEEE Jordan International Joint Conference on Electrical Engineering and Information Technology, JEEIT 2019 - Proceedings, 2019. doi: 10.1109/JEEIT.2019.8717386. F. Haouari, M. Hasanain, R. Suwaileh, and T. Elsayed, “ArCOV19-rumors: Arabic COVID-19 twitter dataset for misinformation detection,” arXiv preprint arXiv:2010.08768, 2020. S. Althabiti, M. Alsalka, and E. Atwell, “SCUoL at CheckThat! 2021: An AraBERT model for check-worthiness of Arabic tweets,” in CEUR Workshop Proceedings, 2021. S. Althabiti, M. A. Alsalka, and E. Atwell, “Detecting Arabic Fake News on Social Media using Sarcasm and Hate Speech in Comments,” International Journal on Islamic Applications in Computer Science And Technology, vol. 10, no. 4, pp. 28–36, 2022, [Online]. Available: http://sign-ific-ance.co.uk/index.php/IJASAT/article/view/2563 S. Althabiti, M. A. Alsalka, and E. Atwell, “A Survey: Datasets and Methods for Arabic Fake News Detection,” International Journal on Islamic Applications in Computer Science And Technology, vol. 11, no. 1, 2023, [Online]. Available: http://www.sign-ific-ance.co.uk/index.php/IJASAT/article/view/2577 T. Alhindi, S. Petridis, and S. Muresan, “Where is your evidence: Improving fact-checking by justification modeling,” in Proceedings of the first workshop on fact extraction and verification (FEVER), 2018, pp. 85–90. W. Y. Wang, “‘ liar, liar pants on fire’: A new benchmark dataset for fake news detection,” arXiv preprint arXiv:1705.00648, 2017. N. Kotonya and F. Toni, “Explainable automated fact-checking for public health claims,” arXiv preprint arXiv:2010.09926, 2020. D. Stammbach and E. Ash, “e-fever: Explanations and summaries for automated fact checking,” Proceedings of the 2020 Truth and Trust Online (TTO 2020), pp. 32–43, 2020. J. Thorne, A. Vlachos, C. Christodoulopoulos, and A. Mittal, “Fever: a large-scale dataset for fact extraction and verification,” arXiv preprint arXiv:1803.05355, 2018. S. Althabiti, M. A. Alsalka, and E. Atwell, “Generative AI for Explainable Automated Fact Checking on the FactEx: A New Benchmark Dataset,” in Multidisciplinary International Symposium on Disinformation in Open Online Media, Springer, 2023, pp. 1–13. Y. Zhang et al., “Tanbih: Get to know what you are reading,” arXiv preprint arXiv:1910.02028, 2019. [↗](https://doi.org/10.1109/jeeit.2019.8717386) [our card](https://md.research.elamur.ai/free/en/papers/thorne-2018-fever-large-scale-dataset-fact/) 3. Z. Sheikh Ali, W. Mansour, F. Haouari, M. Hasanain, T. Elsayed, and A. Al-Ali, “Tahaqqaq: A Real-Time System for Assisting Twitter Users in Arabic Claim Verification,” in Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2023, pp. 3170–3174. Z. S. Ali, W. Mansour, T. Elsayed, and A. Al‐Ali, “AraFacts: The First Large Arabic Dataset of Naturally Occurring Claims,” in Proceedings of the Sixth Arabic Natural Language Processing Workshop, 2021, pp. 231–236. H. Hassani and E. S. Silva, “The role of ChatGPT in data science: how ai-assisted conversational interfaces are revolutionizing the field,” Big data and cognitive computing, vol. 7, no. 2, p. 62, 2023. M. Zong and B. Krishnamachari, “a survey on GPT-3,” arXiv preprint arXiv:2212.00857, 2022. S. Althabiti, M. A. Alsalka, and E. Atwell, “Google Snippets and Twitter Posts; Examining Similarities to Identify Misinformation,” in Proceedings of the Workshop on NLP applied to Misinformation, CEUR, 2023, pp. 36–44. F. Haouari, M. Hasanain, R. Suwaileh, and T. Elsayed, “Arcov-19: The first arabic covid-19 twitter dataset with propagation networks,” arXiv preprint arXiv:2004.05861, 2020. C.-Y. Lin, “Rouge: A package for automatic evaluation of summaries,” in Text summarization branches out, 2004, pp. 74–81. [↗](https://arxiv.org/abs/2212.00857)