---
title: "Kashif-AI at AbjadGenEval Shared Task: A Transformer-based"
description: "Kashif-AI is a system for AbjadGenEval Task 1, which asks whether an Arabic news article was written by a… 66.29% F1. 21 references."
authors: ["Emad Eldin, Fatimah Mohamed"]
year: 2026
venue: "Proceedings of the 2nd Workshop on NLP for Languages Using Arabic Script"
source: "https://aclanthology.org/2026.abjadnlp-1.60/"
doi: "10.18653/v1/2026.abjadnlp-1.60"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/eldin-2026-kashif-ai-abjadgeneval-shared-task/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/eldin-2026-kashif-ai-abjadgeneval-shared-task/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/33/) / Kashif-AI at AbjadGenEval Shared Task: A Transformer-based Approach for Arabic AI-Generated Text Detection

# Kashif-AI at AbjadGenEval Shared Task: A Transformer-based Approach for Arabic AI-Generated Text Detection

**Authors:** [Emad Eldin, Fatimah Mohamed](https://md.research.elamur.ai/free/en/authors/emad-eldin-fatimah-mohamed/) · **Year:** 2026 · **Venue:** [Proceedings of the 2nd Workshop on NLP for Languages Using Arabic Script](https://md.research.elamur.ai/free/en/venues/abjadnlp/) ([Workshop on NLP for Languages Using Arabic Script 2026](https://md.research.elamur.ai/free/en/venues/abjadnlp/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.abjadnlp-1.60/) · [PDF](https://aclanthology.org/2026.abjadnlp-1.60.pdf) · [DOI 10.18653/v1/2026.abjadnlp-1.60](https://doi.org/10.18653/v1/2026.abjadnlp-1.60) · [OpenAlex](https://openalex.org/W7140125439)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Kashif-AI is a system for AbjadGenEval Task 1, which asks whether an Arabic news article was written by a person or produced by an LLM. The author frames it as supervised binary classification. Raw text first passes through regex cleaning that strips URLs and tatweel. A fine-tuned Arabic encoder, either MARBERT or CAMeLBERT-Mix, then reads the text, and its [CLS] vector feeds the label decision. The main experiment is an ablation crossing the two encoders with two training sets: the balanced official data, and a much larger merge that adds an external corpus heavily skewed toward machine text.

On internal validation every configuration looked almost flawless, but the official blind test was far less kind. The best result came from CAMeLBERT-Mix trained only on the official data, with 66.29% F1 and 70.5% accuracy. Adding the external corpus made blind-test results worse. The author blames class imbalance and source-specific artifacts that the model learned as shortcuts. The paper's value is a warning: more data from a different source can hurt, and in-distribution validation can badly overstate real performance.

## Key points
- **Data:**
  - The official training set has 4,800 articles, split evenly between human and machine. Its machine texts come from generators such as GPT-3.5, GPT-4 and Claude.
  - The external set (Al-Shaibani and Ahmed, 2025) has 41,940 samples: 8,637 human and 33,303 machine.
  - The two together give a Combined set of 46,740 samples.
  - The blind test has 2,000 items.
- **Training setup:**
  - AdamW, learning rate 2e-5, weight decay 0.01, linear scheduler.
  - 3 epochs, 512-token maximum length, batch size 16.
  - Stratified 80/20 train/validation split, keeping the checkpoint with the best validation F1.
- **Validation vs. blind test:** validation F1 was 0.992–0.998 everywhere. On the blind test:
  - MARBERT reached F1 0.632 and accuracy 0.650 on Base data, versus 0.636 and 0.570 on Combined data.
  - CAMeLBERT reached 0.663 and 0.705 on Base data, versus 0.561 and 0.570 on Combined data.
- **Encoder choice:** CAMeLBERT-Mix beat MARBERT on blind data. The author links this to its formal MSA pre-training matching news better than MARBERT's tweet-heavy pre-training.
- **Per-generator errors:** on a validation split of N = 7,086, OpenAI-generated texts were caught every time. Llama and JAIS outputs had slightly higher error rates, 1.60% and 1.25%, which hints at easily learnable generator fingerprints.

## Interesting ideas
- Almost perfect scores on a validation split taken from the training distribution said little about the blind test, so only a truly external test set reveals real generalization.
- Merging a large external corpus lowered blind-test quality because it shifted the label balance and brought in source-specific quirks, so an ablation with and without each added source is worth running.
- Splitting errors by generator showed which sources were "too easy," a sign of shortcut artifacts rather than genuine skill.
- The encoder whose pre-training domain matched the target texts (formal news) generalized better than one pre-trained mostly on social media.
- A light regex pass removing noise like URLs and decorative letter elongation ran before tokenization, keeping the model's input clean.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Always score each training run on a held-out set drawn from a different source than the silver data, treating it as a negative control against in-distribution overconfidence.
- **I2** (learn): Before folding a new silver or LLM-teacher batch into training, compare its label and construction distribution with the existing data and test the merge by ablation rather than assuming more is better.
- **I3** (tree): Break parser errors down by source treebank or text genre, so suspiciously perfect subsets that may reflect annotation conventions rather than syntax stand out.
- **I4** (tree): When picking or pre-training the encoder for gparse, prefer corpora close to the target narrative and school-text genres over web chatter.
- **I5** (tokens): A similar explicit, logged normalization step for URLs and stretched spellings in English would keep the tokenizer and tagger stable while staying explainable.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Cairo University](https://md.research.elamur.ai/free/en/universities/cairo-university/) — University, Giza, Egypt · 42 works · 1,350 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works

## Cite

Cite as: Emad Eldin, Fatimah Mohamed. 2026. Kashif-AI at AbjadGenEval Shared Task: A Transformer-based Approach for Arabic AI-Generated Text Detection. Proceedings of the 2nd Workshop on NLP for Languages Using Arabic Script.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{mohamed2026kashif,
  title = {{Kashif-AI at AbjadGenEval Shared Task: A Transformer-based Approach for Arabic AI-Generated Text Detection}},
  author = {Emad Eldin, Fatimah Mohamed},
  booktitle = {Proceedings of the 2nd Workshop on NLP for Languages Using Arabic Script},
  year = {2026},
  doi = {10.18653/v1/2026.abjadnlp-1.60},
  url = {https://doi.org/10.18653/v1/2026.abjadnlp-1.60}
}
```

- [← previous KARL: Reinforcement Learning for LLM Agents on Multi-Turn Knowledge-Intensive Agentic Tasks](https://md.research.elamur.ai/free/en/papers/sun-2026-karl-reinforcement-learning-llm-agents/)
- [next → KCLarity at SemEval-2026 Task 6: Encoder and Zero-Shot Approaches to Political Evasion Detection](https://md.research.elamur.ai/free/en/papers/sage-2026-kclarity-semeval-2026-task-6/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/33/)

## References

21 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. [ARBERT & MARBERT: Deep bidirectional transformers for Arabic](https://md.research.elamur.ai/free/en/papers/abdul-mageed-2021-arbert-marbert-deep-bidirectional-transformers/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2021.acl-long.551)
2. Shadi Abudalfa, Saad Ezzini, Ahmed Abdelali, Hamza Alami, Abdessamad Benlahbib, Salmane Chafik, Mo El-Haj, Abdelkader El Mahdaouy, Mustafa Jarrar, Salima Lamsiyah, and Hamzah Luqman. 2025. [The AraGenEval shared task on Arabic authorship style transfer and AI generated text detection](https://md.research.elamur.ai/free/en/papers/abudalfa-2025-arageneval-shared-task-arabic-authorship/). In Proceedings of The Third Arabic Natural Language Processing Conference at EMNLP 2025, Suzhou, China. Association for Computational Linguistics.
3. Maged S Al-Shaibani and Moataz Ahmed. 2025. [The arabic AI fingerprint: Stylometric analysis and detection of large language models text](https://md.research.elamur.ai/free/en/papers/al-shaibani-2025-arabic-ai-fingerprint-stylometric-analysis/). [↗](https://doi.org/10.48550/arxiv.2505.23276)
4. Hamed Alshammari and Khaled Elleithy. 2024. Toward robust arabic ai-generated text detection: Tackling diacritics challenges. Information, 15(7). [↗](https://doi.org/10.3390/info15070419)
5. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. [AraBERT: Transformer-based model for Arabic language understanding](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/). In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. [↗](https://doi.org/10.48550/arxiv.2003.00104)
6. Evan Crothers, Nathalie Japkowicz, and Herna Viktor. 2022. Machine generated text: A comprehensive survey of threat models and detection methods. [↗](https://doi.org/10.1109/access.2023.3294090)
7. In this paper, Kashif-AI was presented as a robust baseline for Arabic AI text detection. Through Jacob Devlin, Ming-Wei Chang, Kenton Lee, and a systematic ablation study, it was demonstrated Kristina Toutanova. 2019. BERT: Pre-training of that fine-tuning CAMeLBERT-Mix on the balanced deep bidirectional transformers for language under-official dataset yielded the highest performance, standing. In Proceedings of the 2019 Conference of achieving 70.5% accuracy. The study further re-the North American Chapter of the Association for 486 Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. Detection of AI-Generated Text. In Proceedings of the Shared Task on Multi-Domain Detection of AI-Generated Text, pages 1–9, Varna, Bulgaria. IN-COMA Ltd., Shoumen, Bulgaria. [find](https://scholar.google.com/scholar?q=In+this+paper%2C+Kashif%2DAI+was+presented+as+a+robust+baseline+for+Arabic+AI+text+detection%2E+Through+Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+a+systematic+ablation+study%2C+it+was+demonstrated+Kristin)
8. Saad Ezzini, Irfan Ahmed, Salmane Chafik, Shadi Abudalfa, Mo El-Haj, Ahmed Abdelali, Mustafa Jarrar, Nadir Durrani, Hassan Sajjad, and Farah Adeeba. 2026. [Abjadgeneval: Abjad ai generated text detection shared task for languages using arabic script at abjadnlp 2026](https://md.research.elamur.ai/free/en/papers/ezzini-2026-abjadgeneval-abjad-ai-generated-text/). In Proceedings of the 2nd Workshop on NLP for Languages Using Arabic Script (AbjadNLP 2026), co-located with the 19th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2026), Rabat, Morocco.
9. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Ro{bert}a: A robustly optimized {bert} pretraining approach. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Ro%7Bbert%7Da%3A+A+robustly+optimized+%7Bbert%7D+pretraining+approac)
10. Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. 2019. [GLTR: Statistical detection and visualization of generated text](https://md.research.elamur.ai/free/en/papers/gehrmann-2019-gltr-statistical-detection-visualization-generated/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 111–116, Florence, Italy. Association for Computational Linguistics.
11. Cyril Goutte and Eric Gaussier. 2005. A probabilistic interpretation of precision, recall and f-score, with implication for evaluation. ECIR’05, page 345–359, Berlin, Heidelberg. Springer-Verlag. [find](https://scholar.google.com/scholar?q=Cyril+Goutte+and+Eric+Gaussier%2E+2005%2E+A+probabilistic+interpretation+of+precision%2C+recall+and+f%2Dscore%2C+with+implication+for+evaluation%2E+ECIR%E2%80%9905%2C+page+345%E2%80%93359%2C+Berlin%2C+Heidelberg%2E+Springer%2DVerlag%2E)
12. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. [The interplay of variant, size, and task type in Arabic pre-trained language models](https://md.research.elamur.ai/free/en/papers/inoue-2021-interplay-variant-size-task-type/). In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92– 104, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
13. Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. 2020. [Automatic detection of generated text is easiest when humans are fooled](https://md.research.elamur.ai/free/en/papers/ippolito-2019-automatic-detection-generated-text-easiest/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1808–1822, Online. Association for Computational Linguistics.
14. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. [↗](https://doi.org/10.48550/arxiv.1711.05101)
15. Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. 2023. [DetectGPT: Zero-shot machine-generated text detection using probability curvature](https://md.research.elamur.ai/free/en/papers/mitchell-2023-detectgpt-zero-shot-machine-generated/).
16. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Alham Fikri Aji, Zhengzhong Liu, Andy Hock, Andrew Feldman, Jonathan Lee, Andrew Jackson, and 3 others. 2023. [Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models](https://md.research.elamur.ai/free/en/papers/sengupta-2023-jais-jais-chat-arabic-centric/). Preprint, arXiv:2308.16149. [↗](https://arxiv.org/abs/2308.16149)
17. Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, Miles McCain, Alex Newhouse, Jason Blazakis, Kris McGuffie, and Jasmine Wang. 2019. [Release strategies and the social impacts of language models](https://md.research.elamur.ai/free/en/papers/solaiman-2019-release-strategies-social-impacts-language/). [↗](https://doi.org/10.48550/arxiv.1908.09203)
18. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. [Defending against neural fake news](https://md.research.elamur.ai/free/en/papers/zellers-2019-defending-against-neural-fake-news/). [↗](https://doi.org/10.48550/arxiv.1905.12616)
19. Ganesh Jawahar, Muhammad Abdul-Mageed, and Laks Lakshmanan, V.S. 2020. [Automatic detection of machine generated text: A critical survey](https://md.research.elamur.ai/free/en/papers/jawahar-2020-automatic-detection-machine-generated-text/). In Proceedings of the 28th International Conference on Computational Linguistics, pages 2296–2309, Barcelona, Spain (Online). International Committee on Computational Linguistics.
20. John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. 2023. [A watermark for large language models](https://md.research.elamur.ai/free/en/papers/kirchenbauer-2023-watermark-large-language-models/). In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 17061–17084. PMLR. [↗](https://doi.org/10.48550/arxiv.2301.10226)
21. Salima Lamsiyah, Saad Ezzini, Abdelkader El Mahdaouy, Hamza Alami, Abdessamad Benlahbib, Samir El amrany, Salmane Chafik, and Hicham Hammouchi. 2025. M-DAIGT: A Shared Task on Multi-Domain [find](https://scholar.google.com/scholar?q=Salima+Lamsiyah%2C+Saad+Ezzini%2C+Abdelkader+El+Mahdaouy%2C+Hamza+Alami%2C+Abdessamad+Benlahbib%2C+Samir+El+amrany%2C+Salmane+Chafik%2C+and+Hicham+Hammouchi%2E+2025%2E+M%2DDAIGT%3A+A+Shared+Task+on+Multi%2DDomain)

*Updated 1 November 2026*