---
title: "PTUK-HULAT at ArAIEval Shared Task Fine-tuned Distilbert to"
description: "The paper describes the PTUK-HULAT entry for Subtask 2A of the ArAIEval shared task at… 87% F1. 19 references, cited by 2 papers here."
authors: ["Areej Jaber", "Paloma Martinez"]
year: 2023
venue: "Proceedings of ArabicNLP 2023"
source: "https://aclanthology.org/2023.arabicnlp-1.50/"
doi: "10.18653/v1/2023.arabicnlp-1.50"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/jaber-2023-ptuk-hulat-araieval-shared-task/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jaber-2023-ptuk-hulat-araieval-shared-task/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/12/) / PTUK-HULAT at ArAIEval Shared Task Fine-tuned Distilbert to Predict Disinformative Tweets

# PTUK-HULAT at ArAIEval Shared Task Fine-tuned Distilbert to Predict Disinformative Tweets

**Authors:** [Areej Jaber](https://md.research.elamur.ai/free/en/authors/areej-jaber/) and [Paloma Martinez](https://md.research.elamur.ai/free/en/authors/paloma-martinez/) · **Year:** 2023 · **Venue:** [Proceedings of ArabicNLP 2023](https://md.research.elamur.ai/free/en/venues/arabicnlp/) ([ArabicNLP 2023](https://md.research.elamur.ai/free/en/venues/arabicnlp/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.arabicnlp-1.50/) · [PDF](https://aclanthology.org/2023.arabicnlp-1.50.pdf) · [DOI 10.18653/v1/2023.arabicnlp-1.50](https://doi.org/10.18653/v1/2023.arabicnlp-1.50) · [OpenAlex](https://openalex.org/W4389518374)

**Citations:** 4 citations (OpenAlex, 2026-10-14) · [2](https://md.research.elamur.ai/free/en/papers/jaber-2023-ptuk-hulat-araieval-shared-task/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper describes the PTUK-HULAT entry for Subtask 2A of the ArAIEval shared task at ArabicNLP 2023. The task is binary: decide whether an Arabic tweet is disinformative. The organisers' dataset was gathered with Twarc using the Arabic word for "corona" during February–March 2020, and it is strongly skewed toward the non-disinformative class. The authors did light cleaning, stripping punctuation, hashtags, URLs and special characters. They then fine-tuned multilingual DistilBERT, choosing it to cope with scarce Arabic resources and with the computational cost of full-size BERT.

On the test set the system reached about 87% F1 micro and 80% F1 macro. That clearly beats the random and majority baselines and placed 11th among participants. Per-class results show a sharp gap between the minority class (68%) and the majority class (92%). Error analysis points to two culprits: English words written in Arabic script, and letters stretched out to convey emotion.

The work matters as a simple, reproducible reference point for low-resource Arabic social-media classification.

## Key points
- **Data:** ArAIEval Subtask 2A data splits into 14,147 training, 2,115 development and 3,729 test tweets. Only about a fifth carry the "disinfo" label, for example 2,656 in training and 876 in test.
- **Model:** multilingual DistilBERT (Sanh et al., 2019), pretrained on Wikipedia in 104 languages. It has 6 layers, 768 hidden dimensions and 12 attention heads, and was fine-tuned for 2 epochs at learning rate 2e-5.
- **Test scores:** F1 micro 0.8675 and F1 macro 0.7992. The majority baseline reached 0.7651 / 0.4335 and the random baseline 0.5154 / 0.4764. Development scores were lower, at 81% / 72%.
- **Errors:** 494 test tweets were misclassified. Of these, 346 were gold "disinfo" and 148 gold "no-disinfo".
- **Failure sources:** transliterated English (for example "follow" spelled in Arabic letters) and repeated characters used for emotional emphasis.

## Interesting ideas
- Placing a majority baseline beside micro and macro F1 shows how much the micro score is inflated by class imbalance. The majority baseline's micro F1 looks respectable, yet its macro F1 collapses.
- Splitting the errors by gold class revealed that roughly 70% of mistakes fell on the minority label. That points to where effort should go next.
- Letters repeated for emotional effect broke the classifier, which suggests elongation should be handled explicitly rather than left to subword splitting.
- Foreign words written in another script, such as English in Arabic letters, were a distinct error category worth isolating.
- The authors chose a distilled student model because it keeps most of the teacher's ability while being smaller and faster.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): every evaluation report could automatically include majority and random negative controls plus macro-averaged scores, so imbalanced test sets cannot hide weak handling of rare categories.
- **I2** (learn): self-induced repair rules could be ranked by which gold class their errors concentrate in, prioritising rare constructions over already-saturated frequent ones.
- **I3** (tokens): the tokenizer could collapse forms like "sooooo" to the standard lemma while recording an intensity feature, which crate prag could later use as an emphasis or irony cue.
- **I4** (tokens): flag loan or transliterated tokens and nonstandard spellings like "gonna" as a separate class with mapping to canonical lemmas, instead of letting them fall through as unknown words.
- **I5** (learn): the same student-mimics-teacher setup supports compressing LLM teacher tiers into a fast local model, such as a lighter gparse, as long as its outputs stay inspectable.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Palestine Technical University - Kadoorie](https://md.research.elamur.ai/free/en/universities/palestine-technical-university-kadoorie/) — University, Tulkarm, Palestine · 4 works · 20 citations
- [Universidad Carlos III de Madrid](https://md.research.elamur.ai/free/en/universities/universidad-carlos-iii-de-madrid/) — University, Madrid, Spain · 4 works · 11 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works · [🤝 Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) · 577 works

🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works

## Cite

Cite as: Areej Jaber and Paloma Martinez. 2023. PTUK-HULAT at ArAIEval Shared Task Fine-tuned Distilbert to Predict Disinformative Tweets. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{jaber2023ptuk,
  title = {{PTUK-HULAT at ArAIEval Shared Task Fine-tuned Distilbert to Predict Disinformative Tweets}},
  author = {Areej Jaber and Paloma Martinez},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.50},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.50}
}
```

- [← previous PairSpanBERT: An Enhanced Language Model for Bridging Resolution](https://md.research.elamur.ai/free/en/papers/kobayashi-2023-pairspanbert-enhanced-language-model-bridging/)
- [next → The DARPA Wikidata Overlay: Wikidata as an ontology for natural language processing](https://md.research.elamur.ai/free/en/papers/spaulding-2023-darpa-wikidata-overlay-wikidata-as/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/12/)

## References

19 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Farah A Abdulghani and Nada AZ Abdullah. 2022. A survey on arabic text classification using deep and machine learning algorithms. Iraqi Journal of Science, pages 409--419. [↗](https://doi.org/10.24996/ijs.2022.63.1.37)
2. Firoj Alam, Hamdy Mubarak, Wajdi Zaghouani, Giovanni Da San Martino, and Preslav Nakov. 2022. [Overview of the WANLP 2022 shared task on propaganda detection in arabic](https://md.research.elamur.ai/free/en/papers/alam-2022-overview-wanlp-2022-shared-task/). In Proceedings of the The Seventh Arabic Natural Language Processing Workshop, WANLP@EMNLP 2022, Abu Dhabi, United Arab Emirates (Hybrid), December 8, 2022, pages 108--118. Association for Computational Linguistics.
3. Maysoon Alkhair, Karima Meftouh, Kamel Smaïli, and Nouha Othman. 2019. An arabic corpus of fake news: Collection, analysis and classification. In Arabic Language Processing: From Theory to Practice: 7th International Conference, ICALP 2019, Nancy, France, October 16--17, 2019, Proceedings 7, pages 292--302. Springer. [find](https://scholar.google.com/scholar?q=Maysoon+Alkhair%2C+Karima+Meftouh%2C+Kamel+Sma%C3%AFli%2C+and+Nouha+Othman%2E+2019%2E+An+arabic+corpus+of+fake+news%3A+Collection%2C+analysis+and+classification%2E+In+Arabic+Language+Processing%3A+From+Theory+to+Practice%3A+7)
4. Fouzi Harrag and Mohamed Khalil Djahli. 2022. Arabic fake news detection: A fact checking based deep learning approach. 21(4). [↗](https://doi.org/10.1145/3501401)
5. Maram Hasanain, Firoj Alam, Hamdy Mubarak, Samir Abdaljalil, Wajdi Zaghouani, Preslav Nakov, Giovanni Da San Martino, and Abdelhakim Freihat. 2023. [Araieval shared task: Persuasion techniques and disinformation detection in arabic text](https://md.research.elamur.ai/free/en/papers/hasanain-2023-araieval-shared-task-persuasion-techniques/). In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore. Association for Computational Linguistics.
6. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. [The state and fate of linguistic diversity and inclusion in the nlp world](https://md.research.elamur.ai/free/en/papers/joshi-2020-state-fate-linguistic-diversity-inclusion/). arXiv preprint arXiv:2004.09095. [↗](https://arxiv.org/abs/2004.09095)
7. Karima Kadaoui, Samar M Magdy, Abdul Waheed, Md Tawkat Islam Khondaker, Ahmed Oumar El-Shangiti, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. [Tarjamat: Evaluation of bard and chatgpt on machine translation of ten arabic varieties](https://md.research.elamur.ai/free/en/papers/kadaoui-2023-tarjamat-evaluation-bard-chatgpt-machine/). arXiv preprint arXiv:2308.03051. [↗](https://arxiv.org/abs/2308.03051)
8. Yida Mu, Pu Niu, and Nikolaos Aletras. 2022. Identifying and characterizing active citizens who refute misinformation in social media. In Proceedings of the 14th ACM Web Science Conference 2022, pages 401--410. [find](https://scholar.google.com/scholar?q=Yida+Mu%2C+Pu+Niu%2C+and+Nikolaos+Aletras%2E+2022%2E+Identifying+and+characterizing+active+citizens+who+refute+misinformation+in+social+media%2E+In+Proceedings+of+the+14th+ACM+Web+Science+Conference+2022%2C+pages)
9. Hamdy Mubarak, Samir Abdaljalil, Azza Nassar, and Firoj Alam. 2023. Detecting and reasoning of deleted tweets before they are posted. arXiv preprint arXiv:2305.04927. [↗](https://arxiv.org/abs/2305.04927)
10. Mona Nasery, Ofir Turel, and Yufei Yuan. 2023. Combating fake news on social media: A framework, review, and future opportunities. Communications of the Association for Information Systems, 53(1):9. [↗](https://doi.org/10.17705/1cais.05335)
11. Ramy Baly, Mitra Mohtarami, James Glass, Lluís Màrquez, Alessandro Moschitti, and Preslav Nakov. 2018. [Integrating stance detection and fact checking in a unified corpus](https://md.research.elamur.ai/free/en/papers/baly-2018-integrating-stance-detection-fact-checking/). arXiv preprint arXiv:1804.08012. [↗](https://arxiv.org/abs/1804.08012)
12. Ali Bou Nassif, Ashraf Elnagar, Omar Elgendy, and Yaman Afadar. 2022. [Arabic fake news detection based on deep contextualized embedding models](https://md.research.elamur.ai/free/en/papers/nassif-2022-arabic-fake-news-detection-based/). Neural Computing and Applications, 34(18):16019--16032. [↗](https://doi.org/10.1007/s00521-022-07206-4)
13. GHIZLANE Bourahouat, MANAR ABOUREZQ, and NAJIMA DAOUDI. 2023. Systematic review of the [arabic natural language processing](https://md.research.elamur.ai/free/en/papers/habash-2022-arabic-natural-language-processing/): Challenges, techniques and new trends. Journal of Theoretical and Applied Information Technology, 101(3).
14. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108. [↗](https://arxiv.org/abs/1910.01108)
15. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30.
16. Dr Bharat Dhiman. 2023. Ethical issues and challenges in social media: A current scenario. Available at SSRN 4406610. [↗](https://doi.org/10.20944/preprints202303.0513.v1)
17. Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. 2021. Knowledge distillation: A survey. Int. J. Comput. Vis., 129(6):1789--1819. [↗](https://doi.org/10.1007/s11263-021-01453-z)
18. Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Yuan Yao, Ao Zhang, Liang Zhang, et al. 2021. Pre-trained models: Past, present and future. AI Open, 2:225--250. [↗](https://doi.org/10.1016/j.aiopen.2021.08.002)
19. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. [Transformers: State-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38--45.

*Updated 1 November 2026*