--- title: "Benchmarking Transformer-based Language Models — Farha 2021" description: "Many transformer language models for Arabic have recently appeared, but each was tested on its own small set of tasks, so there was little head-to-head…" authors: ["Ibrahim Abu Farha", "Walid Magdy"] year: 2021 venue: "Proceedings of the Sixth Arabic Natural Language Processing Workshop" source: "https://aclanthology.org/2021.wanlp-1.3/" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/papers/farha-2021-benchmarking-transformer-based-language-models/" --- [Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/farha-2021-benchmarking-transformer-based-language-models/) [home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2021](https://md.research.elamur.ai/free/papers/2021/) / Benchmarking Transformer-based Language Models for Arabic Sentiment and Sarcasm Detection # Benchmarking Transformer-based Language Models for Arabic Sentiment and Sarcasm Detection **Authors:** [Ibrahim Abu Farha](https://md.research.elamur.ai/free/authors/ibrahim-abu-farha/), [Walid Magdy](https://md.research.elamur.ai/free/authors/walid-magdy/) · **Year:** 2021 · **Venue:** Proceedings of the Sixth Arabic Natural Language Processing Workshop · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2021.wanlp-1.3/) · [PDF](https://aclanthology.org/2021.wanlp-1.3.pdf) · [OpenAlex](https://openalex.org/W3153540814) **Citations:** 44 citations (openalex-snapshot · 2026-10-14) · [19](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Many transformer language models for Arabic have recently appeared, but each was tested on its own small set of tasks, so there was little head-to-head evidence on which works best. Abu Farha and Magdy address this by benchmarking 24 Arabic-capable models, including multilingual and Arabic-specific BERT, ELECTRA, ALBERT and GPT2 variants. They fine-tune each on two classification tasks, sentiment analysis and sarcasm detection, using the ArSarcasm-v2 dataset from the Arabic sarcasm shared task. They also record fine-tuning time. Models pre-trained only on Arabic, especially those whose corpora include dialectal social-media text, perform best. MARBERT tops both tasks. Monolingual models beat multilingual ones, and larger models generally win when trained on comparable data. AraELECTRA nearly matches much bigger models at a fraction of the cost. AraGPT2 variants trail the BERT-style encoders, which suggests causal language models suit classification poorly. The study, summarised in a single comparison table, gives practical guidance for choosing models in Arabic NLP. It is particularly useful when compute is limited or the input text is dialectal. ## Key points - **Best results:** MARBERT, pre-trained on about 128GB of text of which half is tweets, scored F PN 0.724 on sentiment and F1-sarcastic 0.584 on sarcasm. QARiB (0.72) and AraBERT-large v2 (0.724) were close on sentiment. - **Multilingual models lag:** mBERT scored lowest (0.625 sentiment, 0.425 sarcasm). Large multilingual models such as XLM-R-large only reach the level of smaller Arabic-specific models. - **Efficiency:** The AraELECTRA discriminator has 136M parameters and fine-tunes in about 2:40 per epoch (batch size 4). It scored 0.709 and 0.578, close to AraBERT-large, which has 371M parameters and needs roughly 5:50 per epoch. - **Fine-tuning time:** Time grows steadily with parameter count. Arabic ALBERT-base was fastest (1:56 per epoch) and AraGPT2-large (792M) was slowest (11:27). AraGPT2-Mega could not be fine-tuned at all on the authors' hardware because it ran out of memory. - **AraGPT2 underperforms:** All AraGPT2 variants scored about 0.63–0.67 on sentiment and about 0.52 on sarcasm, below the masked-LM encoders. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) ## Cited by Cited by 19 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 17 not yet. - [Whose Pragmatics? Cultural Grounding as a Bottleneck for Stereotype Detection in Egyptian Arabic Social Media](https://md.research.elamur.ai/free/papers/assem-2026-whose-pragmatics-cultural-grounding-as/) - [Sarcasm Detection is Way Too Easy! An Empirical Comparison of Human and Machine Sarcasm Detection](https://md.research.elamur.ai/free/papers/farha-2022-sarcasm-detection-is-way-too/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{farha2021benchmarking, title = {{Benchmarking Transformer-based Language Models for Arabic Sentiment and Sarcasm Detection}}, author = {Farha, Ibrahim Abu and Magdy, Walid}, booktitle = {Proceedings of the Sixth Arabic Natural Language Processing Workshop}, year = {2021}, url = {https://aclanthology.org/2021.wanlp-1.3/} } ``` - [← previous BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://md.research.elamur.ai/free/papers/thakur-2021-beir-heterogenous-benchmark-zero/) - [next → Biaffine Dependency and Semantic Graph Parsing for EnhancedUniversal Dependencies](https://md.research.elamur.ai/free/papers/attardi-2021-biaffine-dependency-semantic-graph-parsing/) - [↑ back to papers of 2021](https://md.research.elamur.ai/free/papers/2021/) ## References 1 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Ahmed Abbasi, Hsinchun Chen, and Arab Salem. 2008. Sentiment analysis in multiple languages: Feature selection for opinion classification in web forums. ACM Transactions on Information Systems (TOIS), 26(3):1–34. [find](https://scholar.google.com/scholar?q=Ahmed+Abbasi%2C+Hsinchun+Chen%2C+and+Arab+Salem%2E+2008%2E+Sentiment+analysis+in+multiple+languages%3A+Feature+selection+for+opinion+classification+in+web+forums%2E+ACM+Transactions+on+Information+Systems+%28TOIS%29%2C)