---
title: "When Bigger Isn't Better — paper by Ibrahim et al. (2026)"
description: "The authors compare CAMeL-MSA, a CAMeL-BERT model of about 110M parameters fine-tuned for… 86.7% accuracy. 15 references."
authors: ["Mohamed Ibrahim", "Abdullah Makki", "Youssef Barakat", "Nour Samy", "Sarah AlHumoud"]
year: 2026
venue: "The 7th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT7) with 5 Shared Tasks"
source: "https://aclanthology.org/2026.osact-1.4/"
doi: "10.63317/2kimof4u6y8x"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ibrahim-2026-when-bigger-isn-t-better/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ibrahim-2026-when-bigger-isn-t-better/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/56/) / When Bigger Isn't Better: Evaluating LLMs for Arabic Sentiment Analysis

# When Bigger Isn't Better: Evaluating LLMs for Arabic Sentiment Analysis

**Authors:** [Mohamed Ibrahim](https://md.research.elamur.ai/free/en/authors/mohamed-ibrahim/), Abdullah Makki, Youssef Barakat, Nour Samy and [Sarah AlHumoud](https://md.research.elamur.ai/free/en/authors/sarah-alhumoud/) · **Year:** 2026 · **Venue:** [The 7th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT7) with 5 Shared Tasks](https://md.research.elamur.ai/free/en/venues/osact/) ([Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks 2026](https://md.research.elamur.ai/free/en/venues/osact/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.osact-1.4/) · [PDF](https://aclanthology.org/2026.osact-1.4.pdf) · [DOI 10.63317/2kimof4u6y8x](https://doi.org/10.63317/2kimof4u6y8x) · [OpenAlex](https://openalex.org/W7166532752)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
The paper asks whether a small transformer fine-tuned for Arabic sentiment still beats general-purpose LLMs used through zero-shot prompting. The authors compare CAMeL-MSA, a CAMeL-BERT model of about 110M parameters fine-tuned for sentiment, against eight LLMs from the Qwen, BLOOMz, AraGPT2 and Gemma families. The test bed has six corpora under one three-class protocol (positive, negative, neutral), scored with Accuracy and Macro F1:

- three classic benchmarks (LABR, ASTD, ArSAS);
- a small set of Saudi customer-feedback audio, transcribed automatically by their ASR system;
- two synthetic sets generated with GPT-4.1 Nano.

The fine-tuned model wins clearly on five of the six datasets. On ASTD every system does badly. The LLMs mostly stay in a band of roughly 28–46% accuracy, close to chance for balanced three-class data. The authors also document a labeling problem in LABR that hurts reproducibility. For practitioners choosing between prompting a big model and deploying a compact specialized one, the study suggests that size alone does not buy task accuracy.

## Key points
- **Models.** The baseline is CAMeL-MSA (CAMeL-BERT, 110M). The eight LLMs are Qwen2.5-3B, Qwen2.5-7B-Instruct, Qwen3-8B, AraGPT2-medium, BLOOMz-1b7, BLOOMz-3b, BLOOMz-7b1 and Gemma-3-4b-it. All LLMs were run zero-shot.
- **Datasets.** All were balanced through stratified sampling:
  - LABR: 7,056 reviews; stars 1–2 map to negative, 3 to neutral, 4–5 to positive.
  - ASTD: 2,415 tweets; objective and mixed labels map to neutral.
  - ArSAS: 13,200 tweets.
  - Transcribed: 97 ASR samples.
  - Saudi: 300 synthetic samples.
  - Multilingual multi-dialect: 15,000 synthetic samples, including Arabic–English mixing.
- **ArSAS.** CAMeL-MSA scored 86.7% accuracy and 0.866 Macro F1. The strongest LLM, BLOOMz-3b, reached 44.1% and 0.35.
- **LABR.** CAMeL-MSA scored 47% and 0.45, against 36.3% and 0.28 for Qwen2.5-3B.
- **Other sets.** CAMeL-MSA reached 97.2% on Multilingual, 93.3% on Transcribed and 100% on Saudi. On ASTD it fell to 18.4%, below AraGPT2-medium at 34.5%.
- **LABR labeling issue.** Some LABR raters apparently treated 1 star as the best score, so the star-to-class mapping is unreliable. The authors flag this as a reproducibility risk.
- **Caveats from the authors.** The small and synthetic sets may inflate scores and reflect generator bias. LLM results may also depend on prompt and decoding choices.

## Interesting ideas
- Feeding raw, uncorrected ASR transcripts into evaluation turns real transcription errors into a built-in robustness test, instead of testing only on clean text.
- The authors audited the label mapping itself and found that the direction of the star scale was inconsistent between raters, which silently corrupts gold labels.
- On balanced three-class data, many LLM scores sit near 33%, so chance level is the natural yardstick for whether a system learned anything.
- The authors treat tiny or LLM-generated sets as supplementary stress tests, explicitly not as proof of readiness, because perfect scores there may only reflect generator bias.
- The study runs one fixed protocol (same labels, metrics and balancing) across many corpora instead of reporting a single benchmark, which exposes dataset-dependent swings such as the ASTD collapse.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tokens): Add a noisy-input test set (ASR-like or OCR-like text) so the tokenizer, tagger and lemmatizer are measured on degraded input, not only on edited prose.
- **I2** (tree): When converting treebanks between UD dialects or annotation schemes, record every mapping rule and check samples for flipped or ambiguous conventions before trusting converted gold data.
- **I3** (learn): Report a chance or majority baseline next to every metric as a negative control, so near-random components are caught at once.
- **I4** (learn): Data produced by LLM teacher tiers should be scored and reported separately from human-built tests, with near-perfect results treated as a warning sign.
- **I5** (answer): Evaluate readers across several distinct test families (school tests, maths word problems, fables) under one shared scoring protocol, so a gain on one family is not mistaken for general progress.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · 1,035 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 3,066 works

## Cite

Cite as: Mohamed Ibrahim, Abdullah Makki, Youssef Barakat, Nour Samy and Sarah AlHumoud. 2026. When Bigger Isn't Better: Evaluating LLMs for Arabic Sentiment Analysis. The 7th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT7) with 5 Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ibrahim2026when,
  title = {{When Bigger Isn't Better: Evaluating LLMs for Arabic Sentiment Analysis}},
  author = {Mohamed Ibrahim, Abdullah Makki, Youssef Barakat, Nour Samy and Sarah AlHumoud},
  booktitle = {The 7th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT7) with 5 Shared Tasks},
  year = {2026},
  doi = {10.63317/2kimof4u6y8x},
  url = {https://doi.org/10.63317/2kimof4u6y8x}
}
```

- [← previous When Agents Look the Same: Quantifying Distillation-Induced Similarity in Tool-Use Behaviors](https://md.research.elamur.ai/free/en/papers/yang-2026-when-agents-look-same-quantifying/)
- [next → When Correct Beliefs Collapse: Epistemic Resilience of LLMs under Clinical Pressure](https://md.research.elamur.ai/free/en/papers/xiao-2026-when-correct-beliefs-collapse-epistemic/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/56/)

## References

15 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alahmadi, D. (2025). Human Versus AI: A Comparative Study of Zero-Shot LLMs and Transformer Models Against Human Annotations for Arabic Sentiment Analysis. International Journal of Advanced Computer Science and Applications, 16(8). (Match pages/DOI to the published PDF.) [find](https://scholar.google.com/scholar?q=Alahmadi%2C+D%2E+%282025%29%2E+Human+Versus+AI%3A+A+Comparative+Study+of+Zero%2DShot+LLMs+and+Transformer+Models+Against+Human+Annotations+for+Arabic+Sentiment+Analysis%2E+International+Journal+of+Advanced+Computer+S)
2. Alduailej, A., & Alothaim, A. (2022). AraXLNet: Pre-trained language model for sentiment analysis of Arabic. Journal of Big Data, 9(1), 72. https://doi.org/10.1186/s40537-022-00625-z [↗](https://doi.org/10.1186/s40537-022-00625-z)
3. Al-Twairesh, N., Al-Khalifa, H., & Al-Salman, A. (2017). ArSAS: An Arabic Saudi sentiment analysis dataset. In Proceedings of the 24th International Conference on Neural Information Processing (ICONIP). [find](https://scholar.google.com/scholar?q=Al%2DTwairesh%2C+N%2E%2C+Al%2DKhalifa%2C+H%2E%2C+%26+Al%2DSalman%2C+A%2E+%282017%29%2E+ArSAS%3A+An+Arabic+Saudi+sentiment+analysis+dataset%2E+In+Proceedings+of+the+24th+International+Conference+on+Neural+Information+Processing+%28ICONIP)
4. Aly, M., & Atiya, A. (2013). [LABR: A large-scale Arabic book reviews dataset](https://md.research.elamur.ai/free/en/papers/aly-2013-labr-large-scale-arabic-book/). In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) (pp. 494–498). Association for Computational Linguistics. https://aclanthology.org/P13-2088 [↗](https://aclanthology.org/P13-2088/)
5. Antoun, W., Baly, F., & Hajj, H. (2020). [AraBERT: Transformer-based model for Arabic language understanding](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/). In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection (pp. 9–15). European Language Resources Association. https://aclanthology.org/2020.osact-1.2 [↗](https://aclanthology.org/2020.osact-1.2/)
6. Antoun, W., Baly, F., & Hajj, H. (2021). [AraGPT2: Pre-trained transformer for Arabic language generation](https://md.research.elamur.ai/free/en/papers/antoun-2021-aragpt2-pre-trained-transformer-arabic/). In Proceedings of the Sixth Arabic Natural Language Processing Workshop (pp. 196–207). Association for Computational Linguistics. https://aclanthology.org/2021.wanlp-1.21 [↗](https://aclanthology.org/2021.wanlp-1.21/)
7. Gemma Team. (2025). Gemma 3 technical report. arXiv Preprint arXiv:2503.19786. https://arxiv.org/abs/2503.19786 [↗](https://arxiv.org/abs/2503.19786)
8. Hasan, M. A., Das, S., Anjum, A., Alam, F., Anjum, A., Sarker, A., & Noori, S. R. H. (2024). Zero- and few-shot prompting with LLMs: A comparative study with fine-tuned models for Bangla sentiment analysis. In Proceedings of LREC-COLING 2024 (pp. 17808–17818). [find](https://scholar.google.com/scholar?q=Hasan%2C+M%2E+A%2E%2C+Das%2C+S%2E%2C+Anjum%2C+A%2E%2C+Alam%2C+F%2E%2C+Anjum%2C+A%2E%2C+Sarker%2C+A%2E%2C+%26+Noori%2C+S%2E+R%2E+H%2E+%282024%29%2E+Zero%2D+and+few%2Dshot+prompting+with+LLMs%3A+A+comparative+study+with+fine%2Dtuned+models+for+Bangla+sentiment+ana)
9. Inoue, G., Alhafni, B., Baimukan, N., Bouamor, H., & Habash, N. (2021). [The interplay of variant, size, and task type in Arabic pre-trained language models](https://md.research.elamur.ai/free/en/papers/inoue-2021-interplay-variant-size-task-type/). In Proceedings of the Sixth Arabic Natural Language Processing Workshop (pp. 92–104). Association for Computational Linguistics. https://aclanthology.org/2021.wanlp-1.10 Muennighoff, N., Wang, T., Sutawika, L., Roberts, A., Biderman, S., Le Scao, T., Bari, M. S., Shen, S., Yong, Z.-X., Schoelkopf, H., Tang, X., Radev, D., Aji, A. F., Almubarak, K., Albanie, S., Alyafeai, Z., Webson, A., Raff, E., & Raffel, C. [↗](https://aclanthology.org/2021.wanlp-1.10/)
10. (2022). [Crosslingual generalization through multitask finetuning](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-crosslingual-generalization-through-multitask-finetuning/). arXiv Preprint arXiv:2211.01786. https://arxiv.org/abs/2211.01786 [↗](https://arxiv.org/abs/2211.01786)
11. Nabil, M., Aly, M., & Atiya, A. (2015). [ASTD: Arabic sentiment tweets dataset](https://md.research.elamur.ai/free/en/papers/nabil-2015-astd-arabic-sentiment-tweets-dataset/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (pp. 2515–2519). Association for Computational Linguistics. https://aclanthology.org/D15-1299 [↗](https://aclanthology.org/D15-1299/)
12. Polock, S. I. S. (2025). Sentiment analysis for low-resource languages using large language models with zero-shot, N-shot prompting, and LoRA fine-tuning [Master's thesis, SIS Polock]. [https://oulurepo.oulu.fi/handle/10024/56755](https://oulurepo.oulu.fi/handle/10024/56755) [↗](https://oulurepo.oulu.fi/handle/10024/56755)
13. Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., ... Qiu, Z. (2024). Qwen2.5 technical report. arXiv Preprint arXiv:2412.15115. https://arxiv.org/abs/2412.15115 [↗](https://arxiv.org/abs/2412.15115)
14. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., Zheng, C., Liu, D., Zhou, F., Huang, F., Hu, F., Ge, H., Wei, H., Lin, H., Tang, J., ... Qiu, Z. (2025). [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv Preprint arXiv:2505.09388. https://arxiv.org/abs/2505.09388 [↗](https://arxiv.org/abs/2505.09388)
15. Zouidine, M., & Khalil, M. (2025). Large language models for Arabic sentiment analysis and machine translation. Engineering, Technology & Applied Science Research, 15(2), 20737–20742. [find](https://scholar.google.com/scholar?q=Zouidine%2C+M%2E%2C+%26+Khalil%2C+M%2E+%282025%29%2E+Large+language+models+for+Arabic+sentiment+analysis+and+machine+translation%2E+Engineering%2C+Technology+%26+Applied+Science+Research%2C+15%282%29%2C+20737%E2%80%9320742%2E)

*Updated 2 November 2026*