---
title: "A Survey of Large Language Models for Arabic Language and its"
description: "Arabic is hard for language modeling for three reasons: its formal written varieties (Classical Arabic… 80 references, cited by 14 papers here."
authors: ["Malak Mashaabi", "Shahad Al-Khalifa", "Hend Al-Khalifa"]
year: 2024
venue: "arXiv:2410.20238"
source: "https://arxiv.org/abs/2410.20238"
doi: "10.1145/3807946"
arxiv: "2410.20238"
licence: "CC BY-NC-SA 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-nc-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mashaabi-2024-survey-large-language-models-arabic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mashaabi-2024-survey-large-language-models-arabic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/) / A Survey of Large Language Models for Arabic Language and its Dialects

# A Survey of Large Language Models for Arabic Language and its Dialects

**Authors:** Malak Mashaabi, [Shahad Al-Khalifa](https://md.research.elamur.ai/free/en/authors/shahad-al-khalifa/) and [Hend Al-Khalifa](https://md.research.elamur.ai/free/en/authors/hend-al-khalifa/) · **Year:** 2024 · **Venue:** [arXiv:2410.20238](https://md.research.elamur.ai/free/en/venues/acm-transactions-on-asian-and-low-resource-language-information-processing/) ([ACM Transactions on Asian and Low-Resource Language Information Processing 2024](https://md.research.elamur.ai/free/en/venues/acm-transactions-on-asian-and-low-resource-language-information-processing/2024/)) · **Publisher:** [Association for Computing Machinery](https://md.research.elamur.ai/free/en/publishers/association-for-computing-machinery/) · **Licence:** [CC BY-NC-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-nc-sa/4.0/)

**Links:** [Publisher's page](https://doi.org/10.1145/3807946) · [PDF](https://arxiv.org/pdf/2410.20238) · [DOI 10.1145/3807946](https://doi.org/10.1145/3807946) · [arXiv 2410.20238](https://arxiv.org/abs/2410.20238) · [OpenAlex](https://openalex.org/W4404313857)

**Citations:** 10 citations (OpenAlex, 2026-10-14) · [14](https://md.research.elamur.ai/free/en/papers/mashaabi-2024-survey-large-language-models-arabic/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Arabic is hard for language modeling for three reasons: its formal written varieties (Classical Arabic and Modern Standard Arabic) diverge sharply from spoken dialects, dialects vary widely across regions, and the morphology is rich. This survey maps how large language models have been built for Arabic and its dialects. The authors searched Google Scholar, Semantic Scholar and the citation tracker Litmaps up to February 2025, verified relevance from abstracts and conclusions, and traced related-work citations. This reduced an initial set of 80 papers to 34. They group the models by architecture (encoder-only, decoder-only, encoder-decoder), by language scope (monolingual, bilingual, multilingual), and by the Arabic variety in their pretraining data (CA, MSA, DA, or mixtures). They also score each model's openness across code, data, weights, license, documentation and access. MSA data is plentiful, while CA and most dialects are thin. Documentation, especially model cards, is often missing. The paper calls for more dialect data, Arabic-aware tokenization and objectives, and shared benchmarks.

## Key points
- The inclusion rule required a published technical paper. Models known only from HuggingFace or news, such as Noor, Noon, SILMA, Mulhem, Huawei's Arabic LLM and Mistral Saba, were left out.
- Architectures covered include:
  - BERT-style models: AraBERT, MARBERT, QARiB.
  - ELECTRA-style models: AraELECTRA.
  - RoBERTa-style models: AraRoBERTa.
  - GPT-style models: AraGPT2, Jais, AceGPT, ALLaM.
  - T5-style models: AraT5.
- In the openness audit, only 12 of 36 models had fully documented model cards. ARBERT, QARiB and SudaBERT omit details such as vocabulary and context size.
- Seven models provide no form of access whatsoever: AraMUS, JASMINE, SABER, AraQA, AraQA-BERT, AraLegal-BERT and SudaBERT. Most other models can be reached through HuggingFace or GitHub.
- The downstream tasks surveyed include sentiment analysis, NER, question answering and machine translation. The authors note a lack of standardized Arabic evaluation benchmarks.

## Interesting ideas
- The survey scores each model on a checklist of separate openness components (code, training data, weights, RL data and weights, license, architecture docs, paper, model card, API, package) rather than a single open/closed label, which exposes exactly where reproducibility breaks.
- Pretraining corpora are classified by language variety (classical, standard, dialectal, mixed), which makes coverage gaps visible instead of hiding them in one token count.
- ELECTRA-style replaced token detection has a discriminator judge every token as original or swapped by a small generator, so each position yields a training signal.
- The authors stress that morphologically rich languages may need their own tokenization and pretraining objectives rather than generic subword schemes.
- Models were counted only if a technical paper documented them, which filters out unverifiable claims.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Each witness parser and LLM teacher tier could carry such a checklist, so silver data is traceable to components whose weights, data and license are actually known.
- **I2** (tree): Tagging treebanks and converted UD dialect data by register or genre would show which English varieties the perceptron parser and gparse are actually trained on.
- **I3** (learn): Plausible word swaps could serve as negative controls, checking whether the tree, frames or reader notice that the meaning changed.
- **I4** (tokens): This supports keeping the en crate's morphology-aware lemmatizer and tagger explicit and rule-checkable, instead of relying on opaque subword splits.
- **I5** (learn): A similar admission rule for external teachers or knowledge sources would keep every learned rule tied to documented provenance.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [King Saud University](https://md.research.elamur.ai/free/en/universities/king-saud-university/) — University, Riyadh, Saudi Arabia · 37 works · 527 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works

## Cite

Cite as: Malak Mashaabi, Shahad Al-Khalifa and Hend Al-Khalifa. 2024. A Survey of Large Language Models for Arabic Language and its Dialects. arXiv:2410.20238.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{mashaabi2024survey,
  title = {{A Survey of Large Language Models for Arabic Language and its Dialects}},
  author = {Malak Mashaabi, Shahad Al-Khalifa and Hend Al-Khalifa},
  year = {2024},
  doi = {10.1145/3807946},
  eprint = {2410.20238},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1145/3807946}
}
```

- [← previous TILP: Differentiable Learning of Temporal Logical Rules on Knowledge Graphs](https://md.research.elamur.ai/free/en/papers/xiong-2024-tilp-differentiable-learning-temporal-logical/)
- [next → Assessing the Capabilities of Large Language Models in Coreference: An Evaluation](https://md.research.elamur.ai/free/en/papers/gan-2024-assessing-capabilities-large-language-models/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/)

## References

80 references, 46 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. O. ElJundi, W. Antoun, N. El Droubi, H. Hajj, W. El-Hajj, and K. Shaban, “[hULMonA: The Universal Language Model in Arabic](https://md.research.elamur.ai/free/en/papers/eljundi-2019-hulmona-universal-language-model-arabic/),” in Proceedings of the Fourth Arabic Natural Language Processing Workshop, W. El-Hajj, L. H. Belguith, F. Bougares, W. Magdy, I. Zitouni, N. Tomeh, M. El-Haj, and W. Zaghouani, Eds., Florence, Italy: Association for Computational Linguistics, Aug. 2019, pp. 68–77. doi: 10.18653/v1/W19-4608. [↗](https://doi.org/10.18653/v1/w19-4608)
2. W. Antoun, F. Baly, and H. Hajj, “[AraBERT: Transformer-based Model for Arabic Language Understanding](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/),” in Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, H. Al-Khalifa, W. Magdy, K. Darwish, T. Elsayed, and H. Mubarak, Eds., Marseille, France: European Language Resource Association, May 2020, pp. 9–15. Accessed: May 09, 2024. [Online]. Available: https://aclanthology.org/2020.osact-1.2 [↗](https://aclanthology.org/2020.osact-1.2/)
3. S. A. Chowdhury, A. Abdelali, K. Darwish, J. Soon-Gyo, J. Salminen, and B. J. Jansen, “Qarib,” in Proceedings of the Fifth Arabic Natural Language Processing Workshop, I. Zitouni, M. Abdul-Mageed, H. Bouamor, F. Bougares, M. El-Haj, N. Tomeh, and W. Zaghouani, Eds., Barcelona, Spain (Online): Association for Computational Linguistics, Dec. 2020, pp. 226–236. Accessed: Aug. 28, 2024. [Online]. Available: https://aclanthology.org/2020.wanlp-1.21 — [Improving Arabic Text Categorization Using Transformer Training Diversification](https://md.research.elamur.ai/free/en/papers/chowdhury-2020-improving-arabic-text-categorization-transformer/) [↗](https://aclanthology.org/2020.wanlp-1.21/)
4. M. Abdul-Mageed, A. Elmadany, and E. M. B. Nagoudi, “[ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic](https://md.research.elamur.ai/free/en/papers/abdul-mageed-2021-arbert-marbert-deep-bidirectional-transformers/),” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), C. Zong, F. Xia, W. Li, and R. Navigli, Eds., Online: Association for Computational Linguistics, Aug. 2021, pp. 7088–7105. doi: 10.18653/v1/2021.acl-long.551. [↗](https://doi.org/10.18653/v1/2021.acl-long.551)
5. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “[BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/),” May 24, 2019, arXiv: arXiv:1810.04805. doi: 10.48550/arXiv.1810.04805. [↗](https://doi.org/10.48550/arxiv.1810.04805)
6. K. Clark, M.-T. Luong, Q. V. Le, and C. D. Manning, “ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators,” Mar. 23, 2020, arXiv: arXiv:2003.10555. doi: 10.48550/arXiv.2003.10555. [↗](https://doi.org/10.48550/arxiv.2003.10555)
7. Y. Liu et al., “[RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/),” Jul. 26, 2019, arXiv: arXiv:1907.11692. Accessed: Sep. 16, 2024. [Online]. Available: http://arxiv.org/abs/1907.11692 [↗](https://arxiv.org/abs/1907.11692)
8. C. Raffel et al., “[Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/),” Sep. 19, 2023, arXiv: arXiv:1910.10683. doi: 10.48550/arXiv.1910.10683. [↗](https://doi.org/10.48550/arxiv.1910.10683)
9. “NOOR, the new largest NLP model for the Arabic language | MultiLingual.” Accessed: Oct. 07, 2024. [Online]. Available: [https://multilingual.com/noor-the-new-largest-nlp-model-for-the-arabic-language/](https://multilingual.com/noor-the-new-largest-nlp-model-for-the-arabic-language/) [↗](https://multilingual.com/noor-the-new-largest-nlp-model-for-the-arabic-language/)
10. H. Ramadan, “Naseej Launches its Innovative Arabic AI Language Model ‘Noon’ as an open-source initiative,” Naseej For Technology. Accessed: Oct. 07, 2024. [Online]. Available: [https://naseej.com/news/2023/06/naseej-launches-its-innovative-arabic-ai-language-model-noon-as-an-open-source-initiative/](https://naseej.com/news/2023/06/naseej-launches-its-innovative-arabic-ai-language-model-noon-as-an-open-source-initiative/) [↗](https://naseej.com/news/2023/06/naseej-launches-its-innovative-arabic-ai-language-model-noon-as-an-open-source-initiative/)
11. “silma-ai/SILMA-9B-Instruct-v1.0 · Hugging Face.” Accessed: Oct. 07, 2024. [Online]. Available: [https://huggingface.co/silma-ai/SILMA-9B-Instruct-v1.0](https://huggingface.co/silma-ai/SILMA-9B-Instruct-v1.0) [↗](https://huggingface.co/silma-ai/SILMA-9B-Instruct-v1.0)
12. “Watad.” Accessed: Aug. 30, 2024. [Online]. Available: [https://watadenergy.com/mulhem/](https://watadenergy.com/mulhem/) [↗](https://watadenergy.com/mulhem/)
13. W. Antoun, F. Baly, and H. Hajj, “[AraELECTRA: Pre-Training Text Discriminators for Arabic Language Understanding](https://md.research.elamur.ai/free/en/papers/antoun-2021-araelectra-pre-training-text-discriminators/),” in Proceedings of the Sixth Arabic Natural Language Processing Workshop, N. Habash, H. Bouamor, H. Hajj, W. Magdy, W. Zaghouani, F. Bougares, N. Tomeh, I. Abu Farha, and S. Touileb, Eds., Kyiv, Ukraine (Virtual): Association for Computational Linguistics, Apr. 2021, pp. 191–195. Accessed: Aug. 27, 2024. [Online]. Available: https://aclanthology.org/2021.wanlp-1.20 [↗](https://aclanthology.org/2021.wanlp-1.20/)
14. R. AlYami and R. Al-Zaidy, “araRoperta,” in Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), H. Bouamor, H. Al-Khalifa, K. Darwish, O. Rambow, F. Bougares, A. Abdelali, N. Tomeh, S. Khalifa, and W. Zaghouani, Eds., Abu Dhabi, United Arab Emirates (Hybrid): Association for Computational Linguistics, Dec. 2022, pp. 260–272. doi: 10.18653/v1/2022.wanlp-1.24. — [Weakly and Semi-Supervised Learning for Arabic Text Classification using Monodialectal Language Models](https://md.research.elamur.ai/free/en/papers/alyami-2022-weakly-semi-supervised-learning-arabic/) [↗](https://doi.org/10.18653/v1/2022.wanlp-1.24)
15. W. Lan, Y. Chen, W. Xu, and A. Ritter, “gigabert,” in Proceedings of the 2020 [Conference on Empirical Methods in Natural Language Processing (EMNLP), B](https://md.research.elamur.ai/free/en/papers/lan-2020-empirical-study-pre-trained-transformers/). Webber, T. Cohn, Y. He, and Y. Liu, Eds., Online: Association for Computational Linguistics, Nov. 2020, pp. 4727–4734. doi: 10.18653/v1/2020.emnlp-main.382. [↗](https://doi.org/10.18653/v1/2020.emnlp-main.382)
16. A. Safaya, M. Abdullatif, and D. Yuret, “[KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Media](https://md.research.elamur.ai/free/en/papers/safaya-2020-kuisail-semeval-2020-task-12/),” in Proceedings of the Fourteenth Workshop on Semantic Evaluation, A. Herbelot, X. Zhu, A. Palmer, N. Schneider, J. May, and E. Shutova, Eds., Barcelona (online): International Committee for Computational Linguistics, Dec. 2020, pp. 2054–2059. doi: 10.18653/v1/2020.semeval-1.271. [↗](https://doi.org/10.18653/v1/2020.semeval-1.271)
17. G. Inoue, B. Alhafni, N. Baimukan, H. Bouamor, and N. Habash, “camelbert,” in Proceedings of the Sixth Arabic Natural Language Processing Workshop, N. Habash, H. Bouamor, H. Hajj, W. Magdy, W. Zaghouani, F. Bougares, N. Tomeh, I. Abu Farha, and S. Touileb, Eds., Kyiv, Ukraine (Virtual): Association for Computational Linguistics, Apr. 2021, pp. 92–104. Accessed: Oct. 22, 2024. [Online]. Available: https://aclanthology.org/2021.wanlp-1.10 [↗](https://aclanthology.org/2021.wanlp-1.10/)
18. M. Elgezouli, K. N. Elmadani, and M. Saeed, “SudaBERT: A Pre-trained Encoder Representation For Sudanese Arabic Dialect,” in 2020 International Conference on Computer, Control, Electrical, and Electronics Engineering (ICCCEEE), Feb. 2021, pp. 1–4. doi: 10.1109/ICCCEEE49695.2021.9429651. [↗](https://doi.org/10.1109/iccceee49695.2021.9429651)
19. M. Al-qurishi, S. Alqaseemi, and R. Souissi, “[AraLegal-BERT: A pretrained language model for Arabic Legal text](https://md.research.elamur.ai/free/en/papers/qurishi-2022-aralegal-bert-pretrained-language-model/),” in Proceedings of the Natural Legal Language Processing Workshop 2022, N. Aletras, I. Chalkidis, L. Barrett, C. Goan\textcommabelowtă, and D. Preo\textcommabelowtiuc-Pietro, Eds., Abu Dhabi, United Arab Emirates (Hybrid): Association for Computational Linguistics, Dec. 2022, pp. 338– 344. doi: 10.18653/v1/2022.nllp-1.31. [↗](https://doi.org/10.18653/v1/2022.nllp-1.31)
20. A. Ghaddar et al., “jaber and saber,” in Proceedings of the 2022 [Conference on Empirical Methods in Natural Language Processing, Y](https://md.research.elamur.ai/free/en/papers/ghaddar-2022-revisiting-pre-trained-language-models/). Goldberg, Z. Kozareva, and Y. Zhang, Eds., Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, Dec. 2022, pp. 3135–3151. doi: 10.18653/v1/2022.emnlp-main.205. [↗](https://doi.org/10.18653/v1/2022.emnlp-main.205)
21. A. Abdaoui, M. Berrimi, M. Oussalah, and A. Moussaoui, “[DziriBERT: a Pre-trained Language Model for the Algerian Dialect](https://md.research.elamur.ai/free/en/papers/abdaoui-2021-dziribert-pre-trained-language-model/),” Dec. 12, 2022, arXiv: arXiv:2109.12346. doi: 10.48550/arXiv.2109.12346. [↗](https://doi.org/10.48550/arxiv.2109.12346)
22. M. Kamal Eddine, N. Tomeh, N. Habash, J. Le Roux, and M. Vazirgiannis, “[AraBART: a Pretrained Arabic Sequence-to-Sequence Model for Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/eddine-2022-arabart-pretrained-arabic-sequence-sequence/),” in Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), H. Bouamor, H. Al-Khalifa, K. Darwish, O. Rambow, F. Bougares, A. Abdelali, N. Tomeh, S. Khalifa, and W. Zaghouani, Eds., Abu Dhabi, United Arab Emirates (Hybrid): Oct. 23, 2024. [Online]. Available: Association for Computational Linguistics, Dec. 2022, pp. 31–42. doi: 10.18653/v1/2022.wanlp-1.4. [↗](https://doi.org/10.18653/v1/2022.wanlp-1.4)
23. H. Haddad et al., “TunBERT: Pretraining BERT for Tunisian Dialect Understanding,” SN Comput. Sci., vol. 4, no. 2, p. 194, Feb. 2023, doi: 10.1007/s42979-022-01541-y. [↗](https://doi.org/10.1007/s42979-022-01541-y)
24. O. Moussaoui and Y. El Younoussi, “Pre-training Two BERT-Like Models for Moroccan Dialect: MorRoBERTa and MorrBERT,” MENDEL, vol. 29, pp. 55–61, Jun. 2023, doi: 10.13164/mendel.2023.1.055. [↗](https://doi.org/10.13164/mendel.2023.1.055)
25. F. Qarah, “[SaudiBERT: A Large Language Model Pretrained on Saudi Dialect Corpora](https://md.research.elamur.ai/free/en/papers/qarah-2024-saudibert-large-language-model-pretrained/),” May 10, 2024, arXiv: arXiv:2405.06239. doi: 10.48550/arXiv.2405.06239. [↗](https://doi.org/10.48550/arxiv.2405.06239)
26. M. Ahmed, S. Alfasly, B. Wen, J. Addeen, M. Ahmed, and Y. Liu, “[AlclaM: Arabic Dialect Language Model](https://md.research.elamur.ai/free/en/papers/ahmed-2024-alclam-arabic-dialect-language-model/),” in Proceedings of The Second Arabic Natural Language Processing Conference, N. Habash, H. Bouamor, R. Eskander, N. Tomeh, I. Abu Farha, A. Abdelali, S. Touileb, I. Hamed, Y. Onaizan, B. Alhafni, W. Antoun, S. Khalifa, H. Haddad, I. Zitouni, B. AlKhamissi, R. Almatham, and K. Mrini, Eds., Bangkok, Thailand: Association for Computational Linguistics, Aug. 2024, pp. 153–159. doi: 10.18653/v1/2024.arabicnlp-1.14. [↗](https://doi.org/10.18653/v1/2024.arabicnlp-1.14)
27. F. Qarah, “[EgyBERT: A Large Language Model Pretrained on Egyptian Dialect Corpora](https://md.research.elamur.ai/free/en/papers/qarah-2024-egybert-large-language-model-pretrained/),” Aug. 06, 2024, arXiv: arXiv:2408.03524. doi: 10.48550/arXiv.2408.03524. [↗](https://doi.org/10.48550/arxiv.2408.03524)
28. F. Qarah, “[AraPoemBERT: A Pretrained Language Model for Arabic Poetry Analysis](https://md.research.elamur.ai/free/en/papers/qarah-2024-arapoembert-pretrained-language-model-arabic/),” Mar. 18, 2024, arXiv: arXiv:2403.12392. doi: 10.48550/arXiv.2403.12392. [↗](https://doi.org/10.48550/arxiv.2403.12392)
29. K. Gaanoun, A. M. Naira, A. Allak, and I. Benelallam, “DarijaBERT: a step forward in NLP for the written Moroccan dialect,” Int. J. Data Sci. Anal., Jan. 2024, doi: 10.1007/s41060-023-00498-2. [↗](https://doi.org/10.1007/s41060-023-00498-2)
30. A. Dasari, “Understanding Encoder, Decoder, and Autoregressive Models in AI,” Medium. Accessed: Sep. 17, 2024. [Online]. Available: [https://medium.com/@anusaid/understanding-encoder-decoder-and-autoregressive-models-in-ai-8da6ce9d4901](https://medium.com/@anusaid/understanding-encoder-decoder-and-autoregressive-models-in-ai-8da6ce9d4901) [↗](https://medium.com/@anusaid/understanding-encoder-decoder-and-autoregressive-models-in-ai-8da6ce9d4901)
31. P. P. Ray, “ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope,” Internet Things Cyber-Phys. Syst., vol. 3, pp. 121–154, Jan. 2023, doi: 10.1016/j.iotcps.2023.04.003. [↗](https://doi.org/10.1016/j.iotcps.2023.04.003)
32. W. Antoun, F. Baly, and H. Hajj, “[AraGPT2: Pre-Trained Transformer for Arabic Language Generation](https://md.research.elamur.ai/free/en/papers/antoun-2021-aragpt2-pre-trained-transformer-arabic/),” in Proceedings of the Sixth Arabic Natural Language Processing Workshop, N. Habash, H. Bouamor, H. Hajj, W. Magdy, W. Zaghouani, F. Bougares, N. Tomeh, I. Abu Farha, and S. Touileb, Eds., Kyiv, Ukraine (Virtual): Association for Computational Linguistics, Apr. 2021, pp. 196–207. Accessed: Oct. 22, 2024. [Online]. Available: https://aclanthology.org/2021.wanlp-1.21 [↗](https://aclanthology.org/2021.wanlp-1.21/)
33. E. M. Billah Nagoudi, M. Abdul-Mageed, A. Elmadany, A. Inciarte, and M. T. Islam Khondaker, “[JASMINE: Arabic GPT Models for Few-Shot Learning](https://md.research.elamur.ai/free/en/papers/nagoudi-2023-jasmine-arabic-gpt-models-few/),” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali, Eds., Singapore: Association for Computational Linguistics, Dec. 2023, pp. 16721–16744. doi: 10.18653/v1/2023.emnlp-main.1040. [↗](https://doi.org/10.18653/v1/2023.emnlp-main.1040)
34. N. Sengupta et al., “[Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models](https://md.research.elamur.ai/free/en/papers/sengupta-2023-jais-jais-chat-arabic-centric/),” Sep. 29, 2023, arXiv: arXiv:2308.16149. doi: 10.48550/arXiv.2308.16149. [↗](https://doi.org/10.48550/arxiv.2308.16149)
35. H. Touvron et al., “[LLaMA: Open and Efficient Foundation Language Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/),” Feb. 27, 2023, arXiv: arXiv:2302.13971. doi: 10.48550/arXiv.2302.13971. [↗](https://doi.org/10.48550/arxiv.2302.13971)
36. H. Touvron et al., “[Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/),” Jul. 19, 2023, arXiv: arXiv:2307.09288. doi: 10.48550/arXiv.2307.09288. [↗](https://doi.org/10.48550/arxiv.2307.09288)
37. H. Huang et al., “[AceGPT, Localizing Large Language Models in Arabic](https://md.research.elamur.ai/free/en/papers/huang-2024-acegpt-localizing-large-language-models/),” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), K. Duh, H. Gomez, and S. Bethard, Eds., Mexico City, Mexico: Association for Computational Linguistics, Jun. 2024, pp. 8139–8163. doi: 10.18653/v1/2024.naacl-long.450. [↗](https://doi.org/10.18653/v1/2024.naacl-long.450)
38. D. Groeneveld et al., “OLMo: Accelerating the Science of Language Models,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L.-W. Ku, A. Martins, and V. Srikumar, Eds., Bangkok, Thailand: Association for Computational Linguistics, Aug. 2024, pp. 15789–15809. doi: 10.18653/v1/2024.acl-long.841. [↗](https://doi.org/10.18653/v1/2024.acl-long.841)
39. F. Team et al., “[Fanar: An Arabic-Centric Multimodal Generative AI Platform](https://md.research.elamur.ai/free/en/papers/abbas-2025-fanar-arabic-centric-multimodal-generative/),” Jan. 18, 2025, arXiv: arXiv:2501.13944. doi: 10.48550/arXiv.2501.13944. [↗](https://doi.org/10.48550/arxiv.2501.13944)
40. G. Team et al., “Gemma 2: Improving Open Language Models at a Practical Size,” Oct. 02, 2024, arXiv: arXiv:2408.00118. Accessed: Oct. 15, 2024. [Online]. Available: http://arxiv.org/abs/2408.00118 [↗](https://arxiv.org/abs/2408.00118)
41. G. Shang et al., “[Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan Arabic Dialect](https://md.research.elamur.ai/free/en/papers/shang-2025-atlas-chat-adapting-large-language/),” Sep. 26, 2024, arXiv: arXiv:2409.17912. Accessed: Oct. 02, 2024. [Online]. Available: http://arxiv.org/abs/2409.17912 [↗](https://arxiv.org/abs/2409.17912)
42. “[Stable LM 2 1.6B Technical Report](https://md.research.elamur.ai/free/en/papers/bellagente-2024-stable-lm-2-1-6b/).” Accessed: Feb. 16, 2025. [Online]. Available: [https://arxiv.org/html/2402.17834v1](https://arxiv.org/html/2402.17834v1) [↗](https://arxiv.org/html/2402.17834v1)
43. Z. Alyafeai et al., “[Arabic Stable LM: Adapting Stable LM 2 1.6B to Arabic](https://md.research.elamur.ai/free/en/papers/alyafeai-2024-arabic-stable-lm-adapting-stable/),” Dec. 05, 2024, arXiv: arXiv:2412.04277. doi: 10.48550/arXiv.2412.04277. [↗](https://doi.org/10.48550/arxiv.2412.04277)
44. A. Alghamdi et al., “[AraMUS: Pushing the Limits of Data and Model Scale for Arabic Natural Language Processing](https://md.research.elamur.ai/free/en/papers/alghamdi-2023-aramus-pushing-limits-data-model/),” in Findings of the Association for Computational Linguistics: ACL 2023, A. Rogers, J. Boyd-Graber, and N. Okazaki, Eds., Toronto, Canada: Association for Computational Linguistics, Jul. 2023, pp. 2883–2894. doi: 10.18653/v1/2023.findings-acl.181. [↗](https://doi.org/10.18653/v1/2023.findings-acl.181)
45. E. M. B. Nagoudi, A. Elmadany, and M. Abdul-Mageed, “[AraT5: Text-to-Text Transformers for Arabic Language Generation](https://md.research.elamur.ai/free/en/papers/nagoudi-2022-arat5-text-text-transformers-arabic/),” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), S. Muresan, P. Nakov, and A. Villavicencio, Eds., Dublin, Ireland: Association for Computational Linguistics, May 2022, pp. 628–647. doi: 10.18653/v1/2022.acl-long.47. [↗](https://doi.org/10.18653/v1/2022.acl-long.47)
46. Y. Adel et al., “AraQA: An Arabic Generative Question-Answering Model for Authentic Religious Text,” in 2023 11th International Japan-Africa Conference on Electronics, Communications, and Computations (JAC-ECC), Dec. 2023, pp. 235–239. doi: 10.1109/JAC-ECC61002.2023.10479645. [↗](https://doi.org/10.1109/jac-ecc61002.2023.10479645)
47. A. Koubaa, A. Ammar, L. Ghouti, O. Najar, and S. Sibaee, “[ArabianGPT: Native Arabic GPT-based Large Language Model](https://md.research.elamur.ai/free/en/papers/koubaa-2024-arabiangpt-native-arabic-gpt-based/),” Feb. 26, 2024, arXiv: arXiv:2402.15313. doi: 10.48550/arXiv.2402.15313. [↗](https://doi.org/10.48550/arxiv.2402.15313)
48. A. El-Shangiti, F. Alwajih, and M. Abdul-Mageed, “[Arabic Automatic Story Generation with Large Language Models](https://md.research.elamur.ai/free/en/papers/shangiti-2024-arabic-automatic-story-generation-large/),” in Proceedings of The Second Arabic Natural Language Processing Conference, N. Habash, H. Bouamor, R. Eskander, N. Tomeh, I. Abu Farha, A. Abdelali, S. Touileb, I. Hamed, Y. Onaizan, B. Alhafni, W. Antoun, S. Khalifa, H. Haddad, I. Zitouni, B. AlKhamissi, R. Almatham, and K. Mrini, Eds., Bangkok, Thailand: Association for Computational Linguistics, Aug. 2024, pp. 140–152. Accessed: Sep. 06, 2024. [Online]. Available: https://aclanthology.org/2024.arabicnlp-1.13 [↗](https://aclanthology.org/2024.arabicnlp-1.13/)
49. M. S. Bari et al., “[ALLaM: Large Language Models for Arabic and English](https://md.research.elamur.ai/free/en/papers/bari-2024-allam-large-language-models-arabic/),” Jul. 22, 2024, arXiv: arXiv:2407.15390. Accessed: Aug. 28, 2024. [Online]. Available: http://arxiv.org/abs/2407.15390 [↗](https://arxiv.org/abs/2407.15390)
50. J. Zhu et al., “[Second Language (Arabic) Acquisition of LLMs via Progressive Vocabulary Expansion](https://md.research.elamur.ai/free/en/papers/zhu-2025-second-language-arabic-acquisition-llms/),” Dec. 16, 2024, arXiv: arXiv:2412.12310. doi: 10.48550/arXiv.2412.12310. [↗](https://doi.org/10.48550/arxiv.2412.12310)
51. “Modern Standard Arabic,” Wikipedia. Aug. 15, 2024. Accessed: [https://en.wikipedia.org/w/index.php?title=Modern_Standard_Arabic&oldid=1240502833](https://en.wikipedia.org/w/index.php?title=Modern_Standard_Arabic&oldid=1240502833) [↗](https://en.wikipedia.org/w/index.php?title=Modern_Standard_Arabic&oldid=1240502833)
52. W. Zaghouani and A. Charfi, “Arap-Tweet: A Large Multi-Dialect Twitter Corpus for Gender, Age and Language Variety Identification,” in Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), N. Calzolari, K. Choukri, C. Cieri, T. Declerck, S. Goggi, K. Hasida, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, S. Piperidis, and T. Tokunaga, Eds., Miyazaki, Japan: European Language Resources Association (ELRA), May 2018. Accessed: Sep. 16, 2024. [Online]. Available: https://aclanthology.org/L18-1111 [↗](https://aclanthology.org/L18-1111/)
53. A. Conneau et al., “Unsupervised Cross-lingual Representation Learning at Scale,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online: Association for Computational Linguistics, 2020. doi: 10.18653/v1/2020.acl-main.747. [↗](https://doi.org/10.18653/v1/2020.acl-main.747)
54. G. Inoue, B. Alhafni, N. Baimukan, H. Bouamor, and N. Habash, “[The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models](https://md.research.elamur.ai/free/en/papers/inoue-2021-interplay-variant-size-task-type/),” Sep. 04, 2021, arXiv: arXiv:2103.06678. doi: 10.48550/arXiv.2103.06678. [↗](https://doi.org/10.48550/arxiv.2103.06678)
55. B. Talafha et al., “[Multi-dialect Arabic BERT for Country-level Dialect Identification](https://md.research.elamur.ai/free/en/papers/talafha-2020-multi-dialect-arabic-bert-country/),” in Proceedings of the Fifth Arabic Natural Language Processing Workshop, I. Zitouni, M. Abdul-Mageed, H. Bouamor, F. Bougares, M. El-Haj, N. Tomeh, and W. Zaghouani, Eds., Barcelona, Spain (Online): Association for Computational Linguistics, Dec. 2020, pp. 111–118. Accessed: Sep. 20, 2024. [Online]. Available: https://aclanthology.org/2020.wanlp-1.10 [↗](https://aclanthology.org/2020.wanlp-1.10/)
56. Y. Liu et al., “[RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/),” Jul. 26, 2019, arXiv: arXiv:1907.11692. doi: 10.48550/arXiv.1907.11692. [↗](https://doi.org/10.48550/arxiv.1907.11692)
57. F. Feng, Y. Yang, D. Cer, N. Arivazhagan, and W. Wang, “[Language-agnostic BERT Sentence Embedding](https://md.research.elamur.ai/free/en/papers/feng-2020-language-agnostic-bert-sentence-embedding/),” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), S. Muresan, P. Nakov, and A. Villavicencio, Eds., Dublin, Ireland: Association for Computational Linguistics, May 2022, pp. 878–891. doi: 10.18653/v1/2022.acl-long.62. [↗](https://doi.org/10.18653/v1/2022.acl-long.62)
58. T. Kudo and J. Richardson, “[SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/),” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, E. Blanco and W. Lu, Eds., Brussels, Belgium: Association for Computational Linguistics, Nov. 2018, pp. 66–71. doi: 10.18653/v1/D18-2012. [↗](https://doi.org/10.18653/v1/d18-2012)
59. R. Zellers et al., “[Defending against neural fake news](https://md.research.elamur.ai/free/en/papers/zellers-2019-defending-against-neural-fake-news/),” in Proceedings of the 33rd International Conference on Neural Information Processing Systems, Red Hook, NY, USA: Curran Associates Inc., 2019, pp. 9054–9065. [↗](https://doi.org/10.48550/arxiv.1905.12616)
60. O. Shliazhko, A. Fenogenova, M. Tikhonova, A. Kozlova, V. Mikhailov, and T. Shavrina, “[mGPT: Few-Shot Learners Go Multilingual](https://md.research.elamur.ai/free/en/papers/shliazhko-2022-mgpt-few-shot-learners-go/),” Trans. Assoc. Comput. Linguist., vol. 12, pp. 58–79, 2024, doi: 10.1162/tacl_a_00633. [↗](https://doi.org/10.1162/tacl_a_00633)
61. S. Rothe, S. Narayan, and A. Severyn, “[Leveraging Pre-trained Checkpoints for Sequence Generation Tasks](https://md.research.elamur.ai/free/en/papers/rothe-2019-leveraging-pre-trained-checkpoints-sequence/),” Trans. Assoc. Comput. Linguist., vol. 8, pp. 264–280, Dec. 2020, doi: 10.1162/tacl_a_00313. [↗](https://doi.org/10.1162/tacl_a_00313)
62. Y. Liu et al., “Multilingual Denoising Pre-training for Neural Machine Translation,” Trans. Assoc. Comput. Linguist., vol. 8, pp. 726–742, Dec. 2020, doi: 10.1162/tacl_a_00343. [↗](https://doi.org/10.1162/tacl_a_00343)
63. L. Xue et al., “[mT5: A massively multilingual pre-trained text-to-text transformer](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/),” Mar. 11, 2021, arXiv: arXiv:2010.11934. Accessed: Oct. 04, 2024. [Online]. Available: http://arxiv.org/abs/2010.11934 [↗](https://arxiv.org/abs/2010.11934)
64. A. Vaswani et al., “[Attention Is All You Need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/),” Aug. 01, 2023, arXiv: arXiv:1706.03762. Accessed: Oct. 04, 2024. [Online]. Available: http://arxiv.org/abs/1706.03762 [↗](https://arxiv.org/abs/1706.03762)
65. A. Üstün et al., “[Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://md.research.elamur.ai/free/en/papers/ustun-2024-aya-model-instruction-finetuned-open/),” Feb. 12, 2024, arXiv: arXiv:2402.07827. Accessed: Oct. 12, 2024. [Online]. Available: http://arxiv.org/abs/2402.07827 [↗](https://arxiv.org/abs/2402.07827)
66. “Command R: RAG at Production Scale,” Cohere. Accessed: Oct. 12, 2024. [Online]. Available: [https://cohere.com/blog/command-r](https://cohere.com/blog/command-r) [↗](https://cohere.com/blog/command-r)
67. “BLOOM.” Accessed: Oct. 12, 2024. [Online]. Available: [https://bigscience.huggingface.co/blog/bloom](https://bigscience.huggingface.co/blog/bloom) [↗](https://bigscience.huggingface.co/blog/bloom)
68. “Qwen (Qwen).” Accessed: Oct. 12, 2024. [Online]. Available: [https://huggingface.co/Qwen](https://huggingface.co/Qwen) [↗](https://huggingface.co/Qwen)
69. “Cohere | The leading AI platform for enterprise.” Accessed: Oct. 12, 2024. [Online]. Available: [https://cohere.com/](https://cohere.com/) [↗](https://cohere.com/)
70. “Alibaba Cloud: Reliable Secure Cloud Solutions to Empower Your Global Business.” Accessed: Oct. 12, 2024. [Online]. Available: [https://www.alibabacloud.com/en?_p_lc=7](https://www.alibabacloud.com/en?_p_lc=7) [↗](https://www.alibabacloud.com/en?_p_lc=7)
71. “Introducing Gemini: our largest and most capable AI model,” Google. Accessed: Oct. 12, 2024. [Online]. Available: [https://blog.google/technology/ai/google-gemini-ai/Aug.29](https://blog.google/technology/ai/google-gemini-ai/Aug.29), 2024. [Online]. Available: [↗](https://blog.google/technology/ai/google-gemini-ai/Aug.29)
72. J.-C. Burgelman et al., “Open Science, Open Data, and Open Scholarship: European Policies to Make Science Fit for the Twenty-First Century,” Front. Big Data, vol. 2, Dec. 2019, doi: 10.3389/fdata.2019.00043. [↗](https://doi.org/10.3389/fdata.2019.00043)
73. A. Birhane, A. Kasirzadeh, D. Leslie, and S. Wachter, “Science in the age of large language models,” Nat. Rev. Phys., vol. 5, no. 5, pp. 277– 280, Apr. 2023, doi: 10.1038/s42254-023-00581-4. [↗](https://doi.org/10.1038/s42254-023-00581-4)
74. D. Gray Widder, S. West, and M. Whittaker, “Open (For Business): Big Tech, Concentrated Power, and the Political Economy of Open AI,” SSRN Electron. J., 2023, doi: 10.2139/ssrn.4543807. [↗](https://doi.org/10.2139/ssrn.4543807)
75. A. Liesenfeld and M. Dingemanse, “Rethinking open source generative AI: open washing and the EU AI Act,” in The 2024 ACM Conference on Fairness, Accountability, and Transparency, Rio de Janeiro Brazil: ACM, Jun. 2024, pp. 1774–1787. doi: 10.1145/3630106.3659005. [↗](https://doi.org/10.1145/3630106.3659005)
76. “EU AI Act: first regulation on artificial intelligence,” Topics | European Parliament. Accessed: Oct. 12, 2024. [Online]. Available: [https://www.europarl.europa.eu/topics/en/article/20230601STO93804/eu-ai-act-first-regulation-on-artificial-intelligence](https://www.europarl.europa.eu/topics/en/article/20230601STO93804/eu-ai-act-first-regulation-on-artificial-intelligence) [↗](https://www.europarl.europa.eu/topics/en/article/20230601STO93804/eu-ai-act-first-regulation-on-artificial-intelligence)
77. M. White, I. Haddad, C. Osborne, X.-Y. L. Yanglet, A. Abdelmonsef, and S. Varghese, “The Model Openness Framework: Promoting Completeness and Openness for Reproducibility, Transparency, and Usability in Artificial Intelligence,” Oct. 02, 2024, arXiv: arXiv:2403.13784. Accessed: Oct. 16, 2024. [Online]. Available: http://arxiv.org/abs/2403.13784 [↗](https://arxiv.org/abs/2403.13784)
78. R. Kitchin, The Data Revolution: Big Data, Open Data, Data Infrastructures and Their Consequences. SAGE, 2014. [find](https://scholar.google.com/scholar?q=R%2E+Kitchin%2C+The+Data+Revolution%3A+Big+Data%2C+Open+Data%2C+Data+Infrastructures+and+Their+Consequences%2E+SAGE%2C+2014%2E)
79. C. Osborne, J. Ding, and H. R. Kirk, “The AI Community Building the Future? A Quantitative Analysis of Development Activity on Hugging Face Hub,” Jun. 05, 2024, arXiv: arXiv:2405.13058. Accessed: Oct. 16, 2024. [Online]. Available: http://arxiv.org/abs/2405.13058 [↗](https://arxiv.org/abs/2405.13058)
80. M. Mitchell et al., “Model Cards for Model Reporting,” Jan. 14, 2019, arXiv: arXiv:1810.03993. Accessed: Oct. 16, 2024. [Online]. Available: http://arxiv.org/abs/1810.03993 [↗](https://arxiv.org/abs/1810.03993)

*Updated 1 November 2026*