---
title: "EgyBERT: A Large Language Model Pretrained on — Qarah 2024"
description: "Egyptian Arabic is one of the most widely used Arabic varieties."
authors: ["Faisal Qarah"]
year: 2024
venue: "arXiv:2408.03524"
source: "https://arxiv.org/abs/2408.03524"
doi: "10.48550/arxiv.2408.03524"
arxiv: "2408.03524"
licence: "CC BY-NC-SA 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-nc-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/qarah-2024-egybert-large-language-model-pretrained/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/qarah-2024-egybert-large-language-model-pretrained/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / EgyBERT: A Large Language Model Pretrained on Egyptian Dialect Corpora

# EgyBERT: A Large Language Model Pretrained on Egyptian Dialect Corpora

**Authors:** [Faisal Qarah](https://md.research.elamur.ai/free/en/authors/faisal-qarah/) · **Year:** 2024 · **Venue:** arXiv:2408.03524 · **Licence:** [CC BY-NC-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-nc-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2408.03524) · [PDF](https://arxiv.org/pdf/2408.03524) · [DOI 10.48550/arxiv.2408.03524](https://doi.org/10.48550/arxiv.2408.03524) · [arXiv 2408.03524](https://arxiv.org/abs/2408.03524) · [OpenAlex](https://openalex.org/W4403625603)

**Citations:** 4 citations (OpenAlex, 2026-10-14) · 6 in our library, of the 140884 papers whose bibliographies we read

## Summary
Egyptian Arabic is one of the most widely used Arabic varieties. Existing Arabic language models, however, are trained mostly on Modern Standard Arabic (MSA) or on mixtures of many dialects, and earlier Egyptian corpora are too small to pretrain a BERT-style model. Qarah addresses this gap by building two new Egyptian dialect corpora. The Egyptian Tweets Corpus (ETC) was filtered from a large in-house tweet collection using the users' self-reported location field. The Egyptian Forums Corpus (EFC) was collected from Egyptian online forums. Together they supply 10.4 GB of text, which is used to pretrain a single-dialect model, EgyBERT. The model is compared with five multidialect Arabic models on ten Egyptian-dialect classification datasets covering sentiment, emotion, sarcasm, abuse, gender and topic. EgyBERT reaches the best average macro F1 and accuracy, narrowly ahead of MARBERTv2, although its rivals were pretrained on 5–12 times more text. The work suggests that a focused dialect corpus can beat much larger general ones. The model and data are publicly released.

## Key points
- Pretraining data: ETC has 34.33M tweets (24.89M sentences, 2.5 GB), selected from more than 800M Arabic tweets by matching the location field against 52 hand-compiled Arabic and English Egypt-related terms. EFC has 44.42M sentences (7.9 GB) from Egyptian forums. The author reports these as the largest Egyptian dialect corpora to date.
- Baselines: AraBERTv0.2-Twitter, QARiB, CAMeLBERT-DA, MARBERTv1 and MARBERTv2. The ten evaluation tasks draw on ArSAS (used twice, for sentiment and for topic), EmotionalTone, Egyptian Companies Reviews, ArabicEgyptianCorpus2, ArabicEgyptianTweets, EDGAD, the Nile-dialect portion of iSarcasmEval, AbusiveLanguage and SDC-EDC. Scores are reported on validation sets.
- Averages: EgyBERT scores 84.25 macro F1 and 87.33 accuracy, against 83.68 and 87.19 for MARBERTv2. CAMeLBERT-DA is clearly last at 81.60 F1 and 85.09 accuracy.
- EgyBERT has the top F1 on 8 of 10 tasks and the top accuracy on 7. It trails on the ArSAS tasks, whose tweets mix MSA and other dialects with Egyptian.
- Releases: EgyBERT, ETC and a reduced EFC-mini are on Hugging Face (faisalq). The code is on GitHub (FaisalQarah/EgyBERT).

## Interesting ideas
- Mining a big generic pool for one language variety by filtering on cleaned user metadata (location strings in two scripts) produces a large single-variety corpus without manual labelling. — For Mova (tree): The same metadata filtering could collect regional or register-specific English (e.g., Indian or British forum text) to test and tune UD dialect conversion and the perceptron parser on out-of-variety input.
- A small but well-targeted pretraining set beat models trained on several times more mixed data, so matching data to the target domain can matter more than sheer volume. — For Mova (learn): When building silver data agreed by witness parsers, prioritise sentences from the target genres (fables, school tests, word problems) over bulk web text.
- The model's weak spots were explained by checking test-set composition: the dataset where it lost contained a lot of off-target language varieties. — For Mova (learn): Evaluation could be split by register or variety, with off-domain slices used as negative controls, so that drops are traced to data mismatch rather than to model bugs.
- Reporting macro F1 alongside accuracy exposes imbalance problems. On Egyptian Companies Reviews (only 4.8% neutral), accuracy sits around 85 while F1 stays near 66. — For Mova (answer): Test suites with skewed answer classes (e.g., yes/no QA on tales) should report per-class or macro scores so a majority-guessing reader cannot look strong.
- Publishing a reduced "mini" version of a corpus next to the full model lets others reproduce and reuse the work even when the full data cannot be fully released. — For Mova (learn): Silver treebanks or teacher-labelled sets could ship as small public samples with provenance, keeping the training pipeline auditable and explainable.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{qarah2024egybert,
  title = {{EgyBERT: A Large Language Model Pretrained on Egyptian Dialect Corpora}},
  author = {Qarah, Faisal},
  year = {2024},
  doi = {10.48550/arxiv.2408.03524},
  eprint = {2408.03524},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2408.03524}
}
```

- [← previous Challenges to Evaluating the Generalization of Coreference Resolution Models: A Measurement Modeling Perspective](https://md.research.elamur.ai/free/en/papers/porada-2024-challenges-evaluating-generalization-coreference-resolution/)
- [next → How Often Are Errors in Natural Language Reasoning Due to Paraphrastic Variability?](https://md.research.elamur.ai/free/en/papers/srikanth-2024-how-often-are-errors-natural/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

43 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Diab, M., Habash, N., Rambow, O., Altantawy, M., Benajiba, Y.: Colaba: Arabic dialect annotation and processing. In: Lrec Workshop on Semitic Language Processing, pp. 66–74 (2010). Citeseer [find](https://scholar.google.com/scholar?q=Diab%2C+M%2E%2C+Habash%2C+N%2E%2C+Rambow%2C+O%2E%2C+Altantawy%2C+M%2E%2C+Benajiba%2C+Y%2E%3A+Colaba%3A+Arabic+dialect+annotation+and+processing%2E+In%3A+Lrec+Workshop+on+Semitic+Language+Processing%2C+pp%2E+66%E2%80%9374+%282010%29%2E+Citeseer)
2. Zaidan, O., Callison-Burch, C.: The arabic online commentary dataset: an annotated dataset of informal arabic with high dialectal content. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 37–41 (2011) [find](https://scholar.google.com/scholar?q=Zaidan%2C+O%2E%2C+Callison%2DBurch%2C+C%2E%3A+The+arabic+online+commentary+dataset%3A+an+annotated+dataset+of+informal+arabic+with+high+dialectal+content%2E+In%3A+Proceedings+of+the+49th+Annual+Meeting+of+the+Association) [our card](https://md.research.elamur.ai/free/en/papers/zaidan-2011-arabic-online-commentary-dataset-annotated/)
3. Al-Twairesh, N., Al-Khalifa, H., Al-Salman, A., Al-Ohali, Y.: Arasenti-tweet: A corpus for arabic sentiment analysis of saudi tweets. Procedia Computer Science 117, 63–72 (2017) [find](https://scholar.google.com/scholar?q=Al%2DTwairesh%2C+N%2E%2C+Al%2DKhalifa%2C+H%2E%2C+Al%2DSalman%2C+A%2E%2C+Al%2DOhali%2C+Y%2E%3A+Arasenti%2Dtweet%3A+A+corpus+for+arabic+sentiment+analysis+of+saudi+tweets%2E+Procedia+Computer+Science+117%2C+63%E2%80%9372+%282017%29)
4. Mubarak, H., Darwish, K.: Using twitter to collect a multi-dialectal corpus of arabic. In: Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pp. 1–7 (2014) [find](https://scholar.google.com/scholar?q=Mubarak%2C+H%2E%2C+Darwish%2C+K%2E%3A+Using+twitter+to+collect+a+multi%2Ddialectal+corpus+of+arabic%2E+In%3A+Proceedings+of+the+EMNLP+2014+Workshop+on+Arabic+Natural+Language+Processing+%28ANLP%29%2C+pp%2E+1%E2%80%937+%282014%29)
5. Al-Sabbagh, R., Girju, R.: Yadac: Yet another dialectal arabic corpus. In: LREC, pp. 2882–2889 (2012) [find](https://scholar.google.com/scholar?q=Al%2DSabbagh%2C+R%2E%2C+Girju%2C+R%2E%3A+Yadac%3A+Yet+another+dialectal+arabic+corpus%2E+In%3A+LREC%2C+pp%2E+2882%E2%80%932889+%282012%29)
6. Al-Sabbagh, R., Girju, R.: A supervised pos tagger for written arabic social networking corpora. In: KONVENS, pp. 39–52 (2012) [find](https://scholar.google.com/scholar?q=Al%2DSabbagh%2C+R%2E%2C+Girju%2C+R%2E%3A+A+supervised+pos+tagger+for+written+arabic+social+networking+corpora%2E+In%3A+KONVENS%2C+pp%2E+39%E2%80%9352+%282012%29)
7. Hussein, S., Farouk, M., Hemayed, E.: Gender identification of egyptian dialect in twitter. Egyptian Informatics Journal 20(2), 109–116 (2019) [find](https://scholar.google.com/scholar?q=Hussein%2C+S%2E%2C+Farouk%2C+M%2E%2C+Hemayed%2C+E%2E%3A+Gender+identification+of+egyptian+dialect+in+twitter%2E+Egyptian+Informatics+Journal+20%282%29%2C+109%E2%80%93116+%282019%29)
8. Kora, R., Mohammed, A.: Corpus on Arabic Egyptian Tweets. https://doi.org/10.7910/DVN/LBXV9O . https://doi.org/10.7910/DVN/LBXV9O [↗](https://doi.org/10.7910/dvn/lbxv9o)
9. Kora, R., Mohammed, A.: An enhanced approach for sentiment analysis based on meta-ensemble deep learning. Social Network Analysis and Mining 13(1), 38 (2023) [find](https://scholar.google.com/scholar?q=Kora%2C+R%2E%2C+Mohammed%2C+A%2E%3A+An+enhanced+approach+for+sentiment+analysis+based+on+meta%2Densemble+deep+learning%2E+Social+Network+Analysis+and+Mining+13%281%29%2C+38+%282023%29)
10. Tarmom, T., Teahan, W., Atwell, E., Alsalka, M.A.: Compression versus traditional machine learning classifiers to detect code-switching in varieties and dialects: Arabic as a case study. Natural Language Engineering 26(6), 663–676 (2020) [find](https://scholar.google.com/scholar?q=Tarmom%2C+T%2E%2C+Teahan%2C+W%2E%2C+Atwell%2C+E%2E%2C+Alsalka%2C+M%2EA%2E%3A+Compression+versus+traditional+machine+learning+classifiers+to+detect+code%2Dswitching+in+varieties+and+dialects%3A+Arabic+as+a+case+study%2E+Natural+Langu)
11. Fashwan, A., Alansary, S.: A morphologically annotated corpus and a morphological analyzer for egyptian arabic. Procedia Computer Science 189, 203–210 (2021) [find](https://scholar.google.com/scholar?q=Fashwan%2C+A%2E%2C+Alansary%2C+S%2E%3A+A+morphologically+annotated+corpus+and+a+morphological+analyzer+for+egyptian+arabic%2E+Procedia+Computer+Science+189%2C+203%E2%80%93210+%282021%29) [our card](https://md.research.elamur.ai/free/en/papers/lingbulk-2012-habash-morphological-analyzer-egyptian-arabic/)
12. Fashwan, A., Alansary, S.: Developing a tag-set and extracting the morphological lexicons to build a morphological analyzer for egyptian arabic. In: Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pp. 142–160 (2022) [find](https://scholar.google.com/scholar?q=Fashwan%2C+A%2E%2C+Alansary%2C+S%2E%3A+Developing+a+tag%2Dset+and+extracting+the+morphological+lexicons+to+build+a+morphological+analyzer+for+egyptian+arabic%2E+In%3A+Proceedings+of+the+Seventh+Arabic+Natural+Language+) [our card](https://md.research.elamur.ai/free/en/papers/lingbulk-2012-habash-morphological-analyzer-egyptian-arabic/)
13. Faheem, M.A., Wassif, K.T., Bayomi, H., Abdou, S.M.: Improving neural machine translation for low resource languages through non-parallel corpora: a case study of egyptian dialect to modern standard arabic translation. Scientific Reports 14(1), 2265 (2024) [find](https://scholar.google.com/scholar?q=Faheem%2C+M%2EA%2E%2C+Wassif%2C+K%2ET%2E%2C+Bayomi%2C+H%2E%2C+Abdou%2C+S%2EM%2E%3A+Improving+neural+machine+translation+for+low+resource+languages+through+non%2Dparallel+corpora%3A+a+case+study+of+egyptian+dialect+to+modern+standard+a)
14. Antoun, W., Baly, F., Hajj, H.: Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104 (2020) [↗](https://arxiv.org/abs/2003.00104) [our card](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/)
15. Safaya, A., Abdullatif, M., Yuret, D.: Kuisail at semeval-2020 task 12: Bert-cnn for offensive speech identification in social media. In: Proceedings of the Fourteenth Workshop on Semantic Evaluation, pp. 2054–2059 (2020) [find](https://scholar.google.com/scholar?q=Safaya%2C+A%2E%2C+Abdullatif%2C+M%2E%2C+Yuret%2C+D%2E%3A+Kuisail+at+semeval%2D2020+task+12%3A+Bert%2Dcnn+for+offensive+speech+identification+in+social+media%2E+In%3A+Proceedings+of+the+Fourteenth+Workshop+on+Semantic+Evaluation%2C) [our card](https://md.research.elamur.ai/free/en/papers/safaya-2020-kuisail-semeval-2020-task-12/)
16. Antoun, W., Baly, F., Hajj, H.: Aragpt2: Pre-trained transformer for arabic language generation. In: Proceedings of the Sixth Arabic Natural Language Processing Workshop, pp. 196–207 (2021) [find](https://scholar.google.com/scholar?q=Antoun%2C+W%2E%2C+Baly%2C+F%2E%2C+Hajj%2C+H%2E%3A+Aragpt2%3A+Pre%2Dtrained+transformer+for+arabic+language+generation%2E+In%3A+Proceedings+of+the+Sixth+Arabic+Natural+Language+Processing+Workshop%2C+pp%2E+196%E2%80%93207+%282021%29)
17. Abdul-Mageed, M., Elmadany, A., Nagoudi, E.M.B.: Arbert & marbert: deep bidirectional transformers for arabic. arXiv preprint arXiv:2101.01785 (2020) [↗](https://arxiv.org/abs/2101.01785) [our card](https://md.research.elamur.ai/free/en/papers/abdul-2021-mageed-arbert-marbert-deep-bidirectional-transformers/)
18. Elmadany, A., Abdul-Mageed, M., et al.: Arat5: Text-to-text transformers for arabic language generation. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 628–647 (2022) [find](https://scholar.google.com/scholar?q=Elmadany%2C+A%2E%2C+Abdul%2DMageed%2C+M%2E%2C+et+al%2E%3A+Arat5%3A+Text%2Dto%2Dtext+transformers+for+arabic+language+generation%2E+In%3A+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Vo) [our card](https://md.research.elamur.ai/free/en/papers/nagoudi-2022-arat5-text-text-transformers-arabic/)
19. Eddine, M.K., Tomeh, N., Habash, N., Le Roux, J., Vazirgiannis, M.: Arabart: a pretrained arabic sequence-to-sequence model for abstractive summarization. In: Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pp. 31–42 (2022) [find](https://scholar.google.com/scholar?q=Eddine%2C+M%2EK%2E%2C+Tomeh%2C+N%2E%2C+Habash%2C+N%2E%2C+Le+Roux%2C+J%2E%2C+Vazirgiannis%2C+M%2E%3A+Arabart%3A+a+pretrained+arabic+sequence%2Dto%2Dsequence+model+for+abstractive+summarization%2E+In%3A+Proceedings+of+the+Seventh+Arabic+Natural)
20. Al-Qurishi, M., Alqaseemi, S., Souissi, R.: Aralegal-bert: A pretrained language model for arabic legal text. In: Proceedings of the Natural Legal Language Processing Workshop 2022, pp. 338–344 (2022) [find](https://scholar.google.com/scholar?q=Al%2DQurishi%2C+M%2E%2C+Alqaseemi%2C+S%2E%2C+Souissi%2C+R%2E%3A+Aralegal%2Dbert%3A+A+pretrained+language+model+for+arabic+legal+text%2E+In%3A+Proceedings+of+the+Natural+Legal+Language+Processing+Workshop+2022%2C+pp%2E+338%E2%80%93344+%282022%29)
21. Boudjellal, N., Zhang, H., Khan, A., Ahmad, A., Naseem, R., Shang, J., Dai, L.: Abioner: A bert-based model for arabic biomedical named-entity recognition. Complexity 2021(1), 6633213 (2021) [find](https://scholar.google.com/scholar?q=Boudjellal%2C+N%2E%2C+Zhang%2C+H%2E%2C+Khan%2C+A%2E%2C+Ahmad%2C+A%2E%2C+Naseem%2C+R%2E%2C+Shang%2C+J%2E%2C+Dai%2C+L%2E%3A+Abioner%3A+A+bert%2Dbased+model+for+arabic+biomedical+named%2Dentity+recognition%2E+Complexity+2021%281%29%2C+6633213+%282021%29)
22. Qarah, F.: Arapoembert: A pretrained language model for arabic poetry analysis. arXiv preprint arXiv:2403.12392 (2024) [↗](https://arxiv.org/abs/2403.12392) [our card](https://md.research.elamur.ai/free/en/papers/qarah-2024-arapoembert-pretrained-language-model-arabic/)
23. Zeroual, I., Goldhahn, D., Eckart, T., Lakhouaja, A.: Osian: Open source international arabic news corpus-preparation and integration into the clarin-infrastructure. In: Proceedings of the Fourth Arabic Natural Language Processing Workshop, pp. 175–182 (2019) [find](https://scholar.google.com/scholar?q=Zeroual%2C+I%2E%2C+Goldhahn%2C+D%2E%2C+Eckart%2C+T%2E%2C+Lakhouaja%2C+A%2E%3A+Osian%3A+Open+source+international+arabic+news+corpus%2Dpreparation+and+integration+into+the+clarin%2Dinfrastructure%2E+In%3A+Proceedings+of+the+Fourth+Arab)
24. El-Khair, I.A.: 1.5 billion words arabic corpus. arXiv preprint arXiv:1611.04033 (2016) [↗](https://arxiv.org/abs/1611.04033)
25. Suárez, P.J.O., Sagot, B., Romary, L.: Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. In: 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7) (2019). Leibniz-Institut für Deutsche Sprache [find](https://scholar.google.com/scholar?q=Sua%CC%81rez%2C+P%2EJ%2EO%2E%2C+Sagot%2C+B%2E%2C+Romary%2C+L%2E%3A+Asynchronous+pipeline+for+processing+huge+corpora+on+medium+to+low+resource+infrastructures%2E+In%3A+7th+Workshop+on+the+Challenges+in+the+Management+of+Large+Corpo)
26. Chowdhury, S.A., Abdelali, A., Darwish, K., Soon-Gyo, J., Salminen, J., Jansen, B.J.: Improving arabic text categorization using transformer training diversification. In: Proceedings of the Fifth Arabic Natural Language Processing Workshop, pp. 226–236 (2020) [find](https://scholar.google.com/scholar?q=Chowdhury%2C+S%2EA%2E%2C+Abdelali%2C+A%2E%2C+Darwish%2C+K%2E%2C+Soon%2DGyo%2C+J%2E%2C+Salminen%2C+J%2E%2C+Jansen%2C+B%2EJ%2E%3A+Improving+arabic+text+categorization+using+transformer+training+diversification%2E+In%3A+Proceedings+of+the+Fifth+Arab)
27. Parker, R., et al.: Arabic Gigaword Fifth Edition LDC2011T11. Linguistic Data Consortium, Philadelphia (2011) [find](https://scholar.google.com/scholar?q=Parker%2C+R%2E%2C+et+al%2E%3A+Arabic+Gigaword+Fifth+Edition+LDC2011T11%2E+Linguistic+Data+Consortium%2C+Philadelphia+%282011%29)
28. Lison, P., Tiedemann, J.: Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles. In: Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pp. 923–929 (2016) [find](https://scholar.google.com/scholar?q=Lison%2C+P%2E%2C+Tiedemann%2C+J%2E%3A+Opensubtitles2016%3A+Extracting+large+parallel+corpora+from+movie+and+tv+subtitles%2E+In%3A+Proceedings+of+the+Tenth+International+Conference+on+Language+Resources+and+Evaluation+%28) [our card](https://md.research.elamur.ai/free/en/papers/lison-2016-opensubtitles2016-extracting-large-parallel-corpora/)
29. Elmadany, A., Nagoudi, E.B., Mageed, M.A.: Orca: A challenging benchmark for arabic language understanding. In: Findings of the Association for Computational Linguistics: ACL 2023, pp. 9559–9586 (2023) [find](https://scholar.google.com/scholar?q=Elmadany%2C+A%2E%2C+Nagoudi%2C+E%2EB%2E%2C+Mageed%2C+M%2EA%2E%3A+Orca%3A+A+challenging+benchmark+for+arabic+language+understanding%2E+In%3A+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL+2023%2C+pp%2E+9559%E2%80%939586+%282023)
30. Inoue, G., Alhafni, B., Baimukan, N., Bouamor, H., Habash, N.: The interplay of variant, size, and task type in arabic pre-trained language models. arXiv preprint arXiv:2103.06678 (2021) [↗](https://arxiv.org/abs/2103.06678) [our card](https://md.research.elamur.ai/free/en/papers/inoue-2021-interplay-variant-size-task-type/)
31. Abdaoui, A., Berrimi, M., Oussalah, M., Moussaoui, A.: Dziribert: a pre-trained language model for the algerian dialect. arXiv preprint arXiv:2109.12346 (2021) [↗](https://arxiv.org/abs/2109.12346) [our card](https://md.research.elamur.ai/free/en/papers/abdaoui-2021-dziribert-pre-trained-language-model/)
32. Haddad, H., Rouhou, A.C., Messaoudi, A., Korched, A., Fourati, C., Sellami, A., Ben HajHmida, M., Ghriss, F.: Tunbert: pretraining bert for tunisian dialect understanding. SN Computer Science 4(2), 194 (2023) [find](https://scholar.google.com/scholar?q=Haddad%2C+H%2E%2C+Rouhou%2C+A%2EC%2E%2C+Messaoudi%2C+A%2E%2C+Korched%2C+A%2E%2C+Fourati%2C+C%2E%2C+Sellami%2C+A%2E%2C+Ben+HajHmida%2C+M%2E%2C+Ghriss%2C+F%2E%3A+Tunbert%3A+pretraining+bert+for+tunisian+dialect+understanding%2E+SN+Computer+Science+4%282%29%2C+19)
33. Gaanoun, K., Naira, A.M., Allak, A., Benelallam, I.: Darijabert: a step forward in nlp for the written moroccan dialect. International Journal of Data Science and Analytics, 1–13 (2024) [find](https://scholar.google.com/scholar?q=Gaanoun%2C+K%2E%2C+Naira%2C+A%2EM%2E%2C+Allak%2C+A%2E%2C+Benelallam%2C+I%2E%3A+Darijabert%3A+a+step+forward+in+nlp+for+the+written+moroccan+dialect%2E+International+Journal+of+Data+Science+and+Analytics%2C+1%E2%80%9313+%282024%29)
34. Qarah, F.: Saudibert: A large language model pretrained on saudi dialect corpora. arXiv preprint arXiv:2405.06239 (2024) [↗](https://arxiv.org/abs/2405.06239) [our card](https://md.research.elamur.ai/free/en/papers/qarah-2024-saudibert-large-language-model-pretrained/)
35. Richardson, L.: Beautiful soup documentation. April (2007) [find](https://scholar.google.com/scholar?q=Richardson%2C+L%2E%3A+Beautiful+soup+documentation%2E+April+%282007%29)
36. Kudo, T., Richardson, J.: Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226 (2018) [↗](https://arxiv.org/abs/1808.06226) [our card](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/)
37. Qarah, F., Alsanoosy, T.: A comprehensive analysis of various tokenizers for arabic large language models. Applied Sciences 14(13), 5696 (2024) [find](https://scholar.google.com/scholar?q=Qarah%2C+F%2E%2C+Alsanoosy%2C+T%2E%3A+A+comprehensive+analysis+of+various+tokenizers+for+arabic+large+language+models%2E+Applied+Sciences+14%2813%29%2C+5696+%282024%29)
38. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al.: Transformers: State-of-the-art natural language processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38–45 (2020) [find](https://scholar.google.com/scholar?q=Wolf%2C+T%2E%2C+Debut%2C+L%2E%2C+Sanh%2C+V%2E%2C+Chaumond%2C+J%2E%2C+Delangue%2C+C%2E%2C+Moi%2C+A%2E%2C+Cistac%2C+P%2E%2C+Rault%2C+T%2E%2C+Louf%2C+R%2E%2C+Funtowicz%2C+M%2E%2C+et+al%2E%3A+Transformers%3A+State%2Dof%2Dthe%2Dart+natural+language+processing%2E+In%3A+Proceedings+) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
39. Mubarak, H., Darwish, K., Magdy, W.: Abusive language detection on arabic social media. In: Proceedings of the First Workshop on Abusive Language Online, pp. 52–56 (2017) [find](https://scholar.google.com/scholar?q=Mubarak%2C+H%2E%2C+Darwish%2C+K%2E%2C+Magdy%2C+W%2E%3A+Abusive+language+detection+on+arabic+social+media%2E+In%3A+Proceedings+of+the+First+Workshop+on+Abusive+Language+Online%2C+pp%2E+52%E2%80%9356+%282017%29) [our card](https://md.research.elamur.ai/free/en/papers/mubarak-2017-abusive-language-detection-arabic-social/)
40. Farha, I.A., Oprea, S., Wilson, S., Magdy, W.: Semeval-2022 task 6: isarcasmeval, intended sarcasm detection in english and arabic. In: The 16th International Workshop on Semantic Evaluation 2022, pp. 802–814 (2022). Association for Computational Linguistics [find](https://scholar.google.com/scholar?q=Farha%2C+I%2EA%2E%2C+Oprea%2C+S%2E%2C+Wilson%2C+S%2E%2C+Magdy%2C+W%2E%3A+Semeval%2D2022+task+6%3A+isarcasmeval%2C+intended+sarcasm+detection+in+english+and+arabic%2E+In%3A+The+16th+International+Workshop+on+Semantic+Evaluation+2022%2C+pp%2E) [our card](https://md.research.elamur.ai/free/en/papers/farha-2022-semeval-2022-task-6-isarcasmeval/)
41. Al-Khatib, A., El-Beltagy, S.R.: Emotional tone detection in arabic tweets. In: Computational Linguistics and Intelligent Text Processing: 18th International Conference, CICLing 2017, Budapest, Hungary, April 17–23, 2017, Revised Selected Papers, Part II 18, pp. 105–114 (2018). Springer [find](https://scholar.google.com/scholar?q=Al%2DKhatib%2C+A%2E%2C+El%2DBeltagy%2C+S%2ER%2E%3A+Emotional+tone+detection+in+arabic+tweets%2E+In%3A+Computational+Linguistics+and+Intelligent+Text+Processing%3A+18th+International+Conference%2C+CICLing+2017%2C+Budapest%2C+Hungar)
42. AbdelRahim, H.M.E., Magdy, W.: Arsas: An arabic speech-act and sentiment corpus of tweets. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Paris, France. European Language Resources Association (ELRA) (2018) [find](https://scholar.google.com/scholar?q=AbdelRahim%2C+H%2EM%2EE%2E%2C+Magdy%2C+W%2E%3A+Arsas%3A+An+arabic+speech%2Dact+and+sentiment+corpus+of+tweets%2E+In%3A+Proceedings+of+the+Eleventh+International+Conference+on+Language+Resources+and+Evaluation+%28LREC+2018%29%2C+Pa)
43. EgyptionCompaniesReviews-Sentiment-analysis. [https://github.com/zahran1234/EgyptionCompaniesReviews](https://github.com/zahran1234/EgyptionCompaniesReviews) Sentiment analysis/tree/main. Accessed: 2024-07-21 [↗](https://github.com/zahran1234/EgyptionCompaniesReviews)