---
title: "The Interplay of Variant, Size, and Task Type — Inoue 2021"
description: "Arabic has many BERT-style models, but they differ in data size, variant, tokenizer, vocabulary and training length."
authors: ["Go Inoue", "Bashar Alhafni", "Nurpeiis Baimukan", "Houda Bouamor", "Nizar Habash"]
year: 2021
venue: "arXiv:2103.06678"
source: "https://arxiv.org/abs/2103.06678"
doi: "10.48550/arxiv.2103.06678"
arxiv: "2103.06678"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/inoue-2021-interplay-variant-size-task-type/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/inoue-2021-interplay-variant-size-task-type/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/5/) / The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models

# The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models

**Authors:** [Go Inoue](https://md.research.elamur.ai/free/en/authors/go-inoue/), [Bashar Alhafni](https://md.research.elamur.ai/free/en/authors/bashar-alhafni/), [Nurpeiis Baimukan](https://md.research.elamur.ai/free/en/authors/nurpeiis-baimukan/), [Houda Bouamor](https://md.research.elamur.ai/free/en/authors/houda-bouamor/), [Nizar Habash](https://md.research.elamur.ai/free/en/authors/nizar-habash/) · **Year:** 2021 · **Venue:** arXiv:2103.06678 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2103.06678) · [PDF](https://arxiv.org/pdf/2103.06678) · [DOI 10.48550/arxiv.2103.06678](https://doi.org/10.48550/arxiv.2103.06678) · [arXiv 2103.06678](https://arxiv.org/abs/2103.06678) · [OpenAlex](https://openalex.org/W3134155512)

**Citations:** 135 citations (openalex-snapshot · 2026-10-14) · [230](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Arabic has many BERT-style models, but they differ in data size, variant, tokenizer, vocabulary and training length. This makes it unclear which factors actually drive downstream quality. The authors run a controlled study. They pre-train CAMeLBERT models with the original BERT hyperparameters (WordPiece, 30k vocabulary, 1M steps) and vary only the training data: Modern Standard Arabic (MSA), dialectal Arabic (DA), classical Arabic (CA), a mixture of all three, and smaller MSA subsets. These eight models and eight public Arabic models are fine-tuned and compared on 12 datasets spanning five tasks: named entity recognition, part-of-speech tagging, classifying sentiment, identifying the dialect, and classifying poetry. The central finding is that matching the pre-training variant to the task data matters more than corpus size. Existing models show the same pattern: the 24GB AraBERTv0.1 outperforms ~95GB models on average. Building on this, the authors propose CAMeLBERT-Star, a simple rule that picks the CA, MSA or Mix model according to the task's variant. Models and code are openly released.

## Key points
- Corpus sizes per model:
  - CAMeLBERT-Mix: 167GB, 17.3B tokens, the largest Arabic pre-training set at the time.
  - CAMeLBERT-MSA: 107GB, 12.6B tokens.
  - CAMeLBERT-DA: 54GB, 5.8B tokens.
  - CAMeLBERT-CA: 6GB, 847M tokens, from OpenITI v1.2.
- Training setup:
  - Each model trained on a TPU v3-8 for about 4.5 days.
  - Schedule: 900k steps at sequence length 128 (batch 1,024), then 100k steps at length 512 (batch 256).
  - Whole word masking was used.
  - Diacritics and kashida were removed with CAMeL Tools.
- Evaluation sets: ANERcorp, PATB, ARZTB, Gumar, ASTD, ArSAS, SemEval, MADAR-26/-6/-Twitter-5, NADI and APCD. Each dataset's variant was partly estimated from the share of MSA sentences it contains.
- Head-to-head results:
  - AraBERTv0.2 has the best average and wins or ties 6 of 12 subtasks.
  - CAMeLBERT-Star ranks second, with 4 wins or ties.
  - The two are complementary and together cover 9 of 12 subtasks.
- Variant effects in existing models:
  - MSA-only models do slightly better on MSA subtasks (82.6% vs 82.1%).
  - Mixed models do better on DA subtasks (73.9% vs 73.5%).
  - MARBERT vs ARBERT (78.9% vs 78.5%) does not fully follow this pattern.
  - Models are published on Hugging Face under CAMeL-Lab.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

230 papers of our library cite it (of the 140884 whose bibliographies we read); 61 of them have a page here, 169 not yet.

- [Arabic Dialect Classification using RNNs, Transformers, and Large Language Models: A Comparative Analysis](https://md.research.elamur.ai/free/en/papers/a-2025-essameldin-arabic-dialect-classification-using-rnns/)
- [Voice Conversion Improves Cross-Domain Robustness for Spoken Arabic Dialect Identification](https://md.research.elamur.ai/free/en/papers/abdullah-2025-voice-conversion-improves-cross-domain/)
- [TTLab at AlexandriaX-2026: A Fine-Tuned Surface Tagger for Arabic Machine-Translation Error-Span Detection and Classification](https://md.research.elamur.ai/free/en/papers/abusaleh-2026-ttlab-alexandriax-2026-fine-tuned/)
- [Out-of-Domain Dependency Parsing for Dialects of Arabic: A Case Study](https://md.research.elamur.ai/free/en/papers/aclbulk-2024-mokh-out-domain-dependency-parsing-dialects/)
- [Parsing Arabic Dialects Revisited: New Benchmarks, Models, and Insights](https://md.research.elamur.ai/free/en/papers/aclbulk-2026-elshabrawy-parsing-arabic-dialects-revisited-new/)
- [Do Diacritics Matter? Evaluating the Impact of Arabic Diacritics on Tokenization and LLM Benchmarks](https://md.research.elamur.ai/free/en/papers/aclbulk-2026-inoue-do-diacritics-matter-evaluating-impact/)
- [AlcLaM: Arabic Dialectal Language Model](https://md.research.elamur.ai/free/en/papers/ahmed-2024-alclam-arabic-dialectal-language-model/)
- [The Landscape of Arabic Large Language Models (ALLMs): A New Era for Arabic Language Technology](https://md.research.elamur.ai/free/en/papers/al-2025-khalifa-landscape-arabic-large-language-models/)
- [Evaluation of Adversarial Robustness in Arabic Language Models](https://md.research.elamur.ai/free/en/papers/alajmi-2026-evaluation-adversarial-robustness-arabic-language/)
- [High Tech team at SemEval-2022 Task 6: Intended Sarcasm Detection for Arabic texts](https://md.research.elamur.ai/free/en/papers/alami-2022-high-tech-team-semeval-2022/)
- [GHAD NLP at AdabEval2026: Transformer-Based Approach for Arabic Politeness and Pragmatic Category Classification](https://md.research.elamur.ai/free/en/papers/alfattni-2026-ghad-nlp-adabeval2026-transformer-based/)
- [Advancements in Arabic Grammatical Error Detection and Correction: An Empirical Investigation](https://md.research.elamur.ai/free/en/papers/alhafni-2023-advancements-arabic-grammatical-error-detection/)
- [The User-Aware Arabic Gender Rewriter](https://md.research.elamur.ai/free/en/papers/alhafni-2023-user-aware-arabic-gender-rewriter/)
- [A Bilingual Bimodal Benchmark for Arabic-English NLP across Grammatical Correction, Essay Scoring, Morphological Tagging, and Speech Recognition](https://md.research.elamur.ai/free/en/papers/alhafni-2026-bilingual-bimodal-benchmark-arabic-english/)
- [MentalMARBERT: Domain-Adaptive Pre-training and Two-Stage Fine-Tuning for Arabic Mental Health Disorders Detection](https://md.research.elamur.ai/free/en/papers/almalki-2026-mentalmarbert-domain-adaptive-pre-training/)
- [Harnessing Multilingual Resources to Question Answering in Arabic](https://md.research.elamur.ai/free/en/papers/alnajjar-2022-harnessing-multilingual-resources-question-answering/)
- [Towards the Development of Balanced Synthetic Data for Correcting Grammatical Errors in Arabic: An Approach Based on Error Tagging Model and Synthetic Data Generating Model](https://md.research.elamur.ai/free/en/papers/alrehili-2025-towards-development-balanced-synthetic-data/)
- [StanceEval 2024: The First Arabic Stance Detection Shared Task](https://md.research.elamur.ai/free/en/papers/alturayeif-2024-stanceeval-2024-first-arabic-stance/)
- [Investigating Persuasion Techniques in Arabic: An Empirical Study Leveraging Large Language Models](https://md.research.elamur.ai/free/en/papers/alzahrani-2024-investigating-persuasion-techniques-arabic-empirical/)
- [Content-Localization based Neural Machine Translation for Informal Dialectal Arabic: Spanish/French to Levantine/Gulf Arabic](https://md.research.elamur.ai/free/en/papers/alzamzami-2023-content-localization-based-neural-machine/)
- [Content-Localization based System for Analyzing Sentiment and Hate Behaviors in Low-Resource Dialectal Arabic: English to Levantine and Gulf](https://md.research.elamur.ai/free/en/papers/alzamzami-2023-content-localization-based-system-analyzing/)
- [Hierarchical Aggregation of Dialectal Data for Arabic Dialect Identification](https://md.research.elamur.ai/free/en/papers/baimukan-2022-hierarchical-aggregation-dialectal-data-arabic/)
- [Enhanced Arabic Text Retrieval with Attentive Relevance Scoring](https://md.research.elamur.ai/free/en/papers/bekhouche-2025-enhanced-arabic-text-retrieval-attentive/)
- [Camel Morph MSA: A Large-Scale Open-Source Morphological Analyzer for Modern Standard Arabic](https://md.research.elamur.ai/free/en/papers/coling-2024-khairallah-camel-morph-msa-large-scale/)
- [Lemmatization as a Classification Task: Results from Arabic across Multiple Genres](https://md.research.elamur.ai/free/en/papers/ctx-2025-saeed-lemmatization-as-classification-task-results/)
- [Extractive Summarization for Arabic Documents Using SAraBERT with a Semantic Siamese Similarity Evaluation Metric](https://md.research.elamur.ai/free/en/papers/deen-2026-extractive-summarization-arabic-documents-using/)
- [Sentiment Classification of Gaza War Headlines: A Comparative Analysis of Large Language Models and Arabic Fine-Tuned BERT Models](https://md.research.elamur.ai/free/en/papers/eleraqi-2026-sentiment-classification-gaza-war-headlines/)
- [MuDRiC: Multi-Dialect Reasoning for Arabic Commonsense Validation](https://md.research.elamur.ai/free/en/papers/elozeiri-2025-mudric-multi-dialect-reasoning-arabic/)
- [Grounding Arabic LLMs in the Doha Historical Dictionary: Retrieval-Augmented Understanding of Quran and Hadith](https://md.research.elamur.ai/free/en/papers/eltanbouly-2026-grounding-arabic-llms-doha-historical/)
- [JABER and SABER: Junior and Senior Arabic BERt](https://md.research.elamur.ai/free/en/papers/ghaddar-2021-jaber-saber-junior-senior-arabic/)
- [Revisiting Pre-trained Language Models and their Evaluation for Arabic Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/ghaddar-2022-revisiting-pre-trained-language-models-their/)
- [On the importance of Data Scale in Pretraining Arabic Language Models](https://md.research.elamur.ai/free/en/papers/ghaddar-2024-importance-data-scale-pretraining-arabic/)
- [Benchmarking Arabic Authorship Attribution and Style Transfer with Large Language Models](https://md.research.elamur.ai/free/en/papers/hamed-2026-benchmarking-arabic-authorship-attribution-style/)
- [Detecting Stance of Authorities towards Rumors in Arabic Tweets: A Preliminary Study](https://md.research.elamur.ai/free/en/papers/haouari-2023-detecting-stance-authorities-towards-rumors/)
- [SMASH at StanceEval 2024: Prompt Engineering LLMs for Arabic Stance Detection](https://md.research.elamur.ai/free/en/papers/hariri-2024-smash-stanceeval-2024-prompt-engineering/)
- [TARJAMAT: Evaluation of Bard and ChatGPT on Machine Translation of Ten Arabic Varieties](https://md.research.elamur.ai/free/en/papers/kadaoui-2023-tarjamat-evaluation-bard-chatgpt-machine/)
- [Exploring Data and Parameter Efficient Strategies for Arabic Dialect Identifications](https://md.research.elamur.ai/free/en/papers/kanjirangat-2025-exploring-data-parameter-efficient-strategies/)
- [ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic](https://md.research.elamur.ai/free/en/papers/koto-2024-arabicmmlu-assessing-massive-multitask-language/)
- [ArabianGPT: Native Arabic GPT-based Large Language Model](https://md.research.elamur.ai/free/en/papers/koubaa-2024-arabiangpt-native-arabic-gpt-based/)
- [Arabic Morphosyntactic Tagging and Dependency Parsing with Large Language Models](https://md.research.elamur.ai/free/en/papers/langstream-2026-adel-arabic-morphosyntactic-tagging-dependency-parsing/)
- [Arabic Sentence Segmentation Across Genres and Punctuation Conditions](https://md.research.elamur.ai/free/en/papers/langstream-2026-elkholy-arabic-sentence-segmentation-across-genres/)
- [HEBATRON: A Hebrew-Specialized Open-Weight Mixture-of-Experts Language Model](https://md.research.elamur.ai/free/en/papers/langstream-2026-kayzer-hebatron-hebrew-specialized-open-weight/)
- [Camel Treebank: An Open Multi-genre Arabic Dependency Treebank](https://md.research.elamur.ai/free/en/papers/lingbulk-2022-habash-camel-treebank-open-multi-genre/)
- [STAGEET: Stage-wise Typed Edit Tagging for Grammatical Error Correction with Arabic as a Case Study](https://md.research.elamur.ai/free/en/papers/lou-2026-stageet-stage-wise-typed-edit/)
- [Assessing Suicide Risk in Arabic Crisis Helpline Calls: A Comparison of Arabic and English Large Language Models](https://md.research.elamur.ai/free/en/papers/ma-2026-assessing-suicide-risk-arabic-crisis/)
- [CS-UM6P at SemEval-2022 Task 6: Transformer-based Models for Intended Sarcasm Detection in English and Arabic](https://md.research.elamur.ai/free/en/papers/mahdaouy-2022-cs-um6p-semeval-2022-task/)
- [CARMA: Comprehensive Automatically-annotated Reddit Mental Health Dataset for Arabic](https://md.research.elamur.ai/free/en/papers/mankarious-2025-carma-comprehensive-automatically-annotated-reddit/)
- [EDRAC: Benchmarking Arabic Dialect Reading Comprehension](https://md.research.elamur.ai/free/en/papers/mokh-2026-edrac-benchmarking-arabic-dialect-reading/)
- [Beyond Poetry: Can Large Language Models Generate Classical Arabic Maqamat?](https://md.research.elamur.ai/free/en/papers/morsy-2026-poetry-large-language-models-generate/)
- [ANLPers at AraGenEval Shared Task: Descriptive Author Tokens for Transparent Arabic Authorship Style Transfer](https://md.research.elamur.ai/free/en/papers/nacar-2025-anlpers-arageneval-shared-task-descriptive/)
- [AraT5: Text-to-Text Transformers for Arabic Language Generation](https://md.research.elamur.ai/free/en/papers/nagoudi-2022-arat5-text-text-transformers-arabic/)
- [Having Beer after Prayer? Measuring Cultural Bias in Large Language Models](https://md.research.elamur.ai/free/en/papers/naous-2023-having-beer-after-prayer-measuring/)
- [Plumeria at SemEval-2022 Task 6: Robust Approaches for Sarcasm Detection for English and Arabic Using Transformers and Data Augmentation](https://md.research.elamur.ai/free/en/papers/nigam-2022-plumeria-semeval-2022-task-6/)
- [AraPoemBERT: A Pretrained Language Model for Arabic Poetry Analysis](https://md.research.elamur.ai/free/en/papers/qarah-2024-arapoembert-pretrained-language-model-arabic/)
- [EgyBERT: A Large Language Model Pretrained on Egyptian Dialect Corpora](https://md.research.elamur.ai/free/en/papers/qarah-2024-egybert-large-language-model-pretrained/)
- [SaudiBERT: A Large Language Model Pretrained on Saudi Dialect Corpora](https://md.research.elamur.ai/free/en/papers/qarah-2024-saudibert-large-language-model-pretrained/)
- [Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models](https://md.research.elamur.ai/free/en/papers/sengupta-2023-jais-jais-chat-arabic-centric/)
- [Plumeria at SemEval-2022 Task 6: Sarcasm Detection for English and Arabic Using Transformers and Data Augmentation](https://md.research.elamur.ai/free/en/papers/shaheen-2022-plumeria-semeval-2022-task-6/)
- [Arabic Text Diacritization In The Age Of Transfer Learning: Token Classification Is All You Need](https://md.research.elamur.ai/free/en/papers/skiredj-2024-arabic-text-diacritization-age-transfer/)
- [DarijaBanking: A New Resource for Overcoming Language Barriers in Banking Intent Detection for Moroccan Arabic Speakers](https://md.research.elamur.ai/free/en/papers/skiredj-2024-darijabanking-new-resource-overcoming-language/)
- [CamelParser2.0: A State-of-the-Art Dependency Parser for Arabic](https://md.research.elamur.ai/free/en/papers/ud-2023-elshabrawy-camelparser2-0-state-art-dependency/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{inoue2021interplay,
  title = {{The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models}},
  author = {Inoue, Go and Alhafni, Bashar and Baimukan, Nurpeiis and Bouamor, Houda and Habash, Nizar},
  year = {2021},
  doi = {10.48550/arxiv.2103.06678},
  eprint = {2103.06678},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2103.06678}
}
```

- [← previous The Importance of Modeling Social Factors of Language: Theory and Practice](https://md.research.elamur.ai/free/en/papers/hovy-2021-importance-modeling-social-factors-language/)
- [next → The Lean 4 Theorem Prover and Programming Language (System Description)](https://md.research.elamur.ai/free/en/papers/moura-2021-lean-4-theorem-prover-programming/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/5/)

## References

67 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for Arabic. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 11–16, San Diego, California. [find](https://scholar.google.com/scholar?q=Ahmed+Abdelali%2C+Kareem+Darwish%2C+Nadir+Durrani%2C+and+Hamdy+Mubarak%2E+2016%2E+Farasa%3A+A+fast+and+furious+segmenter+for+Arabic%2E+In+Proceedings+of+the+Conference+of+the+North+American+Chapter+of+the+Associati) [our card](https://md.research.elamur.ai/free/en/papers/aclbulk-2016-abdelali-farasa-fast-furious-segmenter-arabic/)
2. Ahmed Abdelali, Hamdy Mubarak, Younes Samih, Sabit Hassan, and Kareem Darwish. 2020. Arabic dialect identification in the wild. [find](https://scholar.google.com/scholar?q=Ahmed+Abdelali%2C+Hamdy+Mubarak%2C+Younes+Samih%2C+Sabit+Hassan%2C+and+Kareem+Darwish%2E+2020%2E+Arabic+dialect+identification+in+the+wild%2E) [our card](https://md.research.elamur.ai/free/en/papers/abdelali-2020-arabic-dialect-identification-wild/)
3. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2020a. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. [find](https://scholar.google.com/scholar?q=Muhammad+Abdul%2DMageed%2C+AbdelRahim+Elmadany%2C+and+El+Moatez+Billah+Nagoudi%2E+2020a%2E+ARBERT+%26+MARBERT%3A+Deep+bidirectional+transformers+for+Arabic%2E) [our card](https://md.research.elamur.ai/free/en/papers/abdul-2021-mageed-arbert-marbert-deep-bidirectional-transformers/)
4. Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020b. NADI 2020: The First Nuanced Arabic Dialect Identification Shared Task. In Proceedings of the Fifth Arabic Natural Language Processing Workshop (WANLP 2020), Barcelona, Spain. [find](https://scholar.google.com/scholar?q=Muhammad+Abdul%2DMageed%2C+Chiyu+Zhang%2C+Houda+Bouamor%2C+and+Nizar+Habash%2E+2020b%2E+NADI+2020%3A+The+First+Nuanced+Arabic+Dialect+Identification+Shared+Task%2E+In+Proceedings+of+the+Fifth+Arabic+Natural+Language+)
5. Ibrahim Abu Farha and Walid Magdy. 2019. Mazajak: An online Arabic sentiment analyser. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 192–198, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ibrahim+Abu+Farha+and+Walid+Magdy%2E+2019%2E+Mazajak%3A+An+online+Arabic+sentiment+analyser%2E+In+Proceedings+of+the+Fourth+Arabic+Natural+Language+Processing+Workshop%2C+pages+192%E2%80%93198%2C+Florence%2C+Italy%2E+Associa)
6. Kathrein Abu Kwaik, Motaz Saad, Stergios Chatzikyriakidis, and Simon Dobnik. 2018. Shami: A corpus of Levantine Arabic dialects. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Kathrein+Abu+Kwaik%2C+Motaz+Saad%2C+Stergios+Chatzikyriakidis%2C+and+Simon+Dobnik%2E+2018%2E+Shami%3A+A+corpus+of+Levantine+Arabic+dialects%2E+In+Proceedings+of+the+Eleventh+International+Conference+on+Language+Res)
7. Nora Al-Twairesh, Rawan Al-Matham, Nora Madi, Nada Almugren, Al-Hanouf Al-Aljmi, Shahad Alshalan, Raghad Alshalan, Nafla Alrumayyan, Shams Al-Manea, Sumayah Bawazeer, Nourah Al-Mutlaq, Nada Almanea, Waad Bin Huwaymil, Dalal Alqusair, Reem Alotaibi, Suha Al-Senaydi, and Abeer Alfutamani. 2018. SUAR: Towards building a corpus for the Saudi dialect. In Proceedings of the International Conference on Arabic Computational Linguistics (ACLing). [find](https://scholar.google.com/scholar?q=Nora+Al%2DTwairesh%2C+Rawan+Al%2DMatham%2C+Nora+Madi%2C+Nada+Almugren%2C+Al%2DHanouf+Al%2DAljmi%2C+Shahad+Alshalan%2C+Raghad+Alshalan%2C+Nafla+Alrumayyan%2C+Shams+Al%2DManea%2C+Sumayah+Bawazeer%2C+Nourah+Al%2DMutlaq%2C+Nada+Almanea%2C+W)
8. Ahmed Ali, Preslav Nakov, Peter Bell, and Steve Renals. 2017. WERD: using social text spelling variants for evaluating dialectal speech recognition. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2017, Okinawa, Japan, December 16-20, 2017, pages 141– 148. IEEE. [find](https://scholar.google.com/scholar?q=Ahmed+Ali%2C+Preslav+Nakov%2C+Peter+Bell%2C+and+Steve+Renals%2E+2017%2E+WERD%3A+using+social+text+spelling+variants+for+evaluating+dialectal+speech+recognition%2E+In+2017+IEEE+Automatic+Speech+Recognition+and+Under)
9. Khalid Almeman and Mark Lee. 2013. Automatic building of Arabic multi dialect text corpora by bootstrapping dialect words. In Proceedings of the International Conference on Communications, Signal Processing, and their Applications (ICCSPA), pages 1–6. [find](https://scholar.google.com/scholar?q=Khalid+Almeman+and+Mark+Lee%2E+2013%2E+Automatic+building+of+Arabic+multi+dialect+text+corpora+by+bootstrapping+dialect+words%2E+In+Proceedings+of+the+International+Conference+on+Communications%2C+Signal+Proc)
10. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. [find](https://scholar.google.com/scholar?q=Wissam+Antoun%2C+Fady+Baly%2C+and+Hazem+Hajj%2E+2020%2E+AraBERT%3A+Transformer%2Dbased+model+for+Arabic+language+understanding%2E+In+Proceedings+of+the+4th+Workshop+on+Open%2DSource+Arabic+Corpora+and+Processing+Tool) [our card](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/)
11. Appen Pty Ltd. 2006a. Gulf Arabic Conversational Telephone Speech, Transcripts LDC2006T15. [find](https://scholar.google.com/scholar?q=Appen+Pty+Ltd%2E+2006a%2E+Gulf+Arabic+Conversational+Telephone+Speech%2C+Transcripts+LDC2006T15%2E)
12. Appen Pty Ltd. 2006b. Iraqi Arabic Conversational Telephone Speech, Transcripts LDC2006T16. [find](https://scholar.google.com/scholar?q=Appen+Pty+Ltd%2E+2006b%2E+Iraqi+Arabic+Conversational+Telephone+Speech%2C+Transcripts+LDC2006T16%2E)
13. Appen Pty Ltd. 2007. Levantine Arabic Conversational Telephone Speech, Transcripts LDC2007T01. [find](https://scholar.google.com/scholar?q=Appen+Pty+Ltd%2E+2007%2E+Levantine+Arabic+Conversational+Telephone+Speech%2C+Transcripts+LDC2007T01%2E)
14. Ramy Baly, Alaa Khaddaj, Hazem Hajj, Wassim El-Hajj, and Khaled Bashir Shaban. 2019. ArSentD-LEV: A multi-topic corpus for target-based sentiment analysis in Arabic Levantine tweets. [find](https://scholar.google.com/scholar?q=Ramy+Baly%2C+Alaa+Khaddaj%2C+Hazem+Hajj%2C+Wassim+El%2DHajj%2C+and+Khaled+Bashir+Shaban%2E+2019%2E+ArSentD%2DLEV%3A+A+multi%2Dtopic+corpus+for+target%2Dbased+sentiment+analysis+in+Arabic+Levantine+tweets%2E) [our card](https://md.research.elamur.ai/free/en/papers/baly-2019-arsentd-lev-multi-topic-corpus/)
15. BBN Technologies. 2005. BBN/AUB DARPA Babylon Levantine Arabic Speech and Transcripts LDC2005S08. [find](https://scholar.google.com/scholar?q=BBN+Technologies%2E+2005%2E+BBN%2FAUB+DARPA+Babylon+Levantine+Arabic+Speech+and+Transcripts+LDC2005S08%2E)
16. Yassine Benajiba, Paolo Rosso, and José Miguel Benedı́ Ruiz. 2007. ANERsys: An Arabic Named Entity Recognition System Based on Maximum Entropy. In Computational Linguistics and Intelligent Text Processing, pages 143–153, Berlin, Heidelberg. Springer Berlin Heidelberg. [find](https://scholar.google.com/scholar?q=Yassine+Benajiba%2C+Paolo+Rosso%2C+and+Jose%CC%81+Miguel+Bened%C4%B1%CC%81+Ruiz%2E+2007%2E+ANERsys%3A+An+Arabic+Named+Entity+Recognition+System+Based+on+Maximum+Entropy%2E+In+Computational+Linguistics+and+Intelligent+Text+Proce)
17. Houda Bouamor, Nizar Habash, and Kemal Oflazer. 2014. A multidialectal parallel corpus of Arabic. In Proceedings of the Language Resources and Evaluation Conference (LREC), Reykjavik, Iceland. [find](https://scholar.google.com/scholar?q=Houda+Bouamor%2C+Nizar+Habash%2C+and+Kemal+Oflazer%2E+2014%2E+A+multidialectal+parallel+corpus+of+Arabic%2E+In+Proceedings+of+the+Language+Resources+and+Evaluation+Conference+%28LREC%29%2C+Reykjavik%2C+Iceland%2E) [our card](https://md.research.elamur.ai/free/en/papers/bouamor-2014-multidialectal-parallel-corpus-arabic/)
18. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The MADAR Arabic Dialect Corpus and Lexicon. In Proceedings of the Language Resources and Evaluation Conference (LREC), Miyazaki, Japan. [find](https://scholar.google.com/scholar?q=Houda+Bouamor%2C+Nizar+Habash%2C+Mohammad+Salameh%2C+Wajdi+Zaghouani%2C+Owen+Rambow%2C+Dana+Abdulrahim%2C+Ossama+Obeid%2C+Salam+Khalifa%2C+Fadhl+Eryani%2C+Alexander+Erdmann%2C+and+Kemal+Oflazer%2E+2018%2E+The+MADAR+Arabic+Di)
19. Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. The MADAR shared task on Arabic fine-grained dialect identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 199–207, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Houda+Bouamor%2C+Sabit+Hassan%2C+and+Nizar+Habash%2E+2019%2E+The+MADAR+shared+task+on+Arabic+fine%2Dgrained+dialect+identification%2E+In+Proceedings+of+the+Fourth+Arabic+Natural+Language+Processing+Workshop%2C+page) [our card](https://md.research.elamur.ai/free/en/papers/bouamor-2019-madar-shared-task-arabic-fine/)
20. Song Chen, Dana Fore, Stephanie Strassel, Haejoong Lee, and Jonathan Wright. 2017. BOLT Egyptian Arabic SMS/Chat and Transliteration LDC2017T07. [find](https://scholar.google.com/scholar?q=Song+Chen%2C+Dana+Fore%2C+Stephanie+Strassel%2C+Haejoong+Lee%2C+and+Jonathan+Wright%2E+2017%2E+BOLT+Egyptian+Arabic+SMS%2FChat+and+Transliteration+LDC2017T07%2E)
21. Ethan A. Chi, John Hewitt, and Christopher D. Manning. 2020. Finding universal grammatical relations in multilingual BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5564–5577, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ethan+A%2E+Chi%2C+John+Hewitt%2C+and+Christopher+D%2E+Manning%2E+2020%2E+Finding+universal+grammatical+relations+in+multilingual+BERT%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computationa)
22. Ryan Cotterell and Chris Callison-Burch. 2014. A Multi-Dialect, Multi-Genre Corpus of Informal Written Arabic. In Proceedings of the Language Resources and Evaluation Conference (LREC), pages 241–245, Reykjavik, Iceland. [find](https://scholar.google.com/scholar?q=Ryan+Cotterell+and+Chris+Callison%2DBurch%2E+2014%2E+A+Multi%2DDialect%2C+Multi%2DGenre+Corpus+of+Informal+Written+Arabic%2E+In+Proceedings+of+the+Language+Resources+and+Evaluation+Conference+%28LREC%29%2C+pages+241%E2%80%93245%2C)
23. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1423)
24. Mahmoud El-Haj. 2020. Habibi - a multi dialect multi national Arabic song lyrics corpus. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 1318–1326, Marseille, France. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Mahmoud+El%2DHaj%2E+2020%2E+Habibi+%2D+a+multi+dialect+multi+national+Arabic+song+lyrics+corpus%2E+In+Proceedings+of+The+12th+Language+Resources+and+Evaluation+Conference%2C+pages+1318%E2%80%931326%2C+Marseille%2C+France%2E+Eu) [our card](https://md.research.elamur.ai/free/en/papers/haj-2020-habibi-multi-dialect-multi-national/)
25. Ibrahim Abu El-Khair. 2016. 1.5 billion words Arabic corpus. CoRR, abs/1611.04033. [↗](https://arxiv.org/abs/1611.04033)
26. AbdelRahim Elmadany, Hamdy Mubarak, and Walid Magdy. 2018. ArSAS: An Arabic speech-act and sentiment corpus of tweets. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Paris, France. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=AbdelRahim+Elmadany%2C+Hamdy+Mubarak%2C+and+Walid+Magdy%2E+2018%2E+ArSAS%3A+An+Arabic+speech%2Dact+and+sentiment+corpus+of+tweets%2E+In+Proceedings+of+the+Eleventh+International+Conference+on+Language+Resources+and)
27. Hassan Gadalla, Hanaa Kilany, Howaida Arram, Ashraf Yacoub, Alaa El-Habashi, Amr Shalaby, Krisjanis Karins, Everett Rowson, Robert MacIntyre, Paul Kingsbury, David Graff, and Cynthia McLemore. 1997. CALLHOME Egyptian Arabic transcripts LDC97T19. Web Download. Philadelphia: Linguistic Data Consortium. [find](https://scholar.google.com/scholar?q=Hassan+Gadalla%2C+Hanaa+Kilany%2C+Howaida+Arram%2C+Ashraf+Yacoub%2C+Alaa+El%2DHabashi%2C+Amr+Shalaby%2C+Krisjanis+Karins%2C+Everett+Rowson%2C+Robert+MacIntyre%2C+Paul+Kingsbury%2C+David+Graff%2C+and+Cynthia+McLemore%2E+1997%2E+C)
28. John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129–4138, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1419) [our card](https://md.research.elamur.ai/free/en/papers/hewitt-2019-structural-probe-finding-syntax-word/)
29. Mustafa Jarrar, Nizar Habash, Faeq Alrimawi, Diyam Akra, and Nasser Zalmout. 2016. Curras: an annotated corpus for the Palestinian Arabic dialect. Language Resources and Evaluation, pages 1–31. [find](https://scholar.google.com/scholar?q=Mustafa+Jarrar%2C+Nizar+Habash%2C+Faeq+Alrimawi%2C+Diyam+Akra%2C+and+Nasser+Zalmout%2E+2016%2E+Curras%3A+an+annotated+corpus+for+the+Palestinian+Arabic+dialect%2E+Language+Resources+and+Evaluation%2C+pages+1%E2%80%9331%2E)
30. Ganesh Jawahar, Benoı̂t Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651–3657, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ganesh+Jawahar%2C+Beno%C4%B1%CC%82t+Sagot%2C+and+Djame%CC%81+Seddah%2E+2019%2E+What+does+BERT+learn+about+the+structure+of+language%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics) [our card](https://md.research.elamur.ai/free/en/papers/jawahar-2019-what-does-bert-learn-about/)
31. Salam Khalifa, Nizar Habash, Fadhl Eryani, Ossama Obeid, Dana Abdulrahim, and Meera Al Kaabi. 2018. A morphologically annotated corpus of Emirati Arabic. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Salam+Khalifa%2C+Nizar+Habash%2C+Fadhl+Eryani%2C+Ossama+Obeid%2C+Dana+Abdulrahim%2C+and+Meera+Al+Kaabi%2E+2018%2E+A+morphologically+annotated+corpus+of+Emirati+Arabic%2E+In+Proceedings+of+the+Eleventh+International+C)
32. Artur Kulmizev, Vinit Ravishankar, Mostafa Abdou, and Joakim Nivre. 2020. Do neural language models show preferences for syntactic formalisms? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4077– 4091, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Artur+Kulmizev%2C+Vinit+Ravishankar%2C+Mostafa+Abdou%2C+and+Joakim+Nivre%2E+2020%2E+Do+neural+language+models+show+preferences+for+syntactic+formalisms%3F+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Associat)
33. Wuwei Lan, Yang Chen, Wei Xu, and Alan Ritter. 2020. An empirical study of pre-trained transformers for Arabic information extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4727–4734, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Wuwei+Lan%2C+Yang+Chen%2C+Wei+Xu%2C+and+Alan+Ritter%2E+2020%2E+An+empirical+study+of+pre%2Dtrained+transformers+for+Arabic+information+extraction%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Nat)
34. Linguistic Data Consortium. 2002. CALL-HOME Egyptian Arabic Transcripts Supplement LDC2002T38. [find](https://scholar.google.com/scholar?q=Linguistic+Data+Consortium%2E+2002%2E+CALL%2DHOME+Egyptian+Arabic+Transcripts+Supplement+LDC2002T38%2E)
35. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019a. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1073–1094, Minneapolis, Minnesota. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nelson+F%2E+Liu%2C+Matt+Gardner%2C+Yonatan+Belinkov%2C+Matthew+E%2E+Peters%2C+and+Noah+A%2E+Smith%2E+2019a%2E+Linguistic+knowledge+and+transferability+of+contextual+representations%2E+In+Proceedings+of+the+2019+Conferenc) [our card](https://md.research.elamur.ai/free/en/papers/liu-2019-linguistic-knowledge-transferability-contextual-representations/)
36. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. RoBERTa: A robustly optimized BERT pretraining approach. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019b%2E+RoBERTa%3A+A+robustly+optimized+BERT+pretraining+approach%2E)
37. Mohamed Maamouri, Ann Bies, Tim Buckwalter, and Wigdan Mekki. 2004. The Penn Arabic Treebank: Building a Large-Scale Annotated Arabic Corpus. In Proceedings of the International Conference on Arabic Language Resources and Tools, pages 102– 109, Cairo, Egypt. [find](https://scholar.google.com/scholar?q=Mohamed+Maamouri%2C+Ann+Bies%2C+Tim+Buckwalter%2C+and+Wigdan+Mekki%2E+2004%2E+The+Penn+Arabic+Treebank%3A+Building+a+Large%2DScale+Annotated+Arabic+Corpus%2E+In+Proceedings+of+the+International+Conference+on+Arabic+L)
38. Mohamed Maamouri, Ann Bies, Seth Kulick, Dalila Tabessi, and Sondos Krouna. 2012. Egyptian Arabic Treebank DF Parts 1-8 V2.0 - LDC catalog numbers LDC2012E93, LDC2012E98, LDC2012E89, LDC2012E99, LDC2012E107, LDC2012E125, LDC2013E12, LDC2013E21. [find](https://scholar.google.com/scholar?q=Mohamed+Maamouri%2C+Ann+Bies%2C+Seth+Kulick%2C+Dalila+Tabessi%2C+and+Sondos+Krouna%2E+2012%2E+Egyptian+Arabic+Treebank+DF+Parts+1%2D8+V2%2E0+%2D+LDC+catalog+numbers+LDC2012E93%2C+LDC2012E98%2C+LDC2012E89%2C+LDC2012E99%2C+LDC20)
39. Mohamed Maamouri, Tim Buckwalter, David Graff, and Hubert Jin. 2006. Levantine Arabic QT Training Data Set 5, Transcripts LDC2006T07. [find](https://scholar.google.com/scholar?q=Mohamed+Maamouri%2C+Tim+Buckwalter%2C+David+Graff%2C+and+Hubert+Jin%2E+2006%2E+Levantine+Arabic+QT+Training+Data+Set+5%2C+Transcripts+LDC2006T07%2E)
40. Mohamed Maamouri, Tim Buckwalter, David Graff, and Hubert Jin. 2007. Fisher Levantine Arabic Conversational Telephone Speech, Transcripts LDC2007T04. [find](https://scholar.google.com/scholar?q=Mohamed+Maamouri%2C+Tim+Buckwalter%2C+David+Graff%2C+and+Hubert+Jin%2E+2007%2E+Fisher+Levantine+Arabic+Conversational+Telephone+Speech%2C+Transcripts+LDC2007T04%2E)
41. Mohamed Maamouri, Tim Buckwalter, and Hubert Jin. 2005. Levantine Arabic QT Training Data Set 4 (Speech + Transcripts) LDC2005S14. [find](https://scholar.google.com/scholar?q=Mohamed+Maamouri%2C+Tim+Buckwalter%2C+and+Hubert+Jin%2E+2005%2E+Levantine+Arabic+QT+Training+Data+Set+4+%28Speech+%2B+Transcripts%29+LDC2005S14%2E)
42. Karima Meftouh, Salima Harrat, Salma Jamoussi, Mourad Abbas, and Kamel Smaili. 2015. Machine translation experiments on PADIC: A parallel Arabic dialect corpus. In Proceedings of the Pacific Asia Conference on Language, Information and Computation. [find](https://scholar.google.com/scholar?q=Karima+Meftouh%2C+Salima+Harrat%2C+Salma+Jamoussi%2C+Mourad+Abbas%2C+and+Kamel+Smaili%2E+2015%2E+Machine+translation+experiments+on+PADIC%3A+A+parallel+Arabic+dialect+corpus%2E+In+Proceedings+of+the+Pacific+Asia+Conf)
43. Vincent Micheli, Martin d’Hoffschmidt, and François Fleuret. 2020. On the importance of pre-training data volume for compact language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7853–7858, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Vincent+Micheli%2C+Martin+d%E2%80%99Hoffschmidt%2C+and+Franc%CC%A7ois+Fleuret%2E+2020%2E+On+the+importance+of+pre%2Dtraining+data+volume+for+compact+language+models%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Method)
44. Mahmoud Nabil, Mohamed Aly, and Amir Atiya. 2015. ASTD: Arabic sentiment tweets dataset. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2515– 2519, Lisbon, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mahmoud+Nabil%2C+Mohamed+Aly%2C+and+Amir+Atiya%2E+2015%2E+ASTD%3A+Arabic+sentiment+tweets+dataset%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+2515%E2%80%93+2519%2C+Li) [our card](https://md.research.elamur.ai/free/en/papers/nabil-2015-astd-arabic-sentiment-tweets-dataset/)
45. Lorenz Nigst, Maxim Romanov, Sarah Bowen Savant, Masoumeh Seydi, and Peter Verkinderen. 2020. OpenITI: a Machine-Readable Corpus of Islamicate Texts. [find](https://scholar.google.com/scholar?q=Lorenz+Nigst%2C+Maxim+Romanov%2C+Sarah+Bowen+Savant%2C+Masoumeh+Seydi%2C+and+Peter+Verkinderen%2E+2020%2E+OpenITI%3A+a+Machine%2DReadable+Corpus+of+Islamicate+Texts%2E)
46. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL tools: An open source python toolkit for Arabic natural language processing. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 7022–7032, Marseille, France. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Ossama+Obeid%2C+Nasser+Zalmout%2C+Salam+Khalifa%2C+Dima+Taji%2C+Mai+Oudah%2C+Bashar+Alhafni%2C+Go+Inoue%2C+Fadhl+Eryani%2C+Alexander+Erdmann%2C+and+Nizar+Habash%2E+2020%2E+CAMeL+tools%3A+An+open+source+python+toolkit+for+Ara) [our card](https://md.research.elamur.ai/free/en/papers/obeid-2020-camel-tools-open-source-python/)
47. Pedro Javier Ortiz Suárez, Laurent Romary, and Benoı̂t Sagot. 2020. A monolingual approach to contextualized word embeddings for mid-resource languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1703–1714, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Pedro+Javier+Ortiz+Sua%CC%81rez%2C+Laurent+Romary%2C+and+Beno%C4%B1%CC%82t+Sagot%2E+2020%2E+A+monolingual+approach+to+contextualized+word+embeddings+for+mid%2Dresource+languages%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+t) [our card](https://md.research.elamur.ai/free/en/papers/suarez-2020-monolingual-approach-contextualized-word-embeddings/)
48. Robert Parker, David Graff, Ke Chen, Junbo Kong, and Kazuaki Maeda. 2011. Arabic Gigaword Fifth Edition. LDC catalog number No. LDC2011T11, ISBN 1-58563-595-2. [find](https://scholar.google.com/scholar?q=Robert+Parker%2C+David+Graff%2C+Ke+Chen%2C+Junbo+Kong%2C+and+Kazuaki+Maeda%2E+2011%2E+Arabic+Gigaword+Fifth+Edition%2E+LDC+catalog+number+No%2E+LDC2011T11%2C+ISBN+1%2D58563%2D595%2D2%2E)
49. Arfath Pasha, Mohamed Al-Badrashiny, Mona Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. MADAMIRA: A fast, comprehensive tool for morphological analysis and disambiguation of Arabic. In Proceedings of the Language Resources and Evaluation Conference (LREC), pages 1094–1101, Reykjavik, Iceland. [find](https://scholar.google.com/scholar?q=Arfath+Pasha%2C+Mohamed+Al%2DBadrashiny%2C+Mona+Diab%2C+Ahmed+El+Kholy%2C+Ramy+Eskander%2C+Nizar+Habash%2C+Manoj+Pooleery%2C+Owen+Rambow%2C+and+Ryan+Roth%2E+2014%2E+MADAMIRA%3A+A+fast%2C+comprehensive+tool+for+morphological+an) [our card](https://md.research.elamur.ai/free/en/papers/lingbulk-2014-pasha-madamira-fast-comprehensive-tool-morphological/)
50. Raytheon BBN Technologies, Linguistic Data Consortium, and Sakhr Software. 2012. Arabic-Dialect/English Parallel Text LDC2012T09. [find](https://scholar.google.com/scholar?q=Raytheon+BBN+Technologies%2C+Linguistic+Data+Consortium%2C+and+Sakhr+Software%2E+2012%2E+Arabic%2DDialect%2FEnglish+Parallel+Text+LDC2012T09%2E)
51. Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. Semeval-2017 task 4: Sentiment analysis in twitter. In Proceedings of the International Workshop on Semantic Evaluation (SemEval), pages 501–516, Vancouver, Canada. [find](https://scholar.google.com/scholar?q=Sara+Rosenthal%2C+Noura+Farra%2C+and+Preslav+Nakov%2E+2017%2E+Semeval%2D2017+task+4%3A+Sentiment+analysis+in+twitter%2E+In+Proceedings+of+the+International+Workshop+on+Semantic+Evaluation+%28SemEval%29%2C+pages+501%E2%80%93516%2C+) [our card](https://md.research.elamur.ai/free/en/papers/rosenthal-2017-semeval-2017-task-4-sentiment/)
52. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2020. How good is your tokenizer? on the monolingual performance of multilingual language models. [find](https://scholar.google.com/scholar?q=Phillip+Rust%2C+Jonas+Pfeiffer%2C+Ivan+Vulic%CC%81%2C+Sebastian+Ruder%2C+and+Iryna+Gurevych%2E+2020%2E+How+good+is+your+tokenizer%3F+on+the+monolingual+performance+of+multilingual+language+models%2E) [our card](https://md.research.elamur.ai/free/en/papers/rust-2021-how-good-is-your-tokenizer/)
53. Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. KUISAIL at SemEval-2020 task 12: BERT-CNN for offensive speech identification in social media. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2054–2059, Barcelona (online). International Committee for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ali+Safaya%2C+Moutasem+Abdullatif%2C+and+Deniz+Yuret%2E+2020%2E+KUISAIL+at+SemEval%2D2020+task+12%3A+BERT%2DCNN+for+offensive+speech+identification+in+social+media%2E+In+Proceedings+of+the+Fourteenth+Workshop+on+Sema) [our card](https://md.research.elamur.ai/free/en/papers/safaya-2020-kuisail-semeval-2020-task-12/)
54. Ahmed Salama, Houda Bouamor, Behrang Mohit, and Kemal Oflazer. 2014. YouDACC: the Youtube Dialectal Arabic Comment Corpus. In Proceedings of the Language Resources and Evaluation Conference (LREC), pages 1246–1251, Reykjavik, Iceland. [find](https://scholar.google.com/scholar?q=Ahmed+Salama%2C+Houda+Bouamor%2C+Behrang+Mohit%2C+and+Kemal+Oflazer%2E+2014%2E+YouDACC%3A+the+Youtube+Dialectal+Arabic+Comment+Corpus%2E+In+Proceedings+of+the+Language+Resources+and+Evaluation+Conference+%28LREC%29%2C+pa)
55. Mohammad Salameh, Houda Bouamor, and Nizar Habash. 2018. Fine-grained Arabic dialect identification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1332–1344, Santa Fe, New Mexico, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mohammad+Salameh%2C+Houda+Bouamor%2C+and+Nizar+Habash%2E+2018%2E+Fine%2Dgrained+Arabic+dialect+identification%2E+In+Proceedings+of+the+27th+International+Conference+on+Computational+Linguistics%2C+pages+1332%E2%80%931344%2C+) [our card](https://md.research.elamur.ai/free/en/papers/salameh-2018-fine-grained-arabic-dialect-identification/)
56. Mike Schuster and Kaisuke Nakajima. 2012. Japanese and Korean voice search. In International Conference on Acoustics, Speech and Signal Processing, pages 5149–5152. [find](https://scholar.google.com/scholar?q=Mike+Schuster+and+Kaisuke+Nakajima%2E+2012%2E+Japanese+and+Korean+voice+search%2E+In+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing%2C+pages+5149%E2%80%935152%2E)
57. Bashar Talafha, Mohammad Ali, Muhy Eddin Za’ter, Haitham Seelawi, Ibraheem Tuffaha, Mostafa Samir, Wael Farhan, and Hussein Al-Natsheh. 2020. Multi-dialect Arabic BERT for country-level dialect identification. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 111–118, Barcelona, Spain (Online). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Bashar+Talafha%2C+Mohammad+Ali%2C+Muhy+Eddin+Za%E2%80%99ter%2C+Haitham+Seelawi%2C+Ibraheem+Tuffaha%2C+Mostafa+Samir%2C+Wael+Farhan%2C+and+Hussein+Al%2DNatsheh%2E+2020%2E+Multi%2Ddialect+Arabic+BERT+for+country%2Dlevel+dialect+identi)
58. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019a. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593– 4601, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ian+Tenney%2C+Dipanjan+Das%2C+and+Ellie+Pavlick%2E+2019a%2E+BERT+rediscovers+the+classical+NLP+pipeline%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+4593%E2%80%93) [our card](https://md.research.elamur.ai/free/en/papers/tenney-2019-bert-rediscovers-classical-nlp-pipeline/)
59. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019b. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Ian+Tenney%2C+Patrick+Xia%2C+Berlin+Chen%2C+Alex+Wang%2C+Adam+Poliak%2C+R+Thomas+McCoy%2C+Najoung+Kim%2C+Benjamin+Van+Durme%2C+Sam+Bowman%2C+Dipanjan+Das%2C+and+Ellie+Pavlick%2E+2019b%2E+What+do+you+learn+from+context%3F+probi) [our card](https://md.research.elamur.ai/free/en/papers/tenney-2019-learn-context-probing-sentence-structure/)
60. Ivan Vulić, Edoardo Maria Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. 2020. Probing pretrained language models for lexical semantics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7222–7240, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ivan+Vulic%CC%81%2C+Edoardo+Maria+Ponti%2C+Robert+Litschko%2C+Goran+Glavas%CC%8C%2C+and+Anna+Korhonen%2E+2020%2E+Probing+pretrained+language+models+for+lexical+semantics%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+) [our card](https://md.research.elamur.ai/free/en/papers/vulic-2020-probing-pretrained-language-models-lexical/)
61. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Huggingface’s transformers: State-of-the-art natural language processing. [find](https://scholar.google.com/scholar?q=Thomas+Wolf%2C+Lysandre+Debut%2C+Victor+Sanh%2C+Julien+Chaumond%2C+Clement+Delangue%2C+Anthony+Moi%2C+Pier%2Dric+Cistac%2C+Tim+Rault%2C+Re%CC%81mi+Louf%2C+Morgan+Funtowicz%2C+Joe+Davison%2C+Sam+Shleifer%2C+Patrick+von+Platen%2C+Clara) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
62. Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833–844, Hong Kong, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shijie+Wu+and+Mark+Dredze%2E+2019%2E+Beto%2C+bentz%2C+becas%3A+The+surprising+cross%2Dlingual+effectiveness+of+BERT%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+t) [our card](https://md.research.elamur.ai/free/en/papers/wu-2019-beto-bentz-becas-surprising-cross/)
63. Waleed A. Yousef, Omar M. Ibrahime, Taha M. Madbouly, and Moustafa A. Mahmoud. 2019. Learning meters of Arabic and English poems with recurrent neural networks: a step forward for language understanding and synthesis. [find](https://scholar.google.com/scholar?q=Waleed+A%2E+Yousef%2C+Omar+M%2E+Ibrahime%2C+Taha+M%2E+Madbouly%2C+and+Moustafa+A%2E+Mahmoud%2E+2019%2E+Learning+meters+of+Arabic+and+English+poems+with+recurrent+neural+networks%3A+a+step+forward+for+language+understandi)
64. Wajdi Zaghouani and Anis Charfi. 2018. ArapTweet: A Large Multi-Dialect Twitter Corpus for Gender, Age and Language Variety Identification. In Proceedings of the Language Resources and Evaluation Conference (LREC), Miyazaki, Japan. [find](https://scholar.google.com/scholar?q=Wajdi+Zaghouani+and+Anis+Charfi%2E+2018%2E+ArapTweet%3A+A+Large+Multi%2DDialect+Twitter+Corpus+for+Gender%2C+Age+and+Language+Variety+Identification%2E+In+Proceedings+of+the+Language+Resources+and+Evaluation+Conf)
65. Omar F Zaidan and Chris Callison-Burch. 2011. The Arabic Online Commentary Dataset: an Annotated Dataset of Informal Arabic With High Dialectal Content. In Proceedings of the Conference of the Association for Computational Linguistics (ACL), pages 37–41. [find](https://scholar.google.com/scholar?q=Omar+F+Zaidan+and+Chris+Callison%2DBurch%2E+2011%2E+The+Arabic+Online+Commentary+Dataset%3A+an+Annotated+Dataset+of+Informal+Arabic+With+High+Dialectal+Content%2E+In+Proceedings+of+the+Conference+of+the+Associa) [our card](https://md.research.elamur.ai/free/en/papers/zaidan-2011-arabic-online-commentary-dataset-annotated/)
66. Imad Zeroual, Dirk Goldhahn, Thomas Eckart, and Abdelhak Lakhouaja. 2019. OSIAN: Open source international Arabic news corpus - preparation and integration into the CLARIN-infrastructure. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 175–182, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Imad+Zeroual%2C+Dirk+Goldhahn%2C+Thomas+Eckart%2C+and+Abdelhak+Lakhouaja%2E+2019%2E+OSIAN%3A+Open+source+international+Arabic+news+corpus+%2D+preparation+and+integration+into+the+CLARIN%2Dinfrastructure%2E+In+Proceedin)
67. Yian Zhang, Alex Warstadt, Haau-Sing Li, and Samuel R. Bowman. 2020. When do you need billions of words of pretraining data? [find](https://scholar.google.com/scholar?q=Yian+Zhang%2C+Alex+Warstadt%2C+Haau%2DSing+Li%2C+and+Samuel+R%2E+Bowman%2E+2020%2E+When+do+you+need+billions+of+words+of+pretraining+data%3F)