---
title: "AraBERT: Transformer-based Model for Arabic — Antoun 2020"
description: "Arabic has rich morphology, limited NLP resources and little in common with Latin-script languages."
authors: ["Wissam Antoun", "Fady Baly", "Hazem Hajj"]
year: 2020
venue: "arXiv:2003.00104"
source: "https://arxiv.org/abs/2003.00104"
doi: "10.48550/arxiv.2003.00104"
arxiv: "2003.00104"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/) / AraBERT: Transformer-based Model for Arabic Language Understanding

# AraBERT: Transformer-based Model for Arabic Language Understanding

**Authors:** Wissam Antoun, Fady Baly, [Hazem Hajj](https://md.research.elamur.ai/free/en/authors/hazem-hajj/) · **Year:** 2020 · **Venue:** arXiv:2003.00104 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2003.00104) · [PDF](https://arxiv.org/pdf/2003.00104) · [DOI 10.48550/arxiv.2003.00104](https://doi.org/10.48550/arxiv.2003.00104) · [arXiv 2003.00104](https://arxiv.org/abs/2003.00104) · [OpenAlex](https://openalex.org/W3008110149)

**Citations:** 680 citations (openalex-snapshot · 2026-10-14) · [617](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary

Arabic has rich morphology, limited NLP resources and little in common with Latin-script languages. As a result, multilingual models such as Google's multilingual BERT represent it poorly. The authors address this by training a BERT model only on Arabic, called AraBERT. It uses the standard BERT-base architecture and two pre-training objectives: Masked Language Modeling with whole-word masking, and Next Sentence Prediction. The training data is about 24GB of deduplicated text, combining news articles scraped by the authors with two public collections, the 1.5 billion words Arabic Corpus and OSIAN.

Arabic often attaches prefixes such as the article "Al-" directly to words, which can fill a vocabulary with near-duplicate tokens. To reduce this, the text is first split into stems and affixes with Farasa, and a SentencePiece vocabulary is then trained on the segmented text. A second variant, AraBERTv0.1, skips segmentation for comparison.

AraBERT was evaluated on three downstream tasks (classifying sentiment, identifying named entities, and answering questions) with datasets covering both Modern Standard Arabic and dialects. It beat multilingual BERT and reached state-of-the-art results on most benchmarks. The models are publicly released.

## Key points

- **Architecture:** BERT-base with 12 encoder layers, 768 hidden units, 12 attention heads, sequences up to 512 tokens, and roughly 110M parameters. In masking, 15% of tokens are selected, and these are replaced with [MASK], a random token or the original token in an 80/10/10 split.
- **Pre-training corpus:** about 70 million sentences (~24GB). Sources are scraped Arabic news, the 1.5 billion words Arabic Corpus (El-Khair, 2016; over 5M articles, 10 sources, 8 countries) and OSIAN (3.5M articles, ~1B tokens, 31 sources, 24 Arab countries). Words containing Latin characters were kept.
- **Tokenization:** Farasa segmentation followed by SentencePiece in unigram mode, giving about 60K subwords. The final vocabulary has 64K entries, including about 4K unused slots reserved for later pre-training. AraBERTv0.1 is trained on unsegmented text.
- **Evaluation:** tasks are SA, NER (IOB2 tagging, using only the first sub-token of each word) and QA (predicting answer start and end positions). The SA datasets include HARD, which has 93,700 hotel reviews in MSA and dialects.
- **Release:** the models are available at github.com/aub-mind/araBERT and through popular NLP libraries.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

617 papers of our library cite it (of the 140884 whose bibliographies we read); 157 of them have a page here, the first 100 below, 460 not yet.

- [Arabic Dialect Classification using RNNs, Transformers, and Large Language Models: A Comparative Analysis](https://md.research.elamur.ai/free/en/papers/a-2025-essameldin-arabic-dialect-classification-using-rnns/)
- [DziriBERT: a Pre-trained Language Model for the Algerian Dialect](https://md.research.elamur.ai/free/en/papers/abdaoui-2021-dziribert-pre-trained-language-model/)
- [Arabic Dialect Identification in the Wild](https://md.research.elamur.ai/free/en/papers/abdelali-2020-arabic-dialect-identification-wild/)
- [Pre-Training BERT on Arabic Tweets: Practical Considerations](https://md.research.elamur.ai/free/en/papers/abdelali-2021-pre-training-bert-arabic-tweets/)
- [ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic](https://md.research.elamur.ai/free/en/papers/abdul-2021-mageed-arbert-marbert-deep-bidirectional-transformers/)
- [Detecting Suicidality in Arabic Tweets Using Machine Learning and Deep Learning Techniques](https://md.research.elamur.ai/free/en/papers/abdulsalam-2023-detecting-suicidality-arabic-tweets-using/)
- [The AraGenEval Shared Task on Arabic Authorship Style Transfer and AI Generated Text Detection](https://md.research.elamur.ai/free/en/papers/abudalfa-2025-arageneval-shared-task-arabic-authorship/)
- [Sentiment Analysis in Poems in Misurata Sub-dialect -- A Sentiment Detection in an Arabic Sub-dialect](https://md.research.elamur.ai/free/en/papers/abugharsa-2021-sentiment-analysis-poems-misurata-sub/)
- [TTLab at AlexandriaX-2026: A Fine-Tuned Surface Tagger for Arabic Machine-Translation Error-Span Detection and Classification](https://md.research.elamur.ai/free/en/papers/abusaleh-2026-ttlab-alexandriax-2026-fine-tuned/)
- [Improving Arabic Diacritization with Regularized Decoding and Adversarial Training](https://md.research.elamur.ai/free/en/papers/acl-2021-qin-improving-arabic-diacritization-regularized-decoding/)
- [AlephBERT: Language Model Pre-training and Evaluation from Sub-Word to Sentence Level](https://md.research.elamur.ai/free/en/papers/acl-2022-seker-alephbert-language-model-pre-training/)
- [A Transformer-based parser for Syriac morphology](https://md.research.elamur.ai/free/en/papers/aclbulk-2023-naaijer-transformer-based-parser-syriac-morphology/)
- [Do Diacritics Matter? Evaluating the Impact of Arabic Diacritics on Tokenization and LLM Benchmarks](https://md.research.elamur.ai/free/en/papers/aclbulk-2026-inoue-do-diacritics-matter-evaluating-impact/)
- [AlcLaM: Arabic Dialectal Language Model](https://md.research.elamur.ai/free/en/papers/ahmed-2024-alclam-arabic-dialectal-language-model/)
- [Leveraging BERT Language Model for Arabic Long Document Classification](https://md.research.elamur.ai/free/en/papers/al-2023-qurishi-leveraging-bert-language-model-arabic/)
- [Cross-Lingual SynthDocs: A Large-Scale Synthetic Corpus for Any to Arabic OCR and Document Understanding](https://md.research.elamur.ai/free/en/papers/al-2025-homoud-cross-lingual-synthdocs-large-scale/)
- [The Landscape of Arabic Large Language Models (ALLMs): A New Era for Arabic Language Technology](https://md.research.elamur.ai/free/en/papers/al-2025-khalifa-landscape-arabic-large-language-models/)
- [Evaluation of Adversarial Robustness in Arabic Language Models](https://md.research.elamur.ai/free/en/papers/alajmi-2026-evaluation-adversarial-robustness-arabic-language/)
- [Propaganda to Hate: A Multimodal Analysis of Arabic Memes with Multi-Agent LLMs](https://md.research.elamur.ai/free/en/papers/alam-2024-propaganda-hate-multimodal-analysis-arabic/)
- [High Tech team at SemEval-2022 Task 6: Intended Sarcasm Detection for Arabic texts](https://md.research.elamur.ai/free/en/papers/alami-2022-high-tech-team-semeval-2022/)
- [EHSAN: Leveraging ChatGPT in a Hybrid Framework for Arabic Aspect-Based Sentiment Analysis in Healthcare](https://md.research.elamur.ai/free/en/papers/alamoudi-2025-ehsan-leveraging-chatgpt-hybrid-framework/)
- [Overview of the Arabic Sentiment Analysis 2021 Competition at KAUST](https://md.research.elamur.ai/free/en/papers/alamro-2021-overview-arabic-sentiment-analysis-2021/)
- [Multi-task Learning with Active Learning for Arabic Offensive Speech Detection](https://md.research.elamur.ai/free/en/papers/alansari-2025-multi-task-learning-active-learning/)
- [Detecting Hope, Hate, and Emotion in Arabic Textual Speech and Multi-modal Memes Using Large Language Models](https://md.research.elamur.ai/free/en/papers/aldahoul-2025-detecting-hope-hate-emotion-arabic/)
- [Comparing Open Arabic Named Entity Recognition Tools](https://md.research.elamur.ai/free/en/papers/aldumaykhi-2022-comparing-open-arabic-named-entity/)
- [Impact of emoji exclusion on the performance of Arabic sarcasm detection models](https://md.research.elamur.ai/free/en/papers/aleryani-2024-impact-emoji-exclusion-performance-arabic/)
- [GHAD NLP at AdabEval2026: Transformer-Based Approach for Arabic Politeness and Pragmatic Category Classification](https://md.research.elamur.ai/free/en/papers/alfattni-2026-ghad-nlp-adabeval2026-transformer-based/)
- [Domain Adaptation for Arabic Machine Translation: The Case of Financial Texts](https://md.research.elamur.ai/free/en/papers/alghamdi-2023-domain-adaptation-arabic-machine-translation/)
- [Advancements in Arabic Grammatical Error Detection and Correction: An Empirical Investigation](https://md.research.elamur.ai/free/en/papers/alhafni-2023-advancements-arabic-grammatical-error-detection/)
- [ASAD: A Twitter-based Benchmark Arabic Sentiment Analysis Dataset](https://md.research.elamur.ai/free/en/papers/alharbi-2020-asad-twitter-based-benchmark-arabic/)
- [MentalMARBERT: Domain-Adaptive Pre-training and Two-Stage Fine-Tuning for Arabic Mental Health Disorders Detection](https://md.research.elamur.ai/free/en/papers/almalki-2026-mentalmarbert-domain-adaptive-pre-training/)
- [ArMIS - The Arabic Misogyny and Sexism Corpus with Annotator Subjective Disagreements](https://md.research.elamur.ai/free/en/papers/almanea-2022-armis-arabic-misogyny-sexism-corpus/)
- [BARRAC: Adaptation of an English Aspect-based Sentiment Analysis Approach for Classification Tasks in Arabic Dialects](https://md.research.elamur.ai/free/en/papers/almutairi-2026-barrac-adaptation-english-aspect-based/)
- [Harnessing Multilingual Resources to Question Answering in Arabic](https://md.research.elamur.ai/free/en/papers/alnajjar-2022-harnessing-multilingual-resources-question-answering/)
- [Neural Coreference Resolution for Arabic](https://md.research.elamur.ai/free/en/papers/aloraini-2020-neural-coreference-resolution-arabic/)
- [Spam and Sentiment Detection in Arabic Tweets Using MARBERT Model](https://md.research.elamur.ai/free/en/papers/alotaibi-2026-spam-sentiment-detection-arabic-tweets/)
- [Are Arabic Benchmarks Reliable? QIMMA's Quality-First Approach to LLM Evaluation](https://md.research.elamur.ai/free/en/papers/alqadi-2026-arabic-benchmarks-reliable-qimma-s/)
- [Understanding the Sociocultural Dimensions of Mental Health Discourse in Arabic-Language X Communities](https://md.research.elamur.ai/free/en/papers/alqahtani-2026-understanding-sociocultural-dimensions-mental-health-discourse/)
- [Assessing the Linguistic Knowledge in Arabic Pre-trained Language Models Using Minimal Pairs](https://md.research.elamur.ai/free/en/papers/alrajhi-2022-assessing-linguistic-knowledge-arabic-pre/)
- [ArbESC+: Arabic Enhanced Edit Selection System Combination for Grammatical Error Correction Resolving conflict and improving system combination in Arabic GEC](https://md.research.elamur.ai/free/en/papers/alrehili-2025-arbesc-arabic-enhanced-edit-selection/)
- [Towards the Development of Balanced Synthetic Data for Correcting Grammatical Errors in Arabic: An Approach Based on Error Tagging Model and Synthetic Data Generating Model](https://md.research.elamur.ai/free/en/papers/alrehili-2025-towards-development-balanced-synthetic-data/)
- [A Novel Dialect-Aware Framework for the Classification of Arabic Dialects and Emotions](https://md.research.elamur.ai/free/en/papers/alsadhan-2025-novel-dialect-aware-framework-classification/)
- [Is AI Catching Up to Human Expression? Exploring Emotion, Personality, Authorship, and Linguistic Style in English and Arabic with Six Large Language Models](https://md.research.elamur.ai/free/en/papers/alsadhan-2026-ai-catching-up-human-expression/)
- [Rasid at StanceEval: Fine-tuning MARBERT for Arabic Stance Detection](https://md.research.elamur.ai/free/en/papers/alshenaifi-2024-rasid-stanceeval-fine-tuning-marbert/)
- [Optimizing RAG Pipelines for Arabic: A Systematic Analysis of Core Components](https://md.research.elamur.ai/free/en/papers/alsubhi-2025-optimizing-rag-pipelines-arabic-systematic/)
- [ArPanEmo: An Open-Source Dataset for Fine-Grained Emotion Recognition in Arabic Online Content during COVID-19 Pandemic](https://md.research.elamur.ai/free/en/papers/althobaiti-2023-arpanemo-open-source-dataset-fine/)
- [StanceEval 2024: The First Arabic Stance Detection Shared Task](https://md.research.elamur.ai/free/en/papers/alturayeif-2024-stanceeval-2024-first-arabic-stance/)
- [Evaluating Various Tokenizers for Arabic Text Classification](https://md.research.elamur.ai/free/en/papers/alyafeai-2021-evaluating-various-tokenizers-arabic-text/)
- [Masader: Metadata Sourcing for Arabic Text and Speech Data Resources](https://md.research.elamur.ai/free/en/papers/alyafeai-2022-masader-metadata-sourcing-arabic-text/)
- [Investigating Persuasion Techniques in Arabic: An Empirical Study Leveraging Large Language Models](https://md.research.elamur.ai/free/en/papers/alzahrani-2024-investigating-persuasion-techniques-arabic-empirical/)
- [AraCOVID19-MFH: Arabic COVID-19 Multi-label Fake News and Hate Speech Detection Dataset](https://md.research.elamur.ai/free/en/papers/ameur-2021-aracovid19-mfh-arabic-covid-19/)
- [AraCOVID19-SSD: Arabic COVID-19 Sentiment and Sarcasm Detection Dataset](https://md.research.elamur.ai/free/en/papers/ameur-2021-aracovid19-ssd-arabic-covid-19/)
- [Multilingual Nonce Dependency Treebanks: Understanding how Language Models Represent and Process Syntactic Structure](https://md.research.elamur.ai/free/en/papers/arps-2024-multilingual-nonce-dependency-treebanks-understanding/)
- [Improving Arabic Multi-Label Emotion Classification using Stacked Embeddings and Hybrid Loss Function](https://md.research.elamur.ai/free/en/papers/aslam-2024-improving-arabic-multi-label-emotion/)
- [Arabic Sentiment Analysis with Noisy Deep Explainable Model](https://md.research.elamur.ai/free/en/papers/atabuzzaman-2023-arabic-sentiment-analysis-noisy-deep/)
- [Optimized Quran Passage Retrieval Using an Expanded QA Dataset and Fine-Tuned Language Models](https://md.research.elamur.ai/free/en/papers/basem-2024-optimized-quran-passage-retrieval-using/)
- [ArSarcasm Shared Task: An Ensemble BERT Model for SarcasmDetection in Arabic Tweets](https://md.research.elamur.ai/free/en/papers/bashmal-2021-arsarcasm-shared-task-ensemble-bert/)
- [Enhanced Arabic Text Retrieval with Attentive Relevance Scoring](https://md.research.elamur.ai/free/en/papers/bekhouche-2025-enhanced-arabic-text-retrieval-attentive/)
- [Dhati+: Fine-tuned Large Language Models for Arabic Subjectivity Evaluation](https://md.research.elamur.ai/free/en/papers/bellaouar-2025-dhati-fine-tuned-large-language/)
- [BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla](https://md.research.elamur.ai/free/en/papers/bhattacharjee-2022-banglabert-language-model-pretraining-benchmarks/)
- [Beyond MCQ: An Open-Ended Arabic Cultural QA Benchmark with Dialect Variants](https://md.research.elamur.ai/free/en/papers/bhatti-2026-beyond-mcq-open-ended-arabic/)
- [Personality Analysis for Social Media Users using Arabic language and its Effect on Sentiment Analysis](https://md.research.elamur.ai/free/en/papers/dandash-2024-personality-analysis-social-media-users/)
- [Improving Natural Language Inference in Arabic using Transformer Models and Linguistically Informed Pre-Training](https://md.research.elamur.ai/free/en/papers/deen-2023-improving-natural-language-inference-arabic/)
- [Extractive Summarization for Arabic Documents Using SAraBERT with a Semantic Siamese Similarity Evaluation Metric](https://md.research.elamur.ai/free/en/papers/deen-2026-extractive-summarization-arabic-documents-using/)
- [Nahw: A Comprehensive Benchmark of Arabic Grammar Understanding, Error Detection, Correction, and Explanation](https://md.research.elamur.ai/free/en/papers/eacl-2026-mubarak-nahw-comprehensive-benchmark-arabic-grammar/)
- [Exploring Retrieval Augmented Generation in Arabic](https://md.research.elamur.ai/free/en/papers/el-2024-beltagy-exploring-retrieval-augmented-generation-arabic/)
- [AraFinNews: Arabic Financial Summarisation with Domain-Adapted LLMs](https://md.research.elamur.ai/free/en/papers/el-2025-haj-arafinnews-arabic-financial-summarisation-domain/)
- [Lexicon-Enriched Graph Modeling for Arabic Document Readability Prediction](https://md.research.elamur.ai/free/en/papers/elchafei-2025-lexicon-enriched-graph-modeling-arabic/)
- [Sentiment Classification of Gaza War Headlines: A Comparative Analysis of Large Language Models and Arabic Fine-Tuned BERT Models](https://md.research.elamur.ai/free/en/papers/eleraqi-2026-sentiment-classification-gaza-war-headlines/)
- [MuDRiC: Multi-Dialect Reasoning for Arabic Commonsense Validation](https://md.research.elamur.ai/free/en/papers/elozeiri-2025-mudric-multi-dialect-reasoning-arabic/)
- [Overview of the WANLP 2021 Shared Task on Sarcasm and Sentiment Detection in Arabic](https://md.research.elamur.ai/free/en/papers/farha-2021-overview-wanlp-2021-shared-task/)
- [SemEval-2022 Task 6: iSarcasmEval, Intended Sarcasm Detection in English and Arabic](https://md.research.elamur.ai/free/en/papers/farha-2022-semeval-2022-task-6-isarcasmeval/)
- [Multilingual Sequence-to-Sequence Models for Hebrew NLP](https://md.research.elamur.ai/free/en/papers/findings-2023-eyal-multilingual-sequence-sequence-models-hebrew/)
- [Sarcasm and Sentiment Detection in Arabic language A Hybrid Approach Combining Embeddings and Rule-based Features](https://md.research.elamur.ai/free/en/papers/gaanoun-2021-sarcasm-sentiment-detection-arabic-language/)
- [Team_Zero at StanceEval2024: Frozen PLMs for Arabic Stance Detection](https://md.research.elamur.ai/free/en/papers/galal-2024-team-zero-stanceeval2024-frozen-plms/)
- [Revisiting Pre-trained Language Models and their Evaluation for Arabic Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/ghaddar-2022-revisiting-pre-trained-language-models-their/)
- [On the importance of Data Scale in Pretraining Arabic Language Models](https://md.research.elamur.ai/free/en/papers/ghaddar-2024-importance-data-scale-pretraining-arabic/)
- [Automated essay scoring in Arabic: a dataset and analysis of a BERT-based system](https://md.research.elamur.ai/free/en/papers/ghazawi-2024-automated-essay-scoring-arabic-dataset/)
- [Exploring Author Style in Nakba Short Stories: A Comparative Study of Transformer-Based Models](https://md.research.elamur.ai/free/en/papers/hamed-2025-exploring-author-style-nakba-short/)
- [Benchmarking Arabic Authorship Attribution and Style Transfer with Large Language Models](https://md.research.elamur.ai/free/en/papers/hamed-2026-benchmarking-arabic-authorship-attribution-style/)
- [New Arabic Medical Dataset for Diseases Classification](https://md.research.elamur.ai/free/en/papers/hammoud-2021-new-arabic-medical-dataset-diseases/)
- [SinaTools: Open Source Toolkit for Arabic Natural Language Processing](https://md.research.elamur.ai/free/en/papers/hammouda-2024-sinatools-open-source-toolkit-arabic/)
- [Detecting Stance of Authorities towards Rumors in Arabic Tweets: A Preliminary Study](https://md.research.elamur.ai/free/en/papers/haouari-2023-detecting-stance-authorities-towards-rumors/)
- [Towards Arabic Multimodal Dataset for Sentiment Analysis](https://md.research.elamur.ai/free/en/papers/haouhat-2023-towards-arabic-multimodal-dataset-sentiment/)
- [SMASH at StanceEval 2024: Prompt Engineering LLMs for Arabic Stance Detection](https://md.research.elamur.ai/free/en/papers/hariri-2024-smash-stanceeval-2024-prompt-engineering/)
- [AIN: The Arabic INclusive Large Multimodal Model](https://md.research.elamur.ai/free/en/papers/heakl-2025-ain-arabic-inclusive-large-multimodal/)
- [Combining Context-Free and Contextualized Representations for Arabic Sarcasm Detection and Sentiment Identification](https://md.research.elamur.ai/free/en/papers/hengle-2021-combining-context-free-contextualized-representations/)
- [Kuwain 1.5B: An Arabic SLM via Language Injection](https://md.research.elamur.ai/free/en/papers/hennara-2025-kuwain-1-5b-arabic-slm/)
- [Challenges and Strategies in Cross-Cultural NLP](https://md.research.elamur.ai/free/en/papers/hershcovich-2022-challenges-strategies-cross-cultural-nlp/)
- [The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models](https://md.research.elamur.ai/free/en/papers/inoue-2021-interplay-variant-size-task-type/)
- [The IDC System for Sentiment Classification and Sarcasm Detection in Arabic](https://md.research.elamur.ai/free/en/papers/israeli-2021-idc-system-sentiment-classification-sarcasm/)
- [Wojood: Nested Arabic Named Entity Corpus and Recognition using BERT](https://md.research.elamur.ai/free/en/papers/jarrar-2022-wojood-nested-arabic-named-entity/)
- [Optimizing the Size of Subword Vocabularies in Dialect Classification](https://md.research.elamur.ai/free/en/papers/kanjirangat-2023-optimizing-size-subword-vocabularies-dialect/)
- [Exploring Data and Parameter Efficient Strategies for Arabic Dialect Identifications](https://md.research.elamur.ai/free/en/papers/kanjirangat-2025-exploring-data-parameter-efficient-strategies/)
- [Learning the Arabic Dialect Continuum as a Continuous Space: A Regression Approach to Speaker Origin Prediction](https://md.research.elamur.ai/free/en/papers/khadraoui-2026-learning-arabic-dialect-continuum-continuous/)
- [ALHD: A Large-Scale and Multigenre Benchmark Dataset for Arabic LLM-Generated Text Detection](https://md.research.elamur.ai/free/en/papers/khairallah-2025-alhd-large-scale-multigenre-benchmark/)
- [ARAFA: An LLM-Generated Arabic Fact-Checking Dataset](https://md.research.elamur.ai/free/en/papers/khalil-2026-arafa-llm-generated-arabic-fact/)
- [Towards Arabic Sentence Simplification via Classification and Generative Approaches](https://md.research.elamur.ai/free/en/papers/khallaf-2022-arabic-sentence-simplification-classification-generative/)
- [ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic](https://md.research.elamur.ai/free/en/papers/koto-2024-arabicmmlu-assessing-massive-multitask-language/)
- [ArabianGPT: Native Arabic GPT-based Large Language Model](https://md.research.elamur.ai/free/en/papers/koubaa-2024-arabiangpt-native-arabic-gpt-based/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{antoun2020arabert,
  title = {{AraBERT: Transformer-based Model for Arabic Language Understanding}},
  author = {Antoun, Wissam and Baly, Fady and Hajj, Hazem},
  year = {2020},
  doi = {10.48550/arxiv.2003.00104},
  eprint = {2003.00104},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2003.00104}
}
```

- [← previous AR-ASAG An ARabic Dataset for Automatic Short Answer Grading Evaluation](https://md.research.elamur.ai/free/en/papers/ouahrani-2020-ar-asag-arabic-dataset-automatic/)
- [next → Arabic Diacritic Recovery Using a Feature-Rich biLSTM Model](https://md.research.elamur.ai/free/en/papers/darwish-2020-arabic-diacritic-recovery-using-feature/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/)

## References

49 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abdelali, A., Darwish, K., Durrani, N., and Mubarak, H. (2016). Farasa: A fast and furious segmenter for arabic. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 11–16. [find](https://scholar.google.com/scholar?q=Abdelali%2C+A%2E%2C+Darwish%2C+K%2E%2C+Durrani%2C+N%2E%2C+and+Mubarak%2C+H%2E+%282016%29%2E+Farasa%3A+A+fast+and+furious+segmenter+for+arabic%2E+In+Proceedings+of+the+2016+Conference+of+the+North+American+Chapter+of+the+Association+) [our card](https://md.research.elamur.ai/free/en/papers/aclbulk-2016-abdelali-farasa-fast-furious-segmenter-arabic/)
2. Abdul-Mageed, M., Alhuzali, H., and Elaraby, M. (2018). You tweet what you speak: A city-level dataset of arabic dialects. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). [find](https://scholar.google.com/scholar?q=Abdul%2DMageed%2C+M%2E%2C+Alhuzali%2C+H%2E%2C+and+Elaraby%2C+M%2E+%282018%29%2E+You+tweet+what+you+speak%3A+A+city%2Dlevel+dataset+of+arabic+dialects%2E+In+Proceedings+of+the+Eleventh+International+Conference+on+Language+Resources)
3. Abu Farha, I. and Magdy, W. (2019). Mazajak: An online Arabic sentiment analyser. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 192–198, Florence, Italy, August. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Abu+Farha%2C+I%2E+and+Magdy%2C+W%2E+%282019%29%2E+Mazajak%3A+An+online+Arabic+sentiment+analyser%2E+In+Proceedings+of+the+Fourth+Arabic+Natural+Language+Processing+Workshop%2C+pages+192%E2%80%93198%2C+Florence%2C+Italy%2C+August%2E+Asso)
4. Adiwardana, D., Luong, M.-T., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y., and Le, Q. V. (2020). Towards a human-like open-domain chatbot. [find](https://scholar.google.com/scholar?q=Adiwardana%2C+D%2E%2C+Luong%2C+M%2E%2DT%2E%2C+So%2C+D%2E+R%2E%2C+Hall%2C+J%2E%2C+Fiedel%2C+N%2E%2C+Thoppilan%2C+R%2E%2C+Yang%2C+Z%2E%2C+Kulshreshtha%2C+A%2E%2C+Nemade%2C+G%2E%2C+Lu%2C+Y%2E%2C+and+Le%2C+Q%2E+V%2E+%282020%29%2E+Towards+a+human%2Dlike+open%2Ddomain+chatbot%2E) [our card](https://md.research.elamur.ai/free/en/papers/adiwardana-2020-towards-human-like-open-domain/)
5. Al Sallab, A., Hajj, H., Badaro, G., Baly, R., El-Hajj, W., and Shaban, K. (2015). Deep learning models for sentiment analysis in arabic. In Proceedings of the second workshop on Arabic natural language processing, pages 9–17. [find](https://scholar.google.com/scholar?q=Al+Sallab%2C+A%2E%2C+Hajj%2C+H%2E%2C+Badaro%2C+G%2E%2C+Baly%2C+R%2E%2C+El%2DHajj%2C+W%2E%2C+and+Shaban%2C+K%2E+%282015%29%2E+Deep+learning+models+for+sentiment+analysis+in+arabic%2E+In+Proceedings+of+the+second+workshop+on+Arabic+natural+langua)
6. Al-Sallab, A., Baly, R., Hajj, H., Shaban, K. B., El-Hajj, W., and Badaro, G. (2017). Aroma: A recursive deep learning model for opinion mining in arabic as a low resource language. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 16(4):1–20. [find](https://scholar.google.com/scholar?q=Al%2DSallab%2C+A%2E%2C+Baly%2C+R%2E%2C+Hajj%2C+H%2E%2C+Shaban%2C+K%2E+B%2E%2C+El%2DHajj%2C+W%2E%2C+and+Badaro%2C+G%2E+%282017%29%2E+Aroma%3A+A+recursive+deep+learning+model+for+opinion+mining+in+arabic+as+a+low+resource+language%2E+ACM+Transactions+o)
7. Alomari, K. M., ElSherif, H. M., and Shaalan, K. (2017). Arabic tweets sentimental analysis using machine learning. In International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems, pages 602–610. Springer. [find](https://scholar.google.com/scholar?q=Alomari%2C+K%2E+M%2E%2C+ElSherif%2C+H%2E+M%2E%2C+and+Shaalan%2C+K%2E+%282017%29%2E+Arabic+tweets+sentimental+analysis+using+machine+learning%2E+In+International+Conference+on+Industrial%2C+Engineering+and+Other+Applications+of+App)
8. Aly, M. and Atiya, A. (2013). LABR: A large scale Arabic book reviews dataset. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 494–498, Sofia, Bulgaria, August. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Aly%2C+M%2E+and+Atiya%2C+A%2E+%282013%29%2E+LABR%3A+A+large+scale+Arabic+book+reviews+dataset%2E+In+Proceedings+of+the+51st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+2%3A+Short+Papers%29%2C+page)
9. Badaro, G., Baly, R., Hajj, H., Habash, N., and El-Hajj, W. (2014). A large scale arabic sentiment lexicon for arabic opinion mining. In Proceedings of the EMNLP 2014 workshop on arabic natural language processing (ANLP), pages 165–173. [find](https://scholar.google.com/scholar?q=Badaro%2C+G%2E%2C+Baly%2C+R%2E%2C+Hajj%2C+H%2E%2C+Habash%2C+N%2E%2C+and+El%2DHajj%2C+W%2E+%282014%29%2E+A+large+scale+arabic+sentiment+lexicon+for+arabic+opinion+mining%2E+In+Proceedings+of+the+EMNLP+2014+workshop+on+arabic+natural+langua)
10. Baly, R., Hajj, H., Habash, N., Shaban, K. B., and El-Hajj, W. (2017). A sentiment treebank and morphologically enriched recursive deep models for effective sentiment analysis in arabic. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 16(4):1–21. [find](https://scholar.google.com/scholar?q=Baly%2C+R%2E%2C+Hajj%2C+H%2E%2C+Habash%2C+N%2E%2C+Shaban%2C+K%2E+B%2E%2C+and+El%2DHajj%2C+W%2E+%282017%29%2E+A+sentiment+treebank+and+morphologically+enriched+recursive+deep+models+for+effective+sentiment+analysis+in+arabic%2E+ACM+Transacti)
11. Baly, R., Khaddaj, A., Hajj, H., El-Hajj, W., and Shaban, K. B. (2018). Arsentd-lev: A multi-topic corpus for target-based sentiment analysis in arabic levantine tweets. In OSACT 3: The 3rd Workshop on Open-Source Arabic Corpora and Processing Tools, page 37. [find](https://scholar.google.com/scholar?q=Baly%2C+R%2E%2C+Khaddaj%2C+A%2E%2C+Hajj%2C+H%2E%2C+El%2DHajj%2C+W%2E%2C+and+Shaban%2C+K%2E+B%2E+%282018%29%2E+Arsentd%2Dlev%3A+A+multi%2Dtopic+corpus+for+target%2Dbased+sentiment+analysis+in+arabic+levantine+tweets%2E+In+OSACT+3%3A+The+3rd+Workshop+o) [our card](https://md.research.elamur.ai/free/en/papers/baly-2019-arsentd-lev-multi-topic-corpus/)
12. Benajiba, Y. and Rosso, P. (2007). Anersys 2.0: Conquering the ner task for the arabic language by combining the maximum entropy with pos-tag information. In IICAI, pages 1814–1823. [find](https://scholar.google.com/scholar?q=Benajiba%2C+Y%2E+and+Rosso%2C+P%2E+%282007%29%2E+Anersys+2%2E0%3A+Conquering+the+ner+task+for+the+arabic+language+by+combining+the+maximum+entropy+with+pos%2Dtag+information%2E+In+IICAI%2C+pages+1814%E2%80%931823%2E)
13. Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T. (2017). Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. [find](https://scholar.google.com/scholar?q=Bojanowski%2C+P%2E%2C+Grave%2C+E%2E%2C+Joulin%2C+A%2E%2C+and+Mikolov%2C+T%2E+%282017%29%2E+Enriching+word+vectors+with+subword+information%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+5%3A135%E2%80%93146%2E)
14. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. (2019). Unsupervised cross-lingual representation learning at scale. [find](https://scholar.google.com/scholar?q=Conneau%2C+A%2E%2C+Khandelwal%2C+K%2E%2C+Goyal%2C+N%2E%2C+Chaudhary%2C+V%2E%2C+Wenzek%2C+G%2E%2C+Guzma%CC%81n%2C+F%2E%2C+Grave%2C+E%2E%2C+Ott%2C+M%2E%2C+Zettlemoyer%2C+L%2E%2C+and+Stoyanov%2C+V%2E+%282019%29%2E+Unsupervised+cross%2Dlingual+representation+learning+at+scal)
15. Dahou, A., Elaziz, M. A., Zhou, J., and Xiong, S. (2019a). Arabic sentiment classification using convolutional neural network and differential evolution algorithm. Computational intelligence and neuroscience, 2019. [find](https://scholar.google.com/scholar?q=Dahou%2C+A%2E%2C+Elaziz%2C+M%2E+A%2E%2C+Zhou%2C+J%2E%2C+and+Xiong%2C+S%2E+%282019a%29%2E+Arabic+sentiment+classification+using+convolutional+neural+network+and+differential+evolution+algorithm%2E+Computational+intelligence+and+neuro)
16. Dahou, A., Xiong, S., Zhou, J., and Elaziz, M. A. (2019b). Multi-channel embedding convolutional neural network model for arabic sentiment classification. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 18(4):1–23. [find](https://scholar.google.com/scholar?q=Dahou%2C+A%2E%2C+Xiong%2C+S%2E%2C+Zhou%2C+J%2E%2C+and+Elaziz%2C+M%2E+A%2E+%282019b%29%2E+Multi%2Dchannel+embedding+convolutional+neural+network+model+for+arabic+sentiment+classification%2E+ACM+Transactions+on+Asian+and+Low%2DResource+La)
17. de Vries, W., van Cranenburgh, A., Bisazza, A., Caselli, T., van Noord, G., and Nissim, M. (2019). Bertje: A dutch bert model. arXiv preprint arXiv:1912.09582. [↗](https://arxiv.org/abs/1912.09582) [our card](https://md.research.elamur.ai/free/en/papers/de-2019-vries-bertje-dutch-bert-model/)
18. DeepsetAI. (2019). Open sourcing german bert. [find](https://scholar.google.com/scholar?q=DeepsetAI%2E+%282019%29%2E+Open+sourcing+german+bert%2E)
19. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. [↗](https://arxiv.org/abs/1810.04805)
20. El Bazi, I. and Laachfoubi, N. (2019). Arabic named entity recognition using deep learning approach. International Journal of Electrical & Computer Engineering (2088-8708), 9(3). [find](https://scholar.google.com/scholar?q=El+Bazi%2C+I%2E+and+Laachfoubi%2C+N%2E+%282019%29%2E+Arabic+named+entity+recognition+using+deep+learning+approach%2E+International+Journal+of+Electrical+%26+Computer+Engineering+%282088%2D8708%29%2C+9%283%29%2E)
21. El-Khair, I. A. (2016). 1.5 billion words arabic corpus. arXiv preprint arXiv:1611.04033. [↗](https://arxiv.org/abs/1611.04033)
22. ElJundi, O., Antoun, W., El Droubi, N., Hajj, H., El-Hajj, W., and Shaban, K. (2019). hulmona: The universal language model in arabic. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 68– 77. [find](https://scholar.google.com/scholar?q=ElJundi%2C+O%2E%2C+Antoun%2C+W%2E%2C+El+Droubi%2C+N%2E%2C+Hajj%2C+H%2E%2C+El%2DHajj%2C+W%2E%2C+and+Shaban%2C+K%2E+%282019%29%2E+hulmona%3A+The+universal+language+model+in+arabic%2E+In+Proceedings+of+the+Fourth+Arabic+Natural+Language+Processing+W)
23. Elnagar, A., Khalifa, Y. S., and Einea, A. (2018). Hotel arabic-reviews dataset construction for sentiment analysis applications. In Intelligent Natural Language Processing: Trends and Applications, pages 35–52. Springer. [find](https://scholar.google.com/scholar?q=Elnagar%2C+A%2E%2C+Khalifa%2C+Y%2E+S%2E%2C+and+Einea%2C+A%2E+%282018%29%2E+Hotel+arabic%2Dreviews+dataset+construction+for+sentiment+analysis+applications%2E+In+Intelligent+Natural+Language+Processing%3A+Trends+and+Applications%2C+p)
24. Erdmann, A., Zalmout, N., and Habash, N. (2018). Addressing noise in multidialectal word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 558–565. [find](https://scholar.google.com/scholar?q=Erdmann%2C+A%2E%2C+Zalmout%2C+N%2E%2C+and+Habash%2C+N%2E+%282018%29%2E+Addressing+noise+in+multidialectal+word+embeddings%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+)
25. Howard, J. and Ruder, S. (2018). Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146. [↗](https://arxiv.org/abs/1801.06146)
26. Huang, Z., Xu, W., and Yu, K. (2015). Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991. [↗](https://arxiv.org/abs/1508.01991) [our card](https://md.research.elamur.ai/free/en/papers/huang-2015-bidirectional-lstm-crf-models-sequence/)
27. Kudo, T. (2018). Subword regularization: Improving neural network translation models with multiple subword candidates. [find](https://scholar.google.com/scholar?q=Kudo%2C+T%2E+%282018%29%2E+Subword+regularization%3A+Improving+neural+network+translation+models+with+multiple+subword+candidates%2E)
28. Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Kelcey, M., Devlin, J., Lee, K., Toutanova, K. N., Jones, L., Chang, M.-W., Dai, A., Uszkoreit, J., Le, Q., and Petrov, S. (2019). Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kwiatkowski%2C+T%2E%2C+Palomaki%2C+J%2E%2C+Redfield%2C+O%2E%2C+Collins%2C+M%2E%2C+Parikh%2C+A%2E%2C+Alberti%2C+C%2E%2C+Epstein%2C+D%2E%2C+Polosukhin%2C+I%2E%2C+Kelcey%2C+M%2E%2C+Devlin%2C+J%2E%2C+Lee%2C+K%2E%2C+Toutanova%2C+K%2E+N%2E%2C+Jones%2C+L%2E%2C+Chang%2C+M%2E%2DW%2E%2C+Dai%2C+A%2E%2C+Usz) [our card](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
29. Lafferty, J. D., McCallum, A., and Pereira, F. C. (2001). Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In ICML. [find](https://scholar.google.com/scholar?q=Lafferty%2C+J%2E+D%2E%2C+McCallum%2C+A%2E%2C+and+Pereira%2C+F%2E+C%2E+%282001%29%2E+Conditional+random+fields%3A+Probabilistic+models+for+segmenting+and+labeling+sequence+data%2E+In+ICML%2E)
30. Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., and Dyer, C. (2016). Neural architectures for named entity recognition. arXiv preprint arXiv:1603.01360. [↗](https://arxiv.org/abs/1603.01360)
31. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. (2019). Albert: A lite bert for self-supervised learning of language representations. [find](https://scholar.google.com/scholar?q=Lan%2C+Z%2E%2C+Chen%2C+M%2E%2C+Goodman%2C+S%2E%2C+Gimpel%2C+K%2E%2C+Sharma%2C+P%2E%2C+and+Soricut%2C+R%2E+%282019%29%2E+Albert%3A+A+lite+bert+for+self%2Dsupervised+learning+of+language+representations%2E)
32. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. [find](https://scholar.google.com/scholar?q=Liu%2C+Y%2E%2C+Ott%2C+M%2E%2C+Goyal%2C+N%2E%2C+Du%2C+J%2E%2C+Joshi%2C+M%2E%2C+Chen%2C+D%2E%2C+Levy%2C+O%2E%2C+Lewis%2C+M%2E%2C+Zettlemoyer%2C+L%2E%2C+and+Stoyanov%2C+V%2E+%282019%29%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E)
33. Martin, L., Muller, B., Suárez, P. J. O., Dupont, Y., Romary, L., Éric Villemonte de la Clergerie, Seddah, D., and Sagot, B. (2019). Camembert: a tasty french language model. [find](https://scholar.google.com/scholar?q=Martin%2C+L%2E%2C+Muller%2C+B%2E%2C+Sua%CC%81rez%2C+P%2E+J%2E+O%2E%2C+Dupont%2C+Y%2E%2C+Romary%2C+L%2E%2C+E%CC%81ric+Villemonte+de+la+Clergerie%2C+Seddah%2C+D%2E%2C+and+Sagot%2C+B%2E+%282019%29%2E+Camembert%3A+a+tasty+french+language+model%2E) [our card](https://md.research.elamur.ai/free/en/papers/martin-2019-camembert-tasty-french-language-model/)
34. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. (2013). Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111– 3119. [find](https://scholar.google.com/scholar?q=Mikolov%2C+T%2E%2C+Sutskever%2C+I%2E%2C+Chen%2C+K%2E%2C+Corrado%2C+G%2E+S%2E%2C+and+Dean%2C+J%2E+%282013%29%2E+Distributed+representations+of+words+and+phrases+and+their+compositionality%2E+In+Advances+in+neural+information+processing+sys)
35. Mikolov, T., Grave, E., Bojanowski, P., Puhrsch, C., and Joulin, A. (2017). Advances in pre-training distributed word representations. arXiv preprint arXiv:1712.09405. [↗](https://arxiv.org/abs/1712.09405) [our card](https://md.research.elamur.ai/free/en/papers/mikolov-2017-advances-pre-training-distributed-word/)
36. Mozannar, H., Maamary, E., El Hajal, K., and Hajj, H. (2019). Neural arabic question answering. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 108–118. [find](https://scholar.google.com/scholar?q=Mozannar%2C+H%2E%2C+Maamary%2C+E%2E%2C+El+Hajal%2C+K%2E%2C+and+Hajj%2C+H%2E+%282019%29%2E+Neural+arabic+question+answering%2E+In+Proceedings+of+the+Fourth+Arabic+Natural+Language+Processing+Workshop%2C+pages+108%E2%80%93118%2E)
37. Nabil, M., Aly, M., and Atiya, A. (2015). ASTD: Arabic sentiment tweets dataset. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2515–2519, Lisbon, Portugal, September. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nabil%2C+M%2E%2C+Aly%2C+M%2E%2C+and+Atiya%2C+A%2E+%282015%29%2E+ASTD%3A+Arabic+sentiment+tweets+dataset%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+2515%E2%80%932519%2C+Lisbon%2C+Por) [our card](https://md.research.elamur.ai/free/en/papers/nabil-2015-astd-arabic-sentiment-tweets-dataset/)
38. Pennington, J., Socher, R., and Manning, C. (2014). Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532– 1543. [find](https://scholar.google.com/scholar?q=Pennington%2C+J%2E%2C+Socher%2C+R%2E%2C+and+Manning%2C+C%2E+%282014%29%2E+Glove%3A+Global+vectors+for+word+representation%2E+In+Proceedings+of+the+2014+conference+on+empirical+methods+in+natural+language+processing+%28EMNLP%29%2C+pa)
39. Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018). Deep contextualized word representations. In Proceedings of NAACL-HLT, pages 2227–2237. [find](https://scholar.google.com/scholar?q=Peters%2C+M%2E+E%2E%2C+Neumann%2C+M%2E%2C+Iyyer%2C+M%2E%2C+Gardner%2C+M%2E%2C+Clark%2C+C%2E%2C+Lee%2C+K%2E%2C+and+Zettlemoyer%2C+L%2E+%282018%29%2E+Deep+contextualized+word+representations%2E+In+Proceedings+of+NAACL%2DHLT%2C+pages+2227%E2%80%932237%2E)
40. Polignano, M., Basile, P., de Gemmis, M., Semeraro, G., and Basile, V. (2019). AlBERTo: Italian BERT Language Understanding Model for NLP Challenging Tasks Based on Tweets. In Proceedings of the Sixth Italian Conference on Computational Linguistics (CLiC-it 2019), volume 2481. CEUR. [find](https://scholar.google.com/scholar?q=Polignano%2C+M%2E%2C+Basile%2C+P%2E%2C+de+Gemmis%2C+M%2E%2C+Semeraro%2C+G%2E%2C+and+Basile%2C+V%2E+%282019%29%2E+AlBERTo%3A+Italian+BERT+Language+Understanding+Model+for+NLP+Challenging+Tasks+Based+on+Tweets%2E+In+Proceedings+of+the+Sixth) [our card](https://md.research.elamur.ai/free/en/papers/polignano-2019-alberto-italian-bert-language-understanding/)
41. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. (2019). Exploring the limits of transfer learning with a unified text-to-text transformer. [find](https://scholar.google.com/scholar?q=Raffel%2C+C%2E%2C+Shazeer%2C+N%2E%2C+Roberts%2C+A%2E%2C+Lee%2C+K%2E%2C+Narang%2C+S%2E%2C+Matena%2C+M%2E%2C+Zhou%2C+Y%2E%2C+Li%2C+W%2E%2C+and+Liu%2C+P%2E+J%2E+%282019%29%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtext+transformer%2E)
42. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250. [↗](https://arxiv.org/abs/1606.05250) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
43. Ratnaparkhi, A. (1998). Maximum entropy models for natural language ambiguity resolution. [find](https://scholar.google.com/scholar?q=Ratnaparkhi%2C+A%2E+%281998%29%2E+Maximum+entropy+models+for+natural+language+ambiguity+resolution%2E)
44. Sang, E. F. and De Meulder, F. (2003). Introduction to the conll-2003 shared task: Language-independent named entity recognition. arXiv preprint cs/0306050. [find](https://scholar.google.com/scholar?q=Sang%2C+E%2E+F%2E+and+De+Meulder%2C+F%2E+%282003%29%2E+Introduction+to+the+conll%2D2003+shared+task%3A+Language%2Dindependent+named+entity+recognition%2E+arXiv+preprint+cs%2F0306050%2E)
45. Soliman, A. B., Eissa, K., and El-Beltagy, S. R. (2017). Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science, 117:256–265. [find](https://scholar.google.com/scholar?q=Soliman%2C+A%2E+B%2E%2C+Eissa%2C+K%2E%2C+and+El%2DBeltagy%2C+S%2E+R%2E+%282017%29%2E+Aravec%3A+A+set+of+arabic+word+embedding+models+for+use+in+arabic+nlp%2E+Procedia+Computer+Science%2C+117%3A256%E2%80%93265%2E)
46. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. [find](https://scholar.google.com/scholar?q=Vaswani%2C+A%2E%2C+Shazeer%2C+N%2E%2C+Parmar%2C+N%2E%2C+Uszkoreit%2C+J%2E%2C+Jones%2C+L%2E%2C+Gomez%2C+A%2E+N%2E%2C+Kaiser%2C+L%2E%2C+and+Polosukhin%2C+I%2E+%282017%29%2E+Attention+is+all+you+need%2E)
47. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., and Le, Q. V. (2019). Xlnet: Generalized autoregressive pretraining for language understanding. [find](https://scholar.google.com/scholar?q=Yang%2C+Z%2E%2C+Dai%2C+Z%2E%2C+Yang%2C+Y%2E%2C+Carbonell%2C+J%2E%2C+Salakhutdinov%2C+R%2E%2C+and+Le%2C+Q%2E+V%2E+%282019%29%2E+Xlnet%3A+Generalized+autoregressive+pretraining+for+language+understanding%2E) [our card](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/)
48. Zeroual, I., Goldhahn, D., Eckart, T., and Lakhouaja, A. (2019). OSIAN: Open source international Arabic news corpus - preparation and integration into the CLARIN-infrastructure. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 175–182, Florence, Italy, August. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zeroual%2C+I%2E%2C+Goldhahn%2C+D%2E%2C+Eckart%2C+T%2E%2C+and+Lakhouaja%2C+A%2E+%282019%29%2E+OSIAN%3A+Open+source+international+Arabic+news+corpus+%2D+preparation+and+integration+into+the+CLARIN%2Dinfrastructure%2E+In+Proceedings+of+the)
49. Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., and Fidler, S. (2015). Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. [find](https://scholar.google.com/scholar?q=Zhu%2C+Y%2E%2C+Kiros%2C+R%2E%2C+Zemel%2C+R%2E%2C+Salakhutdinov%2C+R%2E%2C+Urtasun%2C+R%2E%2C+Torralba%2C+A%2E%2C+and+Fidler%2C+S%2E+%282015%29%2E+Aligning+books+and+movies%3A+Towards+story%2Dlike+visual+explanations+by+watching+movies+and+reading+boo) [our card](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/)