--- title: "Post-hoc analysis of Arabic transformer models — Abdelali 2022" description: "Arabic has many pre-trained transformers. 12 references, cited by 6 papers here. Among them: 2." authors: ["Ahmed Abdelali", "Nadir Durrani", "Fahim Dalvi", "Hassan Sajjad"] year: 2022 venue: "Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP" source: "https://aclanthology.org/2022.blackboxnlp-1.8/" doi: "10.18653/v1/2022.blackboxnlp-1.8" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/abdelali-2022-post-hoc-analysis-arabic-transformer/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/abdelali-2022-post-hoc-analysis-arabic-transformer/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/7/) / Post-hoc analysis of Arabic transformer models # Post-hoc analysis of Arabic transformer models **Authors:** [Ahmed Abdelali](https://md.research.elamur.ai/free/en/authors/ahmed-abdelali/), [Nadir Durrani](https://md.research.elamur.ai/free/en/authors/nadir-durrani/), [Fahim Dalvi](https://md.research.elamur.ai/free/en/authors/fahim-dalvi/) and [Hassan Sajjad](https://md.research.elamur.ai/free/en/authors/hassan-sajjad/) · **Year:** 2022 · **Venue:** [Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP](https://md.research.elamur.ai/free/en/venues/blackboxnlp/) ([BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP 2022](https://md.research.elamur.ai/free/en/venues/blackboxnlp/2022/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2022.blackboxnlp-1.8/) · [PDF](https://aclanthology.org/2022.blackboxnlp-1.8.pdf) · [DOI 10.18653/v1/2022.blackboxnlp-1.8](https://doi.org/10.18653/v1/2022.blackboxnlp-1.8) · [OpenAlex](https://openalex.org/W4385572948) **Citations:** 2 citations (OpenAlex, 2026-10-14) · [6](https://md.research.elamur.ai/free/en/papers/abdelali-2022-post-hoc-analysis-arabic-transformer/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Arabic has many pre-trained transformers. They differ mainly in how much Modern Standard Arabic (MSA), dialectal and Classical text they were trained on. Earlier comparisons looked only at downstream scores. This paper instead inspects what the models encode internally. The authors study nine models: - five Arabic BERT variants: AraBERT, ArabicBERT, CAMeLBERT, MARBERT and QARiB - Arabic ALBERT and AraELECTRA - two multilingual models: mBERT and XLM They freeze each model, extract its activations and train logistic-regression probes on every layer. For neuron-level analysis they use Linguistic Correlation Analysis (LCA). LCA is an elastic-net probe whose weights rank neurons, and it yields a small neuron subset that nearly matches the full-network oracle. The probing tasks are: - POS tagging on MSA (ATB) - POS tagging on several dialects (GMR, CRS, DIA) - dialect identification (DID) The main findings: - Morphology is strongest in lower and middle layers. - Dialect identification depends on upper layers. - MSA-only models miss dialectal nuances even though their vocabulary overlaps heavily with dialects. - Embedding-layer neurons serve many properties, while middle-layer neurons are more specialised. These results help practitioners choose models and layers for feature-based transfer. ## Key points - Probing data: ATB (Arabic Treebank Part1 v2.0 + Part3 v1.0, about 515k tokens, 42 tags); GMR (15,225 Emirati sentences from the Gumar Corpus, 169 tags); CRS (Curras, about 5K Palestinian sentences, 260 tags); DIA (1.4k tweets in EGY, LEV, GLF and MGR); plus a dialect identification task (DID). - Training data differs widely across models: AraBERT, ArabicBERT and AraELECTRA are MSA-based; MARBERT and QARiB add tweets; CAMeLBERT-Mix combines MSA, CA and DA (17.3B tokens); XLM was trained on 2.5TB of CommonCrawl. AraELECTRA has 256-unit hidden layers, while the other models have 768. - In most models, salient neurons come mainly from the embedding layer and the final layers. For most properties, the middle layers contributed fewer than 20 neurons. ALBERT is the exception: its embedding layer contributes almost nothing, which the authors attribute to parameter sharing and factorised embeddings. - Noun neurons cluster in the embedding layer, while verb neurons cluster in the final layers. Closed-class tags such as PRP need only a few neurons, whereas open-class tags such as VBD need many. - In QARiB, the top neurons for DT and NN overlap by about 54%. The authors link this to clitic determiners that attach to nouns. Such shared neurons are most common in the embedding layer and become rarer higher up. ## Interesting ideas - Instead of reporting only accuracy, find the smallest ranked neuron subset whose probe stays within a set margin of the full-feature oracle; this measures how compactly a concept is stored. - Probe one frozen model on parallel tasks drawn from different language varieties, so a representation gap appears even when the vocabularies overlap. - Count how many neurons each tag needs, which separates closed-class categories (cheap, local) from open-class ones (costly, context-dependent). - Build per-layer overlap matrices of the top neurons for each tag, to expose entangled categories that come from the morphology, such as clitic determiner plus noun. - The authors found verb information concentrated in upper layers and nouns in the embedding layer, consistent with verbs acting as structural hubs. This suggests drawing features from different layers for different word classes. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (learn): Apply this as an evaluation of gparse hidden states per deprel, to see which relations rest on a few units and which are diffuse, and to guide pruning or explanations. - **I2** (tree): Probe the gparse encoder separately on treebanks of different genres, and on dialect-converted UD data, to locate where its features stop transferring before retraining. - **I3** (tokens): Use the same contrast to divide the tagger's work, with lexicon-style rules for closed classes and context features reserved for open-class tags. - **I4** (tokens): Tag pairs with high overlap can point to tokenizer splits worth revisiting, such as contractions and possessive 's, where one surface token mixes two categories. - **I5** (frames): When feeding transformer features to role labelling, take predicate representations from upper layers and argument head-noun representations from lower layers. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Hamad bin Khalifa University](https://md.research.elamur.ai/free/en/universities/hamad-bin-khalifa-university/) — University, Doha, Qatar · 144 works · 14,175 citations - [Dalhousie University](https://md.research.elamur.ai/free/en/universities/dalhousie-university/) — University, Halifax, Canada · 9 works · 480 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works 🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 3,066 works ## Cite Cite as: Ahmed Abdelali, Nadir Durrani, Fahim Dalvi and Hassan Sajjad. 2022. Post-hoc analysis of Arabic transformer models. Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{abdelali2022post, title = {{Post-hoc analysis of Arabic transformer models}}, author = {Ahmed Abdelali, Nadir Durrani, Fahim Dalvi and Hassan Sajjad}, booktitle = {Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP}, year = {2022}, doi = {10.18653/v1/2022.blackboxnlp-1.8}, url = {https://doi.org/10.18653/v1/2022.blackboxnlp-1.8} } ``` - [← previous Optimizing Naive Bayes for Arabic Dialect Identification](https://md.research.elamur.ai/free/en/papers/jauhiainen-2022-optimizing-naive-bayes-arabic-dialect/) - [next → Psychometric Properties of Self-Regulated Learning Strategies in Learning English Grammar and English Grammar Self-Efficacy Scales](https://md.research.elamur.ai/free/en/papers/truong-2022-psychometric-properties-self-regulated-learning/) - [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/7/) ## References 12 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. ## Related Work Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. [Farasa: A fast and furious segmenter for Arabic](https://md.research.elamur.ai/free/en/papers/abdelali-2016-farasa-fast-furious-segmenter-arabic/). In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 11–16, San Diego, California. Association for Computational Linguistics. 2. Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. 2021. [Pre-training bert on arabic tweets: Practical considerations](https://md.research.elamur.ai/free/en/papers/abdelali-2021-pre-training-bert-arabic-tweets/). 3. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2020. [Arbert & marbert: Deep bidirectional transformers for arabic](https://md.research.elamur.ai/free/en/papers/abdul-mageed-2021-arbert-marbert-deep-bidirectional-transformers/). 4. Sultan Alrowili and Vijay Shanker. 2021. [ArabicTransformer: Efficient large Arabic language model with funnel transformer and ELECTRA objective](https://md.research.elamur.ai/free/en/papers/alrowili-2021-arabictransformer-efficient-large-arabic-language/). In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1255–1261, Punta Cana, Dominican Republic. Association for Computational Linguistics. 5. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. [Arabert: Transformer-based model for arabic language understanding](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/). In LREC 2020 Workshop Language Resources and Evaluation Conference 11– 16 May 2020, page 9. 6. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. [Araelectra: Pre-training text discriminators for arabic language understanding](https://md.research.elamur.ai/free/en/papers/antoun-2021-araelectra-pre-training-text-discriminators/). 7. D. Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2019. [Identifying and Controlling Important Neurons in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/bau-2018-identifying-controlling-important-neurons-neural/). arXiv preprint arXiv:1811.01157. [↗](https://arxiv.org/abs/1811.01157) 8. Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017. [What do Neural Machine Translation Models Learn about Morphology?](https://md.research.elamur.ai/free/en/papers/belinkov-2017-what-do-neural-machine-translation/) In Proceedings of ACL, Vancouver. Association for Computational Linguistics. 9. Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2020. On the linguistic representational power of neural machine translation models. Computational Linguistics, 46(1):1–52. [find](https://scholar.google.com/scholar?q=Yonatan+Belinkov%2C+Nadir+Durrani%2C+Fahim+Dalvi%2C+Hassan+Sajjad%2C+and+James+Glass%2E+2020%2E+On+the+linguistic+representational+power+of+neural+machine+translation+models%2E+Computational+Linguistics%2C+46%281%29%3A1%E2%80%9352) 10. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Minh%2DThang+Luong%2C+Quoc+V%2E+Le%2C+and+Christopher+D%2E+Manning%2E+2020%2E+Electra%3A+Pre%2Dtraining+text+encoders+as+discriminators+rather+than+generators%2E) 11. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. [Unsupervised cross-lingual representation learning at scale](https://md.research.elamur.ai/free/en/papers/conneau-2020-unsupervised-cross-lingual-representation-learning/). 12. Alexis Conneau, German Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. 2018. [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://md.research.elamur.ai/free/en/papers/conneau-2018-what-you-can-cram-into/). In Proceedings of ACL. *Updated 2 November 2026*