---
title: "AraSSM — paper by Aliane et al. (2026) — summary and ideas"
description: "The authors build AraSSM, a 12-layer bidirectional Mamba encoder. 0.03% accuracy. 22 references."
authors: ["Ahmed Amine Aliane", "Hassina Aliane", "Nasredine Semmar"]
year: 2026
venue: "arXiv:2608.08256"
source: "https://arxiv.org/abs/2608.08256"
doi: "10.48550/arxiv.2608.08256"
arxiv: "2608.08256"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/aliane-2026-arassm-bidirectional-state-space-encoder/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/aliane-2026-arassm-bidirectional-state-space-encoder/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/22/) / AraSSM: A bidirectional state-space encoder for Arabic masked language modeling

# AraSSM: A bidirectional state-space encoder for Arabic masked language modeling

**Authors:** Ahmed Amine Aliane, [Hassina Aliane](https://md.research.elamur.ai/free/en/authors/hassina-aliane/) and [Nasredine Semmar](https://md.research.elamur.ai/free/en/authors/nasredine-semmar/) · **Year:** 2026 · **Venue:** arXiv:2608.08256 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2608.08256) · [PDF](https://arxiv.org/pdf/2608.08256) · [DOI 10.48550/arxiv.2608.08256](https://doi.org/10.48550/arxiv.2608.08256) · [arXiv 2608.08256](https://arxiv.org/abs/2608.08256) · [OpenAlex](https://openalex.org/W7202183694)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Today, Arabic natural language understanding is dominated by Transformer-based encoders, including AraBERT, MARBERT and CAMeLBERT. Their attention cost grows quadratically with input length. Earlier attempts to use Mamba for Arabic skipped language-specific pretraining and fell behind. The authors build AraSSM, a 12-layer bidirectional Mamba encoder. Each block runs one selective scan left-to-right and another right-to-left, then fuses the two with a learned projection, so cost stays linear in sequence length. The model is pretrained from scratch with masked language modeling on roughly 80GB of cleaned, chunk-level deduplicated Arabic Wikipedia and CulturaX text. Training used only four RTX 2080Ti cards for about ten days. The authors fine-tune it under the AraBERT per-task protocol with three seeds. It equals or beats base-sized Transformers on HARD sentiment, comes close on ARCD question answering and ANERcorp NER, and trails by a few points on XNLI-ar. The work shows that a modest academic compute budget can produce a nearly competitive Arabic encoder, and the checkpoint is public. A matched-compute Transformer baseline and long-sequence speed tests are left for future work.

## Key points
- AraSSM has 12 layers, hidden size 512 and 105.0M parameters, close in size to AraBERT-base (~110M, hidden size 768). It reuses the AraBERTv02 tokenizer so that results stay comparable with the AraBERT family.
- All pretraining and fine-tuning ran on 4× RTX 2080Ti (11GB) for about ten days, roughly 960 GPU-hours.
- Results (mean ± std over three seeds):
  - HARD: 96.37 ± 0.03% accuracy
  - ARCD: 32.19 ± 1.07 EM and 63.79 ± 0.25 F1
  - ANERcorp: 81.54 ± 0.30 entity-level F1
  - XNLI-ar: 72.83 ± 0.07% accuracy
- The authors flag several benchmark caveats:
  - ANERcorp has known label noise.
  - XNLI-ar and Arabic-SQuAD (the latter used for prefinetuning) contain machine-translation artifacts.
  - ARCD has a small test set, and F1 is treated as more informative than EM on it.
- Evaluation covers only the standard 512-token regime. The checkpoint is released as aliane29/arassm-base on the Hugging Face Hub.

## Interesting ideas
- When reversing a padded batch for the backward pass, flip only the real tokens in place, and mask padding both before the backward scan and after the block, so a sentence's output never depends on what else is in the batch.
- Keep the tokenizer of the baseline fixed so that any score difference can be credited to the architecture rather than to segmentation.
- Report every number as mean ± standard deviation across several fine-tuning seeds instead of a single lucky run.
- Name each benchmark's built-in ceiling (label noise, translation artifacts, tiny test sets) and favour token-level F1 over exact match when valid answers differ only in surface form.
- A linear-time bidirectional encoder keeps whole-document context affordable where attention would force truncation.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tree): gparse could adopt the same discipline for any right-to-left component and add a test that a sentence gets an identical parse whether run alone or inside a mixed-length batch.
- **I2** (learn): when comparing parser or tagger variants, freeze the crate en tokenization and tagging layer so that gains are attributed to the component actually changed.
- **I3** (learn): training runs for the perceptron parser and gparse could log multi-seed spreads, so small LAS gains are judged against seed noise.
- **I4** (answer): QA on fables and tales could score with partial-overlap F1 next to exact match and document known gold-label errors, so misses are not wrongly blamed on the reader.
- **I5** (frames): coreference across a full tale could use such an encoder to see all mentions at once without chunking the text.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Commissariat à l'Énergie Atomique et aux Énergies Alternatives](https://md.research.elamur.ai/free/en/government/commissariat-a-l-energie-atomique-et-aux-energies-alternatives/) — Government body, Paris, France · 11 works · 416 citations
- [Laboratoire d'Intégration des Systèmes et des Technologies](https://md.research.elamur.ai/free/en/government/laboratoire-d-integration-des-systemes-et-des-technologies/) — Government body, Palaiseau, France · 8 works · 321 citations
- [Centre de Recherche sur l'Information Scientifique et Technique](https://md.research.elamur.ai/free/en/labs/centre-de-recherche-sur-l-information-scientifique-et-technique/) — Research institute, Algiers, Algeria · 3 works · 32 citations
- [University of Algiers Benyoucef Benkhedda](https://md.research.elamur.ai/free/en/universities/university-of-algiers-benyoucef-benkhedda/) — University, Algiers, Algeria · 4 works · 4 citations
- [CEA Paris-Saclay - Etablissement de Saclay](https://md.research.elamur.ai/free/en/labs/cea-paris-saclay-etablissement-de-saclay/) — Research institute, Saclay, France · 4 works · 207 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works

## Cite

Cite as: Ahmed Amine Aliane, Hassina Aliane and Nasredine Semmar. 2026. AraSSM: A bidirectional state-space encoder for Arabic masked language modeling. arXiv:2608.08256.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{aliane2026arassm,
  title = {{AraSSM: A bidirectional state-space encoder for Arabic masked language modeling}},
  author = {Ahmed Amine Aliane, Hassina Aliane and Nasredine Semmar},
  year = {2026},
  doi = {10.48550/arxiv.2608.08256},
  eprint = {2608.08256},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2608.08256}
}
```

- [← previous AraSentEval 2026: A Shared Task on Sentiment Analysis and Swapping in Arabic](https://md.research.elamur.ai/free/en/papers/ezzini-2026-arasenteval-2026-shared-task-sentiment/)
- [next → Archaeology at MWE-2026 PARSEME 2.0 Subtasks 1 and 2: Parsing is for Encoders, Paraphrasing is for LLMs](https://md.research.elamur.ai/free/en/papers/roscan-2026-archaeology-mwe-2026-parseme-2/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/22/)

## References

22 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Wissam Antoun, Fady Baly, and Hazem Hajj. [AraBERT: Transformer-based model for Arabic language understanding](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/). In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, pages 9–15, 2020. [↗](https://doi.org/10.48550/arxiv.2003.00104)
2. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. [ARBERT & MARBERT: Deep bidirectional transformers for Arabic](https://md.research.elamur.ai/free/en/papers/abdul-mageed-2021-arbert-marbert-deep-bidirectional-transformers/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, pages 7085–7105, 2021. [↗](https://doi.org/10.48550/arxiv.2101.01785)
3. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. [The interplay of variant, size, and task type in Arabic pre-trained language models](https://md.research.elamur.ai/free/en/papers/inoue-2021-interplay-variant-size-task-type/). In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92–104, 2021.
4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of NAACL-HLT, pages 4171–4186, 2019.
5. Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arxiv preprint arxiv:2312.00752, 2023. [↗](https://arxiv.org/abs/2312.00752)
6. Ling Yue, Sixue Xing, Yingzhou Lu, and Tianfan Fu. BioMamba: A pre-trained biomedical language representation model leveraging mamba. arxiv preprint arxiv:2408.02600, 2024. [↗](https://arxiv.org/abs/2408.02600)
7. Zhichao Yang, Avijit Mitra, Sunjae Kwon, and Hong Yu. ClinicalMamba: A generative clinical language model on longitudinal clinical notes. arxiv preprint arxiv:2403.05795, 2024. [↗](https://arxiv.org/abs/2403.05795)
8. Ali. Al-Zawqari, Mohamed Ahmed, Abul Gabbar Al-Sharafi, Mohammad M. Khader, Ali Safa, and Gerd Vandersteen. Neural classification of argument elements and styles in Arabic competitive debates. In IEEE Access, 2025. [find](https://scholar.google.com/scholar?q=Ali%2E+Al%2DZawqari%2C+Mohamed+Ahmed%2C+Abul+Gabbar+Al%2DSharafi%2C+Mohammad+M%2E+Khader%2C+Ali+Safa%2C+and+Gerd+Vandersteen%2E+Neural+classification+of+argument+elements+and+styles+in+Arabic+competitive+debates%2E+In+IEEE)
9. Wissam Antoun, Fady Baly, and Hazem Hajj. [AraELECTRA: Pre-training text discriminators for Arabic language understanding](https://md.research.elamur.ai/free/en/papers/antoun-2021-araelectra-pre-training-text-discriminators/). In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 191–195, 2021. [↗](https://doi.org/10.48550/arxiv.2012.15516)
10. Haitham Seelawi, Ibraheem Tuffaha, Mahmoud Gzawi, Wael Farhan, Bashar Talafha, Riham Badawi, Zyad Sober, Oday Al-Dweik, Abed Alhakim Freihat, and Hussein Al-Natsheh. [ALUE: Arabic language understanding evaluation](https://md.research.elamur.ai/free/en/papers/seelawi-2021-alue-arabic-language-understanding-evaluation/). In Proceedings of the Sixth Arabic Natural Language Processing Workshop, 2021.
11. AbdelRahim Elmadany, ElMoatez Billah Nagoudi, and Muhammad Abdul-Mageed. [ORCA: A challenging benchmark for Arabic language understanding](https://md.research.elamur.ai/free/en/papers/elmadany-2023-orca-challenging-benchmark-arabic-language/). arxiv preprint arxiv:2212.10758, 2023. [↗](https://arxiv.org/abs/2212.10758)
12. Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arxiv preprint arxiv:2111.00396, 2021. [↗](https://arxiv.org/abs/2111.00396)
13. Lainghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision Mamba: Efficient visual representation learning with bidirectional state space model. arxiv preprint arxiv:2401.0917, 2024. [↗](https://arxiv.org/abs/2401.0917)
14. Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, Jianbin Jiao, and Yunfan Liu. VMamba visual state space model. arxiv preprint arxiv:2401.10166, 2024. [↗](https://arxiv.org/abs/2401.10166)
15. Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu MAn, Nghia Thrung Ngo, Frank Dernoncourt, Ryan A. Rossi, and Nguyen Huu. CulturaX:a cleaned,enormous and multilingual dataset for large language models in 167 languages. arxi preprint arxiv:2309.[09400, 2023](https://md.research.elamur.ai/free/en/papers/nguyen-2023-culturax-cleaned-enormous-multilingual-dataset/). [↗](https://arxiv.org/abs/2309.09400)
16. Ashraf Elnagar, Yasmin S. Khalifa, and Anas Einea. Hotel Arabic-reviews dataset construction for sentiment analysis applications. In Intelligent Natural Language Processing: Trends and Applications, 2018. [↗](https://doi.org/10.1007/978-3-319-67056-0_3)
17. Yassine Benajiba, Paolo Rosso, and José Miguel Benedí. ANERsys: An Arabic named entity recognition system based on maximum entropy. In Proceedings of CICLing, 2007. [find](https://scholar.google.com/scholar?q=Yassine+Benajiba%2C+Paolo+Rosso%2C+and+Jos%C3%A9+Miguel+Bened%C3%AD%2E+ANERsys%3A+An+Arabic+named+entity+recognition+system+based+on+maximum+entropy%2E+In+Proceedings+of+CICLing%2C+2007%2E)
18. Rami Al-Rfou, Vivek Kulkarni, Bryan Perozzi, and Steven Skiena. Polyglot-NER: Massive multilingual named entity recognition. In Proceedings of the SIAM International Conference on Data Mining, 2015. [↗](https://doi.org/10.48550/arxiv.1410.3791)
19. Mashael Al-Duwais, Hend Al-Khalifa, and Abdulmalik Al-Salman. CleanANERCorp: Identifying and correcting incorrect labels in the anercorp dataset. In OSACT 2024 Workshop, pages 13–19, 2024. [find](https://scholar.google.com/scholar?q=Mashael+Al%2DDuwais%2C+Hend+Al%2DKhalifa%2C+and+Abdulmalik+Al%2DSalman%2E+CleanANERCorp%3A+Identifying+and+correcting+incorrect+labels+in+the+anercorp+dataset%2E+In+OSACT+2024+Workshop%2C+pages+13%E2%80%9319%2C+2024%2E)
20. Hussein Mozannar, Elie Maamary, Karl El Hajal, and Hazem Hajj. [Neural Arabic question answering](https://md.research.elamur.ai/free/en/papers/mozannar-2019-neural-arabic-question-answering/). In Proceedings of the Fourth Arabic Natural Language Processing Workshop, 2019.
21. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. [TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages](https://md.research.elamur.ai/free/en/papers/clark-2020-tydi-qa-benchmark-information-seeking/). Transactions of the Association for Computational Linguistics, 8:454–470, 2020. [↗](https://doi.org/10.48550/arxiv.2003.05002)
22. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of EMNLP, 2018. [↗](https://doi.org/10.48550/arxiv.1809.05053)

*Updated 1 November 2026*