---
title: "AraModernBERT — paper by Elshehy et al. (2026)"
description: "Modern encoder designs such as ModernBERT have mostly been built and tested for English. 0.52 MRR. 20 references, cited by 2 papers here."
authors: ["Omar Elshehy", "Omer Nacar", "Abdelbasset Djamai", "Muhammed Ragab", "Khloud Al Jallad", "Mona Abdelazim"]
year: 2026
venue: "Proceedings of the 2nd Workshop on NLP for Languages Using Arabic Script"
source: "https://aclanthology.org/2026.abjadnlp-1.39/"
doi: "10.18653/v1/2026.abjadnlp-1.39"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/elshehy-2026-aramodernbert-transtokenized-initialization-long-context/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/elshehy-2026-aramodernbert-transtokenized-initialization-long-context/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/6/) / AraModernBERT: Transtokenized Initialization and Long-Context Encoder Modeling for Arabic

# AraModernBERT: Transtokenized Initialization and Long-Context Encoder Modeling for Arabic

**Authors:** [Omar Elshehy](https://md.research.elamur.ai/free/en/authors/omar-elshehy/), [Omer Nacar](https://md.research.elamur.ai/free/en/authors/omer-nacar/), [Abdelbasset Djamai](https://md.research.elamur.ai/free/en/authors/abdelbasset-djamai/), Muhammed Ragab, [Khloud Al Jallad](https://md.research.elamur.ai/free/en/authors/khloud-al-jallad/) and Mona Abdelazim · **Year:** 2026 · **Venue:** [Proceedings of the 2nd Workshop on NLP for Languages Using Arabic Script](https://md.research.elamur.ai/free/en/venues/abjadnlp/) ([Workshop on NLP for Languages Using Arabic Script 2026](https://md.research.elamur.ai/free/en/venues/abjadnlp/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.abjadnlp-1.39/) · [PDF](https://aclanthology.org/2026.abjadnlp-1.39.pdf) · [DOI 10.18653/v1/2026.abjadnlp-1.39](https://doi.org/10.18653/v1/2026.abjadnlp-1.39) · [OpenAlex](https://openalex.org/W7140103017)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [2](https://md.research.elamur.ai/free/en/papers/elshehy-2026-aramodernbert-transtokenized-initialization-long-context/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Modern encoder designs such as ModernBERT have mostly been built and tested for English. Classic Arabic BERT models are still limited to 512 tokens and use absolute positions. The authors port ModernBERT to Arabic as AraModernBERT, a ~149M-parameter encoder that reads up to 8,192 tokens natively. It uses a new Arabic BPE tokenizer and is pretrained on roughly 100 GB of Arabic text.

Swapping in a new tokenizer normally means the embedding table must be reinitialized. To avoid this, they use transtokenization: each new Arabic token starts from a weighted mix of the pretrained embeddings of source tokens it aligns with in a parallel corpus.

An ablation shows that skipping this step causes catastrophic collapse of masked language modeling quality. With it, training is stable, and intrinsic MLM performance improves at longer sequence lengths. Downstream tests cover NLI, offensive language detection, question–question similarity, NER and short-text retrieval, and show solid transfer, especially on large, clean datasets.

The broader lesson is that tokenizer replacement should be treated as a core modeling decision for Arabic-script languages, not as a preprocessing detail.

## Key points
- Architecture follows ModernBERT:
  - 22 layers, hidden size 768, 12 heads, about 149M parameters.
  - Global attention in every third layer; the other layers use a 128-token sliding window.
  - Rotary position embeddings and a native context of 8,192 tokens.
- The tokenizer is a dedicated Arabic BPE vocabulary of 50,280 tokens.
- How the embeddings are initialized:
  - Each token's embedding is the alignment-count-normalized average of the source embeddings it aligns to.
  - Unaligned tokens (digits, punctuation, special symbols) get fixed fallback mappings.
- On MQ2Q short-text retrieval, AraBERT-base edges ahead slightly: R@1 0.54 vs 0.52, MRR 0.73 vs 0.72. Both models reach R@5 0.97 and R@10 0.99.
- NER test F1, averaged over three seeds:
  - WikiAnn (ar): 0.8576
  - ANERCorp: 0.6827
  - AQMAR: 0.5929
  - Twitter NER: 0.4919

## Interesting ideas
- When a vocabulary is replaced, build each new token's vector as a count-weighted average of the old-vocabulary vectors it aligns with, instead of starting from random noise; this keeps semantic structure and prevents training collapse.
- Cover tokens that cannot be aligned (numbers, punctuation, special symbols) with an explicit, hand-defined fallback table, so no entry is left uninitialized or noisy.
- Mix many cheap windowed attention layers with a few full-sequence layers; this gives document-scale reach without paying full attention cost everywhere.
- Run an ablation with and without the initialization trick and report it as a headline result, rather than hiding tokenizer choices as plumbing.
- Note openly that gains in intrinsic long-context modeling went untested on tasks that truly need distant context, and that short lexical retrieval favored the older model.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tokens): When the en crate's tokenizer or lemma inventory is revised, new entries could inherit initial weights from aligned old entries instead of starting cold.
- **I2** (tree): gparse could keep a small deterministic fallback map for symbols and numerals whenever its input vocabulary changes, keeping parses of such tokens stable across retraining.
- **I3** (frames): A gparse-style encoder could use mostly local layers for syntax and periodic global layers that carry long-range signals to coreference across a whole story.
- **I4** (learn): Every tokenizer or vocabulary change in training could ship with a negative-control run that skips the transfer step, making its contribution measurable.
- **I5** (answer): Fable and tale QA evaluation should include items whose answer depends on an early sentence, so long-range world tracking is actually tested.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Saarland University](https://md.research.elamur.ai/free/en/universities/saarland-university/) — University, Saarbrücken, Germany · 148 works · 18,287 citations
- [Arab International University](https://md.research.elamur.ai/free/en/universities/arab-international-university/) — University, Ghabāghib, Syria · 13 works · 46 citations
- [Ain Shams University](https://md.research.elamur.ai/free/en/universities/ain-shams-university/) — University, Cairo, Egypt · 12 works · 109 citations
- [Tuwaiq Academy](https://md.research.elamur.ai/free/en/universities/tuwaiq-academy/) — University, Riyadh, Saudi Arabia · 11 works · 7 citations

Also on the byline: [DataStrategy](https://ror.org/01h8x8p33) · [Leibniz Institute for Educational Media | Georg Eckert Institute](https://ror.org/01hnv8x68).

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works

## Cite

Cite as: Omar Elshehy, Omer Nacar, Abdelbasset Djamai, Muhammed Ragab, Khloud Al Jallad and Mona Abdelazim. 2026. AraModernBERT: Transtokenized Initialization and Long-Context Encoder Modeling for Arabic. Proceedings of the 2nd Workshop on NLP for Languages Using Arabic Script.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{elshehy2026aramodernbert,
  title = {{AraModernBERT: Transtokenized Initialization and Long-Context Encoder Modeling for Arabic}},
  author = {Omar Elshehy, Omer Nacar, Abdelbasset Djamai, Muhammed Ragab, Khloud {Al Jallad} and Mona Abdelazim},
  booktitle = {Proceedings of the 2nd Workshop on NLP for Languages Using Arabic Script},
  year = {2026},
  doi = {10.18653/v1/2026.abjadnlp-1.39},
  url = {https://doi.org/10.18653/v1/2026.abjadnlp-1.39}
}
```

- [← previous Ara-HOPE: Human-Centric Post-Editing Evaluation for Dialectal Arabic to Modern Standard Arabic Translation](https://md.research.elamur.ai/free/en/papers/alabdullah-2026-ara-hope-human-centric-post/)
- [next → Are LLMs Reliable Rankers? Rank Manipulation via Two-Stage Token Optimization](https://md.research.elamur.ai/free/en/papers/xing-2026-are-llms-reliable-rankers-rank/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/6/)

## References

20 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. [Arbert & marbert: Deep bidirectional transformers for arabic](https://md.research.elamur.ai/free/en/papers/abdul-mageed-2021-arbert-marbert-deep-bidirectional-transformers/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, pages 7088– 7105. [↗](https://doi.org/10.48550/arxiv.2101.01785)
2. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. [Arabert: Transformer-based model for arabic language understanding](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/). In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, pages 9–15. [↗](https://doi.org/10.48550/arxiv.2003.00104)
3. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. [On the cross-lingual transferability of monolingual representations](https://md.research.elamur.ai/free/en/papers/artetxe-2020-cross-lingual-transferability-monolingual-representations/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637.
4. Yassine Benajiba, Paolo Rosso, and José Miguel BenedíRuiz. 2007. ANERsys: An Arabic Named Entity Recognition System Based on Maximum Entropy. In Computational Linguistics and Intelligent Text Processing, pages 143–153, Berlin, Heidelberg. Springer. [find](https://scholar.google.com/scholar?q=Yassine+Benajiba%2C+Paolo+Rosso%2C+and+Jos%C3%A9+Miguel+Bened%C3%ADRuiz%2E+2007%2E+ANERsys%3A+An+Arabic+Named+Entity+Recognition+System+Based+on+Maximum+Entropy%2E+In+Computational+Linguistics+and+Intelligent+Text+Processi)
5. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1809.05053)
6. Kareem Darwish. 2013. [Named Entity Recognition using Cross-lingual Resources: Arabic as an Example](https://md.research.elamur.ai/free/en/papers/darwish-2013-named-entity-recognition-cross-lingual/). In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1558–1567, Sofia, Bulgaria. Association for Computational Linguistics.
7. Wietse de Vries and Malvina Nissim. 2021. [As good as new: How to successfully recycle english gpt-2 to make models for other languages](https://md.research.elamur.ai/free/en/papers/vries-2021-as-good-as-new-how/). In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 836–846.
8. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. [The interplay of variant, size, and task type in arabic pre-trained language models](https://md.research.elamur.ai/free/en/papers/inoue-2021-interplay-variant-size-task-type/). In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92– 104.
9. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. [Dense passage retrieval for open-domain question answering](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6769–6781.
10. Omar Khattab and Matei Zaharia. 2020. [Colbert: Efficient and effective passage search via contextualized late interaction over bert](https://md.research.elamur.ai/free/en/papers/khattab-2020-colbert-efficient-effective-passage/). In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 39–48. [↗](https://doi.org/10.48550/arxiv.2004.12832)
11. Behrang Mohit, Nathan Schneider, Rishav Bhowmick, Kemal Oflazer, and Noah A. Smith. 2012. [Recall-oriented learning of named entities in Arabic Wikipedia](https://md.research.elamur.ai/free/en/papers/mohit-2012-recall-oriented-learning-named-entities/). In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 162–173, Avignon, France. Association for Computational Linguistics.
12. Hamdy Mubarak, Kareem Darwish, Walid Magdy, Tamer Elsayed, and Hend Al-Khalifa. 2020. [Overview of OSACT4 Arabic offensive language detection shared task](https://md.research.elamur.ai/free/en/papers/mubarak-2020-overview-osact4-arabic-offensive-language/). In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 48–52, Marseille, France. European Language Resource Association.
13. Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr, and Adel Bibi. 2023. [Language model tokenizers introduce unfairness between languages](https://md.research.elamur.ai/free/en/papers/petrov-2023-language-model-tokenizers-introduce-unfairness/). In Advances in Neural Information Processing Systems.
14. Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019. [Massively Multilingual Transfer for NER](https://md.research.elamur.ai/free/en/papers/rahimi-2019-massively-multilingual-transfer-ner/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 151–164, Florence, Italy. Association for Computational Linguistics.
15. François Remy, Pieter Delobelle, Hayastan Avetisyan, Alfiya Khabibullina, Miryam de Lhoneux, and Thomas Demeester. 2024. Trans-tokenization and cross-lingual vocabulary transfers: Language adaptation of llms for low-resource nlp. arXiv preprint arXiv:2408.04303. [↗](https://arxiv.org/abs/2408.04303)
16. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. [How good is your tokenizer? on the monolingual performance of multilingual language models](https://md.research.elamur.ai/free/en/papers/rust-2021-how-good-is-your-tokenizer/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, pages 3118–3135.
17. Nathan Schneider, Behrang Mohit, Kemal Oflazer, and Noah A. Smith. 2012. [Coarse Lexical Semantic Annotation with Supersenses: An Arabic Case Study](https://md.research.elamur.ai/free/en/papers/schneider-2012-coarse-lexical-semantic-annotation-supersenses/). In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 253–258, Jeju Island, Korea. Association for Computational Linguistics.
18. Haitham Seelawi, Ahmad Mustafa, Hesham Al-Bataineh, Wael Farhan, and Hussein T. Al-Natsheh. 2019. [NSURL-2019 task 8: Semantic question similarity in Arabic](https://md.research.elamur.ai/free/en/papers/seelawi-2019-nsurl-2019-task-8-semantic/). In Proceedings of the First International Workshop on NLP Solutions for Under Resourced Languages (NSURL 2019) co-located with ICNLSP 2019 - Short Papers, pages 1–8, Trento, Italy. Association for Computational Linguistics.
19. Jianlin Su, Yu Lu, Shengfeng Pan, Murtadha Ahmed, Bo Wen, and Yunfeng Liu. 2021. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). arXiv preprint arXiv:2104.09864. [↗](https://arxiv.org/abs/2104.09864)
20. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, and 1 others. 2025. [Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference](https://md.research.elamur.ai/free/en/papers/warner-2024-smarter-better-faster-longer-modern/). In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2526–2547.

*Updated 1 November 2026*