---
title: "A New State-of-the-Art BERT Model for Judeo-Arabic (2026)"
description: "The authors build JABERT, the first BERT encoder pretrained specifically for this language. 16 references."
authors: ["Elisha Rosensweig", "Yitzchak Lindenbaum", "Hillel Gershuni", "Vered Raziel-Kretzmer", "Daniel Caine", "Avi Shmidman"]
year: 2026
venue: "Proceedings of the Fourth Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA 2026) @ LREC 2026"
source: "https://aclanthology.org/2026.lt4hala-1.6/"
doi: "10.63317/52qpsk7c5bfc"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/rosensweig-2026-new-state-art-bert-model/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/rosensweig-2026-new-state-art-bert-model/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/20/) / A New State-of-the-Art BERT Model for Judeo-Arabic

# A New State-of-the-Art BERT Model for Judeo-Arabic

**Authors:** [Elisha Rosensweig](https://md.research.elamur.ai/free/en/authors/elisha-rosensweig/), [Yitzchak Lindenbaum](https://md.research.elamur.ai/free/en/authors/yitzchak-lindenbaum/), [Hillel Gershuni](https://md.research.elamur.ai/free/en/authors/hillel-gershuni/), Vered Raziel-Kretzmer, Daniel Caine and [Avi Shmidman](https://md.research.elamur.ai/free/en/authors/avi-shmidman/) · **Year:** 2026 · **Venue:** [Proceedings of the Fourth Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA 2026) @ LREC 2026](https://md.research.elamur.ai/free/en/venues/lt4hala/) ([Workshop on Language Technologies for Historical and Ancient Languages @ LREC 2026](https://md.research.elamur.ai/free/en/venues/lt4hala/2026/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.lt4hala-1.6/) · [PDF](https://aclanthology.org/2026.lt4hala-1.6.pdf) · [DOI 10.63317/52qpsk7c5bfc](https://doi.org/10.63317/52qpsk7c5bfc) · [OpenAlex](https://openalex.org/W7166517398)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Judeo-Arabic is the family of Arabic varieties that Jewish communities across the Middle East and North Africa wrote in Hebrew characters from about the 8th century onward. It has a large medieval textual heritage but very little NLP support. Digital corpora are small, spelling varies widely, and Arabic-script tools can only handle it after error-prone transliteration.

The authors build JABERT, the first BERT encoder pretrained specifically for this language. The training corpus combines native Judeo-Arabic material with a far larger body of medieval Arabic from OpenITI. The OpenITI texts were converted into Hebrew script, with randomized spelling choices that imitate scribal habits. The tokenizer has a 128K WordPiece vocabulary and includes special tokens that mark gaps in damaged manuscripts.

For evaluation, the team created two new resources: expert-transcribed Cairo Genizah lines for masked-word prediction, and the first annotated Judeo-Arabic homograph test set. On homograph disambiguation, JABERT beats Arabic and multilingual BERTs, transliteration-based pipelines, and the Hebrew-script HeArBERT. It also reads mixed Hebrew–Arabic text directly. The model and the test set are released openly.

## Key points
- **Training sources:**
  - Princeton Genizah Project (~1.6M words)
  - Friedberg Judeo-Arabic Project (FJMS, ~3.1M words)
  - Quotations from Blau's Dictionary of Mediaeval Judaeo-Arabic Texts (~42K words)
  - OpenITI texts transliterated into Hebrew script (~1.2B words; authors who died 390–905 AH)
- **Tokenizer:** 128K WordPiece vocabulary with two lacuna tokens. `[ONEGAP]` marks a single damaged word, and `[GAP]` marks a longer or indeterminate gap.
- **Pretraining:** four NVIDIA A40 GPUs, 86,010 iterations, 106.56B tokens, in three phases (128-token sequences, then 512-token sequences).
- **New evaluation data:**
  - 3,862 Cairo Genizah lines from Cambridge Taylor-Schechter boxes.
  - A homograph set of 16 words, 37 meanings and 1,119 samples, labelled by two experts. Samples where the experts disagreed were dropped.
- **Release:** JABERT is available on HuggingFace (MiDRASH-ERC/JABERT) under CC BY 4.0.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Bar-Ilan University](https://md.research.elamur.ai/free/en/universities/bar-ilan-university/) — University, Ramat Gan, Israel · 141 works · 23,225 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works · [🕎 Hebrew & Aramaic](https://md.research.elamur.ai/free/en/topics/hebrew-aramaic/) · 496 works

## Cite

Cite as: Elisha Rosensweig, Yitzchak Lindenbaum, Hillel Gershuni, Vered Raziel-Kretzmer, Daniel Caine and Avi Shmidman. 2026. A New State-of-the-Art BERT Model for Judeo-Arabic. Proceedings of the Fourth Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA 2026) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{rosensweig2026new,
  title = {{A New State-of-the-Art BERT Model for Judeo-Arabic}},
  author = {Elisha Rosensweig, Yitzchak Lindenbaum, Hillel Gershuni, Vered Raziel-Kretzmer, Daniel Caine and Avi Shmidman},
  booktitle = {Proceedings of the Fourth Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA 2026) @ LREC 2026},
  year = {2026},
  doi = {10.63317/52qpsk7c5bfc},
  url = {https://doi.org/10.63317/52qpsk7c5bfc}
}
```

- [← previous A Mīmāṃsā-Based Framework for Disambiguating Compounds in Sanskrit](https://md.research.elamur.ai/free/en/papers/suraj-2026-mima-sa-based-framework-disambiguating/)
- [next → A Novel Dataset and Three Ways to Approach Automatic Metaphor Detection in German Religious Online Forums](https://md.research.elamur.ai/free/en/papers/reimann-2026-novel-dataset-three-ways-approach/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/20/)

## References

16 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. [ARBERT & MARBERT: Deep bidirectional transformers for Arabic](https://md.research.elamur.ai/free/en/papers/abdul-mageed-2021-arbert-marbert-deep-bidirectional-transformers/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2021.acl-long.551)
2. Bashar Alhafni, Go Inoue, Christian Khairallah, and Nizar Habash. 2023. [Advancements in Arabic grammatical error detection and correction: An empirical investigation](https://md.research.elamur.ai/free/en/papers/alhafni-2023-advancements-arabic-grammatical-error-detection/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6430–6448, Singapore. Association for Computational Linguistics.
3. Wissam Antoun, Fady Baly, and Hazem Hajj. [Arabert: Transformer-based model for arabic language understanding](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/). In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. [↗](https://doi.org/10.48550/arxiv.2003.00104)
4. Kfir Bar, Nachum Dershowitz, Lior Wolf, Yackov Lubarsky, and Yaacov Choueka. 2015. Processing judeo-arabic texts. In 2015 First International Conference on Arabic Computational Linguistics (ACLing), pages 138–144. [find](https://scholar.google.com/scholar?q=Kfir+Bar%2C+Nachum+Dershowitz%2C+Lior+Wolf%2C+Yackov+Lubarsky%2C+and+Yaacov+Choueka%2E+2015%2E+Processing+judeo%2Darabic+texts%2E+In+2015+First+International+Conference+on+Arabic+Computational+Linguistics+%28ACLing%29%2C+p)
5. Joshua Blau. 2006. A Dictionary of Mediaeval Judaeo-Arabic Texts. The Academy of the Hebrew Language; The Israel Academy of Sciences and Humanities. [find](https://scholar.google.com/scholar?q=Joshua+Blau%2E+2006%2E+A+Dictionary+of+Mediaeval+Judaeo%2DArabic+Texts%2E+The+Academy+of+the+Hebrew+Language%3B+The+Israel+Academy+of+Sciences+and+Humanities%2E)
6. Juan Moreno Gonzalez, Bashar Alhafni, and Nizar Habash. 2025. [A tale of two scripts: Transliteration and post-correction for judeo-arabic](https://md.research.elamur.ai/free/en/papers/gonzalez-2026-tale-two-scripts-transliteration-post/).
7. Eylon Gueta, Avi Shmidman, Shaltiel Shmidman, Cheyn Shmuel Shmidman, Joshua Guedalia, Moshe Koppel, Dan Bareket, Amit Seker, and Reut Tsarfaty. 2023. [Large pre-trained models with extra-large vocabularies: A contrastive analysis of Hebrew BERT models and a new one to outperform them all](https://md.research.elamur.ai/free/en/papers/gueta-2022-large-pre-trained-models-extra/). [↗](https://doi.org/10.48550/arxiv.2211.15199)
8. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. [The interplay of variant, size, and task type in Arabic pre-trained language models](https://md.research.elamur.ai/free/en/papers/inoue-2021-interplay-variant-size-task-type/). In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine (Online). Association for Computational Linguistics.
9. Princeton Geniza Project. 2021. Transcription history and conventions. Google Docs. Archived at [https://web.archive.org/web/2026](https://web.archive.org/web/2026) 0204094447/https://docs.google.com/document/d/1pyFajrGRYTzRsb2yFjfX0X mNqpz1kUO9mhlISGwfpxk/edit?tab=t.0. Accessed March 25, 2025. [↗](https://web.archive.org/web/2026)
10. Aviad Rom and Kfir Bar. 2024. Training a bilingual language model by mapping tokens onto a shared character space. arXiv preprint arXiv:2402.16065. [↗](https://arxiv.org/abs/2402.16065)
11. Maxim Romanov. 2023. OpenITI mARkdown. Accessed April 7, 2025. [find](https://scholar.google.com/scholar?q=Maxim+Romanov%2E+2023%2E+OpenITI+mARkdown%2E+Accessed+April+7%2C+2025%2E)
12. Xinying Song, Alex Salcianu, Yang Song, Dave Dopson, and Denny Zhou. 2021. Fast wordpiece tokenization. [find](https://scholar.google.com/scholar?q=Xinying+Song%2C+Alex+Salcianu%2C+Yang+Song%2C+Dave+Dopson%2C+and+Denny+Zhou%2E+2021%2E+Fast+wordpiece+tokenization%2E)
13. Ori Terner, Kfir Bar, and Nachum Dershowitz. 2020. [Transliteration of Judeo-Arabic texts into Arabic script using recurrent neural networks](https://md.research.elamur.ai/free/en/papers/terner-2020-transliteration-judeo-arabic-texts-into/). In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 85–96, Barcelona, Spain (Online). Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2004.11405)
14. Daniel Weisberg Mitelman, Nachum Dershowitz, and Kfir Bar. 2024. [Code-switching and back-transliteration using a bilingual model](https://md.research.elamur.ai/free/en/papers/mitelman-2024-code-switching-back-transliteration-bilingual/). In Findings of the Association for Computational Linguistics: EACL 2024, pages 1501–1511, St. Julian’s, Malta. Association for Computational Linguistics. 10.
15. Language Resource References Romanov, Maxim G. and Miller, Matthew Thomas and Savant, Sarah Bowen and Verkinderen, Peter. 2024. OpenITI: a Machine-Readable Corpus of Islamicate Texts. Zenodo, 2023.1.8. Accessed September 25, 2024. [find](https://scholar.google.com/scholar?q=Language+Resource+References+Romanov%2C+Maxim+G%2E+and+Miller%2C+Matthew+Thomas+and+Savant%2C+Sarah+Bowen+and+Verkinderen%2C+Peter%2E+2024%2E+OpenITI%3A+a+Machine%2DReadable+Corpus+of+Islamicate+Texts%2E+Zenodo%2C+2023%2E1%2E8)
16. Rustow, Marina and Koeser, Rebecca Sutton. 2025. Princeton Geniza Project. Center for Digital Humanities at Princeton, 4.28.0. Accessed March 25, 2025. Text data retrieved via [https://github.com/princetongenizalab/pgp-text](https://github.com/princetongenizalab/pgp-text). [↗](https://github.com/princetongenizalab/pgp-text)

*Updated 1 November 2026*