---
title: "Large Pre-Trained Models with Extra-Large Vocabularies — Gueta 2022"
description: "The authors train AlephBERTGimmel, a Modern Hebrew BERT with a 128K word-piece vocabulary… 12 references, cited by 23 papers here."
authors: ["Eylon Gueta", "Avi Shmidman", "Shaltiel Shmidman", "Cheyn Shmuel Shmidman", "Joshua Guedalia", "Moshe Koppel", "Dan Bareket", "Amit Seker", "Reut Tsarfaty"]
year: 2022
venue: "arXiv:2211.15199"
source: "https://arxiv.org/abs/2211.15199"
arxiv: "2211.15199"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/gueta-2022-large-pre-trained-models-extra/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gueta-2022-large-pre-trained-models-extra/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/) / Large Pre-Trained Models with Extra-Large Vocabularies: A Contrastive Analysis of Hebrew BERT Models and a New One to Outperform Them All

# Large Pre-Trained Models with Extra-Large Vocabularies: A Contrastive Analysis of Hebrew BERT Models and a New One to Outperform Them All

**Authors:** [Eylon Gueta](https://md.research.elamur.ai/free/en/authors/eylon-gueta/), [Avi Shmidman](https://md.research.elamur.ai/free/en/authors/avi-shmidman/), [Shaltiel Shmidman](https://md.research.elamur.ai/free/en/authors/shaltiel-shmidman/), [Cheyn Shmuel Shmidman](https://md.research.elamur.ai/free/en/authors/cheyn-shmuel-shmidman/), [Joshua Guedalia](https://md.research.elamur.ai/free/en/authors/joshua-guedalia/), [Moshe Koppel](https://md.research.elamur.ai/free/en/authors/moshe-koppel/), [Dan Bareket](https://md.research.elamur.ai/free/en/authors/dan-bareket/), [Amit Seker](https://md.research.elamur.ai/free/en/authors/amit-seker/) and [Reut Tsarfaty](https://md.research.elamur.ai/free/en/authors/reut-tsarfaty/) · **Year:** 2022 · **Venue:** arXiv 2211.15199 **Link:** [arxiv.org/abs/2211.15199](https://arxiv.org/abs/2211.15199) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)

[**5** citations](https://openalex.org/works?filter=cites:W4320169836) OpenAlex, 2026-10-14 · [**23** citations in our library](https://md.research.elamur.ai/free/en/papers/gueta-2022-large-pre-trained-models-extra/cited-by/) of the 140,884 papers whose bibliographies we read · [arXiv:2211.15199](https://openalex.org/S4306400194) · [licence](http://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://arxiv.org/abs/2211.15199)
- [PDF](https://arxiv.org/pdf/2211.15199)
- [arXiv 2211.15199](https://arxiv.org/abs/2211.15199)
- [OpenAlex](https://openalex.org/W4320169836)

**Authors (9)**

1. Eylon Gueta
2. [Avi Shmidman](https://openalex.org/A5013212409) [Semantic Scholar](https://www.semanticscholar.org/author/3361477) · h 5 · 114 citations · 53 works
3. [Shaltiel Shmidman](https://openalex.org/A5041361841) [Semantic Scholar](https://www.semanticscholar.org/author/1680059376) · h 3 · 27 citations · 21 works
4. [Cheyn Shmuel Shmidman](https://openalex.org/A5015862561) [Semantic Scholar](https://www.semanticscholar.org/author/2180164925) · h 3 · 13 citations · 6 works
5. [Joshua Guedalia](https://openalex.org/A5059119329) [ORCID](https://orcid.org/0000-0002-3458-7896) · [Semantic Scholar](https://www.semanticscholar.org/author/1490908142) · h 14 · 487 citations · 48 works
6. [Moshe Koppel](https://openalex.org/A5019908505) [Semantic Scholar](https://www.semanticscholar.org/author/145200175) · h 37 · 6,949 citations · 150 works
7. [Dan Bareket](https://openalex.org/A5005137956) [ORCID](https://orcid.org/0009-0008-2364-5676) · [Semantic Scholar](https://www.semanticscholar.org/author/1668035914) · h 5 · 117 citations · 15 works
8. [Amit Seker](https://openalex.org/A5049561362) [Semantic Scholar](https://www.semanticscholar.org/author/2177708) · h 7 · 181 citations · 13 works
9. [Reut Tsarfaty](https://openalex.org/A5063283689) [ORCID](https://orcid.org/0000-0001-6406-6568) · [Semantic Scholar](https://www.semanticscholar.org/author/2799181) · h 24 · 2,832 citations · 217 works

Citations and author metrics: OpenAlex, 2026-10-14; authors with the same name are not always told apart.

## What it is about

The authors train AlephBERTGimmel, a Modern Hebrew BERT with a 128K word-piece vocabulary (earlier models had 2.4K–52K), on the same ~2 billion words as AlephBERT. They also merge the UD, SPMRL and NEMO annotations of the same Hebrew texts into one consistent benchmark (UD-NEMO). The large-vocabulary model reaches the best results on segmentation, POS, morphological features, named entities and sentiment. Their analysis shows that the more pieces a word is split into, the worse every model does on it, so a vocabulary that keeps more words whole helps most, especially on semantic tasks.

## Ideas

- I2. Use the UD-NEMO test set (if its licence allows tests) to measure a Modern Hebrew parser's segmentation, and the morpheme-level "mset" score as the metric.

## For Mova

- I1. For Mova's own Hebrew analysis, look words up whole first (a full-form lexicon from the Torah trees and the dictionary) and only then split off prefixes and suffixes; report accuracy by number of segments per word, the paper's diagnostic.

The model is a neural Modern Hebrew model and cannot be part of Mova at answer time; it could at most serve as a teacher or a baseline. I1 bears on the word segmentation described in [languages/hbo/13-mova-tools.md](https://mova.elamur.ai/free/en/docs/languages/hbo/13-mova-tools/) §4 (tokens) and on the Modern Hebrew manual (`languages/he/`); I2 on the measurements in `12-biblical-mishnaic-modern.md` ("why a Modern Hebrew parser misreads the Bible").

## Institutions

[🔬**Allen Institute for Artificial Intelligence** Seattle · United States 169 works · 65,355 citations](https://md.research.elamur.ai/free/en/labs/allen-institute-for-artificial-intelligence/) · [🎓**Bar-Ilan University** Ramat Gan · Israel 102 works · 17,587 citations](https://md.research.elamur.ai/free/en/universities/bar-ilan-university/)

📚 Shelves 2[Linguistics & languages · 4,876 works](https://md.research.elamur.ai/free/en/shelves/linguistics/) · [Ancient Hebrew · 226 works](https://md.research.elamur.ai/free/en/shelves/hbo/)

🧭 Topics 2[Semitic languages · 1,334 works](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · [Hebrew & Aramaic · 438 works](https://md.research.elamur.ai/free/en/topics/hebrew-aramaic/)

## Cite

Cite as Eylon Gueta, Avi Shmidman, Shaltiel Shmidman, Cheyn Shmuel Shmidman, Joshua Guedalia, Moshe Koppel, Dan Bareket, Amit Seker and Reut Tsarfaty. 2022. Large Pre-Trained Models with Extra-Large Vocabularies: A Contrastive Analysis of Hebrew BERT Models and a New One to Outperform Them All. arXiv:2211.15199.

The paper itself, not this card. BibTeX from the library's record of the work. [arXiv 2211.15199](https://arxiv.org/abs/2211.15199)

```bibtex
@misc{gueta2022large,
  title = {{Large Pre-Trained Models with Extra-Large Vocabularies: A Contrastive Analysis of Hebrew BERT Models and a New One to Outperform Them All}},
  author = {Eylon Gueta, Avi Shmidman, Shaltiel Shmidman, Cheyn Shmuel Shmidman, Joshua Guedalia, Moshe Koppel, Dan Bareket, Amit Seker and Reut Tsarfaty},
  year = {2022},
  eprint = {2211.15199},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.15199}
}
```

- [← previous Context Matters: A Pragmatic Study of PLMs' Negation Understanding](https://md.research.elamur.ai/free/en/papers/gubelmann-2022-context-matters-pragmatic-study-plms/)
- [next → Masader: Metadata Sourcing for Arabic Text and Speech Data Resources](https://md.research.elamur.ai/free/en/papers/alyafeai-2022-masader-metadata-sourcing-arabic-text/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/)

## References

12 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Antoine Nzeyimana and Andre Niyongabo Rubungo. 2022. KinyaBERT: a morphology-aware Kinyarwanda language model. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5347 5363, Dublin, Ireland. Association for Compu tational Linguistics. [find](https://scholar.google.com/scholar?q=Antoine+Nzeyimana+and+Andre+Niyongabo+Rubungo%2E+2022%2E+KinyaBERT%3A+a+morphology%2Daware+Kinyarwanda+language+model%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computational+Linguistic)
2. Adam Amram, Anat Ben David, and Reut Tsarfaty. 2018. Representations and architectures in neural sentiment analysis for morphologically rich languages: A case study from modern hebrew. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2242 2252. [our card](https://md.research.elamur.ai/free/en/papers/amram-2018-representations-architectures-neural-sentiment-analysis/)
3. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. [our card](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/)
4. Dan Bareket and Reut Tsarfaty. 2021. Neural modeling for named entities and morphology (nemo^2). Transactions of the Association for Computational Linguistics, 9:909–928. [find](https://scholar.google.com/scholar?q=Dan+Bareket+and+Reut+Tsarfaty%2E+2021%2E+Neural+modeling+for+named+entities+and+morphology+%28nemo%5E2%29%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+9%3A909%E2%80%93928%2E)
5. Shoval Sade, Amit Seker, and Reut Tsarfaty. 2018. The hebrew universal dependency treebank: Past present and future. In Proceedings of the Second Workshop on Universal Dependencies (UDW 2018), pages 133 143. [our card](https://md.research.elamur.ai/free/en/papers/sade-2018-hebrew-universal-dependency-treebank-past/)
6. Idan Brusilovsky and Reut Tsarfaty. 2022. Neural token segmentation for high token-internal complexity. arXiv preprint arXiv:2203.10845. [↗](https://arxiv.org/abs/2203.10845)
7. Djamé Seddah, Reut Tsarfaty, Sandra Kübler, Marie Candito, Jinho D. Choi, Richárd Farkas, Jennifer Foster, Iakes Goenaga, Koldo Gojenola Galletebeitia, Yoav Goldberg, Spence Green, Nizar Habash, Marco Kuhlmann, Wolfgang Maier, Joakim Nivre, Adam Przepiórkowski, Ryan Roth, Wolfgang Seeker, Yannick Versley, Veronika Vincze, Marcin Woliński, Alina Wróblewska, and Eric Villemonte de la Clergerie. 2013. Overview of the SPMRL 2013 shared task: A cross-framework evaluation of parsing morphologically rich languages. In Proceedings of the Fourth Workshop on Statistical Parsing of Morphologically-Rich Languages, pages 146–182, Seattle, Washington, USA. Association for Computa tional Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/seddah-2013-overview-spmrl-2013-shared-task/)
8. Avihay Chriqui and Inbal Yahav. 2021. Hebert & hebemo: a hebrew bert model and a tool for polarity analysis and emotion recognition. [our card](https://md.research.elamur.ai/free/en/papers/chriqui-2021-hebert-hebemo-hebrew-bert-model/)
9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1423) [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
10. Omri Keren, Tal Avinari, Reut Tsarfaty, and Omer Levy. 2022. Breaking character: Are subwords good enough for mrls after all? [find](https://scholar.google.com/scholar?q=Omri+Keren%2C+Tal+Avinari%2C+Reut+Tsarfaty%2C+and+Omer+Levy%2E+2022%2E+Breaking+character%3A+Are+subwords+good+enough+for+mrls+after+all%3F)
11. Amit Seker, Elron Bandel, Dan Bareket, Idan Brusilovsky, Refael Shaked Greenfeld, and Reut Tsarfaty. 2021. Alephbert: A hebrew large pre-trained language model to start-off your hebrew NLP application with. CoRR, abs/2104.04052. [↗](https://arxiv.org/abs/2104.04052) [our card](https://md.research.elamur.ai/free/en/papers/seker-2021-alephbert-hebrew-large-pre-trained/)
12. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State of the-art natural language processing. CoRR, abs/1910.03771. [↗](https://arxiv.org/abs/1910.03771) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)

Updated 23 October 2026