---
title: "Vocab Diet — paper by Reif et al. (2026) — summary and ideas"
description: "The authors first test whether LLM embeddings already encode this variation as linear offsets. 98 references, cited by 1 paper here."
authors: ["Yuval Reif", "Guy Kaplan", "Roy Schwartz"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.1618/"
doi: "10.18653/v1/2026.findings-acl.1618"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/reif-2026-vocab-diet-reshaping-vocabulary-llms/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/reif-2026-vocab-diet-reshaping-vocabulary-llms/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/19/) / Vocab Diet: Reshaping the Vocabulary of LLMs via Vector Arithmetic

# Vocab Diet: Reshaping the Vocabulary of LLMs via Vector Arithmetic

**Authors:** Yuval Reif, Guy Kaplan and [Roy Schwartz](https://md.research.elamur.ai/free/en/authors/roy-schwartz/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1618/) · [PDF](https://aclanthology.org/2026.findings-acl.1618.pdf) · [DOI 10.18653/v1/2026.findings-acl.1618](https://doi.org/10.18653/v1/2026.findings-acl.1618) · [OpenAlex](https://openalex.org/W4415961110)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary

BPE-style tokenizers give separate vocabulary slots to many surface variants of one word, such as inflections, derivations and capitalized forms. This uses up a capped vocabulary that could otherwise cover rare words, other languages and other domains.

The authors first test whether LLM embeddings already encode this variation as linear offsets. They collect base/surface pairs from UniMorph for five morphologically diverse languages and average the offset for each relation. Using Patchscopes-style interpretation, they find that a base embedding plus the offset is usually read as the expected form. This holds even when the target word has no token of its own.

Building on this, they restructure the vocabulary. Both the input embedding and the output logit of a word are computed as a base-form component plus a small set of transformation components. The scheme works as a post-hoc adaptation with the backbone frozen, and also when pretraining from scratch. It frees 10–40% of token slots and extends coverage to out-of-vocabulary forms, with little loss on downstream tasks.

This shows that a fixed vocabulary budget can be spent much more efficiently, which matters especially for multilingual models.

## Key points

- **Redundancy in the GPT-4 tokenizer:** about 24.6k English whole-word tokens shrink to 17.7k when case is ignored. Merging inflectional and derivational relations leaves 14.3k base forms, a 42% reduction. The same bases and transformations could also compose about 98k words that are currently out of vocabulary.
- **Decomposition map and output scoring:** the map is built from UniMorph tags (e.g., V;PST) plus extra rules for capitalization. At output, the unembedding matrix is split into a base part and a transformation part, and a single softmax runs over all composable words.
- **Post-hoc regime:** only the transformation embeddings and LoRA adapters on the last 8 transformer blocks are trained. Across five models and five languages, this removes up to about 10% of vocabulary tokens while mostly preserving downstream performance.
- **Pretraining regime:** proof-of-concept models trained from scratch drop 41% of BPE vocabulary entries with comparable performance.
- **Linear morphology and vocabulary size:** models with small English vocabularies (8–10k words, e.g., Llama2, Mistral) encode morphology as consistent linear offsets. Models with large ones (about 40k, e.g., Falcon3, Gemma2-9B) store inflected forms more as separate units, and weight tying strengthens this trend.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Hebrew University of Jerusalem](https://md.research.elamur.ai/free/en/universities/hebrew-university-of-jerusalem/) — University, Jerusalem, Israel · 65 works · 9,387 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🧩 Morphology](https://md.research.elamur.ai/free/en/topics/morphology/) · 405 works

## Cite

Cite as: Yuval Reif, Guy Kaplan and Roy Schwartz. 2026. Vocab Diet: Reshaping the Vocabulary of LLMs via Vector Arithmetic. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{reif2026vocab,
  title = {{Vocab Diet: Reshaping the Vocabulary of LLMs via Vector Arithmetic}},
  author = {Yuval Reif, Guy Kaplan and Roy Schwartz},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1618},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1618}
}
```

- [← previous VLN-NF: Feasibility-Aware Vision-and-Language Navigation with False-Premise Instructions](https://md.research.elamur.ai/free/en/papers/su-2026-vln-nf-feasibility-aware-vision/)
- [next → VocabTailor: Dynamic Vocabulary Selection for Downstream Tasks in Small Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2026-vocabtailor-dynamic-vocabulary-selection-downstream/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/19/)

## References

98 references, 46 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. [Farasa: A fast and furious segmenter for Arabic](https://md.research.elamur.ai/free/en/papers/abdelali-2016-farasa-fast-furious-segmenter-arabic/). In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 11–16, San Diego, California. Associati…
2. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, and 1 others. 2024. [Phi-4 technical report](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-4-technical-report/). arXiv preprint arXiv:2412.08905.
3. Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. 2023. [Do all languages cost the same? tokenization in the era of commercial language models](https://md.research.elamur.ai/free/en/papers/ahia-2023-do-all-languages-cost-same/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9904…
4. Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max Lübbering, Johannes Leveling, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Buschhoff, Charvi Jain, Alexander Weber, Lena Jurkschat, Hammam Abdelwahab, Chelsea John, Pedro Ortiz Suarez, Malte Ostendorff, Samuel Weinbach, Rafet Sifa, and… — [Tokenizer Choice For LLM Training: Negligible or Crucial?](https://md.research.elamur.ai/free/en/papers/ali-2024-tokenizer-choice-llm-training-negligible/)
5. Mohamed Taher Alrefaie, Nour Eldin Morsy, and Nada Samir. 2024. [Exploring tokenization strategies and vocabulary sizes for enhanced arabic language models](https://md.research.elamur.ai/free/en/papers/alrefaie-2024-exploring-tokenization-strategies-vocabulary-sizes/). arXiv preprint arXiv:2403.11130.
6. Zaid Alyafeai, Maged S. Al-Shaibani, Mustafa Ghaleb, and Irfan Ahmad. 2021. [Evaluating various tokenizers for arabic text classification](https://md.research.elamur.ai/free/en/papers/alyafeai-2021-evaluating-various-tokenizers-arabic-text/). Neural Processing Letters, 55:2911–2933.
7. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. [On the cross-lingual transferability of monolingual representations](https://md.research.elamur.ai/free/en/papers/artetxe-2020-cross-lingual-transferability-monolingual-representations/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637, Online. Association for Computational Linguistics.
8. Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Jon Ander Campos, Yi Chern Tan, and 1 others. 2024. [Aya 23: Open weight releases to further multilingual progress](https://md.research.elamur.ai/free/en/papers/aryabumi-2024-aya-23-open-weight-releases/). arXiv preprint arXiv:2405.15032.
9. Ehsaneddin Asgari, Yassine El Kheir, and Mohammad Ali Sadraei Javaheri. 2025. MorphBPE: A morpho-aware tokenizer bridging linguistic complexity for efficient llm training across morphologies. Preprint, arXiv:2502.00894. — [arXiv:2502.00894](https://arxiv.org/abs/2502.00894) [find](https://scholar.google.com/scholar?q=Ehsaneddin+Asgari%2C+Yassine+El+Kheir%2C+and+Mohammad+Ali+Sadraei+Javaheri%2E+2025%2E+MorphBPE%3A+A+morpho%2Daware+tokenizer+bridging+linguistic+complexity+for+efficient+llm+training+across+morphologies%2E+Preprint)
10. M Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham Abdullah Alyahya, Sultan AlRashed, Faisal Abdulrahman Mirza, Shaykhah Z. Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef Almushayqih, Raneem Alnajim, Salman Alsubaihi, Maryam Al Mansour, Saad Ami… [find](https://scholar.google.com/scholar?q=M+Saiful+Bari%2C+Yazeed+Alnumay%2C+Norah+A%2E+Alzahrani%2C+Nouf+M%2E+Alotaibi%2C+Hisham+Abdullah+Alyahya%2C+Sultan+AlRashed%2C+Faisal+Abdulrahman+Mirza%2C+Shaykhah+Z%2E+Alsubaie%2C+Hassan+A%2E+Alahmed%2C+Ghadah+Alabduljabbar%2C+)
11. In The Thirteenth International Conference on Learning Representations. Khuyagbaatar Batsuren, Omer Goldman, Salam Khalifa, Nizar Habash, Witold Kieraś, Gábor Bella, Brian Leonard, Garrett Nicolai, Kyle Gorman, Yustinus Ghanggo Ate, Maria Ryskina, Sabrina Mielke, Elena Budianskaya, Charbel El-Khais… — [UniMorph 4.0: Universal Morphology](https://md.research.elamur.ai/free/en/papers/batsuren-2022-unimorph-4/)
12. Thomas Bauwens and Pieter Delobelle. 2024. BPE-knockout: Pruning pre-existing BPE tokenisers with backwards-compatible morphological semi-supervision. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Vo… [find](https://scholar.google.com/scholar?q=Thomas+Bauwens+and+Pieter+Delobelle%2E+2024%2E+BPE%2Dknockout%3A+Pruning+pre%2Dexisting+BPE+tokenisers+with+backwards%2Dcompatible+morphological+semi%2Dsupervision%2E+In+Proceedings+of+the+2024+Conference+of+the+Nort)
13. Amit Ben-Artzy and Roy Schwartz. 2025. Spellm: Character-level multi-head decoding. Preprint, arXiv:2507.16323. — [arXiv:2507.16323](https://arxiv.org/abs/2507.16323) [find](https://scholar.google.com/scholar?q=Amit+Ben%2DArtzy+and+Roy+Schwartz%2E+2025%2E+Spellm%3A+Character%2Dlevel+multi%2Dhead+decoding%2E+Preprint%2C+arXiv%3A2507%2E16323%2E+%E2%80%94+%5BarXiv%3A2507%2E16323%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2507%2E16323%29)
14. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. [Piqa: Reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). In Thirty-Fourth AAAI Conference on Artificial Intelligence.
15. Terra Blevins and Luke Zettlemoyer. 2019. Better character language modeling through morphology. arXiv preprint arXiv:1906.01037. [find](https://scholar.google.com/scholar?q=Terra+Blevins+and+Luke+Zettlemoyer%2E+2019%2E+Better+character+language+modeling+through+morphology%2E+arXiv+preprint+arXiv%3A1906%2E01037%2E)
16. Danqi Chen, Jason Bolton, and Christopher D. Manning. 2016. [A thorough examination of the CNN/Daily Mail reading comprehension task](https://md.research.elamur.ai/free/en/papers/chen-2016-thorough-examination-cnn-daily-mail/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2358–2367, Berlin, Germany. Association for C…
17. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. [BoolQ: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:…
18. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). ArXiv, abs/1803.05457.
19. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Bel… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Ruty+Rinott%2C+Guillaume+Lample%2C+Adina+Williams%2C+Samuel+Bowman%2C+Holger+Schwenk%2C+and+Veselin+Stoyanov%2E+2018%2E+XNLI%3A+Evaluating+cross%2Dlingual+sentence+representations%2E+In+Proceedings+of+the)
20. Mathias Creutz and Krista Lagus. 2002. [Unsupervised discovery of morphemes](https://md.research.elamur.ai/free/en/papers/creutz-2002-unsupervised-discovery-morphemes/). In Proceedings of the ACL-02 Workshop on Morphological and Phonological Learning, pages 21–30. Association for Computational Linguistics.
21. Mathias Creutz and Krista Lagus. 2007. Unsupervised models for morpheme segmentation and morphology learning. ACM Transactions on Speech and Language Processing (TSLP), 4(1):1–34. [find](https://scholar.google.com/scholar?q=Mathias+Creutz+and+Krista+Lagus%2E+2007%2E+Unsupervised+models+for+morpheme+segmentation+and+morphology+learning%2E+ACM+Transactions+on+Speech+and+Language+Processing+%28TSLP%29%2C+4%281%29%3A1%E2%80%9334%2E)
22. Yiming Cui, Wanxiang Che, Shijin Wang, and Ting Liu. 2022. Lert: A linguistically-motivated pre-trained language model. arXiv preprint arXiv:2211.05344. — [arXiv:2211.05344](https://arxiv.org/abs/2211.05344) [find](https://scholar.google.com/scholar?q=Yiming+Cui%2C+Wanxiang+Che%2C+Shijin+Wang%2C+and+Ting+Liu%2E+2022%2E+Lert%3A+A+linguistically%2Dmotivated+pre%2Dtrained+language+model%2E+arXiv+preprint+arXiv%3A2211%2E05344%2E+%E2%80%94+%5BarXiv%3A2211%2E05344%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2211)
23. Gautier Dagan, Gabriele Synnaeve, and Baptiste Rozière. 2024. [Getting the most out of your tokenizer for pre-training and domain adaptation](https://md.research.elamur.ai/free/en/papers/dagan-2024-getting-most-out-tokenizer-pre/). arXiv preprint arXiv:2402.01035.
24. John Dang, Shivalika Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, and 1 others. 2024. [Aya expanse: Combining research breakthroughs for a new multilingual frontier](https://md.research.elamur.ai/free/en/papers/dang-2024-aya-expanse-combining-research-breakthroughs/). arXiv preprint arXiv:2412.04261.
25. Konstantin Dobler and Gerard de Melo. 2023. [FOCUS: Effective embedding initialization for monolingual specialization of multilingual models](https://md.research.elamur.ai/free/en/papers/dobler-2023-focus-effective-embedding-initialization-monolingual/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13440–13454, Singapore. Association for Computational Linguist…
26. Sheridan Feucht, David Atkinson, Byron C Wallace, and David Bau. 2024. Token erasure as a footprint of implicit vocabulary items in LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9727–9739, Miami, Florida, USA. Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Sheridan+Feucht%2C+David+Atkinson%2C+Byron+C+Wallace%2C+and+David+Bau%2E+2024%2E+Token+erasure+as+a+footprint+of+implicit+vocabulary+items+in+LLMs%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+)
27. Negar Foroutan, Clara Meister, Debjit Paul, Joel Niklaus, Sina Ahmadi, Antoine Bosselut, and Rico Sennrich. 2025. [Parity-aware byte-pair encoding: Improving cross-lingual fairness in tokenization](https://md.research.elamur.ai/free/en/papers/foroutan-2026-parity-aware-byte-pair-encoding/).
28. Guobing Gan, Peng Zhang, Sunzhu Li, Xiuqing Lu, and Benyou Wang. 2022. Morphte: Injecting morphology in tensorized embeddings. Advances in Neural Information Processing Systems, 35:33186–33200. [find](https://scholar.google.com/scholar?q=Guobing+Gan%2C+Peng+Zhang%2C+Sunzhu+Li%2C+Xiuqing+Lu%2C+and+Benyou+Wang%2E+2022%2E+Morphte%3A+Injecting+morphology+in+tensorized+embeddings%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A33186%E2%80%9333200%2E)
29. Bar Gazit, Shaltiel Shmidman, Avi Shmidman, and Yuval Pinter. 2025. Splintering nonconcatenative languages for better tokenization. In Findings of the Association for Computational Linguistics: ACL 2025, pages 22405–22417, Vienna, Austria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Bar+Gazit%2C+Shaltiel+Shmidman%2C+Avi+Shmidman%2C+and+Yuval+Pinter%2E+2025%2E+Splintering+nonconcatenative+languages+for+better+tokenization%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL+20)
30. Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. 2024. Patchscopes: A unifying framework for inspecting hidden representations of language models. arXiv preprint arXiv:2401.06102. [find](https://scholar.google.com/scholar?q=Asma+Ghandeharioun%2C+Avi+Caciularu%2C+Adam+Pearce%2C+Lucas+Dixon%2C+and+Mor+Geva%2E+2024%2E+Patchscopes%3A+A+unifying+framework+for+inspecting+hidden+representations+of+language+models%2E+arXiv+preprint+arXiv%3A2401%2E0)
31. Anna Gladkova, Aleksandr Drozd, and Satoshi Matsuoka. 2016. [Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn’t](https://md.research.elamur.ai/free/en/papers/gladkova-2016-analogy-based-detection-morphological-semantic/). In Proceedings of the NAACL Student Research Workshop, pages 8–15.
32. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
33. Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. 2023. Finding neurons in a haystack: Case studies with sparse probing. Transactions on Machine Learning Research. [find](https://scholar.google.com/scholar?q=Wes+Gurnee%2C+Neel+Nanda%2C+Matthew+Pauly%2C+Katherine+Harvey%2C+Dmitrii+Troitskii%2C+and+Dimitris+Bertsimas%2E+2023%2E+Finding+neurons+in+a+haystack%3A+Case+studies+with+sparse+probing%2E+Transactions+on+Machine+Learn)
34. HyoJung Han, Akiko Eriguchi, Haoran Xu, Hieu Hoang, Marine Carpuat, and Huda Khayrallah. 2025. Adapters for altering LLM vocabularies: What languages benefit the most? In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=HyoJung+Han%2C+Akiko+Eriguchi%2C+Haoran+Xu%2C+Hieu+Hoang%2C+Marine+Carpuat%2C+and+Huda+Khayrallah%2E+2025%2E+Adapters+for+altering+LLM+vocabularies%3A+What+languages+benefit+the+most%3F+In+The+Thirteenth+International+)
35. Roee Hendel, Mor Geva, and Amir Globerson. 2023. [In-context learning creates task vectors](https://md.research.elamur.ai/free/en/papers/hendel-2023-context-learning-creates-task-vectors/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9318–9333, Singapore. Association for Computational Linguistics.
36. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). Proceedings of the International Conference on Learning Representations (ICLR).
37. Evan Hernandez, Belinda Z. Li, and Jacob Andreas. 2024. Inspecting and editing knowledge representations in language models. In First Conference on Language Modeling. [find](https://scholar.google.com/scholar?q=Evan+Hernandez%2C+Belinda+Z%2E+Li%2C+and+Jacob+Andreas%2E+2024%2E+Inspecting+and+editing+knowledge+representations+in+language+models%2E+In+First+Conference+on+Language+Modeling%2E)
38. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. [Distilling the knowledge in a neural network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/). arXiv preprint arXiv:1503.02531.
39. Valentin Hofmann, Janet Pierrehumbert, and Hinrich Schütze. 2020. DagoBERT: Generating derivational morphology with a pretrained language model. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3848–3861, Online. Association for Computational L… [find](https://scholar.google.com/scholar?q=Valentin+Hofmann%2C+Janet+Pierrehumbert%2C+and+Hinrich+Sch%C3%BCtze%2E+2020%2E+DagoBERT%3A+Generating+derivational+morphology+with+a+pretrained+language+model%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Meth)
40. Valentin Hofmann, Janet Pierrehumbert, and Hinrich Schütze. 2021. [Superbizarre is not superb: Derivational morphology improves BERT’s interpretation of complex words](https://md.research.elamur.ai/free/en/papers/hofmann-2021-superbizarre-derivational/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conferenc…
41. Valentin Hofmann, Hinrich Schuetze, and Janet Pierrehumbert. 2022. An embarrassingly simple method to mitigate undesirable properties of pretrained language model tokenizers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 38… [find](https://scholar.google.com/scholar?q=Valentin+Hofmann%2C+Hinrich+Schuetze%2C+and+Janet+Pierrehumbert%2E+2022%2E+An+embarrassingly+simple+method+to+mitigate+undesirable+properties+of+pretrained+language+model+tokenizers%2E+In+Proceedings+of+the+60t)
42. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. [The curious case of neural text degeneration](https://md.research.elamur.ai/free/en/papers/holtzman-2019-curious-case-neural-text-degeneration/). In International Conference on Learning Representations.
43. Hongzhi Huang, Defa Zhu, Banggu Wu, Yutao Zeng, Ya Wang, Qiyang Min, and Xun Zhou. 2025. Over-tokenized transformer: Vocabulary is generally worth scaling. arXiv preprint arXiv:2501.16975. — [arXiv:2501.16975](https://arxiv.org/abs/2501.16975) [find](https://scholar.google.com/scholar?q=Hongzhi+Huang%2C+Defa+Zhu%2C+Banggu+Wu%2C+Yutao+Zeng%2C+Ya+Wang%2C+Qiyang+Min%2C+and+Xun+Zhou%2E+2025%2E+Over%2Dtokenized+transformer%3A+Vocabulary+is+generally+worth+scaling%2E+arXiv+preprint+arXiv%3A2501%2E16975%2E+%E2%80%94+%5BarXiv%3A25)
44. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
45. Keller Jordan, Jeremy Bernstein, Brendan Rappazzo, @fernbear.bsky.social, Boza Vlado, You Jiacheng, Franz Cesista, Braden Koszarsky, and @Grad62304977. 2024. modded-nanogpt: Speedrunning the nanogpt baseline. [find](https://scholar.google.com/scholar?q=Keller+Jordan%2C+Jeremy+Bernstein%2C+Brendan+Rappazzo%2C+%40fernbear%2Ebsky%2Esocial%2C+Boza+Vlado%2C+You+Jiacheng%2C+Franz+Cesista%2C+Braden+Koszarsky%2C+and+%40Grad62304977%2E+2024%2E+modded%2Dnanogpt%3A+Speedrunning+the+nanogpt+b)
46. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. [TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver…
47. Guy Kaplan, Matanel Oren, Yuval Reif, and Roy Schwartz. 2025. From tokens to words: On the inner lexicon of LLMs. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Guy+Kaplan%2C+Matanel+Oren%2C+Yuval+Reif%2C+and+Roy+Schwartz%2E+2025%2E+From+tokens+to+words%3A+On+the+inner+lexicon+of+LLMs%2E+In+The+Thirteenth+International+Conference+on+Learning+Representations%2E)
48. Pride Kavumba, Naoya Inoue, Benjamin Heinzerling, Keshav Singh, Paul Reisert, and Kentaro Inui. 2019. When choosing plausible alternatives, clever hans can be clever. In Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing, pages 33–42, Hong Kong, China. Associat… [find](https://scholar.google.com/scholar?q=Pride+Kavumba%2C+Naoya+Inoue%2C+Benjamin+Heinzerling%2C+Keshav+Singh%2C+Paul+Reisert%2C+and+Kentaro+Inui%2E+2019%2E+When+choosing+plausible+alternatives%2C+clever+hans+can+be+clever%2E+In+Proceedings+of+the+First+Works)
49. Seungduk Kim, Seungtaek Choi, and Myeongho Jeong. 2024. [Efficient and effective vocabulary expansion towards multilingual large language models](https://md.research.elamur.ai/free/en/papers/kim-2024-efficient-effective-vocabulary-expansion-towards/). arXiv preprint arXiv:2402.14714.
50. Stav Klein and Reut Tsarfaty. 2020. Getting the \#\#life out of living: How adequate are word-pieces for modelling complex morphology? In Proceedings of the 17th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 204–209, Online. Association for Computational… [find](https://scholar.google.com/scholar?q=Stav+Klein+and+Reut+Tsarfaty%2E+2020%2E+Getting+the+%5C%23%5C%23life+out+of+living%3A+How+adequate+are+word%2Dpieces+for+modelling+complex+morphology%3F+In+Proceedings+of+the+17th+SIGMORPHON+Workshop+on+Computational+R)
51. Paul Lerner and François Yvon. 2025. Unlike “likely”, “unlike” is unlikely: BPE-based segmentation hurts morphological derivations in LLMs. In Proceedings of the 31st International Conference on Computational Linguistics, pages 5181–5190, Abu Dhabi, UAE. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Paul+Lerner+and+Fran%C3%A7ois+Yvon%2E+2025%2E+Unlike+%E2%80%9Clikely%E2%80%9D%2C+%E2%80%9Cunlike%E2%80%9D+is+unlikely%3A+BPE%2Dbased+segmentation+hurts+morphological+derivations+in+LLMs%2E+In+Proceedings+of+the+31st+International+Conference+on+Compu)
52. Omer Levy and Yoav Goldberg. 2014. [Linguistic regularities in sparse and explicit word representations](https://md.research.elamur.ai/free/en/papers/levy-2014-linguistic-regularities-sparse-explicit-word/). In Conference on Computational Natural Language Learning.
53. Tomasz Limisiewicz, Jivr’i Balhar, and David Marevcek. 2023. Tokenization impacts multilingual language modeling: Assessing vocabulary allocation and overlap across languages. In Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tomasz+Limisiewicz%2C+Jivr%E2%80%99i+Balhar%2C+and+David+Marevcek%2E+2023%2E+Tokenization+impacts+multilingual+language+modeling%3A+Assessing+vocabulary+allocation+and+overlap+across+languages%2E+In+Annual+Meeting+of+the)
54. Alisa Liu, Jonathan Hayase, Valentin Hofmann, Sewoong Oh, Noah A. Smith, and Yejin Choi. 2025. SuperBPE: Space travel for language models. ArXiv, abs/2503.13423. — [arXiv:2503.13423](https://arxiv.org/abs/2503.13423) [find](https://scholar.google.com/scholar?q=Alisa+Liu%2C+Jonathan+Hayase%2C+Valentin+Hofmann%2C+Sewoong+Oh%2C+Noah+A%2E+Smith%2C+and+Yejin+Choi%2E+2025%2E+SuperBPE%3A+Space+travel+for+language+models%2E+ArXiv%2C+abs%2F2503%2E13423%2E+%E2%80%94+%5BarXiv%3A2503%2E13423%5D%28https%3A%2F%2Farxiv%2Eorg)
55. Chengyuan Liu, Shihang Wang, Lizhi Qing, Kun Kuang, Yangyang Kang, Changlong Sun, and Fei Wu. 2024a. Gold panning in vocabulary: An adaptive method for vocabulary expansion of domain-specific LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7442–… [find](https://scholar.google.com/scholar?q=Chengyuan+Liu%2C+Shihang+Wang%2C+Lizhi+Qing%2C+Kun+Kuang%2C+Yangyang+Kang%2C+Changlong+Sun%2C+and+Fei+Wu%2E+2024a%2E+Gold+panning+in+vocabulary%3A+An+adaptive+method+for+vocabulary+expansion+of+domain%2Dspecific+LLMs%2E+In)
56. Yihong Liu, Peiqin Lin, Mingyang Wang, and Hinrich Schuetze. 2024b. OFA: A framework of initializing unseen subword embeddings for efficient large-scale multilingual continued pretraining. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 1067–1097, Mexico City, Mexico.… [find](https://scholar.google.com/scholar?q=Yihong+Liu%2C+Peiqin+Lin%2C+Mingyang+Wang%2C+and+Hinrich+Schuetze%2E+2024b%2E+OFA%3A+A+framework+of+initializing+unseen+subword+embeddings+for+efficient+large%2Dscale+multilingual+continued+pretraining%2E+In+Findings)
57. Manuel Mager, Arturo Oncevay, Elisabeth Mager, Katharina Kann, and Thang Vu. 2022. BPE vs. morphological segmentation: A case study on machine translation of four polysynthetic languages. In Findings of the Association for Computational Linguistics: ACL 2022, pages 961–971, Dublin, Ireland. Associat… [find](https://scholar.google.com/scholar?q=Manuel+Mager%2C+Arturo+Oncevay%2C+Elisabeth+Mager%2C+Katharina+Kann%2C+and+Thang+Vu%2E+2022%2E+BPE+vs%2E+morphological+segmentation%3A+A+case+study+on+machine+translation+of+four+polysynthetic+languages%2E+In+Findings+)
58. Marion Di Marco and Alexander Fraser. 2024. Subword segmentation in LLMs: Looking at inflection and consistency. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12050–12060, Miami, Florida, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Marion+Di+Marco+and+Alexander+Fraser%2E+2024%2E+Subword+segmentation+in+LLMs%3A+Looking+at+inflection+and+consistency%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Natural+Language+Processi)
59. Samuel Marks and Max Tegmark. 2024. [The geometry of truth: Emergent linear structure in large language model representations of true/false datasets](https://md.research.elamur.ai/free/en/papers/marks-2023-geometry-truth-emergent-linear-structure/). In First Conference on Language Modeling.
60. Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M Guerreiro, Ricardo Rei, Duarte M Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, and 1 others. 2025. [Eurollm: Multilingual language models for europe](https://md.research.elamur.ai/free/en/papers/martins-2024-eurollm-multilingual-language-models-europe/). Procedia Computer Science, 255:53–62.
61. Austin Matthews, Graham Neubig, and Chris Dyer. 2018. Using morphological knowledge in open-vocabulary neural language models. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pa… [find](https://scholar.google.com/scholar?q=Austin+Matthews%2C+Graham+Neubig%2C+and+Chris+Dyer%2E+2018%2E+Using+morphological+knowledge+in+open%2Dvocabulary+neural+language+models%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+th)
62. Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. 2023. Language models implement simple word2vec-style vector arithmetic. arXiv preprint arXiv:2305.16130. [find](https://scholar.google.com/scholar?q=Jack+Merullo%2C+Carsten+Eickhoff%2C+and+Ellie+Pavlick%2E+2023%2E+Language+models+implement+simple+word2vec%2Dstyle+vector+arithmetic%2E+arXiv+preprint+arXiv%3A2305%2E16130%2E)
63. Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013a. [Distributed representations of words and phrases and their compositionality](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/). In Neural Information Processing Systems.
64. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013b. [Linguistic regularities in continuous space word representations](https://md.research.elamur.ai/free/en/papers/mikolov-2013-linguistic-regularities-continuous-space-word/). In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 746–751, Atlanta, Georgia.…
65. Benjamin Minixhofer, Edoardo Ponti, and Ivan Vulić. 2024. Zero-shot tokenizer transfer. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Benjamin+Minixhofer%2C+Edoardo+Ponti%2C+and+Ivan+Vulic%CC%81%2E+2024%2E+Zero%2Dshot+tokenizer+transfer%2E+In+The+Thirty%2Deighth+Annual+Conference+on+Neural+Information+Processing+Systems%2E)
66. Itay Nakash, Nitay Calderon, Eyal Ben-David, Elad Hoffer, and Roi Reichart. 2025. Adaptivocab: Enhancing LLM efficiency in focused domains through lightweight vocabulary adaptation. In Second Conference on Language Modeling. [find](https://scholar.google.com/scholar?q=Itay+Nakash%2C+Nitay+Calderon%2C+Eyal+Ben%2DDavid%2C+Elad+Hoffer%2C+and+Roi+Reichart%2E+2025%2E+Adaptivocab%3A+Enhancing+LLM+efficiency+in+focused+domains+through+lightweight+vocabulary+adaptation%2E+In+Second+Conferen)
67. Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, and 21 others. 2024. 2 ol… [find](https://scholar.google.com/scholar?q=Team+OLMo%2C+Pete+Walsh%2C+Luca+Soldaini%2C+Dirk+Groeneveld%2C+Kyle+Lo%2C+Shane+Arora%2C+Akshita+Bhagia%2C+Yuling+Gu%2C+Shengyi+Huang%2C+Matt+Jordan%2C+Nathan+Lambert%2C+Dustin+Schwenk%2C+Oyvind+Tafjord%2C+Taira+Anderson%2C+Davi)
68. OpenAI. 2024. tiktoken: A fast BPE tokeniser for use with openai’s models. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2024%2E+tiktoken%3A+A+fast+BPE+tokeniser+for+use+with+openai%E2%80%99s+models%2E)
69. Hyunji Hayley Park, Katherine J. Zhang, Coleman Haley, Kenneth Steimel, Han Liu, and Lane Schwartz. 2021. [Morphology matters: A multilingual language modeling analysis](https://md.research.elamur.ai/free/en/papers/park-2021-morphology-matters-multilingual-language-modeling/). Transactions of the Association for Computational Linguistics, 9:261–276.
70. Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. 2025. The geometry of categorical and hierarchical concepts in large language models. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Kiho+Park%2C+Yo+Joong+Choe%2C+Yibo+Jiang%2C+and+Victor+Veitch%2E+2025%2E+The+geometry+of+categorical+and+hierarchical+concepts+in+large+language+models%2E+In+The+Thirteenth+International+Conference+on+Learning+Re)
71. Kiho Park, Yo Joong Choe, and Victor Veitch. 2024. [The linear representation hypothesis and the geometry of large language models](https://md.research.elamur.ai/free/en/papers/park-2023-linear-representation-hypothesis-geometry-large/). In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 39643–39666. PMLR.
72. Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, and 1 others. 2024. [The fineweb datasets: Decanting the web for the finest text data at scale](https://md.research.elamur.ai/free/en/papers/penedo-2024-fineweb-datasets-decanting-web-finest/). Advances in Neural Information Processing Systems, 37:30811–30849.
73. Guilherme Penedo, Hynek Kydlíček, Vinko Sabolčec, Bettina Messmer, Negar Foroutan, Amir Hossein Kargaran, Colin Raffel, Martin Jaggi, Leandro Von Werra, and Thomas Wolf. 2025. [Fineweb2: One pipeline to scale them all — adapting pre-training data processing to every language](https://md.research.elamur.ai/free/en/papers/penedo-2025-fineweb2-one-pipeline-scale-them/). In Second Conference o…
74. Hao Peng, Roy Schwartz, and Noah A. Smith. 2019. PaLM: A hybrid parser and language model. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3644– 3651, Hong Kong,… [find](https://scholar.google.com/scholar?q=Hao+Peng%2C+Roy+Schwartz%2C+and+Noah+A%2E+Smith%2E+2019%2E+PaLM%3A+A+hybrid+parser+and+language+model%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+the+9th+Interna)
75. Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi. 2023. [Language model tokenizers introduce unfairness between languages](https://md.research.elamur.ai/free/en/papers/petrov-2023-language-model-tokenizers-introduce-unfairness/). Advances in neural information processing systems, 36:36963– 36990.
76. Marko Pranjić, Marko Robnik-Šikonja, and Senja Pollak. 2024. LLMSegm: Surface-level morphological segmentation using large language model. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 10665– 10674… [find](https://scholar.google.com/scholar?q=Marko+Pranjic%CC%81%2C+Marko+Robnik%2D%C5%A0ikonja%2C+and+Senja+Pollak%2E+2024%2E+LLMSegm%3A+Surface%2Dlevel+morphological+segmentation+using+large+language+model%2E+In+Proceedings+of+the+2024+Joint+International+Conference+on)
77. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E)
78. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [SQuAD: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
79. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2021%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+Communications+of+the+ACM%2C+64%289%29%3A99%E2%80%93106%2E)
80. Lisa Schut, Yarin Gal, and Sebastian Farquhar. 2025. Do multilingual llms think in english? arXiv preprint arXiv:2502.15603. — [arXiv:2502.15603](https://arxiv.org/abs/2502.15603) [find](https://scholar.google.com/scholar?q=Lisa+Schut%2C+Yarin+Gal%2C+and+Sebastian+Farquhar%2E+2025%2E+Do+multilingual+llms+think+in+english%3F+arXiv+preprint+arXiv%3A2502%2E15603%2E+%E2%80%94+%5BarXiv%3A2502%2E15603%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2502%2E15603%29)
81. Amit Seker, Elron Bandel, Dan Bareket, Idan Brusilovsky, Refael Greenfeld, and Reut Tsarfaty. 2022. [AlephBERT: Language model pre-training and evaluation from sub-word to sentence level](https://md.research.elamur.ai/free/en/papers/seker-2022-alephbert-language-model-pre-training/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers…
82. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Lin…
83. Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Sebastian Ruder, Wei-Yin Ko, Antoine Bosselut, Alice Oh, Andre Martins, Leshem Chosh… — [Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation](https://md.research.elamur.ai/free/en/papers/singh-2025-global-mmlu-understanding-addressing-cultural/)
84. Peter Smit, Sami Virpioja, Stig-Arne Grönroos, and Mikko Kurimo. 2014. [Morfessor 2.0: Toolkit for statistical morphological segmentation](https://md.research.elamur.ai/free/en/papers/smit-2014-morfessor-2-0-toolkit-statistical/). In Proceedings of the Demonstrations at the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 21–24, Gothenburg, Swe…
85. Nishant Subramani, Nivedita Suresh, and Matthew Peters. 2022. [Extracting latent steering vectors from pretrained language models](https://md.research.elamur.ai/free/en/papers/subramani-2022-extracting-latent-steering-vectors-pretrained/). In Findings of the Association for Computational Linguistics: ACL 2022, pages 566–581, Dublin, Ireland. Association for Computational Linguistics.
86. Sho Takase, Ryokan Ri, Shun Kiyono, and Takuya Kato. 2024. Large vocabulary size improves large language models. arXiv preprint arXiv:2406.16508. [find](https://scholar.google.com/scholar?q=Sho+Takase%2C+Ryokan+Ri%2C+Shun+Kiyono%2C+and+Takuya+Kato%2E+2024%2E+Large+vocabulary+size+improves+large+language+models%2E+arXiv+preprint+arXiv%3A2406%2E16508%2E)
87. Chaofan Tao, Qian Liu, Longxu Dou, Niklas Muennighoff, Zhongwei Wan, Ping Luo, Min Lin, and Ngai Wong. 2024. [Scaling laws with vocabulary: Larger models deserve larger vocabularies](https://md.research.elamur.ai/free/en/papers/tao-2024-scaling-laws-vocabulary-larger-models/). In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
88. Falcon-LLM Team. 2024. The falcon 3 family of open models. [find](https://scholar.google.com/scholar?q=Falcon%2DLLM+Team%2E+2024%2E+The+falcon+3+family+of+open+models%2E)
89. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiroaju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, and 1 others. 2024a. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295. [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Thomas+Mesnard%2C+Cassidy+Hardin%2C+Robert+Dadashi%2C+Surya+Bhupatiroaju%2C+Shreya+Pathak%2C+Laurent+Sifre%2C+Morgane+Rivi%C3%A8re%2C+Mihir+Sanjay+Kale%2C+Juliette+Love%2C+and+1+others%2E+2024a%2E+Gemma%3A+Open+models)
90. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, and 1 others. 2024b. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Morgane+Riviere%2C+Shreya+Pathak%2C+Pier+Giuseppe+Sessa%2C+Cassidy+Hardin%2C+Surya+Bhupatiraju%2C+L%C3%A9onard+Hussenot%2C+Thomas+Mesnard%2C+Bobak+Shahriari%2C+Alexandre+Ram%C3%A9%2C+and+1+others%2E+2024b%2E+Gemma+2%3A+Imp)
91. Eric Todd, Millicent Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. 2024. Function vectors in large language models. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Eric+Todd%2C+Millicent+Li%2C+Arnab+Sen+Sharma%2C+Aaron+Mueller%2C+Byron+C+Wallace%2C+and+David+Bau%2E+2024%2E+Function+vectors+in+large+language+models%2E+In+The+Twelfth+International+Conference+on+Learning+Represent)
92. Cagri Toraman, Eyup Halit Yilmaz, Furkan Şahinuç, and Oguzhan Ozcelik. 2022. Impact of tokenization on language models: An analysis for turkish. ACM Transactions on Asian and Low-Resource Language Information Processing, 22:1 – 21. [find](https://scholar.google.com/scholar?q=Cagri+Toraman%2C+Eyup+Halit+Yilmaz%2C+Furkan+S%CC%A7ahinu%C3%A7%2C+and+Oguzhan+Ozcelik%2E+2022%2E+Impact+of+tokenization+on+language+models%3A+An+analysis+for+turkish%2E+ACM+Transactions+on+Asian+and+Low%2DResource+Language+In)
93. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288.
94. Reut Tsarfaty, Shoval Sadde, Stav Klein, and Amit Seker. 2019. [What’s wrong with Hebrew NLP?](https://md.research.elamur.ai/free/en/papers/tsarfaty-2019-what-s-wrong-hebrew-nlp/) and how to make it right. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): S…
95. Ekaterina Vylomova, Laura Rimell, Trevor Cohn, and Timothy Baldwin. 2015. Take and took, gaggle and goose, book and read: Evaluating the utility of vector differences for lexical relation learning. ArXiv, abs/1509.01692. [find](https://scholar.google.com/scholar?q=Ekaterina+Vylomova%2C+Laura+Rimell%2C+Trevor+Cohn%2C+and+Timothy+Baldwin%2E+2015%2E+Take+and+took%2C+gaggle+and+goose%2C+book+and+read%3A+Evaluating+the+utility+of+vector+differences+for+lexical+relation+learning%2E+Ar)
96. Erik Wijmans, Brody Huval, Alexander Hertzberg, Vladlen Koltun, and Philipp Kraehenbuehl. 2025. Cut your losses in large-vocabulary language models. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Erik+Wijmans%2C+Brody+Huval%2C+Alexander+Hertzberg%2C+Vladlen+Koltun%2C+and+Philipp+Kraehenbuehl%2E+2025%2E+Cut+your+losses+in+large%2Dvocabulary+language+models%2E+In+The+Thirteenth+International+Conference+on+Learn)
97. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+and+1+others%2E+2024%2E+Qwen2%2E+5+technical+report%2E+arXiv+preprint+arXiv%3A2412%2E15115)
98. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+HellaSwag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Com)

*Updated 31 October 2026*