---
title: "Egalitarian Language Representation in Language Models: It All"
description: "As a fix, they propose Grapheme Pair Encoding (GPE), a BPE variant whose atomic units are… 38 references, cited by 11 papers here."
authors: ["Menan Velayuthan", "Kengatharaiyer Sarveswaran"]
year: 2025
venue: "Proceedings of the 31st International Conference on Computational Linguistics"
source: "https://aclanthology.org/2025.coling-main.400/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/velayuthan-2025-egalitarian-language-representation-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/velayuthan-2025-egalitarian-language-representation-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/7/) / Egalitarian Language Representation in Language Models: It All Begins with Tokenizers

# Egalitarian Language Representation in Language Models: It All Begins with Tokenizers

**Authors:** [Menan Velayuthan](https://md.research.elamur.ai/free/en/authors/menan-velayuthan/) and [Kengatharaiyer Sarveswaran](https://md.research.elamur.ai/free/en/authors/kengatharaiyer-sarveswaran/) · **Year:** 2025 · **Venue:** [Proceedings of the 31st International Conference on Computational Linguistics](https://md.research.elamur.ai/free/en/venues/coling/) ([COLING 2025](https://md.research.elamur.ai/free/en/venues/coling/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.coling-main.400/) · [PDF](https://aclanthology.org/2025.coling-main.400.pdf) · [OpenAlex](https://openalex.org/W4403709782)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [11](https://md.research.elamur.ai/free/en/papers/velayuthan-2025-egalitarian-language-representation-language-models/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Tokenizers from English-centric LLMs handle complex abugida scripts such as Tamil, Sinhala and Hindi poorly. In these scripts, one visible character often spans several Unicode codepoints, and tokenizers split those codepoints apart. The result is more tokens, higher usage cost, more latency and heavier context-window demands. The authors argue that the main cause is the pre-tokenization stage, not the subword algorithm. The chunks a pre-tokenizer produces cap how long any learned token can be, so its output gives an upper bound on achievable compression. They audit pre-trained tokenizers from English-centric and multilingual models, measuring Compression Ratio and Tokenization Parity against English. As a fix, they propose Grapheme Pair Encoding (GPE), a BPE variant whose atomic units are grapheme clusters rather than bytes or codepoints. For these scripts, grapheme-based character extraction is reported to beat byte-level approaches such as ByT5 and CANINE. The work matters for fairer and cheaper adaptation of LLMs to under-served scripts. Code and resources are public on GitHub.

## Key points
- **Tokenizers audited.** English-centric: GPT-2, GPT-4, Llama 3, FLAN-T5, Gemma 2. Multilingual: Aya, mBERT, mT5, mBART, NLLBm. Evaluation covers Tamil, Sinhala and Hindi, with English as reference.
- **Pre-tokenizer splits.** The regex-based pre-tokenizers of GPT-2, GPT-4 and Llama 3 break a simple greeting in all three languages into fragments, separating vowel modifiers and the Tamil pulli from their consonants. The pre-tokenizers of BERT-style and T5/mBART-style models keep these words whole.
- **Metrics.** Two metrics are used:
  - Compression Ratio, a bytes-per-token style measure, which becomes a Maximum Compression Ratio (CR max) when bounded by the pre-tokens.
  - Tokenization Parity from Petrov et al. (2024).
- **Why graphemes matter.** A single Tamil Grantha grapheme uses 4 codepoints. A Sinhala character can use 5 codepoints, including a Zero-Width Joiner (x200D).
- **GPE implementation.** GPE is built on Karpathy's BPE implementation, with the `grapheme` Python library for clustering and the HuggingFace `tokenizers` library for the baselines. The extracted text omits the numeric results tables, so no scores are given here.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Jaffna](https://md.research.elamur.ai/free/en/universities/university-of-jaffna/) — University, Jaffna, Sri Lanka · 7 works · 70 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [✂️ Tokenization & segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/) · 119 works

## Cite

Cite as: Menan Velayuthan and Kengatharaiyer Sarveswaran. 2025. Egalitarian Language Representation in Language Models: It All Begins with Tokenizers. Proceedings of the 31st International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{velayuthan2025egalitarian,
  title = {{Egalitarian Language Representation in Language Models: It All Begins with Tokenizers}},
  author = {Menan Velayuthan and Kengatharaiyer Sarveswaran},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-main.400/}
}
```

- [← previous Do Large Language Models Mirror Cognitive Language Processing?](https://md.research.elamur.ai/free/en/papers/ren-2025-do-large-language-models-mirror/)
- [next → Explainability and Interpretability of Multilingual Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/resck-2025-explainability-interpretability-multilingual-large-language/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/7/)

## References

38 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, ..., and Jason Wei.… [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Yunxuan+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+Albert+Webson%2C+Shixiang+Shane+Gu%2C+Zhuyun+Dai%2C+Mirac+Suzgun%2C+X)
2. Jonathan H. Clark, Dan Garrette, Iulia Turc, and John Wieting. 2022. [Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation](https://md.research.elamur.ai/free/en/papers/clark-2022-canine-pre-training-efficient-tokenization/). Transactions of the Association for Computational Linguistics, 10:73–91.
3. Raj Dabre, Chenhui Chu, and Anoop Kunchukuttan. 2020. A Survey of Multilingual Neural Machine Translation. ACM Comput. Surv., 53(5). [find](https://scholar.google.com/scholar?q=Raj+Dabre%2C+Chenhui+Chu%2C+and+Anoop+Kunchukuttan%2E+2020%2E+A+Survey+of+Multilingual+Neural+Machine+Translation%2E+ACM+Comput%2E+Surv%2E%2C+53%285%29%2E)
4. Gautier Dagan, Gabriel Synnaeve, and Baptiste Rozière. 2024. [Getting the most out of your tokenizer for pre-training and domain adaptation](https://md.research.elamur.ai/free/en/papers/dagan-2024-getting-most-out-tokenizer-pre/). Preprint, arXiv:2402.01035.
5. Peter T Daniels, Mark Aronoff, and Janie Rees-Miller. 2003. Writing Systems, page 43–80. Blackwell. [find](https://scholar.google.com/scholar?q=Peter+T+Daniels%2C+Mark+Aronoff%2C+and+Janie+Rees%2DMiller%2E+2003%2E+Writing+Systems%2C+page+43%E2%80%9380%2E+Blackwell%2E)
6. Björn Deiseroth, Manuel Brack, Patrick Schramowski, Kristian Kersting, and Samuel Weinbach. 2024. T-FREE: Tokenizer-Free Generative LLMs via Sparse Representations for Memory-Efficient Embeddings. Preprint, arXiv:2406.19223. — [arXiv:2406.19223](https://arxiv.org/abs/2406.19223) [find](https://scholar.google.com/scholar?q=Bj%C3%B6rn+Deiseroth%2C+Manuel+Brack%2C+Patrick+Schramowski%2C+Kristian+Kersting%2C+and+Samuel+Weinbach%2E+2024%2E+T%2DFREE%3A+Tokenizer%2DFree+Generative+LLMs+via+Sparse+Representations+for+Memory%2DEfficient+Embeddings%2E+Pre)
7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). Preprint, arXiv:1810.04805.
8. Jonathan Dortheimer, Nik Martelaro, Aaron Sprecher, and Gerhard Schubert. 2024. Evaluating large-language-model chatbots to engage communities in large-scale design projects. Artificial Intelligence for Engineering Design, Analysis and Manufacturing, 38:e4. [find](https://scholar.google.com/scholar?q=Jonathan+Dortheimer%2C+Nik+Martelaro%2C+Aaron+Sprecher%2C+and+Gerhard+Schubert%2E+2024%2E+Evaluating+large%2Dlanguage%2Dmodel+chatbots+to+engage+communities+in+large%2Dscale+design+projects%2E+Artificial+Intelligence+f)
9. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, ..., and Zhiwei Zha… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
10. Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. 2023. GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models. Preprint, arXiv:2303.10130. — [arXiv:2303.10130](https://arxiv.org/abs/2303.10130) [find](https://scholar.google.com/scholar?q=Tyna+Eloundou%2C+Sam+Manning%2C+Pamela+Mishkin%2C+and+Daniel+Rock%2E+2023%2E+GPTs+are+GPTs%3A+An+Early+Look+at+the+Labor+Market+Impact+Potential+of+Large+Language+Models%2E+Preprint%2C+arXiv%3A2303%2E10130%2E+%E2%80%94+%5BarXiv%3A2303)
11. Omer Goldman, Avi Caciularu, Matan Eyal, Kris Cao, Idan Szpektor, and Reut Tsarfaty. 2024. Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance. In Findings of the Association for Computational Linguistics ACL 2024, pages 2274–2286, Bangkok, Thailand and vir… [find](https://scholar.google.com/scholar?q=Omer+Goldman%2C+Avi+Caciularu%2C+Matan+Eyal%2C+Kris+Cao%2C+Idan+Szpektor%2C+and+Reut+Tsarfaty%2E+2024%2E+Unpacking+Tokenization%3A+Evaluating+Text+Compression+and+its+Correlation+with+Model+Performance%2E+In+Findings+o)
12. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. [Measuring Massive Multitask Language Understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). Preprint, arXiv:2009.03300.
13. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
14. Andrej Karpathy. 2024. minbpe. [https://github.com/karpathy/minbpe](https://github.com/karpathy/minbpe). [↗](https://github.com/karpathy/minbpe)
15. Taku Kudo. 2018. [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://md.research.elamur.ai/free/en/papers/kudo-2018-subword-regularization-improving-neural-network/). Preprint, arXiv:1804.10959.
16. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, ..., an… [find](https://scholar.google.com/scholar?q=Raymond+Li%2C+Loubna+Ben+Allal%2C+Yangtian+Zi%2C+Niklas+Muennighoff%2C+Denis+Kocetkov%2C+Chenghao+Mou%2C+Marc+Marone%2C+Christopher+Akiki%2C+Jia+Li%2C+Jenny+Chim%2C+Qian+Liu%2C+Evgenii+Zheltonozhskii%2C+Terry+Yue+Zhuo%2C+Thoma)
17. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual Denoising Pre-training for Neural Machine Translation. Preprint, arXiv:2001.08210. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Jiatao+Gu%2C+Naman+Goyal%2C+Xian+Li%2C+Sergey+Edunov%2C+Marjan+Ghazvininejad%2C+Mike+Lewis%2C+and+Luke+Zettlemoyer%2E+2020%2E+Multilingual+Denoising+Pre%2Dtraining+for+Neural+Machine+Translation%2E+Preprint%2C+)
18. Lalita Lowphansirikul, Charin Polpanumas, Nawat Jantrakulchai, and Sarana Nutanong. 2021. WangchanBERTa: Pretraining transformer-based Thai Language Models. Preprint, arXiv:2101.09635. — [arXiv:2101.09635](https://arxiv.org/abs/2101.09635) [find](https://scholar.google.com/scholar?q=Lalita+Lowphansirikul%2C+Charin+Polpanumas%2C+Nawat+Jantrakulchai%2C+and+Sarana+Nutanong%2E+2021%2E+WangchanBERTa%3A+Pretraining+transformer%2Dbased+Thai+Language+Models%2E+Preprint%2C+arXiv%3A2101%2E09635%2E+%E2%80%94+%5BarXiv%3A2101%2E0)
19. Jesse G. Meyer, Ryan J. Urbanowicz, Patrick C. N. Martin, Karen O’Connor, Ruowang Li, Pei-Chen Peng, Tiffani J. Bright, Nicholas Tatonetti, Kyoung Jae Won, Graciela Gonzalez-Hernandez, and Jason H. Moore. 2023. ChatGPT and large language models in academia: opportunities and challenges. BioData Mini… [find](https://scholar.google.com/scholar?q=Jesse+G%2E+Meyer%2C+Ryan+J%2E+Urbanowicz%2C+Patrick+C%2E+N%2E+Martin%2C+Karen+O%E2%80%99Connor%2C+Ruowang+Li%2C+Pei%2DChen+Peng%2C+Tiffani+J%2E+Bright%2C+Nicholas+Tatonetti%2C+Kyoung+Jae+Won%2C+Graciela+Gonzalez%2DHernandez%2C+and+Jason+H%2E+Mo)
20. OpenAI. 2022. ChatGPT: Optimizing Language Models for Dialogue. Accessed: 2024-08-15. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2022%2E+ChatGPT%3A+Optimizing+Language+Models+for+Dialogue%2E+Accessed%3A+2024%2D08%2D15%2E)
21. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, ..., and Barret… — [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
22. Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi. 2024. [Language model tokenizers introduce unfairness between languages](https://md.research.elamur.ai/free/en/papers/petrov-2023-language-model-tokenizers-introduce-unfairness/). Advances in Neural Information Processing Systems, 36.
23. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
24. Gowtham Ramesh, Sumanth Doddapaneni, Aravinth Bheemaraj, Mayank Jobanputra, Raghavan AK, Ajitesh Sharma, Sujit Sahoo, Harshita Diddee, Mahalakshmi J, Divyanshu Kakwani, Navneet Kumar, Aswin Pradeep, Srihari Nagaraj, Kumar Deepak, Vivek Raghavan, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh Shantad… — [Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages](https://md.research.elamur.ai/free/en/papers/ramesh-2022-samanantar-largest-publicly-available-parallel/)
25. Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, .… — [Code Llama: Open Foundation Models for Code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/)
26. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://md.research.elamur.ai/free/en/papers/rust-2021-how-good-is-your-tokenizer/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Join…
27. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. [Neural Machine Translation of Rare Words with Subword Units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computational Li…
28. AI Singapore. 2024. [SEA-LION (Southeast Asian Languages In One Network): A Family of Large Language Models for Southeast Asia](https://md.research.elamur.ai/free/en/papers/ng-2025-sea-lion-southeast-asian-languages/). [https://github.com/aisingapore/sealion](https://github.com/aisingapore/sealion). [↗](https://github.com/aisingapore/sealion)
29. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, ..., an… [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Morgane+Riviere%2C+Shreya+Pathak%2C+Pier+Giuseppe+Sessa%2C+Cassidy+Hardin%2C+Surya+Bhupatiraju%2C+L%C3%A9onard+Hussenot%2C+Thomas+Mesnard%2C+Bobak+Shahriari%2C+Alexandre+Ram%C3%A9%2C+Johan+Ferret%2C+Peter+Liu%2C+Pouya+Ta)
30. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, ..., and… [find](https://scholar.google.com/scholar?q=NLLB+Team%2C+Marta+R%2E+Costa%2Djuss%C3%A0%2C+James+Cross%2C+Onur+%C3%87elebi%2C+Maha+Elbayad%2C+Kenneth+Heafield%2C+Kevin+Heffernan%2C+Elahe+Kalbassi%2C+Janice+Lam%2C+Daniel+Licht%2C+Jean+Maillard%2C+Anna+Sun%2C+Skyler+Wang%2C+Guillaume+We)
31. Timm Teubner, Christoph M. Flath, Christof Weinhardt, Wil van der Aalst, and Oliver Hinz. 2023. Welcome to the Era of ChatGPT et al. Business & Information Systems Engineering, 65(2):95–101. [find](https://scholar.google.com/scholar?q=Timm+Teubner%2C+Christoph+M%2E+Flath%2C+Christof+Weinhardt%2C+Wil+van+der+Aalst%2C+and+Oliver+Hinz%2E+2023%2E+Welcome+to+the+Era+of+ChatGPT+et+al%2E+Business+%26+Information+Systems+Engineering%2C+65%282%29%3A95%E2%80%93101%2E)
32. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… — [Transformers: State-of-the-Art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
33. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, ..., and Jeffrey Dean. 2016. Go… [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V%2E+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+Jeff+Klingner%2C+Apurva+Shah%2C+Melvin+Johnson%2C+Xiaobing+Liu%2C+%C5%81uk)
34. Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. ByT5: Towards a token-free future with pre-trained byte-to-byte models. Preprint, arXiv:2105.13626. [find](https://scholar.google.com/scholar?q=Linting+Xue%2C+Aditya+Barua%2C+Noah+Constant%2C+Rami+Al%2DRfou%2C+Sharan+Narang%2C+Mihir+Kale%2C+Adam+Roberts%2C+and+Colin+Raffel%2E+2022%2E+ByT5%3A+Towards+a+token%2Dfree+future+with+pre%2Dtrained+byte%2Dto%2Dbyte+models%2E+Preprin)
35. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. [mT5: A massively multilingual pre-trained text-to-text transformer](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/). Preprint, arXiv:2010.11934.
36. Lili Yu, Dániel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. 2023. MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers. Preprint, arXiv:2305.07185. [find](https://scholar.google.com/scholar?q=Lili+Yu%2C+D%C3%A1niel+Simig%2C+Colin+Flaherty%2C+Armen+Aghajanyan%2C+Luke+Zettlemoyer%2C+and+Mike+Lewis%2E+2023%2E+MEGABYTE%3A+Predicting+Million%2Dbyte+Sequences+with+Multiscale+Transformers%2E+Preprint%2C+arXiv%3A2305%2E07185%2E)
37. Jun Zhao, Zhihao Zhang, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024. Llama beyond english: An empirical study on language capability transfer. Preprint, arXiv:2401.01055. [find](https://scholar.google.com/scholar?q=Jun+Zhao%2C+Zhihao+Zhang%2C+Qi+Zhang%2C+Tao+Gui%2C+and+Xuanjing+Huang%2E+2024%2E+Llama+beyond+english%3A+An+empirical+study+on+language+capability+transfer%2E+Preprint%2C+arXiv%3A2401%2E01055%2E)
38. Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. [Aya Model: An Instruction Fine…](https://md.research.elamur.ai/free/en/papers/ustun-2024-aya-model-instruction-finetuned-open/)

*Updated 31 October 2026*