---
title: "Tokenizer Choice For LLM Training: Negligible or Crucial? — Ali 2024"
description: "Tokenizers are rarely examined when building LLMs. 73 references, cited by 72 papers here."
authors: ["Mehdi Ali", "Michael Fromm", "Klaudia Thellmann", "Richard Rutmann", "Max Lübbering", "Johannes Leveling", "Katrin Klug", "Jan Ebert", "Niclas Doll", "Jasper Buschhoff", "Charvi Jain", "Alexander Weber", "Lena Jurkschat", "Hammam Abdelwahab", "Chelsea John", "Pedro Ortiz Suarez", "Malte Ostendorff", "Samuel Weinbach", "Rafet Sifa", "Stefan Kesselheim", "Nicolas Flores-Herr"]
year: 2024
venue: "Findings of the Association for Computational Linguistics: NAACL 2024"
source: "https://aclanthology.org/2024.findings-naacl.247/"
doi: "10.18653/v1/2024.findings-naacl.247"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ali-2024-tokenizer-choice-llm-training-negligible/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ali-2024-tokenizer-choice-llm-training-negligible/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / Tokenizer Choice For LLM Training: Negligible or Crucial?

# Tokenizer Choice For LLM Training: Negligible or Crucial?

**Authors:** Mehdi Ali, Michael Fromm, [Klaudia Thellmann](https://md.research.elamur.ai/free/en/authors/klaudia-thellmann/), Richard Rutmann, Max Lübbering, [Johannes Leveling](https://md.research.elamur.ai/free/en/authors/johannes-leveling/), Katrin Klug, [Jan Ebert](https://md.research.elamur.ai/free/en/authors/jan-ebert/), Niclas Doll, Jasper Buschhoff, Charvi Jain, Alexander Weber, Lena Jurkschat, Hammam Abdelwahab, Chelsea John, [Pedro Ortiz Suarez](https://md.research.elamur.ai/free/en/authors/pedro-ortiz-suarez/), Malte Ostendorff, Samuel Weinbach, [Rafet Sifa](https://md.research.elamur.ai/free/en/authors/rafet-sifa/), Stefan Kesselheim and Nicolas Flores-Herr · **Year:** 2024 · **Venue:** Findings of the Association for Computational Linguistics: NAACL 2024 · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.findings-naacl.247/) · [PDF](https://aclanthology.org/2024.findings-naacl.247.pdf) · [DOI 10.18653/v1/2024.findings-naacl.247](https://doi.org/10.18653/v1/2024.findings-naacl.247) · [OpenAlex](https://openalex.org/W4401042479)

**Citations:** 30 citations (OpenAlex, 2026-10-14) · [72](https://md.research.elamur.ai/free/en/papers/ali-2024-tokenizer-choice-llm-training-negligible/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
Tokenizers are rarely examined when building LLMs. This paper asks whether the choice of tokenizer measurably affects decoder-only model quality and training cost, in both English-only and multilingual settings. The authors built two training corpora of 70B words each, with matched domain mixtures (80% web-crawled, 20% curated). One is English only. The other covers English, German, French, Italian and Spanish in balanced proportions. On these they trained 24 tokenizers, varying four factors: algorithm (BPE vs. Unigram), library (Huggingface vs. SentencePiece), language coverage and vocabulary size. For each tokenizer they trained a separate 2.6B-parameter model with all other settings fixed, and added two GPT-2-tokenizer baselines. They then compared intrinsic metrics (fertility, parity) with zero-shot downstream results and training compute. The tokenizer turned out to matter considerably. Fertility and parity only partly predict model quality, so a low fertility score looks necessary but not sufficient. Reusing an English-centric tokenizer for multilingual training lowers accuracy and raises training cost. This gives practical guidance for anyone building LLMs beyond English, including for under-served languages.

## Key points
- The study trained 26 models (24 tokenizer variants plus 2 GPT-2 baselines) on up to 52B tokens each, following Chinchilla-style scaling, at a cost of roughly 59,000 GPU hours.
- Fertility was computed on 10,000 held-out documents, and parity on the FLORES-200 parallel corpus relative to English. Their correlation with downstream scores is weak and depends on the task.
- Applying an English-oriented tokenizer to multilingual data increases training compute by up to 68% and noticeably hurts performance.
- About 33k vocabulary entries suffice for English, while the five-language setup needs roughly three times as many. BPE works well in both cases, and SentencePiece beat the Huggingface library.
- BPE-SP-33 was the strongest monolingual tokenizer, the same configuration later used for much larger models such as LLaMA. Limitations include a single seed, no hyperparameter tuning, Latin-script languages only and zero-shot evaluation only.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · [✂️ Tokenization & segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/)

## Institutions

- [Technische Universität Dresden](https://md.research.elamur.ai/free/en/universities/technische-universitat-dresden/) — University, Dresden, Germany · 6 works · 7,135 citations
- [German Research Centre for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/german-research-centre-for-artificial-intelligence/) — Research institute, Kaiserslautern, Germany · 39 works · 5,199 citations
- [Fraunhofer Institute for Intelligent Analysis and Information Systems](https://md.research.elamur.ai/free/en/labs/fraunhofer-institute-for-intelligent-analysis-and-information-systems/) — Research institute, Sankt Augustin, Germany · 3 works · 270 citations

Also on the byline: [Lamarr Institute for Machine Learning and Artificial Intelligence](https://ror.org/04s11ea33).

## Cite

Cite as: Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max Lübbering, Johannes Leveling, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Buschhoff, Charvi Jain, Alexander Weber, Lena Jurkschat, Hammam Abdelwahab, Chelsea John, Pedro Ortiz Suarez, Malte Ostendorff, Samuel Weinbach, Rafet Sifa, Stefan Kesselheim and Nicolas Flores-Herr. 2024. Tokenizer Choice For LLM Training: Negligible or Crucial?. Findings of the Association for Computational Linguistics: NAACL 2024.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ali2024tokenizer,
  title = {{Tokenizer Choice For LLM Training: Negligible or Crucial?}},
  author = {Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max Lübbering, Johannes Leveling, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Buschhoff, Charvi Jain, Alexander Weber, Lena Jurkschat, Hammam Abdelwahab, Chelsea John, Pedro Ortiz Suarez, Malte Ostendorff, Samuel Weinbach, Rafet Sifa, Stefan Kesselheim and Nicolas Flores-Herr},
  booktitle = {Findings of the Association for Computational Linguistics: NAACL 2024},
  year = {2024},
  doi = {10.18653/v1/2024.findings-naacl.247},
  url = {https://doi.org/10.18653/v1/2024.findings-naacl.247}
}
```

- [← previous Nemotron-4 340B Technical Report](https://md.research.elamur.ai/free/en/papers/adler-2024-nemotron-4-340b-technical-report/)
- [next → LongAlign: A Recipe for Long Context Alignment of Large Language Models](https://md.research.elamur.ai/free/en/papers/bai-2024-longalign-recipe-long-context/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

73 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Julien Abadji, Pedro Javier Ortiz Suárez, Laurent Romary, and Benoît Sagot. 2021. Ungoliant: An optimized pipeline for the generation of a very large-scale multilingual web corpus. In Harald Lüngen, Marc Kupietz, Piotr Bański, Adrien Barbaresi, Simon Clematide, and Ines Pisetta, editors, Proceeding… [find](https://scholar.google.com/scholar?q=Julien+Abadji%2C+Pedro+Javier+Ortiz+Su%C3%A1rez%2C+Laurent+Romary%2C+and+Beno%C3%AEt+Sagot%2E+2021%2E+Ungoliant%3A+An+optimized+pipeline+for+the+generation+of+a+very+large%2Dscale+multilingual+web+corpus%2E+In+Harald+L%C3%BCngen%2C+M)
2. Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R. Mortensen, Noah A. Smith, and Yulia Tsvetkov. 2023. Do all languages cost the same? tokenization in the era of commercial language models. In EMNLP, pages 9904–9923. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/ahia-2023-do-all-languages-cost-same/)
3. Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. 2000. A neural probabilistic language model. Advances in neural information processing systems, 13. [find](https://scholar.google.com/scholar?q=Yoshua+Bengio%2C+R%C3%A9jean+Ducharme%2C+and+Pascal+Vincent%2E+2000%2E+A+neural+probabilistic+language+model%2E+Advances+in+neural+information+processing+systems%2C+13%2E)
4. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439. [our card](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/)
5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020a. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. [our card](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
6. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… [our card](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
7. Nadezhda Chirkova and Sergey Troshin. 2022. CodeBPE: Investigating subtokenization options for large language model pretraining on source code. In Deep Learning for Code Workshop. [find](https://scholar.google.com/scholar?q=Nadezhda+Chirkova+and+Sergey+Troshin%2E+2022%2E+CodeBPE%3A+Investigating+subtokenization+options+for+large+language+model+pretraining+on+source+code%2E+In+Deep+Learning+for+Code+Workshop%2E)
8. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL-HLT (1), pages 2924–2936. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/)
9. Jonathan H. Clark, Dan Garrette, Iulia Turc, and John Wieting. 2022. Canine: Pre-training an efficient tokenization-free encoder for language representation. Transactions of the Association for Computational Linguistics, 10:73–91. [our card](https://md.research.elamur.ai/free/en/papers/clark-2022-canine-pre-training-efficient-tokenization/)
10. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457. [find](https://scholar.google.com/scholar?q=Peter+Clark%2C+Isaac+Cowhey%2C+Oren+Etzioni%2C+Tushar+Khot%2C+Ashish+Sabharwal%2C+Carissa+Schoenick%2C+and+Oyvind+Tafjord%2E+2018%2E+Think+you+have+solved+question+answering%3F+try+arc%2C+the+AI2+reasoning+challenge%2E+CoR)
11. Together Computer. 2023. Redpajama: An open source recipe to reproduce llama training dataset. [find](https://scholar.google.com/scholar?q=Together+Computer%2E+2023%2E+Redpajama%3A+An+open+source+recipe+to+reproduce+llama+training+dataset%2E)
12. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: evaluating cross-lingual sentence representations. In EMNLP, pages 2475–2485. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Ruty+Rinott%2C+Guillaume+Lample%2C+Adina+Williams%2C+Samuel+R%2E+Bowman%2C+Holger+Schwenk%2C+and+Veselin+Stoyanov%2E+2018%2E+XNLI%3A+evaluating+cross%2Dlingual+sentence+representations%2E+In+EMNLP%2C+pages+24)
13. Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. 2019. The commitmentbank: Investigating projection in naturally occurring discourse. [our card](https://md.research.elamur.ai/free/en/papers/marneffe-2019-commitmentbank-investigating-projection-naturally/)
14. In proceedings of Sinn und Bedeutung, volume 23, pages 107–124. [find](https://scholar.google.com/scholar?q=In+proceedings+of+Sinn+und+Bedeutung%2C+volume+23%2C+pages+107%E2%80%93124%2E)
15. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
16. William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005). [our card](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/)
17. Ariel Ekgren, Amaru Cuba Gyllensten, Felix Stollenwerk, Joey Öhman, Tim Isbister, Evangelia Gogoulou, Fredrik Carlsson, Alice Heiman, Judit Casademont, and Magnus Sahlgren. 2023. Gpt-sw3: An autoregressive language model for the nordic languages. [find](https://scholar.google.com/scholar?q=Ariel+Ekgren%2C+Amaru+Cuba+Gyllensten%2C+Felix+Stollenwerk%2C+Joey+%C3%96hman%2C+Tim+Isbister%2C+Evangelia+Gogoulou%2C+Fredrik+Carlsson%2C+Alice+Heiman%2C+Judit+Casademont%2C+and+Magnus+Sahlgren%2E+2023%2E+Gpt%2Dsw3%3A+An+autoregre)
18. Philip Gage. 1994. A new algorithm for data compression. The C Users Journal archive, 12:23–38. [find](https://scholar.google.com/scholar?q=Philip+Gage%2E+1994%2E+A+new+algorithm+for+data+compression%2E+The+C+Users+Journal+archive%2C+12%3A23%E2%80%9338%2E)
19. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020a. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+Shawn+Presser%2C+and+Connor+Leahy%2E+2020a%2E+The+Pile%3A+An+800gb+data)
20. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Ani… [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Baber+Abbasi%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Alain+Le+Noac%E2%80%99h%2C+Haonan+Li%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Chris+Oci)
21. Yingqiang Gao, Nikola I. Nikolov, Yuhuang Hu, and Richard H.R. Hahnloser. 2020b. Character-level translation with self-attention. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1591–1604, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yingqiang+Gao%2C+Nikola+I%2E+Nikolov%2C+Yuhuang+Hu%2C+and+Richard+H%2ER%2E+Hahnloser%2E+2020b%2E+Character%2Dlevel+translation+with+self%2Dattention%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Compu)
22. Joshua Goodman. 2001. A bit of progress in language modeling. CoRR, cs.CL/0108005v1. [find](https://scholar.google.com/scholar?q=Joshua+Goodman%2E+2001%2E+A+bit+of+progress+in+language+modeling%2E+CoRR%2C+cs%2ECL%2F0108005v1%2E)
23. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation. Transactions of the Association for Computat… [find](https://scholar.google.com/scholar?q=Naman+Goyal%2C+Cynthia+Gao%2C+Vishrav+Chaudhary%2C+Peng%2DJen+Chen%2C+Guillaume+Wenzek%2C+Da+Ju%2C+Sanjana+Krishnan%2C+Marc%E2%80%99Aurelio+Ranzato%2C+Francisco+Guzm%C3%A1n%2C+and+Angela+Fan%2E+2022%2E+The+Flores%2D101+Evaluation+Benchmark)
24. Johannes Graën, Tannon Kew, Anastassia Shaitarova, and Martin Volk. 2019. Modelling large parallel corpora: The zurich parallel corpus collection. In Proceedings of the 7th Workshop on Challenges in the Management of Large Corpora (CMLC), pages 1–8. Leibniz-Institut für Deutsche Sprache. [find](https://scholar.google.com/scholar?q=Johannes+Gra%C3%ABn%2C+Tannon+Kew%2C+Anastassia+Shaitarova%2C+and+Martin+Volk%2E+2019%2E+Modelling+large+parallel+corpora%3A+The+zurich+parallel+corpus+collection%2E+In+Proceedings+of+the+7th+Workshop+on+Challenges+in+t)
25. J. Graën, D. Batinic, and M. Volk. 2014. Cleaning the Europarl corpus for linguistic applications. In Konvens 2014. Stiftung Universität Hildesheim. [find](https://scholar.google.com/scholar?q=J%2E+Gra%C3%ABn%2C+D%2E+Batinic%2C+and+M%2E+Volk%2E+2014%2E+Cleaning+the+Europarl+corpus+for+linguistic+applications%2E+In+Konvens+2014%2E+Stiftung+Universit%C3%A4t+Hildesheim%2E)
26. Najeh Hajlaoui, David Kolovratnik, Jaakko Vaeyrynen, Ralf Steinberger, and Dániel Varga. 2014. DCEP -Digital corpus of the European parliament. In Proc. [find](https://scholar.google.com/scholar?q=Najeh+Hajlaoui%2C+David+Kolovratnik%2C+Jaakko+Vaeyrynen%2C+Ralf+Steinberger%2C+and+D%C3%A1niel+Varga%2E+2014%2E+DCEP+%2DDigital+corpus+of+the+European+parliament%2E+In+Proc%2E)
27. LREC 2014 (Language Resources and Evaluation Conference). Reykjavik, Iceland, pages 3164–3171. [find](https://scholar.google.com/scholar?q=LREC+2014+%28Language+Resources+and+Evaluation+Conference%29%2E+Reykjavik%2C+Iceland%2C+pages+3164%E2%80%933171%2E)
28. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karén Simo… [find](https://scholar.google.com/scholar?q=Jordan+Hoffmann%2C+Sebastian+Borgeaud%2C+Arthur+Mensch%2C+Elena+Buchatskaya%2C+Trevor+Cai%2C+Eliza+Rutherford%2C+Diego+de+Las+Casas%2C+Lisa+Anne+Hendricks%2C+Johannes+Welbl%2C+Aidan+Clark%2C+Thomas+Hennigan%2C+Eric+Noland%2C)
29. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… [our card](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
30. Stefan Höfler and Michael Piotrowski. 2011. Building corpora for the philological study of Swiss legal texts. Journal for Language Technology and Computational Linguistics, 26(2):77–89. [find](https://scholar.google.com/scholar?q=Stefan+H%C3%B6fler+and+Michael+Piotrowski%2E+2011%2E+Building+corpora+for+the+philological+study+of+Swiss+legal+texts%2E+Journal+for+Language+Technology+and+Computational+Linguistics%2C+26%282%29%3A77%E2%80%9389%2E)
31. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. PubMedQA: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proce… [our card](https://md.research.elamur.ai/free/en/papers/jin-2019-pubmedqa-dataset-biomedical-research-question/)
32. P. Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Machine Translation Summit, volume 5, pages 79––86. Asia-Pacific Association for Machine Translation (AAMT). [find](https://scholar.google.com/scholar?q=P%2E+Koehn%2E+2005%2E+Europarl%3A+A+parallel+corpus+for+statistical+machine+translation%2E+In+Machine+Translation+Summit%2C+volume+5%2C+pages+79%E2%80%93%E2%80%9386%2E+Asia%2DPacific+Association+for+Machine+Translation+%28AAMT%29%2E)
33. Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75, Melbourne, Australia. Association for Computational… [our card](https://md.research.elamur.ai/free/en/papers/kudo-2018-subword-regularization-improving-neural-network/)
34. Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. [our card](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/)
35. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785– 794, Copenhagen, Denmark. Association for Computational Lin… [our card](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/)
36. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning. [find](https://scholar.google.com/scholar?q=Hector+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+2012%2E+The+winograd+schema+challenge%2E+In+Thirteenth+international+conference+on+the+principles+of+knowledge+representation+and+reasoning%2E)
37. Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, et al. 2022. Few-shot learning with multilingual generative language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Proc… [our card](https://md.research.elamur.ai/free/en/papers/lin-2021-few-shot-learning-multilingual-language/)
38. Pierre Lison and Jörg Tiedemann. 2016. OpenSubtitles2016: Extracting large parallel corpora from movie and tv subtitles. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC-2016). [our card](https://md.research.elamur.ai/free/en/papers/lison-2016-opensubtitles2016-extracting-large-parallel-corpora/)
39. Sabrina J. Mielke, Zaid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja, Chenglei Si, Wilson Y. Lee, Benoît Sagot, and Samson Tan. 2021. Between words and characters: A brief history of open-vocabulary modeling and tokenization in NLP. CoRR, abs/2112.10508. [our card](https://md.research.elamur.ai/free/en/papers/mielke-2021-between-words-characters-brief-history/)
40. Anthony Moi and Nicolas Patry. 2023. HuggingFace’s Tokenizers. [find](https://scholar.google.com/scholar?q=Anthony+Moi+and+Nicolas+Patry%2E+2023%2E+HuggingFace%E2%80%99s+Tokenizers%2E)
41. Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. 2021. Efficient large-scale language model training on gpu clusters using megatron-lm.… [find](https://scholar.google.com/scholar?q=Deepak+Narayanan%2C+Mohammad+Shoeybi%2C+Jared+Casper%2C+Patrick+LeGresley%2C+Mostofa+Patwary%2C+Vijay+Korthikanti%2C+Dmitri+Vainbrand%2C+Prethvi+Kashinkunti%2C+Julie+Bernauer%2C+Bryan+Catanzaro%2C+Amar+Phanishayee%2C+and+M)
42. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. The LAMBADA dataset: Word prediction requiring a broad discourse context. In ACL (1). The Association for Computer Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/paperno-2016-lambada-dataset-word-prediction-requiring/)
43. Aleksandar Petrov, Emanuele La Malfa, Philip HS Torr, and Adel Bibi. 2023. Language model tokenizers introduce unfairness between languages. arXiv preprint arXiv:2305.15425. [our card](https://md.research.elamur.ai/free/en/papers/petrov-2023-language-model-tokenizers-introduce-unfairness/)
44. Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. WiC: the word-in-context dataset for evaluating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum… [our card](https://md.research.elamur.ai/free/en/papers/pilehvar-2019-wic-word-context-dataset-evaluating/)
45. Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2362–2376, Online. Association… [our card](https://md.research.elamur.ai/free/en/papers/ponti-2020-xcopa-multilingual-dataset-causal-commonsense/)
46. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language understanding by generative pre-training. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+Ilya+Sutskever%2C+et+al%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E)
47. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series. [find](https://scholar.google.com/scholar?q=Melissa+Roemmele%2C+Cosmin+Adrian+Bejan%2C+and+Andrew+S+Gordon%2E+2011%2E+Choice+of+plausible+alternatives%3A+An+evaluation+of+commonsense+causal+reasoning%2E+In+2011+AAAI+Spring+Symposium+Series%2E)
48. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Join… [our card](https://md.research.elamur.ai/free/en/papers/rust-2021-how-good-is-your-tokenizer/)
49. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. In AAAI, pages 8732–8740. AAAI Press. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2020%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+In+AAAI%2C+pages+8732%E2%80%938740%2E+AAAI+Press%2E)
50. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighof… [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%2C+Daniel+Hesslow%2C+Roman+Castagn%C3%A9%2C+Alexandra+Sasha+Luccioni%2C+Fran%C3%A7ois+Yvon%2C+Matthias+Gall%C3%A9%2C+Jonathan+Tow%2C+Alexander+M%2E+Rush%2C+Stel)
51. Dietmar Schabus, Marcin Skowron, and Martin Trapp. 2017. One million posts: A data set of german online discussions. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pages 1241–1244, Tokyo, Japan. [find](https://scholar.google.com/scholar?q=Dietmar+Schabus%2C+Marcin+Skowron%2C+and+Martin+Trapp%2E+2017%2E+One+million+posts%3A+A+data+set+of+german+online+discussions%2E+In+Proceedings+of+the+40th+International+ACM+SIGIR+Conference+on+Research+and+Devel)
52. Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In 2012 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5149–5152. IEEE. [find](https://scholar.google.com/scholar?q=Mike+Schuster+and+Kaisuke+Nakajima%2E+2012%2E+Japanese+and+korean+voice+search%2E+In+2012+IEEE+international+conference+on+acoustics%2C+speech+and+signal+processing+%28ICASSP%29%2C+pages+5149%E2%80%935152%2E+IEEE%2E)
53. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909. [our card](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/)
54. Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, and Tatiana Shavrina. 2022. mgpt: Few-shot learners go multilingual. arXiv preprint arXiv:2204.07580. [our card](https://md.research.elamur.ai/free/en/papers/shliazhko-2022-mgpt-few-shot-learners-go/)
55. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–… [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Alex+Perelygin%2C+Jean+Wu%2C+Jason+Chuang%2C+Christopher+D%2E+Manning%2C+Andrew+Ng%2C+and+Christopher+Potts%2E+2013%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E+In)
56. Felix Stollenwerk. 2023. Training and evaluation of a multilingual tokenizer for gpt-sw3. [find](https://scholar.google.com/scholar?q=Felix+Stollenwerk%2E+2023%2E+Training+and+evaluation+of+a+multilingual+tokenizer+for+gpt%2Dsw3%2E)
57. Yi Tay, Vinh Q Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. 2021. Charformer: Fast character transformers via gradient-based subword tokenization. In International Conference on Learning Representations. [our card](https://md.research.elamur.ai/free/en/papers/tay-2021-charformer-fast-character-transformers-gradient/)
58. Cagri Toraman, Eyup Halit Yilmaz, Furkan Sahinuc, and Oguzhan Ozcelik. 2023. Impact of tokenization on language models: An analysis for turkish. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 22(4). [find](https://scholar.google.com/scholar?q=Cagri+Toraman%2C+Eyup+Halit+Yilmaz%2C+Furkan+Sahinuc%2C+and+Oguzhan+Ozcelik%2E+2023%2E+Impact+of+tokenization+on+language+models%3A+An+analysis+for+turkish%2E+ACM+Trans%2E+Asian+Low%2DResour%2E+Lang%2E+Inf%2E+Process%2E%2C+22%284%29)
59. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: open and efficient foundation language models, 2023. URL https://arxiv.org/abs/2302.13971. [↗](https://arxiv.org/abs/2302.13971) [our card](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/)
60. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+Dan+Bikel%2C+Lukas+Blecher%2C+Cristian+Canton%2DFe)
61. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS, pages 5998–6008. [our card](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/)
62. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,… [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
63. Changhan Wang, Kyunghyun Cho, and Jiatao Gu. 2020. Neural machine translation with byte-level subwords. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):9154–9160. [our card](https://md.research.elamur.ai/free/en/papers/wang-2019-neural-machine-translation-byte-level/)
64. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa… [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chap)
65. Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. Byt5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291–306. [find](https://scholar.google.com/scholar?q=Linting+Xue%2C+Aditya+Barua%2C+Noah+Constant%2C+Rami+Al%2DRfou%2C+Sharan+Narang%2C+Mihir+Kale%2C+Adam+Roberts%2C+and+Colin+Raffel%2E+2022%2E+Byt5%3A+Towards+a+token%2Dfree+future+with+pre%2Dtrained+byte%2Dto%2Dbyte+models%2E+Transac)
66. Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processin… [our card](https://md.research.elamur.ai/free/en/papers/yang-2019-paws-x-cross-lingual-adversarial/)
67. Shaked Yehezkel and Yuval Pinter. 2023. Incorporating context into subword vocabularies. In EACL, pages 623–635. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shaked+Yehezkel+and+Yuval+Pinter%2E+2023%2E+Incorporating+context+into+subword+vocabularies%2E+In+EACL%2C+pages+623%E2%80%93635%2E+Association+for+Computational+Linguistics%2E)
68. Lili Yu, Dániel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. 2023. Megabyte: Predicting million-byte sequences with multiscale transformers. arXiv preprint arXiv:2305.07185. [find](https://scholar.google.com/scholar?q=Lili+Yu%2C+D%C3%A1niel+Simig%2C+Colin+Flaherty%2C+Armen+Aghajanyan%2C+Luke+Zettlemoyer%2C+and+Mike+Lewis%2E+2023%2E+Megabyte%3A+Predicting+million%2Dbyte+sequences+with+multiscale+transformers%2E+arXiv+preprint+arXiv%3A2305%2E071)
69. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In ACL (1), pages 4791–4800. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+In+ACL+%281%29%2C+pages+4791%E2%80%934800%2E+Association+for+Computational+Linguisti)
70. Shiyue Zhang, Vishrav Chaudhary, Naman Goyal, James Cross, Guillaume Wenzek, Mohit Bansal, and Francisco Guzman. 2022. How robust is neural machine translation to language imbalance in multilingual tokenizer training? In Proceedings of the 15th biennial conference of the Association for Machine Tran… [find](https://scholar.google.com/scholar?q=Shiyue+Zhang%2C+Vishrav+Chaudhary%2C+Naman+Goyal%2C+James+Cross%2C+Guillaume+Wenzek%2C+Mohit+Bansal%2C+and+Francisco+Guzman%2E+2022%2E+How+robust+is+neural+machine+translation+to+language+imbalance+in+multilingual+to)
71. Name Language Oscar Oscar Oscar Oscar Oscar DE ES EN IT FR Pile Pile Pile Pile Pile Name Language 11.200.000.000 11.200.000.000 11.200.000.000 11.200.000.000 11.200.000.000 Oscar Pile RedPajama RedPajama Misc EN EN EN Code EN DE ES EN IT FR 13.838.432 21.990.512 4.334.313.669 7.946.402 15.857.811 To… [find](https://scholar.google.com/scholar?q=Name+Language+Oscar+Oscar+Oscar+Oscar+Oscar+DE+ES+EN+IT+FR+Pile+Pile+Pile+Pile+Pile+Name+Language+11%2E200%2E000%2E000+11%2E200%2E000%2E000+11%2E200%2E000%2E000+11%2E200%2E000%2E000+11%2E200%2E000%2E000+Oscar+Pile+RedPajama+RedPaj)
72. Our web documents in the corpora consist of Oscars 1 (Abadji et al., 2021), that were generated by the ungoliant pipeline 2 based on three Common Crawl WET Archives (2022-27, 2022-49 and 2023-14). The curated datasets consist of The Pile (Gao et al., 2020a), RedPajama (Computer, 2023), and single da… [find](https://scholar.google.com/scholar?q=Our+web+documents+in+the+corpora+consist+of+Oscars+1+%28Abadji+et+al%2E%2C+2021%29%2C+that+were+generated+by+the+ungoliant+pipeline+2+based+on+three+Common+Crawl+WET+Archives+%282022%2D27%2C+2022%2D49+and+2023%2D14%29%2E+The)
73. Corpora 56.000.000.000 4.893.724.288 5.308.974.750 2.299.301.635 1.497.999.327 *Table 7: Overview of the English 70B words dataset with language, number of sampled words* *Table 6: Overview of the multilingual 70B words dataset with language, number of sampled words* [find](https://scholar.google.com/scholar?q=Corpora+56%2E000%2E000%2E000+4%2E893%2E724%2E288+5%2E308%2E974%2E750+2%2E299%2E301%2E635+1%2E497%2E999%2E327+%2ATable+7%3A+Overview+of+the+English+70B+words+dataset+with+language%2C+number+of+sampled+words%2A+%2ATable+6%3A+Overview+of+the+mul)