---
title: "Getting the most out of your tokenizer for pre-training and domain adaptation — Dagan 2024"
description: "LLM developers rarely tune the tokenizer. 46 references, cited by 44 papers here."
authors: ["Gautier Dagan", "Gabriel Synnaeve", "Baptiste Roziere"]
year: 2024
venue: "arXiv:2402.01035"
source: "https://arxiv.org/abs/2402.01035"
doi: "10.48550/arxiv.2402.01035"
arxiv: "2402.01035"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/dagan-2024-getting-most-out-tokenizer-pre/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/dagan-2024-getting-most-out-tokenizer-pre/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / Getting the most out of your tokenizer for pre-training and domain adaptation

# Getting the most out of your tokenizer for pre-training and domain adaptation

**Authors:** Gautier Dagan, [Gabriel Synnaeve](https://md.research.elamur.ai/free/en/authors/gabriel-synnaeve/) and [Baptiste Roziere](https://md.research.elamur.ai/free/en/authors/baptiste-roziere/) · **Year:** 2024 · **Venue:** arXiv:2402.01035 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2402.01035) · [PDF](https://arxiv.org/pdf/2402.01035) · [DOI 10.48550/arxiv.2402.01035](https://doi.org/10.48550/arxiv.2402.01035) · [arXiv 2402.01035](https://arxiv.org/abs/2402.01035) · [OpenAlex](https://openalex.org/W4391555698)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [44](https://md.research.elamur.ai/free/en/papers/dagan-2024-getting-most-out-tokenizer-pre/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
LLM developers rarely tune the tokenizer. They typically borrow one and keep it when adapting a base model to a new domain. Code Llama, for example, still uses the original Llama tokenizer. Dagan, Synnaeve and Rozière examine how three tokenizer design choices affect compression, speed, context length, memory and code-generation quality: the tokenizer's training data, its pre-tokenization regular expression and its vocabulary size.

They trained BPE tokenizers with HuggingFace tokenizers on CCnet, Wikipedia and the Stack, measuring compression with Normalized Sequence Length (NSL) relative to Llama. They then trained 1.5B and 7B code models from scratch or fine-tuned them with new tokenizers, and evaluated them on HumanEval and MBPP.

The main findings:
- Vocabulary size barely affects coding accuracy.
- Pre-tokenization strongly influences both compression and accuracy.
- A pre-trained model's tokenizer can be replaced with little loss once fine-tuning exceeds about 50B tokens.

This gives practitioners a practical way to make domain-adapted models faster and to fit more text into their context window.

## Key points
- Code tokenizers can shorten code sequences by over 40% versus Llama. They gain more than 25% with no accuracy loss when using the GPT-4 regex at a 256k vocabulary.
- The authors recommend the GPT-4 pre-tokenization pattern. It compresses about 5% better than Punct with similar scores. Identity (no pre-tokenization) compresses most but hurts accuracy.
- Vocabulary size and HumanEval Pass@1 showed only a weak, non-significant correlation (−0.13, p = 0.87).
- Llama 2 7B was retrained on 500B code tokens with 80k Punct, GPT-4 and Merged tokenizers. These models reached 30.4–31.2% HumanEval Pass@1, versus 32.1% for Code Llama 7B.
- The final tokenizers were trained on 70% code and 30% English. Evaluation used held-out sets covering 30 programming languages and 28 natural languages.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) · [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

Topics: [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · [✂️ Tokenization & segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/)

## Cite

Cite as: Gautier Dagan, Gabriel Synnaeve and Baptiste Roziere. 2024. Getting the most out of your tokenizer for pre-training and domain adaptation. arXiv:2402.01035.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{dagan2024getting,
  title = {{Getting the most out of your tokenizer for pre-training and domain adaptation}},
  author = {Gautier Dagan, Gabriel Synnaeve and Baptiste Roziere},
  year = {2024},
  doi = {10.48550/arxiv.2402.01035},
  eprint = {2402.01035},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.01035}
}
```

- [← previous Bridging the Preference Gap between Retrievers and LLMs](https://md.research.elamur.ai/free/en/papers/ke-2024-bridging-preference-gap-between-retrievers/)
- [next → Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies](https://md.research.elamur.ai/free/en/papers/tao-2024-scaling-laws-vocabulary-larger-models/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

46 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. 01.AI. Yi series models: Large language models. [our card](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/)
2. Hugging Face Model Repository, 2023. URL [https://huggingface.co/01-ai/Yi-6B](https://huggingface.co/01-ai/Yi-6B). Accessed: 17/11/2023. [↗](https://huggingface.co/01-ai/Yi-6B)
3. Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397–2430. PMLR, 20… [find](https://scholar.google.com/scholar?q=Biderman%2C+S%2E%2C+Schoelkopf%2C+H%2E%2C+Anthony%2C+Q%2E+G%2E%2C+Bradley%2C+H%2E%2C+O%E2%80%99Brien%2C+K%2E%2C+Hallahan%2C+E%2E%2C+Khan%2C+M%2E+A%2E%2C+Purohit%2C+S%2E%2C+Prashanth%2C+U%2E+S%2E%2C+Raff%2C+E%2E%2C+et+al%2E+Pythia%3A+A+suite+for+analyzing+large+language+models+a)
4. Ahmad, W., Chakraborty, S., Ray, B., and Chang, K.-W. Unified pre-training for program understanding and generation. In Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-Tur, D., Beltagy, I., Bethard, S., Cotterell, R., Chakraborty, T., and Zhou, Y. (eds.), Proceedings of the 2021 Conference of… [our card](https://md.research.elamur.ai/free/en/papers/ahmad-2021-unified-pre-training-program-understanding/)
5. Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., Pieler, M., Prashanth, U. S., Purohit, S., Reynolds, L., Tow, J., Wang, B., and Weinbach, S. GPT-NeoX-20B: An open-source autoregressive language model. In Fan, A., Ilic, S., Wolf, T… [our card](https://md.research.elamur.ai/free/en/papers/black-2022-gpt-neox-20b-open-source/)
6. Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebron, F., and Sanghai, S. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Proce… [our card](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/)
7. Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C.… [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
8. Allal, L. B., Li, R., Kocetkov, D., Mou, C., Akiki, C., Ferrandis, C. M., Muennighoff, N., Mishra, M., Gu, A., Dey, M., Umapathi, L. K., Anderson, C. J., Zi, Y., Lamy-Poirier, J., Schoelkopf, H., Troshin, S., Abulkhanov, D., Romero, M., Lappert, M., Toni, F. D., del Rı́o, B. G., Liu, Q., Bose, S., B… [our card](https://md.research.elamur.ai/free/en/papers/allal-2023-santacoder-don-t-reach-stars/)
9. Chirkova, N. and Troshin, S. Codebpe: Investigating subtok-enization options for large language model pretraining on source code. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. [find](https://scholar.google.com/scholar?q=Chirkova%2C+N%2E+and+Troshin%2C+S%2E+Codebpe%3A+Investigating+subtok%2Denization+options+for+large+language+model+pretraining+on+source+code%2E+In+The+Eleventh+International+Conference+on+Learning+Representations%2C+)
10. OpenReview.net, 2023. URL [https://openreview.net/pdf?id=htL4UZ344nF](https://openreview.net/pdf?id=htL4UZ344nF). [↗](https://openreview.net/pdf?id=htL4UZ344nF)
11. Almazrouei, E., Alobeidli, H., Alshamsi, A., Cappelli, A., Cojocaru, R., Debbah, M., Étienne Goffinet, Hesslow, D., Launay, J., Malartic, Q., Mazzotta, D., Noune, B., Pannier, B., and Penedo, G. The falcon series of open language models, 2023. [our card](https://md.research.elamur.ai/free/en/papers/almazrouei-2023-falcon-series-open-language-models/)
12. Deci. Introducing decicoder: The new gold standard in efficient and accurate code generation, August 2023. URL [https://deci.ai/blog/](https://deci.ai/blog/). Accessed: 2024-01-18. [↗](https://deci.ai/blog/)
13. Anthropic. Introducing claude. Anthropic Blog, March 2023. URL [https://www.anthropic.com/index/introducing-claude](https://www.anthropic.com/index/introducing-claude). Accessed: 17/11/2023. [↗](https://www.anthropic.com/index/introducing-claude)
14. DeepSeek AI. Deepseek coder: A series of code language models. DeepSeek Coder Official Website, 2023. URL [https://deepseekcoder.github.io/](https://deepseekcoder.github.io/). Accessed: 17/11/2023. [↗](https://deepseekcoder.github.io/)
15. Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In Burstein, J., Doran, C., and Solorio, T. (eds… [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
16. Goyal, S., Ji, Z., Rawat, A. S., Menon, A. K., Kumar, S., and Nagarajan, V. Think before you speak: Training language models with pause tokens. CoRR, abs/2310.02226, 2023. doi: 10.48550/ARXIV.2310.02226. URL https://doi.org/10.48550/arXiv.2310.02226. guidance-ai. A guidance language for controlling… [↗](https://doi.org/10.48550/arXiv.2310.02226)
17. Elsen, E., Odena, A., Nye, M., Taşırlar, S., Dao, T., Hawthorne, C., Moparthi, D., and Somani, A. Releasing Persimmon-8B, 2023. URL [https://www.adept.ai/blog/persimmon-8b](https://www.adept.ai/blog/persimmon-8b). [↗](https://www.adept.ai/blog/persimmon-8b)
18. Forsythe, A. Tokenmonster: Ungreedy subword tokenizer and vocabulary trainer for python, go and javascript. [https://github.com/alasdairforsythe/](https://github.com/alasdairforsythe/) tokenmonster, 2023. [↗](https://github.com/alasdairforsythe/)
19. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de Las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b. CoRR, abs/2310.06825, 2023. doi: 10.4855… [find](https://scholar.google.com/scholar?q=Jiang%2C+A%2E+Q%2E%2C+Sablayrolles%2C+A%2E%2C+Mensch%2C+A%2E%2C+Bamford%2C+C%2E%2C+Chaplot%2C+D%2E+S%2E%2C+de+Las+Casas%2C+D%2E%2C+Bressand%2C+F%2E%2C+Lengyel%2C+G%2E%2C+Lample%2C+G%2E%2C+Saulnier%2C+L%2E%2C+Lavaud%2C+L%2E+R%2E%2C+Lachaux%2C+M%2E%2C+Stock%2C+P%2E%2C+Scao%2C+T%2E+L%2E%2C+Lavr)
20. Kocetkov, D., Li, R., Allal, L. B., Li, J., Mou, C., Ferrandis, C. M., Jernite, Y., Mitchell, M., Hughes, S., Wolf, T., Bahdanau, D., von Werra, L., and de Vries, H. The stack: 3 TB of permissively licensed source code. CoRR, abs/2211.15533, 2022. doi: 10.48550/ARXIV. 2211.15533. URL [https://doi.org…](https://doi.org…) [↗](https://doi.org…)
21. Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., Yih, S., Zettlemoyer, L., and Lewis, M. Incoder: A generative model for code infilling and synthesis. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. [find](https://scholar.google.com/scholar?q=Fried%2C+D%2E%2C+Aghajanyan%2C+A%2E%2C+Lin%2C+J%2E%2C+Wang%2C+S%2E%2C+Wallace%2C+E%2E%2C+Shi%2C+F%2E%2C+Zhong%2C+R%2E%2C+Yih%2C+S%2E%2C+Zettlemoyer%2C+L%2E%2C+and+Lewis%2C+M%2E+Incoder%3A+A+generative+model+for+code+infilling+and+synthesis%2E+In+The+Eleventh+Int)
22. OpenReview.net, 2023. URL [https://openreview.net/pdf?id=hQwb-lbM6EL](https://openreview.net/pdf?id=hQwb-lbM6EL). [↗](https://openreview.net/pdf?id=hQwb-lbM6EL)
23. Gee, L., Zugarini, A., Rigutini, L., and Torroni, P. Fast vocabulary transfer for language model compression. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp. 409–416, Abu Dhabi, UAE, December 2022. Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Gee%2C+L%2E%2C+Zugarini%2C+A%2E%2C+Rigutini%2C+L%2E%2C+and+Torroni%2C+P%2E+Fast+vocabulary+transfer+for+language+model+compression%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Natural+Language+Processing%3A)
24. Gee, L., Rigutini, L., Ernandes, M., and Zugarini, A. Multi-word tokenization for sequence compression. In Wang, M. and Zitouni, I. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp. 612–621, Singapore, December 2023. Association for C… [find](https://scholar.google.com/scholar?q=Gee%2C+L%2E%2C+Rigutini%2C+L%2E%2C+Ernandes%2C+M%2E%2C+and+Zugarini%2C+A%2E+Multi%2Dword+tokenization+for+sequence+compression%2E+In+Wang%2C+M%2E+and+Zitouni%2C+I%2E+%28eds%2E%29%2C+Proceedings+of+the+2023+Conference+on+Empirical+Methods+in+N)
25. Gowda, T. and May, J. Finding the optimal vocabulary size for neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 3955–3964, Online, Nov 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.findings-emnlp.352. URL https://aclant… [our card](https://md.research.elamur.ai/free/en/papers/gowda-2020-finding-optimal-vocabulary-size-neural/)
26. Kudo, T. Subword regularization: Improving neural network translation models with multiple subword candidates. In Gurevych, I. and Miyao, Y. (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Lo… [our card](https://md.research.elamur.ai/free/en/papers/kudo-2018-subword-regularization-improving-neural-network/)
27. Kudo, T. and Richardson, J. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Blanco, E. and Lu, W. (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018: System Demonstrations, Bruss… [our card](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/)
28. Kulal, S., Pasupat, P., Chandra, K., Lee, M., Padon, O., Aiken, A., and Liang, P. Spoc: Search-based pseudocode to code. In Wallach, H. M., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E. B., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32: Annual Conference o… — [doi:10.1016/j.artint.2023.103860](https://doi.org/10.1016/j.artint.2023.103860) [find](https://scholar.google.com/scholar?q=Kulal%2C+S%2E%2C+Pasupat%2C+P%2E%2C+Chandra%2C+K%2E%2C+Lee%2C+M%2E%2C+Padon%2C+O%2E%2C+Aiken%2C+A%2E%2C+and+Liang%2C+P%2E+Spoc%3A+Search%2Dbased+pseudocode+to+code%2E+In+Wallach%2C+H%2E+M%2E%2C+Larochelle%2C+H%2E%2C+Beygelzimer%2C+A%2E%2C+d%E2%80%99Alche%CC%81%2DBuc%2C+F%2E%2C+Fox%2C+E%2E+B)
29. OpenReview.net, 2023. URL https: J., Patel, S. S., Abulkhanov, D., Zocca, M., Dey, M., //openreview.net/pdf?id=iaYcJKpY2B_. Zhang, Z., Moustafa-Fahmy, N., Bhattacharyya, U., Yu, W., Singh, S., Luccioni, S., Villegas, P., Kunakov, M., Nogueira, R. F., Jiang, Z., and Lin, J. Investigating the Zhdanov,… [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
30. Replit Blog, October 2023. URL https: languages. In Rogers, A., Boyd-Graber, J., and //blog.replit.com/replit-code-v1_5. Ac-Okazaki, N. (eds.), Findings of the Association for cessed: 17/11/2023. [find](https://scholar.google.com/scholar?q=Replit+Blog%2C+October+2023%2E+URL+https%3A+languages%2E+In+Rogers%2C+A%2E%2C+Boyd%2DGraber%2C+J%2E%2C+and+%2F%2Fblog%2Ereplit%2Ecom%2Freplit%2Dcode%2Dv1%5F5%2E+Ac%2DOkazaki%2C+N%2E+%28eds%2E%29%2C+Findings+of+the+Association+for+cessed%3A+17%2F11%2F2023%2E)
31. Computational Linguistics: ACL 2023, pp. 5661–5681, Rozière, B., Lachaux, M., Chanussot, L., and Lam-Toronto, Canada, July 2023. Association for Computa-ple, G. Unsupervised translation of programming tional Linguistics. doi: 10.18653/v1/2023.findings-acl. languages. In Larochelle, H., Ranzato, M.,… — [doi:10.18653/v1/2023.findings-acl](https://doi.org/10.18653/v1/2023.findings-acl) [find](https://scholar.google.com/scholar?q=Computational+Linguistics%3A+ACL+2023%2C+pp%2E+5661%E2%80%935681%2C+Rozie%CC%80re%2C+B%2E%2C+Lachaux%2C+M%2E%2C+Chanussot%2C+L%2E%2C+and+Lam%2DToronto%2C+Canada%2C+July+2023%2E+Association+for+Computa%2Dple%2C+G%2E+Unsupervised+translation+of+programmin)
32. OpenReview.net, 2019. URL https: neurips.cc/paper/2020/hash/ //openreview.net/forum?id=Bkg6RiCqY7. ed23fbf18c2cd35f8c7f8de44f85c08d-Abstract. html. MosaicML. Mpt-7b-instruct: A model for short-form instruction following. MosaicML Blog, May Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I.… [our card](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/)
33. Rust, P., Pfeiffer, J., Vulić, I., Ruder, S., and Gurevych, I. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natura… [our card](https://md.research.elamur.ai/free/en/papers/rust-2021-how-good-is-your-tokenizer/)
34. Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ili’c, S., Hesslow, D., Castagn’e, R., Luccioni, A. S., Yvon, F., Gallé, M., Tow, J., Rush, A. M., Biderman, S. R., Webson, A., Ammanamanchi, P. S., Wang, T., Sagot, B., Muennighoff, N., del Moral, A. V., Ruwase, O., Bawden, R., Bekman, S., McMillan-Maj… [find](https://scholar.google.com/scholar?q=Scao%2C+T%2E+L%2E%2C+Fan%2C+A%2E%2C+Akiki%2C+C%2E%2C+Pavlick%2C+E%2E%2C+Ili%E2%80%99c%2C+S%2E%2C+Hesslow%2C+D%2E%2C+Castagn%E2%80%99e%2C+R%2E%2C+Luccioni%2C+A%2E+S%2E%2C+Yvon%2C+F%2E%2C+Galle%CC%81%2C+M%2E%2C+Tow%2C+J%2E%2C+Rush%2C+A%2E+M%2E%2C+Biderman%2C+S%2E+R%2E%2C+Webson%2C+A%2E%2C+Ammanamanchi%2C+P%2E+S%2E%2C+Wang)
35. Sennrich, R., Haddow, B., and Birch, A. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1715–1725, Berlin, Germany, August 2016. Association for Computational Linguist… [our card](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/)
36. Wenzek, G., Lachaux, M.-A., Conneau, A., Chaudhary, V., Guzmán, F., Joulin, A., and Grave, E. CCNet: Extracting high quality monolingual datasets from web crawl data. In Calzolari, N., Béchet, F., Blache, P., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Isahara, H., Maegaard, B., Mariani, J.,… [our card](https://md.research.elamur.ai/free/en/papers/wenzek-2020-ccnet-extracting-high-quality-monolingual/)
37. Thawani, A., Pujara, J., Ilievski, F., and Szekely, P. Representing numbers in NLP: a survey and a vision. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 644–656, Online, June 2021. Association fo… [find](https://scholar.google.com/scholar?q=Thawani%2C+A%2E%2C+Pujara%2C+J%2E%2C+Ilievski%2C+F%2E%2C+and+Szekely%2C+P%2E+Representing+numbers+in+NLP%3A+a+survey+and+a+vision%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+the+Association+for+Co)
38. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023a. doi: 10.48550/ARXIV.2302.13971. URL ht… [our card](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/)
39. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Canton-Ferrer, C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Ho… [find](https://scholar.google.com/scholar?q=Touvron%2C+H%2E%2C+Martin%2C+L%2E%2C+Stone%2C+K%2E%2C+Albert%2C+P%2E%2C+Almahairi%2C+A%2E%2C+Babaei%2C+Y%2E%2C+Bashlykov%2C+N%2E%2C+Batra%2C+S%2E%2C+Bhargava%2C+P%2E%2C+Bhosale%2C+S%2E%2C+Bikel%2C+D%2E%2C+Blecher%2C+L%2E%2C+Canton%2DFerrer%2C+C%2E%2C+Chen%2C+M%2E%2C+Cucurull%2C+G%2E%2C+Esiob)
40. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., and Brew, J. Huggingface’s transformers: State-of-the-art natural language processing. CoRR, abs/1910.03771, 2019. URL [http://arxiv.org/](http://arxiv.org/) abs/1910.03771. [↗](http://arxiv.org/) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
41. Xue, L., Barua, A., Constant, N., Al-Rfou, R., Narang, S., Kale, M., Roberts, A., and Raffel, C. ByT5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291–306, 2022. doi: 10. 1162/tacl a 00461. URL https://aclantholo… [find](https://scholar.google.com/scholar?q=Xue%2C+L%2E%2C+Barua%2C+A%2E%2C+Constant%2C+N%2E%2C+Al%2DRfou%2C+R%2E%2C+Narang%2C+S%2E%2C+Kale%2C+M%2E%2C+Roberts%2C+A%2E%2C+and+Raffel%2C+C%2E+ByT5%3A+Towards+a+token%2Dfree+future+with+pre%2Dtrained+byte%2Dto%2Dbyte+models%2E+Transactions+of+the+Association)
42. Zheng, Q., Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Shen, L., Wang, Z., Wang, A., Li, Y., Su, T., Yang, Z., and Tang, J. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. In Singh, A. K., Sun, Y., Akoglu, L., Gunopulos, D., Yan, X., Kumar, R., Ozc… [our card](https://md.research.elamur.ai/free/en/papers/zheng-2023-codegeex-pre-trained-model-code/)
43. ACM, 2023. doi: 10.1145/3580305.3599790. URL https://doi.org/10.1145/3580305.3599790. — [doi:10.1145/3580305.3599790](https://doi.org/10.1145/3580305.3599790) [↗](https://doi.org/10.1145/3580305.3599790)
44. Zipf, G. K. Human Behavior and the Principle of Least Effort. Addison-Wesley, 1949. [find](https://scholar.google.com/scholar?q=Zipf%2C+G%2E+K%2E+Human+Behavior+and+the+Principle+of+Least+Effort%2E+Addison%2DWesley%2C+1949%2E)
45. Zouhar, V., Meister, C., Gastaldi, J. L., Du, L., Sachan, M., and Cotterell, R. Tokenization and the noiseless channel. In Rogers, A., Boyd-Graber, J. L., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023… [find](https://scholar.google.com/scholar?q=Zouhar%2C+V%2E%2C+Meister%2C+C%2E%2C+Gastaldi%2C+J%2E+L%2E%2C+Du%2C+L%2E%2C+Sachan%2C+M%2E%2C+and+Cotterell%2C+R%2E+Tokenization+and+the+noiseless+channel%2E+In+Rogers%2C+A%2E%2C+Boyd%2DGraber%2C+J%2E+L%2E%2C+and+Okazaki%2C+N%2E+%28eds%2E%29%2C+Proceedings+of+the+61)
46. Wang, Y., Wang, W., Joty, S., and Hoi, S. C. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 8696–8708, Online and Punta Cana, Dominican Republic,… [find](https://scholar.google.com/scholar?q=Wang%2C+Y%2E%2C+Wang%2C+W%2E%2C+Joty%2C+S%2E%2C+and+Hoi%2C+S%2E+C%2E+Codet5%3A+Identifier%2Daware+unified+pre%2Dtrained+encoder%2Ddecoder+models+for+code+understanding+and+generation%2E+In+Proceedings+of+the+2021+Conference+on+Empiric)