---
title: "Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs — Singh 2024"
description: "The paper asks whether the way numbers are split into tokens affects how well frontier LLMs do arithmetic. 41 references, cited by 44 papers here."
authors: ["Aaditya K. Singh", "DJ Strouse"]
year: 2024
venue: "arXiv:2402.14903"
source: "https://arxiv.org/abs/2402.14903"
doi: "10.48550/arxiv.2402.14903"
arxiv: "2402.14903"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/singh-2024-tokenization-counts-impact-tokenization-arithmetic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/singh-2024-tokenization-counts-impact-tokenization-arithmetic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs

# Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs

**Authors:** Aaditya K. Singh and DJ Strouse · **Year:** 2024 · **Venue:** arXiv:2402.14903 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2402.14903) · [PDF](https://arxiv.org/pdf/2402.14903) · [DOI 10.48550/arxiv.2402.14903](https://doi.org/10.48550/arxiv.2402.14903) · [arXiv 2402.14903](https://arxiv.org/abs/2402.14903) · [OpenAlex](https://openalex.org/W4392181746)

**Citations:** 9 citations (OpenAlex, 2026-10-14) · 44 in our library, of the 140,884 papers whose bibliographies we read

## Summary
The paper asks whether the way numbers are split into tokens affects how well frontier LLMs do arithmetic. GPT-3.5 and GPT-4 use the cl100k_base tokenizer, which has a token for every 1-, 2- and 3-digit string and chunks long numbers into groups of three starting from the left (L2R). By contrast, LLaMa, PaLM and Mistral use single-digit tokens, and older GPT-3 relied on irregular pure BPE splits. The authors ran few-shot addition problems through OpenAI chat models with greedy decoding. They compared the default L2R chunking with right-to-left (R2L) chunking, which they forced at inference time by inserting thousands-separator commas. R2L inputs gave much higher accuracy. The authors trace the L2R failures to misalignment between how the addends and the answer are tokenized, and they show that these errors follow systematic patterns rather than random noise. Asking the model to first rewrite numbers in R2L form recovers most of the lost accuracy. The work shows that tokenization is a real inductive bias that model builders should test deliberately.

## Key points
- Accuracy on the headline addition task (March 2023 checkpoints), L2R vs. comma-enforced R2L:
  - GPT-3.5: 75.6% vs. 97.8%
  - GPT-4: 84.4% vs. 98.9%
  - Overall, R2L gains reach about 20 points.
- Models tested: gpt-3.5-turbo-0301 (main), gpt-3.5-turbo-0613, gpt-4-0314 and gpt-4-0613. Tokenizers compared: p50k_base (GPT-3) and cl100k_base (GPT-3.5/4).
- Under L2R, GPT-3.5 shows a strikingly regular error pattern, the "digit 4" error, which suggests flawed but systematic internal computation.
- A chain-of-thought-style mitigation, where the model repeats the input in R2L format before answering, closes most of the accuracy gap.
- The L2R/R2L gap narrows in larger models, so smaller or distilled models are more exposed to this bias. Code and results are available at github.com/aadityasingh/TokenizationCounts.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Mathematics & statistics](https://md.research.elamur.ai/free/en/shelves/mathematics/)

Topics: [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · [✂️ Tokenization & segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/)

## Cite

Cite as: Aaditya K. Singh and DJ Strouse. 2024. Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs. arXiv:2402.14903.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{singh2024tokenization,
  title = {{Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs}},
  author = {Aaditya K. Singh and DJ Strouse},
  year = {2024},
  doi = {10.48550/arxiv.2402.14903},
  eprint = {2402.14903},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.14903}
}
```

- [← previous Tokenization Is More Than Compression](https://md.research.elamur.ai/free/en/papers/schmidt-2024-tokenization-is-more-than-compression/)
- [next → VariErr NLI: Separating Annotation Error from Human Label Variation](https://md.research.elamur.ai/free/en/papers/genzel-2024-varierr-nli-separating-annotation-error/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

41 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anil, C., Wu, Y., Andreassen, A. J., Lewkowycz, A., Misra, V., Ramasesh, V. V., Slone, A., Gur-Ari, G., Dyer, E., and Neyshabur, B. Exploring length generalization in large language models. Neural Information Processing Systems (NeurIPS), 2022. URL [https://openreview.net/](https://openreview.net/) forum?id=zSkYVeX7bC4. [↗](https://openreview.net/) [our card](https://md.research.elamur.ai/free/en/papers/anil-2022-exploring-length-generalization-large-language/)
2. URL Deng, Y., Zhang, W., Chen, Z., and Gu, Q. Rephrase and respond: Let large language models ask better questions for themselves. arXiv:2311.04205, 2023. URL https://arxiv.org/abs/2311.04205. [↗](https://arxiv.org/abs/2311.04205)
3. Gage, P. A new algorithm for data compression. C Users Journal, 1994. [find](https://scholar.google.com/scholar?q=Gage%2C+P%2E+A+new+algorithm+for+data+compression%2E+C+Users+Journal%2C+1994%2E)
4. Bostrom, K. and Durrett, G. Byte pair encoding is suboptimal for language model pretraining. Empirical Methods in Natural Language Processing (EMNLP), 2020. URL https://aclanthology.org/ 2020.findings-emnlp.414. [↗](https://aclanthology.org/) [our card](https://md.research.elamur.ai/free/en/papers/bostrom-2020-byte-pair-encoding-is-suboptimal/)
5. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The Pile: An 800gb dataset of diverse text for language modeling. arXiv:2101.00027, 2020. URL https://arxiv.org/abs/2101.00027. [↗](https://arxiv.org/abs/2101.00027)
6. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess… [our card](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
7. Gemini Team, Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., Silver, D., Petrov, S., Johnson, M., Antonoglou, I., Schrittwieser, J., Glaese, A., Chen, J., Pitler, E., Lillicrap, T., Lazaridou, A., Firat, O., Molloy, J., Isard,… [our card](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/)
8. Golkar, S., Pettee, M., Eickenberg, M., Bietti, A., Cranmer, M., Krawezik, G., Lanusse, F., McCabe, M., Ohana, R., Parker, L., Blancard, B. R.-S., Tesileanu, T., Cho, K., and Ho, S. xval: A continuous number encoding for large language models. Neural Information Processing Systems (NeurIPS) AI for S… [find](https://scholar.google.com/scholar?q=Golkar%2C+S%2E%2C+Pettee%2C+M%2E%2C+Eickenberg%2C+M%2E%2C+Bietti%2C+A%2E%2C+Cranmer%2C+M%2E%2C+Krawezik%2C+G%2E%2C+Lanusse%2C+F%2E%2C+McCabe%2C+M%2E%2C+Ohana%2C+R%2E%2C+Parker%2C+L%2E%2C+Blancard%2C+B%2E+R%2E%2DS%2E%2C+Tesileanu%2C+T%2E%2C+Cho%2C+K%2E%2C+and+Ho%2C+S%2E+xval%3A+A+continuous)
9. Goyal, S., Ji, Z., Rawat, A. S., Menon, A. K., Kumar, S., and Nagarajan, V. Think before you speak: Training language models with pause tokens. International Conference on Learning Representations (ICLR), 2024. URL [https://openreview.net/forum?id=ph04CRkPdC](https://openreview.net/forum?id=ph04CRkPdC). [↗](https://openreview.net/forum?id=ph04CRkPdC)
10. Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang, Y., Arora, S., Atkinson, D., Authur, R., Chandu, K. R., Cohan, A., Dumas, J., Elazar, Y., Gu, Y., Hessel, J., Khot, T., Merrill, W., Morrison, J., Muennighoff, N., Naik, A., Nam,… [find](https://scholar.google.com/scholar?q=Groeneveld%2C+D%2E%2C+Beltagy%2C+I%2E%2C+Walsh%2C+P%2E%2C+Bhagia%2C+A%2E%2C+Kinney%2C+R%2E%2C+Tafjord%2C+O%2E%2C+Jha%2C+A%2E+H%2E%2C+Ivison%2C+H%2E%2C+Magnusson%2C+I%2E%2C+Wang%2C+Y%2E%2C+Arora%2C+S%2E%2C+Atkinson%2C+D%2E%2C+Authur%2C+R%2E%2C+Chandu%2C+K%2E+R%2E%2C+Cohan%2C+A%2E%2C+Dumas%2C+J%2E%2C+)
11. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the MATH dataset. Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2021. URL [https://openreview.net/forum?id=7Bywt2mQsCe](https://openreview.net/forum?id=7Bywt2mQsCe). [↗](https://openreview.net/forum?id=7Bywt2mQsCe) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/)
12. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b. arXiv:2310.06825, 2023. URL https://a… [find](https://scholar.google.com/scholar?q=Jiang%2C+A%2E+Q%2E%2C+Sablayrolles%2C+A%2E%2C+Mensch%2C+A%2E%2C+Bamford%2C+C%2E%2C+Chaplot%2C+D%2E+S%2E%2C+de+las+Casas%2C+D%2E%2C+Bressand%2C+F%2E%2C+Lengyel%2C+G%2E%2C+Lample%2C+G%2E%2C+Saulnier%2C+L%2E%2C+Lavaud%2C+L%2E+R%2E%2C+Lachaux%2C+M%2E%2DA%2E%2C+Stock%2C+P%2E%2C+Scao%2C+T%2E+L%2E%2C+L)
13. Muffo, M., Cocco, A., and Bertino, E. Evaluating transformer language models on arithmetic operations using number decomposition. Language Resources and Evaluation Conference (LREC), 2022. URL https://aclanthology.org/2022.lrec-1.30. [↗](https://aclanthology.org/2022.lrec-1.30)
14. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. Neural Information Processing Systems (NeurIPS), 2022. URL [https://openreview.net/](https://openreview.net/) forum?id=e2TBb5y0yFf. [↗](https://openreview.net/) [our card](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/)
15. Nogueira, R., Jiang, Z., and Lin, J. Investigating the limitations of transformers with simple arithmetic tasks. arXiv:2102.13019, 2021. URL https://arxiv.org/abs/2102.13019. [↗](https://arxiv.org/abs/2102.13019)
16. Kudo, T. Subword regularization: Improving neural network translation models with multiple subword candidates. Association for Computational Linguistics (ACL), 2018. URL https://aclanthology.org/P18-1007. [↗](https://aclanthology.org/P18-1007) [our card](https://md.research.elamur.ai/free/en/papers/kudo-2018-subword-regularization-improving-neural-network/)
17. Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., et al. Show your work: Scratchpads for intermediate computation with language models. arXiv:2112.00114, 2021. URL https://arxiv.org/abs/2112.00114. [↗](https://arxiv.org/abs/2112.00114) [our card](https://md.research.elamur.ai/free/en/papers/al-2021-show-your-work-scratchpads-intermediate/)
18. Lanham, T., Chen, A., Radhakrishnan, A., Steiner, B., Denison, C., Hernandez, D., Li, D., Durmus, E., Hubinger, E., Kernion, J., et al. Measuring faithfulness in chain-of-thought reasoning. arXiv:2307.13702, 2023. URL https://arxiv.org/abs/2307.13702. [↗](https://arxiv.org/abs/2307.13702) [our card](https://md.research.elamur.ai/free/en/papers/lanham-2023-measuring-faithfulness-chain-thought-reasoning/)
19. OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Balaji, S., Balcom, V., Baltescu, P., Bao, H., Bavarian, M., Belgum, J., Bello, I., Berdine, J., Bernadett-Shapiro, G., Berner, C., Bog… [find](https://scholar.google.com/scholar?q=OpenAI%2C+Achiam%2C+J%2E%2C+Adler%2C+S%2E%2C+Agarwal%2C+S%2E%2C+Ahmad%2C+L%2E%2C+Akkaya%2C+I%2E%2C+Aleman%2C+F%2E+L%2E%2C+Almeida%2C+D%2E%2C+Altenschmidt%2C+J%2E%2C+Altman%2C+S%2E%2C+Anadkat%2C+S%2E%2C+Avila%2C+R%2E%2C+Babuschkin%2C+I%2E%2C+Balaji%2C+S%2E%2C+Balcom%2C+V%2E%2C+Baltescu%2C+P)
20. Lewkowycz, A., Andreassen, A. J., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V. V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., Wu, Y., Neyshabur, B., Gur-Ari, G., and Misra, V. Solving quantitative reasoning problems with language models. Neural Information Processing Systems (NeurIPS),… [our card](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/)
21. Li, Y., Bubeck, S., Eldan, R., Del Giorno, A., Gunasekar, S., and Lee, Y. T. Textbooks Are All You Need II: phi-1.5 technical report. arXiv preprint arXiv:2309.05463, 2023. [our card](https://md.research.elamur.ai/free/en/papers/li-2023-textbooks-all-need-ii-phi/)
22. Liu, R., Garrette, D., Saharia, C., Chan, W., Roberts, A., Narang, S., Blok, I., Mical, R., Norouzi, M., and Constant, N. Character-aware models improve visual text rendering. Association for Computational Linguistics (ACL), 2023. URL https://aclanthology.org/ 2023.acl-long.900. [↗](https://aclanthology.org/)
23. Lundberg, S. The art of prompt design: Prompt boundaries and token healing, 2023. URL [https://towardsdatascience.com/the-art-of-prompt-design-prompt-boundaries-and-token-healing-3b2448b0be38](https://towardsdatascience.com/the-art-of-prompt-design-prompt-boundaries-and-token-healing-3b2448b0be38). [↗](https://towardsdatascience.com/the-art-of-prompt-design-prompt-boundaries-and-token-healing-3b2448b0be38)
24. McCoy, R. T., Yao, S., Friedman, D., Hardy, M., and Griffiths, T. L. Embers of autoregression: Understanding large language models through the problem they Mély, D., Nair, A., Nakano, R., Nayak, R., Neelakantan, A., Ngo, R., Noh, H., Ouyang, L., O’Keefe, C., Pachocki, J., Paino, A., Palermo, J., Pa… [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
25. Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., and Singh, S. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. Empirical Methods in Natural Language Processing (EMNLP), 2020. URL https://aclanthology.org/2020.emnlp-main.346. [↗](https://aclanthology.org/2020.emnlp-main.346) [our card](https://md.research.elamur.ai/free/en/papers/shin-2020-autoprompt-eliciting-knowledge-language-models/)
26. Stolfo, A., Belinkov, Y., and Sachan, M. A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis. Empirical Methods in Natural Language Processing (EMNLP), 2023. URL [https://openreview.net/](https://openreview.net/) forum?id=aB3Hwh4UzP. [↗](https://openreview.net/)
27. Sun, K., Qi, P., Zhang, Y., Liu, L., Wang, W., and Huang, Z. Tokenization consistency matters for generative models on extractive NLP tasks. Empirical Methods in Natural Language Processing (EMNLP), 2023. URL https://aclanthology.org/ 2023.findings-emnlp.887. [↗](https://aclanthology.org/)
28. Teknium. How did the gpt tokenizer get created?, 2023. URL [https://twitter.com/Teknium1/](https://twitter.com/Teknium1/) status/1634667026739527680?s=20. [↗](https://twitter.com/Teknium1/)
29. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv:2302.13971, 2023a. URL https://arxiv.org/abs/2302.13971. [↗](https://arxiv.org/abs/2302.13971) [our card](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/)
30. Paster, K. Testing language models on a held-out high school national finals exam. [https://huggingface.co/datasets/keirp/](https://huggingface.co/datasets/keirp/) hungarian national hs finals exam, 2023. [↗](https://huggingface.co/datasets/keirp/)
31. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hossei… [find](https://scholar.google.com/scholar?q=Touvron%2C+H%2E%2C+Martin%2C+L%2E%2C+Stone%2C+K%2E%2C+Albert%2C+P%2E%2C+Almahairi%2C+A%2E%2C+Babaei%2C+Y%2E%2C+Bashlykov%2C+N%2E%2C+Batra%2C+S%2E%2C+Bhargava%2C+P%2E%2C+Bhosale%2C+S%2E%2C+Bikel%2C+D%2E%2C+Blecher%2C+L%2E%2C+Ferrer%2C+C%2E+C%2E%2C+Chen%2C+M%2E%2C+Cucurull%2C+G%2E%2C+Esiobu%2C+D)
32. Razeghi, Y., Logan IV, R. L., Gardner, M., and Singh, S. Impact of pretraining term frequencies on few-shot numerical reasoning. Empirical Methods in Natural Language Processing (EMNLP), 2022. URL https://aclanthology.org/ 2022.findings-emnlp.59. [↗](https://aclanthology.org/) [our card](https://md.research.elamur.ai/free/en/papers/razeghi-2022-impact-pretraining-term-frequencies-few/)
33. Rumbelow, J. and mwatkins. Solidgoldmagikarp (plus, prompt generation), 2023. URL [https://www.lesswrong.com/posts/](https://www.lesswrong.com/posts/) aPeJE8bSo6rAFoLqg/solidgoldmagikarp-plus-prompt-generation. [↗](https://www.lesswrong.com/posts/)
34. Saxton, D., Grefenstette, E., Hill, F., and Kohli, P. Analysing mathematical reasoning abilities of neural models. International Conference on Learning Representations (ICLR), 2019. URL [https://openreview.net/](https://openreview.net/) forum?id=H1gR5iR5FX. [↗](https://openreview.net/) [our card](https://md.research.elamur.ai/free/en/papers/saxton-2019-analysing-mathematical-reasoning-abilities-neural/)
35. Wang, B. and Komatsuzaki, A. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model, 2021. URL [https://huggingface.co/EleutherAI/](https://huggingface.co/EleutherAI/) gpt-j-6b. [↗](https://huggingface.co/EleutherAI/)
36. Sennrich, R., Haddow, B., and Birch, A. Neural machine translation of rare words with subword units. Association for Computational Linguistics (ACL), 2016. URL https://aclanthology.org/P16-1162. [↗](https://aclanthology.org/P16-1162) [our card](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/)
37. Wei, J. Sorting a list of words by the second letter, 2023. URL [https://x.com/](https://x.com/) jasonwei/ status/1661781746759909376?s=20. [↗](https://x.com/)
38. Wei, J., Wang, X., Schuurmans, D., Bosma, M., brian ichter, Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. Neural Information Processing Systems (NeurIPS), 2022. URL [https://openreview.net/](https://openreview.net/) forum?id= VjQlMeSB J. [↗](https://openreview.net/) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
39. Yu, L., Simig, D., Flaherty, C., Aghajanyan, A., Zettlemoyer, L., and Lewis, M. MEGABYTE: Predicting million-byte sequences with multiscale transformers. Neural Information Processing Systems (NeurIPS), 2023. URL [https://openreview.net/forum?id=JTmO2V9Xpz](https://openreview.net/forum?id=JTmO2V9Xpz). [↗](https://openreview.net/forum?id=JTmO2V9Xpz)
40. Zhou, H., Nova, A., Larochelle, H., Courville, A., Neyshabur, B., and Sedghi, H. Teaching algorithmic reasoning via in-context learning. arXiv:2211.09066, 2022. URL https://arxiv.org/abs/2211.09066. [↗](https://arxiv.org/abs/2211.09066)
41. Zhou, H., Bradley, A., Littwin, E., Razin, N., Saremi, O., Susskind, J., Bengio, S., and Nakkiran, P. What algorithms can transformers learn? a study in length generalization. arXiv:2310.16028, 2023. URL https://arxiv.org/abs/2310.16028. [↗](https://arxiv.org/abs/2310.16028)