---
title: "Positional Description Matters for Transformers Arithmetic (2023)"
description: "Transformers often struggle with arithmetic. 25 references, cited by 27 papers here. Among them: 3."
authors: ["Ruoqi Shen", "Sébastien Bubeck", "Ronen Eldan", "Yin Tat Lee", "Yuanzhi Li", "Yi Zhang"]
year: 2023
venue: "arXiv:2311.14737"
source: "https://arxiv.org/abs/2311.14737"
doi: "10.48550/arxiv.2311.14737"
arxiv: "2311.14737"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shen-2023-positional-description-matters-transformers-arithmetic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shen-2023-positional-description-matters-transformers-arithmetic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/12/) / Positional Description Matters for Transformers Arithmetic

# Positional Description Matters for Transformers Arithmetic

**Authors:** Ruoqi Shen, [Sébastien Bubeck](https://md.research.elamur.ai/free/en/authors/sebastien-bubeck/), [Ronen Eldan](https://md.research.elamur.ai/free/en/authors/ronen-eldan/), [Yin Tat Lee](https://md.research.elamur.ai/free/en/authors/yin-tat-lee/), [Yuanzhi Li](https://md.research.elamur.ai/free/en/authors/yuanzhi-li/) and [Yi Zhang](https://md.research.elamur.ai/free/en/authors/yi-zhang/) · **Year:** 2023 · **Venue:** arXiv:2311.14737 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2311.14737) · [PDF](https://arxiv.org/pdf/2311.14737) · [DOI 10.48550/arxiv.2311.14737](https://doi.org/10.48550/arxiv.2311.14737) · [arXiv 2311.14737](https://arxiv.org/abs/2311.14737) · [OpenAlex](https://openalex.org/W4389115202)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [27](https://md.research.elamur.ai/free/en/papers/shen-2023-positional-description-matters-transformers-arithmetic/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Transformers often struggle with arithmetic. The authors attribute much of this to over-reliance on absolute token positions. A model trained on short numbers learns to align digits by their fixed locations, and that strategy fails on longer numbers or when the same sum appears inside ordinary text.

The paper tests two remedies. The first changes the positional encoding itself, either by dropping absolute embeddings or by adding a randomized embedding. The second changes the data representation so that standard encodings are used differently, for example by inserting random spaces. All experiments use small GPT2-small-sized models trained from scratch, covering three settings:
- direct multiplication with no scratchpad,
- addition on longer numbers than seen in training,
- addition embedded in teacher–student dialogues.

With 300k training examples, a model can multiply 15-digit numbers directly and is nearly flawless up to 12 digits; conventional training fails at 4 digits. With 120k addition samples, the methods allow generalization from 10 to 12 digits and near-perfect dialogue addition up to 5 digits. This suggests that small models can learn arithmetic if positional cues are handled carefully.

## Key points
- **Model and data:** GPT2-small-sized models (roughly 100M–124M parameters), with each digit spaced so it becomes its own token. Multiplication uses 300k samples and addition uses 120k.
- **Baselines:** GPT-4, prompted to multiply directly, scores 0.05 on 4×4-digit and 0.00 on 5×5-digit problems. A model trained on the plain "Basic" format with up to 5 digits reaches only 0.01 on 5×5.
- **Multiplication result:** with the modified setup, the model handles 15×15-digit multiplication without intermediate steps and is essentially perfect up to 12 digits.
- **Mixing dialogue and addition data:** at 100 epochs, 5-digit dialogue accuracy depends heavily on the setup.

  | Setup | 5-digit accuracy |
  |---|---|
  | Dia+Basic, absolute position embedding | 0.12 |
  | Dia+Basic, no position embedding | 0.99 |
  | Dia+Basic, random embedding | 0.86 |
  | Dia+RandomSpace, absolute position embedding | 0.82 |

- **Trade-off between fixes:** removing position embeddings can make learning unstable. Random embedding offsets this and does better on pure addition, while the no-embedding variant is slightly better in dialogue.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) — Company, Redmond, United States · 404 works · 163,863 citations
- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 422 works · 213,538 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works · [➗ Mathematics](https://md.research.elamur.ai/free/en/shelves/mathematics/) · 233 works · [➗ Statistics and Probability](https://md.research.elamur.ai/free/en/shelves/mathematics/statistics-and-probability/) · 23 works

## Cite

Cite as: Ruoqi Shen, Sébastien Bubeck, Ronen Eldan, Yin Tat Lee, Yuanzhi Li and Yi Zhang. 2023. Positional Description Matters for Transformers Arithmetic. arXiv:2311.14737.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{shen2023positional,
  title = {{Positional Description Matters for Transformers Arithmetic}},
  author = {Ruoqi Shen, Sébastien Bubeck, Ronen Eldan, Yin Tat Lee, Yuanzhi Li and Yi Zhang},
  year = {2023},
  doi = {10.48550/arxiv.2311.14737},
  eprint = {2311.14737},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2311.14737}
}
```

- [← previous No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models](https://md.research.elamur.ai/free/en/papers/kaddour-2023-no-train-no-gain-revisiting/)
- [next → Privacy Issues in Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/neel-2023-privacy-issues-large-language-models/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/12/)

## References

25 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023). [Sparks of artificial general intelligence: Early experiments with gpt-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/). arXiv preprint arXiv:2303.12712.
2. Charton, F. (2021). [Linear algebra with transformers](https://md.research.elamur.ai/free/en/papers/charton-2021-linear-algebra-transformers/). arXiv preprint arXiv:2112.01898.
3. Charton, F. (2022). [What is my math transformer doing?–three results on interpretability and generalization](https://md.research.elamur.ai/free/en/papers/charton-2022-my-math-transformer-doing-three/). arXiv preprint arXiv:2211.00170.
4. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021). [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
5. Csordás, R., Irie, K., and Schmidhuber, J. (2021). The neural data router: Adaptive control flow in transformers improves systematic generalization. arXiv preprint arXiv:2110.07732. [find](https://scholar.google.com/scholar?q=Csorda%CC%81s%2C+R%2E%2C+Irie%2C+K%2E%2C+and+Schmidhuber%2C+J%2E+%282021%29%2E+The+neural+data+router%3A+Adaptive+control+flow+in+transformers+improves+systematic+generalization%2E+arXiv+preprint+arXiv%3A2110%2E07732%2E)
6. Dziri, N., Lu, X., Sclar, M., Li, X. L., Jian, L., Lin, B. Y., West, P., Bhagavatula, C., Bras, R. L., Hwang, J. D., et al. (2023). [Faith and fate: Limits of transformers on compositionality](https://md.research.elamur.ai/free/en/papers/dziri-2023-faith-fate-limits-transformers-compositionality/). arXiv preprint arXiv:2305.18654.
7. Hanna, M., Liu, O., and Variengien, A. (2023). [How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model](https://md.research.elamur.ai/free/en/papers/hanna-2023-gpt-2-compute-greater-than/). arXiv preprint arXiv:2305.00586.
8. Jelassi, S., d’Ascoli, S., Domingo-Enrich, C., Wu, Y., Li, Y., and Charton, F. (2023). [Length generalization in arithmetic transformers](https://md.research.elamur.ai/free/en/papers/jelassi-2023-length-generalization-arithmetic-transformers/). arXiv preprint arXiv:2306.15400.
9. Kazemnejad, A., Padhi, I., Ramamurthy, K. N., Das, P., and Reddy, S. (2023). [The impact of positional encoding on length generalization in transformers](https://md.research.elamur.ai/free/en/papers/kazemnejad-2023-impact-positional-encoding-length-generalization/). arXiv preprint arXiv:2305.19466.
10. Kiyono, S., Kobayashi, S., Suzuki, J., and Inui, K. (2021). Shape: Shifted absolute position embedding for transformers. arXiv preprint arXiv:2109.05644. [find](https://scholar.google.com/scholar?q=Kiyono%2C+S%2E%2C+Kobayashi%2C+S%2E%2C+Suzuki%2C+J%2E%2C+and+Inui%2C+K%2E+%282021%29%2E+Shape%3A+Shifted+absolute+position+embedding+for+transformers%2E+arXiv+preprint+arXiv%3A2109%2E05644%2E)
11. Lee, N., Sreenivasan, K., Lee, J. D., Lee, K., and Papailiopoulos, D. (2023). [Teaching arithmetic to small transformers](https://md.research.elamur.ai/free/en/papers/lee-2023-teaching-arithmetic-small-transformers/). arXiv preprint arXiv:2307.03381.
12. Li, Y. and McClelland, J. L. (2022). Systematic generalization and emergent structures in transformers trained on structured tasks. arXiv preprint arXiv:2210.00400. — [arXiv:2210.00400](https://arxiv.org/abs/2210.00400) [find](https://scholar.google.com/scholar?q=Li%2C+Y%2E+and+McClelland%2C+J%2E+L%2E+%282022%29%2E+Systematic+generalization+and+emergent+structures+in+transformers+trained+on+structured+tasks%2E+arXiv+preprint+arXiv%3A2210%2E00400%2E+%E2%80%94+%5BarXiv%3A2210%2E00400%5D%28https%3A%2F%2Farxiv%2E)
13. Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. (2023). [Let’s verify step by step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/). arXiv preprint arXiv:2305.20050.
14. Nogueira, R., Jiang, Z., and Lin, J. (2021). [Investigating the limitations of transformers with simple arithmetic tasks](https://md.research.elamur.ai/free/en/papers/nogueira-2021-investigating-limitations-transformers-simple-arithmetic/). arXiv preprint arXiv:2102.13019.
15. OpenAI (2023). [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/li-2023-textbooks-all-need-ii-phi/). arXiv preprint arXiv:2309.05463.
16. Press, O., Smith, N. A., and Lewis, M. (2021). Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409. [find](https://scholar.google.com/scholar?q=Press%2C+O%2E%2C+Smith%2C+N%2E+A%2E%2C+and+Lewis%2C+M%2E+%282021%29%2E+Train+short%2C+test+long%3A+Attention+with+linear+biases+enables+input+length+extrapolation%2E+arXiv+preprint+arXiv%3A2108%2E12409%2E)
17. Qian, J., Wang, H., Li, Z., Li, S., and Yan, X. (2022). Limitations of language models in arithmetic and symbolic induction. arXiv preprint arXiv:2208.05051. [find](https://scholar.google.com/scholar?q=Qian%2C+J%2E%2C+Wang%2C+H%2E%2C+Li%2C+Z%2E%2C+Li%2C+S%2E%2C+and+Yan%2C+X%2E+%282022%29%2E+Limitations+of+language+models+in+arithmetic+and+symbolic+induction%2E+arXiv+preprint+arXiv%3A2208%2E05051%2E)
18. Ruoss, A., Delétang, G., Genewein, T., Grau-Moya, J., Csordás, R., Bennani, M., Legg, S., and Veness, J. (2023). Randomized positional encodings boost length generalization of transformers. arXiv preprint arXiv:2305.16843. [find](https://scholar.google.com/scholar?q=Ruoss%2C+A%2E%2C+Dele%CC%81tang%2C+G%2E%2C+Genewein%2C+T%2E%2C+Grau%2DMoya%2C+J%2E%2C+Csorda%CC%81s%2C+R%2E%2C+Bennani%2C+M%2E%2C+Legg%2C+S%2E%2C+and+Veness%2C+J%2E+%282023%29%2E+Randomized+positional+encodings+boost+length+generalization+of+transformers%2E+arXiv+pr)
19. Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Y. (2021). [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). arXiv preprint arXiv:2104.09864.
20. Testolin, A. (2023). Can neural networks do arithmetic? a survey on the elementary numerical skills of state-of-the-art deep learning models. arXiv preprint arXiv:2303.07735. — [arXiv:2303.07735](https://arxiv.org/abs/2303.07735) [find](https://scholar.google.com/scholar?q=Testolin%2C+A%2E+%282023%29%2E+Can+neural+networks+do+arithmetic%3F+a+survey+on+the+elementary+numerical+skills+of+state%2Dof%2Dthe%2Dart+deep+learning+models%2E+arXiv+preprint+arXiv%3A2303%2E07735%2E+%E2%80%94+%5BarXiv%3A2303%2E07735%5D%28http)
21. Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang, L., Creswell, A., Irving, G., and Higgins, I. (2022). [Solving math word problems with process-and outcome-based feedback](https://md.research.elamur.ai/free/en/papers/uesato-2022-solving-math-word-problems-process/). arXiv preprint arXiv:2211.14275.
22. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. (2022). [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in Neural Information Processing Systems, 35:24824–24837.
23. Yang, Z., Ding, M., Lv, Q., Jiang, Z., He, Z., Guo, Y., Bai, J., and Tang, J. (2023). [Gpt can solve mathematical problems without a calculator](https://md.research.elamur.ai/free/en/papers/yang-2023-gpt-solve-mathematical-problems-calculator/). arXiv preprint arXiv:2309.03241.
24. Zhang, Y., Backurs, A., Bubeck, S., Eldan, R., Gunasekar, S., and Wagner, T. (2022). Unveiling transformers with lego: a synthetic reasoning task. arXiv preprint arXiv:2206.04301. [find](https://scholar.google.com/scholar?q=Zhang%2C+Y%2E%2C+Backurs%2C+A%2E%2C+Bubeck%2C+S%2E%2C+Eldan%2C+R%2E%2C+Gunasekar%2C+S%2E%2C+and+Wagner%2C+T%2E+%282022%29%2E+Unveiling+transformers+with+lego%3A+a+synthetic+reasoning+task%2E+arXiv+preprint+arXiv%3A2206%2E04301%2E)
25. Zhou, H., Nova, A., Larochelle, H., Courville, A., Neyshabur, B., and Sedghi, H. (2022). [Teaching algorithmic reasoning via in-context learning](https://md.research.elamur.ai/free/en/papers/zhou-2022-teaching-algorithmic-reasoning-context-learning/). arXiv preprint arXiv:2211.09066.

*Updated 3 November 2026*