---
title: "Arithmetic with Language Models: from Memorization to"
description: "The paper asks whether a language model trained on next-token prediction can genuinely compute… 2 references, cited by 6 papers here."
authors: ["Davide Maltoni", "Matteo Ferrara"]
year: 2023
venue: "arXiv:2308.01154 · \\Neural Networks, vol. 179, 2024"
source: "https://arxiv.org/abs/2308.01154"
doi: "10.1016/j.neunet.2024.106550"
arxiv: "2308.01154"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/maltoni-2023-arithmetic-language-models-memorization-computation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/maltoni-2023-arithmetic-language-models-memorization-computation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/14/) / Arithmetic with Language Models: from Memorization to Computation

# Arithmetic with Language Models: from Memorization to Computation

**Authors:** Davide Maltoni and Matteo Ferrara · **Year:** 2023 · **Venue:** [arXiv:2308.01154 · \\Neural Networks, vol. 179, 2024](https://md.research.elamur.ai/free/en/venues/neural-networks/) ([Neural Networks 2023](https://md.research.elamur.ai/free/en/venues/neural-networks/2023/)) · **Publisher:** [Elsevier BV](https://md.research.elamur.ai/free/en/publishers/elsevier-bv/) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://doi.org/10.1016/j.neunet.2024.106550) · [PDF](https://arxiv.org/pdf/2308.01154) · [DOI 10.1016/j.neunet.2024.106550](https://doi.org/10.1016/j.neunet.2024.106550) · [arXiv 2308.01154](https://arxiv.org/abs/2308.01154) · [OpenAlex](https://openalex.org/W4400723242)

**Citations:** 7 citations (OpenAlex, 2026-10-14) · [6](https://md.research.elamur.ai/free/en/papers/maltoni-2023-arithmetic-language-models-memorization-computation/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks whether a language model trained on next-token prediction can genuinely compute arithmetic results, rather than recalling or smoothly interpolating seen examples. The authors chose binary addition and multiplication as a controlled testbed. These tasks use a tiny vocabulary, and a single flipped input bit can change many output bits, so nearest-neighbour style interpolation fails on new inputs. They trained small, non-pretrained Transformers from scratch on 7-bit operands: the original encoder-decoder model and the decoder-only nanoGPT. Both models learned the tasks. The authors then removed chosen input regions from training to test extrapolation, analysed correlations in internal embeddings, and applied amnesic probing. The evidence points to an Encoding-Regression-Decoding (ERD) mechanism. Input tokens are first mapped into an internal numeric "value" representation, the computation happens as a regression in that space, and the result is decoded back into output tokens. The work helps explain how LMs might reach arithmetic generalization beyond memorization.

## Key points
- **Task format:** operands range over 0–127 and are written as 15-token prompts (two 7-bit strings plus an operator). The vocabulary has about 5 tokens, and outputs are 8 bits for addition and 14 bits for multiplication. Both inputs and outputs are written least-significant bit first.
- **Bit order:** reverse (little-endian) output order made training much faster and more stable than plain order. Input order made no difference.
- **nanoGPT setup and accuracy:** the model has 298K parameters, d_model 64, 8 heads and 6 layers. On the VS_t held-out subset it reached 100% for addition and 99.9% for multiplication. On VS_v it dropped to 82.0% and 80.6%.
- **Amnesic probing setup:** a linear regressor recovered the sum from decoder layer-3 embeddings with an rmse of 0.28.
- **Amnesic probing result:** removing that value information cut addition sequence accuracy from 100% to 0.13%. Removing the same amount of information along random directions left accuracy at 100%.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Bologna](https://md.research.elamur.ai/free/en/universities/university-of-bologna/) — University, Bologna, Italy · 22 works · 2,122 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 4,145 works

## Cite

Cite as: Davide Maltoni and Matteo Ferrara. 2023. Arithmetic with Language Models: from Memorization to Computation. arXiv:2308.01154 · \\Neural Networks, vol. 179, 2024.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{maltoni2023arithmetic,
  title = {{Arithmetic with Language Models: from Memorization to Computation}},
  author = {Davide Maltoni and Matteo Ferrara},
  year = {2023},
  doi = {10.1016/j.neunet.2024.106550},
  eprint = {2308.01154},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1016/j.neunet.2024.106550}
}
```

- [← previous Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression](https://md.research.elamur.ai/free/en/papers/fu-2023-transformers-learn-achieve-second-order/)
- [next → Automated multilingual detection of Pro-Kremlin propaganda in newspapers and Telegram posts](https://md.research.elamur.ai/free/en/papers/solopova-2023-automated-multilingual-detection-pro-kremlin/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/14/)

## References

2 references, 2 of them with a card of ours. ↗ leads to the work.

1. Belinkov Y., Probing Classifiers: Promises, Shortcomings, and Advances, Computational Linguistics, vol. 48, pp. 207-219, 2022. [Bubeck S., Chandrasekaran V., Eldan R., Gehrke J., Horvitz E., Kamar E., Lee P., Lee Y., Li Y., Lundberg S., Nori H., Palangi H., Ribeiro M., Zhang Y., Sparks of Artificial…](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/)
2. Räuker T., Ho A., Casper S., Hadfield-Menell D., Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks, arXiv: 2207.[13243, 2023](https://md.research.elamur.ai/free/en/papers/yuan-2023-well-large-language-models-perform/). Ravfogel S., Elazar Y., Gonen H., Twiton M., Goldberg Y., Null It Out: Guarding Protected Attributes by Iterative Nullspace Project…

*Updated 3 November 2026*