---
title: "Evaluating Transformer Language Models on Arithmetic"
description: "GPT-3 does well on two- and three-digit arithmetic in few-shot mode but performs poorly on four- and… 20 references, cited by 12 papers here."
authors: ["Matteo Muffo", "Aldo Cocco", "Enrico Bertino"]
year: 2022
venue: "arXiv:2304.10977 · Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 291-297"
source: "https://arxiv.org/abs/2304.10977"
doi: "10.48550/arxiv.2304.10977"
arxiv: "2304.10977"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/muffo-2023-evaluating-transformer-language-models-arithmetic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/muffo-2023-evaluating-transformer-language-models-arithmetic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/5/) / Evaluating Transformer Language Models on Arithmetic Operations Using Number Decomposition

# Evaluating Transformer Language Models on Arithmetic Operations Using Number Decomposition

**Authors:** Matteo Muffo, Aldo Cocco and Enrico Bertino · **Year:** 2022 · **Venue:** [arXiv:2304.10977 · Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 291-297](https://md.research.elamur.ai/free/en/venues/lrec/) ([LREC 2022](https://md.research.elamur.ai/free/en/venues/lrec/2022/)) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2304.10977) · [PDF](https://arxiv.org/pdf/2304.10977) · [DOI 10.48550/arxiv.2304.10977](https://doi.org/10.48550/arxiv.2304.10977) · [arXiv 2304.10977](https://arxiv.org/abs/2304.10977) · [OpenAlex](https://openalex.org/W4366835609)

**Citations:** 7 citations (OpenAlex, 2026-10-14) · [12](https://md.research.elamur.ai/free/en/papers/muffo-2023-evaluating-transformer-language-models-arithmetic/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
GPT-3 does well on two- and three-digit arithmetic in few-shot mode but performs poorly on four- and five-digit operands. This suggests it has not learned a procedure that generalizes. The authors ask whether a Transformer LM can learn arithmetic on unseen numbers if it is taught to compute by place value, the way pupils handle units, tens and hundreds separately.

They fine-tune GPT-2 Small on synthetic natural-language traces. In each trace, both operands are rewritten as place-value decompositions, combined position by position, and reassembled into the final number. They call these models Calculon. Two controls are trained on the same number pairs: a baseline that maps the question straight to the answer, and a "spaced" variant that separates the digits so the BPE tokenizer handles them one at a time. All models are evaluated on the GPT-3 arithmetic test sets.

On five-digit addition, Calculon improves accuracy over GPT-3 by 63%, while the baseline scores 0%. Adding decomposition examples to GPT-3's prompt does not help. The authors conclude that addition and subtraction are learnable this way, but multiplication needs stronger reasoning. The work shows that the format of intermediate steps can matter more than model scale.

## Key points
- **Tasks:** addition and subtraction of 2–5-digit numbers, and multiplication of 2-digit numbers. Test sets are the public GPT-3 arithmetic sets from Brown et al. (2020).
- **Training data:** 12,000 random pairs per operation for addition and subtraction (3,000 per digit length) and 3,000 pairs for multiplication. This yields 9 training sets (3 operations × 3 approaches). No training pair appears in the test set for the same operation.
- **Setup:** GPT-2 Small (~117M parameters), fine-tuned for 25 epochs with learning rate 10⁻⁴ and batch size 32, using Huggingface Transformers.
- **Difference from Nogueira et al. (2021):** that T5-based work gives the model preprocessed numbers as input. Here, inputs are raw numbers and the model must generate the decomposition itself.
- **Results:**
  - Calculon gains 63% in accuracy over GPT-3 on five-digit addition.
  - The non-decomposed baseline reaches 0% on the same task.
  - GPT-3 (175B) prompted with 4 decomposition examples does worse than in its original evaluation.
  - Code and data are released on GitHub (mmuffo94).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [➗ Mathematics](https://md.research.elamur.ai/free/en/shelves/mathematics/) · 177 works · [➗ Statistics and Probability](https://md.research.elamur.ai/free/en/shelves/mathematics/statistics-and-probability/) · 16 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

## Cite

Cite as: Matteo Muffo, Aldo Cocco and Enrico Bertino. 2022. Evaluating Transformer Language Models on Arithmetic Operations Using Number Decomposition. arXiv:2304.10977 · Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 291-297.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{muffo2022evaluating,
  title = {{Evaluating Transformer Language Models on Arithmetic Operations Using Number Decomposition}},
  author = {Matteo Muffo, Aldo Cocco and Enrico Bertino},
  year = {2022},
  doi = {10.48550/arxiv.2304.10977},
  eprint = {2304.10977},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2304.10977}
}
```

- [← previous A Survey on Automated Sarcasm Detection on Twitter](https://md.research.elamur.ai/free/en/papers/moores-2022-survey-automated-sarcasm-detection-twitter/)
- [next → Offensive Language Detection in Under-resourced Algerian Dialectal Arabic Language](https://md.research.elamur.ai/free/en/papers/boucherit-2022-offensive-language-detection-under-resourced/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/5/)

## References

20 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alammar, J. (2021). Ecco: An open source library for the explainability of transformer language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations. Associ… [find](https://scholar.google.com/scholar?q=Alammar%2C+J%2E+%282021%29%2E+Ecco%3A+An+open+source+library+for+the+explainability+of+transformer+language+models%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+t)
2. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Ch… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
3. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/).
4. Geva, M., Gupta, A., and Berant, J. (2020). [Injecting numerical reasoning skills into language models](https://md.research.elamur.ai/free/en/papers/geva-2020-injecting-numerical-reasoning-skills-into/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 946– 958, Online, July. Association for Computational Linguistics.
5. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. (2021). [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/).
6. Kingma, D. P. and Ba, J. (2017). Adam: A method for stochastic optimization. [find](https://scholar.google.com/scholar?q=Kingma%2C+D%2E+P%2E+and+Ba%2C+J%2E+%282017%29%2E+Adam%3A+A+method+for+stochastic+optimization%2E)
7. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. (2019). [Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/).
8. Naik, A., Ravichander, A., Rose, C., and Hovy, E. (2019). Exploring numeracy in word embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3374–3380, Florence, Italy, July. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Naik%2C+A%2E%2C+Ravichander%2C+A%2E%2C+Rose%2C+C%2E%2C+and+Hovy%2C+E%2E+%282019%29%2E+Exploring+numeracy+in+word+embeddings%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+3374%E2%80%93)
9. Nogueira, R., Jiang, Z., and Lin, J. (2021). Investigating the limitations of transformers with simple arithmetic tasks. [find](https://scholar.google.com/scholar?q=Nogueira%2C+R%2E%2C+Jiang%2C+Z%2E%2C+and+Lin%2C+J%2E+%282021%29%2E+Investigating+the+limitations+of+transformers+with+simple+arithmetic+tasks%2E)
10. Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018). [Deep contextualized word representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/).
11. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Wu%2C+J%2E%2C+Child%2C+R%2E%2C+Luan%2C+D%2E%2C+Amodei%2C+D%2E%2C+and+Sutskever%2C+I%2E+%282019%29%2E+Language+models+are+unsupervised+multitask+learners%2E)
12. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. (2020). [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/).
13. Saxton, D., Grefenstette, E., Hill, F., and Kohli, P. (2019). [Analysing mathematical reasoning abilities of neural models](https://md.research.elamur.ai/free/en/papers/saxton-2019-analysing-mathematical-reasoning-abilities-neural/).
14. Sennrich, R., Haddow, B., and Birch, A. (2016). [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany, August. Association for Computational Lin…
15. Shrikumar, A., Greenside, P., and Kundaje, A. (2017). [Learning important features through propagating activation differences](https://md.research.elamur.ai/free/en/papers/shrikumar-2017-learning-important-features-through-propagating/). CoRR, abs/1704.02685.
16. Thawani, A., Pujara, J., Szekely, P. A., and Ilievski, F. (2021). Representing numbers in nlp: a survey and a vision. [find](https://scholar.google.com/scholar?q=Thawani%2C+A%2E%2C+Pujara%2C+J%2E%2C+Szekely%2C+P%2E+A%2E%2C+and+Ilievski%2C+F%2E+%282021%29%2E+Representing+numbers+in+nlp%3A+a+survey+and+a+vision%2E)
17. Wallace, E., Wang, Y., Li, S., Singh, S., and Gardner, M. (2019). [Do NLP models know numbers? probing numeracy in embeddings](https://md.research.elamur.ai/free/en/papers/wallace-2019-do-nlp-models-know-numbers/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJC…
18. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T. L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. M. (2020). [Huggingface’s transformers: Sta…](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
19. Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., Klingner, J., Shah, A., Johnson, M., Liu, X., Łukasz Kaiser, Gouws, S., Kato, Y., Kudo, T., Kazawa, H., Stevens, K., Kurian, G., Patil, N., Wang, W., Young, C., Smith, J., Riesa, J., Rud… [find](https://scholar.google.com/scholar?q=Wu%2C+Y%2E%2C+Schuster%2C+M%2E%2C+Chen%2C+Z%2E%2C+Le%2C+Q%2E+V%2E%2C+Norouzi%2C+M%2E%2C+Macherey%2C+W%2E%2C+Krikun%2C+M%2E%2C+Cao%2C+Y%2E%2C+Gao%2C+Q%2E%2C+Macherey%2C+K%2E%2C+Klingner%2C+J%2E%2C+Shah%2C+A%2E%2C+Johnson%2C+M%2E%2C+Liu%2C+X%2E%2C+%C5%81ukasz+Kaiser%2C+Gouws%2C+S%2E%2C+Kato%2C+Y%2E%2C+Kudo)
20. Zhang, X., Ramachandran, D., Tenney, I., Elazar, Y., and Roth, D. (2020). Do language embeddings capture scales? In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4889– 4896, Online, November. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zhang%2C+X%2E%2C+Ramachandran%2C+D%2E%2C+Tenney%2C+I%2E%2C+Elazar%2C+Y%2E%2C+and+Roth%2C+D%2E+%282020%29%2E+Do+language+embeddings+capture+scales%3F+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2020%2C+pages+4889%E2%80%93+4)

*Updated 31 October 2026*