---
title: "Teaching Arithmetic to Small Transformers — Lee 2023"
description: "The work isolates how data design, rather than scale alone, drives emergent skills. 100% accuracy. 71 references, cited by 50 papers here."
authors: ["Nayoung Lee", "Kartik Sreenivasan", "Jason D. Lee", "Kangwook Lee", "Dimitris Papailiopoulos"]
year: 2023
venue: "arXiv:2307.03381"
source: "https://arxiv.org/abs/2307.03381"
doi: "10.48550/arxiv.2307.03381"
arxiv: "2307.03381"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lee-2023-teaching-arithmetic-small-transformers/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lee-2023-teaching-arithmetic-small-transformers/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/) / Teaching Arithmetic to Small Transformers

# Teaching Arithmetic to Small Transformers

**Authors:** Nayoung Lee, Kartik Sreenivasan, [Jason D. Lee](https://md.research.elamur.ai/free/en/authors/jason-d-lee/), [Kangwook Lee](https://md.research.elamur.ai/free/en/authors/kangwook-lee/) and Dimitris Papailiopoulos · **Year:** 2023 · **Venue:** arXiv:2307.03381 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2307.03381) · [PDF](https://arxiv.org/pdf/2307.03381) · [DOI 10.48550/arxiv.2307.03381](https://doi.org/10.48550/arxiv.2307.03381) · [arXiv 2307.03381](https://arxiv.org/abs/2307.03381) · [OpenAlex](https://openalex.org/W4383860315)

**Citations:** 7 citations (OpenAlex, 2026-10-14) · [50](https://md.research.elamur.ai/free/en/papers/lee-2023-teaching-arithmetic-small-transformers/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks which factors let a next-token predictor acquire arithmetic. It uses a controlled setting: small decoder-only transformers (NanoGPT and GPT-2, from 10.6M up to 124M parameters) trained from random initialization with character-level tokens. Tasks include addition, subtraction, multiplication, square root and sine.

The authors run extensive ablations over:
- **Data format:** plain, reversed output, and two scratchpad variants.
- **Sampling:** balancing examples by digit count and number of carries.
- **Other factors:** mixing in Shakespeare text, few-shot prompting, fine-tuning pretrained GPT-2 and GPT-3, and model scale.

**Main results.** Writing sums the conventional way is a poor teaching signal. Emitting the answer starting from the lowest digit, or spelling out intermediate steps, cuts the number of examples needed and raises accuracy, even with no pretraining. Accuracy jumps abruptly as training data grows, which the authors link to low-rank matrix completion.

**Limitations.** Detailed formats cost many more tokens. The models also fail on digit lengths they never saw, which suggests they learn a length-bound mapping rather than a true algorithm.

**Why it matters.** The work isolates how data design, rather than scale alone, drives emergent skills.

## Key points
- Four addition formats are compared: Plain, Reverse, Simplified Scratchpad and Detailed Scratchpad. Plain never hits 100% accuracy, and the sample cost of the others falls as the format becomes more detailed.
- An n-digit addition table forms a rank-2 matrix. This makes learning addition analogous to low-rank matrix completion, and NanoGPT generalizes better than standard completion solutions.
- Joint training on all five operations helps each individual task. Moving from zero-shot to 1-shot prompting gives a large accuracy boost, while additional examples add little.
- Fine-tuning pretrained GPT-2 and GPT-3 (davinci) works with few samples, but non-standard formats such as Reverse can clash with the patterns learned in pretraining. Larger scale helps but is not required.
- No length generalization was observed beyond the trained digit lengths. Code is available at github.com/lee-ny/teaching_arithmetic.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Princeton University](https://md.research.elamur.ai/free/en/universities/princeton-university/) — University, Princeton, United States · 113 works · 48,612 citations
- [University of Wisconsin–Madison](https://md.research.elamur.ai/free/en/universities/university-of-wisconsin-madison/) — University, Madison, United States · 54 works · 49,089 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

## Cite

Cite as: Nayoung Lee, Kartik Sreenivasan, Jason D. Lee, Kangwook Lee and Dimitris Papailiopoulos. 2023. Teaching Arithmetic to Small Transformers. arXiv:2307.03381.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{lee2023teaching,
  title = {{Teaching Arithmetic to Small Transformers}},
  author = {Nayoung Lee, Kartik Sreenivasan, Jason D. Lee, Kangwook Lee and Dimitris Papailiopoulos},
  year = {2023},
  doi = {10.48550/arxiv.2307.03381},
  eprint = {2307.03381},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.03381}
}
```

- [← previous SuperCLUE: A Comprehensive Chinese Large Language Model Benchmark](https://md.research.elamur.ai/free/en/papers/xu-2023-superclue-comprehensive-chinese-large/)
- [next → Parsel: Algorithmic Reasoning with Language Models by Composing Decompositions](https://md.research.elamur.ai/free/en/papers/zelikman-2023-parsel-algorithmic-reasoning-language-models/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/)

## References

71 references, 35 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anil, C., Wu, Y., Andreassen, A., Lewkowycz, A., Misra, V., Ramasesh, V., Slone, A., Gur-Ari, G., Dyer, E., and Neyshabur, B. [Exploring length generalization in large language models](https://md.research.elamur.ai/free/en/papers/anil-2022-exploring-length-generalization-large-language/). arXiv preprint arXiv:2207.04901, 2022.
2. Bowman, S. R. Can recursive neural tensor networks learn logical reasoning? arXiv preprint arXiv:1312.6192, 2013. [find](https://scholar.google.com/scholar?q=Bowman%2C+S%2E+R%2E+Can+recursive+neural+tensor+networks+learn+logical+reasoning%3F+arXiv+preprint+arXiv%3A1312%2E6192%2C+2013%2E)
3. Bowman, S. R., Potts, C., and Manning, C. D. Recursive neural networks for learning logical semantics. CoRR, abs/1406.1827, 5, 2014. [find](https://scholar.google.com/scholar?q=Bowman%2C+S%2E+R%2E%2C+Potts%2C+C%2E%2C+and+Manning%2C+C%2E+D%2E+Recursive+neural+networks+for+learning+logical+semantics%2E+CoRR%2C+abs%2F1406%2E1827%2C+5%2C+2014%2E)
4. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020.
5. Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. [Sparks of artificial general intelligence: Early experiments with gpt-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/). arXiv preprint arXiv:2303.12712, 2023.
6. Cai, J., Shin, R., and Song, D. Making neural programming architectures generalize via recursion. arXiv preprint arXiv:1704.06611, 2017. [find](https://scholar.google.com/scholar?q=Cai%2C+J%2E%2C+Shin%2C+R%2E%2C+and+Song%2C+D%2E+Making+neural+programming+architectures+generalize+via+recursion%2E+arXiv+preprint+arXiv%3A1704%2E06611%2C+2017%2E)
7. Charton, F. [Linear algebra with transformers](https://md.research.elamur.ai/free/en/papers/charton-2021-linear-algebra-transformers/). arXiv preprint arXiv:2112.01898, 2021.
8. Charton, F. What is my math transformer doing?–three results on interpretability and generalization. arXiv preprint arXiv:2211.00170, 2022. [find](https://scholar.google.com/scholar?q=Charton%2C+F%2E+What+is+my+math+transformer+doing%3F%E2%80%93three+results+on+interpretability+and+generalization%2E+arXiv+preprint+arXiv%3A2211%2E00170%2C+2022%2E)
9. Chen, X., Liu, C., and Song, D. Towards synthesizing complex programs from input-output examples. arXiv preprint arXiv:1706.01284, 2017. [find](https://scholar.google.com/scholar?q=Chen%2C+X%2E%2C+Liu%2C+C%2E%2C+and+Song%2C+D%2E+Towards+synthesizing+complex+programs+from+input%2Doutput+examples%2E+arXiv+preprint+arXiv%3A1706%2E01284%2C+2017%2E)
10. Chen, X., Lin, M., Schärli, N., and Zhou, D. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128, 2023. [find](https://scholar.google.com/scholar?q=Chen%2C+X%2E%2C+Lin%2C+M%2E%2C+Sch%C3%A4rli%2C+N%2E%2C+and+Zhou%2C+D%2E+Teaching+large+language+models+to+self%2Ddebug%2E+arXiv+preprint+arXiv%3A2304%2E05128%2C+2023%2E)
11. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv preprint arXiv:2204.02311, 2022.
12. Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. [find](https://scholar.google.com/scholar?q=Chung%2C+H%2E+W%2E%2C+Hou%2C+L%2E%2C+Longpre%2C+S%2E%2C+Zoph%2C+B%2E%2C+Tay%2C+Y%2E%2C+Fedus%2C+W%2E%2C+Li%2C+E%2E%2C+Wang%2C+X%2E%2C+Dehghani%2C+M%2E%2C+Brahma%2C+S%2E%2C+et+al%2E+Scaling+instruction%2Dfinetuned+language+models%2E+arXiv+preprint+arXiv%3A2210%2E11416%2C+202)
13. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021.
14. Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., and Kaiser, Ł. Universal transformers. arXiv preprint arXiv:1807.03819, 2018. [find](https://scholar.google.com/scholar?q=Dehghani%2C+M%2E%2C+Gouws%2C+S%2E%2C+Vinyals%2C+O%2E%2C+Uszkoreit%2C+J%2E%2C+and+Kaiser%2C+%C5%81%2E+Universal+transformers%2E+arXiv+preprint+arXiv%3A1807%2E03819%2C+2018%2E)
15. Drozdov, A., Schärli, N., Akyürek, E., Scales, N., Song, X., Chen, X., Bousquet, O., and Zhou, D. Compositional semantic parsing with large language models. arXiv preprint arXiv:2209.15003, 2022. [find](https://scholar.google.com/scholar?q=Drozdov%2C+A%2E%2C+Sch%C3%A4rli%2C+N%2E%2C+Aky%C3%BCrek%2C+E%2E%2C+Scales%2C+N%2E%2C+Song%2C+X%2E%2C+Chen%2C+X%2E%2C+Bousquet%2C+O%2E%2C+and+Zhou%2C+D%2E+Compositional+semantic+parsing+with+large+language+models%2E+arXiv+preprint+arXiv%3A2209%2E15003%2C+2022%2E)
16. Gadre, S. Y., Ilharco, G., Fang, A., Hayase, J., Smyrnis, G., Nguyen, T., Marten, R., Wortsman, M., Ghosh, D., Zhang, J., et al. [Datacomp: In search of the next generation of multimodal datasets](https://md.research.elamur.ai/free/en/papers/gadre-2023-datacomp-search-next-generation-multimodal/). arXiv preprint arXiv:2304.14108, 2023.
17. Giannou, A., Rajput, S., Sohn, J.-y., Lee, K., Lee, J. D., and Papailiopoulos, D. Looped transformers as programmable computers. arXiv preprint arXiv:2301.13196, 2023. [find](https://scholar.google.com/scholar?q=Giannou%2C+A%2E%2C+Rajput%2C+S%2E%2C+Sohn%2C+J%2E%2Dy%2E%2C+Lee%2C+K%2E%2C+Lee%2C+J%2E+D%2E%2C+and+Papailiopoulos%2C+D%2E+Looped+transformers+as+programmable+computers%2E+arXiv+preprint+arXiv%3A2301%2E13196%2C+2023%2E)
18. Hajij, M., Zamzmi, G., Ramamurthy, K. N., and Saenz, A. G. Data-centric ai requires rethinking data notion. arXiv preprint arXiv:2110.02491, 2021. — [arXiv:2110.02491](https://arxiv.org/abs/2110.02491) [find](https://scholar.google.com/scholar?q=Hajij%2C+M%2E%2C+Zamzmi%2C+G%2E%2C+Ramamurthy%2C+K%2E+N%2E%2C+and+Saenz%2C+A%2E+G%2E+Data%2Dcentric+ai+requires+rethinking+data+notion%2E+arXiv+preprint+arXiv%3A2110%2E02491%2C+2021%2E+%E2%80%94+%5BarXiv%3A2110%2E02491%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2110%2E02491)
19. Hanna, M., Liu, O., and Variengien, A. [How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model](https://md.research.elamur.ai/free/en/papers/hanna-2023-gpt-2-compute-greater-than/). arXiv preprint arXiv:2305.00586, 2023.
20. Huang, J., Gu, S. S., Hou, L., Wu, Y., Wang, X., Yu, H., and Han, J. [Large language models can self-improve](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-can-self/). arXiv preprint arXiv:2210.11610, 2022.
21. Kaiser, Ł. and Sutskever, I. Neural gpus learn algorithms. arXiv preprint arXiv:1511.08228, 2015. [find](https://scholar.google.com/scholar?q=Kaiser%2C+%C5%81%2E+and+Sutskever%2C+I%2E+Neural+gpus+learn+algorithms%2E+arXiv+preprint+arXiv%3A1511%2E08228%2C+2015%2E)
22. Karpathy, A. char-rnn. [https://github.com/karpathy/char-rnn](https://github.com/karpathy/char-rnn), 2015. [↗](https://github.com/karpathy/char-rnn)
23. Karpathy, A. Andrej karpathy’s lightweight implementation of medium-sized gpts. GitHub, 2022. URL [https://github.com/karpathy/nanoGPT](https://github.com/karpathy/nanoGPT). [↗](https://github.com/karpathy/nanoGPT)
24. Kim, J., Hong, G., Kim, K.-m., Kang, J., and Myaeng, S.-H. Have you seen that number? investigating extrapolation in question answering models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7031–7037, 2021. [find](https://scholar.google.com/scholar?q=Kim%2C+J%2E%2C+Hong%2C+G%2E%2C+Kim%2C+K%2E%2Dm%2E%2C+Kang%2C+J%2E%2C+and+Myaeng%2C+S%2E%2DH%2E+Have+you+seen+that+number%3F+investigating+extrapolation+in+question+answering+models%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Metho)
25. Király, F. J., Theran, L., and Tomioka, R. The algebraic combinatorial approach for low-rank matrix completion. J. Mach. Learn. Res., 16(1):1391–1436, 2015. [find](https://scholar.google.com/scholar?q=Kir%C3%A1ly%2C+F%2E+J%2E%2C+Theran%2C+L%2E%2C+and+Tomioka%2C+R%2E+The+algebraic+combinatorial+approach+for+low%2Drank+matrix+completion%2E+J%2E+Mach%2E+Learn%2E+Res%2E%2C+16%281%29%3A1391%E2%80%931436%2C+2015%2E)
26. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). arXiv preprint arXiv:2205.11916, 2022.
27. Lake, B. and Baroni, M. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In International conference on machine learning, pp. 2873–2882. PMLR, 2018. [find](https://scholar.google.com/scholar?q=Lake%2C+B%2E+and+Baroni%2C+M%2E+Generalization+without+systematicity%3A+On+the+compositional+skills+of+sequence%2Dto%2Dsequence+recurrent+networks%2E+In+International+conference+on+machine+learning%2C+pp%2E+2873%E2%80%932882%2E+PM)
28. Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let’s verify step by step. arXiv preprint arXiv:2305.[20050, 2023](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/).
29. Ling, W., Yogatama, D., Dyer, C., and Blunsom, P. Program induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv preprint arXiv:1705.04146, 2017. [find](https://scholar.google.com/scholar?q=Ling%2C+W%2E%2C+Yogatama%2C+D%2E%2C+Dyer%2C+C%2E%2C+and+Blunsom%2C+P%2E+Program+induction+by+rationale+generation%3A+Learning+to+solve+and+explain+algebraic+word+problems%2E+arXiv+preprint+arXiv%3A1705%2E04146%2C+2017%2E)
30. Liu, B., Ash, J. T., Goel, S., Krishnamurthy, A., and Zhang, C. Exposing attention glitches with flip-flop language modeling. arXiv preprint arXiv:2306.00946, 2023. [find](https://scholar.google.com/scholar?q=Liu%2C+B%2E%2C+Ash%2C+J%2E+T%2E%2C+Goel%2C+S%2E%2C+Krishnamurthy%2C+A%2E%2C+and+Zhang%2C+C%2E+Exposing+attention+glitches+with+flip%2Dflop+language+modeling%2E+arXiv+preprint+arXiv%3A2306%2E00946%2C+2023%2E)
31. Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. [Rethinking the role of demonstrations: What makes in-context learning work?](https://md.research.elamur.ai/free/en/papers/min-2022-rethinking-role-demonstrations-what-makes/) arXiv preprint arXiv:2202.12837, 2022.
32. MosaicML. Introducing mpt-7b: A new standard for open source, commercially usable llms, 2023. URL www.mosaicml.com/blog/mpt-7b. Accessed: 2023-05-05. [find](https://scholar.google.com/scholar?q=MosaicML%2E+Introducing+mpt%2D7b%3A+A+new+standard+for+open+source%2C+commercially+usable+llms%2C+2023%2E+URL+www%2Emosaicml%2Ecom%2Fblog%2Fmpt%2D7b%2E+Accessed%3A+2023%2D05%2D05%2E)
33. Motamedi, M., Sakharnykh, N., and Kaldewey, T. A data-centric approach for training deep neural networks with less data. arXiv preprint arXiv:2110.03613, 2021. — [arXiv:2110.03613](https://arxiv.org/abs/2110.03613) [find](https://scholar.google.com/scholar?q=Motamedi%2C+M%2E%2C+Sakharnykh%2C+N%2E%2C+and+Kaldewey%2C+T%2E+A+data%2Dcentric+approach+for+training+deep+neural+networks+with+less+data%2E+arXiv+preprint+arXiv%3A2110%2E03613%2C+2021%2E+%E2%80%94+%5BarXiv%3A2110%2E03613%5D%28https%3A%2F%2Farxiv%2Eorg%2Fa)
34. Nogueira, R., Jiang, Z., and Lin, J. [Investigating the limitations of transformers with simple arithmetic tasks](https://md.research.elamur.ai/free/en/papers/nogueira-2021-investigating-limitations-transformers-simple-arithmetic/). arXiv preprint arXiv:2102.13019, 2021.
35. Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., et al. [Show your work: Scratchpads for intermediate computation with language models](https://md.research.elamur.ai/free/en/papers/al-2021-show-your-work-scratchpads-intermediate/). arXiv preprint arXiv:2112.00114, 2021.
36. Ontanón, S., Ainslie, J., Cvicek, V., and Fisher, Z. Making transformers solve compositional tasks. arXiv preprint arXiv:2108.04378, 2021. [find](https://scholar.google.com/scholar?q=Ontan%C3%B3n%2C+S%2E%2C+Ainslie%2C+J%2E%2C+Cvicek%2C+V%2E%2C+and+Fisher%2C+Z%2E+Making+transformers+solve+compositional+tasks%2E+arXiv+preprint+arXiv%3A2108%2E04378%2C+2021%2E)
37. Pérez, J., Barceló, P., and Marinkovic, J. Attention is turing complete. The Journal of Machine Learning Research, 22(1):3463–3497, 2021. [find](https://scholar.google.com/scholar?q=P%C3%A9rez%2C+J%2E%2C+Barcel%C3%B3%2C+P%2E%2C+and+Marinkovic%2C+J%2E+Attention+is+turing+complete%2E+The+Journal+of+Machine+Learning+Research%2C+22%281%29%3A3463%E2%80%933497%2C+2021%2E)
38. Peterson, J., Meylan, S., and Bourgin, D. Open clone of openai’s unreleased webtext dataset scraper. GitHub, 2019. URL [https://github.com/jcpeterson/openwebtext](https://github.com/jcpeterson/openwebtext). [↗](https://github.com/jcpeterson/openwebtext)
39. Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Heek, J., Xiao, K., Agrawal, S., and Dean, J. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5, 2023. [find](https://scholar.google.com/scholar?q=Pope%2C+R%2E%2C+Douglas%2C+S%2E%2C+Chowdhery%2C+A%2E%2C+Devlin%2C+J%2E%2C+Bradbury%2C+J%2E%2C+Heek%2C+J%2E%2C+Xiao%2C+K%2E%2C+Agrawal%2C+S%2E%2C+and+Dean%2C+J%2E+Efficiently+scaling+transformer+inference%2E+Proceedings+of+Machine+Learning+and+Systems%2C+5%2C)
40. Qian, J., Wang, H., Li, Z., Li, S., and Yan, X. Limitations of language models in arithmetic and symbolic induction. arXiv preprint arXiv:2208.05051, 2022. [find](https://scholar.google.com/scholar?q=Qian%2C+J%2E%2C+Wang%2C+H%2E%2C+Li%2C+Z%2E%2C+Li%2C+S%2E%2C+and+Yan%2C+X%2E+Limitations+of+language+models+in+arithmetic+and+symbolic+induction%2E+arXiv+preprint+arXiv%3A2208%2E05051%2C+2022%2E)
41. Radford, A. and Narasimhan, K. Improving language understanding by generative pre-training. 2018. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E+and+Narasimhan%2C+K%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+2018%2E)
42. Rajani, N. F., McCann, B., Xiong, C., and Socher, R. [Explain yourself! leveraging language models for commonsense reasoning](https://md.research.elamur.ai/free/en/papers/rajani-2019-explain-yourself-leveraging-language-models/). arXiv preprint arXiv:1906.02361, 2019.
43. Razeghi, Y., Logan IV, R. L., Gardner, M., and Singh, S. Impact of pretraining term frequencies on few-shot reasoning. arXiv preprint arXiv:2202.07206, 2022. — [arXiv:2202.07206](https://arxiv.org/abs/2202.07206) [find](https://scholar.google.com/scholar?q=Razeghi%2C+Y%2E%2C+Logan+IV%2C+R%2E+L%2E%2C+Gardner%2C+M%2E%2C+and+Singh%2C+S%2E+Impact+of+pretraining+term+frequencies+on+few%2Dshot+reasoning%2E+arXiv+preprint+arXiv%3A2202%2E07206%2C+2022%2E+%E2%80%94+%5BarXiv%3A2202%2E07206%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs)
44. Recht, B. A simpler approach to matrix completion. Journal of Machine Learning Research, 12(12), 2011. [find](https://scholar.google.com/scholar?q=Recht%2C+B%2E+A+simpler+approach+to+matrix+completion%2E+Journal+of+Machine+Learning+Research%2C+12%2812%29%2C+2011%2E)
45. Reed, S. and De Freitas, N. Neural programmer-interpreters. arXiv preprint arXiv:1511.06279, 2015. [find](https://scholar.google.com/scholar?q=Reed%2C+S%2E+and+De+Freitas%2C+N%2E+Neural+programmer%2Dinterpreters%2E+arXiv+preprint+arXiv%3A1511%2E06279%2C+2015%2E)
46. Roy, S. and Roth, D. [Solving general arithmetic word problems](https://md.research.elamur.ai/free/en/papers/roy-2015-solving-general-arithmetic-word-problems/). arXiv preprint arXiv:1608.01413, 2016.
47. Shaw, P., Uszkoreit, J., and Vaswani, A. [Self-attention with relative position representations](https://md.research.elamur.ai/free/en/papers/shaw-2018-self-attention-relative-position-representations/). arXiv preprint arXiv:1803.02155, 2018.
48. Shi, F., Suzgun, M., Freitag, M., Wang, X., Srivats, S., Vosoughi, S., Chung, H. W., Tay, Y., Ruder, S., Zhou, D., et al. [Language models are multilingual chain-of-thought reasoners](https://md.research.elamur.ai/free/en/papers/shi-2022-language-models-multilingual-chain-thought/). arXiv preprint arXiv:2210.03057, 2022.
49. Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. [Beyond the imitation game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/). arXiv preprint arXiv:2206.04615, 2022.
50. Sun, Y., Dong, L., Patra, B., Ma, S., Huang, S., Benhaim, A., Chaudhary, V., Song, X., and Wei, F. [A length-extrapolatable transformer](https://md.research.elamur.ai/free/en/papers/sun-2023-length-extrapolatable-transformer/). arXiv preprint arXiv:2212.10554, 2022.
51. Sutskever, I., Vinyals, O., and Le, Q. V. [Sequence to sequence learning with neural networks](https://md.research.elamur.ai/free/en/papers/sutskever-2014-sequence-sequence-learning-neural-networks/). Advances in neural information processing systems, 27, 2014.
52. Talmor, A., Tafjord, O., Clark, P., Goldberg, Y., and Berant, J. [Leap-of-thought: Teaching pre-trained models to systematically reason over implicit knowledge](https://md.research.elamur.ai/free/en/papers/talmor-2020-leap-thought-teaching-pre-trained/). Advances in Neural Information Processing Systems, 33:20227–20237, 2020.
53. Tay, Y., Wei, J., Chung, H. W., Tran, V. Q., So, D. R., Shakeri, S., Garcia, X., Zheng, H. S., Rao, J., Chowdhery, A., et al. [Transcending scaling laws with 0.1% extra compute](https://md.research.elamur.ai/free/en/papers/tay-2022-transcending-scaling-laws-0-1/). arXiv preprint arXiv:2210.11399, 2022.
54. Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. [Lamda: Language models for dialog applications](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/). arXiv preprint arXiv:2201.08239, 2022.
55. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/), 2023.
56. Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang, L., Creswell, A., Irving, G., and Higgins, I. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.[14275, 2022](https://md.research.elamur.ai/free/en/papers/uesato-2022-solving-math-word-problems-process/).
57. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30, 2017.
58. Wallace, E., Wang, Y., Li, S., Singh, S., and Gardner, M. [Do nlp models know numbers? probing numeracy in embeddings](https://md.research.elamur.ai/free/en/papers/wallace-2019-do-nlp-models-know-numbers/). arXiv preprint arXiv:1909.07940, 2019.
59. Wang, C., Zheng, B., Niu, Y., and Zhang, Y. Exploring generalization ability of pretrained language models on arithmetic and logical reasoning. In Natural Language Processing and Chinese Computing: 10th CCF International Conference, NLPCC 2021, Qingdao, China, October 13–17, 2021, Proceedings, Part… [find](https://scholar.google.com/scholar?q=Wang%2C+C%2E%2C+Zheng%2C+B%2E%2C+Niu%2C+Y%2E%2C+and+Zhang%2C+Y%2E+Exploring+generalization+ability+of+pretrained+language+models+on+arithmetic+and+logical+reasoning%2E+In+Natural+Language+Processing+and+Chinese+Computing%3A+10)
60. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., and Zhou, D. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). arXiv preprint arXiv:2203.11171, 2022.
61. Webb, T., Holyoak, K. J., and Lu, H. Emergent analogical reasoning in large language models. arXiv preprint arXiv:2212.09196, 2022. [find](https://scholar.google.com/scholar?q=Webb%2C+T%2E%2C+Holyoak%2C+K%2E+J%2E%2C+and+Lu%2C+H%2E+Emergent+analogical+reasoning+in+large+language+models%2E+arXiv+preprint+arXiv%3A2212%2E09196%2C+2022%2E)
62. Wei, C., Chen, Y., and Ma, T. Statistically meaningful approximation: a case study on approximating turing machines with transformers. Advances in Neural Information Processing Systems, 35: 12071–12083, 2022a. [find](https://scholar.google.com/scholar?q=Wei%2C+C%2E%2C+Chen%2C+Y%2E%2C+and+Ma%2C+T%2E+Statistically+meaningful+approximation%3A+a+case+study+on+approximating+turing+machines+with+transformers%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A+12071%E2%80%931208)
63. Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/). arXiv preprint arXiv:2206.07682, 2022b.
64. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.[11903, 2022c](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/).
65. Yuan, Z., Yuan, H., Tan, C., Wang, W., and Huang, S. [How well do large language models perform in arithmetic tasks?](https://md.research.elamur.ai/free/en/papers/yuan-2023-well-large-language-models-perform/) arXiv preprint arXiv:2304.02015, 2023.
66. Yun, C., Bhojanapalli, S., Rawat, A. S., Reddi, S. J., and Kumar, S. [Are transformers universal approximators of sequence-to-sequence functions?](https://md.research.elamur.ai/free/en/papers/yun-2019-transformers-universal-approximators-sequence-sequence/) arXiv preprint arXiv:1912.10077, 2019.
67. Zaremba, W. and Sutskever, I. Learning to execute. arXiv preprint arXiv:1410.4615, 2014. [find](https://scholar.google.com/scholar?q=Zaremba%2C+W%2E+and+Sutskever%2C+I%2E+Learning+to+execute%2E+arXiv+preprint+arXiv%3A1410%2E4615%2C+2014%2E)
68. Zaremba, W., Kurach, K., and Fergus, R. Learning to discover efficient mathematical identities. Advances in Neural Information Processing Systems, 27, 2014. [find](https://scholar.google.com/scholar?q=Zaremba%2C+W%2E%2C+Kurach%2C+K%2E%2C+and+Fergus%2C+R%2E+Learning+to+discover+efficient+mathematical+identities%2E+Advances+in+Neural+Information+Processing+Systems%2C+27%2C+2014%2E)
69. Zelikman, E., Mu, J., Goodman, N. D., and Wu, Y. T. Star: Self-taught reasoner bootstrapping reasoning with reasoning. 2022. [find](https://scholar.google.com/scholar?q=Zelikman%2C+E%2E%2C+Mu%2C+J%2E%2C+Goodman%2C+N%2E+D%2E%2C+and+Wu%2C+Y%2E+T%2E+Star%3A+Self%2Dtaught+reasoner+bootstrapping+reasoning+with+reasoning%2E+2022%2E)
70. Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Bousquet, O., Le, Q., and Chi, E. [Least-to-most prompting enables complex reasoning in large language models](https://md.research.elamur.ai/free/en/papers/al-2022-least-most-prompting-enables-complex/). arXiv preprint arXiv:2205.10625, 2022a.
71. Zhou, H., Nova, A., Larochelle, H., Courville, A., Neyshabur, B., and Sedghi, H. [Teaching algorithmic reasoning via in-context learning](https://md.research.elamur.ai/free/en/papers/zhou-2022-teaching-algorithmic-reasoning-context-learning/). arXiv preprint arXiv:2211.09066, 2022b.

*Updated 2 November 2026*