---
title: "Transformers Can Achieve Length Generalization But Not Robustly"
description: "They test many pairings of position encodings (APE, T5, Alibi, KerpleLog, FIRE, RoPE, NoPE… 52 references, cited by 25 papers here."
authors: ["Yongchao Zhou", "Uri Alon", "Xinyun Chen", "Xuezhi Wang", "Rishabh Agarwal", "Denny Zhou"]
year: 2024
venue: "arXiv:2402.09371"
source: "https://arxiv.org/abs/2402.09371"
doi: "10.48550/arxiv.2402.09371"
arxiv: "2402.09371"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhou-2024-transformers-achieve-length-generalization-but/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhou-2024-transformers-achieve-length-generalization-but/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/10/) / Transformers Can Achieve Length Generalization But Not Robustly

# Transformers Can Achieve Length Generalization But Not Robustly

**Authors:** [Yongchao Zhou](https://md.research.elamur.ai/free/en/authors/yongchao-zhou/), [Uri Alon](https://md.research.elamur.ai/free/en/authors/uri-alon/), [Xinyun Chen](https://md.research.elamur.ai/free/en/authors/xinyun-chen/), [Xuezhi Wang](https://md.research.elamur.ai/free/en/authors/xuezhi-wang/), [Rishabh Agarwal](https://md.research.elamur.ai/free/en/authors/rishabh-agarwal/) and [Denny Zhou](https://md.research.elamur.ai/free/en/authors/denny-zhou/) · **Year:** 2024 · **Venue:** arXiv:2402.09371 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2402.09371) · [PDF](https://arxiv.org/pdf/2402.09371) · [DOI 10.48550/arxiv.2402.09371](https://doi.org/10.48550/arxiv.2402.09371) · [arXiv 2402.09371](https://arxiv.org/abs/2402.09371) · [OpenAlex](https://openalex.org/W4391871437)

**Citations:** 4 citations (OpenAlex, 2026-10-14) · [25](https://md.research.elamur.ai/free/en/papers/zhou-2024-transformers-achieve-length-generalization-but/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Transformers usually fail on inputs longer than those seen in training, which raises doubts about whether they learn real algorithms. Zhou et al. (Google DeepMind, University of Toronto) study this length generalization problem with a controlled synthetic task: adding two decimal integers, using models trained from scratch. They test many pairings of position encodings (APE, T5, Alibi, KerpleLog, FIRE, RoPE, NoPE, Randomized PE) with data formats (reversed digit order, index hints, random space augmentation). The best pairing combines FIRE with randomized positions, writes digits least-significant first, and tags each digit with an index hint. The best model was trained on additions of at most 40 digits. It still solves 100-digit sums with over 98% exact-match accuracy, a 2.5× extrapolation that the authors consider the best result so far for this kind of model. The gain is unstable, though. Results vary sharply between random seeds, depending on weight initialization and the order of training data. The work suggests the obstacle is not the architecture itself, but dependable extrapolation remains an open problem.

## Key points
- The final recipe is FIRE position encoding, Randomized PE (Ruoss et al., 2023), reversed format, and index hints (Zhou et al., 2023). For example, 42+39=81 is written with letter markers like `a4b2+a3b9=a8b1`.
- The length extension ratio is 2.5× (40 to 100 digits, best of 10 trials). Earlier work reached 1.0× (Lee et al.), 1.125× (Kazemnejad et al.), 1.1× (Shen et al.) and 1.5× (Zhou et al., 2023).
- How much a data-formatting trick helps depends mainly on which position encoding is used. A well-designed PE such as FIRE beats NoPE, which contradicts Kazemnejad et al.'s earlier finding.
- Bigger models are not necessarily better. A 5M-parameter model reached about 80% on 100-digit addition and outperformed a 268M model, while a 2M model degraded on long inputs.
- Regularization reduces variance only slightly. Weight decay of 0.1–0.3 helps a little, and dropout of 0.2 seriously hurts extrapolation.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Toronto](https://md.research.elamur.ai/free/en/universities/university-of-toronto/) — University, Toronto, Canada · 121 works · 179,944 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works

## Cite

Cite as: Yongchao Zhou, Uri Alon, Xinyun Chen, Xuezhi Wang, Rishabh Agarwal and Denny Zhou. 2024. Transformers Can Achieve Length Generalization But Not Robustly. arXiv:2402.09371.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhou2024transformers,
  title = {{Transformers Can Achieve Length Generalization But Not Robustly}},
  author = {Yongchao Zhou, Uri Alon, Xinyun Chen, Xuezhi Wang, Rishabh Agarwal and Denny Zhou},
  year = {2024},
  doi = {10.48550/arxiv.2402.09371},
  eprint = {2402.09371},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.09371}
}
```

- [← previous TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generation](https://md.research.elamur.ai/free/en/papers/cook-2024-ticking-all-boxes-generated-checklists/)
- [next → Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models](https://md.research.elamur.ai/free/en/papers/bick-2024-transformers-ssms-distilling-quadratic-knowledge/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/10/)

## References

52 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. E. Abbe, S. Bengio, A. Lotfi, and K. Rizk. Generalization on the unseen, logic reasoning and degree curriculum. arXiv preprint arXiv:2301.13105, 2023. [find](https://scholar.google.com/scholar?q=E%2E+Abbe%2C+S%2E+Bengio%2C+A%2E+Lotfi%2C+and+K%2E+Rizk%2E+Generalization+on+the+unseen%2C+logic+reasoning+and+degree+curriculum%2E+arXiv+preprint+arXiv%3A2301%2E13105%2C+2023%2E)
2. C. Anil, Y. Wu, A. Andreassen, A. Lewkowycz, V. Misra, V. Ramasesh, A. Slone, G. Gur-Ari, E. Dyer, and B. Neyshabur. [Exploring length generalization in large language models](https://md.research.elamur.ai/free/en/papers/anil-2022-exploring-length-generalization-large-language/). Advances in Neural Information Processing Systems, 35:38546–38556, 2022.
3. P. Awasthi and A. Gupta. [Improving length-generalization in transformers via task hinting](https://md.research.elamur.ai/free/en/papers/awasthi-2023-improving-length-generalization-transformers-task/). arXiv preprint arXiv:2310.00726, 2023.
4. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020.
5. S. Chen, S. Wong, L. Chen, and Y. Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595, 2023. [find](https://scholar.google.com/scholar?q=S%2E+Chen%2C+S%2E+Wong%2C+L%2E+Chen%2C+and+Y%2E+Tian%2E+Extending+context+window+of+large+language+models+via+positional+interpolation%2E+arXiv+preprint+arXiv%3A2306%2E15595%2C+2023%2E)
6. T.-C. Chi, T.-H. Fan, P. J. Ramadge, and A. Rudnicky. Kerple: Kernelized relative positional embedding for length extrapolation. Advances in Neural Information Processing Systems, 35:8386–8399, 2022. [find](https://scholar.google.com/scholar?q=T%2E%2DC%2E+Chi%2C+T%2E%2DH%2E+Fan%2C+P%2E+J%2E+Ramadge%2C+and+A%2E+Rudnicky%2E+Kerple%3A+Kernelized+relative+positional+embedding+for+length+extrapolation%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A8386%E2%80%938399%2C+2022%2E)
7. A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). Journal of Machine Learning Research, 24(240):1–113, 2023.
8. Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. V. Le, and R. Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860, 2019. [find](https://scholar.google.com/scholar?q=Z%2E+Dai%2C+Z%2E+Yang%2C+Y%2E+Yang%2C+J%2E+Carbonell%2C+Q%2E+V%2E+Le%2C+and+R%2E+Salakhutdinov%2E+Transformer%2Dxl%3A+Attentive+language+models+beyond+a+fixed%2Dlength+context%2E+arXiv+preprint+arXiv%3A1901%2E02860%2C+2019%2E)
9. G. Deletang, A. Ruoss, J. Grau-Moya, T. Genewein, L. K. Wenliang, E. Catt, C. Cundy, M. Hutter, S. Legg, J. Veness, and P. A. Ortega. [Neural networks and the chomsky hierarchy](https://md.research.elamur.ai/free/en/papers/deletang-2022-neural-networks-chomsky-hierarchy/). In The Eleventh International Conference on Learning Representations, 2023. URL [https://openreview.net/](https://openreview.net/) forum?id=WbxHAzkeQc… [↗](https://openreview.net/)
10. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805, 2018.
11. S. Duan and Y. Shi. From interpolation to extrapolation: Complete length generalization for arithmetic transformers. arXiv preprint arXiv:2310.11984, 2023. [find](https://scholar.google.com/scholar?q=S%2E+Duan+and+Y%2E+Shi%2E+From+interpolation+to+extrapolation%3A+Complete+length+generalization+for+arithmetic+transformers%2E+arXiv+preprint+arXiv%3A2310%2E11984%2C+2023%2E)
12. Y. Dubois, G. Dagan, D. Hupkes, and E. Bruni. Location attention for extrapolation to longer sequences. arXiv preprint arXiv:1911.03872, 2019. [find](https://scholar.google.com/scholar?q=Y%2E+Dubois%2C+G%2E+Dagan%2C+D%2E+Hupkes%2C+and+E%2E+Bruni%2E+Location+attention+for+extrapolation+to+longer+sequences%2E+arXiv+preprint+arXiv%3A1911%2E03872%2C+2019%2E)
13. N. Dziri, X. Lu, M. Sclar, X. L. Li, L. Jian, B. Y. Lin, P. West, C. Bhagavatula, R. L. Bras, J. D. Hwang, et al. [Faith and fate: Limits of transformers on compositionality](https://md.research.elamur.ai/free/en/papers/dziri-2023-faith-fate-limits-transformers-compositionality/). arXiv preprint arXiv:2305.18654, 2023.
14. J. Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635, 2018. [find](https://scholar.google.com/scholar?q=J%2E+Frankle+and+M%2E+Carbin%2E+The+lottery+ticket+hypothesis%3A+Finding+sparse%2C+trainable+neural+networks%2E+arXiv+preprint+arXiv%3A1803%2E03635%2C+2018%2E)
15. Gemini, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805, 2023.
16. A. Graves, G. Wayne, M. Reynolds, T. Harley, I. Danihelka, A. Grabska-Barwinska, S. G. Colmenarejo, E. Grefenstette, T. Ramalho, J. P. Agapiou, A. P. Badia, K. M. Hermann, Y. Zwols, G. Ostrovski, A. Cain, H. King, C. Summerfield, P. Blunsom, K. Kavukcuoglu, and D. Hassabis. Hybrid computing using a… [find](https://scholar.google.com/scholar?q=A%2E+Graves%2C+G%2E+Wayne%2C+M%2E+Reynolds%2C+T%2E+Harley%2C+I%2E+Danihelka%2C+A%2E+Grabska%2DBarwinska%2C+S%2E+G%2E+Colmenarejo%2C+E%2E+Grefenstette%2C+T%2E+Ramalho%2C+J%2E+P%2E+Agapiou%2C+A%2E+P%2E+Badia%2C+K%2E+M%2E+Hermann%2C+Y%2E+Zwols%2C+G%2E+Ostrovski%2C+A%2E+C)
17. A. Haviv, O. Ram, O. Press, P. Izsak, and O. Levy. [Transformer language models without positional encodings still learn positional information](https://md.research.elamur.ai/free/en/papers/haviv-2022-transformer-language-models-positional-encodings/). arXiv preprint arXiv:2203.16634, 2022.
18. D. Hupkes, V. Dankers, M. Mul, and E. Bruni. [Compositionality decomposed: How do neural networks generalise?](https://md.research.elamur.ai/free/en/papers/hupkes-2019-compositionality-decomposed-neural-networks-generalise/) Journal of Artificial Intelligence Research, 67:757–795, 2020.
19. S. Jelassi, S. d’Ascoli, C. Domingo-Enrich, Y. Wu, Y. Li, and F. Charton. [Length generalization in arithmetic transformers](https://md.research.elamur.ai/free/en/papers/jelassi-2023-length-generalization-arithmetic-transformers/). arXiv preprint arXiv:2306.15400, 2023.
20. A. Kazemnejad, I. Padhi, K. N. Ramamurthy, P. Das, and S. Reddy. [The impact of positional encoding on length generalization in transformers](https://md.research.elamur.ai/free/en/papers/kazemnejad-2023-impact-positional-encoding-length-generalization/). arXiv preprint arXiv:2305.19466, 2023.
21. N. Lee, K. Sreenivasan, J. D. Lee, K. Lee, and D. Papailiopoulos. [Teaching arithmetic to small transformers](https://md.research.elamur.ai/free/en/papers/lee-2023-teaching-arithmetic-small-transformers/). arXiv preprint arXiv:2307.03381, 2023.
22. A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al. [Solving quantitative reasoning problems with language models](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/). Advances in Neural Information Processing Systems, 35:3843–3857, 2022.
23. S. Li, C. You, G. Guruganesh, J. Ainslie, S. Ontanon, M. Zaheer, S. Sanghai, Y. Yang, S. Kumar, and S. Bhojanapalli. Functional interpolation for relative positions improves long context transformers. arXiv preprint arXiv:2310.04418, 2023. [find](https://scholar.google.com/scholar?q=S%2E+Li%2C+C%2E+You%2C+G%2E+Guruganesh%2C+J%2E+Ainslie%2C+S%2E+Ontanon%2C+M%2E+Zaheer%2C+S%2E+Sanghai%2C+Y%2E+Yang%2C+S%2E+Kumar%2C+and+S%2E+Bhojanapalli%2E+Functional+interpolation+for+relative+positions+improves+long+context+transformers%2E)
24. Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, et al. [Competition-level code generation with alphacode](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/). Science, 378(6624):1092– 1097, 2022.
25. B. Liu, J. T. Ash, S. Goel, A. Krishnamurthy, and C. Zhang. Exposing attention glitches with flip-flop language modeling. arXiv preprint arXiv:2306.00946, 2023a. [find](https://scholar.google.com/scholar?q=B%2E+Liu%2C+J%2E+T%2E+Ash%2C+S%2E+Goel%2C+A%2E+Krishnamurthy%2C+and+C%2E+Zhang%2E+Exposing+attention+glitches+with+flip%2Dflop+language+modeling%2E+arXiv+preprint+arXiv%3A2306%2E00946%2C+2023a%2E)
26. B. Liu, J. T. Ash, S. Goel, A. Krishnamurthy, and C. Zhang. Transformers learn shortcuts to automata. In The Eleventh International Conference on Learning Representations, 2023b. URL [https://openreview.net/forum?id=De4FYqjFueZ](https://openreview.net/forum?id=De4FYqjFueZ) . [↗](https://openreview.net/forum?id=De4FYqjFueZ)
27. I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. [find](https://scholar.google.com/scholar?q=I%2E+Loshchilov+and+F%2E+Hutter%2E+Decoupled+weight+decay+regularization%2E+arXiv+preprint+arXiv%3A1711%2E05101%2C+2017%2E)
28. M. Motamedi, N. Sakharnykh, and T. Kaldewey. A data-centric approach for training deep neural networks with less data. arXiv preprint arXiv:2110.03613, 2021. — [arXiv:2110.03613](https://arxiv.org/abs/2110.03613) [find](https://scholar.google.com/scholar?q=M%2E+Motamedi%2C+N%2E+Sakharnykh%2C+and+T%2E+Kaldewey%2E+A+data%2Dcentric+approach+for+training+deep+neural+networks+with+less+data%2E+arXiv+preprint+arXiv%3A2110%2E03613%2C+2021%2E+%E2%80%94+%5BarXiv%3A2110%2E03613%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs)
29. V. Nagarajan, A. Andreassen, and B. Neyshabur. Understanding the failure modes of out-of-distribution generalization. arXiv preprint arXiv:2010.15775, 2020. — [arXiv:2010.15775](https://arxiv.org/abs/2010.15775) [find](https://scholar.google.com/scholar?q=V%2E+Nagarajan%2C+A%2E+Andreassen%2C+and+B%2E+Neyshabur%2E+Understanding+the+failure+modes+of+out%2Dof%2Ddistribution+generalization%2E+arXiv+preprint+arXiv%3A2010%2E15775%2C+2020%2E+%E2%80%94+%5BarXiv%3A2010%2E15775%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F)
30. C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y. Bai, A. Chen, et al. [In-context learning and induction heads](https://md.research.elamur.ai/free/en/papers/olsson-2022-context-learning-induction-heads/). arXiv preprint arXiv:2209.11895, 2022.
31. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). ArXiv, abs/2303.08774, 2023. semanticscholar.org/CorpusID:257532815 .
32. URL https://api. B. Peng, J. Quesnelle, H. Fan, and E. Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071, 2023. [find](https://scholar.google.com/scholar?q=URL+https%3A%2F%2Fapi%2E+B%2E+Peng%2C+J%2E+Quesnelle%2C+H%2E+Fan%2C+and+E%2E+Shippole%2E+Yarn%3A+Efficient+context+window+extension+of+large+language+models%2E+arXiv+preprint+arXiv%3A2309%2E00071%2C+2023%2E)
33. A. Power, Y. Burda, H. Edwards, I. Babuschkin, and V. Misra. [Grokking: Generalization beyond overfitting on small algorithmic datasets](https://md.research.elamur.ai/free/en/papers/power-2022-grokking-generalization-overfitting-small-algorithmic/). arXiv preprint arXiv:2201.02177, 2022.
34. O. Press, N. Smith, and M. Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations, 2022. URL [https://openreview.net/forum?id=R8sQPpGCv0](https://openreview.net/forum?id=R8sQPpGCv0) . [↗](https://openreview.net/forum?id=R8sQPpGCv0)
35. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). The Journal of Machine Learning Research, 21(1):5485–5551, 2020.
36. A. Ruoss, G. Delétang, T. Genewein, J. Grau-Moya, R. Csordás, M. Bennani, S. Legg, and J. Veness. Randomized positional encodings boost length generalization of transformers. arXiv preprint arXiv:2305.16843, 2023. [find](https://scholar.google.com/scholar?q=A%2E+Ruoss%2C+G%2E+Del%C3%A9tang%2C+T%2E+Genewein%2C+J%2E+Grau%2DMoya%2C+R%2E+Csord%C3%A1s%2C+M%2E+Bennani%2C+S%2E+Legg%2C+and+J%2E+Veness%2E+Randomized+positional+encodings+boost+length+generalization+of+transformers%2E+arXiv+preprint+arXiv%3A2305)
37. A. Schwarzschild, E. Borgnia, A. Gupta, F. Huang, U. Vishkin, M. Goldblum, and T. Goldstein. Can you learn an algorithm? generalizing from easy to hard problems with recurrent networks. Advances in Neural Information Processing Systems, 34:6695–6706, 2021. [find](https://scholar.google.com/scholar?q=A%2E+Schwarzschild%2C+E%2E+Borgnia%2C+A%2E+Gupta%2C+F%2E+Huang%2C+U%2E+Vishkin%2C+M%2E+Goldblum%2C+and+T%2E+Goldstein%2E+Can+you+learn+an+algorithm%3F+generalizing+from+easy+to+hard+problems+with+recurrent+networks%2E+Advances+in+Ne)
38. P. Shaw, J. Uszkoreit, and A. Vaswani. [Self-attention with relative position representations](https://md.research.elamur.ai/free/en/papers/shaw-2018-self-attention-relative-position-representations/). arXiv preprint arXiv:1803.02155, 2018.
39. N. Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020. [find](https://scholar.google.com/scholar?q=N%2E+Shazeer%2E+Glu+variants+improve+transformer%2E+arXiv+preprint+arXiv%3A2002%2E05202%2C+2020%2E)
40. R. Shen, S. Bubeck, R. Eldan, Y. T. Lee, Y. Li, and Y. Zhang. [Positional description matters for transformers arithmetic](https://md.research.elamur.ai/free/en/papers/shen-2023-positional-description-matters-transformers-arithmetic/). arXiv preprint arXiv:2311.14737, 2023.
41. D. R. So, W. Mańke, H. Liu, Z. Dai, N. Shazeer, and Q. V. Le. Primer: Searching for efficient transformers for language modeling. arXiv preprint arXiv:2109.08668, 2021. [find](https://scholar.google.com/scholar?q=D%2E+R%2E+So%2C+W%2E+Ma%C5%84ke%2C+H%2E+Liu%2C+Z%2E+Dai%2C+N%2E+Shazeer%2C+and+Q%2E+V%2E+Le%2E+Primer%3A+Searching+for+efficient+transformers+for+language+modeling%2E+arXiv+preprint+arXiv%3A2109%2E08668%2C+2021%2E)
42. J. Su. Rectified rotary position embeddings. [https://github.com/bojone/rerope](https://github.com/bojone/rerope) , 2023. [↗](https://github.com/bojone/rerope)
43. J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). Neurocomputing, 568:127063, 2024.
44. R. Thoppilan, D. De Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, A. Jin, T. Bos, L. Baker, Y. Du, et al. [Lamda: Language models for dialog applications](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/). arXiv preprint arXiv:2201.08239, 2022.
45. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30, 2017.
46. P. Veličković, A. P. Badia, D. Budden, R. Pascanu, A. Banino, M. Dashevskiy, R. Hadsell, and C. Blundell. The clrs algorithmic reasoning benchmark. In International Conference on Machine Learning, pages 22084–22102. PMLR, 2022. [find](https://scholar.google.com/scholar?q=P%2E+Veli%C4%8Dkovi%C4%87%2C+A%2E+P%2E+Badia%2C+D%2E+Budden%2C+R%2E+Pascanu%2C+A%2E+Banino%2C+M%2E+Dashevskiy%2C+R%2E+Hadsell%2C+and+C%2E+Blundell%2E+The+clrs+algorithmic+reasoning+benchmark%2E+In+International+Conference+on+Machine+Learning%2C+pag)
47. Y. Wu, A. Q. Jiang, W. Li, M. Rabe, C. Staats, M. Jamnik, and C. Szegedy. [Autoformalization with large language models](https://md.research.elamur.ai/free/en/papers/wu-2022-autoformalization-large-language-models/). Advances in Neural Information Processing Systems, 35:32353–32368, 2022.
48. Y. Zhang, A. Backurs, S. Bubeck, R. Eldan, S. Gunasekar, and T. Wagner. Unveiling transformers with lego: a synthetic reasoning task. arXiv preprint arXiv:2206.04301, 2022. [find](https://scholar.google.com/scholar?q=Y%2E+Zhang%2C+A%2E+Backurs%2C+S%2E+Bubeck%2C+R%2E+Eldan%2C+S%2E+Gunasekar%2C+and+T%2E+Wagner%2E+Unveiling+transformers+with+lego%3A+a+synthetic+reasoning+task%2E+arXiv+preprint+arXiv%3A2206%2E04301%2C+2022%2E)
49. H. Zhou, A. Nova, H. Larochelle, A. Courville, B. Neyshabur, and H. Sedghi. [Teaching algorithmic reasoning via in-context learning](https://md.research.elamur.ai/free/en/papers/zhou-2022-teaching-algorithmic-reasoning-context-learning/). arXiv preprint arXiv:2211.09066, 2022.
50. H. Zhou, A. Bradley, E. Littwin, N. Razin, O. Saremi, J. Susskind, S. Bengio, and P. Nakkiran. [What algorithms can transformers learn? a study in length generalization](https://md.research.elamur.ai/free/en/papers/zhou-2023-algorithms-transformers-learn-study-length/). arXiv preprint arXiv:2310.16028, 2023.
51. A. Positional Encoding A.1. Additive Relative Positional Encoding (RPE) [find](https://scholar.google.com/scholar?q=A%2E+Positional+Encoding+A%2E1%2E+Additive+Relative+Positional+Encoding+%28RPE%29)
52. Shaw et al. (2018) pioneered additive RPE by integrating position encodings into the attention layer’s key, and optionally the value, rather than the input. This concept was further simplified in T5 (Raffel et al., 2020), where the vector representations of relative positions are simplified to scala… [find](https://scholar.google.com/scholar?q=Shaw+et+al%2E+%282018%29+pioneered+additive+RPE+by+integrating+position+encodings+into+the+attention+layer%E2%80%99s+key%2C+and+optionally+the+value%2C+rather+than+the+input%2E+This+concept+was+further+simplified+in+T5+%28)

*Updated 3 November 2026*