---
title: "The Impact of Positional Encoding on Length Generalization in"
description: "This study asks how the choice of positional encoding affects that ability on downstream tasks rather… 62 references, cited by 91 papers here."
authors: ["Amirhossein Kazemnejad", "Inkit Padhi", "Karthikeyan Natesan Ramamurthy", "Payel Das", "Siva Reddy"]
year: 2023
venue: "arXiv:2305.19466"
source: "https://arxiv.org/abs/2305.19466"
doi: "10.48550/arxiv.2305.19466"
arxiv: "2305.19466"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kazemnejad-2023-impact-positional-encoding-length-generalization/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kazemnejad-2023-impact-positional-encoding-length-generalization/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / The Impact of Positional Encoding on Length Generalization in Transformers

# The Impact of Positional Encoding on Length Generalization in Transformers

**Authors:** [Amirhossein Kazemnejad](https://md.research.elamur.ai/free/en/authors/amirhossein-kazemnejad/), [Inkit Padhi](https://md.research.elamur.ai/free/en/authors/inkit-padhi/), [Karthikeyan Natesan Ramamurthy](https://md.research.elamur.ai/free/en/authors/karthikeyan-natesan-ramamurthy/), Payel Das and [Siva Reddy](https://md.research.elamur.ai/free/en/authors/siva-reddy/) · **Year:** 2023 · **Venue:** [arXiv:2305.19466](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/) ([NeurIPS 2023](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/2023/)) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2305.19466) · [PDF](https://arxiv.org/pdf/2305.19466) · [DOI 10.48550/arxiv.2305.19466](https://doi.org/10.48550/arxiv.2305.19466) · [arXiv 2305.19466](https://arxiv.org/abs/2305.19466) · [OpenAlex](https://openalex.org/W7133185274)

**Citations:** 43 citations (OpenAlex, 2026-10-14) · [91](https://md.research.elamur.ai/free/en/papers/kazemnejad-2023-impact-positional-encoding-length-generalization/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

Transformer language models are usually trained on limited context sizes but are expected to handle longer inputs at inference. This study asks how the choice of positional encoding affects that ability on downstream tasks rather than on perplexity. The authors trained decoder-only Transformers (~107M parameters, HuggingFace "base" configuration, same hyperparameters for every variant) from scratch on synthetic reasoning and algorithmic tasks. They trained on short instances and scored longer, unseen ones with exact-match accuracy. They compared five options: APE (sinusoidal), T5's Relative PE, ALiBi, Rotary, and no positional encoding at all (NoPE). Surprisingly, NoPE extrapolated best. T5's Relative PE came second, while the encodings behind popular LLMs lagged behind. A theoretical analysis shows that NoPE can express both absolute and relative positions. In practice, SGD-trained NoPE models behave much like T5's relative scheme. Scratchpad (chain-of-thought) prompting did not reliably fix length generalization, and its format strongly influenced results. The work suggests that explicit position signals may be unnecessary for decoder-only models that must handle longer sequences.

## Key points

- The benchmark covers 10 tasks in three groups:
  - primitive tasks: Copy, Reverse;
  - mathematical/reasoning tasks: Addition, Polynomial Evaluation, Sorting, Summation, Parity, LEGO;
  - classic length-generalization datasets: SCAN, PCFG.
- Aggregate mean reciprocal rank across tasks:
  - NoPE ≈ 0.69;
  - T5's Relative PE ≈ 0.55;
  - ALiBi ≈ 0.50;
  - Rotary ≈ 0.33;
  - APE ≈ 0.22.

  NoPE also needs no extra attention computation.
- Attention patterns differ by encoding:
  - NoPE and T5's Relative PE attend to both near and distant tokens;
  - ALiBi favors recent tokens;
  - Rotary and APE show no clear positional preference.
- The authors argue that perplexity-based evaluation can overstate ALiBi's extrapolation, because its recency bias suits the short-range dependencies common in web text.
- Limitation: the study does not examine the effects of large-scale pretraining, since no comparable public LLMs exist that differ only in positional encoding. The paper appeared at NeurIPS 2023 (Mila/McGill, IBM Research).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

## Cite

Cite as: Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das and Siva Reddy. 2023. The Impact of Positional Encoding on Length Generalization in Transformers. arXiv:2305.19466.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{kazemnejad2023impact,
  title = {{The Impact of Positional Encoding on Length Generalization in Transformers}},
  author = {Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das and Siva Reddy},
  year = {2023},
  doi = {10.48550/arxiv.2305.19466},
  eprint = {2305.19466},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.19466}
}
```

- [← previous Simple synthetic data reduces sycophancy in large language models](https://md.research.elamur.ai/free/en/papers/wei-2023-simple-synthetic-data-reduces-sycophancy/)
- [next → Prompting is not a substitute for probability measurements in large language models](https://md.research.elamur.ai/free/en/papers/hu-2023-prompting-is-not-substitute-probability/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

62 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ekin Akyurek and Afra Feyza Akyurek. 2022. Notes on teaching gpt-3 adding numbers. [find](https://scholar.google.com/scholar?q=Ekin+Akyurek+and+Afra+Feyza+Akyurek%2E+2022%2E+Notes+on+teaching+gpt%2D3+adding+numbers%2E)
2. Ekin Akyurek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. 2023. [What learning algorithm is in-context learning? investigations with linear models](https://md.research.elamur.ai/free/en/papers/akyurek-2022-learning-algorithm-context-learning-investigations/). In The Eleventh International Conference on Learning Representations.
3. Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, Logesh Kumar Umapathi, et al. 2023. [Santacoder: don’t reach for the stars!](https://md.research.elamur.ai/free/en/papers/allal-2023-santacoder-don-t-reach-stars/) ArXiv, abs/2301.03988.
4. Cem Anil, Yuhuai Wu, Anders Johan Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Venkatesh Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. 2022. [Exploring length generalization in large language models](https://md.research.elamur.ai/free/en/papers/anil-2022-exploring-length-generalization-large-language/). In Advances in Neural Information Processing Systems.
5. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. [Layer normalization](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/). ArXiv, abs/1607.06450.
6. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
7. Mirelle Candida Bueno, Carlos Gemmell, Jeff Dalton, Roberto Lotufo, and Rodrigo Nogueira. 2022. Induced natural language rationales and interleaved markup tokens enable extrapolation in large language models. In Proceedings of the 1st Workshop on Mathematical Natural Language Processing (MathNLP), p… [find](https://scholar.google.com/scholar?q=Mirelle+Candida+Bueno%2C+Carlos+Gemmell%2C+Jeff+Dalton%2C+Roberto+Lotufo%2C+and+Rodrigo+Nogueira%2E+2022%2E+Induced+natural+language+rationales+and+interleaved+markup+tokens+enable+extrapolation+in+large+language)
8. Ta-Chung Chi, Ting-Han Fan, Alexander Rudnicky, and Peter Ramadge. 2023. Dissecting transformer length extrapolation via the lens of receptive field analysis. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13522– 13537, Toron… [find](https://scholar.google.com/scholar?q=Ta%2DChung+Chi%2C+Ting%2DHan+Fan%2C+Alexander+Rudnicky%2C+and+Peter+Ramadge%2E+2023%2E+Dissecting+transformer+length+extrapolation+via+the+lens+of+receptive+field+analysis%2E+In+Proceedings+of+the+61st+Annual+Meeting)
9. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). ArXiv, abs/2204.02311.
10. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. ArXiv, abs/2210.11416. [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Eric+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+et+al%2E+2022%2E+Scaling+instruction%2Dfinetuned+language+models%2E+ArXi)
11. Róbert Csordás, Kazuki Irie, and Juergen Schmidhuber. 2021. The devil is in the detail: Simple tricks improve systematic generalization of transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 619–634, Online and Punta Cana, Dominican Republic… [find](https://scholar.google.com/scholar?q=R%C3%B3bert+Csord%C3%A1s%2C+Kazuki+Irie%2C+and+Juergen+Schmidhuber%2E+2021%2E+The+devil+is+in+the+detail%3A+Simple+tricks+improve+systematic+generalization+of+transformers%2E+In+Proceedings+of+the+2021+Conference+on+Empiri)
12. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988, Florence, Italy. Assoc… [find](https://scholar.google.com/scholar?q=Zihang+Dai%2C+Zhilin+Yang%2C+Yiming+Yang%2C+Jaime+Carbonell%2C+Quoc+Le%2C+and+Ruslan+Salakhutdinov%2E+2019%2E+Transformer%2DXL%3A+Attentive+language+models+beyond+a+fixed%2Dlength+context%2E+In+Proceedings+of+the+57th+Annu)
13. Gregoire Deletang, Anian Ruoss, Jordi Grau-Moya, Tim Genewein, Li Kevin Wenliang, Elliot Catt, Chris Cundy, Marcus Hutter, Shane Legg, Joel Veness, and Pedro A Ortega. 2023. [Neural networks and the chomsky hierarchy](https://md.research.elamur.ai/free/en/papers/deletang-2022-neural-networks-chomsky-hierarchy/). In International Conference on Learning Representations.
14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
15. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, et al. 2021. A mathematical framework for transformer circuits. Transformer Circuits Thread. [find](https://scholar.google.com/scholar?q=Nelson+Elhage%2C+Neel+Nanda%2C+Catherine+Olsson%2C+Tom+Henighan%2C+Nicholas+Joseph%2C+Ben+Mann%2C+Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Tom+Conerly%2C+Nova+DasSarma%2C+et+al%2E+2021%2E+A+mathematical+framework+for+transf)
16. Daniel Furrer, Marc van Zee, Nathan Scales, and Nathanael Schärli. 2020. Compositional generalization in semantic parsing: Pre-training vs. specialized architectures. ArXiv, abs/2007.08970. [find](https://scholar.google.com/scholar?q=Daniel+Furrer%2C+Marc+van+Zee%2C+Nathan+Scales%2C+and+Nathanael+Sch%C3%A4rli%2E+2020%2E+Compositional+generalization+in+semantic+parsing%3A+Pre%2Dtraining+vs%2E+specialized+architectures%2E+ArXiv%2C+abs%2F2007%2E08970%2E)
17. Edward Gibson. 1998. Linguistic complexity: locality of syntactic dependencies. Cognition, 68(1):1– 76. [find](https://scholar.google.com/scholar?q=Edward+Gibson%2E+1998%2E+Linguistic+complexity%3A+locality+of+syntactic+dependencies%2E+Cognition%2C+68%281%29%3A1%E2%80%93+76%2E)
18. Nicolas Gontier, Koustuv Sinha, Siva Reddy, and Christopher Pal. 2020. Measuring systematic generalization in neural proof generation with transformers. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-… [find](https://scholar.google.com/scholar?q=Nicolas+Gontier%2C+Koustuv+Sinha%2C+Siva+Reddy%2C+and+Christopher+Pal%2E+2020%2E+Measuring+systematic+generalization+in+neural+proof+generation+with+transformers%2E+In+Advances+in+Neural+Information+Processing+Sy)
19. Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. 2016. Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626):471–476. [find](https://scholar.google.com/scholar?q=Alex+Graves%2C+Greg+Wayne%2C+Malcolm+Reynolds%2C+Tim+Harley%2C+Ivo+Danihelka%2C+Agnieszka+Grabska%2DBarwin%CC%81ska%2C+Sergio+G%C3%B3mez+Colmenarejo%2C+Edward+Grefenstette%2C+Tiago+Ramalho%2C+John+Agapiou%2C+et+al%2E+2016%2E+Hybrid+comp)
20. Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. 2022. [Transformer language models without positional encodings still learn positional information](https://md.research.elamur.ai/free/en/papers/haviv-2022-transformer-language-models-positional-encodings/). In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1382–1390, Abu Dhabi, United Arab Emirates. Association for C…
21. Dan Hendrycks and Kevin Gimpel. 2020. Gaussian error linear units (gelus). ArXiv, abs/1606.08415. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks+and+Kevin+Gimpel%2E+2020%2E+Gaussian+error+linear+units+%28gelus%29%2E+ArXiv%2C+abs%2F1606%2E08415%2E)
22. Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. 2020. [Compositionality decomposed: How do neural networks generalise?](https://md.research.elamur.ai/free/en/papers/hupkes-2019-compositionality-decomposed-neural-networks-generalise/) Journal of Artificial Intelligence Research, 67:757–795.
23. Kazuki Irie, Albert Zeyer, Ralf Schlüter, and Hermann Ney. 2019. Language modeling with deep transformers. In Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019, pages 3905–3909. [find](https://scholar.google.com/scholar?q=Kazuki+Irie%2C+Albert+Zeyer%2C+Ralf+Schl%C3%BCter%2C+and+Hermann+Ney%2E+2019%2E+Language+modeling+with+deep+transformers%2E+In+Interspeech+2019%2C+20th+Annual+Conference+of+the+International+Speech+Communication+Associa)
24. Lukasz Kaiser and Ilya Sutskever. 2016. Neural gpus learn algorithms. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings. [find](https://scholar.google.com/scholar?q=Lukasz+Kaiser+and+Ilya+Sutskever%2E+2016%2E+Neural+gpus+learn+algorithms%2E+In+4th+International+Conference+on+Learning+Representations%2C+ICLR+2016%2C+San+Juan%2C+Puerto+Rico%2C+May+2%2D4%2C+2016%2C+Conference+Track+Pro)
25. Shun Kiyono, Sosuke Kobayashi, Jun Suzuki, and Kentaro Inui. 2021. SHAPE: Shifted absolute position embedding for transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3309–3321, Online and Punta Cana, Dominican Republic. Association for Compu… [find](https://scholar.google.com/scholar?q=Shun+Kiyono%2C+Sosuke+Kobayashi%2C+Jun+Suzuki%2C+and+Kentaro+Inui%2E+2021%2E+SHAPE%3A+Shifted+absolute+position+embedding+for+transformers%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+La)
26. Brenden M. Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 o… [find](https://scholar.google.com/scholar?q=Brenden+M%2E+Lake+and+Marco+Baroni%2E+2018%2E+Generalization+without+systematicity%3A+On+the+compositional+skills+of+sequence%2Dto%2Dsequence+recurrent+networks%2E+In+Proceedings+of+the+35th+International+Conferenc)
27. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, et al. 2023. Starcoder: may the source be with you! ArXiv, abs/2305.06161. [find](https://scholar.google.com/scholar?q=Raymond+Li%2C+Loubna+Ben+Allal%2C+Yangtian+Zi%2C+Niklas+Muennighoff%2C+Denis+Kocetkov%2C+Chenghao+Mou%2C+Marc+Marone%2C+Christopher+Akiki%2C+Jia+Li%2C+Jenny+Chim%2C+Qian+Liu%2C+Evgenii+Zheltonozhskii%2C+Terry+Yue+Zhuo%2C+et+al)
28. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022. Holistic evaluation of language models. ArXiv, abs/2211.09110. [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Rishi+Bommasani%2C+Tony+Lee%2C+Dimitris+Tsipras%2C+Dilara+Soylu%2C+Michihiro+Yasunaga%2C+Yian+Zhang%2C+Deepak+Narayanan%2C+Yuhuai+Wu%2C+Ananya+Kumar%2C+et+al%2E+2022%2E+Holistic+evaluation+of+language+models%2E+)
29. Tatiana Likhomanenko, Qiantong Xu, Gabriel Synnaeve, Ronan Collobert, and Alex Rogozhnikov. 2021. CAPE: Encoding relative positions with continuous augmented positional embeddings. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Tatiana+Likhomanenko%2C+Qiantong+Xu%2C+Gabriel+Synnaeve%2C+Ronan+Collobert%2C+and+Alex+Rogozhnikov%2E+2021%2E+CAPE%3A+Encoding+relative+positions+with+continuous+augmented+positional+embeddings%2E+In+Advances+in+Neur)
30. David Lindner, János Kramár, Matthew Rahtz, Thomas McGrath, and Vladimir Mikulik. 2023. Tracr: Compiled transformers as a laboratory for interpretability. ArXiv, abs/2301.05062. [find](https://scholar.google.com/scholar?q=David+Lindner%2C+J%C3%A1nos+Kram%C3%A1r%2C+Matthew+Rahtz%2C+Thomas+McGrath%2C+and+Vladimir+Mikulik%2E+2023%2E+Tracr%3A+Compiled+transformers+as+a+laboratory+for+interpretability%2E+ArXiv%2C+abs%2F2301%2E05062%2E)
31. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. 2023. [The flan collection: Designing data and methods for effective instruction tuning](https://md.research.elamur.ai/free/en/papers/longpre-2023-flan-collection-designing-data-methods/). ArXiv, abs/2301.13688.
32. Ziyang Luo, Artur Kulmizev, and Xiaoxi Mao. 2021. Positional artefacts propagate through masked language model embeddings. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1:… [find](https://scholar.google.com/scholar?q=Ziyang+Luo%2C+Artur+Kulmizev%2C+and+Xiaoxi+Mao%2E+2021%2E+Positional+artefacts+propagate+through+masked+language+model+embeddings%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computationa)
33. Maxwell Nye, Anders Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. [Show your work: Scratchpads for intermediate computation with language models](https://md.research.elamur.ai/free/en/papers/al-2021-show-your-work-scratchpads-intermediate/). ArXiv, abs/2112.00114.
34. Santiago Ontanon, Joshua Ainslie, Zachary Fisher, and Vaclav Cvicek. 2022. Making transformers solve compositional tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3591–3607, Dublin, Ireland. Association for Computationa… [find](https://scholar.google.com/scholar?q=Santiago+Ontanon%2C+Joshua+Ainslie%2C+Zachary+Fisher%2C+and+Vaclav+Cvicek%2E+2022%2E+Making+transformers+solve+compositional+tasks%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computational)
35. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. [Train…](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
36. Sejun Park, Chulhee Yun, Jaeho Lee, and Jinwoo Shin. 2020. Minimum width for universal approximation. ArXiv. [find](https://scholar.google.com/scholar?q=Sejun+Park%2C+Chulhee+Yun%2C+Jaeho+Lee%2C+and+Jinwoo+Shin%2E+2020%2E+Minimum+width+for+universal+approximation%2E+ArXiv%2E)
37. Ofir Press, Noah A. Smith, and Mike Lewis. 2022. Train short, test long: Attention with linear biases enables input length extrapolation. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. [find](https://scholar.google.com/scholar?q=Ofir+Press%2C+Noah+A%2E+Smith%2C+and+Mike+Lewis%2E+2022%2E+Train+short%2C+test+long%3A+Attention+with+linear+biases+enables+input+length+extrapolation%2E+In+The+Tenth+International+Conference+on+Learning+Representati)
38. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+Blog%2C+1%288%29%3A9%2E)
39. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research, 21:140:1–140:67.
40. David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. 2019. [Analysing mathematical reasoning abilities of neural models](https://md.research.elamur.ai/free/en/papers/saxton-2019-analysing-mathematical-reasoning-abilities-neural/). In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
41. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022a. Bloom: A 176b-parameter open-access multilingual language model. ArXiv, abs/2211.05100. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%CC%81%2C+Daniel+Hesslow%2C+Roman+Castagn%C3%A9%2C+Alexandra+Sasha+Luccioni%2C+Fran%C3%A7ois+Yvon%2C+Matthias+Gall%C3%A9%2C+et+al%2E+2022a%2E+Bloom%3A+A+176b%2Dparamete)
42. Teven Le Scao, Thomas Wang, Daniel Hesslow, Lucile Saulnier, Stas Bekman, M Saiful Bari, Stella Biderman, Hady Elsahar, Jason Phang, Ofir Press, Colin Raffel, Victor Sanh, Sheng Shen, Lintang Sutawika, Jaesung Tae, Zheng Xin Yong, Julien Launay, and Iz Beltagy. 2022b. [What language model to train if…](https://md.research.elamur.ai/free/en/papers/scao-2022-language-model-train-if-have/)
43. Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. [Self-attention with relative position representations](https://md.research.elamur.ai/free/en/papers/shaw-2018-self-attention-relative-position-representations/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 464–468, New…
44. Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, and Chengqi Zhang. 2018. [Disan: Directional self-attention network for rnn/cnn-free language understanding](https://md.research.elamur.ai/free/en/papers/shen-2017-disan-directional-self-attention-network/). In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Arti…
45. Koustuv Sinha, Amirhossein Kazemnejad, Siva Reddy, Joelle Pineau, Dieuwke Hupkes, and Adina Williams. 2022. The curious case of absolute position embeddings. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4449–4472, Abu Dhabi, United Arab Emirates. Association for Co… [find](https://scholar.google.com/scholar?q=Koustuv+Sinha%2C+Amirhossein+Kazemnejad%2C+Siva+Reddy%2C+Joelle+Pineau%2C+Dieuwke+Hupkes%2C+and+Adina+Williams%2E+2022%2E+The+curious+case+of+absolute+position+embeddings%2E+In+Findings+of+the+Association+for+Computa)
46. Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L. Hamilton. 2019. CLUTRR: A diagnostic benchmark for inductive reasoning from text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan… [find](https://scholar.google.com/scholar?q=Koustuv+Sinha%2C+Shagun+Sodhani%2C+Jin+Dong%2C+Joelle+Pineau%2C+and+William+L%2E+Hamilton%2E+2019%2E+CLUTRR%3A+A+diagnostic+benchmark+for+inductive+reasoning+from+text%2E+In+Proceedings+of+the+2019+Conference+on+Empiri)
47. Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. 2021. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). ArXiv, abs/2104.09864.
48. Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. 2022. Scale efficiently: Insights from pretraining and finetuning transformers. In The Tenth International Conference on Learning Representations, IC… [find](https://scholar.google.com/scholar?q=Yi+Tay%2C+Mostafa+Dehghani%2C+Jinfeng+Rao%2C+William+Fedus%2C+Samira+Abnar%2C+Hyung+Won+Chung%2C+Sharan+Narang%2C+Dani+Yogatama%2C+Ashish+Vaswani%2C+and+Donald+Metzler%2E+2022%2E+Scale+efficiently%3A+Insights+from+pretrainin)
49. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. [Galactica: A large language model for science](https://md.research.elamur.ai/free/en/papers/taylor-2022-galactica-large-language-model-science/). ArXiv, abs/2211.09085.
50. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. [LLaMA: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). ArXiv, a…
51. Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Transformer dissection: An unified understanding for transformer’s attention via the lens of kernel. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and… [find](https://scholar.google.com/scholar?q=Yao%2DHung+Hubert+Tsai%2C+Shaojie+Bai%2C+Makoto+Yamada%2C+Louis%2DPhilippe+Morency%2C+and+Ruslan+Salakhutdinov%2E+2019%2E+Transformer+dissection%3A+An+unified+understanding+for+transformer%E2%80%99s+attention+via+the+lens+of+k)
52. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017,…
53. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, et al. 2022. Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks. In Proc. of EMNLP, pages 5… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Swaroop+Mishra%2C+Pegah+Alipoormolabashi%2C+Yeganeh+Kordi%2C+Amirreza+Mirzaei%2C+Atharva+Naik%2C+Arjun+Ashok%2C+Arut+Selvan+Dhanasekaran%2C+Anjana+Arunkumar%2C+David+Stap%2C+et+al%2E+2022%2E+Super%2DNaturalInst)
54. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022a. [Finetuned language models are zero-shot learners](https://md.research.elamur.ai/free/en/papers/wei-2021-finetuned-language-models-zero-shot/). In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenRevie…
55. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022b. [Chain of thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Advances in Neural Information Processing Systems.
56. Gail Weiss, Yoav Goldberg, and Eran Yahav. 2021. Thinking like transformers. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 11080–11090. PMLR. [find](https://scholar.google.com/scholar?q=Gail+Weiss%2C+Yoav+Goldberg%2C+and+Eran+Yahav%2E+2021%2E+Thinking+like+transformers%2E+In+Proceedings+of+the+38th+International+Conference+on+Machine+Learning%2C+ICML+2021%2C+18%2D24+July+2021%2C+Virtual+Event%2C+volume+)
57. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe… — [Transformers: State-of-the-Art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
58. Baosong Yang, Longyue Wang, Derek F. Wong, Lidia S. Chao, and Zhaopeng Tu. 2019. Assessing the ability of self-attention networks to learn word order. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3635–3644, Florence, Italy. Association for Computa… [find](https://scholar.google.com/scholar?q=Baosong+Yang%2C+Longyue+Wang%2C+Derek+F%2E+Wong%2C+Lidia+S%2E+Chao%2C+and+Zhaopeng+Tu%2E+2019%2E+Assessing+the+ability+of+self%2Dattention+networks+to+learn+word+order%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+)
59. Gilad Yehudai, Ethan Fetaya, Eli A. Meirom, Gal Chechik, and Haggai Maron. 2021. From local structures to size generalization in graph neural networks. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Mac… [find](https://scholar.google.com/scholar?q=Gilad+Yehudai%2C+Ethan+Fetaya%2C+Eli+A%2E+Meirom%2C+Gal+Chechik%2C+and+Haggai+Maron%2E+2021%2E+From+local+structures+to+size+generalization+in+graph+neural+networks%2E+In+Proceedings+of+the+38th+International+Confere)
60. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. [Opt: Open pre-trai…](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/)
61. Yi Zhang, Arturs Backurs, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, and Tal Wagner. 2023. Unveiling transformers with lego: a synthetic reasoning task. ArXiv, abs/2206.04301. [find](https://scholar.google.com/scholar?q=Yi+Zhang%2C+Arturs+Backurs%2C+S%C3%A9bastien+Bubeck%2C+Ronen+Eldan%2C+Suriya+Gunasekar%2C+and+Tal+Wagner%2E+2023%2E+Unveiling+transformers+with+lego%3A+a+synthetic+reasoning+task%2E+ArXiv%2C+abs%2F2206%2E04301%2E)
62. FLAN (CoT) Super-Natural Instructions Count Example Length Figure A.1: Histogram of instruction lengths in two instruction finetuning datasets: FLAN (CoT subset) (Longpre et al., 2023) and Super Natural Instructions (Wang et al., 2022). The dotted line indicates the median length of the instructions… [find](https://scholar.google.com/scholar?q=FLAN+%28CoT%29+Super%2DNatural+Instructions+Count+Example+Length+Figure+A%2E1%3A+Histogram+of+instruction+lengths+in+two+instruction+finetuning+datasets%3A+FLAN+%28CoT+subset%29+%28Longpre+et+al%2E%2C+2023%29+and+Super+Natur)

*Updated 2 November 2026*