---
title: "Why Can GPT Learn In-Context? Language Models — Dai 2023"
description: "This paper asks why large GPT-style models can solve new tasks from a few demonstrations without any weight updates."
authors: ["Damai Dai", "Yutao Sun", "Li Dong", "Yaru Hao", "Shuming Ma", "Zhifang Sui", "Furu Wei"]
year: 2023
venue: "Findings of the Association for Computational Linguistics: ACL 2023"
source: "https://aclanthology.org/2023.findings-acl.247/"
doi: "10.18653/v1/2023.findings-acl.247"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/dai-2023-why-can-gpt-learn-context/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/dai-2023-why-can-gpt-learn-context/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/) / Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers

# Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers

**Authors:** [Damai Dai](https://md.research.elamur.ai/free/en/authors/damai-dai/), Yutao Sun, [Li Dong](https://md.research.elamur.ai/free/en/authors/li-dong/), [Yaru Hao](https://md.research.elamur.ai/free/en/authors/yaru-hao/), [Shuming Ma](https://md.research.elamur.ai/free/en/authors/shuming-ma/), [Zhifang Sui](https://md.research.elamur.ai/free/en/authors/zhifang-sui/), [Furu Wei](https://md.research.elamur.ai/free/en/authors/furu-wei/) · **Year:** 2023 · **Venue:** Findings of the Association for Computational Linguistics: ACL 2023 · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.findings-acl.247/) · [PDF](https://aclanthology.org/2023.findings-acl.247.pdf) · [DOI 10.18653/v1/2023.findings-acl.247](https://doi.org/10.18653/v1/2023.findings-acl.247) · [OpenAlex](https://openalex.org/W4385571886)

**Citations:** 147 citations (openalex-snapshot · 2026-10-14) · [105](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
This paper asks why large GPT-style models can solve new tasks from a few demonstrations without any weight updates. The authors simplify Transformer attention by dropping the softmax and scaling, which gives a relaxed linear form. In this form, attending to demonstration tokens works like adding an update matrix to the zero-shot weights. That matches a known dual relationship between linear attention and linear layers trained by gradient descent. They therefore treat a pretrained GPT as a meta-optimizer: its forward pass turns demonstrations into "meta-gradients," and attention applies them, so in-context learning (ICL) acts as implicit finetuning.

To test this, they compare ICL with a tightly matched finetuning baseline on six classification tasks. The comparison examines what the models predict, what their attention layers produce, and how attention is distributed, and the two behave alike on every measure. The analogy also suggests a new design: a momentum-based attention, modelled on gradient descent with momentum. It beats vanilla attention on both perplexity and ICL accuracy. The work offers a mechanistic account of ICL in real models and real NLP tasks, not toy regression settings.

## Key points
- **Theory:** Under relaxed linear attention, the attention output splits into two terms. One is the zero-shot term W_ZSL·q. The other is an update ΔW_ICL built from the demonstration tokens. The values W_V X′ play the role of meta-gradients.
- **Finetuning baseline:** It was built to mirror ICL as closely as possible:
  - only the key and value projections are updated;
  - the training examples are the demonstrations themselves;
  - each example gets one step, in the same order and with the same template;
  - training uses a causal language modeling objective.
- **Empirical comparison:** ICL and finetuning were compared on six classification tasks across four views:
  - the final outputs each model predicts;
  - the results produced by the attention layers;
  - how much attention is placed on the query tokens;
  - how much attention is placed on the training tokens.
  
  Because recording intermediate states is costly, only GPT models of up to 2.7B parameters were analysed.
- **Momentum-based attention:** It lowered perplexity consistently at input lengths of 256, 512 and 1024. It also raised ICL accuracy, using up to 32 demonstrations, on SST5, IMDB, MR, CB, ARC-E and PIQA.
- **Limitations:**
  - The analysis rests on a linear approximation of attention.
  - It covers only Transformers.
  - The main analysis uses only classification tasks.
  
  Code is available at https://aka.ms/icl.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

105 papers of our library cite it (of the 140884 whose bibliographies we read); 11 of them have a page here, 94 not yet.

- [A Survey on In-context Learning](https://md.research.elamur.ai/free/en/papers/dong-2023-survey-context-learning/)
- [Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedback](https://md.research.elamur.ai/free/en/papers/fu-2023-improving-language-model-negotiation-self/)
- [In-Context Learning Creates Task Vectors](https://md.research.elamur.ai/free/en/papers/hendel-2023-context-learning-creates-task-vectors/)
- [Challenges and Applications of Large Language Models](https://md.research.elamur.ai/free/en/papers/kaddour-2023-challenges-applications-large-language-models/)
- [Language Models can Solve Computer Tasks](https://md.research.elamur.ai/free/en/papers/kim-2023-language-models-solve-computer-tasks/)
- [Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning](https://md.research.elamur.ai/free/en/papers/wang-2023-label-words-anchors-information-flow/)
- [Larger language models do in-context learning differently](https://md.research.elamur.ai/free/en/papers/wei-2023-larger-language-models-context-learning/)
- [A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions](https://md.research.elamur.ai/free/en/papers/wu-2025-survey-llm-generated-text-detection/)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://md.research.elamur.ai/free/en/papers/yang-2023-dawn-lmms-preliminary-explorations-gpt/)
- [GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher](https://md.research.elamur.ai/free/en/papers/yuan-2023-gpt-4-too-smart-safe/)
- [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{dai2023why,
  title = {{Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers}},
  author = {Dai, Damai and Sun, Yutao and Dong, Li and Hao, Yaru and Ma, Shuming and Sui, Zhifang and Wei, Furu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.247},
  url = {https://doi.org/10.18653/v1/2023.findings-acl.247}
}
```

- [← previous Whose Opinions Do Language Models Reflect?](https://md.research.elamur.ai/free/en/papers/santurkar-2023-whose-opinions-language-models-reflect/)
- [next → Why Can't Discourse Parsing Generalize? A Thorough Investigation of the Impact of Data Diversity](https://md.research.elamur.ai/free/en/papers/liu-2023-why-can-t-discourse-parsing/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/)

## References

25 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mark A Aizerman, Emmanuil M Braverman, and Lev I Rozonoer. 1964. Theoretical foundation of potential functions method in pattern recognition. Avtomatika i Telemekhanika, 25(6):917–936. [find](https://scholar.google.com/scholar?q=Mark+A+Aizerman%2C+Emmanuil+M+Braverman%2C+and+Lev+I+Rozonoer%2E+1964%2E+Theoretical+foundation+of+potential+functions+method+in+pattern+recognition%2E+Avtomatika+i+Telemekhanika%2C+25%286%29%3A917%E2%80%93936%2E)
2. Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. 2022. What learning algorithm is in-context learning? investigations with linear models. CoRR, abs/2211.15661. [find](https://scholar.google.com/scholar?q=Ekin+Aky%C3%BCrek%2C+Dale+Schuurmans%2C+Jacob+Andreas%2C+Tengyu+Ma%2C+and+Denny+Zhou%2E+2022%2E+What+learning+algorithm+is+in%2Dcontext+learning%3F+investigations+with+linear+models%2E+CoRR%2C+abs%2F2211%2E15661%2E) [our card](https://md.research.elamur.ai/free/en/papers/akyurek-2022-learning-algorithm-context-learning-investigations/)
3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… [find](https://scholar.google.com/scholar?q=Tom+B%2E+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretche)
4. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457. [find](https://scholar.google.com/scholar?q=Peter+Clark%2C+Isaac+Cowhey%2C+Oren+Etzioni%2C+Tushar+Khot%2C+Ashish+Sabharwal%2C+Carissa+Schoenick%2C+and+Oyvind+Tafjord%2E+2018%2E+Think+you+have+solved+question+answering%3F+try+arc%2C+the+AI2+reasoning+challenge%2E+CoR)
5. Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. 2019. The commitmentbank: Investigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pages 107–124. [find](https://scholar.google.com/scholar?q=Marie%2DCatherine+De+Marneffe%2C+Mandy+Simons%2C+and+Judith+Tonhauser%2E+2019%2E+The+commitmentbank%3A+Investigating+projection+in+naturally+occurring+discourse%2E+In+proceedings+of+Sinn+und+Bedeutung%2C+volume+23%2C+p) [our card](https://md.research.elamur.ai/free/en/papers/enprag-2019-marneffe-commitmentbank-investigating-projection-naturally/)
6. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. 2023. A survey for in-context learning. CoRR, abs/2301.00234. [find](https://scholar.google.com/scholar?q=Qingxiu+Dong%2C+Lei+Li%2C+Damai+Dai%2C+Ce+Zheng%2C+Zhiyong+Wu%2C+Baobao+Chang%2C+Xu+Sun%2C+Jingjing+Xu%2C+Lei+Li%2C+and+Zhifang+Sui%2E+2023%2E+A+survey+for+in%2Dcontext+learning%2E+CoRR%2C+abs%2F2301%2E00234%2E) [our card](https://md.research.elamur.ai/free/en/papers/dong-2023-survey-context-learning/)
7. Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. 2022. What can transformers learn in-context? A case study of simple function classes. CoRR, abs/2208.01066. [find](https://scholar.google.com/scholar?q=Shivam+Garg%2C+Dimitris+Tsipras%2C+Percy+Liang%2C+and+Gregory+Valiant%2E+2022%2E+What+can+transformers+learn+in%2Dcontext%3F+A+case+study+of+simple+function+classes%2E+CoRR%2C+abs%2F2208%2E01066%2E)
8. J Stuart Hunter. 1986. The exponentially weighted moving average. Journal of quality technology, 18(4):203–210. [find](https://scholar.google.com/scholar?q=J+Stuart+Hunter%2E+1986%2E+The+exponentially+weighted+moving+average%2E+Journal+of+quality+technology%2C+18%284%29%3A203%E2%80%93210%2E)
9. Kazuki Irie, Róbert Csordás, and Jürgen Schmidhuber. 2022. The dual form of neural networks revisited: Connecting test time predictions to training patterns via spotlights of attention. In International Conference on Machine Learning, ICML 2022, volume 162 of Proceedings of Machine Learning Research… [find](https://scholar.google.com/scholar?q=Kazuki+Irie%2C+R%C3%B3bert+Csord%C3%A1s%2C+and+J%C3%BCrgen+Schmidhuber%2E+2022%2E+The+dual+form+of+neural+networks+revisited%3A+Connecting+test+time+predictions+to+training+patterns+via+spotlights+of+attention%2E+In+Internation)
10. Maurice George Kendall. 1948. Rank correlation methods. [find](https://scholar.google.com/scholar?q=Maurice+George+Kendall%2E+1948%2E+Rank+correlation+methods%2E)
11. Louis Kirsch, James Harrison, Jascha Sohl-Dickstein, and Luke Metz. 2022. General-purpose in-context learning by meta-learning transformers. CoRR, abs/2212.04458. — [arXiv:2212.04458](https://arxiv.org/abs/2212.04458) [find](https://scholar.google.com/scholar?q=Louis+Kirsch%2C+James+Harrison%2C+Jascha+Sohl%2DDickstein%2C+and+Luke+Metz%2E+2022%2E+General%2Dpurpose+in%2Dcontext+learning+by+meta%2Dlearning+transformers%2E+CoRR%2C+abs%2F2212%2E04458%2E+%E2%80%94+%5BarXiv%3A2212%2E04458%5D%28https%3A%2F%2Farxiv%2Eor)
12. Louis Kirsch and Jürgen Schmidhuber. 2021. Meta learning backpropagation and improving it. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, pages 14122– 14134. [find](https://scholar.google.com/scholar?q=Louis+Kirsch+and+J%C3%BCrgen+Schmidhuber%2E+2021%2E+Meta+learning+backpropagation+and+improving+it%2E+In+Advances+in+Neural+Information+Processing+Systems+34%3A+Annual+Conference+on+Neural+Information+Processing+S)
13. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon… [find](https://scholar.google.com/scholar?q=Andrew+L%2E+Maas%2C+Raymond+E%2E+Daly%2C+Peter+T%2E+Pham%2C+Dan+Huang%2C+Andrew+Y%2E+Ng%2C+and+Christopher+Potts%2E+2011%2E+Learning+word+vectors+for+sentiment+analysis%2E+In+Proceedings+of+the+49th+Annual+Meeting+of+the+Ass) [our card](https://md.research.elamur.ai/free/en/papers/maas-2011-learning-word-vectors-sentiment-analysis/)
14. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amode… [find](https://scholar.google.com/scholar?q=Catherine+Olsson%2C+Nelson+Elhage%2C+Neel+Nanda%2C+Nicholas+Joseph%2C+Nova+DasSarma%2C+Tom+Henighan%2C+Ben+Mann%2C+Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Tom+Conerly%2C+Dawn+Drain%2C+Deep+Ganguli%2C+Zac+Hatfield%2DDodds%2C+Da)
15. Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 271–278, Barcelona, Spain. [find](https://scholar.google.com/scholar?q=Bo+Pang+and+Lillian+Lee%2E+2004%2E+A+sentimental+education%3A+Sentiment+analysis+using+subjectivity+summarization+based+on+minimum+cuts%2E+In+Proceedings+of+the+42nd+Annual+Meeting+of+the+Association+for+Comp) [our card](https://md.research.elamur.ai/free/en/papers/pang-2004-sentimental-education-sentiment-analysis-subjectivity/)
16. Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In ACL 2005, 43rd Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, pages 115–124. The Association for Computer Lingui… [find](https://scholar.google.com/scholar?q=Bo+Pang+and+Lillian+Lee%2E+2005%2E+Seeing+stars%3A+Exploiting+class+relationships+for+sentiment+categorization+with+respect+to+rating+scales%2E+In+ACL+2005%2C+43rd+Annual+Meeting+of+the+Association+for+Computat) [our card](https://md.research.elamur.ai/free/en/papers/pang-2005-seeing-stars-exploiting-class-relationships/)
17. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–… [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Alex+Perelygin%2C+Jean+Wu%2C+Jason+Chuang%2C+Christopher+D%2E+Manning%2C+Andrew+Ng%2C+and+Christopher+Potts%2E+2013%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E+In)
18. Ilya Sutskever, James Martens, George E. Dahl, and Geoffrey E. Hinton. 2013. On the importance of initialization and momentum in deep learning. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, volume 28 of JMLR Workshop and Conference Proceedings, pages 1139– 1147.… [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+James+Martens%2C+George+E%2E+Dahl%2C+and+Geoffrey+E%2E+Hinton%2E+2013%2E+On+the+importance+of+initialization+and+momentum+in+deep+learning%2E+In+Proceedings+of+the+30th+International+Conference+on+M)
19. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+Advances+in+Neural+Information+Processi)
20. Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. 2022. Transformers learn in-context by gradient descent. ArXiv preprint, abs/2212.07677. [find](https://scholar.google.com/scholar?q=Johannes+von+Oswald%2C+Eyvind+Niklasson%2C+Ettore+Randazzo%2C+Jo%C3%A3o+Sacramento%2C+Alexander+Mordvintsev%2C+Andrey+Zhmoginov%2C+and+Max+Vladymyrov%2E+2022%2E+Transformers+learn+in%2Dcontext+by+gradient+descent%2E+ArXiv+pre) [our card](https://md.research.elamur.ai/free/en/papers/oswald-2022-transformers-learn-context-gradient-descent/)
21. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emergent abilities of large language models. CoRR, abs/2206.076… [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Yi+Tay%2C+Rishi+Bommasani%2C+Colin+Raffel%2C+Barret+Zoph%2C+Sebastian+Borgeaud%2C+Dani+Yogatama%2C+Maarten+Bosma%2C+Denny+Zhou%2C+Donald+Metzler%2C+Ed+H%2E+Chi%2C+Tatsunori+Hashimoto%2C+Oriol+Vinyals%2C+Percy+Liang%2C) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/)
22. Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2022. An explanation of in-context learning as implicit bayesian inference. In The Tenth International Conference on Learning Representations, ICLR 2022. OpenReview.net. [find](https://scholar.google.com/scholar?q=Sang+Michael+Xie%2C+Aditi+Raghunathan%2C+Percy+Liang%2C+and+Tengyu+Ma%2E+2022%2E+An+explanation+of+in%2Dcontext+learning+as+implicit+bayesian+inference%2E+In+The+Tenth+International+Conference+on+Learning+Represent) [our card](https://md.research.elamur.ai/free/en/papers/xie-2021-explanation-context-learning-implicit-bayesian/)
23. Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, pages 649– 657. [find](https://scholar.google.com/scholar?q=Xiang+Zhang%2C+Junbo+Jake+Zhao%2C+and+Yann+LeCun%2E+2015%2E+Character%2Dlevel+convolutional+networks+for+text+classification%2E+In+Advances+in+Neural+Information+Processing+Systems+28%3A+Annual+Conference+on+Neural) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/)
24. Boris T Polyak. 1964. Some methods of speeding up the convergence of iteration methods. Ussr computational mathematics and mathematical physics, 4(5):1–17. [find](https://scholar.google.com/scholar?q=Boris+T+Polyak%2E+1964%2E+Some+methods+of+speeding+up+the+convergence+of+iteration+methods%2E+Ussr+computational+mathematics+and+mathematical+physics%2C+4%285%29%3A1%E2%80%9317%2E)
25. Ofir Press, Noah A. Smith, and Mike Lewis. 2022. Train short, test long: Attention with linear biases enables input length extrapolation. In The Tenth International Conference on Learning Representations, ICLR 2022. OpenReview.net. [find](https://scholar.google.com/scholar?q=Ofir+Press%2C+Noah+A%2E+Smith%2C+and+Mike+Lewis%2E+2022%2E+Train+short%2C+test+long%3A+Attention+with+linear+biases+enables+input+length+extrapolation%2E+In+The+Tenth+International+Conference+on+Learning+Representati)