---
title: "Transformers Learn to Achieve Second-Order Convergence"
description: "Together, these results offer a mechanistic account of in-context learning that fits observed behaviour… 62 references, cited by 12 papers here."
authors: ["Deqing Fu", "Tian-Qi Chen", "Robin Jia", "Vatsal Sharan"]
year: 2023
venue: "arXiv:2310.17086"
source: "https://arxiv.org/abs/2310.17086"
doi: "10.48550/arxiv.2310.17086"
arxiv: "2310.17086"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/fu-2023-transformers-learn-achieve-second-order/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/fu-2023-transformers-learn-achieve-second-order/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/14/) / Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression

# Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression

**Authors:** Deqing Fu, Tian-Qi Chen, [Robin Jia](https://md.research.elamur.ai/free/en/authors/robin-jia/) and Vatsal Sharan · **Year:** 2023 · **Venue:** [arXiv:2310.17086](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/) ([NeurIPS 2023](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/2023/)) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2310.17086) · [PDF](https://arxiv.org/pdf/2310.17086) · [DOI 10.48550/arxiv.2310.17086](https://doi.org/10.48550/arxiv.2310.17086) · [arXiv 2310.17086](https://arxiv.org/abs/2310.17086) · [OpenAlex](https://openalex.org/W4415796413)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [12](https://md.research.elamur.ai/free/en/papers/fu-2023-transformers-learn-achieve-second-order/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks which algorithm Transformers effectively run when they solve linear regression in context, using only example pairs in the prompt and no weight updates. The prevailing explanation is that they carry out Gradient Descent internally. The authors test an alternative: that they behave like a second-order optimizer. They train Transformers on synthetic regression prompts, following the Garg et al. setup with Gaussian weights and inputs, including ill-conditioned covariance matrices. They then compare the predictions of each layer with successive iterates of Gradient Descent, Online GD, and Iterative Newton's Method (Newton–Schulz).

Layer-wise predictions line up with Newton iterations in a linear fashion. Matching them with GD would take a number of steps that grows exponentially with depth. Transformers also stay accurate on badly conditioned problems, where GD slows down. On the theory side, the authors give an explicit construction in which one Transformer layer performs one Newton update. Together, these results offer a mechanistic account of in-context learning that fits observed behaviour better than the GD hypothesis.

## Key points
- Each middle Transformer layer does roughly the work of about 3 Newton iterations. Some single layers make as much progress as hundreds of GD steps.
- Iterative Newton needs O(log κ + log log(1/ϵ)) steps to converge, while GD needs O(κ log(1/ϵ)). Transformers track the faster, Newton-like rate.
- The construction implements k Newton iterations with k + 8 layers (3 for initialization, 5 for readout) and hidden size O(d). An earlier looped-Transformer construction by Giannou et al. needed 13 layers per iteration.
- LSTMs do not show this behaviour: their predictions do not improve from layer to layer.
- Code is available at github.com/DeqingFu/transformers-icl-second-order.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works

## Cite

Cite as: Deqing Fu, Tian-Qi Chen, Robin Jia and Vatsal Sharan. 2023. Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression. arXiv:2310.17086.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{fu2023transformers,
  title = {{Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression}},
  author = {Deqing Fu, Tian-Qi Chen, Robin Jia and Vatsal Sharan},
  year = {2023},
  doi = {10.48550/arxiv.2310.17086},
  eprint = {2310.17086},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.17086}
}
```

- [← previous Tailoring Self-Rationalizers with Multi-Reward Distillation](https://md.research.elamur.ai/free/en/papers/ramnath-2023-tailoring-self-rationalizers-multi-reward/)
- [next → Arithmetic with Language Models: from Memorization to Computation](https://md.research.elamur.ai/free/en/papers/maltoni-2023-arithmetic-language-models-memorization-computation/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/14/)

## References

62 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. [Transformers learn to implement preconditioned gradient descent for in-context learning](https://md.research.elamur.ai/free/en/papers/ahn-2023-transformers-learn-implement-preconditioned-gradient/). ArXiv, abs/2306.00297, 2023. 1, 2
2. Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. [What learning algorithm is in-context learning? investigations with linear models](https://md.research.elamur.ai/free/en/papers/akyurek-2022-learning-algorithm-context-learning-investigations/). ArXiv, abs/2211.15661, 2022. 2, 3.3, 5.1, B.1, B.4, B.2, B.5
3. Yossi Arjevani, Shai Shalev-Shwartz, and Ohad Shamir. On lower and upper bounds in smooth and strongly convex optimization. Journal of Machine Learning Research, 17(126):1–51, 2016. URL [http://jmlr.org/papers/v17/15-106.html](http://jmlr.org/papers/v17/15-106.html). 4.2 [↗](http://jmlr.org/papers/v17/15-106.html)
4. Yu Bai, Fan Chen, Haiquan Wang, Caiming Xiong, and Song Mei. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. ArXiv, abs/2306.04637, 2023. 2 — [arXiv:2306.04637](https://arxiv.org/abs/2306.04637) [find](https://scholar.google.com/scholar?q=Yu+Bai%2C+Fan+Chen%2C+Haiquan+Wang%2C+Caiming+Xiong%2C+and+Song+Mei%2E+Transformers+as+statisticians%3A+Provable+in%2Dcontext+learning+with+in%2Dcontext+algorithm+selection%2E+ArXiv%2C+abs%2F2306%2E04637%2C+2023%2E+2+%E2%80%94+%5BarXiv%3A23)
5. Adi Ben-Israel. An iterative method for computing the generalized inverse of an arbitrary matrix. Mathematics of Computation, 19(91):452–455, 1965. ISSN 00255718, 10886842. URL [http://www.jstor.org/stable/2003676](http://www.jstor.org/stable/2003676). 3.1 [↗](http://www.jstor.org/stable/2003676)
6. Stephen P Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004. 3.1 [find](https://scholar.google.com/scholar?q=Stephen+P+Boyd+and+Lieven+Vandenberghe%2E+Convex+optimization%2E+Cambridge+university+press%2C+2004%2E+3%2E1)
7. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretchen)
8. Curran Associates, Inc., 2020. URL [https://proceedings.neurips.cc/paper_files/paper/2020/](https://proceedings.neurips.cc/paper_files/paper/2020/) file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf. 1, 2 [↗](https://proceedings.neurips.cc/paper_files/paper/2020/)
9. Han Cai, Chuang Gan, and Song Han. Efficientvit: Enhanced linear attention for high-resolution low-computation visual recognition. ArXiv, abs/2205.14756, 2022. 3.2 [find](https://scholar.google.com/scholar?q=Han+Cai%2C+Chuang+Gan%2C+and+Song+Han%2E+Efficientvit%3A+Enhanced+linear+attention+for+high%2Dresolution+low%2Dcomputation+visual+recognition%2E+ArXiv%2C+abs%2F2205%2E14756%2C+2022%2E+3%2E2)
10. Ting-Yun Chang and Robin Jia. Data curation alone can stabilize in-context learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8123–8144, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.186… [find](https://scholar.google.com/scholar?q=Ting%2DYun+Chang+and+Robin+Jia%2E+Data+curation+alone+can+stabilize+in%2Dcontext+learning%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Papers%29%2C)
11. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… — [PaLM: Scaling Language Modeling with Pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
12. Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. [Dola: Decoding by contrasting layers improves factuality in large language models](https://md.research.elamur.ai/free/en/papers/chuang-2023-dola-decoding-contrasting-layers-improves/), 2023. 4.1
13. Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability, 2023. 2 [find](https://scholar.google.com/scholar?q=Arthur+Conmy%2C+Augustine+N%2E+Mavor%2DParker%2C+Aengus+Lynch%2C+Stefan+Heimersheim%2C+and+Adri%C3%A0+Garriga%2DAlonso%2E+Towards+automated+circuit+discovery+for+mechanistic+interpretability%2C+2023%2E+2)
14. Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. [Why can gpt learn in-context? language models secretly perform gradient descent as meta-optimizers](https://md.research.elamur.ai/free/en/papers/dai-2023-why-can-gpt-learn-context/). ArXiv, abs/2212.10559, 2023. 1, 2
15. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum…
16. Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. [What can transformers learn in-context? a case study of simple function classes](https://md.research.elamur.ai/free/en/papers/garg-2022-transformers-learn-context-case-study/). ArXiv, abs/2208.01066, 2022. 1, 2, 3.2, 4, A.3.3
17. Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. [Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space](https://md.research.elamur.ai/free/en/papers/geva-2022-transformer-feed-forward-layers-build/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 30–45, Abu Dhabi, United Arab Emirates, December 2…
18. Angeliki Giannou, Shashank Rajput, Jy-Yong Sohn, Kangwook Lee, Jason D. Lee, and Dimitris Papailiopoulos. Looped transformers as programmable computers. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th Interna… [find](https://scholar.google.com/scholar?q=Angeliki+Giannou%2C+Shashank+Rajput%2C+Jy%2DYong+Sohn%2C+Kangwook+Lee%2C+Jason+D%2E+Lee%2C+and+Dimitris+Papailiopoulos%2E+Looped+transformers+as+programmable+computers%2E+In+Andreas+Krause%2C+Emma+Brunskill%2C+Kyunghyun+Ch)
19. Chi Han, Ziqi Wang, Han Zhao, and Heng Ji. In-context learning of large language models explained as kernel regression, 2023. 2 [find](https://scholar.google.com/scholar?q=Chi+Han%2C+Ziqi+Wang%2C+Han+Zhao%2C+and+Heng+Ji%2E+In%2Dcontext+learning+of+large+language+models+explained+as+kernel+regression%2C+2023%2E+2)
20. Michael Hassid, Hao Peng, Daniel Rotem, Jungo Kasai, Ivan Montero, Noah A. Smith, and Roy Schwartz. How much does attention actually attend? questioning the importance of attention in pretrained transformers, 2022. 2 [find](https://scholar.google.com/scholar?q=Michael+Hassid%2C+Hao+Peng%2C+Daniel+Rotem%2C+Jungo+Kasai%2C+Ivan+Montero%2C+Noah+A%2E+Smith%2C+and+Roy+Schwartz%2E+How+much+does+attention+actually+attend%3F+questioning+the+importance+of+attention+in+pretrained+trans)
21. Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8): 1735–1780, 11 1997. ISSN 0899-7667. doi: 10.1162/neco.1997.9.8.1735. URL https://doi.org/ 10.1162/neco.1997.9.8.1735. A.1 [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+J%C3%BCrgen+Schmidhuber%2E+Long+Short%2DTerm+Memory%2E+Neural+Computation%2C+9%288%29%3A+1735%E2%80%931780%2C+11+1997%2E+ISSN+0899%2D7667%2E+doi%3A+10%2E1162%2Fneco%2E1997%2E9%2E8%2E1735%2E+URL+https%3A%2F%2Fdoi%2Eorg%2F+10%2E1162%2Fneco%2E1997%2E9%2E)
22. Ivan Lee, Nan Jiang, and Taylor Berg-Kirkpatrick. Exploring the relationship between model architecture and in-context learning ability, 2023. 2 [find](https://scholar.google.com/scholar?q=Ivan+Lee%2C+Nan+Jiang%2C+and+Taylor+Berg%2DKirkpatrick%2E+Exploring+the+relationship+between+model+architecture+and+in%2Dcontext+learning+ability%2C+2023%2E+2)
23. Yingcong Li, Muhammed Emrullah Ildiz, Dimitris Papailiopoulos, and Samet Oymak. Transformers as algorithms: Generalization and stability in in-context learning. In International Conference on Machine Learning, 2023. 2, 6 [find](https://scholar.google.com/scholar?q=Yingcong+Li%2C+Muhammed+Emrullah+Ildiz%2C+Dimitris+Papailiopoulos%2C+and+Samet+Oymak%2E+Transformers+as+algorithms%3A+Generalization+and+stability+in+in%2Dcontext+learning%2E+In+International+Conference+on+Machine+)
24. Dong C. Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical Programming, 45:503–528, 1989. URL [https://api.semanticscholar.org/](https://api.semanticscholar.org/) CorpusID:5681609. 4.2 [↗](https://api.semanticscholar.org/)
25. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. [What makes good in-context examples for GPT-3?](https://md.research.elamur.ai/free/en/papers/liu-2022-what-makes-good-context-examples/) In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 100–114, Dubl…
26. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. [Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity](https://md.research.elamur.ai/free/en/papers/lu-2022-fantastically-ordered-prompts-where-find/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8…
27. Arvind V. Mahankali, Tatsunori Hashimoto, and Tengyu Ma. [One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention](https://md.research.elamur.ai/free/en/papers/mahankali-2023-one-step-gradient-descent-provably/). In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/](https://openreview.net/) forum?id=8p3fu56lKc. 1, 2 [↗](https://openreview.net/)
28. Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. [Noisy channel language model prompting for few-shot text classification](https://md.research.elamur.ai/free/en/papers/min-2021-noisy-channel-language-model-prompting/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5316–5330, Dublin, Ireland, May 2022a.…
29. Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. MetaICL: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2791–2809, Seattle, United States, July 20… [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Hannaneh+Hajishirzi%2E+MetaICL%3A+Learning+to+learn+in+context%2E+In+Proceedings+of+the+2022+Conference+of+the+North+American+Chapter+of+the+Association+for+Comp)
30. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. [Rethinking the role of demonstrations: What makes in-context learning work?](https://md.research.elamur.ai/free/en/papers/min-2022-rethinking-role-demonstrations-what-makes/) In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048–11064, Abu Dhab…
31. E.H Moore. On the reciprocal of the general algebraic matrix. Bulletin of American Mathematical Society, 26:394–395, 1920. 3.1 [find](https://scholar.google.com/scholar?q=E%2EH+Moore%2E+On+the+reciprocal+of+the+general+algebraic+matrix%2E+Bulletin+of+American+Mathematical+Society%2C+26%3A394%E2%80%93395%2C+1920%2E+3%2E1)
32. Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. [Progress measures for grokking via mechanistic interpretability](https://md.research.elamur.ai/free/en/papers/nanda-2023-progress-measures-grokking-mechanistic-interpretability/), 2023. 2
33. A.S. Nemirovski and D.B Yudin. Problem complexity and method efficiency in optimization. 1983. 4.2 [find](https://scholar.google.com/scholar?q=A%2ES%2E+Nemirovski+and+D%2EB+Yudin%2E+Problem+complexity+and+method+efficiency+in+optimization%2E+1983%2E+4%2E2)
34. Tai Nguyen and Eric Wong. arXiv:2302.11042, 2023. 2 — [arXiv:2302.11042](https://arxiv.org/abs/2302.11042) [find](https://scholar.google.com/scholar?q=Tai+Nguyen+and+Eric+Wong%2E+arXiv%3A2302%2E11042%2C+2023%2E+2+%E2%80%94+%5BarXiv%3A2302%2E11042%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2302%2E11042%29)
35. In-context example selection with influences. arXiv preprint Jorge Nocedal and Stephen J Wright. Numerical optimization. Springer, 1999. 4.2, A.2.3 [find](https://scholar.google.com/scholar?q=In%2Dcontext+example+selection+with+influences%2E+arXiv+preprint+Jorge+Nocedal+and+Stephen+J+Wright%2E+Numerical+optimization%2E+Springer%2C+1999%2E+4%2E2%2C+A%2E2%2E3)
36. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), 2023. URL http://arxiv.org/abs/2303.08774v3. 1 [↗](http://arxiv.org/abs/2303.08774v3)
37. Victor Y. Pan and Robert S. Schreiber. An improved newton iteration for the generalized inverse of a matrix, with applications. SIAM J. Sci. Comput., 12:1109–1130, 1991. 3.1 [find](https://scholar.google.com/scholar?q=Victor+Y%2E+Pan+and+Robert+S%2E+Schreiber%2E+An+improved+newton+iteration+for+the+generalized+inverse+of+a+matrix%2C+with+applications%2E+SIAM+J%2E+Sci%2E+Comput%2E%2C+12%3A1109%E2%80%931130%2C+1991%2E+3%2E1)
38. Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. [Grokking: Generalization beyond overfitting on small algorithmic datasets](https://md.research.elamur.ai/free/en/papers/power-2022-grokking-generalization-overfitting-small-algorithmic/), 2022. 2
39. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. 3.2 [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+Language+models+are+unsupervised+multitask+learners%2E+2019%2E+3%2E2)
40. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen… [find](https://scholar.google.com/scholar?q=Jack+W%2E+Rae%2C+Sebastian+Borgeaud%2C+Trevor+Cai%2C+Katie+Millican%2C+Jordan+Hoffmann%2C+Francis+Song%2C+John+Aslanides%2C+Sarah+Henderson%2C+Roman+Ring%2C+Susannah+Young%2C+Eliza+Rutherford%2C+Tom+Hennigan%2C+Jacob+Menick%2C+A)
41. Allan Raventós, Mansheej Paul, Feng Chen, and Surya Ganguli. Pretraining task diversity and the emergence of non-bayesian in-context learning for regression, 2023. 2 [find](https://scholar.google.com/scholar?q=Allan+Ravent%C3%B3s%2C+Mansheej+Paul%2C+Feng+Chen%2C+and+Surya+Ganguli%2E+Pretraining+task+diversity+and+the+emergence+of+non%2Dbayesian+in%2Dcontext+learning+for+regression%2C+2023%2E+2)
42. Ohad Rubin, Jonathan Herzig, and Jonathan Berant. [Learning to retrieve prompts for in-context learning](https://md.research.elamur.ai/free/en/papers/rubin-2021-learning-retrieve-prompts-context-learning/). In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2655–2671, Seattle, United States, July 202…
43. Günther Schulz. Iterative berechung der reziproken matrix. Zeitschrift für Angewandte Mathematik und Mechanik (Journal of Applied Mathematics and Mechanics), 13:57–59, 1933. 3.1 [find](https://scholar.google.com/scholar?q=G%C3%BCnther+Schulz%2E+Iterative+berechung+der+reziproken+matrix%2E+Zeitschrift+f%C3%BCr+Angewandte+Mathematik+und+Mechanik+%28Journal+of+Applied+Mathematics+and+Mechanics%29%2C+13%3A57%E2%80%9359%2C+1933%2E+3%2E1)
44. Vatsal Sharan, Aaron Sidford, and Gregory Valiant. Memory-sample tradeoffs for linear regression with small error. In Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing, pages 890–901, 2019. 4.3 [find](https://scholar.google.com/scholar?q=Vatsal+Sharan%2C+Aaron+Sidford%2C+and+Gregory+Valiant%2E+Memory%2Dsample+tradeoffs+for+linear+regression+with+small+error%2E+In+Proceedings+of+the+51st+Annual+ACM+SIGACT+Symposium+on+Theory+of+Computing%2C+pages+)
45. Kai Shen, Junliang Guo, Xu Tan, Siliang Tang, Rui Wang, and Jiang Bian. A study on relu and softmax in transformer, 2023a. 3.2 [find](https://scholar.google.com/scholar?q=Kai+Shen%2C+Junliang+Guo%2C+Xu+Tan%2C+Siliang+Tang%2C+Rui+Wang%2C+and+Jiang+Bian%2E+A+study+on+relu+and+softmax+in+transformer%2C+2023a%2E+3%2E2)
46. Lingfeng Shen, Aayush Mishra, and Daniel Khashabi. Do pretrained transformers really learn in-context by gradient descent?, 2023b. 2 [find](https://scholar.google.com/scholar?q=Lingfeng+Shen%2C+Aayush+Mishra%2C+and+Daniel+Khashabi%2E+Do+pretrained+transformers+really+learn+in%2Dcontext+by+gradient+descent%3F%2C+2023b%2E+2)
47. Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zhang, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwa… — [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://md.research.elamur.ai/free/en/papers/smith-2022-using-deepspeed-megatron-train-megatron/)
48. Torsten Soderstrom and G. W. Stewart. On the numerical properties of an iterative method for computing the moore- penrose generalized inverse. SIAM Journal on Numerical Analysis, 11(1): 61–74, 1974. ISSN 00361429. URL [http://www.jstor.org/stable/2156431](http://www.jstor.org/stable/2156431). 3.1 [↗](http://www.jstor.org/stable/2156431)
49. Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. Selective annotation makes language models better few-shot learners. In The Eleventh International Conference on Learning Representations, 2023. URL htt… [find](https://scholar.google.com/scholar?q=Hongjin+Su%2C+Jungo+Kasai%2C+Chen+Henry+Wu%2C+Weijia+Shi%2C+Tianlu+Wang%2C+Jiayi+Xin%2C+Rui+Zhang%2C+Mari+Ostendorf%2C+Luke+Zettlemoyer%2C+Noah+A%2E+Smith%2C+and+Tao+Yu%2E+Selective+annotation+makes+language+models+better+fe)
50. Davoud Ataee Tarzanagh, Yingcong Li, Christos Thrampoulidis, and Samet Oymak. Transformers as support vector machines. ArXiv, abs/2308.16898, 2023a. 2, 6 — [arXiv:2308.16898](https://arxiv.org/abs/2308.16898) [find](https://scholar.google.com/scholar?q=Davoud+Ataee+Tarzanagh%2C+Yingcong+Li%2C+Christos+Thrampoulidis%2C+and+Samet+Oymak%2E+Transformers+as+support+vector+machines%2E+ArXiv%2C+abs%2F2308%2E16898%2C+2023a%2E+2%2C+6+%E2%80%94+%5BarXiv%3A2308%2E16898%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F230)
51. Davoud Ataee Tarzanagh, Yingcong Li, Xuechen Zhang, and Samet Oymak. Max-margin token selection in attention mechanism, 2023b. 2 [find](https://scholar.google.com/scholar?q=Davoud+Ataee+Tarzanagh%2C+Yingcong+Li%2C+Xuechen+Zhang%2C+and+Samet+Oymak%2E+Max%2Dmargin+token+selection+in+attention+mechanism%2C+2023b%2E+2)
52. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzho… — [LaMDA: Language Models for Dialog Applications](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/)
53. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing…
54. Max Vladymyrov, Johannes von Oswald, Mark Sandler, and Rong Ge. Linear transformers are versatile in-context learners, 2024. 1 [find](https://scholar.google.com/scholar?q=Max+Vladymyrov%2C+Johannes+von+Oswald%2C+Mark+Sandler%2C+and+Rong+Ge%2E+Linear+transformers+are+versatile+in%2Dcontext+learners%2C+2024%2E+1)
55. Johannes von Oswald, Eyvind Niklasson, E. Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. [Transformers learn in-context by gradient descent](https://md.research.elamur.ai/free/en/papers/oswald-2022-transformers-learn-context-gradient-descent/). In International Conference on Machine Learning, 2022. 1, 2, 4.1
56. Johannes von Oswald, Eyvind Niklasson, Maximilian Schlegel, Seijin Kobayashi, Nicolas Zucchet, Nino Scherrer, Nolan Miller, Mark Sandler, Blaise Agüera y Arcas, Max Vladymyrov, Razvan Pascanu, and Joao Sacramento. Uncovering mesa-optimization algorithms in transformers. ArXiv, abs/2309.05858, 2023.… — [arXiv:2309.05858](https://arxiv.org/abs/2309.05858) [find](https://scholar.google.com/scholar?q=Johannes+von+Oswald%2C+Eyvind+Niklasson%2C+Maximilian+Schlegel%2C+Seijin+Kobayashi%2C+Nicolas+Zucchet%2C+Nino+Scherrer%2C+Nolan+Miller%2C+Mark+Sandler%2C+Blaise+Ag%C3%BCera+y+Arcas%2C+Max+Vladymyrov%2C+Razvan+Pascanu%2C+and+Joa)
57. Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small, 2022. 2 [find](https://scholar.google.com/scholar?q=Kevin+Wang%2C+Alexandre+Variengien%2C+Arthur+Conmy%2C+Buck+Shlegeris%2C+and+Jacob+Steinhardt%2E+Interpretability+in+the+wild%3A+a+circuit+for+indirect+object+identification+in+gpt%2D2+small%2C+2022%2E+2)
58. Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, and Tengyu Ma. [Larger language models do in-context learning differently](https://md.research.elamur.ai/free/en/papers/wei-2023-larger-language-models-context-learning/), 2023. 2
59. Mitchell Wortsman, Jaehoon Lee, Justin Gilmer, and Simon Kornblith. Replacing softmax with relu in vision transformers, 2023. 3.2 [find](https://scholar.google.com/scholar?q=Mitchell+Wortsman%2C+Jaehoon+Lee%2C+Justin+Gilmer%2C+and+Simon+Kornblith%2E+Replacing+softmax+with+relu+in+vision+transformers%2C+2023%2E+3%2E2)
60. Kang Min Yoo, Junyeob Kim, Hyuhng Joon Kim, Hyunsoo Cho, Hwiyeol Jo, Sang-Woo Lee, Sang-goo Lee, and Taeuk Kim. [Ground-truth labels matter: A deeper look into input-label demonstrations](https://md.research.elamur.ai/free/en/papers/yoo-2022-ground-truth-labels-matter-deeper/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2422–2437, Abu D…
61. Ruiqi Zhang, Spencer Frei, and Peter L. Bartlett. Trained transformers learn linear models in-context. ArXiv, abs/2306.09927, 2023. 2 — [arXiv:2306.09927](https://arxiv.org/abs/2306.09927) [find](https://scholar.google.com/scholar?q=Ruiqi+Zhang%2C+Spencer+Frei%2C+and+Peter+L%2E+Bartlett%2E+Trained+transformers+learn+linear+models+in%2Dcontext%2E+ArXiv%2C+abs%2F2306%2E09927%2C+2023%2E+2+%E2%80%94+%5BarXiv%3A2306%2E09927%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2306%2E09927%29)
62. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. [Calibrate before use: Improving few-shot performance of language models](https://md.research.elamur.ai/free/en/papers/zhao-2021-calibrate-before-use-improving-few/). In International Conference on Machine Learning, pages 12697–12706. PMLR, 2021. 2

*Updated 3 November 2026*