---
title: "Transformers learn in-context by gradient — Oswald 2022"
description: "It is still not well understood how Transformers learn from examples placed directly in their input."
authors: ["Johannes von Oswald", "Eyvind Niklasson", "Ettore Randazzo", "Joao Sacramento", "Alexander Mordvintsev", "Andrey Zhmoginov", "Max Vladymyrov"]
year: 2022
venue: "arXiv:2212.07677"
source: "https://arxiv.org/abs/2212.07677"
doi: "10.48550/arxiv.2212.07677"
arxiv: "2212.07677"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/oswald-2022-transformers-learn-context-gradient-descent/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/oswald-2022-transformers-learn-context-gradient-descent/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) / Transformers learn in-context by gradient descent

# Transformers learn in-context by gradient descent

**Authors:** Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, Joao Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, Max Vladymyrov · **Year:** 2022 · **Venue:** arXiv:2212.07677 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2212.07677) · [PDF](https://arxiv.org/pdf/2212.07677) · [DOI 10.48550/arxiv.2212.07677](https://doi.org/10.48550/arxiv.2212.07677) · [arXiv 2212.07677](https://arxiv.org/abs/2212.07677) · [OpenAlex](https://openalex.org/W4311726128)

**Citations:** 87 citations (openalex-snapshot · 2026-10-14) · [149](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
It is still not well understood how Transformers learn from examples placed directly in their input. The authors argue that, at least for regression, this in-context learning amounts to running gradient descent inside the model's forward pass. First, they write down explicit weights for a linear self-attention layer, which is attention with the softmax removed. They show that its effect on the tokens equals one gradient-descent (GD) step on a squared-error loss for a linear model. In this view, the step changes the regression targets rather than the model weights. Next, they train small self-attention-only Transformers on synthetic linear regression tasks. The trained weights either recover the hand-built construction or produce predictions that closely match GD-trained models, both in-distribution and out-of-distribution. Deeper models learn GD++, a variant with curvature correction that beats plain GD. Adding MLP blocks lets the model solve non-linear tasks by fitting a linear model on learned features. The result is a concrete, testable account of trained Transformers acting as mesa-optimizers.

## Key points
- With the constructed weights, one linear self-attention layer performs exactly one GD step on a mean squared error loss. A stack of K layers lines up with K steps of GD.
- In multi-layer trained models, the learned updates match GD++, an accelerated form of GD that corrects for curvature step by step and outperforms vanilla GD.
- On sine-wave regression, Transformers with MLPs behave like a meta-learned MLP whose output layer gets one GD step. Their predictions and partial derivatives closely align with that baseline.
- Early layers appear to copy neighbouring tokens into input–target pairs, which then feed the later layers that perform GD. The authors relate this to the induction heads described by Olsson et al. (2022).
- The paper appeared at ICML 2023 (PMLR 202), with notebooks in google-research/self-organising-systems. Its scope is small models and simple, noise-free regression problems.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

149 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 142 not yet.

- [Many-Shot In-Context Learning](https://md.research.elamur.ai/free/en/papers/agarwal-2024-many-shot-context-learning/)
- [A Survey on In-context Learning](https://md.research.elamur.ai/free/en/papers/dong-2023-survey-context-learning/)
- [Challenges and Applications of Large Language Models](https://md.research.elamur.ai/free/en/papers/kaddour-2023-challenges-applications-large-language-models/)
- [Language Models can Solve Computer Tasks](https://md.research.elamur.ai/free/en/papers/kim-2023-language-models-solve-computer-tasks/)
- [Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning](https://md.research.elamur.ai/free/en/papers/wang-2023-label-words-anchors-information-flow/)
- [Larger language models do in-context learning differently](https://md.research.elamur.ai/free/en/papers/wei-2023-larger-language-models-context-learning/)
- [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{oswald2022transformers,
  title = {{Transformers learn in-context by gradient descent}},
  author = {Oswald, Johannes von and Niklasson, Eyvind and Randazzo, Ettore and Sacramento, Joao and Mordvintsev, Alexander and Zhmoginov, Andrey and Vladymyrov, Max},
  year = {2022},
  doi = {10.48550/arxiv.2212.07677},
  eprint = {2212.07677},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2212.07677}
}
```

- [← previous Transformer Memory as a Differentiable Search Index](https://md.research.elamur.ai/free/en/papers/tay-2022-transformer-memory-differentiable-search-index/)
- [next → TRUE: Re-evaluating Factual Consistency Evaluation](https://md.research.elamur.ai/free/en/papers/honovich-2022-true-re-evaluating-factual-consistency/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/)

## References

56 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Akyürek, E., Schuurmans, D., Andreas, J., Ma, T., and Zhou, D. What learning algorithm is in-context learning? investigations with linear models. In The Eleventh International Conference on Learning Representations, 2023. URL [https://openreview.net/forum?id=0g0X4H8yN4I](https://openreview.net/forum?id=0g0X4H8yN4I). [↗](https://openreview.net/forum?id=0g0X4H8yN4I) [our card](https://md.research.elamur.ai/free/en/papers/akyurek-2022-learning-algorithm-context-learning-investigations/)
2. Amos, B. and Kolter, J. Z. Optnet: Differentiable optimization as a layer in neural networks. In International Conference on Machine Learning, 2017. [find](https://scholar.google.com/scholar?q=Amos%2C+B%2E+and+Kolter%2C+J%2E+Z%2E+Optnet%3A+Differentiable+optimization+as+a+layer+in+neural+networks%2E+In+International+Conference+on+Machine+Learning%2C+2017%2E)
3. Andrychowicz, M., Denil, M., Gomez, S., Hoffman, M. W., Pfau, D., Schaul, T., Shillingford, B., and de Freitas, N. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems, 2016. [find](https://scholar.google.com/scholar?q=Andrychowicz%2C+M%2E%2C+Denil%2C+M%2E%2C+Gomez%2C+S%2E%2C+Hoffman%2C+M%2E+W%2E%2C+Pfau%2C+D%2E%2C+Schaul%2C+T%2E%2C+Shillingford%2C+B%2E%2C+and+de+Freitas%2C+N%2E+Learning+to+learn+by+gradient+descent+by+gradient+descent%2E+In+Advances+in+Neural+Info)
4. Ba, J., Hinton, G. E., Mnih, V., Leibo, J. Z., and Ionescu, C. Using fast weights to attend to the recent past. In Advances in Neural Information Processing Systems 29, 2016. [find](https://scholar.google.com/scholar?q=Ba%2C+J%2E%2C+Hinton%2C+G%2E+E%2E%2C+Mnih%2C+V%2E%2C+Leibo%2C+J%2E+Z%2E%2C+and+Ionescu%2C+C%2E+Using+fast+weights+to+attend+to+the+recent+past%2E+In+Advances+in+Neural+Information+Processing+Systems+29%2C+2016%2E)
5. Bai, S., Kolter, J. Z., and Koltun, V. Deep equilibrium models. Advances in Neural Information Processing Systems, 2019. [find](https://scholar.google.com/scholar?q=Bai%2C+S%2E%2C+Kolter%2C+J%2E+Z%2E%2C+and+Koltun%2C+V%2E+Deep+equilibrium+models%2E+Advances+in+Neural+Information+Processing+Systems%2C+2019%2E)
6. Bengio, Y., Bengio, S., and Cloutier, J. Learning a synaptic learning rule. Technical report, Université de Montréal, Département d’Informatique et de Recherche opérationnelle, 1990. [find](https://scholar.google.com/scholar?q=Bengio%2C+Y%2E%2C+Bengio%2C+S%2E%2C+and+Cloutier%2C+J%2E+Learning+a+synaptic+learning+rule%2E+Technical+report%2C+Universite%CC%81+de+Montre%CC%81al%2C+De%CC%81partement+d%E2%80%99Informatique+et+de+Recherche+ope%CC%81rationnelle%2C+1990%2E)
7. Benzing, F., Schug, S., Meier, R., von Oswald, J., Akram, Y., Zucchet, N., Aitchison, L., and Steger, A. Random initialisations performing above chance and how to find them. OPT2022: 14th Annual Workshop on Optimization for Machine Learning, 2022. [find](https://scholar.google.com/scholar?q=Benzing%2C+F%2E%2C+Schug%2C+S%2E%2C+Meier%2C+R%2E%2C+von+Oswald%2C+J%2E%2C+Akram%2C+Y%2E%2C+Zucchet%2C+N%2E%2C+Aitchison%2C+L%2E%2C+and+Steger%2C+A%2E+Random+initialisations+performing+above+chance+and+how+to+find+them%2E+OPT2022%3A+14th+Annual+Works)
8. Bertinetto, L., Henriques, J. F., Torr, P. H. S., and Vedaldi, A. Meta-learning with differentiable closed-form solvers. In International Conference on Learning Representations, 2019. [find](https://scholar.google.com/scholar?q=Bertinetto%2C+L%2E%2C+Henriques%2C+J%2E+F%2E%2C+Torr%2C+P%2E+H%2E+S%2E%2C+and+Vedaldi%2C+A%2E+Meta%2Dlearning+with+differentiable+closed%2Dform+solvers%2E+In+International+Conference+on+Learning+Representations%2C+2019%2E)
9. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Ch… [find](https://scholar.google.com/scholar?q=Brown%2C+T%2E+B%2E%2C+Mann%2C+B%2E%2C+Ryder%2C+N%2E%2C+Subbiah%2C+M%2E%2C+Kaplan%2C+J%2E%2C+Dhariwal%2C+P%2E%2C+Neelakantan%2C+A%2E%2C+Shyam%2C+P%2E%2C+Sastry%2C+G%2E%2C+Askell%2C+A%2E%2C+Agarwal%2C+S%2E%2C+Herbert%2DVoss%2C+A%2E%2C+Krueger%2C+G%2E%2C+Henighan%2C+T%2E%2C+Child%2C+R%2E%2C+Rames)
10. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. End-to-end object detection with transformers. In Computer Vision – ECCV 2020. [find](https://scholar.google.com/scholar?q=Carion%2C+N%2E%2C+Massa%2C+F%2E%2C+Synnaeve%2C+G%2E%2C+Usunier%2C+N%2E%2C+Kirillov%2C+A%2E%2C+and+Zagoruyko%2C+S%2E+End%2Dto%2Dend+object+detection+with+transformers%2E+In+Computer+Vision+%E2%80%93+ECCV+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/carion-2020-end-end-object-detection-transformers/)
11. Springer International Publishing, 2020. [find](https://scholar.google.com/scholar?q=Springer+International+Publishing%2C+2020%2E)
12. Chalmers, D. J. The evolution of learning: an experiment in genetic connectionism. In Touretzky, D. S., Elman, J. L., Sejnowski, T. J., and Hinton, G. E. (eds.), Connectionist Models, pp. 81–90. Morgan Kaufmann, 1991. [find](https://scholar.google.com/scholar?q=Chalmers%2C+D%2E+J%2E+The+evolution+of+learning%3A+an+experiment+in+genetic+connectionism%2E+In+Touretzky%2C+D%2E+S%2E%2C+Elman%2C+J%2E+L%2E%2C+Sejnowski%2C+T%2E+J%2E%2C+and+Hinton%2C+G%2E+E%2E+%28eds%2E%29%2C+Connectionist+Models%2C+pp%2E+81%E2%80%9390%2E+Morga)
13. Chan, S. C. Y., Dasgupta, I., Kim, J., Kumaran, D., Lampinen, A. K., and Hill, F. Transformers generalize differently from information stored in context vs in weights. arXiv preprint arXiv:2210.05675, 2022a. — [arXiv:2210.05675](https://arxiv.org/abs/2210.05675) [find](https://scholar.google.com/scholar?q=Chan%2C+S%2E+C%2E+Y%2E%2C+Dasgupta%2C+I%2E%2C+Kim%2C+J%2E%2C+Kumaran%2C+D%2E%2C+Lampinen%2C+A%2E+K%2E%2C+and+Hill%2C+F%2E+Transformers+generalize+differently+from+information+stored+in+context+vs+in+weights%2E+arXiv+preprint+arXiv%3A2210%2E05675%2C)
14. Chan, S. C. Y., Santoro, A., Lampinen, A. K., Wang, J. X., Singh, A., Richemond, P. H., McClelland, J., and Hill, F. Data distributional properties drive emergent in-context learning in transformers. Advances in Neural Information Processing Systems, 2022b. [find](https://scholar.google.com/scholar?q=Chan%2C+S%2E+C%2E+Y%2E%2C+Santoro%2C+A%2E%2C+Lampinen%2C+A%2E+K%2E%2C+Wang%2C+J%2E+X%2E%2C+Singh%2C+A%2E%2C+Richemond%2C+P%2E+H%2E%2C+McClelland%2C+J%2E%2C+and+Hill%2C+F%2E+Data+distributional+properties+drive+emergent+in%2Dcontext+learning+in+transformers%2E+)
15. Choromanski, K. M., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J. Q., Mohiuddin, A., Kaiser, L., Belanger, D. B., Colwell, L. J., and Weller, A. Rethinking attention with performers. In International Conference on Learning Representations, 2021. URL https://ope… [find](https://scholar.google.com/scholar?q=Choromanski%2C+K%2E+M%2E%2C+Likhosherstov%2C+V%2E%2C+Dohan%2C+D%2E%2C+Song%2C+X%2E%2C+Gane%2C+A%2E%2C+Sarlos%2C+T%2E%2C+Hawkins%2C+P%2E%2C+Davis%2C+J%2E+Q%2E%2C+Mohiuddin%2C+A%2E%2C+Kaiser%2C+L%2E%2C+Belanger%2C+D%2E+B%2E%2C+Colwell%2C+L%2E+J%2E%2C+and+Weller%2C+A%2E+Rethinking+atten)
16. Dai, D., Sun, Y., Dong, L., Hao, Y., Ma, S., Sui, Z., and Wei, F. Why can GPT learn in-context? language models implicitly perform gradient descent as meta-optimizers. In ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2023. URL [https://openreview.net/forum?id=f…](https://openreview.net/forum?id=f…) [↗](https://openreview.net/forum?id=f…)
17. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representatio… [find](https://scholar.google.com/scholar?q=Dosovitskiy%2C+A%2E%2C+Beyer%2C+L%2E%2C+Kolesnikov%2C+A%2E%2C+Weissenborn%2C+D%2E%2C+Zhai%2C+X%2E%2C+Unterthiner%2C+T%2E%2C+Dehghani%2C+M%2E%2C+Minderer%2C+M%2E%2C+Heigold%2C+G%2E%2C+Gelly%2C+S%2E%2C+Uszkoreit%2C+J%2E%2C+and+Houlsby%2C+N%2E+An+image+is+worth+16x16+words)
18. Entezari, R., Sedghi, H., Saukh, O., and Neyshabur, B. The role of permutation invariance in linear mode connectivity of neural networks. arXiv preprint arXiv:2110.06296, 2021. — [arXiv:2110.06296](https://arxiv.org/abs/2110.06296) [find](https://scholar.google.com/scholar?q=Entezari%2C+R%2E%2C+Sedghi%2C+H%2E%2C+Saukh%2C+O%2E%2C+and+Neyshabur%2C+B%2E+The+role+of+permutation+invariance+in+linear+mode+connectivity+of+neural+networks%2E+arXiv+preprint+arXiv%3A2110%2E06296%2C+2021%2E+%E2%80%94+%5BarXiv%3A2110%2E06296%5D%28ht)
19. Finn, C. and Levine, S. Meta-learning and universality: Deep representations and gradient descent can approximate any learning algorithm. In International Conference on Learning Representations, 2018. URL [https://openreview.net/forum?id=HyjC5yWCW](https://openreview.net/forum?id=HyjC5yWCW). [↗](https://openreview.net/forum?id=HyjC5yWCW)
20. Finn, C., Abbeel, P., and Levine, S. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, 2017. [find](https://scholar.google.com/scholar?q=Finn%2C+C%2E%2C+Abbeel%2C+P%2E%2C+and+Levine%2C+S%2E+Model%2Dagnostic+meta%2Dlearning+for+fast+adaptation+of+deep+networks%2E+In+International+Conference+on+Machine+Learning%2C+2017%2E)
21. Flennerhag, S., Rusu, A. A., Pascanu, R., Visin, F., Yin, H., and Hadsell, R. Meta-learning with warped gradient descent. In International Conference on Learning Representations, 2020. [find](https://scholar.google.com/scholar?q=Flennerhag%2C+S%2E%2C+Rusu%2C+A%2E+A%2E%2C+Pascanu%2C+R%2E%2C+Visin%2C+F%2E%2C+Yin%2C+H%2E%2C+and+Hadsell%2C+R%2E+Meta%2Dlearning+with+warped+gradient+descent%2E+In+International+Conference+on+Learning+Representations%2C+2020%2E)
22. Garg, S., Tsipras, D., Liang, P., and Valiant, G. What can transformers learn in-context? a case study of simple function classes. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL [https://openreview.net/forum?id=flNZJ2eOet](https://openreview.net/forum?id=flNZJ2eOet). [↗](https://openreview.net/forum?id=flNZJ2eOet)
23. Gordon, J., Bronskill, J., Bauer, M., Nowozin, S., and Turner, R. Meta-learning probabilistic inference for prediction. In International Conference on Learning Representations, 2019. URL [https://openreview.net/forum?id=HkxStoC5F7](https://openreview.net/forum?id=HkxStoC5F7). [↗](https://openreview.net/forum?id=HkxStoC5F7)
24. Gould, S., Hartley, R., and Campbell, D. J. Deep declarative networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021. [find](https://scholar.google.com/scholar?q=Gould%2C+S%2E%2C+Hartley%2C+R%2E%2C+and+Campbell%2C+D%2E+J%2E+Deep+declarative+networks%2E+IEEE+Transactions+on+Pattern+Analysis+and+Machine+Intelligence%2C+2021%2E)
25. Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., and Pang, R. Conformer: Convolution-augmented transformer for speech recognition. arXiv preprint arXiv:2005.08100, 2020. [find](https://scholar.google.com/scholar?q=Gulati%2C+A%2E%2C+Qin%2C+J%2E%2C+Chiu%2C+C%2E%2DC%2E%2C+Parmar%2C+N%2E%2C+Zhang%2C+Y%2E%2C+Yu%2C+J%2E%2C+Han%2C+W%2E%2C+Wang%2C+S%2E%2C+Zhang%2C+Z%2E%2C+Wu%2C+Y%2E%2C+and+Pang%2C+R%2E+Conformer%3A+Convolution%2Daugmented+transformer+for+speech+recognition%2E+arXiv+preprint+) [our card](https://md.research.elamur.ai/free/en/papers/gulati-2020-conformer-convolution-augmented-transformer-speech/)
26. Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016. [find](https://scholar.google.com/scholar?q=Hendrycks%2C+D%2E+and+Gimpel%2C+K%2E+Gaussian+error+linear+units+%28gelus%29%2E+arXiv+preprint+arXiv%3A1606%2E08415%2C+2016%2E)
27. Hinton, G. E. and Plaut, D. C. Using fast weights to deblur old memories. 1987. [find](https://scholar.google.com/scholar?q=Hinton%2C+G%2E+E%2E+and+Plaut%2C+D%2E+C%2E+Using+fast+weights+to+deblur+old+memories%2E+1987%2E)
28. Hochreiter, S., Younger, A. S., and Conwell, P. R. Learning to learn using gradient descent. In Dorffner, G., Bischof, H., and Hornik, K. (eds.), Artificial Neural Networks — ICANN 2001, pp. 87–94, Berlin, Heidelberg, 2001. Springer Berlin Heidelberg. ISBN 978-3-540-44668-2. [find](https://scholar.google.com/scholar?q=Hochreiter%2C+S%2E%2C+Younger%2C+A%2E+S%2E%2C+and+Conwell%2C+P%2E+R%2E+Learning+to+learn+using+gradient+descent%2E+In+Dorffner%2C+G%2E%2C+Bischof%2C+H%2E%2C+and+Hornik%2C+K%2E+%28eds%2E%29%2C+Artificial+Neural+Networks+%E2%80%94+ICANN+2001%2C+pp%2E+87%E2%80%9394%2C+Be)
29. Hubinger, E., van Merwijk, C., Mikulik, V., Skalse, J., and Garrabrant, S. Risks from learned optimization in advanced machine learning systems. arXiv [cs.AI], Jun 2019. URL http://arxiv.org/abs/1906.01820. — [arXiv:1906.01820](https://arxiv.org/abs/1906.01820) [↗](http://arxiv.org/abs/1906.01820)
30. Irie, K., Schlag, I., Csordás, R., and Schmidhuber, J. Going beyond linear transformers with recurrent fast weight programmers. CoRR, abs/2106.06295, 2021. URL https://arxiv.org/abs/2106.06295. — [arXiv:2106.06295](https://arxiv.org/abs/2106.06295) [↗](https://arxiv.org/abs/2106.06295)
31. Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization, 2014. [find](https://scholar.google.com/scholar?q=Kingma%2C+D%2E+P%2E+and+Ba%2C+J%2E+Adam%3A+A+method+for+stochastic+optimization%2C+2014%2E)
32. Kirsch, L. and Schmidhuber, J. Meta learning backpropagation and improving it. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021. URL [https://openreview.net/forum?id=hhU9TEvB6AF](https://openreview.net/forum?id=hhU9TEvB6AF). [↗](https://openreview.net/forum?id=hhU9TEvB6AF)
33. Kirsch, L., Harrison, J., Sohl-Dickstein, J., and Metz, L. General-purpose in-context learning by meta-learning transformers. In Sixth Workshop on Meta-Learning at the Conference on Neural Information Processing Systems, 2022. URL [https://openreview.net/forum?id=t6tA-KB4dO](https://openreview.net/forum?id=t6tA-KB4dO). [↗](https://openreview.net/forum?id=t6tA-KB4dO)
34. Lee, K., Maji, S., Ravichandran, A., and Soatto, S. Meta-learning with differentiable convex optimization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019. [find](https://scholar.google.com/scholar?q=Lee%2C+K%2E%2C+Maji%2C+S%2E%2C+Ravichandran%2C+A%2E%2C+and+Soatto%2C+S%2E+Meta%2Dlearning+with+differentiable+convex+optimization%2E+In+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+2019%2E)
35. Lee, Y. and Choi, S. Gradient-based meta-learning with learned layerwise metric and subspace. In International Conference on Machine Learning, 2018. [find](https://scholar.google.com/scholar?q=Lee%2C+Y%2E+and+Choi%2C+S%2E+Gradient%2Dbased+meta%2Dlearning+with+learned+layerwise+metric+and+subspace%2E+In+International+Conference+on+Machine+Learning%2C+2018%2E)
36. Li, Z., Zhou, F., Chen, F., and Li, H. Meta-SGD: Learning to learn quickly for few shot learning. arXiv preprint arXiv:1707.09835, 2017. — [arXiv:1707.09835](https://arxiv.org/abs/1707.09835) [find](https://scholar.google.com/scholar?q=Li%2C+Z%2E%2C+Zhou%2C+F%2E%2C+Chen%2C+F%2E%2C+and+Li%2C+H%2E+Meta%2DSGD%3A+Learning+to+learn+quickly+for+few+shot+learning%2E+arXiv+preprint+arXiv%3A1707%2E09835%2C+2017%2E+%E2%80%94+%5BarXiv%3A1707%2E09835%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1707%2E09835%29)
37. Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586, 2021. [find](https://scholar.google.com/scholar?q=Liu%2C+P%2E%2C+Yuan%2C+W%2E%2C+Fu%2C+J%2E%2C+Jiang%2C+Z%2E%2C+Hayashi%2C+H%2E%2C+and+Neubig%2C+G%2E+Pre%2Dtrain%2C+prompt%2C+and+predict%3A+A+systematic+survey+of+prompting+methods+in+natural+language+processing%2E+arXiv+preprint+arXiv%3A2107%2E135) [our card](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/)
38. Nadaraya, E. A. On estimating regression. Theory of Probability & its Applications, 9(1):141–142, 1964. [find](https://scholar.google.com/scholar?q=Nadaraya%2C+E%2E+A%2E+On+estimating+regression%2E+Theory+of+Probability+%26+its+Applications%2C+9%281%29%3A141%E2%80%93142%2C+1964%2E)
39. Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Johnston, S., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McC… [find](https://scholar.google.com/scholar?q=Olsson%2C+C%2E%2C+Elhage%2C+N%2E%2C+Nanda%2C+N%2E%2C+Joseph%2C+N%2E%2C+DasSarma%2C+N%2E%2C+Henighan%2C+T%2E%2C+Mann%2C+B%2E%2C+Askell%2C+A%2E%2C+Bai%2C+Y%2E%2C+Chen%2C+A%2E%2C+Conerly%2C+T%2E%2C+Drain%2C+D%2E%2C+Ganguli%2C+D%2E%2C+Hatfield%2DDodds%2C+Z%2E%2C+Hernandez%2C+D%2E%2C+Johnston%2C+S%2E)
40. Park, E. and Oliva, J. B. Meta-curvature. In Advances in Neural Information Processing Systems, 2019. [find](https://scholar.google.com/scholar?q=Park%2C+E%2E+and+Oliva%2C+J%2E+B%2E+Meta%2Dcurvature%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2019%2E)
41. Power, A., Burda, Y., Edwards, H., Babuschkin, I., and Misra, V. Grokking: Generalization beyond overfitting on small algorithmic datasets. abs/2201.02177, 2022. [find](https://scholar.google.com/scholar?q=Power%2C+A%2E%2C+Burda%2C+Y%2E%2C+Edwards%2C+H%2E%2C+Babuschkin%2C+I%2E%2C+and+Misra%2C+V%2E+Grokking%3A+Generalization+beyond+overfitting+on+small+algorithmic+datasets%2E+abs%2F2201%2E02177%2C+2022%2E)
42. Raghu, A., Raghu, M., Bengio, S., and Vinyals, O. Rapid learning or feature reuse? Towards understanding the effectiveness of MAML. In International Conference on Learning Representations, 2020. [find](https://scholar.google.com/scholar?q=Raghu%2C+A%2E%2C+Raghu%2C+M%2E%2C+Bengio%2C+S%2E%2C+and+Vinyals%2C+O%2E+Rapid+learning+or+feature+reuse%3F+Towards+understanding+the+effectiveness+of+MAML%2E+In+International+Conference+on+Learning+Representations%2C+2020%2E)
43. Ramsauer, H., Schäfl, B., Lehner, J., Seidl, P., Widrich, M., Adler, T., Gruber, L., Holzleitner, M., Pavlović, M., Sandve, G. K., Greiff, V., Kreil, D., Kopp, M., Klambauer, G., Brandstetter, J., and Hochreiter, S. Hopfield networks is all you need. arXiv preprint arXiv:2008.02217, 2020. [find](https://scholar.google.com/scholar?q=Ramsauer%2C+H%2E%2C+Scha%CC%88fl%2C+B%2E%2C+Lehner%2C+J%2E%2C+Seidl%2C+P%2E%2C+Widrich%2C+M%2E%2C+Adler%2C+T%2E%2C+Gruber%2C+L%2E%2C+Holzleitner%2C+M%2E%2C+Pavlovic%CC%81%2C+M%2E%2C+Sandve%2C+G%2E+K%2E%2C+Greiff%2C+V%2E%2C+Kreil%2C+D%2E%2C+Kopp%2C+M%2E%2C+Klambauer%2C+G%2E%2C+Brandstetter%2C+J%2E%2C+a)
44. Rusu, A. A., Rao, D., Sygnowski, J., Vinyals, O., Pascanu, R., Osindero, S., and Hadsell, R. Meta-learning with latent embedding optimization. In International Conference on Learning Representations, 2019. [find](https://scholar.google.com/scholar?q=Rusu%2C+A%2E+A%2E%2C+Rao%2C+D%2E%2C+Sygnowski%2C+J%2E%2C+Vinyals%2C+O%2E%2C+Pascanu%2C+R%2E%2C+Osindero%2C+S%2E%2C+and+Hadsell%2C+R%2E+Meta%2Dlearning+with+latent+embedding+optimization%2E+In+International+Conference+on+Learning+Representations%2C+)
45. Schlag, I., Irie, K., and Schmidhuber, J. Linear transformers are secretly fast weight programmers. In ICML, 2021. [find](https://scholar.google.com/scholar?q=Schlag%2C+I%2E%2C+Irie%2C+K%2E%2C+and+Schmidhuber%2C+J%2E+Linear+transformers+are+secretly+fast+weight+programmers%2E+In+ICML%2C+2021%2E)
46. Schmidhuber, J. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-... hook. Diploma thesis, Institut für Informatik, Technische Universität München, 1987. [find](https://scholar.google.com/scholar?q=Schmidhuber%2C+J%2E+Evolutionary+principles+in+self%2Dreferential+learning%2C+or+on+learning+how+to+learn%3A+the+meta%2Dmeta%2D%2E%2E%2E+hook%2E+Diploma+thesis%2C+Institut+fu%CC%88r+Informatik%2C+Technische+Universita%CC%88t+Mu%CC%88nchen%2C+1)
47. Schmidhuber, J. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131–139, 1992. doi: 10.1162/neco.1992.4.1.131. — [doi:10.1162/neco.1992.4.1.131](https://doi.org/10.1162/neco.1992.4.1.131) [find](https://scholar.google.com/scholar?q=Schmidhuber%2C+J%2E+Learning+to+control+fast%2Dweight+memories%3A+An+alternative+to+dynamic+recurrent+networks%2E+Neural+Computation%2C+4%281%29%3A131%E2%80%93139%2C+1992%2E+doi%3A+10%2E1162%2Fneco%2E1992%2E4%2E1%2E131%2E+%E2%80%94+%5Bdoi%3A10%2E1162%2Fneco%2E1992)
48. Thrun, S. and Pratt, L. Learning to learn. Springer US, 1998. [find](https://scholar.google.com/scholar?q=Thrun%2C+S%2E+and+Pratt%2C+L%2E+Learning+to+learn%2E+Springer+US%2C+1998%2E)
49. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need, 2017. von Oswald, J., Zhao, D., Kobayashi, S., Schug, S., Caccia, M., Zucchet, N., and Sacramento, J. Learning where to learn: Gradient sparsity in meta and continu… [find](https://scholar.google.com/scholar?q=Vaswani%2C+A%2E%2C+Shazeer%2C+N%2E%2C+Parmar%2C+N%2E%2C+Uszkoreit%2C+J%2E%2C+Jones%2C+L%2E%2C+Gomez%2C+A%2E+N%2E%2C+Kaiser%2C+L%2E%2C+and+Polosukhin%2C+I%2E+Attention+is+all+you+need%2C+2017%2E+von+Oswald%2C+J%2E%2C+Zhao%2C+D%2E%2C+Kobayashi%2C+S%2E%2C+Schug%2C+S%2E%2C+Caccia)
50. Watson, G. S. Smooth regression analysis. Sankhyā: The Indian Journal of Statistics, Series A, pp. 359–372, 1964. [find](https://scholar.google.com/scholar?q=Watson%2C+G%2E+S%2E+Smooth+regression+analysis%2E+Sankhya%CC%84%3A+The+Indian+Journal+of+Statistics%2C+Series+A%2C+pp%2E+359%E2%80%93372%2C+1964%2E)
51. Widrow, B. and Hoff, M. E. Adaptive switching circuits. In 1960 IRE WESCON Convention Record, Part 4, pp. 96–104, New York, 1960. IRE. [find](https://scholar.google.com/scholar?q=Widrow%2C+B%2E+and+Hoff%2C+M%2E+E%2E+Adaptive+switching+circuits%2E+In+1960+IRE+WESCON+Convention+Record%2C+Part+4%2C+pp%2E+96%E2%80%93104%2C+New+York%2C+1960%2E+IRE%2E)
52. Yun, S., Jeong, M., Kim, R., Kang, J., and Kim, H. J. Graph transformer networks. In Wallach, H., Larochelle, H., Beygelzimer, A., dÁlché-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, 2019. [find](https://scholar.google.com/scholar?q=Yun%2C+S%2E%2C+Jeong%2C+M%2E%2C+Kim%2C+R%2E%2C+Kang%2C+J%2E%2C+and+Kim%2C+H%2E+J%2E+Graph+transformer+networks%2E+In+Wallach%2C+H%2E%2C+Larochelle%2C+H%2E%2C+Beygelzimer%2C+A%2E%2C+dA%CC%81lche%CC%81%2DBuc%2C+F%2E%2C+Fox%2C+E%2E%2C+and+Garnett%2C+R%2E+%28eds%2E%29%2C+Advances+in+Neural)
53. Zhang, A., Lipton, Z. C., Li, M., and Smola, A. J. Dive into deep learning. arXiv preprint arXiv:2106.11342, 2021. [find](https://scholar.google.com/scholar?q=Zhang%2C+A%2E%2C+Lipton%2C+Z%2E+C%2E%2C+Li%2C+M%2E%2C+and+Smola%2C+A%2E+J%2E+Dive+into+deep+learning%2E+arXiv+preprint+arXiv%3A2106%2E11342%2C+2021%2E)
54. Zhao, D., Kobayashi, S., Sacramento, J., and von Oswald, J. Meta-learning via hypernetworks. In NeurIPS Workshop on Meta-Learning, 2020. [find](https://scholar.google.com/scholar?q=Zhao%2C+D%2E%2C+Kobayashi%2C+S%2E%2C+Sacramento%2C+J%2E%2C+and+von+Oswald%2C+J%2E+Meta%2Dlearning+via+hypernetworks%2E+In+NeurIPS+Workshop+on+Meta%2DLearning%2C+2020%2E)
55. Zhmoginov, A., Sandler, M., and Vladymyrov, M. Hy-perTransformer: Model generation for supervised and semi-supervised few-shot learning. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learning, volu… [find](https://scholar.google.com/scholar?q=Zhmoginov%2C+A%2E%2C+Sandler%2C+M%2E%2C+and+Vladymyrov%2C+M%2E+Hy%2DperTransformer%3A+Model+generation+for+supervised+and+semi%2Dsupervised+few%2Dshot+learning%2E+In+Chaudhuri%2C+K%2E%2C+Jegelka%2C+S%2E%2C+Song%2C+L%2E%2C+Szepesvari%2C+C%2E%2C+Niu%2C+G)
56. Zucchet, N. and Sacramento, J. Beyond backpropagation: bilevel optimization through implicit differentiation and equilibrium propagation. Neural Computation, 34(12), December 2022. [find](https://scholar.google.com/scholar?q=Zucchet%2C+N%2E+and+Sacramento%2C+J%2E+Beyond+backpropagation%3A+bilevel+optimization+through+implicit+differentiation+and+equilibrium+propagation%2E+Neural+Computation%2C+34%2812%29%2C+December+2022%2E)