---
title: "Can Mamba Learn How to Learn? A Comparative Study on"
description: "To combine both strengths, the authors propose MambaFormer, a hybrid that substitutes Mamba… 59 references, cited by 31 papers here."
authors: ["Jongho Park", "Jaeseung Park", "Zheyang Xiong", "Nayoung Lee", "Jaewoong Cho", "Samet Oymak", "Kangwook Lee", "Dimitris Papailiopoulos"]
year: 2024
venue: "arXiv:2402.04248"
source: "https://arxiv.org/abs/2402.04248"
doi: "10.48550/arxiv.2402.04248"
arxiv: "2402.04248"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/park-2024-mamba-learn-learn-comparative-study/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/park-2024-mamba-learn-learn-comparative-study/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/) / Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks

# Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks

**Authors:** Jongho Park, Jaeseung Park, Zheyang Xiong, [Nayoung Lee](https://md.research.elamur.ai/free/en/authors/nayoung-lee/), [Jaewoong Cho](https://md.research.elamur.ai/free/en/authors/jaewoong-cho/), [Samet Oymak](https://md.research.elamur.ai/free/en/authors/samet-oymak/), [Kangwook Lee](https://md.research.elamur.ai/free/en/authors/kangwook-lee/) and [Dimitris Papailiopoulos](https://md.research.elamur.ai/free/en/authors/dimitris-papailiopoulos/) · **Year:** 2024 · **Venue:** arXiv:2402.04248 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2402.04248) · [PDF](https://arxiv.org/pdf/2402.04248) · [DOI 10.48550/arxiv.2402.04248](https://doi.org/10.48550/arxiv.2402.04248) · [arXiv 2402.04248](https://arxiv.org/abs/2402.04248) · [OpenAlex](https://openalex.org/W4391632210)

**Citations:** 10 citations (OpenAlex, 2026-10-14) · [31](https://md.research.elamur.ai/free/en/papers/park-2024-mamba-learn-learn-comparative-study/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
In-context learning (ICL) has mostly been studied in Transformers. This paper asks whether attention-free state-space models (SSMs), chiefly Mamba and also S4, can acquire the same ability. Large-scale ICL is hard to test cheaply, so the authors follow the Garg et al. (2022) setup: small models are trained from scratch to solve synthetic function classes from in-prompt examples. The suite covers several regression variants, decision trees, sparse parity, Chain-of-Thought I/O and vector-valued MQAR (a retrieval task). It is complemented by formal-language benchmarks (GINC, RegBench).

Mamba roughly matches Transformers on standard regression and solves sparse parity, which Transformers fail. However, it lags on decision trees and breaks down on MQAR retrieval. To combine both strengths, the authors propose MambaFormer, a hybrid that substitutes Mamba blocks for MLP blocks, begins with a Mamba layer and needs no positional encoding. It performs competitively across every task. The work suggests SSMs face no fundamental barrier to ICL apart from retrieval, and it points to hybrids as a promising design direction.

## Key points
- Nine synthetic ICL tasks (e.g., linear, sparse linear and 2NN regression, decision tree, outlier regression, sparse parity, vector MQAR) were used. Models were mostly trained for 500,000 iterations.
- On sparse parity, Transformers did no better than random guessing, while Mamba succeeded. On vector-valued MQAR, Mamba could not reliably retrieve values.
- On RegBench after 120 epochs, accuracy was: Transformer 98.9, Mamba 87.8, MambaFormer 99.8, Standard Hybrid 99.9. The hybrids also converged faster.
- On GINC, ICL accuracy was: LSTM 96.4, Mamba 87.1, Standard Hybrid 85.0, Transformer 84.2, MambaFormer 79.6.
- Code is available at github.com/krafton-ai/mambaformer-icl. The authors note that the study is limited to small models and mostly non-language tasks.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Michigan](https://md.research.elamur.ai/free/en/universities/university-of-michigan/) — University, Ann Arbor, United States · 94 works · 18,548 citations
- [Seoul National University](https://md.research.elamur.ai/free/en/universities/seoul-national-university/) — University, Seoul, South Korea · 47 works · 46,072 citations
- [University of Wisconsin–Madison](https://md.research.elamur.ai/free/en/universities/university-of-wisconsin-madison/) — University, Madison, United States · 60 works · 49,213 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 4,145 works

## Cite

Cite as: Jongho Park, Jaeseung Park, Zheyang Xiong, Nayoung Lee, Jaewoong Cho, Samet Oymak, Kangwook Lee and Dimitris Papailiopoulos. 2024. Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks. arXiv:2402.04248.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{park2024can,
  title = {{Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks}},
  author = {Jongho Park, Jaeseung Park, Zheyang Xiong, Nayoung Lee, Jaewoong Cho, Samet Oymak, Kangwook Lee and Dimitris Papailiopoulos},
  year = {2024},
  doi = {10.48550/arxiv.2402.04248},
  eprint = {2402.04248},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.04248}
}
```

- [← previous Auxiliary task demands mask the capabilities of smaller language models](https://md.research.elamur.ai/free/en/papers/hu-2024-auxiliary-task-demands-mask-capabilities/)
- [next → Do Language Models Enjoy Their Own Stories? Prompting Large Language Models for Automatic Story Evaluation](https://md.research.elamur.ai/free/en/papers/chhun-2024-do-language-models-enjoy-their/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/)

## References

59 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ahn, K., Cheng, X., Daneshmand, H., and Sra, S. [Transformers learn to implement preconditioned gradient descent for in-context learning](https://md.research.elamur.ai/free/en/papers/ahn-2023-transformers-learn-implement-preconditioned-gradient/). arXiv preprint arXiv:2306.00297, 2023. 3
2. Akyürek, E., Schuurmans, D., Andreas, J., Ma, T., and Zhou, D. [What learning algorithm is in-context learning? investigations with linear models](https://md.research.elamur.ai/free/en/papers/akyurek-2022-learning-algorithm-context-learning-investigations/). In The Eleventh International Conference on Learning Representations, 2022. 1, 3
3. Akyürek, E., Wang, B., Kim, Y., and Andreas, J. In-context language learning: Architectures and algorithms. arXiv preprint arXiv:2401.12973, 2024. URL https://arxiv.org/abs/2401. 12973. 3, 4, 6, 13, 15 [↗](https://arxiv.org/abs/2401)
4. Arora, S., Eyuboglu, S., Timalsina, A., Johnson, I., Poli, M., Zou, J., Rudra, A., and Ré, C. [Zoology: Measuring and improving recall in efficient language models](https://md.research.elamur.ai/free/en/papers/arora-2023-zoology-measuring-improving-recall-efficient/). arXiv preprint arXiv:2312.04927, 2023. URL https://arxiv.org/abs/2312.04927. 3, 4, 6, 10 [↗](https://arxiv.org/abs/2312.04927)
5. Bai, Y., Chen, F., Wang, H., Xiong, C., and Mei, S. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. arXiv preprint arXiv:2306.04637, 2023. 1 — [arXiv:2306.04637](https://arxiv.org/abs/2306.04637) [find](https://scholar.google.com/scholar?q=Bai%2C+Y%2E%2C+Chen%2C+F%2E%2C+Wang%2C+H%2E%2C+Xiong%2C+C%2E%2C+and+Mei%2C+S%2E+Transformers+as+statisticians%3A+Provable+in%2Dcontext+learning+with+in%2Dcontext+algorithm+selection%2E+arXiv+preprint+arXiv%3A2306%2E04637%2C+2023%2E+1+%E2%80%94+%5BarXiv%3A2)
6. Barak, B., Edelman, B., Goel, S., Kakade, S., Malach, E., and Zhang, C. Hidden progress in deep learning: Sgd learns parities near the computational limit. Advances in Neural Information Processing Systems, 35:21750–21764, 2022. 12 [find](https://scholar.google.com/scholar?q=Barak%2C+B%2E%2C+Edelman%2C+B%2E%2C+Goel%2C+S%2E%2C+Kakade%2C+S%2E%2C+Malach%2C+E%2E%2C+and+Zhang%2C+C%2E+Hidden+progress+in+deep+learning%3A+Sgd+learns+parities+near+the+computational+limit%2E+Advances+in+Neural+Information+Processing+Sy)
7. Beltagy, I., Peters, M. E., and Cohan, A. [Longformer: The long-document transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/). arXiv preprint arXiv:2004.05150, 2020. 3
8. Bhattamishra, S., Ahuja, K., and Goyal, N. [On the ability and limitations of transformers to recognize formal languages](https://md.research.elamur.ai/free/en/papers/bhattamishra-2020-ability-limitations-transformers-recognize-formal/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7096–7116. Association for Computational Linguistics, 2020. doi: 10.18653/v1…
9. Bhattamishra, S., Patel, A., Blunsom, P., and Kanade, V. Understanding in-context learning in transformers and llms by learning to learn discrete functions. arXiv preprint arXiv:2310.03016, 2023. URL https://arxiv.org/abs/2310.03016. 3, 6, 10, 12, 24 — [arXiv:2310.03016](https://arxiv.org/abs/2310.03016) [↗](https://arxiv.org/abs/2310.03016)
10. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020. 1
11. Chan, S., Santoro, A., Lampinen, A., Wang, J., Singh, A., Richemond, P., McClelland, J., and Hill, F. Data distributional properties drive emergent in-context learning in transformers. Advances in Neural Information Processing Systems, 35:18878–18891, 2022. 1 [find](https://scholar.google.com/scholar?q=Chan%2C+S%2E%2C+Santoro%2C+A%2E%2C+Lampinen%2C+A%2E%2C+Wang%2C+J%2E%2C+Singh%2C+A%2E%2C+Richemond%2C+P%2E%2C+McClelland%2C+J%2E%2C+and+Hill%2C+F%2E+Data+distributional+properties+drive+emergent+in%2Dcontext+learning+in+transformers%2E+Advances+in+Neu)
12. Dai, D., Sun, Y., Dong, L., Hao, Y., Ma, S., Sui, Z., and Wei, F. [Why can gpt learn in-context? language models secretly perform gradient descent as meta-optimizers](https://md.research.elamur.ai/free/en/papers/dai-2023-why-can-gpt-learn-context/). In Findings of the Association for Computational Linguistics: ACL 2023, pp. 4005–4019, 2023. 3
13. Dao, T., Fu, D. Y., Saab, K. K., Thomas, A. W., Rudra, A., and Ré, C. Hungry hungry hippos: Towards language modeling with state space models. arXiv preprint arXiv:2212.14052, 2022. URL https://arxiv.org/abs/2212.14052. 2, 4 [↗](https://arxiv.org/abs/2212.14052)
14. Deletang, G., Ruoss, A., Grau-Moya, J., Genewein, T., Wenliang, L. K., Catt, E., Cundy, C., Hutter, M., Legg, S., Veness, J., et al. [Neural networks and the chomsky hierarchy](https://md.research.elamur.ai/free/en/papers/deletang-2022-neural-networks-chomsky-hierarchy/). In The Eleventh International Conference on Learning Representations, 2022. 14
15. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. The pile: An 800gb dataset of diverse text for language modeling, 2020. 2 [find](https://scholar.google.com/scholar?q=Gao%2C+L%2E%2C+Biderman%2C+S%2E%2C+Black%2C+S%2E%2C+Golding%2C+L%2E%2C+Hoppe%2C+T%2E%2C+Foster%2C+C%2E%2C+Phang%2C+J%2E%2C+He%2C+H%2E%2C+Thite%2C+A%2E%2C+Nabeshima%2C+N%2E%2C+Presser%2C+S%2E%2C+and+Leahy%2C+C%2E+The+pile%3A+An+800gb+dataset+of+diverse+text+for+language+mo)
16. Garg, S., Tsipras, D., Liang, P. S., and Valiant, G. [What can transformers learn in-context? a case study of simple function classes](https://md.research.elamur.ai/free/en/papers/garg-2022-transformers-learn-context-case-study/). Advances in Neural Information Processing Systems, 35: 30583–30598, 2022. 1, 2, 3, 5, 20, 24
17. Giannou, A., Rajput, S., Sohn, J.-y., Lee, K., Lee, J. D., and Papailiopoulos, D. Looped transformers as programmable computers. In International Conference on Machine Learning, pp. 11398–11442. PMLR, 2023. 1 [find](https://scholar.google.com/scholar?q=Giannou%2C+A%2E%2C+Rajput%2C+S%2E%2C+Sohn%2C+J%2E%2Dy%2E%2C+Lee%2C+K%2E%2C+Lee%2C+J%2E+D%2E%2C+and+Papailiopoulos%2C+D%2E+Looped+transformers+as+programmable+computers%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+11398%E2%80%9311442%2E+PMLR%2C)
18. Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. URL https://arxiv.org/abs/2312.00752. 1, 2, 4, 5, 9, 11, 12, 15, 20 [↗](https://arxiv.org/abs/2312.00752)
19. Gu, A., Goel, K., Gupta, A., and Ré, C. On the parameterization and initialization of diagonal state space models. In Advances in Neural Information Processing Systems, volume 35, pp. 35971–35983, 2022a. 3, 4 [find](https://scholar.google.com/scholar?q=Gu%2C+A%2E%2C+Goel%2C+K%2E%2C+Gupta%2C+A%2E%2C+and+R%C3%A9%2C+C%2E+On+the+parameterization+and+initialization+of+diagonal+state+space+models%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+35%2C+pp%2E+35971%E2%80%9335983%2C+202)
20. Gu, A., Goel, K., and Ré, C. Efficiently modeling long sequences with structured state spaces. In The Tenth International Conference on Learning Representations, ICLR 2022, 2022b. URL [https://openreview.net/forum?id=uYLFoz1vlAC](https://openreview.net/forum?id=uYLFoz1vlAC). 2, 3 [↗](https://openreview.net/forum?id=uYLFoz1vlAC)
21. Hahn, M. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics, 8:156–171, 2020. 10 [find](https://scholar.google.com/scholar?q=Hahn%2C+M%2E+Theoretical+limitations+of+self%2Dattention+in+neural+sequence+models%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+8%3A156%E2%80%93171%2C+2020%2E+10)
22. Hahn, M. and Goyal, N. A theory of emergent in-context learning as implicit structure induction. arXiv preprint arXiv:2303.07971, 2023. URL https://arxiv.org/abs/2303.07971. 13 — [arXiv:2303.07971](https://arxiv.org/abs/2303.07971) [↗](https://arxiv.org/abs/2303.07971)
23. Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. 4 [find](https://scholar.google.com/scholar?q=Hochreiter%2C+S%2E+and+Schmidhuber%2C+J%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A1735%E2%80%931780%2C+1997%2E+4)
24. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). arXiv preprint arXiv:2001.08361, 2020. 1, 5, 21
25. Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are rnns: Fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, volume 119 of Proceedings of Machine Learning Research, pp. 5156–5165, 2020. URL… [find](https://scholar.google.com/scholar?q=Katharopoulos%2C+A%2E%2C+Vyas%2C+A%2E%2C+Pappas%2C+N%2E%2C+and+Fleuret%2C+F%2E+Transformers+are+rnns%3A+Fast+autoregressive+transformers+with+linear+attention%2E+In+Proceedings+of+the+37th+International+Conference+on+Machine+L)
26. Kingma, D. P. and Ba, J. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). arXiv:1412.6980, 2014. 5, 20 arXiv preprint Kol, G., Raz, R., and Tal, A. Time-space hardness of learning sparse parities. In Proceedings of the 49th Annual ACM SIGACT Symposium on Theory of Computing, pp. 1067–1080, 2017. 10
27. Li, Y., Ildiz, M. E., Papailiopoulos, D., and Oymak, S. Transformers as algorithms: Generalization and stability in in-context learning, 2023a. 1 [find](https://scholar.google.com/scholar?q=Li%2C+Y%2E%2C+Ildiz%2C+M%2E+E%2E%2C+Papailiopoulos%2C+D%2E%2C+and+Oymak%2C+S%2E+Transformers+as+algorithms%3A+Generalization+and+stability+in+in%2Dcontext+learning%2C+2023a%2E+1)
28. Li, Y., Sreenivasan, K., Giannou, A., Papailiopoulos, D., and Oymak, S. Dissecting chain-of-thought: Compositionality through in-context filtering and learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023b. 6, 21 [find](https://scholar.google.com/scholar?q=Li%2C+Y%2E%2C+Sreenivasan%2C+K%2E%2C+Giannou%2C+A%2E%2C+Papailiopoulos%2C+D%2E%2C+and+Oymak%2C+S%2E+Dissecting+chain%2Dof%2Dthought%3A+Compositionality+through+in%2Dcontext+filtering+and+learning%2E+In+Thirty%2Dseventh+Conference+on+Neural+)
29. Liu, B., Ash, J. T., Goel, S., Krishnamurthy, A., and Zhang, C. Transformers learn shortcuts to automata. In The Eleventh International Conference on Learning Representations, 2022. 1 [find](https://scholar.google.com/scholar?q=Liu%2C+B%2E%2C+Ash%2C+J%2E+T%2E%2C+Goel%2C+S%2E%2C+Krishnamurthy%2C+A%2E%2C+and+Zhang%2C+C%2E+Transformers+learn+shortcuts+to+automata%2E+In+The+Eleventh+International+Conference+on+Learning+Representations%2C+2022%2E+1)
30. Liu, B., Ash, J. T., Goel, S., Krishnamurthy, A., and Zhang, C. Exposing attention glitches with flip-flop language modeling. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. 4 [find](https://scholar.google.com/scholar?q=Liu%2C+B%2E%2C+Ash%2C+J%2E+T%2E%2C+Goel%2C+S%2E%2C+Krishnamurthy%2C+A%2E%2C+and+Zhang%2C+C%2E+Exposing+attention+glitches+with+flip%2Dflop+language+modeling%2E+In+Thirty%2Dseventh+Conference+on+Neural+Information+Processing+Systems%2C+202)
31. Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018. 22 [find](https://scholar.google.com/scholar?q=Loshchilov%2C+I%2E+and+Hutter%2C+F%2E+Decoupled+weight+decay+regularization%2E+In+International+Conference+on+Learning+Representations%2C+2018%2E+22)
32. Mahankali, A., Hashimoto, T. B., and Ma, T. [One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention](https://md.research.elamur.ai/free/en/papers/mahankali-2023-one-step-gradient-descent-provably/). arXiv preprint arXiv:2307.03576, 2023. 3
33. Mehta, H., Gupta, A., Cutkosky, A., and Neyshabur, B. Long range language modeling via gated state spaces. arXiv preprint arXiv:2206.13947, 2022. 12 — [arXiv:2206.13947](https://arxiv.org/abs/2206.13947) [find](https://scholar.google.com/scholar?q=Mehta%2C+H%2E%2C+Gupta%2C+A%2E%2C+Cutkosky%2C+A%2E%2C+and+Neyshabur%2C+B%2E+Long+range+language+modeling+via+gated+state+spaces%2E+arXiv+preprint+arXiv%3A2206%2E13947%2C+2022%2E+12+%E2%80%94+%5BarXiv%3A2206%2E13947%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2206%2E139)
34. Min, S., Lewis, M., Zettlemoyer, L., and Hajishirzi, H. Metaicl: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2791–2809, 2022a. 1, 3 [find](https://scholar.google.com/scholar?q=Min%2C+S%2E%2C+Lewis%2C+M%2E%2C+Zettlemoyer%2C+L%2E%2C+and+Hajishirzi%2C+H%2E+Metaicl%3A+Learning+to+learn+in+context%2E+In+Proceedings+of+the+2022+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+)
35. Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. [Rethinking the role of demonstrations: What makes in-context learning work?](https://md.research.elamur.ai/free/en/papers/min-2022-rethinking-role-demonstrations-what-makes/) arXiv preprint arXiv:2202.12837, 2022b. URL https://arxiv.org/abs/2202.12837. 1 [↗](https://arxiv.org/abs/2202.12837)
36. Muennighoff, N., Rush, A. M., Barak, B., Scao, T. L., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., and Raffel, C. Scaling data-constrained language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL [https://openreview.net/forum?id=](https://openreview.net/forum?id=) j5BuTrEj35. 1, 21 [↗](https://openreview.net/forum?id=)
37. Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., and et al., A. C. [In-context learning and induction heads](https://md.research.elamur.ai/free/en/papers/olsson-2022-context-learning-induction-heads/). arXiv preprint arXiv:2209.11895, 2022. URL https://arxiv.org/abs/2209.11895. 3, 4, 6 [↗](https://arxiv.org/abs/2209.11895)
38. Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Cao, H., Cheng, X., Chung, M., Grella, M., Kiran, K. G., et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048, 2023. URL https://arxiv.org/abs/2305.13048. 2, 4 [↗](https://arxiv.org/abs/2305.13048)
39. Pilault, J., Fathi, M., Firat, O., Pal, C., Bacon, P.-L., and Goroshin, R. [Block-state transformers](https://md.research.elamur.ai/free/en/papers/fathi-2023-block-state-transformers/). In Thirty-seventh Conference on Neural Information Processing Systems, 2023. 12
40. Poli, M., Massaroli, S., Nguyen, E., Fu, D. Y., Dao, T., Baccus, S., Bengio, Y., Ermon, S., and Ré, C. Hyena hierarchy: Towards larger convolutional language models. arXiv preprint arXiv:2302.10866, 2023. URL https://arxiv.org/abs/2302.10866. 2, 4 [↗](https://arxiv.org/abs/2302.10866)
41. Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N. A., and Lewis, M. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350, 2022. 12 [find](https://scholar.google.com/scholar?q=Press%2C+O%2E%2C+Zhang%2C+M%2E%2C+Min%2C+S%2E%2C+Schmidt%2C+L%2E%2C+Smith%2C+N%2E+A%2E%2C+and+Lewis%2C+M%2E+Measuring+and+narrowing+the+compositionality+gap+in+language+models%2E+arXiv+preprint+arXiv%3A2210%2E03350%2C+2022%2E+12)
42. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. 20 [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Wu%2C+J%2E%2C+Child%2C+R%2E%2C+Luan%2C+D%2E%2C+Amodei%2C+D%2E%2C+Sutskever%2C+I%2E%2C+et+al%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2C+2019%2E+20)
43. Ravi, S. and Larochelle, H. Optimization as a model for few-shot learning. In International conference on learning representations, 2016. 1, 3 [find](https://scholar.google.com/scholar?q=Ravi%2C+S%2E+and+Larochelle%2C+H%2E+Optimization+as+a+model+for+few%2Dshot+learning%2E+In+International+conference+on+learning+representations%2C+2016%2E+1%2C+3)
44. Raz, R. Fast learning requires good memory: A time-space lower bound for parity learning. In 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS), pp. 266–275. [find](https://scholar.google.com/scholar?q=Raz%2C+R%2E+Fast+learning+requires+good+memory%3A+A+time%2Dspace+lower+bound+for+parity+learning%2E+In+2016+IEEE+57th+Annual+Symposium+on+Foundations+of+Computer+Science+%28FOCS%29%2C+pp%2E+266%E2%80%93275%2E)
45. Schaeffer, R., Miranda, B., and Koyejo, S. [Are emergent abilities of large language models a mirage?](https://md.research.elamur.ai/free/en/papers/schaeffer-2023-emergent-abilities-large-language-models/) In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL [https://openreview.net/forum?id=ITw9edRDlD](https://openreview.net/forum?id=ITw9edRDlD). 1 [↗](https://openreview.net/forum?id=ITw9edRDlD)
46. Schmidhuber, J., Zhao, J., and Wiering, M. Shifting inductive bias with success-story algorithm, adaptive levin search, and incremental self-improvement. Machine Learning, 28:105–130, 1997. 1 [find](https://scholar.google.com/scholar?q=Schmidhuber%2C+J%2E%2C+Zhao%2C+J%2E%2C+and+Wiering%2C+M%2E+Shifting+inductive+bias+with+success%2Dstory+algorithm%2C+adaptive+levin+search%2C+and+incremental+self%2Dimprovement%2E+Machine+Learning%2C+28%3A105%E2%80%93130%2C+1997%2E+1)
47. Sun, Y., Dong, L., Huang, S., Ma, S., Xia, Y., Xue, J., Wang, J., and Wei, F. [Retentive network: A successor to transformer for large language models](https://md.research.elamur.ai/free/en/papers/sun-2023-retentive-network-successor-transformer-large/). arXiv preprint arXiv:2307.08621, 2023. URL https://arxiv.org/abs/2307.08621. 2, 4 [↗](https://arxiv.org/abs/2307.08621)
48. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023. 22
49. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pp. 5998–6008, 2017. URL https://proceedin…
50. PMLR, 2023a. 1 von Oswald, J., Niklasson, E., Schlegel, M., Kobayashi, S., Zucchet, N., Scherrer, N., Miller, N., Sandler, M., Vladymyrov, M., Pascanu, R., et al. Uncovering mesa-optimization algorithms in transformers. arXiv preprint arXiv:2309.05858, 2023b. URL https://arxiv.org/abs/2309. 05858. 1… — [arXiv:2309.05858](https://arxiv.org/abs/2309.05858) [↗](https://arxiv.org/abs/2309)
51. Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020. 3 [find](https://scholar.google.com/scholar?q=Wang%2C+S%2E%2C+Li%2C+B%2E+Z%2E%2C+Khabsa%2C+M%2E%2C+Fang%2C+H%2E%2C+and+Ma%2C+H%2E+Linformer%3A+Self%2Dattention+with+linear+complexity%2E+arXiv+preprint+arXiv%3A2006%2E04768%2C+2020%2E+3)
52. Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/). arXiv preprint arXiv:2206.07682, 2022. 1
53. Xie, S. M., Raghunathan, A., Liang, P., and Ma, T. [An explanation of in-context learning as implicit bayesian inference](https://md.research.elamur.ai/free/en/papers/xie-2021-explanation-context-learning-implicit-bayesian/). In International Conference on Learning Representations, 2021. 3, 4, 6, 13, 14
54. Yang, L., Lee, K., Nowak, R., and Papailiopoulos, D. Looped transformers are better at learning learning algorithms, 2023a. 1 [find](https://scholar.google.com/scholar?q=Yang%2C+L%2E%2C+Lee%2C+K%2E%2C+Nowak%2C+R%2E%2C+and+Papailiopoulos%2C+D%2E+Looped+transformers+are+better+at+learning+learning+algorithms%2C+2023a%2E+1)
55. Yang, S., Wang, B., Shen, Y., Panda, R., and Kim, Y. Gated linear attention transformers with hardware-efficient training. arXiv preprint arXiv:2312.06635, 2023b. 2, 4 [find](https://scholar.google.com/scholar?q=Yang%2C+S%2E%2C+Wang%2C+B%2E%2C+Shen%2C+Y%2E%2C+Panda%2C+R%2E%2C+and+Kim%2C+Y%2E+Gated+linear+attention+transformers+with+hardware%2Defficient+training%2E+arXiv+preprint+arXiv%3A2312%2E06635%2C+2023b%2E+2%2C+4)
56. Yu, W., Luo, M., Zhou, P., Si, C., Zhou, Y., Wang, X., Feng, J., and Yan, S. Metaformer is actually what you need for vision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10819–10829, 2022. 10 [find](https://scholar.google.com/scholar?q=Yu%2C+W%2E%2C+Luo%2C+M%2E%2C+Zhou%2C+P%2E%2C+Si%2C+C%2E%2C+Zhou%2C+Y%2E%2C+Wang%2C+X%2E%2C+Feng%2C+J%2E%2C+and+Yan%2C+S%2E+Metaformer+is+actually+what+you+need+for+vision%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+computer+vision+and+pattern+r)
57. Zhai, S., Talbott, W., Srivastava, N., Huang, C., Goh, H., Zhang, R., and Susskind, J. An attention free transformer. arXiv preprint arXiv:2105.14103, 2021. 2 — [arXiv:2105.14103](https://arxiv.org/abs/2105.14103) [find](https://scholar.google.com/scholar?q=Zhai%2C+S%2E%2C+Talbott%2C+W%2E%2C+Srivastava%2C+N%2E%2C+Huang%2C+C%2E%2C+Goh%2C+H%2E%2C+Zhang%2C+R%2E%2C+and+Susskind%2C+J%2E+An+attention+free+transformer%2E+arXiv+preprint+arXiv%3A2105%2E14103%2C+2021%2E+2+%E2%80%94+%5BarXiv%3A2105%2E14103%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
58. Zhou, H., Bradley, A., Littwin, E., Razin, N., Saremi, O., Susskind, J., Bengio, S., and Nakkiran, P. [What algorithms can transformers learn? a study in length generalization](https://md.research.elamur.ai/free/en/papers/zhou-2023-algorithms-transformers-learn-study-length/). arXiv preprint arXiv:2310.16028, 2023. 1
59. Zuo, S., Liu, X., Jiao, J., Charles, D., Manavoglu, E., Zhao, T., and Gao, J. Efficient long sequence modeling via state space augmented transformer. arXiv preprint arXiv:2212.08136, 2022. 12 — [arXiv:2212.08136](https://arxiv.org/abs/2212.08136) [find](https://scholar.google.com/scholar?q=Zuo%2C+S%2E%2C+Liu%2C+X%2E%2C+Jiao%2C+J%2E%2C+Charles%2C+D%2E%2C+Manavoglu%2C+E%2E%2C+Zhao%2C+T%2E%2C+and+Gao%2C+J%2E+Efficient+long+sequence+modeling+via+state+space+augmented+transformer%2E+arXiv+preprint+arXiv%3A2212%2E08136%2C+2022%2E+12+%E2%80%94+%5BarXiv)

*Updated 3 November 2026*