---
title: "xLSTM: Extended Long Short-Term Memory (2024)"
description: "They propose exponential gating with a normalizer state and stabilization against numerical overflow… 128 references, cited by 63 papers here."
authors: ["Maximilian Beck", "Korbinian Pöppel", "Markus Spanring", "Andreas Auer", "Oleksandra Prudnikova", "Michael Kopp", "Günter Klambauer", "Johannes Brandstetter", "Sepp Hochreiter"]
year: 2024
venue: "arXiv:2405.04517"
source: "https://arxiv.org/abs/2405.04517"
doi: "10.48550/arxiv.2405.04517"
arxiv: "2405.04517"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/beck-2024-xlstm-extended-long-short-term/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/beck-2024-xlstm-extended-long-short-term/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/) / xLSTM: Extended Long Short-Term Memory

# xLSTM: Extended Long Short-Term Memory

**Authors:** Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter and [Sepp Hochreiter](https://md.research.elamur.ai/free/en/authors/sepp-hochreiter/) · **Year:** 2024 · **Venue:** [arXiv:2405.04517](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/) ([NeurIPS 2024](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/2024/)) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2405.04517) · [PDF](https://arxiv.org/pdf/2405.04517) · [DOI 10.48550/arxiv.2405.04517](https://doi.org/10.48550/arxiv.2405.04517) · [arXiv 2405.04517](https://arxiv.org/abs/2405.04517) · [OpenAlex](https://openalex.org/W4415795940)

**Citations:** 233 citations (OpenAlex, 2026-10-14) · [63](https://md.research.elamur.ai/free/en/papers/beck-2024-xlstm-extended-long-short-term/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The authors examine whether the classic LSTM recurrent network, which has been displaced by Transformers, can compete with modern large language models. To this end, they address its three known weaknesses: the model cannot revise a storage decision once made, its scalar memory has limited capacity, and its computations must be performed sequentially. They propose exponential gating with a normalizer state and stabilization against numerical overflow, as well as two new cells. sLSTM has scalar memory and a new form of memory mixing within individual "heads". mLSTM stores key–value pairs in a matrix memory using a covariance update rule and is fully parallelizable during training. Both cells are embedded in residual blocks, and a stack of such blocks forms the xLSTM architecture. Models ranging from 125M to 1.3B parameters were trained on 300B tokens of SlimPajama and compared with Llama, Mamba and RWKV-4. xLSTM achieved lower perplexity, better scaling curves and linear generation time. The work shows that recurrent networks can once again be a real alternative to attention. Moreover, they use a constant amount of memory during inference.

## Key points
- The architecture has two new cell variants. sLSTM with memory mixing cannot be parallelized, but its CUDA kernel is less than twice as slow as mLSTM. mLSTM uses a d×d matrix and fully parallel training. The code is available at github.com/NX-AI/xlstm.
- On the PALOMA benchmark with 571 text domains, xLSTM achieves lower perplexity than Llama in 486 domains (85.1%) and lower than RWKV-4 in 570 domains (99.8%).
- Scaling laws on the SlimPajama validation set give the following ordering from worst to best model: RWKV-4, Llama, Mamba, xLSTM. The margin of xLSTM over Mamba is roughly the same as that of Mamba over Llama.
- Inference was tested on 1.3B models on an A100-80GB. Generation time grows linearly for xLSTM and quadratically for Llama. Llama runs out of memory (OOM) already at small batch sizes, while xLSTM[1:0] has higher throughput than Mamba.
- The authors identify the following limitations. The unoptimized mLSTM kernels are about four times slower than FlashAttention. Forget gate initialization is sensitive. The architecture and hyperparameters for large models were not fully tuned. No memory overload was observed for contexts up to 16k.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

## Cite

Cite as: Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter and Sepp Hochreiter. 2024. xLSTM: Extended Long Short-Term Memory. arXiv:2405.04517.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{beck2024xlstm,
  title = {{xLSTM: Extended Long Short-Term Memory}},
  author = {Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter and Sepp Hochreiter},
  year = {2024},
  doi = {10.48550/arxiv.2405.04517},
  eprint = {2405.04517},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.04517}
}
```

- [← previous Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting](https://md.research.elamur.ai/free/en/papers/qin-2024-large-language-models-are-effective/)
- [next → KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization](https://md.research.elamur.ai/free/en/papers/hooper-2024-kvquant-towards-10-million-context/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/)

## References

128 references, 43 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. J. Achiam, S. Adler, S. Agarwal, et al. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). ArXiv, 2303.08774, 2023.
2. J. Anderson, J. Silverstein, S. Ritz, and R. Jones. Distinctive features, categorical perception, and probability learning: Some applications of a neural model. Psychological Review, 84:413–451, 1977. doi: 10.1037/0033-295X.84.5.413. — [doi:10.1037/0033-295x.84.5.413](https://doi.org/10.1037/0033-295x.84.5.413) [find](https://scholar.google.com/scholar?q=J%2E+Anderson%2C+J%2E+Silverstein%2C+S%2E+Ritz%2C+and+R%2E+Jones%2E+Distinctive+features%2C+categorical+perception%2C+and+probability+learning%3A+Some+applications+of+a+neural+model%2E+Psychological+Review%2C+84%3A413%E2%80%93451%2C+1977%2E)
3. J. A. Anderson. A simple neural network generating an interactive memory. Mathematical Bio-sciences, 14, 1972. doi: 10.1016/0025-5564(72)90075-2. — [doi:10.1016/0025-5564(72)90075-2](https://doi.org/10.1016/0025-5564%2872%2990075-2) [find](https://scholar.google.com/scholar?q=J%2E+A%2E+Anderson%2E+A+simple+neural+network+generating+an+interactive+memory%2E+Mathematical+Bio%2Dsciences%2C+14%2C+1972%2E+doi%3A+10%2E1016%2F0025%2D5564%2872%2990075%2D2%2E+%E2%80%94+%5Bdoi%3A10%2E1016%2F0025%2D5564%2872%2990075%2D2%5D%28https%3A%2F%2Fdoi%2Eorg%2F1)
4. S. Arora, S. Eyuboglu, A. Timalsina, I. Johnson, M. Poli, J. Zou, A. Rudra, and C. Ré. [Zoology: Measuring and improving recall in efficient language models](https://md.research.elamur.ai/free/en/papers/arora-2023-zoology-measuring-improving-recall-efficient/). ArXiv, 2312.04927, 2023.
5. J. Ba, G. E. Hinton, V. Mnih, J. Z. Leibo, and C. Ionescu. Using fast weights to attend to the recent past. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 4331–4339. Curran Associates, Inc., 2016a. [find](https://scholar.google.com/scholar?q=J%2E+Ba%2C+G%2E+E%2E+Hinton%2C+V%2E+Mnih%2C+J%2E+Z%2E+Leibo%2C+and+C%2E+Ionescu%2E+Using+fast+weights+to+attend+to+the+recent+past%2E+In+D%2E+D%2E+Lee%2C+M%2E+Sugiyama%2C+U%2E+V%2E+Luxburg%2C+I%2E+Guyon%2C+and+R%2E+Garnett+%28eds%2E%29%2C+Advances+in+Neura)
6. J. Ba, J. R. Kiros, and G. Hinton. [Layer normalization](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/). ArXiv, 1607.06450, 2016b.
7. A. Bau, Y. Belinkov, H. Sajjad, N. Durrani, F. Dalvi, and J. Glass. [Identifying and controlling important neurons in neural machine translation](https://md.research.elamur.ai/free/en/papers/bau-2018-identifying-controlling-important-neurons-neural/). In International Conference on Learning Representations (ICLR), 2019. URL [https://openreview.net/forum?id=H1z-PsR5KX](https://openreview.net/forum?id=H1z-PsR5KX). [↗](https://openreview.net/forum?id=H1z-PsR5KX)
8. Y. Bisk, R. Zellers, R. LeBras, J. Gao, and Y. Choi. [Piqa: Reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). In AAAI Conference on Artificial Intelligence, volume 34, pp. 7432–7439, 2020.
9. S. L. Blodgett, L. Green, and B. O’Connor. [Demographic dialectal variation in social media: A case study of African-American English](https://md.research.elamur.ai/free/en/papers/blodgett-2016-demographic-dialectal-variation-social-media/). In Conference on Empirical Methods in Natural Language Processing, pp. 1119–1130, 2016. doi: 10.18653/v1/D16-1120.
10. T. Brown, B. Mann, N. Ryder, et al. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877–1901. [find](https://scholar.google.com/scholar?q=T%2E+Brown%2C+B%2E+Mann%2C+N%2E+Ryder%2C+et+al%2E+Language+models+are+few%2Dshot+learners%2E+In+H%2E+Larochelle%2C+M%2E+Ranzato%2C+R%2E+Hadsell%2C+M%2EF%2E+Balcan%2C+and+H%2E+Lin+%28eds%2E%29%2C+Advances+in+Neural+Information+Processing+Systems%2C+)
11. Curran Associates, Inc., 2020. [find](https://scholar.google.com/scholar?q=Curran+Associates%2C+Inc%2E%2C+2020%2E)
12. K. M. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlós, P. Hawkins, J. Q. Davis, A. Mohiuddin, L. Kaiser, D. B. Belanger, L. J. Colwell, and A. Weller. [Rethinking attention with performers](https://md.research.elamur.ai/free/en/papers/choromanski-2020-rethinking-attention-performers/). In 9th International Conference on Learning Representations (ICLR). OpenReview.net, 2021.…
13. A. Chowdhery, S. Narang, J. Devlin, et al. [PaLM: scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). ArXiv, 2204.02311, 2022.
14. A. Chronopoulou, M. Peters, and J. Dodge. Efficient hierarchical domain adaptation for pretrained language models. In Conference of the North American Chapter of the Association for Computational Linguistics, pp. 1336–1351, 2022. doi: 10.18653/v1/2022.naacl-main.96. [find](https://scholar.google.com/scholar?q=A%2E+Chronopoulou%2C+M%2E+Peters%2C+and+J%2E+Dodge%2E+Efficient+hierarchical+domain+adaptation+for+pretrained+language+models%2E+In+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Ling)
15. P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. [Think you have solved question answering? Try ARC, the AI2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). ArXiv, 1803.05457, 2018.
16. T. M. Cover. Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition. Electronic Computers, IEEE Transactions on, EC-14(3):326–334, 1965. [find](https://scholar.google.com/scholar?q=T%2E+M%2E+Cover%2E+Geometrical+and+statistical+properties+of+systems+of+linear+inequalities+with+applications+in+pattern+recognition%2E+Electronic+Computers%2C+IEEE+Transactions+on%2C+EC%2D14%283%29%3A326%E2%80%93334%2C+1965%2E)
17. T. Dao. [Flashattention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/). In International Conference on Learning Representations (ICLR), volume 12, 2024. URL [https://openreview.net/forum?id=mZn2Xyh9Ec](https://openreview.net/forum?id=mZn2Xyh9Ec). [↗](https://openreview.net/forum?id=mZn2Xyh9Ec)
18. T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré. Flashattention: Fast and memory-efficient exact attention with IO-awareness. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho (eds.), Advances in Neural Information Processing Systems (NeurIPS), 2022. URL [https://openreview.net/](https://openreview.net/) forum?id=H4DqfPSibmx. [↗](https://openreview.net/)
19. P. Dayan and D. J. Willshaw. Optimising synaptic learning rules in linear associative memories. Biological Cybernetics, 65, 1991. doi: 10.1007/bf00206223. — [doi:10.1007/bf00206223](https://doi.org/10.1007/bf00206223) [find](https://scholar.google.com/scholar?q=P%2E+Dayan+and+D%2E+J%2E+Willshaw%2E+Optimising+synaptic+learning+rules+in+linear+associative+memories%2E+Biological+Cybernetics%2C+65%2C+1991%2E+doi%3A+10%2E1007%2Fbf00206223%2E+%E2%80%94+%5Bdoi%3A10%2E1007%2Fbf00206223%5D%28https%3A%2F%2Fdoi%2Eorg%2F10)
20. S. De, S. L. Smith, A. Fernando, A. Botev, G. Cristian-Muraru, A. Gu, R. Haroun, L. Berrada, Y. Chen, S. Srinivasan, G. Desjardins, A. Doucet, D. Budden, Y. W. Teh, R. Pascanu, N. DeFreitas, and C. Gulcehre. Griffin: Mixing gated linear recurrences with local attention for efficient language models.… [find](https://scholar.google.com/scholar?q=S%2E+De%2C+S%2E+L%2E+Smith%2C+A%2E+Fernando%2C+A%2E+Botev%2C+G%2E+Cristian%2DMuraru%2C+A%2E+Gu%2C+R%2E+Haroun%2C+L%2E+Berrada%2C+Y%2E+Chen%2C+S%2E+Srinivasan%2C+G%2E+Desjardins%2C+A%2E+Doucet%2C+D%2E+Budden%2C+Y%2E+W%2E+Teh%2C+R%2E+Pascanu%2C+N%2E+DeFreitas%2C+and+C%2E+Gu)
21. J. Degrave, F. Felici, J. Buchli, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602:414–419, 2022. doi: 10.1038/s41586-021-04301-9. [find](https://scholar.google.com/scholar?q=J%2E+Degrave%2C+F%2E+Felici%2C+J%2E+Buchli%2C+et+al%2E+Magnetic+control+of+tokamak+plasmas+through+deep+reinforcement+learning%2E+Nature%2C+602%3A414%E2%80%93419%2C+2022%2E+doi%3A+10%2E1038%2Fs41586%2D021%2D04301%2D9%2E)
22. G. Delétang, A. Ruoss, J. Grau-Moya, T. Genewein, L. K. Wenliang, E. Catt, C. Cundy, M. Hutter, S. Legg, J. Veness, and P. A. Ortega. [Neural networks and the Chomsky hierarchy](https://md.research.elamur.ai/free/en/papers/deletang-2022-neural-networks-chomsky-hierarchy/). In International Conference on Learning Representations (ICLR), volume 11, 2023. URL https://openreview. net/forum?id=WbxHA…
23. N. Du, Y. Huang, A. M. Dai, et al. [GLaM: efficient scaling of language models with mixture-of-experts](https://md.research.elamur.ai/free/en/papers/du-2021-glam-efficient-scaling-language-models/). ArXiv, 2112.06905, 2021.
24. D. Y. Fu, T. Dao, K. K. Saab, A. W. Thomas, A. Rudra, and C. Re. Hungry hungry hippos: Towards language modeling with state space models. In The Eleventh International Conference on Learning Representations, 2023. URL [https://openreview.net/forum?id=COZDy0WYGg](https://openreview.net/forum?id=COZDy0WYGg). [↗](https://openreview.net/forum?id=COZDy0WYGg)
25. L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy. The Pile: An 800gb dataset of diverse text for language modeling. ArXiv, 2101.00027, 2021. [find](https://scholar.google.com/scholar?q=L%2E+Gao%2C+S%2E+Biderman%2C+S%2E+Black%2C+L%2E+Golding%2C+T%2E+Hoppe%2C+C%2E+Foster%2C+J%2E+Phang%2C+H%2E+He%2C+A%2E+Thite%2C+N%2E+Nabeshima%2C+S%2E+Presser%2C+and+C%2E+Leahy%2E+The+Pile%3A+An+800gb+dataset+of+diverse+text+for+language+modeling%2E+ArX)
26. F. A. Gers, J. Schmidhuber, and F. Cummins. Learning to forget: Continual prediction with LSTM. Neural Compututation, 12(10):2451–2471, 2000. [find](https://scholar.google.com/scholar?q=F%2E+A%2E+Gers%2C+J%2E+Schmidhuber%2C+and+F%2E+Cummins%2E+Learning+to+forget%3A+Continual+prediction+with+LSTM%2E+Neural+Compututation%2C+12%2810%29%3A2451%E2%80%932471%2C+2000%2E)
27. Gemini Team Google. [Gemini: A family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). ArXiv, 2312.11805, 2023.
28. A. Graves. Generating sequences with recurrent neural networks. ArXiv, 1308.0850, 2013. [find](https://scholar.google.com/scholar?q=A%2E+Graves%2E+Generating+sequences+with+recurrent+neural+networks%2E+ArXiv%2C+1308%2E0850%2C+2013%2E)
29. S. Greenbaum and G. Nelson. The international corpus of English (ICE) project. World Englishes, 15(1):3–15, 1996. [find](https://scholar.google.com/scholar?q=S%2E+Greenbaum+and+G%2E+Nelson%2E+The+international+corpus+of+English+%28ICE%29+project%2E+World+Englishes%2C+15%281%29%3A3%E2%80%9315%2C+1996%2E)
30. K. Greff, R. K. Srivastava, J. Koutník, B. R. Steunebrink, and J. Schmidhuber. [LSTM: A search space odyssey](https://md.research.elamur.ai/free/en/papers/greff-2015-lstm-search-space-odyssey/). ArXiv, 1503.04069, 2015.
31. A. Gu and T. Dao. Mamba: Linear-time sequence modeling with selective state spaces. ArXiv, 2312.00752, 2023. [find](https://scholar.google.com/scholar?q=A%2E+Gu+and+T%2E+Dao%2E+Mamba%3A+Linear%2Dtime+sequence+modeling+with+selective+state+spaces%2E+ArXiv%2C+2312%2E00752%2C+2023%2E)
32. A. Gu, K. Goel, and C. Ré. Efficiently modeling long sequences with structured state spaces. ArXiv, 2111.00396, 2021. [find](https://scholar.google.com/scholar?q=A%2E+Gu%2C+K%2E+Goel%2C+and+C%2E+R%C3%A9%2E+Efficiently+modeling+long+sequences+with+structured+state+spaces%2E+ArXiv%2C+2111%2E00396%2C+2021%2E)
33. A. Gupta, A. Gu, and J. Berant. Diagonal state spaces are as effective as structured state spaces. ArXiv, 2203.14343, 2022. [find](https://scholar.google.com/scholar?q=A%2E+Gupta%2C+A%2E+Gu%2C+and+J%2E+Berant%2E+Diagonal+state+spaces+are+as+effective+as+structured+state+spaces%2E+ArXiv%2C+2203%2E14343%2C+2022%2E)
34. K. He, X. Zhang, S. Ren, and J. Sun. [Deep residual learning for image recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/). In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778, 2016.
35. S. Hochreiter. Untersuchungen zu dynamischen neuronalen Netzen. Master’s thesis, Technische Universität München, 1991. [find](https://scholar.google.com/scholar?q=S%2E+Hochreiter%2E+Untersuchungen+zu+dynamischen+neuronalen+Netzen%2E+Master%E2%80%99s+thesis%2C+Technische+Universit%C3%A4t+M%C3%BCnchen%2C+1991%2E)
36. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735–1780, 1997a. [find](https://scholar.google.com/scholar?q=S%2E+Hochreiter+and+J%2E+Schmidhuber%2E+Long+short%2Dterm+memory%2E+Neural+Computation%2C+9%288%29%3A1735%E2%80%931780%2C+1997a%2E)
37. S. Hochreiter and J. Schmidhuber. LSTM can solve hard long time lag problems. In M. C. Mozer, M. I. Jordan, and T. Petsche (eds.), Advances in Neural Information Processing Systems (NeurIPS), volume 9, pp. 473–479. MIT Press, Cambridge MA, 1997b. [find](https://scholar.google.com/scholar?q=S%2E+Hochreiter+and+J%2E+Schmidhuber%2E+LSTM+can+solve+hard+long+time+lag+problems%2E+In+M%2E+C%2E+Mozer%2C+M%2E+I%2E+Jordan%2C+and+T%2E+Petsche+%28eds%2E%29%2C+Advances+in+Neural+Information+Processing+Systems+%28NeurIPS%29%2C+volume+9)
38. S. Hochreiter, Y. Bengio, P. Frasconi, and J. Schmidhuber. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. In J. Kolen and S. Kremer (eds.), A Field Guide to Dynamical Recurrent Networks. IEEE, 2000. [find](https://scholar.google.com/scholar?q=S%2E+Hochreiter%2C+Y%2E+Bengio%2C+P%2E+Frasconi%2C+and+J%2E+Schmidhuber%2E+Gradient+flow+in+recurrent+nets%3A+the+difficulty+of+learning+long%2Dterm+dependencies%2E+In+J%2E+Kolen+and+S%2E+Kremer+%28eds%2E%29%2C+A+Field+Guide+to+Dynami)
39. S. Hochreiter, A. Steven Younger, and Peter R. Conwell. Learning to learn using gradient descent. In G. Dorffner, H. Bischof, and K. Hornik (eds.), Proc. Int. Conf. on Artificial Neural Networks (ICANN 2001), pp. 87–94. [find](https://scholar.google.com/scholar?q=S%2E+Hochreiter%2C+A%2E+Steven+Younger%2C+and+Peter+R%2E+Conwell%2E+Learning+to+learn+using+gradient+descent%2E+In+G%2E+Dorffner%2C+H%2E+Bischof%2C+and+K%2E+Hornik+%28eds%2E%29%2C+Proc%2E+Int%2E+Conf%2E+on+Artificial+Neural+Networks+%28ICAN)
40. S. Hochreiter, M. Heusel, and K. Obermayer. Fast model-based protein homology detection without alignment. Bioinformatics, 23(14):1728–1736, 2007. [find](https://scholar.google.com/scholar?q=S%2E+Hochreiter%2C+M%2E+Heusel%2C+and+K%2E+Obermayer%2E+Fast+model%2Dbased+protein+homology+detection+without+alignment%2E+Bioinformatics%2C+23%2814%29%3A1728%E2%80%931736%2C+2007%2E)
41. J. Hoffmann, S. Borgeaud, A. Mensch, et al. [Training compute-optimal large language models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/). ArXiv, 2203.15556, 2022.
42. M. D. Hossain, F. Sohel, M. F. Shiratuddin, and H. Laga. A comprehensive survey of deep learning for image captioning. ACM Computing Surveys (CSUR), 51(6):118, 2019. [find](https://scholar.google.com/scholar?q=M%2E+D%2E+Hossain%2C+F%2E+Sohel%2C+M%2E+F%2E+Shiratuddin%2C+and+H%2E+Laga%2E+A+comprehensive+survey+of+deep+learning+for+image+captioning%2E+ACM+Computing+Surveys+%28CSUR%29%2C+51%286%29%3A118%2C+2019%2E)
43. J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). ArXiv, 2001.08361, 2020.
44. A. Karpathy. The unreasonable effectiveness of [http://karpathy.github.io/2015/05/21/rnn-effectiveness/](http://karpathy.github.io/2015/05/21/rnn-effectiveness/), 2015. recurrent neural networks. [↗](http://karpathy.github.io/2015/05/21/rnn-effectiveness/)
45. A. Karpathy. OpenAI Five defeats Dota 2 world champions. [https://openai.com/research/openai-five-defeats-dota-2-world-champions](https://openai.com/research/openai-five-defeats-dota-2-world-champions), 2019. [↗](https://openai.com/research/openai-five-defeats-dota-2-world-champions)
46. A. Karpathy and L. Fei-Fei. [Deep visual-semantic alignments for generating image descriptions](https://md.research.elamur.ai/free/en/papers/karpathy-2014-deep-visual-semantic-alignments-generating/). In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3128–3137, 2015.
47. A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In E. H. Daumé III and A. Singh (eds.), International Conference on Machine Learning (ICML), volume 119 of Proceedings of Machine Learning Research, pp. 5156–5165. PML… [find](https://scholar.google.com/scholar?q=A%2E+Katharopoulos%2C+A%2E+Vyas%2C+N%2E+Pappas%2C+and+F%2E+Fleuret%2E+Transformers+are+RNNs%3A+Fast+autoregressive+transformers+with+linear+attention%2E+In+E%2E+H%2E+Daum%C3%A9+III+and+A%2E+Singh+%28eds%2E%29%2C+International+Conference+on)
48. T. Katsch. GateLoop: Fully data-controlled linear recurrence for sequence modeling. ArXiv, 2311.01927, 2023. [find](https://scholar.google.com/scholar?q=T%2E+Katsch%2E+GateLoop%3A+Fully+data%2Dcontrolled+linear+recurrence+for+sequence+modeling%2E+ArXiv%2C+2311%2E01927%2C+2023%2E)
49. D. Kocetkov, R. Li, L. BenAllal, J. Li, C. Mou, C. Mu nozFerrandis, Y. Jernite, M. Mitchell, S. Hughes, T. Wolf, D. Bahdanau, L. vonWerra, and H. deVries. The Stack: 3 TB of permissively licensed source code. ArXiv, 2211.15533, 2022. [find](https://scholar.google.com/scholar?q=D%2E+Kocetkov%2C+R%2E+Li%2C+L%2E+BenAllal%2C+J%2E+Li%2C+C%2E+Mou%2C+C%2E+Mu+nozFerrandis%2C+Y%2E+Jernite%2C+M%2E+Mitchell%2C+S%2E+Hughes%2C+T%2E+Wolf%2C+D%2E+Bahdanau%2C+L%2E+vonWerra%2C+and+H%2E+deVries%2E+The+Stack%3A+3+TB+of+permissively+licensed+sour)
50. T. Kohonen. Correlation matrix memories. IEEE Transactions on Computers, C-21(4), 1972. doi: 10.1109/tc.1972.5008975. — [doi:10.1109/tc.1972.5008975](https://doi.org/10.1109/tc.1972.5008975) [find](https://scholar.google.com/scholar?q=T%2E+Kohonen%2E+Correlation+matrix+memories%2E+IEEE+Transactions+on+Computers%2C+C%2D21%284%29%2C+1972%2E+doi%3A+10%2E1109%2Ftc%2E1972%2E5008975%2E+%E2%80%94+%5Bdoi%3A10%2E1109%2Ftc%2E1972%2E5008975%5D%28https%3A%2F%2Fdoi%2Eorg%2F10%2E1109%2Ftc%2E1972%2E5008975%29)
51. F. Kratzert, D. Klotz, C. Brenner, K. Schulz, and M. Herrnegger. Rainfall-runoff modelling using long short-term memory (LSTM) networks. Hydrology and Earth System Sciences, 22(11):6005–6022, 2018. [find](https://scholar.google.com/scholar?q=F%2E+Kratzert%2C+D%2E+Klotz%2C+C%2E+Brenner%2C+K%2E+Schulz%2C+and+M%2E+Herrnegger%2E+Rainfall%2Drunoff+modelling+using+long+short%2Dterm+memory+%28LSTM%29+networks%2E+Hydrology+and+Earth+System+Sciences%2C+22%2811%29%3A6005%E2%80%936022%2C+2018%2E)
52. F. Kratzert, D. Klotz, G. Shalev, G. Klambauer, S. Hochreiter, and G. Nearing. Benchmarking a catchment-aware long short-term memory network (LSTM) for large-scale hydrological modeling. ArXiv, 1907.08456, 2019. [find](https://scholar.google.com/scholar?q=F%2E+Kratzert%2C+D%2E+Klotz%2C+G%2E+Shalev%2C+G%2E+Klambauer%2C+S%2E+Hochreiter%2C+and+G%2E+Nearing%2E+Benchmarking+a+catchment%2Daware+long+short%2Dterm+memory+network+%28LSTM%29+for+large%2Dscale+hydrological+modeling%2E+ArXiv%2C+1907%2E0)
53. A. Krizhevsky. Learning multiple layers of features from tiny images. Master’s thesis, Deptartment of Computer Science, University of Toronto, 2009. [find](https://scholar.google.com/scholar?q=A%2E+Krizhevsky%2E+Learning+multiple+layers+of+features+from+tiny+images%2E+Master%E2%80%99s+thesis%2C+Deptartment+of+Computer+Science%2C+University+of+Toronto%2C+2009%2E)
54. D. Krotov and J. J. Hopfield. Dense associative memory for pattern recognition. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, pp. 1172–1180. Curran Associates, Inc., 2016. [find](https://scholar.google.com/scholar?q=D%2E+Krotov+and+J%2E+J%2E+Hopfield%2E+Dense+associative+memory+for+pattern+recognition%2E+In+D%2E+D%2E+Lee%2C+M%2E+Sugiyama%2C+U%2E+V%2E+Luxburg%2C+I%2E+Guyon%2C+and+R%2E+Garnett+%28eds%2E%29%2C+Advances+in+Neural+Information+Processing+Sys)
55. D. Krotov and J. J. Hopfield. Dense associative memory is robust to adversarial inputs. ArXiv, 1701.00939, 2017. [find](https://scholar.google.com/scholar?q=D%2E+Krotov+and+J%2E+J%2E+Hopfield%2E+Dense+associative+memory+is+robust+to+adversarial+inputs%2E+ArXiv%2C+1701%2E00939%2C+2017%2E)
56. Y. Lakretz, G. Kruszewski, T. Desbordes, D. Hupkes, S. Dehaene, and M. Baroni. [The emergence of number and syntax units in LSTM language models](https://md.research.elamur.ai/free/en/papers/lakretz-2019-emergence-number-syntax-units-lstm/). In J. Burstein, C. Doran, and T. Solorio (eds.), Conference of the North American Chapter of the Association for Computational Linguistics, pp. 11–20. Asso…
57. Y. Li, T. Cai, Y. Zhang, D. Chen, and D. Dey. What makes convolutional models great on long sequence modeling? ArXiv, 2210.09298, 2022. [find](https://scholar.google.com/scholar?q=Y%2E+Li%2C+T%2E+Cai%2C+Y%2E+Zhang%2C+D%2E+Chen%2C+and+D%2E+Dey%2E+What+makes+convolutional+models+great+on+long+sequence+modeling%3F+ArXiv%2C+2210%2E09298%2C+2022%2E)
58. P. Liang, R. Bommasani, T. Lee, et al. Holistic evaluation of language models. Annals of the New York Academy of Sciences, 1525:140–146, 2023. [find](https://scholar.google.com/scholar?q=P%2E+Liang%2C+R%2E+Bommasani%2C+T%2E+Lee%2C+et+al%2E+Holistic+evaluation+of+language+models%2E+Annals+of+the+New+York+Academy+of+Sciences%2C+1525%3A140%E2%80%93146%2C+2023%2E)
59. J. Lin, R. Men, A. Yang, C. Zhou, M. Ding, Y. Zhang, P. Wang, A. Wang, L. Jiang, X. Jia, J. Zhang, J. Zhang, X. Zou, Z. Li, X. Deng, J. Liu, J. Xue, H. Zhou, J. Ma, j. Yu, Y. Li, W. Lin, J. Zhou, J. Tang, and H. Yang. M6: A Chinese multimodal pretrainer. ArXiv, 2103.00823, 2021. [find](https://scholar.google.com/scholar?q=J%2E+Lin%2C+R%2E+Men%2C+A%2E+Yang%2C+C%2E+Zhou%2C+M%2E+Ding%2C+Y%2E+Zhang%2C+P%2E+Wang%2C+A%2E+Wang%2C+L%2E+Jiang%2C+X%2E+Jia%2C+J%2E+Zhang%2C+J%2E+Zhang%2C+X%2E+Zou%2C+Z%2E+Li%2C+X%2E+Deng%2C+J%2E+Liu%2C+J%2E+Xue%2C+H%2E+Zhou%2C+J%2E+Ma%2C+j%2E+Yu%2C+Y%2E+Li%2C+W%2E+Lin%2C+J%2E+Zhou%2C+J%2E+T)
60. D. Linsley, J. Kim, V. Veerabadran, C. Windolf, and T. Serre. Learning long-range spatial dependencies with horizontal gated recurrent units. Advances in Neural Information Processing Systems (NeurIPS), 31, 2018. [find](https://scholar.google.com/scholar?q=D%2E+Linsley%2C+J%2E+Kim%2C+V%2E+Veerabadran%2C+C%2E+Windolf%2C+and+T%2E+Serre%2E+Learning+long%2Drange+spatial+dependencies+with+horizontal+gated+recurrent+units%2E+Advances+in+Neural+Information+Processing+Systems+%28NeurIPS)
61. I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019. URL [https://openreview.net/forum?id=](https://openreview.net/forum?id=) Bkg6RiCqY7. [↗](https://openreview.net/forum?id=)
62. X. Ma, C. Zhou, X. Kong, J. He, L. Gui, G. Neubig, J. May, and L. Zettlemoyer. Mega: Moving average equipped gated attention. ArXiv, 2209.10655, 2022. [find](https://scholar.google.com/scholar?q=X%2E+Ma%2C+C%2E+Zhou%2C+X%2E+Kong%2C+J%2E+He%2C+L%2E+Gui%2C+G%2E+Neubig%2C+J%2E+May%2C+and+L%2E+Zettlemoyer%2E+Mega%3A+Moving+average+equipped+gated+attention%2E+ArXiv%2C+2209%2E10655%2C+2022%2E)
63. A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts. [Learning word vectors for sentiment analysis](https://md.research.elamur.ai/free/en/papers/maas-2011-learning-word-vectors-sentiment-analysis/). In Annual Meeting of the Association for Computational Linguistics, volume 49, pp. 142–150, 2011.
64. I. Magnusson, A. Bhagia, V. Hofmann, et al. [Paloma: A benchmark for evaluating language model fit](https://md.research.elamur.ai/free/en/papers/magnusson-2023-paloma-benchmark-evaluating-language-model/). ArXiv, 2312.10523, 2023.
65. H. Mehta, A. Gupta, A. Cutkosky, and B. Neyshabur. Long range language modeling via gated state spaces. ArXiv, 2206.13947, 2022. [find](https://scholar.google.com/scholar?q=H%2E+Mehta%2C+A%2E+Gupta%2C+A%2E+Cutkosky%2C+and+B%2E+Neyshabur%2E+Long+range+language+modeling+via+gated+state+spaces%2E+ArXiv%2C+2206%2E13947%2C+2022%2E)
66. S. Merity, C. Xiong, J. Bradbury, and R. Socher. [Pointer sentinel mixture models](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/). In International Conference on Learning Representations (ICRL), 2017. URL [https://openreview.net/](https://openreview.net/) forum?id=Byj72udxe. [↗](https://openreview.net/)
67. W. Merrill and A. Sabharwal. The parallelism tradeoff: Limitations of log-precision transformers. Transactions of the Association for Computational Linguistics, 11:531–545, 2023. doi: 10.1162/ tacl_a_00562. [find](https://scholar.google.com/scholar?q=W%2E+Merrill+and+A%2E+Sabharwal%2E+The+parallelism+tradeoff%3A+Limitations+of+log%2Dprecision+transformers%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+11%3A531%E2%80%93545%2C+2023%2E+doi%3A+10%2E1162%2F+tacl%5Fa%5F)
68. W. Merrill, J. Petty, and A. Sabharwal. The illusion of state in state-space models. ArXiv, 2404.08819, 2024. [find](https://scholar.google.com/scholar?q=W%2E+Merrill%2C+J%2E+Petty%2C+and+A%2E+Sabharwal%2E+The+illusion+of+state+in+state%2Dspace+models%2E+ArXiv%2C+2404%2E08819%2C+2024%2E)
69. M. Milakov and N. Gimelshein. [Online normalizer calculation for softmax](https://md.research.elamur.ai/free/en/papers/milakov-2018-online-normalizer-calculation-softmax/). ArXiv, 1805.02867, 2018.
70. K. Nakano. Associatron – a model of associative memory. IEEE Transactions on Systems, Man, and Cybernetics, SMC-2(3):380–388, 1972. doi: 10.1109/TSMC.1972.4309133. — [doi:10.1109/tsmc.1972.4309133](https://doi.org/10.1109/tsmc.1972.4309133) [find](https://scholar.google.com/scholar?q=K%2E+Nakano%2E+Associatron+%E2%80%93+a+model+of+associative+memory%2E+IEEE+Transactions+on+Systems%2C+Man%2C+and+Cybernetics%2C+SMC%2D2%283%29%3A380%E2%80%93388%2C+1972%2E+doi%3A+10%2E1109%2FTSMC%2E1972%2E4309133%2E+%E2%80%94+%5Bdoi%3A10%2E1109%2Ftsmc%2E1972%2E4309133%5D%28ht)
71. G. Nearing, D. Cohen, V. Dube, M. Gauch, O. Gilon, S. Harrigan, A. Hassidim, D. Klotz, F. Kratzert, A. Metzger, S. Nevo, F. Pappenberger, C. Prudhomme, G. Shalev, S. Shenzis, T. Y. Tekalign, D. Weitzner, and Y. M. B. Kosko. Global prediction of extreme floods in ungauged watersheds. Nature, 627:559–… — [doi:10.1038/s41586-024-07145-1](https://doi.org/10.1038/s41586-024-07145-1) [find](https://scholar.google.com/scholar?q=G%2E+Nearing%2C+D%2E+Cohen%2C+V%2E+Dube%2C+M%2E+Gauch%2C+O%2E+Gilon%2C+S%2E+Harrigan%2C+A%2E+Hassidim%2C+D%2E+Klotz%2C+F%2E+Kratzert%2C+A%2E+Metzger%2C+S%2E+Nevo%2C+F%2E+Pappenberger%2C+C%2E+Prudhomme%2C+G%2E+Shalev%2C+S%2E+Shenzis%2C+T%2E+Y%2E+Tekalign%2C+D%2E+Weitzn)
72. C. Olsson, N. Elhage, N. Nanda, et al. [In-context learning and induction heads](https://md.research.elamur.ai/free/en/papers/olsson-2022-context-learning-induction-heads/). ArXiv, 2209.11895, 2022.
73. A. Orvieto, S. L. Smith, A. Gu, A. Fernando, C. Gulcehre, R. Pascanu, and S. De. [Resurrecting recurrent neural networks for long sequences](https://md.research.elamur.ai/free/en/papers/orvieto-2023-resurrecting-recurrent-neural-networks-long/). In Proceedings of the 40th International Conference on Machine Learning (ICML). JMLR.org, 2023. doi: 10.5555/3618408.3619518.
74. A. Papasavva, S. Zannettou, E. DeCristofaro, G. Stringhini, and J. Blackburn. Raiders of the lost KeK: 3.5 years of augmented 4chan posts from the politically incorrect board. In International AAAI Conference on Web and Social Media (ICWSM), volume 14, pp. 885–894, 2020. [find](https://scholar.google.com/scholar?q=A%2E+Papasavva%2C+S%2E+Zannettou%2C+E%2E+DeCristofaro%2C+G%2E+Stringhini%2C+and+J%2E+Blackburn%2E+Raiders+of+the+lost+KeK%3A+3%2E5+years+of+augmented+4chan+posts+from+the+politically+incorrect+board%2E+In+International+AAAI+Co)
75. D. Paperno, G. Kruszewski, A. Lazaridou, N.-Q. Pham, R. Bernardi, S. Pezzelle, M. Baroni, Gemma G. Boleda, and R. Fernández. [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://md.research.elamur.ai/free/en/papers/paperno-2016-lambada-dataset-word-prediction-requiring/). In Annual Meeting of the Association for Computational Linguistics, volume 1, pp. 1525–1534, 2016.
76. G. Penedo, Q. Malartic, D. Hesslow, R. Cojocaru, A. Cappelli, H. Alobeidli, B. Pannier, E. Almazrouei, and J. Launay. The RefinedWeb dataset for Falcon LLM: Outperforming curated corpora with web data, and web data only. ArXiv, 2306.01116, 2023. [find](https://scholar.google.com/scholar?q=G%2E+Penedo%2C+Q%2E+Malartic%2C+D%2E+Hesslow%2C+R%2E+Cojocaru%2C+A%2E+Cappelli%2C+H%2E+Alobeidli%2C+B%2E+Pannier%2C+E%2E+Almazrouei%2C+and+J%2E+Launay%2E+The+RefinedWeb+dataset+for+Falcon+LLM%3A+Outperforming+curated+corpora+with+web+data)
77. B. Peng, E. Alcaide, Q. Anthony, et al. RWKV: Reinventing RNNs for the transformer era. ArXiv, 2305.13048, 2023. [find](https://scholar.google.com/scholar?q=B%2E+Peng%2C+E%2E+Alcaide%2C+Q%2E+Anthony%2C+et+al%2E+RWKV%3A+Reinventing+RNNs+for+the+transformer+era%2E+ArXiv%2C+2305%2E13048%2C+2023%2E)
78. B. Peng, D. Goldstein, Q. Anthony, et al. Eagle and Finch: RWKV with matrix-valued states and dynamic recurrence. ArXiv, 2404.05892, 2024. [find](https://scholar.google.com/scholar?q=B%2E+Peng%2C+D%2E+Goldstein%2C+Q%2E+Anthony%2C+et+al%2E+Eagle+and+Finch%3A+RWKV+with+matrix%2Dvalued+states+and+dynamic+recurrence%2E+ArXiv%2C+2404%2E05892%2C+2024%2E)
79. M. Poli, S. Massaroli, E. Nguyen, D. Y. Fu, T. Dao, S. Baccus, Y. Bengio, S. Ermon, and C. Ré. Hyena hierarchy: Towards larger convolutional language models. In Proceedings of the 40th International Conference on Machine Learning (ICML). JMLR.org, 2023. doi: 10.5555/3618408.3619572. [find](https://scholar.google.com/scholar?q=M%2E+Poli%2C+S%2E+Massaroli%2C+E%2E+Nguyen%2C+D%2E+Y%2E+Fu%2C+T%2E+Dao%2C+S%2E+Baccus%2C+Y%2E+Bengio%2C+S%2E+Ermon%2C+and+C%2E+R%C3%A9%2E+Hyena+hierarchy%3A+Towards+larger+convolutional+language+models%2E+In+Proceedings+of+the+40th+International+C)
80. M. Poli, A. W. Thomas, E. Nguyen, P. Ponnusamy, B. Deiseroth, K. Kersting, T. Suzuki, B. Hie, S. Ermon, C. Ré, C. Zhang, and S. Massaroli. Mechanistic design and scaling of hybrid architectures. ArXiv, 2403.17844, 2024. [find](https://scholar.google.com/scholar?q=M%2E+Poli%2C+A%2E+W%2E+Thomas%2C+E%2E+Nguyen%2C+P%2E+Ponnusamy%2C+B%2E+Deiseroth%2C+K%2E+Kersting%2C+T%2E+Suzuki%2C+B%2E+Hie%2C+S%2E+Ermon%2C+C%2E+R%C3%A9%2C+C%2E+Zhang%2C+and+S%2E+Massaroli%2E+Mechanistic+design+and+scaling+of+hybrid+architectures%2E+ArXiv)
81. Z. Qin, S. Yang, and Y. Zhong. Hierarchically gated recurrent neural network for sequence modeling. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, 2023. URL [https://openreview.net/forum?id=P1TCHxJwLB](https://openreview.net/forum?id=P1TCHxJwLB). [↗](https://openreview.net/forum?id=P1TCHxJwLB)
82. Z. Qin, S. Yang, W. Sun, X. Shen, D. Li, W. Sun, and Y. Zhong. HGRN2: Gated linear RNNs with state expansion. ArXiv, 2404.07904, 2024. [find](https://scholar.google.com/scholar?q=Z%2E+Qin%2C+S%2E+Yang%2C+W%2E+Sun%2C+X%2E+Shen%2C+D%2E+Li%2C+W%2E+Sun%2C+and+Y%2E+Zhong%2E+HGRN2%3A+Gated+linear+RNNs+with+state+expansion%2E+ArXiv%2C+2404%2E07904%2C+2024%2E)
83. D. R. Radev, P. Muthukrishnan, and V. Qazvinian. The ACL anthology network corpus. In Workshop on Text and Citation Analysis for Scholarly Digital Libraries (NLPIR4DL), pp. 54–61. Association for Computational Linguistics, 2009. [find](https://scholar.google.com/scholar?q=D%2E+R%2E+Radev%2C+P%2E+Muthukrishnan%2C+and+V%2E+Qazvinian%2E+The+ACL+anthology+network+corpus%2E+In+Workshop+on+Text+and+Citation+Analysis+for+Scholarly+Digital+Libraries+%28NLPIR4DL%29%2C+pp%2E+54%E2%80%9361%2E+Association+for+Comp)
84. A. Radford, R. Jozefowicz, and I. Sutskever. [Learning to generate reviews and discovering sentiment](https://md.research.elamur.ai/free/en/papers/radford-2017-learning-generate-reviews-discovering-sentiment/). ArXiv, 1704.01444, 2017.
85. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. [https://openai.com/index/better-language-models](https://openai.com/index/better-language-models), 2019. [↗](https://openai.com/index/better-language-models)
86. J. W. Rae, S. Borgeaud, T. Cai, et al. Scaling language models: Methods, analysis & insights from training Gopher. ArXiv, 2112.11446, 2021. [find](https://scholar.google.com/scholar?q=J%2E+W%2E+Rae%2C+S%2E+Borgeaud%2C+T%2E+Cai%2C+et+al%2E+Scaling+language+models%3A+Methods%2C+analysis+%26+insights+from+training+Gopher%2E+ArXiv%2C+2112%2E11446%2C+2021%2E)
87. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). ArXiv, 1910.10683, 2019.
88. H. Ramsauer, B. Schäfl, J. Lehner, P. Seidl, M. Widrich, L. Gruber, M. Holzleitner, M. Pavlović, G. K. Sandve, V. Greiff, D. Kreil, M. Kopp, G. Klambauer, J. Brandstetter, and S. Hochreiter. Hopfield networks is all you need. In International Conference on Learning Representations (ICLR). OpenRevie… [find](https://scholar.google.com/scholar?q=H%2E+Ramsauer%2C+B%2E+Sch%C3%A4fl%2C+J%2E+Lehner%2C+P%2E+Seidl%2C+M%2E+Widrich%2C+L%2E+Gruber%2C+M%2E+Holzleitner%2C+M%2E+Pavlovic%CC%81%2C+G%2E+K%2E+Sandve%2C+V%2E+Greiff%2C+D%2E+Kreil%2C+M%2E+Kopp%2C+G%2E+Klambauer%2C+J%2E+Brandstetter%2C+and+S%2E+Hochreiter%2E+Hopfield)
89. M. Reid, V. Zhong, S. Gururangan, and L. Zettlemoyer. M2D2: A massively multi-domain language modeling dataset. In Conference on Empirical Methods in Natural Language Processing, pp. 964–975, 2022. [find](https://scholar.google.com/scholar?q=M%2E+Reid%2C+V%2E+Zhong%2C+S%2E+Gururangan%2C+and+L%2E+Zettlemoyer%2E+M2D2%3A+A+massively+multi%2Ddomain+language+modeling+dataset%2E+In+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pp%2E+964%E2%80%93975%2C+2022%2E)
90. M. Reid, N. Savinov, D. Teplyashin, et al. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/). ArXiv, 2403.05530, 2024.
91. M. H. Ribeiro, J. Blackburn, B. Bradlyn, E. DeCristofaro, G. Stringhini, S. Long, S. Greenberg, and S. Zannettou. The evolution of the manosphere across the web. In Proceedings of the international AAAI conference on web and social media, volume 15, pp. 196–207, 2021. [find](https://scholar.google.com/scholar?q=M%2E+H%2E+Ribeiro%2C+J%2E+Blackburn%2C+B%2E+Bradlyn%2C+E%2E+DeCristofaro%2C+G%2E+Stringhini%2C+S%2E+Long%2C+S%2E+Greenberg%2C+and+S%2E+Zannettou%2E+The+evolution+of+the+manosphere+across+the+web%2E+In+Proceedings+of+the+international+AA)
92. K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021. [find](https://scholar.google.com/scholar?q=K%2E+Sakaguchi%2C+R%2E+L%2E+Bras%2C+C%2E+Bhagavatula%2C+and+Y%2E+Choi%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+Communications+of+the+ACM%2C+64%289%29%3A99%E2%80%93106%2C+2021%2E)
93. T. L. Scao, A. Fan, C. Akiki, et al. BLOOM: A 176B-parameter open-access multilingual language model. ArXiv, 2211.05100, 2022. [find](https://scholar.google.com/scholar?q=T%2E+L%2E+Scao%2C+A%2E+Fan%2C+C%2E+Akiki%2C+et+al%2E+BLOOM%3A+A+176B%2Dparameter+open%2Daccess+multilingual+language+model%2E+ArXiv%2C+2211%2E05100%2C+2022%2E)
94. I. Schlag, K. Irie, and J. Schmidhuber. [Linear transformers are secretly fast weight programmers](https://md.research.elamur.ai/free/en/papers/schlag-2021-linear-transformers-secretly-fast-weight/). In M. Meila and T. Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning (ICML), volume 139 of Proceedings of Machine Learning Research, pp. 9355–9366. PMLR, 2021.
95. J. Schmidhuber. Learning to control fast-weight memories: An alternative to recurrent nets. Neural Computation, 4(1):131–139, 1992. [find](https://scholar.google.com/scholar?q=J%2E+Schmidhuber%2E+Learning+to+control+fast%2Dweight+memories%3A+An+alternative+to+recurrent+nets%2E+Neural+Computation%2C+4%281%29%3A131%E2%80%93139%2C+1992%2E)
96. J. Schmidhuber. Deep learning in neural networks: An overview. Neural Networks, 61:85–117, 2015. doi: 10.1016/j.neunet.2014.09.003. [find](https://scholar.google.com/scholar?q=J%2E+Schmidhuber%2E+Deep+learning+in+neural+networks%3A+An+overview%2E+Neural+Networks%2C+61%3A85%E2%80%93117%2C+2015%2E+doi%3A+10%2E1016%2Fj%2Eneunet%2E2014%2E09%2E003%2E)
97. J. Schulman, B. Zoph, C. Kim, J. Hilton, et al. ChatGPT: Optimizing language models for dialogue. [https://openai.com/blog/chatgpt/](https://openai.com/blog/chatgpt/), 2022. OpenAI Research. [↗](https://openai.com/blog/chatgpt/)
98. T. J. Sejnowski. Storing covariance with nonlinearly interacting neurons. Journal of Mathematical Biology, 4, 1977. doi: 10.1007/BF00275079. — [doi:10.1007/bf00275079](https://doi.org/10.1007/bf00275079) [find](https://scholar.google.com/scholar?q=T%2E+J%2E+Sejnowski%2E+Storing+covariance+with+nonlinearly+interacting+neurons%2E+Journal+of+Mathematical+Biology%2C+4%2C+1977%2E+doi%3A+10%2E1007%2FBF00275079%2E+%E2%80%94+%5Bdoi%3A10%2E1007%2Fbf00275079%5D%28https%3A%2F%2Fdoi%2Eorg%2F10%2E1007%2Fbf002750)
99. M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-LM: Training multi-billion parameter language models using model parallelism. ArXiv, 1909.08053, 2019. [find](https://scholar.google.com/scholar?q=M%2E+Shoeybi%2C+M%2E+Patwary%2C+R%2E+Puri%2C+P%2E+LeGresley%2C+J%2E+Casper%2C+and+B%2E+Catanzaro%2E+Megatron%2DLM%3A+Training+multi%2Dbillion+parameter+language+models+using+model+parallelism%2E+ArXiv%2C+1909%2E08053%2C+2019%2E)
100. J. T. H. Smith, A. Warrington, and S. W. Linderman. Simplified state space layers for sequence modeling. ArXiv, 2208.04933, 2022. [find](https://scholar.google.com/scholar?q=J%2E+T%2E+H%2E+Smith%2C+A%2E+Warrington%2C+and+S%2E+W%2E+Linderman%2E+Simplified+state+space+layers+for+sequence+modeling%2E+ArXiv%2C+2208%2E04933%2C+2022%2E)
101. D. Soboleva, F. Al-Khateeb, R. Myers, J. R. Steeves, J. Hestness, and N. Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. [https://www.cerebras.net/blog/](https://www.cerebras.net/blog/) slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama, 2023. URL https://huggingface.co/datasets/cerebr… [↗](https://www.cerebras.net/blog/)
102. L. Soldaini, R. Kinney, A. Bhagia, et al. Dolma: an open corpus of three trillion tokens for language model pretraining research. ArXiv, 2306.01116, 2023. [find](https://scholar.google.com/scholar?q=L%2E+Soldaini%2C+R%2E+Kinney%2C+A%2E+Bhagia%2C+et+al%2E+Dolma%3A+an+open+corpus+of+three+trillion+tokens+for+language+model+pretraining+research%2E+ArXiv%2C+2306%2E01116%2C+2023%2E)
103. S. Soltan, S. Ananthakrishnan, J. FitzGerald, R. Gupta, W. Hamza, H. Khan, C. Peris, S. Rawls, A. Rosenbaum, A. Rumshisky, C. S. Prakash, M. Sridhar, F. Triefenbach, A. Verma, G. Tur, and P. Natarajan. [AlexaTM 20B: Few-shot learning using a large-scale multilingual Seq2Seq model](https://md.research.elamur.ai/free/en/papers/soltan-2022-alexatm-20b-few-shot-learning/). ArXiv, 2208.01448,…
104. R. K. Srivastava, K. Greff, and J. Schmidhuber. Training very deep networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems (NeurIPS), volume 28. Curran Associates, Inc., 2015. [find](https://scholar.google.com/scholar?q=R%2E+K%2E+Srivastava%2C+K%2E+Greff%2C+and+J%2E+Schmidhuber%2E+Training+very+deep+networks%2E+In+C%2E+Cortes%2C+N%2E+Lawrence%2C+D%2E+Lee%2C+M%2E+Sugiyama%2C+and+R%2E+Garnett+%28eds%2E%29%2C+Advances+in+Neural+Information+Processing+Systems+%28N)
105. Y. Sun, L. Dong, S. Huang, S. Ma, Y. Xia, J. Xue, J. Wang, and F. Wei. [Retentive network: A successor to transformer for large language models](https://md.research.elamur.ai/free/en/papers/sun-2023-retentive-network-successor-transformer-large/). ArXiv, 2307.08621, 2023.
106. L. Sutawika, L. Gao, H. Schoelkopf, et al. EleutherAI/lm-evaluation-harness: Major refactor, 2023. [find](https://scholar.google.com/scholar?q=L%2E+Sutawika%2C+L%2E+Gao%2C+H%2E+Schoelkopf%2C+et+al%2E+EleutherAI%2Flm%2Devaluation%2Dharness%3A+Major+refactor%2C+2023%2E)
107. L. Sutawika, H. Schoelkopf, L. Gao, B. Abbasi, S. Biderman, J. Tow, B. fattori, C. Lovering, farzanehnakhaee70, J. Phang, A. Thite, Fazz, T. Wang, N. Muennighoff, Aflah, sdtblck, nopperl, gakada, tttyuntian, researcher2, Chris, J. Etxaniz, H. A. Lee, Z. Kasner, Khalid, J. Hsu, A. Kanekar, P. S. Amma… [find](https://scholar.google.com/scholar?q=L%2E+Sutawika%2C+H%2E+Schoelkopf%2C+L%2E+Gao%2C+B%2E+Abbasi%2C+S%2E+Biderman%2C+J%2E+Tow%2C+B%2E+fattori%2C+C%2E+Lovering%2C+farzanehnakhaee70%2C+J%2E+Phang%2C+A%2E+Thite%2C+Fazz%2C+T%2E+Wang%2C+N%2E+Muennighoff%2C+Aflah%2C+sdtblck%2C+nopperl%2C+gakada%2C+ttty)
108. I. Sutskever, O. Vinyals, and Q. V. V. Le. [Sequence to sequence learning with neural networks](https://md.research.elamur.ai/free/en/papers/sutskever-2014-sequence-sequence-learning-neural-networks/). In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27 (NIPS’13), pp. 3104–3112. Curran Associates, Inc., 2014.
109. Y. Tay, D. Bahri, D. Metzler, D.-C. Juan, Z. Zhao, and C. Zheng. Synthesizer: Rethinking self-attention in transformer models. ArXiv, 2005.00743, 2020. [find](https://scholar.google.com/scholar?q=Y%2E+Tay%2C+D%2E+Bahri%2C+D%2E+Metzler%2C+D%2E%2DC%2E+Juan%2C+Z%2E+Zhao%2C+and+C%2E+Zheng%2E+Synthesizer%3A+Rethinking+self%2Dattention+in+transformer+models%2E+ArXiv%2C+2005%2E00743%2C+2020%2E)
110. Y. Tay, M. Dehghani, S. Abnar, Y. Shen, D. Bahri, P. Pham, J. Rao, L. Yang, S. Ruder, and D. Metzler. [Long range arena: A benchmark for efficient transformers](https://md.research.elamur.ai/free/en/papers/tay-2020-long-range-arena-benchmark-efficient/). In International Conference on Learning Representations (ICRL), 2021. URL [https://openreview.net/forum?id=qVyeW-grC2k](https://openreview.net/forum?id=qVyeW-grC2k). [↗](https://openreview.net/forum?id=qVyeW-grC2k)
111. R. Thoppilan, D. deFreitas, J. Hall, et al. [LaMDA: Language models for dialog applications](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/). ArXiv, 2201.08239, 2022.
112. TogetherComputer. [Redpajama: an open dataset for training large language models](https://md.research.elamur.ai/free/en/papers/weber-2024-redpajama-open-dataset-training-large/), 2023. URL [https://github.com/togethercomputer/RedPajama-Data](https://github.com/togethercomputer/RedPajama-Data). [↗](https://github.com/togethercomputer/RedPajama-Data)
113. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). ArXiv, 2302.1397, 2023.
114. D. Vadas and J. R. Curran. [Parsing noun phrases in the Penn Treebank](https://md.research.elamur.ai/free/en/papers/vadas-2011-parsing-noun-phrases-penn-treebank/). Computational Linguistics, 37(4):753–809, 2011.
115. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems (NeurIPS), volume 30, pp. 5998–6008. Curran Associates, Inc., 2017.
116. O. Vinyals, T. Ewalds, S. Bartunov, et al. Starcraft II: A new challenge for reinforcement learning. ArXiv, 1708.04782, 2017. [find](https://scholar.google.com/scholar?q=O%2E+Vinyals%2C+T%2E+Ewalds%2C+S%2E+Bartunov%2C+et+al%2E+Starcraft+II%3A+A+new+challenge+for+reinforcement+learning%2E+ArXiv%2C+1708%2E04782%2C+2017%2E)
117. J. Wang, J. N. Yan, A. Gu, and A. M. Rush. Pretraining without attention. ArXiv, 2212.10544, 2022. [find](https://scholar.google.com/scholar?q=J%2E+Wang%2C+J%2E+N%2E+Yan%2C+A%2E+Gu%2C+and+A%2E+M%2E+Rush%2E+Pretraining+without+attention%2E+ArXiv%2C+2212%2E10544%2C+2022%2E)
118. S. Wang, B. Z. Li, M. Khabsa, H. Fang, and H. Ma. Linformer: Self-attention with linear complexity. ArXiv, 2006.04768, 2020. [find](https://scholar.google.com/scholar?q=S%2E+Wang%2C+B%2E+Z%2E+Li%2C+M%2E+Khabsa%2C+H%2E+Fang%2C+and+H%2E+Ma%2E+Linformer%3A+Self%2Dattention+with+linear+complexity%2E+ArXiv%2C+2006%2E04768%2C+2020%2E)
119. S. Wang, Y. Sun, Y. Xiang, et al. [ERNIE 3.0 Titan: Exploring larger-scale knowledge enhanced pre-training for language understanding and generation](https://md.research.elamur.ai/free/en/papers/wang-2021-ernie-3-0-titan/). ArXiv, 2112.12731, 2021.
120. Y. Wu and K. He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3–19, 2018. [find](https://scholar.google.com/scholar?q=Y%2E+Wu+and+K%2E+He%2E+Group+normalization%2E+In+Proceedings+of+the+European+conference+on+computer+vision+%28ECCV%29%2C+pp%2E+3%E2%80%9319%2C+2018%2E)
121. L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, and C. Raffel. [mT5: A massively multilingual pre-trained text-to-text transformer](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/). In Conference of the North American Chapter of the Association for Computational Linguistics, pp. 483–498, 2021. doi: 10.18653/v1/2021.naacl…
122. S. Yang and Y. Zhang. FLA: A Triton-based library for hardware-efficient implementations of linear attention mechanism, 2024. URL [https://github.com/sustcsonglin/](https://github.com/sustcsonglin/) flash-linear-attention. [↗](https://github.com/sustcsonglin/)
123. S. Yang, B. Wang, Y. Shen, R. Panda, and Y. Kim. Gated linear attention transformers with hardware-efficient training. ArXiv, 2312.06635, 2023. [find](https://scholar.google.com/scholar?q=S%2E+Yang%2C+B%2E+Wang%2C+Y%2E+Shen%2C+R%2E+Panda%2C+and+Y%2E+Kim%2E+Gated+linear+attention+transformers+with+hardware%2Defficient+training%2E+ArXiv%2C+2312%2E06635%2C+2023%2E)
124. S. Zannettou, B. Bradlyn, E. DeCristofaro, H. Kwak, M. Sirivianos, G. Stringini, and J. Blackburn. What is Gab: A bastion of free speech or an alt-right echo chamber. In The Web Conference, pp. 1007–1014, 2018. doi: 10.1145/3184558.3191531. — [doi:10.1145/3184558.3191531](https://doi.org/10.1145/3184558.3191531) [find](https://scholar.google.com/scholar?q=S%2E+Zannettou%2C+B%2E+Bradlyn%2C+E%2E+DeCristofaro%2C+H%2E+Kwak%2C+M%2E+Sirivianos%2C+G%2E+Stringini%2C+and+J%2E+Blackburn%2E+What+is+Gab%3A+A+bastion+of+free+speech+or+an+alt%2Dright+echo+chamber%2E+In+The+Web+Conference%2C+pp%2E+1007%E2%80%931)
125. W. Zaremba and I. Sutskever. Learning to execute. ArXiv, 1410.4615, 2014. [find](https://scholar.google.com/scholar?q=W%2E+Zaremba+and+I%2E+Sutskever%2E+Learning+to+execute%2E+ArXiv%2C+1410%2E4615%2C+2014%2E)
126. R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In Annual Meeting of the Association for Computational Linguistics, pp. 4791–4800, 2019. [find](https://scholar.google.com/scholar?q=R%2E+Zellers%2C+A%2E+Holtzman%2C+Y%2E+Bisk%2C+A%2E+Farhadi%2C+and+Y%2E+Choi%2E+HellaSwag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pp%2E+4791%E2%80%934800%2C+2019)
127. A. Zeng, X. Liu, Z. Du, et al. [GLM-130B: An open bilingual pre-trained model](https://md.research.elamur.ai/free/en/papers/zeng-2022-glm-130b-open-bilingual-pre/). ArXiv, 2210.02414, 2022.
128. S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. [OPT: Open pre-trained transformer language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/). ArXiv, 2205.01068, 2022.

*Updated 2 November 2026*