---
title: "Samba — paper by Ren et al. (2024) — summary and ideas"
description: "The authors propose Samba, a hybrid that stacks Mamba layers, SwiGLU MLPs and Sliding… 112 references, cited by 52 papers here."
authors: ["Liliang Ren", "Yang Liu", "Yadong Lu", "Yelong Shen", "Chen Liang", "Weizhu Chen"]
year: 2024
venue: "arXiv:2406.07522"
source: "https://arxiv.org/abs/2406.07522"
doi: "10.48550/arxiv.2406.07522"
arxiv: "2406.07522"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ren-2024-samba-simple-hybrid-state-space/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ren-2024-samba-simple-hybrid-state-space/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling

# Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling

**Authors:** [Liliang Ren](https://md.research.elamur.ai/free/en/authors/liliang-ren/), [Yang Liu](https://md.research.elamur.ai/free/en/authors/yang-liu/), Yadong Lu, [Yelong Shen](https://md.research.elamur.ai/free/en/authors/yelong-shen/), [Chen Liang](https://md.research.elamur.ai/free/en/authors/chen-liang/) and [Weizhu Chen](https://md.research.elamur.ai/free/en/authors/weizhu-chen/) · **Year:** 2024 · **Venue:** arXiv:2406.07522 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2406.07522) · [PDF](https://arxiv.org/pdf/2406.07522) · [DOI 10.48550/arxiv.2406.07522](https://doi.org/10.48550/arxiv.2406.07522) · [arXiv 2406.07522](https://arxiv.org/abs/2406.07522) · [OpenAlex](https://openalex.org/W4399655853)

**Citations:** 7 citations (OpenAlex, 2026-10-14) · [52](https://md.research.elamur.ai/free/en/papers/ren-2024-samba-simple-hybrid-state-space/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Language models struggle with very long inputs. Full attention scales quadratically with sequence length, while state space models (SSMs) are linear but recall specific details poorly. The authors propose Samba, a hybrid that stacks Mamba layers, SwiGLU MLPs and Sliding Window Attention (SWA, window 2048) in alternating order. The recurrent Mamba blocks carry a compressed long-range state. The local attention blocks retrieve exact content from the recent past. The MLPs handle factual knowledge. They trained models from 421M to 3.8B parameters on SlimPajama, Phi-2 and Phi-3 data, the largest on 3.2T tokens. They compared Samba with Llama-3, Mistral, Mamba and other hybrid layouts under identical setups. Samba outperformed Transformer baselines on short-context benchmarks. Trained only on 4K sequences, it extrapolated far beyond that length while keeping linear-time decoding. This is notable because a linear-complexity model rarely matches strong Transformers at scale while also handling near-unlimited context cheaply. The training code is open source.

## Key points
- The 3.8B post-trained Samba scores 71.9 on MMLU, 62.8 on HumanEval and 87.6 on GSM8K. It beats Phi-3-mini trained with the same data and recipe.
- Pretrained at 4K context, its zero-shot perplexity on Proof-Pile keeps improving up to 1M tokens, a 256× extrapolation.
- After 500 instruction-tuning steps at 4K length, it achieves perfect Passkey Retrieval up to 256K tokens. It also extrapolates better than full-attention models on Phonebook.
- On 128K-token prompts, throughput is 3.73× higher than Transformers with grouped-query attention. Generating 64K tokens is 3.64× faster.
- Limitations: the base model's retrieval is close to SWA-only models. Mamba-SWA-MLP does better on some tasks, such as WinoGrande, SIQA and GSM8K. The code is at github.com/microsoft/Samba.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Illinois Urbana-Champaign](https://md.research.elamur.ai/free/en/universities/university-of-illinois-urbana-champaign/) — University, Urbana, United States · 184 works · 69,372 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang and Weizhu Chen. 2024. Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling. arXiv:2406.07522.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{ren2024samba,
  title = {{Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling}},
  author = {Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang and Weizhu Chen},
  year = {2024},
  doi = {10.48550/arxiv.2406.07522},
  eprint = {2406.07522},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.07522}
}
```

- [← previous NADI 2024: The Fifth Nuanced Arabic Dialect Identification Shared Task](https://md.research.elamur.ai/free/en/papers/mageed-2024-nadi-2024-fifth-nuanced-arabic/)
- [next → Simple linear attention language models balance the recall-throughput tradeoff](https://md.research.elamur.ai/free/en/papers/arora-2024-simple-linear-attention-language-models/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/)

## References

112 references, 47 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Alli… — [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-3-technical-report-highly/)
2. J. Ainslie, J. Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebr’on, and Sumit K. Sanghai. [Gqa: Training generalized multi-query transformer models from multi-head checkpoints](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/). Conference on Empirical Methods in Natural Language Processing, 2023. doi: 10.48550/arXiv.2305.13245. URL https:/…
3. Ekin Akyürek, Bailin Wang, Yoon Kim, and Jacob Andreas. In-context language learning: Architectures and algorithms. arXiv preprint arXiv: 2401.12973, 2024. URL https://arxiv.org/abs/2401.12973v2. [↗](https://arxiv.org/abs/2401.12973v2)
4. Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. [Zoology: Measuring and improving recall in efficient language models](https://md.research.elamur.ai/free/en/papers/arora-2023-zoology-measuring-improving-recall-efficient/). arXiv preprint arXiv: 2312.04927, 2023. URL https://arxiv.org/abs/2312.04927v1. [↗](https://arxiv.org/abs/2312.04927v1)
5. Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher Ré. [Simple linear attention language models balance the recall-throughput tradeoff](https://md.research.elamur.ai/free/en/papers/arora-2024-simple-linear-attention-language-models/). arXiv preprint arXiv:2402.18668, 2024.
6. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv: 2108.07732, 2021. URL https://arxiv.org/abs/2108.07732v1. [↗](https://arxiv.org/abs/2108.07732v1)
7. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). International Conference On Learning Representations, 2014. URL https://arxiv.org/abs/1409.0473v7. [↗](https://arxiv.org/abs/1409.0473v7)
8. Iz Beltagy, Matthew E. Peters, and Arman Cohan. [Longformer: The long-document transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/). arXiv preprint arXiv: Arxiv-2004.05150, 2020. URL https://arxiv.org/abs/2004.05150v2. [↗](https://arxiv.org/abs/2004.05150v2)
9. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. [PIQA: reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI…
10. AAAI Press, 2020. doi: 10.1609/AAAI.V34I05.6239. URL https://doi.org/10.1609/aaai.v34i05.6239. — [doi:10.1609/aaai.v34i05.6239](https://doi.org/10.1609/aaai.v34i05.6239) [↗](https://doi.org/10.1609/aaai.v34i05.6239)
11. Aleksandar Botev, Soham De, Samuel L Smith, Anushan Fernando, George-Cristian Muraru, Ruba Haroun, Leonard Berrada, Razvan Pascanu, Pier Giuseppe Sessa, Robert Dadashi, Léonard Hussenot, Johan Ferret, Sertan Girgin, Olivier Bachem, Alek Andreev, Kathleen Kenealy, Thomas Mesnard, Cassidy Hardin, Sury… [find](https://scholar.google.com/scholar?q=Aleksandar+Botev%2C+Soham+De%2C+Samuel+L+Smith%2C+Anushan+Fernando%2C+George%2DCristian+Muraru%2C+Ruba+Haroun%2C+Leonard+Berrada%2C+Razvan+Pascanu%2C+Pier+Giuseppe+Sessa%2C+Robert+Dadashi%2C+L%C3%A9onard+Hussenot%2C+Johan+Ferret%2C)
12. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020. URL https://arxiv.org/abs/ 20… [↗](https://arxiv.org/abs/)
13. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. [Sparks of artificial general intelligence: Early experiments with gpt-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/). arXiv preprint ar…
14. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
15. Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv: 2306.15595, 2023a. URL https://arxiv.org/abs/2306.15595v2. [↗](https://arxiv.org/abs/2306.15595v2)
16. Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. [Longlora: Efficient fine-tuning of long-context large language models](https://md.research.elamur.ai/free/en/papers/chen-2023-longlora-efficient-fine-tuning-long/). International Conference on Learning Representations, 2023b. doi: 10.48550/arXiv.2309.12307. URL https://arxiv.org/abs/2309.12307v1. [↗](https://arxiv.org/abs/2309.12307v1)
17. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. [Generating long sequences with sparse transformers](https://md.research.elamur.ai/free/en/papers/child-2019-generating-long-sequences-sparse-transformers/). PREPRINT, 2019. URL https://arxiv.org/abs/1904.10509v1. [↗](https://arxiv.org/abs/1904.10509v1)
18. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. [Boolq: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapte…
19. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv preprint arXiv: 1803.05457, 2018. URL https://arxiv.org/abs/1803.05457v1. [↗](https://arxiv.org/abs/1803.05457v1)
20. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv: 2110.14168, 2021. URL [https://arxiv.org/a…](https://arxiv.org/a…) [↗](https://arxiv.org/a…)
21. Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. ACL, 2022. URL https://arxiv.org/abs/2104.08696v2. [↗](https://arxiv.org/abs/2104.08696v2)
22. Tri Dao. [Flashattention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/). arXiv preprint arXiv: 2307.08691, 2023. URL https://arxiv.org/abs/2307.08691v1. [↗](https://arxiv.org/abs/2307.08691v1)
23. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems, 2022a. [find](https://scholar.google.com/scholar?q=Tri+Dao%2C+Daniel+Y%2E+Fu%2C+Stefano+Ermon%2C+Atri+Rudra%2C+and+Christopher+R%C3%A9%2E+FlashAttention%3A+Fast+and+memory%2Defficient+exact+attention+with+IO%2Dawareness%2E+In+Advances+in+Neural+Information+Processing+Systems%2C)
24. Tri Dao, Daniel Y. Fu, Khaled Kamal Saab, A. Thomas, A. Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models. International Conference On Learning Representations, 2022b. doi: 10.48550/arXiv.2212.14052. URL https://arxiv.org/abs/2212.14052v3. [↗](https://arxiv.org/abs/2212.14052v3)
25. Y. Dauphin, Angela Fan, Michael Auli, and David Grangier. [Language modeling with gated convolutional networks](https://md.research.elamur.ai/free/en/papers/dauphin-2016-language-modeling-gated-convolutional-networks/). International Conference On Machine Learning, 2016. URL https://arxiv.org/abs/1612. 08083v3. [↗](https://arxiv.org/abs/1612)
26. Soham De, Samuel L. Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, Guillaume Desjardins, Arnaud Doucet, David Budden, Yee Whye Teh, Razvan Pascanu, Nando De Freitas, and Caglar Gulcehre. Griffin: Mixing g… [find](https://scholar.google.com/scholar?q=Soham+De%2C+Samuel+L%2E+Smith%2C+Anushan+Fernando%2C+Aleksandar+Botev%2C+George+Cristian%2DMuraru%2C+Albert+Gu%2C+Ruba+Haroun%2C+Leonard+Berrada%2C+Yutian+Chen%2C+Srivatsan+Srinivasan%2C+Guillaume+Desjardins%2C+Arnaud+Doucet%2C+)
27. Yiran Ding, L. Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. [Longrope: Extending llm context window beyond 2 million tokens](https://md.research.elamur.ai/free/en/papers/ding-2024-longrope-extending-llm-context/). International Conference on Machine Learning, 2024. doi: 10.48550/arXiv.2402.13753.
28. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez,… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
29. Stefan Elfwing, E. Uchibe, and K. Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Networks, 2017. doi: 10.1016/j.neunet.2017.12.012. — [doi:10.1016/j.neunet.2017.12.012](https://doi.org/10.1016/j.neunet.2017.12.012) [find](https://scholar.google.com/scholar?q=Stefan+Elfwing%2C+E%2E+Uchibe%2C+and+K%2E+Doya%2E+Sigmoid%2Dweighted+linear+units+for+neural+network+function+approximation+in+reinforcement+learning%2E+Neural+Networks%2C+2017%2E+doi%3A+10%2E1016%2Fj%2Eneunet%2E2017%2E12%2E012%2E+%E2%80%94+%5B)
30. Mahan Fathi, Jonathan Pilault, Orhan Firat, Christopher Pal, Pierre-Luc Bacon, and Ross Goroshin. [Block-state transformers](https://md.research.elamur.ai/free/en/papers/fathi-2023-block-state-transformers/). NEURIPS, 2023. URL https://arxiv.org/abs/2306.09539v4. [↗](https://arxiv.org/abs/2306.09539v4)
31. Daniel Y Fu, Tri Dao, Khaled Kamal Saab, Armin W Thomas, Atri Rudra, and Christopher Re. Hungry hungry hippos: Towards language modeling with state space models. In The Eleventh International Conference on Learning Representations, 2023. URL [https://openreview.net/forum?id=COZDy0WYGg](https://openreview.net/forum?id=COZDy0WYGg). [↗](https://openreview.net/forum?id=COZDy0WYGg)
32. Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. [find](https://scholar.google.com/scholar?q=Albert+Gu+and+Tri+Dao%2E+Mamba%3A+Linear%2Dtime+sequence+modeling+with+selective+state+spaces%2E+arXiv+preprint+arXiv%3A2312%2E00752%2C+2023%2E)
33. Albert Gu, Karan Goel, and Christopher R’e. Efficiently modeling long sequences with structured state spaces. International Conference On Learning Representations, 2021. [find](https://scholar.google.com/scholar?q=Albert+Gu%2C+Karan+Goel%2C+and+Christopher+R%E2%80%99e%2E+Efficiently+modeling+long+sequences+with+structured+state+spaces%2E+International+Conference+On+Learning+Representations%2C+2021%2E)
34. Albert Gu, Ankit Gupta, Karan Goel, and Christopher Ré. On the parameterization and initialization of diagonal state space models. ARXIV.ORG, 2022. doi: 10.48550/arXiv.2206.11893. — [doi:10.48550/arxiv.2206.11893](https://doi.org/10.48550/arxiv.2206.11893) [find](https://scholar.google.com/scholar?q=Albert+Gu%2C+Ankit+Gupta%2C+Karan+Goel%2C+and+Christopher+R%C3%A9%2E+On+the+parameterization+and+initialization+of+diagonal+state+space+models%2E+ARXIV%2EORG%2C+2022%2E+doi%3A+10%2E48550%2FarXiv%2E2206%2E11893%2E+%E2%80%94+%5Bdoi%3A10%2E48550%2Farxi)
35. Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Simple on-the-fly length generalization for large language models. arXiv preprint arXiv: 2308.16137, 2023. URL https://arxiv.org/abs/2308.16137v3. — [arXiv:2308.16137](https://arxiv.org/abs/2308.16137) [↗](https://arxiv.org/abs/2308.16137v3)
36. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. [Deep residual learning for image recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/). CVPR, 2016. URL https://arxiv.org/abs/1512.03385v1. [↗](https://arxiv.org/abs/1512.03385v1)
37. Yihui He, Jianing Qian, Jianren Wang, Cindy X. Le, Congrui Hetang, Qi Lyu, Wenping Wang, and Tianwei Yue. Depth-wise decomposition for accelerating separable convolutions in efficient convolutional neural networks. arXiv preprint arXiv: 1910.09455, 2019. URL https://arxiv.org/abs/1910.09455v3. — [arXiv:1910.09455](https://arxiv.org/abs/1910.09455) [↗](https://arxiv.org/abs/1910.09455v3)
38. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.
39. OpenReview.net, 2021. URL [https://openreview.net/forum?id=d7KBjmI3GmQ](https://openreview.net/forum?id=d7KBjmI3GmQ). [↗](https://openreview.net/forum?id=d7KBjmI3GmQ)
40. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. [The curious case of neural text degeneration](https://md.research.elamur.ai/free/en/papers/holtzman-2019-curious-case-neural-text-degeneration/). International Conference on Learning Representations, 2019. URL https://arxiv.org/abs/1904.09751v2. [↗](https://arxiv.org/abs/1904.09751v2)
41. Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. [Efficient attentions for long document summarization](https://md.research.elamur.ai/free/en/papers/huang-2021-efficient-attentions-long-document-summarization/). Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1419–1436, 2021.
42. Samy Jelassi, David Brandfonbrener, Sham M. Kakade, and Eran Malach. Repeat after me: Transformers are better than state space models at copying. arXiv preprint arXiv: 2402.01032, 2024. URL https://arxiv.org/abs/2402.01032v1. [↗](https://arxiv.org/abs/2402.01032v1)
43. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
44. Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu. [Llm maybe longlm: Self-extend llm context window without tuning](https://md.research.elamur.ai/free/en/papers/jin-2024-llm-maybe-longlm-self-extend/). arXiv preprint arXiv: 2401.01325, 2024. URL https://arxiv.org/abs/2401.01325v1. [↗](https://arxiv.org/abs/2401.01325v1)
45. Tobias Katsch. Gateloop: Fully data-controlled linear recurrence for sequence modeling. arXiv preprint arXiv: 2311.01927, 2023. URL https://arxiv.org/abs/2311.01927v1. — [arXiv:2311.01927](https://arxiv.org/abs/2311.01927) [↗](https://arxiv.org/abs/2311.01927v1)
46. Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. [Reformer: The efficient transformer](https://md.research.elamur.ai/free/en/papers/kitaev-2020-reformer-efficient-transformer/). arXiv preprint arXiv:2001.04451, 2020.
47. Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. [Textbooks are all you need ii: phi-1.5 technical report](https://md.research.elamur.ai/free/en/papers/li-2023-textbooks-all-need-ii-phi/). arXiv preprint arXiv: 2309.05463, 2023. URL https://arxiv.org/abs/2309.05463v1. [↗](https://arxiv.org/abs/2309.05463v1)
48. Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avashalom Manevich, Nir Ratner, Noam Rozen, Erez Shwartz, Mor Zusman,… — [Jamba: A Hybrid Transformer-Mamba Language Model](https://md.research.elamur.ai/free/en/papers/lieber-2024-jamba-hybrid-transformer-mamba-language/)
49. Stephanie Lin, Jacob Hilton, and Owain Evans. [TruthfulQA: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3214–3252, D…
50. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+Decoupled+weight+decay+regularization%2E+In+International+Conference+on+Learning+Representations%2C+2018%2E)
51. Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. Mega: Moving average equipped gated attention. In The Eleventh International Conference on Learning Representations, 2023. URL [https://openreview.net/forum?id=qNLe3iq2El](https://openreview.net/forum?id=qNLe3iq2El). [↗](https://openreview.net/forum?id=qNLe3iq2El)
52. Xuezhe Ma, Xiaomeng Yang, Wenhan Xiong, Beidi Chen, Lili Yu, Hao Zhang, Jonathan May, Luke Zettlemoyer, Omer Levy, and Chunting Zhou. Megalodon: Efficient llm pretraining and inference with unlimited context length. arXiv preprint arXiv: 2404.08801, 2024. URL https://arxiv.org/abs/2404.08801v1. [↗](https://arxiv.org/abs/2404.08801v1)
53. Eric Martin and Chris Cundy. Parallelizing linear recurrent neural nets over sequence length. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. [find](https://scholar.google.com/scholar?q=Eric+Martin+and+Chris+Cundy%2E+Parallelizing+linear+recurrent+neural+nets+over+sequence+length%2E+In+6th+International+Conference+on+Learning+Representations%2C+ICLR+2018%2C+Vancouver%2C+BC%2C+Canada%2C+April+30+%2D+)
54. OpenReview.net, 2018. URL [https://openreview.net/forum?id=HyUNwulC-](https://openreview.net/forum?id=HyUNwulC-). [↗](https://openreview.net/forum?id=HyUNwulC-)
55. Harsh Mehta, Ankit Gupta, Ashok Cutkosky, and Behnam Neyshabur. Long range language modeling via gated state spaces. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. [find](https://scholar.google.com/scholar?q=Harsh+Mehta%2C+Ankit+Gupta%2C+Ashok+Cutkosky%2C+and+Behnam+Neyshabur%2E+Long+range+language+modeling+via+gated+state+spaces%2E+In+The+Eleventh+International+Conference+on+Learning+Representations%2C+ICLR+2023%2C+Ki)
56. OpenReview.net, 2023. URL [https://openreview.net/forum?id=5MkYIYCbva](https://openreview.net/forum?id=5MkYIYCbva). [↗](https://openreview.net/forum?id=5MkYIYCbva)
57. William Merrill, Jackson Petty, and Ashish Sabharwal. The illusion of state in state-space models. arXiv preprint arXiv: 2404.08819, 2024. URL https://arxiv.org/abs/2404.08819v1. [↗](https://arxiv.org/abs/2404.08819v1)
58. MetaAI. Introducing meta llama 3: The most capable openly available llm to date, 2024. URL: [https://ai.meta.com/blog/meta-llama-3/](https://ai.meta.com/blog/meta-llama-3/). [↗](https://ai.meta.com/blog/meta-llama-3/)
59. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. Conference on Empirical Methods in Natural Language Processing, 2018. doi: 10.18653/v1/D18-1260. URL https://arxiv.org/abs/1809.02789v1. [↗](https://arxiv.org/abs/1809.02789v1)
60. Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access infinite context length for transformers. arXiv preprint arXiv: 2305.16300, 2023. URL https://arxiv.org/abs/2305.16300v2. [↗](https://arxiv.org/abs/2305.16300v2)
61. Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. Leave no context behind: Efficient infinite context transformers with infini-attention. arXiv preprint arXiv: 2404.07143, 2024. URL https://arxiv.org/abs/2404.07143v1. [↗](https://arxiv.org/abs/2404.07143v1)
62. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). PREPRINT, 2023. URL https://arxiv.org/abs/2303.08774v4. [↗](https://arxiv.org/abs/2303.08774v4)
63. Antonio Orvieto, Samuel L. Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. International Conference on Machine Learning, 2023. doi: 10.48550/arXiv.2303.06349. URL https://arxiv.org/abs/2303.06349v1. [↗](https://arxiv.org/abs/2303.06349v1)
64. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Q. N. Pham, R. Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and R. Fernández. [The lambada dataset: Word prediction requiring a broad discourse context](https://md.research.elamur.ai/free/en/papers/paperno-2016-lambada-dataset-word-prediction-requiring/). Annual Meeting of the Association for Computational Linguistics, 2016. doi: 10.18653…
65. Jongho Park, Jaeseung Park, Zheyang Xiong, Nayoung Lee, Jaewoong Cho, Samet Oymak, Kangwook Lee, and Dimitris Papailiopoulos. Can mamba learn how to learn? a comparative study on in-context learning tasks. arXiv preprint arXiv: 2402.04248, 2024. URL https://arxiv.org/abs/2402.04248v1. [↗](https://arxiv.org/abs/2402.04248v1)
66. Michael Poli, Stefano Massaroli, Eric Q. Nguyen, Daniel Y. Fu, Tri Dao, S. Baccus, Y. Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. International Conference On Machine Learning, 2023. doi: 10.48550/arXiv.2302.10866. URL [https://arxiv.org/ab…](https://arxiv.org/ab…) [↗](https://arxiv.org/ab…)
67. Ofir Press, Noah A. Smith, and M. Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. International Conference On Learning Representations, 2021. URL https://arxiv.org/abs/2108.12409v2. [↗](https://arxiv.org/abs/2108.12409v2)
68. Zhen Qin, Songlin Yang, and Yiran Zhong. Hierarchically gated recurrent neural network for sequence modeling. Neural Information Processing Systems, 2023. doi: 10.48550/arXiv.2311.04823. URL https://arxiv.org/abs/2311.04823v1. [↗](https://arxiv.org/abs/2311.04823v1)
69. Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, and Yiran Zhong. Hgrn2: Gated linear rnns with state expansion. arXiv preprint arXiv: 2404.07904, 2024. URL https://arxiv.org/abs/ 2404.07904v1. — [arXiv:2404.07904](https://arxiv.org/abs/2404.07904) [↗](https://arxiv.org/abs/)
70. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. arXiv preprint, 2019. URL [https://api.semanticscholar.org/](https://api.semanticscholar.org/) CorpusID:160025533. [↗](https://api.semanticscholar.org/)
71. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. [Squad: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). EMNLP, 2016. URL https://arxiv.org/abs/1606.05250v3. [↗](https://arxiv.org/abs/1606.05250v3)
72. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof qa benchmark. arXiv preprint arXiv: 2311.[12022, 2023](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/). URL https://arxiv.org/abs/2311.12022v1. [↗](https://arxiv.org/abs/2311.12022v1)
73. Liliang Ren, Yang Liu, Shuohang Wang, Yichong Xu, Chenguang Zhu, and ChengXiang Zhai. Sparse modular activation for efficient sequence modeling. NEURIPS, 2023. URL https://arxiv.org/abs/2306.11197v1. — [arXiv:2306.11197](https://arxiv.org/abs/2306.11197) [↗](https://arxiv.org/abs/2306.11197v1)
74. Aurko Roy, M. Saffar, Ashish Vaswani, and David Grangier. [Efficient content-based sparse attention with routing transformers](https://md.research.elamur.ai/free/en/papers/roy-2020-efficient-content-based-sparse-attention/). International Conference On Topology, Algebra And Categories In Logic, 2020. doi: 10.1162/tacl_a_00353.
75. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021. URL https://arxiv.org/abs/1907.10641v2. [↗](https://arxiv.org/abs/1907.10641v2)
76. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. [Socialiqa: Commonsense reasoning about social interactions](https://md.research.elamur.ai/free/en/papers/sap-2019-socialiqa-commonsense-reasoning-about-social/). arXiv preprint arXiv: 1904.09728, 2019. URL https://arxiv.org/abs/1904. 09728v3. [↗](https://arxiv.org/abs/1904)
77. Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine L… [find](https://scholar.google.com/scholar?q=Imanol+Schlag%2C+Kazuki+Irie%2C+and+J%C3%BCrgen+Schmidhuber%2E+Linear+transformers+are+secretly+fast+weight+programmers%2E+In+Marina+Meila+and+Tong+Zhang+%28eds%2E%29%2C+Proceedings+of+the+38th+International+Conference+on)
78. PMLR, 2021. URL [http://proceedings.mlr.press/v139/schlag21a.html](http://proceedings.mlr.press/v139/schlag21a.html). [↗](http://proceedings.mlr.press/v139/schlag21a.html)
79. Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. [Flashattention-3: Fast and accurate attention with asynchrony and low-precision](https://md.research.elamur.ai/free/en/papers/shah-2024-flashattention-3-fast-accurate-attention/). arXiv preprint arXiv: 2407.08608, 2024. URL https://arxiv.org/abs/2407.08608v2. [↗](https://arxiv.org/abs/2407.08608v2)
80. Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. [Zeroscrolls: A zero-shot benchmark for long text understanding](https://md.research.elamur.ai/free/en/papers/shaham-2023-zeroscrolls-zero-shot-benchmark-long/). Conference on Empirical Methods in Natural Language Processing, 2023. doi: 10.48550/arXiv.2305.14196. URL https://arxiv.org/abs/2305.14196v2. [↗](https://arxiv.org/abs/2305.14196v2)
81. Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv: 1911.02150, 2019. URL https://arxiv.org/abs/1911.02150v1. [↗](https://arxiv.org/abs/1911.02150v1)
82. Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv: 2002.05202, 2020. URL https://arxiv.org/abs/2002.05202v1. [↗](https://arxiv.org/abs/2002.05202v1)
83. Jimmy T.H. Smith, Andrew Warrington, and Scott Linderman. Simplified state space layers for sequence modeling. In The Eleventh International Conference on Learning Representations, 2023. URL [https://openreview.net/forum?id=Ai8Hw3AXqks](https://openreview.net/forum?id=Ai8Hw3AXqks). [↗](https://openreview.net/forum?id=Ai8Hw3AXqks)
84. Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. Slimpajama: A 627b token cleaned and deduplicated version of redpajama, 2023. URL: [https://www.cerebras.net/](https://www.cerebras.net/) blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama. [↗](https://www.cerebras.net/)
85. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). arXiv preprint arXiv: 2104.09864, 2021.
86. Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. [Retentive network: A successor to transformer for large language models](https://md.research.elamur.ai/free/en/papers/sun-2023-retentive-network-successor-transformer-large/). arXiv preprint arXiv:2307.08621, 2023.
87. Gemma Team. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv: 2403.08295, 2024. URL https://arxiv.org/abs/2403.08295v1. [↗](https://arxiv.org/abs/2403.08295v1)
88. Jamba Team, Barak Lenz, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, Daniel Gissin, Daniel Jannai, Dor Muhlgay, Dor Zimberg, Edden M Gerber, Elad Dolev, Eran Krakovsky, Erez Safahi, Erez Schwartz, Gal Cohen, Gal Shachaf, Haim Rozenblu… — [Jamba-1.5: Hybrid Transformer-Mamba Models at Scale](https://md.research.elamur.ai/free/en/papers/lenz-2024-jamba-1-5-hybrid-transformer/)
89. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
90. Szymon Tworkowski, Konrad Staniszewski, Mikołaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Miłoś. Focused transformer: Contrastive training for context scaling. NEURIPS, 2023. URL https://arxiv.org/abs/2307.03170v2. — [arXiv:2307.03170](https://arxiv.org/abs/2307.03170) [↗](https://arxiv.org/abs/2307.03170v2)
91. Dusan Varis and Ondřej Bojar. Sequence length is a domain: Length-based overfitting in transformer models. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 8246–8257, Onli… [find](https://scholar.google.com/scholar?q=Dusan+Varis+and+Ondr%CC%8Cej+Bojar%2E+Sequence+length+is+a+domain%3A+Length%2Dbased+overfitting+in+transformer+models%2E+In+Marie%2DFrancine+Moens%2C+Xuanjing+Huang%2C+Lucia+Specia%2C+and+Scott+Wen%2Dtau+Yih+%28eds%2E%29%2C+Proceed)
92. Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). NIPS, 2017.
93. Alex Wang, Richard Yuanzhe Pang, Angelica Chen, Jason Phang, and Samuel R. Bowman. Squality: Building a long-document summarization dataset the hard way. Conference on Empirical Methods in Natural Language Processing, 2022. doi: 10.48550/arXiv.2205.11465. URL https://arxiv.org/abs/2205.11465v1. [↗](https://arxiv.org/abs/2205.11465v1)
94. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. [Mmlu-pro: A more robust and challenging multi-task language understanding benchmark](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/). arXi…
95. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, E. Chi, F. Xia, Quoc Le, and Denny Zhou. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Neural Information Processing Systems, 2022. URL https://arxiv.org/abs/2201.11903v6. [↗](https://arxiv.org/abs/2201.11903v6)
96. Kaiyue Wen, Xingyu Dang, and Kaifeng Lyu. Rnns are not transformers (yet): The key bottleneck on in-context retrieval. arXiv preprint arXiv: 2402.18510, 2024. URL https://arxiv.org/abs/2402.18510v1. [↗](https://arxiv.org/abs/2402.18510v1)
97. Yuhuai Wu, Markus N. Rabe, DeLesley S. Hutchins, and Christian Szegedy. Memorizing transformers. International Conference On Learning Representations, 2022. doi: 10.48550/arXiv.2203.08913. URL https://arxiv.org/abs/2203.08913v1. [↗](https://arxiv.org/abs/2203.08913v1)
98. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv: 2309.17453, 2023. URL https://arxiv.org/abs/2309. 17453v1. [↗](https://arxiv.org/abs/2309)
99. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. [find](https://scholar.google.com/scholar?q=Guangxuan+Xiao%2C+Yuandong+Tian%2C+Beidi+Chen%2C+Song+Han%2C+and+Mike+Lewis%2E+Efficient+streaming+language+models+with+attention+sinks%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2C+ICLR)
100. OpenReview.net, 2024. URL [https://openreview.net/forum?id=NG7sS51zVF](https://openreview.net/forum?id=NG7sS51zVF). [↗](https://openreview.net/forum?id=NG7sS51zVF)
101. Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie-Yan Liu. On layer normalization in the transformer architecture. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event,… [find](https://scholar.google.com/scholar?q=Ruibin+Xiong%2C+Yunchang+Yang%2C+Di+He%2C+Kai+Zheng%2C+Shuxin+Zheng%2C+Chen+Xing%2C+Huishuai+Zhang%2C+Yanyan+Lan%2C+Liwei+Wang%2C+and+Tie%2DYan+Liu%2E+On+layer+normalization+in+the+transformer+architecture%2E+In+Proceedings+)
102. PMLR, 2020. URL [http://proceedings.mlr.press/v119/xiong20b.html](http://proceedings.mlr.press/v119/xiong20b.html). [↗](http://proceedings.mlr.press/v119/xiong20b.html)
103. Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma.… — [Effective Long-Context Scaling of Foundation Models](https://md.research.elamur.ai/free/en/papers/xiong-2023-effective-long-context-scaling-foundation/)
104. Songlin Yang and Yu Zhang. Fla: A triton-based library for hardware-efficient implementations of linear attention mechanism, January 2024. URL [https://github.com/sustcsonglin/flash-linear-attention](https://github.com/sustcsonglin/flash-linear-attention). [↗](https://github.com/sustcsonglin/flash-linear-attention)
105. Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. arXiv preprint arXiv:2312.06635, 2023. [find](https://scholar.google.com/scholar?q=Songlin+Yang%2C+Bailin+Wang%2C+Yikang+Shen%2C+Rameswar+Panda%2C+and+Yoon+Kim%2E+Gated+linear+attention+transformers+with+hardware%2Defficient+training%2E+arXiv+preprint+arXiv%3A2312%2E06635%2C+2023%2E)
106. Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. arXiv preprint arXiv: 2406.06484, 2024. URL https://arxiv.org/abs/ 2406.06484v1. [↗](https://arxiv.org/abs/)
107. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. [Big bird: Transformers for longer sequences](https://md.research.elamur.ai/free/en/papers/zaheer-2020-big-bird-transformers-longer-sequences/). Advances in neural information processing systems, 33:17283–17297, 2020. URL [https://arxiv.org/…](https://arxiv.org/…) [↗](https://arxiv.org/…)
108. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? Annual Meeting of the Association for Computational Linguistics, 2019. doi: 10.18653/v1/P19-1472. URL https://arxiv.org/abs/1905.07830v1. [↗](https://arxiv.org/abs/1905.07830v1)
109. Biao Zhang and Rico Sennrich. Root mean square layer normalization. Neural Information Processing Systems, 2019. doi: 10.5167/UZH-177483. URL https://arxiv.org/abs/1910.07467v1. [↗](https://arxiv.org/abs/1910.07467v1)
110. Edward Ayers Zhangir Azerbayev and Bartosz Piotrowski. Proof-pile, 2022. URL: [https://github.com/](https://github.com/) zhangir-azerbayev/proof-pile. [↗](https://github.com/)
111. Hao Zheng, Zhanlei Yang, Wenju Liu, Jizhong Liang, and Yanpeng Li. Improving deep neural networks using softplus units. 2015 International Joint Conference on Neural Networks (IJCNN), pp. 1–4, 2015. doi: 10.1109/IJCNN.2015.7280459. URL [https://ieeexplore.ieee.org/document/7280459](https://ieeexplore.ieee.org/document/7280459). — [doi:10.1109/ijcnn.2015.7280459](https://doi.org/10.1109/ijcnn.2015.7280459) [↗](https://ieeexplore.ieee.org/document/7280459)
112. Simiao Zuo, Xiaodong Liu, Jian Jiao, Denis Charles, Eren Manavoglu, Tuo Zhao, and Jianfeng Gao. Efficient long sequence modeling via state space augmented transformer. arXiv preprint arXiv: 2212.08136, 2022. URL https://arxiv.org/abs/2212.08136v1. — [arXiv:2212.08136](https://arxiv.org/abs/2212.08136) [↗](https://arxiv.org/abs/2212.08136v1)

*Updated 30 October 2026*