---
title: "Linearizing Large Language Models — Mercat 2024"
description: "The authors propose SUPRA (Scalable UPtraining for Recurrent Attention), which converts an… 44 references, cited by 16 papers here."
authors: ["Jean Mercat", "Igor Vasiljevic", "Sedrick Keh", "Kushal Arora", "Achal Dave", "Adrien Gaidon", "Thomas Kollar"]
year: 2024
venue: "arXiv:2405.06640"
source: "https://arxiv.org/abs/2405.06640"
doi: "10.48550/arxiv.2405.06640"
arxiv: "2405.06640"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mercat-2024-linearizing-large-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mercat-2024-linearizing-large-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/) / Linearizing Large Language Models

# Linearizing Large Language Models

**Authors:** [Jean Mercat](https://md.research.elamur.ai/free/en/authors/jean-mercat/), Igor Vasiljevic, [Sedrick Keh](https://md.research.elamur.ai/free/en/authors/sedrick-keh/), [Kushal Arora](https://md.research.elamur.ai/free/en/authors/kushal-arora/), Achal Dave, Adrien Gaidon and [Thomas Kollar](https://md.research.elamur.ai/free/en/authors/thomas-kollar/) · **Year:** 2024 · **Venue:** arXiv:2405.06640 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2405.06640) · [PDF](https://arxiv.org/pdf/2405.06640) · [DOI 10.48550/arxiv.2405.06640](https://doi.org/10.48550/arxiv.2405.06640) · [arXiv 2405.06640](https://arxiv.org/abs/2405.06640) · [OpenAlex](https://openalex.org/W4396907554)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [16](https://md.research.elamur.ai/free/en/papers/mercat-2024-linearizing-large-language-models/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Linear-attention models and other recurrent models keep a fixed-size state during generation, which makes inference cheaper than in softmax transformers. Training them from scratch at LLM scale, however, takes a great deal of compute and high-quality data. The authors propose SUPRA (Scalable UPtraining for Recurrent Attention), which converts an existing pre-trained transformer into an RNN through continued training on the modified architecture. Softmax attention is swapped for a linear kernel: a trainable ReLU MLP, shared by queries and keys, followed by RoPE, a fixed per-head decay and GroupNorm in place of the usual normalization factor. The model still trains in parallel like a transformer but runs recurrently at inference. Llama2-7B and Mistral-7B were uptrained on RefinedWeb at about 5% of pre-training cost. The resulting models beat RWKV-5 and roughly match a Mamba-7B trained from scratch on standard NLU benchmarks. Both uptrained and natively recurrent models still fall short on in-context learning and long-context tasks. This makes SUPRA a cheap testbed for studying recurrent LLMs.

## Key points
- SUPRA builds on T2R (Kasai et al., 2021). T2R needed about 20% of pre-training tokens and lost 5–10% on benchmarks, and it was unstable beyond roughly the 100M scale. SUPRA's GroupNorm-based normalization lets large models uptrain stably.
- As a strong recurrent baseline, the authors trained a Mamba-7B from scratch for 1.2T tokens on RefinedWeb. They release it along with Mistral-SUPRA and their code (TRI-ML/linear_open_lm).
- Persistent weaknesses of the linearized models include MMLU (in-context learning) and the long-context tasks NarrativeQA and Qasper. These abilities do not carry over from the softmax base model.
- The training setup uses a fork of OpenLM with Lightning Attention 2 kernels and evaluation with the Eleuther harness. The default sequence length is 2048. Hardware ranged from 4 to 32 nodes of 8×H100, with about 4300 tokens/s per GPU for 7B uptraining.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Toyota Research Institute](https://md.research.elamur.ai/free/en/labs/toyota-research-institute/) — Research institute, Palo Alto, United States · 3 works · 116 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Jean Mercat, Igor Vasiljevic, Sedrick Keh, Kushal Arora, Achal Dave, Adrien Gaidon and Thomas Kollar. 2024. Linearizing Large Language Models. arXiv:2405.06640.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{mercat2024linearizing,
  title = {{Linearizing Large Language Models}},
  author = {Jean Mercat, Igor Vasiljevic, Sedrick Keh, Kushal Arora, Achal Dave, Adrien Gaidon and Thomas Kollar},
  year = {2024},
  doi = {10.48550/arxiv.2405.06640},
  eprint = {2405.06640},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.06640}
}
```

- [← previous LexSumm and LexT5: Benchmarking and Modeling Legal Summarization Tasks in English](https://md.research.elamur.ai/free/en/papers/s-2024-lexsumm-lext5-benchmarking-modeling-legal/)
- [next → LitLLMs, LLMs for Literature Review: Are we there yet?](https://md.research.elamur.ai/free/en/papers/agarwal-2024-litllms-llms-literature-review-there/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/)

## References

44 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. [Gqa: Training generalized multi-query transformer models from multi-head checkpoints](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/). arXiv preprint arXiv:2305.13245, 2023.
2. Ekin Akyürek, Bailin Wang, Yoon Kim, and Jacob Andreas. In-context language learning: Arhitectures and algorithms. arXiv preprint arXiv:2401.12973, 2024. [find](https://scholar.google.com/scholar?q=Ekin+Akyu%CC%88rek%2C+Bailin+Wang%2C+Yoon+Kim%2C+and+Jacob+Andreas%2E+In%2Dcontext+language+learning%3A+Arhitectures+and+algorithms%2E+arXiv+preprint+arXiv%3A2401%2E12973%2C+2024%2E)
3. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. [Layer normalization](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/). arXiv preprint arXiv:1607.06450, 2016.
4. Iz Beltagy, Matthew E Peters, and Arman Cohan. [Longformer: The long-document transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/). arXiv preprint arXiv:2004.05150, 2020.
5. Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. Gpt-neox-20b: An open-source autoregressive… — [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://md.research.elamur.ai/free/en/papers/black-2022-gpt-neox-20b-open-source/)
6. Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. [Rethinking attention with performers](https://md.research.elamur.ai/free/en/papers/choromanski-2020-rethinking-attention-performers/). arXiv preprint arXiv:2009.14794, 2020.
7. Tri Dao, Daniel Y Fu, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models. arXiv preprint arXiv:2212.14052, 2022. [find](https://scholar.google.com/scholar?q=Tri+Dao%2C+Daniel+Y+Fu%2C+Khaled+K+Saab%2C+Armin+W+Thomas%2C+Atri+Rudra%2C+and+Christopher+Re%CC%81%2E+Hungry+hungry+hippos%3A+Towards+language+modeling+with+state+space+models%2E+arXiv+preprint+arXiv%3A2212%2E14052%2C+2022%2E)
8. Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. [A dataset of information-seeking questions and answers anchored in research papers](https://md.research.elamur.ai/free/en/papers/dasigi-2021-dataset-information-seeking-questions-answers/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language…
9. Soham De, Samuel L Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, et al. Griffin: Mixing gated linear recurrences with local attention for efficient language models. arXiv preprint arXiv:2402.19427, 2024.… [find](https://scholar.google.com/scholar?q=Soham+De%2C+Samuel+L+Smith%2C+Anushan+Fernando%2C+Aleksandar+Botev%2C+George+Cristian%2DMuraru%2C+Albert+Gu%2C+Ruba+Haroun%2C+Leonard+Berrada%2C+Yutian+Chen%2C+Srivatsan+Srinivasan%2C+et+al%2E+Griffin%3A+Mixing+gated+linear+re)
10. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Ani… [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Baber+Abbasi%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Alain+Le+Noac%E2%80%99h%2C+Haonan+Li%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Chris+Oci)
11. Alexsandar Botev Griffin Team, Soham De, Samuel L Smith, Anushan Fernando, George-Christian Muraru, Ruba Haroun, and Leonard Berrada et al. Recurrentgemma. arXiv preprint arXiv:2404.07839, 2024. [find](https://scholar.google.com/scholar?q=Alexsandar+Botev+Griffin+Team%2C+Soham+De%2C+Samuel+L+Smith%2C+Anushan+Fernando%2C+George%2DChristian+Muraru%2C+Ruba+Haroun%2C+and+Leonard+Berrada+et+al%2E+Recurrentgemma%2E+arXiv+preprint+arXiv%3A2404%2E07839%2C+2024%2E)
12. Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. [find](https://scholar.google.com/scholar?q=Albert+Gu+and+Tri+Dao%2E+Mamba%3A+Linear%2Dtime+sequence+modeling+with+selective+state+spaces%2E+arXiv+preprint+arXiv%3A2312%2E00752%2C+2023%2E)
13. Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021. [find](https://scholar.google.com/scholar?q=Albert+Gu%2C+Karan+Goel%2C+and+Christopher+Re%CC%81%2E+Efficiently+modeling+long+sequences+with+structured+state+spaces%2E+arXiv+preprint+arXiv%3A2111%2E00396%2C+2021%2E)
14. Suchin Gururangan, Mitchell Wortsman, Samir Yitzhak Gadre, Achal Dave, Maciej Kilian, Weijia Shi, Jean Mercat, Georgios Smyrnis, Gabriel Ilharco, Matt Jordan, Reinhard Heckel, Alex Dimakis, Ali Farhadi, Vaishaal Shankar, and Ludwig Schmidt. OpenLM: a minimal but performative language modeling (lm) r… [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Mitchell+Wortsman%2C+Samir+Yitzhak+Gadre%2C+Achal+Dave%2C+Maciej+Kilian%2C+Weijia+Shi%2C+Jean+Mercat%2C+Georgios+Smyrnis%2C+Gabriel+Ilharco%2C+Matt+Jordan%2C+Reinhard+Heckel%2C+Alex+Dimakis%2C+Ali+Farhad)
15. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735–1780, 1997. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A+1735%E2%80%931780%2C+1997%2E)
16. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+Mistral+7b)
17. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). arXiv preprint arXiv:2001.08361, 2020.
18. Jungo Kasai, Hao Peng, Yizhe Zhang, Dani Yogatama, Gabriel Ilharco, Nikolaos Pappas, Yi Mao, Weizhu Chen, and Noah A Smith. Finetuning pretrained transformers into rnns. arXiv preprint arXiv:2103.13076, 2021. [find](https://scholar.google.com/scholar?q=Jungo+Kasai%2C+Hao+Peng%2C+Yizhe+Zhang%2C+Dani+Yogatama%2C+Gabriel+Ilharco%2C+Nikolaos+Pappas%2C+Yi+Mao%2C+Weizhu+Chen%2C+and+Noah+A+Smith%2E+Finetuning+pretrained+transformers+into+rnns%2E+arXiv+preprint+arXiv%3A2103%2E1307)
19. Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pp. 5156–5165. PMLR, 2020. [find](https://scholar.google.com/scholar?q=Angelos+Katharopoulos%2C+Apoorv+Vyas%2C+Nikolaos+Pappas%2C+and+Franc%CC%A7ois+Fleuret%2E+Transformers+are+rnns%3A+Fast+autoregressive+transformers+with+linear+attention%2E+In+International+conference+on+machine+learni)
20. Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. [The NarrativeQA reading comprehension challenge](https://md.research.elamur.ai/free/en/papers/kocisky-2018-narrativeqa-reading-comprehension-challenge/). Transactions of the Association for Computational Linguistics, 6:317–328, 2018. doi: 10.1162/tacl a 00023. URL https://aclanthol…
21. Hao Liu, Matei Zaharia, and Pieter Abbeel. [Ring attention with blockwise transformers for near-infinite context](https://md.research.elamur.ai/free/en/papers/liu-2023-ring-attention-blockwise-transformers-near/). arXiv preprint arXiv:2310.01889, 2023.
22. Xuezhe Ma, Xiang Kong, Sinong Wang, Chunting Zhou, Jonathan May, Hao Ma, and Luke Zettlemoyer. Luna: Linear unified nested attention. Advances in Neural Information Processing Systems, 34:2441–2453, 2021. [find](https://scholar.google.com/scholar?q=Xuezhe+Ma%2C+Xiang+Kong%2C+Sinong+Wang%2C+Chunting+Zhou%2C+Jonathan+May%2C+Hao+Ma%2C+and+Luke+Zettlemoyer%2E+Luna%3A+Linear+unified+nested+attention%2E+Advances+in+Neural+Information+Processing+Systems%2C+34%3A2441%E2%80%932453%2C+2)
23. Jean Mercat. Higher order linear transformer. arXiv preprint arXiv:2010.14816, 2020. — [arXiv:2010.14816](https://arxiv.org/abs/2010.14816) [find](https://scholar.google.com/scholar?q=Jean+Mercat%2E+Higher+order+linear+transformer%2E+arXiv+preprint+arXiv%3A2010%2E14816%2C+2020%2E+%E2%80%94+%5BarXiv%3A2010%2E14816%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2010%2E14816%29)
24. Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. Leave no context behind: Efficient infinite context transformers with infini-attention. arXiv preprint arXiv:2404.07143, 2024. [find](https://scholar.google.com/scholar?q=Tsendsuren+Munkhdalai%2C+Manaal+Faruqui%2C+and+Siddharth+Gopal%2E+Leave+no+context+behind%3A+Efficient+infinite+context+transformers+with+infini%2Dattention%2E+arXiv+preprint+arXiv%3A2404%2E07143%2C+2024%2E)
25. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116… [find](https://scholar.google.com/scholar?q=Guilherme+Penedo%2C+Quentin+Malartic%2C+Daniel+Hesslow%2C+Ruxandra+Cojocaru%2C+Alessandro+Cappelli%2C+Hamza+Alobeidli%2C+Baptiste+Pannier%2C+Ebtesam+Almazrouei%2C+and+Julien+Launay%2E+The+refinedweb+dataset+for+falcon+)
26. Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048, 2023a. [find](https://scholar.google.com/scholar?q=Bo+Peng%2C+Eric+Alcaide%2C+Quentin+Anthony%2C+Alon+Albalak%2C+Samuel+Arcadinho%2C+Huanqi+Cao%2C+Xin+Cheng%2C+Michael+Chung%2C+Matteo+Grella%2C+Kranthi+Kiran+GV%2C+et+al%2E+Rwkv%3A+Reinventing+rnns+for+the+transformer+era%2E+ar)
27. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071, 2023b. [find](https://scholar.google.com/scholar?q=Bowen+Peng%2C+Jeffrey+Quesnelle%2C+Honglu+Fan%2C+and+Enrico+Shippole%2E+Yarn%3A+Efficient+context+window+extension+of+large+language+models%2E+arXiv+preprint+arXiv%3A2309%2E00071%2C+2023b%2E)
28. Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A Smith, and Lingpeng Kong. Random feature attention. arXiv preprint arXiv:2103.02143, 2021. [find](https://scholar.google.com/scholar?q=Hao+Peng%2C+Nikolaos+Pappas%2C+Dani+Yogatama%2C+Roy+Schwartz%2C+Noah+A+Smith%2C+and+Lingpeng+Kong%2E+Random+feature+attention%2E+arXiv+preprint+arXiv%3A2103%2E02143%2C+2021%2E)
29. Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. arXiv preprint arXiv:2302.10866, 2023. [find](https://scholar.google.com/scholar?q=Michael+Poli%2C+Stefano+Massaroli%2C+Eric+Nguyen%2C+Daniel+Y+Fu%2C+Tri+Dao%2C+Stephen+Baccus%2C+Yoshua+Bengio%2C+Stefano+Ermon%2C+and+Christopher+Re%CC%81%2E+Hyena+hierarchy%3A+Towards+larger+convolutional+language+models%2E+ar)
30. Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5, 2023. [find](https://scholar.google.com/scholar?q=Reiner+Pope%2C+Sholto+Douglas%2C+Aakanksha+Chowdhery%2C+Jacob+Devlin%2C+James+Bradbury%2C+Jonathan+Heek%2C+Kefan+Xiao%2C+Shivani+Agrawal%2C+and+Jeff+Dean%2E+Efficiently+scaling+transformer+inference%2E+Proceedings+of+Mac)
31. Zhen Qin, Xiaodong Han, Weixuan Sun, Dongxu Li, Lingpeng Kong, Nick Barnes, and Yiran Zhong. [The devil in linear transformer](https://md.research.elamur.ai/free/en/papers/qin-2022-devil-linear-transformer/). arXiv preprint arXiv:2210.10340, 2022a.
32. Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong. cosformer: Rethinking softmax in attention. arXiv preprint arXiv:2202.08791, 2022b. — [arXiv:2202.08791](https://arxiv.org/abs/2202.08791) [find](https://scholar.google.com/scholar?q=Zhen+Qin%2C+Weixuan+Sun%2C+Hui+Deng%2C+Dongxu+Li%2C+Yunshen+Wei%2C+Baohong+Lv%2C+Junjie+Yan%2C+Lingpeng+Kong%2C+and+Yiran+Zhong%2E+cosformer%3A+Rethinking+softmax+in+attention%2E+arXiv+preprint+arXiv%3A2202%2E08791%2C+2022b%2E+%E2%80%94+%5B)
33. Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong. [Lightning attention-2: A free lunch for handling unlimited sequence lengths in large language models](https://md.research.elamur.ai/free/en/papers/qin-2024-lightning-attention-2-free/). arXiv preprint arXiv:2401.04658, 2024.
34. Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, and Omer Levy. [SCROLLS: Standardized CompaRison over long language sequences](https://md.research.elamur.ai/free/en/papers/shaham-2022-scrolls-standardized-comparison-over-long/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1200…
35. Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, et al. [Scrolls: Standardized comparison over long language sequences](https://md.research.elamur.ai/free/en/papers/shaham-2022-scrolls-standardized-comparison-over-long/). arXiv preprint arXiv:2201.03533, 2022b.
36. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). arXiv preprint arXiv:2104.09864, 2021.
37. Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. [Retentive network: A successor to transformer for large language models](https://md.research.elamur.ai/free/en/papers/sun-2023-retentive-network-successor-transformer-large/). arXiv preprint arXiv:2307.08621, 2023.
38. Philippe Tillet, H. T. Kung, and David Cox. [Triton: an intermediate language and compiler for tiled neural network computations](https://md.research.elamur.ai/free/en/papers/tillet-2019-triton-intermediate-language-compiler-tiled/). In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, pp. 10–19, New York, NY, USA, 2019. Association for…
39. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023.
40. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30, 2017.
41. Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3–19, 2018. [find](https://scholar.google.com/scholar?q=Yuxin+Wu+and+Kaiming+He%2E+Group+normalization%2E+In+Proceedings+of+the+European+conference+on+computer+vision+%28ECCV%29%2C+pp%2E+3%E2%80%9319%2C+2018%2E)
42. Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma.… — [Effective Long-Context Scaling of Foundation Models](https://md.research.elamur.ai/free/en/papers/xiong-2023-effective-long-context-scaling-foundation/)
43. Shuangfei Zhai, Walter Talbott, Nitish Srivastava, Chen Huang, Hanlin Goh, Ruixiang Zhang, and Josh Susskind. An attention free transformer. arXiv preprint arXiv:2105.14103, 2021. — [arXiv:2105.14103](https://arxiv.org/abs/2105.14103) [find](https://scholar.google.com/scholar?q=Shuangfei+Zhai%2C+Walter+Talbott%2C+Nitish+Srivastava%2C+Chen+Huang%2C+Hanlin+Goh%2C+Ruixiang+Zhang%2C+and+Josh+Susskind%2E+An+attention+free+transformer%2E+arXiv+preprint+arXiv%3A2105%2E14103%2C+2021%2E+%E2%80%94+%5BarXiv%3A2105%2E14103%5D)
44. Michael Zhang, Kush Bhatia, Hermann Kumbong, and Christopher Ré. The hedgehog & the porcupine: Expressive linear attentions with softmax mimicry. arXiv preprint arXiv:2402.04347, 2024. — [arXiv:2402.04347](https://arxiv.org/abs/2402.04347) [find](https://scholar.google.com/scholar?q=Michael+Zhang%2C+Kush+Bhatia%2C+Hermann+Kumbong%2C+and+Christopher+Re%CC%81%2E+The+hedgehog+%26+the+porcupine%3A+Expressive+linear+attentions+with+softmax+mimicry%2E+arXiv+preprint+arXiv%3A2402%2E04347%2C+2024%2E+%E2%80%94+%5BarXiv%3A2402%2E)

*Updated 30 October 2026*