---
title: "Hymba: A Hybrid-head Architecture for Small Language Models"
description: "The design adds three further pieces: 128 learnable meta tokens are prepended to every… 4.4 points. 89 references, cited by 48 papers here."
authors: ["Xin Dong", "Yonggan Fu", "Shizhe Diao", "Wonmin Byeon", "Zijia Chen", "Ameya Sunil Mahabaleshwarkar", "Shih-Yang Liu", "Matthijs Van Keirsbilck", "Min-Hung Chen", "Yoshi Suhara", "Yingyan Lin", "Jan Kautz", "Pavlo Molchanov"]
year: 2024
venue: "arXiv:2411.13676"
source: "https://arxiv.org/abs/2411.13676"
doi: "10.48550/arxiv.2411.13676"
arxiv: "2411.13676"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/dong-2024-hymba-hybrid-head-architecture-small/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/dong-2024-hymba-hybrid-head-architecture-small/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) / Hymba: A Hybrid-head Architecture for Small Language Models

# Hymba: A Hybrid-head Architecture for Small Language Models

**Authors:** [Xin Dong](https://md.research.elamur.ai/free/en/authors/xin-dong/), [Yonggan Fu](https://md.research.elamur.ai/free/en/authors/yonggan-fu/), [Shizhe Diao](https://md.research.elamur.ai/free/en/authors/shizhe-diao/), Wonmin Byeon, [Zijia Chen](https://md.research.elamur.ai/free/en/authors/zijia-chen/), [Ameya Sunil Mahabaleshwarkar](https://md.research.elamur.ai/free/en/authors/ameya-sunil-mahabaleshwarkar/), Shih-Yang Liu, Matthijs Van Keirsbilck, Min-Hung Chen, [Yoshi Suhara](https://md.research.elamur.ai/free/en/authors/yoshi-suhara/), [Yingyan Lin](https://md.research.elamur.ai/free/en/authors/yingyan-celine-lin/), [Jan Kautz](https://md.research.elamur.ai/free/en/authors/jan-kautz/) and Pavlo Molchanov · **Year:** 2024 · **Venue:** arXiv:2411.13676 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2411.13676) · [PDF](https://arxiv.org/pdf/2411.13676) · [DOI 10.48550/arxiv.2411.13676](https://doi.org/10.48550/arxiv.2411.13676) · [arXiv 2411.13676](https://arxiv.org/abs/2411.13676) · [OpenAlex](https://openalex.org/W4404649984)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [48](https://md.research.elamur.ai/free/en/papers/dong-2024-hymba-hybrid-head-architecture-small/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Small language models must balance accuracy against memory and speed. Transformers recall context well, but their compute cost and KV cache grow quickly with sequence length. State space models such as Mamba run cheaply but recall poorly. Earlier hybrids stacked the two layer types one after another, so a weak layer type could become a bottleneck. Hymba, from NVIDIA, instead places attention heads and Mamba-style SSM heads side by side in each layer. Both read the same input, and their outputs are normalized, rescaled and averaged.

The design adds three further pieces:
- 128 learnable meta tokens are prepended to every prompt. They absorb surplus attention and appear to store compressed general knowledge.
- Neighbouring layers share their KV cache.
- Most layers use sliding-window attention, with only a few kept global.

The authors developed the architecture through controlled ablations at 300M parameters and then trained 350M and 1.5B models on 1.5T tokens. Hymba-1.5B-Base outperforms every public model under 2B parameters and even Llama-3.2-3B, while using far less cache and running much faster. This matters for on-device deployment, where memory is the main constraint.

## Key points
- Against Llama-3.2-3B, Hymba-1.5B-Base scores 1.32% higher in average commonsense-reasoning accuracy, needs an 11.67× smaller cache and achieves 3.49× higher throughput (A100, 8k sequence length).
- In the 300M ablations, combining local/global attention with cross-layer KV sharing roughly tripled throughput and cut cache size by almost 4×. Parallel head fusion beat sequential stacking on both reasoning and recall.
- Hymba-1.5B-Instruct, trained with SFT and DPO, reaches the best average (49.22) among lightweight instruct models. It leads on GSM8K (58.76), GPQA (31.03) and BFCLv2 (46.40), ahead of Llama-3.2-1B, Qwen2.5-1.5B and SmolLM2-1.7B.
- A DoRA-finetuned Hymba-1.5B scores 40.0 / 37.9 on RoleBench (instruction / role generalization), beating RoleLlama-7B by 4.5 and 4.4 points.
- In the head-importance analysis, removing the first-layer SSM head drops accuracy to chance level. The Hymba-1.5B-Base and Hymba-1.5B-Instruct models are released on Hugging Face.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Georgia Institute of Technology](https://md.research.elamur.ai/free/en/universities/georgia-institute-of-technology/) — University, Atlanta, United States · 133 works · 24,497 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Xin Dong, Yonggan Fu, Shizhe Diao, Wonmin Byeon, Zijia Chen, Ameya Sunil Mahabaleshwarkar, Shih-Yang Liu, Matthijs Van Keirsbilck, Min-Hung Chen, Yoshi Suhara, Yingyan Lin, Jan Kautz and Pavlo Molchanov. 2024. Hymba: A Hybrid-head Architecture for Small Language Models. arXiv:2411.13676.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{dong2024hymba,
  title = {{Hymba: A Hybrid-head Architecture for Small Language Models}},
  author = {Xin Dong, Yonggan Fu, Shizhe Diao, Wonmin Byeon, Zijia Chen, Ameya Sunil Mahabaleshwarkar, Shih-Yang Liu, Matthijs {Van Keirsbilck}, Min-Hung Chen, Yoshi Suhara, Yingyan Lin, Jan Kautz and Pavlo Molchanov},
  year = {2024},
  doi = {10.48550/arxiv.2411.13676},
  eprint = {2411.13676},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2411.13676}
}
```

- [← previous Cultural Adaptation of Recipes](https://md.research.elamur.ai/free/en/papers/cao-2024-cultural-adaptation-recipes/)
- [next → mDPO: Conditional Preference Optimization for Multimodal Large Language Models](https://md.research.elamur.ai/free/en/papers/wang-2024-mdpo-conditional-preference-optimization-multimodal/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/)

## References

89 references, 36 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. [1] A Vaswani. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in Neural Information Processing Systems, 2017.
2. [2] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. [find](https://scholar.google.com/scholar?q=%5B2%5D+Albert+Gu+and+Tri+Dao%2E+Mamba%3A+Linear%2Dtime+sequence+modeling+with+selective+state+spaces%2E+arXiv+preprint+arXiv%3A2312%2E00752%2C+2023%2E)
3. [3] Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060, 2024. [find](https://scholar.google.com/scholar?q=%5B3%5D+Tri+Dao+and+Albert+Gu%2E+Transformers+are+ssms%3A+Generalized+models+and+efficient+algorithms+through+structured+state+space+duality%2E+arXiv+preprint+arXiv%3A2405%2E21060%2C+2024%2E)
4. [4] Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, et al. An empirical study of mamba-based language models. arXiv preprint arXiv:2406.07887, 2024. [find](https://scholar.google.com/scholar?q=%5B4%5D+Roger+Waleffe%2C+Wonmin+Byeon%2C+Duncan+Riach%2C+Brandon+Norick%2C+Vijay+Korthikanti%2C+Tri+Dao%2C+Albert+Gu%2C+Ali+Hatamizadeh%2C+Sudhakar+Singh%2C+Deepak+Narayanan%2C+et+al%2E+An+empirical+study+of+mamba%2Dbased+langua)
5. [5] Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher Ré. [Simple linear attention language models balance the recall-throughput tradeoff](https://md.research.elamur.ai/free/en/papers/arora-2024-simple-linear-attention-language-models/). arXiv preprint arXiv:2402.18668, 2024.
6. [6] Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. [Jamba: A hybrid transformer-mamba language model](https://md.research.elamur.ai/free/en/papers/lieber-2024-jamba-hybrid-transformer-mamba-language/). arXiv preprint arXiv:2403.19887, 2024.
7. [7] Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, and Weizhu Chen. [Samba: Simple hybrid state space models for efficient unlimited context language modeling](https://md.research.elamur.ai/free/en/papers/ren-2024-samba-simple-hybrid-state-space/). arXiv preprint arXiv:2406.07522, 2024.
8. [8] Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Quantizable transformers: Removing outliers by helping attention heads do nothing. Advances in Neural Information Processing Systems, 36:75067–75096, 2023. [find](https://scholar.google.com/scholar?q=%5B8%5D+Yelysei+Bondarenko%2C+Markus+Nagel%2C+and+Tijmen+Blankevoort%2E+Quantizable+transformers%3A+Removing+outliers+by+helping+attention+heads+do+nothing%2E+Advances+in+Neural+Information+Processing+Systems%2C+36%3A7)
9. [9] Evan Miller. Attention is off by one. [10] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023. [find](https://scholar.google.com/scholar?q=%5B9%5D+Evan+Miller%2E+Attention+is+off+by+one%2E+%5B10%5D+Guangxuan+Xiao%2C+Yuandong+Tian%2C+Beidi+Chen%2C+Song+Han%2C+and+Mike+Lewis%2E+Efficient+streaming+language+models+with+attention+sinks%2E+arXiv+preprint+arXiv%3A2309%2E)
10. [11] William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, and Jonathan Ragan Kelly. Reducing transformer key-value cache size with cross-layer attention. arXiv preprint arXiv:2405.12981, 2024. [find](https://scholar.google.com/scholar?q=%5B11%5D+William+Brandon%2C+Mayank+Mishra%2C+Aniruddha+Nrusimha%2C+Rameswar+Panda%2C+and+Jonathan+Ragan+Kelly%2E+Reducing+transformer+key%2Dvalue+cache+size+with+cross%2Dlayer+attention%2E+arXiv+preprint+arXiv%3A2405%2E12981)
11. [12] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). Advances in Neural Information Processing Systems, 36, 2024.
12. [13] Shih-yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024. [find](https://scholar.google.com/scholar?q=%5B13%5D+Shih%2Dyang+Liu%2C+Chien%2DYi+Wang%2C+Hongxu+Yin%2C+Pavlo+Molchanov%2C+Yu%2DChiang+Frank+Wang%2C+Kwang%2DTing+Cheng%2C+and+Min%2DHung+Chen%2E+Dora%3A+Weight%2Ddecomposed+low%2Drank+adaptation%2E+In+Forty%2Dfirst+International+Con)
13. [14] Zekun Moore Wang, Zhongyuan Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Man Zhang, et al. [Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models](https://md.research.elamur.ai/free/en/papers/wang-2023-rolellm-benchmarking-eliciting-enhancing-role/). arXiv preprint arXiv:2310.00746, 2023.
14. [15] Samy Jelassi, David Brandfonbrener, Sham M Kakade, and Eran Malach. Repeat after me: Transformers are better than state space models at copying. arXiv preprint arXiv:2402.01032, 2024. [find](https://scholar.google.com/scholar?q=%5B15%5D+Samy+Jelassi%2C+David+Brandfonbrener%2C+Sham+M+Kakade%2C+and+Eran+Malach%2E+Repeat+after+me%3A+Transformers+are+better+than+state+space+models+at+copying%2E+arXiv+preprint+arXiv%3A2402%2E01032%2C+2024%2E)
15. [16] Assaf Ben-Kish, Itamar Zimerman, Shady Abu-Hussein, Nadav Cohen, Amir Globerson, Lior Wolf, and Raja Giryes. Decimamba: Exploring the length extrapolation potential of mamba, 2024. [find](https://scholar.google.com/scholar?q=%5B16%5D+Assaf+Ben%2DKish%2C+Itamar+Zimerman%2C+Shady+Abu%2DHussein%2C+Nadav+Cohen%2C+Amir+Globerson%2C+Lior+Wolf%2C+and+Raja+Giryes%2E+Decimamba%3A+Exploring+the+length+extrapolation+potential+of+mamba%2C+2024%2E)
16. [17] Paolo Glorioso, Quentin Anthony, Yury Tokpanov, James Whittington, Jonathan Pilault, Adam Ibrahim, and Beren Millidge. Zamba: A compact 7b ssm hybrid model. arXiv preprint arXiv:2405.16712, 2024. [find](https://scholar.google.com/scholar?q=%5B17%5D+Paolo+Glorioso%2C+Quentin+Anthony%2C+Yury+Tokpanov%2C+James+Whittington%2C+Jonathan+Pilault%2C+Adam+Ibrahim%2C+and+Beren+Millidge%2E+Zamba%3A+A+compact+7b+ssm+hybrid+model%2E+arXiv+preprint+arXiv%3A2405%2E16712%2C+2024%2E)
17. [18] Ang Lv, Kaiyi Zhang, Yuhan Chen, Yulong Wang, Lifeng Liu, Ji-Rong Wen, Jian Xie, and Rui Yan. Interpreting key mechanisms of factual recall in transformer-based language models. arXiv preprint arXiv:2403.19521, 2024. — [arXiv:2403.19521](https://arxiv.org/abs/2403.19521) [find](https://scholar.google.com/scholar?q=%5B18%5D+Ang+Lv%2C+Kaiyi+Zhang%2C+Yuhan+Chen%2C+Yulong+Wang%2C+Lifeng+Liu%2C+Ji%2DRong+Wen%2C+Jian+Xie%2C+and+Rui+Yan%2E+Interpreting+key+mechanisms+of+factual+recall+in+transformer%2Dbased+language+models%2E+arXiv+preprint+ar)
18. Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. [find](https://scholar.google.com/scholar?q=Aviya+Skowron%2C+Lintang+Sutawika%2C+Eric+Tang%2C+Anish+Thite%2C+Ben+Wang%2C+Kevin+Wang%2C+and+Andy+Zou%2E+A+framework+for+few%2Dshot+language+model+evaluation%2C+12+2023%2E)
19. [19] Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Talking heads: Understanding inter-layer communication in transformer language models. arXiv preprint arXiv:2406.09519, 2024. [find](https://scholar.google.com/scholar?q=%5B19%5D+Jack+Merullo%2C+Carsten+Eickhoff%2C+and+Ellie+Pavlick%2E+Talking+heads%3A+Understanding+inter%2Dlayer+communication+in+transformer+language+models%2E+arXiv+preprint+arXiv%3A2406%2E09519%2C+2024%2E)
20. [29] Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher Ré. [Simple linear attention language models balance the recall-throughput tradeoff](https://md.research.elamur.ai/free/en/papers/arora-2024-simple-linear-attention-language-models/), 2024.
21. [20] Ameen Ali, Itamar Zimerman, and Lior Wolf. [The hidden attention of mamba models](https://md.research.elamur.ai/free/en/papers/ali-2025-hidden-attention-mamba-models/). arXiv preprint arXiv:2403.01590, 2024.
22. [30] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309.16588, 2023. — [arXiv:2309.16588](https://arxiv.org/abs/2309.16588) [find](https://scholar.google.com/scholar?q=%5B30%5D+Timoth%C3%A9e+Darcet%2C+Maxime+Oquab%2C+Julien+Mairal%2C+and+Piotr+Bojanowski%2E+Vision+transformers+need+registers%2E+arXiv+preprint+arXiv%3A2309%2E16588%2C+2023%2E+%E2%80%94+%5BarXiv%3A2309%2E16588%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2309%2E1658)
23. [21] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. [find](https://scholar.google.com/scholar?q=%5B21%5D+Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Comp)
24. [31] Evan Miller. Attention if off by one, 2023. [find](https://scholar.google.com/scholar?q=%5B31%5D+Evan+Miller%2E+Attention+if+off+by+one%2C+2023%2E)
25. [22] Iz Beltagy, Matthew E Peters, and Arman Cohan. [Longformer: The long-document transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/). arXiv preprint arXiv:2004.05150, 2020.
26. [23] Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, and Bohan Zhuang. Minicache: Kv cache compression in depth dimension for large language models. arXiv preprint arXiv:2405.14366, 2024. — [arXiv:2405.14366](https://arxiv.org/abs/2405.14366) [find](https://scholar.google.com/scholar?q=%5B23%5D+Akide+Liu%2C+Jing+Liu%2C+Zizheng+Pan%2C+Yefei+He%2C+Gholamreza+Haffari%2C+and+Bohan+Zhuang%2E+Minicache%3A+Kv+cache+compression+in+depth+dimension+for+large+language+models%2E+arXiv+preprint+arXiv%3A2405%2E14366%2C+20)
27. [24] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. [SQuAD: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In Jian Su, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, T…
28. [25] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021.
29. [26] CodeParrot. Codeparrot/github-code · datasets at hugging face. [27] Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. [Lm-infinite: Zero-shot extreme length generalization for large language models](https://md.research.elamur.ai/free/en/papers/han-2024-lm-infinite-zero-shot-extreme/). In Proceedings of the 2024 Conference of the North American Chapter…
30. [28] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, [32] Brian Lester, Rami Al-Rfou, and Noah… — [The Power of Scale for Parameter-Efficient Prompt Tuning](https://md.research.elamur.ai/free/en/papers/lester-2021-power-scale-parameter-efficient-prompt/)
31. [33] Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. [Ppt: Pre-trained prompt tuning for few-shot learning](https://md.research.elamur.ai/free/en/papers/gu-2022-ppt-pre-trained-prompt-tuning/). arXiv preprint arXiv:2109.04332, 2021.
32. [34] Charles Dickens. The Adventures of Oliver Twist. Ticknor and Fields, 1868. [find](https://scholar.google.com/scholar?q=%5B34%5D+Charles+Dickens%2E+The+Adventures+of+Oliver+Twist%2E+Ticknor+and+Fields%2C+1868%2E)
33. [35] Itamar Zimerman, Ameen Ali, and Lior Wolf. A unified implicit attention formulation for gated-linear recurrent sequence models. arXiv preprint arXiv:2405.16504, 2024. — [arXiv:2405.16504](https://arxiv.org/abs/2405.16504) [find](https://scholar.google.com/scholar?q=%5B35%5D+Itamar+Zimerman%2C+Ameen+Ali%2C+and+Lior+Wolf%2E+A+unified+implicit+attention+formulation+for+gated%2Dlinear+recurrent+sequence+models%2E+arXiv+preprint+arXiv%3A2405%2E16504%2C+2024%2E+%E2%80%94+%5BarXiv%3A2405%2E16504%5D%28https%3A%2F)
34. [36] Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna N… — [DataComp-LM: In search of the next generation of training sets for language models](https://md.research.elamur.ai/free/en/papers/li-2024-datacomp-lm-search-next-generation/)
35. [37] Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. Smollm-corpus, 2024. [find](https://scholar.google.com/scholar?q=%5B37%5D+Loubna+Ben+Allal%2C+Anton+Lozhkov%2C+Guilherme+Penedo%2C+Thomas+Wolf%2C+and+Leandro+von+Werra%2E+Smollm%2Dcorpus%2C+2024%2E)
36. [38] Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. [Minicpm: Unveiling the potential of small language models with scalable training strategies](https://md.research.elamur.ai/free/en/papers/hu-2024-minicpm-unveiling-potential-small-language/). arXiv preprint arXiv:2404.06395, 2024.
37. [39] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783, 2024.
38. [40] Yikang Shen, Zhen Guo, Tianle Cai, and Zengyi Qin. Jetmoe: Reaching llama2 performance with 0.1 m dollars. arXiv preprint arXiv:2404.[07413, 2024](https://md.research.elamur.ai/free/en/papers/shen-2024-jetmoe-reaching-llama2-performance-0/).
39. [41] bloc97. Dynamically scaled rope further increases performance of long context llama with zero finetuning, July 2023. [find](https://scholar.google.com/scholar?q=%5B41%5D+bloc97%2E+Dynamically+scaled+rope+further+increases+performance+of+long+context+llama+with+zero+finetuning%2C+July+2023%2E)
40. [42] Meta AI. Llama 3.2: Revolutionizing edge AI and vision with open, customizable models. 2024. [find](https://scholar.google.com/scholar?q=%5B42%5D+Meta+AI%2E+Llama+3%2E2%3A+Revolutionizing+edge+AI+and+vision+with+open%2C+customizable+models%2E+2024%2E)
41. [43] Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Leandro von Werra, and Thomas Wolf. Smollm -blazingly fast and remarkably powerful, 2024. [find](https://scholar.google.com/scholar?q=%5B43%5D+Loubna+Ben+Allal%2C+Anton+Lozhkov%2C+Elie+Bakouch%2C+Leandro+von+Werra%2C+and+Thomas+Wolf%2E+Smollm+%2Dblazingly+fast+and+remarkably+powerful%2C+2024%2E)
42. [44] Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Lewis Tunstall, Agustín Piqueres, Andres Marafioti, Cyril Zakka, Leandro von Werra, and Thomas Wolf. Smollm2 - with great data, comes great performance, 2024. [find](https://scholar.google.com/scholar?q=%5B44%5D+Loubna+Ben+Allal%2C+Anton+Lozhkov%2C+Elie+Bakouch%2C+Gabriel+Mart%C3%ADn+Bl%C3%A1zquez%2C+Lewis+Tunstall%2C+Agust%C3%ADn+Piqueres%2C+Andres+Marafioti%2C+Cyril+Zakka%2C+Leandro+von+Werra%2C+and+Thomas+Wolf%2E+Smollm2+%2D+with+great+d)
43. [45] Jiang Liu, Jialian Wu, Prakamya Mishra, Zicheng Liu, Sudhanshu Ranjan, Pratik Prabhanjan Brahma, Yusheng Su, Gowtham Ramesh, Peng Sun, Zhe Li, Dong Li, Lu Tian, and Emad Barsoum. Amd-olmo: A series of 1b language models trained from scratch by amd on amd instinct™ mi250 gpus., October 2024. [find](https://scholar.google.com/scholar?q=%5B45%5D+Jiang+Liu%2C+Jialian+Wu%2C+Prakamya+Mishra%2C+Zicheng+Liu%2C+Sudhanshu+Ranjan%2C+Pratik+Prabhanjan+Brahma%2C+Yusheng+Su%2C+Gowtham+Ramesh%2C+Peng+Sun%2C+Zhe+Li%2C+Dong+Li%2C+Lu+Tian%2C+and+Emad+Barsoum%2E+Amd%2Dolmo%3A+A+seri)
44. [46] Marco Bellagente, Jonathan Tow, Dakota Mahan, Duy Phung, Maksym Zhuravinskyi, Reshinth Adithyan, James Baicoianu, Ben Brooks, Nathan Cooper, Ashish Datta, et al. Stable lm 2 1.6 b technical report. arXiv preprint arXiv:2402.[17834, 2024](https://md.research.elamur.ai/free/en/papers/bellagente-2024-stable-lm-2-1-6b/).
45. [47] Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, Will… [find](https://scholar.google.com/scholar?q=%5B47%5D+Dirk+Groeneveld%2C+Iz+Beltagy%2C+Pete+Walsh%2C+Akshita+Bhagia%2C+Rodney+Kinney%2C+Oyvind+Tafjord%2C+Ananya+Harsh+Jha%2C+Hamish+Ivison%2C+Ian+Magnusson%2C+Yizhong+Wang%2C+Shane+Arora%2C+David+Atkinson%2C+Russell+Authur%2C+)
46. [48] Huggingface. HuggingFaceTB/cosmo-1b. 2024. [find](https://scholar.google.com/scholar?q=%5B48%5D+Huggingface%2E+HuggingFaceTB%2Fcosmo%2D1b%2E+2024%2E)
47. [49] Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. [Textbooks are all you need ii: phi-1.5 technical report](https://md.research.elamur.ai/free/en/papers/li-2023-textbooks-all-need-ii-phi/). arXiv preprint arXiv:2309.05463, 2023.
48. [50] Philipp Singer, Pascal Pfeiffer, Yauhen Babakhin, Maximilian Jeblick, Nischay Dhankhar, Gabor Fodor, and Sri Satish Ambati. H2o-danube-1.8 b technical report. arXiv preprint arXiv:2401.16818, 2024. — [arXiv:2401.16818](https://arxiv.org/abs/2401.16818) [find](https://scholar.google.com/scholar?q=%5B50%5D+Philipp+Singer%2C+Pascal+Pfeiffer%2C+Yauhen+Babakhin%2C+Maximilian+Jeblick%2C+Nischay+Dhankhar%2C+Gabor+Fodor%2C+and+Sri+Satish+Ambati%2E+H2o%2Ddanube%2D1%2E8+b+technical+report%2E+arXiv+preprint+arXiv%3A2401%2E16818%2C+202)
49. [51] Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Seyed Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, et al. Openelm: An efficient language model family with open training and inference framework. In Workshop on Efficient Systems for… [find](https://scholar.google.com/scholar?q=%5B51%5D+Sachin+Mehta%2C+Mohammad+Hossein+Sekhavat%2C+Qingqing+Cao%2C+Maxwell+Horton%2C+Yanzi+Jin%2C+Chenfan+Sun%2C+Seyed+Iman+Mirzadeh%2C+Mahyar+Najibi%2C+Dmitry+Belenko%2C+Peter+Zatloukal%2C+et+al%2E+Openelm%3A+An+efficient+la)
50. [52] Cartesia AI. The On-Device Intelligence Update. 2024. [find](https://scholar.google.com/scholar?q=%5B52%5D+Cartesia+AI%2E+The+On%2DDevice+Intelligence+Update%2E+2024%2E)
51. [53] Meta AI. Introducing Meta Llama 3: The most capable openly available LLM to date. 2024. [find](https://scholar.google.com/scholar?q=%5B53%5D+Meta+AI%2E+Introducing+Meta+Llama+3%3A+The+most+capable+openly+available+LLM+to+date%2E+2024%2E)
52. [54] Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. [The fineweb datasets: Decanting the web for the finest text data at scale](https://md.research.elamur.ai/free/en/papers/penedo-2024-fineweb-datasets-decanting-web-finest/), 2024.
53. [55] Colin Lockard, Prashant Shiralkar, and Xin Luna Dong. [OpenCeres: When open information extraction meets the semi-structured web](https://md.research.elamur.ai/free/en/papers/lockard-2019-openceres-when-open-information-extraction/). In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Li…
54. [56] Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for squad, 2018. — [Know What You Don't Know: Unanswerable Questions for SQuAD](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/)
55. [57] Xiaoran Liu, Hang Yan, Shuo Zhang, Chenxin An, Xipeng Qiu, and Dahua Lin. [Scaling laws of rope-based extrapolation](https://md.research.elamur.ai/free/en/papers/liu-2023-scaling-laws-rope-based/). arXiv preprint arXiv:2310.05209, 2023.
56. [58] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071, 2023. [find](https://scholar.google.com/scholar?q=%5B58%5D+Bowen+Peng%2C+Jeffrey+Quesnelle%2C+Honglu+Fan%2C+and+Enrico+Shippole%2E+Yarn%3A+Efficient+context+window+extension+of+large+language+models%2E+arXiv+preprint+arXiv%3A2309%2E00071%2C+2023%2E)
57. [59] Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, et al. Inf bench: Extending long context evaluation beyond 100k tokens. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon… [find](https://scholar.google.com/scholar?q=%5B59%5D+Xinrong+Zhang%2C+Yingfa+Chen%2C+Shengding+Hu%2C+Zihang+Xu%2C+Junhao+Chen%2C+Moo+Hao%2C+Xu+Han%2C+Zhen+Thai%2C+Shuo+Wang%2C+Zhiyuan+Liu%2C+et+al%2E+Inf+bench%3A+Extending+long+context+evaluation+beyond+100k+tokens%2E+In+Pr)
58. [60] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. [Lost in the middle: How language models use long contexts](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/). Transactions of the Association for Computational Linguistics, 12:157–173, 2024.
59. [61] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944, 2023. [find](https://scholar.google.com/scholar?q=%5B61%5D+Lewis+Tunstall%2C+Edward+Beeching%2C+Nathan+Lambert%2C+Nazneen+Rajani%2C+Kashif+Rasul%2C+Younes+Belkada%2C+Shengyi+Huang%2C+Leandro+von+Werra%2C+Cl%C3%A9mentine+Fourrier%2C+Nathan+Habib%2C+et+al%2E+Zephyr%3A+Direct+distillat)
60. [62] Shizhe Diao, Rui Pan, Hanze Dong, Kashun Shum, Jipeng Zhang, Wei Xiong, and Tong Zhang. Lmflow: An extensible toolkit for finetuning and inference of large foundation models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Hu… [find](https://scholar.google.com/scholar?q=%5B62%5D+Shizhe+Diao%2C+Rui+Pan%2C+Hanze+Dong%2C+Kashun+Shum%2C+Jipeng+Zhang%2C+Wei+Xiong%2C+and+Tong+Zhang%2E+Lmflow%3A+An+extensible+toolkit+for+finetuning+and+inference+of+large+foundation+models%2E+In+Proceedings+of+th)
61. [63] Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.07863, 2024. [find](https://scholar.google.com/scholar?q=%5B63%5D+Hanze+Dong%2C+Wei+Xiong%2C+Bo+Pang%2C+Haoxiang+Wang%2C+Han+Zhao%2C+Yingbo+Zhou%2C+Nan+Jiang%2C+Doyen+Sahoo%2C+Caiming+Xiong%2C+and+Tong+Zhang%2E+Rlhf+workflow%3A+From+reward+modeling+to+online+rlhf%2E+arXiv+preprint+arX)
62. [64] Qwen Team. Qwen2.5: A party of foundation models, September 2024. [find](https://scholar.google.com/scholar?q=%5B64%5D+Qwen+Team%2E+Qwen2%2E5%3A+A+party+of+foundation+models%2C+September+2024%2E)
63. [65] Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. Berkeley function calling leaderboard. [https://gorilla.cs.berkeley.edu/blogs/](https://gorilla.cs.berkeley.edu/blogs/) 8_berkeley_function_calling_leaderboard.html, 2024. [↗](https://gorilla.cs.berkeley.edu/blogs/)
64. [69] Zhen Qin, Dong Li, Weigao Sun, Weixuan Sun, Xuyang Shen, Xiaodong Han, Yunshen Wei, Baohong Lv, Fei Yuan, Xiao Luo, et al. Scaling transnormer to 175 billion parameters. arXiv preprint arXiv:2307.[14995, 2023](https://md.research.elamur.ai/free/en/papers/qin-2023-transnormerllm-faster-better-large/).
65. [70] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). arXiv preprint arXiv:2001.08361, 2020.
66. [71] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference… [find](https://scholar.google.com/scholar?q=%5B71%5D+Stella+Biderman%2C+Hailey+Schoelkopf%2C+Quentin+Gregory+Anthony%2C+Herbie+Bradley%2C+Kyle+O%E2%80%99Brien%2C+Eric+Hallahan%2C+Mohammad+Aflah+Khan%2C+Shivanshu+Purohit%2C+USVSN+Sai+Prashanth%2C+Edward+Raff%2C+et+al%2E+Pythia%3A+)
67. [72] Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. [Glm: General language model pretraining with autoregressive blank infilling](https://md.research.elamur.ai/free/en/papers/du-2022-glm-general-language-model-pretraining/). arXiv preprint arXiv:2103.10360, 2021.
68. [73] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. [Opt: Open pre-trained transformer language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/). arXiv preprint arXiv:2205.01068, 2022.
69. [66] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805, 2018.
70. [74] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. [find](https://scholar.google.com/scholar?q=%5B74%5D+Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+Mistr)
71. [67] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research, 21(140):1–67, 2020.
72. [75] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023.
73. [68] Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowycz, Alex Salcianu, Marc van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvy… [find](https://scholar.google.com/scholar?q=%5B68%5D+Adam+Roberts%2C+Hyung+Won+Chung%2C+Anselm+Levskaya%2C+Gaurav+Mishra%2C+James+Bradbury%2C+Daniel+Andor%2C+Sharan+Narang%2C+Brian+Lester%2C+Colin+Gaffney%2C+Afroz+Mohiuddin%2C+Curtis+Hawthorne%2C+Aitor+Lewkowycz%2C+Alex+S)
74. [76] Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024. [find](https://scholar.google.com/scholar?q=%5B76%5D+Gemma+Team%2C+Morgane+Riviere%2C+Shreya+Pathak%2C+Pier+Giuseppe+Sessa%2C+Cassidy+Hardin%2C+Surya+Bhupatiraju%2C+L%C3%A9onard+Hussenot%2C+Thomas+Mesnard%2C+Bobak+Shahriari%2C+Alexandre+Ram%C3%A9%2C+et+al%2E+Gemma+2%3A+Improving+op)
75. [77] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774, 2023.
76. [78] Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048, 2023. [find](https://scholar.google.com/scholar?q=%5B78%5D+Bo+Peng%2C+Eric+Alcaide%2C+Quentin+Anthony%2C+Alon+Albalak%2C+Samuel+Arcadinho%2C+Stella+Biderman%2C+Huanqi+Cao%2C+Xin+Cheng%2C+Michael+Chung%2C+Matteo+Grella%2C+et+al%2E+Rwkv%3A+Reinventing+rnns+for+the+transformer+era)
77. [79] Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. [Retentive network: A successor to transformer for large language models](https://md.research.elamur.ai/free/en/papers/sun-2023-retentive-network-successor-transformer-large/). arXiv preprint arXiv:2307.08621, 2023.
78. [80] Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. arXiv preprint arXiv:2312.06635, 2023. [find](https://scholar.google.com/scholar?q=%5B80%5D+Songlin+Yang%2C+Bailin+Wang%2C+Yikang+Shen%2C+Rameswar+Panda%2C+and+Yoon+Kim%2E+Gated+linear+attention+transformers+with+hardware%2Defficient+training%2E+arXiv+preprint+arXiv%3A2312%2E06635%2C+2023%2E)
79. [91] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang… [find](https://scholar.google.com/scholar?q=%5B91%5D+Leo+Gao%2C+Jonathan+Tow%2C+Baber+Abbasi%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Alain+Le+Noac%E2%80%99h%2C+Haonan+Li%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Chri)
80. [81] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pages 5156–5165. PMLR, 2020. [find](https://scholar.google.com/scholar?q=%5B81%5D+Angelos+Katharopoulos%2C+Apoorv+Vyas%2C+Nikolaos+Pappas%2C+and+Fran%C3%A7ois+Fleuret%2E+Transformers+are+rnns%3A+Fast+autoregressive+transformers+with+linear+attention%2E+In+International+conference+on+machine+le)
81. [82] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021. [find](https://scholar.google.com/scholar?q=%5B82%5D+Albert+Gu%2C+Karan+Goel%2C+and+Christopher+R%C3%A9%2E+Efficiently+modeling+long+sequences+with+structured+state+spaces%2E+arXiv+preprint+arXiv%3A2111%2E00396%2C+2021%2E)
82. [83] Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in neural information processing systems, 34:572–585, 2021. [find](https://scholar.google.com/scholar?q=%5B83%5D+Albert+Gu%2C+Isys+Johnson%2C+Karan+Goel%2C+Khaled+Saab%2C+Tri+Dao%2C+Atri+Rudra%2C+and+Christopher+R%C3%A9%2E+Combining+recurrent%2C+convolutional%2C+and+continuous%2Dtime+models+with+linear+state+space+layers%2E+Advances+)
83. [84] Jongho Park, Jaeseung Park, Zheyang Xiong, Nayoung Lee, Jaewoong Cho, Samet Oymak, Kangwook Lee, and Dimitris Papailiopoulos. Can mamba learn how to learn? a comparative study on in-context learning tasks. arXiv preprint arXiv:2402.04248, 2024. [find](https://scholar.google.com/scholar?q=%5B84%5D+Jongho+Park%2C+Jaeseung+Park%2C+Zheyang+Xiong%2C+Nayoung+Lee%2C+Jaewoong+Cho%2C+Samet+Oymak%2C+Kangwook+Lee%2C+and+Dimitris+Papailiopoulos%2E+Can+mamba+learn+how+to+learn%3F+a+comparative+study+on+in%2Dcontext+learn)
84. [85] Soham De, Samuel L Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, et al. Griffin: Mixing gated linear recurrences with local attention for efficient language models. arXiv preprint arXiv:2402.19427,… [find](https://scholar.google.com/scholar?q=%5B85%5D+Soham+De%2C+Samuel+L+Smith%2C+Anushan+Fernando%2C+Aleksandar+Botev%2C+George+Cristian%2DMuraru%2C+Albert+Gu%2C+Ruba+Haroun%2C+Leonard+Berrada%2C+Yutian+Chen%2C+Srivatsan+Srinivasan%2C+et+al%2E+Griffin%3A+Mixing+gated+line)
85. [86] Jonathan Pilault, Mahan Fathi, Orhan Firat, Chris Pal, Pierre-Luc Bacon, and Ross Goroshin. [Block-state transformers](https://md.research.elamur.ai/free/en/papers/fathi-2023-block-state-transformers/). Advances in Neural Information Processing Systems, 36, 2024.
86. [87] George Saon, Ankit Gupta, and Xiaodong Cui. Diagonal state space augmented transformers for speech recognition. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. [find](https://scholar.google.com/scholar?q=%5B87%5D+George+Saon%2C+Ankit+Gupta%2C+and+Xiaodong+Cui%2E+Diagonal+state+space+augmented+transformers+for+speech+recognition%2E+In+ICASSP+2023%2D2023+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+P)
87. [88] Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. arXiv preprint arXiv:2406.06484, 2024. [find](https://scholar.google.com/scholar?q=%5B88%5D+Songlin+Yang%2C+Bailin+Wang%2C+Yu+Zhang%2C+Yikang+Shen%2C+and+Yoon+Kim%2E+Parallelizing+linear+transformers+with+the+delta+rule+over+sequence+length%2E+arXiv+preprint+arXiv%3A2406%2E06484%2C+2024%2E)
88. [89] Alexey Dosovitskiy. [An image is worth 16x16 words: Transformers for image recognition at scale](https://md.research.elamur.ai/free/en/papers/dosovitskiy-2020-image-worth-16x16-words-transformers/). arXiv preprint arXiv:2010.11929, 2020.
89. [90] Mikhail S Burtsev, Yuri Kuratov, Anton Peganov, and Grigory V Sapunov. Memory transformer. arXiv preprint arXiv:2006.11527, 2020. — [arXiv:2006.11527](https://arxiv.org/abs/2006.11527) [find](https://scholar.google.com/scholar?q=%5B90%5D+Mikhail+S+Burtsev%2C+Yuri+Kuratov%2C+Anton+Peganov%2C+and+Grigory+V+Sapunov%2E+Memory+transformer%2E+arXiv+preprint+arXiv%3A2006%2E11527%2C+2020%2E+%E2%80%94+%5BarXiv%3A2006%2E11527%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2006%2E11527%29)

*Updated 1 November 2026*