---
title: "Leave No Context Behind — paper by Munkhdalai et al. (2024)"
description: "The authors propose Infini-attention, which changes a single attention layer so it handles both nearby… 64 references, cited by 65 papers here."
authors: ["Tsendsuren Munkhdalai", "Manaal Faruqui", "Siddharth Gopal"]
year: 2024
venue: "arXiv:2404.07143"
source: "https://arxiv.org/abs/2404.07143"
doi: "10.48550/arxiv.2404.07143"
arxiv: "2404.07143"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/munkhdalai-2024-leave-no-context-behind-efficient/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/munkhdalai-2024-leave-no-context-behind-efficient/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) / Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

# Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

**Authors:** [Tsendsuren Munkhdalai](https://md.research.elamur.ai/free/en/authors/tsendsuren-munkhdalai/), [Manaal Faruqui](https://md.research.elamur.ai/free/en/authors/manaal-faruqui/) and Siddharth Gopal · **Year:** 2024 · **Venue:** arXiv:2404.07143 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2404.07143) · [PDF](https://arxiv.org/pdf/2404.07143) · [DOI 10.48550/arxiv.2404.07143](https://doi.org/10.48550/arxiv.2404.07143) · [arXiv 2404.07143](https://arxiv.org/abs/2404.07143) · [OpenAlex](https://openalex.org/W4394737116)

**Citations:** 15 citations (OpenAlex, 2026-10-14) · [65](https://md.research.elamur.ai/free/en/papers/munkhdalai-2024-leave-no-context-behind-efficient/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Standard Transformer attention scales quadratically with input length, and its key-value cache grows with every token. This makes million-token contexts expensive to train and serve. The authors propose Infini-attention, which changes a single attention layer so it handles both nearby and distant context. Text is processed segment by segment. Inside each segment, ordinary causal dot-product attention runs as usual. Instead of discarding past key-value states, the layer folds them into a fixed-size associative memory matrix per head. Later queries read from that memory through linear attention, and a learned gate mixes the retrieved content with the local attention output. Because the memory reuses the existing Q, K and V projections, existing LLMs can adopt it through continual pre-training and fine-tuning.

The resulting Infini-Transformers beat memory-based baselines on long-context language modeling while using a much smaller memory. A 1B model solves passkey retrieval at 1M tokens, and an 8B model reaches state-of-the-art results on BookSum with 500K-token inputs. This points to streaming, constant-memory long-context LLMs as a practical option.

## Key points
- **Memory design:** The memory is an associative matrix updated incrementally, following the linear-attention approach of Katharopoulos et al. (2020), with ELU + 1 activation and a normalization term. Its footprint is d_key × (d_value + 1) × H × l regardless of sequence length.
- **Baselines:** The paper compares against Transformer-XL, Compressive Transformer, Memorizing Transformers, RMT and AutoCompressors. It reports roughly 114x compression in memory size relative to baselines, and perplexity improves further when training with 100K-token sequences.
- **Length generalization:** A 1B model fine-tuned on passkey instances of only up to 5K tokens solved the passkey retrieval task at 1M tokens.
- **BookSum results:** An 8B model, after continual pre-training, was fine-tuned on BookSum with 32K-token inputs and evaluated with 500K-token inputs (temperature 0.5, top-p 0.95, 1024 decoding steps). It set a new state of the art, and its Rouge scores rose as more of each book was given as input.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

## Cite

Cite as: Tsendsuren Munkhdalai, Manaal Faruqui and Siddharth Gopal. 2024. Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention. arXiv:2404.07143.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{munkhdalai2024leave,
  title = {{Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention}},
  author = {Tsendsuren Munkhdalai, Manaal Faruqui and Siddharth Gopal},
  year = {2024},
  doi = {10.48550/arxiv.2404.07143},
  eprint = {2404.07143},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.07143}
}
```

- [← previous Lean Workbook: A large-scale Lean problem set formalized from natural language math problems](https://md.research.elamur.ai/free/en/papers/ying-2024-lean-workbook-large-scale-lean/)
- [next → NanoFlow: Towards Optimal Large Language Model Serving Throughput](https://md.research.elamur.ai/free/en/papers/zhu-2024-nanoflow-optimal-large-language-model/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/)

## References

64 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. [Palm 2 technical report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/). arXiv preprint arXiv:2305.10403, 2023.
2. Jimmy Ba, Geoffrey E Hinton, Volodymyr Mnih, Joel Z Leibo, and Catalin Ionescu. Using fast weights to attend to the recent past. Advances in neural information processing systems, 29, 2016. [find](https://scholar.google.com/scholar?q=Jimmy+Ba%2C+Geoffrey+E+Hinton%2C+Volodymyr+Mnih%2C+Joel+Z+Leibo%2C+and+Catalin+Ionescu%2E+Using+fast+weights+to+attend+to+the+recent+past%2E+Advances+in+neural+information+processing+systems%2C+29%2C+2016%2E)
3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). arXiv preprint arXiv:1409.0473, 2014.
4. Iz Beltagy, Matthew E Peters, and Arman Cohan. [Longformer: The long-document transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/). arXiv preprint arXiv:2004.05150, 2020.
5. Amanda Bertsch, Uri Alon, Graham Neubig, and Matthew Gormley. [Unlimiformer: Long-range transformers with unlimited length input](https://md.research.elamur.ai/free/en/papers/bertsch-2023-unlimiformer-long-range-transformers-unlimited/). Advances in Neural Information Processing Systems, 36, 2024.
6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020.
7. Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. Recurrent memory transformer. Advances in Neural Information Processing Systems, 35:11079–11091, 2022. [find](https://scholar.google.com/scholar?q=Aydar+Bulatov%2C+Yury+Kuratov%2C+and+Mikhail+Burtsev%2E+Recurrent+memory+transformer%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A11079%E2%80%9311091%2C+2022%2E)
8. Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595, 2023a. [find](https://scholar.google.com/scholar?q=Shouyuan+Chen%2C+Sherman+Wong%2C+Liangjian+Chen%2C+and+Yuandong+Tian%2E+Extending+context+window+of+large+language+models+via+positional+interpolation%2E+arXiv+preprint+arXiv%3A2306%2E15595%2C+2023a%2E)
9. Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. [Longlora: Efficient fine-tuning of long-context large language models](https://md.research.elamur.ai/free/en/papers/chen-2023-longlora-efficient-fine-tuning-long/). arXiv preprint arXiv:2309.12307, 2023b.
10. Jianpeng Cheng, Li Dong, and Mirella Lapata. [Long short-term memory-networks for machine reading](https://md.research.elamur.ai/free/en/papers/cheng-2016-long-short-term-memory-networks/). arXiv preprint arXiv:1601.06733, 2016.
11. Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. [Adapting language models to compress contexts](https://md.research.elamur.ai/free/en/papers/chevalier-2023-adapting-language-models-compress-contexts/). arXiv preprint arXiv:2305.14788, 2023.
12. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. [Generating long sequences with sparse transformers](https://md.research.elamur.ai/free/en/papers/child-2019-generating-long-sequences-sparse-transformers/). arXiv preprint arXiv:1904.10509, 2019.
13. Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. [Fast and accurate deep network learning by exponential linear units (elus)](https://md.research.elamur.ai/free/en/papers/clevert-2015-fast-accurate-deep-network-learning/). arXiv preprint arXiv:1511.07289, 2015.
14. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860, 2019. [find](https://scholar.google.com/scholar?q=Zihang+Dai%2C+Zhilin+Yang%2C+Yiming+Yang%2C+Jaime+Carbonell%2C+Quoc+V+Le%2C+and+Ruslan+Salakhutdinov%2E+Transformer%2Dxl%3A+Attentive+language+models+beyond+a+fixed%2Dlength+context%2E+arXiv+preprint+arXiv%3A1901%2E02860%2C+20)
15. Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344–16359, 2022. [find](https://scholar.google.com/scholar?q=Tri+Dao%2C+Dan+Fu%2C+Stefano+Ermon%2C+Atri+Rudra%2C+and+Christopher+Re%CC%81%2E+Flashattention%3A+Fast+and+memory%2Defficient+exact+attention+with+io%2Dawareness%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A1634)
16. Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Nanning Zheng, and Furu Wei. [Longnet: Scaling transformers to 1,000,000,000 tokens](https://md.research.elamur.ai/free/en/papers/ding-2023-longnet-scaling-transformers-1-000/). arXiv preprint arXiv:2307.02486, 2023.
17. Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. [Data engineering for scaling language models to 128k context](https://md.research.elamur.ai/free/en/papers/fu-2024-data-engineering-scaling-language-models/). arXiv preprint arXiv:2402.10171, 2024.
18. Tao Ge, Jing Hu, Xun Wang, Si-Qing Chen, and Furu Wei. [In-context autoencoder for context compression in a large language model](https://md.research.elamur.ai/free/en/papers/ge-2023-context-autoencoder-context-compression-large/). arXiv preprint arXiv:2307.06945, 2023.
19. Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014. [find](https://scholar.google.com/scholar?q=Alex+Graves%2C+Greg+Wayne%2C+and+Ivo+Danihelka%2E+Neural+turing+machines%2E+arXiv+preprint+arXiv%3A1410%2E5401%2C+2014%2E)
20. Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. Olmo: Accelerating the science of language models. arXiv preprint arXiv:2402.00838, 2024. [find](https://scholar.google.com/scholar?q=Dirk+Groeneveld%2C+Iz+Beltagy%2C+Pete+Walsh%2C+Akshita+Bhagia%2C+Rodney+Kinney%2C+Oyvind+Tafjord%2C+Ananya+Harsh+Jha%2C+Hamish+Ivison%2C+Ian+Magnusson%2C+Yizhong+Wang%2C+et+al%2E+Olmo%3A+Accelerating+the+science+of+language+)
21. Donald Olding Hebb. The organization of behavior: A neuropsychological theory. Psychology press, 2005. [find](https://scholar.google.com/scholar?q=Donald+Olding+Hebb%2E+The+organization+of+behavior%3A+A+neuropsychological+theory%2E+Psychology+press%2C+2005%2E)
22. Geoffrey E Hinton and David C Plaut. Using fast weights to deblur old memories. In Proceedings of the ninth annual conference of the Cognitive Science Society, pp. 177–186, 1987. [find](https://scholar.google.com/scholar?q=Geoffrey+E+Hinton+and+David+C+Plaut%2E+Using+fast+weights+to+deblur+old+memories%2E+In+Proceedings+of+the+ninth+annual+conference+of+the+Cognitive+Science+Society%2C+pp%2E+177%E2%80%93186%2C+1987%2E)
23. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9 (8):1735–1780, 1997. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9+%288%29%3A1735%E2%80%931780%2C+1997%2E)
24. John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8):2554–2558, 1982. [find](https://scholar.google.com/scholar?q=John+J+Hopfield%2E+Neural+networks+and+physical+systems+with+emergent+collective+computational+abilities%2E+Proceedings+of+the+national+academy+of+sciences%2C+79%288%29%3A2554%E2%80%932558%2C+1982%2E)
25. Dongseong Hwang, Weiran Wang, Zhuoyuan Huo, Khe Chai Sim, and Pedro Moreno Mengibar. Transformerfam: Feedback attention is working memory. arXiv preprint arXiv:2404.09173, 2024. — [arXiv:2404.09173](https://arxiv.org/abs/2404.09173) [find](https://scholar.google.com/scholar?q=Dongseong+Hwang%2C+Weiran+Wang%2C+Zhuoyuan+Huo%2C+Khe+Chai+Sim%2C+and+Pedro+Moreno+Mengibar%2E+Transformerfam%3A+Feedback+attention+is+working+memory%2E+arXiv+preprint+arXiv%3A2404%2E09173%2C+2024%2E+%E2%80%94+%5BarXiv%3A2404%2E09173%5D%28h)
26. Łukasz Kaiser and Ilya Sutskever. arXiv:1511.08228, 2015. [find](https://scholar.google.com/scholar?q=%C5%81ukasz+Kaiser+and+Ilya+Sutskever%2E+arXiv%3A1511%2E08228%2C+2015%2E)
27. Pentti Kanerva. Sparse distributed memory. MIT press, 1988. [find](https://scholar.google.com/scholar?q=Pentti+Kanerva%2E+Sparse+distributed+memory%2E+MIT+press%2C+1988%2E)
28. Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pp. 5156–5165. PMLR, 2020. [find](https://scholar.google.com/scholar?q=Angelos+Katharopoulos%2C+Apoorv+Vyas%2C+Nikolaos+Pappas%2C+and+Franc%CC%A7ois+Fleuret%2E+Transformers+are+rnns%3A+Fast+autoregressive+transformers+with+linear+attention%2E+In+International+conference+on+machine+learni)
29. Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. [The impact of positional encoding on length generalization in transformers](https://md.research.elamur.ai/free/en/papers/kazemnejad-2023-impact-positional-encoding-length-generalization/). Advances in Neural Information Processing Systems, 36, 2024.
30. Wojciech Kryściński, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. Booksum: A collection of datasets for long-form narrative summarization. arXiv preprint arXiv:2105.08209, 2021. arXiv preprint Neural gpus learn algorithms. [find](https://scholar.google.com/scholar?q=Wojciech+Krys%CC%81cin%CC%81ski%2C+Nazneen+Rajani%2C+Divyansh+Agarwal%2C+Caiming+Xiong%2C+and+Dragomir+Radev%2E+Booksum%3A+A+collection+of+datasets+for+long%2Dform+narrative+summarization%2E+arXiv+preprint+arXiv%3A2105%2E08209%2C+20)
31. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. [Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). arXiv preprint arXiv:1910.13461, 2019.
32. Hao Liu, Matei Zaharia, and Pieter Abbeel. [Ring attention with blockwise transformers for near-infinite context](https://md.research.elamur.ai/free/en/papers/liu-2023-ring-attention-blockwise-transformers-near/). arXiv preprint arXiv:2310.01889, 2023.
33. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. [Lost in the middle: How language models use long contexts](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/). Transactions of the Association for Computational Linguistics, 12:157–173, 2024.
34. Wolfgang Maass, Thomas Natschläger, and Henry Markram. Real-time computing without stable states: A new framework for neural computation based on perturbations. Neural computation, 14(11):2531–2560, 2002. [find](https://scholar.google.com/scholar?q=Wolfgang+Maass%2C+Thomas+Natschla%CC%88ger%2C+and+Henry+Markram%2E+Real%2Dtime+computing+without+stable+states%3A+A+new+framework+for+neural+computation+based+on+perturbations%2E+Neural+computation%2C+14%2811%29%3A2531%E2%80%932560%2C+)
35. Thomas Miconi, Kenneth Stanley, and Jeff Clune. Differentiable plasticity: training plastic neural networks with backpropagation. In International Conference on Machine Learning, pp. 3559–3568. PMLR, 2018. [find](https://scholar.google.com/scholar?q=Thomas+Miconi%2C+Kenneth+Stanley%2C+and+Jeff+Clune%2E+Differentiable+plasticity%3A+training+plastic+neural+networks+with+backpropagation%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+3559%E2%80%933568%2E+PMLR%2C+)
36. Amirkeivan Mohtashami and Martin Jaggi. Random-access infinite context length for transformers. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Amirkeivan+Mohtashami+and+Martin+Jaggi%2E+Random%2Daccess+infinite+context+length+for+transformers%2E+Advances+in+Neural+Information+Processing+Systems%2C+36%2C+2024%2E)
37. Jesse Mu, Xiang Li, and Noah Goodman. [Learning to compress prompts with gist tokens](https://md.research.elamur.ai/free/en/papers/mu-2023-learning-compress-prompts-gist-tokens/). Advances in Neural Information Processing Systems, 36, 2024.
38. Tsendsuren Munkhdalai and Hong Yu. Meta networks. In International conference on machine learning, pp. 2554–2563. PMLR, 2017a. [find](https://scholar.google.com/scholar?q=Tsendsuren+Munkhdalai+and+Hong+Yu%2E+Meta+networks%2E+In+International+conference+on+machine+learning%2C+pp%2E+2554%E2%80%932563%2E+PMLR%2C+2017a%2E)
39. Tsendsuren Munkhdalai and Hong Yu. Neural semantic encoders. In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 1, pp. 397. NIH Public Access, 2017b. [find](https://scholar.google.com/scholar?q=Tsendsuren+Munkhdalai+and+Hong+Yu%2E+Neural+semantic+encoders%2E+In+Proceedings+of+the+conference%2E+Association+for+Computational+Linguistics%2E+Meeting%2C+volume+1%2C+pp%2E+397%2E+NIH+Public+Access%2C+2017b%2E)
40. Tsendsuren Munkhdalai, John P Lalor, and Hong Yu. Citation analysis with neural attention models. In Proceedings of the Seventh International Workshop on Health Text Mining and Information Analysis, pp. 69–77, 2016. [find](https://scholar.google.com/scholar?q=Tsendsuren+Munkhdalai%2C+John+P+Lalor%2C+and+Hong+Yu%2E+Citation+analysis+with+neural+attention+models%2E+In+Proceedings+of+the+Seventh+International+Workshop+on+Health+Text+Mining+and+Information+Analysis%2C+p)
41. Tsendsuren Munkhdalai, Alessandro Sordoni, Tong Wang, and Adam Trischler. Metalearned neural memory. Advances in Neural Information Processing Systems, 32, 2019. [find](https://scholar.google.com/scholar?q=Tsendsuren+Munkhdalai%2C+Alessandro+Sordoni%2C+Tong+Wang%2C+and+Adam+Trischler%2E+Metalearned+neural+memory%2E+Advances+in+Neural+Information+Processing+Systems%2C+32%2C+2019%2E)
42. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071, 2023. [find](https://scholar.google.com/scholar?q=Bowen+Peng%2C+Jeffrey+Quesnelle%2C+Honglu+Fan%2C+and+Enrico+Shippole%2E+Yarn%3A+Efficient+context+window+extension+of+large+language+models%2E+arXiv+preprint+arXiv%3A2309%2E00071%2C+2023%2E)
43. Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5, 2023. [find](https://scholar.google.com/scholar?q=Reiner+Pope%2C+Sholto+Douglas%2C+Aakanksha+Chowdhery%2C+Jacob+Devlin%2C+James+Bradbury%2C+Jonathan+Heek%2C+Kefan+Xiao%2C+Shivani+Agrawal%2C+and+Jeff+Dean%2E+Efficiently+scaling+transformer+inference%2E+Proceedings+of+Mac)
44. Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409, 2021. [find](https://scholar.google.com/scholar?q=Ofir+Press%2C+Noah+A+Smith%2C+and+Mike+Lewis%2E+Train+short%2C+test+long%3A+Attention+with+linear+biases+enables+input+length+extrapolation%2E+arXiv+preprint+arXiv%3A2108%2E12409%2C+2021%2E)
45. Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. [Compressive transformers for long-range sequence modelling](https://md.research.elamur.ai/free/en/papers/rae-2019-compressive-transformers-long-range-sequence/). arXiv preprint arXiv:1911.05507, 2019.
46. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). The Journal of Machine Learning Research, 21(1):5485–5551, 2020.
47. Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Omri Abend, Ehud Karpas, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. Parallel context windows improve in-context learning of large language models. arXiv preprint arXiv:2212.10947, 2022. — [arXiv:2212.10947](https://arxiv.org/abs/2212.10947) [find](https://scholar.google.com/scholar?q=Nir+Ratner%2C+Yoav+Levine%2C+Yonatan+Belinkov%2C+Ori+Ram%2C+Omri+Abend%2C+Ehud+Karpas%2C+Amnon+Shashua%2C+Kevin+Leyton%2DBrown%2C+and+Yoav+Shoham%2E+Parallel+context+windows+improve+in%2Dcontext+learning+of+large+language+)
48. Imanol Schlag, Paul Smolensky, Roland Fernandez, Nebojsa Jojic, Jürgen Schmidhuber, and Jianfeng Gao. Enhancing the transformer with explicit relational encoding for math problem solving. arXiv preprint arXiv:1910.06611, 2019. — [arXiv:1910.06611](https://arxiv.org/abs/1910.06611) [find](https://scholar.google.com/scholar?q=Imanol+Schlag%2C+Paul+Smolensky%2C+Roland+Fernandez%2C+Nebojsa+Jojic%2C+Ju%CC%88rgen+Schmidhuber%2C+and+Jianfeng+Gao%2E+Enhancing+the+transformer+with+explicit+relational+encoding+for+math+problem+solving%2E+arXiv+prepr)
49. Imanol Schlag, Tsendsuren Munkhdalai, and Jürgen Schmidhuber. Learning associative inference using fast weight memory. arXiv preprint arXiv:2011.07831, 2020. — [arXiv:2011.07831](https://arxiv.org/abs/2011.07831) [find](https://scholar.google.com/scholar?q=Imanol+Schlag%2C+Tsendsuren+Munkhdalai%2C+and+Ju%CC%88rgen+Schmidhuber%2E+Learning+associative+inference+using+fast+weight+memory%2E+arXiv+preprint+arXiv%3A2011%2E07831%2C+2020%2E+%E2%80%94+%5BarXiv%3A2011%2E07831%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
50. Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. [Linear transformers are secretly fast weight programmers](https://md.research.elamur.ai/free/en/papers/schlag-2021-linear-transformers-secretly-fast-weight/). In International Conference on Machine Learning, pp. 9355–9366. PMLR, 2021.
51. Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131–139, 1992. [find](https://scholar.google.com/scholar?q=Ju%CC%88rgen+Schmidhuber%2E+Learning+to+control+fast%2Dweight+memories%3A+An+alternative+to+dynamic+recurrent+networks%2E+Neural+Computation%2C+4%281%29%3A131%E2%80%93139%2C+1992%2E)
52. Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp. 4596–4604. PMLR, 2018. [find](https://scholar.google.com/scholar?q=Noam+Shazeer+and+Mitchell+Stern%2E+Adafactor%3A+Adaptive+learning+rates+with+sublinear+memory+cost%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+4596%E2%80%934604%2E+PMLR%2C+2018%2E)
53. Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. arXiv preprint arXiv:1812.01243, 2018. — [arXiv:1812.01243](https://arxiv.org/abs/1812.01243) [find](https://scholar.google.com/scholar?q=Zhuoran+Shen%2C+Mingyuan+Zhang%2C+Haiyu+Zhao%2C+Shuai+Yi%2C+and+Hongsheng+Li%2E+Efficient+attention%3A+Attention+with+linear+complexities%2E+arXiv+preprint+arXiv%3A1812%2E01243%2C+2018%2E+%E2%80%94+%5BarXiv%3A1812%2E01243%5D%28https%3A%2F%2Farxiv)
54. Paul Smolensky. Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial intelligence, 46(1-2):159–216, 1990. [find](https://scholar.google.com/scholar?q=Paul+Smolensky%2E+Tensor+product+variable+binding+and+the+representation+of+symbolic+structures+in+connectionist+systems%2E+Artificial+intelligence%2C+46%281%2D2%29%3A159%E2%80%93216%2C+1990%2E)
55. Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. End-to-end memory networks. Advances in neural information processing systems, 28, 2015. [find](https://scholar.google.com/scholar?q=Sainbayar+Sukhbaatar%2C+Jason+Weston%2C+Rob+Fergus%2C+et+al%2E+End%2Dto%2Dend+memory+networks%2E+Advances+in+neural+information+processing+systems%2C+28%2C+2015%2E)
56. Sainbayar Sukhbaatar, Da Ju, Spencer Poff, Stephen Roller, Arthur Szlam, Jason Weston, and Angela Fan. Not all memories are created equal: Learning to forget by expiring. In International Conference on Machine Learning, pp. 9902–9912. PMLR, 2021. [find](https://scholar.google.com/scholar?q=Sainbayar+Sukhbaatar%2C+Da+Ju%2C+Spencer+Poff%2C+Stephen+Roller%2C+Arthur+Szlam%2C+Jason+Weston%2C+and+Angela+Fan%2E+Not+all+memories+are+created+equal%3A+Learning+to+forget+by+expiring%2E+In+International+Conference+o)
57. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023.
58. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30, 2017.
59. Paul J Werbos. Generalization of backpropagation with application to a recurrent gas market model. Neural networks, 1(4):339–356, 1988. [find](https://scholar.google.com/scholar?q=Paul+J+Werbos%2E+Generalization+of+backpropagation+with+application+to+a+recurrent+gas+market+model%2E+Neural+networks%2C+1%284%29%3A339%E2%80%93356%2C+1988%2E)
60. Yuhuai Wu, Markus N Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing transformers. arXiv preprint arXiv:2203.08913, 2022. [find](https://scholar.google.com/scholar?q=Yuhuai+Wu%2C+Markus+N+Rabe%2C+DeLesley+Hutchins%2C+and+Christian+Szegedy%2E+Memorizing+transformers%2E+arXiv+preprint+arXiv%3A2203%2E08913%2C+2022%2E)
61. Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, Song Han, and Maosong Sun. Infllm: Unveiling the intrinsic capacity of llms for understanding extremely long sequences with training-free memory. arXiv preprint arXiv:2402.[04617, 2024](https://md.research.elamur.ai/free/en/papers/xiao-2024-infllm-training-free-long-context/).
62. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023. [find](https://scholar.google.com/scholar?q=Guangxuan+Xiao%2C+Yuandong+Tian%2C+Beidi+Chen%2C+Song+Han%2C+and+Mike+Lewis%2E+Efficient+streaming+language+models+with+attention+sinks%2E+arXiv+preprint+arXiv%3A2309%2E17453%2C+2023%2E)
63. Wen Xiao, Iz Beltagy, Giuseppe Carenini, and Arman Cohan. [Primera: Pyramid-based masked sentence pre-training for multi-document summarization](https://md.research.elamur.ai/free/en/papers/xiao-2022-primera-pyramid-based-masked-sentence/). arXiv preprint arXiv:2110.08499, 2021.
64. Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. [Effective long-context scaling of foundation models](https://md.research.elamur.ai/free/en/papers/xiong-2023-effective-long-context-scaling-foundation/). arXiv preprint arXiv:2309.16039, 2023.

*Updated 2 November 2026*