---
title: "Associative Recurrent Memory Transformer — Rodkin 2024"
description: "They propose the Associative Recurrent Memory Transformer (ARMT), which builds… 79.9% accuracy. 31 references, cited by 8 papers here."
authors: ["Ivan Rodkin", "Yuri Kuratov", "Aydar Bulatov", "Mikhail Burtsev"]
year: 2024
venue: "arXiv:2407.04841"
source: "https://arxiv.org/abs/2407.04841"
doi: "10.48550/arxiv.2407.04841"
arxiv: "2407.04841"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/rodkin-2024-associative-recurrent-memory-transformer/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/rodkin-2024-associative-recurrent-memory-transformer/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/10/) / Associative Recurrent Memory Transformer

# Associative Recurrent Memory Transformer

**Authors:** [Ivan Rodkin](https://md.research.elamur.ai/free/en/authors/ivan-rodkin/), [Yuri Kuratov](https://md.research.elamur.ai/free/en/authors/yuri-kuratov/), [Aydar Bulatov](https://md.research.elamur.ai/free/en/authors/aydar-bulatov/) and [Mikhail Burtsev](https://md.research.elamur.ai/free/en/authors/mikhail-burtsev/) · **Year:** 2024 · **Venue:** arXiv:2407.04841 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2407.04841) · [PDF](https://arxiv.org/pdf/2407.04841) · [DOI 10.48550/arxiv.2407.04841](https://doi.org/10.48550/arxiv.2407.04841) · [arXiv 2407.04841](https://arxiv.org/abs/2407.04841) · [OpenAlex](https://openalex.org/W4400517511)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [8](https://md.research.elamur.ai/free/en/papers/rodkin-2024-associative-recurrent-memory-transformer/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The authors address a core difficulty in long-context modeling: building a network whose cost for each new chunk of input stays fixed, however long the sequence grows. They propose the Associative Recurrent Memory Transformer (ARMT), which builds on the segment-level recurrent model RMT. Ordinary self-attention handles each segment locally. In addition, every layer gets its own associative key-value matrix. Memory tokens from the previous segment are written into this matrix, and current tokens read from it. The update resembles linear attention with the DPFP-3 feature map. A correction to the normalization vector prevents forgetting after many overwrite operations. On synthetic associative retrieval tests, ARMT holds more key-value pairs than Mamba, despite having an equal recurrent state size, and it copes with repeated rewrites. Built on GPT-2 (137M) and trained on only 16k-token inputs, it sets a record on BABILong by answering questions at contexts of up to 50 million tokens. This shows that compact recurrent memory can generalize far beyond its training length.

## Key points
- **Architecture and ablations:**
  - ARMT is RMT plus a layerwise associative block, using the DPFP-3 nonlinearity.
  - An ablation shows that the normalization fix is needed for hundreds of erase-insert operations.
  - The PRMT variant has layerwise memory but no associative block. It gives no gain over RMT, which indicates that the associative mechanism drives the improvement.
- **Rewrite task:** ARMT was trained on 50 key-value pairs and still recalls perfectly with 500 updates, a 10x generalization.
- **BABILong:**
  - On QA1, ARMT reaches 79.9% accuracy at 50M tokens.
  - On the multi-hop tasks QA2–QA5, it performs strongly up to 10M tokens.
- **Length generalization:** ARMT generalizes about 60x beyond its training length (1M/16k), compared with roughly 8x for Mamba-130m (128k/16k).
- **Baselines:** The comparisons cover Mamba and RMT. The authors could not train RWKV successfully. The paper appeared at an ICML 2024 workshop, and the code is released on GitHub.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Moscow Institute of Physics and Technology](https://md.research.elamur.ai/free/en/universities/moscow-institute-of-physics-and-technology/) — University, Dolgoprudny, Russia · 8 works · 2,761 citations
- [AIRI - Artificial Intelligence Research Institute](https://md.research.elamur.ai/free/en/ngos/airi-artificial-intelligence-research-institute/) — Nonprofit, Moscow, Russia · 5 works · 29 citations
- [London Institute for Mathematical Sciences](https://md.research.elamur.ai/free/en/universities/london-institute-for-mathematical-sciences/) — University, London, United Kingdom · 2 works · 8 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

## Cite

Cite as: Ivan Rodkin, Yuri Kuratov, Aydar Bulatov and Mikhail Burtsev. 2024. Associative Recurrent Memory Transformer. arXiv:2407.04841.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{rodkin2024associative,
  title = {{Associative Recurrent Memory Transformer}},
  author = {Ivan Rodkin, Yuri Kuratov, Aydar Bulatov and Mikhail Burtsev},
  year = {2024},
  doi = {10.48550/arxiv.2407.04841},
  eprint = {2407.04841},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2407.04841}
}
```

- [← previous Assessing the Role of Context in Chat Translation Evaluation: Is Context Helpful and Under What Conditions?](https://md.research.elamur.ai/free/en/papers/agrawal-2024-assessing-role-context-chat-translation/)
- [next → Bilingual Rhetorical Structure Parsing with Large Parallel Annotations](https://md.research.elamur.ai/free/en/papers/chistova-2024-bilingual-rhetorical-structure-parsing-large/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/10/)

## References

31 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, p… [find](https://scholar.google.com/scholar?q=Sebastian+Borgeaud%2C+Arthur+Mensch%2C+Jordan+Hoffmann%2C+Trevor+Cai%2C+Eliza+Rutherford%2C+Katie+Millican%2C+George+Bm+Van+Den+Driessche%2C+Jean%2DBaptiste+Lespiau%2C+Bogdan+Damoc%2C+Aidan+Clark%2C+et+al%2E+Improving+langua)
2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020.
3. Aydar Bulatov, Yuri Kuratov, and Mikhail S. Burtsev. Recurrent memory transformer, 2022. [find](https://scholar.google.com/scholar?q=Aydar+Bulatov%2C+Yuri+Kuratov%2C+and+Mikhail+S%2E+Burtsev%2E+Recurrent+memory+transformer%2C+2022%2E)
4. Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. [Longlora: Efficient fine-tuning of long-context large language models](https://md.research.elamur.ai/free/en/papers/chen-2023-longlora-efficient-fine-tuning-long/). In The Twelfth International Conference on Learning Representations, 2023.
5. Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. [Adapting language models to compress contexts](https://md.research.elamur.ai/free/en/papers/chevalier-2023-adapting-language-models-compress-contexts/), 2023.
6. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988, 2019. [find](https://scholar.google.com/scholar?q=Zihang+Dai%2C+Zhilin+Yang%2C+Yiming+Yang%2C+Jaime+G+Carbonell%2C+Quoc+Le%2C+and+Ruslan+Salakhutdinov%2E+Transformer%2Dxl%3A+Attentive+language+models+beyond+a+fixed%2Dlength+context%2E+In+Proceedings+of+the+57th+Annual+M)
7. Soham De, Samuel L. Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, Guillaume Desjardins, Arnaud Doucet, David Budden, Yee Whye Teh, Razvan Pascanu, Nando De Freitas, and Caglar Gulcehre. Griffin: Mixing g… [find](https://scholar.google.com/scholar?q=Soham+De%2C+Samuel+L%2E+Smith%2C+Anushan+Fernando%2C+Aleksandar+Botev%2C+George+Cristian%2DMuraru%2C+Albert+Gu%2C+Ruba+Haroun%2C+Leonard+Berrada%2C+Yutian+Chen%2C+Srivatsan+Srinivasan%2C+Guillaume+Desjardins%2C+Arnaud+Doucet%2C+)
8. Daniel Y. Fu, Tri Dao, Khaled K. Saab, Armin W. Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models, 2023. [find](https://scholar.google.com/scholar?q=Daniel+Y%2E+Fu%2C+Tri+Dao%2C+Khaled+K%2E+Saab%2C+Armin+W%2E+Thomas%2C+Atri+Rudra%2C+and+Christopher+Re%CC%81%2E+Hungry+hungry+hippos%3A+Towards+language+modeling+with+state+space+models%2C+2023%2E)
9. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+Shawn+Presser%2C+and+Connor+Leahy%2E+The+pile%3A+An+800gb+dataset+of+)
10. Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2023. [find](https://scholar.google.com/scholar?q=Albert+Gu+and+Tri+Dao%2E+Mamba%3A+Linear%2Dtime+sequence+modeling+with+selective+state+spaces%2C+2023%2E)
11. Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2021. [find](https://scholar.google.com/scholar?q=Albert+Gu%2C+Karan+Goel%2C+and+Christopher+Re%2E+Efficiently+modeling+long+sequences+with+structured+state+spaces%2E+In+International+Conference+on+Learning+Representations%2C+2021%2E)
12. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning, pages 3929–3938. PMLR, 2020. — [REALM: Retrieval-Augmented Language Model Pre-Training](https://md.research.elamur.ai/free/en/papers/guu-2020-realm-retrieval-augmented-language-model/)
13. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 9 (8):1735–1780, November 1997. ISSN 0899-7667. doi: 10.1162/neco.1997.9.8.1735. URL https://doi.org/10.1162/neco.1997.9.8.1735. [↗](https://doi.org/10.1162/neco.1997.9.8.1735)
14. Samy Jelassi, David Brandfonbrener, Sham M. Kakade, and Eran Malach. Repeat after me: Transformers are better than state space models at copying, 2024. [find](https://scholar.google.com/scholar?q=Samy+Jelassi%2C+David+Brandfonbrener%2C+Sham+M%2E+Kakade%2C+and+Eran+Malach%2E+Repeat+after+me%3A+Transformers+are+better+than+state+space+models+at+copying%2C+2024%2E)
15. Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention, 2020. [find](https://scholar.google.com/scholar?q=Angelos+Katharopoulos%2C+Apoorv+Vyas%2C+Nikolaos+Pappas%2C+and+Franc%CC%A7ois+Fleuret%2E+Transformers+are+rnns%3A+Fast+autoregressive+transformers+with+linear+attention%2C+2020%2E)
16. Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. In search of needles in a 11m haystack: Recurrent memory finds what llms miss, 2024. [find](https://scholar.google.com/scholar?q=Yuri+Kuratov%2C+Aydar+Bulatov%2C+Petr+Anokhin%2C+Dmitry+Sorokin%2C+Artyom+Sorokin%2C+and+Mikhail+Burtsev%2E+In+search+of+needles+in+a+11m+haystack%3A+Recurrent+memory+finds+what+llms+miss%2C+2024%2E)
17. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Ja… — [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
18. Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Jiaju Lin, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Krishna Sri Ipsit Mantri, F… [find](https://scholar.google.com/scholar?q=Bo+Peng%2C+Eric+Alcaide%2C+Quentin+Anthony%2C+Alon+Albalak%2C+Samuel+Arcadinho%2C+Stella+Biderman%2C+Huanqi+Cao%2C+Xin+Cheng%2C+Michael+Chung%2C+Matteo+Grella%2C+Kranthi+Kiran+GV%2C+Xuzheng+He%2C+Haowen+Hou%2C+Jiaju+Lin%2C+Przem)
19. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations, 2023. [find](https://scholar.google.com/scholar?q=Bowen+Peng%2C+Jeffrey+Quesnelle%2C+Honglu+Fan%2C+and+Enrico+Shippole%2E+Yarn%3A+Efficient+context+window+extension+of+large+language+models%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2C+)
20. Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap. [Compressive transformers for long-range sequence modelling](https://md.research.elamur.ai/free/en/papers/rae-2019-compressive-transformers-long-range-sequence/). arXiv preprint, 2019. URL https://arxiv.org/abs/1911.05507. [↗](https://arxiv.org/abs/1911.05507)
21. Ohad Rubin and Jonathan Berant. Long-range language modeling with self-retrieval. arXiv preprint arXiv:2306.13421, 2023. — [arXiv:2306.13421](https://arxiv.org/abs/2306.13421) [find](https://scholar.google.com/scholar?q=Ohad+Rubin+and+Jonathan+Berant%2E+Long%2Drange+language+modeling+with+self%2Dretrieval%2E+arXiv+preprint+arXiv%3A2306%2E13421%2C+2023%2E+%E2%80%94+%5BarXiv%3A2306%2E13421%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2306%2E13421%29)
22. Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers, 2021. [find](https://scholar.google.com/scholar?q=Imanol+Schlag%2C+Kazuki+Irie%2C+and+Ju%CC%88rgen+Schmidhuber%2E+Linear+transformers+are+secretly+fast+weight+programmers%2C+2021%2E)
23. Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. [Replug: Retrieval-augmented black-box language models](https://md.research.elamur.ai/free/en/papers/shi-2023-replug-retrieval-augmented-black-box/). arXiv preprint arXiv:2301.12652, 2023.
24. Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. End-to-end memory networks, 2015. [find](https://scholar.google.com/scholar?q=Sainbayar+Sukhbaatar%2C+Arthur+Szlam%2C+Jason+Weston%2C+and+Rob+Fergus%2E+End%2Dto%2Dend+memory+networks%2C+2015%2E)
25. Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. [Retentive network: A successor to transformer for large language models](https://md.research.elamur.ai/free/en/papers/sun-2023-retentive-network-successor-transformer-large/), 2023.
26. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim-othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023.
27. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. [Attention is All you Need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in neural information processing systems, pages 5998–6008, 2017. URL [http://papers.nips.cc/paper/7181-attention-is-all-you-need](http://papers.nips.cc/paper/7181-attention-is-all-you-need). [↗](http://papers.nips.cc/paper/7181-attention-is-all-you-need)
28. Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/1410.3916. [↗](http://arxiv.org/abs/1410.3916)
29. Yuhuai Wu, Markus Norman Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing transformers. In International Conference on Learning Representations, 2022. URL [https://openreview.net/forum?id=TrjbxzRcnf-](https://openreview.net/forum?id=TrjbxzRcnf-). [↗](https://openreview.net/forum?id=TrjbxzRcnf-)
30. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks, 2023. [find](https://scholar.google.com/scholar?q=Guangxuan+Xiao%2C+Yuandong+Tian%2C+Beidi+Chen%2C+Song+Han%2C+and+Mike+Lewis%2E+Efficient+streaming+language+models+with+attention+sinks%2C+2023%2E)
31. Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, and Zhicheng Dou. Soaring from 4k to 400k: Extending llm’s context with activation beacon, 2024. [find](https://scholar.google.com/scholar?q=Peitian+Zhang%2C+Zheng+Liu%2C+Shitao+Xiao%2C+Ninglu+Shao%2C+Qiwei+Ye%2C+and+Zhicheng+Dou%2E+Soaring+from+4k+to+400k%3A+Extending+llm%E2%80%99s+context+with+activation+beacon%2C+2024%2E)

*Updated 1 November 2026*