---
title: "MEMORYLLM: Towards Self-Updatable Large Language Models"
description: "This paper proposes MEMORYLLM, which splits the model into two parts. 27 references, cited by 23 papers here."
authors: ["Yu Wang", "Yifan Gao", "Xiusi Chen", "Haoming Jiang", "Shiyang Li", "Jingfeng Yang", "Qingyu Yin", "Zheng Li", "Xian Li", "Bing Yin", "Jingbo Shang", "Julian McAuley"]
year: 2024
venue: "arXiv:2402.04624"
source: "https://arxiv.org/abs/2402.04624"
doi: "10.48550/arxiv.2402.04624"
arxiv: "2402.04624"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2024-memoryllm-towards-self-updatable-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2024-memoryllm-towards-self-updatable-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/) / MEMORYLLM: Towards Self-Updatable Large Language Models

# MEMORYLLM: Towards Self-Updatable Large Language Models

**Authors:** [Yu Wang](https://md.research.elamur.ai/free/en/authors/yu-wang/), [Yifan Gao](https://md.research.elamur.ai/free/en/authors/yifan-gao/), [Xiusi Chen](https://md.research.elamur.ai/free/en/authors/xiusi-chen/), [Haoming Jiang](https://md.research.elamur.ai/free/en/authors/haoming-jiang/), [Shiyang Li](https://md.research.elamur.ai/free/en/authors/shiyang-li/), [Jingfeng Yang](https://md.research.elamur.ai/free/en/authors/jingfeng-yang/), [Qingyu Yin](https://md.research.elamur.ai/free/en/authors/qingyu-yin/), [Zheng Li](https://md.research.elamur.ai/free/en/authors/zheng-li/), [Xian Li](https://md.research.elamur.ai/free/en/authors/xian-li/), [Bing Yin](https://md.research.elamur.ai/free/en/authors/bing-yin/), [Jingbo Shang](https://md.research.elamur.ai/free/en/authors/jingbo-shang/) and [Julian McAuley](https://md.research.elamur.ai/free/en/authors/julian-mcauley/) · **Year:** 2024 · **Venue:** arXiv:2402.04624 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2402.04624) · [PDF](https://arxiv.org/pdf/2402.04624) · [DOI 10.48550/arxiv.2402.04624](https://doi.org/10.48550/arxiv.2402.04624) · [arXiv 2402.04624](https://arxiv.org/abs/2402.04624) · [OpenAlex](https://openalex.org/W4391673048)

**Citations:** 4 citations (OpenAlex, 2026-10-14) · [23](https://md.research.elamur.ai/free/en/papers/wang-2024-memoryllm-towards-self-updatable-large/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Large language models are usually frozen once deployed, which makes adding fresh knowledge difficult. Current workarounds each have drawbacks. Retrieval stores keep growing, fact-level editing struggles with longer passages, and long-context approaches eventually run out of context length. This paper proposes MEMORYLLM, which splits the model into two parts. One is a frozen backbone (Llama2). The other is a constant-size pool of latent "memory tokens" placed in every transformer layer, which serves as the updatable parameters. When the model reads a new text passage, the backbone itself compresses it into a few new memory tokens per layer. An equal number of existing tokens is discarded at random, so older content fades roughly exponentially and the pool never grows. Updates require no back-propagation. The authors report:
- gains over model-editing and long-context baselines,
- strong recall in their own knowledge-retention experiments,
- stable behaviour after close to one million updates.

The work offers a bounded-memory way to keep deployed LLMs current without retraining or external knowledge bases.

## Key points
- **Architecture:** a Llama2 7B backbone (parameters ϕ, kept fixed) is paired with a memory pool of about 1B parameters (θ). The pool consists of N memory tokens of the embedding dimension d in each layer.
- **Generation:** every input token attends to the full memory pool, so cost grows linearly with pool size.
- **Self-update:** the last K memory tokens are combined with the new passage's hidden states. The last K outputs become the new memory, and K randomly chosen old tokens are dropped. Knowledge therefore decays at rate K/N, a design inspired by the Ebbinghaus forgetting curve.
- **Evaluation:** model-editing benchmarks (compared against MEND, ROME and IKE), long-context QA benchmarks, custom retention tests, and a robustness run of almost a million updates with no observed degradation.
- **Availability:** code and model are open-sourced at github.com/wangyu-ustc/MemoryLLM (ICML 2024).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of California, Los Angeles](https://md.research.elamur.ai/free/en/universities/university-of-california-los-angeles/) — University, Los Angeles, United States · 87 works · 54,961 citations
- [University of California San Diego](https://md.research.elamur.ai/free/en/universities/university-of-california-san-diego/) — University, San Diego, United States · 73 works · 51,948 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Yu Wang, Yifan Gao, Xiusi Chen, Haoming Jiang, Shiyang Li, Jingfeng Yang, Qingyu Yin, Zheng Li, Xian Li, Bing Yin, Jingbo Shang and Julian McAuley. 2024. MEMORYLLM: Towards Self-Updatable Large Language Models. arXiv:2402.04624.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{wang2024memoryllm,
  title = {{MEMORYLLM: Towards Self-Updatable Large Language Models}},
  author = {Yu Wang, Yifan Gao, Xiusi Chen, Haoming Jiang, Shiyang Li, Jingfeng Yang, Qingyu Yin, Zheng Li, Xian Li, Bing Yin, Jingbo Shang and Julian McAuley},
  year = {2024},
  doi = {10.48550/arxiv.2402.04624},
  eprint = {2402.04624},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.04624}
}
```

- [← previous Low-Resource Machine Translation through Retrieval-Augmented LLM Prompting: A Study on the Mambai Language](https://md.research.elamur.ai/free/en/papers/merx-2024-low-resource-machine-translation-through/)
- [next → MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response](https://md.research.elamur.ai/free/en/papers/deng-2024-musilingo-bridging-music-text-pre/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/)

## References

27 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bulatov, A., Kuratov, Y., and Burtsev, M. S. Recurrent memory transformer. In NeurIPS, 2022. [find](https://scholar.google.com/scholar?q=Bulatov%2C+A%2E%2C+Kuratov%2C+Y%2E%2C+and+Burtsev%2C+M%2E+S%2E+Recurrent+memory+transformer%2E+In+NeurIPS%2C+2022%2E)
2. Burtsev, M. S. and Sapunov, G. V. Memory transformer. CoRR, abs/2006.11527, 2020. URL https://arxiv.org/abs/2006.11527. — [arXiv:2006.11527](https://arxiv.org/abs/2006.11527) [↗](https://arxiv.org/abs/2006.11527)
3. Cao, N. D., Aziz, W., and Titov, I. Editing factual knowledge in language models. In EMNLP (1), pp. 6491–6506. Association for Computational Linguistics, 2021. [find](https://scholar.google.com/scholar?q=Cao%2C+N%2E+D%2E%2C+Aziz%2C+W%2E%2C+and+Titov%2C+I%2E+Editing+factual+knowledge+in+language+models%2E+In+EMNLP+%281%29%2C+pp%2E+6491%E2%80%936506%2E+Association+for+Computational+Linguistics%2C+2021%2E)
4. Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359–17372, 2022. [find](https://scholar.google.com/scholar?q=Meng%2C+K%2E%2C+Bau%2C+D%2E%2C+Andonian%2C+A%2E%2C+and+Belinkov%2C+Y%2E+Locating+and+editing+factual+associations+in+gpt%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A17359%E2%80%9317372%2C+2022%2E)
5. Mitchell, E., Lin, C., Bosselut, A., Finn, C., and Manning, C. D. Fast model editing at scale. In ICLR. OpenReview.net, 2022. [find](https://scholar.google.com/scholar?q=Mitchell%2C+E%2E%2C+Lin%2C+C%2E%2C+Bosselut%2C+A%2E%2C+Finn%2C+C%2E%2C+and+Manning%2C+C%2E+D%2E+Fast+model+editing+at+scale%2E+In+ICLR%2E+OpenReview%2Enet%2C+2022%2E)
6. Chen, S., Wong, S., Chen, L., and Tian, Y. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595, 2023a. [find](https://scholar.google.com/scholar?q=Chen%2C+S%2E%2C+Wong%2C+S%2E%2C+Chen%2C+L%2E%2C+and+Tian%2C+Y%2E+Extending+context+window+of+large+language+models+via+positional+interpolation%2E+arXiv+preprint+arXiv%3A2306%2E15595%2C+2023a%2E)
7. Modarressi, A., Imani, A., Fayyaz, M., and Schütze, H. Ret-llm: Towards a general read-write memory for large language models. arXiv preprint arXiv:2305.14322, 2023. — [arXiv:2305.14322](https://arxiv.org/abs/2305.14322) [find](https://scholar.google.com/scholar?q=Modarressi%2C+A%2E%2C+Imani%2C+A%2E%2C+Fayyaz%2C+M%2E%2C+and+Schu%CC%88tze%2C+H%2E+Ret%2Dllm%3A+Towards+a+general+read%2Dwrite+memory+for+large+language+models%2E+arXiv+preprint+arXiv%3A2305%2E14322%2C+2023%2E+%E2%80%94+%5BarXiv%3A2305%2E14322%5D%28https%3A%2F%2Farxi)
8. Chen, Y., Qian, S., Tang, H., Lai, X., Liu, Z., Han, S., and Jia, J. [Longlora: Efficient fine-tuning of long-context large language models](https://md.research.elamur.ai/free/en/papers/chen-2023-longlora-efficient-fine-tuning-long/). arXiv preprint arXiv:2309.12307, 2023b.
9. Moro, G., Ragazzi, L., Valgimigli, L., Frisoni, G., Sartori, C., and Marfia, G. Efficient memory-enhanced transformer for long-document summarization in low-resource regimes. Sensors, 23(7):3542, 2023. [find](https://scholar.google.com/scholar?q=Moro%2C+G%2E%2C+Ragazzi%2C+L%2E%2C+Valgimigli%2C+L%2E%2C+Frisoni%2C+G%2E%2C+Sartori%2C+C%2E%2C+and+Marfia%2C+G%2E+Efficient+memory%2Denhanced+transformer+for+long%2Ddocument+summarization+in+low%2Dresource+regimes%2E+Sensors%2C+23%287%29%3A3542%2C+2023)
10. Child, R., Gray, S., Radford, A., and Sutskever, I. [Generating long sequences with sparse transformers](https://md.research.elamur.ai/free/en/papers/child-2019-generating-long-sequences-sparse-transformers/). CoRR, abs/1904.10509, 2019. URL [http://arxiv.org/](http://arxiv.org/) abs/1904.10509. [↗](http://arxiv.org/)
11. Press, O., Smith, N. A., and Lewis, M. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409, 2021. [find](https://scholar.google.com/scholar?q=Press%2C+O%2E%2C+Smith%2C+N%2E+A%2E%2C+and+Lewis%2C+M%2E+Train+short%2C+test+long%3A+Attention+with+linear+biases+enables+input+length+extrapolation%2E+arXiv+preprint+arXiv%3A2108%2E12409%2C+2021%2E)
12. Computer, T. [Redpajama: an open dataset for training large language models](https://md.research.elamur.ai/free/en/papers/weber-2024-redpajama-open-dataset-training-large/), 2023. URL [https://github.com/togethercomputer/RedPajama-Data](https://github.com/togethercomputer/RedPajama-Data). [↗](https://github.com/togethercomputer/RedPajama-Data)
13. Cornia, M., Stefanini, M., Baraldi, L., and Cucchiara, R. Meshed-memory transformer for image captioning. In CVPR, pp. 10575–10584. Computer Vision Foundation / IEEE, 2020. [find](https://scholar.google.com/scholar?q=Cornia%2C+M%2E%2C+Stefanini%2C+M%2E%2C+Baraldi%2C+L%2E%2C+and+Cucchiara%2C+R%2E+Meshed%2Dmemory+transformer+for+image+captioning%2E+In+CVPR%2C+pp%2E+10575%E2%80%9310584%2E+Computer+Vision+Foundation+%2F+IEEE%2C+2020%2E)
14. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). J. Mach. Learn. Res., 21:140:1–140:67, 2020.
15. Sukhbaatar, S., Weston, J., Fergus, R., et al. End-to-end memory networks. Advances in neural information processing systems, 28, 2015. [find](https://scholar.google.com/scholar?q=Sukhbaatar%2C+S%2E%2C+Weston%2C+J%2E%2C+Fergus%2C+R%2E%2C+et+al%2E+End%2Dto%2Dend+memory+networks%2E+Advances+in+neural+information+processing+systems%2C+28%2C+2015%2E)
16. Sun, Y., Dong, L., Patra, B., Ma, S., Huang, S., Benhaim, A., Chaudhary, V., Song, X., and Wei, F. [A length-extrapolatable transformer](https://md.research.elamur.ai/free/en/papers/sun-2023-length-extrapolatable-transformer/). In ACL (1), pp. 14590–14604. Association for Computational Linguistics, 2023.
17. Zhong, W., Guo, L., Gao, Q., and Wang, Y. Memorybank: Enhancing large language models with long-term memory. arXiv preprint arXiv:2305.10250, 2023. [find](https://scholar.google.com/scholar?q=Zhong%2C+W%2E%2C+Guo%2C+L%2E%2C+Gao%2C+Q%2E%2C+and+Wang%2C+Y%2E+Memorybank%3A+Enhancing+large+language+models+with+long%2Dterm+memory%2E+arXiv+preprint+arXiv%3A2305%2E10250%2C+2023%2E)
18. Zhu, C., Rawat, A. S., Zaheer, M., Bhojanapalli, S., Li, D., Yu, F. X., and Kumar, S. Modifying memories in transformer models. CoRR, abs/2012.00363, 2020. [find](https://scholar.google.com/scholar?q=Zhu%2C+C%2E%2C+Rawat%2C+A%2E+S%2E%2C+Zaheer%2C+M%2E%2C+Bhojanapalli%2C+S%2E%2C+Li%2C+D%2E%2C+Yu%2C+F%2E+X%2E%2C+and+Kumar%2C+S%2E+Modifying+memories+in+transformer+models%2E+CoRR%2C+abs%2F2012%2E00363%2C+2020%2E)
19. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Canton-Ferrer, C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Ho… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
20. Tworkowski, S., Staniszewski, K., Pacek, M., Wu, Y., Michalewski, H., and Miłoś, P. Focused transformer: Contrastive training for context scaling. arXiv preprint arXiv:2307.03170, 2023. — [arXiv:2307.03170](https://arxiv.org/abs/2307.03170) [find](https://scholar.google.com/scholar?q=Tworkowski%2C+S%2E%2C+Staniszewski%2C+K%2E%2C+Pacek%2C+M%2E%2C+Wu%2C+Y%2E%2C+Michalewski%2C+H%2E%2C+and+Mi%C5%82os%CC%81%2C+P%2E+Focused+transformer%3A+Contrastive+training+for+context+scaling%2E+arXiv+preprint+arXiv%3A2307%2E03170%2C+2023%2E+%E2%80%94+%5BarXiv%3A2307)
21. Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020. [find](https://scholar.google.com/scholar?q=Wang%2C+S%2E%2C+Li%2C+B%2E+Z%2E%2C+Khabsa%2C+M%2E%2C+Fang%2C+H%2E%2C+and+Ma%2C+H%2E+Linformer%3A+Self%2Dattention+with+linear+complexity%2E+arXiv+preprint+arXiv%3A2006%2E04768%2C+2020%2E)
22. Wang, W., Dong, L., Cheng, H., Liu, X., Yan, X., Gao, J., and Wei, F. Augmenting language models with long-term memory. arXiv preprint arXiv:2306.07174, 2023. [find](https://scholar.google.com/scholar?q=Wang%2C+W%2E%2C+Dong%2C+L%2E%2C+Cheng%2C+H%2E%2C+Liu%2C+X%2E%2C+Yan%2C+X%2E%2C+Gao%2C+J%2E%2C+and+Wei%2C+F%2E+Augmenting+language+models+with+long%2Dterm+memory%2E+arXiv+preprint+arXiv%3A2306%2E07174%2C+2023%2E)
23. Weston, J., Chopra, S., and Bordes, A. Memory networks. arXiv preprint arXiv:1410.3916, 2014. [find](https://scholar.google.com/scholar?q=Weston%2C+J%2E%2C+Chopra%2C+S%2E%2C+and+Bordes%2C+A%2E+Memory+networks%2E+arXiv+preprint+arXiv%3A1410%2E3916%2C+2014%2E)
24. Wu, Q., Lan, Z., Qian, K., Gu, J., Geramifard, A., and Yu, Z. Memformer: A memory-augmented transformer for sequence modeling. In AACL/IJCNLP (Findings), pp. 308–318. Association for Computational Linguistics, 2022. [find](https://scholar.google.com/scholar?q=Wu%2C+Q%2E%2C+Lan%2C+Z%2E%2C+Qian%2C+K%2E%2C+Gu%2C+J%2E%2C+Geramifard%2C+A%2E%2C+and+Yu%2C+Z%2E+Memformer%3A+A+memory%2Daugmented+transformer+for+sequence+modeling%2E+In+AACL%2FIJCNLP+%28Findings%29%2C+pp%2E+308%E2%80%93318%2E+Association+for+Computational+Lin)
25. Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oguz, B., et al. [Effective long-context scaling of foundation models](https://md.research.elamur.ai/free/en/papers/xiong-2023-effective-long-context-scaling-foundation/). arXiv preprint arXiv:2309.16039, 2023.
26. Yao, Y., Wang, P., Tian, B., Cheng, S., Li, Z., Deng, S., Chen, H., and Zhang, N. Editing large language models: Problems, methods, and opportunities. CoRR, abs/2305.13172, 2023. [find](https://scholar.google.com/scholar?q=Yao%2C+Y%2E%2C+Wang%2C+P%2E%2C+Tian%2C+B%2E%2C+Cheng%2C+S%2E%2C+Li%2C+Z%2E%2C+Deng%2C+S%2E%2C+Chen%2C+H%2E%2C+and+Zhang%2C+N%2E+Editing+large+language+models%3A+Problems%2C+methods%2C+and+opportunities%2E+CoRR%2C+abs%2F2305%2E13172%2C+2023%2E)
27. Zheng, C., Li, L., Dong, Q., Fan, Y., Wu, Z., Xu, J., and Chang, B. Can we edit factual knowledge by in-context learning? arXiv preprint arXiv:2305.12740, 2023. [find](https://scholar.google.com/scholar?q=Zheng%2C+C%2E%2C+Li%2C+L%2E%2C+Dong%2C+Q%2E%2C+Fan%2C+Y%2E%2C+Wu%2C+Z%2E%2C+Xu%2C+J%2E%2C+and+Chang%2C+B%2E+Can+we+edit+factual+knowledge+by+in%2Dcontext+learning%3F+arXiv+preprint+arXiv%3A2305%2E12740%2C+2023%2E)

*Updated 30 October 2026*