---
title: "CoMeT — paper by Zhao et al. (2026) — summary and ideas"
description: "To make fine-tuning on very long sequences practical, the authors also propose layer-level pipeline parallelism. 63 references."
authors: ["Runsong Zhao", "Shilei Liu", "Jiwei Tang", "Langming Liu", "Haibin Chen", "Weidong Zhang", "Yujin Yuan", "Tong Xiao", "JingBo Zhu", "Wenbo Su", "Bo Zheng"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.1779/"
doi: "10.18653/v1/2026.acl-long.1779"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhao-2026-comet-collaborative-memory-transformer-efficient/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhao-2026-comet-collaborative-memory-transformer-efficient/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/27/) / CoMeT: Collaborative Memory Transformer for Efficient Long Context Modeling

# CoMeT: Collaborative Memory Transformer for Efficient Long Context Modeling

**Authors:** [Runsong Zhao](https://md.research.elamur.ai/free/en/authors/runsong-zhao/), [Shilei Liu](https://md.research.elamur.ai/free/en/authors/shilei-liu/), [Jiwei Tang](https://md.research.elamur.ai/free/en/authors/jiwei-tang/), [Langming Liu](https://md.research.elamur.ai/free/en/authors/langming-liu/), [Haibin Chen](https://md.research.elamur.ai/free/en/authors/haibin-chen/), [Weidong Zhang](https://md.research.elamur.ai/free/en/authors/weidong-zhang/), [Yujin Yuan](https://md.research.elamur.ai/free/en/authors/yujin-yuan/), [Tong Xiao](https://md.research.elamur.ai/free/en/authors/tong-xiao/), [JingBo Zhu](https://md.research.elamur.ai/free/en/authors/jingbo-zhu/), [Wenbo Su](https://md.research.elamur.ai/free/en/authors/wenbo-su/) and [Bo Zheng](https://md.research.elamur.ai/free/en/authors/bo-zheng/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.1779/) · [PDF](https://aclanthology.org/2026.acl-long.1779.pdf) · [DOI 10.18653/v1/2026.acl-long.1779](https://doi.org/10.18653/v1/2026.acl-long.1779) · [OpenAlex](https://openalex.org/W7166884820)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Standard Transformers struggle with very long inputs because attention cost grows quadratically and the KV cache keeps expanding. CoMeT is a lightweight plug-in that lets an existing pre-trained LLM read input chunk by chunk while keeping memory constant and time linear. Two memories are prepended to each chunk as a soft prompt. A temporary memory, organized as a FIFO queue, keeps detailed traces of recent chunks. A fixed-size global memory uses a gated update so that important older facts are not overwritten. To make fine-tuning on very long sequences practical, the authors also propose layer-level pipeline parallelism. After training on only 32k-token contexts, CoMeT finds a passkey anywhere in a 1M-token input. Among efficient methods it achieves the best average score on SCROLLS, and on summarization it matches full attention. It also proves useful on real-world agent and user-behavior QA. This matters because it offers an affordable way to extend existing LLMs to arbitrarily long contexts without retraining them from scratch.

## Key points
- **Passkey extrapolation:** trained on 32k contexts, CoMeT retrieves the passkey at any depth in 1M tokens. At that length it runs about 21× faster than full attention and uses roughly 10× less memory.
- **Training efficiency:** compared with a naive context-parallel setup, the layer-level pipeline parallelism delivers a 2.7× speedup, making fine-tuning on up to 128k tokens possible with 16×80GB GPUs.
- **Inference with Qwen3-4B-Instruct-2507:**
  - CoMeT keeps peak memory at about 10GB and decodes at roughly 22ms per token.
  - Full attention reaches 104ms per token at 65k and runs out of memory at 128k.
- **Ablations:**
  - On SCROLLS, gains from temporary memory level off at 2,048 tokens.
  - Global memory drives length extrapolation, and removing its gate causes passkey retrieval to collapse.
- **Hyperparameters:** a Residual Low-Rank Adapter (RLA) rank of 8 gives the best SCROLLS average (40.10). Summarization with about 2.5k memory tokens performs on par with full attention.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 415 works · 94,028 citations
- [Alibaba Group](https://md.research.elamur.ai/free/en/companies/alibaba-group/) — Company, Hangzhou, China · 201 works · 17,173 citations
- [Northeastern University](https://md.research.elamur.ai/free/en/universities/northeastern-university-cn/) — University, Shenyang, China · 53 works · 2,205 citations
- [Alibaba Group (United States)](https://md.research.elamur.ai/free/en/companies/alibaba-group-united-states/) — Company, San Mateo, United States · 8 works · 329 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

## Cite

Cite as: Runsong Zhao, Shilei Liu, Jiwei Tang, Langming Liu, Haibin Chen, Weidong Zhang, Yujin Yuan, Tong Xiao, JingBo Zhu, Wenbo Su and Bo Zheng. 2026. CoMeT: Collaborative Memory Transformer for Efficient Long Context Modeling. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhao2026comet,
  title = {{CoMeT: Collaborative Memory Transformer for Efficient Long Context Modeling}},
  author = {Runsong Zhao, Shilei Liu, Jiwei Tang, Langming Liu, Haibin Chen, Weidong Zhang, Yujin Yuan, Tong Xiao, JingBo Zhu, Wenbo Su and Bo Zheng},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1779},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1779}
}
```

- [← previous Combining Semantic Embeddings and Knowledge Graphs for Identifying Decision Patterns in Brazilian Judicial Decisions](https://md.research.elamur.ai/free/en/papers/silva-2026-combining-semantic-embeddings-knowledge-graphs/)
- [next → ComfyFlow: Benchmarking LLMs for AIGC Workflow Generation](https://md.research.elamur.ai/free/en/papers/xu-2026-comfyflow-benchmarking-llms-aigc-workflow/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/27/)

## References

63 references, 33 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. 2022. [SummScreen: A dataset for abstractive screenplay summarization](https://md.research.elamur.ai/free/en/papers/chen-2022-summscreen-dataset-abstractive-screenplay-summarization/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8602–8615, Dublin, Ireland. Association for Computati…
2. Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. 2023. [Adapting language models to compress contexts](https://md.research.elamur.ai/free/en/papers/chevalier-2023-adapting-language-models-compress-contexts/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3829–3846, Singapore. Association for Computational Linguistics.
3. Yu-Neng Chuang, Tianwei Xing, Chia-Yuan Chang, Zirui Liu, Xun Chen, and Xia Hu. 2024. Learning to compress prompt in natural language formats. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1:… [find](https://scholar.google.com/scholar?q=Yu%2DNeng+Chuang%2C+Tianwei+Xing%2C+Chia%2DYuan+Chang%2C+Zirui+Liu%2C+Xun+Chen%2C+and+Xia+Hu%2E+2024%2E+Learning+to+compress+prompt+in+natural+language+formats%2E+In+Proceedings+of+the+2024+Conference+of+the+North+Americ)
4. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 2978–2988. [find](https://scholar.google.com/scholar?q=Zihang+Dai%2C+Zhilin+Yang%2C+Yiming+Yang%2C+Jaime+G+Carbonell%2C+Quoc+Le%2C+and+Ruslan+Salakhutdinov%2E+2019%2E+Transformer%2Dxl%3A+Attentive+language+models+beyond+a+fixed%2Dlength+context%2E+In+Proceedings+of+the+57th+an)
5. Tri Dao and Albert Gu. 2024. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 10041–10071. PMLR. [find](https://scholar.google.com/scholar?q=Tri+Dao+and+Albert+Gu%2E+2024%2E+Transformers+are+SSMs%3A+Generalized+models+and+efficient+algorithms+through+structured+state+space+duality%2E+In+Proceedings+of+the+41st+International+Conference+on+Machine+L)
6. Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. [A dataset of information-seeking questions and answers anchored in research papers](https://md.research.elamur.ai/free/en/papers/dasigi-2021-dataset-information-seeking-questions-answers/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan…
7. Siyu Ding, Junyuan Shang, Shuohuan Wang, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2021. [Ernie-doc: A retrospective long-document modeling transformer](https://md.research.elamur.ai/free/en/papers/ding-2020-ernie-doc-retrospective-long/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural…
8. Jun Gao, Ziqiang Cao, and Wenjie Li. 2024. Selfcp: Compressing over-limit prompt via the frozen large language model itself. Inf. Process. Manag., 61:103873. [find](https://scholar.google.com/scholar?q=Jun+Gao%2C+Ziqiang+Cao%2C+and+Wenjie+Li%2E+2024%2E+Selfcp%3A+Compressing+over%2Dlimit+prompt+via+the+frozen+large+language+model+itself%2E+Inf%2E+Process%2E+Manag%2E%2C+61%3A103873%2E)
9. Tao Ge, Hu Jing, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. 2024. [In-context autoencoder for context compression in a large language model](https://md.research.elamur.ai/free/en/papers/ge-2023-context-autoencoder-context-compression-large/). In The Twelfth International Conference on Learning Representations.
10. Albert Gu and Tri Dao. 2024. Mamba: Linear-time sequence modeling with selective state spaces. In First conference on language modeling. [find](https://scholar.google.com/scholar?q=Albert+Gu+and+Tri+Dao%2E+2024%2E+Mamba%3A+Linear%2Dtime+sequence+modeling+with+selective+state+spaces%2E+In+First+conference+on+language+modeling%2E)
11. Albert Gu, Karan Goel, and Christopher Re. 2022. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Albert+Gu%2C+Karan+Goel%2C+and+Christopher+Re%2E+2022%2E+Efficiently+modeling+long+sequences+with+structured+state+spaces%2E+In+International+Conference+on+Learning+Representations%2E)
12. Zifan He, Yingqi Cao, Zongyue Qin, Neha Prakriya, Yizhou Sun, and Jason Cong. 2025. HMT: Hierarchical memory transformer for efficient long context language processing. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Huma… [find](https://scholar.google.com/scholar?q=Zifan+He%2C+Yingqi+Cao%2C+Zongyue+Qin%2C+Neha+Prakriya%2C+Yizhou+Sun%2C+and+Jason+Cong%2E+2025%2E+HMT%3A+Hierarchical+memory+transformer+for+efficient+long+context+language+processing%2E+In+Proceedings+of+the+2025+Conf)
13. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. [Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps](https://md.research.elamur.ai/free/en/papers/ho-2020-constructing-multi-hop-qa-dataset/). In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609–6625, Barcelona, Spain (Online). International Comm…
14. Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021. [Efficient attentions for long document summarization](https://md.research.elamur.ai/free/en/papers/huang-2021-efficient-attentions-long-document-summarization/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1419–1436, Online.…
15. Taeho Hwang, Sukmin Cho, Soyeong Jeong, Hoyun Song, SeungYoon Han, and Jong C. Park. 2025. Exit: Context-aware extractive compression for enhancing retrieval-augmented generation. Preprint, arXiv:2412.12559. [find](https://scholar.google.com/scholar?q=Taeho+Hwang%2C+Sukmin+Cho%2C+Soyeong+Jeong%2C+Hoyun+Song%2C+SeungYoon+Han%2C+and+Jong+C%2E+Park%2E+2025%2E+Exit%3A+Context%2Daware+extractive+compression+for+enhancing+retrieval%2Daugmented+generation%2E+Preprint%2C+arXiv%3A2412)
16. Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. [LLMLingua: Compressing prompts for accelerated inference of large language models](https://md.research.elamur.ai/free/en/papers/jiang-2023-llmlingua-compressing-prompts-accelerated-inference/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13358–13376, Singapore. Association for Compu…
17. Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2024. [LongLLMLingua: Accelerating and enhancing LLMs in long context scenarios via prompt compression](https://md.research.elamur.ai/free/en/papers/jiang-2023-longllmlingua-accelerating-enhancing-llms-long/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1:…
18. Dongwon Jung, Qin Liu, Tenghao Huang, Ben Zhou, and Muhao Chen. 2025. Familiarity-aware evidence compression for retrieval-augmented generation. Preprint, arXiv:2409.12468. [find](https://scholar.google.com/scholar?q=Dongwon+Jung%2C+Qin+Liu%2C+Tenghao+Huang%2C+Ben+Zhou%2C+and+Muhao+Chen%2E+2025%2E+Familiarity%2Daware+evidence+compression+for+retrieval%2Daugmented+generation%2E+Preprint%2C+arXiv%3A2409%2E12468%2E)
19. Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. 2020. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pages 5156–5165. PMLR. [find](https://scholar.google.com/scholar?q=Angelos+Katharopoulos%2C+Apoorv+Vyas%2C+Nikolaos+Pappas%2C+and+Fran%C3%A7ois+Fleuret%2E+2020%2E+Transformers+are+rnns%3A+Fast+autoregressive+transformers+with+linear+attention%2E+In+International+conference+on+machine+l)
20. Yuta Koreeda and Christopher Manning. 2021. [ContractNLI: A dataset for document-level natural language inference for contracts](https://md.research.elamur.ai/free/en/papers/koreeda-2021-contractnli-dataset-document-level-natural/). In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1907–1919, Punta Cana, Dominican Republic. Association for Computational Linguistics.
21. Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. [The narrativeqa reading comprehension challenge](https://md.research.elamur.ai/free/en/papers/kocisky-2018-narrativeqa-reading-comprehension-challenge/). Transactions of the Association for Computational Linguistics, 6:317–328.
22. Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville. 2025. [Llms get lost in multi-turn conversation](https://md.research.elamur.ai/free/en/papers/laban-2025-llms-get-lost-multi-turn/). Preprint, arXiv:2505.06120.
23. Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. 2023. [Compressing context to enhance inference efficiency of large language models](https://md.research.elamur.ai/free/en/papers/li-2023-compressing-context-enhance-inference-efficiency/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6342–6353, Singapore. Association for Computational Linguistics.
24. Zongqian Li, Yixuan Su, and Nigel Collier. 2025. [500xCompressor: Generalized prompt compression for large language models](https://md.research.elamur.ai/free/en/papers/li-2025-500xcompressor-generalized-prompt-compression-large/). In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 25081–25091, Vienna, Austria. Association for Computati…
25. Xin Liu, Runsong Zhao, Pengcheng Huang, Xinyu Liu, Junyi Xiao, Chunyang Xiao, Tong Xiao, Shengxiang Gao, Zhengtao Yu, and Jingbo Zhu. 2025. Autoencoding-free context compression for llms via contextual semantic anchors. Preprint, arXiv:2510.08907. — [arXiv:2510.08907](https://arxiv.org/abs/2510.08907) [find](https://scholar.google.com/scholar?q=Xin+Liu%2C+Runsong+Zhao%2C+Pengcheng+Huang%2C+Xinyu+Liu%2C+Junyi+Xiao%2C+Chunyang+Xiao%2C+Tong+Xiao%2C+Shengxiang+Gao%2C+Zhengtao+Yu%2C+and+Jingbo+Zhu%2E+2025%2E+Autoencoding%2Dfree+context+compression+for+llms+via+contextua)
26. Jesse Mu, Xiang Li, and Noah Goodman. 2023a. [Learning to compress prompts with gist tokens](https://md.research.elamur.ai/free/en/papers/mu-2023-learning-compress-prompts-gist-tokens/). Advances in Neural Information Processing Systems, 36:19327– 19352.
27. Jesse Mu, Xiang Lisa Li, and Noah Goodman. 2023b. [Learning to compress prompts with gist tokens](https://md.research.elamur.ai/free/en/papers/mu-2023-learning-compress-prompts-gist-tokens/). In Thirty-seventh Conference on Neural Information Processing Systems.
28. Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. 2024. [Leave no context behind: Efficient infinite context transformers with infini-attention](https://md.research.elamur.ai/free/en/papers/munkhdalai-2024-leave-no-context-behind-efficient/). arXiv preprint arXiv:2404.07143, 101.
29. Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. 2023. [Resurrecting recurrent neural networks for long sequences](https://md.research.elamur.ai/free/en/papers/orvieto-2023-resurrecting-recurrent-neural-networks-long/). In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Researc…
30. Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Ruhle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. 2024. [LLMLingua-2: Data distillation for efficient and faithful task-agnostic prompt compression](https://md.research.elamur.ai/free/en/papers/pan-2024-llmlingua-2-data-distillation-efficient/). In Findings of the Associat…
31. Bo Pang, Erik Nijkamp, Wojciech Kryscinski, Silvio Savarese, Yingbo Zhou, and Caiming Xiong. 2023. Long document summarization with top-down and bottom-up inference. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1267–1284, Dubrovnik, Croatia. Association for Computat… [find](https://scholar.google.com/scholar?q=Bo+Pang%2C+Erik+Nijkamp%2C+Wojciech+Kryscinski%2C+Silvio+Savarese%2C+Yingbo+Zhou%2C+and+Caiming+Xiong%2E+2023%2E+Long+document+summarization+with+top%2Ddown+and+bottom%2Dup+inference%2E+In+Findings+of+the+Association+for)
32. Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, and Samuel Bowman. 2022. [QuALITY: Question answering with long input texts, yes!](https://md.research.elamur.ai/free/en/papers/pang-2022-quality-question-answering-long-input/) In Proceedings of the 2022 Conference of the North American Chapter of…
33. Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Gv, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartłomiej Koptyra, and 13 others. 2023. RWKV:… [find](https://scholar.google.com/scholar?q=Bo+Peng%2C+Eric+Alcaide%2C+Quentin+Anthony%2C+Alon+Albalak%2C+Samuel+Arcadinho%2C+Stella+Biderman%2C+Huanqi+Cao%2C+Xin+Cheng%2C+Michael+Chung%2C+Leon+Derczynski%2C+Xingjian+Du%2C+Matteo+Grella%2C+Kranthi+Gv%2C+Xuzheng+He%2C+Haow)
34. Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Xingjian Du, Teddy Ferdinan, Haowen Hou, Przemysław Kazienko, Kranthi Kiran GV, Jan Kocoń, Bartłomiej Koptyra, Satyapriya Krishna, Ronald Mc-Clelland Jr., Jiaju Lin, Niklas Muennighoff, Fares Obei… [find](https://scholar.google.com/scholar?q=Bo+Peng%2C+Daniel+Goldstein%2C+Quentin+Anthony%2C+Alon+Albalak%2C+Eric+Alcaide%2C+Stella+Biderman%2C+Eugene+Cheah%2C+Xingjian+Du%2C+Teddy+Ferdinan%2C+Haowen+Hou%2C+Przemys%C5%82aw+Kazienko%2C+Kranthi+Kiran+GV%2C+Jan+Kocon%CC%81%2C+Bart%C5%82)
35. Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, and Yiran Zhong. 2024. Hgrn2: Gated linear rnns with state expansion. arXiv preprint arXiv:2404.07904. — [arXiv:2404.07904](https://arxiv.org/abs/2404.07904) [find](https://scholar.google.com/scholar?q=Zhen+Qin%2C+Songlin+Yang%2C+Weixuan+Sun%2C+Xuyang+Shen%2C+Dong+Li%2C+Weigao+Sun%2C+and+Yiran+Zhong%2E+2024%2E+Hgrn2%3A+Gated+linear+rnns+with+state+expansion%2E+arXiv+preprint+arXiv%3A2404%2E07904%2E+%E2%80%94+%5BarXiv%3A2404%2E07904%5D%28https)
36. Zhen Qin, Songlin Yang, and Yiran Zhong. 2023. Hierarchically gated recurrent neural network for sequence modeling. In Advances in Neural Information Processing Systems, volume 36, pages 33202–33221. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Zhen+Qin%2C+Songlin+Yang%2C+and+Yiran+Zhong%2E+2023%2E+Hierarchically+gated+recurrent+neural+network+for+sequence+modeling%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+36%2C+pages+33202%E2%80%9333221%2E+)
37. Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. 2019. [Compressive transformers for long-range sequence modelling](https://md.research.elamur.ai/free/en/papers/rae-2019-compressive-transformers-long-range-sequence/). arXiv preprint arXiv:1911.05507.
38. Ivan Rodkin, Yuri Kuratov, Aydar Bulatov, and Mikhail Burtsev. 2024. [Associative recurrent memory transformer](https://md.research.elamur.ai/free/en/papers/rodkin-2024-associative-recurrent-memory-transformer/). Preprint, arXiv:2407.04841.
39. Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, and Omer Levy. 2022. [SCROLLS: Standardized CompaRison over long language sequences](https://md.research.elamur.ai/free/en/papers/shaham-2022-scrolls-standardized-comparison-over-long/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pa…
40. Claude E Shannon. 1948. A mathematical theory of communication. The Bell system technical journal, 27(3):379–423. [find](https://scholar.google.com/scholar?q=Claude+E+Shannon%2E+1948%2E+A+mathematical+theory+of+communication%2E+The+Bell+system+technical+journal%2C+27%283%29%3A379%E2%80%93423%2E)
41. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053. [find](https://scholar.google.com/scholar?q=Mohammad+Shoeybi%2C+Mostofa+Patwary%2C+Raul+Puri%2C+Patrick+LeGresley%2C+Jared+Casper%2C+and+Bryan+Catanzaro%2E+2019%2E+Megatron%2Dlm%3A+Training+multi%2Dbillion+parameter+language+models+using+model+parallelism%2E+arXiv+p)
42. Jimmy T.H. Smith, Andrew Warrington, and Scott Linderman. 2023. Simplified state space layers for sequence modeling. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Jimmy+T%2EH%2E+Smith%2C+Andrew+Warrington%2C+and+Scott+Linderman%2E+2023%2E+Simplified+state+space+layers+for+sequence+modeling%2E+In+The+Eleventh+International+Conference+on+Learning+Representations%2E)
43. Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. 2023. [Retentive network: A successor to transformer for large language models](https://md.research.elamur.ai/free/en/papers/sun-2023-retentive-network-successor-transformer-large/). arXiv preprint arXiv:2307.08621.
44. Sijun Tan, Xiuyu Li, Shishir G Patil, Ziyang Wu, Tianjun Zhang, Kurt Keutzer, Joseph E. Gonzalez, and Raluca Ada Popa. 2024. LLoCO: Learning long contexts offline. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17605–17621, Miami, Florida, USA. Assoc… [find](https://scholar.google.com/scholar?q=Sijun+Tan%2C+Xiuyu+Li%2C+Shishir+G+Patil%2C+Ziyang+Wu%2C+Tianjun+Zhang%2C+Kurt+Keutzer%2C+Joseph+E%2E+Gonzalez%2C+and+Raluca+Ada+Popa%2E+2024%2E+LLoCO%3A+Learning+long+contexts+offline%2E+In+Proceedings+of+the+2024+Conferenc)
45. Jiwei Tang, Jin Xu, Tingwei Lu, Zhicheng Zhang, Yiming Zhao, Lin Hai, and Hai-Tao Zheng. 2025a. Perception compressor: A training-free prompt compression framework in long context scenarios. In NAACL (Findings), pages 4093–4108. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jiwei+Tang%2C+Jin+Xu%2C+Tingwei+Lu%2C+Zhicheng+Zhang%2C+Yiming+Zhao%2C+Lin+Hai%2C+and+Hai%2DTao+Zheng%2E+2025a%2E+Perception+compressor%3A+A+training%2Dfree+prompt+compression+framework+in+long+context+scenarios%2E+In+NAACL+)
46. Jiwei Tang, Zhicheng Zhang, Shunlong Wu, Jingheng Ye, Lichen Bai, Zitai Wang, Tingwei Lu, Jiaqi Chen, Lin Hai, Hai-Tao Zheng, and Hong-Gee Kim. 2025b. [GMSA: enhancing context compression via group merging and layer semantic alignment](https://md.research.elamur.ai/free/en/papers/tang-2026-gmsa-enhancing-context-compression-group/). CoRR, abs/2505.12215.
47. Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2022. [Efficient transformers: A survey](https://md.research.elamur.ai/free/en/papers/tay-2020-efficient-transformers-survey/).
48. Qwen Team. 2025a. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). Preprint, arXiv:2505.09388.
49. The Terminal-Bench Team. 2025b. Terminal-bench: A benchmark for ai agents in terminal environments. [find](https://scholar.google.com/scholar?q=The+Terminal%2DBench+Team%2E+2025b%2E+Terminal%2Dbench%3A+A+benchmark+for+ai+agents+in+terminal+environments%2E)
50. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30.
51. Qingyang Wu, Zhenzhong Lan, Kun Qian, Jing Gu, Alborz Geramifard, and Zhou Yu. 2022. Memformer: A memory-augmented transformer for sequence modeling. In Findings of the association for computational linguistics: AACL-IJCNLP 2022, pages 308–318. [find](https://scholar.google.com/scholar?q=Qingyang+Wu%2C+Zhenzhong+Lan%2C+Kun+Qian%2C+Jing+Gu%2C+Alborz+Geramifard%2C+and+Zhou+Yu%2E+2022%2E+Memformer%3A+A+memory%2Daugmented+transformer+for+sequence+modeling%2E+In+Findings+of+the+association+for+computational+l)
52. Tong Xiao and Jingbo Zhu. 2023. Introduction to transformers: an nlp perspective. [find](https://scholar.google.com/scholar?q=Tong+Xiao+and+Jingbo+Zhu%2E+2023%2E+Introduction+to+transformers%3A+an+nlp+perspective%2E)
53. Songlin Yang, Jan Kautz, and Ali Hatamizadeh. 2024a. Gated delta networks: Improving mamba2 with delta rule. arXiv preprint arXiv:2412.06464. [find](https://scholar.google.com/scholar?q=Songlin+Yang%2C+Jan+Kautz%2C+and+Ali+Hatamizadeh%2E+2024a%2E+Gated+delta+networks%3A+Improving+mamba2+with+delta+rule%2E+arXiv+preprint+arXiv%3A2412%2E06464%2E)
54. Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. 2024b. Parallelizing linear transformers with the delta rule over sequence length. Advances in neural information processing systems, 37:115491–115522. [find](https://scholar.google.com/scholar?q=Songlin+Yang%2C+Bailin+Wang%2C+Yu+Zhang%2C+Yikang+Shen%2C+and+Yoon+Kim%2E+2024b%2E+Parallelizing+linear+transformers+with+the+delta+rule+over+sequence+length%2E+Advances+in+neural+information+processing+systems%2C+37)
55. Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, and Zhicheng Dou. 2025b. [Long context compression with activation beacon](https://md.research.elamur.ai/free/en/papers/zhang-2024-long-context-compression-activation-beacon/). In The Thirteenth International Conference on Learning Representations.
56. Yu Zhang, Songlin Yang, Ruijie Zhu, Yue Zhang, Leyang Cui, Yiqiao Wang, Bolun Wang, Freda Shi, Bailin Wang, Wei Bi, Peng Zhou, and Guohong Fu. 2024. Gated slot attention for efficient linear-time sequence modeling. ArXiv, abs/2409.07146. — [arXiv:2409.07146](https://arxiv.org/abs/2409.07146) [find](https://scholar.google.com/scholar?q=Yu+Zhang%2C+Songlin+Yang%2C+Ruijie+Zhu%2C+Yue+Zhang%2C+Leyang+Cui%2C+Yiqiao+Wang%2C+Bolun+Wang%2C+Freda+Shi%2C+Bailin+Wang%2C+Wei+Bi%2C+Peng+Zhou%2C+and+Guohong+Fu%2E+2024%2E+Gated+slot+attention+for+efficient+linear%2Dtime+sequ)
57. Runsong Zhao, Xin Liu, Xinyu Liu, Pengcheng Huang, Chunyang Xiao, Tong Xiao, and JingBo Zhu. 2025. Position IDs matter: An enhanced position layout for efficient context compression in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 17715– 17734… [find](https://scholar.google.com/scholar?q=Runsong+Zhao%2C+Xin+Liu%2C+Xinyu+Liu%2C+Pengcheng+Huang%2C+Chunyang+Xiao%2C+Tong+Xiao%2C+and+JingBo+Zhu%2E+2025%2E+Position+IDs+matter%3A+An+enhanced+position+layout+for+efficient+context+compression+in+large+language+)
58. Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021. [QMSum: A new benchmark for query-based multi-domain meeting summarization](https://md.research.elamur.ai/free/en/papers/zhong-2021-qmsum-new-benchmark-query-based/). In Proceedings of the 2021 Conference of the North American Chapte…
59. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. [HotpotQA: A dataset for diverse, explainable multi-hop question answering](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369…
60. Zihao Yi, Jiarui Ouyang, Yuwen Liu, Tianhao Liao, Zhe Xu, and Ying Shen. 2024. [A survey on recent advances in llm-based multi-turn dialogue systems](https://md.research.elamur.ai/free/en/papers/yi-2024-survey-recent-advances-llm-based/). arXiv preprint arXiv:2402.18013.
61. Chanwoong Yoon, Taewhoo Lee, Hyeon Hwang, Minbyul Jeong, and Jaewoo Kang. 2024. [Compact: Compressing retrieved documents actively for question answering](https://md.research.elamur.ai/free/en/papers/yoon-2024-compact-compressing-retrieved-documents-actively/). Preprint, arXiv:2407.09014.
62. Zhiqiang Yuan, Junwei Liu, Qiancheng Zi, Ming-wei Liu, Xin Peng, and Yiling Lou. 2023. Evaluating instruction-tuned large language models on code comprehension and generation. Preprint, arXiv:2308.01240. — [arXiv:2308.01240](https://arxiv.org/abs/2308.01240) [find](https://scholar.google.com/scholar?q=Zhiqiang+Yuan%2C+Junwei+Liu%2C+Qiancheng+Zi%2C+Ming%2Dwei+Liu%2C+Xin+Peng%2C+and+Yiling+Lou%2E+2023%2E+Evaluating+instruction%2Dtuned+large+language+models+on+code+comprehension+and+generation%2E+Preprint%2C+arXiv%3A2308%2E012)
63. Jiajie Zhang, Yushi Bai, Xin Lv, Wanjun Gu, Danqing Liu, Minhao Zou, Shulin Cao, Lei Hou, Yuxiao Dong, Ling Feng, and Juanzi Li. 2025a. LongCite: Enabling LLMs to generate fine-grained citations in long-context QA. In Findings of the Association for Computational Linguistics: ACL 2025, pages 5098–51… [find](https://scholar.google.com/scholar?q=Jiajie+Zhang%2C+Yushi+Bai%2C+Xin+Lv%2C+Wanjun+Gu%2C+Danqing+Liu%2C+Minhao+Zou%2C+Shulin+Cao%2C+Lei+Hou%2C+Yuxiao+Dong%2C+Ling+Feng%2C+and+Juanzi+Li%2E+2025a%2E+LongCite%3A+Enabling+LLMs+to+generate+fine%2Dgrained+citations+in+lo)

*Updated 2 November 2026*