---
title: "Latent-Condensed Transformer for Efficient Long Context Modeling"
description: "The authors introduce Latent-Condensed Attention (LCA), which compresses older context directly inside MLA's latent space. 41 references."
authors: ["Zeng You", "Yaofo Chen", "Qiuwu Chen", "Ying Sun", "Shuhai Zhang", "Yingjian Li", "Yaowei Wang", "Mingkui Tan"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.1176/"
doi: "10.18653/v1/2026.acl-long.1176"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/you-2026-latent-condensed-transformer-efficient-long/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/you-2026-latent-condensed-transformer-efficient-long/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/38/) / Latent-Condensed Transformer for Efficient Long Context Modeling

# Latent-Condensed Transformer for Efficient Long Context Modeling

**Authors:** Zeng You, Yaofo Chen, Qiuwu Chen, [Ying Sun](https://md.research.elamur.ai/free/en/authors/ying-sun/), Shuhai Zhang, Yingjian Li, Yaowei Wang and Mingkui Tan · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.1176/) · [PDF](https://aclanthology.org/2026.acl-long.1176.pdf) · [DOI 10.18653/v1/2026.acl-long.1176](https://doi.org/10.18653/v1/2026.acl-long.1176) · [OpenAlex](https://openalex.org/W7166877006)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Long-context LLMs face two separate costs. The KV cache grows with sequence length, and self-attention scales quadratically. Multi-head Latent Attention (MLA) shrinks the per-token cache by storing low-dimensional latents, but it still attends over every token. Sparse-attention techniques, in turn, need full-dimensional keys and values, so applying them to MLA discards its compression advantage.

The authors introduce Latent-Condensed Attention (LCA), which compresses older context directly inside MLA's latent space. The context is split into groups. Within each group, the semantic latents C KV are merged through pooling weighted by query relevance. The RoPE positional key K R is instead taken from the single most relevant token, so positional signals are not blended. A recent local window stays uncompressed.

LCA adds no parameters and needs only light fine-tuning. It comes with an approximation-error bound that does not depend on length, and it also transfers to GQA. At 128K tokens it delivers up to 2.5× faster prefilling and about 90% smaller KV cache while keeping scores competitive, which makes long-context deployment cheaper.

## Key points
- At 128K context, LCA reaches up to 2.5× prefill speedup and roughly 90% KV cache reduction with comparable task performance. It also extends beyond MLA to GQA.
- On MiniCPM3-4B, a second MLA-based model, LCA makes prefilling 2.2× faster and cuts the KV cache by 93% at 128K.
- Ablations on LongBench-E support the design choices:
  - Attention-weighted pooling for semantics combined with max selection for positional anchors scores best.
  - The default group size is g = 16 and the default local window is w = 1024.
  - Using 16 summary queries gives an average of 29.09, versus 29.26 with 32 queries and 28.43 with 8.
- A custom Triton kernel is needed to realize the speedups in practice, compared against a PyTorch implementation.
- The authors note several limitations:
  - Accuracy drops modestly on precise token-level retrieval under aggressive condensation, for example in multi-document QA.
  - Lower-precision formats such as int8 have not been explored.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Harbin Institute of Technology](https://md.research.elamur.ai/free/en/universities/harbin-institute-of-technology/) — University, Harbin, China · 185 works · 29,140 citations
- [Peng Cheng Laboratory](https://md.research.elamur.ai/free/en/labs/peng-cheng-laboratory/) — Research institute, Shenzhen, China · 47 works · 2,699 citations
- [South China University of Technology](https://md.research.elamur.ai/free/en/universities/south-china-university-of-technology/) — University, Guangzhou, China · 27 works · 1,554 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (1): [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 297 works

## Cite

Cite as: Zeng You, Yaofo Chen, Qiuwu Chen, Ying Sun, Shuhai Zhang, Yingjian Li, Yaowei Wang and Mingkui Tan. 2026. Latent-Condensed Transformer for Efficient Long Context Modeling. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{you2026latent,
  title = {{Latent-Condensed Transformer for Efficient Long Context Modeling}},
  author = {Zeng You, Yaofo Chen, Qiuwu Chen, Ying Sun, Shuhai Zhang, Yingjian Li, Yaowei Wang and Mingkui Tan},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1176},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1176}
}
```

- [← previous Latent Narratives at AR-MS NakbaNLP 2026: Reducing Character Errors in Arabic Manuscript Transcription: A CER Oriented System](https://md.research.elamur.ai/free/en/papers/shaza-2026-latent-narratives-ar-ms-nakbanlp/)
- [next → LatentGate: Low-Latency Semantic Routing via Frozen-Backbone Probing of Small Language Models](https://md.research.elamur.ai/free/en/papers/ratnakar-2026-latentgate-low-latency-semantic-routing/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/38/)

## References

41 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. 2023. [GQA: Training generalized multi-query transformer models from multi-head checkpoints](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895– 4…
2. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and 1 others. 2021. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732.
3. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2023. [Longbench: A bilingual, multitask benchmark for long context understanding](https://md.research.elamur.ai/free/en/papers/bai-2023-longbench-bilingual-multitask-benchmark/). arXiv preprint arXiv:2308.14508.
4. Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. [Longformer: The long-document transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/). arXiv preprint arXiv:2004.05150.
5. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
6. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Tri+Dao%2C+Daniel+Y%2E+Fu%2C+Stefano+Ermon%2C+Atri+Rudra%2C+and+Christopher+R%C3%A9%2E+2022%2E+Flashattention%3A+Fast+and+memory%2Defficient+exact+attention+with+io%2Dawareness%2E+In+Advances+in+Neural+Information+Processing+Sy)
7. DeepSeek-AI. 2025. [Deepseek-v3.2: Pushing the frontier of open large language models](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-v3-2-pushing/). Preprint, arXiv:2512.02556.
8. DeepSeek-AI, Aixin Liu, Bei Feng, and et.al. 2024. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. Preprint, arXiv:2405.04434. [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2C+Aixin+Liu%2C+Bei+Feng%2C+and+et%2Eal%2E+2024%2E+Deepseek%2Dv2%3A+A+strong%2C+economical%2C+and+efficient+mixture%2Dof%2Dexperts+language+model%2E+Preprint%2C+arXiv%3A2405%2E04434%2E)
9. Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Nanning Zheng, and Furu Wei. 2023. [Longnet: Scaling transformers to 1,000,000,000 tokens](https://md.research.elamur.ai/free/en/papers/ding-2023-longnet-scaling-transformers-1-000/). arXiv preprint arXiv:2307.02486.
10. Zhanzhou Feng and Shiliang Zhang. 2023. Efficient vision transformer via token merger. IEEE Transactions on Image Processing, 32:4156–4169. [find](https://scholar.google.com/scholar?q=Zhanzhou+Feng+and+Shiliang+Zhang%2E+2023%2E+Efficient+vision+transformer+via+token+merger%2E+IEEE+Transactions+on+Image+Processing%2C+32%3A4156%E2%80%934169%2E)
11. Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. 2024. [Data engineering for scaling language models to 128k context](https://md.research.elamur.ai/free/en/papers/fu-2024-data-engineering-scaling-language-models/). arXiv preprint arXiv:2402.10171.
12. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
13. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseekr1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint. [find](https://scholar.google.com/scholar?q=Daya+Guo%2C+Dejian+Yang%2C+Haowei+Zhang%2C+Junxiao+Song%2C+Ruoyu+Zhang%2C+Runxin+Xu%2C+Qihao+Zhu%2C+Shirong+Ma%2C+Peiyi+Wang%2C+Xiao+Bi%2C+and+1+others%2E+2025%2E+Deepseekr1%3A+Incentivizing+reasoning+capability+in+llms+via+re)
14. Jitai Hao, Yuke Zhu, Tian Wang, Jun Yu, Xin Xin, Bo Zheng, Zhaochun Ren, and Sheng Guo. 2025. Omnikv: Dynamic context selection for efficient long-context llms. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Jitai+Hao%2C+Yuke+Zhu%2C+Tian+Wang%2C+Jun+Yu%2C+Xin+Xin%2C+Bo+Zheng%2C+Zhaochun+Ren%2C+and+Sheng+Guo%2E+2025%2E+Omnikv%3A+Dynamic+context+selection+for+efficient+long%2Dcontext+llms%2E+In+International+Conference+on+Learning)
15. Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. 2023. Token merging: Your vit but faster. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Daniel+Bolya%2C+Cheng%2DYang+Fu%2C+Xiaoliang+Dai%2C+Peizhao+Zhang%2C+Christoph+Feichtenhofer%2C+and+Judy+Hoffman%2E+2023%2E+Token+merging%3A+Your+vit+but+faster%2E+In+International+Conference+on+Learning+Representations%2E)
16. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In International Conference on Learning Representations.
17. Yaofo Chen, Zeng You, Shuhai Zhang, Haokun Li, Yirui Li, Yaowei Wang, and Mingkui Tan. 2025. Core context aware transformers for long context language modeling. In International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Yaofo+Chen%2C+Zeng+You%2C+Shuhai+Zhang%2C+Haokun+Li%2C+Yirui+Li%2C+Yaowei+Wang%2C+and+Mingkui+Tan%2E+2025%2E+Core+context+aware+transformers+for+long+context+language+modeling%2E+In+International+Conference+on+Machine+)
18. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. [Ruler: What’s the real context size of your long-context language models?](https://md.research.elamur.ai/free/en/papers/hsieh-2024-ruler-s-real-context-size/) arXiv preprint arXiv:2404.06654.
19. Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, and 1 others. 2024. [Minicpm: Unveiling the potential of small language models with scalable training strategies](https://md.research.elamur.ai/free/en/papers/hu-2024-minicpm-unveiling-potential-small-language/). arXiv preprint arXiv:2404.06395.
20. Huiqiang Jiang, YUCHENG LI, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2024. [MInference 1.0: Accelerating pre-filling for long-context LLMs via dynamic sparse attention](https://md.research.elamur.ai/free/en/papers/jiang-2024-minference-1-0-accelerating-pre/). In Advances in Neural Information Pr…
21. Kimi, Yifan Bai, Yiping Bao, and et.al. 2025a. [Kimi k2: Open agentic intelligence](https://md.research.elamur.ai/free/en/papers/kimi-team-2025-kimi-k2-open-agentic/). arXiv.
22. Kimi, Yu Zhang, Zongyu Lin, and et.al. 2025b. [Kimi linear: An expressive, efficient attention architecture](https://md.research.elamur.ai/free/en/papers/kimi-team-2025-kimi-linear-expressive-efficient/). Preprint, arXiv:2510.26692.
23. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626.
24. Xunhao Lai, Jianqiao Lu, Yao Luo, Yiyuan Ma, and Xun Zhou. 2025. Flexprefill: A context-aware sparse attention mechanism for efficient long-sequence inference. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Xunhao+Lai%2C+Jianqiao+Lu%2C+Yao+Luo%2C+Yiyuan+Ma%2C+and+Xun+Zhou%2E+2025%2E+Flexprefill%3A+A+context%2Daware+sparse+attention+mechanism+for+efficient+long%2Dsequence+inference%2E+In+International+Conference+on+Learning+)
25. Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. [Snapkv: Llm knows what you are looking for before generation](https://md.research.elamur.ai/free/en/papers/li-2024-snapkv-llm-knows-looking-before/). Advances in Neural Information Processing Systems, 37:22947–22970.
26. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. [find](https://scholar.google.com/scholar?q=Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+Bingxuan+Wang%2C+Bochao+Wu%2C+Chengda+Lu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chenyu+Zhang%2C+Chong+Ruan%2C+and+1+others%2E+2024%2E+Deepseek%2Dv3+technical+report%2E+arXiv+preprint+arXiv%3A2412)
27. Xiang Liu, Zhenheng Tang, Peijie Dong, Zeyu Li, Yue Liu, Bo Li, Xuming Hu, and Xiaowen Chu. 2025. Chunkkv: Semantic-preserving kv cache compression for efficient long-context llm inference. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Xiang+Liu%2C+Zhenheng+Tang%2C+Peijie+Dong%2C+Zeyu+Li%2C+Yue+Liu%2C+Bo+Li%2C+Xuming+Hu%2C+and+Xiaowen+Chu%2E+2025%2E+Chunkkv%3A+Semantic%2Dpreserving+kv+cache+compression+for+efficient+long%2Dcontext+llm+inference%2E+In+Advance)
28. Potsawee Manakul and Mark J. F. Gales. 2021. Long-span summarization via local attention and content selection. In Association for Computational Linguistics, pages 6026–6041. [find](https://scholar.google.com/scholar?q=Potsawee+Manakul+and+Mark+J%2E+F%2E+Gales%2E+2021%2E+Long%2Dspan+summarization+via+local+attention+and+content+selection%2E+In+Association+for+Computational+Linguistics%2C+pages+6026%E2%80%936041%2E)
29. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). Neurocomputing, 568:127063.
30. Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. 2024. [QUEST: Query-aware sparsity for efficient long-context LLM inference](https://md.research.elamur.ai/free/en/papers/tang-2024-quest-query-aware-sparsity-efficient/). In International Conference on Machine Learning.
31. Philippe Tillet, Hsiang-Tsung Kung, and David D. Cox. 2019. [Triton: an intermediate language and compiler for tiled neural network computations](https://md.research.elamur.ai/free/en/papers/tillet-2019-triton-intermediate-language-compiler-tiled/). In Proceedings of the International Workshop on Machine Learning and Programming Languages, pages 10– 19.
32. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, pages 5998–6008.
33. Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Shang Yang, Haotian Tang, Yao Fu, Song Han, and 1 others. 2025. [Duoattention: Efficient long-context llm inference with retrieval and streaming heads](https://md.research.elamur.ai/free/en/papers/xiao-2024-duoattention-efficient-long-context-llm/). In International Conference on Learning Representations.
34. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient streaming language models with attention sinks. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Guangxuan+Xiao%2C+Yuandong+Tian%2C+Beidi+Chen%2C+Song+Han%2C+and+Mike+Lewis%2E+2024%2E+Efficient+streaming+language+models+with+attention+sinks%2E+In+International+Conference+on+Learning+Representations%2E)
35. Ruyi Xu, Guangxuan Xiao, Haofeng Huang, Junxian Guo, and Song Han. 2025. [Xattention: Block sparse attention with antidiagonal scoring](https://md.research.elamur.ai/free/en/papers/xu-2025-xattention-block-sparse-attention-antidiagonal/). In International Conference on Machine Learning.
36. Baosong Yang, Longyue Wang, Derek F Wong, Shuming Shi, and Zhaopeng Tu. 2021. Context-aware self-attention networks for natural language processing. Neurocomputing, 458:157–169. [find](https://scholar.google.com/scholar?q=Baosong+Yang%2C+Longyue+Wang%2C+Derek+F+Wong%2C+Shuming+Shi%2C+and+Zhaopeng+Tu%2E+2021%2E+Context%2Daware+self%2Dattention+networks+for+natural+language+processing%2E+Neurocomputing%2C+458%3A157%E2%80%93169%2E)
37. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and 1 others. 2020. [Big bird: Transformers for longer sequences](https://md.research.elamur.ai/free/en/papers/zaheer-2020-big-bird-transformers-longer-sequences/). Advances in Neural Information Processing Systems, 33:17283–17297.
38. Shuhai Zhang, Zeng You, Yaofo Chen, Zhiquan Wen, Qianyue Wang, Zhijie Qiu, Yuanqing Li, and Mingkui Tan. 2025. Curse of high dimensionality issue in transformer for long context modeling. In International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Shuhai+Zhang%2C+Zeng+You%2C+Yaofo+Chen%2C+Zhiquan+Wen%2C+Qianyue+Wang%2C+Zhijie+Qiu%2C+Yuanqing+Li%2C+and+Mingkui+Tan%2E+2025%2E+Curse+of+high+dimensionality+issue+in+transformer+for+long+context+modeling%2E+In+Internati)
39. OpenAI. 2023. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint.
40. Noam Shazeer. 2019. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2E+2019%2E+Fast+transformer+decoding%3A+One+write%2Dhead+is+all+you+need%2E+CoRR%2C+abs%2F1911%2E02150%2E)
41. In the supplementary, we provide a detailed proof, more details on the LCA method. We organize our supplementary as follows. In Section A, we provide the detailed proof for Proposition 1 and Theorem 1. In Section B, we analysis the computation and memory complexity of LCA in detail. In Section C, we… [find](https://scholar.google.com/scholar?q=In+the+supplementary%2C+we+provide+a+detailed+proof%2C+more+details+on+the+LCA+method%2E+We+organize+our+supplementary+as+follows%2E+In+Section+A%2C+we+provide+the+detailed+proof+for+Proposition+1+and+Theorem+1)

*Updated 2 November 2026*