---
title: "A Survey of Retentive Network — paper by Yang et al. (2026)"
description: "The authors say no systematic review of the growing RetNet literature existed, so they present what they call the first survey… 107 references."
authors: ["Haiqi Yang", "Zhiyuan Li", "Yi Chang", "Yuan Wu"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.256/"
doi: "10.18653/v1/2026.findings-acl.256"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yang-2026-survey-retentive-network/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yang-2026-survey-retentive-network/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/20/) / A Survey of Retentive Network

# A Survey of Retentive Network

**Authors:** Haiqi Yang, [Zhiyuan Li](https://md.research.elamur.ai/free/en/authors/zhiyuan-li/), [Yi Chang](https://md.research.elamur.ai/free/en/authors/yi-chang/) and [Yuan Wu](https://md.research.elamur.ai/free/en/authors/yuan-wu/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.256/) · [PDF](https://aclanthology.org/2026.findings-acl.256.pdf) · [DOI 10.18653/v1/2026.findings-acl.256](https://doi.org/10.18653/v1/2026.findings-acl.256) · [OpenAlex](https://openalex.org/W4417119786)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Transformers scale poorly on long inputs: self-attention costs grow quadratically with sequence length during training, and the per-token KV cache inflates memory use at inference. RetNet (Sun et al., 2023) addresses this with a retention mechanism that merges recurrent-style inductive bias with attention-like parallel training. The authors say no systematic review of the growing RetNet literature existed, so they present what they call the first survey of it.

The paper has no new dataset or experiments. Instead, it does three things:
- **Explains the architecture.** It covers the parallel, recurrent and chunkwise recurrent forms of retention, gated Multi-Scale Retention, and the overall block design.
- **Organizes the applications.** It groups RetNet-based models into NLP, computer vision, audio, natural science, and spatio-temporal/environmental modeling.
- **Discusses open problems.** These include weak associative recall caused by lossy state compression, unclear scaling behavior and training stability, hybrid designs, and robustness of the recurrent state.

The resulting map is useful for anyone weighing efficient Transformer alternatives for long-sequence or resource-constrained settings.

## Key points
- **Block structure:** RetNet stacks L identical blocks, each pairing Multi-Scale Retention (MSR) with an FFN. Each head gets a fixed decay γ = 1 − 2^(−5−arange(0,h)), followed by a swish gate and per-head GroupNorm.
- **Complexity:** The chunkwise form makes training linear in sequence length, O(N) for a fixed chunk size B, while recurrent inference runs in constant time, O(1) per step.
- **Surveyed models:** Examples include YOCO, LION and STSR (NLP); RMT, ViR, RetViT and SegRet (CV); RetNet-EEND, LS-EEND and LRetUNet (audio); CellFM, GR-pKa and DeepHapNet (science); and ST-RetNet, H-RetNet and FisH (spatio-temporal).
- **Proposed fixes for recall loss:** The authors point to adaptive or input-gated decay (citing Mamba, GLA and YOCO as examples) and to hybrids such as KARMA, which combines retention with KANs.
- **Publication:** Findings of ACL 2026, pp. 5200–5216, by authors from Jilin University.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Jilin University](https://md.research.elamur.ai/free/en/universities/jilin-university/) — University, Changchun, China · 30 works · 5,252 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

## Cite

Cite as: Haiqi Yang, Zhiyuan Li, Yi Chang and Yuan Wu. 2026. A Survey of Retentive Network. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yang2026survey,
  title = {{A Survey of Retentive Network}},
  author = {Haiqi Yang, Zhiyuan Li, Yi Chang and Yuan Wu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.256},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.256}
}
```

- [← previous A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutions](https://md.research.elamur.ai/free/en/papers/yu-2026-survey-reinforcement-learning-large-language/)
- [next → A Survey of the Digitisation of German Newspapers in interwar Lithuania (1918–1940)](https://md.research.elamur.ai/free/en/papers/plausinaityte-2026-survey-digitisation-german-newspapers-interwar/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/20/)

## References

107 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Arshia Afzal, Elias Abad Rocamora, Leyla Naz Can-dogan, Pol Puigdemont, Francesco Tonin, Yongtao Wu, Mahsa Shoaran, and Volkan Cevher. Lion: A bidirectional framework that trains like a transformer and infers like an rnn. [find](https://scholar.google.com/scholar?q=Arshia+Afzal%2C+Elias+Abad+Rocamora%2C+Leyla+Naz+Can%2Ddogan%2C+Pol+Puigdemont%2C+Francesco+Tonin%2C+Yongtao+Wu%2C+Mahsa+Shoaran%2C+and+Volkan+Cevher%2E+Lion%3A+A+bidirectional+framework+that+trains+like+a+transformer+an)
2. Arshia Afzal, Elias Abad Rocamora, Leyla Naz Cando-gan, Pol Puigdemont, Francesco Tonin, Yongtao Wu, Mahsa Shoaran, and Volkan Cevher. 2025. Linear attention for efficient bidirectional sequence modeling. arXiv preprint arXiv:2502.16249. — [arXiv:2502.16249](https://arxiv.org/abs/2502.16249) [find](https://scholar.google.com/scholar?q=Arshia+Afzal%2C+Elias+Abad+Rocamora%2C+Leyla+Naz+Cando%2Dgan%2C+Pol+Puigdemont%2C+Francesco+Tonin%2C+Yongtao+Wu%2C+Mahsa+Shoaran%2C+and+Volkan+Cevher%2E+2025%2E+Linear+attention+for+efficient+bidirectional+sequence+model)
3. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. [Layer normalization](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/). arXiv preprint arXiv:1607.06450.
4. Yoshua Bengio, Patrice Simard, and Paolo Frasconi. 1994. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157–166. [find](https://scholar.google.com/scholar?q=Yoshua+Bengio%2C+Patrice+Simard%2C+and+Paolo+Frasconi%2E+1994%2E+Learning+long%2Dterm+dependencies+with+gradient+descent+is+difficult%2E+IEEE+transactions+on+neural+networks%2C+5%282%29%3A157%E2%80%93166%2E)
5. Yueyang Cang, Pingge Hu, Xiaoteng Zhang, Xingtong Wang, Yuhang Liu, and Li Shi. 2024. Retcompletion: High-speed inference image completion with retentive network. arXiv preprint arXiv:2410.04056. — [arXiv:2410.04056](https://arxiv.org/abs/2410.04056) [find](https://scholar.google.com/scholar?q=Yueyang+Cang%2C+Pingge+Hu%2C+Xiaoteng+Zhang%2C+Xingtong+Wang%2C+Yuhang+Liu%2C+and+Li+Shi%2E+2024%2E+Retcompletion%3A+High%2Dspeed+inference+image+completion+with+retentive+network%2E+arXiv+preprint+arXiv%3A2410%2E04056%2E+%E2%80%94+%5Ba)
6. Qian Chang, Xia Li, and Xiufeng Cheng. 2024. Graph retention networks for dynamic graphs. arXiv preprint arXiv:2411.11259. [find](https://scholar.google.com/scholar?q=Qian+Chang%2C+Xia+Li%2C+and+Xiufeng+Cheng%2E+2024%2E+Graph+retention+networks+for+dynamic+graphs%2E+arXiv+preprint+arXiv%3A2411%2E11259%2E)
7. Jun Cheng, Tao Meng, Xiao Ao, and Xiaohua Wu. 2024. Pre-training retnet of simulating entities and relations as sentences for knowledge graph reasoning. In 2024 4th Asia Conference on Information Engineering (ACIE), pages 6–10. IEEE. [find](https://scholar.google.com/scholar?q=Jun+Cheng%2C+Tao+Meng%2C+Xiao+Ao%2C+and+Xiaohua+Wu%2E+2024%2E+Pre%2Dtraining+retnet+of+simulating+entities+and+relations+as+sentences+for+knowledge+graph+reasoning%2E+In+2024+4th+Asia+Conference+on+Information+Engi)
8. Lei Cheng and Siyang Cao. 2025. Transrad: Retentive vision transformer for enhanced radar object detection. IEEE Transactions on Radar Systems. [find](https://scholar.google.com/scholar?q=Lei+Cheng+and+Siyang+Cao%2E+2025%2E+Transrad%3A+Retentive+vision+transformer+for+enhanced+radar+object+detection%2E+IEEE+Transactions+on+Radar+Systems%2E)
9. Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, and 1 others. 2020. [Rethinking attention with performers](https://md.research.elamur.ai/free/en/papers/choromanski-2020-rethinking-attention-performers/). arXiv preprint arXiv:2009.14794.
10. Jinghui Chu, Wangtao Liu, Qi Tian, and Wei Lu. 2024. Pfprnet: A phase-wise feature pyramid with retention network for polyp segmentation. IEEE Journal of Biomedical and Health Informatics. [find](https://scholar.google.com/scholar?q=Jinghui+Chu%2C+Wangtao+Liu%2C+Qi+Tian%2C+and+Wei+Lu%2E+2024%2E+Pfprnet%3A+A+phase%2Dwise+feature+pyramid+with+retention+network+for+polyp+segmentation%2E+IEEE+Journal+of+Biomedical+and+Health+Informatics%2E)
11. Lior Cohen, Kaixin Wang, Bingyi Kang, and Shie Mannor. 2024. Improving token-based world models with parallel observation prediction. arXiv preprint arXiv:2402.05643. [find](https://scholar.google.com/scholar?q=Lior+Cohen%2C+Kaixin+Wang%2C+Bingyi+Kang%2C+and+Shie+Mannor%2E+2024%2E+Improving+token%2Dbased+world+models+with+parallel+observation+prediction%2E+arXiv+preprint+arXiv%3A2402%2E05643%2E)
12. Soham De, Samuel L Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, and 1 others. 2024. Griffin: Mixing gated linear recurrences with local attention for efficient language models. arXiv preprint arXiv:2402… [find](https://scholar.google.com/scholar?q=Soham+De%2C+Samuel+L+Smith%2C+Anushan+Fernando%2C+Aleksandar+Botev%2C+George+Cristian%2DMuraru%2C+Albert+Gu%2C+Ruba+Haroun%2C+Leonard+Berrada%2C+Yutian+Chen%2C+Srivatsan+Srinivasan%2C+and+1+others%2E+2024%2E+Griffin%3A+Mixing+ga)
13. JE Domínguez-Vidal and Alberto Sanfeliu. 2024. Force and velocity prediction in human-robot collaborative transportation tasks through video retentive networks. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9307–9313. IEEE. [find](https://scholar.google.com/scholar?q=JE+Dom%C3%ADnguez%2DVidal+and+Alberto+Sanfeliu%2E+2024%2E+Force+and+velocity+prediction+in+human%2Drobot+collaborative+transportation+tasks+through+video+retentive+networks%2E+In+2024+IEEE%2FRSJ+International+Conferen)
14. Shreyas Dongre and Shrushti Mehta. 2024. Retvit: Retentive vision transformers. In 2024 15th International Conference on Computing Communication and Networking Technologies (ICCCNT), pages 1–8. IEEE. [find](https://scholar.google.com/scholar?q=Shreyas+Dongre+and+Shrushti+Mehta%2E+2024%2E+Retvit%3A+Retentive+vision+transformers%2E+In+2024+15th+International+Conference+on+Computing+Communication+and+Networking+Technologies+%28ICCCNT%29%2C+pages+1%E2%80%938%2E+IEEE%2E)
15. Khaled ELKarazle, Valliappan Raman, Caslon Chua, and Patrick Then. 2023. Retseg: Retention-based colorectal polyps segmentation network. arXiv preprint arXiv:2310.05446. — [arXiv:2310.05446](https://arxiv.org/abs/2310.05446) [find](https://scholar.google.com/scholar?q=Khaled+ELKarazle%2C+Valliappan+Raman%2C+Caslon+Chua%2C+and+Patrick+Then%2E+2023%2E+Retseg%3A+Retention%2Dbased+colorectal+polyps+segmentation+network%2E+arXiv+preprint+arXiv%3A2310%2E05446%2E+%E2%80%94+%5BarXiv%3A2310%2E05446%5D%28https%3A%2F%2Fa)
16. Gopi Krishna Erabati and Helder Araujo. 2024. Retformer: Embracing point cloud transformer with retentive network. IEEE Transactions on Intelligent Vehicles. [find](https://scholar.google.com/scholar?q=Gopi+Krishna+Erabati+and+Helder+Araujo%2E+2024%2E+Retformer%3A+Embracing+point+cloud+transformer+with+retentive+network%2E+IEEE+Transactions+on+Intelligent+Vehicles%2E)
17. Gopi Krishna Erabati and Helder Araujo. 2025. Retseg3d: Retention-based 3d semantic segmentation for autonomous driving. Computer Vision and Image Understanding, 250:104231. [find](https://scholar.google.com/scholar?q=Gopi+Krishna+Erabati+and+Helder+Araujo%2E+2025%2E+Retseg3d%3A+Retention%2Dbased+3d+semantic+segmentation+for+autonomous+driving%2E+Computer+Vision+and+Image+Understanding%2C+250%3A104231%2E)
18. Lily Erickson. 2023. Cross-axis transformer with 3d rotary positional embeddings. Preprint, arXiv:2311.07184. — [arXiv:2311.07184](https://arxiv.org/abs/2311.07184) [find](https://scholar.google.com/scholar?q=Lily+Erickson%2E+2023%2E+Cross%2Daxis+transformer+with+3d+rotary+positional+embeddings%2E+Preprint%2C+arXiv%3A2311%2E07184%2E+%E2%80%94+%5BarXiv%3A2311%2E07184%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2311%2E07184%29)
19. Dengxin Fan and Songyan Liu. 2024. Monoretnet: A self-supervised model for monocular depth estimation with bidirectional half-duplex retention. In International Conference on Intelligent Computing, pages 361–372. Springer. [find](https://scholar.google.com/scholar?q=Dengxin+Fan+and+Songyan+Liu%2E+2024%2E+Monoretnet%3A+A+self%2Dsupervised+model+for+monocular+depth+estimation+with+bidirectional+half%2Dduplex+retention%2E+In+International+Conference+on+Intelligent+Computing%2C+pa)
20. Qihang Fan, Huaibo Huang, Mingrui Chen, Hongmin Liu, and Ran He. 2024. Rmt: Retentive networks meet vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5641–5651. [find](https://scholar.google.com/scholar?q=Qihang+Fan%2C+Huaibo+Huang%2C+Mingrui+Chen%2C+Hongmin+Liu%2C+and+Ran+He%2E+2024%2E+Rmt%3A+Retentive+networks+meet+vision+transformers%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+computer+vision+and+pattern+recogn)
21. Meng Feiyu. 2024. Cfpsg: Collaborative filtering poi similarity graph enhanced retentive network for next poi recommendation. In 2024 21st International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP), pages 1–4. IEEE. [find](https://scholar.google.com/scholar?q=Meng+Feiyu%2E+2024%2E+Cfpsg%3A+Collaborative+filtering+poi+similarity+graph+enhanced+retentive+network+for+next+poi+recommendation%2E+In+2024+21st+International+Computer+Conference+on+Wavelet+Active+Media+Tec)
22. Albert Gu and Tri Dao. 2023. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752. [find](https://scholar.google.com/scholar?q=Albert+Gu+and+Tri+Dao%2E+2023%2E+Mamba%3A+Linear%2Dtime+sequence+modeling+with+selective+state+spaces%2E+arXiv+preprint+arXiv%3A2312%2E00752%2E)
23. Albert Gu, Karan Goel, and Christopher Ré. 2021. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396. [find](https://scholar.google.com/scholar?q=Albert+Gu%2C+Karan+Goel%2C+and+Christopher+R%C3%A9%2E+2021%2E+Efficiently+modeling+long+sequences+with+structured+state+spaces%2E+arXiv+preprint+arXiv%3A2111%2E00396%2E)
24. Meisheng Guan, Haiyong Xu, Yeyao Chen, Ting Luo, Yang Song, and Huaping Wang. 2025. Retuie: Retention-based underwater image enhancement. In 2025 International Symposium on Machine Learning and Media Computing (MLMC), pages 1–10. IEEE. [find](https://scholar.google.com/scholar?q=Meisheng+Guan%2C+Haiyong+Xu%2C+Yeyao+Chen%2C+Ting+Luo%2C+Yang+Song%2C+and+Huaping+Wang%2E+2025%2E+Retuie%3A+Retention%2Dbased+underwater+image+enhancement%2E+In+2025+International+Symposium+on+Machine+Learning+and+Media+)
25. Chen Guo, Xinran Li, Jiaman Ma, Yimeng Li, Yuefan Liu, Haiying Qi, Li Zhang, and Yuhan Jin. 2024. Vl-mfer: A vision-language multimodal pretrained model with multiway-fuzzy-experts bidirectional retention network. IEEE Transactions on Fuzzy Systems. [find](https://scholar.google.com/scholar?q=Chen+Guo%2C+Xinran+Li%2C+Jiaman+Ma%2C+Yimeng+Li%2C+Yuefan+Liu%2C+Haiying+Qi%2C+Li+Zhang%2C+and+Yuhan+Jin%2E+2024%2E+Vl%2Dmfer%3A+A+vision%2Dlanguage+multimodal+pretrained+model+with+multiway%2Dfuzzy%2Dexperts+bidirectional+reten)
26. Tianjun Guo, Weixin Zhao, and Jian Peng. 2026. Ret-unet: Enhancing medical image segmentation with self-retention. Array, 29:100653. [find](https://scholar.google.com/scholar?q=Tianjun+Guo%2C+Weixin+Zhao%2C+and+Jian+Peng%2E+2026%2E+Ret%2Dunet%3A+Enhancing+medical+image+segmentation+with+self%2Dretention%2E+Array%2C+29%3A100653%2E)
27. Ayse Asu Guvenli and Bora Isildak. 2024. B-jet tagging with retentive networks: A novel approach and comparative study. arXiv preprint arXiv:2412.08134. — [arXiv:2412.08134](https://arxiv.org/abs/2412.08134) [find](https://scholar.google.com/scholar?q=Ayse+Asu+Guvenli+and+Bora+Isildak%2E+2024%2E+B%2Djet+tagging+with+retentive+networks%3A+A+novel+approach+and+comparative+study%2E+arXiv+preprint+arXiv%3A2412%2E08134%2E+%E2%80%94+%5BarXiv%3A2412%2E08134%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2412)
28. Jia Han, Zhiyong Yu, and Jian Yang. 2025. Radio frequency-retentive network for automatic modulation classification. Electronics Letters, 61(1):e70203. [find](https://scholar.google.com/scholar?q=Jia+Han%2C+Zhiyong+Yu%2C+and+Jian+Yang%2E+2025%2E+Radio+frequency%2Dretentive+network+for+automatic+modulation+classification%2E+Electronics+Letters%2C+61%281%29%3Ae70203%2E)
29. Zhu Han, Shuyi Xu, Lianru Gao, Zhi Li, and Bing Zhang. 2024. Gretnet: Gaussian retentive network for hyperspectral image classification. IEEE Geoscience and Remote Sensing Letters. [find](https://scholar.google.com/scholar?q=Zhu+Han%2C+Shuyi+Xu%2C+Lianru+Gao%2C+Zhi+Li%2C+and+Bing+Zhang%2E+2024%2E+Gretnet%3A+Gaussian+retentive+network+for+hyperspectral+image+classification%2E+IEEE+Geoscience+and+Remote+Sensing+Letters%2E)
30. Ali Hatamizadeh, Michael Ranzinger, and Jan Kautz. 2023a. Vir: Vision retention networks. arXiv. org. [find](https://scholar.google.com/scholar?q=Ali+Hatamizadeh%2C+Michael+Ranzinger%2C+and+Jan+Kautz%2E+2023a%2E+Vir%3A+Vision+retention+networks%2E+arXiv%2E+org%2E)
31. Ali Hatamizadeh, Michael Ranzinger, Shiyi Lan, Jose M Alvarez, Sanja Fidler, and Jan Kautz. 2023b. Vir: Towards efficient vision retention backbones. arXiv preprint arXiv:2310.19731. — [arXiv:2310.19731](https://arxiv.org/abs/2310.19731) [find](https://scholar.google.com/scholar?q=Ali+Hatamizadeh%2C+Michael+Ranzinger%2C+Shiyi+Lan%2C+Jose+M+Alvarez%2C+Sanja+Fidler%2C+and+Jan+Kautz%2E+2023b%2E+Vir%3A+Towards+efficient+vision+retention+backbones%2E+arXiv+preprint+arXiv%3A2310%2E19731%2E+%E2%80%94+%5BarXiv%3A2310%2E197)
32. Wei He, Kai Han, Yehui Tang, Chengcheng Wang, Yujie Yang, Tianyu Guo, and Yunhe Wang. 2024. Densemamba: State space models with dense hidden connection for efficient large language models. arXiv preprint arXiv:2403.00818. — [arXiv:2403.00818](https://arxiv.org/abs/2403.00818) [find](https://scholar.google.com/scholar?q=Wei+He%2C+Kai+Han%2C+Yehui+Tang%2C+Chengcheng+Wang%2C+Yujie+Yang%2C+Tianyu+Guo%2C+and+Yunhe+Wang%2E+2024%2E+Densemamba%3A+State+space+models+with+dense+hidden+connection+for+efficient+large+language+models%2E+arXiv+prepr)
33. Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks+and+Kevin+Gimpel%2E+2016%2E+Gaussian+error+linear+units+%28gelus%29%2E+arXiv+preprint+arXiv%3A1606%2E08415%2E)
34. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735– 1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+J%C3%BCrgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A1735%E2%80%93+1780%2E)
35. Jia Cheng Hu, Roberto Cavicchioli, and Alessandro Capotondi. 2024a. Shifted window fourier transform and retention for image captioning. arXiv preprint arXiv:2408.13963. — [arXiv:2408.13963](https://arxiv.org/abs/2408.13963) [find](https://scholar.google.com/scholar?q=Jia+Cheng+Hu%2C+Roberto+Cavicchioli%2C+and+Alessandro+Capotondi%2E+2024a%2E+Shifted+window+fourier+transform+and+retention+for+image+captioning%2E+arXiv+preprint+arXiv%3A2408%2E13963%2E+%E2%80%94+%5BarXiv%3A2408%2E13963%5D%28https%3A%2F%2Fa)
36. Jingjing Hu, Dan Guo, Kun Li, Zhan Si, Xun Yang, and Meng Wang. 2024b. Maskable retentive network for video moment retrieval. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 1476–1485. [find](https://scholar.google.com/scholar?q=Jingjing+Hu%2C+Dan+Guo%2C+Kun+Li%2C+Zhan+Si%2C+Xun+Yang%2C+and+Meng+Wang%2E+2024b%2E+Maskable+retentive+network+for+video+moment+retrieval%2E+In+Proceedings+of+the+32nd+ACM+International+Conference+on+Multimedia%2C+pag)
37. Jianan Huang, Xuebing Liu, Qing Zhu, Yaonan Wang, Mingtao Feng, Jiaming Zhou, Zhen Zhou, Lin Chen, and Danwei Wang. 2025a. Rampgrasp: Retentive attention-based multiscale perception grasp detection network. IEEE Transactions on Circuits and Systems for Video Technology. [find](https://scholar.google.com/scholar?q=Jianan+Huang%2C+Xuebing+Liu%2C+Qing+Zhu%2C+Yaonan+Wang%2C+Mingtao+Feng%2C+Jiaming+Zhou%2C+Zhen+Zhou%2C+Lin+Chen%2C+and+Danwei+Wang%2E+2025a%2E+Rampgrasp%3A+Retentive+attention%2Dbased+multiscale+perception+grasp+detection+ne)
38. Jingjia Huang, Jingyan Tu, Ge Meng, Yingying Wang, Yuhang Dong, Xiaotong Tu, Xinghao Ding, and Yue Huang. 2024a. Efficient perceiving local details via adaptive spatial-frequency information integration for multi-focus image fusion. In Proceedings of the 32nd ACM International Conference on Multimed… [find](https://scholar.google.com/scholar?q=Jingjia+Huang%2C+Jingyan+Tu%2C+Ge+Meng%2C+Yingying+Wang%2C+Yuhang+Dong%2C+Xiaotong+Tu%2C+Xinghao+Ding%2C+and+Yue+Huang%2E+2024a%2E+Efficient+perceiving+local+details+via+adaptive+spatial%2Dfrequency+information+integrati)
39. Kai-Wei Huang and Chia-Ping Chen. 2024. Long audio file speaker diarization with feasible end-to-end models. In 2024 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pages 1–6. IEEE. [find](https://scholar.google.com/scholar?q=Kai%2DWei+Huang+and+Chia%2DPing+Chen%2E+2024%2E+Long+audio+file+speaker+diarization+with+feasible+end%2Dto%2Dend+models%2E+In+2024+Asia+Pacific+Signal+and+Information+Processing+Association+Annual+Summit+and+Confer)
40. Qihe Huang, Zhengyang Zhou, Kuo Yang, Gengyu Lin, Zhongchao Yi, and Yang Wang. 2024b. Leret: Language-empowered retentive network for time series forecasting. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24. [find](https://scholar.google.com/scholar?q=Qihe+Huang%2C+Zhengyang+Zhou%2C+Kuo+Yang%2C+Gengyu+Lin%2C+Zhongchao+Yi%2C+and+Yang+Wang%2E+2024b%2E+Leret%3A+Language%2Dempowered+retentive+network+for+time+series+forecasting%2E+In+Proceedings+of+the+Thirty%2DThird+Intern)
41. Yuyao Huang, Kai Chen, Wei Tian, and Lu Xiong. 2025b. Boost query-centric network efficiency for multi-agent motion forecasting. IEEE Robotics and Automation Letters. [find](https://scholar.google.com/scholar?q=Yuyao+Huang%2C+Kai+Chen%2C+Wei+Tian%2C+and+Lu+Xiong%2E+2025b%2E+Boost+query%2Dcentric+network+efficiency+for+multi%2Dagent+motion+forecasting%2E+IEEE+Robotics+and+Automation+Letters%2E)
42. Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. 2020. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pages 5156–5165. PMLR. [find](https://scholar.google.com/scholar?q=Angelos+Katharopoulos%2C+Apoorv+Vyas%2C+Nikolaos+Pappas%2C+and+Fran%C3%A7ois+Fleuret%2E+2020%2E+Transformers+are+rnns%3A+Fast+autoregressive+transformers+with+linear+attention%2E+In+International+conference+on+machine+l)
43. Sangwon Kim, In-su Jang, and Byoung Chul Ko. 2024. Domain-free fire detection using the spatial–temporal attention transform of the yolo backbone. Pattern Analysis and Applications, 27(2):45. [find](https://scholar.google.com/scholar?q=Sangwon+Kim%2C+In%2Dsu+Jang%2C+and+Byoung+Chul+Ko%2E+2024%2E+Domain%2Dfree+fire+detection+using+the+spatial%E2%80%93temporal+attention+transform+of+the+yolo+backbone%2E+Pattern+Analysis+and+Applications%2C+27%282%29%3A45%2E)
44. Bin Lei, Caiwen Ding, and 1 others. 2023. Flashvideo: A framework for swift inference in text-to-video generation. arXiv preprint arXiv:2401.00869. — [arXiv:2401.00869](https://arxiv.org/abs/2401.00869) [find](https://scholar.google.com/scholar?q=Bin+Lei%2C+Caiwen+Ding%2C+and+1+others%2E+2023%2E+Flashvideo%3A+A+framework+for+swift+inference+in+text%2Dto%2Dvideo+generation%2E+arXiv+preprint+arXiv%3A2401%2E00869%2E+%E2%80%94+%5BarXiv%3A2401%2E00869%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2401%2E0086)
45. Sihan Li and Juhua Huang. 2025. Resgdanet: An efficient residual group attention neural network for medical image classification. Applied Sciences, 15(5):2693. [find](https://scholar.google.com/scholar?q=Sihan+Li+and+Juhua+Huang%2E+2025%2E+Resgdanet%3A+An+efficient+residual+group+attention+neural+network+for+medical+image+classification%2E+Applied+Sciences%2C+15%285%29%3A2693%2E)
46. Xing Li and Yuequan Bao. Adaptive gated meta graph retention network: A model for urban traffic flow prediction. Available at SSRN 5170149. [find](https://scholar.google.com/scholar?q=Xing+Li+and+Yuequan+Bao%2E+Adaptive+gated+meta+graph+retention+network%3A+A+model+for+urban+traffic+flow+prediction%2E+Available+at+SSRN+5170149%2E)
47. Zhiyuan Li, Yi Chang, and Yuan Wu. 2025. Segret: An efficient design for semantic segmentation with retentive network. arXiv preprint arXiv:2502.14014. — [arXiv:2502.14014](https://arxiv.org/abs/2502.14014) [find](https://scholar.google.com/scholar?q=Zhiyuan+Li%2C+Yi+Chang%2C+and+Yuan+Wu%2E+2025%2E+Segret%3A+An+efficient+design+for+semantic+segmentation+with+retentive+network%2E+arXiv+preprint+arXiv%3A2502%2E14014%2E+%E2%80%94+%5BarXiv%3A2502%2E14014%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2502%2E)
48. Zhiyuan Li, Tingyu Xia, Yi Chang, and Yuan Wu. 2024. A survey of rwkv. arXiv preprint arXiv:2412.14847. — [arXiv:2412.14847](https://arxiv.org/abs/2412.14847) [find](https://scholar.google.com/scholar?q=Zhiyuan+Li%2C+Tingyu+Xia%2C+Yi+Chang%2C+and+Yuan+Wu%2E+2024%2E+A+survey+of+rwkv%2E+arXiv+preprint+arXiv%3A2412%2E14847%2E+%E2%80%94+%5BarXiv%3A2412%2E14847%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2412%2E14847%29)
49. Di Liang and Xiaofei Li. 2024. Ls-eend: Long-form streaming end-to-end neural diarization with online attractor extraction. arXiv preprint arXiv:2410.06670. — [arXiv:2410.06670](https://arxiv.org/abs/2410.06670) [find](https://scholar.google.com/scholar?q=Di+Liang+and+Xiaofei+Li%2E+2024%2E+Ls%2Deend%3A+Long%2Dform+streaming+end%2Dto%2Dend+neural+diarization+with+online+attractor+extraction%2E+arXiv+preprint+arXiv%3A2410%2E06670%2E+%E2%80%94+%5BarXiv%3A2410%2E06670%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F)
50. Ruixing Lin and Shunmei Piao. 2024. Change detection of building remote sensing images based on rmt-bit. In 2024 4th International Conference on Electronic Information Engineering and Computer Science (EIECS), pages 428–432. IEEE. [find](https://scholar.google.com/scholar?q=Ruixing+Lin+and+Shunmei+Piao%2E+2024%2E+Change+detection+of+building+remote+sensing+images+based+on+rmt%2Dbit%2E+In+2024+4th+International+Conference+on+Electronic+Information+Engineering+and+Computer+Science)
51. Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. 2022. A survey of transformers. AI open, 3:111– 132. [find](https://scholar.google.com/scholar?q=Tianyang+Lin%2C+Yuxin+Wang%2C+Xiangyang+Liu%2C+and+Xipeng+Qiu%2E+2022%2E+A+survey+of+transformers%2E+AI+open%2C+3%3A111%E2%80%93+132%2E)
52. Zhicheng Lin, Rongpu Cui, Limiao Ning, and Jian Peng. 2025a. Temporal features-fused vision retentive network for echocardiography image segmentation. Sensors, 25(6):1909. [find](https://scholar.google.com/scholar?q=Zhicheng+Lin%2C+Rongpu+Cui%2C+Limiao+Ning%2C+and+Jian+Peng%2E+2025a%2E+Temporal+features%2Dfused+vision+retentive+network+for+echocardiography+image+segmentation%2E+Sensors%2C+25%286%29%3A1909%2E)
53. Zhijie Lin, Zilong Zhu, Lingling Guo, Jingjing Chen, and Jiyi Wu. 2025b. Disease detection algorithm for tea health protection based on improved real-time detection transformer. Applied Sciences (2076-3417), 15(4). [find](https://scholar.google.com/scholar?q=Zhijie+Lin%2C+Zilong+Zhu%2C+Lingling+Guo%2C+Jingjing+Chen%2C+and+Jiyi+Wu%2E+2025b%2E+Disease+detection+algorithm+for+tea+health+protection+based+on+improved+real%2Dtime+detection+transformer%2E+Applied+Sciences+%282076)
54. Zhou Linhao, Zhong Shenghua, and Xiao Zhijiao. 2024. Discovering multi-relational integration for knowledge tracing with retentive networks. In Proceedings of the 2024 International Conference on Multimedia Retrieval, pages 960–968. [find](https://scholar.google.com/scholar?q=Zhou+Linhao%2C+Zhong+Shenghua%2C+and+Xiao+Zhijiao%2E+2024%2E+Discovering+multi%2Drelational+integration+for+knowledge+tracing+with+retentive+networks%2E+In+Proceedings+of+the+2024+International+Conference+on+Mult)
55. Li, and Shiliang Li. 2024. Gr-p k a: a message-passing neural network with retention mechanism for p k a prediction. Briefings in Bioinformatics, 25(5):bbae408. [find](https://scholar.google.com/scholar?q=Li%2C+and+Shiliang+Li%2E+2024%2E+Gr%2Dp+k+a%3A+a+message%2Dpassing+neural+network+with+retention+mechanism+for+p+k+a+prediction%2E+Briefings+in+Bioinformatics%2C+25%285%29%3Abbae408%2E)
56. Zhaoyi Min, Qianqian Xiao, Muhammad Abbas, and Duanjin Zhang. 2025. Retentive network-based time series anomaly detection in cyber-physical systems. Engineering Applications of Artificial Intelligence, 145:110215. [find](https://scholar.google.com/scholar?q=Zhaoyi+Min%2C+Qianqian+Xiao%2C+Muhammad+Abbas%2C+and+Duanjin+Zhang%2E+2025%2E+Retentive+network%2Dbased+time+series+anomaly+detection+in+cyber%2Dphysical+systems%2E+Engineering+Applications+of+Artificial+Intelligence)
57. SIMONE MOSCO. 2023. Exploiting retentive networks in 3d lidar semantic segmentation. [find](https://scholar.google.com/scholar?q=SIMONE+MOSCO%2E+2023%2E+Exploiting+retentive+networks+in+3d+lidar+semantic+segmentation%2E)
58. Cheng Nian, Weiyi Zhang, Fasih Ud Din Farrukh, Liting Niu, Dapeng Jiang, Fei Chen, and Chun Zhang. 2024. A 77.79 gops/w retentive network fpga inference accelerator with optimized workload. In IECON 2024-50th Annual Conference of the IEEE Industrial Electronics Society, pages 1–7. IEEE. [find](https://scholar.google.com/scholar?q=Cheng+Nian%2C+Weiyi+Zhang%2C+Fasih+Ud+Din+Farrukh%2C+Liting+Niu%2C+Dapeng+Jiang%2C+Fei+Chen%2C+and+Chun+Zhang%2E+2024%2E+A+77%2E79+gops%2Fw+retentive+network+fpga+inference+accelerator+with+optimized+workload%2E+In+IECON+2)
59. Jiayuan Liu, Bo Zhou, Xue Wan, Yan Pan, Zicong Li, and Yuanbin Shao. 2025. Mar-vo: A match-and-refine framework for uav’s monocular visual odometry in planetary environments. IEEE Transactions on Geoscience and Remote Sensing. [find](https://scholar.google.com/scholar?q=Jiayuan+Liu%2C+Bo+Zhou%2C+Xue+Wan%2C+Yan+Pan%2C+Zicong+Li%2C+and+Yuanbin+Shao%2E+2025%2E+Mar%2Dvo%3A+A+match%2Dand%2Drefine+framework+for+uav%E2%80%99s+monocular+visual+odometry+in+planetary+environments%2E+IEEE+Transactions+on+Geos)
60. Masashi Okada, Mayumi Komatsu, and Tadahiro Taniguchi. 2024. A contact model based on denoising diffusion to learn variable impedance control for contact-rich manipulation. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7286–7293. IEEE. [find](https://scholar.google.com/scholar?q=Masashi+Okada%2C+Mayumi+Komatsu%2C+and+Tadahiro+Taniguchi%2E+2024%2E+A+contact+model+based+on+denoising+diffusion+to+learn+variable+impedance+control+for+contact%2Drich+manipulation%2E+In+2024+IEEE%2FRSJ+Internatio)
61. Jing Liu, Donglin Jing, Yanyan Cao, Ying Wang, Chaoping Guo, Peijun Shi, and Haijing Zhang. 2024a. Lightweight progressive fusion calibration network for rotated object detection in remote sensing images. Electronics, 13(16):3172. [find](https://scholar.google.com/scholar?q=Jing+Liu%2C+Donglin+Jing%2C+Yanyan+Cao%2C+Ying+Wang%2C+Chaoping+Guo%2C+Peijun+Shi%2C+and+Haijing+Zhang%2E+2024a%2E+Lightweight+progressive+fusion+calibration+network+for+rotated+object+detection+in+remote+sensing+ima)
62. Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, and 1 others. 2023. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048. [find](https://scholar.google.com/scholar?q=Bo+Peng%2C+Eric+Alcaide%2C+Quentin+Anthony%2C+Alon+Albalak%2C+Samuel+Arcadinho%2C+Stella+Biderman%2C+Huanqi+Cao%2C+Xin+Cheng%2C+Michael+Chung%2C+Matteo+Grella%2C+and+1+others%2E+2023%2E+Rwkv%3A+Reinventing+rnns+for+the+transfo)
63. Zhe Liu, Jinghua Hou, Xinyu Wang, Xiaoqing Ye, Jingdong Wang, Hengshuang Zhao, and Xiang Bai. 2024b. Lion: Linear group rnn for 3d object detection in point clouds. Advances in Neural Information Processing Systems, 37:13601–13626. [find](https://scholar.google.com/scholar?q=Zhe+Liu%2C+Jinghua+Hou%2C+Xinyu+Wang%2C+Xiaoqing+Ye%2C+Jingdong+Wang%2C+Hengshuang+Zhao%2C+and+Xiang+Bai%2E+2024b%2E+Lion%3A+Linear+group+rnn+for+3d+object+detection+in+point+clouds%2E+Advances+in+Neural+Information+Proc)
64. Lihong Peng, Xin Liu, Min Chen, Wen Liao, Jiale Mao, and Liqian Zhou. 2024. Mgndti: A drug-target interaction prediction framework based on multimodal representation learning and the gating mechanism. Journal of Chemical Information and Modeling, 64(16):6684–6698. [find](https://scholar.google.com/scholar?q=Lihong+Peng%2C+Xin+Liu%2C+Min+Chen%2C+Wen+Liao%2C+Jiale+Mao%2C+and+Liqian+Zhou%2E+2024%2E+Mgndti%3A+A+drug%2Dtarget+interaction+prediction+framework+based+on+multimodal+representation+learning+and+the+gating+mechanism%2E)
65. Ziyu Liu, Yi Shen, Yunliang Jiang, Hancan Zhu, Hailong Hu, Yanlei Kang, Ming Chen, and Zhong Li. 2024c. Variation and evolution analysis of sars-cov-2 using self-game sequence optimization. Frontiers in Microbiology, 15:1485748. [find](https://scholar.google.com/scholar?q=Ziyu+Liu%2C+Yi+Shen%2C+Yunliang+Jiang%2C+Hancan+Zhu%2C+Hailong+Hu%2C+Yanlei+Kang%2C+Ming+Chen%2C+and+Zhong+Li%2E+2024c%2E+Variation+and+evolution+analysis+of+sars%2Dcov%2D2+using+self%2Dgame+sequence+optimization%2E+Frontiers+)
66. Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. 2023. Hyena hierarchy: Towards larger convolutional language models. Preprint, arXiv:2302.10866. [find](https://scholar.google.com/scholar?q=Michael+Poli%2C+Stefano+Massaroli%2C+Eric+Nguyen%2C+Daniel+Y%2E+Fu%2C+Tri+Dao%2C+Stephen+Baccus%2C+Yoshua+Bengio%2C+Stefano+Ermon%2C+and+Christopher+R%C3%A9%2E+2023%2E+Hyena+hierarchy%3A+Towards+larger+convolutional+language+mode)
67. Baichao Long, Wang Zhu, and Jianli Xiao. 2024. St-retnet: A long-term spatial-temporal traffic flow prediction method. In Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pages 3– 16. Springer. [find](https://scholar.google.com/scholar?q=Baichao+Long%2C+Wang+Zhu%2C+and+Jianli+Xiao%2E+2024%2E+St%2Dretnet%3A+A+long%2Dterm+spatial%2Dtemporal+traffic+flow+prediction+method%2E+In+Chinese+Conference+on+Pattern+Recognition+and+Computer+Vision+%28PRCV%29%2C+pages+3%E2%80%93)
68. Zhen Qin, Songlin Yang, and Yiran Zhong. 2023. Hierarchically gated recurrent neural network for sequence modeling. Advances in Neural Information Processing Systems, 36:33202–33221. [find](https://scholar.google.com/scholar?q=Zhen+Qin%2C+Songlin+Yang%2C+and+Yiran+Zhong%2E+2023%2E+Hierarchically+gated+recurrent+neural+network+for+sequence+modeling%2E+Advances+in+Neural+Information+Processing+Systems%2C+36%3A33202%E2%80%9333221%2E)
69. Junwei Luo, Jiaojiao Wang, Jingjing Wei, Chaokun Yan, and Huimin Luo. 2025. Deephapnet: a haplotype assembly method based on retnet and deep spectral clustering. Briefings in Bioinformatics, 26(1):bbae656. [find](https://scholar.google.com/scholar?q=Junwei+Luo%2C+Jiaojiao+Wang%2C+Jingjing+Wei%2C+Chaokun+Yan%2C+and+Huimin+Luo%2E+2025%2E+Deephapnet%3A+a+haplotype+assembly+method+based+on+retnet+and+deep+spectral+clustering%2E+Briefings+in+Bioinformatics%2C+26%281%29%3Abba)
70. Yufan Qiu, Yaping Liu, and Shuo Zhang. 2024. Rn-ete: A retentive network-based encryption traffic encoder. In Proceedings of the 2024 3rd International Conference on Cryptography, Network Security and Communication Technology, pages 214–219. [find](https://scholar.google.com/scholar?q=Yufan+Qiu%2C+Yaping+Liu%2C+and+Shuo+Zhang%2E+2024%2E+Rn%2Dete%3A+A+retentive+network%2Dbased+encryption+traffic+encoder%2E+In+Proceedings+of+the+2024+3rd+International+Conference+on+Cryptography%2C+Network+Security+and)
71. Omayma Mahjoub, Sasha Abramowitz, Ruan de Kock, Wiem Khlifi, Simon du Toit, Jemma Daniel, Louay Ben Nessir, Louise Beyers, Claude Formanek, Liam Clark, and Arnu Pretorius. 2025. Sable: a performant, efficient and scalable sequence model for marl. Preprint, arXiv:2410.01706. — [arXiv:2410.01706](https://arxiv.org/abs/2410.01706) [find](https://scholar.google.com/scholar?q=Omayma+Mahjoub%2C+Sasha+Abramowitz%2C+Ruan+de+Kock%2C+Wiem+Khlifi%2C+Simon+du+Toit%2C+Jemma+Daniel%2C+Louay+Ben+Nessir%2C+Louise+Beyers%2C+Claude+Formanek%2C+Liam+Clark%2C+and+Arnu+Pretorius%2E+2025%2E+Sable%3A+a+performant%2C+e)
72. Runyu Miao, Danlin Liu, Liyun Mao, Xingyu Chen, Leihao Zhang, Zhen Yuan, Shanshan Shi, Honglin Prajit Ramachandran, Barret Zoph, and Quoc V Le. 2017. Swish: a self-gated activation function. arXiv preprint arXiv:1710.05941, 7(1):5. [find](https://scholar.google.com/scholar?q=Runyu+Miao%2C+Danlin+Liu%2C+Liyun+Mao%2C+Xingyu+Chen%2C+Leihao+Zhang%2C+Zhen+Yuan%2C+Shanshan+Shi%2C+Honglin+Prajit+Ramachandran%2C+Barret+Zoph%2C+and+Quoc+V+Le%2E+2017%2E+Swish%3A+a+self%2Dgated+activation+function%2E+arXiv+pre)
73. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053. [find](https://scholar.google.com/scholar?q=Mohammad+Shoeybi%2C+Mostofa+Patwary%2C+Raul+Puri%2C+Patrick+LeGresley%2C+Jared+Casper%2C+and+Bryan+Catanzaro%2E+2019%2E+Megatron%2Dlm%3A+Training+multi%2Dbillion+parameter+language+models+using+model+parallelism%2E+arXiv+p)
74. Ruilin Su, Wanshan Zhang, and Dunhui Yu. 2024. Sequence-to-sequence multi-hop knowledge reasoning based on retentive network. In 2024 7th International Conference on Computer Information Science and Application Technology (CISAT), pages 360–366. IEEE. [find](https://scholar.google.com/scholar?q=Ruilin+Su%2C+Wanshan+Zhang%2C+and+Dunhui+Yu%2E+2024%2E+Sequence%2Dto%2Dsequence+multi%2Dhop+knowledge+reasoning+based+on+retentive+network%2E+In+2024+7th+International+Conference+on+Computer+Information+Science+and+A)
75. Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. 2023. [Retentive network: A successor to transformer for large language models](https://md.research.elamur.ai/free/en/papers/sun-2023-retentive-network-successor-transformer-large/). arXiv preprint arXiv:2307.08621.
76. Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. 2022. [A length-extrapolatable transformer](https://md.research.elamur.ai/free/en/papers/sun-2023-length-extrapolatable-transformer/). arXiv preprint arXiv:2212.10554.
77. Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei. 2024. You only cache once: Decoder-decoder architectures for language models. Advances in Neural Information Processing Systems, 37:7339– 7361. [find](https://scholar.google.com/scholar?q=Yutao+Sun%2C+Li+Dong%2C+Yi+Zhu%2C+Shaohan+Huang%2C+Wenhui+Wang%2C+Shuming+Ma%2C+Quanlu+Zhang%2C+Jianyong+Wang%2C+and+Furu+Wei%2E+2024%2E+You+only+cache+once%3A+Decoder%2Ddecoder+architectures+for+language+models%2E+Advances+in)
78. Paweł Tomiło. 2025. Retention mechanism based neural network model for measuring aircraft landing distance. In 2025 IEEE 12th International Workshop on Metrology for AeroSpace (MetroAeroSpace), pages 321–326. IEEE. [find](https://scholar.google.com/scholar?q=Pawe%C5%82+Tomi%C5%82o%2E+2025%2E+Retention+mechanism+based+neural+network+model+for+measuring+aircraft+landing+distance%2E+In+2025+IEEE+12th+International+Workshop+on+Metrology+for+AeroSpace+%28MetroAeroSpace%29%2C+pages+)
79. Paweł Tomiło, Jan Laskowski, and Agnieszka Laskowska. 2025. Artificial neural network model based on kolmogorov-arnold representation theorem and retention mechanism for real-time aircraft flight phases classification. Engineering Applications of Artificial Intelligence, 160:112004. [find](https://scholar.google.com/scholar?q=Pawe%C5%82+Tomi%C5%82o%2C+Jan+Laskowski%2C+and+Agnieszka+Laskowska%2E+2025%2E+Artificial+neural+network+model+based+on+kolmogorov%2Darnold+representation+theorem+and+retention+mechanism+for+real%2Dtime+aircraft+flight+phas)
80. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30.
81. Siyu Wang, Xiaocong Chen, and Lina Yao. 2024b. Retentive decision transformer with adaptive masking for reinforcement learning based recommendation systems. arXiv preprint arXiv:2403.17634. — [arXiv:2403.17634](https://arxiv.org/abs/2403.17634) [find](https://scholar.google.com/scholar?q=Siyu+Wang%2C+Xiaocong+Chen%2C+and+Lina+Yao%2E+2024b%2E+Retentive+decision+transformer+with+adaptive+masking+for+reinforcement+learning+based+recommendation+systems%2E+arXiv+preprint+arXiv%3A2403%2E17634%2E+%E2%80%94+%5BarXiv%3A2)
82. Zeyu Wang, Libo Zhao, Jizheng Zhang, Rui Song, Haiyu Song, Jiana Meng, and Shidong Wang. 2025c. Multi-text guidance is important: Multi-modality image fusion via large generative vision-language model. International Journal of Computer Vision, pages 1–23. [find](https://scholar.google.com/scholar?q=Zeyu+Wang%2C+Libo+Zhao%2C+Jizheng+Zhang%2C+Rui+Song%2C+Haiyu+Song%2C+Jiana+Meng%2C+and+Shidong+Wang%2E+2025c%2E+Multi%2Dtext+guidance+is+important%3A+Multi%2Dmodality+image+fusion+via+large+generative+vision%2Dlanguage+model)
83. Tengqing Wu. 2024. A diffusion data enhancement retentive model for sequential recommendation. In 2024 7th International Conference on Computer Information Science and Application Technology (CISAT), pages 114–118. IEEE. [find](https://scholar.google.com/scholar?q=Tengqing+Wu%2E+2024%2E+A+diffusion+data+enhancement+retentive+model+for+sequential+recommendation%2E+In+2024+7th+International+Conference+on+Computer+Information+Science+and+Application+Technology+%28CISAT%29%2C+)
84. Yuxin Wu and Kaiming He. 2018. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pages 3–19. [find](https://scholar.google.com/scholar?q=Yuxin+Wu+and+Kaiming+He%2E+2018%2E+Group+normalization%2E+In+Proceedings+of+the+European+conference+on+computer+vision+%28ECCV%29%2C+pages+3%E2%80%9319%2E)
85. Yunyang Xie, Kai Chen, Shenghui Li, Bingqian Li, and Ning Zhang. 2024a. Uarc: Unsupervised anomalous traffic detection with improved u-shaped autoencoder and retnet based multi-clustering. In International Conference on Information and Communications Security, pages 187–207. Springer. [find](https://scholar.google.com/scholar?q=Yunyang+Xie%2C+Kai+Chen%2C+Shenghui+Li%2C+Bingqian+Li%2C+and+Ning+Zhang%2E+2024a%2E+Uarc%3A+Unsupervised+anomalous+traffic+detection+with+improved+u%2Dshaped+autoencoder+and+retnet+based+multi%2Dclustering%2E+In+Internat)
86. Zexun Xie, Min Xu, Shudong Zhang, and Lijuan Zhou. 2024b. Rcat: Retentive clip adapter tuning for improved video recognition. Electronics, 13(5):965. [find](https://scholar.google.com/scholar?q=Zexun+Xie%2C+Min+Xu%2C+Shudong+Zhang%2C+and+Lijuan+Zhou%2E+2024b%2E+Rcat%3A+Retentive+clip+adapter+tuning+for+improved+video+recognition%2E+Electronics%2C+13%285%29%3A965%2E)
87. Yimo Yan, Songyi Cui, Jiahui Liu, Yaping Zhao, Bodong Zhou, and Yong-Hong Kuo. 2025. Multimodal fusion for large-scale traffic prediction with heterogeneous retentive networks. Information Fusion, 114:102695. [find](https://scholar.google.com/scholar?q=Yimo+Yan%2C+Songyi+Cui%2C+Jiahui+Liu%2C+Yaping+Zhao%2C+Bodong+Zhou%2C+and+Yong%2DHong+Kuo%2E+2025%2E+Multimodal+fusion+for+large%2Dscale+traffic+prediction+with+heterogeneous+retentive+networks%2E+Information+Fusion%2C+114)
88. Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. 2024a. Gated linear attention transformers with hardware-efficient training. Preprint, arXiv:2312.06635. [find](https://scholar.google.com/scholar?q=Songlin+Yang%2C+Bailin+Wang%2C+Yikang+Shen%2C+Rameswar+Panda%2C+and+Yoon+Kim%2E+2024a%2E+Gated+linear+attention+transformers+with+hardware%2Defficient+training%2E+Preprint%2C+arXiv%3A2312%2E06635%2E)
89. Baoquan Wang, Yan Zeng, and Dongming Feng. 2025a. Deep learning-based damage assessment of hinge joints for multi-girder bridges utilizing vehicle-induced bridge responses. Engineering Structures, 333:120148. [find](https://scholar.google.com/scholar?q=Baoquan+Wang%2C+Yan+Zeng%2C+and+Dongming+Feng%2E+2025a%2E+Deep+learning%2Dbased+damage+assessment+of+hinge+joints+for+multi%2Dgirder+bridges+utilizing+vehicle%2Dinduced+bridge+responses%2E+Engineering+Structures%2C+333)
90. Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. 2025. Parallelizing linear transformers with the delta rule over sequence length. Preprint, arXiv:2406.06484. [find](https://scholar.google.com/scholar?q=Songlin+Yang%2C+Bailin+Wang%2C+Yu+Zhang%2C+Yikang+Shen%2C+and+Yoon+Kim%2E+2025%2E+Parallelizing+linear+transformers+with+the+delta+rule+over+sequence+length%2E+Preprint%2C+arXiv%3A2406%2E06484%2E)
91. Bin Wang, Fei Deng, and Peifan Jiang. 2024a. Eegdir: Electroencephalogram denoising network for temporal information storage and global modeling through retentive network. Computers in Biology and Medicine, 177:108626. [find](https://scholar.google.com/scholar?q=Bin+Wang%2C+Fei+Deng%2C+and+Peifan+Jiang%2E+2024a%2E+Eegdir%3A+Electroencephalogram+denoising+network+for+temporal+information+storage+and+global+modeling+through+retentive+network%2E+Computers+in+Biology+and+Med)
92. Wenkui Yang, Zhida Zhang, Xiaoqiang Zhou, Junxian Duan, and Jie Cao. 2024b. Tt-df: A large-scale diffusion-based dataset and benchmark for human body forgery detection. In Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pages 429–443. Springer. [find](https://scholar.google.com/scholar?q=Wenkui+Yang%2C+Zhida+Zhang%2C+Xiaoqiang+Zhou%2C+Junxian+Duan%2C+and+Jie+Cao%2E+2024b%2E+Tt%2Ddf%3A+A+large%2Dscale+diffusion%2Dbased+dataset+and+benchmark+for+human+body+forgery+detection%2E+In+Chinese+Conference+on+Patter)
93. Junliang Wang, Wenlong Hang, Shuang Liang, Qiong Wang, Badong Chen, and Jing Qin. 2025b. Convolutional retentive network for eeg decoding. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. [find](https://scholar.google.com/scholar?q=Junliang+Wang%2C+Wenlong+Hang%2C+Shuang+Liang%2C+Qiong+Wang%2C+Badong+Chen%2C+and+Jing+Qin%2E+2025b%2E+Convolutional+retentive+network+for+eeg+decoding%2E+In+ICASSP+2025%2D2025+IEEE+International+Conference+on+Acoustic)
94. Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768. [find](https://scholar.google.com/scholar?q=Sinong+Wang%2C+Belinda+Z+Li%2C+Madian+Khabsa%2C+Han+Fang%2C+and+Hao+Ma%2E+2020%2E+Linformer%3A+Self%2Dattention+with+linear+complexity%2E+arXiv+preprint+arXiv%3A2006%2E04768%2E)
95. Xuan Yang, Yunxuan Dong, Lina Yang, and Thomas Wu. 2024c. Short-term photovoltaic forecasting model for qualifying uncertainty during hazy weather. arXiv preprint arXiv:2407.19663. — [arXiv:2407.19663](https://arxiv.org/abs/2407.19663) [find](https://scholar.google.com/scholar?q=Xuan+Yang%2C+Yunxuan+Dong%2C+Lina+Yang%2C+and+Thomas+Wu%2E+2024c%2E+Short%2Dterm+photovoltaic+forecasting+model+for+qualifying+uncertainty+during+hazy+weather%2E+arXiv+preprint+arXiv%3A2407%2E19663%2E+%E2%80%94+%5BarXiv%3A2407%2E19663)
96. Xuan Yang, Tao Peng, Haijia Bi, and Jiayu Han. 2024d. Span-level bidirectional retention scheme for aspect sentiment triplet extraction. Information Processing & Management, 61(5):103823. [find](https://scholar.google.com/scholar?q=Xuan+Yang%2C+Tao+Peng%2C+Haijia+Bi%2C+and+Jiayu+Han%2E+2024d%2E+Span%2Dlevel+bidirectional+retention+scheme+for+aspect+sentiment+triplet+extraction%2E+Information+Processing+%26+Management%2C+61%285%29%3A103823%2E)
97. Yuansong Zeng, Jiancong Xie, Zhuoyi Wei, Yun Su, Ningyuan Shangguan, Shuangyu Yang, Chengyang Zhang, Wenbing Li, Jinbo Zhang, Nan Fang, and 1 others. 2024. Cellfm: a large-scale foundation model pre-trained on transcriptomics of 100 million human cells. bioRxiv, pages 2024–06. [find](https://scholar.google.com/scholar?q=Yuansong+Zeng%2C+Jiancong+Xie%2C+Zhuoyi+Wei%2C+Yun+Su%2C+Ningyuan+Shangguan%2C+Shuangyu+Yang%2C+Chengyang+Zhang%2C+Wenbing+Li%2C+Jinbo+Zhang%2C+Nan+Fang%2C+and+1+others%2E+2024%2E+Cellfm%3A+a+large%2Dscale+foundation+model+pre%2Dt)
98. Fanlong Zhang, Huanming Chen, Quan Chen, and Jianqi Liu. 2025a. Cloud software code generation via knowledge graphs and multi-modal learning. [find](https://scholar.google.com/scholar?q=Fanlong+Zhang%2C+Huanming+Chen%2C+Quan+Chen%2C+and+Jianqi+Liu%2E+2025a%2E+Cloud+software+code+generation+via+knowledge+graphs+and+multi%2Dmodal+learning%2E)
99. Lingzhe Zhang, Zewen Xiao, and Huaiyuan Wang. 2024a. Transient stability assessment of power system based on time-adapative retnet. In 2024 3rd Asia Power and Electrical Technology Conference (APET), pages 391–396. IEEE. [find](https://scholar.google.com/scholar?q=Lingzhe+Zhang%2C+Zewen+Xiao%2C+and+Huaiyuan+Wang%2E+2024a%2E+Transient+stability+assessment+of+power+system+based+on+time%2Dadapative+retnet%2E+In+2024+3rd+Asia+Power+and+Electrical+Technology+Conference+%28APET%29%2C+)
100. Tianning Zhang, Feng Liu, Yuming Yuan, Rui Su, Wanli Ouyang, and Lei Bai. 2024b. Fast information streaming handler (fish): A unified seismic neural network for single station real-time earthquake early warning. arXiv preprint arXiv:2408.06629. — [arXiv:2408.06629](https://arxiv.org/abs/2408.06629) [find](https://scholar.google.com/scholar?q=Tianning+Zhang%2C+Feng+Liu%2C+Yuming+Yuan%2C+Rui+Su%2C+Wanli+Ouyang%2C+and+Lei+Bai%2E+2024b%2E+Fast+information+streaming+handler+%28fish%29%3A+A+unified+seismic+neural+network+for+single+station+real%2Dtime+earthquake+ear)
101. Yuxuan Zhang, Zipeng Zhang, Weiwei Guo, Wei Chen, Zhaohai Liu, and Houguang Liu. 2025b. Lretunet: A u-net-based retentive network for single-channel speech enhancement. Computer Speech & Language, page 101798. [find](https://scholar.google.com/scholar?q=Yuxuan+Zhang%2C+Zipeng+Zhang%2C+Weiwei+Guo%2C+Wei+Chen%2C+Zhaohai+Liu%2C+and+Houguang+Liu%2E+2025b%2E+Lretunet%3A+A+u%2Dnet%2Dbased+retentive+network+for+single%2Dchannel+speech+enhancement%2E+Computer+Speech+%26+Language%2C+pag)
102. Zipeng Zhang, Zhencai Zhu, Bin Meng, Zheng Yang, Mingke Wu, Xinyu Cheng, Binhong Li, and Houguang Liu. 2025c. Intelligent coal gangue identification: A novel amplitude frequency sensitive neural network. Expert Systems with Applications, 274:126880. [find](https://scholar.google.com/scholar?q=Zipeng+Zhang%2C+Zhencai+Zhu%2C+Bin+Meng%2C+Zheng+Yang%2C+Mingke+Wu%2C+Xinyu+Cheng%2C+Binhong+Li%2C+and+Houguang+Liu%2E+2025c%2E+Intelligent+coal+gangue+identification%3A+A+novel+amplitude+frequency+sensitive+neural+netwo)
103. Kaili Zheng, Feixiang Lu, Yihao Lv, Liangjun Zhang, Chenyi Guo, and Ji Wu. 2024. 3d human pose estimation via non-causal retentive networks. In European Conference on Computer Vision, pages 111– 128. Springer. [find](https://scholar.google.com/scholar?q=Kaili+Zheng%2C+Feixiang+Lu%2C+Yihao+Lv%2C+Liangjun+Zhang%2C+Chenyi+Guo%2C+and+Ji+Wu%2E+2024%2E+3d+human+pose+estimation+via+non%2Dcausal+retentive+networks%2E+In+European+Conference+on+Computer+Vision%2C+pages+111%E2%80%93+128%2E+)
104. Li Zhou, Dayang Wang, Yongshun Xu, Shuo Han, Bahareh Morovati, Shuyi Fan, and Hengyong Yu. 2024. Gradient guided co-retention feature pyramid network for ldct image denoising. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 153–163. Springer. [find](https://scholar.google.com/scholar?q=Li+Zhou%2C+Dayang+Wang%2C+Yongshun+Xu%2C+Shuo+Han%2C+Bahareh+Morovati%2C+Shuyi+Fan%2C+and+Hengyong+Yu%2E+2024%2E+Gradient+guided+co%2Dretention+feature+pyramid+network+for+ldct+image+denoising%2E+In+International+Confere)
105. Wang Zhu, Baichao Long, and Jianli Xiao. 2024. Spatial-temporal retentive heterogeneous graph convolutional network for traffic flow prediction. In 2024 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE. [find](https://scholar.google.com/scholar?q=Wang+Zhu%2C+Baichao+Long%2C+and+Jianli+Xiao%2E+2024%2E+Spatial%2Dtemporal+retentive+heterogeneous+graph+convolutional+network+for+traffic+flow+prediction%2E+In+2024+International+Joint+Conference+on+Neural+Networ)
106. Zijie Zhu, Feng Ding, Chenglong Chu, and Fangming Zhong. 2025. Retention enhanced cross-modal attention for multi-hop vqa. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. [find](https://scholar.google.com/scholar?q=Zijie+Zhu%2C+Feng+Ding%2C+Chenglong+Chu%2C+and+Fangming+Zhong%2E+2025%2E+Retention+enhanced+cross%2Dmodal+attention+for+multi%2Dhop+vqa%2E+In+ICASSP+2025%2D2025+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Si)
107. Zhongliang Zou, Shuzhen Yang, Mansheng Wang, and Bo Song. 2025. Multilevel retentive networks with edge enhanced attention for infrastructure crack detection. Engineering Structures, 343:121191. [find](https://scholar.google.com/scholar?q=Zhongliang+Zou%2C+Shuzhen+Yang%2C+Mansheng+Wang%2C+and+Bo+Song%2E+2025%2E+Multilevel+retentive+networks+with+edge+enhanced+attention+for+infrastructure+crack+detection%2E+Engineering+Structures%2C+343%3A121191%2E)

*Updated 2 November 2026*