---
title: "Vision Mamba — paper by Zhu et al. (2024) — summary and ideas"
description: "They propose Vision Mamba (Vim), a plain, non-hierarchical backbone built only from state space… 80 references, cited by 86 papers here."
authors: ["Lianghui Zhu", "Bencheng Liao", "Qian Zhang", "Xinlong Wang", "Wenyu Liu", "Xinggang Wang"]
year: 2024
venue: "arXiv:2401.09417"
source: "https://arxiv.org/abs/2401.09417"
doi: "10.48550/arxiv.2401.09417"
arxiv: "2401.09417"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhu-2024-vision-mamba-efficient-visual-representation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhu-2024-vision-mamba-efficient-visual-representation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/) / Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model

# Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model

**Authors:** [Lianghui Zhu](https://md.research.elamur.ai/free/en/authors/lianghui-zhu/), Bencheng Liao, [Qian Zhang](https://md.research.elamur.ai/free/en/authors/qian-zhang/), [Xinlong Wang](https://md.research.elamur.ai/free/en/authors/xinlong-wang/), [Wenyu Liu](https://md.research.elamur.ai/free/en/authors/wenyu-liu/) and [Xinggang Wang](https://md.research.elamur.ai/free/en/authors/xinggang-wang/) · **Year:** 2024 · **Venue:** arXiv:2401.09417 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2401.09417) · [PDF](https://arxiv.org/pdf/2401.09417) · [DOI 10.48550/arxiv.2401.09417](https://doi.org/10.48550/arxiv.2401.09417) · [arXiv 2401.09417](https://arxiv.org/abs/2401.09417) · [OpenAlex](https://openalex.org/W4391013663)

**Citations:** 414 citations (OpenAlex, 2026-10-14) · [86](https://md.research.elamur.ai/free/en/papers/zhu-2024-vision-mamba-efficient-visual-representation/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Vision Transformers give each patch global, input-dependent context, but self-attention becomes slow and memory-hungry on high-resolution images with many tokens. The authors ask whether attention is needed at all. They propose Vision Mamba (Vim), a plain, non-hierarchical backbone built only from state space model (SSM) blocks derived from Mamba. Vim splits an image into patches, projects them linearly, adds position embeddings, and runs the token sequence through blocks that scan both forward and backward. This addresses two weaknesses of Mamba for images: it reads in only one direction, and it does not know where each token sits. Vim is pretrained with supervision on ImageNet and then used as a backbone for COCO detection and instance segmentation and ADE20K semantic segmentation. It beats DeiT on these tasks while being much cheaper at high resolution. This matters because Vim keeps the sequence-style, modality-agnostic design of ViT without its quadratic cost, which makes it a candidate backbone for high-resolution and multimodal foundation models.

## Key points
- On batch feature extraction at 1248×1248 (6084 tokens per image), Vim runs 2.8× faster than DeiT and uses 86.8% less GPU memory.
- Vim's computation grows sub-quadratically and its memory grows linearly with sequence length. The authors present it as the first pure-SSM model to handle dense prediction.
- Bidirectional design ablation:
  - A vanilla unidirectional Mamba block scores 73.2 top-1 on ImageNet and 32.3 mIoU on ADE20K.
  - The default, which adds a backward SSM preceded by a backward Conv1d, reaches 73.9 top-1 and 35.9 mIoU.
- Classification head ablation on ImageNet-1K:
  - A class token in the middle of the sequence gives 76.1 top-1.
  - A head class token gives 75.2, a double token 74.3, mean pooling 73.9, and max pooling 73.4.
- Detection and instance segmentation follow the ViTDet framework on COCO 2017. Code and models are released at github.com/hustvl/Vim (ICML 2024).

## Interesting ideas
- **Add a reverse scan instead of only flipping data.** Randomly reversing sequences or alternating block directions was weaker than giving each block its own backward SSM branch. Any causal, recurrent model used for non-causal understanding tasks could benefit from this, for example long-text encoders.
- **Place the summary token where recurrence sees the most.** In a left-to-right recurrent model, a token at the start sees nothing. Putting the class token in the middle lets it gather context from both halves, and on ImageNet it also matches the tendency for objects to be centred. Where to insert special tokens is worth testing whenever a recurrent model must produce one pooled representation.
- **Ablate on both global and dense tasks.** The unidirectional block looked fine for classification but lagged clearly on segmentation, and one bidirectional variant showed the opposite pattern. Judging an architecture change by a single global metric can hide weaknesses that only appear in per-token tasks.
- **Benchmark efficiency as input length grows.** Reporting speed and memory across increasing resolutions, up to the point where the baseline runs out of memory, makes the real scaling advantage visible. This is a useful protocol for any comparison of long-sequence models.
- **Avoid domain-specific priors to stay modality-agnostic.** Vim deliberately skips 2D windows and pyramids and treats the image as a plain token sequence. This keeps the backbone compatible with text-style models and future multimodal pretraining, such as CLIP-style or masked modeling.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Huazhong University of Science and Technology](https://md.research.elamur.ai/free/en/universities/huazhong-university-of-science-and-technology/) — University, Wuhan, China · 59 works · 8,310 citations
- [Beijing Academy of Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/beijing-academy-of-artificial-intelligence/) — Research institute, Beijing, China · 46 works · 8,682 citations

Also on the byline: [Horizon Robotics](https://ror.org/05cmv6g68).

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

## Cite

Cite as: Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu and Xinggang Wang. 2024. Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model. arXiv:2401.09417.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhu2024vision,
  title = {{Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model}},
  author = {Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu and Xinggang Wang},
  year = {2024},
  doi = {10.48550/arxiv.2401.09417},
  eprint = {2401.09417},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2401.09417}
}
```

- [← previous Phi-4 Technical Report](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-4-technical-report/)
- [next → Solving olympiad geometry without human demonstrations](https://md.research.elamur.ai/free/en/papers/trinh-2024-solving-olympiad-geometry-without-human/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/)

## References

80 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bao, H., Dong, L., Piao, S., and Wei, F. Beit: BERT pre-training of image transformers. In ICLR, 2022. URL [https://openreview.net/forum?id=p-BhZSz59o4](https://openreview.net/forum?id=p-BhZSz59o4). [↗](https://openreview.net/forum?id=p-BhZSz59o4)
2. Baron, E., Zimerman, I., and Wolf, L. 2-d ssm: A general spatial layer for visual transformers. arXiv preprint arXiv:2306.06635, 2023. — [arXiv:2306.06635](https://arxiv.org/abs/2306.06635) [find](https://scholar.google.com/scholar?q=Baron%2C+E%2E%2C+Zimerman%2C+I%2E%2C+and+Wolf%2C+L%2E+2%2Dd+ssm%3A+A+general+spatial+layer+for+visual+transformers%2E+arXiv+preprint+arXiv%3A2306%2E06635%2C+2023%2E+%E2%80%94+%5BarXiv%3A2306%2E06635%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2306%2E06635%29)
3. Bavishi, R., Elsen, E., Hawthorne, C., Nye, M., Odena, A., Somani, A., and Taşırlar, S. Introducing our multimodal models, 2023. URL [https://www.adept.ai/blog/fuyu-8b](https://www.adept.ai/blog/fuyu-8b). [↗](https://www.adept.ai/blog/fuyu-8b)
4. Cai, Z. and Vasconcelos, N. Cascade r-cnn: High quality object detection and instance segmentation. TPAMI, 2019. [find](https://scholar.google.com/scholar?q=Cai%2C+Z%2E+and+Vasconcelos%2C+N%2E+Cascade+r%2Dcnn%3A+High+quality+object+detection+and+instance+segmentation%2E+TPAMI%2C+2019%2E)
5. Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. [Emerging properties in self-supervised vision transformers](https://md.research.elamur.ai/free/en/papers/caron-2021-emerging-properties-self-supervised-vision/). In ICCV, 2021.
6. Child, R., Gray, S., Radford, A., and Sutskever, I. [Generating long sequences with sparse transformers](https://md.research.elamur.ai/free/en/papers/child-2019-generating-long-sequences-sparse-transformers/). arXiv preprint arXiv:1904.10509, 2019.
7. Choromanski, K. M., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J. Q., Mohiuddin, A., Kaiser, L., Belanger, D. B., Colwell, L. J., and Weller, A. [Rethinking attention with performers](https://md.research.elamur.ai/free/en/papers/choromanski-2020-rethinking-attention-performers/). In ICLR, 2021. URL [https://openreview.net/forum?id=Ua6zuk0WRH](https://openreview.net/forum?id=Ua6zuk0WRH). [↗](https://openreview.net/forum?id=Ua6zuk0WRH)
8. Dai, Z., Liu, H., Le, Q. V., and Tan, M. Coatnet: Marrying convolution and attention for all data sizes. NeurIPS, 34, 2021. [find](https://scholar.google.com/scholar?q=Dai%2C+Z%2E%2C+Liu%2C+H%2E%2C+Le%2C+Q%2E+V%2E%2C+and+Tan%2C+M%2E+Coatnet%3A+Marrying+convolution+and+attention+for+all+data+sizes%2E+NeurIPS%2C+34%2C+2021%2E)
9. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. [find](https://scholar.google.com/scholar?q=Deng%2C+J%2E%2C+Dong%2C+W%2E%2C+Socher%2C+R%2E%2C+Li%2C+L%2E%2DJ%2E%2C+Li%2C+K%2E%2C+and+Fei%2DFei%2C+L%2E+Imagenet%3A+A+large%2Dscale+hierarchical+image+database%2E+In+CVPR%2C+2009%2E)
10. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805, 2018.
11. Ding, J., Ma, S., Dong, L., Zhang, X., Huang, S., Wang, W., Zheng, N., and Wei, F. [Longnet: Scaling transformers to 1,000,000,000 tokens](https://md.research.elamur.ai/free/en/papers/ding-2023-longnet-scaling-transformers-1-000/). arXiv preprint arXiv:2307.02486, 2023.
12. Ding, X., Zhang, X., Han, J., and Ding, G. Scaling up your kernels to 31x31: Revisiting large kernel design in cnns. In CVPR, 2022. [find](https://scholar.google.com/scholar?q=Ding%2C+X%2E%2C+Zhang%2C+X%2E%2C+Han%2C+J%2E%2C+and+Ding%2C+G%2E+Scaling+up+your+kernels+to+31x31%3A+Revisiting+large+kernel+design+in+cnns%2E+In+CVPR%2C+2022%2E)
13. Dong, X., Bao, J., Chen, D., Zhang, W., Yu, N., Yuan, L., Chen, D., and Guo, B. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In CVPR, 2022. [find](https://scholar.google.com/scholar?q=Dong%2C+X%2E%2C+Bao%2C+J%2E%2C+Chen%2C+D%2E%2C+Zhang%2C+W%2E%2C+Yu%2C+N%2E%2C+Yuan%2C+L%2E%2C+Chen%2C+D%2E%2C+and+Guo%2C+B%2E+Cswin+transformer%3A+A+general+vision+transformer+backbone+with+cross%2Dshaped+windows%2E+In+CVPR%2C+2022%2E)
14. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. [An image is worth 16x16 words: Transformers for image recognition at scale](https://md.research.elamur.ai/free/en/papers/dosovitskiy-2020-image-worth-16x16-words-transformers/). In ICLR, 2020. d’Ascoli, S., Touvron, H., Leavitt, M. L., Morcos, A. S., Biro…
15. Fang, J., Xie, L., Wang, X., Zhang, X., Liu, W., and Tian, Q. Msg-transformer: Exchanging local spatial information by manipulating messenger tokens. In CVPR, 2022. [find](https://scholar.google.com/scholar?q=Fang%2C+J%2E%2C+Xie%2C+L%2E%2C+Wang%2C+X%2E%2C+Zhang%2C+X%2E%2C+Liu%2C+W%2E%2C+and+Tian%2C+Q%2E+Msg%2Dtransformer%3A+Exchanging+local+spatial+information+by+manipulating+messenger+tokens%2E+In+CVPR%2C+2022%2E)
16. Fang, Y., Wang, W., Xie, B., Sun, Q., Wu, L., Wang, X., Huang, T., Wang, X., and Cao, Y. Eva: Exploring the limits of masked visual representation learning at scale. In CVPR, 2023. [find](https://scholar.google.com/scholar?q=Fang%2C+Y%2E%2C+Wang%2C+W%2E%2C+Xie%2C+B%2E%2C+Sun%2C+Q%2E%2C+Wu%2C+L%2E%2C+Wang%2C+X%2E%2C+Huang%2C+T%2E%2C+Wang%2C+X%2E%2C+and+Cao%2C+Y%2E+Eva%3A+Exploring+the+limits+of+masked+visual+representation+learning+at+scale%2E+In+CVPR%2C+2023%2E)
17. Fu, D. Y., Dao, T., Saab, K. K., Thomas, A. W., Rudra, A., and Re, C. Hungry hungry hippos: Towards language modeling with state space models. In ICLR, 2023. URL [https://openreview.net/forum?id=COZDy0WYGg](https://openreview.net/forum?id=COZDy0WYGg). [↗](https://openreview.net/forum?id=COZDy0WYGg)
18. Ghiasi, G., Cui, Y., Srinivas, A., Qian, R., Lin, T.-Y., Cubuk, E. D., Le, Q. V., and Zoph, B. Simple copy-paste is a strong data augmentation method for instance segmentation. In CVPR, 2021. [find](https://scholar.google.com/scholar?q=Ghiasi%2C+G%2E%2C+Cui%2C+Y%2E%2C+Srinivas%2C+A%2E%2C+Qian%2C+R%2E%2C+Lin%2C+T%2E%2DY%2E%2C+Cubuk%2C+E%2E+D%2E%2C+Le%2C+Q%2E+V%2E%2C+and+Zoph%2C+B%2E+Simple+copy%2Dpaste+is+a+strong+data+augmentation+method+for+instance+segmentation%2E+In+CVPR%2C+2021%2E)
19. Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. [find](https://scholar.google.com/scholar?q=Gu%2C+A%2E+and+Dao%2C+T%2E+Mamba%3A+Linear%2Dtime+sequence+modeling+with+selective+state+spaces%2E+arXiv+preprint+arXiv%3A2312%2E00752%2C+2023%2E)
20. Gu, A., Goel, K., and Ré, C. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021a. [find](https://scholar.google.com/scholar?q=Gu%2C+A%2E%2C+Goel%2C+K%2E%2C+and+Re%CC%81%2C+C%2E+Efficiently+modeling+long+sequences+with+structured+state+spaces%2E+arXiv+preprint+arXiv%3A2111%2E00396%2C+2021a%2E)
21. Gu, A., Johnson, I., Goel, K., Saab, K., Dao, T., Rudra, A., and Ré, C. Combining recurrent, convolutional, and continuous-time models with linear state space layers. In NeurIPS, 2021b. [find](https://scholar.google.com/scholar?q=Gu%2C+A%2E%2C+Johnson%2C+I%2E%2C+Goel%2C+K%2E%2C+Saab%2C+K%2E%2C+Dao%2C+T%2E%2C+Rudra%2C+A%2E%2C+and+Re%CC%81%2C+C%2E+Combining+recurrent%2C+convolutional%2C+and+continuous%2Dtime+models+with+linear+state+space+layers%2E+In+NeurIPS%2C+2021b%2E)
22. Gu, A., Goel, K., Gupta, A., and Ré, C. On the parameterization and initialization of diagonal state space models. In NeurIPS, 2022. [find](https://scholar.google.com/scholar?q=Gu%2C+A%2E%2C+Goel%2C+K%2E%2C+Gupta%2C+A%2E%2C+and+Re%CC%81%2C+C%2E+On+the+parameterization+and+initialization+of+diagonal+state+space+models%2E+In+NeurIPS%2C+2022%2E)
23. Gupta, A., Gu, A., and Berant, J. Diagonal state spaces are as effective as structured state spaces. In NeurIPS, 2022. [find](https://scholar.google.com/scholar?q=Gupta%2C+A%2E%2C+Gu%2C+A%2E%2C+and+Berant%2C+J%2E+Diagonal+state+spaces+are+as+effective+as+structured+state+spaces%2E+In+NeurIPS%2C+2022%2E)
24. He, K., Zhang, X., Ren, S., and Sun, J. [Deep residual learning for image recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/). In CVPR, 2016.
25. Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. Densely connected convolutional networks. In CVPR, 2017. [find](https://scholar.google.com/scholar?q=Huang%2C+G%2E%2C+Liu%2C+Z%2E%2C+Van+Der+Maaten%2C+L%2E%2C+and+Weinberger%2C+K%2E+Q%2E+Densely+connected+convolutional+networks%2E+In+CVPR%2C+2017%2E)
26. Islam, M. M. and Bertasius, G. Long movie clip classification with state-space video models. In ECCV, 2022. [find](https://scholar.google.com/scholar?q=Islam%2C+M%2E+M%2E+and+Bertasius%2C+G%2E+Long+movie+clip+classification+with+state%2Dspace+video+models%2E+In+ECCV%2C+2022%2E)
27. Islam, M. M., Hasan, M., Athrey, K. S., Braskich, T., and Bertasius, G. Efficient movie scene detection using state-space transformers. In CVPR, 2023. [find](https://scholar.google.com/scholar?q=Islam%2C+M%2E+M%2E%2C+Hasan%2C+M%2E%2C+Athrey%2C+K%2E+S%2E%2C+Braskich%2C+T%2E%2C+and+Bertasius%2C+G%2E+Efficient+movie+scene+detection+using+state%2Dspace+transformers%2E+In+CVPR%2C+2023%2E)
28. Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q., Sung, Y.-H., Li, Z., and Duerig, T. [Scaling up visual and vision-language representation learning with noisy text supervision](https://md.research.elamur.ai/free/en/papers/jia-2021-scaling-up-visual-vision-language/). In ICML, 2021.
29. Kalman, R. E. A new approach to linear filtering and prediction problems. 1960. [find](https://scholar.google.com/scholar?q=Kalman%2C+R%2E+E%2E+A+new+approach+to+linear+filtering+and+prediction+problems%2E+1960%2E)
30. Kenton, J. D. M.-W. C. and Toutanova, L. K. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In NAACL-HLT, 2019.
31. Kitaev, N., Kaiser, L., and Levskaya, A. [Reformer: The efficient transformer](https://md.research.elamur.ai/free/en/papers/kitaev-2020-reformer-efficient-transformer/). In ICLR, 2020. URL [https://openreview.net/forum?id=rkgNKkHtvB](https://openreview.net/forum?id=rkgNKkHtvB). [↗](https://openreview.net/forum?id=rkgNKkHtvB)
32. Krizhevsky, A., Sutskever, I., and Hinton, G. E. [Imagenet classification with deep convolutional neural networks](https://md.research.elamur.ai/free/en/papers/krizhevsky-2012-imagenet-classification-deep-convolutional/). In NeurIPS, 2012.
33. LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. [find](https://scholar.google.com/scholar?q=LeCun%2C+Y%2E%2C+Bottou%2C+L%2E%2C+Bengio%2C+Y%2E%2C+and+Haffner%2C+P%2E+Gradient%2Dbased+learning+applied+to+document+recognition%2E+Proceedings+of+the+IEEE%2C+86%2811%29%3A2278%E2%80%932324%2C+1998%2E)
34. Li, J., Li, D., Xiong, C., and Hoi, S. [Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation](https://md.research.elamur.ai/free/en/papers/li-2022-blip-bootstrapping-language-image-pre/). In ICML, 2022a.
35. Li, J., Li, D., Savarese, S., and Hoi, S. [Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/). arXiv preprint arXiv:2301.12597, 2023.
36. Li, Y., Cai, T., Zhang, Y., Chen, D., and Dey, D. What makes convolutional models great on long sequence modeling? In ICLR, 2022b. [find](https://scholar.google.com/scholar?q=Li%2C+Y%2E%2C+Cai%2C+T%2E%2C+Zhang%2C+Y%2E%2C+Chen%2C+D%2E%2C+and+Dey%2C+D%2E+What+makes+convolutional+models+great+on+long+sequence+modeling%3F+In+ICLR%2C+2022b%2E)
37. Li, Y., Mao, H., Girshick, R., and He, K. Exploring plain vision transformer backbones for object detection. In ECCV, 2022c. [find](https://scholar.google.com/scholar?q=Li%2C+Y%2E%2C+Mao%2C+H%2E%2C+Girshick%2C+R%2E%2C+and+He%2C+K%2E+Exploring+plain+vision+transformer+backbones+for+object+detection%2E+In+ECCV%2C+2022c%2E)
38. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. [Microsoft coco: Common objects in context](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/). In ECCV, 2014.
39. Liu, H., Li, C., Wu, Q., and Lee, Y. J. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). arXiv preprint arXiv:2304.08485, 2023.
40. Liu, S., Chen, T., Chen, X., Chen, X., Xiao, Q., Wu, B., Kärkkäinen, T., Pechenizkiy, M., Mocanu, D., and Wang, Z. More convnets in the 2020s: Scaling up kernels beyond 51x51 using sparsity. arXiv preprint arXiv:2207.03620, 2022a. — [arXiv:2207.03620](https://arxiv.org/abs/2207.03620) [find](https://scholar.google.com/scholar?q=Liu%2C+S%2E%2C+Chen%2C+T%2E%2C+Chen%2C+X%2E%2C+Chen%2C+X%2E%2C+Xiao%2C+Q%2E%2C+Wu%2C+B%2E%2C+Ka%CC%88rkka%CC%88inen%2C+T%2E%2C+Pechenizkiy%2C+M%2E%2C+Mocanu%2C+D%2E%2C+and+Wang%2C+Z%2E+More+convnets+in+the+2020s%3A+Scaling+up+kernels+beyond+51x51+using+sparsity%2E+arXiv+p)
41. Liu, Y., Tian, Y., Zhao, Y., Yu, H., Xie, L., Wang, Y., Ye, Q., and Liu, Y. Vmamba: Visual state space model. arXiv preprint arXiv:2401.10166, 2024. [find](https://scholar.google.com/scholar?q=Liu%2C+Y%2E%2C+Tian%2C+Y%2E%2C+Zhao%2C+Y%2E%2C+Yu%2C+H%2E%2C+Xie%2C+L%2E%2C+Wang%2C+Y%2E%2C+Ye%2C+Q%2E%2C+and+Liu%2C+Y%2E+Vmamba%3A+Visual+state+space+model%2E+arXiv+preprint+arXiv%3A2401%2E10166%2C+2024%2E)
42. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. [Swin transformer: Hierarchical vision transformer using shifted windows](https://md.research.elamur.ai/free/en/papers/liu-2021-swin-transformer-hierarchical-vision-transformer/). In ICCV, 2021.
43. Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., and Xie, S. A convnet for the 2020s. In CVPR, 2022b. [find](https://scholar.google.com/scholar?q=Liu%2C+Z%2E%2C+Mao%2C+H%2E%2C+Wu%2C+C%2E%2DY%2E%2C+Feichtenhofer%2C+C%2E%2C+Darrell%2C+T%2E%2C+and+Xie%2C+S%2E+A+convnet+for+the+2020s%2E+In+CVPR%2C+2022b%2E)
44. Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In ICLR, 2019. [find](https://scholar.google.com/scholar?q=Loshchilov%2C+I%2E+and+Hutter%2C+F%2E+Decoupled+weight+decay+regularization%2E+In+ICLR%2C+2019%2E)
45. Ma, J., Li, F., and Wang, B. U-mamba: Enhancing long-range dependency for biomedical image segmentation. arXiv preprint arXiv:2401.04722, 2024. — [arXiv:2401.04722](https://arxiv.org/abs/2401.04722) [find](https://scholar.google.com/scholar?q=Ma%2C+J%2E%2C+Li%2C+F%2E%2C+and+Wang%2C+B%2E+U%2Dmamba%3A+Enhancing+long%2Drange+dependency+for+biomedical+image+segmentation%2E+arXiv+preprint+arXiv%3A2401%2E04722%2C+2024%2E+%E2%80%94+%5BarXiv%3A2401%2E04722%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2401%2E04722%29)
46. Mehta, H., Gupta, A., Cutkosky, A., and Neyshabur, B. Long range language modeling via gated state spaces. In ICLR, 2023. URL [https://openreview.net/forum?id=5MkYIYCbva](https://openreview.net/forum?id=5MkYIYCbva). [↗](https://openreview.net/forum?id=5MkYIYCbva)
47. Nguyen, E., Goel, K., Gu, A., Downs, G., Shah, P., Dao, T., Baccus, S., and Ré, C. S4nd: Modeling images and videos as multidimensional signals with state spaces. In NeurIPS, 2022. [find](https://scholar.google.com/scholar?q=Nguyen%2C+E%2E%2C+Goel%2C+K%2E%2C+Gu%2C+A%2E%2C+Downs%2C+G%2E%2C+Shah%2C+P%2E%2C+Dao%2C+T%2E%2C+Baccus%2C+S%2E%2C+and+Re%CC%81%2C+C%2E+S4nd%3A+Modeling+images+and+videos+as+multidimensional+signals+with+state+spaces%2E+In+NeurIPS%2C+2022%2E)
48. Qin, Z., Yang, S., and Zhong, Y. Hierarchically gated recurrent neural network for sequence modeling. In NeurIPS, 2023. URL [https://openreview.net/forum?id=P1TCHxJwLB](https://openreview.net/forum?id=P1TCHxJwLB). [↗](https://openreview.net/forum?id=P1TCHxJwLB)
49. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). In ICML, 2021.
50. Radosavovic, I., Kosaraju, R. P., Girshick, R., He, K., and Dollár, P. [Designing network design spaces](https://md.research.elamur.ai/free/en/papers/radosavovic-2020-designing-network-design-spaces/). In CVPR, 2020.
51. Rao, Y., Zhao, W., Zhu, Z., Lu, J., and Zhou, J. Global filter networks for image classification. Advances in neural information processing systems, 34:980–993, 2021. [find](https://scholar.google.com/scholar?q=Rao%2C+Y%2E%2C+Zhao%2C+W%2E%2C+Zhu%2C+Z%2E%2C+Lu%2C+J%2E%2C+and+Zhou%2C+J%2E+Global+filter+networks+for+image+classification%2E+Advances+in+neural+information+processing+systems%2C+34%3A980%E2%80%93993%2C+2021%2E)
52. Simonyan, K. and Zisserman, A. [Very deep convolutional networks for large-scale image recognition](https://md.research.elamur.ai/free/en/papers/simonyan-2014-very-deep-convolutional-networks-large/). arXiv preprint arXiv:1409.1556, 2014.
53. Smith, J. T., De Mello, S., Kautz, J., Linderman, S., and Byeon, W. Convolutional state space models for long-range spatiotemporal modeling. In NeurIPS, 2023a. [find](https://scholar.google.com/scholar?q=Smith%2C+J%2E+T%2E%2C+De+Mello%2C+S%2E%2C+Kautz%2C+J%2E%2C+Linderman%2C+S%2E%2C+and+Byeon%2C+W%2E+Convolutional+state+space+models+for+long%2Drange+spatiotemporal+modeling%2E+In+NeurIPS%2C+2023a%2E)
54. Smith, J. T., Warrington, A., and Linderman, S. Simplified state space layers for sequence modeling. In ICLR, 2023b. URL [https://openreview.net/forum?id=Ai8Hw3AXqks](https://openreview.net/forum?id=Ai8Hw3AXqks). [↗](https://openreview.net/forum?id=Ai8Hw3AXqks)
55. Strudel, R., Garcia, R., Laptev, I., and Schmid, C. Segmenter: Transformer for semantic segmentation. In ICCV, 2021. [find](https://scholar.google.com/scholar?q=Strudel%2C+R%2E%2C+Garcia%2C+R%2E%2C+Laptev%2C+I%2E%2C+and+Schmid%2C+C%2E+Segmenter%3A+Transformer+for+semantic+segmentation%2E+In+ICCV%2C+2021%2E)
56. Sun, Y., Dong, L., Huang, S., Ma, S., Xia, Y., Xue, J., Wang, J., and Wei, F. [Retentive network: A successor to transformer for large language models](https://md.research.elamur.ai/free/en/papers/sun-2023-retentive-network-successor-transformer-large/)s. arXiv preprint arXiv:2307.08621, 2023.
57. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. Going deeper with convolutions. In CVPR, 2015. [find](https://scholar.google.com/scholar?q=Szegedy%2C+C%2E%2C+Liu%2C+W%2E%2C+Jia%2C+Y%2E%2C+Sermanet%2C+P%2E%2C+Reed%2C+S%2E%2C+Anguelov%2C+D%2E%2C+Erhan%2C+D%2E%2C+Vanhoucke%2C+V%2E%2C+and+Rabinovich%2C+A%2E+Going+deeper+with+convolutions%2E+In+CVPR%2C+2015%2E)
58. Tan, M. and Le, Q. [Efficientnet: Rethinking model scaling for convolutional neural networks](https://md.research.elamur.ai/free/en/papers/tan-2019-efficientnet-rethinking-model-scaling-convolutional/). In ICML, 2019.
59. Tan, M. and Le, Q. Efficientnetv2: Smaller models and faster training. In ICML, 2021. [find](https://scholar.google.com/scholar?q=Tan%2C+M%2E+and+Le%2C+Q%2E+Efficientnetv2%3A+Smaller+models+and+faster+training%2E+In+ICML%2C+2021%2E)
60. Tolstikhin, I. O., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., Yung, J., Steiner, A., Keysers, D., Uszkoreit, J., et al. Mlp-mixer: An all-mlp architecture for vision. In NeurIPS, 2021. [find](https://scholar.google.com/scholar?q=Tolstikhin%2C+I%2E+O%2E%2C+Houlsby%2C+N%2E%2C+Kolesnikov%2C+A%2E%2C+Beyer%2C+L%2E%2C+Zhai%2C+X%2E%2C+Unterthiner%2C+T%2E%2C+Yung%2C+J%2E%2C+Steiner%2C+A%2E%2C+Keysers%2C+D%2E%2C+Uszkoreit%2C+J%2E%2C+et+al%2E+Mlp%2Dmixer%3A+An+all%2Dmlp+architecture+for+vision%2E+In+NeurIP)
61. Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. [Training data-efficient image transformers & distillation through attention](https://md.research.elamur.ai/free/en/papers/touvron-2020-training-data-efficient-image-transformers/). In ICML, 2021a.
62. Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. [Training data-efficient image transformers & distillation through attention](https://md.research.elamur.ai/free/en/papers/touvron-2020-training-data-efficient-image-transformers/). In ICML, 2021b.
63. Touvron, H., Bojanowski, P., Caron, M., Cord, M., El-Nouby, A., Grave, E., Izacard, G., Joulin, A., Synnaeve, G., Verbeek, J., et al. Resmlp: Feedforward networks for image classification with data-efficient training. TPAMI, 2022. [find](https://scholar.google.com/scholar?q=Touvron%2C+H%2E%2C+Bojanowski%2C+P%2E%2C+Caron%2C+M%2E%2C+Cord%2C+M%2E%2C+El%2DNouby%2C+A%2E%2C+Grave%2C+E%2E%2C+Izacard%2C+G%2E%2C+Joulin%2C+A%2E%2C+Synnaeve%2C+G%2E%2C+Verbeek%2C+J%2E%2C+et+al%2E+Resmlp%3A+Feedforward+networks+for+image+classification+with+data%2Def)
64. Wang, J., Sun, K., Cheng, T., Jiang, B., Deng, C., Zhao, Y., Liu, D., Mu, Y., Tan, M., Wang, X., et al. Deep high-resolution representation learning for visual recognition. TPAMI, 2020a. [find](https://scholar.google.com/scholar?q=Wang%2C+J%2E%2C+Sun%2C+K%2E%2C+Cheng%2C+T%2E%2C+Jiang%2C+B%2E%2C+Deng%2C+C%2E%2C+Zhao%2C+Y%2E%2C+Liu%2C+D%2E%2C+Mu%2C+Y%2E%2C+Tan%2C+M%2E%2C+Wang%2C+X%2E%2C+et+al%2E+Deep+high%2Dresolution+representation+learning+for+visual+recognition%2E+TPAMI%2C+2020a%2E)
65. Wang, J., Yan, J. N., Gu, A., and Rush, A. M. Pretraining without attention. arXiv preprint arXiv:2212.10544, 2022. [find](https://scholar.google.com/scholar?q=Wang%2C+J%2E%2C+Yan%2C+J%2E+N%2E%2C+Gu%2C+A%2E%2C+and+Rush%2C+A%2E+M%2E+Pretraining+without+attention%2E+arXiv+preprint+arXiv%3A2212%2E10544%2C+2022%2E)
66. Wang, J., Zhu, W., Wang, P., Yu, X., Liu, L., Omar, M., and Hamid, R. Selective structured state-spaces for long-form video understanding. In CVPR, 2023a. [find](https://scholar.google.com/scholar?q=Wang%2C+J%2E%2C+Zhu%2C+W%2E%2C+Wang%2C+P%2E%2C+Yu%2C+X%2E%2C+Liu%2C+L%2E%2C+Omar%2C+M%2E%2C+and+Hamid%2C+R%2E+Selective+structured+state%2Dspaces+for+long%2Dform+video+understanding%2E+In+CVPR%2C+2023a%2E)
67. Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020b. [find](https://scholar.google.com/scholar?q=Wang%2C+S%2E%2C+Li%2C+B%2E+Z%2E%2C+Khabsa%2C+M%2E%2C+Fang%2C+H%2E%2C+and+Ma%2C+H%2E+Linformer%3A+Self%2Dattention+with+linear+complexity%2E+arXiv+preprint+arXiv%3A2006%2E04768%2C+2020b%2E)
68. Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., and Shao, L. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, 2021. [find](https://scholar.google.com/scholar?q=Wang%2C+W%2E%2C+Xie%2C+E%2E%2C+Li%2C+X%2E%2C+Fan%2C+D%2E%2DP%2E%2C+Song%2C+K%2E%2C+Liang%2C+D%2E%2C+Lu%2C+T%2E%2C+Luo%2C+P%2E%2C+and+Shao%2C+L%2E+Pyramid+vision+transformer%3A+A+versatile+backbone+for+dense+prediction+without+convolutions%2E+In+ICCV%2C+2021%2E)
69. Wang, W., Dai, J., Chen, Z., Huang, Z., Li, Z., Zhu, X., Hu, X., Lu, T., Lu, L., Li, H., et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In CVPR, 2023b. [find](https://scholar.google.com/scholar?q=Wang%2C+W%2E%2C+Dai%2C+J%2E%2C+Chen%2C+Z%2E%2C+Huang%2C+Z%2E%2C+Li%2C+Z%2E%2C+Zhu%2C+X%2E%2C+Hu%2C+X%2E%2C+Lu%2C+T%2E%2C+Lu%2C+L%2E%2C+Li%2C+H%2E%2C+et+al%2E+Internimage%3A+Exploring+large%2Dscale+vision+foundation+models+with+deformable+convolutions%2E+In+CVPR%2C+2023b)
70. Wang, W., Ma, S., Xu, H., Usuyama, N., Ding, J., Poon, H., and Wei, F. When an image is worth 1,024 x 1,024 words: A case study in computational pathology. arXiv preprint arXiv:2312.03558, 2023c. [find](https://scholar.google.com/scholar?q=Wang%2C+W%2E%2C+Ma%2C+S%2E%2C+Xu%2C+H%2E%2C+Usuyama%2C+N%2E%2C+Ding%2C+J%2E%2C+Poon%2C+H%2E%2C+and+Wei%2C+F%2E+When+an+image+is+worth+1%2C024+x+1%2C024+words%3A+A+case+study+in+computational+pathology%2E+arXiv+preprint+arXiv%3A2312%2E03558%2C+2023c%2E)
71. Wu, H., Xiao, B., Codella, N., Liu, M., Dai, X., Yuan, L., and Zhang, L. Cvt: Introducing convolutions to vision transformers. In ICCV, 2021. [find](https://scholar.google.com/scholar?q=Wu%2C+H%2E%2C+Xiao%2C+B%2E%2C+Codella%2C+N%2E%2C+Liu%2C+M%2E%2C+Dai%2C+X%2E%2C+Yuan%2C+L%2E%2C+and+Zhang%2C+L%2E+Cvt%3A+Introducing+convolutions+to+vision+transformers%2E+In+ICCV%2C+2021%2E)
72. Xiao, T., Liu, Y., Zhou, B., Jiang, Y., and Sun, J. Unified perceptual parsing for scene understanding. In ECCV, 2018a. [find](https://scholar.google.com/scholar?q=Xiao%2C+T%2E%2C+Liu%2C+Y%2E%2C+Zhou%2C+B%2E%2C+Jiang%2C+Y%2E%2C+and+Sun%2C+J%2E+Unified+perceptual+parsing+for+scene+understanding%2E+In+ECCV%2C+2018a%2E)
73. Xiao, T., Liu, Y., Zhou, B., Jiang, Y., and Sun, J. Unified perceptual parsing for scene understanding. In ECCV, 2018b. [find](https://scholar.google.com/scholar?q=Xiao%2C+T%2E%2C+Liu%2C+Y%2E%2C+Zhou%2C+B%2E%2C+Jiang%2C+Y%2E%2C+and+Sun%2C+J%2E+Unified+perceptual+parsing+for+scene+understanding%2E+In+ECCV%2C+2018b%2E)
74. Xie, S., Girshick, R., Dollár, P., Tu, Z., and He, K. Aggregated residual transformations for deep neural networks. In CVPR, 2017. [find](https://scholar.google.com/scholar?q=Xie%2C+S%2E%2C+Girshick%2C+R%2E%2C+Dolla%CC%81r%2C+P%2E%2C+Tu%2C+Z%2E%2C+and+He%2C+K%2E+Aggregated+residual+transformations+for+deep+neural+networks%2E+In+CVPR%2C+2017%2E)
75. Xing, Z., Ye, T., Yang, Y., Liu, G., and Zhu, L. Segmamba: Long-range sequential modeling mamba for 3d medical image segmentation. arXiv preprint arXiv:2401.13560, 2024. — [arXiv:2401.13560](https://arxiv.org/abs/2401.13560) [find](https://scholar.google.com/scholar?q=Xing%2C+Z%2E%2C+Ye%2C+T%2E%2C+Yang%2C+Y%2E%2C+Liu%2C+G%2E%2C+and+Zhu%2C+L%2E+Segmamba%3A+Long%2Drange+sequential+modeling+mamba+for+3d+medical+image+segmentation%2E+arXiv+preprint+arXiv%3A2401%2E13560%2C+2024%2E+%E2%80%94+%5BarXiv%3A2401%2E13560%5D%28https%3A%2F%2Fa)
76. Yan, J. N., Gu, J., and Rush, A. M. Diffusion models without attention. arXiv preprint arXiv:2311.18257, 2023. — [arXiv:2311.18257](https://arxiv.org/abs/2311.18257) [find](https://scholar.google.com/scholar?q=Yan%2C+J%2E+N%2E%2C+Gu%2C+J%2E%2C+and+Rush%2C+A%2E+M%2E+Diffusion+models+without+attention%2E+arXiv+preprint+arXiv%3A2311%2E18257%2C+2023%2E+%E2%80%94+%5BarXiv%3A2311%2E18257%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2311%2E18257%29)
77. Yang, J., Li, C., Zhang, P., Dai, X., Xiao, B., Yuan, L., and Gao, J. Focal self-attention for local-global interactions in vision transformers. arXiv preprint arXiv:2107.00641, 2021. — [arXiv:2107.00641](https://arxiv.org/abs/2107.00641) [find](https://scholar.google.com/scholar?q=Yang%2C+J%2E%2C+Li%2C+C%2E%2C+Zhang%2C+P%2E%2C+Dai%2C+X%2E%2C+Xiao%2C+B%2E%2C+Yuan%2C+L%2E%2C+and+Gao%2C+J%2E+Focal+self%2Dattention+for+local%2Dglobal+interactions+in+vision+transformers%2E+arXiv+preprint+arXiv%3A2107%2E00641%2C+2021%2E+%E2%80%94+%5BarXiv%3A2107%2E00)
78. Yu, W., Luo, M., Zhou, P., Si, C., Zhou, Y., Wang, X., Feng, J., and Yan, S. Metaformer is actually what you need for vision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10819– 10829, 2022. [find](https://scholar.google.com/scholar?q=Yu%2C+W%2E%2C+Luo%2C+M%2E%2C+Zhou%2C+P%2E%2C+Si%2C+C%2E%2C+Zhou%2C+Y%2E%2C+Wang%2C+X%2E%2C+Feng%2C+J%2E%2C+and+Yan%2C+S%2E+Metaformer+is+actually+what+you+need+for+vision%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+computer+vision+and+pattern+r)
79. Zhou, B., Zhao, H., Puig, X., Xiao, T., Fidler, S., Barriuso, A., and Torralba, A. Semantic understanding of scenes through the ade20k dataset. IJCV, 2019. [find](https://scholar.google.com/scholar?q=Zhou%2C+B%2E%2C+Zhao%2C+H%2E%2C+Puig%2C+X%2E%2C+Xiao%2C+T%2E%2C+Fidler%2C+S%2E%2C+Barriuso%2C+A%2E%2C+and+Torralba%2C+A%2E+Semantic+understanding+of+scenes+through+the+ade20k+dataset%2E+IJCV%2C+2019%2E)
80. A. Visualization GT Figure 5: Visualization comparison of DeiT-Ti (Touvron et al., 2021b) and our Vim-Ti on the Cascade Mask R-CNN (Cai & Vasconcelos, 2019) framework. Thanks to the long-range context learning of SSM, we can capture the very large object in the image, which the DeiT-Ti counterpart f… [find](https://scholar.google.com/scholar?q=A%2E+Visualization+GT+Figure+5%3A+Visualization+comparison+of+DeiT%2DTi+%28Touvron+et+al%2E%2C+2021b%29+and+our+Vim%2DTi+on+the+Cascade+Mask+R%2DCNN+%28Cai+%26+Vasconcelos%2C+2019%29+framework%2E+Thanks+to+the+long%2Drange+context)

*Updated 2 November 2026*