---
title: "HERMES — paper by Zhang et al. (2026) — summary and ideas"
description: "Multimodal LLMs handle offline video well, but streaming video is harder. 55 references, cited by 4 papers here."
authors: ["Haowei Zhang", "Shudong Yang", "Jinlan Fu", "See-Kiong Ng", "Xipeng Qiu"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.381/"
doi: "10.18653/v1/2026.acl-long.381"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2026-hermes-kv-cache-as-hierarchical/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2026-hermes-kv-cache-as-hierarchical/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/2/) / HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding

# HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding

**Authors:** [Haowei Zhang](https://md.research.elamur.ai/free/en/authors/haowei-zhang/), Shudong Yang, [Jinlan Fu](https://md.research.elamur.ai/free/en/authors/jinlan-fu/), [See-Kiong Ng](https://md.research.elamur.ai/free/en/authors/see-kiong-ng/) and [Xipeng Qiu](https://md.research.elamur.ai/free/en/authors/xipeng-qiu/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.381/) · [PDF](https://aclanthology.org/2026.acl-long.381.pdf) · [DOI 10.18653/v1/2026.acl-long.381](https://doi.org/10.18653/v1/2026.acl-long.381) · [OpenAlex](https://openalex.org/W7166826313)

**Citations:** 5 citations (OpenAlex, 2026-10-14) · 4 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Multimodal LLMs handle offline video well, but streaming video is harder. A system must stay accurate, answer quickly, and keep GPU memory bounded, even though future frames and questions are unknown. Earlier cache-based methods such as ReKV and LiveVLM move KV entries off the GPU and retrieve them when a question arrives, which slows responses. HERMES is a training-free, plug-and-play alternative. It is based on an attention analysis that treats the KV cache as layered memory:
- **Shallow layers** behave like sensory memory and favour the newest frames.
- **Middle layers** act as working memory.
- **Deep layers** keep sparse per-frame anchor tokens, functioning as long-term memory.

The method has three parts: layer-specific token eviction, cross-layer memory smoothing, and position re-indexing. The compact cache is reused directly, so nothing extra is computed when a query arrives. Compared with the previous training-free SOTA, time-to-first-token is about 10× lower, and GPU memory stays constant. Accuracy matches or beats uniform sampling while using up to 68% fewer video tokens. This makes HERMES practical for real-time video assistants with limited hardware.

## Key points
- **Attention study:** The authors analysed 300 randomly sampled VideoMME video–question pairs with LLaVA-OV-7B. In deep layers, attention peaks recur every 196 tokens, which equals one frame's token count. The authors read these peaks as frame-level anchors.
- **Eviction scoring by layer:**
  - Shallow layers score tokens with an exponential forgetting curve inspired by Ebbinghaus.
  - Deep layers score tokens by attention to a generic guidance prompt that stands in for the unknown query.
  - In deep layers, evicted tokens are merged into a summary token.
- **Efficiency:** On a single A800 GPU, TTFT stays below 30 ms as the number of frames grows, with no OOM risk and no external CPU or disk storage.
- **Evaluation and accuracy:**
  - Evaluated on StreamingBench, OVO-Bench and VideoMME with LLaVA-OV-7B and Qwen2.5-VL-7B.
  - Gains reach up to 11.4% on streaming benchmarks.
  - VideoMME average for LLaVA-OV-7B rises from 57.67 to 58.44.
- **Ablations and limits:**
  - Lazy re-indexing works best for streaming and eager re-indexing for offline use.
  - Results barely change when the guidance prompt is reworded.
  - 72B models were not tested.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (1): [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 297 works

## Cite

Cite as: Haowei Zhang, Shudong Yang, Jinlan Fu, See-Kiong Ng and Xipeng Qiu. 2026. HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhang2026hermes,
  title = {{HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding}},
  author = {Haowei Zhang, Shudong Yang, Jinlan Fu, See-Kiong Ng and Xipeng Qiu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.381},
  url = {https://doi.org/10.18653/v1/2026.acl-long.381}
}
```

- [← previous Findings of the BEA 2026 Shared Task on Vocabulary Difficulty Prediction for English Learners](https://md.research.elamur.ai/free/en/papers/felice-2026-findings-bea-2026-shared-task/)
- [next → Knowing When to Abstain: Medical LLMs Under Clinical Uncertainty](https://md.research.elamur.ai/free/en/papers/machcha-2026-knowing-when-abstain-medical-llms/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/2/)

## References

55 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anthropic. 2024. Claude 3.5 sonnet. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2024%2E+Claude+3%2E5+sonnet%2E)
2. R.C. Atkinson and R.M. Shiffrin. 1968. Human memory: A proposed system and its control processes. [find](https://scholar.google.com/scholar?q=R%2EC%2E+Atkinson+and+R%2EM%2E+Shiffrin%2E+1968%2E+Human+memory%3A+A+proposed+system+and+its+control+processes%2E)
3. Alan D. Baddeley and Graham Hitch. 1974. Working memory. [find](https://scholar.google.com/scholar?q=Alan+D%2E+Baddeley+and+Graham+Hitch%2E+1974%2E+Working+memory%2E)
4. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025a. [Qwen3-vl technical report](https://md.research.elamur.ai/free/en/papers/bai-2025-qwen3-vl-technical-report/). Preprint, a…
5. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025b. [Qwen2.5-vl technical report](https://md.research.elamur.ai/free/en/papers/bai-2025-qwen2-5-vl-technical/). Preprint, arXiv:2502.1…
6. Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. 2024. Titans: Learning to memorize at test time. Preprint, arXiv:2501.00663. [find](https://scholar.google.com/scholar?q=Ali+Behrouz%2C+Peilin+Zhong%2C+and+Vahab+Mirrokni%2E+2024%2E+Titans%3A+Learning+to+memorize+at+test+time%2E+Preprint%2C+arXiv%3A2501%2E00663%2E)
7. Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. 2024a. Videollm-online: Online video large language model for streaming video. Preprint, arXiv:2406.11816. [find](https://scholar.google.com/scholar?q=Joya+Chen%2C+Zhaoyang+Lv%2C+Shiwei+Wu%2C+Kevin+Qinghong+Lin%2C+Chenan+Song%2C+Difei+Gao%2C+Jia%2DWei+Liu%2C+Ziteng+Gao%2C+Dongxing+Mao%2C+and+Mike+Zheng+Shou%2E+2024a%2E+Videollm%2Donline%3A+Online+video+large+language+model+for)
8. Xueyi Chen, Keda Tao, Kele Shao, and Huan Wang. 2025a. Streamingtom: Streaming token compression for efficient video understanding. Preprint, arXiv:2510.18269. — [arXiv:2510.18269](https://arxiv.org/abs/2510.18269) [find](https://scholar.google.com/scholar?q=Xueyi+Chen%2C+Keda+Tao%2C+Kele+Shao%2C+and+Huan+Wang%2E+2025a%2E+Streamingtom%3A+Streaming+token+compression+for+efficient+video+understanding%2E+Preprint%2C+arXiv%3A2510%2E18269%2E+%E2%80%94+%5BarXiv%3A2510%2E18269%5D%28https%3A%2F%2Farxiv%2Eorg%2Fa)
9. Yilong Chen, Xiang Bai, Zhibin Wang, Chengyu Bai, Yuhan Dai, Ming Lu, and Shanghang Zhang. 2025b. Streamkv: Streaming video question-answering with segment-based kv cache retrieval and compression. Preprint, arXiv:2511.07278. — [arXiv:2511.07278](https://arxiv.org/abs/2511.07278) [find](https://scholar.google.com/scholar?q=Yilong+Chen%2C+Xiang+Bai%2C+Zhibin+Wang%2C+Chengyu+Bai%2C+Yuhan+Dai%2C+Ming+Lu%2C+and+Shanghang+Zhang%2E+2025b%2E+Streamkv%3A+Streaming+video+question%2Danswering+with+segment%2Dbased+kv+cache+retrieval+and+compression%2E+Pr)
10. Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang, Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, and 16 others. 2024b. [How far are we to gpt-4v?](https://md.research.elamur.ai/free/en/papers/chen-2024-far-gpt-4v-closing/) closing the gap to commercial m…
11. Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. 2024. [Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms](https://md.research.elamur.ai/free/en/papers/cheng-2024-videollama-2-advancing-spatial-temporal/). Preprint, arXiv:2406.07476.
12. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan-Jiang Jiang, and 3416 [other…](https://md.research.elamur.ai/free/en/papers/comanici-2025-gemini-2-5-pushing-frontier/)
13. Shangzhe Di, Zhelun Yu, Guanghao Zhang, Haoyuan Li, Tao Zhong, Hao Cheng, Bolin Li, Wanggui He, Fangxun Shu, and Hao Jiang. 2025. Streaming video question-answering with in-context video kv-cache retrieval. Preprint, arXiv:2503.00540. [find](https://scholar.google.com/scholar?q=Shangzhe+Di%2C+Zhelun+Yu%2C+Guanghao+Zhang%2C+Haoyuan+Li%2C+Tao+Zhong%2C+Hao+Cheng%2C+Bolin+Li%2C+Wanggui+He%2C+Fangxun+Shu%2C+and+Hao+Jiang%2E+2025%2E+Streaming+video+question%2Danswering+with+in%2Dcontext+video+kv%2Dcache+retr)
14. Hermann Ebbinghaus. 2013. Memory: A contribution to experimental psychology. Annals of neurosciences, 20(4):155. [find](https://scholar.google.com/scholar?q=Hermann+Ebbinghaus%2E+2013%2E+Memory%3A+A+contribution+to+experimental+psychology%2E+Annals+of+neurosciences%2C+20%284%29%3A155%2E)
15. Jiajun Fei, Dian Li, Zhidong Deng, Zekun Wang, Gang Liu, and Hui Wang. 2024. Video-ccam: Enhancing video-language understanding with causal cross-attention masks for short and long videos. Preprint, arXiv:2408.14023. — [arXiv:2408.14023](https://arxiv.org/abs/2408.14023) [find](https://scholar.google.com/scholar?q=Jiajun+Fei%2C+Dian+Li%2C+Zhidong+Deng%2C+Zekun+Wang%2C+Gang+Liu%2C+and+Hui+Wang%2E+2024%2E+Video%2Dccam%3A+Enhancing+video%2Dlanguage+understanding+with+causal+cross%2Dattention+masks+for+short+and+long+videos%2E+Preprint%2C+a)
16. Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Caifeng Shan, and 2 others. 2025. [Video-mme: The first-ever comprehensive evaluati…](https://md.research.elamur.ai/free/en/papers/fu-2024-video-mme-first-ever-comprehensive/)
17. Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong X… [find](https://scholar.google.com/scholar?q=Kristen+Grauman%2C+Andrew+Westbury%2C+Eugene+Byrne%2C+Zachary+Chavis%2C+Antonino+Furnari%2C+Rohit+Girdhar%2C+Jackson+Hamburger%2C+Hao+Jiang%2C+Miao+Liu%2C+Xingyu+Liu%2C+Miguel+Martin%2C+Tushar+Nagarajan%2C+Ilija+Radosavovic%2C)
18. Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, and 6 others. 2024. [Minicpm: Unveiling the potential of small lan…](https://md.research.elamur.ai/free/en/papers/hu-2024-minicpm-unveiling-potential-small-language/)
19. Jiajun Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xiaoqi Ma, Xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. 2024b. Kangaroo: A powerful video-language model supporting long-context video input. Preprint, arXiv:2408.15542. — [arXiv:2408.15542](https://arxiv.org/abs/2408.15542) [find](https://scholar.google.com/scholar?q=Jiajun+Liu%2C+Yibing+Wang%2C+Hanghang+Ma%2C+Xiaoping+Wu%2C+Xiaoqi+Ma%2C+Xiaoming+Wei%2C+Jianbin+Jiao%2C+Enhua+Wu%2C+and+Jie+Hu%2E+2024b%2E+Kangaroo%3A+A+powerful+video%2Dlanguage+model+supporting+long%2Dcontext+video+input%2E+Pr)
20. Yuyang Hu, Shichun Liu, Yanwei Yue, Guibin Zhang, Boyang Liu, Fangyi Zhu, Jiahang Lin, Honglin Guo, Shihan Dou, Zhiheng Xi, Senjie Jin, Jiejun Tan, Yanbin Yin, Jiongnan Liu, Zeyu Zhang, Zhongxiang Sun, Yutao Zhu, Hao Sun, Boci Peng, and 28 others. 2025. Memory in the age of ai agents. Preprint, arXi… — [arXiv:2512.13564](https://arxiv.org/abs/2512.13564) [find](https://scholar.google.com/scholar?q=Yuyang+Hu%2C+Shichun+Liu%2C+Yanwei+Yue%2C+Guibin+Zhang%2C+Boyang+Liu%2C+Fangyi+Zhu%2C+Jiahang+Lin%2C+Honglin+Guo%2C+Shihan+Dou%2C+Zhiheng+Xi%2C+Senjie+Jin%2C+Jiejun+Tan%2C+Yanbin+Yin%2C+Jiongnan+Liu%2C+Zeyu+Zhang%2C+Zhongxiang+Sun)
21. Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. 2023. [Egoschema: A diagnostic benchmark for very long-form video language understanding](https://md.research.elamur.ai/free/en/papers/mangalam-2023-egoschema-diagnostic-benchmark-very-long/). Preprint, arXiv:2308.09126.
22. Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin. 2020. Movienet: A holistic dataset for movie understanding. Preprint, arXiv:2007.10937. — [arXiv:2007.10937](https://arxiv.org/abs/2007.10937) [find](https://scholar.google.com/scholar?q=Qingqiu+Huang%2C+Yu+Xiong%2C+Anyi+Rao%2C+Jiaze+Wang%2C+and+Dahua+Lin%2E+2020%2E+Movienet%3A+A+holistic+dataset+for+movie+understanding%2E+Preprint%2C+arXiv%3A2007%2E10937%2E+%E2%80%94+%5BarXiv%3A2007%2E10937%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2007%2E10)
23. Zhenyu Ning, Guangda Liu, Qihao Jin, Wenchao Ding, Minyi Guo, and Jieru Zhao. 2025. Livevlm: Efficient online video understanding via streaming-oriented kv cache and retrieval. Preprint, arXiv:2505.15269. — [arXiv:2505.15269](https://arxiv.org/abs/2505.15269) [find](https://scholar.google.com/scholar?q=Zhenyu+Ning%2C+Guangda+Liu%2C+Qihao+Jin%2C+Wenchao+Ding%2C+Minyi+Guo%2C+and+Jieru+Zhao%2E+2025%2E+Livevlm%3A+Efficient+online+video+understanding+via+streaming%2Doriented+kv+cache+and+retrieval%2E+Preprint%2C+arXiv%3A2505%2E15)
24. Minsoo Kim, Kyuhong Shim, Jungwook Choi, and Simyung Chang. 2024. Infinipot: Infinite context processing on memory-constrained llms. Preprint, arXiv:2410.01518. [find](https://scholar.google.com/scholar?q=Minsoo+Kim%2C+Kyuhong+Shim%2C+Jungwook+Choi%2C+and+Simyung+Chang%2E+2024%2E+Infinipot%3A+Infinite+context+processing+on+memory%2Dconstrained+llms%2E+Preprint%2C+arXiv%3A2410%2E01518%2E)
25. OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, and 401 others. 2024. [Gpt-4o system card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/).…
26. Minsoo Kim, Kyuhong Shim, Jungwook Choi, and Simyung Chang. 2025. Infinipot-v: Memory-constrained kv cache compression for streaming video understanding. Preprint, arXiv:2506.15745. [find](https://scholar.google.com/scholar?q=Minsoo+Kim%2C+Kyuhong+Shim%2C+Jungwook+Choi%2C+and+Simyung+Chang%2E+2025%2E+Infinipot%2Dv%3A+Memory%2Dconstrained+kv+cache+compression+for+streaming+video+understanding%2E+Preprint%2C+arXiv%3A2506%2E15745%2E)
27. Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2024. Memgpt: Towards llms as operating systems. Preprint, arXiv:2310.08560. [find](https://scholar.google.com/scholar?q=Charles+Packer%2C+Sarah+Wooders%2C+Kevin+Lin%2C+Vivian+Fang%2C+Shishir+G%2E+Patil%2C+Ion+Stoica%2C+and+Joseph+E%2E+Gonzalez%2E+2024%2E+Memgpt%3A+Towards+llms+as+operating+systems%2E+Preprint%2C+arXiv%3A2310%2E08560%2E)
28. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024a. [Llava-onevision: Easy visual task transfer](https://md.research.elamur.ai/free/en/papers/li-2024-llava-onevision-easy-visual-task/). Preprint, arXiv:2408.03326.
29. Viorica Pătrăucean, Lucas Smaira, Ankush Gupta, Adrià Recasens Continente, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, Carl Doersch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alex Frechette, Hanna Klimczak, Raphael Koster, Junlin Zhang, Step… [find](https://scholar.google.com/scholar?q=Viorica+Pa%CC%86tra%CC%86ucean%2C+Lucas+Smaira%2C+Ankush+Gupta%2C+Adri%C3%A0+Recasens+Continente%2C+Larisa+Markeeva%2C+Dylan+Banarse%2C+Skanda+Koppula%2C+Joseph+Heyward%2C+Mateusz+Malinowski%2C+Yi+Yang%2C+Carl+Doersch%2C+Tatiana+Matejovi)
30. Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. 2024b. [Mvbench: A comprehensive multimodal video understanding benchmark](https://md.research.elamur.ai/free/en/papers/li-2023-mvbench-comprehensive-multi-modal-video/). Preprint, arXiv:2311.17005.
31. Yifei Li, Junbo Niu, Ziyang Miao, Chunjiang Ge, Yuanhang Zhou, Qihao He, Xiaoyi Dong, Haodong Duan, Shuangrui Ding, Rui Qian, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, and Jiaqi Wang. 2025. Ovo-bench: How far is your video-llms from real-world online video understanding? Preprint, arXiv:2501.0… — [arXiv:2501.05510](https://arxiv.org/abs/2501.05510) [find](https://scholar.google.com/scholar?q=Yifei+Li%2C+Junbo+Niu%2C+Ziyang+Miao%2C+Chunjiang+Ge%2C+Yuanhang+Zhou%2C+Qihao+He%2C+Xiaoyi+Dong%2C+Haodong+Duan%2C+Shuangrui+Ding%2C+Rui+Qian%2C+Pan+Zhang%2C+Yuhang+Zang%2C+Yuhang+Cao%2C+Conghui+He%2C+and+Jiaqi+Wang%2E+2025%2E+Ovo%2D)
32. Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. 2024a. [Vila: On pre-training for visual language models](https://md.research.elamur.ai/free/en/papers/lin-2023-vila-pre-training-visual-language/). Preprint, arXiv:2312.07533.
33. Rui Qian, Shuangrui Ding, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. 2025. Dispider: Enabling video llms with active real-time interaction via disentangled perception, decision, and reaction. Preprint, arXiv:2501.03218. — [arXiv:2501.03218](https://arxiv.org/abs/2501.03218) [find](https://scholar.google.com/scholar?q=Rui+Qian%2C+Shuangrui+Ding%2C+Xiaoyi+Dong%2C+Pan+Zhang%2C+Yuhang+Zang%2C+Yuhang+Cao%2C+Dahua+Lin%2C+and+Jiaqi+Wang%2E+2025%2E+Dispider%3A+Enabling+video+llms+with+active+real%2Dtime+interaction+via+disentangled+perception%2C)
34. Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang. 2016. Ntu rgb+d: A large scale dataset for 3d human activity analysis. Preprint, arXiv:1604.02808. [find](https://scholar.google.com/scholar?q=Amir+Shahroudy%2C+Jun+Liu%2C+Tian%2DTsong+Ng%2C+and+Gang+Wang%2E+2016%2E+Ntu+rgb%2Bd%3A+A+large+scale+dataset+for+3d+human+activity+analysis%2E+Preprint%2C+arXiv%3A1604%2E02808%2E)
35. Lianlei Shan, Shixian Luo, Zezhou Zhu, Yu Yuan, and Yong Wu. 2025. Cognitive memory in large language models. Preprint, arXiv:2504.02441. — [arXiv:2504.02441](https://arxiv.org/abs/2504.02441) [find](https://scholar.google.com/scholar?q=Lianlei+Shan%2C+Shixian+Luo%2C+Zezhou+Zhu%2C+Yu+Yuan%2C+and+Yong+Wu%2E+2025%2E+Cognitive+memory+in+large+language+models%2E+Preprint%2C+arXiv%3A2504%2E02441%2E+%E2%80%94+%5BarXiv%3A2504%2E02441%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2504%2E02441%29)
36. Junming Lin, Zheng Fang, Chi Chen, Zihao Wan, Fuwen Luo, Peng Li, Yang Liu, and Maosong Sun. 2024b. Streamingbench: Assessing the gap for mllms to achieve streaming video understanding. Preprint, arXiv:2411.03628. — [arXiv:2411.03628](https://arxiv.org/abs/2411.03628) [find](https://scholar.google.com/scholar?q=Junming+Lin%2C+Zheng+Fang%2C+Chi+Chen%2C+Zihao+Wan%2C+Fuwen+Luo%2C+Peng+Li%2C+Yang+Liu%2C+and+Maosong+Sun%2E+2024b%2E+Streamingbench%3A+Assessing+the+gap+for+mllms+to+achieve+streaming+video+understanding%2E+Preprint%2C+arXi)
37. Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, Zhuang Liu, Hu Xu, Hyunwoo J. Kim, Bilge Soran, Raghuraman Krishnamoorthi, Mohamed Elhoseiny, and Vikas Chandra. 2024. Longvu: Spatiotemporal adaptive c… — [arXiv:2410.17434](https://arxiv.org/abs/2410.17434) [find](https://scholar.google.com/scholar?q=Xiaoqian+Shen%2C+Yunyang+Xiong%2C+Changsheng+Zhao%2C+Lemeng+Wu%2C+Jun+Chen%2C+Chenchen+Zhu%2C+Zechun+Liu%2C+Fanyi+Xiao%2C+Balakrishnan+Varadarajan%2C+Florian+Bordes%2C+Zhuang+Liu%2C+Hu+Xu%2C+Hyunwoo+J%2E+Kim%2C+Bilge+Soran%2C+Ragh)
38. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024a. Llava-next: Improved reasoning, ocr, and world knowledge. [find](https://scholar.google.com/scholar?q=Haotian+Liu%2C+Chunyuan+Li%2C+Yuheng+Li%2C+Bo+Li%2C+Yuanhan+Zhang%2C+Sheng+Shen%2C+and+Yong+Jae+Lee%2E+2024a%2E+Llava%2Dnext%3A+Improved+reasoning%2C+ocr%2C+and+world+knowledge%2E)
39. Haoran Sun and Shaoning Zeng. 2025. Hierarchical memory for high-efficiency long-term reasoning in llm agents. Preprint, arXiv:2507.22925. — [arXiv:2507.22925](https://arxiv.org/abs/2507.22925) [find](https://scholar.google.com/scholar?q=Haoran+Sun+and+Shaoning+Zeng%2E+2025%2E+Hierarchical+memory+for+high%2Defficiency+long%2Dterm+reasoning+in+llm+agents%2E+Preprint%2C+arXiv%3A2507%2E22925%2E+%E2%80%94+%5BarXiv%3A2507%2E22925%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2507%2E22925%29)
40. Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. 2025. Dycoke: Dynamic compression of tokens for fast video large language models. Preprint, arXiv:2411.15024. — [arXiv:2411.15024](https://arxiv.org/abs/2411.15024) [find](https://scholar.google.com/scholar?q=Keda+Tao%2C+Can+Qin%2C+Haoxuan+You%2C+Yang+Sui%2C+and+Huan+Wang%2E+2025%2E+Dycoke%3A+Dynamic+compression+of+tokens+for+fast+video+large+language+models%2E+Preprint%2C+arXiv%3A2411%2E15024%2E+%E2%80%94+%5BarXiv%3A2411%2E15024%5D%28https%3A%2F%2Farxi)
41. Haibo Wang, Bo Feng, Zhengfeng Lai, Mingze Xu, Shiyu Li, Weifeng Ge, Afshin Dehghan, Meng Cao, and Ping Huang. 2025a. Streambridge: Turning your offline video large language model into a proactive streaming assistant. Preprint, arXiv:2505.05467. — [arXiv:2505.05467](https://arxiv.org/abs/2505.05467) [find](https://scholar.google.com/scholar?q=Haibo+Wang%2C+Bo+Feng%2C+Zhengfeng+Lai%2C+Mingze+Xu%2C+Shiyu+Li%2C+Weifeng+Ge%2C+Afshin+Dehghan%2C+Meng+Cao%2C+and+Ping+Huang%2E+2025a%2E+Streambridge%3A+Turning+your+offline+video+large+language+model+into+a+proactive+str)
42. Han Wang, Yuxiang Nie, Yongjie Ye, Deng GuanYu, Yanjie Wang, Shuai Li, Haiyang Yu, Jinghui Lu, and Can Huang. 2024a. Dynamic-vlm: Simple dynamic visual token compression for videollm. Preprint, arXiv:2412.09530. — [arXiv:2412.09530](https://arxiv.org/abs/2412.09530) [find](https://scholar.google.com/scholar?q=Han+Wang%2C+Yuxiang+Nie%2C+Yongjie+Ye%2C+Deng+GuanYu%2C+Yanjie+Wang%2C+Shuai+Li%2C+Haiyang+Yu%2C+Jinghui+Lu%2C+and+Can+Huang%2E+2024a%2E+Dynamic%2Dvlm%3A+Simple+dynamic+visual+token+compression+for+videollm%2E+Preprint%2C+arXiv%3A)
43. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024b. [Qwen2-vl: Enhancing vision-language model’s perception of the…](https://md.research.elamur.ai/free/en/papers/wang-2024-qwen2-vl-enhancing-vision/)
44. Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. 2025b. Videotree: Adaptive tree-based video representation for llm reasoning on long videos. Preprint, arXiv:2405.19209. [find](https://scholar.google.com/scholar?q=Ziyang+Wang%2C+Shoubin+Yu%2C+Elias+Stengel%2DEskin%2C+Jaehong+Yoon%2C+Feng+Cheng%2C+Gedas+Bertasius%2C+and+Mohit+Bansal%2E+2025b%2E+Videotree%3A+Adaptive+tree%2Dbased+video+representation+for+llm+reasoning+on+long+videos%2E+)
45. Haomiao Xiong, Zongxin Yang, Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Jiawen Zhu, and Huchuan Lu. 2025. Streaming video understanding and multi-round interaction with memory-enhanced knowledge. Preprint, arXiv:2501.13468. — [arXiv:2501.13468](https://arxiv.org/abs/2501.13468) [find](https://scholar.google.com/scholar?q=Haomiao+Xiong%2C+Zongxin+Yang%2C+Jiazuo+Yu%2C+Yunzhi+Zhuge%2C+Lu+Zhang%2C+Jiawen+Zhu%2C+and+Huchuan+Lu%2E+2025%2E+Streaming+video+understanding+and+multi%2Dround+interaction+with+memory%2Denhanced+knowledge%2E+Preprint%2C+ar)
46. Ruyi Xu, Guangxuan Xiao, Yukang Chen, Liuning He, Kelly Peng, Yao Lu, and Song Han. 2025. Stream-ingvlm: Real-time understanding for infinite video streams. Preprint, arXiv:2510.09608. [find](https://scholar.google.com/scholar?q=Ruyi+Xu%2C+Guangxuan+Xiao%2C+Yukang+Chen%2C+Liuning+He%2C+Kelly+Peng%2C+Yao+Lu%2C+and+Song+Han%2E+2025%2E+Stream%2Dingvlm%3A+Real%2Dtime+understanding+for+infinite+video+streams%2E+Preprint%2C+arXiv%3A2510%2E09608%2E)
47. Haolin Yang, Feilong Tang, Lingxiao Zhao, Xiang An, Ming Hu, Huifa Li, Xinlin Zhuang, Yifan Lu, Xiaofeng Zhang, Abdalla Swikir, Junjun He, Zongyuan Ge, and Imran Razzak. 2025a. Streamagent: Towards anticipatory agents for streaming video understanding. Preprint, arXiv:2508.01875. — [arXiv:2508.01875](https://arxiv.org/abs/2508.01875) [find](https://scholar.google.com/scholar?q=Haolin+Yang%2C+Feilong+Tang%2C+Lingxiao+Zhao%2C+Xiang+An%2C+Ming+Hu%2C+Huifa+Li%2C+Xinlin+Zhuang%2C+Yifan+Lu%2C+Xiaofeng+Zhang%2C+Abdalla+Swikir%2C+Junjun+He%2C+Zongyuan+Ge%2C+and+Imran+Razzak%2E+2025a%2E+Streamagent%3A+Towards+an)
48. Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. 2024. Visionzip: Longer is better but not necessary in vision language models. Preprint, arXiv:2412.04467. — [arXiv:2412.04467](https://arxiv.org/abs/2412.04467) [find](https://scholar.google.com/scholar?q=Senqiao+Yang%2C+Yukang+Chen%2C+Zhuotao+Tian%2C+Chengyao+Wang%2C+Jingyao+Li%2C+Bei+Yu%2C+and+Jiaya+Jia%2E+2024%2E+Visionzip%3A+Longer+is+better+but+not+necessary+in+vision+language+models%2E+Preprint%2C+arXiv%3A2412%2E04467%2E+%E2%80%94+)
49. Yanlai Yang, Zhuokai Zhao, Satya Narayan Shukla, Aashu Singh, Shlok Kumar Mishra, Lizhu Zhang, and Mengye Ren. 2025b. Streammem: Query-agnostic kv cache memory for streaming video understanding. Preprint, arXiv:2508.15717. — [arXiv:2508.15717](https://arxiv.org/abs/2508.15717) [find](https://scholar.google.com/scholar?q=Yanlai+Yang%2C+Zhuokai+Zhao%2C+Satya+Narayan+Shukla%2C+Aashu+Singh%2C+Shlok+Kumar+Mishra%2C+Lizhu+Zhang%2C+and+Mengye+Ren%2E+2025b%2E+Streammem%3A+Query%2Dagnostic+kv+cache+memory+for+streaming+video+understanding%2E+Prepr)
50. Linli Yao, Yicheng Li, Yuancheng Wei, Lei Li, Shuhuai Ren, Yuanxin Liu, Kun Ouyang, Lean Wang, Shicheng Li, Sida Li, Lingpeng Kong, Qi Liu, Yuanxing Zhang, and Xu Sun. 2025. Timechat-online: 80% visual tokens are naturally redundant in streaming videos. Preprint, arXiv:2504.17343. — [arXiv:2504.17343](https://arxiv.org/abs/2504.17343) [find](https://scholar.google.com/scholar?q=Linli+Yao%2C+Yicheng+Li%2C+Yuancheng+Wei%2C+Lei+Li%2C+Shuhuai+Ren%2C+Yuanxin+Liu%2C+Kun+Ouyang%2C+Lean+Wang%2C+Shicheng+Li%2C+Sida+Li%2C+Lingpeng+Kong%2C+Qi+Liu%2C+Yuanxing+Zhang%2C+and+Xu+Sun%2E+2025%2E+Timechat%2Donline%3A+80%25+visua)
51. Xiangyu Zeng, Kefan Qiu, Qingyu Zhang, Xinhao Li, Jing Wang, Jiaxin Li, Ziang Yan, Kun Tian, Meng Tian, Xinhai Zhao, Yi Wang, and Limin Wang. 2025. Streamforest: Efficient online video understanding with persistent event memory. Preprint, arXiv:2509.24871. — [arXiv:2509.24871](https://arxiv.org/abs/2509.24871) [find](https://scholar.google.com/scholar?q=Xiangyu+Zeng%2C+Kefan+Qiu%2C+Qingyu+Zhang%2C+Xinhao+Li%2C+Jing+Wang%2C+Jiaxin+Li%2C+Ziang+Yan%2C+Kun+Tian%2C+Meng+Tian%2C+Xinhai+Zhao%2C+Yi+Wang%2C+and+Limin+Wang%2E+2025%2E+Streamforest%3A+Efficient+online+video+understanding+w)
52. Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. 2024a. Flash-vstream: Memory-based real-time understanding for long video streams. Preprint, arXiv:2406.08085. — [arXiv:2406.08085](https://arxiv.org/abs/2406.08085) [find](https://scholar.google.com/scholar?q=Haoji+Zhang%2C+Yiqin+Wang%2C+Yansong+Tang%2C+Yong+Liu%2C+Jiashi+Feng%2C+Jifeng+Dai%2C+and+Xiaojie+Jin%2E+2024a%2E+Flash%2Dvstream%3A+Memory%2Dbased+real%2Dtime+understanding+for+long+video+streams%2E+Preprint%2C+arXiv%3A2406%2E08085)
53. Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. 2024b. Flash-vstream: Memory-based real-time understanding for long video streams. Preprint, arXiv:2406.08085. — [arXiv:2406.08085](https://arxiv.org/abs/2406.08085) [find](https://scholar.google.com/scholar?q=Haoji+Zhang%2C+Yiqin+Wang%2C+Yansong+Tang%2C+Yong+Liu%2C+Jiashi+Feng%2C+Jifeng+Dai%2C+and+Xiaojie+Jin%2E+2024b%2E+Flash%2Dvstream%3A+Memory%2Dbased+real%2Dtime+understanding+for+long+video+streams%2E+Preprint%2C+arXiv%3A2406%2E08085)
54. Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. 2024c. [Long context transfer from language to vision](https://md.research.elamur.ai/free/en/papers/zhang-2024-long-context-transfer-language-vision/). Preprint, arXiv:2406.16852.
55. Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. 2025. [Llava-video: Video instruction tuning with synthetic data](https://md.research.elamur.ai/free/en/papers/zhang-2024-llava-video-video-instruction-tuning/). Preprint, arXiv:2410.02713.

*Updated 2 November 2026*