---
title: "Long Context Transfer from Language to Vision — Zhang 2024"
description: "Current large multimodal models (LMMs) struggle with very long videos. 93 references, cited by 71 papers here."
authors: ["Peiyuan Zhang", "Kaichen Zhang", "Bo Li", "Guangtao Zeng", "Jingkang Yang", "Yuanhan Zhang", "Ziyue Wang", "Haoran Tan", "Chunyuan Li", "Ziwei Liu"]
year: 2024
venue: "arXiv:2406.16852"
source: "https://arxiv.org/abs/2406.16852"
doi: "10.48550/arxiv.2406.16852"
arxiv: "2406.16852"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2024-long-context-transfer-language-vision/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2024-long-context-transfer-language-vision/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) / Long Context Transfer from Language to Vision

# Long Context Transfer from Language to Vision

**Authors:** [Peiyuan Zhang](https://md.research.elamur.ai/free/en/authors/peiyuan-zhang/), [Kaichen Zhang](https://md.research.elamur.ai/free/en/authors/kaichen-zhang/), [Bo Li](https://md.research.elamur.ai/free/en/authors/bo-li/), [Guangtao Zeng](https://md.research.elamur.ai/free/en/authors/guangtao-zeng/), [Jingkang Yang](https://md.research.elamur.ai/free/en/authors/jingkang-yang/), [Yuanhan Zhang](https://md.research.elamur.ai/free/en/authors/yuanhan-zhang/), [Ziyue Wang](https://md.research.elamur.ai/free/en/authors/ziyue-wang/), [Haoran Tan](https://md.research.elamur.ai/free/en/authors/haoran-tan/), [Chunyuan Li](https://md.research.elamur.ai/free/en/authors/chunyuan-li/) and [Ziwei Liu](https://md.research.elamur.ai/free/en/authors/ziwei-liu/) · **Year:** 2024 · **Venue:** arXiv:2406.16852 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2406.16852) · [PDF](https://arxiv.org/pdf/2406.16852) · [DOI 10.48550/arxiv.2406.16852](https://doi.org/10.48550/arxiv.2406.16852) · [arXiv 2406.16852](https://arxiv.org/abs/2406.16852) · [OpenAlex](https://openalex.org/W4400024885)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [71](https://md.research.elamur.ai/free/en/papers/zhang-2024-long-context-transfer-language-vision/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Current large multimodal models (LMMs) struggle with very long videos. Each frame yields many visual tokens, and most work responds by compressing those tokens. This paper locates the bottleneck elsewhere: the limited context window of the underlying language model. The authors first lengthen the context of Qwen2-7B-Instruct using only text. They then align it with vision using short image data alone, with no long video–text pairs. They find that the extended text context carries over to the visual side, which they name "long context transfer."

A new encoding scheme, UniRes, treats a video as a long image made of frame grids, so images and videos share one representation. To measure how far visual context actually reaches, they build V-NIAH, a synthetic needle-retrieval benchmark for video.

The resulting model, LongVA, retrieves information across 2000 frames, more than 200K visual tokens. It reaches state-of-the-art results among 7B models on Video-MME and MLVU. This makes long-video understanding achievable on academic compute, and the code is open-sourced.

## Key points
- **Language-side extension:** Qwen2-7B-Instruct underwent continued pretraining at 224K context.
  - Data: 900M tokens from Slimpajama, with documents longer than 4096 tokens upsampled.
  - Settings: RoPE base raised to 1B, learning rate 1e-5, 1M-token batches, 1,000 steps.
  - Cost: about 2 days on 8×A100-80G, using FlashAttention-2 and Ring Attention, without LoRA.
- **UniRes encoding:**
  - Images are cut into 336×336 grids, up to a maximum of 49.
  - Each grid is encoded with CLIP-ViT-L-336px and passed through a 2-layer MLP, giving 576 features.
  - 2×2 average pooling reduces this to 144 tokens per grid.
  - Video frames are handled as successive grids.
- **V-NIAH:** a synthetic Visual Needle-In-A-Haystack benchmark, modeled on the text NIAH test, for probing effective visual context length. LongVA succeeds at retrieval over 2000 frames.
- **Image results:** compared with an AnyRes baseline, UniRes improves InfoVQA (49.4 vs 43.3) because high-resolution images get more grids. It loses ground on AI2D (70.7 vs 73.1) and ChartQA (70.4 vs 74.4), whose images are smaller.
- **Short-video results:** sampling more frames helps only modestly on short benchmarks such as NeXTQA and ActivityNetQA. A light DPO step on LLaVA-Hound-DPO (LongVA-DPO) improves scores on the GPT-judged VideoChatGPT and Video-DD benchmarks.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

## Cite

Cite as: Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li and Ziwei Liu. 2024. Long Context Transfer from Language to Vision. arXiv:2406.16852.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhang2024long,
  title = {{Long Context Transfer from Language to Vision}},
  author = {Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li and Ziwei Liu},
  year = {2024},
  doi = {10.48550/arxiv.2406.16852},
  eprint = {2406.16852},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.16852}
}
```

- [← previous Iterative Translation Refinement with Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2024-iterative-translation-refinement-large-language/)
- [next → Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming](https://md.research.elamur.ai/free/en/papers/xie-2024-mini-omni-language-models-hear/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/)

## References

93 references, 49 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Arize AI. Llm testneedleinahaystack, 2023. [find](https://scholar.google.com/scholar?q=Arize+AI%2E+Llm+testneedleinahaystack%2C+2023%2E)
2. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L Menick, Sebastian Borgeaud, Andy B… — [Flamingo: a Visual Language Model for Few-Shot Learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/)
3. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425–2433, 2015. [find](https://scholar.google.com/scholar?q=Stanislaw+Antol%2C+Aishwarya+Agrawal%2C+Jiasen+Lu%2C+Margaret+Mitchell%2C+Dhruv+Batra%2C+C+Lawrence+Zitnick%2C+and+Devi+Parikh%2E+Vqa%3A+Visual+question+answering%2E+In+Proceedings+of+the+IEEE+international+conference+)
4. Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo: An open-source framework for training large… — [OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models](https://md.research.elamur.ai/free/en/papers/awadalla-2023-openflamingo-open-source-framework-training/)
5. Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. [Longalign: A recipe for long context alignment of large language models](https://md.research.elamur.ai/free/en/papers/bai-2024-longalign-recipe-long-context/), 2024.
6. bloc97. ntkaware scaled rope allows llama models to have, 2023. [find](https://scholar.google.com/scholar?q=bloc97%2E+ntkaware+scaled+rope+allows+llama+models+to+have%2C+2023%2E)
7. bloc97. Ntk-aware scaled rope allows llama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation., 2024. [find](https://scholar.google.com/scholar?q=bloc97%2E+Ntk%2Daware+scaled+rope+allows+llama+models+to+have+extended+%288k%2B%29+context+size+without+any+fine%2Dtuning+and+minimal+perplexity+degradation%2E%2C+2024%2E)
8. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
9. Mu Cai, Jianwei Yang, Jianfeng Gao, and Yong Jae Lee. Matryoshka multimodal models, 2024. [find](https://scholar.google.com/scholar?q=Mu+Cai%2C+Jianwei+Yang%2C+Jianfeng+Gao%2C+and+Yong+Jae+Lee%2E+Matryoshka+multimodal+models%2C+2024%2E)
10. Cerebras. cerebras slimpajama-627b, 2023. [find](https://scholar.google.com/scholar?q=Cerebras%2E+cerebras+slimpajama%2D627b%2C+2023%2E)
11. Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. [An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models](https://md.research.elamur.ai/free/en/papers/chen-2024-image-worth-1-2-tokens/), 2024.
12. Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, Li Yuan, Yu Qiao, Dahua Lin, Feng Zhao, and Jiaqi Wang. Sharegpt4video: Improving video understanding and generation with better captions, 2024. [find](https://scholar.google.com/scholar?q=Lin+Chen%2C+Xilin+Wei%2C+Jinsong+Li%2C+Xiaoyi+Dong%2C+Pan+Zhang%2C+Yuhang+Zang%2C+Zehui+Chen%2C+Haodong+Duan%2C+Bin+Lin%2C+Zhenyu+Tang%2C+Li+Yuan%2C+Yu+Qiao%2C+Dahua+Lin%2C+Feng+Zhao%2C+and+Jiaqi+Wang%2E+Sharegpt4video%3A+Improving+)
13. Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595, 2023. [find](https://scholar.google.com/scholar?q=Shouyuan+Chen%2C+Sherman+Wong%2C+Liangjian+Chen%2C+and+Yuandong+Tian%2E+Extending+context+window+of+large+language+models+via+positional+interpolation%2E+arXiv+preprint+arXiv%3A2306%2E15595%2C+2023%2E)
14. Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. [Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks](https://md.research.elamur.ai/free/en/papers/chen-2023-internvl-scaling-up-vision/). arXiv preprint arXiv:2312.14238, 2023.
15. Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. [Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms](https://md.research.elamur.ai/free/en/papers/cheng-2024-videollama-2-advancing-spatial-temporal/), 2024.
16. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. [Generating long sequences with sparse transformers](https://md.research.elamur.ai/free/en/papers/child-2019-generating-long-sequences-sparse-transformers/), 2019.
17. Cohere. Introducing command r+: A scalable llm built for business, 2024. [find](https://scholar.google.com/scholar?q=Cohere%2E+Introducing+command+r%2B%3A+A+scalable+llm+built+for+business%2C+2024%2E)
18. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. [Instructblip: Towards general-purpose vision-language models with instruction tuning](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/), 2023.
19. Tri Dao. [Flashattention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/), 2023.
20. Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. [Longrope: Extending llm context window beyond 2 million tokens](https://md.research.elamur.ai/free/en/papers/ding-2024-longrope-extending-llm-context/). arXiv preprint arXiv:2402.13753, 2024.
21. Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. Video-mme: The first-ever comprehensive evaluation bench… — [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://md.research.elamur.ai/free/en/papers/fu-2024-video-mme-first-ever-comprehensive/)
22. Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. [Data engineering for scaling language models to 128k context](https://md.research.elamur.ai/free/en/papers/fu-2024-data-engineering-scaling-language-models/), 2024.
23. Kamradt Gregory. Llmtest needleinahaystack, 2024. [find](https://scholar.google.com/scholar?q=Kamradt+Gregory%2E+Llmtest+needleinahaystack%2C+2024%2E)
24. Madeleine Grunde-McLaughlin, Ranjay Krishna, and Maneesh Agrawala. Agqa: A benchmark for compositional spatio-temporal reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11287–11297, 2021. [find](https://scholar.google.com/scholar?q=Madeleine+Grunde%2DMcLaughlin%2C+Ranjay+Krishna%2C+and+Maneesh+Agrawala%2E+Agqa%3A+A+benchmark+for+compositional+spatio%2Dtemporal+reasoning%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Patte)
25. Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-lmm: Memory-augmented large multimodal model for long-term video understanding, 2024. [find](https://scholar.google.com/scholar?q=Bo+He%2C+Hengduo+Li%2C+Young+Kyun+Jang%2C+Menglin+Jia%2C+Xuefei+Cao%2C+Ashish+Shah%2C+Abhinav+Shrivastava%2C+and+Ser%2DNam+Lim%2E+Ma%2Dlmm%3A+Memory%2Daugmented+large+multimodal+model+for+long%2Dterm+video+understanding%2C+2024%2E)
26. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/), 2021.
27. Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. [Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models](https://md.research.elamur.ai/free/en/papers/jacobs-2023-deepspeed-ulysses-system-optimizations-enabling/), 2023.
28. Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio-temporal reasoning in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2758–2766, 2017. [find](https://scholar.google.com/scholar?q=Yunseok+Jang%2C+Yale+Song%2C+Youngjae+Yu%2C+Youngjin+Kim%2C+and+Gunhee+Kim%2E+Tgif%2Dqa%3A+Toward+spatio%2Dtemporal+reasoning+in+visual+question+answering%2E+In+Proceedings+of+the+IEEE+conference+on+computer+vision+and)
29. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
30. Peng Jin, Ryuichi Takanobu, Caiwan Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. arXiv preprint arXiv:2311.08046, 2023. — [arXiv:2311.08046](https://arxiv.org/abs/2311.08046) [find](https://scholar.google.com/scholar?q=Peng+Jin%2C+Ryuichi+Takanobu%2C+Caiwan+Zhang%2C+Xiaochun+Cao%2C+and+Li+Yuan%2E+Chat%2Dunivi%3A+Unified+visual+representation+empowers+large+language+models+with+image+and+video+understanding%2E+arXiv+preprint+arXiv%3A2)
31. Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding, 2024. [find](https://scholar.google.com/scholar?q=Peng+Jin%2C+Ryuichi+Takanobu%2C+Wancai+Zhang%2C+Xiaochun+Cao%2C+and+Li+Yuan%2E+Chat%2Dunivi%3A+Unified+visual+representation+empowers+large+language+models+with+image+and+video+understanding%2C+2024%2E)
32. Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. [A diagram is worth a dozen images](https://md.research.elamur.ai/free/en/papers/kembhavi-2016-diagram-worth-dozen-images/), 2016.
33. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
34. Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. [Obelics: An open web-scale filtered dataset of interleaved image-text documents](https://md.research.elamur.ai/free/en/papers/lauren-ccon-2023-obelics-open-web-scale-filtered/), 2023.
35. Bo Li, Hao Zhang, Kaichen Zhang, Dong Guo, Yuanhan Zhang, Renrui Zhang, Feng Li, Ziwei Liu, and Chunyuan Li. Llava-next: What else influences visual instruction tuning beyond data?, May 2024. [find](https://scholar.google.com/scholar?q=Bo+Li%2C+Hao+Zhang%2C+Kaichen+Zhang%2C+Dong+Guo%2C+Yuanhan+Zhang%2C+Renrui+Zhang%2C+Feng+Li%2C+Ziwei+Liu%2C+and+Chunyuan+Li%2E+Llava%2Dnext%3A+What+else+influences+visual+instruction+tuning+beyond+data%3F%2C+May+2024%2E)
36. Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger llms supercharge multimodal capabilities in the wild, May 2024. [find](https://scholar.google.com/scholar?q=Bo+Li%2C+Kaichen+Zhang%2C+Hao+Zhang%2C+Dong+Guo%2C+Renrui+Zhang%2C+Feng+Li%2C+Yuanhan+Zhang%2C+Ziwei+Liu%2C+and+Chunyuan+Li%2E+Llava%2Dnext%3A+Stronger+llms+supercharge+multimodal+capabilities+in+the+wild%2C+May+2024%2E)
37. Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. [Otter: A multi-modal model with in-context instruction tuning](https://md.research.elamur.ai/free/en/papers/li-2023-otter-multi-modal-model-context/), 2023.
38. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. [Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/), 2023.
39. KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. [Videochat: Chat-centric video understanding](https://md.research.elamur.ai/free/en/papers/li-2023-videochat-chat-centric-video-understanding/), 2024.
40. Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li, and Yang You. [Sequence parallelism: Long sequence training from system perspective](https://md.research.elamur.ai/free/en/papers/li-2023-sequence-parallelism-long-sequence-training/). In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volum…
41. Yanwei Li, Chengyao Wang, and Jiaya Jia. [Llama-vid: An image is worth 2 tokens in large language models](https://md.research.elamur.ai/free/en/papers/li-2023-llama-vid-image-worth-2/), 2023.
42. Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. [Video-llava: Learning united visual representation by alignment before projection](https://md.research.elamur.ai/free/en/papers/lin-2023-video-llava-learning-united-visual/), 2023.
43. Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. [Vila: On pre-training for visual language models](https://md.research.elamur.ai/free/en/papers/lin-2023-vila-pre-training-visual-language/), 2023.
44. Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv preprint, 2024. [find](https://scholar.google.com/scholar?q=Hao+Liu%2C+Wilson+Yan%2C+Matei+Zaharia%2C+and+Pieter+Abbeel%2E+World+model+on+million%2Dlength+video+and+language+with+ringattention%2E+arXiv+preprint%2C+2024%2E)
45. Hao Liu, Matei Zaharia, and Pieter Abbeel. [Ring attention with blockwise transformers for near-infinite context](https://md.research.elamur.ai/free/en/papers/liu-2023-ring-attention-blockwise-transformers-near/), 2023.
46. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. [Improved baselines with visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/), 2023.
47. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. [find](https://scholar.google.com/scholar?q=Haotian+Liu%2C+Chunyuan+Li%2C+Yuheng+Li%2C+Bo+Li%2C+Yuanhan+Zhang%2C+Sheng+Shen%2C+and+Yong+Jae+Lee%2E+Llava%2Dnext%3A+Improved+reasoning%2C+ocr%2C+and+world+knowledge%2C+January+2024%2E)
48. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/), 2023.
49. Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. St-llm: Large language models are effective temporal learners, 2024. [find](https://scholar.google.com/scholar?q=Ruyang+Liu%2C+Chen+Li%2C+Haoran+Tang%2C+Yixiao+Ge%2C+Ying+Shan%2C+and+Ge+Li%2E+St%2Dllm%3A+Large+language+models+are+effective+temporal+learners%2C+2024%2E)
50. Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. [Video-chatgpt: Towards detailed video understanding via large vision and language models](https://md.research.elamur.ai/free/en/papers/maaz-2023-video-chatgpt-towards-detailed-video/), 2023.
51. Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. [Egoschema: A diagnostic benchmark for very long-form video language understanding](https://md.research.elamur.ai/free/en/papers/mangalam-2023-egoschema-diagnostic-benchmark-very-long/), 2023.
52. Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. [Chartqa: A benchmark for question answering about charts with visual and logical reasoning](https://md.research.elamur.ai/free/en/papers/masry-2022-chartqa-benchmark-question-answering-about/), 2022.
53. Minesh Mathew, Dimosthenis Karatzas, R Manmatha, and CV Jawahar. [Docvqa: A dataset for vqa on document images](https://md.research.elamur.ai/free/en/papers/mathew-2020-docvqa-dataset-vqa-document-images/). corr abs/2007.00398 (2020). arXiv preprint arXiv:2007.00398, 2020.
54. Mistral. Mixtral 8x22b: Cheaper, better, faster, stronger, 2024. [find](https://scholar.google.com/scholar?q=Mistral%2E+Mixtral+8x22b%3A+Cheaper%2C+better%2C+faster%2C+stronger%2C+2024%2E)
55. OpenAI. Hello gpt-4o, 2024. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Hello+gpt%2D4o%2C+2024%2E)
56. Aitor Ormazabal, Che Zheng, Cyprien de Masson d’Autume, Dani Yogatama, Deyu Fu, Donovan Ong, Eric Chen, Eugenie Lamprecht, Hai Pham, Isaac Ong, et al. [Reka core, flash, and edge: A series of powerful multimodal language models](https://md.research.elamur.ai/free/en/papers/ormazabal-2024-reka-core-flash-edge-series/). arXiv preprint arXiv:2404.12387, 2024.
57. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations, 2023. [find](https://scholar.google.com/scholar?q=Bowen+Peng%2C+Jeffrey+Quesnelle%2C+Honglu+Fan%2C+and+Enrico+Shippole%2E+Yarn%3A+Efficient+context+window+extension+of+large+language+models%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2C+)
58. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/), 2021.
59. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models, 2020. [find](https://scholar.google.com/scholar?q=Samyam+Rajbhandari%2C+Jeff+Rasley%2C+Olatunji+Ruwase%2C+and+Yuxiong+He%2E+Zero%3A+Memory+optimizations+toward+training+trillion+parameter+models%2C+2020%2E)
60. Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding, 2024. [find](https://scholar.google.com/scholar?q=Shuhuai+Ren%2C+Linli+Yao%2C+Shicheng+Li%2C+Xu+Sun%2C+and+Lu+Hou%2E+Timechat%3A+A+time%2Dsensitive+multimodal+large+language+model+for+long+video+understanding%2C+2024%2E)
61. Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, J… — [Code Llama: Open Foundation Models for Code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/)
62. Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models, 2024. [find](https://scholar.google.com/scholar?q=Yuzhang+Shang%2C+Mu+Cai%2C+Bingxin+Xu%2C+Yong+Jae+Lee%2C+and+Yan+Yan%2E+Llava%2Dprumerge%3A+Adaptive+token+reduction+for+efficient+large+multimodal+models%2C+2024%2E)
63. Dingjie Song, Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, and Benyou Wang. Milebench: Benchmarking mllms in long context, 2024. [find](https://scholar.google.com/scholar?q=Dingjie+Song%2C+Shunian+Chen%2C+Guiming+Hardy+Chen%2C+Fei+Yu%2C+Xiang+Wan%2C+and+Benyou+Wang%2E+Milebench%3A+Benchmarking+mllms+in+long+context%2C+2024%2E)
64. Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, Yan Lu, Jenq-Neng Hwang, and Gaoang Wang. Moviechat: From dense token to sparse memory for long video understanding, 2024. [find](https://scholar.google.com/scholar?q=Enxin+Song%2C+Wenhao+Chai%2C+Guanhong+Wang%2C+Yucheng+Zhang%2C+Haoyang+Zhou%2C+Feiyang+Wu%2C+Haozhe+Chi%2C+Xun+Guo%2C+Tian+Ye%2C+Yanting+Zhang%2C+Yan+Lu%2C+Jenq%2DNeng+Hwang%2C+and+Gaoang+Wang%2E+Moviechat%3A+From+dense+token+to+s)
65. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/), 2023.
66. Gemini Team. [Gemini: A family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/), 2024.
67. PaLM Team. [Palm 2 technical report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/), 2023.
68. Qwen Team. Introducing qwen-vl, 2024. [find](https://scholar.google.com/scholar?q=Qwen+Team%2E+Introducing+qwen%2Dvl%2C+2024%2E)
69. Qwen2 Team. Qwen2 technical report, 2024. [find](https://scholar.google.com/scholar?q=Qwen2+Team%2E+Qwen2+technical+report%2C+2024%2E)
70. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/), 2023.
71. Hengyi Wang, Haizhou Shi, Shiwei Tan, Weiyi Qin, Wenyuan Wang, Tunyu Zhang, Akshay Nambi, Tanuja Ganu, and Hao Wang. [Multimodal needle in a haystack: Benchmarking long-context capability of multimodal large language models](https://md.research.elamur.ai/free/en/papers/wang-2025-multimodal-needle-haystack-benchmarking-long/), 2024.
72. Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. [Lvbench: An extreme long video understanding benchmark](https://md.research.elamur.ai/free/en/papers/wang-2024-lvbench-extreme-long-video-understanding/), 2024.
73. Weiyun Wang, Shuibo Zhang, Yiming Ren, Yuchen Duan, Tiantong Li, Shuo Liu, Mengkang Hu, Zhe Chen, Kaipeng Zhang, Lewei Lu, Xizhou Zhu, Ping Luo, Yu Qiao, Jifeng Dai, Wenqi Shao, and Wenhai Wang. Needle in a multimodal haystack, 2024. [find](https://scholar.google.com/scholar?q=Weiyun+Wang%2C+Shuibo+Zhang%2C+Yiming+Ren%2C+Yuchen+Duan%2C+Tiantong+Li%2C+Shuo+Liu%2C+Mengkang+Hu%2C+Zhe+Chen%2C+Kaipeng+Zhang%2C+Lewei+Lu%2C+Xizhou+Zhu%2C+Ping+Luo%2C+Yu+Qiao%2C+Jifeng+Dai%2C+Wenqi+Shao%2C+and+Wenhai+Wang%2E+Needl)
74. Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark for situated reasoning in real-world videos. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. [find](https://scholar.google.com/scholar?q=Bo+Wu%2C+Shoubin+Yu%2C+Zhenfang+Chen%2C+Joshua+B+Tenenbaum%2C+and+Chuang+Gan%2E+Star%3A+A+benchmark+for+situated+reasoning+in+real%2Dworld+videos%2E+In+Thirty%2Dfifth+Conference+on+Neural+Information+Processing+Systems)
75. xAI. Grok-1.5 vision preview, apr 2024. [find](https://scholar.google.com/scholar?q=xAI%2E+Grok%2D1%2E5+vision+preview%2C+apr+2024%2E)
76. Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9777–9786, 2021. [find](https://scholar.google.com/scholar?q=Junbin+Xiao%2C+Xindi+Shang%2C+Angela+Yao%2C+and+Tat%2DSeng+Chua%2E+Next%2Dqa%3A+Next+phase+of+question%2Danswering+to+explaining+temporal+actions%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+computer+vision+and+patt)
77. Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa:next phase of question-answering to explaining temporal actions, 2021. [find](https://scholar.google.com/scholar?q=Junbin+Xiao%2C+Xindi+Shang%2C+Angela+Yao%2C+and+Tat%2DSeng+Chua%2E+Next%2Dqa%3Anext+phase+of+question%2Danswering+to+explaining+temporal+actions%2C+2021%2E)
78. Binzhu Xie, Sicheng Zhang, Zitang Zhou, Bo Li, Yuanhan Zhang, Jack Hessel, Jingkang Yang, and Ziwei Liu. Funqa: Towards surprising video comprehension. GitHub repository, 2023. [find](https://scholar.google.com/scholar?q=Binzhu+Xie%2C+Sicheng+Zhang%2C+Zitang+Zhou%2C+Bo+Li%2C+Yuanhan+Zhang%2C+Jack+Hessel%2C+Jingkang+Yang%2C+and+Ziwei+Liu%2E+Funqa%3A+Towards+surprising+video+comprehension%2E+GitHub+repository%2C+2023%2E)
79. Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma.… — [Effective Long-Context Scaling of Foundation Models](https://md.research.elamur.ai/free/en/papers/xiong-2023-effective-long-context-scaling-foundation/)
80. Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference on Multimedia, pages 1645–1653, 2017. [find](https://scholar.google.com/scholar?q=Dejing+Xu%2C+Zhou+Zhao%2C+Jun+Xiao%2C+Fei+Wu%2C+Hanwang+Zhang%2C+Xiangnan+He%2C+and+Yueting+Zhuang%2E+Video+question+answering+via+gradually+refined+attention+over+appearance+and+motion%2E+In+Proceedings+of+the+25th+)
81. Li Xu, He Huang, and Jun Liu. Sutd-trafficqa: A question answering benchmark and an efficient network for video reasoning over traffic events. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9878–9888, 2021. [find](https://scholar.google.com/scholar?q=Li+Xu%2C+He+Huang%2C+and+Jun+Liu%2E+Sutd%2Dtrafficqa%3A+A+question+answering+benchmark+and+an+efficient+network+for+video+reasoning+over+traffic+events%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vis)
82. Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. [mplug-owl: Modularization empowers large language models with multimodality](https://md.research.elamur.ai/free/en/papers/ye-2023-mplug-owl-modularization-empowers-large/), 2024…
83. Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv:1910.01442, 2019. [find](https://scholar.google.com/scholar?q=Kexin+Yi%2C+Chuang+Gan%2C+Yunzhu+Li%2C+Pushmeet+Kohli%2C+Jiajun+Wu%2C+Antonio+Torralba%2C+and+Joshua+B+Tenenbaum%2E+Clevrer%3A+Collision+events+for+video+representation+and+reasoning%2E+arXiv+preprint+arXiv%3A1910%2E01442%2C)
84. Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. [Activitynet-qa: A dataset for understanding complex web videos via question answering](https://md.research.elamur.ai/free/en/papers/yu-2019-activitynet-qa-dataset-understanding-complex/). In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9127–9134, 2019.
85. Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. [Activitynet-qa: A dataset for understanding complex web videos via question answering](https://md.research.elamur.ai/free/en/papers/yu-2019-activitynet-qa-dataset-understanding-complex/), 2019.
86. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline… — [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/)
87. Hang Zhang, Xin Li, and Lidong Bing. [Video-llama: An instruction-tuned audio-visual language model for video understanding](https://md.research.elamur.ai/free/en/papers/zhang-2023-video-llama-instruction-tuned-audio/), 2023.
88. Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, and Yiming Yang. [Direct preference optimization of video large multimodal models from language model reward](https://md.research.elamur.ai/free/en/papers/zhang-2025-direct-preference-optimization-video-large/), 2024.
89. Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024. [find](https://scholar.google.com/scholar?q=Yuanhan+Zhang%2C+Bo+Li%2C+haotian+Liu%2C+Yong+jae+Lee%2C+Liangke+Gui%2C+Di+Fu%2C+Jiashi+Feng%2C+Ziwei+Liu%2C+and+Chunyuan+Li%2E+Llava%2Dnext%3A+A+strong+zero%2Dshot+video+understanding+model%2C+April+2024%2E)
90. Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding, 2024. [find](https://scholar.google.com/scholar?q=Junjie+Zhou%2C+Yan+Shu%2C+Bo+Zhao%2C+Boya+Wu%2C+Shitao+Xiao%2C+Xi+Yang%2C+Yongping+Xiong%2C+Bo+Zhang%2C+Tiejun+Huang%2C+and+Zheng+Liu%2E+Mlvu%3A+A+comprehensive+benchmark+for+multi%2Dtask+long+video+understanding%2C+2024%2E)
91. Xingyi Zhou, Anurag Arnab, Shyamal Buch, Shen Yan, Austin Myers, Xuehan Xiong, Arsha Nagrani, and Cordelia Schmid. Streaming dense video captioning, 2024. [find](https://scholar.google.com/scholar?q=Xingyi+Zhou%2C+Anurag+Arnab%2C+Shyamal+Buch%2C+Shen+Yan%2C+Austin+Myers%2C+Xuehan+Xiong%2C+Arsha+Nagrani%2C+and+Cordelia+Schmid%2E+Streaming+dense+video+captioning%2C+2024%2E)
92. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. [Minigpt-4: Enhancing vision-language understanding with advanced large language models](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/), 2023.
93. Zilin Zhu. Ring flash attention, 2024. [https://github.com/zhuzilin/ring-flash-attention](https://github.com/zhuzilin/ring-flash-attention). [↗](https://github.com/zhuzilin/ring-flash-attention)

*Updated 2 November 2026*