---
title: "EgoSchema — paper by Mangalam et al. (2023)"
description: "Video-language benchmarks often label themselves \"long-form,\" but many can be… 32% accuracy. 65 references, cited by 123 papers here."
authors: ["Karttikeya Mangalam", "Raiymbek Akshulakov", "Jitendra Malik"]
year: 2023
venue: "arXiv:2308.09126"
source: "https://arxiv.org/abs/2308.09126"
doi: "10.48550/arxiv.2308.09126"
arxiv: "2308.09126"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mangalam-2023-egoschema-diagnostic-benchmark-very-long/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mangalam-2023-egoschema-diagnostic-benchmark-very-long/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/) / EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding

# EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding

**Authors:** Karttikeya Mangalam, Raiymbek Akshulakov and [Jitendra Malik](https://md.research.elamur.ai/free/en/authors/jitendra-malik/) · **Year:** 2023 · **Venue:** [arXiv:2308.09126](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/) ([NeurIPS 2023](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/2023/)) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2308.09126) · [PDF](https://arxiv.org/pdf/2308.09126) · [DOI 10.48550/arxiv.2308.09126](https://doi.org/10.48550/arxiv.2308.09126) · [arXiv 2308.09126](https://arxiv.org/abs/2308.09126) · [OpenAlex](https://openalex.org/W7133198646)

**Citations:** 72 citations (OpenAlex, 2026-10-14) · [123](https://md.research.elamur.ai/free/en/papers/mangalam-2023-egoschema-diagnostic-benchmark-very-long/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Video-language benchmarks often label themselves "long-form," but many can be answered from a few seconds of footage. This paper addresses that gap with EgoSchema, a multiple-choice question-answering benchmark built from Ego4D egocentric recordings. Each item pairs a three-minute clip with a question and five candidate answers. The authors produced the items in four stages:

- selecting densely narrated videos;
- prompting an LLM in a chain over the narrations to write questions, correct answers and distractors;
- automatic and LLM-based filtering;
- two rounds of human curation.

To measure how much video a task truly requires, they define temporal certificate sets: the minimal footage a human needs to verify an annotation. Using this measure, EgoSchema's median certificate is about 100 seconds, several times longer than any of the other datasets they compare. In zero-shot tests, billion-parameter video-language models stay near 32% accuracy, barely above chance, while people reach roughly 76%. The benchmark therefore exposes a large gap in long-horizon video reasoning and offers a reusable way to judge how temporally demanding a dataset really is.

## Key points
- **Scale and format:** EgoSchema has over 5000 curated questions covering more than 250 hours of Ego4D video. Each item uses a 3-minute clip with 5 answer options, so random guessing scores 20%.
- **Taxonomy:** Certificate length sorts tasks into three bands:
  - short (~1 s);
  - long-form (~10 s);
  - very long-form (~100 s).
- **Certificate comparison:** The authors estimated certificate lengths for 15 datasets. EgoSchema's is about 5–5.7× longer than the runner-up (LVU) and 10–100× longer than the rest; the paper's exact figures vary between sections.
- **Zero-shot model results:**
  - InternVideo: best score, 32.1% at 90 frames;
  - mPLUG-Owl: 31.1% at 5 frames;
  - FrozenBiLM: 26.9%;
  - VIOLET: about 20%.
- **Human results:**
  - 75.0% with no constraints;
  - 76.2% when watching the video before reading the question;
  - about 67% under a 1-minute limit or with 1 fps video.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 811 works

## Cite

Cite as: Karttikeya Mangalam, Raiymbek Akshulakov and Jitendra Malik. 2023. EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding. arXiv:2308.09126.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{mangalam2023egoschema,
  title = {{EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding}},
  author = {Karttikeya Mangalam, Raiymbek Akshulakov and Jitendra Malik},
  year = {2023},
  doi = {10.48550/arxiv.2308.09126},
  eprint = {2308.09126},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.09126}
}
```

- [← previous MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action](https://md.research.elamur.ai/free/en/papers/yang-2023-mm-react-prompting-chatgpt-multimodal/)
- [next → Making Retrieval-Augmented Language Models Robust to Irrelevant Context](https://md.research.elamur.ai/free/en/papers/yoran-2023-making-retrieval-augmented-language-models/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/)

## References

65 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan. Youtube-8m: A large-scale video classification benchmark. arXiv preprint arXiv:1609.08675, 2016. — [arXiv:1609.08675](https://arxiv.org/abs/1609.08675) [find](https://scholar.google.com/scholar?q=Sami+Abu%2DEl%2DHaija%2C+Nisarg+Kothari%2C+Joonseok+Lee%2C+Paul+Natsev%2C+George+Toderici%2C+Balakrishnan+Varadarajan%2C+and+Sudheendra+Vijayanarasimhan%2E+Youtube%2D8m%3A+A+large%2Dscale+video+classification+benchmark%2E+arXi)
2. anthropic. Introducing claude, 2023. [find](https://scholar.google.com/scholar?q=anthropic%2E+Introducing+claude%2C+2023%2E)
3. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proc. ICCV, 2015. [find](https://scholar.google.com/scholar?q=Stanislaw+Antol%2C+Aishwarya+Agrawal%2C+Jiasen+Lu%2C+Margaret+Mitchell%2C+Dhruv+Batra%2C+C+Lawrence+Zitnick%2C+and+Devi+Parikh%2E+Vqa%3A+Visual+question+answering%2E+In+Proc%2E+ICCV%2C+2015%2E)
4. Sanjeev Arora and Boaz Barak. Computational complexity: a modern approach. Cambridge University Press, 2009. [find](https://scholar.google.com/scholar?q=Sanjeev+Arora+and+Boaz+Barak%2E+Computational+complexity%3A+a+modern+approach%2E+Cambridge+University+Press%2C+2009%2E)
5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 2020.
6. Chris Callison-Burch, Miles Osborne, and Philipp Koehn. [Re-evaluating the role of bleu in machine translation research](https://md.research.elamur.ai/free/en/papers/burch-2006-re-evaluating-role-bleu-machine/). In Conference of the European chapter of the association for computational linguistics, 2006.
7. Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017. [find](https://scholar.google.com/scholar?q=Joao+Carreira+and+Andrew+Zisserman%2E+Quo+vadis%2C+action+recognition%3F+a+new+model+and+the+kinetics+dataset%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+2017%2E)
8. Santiago Castro, Mahmoud Azab, Jonathan Stroud, Cristina Noujaim, Ruoyao Wang, Jia Deng, and Rada Mihalcea. Lifeqa: A real-life dataset for video question answering. In Proceedings of the Language Resources and Evaluation Conference, 2020. [find](https://scholar.google.com/scholar?q=Santiago+Castro%2C+Mahmoud+Azab%2C+Jonathan+Stroud%2C+Cristina+Noujaim%2C+Ruoyao+Wang%2C+Jia+Deng%2C+and+Rada+Mihalcea%2E+Lifeqa%3A+A+real%2Dlife+dataset+for+video+question+answering%2E+In+Proceedings+of+the+Language+Res)
9. Santiago Castro, Naihao Deng, Pingxuan Huang, Mihai Burzo, and Rada Mihalcea. Wildqa: In-the-wild video question answering. arXiv preprint arXiv:2209.06650, 2022. [find](https://scholar.google.com/scholar?q=Santiago+Castro%2C+Naihao+Deng%2C+Pingxuan+Huang%2C+Mihai+Burzo%2C+and+Rada+Mihalcea%2E+Wildqa%3A+In%2Dthe%2Dwild+video+question+answering%2E+arXiv+preprint+arXiv%3A2209%2E06650%2C+2022%2E)
10. Santiago Castro, Ruoyao Wang, Pingxuan Huang, Ian Stewart, Oana Ignat, Nan Liu, Jonathan C Stroud, and Rada Mihalcea. Fiber: Fill-in-the-blanks as a challenging video understanding evaluation framework. arXiv preprint arXiv:2104.04182, 2021. [find](https://scholar.google.com/scholar?q=Santiago+Castro%2C+Ruoyao+Wang%2C+Pingxuan+Huang%2C+Ian+Stewart%2C+Oana+Ignat%2C+Nan+Liu%2C+Jonathan+C+Stroud%2C+and+Rada+Mihalcea%2E+Fiber%3A+Fill%2Din%2Dthe%2Dblanks+as+a+challenging+video+understanding+evaluation+framewor)
11. Seongho Choi, Kyoung-Woon On, Yu-Jung Heo, Ahjeong Seo, Youwon Jang, Minsu Lee, and Byoung-Tak Zhang. Dramaqa: Character-centered video story understanding with hierarchical qa. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021. [find](https://scholar.google.com/scholar?q=Seongho+Choi%2C+Kyoung%2DWoon+On%2C+Yu%2DJung+Heo%2C+Ahjeong+Seo%2C+Youwon+Jang%2C+Minsu+Lee%2C+and+Byoung%2DTak+Zhang%2E+Dramaqa%3A+Character%2Dcentered+video+story+understanding+with+hierarchical+qa%2E+In+Proceedings+of+the+)
12. Ali Diba, Mohsen Fayyaz, Vivek Sharma, Manohar Paluri, Jürgen Gall, Rainer Stiefelhagen, and Luc Van Gool. Large scale holistic video understanding. In Proceedings of the European Conference on Computer Vision, 2020. [find](https://scholar.google.com/scholar?q=Ali+Diba%2C+Mohsen+Fayyaz%2C+Vivek+Sharma%2C+Manohar+Paluri%2C+J%C3%BCrgen+Gall%2C+Rainer+Stiefelhagen%2C+and+Luc+Van+Gool%2E+Large+scale+holistic+video+understanding%2E+In+Proceedings+of+the+European+Conference+on+Comput)
13. Zhiyuan Fang, Tejas Gokhale, Pratyay Banerjee, Chitta Baral, and Yezhou Yang. Video2commonsense: Generating commonsense descriptions to enrich video captioning. arXiv preprint arXiv:2003.05162, 2020. [find](https://scholar.google.com/scholar?q=Zhiyuan+Fang%2C+Tejas+Gokhale%2C+Pratyay+Banerjee%2C+Chitta+Baral%2C+and+Yezhou+Yang%2E+Video2commonsense%3A+Generating+commonsense+descriptions+to+enrich+video+captioning%2E+arXiv+preprint+arXiv%3A2003%2E05162%2C+2020%2E)
14. Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. An empirical study of end-to-end video-language transformers with masked visual modeling. arXiv preprint arXiv:2209.01540, 2022. [find](https://scholar.google.com/scholar?q=Tsu%2DJui+Fu%2C+Linjie+Li%2C+Zhe+Gan%2C+Kevin+Lin%2C+William+Yang+Wang%2C+Lijuan+Wang%2C+and+Zicheng+Liu%2E+An+empirical+study+of+end%2Dto%2Dend+video%2Dlanguage+transformers+with+masked+visual+modeling%2E+arXiv+preprint+arX)
15. Difei Gao, Ruiping Wang, Ziyi Bai, and Xilin Chen. Env-qa: a video question answering benchmark for comprehensive understanding of dynamic environments. In Proc. ICCV, 2021. [find](https://scholar.google.com/scholar?q=Difei+Gao%2C+Ruiping+Wang%2C+Ziyi+Bai%2C+and+Xilin+Chen%2E+Env%2Dqa%3A+a+video+question+answering+benchmark+for+comprehensive+understanding+of+dynamic+environments%2E+In+Proc%2E+ICCV%2C+2021%2E)
16. Noa Garcia, Mayu Otani, Chenhui Chu, and Yuta Nakashima. Knowit vqa: Answering knowledge-based questions about videos. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020. [find](https://scholar.google.com/scholar?q=Noa+Garcia%2C+Mayu+Otani%2C+Chenhui+Chu%2C+and+Yuta+Nakashima%2E+Knowit+vqa%3A+Answering+knowledge%2Dbased+questions+about+videos%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+2020%2E)
17. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 2021. [find](https://scholar.google.com/scholar?q=Timnit+Gebru%2C+Jamie+Morgenstern%2C+Briana+Vecchione%2C+Jennifer+Wortman+Vaughan%2C+Hanna+Wallach%2C+Hal+Daum%C3%A9+Iii%2C+and+Kate+Crawford%2E+Datasheets+for+datasets%2E+Communications+of+the+ACM%2C+2021%2E)
18. Google. An important next step on our ai journey, 2023. [find](https://scholar.google.com/scholar?q=Google%2E+An+important+next+step+on+our+ai+journey%2C+2023%2E)
19. Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE/C… [find](https://scholar.google.com/scholar?q=Raghav+Goyal%2C+Samira+Ebrahimi+Kahou%2C+Vincent+Michalski%2C+Joanna+Materzynska%2C+Susanne+Westphal%2C+Heuna+Kim%2C+Valentin+Haenel%2C+Ingo+Fruend%2C+Peter+Yianilos%2C+Moritz+Mueller%2DFreitag%2C+et+al%2E+The%22+something+som)
20. Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition… [find](https://scholar.google.com/scholar?q=Kristen+Grauman%2C+Andrew+Westbury%2C+Eugene+Byrne%2C+Zachary+Chavis%2C+Antonino+Furnari%2C+Rohit+Girdhar%2C+Jackson+Hamburger%2C+Hao+Jiang%2C+Miao+Liu%2C+Xingyu+Liu%2C+et+al%2E+Ego4d%3A+Around+the+world+in+3%2C000+hours+of+eg)
21. Madeleine Grunde-McLaughlin, Ranjay Krishna, and Maneesh Agrawala. Agqa: A benchmark for compositional spatio-temporal reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021. [find](https://scholar.google.com/scholar?q=Madeleine+Grunde%2DMcLaughlin%2C+Ranjay+Krishna%2C+and+Maneesh+Agrawala%2E+Agqa%3A+A+benchmark+for+compositional+spatio%2Dtemporal+reasoning%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Patte)
22. Madeleine Grunde-McLaughlin, Ranjay Krishna, and Maneesh Agrawala. Agqa 2.0: An updated benchmark for compositional spatio-temporal reasoning. arXiv preprint arXiv:2204.06105, 2022. [find](https://scholar.google.com/scholar?q=Madeleine+Grunde%2DMcLaughlin%2C+Ranjay+Krishna%2C+and+Maneesh+Agrawala%2E+Agqa+2%2E0%3A+An+updated+benchmark+for+compositional+spatio%2Dtemporal+reasoning%2E+arXiv+preprint+arXiv%3A2204%2E06105%2C+2022%2E)
23. Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et al. Ava: A video dataset of spatio-temporally localized atomic visual actions. In Proceedings of the IEEE/CVF Conference on Computer Vis… [find](https://scholar.google.com/scholar?q=Chunhui+Gu%2C+Chen+Sun%2C+David+A+Ross%2C+Carl+Vondrick%2C+Caroline+Pantofaru%2C+Yeqing+Li%2C+Sudheendra+Vijayanarasimhan%2C+George+Toderici%2C+Susanna+Ricco%2C+Rahul+Sukthankar%2C+et+al%2E+Ava%3A+A+video+dataset+of+spatio%2Dt)
24. Pranay Gupta and Manish Gupta. Newskvqa: Knowledge-aware news video question answering. In Advances in Knowledge Discovery and Data Mining. Springer. [find](https://scholar.google.com/scholar?q=Pranay+Gupta+and+Manish+Gupta%2E+Newskvqa%3A+Knowledge%2Daware+news+video+question+answering%2E+In+Advances+in+Knowledge+Discovery+and+Data+Mining%2E+Springer%2E)
25. Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017. [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Georgia+Gkioxari%2C+Piotr+Doll%C3%A1r%2C+and+Ross+Girshick%2E+Mask+r%2Dcnn%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+2017%2E)
26. Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin. Movienet: A holistic dataset for movie understanding. In Proceedings of the European Conference on Computer Vision, 2020. [find](https://scholar.google.com/scholar?q=Qingqiu+Huang%2C+Yu+Xiong%2C+Anyi+Rao%2C+Jiaze+Wang%2C+and+Dahua+Lin%2E+Movienet%3A+A+holistic+dataset+for+movie+understanding%2E+In+Proceedings+of+the+European+Conference+on+Computer+Vision%2C+2020%2E)
27. Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio-temporal reasoning in visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017. [find](https://scholar.google.com/scholar?q=Yunseok+Jang%2C+Yale+Song%2C+Youngjae+Yu%2C+Youngjin+Kim%2C+and+Gunhee+Kim%2E+Tgif%2Dqa%3A+Toward+spatio%2Dtemporal+reasoning+in+visual+question+answering%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision)
28. Yu-Gang Jiang, Jingen Liu, A Roshan Zamir, George Toderici, Ivan Laptev, Mubarak Shah, and Rahul Sukthankar. Thumos challenge: action recognition with a large number of classes (2014), 2014. [find](https://scholar.google.com/scholar?q=Yu%2DGang+Jiang%2C+Jingen+Liu%2C+A+Roshan+Zamir%2C+George+Toderici%2C+Ivan+Laptev%2C+Mubarak+Shah%2C+and+Rahul+Sukthankar%2E+Thumos+challenge%3A+action+recognition+with+a+large+number+of+classes+%282014%29%2C+2014%2E)
29. Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. [Tvqa: Localized, compositional video question answering](https://md.research.elamur.ai/free/en/papers/lei-2018-tvqa-localized-compositional-video-question/). arXiv preprint arXiv:1809.01696, 2018.
30. Jie Lei, Licheng Yu, Tamara L Berg, and Mohit Bansal. Tvqa+: Spatio-temporal grounding for video question answering. arXiv preprint arXiv:1904.11574, 2019. [find](https://scholar.google.com/scholar?q=Jie+Lei%2C+Licheng+Yu%2C+Tamara+L+Berg%2C+and+Mohit+Bansal%2E+Tvqa%2B%3A+Spatio%2Dtemporal+grounding+for+video+question+answering%2E+arXiv+preprint+arXiv%3A1904%2E11574%2C+2019%2E)
31. Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. Learning to answer questions in dynamic audio-visual scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. [find](https://scholar.google.com/scholar?q=Guangyao+Li%2C+Yake+Wei%2C+Yapeng+Tian%2C+Chenliang+Xu%2C+Ji%2DRong+Wen%2C+and+Di+Hu%2E+Learning+to+answer+questions+in+dynamic+audio%2Dvisual+scenarios%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+a)
32. Chin-Yew Lin. [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Proceedings of the annual meeting of the Association for Computational Linguistics, 2004.
33. Feng Liu, Tao Xiang, Timothy M Hospedales, Wankou Yang, and Changyin Sun. ivqa: Inverse visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018. [find](https://scholar.google.com/scholar?q=Feng+Liu%2C+Tao+Xiang%2C+Timothy+M+Hospedales%2C+Wankou+Yang%2C+and+Changyin+Sun%2E+ivqa%3A+Inverse+visual+question+answering%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition%2C)
34. Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In Proc. ICCV, 2019. [find](https://scholar.google.com/scholar?q=Antoine+Miech%2C+Dimitri+Zhukov%2C+Jean%2DBaptiste+Alayrac%2C+Makarand+Tapaswi%2C+Ivan+Laptev%2C+and+Josef+Sivic%2E+Howto100m%3A+Learning+a+text%2Dvideo+embedding+by+watching+hundred+million+narrated+video+clips%2E+In+Pr)
35. Ravi Teja Mullapudi, Steven Chen, Keyi Zhang, Deva Ramanan, and Kayvon Fatahalian. Online model distillation for efficient video inference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019. [find](https://scholar.google.com/scholar?q=Ravi+Teja+Mullapudi%2C+Steven+Chen%2C+Keyi+Zhang%2C+Deva+Ramanan%2C+and+Kayvon+Fatahalian%2E+Online+model+distillation+for+efficient+video+inference%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision)
36. Jonghwan Mun, Paul Hongsuck Seo, Ilchae Jung, and Bohyung Han. Marioqa: Answering questions by watching gameplay videos. In Proc. ICCV, 2017. [find](https://scholar.google.com/scholar?q=Jonghwan+Mun%2C+Paul+Hongsuck+Seo%2C+Ilchae+Jung%2C+and+Bohyung+Han%2E+Marioqa%3A+Answering+questions+by+watching+gameplay+videos%2E+In+Proc%2E+ICCV%2C+2017%2E)
37. OpenAI. Introducing chatgpt, 2022. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Introducing+chatgpt%2C+2022%2E)
38. R OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv, 2023.
39. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the annual meeting of the Association for Computational Linguistics, 2002.
40. Mahima Pushkarna, Andrew Zaldivar, and Oddur Kjartansson. Data cards: Purposeful and transparent dataset documentation for responsible ai. In 2022 ACM Conference on Fairness, Accountability, and Transparency, 2022. [find](https://scholar.google.com/scholar?q=Mahima+Pushkarna%2C+Andrew+Zaldivar%2C+and+Oddur+Kjartansson%2E+Data+cards%3A+Purposeful+and+transparent+dataset+documentation+for+responsible+ai%2E+In+2022+ACM+Conference+on+Fairness%2C+Accountability%2C+and+Trans)
41. Arka Sadhu, Kan Chen, and Ram Nevatia. Video question answering with phrases via semantic roles. arXiv preprint arXiv:2104.03762, 2021. [find](https://scholar.google.com/scholar?q=Arka+Sadhu%2C+Kan+Chen%2C+and+Ram+Nevatia%2E+Video+question+answering+with+phrases+via+semantic+roles%2E+arXiv+preprint+arXiv%3A2104%2E03762%2C+2021%2E)
42. Mike Zheng Shou, Stan Weixian Lei, Weiyao Wang, Deepti Ghadiyaram, and Matt Feiszli. Generic event boundary detection: A benchmark for event segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021. [find](https://scholar.google.com/scholar?q=Mike+Zheng+Shou%2C+Stan+Weixian+Lei%2C+Weiyao+Wang%2C+Deepti+Ghadiyaram%2C+and+Matt+Feiszli%2E+Generic+event+boundary+detection%3A+A+benchmark+for+event+segmentation%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+)
43. Mattia Soldan, Alejandro Pardo, Juan León Alcázar, Fabian Caba, Chen Zhao, Silvio Giancola, and Bernard Ghanem. Mad: A scalable dataset for language grounding in videos from movie audio descriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. [find](https://scholar.google.com/scholar?q=Mattia+Soldan%2C+Alejandro+Pardo%2C+Juan+Le%C3%B3n+Alc%C3%A1zar%2C+Fabian+Caba%2C+Chen+Zhao%2C+Silvio+Giancola%2C+and+Bernard+Ghanem%2E+Mad%3A+A+scalable+dataset+for+language+grounding+in+videos+from+movie+audio+descriptions%2E+)
44. Xiaomeng Song, Yucheng Shi, Xin Chen, and Yahong Han. Explore multi-step reasoning in video question answering. In Proceedings of the ACM international conference on Multimedia, 2018. [find](https://scholar.google.com/scholar?q=Xiaomeng+Song%2C+Yucheng+Shi%2C+Xin+Chen%2C+and+Yahong+Han%2E+Explore+multi%2Dstep+reasoning+in+video+question+answering%2E+In+Proceedings+of+the+ACM+international+conference+on+Multimedia%2C+2018%2E)
45. Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012. — [arXiv:1212.0402](https://arxiv.org/abs/1212.0402) [find](https://scholar.google.com/scholar?q=Khurram+Soomro%2C+Amir+Roshan+Zamir%2C+and+Mubarak+Shah%2E+Ucf101%3A+A+dataset+of+101+human+actions+classes+from+videos+in+the+wild%2E+arXiv+preprint+arXiv%3A1212%2E0402%2C+2012%2E+%E2%80%94+%5BarXiv%3A1212%2E0402%5D%28https%3A%2F%2Farxiv%2Eorg)
46. Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Movieqa: Understanding stories in movies through question-answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016. [find](https://scholar.google.com/scholar?q=Makarand+Tapaswi%2C+Yukun+Zhu%2C+Rainer+Stiefelhagen%2C+Antonio+Torralba%2C+Raquel+Urtasun%2C+and+Sanja+Fidler%2E+Movieqa%3A+Understanding+stories+in+movies+through+question%2Danswering%2E+In+Proceedings+of+the+IEEE%2FCV)
47. Paul Vicol, Makarand Tapaswi, Lluis Castrejon, and Sanja Fidler. Moviegraphs: Towards understanding human-centric situations from videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018. [find](https://scholar.google.com/scholar?q=Paul+Vicol%2C+Makarand+Tapaswi%2C+Lluis+Castrejon%2C+and+Sanja+Fidler%2E+Moviegraphs%3A+Towards+understanding+human%2Dcentric+situations+from+videos%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+a)
48. Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. [Internvideo: General video foundation models via generative and discriminative learning](https://md.research.elamur.ai/free/en/papers/wang-2022-internvideo-general-video-foundation/). arXiv preprint arXiv:2212.03191, 2022.
49. Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark for situated reasoning in real-world videos. In Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021. [find](https://scholar.google.com/scholar?q=Bo+Wu%2C+Shoubin+Yu%2C+Zhenfang+Chen%2C+Joshua+B+Tenenbaum%2C+and+Chuang+Gan%2E+Star%3A+A+benchmark+for+situated+reasoning+in+real%2Dworld+videos%2E+In+Conference+on+Neural+Information+Processing+Systems+Datasets+and)
50. Chao-Yuan Wu and Philipp Krahenbuhl. Towards long-form video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021. [find](https://scholar.google.com/scholar?q=Chao%2DYuan+Wu+and+Philipp+Krahenbuhl%2E+Towards+long%2Dform+video+understanding%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+2021%2E)
51. Tianran Wu, Noa Garcia, Mayu Otani, Chenhui Chu, Yuta Nakashima, and Haruo Takemura. Transferring domain-agnostic knowledge in video question answering. arXiv preprint arXiv:2110.13395, 2021. — [arXiv:2110.13395](https://arxiv.org/abs/2110.13395) [find](https://scholar.google.com/scholar?q=Tianran+Wu%2C+Noa+Garcia%2C+Mayu+Otani%2C+Chenhui+Chu%2C+Yuta+Nakashima%2C+and+Haruo+Takemura%2E+Transferring+domain%2Dagnostic+knowledge+in+video+question+answering%2E+arXiv+preprint+arXiv%3A2110%2E13395%2C+2021%2E+%E2%80%94+%5BarXiv)
52. Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021. [find](https://scholar.google.com/scholar?q=Junbin+Xiao%2C+Xindi+Shang%2C+Angela+Yao%2C+and+Tat%2DSeng+Chua%2E+Next%2Dqa%3A+Next+phase+of+question%2Danswering+to+explaining+temporal+actions%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Patt)
53. Yu Xiong, Qingqiu Huang, Lingfeng Guo, Hang Zhou, Bolei Zhou, and Dahua Lin. A graph-based framework to bridge movies and synopses. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019. [find](https://scholar.google.com/scholar?q=Yu+Xiong%2C+Qingqiu+Huang%2C+Lingfeng+Guo%2C+Hang+Zhou%2C+Bolei+Zhou%2C+and+Dahua+Lin%2E+A+graph%2Dbased+framework+to+bridge+movies+and+synopses%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pat)
54. Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the ACM international conference on Multimedia, pages 1645–1653, 2017. [find](https://scholar.google.com/scholar?q=Dejing+Xu%2C+Zhou+Zhao%2C+Jun+Xiao%2C+Fei+Wu%2C+Hanwang+Zhang%2C+Xiangnan+He%2C+and+Yueting+Zhuang%2E+Video+question+answering+via+gradually+refined+attention+over+appearance+and+motion%2E+In+Proceedings+of+the+ACM+i)
55. Li Xu, He Huang, and Jun Liu. Sutd-trafficqa: A question answering benchmark and an efficient network for video reasoning over traffic events. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021. [find](https://scholar.google.com/scholar?q=Li+Xu%2C+He+Huang%2C+and+Jun+Liu%2E+Sutd%2Dtrafficqa%3A+A+question+answering+benchmark+and+an+efficient+network+for+video+reasoning+over+traffic+events%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vis)
56. Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Learning to answer questions from millions of narrated videos. In Proc. ICCV, 2021. [find](https://scholar.google.com/scholar?q=Antoine+Yang%2C+Antoine+Miech%2C+Josef+Sivic%2C+Ivan+Laptev%2C+and+Cordelia+Schmid%2E+Just+ask%3A+Learning+to+answer+questions+from+millions+of+narrated+videos%2E+In+Proc%2E+ICCV%2C+2021%2E)
57. Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Zero-shot video question answering via frozen bidirectional language models. arXiv preprint arXiv:2206.08155, 2022. — [arXiv:2206.08155](https://arxiv.org/abs/2206.08155) [find](https://scholar.google.com/scholar?q=Antoine+Yang%2C+Antoine+Miech%2C+Josef+Sivic%2C+Ivan+Laptev%2C+and+Cordelia+Schmid%2E+Zero%2Dshot+video+question+answering+via+frozen+bidirectional+language+models%2E+arXiv+preprint+arXiv%3A2206%2E08155%2C+2022%2E+%E2%80%94+%5BarXiv)
58. Pinci Yang, Xin Wang, Xuguang Duan, Hong Chen, Runze Hou, Cong Jin, and Wenwu Zhu. Avqa: A dataset for audio-visual question answering on videos. In Proceedings of the ACM international conference on Multimedia, 2022. [find](https://scholar.google.com/scholar?q=Pinci+Yang%2C+Xin+Wang%2C+Xuguang+Duan%2C+Hong+Chen%2C+Runze+Hou%2C+Cong+Jin%2C+and+Wenwu+Zhu%2E+Avqa%3A+A+dataset+for+audio%2Dvisual+question+answering+on+videos%2E+In+Proceedings+of+the+ACM+international+conference+on+)
59. Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. [mplug-owl: Modularization empowers large language models with multimodality](https://md.research.elamur.ai/free/en/papers/ye-2023-mplug-owl-modularization-empowers-large/). arXiv preprint arXiv:2304.14178, 2023.
60. Yunan Ye, Zhou Zhao, Yimeng Li, Long Chen, Jun Xiao, and Yueting Zhuang. Video question answering via attribute-augmented attention network learning. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval, 2017. [find](https://scholar.google.com/scholar?q=Yunan+Ye%2C+Zhou+Zhao%2C+Yimeng+Li%2C+Long+Chen%2C+Jun+Xiao%2C+and+Yueting+Zhuang%2E+Video+question+answering+via+attribute%2Daugmented+attention+network+learning%2E+In+Proceedings+of+the+International+ACM+SIGIR+Conf)
61. Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv:1910.01442, 2019. [find](https://scholar.google.com/scholar?q=Kexin+Yi%2C+Chuang+Gan%2C+Yunzhu+Li%2C+Pushmeet+Kohli%2C+Jiajun+Wu%2C+Antonio+Torralba%2C+and+Joshua+B+Tenenbaum%2E+Clevrer%3A+Collision+events+for+video+representation+and+reasoning%2E+arXiv+preprint+arXiv%3A1910%2E01442%2C)
62. Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. [Activitynet-qa: A dataset for understanding complex web videos via question answering](https://md.research.elamur.ai/free/en/papers/yu-2019-activitynet-qa-dataset-understanding-complex/). In Proceedings of the AAAI Conference on Artificial Intelligence, 2019.
63. Heeseung Yun, Youngjae Yu, Wonsuk Yang, Kangil Lee, and Gunhee Kim. Pano-avqa: Grounded audio-visual question answering on 360deg videos. In Proc. ICCV, 2021. [find](https://scholar.google.com/scholar?q=Heeseung+Yun%2C+Youngjae+Yu%2C+Wonsuk+Yang%2C+Kangil+Lee%2C+and+Gunhee+Kim%2E+Pano%2Davqa%3A+Grounded+audio%2Dvisual+question+answering+on+360deg+videos%2E+In+Proc%2E+ICCV%2C+2021%2E)
64. Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Philippe Morency. Social-iq: A question answering benchmark for artificial social intelligence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019. [find](https://scholar.google.com/scholar?q=Amir+Zadeh%2C+Michael+Chan%2C+Paul+Pu+Liang%2C+Edmund+Tong%2C+and+Louis%2DPhilippe+Morency%2E+Social%2Diq%3A+A+question+answering+benchmark+for+artificial+social+intelligence%2E+In+Proceedings+of+the+IEEE%2FCVF+Conferenc)
65. Wentian Zhao, Seokhwan Kim, Ning Xu, and Hailin Jin. Video question answering on screencast tutorials. arXiv preprint arXiv:2008.00544, 2020. EgoSchema Datasheet Motivation — [arXiv:2008.00544](https://arxiv.org/abs/2008.00544) [find](https://scholar.google.com/scholar?q=Wentian+Zhao%2C+Seokhwan+Kim%2C+Ning+Xu%2C+and+Hailin+Jin%2E+Video+question+answering+on+screencast+tutorials%2E+arXiv+preprint+arXiv%3A2008%2E00544%2C+2020%2E+EgoSchema+Datasheet+Motivation+%E2%80%94+%5BarXiv%3A2008%2E00544%5D%28https%3A)

*Updated 2 November 2026*