---
title: "OmniBench — paper by Li et al. (2024) — summary and ideas"
description: "The authors call models with this three-way capacity omni-language models (OLMs)… 50% accuracy. 78 references, cited by 16 papers here."
authors: ["Yizhi Li", "Yinghao Ma", "Ge Zhang", "Ruibin Yuan", "Kang Zhu", "Hangyu Guo", "Yiming Liang", "Jiaheng Liu", "Zekun Wang", "Jian Yang", "Siwei Wu", "Xingwei Qu", "Jinjie Shi", "Xinyue Zhang", "Zhenzhu Yang", "Yidan Wen", "Yanghai Wang", "Shihao Li", "Zhaoxiang Zhang", "Zachary Liu", "Emmanouil Benetos", "Wenhao Huang", "Chenghua Lin"]
year: 2024
venue: "arXiv:2409.15272"
source: "https://arxiv.org/abs/2409.15272"
doi: "10.48550/arxiv.2409.15272"
arxiv: "2409.15272"
licence: "CC BY-SA 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/li-2024-omnibench-towards-future-universal-omni/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2024-omnibench-towards-future-universal-omni/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/) / OmniBench: Towards The Future of Universal Omni-Language Models

# OmniBench: Towards The Future of Universal Omni-Language Models

**Authors:** [Yizhi Li](https://md.research.elamur.ai/free/en/authors/yizhi-li/), [Yinghao Ma](https://md.research.elamur.ai/free/en/authors/yinghao-ma/), [Ge Zhang](https://md.research.elamur.ai/free/en/authors/ge-zhang/), [Ruibin Yuan](https://md.research.elamur.ai/free/en/authors/ruibin-yuan/), Kang Zhu, [Hangyu Guo](https://md.research.elamur.ai/free/en/authors/hangyu-guo/), [Yiming Liang](https://md.research.elamur.ai/free/en/authors/yiming-liang/), [Jiaheng Liu](https://md.research.elamur.ai/free/en/authors/jiaheng-liu/), [Zekun Wang](https://md.research.elamur.ai/free/en/authors/zekun-wang/), [Jian Yang](https://md.research.elamur.ai/free/en/authors/jian-yang/), Siwei Wu, [Xingwei Qu](https://md.research.elamur.ai/free/en/authors/xingwei-qu/), Jinjie Shi, Xinyue Zhang, Zhenzhu Yang, Yidan Wen, Yanghai Wang, Shihao Li, [Zhaoxiang Zhang](https://md.research.elamur.ai/free/en/authors/zhaoxiang-zhang/), Zachary Liu, Emmanouil Benetos, [Wenhao Huang](https://md.research.elamur.ai/free/en/authors/wenhao-huang/) and [Chenghua Lin](https://md.research.elamur.ai/free/en/authors/chenghua-lin/) · **Year:** 2024 · **Venue:** [arXiv:2409.15272](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/) ([NeurIPS 2024](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/2024/)) · **Licence:** [CC BY-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2409.15272) · [PDF](https://arxiv.org/pdf/2409.15272) · [DOI 10.48550/arxiv.2409.15272](https://doi.org/10.48550/arxiv.2409.15272) · [arXiv 2409.15272](https://arxiv.org/abs/2409.15272) · [OpenAlex](https://openalex.org/W4403885114)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [16](https://md.research.elamur.ai/free/en/papers/li-2024-omnibench-towards-future-universal-omni/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most multimodal LLMs are built and tested on pairs of modalities (image–text or audio–text), so it is unclear how well they handle an image, an audio clip and a text instruction together. The authors call models with this three-way capacity omni-language models (OLMs) and introduce OmniBench, a human-annotated multiple-choice benchmark. Each question requires evidence from both the picture and the sound, and annotators also wrote rationales for the correct answers. Question categories include object identification, causal story reasoning, plot prediction, counting, and reading text or symbols. The authors evaluated open-source OLMs, proprietary systems, VLMs and ALMs, sometimes replacing images or audio with captions or transcripts. Open models often failed to follow instructions under tri-modal input, and most systems scored under 50% accuracy even with textual substitutes. The team also releases OmniInstruct, a tri-modal instruction-tuning set. The work gives the field a concrete test of integrated audio-visual-text reasoning.

## Key points
- OmniBench (arXiv:2409.15272, NeurIPS 2025) targets simultaneous image + audio + text understanding. Its data, code and a live leaderboard are hosted at m-a-p.ai/OmniBench.
- OmniInstruct contains 84.5K training samples for supervised fine-tuning of OLMs on tri-modal contexts.
- In ablations that replace inputs with text, VLMs generally beat ALMs. Claude-3.5-Sonnet and GPT-4o lead the VLMs. In the pure-text setting, GPT-4o (0513) reaches 60.60%.
- Qwen2-Audio-7B-Instruct improves from 39.05% (transcript only) and 39.67% (caption only) to 47.02% when given both text descriptions. This suggests models reason better over combined text than over raw multimodal signals.
- Tested systems include the UnifiedIO2 family (1.1B/3.2B/6.8B), Gemini-1.5-Pro, Reka-core, and the ALMs LTU, Mu-LLaMA, MusiLingo, Audio-SALMONN, Qwen-Audio and Audio-Flamingo. Models show a bias toward speech audio, and humans do notably better on sound events and abstract concepts.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 539 works · 179,533 citations
- [National University of Singapore](https://md.research.elamur.ai/free/en/universities/national-university-of-singapore/) — University, Singapore, Singapore · 180 works · 34,304 citations
- [Google DeepMind](https://md.research.elamur.ai/free/en/companies/google-deepmind/) — Company, London, United Kingdom · 56 works · 78,880 citations
- [Alibaba Group](https://md.research.elamur.ai/free/en/companies/alibaba-group/) — Company, Hangzhou, China · 168 works · 15,386 citations
- [Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/government/chinese-academy-of-sciences/) — Government body, Beijing, China · 194 works · 23,906 citations
- [University of Michigan](https://md.research.elamur.ai/free/en/universities/university-of-michigan/) — University, Ann Arbor, United States · 70 works · 13,693 citations
- [University of Manchester](https://md.research.elamur.ai/free/en/universities/university-of-manchester/) — University, Manchester, United Kingdom · 87 works · 44,938 citations
- [Nanjing University](https://md.research.elamur.ai/free/en/universities/nanjing-university/) — University, Nanjing, China · 74 works · 7,814 citations
- [Queen Mary University of London](https://md.research.elamur.ai/free/en/universities/queen-mary-university-of-london/) — University, London, United Kingdom · 54 works · 3,542 citations
- [Purdue University West Lafayette](https://md.research.elamur.ai/free/en/universities/purdue-university-west-lafayette/) — University, West Lafayette, United States · 23 works · 1,417 citations
- [Nanjing University of Science and Technology](https://md.research.elamur.ai/free/en/universities/nanjing-university-of-science-and-technology/) — University, Nanjing, China · 14 works · 1,590 citations
- [Northwestern Polytechnical University](https://md.research.elamur.ai/free/en/universities/northwestern-polytechnical-university/) — University, Xi'an, China · 13 works · 67 citations
- [China University of Geosciences](https://md.research.elamur.ai/free/en/universities/china-university-of-geosciences/) — University, Wuhan, China · 2 works · 33 citations
- [China University of Geosciences (Beijing)](https://md.research.elamur.ai/free/en/universities/china-university-of-geosciences-beijing/) — University, Beijing, China · 2 works · 33 citations

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 1,552 works · [💭 Experimental and Cognitive Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/experimental-and-cognitive-psychology/) · 870 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Yizhi Li, Yinghao Ma, Ge Zhang, Ruibin Yuan, Kang Zhu, Hangyu Guo, Yiming Liang, Jiaheng Liu, Zekun Wang, Jian Yang, Siwei Wu, Xingwei Qu, Jinjie Shi, Xinyue Zhang, Zhenzhu Yang, Yidan Wen, Yanghai Wang, Shihao Li, Zhaoxiang Zhang, Zachary Liu, Emmanouil Benetos, Wenhao Huang and Chenghua Lin. 2024. OmniBench: Towards The Future of Universal Omni-Language Models. arXiv:2409.15272.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{li2024omnibench,
  title = {{OmniBench: Towards The Future of Universal Omni-Language Models}},
  author = {Yizhi Li, Yinghao Ma, Ge Zhang, Ruibin Yuan, Kang Zhu, Hangyu Guo, Yiming Liang, Jiaheng Liu, Zekun Wang, Jian Yang, Siwei Wu, Xingwei Qu, Jinjie Shi, Xinyue Zhang, Zhenzhu Yang, Yidan Wen, Yanghai Wang, Shihao Li, Zhaoxiang Zhang, Zachary Liu, Emmanouil Benetos, Wenhao Huang and Chenghua Lin},
  year = {2024},
  doi = {10.48550/arxiv.2409.15272},
  eprint = {2409.15272},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2409.15272}
}
```

- [← previous NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Queries](https://md.research.elamur.ai/free/en/papers/zhang-2024-naturalcodebench-examining-coding-performance-mismatch/)
- [next → Orion-14B: Open-source Multilingual Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2024-orion-14b-open-source/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/)

## References

78 references, 34 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774, 2023.
2. Anthropic. Claude 3.5 sonnet. [https://www.anthropic.com/news/claude-3-5-sonnet](https://www.anthropic.com/news/claude-3-5-sonnet), 2024. [↗](https://www.anthropic.com/news/claude-3-5-sonnet)
3. J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.[12966, 2023](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/).
4. H. Bu, J. Du, X. Na, B. Wu, and H. Zheng. Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline. In 2017 20th conference of the oriental chapter of the international coordinating committee on speech databases and speech I/O systems and assessment (O-COCOSDA), pages 1–5. [find](https://scholar.google.com/scholar?q=H%2E+Bu%2C+J%2E+Du%2C+X%2E+Na%2C+B%2E+Wu%2C+and+H%2E+Zheng%2E+Aishell%2D1%3A+An+open%2Dsource+mandarin+speech+corpus+and+a+speech+recognition+baseline%2E+In+2017+20th+conference+of+the+oriental+chapter+of+the+international+coord)
5. L. Chen, J. Li, X. Dong, P. Zhang, Y. Zang, Z. Chen, H. Duan, J. Wang, Y. Qiao, D. Lin, et al. [Are we on the right way for evaluating large vision-language models?](https://md.research.elamur.ai/free/en/papers/chen-2024-right-way-evaluating-large-vision/) arXiv preprint arXiv:2403.20330, 2024a.
6. S. Chen, Y. Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, and F. Wei. Beats: Audio pre-training with acoustic tokenizers. arXiv preprint arXiv:2212.09058, 2022. — [arXiv:2212.09058](https://arxiv.org/abs/2212.09058) [find](https://scholar.google.com/scholar?q=S%2E+Chen%2C+Y%2E+Wu%2C+C%2E+Wang%2C+S%2E+Liu%2C+D%2E+Tompkins%2C+Z%2E+Chen%2C+and+F%2E+Wei%2E+Beats%3A+Audio+pre%2Dtraining+with+acoustic+tokenizers%2E+arXiv+preprint+arXiv%3A2212%2E09058%2C+2022%2E+%E2%80%94+%5BarXiv%3A2212%2E09058%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs)
7. S. Chen, X. He, L. Guo, X. Zhu, W. Wang, J. Tang, and J. Liu. Valor: Vision-audio-language omni-perception pretraining model and dataset. arXiv preprint arXiv:2304.08345, 2023. — [arXiv:2304.08345](https://arxiv.org/abs/2304.08345) [find](https://scholar.google.com/scholar?q=S%2E+Chen%2C+X%2E+He%2C+L%2E+Guo%2C+X%2E+Zhu%2C+W%2E+Wang%2C+J%2E+Tang%2C+and+J%2E+Liu%2E+Valor%3A+Vision%2Daudio%2Dlanguage+omni%2Dperception+pretraining+model+and+dataset%2E+arXiv+preprint+arXiv%3A2304%2E08345%2C+2023%2E+%E2%80%94+%5BarXiv%3A2304%2E08345%5D%28ht)
8. Z. Chen, W. Wang, H. Tian, S. Ye, Z. Gao, E. Cui, W. Tong, K. Hu, J. Luo, Z. Ma, et al. [How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites](https://md.research.elamur.ai/free/en/papers/chen-2024-far-gpt-4v-closing/). arXiv preprint arXiv:2404.16821, 2024b.
9. H. Chernoff. A measure of asymptotic efficiency for tests of a hypothesis based on the sum of observations. The Annals of Mathematical Statistics, pages 493–507, 1952. [find](https://scholar.google.com/scholar?q=H%2E+Chernoff%2E+A+measure+of+asymptotic+efficiency+for+tests+of+a+hypothesis+based+on+the+sum+of+observations%2E+The+Annals+of+Mathematical+Statistics%2C+pages+493%E2%80%93507%2C+1952%2E)
10. Y. Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou. [Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models](https://md.research.elamur.ai/free/en/papers/chu-2023-qwen-audio-advancing-universal/). arXiv preprint arXiv:2311.07919, 2023a.
11. Y. Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou. [Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models](https://md.research.elamur.ai/free/en/papers/chu-2023-qwen-audio-advancing-universal/). arXiv preprint arXiv:2311.07919, 2023b.
12. Z. Deng, Y. Ma, Y. Liu, R. Guo, G. Zhang, W. Chen, W. Huang, and E. Benetos. [Musilingo: Bridging music and text with pre-trained language models for music captioning and query response](https://md.research.elamur.ai/free/en/papers/deng-2024-musilingo-bridging-music-text-pre/). In NAACL-HLT, 2023. URL [https://api.semanticscholar.org/CorpusID:262043691](https://api.semanticscholar.org/CorpusID:262043691). [↗](https://api.semanticscholar.org/CorpusID:262043691)
13. X. Dong, P. Zhang, Y. Zang, Y. Cao, B. Wang, L. Ouyang, S. Zhang, H. Duan, W. Zhang, Y. Li, et al. [Internlm-xcomposer2-4khd: A pioneering large vision-language model handling resolutions from 336 pixels to 4k hd](https://md.research.elamur.ai/free/en/papers/dong-2024-internlm-xcomposer2-4khd-pioneering/). arXiv preprint arXiv:2404.06512, 2024.
14. A. Dosovitskiy. [An image is worth 16x16 words: Transformers for image recognition at scale](https://md.research.elamur.ai/free/en/papers/dosovitskiy-2020-image-worth-16x16-words-transformers/). arXiv preprint arXiv:2010.11929, 2020.
15. K. Drossos, S. Lipping, and T. Virtanen. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 736–740. [find](https://scholar.google.com/scholar?q=K%2E+Drossos%2C+S%2E+Lipping%2C+and+T%2E+Virtanen%2E+Clotho%3A+An+audio+captioning+dataset%2E+In+ICASSP+2020%2D2020+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+%28ICASSP%29%2C+pages+736%E2%80%93740%2E)
16. J. Du, X. Na, X. Liu, and H. Bu. Aishell-2: Transforming mandarin asr research into industrial scale. arXiv preprint arXiv:1808.10583, 2018. [find](https://scholar.google.com/scholar?q=J%2E+Du%2C+X%2E+Na%2C+X%2E+Liu%2C+and+H%2E+Bu%2E+Aishell%2D2%3A+Transforming+mandarin+asr+research+into+industrial+scale%2E+arXiv+preprint+arXiv%3A1808%2E10583%2C+2018%2E)
17. C. Fu, H. Lin, Z. Long, Y. Shen, M. Zhao, Y. Zhang, S. Dong, X. Wang, D. Yin, L. Ma, et al. Vita: Towards open-source interactive omni multimodal llm. arXiv preprint arXiv:2408.05211, 2024. — [arXiv:2408.05211](https://arxiv.org/abs/2408.05211) [find](https://scholar.google.com/scholar?q=C%2E+Fu%2C+H%2E+Lin%2C+Z%2E+Long%2C+Y%2E+Shen%2C+M%2E+Zhao%2C+Y%2E+Zhang%2C+S%2E+Dong%2C+X%2E+Wang%2C+D%2E+Yin%2C+L%2E+Ma%2C+et+al%2E+Vita%3A+Towards+open%2Dsource+interactive+omni+multimodal+llm%2E+arXiv+preprint+arXiv%3A2408%2E05211%2C+2024%2E+%E2%80%94+%5BarXiv%3A2)
18. C. Fu, H. Lin, X. Wang, Y.-F. Zhang, Y. Shen, X. Liu, Y. Li, Z. Long, H. Gao, K. Li, et al. Vita-1.5: Towards gpt-4o level real-time vision and speech interaction. arXiv preprint arXiv:2501.01957, 2025. — [arXiv:2501.01957](https://arxiv.org/abs/2501.01957) [find](https://scholar.google.com/scholar?q=C%2E+Fu%2C+H%2E+Lin%2C+X%2E+Wang%2C+Y%2E%2DF%2E+Zhang%2C+Y%2E+Shen%2C+X%2E+Liu%2C+Y%2E+Li%2C+Z%2E+Long%2C+H%2E+Gao%2C+K%2E+Li%2C+et+al%2E+Vita%2D1%2E5%3A+Towards+gpt%2D4o+level+real%2Dtime+vision+and+speech+interaction%2E+arXiv+preprint+arXiv%3A2501%2E01957%2C+202)
19. J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 776–780. [find](https://scholar.google.com/scholar?q=J%2E+F%2E+Gemmeke%2C+D%2E+P%2E+Ellis%2C+D%2E+Freedman%2C+A%2E+Jansen%2C+W%2E+Lawrence%2C+R%2E+C%2E+Moore%2C+M%2E+Plakal%2C+and+M%2E+Ritter%2E+Audio+set%3A+An+ontology+and+human%2Dlabeled+dataset+for+audio+events%2E+In+2017+IEEE+international+co)
20. Y. Gong, J. Yu, and J. Glass. Vocalsound: A dataset for improving human vocal sounds recognition. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 151–155. [find](https://scholar.google.com/scholar?q=Y%2E+Gong%2C+J%2E+Yu%2C+and+J%2E+Glass%2E+Vocalsound%3A+A+dataset+for+improving+human+vocal+sounds+recognition%2E+In+ICASSP+2022%2D2022+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+%28ICASSP%29%2C)
21. Y. Gong, H. Luo, A. H. Liu, L. Karlinsky, and J. Glass. Listen, think, and understand. arXiv preprint arXiv:2305.10790, 2023. — [arXiv:2305.10790](https://arxiv.org/abs/2305.10790) [find](https://scholar.google.com/scholar?q=Y%2E+Gong%2C+H%2E+Luo%2C+A%2E+H%2E+Liu%2C+L%2E+Karlinsky%2C+and+J%2E+Glass%2E+Listen%2C+think%2C+and+understand%2E+arXiv+preprint+arXiv%3A2305%2E10790%2C+2023%2E+%E2%80%94+%5BarXiv%3A2305%2E10790%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2305%2E10790%29)
22. I. Guyon, J. Makhoul, R. Schwartz, and V. Vapnik. What size test set gives good error rate estimates? IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(1):52–64, 1998. [find](https://scholar.google.com/scholar?q=I%2E+Guyon%2C+J%2E+Makhoul%2C+R%2E+Schwartz%2C+and+V%2E+Vapnik%2E+What+size+test+set+gives+good+error+rate+estimates%3F+IEEE+Transactions+on+Pattern+Analysis+and+Machine+Intelligence%2C+20%281%29%3A52%E2%80%9364%2C+1998%2E)
23. E. E.-D. Hemdan, W. El-Shafai, and A. Sayed. Cr19: A framework for preliminary detection of covid-19 in cough audio signals using machine learning algorithms for automated medical diagnosis applications. Journal of Ambient Intelligence and Humanized Computing, 14(9):11715–11727, 2023. [find](https://scholar.google.com/scholar?q=E%2E+E%2E%2DD%2E+Hemdan%2C+W%2E+El%2DShafai%2C+and+A%2E+Sayed%2E+Cr19%3A+A+framework+for+preliminary+detection+of+covid%2D19+in+cough+audio+signals+using+machine+learning+algorithms+for+automated+medical+diagnosis+applicatio)
24. D. Jiang, X. He, H. Zeng, C. Wei, M. Ku, Q. Liu, and W. Chen. Mantis: Interleaved multi-image instruction tuning. arXiv preprint arXiv:2405.01483, 2024. [find](https://scholar.google.com/scholar?q=D%2E+Jiang%2C+X%2E+He%2C+H%2E+Zeng%2C+C%2E+Wei%2C+M%2E+Ku%2C+Q%2E+Liu%2C+and+W%2E+Chen%2E+Mantis%3A+Interleaved+multi%2Dimage+instruction+tuning%2E+arXiv+preprint+arXiv%3A2405%2E01483%2C+2024%2E)
25. Z. Kong, A. Goel, R. Badlani, W. Ping, R. Valle, and B. Catanzaro. Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities. arXiv preprint arXiv:2402.01831, 2024. — [arXiv:2402.01831](https://arxiv.org/abs/2402.01831) [find](https://scholar.google.com/scholar?q=Z%2E+Kong%2C+A%2E+Goel%2C+R%2E+Badlani%2C+W%2E+Ping%2C+R%2E+Valle%2C+and+B%2E+Catanzaro%2E+Audio+flamingo%3A+A+novel+audio+language+model+with+few%2Dshot+learning+and+dialogue+abilities%2E+arXiv+preprint+arXiv%3A2402%2E01831%2C+2024%2E+%E2%80%94+)
26. H. Laurençon, L. Tronchon, M. Cord, and V. Sanh. [What matters when building vision-language models?](https://md.research.elamur.ai/free/en/papers/lauren-ccon-2024-matters-building-vision-language-models/) arXiv preprint arXiv:2405.02246, 2024.
27. B. Li, Y. Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, Y. Li, Z. Liu, and C. Li. [Llava-onevision: Easy visual task transfer](https://md.research.elamur.ai/free/en/papers/li-2024-llava-onevision-easy-visual-task/). arXiv preprint arXiv:2408.03326, 2024.
28. G. Li, Y. Wei, Y. Tian, C. Xu, J.-R. Wen, and D. Hu. Learning to answer questions in dynamic audio-visual scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19108–19118, 2022. [find](https://scholar.google.com/scholar?q=G%2E+Li%2C+Y%2E+Wei%2C+Y%2E+Tian%2C+C%2E+Xu%2C+J%2E%2DR%2E+Wen%2C+and+D%2E+Hu%2E+Learning+to+answer+questions+in+dynamic+audio%2Dvisual+scenarios%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognitio)
29. J. Li and W. Lu. A survey on benchmarks of multimodal large language models. ArXiv, abs/2408.08632, 2024. URL [https://api.semanticscholar.org/CorpusID:271892136](https://api.semanticscholar.org/CorpusID:271892136). — [arXiv:2408.08632](https://arxiv.org/abs/2408.08632) [↗](https://api.semanticscholar.org/CorpusID:271892136)
30. J. Li, D. Li, S. Savarese, and S. Hoi. [Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/). In International conference on machine learning, pages 19730–19742. PMLR, 2023a.
31. Y. Li, R. Yuan, G. Zhang, Y. Ma, X. Chen, H. Yin, C. Xiao, C. Lin, A. Ragni, E. Benetos, et al. Mert: Acoustic music understanding model with large-scale self-supervised training. arXiv preprint arXiv:2306.00107, 2023b. — [arXiv:2306.00107](https://arxiv.org/abs/2306.00107) [find](https://scholar.google.com/scholar?q=Y%2E+Li%2C+R%2E+Yuan%2C+G%2E+Zhang%2C+Y%2E+Ma%2C+X%2E+Chen%2C+H%2E+Yin%2C+C%2E+Xiao%2C+C%2E+Lin%2C+A%2E+Ragni%2C+E%2E+Benetos%2C+et+al%2E+Mert%3A+Acoustic+music+understanding+model+with+large%2Dscale+self%2Dsupervised+training%2E+arXiv+preprint+arXiv)
32. Y. Li, J. Liu, T. Zhang, S. Chen, T. Li, Z. Li, L. Liu, L. Ming, G. Dong, D. Pan, et al. [Baichuan-omni-1.5 technical report](https://md.research.elamur.ai/free/en/papers/li-2025-baichuan-omni-1-5/). arXiv preprint arXiv:2501.15368, 2025.
33. J. Liang, I. Nolasco, B. Ghani, H. Phan, E. Benetos, and D. Stowell. Mind the domain gap: a systematic analysis on bioacoustic sound event detection. arXiv preprint arXiv:2403.18638, 2024a. — [arXiv:2403.18638](https://arxiv.org/abs/2403.18638) [find](https://scholar.google.com/scholar?q=J%2E+Liang%2C+I%2E+Nolasco%2C+B%2E+Ghani%2C+H%2E+Phan%2C+E%2E+Benetos%2C+and+D%2E+Stowell%2E+Mind+the+domain+gap%3A+a+systematic+analysis+on+bioacoustic+sound+event+detection%2E+arXiv+preprint+arXiv%3A2403%2E18638%2C+2024a%2E+%E2%80%94+%5BarXiv%3A2)
34. J. Liang, H. Phan, and E. Benetos. Learning from taxonomy: Multi-label few-shot classification for everyday sound recognition. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 771–775. [find](https://scholar.google.com/scholar?q=J%2E+Liang%2C+H%2E+Phan%2C+and+E%2E+Benetos%2E+Learning+from+taxonomy%3A+Multi%2Dlabel+few%2Dshot+classification+for+everyday+sound+recognition%2E+In+ICASSP+2024%2D2024+IEEE+International+Conference+on+Acoustics%2C+Speech+an)
35. F. Liu, T. Zhu, X. Wu, B. Yang, C. You, C. Wang, L. Lu, Z. Liu, Y. Zheng, X. Sun, et al. A medical multimodal large language model for future pandemics. NPJ Digital Medicine, 6(1):226, 2023a. [find](https://scholar.google.com/scholar?q=F%2E+Liu%2C+T%2E+Zhu%2C+X%2E+Wu%2C+B%2E+Yang%2C+C%2E+You%2C+C%2E+Wang%2C+L%2E+Lu%2C+Z%2E+Liu%2C+Y%2E+Zheng%2C+X%2E+Sun%2C+et+al%2E+A+medical+multimodal+large+language+model+for+future+pandemics%2E+NPJ+Digital+Medicine%2C+6%281%29%3A226%2C+2023a%2E)
36. H. Liu, C. Li, Y. Li, B. Li, Y. Zhang, S. Shen, and Y. J. Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024a. URL [https://llava-vl.github.io/blog/](https://llava-vl.github.io/blog/) 2024-01-30-llava-next/. [↗](https://llava-vl.github.io/blog/)
37. H. Liu, C. Li, Q. Wu, and Y. J. Lee. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). Advances in neural information processing systems, 36, 2024b.
38. S. Liu, A. S. Hussain, C. Sun, and Y. Shan. Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioning. arXiv preprint arXiv:2308.11276, 2023b. — [arXiv:2308.11276](https://arxiv.org/abs/2308.11276) [find](https://scholar.google.com/scholar?q=S%2E+Liu%2C+A%2E+S%2E+Hussain%2C+C%2E+Sun%2C+and+Y%2E+Shan%2E+Music+Understanding+LLaMA%3A+Advancing+Text%2Dto%2DMusic+Generation+with+Question+Answering+and+Captioning%2E+arXiv+preprint+arXiv%3A2308%2E11276%2C+2023b%2E+%E2%80%94+%5BarXiv%3A2308%2E)
39. X. Liu, Z. Dong, and P. Zhang. Tackling data bias in music-avqa: Crafting a balanced dataset for unbiased question-answering. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 4478–4487, 2024c. [find](https://scholar.google.com/scholar?q=X%2E+Liu%2C+Z%2E+Dong%2C+and+P%2E+Zhang%2E+Tackling+data+bias+in+music%2Davqa%3A+Crafting+a+balanced+dataset+for+unbiased+question%2Danswering%2E+In+Proceedings+of+the+IEEE%2FCVF+Winter+Conference+on+Applications+of+Comput)
40. Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu, et al. [Mmbench: Is your multi-modal model an all-around player?](https://md.research.elamur.ai/free/en/papers/liu-2023-mmbench-multi-modal-model-all/) arXiv preprint arXiv:2307.06281, 2023c.
41. Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. [Swin transformer: Hierarchical vision transformer using shifted windows](https://md.research.elamur.ai/free/en/papers/liu-2021-swin-transformer-hierarchical-vision-transformer/). In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021.
42. H. Lu, W. Liu, B. Zhang, B. Wang, K. Dong, B. Liu, J. Sun, T. Ren, Z. Li, Y. Sun, et al. [Deepseek-vl: towards real-world vision-language understanding](https://md.research.elamur.ai/free/en/papers/lu-2024-deepseek-vl-real-world/). arXiv preprint arXiv:2403.05525, 2024a.
43. J. Lu, C. Clark, S. Lee, Z. Zhang, S. Khosla, R. Marten, D. Hoiem, and A. Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26439–26455, 2024b. [find](https://scholar.google.com/scholar?q=J%2E+Lu%2C+C%2E+Clark%2C+S%2E+Lee%2C+Z%2E+Zhang%2C+S%2E+Khosla%2C+R%2E+Marten%2C+D%2E+Hoiem%2C+and+A%2E+Kembhavi%2E+Unified%2Dio+2%3A+Scaling+autoregressive+multimodal+models+with+vision+language+audio+and+action%2E+In+Proceedings+of+the+)
44. R. Luo, T.-E. Lin, H. Zhang, Y. Wu, X. Liu, M. Yang, Y. Li, L. Chen, J. Li, L. Zhang, et al. Openomni: Large language models pivot zero-shot omnimodal alignment across language with real-time self-aware emotional speech synthesis. arXiv preprint arXiv:2501.04561, 2025. — [arXiv:2501.04561](https://arxiv.org/abs/2501.04561) [find](https://scholar.google.com/scholar?q=R%2E+Luo%2C+T%2E%2DE%2E+Lin%2C+H%2E+Zhang%2C+Y%2E+Wu%2C+X%2E+Liu%2C+M%2E+Yang%2C+Y%2E+Li%2C+L%2E+Chen%2C+J%2E+Li%2C+L%2E+Zhang%2C+et+al%2E+Openomni%3A+Large+language+models+pivot+zero%2Dshot+omnimodal+alignment+across+language+with+real%2Dtime+self%2Dawa)
45. Y. Ma, A. Øland, A. Ragni, B. M. Del Sette, C. Saitis, C. Donahue, C. Lin, C. Plachouras, E. Benetos, E. Quinton, et al. Foundation models for music: A survey. arXiv preprint arXiv:2408.14340, 2024. — [arXiv:2408.14340](https://arxiv.org/abs/2408.14340) [find](https://scholar.google.com/scholar?q=Y%2E+Ma%2C+A%2E+%C3%98land%2C+A%2E+Ragni%2C+B%2E+M%2E+Del+Sette%2C+C%2E+Saitis%2C+C%2E+Donahue%2C+C%2E+Lin%2C+C%2E+Plachouras%2C+E%2E+Benetos%2C+E%2E+Quinton%2C+et+al%2E+Foundation+models+for+music%3A+A+survey%2E+arXiv+preprint+arXiv%3A2408%2E14340%2C+2024%2E+%E2%80%94)
46. Y. Ma, S. Li, J. Yu, E. Benetos, and A. Maezawa. Cmi-bench: A comprehensive benchmark for evaluating music instruction following. In Proceedings of the 26th International Society for Music Information Retrieval Conference, ISMIR 2025, Daejeon, Korea, and Online, Sept 22-26, 2025, 2025. [find](https://scholar.google.com/scholar?q=Y%2E+Ma%2C+S%2E+Li%2C+J%2E+Yu%2C+E%2E+Benetos%2C+and+A%2E+Maezawa%2E+Cmi%2Dbench%3A+A+comprehensive+benchmark+for+evaluating+music+instruction+following%2E+In+Proceedings+of+the+26th+International+Society+for+Music+Information)
47. B. Meskó. The impact of multimodal large language models on health care’s future. Journal of medical Internet research, 25:e52865, 2023. [find](https://scholar.google.com/scholar?q=B%2E+Mesk%C3%B3%2E+The+impact+of+multimodal+large+language+models+on+health+care%E2%80%99s+future%2E+Journal+of+medical+Internet+research%2C+25%3Ae52865%2C+2023%2E)
48. A. Ormazabal, C. Zheng, C. d. M. d’Autume, D. Yogatama, D. Fu, D. Ong, E. Chen, E. Lamprecht, H. Pham, I. Ong, et al. [Reka core, flash, and edge: A series of powerful multimodal language models](https://md.research.elamur.ai/free/en/papers/ormazabal-2024-reka-core-flash-edge-series/). arXiv preprint arXiv:2404.12387, 2024.
49. V. Panayotov, G. Chen, D. Povey, and S. Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210. [find](https://scholar.google.com/scholar?q=V%2E+Panayotov%2C+G%2E+Chen%2C+D%2E+Povey%2C+and+S%2E+Khudanpur%2E+Librispeech%3A+an+asr+corpus+based+on+public+domain+audio+books%2E+In+2015+IEEE+international+conference+on+acoustics%2C+speech+and+signal+processing+%28ICAS)
50. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). In International conference on machine learning, pages 8748–8763. PMLR, 2021.
51. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever. [Robust speech recognition via large-scale weak supervision](https://md.research.elamur.ai/free/en/papers/radford-2022-robust-speech-recognition-large-scale/). arxiv 2022. arXiv preprint arXiv:2212.04356, 10, 2022.
52. H. Su, W. Qi, J. Chen, C. Yang, J. Sandoval, and M. A. Laribi. Recent advancements in multimodal human–robot interaction. Frontiers in Neurorobotics, 17:1084000, 2023. [find](https://scholar.google.com/scholar?q=H%2E+Su%2C+W%2E+Qi%2C+J%2E+Chen%2C+C%2E+Yang%2C+J%2E+Sandoval%2C+and+M%2E+A%2E+Laribi%2E+Recent+advancements+in+multimodal+human%E2%80%93robot+interaction%2E+Frontiers+in+Neurorobotics%2C+17%3A1084000%2C+2023%2E)
53. G. Sun, W. Yu, C. Tang, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, Y. Wang, and C. Zhang. video-salmonn: Speech-enhanced audio-visual large language models. arXiv preprint arXiv:2406.15704, 2024. — [arXiv:2406.15704](https://arxiv.org/abs/2406.15704) [find](https://scholar.google.com/scholar?q=G%2E+Sun%2C+W%2E+Yu%2C+C%2E+Tang%2C+X%2E+Chen%2C+T%2E+Tan%2C+W%2E+Li%2C+L%2E+Lu%2C+Z%2E+Ma%2C+Y%2E+Wang%2C+and+C%2E+Zhang%2E+video%2Dsalmonn%3A+Speech%2Denhanced+audio%2Dvisual+large+language+models%2E+arXiv+preprint+arXiv%3A2406%2E15704%2C+2024%2E+%E2%80%94+%5BarXiv%3A)
54. H. Sun, X. Liu, K. Xu, J. Miao, and Q. Luo. Emergency vehicles audio detection and localization in autonomous driving. arXiv preprint arXiv:2109.14797, 2021. — [arXiv:2109.14797](https://arxiv.org/abs/2109.14797) [find](https://scholar.google.com/scholar?q=H%2E+Sun%2C+X%2E+Liu%2C+K%2E+Xu%2C+J%2E+Miao%2C+and+Q%2E+Luo%2E+Emergency+vehicles+audio+detection+and+localization+in+autonomous+driving%2E+arXiv+preprint+arXiv%3A2109%2E14797%2C+2021%2E+%E2%80%94+%5BarXiv%3A2109%2E14797%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs)
55. C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang. [Salmonn: Towards generic hearing abilities for large language models](https://md.research.elamur.ai/free/en/papers/tang-2023-salmonn-towards-generic-hearing-abilities/). arXiv preprint arXiv:2310.13289, 2023.
56. G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805, 2023.
57. A. Terenzi, N. Ortolani, I. Nolasco, E. Benetos, and S. Cecchi. Comparison of feature extraction methods for sound-based classification of honey bee activity. IEEE/ACM transactions on audio, speech, and language processing, 30:112–122, 2021. [find](https://scholar.google.com/scholar?q=A%2E+Terenzi%2C+N%2E+Ortolani%2C+I%2E+Nolasco%2C+E%2E+Benetos%2C+and+S%2E+Cecchi%2E+Comparison+of+feature+extraction+methods+for+sound%2Dbased+classification+of+honey+bee+activity%2E+IEEE%2FACM+transactions+on+audio%2C+speech%2C+a)
58. S. Tong, E. Brown, P. Wu, S. Woo, M. Middepogu, S. C. Akula, J. Yang, S. Yang, A. Iyer, X. Pan, A. Wang, R. Fergus, Y. LeCun, and S. Xie. [Cambrian-1: A fully open, vision-centric exploration of multimodal llms](https://md.research.elamur.ai/free/en/papers/tong-2024-cambrian-1-fully-open-vision/), 2024.
59. H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou. [Training data-efficient image transformers & distillation through attention](https://md.research.elamur.ai/free/en/papers/touvron-2020-training-data-efficient-image-transformers/). arxiv 2020. arXiv preprint arXiv:2012.12877, 2(3), 2020.
60. C. Wang, M. Liao, Z. Huang, J. Lu, J. Wu, Y. Liu, C. Zong, and J. Zhang. [Blsp: Bootstrapping language-speech pre-training via behavior alignment of continuation writing](https://md.research.elamur.ai/free/en/papers/wang-2023-blsp-bootstrapping-language-speech-pre/). arXiv preprint arXiv:2309.00916, 2023a.
61. P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, Y. Fan, K. Dang, M. Du, X. Ren, R. Men, D. Liu, C. Zhou, J. Zhou, and J. Lin. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution, 2024a. URL — [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://md.research.elamur.ai/free/en/papers/wang-2024-qwen2-vl-enhancing-vision/) [https://arxiv.org/](https://arxiv.org/) abs/2409.12191. [↗](https://arxiv.org/)
62. W. Wang, Q. Lv, W. Yu, W. Hong, J. Qi, Y. Wang, J. Ji, Z. Yang, L. Zhao, X. Song, J. Xu, B. Xu, J. Li, Y. Dong, M. Ding, and J. Tang. [Cogvlm: Visual expert for pretrained language models](https://md.research.elamur.ai/free/en/papers/wang-2023-cogvlm-visual-expert-pretrained/), 2023b.
63. Z. Wang, K. Zhu, C. Xu, W. Zhou, J. Liu, Y. Zhang, J. Wang, N. Shi, S. Li, Y. Li, H. Que, Z. Zhang, Y. Zhang, G. Zhang, K. Xu, J. Fu, and W. Huang. Mio: A foundation model on multimodal tokens, 2024b. URL https://arxiv.org/abs/2409.17692. [↗](https://arxiv.org/abs/2409.17692)
64. J. Wu, Y. Gaur, Z. Chen, L. Zhou, Y. Zhu, T. Wang, J. Li, S. Liu, B. Ren, L. Liu, et al. On decoder-only architecture for speech-to-text and large language model integration. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1–8. [find](https://scholar.google.com/scholar?q=J%2E+Wu%2C+Y%2E+Gaur%2C+Z%2E+Chen%2C+L%2E+Zhou%2C+Y%2E+Zhu%2C+T%2E+Wang%2C+J%2E+Li%2C+S%2E+Liu%2C+B%2E+Ren%2C+L%2E+Liu%2C+et+al%2E+On+decoder%2Donly+architecture+for+speech%2Dto%2Dtext+and+large+language+model+integration%2E+In+2023+IEEE+Automatic+Sp)
65. S. Wu, Y. Li, K. Zhu, G. Zhang, Y. Liang, K. Ma, C. Xiao, H. Zhang, B. Yang, W. Chen, et al. Scimmir: Benchmarking scientific multi-modal information retrieval. arXiv preprint arXiv:2401.13478, 2024a. [find](https://scholar.google.com/scholar?q=S%2E+Wu%2C+Y%2E+Li%2C+K%2E+Zhu%2C+G%2E+Zhang%2C+Y%2E+Liang%2C+K%2E+Ma%2C+C%2E+Xiao%2C+H%2E+Zhang%2C+B%2E+Yang%2C+W%2E+Chen%2C+et+al%2E+Scimmir%3A+Benchmarking+scientific+multi%2Dmodal+information+retrieval%2E+arXiv+preprint+arXiv%3A2401%2E13478%2C+2024a%2E)
66. S. Wu, K. Zhu, Y. Bai, Y. Liang, Y. Li, H. Wu, J. Liu, R. Liu, X. Qu, X. Cheng, et al. Mmra: A benchmark for multi-granularity multi-image relational association. arXiv preprint arXiv:2407.17379, 2024b. — [arXiv:2407.17379](https://arxiv.org/abs/2407.17379) [find](https://scholar.google.com/scholar?q=S%2E+Wu%2C+K%2E+Zhu%2C+Y%2E+Bai%2C+Y%2E+Liang%2C+Y%2E+Li%2C+H%2E+Wu%2C+J%2E+Liu%2C+R%2E+Liu%2C+X%2E+Qu%2C+X%2E+Cheng%2C+et+al%2E+Mmra%3A+A+benchmark+for+multi%2Dgranularity+multi%2Dimage+relational+association%2E+arXiv+preprint+arXiv%3A2407%2E17379%2C+2024)
67. Y. Wu, K. Chen, T. Zhang, Y. Hui, T. Berg-Kirkpatrick, and S. Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. [find](https://scholar.google.com/scholar?q=Y%2E+Wu%2C+K%2E+Chen%2C+T%2E+Zhang%2C+Y%2E+Hui%2C+T%2E+Berg%2DKirkpatrick%2C+and+S%2E+Dubnov%2E+Large%2Dscale+contrastive+language%2Daudio+pretraining+with+feature+fusion+and+keyword%2Dto%2Dcaption+augmentation%2E+In+ICASSP+2023%2D2023+IE)
68. D. Xu, Z. Zhao, J. Xiao, F. Wu, H. Zhang, X. He, and Y. Zhuang. Video question answering via gradually refined attention over appearance and motion. In ACM Multimedia, 2017. [find](https://scholar.google.com/scholar?q=D%2E+Xu%2C+Z%2E+Zhao%2C+J%2E+Xiao%2C+F%2E+Wu%2C+H%2E+Zhang%2C+X%2E+He%2C+and+Y%2E+Zhuang%2E+Video+question+answering+via+gradually+refined+attention+over+appearance+and+motion%2E+In+ACM+Multimedia%2C+2017%2E)
69. J. Xu, Z. Guo, J. He, H. Hu, T. He, S. Bai, K. Chen, J. Wang, Y. Fan, K. Dang, B. Zhang, X. Wang, Y. Chu, and J. Lin. [Qwen2.5-omni technical report](https://md.research.elamur.ai/free/en/papers/xu-2025-qwen2-5-omni-technical/), 2025. URL https://arxiv.org/abs/2503. 20215. [↗](https://arxiv.org/abs/2503)
70. P. Yang, X. Wang, X. Duan, H. Chen, R. Hou, C. Jin, and W. Zhu. Avqa: A dataset for audio-visual question answering on videos. In Proceedings of the 30th ACM international conference on multimedia, pages 3480–3491, 2022. [find](https://scholar.google.com/scholar?q=P%2E+Yang%2C+X%2E+Wang%2C+X%2E+Duan%2C+H%2E+Chen%2C+R%2E+Hou%2C+C%2E+Jin%2C+and+W%2E+Zhu%2E+Avqa%3A+A+dataset+for+audio%2Dvisual+question+answering+on+videos%2E+In+Proceedings+of+the+30th+ACM+international+conference+on+multimedia%2C+pa)
71. Q. Yang, J. Xu, W. Liu, Y. Chu, Z. Jiang, X. Zhou, Y. Leng, Y. Lv, Z. Zhao, C. Zhou, and J. Zhou. [Air-bench: Benchmarking large audio-language models via generative comprehension](https://md.research.elamur.ai/free/en/papers/yang-2024-air-bench-benchmarking-large-audio/). ArXiv, abs/2402.07729, 2024. URL [https://api.semanticscholar.org/CorpusID:267626820](https://api.semanticscholar.org/CorpusID:267626820). [↗](https://api.semanticscholar.org/CorpusID:267626820)
72. A. Young, B. Chen, C. Li, C. Huang, G. Zhang, G. Zhang, H. Li, J. Zhu, J. Chen, J. Chang, et al. Yi: Open foundation models by 01. ai, 2024. — [Yi: Open Foundation Models by 01.AI](https://md.research.elamur.ai/free/en/papers/ai-2024-yi-open-foundation-models-01/)
73. W. Yu, Z. Yang, L. Li, J. Wang, K. Lin, Z. Liu, X. Wang, and L. Wang. [Mm-vet: Evaluating large multimodal models for integrated capabilities](https://md.research.elamur.ai/free/en/papers/yu-2023-mm-vet-evaluating-large-multimodal/). arXiv preprint arXiv:2308.02490, 2023.
74. X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, et al. [Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556–9567, 2…
75. H. Yun, Y. Yu, W. Yang, K. Lee, and G. Kim. Pano-avqa: Grounded audio-visual question answering on 360deg videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2031–2041, 2021. [find](https://scholar.google.com/scholar?q=H%2E+Yun%2C+Y%2E+Yu%2C+W%2E+Yang%2C+K%2E+Lee%2C+and+G%2E+Kim%2E+Pano%2Davqa%3A+Grounded+audio%2Dvisual+question+answering+on+360deg+videos%2E+In+Proceedings+of+the+IEEE%2FCVF+International+Conference+on+Computer+Vision%2C+pages+2031)
76. X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. [Sigmoid loss for language image pre-training](https://md.research.elamur.ai/free/en/papers/zhai-2023-sigmoid-loss-language-image-pre/). In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023.
77. J. Zhan, J. Dai, J. Ye, Y. Zhou, D. Zhang, Z. Liu, X. Zhang, R. Yuan, G. Zhang, L. Li, et al. [Anygpt: Unified multimodal llm with discrete sequence modeling](https://md.research.elamur.ai/free/en/papers/zhan-2024-anygpt-unified-multimodal-llm-discrete/). arXiv preprint arXiv:2402.12226, 2024.
78. G. Zhang, X. Du, B. Chen, Y. Liang, T. Luo, T. Zheng, K. Zhu, Y. Cheng, C. Xu, S. Guo, H. Zhang, X. Qu, J. Wang, R. Yuan, Y. Li, Z. Wang, Y. Liu, Y.-H. Tsai, F. Zhang, C. Lin, W. Huang, W. Chen, and J. Fu. Cmmmu: A chinese massive multi-discipline multimodal understanding benchmark. ArXiv, abs/2401.… — [arXiv:2401.11944](https://arxiv.org/abs/2401.11944) [find](https://scholar.google.com/scholar?q=G%2E+Zhang%2C+X%2E+Du%2C+B%2E+Chen%2C+Y%2E+Liang%2C+T%2E+Luo%2C+T%2E+Zheng%2C+K%2E+Zhu%2C+Y%2E+Cheng%2C+C%2E+Xu%2C+S%2E+Guo%2C+H%2E+Zhang%2C+X%2E+Qu%2C+J%2E+Wang%2C+R%2E+Yuan%2C+Y%2E+Li%2C+Z%2E+Wang%2C+Y%2E+Liu%2C+Y%2E%2DH%2E+Tsai%2C+F%2E+Zhang%2C+C%2E+Lin%2C+W%2E+Huang%2C+W%2E+Chen%2C+and+J)

*Updated 1 November 2026*