---
title: "MMAU — paper by Sakshi et al. (2024) — summary and ideas"
description: "These tasks show little about whether a model can reason over audio the way a… 52.97% accuracy. 81 references, cited by 67 papers here."
authors: ["S Sakshi", "Utkarsh Tyagi", "Sonal Kumar", "Ashish Seth", "Ramaneswaran Selvakumar", "Oriol Nieto", "Ramani Duraiswami", "Sreyan Ghosh", "Dinesh Manocha"]
year: 2024
venue: "arXiv:2410.19168"
source: "https://arxiv.org/abs/2410.19168"
doi: "10.48550/arxiv.2410.19168"
arxiv: "2410.19168"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sakshi-2024-mmau-massive-multi-task-audio/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sakshi-2024-mmau-massive-multi-task-audio/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/) / MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark

# MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark

**Authors:** [S Sakshi](https://md.research.elamur.ai/free/en/authors/s-sakshi/), [Utkarsh Tyagi](https://md.research.elamur.ai/free/en/authors/utkarsh-tyagi/), [Sonal Kumar](https://md.research.elamur.ai/free/en/authors/sonal-kumar/), [Ashish Seth](https://md.research.elamur.ai/free/en/authors/ashish-seth/), Ramaneswaran Selvakumar, [Oriol Nieto](https://md.research.elamur.ai/free/en/authors/oriol-nieto/), [Ramani Duraiswami](https://md.research.elamur.ai/free/en/authors/ramani-duraiswami/), [Sreyan Ghosh](https://md.research.elamur.ai/free/en/authors/sreyan-ghosh/) and [Dinesh Manocha](https://md.research.elamur.ai/free/en/authors/dinesh-manocha/) · **Year:** 2024 · **Venue:** arXiv:2410.19168 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2410.19168) · [PDF](https://arxiv.org/pdf/2410.19168) · [DOI 10.48550/arxiv.2410.19168](https://doi.org/10.48550/arxiv.2410.19168) · [arXiv 2410.19168](https://arxiv.org/abs/2410.19168) · [OpenAlex](https://openalex.org/W4404311027)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [67](https://md.research.elamur.ai/free/en/papers/sakshi-2024-mmau-massive-multi-task-audio/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Large Audio-Language Models (LALMs) are usually tested on basic tasks such as ASR, acoustic scene classification, or genre labeling. These tasks show little about whether a model can reason over audio the way a skilled human expert would. The authors, from the University of Maryland and Adobe, built MMAU to fill this gap. It is a multiple-choice benchmark of roughly 10,000 audio clips covering speech, environmental sounds, and music, each with human-written questions and answers. The questions fall into two types. Information-extraction items require detailed content analysis and sometimes world knowledge, while reasoning items demand deliberate, multi-step inference. Together they probe 27 distinct skills, and LLM prompts were used to create tricky distractor options. Eighteen open and proprietary models were evaluated, and none of them got much past half the questions right. Because the benchmark exposes how far current systems are from expert-level listening, it gives the community a harder and broader target than existing audio evaluations.

## Key points
- **Scale and scope:** about 10k curated audio–question–answer items, roughly balanced across speech, sound, and music, testing 27 skills.
- **Example skills:** multi-speaker role mapping, emotion flip detection, temporal event reasoning, chord progression identification, and tongue-twister phonological decoding.
- **Top results:** Gemini Pro v1.5 scored highest at 52.97% accuracy. Qwen2-Audio was the strongest open-source model at 52.50%.
- **Analysis findings:** text-only LLMs gain from audio captions (here produced with Qwen2-Audio). LALMs often fail to really attend to the audio input and struggle on complex items.
- **Related benchmarks:** the paper positions MMAU against AIR-Bench, AudioBench, CompA-R, MuChoMusic, and OpenASQA. It argues these are either narrow in reasoning type or centered on foundational tasks.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Maryland, College Park](https://md.research.elamur.ai/free/en/universities/university-of-maryland-college-park/) — University, College Park, United States · 140 works · 29,011 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [🔊 Signal Processing](https://md.research.elamur.ai/free/en/shelves/computer-science/signal-processing/) · 493 works

🧭 Topics (3): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works · [🎙️ Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) · 519 works

## Cite

Cite as: S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh and Dinesh Manocha. 2024. MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark. arXiv:2410.19168.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{sakshi2024mmau,
  title = {{MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark}},
  author = {S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh and Dinesh Manocha},
  year = {2024},
  doi = {10.48550/arxiv.2410.19168},
  eprint = {2410.19168},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.19168}
}
```

- [← previous FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusion](https://md.research.elamur.ai/free/en/papers/chang-2024-flux-fast-software-based-communication/)
- [next → Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue](https://md.research.elamur.ai/free/en/papers/gu-2024-model-editing-harms-general-abilities/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/)

## References

81 references, 32 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774, 2023.
2. Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint arXiv:2301.11325, 2023. — [arXiv:2301.11325](https://arxiv.org/abs/2301.11325) [find](https://scholar.google.com/scholar?q=Andrea+Agostinelli%2C+Timo+I+Denk%2C+Zala%CC%81n+Borsos%2C+Jesse+Engel%2C+Mauro+Verzetti%2C+Antoine+Caillon%2C+Qingqing+Huang%2C+Aren+Jansen%2C+Adam+Roberts%2C+Marco+Tagliasacchi%2C+et+al%2E+Musiclm%3A+Generating+music+from+text%2E)
3. Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Maxamed Axmed, et al. [Mega: Multilingual evaluation of generative ai](https://md.research.elamur.ai/free/en/papers/ahuja-2023-mega-multilingual-evaluation-generative-ai/). arXiv preprint arXiv:2303.12528, 2023.
4. Dmitry Bogdanov, Minz Won, Philip Tovstogan, Alastair Porter, and Xavier Serra. The mtg-jamendo dataset for automatic music tagging. ICML, 2019. [find](https://scholar.google.com/scholar?q=Dmitry+Bogdanov%2C+Minz+Won%2C+Philip+Tovstogan%2C+Alastair+Porter%2C+and+Xavier+Serra%2E+The+mtg%2Djamendo+dataset+for+automatic+music+tagging%2E+ICML%2C+2019%2E)
5. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. [Sparks of artificial general intelligence: Early experiments with gpt-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/). arXiv preprint arXiv:2303.12712, 2023.
6. Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42:335–359, 2008. [find](https://scholar.google.com/scholar?q=Carlos+Busso%2C+Murtaza+Bulut%2C+Chi%2DChun+Lee%2C+Abe+Kazemzadeh%2C+Emily+Mower%2C+Samuel+Kim%2C+Jeannette+N+Chang%2C+Sungbok+Lee%2C+and+Shrikanth+S+Narayanan%2E+Iemocap%3A+Interactive+emotional+dyadic+motion+capture+data)
7. Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. Towards multimodal sarcasm detection (an Obviously perfect paper). In Anna Korhonen, David Traum, and Lluı́s Màrquez (eds.), Proceedings of the 57th Annual Meeting of the Association fo… — [Towards Multimodal Sarcasm Detection (An \textitObviously Perfect Paper)](https://md.research.elamur.ai/free/en/papers/castro-2019-multimodal-sarcasm-detection-textitobviously-perfect/)
8. Zhehuai Chen, He Huang, Andrei Andrusenko, Oleksii Hrinchuk, Krishna C Puvvada, Jason Li, Subhankar Ghosh, Jagadeesh Balam, and Boris Ginsburg. Salm: Speech-augmented language model with in-context learning for speech recognition and translation. In ICASSP 2024-2024 IEEE International Conference on… [find](https://scholar.google.com/scholar?q=Zhehuai+Chen%2C+He+Huang%2C+Andrei+Andrusenko%2C+Oleksii+Hrinchuk%2C+Krishna+C+Puvvada%2C+Jason+Li%2C+Subhankar+Ghosh%2C+Jagadeesh+Balam%2C+and+Boris+Ginsburg%2E+Salm%3A+Speech%2Daugmented+language+model+with+in%2Dcontext+le)
9. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). Journal of Machine Learning Research, 24(240): 1–113, 2023.
10. Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. [Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models](https://md.research.elamur.ai/free/en/papers/chu-2023-qwen-audio-advancing-universal/). arXiv preprint arXiv:2311.07919, 2023.
11. Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. [Qwen2-audio technical report](https://md.research.elamur.ai/free/en/papers/chu-2024-qwen2-audio-technical-report/). arXiv preprint arXiv:2407.10759, 2024.
12. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv preprint arXiv:1803.05457, 2018.
13. Zihao Deng, Yinghao Ma, Yudong Liu, Rongchen Guo, Ge Zhang, Wenhu Chen, Wenhao Huang, and Emmanouil Benetos. [Musilingo: Bridging music and text with pre-trained language models for music captioning and query response](https://md.research.elamur.ai/free/en/papers/deng-2024-musilingo-bridging-music-text-pre/). arXiv preprint arXiv:2309.08730, 2023.
14. Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang. Pengi: An audio language model for audio tasks. Advances in Neural Information Processing Systems, 36:18090–18108, 2023. [find](https://scholar.google.com/scholar?q=Soham+Deshmukh%2C+Benjamin+Elizalde%2C+Rita+Singh%2C+and+Huaming+Wang%2E+Pengi%3A+An+audio+language+model+for+audio+tasks%2E+Advances+in+Neural+Information+Processing+Systems%2C+36%3A18090%E2%80%9318108%2C+2023%2E)
15. Soham Deshmukh, Shuo Han, Hazim Bukhari, Benjamin Elizalde, Hannes Gamper, Rita Singh, and Bhiksha Raj. Audio entailment: Assessing deductive reasoning for audio understanding. arXiv preprint arXiv:2407.18062, 2024. — [arXiv:2407.18062](https://arxiv.org/abs/2407.18062) [find](https://scholar.google.com/scholar?q=Soham+Deshmukh%2C+Shuo+Han%2C+Hazim+Bukhari%2C+Benjamin+Elizalde%2C+Hannes+Gamper%2C+Rita+Singh%2C+and+Bhiksha+Raj%2E+Audio+entailment%3A+Assessing+deductive+reasoning+for+audio+understanding%2E+arXiv+preprint+arXiv%3A24)
16. SeungHeon Doh, Keunwoo Choi, Jongpil Lee, and Juhan Nam. Lp-musiccaps: Llm-based pseudo music captioning. arXiv preprint arXiv:2307.16372, 2023. — [arXiv:2307.16372](https://arxiv.org/abs/2307.16372) [find](https://scholar.google.com/scholar?q=SeungHeon+Doh%2C+Keunwoo+Choi%2C+Jongpil+Lee%2C+and+Juhan+Nam%2E+Lp%2Dmusiccaps%3A+Llm%2Dbased+pseudo+music+captioning%2E+arXiv+preprint+arXiv%3A2307%2E16372%2C+2023%2E+%E2%80%94+%5BarXiv%3A2307%2E16372%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2307%2E16372%29)
17. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783, 2024.
18. Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5. [find](https://scholar.google.com/scholar?q=Benjamin+Elizalde%2C+Soham+Deshmukh%2C+Mahmoud+Al+Ismail%2C+and+Huaming+Wang%2E+Clap+learning+audio+concepts+from+natural+language+supervision%2E+In+ICASSP+2023%2D2023+IEEE+International+Conference+on+Acoustics%2C+)
19. Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. [Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis](https://md.research.elamur.ai/free/en/papers/fu-2024-video-mme-first-ever-comprehensive/). arXiv preprint arXiv:2405.21075, 2024a.
20. Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. [Blink: Multimodal large language models can see but not perceive](https://md.research.elamur.ai/free/en/papers/fu-2024-blink-multimodal-large-language-models/). arXiv preprint arXiv:2404.12390, 2024b.
21. Josh Gardner, Simon Durand, Daniel Stoller, and Rachel M Bittner. Llark: A multimodal foundation model for music. arXiv preprint arXiv:2310.07160, 2023. — [arXiv:2310.07160](https://arxiv.org/abs/2310.07160) [find](https://scholar.google.com/scholar?q=Josh+Gardner%2C+Simon+Durand%2C+Daniel+Stoller%2C+and+Rachel+M+Bittner%2E+Llark%3A+A+multimodal+foundation+model+for+music%2E+arXiv+preprint+arXiv%3A2310%2E07160%2C+2023%2E+%E2%80%94+%5BarXiv%3A2310%2E07160%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2310)
22. Yingqiang Ge, Wenyue Hua, Kai Mei, Juntao Tan, Shuyuan Xu, Zelong Li, Yongfeng Zhang, et al. Openagi: When llm meets domain experts. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Yingqiang+Ge%2C+Wenyue+Hua%2C+Kai+Mei%2C+Juntao+Tan%2C+Shuyuan+Xu%2C+Zelong+Li%2C+Yongfeng+Zhang%2C+et+al%2E+Openagi%3A+When+llm+meets+domain+experts%2E+Advances+in+Neural+Information+Processing+Systems%2C+36%2C+2024%2E)
23. Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 776–780. [find](https://scholar.google.com/scholar?q=Jort+F+Gemmeke%2C+Daniel+PW+Ellis%2C+Dylan+Freedman%2C+Aren+Jansen%2C+Wade+Lawrence%2C+R+Channing+Moore%2C+Manoj+Plakal%2C+and+Marvin+Ritter%2E+Audio+set%3A+An+ontology+and+human%2Dlabeled+dataset+for+audio+events%2E+In+20)
24. Peter Gerhardstein and Carolyn Rovee-Collier. The development of visual search in infants and very young children. Journal of Experimental Child Psychology, 81(2):194–215, 2002. [find](https://scholar.google.com/scholar?q=Peter+Gerhardstein+and+Carolyn+Rovee%2DCollier%2E+The+development+of+visual+search+in+infants+and+very+young+children%2E+Journal+of+Experimental+Child+Psychology%2C+81%282%29%3A194%E2%80%93215%2C+2002%2E)
25. Sreyan Ghosh, Ashish Seth, Sonal Kumar, Utkarsh Tyagi, Chandra Kiran Evuru, S Ramaneswaran, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. Compa: Addressing the gap in compositional reasoning in audio-language models. arXiv preprint arXiv:2310.08753, 2023. — [arXiv:2310.08753](https://arxiv.org/abs/2310.08753) [find](https://scholar.google.com/scholar?q=Sreyan+Ghosh%2C+Ashish+Seth%2C+Sonal+Kumar%2C+Utkarsh+Tyagi%2C+Chandra+Kiran+Evuru%2C+S+Ramaneswaran%2C+S+Sakshi%2C+Oriol+Nieto%2C+Ramani+Duraiswami%2C+and+Dinesh+Manocha%2E+Compa%3A+Addressing+the+gap+in+compositional+rea)
26. Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, Utkarsh Tyagi, Oriol Nieto, Zeyu Jin, and Dinesh Manocha. Vdgd: Mitigating lvlm hallucinations in cognitive prompts by bridging the visual perception gap. arXiv preprint arXiv:2405.15683, 2024a. — [arXiv:2405.15683](https://arxiv.org/abs/2405.15683) [find](https://scholar.google.com/scholar?q=Sreyan+Ghosh%2C+Chandra+Kiran+Reddy+Evuru%2C+Sonal+Kumar%2C+Utkarsh+Tyagi%2C+Oriol+Nieto%2C+Zeyu+Jin%2C+and+Dinesh+Manocha%2E+Vdgd%3A+Mitigating+lvlm+hallucinations+in+cognitive+prompts+by+bridging+the+visual+percept)
27. Sreyan Ghosh, Sonal Kumar, Chandra Kiran Reddy Evuru, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. Reclap: Improving zero shot audio classification by describing sounds. arXiv preprint arXiv:2409.09213, 2024b. — [arXiv:2409.09213](https://arxiv.org/abs/2409.09213) [find](https://scholar.google.com/scholar?q=Sreyan+Ghosh%2C+Sonal+Kumar%2C+Chandra+Kiran+Reddy+Evuru%2C+Oriol+Nieto%2C+Ramani+Duraiswami%2C+and+Dinesh+Manocha%2E+Reclap%3A+Improving+zero+shot+audio+classification+by+describing+sounds%2E+arXiv+preprint+arXiv%3A24)
28. Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. [Gama: A large audio-language model with advanced audio understanding and complex reasoning abilities](https://md.research.elamur.ai/free/en/papers/ghosh-2024-gama-large-audio-language-model/). arXiv preprint arXiv:2406.11768, 2024c.
29. Yuan Gong. From audio perception to understanding: A path towards audio agi. In AI Seminar. Stony Brook University, 2024. URL [https://ai.stonybrook.edu/](https://ai.stonybrook.edu/) Audio-Perception-Understanding-Path-Towards-Audio-AGI. [↗](https://ai.stonybrook.edu/)
30. Yuan Gong, Alexander H. Liu, Hongyin Luo, Leonid Karlinsky, and James Glass. Joint audio and speech understanding. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 1–8, 2023a. doi: 10.1109/ASRU57964.2023.10389742. — [doi:10.1109/asru57964.2023.10389742](https://doi.org/10.1109/asru57964.2023.10389742) [find](https://scholar.google.com/scholar?q=Yuan+Gong%2C+Alexander+H%2E+Liu%2C+Hongyin+Luo%2C+Leonid+Karlinsky%2C+and+James+Glass%2E+Joint+audio+and+speech+understanding%2E+In+2023+IEEE+Automatic+Speech+Recognition+and+Understanding+Workshop+%28ASRU%29%2C+pp%2E+1%E2%80%938%2C)
31. Yuan Gong, Alexander H Liu, Hongyin Luo, Leonid Karlinsky, and James Glass. Joint audio and speech understanding. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 1–8. [find](https://scholar.google.com/scholar?q=Yuan+Gong%2C+Alexander+H+Liu%2C+Hongyin+Luo%2C+Leonid+Karlinsky%2C+and+James+Glass%2E+Joint+audio+and+speech+understanding%2E+In+2023+IEEE+Automatic+Speech+Recognition+and+Understanding+Workshop+%28ASRU%29%2C+pp%2E+1%E2%80%938%2E)
32. Yuan Gong, Hongyin Luo, Alexander H Liu, Leonid Karlinsky, and James Glass. Listen, think, and understand. arXiv preprint arXiv:2305.10790, 2023c. — [arXiv:2305.10790](https://arxiv.org/abs/2305.10790) [find](https://scholar.google.com/scholar?q=Yuan+Gong%2C+Hongyin+Luo%2C+Alexander+H+Liu%2C+Leonid+Karlinsky%2C+and+James+Glass%2E+Listen%2C+think%2C+and+understand%2E+arXiv+preprint+arXiv%3A2305%2E10790%2C+2023c%2E+%E2%80%94+%5BarXiv%3A2305%2E10790%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2305%2E10790)
33. Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 976–980. [find](https://scholar.google.com/scholar?q=Andrey+Guzhov%2C+Federico+Raue%2C+Jo%CC%88rn+Hees%2C+and+Andreas+Dengel%2E+Audioclip%3A+Extending+clip+to+image%2C+text+and+audio%2E+In+ICASSP+2022%2D2022+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Proc)
34. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300, 2020.
35. Shawn Hershey, Daniel PW Ellis, Eduardo Fonseca, Aren Jansen, Caroline Liu, R Channing Moore, and Manoj Plakal. The benefit of temporally-strong labels in audio event classification. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 366–370. [find](https://scholar.google.com/scholar?q=Shawn+Hershey%2C+Daniel+PW+Ellis%2C+Eduardo+Fonseca%2C+Aren+Jansen%2C+Caroline+Liu%2C+R+Channing+Moore%2C+and+Manoj+Plakal%2E+The+benefit+of+temporally%2Dstrong+labels+in+audio+event+classification%2E+In+ICASSP+2021%2D20)
36. Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22170–22183, 2024. [find](https://scholar.google.com/scholar?q=Yutao+Hu%2C+Tianbin+Li%2C+Quanfeng+Lu%2C+Wenqi+Shao%2C+Junjun+He%2C+Yu+Qiao%2C+and+Ping+Luo%2E+Omnimedvqa%3A+A+new+large%2Dscale+comprehensive+evaluation+benchmark+for+medical+lvlm%2E+In+Proceedings+of+the+IEEE%2FCVF+Confe)
37. Atin Sakkeer Hussain, Shansong Liu, Chenshuo Sun, and Ying Shan. M ˆ{2} ugen: Multi-modal music understanding and generation with the power of large language models. arXiv preprint arXiv:2311.11255, 2023. — [arXiv:2311.11255](https://arxiv.org/abs/2311.11255) [find](https://scholar.google.com/scholar?q=Atin+Sakkeer+Hussain%2C+Shansong+Liu%2C+Chenshuo+Sun%2C+and+Ying+Shan%2E+M+%CB%86%7B2%7D+ugen%3A+Multi%2Dmodal+music+understanding+and+generation+with+the+power+of+large+language+models%2E+arXiv+preprint+arXiv%3A2311%2E11255%2C+2)
38. Jaeyeon Kim, Jaeyoon Jung, Jinjoo Lee, and Sang Hoon Woo. Enclap: Combining neural audio codec and audio-text joint embedding for automated audio captioning. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6735– 6739. [find](https://scholar.google.com/scholar?q=Jaeyeon+Kim%2C+Jaeyoon+Jung%2C+Jinjoo+Lee%2C+and+Sang+Hoon+Woo%2E+Enclap%3A+Combining+neural+audio+codec+and+audio%2Dtext+joint+embedding+for+automated+audio+captioning%2E+In+ICASSP+2024%2D2024+IEEE+International+Con)
39. Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro. Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities. arXiv preprint arXiv:2402.01831, 2024. — [arXiv:2402.01831](https://arxiv.org/abs/2402.01831) [find](https://scholar.google.com/scholar?q=Zhifeng+Kong%2C+Arushi+Goel%2C+Rohan+Badlani%2C+Wei+Ping%2C+Rafael+Valle%2C+and+Bryan+Catanzaro%2E+Audio+flamingo%3A+A+novel+audio+language+model+with+few%2Dshot+learning+and+dialogue+abilities%2E+arXiv+preprint+arXiv%3A)
40. Ehsan Latif, Gengchen Mai, Matthew Nyaaba, Xuansheng Wu, Ninghao Liu, Guoyu Lu, Sheng Li, Tianming Liu, and Xiaoming Zhai. Artificial general intelligence (agi) for education. arXiv preprint arXiv:2304.12479, 1, 2023. — [arXiv:2304.12479](https://arxiv.org/abs/2304.12479) [find](https://scholar.google.com/scholar?q=Ehsan+Latif%2C+Gengchen+Mai%2C+Matthew+Nyaaba%2C+Xuansheng+Wu%2C+Ninghao+Liu%2C+Guoyu+Lu%2C+Sheng+Li%2C+Tianming+Liu%2C+and+Xiaoming+Zhai%2E+Artificial+general+intelligence+%28agi%29+for+education%2E+arXiv+preprint+arXiv%3A230)
41. Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. [Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models](https://md.research.elamur.ai/free/en/papers/li-2024-multimodal-arxiv-dataset-improving-scientific/). arXiv preprint arXiv:2403.00231, 2024.
42. Yizhi Li, Ruibin Yuan, Ge Zhang, Yinghao Ma, Xingran Chen, Hanzhi Yin, Chenghao Xiao, Chenghua Lin, Anton Ragni, Emmanouil Benetos, et al. Mert: Acoustic music understanding model with large-scale self-supervised training. arXiv preprint arXiv:2306.00107, 2023. — [arXiv:2306.00107](https://arxiv.org/abs/2306.00107) [find](https://scholar.google.com/scholar?q=Yizhi+Li%2C+Ruibin+Yuan%2C+Ge+Zhang%2C+Yinghao+Ma%2C+Xingran+Chen%2C+Hanzhi+Yin%2C+Chenghao+Xiao%2C+Chenghua+Lin%2C+Anton+Ragni%2C+Emmanouil+Benetos%2C+et+al%2E+Mert%3A+Acoustic+music+understanding+model+with+large%2Dscale+sel)
43. Richard P. Lippmann. Speech recognition by machines and humans. Speech Communication, 22(1):1–15, 1997. ISSN 0167-6393. doi: https://doi.org/10.1016/S0167-6393(97) 00021-6. URL https://www.sciencedirect.com/science/article/pii/ S0167639397000216. — [doi:10.1016/s0167-6393(97)](https://doi.org/10.1016/s0167-6393%2897%29) [↗](https://doi.org/10.1016/S0167-6393(97)
44. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. [Improved baselines with visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/), 2023a.
45. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023b. [find](https://scholar.google.com/scholar?q=Haotian+Liu%2C+Chunyuan+Li%2C+Qingyang+Wu%2C+and+Yong+Jae+Lee%2E+Visual+instruction+tuning%2C+2023b%2E)
46. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024a. URL [https://llava-vl.github.io/blog/2024-01-30-llava-next/](https://llava-vl.github.io/blog/2024-01-30-llava-next/). [↗](https://llava-vl.github.io/blog/2024-01-30-llava-next/)
47. Shansong Liu, Atin Sakkeer Hussain, Chenshuo Sun, and Ying Shan. Music understanding llama: Advancing text-to-music generation with question answering and captioning. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 286–290. [find](https://scholar.google.com/scholar?q=Shansong+Liu%2C+Atin+Sakkeer+Hussain%2C+Chenshuo+Sun%2C+and+Ying+Shan%2E+Music+understanding+llama%3A+Advancing+text%2Dto%2Dmusic+generation+with+question+answering+and+captioning%2E+In+ICASSP+2024%2D2024+IEEE+Internat)
48. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. [Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts](https://md.research.elamur.ai/free/en/papers/lu-2023-mathvista-evaluating-mathematical-reasoning-foundation/). arXiv preprint arXiv:2310.02255, 2023.
49. Ilaria Manco, Benno Weck, Seungheon Doh, Minz Won, Yixiao Zhang, Dmitry Bodganov, Yusong Wu, Ke Chen, Philip Tovstogan, Emmanouil Benetos, et al. The song describer dataset: a corpus of audio captions for music-and-language evaluation. arXiv preprint arXiv:2311.10057, 2023. — [arXiv:2311.10057](https://arxiv.org/abs/2311.10057) [find](https://scholar.google.com/scholar?q=Ilaria+Manco%2C+Benno+Weck%2C+Seungheon+Doh%2C+Minz+Won%2C+Yixiao+Zhang%2C+Dmitry+Bodganov%2C+Yusong+Wu%2C+Ke+Chen%2C+Philip+Tovstogan%2C+Emmanouil+Benetos%2C+et+al%2E+The+song+describer+dataset%3A+a+corpus+of+audio+captions)
50. Jan Melechovsky, Zixun Guo, Deepanway Ghosal, Navonil Majumder, Dorien Herremans, and Soujanya Poria. Mustango: Toward controllable text-to-music generation. arXiv preprint arXiv:2311.08355, 2023. [find](https://scholar.google.com/scholar?q=Jan+Melechovsky%2C+Zixun+Guo%2C+Deepanway+Ghosal%2C+Navonil+Majumder%2C+Dorien+Herremans%2C+and+Soujanya+Poria%2E+Mustango%3A+Toward+controllable+text%2Dto%2Dmusic+generation%2E+arXiv+preprint+arXiv%3A2311%2E08355%2C+2023%2E)
51. Meredith Ringel Morris, Jascha Sohl-Dickstein, Noah Fiedel, Tris Warkentin, Allan Dafoe, Aleksandra Faust, Clement Farabet, and Shane Legg. Position: Levels of AGI for operationalizing progress on the path to AGI. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, J… [find](https://scholar.google.com/scholar?q=Meredith+Ringel+Morris%2C+Jascha+Sohl%2DDickstein%2C+Noah+Fiedel%2C+Tris+Warkentin%2C+Allan+Dafoe%2C+Aleksandra+Faust%2C+Clement+Farabet%2C+and+Shane+Legg%2E+Position%3A+Levels+of+AGI+for+operationalizing+progress+on+the)
52. Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. Voxceleb: a large-scale speaker identification dataset. arXiv preprint arXiv:1706.08612, 2017. [find](https://scholar.google.com/scholar?q=Arsha+Nagrani%2C+Joon+Son+Chung%2C+and+Andrew+Zisserman%2E+Voxceleb%3A+a+large%2Dscale+speaker+identification+dataset%2E+arXiv+preprint+arXiv%3A1706%2E08612%2C+2017%2E)
53. Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan. Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language models. arXiv preprint arXiv:2311.16103, 2023. — [arXiv:2311.16103](https://arxiv.org/abs/2311.16103) [find](https://scholar.google.com/scholar?q=Munan+Ning%2C+Bin+Zhu%2C+Yujia+Xie%2C+Bin+Lin%2C+Jiaxi+Cui%2C+Lu+Yuan%2C+Dongdong+Chen%2C+and+Li+Yuan%2E+Video%2Dbench%3A+A+comprehensive+benchmark+and+toolkit+for+evaluating+video%2Dbased+large+language+models%2E+arXiv+prep)
54. OpenAI, Josh Achiam, and Others. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), 2024. URL [https://arxiv.org/](https://arxiv.org/) abs/2303.08774. [↗](https://arxiv.org/)
55. Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. [Meld: A multimodal multi-party dataset for emotion recognition in conversations](https://md.research.elamur.ai/free/en/papers/poria-2018-meld-multimodal-multi-party-dataset/). arXiv preprint arXiv:1810.02508, 2018.
56. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. [Robust speech recognition via large-scale weak supervision](https://md.research.elamur.ai/free/en/papers/radford-2022-robust-speech-recognition-large-scale/). In International conference on machine learning, pp. 28492–28518. PMLR, 2023.
57. Zafar Rafii, Antoine Liutkus, Fabian-Robert Stöter, Stylianos Ioannis Mimilakis, and Rachel Bittner. The musdb18 corpus for music separation. 2017. [find](https://scholar.google.com/scholar?q=Zafar+Rafii%2C+Antoine+Liutkus%2C+Fabian%2DRobert+Sto%CC%88ter%2C+Stylianos+Ioannis+Mimilakis%2C+and+Rachel+Bittner%2E+The+musdb18+corpus+for+music+separation%2E+2017%2E)
58. Inioluwa Deborah Raji, Emily M Bender, Amandalynne Paullada, Emily Denton, and Alex Hanna. Ai and the everything in the whole wide world benchmark. arXiv preprint arXiv:2111.15366, 2021. [find](https://scholar.google.com/scholar?q=Inioluwa+Deborah+Raji%2C+Emily+M+Bender%2C+Amandalynne+Paullada%2C+Emily+Denton%2C+and+Alex+Hanna%2E+Ai+and+the+everything+in+the+whole+wide+world+benchmark%2E+arXiv+preprint+arXiv%3A2111%2E15366%2C+2021%2E)
59. Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al. [Audiopalm: A large language model that can speak and listen](https://md.research.elamur.ai/free/en/papers/rubenstein-2023-audiopalm-large-language-model-that/). arXiv preprint arXiv:2306.12925, 2023.
60. Ashish Seth, Mayur Hemani, and Chirag Agarwal. Dear: Debiasing vision-language models with additive residuals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6820–6829, 2023. [find](https://scholar.google.com/scholar?q=Ashish+Seth%2C+Mayur+Hemani%2C+and+Chirag+Agarwal%2E+Dear%3A+Debiasing+vision%2Dlanguage+models+with+additive+residuals%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+pp%2E)
61. Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. [Salmonn: Towards generic hearing abilities for large language models](https://md.research.elamur.ai/free/en/papers/tang-2023-salmonn-towards-generic-hearing-abilities/). arXiv preprint arXiv:2310.13289, 2023.
62. Gemini Team, Petko Georgiev, and Others. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/), 2024. URL https://arxiv.org/abs/2403.05530. [↗](https://arxiv.org/abs/2403.05530)
63. H Touvron, T Lavril, G Izacard, X Martinet, MA Lachaux, T Lacroix, B Rozière, N Goyal, E Hambro, F Azhar, et al. Open and efficient foundation language models. Preprint at arXiv. https://doi.org/10.48550/arXiv, 2302, 2023a. — [doi:10.48550/arxiv](https://doi.org/10.48550/arxiv) [↗](https://doi.org/10.48550/arXiv)
64. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/), 2023b. URL h…
65. Harsh Trivedi, Heeyoung Kwon, Tushar Khot, Ashish Sabharwal, and Niranjan Balasubramanian. Repurposing entailment for multi-hop question answering tasks. arXiv preprint arXiv:1904.09380, 2019. [find](https://scholar.google.com/scholar?q=Harsh+Trivedi%2C+Heeyoung+Kwon%2C+Tushar+Khot%2C+Ashish+Sabharwal%2C+and+Niranjan+Balasubramanian%2E+Repurposing+entailment+for+multi%2Dhop+question+answering+tasks%2E+arXiv+preprint+arXiv%3A1904%2E09380%2C+2019%2E)
66. Alex Wang. [Glue: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). arXiv preprint arXiv:1804.07461, 2018.
67. Bin Wang, Xunlong Zou, Geyu Lin, Shuo Sun, Zhuohan Liu, Wenyu Zhang, Zhengyuan Liu, AiTi Aw, and Nancy F. Chen. [Audiobench: A universal benchmark for audio large language models](https://md.research.elamur.ai/free/en/papers/wang-2025-audiobench-universal-benchmark-audio-large/), 2024a. URL https://arxiv.org/abs/2406.16020. [↗](https://arxiv.org/abs/2406.16020)
68. Zihao Wang, Shuyu Li, Tao Zhang, Qi Wang, Pengfei Yu, Jinyang Luo, Yan Liu, Ming Xi, and Kejun Zhang. Muchin: A chinese colloquial description benchmark for evaluating language models in the field of music. arXiv preprint arXiv:2402.09871, 2024b. — [arXiv:2402.09871](https://arxiv.org/abs/2402.09871) [find](https://scholar.google.com/scholar?q=Zihao+Wang%2C+Shuyu+Li%2C+Tao+Zhang%2C+Qi+Wang%2C+Pengfei+Yu%2C+Jinyang+Luo%2C+Yan+Liu%2C+Ming+Xi%2C+and+Kejun+Zhang%2E+Muchin%3A+A+chinese+colloquial+description+benchmark+for+evaluating+language+models+in+the+field+of+)
69. Benno Weck, Ilaria Manco, Emmanouil Benetos, Elio Quinton, George Fazekas, and Dmitry Bogdanov. Muchomusic: Evaluating music understanding in multimodal audio-language models. arXiv preprint arXiv:2408.01337, 2024. — [arXiv:2408.01337](https://arxiv.org/abs/2408.01337) [find](https://scholar.google.com/scholar?q=Benno+Weck%2C+Ilaria+Manco%2C+Emmanouil+Benetos%2C+Elio+Quinton%2C+George+Fazekas%2C+and+Dmitry+Bogdanov%2E+Muchomusic%3A+Evaluating+music+understanding+in+multimodal+audio%2Dlanguage+models%2E+arXiv+preprint+arXiv%3A240)
70. Minz Won, Yun-Ning Hung, and Duc Le. A foundation model for music informatics. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1226–1230. [find](https://scholar.google.com/scholar?q=Minz+Won%2C+Yun%2DNing+Hung%2C+and+Duc+Le%2E+A+foundation+model+for+music+informatics%2E+In+ICASSP+2024%2D2024+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+%28ICASSP%29%2C+pp%2E+1226%E2%80%931230%2E)
71. Yusong Wu*, Ke Chen*, Tianyu Zhang*, Yuchen Hui*, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, 2023. [find](https://scholar.google.com/scholar?q=Yusong+Wu%2A%2C+Ke+Chen%2A%2C+Tianyu+Zhang%2A%2C+Yuchen+Hui%2A%2C+Taylor+Berg%2DKirkpatrick%2C+and+Shlomo+Dubnov%2E+Large%2Dscale+contrastive+language%2Daudio+pretraining+with+feature+fusion+and+keyword%2Dto%2Dcaption+augmentation)
72. Qingyang Xi, Rachel M Bittner, Johan Pauwels, Xuzhou Ye, and Juan Pablo Bello. Guitarset: A dataset for guitar transcription. In ISMIR, pp. 453–460, 2018. [find](https://scholar.google.com/scholar?q=Qingyang+Xi%2C+Rachel+M+Bittner%2C+Johan+Pauwels%2C+Xuzhou+Ye%2C+and+Juan+Pablo+Bello%2E+Guitarset%3A+A+dataset+for+guitar+transcription%2E+In+ISMIR%2C+pp%2E+453%E2%80%93460%2C+2018%2E)
73. Blaise Agüera y Arcas and Peter Norvig. Artificial general intelligence is already here. Noema, October, 2023. [find](https://scholar.google.com/scholar?q=Blaise+Agu%CC%88era+y+Arcas+and+Peter+Norvig%2E+Artificial+general+intelligence+is+already+here%2E+Noema%2C+October%2C+2023%2E)
74. Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, and Jingren Zhou. [Air-bench: Benchmarking large audio-language models via generative comprehension](https://md.research.elamur.ai/free/en/papers/yang-2024-air-bench-benchmarking-large-audio/), 2024. URL https://arxiv.org/abs/ 2402.07729. [↗](https://arxiv.org/abs/)
75. Jiaxuan You, Ge Liu, Yunzhu Li, Song Han, and Dawn Song. How far are we from agi. In ICLR 2024 Workshops, 2024. [find](https://scholar.google.com/scholar?q=Jiaxuan+You%2C+Ge+Liu%2C+Yunzhu+Li%2C+Song+Han%2C+and+Dawn+Song%2E+How+far+are+we+from+agi%2E+In+ICLR+2024+Workshops%2C+2024%2E)
76. Ruibin Yuan, Yinghao Ma, Yizhi Li, Ge Zhang, Xingran Chen, Hanzhi Yin, Yiqi Liu, Jiawen Huang, Zeyue Tian, Binyue Deng, et al. Marble: Music audio representation benchmark for universal evaluation. Advances in Neural Information Processing Systems, 36:39626–39647, 2023. [find](https://scholar.google.com/scholar?q=Ruibin+Yuan%2C+Yinghao+Ma%2C+Yizhi+Li%2C+Ge+Zhang%2C+Xingran+Chen%2C+Hanzhi+Yin%2C+Yiqi+Liu%2C+Jiawen+Huang%2C+Zeyue+Tian%2C+Binyue+Deng%2C+et+al%2E+Marble%3A+Music+audio+representation+benchmark+for+universal+evaluation%2E+Ad)
77. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. [Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Patt…
78. Hang Zhang, Xin Li, and Lidong Bing. [Video-llama: An instruction-tuned audio-visual language model for video understanding](https://md.research.elamur.ai/free/en/papers/zhang-2023-video-llama-instruction-tuned-audio/). arXiv preprint arXiv:2306.02858, 2023a. URL https://arxiv.org/abs/2306.02858. [↗](https://arxiv.org/abs/2306.02858)
79. Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, et al. [Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?](https://md.research.elamur.ai/free/en/papers/zhang-2024-mathverse-multi-modal-llm-truly/) arXiv preprint arXiv:2403.14624, 2024.
80. Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. [Safetybench: Evaluating the safety of large language models](https://md.research.elamur.ai/free/en/papers/zhang-2023-safetybench-evaluating-safety-large/) with multiple choice questions. arXiv preprint arXiv:2309.07045, 2023b.
81. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. [MiniGPT-4: Enhancing vision-language understanding with advanced large language models](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/). In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/](https://openreview.net/) forum?id=1tZbq88f27. [↗](https://openreview.net/)

*Updated 2 November 2026*