---
title: "AIR-Bench — paper by Yang et al. (2024) — summary and ideas"
description: "Large audio-language models (LALMs) such as SALMONN, BLSP and Qwen-Audio can respond to… 66 references, cited by 69 papers here."
authors: ["Qian Yang", "Jin Xu", "Wenrui Liu", "Yunfei Chu", "Ziyue Jiang", "Xiaohuan Zhou", "Yichong Leng", "Yuanjun Lv", "Zhou Zhao", "Chang Zhou", "Jingren Zhou"]
year: 2024
venue: "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.acl-long.109/"
doi: "10.18653/v1/2024.acl-long.109"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yang-2024-air-bench-benchmarking-large-audio/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yang-2024-air-bench-benchmarking-large-audio/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension

# AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension

**Authors:** [Qian Yang](https://md.research.elamur.ai/free/en/authors/qian-yang/), [Jin Xu](https://md.research.elamur.ai/free/en/authors/jin-xu/), Wenrui Liu, [Yunfei Chu](https://md.research.elamur.ai/free/en/authors/yunfei-chu/), Ziyue Jiang, [Xiaohuan Zhou](https://md.research.elamur.ai/free/en/authors/xiaohuan-zhou/), [Yichong Leng](https://md.research.elamur.ai/free/en/authors/yichong-leng/), [Yuanjun Lv](https://md.research.elamur.ai/free/en/authors/yuanjun-lv/), [Zhou Zhao](https://md.research.elamur.ai/free/en/authors/zhou-zhao/), [Chang Zhou](https://md.research.elamur.ai/free/en/authors/chang-zhou/) and [Jingren Zhou](https://md.research.elamur.ai/free/en/authors/jingren-zhou/) · **Year:** 2024 · **Venue:** [Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2024](https://md.research.elamur.ai/free/en/venues/acl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.acl-long.109/) · [PDF](https://aclanthology.org/2024.acl-long.109.pdf) · [DOI 10.18653/v1/2024.acl-long.109](https://doi.org/10.18653/v1/2024.acl-long.109) · [OpenAlex](https://openalex.org/W4402684170)

**Citations:** 39 citations (OpenAlex, 2026-10-14) · [69](https://md.research.elamur.ai/free/en/papers/yang-2024-air-bench-benchmarking-large-audio/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary

Large audio-language models (LALMs) such as SALMONN, BLSP and Qwen-Audio can respond to spoken and other audio input in free text. Until now they were mostly compared on narrow fixed tasks like ASR, or shown through demos, so fair comparison was hard. Dynamic-SUPERB, the only earlier instruction-oriented benchmark, covers speech alone and does not test open-ended answers.

The authors built AIR-Bench, which spans speech, natural sounds and music at two levels:

- **Foundation tier:** diagnoses individual skills using single-choice questions.
- **Chat tier:** tests open-ended question answering on complex audio. Some of this audio is produced by mixing two clips with controlled relative loudness and time offsets, and those mixing settings are recorded as extra metadata.

In both tiers the model must generate its own answer. GPT-4 then judges that answer against references derived from the audio metadata. Each answer is scored twice with the order of hypothesis and reference swapped, which reduces position bias.

Across 9 LALMs, current systems turned out to be weak either in understanding audio or in following instructions. This makes AIR-Bench a useful diagnostic tool for future work.

## Key points

- **Scale:** the foundation benchmark has 19 tasks and about 19k single-choice questions. The chat benchmark holds roughly 2k open-ended audio questions, filtered by GPT-4 and then checked by hand.
- **Agreement with humans (foundation):** on 400 sampled questions answered by Qwen-Audio-Chat, GPT-4 Turbo as judge agreed with three native English annotators 98.2% of the time. GPT-3.5 Turbo reached 96.4%.
- **Agreement with humans (chat):** in pairwise preference tests on 200 questions comparing Qwen-Audio-Chat, SALMONN, BLSP and GPT-4, judge–human agreement was above 70%.
- **Position bias:** the GPT-4 judge clearly scores differently when the hypothesis comes before the reference, which justifies the swap-and-average scoring.
- **Availability and limits:** data and evaluation code are released at github.com/OFA-Sys/AIR-Bench. The benchmark does not yet cover multi-audio comparison or multi-turn dialogue, and it depends on access to the GPT-4 API.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Alibaba Group](https://md.research.elamur.ai/free/en/companies/alibaba-group/) — Company, Hangzhou, China · 64 works · 14,113 citations
- [Zhejiang University](https://md.research.elamur.ai/free/en/universities/zhejiang-university/) — University, Hangzhou, China · 46 works · 11,542 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (1): [🎙️ Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) · 233 works

## Cite

Cite as: Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou and Jingren Zhou. 2024. AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yang2024air,
  title = {{AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension}},
  author = {Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou and Jingren Zhou},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.109},
  url = {https://doi.org/10.18653/v1/2024.acl-long.109}
}
```

- [← previous Orca-Math: Unlocking the potential of SLMs in Grade School Math](https://md.research.elamur.ai/free/en/papers/mitra-2024-orca-math-unlocking-potential-slms/)
- [next → CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation](https://md.research.elamur.ai/free/en/papers/tu-2024-charactereval-chinese-benchmark-role-playing/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

66 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. 2023. Musiclm: Generating music from text. arXiv preprint arXiv:2301.11325. — [arXiv:2301.11325](https://arxiv.org/abs/2301.11325) [find](https://scholar.google.com/scholar?q=Andrea+Agostinelli%2C+Timo+I+Denk%2C+Zal%C3%A1n+Borsos%2C+Jesse+Engel%2C+Mauro+Verzetti%2C+Antoine+Caillon%2C+Qingqing+Huang%2C+Aren+Jansen%2C+Adam+Roberts%2C+Marco+Tagliasacchi%2C+et+al%2E+2023%2E+Musiclm%3A+Generating+music+from+)
2. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. [PaLM 2 technical report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/). arXiv:2305.10403.
3. Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. 2019. [Common voice: A massively-multilingual speech corpus](https://md.research.elamur.ai/free/en/papers/ardila-2019-common-voice-massively-multilingual-speech/). arXiv preprint arXiv:1912.06670.
4. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023a. [Qwen technical report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/). arXiv preprint arXiv:2309.16609.
5. Shuai Bai, Shusheng Yang, Jinze Bai, Peng Wang, Xingxuan Zhang, Junyang Lin, Xinggang Wang, Chang Zhou, and Jingren Zhou. 2023b. Touchstone: Evaluating vision-language models by language models. arXiv preprint arXiv:2308.16890. — [arXiv:2308.16890](https://arxiv.org/abs/2308.16890) [find](https://scholar.google.com/scholar?q=Shuai+Bai%2C+Shusheng+Yang%2C+Jinze+Bai%2C+Peng+Wang%2C+Xingxuan+Zhang%2C+Junyang+Lin%2C+Xinggang+Wang%2C+Chang+Zhou%2C+and+Jingren+Zhou%2E+2023b%2E+Touchstone%3A+Evaluating+vision%2Dlanguage+models+by+language+models%2E+arXiv)
6. Satanjeev Banerjee and Alon Lavie. 2005. [METEOR: an automatic metric for MT evaluation with improved correlation with human judgments](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/). In Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization@ACL 2005, Ann Arbor, Michigan, USA, June 2…
7. Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser. 2020. [Slurp: A spoken language understanding resource package](https://md.research.elamur.ai/free/en/papers/bastianelli-2020-slurp-spoken-language-understanding-resource/). arXiv preprint arXiv:2011.13205.
8. Dmitry Bogdanov, Minz Won, Philip Tovstogan, Alastair Porter, and Xavier Serra. 2019. The mtg-jamendo dataset for automatic music tagging. ICML. [find](https://scholar.google.com/scholar?q=Dmitry+Bogdanov%2C+Minz+Won%2C+Philip+Tovstogan%2C+Alastair+Porter%2C+and+Xavier+Serra%2E+2019%2E+The+mtg%2Djamendo+dataset+for+automatic+music+tagging%2E+ICML%2E)
9. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). NeurIPS.
10. Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42:335–359. [find](https://scholar.google.com/scholar?q=Carlos+Busso%2C+Murtaza+Bulut%2C+Chi%2DChun+Lee%2C+Abe+Kazemzadeh%2C+Emily+Mower%2C+Samuel+Kim%2C+Jeannette+N+Chang%2C+Sungbok+Lee%2C+and+Shrikanth+S+Narayanan%2E+2008%2E+Iemocap%3A+Interactive+emotional+dyadic+motion+captur)
11. Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. 2014. [find](https://scholar.google.com/scholar?q=Houwei+Cao%2C+David+G+Cooper%2C+Michael+K+Keutmann%2C+Ruben+C+Gur%2C+Ani+Nenkova%2C+and+Ragini+Verma%2E+2014%2E)
12. Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE transactions on affective computing, 5(4):377–390. Dynamic-superb: Towards A dynamic, collaborative, and comprehensive instruction-tuning benchmark for speech. CoRR, abs/2309.09510. — [arXiv:2309.09510](https://arxiv.org/abs/2309.09510) [find](https://scholar.google.com/scholar?q=Crema%2Dd%3A+Crowd%2Dsourced+emotional+multimodal+actors+dataset%2E+IEEE+transactions+on+affective+computing%2C+5%284%29%3A377%E2%80%93390%2E+Dynamic%2Dsuperb%3A+Towards+A+dynamic%2C+collaborative%2C+and+comprehensive+instruction%2Dtuni)
13. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. [PaLM: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv:2204.02311.
14. Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, Yi Ren, Zhou Zhao, and Shinji Watanabe. 2023b. Audiogpt: Understanding and generating speech, music, sound, and talking head. CoRR, abs/2304.12995. — [arXiv:2304.12995](https://arxiv.org/abs/2304.12995) [find](https://scholar.google.com/scholar?q=Rongjie+Huang%2C+Mingze+Li%2C+Dongchao+Yang%2C+Jiatong+Shi%2C+Xuankai+Chang%2C+Zhenhui+Ye%2C+Yuning+Wu%2C+Zhiqing+Hong%2C+Jiawei+Huang%2C+Jinglin+Liu%2C+Yi+Ren%2C+Zhou+Zhao%2C+and+Shinji+Watanabe%2E+2023b%2E+Audiogpt%3A+Understand)
15. Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. [Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models](https://md.research.elamur.ai/free/en/papers/chu-2023-qwen-audio-advancing-universal/). CoRR, abs/2311.07919.
16. Christopher Cieri, David Miller, and Kevin Walker. 2004. [The fisher corpus: A resource for the next generations of speech-to-text](https://md.research.elamur.ai/free/en/papers/cieri-2004-fisher-corpus-resource-next-generations/). In LREC, volume 4, pages 69–71.
17. Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. 2022. [FLEURS: few-shot learning evaluation of universal representations of speech](https://md.research.elamur.ai/free/en/papers/conneau-2022-fleurs-few-shot-learning-evaluation/). In IEEE Spoken Language Technology Workshop, SLT 2022, Doha, Qatar, January 9-12, 2023. IEE…
18. Michaël Defferrard, Kirell Benzi, Pierre Vandergheynst, and Xavier Bresson. 2016. Fma: A dataset for music analysis. arXiv preprint arXiv:1612.01840. — [arXiv:1612.01840](https://arxiv.org/abs/1612.01840) [find](https://scholar.google.com/scholar?q=Micha%C3%ABl+Defferrard%2C+Kirell+Benzi%2C+Pierre+Vandergheynst%2C+and+Xavier+Bresson%2E+2016%2E+Fma%3A+A+dataset+for+music+analysis%2E+arXiv+preprint+arXiv%3A1612%2E01840%2E+%E2%80%94+%5BarXiv%3A1612%2E01840%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1612%2E01)
19. Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. 2020. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 736–740. IEEE. [find](https://scholar.google.com/scholar?q=Konstantinos+Drossos%2C+Samuel+Lipping%2C+and+Tuomas+Virtanen%2E+2020%2E+Clotho%3A+An+audio+captioning+dataset%2E+In+ICASSP+2020%2D2020+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+%28ICAS)
20. Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Mohammad Norouzi, Douglas Eck, and Karen Simonyan. 2017. Neural audio synthesis of musical notes with wavenet autoencoders. In International Conference on Machine Learning, pages 1068–1077. PMLR. [find](https://scholar.google.com/scholar?q=Jesse+Engel%2C+Cinjon+Resnick%2C+Adam+Roberts%2C+Sander+Dieleman%2C+Mohammad+Norouzi%2C+Douglas+Eck%2C+and+Karen+Simonyan%2E+2017%2E+Neural+audio+synthesis+of+musical+notes+with+wavenet+autoencoders%2E+In+International)
21. Josh Gardner, Simon Durand, Daniel Stoller, and Rachel M Bittner. 2023. Llark: A multimodal foundation model for music. arXiv preprint arXiv:2310.07160. — [arXiv:2310.07160](https://arxiv.org/abs/2310.07160) [find](https://scholar.google.com/scholar?q=Josh+Gardner%2C+Simon+Durand%2C+Daniel+Stoller%2C+and+Rachel+M+Bittner%2E+2023%2E+Llark%3A+A+multimodal+foundation+model+for+music%2E+arXiv+preprint+arXiv%3A2310%2E07160%2E+%E2%80%94+%5BarXiv%3A2310%2E07160%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2310)
22. Il-Young Jeong and Jeongsoo Park. 2022. Cochlscene: Acquisition of acoustic scene data using crowdsourcing. In 2022 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pages 17–21. IEEE. [find](https://scholar.google.com/scholar?q=Il%2DYoung+Jeong+and+Jeongsoo+Park%2E+2022%2E+Cochlscene%3A+Acquisition+of+acoustic+scene+data+using+crowdsourcing%2E+In+2022+Asia%2DPacific+Signal+and+Information+Processing+Association+Annual+Summit+and+Confere)
23. Ye Jia, Michelle Tadmor Ramanovich, Quan Wang, and Heiga Zen. 2022. CVSS corpus and massively multilingual speech-to-speech translation. In Proceedings of Language Resources and Evaluation Conference (LREC). [find](https://scholar.google.com/scholar?q=Ye+Jia%2C+Michelle+Tadmor+Ramanovich%2C+Quan+Wang%2C+and+Heiga+Zen%2E+2022%2E+CVSS+corpus+and+massively+multilingual+speech%2Dto%2Dspeech+translation%2E+In+Proceedings+of+Language+Resources+and+Evaluation+Conference+)
24. Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. 2019a. [Audiocaps: Generating captions for audios in the wild](https://md.research.elamur.ai/free/en/papers/kim-2019-audiocaps-generating-captions-audios-wild/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short P…
25. Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. 2019b. [Audiocaps: Generating captions for audios in the wild](https://md.research.elamur.ai/free/en/papers/kim-2019-audiocaps-generating-captions-audios-wild/). In NAACL-HLT.
26. Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. 2022. Learning to answer questions in dynamic audio-visual scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19108–19118. [find](https://scholar.google.com/scholar?q=Guangyao+Li%2C+Yake+Wei%2C+Yapeng+Tian%2C+Chenliang+Xu%2C+Ji%2DRong+Wen%2C+and+Di+Hu%2E+2022%2E+Learning+to+answer+questions+in+dynamic+audio%2Dvisual+scenarios%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vi)
27. Chin-Yew Lin. 2004. [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text summarization branches out, pages 74–81.
28. Samuel Lipping, Parthasaarathy Sudarsanam, Konstantinos Drossos, and Tuomas Virtanen. 2022. Clothoaqa: A crowdsourced dataset for audio question answering. In 2022 30th European Signal Processing Conference (EUSIPCO), pages 1140–1144. IEEE. [find](https://scholar.google.com/scholar?q=Samuel+Lipping%2C+Parthasaarathy+Sudarsanam%2C+Konstantinos+Drossos%2C+and+Tuomas+Virtanen%2E+2022%2E+Clothoaqa%3A+A+crowdsourced+dataset+for+audio+question+answering%2E+In+2022+30th+European+Signal+Processing+Conf)
29. Yuan Gong, Hongyin Luo, Alexander H. Liu, Leonid Karlinsky, and James R. Glass. 2023. Listen, think, and understand. CoRR, abs/2305.10790. — [arXiv:2305.10790](https://arxiv.org/abs/2305.10790) [find](https://scholar.google.com/scholar?q=Yuan+Gong%2C+Hongyin+Luo%2C+Alexander+H%2E+Liu%2C+Leonid+Karlinsky%2C+and+James+R%2E+Glass%2E+2023%2E+Listen%2C+think%2C+and+understand%2E+CoRR%2C+abs%2F2305%2E10790%2E+%E2%80%94+%5BarXiv%3A2305%2E10790%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2305%2E10790%29)
30. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023a. Gpteval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634. [find](https://scholar.google.com/scholar?q=Yang+Liu%2C+Dan+Iter%2C+Yichong+Xu%2C+Shuohang+Wang%2C+Ruochen+Xu%2C+and+Chenguang+Zhu%2E+2023a%2E+Gpteval%3A+Nlg+evaluation+using+gpt%2D4+with+better+human+alignment%2E+arXiv+preprint+arXiv%3A2303%2E16634%2E)
31. Yuan Gong, Jin Yu, and James Glass. 2022. Vocalsound: A dataset for improving human vocal sounds recognition. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 151–155. IEEE. [find](https://scholar.google.com/scholar?q=Yuan+Gong%2C+Jin+Yu%2C+and+James+Glass%2E+2022%2E+Vocalsound%3A+A+dataset+for+improving+human+vocal+sounds+recognition%2E+In+ICASSP+2022%2D2022+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processi)
32. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. 2023b. [Mmbench: Is your multi-modal model an all-around player?](https://md.research.elamur.ai/free/en/papers/liu-2023-mmbench-multi-modal-model-all/) arXiv preprint arXiv:2307.06281.
33. Chien-yu Huang, Ke-Han Lu, Shih-Heng Wang, Chi-Yuan Hsiao, Chun-Yi Kuan, Haibin Wu, Siddhant Arora, Kai-Wei Chang, Jiatong Shi, Yifan Peng, Roshan S. Sharma, Shinji Watanabe, Bhiksha Ramakrishnan, Shady Shehata, and Hung-yi Lee. 2023a. [find](https://scholar.google.com/scholar?q=Chien%2Dyu+Huang%2C+Ke%2DHan+Lu%2C+Shih%2DHeng+Wang%2C+Chi%2DYuan+Hsiao%2C+Chun%2DYi+Kuan%2C+Haibin+Wu%2C+Siddhant+Arora%2C+Kai%2DWei+Chang%2C+Jiatong+Shi%2C+Yifan+Peng%2C+Roshan+S%2E+Sharma%2C+Shinji+Watanabe%2C+Bhiksha+Ramakrishnan%2C+Sha)
34. Steven R Livingstone and Frank A Russo. 2018. The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS one, 13(5):e0196391. [find](https://scholar.google.com/scholar?q=Steven+R+Livingstone+and+Frank+A+Russo%2E+2018%2E+The+ryerson+audio%2Dvisual+database+of+emotional+speech+and+song+%28ravdess%29%3A+A+dynamic%2C+multimodal+set+of+facial+and+vocal+expressions+in+north+american+engl)
35. Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, and Zhaopeng Tu. 2023. [Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration](https://md.research.elamur.ai/free/en/papers/lyu-2023-macaw-llm-multi-modal-language/). CoRR, abs/2306.09093.
36. Annamaria Mesaros, Toni Heittola, Aleksandr Diment, Benjamin Elizalde, Ankit Shah, Emmanuel Vincent, Bhiksha Raj, and Tuomas Virtanen. 2017. [find](https://scholar.google.com/scholar?q=Annamaria+Mesaros%2C+Toni+Heittola%2C+Aleksandr+Diment%2C+Benjamin+Elizalde%2C+Ankit+Shah%2C+Emmanuel+Vincent%2C+Bhiksha+Raj%2C+and+Tuomas+Virtanen%2E+2017%2E)
37. Dcase 2017 challenge setup: Tasks, datasets and baseline system. In DCASE 2017-Workshop on Detection and Classification of Acoustic Scenes and Events. [find](https://scholar.google.com/scholar?q=Dcase+2017+challenge+setup%3A+Tasks%2C+datasets+and+baseline+system%2E+In+DCASE+2017%2DWorkshop+on+Detection+and+Classification+of+Acoustic+Scenes+and+Events%2E)
38. Arsha Nagrani, Joon Son Chung, Weidi Xie, and Andrew Zisserman. 2020. Voxceleb: Large-scale speaker verification in the wild. Computer Speech & Language, 60:101027. [find](https://scholar.google.com/scholar?q=Arsha+Nagrani%2C+Joon+Son+Chung%2C+Weidi+Xie%2C+and+Andrew+Zisserman%2E+2020%2E+Voxceleb%3A+Large%2Dscale+speaker+verification+in+the+wild%2E+Computer+Speech+%26+Language%2C+60%3A101027%2E)
39. OpenAI. 2022. Introducing ChatGPT. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2022%2E+Introducing+ChatGPT%2E)
40. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/).
41. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210. IEEE. [find](https://scholar.google.com/scholar?q=Vassil+Panayotov%2C+Guoguo+Chen%2C+Daniel+Povey%2C+and+Sanjeev+Khudanpur%2E+2015%2E+Librispeech%3A+an+asr+corpus+based+on+public+domain+audio+books%2E+In+2015+IEEE+international+conference+on+acoustics%2C+speech+and+)
42. Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2018. [Meld: A multimodal multi-party dataset for emotion recognition in conversations](https://md.research.elamur.ai/free/en/papers/poria-2018-meld-multimodal-multi-party-dataset/). arXiv preprint arXiv:1810.02508.
43. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. [Robust speech recognition via large-scale weak supervision](https://md.research.elamur.ai/free/en/papers/radford-2022-robust-speech-recognition-large-scale/). In International Conference on Machine Learning, pages 28492–28518. PMLR.
44. Ricardo Reimao and Vassilios Tzerpos. 2019. For: A dataset for synthetic speech detection. In 2019 International Conference on Speech Technology and Human-Computer Dialogue (SpeD), pages 1–10. IEEE. [find](https://scholar.google.com/scholar?q=Ricardo+Reimao+and+Vassilios+Tzerpos%2E+2019%2E+For%3A+A+dataset+for+synthetic+speech+detection%2E+In+2019+International+Conference+on+Speech+Technology+and+Human%2DComputer+Dialogue+%28SpeD%29%2C+pages+1%E2%80%9310%2E+IEEE%2E)
45. Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. 2023. [Pandagpt: One model to instruction-follow them all](https://md.research.elamur.ai/free/en/papers/su-2023-pandagpt-one-model-instruction-follow/). arXiv preprint arXiv:2305.16355.
46. Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2023a. [SALMONN: towards generic hearing abilities for large language models](https://md.research.elamur.ai/free/en/papers/tang-2023-salmonn-towards-generic-hearing-abilities/). CoRR, abs/2310.13289.
47. Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2023b. [Salmonn: Towards generic hearing abilities for large language models](https://md.research.elamur.ai/free/en/papers/tang-2023-salmonn-towards-generic-hearing-abilities/). arXiv preprint arXiv:2310.13289.
48. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. [LLaMA: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv:2302.13971.
49. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+Dan+Bikel%2C+Lukas+Blecher%2C+Cristian+Canton%2DFe)
50. Yu Shu, Siwei Dong, Guangyao Chen, Wenhao Huang, Ruihua Zhang, Daochen Shi, Qiqi Xiang, and Yemin Shi. 2023. Llasm: Large language and speech model. arXiv:2308.15930. — [arXiv:2308.15930](https://arxiv.org/abs/2308.15930) [find](https://scholar.google.com/scholar?q=Yu+Shu%2C+Siwei+Dong%2C+Guangyao+Chen%2C+Wenhao+Huang%2C+Ruihua+Zhang%2C+Daochen+Shi%2C+Qiqi+Xiang%2C+and+Yemin+Shi%2E+2023%2E+Llasm%3A+Large+language+and+speech+model%2E+arXiv%3A2308%2E15930%2E+%E2%80%94+%5BarXiv%3A2308%2E15930%5D%28https%3A%2F%2Farxi)
51. Joseph Turian, Jordie Shier, Humair Raj Khan, Bhiksha Raj, Björn W. Schuller, Christian J. Steinmetz, Colin Malloy, George Tzanetakis, Gissel Velarde, Kirk McNally, Max Henry, Nicolas Pinto, Camille Noufi, Christian Clough, Dorien Herremans, Eduardo Fonseca, Jesse H. Engel, Justin Salamon, Philippe… [find](https://scholar.google.com/scholar?q=Joseph+Turian%2C+Jordie+Shier%2C+Humair+Raj+Khan%2C+Bhiksha+Raj%2C+Bj%C3%B6rn+W%2E+Schuller%2C+Christian+J%2E+Steinmetz%2C+Colin+Malloy%2C+George+Tzanetakis%2C+Gissel+Velarde%2C+Kirk+McNally%2C+Max+Henry%2C+Nicolas+Pinto%2C+Camille+N)
52. Shuzheng Si, Wentao Ma, Yuchuan Wu, Yinpei Dai, Haoyu Gao, Ting-En Lin, Hangyu Li, Rui Yan, Fei Huang, and Yongbin Li. 2023. Spokenwoz: A large-scale speech-text benchmark for spoken task-oriented dialogue in multiple domains. arXiv preprint arXiv:2305.13040. — [arXiv:2305.13040](https://arxiv.org/abs/2305.13040) [find](https://scholar.google.com/scholar?q=Shuzheng+Si%2C+Wentao+Ma%2C+Yuchuan+Wu%2C+Yinpei+Dai%2C+Haoyu+Gao%2C+Ting%2DEn+Lin%2C+Hangyu+Li%2C+Rui+Yan%2C+Fei+Huang%2C+and+Yongbin+Li%2E+2023%2E+Spokenwoz%3A+A+large%2Dscale+speech%2Dtext+benchmark+for+spoken+task%2Doriented+dia)
53. Joseph Turian, Jordie Shier, Humair Raj Khan, Bhiksha Raj, Björn W Schuller, Christian J Steinmetz, Colin Malloy, George Tzanetakis, Gissel Velarde, Kirk Mc-Nally, et al. 2022. Hear: Holistic evaluation of audio representations. In NeurIPS 2021 Competitions and Demonstrations Track, pages 125–145. P… [find](https://scholar.google.com/scholar?q=Joseph+Turian%2C+Jordie+Shier%2C+Humair+Raj+Khan%2C+Bhiksha+Raj%2C+Bj%C3%B6rn+W+Schuller%2C+Christian+J+Steinmetz%2C+Colin+Malloy%2C+George+Tzanetakis%2C+Gissel+Velarde%2C+Kirk+Mc%2DNally%2C+et+al%2E+2022%2E+Hear%3A+Holistic+evaluati)
54. Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. [Hugginggpt: Solving AI tasks with chatgpt and its friends in huggingface](https://md.research.elamur.ai/free/en/papers/shen-2023-hugginggpt-solving-ai-tasks-chatgpt/). CoRR, abs/2303.17580.
55. Changhan Wang, Juan Pino, Anne Wu, and Jiatao Gu. 2020a. CoVoST: A diverse multilingual speech-to-text translation corpus. In Proceedings of The 12th Language Resources and Evaluation Conference. [find](https://scholar.google.com/scholar?q=Changhan+Wang%2C+Juan+Pino%2C+Anne+Wu%2C+and+Jiatao+Gu%2E+2020a%2E+CoVoST%3A+A+diverse+multilingual+speech%2Dto%2Dtext+translation+corpus%2E+In+Proceedings+of+The+12th+Language+Resources+and+Evaluation+Conference%2E)
56. Changhan Wang, Anne Wu, and Juan Pino. 2020b. Covost 2 and massively multilingual speech-to-text translation. arXiv preprint arXiv:2007.10310. [find](https://scholar.google.com/scholar?q=Changhan+Wang%2C+Anne+Wu%2C+and+Juan+Pino%2E+2020b%2E+Covost+2+and+massively+multilingual+speech%2Dto%2Dtext+translation%2E+arXiv+preprint+arXiv%3A2007%2E10310%2E)
57. Chen Wang, Minpeng Liao, Zhongqiang Huang, Jinliang Lu, Junhong Wu, Yuchen Liu, Chengqing Zong, and Jiajun Zhang. 2023a. Blsp: Bootstrapping language-speech pre-training via behavior alignment of continuation writing. arXiv preprint arXiv:2309.00916. [find](https://scholar.google.com/scholar?q=Chen+Wang%2C+Minpeng+Liao%2C+Zhongqiang+Huang%2C+Jinliang+Lu%2C+Junhong+Wu%2C+Yuchen+Liu%2C+Chengqing+Zong%2C+and+Jiajun+Zhang%2E+2023a%2E+Blsp%3A+Bootstrapping+language%2Dspeech+pre%2Dtraining+via+behavior+alignment+of+cont)
58. Mingqiu Wang, Wei Han, Izhak Shafran, Zelin Wu, Chung-Cheng Chiu, Yuan Cao, Yongqiang Wang, Nanxin Chen, Yu Zhang, Hagen Soltau, et al. 2023b. Slm: Bridge the thin gap between speech and text foundation models. arXiv:2310.00230. [find](https://scholar.google.com/scholar?q=Mingqiu+Wang%2C+Wei+Han%2C+Izhak+Shafran%2C+Zelin+Wu%2C+Chung%2DCheng+Chiu%2C+Yuan+Cao%2C+Yongqiang+Wang%2C+Nanxin+Chen%2C+Yu+Zhang%2C+Hagen+Soltau%2C+et+al%2E+2023b%2E+Slm%3A+Bridge+the+thin+gap+between+speech+and+text+foundati)
59. Jian Wu, Yashesh Gaur, Zhuo Chen, Long Zhou, Yimeng Zhu, Tianrui Wang, Jinyu Li, Shujie Liu, Bo Ren, Linquan Liu, and Yu Wu. 2023a. On decoder-only architecture for speech-to-text and large language model integration. abs/2307.03917. — [arXiv:2307.03917](https://arxiv.org/abs/2307.03917) [find](https://scholar.google.com/scholar?q=Jian+Wu%2C+Yashesh+Gaur%2C+Zhuo+Chen%2C+Long+Zhou%2C+Yimeng+Zhu%2C+Tianrui+Wang%2C+Jinyu+Li%2C+Shujie+Liu%2C+Bo+Ren%2C+Linquan+Liu%2C+and+Yu+Wu%2E+2023a%2E+On+decoder%2Donly+architecture+for+speech%2Dto%2Dtext+and+large+language+m)
60. Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. 2023b. Next-gpt: Any-to-any multimodal LLM. CoRR, abs/2309.05519. [find](https://scholar.google.com/scholar?q=Shengqiong+Wu%2C+Hao+Fei%2C+Leigang+Qu%2C+Wei+Ji%2C+and+Tat%2DSeng+Chua%2E+2023b%2E+Next%2Dgpt%3A+Any%2Dto%2Dany+multimodal+LLM%2E+CoRR%2C+abs%2F2309%2E05519%2E)
61. Xuenan Xu, Heinrich Dinkel, Mengyue Wu, and Kai Yu. 2021. Text-to-audio grounding: Building correspondence between captions and sound events. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 606–610. IEEE. [find](https://scholar.google.com/scholar?q=Xuenan+Xu%2C+Heinrich+Dinkel%2C+Mengyue+Wu%2C+and+Kai+Yu%2E+2021%2E+Text%2Dto%2Daudio+grounding%3A+Building+correspondence+between+captions+and+sound+events%2E+In+ICASSP+2021%2D2021+IEEE+International+Conference+on+Acous)
62. Pinci Yang, Xin Wang, Xuguang Duan, Hong Chen, Runze Hou, Cong Jin, and Wenwu Zhu. 2022. Avqa: A dataset for audio-visual question answering on videos. In Proceedings of the 30th ACM International Conference on Multimedia, pages 3480–3491. [find](https://scholar.google.com/scholar?q=Pinci+Yang%2C+Xin+Wang%2C+Xuguang+Duan%2C+Hong+Chen%2C+Runze+Hou%2C+Cong+Jin%2C+and+Wenwu+Zhu%2E+2022%2E+Avqa%3A+A+dataset+for+audio%2Dvisual+question+answering+on+videos%2E+In+Proceedings+of+the+30th+ACM+International+Con)
63. Shu-Wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko-tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-Wen Li, Shinji Watanabe, Abdelrahman Mohamed, and Hung-yi Lee.… [find](https://scholar.google.com/scholar?q=Shu%2DWen+Yang%2C+Po%2DHan+Chi%2C+Yung%2DSung+Chuang%2C+Cheng%2DI+Jeff+Lai%2C+Kushal+Lakhotia%2C+Yist+Y%2E+Lin%2C+Andy+T%2E+Liu%2C+Jiatong+Shi%2C+Xuankai+Chang%2C+Guan%2DTing+Lin%2C+Tzu%2DHsien+Huang%2C+Wei%2DCheng+Tseng%2C+Ko%2Dtik+Lee%2C+Da%2DRon)
64. Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y Lin, Andy T Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, et al. 2021b. Superb: Speech processing universal performance benchmark. arXiv preprint arXiv:2105.01051. — [arXiv:2105.01051](https://arxiv.org/abs/2105.01051) [find](https://scholar.google.com/scholar?q=Shu%2Dwen+Yang%2C+Po%2DHan+Chi%2C+Yung%2DSung+Chuang%2C+Cheng%2DI+Jeff+Lai%2C+Kushal+Lakhotia%2C+Yist+Y+Lin%2C+Andy+T+Liu%2C+Jiatong+Shi%2C+Xuankai+Chang%2C+Guan%2DTing+Lin%2C+et+al%2E+2021b%2E+Superb%3A+Speech+processing+universal+perf)
65. Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. 2023. [Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities](https://md.research.elamur.ai/free/en/papers/zhang-2023-speechgpt-empowering-large-language-models/). arXiv preprint arXiv:2305.11000.
66. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685. [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric+Xing%2C+et+al%2E+2023%2E+Judging+llm%2Das%2Da%2Djudge+with+mt%2Dbench+and+chatbot+arena%2E+a)

*Updated 24 October 2026*