--- title: "Benchmarking Open-ended Audio Dialogue Understanding for" description: "Large Audio-Language Models (LALMs) such as GPT-4o can now hold spoken conversations directly… 1 reference, cited by 8 papers here." authors: ["Kuofeng Gao", "Shu-Tao Xia", "Ke Xu", "Philip Torr", "Jindong Gu"] year: 2025 venue: "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)" source: "https://aclanthology.org/2025.acl-long.237/" doi: "10.18653/v1/2025.acl-long.237" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/gao-2025-benchmarking-open-ended-audio-dialogue/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gao-2025-benchmarking-open-ended-audio-dialogue/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/7/) / Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Models # Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Models **Authors:** [Kuofeng Gao](https://md.research.elamur.ai/free/en/authors/kuofeng-gao/), [Shu-Tao Xia](https://md.research.elamur.ai/free/en/authors/shu-tao-xia/), [Ke Xu](https://md.research.elamur.ai/free/en/authors/ke-xu/), [Philip Torr](https://md.research.elamur.ai/free/en/authors/philip-h-s-torr/) and [Jindong Gu](https://md.research.elamur.ai/free/en/authors/jindong-gu/) · **Year:** 2025 · **Venue:** [Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2025](https://md.research.elamur.ai/free/en/venues/acl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2025.acl-long.237/) · [PDF](https://aclanthology.org/2025.acl-long.237.pdf) · [DOI 10.18653/v1/2025.acl-long.237](https://doi.org/10.18653/v1/2025.acl-long.237) · [OpenAlex](https://openalex.org/W4412886727) **Citations:** 4 citations (OpenAlex, 2026-10-14) · [8](https://md.research.elamur.ai/free/en/papers/gao-2025-benchmarking-open-ended-audio-dialogue/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Large Audio-Language Models (LALMs) such as GPT-4o can now hold spoken conversations directly with users. Until this work, however, no benchmark tested how well they understand open-ended audio dialogue. Earlier evaluations covered basic audio tasks, audio QA driven by text instructions, or general-purpose chat only. The authors built ADU-Bench, a collection of four datasets with 20,715 spoken queries. More than 8,000 are real recordings and the rest are synthesized. The datasets cover: - general conversation, - domain skills, - multilingual use, - ambiguity, a first-of-its-kind set where the same words carry different intent depending on intonation, pauses, homophones or repetition. Model replies are converted to text and scored by GPT-4 against reference answers. Each reply is scored twice, with the reference and response positions swapped, to cancel order bias. LLaMA-3-70B-Instruct and Qwen-2-72B-Instruct serve as additional judges. Across 16 LALMs, the main weaknesses were: - math and code involving formulas, - common sense and roleplay, - non-Indo-European languages, - phonetically driven ambiguity. The benchmark gives a standardized way to compare LALMs and to identify where spoken-dialogue systems need work. ## Key points - **Dataset sizes:** ADU-General has 12,000 items (helpful questions, daily questions, daily statements). ADU-Skill has 3,725 items over 12 skills. ADU-Multilingual has 3,600 items in 9 languages. ADU-Ambiguity has 1,390 items in 4 ambiguity types, drawn from phonetics/phonology books. - **Sources:** Alpaca, NQ-Bench, WebGLM, Slue HVB, Common Voice, GSM8K, MATH, MBPP, MMLU, HotpotQA, StrategyQA, among others. - **Judge validation:** GPT-4 judgments matched human pairwise preferences in over 85% of cases. Swapping positions showed real order bias, particularly when the response appeared before the reference. - **Ambiguity results:** Even GPT-4o averaged only 5.22 on pause-based and 6.05 on homophone-based ambiguity. It tended to list several possible readings rather than identify the intended one. - **Ablations:** - Real and synthetic audio gave similar scores on a 1,000-sample comparison. - Larger models scored better on average but sometimes lost to smaller ones. For example, SALMONN-7B beat SALMONN-13B on Code. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 333 works · 87,019 citations - [University of Oxford](https://md.research.elamur.ai/free/en/universities/university-of-oxford/) — University, Oxford, United Kingdom · 104 works · 157,183 citations - [Peng Cheng Laboratory](https://md.research.elamur.ai/free/en/labs/peng-cheng-laboratory/) — Research institute, Shenzhen, China · 38 works · 2,504 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🔊 Signal Processing](https://md.research.elamur.ai/free/en/shelves/computer-science/signal-processing/) · 452 works 🧭 Topics (2): [🗨️ Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) · 624 works · [🎙️ Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) · 480 works ## Cite Cite as: Kuofeng Gao, Shu-Tao Xia, Ke Xu, Philip Torr and Jindong Gu. 2025. Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Models. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{gao2025benchmarking, title = {{Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Models}}, author = {Kuofeng Gao, Shu-Tao Xia, Ke Xu, Philip Torr and Jindong Gu}, booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, year = {2025}, doi = {10.18653/v1/2025.acl-long.237}, url = {https://doi.org/10.18653/v1/2025.acl-long.237} } ``` - [← previous ArithmAttack: Evaluating Robustness of LLMs to Noisy Context in Math Problem Solving](https://md.research.elamur.ai/free/en/papers/abedin-2025-arithmattack-evaluating-robustness-llms-noisy/) - [next → BMIKE-53: Investigating Cross-Lingual Knowledge Editing with In-Context Learning](https://md.research.elamur.ai/free/en/papers/nie-2025-bmike-53-investigating-cross-lingual/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/7/) ## References 1 references, 1 of them with a card of ours. ↗ leads to the work. 1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774. \#\# Conclusion *Updated 1 November 2026*