---
title: "Omni-Embed-Audio — paper by Yoo et al. (2026)"
description: "The authors introduce Omni-Embed-Audio (OEA), which turns a multimodal LLM with built-in audio… 24 references, cited by 1 paper here."
authors: ["HaeJun Yoo", "Yongseop Shin", "Insung Lee", "Myoung-Wan Koo", "Du-Seong Chang"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.1038/"
doi: "10.18653/v1/2026.acl-long.1038"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yoo-2026-omni-embed-audio-leveraging-multimodal/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yoo-2026-omni-embed-audio-leveraging-multimodal/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/15/) / Omni-Embed-Audio: Leveraging Multimodal LLMs for Robust Audio-Text Retrieval

# Omni-Embed-Audio: Leveraging Multimodal LLMs for Robust Audio-Text Retrieval

**Authors:** HaeJun Yoo, Yongseop Shin, Insung Lee, Myoung-Wan Koo and Du-Seong Chang · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.1038/) · [PDF](https://aclanthology.org/2026.acl-long.1038.pdf) · [DOI 10.18653/v1/2026.acl-long.1038](https://doi.org/10.18653/v1/2026.acl-long.1038) · [OpenAlex](https://openalex.org/W7166806903)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most audio-text retrieval systems are built on CLAP and tested with caption-like queries. Real users search differently: they type short tags, ask questions, give commands, or say what they want to exclude. The authors introduce Omni-Embed-Audio (OEA), which turns a multimodal LLM with built-in audio understanding into a single shared encoder for both text and audio. The backbone stays frozen. Only LoRA adapters on the attention layers and modality-specific projection heads into a 512-dimensional space are trained, using a symmetric InfoNCE objective.

To test robustness, they build the User-Intent Queries (UIQ) benchmark on AudioCaps, Clotho and MECAT. It has five query types, and the exclusion-based queries are paired with mined hard negatives. They also propose new discrimination metrics, HNSR and TFR.

On text-to-audio retrieval, OEA roughly matches the state-of-the-art M2D-CLAP. It clearly leads on text-to-text retrieval and on rejecting acoustically similar distractors. The paper therefore suggests M2D-CLAP for in-domain caption-style search and OEA for caption-indexed, exclusion-aware or cross-domain use.

## Key points
- **Backbones and training cost:** OEA is built on Omni-Embed-Nemotron-3B, Qwen2.5-Omni-3B and Qwen2.5-Omni-7B. Only about 11–16M parameters are trained (0.29–0.36% of each model), and OEA-Nemo3B encodes a text query in 2.3 ms.
- **UIQ benchmark:**
  - It has five query types in three groups: Question and Imperative (conversational), Keyphrase and Paraphrase (reformulation), and Negative (exclusionary).
  - The queries were generated with GPT-5.1, with lengths kept within ±2 words of the source captions.
- **Released data and validation:**
  - The release comprises AudioCaps-UIQ, Clotho-UIQ and MECAT-UIQ, totalling 13,053 queries (4,530 / 4,722 / 3,801).
  - In a human check by 9 annotators (675 ratings), queries scored a mean of 4.15.
- **Main results** (OEA-Qwen7B, mean R@5):
  - Text-to-audio: 46.4% vs 47.9% for M2D-CLAP.
  - Text-to-text: 64.8% vs 59.3%.
  - Imperative queries: 49.9% vs 44.7%.
  - Hard-negative discrimination: HNSR@10 of 34.6% vs 30.3%, plus a 34.7% relative gain in TFR@10.
- **Scaling:** The 7B backbone does not consistently beat the 3B ones. The authors conclude that contrastive alignment and dataset-backbone fit matter more than raw model size.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Sogang University](https://md.research.elamur.ai/free/en/universities/sogang-university/) — University, Seoul, South Korea · 3 works · 55 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works · [🔊 Signal Processing](https://md.research.elamur.ai/free/en/shelves/computer-science/signal-processing/) · 452 works

🧭 Topics (4): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 657 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 473 works · [🎙️ Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) · 480 works

## Cite

Cite as: HaeJun Yoo, Yongseop Shin, Insung Lee, Myoung-Wan Koo and Du-Seong Chang. 2026. Omni-Embed-Audio: Leveraging Multimodal LLMs for Robust Audio-Text Retrieval. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yoo2026omni,
  title = {{Omni-Embed-Audio: Leveraging Multimodal LLMs for Robust Audio-Text Retrieval}},
  author = {HaeJun Yoo, Yongseop Shin, Insung Lee, Myoung-Wan Koo and Du-Seong Chang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1038},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1038}
}
```

- [← previous OLA: Output Language Alignment in Code-Switched LLM Interactions](https://md.research.elamur.ai/free/en/papers/oh-2026-ola-output-language-alignment-code/)
- [next → OmniDiagram: Advancing Unified Diagram Code Generation via Visual Interrogation Reward](https://md.research.elamur.ai/free/en/papers/yang-2026-omnidiagram-advancing-unified-diagram-code/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/15/)

## References

24 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Claudio Carpineto and Giovanni Romano. 2012. A survey of automatic query expansion in information retrieval. ACM Computing Surveys, 44(1):1–50. [find](https://scholar.google.com/scholar?q=Claudio+Carpineto+and+Giovanni+Romano%2E+2012%2E+A+survey+of+automatic+query+expansion+in+information+retrieval%2E+ACM+Computing+Surveys%2C+44%281%29%3A1%E2%80%9350%2E)
2. Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2024. [Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models](https://md.research.elamur.ai/free/en/papers/chu-2023-qwen-audio-advancing-universal/). arXiv preprint arXiv:2311.07919.
3. Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. 2020. Clotho: An audio captioning dataset. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 736– 740. IEEE. [find](https://scholar.google.com/scholar?q=Konstantinos+Drossos%2C+Samuel+Lipping%2C+and+Tuomas+Virtanen%2E+2020%2E+Clotho%3A+An+audio+captioning+dataset%2E+In+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+%28ICASSP%29%2C+pages+736%E2%80%93+7)
4. Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. 2023. CLAP: Learning audio concepts from natural language supervision. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. [find](https://scholar.google.com/scholar?q=Benjamin+Elizalde%2C+Soham+Deshmukh%2C+Mahmoud+Al+Ismail%2C+and+Huaming+Wang%2E+2023%2E+CLAP%3A+Learning+audio+concepts+from+natural+language+supervision%2E+In+IEEE+International+Conference+on+Acoustics%2C+Speech+and)
5. Frederic Font, Gerard Roma, and Xavier Serra. 2013. Freesound technical demo. In Proceedings of the 21st ACM International Conference on Multimedia, pages 411–412. ACM. [find](https://scholar.google.com/scholar?q=Frederic+Font%2C+Gerard+Roma%2C+and+Xavier+Serra%2E+2013%2E+Freesound+technical+demo%2E+In+Proceedings+of+the+21st+ACM+International+Conference+on+Multimedia%2C+pages+411%E2%80%93412%2E+ACM%2E)
6. Sreyan Ghosh, Ashish Seth, Sonal Kumar, Utkarsh Tyagi, Chandra Kiran Reddy Evuru, S Ramaneswaran, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. 2024. CompA: Addressing the gap in compositional reasoning in audio-language models. In The Twelfth International Conference on Learning Rep… [find](https://scholar.google.com/scholar?q=Sreyan+Ghosh%2C+Ashish+Seth%2C+Sonal+Kumar%2C+Utkarsh+Tyagi%2C+Chandra+Kiran+Reddy+Evuru%2C+S+Ramaneswaran%2C+S+Sakshi%2C+Oriol+Nieto%2C+Ramani+Duraiswami%2C+and+Dinesh+Manocha%2E+2024%2E+CompA%3A+Addressing+the+gap+in+compo)
7. Yunah Jang, Kang-il Lee, Hyunkyung Bae, Hwanhee Lee, and Kyomin Jung. 2024. Itercqr: Iterative conversational query reformulation with retrieval guidance. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies… [find](https://scholar.google.com/scholar?q=Yunah+Jang%2C+Kang%2Dil+Lee%2C+Hyunkyung+Bae%2C+Hwanhee+Lee%2C+and+Kyomin+Jung%2E+2024%2E+Itercqr%3A+Iterative+conversational+query+reformulation+with+retrieval+guidance%2E+In+Proceedings+of+the+2024+Conference+of+the+)
8. Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. 2019. [AudioCaps: Generating captions for audios in the wild](https://md.research.elamur.ai/free/en/papers/kim-2019-audiocaps-generating-captions-audios-wild/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 119–132.
9. A Sophia Koepke, Andreea-Maria Oncescu, João F Henriques, Zeynep Akata, and Samuel Albanie. 2023. Audio retrieval with natural language queries: A benchmark study. IEEE Transactions on Multimedia, 25:2675–2685. [find](https://scholar.google.com/scholar?q=A+Sophia+Koepke%2C+Andreea%2DMaria+Oncescu%2C+Jo%C3%A3o+F+Henriques%2C+Zeynep+Akata%2C+and+Samuel+Albanie%2E+2023%2E+Audio+retrieval+with+natural+language+queries%3A+A+benchmark+study%2E+IEEE+Transactions+on+Multimedia%2C+25%3A)
10. Sangho Lee, Jiho Bui, Zihang Lu, Yuta Yoshitomo, Sung Ju Chang, Joon Son Hwang, and Gunhee Lee. 2021. ACAV100M: Automatic curation of large-scale datasets for audio-visual video representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10274–10… [find](https://scholar.google.com/scholar?q=Sangho+Lee%2C+Jiho+Bui%2C+Zihang+Lu%2C+Yuta+Yoshitomo%2C+Sung+Ju+Chang%2C+Joon+Son+Hwang%2C+and+Gunhee+Lee%2E+2021%2E+ACAV100M%3A+Automatic+curation+of+large%2Dscale+datasets+for+audio%2Dvisual+video+representation+learnin)
11. Yiming Li, Zhifang Guo, Xiangdong Wang, and Hong Liu. 2024. Advancing multi-grained alignment for contrastive language-audio pre-training. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 7356–7365. [find](https://scholar.google.com/scholar?q=Yiming+Li%2C+Zhifang+Guo%2C+Xiangdong+Wang%2C+and+Hong+Liu%2E+2024%2E+Advancing+multi%2Dgrained+alignment+for+contrastive+language%2Daudio+pre%2Dtraining%2E+In+Proceedings+of+the+32nd+ACM+International+Conference+on+Mu)
12. Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023. [Query rewriting in retrieval-augmented large language models](https://md.research.elamur.ai/free/en/papers/ma-2023-query-rewriting-retrieval-augmented-large/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5303–5315. Association for Computational Linguistics.
13. Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D Plumbley, Yuexian Zou, and Wenwu Wang. 2024. WavCaps: A ChatGPT-assisted weakly-labelled audio captioning dataset for audio-language multimodal research. IEEE/ACM Transactions on Audio, Speech, and Language Processing. [find](https://scholar.google.com/scholar?q=Xinhao+Mei%2C+Chutong+Meng%2C+Haohe+Liu%2C+Qiuqiang+Kong%2C+Tom+Ko%2C+Chengqi+Zhao%2C+Mark+D+Plumbley%2C+Yuexian+Zou%2C+and+Wenwu+Wang%2E+2024%2E+WavCaps%3A+A+ChatGPT%2Dassisted+weakly%2Dlabelled+audio+captioning+dataset+for+a)
14. Daisuke Niizumi, Daiki Takeuchi, Masahiro Yasuda, Binh Thien Nguyen, Yasunori Ohishi, and Noboru Harada. 2025. M2D-CLAP: Exploring general-purpose audio-language representations beyond CLAP. IEEE Access. [find](https://scholar.google.com/scholar?q=Daisuke+Niizumi%2C+Daiki+Takeuchi%2C+Masahiro+Yasuda%2C+Binh+Thien+Nguyen%2C+Yasunori+Ohishi%2C+and+Noboru+Harada%2E+2025%2E+M2D%2DCLAP%3A+Exploring+general%2Dpurpose+audio%2Dlanguage+representations+beyond+CLAP%2E+IEEE+Acce)
15. Gustavo Penha, Arthur Câmara, and Claudia Hauff. 2022. Evaluating the robustness of retrieval pipelines with query variation generators. In Advances in Information Retrieval: 44th European Conference on IR Research (ECIR), pages 397–412. Springer. [find](https://scholar.google.com/scholar?q=Gustavo+Penha%2C+Arthur+C%C3%A2mara%2C+and+Claudia+Hauff%2E+2022%2E+Evaluating+the+robustness+of+retrieval+pipelines+with+query+variation+generators%2E+In+Advances+in+Information+Retrieval%3A+44th+European+Conference+)
16. Ramaneswaran Selvakumar, Sonal Kumar, Hemant Kumar Giri, Nishit Anand, Ashish Seth, Sreyan Ghosh, and Dinesh Manocha. 2024. Do audio-language models understand linguistic variations? arXiv preprint arXiv:2410.16505. [find](https://scholar.google.com/scholar?q=Ramaneswaran+Selvakumar%2C+Sonal+Kumar%2C+Hemant+Kumar+Giri%2C+Nishit+Anand%2C+Ashish+Seth%2C+Sreyan+Ghosh%2C+and+Dinesh+Manocha%2E+2024%2E+Do+audio%2Dlanguage+models+understand+linguistic+variations%3F+arXiv+preprint+ar)
17. Benno Weck and Frederic Font. 2024. The language of sound search: Examining user queries in audio search engines. In Proceedings of the 9th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE). [find](https://scholar.google.com/scholar?q=Benno+Weck+and+Frederic+Font%2E+2024%2E+The+language+of+sound+search%3A+Examining+user+queries+in+audio+search+engines%2E+In+Proceedings+of+the+9th+Workshop+on+Detection+and+Classification+of+Acoustic+Scenes+)
18. Orion Weller, Dawn Lawrie, and Benjamin Van Durme. 2023. NevIR: Negation in neural information retrieval. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), pages 4163–4173. [find](https://scholar.google.com/scholar?q=Orion+Weller%2C+Dawn+Lawrie%2C+and+Benjamin+Van+Durme%2E+2023%2E+NevIR%3A+Negation+in+neural+information+retrieval%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28AC)
19. Anonymous Wu and 1 others. 2025. Mecat: A benchmark for evaluating audio-text retrieval with minimal data contamination. arXiv preprint arXiv:2507.23511. Under review. — [arXiv:2507.23511](https://arxiv.org/abs/2507.23511) [find](https://scholar.google.com/scholar?q=Anonymous+Wu+and+1+others%2E+2025%2E+Mecat%3A+A+benchmark+for+evaluating+audio%2Dtext+retrieval+with+minimal+data+contamination%2E+arXiv+preprint+arXiv%3A2507%2E23511%2E+Under+review%2E+%E2%80%94+%5BarXiv%3A2507%2E23511%5D%28https%3A%2F%2Farx)
20. Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. 2023. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5.… [find](https://scholar.google.com/scholar?q=Yusong+Wu%2C+Ke+Chen%2C+Tianyu+Zhang%2C+Yuchen+Hui%2C+Taylor+Berg%2DKirkpatrick%2C+and+Shlomo+Dubnov%2E+2023%2E+Large%2Dscale+contrastive+language%2Daudio+pretraining+with+feature+fusion+and+keyword%2Dto%2Dcaption+augmentati)
21. Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023. [C-Pack: Packaged resources to advance general chinese embedding](https://md.research.elamur.ai/free/en/papers/xiao-2023-c-pack-packed-resources/). arXiv preprint arXiv:2309.07597.
22. Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. 2025a. [Qwen2.5-omni technical report](https://md.research.elamur.ai/free/en/papers/xu-2025-qwen2-5-omni-technical/). arXiv preprint arXiv:2503.20215.
23. Mengyao Xu, Wenfei Zhou, Yauhen Babakhin, Gabriel Moreira, Ronay Ak, Radek Osmulski, Bo Liu, Even Oldridge, and Benedikt Schifferer. 2025b. Omni-embed-nemotron: A unified multimodal retrieval model for text, image, audio, and video. arXiv preprint arXiv:2510.03458. — [arXiv:2510.03458](https://arxiv.org/abs/2510.03458) [find](https://scholar.google.com/scholar?q=Mengyao+Xu%2C+Wenfei+Zhou%2C+Yauhen+Babakhin%2C+Gabriel+Moreira%2C+Ronay+Ak%2C+Radek+Osmulski%2C+Bo+Liu%2C+Even+Oldridge%2C+and+Benedikt+Schifferer%2E+2025b%2E+Omni%2Dembed%2Dnemotron%3A+A+unified+multimodal+retrieval+model+fo)
24. Wenhao Zhang and 1 others. 2024. ExcluIR: Exclusionary neural information retrieval. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Wenhao+Zhang+and+1+others%2E+2024%2E+ExcluIR%3A+Exclusionary+neural+information+retrieval%2E+In+Proceedings+of+the+62nd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28ACL%29%2E)

*Updated 31 October 2026*