---
title: "Step-Audio 2 Technical Report — paper by Wu et al. (2025)"
description: "Step-Audio 2 is StepFun's end-to-end large audio language model (LALM). 39.26 BLEU on CoVoST 2. 82 references, cited by 30 papers here."
authors: ["Boyong Wu", "Chao Yan", "Chen Hu", "Cheng Yi", "Chengli Feng", "Fei Tian", "Feiyu Shen", "Gang Yu", "Haoyang Zhang", "Jingbei Li", "Mingrui Chen", "Peng Liu", "Wang You", "Xiangyu Tony Zhang", "Xingyuan Li", "Xuerui Yang", "Yayue Deng", "Yechang Huang", "Yuxin Li", "Yuxin Zhang", "Zhao You", "Brian Li", "Changyi Wan", "Hanpeng Hu", "Jiangjie Zhen", "Siyu Chen", "Song Yuan", "Xuelin Zhang", "Yimin Jiang", "Yu Zhou", "Yuxiang Yang", "Bingxin Li", "Buyun Ma", "Changhe Song", "Dongqing Pang", "Guoqiang Hu", "Haiyang Sun", "Kang An", "Na Wang", "Shuli Gao", "Wei Ji", "Wen Li", "Wen Sun", "Xuan Wen", "Yong Ren", "Yuankai Ma", "Yufan Lu", "Bin Wang", "Bo Li", "Changxin Miao", "Che Liu", "Chen Xu", "Dapeng Shi", "Dingyuan Hu", "Donghang Wu", "Enle Liu", "Guanzhe Huang", "Gulin Yan", "Han Zhang", "Hao Nie", "Haonan Jia", "Hongyu Zhou", "Jianjian Sun", "Jiaoren Wu", "Jie Wu", "Jie Yang", "Jin Yang", "Junzhe Lin", "Kaixiang Li", "Lei Yang", "Liying Shi", "Li Zhou", "Longlong Gu", "Ming Li", "Mingliang Li", "Mingxiao Li", "Nan Wu", "Qi Han", "Qinyuan Tan", "Shaoliang Pang", "Shengjie Fan", "Siqi Liu", "Tiancheng Cao", "Wanying Lu", "Wenqing He", "Wuxun Xie", "Xu Zhao", "Xueqi Li", "Yanbo Yu", "Yang Yang", "Yi Liu", "Yifan Lu", "Yilei Wang", "Yuanhao Ding", "Yuanwei Liang", "Yuanwei Lu", "Yuchu Luo", "Yuhe Yin", "Yumeng Zhan", "Yuxiang Zhang", "Zidong Yang", "Zixin Zhang", "Binxing Jiao", "Daxin Jiang", "Heung-Yeung Shum", "Jiansheng Chen", "Jing Li", "Xiangyu Zhang", "Yibo Zhu"]
year: 2025
venue: "arXiv:2507.16632"
source: "https://arxiv.org/abs/2507.16632"
doi: "10.48550/arxiv.2507.16632"
arxiv: "2507.16632"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wu-2025-step-audio-2-technical/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wu-2025-step-audio-2-technical/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/3/) / Step-Audio 2 Technical Report

# Step-Audio 2 Technical Report

**Authors:** [Boyong Wu](https://md.research.elamur.ai/free/en/authors/boyong-wu/), [Chao Yan](https://md.research.elamur.ai/free/en/authors/chao-yan/), [Chen Hu](https://md.research.elamur.ai/free/en/authors/chen-hu/), Cheng Yi, [Chengli Feng](https://md.research.elamur.ai/free/en/authors/chengli-feng/), [Fei Tian](https://md.research.elamur.ai/free/en/authors/fei-tian/), [Feiyu Shen](https://md.research.elamur.ai/free/en/authors/feiyu-shen/), [Gang Yu](https://md.research.elamur.ai/free/en/authors/gang-yu/), [Haoyang Zhang](https://md.research.elamur.ai/free/en/authors/haoyang-zhang/), [Jingbei Li](https://md.research.elamur.ai/free/en/authors/jingbei-li/), [Mingrui Chen](https://md.research.elamur.ai/free/en/authors/mingrui-chen/), [Peng Liu](https://md.research.elamur.ai/free/en/authors/peng-liu/), Wang You, Xiangyu Tony Zhang, Xingyuan Li, Xuerui Yang, [Yayue Deng](https://md.research.elamur.ai/free/en/authors/yayue-deng/), Yechang Huang, [Yuxin Li](https://md.research.elamur.ai/free/en/authors/yuxin-li/), Yuxin Zhang, Zhao You, Brian Li, [Changyi Wan](https://md.research.elamur.ai/free/en/authors/changyi-wan/), Hanpeng Hu, Jiangjie Zhen, [Siyu Chen](https://md.research.elamur.ai/free/en/authors/siyu-chen/), Song Yuan, Xuelin Zhang, Yimin Jiang, [Yu Zhou](https://md.research.elamur.ai/free/en/authors/yu-zhou/), Yuxiang Yang, Bingxin Li, Buyun Ma, Changhe Song, Dongqing Pang, Guoqiang Hu, Haiyang Sun, Kang An, Na Wang, Shuli Gao, Wei Ji, Wen Li, Wen Sun, Xuan Wen, Yong Ren, Yuankai Ma, Yufan Lu, [Bin Wang](https://md.research.elamur.ai/free/en/authors/bin-wang/), [Bo Li](https://md.research.elamur.ai/free/en/authors/bo-li/), Changxin Miao, [Che Liu](https://md.research.elamur.ai/free/en/authors/che-liu/), [Chen Xu](https://md.research.elamur.ai/free/en/authors/chen-xu/), Dapeng Shi, Dingyuan Hu, [Donghang Wu](https://md.research.elamur.ai/free/en/authors/donghang-wu/), Enle Liu, Guanzhe Huang, Gulin Yan, [Han Zhang](https://md.research.elamur.ai/free/en/authors/han-zhang/), Hao Nie, Haonan Jia, Hongyu Zhou, Jianjian Sun, Jiaoren Wu, Jie Wu, [Jie Yang](https://md.research.elamur.ai/free/en/authors/jie-yang/), Jin Yang, Junzhe Lin, Kaixiang Li, [Lei Yang](https://md.research.elamur.ai/free/en/authors/lei-yang/), Liying Shi, [Li Zhou](https://md.research.elamur.ai/free/en/authors/li-zhou/), Longlong Gu, [Ming Li](https://md.research.elamur.ai/free/en/authors/ming-li/), Mingliang Li, Mingxiao Li, Nan Wu, [Qi Han](https://md.research.elamur.ai/free/en/authors/qi-han/), Qinyuan Tan, Shaoliang Pang, Shengjie Fan, Siqi Liu, Tiancheng Cao, Wanying Lu, Wenqing He, Wuxun Xie, [Xu Zhao](https://md.research.elamur.ai/free/en/authors/xu-zhao/), Xueqi Li, Yanbo Yu, [Yang Yang](https://md.research.elamur.ai/free/en/authors/yang-yang/), [Yi Liu](https://md.research.elamur.ai/free/en/authors/yi-liu/), Yifan Lu, Yilei Wang, Yuanhao Ding, Yuanwei Liang, Yuanwei Lu, Yuchu Luo, Yuhe Yin, Yumeng Zhan, [Yuxiang Zhang](https://md.research.elamur.ai/free/en/authors/yuxiang-zhang/), Zidong Yang, Zixin Zhang, [Binxing Jiao](https://md.research.elamur.ai/free/en/authors/binxing-jiao/), [Daxin Jiang](https://md.research.elamur.ai/free/en/authors/daxin-jiang/), [Heung-Yeung Shum](https://md.research.elamur.ai/free/en/authors/heung-yeung-shum/), Jiansheng Chen, [Jing Li](https://md.research.elamur.ai/free/en/authors/jing-li/), [Xiangyu Zhang](https://md.research.elamur.ai/free/en/authors/xiangyu-zhang/) and Yibo Zhu · **Year:** 2025 · **Venue:** arXiv:2507.16632 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2507.16632) · [PDF](https://arxiv.org/pdf/2507.16632) · [DOI 10.48550/arxiv.2507.16632](https://doi.org/10.48550/arxiv.2507.16632) · [arXiv 2507.16632](https://arxiv.org/abs/2507.16632) · [OpenAlex](https://openalex.org/W4415202956)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [30](https://md.research.elamur.ai/free/en/papers/wu-2025-step-audio-2-technical/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Step-Audio 2 is StepFun's end-to-end large audio language model (LALM). It targets three weaknesses of earlier systems: they ignore paralinguistic cues such as emotion and speaking style, they answer mostly in text, and they hallucinate because they lack external knowledge. The model reads raw audio through a latent audio encoder and produces interleaved text and discrete audio tokens. This lets it respond in speech whose tone matches the conversation. Chain-of-thought reasoning and reinforcement learning sharpen its understanding. Retrieval-augmented generation and tool calls ground its answers, including web search and a new audio search tool that retrieves voices so the model can change timbre on request. Training is multi-stage and uses 680 billion text tokens plus 8 million hours of real and synthetic audio. The authors report state-of-the-art results against open-source and commercial rivals on ASR, audio understanding, speech translation and spoken dialogue. This makes Step-Audio 2 a strong open reference for expressive, tool-using speech assistants.

## Key points
- **Speech translation:** On Chinese↔English tasks, Step-Audio 2 averages 39.26 BLEU on CoVoST 2 (speech-to-text), compared with 35.40 for Qwen2.5-Omni and 29.61 for GPT-4o Audio. On CVSS (speech-to-speech) it averages 30.87, compared with 23.68 for GPT-4o Audio.
- **Spoken dialogue (URO-Bench):**
  - Chinese: it leads all compared models, scoring 83.32 in the basic track and 68.25 in the pro track.
  - English: GPT-4o Audio edges it out (84.54 vs 83.90 on the basic track).
- **New benchmark, StepEval-Audio-Toolcall:**
  - Built from 200 synthesized Chinese multi-turn dialogues per tool, each 3–6 turns long, balanced with negative samples.
  - Judged automatically by Qwen3-32B.
  - Released on GitHub together with the scripts and evaluation code.
- **Tool calling:** With speech input, Step-Audio 2 matches text-input Qwen3-32B on tool calling overall. It is clearly better at triggering audio search (precision 86.8 vs 67.5).
- **Model size:** Step-Audio 2 has fewer parameters than its 130B predecessor Step-Audio. Code and further details are available at github.com/stepfun-ai/Step-Audio2.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🔊 Signal Processing](https://md.research.elamur.ai/free/en/shelves/computer-science/signal-processing/) · 452 works

🧭 Topics (1): [🎙️ Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) · 480 works

## Cite

Cite as: Boyong Wu, Chao Yan, Chen Hu, Cheng Yi, Chengli Feng, Fei Tian, Feiyu Shen, Gang Yu, Haoyang Zhang, Jingbei Li, Mingrui Chen, Peng Liu, Wang You, Xiangyu Tony Zhang, Xingyuan Li, Xuerui Yang, Yayue Deng, Yechang Huang, Yuxin Li, Yuxin Zhang, Zhao You, Brian Li, Changyi Wan, Hanpeng Hu, Jiangjie Zhen, Siyu Chen, Song Yuan, Xuelin Zhang, Yimin Jiang, Yu Zhou, Yuxiang Yang, Bingxin Li, Buyun Ma, Changhe Song, Dongqing Pang, Guoqiang Hu, Haiyang Sun, Kang An, Na Wang, Shuli Gao, Wei Ji, Wen Li, Wen Sun, Xuan Wen, Yong Ren, Yuankai Ma, Yufan Lu, Bin Wang, Bo Li, Changxin Miao, Che Liu, Chen Xu, Dapeng Shi, Dingyuan Hu, Donghang Wu, Enle Liu, Guanzhe Huang, Gulin Yan, Han Zhang, Hao Nie, Haonan Jia, Hongyu Zhou, Jianjian Sun, Jiaoren Wu, Jie Wu, Jie Yang, Jin Yang, Junzhe Lin, Kaixiang Li, Lei Yang, Liying Shi, Li Zhou, Longlong Gu, Ming Li, Mingliang Li, Mingxiao Li, Nan Wu, Qi Han, Qinyuan Tan, Shaoliang Pang, Shengjie Fan, Siqi Liu, Tiancheng Cao, Wanying Lu, Wenqing He, Wuxun Xie, Xu Zhao, Xueqi Li, Yanbo Yu, Yang Yang, Yi Liu, Yifan Lu, Yilei Wang, Yuanhao Ding, Yuanwei Liang, Yuanwei Lu, Yuchu Luo, Yuhe Yin, Yumeng Zhan, Yuxiang Zhang, Zidong Yang, Zixin Zhang, Binxing Jiao, Daxin Jiang, Heung-Yeung Shum, Jiansheng Chen, Jing Li, Xiangyu Zhang and Yibo Zhu. 2025. Step-Audio 2 Technical Report. arXiv:2507.16632.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{wu2025step,
  title = {{Step-Audio 2 Technical Report}},
  author = {Boyong Wu, Chao Yan, Chen Hu, Cheng Yi, Chengli Feng, Fei Tian, Feiyu Shen, Gang Yu, Haoyang Zhang, Jingbei Li, Mingrui Chen, Peng Liu, Wang You, Xiangyu Tony Zhang, Xingyuan Li, Xuerui Yang, Yayue Deng, Yechang Huang, Yuxin Li, Yuxin Zhang, Zhao You, Brian Li, Changyi Wan, Hanpeng Hu, Jiangjie Zhen, Siyu Chen, Song Yuan, Xuelin Zhang, Yimin Jiang, Yu Zhou, Yuxiang Yang, Bingxin Li, Buyun Ma, Changhe Song, Dongqing Pang, Guoqiang Hu, Haiyang Sun, Kang An, Na Wang, Shuli Gao, Wei Ji, Wen Li, Wen Sun, Xuan Wen, Yong Ren, Yuankai Ma, Yufan Lu, Bin Wang, Bo Li, Changxin Miao, Che Liu, Chen Xu, Dapeng Shi, Dingyuan Hu, Donghang Wu, Enle Liu, Guanzhe Huang, Gulin Yan, Han Zhang, Hao Nie, Haonan Jia, Hongyu Zhou, Jianjian Sun, Jiaoren Wu, Jie Wu, Jie Yang, Jin Yang, Junzhe Lin, Kaixiang Li, Lei Yang, Liying Shi, Li Zhou, Longlong Gu, Ming Li, Mingliang Li, Mingxiao Li, Nan Wu, Qi Han, Qinyuan Tan, Shaoliang Pang, Shengjie Fan, Siqi Liu, Tiancheng Cao, Wanying Lu, Wenqing He, Wuxun Xie, Xu Zhao, Xueqi Li, Yanbo Yu, Yang Yang, Yi Liu, Yifan Lu, Yilei Wang, Yuanhao Ding, Yuanwei Liang, Yuanwei Lu, Yuchu Luo, Yuhe Yin, Yumeng Zhan, Yuxiang Zhang, Zidong Yang, Zixin Zhang, Binxing Jiao, Daxin Jiang, Heung-Yeung Shum, Jiansheng Chen, Jing Li, Xiangyu Zhang and Yibo Zhu},
  year = {2025},
  doi = {10.48550/arxiv.2507.16632},
  eprint = {2507.16632},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2507.16632}
}
```

- [← previous RAG-Star: Enhancing Deliberative Reasoning with Retrieval Augmented Verification and Refinement](https://md.research.elamur.ai/free/en/papers/jiang-2025-rag-star-enhancing-deliberative-reasoning/)
- [next → Benchmark Self-Evolving: A Multi-Agent Framework for Dynamic LLM Evaluation](https://md.research.elamur.ai/free/en/papers/wang-2025-benchmark-self-evolving-multi-agent/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/3/)

## References

82 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Philip Anastassiou et al. “[Seed-tts: A family of high-quality versatile speech generation models](https://md.research.elamur.ai/free/en/papers/anastassiou-2024-seed-tts-family-high/)”. In: arXiv preprint arXiv:2406.02430 (2024).
2. Rohan Anil et al. [PaLM 2 Technical Report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/). 2023. arXiv: 2305.10403 [cs.CL]. URL : https://arxiv.org/abs/2305.10403. [↗](https://arxiv.org/abs/2305.10403)
3. Alexei Baevski et al. [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://md.research.elamur.ai/free/en/papers/baevski-2020-wav2vec-2-0-framework-self/). 2020. arXiv: 2006.11477 [cs.CL]. URL : https://arxiv.org/abs/2006.11477. [↗](https://arxiv.org/abs/2006.11477)
4. Jinze Bai et al. [Qwen Technical Report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/). 2023. arXiv: 2309.16609 [cs.CL]. URL : https://arxiv.org/abs/2309.16609. [↗](https://arxiv.org/abs/2309.16609)
5. Ye Bai et al. “Seed-asr: Understanding diverse speech and contexts with llm-based speech recognition”. In: arXiv preprint arXiv:2407.04675 (2024). — [arXiv:2407.04675](https://arxiv.org/abs/2407.04675) [find](https://scholar.google.com/scholar?q=Ye+Bai+et+al%2E+%E2%80%9CSeed%2Dasr%3A+Understanding+diverse+speech+and+contexts+with+llm%2Dbased+speech+recognition%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2407%2E04675+%282024%29%2E+%E2%80%94+%5BarXiv%3A2407%2E04675%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2407%2E04675)
6. James Betker. Better speech synthesis through scaling. 2023. arXiv: 2305 . 07243 [cs.SD]. URL : https : //arxiv.org/abs/2305.07243. — [arXiv:2305.07243](https://arxiv.org/abs/2305.07243) [find](https://scholar.google.com/scholar?q=James+Betker%2E+Better+speech+synthesis+through+scaling%2E+2023%2E+arXiv%3A+2305+%2E+07243+%5Bcs%2ESD%5D%2E+URL+%3A+https+%3A+%2F%2Farxiv%2Eorg%2Fabs%2F2305%2E07243%2E+%E2%80%94+%5BarXiv%3A2305%2E07243%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2305%2E07243%29)
7. Zalán Borsos et al. “[Audiolm: a language modeling approach to audio generation](https://md.research.elamur.ai/free/en/papers/borsos-2022-audiolm-language-modeling-approach-audio/)”. In: IEEE/ACM transactions on audio, speech, and language processing 31 (2023), pp. 2523–2533.
8. Guoguo Chen et al. “GigaSpeech: An Evolving, Multi-Domain ASR Corpus with 10,000 Hours of Transcribed Audio”. In: Interspeech 2021. ISCA, Aug. 2021. DOI : 10.21437/interspeech.2021- 1965. URL : http://dx.doi.org/10.21437/Interspeech.2021-1965. — [doi:10.21437/interspeech.2021-](https://doi.org/10.21437/interspeech.2021-) [↗](http://dx.doi.org/10.21437/Interspeech.2021-1965)
9. Qian Chen et al. “Minmo: A multimodal large language model for seamless voice interaction”. In: arXiv preprint arXiv:2501.06282 (2025). — [arXiv:2501.06282](https://arxiv.org/abs/2501.06282) [find](https://scholar.google.com/scholar?q=Qian+Chen+et+al%2E+%E2%80%9CMinmo%3A+A+multimodal+large+language+model+for+seamless+voice+interaction%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2501%2E06282+%282025%29%2E+%E2%80%94+%5BarXiv%3A2501%2E06282%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2501%2E06282%29)
10. Sanyuan Chen et al. BEATs: Audio Pre-Training with Acoustic Tokenizers. 2022. arXiv: 2212.09058 [eess.AS]. URL : https://arxiv.org/abs/2212.09058. — [arXiv:2212.09058](https://arxiv.org/abs/2212.09058) [↗](https://arxiv.org/abs/2212.09058)
11. Sanyuan Chen et al. “WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing”. In: IEEE Journal of Selected Topics in Signal Processing 16.6 (Oct. 2022), pp. 1505–1518. ISSN : 1941-0484. DOI : 10.1109/jstsp.2022.3188113. URL : http://dx.doi.org/10.1109/JSTSP.2022.3188113. [↗](http://dx.doi.org/10.1109/JSTSP.2022.3188113)
12. Yunfei Chu et al. “[Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models](https://md.research.elamur.ai/free/en/papers/chu-2023-qwen-audio-advancing-universal/)”. In: arXiv preprint arXiv:2311.07919 (2023).
13. Yunfei Chu et al. “[Qwen2-audio technical report](https://md.research.elamur.ai/free/en/papers/chu-2024-qwen2-audio-technical-report/)”. In: arXiv preprint arXiv:2407.10759 (2024).
14. Jade Copet et al. Simple and Controllable Music Generation. 2024. arXiv: 2306.05284 [cs.SD]. URL : https://arxiv.org/abs/2306.05284. — [arXiv:2306.05284](https://arxiv.org/abs/2306.05284) [↗](https://arxiv.org/abs/2306.05284)
15. Alexandre Défossez et al. “[High fidelity neural audio compression](https://md.research.elamur.ai/free/en/papers/defossez-2022-high-fidelity-neural-audio-compression/)”. In: arXiv preprint arXiv:2210.13438 (2022).
16. Alexandre Défossez et al. “[Moshi: a speech-text foundation model for real-time dialogue](https://md.research.elamur.ai/free/en/papers/defossez-2024-moshi-speech-text-foundation-model/)”. In: arXiv preprint arXiv:2410.00037 (2024).
17. Soham Deshmukh et al. “Pengi: An audio language model for audio tasks”. In: Advances in Neural Information Processing Systems 36 (2023), pp. 18090–18108. [find](https://scholar.google.com/scholar?q=Soham+Deshmukh+et+al%2E+%E2%80%9CPengi%3A+An+audio+language+model+for+audio+tasks%E2%80%9D%2E+In%3A+Advances+in+Neural+Information+Processing+Systems+36+%282023%29%2C+pp%2E+18090%E2%80%9318108%2E)
18. Ding Ding et al. “[Kimi-audio technical report](https://md.research.elamur.ai/free/en/papers/kimiteam-2025-kimi-audio-technical-report/)”. In: arXiv preprint arXiv:2504.18425 (2025).
19. Zhihao Du et al. “Cosyvoice 2: Scalable streaming speech synthesis with large language models”. In: arXiv preprint arXiv:2412.10117 (2024). [find](https://scholar.google.com/scholar?q=Zhihao+Du+et+al%2E+%E2%80%9CCosyvoice+2%3A+Scalable+streaming+speech+synthesis+with+large+language+models%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2412%2E10117+%282024%29%2E)
20. Zhihao Du et al. CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens. 2024. arXiv: 2407.05407 [cs.SD]. URL : https://arxiv.org/abs/2407.05407. [↗](https://arxiv.org/abs/2407.05407)
21. Qingkai Fang et al. “Llama-omni: Seamless speech interaction with large language models”. In: arXiv preprint arXiv:2409.06666 (2024). [find](https://scholar.google.com/scholar?q=Qingkai+Fang+et+al%2E+%E2%80%9CLlama%2Domni%3A+Seamless+speech+interaction+with+large+language+models%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2409%2E06666+%282024%29%2E)
22. Heting Gao et al. LUCY: Linguistic Understanding and Control Yielding Early Stage of Her. 2025. arXiv: 2501.16327 [cs.CL]. URL : https://arxiv.org/abs/2501.16327. — [arXiv:2501.16327](https://arxiv.org/abs/2501.16327) [↗](https://arxiv.org/abs/2501.16327)
23. Jort F. Gemmeke et al. “Audio Set: An ontology and human-labeled dataset for audio events”. In: 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2017, pp. 776–780. DOI : 10.1109/ICASSP.2017.7952261. — [doi:10.1109/icassp.2017.7952261](https://doi.org/10.1109/icassp.2017.7952261) [find](https://scholar.google.com/scholar?q=Jort+F%2E+Gemmeke+et+al%2E+%E2%80%9CAudio+Set%3A+An+ontology+and+human%2Dlabeled+dataset+for+audio+events%E2%80%9D%2E+In%3A+2017+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+%28ICASSP%29%2E+2017%2C+pp%2E+776%E2%80%9378)
24. Sreyan Ghosh et al. Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities. 2025. arXiv: 2503.03983 [cs.SD]. URL : https://arxiv.org/abs/2503.03983. — [arXiv:2503.03983](https://arxiv.org/abs/2503.03983) [↗](https://arxiv.org/abs/2503.03983)
25. Arushi Goel et al. Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models. 2025. arXiv: 2507.08128 [cs.SD]. URL : https://arxiv.org/abs/2507.08128. — [arXiv:2507.08128](https://arxiv.org/abs/2507.08128) [↗](https://arxiv.org/abs/2507.08128)
26. Yuan Gong, Jin Yu, and James Glass. “Vocalsound: A Dataset for Improving Human Vocal Sounds Recognition”. In: ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2022, pp. 151–155. DOI : 10.1109/ICASSP43922.2022.9746828. — [doi:10.1109/icassp43922.2022.9746828](https://doi.org/10.1109/icassp43922.2022.9746828) [find](https://scholar.google.com/scholar?q=Yuan+Gong%2C+Jin+Yu%2C+and+James+Glass%2E+%E2%80%9CVocalsound%3A+A+Dataset+for+Improving+Human+Vocal+Sounds+Recognition%E2%80%9D%2E+In%3A+ICASSP+2022+%2D+2022+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processin)
27. Yuan Gong et al. “Joint audio and speech understanding”. In: 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE. 2023, pp. 1–8. [find](https://scholar.google.com/scholar?q=Yuan+Gong+et+al%2E+%E2%80%9CJoint+audio+and+speech+understanding%E2%80%9D%2E+In%3A+2023+IEEE+Automatic+Speech+Recognition+and+Understanding+Workshop+%28ASRU%29%2E+IEEE%2E+2023%2C+pp%2E+1%E2%80%938%2E)
28. Yuan Gong et al. “Listen, think, and understand”. In: arXiv preprint arXiv:2305.10790 (2023). — [arXiv:2305.10790](https://arxiv.org/abs/2305.10790) [find](https://scholar.google.com/scholar?q=Yuan+Gong+et+al%2E+%E2%80%9CListen%2C+think%2C+and+understand%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2305%2E10790+%282023%29%2E+%E2%80%94+%5BarXiv%3A2305%2E10790%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2305%2E10790%29)
29. Aaron Grattafiori et al. [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). 2024. arXiv: 2407 . 21783 [cs.AI]. URL : https : //arxiv.org/abs/2407.21783.
30. Wei-Ning Hsu et al. [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://md.research.elamur.ai/free/en/papers/hsu-2021-hubert-self-supervised-speech-representation/). 2021. arXiv: 2106.07447 [cs.CL]. URL : https://arxiv.org/abs/2106.07447. [↗](https://arxiv.org/abs/2106.07447)
31. Ailin Huang et al. “Step-Audio-AQAA: a Fully End-to-End Expressive Large Audio Language Model”. In: arXiv preprint arXiv:2506.08967 (2025). — [arXiv:2506.08967](https://arxiv.org/abs/2506.08967) [find](https://scholar.google.com/scholar?q=Ailin+Huang+et+al%2E+%E2%80%9CStep%2DAudio%2DAQAA%3A+a+Fully+End%2Dto%2DEnd+Expressive+Large+Audio+Language+Model%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2506%2E08967+%282025%29%2E+%E2%80%94+%5BarXiv%3A2506%2E08967%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2506%2E08967%29)
32. Ailin Huang et al. “[Step-audio: Unified understanding and generation in intelligent speech interaction](https://md.research.elamur.ai/free/en/papers/huang-2025-step-audio-unified-understanding/)”. In: arXiv preprint arXiv:2502.11946 (2025).
33. Rongjie Huang et al. “Audiogpt: Understanding and generating speech, music, sound, and talking head”. In: Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. 21. 2024, pp. 23802–23804. [find](https://scholar.google.com/scholar?q=Rongjie+Huang+et+al%2E+%E2%80%9CAudiogpt%3A+Understanding+and+generating+speech%2C+music%2C+sound%2C+and+talking+head%E2%80%9D%2E+In%3A+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2E+Vol%2E+38%2E+21%2E+2024%2C+pp%2E+23802%E2%80%9323)
34. Aaron Hurst et al. “[Gpt-4o system card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/)”. In: arXiv preprint arXiv:2410.21276 (2024).
35. Il-Young Jeong and Jeongsoo Park. “CochlScene: Acquisition of acoustic scene data using crowdsourcing”. In: 2022 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC). 2022, pp. 17–21. DOI : 10.23919/APSIPAASC55919.2022.9979822. — [doi:10.23919/apsipaasc55919.2022.9979822](https://doi.org/10.23919/apsipaasc55919.2022.9979822) [find](https://scholar.google.com/scholar?q=Il%2DYoung+Jeong+and+Jeongsoo+Park%2E+%E2%80%9CCochlScene%3A+Acquisition+of+acoustic+scene+data+using+crowdsourcing%E2%80%9D%2E+In%3A+2022+Asia%2DPacific+Signal+and+Information+Processing+Association+Annual+Summit+and+Conference)
36. Shengpeng Ji et al. “Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling”. In: arXiv preprint arXiv:2408.16532 (2024). — [arXiv:2408.16532](https://arxiv.org/abs/2408.16532) [find](https://scholar.google.com/scholar?q=Shengpeng+Ji+et+al%2E+%E2%80%9CWavtokenizer%3A+an+efficient+acoustic+discrete+codec+tokenizer+for+audio+language+modeling%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2408%2E16532+%282024%29%2E+%E2%80%94+%5BarXiv%3A2408%2E16532%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2)
37. Ye Jia et al. CVSS Corpus and Massively Multilingual Speech-to-Speech Translation. 2022. arXiv: 2201.03713 [cs.CL]. URL : https://arxiv.org/abs/2201.03713. [↗](https://arxiv.org/abs/2201.03713)
38. Ye Jia et al. “Direct speech-to-speech translation with a sequence-to-sequence model”. In: arXiv preprint arXiv:1904.06037 (2019). [find](https://scholar.google.com/scholar?q=Ye+Jia+et+al%2E+%E2%80%9CDirect+speech%2Dto%2Dspeech+translation+with+a+sequence%2Dto%2Dsequence+model%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A1904%2E06037+%282019%29%2E)
39. Ye Jia et al. “Translatotron 2: High-quality direct speech-to-speech translation with voice preservation”. In: International conference on machine learning. PMLR. 2022, pp. 10120–10134. [find](https://scholar.google.com/scholar?q=Ye+Jia+et+al%2E+%E2%80%9CTranslatotron+2%3A+High%2Dquality+direct+speech%2Dto%2Dspeech+translation+with+voice+preservation%E2%80%9D%2E+In%3A+International+conference+on+machine+learning%2E+PMLR%2E+2022%2C+pp%2E+10120%E2%80%9310134%2E)
40. Eugene Kharitonov et al. [Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision](https://md.research.elamur.ai/free/en/papers/kharitonov-2023-speak-read-prompt-high-fidelity/). 2023. arXiv: 2302.03540 [cs.SD]. URL : https://arxiv.org/abs/2302.03540. [↗](https://arxiv.org/abs/2302.03540)
41. Chris Dongjoo Kim et al. “[Audiocaps: Generating captions for audios in the wild](https://md.research.elamur.ai/free/en/papers/kim-2019-audiocaps-generating-captions-audios-wild/)”. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019, pp. 119–132.
42. Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis. 2020. arXiv: 2010.05646 [cs.SD]. URL : https://arxiv.org/abs/2010.05646. [↗](https://arxiv.org/abs/2010.05646)
43. Zhifeng Kong et al. Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities. 2024. arXiv: 2402.01831 [cs.SD]. URL : https://arxiv.org/abs/2402.01831. — [arXiv:2402.01831](https://arxiv.org/abs/2402.01831) [↗](https://arxiv.org/abs/2402.01831)
44. Chenyang Le et al. “Transvip: Speech to speech translation system with voice and isochrony preservation”. In: Advances in Neural Information Processing Systems 37 (2024), pp. 89682–89705. [find](https://scholar.google.com/scholar?q=Chenyang+Le+et+al%2E+%E2%80%9CTransvip%3A+Speech+to+speech+translation+system+with+voice+and+isochrony+preservation%E2%80%9D%2E+In%3A+Advances+in+Neural+Information+Processing+Systems+37+%282024%29%2C+pp%2E+89682%E2%80%9389705%2E)
45. Ann Lee et al. “Textless speech-to-speech translation on real data”. In: arXiv preprint arXiv:2112.08352 (2021). [find](https://scholar.google.com/scholar?q=Ann+Lee+et+al%2E+%E2%80%9CTextless+speech%2Dto%2Dspeech+translation+on+real+data%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2112%2E08352+%282021%29%2E)
46. Sang-gil Lee et al. BigVGAN: A Universal Neural Vocoder with Large-Scale Training. 2023. arXiv: 2206.04658 [cs.SD]. URL : https://arxiv.org/abs/2206.04658. — [arXiv:2206.04658](https://arxiv.org/abs/2206.04658) [↗](https://arxiv.org/abs/2206.04658)
47. Guan-Ting Lin, Cheng-Han Chiang, and Hung-yi Lee. “Advancing large language models to capture varied speaking styles and respond properly in spoken conversations”. In: arXiv preprint arXiv:2402.12786 (2024). [find](https://scholar.google.com/scholar?q=Guan%2DTing+Lin%2C+Cheng%2DHan+Chiang%2C+and+Hung%2Dyi+Lee%2E+%E2%80%9CAdvancing+large+language+models+to+capture+varied+speaking+styles+and+respond+properly+in+spoken+conversations%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2402%2E12786+)
48. Guan-Ting Lin et al. “Paralinguistics-enhanced large language modeling of spoken dialogue”. In: ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE. 2024, pp. 10316–10320. [find](https://scholar.google.com/scholar?q=Guan%2DTing+Lin+et+al%2E+%E2%80%9CParalinguistics%2Denhanced+large+language+modeling+of+spoken+dialogue%E2%80%9D%2E+In%3A+ICASSP+2024%2D2024+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+%28ICASSP%29%2E+IEEE)
49. Tu Anh Nguyen et al. Spirit LM: Interleaved Spoken and Written Language Model. 2024. [arXiv: 2402.05755 cs.CL](https://md.research.elamur.ai/free/en/papers/nguyen-2025-spirit-lm-interleaved-spoken-written/). URL : https://arxiv.org/abs/2402.05755. [↗](https://arxiv.org/abs/2402.05755)
50. OpenAI. [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). [https://openai.com/research/gpt-4](https://openai.com/research/gpt-4). Accessed: 2025-07-11. 2023. [↗](https://openai.com/research/gpt-4)
51. OpenAI. Introducing ChatGPT. Accessed: 2025-07-11. 2022. URL : [https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt). [↗](https://openai.com/blog/chatgpt)
52. Wei Ping et al. “Deep voice 3: 2000-speaker neural text-to-speech”. In: proc. ICLR. Vol. 79. 2018, pp. 1094–1099. [find](https://scholar.google.com/scholar?q=Wei+Ping+et+al%2E+%E2%80%9CDeep+voice+3%3A+2000%2Dspeaker+neural+text%2Dto%2Dspeech%E2%80%9D%2E+In%3A+proc%2E+ICLR%2E+Vol%2E+79%2E+2018%2C+pp%2E+1094%E2%80%931099%2E)
53. Alec Radford et al. “[Robust speech recognition via large-scale weak supervision](https://md.research.elamur.ai/free/en/papers/radford-2022-robust-speech-recognition-large-scale/)”. In: International conference on machine learning. PMLR. 2023, pp. 28492–28518.
54. Rafael Rafailov et al. “[Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/)”. In: Advances in Neural Information Processing Systems 36 (2023), pp. 53728–53741.
55. Yi Ren et al. “Fastspeech 2: Fast and high-quality end-to-end text to speech”. In: arXiv preprint arXiv:2006.04558 (2020). [find](https://scholar.google.com/scholar?q=Yi+Ren+et+al%2E+%E2%80%9CFastspeech+2%3A+Fast+and+high%2Dquality+end%2Dto%2Dend+text+to+speech%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2006%2E04558+%282020%29%2E)
56. Andrew Rouditchenko et al. “Omni-R1: Do You Really Need Audio to Fine-Tune Your Audio LLM?” In: arXiv preprint arXiv:2505.09439 (2025). — [arXiv:2505.09439](https://arxiv.org/abs/2505.09439) [find](https://scholar.google.com/scholar?q=Andrew+Rouditchenko+et+al%2E+%E2%80%9COmni%2DR1%3A+Do+You+Really+Need+Audio+to+Fine%2DTune+Your+Audio+LLM%3F%E2%80%9D+In%3A+arXiv+preprint+arXiv%3A2505%2E09439+%282025%29%2E+%E2%80%94+%5BarXiv%3A2505%2E09439%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2505%2E09439%29)
57. Paul K Rubenstein et al. “[Audiopalm: A large language model that can speak and listen](https://md.research.elamur.ai/free/en/papers/rubenstein-2023-audiopalm-large-language-model-that/)”. In: arXiv preprint arXiv:2306.12925 (2023).
58. S Sakshi et al. “Mmau: A massive multi-task audio understanding and reasoning benchmark”. In: arXiv preprint arXiv:2410.19168 (2024). [find](https://scholar.google.com/scholar?q=S+Sakshi+et+al%2E+%E2%80%9CMmau%3A+A+massive+multi%2Dtask+audio+understanding+and+reasoning+benchmark%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2410%2E19168+%282024%29%2E)
59. Jonathan Shen et al. “Natural tts synthesis by conditioning wavenet on mel spectrogram predictions”. In: 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE. 2018, pp. 4779–4783. [find](https://scholar.google.com/scholar?q=Jonathan+Shen+et+al%2E+%E2%80%9CNatural+tts+synthesis+by+conditioning+wavenet+on+mel+spectrogram+predictions%E2%80%9D%2E+In%3A+2018+IEEE+international+conference+on+acoustics%2C+speech+and+signal+processing+%28ICASSP%29%2E+IEEE%2E+2)
60. Hubert Siuzdak, Florian Grötschla, and Luca A Lanzendörfer. “Snac: Multi-scale neural audio codec”. In: arXiv preprint arXiv:2410.14411 (2024). — [arXiv:2410.14411](https://arxiv.org/abs/2410.14411) [find](https://scholar.google.com/scholar?q=Hubert+Siuzdak%2C+Florian+Gr%C3%B6tschla%2C+and+Luca+A+Lanzend%C3%B6rfer%2E+%E2%80%9CSnac%3A+Multi%2Dscale+neural+audio+codec%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2410%2E14411+%282024%29%2E+%E2%80%94+%5BarXiv%3A2410%2E14411%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2410%2E14411%29)
61. Changli Tang et al. “[Salmonn: Towards generic hearing abilities for large language models](https://md.research.elamur.ai/free/en/papers/tang-2023-salmonn-towards-generic-hearing-abilities/)”. In: arXiv preprint arXiv:2310.13289 (2023).
62. Wolfgang Wahlster. Verbmobil: foundations of speech-to-speech translation. Springer Science & Business Media, 2013. [find](https://scholar.google.com/scholar?q=Wolfgang+Wahlster%2E+Verbmobil%3A+foundations+of+speech%2Dto%2Dspeech+translation%2E+Springer+Science+%26+Business+Media%2C+2013%2E)
63. Changhan Wang, Anne Wu, and Juan Pino. CoVoST 2 and Massively Multilingual Speech-to-Text Translation. 2020. arXiv: 2007.10310 [cs.CL]. URL : https://arxiv.org/abs/2007.10310. [↗](https://arxiv.org/abs/2007.10310)
64. Chengyi Wang et al. “[Neural codec language models are zero-shot text to speech synthesizers](https://md.research.elamur.ai/free/en/papers/wang-2023-neural-codec-language-models-zero/)”. In: arXiv preprint arXiv:2301.02111 (2023).
65. Xinsheng Wang et al. “Spark-tts: An efficient llm-based text-to-speech model with single-stream decoupled speech tokens”. In: arXiv preprint arXiv:2503.01710 (2025). — [arXiv:2503.01710](https://arxiv.org/abs/2503.01710) [find](https://scholar.google.com/scholar?q=Xinsheng+Wang+et+al%2E+%E2%80%9CSpark%2Dtts%3A+An+efficient+llm%2Dbased+text%2Dto%2Dspeech+model+with+single%2Dstream+decoupled+speech+tokens%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2503%2E01710+%282025%29%2E+%E2%80%94+%5BarXiv%3A2503%2E01710%5D%28https%3A%2F%2Farxiv)
66. Xiong Wang et al. “Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen llm”. In: arXiv preprint arXiv:2411.00774 (2024). [find](https://scholar.google.com/scholar?q=Xiong+Wang+et+al%2E+%E2%80%9CFreeze%2Domni%3A+A+smart+and+low+latency+speech%2Dto%2Dspeech+dialogue+model+with+frozen+llm%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2411%2E00774+%282024%29%2E)
67. Yuancheng Wang et al. “Maskgct: Zero-shot text-to-speech with masked generative codec transformer”. In: arXiv preprint arXiv:2409.00750 (2024). — [arXiv:2409.00750](https://arxiv.org/abs/2409.00750) [find](https://scholar.google.com/scholar?q=Yuancheng+Wang+et+al%2E+%E2%80%9CMaskgct%3A+Zero%2Dshot+text%2Dto%2Dspeech+with+masked+generative+codec+transformer%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2409%2E00750+%282024%29%2E+%E2%80%94+%5BarXiv%3A2409%2E00750%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2409%2E00750%29)
68. Yuxuan Wang et al. “Tacotron: Towards end-to-end speech synthesis”. In: arXiv preprint arXiv:1703.10135 (2017). [find](https://scholar.google.com/scholar?q=Yuxuan+Wang+et+al%2E+%E2%80%9CTacotron%3A+Towards+end%2Dto%2Dend+speech+synthesis%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A1703%2E10135+%282017%29%2E)
69. Jason Wei et al. “[Finetuned language models are zero-shot learners](https://md.research.elamur.ai/free/en/papers/wei-2021-finetuned-language-models-zero-shot/)”. In: arXiv preprint arXiv:2109.01652 (2021).
70. Yonghui Wu et al. “Google’s neural machine translation system: Bridging the gap between human and machine translation”. In: arXiv preprint arXiv:1609.08144 (2016). [find](https://scholar.google.com/scholar?q=Yonghui+Wu+et+al%2E+%E2%80%9CGoogle%E2%80%99s+neural+machine+translation+system%3A+Bridging+the+gap+between+human+and+machine+translation%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A1609%2E08144+%282016%29%2E)
71. Zhifei Xie and Changqiao Wu. “Mini-omni: Language models can hear, talk while thinking in streaming”. In: arXiv preprint arXiv:2408.16725 (2024). [find](https://scholar.google.com/scholar?q=Zhifei+Xie+and+Changqiao+Wu%2E+%E2%80%9CMini%2Domni%3A+Language+models+can+hear%2C+talk+while+thinking+in+streaming%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2408%2E16725+%282024%29%2E)
72. Zhifei Xie and Changqiao Wu. “Mini-omni2: Towards open-source gpt-4o with vision, speech and duplex capabilities”. In: arXiv preprint arXiv:2410.11190 (2024). — [arXiv:2410.11190](https://arxiv.org/abs/2410.11190) [find](https://scholar.google.com/scholar?q=Zhifei+Xie+and+Changqiao+Wu%2E+%E2%80%9CMini%2Domni2%3A+Towards+open%2Dsource+gpt%2D4o+with+vision%2C+speech+and+duplex+capabilities%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2410%2E11190+%282024%29%2E+%E2%80%94+%5BarXiv%3A2410%2E11190%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
73. Detai Xin et al. “Bigcodec: Pushing the limits of low-bitrate neural speech codec”. In: arXiv preprint arXiv:2409.05377 (2024). — [arXiv:2409.05377](https://arxiv.org/abs/2409.05377) [find](https://scholar.google.com/scholar?q=Detai+Xin+et+al%2E+%E2%80%9CBigcodec%3A+Pushing+the+limits+of+low%2Dbitrate+neural+speech+codec%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2409%2E05377+%282024%29%2E+%E2%80%94+%5BarXiv%3A2409%2E05377%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2409%2E05377%29)
74. Jin Xu et al. [Qwen2.5-Omni Technical Report](https://md.research.elamur.ai/free/en/papers/xu-2025-qwen2-5-omni-technical/). 2025. arXiv: 2503.20215 [cs.CL]. URL : https://arxiv.org/abs/2503.20215. [↗](https://arxiv.org/abs/2503.20215)
75. Ruiqi Yan et al. URO-Bench: A Comprehensive Benchmark for End-to-End Spoken Dialogue Models. 2025. arXiv: 2502.17810 [cs.CL]. URL : https://arxiv.org/abs/2502.17810. — [arXiv:2502.17810](https://arxiv.org/abs/2502.17810) [↗](https://arxiv.org/abs/2502.17810)
76. Neil Zeghidour et al. “[Soundstream: An end-to-end neural audio codec](https://md.research.elamur.ai/free/en/papers/zeghidour-2021-soundstream-end-end-neural-audio/)”. In: IEEE/ACM Transactions on Audio, Speech, and Language Processing 30 (2021), pp. 495–507.
77. Aohan Zeng et al. “[Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot](https://md.research.elamur.ai/free/en/papers/zeng-2024-glm-4-voice-intelligent/)”. In: arXiv preprint arXiv:2412.02612 (2024).
78. Binbin Zhang et al. WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognition. 2022. arXiv: 2110.03370 [cs.SD]. URL : https://arxiv.org/abs/2110.03370. — [arXiv:2110.03370](https://arxiv.org/abs/2110.03370) [↗](https://arxiv.org/abs/2110.03370)
79. Bowen Zhang et al. “Minimax-speech: Intrinsic zero-shot text-to-speech with a learnable speaker encoder”. In: arXiv preprint arXiv:2505.07916 (2025). — [arXiv:2505.07916](https://arxiv.org/abs/2505.07916) [find](https://scholar.google.com/scholar?q=Bowen+Zhang+et+al%2E+%E2%80%9CMinimax%2Dspeech%3A+Intrinsic+zero%2Dshot+text%2Dto%2Dspeech+with+a+learnable+speaker+encoder%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2505%2E07916+%282025%29%2E+%E2%80%94+%5BarXiv%3A2505%2E07916%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2505%2E07)
80. Dong Zhang et al. “[Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities](https://md.research.elamur.ai/free/en/papers/zhang-2023-speechgpt-empowering-large-language-models/)”. In: arXiv preprint arXiv:2305.11000 (2023).
81. Xiangyu Zhang et al. “Distinctive Feature Codec: Adaptive Segmentation for Efficient Speech Representation”. In: arXiv preprint arXiv:2505.18516 (2025). — [arXiv:2505.18516](https://arxiv.org/abs/2505.18516) [find](https://scholar.google.com/scholar?q=Xiangyu+Zhang+et+al%2E+%E2%80%9CDistinctive+Feature+Codec%3A+Adaptive+Segmentation+for+Efficient+Speech+Representation%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2505%2E18516+%282025%29%2E+%E2%80%94+%5BarXiv%3A2505%2E18516%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2505)
82. Xin Zhang et al. “[Speechtokenizer: Unified speech tokenizer for speech large language models](https://md.research.elamur.ai/free/en/papers/zhang-2023-speechtokenizer-unified-speech-tokenizer-speech/)”. In: arXiv preprint arXiv:2308.16692 (2023).

*Updated 1 November 2026*