---
title: "Mini-Omni — paper by Xie and Wu (2024) — summary and ideas"
description: "Open speech assistants usually produce a text reply first and then pass it to a separate TTS engine. 31 references, cited by 69 papers here."
authors: ["Zhifei Xie", "Changqiao Wu"]
year: 2024
venue: "arXiv:2408.16725"
source: "https://arxiv.org/abs/2408.16725"
doi: "10.48550/arxiv.2408.16725"
arxiv: "2408.16725"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xie-2024-mini-omni-language-models-hear/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xie-2024-mini-omni-language-models-hear/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) / Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming

# Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming

**Authors:** Zhifei Xie and Changqiao Wu · **Year:** 2024 · **Venue:** arXiv:2408.16725 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2408.16725) · [PDF](https://arxiv.org/pdf/2408.16725) · [DOI 10.48550/arxiv.2408.16725](https://doi.org/10.48550/arxiv.2408.16725) · [arXiv 2408.16725](https://arxiv.org/abs/2408.16725) · [OpenAlex](https://openalex.org/W4403556012)

**Citations:** 4 citations (OpenAlex, 2026-10-14) · [69](https://md.research.elamur.ai/free/en/papers/xie-2024-mini-omni-language-models-hear/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Open speech assistants usually produce a text reply first and then pass it to a separate TTS engine. That pipeline adds noticeable delay before the user hears anything. Mini-Omni addresses this with a fully end-to-end, open-source model that takes audio in and streams audio out. It is built on a small 0.5B-parameter LLM. Instead of finishing text before speech, the model produces text and audio tokens simultaneously, and the text slightly leads the audio so it can steer speech generation like an online TTS. A batch-parallel decoding variant passes stronger text-side reasoning into the spoken answer. The authors call their adapter-based recipe "Any Model Can Talk". It adds speech capability to an existing LLM with little modification and modest synthetic data, through staged training of input and output adapters. They also built VoiceAssistant-400K, a GPT-4o-generated set of assistant-style answers without code or symbols. On LibriSpeech ASR, Mini-Omni trails Whisper-small only slightly. The work gives the research community a reproducible, low-latency baseline for speech-to-speech dialogue.

## Key points
- Architecture: Qwen2-0.5B base (24 blocks, hidden size 896) with a Whisper-small encoder and a two-layer MLP ASR adapter. Six extra transformer blocks form the TTS adapter, and audio is tokenized with SNAC (8 codebook layers).
- Training data: about 8,000 hours of ASR speech (LibriTTS, VCTK, Multilingual LibriSpeech), 2M Open-Orca text QA items, and 1.5M speech QA pairs synthesized from Moss-002-sft-data. Final fine-tuning uses VoiceAssistant-400K (over 400K entries).
- Training runs in stages: speech adapter training, then text and audio QA, then multimodal interaction, and finally annealing with SFT. All stages ran on 8 A100 GPUs.
- LibriSpeech WER (test-clean / test-other / dev-clean / dev-other): Mini-Omni scores 4.5 / 9.7 / 4.6 / 9.2. Whisper-small scores 3.4 / 7.6 on the test sets, and VITA scores 8.14 / 18.41 / 7.57 / 16.57.
- Code and data are released at github.com/gpt-omni/mini-omni. The authors admit that spoken reasoning remains weaker than text reasoning.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 415 works · 94,028 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🔊 Signal Processing](https://md.research.elamur.ai/free/en/shelves/computer-science/signal-processing/) · 493 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Zhifei Xie and Changqiao Wu. 2024. Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming. arXiv:2408.16725.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{xie2024mini,
  title = {{Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming}},
  author = {Zhifei Xie and Changqiao Wu},
  year = {2024},
  doi = {10.48550/arxiv.2408.16725},
  eprint = {2408.16725},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2408.16725}
}
```

- [← previous Long Context Transfer from Language to Vision](https://md.research.elamur.ai/free/en/papers/zhang-2024-long-context-transfer-language-vision/)
- [next → Repetition Improves Language Model Embeddings](https://md.research.elamur.ai/free/en/papers/springer-2024-repetition-improves-language-model-embeddings/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/)

## References

31 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. [Flamingo: a visual language model for few-shot learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/). Advances in neural information processing systems, 35:23716–23736, 2022.
2. Anthropic. [https://huggingface.co/datasets/anthropic/hh-rlhf](https://huggingface.co/datasets/anthropic/hh-rlhf), 2024. [↗](https://huggingface.co/datasets/anthropic/hh-rlhf)
3. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. [wav2vec 2.0: A framework for self-supervised learning of speech representations](https://md.research.elamur.ai/free/en/papers/baevski-2020-wav2vec-2-0-framework-self/). Advances in neural information processing systems, 33:12449–12460, 2020.
4. Qian Chen, Yunfei Chu, Zhifu Gao, Zerui Li, Kai Hu, Xiaohuan Zhou, Jin Xu, Ziyang Ma, Wen Wang, Siqi Zheng, et al. Lauragpt: Listen, attend, understand, and regenerate audio with gpt. arXiv preprint arXiv:2310.04673, 2023. — [arXiv:2310.04673](https://arxiv.org/abs/2310.04673) [find](https://scholar.google.com/scholar?q=Qian+Chen%2C+Yunfei+Chu%2C+Zhifu+Gao%2C+Zerui+Li%2C+Kai+Hu%2C+Xiaohuan+Zhou%2C+Jin+Xu%2C+Ziyang+Ma%2C+Wen+Wang%2C+Siqi+Zheng%2C+et+al%2E+Lauragpt%3A+Listen%2C+attend%2C+understand%2C+and+regenerate+audio+with+gpt%2E+arXiv+preprint+a)
5. Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, and Furu Wei. Beats: Audio pre-training with acoustic tokenizers. arXiv preprint arXiv:2212.09058, 2022. — [arXiv:2212.09058](https://arxiv.org/abs/2212.09058) [find](https://scholar.google.com/scholar?q=Sanyuan+Chen%2C+Yu+Wu%2C+Chengyi+Wang%2C+Shujie+Liu%2C+Daniel+Tompkins%2C+Zhuo+Chen%2C+and+Furu+Wei%2E+Beats%3A+Audio+pre%2Dtraining+with+acoustic+tokenizers%2E+arXiv+preprint+arXiv%3A2212%2E09058%2C+2022%2E+%E2%80%94+%5BarXiv%3A2212%2E09058%5D)
6. Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. [Qwen2-audio technical report](https://md.research.elamur.ai/free/en/papers/chu-2024-qwen2-audio-technical-report/). arXiv preprint arXiv:2407.10759, 2024.
7. Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Jade+Copet%2C+Felix+Kreuk%2C+Itai+Gat%2C+Tal+Remez%2C+David+Kant%2C+Gabriel+Synnaeve%2C+Yossi+Adi%2C+and+Alexandre+D%C3%A9fossez%2E+Simple+and+controllable+music+generation%2E+Advances+in+Neural+Information+Processing+Syste)
8. Nilaksh Das, Saket Dingliwal, Srikanth Ronanki, Rohit Paturi, David Huang, Prashant Mathur, Jie Yuan, Dhanush Bekal, Xing Niu, Sai Muralidhar Jayanthi, et al. [Speechverse: A large-scale generalizable audio language model](https://md.research.elamur.ai/free/en/papers/das-2024-speechverse-large-scale-generalizable-audio/). arXiv preprint arXiv:2405.08295, 2024. datashare. [https://datashare.ed.ac.uk/h…](https://datashare.ed.ac.uk/h…) [↗](https://datashare.ed.ac.uk/h…)
9. Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Meng Zhao, Yifan Zhang, Xiong Wang, Di Yin, Long Ma, Xiawu Zheng, et al. Vita: Towards open-source interactive omni multimodal llm. arXiv preprint arXiv:2408.05211, 2024. — [arXiv:2408.05211](https://arxiv.org/abs/2408.05211) [find](https://scholar.google.com/scholar?q=Chaoyou+Fu%2C+Haojia+Lin%2C+Zuwei+Long%2C+Yunhang+Shen%2C+Meng+Zhao%2C+Yifan+Zhang%2C+Xiong+Wang%2C+Di+Yin%2C+Long+Ma%2C+Xiawu+Zheng%2C+et+al%2E+Vita%3A+Towards+open%2Dsource+interactive+omni+multimodal+llm%2E+arXiv+preprint+arX)
10. Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, et al. Wavllm: Towards robust and adaptive speech large language model. arXiv preprint arXiv:2404.00656, 2024. [find](https://scholar.google.com/scholar?q=Shujie+Hu%2C+Long+Zhou%2C+Shujie+Liu%2C+Sanyuan+Chen%2C+Hongkun+Hao%2C+Jing+Pan%2C+Xunying+Liu%2C+Jinyu+Li%2C+Sunit+Sivasankaran%2C+Linquan+Liu%2C+et+al%2E+Wavllm%3A+Towards+robust+and+adaptive+speech+large+language+model%2E+a)
11. Ziyue Jiang, Yi Ren, Zhenhui Ye, Jinglin Liu, Chen Zhang, Qian Yang, Shengpeng Ji, Rongjie Huang, Chunfeng Wang, Xiang Yin, et al. Mega-tts: Zero-shot text-to-speech at scale with intrinsic inductive bias. arXiv preprint arXiv:2306.03509, 2023. — [arXiv:2306.03509](https://arxiv.org/abs/2306.03509) [find](https://scholar.google.com/scholar?q=Ziyue+Jiang%2C+Yi+Ren%2C+Zhenhui+Ye%2C+Jinglin+Liu%2C+Chen+Zhang%2C+Qian+Yang%2C+Shengpeng+Ji%2C+Rongjie+Huang%2C+Chunfeng+Wang%2C+Xiang+Yin%2C+et+al%2E+Mega%2Dtts%3A+Zero%2Dshot+text%2Dto%2Dspeech+at+scale+with+intrinsic+inductive+)
12. Eugene Kharitonov, Damien Vincent, Zalán Borsos, Raphaël Marinier, Sertan Girgin, Olivier Pietquin, Matt Sharifi, Marco Tagliasacchi, and Neil Zeghidour. [Speak, read and prompt: High-fidelity text-to-speech with minimal supervision](https://md.research.elamur.ai/free/en/papers/kharitonov-2023-speak-read-prompt-high-fidelity/). Transactions of the Association for Computational Linguistics, 11:1…
13. Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al. Voicebox: Text-guided multilingual universal speech generation at scale. Advances in neural information processing systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Matthew+Le%2C+Apoorv+Vyas%2C+Bowen+Shi%2C+Brian+Karrer%2C+Leda+Sari%2C+Rashel+Moritz%2C+Mary+Williamson%2C+Vimal+Manohar%2C+Yossi+Adi%2C+Jay+Mahadeokar%2C+et+al%2E+Voicebox%3A+Text%2Dguided+multilingual+universal+speech+genera)
14. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. [Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation](https://md.research.elamur.ai/free/en/papers/li-2022-blip-bootstrapping-language-image-pre/). In International conference on machine learning, pages 12888–12900. PMLR, 2022.
15. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. [Improved baselines with visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. meta. llama3.1, 2024. URL [https://llama.meta.com/](https://llama.meta.com/). [↗](https://llama.meta.com/)
16. Mihaiii. [https://huggingface.co/datasets/mihaiii/qa-assistant-2](https://huggingface.co/datasets/mihaiii/qa-assistant-2), 2024a. [↗](https://huggingface.co/datasets/mihaiii/qa-assistant-2)
17. Mihaiii. [https://huggingface.co/datasets/mihaiii/triviamultichoice](https://huggingface.co/datasets/mihaiii/triviamultichoice), 2024b. [↗](https://huggingface.co/datasets/mihaiii/triviamultichoice)
18. Mihaiii. [https://huggingface.co/datasets/mihaiii/triviasinglechoice](https://huggingface.co/datasets/mihaiii/triviasinglechoice), 2024c. mixtral. https://mistral.ai/, 2024. [↗](https://huggingface.co/datasets/mihaiii/triviasinglechoice)
19. Eliya Nachmani, Alon Levkovitch, Roy Hirsch, Julian Salazar, Chulayuth Asawaroengchai, Soroosh Mariooryad, Ehud Rivlin, RJ Skerry-Ryan, and Michelle Tadmor Ramanovich. Spoken question answering and speech continuation using spectrogram-powered llm. arXiv preprint arXiv:2305.15255, 2023. openai. http… — [arXiv:2305.15255](https://arxiv.org/abs/2305.15255) [find](https://scholar.google.com/scholar?q=Eliya+Nachmani%2C+Alon+Levkovitch%2C+Roy+Hirsch%2C+Julian+Salazar%2C+Chulayuth+Asawaroengchai%2C+Soroosh+Mariooryad%2C+Ehud+Rivlin%2C+RJ+Skerry%2DRyan%2C+and+Michelle+Tadmor+Ramanovich%2E+Spoken+question+answering+and+sp)
20. OpenAssistan. [https://huggingface.co/datasets/openassistant/oasst1](https://huggingface.co/datasets/openassistant/oasst1), 2024. [↗](https://huggingface.co/datasets/openassistant/oasst1)
21. OpenOrca. [https://huggingface.co/datasets/open-orca/openorca/](https://huggingface.co/datasets/open-orca/openorca/). Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210… [↗](https://huggingface.co/datasets/open-orca/openorca/)
22. Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. [Mls: A large-scale multilingual dataset for speech research](https://md.research.elamur.ai/free/en/papers/pratap-2020-mls-large-scale-multilingual-dataset/). arXiv preprint arXiv:2012.03411, 2020.
23. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. [Robust speech recognition via large-scale weak supervision](https://md.research.elamur.ai/free/en/papers/radford-2022-robust-speech-recognition-large-scale/). In International conference on machine learning, pages 28492–28518. PMLR, 2023. sayan1101. [https://huggingface.co/datasets/sayan1101/identity-finetu…](https://huggingface.co/datasets/sayan1101/identity-finetu…) [↗](https://huggingface.co/datasets/sayan1101/identity-finetu…)
24. Hubert Siuzdak. [https://github.com/hubertsiuzdak/snac/](https://github.com/hubertsiuzdak/snac/), 2024. [↗](https://github.com/hubertsiuzdak/snac/)
25. Tianxiang Sun, Xiaotian Zhang, Zhengfu He, Peng Li, Qinyuan Cheng, Xiangyang Liu, Hang Yan, Yunfan Shao, Qiong Tang, Shiduo Zhang, et al. Moss: An open conversational large language model. Machine Intelligence Research, pages 1–18, 2024. [find](https://scholar.google.com/scholar?q=Tianxiang+Sun%2C+Xiaotian+Zhang%2C+Zhengfu+He%2C+Peng+Li%2C+Qinyuan+Cheng%2C+Xiangyang+Liu%2C+Hang+Yan%2C+Yunfan+Shao%2C+Qiong+Tang%2C+Shiduo+Zhang%2C+et+al%2E+Moss%3A+An+open+conversational+large+language+model%2E+Machine+Int)
26. Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. [Salmonn: Towards generic hearing abilities for large language models](https://md.research.elamur.ai/free/en/papers/tang-2023-salmonn-towards-generic-hearing-abilities/). arXiv preprint arXiv:2310.13289, 2023. vicgalle. [https://huggingface.co/datasets/vicgalle/alpaca-gpt4](https://huggingface.co/datasets/vicgalle/alpaca-gpt4), 2024. [↗](https://huggingface.co/datasets/vicgalle/alpaca-gpt4)
27. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024a. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chang+Zhou%2C+Chengpeng+Li%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+et+al%2E+Qwen2+technical+report%2E+arXiv+preprint+arXiv%3A2407%2E10671%2C+2024a%2E)
28. Dongchao Yang, Songxiang Liu, Rongjie Huang, Chao Weng, and Helen Meng. Instructtts: Modelling expressive tts in discrete latent space with natural language style prompt. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024b. [find](https://scholar.google.com/scholar?q=Dongchao+Yang%2C+Songxiang+Liu%2C+Rongjie+Huang%2C+Chao+Weng%2C+and+Helen+Meng%2E+Instructtts%3A+Modelling+expressive+tts+in+discrete+latent+space+with+natural+language+style+prompt%2E+IEEE%2FACM+Transactions+on+Audi)
29. Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. [Libritts: A corpus derived from librispeech for text-to-speech](https://md.research.elamur.ai/free/en/papers/zen-2019-libritts-corpus-derived-librispeech-text/). arXiv preprint arXiv:1904.02882, 2019.
30. Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. [Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities](https://md.research.elamur.ai/free/en/papers/zhang-2023-speechgpt-empowering-large-language-models/). arXiv preprint arXiv:2305.11000, 2023a.
31. Hang Zhang, Xin Li, and Lidong Bing. [Video-llama: An instruction-tuned audio-visual language model for video understanding](https://md.research.elamur.ai/free/en/papers/zhang-2023-video-llama-instruction-tuned-audio/). arXiv preprint arXiv:2306.02858, 2023b.

*Updated 2 November 2026*