---
title: "UniSonate — paper by Qiang et al. (2026) — summary and ideas"
description: "UniSonate replaces these with one conditional flow-matching model, a dual-stream… 2.24% WER. 45 references, cited by 1 paper here."
authors: ["Chunyu Qiang", "Xiaopeng Wang", "Kang Yin", "Yuzhe Liang", "Yuxin Guo", "Teng Ma", "Ziyu Zhang", "Tianrui Wang", "Cheng Gong", "Yushen Chen", "Ruibo Fu", "Longbiao Wang", "Jianwu Dang"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.1293/"
doi: "10.18653/v1/2026.acl-long.1293"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/qiang-2026-unisonate-unified-model-speech-music/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/qiang-2026-unisonate-unified-model-speech-music/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/19/) / UniSonate: A Unified Model for Speech, Music, and Sound Effect Generation with Text Instructions

# UniSonate: A Unified Model for Speech, Music, and Sound Effect Generation with Text Instructions

**Authors:** [Chunyu Qiang](https://md.research.elamur.ai/free/en/authors/chunyu-qiang/), Xiaopeng Wang, Kang Yin, [Yuzhe Liang](https://md.research.elamur.ai/free/en/authors/yuzhe-liang/), [Yuxin Guo](https://md.research.elamur.ai/free/en/authors/yuxin-guo/), Teng Ma, Ziyu Zhang, [Tianrui Wang](https://md.research.elamur.ai/free/en/authors/tianrui-wang/), Cheng Gong, [Yushen Chen](https://md.research.elamur.ai/free/en/authors/yushen-chen/), Ruibo Fu, [Longbiao Wang](https://md.research.elamur.ai/free/en/authors/longbiao-wang/) and Jianwu Dang · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.1293/) · [PDF](https://aclanthology.org/2026.acl-long.1293.pdf) · [DOI 10.18653/v1/2026.acl-long.1293](https://doi.org/10.18653/v1/2026.acl-long.1293) · [OpenAlex](https://openalex.org/W7166830657)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Audio generation is usually handled by separate systems for speech (TTS), music (TTM) and sound effects (TTA), each with its own kind of input. TTS often needs reference audio, TTM needs lyrics or tags, and TTA works from free-form captions. UniSonate replaces these with one conditional flow-matching model, a dual-stream Multimodal Diffusion Transformer (MM-DiT) operating on Mel-VAE latents. Every task is driven by a natural-language instruction, with no reference audio.

The input is split into two streams: an instruction describing attributes such as timbre, mood or genre, and a content sequence that sets temporal structure. Speech and music use phonemes from transcripts or lyrics as content. Sound effects have no phonemes, so the model inserts learnable [SFX] tokens instead, which lets it control their duration the same way it controls speech. A curriculum moves training from speech to music and then to sound effects, reducing conflicts between these modalities.

UniSonate reports state-of-the-art instruction-based TTS and TTM, and reasonable TTA quality. Joint training also improved speech and music over single-task versions of the same model.

## Key points
- **Results:** instruction-based TTS reaches WER 1.47%, and TTM reaches SongEval Coherence 3.18.
- **Ablation:** a TTS-only version of the identical architecture scores 2.24% WER versus 1.47% with joint training. The joint model also beats a TTM-only variant on all SongEval metrics, which the authors credit partly to 50K hours of speech data.
- **Main method:** the key contributions are Dynamic Token Injection (learnable [SFX] tokens) and an Instruction-Content Alignment input format.
- **Sound effects still lag:** FAD is 4.21, compared with 2.07 for the specialized GenAU-L.
- **Limitations:** the model has 1.3B parameters and is slow at inference. It was trained and evaluated on 2–20 s clips, so long-form audio remains difficult. Text-only prompts are ambiguous, since one description can match many very different sounds.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/government/chinese-academy-of-sciences/) — Government body, Beijing, China · 194 works · 23,906 citations
- [Institute of Automation](https://md.research.elamur.ai/free/en/labs/institute-of-automation/) — Research institute, Beijing, China · 70 works · 14,251 citations
- [Tianjin University](https://md.research.elamur.ai/free/en/universities/tianjin-university/) — University, Tianjin, China · 41 works · 1,128 citations
- [Kuaishou](https://md.research.elamur.ai/free/en/companies/kuaishou/) — Company, Beijing, China · 15 works · 95 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works · [🔊 Signal Processing](https://md.research.elamur.ai/free/en/shelves/computer-science/signal-processing/) · 452 works

## Cite

Cite as: Chunyu Qiang, Xiaopeng Wang, Kang Yin, Yuzhe Liang, Yuxin Guo, Teng Ma, Ziyu Zhang, Tianrui Wang, Cheng Gong, Yushen Chen, Ruibo Fu, Longbiao Wang and Jianwu Dang. 2026. UniSonate: A Unified Model for Speech, Music, and Sound Effect Generation with Text Instructions. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{qiang2026unisonate,
  title = {{UniSonate: A Unified Model for Speech, Music, and Sound Effect Generation with Text Instructions}},
  author = {Chunyu Qiang, Xiaopeng Wang, Kang Yin, Yuzhe Liang, Yuxin Guo, Teng Ma, Ziyu Zhang, Tianrui Wang, Cheng Gong, Yushen Chen, Ruibo Fu, Longbiao Wang and Jianwu Dang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1293},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1293}
}
```

- [← previous UniCreative: Unifying Long-form Logic and Short-form Sparkle via Reference-Free Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/wei-2026-unicreative-unifying-long-form-logic/)
- [next → UniTor at Cruciverb-IT: Retrieval-Augmented Two-Step Reasoning for Italian Crossword Clue Answering](https://md.research.elamur.ai/free/en/papers/shcherbakov-2026-unitor-cruciverb-it-retrieval-augmented/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/19/)

## References

45 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Suno AI. 2024a. Suno: Ai music generation platform. [https://suno.com](https://suno.com). [↗](https://suno.com)
2. Udio AI. 2024b. Udio: Ai music creation platform. [https://udio.com](https://udio.com). [↗](https://udio.com)
3. Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, et al. 2024. [Seed-tts: A family of high-quality versatile speech generation models](https://md.research.elamur.ai/free/en/papers/anastassiou-2024-seed-tts-family-high/). arXiv preprint arXiv:2406.02430.
4. Huakang Chen, Yuepeng Jiang, Guobin Ma, Chunbo Hao, Shuai Wang, Jixun Yao, Ziqian Ning, Meng Meng, Jian Luan, and Lei Xie. 2025a. Diffrhythm+: Controllable and flexible full-length song generation with preference optimization. arXiv preprint arXiv:2507.12890. — [arXiv:2507.12890](https://arxiv.org/abs/2507.12890) [find](https://scholar.google.com/scholar?q=Huakang+Chen%2C+Yuepeng+Jiang%2C+Guobin+Ma%2C+Chunbo+Hao%2C+Shuai+Wang%2C+Jixun+Yao%2C+Ziqian+Ning%2C+Meng+Meng%2C+Jian+Luan%2C+and+Lei+Xie%2E+2025a%2E+Diffrhythm%2B%3A+Controllable+and+flexible+full%2Dlength+song+generation+wit)
5. Sanyuan Chen, Chengyi Wang, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. 2025b. [Neural codec language models are zero-shot text to speech synthesizers](https://md.research.elamur.ai/free/en/papers/wang-2023-neural-codec-language-models-zero/). IEEE Transactions on Audio, Speech and Language Processing, 33:705…
6. Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, and Xie Chen. 2024. [F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching](https://md.research.elamur.ai/free/en/papers/chen-2025-f5-tts-fairytaler-that-fakes/). arXiv preprint arXiv:2410.06885.
7. Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing, and Yuki Mitsufuji. 2025. Mmaudio: Taming multimodal joint training for high-quality video-to-audio synthesis. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 28901–28911. [find](https://scholar.google.com/scholar?q=Ho+Kei+Cheng%2C+Masato+Ishii%2C+Akio+Hayakawa%2C+Takashi+Shibuya%2C+Alexander+Schwing%2C+and+Yuki+Mitsufuji%2E+2025%2E+Mmaudio%3A+Taming+multimodal+joint+training+for+high%2Dquality+video%2Dto%2Daudio+synthesis%2E+In+Proceed)
8. Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. 2023. Simple and controllable music generation. Advances in Neural Information Processing Systems, 36:47704–47720. [find](https://scholar.google.com/scholar?q=Jade+Copet%2C+Felix+Kreuk%2C+Itai+Gat%2C+Tal+Remez%2C+David+Kant%2C+Gabriel+Synnaeve%2C+Yossi+Adi%2C+and+Alexandre+D%C3%A9fossez%2E+2023%2E+Simple+and+controllable+music+generation%2E+Advances+in+Neural+Information+Processing)
9. Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. 2020. Jukebox: A generative model for music. arXiv preprint arXiv:2005.00341. [find](https://scholar.google.com/scholar?q=Prafulla+Dhariwal%2C+Heewoo+Jun%2C+Christine+Payne%2C+Jong+Wook+Kim%2C+Alec+Radford%2C+and+Ilya+Sutskever%2E+2020%2E+Jukebox%3A+A+generative+model+for+music%2E+arXiv+preprint+arXiv%3A2005%2E00341%2E)
10. Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. 2024a. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.05407. [find](https://scholar.google.com/scholar?q=Zhihao+Du%2C+Qian+Chen%2C+Shiliang+Zhang%2C+Kai+Hu%2C+Heng+Lu%2C+Yexin+Yang%2C+Hangrui+Hu%2C+Siqi+Zheng%2C+Yue+Gu%2C+Ziyang+Ma%2C+et+al%2E+2024a%2E+Cosyvoice%3A+A+scalable+multilingual+zero%2Dshot+text%2Dto%2Dspeech+synthesizer+base)
11. Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, et al. 2024b. Cosyvoice 2: Scalable streaming speech synthesis with large language models. arXiv preprint arXiv:2412.10117. [find](https://scholar.google.com/scholar?q=Zhihao+Du%2C+Yuxuan+Wang%2C+Qian+Chen%2C+Xian+Shi%2C+Xiang+Lv%2C+Tianyu+Zhao%2C+Zhifu+Gao%2C+Yexin+Yang%2C+Changfeng+Gao%2C+Hui+Wang%2C+et+al%2E+2024b%2E+Cosyvoice+2%3A+Scalable+streaming+speech+synthesis+with+large+language+m)
12. Sefik Emre Eskimez, Xiaofei Wang, Manthan Thakker, Canrun Li, Chung-Hsien Tsai, Zhen Xiao, Hemin Yang, Zirun Zhu, Min Tang, Xu Tan, et al. 2024. E2 tts: Embarrassingly easy fully non-autoregressive zero-shot tts. In 2024 IEEE Spoken Language Technology Workshop (SLT), pages 682–689. IEEE. [find](https://scholar.google.com/scholar?q=Sefik+Emre+Eskimez%2C+Xiaofei+Wang%2C+Manthan+Thakker%2C+Canrun+Li%2C+Chung%2DHsien+Tsai%2C+Zhen+Xiao%2C+Hemin+Yang%2C+Zirun+Zhu%2C+Min+Tang%2C+Xu+Tan%2C+et+al%2E+2024%2E+E2+tts%3A+Embarrassingly+easy+fully+non%2Dautoregressive+ze)
13. Zach Evans, Julian D Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. 2025. Stable audio open. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. [find](https://scholar.google.com/scholar?q=Zach+Evans%2C+Julian+D+Parker%2C+CJ+Carr%2C+Zack+Zukowski%2C+Josiah+Taylor%2C+and+Jordi+Pons%2E+2025%2E+Stable+audio+open%2E+In+ICASSP+2025%2D2025+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processin)
14. Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, and Soujanya Poria. 2023. Text-to-audio generation using instruction guided latent diffusion model. In Proceedings of the 31st ACM International Conference on Multimedia, pages 3590–3598. [find](https://scholar.google.com/scholar?q=Deepanway+Ghosal%2C+Navonil+Majumder%2C+Ambuj+Mehrish%2C+and+Soujanya+Poria%2E+2023%2E+Text%2Dto%2Daudio+generation+using+instruction+guided+latent+diffusion+model%2E+In+Proceedings+of+the+31st+ACM+International+Conf)
15. Junmin Gong, Sean Zhao, Sen Wang, Shengyuan Xu, and Joe Guo. 2025. ACE-Step: A step towards music generation foundation model. arXiv preprint arXiv:2506.00045. [find](https://scholar.google.com/scholar?q=Junmin+Gong%2C+Sean+Zhao%2C+Sen+Wang%2C+Shengyuan+Xu%2C+and+Joe+Guo%2E+2025%2E+ACE%2DStep%3A+A+step+towards+music+generation+foundation+model%2E+arXiv+preprint+arXiv%3A2506%2E00045%2E)
16. Zhifang Guo, Yichong Leng, Yihan Wu, Sheng Zhao, and Xu Tan. 2023. Prompttts: Controllable text-to-speech with text descriptions. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. [find](https://scholar.google.com/scholar?q=Zhifang+Guo%2C+Yichong+Leng%2C+Yihan+Wu%2C+Sheng+Zhao%2C+and+Xu+Tan%2E+2023%2E+Prompttts%3A+Controllable+text%2Dto%2Dspeech+with+text+descriptions%2E+In+ICASSP+2023%2D2023+IEEE+International+Conference+on+Acoustics%2C+Speech)
17. Jiarui Hai, Yong Xu, Hao Zhang, Chenxing Li, Helin Wang, Mounya Elhilali, and Dong Yu. 2024. Ezaudio: Enhancing text-to-audio generation with efficient diffusion transformer. arXiv preprint arXiv:2409.10819. — [arXiv:2409.10819](https://arxiv.org/abs/2409.10819) [find](https://scholar.google.com/scholar?q=Jiarui+Hai%2C+Yong+Xu%2C+Hao+Zhang%2C+Chenxing+Li%2C+Helin+Wang%2C+Mounya+Elhilali%2C+and+Dong+Yu%2E+2024%2E+Ezaudio%3A+Enhancing+text%2Dto%2Daudio+generation+with+efficient+diffusion+transformer%2E+arXiv+preprint+arXiv%3A2409)
18. Moayed Haji-Ali, Willi Menapace, Aliaksandr Siarohin, Guha Balakrishnan, and Vicente Ordonez. 2024. Taming data and transformers for audio generation. arXiv preprint arXiv:2406.19388. — [arXiv:2406.19388](https://arxiv.org/abs/2406.19388) [find](https://scholar.google.com/scholar?q=Moayed+Haji%2DAli%2C+Willi+Menapace%2C+Aliaksandr+Siarohin%2C+Guha+Balakrishnan%2C+and+Vicente+Ordonez%2E+2024%2E+Taming+data+and+transformers+for+audio+generation%2E+arXiv+preprint+arXiv%3A2406%2E19388%2E+%E2%80%94+%5BarXiv%3A2406%2E19)
19. Jiawei Huang, Yi Ren, Rongjie Huang, Dongchao Yang, Zhenhui Ye, Chen Zhang, Jinglin Liu, Xiang Yin, Zejun Ma, and Zhou Zhao. 2023. Make-an-audio 2: Temporal-enhanced text-to-audio generation. arXiv preprint arXiv:2305.18474. — [arXiv:2305.18474](https://arxiv.org/abs/2305.18474) [find](https://scholar.google.com/scholar?q=Jiawei+Huang%2C+Yi+Ren%2C+Rongjie+Huang%2C+Dongchao+Yang%2C+Zhenhui+Ye%2C+Chen+Zhang%2C+Jinglin+Liu%2C+Xiang+Yin%2C+Zejun+Ma%2C+and+Zhou+Zhao%2E+2023%2E+Make%2Dan%2Daudio+2%3A+Temporal%2Denhanced+text%2Dto%2Daudio+generation%2E+arXiv+pr)
20. Shengpeng Ji, Jialong Zuo, Wen Wang, Minghui Fang, Siqi Zheng, Qian Chen, Ziyue Jiang, Hai Huang, Zehan Wang, Xize Cheng, et al. 2024. Controlspeech: Towards simultaneous zero-shot speaker cloning and zero-shot language style control with decoupled codec. arXiv preprint arXiv:2406.01205. — [arXiv:2406.01205](https://arxiv.org/abs/2406.01205) [find](https://scholar.google.com/scholar?q=Shengpeng+Ji%2C+Jialong+Zuo%2C+Wen+Wang%2C+Minghui+Fang%2C+Siqi+Zheng%2C+Qian+Chen%2C+Ziyue+Jiang%2C+Hai+Huang%2C+Zehan+Wang%2C+Xize+Cheng%2C+et+al%2E+2024%2E+Controlspeech%3A+Towards+simultaneous+zero%2Dshot+speaker+cloning+and)
21. Diederik P. Kingma and Jimmy Ba. 2014. [A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). abs/1412.6980.
22. Adam: CoRR, Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D Plumbley. 2020. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2880– 2894. [find](https://scholar.google.com/scholar?q=Adam%3A+CoRR%2C+Qiuqiang+Kong%2C+Yin+Cao%2C+Turab+Iqbal%2C+Yuxuan+Wang%2C+Wenwu+Wang%2C+and+Mark+D+Plumbley%2E+2020%2E+Panns%3A+Large%2Dscale+pretrained+audio+neural+networks+for+audio+pattern+recognition%2E+IEEE%2FACM+Transac)
23. Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. 2022. Audiogen: Textually guided audio generation. arXiv preprint arXiv:2209.15352. [find](https://scholar.google.com/scholar?q=Felix+Kreuk%2C+Gabriel+Synnaeve%2C+Adam+Polyak%2C+Uriel+Singer%2C+Alexandre+D%C3%A9fossez%2C+Jade+Copet%2C+Devi+Parikh%2C+Yaniv+Taigman%2C+and+Yossi+Adi%2E+2022%2E+Audiogen%3A+Textually+guided+audio+generation%2E+arXiv+preprint+a)
24. Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. 2022. [Flow matching for generative modeling](https://md.research.elamur.ai/free/en/papers/lipman-2022-flow-matching-generative-modeling/). arXiv preprint arXiv:2210.02747.
25. Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. 2023. AudioLDM: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301.12503. [find](https://scholar.google.com/scholar?q=Haohe+Liu%2C+Zehua+Chen%2C+Yi+Yuan%2C+Xinhao+Mei%2C+Xubo+Liu%2C+Danilo+Mandic%2C+Wenwu+Wang%2C+and+Mark+D+Plumbley%2E+2023%2E+AudioLDM%3A+Text%2Dto%2Daudio+generation+with+latent+diffusion+models%2E+arXiv+preprint+arXiv%3A2301%2E1)
26. Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D Plumbley. 2024. AudioLDM 2: Learning holistic audio generation with self-supervised pretraining. IEEE/ACM Transactions on Audio, Speech, and Language Processing. [find](https://scholar.google.com/scholar?q=Haohe+Liu%2C+Yi+Yuan%2C+Xubo+Liu%2C+Xinhao+Mei%2C+Qiuqiang+Kong%2C+Qiao+Tian%2C+Yuping+Wang%2C+Wenwu+Wang%2C+Yuxuan+Wang%2C+and+Mark+D+Plumbley%2E+2024%2E+AudioLDM+2%3A+Learning+holistic+audio+generation+with+self%2Dsupervised)
27. Zihan Liu, Shuangrui Ding, Zhixiong Zhang, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. 2025. Songgen: A single stage auto-regressive transformer for text-to-song generation. arXiv preprint arXiv:2502.13128. — [arXiv:2502.13128](https://arxiv.org/abs/2502.13128) [find](https://scholar.google.com/scholar?q=Zihan+Liu%2C+Shuangrui+Ding%2C+Zhixiong+Zhang%2C+Xiaoyi+Dong%2C+Pan+Zhang%2C+Yuhang+Zang%2C+Yuhang+Cao%2C+Dahua+Lin%2C+and+Jiaqi+Wang%2E+2025%2E+Songgen%3A+A+single+stage+auto%2Dregressive+transformer+for+text%2Dto%2Dsong+genera)
28. Jan Melechovsky, Zixun Guo, Deepanway Ghosal, Navonil Majumder, Dorien Herremans, and Soujanya Poria. 2024. Mustango: Toward controllable text-to-music generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tec… [find](https://scholar.google.com/scholar?q=Jan+Melechovsky%2C+Zixun+Guo%2C+Deepanway+Ghosal%2C+Navonil+Majumder%2C+Dorien+Herremans%2C+and+Soujanya+Poria%2E+2024%2E+Mustango%3A+Toward+controllable+text%2Dto%2Dmusic+generation%2E+In+Proceedings+of+the+2024+Conferenc)
29. Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. 2023. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (IC… [find](https://scholar.google.com/scholar?q=Yusong+Wu%2C+Ke+Chen%2C+Tianyu+Zhang%2C+Yuchen+Hui%2C+Taylor+Berg%2DKirkpatrick%2C+and+Shlomo+Dubnov%2E+2023%2E+Large%2Dscale+contrastive+language%2Daudio+pretraining+with+feature+fusion+and+keyword%2Dto%2Dcaption+augmentati)
30. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, and et al. Bo Zheng. 2025a. Qwen2.5:technical report. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+and+et+al%2E+Bo+Zheng%2E+2025a%2E+Qwen2%2E5%3Atechnical+report%2E)
31. Ziqian Ning, Huakang Chen, Yuepeng Jiang, Chunbo Hao, Guobin Ma, Shuai Wang, Jixun Yao, and Lei Xie. 2025. DiffRhythm: Blazingly fast and embarrassingly simple end-to-end full-length song generation with latent diffusion. arXiv preprint arXiv:2503.01183. — [arXiv:2503.01183](https://arxiv.org/abs/2503.01183) [find](https://scholar.google.com/scholar?q=Ziqian+Ning%2C+Huakang+Chen%2C+Yuepeng+Jiang%2C+Chunbo+Hao%2C+Guobin+Ma%2C+Shuai+Wang%2C+Jixun+Yao%2C+and+Lei+Xie%2E+2025%2E+DiffRhythm%3A+Blazingly+fast+and+embarrassingly+simple+end%2Dto%2Dend+full%2Dlength+song+generation+w)
32. Dongchao Yang, Songxiang Liu, Rongjie Huang, Chao Weng, and Helen Meng. 2024. Instructtts: Modelling expressive tts in discrete latent space with natural language style prompt. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32:2913– 2925. [find](https://scholar.google.com/scholar?q=Dongchao+Yang%2C+Songxiang+Liu%2C+Rongjie+Huang%2C+Chao+Weng%2C+and+Helen+Meng%2E+2024%2E+Instructtts%3A+Modelling+expressive+tts+in+discrete+latent+space+with+natural+language+style+prompt%2E+IEEE%2FACM+Transactions+o)
33. Chunyu Qiang, Peng Yang, Hao Che, Jinba Xiao, Xiaorui Wang, and Zhongyuan Wang. 2022. Back-translation-style data augmentation for mandarin chinese polyphone disambiguation. In 2022 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pages 1915–1919.… [find](https://scholar.google.com/scholar?q=Chunyu+Qiang%2C+Peng+Yang%2C+Hao+Che%2C+Jinba+Xiao%2C+Xiaorui+Wang%2C+and+Zhongyuan+Wang%2E+2022%2E+Back%2Dtranslation%2Dstyle+data+augmentation+for+mandarin+chinese+polyphone+disambiguation%2E+In+2022+Asia%2DPacific+Signa)
34. Dongchao Yang, Jinchuan Tian, Xu Tan, Rongjie Huang, Songxiang Liu, Xuankai Chang, Jiatong Shi, Sheng Zhao, Jiang Bian, Xixin Wu, et al. 2023. Uniaudio: An audio foundation model toward universal audio generation. arXiv preprint arXiv:2310.00704. — [arXiv:2310.00704](https://arxiv.org/abs/2310.00704) [find](https://scholar.google.com/scholar?q=Dongchao+Yang%2C+Jinchuan+Tian%2C+Xu+Tan%2C+Rongjie+Huang%2C+Songxiang+Liu%2C+Xuankai+Chang%2C+Jiatong+Shi%2C+Sheng+Zhao%2C+Jiang+Bian%2C+Xixin+Wu%2C+et+al%2E+2023%2E+Uniaudio%3A+An+audio+foundation+model+toward+universal+audi)
35. Chunyu Qiang, Kang Yin, Xiaopeng Wang, Yuzhe Liang, Jiahui Zhao, Ruibo Fu, Tianrui Wang, Cheng Gong, Chen Zhang, Longbiao Wang, et al. 2025. Instructaudio: Unified speech and music generation with natural language instruction. arXiv preprint arXiv:2511.18487. — [arXiv:2511.18487](https://arxiv.org/abs/2511.18487) [find](https://scholar.google.com/scholar?q=Chunyu+Qiang%2C+Kang+Yin%2C+Xiaopeng+Wang%2C+Yuzhe+Liang%2C+Jiahui+Zhao%2C+Ruibo+Fu%2C+Tianrui+Wang%2C+Cheng+Gong%2C+Chen+Zhang%2C+Longbiao+Wang%2C+et+al%2E+2025%2E+Instructaudio%3A+Unified+speech+and+music+generation+with+nat)
36. Guanrou Yang, Chen Yang, Qian Chen, Ziyang Ma, Wenxi Chen, Wen Wang, Tianrui Wang, Yifan Yang, Zhikang Niu, Wenrui Liu, et al. 2025b. Emovoice: Llm-based emotional text-to-speech model with freestyle text prompting. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 10748–1… [find](https://scholar.google.com/scholar?q=Guanrou+Yang%2C+Chen+Yang%2C+Qian+Chen%2C+Ziyang+Ma%2C+Wenxi+Chen%2C+Wen+Wang%2C+Tianrui+Wang%2C+Yifan+Yang%2C+Zhikang+Niu%2C+Wenrui+Liu%2C+et+al%2E+2025b%2E+Emovoice%3A+Llm%2Dbased+emotional+text%2Dto%2Dspeech+model+with+freestyle+)
37. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). Neurocomputing, 568:127063.
38. Jixun Yao, Guobin Ma, Huixin Xue, Huakang Chen, Chunbo Hao, Yuepeng Jiang, Haohe Liu, Ruibin Yuan, Jin Xu, Wei Xue, et al. 2025. Songeval: A benchmark dataset for song aesthetics evaluation. arXiv preprint arXiv:2505.10793. — [arXiv:2505.10793](https://arxiv.org/abs/2505.10793) [find](https://scholar.google.com/scholar?q=Jixun+Yao%2C+Guobin+Ma%2C+Huixin+Xue%2C+Huakang+Chen%2C+Chunbo+Hao%2C+Yuepeng+Jiang%2C+Haohe+Liu%2C+Ruibin+Yuan%2C+Jin+Xu%2C+Wei+Xue%2C+et+al%2E+2025%2E+Songeval%3A+A+benchmark+dataset+for+song+aesthetics+evaluation%2E+arXiv+pre)
39. Apoorv Vyas, Bowen Shi, Matthew Le, Andros Tjandra, Yi-Chiao Wu, Baishan Guo, Jiemin Zhang, Xinyue Zhang, Robert Adkins, William Ngan, et al. 2023. Audiobox: Unified audio generation with natural language prompts. arXiv preprint arXiv:2312.15821. — [arXiv:2312.15821](https://arxiv.org/abs/2312.15821) [find](https://scholar.google.com/scholar?q=Apoorv+Vyas%2C+Bowen+Shi%2C+Matthew+Le%2C+Andros+Tjandra%2C+Yi%2DChiao+Wu%2C+Baishan+Guo%2C+Jiemin+Zhang%2C+Xinyue+Zhang%2C+Robert+Adkins%2C+William+Ngan%2C+et+al%2E+2023%2E+Audiobox%3A+Unified+audio+generation+with+natural+lang)
40. Ruibin Yuan, Hanfeng Lin, Shuyue Guo, Ge Zhang, Jiahao Pan, Yongyi Zang, Haohe Liu, Yiming Liang, Wenye Ma, Xingjian Du, et al. 2025. Yue: Scaling open foundation models for long-form music generation. arXiv preprint arXiv:2503.08638. — [arXiv:2503.08638](https://arxiv.org/abs/2503.08638) [find](https://scholar.google.com/scholar?q=Ruibin+Yuan%2C+Hanfeng+Lin%2C+Shuyue+Guo%2C+Ge+Zhang%2C+Jiahao+Pan%2C+Yongyi+Zang%2C+Haohe+Liu%2C+Yiming+Liang%2C+Wenye+Ma%2C+Xingjian+Du%2C+et+al%2E+2025%2E+Yue%3A+Scaling+open+foundation+models+for+long%2Dform+music+generation)
41. Jun Wang, Xijuan Zeng, Chunyu Qiang, Ruilong Chen, Shiyao Wang, Le Wang, Wangjing Zhou, Pengfei Cai, Jiahui Zhao, Nan Li, et al. 2025a. Kling-foley: Multimodal diffusion transformer for high-quality video-to-audio generation. arXiv preprint arXiv:2506.19774. — [arXiv:2506.19774](https://arxiv.org/abs/2506.19774) [find](https://scholar.google.com/scholar?q=Jun+Wang%2C+Xijuan+Zeng%2C+Chunyu+Qiang%2C+Ruilong+Chen%2C+Shiyao+Wang%2C+Le+Wang%2C+Wangjing+Zhou%2C+Pengfei+Cai%2C+Jiahui+Zhao%2C+Nan+Li%2C+et+al%2E+2025a%2E+Kling%2Dfoley%3A+Multimodal+diffusion+transformer+for+high%2Dquality+v)
42. Xueyao Zhang, Junan Zhang, Yuancheng Wang, Chaoren Wang, Yuanzhe Chen, Dongya Jia, Zhuo Chen, and Zhizheng Wu. 2025. Vevo2: Bridging controllable speech and singing voice generation via unified prosody learning. arXiv preprint arXiv:2508.16332. — [arXiv:2508.16332](https://arxiv.org/abs/2508.16332) [find](https://scholar.google.com/scholar?q=Xueyao+Zhang%2C+Junan+Zhang%2C+Yuancheng+Wang%2C+Chaoren+Wang%2C+Yuanzhe+Chen%2C+Dongya+Jia%2C+Zhuo+Chen%2C+and+Zhizheng+Wu%2E+2025%2E+Vevo2%3A+Bridging+controllable+speech+and+singing+voice+generation+via+unified+prosod)
43. Xinsheng Wang, Mingqi Jiang, Ziyang Ma, Ziyu Zhang, Songxiang Liu, Linqin Li, Zheng Liang, Qixi Zheng, Rui Wang, Xiaoqin Feng, et al. 2025b. Spark-tts: An efficient llm-based text-to-speech model with single-stream decoupled speech tokens. arXiv preprint arXiv:2503.01710. — [arXiv:2503.01710](https://arxiv.org/abs/2503.01710) [find](https://scholar.google.com/scholar?q=Xinsheng+Wang%2C+Mingqi+Jiang%2C+Ziyang+Ma%2C+Ziyu+Zhang%2C+Songxiang+Liu%2C+Linqin+Li%2C+Zheng+Liang%2C+Qixi+Zheng%2C+Rui+Wang%2C+Xiaoqin+Feng%2C+et+al%2E+2025b%2E+Spark%2Dtts%3A+An+efficient+llm%2Dbased+text%2Dto%2Dspeech+model+with)
44. Siyi Zhou, Yiquan Zhou, Yi He, Xun Zhou, Jinchao Wang, Wei Deng, and Jingchen Shu. 2025. Indextts2: A breakthrough in emotionally expressive and duration-controlled auto-regressive zero-shot text-to-speech. arXiv preprint arXiv:2506.21619. — [arXiv:2506.21619](https://arxiv.org/abs/2506.21619) [find](https://scholar.google.com/scholar?q=Siyi+Zhou%2C+Yiquan+Zhou%2C+Yi+He%2C+Xun+Zhou%2C+Jinchao+Wang%2C+Wei+Deng%2C+and+Jingchen+Shu%2E+2025%2E+Indextts2%3A+A+breakthrough+in+emotionally+expressive+and+duration%2Dcontrolled+auto%2Dregressive+zero%2Dshot+text%2Dto%2Ds)
45. Yuancheng Wang, Haoyue Zhan, Liwei Liu, Ruihong Zeng, Haotian Guo, Jiachen Zheng, Qiang Zhang, Xueyao Zhang, Shunsi Zhang, and Zhizheng Wu. 2024. Maskgct: Zero-shot text-to-speech with Han Zhu, Wei Kang, Zengwei Yao, Liyong Guo, Fangjun Kuang, Zhaoqing Li, Weiji Zhuang, Long Lin, and Daniel Povey. 2… — [arXiv:2506.13053](https://arxiv.org/abs/2506.13053) [find](https://scholar.google.com/scholar?q=Yuancheng+Wang%2C+Haoyue+Zhan%2C+Liwei+Liu%2C+Ruihong+Zeng%2C+Haotian+Guo%2C+Jiachen+Zheng%2C+Qiang+Zhang%2C+Xueyao+Zhang%2C+Shunsi+Zhang%2C+and+Zhizheng+Wu%2E+2024%2E+Maskgct%3A+Zero%2Dshot+text%2Dto%2Dspeech+with+Han+Zhu%2C+Wei+Ka)

*Updated 1 November 2026*