---
title: "ControlSpeech — paper by Ji et al. (2025) — summary and ideas"
description: "To address this, they propose a component called SMSD (Style Mixture Semantic Density) that draws… 56 references, cited by 7 papers here."
authors: ["Shengpeng Ji", "Qian Chen", "Wen Wang", "Jialong Zuo", "Minghui Fang", "Ziyue Jiang", "Hai Huang", "Zehan Wang", "Xize Cheng", "Siqi Zheng", "Zhou Zhao"]
year: 2025
venue: "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2025.acl-long.346/"
doi: "10.18653/v1/2025.acl-long.346"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ji-2025-controlspeech-simultaneous-independent-zero-shot/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ji-2025-controlspeech-simultaneous-independent-zero-shot/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/8/) / ControlSpeech: Towards Simultaneous and Independent Zero-shot Speaker Cloning and Zero-shot Language Style Control

# ControlSpeech: Towards Simultaneous and Independent Zero-shot Speaker Cloning and Zero-shot Language Style Control

**Authors:** [Shengpeng Ji](https://md.research.elamur.ai/free/en/authors/shengpeng-ji/), [Qian Chen](https://md.research.elamur.ai/free/en/authors/qian-chen/), [Wen Wang](https://md.research.elamur.ai/free/en/authors/wen-wang/), Jialong Zuo, Minghui Fang, [Ziyue Jiang](https://md.research.elamur.ai/free/en/authors/ziyue-jiang/), Hai Huang, Zehan Wang, [Xize Cheng](https://md.research.elamur.ai/free/en/authors/xize-cheng/), [Siqi Zheng](https://md.research.elamur.ai/free/en/authors/siqi-zheng/) and [Zhou Zhao](https://md.research.elamur.ai/free/en/authors/zhou-zhao/) · **Year:** 2025 · **Venue:** [Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2025](https://md.research.elamur.ai/free/en/venues/acl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.acl-long.346/) · [PDF](https://aclanthology.org/2025.acl-long.346.pdf) · [DOI 10.18653/v1/2025.acl-long.346](https://doi.org/10.18653/v1/2025.acl-long.346) · [OpenAlex](https://openalex.org/W4412945647)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [7](https://md.research.elamur.ai/free/en/papers/ji-2025-controlspeech-simultaneous-independent-zero-shot/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Text-to-speech systems have usually offered one of two abilities. Zero-shot models can copy an unseen speaker's voice but cannot change how that voice speaks. Style-controllable models follow textual style descriptions but cannot reproduce a chosen speaker's timbre. ControlSpeech combines both. It takes three inputs: a speech prompt for timbre, a content prompt and a textual style prompt. It produces codec tokens in a pre-trained, disentangled discrete codec space using bidirectional attention and mask-based parallel (non-autoregressive) decoding.

The authors also point out a many-to-many relation in style control. Different descriptions can denote the same speech, and one description can cover many intensities of a style. To address this, they propose a component called SMSD (Style Mixture Semantic Density) that draws samples from a Gaussian mixture density network and injects noise perturbation to increase diversity. They also release VccmDataset, derived from TextrolSpeech.

ControlSpeech matches or beats specialised baselines on style accuracy, timbre similarity, quality and robustness. This makes independent zero-shot control of timbre, content and style practical.

## Key points
- The disentangled codec representation comes from NaturalSpeech 3 (Ju et al., 2024) and was pre-trained on about 60,000 hours of multi-speaker speech. This pre-training underpins the zero-shot cloning ability.
- On VccmDataset test set A, ControlSpeech compared with style-controllable baselines (Salle, PromptStyle, InstructTTS, PromptTTS 2) as follows:
  - It led on speed (0.829), volume (0.894) and emotion accuracy (0.557 vs. at most 0.412 for baselines).
  - It also led on WER (2.9), Spk-sv (0.89) and MOS-Q (3.91).
  - Its pitch accuracy (0.833) trailed PromptTTS 2 (0.867).
- In out-of-domain timbre cloning (test set B), ControlSpeech had a WER of 3.3, compared with 6.7 for VALL-E and 4.1 for MobileSpeech. Its MOS-S (3.96) was close to MobileSpeech's 4.01.
- On the many-to-many test set D, SMSD raised style accuracy (MOS-SA) from 3.59 to 3.84 and style diversity (MOS-SD) from 3.66 to 4.05, compared with a variant without SMSD.
- Removing codec decoupling sharply reduced controllability, for example emotion accuracy fell from 0.557 to 0.237. This indicates that the speech and style prompts interfere with each other when they are not separated.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works · [🔊 Signal Processing](https://md.research.elamur.ai/free/en/shelves/computer-science/signal-processing/) · 452 works

## Cite

Cite as: Shengpeng Ji, Qian Chen, Wen Wang, Jialong Zuo, Minghui Fang, Ziyue Jiang, Hai Huang, Zehan Wang, Xize Cheng, Siqi Zheng and Zhou Zhao. 2025. ControlSpeech: Towards Simultaneous and Independent Zero-shot Speaker Cloning and Zero-shot Language Style Control. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ji2025controlspeech,
  title = {{ControlSpeech: Towards Simultaneous and Independent Zero-shot Speaker Cloning and Zero-shot Language Style Control}},
  author = {Shengpeng Ji, Qian Chen, Wen Wang, Jialong Zuo, Minghui Fang, Ziyue Jiang, Hai Huang, Zehan Wang, Xize Cheng, Siqi Zheng and Zhou Zhao},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.346},
  url = {https://doi.org/10.18653/v1/2025.acl-long.346}
}
```

- [← previous Can Graph Descriptive Order Affect Solving Graph Problems with LLMs?](https://md.research.elamur.ai/free/en/papers/ge-2025-can-graph-descriptive-order-affect/)
- [next → Could Thinking Multilingually Empower LLM Reasoning?](https://md.research.elamur.ai/free/en/papers/gao-2025-could-thinking-multilingually-empower-llm/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/8/)

## References

56 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Sunghwan Ahn, Beom Jun Woo, Min Hyun Han, Chanyeong Moon, and Nam Soo Kim. 2024. Hilcodec: High fidelity and lightweight neural audio codec. arXiv preprint arXiv:2405.04752. — [arXiv:2405.04752](https://arxiv.org/abs/2405.04752) [find](https://scholar.google.com/scholar?q=Sunghwan+Ahn%2C+Beom+Jun+Woo%2C+Min+Hyun+Han%2C+Chanyeong+Moon%2C+and+Nam+Soo+Kim%2E+2024%2E+Hilcodec%3A+High+fidelity+and+lightweight+neural+audio+codec%2E+arXiv+preprint+arXiv%3A2405%2E04752%2E+%E2%80%94+%5BarXiv%3A2405%2E04752%5D%28https)
2. Zalán Borsos, Matt Sharifi, Damien Vincent, Eugene Kharitonov, Neil Zeghidour, and Marco Tagliasacchi. 2023. Soundstorm: Efficient parallel audio generation. arXiv preprint arXiv:2305.09636. — [arXiv:2305.09636](https://arxiv.org/abs/2305.09636) [find](https://scholar.google.com/scholar?q=Zal%C3%A1n+Borsos%2C+Matt+Sharifi%2C+Damien+Vincent%2C+Eugene+Kharitonov%2C+Neil+Zeghidour%2C+and+Marco+Tagliasacchi%2E+2023%2E+Soundstorm%3A+Efficient+parallel+audio+generation%2E+arXiv+preprint+arXiv%3A2305%2E09636%2E+%E2%80%94+%5BarXiv%3A)
3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
4. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. [Sparks of artificial general intelligence: Early experiments with gpt-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/). arXiv preprint arXiv:2303.12712.
5. Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. 2022. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11315–11325. [find](https://scholar.google.com/scholar?q=Huiwen+Chang%2C+Han+Zhang%2C+Lu+Jiang%2C+Ce+Liu%2C+and+William+T+Freeman%2E+2022%2E+Maskgit%3A+Masked+generative+image+transformer%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recogniti)
6. Mingjian Chen, Xu Tan, Bohan Li, Yanqing Liu, Tao Qin, Sheng Zhao, and Tie-Yan Liu. 2021. Adaspeech: Adaptive text to speech for custom voice. arXiv preprint arXiv:2103.00993. — [arXiv:2103.00993](https://arxiv.org/abs/2103.00993) [find](https://scholar.google.com/scholar?q=Mingjian+Chen%2C+Xu+Tan%2C+Bohan+Li%2C+Yanqing+Liu%2C+Tao+Qin%2C+Sheng+Zhao%2C+and+Tie%2DYan+Liu%2E+2021%2E+Adaspeech%3A+Adaptive+text+to+speech+for+custom+voice%2E+arXiv+preprint+arXiv%3A2103%2E00993%2E+%E2%80%94+%5BarXiv%3A2103%2E00993%5D%28htt)
7. Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. 2022. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):1505–1518. [find](https://scholar.google.com/scholar?q=Sanyuan+Chen%2C+Chengyi+Wang%2C+Zhengyang+Chen%2C+Yu+Wu%2C+Shujie+Liu%2C+Zhuo+Chen%2C+Jinyu+Li%2C+Naoyuki+Kanda%2C+Takuya+Yoshioka%2C+Xiong+Xiao%2C+et+al%2E+2022%2E+Wavlm%3A+Large%2Dscale+self%2Dsupervised+pre%2Dtraining+for+full+st)
8. Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. 2022. [High fidelity neural audio compression](https://md.research.elamur.ai/free/en/papers/defossez-2022-high-fidelity-neural-audio-compression/). arXiv preprint arXiv:2210.13438.
9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805.
10. Chenpeng Du and Kai Yu. 2021. Phone-level prosody modelling with gmm-based mdn for diverse and controllable speech synthesis. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:190– 201. [find](https://scholar.google.com/scholar?q=Chenpeng+Du+and+Kai+Yu%2E+2021%2E+Phone%2Dlevel+prosody+modelling+with+gmm%2Dbased+mdn+for+diverse+and+controllable+speech+synthesis%2E+IEEE%2FACM+Transactions+on+Audio%2C+Speech%2C+and+Language+Processing%2C+30%3A190%E2%80%93+2)
11. Zhihao Du, Shiliang Zhang, Kai Hu, and Siqi Zheng. 2024. Funcodec: A fundamental, reproducible and integrable open-source toolkit for neural speech codec. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 591–595. IEEE. [find](https://scholar.google.com/scholar?q=Zhihao+Du%2C+Shiliang+Zhang%2C+Kai+Hu%2C+and+Siqi+Zheng%2E+2024%2E+Funcodec%3A+A+fundamental%2C+reproducible+and+integrable+open%2Dsource+toolkit+for+neural+speech+codec%2E+In+ICASSP+2024%2D2024+IEEE+International+Confer)
12. Tony Duan. 2019. tonyduan/mdn. Original-date: 2019-03-06T01:59:39Z. [find](https://scholar.google.com/scholar?q=Tony+Duan%2E+2019%2E+tonyduan%2Fmdn%2E+Original%2Ddate%3A+2019%2D03%2D06T01%3A59%3A39Z%2E)
13. Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. 2020. [Conformer: Convolution-augmented transformer for speech recognition](https://md.research.elamur.ai/free/en/papers/gulati-2020-conformer-convolution-augmented-transformer-speech/). arXiv preprint arXiv:2005.08100.
14. Zhifang Guo, Yichong Leng, Yihan Wu, Sheng Zhao, and Xu Tan. 2023. Prompttts: Controllable text-to-speech with text descriptions. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. [find](https://scholar.google.com/scholar?q=Zhifang+Guo%2C+Yichong+Leng%2C+Yihan+Wu%2C+Sheng+Zhao%2C+and+Xu+Tan%2E+2023%2E+Prompttts%3A+Controllable+text%2Dto%2Dspeech+with+text+descriptions%2E+In+ICASSP+2023%2D2023+IEEE+International+Conference+on+Acoustics%2C+Speech)
15. Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840– 6851. [find](https://scholar.google.com/scholar?q=Jonathan+Ho%2C+Ajay+Jain%2C+and+Pieter+Abbeel%2E+2020%2E+Denoising+diffusion+probabilistic+models%2E+Advances+in+neural+information+processing+systems%2C+33%3A6840%E2%80%93+6851%2E)
16. Rongjie Huang, Chunlei Zhang, Yongqi Wang, Dongchao Yang, Luping Liu, Zhenhui Ye, Ziyue Jiang, Chao Weng, Zhou Zhao, and Dong Yu. 2023. Make-a-voice: Unified voice synthesis with discrete representation. arXiv preprint arXiv:2305.19269. — [arXiv:2305.19269](https://arxiv.org/abs/2305.19269) [find](https://scholar.google.com/scholar?q=Rongjie+Huang%2C+Chunlei+Zhang%2C+Yongqi+Wang%2C+Dongchao+Yang%2C+Luping+Liu%2C+Zhenhui+Ye%2C+Ziyue+Jiang%2C+Chao+Weng%2C+Zhou+Zhao%2C+and+Dong+Yu%2E+2023%2E+Make%2Da%2Dvoice%3A+Unified+voice+synthesis+with+discrete+representati)
17. Zhichao Huang, Chutong Meng, and Tom Ko. 2024. Repcodec: A speech representation codec for speech tokenization. Preprint, arXiv:2309.00169. [find](https://scholar.google.com/scholar?q=Zhichao+Huang%2C+Chutong+Meng%2C+and+Tom+Ko%2E+2024%2E+Repcodec%3A+A+speech+representation+codec+for+speech+tokenization%2E+Preprint%2C+arXiv%3A2309%2E00169%2E)
18. Min-Jae Hwang, Eunwoo Song, Ryuichi Yamamoto, Frank Soong, and Hong-Goo Kang. 2020. Improving lpcnet-based text-to-speech with linear prediction-structured mixture density network. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7219– 7223… [find](https://scholar.google.com/scholar?q=Min%2DJae+Hwang%2C+Eunwoo+Song%2C+Ryuichi+Yamamoto%2C+Frank+Soong%2C+and+Hong%2DGoo+Kang%2E+2020%2E+Improving+lpcnet%2Dbased+text%2Dto%2Dspeech+with+linear+prediction%2Dstructured+mixture+density+network%2E+In+ICASSP+2020%2D2020)
19. Shengpeng Ji, Minghui Fang, Ziyue Jiang, Rongjie Huang, Jialung Zuo, Shulei Wang, and Zhou Zhao. 2024a. Language-codec: Reducing the gaps between discrete codec representation and speech language models. arXiv preprint arXiv:2402.12208. [find](https://scholar.google.com/scholar?q=Shengpeng+Ji%2C+Minghui+Fang%2C+Ziyue+Jiang%2C+Rongjie+Huang%2C+Jialung+Zuo%2C+Shulei+Wang%2C+and+Zhou+Zhao%2E+2024a%2E+Language%2Dcodec%3A+Reducing+the+gaps+between+discrete+codec+representation+and+speech+language+mode)
20. Shengpeng Ji, Ziyue Jiang, Xize Cheng, Yifu Chen, Minghui Fang, Jialong Zuo, Qian Yang, Ruiqi Li, Ziang Zhang, Xiaoda Yang, et al. 2024b. Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling. arXiv preprint arXiv:2408.16532. — [arXiv:2408.16532](https://arxiv.org/abs/2408.16532) [find](https://scholar.google.com/scholar?q=Shengpeng+Ji%2C+Ziyue+Jiang%2C+Xize+Cheng%2C+Yifu+Chen%2C+Minghui+Fang%2C+Jialong+Zuo%2C+Qian+Yang%2C+Ruiqi+Li%2C+Ziang+Zhang%2C+Xiaoda+Yang%2C+et+al%2E+2024b%2E+Wavtokenizer%3A+an+efficient+acoustic+discrete+codec+tokenizer+f)
21. Shengpeng Ji, Ziyue Jiang, Hanting Wang, Jialong Zuo, and Zhou Zhao. 2024c. Mobilespeech: A fast and high-fidelity framework for mobile zero-shot text-to-speech. arXiv preprint arXiv:2402.09378. [find](https://scholar.google.com/scholar?q=Shengpeng+Ji%2C+Ziyue+Jiang%2C+Hanting+Wang%2C+Jialong+Zuo%2C+and+Zhou+Zhao%2E+2024c%2E+Mobilespeech%3A+A+fast+and+high%2Dfidelity+framework+for+mobile+zero%2Dshot+text%2Dto%2Dspeech%2E+arXiv+preprint+arXiv%3A2402%2E09378%2E)
22. Shengpeng Ji, Jialong Zuo, Minghui Fang, Ziyue Jiang, Feiyang Chen, Xinyu Duan, Baoxing Huai, and Zhou Zhao. 2023. Textrolspeech: A text style control speech corpus with codec language text-to-speech models. arXiv preprint arXiv:2308.14430. — [arXiv:2308.14430](https://arxiv.org/abs/2308.14430) [find](https://scholar.google.com/scholar?q=Shengpeng+Ji%2C+Jialong+Zuo%2C+Minghui+Fang%2C+Ziyue+Jiang%2C+Feiyang+Chen%2C+Xinyu+Duan%2C+Baoxing+Huai%2C+and+Zhou+Zhao%2E+2023%2E+Textrolspeech%3A+A+text+style+control+speech+corpus+with+codec+language+text%2Dto%2Dspeech+)
23. Ziyue Jiang, Jinglin Liu, Yi Ren, Jinzheng He, Zhenhui Ye, Shengpeng Ji, Qian Yang, Chen Zhang, Pengfei Wei, Chunfeng Wang, et al. 2023a. Boosting prompting mechanisms for zero-shot speech synthesis. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Ziyue+Jiang%2C+Jinglin+Liu%2C+Yi+Ren%2C+Jinzheng+He%2C+Zhenhui+Ye%2C+Shengpeng+Ji%2C+Qian+Yang%2C+Chen+Zhang%2C+Pengfei+Wei%2C+Chunfeng+Wang%2C+et+al%2E+2023a%2E+Boosting+prompting+mechanisms+for+zero%2Dshot+speech+synthesis%2E+)
24. Ziyue Jiang, Jinglin Liu, Yi Ren, Jinzheng He, Chen Zhang, Zhenhui Ye, Pengfei Wei, Chunfeng Wang, Xiang Yin, Zejun Ma, et al. 2023b. Mega-tts 2: Zero-shot text-to-speech with arbitrary length speech prompts. arXiv preprint arXiv:2307.07218. — [arXiv:2307.07218](https://arxiv.org/abs/2307.07218) [find](https://scholar.google.com/scholar?q=Ziyue+Jiang%2C+Jinglin+Liu%2C+Yi+Ren%2C+Jinzheng+He%2C+Chen+Zhang%2C+Zhenhui+Ye%2C+Pengfei+Wei%2C+Chunfeng+Wang%2C+Xiang+Yin%2C+Zejun+Ma%2C+et+al%2E+2023b%2E+Mega%2Dtts+2%3A+Zero%2Dshot+text%2Dto%2Dspeech+with+arbitrary+length+speech+)
25. Ziyue Jiang, Yi Ren, Zhenhui Ye, Jinglin Liu, Chen Zhang, Qian Yang, Shengpeng Ji, Rongjie Huang, Chunfeng Wang, Xiang Yin, et al. 2023c. Megatts: Zero-shot text-to-speech at scale with intrinsic inductive bias. arXiv preprint arXiv:2306.03509. — [arXiv:2306.03509](https://arxiv.org/abs/2306.03509) [find](https://scholar.google.com/scholar?q=Ziyue+Jiang%2C+Yi+Ren%2C+Zhenhui+Ye%2C+Jinglin+Liu%2C+Chen+Zhang%2C+Qian+Yang%2C+Shengpeng+Ji%2C+Rongjie+Huang%2C+Chunfeng+Wang%2C+Xiang+Yin%2C+et+al%2E+2023c%2E+Megatts%3A+Zero%2Dshot+text%2Dto%2Dspeech+at+scale+with+intrinsic+indu)
26. Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, et al. 2024. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. arXiv preprint arXiv:2403.03100. — [arXiv:2403.03100](https://arxiv.org/abs/2403.03100) [find](https://scholar.google.com/scholar?q=Zeqian+Ju%2C+Yuancheng+Wang%2C+Kai+Shen%2C+Xu+Tan%2C+Detai+Xin%2C+Dongchao+Yang%2C+Yanqing+Liu%2C+Yichong+Leng%2C+Kaitao+Song%2C+Siliang+Tang%2C+et+al%2E+2024%2E+Naturalspeech+3%3A+Zero%2Dshot+speech+synthesis+with+factorized+co)
27. Jacob Kahn, Morgane Rivière, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Pierre-Emmanuel Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, et al. 2020. Libri-light: A benchmark for asr with limited or no supervision. In ICASSP 2020-2020 IEEE International Conference o… [find](https://scholar.google.com/scholar?q=Jacob+Kahn%2C+Morgane+Rivi%C3%A8re%2C+Weiyi+Zheng%2C+Evgeny+Kharitonov%2C+Qiantong+Xu%2C+Pierre%2DEmmanuel+Mazar%C3%A9%2C+Julien+Karadayi%2C+Vitaliy+Liptchinsky%2C+Ronan+Collobert%2C+Christian+Fuegen%2C+et+al%2E+2020%2E+Libri%2Dlight%3A+A+b)
28. Eugene Kharitonov, Damien Vincent, Zalán Borsos, Raphaël Marinier, Sertan Girgin, Olivier Pietquin, Matt Sharifi, Marco Tagliasacchi, and Neil Zeghidour. 2023. [Speak, read and prompt: High-fidelity text-to-speech with minimal supervision](https://md.research.elamur.ai/free/en/papers/kharitonov-2023-speak-read-prompt-high-fidelity/). arXiv preprint arXiv:2302.03540.
29. Jaehyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. 2020. Glow-tts: A generative flow for text-to-speech via monotonic alignment search. Advances in Neural Information Processing Systems, 33:8067–8077. [find](https://scholar.google.com/scholar?q=Jaehyeon+Kim%2C+Sungwon+Kim%2C+Jungil+Kong%2C+and+Sungroh+Yoon%2E+2020%2E+Glow%2Dtts%3A+A+generative+flow+for+text%2Dto%2Dspeech+via+monotonic+alignment+search%2E+Advances+in+Neural+Information+Processing+Systems%2C+33%3A806)
30. Jaehyeon Kim, Jungil Kong, and Juhee Son. 2021. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In International Conference on Machine Learning, pages 5530–5540. PMLR. [find](https://scholar.google.com/scholar?q=Jaehyeon+Kim%2C+Jungil+Kong%2C+and+Juhee+Son%2E+2021%2E+Conditional+variational+autoencoder+with+adversarial+learning+for+end%2Dto%2Dend+text%2Dto%2Dspeech%2E+In+International+Conference+on+Machine+Learning%2C+pages+5530)
31. Sungwon Kim, Kevin Shih, Joao Felipe Santos, Evelina Bakhturina, Mikyas Desta, Rafael Valle, Sungroh Yoon, Bryan Catanzaro, et al. 2024. P-flow: A fast and data-efficient zero-shot tts through speech prompting. Advances in Neural Information Processing Systems, 36. [find](https://scholar.google.com/scholar?q=Sungwon+Kim%2C+Kevin+Shih%2C+Joao+Felipe+Santos%2C+Evelina+Bakhturina%2C+Mikyas+Desta%2C+Rafael+Valle%2C+Sungroh+Yoon%2C+Bryan+Catanzaro%2C+et+al%2E+2024%2E+P%2Dflow%3A+A+fast+and+data%2Defficient+zero%2Dshot+tts+through+speech+)
32. Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. 2024. High-fidelity audio compression with improved rvqgan. Advances in Neural Information Processing Systems, 36. [find](https://scholar.google.com/scholar?q=Rithesh+Kumar%2C+Prem+Seetharaman%2C+Alejandro+Luebs%2C+Ishaan+Kumar%2C+and+Kundan+Kumar%2E+2024%2E+High%2Dfidelity+audio+compression+with+improved+rvqgan%2E+Advances+in+Neural+Information+Processing+Systems%2C+36%2E)
33. Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al. 2023. Voicebox: Text-guided multilingual universal speech generation at scale. arXiv preprint arXiv:2306.15687. — [arXiv:2306.15687](https://arxiv.org/abs/2306.15687) [find](https://scholar.google.com/scholar?q=Matthew+Le%2C+Apoorv+Vyas%2C+Bowen+Shi%2C+Brian+Karrer%2C+Leda+Sari%2C+Rashel+Moritz%2C+Mary+Williamson%2C+Vimal+Manohar%2C+Yossi+Adi%2C+Jay+Mahadeokar%2C+et+al%2E+2023%2E+Voicebox%3A+Text%2Dguided+multilingual+universal+speech+)
34. Sang-gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon. 2022. Bigvgan: A universal neural vocoder with large-scale training. arXiv preprint arXiv:2206.04658. — [arXiv:2206.04658](https://arxiv.org/abs/2206.04658) [find](https://scholar.google.com/scholar?q=Sang%2Dgil+Lee%2C+Wei+Ping%2C+Boris+Ginsburg%2C+Bryan+Catanzaro%2C+and+Sungroh+Yoon%2E+2022%2E+Bigvgan%3A+A+universal+neural+vocoder+with+large%2Dscale+training%2E+arXiv+preprint+arXiv%3A2206%2E04658%2E+%E2%80%94+%5BarXiv%3A2206%2E04658%5D%28ht)
35. Yichong Leng, Zhifang Guo, Kai Shen, Xu Tan, Zeqian Ju, Yanqing Liu, Yufei Liu, Dongchao Yang, Leying Zhang, Kaitao Song, et al. 2023. Prompttts 2: Describing and generating voices with text prompt. arXiv preprint arXiv:2309.02285. — [arXiv:2309.02285](https://arxiv.org/abs/2309.02285) [find](https://scholar.google.com/scholar?q=Yichong+Leng%2C+Zhifang+Guo%2C+Kai+Shen%2C+Xu+Tan%2C+Zeqian+Ju%2C+Yanqing+Liu%2C+Yufei+Liu%2C+Dongchao+Yang%2C+Leying+Zhang%2C+Kaitao+Song%2C+et+al%2E+2023%2E+Prompttts+2%3A+Describing+and+generating+voices+with+text+prompt%2E+a)
36. Hanzhao Li, Liumeng Xue, Haohan Guo, Xinfa Zhu, Yuanjun Lv, Lei Xie, Yunlin Chen, Hao Yin, and Zhifei Li. 2024. Single-codec: Single-codebook speech codec towards high-performance speech generation. arXiv preprint arXiv:2406.07422. — [arXiv:2406.07422](https://arxiv.org/abs/2406.07422) [find](https://scholar.google.com/scholar?q=Hanzhao+Li%2C+Liumeng+Xue%2C+Haohan+Guo%2C+Xinfa+Zhu%2C+Yuanjun+Lv%2C+Lei+Xie%2C+Yunlin+Chen%2C+Hao+Yin%2C+and+Zhifei+Li%2E+2024%2E+Single%2Dcodec%3A+Single%2Dcodebook+speech+codec+towards+high%2Dperformance+speech+generation%2E+a)
37. Guanghou Liu, Yongmao Zhang, Yi Lei, Yunlin Chen, Rui Wang, Zhifei Li, and Lei Xie. 2023. Promptstyle: Controllable style transfer for text-to-speech with natural language descriptions. arXiv preprint arXiv:2305.19522. — [arXiv:2305.19522](https://arxiv.org/abs/2305.19522) [find](https://scholar.google.com/scholar?q=Guanghou+Liu%2C+Yongmao+Zhang%2C+Yi+Lei%2C+Yunlin+Chen%2C+Rui+Wang%2C+Zhifei+Li%2C+and+Lei+Xie%2E+2023%2E+Promptstyle%3A+Controllable+style+transfer+for+text%2Dto%2Dspeech+with+natural+language+descriptions%2E+arXiv+preprint)
38. Haohe Liu, Xuenan Xu, Yi Yuan, Mengyue Wu, Wenwu Wang, and Mark D Plumbley. 2024. Semanticodec: An ultra low bitrate semantic audio codec for general sound. arXiv preprint arXiv:2405.00233. — [arXiv:2405.00233](https://arxiv.org/abs/2405.00233) [find](https://scholar.google.com/scholar?q=Haohe+Liu%2C+Xuenan+Xu%2C+Yi+Yuan%2C+Mengyue+Wu%2C+Wenwu+Wang%2C+and+Mark+D+Plumbley%2E+2024%2E+Semanticodec%3A+An+ultra+low+bitrate+semantic+audio+codec+for+general+sound%2E+arXiv+preprint+arXiv%3A2405%2E00233%2E+%E2%80%94+%5BarXiv%3A2)
39. Ziyang Ma, Zhisheng Zheng, Jiaxin Ye, Jinchao Li, Zhifu Gao, Shiliang Zhang, and Xie Chen. 2023. emotion2vec: Self-supervised pre-training for speech emotion representation. arXiv preprint arXiv:2312.15185. [find](https://scholar.google.com/scholar?q=Ziyang+Ma%2C+Zhisheng+Zheng%2C+Jiaxin+Ye%2C+Jinchao+Li%2C+Zhifu+Gao%2C+Shiliang+Zhang%2C+and+Xie+Chen%2E+2023%2E+emotion2vec%3A+Self%2Dsupervised+pre%2Dtraining+for+speech+emotion+representation%2E+arXiv+preprint+arXiv%3A2312%2E)
40. Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. 2017. Montreal forced aligner: Trainable text-speech alignment using kaldi. In Interspeech, volume 2017, pages 498–502. [find](https://scholar.google.com/scholar?q=Michael+McAuliffe%2C+Michaela+Socolof%2C+Sarah+Mihuc%2C+Michael+Wagner%2C+and+Morgan+Sonderegger%2E+2017%2E+Montreal+forced+aligner%3A+Trainable+text%2Dspeech+alignment+using+kaldi%2E+In+Interspeech%2C+volume+2017%2C+pages)
41. Yu Pan, Lei Ma, and Jianjun Zhao. 2024. Promptcodec: High-fidelity neural speech codec using disentangled representation learning based adaptive feature-aware prompt encoders. arXiv preprint arXiv:2404.02702. — [arXiv:2404.02702](https://arxiv.org/abs/2404.02702) [find](https://scholar.google.com/scholar?q=Yu+Pan%2C+Lei+Ma%2C+and+Jianjun+Zhao%2E+2024%2E+Promptcodec%3A+High%2Dfidelity+neural+speech+codec+using+disentangled+representation+learning+based+adaptive+feature%2Daware+prompt+encoders%2E+arXiv+preprint+arXiv%3A240)
42. Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. 2020. Fastspeech 2: Fast and high-quality end-to-end text to speech. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yi+Ren%2C+Chenxu+Hu%2C+Xu+Tan%2C+Tao+Qin%2C+Sheng+Zhao%2C+Zhou+Zhao%2C+and+Tie%2DYan+Liu%2E+2020%2E+Fastspeech+2%3A+Fast+and+high%2Dquality+end%2Dto%2Dend+text+to+speech%2E+In+International+Conference+on+Learning+Representations)
43. Yong Ren, Tao Wang, Jiangyan Yi, Le Xu, Jianhua Tao, Chu Yuan Zhang, and Junzuo Zhou. 2024. Fewer-token neural speech codec with time-invariant codes. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 12737–12741. IEEE. [find](https://scholar.google.com/scholar?q=Yong+Ren%2C+Tao+Wang%2C+Jiangyan+Yi%2C+Le+Xu%2C+Jianhua+Tao%2C+Chu+Yuan+Zhang%2C+and+Junzuo+Zhou%2E+2024%2E+Fewer%2Dtoken+neural+speech+codec+with+time%2Dinvariant+codes%2E+In+ICASSP+2024%2D2024+IEEE+International+Conference)
44. Kai Shen, Zeqian Ju, Xu Tan, Yanqing Liu, Yichong Leng, Lei He, Tao Qin, Sheng Zhao, and Jiang Bian. 2023. Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. arXiv preprint arXiv:2304.09116. — [arXiv:2304.09116](https://arxiv.org/abs/2304.09116) [find](https://scholar.google.com/scholar?q=Kai+Shen%2C+Zeqian+Ju%2C+Xu+Tan%2C+Yanqing+Liu%2C+Yichong+Leng%2C+Lei+He%2C+Tao+Qin%2C+Sheng+Zhao%2C+and+Jiang+Bian%2E+2023%2E+Naturalspeech+2%3A+Latent+diffusion+models+are+natural+and+zero%2Dshot+speech+and+singing+synthes)
45. Hubert Siuzdak. 2023. Vocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis. arXiv preprint arXiv:2306.00814. — [arXiv:2306.00814](https://arxiv.org/abs/2306.00814) [find](https://scholar.google.com/scholar?q=Hubert+Siuzdak%2E+2023%2E+Vocos%3A+Closing+the+gap+between+time%2Ddomain+and+fourier%2Dbased+neural+vocoders+for+high%2Dquality+audio+synthesis%2E+arXiv+preprint+arXiv%3A2306%2E00814%2E+%E2%80%94+%5BarXiv%3A2306%2E00814%5D%28https%3A%2F%2Farxiv)
46. Xu Tan, Jiawei Chen, Haohe Liu, Jian Cong, Chen Zhang, Yanqing Liu, Xi Wang, Yichong Leng, Yuanhao Yi, Lei He, et al. 2024. Naturalspeech: End-to-end text-to-speech synthesis with human-level quality. IEEE Transactions on Pattern Analysis and Machine Intelligence. [find](https://scholar.google.com/scholar?q=Xu+Tan%2C+Jiawei+Chen%2C+Haohe+Liu%2C+Jian+Cong%2C+Chen+Zhang%2C+Yanqing+Liu%2C+Xi+Wang%2C+Yichong+Leng%2C+Yuanhao+Yi%2C+Lei+He%2C+et+al%2E+2024%2E+Naturalspeech%3A+End%2Dto%2Dend+text%2Dto%2Dspeech+synthesis+with+human%2Dlevel+quality%2E)
47. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971.
48. Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. 2022. Phenaki: Variable length video generation from open domain textual description. arXiv preprint arXiv:2210.02399. — [arXiv:2210.02399](https://arxiv.org/abs/2210.02399) [find](https://scholar.google.com/scholar?q=Ruben+Villegas%2C+Mohammad+Babaeizadeh%2C+Pieter%2DJan+Kindermans%2C+Hernan+Moraldo%2C+Han+Zhang%2C+Mohammad+Taghi+Saffar%2C+Santiago+Castro%2C+Julius+Kunze%2C+and+Dumitru+Erhan%2E+2022%2E+Phenaki%3A+Variable+length+video+ge)
49. Apoorv Vyas, Bowen Shi, Matthew Le, Andros Tjandra, Yi-Chiao Wu, Baishan Guo, Jiemin Zhang, Xinyue Zhang, Robert Adkins, William Ngan, et al. 2023. Audiobox: Unified audio generation with natural language prompts. arXiv preprint arXiv:2312.15821. — [arXiv:2312.15821](https://arxiv.org/abs/2312.15821) [find](https://scholar.google.com/scholar?q=Apoorv+Vyas%2C+Bowen+Shi%2C+Matthew+Le%2C+Andros+Tjandra%2C+Yi%2DChiao+Wu%2C+Baishan+Guo%2C+Jiemin+Zhang%2C+Xinyue+Zhang%2C+Robert+Adkins%2C+William+Ngan%2C+et+al%2E+2023%2E+Audiobox%3A+Unified+audio+generation+with+natural+lang)
50. Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. 2023. [Neural codec language models are zero-shot text to speech synthesizers](https://md.research.elamur.ai/free/en/papers/wang-2023-neural-codec-language-models-zero/). arXiv preprint arXiv:2301.02111.
51. Yi-Chiao Wu, Israel D Gebru, Dejan Marković, and Alexander Richard. 2023. Audiodec: An open-source streaming high-fidelity neural audio codec. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. [find](https://scholar.google.com/scholar?q=Yi%2DChiao+Wu%2C+Israel+D+Gebru%2C+Dejan+Markovic%CC%81%2C+and+Alexander+Richard%2E+2023%2E+Audiodec%3A+An+open%2Dsource+streaming+high%2Dfidelity+neural+audio+codec%2E+In+ICASSP+2023%2D2023+IEEE+International+Conference+on+Aco)
52. Dongchao Yang, Songxiang Liu, Rongjie Huang, Jinchuan Tian, Chao Weng, and Yuexian Zou. 2023a. Hifi-codec: Group-residual vector quantization for high fidelity audio codec. arXiv preprint arXiv:2305.02765. [find](https://scholar.google.com/scholar?q=Dongchao+Yang%2C+Songxiang+Liu%2C+Rongjie+Huang%2C+Jinchuan+Tian%2C+Chao+Weng%2C+and+Yuexian+Zou%2E+2023a%2E+Hifi%2Dcodec%3A+Group%2Dresidual+vector+quantization+for+high+fidelity+audio+codec%2E+arXiv+preprint+arXiv%3A2305%2E0)
53. Dongchao Yang, Songxiang Liu, Rongjie Huang, Chao Weng, and Helen Meng. 2023b. Instructtts: Modelling expressive tts in discrete latent space with natural language style prompt. arXiv preprint arXiv:2301.13662. — [arXiv:2301.13662](https://arxiv.org/abs/2301.13662) [find](https://scholar.google.com/scholar?q=Dongchao+Yang%2C+Songxiang+Liu%2C+Rongjie+Huang%2C+Chao+Weng%2C+and+Helen+Meng%2E+2023b%2E+Instructtts%3A+Modelling+expressive+tts+in+discrete+latent+space+with+natural+language+style+prompt%2E+arXiv+preprint+arXiv%3A2)
54. Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. 2021. [Soundstream: An end-to-end neural audio codec](https://md.research.elamur.ai/free/en/papers/zeghidour-2021-soundstream-end-end-neural-audio/). IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495–507.
55. Heiga Zen and Andrew Senior. 2014. Deep mixture density networks for acoustic modeling in statistical parametric speech synthesis. In 2014 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 3844–3848. IEEE. [find](https://scholar.google.com/scholar?q=Heiga+Zen+and+Andrew+Senior%2E+2014%2E+Deep+mixture+density+networks+for+acoustic+modeling+in+statistical+parametric+speech+synthesis%2E+In+2014+IEEE+international+conference+on+acoustics%2C+speech+and+signal)
56. Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. 2023. [Speechtokenizer: Unified speech tokenizer for speech large language models](https://md.research.elamur.ai/free/en/papers/zhang-2023-speechtokenizer-unified-speech-tokenizer-speech/). arXiv preprint arXiv:2308.16692.

*Updated 1 November 2026*