---
title: "Text-Free Prosody-Aware Generative Spoken Language Modeling"
description: "This paper introduces pGSLM, a text-free spoken language model that handles phonetic content and… 48 references, cited by 49 papers here."
authors: ["Eugene Kharitonov", "Ann Lee", "Adam Polyak", "Yossi Adi", "Jade Copet", "Kushal Lakhotia", "Tu Anh Nguyen", "Morgane Riviere", "Abdelrahman Mohamed", "Emmanuel Dupoux", "Wei-Ning Hsu"]
year: 2022
venue: "Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2022.acl-long.593/"
doi: "10.18653/v1/2022.acl-long.593"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kharitonov-2022-text-free-prosody-aware-generative/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kharitonov-2022-text-free-prosody-aware-generative/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/) / Text-Free Prosody-Aware Generative Spoken Language Modeling

# Text-Free Prosody-Aware Generative Spoken Language Modeling

**Authors:** [Eugene Kharitonov](https://md.research.elamur.ai/free/en/authors/eugene-kharitonov/), [Ann Lee](https://md.research.elamur.ai/free/en/authors/ann-lee/), [Adam Polyak](https://md.research.elamur.ai/free/en/authors/adam-polyak/), [Yossi Adi](https://md.research.elamur.ai/free/en/authors/yossi-adi/), [Jade Copet](https://md.research.elamur.ai/free/en/authors/jade-copet/), [Kushal Lakhotia](https://md.research.elamur.ai/free/en/authors/kushal-lakhotia/), [Tu Anh Nguyen](https://md.research.elamur.ai/free/en/authors/tu-anh-nguyen/), [Morgane Riviere](https://md.research.elamur.ai/free/en/authors/morgane-riviere/), [Abdelrahman Mohamed](https://md.research.elamur.ai/free/en/authors/abdelrahman-mohamed/), [Emmanuel Dupoux](https://md.research.elamur.ai/free/en/authors/emmanuel-dupoux/) and [Wei-Ning Hsu](https://md.research.elamur.ai/free/en/authors/wei-ning-hsu/) · **Year:** 2022 · **Venue:** [Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2022](https://md.research.elamur.ai/free/en/venues/acl/2022/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.acl-long.593/) · [PDF](https://aclanthology.org/2022.acl-long.593.pdf) · [DOI 10.18653/v1/2022.acl-long.593](https://doi.org/10.18653/v1/2022.acl-long.593) · [OpenAlex](https://openalex.org/W3198217962)

**Citations:** 52 citations (OpenAlex, 2026-10-14) · [49](https://md.research.elamur.ai/free/en/papers/kharitonov-2022-text-free-prosody-aware-generative/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Most speech pre-training work targets classification, and the one earlier generative approach, GSLM, models speech through discovered phone-like units that drop most prosodic information. As a result, it neither uses intonation and rhythm for comprehension nor produces expressive continuations. This paper introduces pGSLM, a text-free spoken language model that handles phonetic content and prosody jointly.

Content is encoded as discrete HuBERT units. Prosody is encoded as unit duration and speaker-mean-normalized log F0, which keeps expressiveness while removing much of the speaker identity. A Multi-Stream Transformer LM (MS-TLM) autoregressively predicts all of these streams, and an adapted unit HiFi-GAN converts its output into waveforms.

The authors reuse GSLM's content metrics and propose new prosody metrics, including teacher-forced MAE and continuation correctness, consistency and expressiveness. Feeding prosody back as input improves both prosody and content modeling. Quantized prosody streams produce the most natural and meaningful speech continuations.

The work matters because it shows that unlabeled audio alone can support coherent, expressive prompted speech generation. This is relevant to unwritten or low-resource languages and to future speech-based dialogue systems.

## Key points
- **Phonetic representation:** 100 k-means clusters over the 6th transformer layer of base HuBERT, at 50 units per second, run-length encoded into (unit, duration) pairs.
- **Prosody representation:**
  - Log F0 is normalized only by the speaker mean, so variance is kept as a cue for expressiveness.
  - Unvoiced frames are set to 0.
- **Effect of prosodic input on continuation (large models trained on LL-6k):**
  - lf MAE falls from 0.096 to 0.077.
  - For the continuous models, duration correlation rises from 0.176 to 0.344 and F0 correlation from 0.093 to 0.494.
  - Word-level BLEU also improves.
- **Quantized vs. continuous prosody:**
  - Quantized models benefit from sampling several candidates and reach lower min-MAE.
  - They produce up to twice the F0 diversity, which the authors attribute to better coverage of multi-modal distributions.
- **Human evaluation and release:**
  - The variant that takes prosody as input and uses discretized prosody streams achieves the top MOS and M-MOS scores, and it is on par with the continuous version on P-MOS.
  - Code and models are released in fairseq (examples/textless_nlp/pgslm).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Meta](https://md.research.elamur.ai/free/en/companies/meta/) — Company, Menlo Park, United States · 158 works · 211,459 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🔊 Signal Processing](https://md.research.elamur.ai/free/en/shelves/computer-science/signal-processing/) · 213 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works · [🎙️ Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) · 233 works

## Cite

Cite as: Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Morgane Riviere, Abdelrahman Mohamed, Emmanuel Dupoux and Wei-Ning Hsu. 2022. Text-Free Prosody-Aware Generative Spoken Language Modeling. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{kharitonov2022text,
  title = {{Text-Free Prosody-Aware Generative Spoken Language Modeling}},
  author = {Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Morgane Riviere, Abdelrahman Mohamed, Emmanuel Dupoux and Wei-Ning Hsu},
  booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.593},
  url = {https://doi.org/10.18653/v1/2022.acl-long.593}
}
```

- [← previous FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/wan-2022-factpegasus-factuality-aware-pre-training/)
- [next → Time Waits for No One! Analysis and Challenges of Temporal Misalignment](https://md.research.elamur.ai/free/en/papers/luu-2022-time-waits-no-one-analysis/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/)

## References

48 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Adaeze Adigwe, Noé Tits, Kevin El Haddad, Sarah Ostadabbas, and Thierry Dutoit. 2018. The emotional voices database: Towards controlling the emotion dimension in voice generation systems. arXiv preprint arXiv:1806.09514. — [arXiv:1806.09514](https://arxiv.org/abs/1806.09514) [find](https://scholar.google.com/scholar?q=Adaeze+Adigwe%2C+No%C3%A9+Tits%2C+Kevin+El+Haddad%2C+Sarah+Ostadabbas%2C+and+Thierry+Dutoit%2E+2018%2E+The+emotional+voices+database%3A+Towards+controlling+the+emotion+dimension+in+voice+generation+systems%2E+arXiv+prepri)
2. Wei-Ning Hsu, Yao-Hung Hubert Tsai, Benjamin Bolte, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021c. HuBERT: How much can a bad teacher benefit ASR pre-training? In Neural Information Processing Systems Workshop on Self-Supervised Learning for Speech and Audio Processing Workshop, pages 6533–65… [find](https://scholar.google.com/scholar?q=Wei%2DNing+Hsu%2C+Yao%2DHung+Hubert+Tsai%2C+Benjamin+Bolte%2C+Ruslan+Salakhutdinov%2C+and+Abdelrahman+Mohamed%2E+2021c%2E+HuBERT%3A+How+much+can+a+bad+teacher+benefit+ASR+pre%2Dtraining%3F+In+Neural+Information+Processing+)
3. Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. [wav2vec 2.0: A framework for self-supervised learning of speech representations](https://md.research.elamur.ai/free/en/papers/baevski-2020-wav2vec-2-0-framework-self/). arXiv preprint arXiv:2006.11477.
4. Songfang Huang and Steve Renals. 2007. Using prosodic features in language models for meetings. In International Workshop on Machine Learning for Multimodal Interaction, pages 192–203. Springer. [find](https://scholar.google.com/scholar?q=Songfang+Huang+and+Steve+Renals%2E+2007%2E+Using+prosodic+features+in+language+models+for+meetings%2E+In+International+Workshop+on+Machine+Learning+for+Multimodal+Interaction%2C+pages+192%E2%80%93203%2E+Springer%2E)
5. Douglas Biber. 1991. Variation across speech and writing. Cambridge University Press. [find](https://scholar.google.com/scholar?q=Douglas+Biber%2E+1991%2E+Variation+across+speech+and+writing%2E+Cambridge+University+Press%2E)
6. Keith Ito and Linda Johnson. 2017. The lj speech dataset. [https://keithito.com/](https://keithito.com/) LJ-Speech-Dataset/. [↗](https://keithito.com/)
7. [Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,…](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
8. Jacob Kahn et al. 2020. Libri-light: A benchmark for asr with limited or no supervision. In ICASSP. [find](https://scholar.google.com/scholar?q=Jacob+Kahn+et+al%2E+2020%2E+Libri%2Dlight%3A+A+benchmark+for+asr+with+limited+or+no+supervision%2E+In+ICASSP%2E)
9. Eugene Kharitonov, Morgane Rivière, Gabriel Synnaeve, Lior Wolf, Pierre-Emmanuel Mazaré, Matthijs Douze, and Emmanuel Dupoux. 2021. Data augmenting contrastive learning of speech representations in the time domain. arXiv preprint arXiv:2007.00991, pages 215–222. — [arXiv:2007.00991](https://arxiv.org/abs/2007.00991) [find](https://scholar.google.com/scholar?q=Eugene+Kharitonov%2C+Morgane+Rivi%C3%A8re%2C+Gabriel+Synnaeve%2C+Lior+Wolf%2C+Pierre%2DEmmanuel+Mazar%C3%A9%2C+Matthijs+Douze%2C+and+Emmanuel+Dupoux%2E+2021%2E+Data+augmenting+contrastive+learning+of+speech+representations+in+th)
10. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. [find](https://scholar.google.com/scholar?q=Diederik+P+Kingma+and+Jimmy+Ba%2E+2014%2E+Adam%3A+A+method+for+stochastic+optimization%2E+arXiv+preprint+arXiv%3A1412%2E6980%2E)
11. Wei Chu and Abeer Alwan. 2009. Reducing f0 frame error of f0 tracking algorithms under noisy conditions with an unvoiced/voiced classification frontend. In 2009 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 3969–3972. IEEE. [find](https://scholar.google.com/scholar?q=Wei+Chu+and+Abeer+Alwan%2E+2009%2E+Reducing+f0+frame+error+of+f0+tracking+algorithms+under+noisy+conditions+with+an+unvoiced%2Fvoiced+classification+frontend%2E+In+2009+IEEE+International+Conference+on+Acoust)
12. Kushal Lakhotia, Evgeny Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Adelrahman Mohamed, et al. 2021. Generative spoken language modeling from raw audio. arXiv preprint arXiv:2102.01192. — [arXiv:2102.01192](https://arxiv.org/abs/2102.01192) [find](https://scholar.google.com/scholar?q=Kushal+Lakhotia%2C+Evgeny+Kharitonov%2C+Wei%2DNing+Hsu%2C+Yossi+Adi%2C+Adam+Polyak%2C+Benjamin+Bolte%2C+Tu%2DAnh+Nguyen%2C+Jade+Copet%2C+Alexei+Baevski%2C+Adelrahman+Mohamed%2C+et+al%2E+2021%2E+Generative+spoken+language+modelin)
13. Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James Glass. 2019. An unsupervised autoregressive model for speech representation learning. In Proc. INTER-SPEECH, pages 146–150. [find](https://scholar.google.com/scholar?q=Yu%2DAn+Chung%2C+Wei%2DNing+Hsu%2C+Hao+Tang%2C+and+James+Glass%2E+2019%2E+An+unsupervised+autoregressive+model+for+speech+representation+learning%2E+In+Proc%2E+INTER%2DSPEECH%2C+pages+146%E2%80%93150%2E)
14. Adrian Łańcucki. 2021. Fastpitch: Parallel text-to-speech with pitch prediction. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6588–6592. IEEE. [find](https://scholar.google.com/scholar?q=Adrian+%C5%81an%CC%81cucki%2E+2021%2E+Fastpitch%3A+Parallel+text%2Dto%2Dspeech+with+pitch+prediction%2E+In+ICASSP+2021%2D2021+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+%28ICASSP%29%2C+pages+6588%E2%80%936592)
15. Jacob T Cohen, Alma Cohen, Limor Benyamini, Yossi Adi, and Joseph Keshet. 2019. Predicting glottal closure insufficiency using fundamental frequency contour analysis. Head & neck, 41(7):2324–2331. [find](https://scholar.google.com/scholar?q=Jacob+T+Cohen%2C+Alma+Cohen%2C+Limor+Benyamini%2C+Yossi+Adi%2C+and+Joseph+Keshet%2E+2019%2E+Predicting+glottal+closure+insufficiency+using+fundamental+frequency+contour+analysis%2E+Head+%26+neck%2C+41%287%29%3A2324%E2%80%932331%2E)
16. M. Paul Lewis, Gary F. Simon, and Charles D. Fennig. 2016. Ethnologue: Languages of the World, Nineteenth edition. SIL International. Online version: [http://www.ethnologue.com](http://www.ethnologue.com). [↗](http://www.ethnologue.com)
17. Anne Cutler, Delphine Dahan, and Wilma Van Donselaar. 1997. Prosody in the comprehension of spoken language: A literature review. Language and speech, 40(2):141–201. [find](https://scholar.google.com/scholar?q=Anne+Cutler%2C+Delphine+Dahan%2C+and+Wilma+Van+Donselaar%2E+1997%2E+Prosody+in+the+comprehension+of+spoken+language%3A+A+literature+review%2E+Language+and+speech%2C+40%282%29%3A141%E2%80%93201%2E)
18. Shaoshi Ling and Yuzong Liu. 2020. DeCoAR 2.0: Deep contextualized acoustic representations with vector quantization. arXiv preprint arXiv:2012.06659. — [arXiv:2012.06659](https://arxiv.org/abs/2012.06659) [find](https://scholar.google.com/scholar?q=Shaoshi+Ling+and+Yuzong+Liu%2E+2020%2E+DeCoAR+2%2E0%3A+Deep+contextualized+acoustic+representations+with+vector+quantization%2E+arXiv+preprint+arXiv%3A2012%2E06659%2E+%E2%80%94+%5BarXiv%3A2012%2E06659%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2012%2E0)
19. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021a. [Hubert: Self-supervised speech representation learning by masked prediction of hidden units](https://md.research.elamur.ai/free/en/papers/hsu-2021-hubert-self-supervised-speech-representation/). arXiv preprint arXiv:2106.07447.
20. A. T. Liu, S. Yang, P. Chi, P. Hsu, and H. Lee. 2020. Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6419–6423. [find](https://scholar.google.com/scholar?q=A%2E+T%2E+Liu%2C+S%2E+Yang%2C+P%2E+Chi%2C+P%2E+Hsu%2C+and+H%2E+Lee%2E+2020%2E+Mockingjay%3A+Unsupervised+speech+representation+learning+with+deep+bidirectional+transformer+encoders%2E+In+IEEE+International+Conference+on+Acoustic)
21. Wei-Ning Hsu, Anuroop Sriram, Alexei Baevski, Tatiana Likhomanenko, Qiantong Xu, Vineel Pratap, Jacob Kahn, Ann Lee, Ronan Collobert, Gabriel Synnaeve, et al. 2021b. Robust wav2vec 2.0: Analyzing domain shift in self-supervised pre-training. In Interspeech. [find](https://scholar.google.com/scholar?q=Wei%2DNing+Hsu%2C+Anuroop+Sriram%2C+Alexei+Baevski%2C+Tatiana+Likhomanenko%2C+Qiantong+Xu%2C+Vineel+Pratap%2C+Jacob+Kahn%2C+Ann+Lee%2C+Ronan+Collobert%2C+Gabriel+Synnaeve%2C+et+al%2E+2021b%2E+Robust+wav2vec+2%2E0%3A+Analyzing+doma)
22. Tomohiro Nakatani, Shigeaki Amano, Toshio Irino, Kentaro Ishizuka, and Tadahisa Kondo. 2008. A method for fundamental frequency estimation and voicing decision: Application to infant utterances recorded in real acoustical environments. Speech Communication, 50(3):203–214. [find](https://scholar.google.com/scholar?q=Tomohiro+Nakatani%2C+Shigeaki+Amano%2C+Toshio+Irino%2C+Kentaro+Ishizuka%2C+and+Tadahisa+Kondo%2E+2008%2E+A+method+for+fundamental+frequency+estimation+and+voicing+decision%3A+Application+to+infant+utterances+record)
23. Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. 2016. [WaveNet: A generative model for raw audio](https://md.research.elamur.ai/free/en/papers/oord-2016-wavenet-generative-model-raw-audio/). arXiv preprint arXiv:1609.03499.
24. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. [Representation learning with contrastive predictive coding](https://md.research.elamur.ai/free/en/papers/oord-2018-representation-learning-contrastive-predictive-coding/). arXiv preprint arXiv:1807.03748.
25. Mari Ostendorf, Izhak Shafran, and Rebecca Bates. 2003. Prosody models for conversational speech recognition. [find](https://scholar.google.com/scholar?q=Mari+Ostendorf%2C+Izhak+Shafran%2C+and+Rebecca+Bates%2E+2003%2E+Prosody+models+for+conversational+speech+recognition%2E)
26. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based on public domain audio books. In ICASSP. [find](https://scholar.google.com/scholar?q=Vassil+Panayotov%2C+Guoguo+Chen%2C+Daniel+Povey%2C+and+Sanjeev+Khudanpur%2E+2015%2E+Librispeech%3A+an+asr+corpus+based+on+public+domain+audio+books%2E+In+ICASSP%2E)
27. Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdelrahman Mohamed, and Emmanuel Dupoux. 2021. Speech resynthesis from discrete disentangled self-supervised representations. In Proc. INTER-SPEECH. [find](https://scholar.google.com/scholar?q=Adam+Polyak%2C+Yossi+Adi%2C+Jade+Copet%2C+Eugene+Kharitonov%2C+Kushal+Lakhotia%2C+Wei%2DNing+Hsu%2C+Abdelrahman+Mohamed%2C+and+Emmanuel+Dupoux%2E+2021%2E+Speech+resynthesis+from+discrete+disentangled+self%2Dsupervised+repr)
28. Adam Polyak, Lior Wolf, Yossi Adi, and Yaniv Taigman. 2020. Unsupervised cross-domain singing voice conversion. In Proc. INTERSPEECH, pages 801–805. [find](https://scholar.google.com/scholar?q=Adam+Polyak%2C+Lior+Wolf%2C+Yossi+Adi%2C+and+Yaniv+Taigman%2E+2020%2E+Unsupervised+cross%2Ddomain+singing+voice+conversion%2E+In+Proc%2E+INTERSPEECH%2C+pages+801%E2%80%93805%2E)
29. Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, et al. 2011. The kaldi speech recognition toolkit. In IEEE 2011 workshop on automatic speech recognition and understanding, CONF. IEEE Signal Process… [find](https://scholar.google.com/scholar?q=Daniel+Povey%2C+Arnab+Ghoshal%2C+Gilles+Boulianne%2C+Lukas+Burget%2C+Ondrej+Glembek%2C+Nagendra+Goel%2C+Mirko+Hannemann%2C+Petr+Motlicek%2C+Yanmin+Qian%2C+Petr+Schwarz%2C+et+al%2E+2011%2E+The+kaldi+speech+recognition+toolkit)
30. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E)
31. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
32. Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. 2020. Fastspeech 2: Fast and high-quality end-to-end text to speech. arXiv preprint arXiv:2006.04558. [find](https://scholar.google.com/scholar?q=Yi+Ren%2C+Chenxu+Hu%2C+Xu+Tan%2C+Tao+Qin%2C+Sheng+Zhao%2C+Zhou+Zhao%2C+and+Tie%2DYan+Liu%2E+2020%2E+Fastspeech+2%3A+Fast+and+high%2Dquality+end%2Dto%2Dend+text+to+speech%2E+arXiv+preprint+arXiv%3A2006%2E04558%2E)
33. Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. 2019. wav2vec: Unsupervised pre-training for speech recognition. arXiv preprint arXiv:1904.05862. — [arXiv:1904.05862](https://arxiv.org/abs/1904.05862) [find](https://scholar.google.com/scholar?q=Steffen+Schneider%2C+Alexei+Baevski%2C+Ronan+Collobert%2C+and+Michael+Auli%2E+2019%2E+wav2vec%3A+Unsupervised+pre%2Dtraining+for+speech+recognition%2E+arXiv+preprint+arXiv%3A1904%2E05862%2E+%E2%80%94+%5BarXiv%3A1904%2E05862%5D%28https%3A%2F%2Farx)
34. Elizabeth Shriberg and Andreas Stolcke. 2004. Prosody modeling for automatic speech recognition and understanding. In Mathematical Foundations of Speech and Language Processing, pages 105–114. Springer. [find](https://scholar.google.com/scholar?q=Elizabeth+Shriberg+and+Andreas+Stolcke%2E+2004%2E+Prosody+modeling+for+automatic+speech+recognition+and+understanding%2E+In+Mathematical+Foundations+of+Speech+and+Language+Processing%2C+pages+105%E2%80%93114%2E+Springe)
35. Elizabeth Shriberg, Andreas Stolcke, Dilek Hakkani-Tür, and Gökhan Tür. 2000. Prosody-based automatic segmentation of speech into sentences and topics. Speech communication, 32(1-2):127–154. [find](https://scholar.google.com/scholar?q=Elizabeth+Shriberg%2C+Andreas+Stolcke%2C+Dilek+Hakkani%2DT%C3%BCr%2C+and+G%C3%B6khan+T%C3%BCr%2E+2000%2E+Prosody%2Dbased+automatic+segmentation+of+speech+into+sentences+and+topics%2E+Speech+communication%2C+32%281%2D2%29%3A127%E2%80%93154%2E)
36. Elizabeth Shriberg, Andreas Stolcke, Daniel Jurafsky, Noah Coccaro, Marie Meteer, Rebecca Bates, Paul Taylor, Klaus Ries, Rachel Martin, and Carol Van Ess-Dykema. 1998. Can prosody aid the automatic classification of dialog acts in conversational speech? Language and speech, 41(3-4):443–492. [find](https://scholar.google.com/scholar?q=Elizabeth+Shriberg%2C+Andreas+Stolcke%2C+Daniel+Jurafsky%2C+Noah+Coccaro%2C+Marie+Meteer%2C+Rebecca+Bates%2C+Paul+Taylor%2C+Klaus+Ries%2C+Rachel+Martin%2C+and+Carol+Van+Ess%2DDykema%2E+1998%2E+Can+prosody+aid+the+automatic+c)
37. Yi Su and Frederick Jelinek. 2008. Exploiting prosodic breaks in language modeling with random forests. In Speech Prosody. Citeseer. [find](https://scholar.google.com/scholar?q=Yi+Su+and+Frederick+Jelinek%2E+2008%2E+Exploiting+prosodic+breaks+in+language+modeling+with+random+forests%2E+In+Speech+Prosody%2E+Citeseer%2E)
38. SynSIG. Blizzard Challenge 2013. [https://www.synsig.org/index.php/](https://www.synsig.org/index.php/) Blizzard_Challenge_2013. Accessed: 2021-08-23. [↗](https://www.synsig.org/index.php/)
39. Andros Tjandra, Ruoming Pang, Yu Zhang, and Shigeki Karita. 2020. Unsupervised learning of disentangled speech content and style representation. arXiv preprint arXiv:2010.12973. — [arXiv:2010.12973](https://arxiv.org/abs/2010.12973) [find](https://scholar.google.com/scholar?q=Andros+Tjandra%2C+Ruoming+Pang%2C+Yu+Zhang%2C+and+Shigeki+Karita%2E+2020%2E+Unsupervised+learning+of+disentangled+speech+content+and+style+representation%2E+arXiv+preprint+arXiv%3A2010%2E12973%2E+%E2%80%94+%5BarXiv%3A2010%2E12973%5D%28h)
40. Trang Tran, Shubham Toshniwal, Mohit Bansal, Kevin Gimpel, Karen Livescu, and Mari Ostendorf. 2017. Parsing speech: a neural approach to integrating lexical and acoustic-prosodic information. arXiv preprint arXiv:1704.07287. [find](https://scholar.google.com/scholar?q=Trang+Tran%2C+Shubham+Toshniwal%2C+Mohit+Bansal%2C+Kevin+Gimpel%2C+Karen+Livescu%2C+and+Mari+Ostendorf%2E+2017%2E+Parsing+speech%3A+a+neural+approach+to+integrating+lexical+and+acoustic%2Dprosodic+information%2E+arXiv+pr)
41. Aaron van den Oord, Oriol Vinyals, et al. 2017. Neural discrete representation learning. In NeurIPS. [find](https://scholar.google.com/scholar?q=Aaron+van+den+Oord%2C+Oriol+Vinyals%2C+et+al%2E+2017%2E+Neural+discrete+representation+learning%2E+In+NeurIPS%2E)
42. Christophe Veaux, Junichi Yamagishi, Kirsten Mac-Donald, et al. 2016. Superseded-cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit. [find](https://scholar.google.com/scholar?q=Christophe+Veaux%2C+Junichi+Yamagishi%2C+Kirsten+Mac%2DDonald%2C+et+al%2E+2016%2E+Superseded%2Dcstr+vctk+corpus%3A+English+multi%2Dspeaker+corpus+for+cstr+voice+cloning+toolkit%2E)
43. Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. 2017. Tacotron: Towards end-to-end speech synthesis. arXiv preprint arXiv:1703.10135. [find](https://scholar.google.com/scholar?q=Yuxuan+Wang%2C+RJ+Skerry%2DRyan%2C+Daisy+Stanton%2C+Yonghui+Wu%2C+Ron+J+Weiss%2C+Navdeep+Jaitly%2C+Zongheng+Yang%2C+Ying+Xiao%2C+Zhifeng+Chen%2C+Samy+Bengio%2C+et+al%2E+2017%2E+Tacotron%3A+Towards+end%2Dto%2Dend+speech+synthesis%2E+ar)
44. F. Ribeiro, D. Florêncio, C. Zhang, and M. Seltzer. 2011. CROWDMOS: An approach for crowdsourcing mean opinion score studies. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2416–2419. [find](https://scholar.google.com/scholar?q=F%2E+Ribeiro%2C+D%2E+Flor%C3%AAncio%2C+C%2E+Zhang%2C+and+M%2E+Seltzer%2E+2011%2E+CROWDMOS%3A+An+approach+for+crowdsourcing+mean+opinion+score+studies%2E+In+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processin)
45. Nigel Ward and Wataru Tsukahara. 2000. Prosodic features which cue back-channel responses in english and japanese. Journal of pragmatics, 32(8):1177– 1207. [find](https://scholar.google.com/scholar?q=Nigel+Ward+and+Wataru+Tsukahara%2E+2000%2E+Prosodic+features+which+cue+back%2Dchannel+responses+in+english+and+japanese%2E+Journal+of+pragmatics%2C+32%288%29%3A1177%E2%80%93+1207%2E)
46. Morgane Rivière and Emmanuel Dupoux. 2020. Towards unsupervised learning of speech features in the wild. In IEEE Spoken Language Technology Workshop (SLT), pages 156–163. [find](https://scholar.google.com/scholar?q=Morgane+Rivi%C3%A8re+and+Emmanuel+Dupoux%2E+2020%2E+Towards+unsupervised+learning+of+speech+features+in+the+wild%2E+In+IEEE+Spoken+Language+Technology+Workshop+%28SLT%29%2C+pages+156%E2%80%93163%2E)
47. Nigel G Ward, Alejandro Vega, and Timo Baumann. 2012. Prosodic and temporal features for language modeling for dialog. Speech Communication, 54(2):161–174. [find](https://scholar.google.com/scholar?q=Nigel+G+Ward%2C+Alejandro+Vega%2C+and+Timo+Baumann%2E+2012%2E+Prosodic+and+temporal+features+for+language+modeling+for+dialog%2E+Speech+Communication%2C+54%282%29%3A161%E2%80%93174%2E)
48. Shu-Wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko-tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-Wen Li, Shinji Watanabe, Abdelrahman Mohamed, and Hung-yi Lee.… — [arXiv:2105.01051](https://arxiv.org/abs/2105.01051) [find](https://scholar.google.com/scholar?q=Shu%2DWen+Yang%2C+Po%2DHan+Chi%2C+Yung%2DSung+Chuang%2C+Cheng%2DI+Jeff+Lai%2C+Kushal+Lakhotia%2C+Yist+Y%2E+Lin%2C+Andy+T%2E+Liu%2C+Jiatong+Shi%2C+Xuankai+Chang%2C+Guan%2DTing+Lin%2C+Tzu%2DHsien+Huang%2C+Wei%2DCheng+Tseng%2C+Ko%2Dtik+Lee%2C+Da%2DRon)

*Updated 24 October 2026*