---
title: "NonverbalTTS — paper by Borisov et al. (2025)"
description: "The authors build NonverbalTTS (NVTTS), an openly released English corpus of about 17 hours drawn… 31 references, cited by 2 papers here."
authors: ["Maksim Borisov", "Egor Spirin", "Daria Diatlova"]
year: 2025
venue: "arXiv:2507.13155"
source: "https://arxiv.org/abs/2507.13155"
doi: "10.48550/arxiv.2507.13155"
arxiv: "2507.13155"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/borisov-2025-nonverbaltts-public-english-corpus-text/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/borisov-2025-nonverbaltts-public-english-corpus-text/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/12/) / NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speech

# NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speech

**Authors:** Maksim Borisov, Egor Spirin and Daria Diatlova · **Year:** 2025 · **Venue:** arXiv:2507.13155 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2507.13155) · [PDF](https://arxiv.org/pdf/2507.13155) · [DOI 10.48550/arxiv.2507.13155](https://doi.org/10.48550/arxiv.2507.13155) · [arXiv 2507.13155](https://arxiv.org/abs/2507.13155) · [OpenAlex](https://openalex.org/W7084103406)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · 2 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Expressive speech synthesis needs training data in which nonverbal sounds such as laughs, coughs or sighs are marked in the transcript. Open corpora rarely provide this. They either cover only a couple of sound types, have poor audio (old telephone recordings), or leave the sounds unmarked in the text. The authors build NonverbalTTS (NVTTS), an openly released English corpus of about 17 hours drawn from VoxCeleb and Expresso. Each sample carries inline tags for 10 nonverbal vocalization types and one of 8 emotion labels.

The pipeline works in stages:
- An ASR model transcribes the audio where no transcript exists.
- A sound event detector finds nonverbal events, and forced alignment places their tags between words.
- A classifier assigns an emotion label.
- Several human annotators correct the result.
- A character-level merge-align-vote procedure fuses their versions into one.

Open TTS models fine-tuned on NVTTS reach parity with the closed CosyVoice2 system, judged by human ratings and automatic measures such as speaker similarity and fidelity of the vocalizations. The corpus and annotation guidelines are public, which makes controllable nonverbal TTS reproducible.

## Key points
- NVTTS is about 17 h in total: roughly 13.7 h from VoxCeleb (about 2,292 speakers kept) and 3.9 h from Expresso (4 speakers). It is released at huggingface.co/datasets/deepvk/NonverbalTTS.
- The 10 NV types are breath, laugh, sniff, cough, throat clearing, sigh, groan, sneeze, snore and grunt.
  - Breath (3,612) and laugh (1,018) are the most frequent.
  - Grunt is the rarest; only 7 samples were found at detection time.
- The 8 emotion classes cover happiness, sadness, anger, fear, surprise, disgust and a neutral state, plus a catch-all "other" category. All labels come from emotion2vec+ large, applied to both sources, so the scheme is uniform even though Expresso has its own style tags.
- The tools used were:
  - Canary for ASR on VoxCeleb;
  - BEATs for event detection, with a deliberately low threshold of 0.1;
  - Montreal Forced Aligner for tag placement;
  - Argilla for human review;
  - Pyalign for the dynamic-programming alignment used in fusion.
- Annotators could edit words, NV tags and emotions. They discarded clips that were non-English, had overlapping speakers, or contained nonverbal sounds made by someone other than the main speaker.

## Interesting ideas
- The detector runs at its most permissive threshold because human review comes afterwards, so recall is maximized automatically and precision is restored by validation.
- Multiple annotator hypotheses are fused in three steps: incrementally merge all versions into one superset, align each version back to that superset, then keep each element by majority vote.
- Nonverbal events from an audio detector are inserted into the transcript as bracketed tags at word-aligned positions, so sound events become first-class tokens in the text stream.
- Instead of inheriting each source corpus's own label scheme, one classifier relabels everything so all sources share a single consistent inventory.
- A sample is rejected when the vocalization belongs to someone other than the main speaker, which makes each sound attributable to one person.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): witness parsers could propose candidate arcs or frames with a generous cutoff, leaving filtering to agreement checks or LLM teacher tiers rather than losing rare constructions early.
- **I2** (learn): this fusion can combine silver outputs from several witnesses even when their tokenizations disagree, yielding a per-token vote count that also serves as an explainable confidence signal.
- **I3** (tokens): the tokenizer could recognize tags like [laugh] or [sigh] in dialogue transcripts as dedicated non-word tokens instead of splitting or tagging them as nouns, keeping downstream trees clean.
- **I4** (tree): when mixing treebanks with differing UD dialects, re-deriving labels through one shared converter avoids mixing incompatible conventions in training data.
- **I5** (frames): in dialogue, laughter or sighs could be attached to a specific speaker as an emotional signal, helping speech-act and irony detection decide whose attitude a turn expresses.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 1,552 works · [💭 Experimental and Cognitive Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/experimental-and-cognitive-psychology/) · 870 works · [🔊 Signal Processing](https://md.research.elamur.ai/free/en/shelves/computer-science/signal-processing/) · 452 works

🧭 Topics (3): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works · [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · 972 works · [🎙️ Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) · 480 works

## Cite

Cite as: Maksim Borisov, Egor Spirin and Daria Diatlova. 2025. NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speech. arXiv:2507.13155.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{borisov2025nonverbaltts,
  title = {{NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speech}},
  author = {Maksim Borisov, Egor Spirin and Daria Diatlova},
  year = {2025},
  doi = {10.48550/arxiv.2507.13155},
  eprint = {2507.13155},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2507.13155}
}
```

- [← previous MuSC: Improving Complex Instruction Following with Multi-granularity Self-Contrastive Training](https://md.research.elamur.ai/free/en/papers/huang-2025-musc-improving-complex-instruction-following/)
- [next → On Using Arabic Language Dialects in Recommendation Systems](https://md.research.elamur.ai/free/en/papers/alshabanah-2025-arabic-language-dialects-recommendation-systems/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/12/)

## References

31 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. J. A. Hall, S. A. Andrzejewski, and J. E. Yopchick, “Psychosocial correlates of interpersonal sensitivity: A meta-analysis,” Journal of nonverbal behavior, vol. 33, pp. 149–180, 2009. [↗](https://doi.org/10.1007/s10919-009-0070-5)
2. K. R. Scherer and U. Scherer, “Assessing the ability to recognize facial and vocal expressions of emotion: Construction and validation of the emotion recognition index,” Journal of Nonverbal Behavior, vol. 35, pp. 305–326, 2011. [↗](https://doi.org/10.1007/s10919-011-0115-4)
3. Z. Du, Q. Chen, S. Zhang, K. Hu, H. Lu, Y. Yang, H. Hu, S. Zheng, Y. Gu, Z. Ma et al., “Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens,” arXiv preprint arXiv:2407.05407, 2024. [↗](https://arxiv.org/abs/2407.05407)
4. H.-H. Guo, K. Liu, F.-Y. Shen, Y.-C. Wu, F.-L. Xie, K. Xie, and K.-T. Xu, “Fireredtts: A foundation text-to-speech framework 2 We calculate EMO-SIM on emotional subset of test (i.e without neutral and other) for industry-level generative speech applications,” arXiv preprint arXiv:2409.03283, 2024. [↗](https://arxiv.org/abs/2409.03283)
5. H. Wu, X. Wang, S. E. Eskimez, M. Thakker, D. Tompkins, C.-H. Tsai, C. Li, Z. Xiao, S. Zhao, J. Li et al., “Laugh now cry later: Controlling time-varying emotional states of flow-matching-based zero-shot text-to-speech,” in 2024 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2024, pp. 690– 697. [find](https://scholar.google.com/scholar?q=H%2E+Wu%2C+X%2E+Wang%2C+S%2E+E%2E+Eskimez%2C+M%2E+Thakker%2C+D%2E+Tompkins%2C+C%2E%2DH%2E+Tsai%2C+C%2E+Li%2C+Z%2E+Xiao%2C+S%2E+Zhao%2C+J%2E+Li+et+al%2E%2C+%E2%80%9CLaugh+now+cry+later%3A+Controlling+time%2Dvarying+emotional+states+of+flow%2Dmatching%2Dbased+zero%2Ds)
6. Z. Du, Y. Wang, Q. Chen, X. Shi, X. Lv, T. Zhao, Z. Gao, Y. Yang, C. Gao, H. Wang et al., “Cosyvoice 2: Scalable streaming speech synthesis with large language models,” arXiv preprint arXiv:2412.10117, 2024. [↗](https://arxiv.org/abs/2412.10117)
7. A. Nagrani, J. S. Chung, W. Xie, and A. Zisserman, “Voxceleb: Large-scale speaker verification in the wild,” Computer Speech & Language, vol. 60, p. 101027, 2020. [↗](https://doi.org/10.1016/j.csl.2019.101027)
8. T. A. Nguyen, W.-N. Hsu, A. d’Avirro, B. Shi, I. Gat, M. Fazel-Zarani, T. Remez, J. Copet, G. Synnaeve, M. Hassid et al., “Expresso: A benchmark and analysis of discrete expressive speech resynthesis,” arXiv preprint arXiv:2308.05725, 2023. [↗](https://arxiv.org/abs/2308.05725)
9. W. Kraaij, T. Hain, M. Lincoln, and W. Post, “The ami meeting corpus,” in Proc. International Conference on Methods and Techniques in Behavioral Research, 2005, pp. 1–4. [find](https://scholar.google.com/scholar?q=W%2E+Kraaij%2C+T%2E+Hain%2C+M%2E+Lincoln%2C+and+W%2E+Post%2C+%E2%80%9CThe+ami+meeting+corpus%2C%E2%80%9D+in+Proc%2E+International+Conference+on+Methods+and+Techniques+in+Behavioral+Research%2C+2005%2C+pp%2E+1%E2%80%934%2E)
10. J. J. Godfrey, E. C. Holliman, and J. McDaniel, “Switchboard: Telephone speech corpus for research and development,” in Acoustics, speech, and signal processing, ieee international conference on, vol. 1. IEEE Computer Society, 1992, pp. 517–520. [find](https://scholar.google.com/scholar?q=J%2E+J%2E+Godfrey%2C+E%2E+C%2E+Holliman%2C+and+J%2E+McDaniel%2C+%E2%80%9CSwitchboard%3A+Telephone+speech+corpus+for+research+and+development%2C%E2%80%9D+in+Acoustics%2C+speech%2C+and+signal+processing%2C+ieee+international+conference+on%2C+vol%2E)
11. C. Cieri, D. Miller, and K. Walker, “[The fisher corpus: A resource for the next generations of speech-to-text](https://md.research.elamur.ai/free/en/papers/cieri-2004-fisher-corpus-resource-next-generations/).” in LREC, vol. 4, 2004, pp. 69–71.
12. D. Xin, J. Jiang, S. Takamichi, Y. Saito, A. Aizawa, and H. Saruwatari, “Jvnv: A corpus of japanese emotional speech with verbal content and nonverbal expressions,” IEEE Access, vol. 12, pp. 19 752–19 764, 2024. [↗](https://doi.org/10.1109/access.2024.3360885)
13. Y. Arimoto, H. Kawatsu, S. Ohno, and H. Iida, “Naturalistic emotional speech collection paradigm with online game and its psychological and acoustical assessment,” Acoustical science and technology, vol. 33, no. 6, pp. 359–369, 2012. [↗](https://doi.org/10.1250/ast.33.359)
14. T. A. Nguyen, W.-N. Hsu, A. D’Avirro, B. Shi, I. Gat, M. Fazel-Zarani, T. Remez, J. Copet, G. Synnaeve, M. Hassid, F. Kreuk, Y. Adi, and E. Dupoux, “Expresso: A benchmark and analysis of discrete expressive speech resynthesis,” 2023. [Online]. Available: https://arxiv.org/abs/2308.05725 [↗](https://arxiv.org/abs/2308.05725)
15. H. He, Z. Shang, C. Wang, X. Li, Y. Gu, H. Hua, L. Liu, C. Yang, J. Li, P. Shi, Y. Wang, K. Chen, P. Zhang, and Z. Wu, “Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation,” 2024. [Online]. Available: https://arxiv.org/abs/2407.05361 [↗](https://arxiv.org/abs/2407.05361)
16. H. Zhang, X. Yu, and Y. Lin, “Nsv-tts: Non-speech vocalization modeling and transfer in emotional text-to-speech,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5. [find](https://scholar.google.com/scholar?q=H%2E+Zhang%2C+X%2E+Yu%2C+and+Y%2E+Lin%2C+%E2%80%9CNsv%2Dtts%3A+Non%2Dspeech+vocalization+modeling+and+transfer+in+emotional+text%2Dto%2Dspeech%2C%E2%80%9D+in+ICASSP+2023%2D2023+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Pro)
17. V. Srivastav, S. Majumdar, N. Koluguri, A. Moumen, S. Gandhi et al., “Open automatic speech recognition leaderboard,” [https://huggingface.co/spaces/hf-audio/open](https://huggingface.co/spaces/hf-audio/open) asr leaderboard, 2023. [↗](https://huggingface.co/spaces/hf-audio/open)
18. F. Schmid, T. Morocutti, F. Foscarin, J. Schlüter, P. Primus, and G. Widmer, “Effective pre-training of audio transformers for sound event detection,” arXiv preprint arXiv:2409.09546, 2024. [↗](https://arxiv.org/abs/2409.09546)
19. J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2017, pp. 776–780. [find](https://scholar.google.com/scholar?q=J%2E+F%2E+Gemmeke%2C+D%2E+P%2E+Ellis%2C+D%2E+Freedman%2C+A%2E+Jansen%2C+W%2E+Lawrence%2C+R%2E+C%2E+Moore%2C+M%2E+Plakal%2C+and+M%2E+Ritter%2C+%E2%80%9CAudio+set%3A+An+ontology+and+human%2Dlabeled+dataset+for+audio+events%2C%E2%80%9D+in+2017+IEEE+international+)
20. M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger, “Montreal forced aligner: Trainable text-speech alignment using kaldi.” in Interspeech, vol. 2017, 2017, pp. 498–502. [find](https://scholar.google.com/scholar?q=M%2E+McAuliffe%2C+M%2E+Socolof%2C+S%2E+Mihuc%2C+M%2E+Wagner%2C+and+M%2E+Sonderegger%2C+%E2%80%9CMontreal+forced+aligner%3A+Trainable+text%2Dspeech+alignment+using+kaldi%2E%E2%80%9D+in+Interspeech%2C+vol%2E+2017%2C+2017%2C+pp%2E+498%E2%80%93502%2E)
21. Z. Ma, Z. Zheng, J. Ye, J. Li, Z. Gao, S. Zhang, and X. Chen, “emotion2vec: Self-supervised pre-training for speech emotion representation,” Proc. ACL 2024 Findings, 2024. [find](https://scholar.google.com/scholar?q=Z%2E+Ma%2C+Z%2E+Zheng%2C+J%2E+Ye%2C+J%2E+Li%2C+Z%2E+Gao%2C+S%2E+Zhang%2C+and+X%2E+Chen%2C+%E2%80%9Cemotion2vec%3A+Self%2Dsupervised+pre%2Dtraining+for+speech+emotion+representation%2C%E2%80%9D+Proc%2E+ACL+2024+Findings%2C+2024%2E)
22. V.-S. Daniel and A. Francisco, “Argilla - Open-source framework for data-centric NLP,” Jan. 2023. [Online]. Available: [https://github.com/argilla-io/argilla](https://github.com/argilla-io/argilla) [↗](https://github.com/argilla-io/argilla)
23. A. Nagrani, J. S. Chung, and A. Zisserman, “Voxceleb: A large-scale speaker identification dataset,” in Interspeech 2017. ISCA, aug 2017. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2017-950 [↗](https://doi.org/10.21437/interspeech.2017-950)
24. J. S. Chung, A. Nagrani, and A. Zisserman, “Voxceleb2: Deep speaker recognition,” in Interspeech 2018. ISCA, sep 2018. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2018-1929 [↗](https://doi.org/10.21437/interspeech.2018-1929)
25. A. Diwan, Z. Zheng, D. Harwath, and E. Choi, “Scaling rich style-prompted text-to-speech datasets,” 2025. [Online]. Available: https://arxiv.org/abs/2503.04713 [↗](https://arxiv.org/abs/2503.04713)
26. N. Pallewela, D. Alahakoon, A. Adikari, J. E. Pierce, and M. L. Rose, “Optimizing speech emotion recognition with machine learning based advanced audio cue analysis,” Technologies, vol. 12, no. 7, p. 111, 2024. [↗](https://doi.org/10.3390/technologies12070111)
27. K. Diederik, “[Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/),” (No Title), 2014.
28. S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao et al., “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022. [↗](https://doi.org/10.1109/jstsp.2022.3188113)
29. C. K. Reddy, V. Gopal, and R. Cutler, “Dnsmos: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 6493–6497. [find](https://scholar.google.com/scholar?q=C%2E+K%2E+Reddy%2C+V%2E+Gopal%2C+and+R%2E+Cutler%2C+%E2%80%9CDnsmos%3A+A+non%2Dintrusive+perceptual+objective+speech+quality+metric+to+evaluate+noise+suppressors%2C%E2%80%9D+in+ICASSP+2021%2D2021+IEEE+International+Conference+on+Acoustics)
30. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “[Robust speech recognition via large-scale weak supervision](https://md.research.elamur.ai/free/en/papers/radford-2022-robust-speech-recognition-large-scale/),” 2022. [Online]. Available: https://arxiv.org/abs/2212.04356 [↗](https://arxiv.org/abs/2212.04356)
31. P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, İ. Polat, Y. Feng, E. W. Moore, J. VanderPlas, D. Laxalde, J. Perktold, R. Cimrman, I. Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, and SciPy 1.0 Contributors, “SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python,” Nature Methods, vol. 17, pp. 261–272, 2020. — [SciPy 1.0--Fundamental Algorithms for Scientific Computing in Python](https://md.research.elamur.ai/free/en/papers/virtanen-2019-scipy-1-0-fundamental-algorithms/) [↗](https://doi.org/10.1038/s41592-019-0686-2)

*Updated 1 November 2026*