---
title: "Mhm... Yeah? Okay! Evaluating the Naturalness and"
description: "The authors test whether existing synthesis methods can reproduce such responses so that they sound… 22 references, cited by 1 paper here."
authors: ["Carol Figueroa", "Marcel de Korte", "Magalie Ochs", "Gabriel Skantze"]
year: 2024
venue: "Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue"
source: "https://aclanthology.org/2024.sigdial-1.46/"
doi: "10.18653/v1/2024.sigdial-1.46"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/figueroa-2024-mhm-yeah-okay-evaluating-naturalness/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/figueroa-2024-mhm-yeah-okay-evaluating-naturalness/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/15/) / Mhm... Yeah? Okay! Evaluating the Naturalness and Communicative Function of Synthesized Feedback Responses in Spoken Dialogue

# Mhm... Yeah? Okay! Evaluating the Naturalness and Communicative Function of Synthesized Feedback Responses in Spoken Dialogue

**Authors:** Carol Figueroa, Marcel de Korte, [Magalie Ochs](https://md.research.elamur.ai/free/en/authors/magalie-ochs/) and [Gabriel Skantze](https://md.research.elamur.ai/free/en/authors/gabriel-skantze/) · **Year:** 2024 · **Venue:** [Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue](https://md.research.elamur.ai/free/en/venues/sigdial/) ([SIGDIAL 2024](https://md.research.elamur.ai/free/en/venues/sigdial/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.sigdial-1.46/) · [PDF](https://aclanthology.org/2024.sigdial-1.46.pdf) · [DOI 10.18653/v1/2024.sigdial-1.46](https://doi.org/10.18653/v1/2024.sigdial-1.46) · [OpenAlex](https://openalex.org/W4402715078)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/figueroa-2024-mhm-yeah-okay-evaluating-naturalness/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Conversational agents that listen like humans need short feedback responses such as "mhm", "yeah" or "wow". These tokens carry their meaning through both their words and their prosody. The authors test whether existing synthesis methods can reproduce such responses so that they sound natural, fit their context, and keep their intended communicative function. They do not try to predict prosody. Instead, they assume prosody is known and transplant it from listeners in U.S. English Switchboard telephone calls onto one female voice talent, using two routes:

- **Frame-level signal processing:** TD-PSOLA, WORLD and the Praat Vocal Toolkit.
- **Phone-level TTS:** a fine-tuned FastPitch 1.1 with a HiFiGAN vocoder.

In online listening tests, 86 raters judged original, TTS, signal-processed and pitch-flattened versions in dialogue context. They gave naturalness and appropriateness scores and chose a function label for each response. TTS sounded more natural than signal processing, and the two were rated about equally appropriate. However, TTS preserved the original function worse, even worse than the monotone control. The result suggests that future prosody predictors for feedback should work at the frame level.

## Key points
- Switchboard listener feedback (47 responses in total) was re-synthesized and placed back into 12 conversation clips. Only the listener channel was replaced.
- The TTS voice was pretrained on LJ-Speech and fine-tuned on a mix of target-voice recordings:
  - read CMU ARCTIC sentences (43 min);
  - Taskmaster-2 role-play (4 min);
  - 724 imitated Switchboard feedback tokens (11 min);
  - 34 min of free conversation containing 981 feedback instances.
- Mean naturalness scores (1–5) were Monotone 1.73, SignalP 2.61, TTS 3.0 and Original 4.37. All pairwise differences were significant in a CLMM analysis with Bonferroni-corrected post-hoc tests.
- Raters labeled each response with one of 10 feedback functions from Figueroa et al. (2022), such as Continue, Agree, Sympathy, Mild Surprise and Strong Surprise.
- Agreement between each rater's label for the original and for the synthesized version (Cohen's kappa) was 0.79 for SignalP, 0.74 for Monotone and 0.71 for TTS. TTS mainly lost attitudinal functions such as Sympathy and the two Surprise levels.

## Interesting ideas
- Naturalness, contextual fit and preserved function are measured separately, which shows that output can sound fine and fit the context while still meaning something else.
- A deliberately degraded condition (pitch flattened) serves as a floor, and the TTS scoring below it exposed that TTS carried almost no extra functional signal.
- In the within-participant design, each rater's reading of the original response serves as that rater's own gold label, which absorbs individual variation in how functions are perceived.
- The authors note that two labels can both fit the context yet be different functions, such as Mild versus Strong Surprise, so appropriateness does not equal correctness.
- A fine-grained, ten-way taxonomy of short feedback responses (non-understanding, continue, agree, disagree, yes, no, sympathy, disapproval, two levels of surprise) covers functions carried by very short responses.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (back): Evaluate the text → graph → text compressor on whether regenerated text keeps the same speech act and stance, not only on fluency.
- **I2** (learn): Add stripped-information baselines (e.g., lemmas only, no tree) as negative controls, so any component that fails to beat them is flagged.
- **I3** (learn): When LLM teachers or witness parsers label pragmatic phenomena, compare each witness against its own judgment on the original input, so systematic per-witness bias is separated from real loss.
- **I4** (frames): Let prag keep graded or neighboring speech-act labels and score near-misses differently from outright errors.
- **I5** (frames): Use it as a speech-act inventory for interjections and short replies in dialogue texts, for example in tales, inferring the function from context and punctuation.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [KTH Royal Institute of Technology](https://md.research.elamur.ai/free/en/universities/kth-royal-institute-of-technology/) — University, Stockholm, Sweden · 20 works · 1,723 citations
- [Aix-Marseille Université](https://md.research.elamur.ai/free/en/universities/aix-marseille-universite/) — University, Marseille, France · 25 works · 39,241 citations
- [Laboratoire d’Informatique et Systèmes](https://md.research.elamur.ai/free/en/labs/laboratoire-d-informatique-et-systemes/) — Research institute, Marseille, France · 10 works · 187 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 1,649 works · [💭 Experimental and Cognitive Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/experimental-and-cognitive-psychology/) · 930 works

🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works · [🗨️ Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) · 654 works

## Cite

Cite as: Carol Figueroa, Marcel de Korte, Magalie Ochs and Gabriel Skantze. 2024. Mhm... Yeah? Okay! Evaluating the Naturalness and Communicative Function of Synthesized Feedback Responses in Spoken Dialogue. Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{figueroa2024mhm,
  title = {{Mhm... Yeah? Okay! Evaluating the Naturalness and Communicative Function of Synthesized Feedback Responses in Spoken Dialogue}},
  author = {Carol Figueroa, Marcel {de Korte}, Magalie Ochs and Gabriel Skantze},
  booktitle = {Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue},
  year = {2024},
  doi = {10.18653/v1/2024.sigdial-1.46},
  url = {https://doi.org/10.18653/v1/2024.sigdial-1.46}
}
```

- [← previous MemeMind at ArAIEval Shared Task: Spotting Persuasive Spans in Arabic Text with Persuasion Techniques Identification](https://md.research.elamur.ai/free/en/papers/biswas-2024-mememind-araieval-shared-task-spotting/)
- [next → Morphological Synthesizer for Ge'ez Language: Addressing Morphological Complexity and Resource Limitations](https://md.research.elamur.ai/free/en/papers/gidey-2024-morphological-synthesizer-ge-ez-language/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/15/)

## References

22 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. 2002. Qualtrics. [https://www.qualtrics.com/.2014](https://www.qualtrics.com/.2014). Prolific. https://www.prolific.com/. [↗](https://www.qualtrics.com/.2014)
2. Tuan Dinh, Alexander Kain, and Kris Tjaden. 2019. Using a manifold vocoder for spectral voice and style conversion. In Proc. Interspeech 2019, pages 1388– 1392. [find](https://scholar.google.com/scholar?q=Tuan+Dinh%2C+Alexander+Kain%2C+and+Kris+Tjaden%2E+2019%2E+Using+a+manifold+vocoder+for+spectral+voice+and+style+conversion%2E+In+Proc%2E+Interspeech+2019%2C+pages+1388%E2%80%93+1392%2E)
3. Carol Figueroa, Adaeze Adigwe, Magalie Ochs, and Gabriel Skantze. 2022. Annotation of communicative functions of short feedback tokens in Switchboard. In Proc. LREC, pages 1849–1859. [find](https://scholar.google.com/scholar?q=Carol+Figueroa%2C+Adaeze+Adigwe%2C+Magalie+Ochs%2C+and+Gabriel+Skantze%2E+2022%2E+Annotation+of+communicative+functions+of+short+feedback+tokens+in+Switchboard%2E+In+Proc%2E+LREC%2C+pages+1849%E2%80%931859%2E)
4. John J Godfrey, Edward C Holliman, and Jane Mc-Daniel. 1992. SWITCHBOARD: Telephone speech corpus for research and development. In IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 517–520. IEEE Computer Society. [find](https://scholar.google.com/scholar?q=John+J+Godfrey%2C+Edward+C+Holliman%2C+and+Jane+Mc%2DDaniel%2E+1992%2E+SWITCHBOARD%3A+Telephone+speech+corpus+for+research+and+development%2E+In+IEEE+International+Conference+on+Acoustics%2C+Speech%2C+and+Signal+Proces)
5. Catharine Oertel, Joakim Gustafson, and Alan W Black. 2016. On data driven parametric backchannel synthesis for expressing attentiveness in conversational agents. In Proceedings of the Workshop on Multimodal Analyses Enabling Artificial Agents in Human-Machine Interaction, pages 43–47. [find](https://scholar.google.com/scholar?q=Catharine+Oertel%2C+Joakim+Gustafson%2C+and+Alan+W+Black%2E+2016%2E+On+data+driven+parametric+backchannel+synthesis+for+expressing+attentiveness+in+conversational+agents%2E+In+Proceedings+of+the+Workshop+on+Mul)
6. Sathish Pammi, Marc Schröder, Marcela Charfuelan, Oytun Türk, and Ingmar Steiner. 2010. Synthesis of listener vocalisations with imposed intonation contours. In Proc. 7th ISCA Workshop on Speech Synthesis (SSW 7), pages 240–245. [find](https://scholar.google.com/scholar?q=Sathish+Pammi%2C+Marc+Schr%C3%B6der%2C+Marcela+Charfuelan%2C+Oytun+T%C3%BCrk%2C+and+Ingmar+Steiner%2E+2010%2E+Synthesis+of+listener+vocalisations+with+imposed+intonation+contours%2E+In+Proc%2E+7th+ISCA+Workshop+on+Speech+Synth)
7. Charles Goodwin. 1986. Between and within: Alternative sequential treatments of continuers and assessments. Human Studies, 9(2-3):205–217. [↗](https://doi.org/10.1007/bf00148127)
8. R Core Team. 2023. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. [find](https://scholar.google.com/scholar?q=R+Core+Team%2E+2023%2E+R%3A+A+Language+and+Environment+for+Statistical+Computing%2E+R+Foundation+for+Statistical+Computing%2C+Vienna%2C+Austria%2E)
9. Keith Ito and Linda Johnson. 2017. The LJ speech dataset. [https://keithito.com/](https://keithito.com/) LJ-Speech-Dataset/. [↗](https://keithito.com/)
10. Emanuel A Schegloff. 1982. Discourse as an interactional achievement: Some uses of ‘uh huh’ and other things that come between sentences. Analyzing Discourse: Text and Talk, 71:71–93. [find](https://scholar.google.com/scholar?q=Emanuel+A+Schegloff%2E+1982%2E+Discourse+as+an+interactional+achievement%3A+Some+uses+of+%E2%80%98uh+huh%E2%80%99+and+other+things+that+come+between+sentences%2E+Analyzing+Discourse%3A+Text+and+Talk%2C+71%3A71%E2%80%9393%2E)
11. John Kominek and Alan W Black. 2004. The CMU Arctic speech databases. In Fifth ISCA Workshop on Speech Synthesis. [find](https://scholar.google.com/scholar?q=John+Kominek+and+Alan+W+Black%2E+2004%2E+The+CMU+Arctic+speech+databases%2E+In+Fifth+ISCA+Workshop+on+Speech+Synthesis%2E)
12. Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020. [HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis](https://md.research.elamur.ai/free/en/papers/kong-2020-hifi-gan-generative-adversarial-networks/). Advances in Neural Information Processing Systems, 33:17022– 17033.
13. Divesh Lala, Koji Inoue, Tatsuya Kawahara, and Kei Sawada. 2022. Backchannel generation model for a third party listener agent. In Proceedings of the 10th International Conference on Human-Agent Interaction, pages 114–122. [find](https://scholar.google.com/scholar?q=Divesh+Lala%2C+Koji+Inoue%2C+Tatsuya+Kawahara%2C+and+Kei+Sawada%2E+2022%2E+Backchannel+generation+model+for+a+third+party+listener+agent%2E+In+Proceedings+of+the+10th+International+Conference+on+Human%2DAgent+Inter)
14. Adrian Łańcucki. 2021. Fastpitch: Parallel text-to-speech with pitch prediction. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6588–6592. IEEE. [↗](https://doi.org/10.17023/03nj-k027)
15. Thorsten Stocksmeier, Stefan Kopp, and Dafydd Gibbon. 2007. Synthesis of prosodic attitudinal variants in German backchannel ja. In Proc. Interspeech 2007, pages 1290–1293. [find](https://scholar.google.com/scholar?q=Thorsten+Stocksmeier%2C+Stefan+Kopp%2C+and+Dafydd+Gibbon%2E+2007%2E+Synthesis+of+prosodic+attitudinal+variants+in+German+backchannel+ja%2E+In+Proc%2E+Interspeech+2007%2C+pages+1290%E2%80%931293%2E)
16. Jinhan Wang, Long Chen, Aparna Khare, Anirudh Raju, Pranav Dheram, Di He, Minhua Wu, Andreas Stolcke, and Venkatesh Ravichandran. 2024. Turn-taking and backchannel prediction with acoustic and large language model fusion. arXiv preprint arXiv:2401.14717. [↗](https://arxiv.org/abs/2401.14717)
17. Victor H Yngve. 1970. On getting a word in edgewise. In Papers from the 6th Regional Meeting Chicago Linguistic Society, pages 567–578. [find](https://scholar.google.com/scholar?q=Victor+H+Yngve%2E+1970%2E+On+getting+a+word+in+edgewise%2E+In+Papers+from+the+6th+Regional+Meeting+Chicago+Linguistic+Society%2C+pages+567%E2%80%93578%2E)
18. Åsa Wallers. 2006. Minor sounds of major importance-prosodic manipulation of synthetic backchannels in Swedish. [find](https://scholar.google.com/scholar?q=%C3%85sa+Wallers%2E+2006%2E+Minor+sounds+of+major+importance%2Dprosodic+manipulation+of+synthetic+backchannels+in+Swedish%2E)
19. Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. 2017. Montreal Forced Aligner: Trainable text-speech alignment using Kaldi. In Proc. Interspeech 2017, pages 498–502. [find](https://scholar.google.com/scholar?q=Michael+McAuliffe%2C+Michaela+Socolof%2C+Sarah+Mihuc%2C+Michael+Wagner%2C+and+Morgan+Sonderegger%2E+2017%2E+Montreal+Forced+Aligner%3A+Trainable+text%2Dspeech+alignment+using+Kaldi%2E+In+Proc%2E+Interspeech+2017%2C+pages+4)
20. Kentaro Mitsui, Yukiya Hono, and Kei Sawada. 2023. Towards human-like spoken dialogue generation between ai agents from written dialogue. Preprint, arXiv:2310.01088. [↗](https://arxiv.org/abs/2310.01088)
21. Masanori Morise, Fumiya Yokomori, and Kenji Ozawa. 2016. WORLD: a vocoder-based high-quality speech synthesis system for real-time applications. IE-ICE TRANSACTIONS on Information and Systems, 99(7):1877–1884. [↗](https://doi.org/10.1587/transinf.2015edp7457)
22. Anindita Nath and Nigel Ward. 2022. On the predictability of the prosody of dialog markers from the prosody of the local context. In Proc. Speech Prosody, pages 664–668. [find](https://scholar.google.com/scholar?q=Anindita+Nath+and+Nigel+Ward%2E+2022%2E+On+the+predictability+of+the+prosody+of+dialog+markers+from+the+prosody+of+the+local+context%2E+In+Proc%2E+Speech+Prosody%2C+pages+664%E2%80%93668%2E)

*Updated 2 November 2026*