---
title: "A neural prosody encoder for end-ro-end dialogue act classification"
description: "Dialogue act classification (DAC) decides what a speaker is doing with an utterance: stating, asking… 34 references, cited by 1 paper here."
authors: ["Kai Wei", "Dillon Knox", "Martin Radfar", "Thanh Tran", "Markus Muller", "Grant P. Strimel", "Nathan Susanj", "Athanasios Mouchtaris", "Maurizio Omologo"]
year: 2022
venue: "arXiv:2205.05590"
source: "https://arxiv.org/abs/2205.05590"
doi: "10.48550/arxiv.2205.05590"
arxiv: "2205.05590"
licence: "CC BY-NC-SA 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-nc-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wei-2022-neural-prosody-encoder-end-ro/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wei-2022-neural-prosody-encoder-end-ro/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/6/) / A neural prosody encoder for end-ro-end dialogue act classification

# A neural prosody encoder for end-ro-end dialogue act classification

**Authors:** Kai Wei, Dillon Knox, Martin Radfar, Thanh Tran, Markus Muller, Grant P. Strimel, Nathan Susanj, Athanasios Mouchtaris and Maurizio Omologo · **Year:** 2022 · **Venue:** arXiv:2205.05590 · **Licence:** [CC BY-NC-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-nc-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2205.05590) · [PDF](https://arxiv.org/pdf/2205.05590) · [DOI 10.48550/arxiv.2205.05590](https://doi.org/10.48550/arxiv.2205.05590) · [arXiv 2205.05590](https://arxiv.org/abs/2205.05590) · [OpenAlex](https://openalex.org/W4280598105)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 141,840 papers whose bibliographies we read

## Summary
Dialogue act classification (DAC) decides what a speaker is doing with an utterance: stating, asking, agreeing, or simply signalling attention. Most systems work from transcripts, but transcripts may be missing, are costly to produce, and carry ASR errors. End-to-end (E2E) DAC instead predicts the act straight from audio. The authors argue that intonation, stress and loudness carry intent at two scales: within words or syllables, and across the whole utterance. Their model reflects both scales:
- Per-frame energy and pitch features are merged with log mel filter-bank (LFBE) features through a learned local gate.
- A Bi-LSTM encodes the merged frames.
- A multi-width 2-D CNN summarises prosody over the entire utterance.
- A global gate, built on compositional de-attention, can add prosodic evidence to the acoustic states, subtract it, or suppress it.

The system was tested on DSTC2, DSTC3 and Switchboard. It beat spectral-only E2E baselines and a strong earlier neural prosody encoder, gaining 1.07% absolute accuracy on average across the three datasets. The work shows that non-lexical cues can be fused selectively rather than just appended.

## Key points
- Inputs are computed with Kaldi:
  - LFBE features use 25 ms windows, a 10 ms frame shift and 8 kHz audio.
  - Energy has 3 dimensions: total log energy, and the low-band and high-band halves of the 40 mel bands.
  - Pitch has 3 dimensions: warped NCCF, POV-weighted mean-normalised log-pitch, and the log-pitch derivative.
- The local gate scores each frame from three inputs: the prosody vector, the LFBE vector and their element-wise interaction. That score scales the prosody vector before it is concatenated with the LFBE vector.
- The global prosody encoder is a 2-D CNN with stride 1 and kernel lengths 5, 10, 25 and 50. Its outputs are max-pooled and flattened.
- The global gate learns two matrices between acoustic and prosodic embeddings:
  - a similarity matrix, mean-centred and passed through tanh, which allows adding or subtracting prosody;
  - a dissimilarity matrix based on L1 distance, mean-centred and passed through a sigmoid, which acts as an eraser.
- Evaluation covers DSTC2, DSTC3 and Switchboard. Baselines are spectral-only E2E models and the prosody encoder of Trang Tran's thesis. Ablations isolate the gating and each prosodic feature type.

## Interesting ideas
- A side-feature gate that sees both streams plus their product lets the model choose between plain concatenation and ignoring the extra signal entirely, so a weak cue never drowns the main input.
- Replacing softmax attention with a signed (tanh) affinity plus a separate sigmoid "eraser" makes fusion able to subtract evidence, not only reweight it. Mean-centring keeps both ranges meaningful.
- Modelling cues at two granularities (frame-level and utterance-level with several window widths) captures phenomena that live at different scales inside one unit.
- The motivating example: the same word "yes" can be agreement or a backchannel, and only a non-lexical cue tells them apart. This shows that the act label must not be read off the lexical form alone.
- Ablating the gate and each feature family separately shows which component earns the gain instead of crediting the whole architecture.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tree): gparse could gate auxiliary inputs such as lexicon morphology hints or witness-parser votes per token. The gate values could then be logged to explain when the hint mattered.
- **I2** (learn): when silver labels from several witness parsers are combined, a signed-agreement-plus-veto scheme could let a witness argue against an arc rather than only for it.
- **I3** (frames): speech-act detection could pair token-level text cues (a sentence-final "?", emphasis, "yes" versus "yes?") with sentence-level pattern features.
- **I4** (frames): the prag speech-act classifier should keep agreement and backchannel as distinct readings of "yes". It should record which cue chose between them, so the decision stays explainable.
- **I5** (learn): every new fusion component could ship with a "gate removed / feature removed" control run as a standard negative control in evaluation.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Southern California](https://md.research.elamur.ai/free/en/universities/university-of-southern-california/) — University, Los Angeles, United States · 104 works · 33,876 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🔊 Signal Processing](https://md.research.elamur.ai/free/en/shelves/computer-science/signal-processing/) · 213 works

🧭 Topics (3): [🗨️ Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) · 401 works · [🎙️ Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) · 233 works · [💬 Speech act theory](https://md.research.elamur.ai/free/en/topics/speech-act-theory/) · 160 works

## Cite

Cite as: Kai Wei, Dillon Knox, Martin Radfar, Thanh Tran, Markus Muller, Grant P. Strimel, Nathan Susanj, Athanasios Mouchtaris and Maurizio Omologo. 2022. A neural prosody encoder for end-ro-end dialogue act classification. arXiv:2205.05590.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{wei2022neural,
  title = {{A neural prosody encoder for end-ro-end dialogue act classification}},
  author = {Kai Wei, Dillon Knox, Martin Radfar, Thanh Tran, Markus Muller, Grant P. Strimel, Nathan Susanj, Athanasios Mouchtaris and Maurizio Omologo},
  year = {2022},
  doi = {10.48550/arxiv.2205.05590},
  eprint = {2205.05590},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2205.05590}
}
```

- [← previous A Multi-Dimensional, Cross-Domain and Hierarchy-Aware Neural Architecture for ISO-Standard Dialogue Act Tagging](https://md.research.elamur.ai/free/en/papers/mezza-2022-multi-dimensional-cross-domain-hierarchy/)
- [next → A Romanization System and WebMAUS Aligner for Arabic Varieties](https://md.research.elamur.ai/free/en/papers/tamimi-2022-romanization-system-webmaus-aligner-arabic/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/6/)

## References

34 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. J. Austin, How to do things with words, 1962. [find](https://scholar.google.com/scholar?q=J%2E+Austin%2C+How+to+do+things+with+words%2C+1962%2E)
2. Q. Tran, I. Zukerman, and G. Haffari, “[A hierarchical neural model for learning sequences of dialogue acts](https://md.research.elamur.ai/free/en/papers/tran-2017-hierarchical-neural-model-learning-sequences/),” in ACL, 2017, vol. 1, pp. 428–437.
3. Y. Ji, G. Haffari, and J. Eisenstein, “A latent variable recurrent neural network for discourse relation language models,” NAACL, p. 332–342, 2016. [↗](https://doi.org/10.48550/arxiv.1603.01913)
4. D. Ortega and N. Vu, “[Lexico-acoustic neural-based models for dialog act classification](https://md.research.elamur.ai/free/en/papers/ortega-2018-lexico-acoustic-neural-based-models/),” in ICASSP, 2018, pp. 6194–6198.
5. S. Shen and H. Lee, “Neural attention models for sequence classification: Analysis and application to key term extraction and dialogue act detection.,” in Interspeech 2016, 2016, pp. 2716–2720. [find](https://scholar.google.com/scholar?q=S%2E+Shen+and+H%2E+Lee%2C+%E2%80%9CNeural+attention+models+for+sequence+classification%3A+Analysis+and+application+to+key+term+extraction+and+dialogue+act+detection%2E%2C%E2%80%9D+in+Interspeech+2016%2C+2016%2C+pp%2E+2716%E2%80%932720%2E)
6. V. Dang, T. Zhao, S. Ueno, H. Inaguma, and T. Kawahara, “[End-to-end speech-to-dialog-act recognition](https://md.research.elamur.ai/free/en/papers/dang-2020-end-end-speech-dialog-act/),” in Interspeech 2020, 2020, pp. 3910–3914.
7. X. He, Q. Tran, W. Havard, L. Besacier, I. Zukerman, and G. Haffari, “[Exploring textual and speech information in dialogue act classification with speaker domain adaptation](https://md.research.elamur.ai/free/en/papers/he-2018-exploring-textual-speech-information-dialogue/),” Proceedings of the Australasian Language Technology Association Workshop 2018, pp. 61–65, 2018. [↗](https://doi.org/10.48550/arxiv.1810.07455)
8. F. Julia, K. Iftekharuddin, and A. Islam, “Dialog act classification using acoustic and discourse information of maptask data,” International Journal of Computational Intelligence and Applications, pp. 289–311, 2010. [↗](https://doi.org/10.1142/s1469026810002926)
9. Trang Tran, Neural Models for Integrating Prosody in Spoken Language Understanding, Ph.D. thesis, University of Washington, 2020. [find](https://scholar.google.com/scholar?q=Trang+Tran%2C+Neural+Models+for+Integrating+Prosody+in+Spoken+Language+Understanding%2C+Ph%2ED%2E+thesis%2C+University+of+Washington%2C+2020%2E)
10. S Wallbridge, P. Bell, and C. Lai, “It’s not what you said, it’s how you said it: discriminative perception of speech as a multichannel communication system,” in Interspeech 2021: The 22nd Annual Conference of the International Speech Communication Association, 2021. [find](https://scholar.google.com/scholar?q=S+Wallbridge%2C+P%2E+Bell%2C+and+C%2E+Lai%2C+%E2%80%9CIt%E2%80%99s+not+what+you+said%2C+it%E2%80%99s+how+you+said+it%3A+discriminative+perception+of+speech+as+a+multichannel+communication+system%2C%E2%80%9D+in+Interspeech+2021%3A+The+22nd+Annual+Conf)
11. D. Dahan, “Prosody and language comprehension,” Wiley Interdisciplinary Reviews: Cognitive Science, 2015. [↗](https://doi.org/10.1002/wcs.1355)
12. N. Ward, Prosodic Patterns in English Conversation, 2019. [find](https://scholar.google.com/scholar?q=N%2E+Ward%2C+Prosodic+Patterns+in+English+Conversation%2C+2019%2E)
13. K. Honda, “Physiological factors causing tonal characteristics of speech: from global to local prosody,” in Speech Prosody 2004, International Conference, 2004. [find](https://scholar.google.com/scholar?q=K%2E+Honda%2C+%E2%80%9CPhysiological+factors+causing+tonal+characteristics+of+speech%3A+from+global+to+local+prosody%2C%E2%80%9D+in+Speech+Prosody+2004%2C+International+Conference%2C+2004%2E)
14. E. Shriberg, A. Stolcke, D.Jurafsky, N. Coccaro, M. Meteer, R. Bates, P. Taylor, K. Ries, R. Martin, and C. Ess-Dykema, “Can prosody aid the automatic classification of dialog acts in conversational speech,” Language and Speech, vol. 41, pp. 443–492, 1998. [↗](https://doi.org/10.1177/002383099804100410)
15. M. Zimmermann, “Joint segmentation and classification of dialog acts using conditional random fields.,” in INTERSPEECH, 2009, pp. 864–867. [find](https://scholar.google.com/scholar?q=M%2E+Zimmermann%2C+%E2%80%9CJoint+segmentation+and+classification+of+dialog+acts+using+conditional+random+fields%2E%2C%E2%80%9D+in+INTERSPEECH%2C+2009%2C+pp%2E+864%E2%80%93867%2E)
16. S. Quarteroni, A. Ivanov, and G. Riccardi, “Simultaneous dialog act segmentation and classification from human-human spoken conversations,” in 2011 ICASSP, 2011, pp. 5596–5599. [find](https://scholar.google.com/scholar?q=S%2E+Quarteroni%2C+A%2E+Ivanov%2C+and+G%2E+Riccardi%2C+%E2%80%9CSimultaneous+dialog+act+segmentation+and+classification+from+human%2Dhuman+spoken+conversations%2C%E2%80%9D+in+2011+ICASSP%2C+2011%2C+pp%2E+5596%E2%80%935599%2E)
17. J. Ang, Yang Liu, and E. Shriberg, “Automatic dialog act segmentation and classification in multiparty meetings,” in IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005., 2005, vol. 1, pp. 1061–1064. [find](https://scholar.google.com/scholar?q=J%2E+Ang%2C+Yang+Liu%2C+and+E%2E+Shriberg%2C+%E2%80%9CAutomatic+dialog+act+segmentation+and+classification+in+multiparty+meetings%2C%E2%80%9D+in+IEEE+International+Conference+on+Acoustics%2C+Speech%2C+and+Signal+Processing%2C+2005%2E%2C+2)
18. A. Stolcke, N. Coccaro, R. Bates, P. Taylor, C. Ess-Dykema, K. Ries, E. Shriberg, D. Jurafsky, R. Martin, and M. Meteer, “[Dialogue act modeling for automatic tagging and recognition of conversational speech](https://md.research.elamur.ai/free/en/papers/stolcke-2000-dialogue-act-modeling-automatic-tagging/),” Computational Linguistics, vol. 26, no. 3, pp. 339–373, 2000. [↗](https://doi.org/10.1162/089120100561737)
19. H. Arsikere, A. Sen, A. Prathosh, and V. Tyagi, “Novel acoustic features for automatic dialog-act tagging,” in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2016, pp. 6105–6109. [find](https://scholar.google.com/scholar?q=H%2E+Arsikere%2C+A%2E+Sen%2C+A%2E+Prathosh%2C+and+V%2E+Tyagi%2C+%E2%80%9CNovel+acoustic+features+for+automatic+dialog%2Dact+tagging%2C%E2%80%9D+in+2016+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+%28ICASSP%29%2C+2)
20. A. Gravano, S. Benus, H. Chavez, J. Hirschberg, and L. Wilcox, “On the role of context and prosody in the interpretation of ’okay’,” in Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, 2007, pp. 800–807. [find](https://scholar.google.com/scholar?q=A%2E+Gravano%2C+S%2E+Benus%2C+H%2E+Chavez%2C+J%2E+Hirschberg%2C+and+L%2E+Wilcox%2C+%E2%80%9COn+the+role+of+context+and+prosody+in+the+interpretation+of+%E2%80%99okay%E2%80%99%2C%E2%80%9D+in+Proceedings+of+the+45th+Annual+Meeting+of+the+Association+of+Com)
21. S. Benus, A. Gravano, and J. Hirschberg, “The prosody of backchannels in american english,” ICPhS, 2007. [↗](https://doi.org/10.7916/d8ww7s1p)
22. H. Matthew, T. Blaise, and W. Jason, “The second dialog state tracking challenge,” in Proceedings of the 15th annual meeting of the special interest group on discourse and dialogue (SIG-DIAL), 2014, pp. 263–272. [find](https://scholar.google.com/scholar?q=H%2E+Matthew%2C+T%2E+Blaise%2C+and+W%2E+Jason%2C+%E2%80%9CThe+second+dialog+state+tracking+challenge%2C%E2%80%9D+in+Proceedings+of+the+15th+annual+meeting+of+the+special+interest+group+on+discourse+and+dialogue+%28SIG%2DDIAL%29%2C+2014%2C+p)
23. H. Matthew, T. Blaise, and W. Jason, “The third dialog state tracking challenge,” in 2014 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2014, pp. 324–329. [find](https://scholar.google.com/scholar?q=H%2E+Matthew%2C+T%2E+Blaise%2C+and+W%2E+Jason%2C+%E2%80%9CThe+third+dialog+state+tracking+challenge%2C%E2%80%9D+in+2014+IEEE+Spoken+Language+Technology+Workshop+%28SLT%29%2E+IEEE%2C+2014%2C+pp%2E+324%E2%80%93329%2E)
24. J. Godfrey, E. Holliman, and J. McDaniel, “Switchboard: telephone speech corpus for research and development,” in [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing, 1992, vol. 1, pp. 517–520. [find](https://scholar.google.com/scholar?q=J%2E+Godfrey%2C+E%2E+Holliman%2C+and+J%2E+McDaniel%2C+%E2%80%9CSwitchboard%3A+telephone+speech+corpus+for+research+and+development%2C%E2%80%9D+in+%5BProceedings%5D+ICASSP%2D92%3A+1992+IEEE+International+Conference+on+Acoustics%2C+Speech%2C+and+)
25. D. Povey, A. Ghoshal, G. Boulianne, N. Goel, M. Hannemann, Y. Qian, P. Schwarz, and G. Stemmer, “The kaldi speech recognition toolkit,” in In IEEE 2011 workshop, 2011. [find](https://scholar.google.com/scholar?q=D%2E+Povey%2C+A%2E+Ghoshal%2C+G%2E+Boulianne%2C+N%2E+Goel%2C+M%2E+Hannemann%2C+Y%2E+Qian%2C+P%2E+Schwarz%2C+and+G%2E+Stemmer%2C+%E2%80%9CThe+kaldi+speech+recognition+toolkit%2C%E2%80%9D+in+In+IEEE+2011+workshop%2C+2011%2E)
26. P. Ghahremani, B. BabaAli, D. Povey, K. Riedhammer, J. Trmal, and S. Khudanpur, “A pitch extraction algorithm tuned for automatic speech recognition,” in ICASSP, 2014. [find](https://scholar.google.com/scholar?q=P%2E+Ghahremani%2C+B%2E+BabaAli%2C+D%2E+Povey%2C+K%2E+Riedhammer%2C+J%2E+Trmal%2C+and+S%2E+Khudanpur%2C+%E2%80%9CA+pitch+extraction+algorithm+tuned+for+automatic+speech+recognition%2C%E2%80%9D+in+ICASSP%2C+2014%2E)
27. S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997. [find](https://scholar.google.com/scholar?q=S%2E+Hochreiter+and+J%2E+Schmidhuber%2C+%E2%80%9CLong+short%2Dterm+memory%2C%E2%80%9D+Neural+Computation%2C+vol%2E+9%2C+no%2E+8%2C+pp%2E+1735%E2%80%931780%2C+1997%2E)
28. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polosukhin, “[Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/),” in Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017, vol. 30, pp. 5998–6008.
29. D. Bahdanau, K. Cho, and B. Yoshua, “[Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/),” in ICLR, 2015. [↗](https://doi.org/10.48550/arxiv.1409.0473)
30. T. Yi, L. Tuan, Z. Aston, W. Shuohang, and H.Cheung, “Compositional de-attention networks,” in Advances in Neural Information Processing Systems, 2019. [find](https://scholar.google.com/scholar?q=T%2E+Yi%2C+L%2E+Tuan%2C+Z%2E+Aston%2C+W%2E+Shuohang%2C+and+H%2ECheung%2C+%E2%80%9CCompositional+de%2Dattention+networks%2C%E2%80%9D+in+Advances+in+Neural+Information+Processing+Systems%2C+2019%2E)
31. D. Jurafsky, E Shriberg, and D. Biasca, “Switchboard swbd-damsl shallow-discourse-function annotation coders manual, draft 13. technical report,” 1998, pp. 97–02. [find](https://scholar.google.com/scholar?q=D%2E+Jurafsky%2C+E+Shriberg%2C+and+D%2E+Biasca%2C+%E2%80%9CSwitchboard+swbd%2Ddamsl+shallow%2Ddiscourse%2Dfunction+annotation+coders+manual%2C+draft+13%2E+technical+report%2C%E2%80%9D+1998%2C+pp%2E+97%E2%80%9302%2E)
32. P. Adam et al., “[Pytorch: An imperative style, high-performance deep learning library](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/),” in Advances in Neural Information Processing Systems. 2019.
33. D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in ICLR, 2015. [↗](https://doi.org/10.48550/arxiv.1412.6980)
34. P. McKnight and J. Najab, “Mann-whitney u test,” Corsini Encyclopedia of Psychology, pp. 1–1, 2010. [↗](https://doi.org/10.1002/9780470479216.corpsy0524)

*Updated 24 October 2026*