---
title: "Context-aware Neural-based Dialog Act Classification on"
description: "Earlier work evaluated on clean human transcripts, so the authors test what happens with speech… 29 references, cited by 1 paper here."
authors: ["Daniel Ortega", "Chia-Yu Li", "Gisela Vallejo", "Pavel Denisov", "Ngoc Thang Vu"]
year: 2019
venue: "arXiv:1902.11060"
source: "https://arxiv.org/abs/1902.11060"
doi: "10.48550/arxiv.1902.11060"
arxiv: "1902.11060"
licence: "arXiv non-exclusive distribution licence (read only, no redistribution)"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ortega-2019-context-aware-neural-based-dialog/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ortega-2019-context-aware-neural-based-dialog/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/6/) / Context-aware Neural-based Dialog Act Classification on Automatically Generated Transcriptions

# Context-aware Neural-based Dialog Act Classification on Automatically Generated Transcriptions

**Authors:** [Daniel Ortega](https://md.research.elamur.ai/free/en/authors/daniel-ortega/), Chia-Yu Li, Gisela Vallejo, Pavel Denisov and [Ngoc Thang Vu](https://md.research.elamur.ai/free/en/authors/ngoc-thang-vu/) · **Year:** 2019 · **Venue:** [arXiv:1902.11060](https://md.research.elamur.ai/free/en/venues/ieee-international-conference-on-acoustics-speech-and-signal-processing/) ([IEEE International Conference on Acoustics Speech and Signal Processing 2019](https://md.research.elamur.ai/free/en/venues/ieee-international-conference-on-acoustics-speech-and-signal-processing/2019/)) · **Licence:** [arXiv non-exclusive distribution licence (read only, no redistribution)](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1902.11060) · [PDF](https://arxiv.org/pdf/1902.11060) · [DOI 10.48550/arxiv.1902.11060](https://doi.org/10.48550/arxiv.1902.11060) · [arXiv 1902.11060](https://arxiv.org/abs/1902.11060) · [OpenAlex](https://openalex.org/W2917937435)

**Citations:** 10 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/ortega-2019-context-aware-neural-based-dialog/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper tackles dialog act (DA) classification, which labels each utterance with the speaker's intention, such as a question, statement or backchannel. Earlier work evaluated on clean human transcripts, so the authors test what happens with speech recognizer output. Their model encodes the current utterance and up to five previous ones with a shared CNN, using word2vec embeddings and utterance-wise max pooling. A linear-chain CRF on top then labels the whole utterance sequence jointly. They evaluate on two benchmark corpora, MRDA (multiparty meetings, 5 tags) and SwDA (two-speaker phone calls, 42 tags). They compare manual transcripts with automatic ones from two ASR systems: a Kaldi hybrid TDNN/HMM and an ESPnet End-to-End model. On manual transcripts the CNN+CRF beats a context-free CNN and reaches state-of-the-art accuracy. On ASR output, E2E transcripts give better DA accuracy than TDNN/HMM ones, even though their word error rates are similar. The paper also shows that punctuation, which ASR output lacks, is a major source of train/test mismatch. This matters for any real voice-driven dialog system.

## Key points
- The context-free CNN baseline scores 80.2% on MRDA and 72.0% on SwDA. CNN+CRF with two previous utterances reaches 84.7% and 74.6%, averaged over 5 runs. Longer contexts (3–5) bring no further gain.
- Test WERs are close between the two recognizers:
  - SwDA: 18.02 for TDNN/HMM and 18.80 for E2E.
  - MRDA: 21.48 for TDNN/HMM and 18.80 for E2E.
- Training and testing on E2E transcripts gave the best results on automatic speech. The text reports 76.6% and 68.7%, though the extracted tables are ambiguous about which corpus each figure belongs to.
- A model trained on MRDA manual transcripts with punctuation falls from 84.7% to 59.2% on TDNN/HMM output and 66.1% on E2E output. Removing punctuation from the training data raises these to 69.3% and 76.2%, about 10 points better.
- Splits follow Lee & Dernoncourt (2016). Tuning was done only on validation data, changing one hyperparameter at a time.

## Interesting ideas
- Treat a dialog as a sequence and let a CRF learn which DA label tends to follow which, instead of classifying each utterance alone. Neighbouring acts constrain each other, as when a question is followed by an answer.
- Encode each context utterance separately with the same encoder and then combine the vectors. This keeps context distinguishable from the current utterance, which the cited analysis says is required for context to help.
- Word error rate alone does not predict how well a downstream task will do: two recognizers with similar WER produced different DA accuracy.
- Train on input that looks like test input, for example by stripping punctuation that the real source will never provide. A strong cue that disappears at test time hurts more than it helps.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (frames): The speech-act layer in prag could score act transitions across turns, keeping each transition weight inspectable, so an utterance's act is explained partly by the previous one.
- **I2** (frames): When prag decides a speech act or implicature, previous turns should be kept as separate, labelled evidence rather than merged into one span, so explanations can say which turn contributed.
- **I3** (learn): When comparing witness parsers or teachers, Mova should measure downstream frame and answer quality, not only token-level or LAS-style scores.
- **I4** (tokens): Add an evaluation variant of the tokenizer and tagger on de-punctuated, lowercased text, and train robustness on it, so Mova does not silently depend on punctuation it may not get from transcripts or chat.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Stuttgart](https://md.research.elamur.ai/free/en/universities/university-of-stuttgart/) — University, Stuttgart, Germany · 75 works · 46,113 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [🗨️ Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) · 401 works · [💬 Speech act theory](https://md.research.elamur.ai/free/en/topics/speech-act-theory/) · 160 works

## Cite

Cite as: Daniel Ortega, Chia-Yu Li, Gisela Vallejo, Pavel Denisov and Ngoc Thang Vu. 2019. Context-aware Neural-based Dialog Act Classification on Automatically Generated Transcriptions. arXiv:1902.11060.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{ortega2019context,
  title = {{Context-aware Neural-based Dialog Act Classification on Automatically Generated Transcriptions}},
  author = {Daniel Ortega, Chia-Yu Li, Gisela Vallejo, Pavel Denisov and Ngoc Thang Vu},
  year = {2019},
  doi = {10.48550/arxiv.1902.11060},
  eprint = {1902.11060},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1902.11060}
}
```

- [← previous Acquiring Annotated Data with Cross-lingual Explicitation for Implicit Discourse Relation Classification](https://md.research.elamur.ai/free/en/papers/shi-2019-acquiring-annotated-data-cross-lingual/)
- [next → Investigating Machine Learning Methods for Language and Dialect Identification of Cuneiform Texts](https://md.research.elamur.ai/free/en/papers/doostmohammadi-2019-investigating-machine-learning-methods-language/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/6/)

## References

29 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. J. L. Austin, How to Do Things with Words, Oxford university press, 1976. [find](https://scholar.google.com/scholar?q=J%2E+L%2E+Austin%2C+How+to+Do+Things+with+Words%2C+Oxford+university+press%2C+1976%2E)
2. K. Bach and R. M. Harnish, Linguistic Communication and Speech Acts, MIT Press, 1979. [find](https://scholar.google.com/scholar?q=K%2E+Bach+and+R%2E+M%2E+Harnish%2C+Linguistic+Communication+and+Speech+Acts%2C+MIT+Press%2C+1979%2E)
3. A. Stolcke et al., “[Dialogue act modeling for automatic tagging and recognition of conversational speech](https://md.research.elamur.ai/free/en/papers/stolcke-2000-dialogue-act-modeling-automatic-tagging/),” Computational Linguistics, vol. 26, pp. 339–373, 2000. [↗](https://doi.org/10.1162/089120100561737)
4. M. Zimmermann, “Joint segmentation and classification of dialog acts using conditional random fields,” in Proc. of ISCA, 2009. [find](https://scholar.google.com/scholar?q=M%2E+Zimmermann%2C+%E2%80%9CJoint+segmentation+and+classification+of+dialog+acts+using+conditional+random+fields%2C%E2%80%9D+in+Proc%2E+of+ISCA%2C+2009%2E)
5. J. Y. Lee and F. Dernoncourt, “[Sequential short-text classification with recurrent and convolutional neural networks](https://md.research.elamur.ai/free/en/papers/lee-2016-sequential-short-text-classification-recurrent/),” in Proc. of NAACL, 2016.
6. N. Kalchbrenner and P. Blunsom, “[Recurrent convolutional neural networks for discourse compositionality](https://md.research.elamur.ai/free/en/papers/kalchbrenner-2013-recurrent-convolutional-neural-networks-discourse/),” in Proc. of CVSC, 2013.
7. D. Ortega and N. T. Vu, “[Lexico-acoustic neural-based models for dialog act classification](https://md.research.elamur.ai/free/en/papers/ortega-2018-lexico-acoustic-neural-based-models/),” in Proc. of ICASSP, 2018.
8. D. Ortega and N. T. Vu, “[Neural-based context representation learning for dialog act classification](https://md.research.elamur.ai/free/en/papers/ortega-2017-neural-based-context-representation-learning/),” in Proc. of SIGDIAL, 2017.
9. Eugénio Ribeiro, Ricardo Ribeiro, and David Martins de Mato, “[The influence of context on dialogue act recognition](https://md.research.elamur.ai/free/en/papers/ribeiro-2015-influence-context-dialogue-act-recognition/),” CoRR, 2015. [↗](https://doi.org/10.48550/arxiv.1506.00839)
10. G. Lample et al., “[Neural architectures for named entity recognition](https://md.research.elamur.ai/free/en/papers/lample-2016-neural-architectures-named-entity-recognition/),” in Proc. of NAACL, 2016. [↗](https://doi.org/10.48550/arxiv.1603.01360)
11. D. Andor et al., “[Globally normalized transition-based neural networks](https://md.research.elamur.ai/free/en/papers/andor-2016-globally-normalized-transition-based-neural/),” in Proc. of ACL, 2016.
12. H. Adel et al., “Global normalization of convolutional neural networks for joint entity and relation classification,” in Proc. of EMNLP, 2017. [find](https://scholar.google.com/scholar?q=H%2E+Adel+et+al%2E%2C+%E2%80%9CGlobal+normalization+of+convolutional+neural+networks+for+joint+entity+and+relation+classification%2C%E2%80%9D+in+Proc%2E+of+EMNLP%2C+2017%2E)
13. W. Xiong et al., “The microsoft 2017 conversational speech recognition system,” in Proc. of ICASSP, 2018. [find](https://scholar.google.com/scholar?q=W%2E+Xiong+et+al%2E%2C+%E2%80%9CThe+microsoft+2017+conversational+speech+recognition+system%2C%E2%80%9D+in+Proc%2E+of+ICASSP%2C+2018%2E)
14. A. Graves et al., “Speech recognition with deep recurrent neural networks,” in Proc. of ICASSP, 2013. [↗](https://doi.org/10.48550/arxiv.1303.5778)
15. D. Povey et al., “Purely sequence-trained neural networks for ASR based on lattice-free MMI,” in Proc. of INTERSPEECH, 2016. [find](https://scholar.google.com/scholar?q=D%2E+Povey+et+al%2E%2C+%E2%80%9CPurely+sequence%2Dtrained+neural+networks+for+ASR+based+on+lattice%2Dfree+MMI%2C%E2%80%9D+in+Proc%2E+of+INTERSPEECH%2C+2016%2E)
16. S. Watanabe et al., “ESPnet: end-to-end speech processing toolkit,” in Proc. of INTERSPEECH, 2018. [find](https://scholar.google.com/scholar?q=S%2E+Watanabe+et+al%2E%2C+%E2%80%9CESPnet%3A+end%2Dto%2Dend+speech+processing+toolkit%2C%E2%80%9D+in+Proc%2E+of+INTERSPEECH%2C+2018%2E)
17. Y. Miao et al., “EESEN: end-to-end speech recognition using deep RNN models and wfst-based decoding,” in Proc. of IEEE ASRU, 2015. [find](https://scholar.google.com/scholar?q=Y%2E+Miao+et+al%2E%2C+%E2%80%9CEESEN%3A+end%2Dto%2Dend+speech+recognition+using+deep+RNN+models+and+wfst%2Dbased+decoding%2C%E2%80%9D+in+Proc%2E+of+IEEE+ASRU%2C+2015%2E)
18. D. Bahdanau et al., “End-to-end attention-based large vocabulary speech recognition,” in Proc. of ICASSP, 2016. [find](https://scholar.google.com/scholar?q=D%2E+Bahdanau+et+al%2E%2C+%E2%80%9CEnd%2Dto%2Dend+attention%2Dbased+large+vocabulary+speech+recognition%2C%E2%80%9D+in+Proc%2E+of+ICASSP%2C+2016%2E)
19. A. Janin et al., “The ICSI meeting corpus,” in Proc. of ICASSP, 2003. [find](https://scholar.google.com/scholar?q=A%2E+Janin+et+al%2E%2C+%E2%80%9CThe+ICSI+meeting+corpus%2C%E2%80%9D+in+Proc%2E+of+ICASSP%2C+2003%2E)
20. E. Shriberg et al., “[The ICSI meeting recorder dialog act (MRDA) corpus](https://md.research.elamur.ai/free/en/papers/shriberg-2004-icsi-meeting-recorder-dialog-act/),” in Proc. of SIGdial Workshop on Discourse and Dialogue at HLT-NAACL, 2004. [↗](https://doi.org/10.21236/ada460980)
21. R. Dhillon et al., “Meeting recorder project: Dialog act labeling guide,” Tech. Rep., ICSI Berkeley CA, 2004. [↗](https://doi.org/10.21236/ada607947)
22. J. Ang et al., “Automatic dialog act segmentation and classification in multiparty meetings,” in Proc. of ICASSP, 2005. [find](https://scholar.google.com/scholar?q=J%2E+Ang+et+al%2E%2C+%E2%80%9CAutomatic+dialog+act+segmentation+and+classification+in+multiparty+meetings%2C%E2%80%9D+in+Proc%2E+of+ICASSP%2C+2005%2E)
23. S. Calhoun et al, “The NXT-format switchboard corpus: A rich resource for investigating the syntax, semantics, pragmatics and prosody of dialogue,” Language Resources and Evaluation, vol. 44, pp. 387–419, 2010. [↗](https://doi.org/10.1007/s10579-010-9120-1)
24. B. T. Polyak et al., “Acceleration of stochastic approximation by averaging,” SICON, vol. 30, pp. 838–855, 1992. [↗](https://doi.org/10.1137/0330046)
25. J. Duchi et al., “Adaptive subgradient methods for online learning and stochastic optimization,” Journal of Machine Learning Research, vol. 12, pp. 2121–2159, 2011. [find](https://scholar.google.com/scholar?q=J%2E+Duchi+et+al%2E%2C+%E2%80%9CAdaptive+subgradient+methods+for+online+learning+and+stochastic+optimization%2C%E2%80%9D+Journal+of+Machine+Learning+Research%2C+vol%2E+12%2C+pp%2E+2121%E2%80%932159%2C+2011%2E)
26. T. Mikolov et al., “[Efficient estimation of word representations in vector space](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/),” in Proc. of ICLR, 2013. [↗](https://doi.org/10.48550/arxiv.1301.3781)
27. D. Povey et al., “The Kaldi speech recognition toolkit,” in Proc. of IEEE ASRU, 2011. [find](https://scholar.google.com/scholar?q=D%2E+Povey+et+al%2E%2C+%E2%80%9CThe+Kaldi+speech+recognition+toolkit%2C%E2%80%9D+in+Proc%2E+of+IEEE+ASRU%2C+2011%2E)
28. C. Cieri et al., “[The Fisher corpus: a resource for the next generations of speech-to-text](https://md.research.elamur.ai/free/en/papers/cieri-2004-fisher-corpus-resource-next-generations/),” in Proc. of LREC, 2004.
29. Z. Xiao et al., “Hybrid ctc-attention based end-to-end speech recognition using subword units,” in Proc. of ISCSLP, 2018. [find](https://scholar.google.com/scholar?q=Z%2E+Xiao+et+al%2E%2C+%E2%80%9CHybrid+ctc%2Dattention+based+end%2Dto%2Dend+speech+recognition+using+subword+units%2C%E2%80%9D+in+Proc%2E+of+ISCSLP%2C+2018%2E)

*Updated 24 October 2026*