---
title: "In-Context Evaluation of Unsupervised Dialogue Act Models for"
description: "Ezen-Can and Boyer propose judging such a model by what it enables: driving a… 30.4% accuracy. 18 references, cited by 1 paper here."
authors: ["Aysu Ezen-Can", "Kristy Boyer"]
year: 2013
venue: "Proceedings of the SIGDIAL 2013 Conference"
source: "https://aclanthology.org/W13-4049/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/can-2013-context-evaluation-unsupervised-dialogue-act/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/can-2013-context-evaluation-unsupervised-dialogue-act/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2013](https://md.research.elamur.ai/free/en/papers/2013/page/2/) / In-Context Evaluation of Unsupervised Dialogue Act Models for Tutorial Dialogue

# In-Context Evaluation of Unsupervised Dialogue Act Models for Tutorial Dialogue

**Authors:** [Aysu Ezen-Can](https://md.research.elamur.ai/free/en/authors/aysu-ezen-can/) and [Kristy Boyer](https://md.research.elamur.ai/free/en/authors/kristy-boyer/) · **Year:** 2013 · **Venue:** [Proceedings of the SIGDIAL 2013 Conference](https://md.research.elamur.ai/free/en/venues/sigdial-conference/) ([SIGDIAL Conference 2013](https://md.research.elamur.ai/free/en/venues/sigdial-conference/2013/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W13-4049/) · [PDF](https://aclanthology.org/W13-4049.pdf) · [OpenAlex](https://openalex.org/W2132474242)

**Citations:** 5 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/can-2013-context-evaluation-unsupervised-dialogue-act/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Unsupervised dialogue act models avoid costly labelling. However, they are usually scored by how well they agree with hand-made tags. That brings the annotation cost back and forces data-driven clusters to imitate a human tagset. Ezen-Can and Boyer propose judging such a model by what it enables: driving a simple dialogue policy.

The data is a corpus of typed human-human tutoring sessions about programming. The authors cluster student turns with a query-likelihood method: bigram features plus top-level Stanford parser constituents, fed to k-means, with 21 clusters chosen at a knee of the SSE curve. On the 75% training portion, they write one tutor reply per cluster. They then apply this policy to held-out student turns, and seven untrained judges rate how sensible each reply is.

The unsupervised policy scored slightly higher than a comparable policy built on manual tags. This held even though its accuracy at recovering those tags was only modestly above the majority baseline. The study suggests that task-based evaluation can reveal usefulness that label agreement hides, and at a small fraction of the annotation effort.

## Key points
- **Corpus and tags:** the corpus (Boyer et al., 2009) has 1,525 student and 3,332 tutor utterances. Earlier work labelled it with nine tutoring-specific acts (Kappa 0.76), such as Evaluation Question, Lukewarm Feedback and Extra-domain. These tags were never used for training or tuning.
- **Model and protocol:** the model uses query-likelihood clustering (Ezen-Can & Boyer, 2013), with k tested from 2 to 24 and k=21 selected. Of 373 held-out test pairs, 29 were rated by seven judges on a 1–4 scale, in randomized order, interleaved with 29 manual-tag pairs.
- **Ratings:** the unsupervised policy averaged 2.35, with 44.8% rated positive and 65.5% judged at least somewhat sensible. The manual-tag policy averaged 2.22, with 31% positive.
- **Label accuracy:** mapping each cluster to its majority manual tag gave 30.4% accuracy, against a 25.87% majority baseline (Evaluation Question). Judge agreement was only fair: weighted Kappa 0.30, and binary Kappa 0.36.
- **Cost:** judging took about 15 s per item, roughly 1 hour in total across judges, and writing the policy took about 1 hour. Full act annotation had cost about 35 s per utterance, roughly 50 hours per annotator.

## Interesting ideas
- Test an induced category set extrinsically: attach one canned action to each category and have people judge the resulting behaviour on unseen data, with no full end-to-end system needed.
- Represent each utterance by its retrieval-likelihood scores against every other training utterance, then cluster those similarity vectors.
- Show the experimental and comparison conditions to the same judges, interleaved, in a fresh random order per judge, and measure ordinal agreement with weighted Kappa.
- Report evaluation cost (judge-minutes versus annotation hours) next to quality numbers, treating cheap judgement as a goal in its own right.
- When comparing clusters with gold labels, give each cluster its majority tag and always report the majority-class baseline beside the accuracy.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (frames): speech-act labels from prag could be scored by whether a fixed act-to-response mapping produces sensible replies, rather than only by agreement with a gold tagset.
- **I2** (learn): candidate speech-act or construction classes could be induced by ranking sentences by similarity over UD subtree features, keeping the grouping explainable.
- **I3** (learn): human checks of answers or generated text could mix Mova outputs with a baseline's in a blind shuffle, which acts as a built-in negative control.
- **I4** (learn): logging the human time spent per evaluation round would help decide where LLM-teacher or silver checks can replace manual gold labelling.
- **I5** (tokens): any unsupervised tag or lemma-class induction in crate en could use this mapping plus baseline to avoid overstating weak gains.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [North Carolina State University](https://md.research.elamur.ai/free/en/universities/north-carolina-state-university/) — University, Raleigh, United States · 7 works · 376 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (3): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 685 works · [🗨️ Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) · 401 works · [💬 Speech act theory](https://md.research.elamur.ai/free/en/topics/speech-act-theory/) · 160 works

## Cite

Cite as: Aysu Ezen-Can and Kristy Boyer. 2013. In-Context Evaluation of Unsupervised Dialogue Act Models for Tutorial Dialogue. Proceedings of the SIGDIAL 2013 Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ezencan2013context,
  title = {{In-Context Evaluation of Unsupervised Dialogue Act Models for Tutorial Dialogue}},
  author = {Aysu Ezen-Can and Kristy Boyer},
  booktitle = {Proceedings of the SIGDIAL 2013 Conference},
  year = {2013},
  url = {https://aclanthology.org/W13-4049/}
}
```

- [← previous Cross-Language Personal Name Mapping](https://md.research.elamur.ai/free/en/papers/yousef-2014-cross-language-personal-name-mapping/)
- [next → Labeled Alignment for Recognizing Textual Entailment](https://md.research.elamur.ai/free/en/papers/wang-2013-labeled-alignment-recognizing-textual-entailment/)
- [↑ back to papers of 2013](https://md.research.elamur.ai/free/en/papers/2013/page/2/)

## References

18 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bangalore, S., Di Fabbrizio, G., & Stent, A. (2008). Learning the Structure of Task-Driven Human-Human Dialogs. IEEE Transactions on Audio, Speech and Language Processing, 16(7), 1249– 1259. [↗](https://doi.org/10.1109/tasl.2008.2001102)
2. Becker, L., Palmer, M., Vuuren, S. Van, & Ward, W. (2012). Learning to Tutor Like a Tutor: Ranking Questions in Context. Proceedings of the International Conference on Intelligent Tutoring Systems, 368–378. [find](https://scholar.google.com/scholar?q=Becker%2C+L%2E%2C+Palmer%2C+M%2E%2C+Vuuren%2C+S%2E+Van%2C+%26+Ward%2C+W%2E+%282012%29%2E+Learning+to+Tutor+Like+a+Tutor%3A+Ranking+Questions+in+Context%2E+Proceedings+of+the+International+Conference+on+Intelligent+Tutoring+Systems%2C+36)
3. Boyer, K. E., Philips, R., Ha, E. Y., Wallis, M. D., Vouk, M. A., & Lester, J. C. (2009). Modeling Dialogue Structure with Adjacency Pair Analysis and Hidden Markov Models. Proceedings of NAACL HLT, 49–52. [find](https://scholar.google.com/scholar?q=Boyer%2C+K%2E+E%2E%2C+Philips%2C+R%2E%2C+Ha%2C+E%2E+Y%2E%2C+Wallis%2C+M%2E+D%2E%2C+Vouk%2C+M%2E+A%2E%2C+%26+Lester%2C+J%2E+C%2E+%282009%29%2E+Modeling+Dialogue+Structure+with+Adjacency+Pair+Analysis+and+Hidden+Markov+Models%2E+Proceedings+of+NAACL+HLT%2C+4)
4. Cohen, J. (1968). Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213-220. [↗](https://doi.org/10.1037/h0026256)
5. Crook, N., Granell, R., & Pulman, S. (2009). Unsupervised Classification of Dialogue Acts Using a Dirichlet Process Mixture Model. Proceedings of SIGDIAL, 341–348. [find](https://scholar.google.com/scholar?q=Crook%2C+N%2E%2C+Granell%2C+R%2E%2C+%26+Pulman%2C+S%2E+%282009%29%2E+Unsupervised+Classification+of+Dialogue+Acts+Using+a+Dirichlet+Process+Mixture+Model%2E+Proceedings+of+SIGDIAL%2C+341%E2%80%93348%2E)
6. Ezen-Can, A., & Boyer, K. E. (2013). Unsupervised Classification of Student Dialogue Acts With Query-likelihood Clustering. International Conference on Educational Data Mining, 20– 27. [find](https://scholar.google.com/scholar?q=Ezen%2DCan%2C+A%2E%2C+%26+Boyer%2C+K%2E+E%2E+%282013%29%2E+Unsupervised+Classification+of+Student+Dialogue+Acts+With+Query%2Dlikelihood+Clustering%2E+International+Conference+on+Educational+Data+Mining%2C+20%E2%80%93+27%2E)
7. Forbes-Riley, K., & Litman, D. J. (2005). Using Bigrams to Identify Relationships Between Student Certainness States and Tutor Responses in a Spoken Dialogue Corpus. Proceedings of SIGDIAL, 87–96. [find](https://scholar.google.com/scholar?q=Forbes%2DRiley%2C+K%2E%2C+%26+Litman%2C+D%2E+J%2E+%282005%29%2E+Using+Bigrams+to+Identify+Relationships+Between+Student+Certainness+States+and+Tutor+Responses+in+a+Spoken+Dialogue+Corpus%2E+Proceedings+of+SIGDIAL%2C+87%E2%80%9396%2E)
8. Henderson, J., Lemon, O., & Georgila, K. (2008). Hybrid Reinforcement / Supervised Learning of Dialogue Policies from Fixed Data Sets. Computational Linguistics, 34(4), 487-511. [↗](https://doi.org/10.1162/coli.2008.07-028-r2-05-82)
9. Klein, D., & Manning, C. D. (2003). [Accurate Unlexicalized Parsing](https://md.research.elamur.ai/free/en/papers/klein-2003-accurate-unlexicalized-parsing/). Proceedings of ACL, 423–430.
10. Landis, J. R., & Koch, G. G. (1994). The Measurement of Observer Agreement for Categorical Data Data for Categorical of Observer Agreement The Measurement. International Biometric Society, 33(1), 159– 174. [find](https://scholar.google.com/scholar?q=Landis%2C+J%2E+R%2E%2C+%26+Koch%2C+G%2E+G%2E+%281994%29%2E+The+Measurement+of+Observer+Agreement+for+Categorical+Data+Data+for+Categorical+of+Observer+Agreement+The+Measurement%2E+International+Biometric+Society%2C+33%281%29%2C+159%E2%80%93)
11. Mayfield, E., Adamson, D., & Rosé, C. P. (2012). Hierarchical Conversation Structure Prediction in Multi-Party Chat. Proceedings of SIGDIAL, 60–69. [find](https://scholar.google.com/scholar?q=Mayfield%2C+E%2E%2C+Adamson%2C+D%2E%2C+%26+Ros%C3%A9%2C+C%2E+P%2E+%282012%29%2E+Hierarchical+Conversation+Structure+Prediction+in+Multi%2DParty+Chat%2E+Proceedings+of+SIGDIAL%2C+60%E2%80%9369%2E)
12. Ritter, A., Cherry, C., & Dolan, B. (2010). [Unsupervised Modeling of Twitter Conversations](https://md.research.elamur.ai/free/en/papers/ritter-2010-unsupervised-modeling-twitter-conversations/). Proceedings of NAACL HLT, 172–180.
13. Rus, V., Moldovan, C., Niraula, N., & Graesser, A. C. (2012). Automated Discovery of Speech Act Categories in Educational Games. Proceedings of the International Conference on Educational Data Mining, 25-32. [find](https://scholar.google.com/scholar?q=Rus%2C+V%2E%2C+Moldovan%2C+C%2E%2C+Niraula%2C+N%2E%2C+%26+Graesser%2C+A%2E+C%2E+%282012%29%2E+Automated+Discovery+of+Speech+Act+Categories+in+Educational+Games%2E+Proceedings+of+the+International+Conference+on+Educational+Data+Mining%2C)
14. Serafin, R., & Di Eugenio, B. (2004). [FLSA : Extending Latent Semantic Analysis with features for dialogue act classification](https://md.research.elamur.ai/free/en/papers/serafin-2004-flsa-extending-latent-semantic-analysis/). Proceedings of ACL, 692–699.
15. Stolcke, A., Ries, K., Coccaro, N., Shriberg, E., Bates, R., Jurafsky, D., Taylor, P., et al. (2000). [Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech](https://md.research.elamur.ai/free/en/papers/stolcke-2000-dialogue-act-modeling-automatic-tagging/). Computational Linguistics, 26(3), 339–373. [↗](https://doi.org/10.1162/089120100561737)
16. Vlachos, A. (2011). Evaluating unsupervised learning for natural language processing tasks. Proceedings of EMNLP, 35–42. [find](https://scholar.google.com/scholar?q=Vlachos%2C+A%2E+%282011%29%2E+Evaluating+unsupervised+learning+for+natural+language+processing+tasks%2E+Proceedings+of+EMNLP%2C+35%E2%80%9342%2E)
17. Williams, J. D., & Young, S. (2003). Using Wizard-of-Oz simulations to bootstrap Reinforcement-Learning- based dialog management systems. Proceedings of SIGDIAL, 135–139. [find](https://scholar.google.com/scholar?q=Williams%2C+J%2E+D%2E%2C+%26+Young%2C+S%2E+%282003%29%2E+Using+Wizard%2Dof%2DOz+simulations+to+bootstrap+Reinforcement%2DLearning%2D+based+dialog+management+systems%2E+Proceedings+of+SIGDIAL%2C+135%E2%80%93139%2E)
18. Woszczyna, M., & Waibel, A. (1994). Inferring linguistic structure in spoken language. Proceedings of ICSLP, 847-850. [find](https://scholar.google.com/scholar?q=Woszczyna%2C+M%2E%2C+%26+Waibel%2C+A%2E+%281994%29%2E+Inferring+linguistic+structure+in+spoken+language%2E+Proceedings+of+ICSLP%2C+847%2D850%2E)

*Updated 24 October 2026*