---
title: "Improving the Quality of Minority Class Identification in Dialog Act"
description: "The paper addresses a common weakness of dialog act (DA) taggers: they score well on overall… 16 references, cited by 8 papers here."
authors: ["Adinoyi Omuya", "Vinodkumar Prabhakaran", "Owen Rambow"]
year: 2013
venue: "Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies"
source: "https://aclanthology.org/N13-1099/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/omuya-2013-improving-quality-minority-class-identification/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/omuya-2013-improving-quality-minority-class-identification/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2013](https://md.research.elamur.ai/free/en/papers/2013/page/2/) / Improving the Quality of Minority Class Identification in Dialog Act Tagging

# Improving the Quality of Minority Class Identification in Dialog Act Tagging

**Authors:** Adinoyi Omuya, [Vinodkumar Prabhakaran](https://md.research.elamur.ai/free/en/authors/vinodkumar-prabhakaran/) and [Owen Rambow](https://md.research.elamur.ai/free/en/authors/owen-rambow/) · **Year:** 2013 · **Venue:** [Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2013](https://md.research.elamur.ai/free/en/venues/naacl/2013/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/N13-1099/) · [PDF](https://aclanthology.org/N13-1099.pdf) · [OpenAlex](https://openalex.org/W2132406104)

**Citations:** 13 citations (OpenAlex, 2026-10-14) · [8](https://md.research.elamur.ai/free/en/papers/omuya-2013-improving-quality-minority-class-identification/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper addresses a common weakness of dialog act (DA) taggers: they score well on overall accuracy but miss rare classes. In email, the rarest is REQUEST-ACTION. The authors start from the SVM tagger of Hu et al. (2009) and use the Enron-derived email corpus from that work, segmented into Dialog Functional Units (DFUs). Each DFU is labelled with one of four DAs.

They change the standard one-against-all multi-class SVM in two ways:
- **Per-class feature optimization.** Each binary classifier gets its own feature set, tuned for its own F-measure.
- **Cascaded decision rule.** The binary classifiers are consulted from the rarest class to the most frequent, and the first positive answer wins. Only when every classifier says no does the system fall back to picking the most confident one.

The final system reduces F-measure error on REQUEST-ACTION by 22.8% and overall error by about 10%. It also improves a downstream task that detects Overt Displays of Power. The broader lesson is that treating a multi-class SVM as an opaque package hides easy gains for minority classes.

## Key points
- **Data.** The corpus has 122 Enron email threads, 360 messages, 20,740 tokens and 1406 DFUs. The class distribution is very uneven:
  - REQUEST-ACTION: 35 (2.5%)
  - REQUEST-INFORMATION: 151
  - CONVENTIONAL: 357
  - INFORM: 853

  Labels occurring 5 or fewer times were dropped.
- **Setup.** The classifiers are linear-kernel SVMs built with ClearTK's SVMLight wrapper and evaluated with 5-fold cross validation. POS tags come from the OpenNLP tagger.
- **Features** fall into three groups:
  - lexical: word, POS and mixed n-grams, first/last POS and lemma, modal count, question mark;
  - verbal: position and identity of the first verb;
  - message-level: position and size of the DFU and the message.

  Feature selection is exhaustive within each group, followed by a search over combinations of groups.
- **Main results.**

  | System | REQUEST-ACTION F | Accuracy |
  |---|---|---|
  | Baseline (BAS) | 40.7 | 91.3 |
  | Per-class optimization (DAC) | 50.0 | 91.7 |
  | DAC + rare-class preference (DAC MP) | 54.2 | 91.8 |
  | DAC + full cascade (DAC CMP) | 54.2 | 92.2 (10.6% error reduction) |

  With the full cascade, every other class improves too.
- **Extrinsic task (Overt Displays of Power).**

  | DA tags used | ODP F-measure |
  |---|---|
  | Gold DA tags | 65.8 |
  | BAS tags | 52.6 |
  | DAC CMP tags | 54.2 |
  | No DA features | 50.0 |

  DAC CMP tags close 12.1% of the gap between BAS and gold, and give a 26.6% error reduction over using no DA features.

## Interesting ideas
- Tuning a separate feature set for each one-vs-rest classifier lets each class use its own best evidence: word n-grams for INFORM and CONVENTIONAL, POS n-grams for the two request types.
- Confidence-based argmax drowns out rare classes, whose classifiers have weaker weights simply because they see less data; asking the rare-class classifiers first fixes this without retraining anything.
- Inspecting the top weights of each trained classifier shows interpretable cues, such as "please" for action requests and a question mark that counts for information requests but against INFORM.
- The extrinsic evaluation feeds predicted rather than gold tags into a downstream task, keeps the cross-validation folds strictly separate, and reports the result against both a gold-tag ceiling and a no-feature floor.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (frames): The speech-act detectors in prag could each keep their own small, inspectable feature set instead of one shared vector, which would make each decision easier to explain.
- **I2** (frames): Speech-act or irony decisions could be ordered as a cascade with rare acts like REQUEST-ACTION checked first, and the explanation trace would record which rule fired.
- **I3** (learn): After training, top-weight dumps per class could serve as a sanity check and as seeds for self-induced repair rules.
- **I4** (learn): When an upstream layer such as tagging or parsing is improved, its value could be measured by how much of the downstream gap between the floor and gold input it closes.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Columbia University](https://md.research.elamur.ai/free/en/universities/columbia-university/) — University, New York, United States · 149 works · 48,852 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [🗨️ Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) · 401 works · [💬 Speech act theory](https://md.research.elamur.ai/free/en/topics/speech-act-theory/) · 160 works

## Cite

Cite as: Adinoyi Omuya, Vinodkumar Prabhakaran and Owen Rambow. 2013. Improving the Quality of Minority Class Identification in Dialog Act Tagging. Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{omuya2013improving,
  title = {{Improving the Quality of Minority Class Identification in Dialog Act Tagging}},
  author = {Adinoyi Omuya, Vinodkumar Prabhakaran and Owen Rambow},
  booktitle = {Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2013},
  doi = {10.13140/2.1.1516.4163},
  url = {https://doi.org/10.13140/2.1.1516.4163}
}
```

- [← previous A Generative Joint, Additive, Sequential Model of Topics and Speech Acts in Patient-Doctor Communication](https://md.research.elamur.ai/free/en/papers/wallace-2013-generative-joint-additive-sequential-model/)
- [next → Multi-document multilingual summarization corpus preparation, Part 2: Czech, Hebrew and Spanish](https://md.research.elamur.ai/free/en/papers/elhadad-2013-multi-document-multilingual-summarization-corpus/)
- [↑ back to papers of 2013](https://md.research.elamur.ai/free/en/papers/2013/page/2/)

## References

16 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Apoorv Agarwal, Adinoyi Omuya, Aaron Harnly, and Owen Rambow. 2012. A Comprehensive Gold Standard for the Enron Organizational Hierarchy. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 161–165, Jeju Island, Korea, July. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Apoorv+Agarwal%2C+Adinoyi+Omuya%2C+Aaron+Harnly%2C+and+Owen+Rambow%2E+2012%2E+A+Comprehensive+Gold+Standard+for+the+Enron+Organizational+Hierarchy%2E+In+Proceedings+of+the+50th+Annual+Meeting+of+the+Association+f)
2. J. L. Austin. 1975. How to Do Things with Words. Harvard University Press, Cambridge, Mass. [↗](https://doi.org/10.1093/acprof:oso/9780198245537.001.0001)
3. William W. Cohen, Vitor R. Carvalho, and Tom M. Mitchell. 2004. [Learning to Classify Email into “Speech Acts”](https://md.research.elamur.ai/free/en/papers/cohen-2004-learning-classify-email-into-speech/). In Dekang Lin and Dekai Wu, editors, Proceedings of EMNLP 2004, pages 309–316, Barcelona, Spain, July. Association for Computational Linguistics.
4. Isabelle Guyon, Jason Weston, Stephen Barnhill, and Vladimir Vapnik. 2002. Gene Selection for Cancer Classification using Support Vector Machines. Mach. Learn., 46:389–422, March. [↗](https://doi.org/10.1023/a:1012487302797)
5. Jun Hu, Rebecca Passonneau, and Owen Rambow. 2009. Contrasting the Interaction Structure of an Email and a Telephone Corpus: A Machine Learning Approach to Annotation of Dialogue Function Units. In Proceedings of the SIGDIAL 2009 Conference, London, UK, September. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jun+Hu%2C+Rebecca+Passonneau%2C+and+Owen+Rambow%2E+2009%2E+Contrasting+the+Interaction+Structure+of+an+Email+and+a+Telephone+Corpus%3A+A+Machine+Learning+Approach+to+Annotation+of+Dialogue+Function+Units%2E+In+Pr)
6. Thorsten Joachims. 1999. Making Large-Scale SVM Learning Practical. In Bernhard Schölkopf, Christopher J.C. Burges, and A. Smola, editors, Advances in Kernel Methods - Support Vector Learning, Cambridge, MA, USA. MIT Press. [↗](https://doi.org/10.17877/de290r-14262)
7. J. Kim, G. Chern, D. Feng, E. Shaw, and E. Hovy. 2006. Mining and Assessing Discussions on the Web Through Speech Act Analysis. In Proceedings of the Workshop on Web Content Mining with Human Language Technologies at the 5th International Semantic Web Conference. [find](https://scholar.google.com/scholar?q=J%2E+Kim%2C+G%2E+Chern%2C+D%2E+Feng%2C+E%2E+Shaw%2C+and+E%2E+Hovy%2E+2006%2E+Mining+and+Assessing+Discussions+on+the+Web+Through+Speech+Act+Analysis%2E+In+Proceedings+of+the+Workshop+on+Web+Content+Mining+with+Human+Language)
8. S.N. Kim, L. Cavedon, and T. Baldwin. 2010a. [Classifying Dialogue Acts in One-on-one Live Chats](https://md.research.elamur.ai/free/en/papers/kim-2010-classifying-dialogue-acts-one-one/). In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 862–871. Association for Computational Linguistics.
9. S.N. Kim, L. Wang, and T. Baldwin. 2010b. Tagging and Linking Web Forum Posts. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, pages 192–202. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=S%2EN%2E+Kim%2C+L%2E+Wang%2C+and+T%2E+Baldwin%2E+2010b%2E+Tagging+and+Linking+Web+Forum+Posts%2E+In+Proceedings+of+the+Fourteenth+Conference+on+Computational+Natural+Language+Learning%2C+pages+192%E2%80%93202%2E+Association+for+Co)
10. Hsuan-Tien Lin, Chih-Jen Lin, and Ruby C. Weng. 2007. A Note on Platt’s Probabilistic Outputs for Support Vector Machines. Mach. Learn., 68:267–276, October. [↗](https://doi.org/10.1007/s10994-007-5018-6)
11. Philip V. Ogren, Philipp G. Wetzler, and Steven Bethard. 2008. ClearTK: A UIMA toolkit for statistical natural language processing. In Towards Enhanced Interoperability for Large HLT Systems: UIMA for NLP workshop at Language Resources and Evaluation Conference (LREC). [find](https://scholar.google.com/scholar?q=Philip+V%2E+Ogren%2C+Philipp+G%2E+Wetzler%2C+and+Steven+Bethard%2E+2008%2E+ClearTK%3A+A+UIMA+toolkit+for+statistical+natural+language+processing%2E+In+Towards+Enhanced+Interoperability+for+Large+HLT+Systems%3A+UIMA+for)
12. Vinodkumar Prabhakaran, Owen Rambow, and Mona Diab. 2012. Predicting Overt Display of Power in Written Dialogs. In Human Language Technologies: The 2012 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Montreal, Canada, June. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Vinodkumar+Prabhakaran%2C+Owen+Rambow%2C+and+Mona+Diab%2E+2012%2E+Predicting+Overt+Display+of+Power+in+Written+Dialogs%2E+In+Human+Language+Technologies%3A+The+2012+Annual+Conference+of+the+North+American+Chapter)
13. J.R. Searle. 1976. A Classification of Illocutionary Acts. Language in society, 5(01):1–23. [↗](https://doi.org/10.1017/s0047404500006837)
14. A. Stolcke, K. Ries, N. Coccaro, E. Shriberg, R. Bates, D. Jurafsky, P. Taylor, R. Martin, C.V. Ess-Dykema, and M. Meteer. 2000. [Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech](https://md.research.elamur.ai/free/en/papers/stolcke-2000-dialogue-act-modeling-automatic-tagging/). Computational linguistics, 26(3):339–373. [↗](https://doi.org/10.1162/089120100561737)
15. J.Y. Yeh and A. Harnly. 2006. Email Thread Reassembly Using Similarity Matching. In Third Conference on Email and Anti-Spam (CEAS), pages 27–28. [↗](https://doi.org/10.7916/d8j3921q)
16. R. Zhang, D. Gao, and W. Li. 2012. [Towards Scalable Speech Act Recognition in Twitter: Tackling Insufficient Training Data](https://md.research.elamur.ai/free/en/papers/zhang-2012-scalable-speech-act-recognition-twitter/). EACL 2012, page 18.

*Updated 24 October 2026*