---
title: "Increasing Robustness for Cross-domain Dialogue Act"
description: "The authors build a benchmark with two test conditions. 30 references, cited by 2 papers here."
authors: ["Marcus Vielsted", "Nikolaj Wallenius", "Rob van der Goot"]
year: 2022
venue: "Proceedings of the Eighth Workshop on Noisy User-generated Text (W-NUT 2022)"
source: "https://aclanthology.org/2022.wnut-1.20/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/vielsted-2022-increasing-robustness-cross-domain-dialogue/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/vielsted-2022-increasing-robustness-cross-domain-dialogue/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/6/) / Increasing Robustness for Cross-domain Dialogue Act Classification on Social Media Data

# Increasing Robustness for Cross-domain Dialogue Act Classification on Social Media Data

**Authors:** Marcus Vielsted, Nikolaj Wallenius and [Rob van der Goot](https://md.research.elamur.ai/free/en/authors/rob-van-der-goot/) · **Year:** 2022 · **Venue:** [Proceedings of the Eighth Workshop on Noisy User-generated Text (W-NUT 2022)](https://md.research.elamur.ai/free/en/venues/wnut/) ([Workshop on Noisy User-generated Text 2022](https://md.research.elamur.ai/free/en/venues/wnut/2022/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.wnut-1.20/) · [PDF](https://aclanthology.org/2022.wnut-1.20.pdf) · [OpenAlex](https://openalex.org/W4412285405)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · 2 in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper looks at how well dialogue act classification (DAC), which assigns each utterance an intent label such as question, disagreement or greeting, holds up on social media text. Most earlier DAC work used spoken one-to-one conversations. Online text is noisy, one-to-many, and its norms shift quickly. The authors build a benchmark with two test conditions. In-domain data comes from the 2006 NPS Chat Corpus. Cross-domain data is a newly annotated Reddit set drawn from Convokit. Both use a label set adapted from the ISO 24617-2:2020 standard. On top of a fine-tuned transformer baseline they test three robustness strategies: lexical normalization, label resampling, and adding context from the preceding utterance. Normalization does not help. Resampling helps only in some settings. Context gives by far the largest gain, about 7 F1 points in-domain and about 20 cross-domain. The work matters because it shows that intent recognition on unseen platforms depends mostly on conversational context, and it releases public data for a task whose earlier social media datasets were unavailable.

## Key points
- **Data:**
  - NPS Chat Corpus: 10,567 utterances in total. The authors split off 4,800 for training, 600 for dev and 600 for test.
  - Reddit cross-domain set: 1,705 utterances from 50 threads (853 dev, 852 test).
  - Utterances are segmented into the smallest functional units, so one post can yield several dialogue acts.
- **Label set:** adapted from ISO 24617-2:2020, with 19 labels, for example:
  - question types: propositionalQuestion, setQuestion, choiceQuestion;
  - other acts: inform, elaborate, agreement, disagreement, correction, offer, acceptAction/declineAction, misc.
- **Model selection:** 17 language models were compared on in-domain dev, and all tuning used in-domain dev only. The Reddit dev set served purely for evaluation, so the target domain never influenced design choices.
- **Context study:** 31 configurations were tested, combining:
  - which context is added: the previous utterance's label, sender and/or text;
  - their order;
  - where it is placed: before or after the input;
  - whether the context label is gold or predicted.

  On cross-domain dev, accuracy rose from about 54 F1 with no context to around 76 F1 in the best setups.
- **Reddit sampling:**
  - kept threads with more than 40 posts and a mean post length of 50–150 characters;
  - took the first 20 consecutive posts from 50 random such threads.

## Interesting ideas
- Keep target-domain data entirely out of tuning and model selection, so cross-domain scores show real transfer and are not inflated by repeated peeking.
- Feed the previous utterance's predicted label, speaker and text into the input, and test whether predicted labels work as well as gold ones; here they often matched or beat gold.
- Before labeling, split a single post into minimal functional segments, so that a greeting, a question and a follow-up each get their own act.
- Adapt an ISO standard by merging rare fine-grained labels until every class is actually present in the data, and redefine "inform" as context-free.
- Run a cheap check that seems helpful (normalizing noisy text) and keep it as an evaluated negative result.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Use held-out genres (e.g., fables vs. school texts) only as final evaluation sets, never for picking repair rules or hyperparameters.
- **I2** (frames): The speech-act classifier in prag can condition on the preceding act and on whether the speaker changed, using its own predictions at inference.
- **I3** (frames): Segment sentences into act-bearing units before speech-act assignment, so mixed turns aren't forced into a single label.
- **I4** (frames): Map Mova's speech-act inventory onto ISO 24617-2 categories, coarsening where the corpus is too thin, so labels stay comparable and explainable.
- **I5** (tokens): Measure whether spelling normalization in the tokenizer actually improves downstream acts or trees before adopting it as a default step.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [IT University of Copenhagen](https://md.research.elamur.ai/free/en/universities/it-university-of-copenhagen/) — University, Copenhagen, Denmark · 16 works · 2,225 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [🗨️ Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) · 401 works · [💬 Speech act theory](https://md.research.elamur.ai/free/en/topics/speech-act-theory/) · 160 works

## Cite

Cite as: Marcus Vielsted, Nikolaj Wallenius and Rob van der Goot. 2022. Increasing Robustness for Cross-domain Dialogue Act Classification on Social Media Data. Proceedings of the Eighth Workshop on Noisy User-generated Text (W-NUT 2022).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{vielsted2022increasing,
  title = {{Increasing Robustness for Cross-domain Dialogue Act Classification on Social Media Data}},
  author = {Marcus Vielsted, Nikolaj Wallenius and Rob {van der Goot}},
  booktitle = {Proceedings of the Eighth Workshop on Noisy User-generated Text (W-NUT 2022)},
  year = {2022},
  url = {https://aclanthology.org/2022.wnut-1.20/}
}
```

- [← previous I2C at SemEval-2022 Task 6: Intended Sarcasm in English using Deep Learning Techniques](https://md.research.elamur.ai/free/en/papers/monterde-2022-i2c-semeval-2022-task-6/)
- [next → Mitigating Dataset Artifacts in Natural Language Inference Through Automatic Contextual Data Augmentation and Learning Optimization](https://md.research.elamur.ai/free/en/papers/mersinias-2022-mitigating-dataset-artifacts-natural-language/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/6/)

## References

30 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Firstly, we found that lexical normalization does not constitute a stochastically dominant feature for cross-domain applications, but rather had a nega-References tive effect on F1 performance. By applying lexical normalization, the performance dropped for both Ali Ahmadvand, Jason Ingyu Choi, and Eugene our in-domain data, while staying the same for our Agichtein. 2019. Contextual dialogue act classifi-cross-domain dataset. Additionally, while it de-cation for open-domain conversational agents. SIGIR 2019 - Proceedings of the 42nd International ACM creased the standard deviation for our in-domain SIGIR Conference on Research and Development in model, it almost doubled the standard deviations Information Retrieval, pages 1273–1276. for our cross-domain model. We can therefore state that lexical normalization does not improve Mikel Artetxe, Sebastian Ruder, Dani Yogatama, Gorka the robustness nor increase the F1 score of either in-Labaka, and Eneko Agirre. 2020. A call for more 188 rigor in unsupervised cross-lingual learning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7375– 7388, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Firstly%2C+we+found+that+lexical+normalization+does+not+constitute+a+stochastically+dominant+feature+for+cross%2Ddomain+applications%2C+but+rather+had+a+nega%2DReferences+tive+effect+on+F1+performance%2E+By+app)
2. Timothy Baldwin, Paul Cook, Marco Lui, Andrew MacKinlay, and Li Wang. 2013. How Noisy Social Media Text, How Different Social Media Sources? [find](https://scholar.google.com/scholar?q=Timothy+Baldwin%2C+Paul+Cook%2C+Marco+Lui%2C+Andrew+MacKinlay%2C+and+Li+Wang%2E+2013%2E+How+Noisy+Social+Media+Text%2C+How+Different+Social+Media+Sources%3F)
3. Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604. [↗](https://doi.org/10.1162/tacl_a_00041)
4. Chandrakant Bothe, Sven Magg, Cornelius Weber, and Stefan Wermter. 2018a. Conversational Analysis using Utterance-level Attention-based Bidirectional Recurrent Neural Networks. Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, 2018-September:996–1000. [find](https://scholar.google.com/scholar?q=Chandrakant+Bothe%2C+Sven+Magg%2C+Cornelius+Weber%2C+and+Stefan+Wermter%2E+2018a%2E+Conversational+Analysis+using+Utterance%2Dlevel+Attention%2Dbased+Bidirectional+Recurrent+Neural+Networks%2E+Proceedings+of+the+Annu)
5. Chandrakant Bothe, Cornelius Weber, Sven Magg, and Stefan Wermter. 2018b. [A context-based approach for dialogue act recognition using simple recurrent neural networks](https://md.research.elamur.ai/free/en/papers/bothe-2018-context-based-approach-dialogue-act/). In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). [↗](https://doi.org/10.63317/2b2w5ksnfqov)
6. Jacob Cohen. 1960. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1):37–46. Linking and Mining Heterogeneous and Multi-view Data, chapter 6, pages 137–160. Springer International Publishing. [↗](https://doi.org/10.1177/001316446002000104)
7. Jacob Eisenstein. 2013. [What to do about bad language on the internet](https://md.research.elamur.ai/free/en/papers/eisenstein-2013-what-do-about-bad-language/). In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 359–369. Association for Computational Linguistics.
8. Eric Forsyth, Jane Lin, and Craig Martell. 2008. The NPS Chat Corpus. [find](https://scholar.google.com/scholar?q=Eric+Forsyth%2C+Jane+Lin%2C+and+Craig+Martell%2E+2008%2E+The+NPS+Chat+Corpus%2E)
9. Rob van der Goot. 2021. We Need to Talk About train-dev-test Splits. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4485–4494. Association for Computational Linguistics. [↗](https://doi.org/10.48448/gcx0-nz17)
10. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://md.research.elamur.ai/free/en/papers/he-2021-debertav3-improving-deberta-using-electra/). [↗](https://doi.org/10.48550/arxiv.2111.09543)
11. ISO 24617-2. 2020. ISO 24617-2 - Language resource management - Semantic annotation framework - Part 2: Dialogue acts. ISO, pages 1–96. [find](https://scholar.google.com/scholar?q=ISO+24617%2D2%2E+2020%2E+ISO+24617%2D2+%2D+Language+resource+management+%2D+Semantic+annotation+framework+%2D+Part+2%3A+Dialogue+acts%2E+ISO%2C+pages+1%E2%80%9396%2E)
12. Chen Li and Yang Liu. 2014. Improving text normalization via unsupervised model and discriminative reranking. In Proceedings of the ACL 2014 Student Research Workshop, pages 86–93, Baltimore, Maryland, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chen+Li+and+Yang+Liu%2E+2014%2E+Improving+text+normalization+via+unsupervised+model+and+discriminative+reranking%2E+In+Proceedings+of+the+ACL+2014+Student+Research+Workshop%2C+pages+86%E2%80%9393%2C+Baltimore%2C+Maryland)
13. CornellNLP. 2021. ConvoKit version 2.5.1. [find](https://scholar.google.com/scholar?q=CornellNLP%2E+2021%2E+ConvoKit+version+2%2E5%2E1%2E)
14. Dan Jurafsky, Liz Shriberg, and Debra Biasca. 1997. Switchboard SWBD-DAMSL Shallow-DiscourseFunction Annotation Coders Manual. SRI International. [find](https://scholar.google.com/scholar?q=Dan+Jurafsky%2C+Liz+Shriberg%2C+and+Debra+Biasca%2E+1997%2E+Switchboard+SWBD%2DDAMSL+Shallow%2DDiscourseFunction+Annotation+Coders+Manual%2E+SRI+International%2E)
15. Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). [↗](https://doi.org/10.18653/v1/n19-1423)
16. NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference, 1:4171– 4186. [find](https://scholar.google.com/scholar?q=NAACL+HLT+2019+%2D+2019+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics%3A+Human+Language+Technologies+%2D+Proceedings+of+the+Conference%2C+1%3A4171%E2%80%93+4186%2E)
17. Rotem Dror, Segev Shlomov, and Roi Reichart. 2019. Deep Dominance - How to Properly Compare Deep Neural Models. ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, pages 2773–2785. [find](https://scholar.google.com/scholar?q=Rotem+Dror%2C+Segev+Shlomov%2C+and+Roi+Reichart%2E+2019%2E+Deep+Dominance+%2D+How+to+Properly+Compare+Deep+Neural+Models%2E+ACL+2019+%2D+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+Proceed)
18. Nathan Duran, Steve Battle, Jim Smith, and Nathan Duran. 2021. Sentence encoding for Dialogue Act classification. Natural Language Engineering, pages 1–30. [↗](https://doi.org/10.1017/s1351324921000310)
19. Subhabrata Dutta, Tanmoy Chakraborty, and Dipankar Das. 2019. How Did the Discussion Go: Discourse Act Classification in Social Media Conversations. In Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [↗](https://arxiv.org/abs/1907.11692)
20. Christopher Potts. 1998. The Switchboard Dialog Act Corpus. [find](https://scholar.google.com/scholar?q=Christopher+Potts%2E+1998%2E+The+Switchboard+Dialog+Act+Corpus%2E)
21. Dat Quoc Nguyen, Thanh Vu, Anh Tuan Nguyen, and VinAI Research. 2020. [BERTweet: A pre-trained language model for English Tweets](https://md.research.elamur.ai/free/en/papers/nguyen-2020-bertweet-pre-trained-language-model/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 9–14. Association for Computational Linguistics.
22. Vipul Raheja and Joel Tetreault. 2019. [Dialogue Act Classification with Context-Aware Self-Attention](https://md.research.elamur.ai/free/en/papers/raheja-2019-dialogue-act-classification-context-aware/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), volume 1, pages 3727–3733. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1904.02594)
23. Tulika Saha, Sriparna Saha, and Pushpak Bhattacharyya. 2019. Tweet Act Classification : A Deep Learning based Classifier for Recognizing Speech Acts in Twitter. In 2019 International Joint Conference on Neural Networks (IJCNN), volume 2019-July. Institute of Electrical and Electronics Engineers Inc. [find](https://scholar.google.com/scholar?q=Tulika+Saha%2C+Sriparna+Saha%2C+and+Pushpak+Bhattacharyya%2E+2019%2E+Tweet+Act+Classification+%3A+A+Deep+Learning+based+Classifier+for+Recognizing+Speech+Acts+in+Twitter%2E+In+2019+International+Joint+Conference+)
24. Elizabeth Shriberg, Raj Dhillon, Sonali Bhagat, Jeremy Ang, and Hannah Carvey. 2004. [The ICSI Meeting Recorder Dialog Act (MRDA) Corpus](https://md.research.elamur.ai/free/en/papers/shriberg-2004-icsi-meeting-recorder-dialog-act/). In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue at HLT-NAACL 2004, pages 97–100. OSD or Non-Service DoD Agency, Association for Computational Linguistics. [↗](https://doi.org/10.21236/ada460980)
25. Dennis Ulmer, Christian Hardmeier, and Jes Frellsen. 2022. deep-significance-easy and meaningful statistical significance testing in the age of neural networks. arXiv preprint arXiv:2204.06815. [↗](https://arxiv.org/abs/2204.06815)
26. Rob Van Der Goot. 2019. MoNoise: A Multi-lingual and Easy-to-use Lexical Normalization Tool. ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of System Demonstrations, pages 201–206. [find](https://scholar.google.com/scholar?q=Rob+Van+Der+Goot%2E+2019%2E+MoNoise%3A+A+Multi%2Dlingual+and+Easy%2Dto%2Duse+Lexical+Normalization+Tool%2E+ACL+2019+%2D+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+Proceedings+of+System+Demo)
27. Rob van der Goot, Alan Ramponi, Arkaitz Zubiaga, Barbara Plank, Benjamin Muller, Iñaki San Vicente Ron-cal, Nikola Ljubešić, Özlem Çetinoğlu, Rahmad Mahendra, Talha Çolakoğlu, Timothy Baldwin, Tommaso Caselli, and Wladimir Sidorenko. 2021. Mul-tiLexNorm: A shared task on multilingual lexical normalization. In Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021), pages 493–509, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rob+van+der+Goot%2C+Alan+Ramponi%2C+Arkaitz+Zubiaga%2C+Barbara+Plank%2C+Benjamin+Muller%2C+I%C3%B1aki+San+Vicente+Ron%2Dcal%2C+Nikola+Ljube%C5%A1ic%CC%81%2C+%C3%96zlem+%C3%87etinog%CC%86lu%2C+Rahmad+Mahendra%2C+Talha+%C3%87olakog%CC%86lu%2C+Timothy+Baldwin%2C+Tomm)
28. Nikolaj Wallenius and Marcus Lind Vielsted. 2021. Dialogue act classification for social media data. [find](https://scholar.google.com/scholar?q=Nikolaj+Wallenius+and+Marcus+Lind+Vielsted%2E+2021%2E+Dialogue+act+classification+for+social+media+data%2E)
29. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. [Transformers: State-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.emnlp-demos.6)
30. Tianhao Wu, Faisal M. Khan, Todd A. Fisher, Lori A. Shuler, and William M. Pottenger. 2005. Posting Act Tagging Using Transformation-Based Learning. In Foundations of Data Mining and knowledge Discovery, pages 319–331. Springer, Berlin, Heidelberg. [↗](https://doi.org/10.1007/11498186_18)

*Updated 24 October 2026*