---
title: "Building Multi-turn Intent Classification with LLM-based Labeling"
description: "The work offers practical guidance for deploying accurate, low-latency multi-turn intent systems. 25 references."
authors: ["Biancen Xie", "Kaiqi Bian", "Jai Ranjan Singh Gusain", "Manikandarajan Ramanathan", "Raj Maragoud"]
year: 2026
venue: "Proceedings of the Second Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNLP4U)"
source: "https://aclanthology.org/2026.customnlp4u-1.8/"
doi: "10.18653/v1/2026.customnlp4u-1.8"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xie-2026-building-multi-turn-intent-classification/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xie-2026-building-multi-turn-intent-classification/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/25/) / Building Multi-turn Intent Classification with LLM-based Labeling

# Building Multi-turn Intent Classification with LLM-based Labeling

**Authors:** Biancen Xie, Kaiqi Bian, Jai Ranjan Singh Gusain, Manikandarajan Ramanathan and Raj Maragoud · **Year:** 2026 · **Venue:** [Proceedings of the Second Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNLP4U)](https://md.research.elamur.ai/free/en/venues/customnlp4u/) ([Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual 2026](https://md.research.elamur.ai/free/en/venues/customnlp4u/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.customnlp4u-1.8/) · [PDF](https://aclanthology.org/2026.customnlp4u-1.8.pdf) · [DOI 10.18653/v1/2026.customnlp4u-1.8](https://doi.org/10.18653/v1/2026.customnlp4u-1.8) · [OpenAlex](https://openalex.org/W7166824616)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
The paper addresses turn-level intent detection for routing e-commerce customer-service chats, where taxonomies are large, users change goals mid-conversation, and labelled multi-turn data is scarce. Instead of one flat label set, the authors split intent into three dimensions: Domain (product), Intent (conversational or actionable) and Issue (slot-like attribute). Each dimension gets its own classifier. This keeps the label space manageable and lets compound requests be represented. Claude 3.7 Sonnet annotates real agent transcripts using two methods: a single pass that gives a label plus an evidence-based explanation, and a two-stage variant in which the model critiques and possibly revises its first answer. Because real transcripts rarely contain topic changes, a controller-driven simulator generates extra dialogues with intent switches, chit-chat and clarifications. RoBERTa models are fine-tuned on these labels with the dialogue history as input. They beat zero-shot LLMs while running far faster. A hybrid setup, which sends low-confidence cases to an LLM restricted to RoBERTa's top candidates, gives the best accuracy. The work offers practical guidance for deploying accurate, low-latency multi-turn intent systems.

## Key points
- **Labelling quality:** The authors manually checked more than 5,500 test cases. Single-stage and two-stage labelling agreed on about 87% of them, and where they differed, the self-critique version was usually right.
- **Offline F1 by dimension:** Results are reported for intent, product and issue.
  - RoBERTa with two-stage labels plus simulated data: 74% / 73% / 69%.
  - Hybrid setup: 76% / 76% / 71%.
  - Claude Sonnet 3.7 zero-shot: 71% / 65% / 52%.
  - Nova Pro zero-shot: 45% / 56% / 38%.
- **Latency (P50):** RoBERTa takes 0.08 s, compared with 1.99 s for the hybrid, 1.98 s for Nova Pro and 5.93 s for Claude. That makes RoBERTa roughly 20–50× faster than zero-shot LLMs.
- **Online A/B test:** A fine-tuned RoBERTa-base was deployed for one month. It raised bot automation by 4.91% and positive customer responses by 7.89%, with production P99 latency of 0.20 s.
- **Limitations:** The data covers a limited set of products, locales and workflows, so results may not transfer to other domains. Noise in the automatic labels may also inflate offline scores.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

## Cite

Cite as: Biancen Xie, Kaiqi Bian, Jai Ranjan Singh Gusain, Manikandarajan Ramanathan and Raj Maragoud. 2026. Building Multi-turn Intent Classification with LLM-based Labeling. Proceedings of the Second Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNLP4U).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{xie2026building,
  title = {{Building Multi-turn Intent Classification with LLM-based Labeling}},
  author = {Biancen Xie, Kaiqi Bian, Jai Ranjan Singh Gusain, Manikandarajan Ramanathan and Raj Maragoud},
  booktitle = {Proceedings of the Second Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNLP4U)},
  year = {2026},
  doi = {10.18653/v1/2026.customnlp4u-1.8},
  url = {https://doi.org/10.18653/v1/2026.customnlp4u-1.8}
}
```

- [← previous Building Collaborative Speech Corpora for Low-Resource Languages: The Galician Dataset in Mozilla Common Voice](https://md.research.elamur.ai/free/en/papers/vladu-2026-building-collaborative-speech-corpora-low/)
- [next → Building the v4 of the Croatian National Corpus](https://md.research.elamur.ai/free/en/papers/tadic-2026-building-v4-croatian-national-corpus/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/25/)

## References

25 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Gaurav Arora, Shreya Jain, and Srujana Merugu. 2024. Intent detection in the age of LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track. [find](https://scholar.google.com/scholar?q=Gaurav+Arora%2C+Shreya+Jain%2C+and+Srujana+Merugu%2E+2024%2E+Intent+detection+in+the+age+of+LLMs%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Natural+Language+Processing%3A+Industry+Track%2E)
2. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Nanyun Raja, Shivang Gulati, Shubham Tan, et al. 2023. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651. [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Prakhar+Gupta%2C+Skyler+Hallinan%2C+Luyu+Gao%2C+Sarah+Wiegreffe%2C+Nanyun+Raja%2C+Shivang+Gulati%2C+Shubham+Tan%2C+et+al%2E+2023%2E+Self%2Drefine%3A+Iterative+refinement+with+self%2Dfeedback%2E+arXiv)
3. Krisztian Balog, Nolwenn Bernard, Saber Zerhoudi, and ChengXiang Zhai. 2025. Theory and toolkits for user simulation in the era of generative AI: User modeling, synthetic data generation, and system evaluation. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development… [find](https://scholar.google.com/scholar?q=Krisztian+Balog%2C+Nolwenn+Bernard%2C+Saber+Zerhoudi%2C+and+ChengXiang+Zhai%2E+2025%2E+Theory+and+toolkits+for+user+simulation+in+the+era+of+generative+AI%3A+User+modeling%2C+synthetic+data+generation%2C+and+system+e)
4. Subhadip Nandi, Neeraj Agrawal, Anshika Singh, and Priyanka Bhatt. 2024. Enhancing customer service chatbots with context-aware nlu through selective attention and multi-task learning. In Proceedings of the 8th International Conference on Data Science and Management of Data (CODS-COMAD 2024), pages… [find](https://scholar.google.com/scholar?q=Subhadip+Nandi%2C+Neeraj+Agrawal%2C+Anshika+Singh%2C+and+Priyanka+Bhatt%2E+2024%2E+Enhancing+customer+service+chatbots+with+context%2Daware+nlu+through+selective+attention+and+multi%2Dtask+learning%2E+In+Proceedings+)
5. Krisztian Balog and ChengXiang Zhai. 2025. User simulation in the era of generative AI: User modeling, synthetic data generation, and system evaluation. arXiv preprint arXiv:2501.04410. — [arXiv:2501.04410](https://arxiv.org/abs/2501.04410) [find](https://scholar.google.com/scholar?q=Krisztian+Balog+and+ChengXiang+Zhai%2E+2025%2E+User+simulation+in+the+era+of+generative+AI%3A+User+modeling%2C+synthetic+data+generation%2C+and+system+evaluation%2E+arXiv+preprint+arXiv%3A2501%2E04410%2E+%E2%80%94+%5BarXiv%3A2501%2E)
6. Haode Qi, Lin Pan, Atin Sood, Abhishek Shah, Ladislav Kunc, Mo Yu, and Saloni Potdar. 2021. Benchmarking commercial intent detection services with practice-driven evaluations. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human… [find](https://scholar.google.com/scholar?q=Haode+Qi%2C+Lin+Pan%2C+Atin+Sood%2C+Abhishek+Shah%2C+Ladislav+Kunc%2C+Mo+Yu%2C+and+Saloni+Potdar%2E+2021%2E+Benchmarking+commercial+intent+detection+services+with+practice%2Ddriven+evaluations%2E+In+Proceedings+of+the+20)
7. Lei Cui, Shaohan Huang, Furu Wei, Chuanqi Tan, Chaoqun Duan, and Ming Zhou. 2017. Superagent: A customer service chatbot for e-commerce websites. In Proceedings of ACL 2017, system demonstrations, pages 97–102. [find](https://scholar.google.com/scholar?q=Lei+Cui%2C+Shaohan+Huang%2C+Furu+Wei%2C+Chuanqi+Tan%2C+Chaoqun+Duan%2C+and+Ming+Zhou%2E+2017%2E+Superagent%3A+A+customer+service+chatbot+for+e%2Dcommerce+websites%2E+In+Proceedings+of+ACL+2017%2C+system+demonstrations%2C+pag)
8. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. Tiny-BERT: Distilling BERT for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4163–4174, Online. Association for Computational Lingui… [find](https://scholar.google.com/scholar?q=Xiaoqi+Jiao%2C+Yichun+Yin%2C+Lifeng+Shang%2C+Xin+Jiang%2C+Xiao+Chen%2C+Linlin+Li%2C+Fang+Wang%2C+and+Qun+Liu%2E+2020%2E+Tiny%2DBERT%3A+Distilling+BERT+for+natural+language+understanding%2E+In+Findings+of+the+Association+for+)
9. Libo Qin, Zhou Chen, Wanxiang Che, Hang Li, and Ting Liu. 2021. Knowing where to leverage: Context-aware graph convolutional network with an adaptive fusion layer for contextual spoken language understanding. [find](https://scholar.google.com/scholar?q=Libo+Qin%2C+Zhou+Chen%2C+Wanxiang+Che%2C+Hang+Li%2C+and+Ting+Liu%2E+2021%2E+Knowing+where+to+leverage%3A+Context%2Daware+graph+convolutional+network+with+an+adaptive+fusion+layer+for+contextual+spoken+language+unders)
10. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). arXiv preprint arXiv:2205.11916.
11. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://md.research.elamur.ai/free/en/papers/sanh-2019-distilbert-distilled-version-bert-smaller/). arXiv preprint arXiv:1910.01108.
12. Hsien-chin Lin, Nurul Lubis, Songbo Hu, Carel van Niekerk, Christian Geishauser, Michael Heck, Shutong Feng, and Milica Gasic. 2021. Domain-independent user simulation with transformers for task-oriented dialogue systems. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Dis… [find](https://scholar.google.com/scholar?q=Hsien%2Dchin+Lin%2C+Nurul+Lubis%2C+Songbo+Hu%2C+Carel+van+Niekerk%2C+Christian+Geishauser%2C+Michael+Heck%2C+Shutong+Feng%2C+and+Milica+Gasic%2E+2021%2E+Domain%2Dindependent+user+simulation+with+transformers+for+task%2Dorien)
13. Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Ye, and Steve Young. 2007. [Agenda-based user simulation for bootstrapping a POMDP dialogue system](https://md.research.elamur.ai/free/en/papers/schatzmann-2007-agenda-based-user-simulation-bootstrapping/). In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volum…
14. Junhua Liu, Yong Keat Tan, Bin Fu, and Kwan Hui Lim. 2024a. Balancing accuracy and efficiency in multi-turn intent classification for LLM-powered dialog systems in production. [find](https://scholar.google.com/scholar?q=Junhua+Liu%2C+Yong+Keat+Tan%2C+Bin+Fu%2C+and+Kwan+Hui+Lim%2E+2024a%2E+Balancing+accuracy+and+efficiency+in+multi%2Dturn+intent+classification+for+LLM%2Dpowered+dialog+systems+in+production%2E)
15. Matteo Antonio Senese, Giuseppe Rizzo, Mauro Dragoni, and Maurizio Morisio. 2020. MTSI-BERT: A session-aware knowledge-based conversational agent. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 717–725, Marseille, France. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Matteo+Antonio+Senese%2C+Giuseppe+Rizzo%2C+Mauro+Dragoni%2C+and+Maurizio+Morisio%2E+2020%2E+MTSI%2DBERT%3A+A+session%2Daware+knowledge%2Dbased+conversational+agent%2E+In+Proceedings+of+the+Twelfth+Language+Resources+and+)
16. Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. [Reflexion: Language agents with verbal reinforcement learning](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/). arXiv preprint arXiv:2303.11366.
17. Weiwei Sun, Shuyu Guo, Shuo Zhang, Pengjie Ren, Zhumin Chen, Maarten de Rijke, and Zhaochun Ren. 2022. Metaphorical user simulators for evaluating task-oriented dialogue systems. arXiv preprint arXiv:2204.00763. — [arXiv:2204.00763](https://arxiv.org/abs/2204.00763) [find](https://scholar.google.com/scholar?q=Weiwei+Sun%2C+Shuyu+Guo%2C+Shuo+Zhang%2C+Pengjie+Ren%2C+Zhumin+Chen%2C+Maarten+de+Rijke%2C+and+Zhaochun+Ren%2E+2022%2E+Metaphorical+user+simulators+for+evaluating+task%2Doriented+dialogue+systems%2E+arXiv+preprint+arXiv%3A)
18. Peiyao Wang, Joyce Fang, and Julia Reinspach. 2021. CS-BERT: a pretrained model for customer service dialogues. In Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI, pages 130– 142, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Peiyao+Wang%2C+Joyce+Fang%2C+and+Julia+Reinspach%2E+2021%2E+CS%2DBERT%3A+a+pretrained+model+for+customer+service+dialogues%2E+In+Proceedings+of+the+3rd+Workshop+on+Natural+Language+Processing+for+Conversational+AI%2C)
19. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Wenhui+Wang%2C+Furu+Wei%2C+Li+Dong%2C+Hangbo+Bao%2C+Nan+Yang%2C+and+Ming+Zhou%2E+2020%2E+MiniLM%3A+Deep+self%2Dattention+distillation+for+task%2Dagnostic+compression+of+pre%2Dtrained+transformers%2E+In+Advances+in+Neural+Inf)
20. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Advances in Neural Information Processing Systems.
21. Ting-Wei Wu, Ruolin Su, and Biing-Hwang Juang. 2021. [A context-aware hierarchical BERT fusion network for multi-turn dialog act detection](https://md.research.elamur.ai/free/en/papers/wu-2021-context-aware-hierarchical-bert-fusion/). In Proceedings of Interspeech 2021, pages 1239–1243.
22. Dian Yu, Luheng He, Yuan Zhang, Xinya Du, Panupong Pasupat, and Qi Li. 2021. Few-shot intent classification and slot filling with retrieved examples. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pag… [find](https://scholar.google.com/scholar?q=Dian+Yu%2C+Luheng+He%2C+Yuan+Zhang%2C+Xinya+Du%2C+Panupong+Pasupat%2C+and+Qi+Li%2E+2021%2E+Few%2Dshot+intent+classification+and+slot+filling+with+retrieved+examples%2E+In+Proceedings+of+the+2021+Conference+of+the+North)
23. Ziji Zhang, Michael Yang, Zhiyu Chen, Yingying Zhuang, Shu-Ting Pi, Qun Liu, Rajashekar Maragoud, Vy Nguyen, and Anurag Beniwal. 2025. REIC: RAG-enhanced intent classification at scale. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 1… [find](https://scholar.google.com/scholar?q=Ziji+Zhang%2C+Michael+Yang%2C+Zhiyu+Chen%2C+Yingying+Zhuang%2C+Shu%2DTing+Pi%2C+Qun+Liu%2C+Rajashekar+Maragoud%2C+Vy+Nguyen%2C+and+Anurag+Beniwal%2E+2025%2E+REIC%3A+RAG%2Denhanced+intent+classification+at+scale%2E+In+Proceedings)
24. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023. [A survey of l…](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)
25. Yunhua Zhou, Jiawei Hong, and Xipeng Qiu. 2023. Towards open environment intent prediction. In Findings of the Association for Computational Linguistics: ACL 2023, pages 2226–2240, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yunhua+Zhou%2C+Jiawei+Hong%2C+and+Xipeng+Qiu%2E+2023%2E+Towards+open+environment+intent+prediction%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL+2023%2C+pages+2226%E2%80%932240%2C+Toronto%2C+Canada%2E+As)

*Updated 2 November 2026*