---
title: "Continuously Learning Neural Dialogue Management — Su 2016"
description: "Task-oriented spoken dialogue systems need a dialogue manager that picks the next system action. 31 references, cited by 49 papers here."
authors: ["Pei-Hao Su", "Milica Gasic", "Nikola Mrksic", "Lina Rojas-Barahona", "Stefan Ultes", "David Vandyke", "Tsung-Hsien Wen", "Steve Young"]
year: 2016
venue: "arXiv:1606.02689"
source: "https://arxiv.org/abs/1606.02689"
doi: "10.48550/arxiv.1606.02689"
arxiv: "1606.02689"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/su-2016-continuously-learning-neural-dialogue-management/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/su-2016-continuously-learning-neural-dialogue-management/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/) / Continuously Learning Neural Dialogue Management

# Continuously Learning Neural Dialogue Management

**Authors:** [Pei-Hao Su](https://md.research.elamur.ai/free/en/authors/pei-hao-su/), [Milica Gasic](https://md.research.elamur.ai/free/en/authors/milica-gasic/), [Nikola Mrksic](https://md.research.elamur.ai/free/en/authors/nikola-mrksic/), Lina Rojas-Barahona, [Stefan Ultes](https://md.research.elamur.ai/free/en/authors/stefan-ultes/), [David Vandyke](https://md.research.elamur.ai/free/en/authors/david-vandyke/), [Tsung-Hsien Wen](https://md.research.elamur.ai/free/en/authors/tsung-hsien-wen/) and [Steve Young](https://md.research.elamur.ai/free/en/authors/steve-young/) · **Year:** 2016 · **Venue:** arXiv:1606.02689 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1606.02689) · [PDF](https://arxiv.org/pdf/1606.02689) · [DOI 10.48550/arxiv.1606.02689](https://doi.org/10.48550/arxiv.1606.02689) · [arXiv 1606.02689](https://arxiv.org/abs/1606.02689) · [OpenAlex](https://openalex.org/W2417401578)

**Citations:** 88 citations (OpenAlex, 2026-10-14) · [49](https://md.research.elamur.ai/free/en/papers/su-2016-continuously-learning-neural-dialogue-management/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Task-oriented spoken dialogue systems need a dialogue manager that picks the next system action. Reinforcement learning alone behaves poorly early in training. Supervised imitation of logged dialogues ignores long-term consequences and may not cover all situations. The authors combine both approaches in one neural policy network that maps the belief state to a full master dialogue action (dialogue act, query slot, offered slots).

The network is first trained by supervision on 720 dialogues from the Cambridge restaurant domain, collected through Amazon Mechanical Turk. The same model, with no change to its architecture, is then refined with policy-gradient RL using episodic Natural Actor-Critic (eNAC) and experience replay. This refinement runs against a simulated user and against paid human callers.

The supervised model already predicts actions accurately. RL raises dialogue success further, most clearly when speech understanding is noisier than in the training data. The work shows that a small corpus can bootstrap a usable policy, which then adapts online to new conditions, in a training scheme similar to AlphaGo's.

## Key points
- **Policy network design:** one tanh hidden layer of 32 units and about 2,600 weights in total.
  - Two softmax outputs: DiaAct (request, offer, confirm, select, bye) and Query (food, pricerange, area, none).
  - Six sigmoid outputs for the Offer slots.
  - The domain covers about 150 venues, and the model was implemented in Theano.
- **Supervised phase:** trained on a 4:1:1 train/validation/test split using Adagrad and early stopping. Weighted F-1 on the test set was:
  - DiaAct: 97.73
  - Query: 87.39
  - Offer: 92.51
- **RL setup:** eNAC natural gradient with a replay pool of 2,000 and mini-batches of 32. Exploration ε was set to 0.1. Each dialogue's return was 20 × success minus the number of turns, normalised to [0, 1], with a cap of 30 turns.
- **Simulation results:** over 6,000 simulated dialogues, success improved by 1–8% across semantic error rates. The largest gains came at high error rates, where the policy learned to confirm more often.
- **Human evaluation:** after 200 RL training dialogues with AMT users, 110 evaluation dialogues per policy were collected. SL+RL beat SL alone:
  - Success: 98.2% vs 94.5%
  - Quality: 4.04 vs 3.97 (on a 0–5 scale)

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Cambridge](https://md.research.elamur.ai/free/en/universities/university-of-cambridge/) — University, Cambridge, United Kingdom · 256 works · 123,373 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🗨️ Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) · 654 works

## Cite

Cite as: Pei-Hao Su, Milica Gasic, Nikola Mrksic, Lina Rojas-Barahona, Stefan Ultes, David Vandyke, Tsung-Hsien Wen and Steve Young. 2016. Continuously Learning Neural Dialogue Management. arXiv:1606.02689.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{su2016continuously,
  title = {{Continuously Learning Neural Dialogue Management}},
  author = {Pei-Hao Su, Milica Gasic, Nikola Mrksic, Lina Rojas-Barahona, Stefan Ultes, David Vandyke, Tsung-Hsien Wen and Steve Young},
  year = {2016},
  doi = {10.48550/arxiv.1606.02689},
  eprint = {1606.02689},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1606.02689}
}
```

- [← previous The Role of Context Types and Dimensionality in Learning Word Embeddings](https://md.research.elamur.ai/free/en/papers/melamud-2016-role-context-types-dimensionality-learning/)
- [next → SemEval-2016 Task 7: Determining Sentiment Intensity of English and Arabic Phrases](https://md.research.elamur.ai/free/en/papers/kiritchenko-2016-semeval-2016-task-7-determining/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/)

## References

31 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Shun-Ichi Amari. 1998. Natural gradient works efficiently in learning. Neural computation, 10(2):251–276. [find](https://scholar.google.com/scholar?q=Shun%2DIchi+Amari%2E+1998%2E+Natural+gradient+works+efficiently+in+learning%2E+Neural+computation%2C+10%282%29%3A251%E2%80%93276%2E)
2. Antoine Bordes and Jason Weston. 2016. Learning end-to-end goal-oriented dialog. arXiv preprint: 1605.07683, May. [find](https://scholar.google.com/scholar?q=Antoine+Bordes+and+Jason+Weston%2E+2016%2E+Learning+end%2Dto%2Dend+goal%2Doriented+dialog%2E+arXiv+preprint%3A+1605%2E07683%2C+May%2E)
3. Heriberto Cuayáhuitl, Simon Keizer, and Oliver Lemon. 2015. Strategic dialogue management via deep reinforcement learning. arXiv preprint arXiv:1511.08099. — [arXiv:1511.08099](https://arxiv.org/abs/1511.08099) [find](https://scholar.google.com/scholar?q=Heriberto+Cuaya%CC%81huitl%2C+Simon+Keizer%2C+and+Oliver+Lemon%2E+2015%2E+Strategic+dialogue+management+via+deep+reinforcement+learning%2E+arXiv+preprint+arXiv%3A1511%2E08099%2E+%E2%80%94+%5BarXiv%3A1511%2E08099%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F)
4. Nils Dahlbäck, Arne Jönsson, and Lars Ahrenberg. 1993. Wizard of oz studies: why and how. In Proc of Intelligent user interfaces. [find](https://scholar.google.com/scholar?q=Nils+Dahlba%CC%88ck%2C+Arne+Jo%CC%88nsson%2C+and+Lars+Ahrenberg%2E+1993%2E+Wizard+of+oz+studies%3A+why+and+how%2E+In+Proc+of+Intelligent+user+interfaces%2E)
5. Lucie Daubigney, Matthieu Geist, Senthilkumar Chandramohan, and Olivier Pietquin. 2014. A comprehensive reinforcement learning framework for dialogue management optimisation. Journal of Selected Topics in Signal Processing, 6(8). [find](https://scholar.google.com/scholar?q=Lucie+Daubigney%2C+Matthieu+Geist%2C+Senthilkumar+Chandramohan%2C+and+Olivier+Pietquin%2E+2014%2E+A+comprehensive+reinforcement+learning+framework+for+dialogue+management+optimisation%2E+Journal+of+Selected+Topic)
6. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 12:2121–2159. [find](https://scholar.google.com/scholar?q=John+Duchi%2C+Elad+Hazan%2C+and+Yoram+Singer%2E+2011%2E+Adaptive+subgradient+methods+for+online+learning+and+stochastic+optimization%2E+The+Journal+of+Machine+Learning+Research%2C+12%3A2121%E2%80%932159%2E)
7. Layla El Asri, Romain Laroche, and Olivier Pietquin. 2014. Task completion transfer learning for reward inference. In Proc of MLIS. [find](https://scholar.google.com/scholar?q=Layla+El+Asri%2C+Romain+Laroche%2C+and+Olivier+Pietquin%2E+2014%2E+Task+completion+transfer+learning+for+reward+inference%2E+In+Proc+of+MLIS%2E)
8. Milica Gašić and Steve Young. 2014. Gaussian processes for pomdp-based dialogue manager optimization. TASLP, 22(1):28–40. [find](https://scholar.google.com/scholar?q=Milica+Gas%CC%8Cic%CC%81+and+Steve+Young%2E+2014%2E+Gaussian+processes+for+pomdp%2Dbased+dialogue+manager+optimization%2E+TASLP%2C+22%281%29%3A28%E2%80%9340%2E)
9. Milica Gašić, Catherine Breslin, Matthew Henderson, Dongho Kim, Martin Szummer, Blaise Thomson, Pirros Tsiakoulis, and Steve J. Young. 2013. On-line policy optimisation of bayesian spoken dialogue systems via human interaction. In Proc of ICASSP. [find](https://scholar.google.com/scholar?q=Milica+Gas%CC%8Cic%CC%81%2C+Catherine+Breslin%2C+Matthew+Henderson%2C+Dongho+Kim%2C+Martin+Szummer%2C+Blaise+Thomson%2C+Pirros+Tsiakoulis%2C+and+Steve+J%2E+Young%2E+2013%2E+On%2Dline+policy+optimisation+of+bayesian+spoken+dialogue+s)
10. M. Henderson, B. Thomson, and J. Williams. 2014a. The Second Dialog State Tracking Challenge. In Proc of SIGdial. [find](https://scholar.google.com/scholar?q=M%2E+Henderson%2C+B%2E+Thomson%2C+and+J%2E+Williams%2E+2014a%2E+The+Second+Dialog+State+Tracking+Challenge%2E+In+Proc+of+SIGdial%2E)
11. M. Henderson, B. Thomson, and S. J. Young. 2014b. [Word-based Dialog State Tracking with Recurrent Neural Networks](https://md.research.elamur.ai/free/en/papers/henderson-2014-word-based-dialog-state-tracking/). In Proc of SIGdial.
12. Filip Jurčı́ček, Blaise Thomson, and Steve Young. 2011. Natural actor and belief critic: Reinforcement algorithm for learning parameters of dialogue systems modelled as pomdps. ACM TSLP, 7(3):6. [find](https://scholar.google.com/scholar?q=Filip+Jurc%CC%8C%C4%B1%CC%81c%CC%8Cek%2C+Blaise+Thomson%2C+and+Steve+Young%2E+2011%2E+Natural+actor+and+belief+critic%3A+Reinforcement+algorithm+for+learning+parameters+of+dialogue+systems+modelled+as+pomdps%2E+ACM+TSLP%2C+7%283%29%3A6%2E)
13. John F. Kelley. 1984. An iterative design methodology for user-friendly natural language office information applications. ACM Transaction on Information Systems. [find](https://scholar.google.com/scholar?q=John+F%2E+Kelley%2E+1984%2E+An+iterative+design+methodology+for+user%2Dfriendly+natural+language+office+information+applications%2E+ACM+Transaction+on+Information+Systems%2E)
14. Esther Levin and Roberto Pieraccini. 1997. A stochastic model of computer-human interaction for learning dialogue strategies. Eurospeech. [find](https://scholar.google.com/scholar?q=Esther+Levin+and+Roberto+Pieraccini%2E+1997%2E+A+stochastic+model+of+computer%2Dhuman+interaction+for+learning+dialogue+strategies%2E+Eurospeech%2E)
15. Long-Ji Lin. 1992. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 8(3-4):293–321. [find](https://scholar.google.com/scholar?q=Long%2DJi+Lin%2E+1992%2E+Self%2Dimproving+reactive+agents+based+on+reinforcement+learning%2C+planning+and+teaching%2E+Machine+learning%2C+8%283%2D4%29%3A293%E2%80%93321%2E)
16. Nikola Mrkšić, Diarmuid Ó Séaghdha, Blaise Thomson, Milica Gašić, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2015. Multi-domain Dialog State Tracking using Recurrent Neural Networks. In Proc of ACL. [find](https://scholar.google.com/scholar?q=Nikola+Mrks%CC%8Cic%CC%81%2C+Diarmuid+O%CC%81+Se%CC%81aghdha%2C+Blaise+Thomson%2C+Milica+Gas%CC%8Cic%CC%81%2C+Pei%2DHao+Su%2C+David+Vandyke%2C+Tsung%2DHsien+Wen%2C+and+Steve+Young%2E+2015%2E+Multi%2Ddomain+Dialog+State+Tracking+using+Recurrent+Neural+Net)
17. Jan Peters and Stefan Schaal. 2006. Policy gradient methods for robotics. In IEEE RSJ. [find](https://scholar.google.com/scholar?q=Jan+Peters+and+Stefan+Schaal%2E+2006%2E+Policy+gradient+methods+for+robotics%2E+In+IEEE+RSJ%2E)
18. Nicholas Roy, Joelle Pineau, and Sebastian Thrun. 2000. Spoken dialogue management using probabilistic reasoning. In Proc of SigDial. [find](https://scholar.google.com/scholar?q=Nicholas+Roy%2C+Joelle+Pineau%2C+and+Sebastian+Thrun%2E+2000%2E+Spoken+dialogue+management+using+probabilistic+reasoning%2E+In+Proc+of+SigDial%2E)
19. Jost Schatzmann, Karl Weilhammer, Matt Stuttle, and Steve Young. 2006. A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies. The knowledge engineering review, 21(02):97–126. [find](https://scholar.google.com/scholar?q=Jost+Schatzmann%2C+Karl+Weilhammer%2C+Matt+Stuttle%2C+and+Steve+Young%2E+2006%2E+A+survey+of+statistical+user+simulation+techniques+for+reinforcement%2Dlearning+of+dialogue+management+strategies%2E+The+knowledge+en)
20. John Schulman, Sergey Levine, Philipp Moritz, Michael I Jordan, and Pieter Abbeel. 2015. Trust region policy optimization. Proc of ICML. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Sergey+Levine%2C+Philipp+Moritz%2C+Michael+I+Jordan%2C+and+Pieter+Abbeel%2E+2015%2E+Trust+region+policy+optimization%2E+Proc+of+ICML%2E)
21. Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2015. [Hierarchical neural network generative models for movie dialogues](https://md.research.elamur.ai/free/en/papers/serban-2015-building-end-end-dialogue-systems/). arXiv preprint arXiv:1507.04808.
22. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. 2016. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484– 489. [find](https://scholar.google.com/scholar?q=David+Silver%2C+Aja+Huang%2C+Chris+J+Maddison%2C+Arthur+Guez%2C+Laurent+Sifre%2C+George+Van+Den+Driessche%2C+Julian+Schrittwieser%2C+Ioannis+Antonoglou%2C+Veda+Panneershelvam%2C+Marc+Lanctot%2C+et+al%2E+2016%2E+Mastering+the)
23. Pei-Hao Su, David Vandyke, Milica Gašić, Dongho Kim, Nikola Mrkšić, Tsung-Hsien Wen, and Steve Young. 2015. Learning from real users: Rating dialogue success with neural networks for reinforcement learning in spoken dialogue systems. In Proc of Interspeech. [find](https://scholar.google.com/scholar?q=Pei%2DHao+Su%2C+David+Vandyke%2C+Milica+Gas%CC%8Cic%CC%81%2C+Dongho+Kim%2C+Nikola+Mrks%CC%8Cic%CC%81%2C+Tsung%2DHsien+Wen%2C+and+Steve+Young%2E+2015%2E+Learning+from+real+users%3A+Rating+dialogue+success+with+neural+networks+for+reinforcement)
24. Pei-Hao Su, Milica Gašić, Nikola Mrkšić, Lina Rojas-Barahona, Stefan Ultes, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. On-line active reward learning for policy optimisation in spoken dialogue systems. In Proc of ACL. [find](https://scholar.google.com/scholar?q=Pei%2DHao+Su%2C+Milica+Gas%CC%8Cic%CC%81%2C+Nikola+Mrks%CC%8Cic%CC%81%2C+Lina+Rojas%2DBarahona%2C+Stefan+Ultes%2C+David+Vandyke%2C+Tsung%2DHsien+Wen%2C+and+Steve+Young%2E+2016%2E+On%2Dline+active+reward+learning+for+policy+optimisation+in+spoken+)
25. Theano Development Team. 2016. [Theano: A Python framework for fast computation of mathematical expressions](https://md.research.elamur.ai/free/en/papers/the-theano-development-team-2016-theano-python-framework-fast-computation/). arXiv eprints, abs/1605.02688.
26. David Vandyke, Pei-Hao Su, Milica Gašić, Nikola Mrkšić, Tsung-Hsien Wen, and Steve Young. 2015. Multi-domain dialogue success classifiers for policy training. In ASRU. [find](https://scholar.google.com/scholar?q=David+Vandyke%2C+Pei%2DHao+Su%2C+Milica+Gas%CC%8Cic%CC%81%2C+Nikola+Mrks%CC%8Cic%CC%81%2C+Tsung%2DHsien+Wen%2C+and+Steve+Young%2E+2015%2E+Multi%2Ddomain+dialogue+success+classifiers+for+policy+training%2E+In+ASRU%2E)
27. Oriol Vinyals and Quoc Le. 2015. [A neural conversational model](https://md.research.elamur.ai/free/en/papers/vinyals-2015-neural-conversational-model/). arXiv preprint arXiv:1506.05869.
28. Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić, Pei-Hao Su, David Vandyke, and Steve Young. 2015. [Semantically conditioned lstm-based natural language generation for spoken dialogue systems](https://md.research.elamur.ai/free/en/papers/wen-2015-semantically-conditioned-lstm-based-natural/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Associat…
29. Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić, Lina M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, David Vandyke, and Steve Young. 2016. [A network-based end-to-end trainable task-oriented dialogue system](https://md.research.elamur.ai/free/en/papers/wen-2017-network-based-end-end-trainable/). arXiv preprint: 1604.04562, April.
30. Jason D. Williams and Steve Young. 2007. Partially observable Markov decision processes for spoken dialog systems. Computer Speech and Language, 21(2):393–422. [find](https://scholar.google.com/scholar?q=Jason+D%2E+Williams+and+Steve+Young%2E+2007%2E+Partially+observable+Markov+decision+processes+for+spoken+dialog+systems%2E+Computer+Speech+and+Language%2C+21%282%29%3A393%E2%80%93422%2E)
31. Steve Young, Milica Gašic, Blaise Thomson, and Jason Williams. 2013. Pomdp-based statistical spoken dialogue systems: a review. In Proc of IEEE, volume 99, pages 1–20. [find](https://scholar.google.com/scholar?q=Steve+Young%2C+Milica+Gas%CC%8Cic%2C+Blaise+Thomson%2C+and+Jason+Williams%2E+2013%2E+Pomdp%2Dbased+statistical+spoken+dialogue+systems%3A+a+review%2E+In+Proc+of+IEEE%2C+volume+99%2C+pages+1%E2%80%9320%2E)

*Updated 2 November 2026*