---
title: "Rethinking Action Spaces for Reinforcement Learning in"
description: "The authors propose Latent Action Reinforcement Learning (LaRL). 48 references, cited by 56 papers here."
authors: ["Tiancheng Zhao", "Kaige Xie", "Maxine Eskenazi"]
year: 2019
venue: "Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)"
source: "https://aclanthology.org/N19-1123/"
doi: "10.18653/v1/n19-1123"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhao-2019-rethinking-action-spaces-reinforcement-learning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhao-2019-rethinking-action-spaces-reinforcement-learning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models

# Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models

**Authors:** [Tiancheng Zhao](https://md.research.elamur.ai/free/en/authors/tiancheng-zhao/), [Kaige Xie](https://md.research.elamur.ai/free/en/authors/kaige-xie/) and [Maxine Eskenazi](https://md.research.elamur.ai/free/en/authors/maxine-eskenazi/) · **Year:** 2019 · **Venue:** [Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2019](https://md.research.elamur.ai/free/en/venues/naacl/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/N19-1123/) · [PDF](https://aclanthology.org/N19-1123.pdf) · [DOI 10.18653/v1/n19-1123](https://doi.org/10.18653/v1/n19-1123) · [OpenAlex](https://openalex.org/W2915295540)

**Citations:** 112 citations (OpenAlex, 2026-10-14) · [56](https://md.research.elamur.ai/free/en/papers/zhao-2019-rethinking-action-spaces-reinforcement-learning/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary

Applying reinforcement learning to end-to-end dialog agents is hard because the action space is usually either a set of handcrafted dialog acts, which does not scale to complex domains, or the full output vocabulary. Treating every generated word as an action, as word-level RL does, tends to corrupt the language a model produces and makes credit assignment slow over long dialogs. The authors propose Latent Action Reinforcement Learning (LaRL). A latent variable z sits between the dialog context and the response, and policy gradients are applied to z rather than to individual words. The decoder is left unchanged during RL, so strategy is separated from surface generation. They compare Gaussian and categorical latent actions, two pre-training objectives based on stochastic variational inference (a full ELBO and a regularized "Lite" variant), and an attention fusion mechanism for discrete codes. On DealOrNoDeal and MultiWoz, LaRL beats word-level policy gradient baselines without degrading language quality. It also sets a new success-rate state of the art on MultiWoz. This makes the approach a promising basis for unsupervised action abstraction in dialog agents.

## Key points

- **Shorter RL horizon:** LaRL factorizes generation as p(x|c) = p(x|z)p(z|c) and runs REINFORCE only on the encoder. This cuts the horizon from T·U word steps to T turns.
- **MultiWoz result:** The method gains 18.2% absolute over the previous state of the art. Word-level RL with RL:SL=off learned to spam entity tokens to inflate the success metric, whereas LiteAttnCat produced fluent, informative replies.
- **Categorical beats Gaussian:** Categorical latent actions (M independent K-way variables, giving K^M combinations) consistently outperformed Gaussian ones. REINFORCE on Gaussian latents was unstable and often diverged.
- **Lite objective and regularization:** The Lite objective outperformed the full ELBO for pre-training, which the authors attribute to exposure bias in the latent space. Regularizing the latent space with β = 0.01 instead of 0 was needed for sufficient exploration during RL.
- **Resources:** Data and code are released at github.com/snakeztc/NeuralDialog-LaRL. The paper also introduces LCR curves for analyzing the trade-off between reward and language quality.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 365 works · 158,048 citations
- [Shanghai Jiao Tong University](https://md.research.elamur.ai/free/en/universities/shanghai-jiao-tong-university/) — University, Shanghai, China · 63 works · 15,047 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🗨️ Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) · 401 works

## Cite

Cite as: Tiancheng Zhao, Kaige Xie and Maxine Eskenazi. 2019. Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhao2019rethinking,
  title = {{Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models}},
  author = {Tiancheng Zhao, Kaige Xie and Maxine Eskenazi},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1123},
  url = {https://doi.org/10.18653/v1/n19-1123}
}
```

- [← previous Curriculum Learning for Domain Adaptation in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/zhang-2019-curriculum-learning-domain-adaptation-neural/)
- [next → Target-Guided Open-Domain Conversation](https://md.research.elamur.ai/free/en/papers/tang-2019-target-guided-open-domain-conversation/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/)

## References

48 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. 2018. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language P… [find](https://scholar.google.com/scholar?q=Pawe%C5%82+Budzianowski%2C+Tsung%2DHsien+Wen%2C+Bo%2DHsiang+Tseng%2C+In%CC%83igo+Casanueva%2C+Stefan+Ultes%2C+Osman+Ramadan%2C+and+Milica+Gasic%2E+2018%2E+Multiwoz%2Da+large%2Dscale+multi%2Ddomain+wizard%2Dof%2Doz+dataset+for+task%2Doriented+)
2. Kris Cao and Stephen Clark. 2017. Latent variable dialogue models and their diversity. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, volume 2, pages 182–187. [find](https://scholar.google.com/scholar?q=Kris+Cao+and+Stephen+Clark%2E+2017%2E+Latent+variable+dialogue+models+and+their+diversity%2E+In+Proceedings+of+the+15th+Conference+of+the+European+Chapter+of+the+Association+for+Computational+Linguistics%3A+V)
3. Yun-Nung Chen, William Yang Wang, and Alexander I Rudnicky. 2013. Unsupervised induction and filling of semantic slots for spoken dialogue systems using frame-semantic parsing. In Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on, pages 120–125. IEEE. [find](https://scholar.google.com/scholar?q=Yun%2DNung+Chen%2C+William+Yang+Wang%2C+and+Alexander+I+Rudnicky%2E+2013%2E+Unsupervised+induction+and+filling+of+semantic+slots+for+spoken+dialogue+systems+using+frame%2Dsemantic+parsing%2E+In+Automatic+Speech+Rec)
4. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078. [find](https://scholar.google.com/scholar?q=Kyunghyun+Cho%2C+Bart+Van+Merrie%CC%88nboer%2C+Caglar+Gulcehre%2C+Dzmitry+Bahdanau%2C+Fethi+Bougares%2C+Holger+Schwenk%2C+and+Yoshua+Bengio%2E+2014%2E+Learning+phrase+representations+using+rnn+encoder%2Ddecoder+for+statisti)
5. Abhishek Das, Satwik Kottur, José MF Moura, Stefan Lee, and Dhruv Batra. 2017. Learning cooperative visual dialog agents with deep reinforcement learning. In Computer Vision (ICCV), 2017 IEEE International Conference on, pages 2970–2979. IEEE. [find](https://scholar.google.com/scholar?q=Abhishek+Das%2C+Satwik+Kottur%2C+Jose%CC%81+MF+Moura%2C+Stefan+Lee%2C+and+Dhruv+Batra%2E+2017%2E+Learning+cooperative+visual+dialog+agents+with+deep+reinforcement+learning%2E+In+Computer+Vision+%28ICCV%29%2C+2017+IEEE+Interna)
6. Bhuwan Dhingra, Lihong Li, Xiujun Li, Jianfeng Gao, Yun-Nung Chen, Faisal Ahmed, and Li Deng. 2017. [Towards end-to-end reinforcement learning of dialogue agents for information access](https://md.research.elamur.ai/free/en/papers/dhingra-2017-end-end-reinforcement-learning-dialogue/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),…
7. Milica Gasic and Steve Young. 2014. Gaussian processes for pomdp-based dialogue manager optimization. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 22(1):28–40. [find](https://scholar.google.com/scholar?q=Milica+Gasic+and+Steve+Young%2E+2014%2E+Gaussian+processes+for+pomdp%2Dbased+dialogue+manager+optimization%2E+IEEE%2FACM+Transactions+on+Audio%2C+Speech%2C+and+Language+Processing%2C+22%281%29%3A28%E2%80%9340%2E)
8. Evan Greensmith, Peter L Bartlett, and Jonathan Baxter. 2004. Variance reduction techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research, 5(Nov):1471–1530. [find](https://scholar.google.com/scholar?q=Evan+Greensmith%2C+Peter+L+Bartlett%2C+and+Jonathan+Baxter%2E+2004%2E+Variance+reduction+techniques+for+gradient+estimates+in+reinforcement+learning%2E+Journal+of+Machine+Learning+Research%2C+5%28Nov%29%3A1471%E2%80%931530%2E)
9. He He, Derek Chen, Anusha Balakrishnan, and Percy Liang. 2018. [Decoupling strategy and generation in negotiation dialogues](https://md.research.elamur.ai/free/en/papers/he-2018-decoupling-strategy-generation-negotiation-dialogues/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2333–2343.
10. Matthew Henderson, Blaise Thomson, and Steve Young. 2014. [Word-based dialog state tracking with recurrent neural networks](https://md.research.elamur.ai/free/en/papers/henderson-2014-word-based-dialog-state-tracking/). In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 292– 299.
11. Eric Jang, Shixiang Gu, and Ben Poole. 2016. [Categorical reparameterization with gumbel-softmax](https://md.research.elamur.ai/free/en/papers/jang-2016-categorical-reparameterization-gumbel-softmax/). arXiv preprint arXiv:1611.01144.
12. Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. 1996. Reinforcement learning: A survey. Journal of artificial intelligence research, 4:237–285. [find](https://scholar.google.com/scholar?q=Leslie+Pack+Kaelbling%2C+Michael+L+Littman%2C+and+Andrew+W+Moore%2E+1996%2E+Reinforcement+learning%3A+A+survey%2E+Journal+of+artificial+intelligence+research%2C+4%3A237%E2%80%93285%2E)
13. Diederik P Kingma and Max Welling. 2013. [Autoencoding variational bayes](https://md.research.elamur.ai/free/en/papers/kingma-2013-auto-encoding-variational-bayes/). arXiv preprint arXiv:1312.6114.
14. Satwik Kottur, José Moura, Stefan Lee, and Dhruv Batra. 2017. Natural language does not emerge nat-urallyin multi-agent dialog. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2962–2967. [find](https://scholar.google.com/scholar?q=Satwik+Kottur%2C+Jose%CC%81+Moura%2C+Stefan+Lee%2C+and+Dhruv+Batra%2E+2017%2E+Natural+language+does+not+emerge+nat%2Durallyin+multi%2Dagent+dialog%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Natural+L)
15. John Langford and Tong Zhang. 2008. The epoch-greedy algorithm for multi-armed bandits with side information. In Advances in neural information processing systems, pages 817–824. [find](https://scholar.google.com/scholar?q=John+Langford+and+Tong+Zhang%2E+2008%2E+The+epoch%2Dgreedy+algorithm+for+multi%2Darmed+bandits+with+side+information%2E+In+Advances+in+neural+information+processing+systems%2C+pages+817%E2%80%93824%2E)
16. Sungjin Lee. 2013. Structured discriminative model for dialog state tracking. In Proceedings of the SIG-DIAL 2013 Conference, pages 442–451. [find](https://scholar.google.com/scholar?q=Sungjin+Lee%2E+2013%2E+Structured+discriminative+model+for+dialog+state+tracking%2E+In+Proceedings+of+the+SIG%2DDIAL+2013+Conference%2C+pages+442%E2%80%93451%2E)
17. Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. 2017. [Deal or no deal? end-to-end learning of negotiation dialogues](https://md.research.elamur.ai/free/en/papers/lewis-2017-deal-or-no-deal-end/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2443–2453.
18. Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016. [Deep reinforcement learning for dialogue generation](https://md.research.elamur.ai/free/en/papers/li-2016-deep-reinforcement-learning-dialogue-generation/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1192– 1202.
19. Bing Liu and Ian Lane. 2017. An end-to-end trainable neural network model with belief tracking for task-oriented dialog. [find](https://scholar.google.com/scholar?q=Bing+Liu+and+Ian+Lane%2E+2017%2E+An+end%2Dto%2Dend+trainable+neural+network+model+with+belief+tracking+for+task%2Doriented+dialog%2E)
20. Proc. Interspeech 2017, pages 2506–2510. [find](https://scholar.google.com/scholar?q=Proc%2E+Interspeech+2017%2C+pages+2506%E2%80%932510%2E)
21. Marilyn A. Walker. 2000. An application of reinforcement learning to dialogue strategy selection in a spoken dialogue system for email. Journal of Artificial Intelligence Research, pages 387–416. [find](https://scholar.google.com/scholar?q=Marilyn+A%2E+Walker%2E+2000%2E+An+application+of+reinforcement+learning+to+dialogue+strategy+selection+in+a+spoken+dialogue+system+for+email%2E+Journal+of+Artificial+Intelligence+Research%2C+pages+387%E2%80%93416%2E)
22. Thang Luong, Hieu Pham, and Christopher D Manning. 2015. [Effective approaches to attention-based neural machine translation](https://md.research.elamur.ai/free/en/papers/luong-2015-effective-approaches-attention-based-neural/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412–1421.
23. Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Lina M Rojas-Barahona, Pei-Hao Su, Stefan Ultes, David Vandyke, and Steve Young. 2016. A network-based end-to-end trainable task-oriented dialogue system. arXiv preprint arXiv:1604.04562. [find](https://scholar.google.com/scholar?q=Tsung%2DHsien+Wen%2C+Milica+Gasic%2C+Nikola+Mrksic%2C+Lina+M+Rojas%2DBarahona%2C+Pei%2DHao+Su%2C+Stefan+Ultes%2C+David+Vandyke%2C+and+Steve+Young%2E+2016%2E+A+network%2Dbased+end%2Dto%2Dend+trainable+task%2Doriented+dialogue+system%2E)
24. Igor Mordatch and Pieter Abbeel. 2017. Emergence of grounded compositional language in multi-agent populations. arXiv preprint arXiv:1703.04908. [find](https://scholar.google.com/scholar?q=Igor+Mordatch+and+Pieter+Abbeel%2E+2017%2E+Emergence+of+grounded+compositional+language+in+multi%2Dagent+populations%2E+arXiv+preprint+arXiv%3A1703%2E04908%2E)
25. Tsung-Hsien Wen, Yishu Miao, Phil Blunsom, and Steve Young. 2017. Latent intention dialogue models. In International Conference on Machine Learning, pages 3732–3741. [find](https://scholar.google.com/scholar?q=Tsung%2DHsien+Wen%2C+Yishu+Miao%2C+Phil+Blunsom%2C+and+Steve+Young%2E+2017%2E+Latent+intention+dialogue+models%2E+In+International+Conference+on+Machine+Learning%2C+pages+3732%E2%80%933741%2E)
26. Antoine Raux, Brian Langner, Dan Bohus, Alan W Black, and Maxine Eskenazi. 2005. Lets go public! taking a spoken dialog system to the real world. In in Proc. of Interspeech 2005. Citeseer. [find](https://scholar.google.com/scholar?q=Antoine+Raux%2C+Brian+Langner%2C+Dan+Bohus%2C+Alan+W+Black%2C+and+Maxine+Eskenazi%2E+2005%2E+Lets+go+public%21+taking+a+spoken+dialog+system+to+the+real+world%2E+In+in+Proc%2E+of+Interspeech+2005%2E+Citeseer%2E)
27. Liliang Ren, Kaige Xie, Lu Chen, and Kai Yu. 2018. [Towards universal dialogue state tracking](https://md.research.elamur.ai/free/en/papers/ren-2018-universal-dialogue-state-tracking/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2780– 2786.
28. Iulian V Serban, Chinnadhurai Sankar, Mathieu Germain, Saizheng Zhang, Zhouhan Lin, Sandeep Subramanian, Taesup Kim, Michael Pieper, Sarath Chandar, Nan Rosemary Ke, et al. 2017a. A deep reinforcement learning chatbot. arXiv preprint arXiv:1709.02349. [find](https://scholar.google.com/scholar?q=Iulian+V+Serban%2C+Chinnadhurai+Sankar%2C+Mathieu+Germain%2C+Saizheng+Zhang%2C+Zhouhan+Lin%2C+Sandeep+Subramanian%2C+Taesup+Kim%2C+Michael+Pieper%2C+Sarath+Chandar%2C+Nan+Rosemary+Ke%2C+et+al%2E+2017a%2E+A+deep+reinforcement)
29. Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016. [Building end-to-end dialogue systems using generative hierarchical neural network models](https://md.research.elamur.ai/free/en/papers/serban-2015-building-end-end-dialogue-systems/). In Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI-16).
30. Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017b. [A hierarchical latent variable encoder-decoder model for generating dialogues](https://md.research.elamur.ai/free/en/papers/serban-2016-hierarchical-latent-variable-encoder-decoder/). In Thirty-First AAAI Conference on Artificial Intelligence.
31. Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. [A neural network approach to context-sensitive generation of conversational responses](https://md.research.elamur.ai/free/en/papers/sordoni-2015-neural-network-approach-context-sensitive/). In Proceedings of the 2015 Conference of the North American Chapte…
32. Pei-Hao Su, Paweł Budzianowski, Stefan Ultes, Milica Gasic, and Steve Young. 2017. Sample-efficient actor-critic reinforcement learning with supervised data for dialogue management. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 147–157. [find](https://scholar.google.com/scholar?q=Pei%2DHao+Su%2C+Pawe%C5%82+Budzianowski%2C+Stefan+Ultes%2C+Milica+Gasic%2C+and+Steve+Young%2E+2017%2E+Sample%2Defficient+actor%2Dcritic+reinforcement+learning+with+supervised+data+for+dialogue+management%2E+In+Proceedings+of+)
33. Oriol Vinyals and Quoc Le. 2015. [A neural conversational model](https://md.research.elamur.ai/free/en/papers/vinyals-2015-neural-conversational-model/). arXiv preprint arXiv:1506.05869.
34. Jason D Williams, Kavosh Asadi, and Geoffrey Zweig. 2017. Hybrid code networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1,… [find](https://scholar.google.com/scholar?q=Jason+D+Williams%2C+Kavosh+Asadi%2C+and+Geoffrey+Zweig%2E+2017%2E+Hybrid+code+networks%3A+practical+and+efficient+end%2Dto%2Dend+dialog+control+with+supervised+and+reinforcement+learning%2E+In+Proceedings+of+the+55th)
35. Jason D Williams and Steve Young. 2007. Partially observable markov decision processes for spoken dialog systems. Computer Speech & Language, 21(2):393–422. [find](https://scholar.google.com/scholar?q=Jason+D+Williams+and+Steve+Young%2E+2007%2E+Partially+observable+markov+decision+processes+for+spoken+dialog+systems%2E+Computer+Speech+%26+Language%2C+21%282%29%3A393%E2%80%93422%2E)
36. Jason D Williams and Geoffrey Zweig. 2016. End-to-end lstm-based dialog control optimized with supervised and reinforcement learning. arXiv preprint arXiv:1606.01269. — [arXiv:1606.01269](https://arxiv.org/abs/1606.01269) [find](https://scholar.google.com/scholar?q=Jason+D+Williams+and+Geoffrey+Zweig%2E+2016%2E+End%2Dto%2Dend+lstm%2Dbased+dialog+control+optimized+with+supervised+and+reinforcement+learning%2E+arXiv+preprint+arXiv%3A1606%2E01269%2E+%E2%80%94+%5BarXiv%3A1606%2E01269%5D%28https%3A%2F%2Farxi)
37. Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229–256. [find](https://scholar.google.com/scholar?q=Ronald+J+Williams%2E+1992%2E+Simple+statistical+gradient%2Dfollowing+algorithms+for+connectionist+reinforcement+learning%2E+Machine+learning%2C+8%283%2D4%29%3A229%E2%80%93256%2E)
38. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. [Hierarchical attention networks for document classification](https://md.research.elamur.ai/free/en/papers/yang-2016-hierarchical-attention-networks-document-classification/). In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 14…
39. Denis Yarats and Mike Lewis. 2017. Hierarchical text generation and planning for strategic dialogue. arXiv preprint arXiv:1712.05846. — [arXiv:1712.05846](https://arxiv.org/abs/1712.05846) [find](https://scholar.google.com/scholar?q=Denis+Yarats+and+Mike+Lewis%2E+2017%2E+Hierarchical+text+generation+and+planning+for+strategic+dialogue%2E+arXiv+preprint+arXiv%3A1712%2E05846%2E+%E2%80%94+%5BarXiv%3A1712%2E05846%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1712%2E05846%29)
40. Stephanie Young, Jost Schatzmann, Karl Weilhammer, and Hui Ye. 2007. The hidden information state approach to dialog management. In Acoustics, Speech and Signal Processing, 2007. [find](https://scholar.google.com/scholar?q=Stephanie+Young%2C+Jost+Schatzmann%2C+Karl+Weilhammer%2C+and+Hui+Ye%2E+2007%2E+The+hidden+information+state+approach+to+dialog+management%2E+In+Acoustics%2C+Speech+and+Signal+Processing%2C+2007%2E)
41. ICASSP 2007. IEEE International Conference on, volume 4, pages IV– 149. IEEE. [find](https://scholar.google.com/scholar?q=ICASSP+2007%2E+IEEE+International+Conference+on%2C+volume+4%2C+pages+IV%E2%80%93+149%2E+IEEE%2E)
42. Steve Young, Milica Gašić, Blaise Thomson, and Jason D Williams. 2013. Pomdp-based statistical spoken dialog systems: A review. Proceedings of the IEEE, 101(5):1160–1179. [find](https://scholar.google.com/scholar?q=Steve+Young%2C+Milica+Gas%CC%8Cic%CC%81%2C+Blaise+Thomson%2C+and+Jason+D+Williams%2E+2013%2E+Pomdp%2Dbased+statistical+spoken+dialog+systems%3A+A+review%2E+Proceedings+of+the+IEEE%2C+101%285%29%3A1160%E2%80%931179%2E)
43. Steve J Young. 2006. Using pomdps for dialog management. In SLT, pages 8–13. [find](https://scholar.google.com/scholar?q=Steve+J+Young%2E+2006%2E+Using+pomdps+for+dialog+management%2E+In+SLT%2C+pages+8%E2%80%9313%2E)
44. Scenario 1 1 book, 1 hat and 3 balls available. Value Function Agent: book=1, hat=6, ball=1. User: book=1, hat=6, ball=1. Dialog Agent: I’d like the hat and 2 balls. User: Deal. Agent: SELECTION Outcome: Agreement Agent: 1 hat and 2 balls; User: 1 book and 1 ball Agent reward = 8; User reward = 2 Sc… [find](https://scholar.google.com/scholar?q=Scenario+1+1+book%2C+1+hat+and+3+balls+available%2E+Value+Function+Agent%3A+book%3D1%2C+hat%3D6%2C+ball%3D1%2E+User%3A+book%3D1%2C+hat%3D6%2C+ball%3D1%2E+Dialog+Agent%3A+I%E2%80%99d+like+the+hat+and+2+balls%2E+User%3A+Deal%2E+Agent%3A+SELECTION+Outco)
45. Tiancheng Zhao and Maxine Eskenazi. 2016. Towards end-to-end learning for dialog state tracking and management using deep reinforcement learning. In 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, page 1. [find](https://scholar.google.com/scholar?q=Tiancheng+Zhao+and+Maxine+Eskenazi%2E+2016%2E+Towards+end%2Dto%2Dend+learning+for+dialog+state+tracking+and+management+using+deep+reinforcement+learning%2E+In+17th+Annual+Meeting+of+the+Special+Interest+Group+o)
46. Tiancheng Zhao and Maxine Eskenazi. 2018. Zero-shot dialog generation with cross-domain latent actions. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, pages 1–10. [find](https://scholar.google.com/scholar?q=Tiancheng+Zhao+and+Maxine+Eskenazi%2E+2018%2E+Zero%2Dshot+dialog+generation+with+cross%2Ddomain+latent+actions%2E+In+Proceedings+of+the+19th+Annual+SIGdial+Meeting+on+Discourse+and+Dialogue%2C+pages+1%E2%80%9310%2E)
47. Tiancheng Zhao, Kyusong Lee, and Maxine Eskenazi. 2018. Unsupervised discrete sentence representation learning for interpretable neural dialog generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1. [find](https://scholar.google.com/scholar?q=Tiancheng+Zhao%2C+Kyusong+Lee%2C+and+Maxine+Eskenazi%2E+2018%2E+Unsupervised+discrete+sentence+representation+learning+for+interpretable+neural+dialog+generation%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+)
48. Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. [Learning discourse-level diversity for neural dialog models using conditional variational autoencoders](https://md.research.elamur.ai/free/en/papers/zhao-2017-learning-discourse-level-diversity-neural/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 654–664.

*Updated 24 October 2026*