---
title: "Adversarial Learning for Neural Dialogue — Li 2017"
description: "Neural chatbots trained with maximum likelihood tend to produce bland, repetitive, short-sighted replies."
authors: ["Jiwei Li", "Will Monroe", "Tianlin Shi", "Sébastien Jean", "Alan Ritter", "Dan Jurafsky"]
year: 2017
venue: "Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/D17-1230/"
doi: "10.18653/v1/d17-1230"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/li-2017-adversarial-learning-neural-dialogue-generation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2017-adversarial-learning-neural-dialogue-generation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/) / Adversarial Learning for Neural Dialogue Generation

# Adversarial Learning for Neural Dialogue Generation

**Authors:** [Jiwei Li](https://md.research.elamur.ai/free/en/authors/jiwei-li/), [Will Monroe](https://md.research.elamur.ai/free/en/authors/will-monroe/), Tianlin Shi, [Sébastien Jean](https://md.research.elamur.ai/free/en/authors/sebastien-jean/), [Alan Ritter](https://md.research.elamur.ai/free/en/authors/alan-ritter/), [Dan Jurafsky](https://md.research.elamur.ai/free/en/authors/dan-jurafsky/) · **Year:** 2017 · **Venue:** Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D17-1230/) · [PDF](https://aclanthology.org/D17-1230.pdf) · [DOI 10.18653/v1/d17-1230](https://doi.org/10.18653/v1/d17-1230) · [OpenAlex](https://openalex.org/W2581637843)

**Citations:** 873 citations (openalex-snapshot · 2026-10-14) · [171](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary

Neural chatbots trained with maximum likelihood tend to produce bland, repetitive, short-sighted replies. Hand-crafted RL rewards only capture a few chosen qualities. The authors instead take inspiration from the Turing test. A SEQ2SEQ generator is trained jointly with a discriminator, a hierarchical LSTM classifier that tries to tell human responses from machine ones given the dialogue context. The discriminator's estimated probability that a reply is human-written becomes the generator's reward, optimized with REINFORCE plus a learned critic baseline.

To give each token its own credit, they propose REGS (reward for every generation step). It works either through Monte Carlo rollouts or through a discriminator trained on partial sequences.

They also design adversarial evaluation (AdverSuc), which measures how often a model fools a trained evaluator. It is paired with a sanity check against random responses.

The adversarial systems beat MLE and mutual-information baselines on AdverSuc. In pairwise human judgments they improve both single-turn and multi-turn quality. This suggests a learned "human-likeness" reward is a more general training signal than manually defined ones.

## Key points

- **Architecture:**
  - The generator is an encoder-decoder RNN.
  - The discriminator is a hierarchical LSTM over the context plus the response, ending in a 2-class softmax.
  - The context is approximated by the two preceding utterances.
- **REGS with Monte Carlo search:** each partial sequence is completed N = 5 times by sampling, and the discriminator's scores on these completions are averaged.
- **AdverSuc results:** Adver-REGS scored 0.098 and Adver-Reinforce 0.080, versus 0.073 for MMI+p(t|s), 0.049 for MLE-greedy and 0.037 for MLE-BS.
  - Sampling reached 0.372, but its machine-vs-random accuracy dropped to 0.679, versus about 0.94–0.95 for the other models.
  - This means the evaluator could not separate sampled replies even from random ones, so the high score is misleading.
- **Human evaluation against the mutual-information system:** 200 items, 3 crowdsourced judges per item.
  - Single-turn: the adversarial model won 62%, lost 18%, tied 20%.
  - Multi-turn (3-turn simulated conversations): won 72%, lost 10%, tied 18%.
- **Machine translation:** preliminary MT experiments showed no clear gain. The authors conjecture that adversarial training helps most when target outputs have high entropy.

## Interesting ideas

- **Learned reward instead of hand-written criteria:** a classifier that separates real from generated outputs can serve as the reward. This avoids enumerating desirable properties by hand and could transfer to other open-ended generation tasks where "good" is hard to specify.
- **Per-token credit assignment:** scoring partial outputs, by rolling them out with Monte Carlo sampling or by training the discriminator on prefixes, gives finer feedback than one sequence-level reward. This trick fits any RL-trained sequence model.
- **Sanity check for evaluators:** fooling a classifier is only meaningful if that classifier can still tell the model's outputs from random responses. Reporting machine-vs-random accuracy next to the main score exposes degenerate "wins" such as noisy sampling. This is a useful safeguard for any learned metric.
- **Learned critic as baseline:** a separate network predicts the expected reward from the dialogue history and is subtracted from the actual reward. This reduces gradient variance without biasing the estimate, a standard but effective stabilizer for policy-gradient training on text.
- **Predicting where adversarial training pays off:** the authors link the method's benefit to output entropy, with dialogue gaining and MT not. This gives a quick heuristic for choosing tasks before investing effort.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

171 papers of our library cite it (of the 140884 whose bibliographies we read); 17 of them have a page here, 154 not yet.

- [Towards a Human-like Open-Domain Chatbot](https://md.research.elamur.ai/free/en/papers/adiwardana-2020-towards-human-like-open-domain/)
- [Robust Neural Machine Translation with Doubly Adversarial Inputs](https://md.research.elamur.ai/free/en/papers/cheng-2019-robust-neural-machine-translation-doubly/)
- [Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack](https://md.research.elamur.ai/free/en/papers/dinan-2019-build-it-break-it-fix/)
- [Decoupling Strategy and Generation in Negotiation Dialogues](https://md.research.elamur.ai/free/en/papers/he-2018-decoupling-strategy-generation-negotiation-dialogues/)
- [Speaker Turn Modeling for Dialogue Act Classification](https://md.research.elamur.ai/free/en/papers/he-2021-speaker-turn-modeling-dialogue-act/)
- [Automatic Detection of Generated Text is Easiest when Humans are Fooled](https://md.research.elamur.ai/free/en/papers/ippolito-2019-automatic-detection-generated-text-easiest/)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://md.research.elamur.ai/free/en/papers/li-2017-dailydialog-manually-labelled-multi-turn/)
- [Paraphrase Generation with Deep Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/li-2018-paraphrase-generation-deep-reinforcement-learning/)
- [A Dual Reinforcement Learning Framework for Unsupervised Text Style Transfer](https://md.research.elamur.ai/free/en/papers/luo-2019-dual-reinforcement-learning-framework-unsupervised/)
- [USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation](https://md.research.elamur.ai/free/en/papers/mehri-2020-usr-unsupervised-reference-free-evaluation/)
- [What makes a good conversation? How controllable attributes affect human judgments](https://md.research.elamur.ai/free/en/papers/see-2019-what-makes-good-conversation-how/)
- [Challenges in Data-to-Document Generation](https://md.research.elamur.ai/free/en/papers/wiseman-2017-challenges-data-document-generation/)
- [A Study of Reinforcement Learning for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/wu-2018-study-reinforcement-learning-neural-machine/)
- [Unsupervised Text Style Transfer using Language Models as Discriminators](https://md.research.elamur.ai/free/en/papers/yang-2018-unsupervised-text-style-transfer-using/)
- [Multi-Turn Response Selection for Chatbots with Deep Attention Matching Network](https://md.research.elamur.ai/free/en/papers/zhou-2018-multi-turn-response-selection-chatbots/)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://md.research.elamur.ai/free/en/papers/zhu-2018-texygen-benchmarking-platform-text-generation/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{li2017adversarial,
  title = {{Adversarial Learning for Neural Dialogue Generation}},
  author = {Li, Jiwei and Monroe, Will and Shi, Tianlin and Jean, Sébastien and Ritter, Alan and Jurafsky, Dan},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1230},
  url = {https://doi.org/10.18653/v1/d17-1230}
}
```

- [← previous Adversarial Examples for Evaluating Reading Comprehension Systems](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/)
- [next → Adversarial Multi-task Learning for Text Classification](https://md.research.elamur.ai/free/en/papers/liu-2017-adversarial-multi-task-learning-text/)
- [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/)

## References

50 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. V. M. Aleksandrov, V. I. Sysoyev, and V. V. Shemeneva. 1968. Stochastic optimization. Engineering Cybernetics 5:11–16. [find](https://scholar.google.com/scholar?q=V%2E+M%2E+Aleksandrov%2C+V%2E+I%2E+Sysoyev%2C+and+V%2E+V%2E+Shemeneva%2E+1968%2E+Stochastic+optimization%2E+Engineering+Cybernetics+5%3A11%E2%80%9316%2E)
2. Alex Lamb, Anirudh Goyal, Ying Zhang, Saizheng Zhang, Aaron Courville, and Yoshua Bengio. 2016. Professor forcing: A new algorithm for training recurrent networks. In Advances In Neural Information Processing Systems. pages 4601–4609. [find](https://scholar.google.com/scholar?q=Alex+Lamb%2C+Anirudh+Goyal%2C+Ying+Zhang%2C+Saizheng+Zhang%2C+Aaron+Courville%2C+and+Yoshua+Bengio%2E+2016%2E+Professor+forcing%3A+A+new+algorithm+for+training+recurrent+networks%2E+In+Advances+In+Neural+Information+Pr)
3. Nabiha Asghar, Pasca Poupart, Jiang Xin, and Hang Li. 2016. Online sequence-to-sequence reinforcement learning for open-domain conversational agents. arXiv preprint arXiv:1612.03929 . — [arXiv:1612.03929](https://arxiv.org/abs/1612.03929) [find](https://scholar.google.com/scholar?q=Nabiha+Asghar%2C+Pasca+Poupart%2C+Jiang+Xin%2C+and+Hang+Li%2E+2016%2E+Online+sequence%2Dto%2Dsequence+reinforcement+learning+for+open%2Ddomain+conversational+agents%2E+arXiv+preprint+arXiv%3A1612%2E03929+%2E+%E2%80%94+%5BarXiv%3A1612%2E03)
4. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A diversity-promoting objective function for neural conversation models. In Proc. of NAACL-HLT. [find](https://scholar.google.com/scholar?q=Jiwei+Li%2C+Michel+Galley%2C+Chris+Brockett%2C+Jianfeng+Gao%2C+and+Bill+Dolan%2E+2016a%2E+A+diversity%2Dpromoting+objective+function+for+neural+conversation+models%2E+In+Proc%2E+of+NAACL%2DHLT%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2016-diversity-promoting-objective-function-neural/)
5. Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017. An actor-critic algorithm for sequence prediction. ICLR . [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Philemon+Brakel%2C+Kelvin+Xu%2C+Anirudh+Goyal%2C+Ryan+Lowe%2C+Joelle+Pineau%2C+Aaron+Courville%2C+and+Yoshua+Bengio%2E+2017%2E+An+actor%2Dcritic+algorithm+for+sequence+prediction%2E+ICLR+%2E) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2016-actor-critic-algorithm-sequence-prediction/)
6. Jiwei Li, Michel Galley, Chris Brockett, Georgios Spithourakis, Jianfeng Gao, and Bill Dolan. 2016b. A persona-based neural conversation model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Berlin, Germany, pages 994–1003. http://… [find](https://scholar.google.com/scholar?q=Jiwei+Li%2C+Michel+Galley%2C+Chris+Brockett%2C+Georgios+Spithourakis%2C+Jianfeng+Gao%2C+and+Bill+Dolan%2E+2016b%2E+A+persona%2Dbased+neural+conversation+model%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Associa) [our card](https://md.research.elamur.ai/free/en/papers/li-2016-persona-based-neural-conversation-model/)
7. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. of ICLR. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2015%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+In+Proc%2E+of+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
8. Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. CoNLL . [find](https://scholar.google.com/scholar?q=Samuel+R+Bowman%2C+Luke+Vilnis%2C+Oriol+Vinyals%2C+Andrew+M+Dai%2C+Rafal+Jozefowicz%2C+and+Samy+Bengio%2E+2016%2E+Generating+sentences+from+a+continuous+space%2E+CoNLL+%2E) [our card](https://md.research.elamur.ai/free/en/papers/bowman-2016-generating-sentences-continuous-space/)
9. Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. 2016a. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances In Neural Information Processing Systems. pages 2172–2180. [find](https://scholar.google.com/scholar?q=Xi+Chen%2C+Yan+Duan%2C+Rein+Houthooft%2C+John+Schulman%2C+Ilya+Sutskever%2C+and+Pieter+Abbeel%2E+2016a%2E+Infogan%3A+Interpretable+representation+learning+by+information+maximizing+generative+adversarial+nets%2E+In+Adv)
10. Xilun Chen, Ben Athiwaratkun, Yu Sun, Kilian Weinberger, and Claire Cardie. 2016b. Adversarial deep averaging networks for cross-lingual sentiment classification. arXiv preprint arXiv:1606.01614 . [find](https://scholar.google.com/scholar?q=Xilun+Chen%2C+Ben+Athiwaratkun%2C+Yu+Sun%2C+Kilian+Weinberger%2C+and+Claire+Cardie%2E+2016b%2E+Adversarial+deep+averaging+networks+for+cross%2Dlingual+sentiment+classification%2E+arXiv+preprint+arXiv%3A1606%2E01614+%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2018-adversarial-deep-averaging-networks-cross/)
11. Emily L Denton, Soumith Chintala, Rob Fergus, et al. 2015. Deep generative image models using a? laplacian pyramid of adversarial networks. In Advances in neural information processing systems. pages 1486–1494. [find](https://scholar.google.com/scholar?q=Emily+L+Denton%2C+Soumith+Chintala%2C+Rob+Fergus%2C+et+al%2E+2015%2E+Deep+generative+image+models+using+a%3F+laplacian+pyramid+of+adversarial+networks%2E+In+Advances+in+neural+information+processing+systems%2E+pages+)
12. Peter W Glynn. 1990. Likelihood ratio gradient estimation for stochastic systems. Communications of the ACM 33(10):75–84. [find](https://scholar.google.com/scholar?q=Peter+W+Glynn%2E+1990%2E+Likelihood+ratio+gradient+estimation+for+stochastic+systems%2E+Communications+of+the+ACM+33%2810%29%3A75%E2%80%9384%2E)
13. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9(8):1735– 1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+computation+9%288%29%3A1735%E2%80%93+1780%2E)
14. Thorsten Joachims. 2002. Learning to classify text using support vector machines: Methods, theory and algorithms. Kluwer Academic Publishers. [find](https://scholar.google.com/scholar?q=Thorsten+Joachims%2E+2002%2E+Learning+to+classify+text+using+support+vector+machines%3A+Methods%2C+theory+and+algorithms%2E+Kluwer+Academic+Publishers%2E)
15. Anjuli Kannan and Oriol Vinyals. 2016. Adversarial evaluation of dialogue models. In NIPS 2016 Workshop on Adversarial Training. [find](https://scholar.google.com/scholar?q=Anjuli+Kannan+and+Oriol+Vinyals%2E+2016%2E+Adversarial+evaluation+of+dialogue+models%2E+In+NIPS+2016+Workshop+on+Adversarial+Training%2E)
16. Jiwei Li, Minh-Thang Luong, and Dan Jurafsky. 2015. A hierarchical neural autoencoder for paragraphs and documents. ACL . [find](https://scholar.google.com/scholar?q=Jiwei+Li%2C+Minh%2DThang+Luong%2C+and+Dan+Jurafsky%2E+2015%2E+A+hierarchical+neural+autoencoder+for+paragraphs+and+documents%2E+ACL+%2E)
17. Jiwei Li, Will Monroe, and Dan Jurafsky. 2016c. A simple, fast diverse decoding algorithm for neural generation. arXiv preprint arXiv:1611.08562 . — [arXiv:1611.08562](https://arxiv.org/abs/1611.08562) [find](https://scholar.google.com/scholar?q=Jiwei+Li%2C+Will+Monroe%2C+and+Dan+Jurafsky%2E+2016c%2E+A+simple%2C+fast+diverse+decoding+algorithm+for+neural+generation%2E+arXiv+preprint+arXiv%3A1611%2E08562+%2E+%E2%80%94+%5BarXiv%3A1611%2E08562%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1611%2E08562)
18. Jiwei Li, Will Monroe, Alan Ritter, and Dan Jurafsky. 2016d. Deep reinforcement learning for dialogue generation. EMNLP . [find](https://scholar.google.com/scholar?q=Jiwei+Li%2C+Will+Monroe%2C+Alan+Ritter%2C+and+Dan+Jurafsky%2E+2016d%2E+Deep+reinforcement+learning+for+dialogue+generation%2E+EMNLP+%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2016-deep-reinforcement-learning-dialogue-generation/)
19. Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. EMNLP . [find](https://scholar.google.com/scholar?q=Chia%2DWei+Liu%2C+Ryan+Lowe%2C+Iulian+V+Serban%2C+Michael+Noseworthy%2C+Laurent+Charlin%2C+and+Joelle+Pineau%2E+2016%2E+How+NOT+to+evaluate+your+dialogue+system%3A+An+empirical+study+of+unsupervised+evaluation+metrics+)
20. Ryan Lowe, Michael Noseworthy, Iulian Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017. Towards an automatic turing test: Learning to evaluate dialogue responses. ACL . [find](https://scholar.google.com/scholar?q=Ryan+Lowe%2C+Michael+Noseworthy%2C+Iulian+Serban%2C+Nicolas+Angelard%2DGontier%2C+Yoshua+Bengio%2C+and+Joelle+Pineau%2E+2017%2E+Towards+an+automatic+turing+test%3A+Learning+to+evaluate+dialogue+responses%2E+ACL+%2E) [our card](https://md.research.elamur.ai/free/en/papers/lowe-2017-towards-automatic-turing-test-learning/)
21. Ryan Lowe, Iulian V Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. On the evaluation of dialogue systems with next utterance classification. SIGDIAL . [find](https://scholar.google.com/scholar?q=Ryan+Lowe%2C+Iulian+V+Serban%2C+Mike+Noseworthy%2C+Laurent+Charlin%2C+and+Joelle+Pineau%2E+2016%2E+On+the+evaluation+of+dialogue+systems+with+next+utterance+classification%2E+SIGDIAL+%2E)
22. Yi Luan, Yangfeng Ji, and Mari Ostendorf. 2016. LSTM based conversation models. arXiv preprint arXiv:1603.09457 . — [arXiv:1603.09457](https://arxiv.org/abs/1603.09457) [find](https://scholar.google.com/scholar?q=Yi+Luan%2C+Yangfeng+Ji%2C+and+Mari+Ostendorf%2E+2016%2E+LSTM+based+conversation+models%2E+arXiv+preprint+arXiv%3A1603%2E09457+%2E+%E2%80%94+%5BarXiv%3A1603%2E09457%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1603%2E09457%29)
23. Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum risk training for neural machine translation. ACL . [find](https://scholar.google.com/scholar?q=Shiqi+Shen%2C+Yong+Cheng%2C+Zhongjun+He%2C+Wei+He%2C+Hua+Wu%2C+Maosong+Sun%2C+and+Yang+Liu%2E+2016%2E+Minimum+risk+training+for+neural+machine+translation%2E+ACL+%2E) [our card](https://md.research.elamur.ai/free/en/papers/shen-2015-minimum-risk-training-neural-machine/)
24. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. ACL . [find](https://scholar.google.com/scholar?q=Minh%2DThang+Luong%2C+Hieu+Pham%2C+and+Christopher+D+Manning%2E+2015%2E+Effective+approaches+to+attention%2Dbased+neural+machine+translation%2E+ACL+%2E)
25. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. 2016. Mastering the game of Go with deep neural networks and tree search. Nature 529(7587):484–489. [find](https://scholar.google.com/scholar?q=David+Silver%2C+Aja+Huang%2C+Chris+J+Maddison%2C+Arthur+Guez%2C+Laurent+Sifre%2C+George+Van+Den+Driessche%2C+Julian+Schrittwieser%2C+Ioannis+Antonoglou%2C+Veda+Panneershelvam%2C+Marc+Lanctot%2C+et+al%2E+2016%2E+Mastering+the)
26. Alec Radford, Luke Metz, and Soumith Chintala. 2015. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 . [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Luke+Metz%2C+and+Soumith+Chintala%2E+2015%2E+Unsupervised+representation+learning+with+deep+convolutional+generative+adversarial+networks%2E+arXiv+preprint+arXiv%3A1511%2E06434+%2E) [our card](https://md.research.elamur.ai/free/en/papers/radford-2015-unsupervised-representation-learning-deep-convolutional/)
27. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence level training with recurrent neural networks. ICLR . [find](https://scholar.google.com/scholar?q=Marc%E2%80%99Aurelio+Ranzato%2C+Sumit+Chopra%2C+Michael+Auli%2C+and+Wojciech+Zaremba%2E+2016%2E+Sequence+level+training+with+recurrent+neural+networks%2E+ICLR+%2E) [our card](https://md.research.elamur.ai/free/en/papers/ranzato-2015-sequence-level-training-recurrent-neural/)
28. Alan Ritter, Colin Cherry, and William B Dolan. 2011. Data-driven response generation in social media. In Proceedings of EMNLP 2011. pages 583–593. [find](https://scholar.google.com/scholar?q=Alan+Ritter%2C+Colin+Cherry%2C+and+William+B+Dolan%2E+2011%2E+Data%2Ddriven+response+generation+in+social+media%2E+In+Proceedings+of+EMNLP+2011%2E+pages+583%E2%80%93593%2E) [our card](https://md.research.elamur.ai/free/en/papers/ritter-2011-data-driven-response-generation-social/)
29. Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. 2016. Improved techniques for training gans. In Advances in Neural Information Processing Systems. pages 2226–2234. [find](https://scholar.google.com/scholar?q=Tim+Salimans%2C+Ian+Goodfellow%2C+Wojciech+Zaremba%2C+Vicki+Cheung%2C+Alec+Radford%2C+and+Xi+Chen%2E+2016%2E+Improved+techniques+for+training+gans%2E+In+Advances+in+Neural+Information+Processing+Systems%2E+pages+2226%E2%80%932) [our card](https://md.research.elamur.ai/free/en/papers/salimans-2016-improved-techniques-training-gans/)
30. Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016a. Building end-to-end dialogue systems using generative hierarchical neural network models. In Proceedings of AAAI. [find](https://scholar.google.com/scholar?q=Iulian+V+Serban%2C+Alessandro+Sordoni%2C+Yoshua+Bengio%2C+Aaron+Courville%2C+and+Joelle+Pineau%2E+2016a%2E+Building+end%2Dto%2Dend+dialogue+systems+using+generative+hierarchical+neural+network+models%2E+In+Proceedings+) [our card](https://md.research.elamur.ai/free/en/papers/serban-2015-building-end-end-dialogue-systems/)
31. Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016b. Building end-to-end dialogue systems using generative hierarchical neural network models. In Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI-16). [find](https://scholar.google.com/scholar?q=Iulian+V+Serban%2C+Alessandro+Sordoni%2C+Yoshua+Bengio%2C+Aaron+Courville%2C+and+Joelle+Pineau%2E+2016b%2E+Building+end%2Dto%2Dend+dialogue+systems+using+generative+hierarchical+neural+network+models%2E+In+Proceedings+) [our card](https://md.research.elamur.ai/free/en/papers/serban-2015-building-end-end-dialogue-systems/)
32. Iulian Vlad Serban, Tim Klinger, Gerald Tesauro, Kartik Talamadupula, Bowen Zhou, Yoshua Bengio, and Aaron Courville. 2016c. Multiresolution recurrent neural networks: An application to dialogue response generation. arXiv preprint arXiv:1606.00776 . — [arXiv:1606.00776](https://arxiv.org/abs/1606.00776) [find](https://scholar.google.com/scholar?q=Iulian+Vlad+Serban%2C+Tim+Klinger%2C+Gerald+Tesauro%2C+Kartik+Talamadupula%2C+Bowen+Zhou%2C+Yoshua+Bengio%2C+and+Aaron+Courville%2E+2016c%2E+Multiresolution+recurrent+neural+networks%3A+An+application+to+dialogue+respo)
33. Iulian Vlad Serban, Ryan Lowe, Laurent Charlin, and Joelle Pineau. 2016d. Generative deep neural networks for dialogue: A short review . [find](https://scholar.google.com/scholar?q=Iulian+Vlad+Serban%2C+Ryan+Lowe%2C+Laurent+Charlin%2C+and+Joelle+Pineau%2E+2016d%2E+Generative+deep+neural+networks+for+dialogue%3A+A+short+review+%2E)
34. Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017. A hierarchical latent variable encoder-decoder model for generating dialogues. AAAI . [find](https://scholar.google.com/scholar?q=Iulian+Vlad+Serban%2C+Alessandro+Sordoni%2C+Ryan+Lowe%2C+Laurent+Charlin%2C+Joelle+Pineau%2C+Aaron+Courville%2C+and+Yoshua+Bengio%2E+2017%2E+A+hierarchical+latent+variable+encoder%2Ddecoder+model+for+generating+dialogu) [our card](https://md.research.elamur.ai/free/en/papers/serban-2016-hierarchical-latent-variable-encoder-decoder/)
35. Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Meg Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. In Proceedings of NAACL-HLT. [find](https://scholar.google.com/scholar?q=Alessandro+Sordoni%2C+Michel+Galley%2C+Michael+Auli%2C+Chris+Brockett%2C+Yangfeng+Ji%2C+Meg+Mitchell%2C+Jian%2DYun+Nie%2C+Jianfeng+Gao%2C+and+Bill+Dolan%2E+2015%2E+A+neural+network+approach+to+context%2Dsensitive+generation+) [our card](https://md.research.elamur.ai/free/en/papers/sordoni-2015-neural-network-approach-context-sensitive/)
36. Pei-Hao Su, Milica Gasic, Nikola Mrksic, Lina Rojas-Barahona, Stefan Ultes, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. Continuously learning neural dialogue management. arxiv . [find](https://scholar.google.com/scholar?q=Pei%2DHao+Su%2C+Milica+Gasic%2C+Nikola+Mrksic%2C+Lina+Rojas%2DBarahona%2C+Stefan+Ultes%2C+David+Vandyke%2C+Tsung%2DHsien+Wen%2C+and+Steve+Young%2E+2016%2E+Continuously+learning+neural+dialogue+management%2E+arxiv+%2E)
37. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems. pages 3104–3112. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Advances+in+neural+information+processing+systems%2E+pages+3104%E2%80%933112%2E)
38. Alan M Turing. 1950. Computing machinery and intelligence. Mind 59(236):433–460. [find](https://scholar.google.com/scholar?q=Alan+M+Turing%2E+1950%2E+Computing+machinery+and+intelligence%2E+Mind+59%28236%29%3A433%E2%80%93460%2E)
39. Oriol Vinyals and Quoc Le. 2015. A neural conversational model. In Proceedings of ICML Deep Learning Workshop. [find](https://scholar.google.com/scholar?q=Oriol+Vinyals+and+Quoc+Le%2E+2015%2E+A+neural+conversational+model%2E+In+Proceedings+of+ICML+Deep+Learning+Workshop%2E) [our card](https://md.research.elamur.ai/free/en/papers/vinyals-2015-neural-conversational-model/)
40. Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Lina M Rojas-Barahona, Pei-Hao Su, Stefan Ultes, David Vandyke, and Steve Young. 2016. A network-based end-to-end trainable task-oriented dialogue system. arXiv preprint arXiv:1604.04562 . [find](https://scholar.google.com/scholar?q=Tsung%2DHsien+Wen%2C+Milica+Gasic%2C+Nikola+Mrksic%2C+Lina+M+Rojas%2DBarahona%2C+Pei%2DHao+Su%2C+Stefan+Ultes%2C+David+Vandyke%2C+and+Steve+Young%2E+2016%2E+A+network%2Dbased+end%2Dto%2Dend+trainable+task%2Doriented+dialogue+system%2E)
41. Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning 8(3-4):229–256. [find](https://scholar.google.com/scholar?q=Ronald+J+Williams%2E+1992%2E+Simple+statistical+gradient%2Dfollowing+algorithms+for+connectionist+reinforcement+learning%2E+Machine+learning+8%283%2D4%29%3A229%E2%80%93256%2E)
42. Sam Wiseman and Alexander M Rush. 2016. Sequence-to-sequence learning as beam-search optimization. ACL . [find](https://scholar.google.com/scholar?q=Sam+Wiseman+and+Alexander+M+Rush%2E+2016%2E+Sequence%2Dto%2Dsequence+learning+as+beam%2Dsearch+optimization%2E+ACL+%2E)
43. Zhen Xu, Bingquan Liu, Baoxun Wang, Chengjie Sun, and Xiaolong Wang. 2016. Incorporating loose-structured knowledge into LSTM with recall gate for conversation modeling. arXiv preprint arXiv:1605.05110 . — [arXiv:1605.05110](https://arxiv.org/abs/1605.05110) [find](https://scholar.google.com/scholar?q=Zhen+Xu%2C+Bingquan+Liu%2C+Baoxun+Wang%2C+Chengjie+Sun%2C+and+Xiaolong+Wang%2E+2016%2E+Incorporating+loose%2Dstructured+knowledge+into+LSTM+with+recall+gate+for+conversation+modeling%2E+arXiv+preprint+arXiv%3A1605%2E0511)
44. Lifeng Shang, Zhengdong Lu, and Hang Li. 2015. Neural responding machine for short-text conversation. In Proceedings of ACL-IJCNLP. pages 1577–1586. [find](https://scholar.google.com/scholar?q=Lifeng+Shang%2C+Zhengdong+Lu%2C+and+Hang+Li%2E+2015%2E+Neural+responding+machine+for+short%2Dtext+conversation%2E+In+Proceedings+of+ACL%2DIJCNLP%2E+pages+1577%E2%80%931586%2E) [our card](https://md.research.elamur.ai/free/en/papers/shang-2015-neural-responding-machine-short-text/)
45. Kaisheng Yao, Geoffrey Zweig, and Baolin Peng. 2015. Attention with intention for a neural network conversation model. In NIPS workshop on Machine Learning for Spoken Language Understanding and Interaction. [find](https://scholar.google.com/scholar?q=Kaisheng+Yao%2C+Geoffrey+Zweig%2C+and+Baolin+Peng%2E+2015%2E+Attention+with+intention+for+a+neural+network+conversation+model%2E+In+NIPS+workshop+on+Machine+Learning+for+Spoken+Language+Understanding+and+Intera)
46. Louis Shao, Stephan Gouws, Denny Britz, Anna Goldie, Brian Strope, and Ray Kurzweil. 2017. Generating long and diverse responses with neural conversational models. ICLR . [find](https://scholar.google.com/scholar?q=Louis+Shao%2C+Stephan+Gouws%2C+Denny+Britz%2C+Anna+Goldie%2C+Brian+Strope%2C+and+Ray+Kurzweil%2E+2017%2E+Generating+long+and+diverse+responses+with+neural+conversational+models%2E+ICLR+%2E)
47. Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. 2016a. Seqgan: sequence generative adversarial nets with policy gradient. arXiv preprint arXiv:1609.05473 . — [arXiv:1609.05473](https://arxiv.org/abs/1609.05473) [find](https://scholar.google.com/scholar?q=Lantao+Yu%2C+Weinan+Zhang%2C+Jun+Wang%2C+and+Yong+Yu%2E+2016a%2E+Seqgan%3A+sequence+generative+adversarial+nets+with+policy+gradient%2E+arXiv+preprint+arXiv%3A1609%2E05473+%2E+%E2%80%94+%5BarXiv%3A1609%2E05473%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1)
48. Zhou Yu, Ziyu Xu, Alan W Black, and Alex I Rudnicky. 2016b. Strategy and policy learning for non-task-oriented conversational systems. In 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue. page 404. [find](https://scholar.google.com/scholar?q=Zhou+Yu%2C+Ziyu+Xu%2C+Alan+W+Black%2C+and+Alex+I+Rudnicky%2E+2016b%2E+Strategy+and+policy+learning+for+non%2Dtask%2Doriented+conversational+systems%2E+In+17th+Annual+Meeting+of+the+Special+Interest+Group+on+Discourse)
49. Yuan Zhang, Regina Barzilay, and Tommi Jaakkola. 2017. Aspect-augmented adversarial networks for domain adaptation. arXiv preprint arXiv:1701.00188 . [find](https://scholar.google.com/scholar?q=Yuan+Zhang%2C+Regina+Barzilay%2C+and+Tommi+Jaakkola%2E+2017%2E+Aspect%2Daugmented+adversarial+networks+for+domain+adaptation%2E+arXiv+preprint+arXiv%3A1701%2E00188+%2E)
50. Input Vanilla-MLE Vanilla-Sample Reinforce MC Input Vanilla-MLE Vanilla-Sample Reinforce MC Input Vanilla-MLE Vanilla-Sample Reinforce MC Input Vanilla-MLE Vanilla-Sample Reinforce MC Input Vanilla-MLE Vanilla-Sample Reinforce MC Input Vanilla-MLE Vanilla-Sample Reinforce MC Input Vanilla-MLE Vanill… [find](https://scholar.google.com/scholar?q=Input+Vanilla%2DMLE+Vanilla%2DSample+Reinforce+MC+Input+Vanilla%2DMLE+Vanilla%2DSample+Reinforce+MC+Input+Vanilla%2DMLE+Vanilla%2DSample+Reinforce+MC+Input+Vanilla%2DMLE+Vanilla%2DSample+Reinforce+MC+Input+Vanilla%2DML)