---
title: "Evaluating Prerequisite Qualities for Learning End-to-End Dialog"
description: "These measures show whether replies sound plausible, not whether a system can reach a goal. 46 references, cited by 65 papers here."
authors: ["Jesse Dodge", "Andreea Gane", "Xiang Zhang", "Antoine Bordes", "Sumit Chopra", "Alexander Miller", "Arthur Szlam", "Jason Weston"]
year: 2015
venue: "arXiv:1511.06931"
source: "https://arxiv.org/abs/1511.06931"
doi: "10.48550/arxiv.1511.06931"
arxiv: "1511.06931"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/dodge-2015-evaluating-prerequisite-qualities-learning-end/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/dodge-2015-evaluating-prerequisite-qualities-learning-end/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/) / Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems

# Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems

**Authors:** [Jesse Dodge](https://md.research.elamur.ai/free/en/authors/jesse-dodge/), [Andreea Gane](https://md.research.elamur.ai/free/en/authors/andreea-gane/), [Xiang Zhang](https://md.research.elamur.ai/free/en/authors/xiang-zhang/), [Antoine Bordes](https://md.research.elamur.ai/free/en/authors/antoine-bordes/), [Sumit Chopra](https://md.research.elamur.ai/free/en/authors/sumit-chopra/), [Alexander Miller](https://md.research.elamur.ai/free/en/authors/alexander-miller/), [Arthur Szlam](https://md.research.elamur.ai/free/en/authors/arthur-d-szlam/) and [Jason Weston](https://md.research.elamur.ai/free/en/authors/jason-weston/) · **Year:** 2015 · **Venue:** arXiv:1511.06931 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1511.06931) · [PDF](https://arxiv.org/pdf/1511.06931) · [DOI 10.48550/arxiv.1511.06931](https://doi.org/10.48550/arxiv.1511.06931) · [arXiv 1511.06931](https://arxiv.org/abs/1511.06931) · [OpenAlex](https://openalex.org/W2175256910)

**Citations:** 46 citations (OpenAlex, 2026-10-14) · [65](https://md.research.elamur.ai/free/en/papers/dodge-2015-evaluating-prerequisite-qualities-learning-end/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
End-to-end neural dialog systems trained on raw human conversations are usually judged by BLEU, crowdsourcing or a few human ratings. These measures show whether replies sound plausible, not whether a system can reach a goal. The synthetic bAbI tasks are diagnostic but too small to reward scalable methods. Researchers at Facebook AI Research address this with the Movie Dialog dataset, built from OMDb, MovieLens and Reddit. It contains four movie-domain tasks plus a joint one:
- factoid QA over a knowledge base of triples
- personalized recommendation
- three-turn dialogs mixing QA and recommendation
- picking the right reply in Reddit threads

Models rank candidate answers and are scored with hits@k, so no human judges are needed. Compared systems include supervised embeddings, LSTMs, IR baselines, SVD, a dedicated QA system and Memory Networks (MemN2N). MemN2N was the strongest general-purpose model, especially where dialog context matters. LSTMs struggled on Reddit, and performing all tasks in one joint model proved harder. MemN2N also clearly outperformed the original baselines on the Ubuntu Dialog Corpus.

## Key points
- The dataset covers about 75k movie entities and about 3.5M training examples. It is available via fb.ai/babi.
- Task 1 (QA) uses about 96k/10k/10k train/dev/test questions across 11 question types. Templates come from SimpleQuestions, and the knowledge base has 8 relation types. Test hits@1 was 90.7 for the QA system of Bordes et al. (2014), 78.9 for MemN2N and 50.9 for supervised embeddings.
- Task 2 (Recommendation) draws on MovieLens ratings of about 11k movies and about 110k training users. It provides 1M generated training examples and is scored with hits@100.
- Task 3 (QA+Recommendation) is scored with hits@10. MemN2N with 2 hops reached 76.8, compared with 56.0 for supervised embeddings and 19.9 for an LSTM.
- Task 4 (Reddit) is also scored with hits@10. MemN2N reached about 29, against 27.6 for supervised embeddings and 23.7 for the best IR baseline. LSTMs tended to favor short replies until hidden states were averaged.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Meta](https://md.research.elamur.ai/free/en/companies/meta/) — Company, Menlo Park, United States · 214 works · 241,492 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works · [🗨️ Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) · 654 works

## Cite

Cite as: Jesse Dodge, Andreea Gane, Xiang Zhang, Antoine Bordes, Sumit Chopra, Alexander Miller, Arthur Szlam and Jason Weston. 2015. Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems. arXiv:1511.06931.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{dodge2015evaluating,
  title = {{Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems}},
  author = {Jesse Dodge, Andreea Gane, Xiang Zhang, Antoine Bordes, Sumit Chopra, Alexander Miller, Arthur Szlam and Jason Weston},
  year = {2015},
  doi = {10.48550/arxiv.1511.06931},
  eprint = {1511.06931},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1511.06931}
}
```

- [← previous A Corpus and Model Integrating Multiword Expressions and Supersenses](https://md.research.elamur.ai/free/en/papers/schneider-2015-corpus-model-integrating-multiword-expressions/)
- [next → Latent Structures for Coreference Resolution](https://md.research.elamur.ai/free/en/papers/martschat-2015-latent-structures-coreference-resolution/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/)

## References

46 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio, Yoshua. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). ICLR 2015, 2015. 11 Results for the baselines from (Lowe et al., 2015) differ to that from the v3 of the arxiv paper, because the corpus has been updated since then. All resu…
2. V ALIDATION ( HITS @1) N / A N / A N / A 57.23 64.28 64.31 64.01 T EST ( HITS @1) 48.81 37.91 55.22 56.25 63.51 63.72 62.82 Bai, Bing, Weston, Jason, Grangier, David, Collobert, Ronan, Sadamasa, Kunihiko, Qi, Yanjun, Chapelle, Olivier, and Weinberger, Kilian. Supervised semantic indexing. In Proceed… [find](https://scholar.google.com/scholar?q=V+ALIDATION+%28+HITS+%401%29+N+%2F+A+N+%2F+A+N+%2F+A+57%2E23+64%2E28+64%2E31+64%2E01+T+EST+%28+HITS+%401%29+48%2E81+37%2E91+55%2E22+56%2E25+63%2E51+63%2E72+62%2E82+Bai%2C+Bing%2C+Weston%2C+Jason%2C+Grangier%2C+David%2C+Collobert%2C+Ronan%2C+Sadamasa%2C+Kunih)
3. Berant, Jonathan and Liang, Percy. [Semantic parsing via paraphrasing](https://md.research.elamur.ai/free/en/papers/berant-2014-semantic-parsing-paraphrasing/). In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL’14), Baltimore, USA, 2014.
4. Berant, Jonathan, Chou, Andrew, Frostig, Roy, and Liang, Percy. [Semantic parsing on freebase from question-answer pairs](https://md.research.elamur.ai/free/en/papers/berant-2013-semantic-parsing-freebase-question-answer/). In EMNLP, pp. 1533–1544, 2013.
5. Bordes, Antoine, Chopra, Sumit, and Weston, Jason. Question answering with subgraph embeddings. In Proc. EMNLP, 2014. [find](https://scholar.google.com/scholar?q=Bordes%2C+Antoine%2C+Chopra%2C+Sumit%2C+and+Weston%2C+Jason%2E+Question+answering+with+subgraph+embeddings%2E+In+Proc%2E+EMNLP%2C+2014%2E)
6. Bordes, Antoine, Usunier, Nicolas, Chopra, Sumit, and Weston, Jason. Large-scale simple question answering with memory networks. arXiv preprint arXiv:1506.02075, 2015. [find](https://scholar.google.com/scholar?q=Bordes%2C+Antoine%2C+Usunier%2C+Nicolas%2C+Chopra%2C+Sumit%2C+and+Weston%2C+Jason%2E+Large%2Dscale+simple+question+answering+with+memory+networks%2E+arXiv+preprint+arXiv%3A1506%2E02075%2C+2015%2E)
7. Cremonesi, Paolo, Koren, Yehuda, and Turrin, Roberto. Performance of recommender algorithms on top-n recommendation tasks. In Proceedings of the fourth ACM conference on Recommender systems, pp. 39–46. ACM, 2010. [find](https://scholar.google.com/scholar?q=Cremonesi%2C+Paolo%2C+Koren%2C+Yehuda%2C+and+Turrin%2C+Roberto%2E+Performance+of+recommender+algorithms+on+top%2Dn+recommendation+tasks%2E+In+Proceedings+of+the+fourth+ACM+conference+on+Recommender+systems%2C+pp%2E+39%E2%80%9346)
8. Fader, Anthony, Zettlemoyer, Luke, and Etzioni, Oren. Open question answering over curated and extracted knowledge bases. In Proceedings of 20th SIGKDD Conference on Knowledge Discovery and Data Mining (KDD’14), New York City, USA, 2014. ACM. [find](https://scholar.google.com/scholar?q=Fader%2C+Anthony%2C+Zettlemoyer%2C+Luke%2C+and+Etzioni%2C+Oren%2E+Open+question+answering+over+curated+and+extracted+knowledge+bases%2E+In+Proceedings+of+20th+SIGKDD+Conference+on+Knowledge+Discovery+and+Data+Minin)
9. Graves, Alex et al. Supervised sequence labelling with recurrent neural networks, volume 385. Springer, 2012. [find](https://scholar.google.com/scholar?q=Graves%2C+Alex+et+al%2E+Supervised+sequence+labelling+with+recurrent+neural+networks%2C+volume+385%2E+Springer%2C+2012%2E)
10. Griol, David, Hurtado, Lluı́s F, Segarra, Encarna, and Sanchis, Emilio. A statistical approach to spoken dialog systems design and evaluation. Speech Communication, 50(8):666–682, 2008. [find](https://scholar.google.com/scholar?q=Griol%2C+David%2C+Hurtado%2C+Llu%C4%B1%CC%81s+F%2C+Segarra%2C+Encarna%2C+and+Sanchis%2C+Emilio%2E+A+statistical+approach+to+spoken+dialog+systems+design+and+evaluation%2E+Speech+Communication%2C+50%288%29%3A666%E2%80%93682%2C+2008%2E)
11. Henderson, Matthew. Machine learning for dialog state tracking: A review. In Proceedings of The First International Workshop on Machine Learning in Spoken Language Processing, 2015. [find](https://scholar.google.com/scholar?q=Henderson%2C+Matthew%2E+Machine+learning+for+dialog+state+tracking%3A+A+review%2E+In+Proceedings+of+The+First+International+Workshop+on+Machine+Learning+in+Spoken+Language+Processing%2C+2015%2E)
12. Henderson, Matthew, Thomson, Blaise, and Williams, Jason. The second dialog state tracking challenge. In 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 263, 2014. [find](https://scholar.google.com/scholar?q=Henderson%2C+Matthew%2C+Thomson%2C+Blaise%2C+and+Williams%2C+Jason%2E+The+second+dialog+state+tracking+challenge%2E+In+15th+Annual+Meeting+of+the+Special+Interest+Group+on+Discourse+and+Dialogue%2C+pp%2E+263%2C+2014%2E)
13. Hermann, Karl Moritz, Kočiský, Tomáš, Grefenstette, Edward, Espeholt, Lasse, Kay, Will, Suleyman, Mustafa, and Blunsom, Phil. [Teaching machines to read and comprehend](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/). In Advances in Neural Information Processing Systems (NIPS), 2015. URL http://arxiv.org/abs/1506.03340. [↗](http://arxiv.org/abs/1506.03340)
14. Hill, Felix, Bordes, Antoine, Chopra, Sumit, and Weston, Jason. The goldilocks principle: Reading children’s books with explicit memory representations. arXiv preprint arXiv:1511.[02301, 2015](https://md.research.elamur.ai/free/en/papers/hill-2015-goldilocks-principle-reading-children-s/).
15. Hirschman, Lynette, Dahl, Deborah A, McKay, Donald P, Norton, Lewis M, and Linebarger, Marcia C. Beyond class a: A proposal for automatic evaluation of discourse. Technical report, DTIC Document, 1990. [find](https://scholar.google.com/scholar?q=Hirschman%2C+Lynette%2C+Dahl%2C+Deborah+A%2C+McKay%2C+Donald+P%2C+Norton%2C+Lewis+M%2C+and+Linebarger%2C+Marcia+C%2E+Beyond+class+a%3A+A+proposal+for+automatic+evaluation+of+discourse%2E+Technical+report%2C+DTIC+Document%2C+1990)
16. Hochreiter, Sepp and Schmidhuber, Jürgen. Long short-term memory. Neural computation, 9(8): 1735–1780, 1997. [find](https://scholar.google.com/scholar?q=Hochreiter%2C+Sepp+and+Schmidhuber%2C+Ju%CC%88rgen%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A+1735%E2%80%931780%2C+1997%2E)
17. Isbell, Charles Lee, Kearns, Michael, Kormann, Dave, Singh, Satinder, and Stone, Peter. Cobot in lambdamoo: A social statistics agent. In AAAI/IAAI, pp. 36–41, 2000. [find](https://scholar.google.com/scholar?q=Isbell%2C+Charles+Lee%2C+Kearns%2C+Michael%2C+Kormann%2C+Dave%2C+Singh%2C+Satinder%2C+and+Stone%2C+Peter%2E+Cobot+in+lambdamoo%3A+A+social+statistics+agent%2E+In+AAAI%2FIAAI%2C+pp%2E+36%E2%80%9341%2C+2000%2E)
18. Jafarpour, Sina, Burges, Christopher JC, and Ritter, Alan. Filter, rank, and transfer the knowledge: Learning to chat. Advances in Ranking, 10, 2010. [find](https://scholar.google.com/scholar?q=Jafarpour%2C+Sina%2C+Burges%2C+Christopher+JC%2C+and+Ritter%2C+Alan%2E+Filter%2C+rank%2C+and+transfer+the+knowledge%3A+Learning+to+chat%2E+Advances+in+Ranking%2C+10%2C+2010%2E)
19. Joulin, Armand and Mikolov, Tomas. Inferring algorithmic patterns with stack-augmented recurrent nets. arXiv preprint: 1503.01007, 2015. [find](https://scholar.google.com/scholar?q=Joulin%2C+Armand+and+Mikolov%2C+Tomas%2E+Inferring+algorithmic+patterns+with+stack%2Daugmented+recurrent+nets%2E+arXiv+preprint%3A+1503%2E01007%2C+2015%2E)
20. Kolomiyets, Oleksandr and Moens, Marie-Francine. A survey on question answering technology from an information retrieval perspective. Information Sciences, 181(24):5412–5434, 2011. [find](https://scholar.google.com/scholar?q=Kolomiyets%2C+Oleksandr+and+Moens%2C+Marie%2DFrancine%2E+A+survey+on+question+answering+technology+from+an+information+retrieval+perspective%2E+Information+Sciences%2C+181%2824%29%3A5412%E2%80%935434%2C+2011%2E)
21. Koren, Yehuda. Factorization meets the neighborhood: a multifaceted collaborative filtering model. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 426–434. ACM, 2008. [find](https://scholar.google.com/scholar?q=Koren%2C+Yehuda%2E+Factorization+meets+the+neighborhood%3A+a+multifaceted+collaborative+filtering+model%2E+In+Proceedings+of+the+14th+ACM+SIGKDD+international+conference+on+Knowledge+discovery+and+data+mining)
22. Koren, Yehuda and Bell, Robert. Advances in collaborative filtering. In Recommender systems handbook, pp. 145–186. Springer, 2011. [find](https://scholar.google.com/scholar?q=Koren%2C+Yehuda+and+Bell%2C+Robert%2E+Advances+in+collaborative+filtering%2E+In+Recommender+systems+handbook%2C+pp%2E+145%E2%80%93186%2E+Springer%2C+2011%2E)
23. Kwiatkowski, Tom, Choi, Eunsol, Artzi, Yoav, and Zettlemoyer, Luke. [Scaling semantic parsers with on-the-fly ontology matching](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2013-scaling-semantic-parsers-fly-ontology/). In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP’13), Seattle, USA, October 2013.
24. Lowe, Ryan, Pow, Nissan, Serban, Iulian, and Pineau, Joelle. [The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems](https://md.research.elamur.ai/free/en/papers/lowe-2015-ubuntu-dialogue-corpus-large-dataset/). arXiv preprint arXiv:1506.08909, 2015.
25. Mikolov, Tomas and Zweig, Geoffrey. Context dependent recurrent neural network language model. In SLT, pp. 234–239, 2012. [find](https://scholar.google.com/scholar?q=Mikolov%2C+Tomas+and+Zweig%2C+Geoffrey%2E+Context+dependent+recurrent+neural+network+language+model%2E+In+SLT%2C+pp%2E+234%E2%80%93239%2C+2012%2E)
26. Mikolov, Tomáš, Kombrink, Stefan, Burget, Lukáš, Černockỳ, Jan Honza, and Khudanpur, Sanjeev. Extensions of recurrent neural network language model. In Acoustics, Speech and Signal Processing (ICASSP), 2011 IEEE International Conference on, pp. 5528–5531. [find](https://scholar.google.com/scholar?q=Mikolov%2C+Toma%CC%81s%CC%8C%2C+Kombrink%2C+Stefan%2C+Burget%2C+Luka%CC%81s%CC%8C%2C+C%CC%8Cernocky%CC%80%2C+Jan+Honza%2C+and+Khudanpur%2C+Sanjeev%2E+Extensions+of+recurrent+neural+network+language+model%2E+In+Acoustics%2C+Speech+and+Signal+Processing+%28I)
27. Mikolov, Tomas, Chen, Kai, Corrado, Greg, and Dean, Jeffrey. [Efficient estimation of word representations in vector space](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/). arXiv:1301.3781, 2013.
28. Narasimhan, Karthik, Kulkarni, Tejas, and Barzilay, Regina. [Language understanding for text-based games using deep reinforcement learning](https://md.research.elamur.ai/free/en/papers/narasimhan-2015-language-understanding-text-based-games/). arXiv preprint arXiv:1506.08941, 2015.
29. Paek, Tim. Empirical methods for evaluating dialog systems. In Proceedings of the workshop on Evaluation for Language and Dialogue Systems-Volume 9, pp. 2. Association for Computational Linguistics, 2001. [find](https://scholar.google.com/scholar?q=Paek%2C+Tim%2E+Empirical+methods+for+evaluating+dialog+systems%2E+In+Proceedings+of+the+workshop+on+Evaluation+for+Language+and+Dialogue+Systems%2DVolume+9%2C+pp%2E+2%2E+Association+for+Computational+Linguistics%2C+2)
30. Ramachandran, Deepak, Yeh, Peter Z, Jarrold, William, Douglas, Benjamin, Ratnaparkhi, Adwait, Provine, Ronald, Mendel, Jeremy, and Emfield, Adam. An end-to-end dialog system for tv program discovery. In Spoken Language Technology Workshop (SLT), 2014 IEEE, pp. 602–607. [find](https://scholar.google.com/scholar?q=Ramachandran%2C+Deepak%2C+Yeh%2C+Peter+Z%2C+Jarrold%2C+William%2C+Douglas%2C+Benjamin%2C+Ratnaparkhi%2C+Adwait%2C+Provine%2C+Ronald%2C+Mendel%2C+Jeremy%2C+and+Emfield%2C+Adam%2E+An+end%2Dto%2Dend+dialog+system+for+tv+program+discovery%2E+)
31. Ritter, Alan, Cherry, Colin, and Dolan, William B. [Data-driven response generation in social media](https://md.research.elamur.ai/free/en/papers/ritter-2011-data-driven-response-generation-social/). In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 583–593. Association for Computational Linguistics, 2011.
32. Rojas-Barahona, Lina M, Lorenzo, Alejandra, and Gardent, Claire. An end-to-end evaluation of two situated dialog systems. In Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 10–19. Association for Computational Linguistics, 2012. [find](https://scholar.google.com/scholar?q=Rojas%2DBarahona%2C+Lina+M%2C+Lorenzo%2C+Alejandra%2C+and+Gardent%2C+Claire%2E+An+end%2Dto%2Dend+evaluation+of+two+situated+dialog+systems%2E+In+Proceedings+of+the+13th+Annual+Meeting+of+the+Special+Interest+Group+on+Dis)
33. Rush, Alexander M, Chopra, Sumit, and Weston, Jason. [A neural attention model for abstractive sentence summarization](https://md.research.elamur.ai/free/en/papers/rush-2015-neural-attention-model-abstractive-sentence/). Proceedings of EMNLP, 2015.
34. Shang, Lifeng, Lu, Zhengdong, and Li, Hang. [Neural responding machine for short-text conversation](https://md.research.elamur.ai/free/en/papers/shang-2015-neural-responding-machine-short-text/). arXiv preprint arXiv:1503.02364, 2015.
35. Sordoni, Alessandro, Galley, Michel, Auli, Michael, Brockett, Chris, Ji, Yangfeng, Mitchell, Margaret, Nie, Jian-Yun, Gao, Jianfeng, and Dolan, Bill. [A neural network approach to context-sensitive generation of conversational responses](https://md.research.elamur.ai/free/en/papers/sordoni-2015-neural-network-approach-context-sensitive/). Proceedings of NAACL, 2015.
36. Sukhbaatar, Sainbayar, Szlam, Arthur, Weston, Jason, and Fergus, Rob. End-to-end memory networks. Proceedings of NIPS, 2015. [find](https://scholar.google.com/scholar?q=Sukhbaatar%2C+Sainbayar%2C+Szlam%2C+Arthur%2C+Weston%2C+Jason%2C+and+Fergus%2C+Rob%2E+End%2Dto%2Dend+memory+networks%2E+Proceedings+of+NIPS%2C+2015%2E)
37. Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc VV. [Sequence to sequence learning with neural networks](https://md.research.elamur.ai/free/en/papers/sutskever-2014-sequence-sequence-learning-neural-networks/). In Advances in neural information processing systems, pp. 3104–3112, 2014.
38. Vinyals, Oriol and Le, Quoc. [A neural conversational model](https://md.research.elamur.ai/free/en/papers/vinyals-2015-neural-conversational-model/). arXiv preprint arXiv:1506.05869, 2015.
39. Walker, Marilyn A, Litman, Diane J, Kamm, Candace A, and Abella, Alicia. [Paradise: A framework for evaluating spoken dialogue agents](https://md.research.elamur.ai/free/en/papers/walker-1997-paradise-framework-evaluating-spoken-dialogue/). In Proceedings of the eighth conference on European chapter of the Association for Computational Linguistics, pp. 271–280. Association for Computational Linguistics,…
40. Walker, Marilyn A, Prasad, Rashmi, and Stent, Amanda. A trainable generator for recommendations in multimodal dialog. In INTERSPEECH, 2003. [find](https://scholar.google.com/scholar?q=Walker%2C+Marilyn+A%2C+Prasad%2C+Rashmi%2C+and+Stent%2C+Amanda%2E+A+trainable+generator+for+recommendations+in+multimodal+dialog%2E+In+INTERSPEECH%2C+2003%2E)
41. Weston, J., Bordes, A., Chopra, S., and Mikolov, T. [Towards AI-complete question answering: A set of prerequisite toy tasks](https://md.research.elamur.ai/free/en/papers/weston-2015-towards-ai-complete-question-answering/). arXiv preprint: 1502.05698, 2015a.
42. Weston, Jason, Yee, Hector, and Weiss, Ron J. Learning to rank recommendations with the k-order statistic loss. In Proceedings of the 7th ACM conference on Recommender systems, pp. 245–248. ACM, 2013. [find](https://scholar.google.com/scholar?q=Weston%2C+Jason%2C+Yee%2C+Hector%2C+and+Weiss%2C+Ron+J%2E+Learning+to+rank+recommendations+with+the+k%2Dorder+statistic+loss%2E+In+Proceedings+of+the+7th+ACM+conference+on+Recommender+systems%2C+pp%2E+245%E2%80%93248%2E+ACM%2C+2013%2E)
43. Weston, Jason, Bordes, Antoine, Chopra, Sumit, and Mikolov, Tomas. [Towards ai-complete question answering: a set of prerequisite toy tasks](https://md.research.elamur.ai/free/en/papers/weston-2015-towards-ai-complete-question-answering/). arXiv preprint arXiv:1502.05698, 2015b.
44. Weston, Jason, Chopra, Sumit, and Bordes, Antoine. Memory networks. Proceedings of ICLR, 2015c. [find](https://scholar.google.com/scholar?q=Weston%2C+Jason%2C+Chopra%2C+Sumit%2C+and+Bordes%2C+Antoine%2E+Memory+networks%2E+Proceedings+of+ICLR%2C+2015c%2E)
45. Whittaker, Steve, Walker, Marilyn A, and Moore, Johanna D. Fish or fowl: A wizard of oz evaluation of dialogue strategies in the restaurant domain. In LREC, 2002. [find](https://scholar.google.com/scholar?q=Whittaker%2C+Steve%2C+Walker%2C+Marilyn+A%2C+and+Moore%2C+Johanna+D%2E+Fish+or+fowl%3A+A+wizard+of+oz+evaluation+of+dialogue+strategies+in+the+restaurant+domain%2E+In+LREC%2C+2002%2E)
46. Williams, Jason, Raux, Antoine, Ramachandran, Deepak, and Black, Alan. The dialog state tracking challenge. In Proceedings of the SIGDIAL 2013 Conference, pp. 404–413, 2013. [find](https://scholar.google.com/scholar?q=Williams%2C+Jason%2C+Raux%2C+Antoine%2C+Ramachandran%2C+Deepak%2C+and+Black%2C+Alan%2E+The+dialog+state+tracking+challenge%2E+In+Proceedings+of+the+SIGDIAL+2013+Conference%2C+pp%2E+404%E2%80%93413%2C+2013%2E)

*Updated 2 November 2026*