---
title: "Wizard of Wikipedia: Knowledge-Powered Conversational agents"
description: "The authors address a long-standing problem in open-domain dialogue: sequence-to-sequence… 28 references, cited by 495 papers here."
authors: ["Emily Dinan", "Stephen Roller", "Kurt Shuster", "Angela Fan", "Michael Auli", "Jason Weston"]
year: 2018
venue: "arXiv:1811.01241"
source: "https://arxiv.org/abs/1811.01241"
doi: "10.48550/arxiv.1811.01241"
arxiv: "1811.01241"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/dinan-2018-wizard-wikipedia-knowledge-powered-conversational/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/dinan-2018-wizard-wikipedia-knowledge-powered-conversational/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/) / Wizard of Wikipedia: Knowledge-Powered Conversational agents

# Wizard of Wikipedia: Knowledge-Powered Conversational agents

**Authors:** [Emily Dinan](https://md.research.elamur.ai/free/en/authors/emily-dinan/), [Stephen Roller](https://md.research.elamur.ai/free/en/authors/stephen-roller/), [Kurt Shuster](https://md.research.elamur.ai/free/en/authors/kurt-shuster/), [Angela Fan](https://md.research.elamur.ai/free/en/authors/angela-fan/), [Michael Auli](https://md.research.elamur.ai/free/en/authors/michael-auli/) and [Jason Weston](https://md.research.elamur.ai/free/en/authors/jason-weston/) · **Year:** 2018 · **Venue:** arXiv:1811.01241 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1811.01241) · [PDF](https://arxiv.org/pdf/1811.01241) · [DOI 10.48550/arxiv.1811.01241](https://doi.org/10.48550/arxiv.1811.01241) · [arXiv 1811.01241](https://arxiv.org/abs/1811.01241) · [OpenAlex](https://openalex.org/W2898875342)

**Citations:** 572 citations (OpenAlex, 2026-10-14) · [495](https://md.research.elamur.ai/free/en/papers/dinan-2018-wizard-wikipedia-knowledge-powered-conversational/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

The authors address a long-standing problem in open-domain dialogue: sequence-to-sequence chatbots tend to produce generic phrases that the model has simply memorized in its weights, and they rarely draw on external knowledge. One reason is the lack of a training set in which the conversation is explicitly grounded in a source of facts. The Facebook AI Research team therefore crowdsourced the Wizard of Wikipedia corpus. In each conversation, two people play different roles: a curious "apprentice" asks questions, while the "wizard" sees automatically retrieved passages from Wikipedia and marks the sentence on which their reply is based. On this data, the authors trained the Transformer Memory Networks architecture in two variants, retrieval and generative. It combines the memory of Memory Networks, from which the model selects the relevant knowledge, with a Transformer that encodes the text and generates the response. The best models conduct knowledgeable conversations on open-domain topics, as confirmed by both automatic metrics and human evaluations. However, the example dialogues contain clear factual errors. The main value of the work is that it provides an open benchmark for knowledge-grounded chatbots.

## Key points

- The corpus contains 22,311 dialogues and 201,999 utterances on 1,365 topics, each linked to a Wikipedia article. The split is as follows: 166,787 utterances in train, 17,715 in valid, and 17,497 in test.
- The test set has two parts. Test Seen covers 533 topics that already appeared in training, while Test Unseen covers 58 entirely new topics.
- The knowledge base comprises 5.4M articles and 93M sentences. Passages for the "wizard" were selected by the TF-IDF retriever from Open-SQuAD (Chen et al., 2017): the top seven articles for the two most recent utterances plus the article on the initial topic.
- The Transformer MemNet models (retrieval, two-stage generative, etc.) were compared with baseline Memory Networks and a Transformer without access to knowledge.
- The data and tasks are published in ParlAI (parl.ai/projects/wizard_of_wikipedia).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🗨️ Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) · 624 works

💡 Concepts (1): [💡 Chatbots](https://md.research.elamur.ai/free/en/concepts/chatbot/) · 54 works

## Cite

Cite as: Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli and Jason Weston. 2018. Wizard of Wikipedia: Knowledge-Powered Conversational agents. arXiv:1811.01241.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{dinan2018wizard,
  title = {{Wizard of Wikipedia: Knowledge-Powered Conversational agents}},
  author = {Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli and Jason Weston},
  year = {2018},
  doi = {10.48550/arxiv.1811.01241},
  eprint = {1811.01241},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1811.01241}
}
```

- [← previous SemEval-2018 Task 1: Affect in Tweets](https://md.research.elamur.ai/free/en/papers/mohammad-2018-semeval-2018-task-1-affect/)
- [next → QuAC: Question Answering in Context](https://md.research.elamur.ai/free/en/papers/choi-2018-quac-question-answering-context/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/)

## References

28 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Niels O Bernsen, Hans Dybkjær, and Laila Dybkjær. Designing interactive speech systems: From first ideas to user testing. Springer Science & Business Media, 2012. [find](https://scholar.google.com/scholar?q=Niels+O+Bernsen%2C+Hans+Dybkj%C3%A6r%2C+and+Laila+Dybkj%C3%A6r%2E+Designing+interactive+speech+systems%3A+From+first+ideas+to+user+testing%2E+Springer+Science+%26+Business+Media%2C+2012%2E)
2. Antoine Bordes, Y-Lan Boureau, and Jason Weston. Learning end-to-end goal-oriented dialog. In Proceedings of the International Conference on Learning Representations (ICLR), 2017. [find](https://scholar.google.com/scholar?q=Antoine+Bordes%2C+Y%2DLan+Boureau%2C+and+Jason+Weston%2E+Learning+end%2Dto%2Dend+goal%2Doriented+dialog%2E+In+Proceedings+of+the+International+Conference+on+Learning+Representations+%28ICLR%29%2C+2017%2E)
3. Daniel Cer, Yinfei Yang, Sheng yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. [Universal sentence encoder](https://md.research.elamur.ai/free/en/papers/cer-2018-universal-sentence-encoder/). arXiv preprint arXiv:1803.11175, 2018.
4. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. [Reading wikipedia to answer open-domain questions](https://md.research.elamur.ai/free/en/papers/chen-2017-reading-wikipedia-answer-open-domain/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pp. 1870–1879. Association for Computational Linguistics, 2017.
5. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. [QuAC: Question answering in context](https://md.research.elamur.ai/free/en/papers/choi-2018-quac-question-answering-context/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 2018.
6. Jesse Dodge, Andreea Gane, Xiang Zhang, Antoine Bordes, Sumit Chopra, Alexander Miller, Arthur Szlam, and Jason Weston. Evaluating prerequisite qualities for learning end-to-end dialog systems. In Proceedings of the International Conference on Learning Representations (ICLR), 2016. [find](https://scholar.google.com/scholar?q=Jesse+Dodge%2C+Andreea+Gane%2C+Xiang+Zhang%2C+Antoine+Bordes%2C+Sumit+Chopra%2C+Alexander+Miller%2C+Arthur+Szlam%2C+and+Jason+Weston%2E+Evaluating+prerequisite+qualities+for+learning+end%2Dto%2Dend+dialog+systems%2E+In+Pro)
7. Layla El Asri, Hannes Schulz, Shikhar Sharma, Jeremie Zumer, Justin Harris, Emery Fine, Rahul Mehrotra, and Kaheer Suleman. [Frames: a corpus for adding memory to goal-oriented dialogue systems](https://md.research.elamur.ai/free/en/papers/asri-2017-frames-corpus-adding-memory-goal/). In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pp. 207–219, Saarbrücken, Ge…
8. Angela Fan, David Grangier, and Michael Auli. [Controllable abstractive summarization](https://md.research.elamur.ai/free/en/papers/fan-2018-controllable-abstractive-summarization/). In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pp. 45–54. Association for Computational Linguistics, 2018.
9. Marjan Ghazvininejad, Chris Brockett, Ming-Wei Chang, Bill Dolan, Jianfeng Gao, Wen-tau Yih, and Michel Galley. A knowledge-grounded neural conversation model. In Proceedings of the Conference on Association for the Advancement of Artificial Intelligence (AAAI), 2018. [find](https://scholar.google.com/scholar?q=Marjan+Ghazvininejad%2C+Chris+Brockett%2C+Ming%2DWei+Chang%2C+Bill+Dolan%2C+Jianfeng+Gao%2C+Wen%2Dtau+Yih%2C+and+Michel+Galley%2E+A+knowledge%2Dgrounded+neural+conversation+model%2E+In+Proceedings+of+the+Conference+on+Asso)
10. Matthew Henderson, Blaise Thomson, and Jason D Williams. The second dialog state tracking challenge. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pp. 263–272, 2014. [find](https://scholar.google.com/scholar?q=Matthew+Henderson%2C+Blaise+Thomson%2C+and+Jason+D+Williams%2E+The+second+dialog+state+tracking+challenge%2E+In+Proceedings+of+the+15th+Annual+Meeting+of+the+Special+Interest+Group+on+Discourse+and+Dialogue+%28)
11. Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun-hsuan Sung, L’aszl’o Luk’acs, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. Efficient Natural Language Response Suggestion for Smart Reply. arXiv preprint arXiv:1705.00652, 2017. [find](https://scholar.google.com/scholar?q=Matthew+Henderson%2C+Rami+Al%2DRfou%2C+Brian+Strope%2C+Yun%2Dhsuan+Sung%2C+L%E2%80%99aszl%E2%80%99o+Luk%E2%80%99acs%2C+Ruiqi+Guo%2C+Sanjiv+Kumar%2C+Balint+Miklos%2C+and+Ray+Kurzweil%2E+Efficient+Natural+Language+Response+Suggestion+for+Smart+Repl)
12. Jiwei Li, Will Monroe, and Daniel Jurafsky. A simple, fast diverse decoding algorithm for neural generation. arXiv preprint arXiv:1611.08562, 2016. [find](https://scholar.google.com/scholar?q=Jiwei+Li%2C+Will+Monroe%2C+and+Daniel+Jurafsky%2E+A+simple%2C+fast+diverse+decoding+algorithm+for+neural+generation%2E+arXiv+preprint+arXiv%3A1611%2E08562%2C+2016%2E)
13. Pierre-Emmanuel Mazaré, Samuel Humeau, Martin Raison, and Antoine Bordes. [Training millions of personalized dialogue agents](https://md.research.elamur.ai/free/en/papers/mazare-2018-training-millions-personalized-dialogue-agents/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 2018. Association for Computational Linguistics.
14. Nikita Moghe, Siddhartha Arora, Suman Banerjee, and Mitesh M. Khapra. [Towards exploiting background knowledge for building conversation systems](https://md.research.elamur.ai/free/en/papers/moghe-2018-exploiting-background-knowledge-building-conversation/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2322–2332. Association for Computational Linguistics, 2018.…
15. Prasanna Parthasarathi and Joelle Pineau. Extending neural generative conversational model using external knowledge sources. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 2018. [find](https://scholar.google.com/scholar?q=Prasanna+Parthasarathi+and+Joelle+Pineau%2E+Extending+neural+generative+conversational+model+using+external+knowledge+sources%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Langu)
16. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. [SQuAD: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383–2392, Austin, Texas, November 2016. Association for Computational Lingui…
17. Rico Sennrich, Barry Haddow, and Alexandra Birch. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pp. 1715–1725, Berlin, Germany, August 2016. Association for Computational Linguistics.
18. Iulian Vlad Serban, Ryan Lowe, Laurent Charlin, and Joelle Pineau. Generative deep neural networks for dialogue: A short review. arXiv preprint arXiv:1611.06216, 2016. — [arXiv:1611.06216](https://arxiv.org/abs/1611.06216) [find](https://scholar.google.com/scholar?q=Iulian+Vlad+Serban%2C+Ryan+Lowe%2C+Laurent+Charlin%2C+and+Joelle+Pineau%2E+Generative+deep+neural+networks+for+dialogue%3A+A+short+review%2E+arXiv+preprint+arXiv%3A1611%2E06216%2C+2016%2E+%E2%80%94+%5BarXiv%3A1611%2E06216%5D%28https%3A%2F%2Farx)
19. Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. [A neural network approach to context-sensitive generation of conversational responses](https://md.research.elamur.ai/free/en/papers/sordoni-2015-neural-network-approach-context-sensitive/). arXiv preprint arXiv:1506.06714, 2015.
20. Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. End-to-end memory networks. In Advances in Neural Information Processing Systems, pp. 2440–2448, 2015. [find](https://scholar.google.com/scholar?q=Sainbayar+Sukhbaatar%2C+Jason+Weston%2C+Rob+Fergus%2C+et+al%2E+End%2Dto%2Dend+memory+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pp%2E+2440%E2%80%932448%2C+2015%2E)
21. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. [Sequence to sequence learning with neural networks](https://md.research.elamur.ai/free/en/papers/sutskever-2014-sequence-sequence-learning-neural-networks/). In Advances in Neural Information Processing Systems, pp. 3104–3112, 2014.
22. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, pp. 5998–6008, 2017.
23. Ashwin K. Vijayakumar, Michael Cogswell, Ramprasaath R. Selvaraju, Qing Sun, Stefan Lee, David J. Crandall, and Dhruv Batra. [Diverse beam search: Decoding diverse solutions from neural sequence models](https://md.research.elamur.ai/free/en/papers/vijayakumar-2016-diverse-beam-search-decoding-diverse/). arXiv preprint arXiv:1610.02424, 2016.
24. Oriol Vinyals and Quoc Le. [A neural conversational model](https://md.research.elamur.ai/free/en/papers/vinyals-2015-neural-conversational-model/). In Proceedings of the 31st International Conference on Machine Learning, Deep Learning Workshop, Lille, France, 2015.
25. Tsung-Hsien Wen, David Vandyke, Nikola Mrksic, Milica Gasic, Lina M Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. [A network-based end-to-end trainable task-oriented dialogue system](https://md.research.elamur.ai/free/en/papers/wen-2017-network-based-end-end-trainable/). arXiv preprint arXiv:1604.04562, 2016.
26. Ledell Wu, Adam Fisch, Sumit Chopra, Keith Adams, Antoine Bordes, and Jason Weston. Starspace: Embed all the things! arXiv preprint arXiv:1709.03856, 2017. — [arXiv:1709.03856](https://arxiv.org/abs/1709.03856) [find](https://scholar.google.com/scholar?q=Ledell+Wu%2C+Adam+Fisch%2C+Sumit+Chopra%2C+Keith+Adams%2C+Antoine+Bordes%2C+and+Jason+Weston%2E+Starspace%3A+Embed+all+the+things%21+arXiv+preprint+arXiv%3A1709%2E03856%2C+2017%2E+%E2%80%94+%5BarXiv%3A1709%2E03856%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1)
27. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. [Personalizing dialogue agents: I have a dog, do you have pets too?](https://md.research.elamur.ai/free/en/papers/zhang-2018-personalizing-dialogue-agents-i-have/) In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pp. 2204–2213, Melbourne, Australia, July 2018. Assoc…
28. Hao Zhou, Tom Young, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. Commonsense knowledge aware conversation generation with graph attention. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI), pp. 4623–4629. International Joint Conferen… [find](https://scholar.google.com/scholar?q=Hao+Zhou%2C+Tom+Young%2C+Minlie+Huang%2C+Haizhou+Zhao%2C+Jingfang+Xu%2C+and+Xiaoyan+Zhu%2E+Commonsense+knowledge+aware+conversation+generation+with+graph+attention%2E+In+Proceedings+of+the+Twenty%2DSeventh+Internatio)

*Updated 29 October 2026*