---
title: "Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset — Byrne 2019"
description: "Data-driven dialog systems have been held back by a shortage of high-quality, goal-oriented… 18.19 BLEU. 33 references, cited by 105 papers here."
authors: ["Bill Byrne", "Karthik Krishnamoorthi", "Chinnadhurai Sankar", "Arvind Neelakantan", "Ben Goodrich", "Daniel Duckworth", "Semih Yavuz", "Amit Dubey", "Kyu-Young Kim", "Andy Cedilnik"]
year: 2019
venue: "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)"
source: "https://aclanthology.org/D19-1459/"
doi: "10.18653/v1/d19-1459"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/byrne-2019-taskmaster-1-realistic-diverse-dialog/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/byrne-2019-taskmaster-1-realistic-diverse-dialog/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/) / Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset

# Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset

**Authors:** [Bill Byrne](https://md.research.elamur.ai/free/en/authors/bill-byrne/), Karthik Krishnamoorthi, Chinnadhurai Sankar, [Arvind Neelakantan](https://md.research.elamur.ai/free/en/authors/arvind-neelakantan/), [Ben Goodrich](https://md.research.elamur.ai/free/en/authors/ben-goodrich/), [Daniel Duckworth](https://md.research.elamur.ai/free/en/authors/daniel-duckworth/), [Semih Yavuz](https://md.research.elamur.ai/free/en/authors/semih-yavuz/), Amit Dubey, Kyu-Young Kim and Andy Cedilnik · **Year:** 2019 · **Venue:** Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D19-1459/) · [PDF](https://aclanthology.org/D19-1459.pdf) · [DOI 10.18653/v1/d19-1459](https://doi.org/10.18653/v1/d19-1459) · [OpenAlex](https://openalex.org/W2985067290)

**Citations:** 139 citations (OpenAlex, 2026-10-14) · [105](https://md.research.elamur.ai/free/en/papers/byrne-2019-taskmaster-1-realistic-diverse-dialog/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
Data-driven dialog systems have been held back by a shortage of high-quality, goal-oriented conversational data. Existing corpora often come from deployed systems that carry their limitations, or they rely on small knowledge bases and tight scripts. The authors built Taskmaster-1, a corpus of 13,215 task-based English dialogs across six everyday service domains, using two collection methods. In the first, crowdworkers spoke with trained human operators who posed as an automated assistant (a spoken Wizard of Oz setup). In the second, individual crowdworkers wrote both sides of a conversation from a scenario ("self-dialogs"). Instead of full dialog-state or dialog-act schemes, the dialogs are annotated with API calls and their arguments, which is cheaper and easier to apply consistently. Compared with MultiWOZ, Taskmaster-1 has a larger vocabulary and more real-world entities, and language models find it harder to fit. Baselines range from n-gram models to a Transformer. Automatic metrics broadly agreed with human judgments. The resource supports research on task-oriented dialog and on spoken versus written conversation.

## Key points
- **Composition:** 5,507 spoken two-person (WOz) dialogs and 7,708 written self-dialogs.
  - Six domains: pizza ordering, auto repair appointments, ride service, movie tickets, coffee orders, and restaurant reservations.
  - Dialogs average roughly 22–23 utterances.
- **Comparison with MultiWOZ:** The self-dialogs contain 21,894 unique words versus 19,175 in MultiWOZ, along with many more unique named entities.
- **Response generation:** Baselines included 3/4-gram, LSTM (with and without attention), fconv Convolution, Transformer, and GPT-2 (117M).
  - The Transformer achieved the best automatic scores (PPL 18.19, BLEU 6.11) and was ranked first by human judges.
  - LSTM-attention received the highest LIKERT rating (3.51).
- **Human evaluation reliability:** Krippendorff's alpha was 0.29 for ranking and 0.21 for rating, so ranking produced more consistent judgments.
- **API argument prediction:** A Transformer with a copy mechanism reached 51.79% micro F1, compared with 48.73% for the plain Transformer. The dataset is available at g.co/dataset/taskmaster-1.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [🗨️ Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/)

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 272 works · 255,532 citations
- [Université de Montréal](https://md.research.elamur.ai/free/en/universities/universite-de-montreal/) — University, Montreal, Canada · 42 works · 71,559 citations
- [University of California, Santa Barbara](https://md.research.elamur.ai/free/en/universities/university-of-california-santa-barbara/) — University, Santa Barbara, United States · 24 works · 7,888 citations

## Cite

Cite as: Bill Byrne, Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Ben Goodrich, Daniel Duckworth, Semih Yavuz, Amit Dubey, Kyu-Young Kim and Andy Cedilnik. 2019. Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{byrne2019taskmaster,
  title = {{Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset}},
  author = {Bill Byrne, Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Ben Goodrich, Daniel Duckworth, Semih Yavuz, Amit Dubey, Kyu-Young Kim and Andy Cedilnik},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1459},
  url = {https://doi.org/10.18653/v1/d19-1459}
}
```

- [← previous Can You Unpack That? Learning to Rewrite Questions-in-Context](https://md.research.elamur.ai/free/en/papers/elgohary-2019-can-you-unpack-that-learning/)
- [next → Training Neural Machine Translation to Apply Terminology Constraints](https://md.research.elamur.ai/free/en/papers/dinu-2019-training-neural-machine-translation-apply/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/)

## References

33 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Amazon. 2019. Alexa skills. [find](https://scholar.google.com/scholar?q=Amazon%2E+2019%2E+Alexa+skills%2E)
2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. ICLR. [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
3. Antoine Bordes, Y-Lan Boureau, and Jason Weston. 2017. Learning end-to-end goal-oriented dialog. ICLR. [find](https://scholar.google.com/scholar?q=Antoine+Bordes%2C+Y%2DLan+Boureau%2C+and+Jason+Weston%2E+2017%2E+Learning+end%2Dto%2Dend+goal%2Doriented+dialog%2E+ICLR%2E)
4. Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. 2018. Multiwoz - a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. EMNLP. [find](https://scholar.google.com/scholar?q=Pawe%C5%82+Budzianowski%2C+Tsung%2DHsien+Wen%2C+Bo%2DHsiang+Tseng%2C+In%CC%83igo+Casanueva%2C+Stefan+Ultes%2C+Osman+Ramadan%2C+and+Milica+Gasic%2E+2018%2E+Multiwoz+%2D+a+large%2Dscale+multi%2Ddomain+wizard%2Dof%2Doz+dataset+for+task%2Doriente)
5. Wallace Chafe and Deborah Tannen. 1987. The relation between written and spoken language. Annual Review of Anthropology. [find](https://scholar.google.com/scholar?q=Wallace+Chafe+and+Deborah+Tannen%2E+1987%2E+The+relation+between+written+and+spoken+language%2E+Annual+Review+of+Anthropology%2E)
6. J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y. N. Dauphin. 2017. Convolutional Sequence to Sequence Learning. Arxiv. [our card](https://md.research.elamur.ai/free/en/papers/gehring-2017-convolutional-sequence-sequence-learning/)
7. Ryan Lowe, Nissan Pow, Iulian V. Serban, and Joelle Pineau. 2015. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. SIGDIAL. [our card](https://md.research.elamur.ai/free/en/papers/lowe-2015-ubuntu-dialogue-corpus-large-dataset/)
8. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. ICLR. [our card](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/)
9. Nikita Moghe, Siddhartha Arora, Suman Banerjee, and Mitesh M. Khapra. 2018. Towards exploiting background knowledge for building conversation systems. EMNLP. [our card](https://md.research.elamur.ai/free/en/papers/moghe-2018-exploiting-background-knowledge-building-conversation/)
10. Aron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alexander Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. 2016. Wavenet: A generative model for raw audio. Arxiv. [our card](https://md.research.elamur.ai/free/en/papers/oord-2016-wavenet-generative-model-raw-audio/)
11. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. NAACL Demonstrations. [our card](https://md.research.elamur.ai/free/en/papers/ott-2019-fairseq-fast-extensible-toolkit-sequence/)
12. Google. 2019. Actions on google. [find](https://scholar.google.com/scholar?q=Google%2E+2019%2E+Actions+on+google%2E)
13. Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. ACL. [our card](https://md.research.elamur.ai/free/en/papers/gu-2016-incorporating-copying-mechanism-sequence-sequence/)
14. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Arxiv. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+Arxiv%2E)
15. Matthew Henderson, Blaise Thomson, and Steve Young. 2013. Deep neural network approach for the dialog state tracking challenge. SIGDIAL. [find](https://scholar.google.com/scholar?q=Matthew+Henderson%2C+Blaise+Thomson%2C+and+Steve+Young%2E+2013%2E+Deep+neural+network+approach+for+the+dialog+state+tracking+challenge%2E+SIGDIAL%2E)
16. Antoine Raux, Brian Langner, Alan W Black, and Maxine Eskenazi. 2003. Lets go: Improving spoken dialog systems for the elderly and non-natives. Eurospeech. [find](https://scholar.google.com/scholar?q=Antoine+Raux%2C+Brian+Langner%2C+Alan+W+Black%2C+and+Maxine+Eskenazi%2E+2003%2E+Lets+go%3A+Improving+spoken+dialog+systems+for+the+elderly+and+non%2Dnatives%2E+Eurospeech%2E)
17. Geoffrey Hinton, Li Deng, Dong Yu, George Dahl, Abdel rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara Sainath, and Brian Kingsbury. 2012. Deep neural networks for acoustic modeling in speech recognition. Signal Processing Magazine. [find](https://scholar.google.com/scholar?q=Geoffrey+Hinton%2C+Li+Deng%2C+Dong+Yu%2C+George+Dahl%2C+Abdel+rahman+Mohamed%2C+Navdeep+Jaitly%2C+Andrew+Senior%2C+Vincent+Vanhoucke%2C+Patrick+Nguyen%2C+Tara+Sainath%2C+and+Brian+Kingsbury%2E+2012%2E+Deep+neural+networks+fo)
18. Lina Maria Rojas-Barahona, Milica Gasic, Nikola Mrksic, Pei-Hao Su, Stefan Ultes, Tsung-Hsien Wen, Steve J. Young, and David Vandyke. 2017. A network-based end-to-end trainable task-oriented dialogue system. EACL. [find](https://scholar.google.com/scholar?q=Lina+Maria+Rojas%2DBarahona%2C+Milica+Gasic%2C+Nikola+Mrksic%2C+Pei%2DHao+Su%2C+Stefan+Ultes%2C+Tsung%2DHsien+Wen%2C+Steve+J%2E+Young%2C+and+David+Vandyke%2E+2017%2E+A+network%2Dbased+end%2Dto%2Dend+trainable+task%2Doriented+dialogue+)
19. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+Computation%2E)
20. Norman P. Jouppi, Cliff Young, Nishant Patil, David Patterson, and Gaurav Agrawal et al. 2017. In-datacenter performance analysis of a tensor processing unit. ISCA. [find](https://scholar.google.com/scholar?q=Norman+P%2E+Jouppi%2C+Cliff+Young%2C+Nishant+Patil%2C+David+Patterson%2C+and+Gaurav+Agrawal+et+al%2E+2017%2E+In%2Ddatacenter+performance+analysis+of+a+tensor+processing+unit%2E+ISCA%2E)
21. John F Kelley. 1984. An iterative design methodology for user-friendly natural language office information applications. ACM Transactions on Information Systems (TOIS). [find](https://scholar.google.com/scholar?q=John+F+Kelley%2E+1984%2E+An+iterative+design+methodology+for+user%2Dfriendly+natural+language+office+information+applications%2E+ACM+Transactions+on+Information+Systems+%28TOIS%29%2E)
22. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. ICLR. [find](https://scholar.google.com/scholar?q=Diederik+P%2E+Kingma+and+Jimmy+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+ICLR%2E)
23. Ben Krause, Marco Damonte, Mihai Dobre, Daniel Duma, Joachim Fainberg, Federico Fancellu, Emmanuel Kahembwe, Jianpeng Cheng, and Bonnie Webber. 2017. Edina: Building an open domain socialbot with self-dialogues. Arxiv. [find](https://scholar.google.com/scholar?q=Ben+Krause%2C+Marco+Damonte%2C+Mihai+Dobre%2C+Daniel+Duma%2C+Joachim+Fainberg%2C+Federico+Fancellu%2C+Emmanuel+Kahembwe%2C+Jianpeng+Cheng%2C+and+Bonnie+Webber%2E+2017%2E+Edina%3A+Building+an+open+domain+socialbot+with+self)
24. Yann Lecun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature. [find](https://scholar.google.com/scholar?q=Yann+Lecun%2C+Yoshua+Bengio%2C+and+Geoffrey+Hinton%2E+2015%2E+Deep+learning%2E+Nature%2E)
25. Iulian Vlad Serban, Ryan Lowe, Peter Henderson, Laurent Charlin, and Joelle Pineau. 2017. A survey of available corpora for building data-driven dialogue systems. nnn. [find](https://scholar.google.com/scholar?q=Iulian+Vlad+Serban%2C+Ryan+Lowe%2C+Peter+Henderson%2C+Laurent+Charlin%2C+and+Joelle+Pineau%2E+2017%2E+A+survey+of+available+corpora+for+building+data%2Ddriven+dialogue+systems%2E+nnn%2E)
26. Dave Steinkrau, Patrice Y. Simard, and Ian Buck. 2005. Using gpus for machine learning algorithms. IC-DAR. [find](https://scholar.google.com/scholar?q=Dave+Steinkrau%2C+Patrice+Y%2E+Simard%2C+and+Ian+Buck%2E+2005%2E+Using+gpus+for+machine+learning+algorithms%2E+IC%2DDAR%2E)
27. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. NeurIPS. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V%2E+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+NeurIPS%2E)
28. Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. 2018. Tensor2tensor for neural machine translation. CoRR. [our card](https://md.research.elamur.ai/free/en/papers/vaswani-2018-tensor2tensor-neural-machine-translation/)
29. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. NeurIPS. [our card](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/)
30. Oriol Vinyals and Quoc V. Le. 2015. A neural conversational model. Arxiv. [our card](https://md.research.elamur.ai/free/en/papers/vinyals-2015-neural-conversational-model/)
31. Joseph Weizenbaum. 1966. Eliza a computer program for the study of natural language communication between man and machine. Computational Linguistics. [find](https://scholar.google.com/scholar?q=Joseph+Weizenbaum%2E+1966%2E+Eliza+a+computer+program+for+the+study+of+natural+language+communication+between+man+and+machine%2E+Computational+Linguistics%2E)
32. Jason Williams, Antoine Raux, and Matthew and Henderson. 2016. The dialog state tracking challenge series: A review. Dialog and Discourse. [find](https://scholar.google.com/scholar?q=Jason+Williams%2C+Antoine+Raux%2C+and+Matthew+and+Henderson%2E+2016%2E+The+dialog+state+tracking+challenge+series%3A+A+review%2E+Dialog+and+Discourse%2E)
33. Jason D. Williams and Steve J. Young. 2007. Partially observable markov decision processes for spoken dialog systems. Computer Speech & Language. [find](https://scholar.google.com/scholar?q=Jason+D%2E+Williams+and+Steve+J%2E+Young%2E+2007%2E+Partially+observable+markov+decision+processes+for+spoken+dialog+systems%2E+Computer+Speech+%26+Language%2E)