--- title: "Task-Oriented Dialogue as Dataflow Synthesis — Andreas 2020" description: "Task-oriented dialogue systems often force a choice between rigid slot-filling states, which are easy to learn but limited, and opaque neural states…" authors: ["Jacob Andreas", "John Bufe", "David Burkett", "Charles Chen", "Josh Clausman", "Jean Crawford", "Kate Crim", "Jordan DeLoach", "Leah Dorner", "Jason Eisner", "Hao Fang", "Alan Guo", "David Hall", "Kristin Hayes", "Kellie Hill", "Diana Ho", "Wendy Iwaszuk", "Smriti Jha", "Dan Klein", "Jayant Krishnamurthy", "Theo Lanman", "Percy Liang", "Christopher H. Lin", "Ilya Lintsbakh", "Andy McGovern", "Aleksandr Nisnevich", "Adam Pauls", "Dmitrij Petters", "Brent Read", "Dan Roth", "Subhro Roy", "Jesse Rusak", "Beth Short", "Div Slomin", "Ben Snyder", "Stephon Striplin", "Yu Su", "Zachary Tellman", "Sam Thomson", "Andrei Vorobev", "Izabela Witoszko", "Jason Wolfe", "Abby Wray", "Yuchen Zhang", "Alexander Zotov"] year: 2020 venue: "Transactions of the Association for Computational Linguistics" source: "https://aclanthology.org/2020.tacl-1.36/" doi: "10.1162/tacl_a_00333" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/andreas-2020-task-oriented-dialogue-as-dataflow/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/andreas-2020-task-oriented-dialogue-as-dataflow/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/) / Task-Oriented Dialogue as Dataflow Synthesis # Task-Oriented Dialogue as Dataflow Synthesis **Authors:** [Jacob Andreas](https://md.research.elamur.ai/free/en/authors/jacob-andreas/), John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, [Jason Eisner](https://md.research.elamur.ai/free/en/authors/jason-eisner/), [Hao Fang](https://md.research.elamur.ai/free/en/authors/hao-fang/), Alan Guo et al. (45 authors) · **Year:** 2020 · **Venue:** Transactions of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2020.tacl-1.36/) · [PDF](https://aclanthology.org/2020.tacl-1.36.pdf) · [DOI 10.1162/tacl\_a\_00333](https://doi.org/10.1162/tacl_a_00333) · [OpenAlex](https://openalex.org/W3087300239) **Citations:** 20 citations (openalex-snapshot · 2026-10-14) · [99](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary Task-oriented dialogue systems often force a choice between rigid slot-filling states, which are easy to learn but limited, and opaque neural states, which are flexible but hard to train. Andreas et al. (Microsoft Semantic Machines) propose a middle path. The dialogue state is a dataflow graph that grows over the conversation. At each turn, the agent converts the user's utterance into an executable program that adds nodes to this graph. Two metacomputation operators handle the context-dependent parts of conversation: - **refer** retrieves a salient earlier node, covering phenomena such as anaphora. - **revise** splices a modified constraint into an earlier computation, covering corrections and follow-ups. Errors, such as an ambiguous attendee, are represented as exceptions that later turns can repair. The authors release SMCalFlow, a large English corpus with program annotations for every turn. On this corpus, the explicit reference machinery noticeably reduces the errors of a standard seq2seq-with-copying model. When MultiWOZ is converted into dataflow form, a generic seq2seq model performs on par with the best task-specific state tracker. The work matters because it shows that compositional, cross-domain user goals can be represented and also learned well. ## Key points - **SMCalFlow corpus:** about 40K English dialogues covering calendar events, weather, places and people. The authors describe it as the largest goal-oriented dialogue dataset at the time. The dataset, code and a public leaderboard are released. - **Effect of explicit reference:** on SMCalFlow, adding explicit reference operators cuts the mistakes made by a sequence-to-sequence model with a copy mechanism (See et al., 2017) by 5.9% across all turns. On turns that refer back to earlier turns, the reduction is 10.9%. - **MultiWOZ results:** the authors provide scripts that automatically convert MultiWOZ into dataflow programs. With this representation, an off-the-shelf seq2seq model matches the strongest existing task-specific state tracking model. - **Core operators:** `refer` and `revise` reuse fragments of the graph non-destructively. Exceptions such as `UnderconstrainedException` block evaluation until a later turn fixes them. - **Error analysis:** a manual review of 100 mispredicted SMCalFlow turns found four main error types: underprediction, entity linking, fencing (out-of-scope detection) and ambiguity. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 99 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 94 not yet. - [Diverse Demonstrations Improve In-context Compositional Generalization](https://md.research.elamur.ai/free/en/papers/levy-2023-diverse-demonstrations-improve-context-compositional/) - [Unified Demonstration Retriever for In-Context Learning](https://md.research.elamur.ai/free/en/papers/li-2023-unified-demonstration-retriever-context-learning/) - [Learning To Retrieve Prompts for In-Context Learning](https://md.research.elamur.ai/free/en/papers/rubin-2021-learning-retrieve-prompts-context-learning/) - [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://md.research.elamur.ai/free/en/papers/yao-2022-webshop-towards-scalable-real-world/) - [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://md.research.elamur.ai/free/en/papers/yao-2024-tau-bench-benchmark-tool/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @article{andreas2020task, title = {{Task-Oriented Dialogue as Dataflow Synthesis}}, author = {Andreas, Jacob and Bufe, John and Burkett, David and Chen, Charles and Clausman, Josh and Crawford, Jean and Crim, Kate and DeLoach, Jordan and Dorner, Leah and Eisner, Jason and Fang, Hao and Guo, Alan and Hall, David and Hayes, Kristin and Hill, Kellie and Ho, Diana and Iwaszuk, Wendy and Jha, Smriti and Klein, Dan and Krishnamurthy, Jayant and Lanman, Theo and Liang, Percy and Lin, Christopher H. and Lintsbakh, Ilya and McGovern, Andy and Nisnevich, Aleksandr and Pauls, Adam and Petters, Dmitrij and Read, Brent and Roth, Dan and Roy, Subhro and Rusak, Jesse and Short, Beth and Slomin, Div and Snyder, Ben and Striplin, Stephon and Su, Yu and Tellman, Zachary and Thomson, Sam and Vorobev, Andrei and Witoszko, Izabela and Wolfe, Jason and Wray, Abby and Zhang, Yuchen and Zotov, Alexander}, journal = {Transactions of the Association for Computational Linguistics}, year = {2020}, doi = {10.1162/tacl\_a\_00333}, url = {https://doi.org/10.1162/tacl_a_00333} } ``` - [← previous TArC: Incrementally and Semi-Automatically Collecting a Tunisian Arabish Corpus](https://md.research.elamur.ai/free/en/papers/gugliotta-2020-tarc-incrementally-semi-automatically-collecting/) - [next → TED-CDB: A Large-Scale Chinese Discourse Relation Dataset on TED Talks](https://md.research.elamur.ai/free/en/papers/long-2020-ted-cdb-large-scale-chinese/) - [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/) ## References 2 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. James F. Allen, Bradford W. Miller, Eric K. Ringger, and Teresa Sikorski. 1996. A robust system for natural spoken dialogue. In Proceedings of the 34th Annual Meeting of the Association for Computational Linguistics, pages 62–70. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=James+F%2E+Allen%2C+Bradford+W%2E+Miller%2C+Eric+K%2E+Ringger%2C+and+Teresa+Sikorski%2E+1996%2E+A+robust+system+for+natural+spoken+dialogue%2E+In+Proceedings+of+the+34th+Annual+Meeting+of+the+Association+for+Computatio) 2. John Langshaw Austin. 1962. How to Do Things with Words. William James Lectures. Oxford University Press. Edited by James O. Urmson. A second edition appeared in 1975. [find](https://scholar.google.com/scholar?q=John+Langshaw+Austin%2E+1962%2E+How+to+Do+Things+with+Words%2E+William+James+Lectures%2E+Oxford+University+Press%2E+Edited+by+James+O%2E+Urmson%2E+A+second+edition+appeared+in+1975%2E)