---
title: "Globally Coherent Text Generation with Neural — Kiddon 2016"
description: "RNN generators produce fluent local text but tend to forget which inputs they have already covered."
authors: ["Chloé Kiddon", "Luke Zettlemoyer", "Yejin Choi"]
year: 2016
venue: "Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/D16-1032/"
doi: "10.18653/v1/d16-1032"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kiddon-2016-globally-coherent-text-generation-neural/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kiddon-2016-globally-coherent-text-generation-neural/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/2/) / Globally Coherent Text Generation with Neural Checklist Models

# Globally Coherent Text Generation with Neural Checklist Models

**Authors:** Chloé Kiddon, [Luke Zettlemoyer](https://md.research.elamur.ai/free/en/authors/luke-zettlemoyer/), [Yejin Choi](https://md.research.elamur.ai/free/en/authors/yejin-choi/) · **Year:** 2016 · **Venue:** Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D16-1032/) · [PDF](https://aclanthology.org/D16-1032.pdf) · [DOI 10.18653/v1/d16-1032](https://doi.org/10.18653/v1/d16-1032) · [OpenAlex](https://openalex.org/W2561658355)

**Citations:** 219 citations (OpenAlex, 2026-10-14) · [82](https://md.research.elamur.ai/free/en/papers/kiddon-2016-globally-coherent-text-generation-neural/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
RNN generators produce fluent local text but tend to forget which inputs they have already covered. In long outputs this leads to omitted, repeated or invented content. Kiddon, Zettlemoyer and Choi address this with the neural checklist model. It is a GRU encoder-decoder that keeps a soft record of which agenda items (for example, recipe ingredients) have been mentioned so far. At every step, a small classifier decides whether the next token should be:
- ordinary language-model output,
- a reference to a not-yet-used item, or
- a reference back to an already-used item.

The output representation mixes these three sources according to the classifier's probabilities. Two attention modules choose the item, one over the "new" items and one over the "used" items. The checklist is updated whenever an item is referenced.

The model is tested on a new recipe generation task (title plus ingredient list as input) and on the hotel and restaurant dialogue-act datasets of Wen et al. (2015). It keeps the local fluency of baseline RNNs while clearly improving coverage of the agenda. On dialogue it gains up to 4 BLEU over prior work. This makes it relevant for any data-to-text setting where every input fact must be said exactly once.

## Key points
- **Architecture:**
  - The goal (recipe title or dialogue act type) is encoded as a bag-of-words.
  - The checklist vector has one entry per agenda item, takes values in [0,1], and starts at all zeros.
  - A three-way softmax with a temperature hyper-parameter gates the GRU, new-item and used-item contributions.
- **Dialogue results (Table 3):**
  - Hotel BLEU: Checklist 90.61, SC-LSTM 86.53, rule-based HDC 55.52.
  - Hotel METEOR: Checklist 62.10, SC-LSTM 60.84.
  - Restaurant BLEU: Checklist 77.82 vs. SC-LSTM 74.49.
  - Restaurant METEOR: Checklist 54.42 vs. SC-LSTM 54.31.
- **Recipes, compared with the EncDec baseline:**
  - The baseline often names ingredients that are not in the list or introduces the same ingredient twice.
  - The Checklist model rarely takes the shortcut of dumping all ingredients into a single step; it produces such phrases only 13% of the time.
- **Limitations reported by the authors:**
  - The model refers imprecisely to sets of items (e.g., it calls both Swiss and Parmesan simply "cheese").
  - It only partly captures the full procedure needed to make the dish.
  - It is sensitive to hyperparameter initialization.

## Interesting ideas
- **Explicit soft coverage memory:** the model keeps a differentiable "already said" vector and feeds the "still available" items back into generation. This is a general recipe against omissions and repetitions, and could carry over to report generation from tables or to summarizing trade records.
- **Separate attention over new and used items:** splitting the input into "not yet mentioned" and "already mentioned" pools lets the model introduce items once and still refer back to them naturally. This could be reused wherever repeated references (anaphora) are legitimate but duplicate introductions are not.
- **Gating by reference type:** the classifier decides *whether* to mention an agenda item using only the hidden state, separately from *which* item to mention. This factorization makes behaviour easier to inspect and could be reused in other copy-like or entity-aware generators.
- **Generate references, don't just copy:** item references are produced through embeddings rather than verbatim copying, so the model can paraphrase (e.g., "steaks" for an ingredient listed as "ribeyes"). This is useful when input strings are noisy or overly formal, as in morphologically rich languages like Ukrainian, where surface forms must inflect.
- **Coverage-oriented error analysis:** beyond BLEU and METEOR, the authors examine hallucinated and duplicated item mentions in outputs. This is a cheap, task-grounded check that is worth adding to any data-to-text evaluation.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Text generation](https://md.research.elamur.ai/free/en/topics/text-generation/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{kiddon2016globally,
  title = {{Globally Coherent Text Generation with Neural Checklist Models}},
  author = {Kiddon, Chloé and Zettlemoyer, Luke and Choi, Yejin},
  booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1032},
  url = {https://doi.org/10.18653/v1/d16-1032}
}
```

- [← previous Which argument is more convincing? Analyzing and predicting convincingness of Web arguments using bidirectional LSTM](https://md.research.elamur.ai/free/en/papers/habernal-2016-which-argument-is-more-convincing/)
- [next → Incorporating Discrete Translation Lexicons into Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/arthur-2016-incorporating-discrete-translation-lexicons-into/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/2/)

## References

26 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Gabor Angeli, Percy Liang, and Dan Klein. 2010. A simple domain-independent probabilistic approach to generation. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 502–512. [find](https://scholar.google.com/scholar?q=Gabor+Angeli%2C+Percy+Liang%2C+and+Dan+Klein%2E+2010%2E+A+simple+domain%2Dindependent+probabilistic+approach+to+generation%2E+In+Proceedings+of+the+2010+Conference+on+Empirical+Methods+in+Natural+Language+Process) [our card](https://md.research.elamur.ai/free/en/papers/angeli-2010-simple-domain-independent-probabilistic-approach/)
2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. In ICLR 2015. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2014%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+In+ICLR+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
3. Niranjan Balasubramanian, Stephen Soderland, Mausam, and Oren Etzioni. 2013. Generating coherent event schemas at scale. In Proceedings of the 2013 Conference on Empirical Methods on Natural Language Processing, pages 1721–1731. [find](https://scholar.google.com/scholar?q=Niranjan+Balasubramanian%2C+Stephen+Soderland%2C+Mausam%2C+and+Oren+Etzioni%2E+2013%2E+Generating+coherent+event+schemas+at+scale%2E+In+Proceedings+of+the+2013+Conference+on+Empirical+Methods+on+Natural+Language+) [our card](https://md.research.elamur.ai/free/en/papers/balasubramanian-2013-generating-coherent-event-schemas-scale/)
4. Regina Barzilay and Mirella Lapata. 2005. Collective content selection for concept-to-text generation. In Proceedings of the 2005 Conference on Empirical Methods in Natural Language Processing, pages 331– 338. [find](https://scholar.google.com/scholar?q=Regina+Barzilay+and+Mirella+Lapata%2E+2005%2E+Collective+content+selection+for+concept%2Dto%2Dtext+generation%2E+In+Proceedings+of+the+2005+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+)
5. Jianpeng Cheng, Li Dong, and Mirella Lapata. 2016. Long short-term memory-networks for machine reading. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengi… [find](https://scholar.google.com/scholar?q=Jianpeng+Cheng%2C+Li+Dong%2C+and+Mirella+Lapata%2E+2016%2E+Long+short%2Dterm+memory%2Dnetworks+for+machine+reading%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2E+Kyung) [our card](https://md.research.elamur.ai/free/en/papers/cheng-2016-long-short-term-memory-networks/)
6. Robert Dale. 1988. Generating Referring Expressions in a Domain of Objects and Processes. Ph.D. thesis, Centre for Cognitive Science, University of Edinburgh. [find](https://scholar.google.com/scholar?q=Robert+Dale%2E+1988%2E+Generating+Referring+Expressions+in+a+Domain+of+Objects+and+Processes%2E+Ph%2ED%2E+thesis%2C+Centre+for+Cognitive+Science%2C+University+of+Edinburgh%2E)
7. Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the EACL 2014 Workshop on Statistical Machine Translation, pages 376–380. [find](https://scholar.google.com/scholar?q=Michael+Denkowski+and+Alon+Lavie%2E+2014%2E+Meteor+universal%3A+Language+specific+translation+evaluation+for+any+target+language%2E+In+Proceedings+of+the+EACL+2014+Workshop+on+Statistical+Machine+Translation%2C) [our card](https://md.research.elamur.ai/free/en/papers/denkowski-2014-meteor-universal-language-specific-translation/)
8. Alex Graves. 2012. Sequence transduction with recurrent neural networks. Representation Learning Worksop, ICML. Çaglar Gülçehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, and Yoshua Bengio. 2016. Pointing the unknown words. In Proceedings of the 54th Annual Meeting of the Association for Computa… [find](https://scholar.google.com/scholar?q=Alex+Graves%2E+2012%2E+Sequence+transduction+with+recurrent+neural+networks%2E+Representation+Learning+Worksop%2C+ICML%2E+C%CC%A7aglar+Gu%CC%88lc%CC%A7ehre%2C+Sungjin+Ahn%2C+Ramesh+Nallapati%2C+Bowen+Zhou%2C+and+Yoshua+Bengio%2E+2016%2E+) [our card](https://md.research.elamur.ai/free/en/papers/graves-2012-sequence-transduction-recurrent-neural-networks/)
9. Kristian J. Hammond. 1986. CHEF: A model of case-based planning. In Proceedings of the Fifth National Conference on Artificial Intelligence (AAAI-86), pages 267–271. [find](https://scholar.google.com/scholar?q=Kristian+J%2E+Hammond%2E+1986%2E+CHEF%3A+A+model+of+case%2Dbased+planning%2E+In+Proceedings+of+the+Fifth+National+Conference+on+Artificial+Intelligence+%28AAAI%2D86%29%2C+pages+267%E2%80%93271%2E)
10. R. Jia and P. Liang. 2016. Data recombination for neural semantic parsing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 12–22. [find](https://scholar.google.com/scholar?q=R%2E+Jia+and+P%2E+Liang%2E+2016%2E+Data+recombination+for+neural+semantic+parsing%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+12%E2%80%9322%2E)
11. Xu Jia, Efstratios Gavves, Basura Fernando, and Tinne Tuytelaars. 2015. Guiding long-short term memory for image caption generation. In Proceedings of the IEEE International Conference on Computer Vision, pages 2407–2415. Chloé Kiddon. 2016. Learning to Interpret and Generate Instructional Recipes.… [find](https://scholar.google.com/scholar?q=Xu+Jia%2C+Efstratios+Gavves%2C+Basura+Fernando%2C+and+Tinne+Tuytelaars%2E+2015%2E+Guiding+long%2Dshort+term+memory+for+image+caption+generation%2E+In+Proceedings+of+the+IEEE+International+Conference+on+Computer+Vis)
12. Ioannis Konstas and Mirella Lapata. 2013. A global model for concept-to-text generation. Journal of Artificial Intelligence Research (JAIR), 48:305–346. [find](https://scholar.google.com/scholar?q=Ioannis+Konstas+and+Mirella+Lapata%2E+2013%2E+A+global+model+for+concept%2Dto%2Dtext+generation%2E+Journal+of+Artificial+Intelligence+Research+%28JAIR%29%2C+48%3A305%E2%80%93346%2E)
13. Percy Liang, Michael I. Jordan, and Dan Klein. 2009. Learning semantic correspondences with less supervision. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1 - Volume 1, page… [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Michael+I%2E+Jordan%2C+and+Dan+Klein%2E+2009%2E+Learning+semantic+correspondences+with+less+supervision%2E+In+Proceedings+of+the+Joint+Conference+of+the+47th+Annual+Meeting+of+the+ACL+and+the+4th+I) [our card](https://md.research.elamur.ai/free/en/papers/liang-2009-learning-semantic-correspondences-less-supervision/)
14. Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412–1421, September. [find](https://scholar.google.com/scholar?q=Minh%2DThang+Luong%2C+Hieu+Pham%2C+and+Christopher+D%2E+Manning%2E+2015%2E+Effective+approaches+to+attention%2Dbased+neural+machine+translation%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Methods+in+Natural)
15. Hongyuan Mei, Mohit Bansal, and Matthew R. Walter. 2016. What to talk about and how? Selective generation using lstms with coarse-to-fine alignment. In The 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 720–73… [find](https://scholar.google.com/scholar?q=Hongyuan+Mei%2C+Mohit+Bansal%2C+and+Matthew+R%2E+Walter%2E+2016%2E+What+to+talk+about+and+how%3F+Selective+generation+using+lstms+with+coarse%2Dto%2Dfine+alignment%2E+In+The+15th+Annual+Conference+of+the+North+American) [our card](https://md.research.elamur.ai/free/en/papers/mei-2016-what-talk-about-how-selective/)
16. Shinsuke Mori, Hirokuni Maeta, Tetsuro Sasada, Koichiro Yoshino, Atsushi Hashimoto, Takuya Funatomi, and Yoko Yamakata. 2014. FlowGraph2Text: Automatic sentence skeleton compilation for procedural text generation. In Proceedings of the 8th International Natural Language Generation Conference, pages… [find](https://scholar.google.com/scholar?q=Shinsuke+Mori%2C+Hirokuni+Maeta%2C+Tetsuro+Sasada%2C+Koichiro+Yoshino%2C+Atsushi+Hashimoto%2C+Takuya+Funatomi%2C+and+Yoko+Yamakata%2E+2014%2E+FlowGraph2Text%3A+Automatic+sentence+skeleton+compilation+for+procedural+tex)
17. Ehud Reiter and Robert Dale. 2000. Building Natural Language Generation Systems. Cambridge University Press, New York, NY, USA. [find](https://scholar.google.com/scholar?q=Ehud+Reiter+and+Robert+Dale%2E+2000%2E+Building+Natural+Language+Generation+Systems%2E+Cambridge+University+Press%2C+New+York%2C+NY%2C+USA%2E)
18. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 379–389. [find](https://scholar.google.com/scholar?q=Alexander+M%2E+Rush%2C+Sumit+Chopra%2C+and+Jason+Weston%2E+2015%2E+A+neural+attention+model+for+abstractive+sentence+summarization%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Methods+in+Natural+Language)
19. Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Meg Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. In Conference of the North American Chapter of the Association for Comp… [find](https://scholar.google.com/scholar?q=Alessandro+Sordoni%2C+Michel+Galley%2C+Michael+Auli%2C+Chris+Brockett%2C+Yangfeng+Ji%2C+Meg+Mitchell%2C+Jian%2DYun+Nie%2C+Jianfeng+Gao%2C+and+Bill+Dolan%2E+2015%2E+A+neural+network+approach+to+context%2Dsensitive+generation+) [our card](https://md.research.elamur.ai/free/en/papers/sordoni-2015-neural-network-approach-context-sensitive/)
20. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 3104–3112. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V%2E+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Z%2E+Ghahramani%2C+M%2E+Welling%2C+C%2E+Cortes%2C+N%2E+D%2E+Lawrence%2C+and+K%2E+Q%2E+Weinberger%2C+editors%2C+Advance)
21. Curran Associates, Inc. Henry S. Thompson. 1977. Strategy and tactics: a model for language production. In Papers from the Thirteenth Regional Meeting of the Chicago Linguistics Society, pages 89–95. [find](https://scholar.google.com/scholar?q=Curran+Associates%2C+Inc%2E+Henry+S%2E+Thompson%2E+1977%2E+Strategy+and+tactics%3A+a+model+for+language+production%2E+In+Papers+from+the+Thirteenth+Regional+Meeting+of+the+Chicago+Linguistics+Society%2C+pages+89%E2%80%9395%2E)
22. Chicago Linguistics Society. Zhaopeng Tu, Yang Liu, Zhengdong Lu, Xiaohua Liu, and Hang Li. 2016a. Context gates for neural machine translation. CoRR, abs/1608.06043. [find](https://scholar.google.com/scholar?q=Chicago+Linguistics+Society%2E+Zhaopeng+Tu%2C+Yang+Liu%2C+Zhengdong+Lu%2C+Xiaohua+Liu%2C+and+Hang+Li%2E+2016a%2E+Context+gates+for+neural+machine+translation%2E+CoRR%2C+abs%2F1608%2E06043%2E)
23. Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016b. Modeling coverage for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 76–85. [find](https://scholar.google.com/scholar?q=Zhaopeng+Tu%2C+Zhengdong+Lu%2C+Yang+Liu%2C+Xiaohua+Liu%2C+and+Hang+Li%2E+2016b%2E+Modeling+coverage+for+neural+machine+translation%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computational+L) [our card](https://md.research.elamur.ai/free/en/papers/tu-2016-modeling-coverage-neural-machine-translation/)
24. Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Pei-hao Su, David Vandyke, and Steve J. Young. 2015. Semantically conditioned LSTM-based natural language generation for spoken dialogue systems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1711–1721. [find](https://scholar.google.com/scholar?q=Tsung%2DHsien+Wen%2C+Milica+Gasic%2C+Nikola+Mrksic%2C+Pei%2Dhao+Su%2C+David+Vandyke%2C+and+Steve+J%2E+Young%2E+2015%2E+Semantically+conditioned+LSTM%2Dbased+natural+language+generation+for+spoken+dialogue+systems%2E+In+Proce) [our card](https://md.research.elamur.ai/free/en/papers/wen-2015-semantically-conditioned-lstm-based-natural/)
25. Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Lina Maria Rojas-Barahona, Pei-hao Su, David Vandyke, and Steve J. Young. 2016. Multi-domain neural network language generation for spoken dialogue systems. In Proceedings of the 15th Annual Conference of the North American Chapter of the Association for… [find](https://scholar.google.com/scholar?q=Tsung%2DHsien+Wen%2C+Milica+Gasic%2C+Nikola+Mrksic%2C+Lina+Maria+Rojas%2DBarahona%2C+Pei%2Dhao+Su%2C+David+Vandyke%2C+and+Steve+J%2E+Young%2E+2016%2E+Multi%2Ddomain+neural+network+language+generation+for+spoken+dialogue+system)
26. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. Proceedings of the 32nd International Conference on Machine Learning, pages 2048–2057. [find](https://scholar.google.com/scholar?q=Kelvin+Xu%2C+Jimmy+Ba%2C+Ryan+Kiros%2C+Kyunghyun+Cho%2C+Aaron+C%2E+Courville%2C+Ruslan+Salakhutdinov%2C+Richard+S%2E+Zemel%2C+and+Yoshua+Bengio%2E+2015%2E+Show%2C+attend+and+tell%3A+Neural+image+caption+generation+with+visual+) [our card](https://md.research.elamur.ai/free/en/papers/xu-2015-show-attend-tell-neural-image/)