---
title: "The E2E Dataset — Novikova 2017"
description: "End-to-end NLG systems handle sentence planning and surface realisation together, learning directly from unaligned pairs of meaning representations (MRs)…"
authors: ["Jekaterina Novikova", "Ondřej Dušek", "Verena Rieser"]
year: 2017
venue: "Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue"
source: "https://aclanthology.org/W17-5525/"
doi: "10.18653/v1/w17-5525"
arxiv: "1706.09254"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/novikova-2017-e2e-dataset-new-challenges-end/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/novikova-2017-e2e-dataset-new-challenges-end/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/page/3/) / The E2E Dataset: New Challenges For End-to-End Generation

# The E2E Dataset: New Challenges For End-to-End Generation

**Authors:** Jekaterina Novikova, [Ondřej Dušek](https://md.research.elamur.ai/free/en/authors/ondrej-dusek/), [Verena Rieser](https://md.research.elamur.ai/free/en/authors/verena-rieser/) · **Year:** 2017 · **Venue:** Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/W17-5525/) · [PDF](https://aclanthology.org/W17-5525.pdf) · [DOI 10.18653/v1/w17-5525](https://doi.org/10.18653/v1/w17-5525) · [arXiv 1706.09254](https://arxiv.org/abs/1706.09254) · [OpenAlex](https://openalex.org/W2732004306)

**Citations:** 220 citations (openalex-snapshot · 2026-10-14) · [241](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
End-to-end NLG systems handle sentence planning and surface realisation together, learning directly from unaligned pairs of meaning representations (MRs) and texts. Until this work, they had been trained mostly on small, delexicalised corpora such as BAGEL and SF Restaurants, which produced fairly template-like output. The authors introduce the E2E dataset, about 50k crowdsourced pairings of dialogue-act-style MRs with human-written restaurant descriptions. The data was gathered on CrowdFlower, and workers were shown pictures instead of textual MRs as prompts, which encourages more natural phrasing. The authors compare the corpus with BAGEL and SFRest using the analysis framework of Perez-Beltrachini and Gardent (2017). E2E turns out to be much larger, lexically richer and syntactically more varied, with longer, multi-sentence references. Workers also often skip or add information, so a generator must learn content selection as well as wording. A TGen seq2seq baseline reaches BLEU scores similar to its results on the older corpora. This suggests that the extra size and multiple references compensate for the harder data. The resource was released for the E2E NLG Challenge as a shared, more demanding benchmark.

## Key points
- The E2E dataset has 50,602 instances and 5,751 unique MRs, with on average 8.1 references per MR (range 2–16). Each MR has 3–8 slots, and the data is split 76.5/8.5/15 into train/validation/test with no MR overlap between sets.
- Lexical richness is higher than in the comparison corpora:
  - MSTTR is 0.75 for E2E, against 0.62 for SFRest and 0.41 for BAGEL.
  - Lexical sophistication (LS) is 0.57, against 0.43 and 0.42.
  - Half of the bigrams and 61% of the trigrams occur only once.
- References run up to 6 sentences and average 20.1 words, against 8.53 for SFRest and 11.54 for BAGEL. E2E also has the largest shares of coordinated sentences (14%) and of the most complex D-Level sentences (16%).
- In a manual check of 50 MR–reference pairs, 22% omitted content and 18% added content, so about 40% do not match the MR exactly. Both SFRest and BAGEL show near-complete coverage.
- The TGen baseline, with delexicalised name/near and evaluated on the dev set, scored:
  - BLEU 0.6925
  - NIST 8.4781
  - METEOR 0.4703
  - ROUGE-L 0.7257
  - CIDEr 2.3987

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

241 papers of our library cite it (of the 140673 whose bibliographies we read); 14 of them have a page here, 227 not yet.

- [SLURP: A Spoken Language Understanding Resource Package](https://md.research.elamur.ai/free/en/papers/bastianelli-2020-slurp-spoken-language-understanding-resource/)
- [Handling Divergent Reference Texts when Evaluating Table-to-Text Generation](https://md.research.elamur.ai/free/en/papers/dhingra-2019-handling-divergent-reference-texts-when/)
- [Towards a Unified View of Parameter-Efficient Transfer Learning](https://md.research.elamur.ai/free/en/papers/he-2021-towards-unified-view-parameter-efficient/)
- [Diffusion-LM Improves Controllable Text Generation](https://md.research.elamur.ai/free/en/papers/li-2022-diffusion-lm-improves-controllable-text/)
- [The First Multilingual Surface Realisation Shared Task (SR'18): Overview and Evaluation Results](https://md.research.elamur.ai/free/en/papers/mille-2018-first-multilingual-surface-realisation-shared/)
- [Underspecified Universal Dependency Structures as Inputs for Multilingual Surface Realisation](https://md.research.elamur.ai/free/en/papers/mille-2018-underspecified-universal-dependency-structures-as/)
- [DART: Open-Domain Structured Data Record to Text Generation](https://md.research.elamur.ai/free/en/papers/nan-2020-dart-open-domain-structured-data/)
- [A Comprehensive Overview of Large Language Models](https://md.research.elamur.ai/free/en/papers/naveed-2023-comprehensive-overview-large-language-models/)
- [ToTTo: A Controlled Table-To-Text Generation Dataset](https://md.research.elamur.ai/free/en/papers/parikh-2020-totto-controlled-table-text-generation/)
- [Pragmatically Informative Text Generation](https://md.research.elamur.ai/free/en/papers/shen-2019-pragmatically-informative-text-generation/)
- [Can Diffusion Model Achieve Better Performance in Text Generation ? Bridging the Gap between Training and Inference !](https://md.research.elamur.ai/free/en/papers/tang-2023-can-diffusion-model-achieve-better/)
- [Evaluating Text Generation from Discourse Representation Structures](https://md.research.elamur.ai/free/en/papers/wang-2021-evaluating-text-generation-discourse-representation/)
- [Generating Sequences by Learning to Self-Correct](https://md.research.elamur.ai/free/en/papers/welleck-2022-generating-sequences-learning-self-correct/)
- [UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models](https://md.research.elamur.ai/free/en/papers/xie-2022-unifiedskg-unifying-multi-tasking-structured/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{novikova2017e2e,
  title = {{The E2E Dataset: New Challenges For End-to-End Generation}},
  author = {Novikova, Jekaterina and Dušek, Ondřej and Rieser, Verena},
  booktitle = {Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue},
  year = {2017},
  doi = {10.18653/v1/w17-5525},
  eprint = {1706.09254},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/w17-5525}
}
```

- [← previous The Corpus as Tutor: Data-driven Persuasive Language Learning](https://md.research.elamur.ai/free/en/papers/hbo-2017-winther-nielsen-corpus-tutor-data-driven-persuasive/)
- [next → The Good, the Bad, and the Disagreement: Complex ground truth in rhetorical structure analysis](https://md.research.elamur.ai/free/en/papers/das-2017-good-bad-disagreement-complex-ground/)
- [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/page/3/)

## References

20 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. D. Bahdanau, K. Cho, and Y. Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations. San Diego, CA, USA. ArXiv:1409.0473. http://arxiv.org/abs/1409.0473. [↗](http://arxiv.org/abs/1409.0473) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
2. Alon Lavie and Abhaya Agarwal. 2007. METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation. Association for Computational Linguistics, Prague, Czech Republic, pages 228– 231. http:/… [find](https://scholar.google.com/scholar?q=Alon+Lavie+and+Abhaya+Agarwal%2E+2007%2E+METEOR%3A+An+automatic+metric+for+MT+evaluation+with+high+levels+of+correlation+with+human+judgments%2E+In+Proceedings+of+the+Second+Workshop+on+Statistical+Machine+Tr) [our card](https://md.research.elamur.ai/free/en/papers/lavie-2007-meteor-automatic-metric-mt-evaluation/)
3. David L. Chen and Raymond J. Mooney. 2008. Learning to sportscast: A test of grounded language acquisition. In Proceedings of the 25th international conference on Machine learning (ICML). Helsinki, Finland, pages 128–135. [http://dl.acm.org/citation.cfm?id=1390173](http://dl.acm.org/citation.cfm?id=1390173). [↗](http://dl.acm.org/citation.cfm?id=1390173)
4. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text summarization branches out: Proceedings of the ACL-The training and development parts of our dataset can be downloaded from [http://www.macs.hw.ac.uk/](http://www.macs.hw.ac.uk/) InteractionLab/E2E/. information presentation for statistical dial… [↗](http://www.macs.hw.ac.uk/)
5. Xiaofei Lu. 2009. Automatic measurement of syntactic complexity in child language acquisition. International Journal of Corpus Linguistics 14(1):3–28. http://doi.org/10.1075/ijcl.14.1.02lu. — [doi:10.1075/ijcl.14.1.02lu](https://doi.org/10.1075/ijcl.14.1.02lu) [↗](http://doi.org/10.1075/ijcl.14.1.02lu)
6. Shikhar Sharma, Jing He, Kaheer Suleman, Hannes Schulz, and Philip Bachman. 2016. Natural language generation in dialogue using lexicalized and delexicalized data. CoRR abs/1606.03632. http://arxiv.org/abs/1606.03632. — [arXiv:1606.03632](https://arxiv.org/abs/1606.03632) [↗](http://arxiv.org/abs/1606.03632)
7. Xiaofei Lu. 2012. The relationship of lexical richness to the quality of ESL learners’ oral narratives. The Modern Language Journal 96(2):190–208. http://doi.org/10.1111/j.1540-4781.2011.01232 1.x. — [doi:10.1111/j.1540-4781.2011.01232](https://doi.org/10.1111/j.1540-4781.2011.01232) [↗](http://doi.org/10.1111/j.1540-4781.2011.01232)
8. Amanda Stent, Rashmi Prasad, and Marilyn Walker. 2004. Trainable sentence planning for complex information presentations in spoken dialog systems. In Proceedings of the 42nd Meeting of the Association for Computational Linguistics. Barcelona, Spain, pages 79–86. [http://aclweb.org/anthology/P04-1011](http://aclweb.org/anthology/P04-1011). [↗](http://aclweb.org/anthology/P04-1011)
9. Xiaofei Lu. 2014. Computational methods for corpus annotation and analysis. Springer. http://doi.org/10.1007/978-94-017-8645-4. — [doi:10.1007/978-94-017-8645-4](https://doi.org/10.1007/978-94-017-8645-4) [↗](http://doi.org/10.1007/978-94-017-8645-4)
10. François Mairesse, Milica Gašić, Filip Jurčı́ček, Simon Keizer, Blaise Thomson, Kai Yu, and Steve Young. 2010. Phrase-based statistical language generation using graphical models and active learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. Upp… [find](https://scholar.google.com/scholar?q=Franc%CC%A7ois+Mairesse%2C+Milica+Gas%CC%8Cic%CC%81%2C+Filip+Jurc%CC%8C%C4%B1%CC%81c%CC%8Cek%2C+Simon+Keizer%2C+Blaise+Thomson%2C+Kai+Yu%2C+and+Steve+Young%2E+2010%2E+Phrase%2Dbased+statistical+language+generation+using+graphical+models+and+active+learn)
11. Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. CIDEr: Consensus-based image description evaluation. In Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, MA, USA, pages 4566–4575. https://doi.org/10.1109/CVPR.2015.7299087. [↗](https://doi.org/10.1109/CVPR.2015.7299087) [our card](https://md.research.elamur.ai/free/en/papers/vedantam-2014-cider-consensus-based-image-description/)
12. Hongyuan Mei, Mohit Bansal, and Matthew R. Walter. 2016. What to talk about and how? Selective generation using LSTMs with coarse-to-fine alignment. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. San… [find](https://scholar.google.com/scholar?q=Hongyuan+Mei%2C+Mohit+Bansal%2C+and+Matthew+R%2E+Walter%2E+2016%2E+What+to+talk+about+and+how%3F+Selective+generation+using+LSTMs+with+coarse%2Dto%2Dfine+alignment%2E+In+Proceedings+of+the+2016+Conference+of+the+North+)
13. Marilyn A Walker, Stephen J Whittaker, Amanda Stent, Preetam Maloor, Johanna Moore, Michael Johnston, and Gunaranjan Vasireddy. 2004. Generation and evaluation of user tailored responses in multimodal dialogue. Cognitive Science 28(5):811–840. https://doi.org/10.1016/j.cogsci.2004.06.002. — [doi:10.1016/j.cogsci.2004.06.002](https://doi.org/10.1016/j.cogsci.2004.06.002) [↗](https://doi.org/10.1016/j.cogsci.2004.06.002)
14. Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić, Lina Maria Rojas-Barahona, Pei-hao Su, David Vandyke, and Steve J. Young. 2016. Multi-domain neural network language generation for spoken dialogue systems. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Co… [find](https://scholar.google.com/scholar?q=Tsung%2DHsien+Wen%2C+Milica+Gas%CC%8Cic%CC%81%2C+Nikola+Mrks%CC%8Cic%CC%81%2C+Lina+Maria+Rojas%2DBarahona%2C+Pei%2Dhao+Su%2C+David+Vandyke%2C+and+Steve+J%2E+Young%2E+2016%2E+Multi%2Ddomain+neural+network+language+generation+for+spoken+dialogue+sy)
15. Jekaterina Novikova, Oliver Lemon, and Verena Rieser. 2016. Crowd-sourcing NLG data: Pictures elicit better data. In Proceedings of the 9th International Natural Language Generation Conference. Edinburgh, UK, pages 265–273. arXiv:1608.00339. [http://aclweb.org/anthology/W16-2302](http://aclweb.org/anthology/W16-2302). [↗](http://aclweb.org/anthology/W16-2302) [our card](https://md.research.elamur.ai/free/en/papers/bojar-2016-results-wmt16-metrics-shared-task/)
16. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Philadelphia, PA, USA, pages 311–3… [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+BLEU%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+the+40th+Annual+Meeting+of+the+Association+for+Comp) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
17. Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić, Pei-Hao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned LSTM-based natural language generation for spoken dialogue systems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Lisbon, Portugal… [find](https://scholar.google.com/scholar?q=Tsung%2DHsien+Wen%2C+Milica+Gas%CC%8Cic%CC%81%2C+Nikola+Mrks%CC%8Cic%CC%81%2C+Pei%2DHao+Su%2C+David+Vandyke%2C+and+Steve+Young%2E+2015%2E+Semantically+conditioned+LSTM%2Dbased+natural+language+generation+for+spoken+dialogue+systems%2E+In+Proc)
18. Laura Perez-Beltrachini and Claire Gardent. 2017. Analysing data-to-text generation benchmarks. In Proceedings of the 10th International Natural Language Generation Conference. Santiago de Compostela, Spain. http://arxiv.org/abs/1705.03802. [↗](http://arxiv.org/abs/1705.03802)
19. Verena Rieser and Oliver Lemon. 2009. Natural language generation as planning under uncertainty for spoken dialogue systems. In Proceedings of the 12th Conference of the European Chapter of the ACL (EACL). Athens, Greece, pages 683–691. [http://aclweb.org/anthology/E09-1078](http://aclweb.org/anthology/E09-1078). [↗](http://aclweb.org/anthology/E09-1078)
20. Verena Rieser, Oliver Lemon, and Simon Keizer. 2014. Natural language generation as incremental planning under uncertainty: Adaptive [find](https://scholar.google.com/scholar?q=Verena+Rieser%2C+Oliver+Lemon%2C+and+Simon+Keizer%2E+2014%2E+Natural+language+generation+as+incremental+planning+under+uncertainty%3A+Adaptive)