---
title: "Semantic Noise Matters for Neural Natural — Dušek 2019"
description: "Neural generators that turn meaning representations (MRs) into text often produce output that does not match their input."
authors: ["Ondřej Dušek", "David M. Howcroft", "Verena Rieser"]
year: 2019
venue: "Proceedings of the 12th International Conference on Natural Language Generation"
source: "https://aclanthology.org/W19-8652/"
doi: "10.18653/v1/w19-8652"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/dusek-2019-semantic-noise-matters-neural-natural/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/dusek-2019-semantic-noise-matters-neural-natural/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / Semantic Noise Matters for Neural Natural Language Generation

# Semantic Noise Matters for Neural Natural Language Generation

**Authors:** [Ondřej Dušek](https://md.research.elamur.ai/free/en/authors/ondrej-dusek/), David M. Howcroft, [Verena Rieser](https://md.research.elamur.ai/free/en/authors/verena-rieser/) · **Year:** 2019 · **Venue:** Proceedings of the 12th International Conference on Natural Language Generation · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/W19-8652/) · [PDF](https://aclanthology.org/W19-8652.pdf) · [DOI 10.18653/v1/w19-8652](https://doi.org/10.18653/v1/w19-8652) · [OpenAlex](https://openalex.org/W2987956546)

**Citations:** 85 citations (openalex-snapshot · 2026-10-14) · [79](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Neural generators that turn meaning representations (MRs) into text often produce output that does not match their input. This paper asks how much of that problem comes from noisy crowdsourced training data. The authors used the E2E restaurant dataset, where about 40% of instances leave out or add information. Rather than rewriting the human references, they automatically re-annotated each MR to match its text, using a regular-expression slot matching script based on Reed et al. (2018). They also removed training and development items that overlapped with the test set. They then trained TGen, in three reranking variants, and SC-LSTM on both the original and the cleaned data. Each model was evaluated on both test sets using word-overlap metrics, slot error rate (SER) and a manual error analysis. Training on cleaned data cut SER by up to 97% and lowered fluency only slightly. This effect was far larger than the benefit of a semantic reranker alone. The dominant error type was omission, not hallucination. The study shows that cleaning data is a cheap and effective way to make neural NLG more faithful to its input.

## Key points
- **Cleaned E2E corpus:** The training set grew from 4,862 to 8,362 distinct MRs. Its references dropped from 42,061 to 33,525 after removing items that overlapped with the test set. Data and scripts are released at github.com/tuetschek/e2e-cleaning.
- **Script accuracy:** On 200 manually checked training instances, the slot matching script had a 4.2% SER. On 400 system outputs, it matched 99.5% of instances correctly.
- **SER gains for TGen (original test set):** With the default reranker, SER fell from 4.27% to 0.12%, about a 97% reduction. Without the reranker (TGen−), it fell from 15.94% to 0.97%, about 94%. Word-overlap scores dipped slightly, except NIST.
- **Model comparison:** SC-LSTM performed much worse than TGen, with SER above 30%. The authors attribute this to how much delexicalisation it needs.
- **Error types:** Missing slots were the main error, and cleaning the missing-slot noise gave the biggest improvement. The cleaned test set proved harder because its MRs are more diverse.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

79 papers of our library cite it (of the 140884 whose bibliographies we read); 8 of them have a page here, 71 not yet.

- [Survey of Hallucination in Natural Language Generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/)
- [Unified Demonstration Retriever for In-Context Learning](https://md.research.elamur.ai/free/en/papers/li-2023-unified-demonstration-retriever-context-learning/)
- [大语言模型的幻觉问题研究综述](https://md.research.elamur.ai/free/en/papers/liu-2024-hallucinations-large-language/)
- [The Third Multilingual Surface Realisation Shared Task (SR'20): Overview and Evaluation Results](https://md.research.elamur.ai/free/en/papers/mille-2020-third-multilingual-surface-realisation-shared/)
- [DART: Open-Domain Structured Data Record to Text Generation](https://md.research.elamur.ai/free/en/papers/nan-2020-dart-open-domain-structured-data/)
- [UL2: Unifying Language Learning Paradigms](https://md.research.elamur.ai/free/en/papers/tay-2022-ul2-unifying-language-learning-paradigms/)
- [SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer](https://md.research.elamur.ai/free/en/papers/vu-2022-spot-better-frozen-model-adaptation/)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://md.research.elamur.ai/free/en/papers/zeng-2022-glm-130b-open-bilingual-pre/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{dusek2019semantic,
  title = {{Semantic Noise Matters for Neural Natural Language Generation}},
  author = {Dušek, Ondřej and Howcroft, David M. and Rieser, Verena},
  booktitle = {Proceedings of the 12th International Conference on Natural Language Generation},
  year = {2019},
  doi = {10.18653/v1/w19-8652},
  url = {https://doi.org/10.18653/v1/w19-8652}
}
```

- [← previous Semantic Neural Machine Translation Using AMR](https://md.research.elamur.ai/free/en/papers/song-2019-semantic-neural-machine-translation-amr/)
- [next → SemEval-2019 Task 3: EmoContext Contextual Emotion Detection in Text](https://md.research.elamur.ai/free/en/papers/chatterjee-2019-semeval-2019-task-3-emocontext/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/)

## References

24 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In 3rd International Conference on Learning Representations (ICLR2015), San Diego, CA, USA. arXiv:1409.0473. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2015%2E+Neural+Machine+Translation+by+Jointly+Learning+to+Align+and+Translate%2E+In+3rd+International+Conference+on+Learning+Representations+%28ICLR2015%29%2C) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
2. George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proceedings of the Second International Conference on Human Language Technology Research, pages 138–145, San Diego, CA, USA. [find](https://scholar.google.com/scholar?q=George+Doddington%2E+2002%2E+Automatic+evaluation+of+machine+translation+quality+using+n%2Dgram+co%2Doccurrence+statistics%2E+In+Proceedings+of+the+Second+International+Conference+on+Human+Language+Technology+R)
3. Ondřej Dušek and Filip Jurčı́ček. 2016. Sequence-to-Sequence Generation for Spoken Dialogue via Deep Syntax Trees and Strings. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 45–51, Berlin, Germany. arXiv:1606.05491. [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Dus%CC%8Cek+and+Filip+Jurc%CC%8C%C4%B1%CC%81c%CC%8Cek%2E+2016%2E+Sequence%2Dto%2DSequence+Generation+for+Spoken+Dialogue+via+Deep+Syntax+Trees+and+Strings%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Comp)
4. Ondřej Dušek, Jekaterina Novikova, and Verena Rieser. 2018. Findings of the E2E NLG Challenge. In Proceedings of the 11th International Conference on Natural Language Generation, pages 322–328, Tilburg, The Netherlands. arXiv:1810.01170. [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Dus%CC%8Cek%2C+Jekaterina+Novikova%2C+and+Verena+Rieser%2E+2018%2E+Findings+of+the+E2E+NLG+Challenge%2E+In+Proceedings+of+the+11th+International+Conference+on+Natural+Language+Generation%2C+pages+322%E2%80%93328%2C+Tilb)
5. Ondřej Dušek, Jekaterina Novikova, and Verena Rieser. 2019. Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge. Computer Speech & Language, 59:123–156. arXiv:1901.07931. [https://ehudreiter.com/](https://ehudreiter.com/) — [arXiv:1901.07931](https://arxiv.org/abs/1901.07931) [↗](https://ehudreiter.com/)
6. Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyag Gao, and Dilek Hakkani-Tur. 2019. MultiWOZ 2.1: Multi-Domain Dialogue State Corrections and State Tracking Baselines. arXiv:1907.01669. [find](https://scholar.google.com/scholar?q=Mihail+Eric%2C+Rahul+Goel%2C+Shachi+Paul%2C+Abhishek+Sethi%2C+Sanchit+Agarwal%2C+Shuyag+Gao%2C+and+Dilek+Hakkani%2DTur%2E+2019%2E+MultiWOZ+2%2E1%3A+Multi%2DDomain+Dialogue+State+Corrections+and+State+Tracking+Baselines%2E+arXi)
7. Feng Nie, Jin-Ge Yao, Jinpeng Wang, Rong Pan, and Chin-Yew Lin. 2019. A simple recipe towards reducing hallucination in neural surface realisation. In Proceedings of the 57th Conference of the Association for Computational Linguistics Volume 1: Long Papers, pages 2673–2679, Florence, Italy. [find](https://scholar.google.com/scholar?q=Feng+Nie%2C+Jin%2DGe+Yao%2C+Jinpeng+Wang%2C+Rong+Pan%2C+and+Chin%2DYew+Lin%2E+2019%2E+A+simple+recipe+towards+reducing+hallucination+in+neural+surface+realisation%2E+In+Proceedings+of+the+57th+Conference+of+the+Associa)
8. Shi Feng, Eric Wallace, Alvin Grissom II, Mohit Iyyer, Pedro Rodriguez, and Jordan Boyd-Graber. 2018. Pathologies of Neural Models Make Interpretations Difficult. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3719–3728, Brussels, Belgium. [find](https://scholar.google.com/scholar?q=Shi+Feng%2C+Eric+Wallace%2C+Alvin+Grissom+II%2C+Mohit+Iyyer%2C+Pedro+Rodriguez%2C+and+Jordan+Boyd%2DGraber%2E+2018%2E+Pathologies+of+Neural+Models+Make+Interpretations+Difficult%2E+In+Proceedings+of+the+2018+Conference)
9. Jekaterina Novikova, Oliver Lemon, and Verena Rieser. 2016. Crowd-sourcing NLG Data: Pictures Elicit Better Data. In The 9th International Natural Language Generation conference INLG, Edinburgh, Scotland, UK. arXiv: 1608.00339. [find](https://scholar.google.com/scholar?q=Jekaterina+Novikova%2C+Oliver+Lemon%2C+and+Verena+Rieser%2E+2016%2E+Crowd%2Dsourcing+NLG+Data%3A+Pictures+Elicit+Better+Data%2E+In+The+9th+International+Natural+Language+Generation+conference+INLG%2C+Edinburgh%2C+Scotl)
10. David M. Howcroft, Dietrich Klakow, and Vera Demberg. 2017. The Extended SPaRKy Restaurant Corpus: Designing a Corpus with Variable Information Density. In Proceedings of Interspeech 2017, pages 3757–3761, Stockholm, Sweden. [find](https://scholar.google.com/scholar?q=David+M%2E+Howcroft%2C+Dietrich+Klakow%2C+and+Vera+Demberg%2E+2017%2E+The+Extended+SPaRKy+Restaurant+Corpus%3A+Designing+a+Corpus+with+Variable+Information+Density%2E+In+Proceedings+of+Interspeech+2017%2C+pages+3757%E2%80%93)
11. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, PA, USA. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+BLEU%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+the+40th+Annual+Meeting+of+the+Association+for+Comp) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
12. Juraj Juraska, Panagiotis Karagiannis, Kevin K. Bowden, and Marilyn A. Walker. 2018. A Deep Ensemble Model with Slot Alignment for Sequence-to-Sequence Natural Language Generation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: H… [find](https://scholar.google.com/scholar?q=Juraj+Juraska%2C+Panagiotis+Karagiannis%2C+Kevin+K%2E+Bowden%2C+and+Marilyn+A%2E+Walker%2E+2018%2E+A+Deep+Ensemble+Model+with+Slot+Alignment+for+Sequence%2Dto%2DSequence+Natural+Language+Generation%2E+In+Proceedings+of+t)
13. Philipp Koehn and Rebecca Knowles. 2017. Six Challenges for Neural Machine Translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28–39, Vancouver, Canada. [find](https://scholar.google.com/scholar?q=Philipp+Koehn+and+Rebecca+Knowles%2E+2017%2E+Six+Challenges+for+Neural+Machine+Translation%2E+In+Proceedings+of+the+First+Workshop+on+Neural+Machine+Translation%2C+pages+28%E2%80%9339%2C+Vancouver%2C+Canada%2E)
14. Alon Lavie and Abhaya Agarwal. 2007. Meteor: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 228–231, Prague, Czech Republic. [find](https://scholar.google.com/scholar?q=Alon+Lavie+and+Abhaya+Agarwal%2E+2007%2E+Meteor%3A+An+Automatic+Metric+for+MT+Evaluation+with+High+Levels+of+Correlation+with+Human+Judgments%2E+In+Proceedings+of+the+Second+Workshop+on+Statistical+Machine+Tr) [our card](https://md.research.elamur.ai/free/en/papers/lavie-2007-meteor-automatic-metric-mt-evaluation/)
15. Katherine Lee, Orhan Firat, Ashish Agarwal, Clara Fannjiang, and David Sussillo. 2019. Hallucinations in neural machine translation. OpenReview. [find](https://scholar.google.com/scholar?q=Katherine+Lee%2C+Orhan+Firat%2C+Ashish+Agarwal%2C+Clara+Fannjiang%2C+and+David+Sussillo%2E+2019%2E+Hallucinations+in+neural+machine+translation%2E+OpenReview%2E)
16. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text summarization branches out: Proceedings of the ACL-04 workshop, pages 74–81, Barcelona, Spain. [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin%2E+2004%2E+ROUGE%3A+A+package+for+automatic+evaluation+of+summaries%2E+In+Text+summarization+branches+out%3A+Proceedings+of+the+ACL%2D04+workshop%2C+pages+74%E2%80%9381%2C+Barcelona%2C+Spain%2E)
17. F. Mairesse, M. Gašić, F. Jurčı́ček, S. Keizer, B. Thomson, K. Yu, and S. Young. 2010. Phrase-based statistical language generation using graphical models and active learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, page 1552–1561, Uppsala, Swe… [find](https://scholar.google.com/scholar?q=F%2E+Mairesse%2C+M%2E+Gas%CC%8Cic%CC%81%2C+F%2E+Jurc%CC%8C%C4%B1%CC%81c%CC%8Cek%2C+S%2E+Keizer%2C+B%2E+Thomson%2C+K%2E+Yu%2C+and+S%2E+Young%2E+2010%2E+Phrase%2Dbased+statistical+language+generation+using+graphical+models+and+active+learning%2E+In+Proceedings+of+th)
18. Hongyuan Mei, Mohit Bansal, and Matthew R. Walter. 2016. What to talk about and how? Selective Generation using LSTMs with Coarse-to-Fine Alignment. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, page… [find](https://scholar.google.com/scholar?q=Hongyuan+Mei%2C+Mohit+Bansal%2C+and+Matthew+R%2E+Walter%2E+2016%2E+What+to+talk+about+and+how%3F+Selective+Generation+using+LSTMs+with+Coarse%2Dto%2DFine+Alignment%2E+In+Proceedings+of+the+2016+Conference+of+the+North+) [our card](https://md.research.elamur.ai/free/en/papers/mei-2016-what-talk-about-how-selective/)
19. Lena Reed, Shereen Oraby, and Marilyn Walker. 2018. Can Neural Generators for Dialogue Learn Sentence Planning and Discourse Structuring? In Proceedings of the 11th International Conference on Natural Language Generation, pages 284–295, Tilburg, The Netherlands. arXiv:1809.03015. [find](https://scholar.google.com/scholar?q=Lena+Reed%2C+Shereen+Oraby%2C+and+Marilyn+Walker%2E+2018%2E+Can+Neural+Generators+for+Dialogue+Learn+Sentence+Planning+and+Discourse+Structuring%3F+In+Proceedings+of+the+11th+International+Conference+on+Natural)
20. Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object hallucination in image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4035–4045, Brussels, Belgium. [find](https://scholar.google.com/scholar?q=Anna+Rohrbach%2C+Lisa+Anne+Hendricks%2C+Kaylee+Burns%2C+Trevor+Darrell%2C+and+Kate+Saenko%2E+2018%2E+Object+hallucination+in+image+captioning%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural) [our card](https://md.research.elamur.ai/free/en/papers/rohrbach-2018-object-hallucination-image-captioning/)
21. Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. CIDEr: Consensus-based image description evaluation. In Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4566–4575, Boston, MA, USA. [find](https://scholar.google.com/scholar?q=Ramakrishna+Vedantam%2C+C%2E+Lawrence+Zitnick%2C+and+Devi+Parikh%2E+2015%2E+CIDEr%3A+Consensus%2Dbased+image+description+evaluation%2E+In+Proceedings+of+the+2015+IEEE+Conference+on+Computer+Vision+and+Pattern+Recogni) [our card](https://md.research.elamur.ai/free/en/papers/vedantam-2014-cider-consensus-based-image-description/)
22. Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Lina M. Rojas-Barahona, Pei-Hao Su, David Vandyke, and Steve Young. 2016. Multi-domain Neural Network Language Generation for Spoken Dialogue Systems. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computationa… [find](https://scholar.google.com/scholar?q=Tsung%2DHsien+Wen%2C+Milica+Gasic%2C+Nikola+Mrksic%2C+Lina+M%2E+Rojas%2DBarahona%2C+Pei%2DHao+Su%2C+David+Vandyke%2C+and+Steve+Young%2E+2016%2E+Multi%2Ddomain+Neural+Network+Language+Generation+for+Spoken+Dialogue+Systems%2E+In+)
23. Tsung-Hsien Wen, Milica Gasic, Nikola Mrkšić, Pei-Hao Su, David Vandyke, and Steve Young. 2015. Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1711–1721, L… [find](https://scholar.google.com/scholar?q=Tsung%2DHsien+Wen%2C+Milica+Gasic%2C+Nikola+Mrks%CC%8Cic%CC%81%2C+Pei%2DHao+Su%2C+David+Vandyke%2C+and+Steve+Young%2E+2015%2E+Semantically+Conditioned+LSTM%2Dbased+Natural+Language+Generation+for+Spoken+Dialogue+Systems%2E+In+Procee) [our card](https://md.research.elamur.ai/free/en/papers/wen-2015-semantically-conditioned-lstm-based-natural/)
24. Sam Wiseman, Stuart M. Shieber, and Alexander M. Rush. 2017. Challenges in Data-to-Document Generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2243–2253, Copenhagen, Denmark. arXiv:1707.08052. [find](https://scholar.google.com/scholar?q=Sam+Wiseman%2C+Stuart+M%2E+Shieber%2C+and+Alexander+M%2E+Rush%2E+2017%2E+Challenges+in+Data%2Dto%2DDocument+Generation%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages) [our card](https://md.research.elamur.ai/free/en/papers/wiseman-2017-challenges-data-document-generation/)