---
title: "Improved Natural Language Generation via Loss Truncation (2020)"
description: "As a practical substitute, they propose loss truncation: during training, examples with the highest log… 51 references, cited by 59 papers here."
authors: ["Daniel Kang", "Tatsunori B. Hashimoto"]
year: 2020
venue: "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/2020.acl-main.66/"
doi: "10.18653/v1/2020.acl-main.66"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kang-2020-improved-natural-language-generation-loss/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kang-2020-improved-natural-language-generation-loss/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/) / Improved Natural Language Generation via Loss Truncation

# Improved Natural Language Generation via Loss Truncation

**Authors:** [Daniel Kang](https://md.research.elamur.ai/free/en/authors/daniel-kang/) and [Tatsunori B. Hashimoto](https://md.research.elamur.ai/free/en/authors/tatsunori-hashimoto/) · **Year:** 2020 · **Venue:** [Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2020](https://md.research.elamur.ai/free/en/venues/acl/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.acl-main.66/) · [PDF](https://aclanthology.org/2020.acl-main.66.pdf) · [DOI 10.18653/v1/2020.acl-main.66](https://doi.org/10.18653/v1/2020.acl-main.66) · [OpenAlex](https://openalex.org/W3035725000)

**Citations:** 70 citations (OpenAlex, 2026-10-14) · [59](https://md.research.elamur.ai/free/en/papers/kang-2020-improved-natural-language-generation-loss/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Conditional text generators are usually trained with log loss (KL divergence). This objective forces the model to cover every reference in the training set, so a small share of noisy or invalid targets can distort what it learns. Examples include titles containing facts absent from the source. Kang and Hashimoto argue that distinguishability, measured as total variation distance between model and human outputs, is a more robust goal. However, it is hard to optimize directly, and GAN-style attempts have lagged behind plain log-loss models. As a practical substitute, they propose loss truncation: during training, examples with the highest log loss are adaptively dropped, which yields an upper bound on distinguishability while keeping training as cheap as standard log loss. They also add a sequence-level rejection sampling step that keeps only high-probability outputs. On Gigaword summarization, the truncated models beat common decoding baselines on HUSE, a human-judged distinguishability metric. Their summaries also reach factual accuracy comparable to human references. The work suggests that robust training objectives can complement architectural fixes for hallucination.

## Key points
- A manual audit of 300 Gigaword titles found that 35% contain facts not supported by the article. These titles average a log loss of 34.3, versus 20.5 for directly entailed titles (over 1.7× higher).
- Among examples with log loss above 40, more than 80% involve some hallucinated content. This shows that high-loss training data is dominated by unfaithful references.
- The authors benchmarked loss truncation, paired with either direct or rejection sampling, against several decoding baselines (beam search, unrestricted sampling from the full distribution, and sampling restricted by top-k or by top-p), using HUSE for distinguishability.
- With rejection sampling, the truncated model surpassed all baselines, including beam search, on human-rated factual accuracy and matched the gold references.
- ROUGE-L (summarization) and BLEU (E2E) were reported as supplementary automatic metrics. Manual inspection showed loss-truncated samples were both varied and accurate, whereas top-k and beam outputs were repetitive and full or top-p sampling introduced invented facts.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 310 works · 284,480 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [✍️ Text generation](https://md.research.elamur.ai/free/en/topics/text-generation/) · 145 works

## Cite

Cite as: Daniel Kang and Tatsunori B. Hashimoto. 2020. Improved Natural Language Generation via Loss Truncation. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{kang2020improved,
  title = {{Improved Natural Language Generation via Loss Truncation}},
  author = {Daniel Kang and Tatsunori B. Hashimoto},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.66},
  url = {https://doi.org/10.18653/v1/2020.acl-main.66}
}
```

- [← previous Mapping Natural Language Instructions to Mobile UI Action Sequences](https://md.research.elamur.ai/free/en/papers/li-2020-mapping-natural-language-instructions-mobile/)
- [next → ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora](https://md.research.elamur.ai/free/en/papers/ouyang-2020-ernie-m-enhanced-multilingual/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/)

## References

51 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anusha Balakrishnan, Jinfeng Rao, Kartikeya Upasani, Michael White, and Rajen Subba. 2019. [Constrained decoding for neural nlg from compositional representations in task-oriented dialogue](https://md.research.elamur.ai/free/en/papers/balakrishnan-2019-constrained-decoding-neural-nlg-compositional/). arXiv preprint arXiv:1906.07220.
2. Eyal Beigman and Beata Beigman Klebanov. 2009. Learning with annotation noise. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1-Volume 1, pages 280–287. Association for Comput… [find](https://scholar.google.com/scholar?q=Eyal+Beigman+and+Beata+Beigman+Klebanov%2E+2009%2E+Learning+with+annotation+noise%2E+In+Proceedings+of+the+Joint+Conference+of+the+47th+Annual+Meeting+of+the+ACL+and+the+4th+International+Joint+Conference+o)
3. Andrew Brock, Jeff Donahue, and Karen Simonyan. 2018. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096. — [arXiv:1809.11096](https://arxiv.org/abs/1809.11096) [find](https://scholar.google.com/scholar?q=Andrew+Brock%2C+Jeff+Donahue%2C+and+Karen+Simonyan%2E+2018%2E+Large+scale+gan+training+for+high+fidelity+natural+image+synthesis%2E+arXiv+preprint+arXiv%3A1809%2E11096%2E+%E2%80%94+%5BarXiv%3A1809%2E11096%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F18)
4. Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin. 2018. Language gans falling short. arXiv preprint arXiv:1811.02549. — [arXiv:1811.02549](https://arxiv.org/abs/1811.02549) [find](https://scholar.google.com/scholar?q=Massimo+Caccia%2C+Lucas+Caccia%2C+William+Fedus%2C+Hugo+Larochelle%2C+Joelle+Pineau%2C+and+Laurent+Charlin%2E+2018%2E+Language+gans+falling+short%2E+arXiv+preprint+arXiv%3A1811%2E02549%2E+%E2%80%94+%5BarXiv%3A1811%2E02549%5D%28https%3A%2F%2Farxiv)
5. Imre Csiszar and János Körner. 2011. Information theory: coding theorems for discrete memoryless systems. Cambridge University Press. [find](https://scholar.google.com/scholar?q=Imre+Csiszar+and+Ja%CC%81nos+Ko%CC%88rner%2E+2011%2E+Information+theory%3A+coding+theorems+for+discrete+memoryless+systems%2E+Cambridge+University+Press%2E)
6. Bo Dai, Sanja Fidler, Raquel Urtasun, and Dahua Lin. 2017. Towards diverse and natural image descriptions via a conditional gan. In Proceedings of the IEEE International Conference on Computer Vision, pages 2970–2979. [find](https://scholar.google.com/scholar?q=Bo+Dai%2C+Sanja+Fidler%2C+Raquel+Urtasun%2C+and+Dahua+Lin%2E+2017%2E+Towards+diverse+and+natural+image+descriptions+via+a+conditional+gan%2E+In+Proceedings+of+the+IEEE+International+Conference+on+Computer+Vision%2C)
7. Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, and William W Cohen. 2019. [Handling divergent reference texts when evaluating table-to-text generation](https://md.research.elamur.ai/free/en/papers/dhingra-2019-handling-divergent-reference-texts-when/). arXiv preprint arXiv:1906.01081.
8. Ilias Diakonikolas, Gautam Kamath, Daniel M Kane, Jerry Li, Jacob Steinhardt, and Alistair Stewart. 2018. Sever: A robust meta-algorithm for stochastic optimization. arXiv preprint arXiv:1803.02815. — [arXiv:1803.02815](https://arxiv.org/abs/1803.02815) [find](https://scholar.google.com/scholar?q=Ilias+Diakonikolas%2C+Gautam+Kamath%2C+Daniel+M+Kane%2C+Jerry+Li%2C+Jacob+Steinhardt%2C+and+Alistair+Stewart%2E+2018%2E+Sever%3A+A+robust+meta%2Dalgorithm+for+stochastic+optimization%2E+arXiv+preprint+arXiv%3A1803%2E02815%2E+%E2%80%94)
9. Michael Gamon, Anthony Aue, and Martine Smets. 2005. Sentence-level mt evaluation without reference translations: Beyond language modeling. In Proceedings of EAMT, pages 103–111. [find](https://scholar.google.com/scholar?q=Michael+Gamon%2C+Anthony+Aue%2C+and+Martine+Smets%2E+2005%2E+Sentence%2Dlevel+mt+evaluation+without+reference+translations%3A+Beyond+language+modeling%2E+In+Proceedings+of+EAMT%2C+pages+103%E2%80%93111%2E)
10. Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. [Bottom-up abstractive summarization](https://md.research.elamur.ai/free/en/papers/gehrmann-2018-bottom-up-abstractive-summarization/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098–4109.
11. Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush. 2019. [Gltr: Statistical detection and visualization of generated text](https://md.research.elamur.ai/free/en/papers/gehrmann-2019-gltr-statistical-detection-visualization-generated/). arXiv preprint arXiv:1906.04043.
12. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems, pages 2672–2680. [find](https://scholar.google.com/scholar?q=Ian+Goodfellow%2C+Jean+Pouget%2DAbadie%2C+Mehdi+Mirza%2C+Bing+Xu%2C+David+Warde%2DFarley%2C+Sherjil+Ozair%2C+Aaron+Courville%2C+and+Yoshua+Bengio%2E+2014%2E+Generative+adversarial+nets%2E+In+Advances+in+neural+information+pr)
13. Tatsunori B Hashimoto, Hugh Zhang, and Percy Liang. 2019. Unifying human and statistical evaluation for natural language generation. North American Chapter of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tatsunori+B+Hashimoto%2C+Hugh+Zhang%2C+and+Percy+Liang%2E+2019%2E+Unifying+human+and+statistical+evaluation+for+natural+language+generation%2E+North+American+Chapter+of+the+Association+for+Computational+Linguis)
14. Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. 2018. [Learning to write with cooperative discriminators](https://md.research.elamur.ai/free/en/papers/holtzman-2018-learning-write-cooperative-discriminators/). arXiv preprint arXiv:1805.06087.
15. Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751. [find](https://scholar.google.com/scholar?q=Ari+Holtzman%2C+Jan+Buys%2C+Maxwell+Forbes%2C+and+Yejin+Choi%2E+2019%2E+The+curious+case+of+neural+text+degeneration%2E+arXiv+preprint+arXiv%3A1904%2E09751%2E)
16. Peter J Huber. 1992. Robust estimation of a location parameter. In Breakthroughs in statistics, pages 492– 518. Springer. [find](https://scholar.google.com/scholar?q=Peter+J+Huber%2E+1992%2E+Robust+estimation+of+a+location+parameter%2E+In+Breakthroughs+in+statistics%2C+pages+492%E2%80%93+518%2E+Springer%2E)
17. Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. [Ctrl: A conditional transformer language model for controllable generation](https://md.research.elamur.ai/free/en/papers/keskar-2019-ctrl-conditional-transformer-language-model/). arXiv preprint arXiv:1909.05858.
18. DL Donoho. 1982. Breakdown properties of multivariate location estimators. The Annals of Statistics. [find](https://scholar.google.com/scholar?q=DL+Donoho%2E+1982%2E+Breakdown+properties+of+multivariate+location+estimators%2E+The+Annals+of+Statistics%2E)
19. Beata Beigman Klebanov and Eyal Beigman. 2010. Some empirical evidence for annotation noise in a benchmarked dataset. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 438–446. Association for Computationa… [find](https://scholar.google.com/scholar?q=Beata+Beigman+Klebanov+and+Eyal+Beigman%2E+2010%2E+Some+empirical+evidence+for+annotation+noise+in+a+benchmarked+dataset%2E+In+Human+Language+Technologies%3A+The+2010+Annual+Conference+of+the+North+American+C)
20. Ondřej Dušek, Jekaterina Novikova, and Verena Rieser. 2019. Evaluating the state-of-the-art of end-to-end natural language generation: The E2E NLG Challenge. arXiv preprint arXiv:1901.11528. [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Dus%CC%8Cek%2C+Jekaterina+Novikova%2C+and+Verena+Rieser%2E+2019%2E+Evaluating+the+state%2Dof%2Dthe%2Dart+of+end%2Dto%2Dend+natural+language+generation%3A+The+E2E+NLG+Challenge%2E+arXiv+preprint+arXiv%3A1901%2E11528%2E)
21. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M Rush. 2017. [Opennmt: Open-source toolkit for neural machine translation](https://md.research.elamur.ai/free/en/papers/klein-2017-opennmt-open-source-toolkit-neural/). arXiv preprint arXiv:1701.02810.
22. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. ACL. [find](https://scholar.google.com/scholar?q=Angela+Fan%2C+Mike+Lewis%2C+and+Yann+Dauphin%2E+2018%2E+Hierarchical+neural+story+generation%2E+ACL%2E)
23. Wojciech Kryściński, Nitish Shirish Keskar, Bryan Mc-Cann, Caiming Xiong, and Richard Socher. 2019. [Neural text summarization: A critical evaluation](https://md.research.elamur.ai/free/en/papers/kryscinski-2019-neural-text-summarization-critical-evaluation/). arXiv preprint arXiv:1908.08960.
24. David L Donoho, Richard C Liu, et al. 1988. The” automatic” robustness of minimum distance functionals. The Annals of Statistics, 16(2):552–586. [find](https://scholar.google.com/scholar?q=David+L+Donoho%2C+Richard+C+Liu%2C+et+al%2E+1988%2E+The%E2%80%9D+automatic%E2%80%9D+robustness+of+minimum+distance+functionals%2E+The+Annals+of+Statistics%2C+16%282%29%3A552%E2%80%93586%2E)
25. Martin A Fischler and Robert C Bolles. 1981. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6):381– 395. [find](https://scholar.google.com/scholar?q=Martin+A+Fischler+and+Robert+C+Bolles%2E+1981%2E+Random+sample+consensus%3A+a+paradigm+for+model+fitting+with+applications+to+image+analysis+and+automated+cartography%2E+Communications+of+the+ACM%2C+24%286%29%3A381%E2%80%93+)
26. Katherine Lee, Orhan Firat, Ashish Agarwal, Clara Fannjiang, and David Sussillo. 2018. Hallucinations in neural machine translation. Interpretability and Robustness in Audio, Speech, and Language Workshop. of the 2015 Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Katherine+Lee%2C+Orhan+Firat%2C+Ashish+Agarwal%2C+Clara+Fannjiang%2C+and+David+Sussillo%2E+2018%2E+Hallucinations+in+neural+machine+translation%2E+Interpretability+and+Robustness+in+Audio%2C+Speech%2C+and+Language+Work)
27. Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. 2017. [Adversarial learning for neural dialogue generation](https://md.research.elamur.ai/free/en/papers/li-2017-adversarial-learning-neural-dialogue-generation/). arXiv preprint arXiv:1701.06547.
28. Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368. [find](https://scholar.google.com/scholar?q=Abigail+See%2C+Peter+J+Liu%2C+and+Christopher+D+Manning%2E+2017%2E+Get+to+the+point%3A+Summarization+with+pointer%2Dgenerator+networks%2E+arXiv+preprint+arXiv%3A1704%2E04368%2E)
29. Chin-Yew Lin and Eduard Hovy. 2003. [Automatic evaluation of summaries using n-gram co-occurrence statistics](https://md.research.elamur.ai/free/en/papers/lin-2003-automatic-evaluation-summaries-n-gram/). In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pages 150–157.
30. Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2015. [Minimum risk training for neural machine translation](https://md.research.elamur.ai/free/en/papers/shen-2015-minimum-risk-training-neural-machine/). arXiv preprint arXiv:1512.02433.
31. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. [Microsoft coco: Common objects in context](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/). In European conference on computer vision, pages 740–755. Springer.
32. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. Advances in NIPS. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+Advances+in+NIPS%2E)
33. Lucas Theis, Aäron van den Oord, and Matthias Bethge. 2016. A note on the evaluation of generative models. ICLR. [find](https://scholar.google.com/scholar?q=Lucas+Theis%2C+Aa%CC%88ron+van+den+Oord%2C+and+Matthias+Bethge%2E+2016%2E+A+note+on+the+evaluation+of+generative+models%2E+ICLR%2E)
34. Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. arXiv preprint arXiv:1602.06023. [find](https://scholar.google.com/scholar?q=Ramesh+Nallapati%2C+Bowen+Zhou%2C+Caglar+Gulcehre%2C+Bing+Xiang%2C+et+al%2E+2016%2E+Abstractive+text+summarization+using+sequence%2Dto%2Dsequence+rnns+and+beyond%2E+arXiv+preprint+arXiv%3A1602%2E06023%2E)
35. Ran Tian, Shashi Narayan, Thibault Sellam, and Ankur P Parikh. 2019. Sticking to the facts: Confident decoding for faithful data-to-text generation. arXiv preprint arXiv:1910.08684. — [arXiv:1910.08684](https://arxiv.org/abs/1910.08684) [find](https://scholar.google.com/scholar?q=Ran+Tian%2C+Shashi+Narayan%2C+Thibault+Sellam%2C+and+Ankur+P+Parikh%2E+2019%2E+Sticking+to+the+facts%3A+Confident+decoding+for+faithful+data%2Dto%2Dtext+generation%2E+arXiv+preprint+arXiv%3A1910%2E08684%2E+%E2%80%94+%5BarXiv%3A1910%2E0868)
36. Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. [Why we need new evaluation metrics for nlg](https://md.research.elamur.ai/free/en/papers/novikova-2017-why-we-need-new-evaluation/). arXiv preprint arXiv:1707.06875.
37. Christoph Tillmann and Hermann Ney. 2003. Word reordering and a dynamic programming beam search algorithm for statistical machine translation. Computational linguistics, 29(1):97–133. [find](https://scholar.google.com/scholar?q=Christoph+Tillmann+and+Hermann+Ney%2E+2003%2E+Word+reordering+and+a+dynamic+programming+beam+search+algorithm+for+statistical+machine+translation%2E+Computational+linguistics%2C+29%281%29%3A97%E2%80%93133%2E)
38. Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics-Volume 1, pages 160–167. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Franz+Josef+Och%2E+2003%2E+Minimum+error+rate+training+in+statistical+machine+translation%2E+In+Proceedings+of+the+41st+Annual+Meeting+on+Association+for+Computational+Linguistics%2DVolume+1%2C+pages+160%E2%80%93167%2E+A)
39. John W Tukey. 1960. A survey of sampling from contaminated distributions. Contributions to probability and statistics, pages 448–485. [find](https://scholar.google.com/scholar?q=John+W+Tukey%2E+1960%2E+A+survey+of+sampling+from+contaminated+distributions%2E+Contributions+to+probability+and+statistics%2C+pages+448%E2%80%93485%2E)
40. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics.
41. Yau-Shian Wang and Hung-Yi Lee. 2018. Learning to encode text as human-readable summaries using generative adversarial networks. arXiv preprint arXiv:1810.02851. [find](https://scholar.google.com/scholar?q=Yau%2DShian+Wang+and+Hung%2DYi+Lee%2E+2018%2E+Learning+to+encode+text+as+human%2Dreadable+summaries+using+generative+adversarial+networks%2E+arXiv+preprint+arXiv%3A1810%2E02851%2E)
42. Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2019. [Neural text generation with unlikelihood training](https://md.research.elamur.ai/free/en/papers/welleck-2019-neural-text-generation-unlikelihood-training/). arXiv preprint arXiv:1908.04319.
43. Yevgeniy Puzikov and Iryna Gurevych. 2018. E2e nlg challenge: Neural models vs. templates. In Proceedings of the 11th International Conference on Natural Language Generation, pages 463–471. [find](https://scholar.google.com/scholar?q=Yevgeniy+Puzikov+and+Iryna+Gurevych%2E+2018%2E+E2e+nlg+challenge%3A+Neural+models+vs%2E+templates%2E+In+Proceedings+of+the+11th+International+Conference+on+Natural+Language+Generation%2C+pages+463%E2%80%93471%2E)
44. Sam Wiseman, Stuart M Shieber, and Alexander M Rush. 2017. [Challenges in data-to-document generation](https://md.research.elamur.ai/free/en/papers/wiseman-2017-challenges-data-document-generation/). arXiv preprint arXiv:1707.08052.
45. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8). [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+Blog%2C+1%288%29%2E)
46. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. [Defending against neural fake news](https://md.research.elamur.ai/free/en/papers/zellers-2019-defending-against-neural-fake-news/). arXiv preprint arXiv:1905.12616.
47. Sai Rajeswar, Sandeep Subramanian, Francis Dutil, Christopher Pal, and Aaron Courville. 2017. Adversarial generation of natural language. arXiv preprint arXiv:1705.10929. [find](https://scholar.google.com/scholar?q=Sai+Rajeswar%2C+Sandeep+Subramanian%2C+Francis+Dutil%2C+Christopher+Pal%2C+and+Aaron+Courville%2E+2017%2E+Adversarial+generation+of+natural+language%2E+arXiv+preprint+arXiv%3A1705%2E10929%2E)
48. Sharon Zhou, Mitchell Gordon, Ranjay Krishna, Austin Narcomey, Li F Fei-Fei, and Michael Bernstein. 2019. Hype: A benchmark for human eye perceptual evaluation of generative models. In Advances in Neural Information Processing Systems, pages 3444–3456. [find](https://scholar.google.com/scholar?q=Sharon+Zhou%2C+Mitchell+Gordon%2C+Ranjay+Krishna%2C+Austin+Narcomey%2C+Li+F+Fei%2DFei%2C+and+Michael+Bernstein%2E+2019%2E+Hype%3A+A+benchmark+for+human+eye+perceptual+evaluation+of+generative+models%2E+In+Advances+in+Neu)
49. Melissa Roemmele. 2016. Writing stories with help from recurrent neural networks. In Thirtieth AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Melissa+Roemmele%2E+2016%2E+Writing+stories+with+help+from+recurrent+neural+networks%2E+In+Thirtieth+AAAI+Conference+on+Artificial+Intelligence%2E)
50. Alexander M Rush, SEAS Harvard, Sumit Chopra, and Jason Weston. 2017. A neural attention model for sentence summarization. In ACLWeb. Proceedings Context: Donna Shalala is sporting a mustache to promote public health. Title: Milk on Her Lip Shalala Raises Eyebrows Context: At least two people have t… [find](https://scholar.google.com/scholar?q=Alexander+M+Rush%2C+SEAS+Harvard%2C+Sumit+Chopra%2C+and+Jason+Weston%2E+2017%2E+A+neural+attention+model+for+sentence+summarization%2E+In+ACLWeb%2E+Proceedings+Context%3A+Donna+Shalala+is+sporting+a+mustache+to+promo)
51. Context: Southwest China’s Sichuan province has decided to build an inter-city high-tech industrial belt to serve development of Western China. Title: Sichuan to Build High-Tech Industrial Belt (a) Example 1. Context: British investment fund Fidelity has increased its stake in Puma, the German maker… [find](https://scholar.google.com/scholar?q=Context%3A+Southwest+China%E2%80%99s+Sichuan+province+has+decided+to+build+an+inter%2Dcity+high%2Dtech+industrial+belt+to+serve+development+of+Western+China%2E+Title%3A+Sichuan+to+Build+High%2DTech+Industrial+Belt+%28a%29+Ex)

*Updated 24 October 2026*