---
title: "Locally Typical Sampling — Meister 2023"
description: "Neural language models assign high likelihood to held-out text, yet the text they generate often reads poorly."
authors: ["Clara Meister", "Tiago Pimentel", "Gian Wiher", "Ryan Cotterell"]
year: 2023
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/2023.tacl-1.7/"
doi: "10.1162/tacl_a_00536"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/meister-2023-locally-typical-sampling/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/meister-2023-locally-typical-sampling/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/) / Locally Typical Sampling

# Locally Typical Sampling

**Authors:** [Clara Meister](https://md.research.elamur.ai/free/en/authors/clara-meister/), [Tiago Pimentel](https://md.research.elamur.ai/free/en/authors/tiago-pimentel/), Gian Wiher, [Ryan Cotterell](https://md.research.elamur.ai/free/en/authors/ryan-cotterell/) · **Year:** 2023 · **Venue:** Transactions of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.tacl-1.7/) · [PDF](https://aclanthology.org/2023.tacl-1.7.pdf) · [DOI 10.1162/tacl\_a\_00536](https://doi.org/10.1162/tacl_a_00536) · [OpenAlex](https://openalex.org/W4317897818)

**Citations:** 70 citations (openalex-snapshot · 2026-10-14) · [82](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary

Neural language models assign high likelihood to held-out text, yet the text they generate often reads poorly. Maximization-based decoding tends toward dull or degenerate output, and ordinary sampling can drift into nonsense. Meister et al. argue that this gap comes from the decision rule used for generation, not from flawed models. They treat generation as a discrete stochastic process, which lets them analyse it with information theory. Drawing on psycholinguistic evidence that people spread information fairly evenly across an utterance, they introduce local typicality: each word's information content should stay near its expected value, which is the model's conditional entropy given the context. Their decoder, locally typical sampling, enforces this by restricting each step's candidates to tokens whose surprisal is close to that entropy. They tested it on abstractive summarization and story generation with automatic metrics and crowdsourced human judgments. Its quality matched or slightly exceeded nucleus and top-k sampling, and it consistently produced fewer degenerate repetitions. The method is cheap, simple, and offers a principled explanation for why the most probable text is often not the most human-like.

## Key points

- The paper formally distinguishes a language model (a distribution over whole strings) from a language process. A process is a valid model only if it is "tight", meaning no probability mass leaks to infinite strings. Their toy example of a non-tight process loses about 0.58 of its mass to such strings.
- Baselines include nucleus (η = 0.9/0.95), top-k (k = 30/40), temperature, Mirostat and beam search. Locally typical sampling is evaluated with τ = 0.2 and τ = 0.95.
- Story generation uses GPT-2 medium and large fine-tuned on WritingPrompts. Evaluation combines human ratings (coherence, fluency, interestingness, relevance) with MAUVE, Zipf's coefficient and the REP repetition measure.
- Typical sampling achieves REP values comparable to human-written text, and human quality scores are on par with or slightly above other stochastic decoders.
- An implementation is available in the HuggingFace Transformers library. Crowdworkers were paid at least $15 per hour.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

82 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 79 not yet.

- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://md.research.elamur.ai/free/en/papers/cai-2024-medusa-simple-llm-inference-acceleration/)
- [ALADAN at IWSLT24 Low-resource Arabic Dialectal Speech Translation Task](https://md.research.elamur.ai/free/en/papers/kheder-2024-aladan-iwslt24-low-resource-arabic/)
- [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{meister2023locally,
  title = {{Locally Typical Sampling}},
  author = {Meister, Clara and Pimentel, Tiago and Wiher, Gian and Cotterell, Ryan},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.1162/tacl\_a\_00536},
  url = {https://doi.org/10.1162/tacl_a_00536}
}
```

- [← previous LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset](https://md.research.elamur.ai/free/en/papers/zheng-2023-lmsys-chat-1m-large-scale/)
- [next → Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning](https://md.research.elamur.ai/free/en/papers/pan-2023-logic-lm-empowering-large-language/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/)

## References

57 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Noam Chomsky. 1995. The Minimalist Program. MIT Press, Cambridge, MA. [find](https://scholar.google.com/scholar?q=Noam+Chomsky%2E+1995%2E+The+Minimalist+Program%2E+MIT+Press%2C+Cambridge%2C+MA%2E)
2. Matthew Aylett and Alice Turk. 2004. The smooth signal redundancy hypothesis: A functional explanation for relationships between redundancy, prosodic prominence, and duration in spontaneous speech. Language and Speech, 47(1):31–56. https://doi.org/10.1177/00238309040470010201, PubMed: 15298329 [↗](https://doi.org/10.1177/00238309040470010201)
3. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… [find](https://scholar.google.com/scholar?q=Aakanksha+Chowdhery%2C+Sharan+Narang%2C+Jacob+Devlin%2C+Maarten+Bosma%2C+Gaurav+Mishra%2C+Adam+Roberts%2C+Paul+Barham%2C+Hyung+Won+Chung%2C+Charles+Sutton%2C+Sebastian+Gehrmann%2C+Parker+Schuh%2C+Kensen+Shi%2C+Sasha+Tsvyashc)
4. Sourya Basu, Govardana Sachitanandam Ramachandran, Nitish Shirish Keskar, and Lav R. Varshney. 2021. Mirostat: A perplexity-controlled neural text decoding algorithm. In Proceedings of the 9th International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Sourya+Basu%2C+Govardana+Sachitanandam+Ramachandran%2C+Nitish+Shirish+Keskar%2C+and+Lav+R%2E+Varshney%2E+2021%2E+Mirostat%3A+A+perplexity%2Dcontrolled+neural+text+decoding+algorithm%2E+In+Proceedings+of+the+9th+Interna)
5. Taylor L. Booth and Richard A. Thompson. 1973. Applying probability measures to abstract languages. IEEE Transactions on Computers, C-22(5):442–450. https://doi.org/10 .1109/T-C.1973.223746 [find](https://scholar.google.com/scholar?q=Taylor+L%2E+Booth+and+Richard+A%2E+Thompson%2E+1973%2E+Applying+probability+measures+to+abstract+languages%2E+IEEE+Transactions+on+Computers%2C+C%2D22%285%29%3A442%E2%80%93450%2E+https%3A%2F%2Fdoi%2Eorg%2F10+%2E1109%2FT%2DC%2E1973%2E223746)
6. Mark Braverman, Xinyi Chen, Sham Kakade, Karthik Narasimhan, Cyril Zhang, and Yi Zhang. 2020. Calibration, entropy rates, and memory in language models. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pages 1089–1099. PMLR. [find](https://scholar.google.com/scholar?q=Mark+Braverman%2C+Xinyi+Chen%2C+Sham+Kakade%2C+Karthik+Narasimhan%2C+Cyril+Zhang%2C+and+Yi+Zhang%2E+2020%2E+Calibration%2C+entropy+rates%2C+and+memory+in+language+models%2E+In+Proceedings+of+the+37th+International+Confer)
7. Leo Breiman. 1957. The individual ergodic theorem of information theory. The Annals of Mathematical Statistics, 28(3):809–811. https://doi.org/10.1214/aoms/1177706899 — [doi:10.1214/aoms/1177706899](https://doi.org/10.1214/aoms/1177706899) [↗](https://doi.org/10.1214/aoms/1177706899)
8. Christophe Coupé, Yoon Mi Oh, Dan Dediu, and François Pellegrino. 2019. Different languages, similar encoding efficiency: Comparable information rates across the human communicative niche. Science Advances, 5(9). https://doi.org/10.1126/sciadv.aaw2594, PubMed: 32047854 — [doi:10.1126/sciadv.aaw2594](https://doi.org/10.1126/sciadv.aaw2594) [↗](https://doi.org/10.1126/sciadv.aaw2594)
9. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chr… [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D%2E+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretche)
10. Thomas M. Cover and Joy A. Thomas. 2012. Elements of Information Theory. John Wiley & Sons. [find](https://scholar.google.com/scholar?q=Thomas+M%2E+Cover+and+Joy+A%2E+Thomas%2E+2012%2E+Elements+of+Information+Theory%2E+John+Wiley+%26+Sons%2E)
11. Alexandra DeLucia, Aaron Mueller, Xiang Lisa Li, and João Sedoc. 2021. Decoding methods for neural narrative generation. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), pages 166–185, Online. Association for Computational Linguistics. https://d… [find](https://scholar.google.com/scholar?q=Alexandra+DeLucia%2C+Aaron+Mueller%2C+Xiang+Lisa+Li%2C+and+Joa%CC%83o+Sedoc%2E+2021%2E+Decoding+methods+for+neural+narrative+generation%2E+In+Proceedings+of+the+1st+Workshop+on+Natural+Language+Generation%2C+Evaluation%2C)
12. Noam Chomsky. 1957. Syntactic Structures. Mouton and Co., The Hague. https://doi.org/10.1515/9783112316009 — [doi:10.1515/9783112316009](https://doi.org/10.1515/9783112316009) [↗](https://doi.org/10.1515/9783112316009)
13. Sander Dieleman. 2020. Musings on typicality. [find](https://scholar.google.com/scholar?q=Sander+Dieleman%2E+2020%2E+Musings+on+typicality%2E)
14. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489–500, Brussels, Belgium. Association for Computational Linguistics. https://doi.org/10 .18653/… [find](https://scholar.google.com/scholar?q=Sergey+Edunov%2C+Myle+Ott%2C+Michael+Auli%2C+and+David+Grangier%2E+2018%2E+Understanding+back%2Dtranslation+at+scale%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pag)
15. John Hale. 2001. A probabilistic Earley parser as a psycholinguistic model. In Second Meeting of the North American Chapter of the Association for Computational Linguistics. https://doi.org/10.3115/1073336 .1073357 [↗](https://doi.org/10.3115/1073336) [our card](https://md.research.elamur.ai/free/en/papers/hale-2001-probabilistic-earley-parser-as-psycholinguistic/)
16. Bryan Eikema and Wilker Aziz. 2020. Is MAP decoding all you need? The inadequacy of the mode in neural machine translation. In Proceedings of the 28th International Conference on Computational Linguistics, COLING, pages 4506–4520, Barcelona, Spain (Online). International Committee on Computational L… [find](https://scholar.google.com/scholar?q=Bryan+Eikema+and+Wilker+Aziz%2E+2020%2E+Is+MAP+decoding+all+you+need%3F+The+inadequacy+of+the+mode+in+neural+machine+translation%2E+In+Proceedings+of+the+28th+International+Conference+on+Computational+Linguis) [our card](https://md.research.elamur.ai/free/en/papers/eikema-2020-is-map-decoding-all-you/)
17. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… [find](https://scholar.google.com/scholar?q=Jordan+Hoffmann%2C+Sebastian+Borgeaud%2C+Arthur+Mensch%2C+Elena+Buchatskaya%2C+Trevor+Cai%2C+Eliza+Rutherford%2C+Diego+de+Las+Casas%2C+Lisa+Anne+Hendricks%2C+Johannes+Welbl%2C+Aidan+Clark%2C+Tom+Hennigan%2C+Eric+Noland%2C+Ka)
18. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898, Melbourne, Australia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Angela+Fan%2C+Mike+Lewis%2C+and+Yann+Dauphin%2E+2018%2E+Hierarchical+neural+story+generation%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Papers%29)
19. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In Proceedings of the 8th International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Ari+Holtzman%2C+Jan+Buys%2C+Li+Du%2C+Maxwell+Forbes%2C+and+Yejin+Choi%2E+2020%2E+The+curious+case+of+neural+text+degeneration%2E+In+Proceedings+of+the+8th+International+Conference+on+Learning+Representations%2E)
20. August Fenk and Gertraud Fenk. 1980. Konstanz im Kurzzeitgedächtnis-Konstanz im sprachlichen Informationsfluß. Zeitschrift für ex-perimentelle und angewandte Psychologie, 27(3):400–414. [find](https://scholar.google.com/scholar?q=August+Fenk+and+Gertraud+Fenk%2E+1980%2E+Konstanz+im+Kurzzeitgeda%CC%88chtnis%2DKonstanz+im+sprachlichen+Informationsflu%C3%9F%2E+Zeitschrift+fu%CC%88r+ex%2Dperimentelle+und+angewandte+Psychologie%2C+27%283%29%3A400%E2%80%93414%2E)
21. Fred Karlsson. 2010. 3. Syntactic recursion and iteration. In Recursion and Human Language. De Gruyter Mouton, Berlin, New York. https://doi.org/10.1515/9783110219258.43 [↗](https://doi.org/10.1515/9783110219258.43)
22. Edward Gibson, Richard Futrell, Steven T. Piantadosi, Isabelle Dautriche, Kyle Mahowald, Leon Bergen, and Roger Levy. 2019. How efficiency shapes human language. Trends in Cognitive Sciences, 23(5):389–407. https://doi.org/10.1016/j.tics.2019.02.003, PubMed: 31006626 — [doi:10.1016/j.tics.2019.02.003](https://doi.org/10.1016/j.tics.2019.02.003) [↗](https://doi.org/10.1016/j.tics.2019.02.003)
23. Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. 2018. Sharp nearby, fuzzy far away: How neural language models use context. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 284–294, Melbourne, Australia. Association for C… [find](https://scholar.google.com/scholar?q=Urvashi+Khandelwal%2C+He+He%2C+Peng+Qi%2C+and+Dan+Jurafsky%2E+2018%2E+Sharp+nearby%2C+fuzzy+far+away%3A+How+neural+language+models+use+context%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Compu)
24. Kartik Goyal, Chris Dyer, and Taylor Berg-Kirkpatrick. 2019. An empirical investigation of global and local normalization for recurrent neural sequence models using a continuous relaxation to beam search. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Comp… [find](https://scholar.google.com/scholar?q=Kartik+Goyal%2C+Chris+Dyer%2C+and+Taylor+Berg%2DKirkpatrick%2E+2019%2E+An+empirical+investigation+of+global+and+local+normalization+for+recurrent+neural+sequence+models+using+a+continuous+relaxation+to+beam+sea)
25. Konrad Knopp. 1954. Theory and Application of Infinite Series. London, Blackie & Son Ltd. [find](https://scholar.google.com/scholar?q=Konrad+Knopp%2E+1954%2E+Theory+and+Application+of+Infinite+Series%2E+London%2C+Blackie+%26+Son+Ltd%2E)
26. Jey Han Lau, Alexander Clark, and Shalom Lappin. 2017. Grammaticality, acceptability, and probability: A probabilistic view of linguistic knowledge. Cognitive Science, 41(5):1202–1241. https://doi.org/10 .1111/cogs.12414, PubMed: 27732744 [find](https://scholar.google.com/scholar?q=Jey+Han+Lau%2C+Alexander+Clark%2C+and+Shalom+Lappin%2E+2017%2E+Grammaticality%2C+acceptability%2C+and+probability%3A+A+probabilistic+view+of+linguistic+knowledge%2E+Cognitive+Science%2C+41%285%29%3A1202%E2%80%931241%2E+https%3A%2F%2Fdoi%2Eorg)
27. Jian Guan, Fei Huang, Zhihao Zhao, Xiaoyan Zhu, and Minlie Huang. 2020. A knowledge-enhanced pretraining model for commonsense Roger Levy and T. Florian Jaeger. 2007. Speakers optimize information density through syntactic reduction. In Advances in Neural Information Processing Systems, volume 19. M… — [doi:10.18653/v1/2020](https://doi.org/10.18653/v1/2020) [find](https://scholar.google.com/scholar?q=Jian+Guan%2C+Fei+Huang%2C+Zhihao+Zhao%2C+Xiaoyan+Zhu%2C+and+Minlie+Huang%2E+2020%2E+A+knowledge%2Denhanced+pretraining+model+for+commonsense+Roger+Levy+and+T%2E+Florian+Jaeger%2E+2007%2E+Speakers+optimize+information+den)
28. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the… [find](https://scholar.google.com/scholar?q=Mike+Lewis%2C+Yinhan+Liu%2C+Naman+Goyal%2C+Marjan+Ghazvininejad%2C+Abdelrahman+Mohamed%2C+Omer+Levy%2C+Veselin+Stoyanov%2C+and+Luke+Zettlemoyer%2E+2020%2E+BART%3A+Denoising+sequence%2Dto%2Dsequence+pre%2Dtraining+for+natural+l) [our card](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/)
29. Clara Meister, Tim Vieira, and Ryan Cotterell. 2020b. If beam search is the answer, what was the question? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.emnlp-main.170 [↗](https://doi.org/10.18653/v1/2020.emnlp-main.170)
30. Clara Meister, Gian Wiher, Tiago Pimentel, and Ryan Cotterell. 2022. On the probability– quality paradox in language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 36–45, Dublin, Ireland. Association for Computat… [find](https://scholar.google.com/scholar?q=Clara+Meister%2C+Gian+Wiher%2C+Tiago+Pimentel%2C+and+Ryan+Cotterell%2E+2022%2E+On+the+probability%E2%80%93+quality+paradox+in+language+generation%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Comput)
31. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pag… [find](https://scholar.google.com/scholar?q=Jiwei+Li%2C+Michel+Galley%2C+Chris+Brockett%2C+Jianfeng+Gao%2C+and+Bill+Dolan%2E+2016%2E+A+diversity%2Dpromoting+objective+function+for+neural+conversation+models%2E+In+Proceedings+of+the+2016+Conference+of+the+North) [our card](https://md.research.elamur.ai/free/en/papers/li-2016-diversity-promoting-objective-function-neural/)
32. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. In Proceedings of the 5th International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Stephen+Merity%2C+Caiming+Xiong%2C+James+Bradbury%2C+and+Richard+Socher%2E+2017%2E+Pointer+sentinel+mixture+models%2E+In+Proceedings+of+the+5th+International+Conference+on+Learning+Representations%2E) [our card](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/)
33. Margaret Li, Stephen Roller, Ilia Kulikov, Sean Welleck, Y-Lan Boureau, Kyunghyun Cho, and Jason Weston. 2020. Don’t say that! Making inconsistent dialogue unlikely with unlikelihood training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4715–4728… [find](https://scholar.google.com/scholar?q=Margaret+Li%2C+Stephen+Roller%2C+Ilia+Kulikov%2C+Sean+Welleck%2C+Y%2DLan+Boureau%2C+Kyunghyun+Cho%2C+and+Jason+Weston%2E+2020%2E+Don%E2%80%99t+say+that%21+Making+inconsistent+dialogue+unlikely+with+unlikelihood+training%2E+In+Proc) [our card](https://md.research.elamur.ai/free/en/papers/li-2020-don-t-say-that-making/)
34. Moin Nadeem, Tianxing He, Kyunghyun Cho, and James Glass. 2020. A systematic characterization of sampling algorithms for open-ended language generation. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint… [find](https://scholar.google.com/scholar?q=Moin+Nadeem%2C+Tianxing+He%2C+Kyunghyun+Cho%2C+and+James+Glass%2E+2020%2E+A+systematic+characterization+of+sampling+algorithms+for+open%2Dended+language+generation%2E+In+Proceedings+of+the+1st+Conference+of+the+Asi)
35. Kyle Mahowald, Evelina Fedorenko, Steven T. Piantadosi, and Edward Gibson. 2013. Info/information theory: Speakers choose shorter words in predictive contexts. Cognition, 126(2):313–318. https://doi.org/10 .1016/j.cognition.2012.09.010, PubMed: 23116925 [find](https://scholar.google.com/scholar?q=Kyle+Mahowald%2C+Evelina+Fedorenko%2C+Steven+T%2E+Piantadosi%2C+and+Edward+Gibson%2E+2013%2E+Info%2Finformation+theory%3A+Speakers+choose+shorter+words+in+predictive+contexts%2E+Cognition%2C+126%282%29%3A313%E2%80%93318%2E+https%3A%2F%2Fdoi%2Eo)
36. Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280–290, Berlin, Germany. Association for… [find](https://scholar.google.com/scholar?q=Ramesh+Nallapati%2C+Bowen+Zhou%2C+Cicero+dos+Santos%2C+Caglar+Gulc%CC%A7ehre%2C+and+Bing+Xiang%2E+2016%2E+Abstractive+text+summarization+using+sequence%2Dto%2Dsequence+RNNs+and+beyond%2E+In+Proceedings+of+The+20th+SIGNLL+Co)
37. Brockway McMillan. 1953. The basic theorems of information theory. The Annals of Mathematical Statistics, 24(2):196–219. https://doi.org/10.1214/aoms/1177729028 — [doi:10.1214/aoms/1177729028](https://doi.org/10.1214/aoms/1177729028) [↗](https://doi.org/10.1214/aoms/1177729028)
38. Clara Meister, Elizabeth Salesky, and Ryan Cotterell. 2020a. Generalized entropy regularization or: There’s nothing special about label smoothing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6870–6886, Online. Association for Computational Lingui… [find](https://scholar.google.com/scholar?q=Clara+Meister%2C+Elizabeth+Salesky%2C+and+Ryan+Cotterell%2E+2020a%2E+Generalized+entropy+regularization+or%3A+There%E2%80%99s+nothing+special+about+label+smoothing%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Asso)
39. Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook FAIR’s WMT19 news translation task submission. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 314–319, Carson T. Schütze. 2016. The empirical b… [find](https://scholar.google.com/scholar?q=Nathan+Ng%2C+Kyra+Yee%2C+Alexei+Baevski%2C+Myle+Ott%2C+Michael+Auli%2C+and+Sergey+Edunov%2E+2019%2E+Facebook+FAIR%E2%80%99s+WMT19+news+translation+task+submission%2E+In+Proceedings+of+the+Fourth+Conference+on+Machine+Transla) [our card](https://md.research.elamur.ai/free/en/papers/ng-2019-facebook-fair-s-wmt19-news/)
40. Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Florence%2C+Italy%2E+Association+for+Computational+Linguistics%2E)
41. Gabriel Pereyra, George Tucker, Jan Chorowski, Łukasz Kaiser, and Geoffrey E. Hinton. 2017. Regularizing neural networks by penalizing confident output distributions. In Proceedings of the 5th International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Gabriel+Pereyra%2C+George+Tucker%2C+Jan+Chorowski%2C+%C5%81ukasz+Kaiser%2C+and+Geoffrey+E%2E+Hinton%2E+2017%2E+Regularizing+neural+networks+by+penalizing+confident+output+distributions%2E+In+Proceedings+of+the+5th+Interna) [our card](https://md.research.elamur.ai/free/en/papers/pereyra-2017-regularizing-neural-networks-penalizing-confident/)
42. Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D. Manning. 2019. Do massively pretrained language models make better storytellers? In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 843–861, Hong Kong, China. Association for Com… [find](https://scholar.google.com/scholar?q=Abigail+See%2C+Aneesh+Pappu%2C+Rohun+Saxena%2C+Akhila+Yerukola%2C+and+Christopher+D%2E+Manning%2E+2019%2E+Do+massively+pretrained+language+models+make+better+storytellers%3F+In+Proceedings+of+the+23rd+Conference+on+C)
43. Steven T. Piantadosi, Harry Tily, and Edward Gibson. 2011. Word lengths are optimized for efficient communication. Proceedings of the National Academy of Sciences, 108(9):3526–3529. https://doi.org/10.1073/pnas.1012551108, PubMed: 21278332 — [doi:10.1073/pnas.1012551108](https://doi.org/10.1073/pnas.1012551108) [↗](https://doi.org/10.1073/pnas.1012551108)
44. Claude E. Shannon. 1948. A mathematical theory of communication. Bell System Technical Journal, 27:623–656. [find](https://scholar.google.com/scholar?q=Claude+E%2E+Shannon%2E+1948%2E+A+mathematical+theory+of+communication%2E+Bell+System+Technical+Journal%2C+27%3A623%E2%80%93656%2E)
45. Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021. MAUVE: Measuring the gap between neural text and human text using divergence frontiers. In Advances in Neural Information Processing Systems, volume 34, pages 4816–4828. Curran As… [find](https://scholar.google.com/scholar?q=Krishna+Pillutla%2C+Swabha+Swayamdipta%2C+Rowan+Zellers%2C+John+Thickstun%2C+Sean+Welleck%2C+Yejin+Choi%2C+and+Zaid+Harchaoui%2E+2021%2E+MAUVE%3A+Measuring+the+gap+between+neural+text+and+human+text+using+divergence+fr) [our card](https://md.research.elamur.ai/free/en/papers/pillutla-2021-mauve-measuring-gap-between-neural/)
46. Claude E. Shannon. 1951. Prediction and entropy of printed English. Bell System Technical Journal, 30(1):50–64. https://doi.org/10 .1002/j.1538-7305.1951.tb01366.x Felix Stahlberg and Bill Byrne. 2019. On NMT search errors and model errors: Cat got your tongue? In Proceedings of the 2019 Conference… [find](https://scholar.google.com/scholar?q=Claude+E%2E+Shannon%2E+1951%2E+Prediction+and+entropy+of+printed+English%2E+Bell+System+Technical+Journal%2C+30%281%29%3A50%E2%80%9364%2E+https%3A%2F%2Fdoi%2Eorg%2F10+%2E1002%2Fj%2E1538%2D7305%2E1951%2Etb01366%2Ex+Felix+Stahlberg+and+Bill+Byrne%2E+2019) [our card](https://md.research.elamur.ai/free/en/papers/stahlberg-2019-nmt-search-errors-model-errors/)
47. Tiago Pimentel, Clara Meister, and Ryan Cotterell. 2022. Cluster-based evaluation of automatically generated text. arXiv preprint arXiv:2205.16001. — [arXiv:2205.16001](https://arxiv.org/abs/2205.16001) [find](https://scholar.google.com/scholar?q=Tiago+Pimentel%2C+Clara+Meister%2C+and+Ryan+Cotterell%2E+2022%2E+Cluster%2Dbased+evaluation+of+automatically+generated+text%2E+arXiv+preprint+arXiv%3A2205%2E16001%2E+%E2%80%94+%5BarXiv%3A2205%2E16001%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2205%2E1600)
48. Tiago Pimentel, Clara Meister, Elizabeth Salesky, Simone Teufel, Damián Blasi, and Ryan Cotterell. 2021. A surprisal–duration trade-off across and within the world’s languages. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 949–962, Online and Punta… [find](https://scholar.google.com/scholar?q=Tiago+Pimentel%2C+Clara+Meister%2C+Elizabeth+Salesky%2C+Simone+Teufel%2C+Damia%CC%81n+Blasi%2C+and+Ryan+Cotterell%2E+2021%2E+A+surprisal%E2%80%93duration+trade%2Doff+across+and+within+the+world%E2%80%99s+languages%2E+In+Proceedings+of+the+)
49. Chris van der Lee, Albert Gatt, Emiel van Miltenburg, Sander Wubben, and Emiel Krahmer. 2019. Best practices for the human evaluation of automatically generated text. In Proceedings of the 12th International Conference on Natural Language Generation, pages 355–368, Tokyo, Japan. Association for Comp… [find](https://scholar.google.com/scholar?q=Chris+van+der+Lee%2C+Albert+Gatt%2C+Emiel+van+Miltenburg%2C+Sander+Wubben%2C+and+Emiel+Krahmer%2E+2019%2E+Best+practices+for+the+human+evaluation+of+automatically+generated+text%2E+In+Proceedings+of+the+12th+Intern)
50. Tiago Pimentel, Brian Roark, and Ryan Cotterell. 2020. Phonotactic complexity and its trade-offs. Transactions of the Association for Computational Linguistics, 8:1–18. https://doi.org/10.1162/tacl a 00296 [↗](https://doi.org/10.1162/tacl)
51. Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020. Neural text generation with unlikelihood training. In Proceedings of the 8th International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Sean+Welleck%2C+Ilia+Kulikov%2C+Stephen+Roller%2C+Emily+Dinan%2C+Kyunghyun+Cho%2C+and+Jason+Weston%2E+2020%2E+Neural+text+generation+with+unlikelihood+training%2E+In+Proceedings+of+the+8th+International+Conference+on) [our card](https://md.research.elamur.ai/free/en/papers/welleck-2019-neural-text-generation-unlikelihood-training/)
52. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E)
53. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Peter A. Reich. 1969. The finiteness of natural language. Language, 45(4):831–843. https://doi.org/10.2307/412337 — [doi:10.2307/412337](https://doi.org/10.2307/412337) [↗](https://doi.org/10.2307/412337)
54. Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language P… [find](https://scholar.google.com/scholar?q=Patrick+von+Platen%2C+Clara+Ma%2C+Yacine+Jernite%2C+Julien+Plu%2C+Canwen+Xu%2C+Teven+Le+Scao%2C+Sylvain+Gugger%2C+Mariama+Drame%2C+Quentin+Lhoest%2C+and+Alexander+M%2E+Rush%2E+2020%2E+Transformers%3A+State%2Dof%2Dthe%2Dart+natural+l)
55. Noga Zaslavsky, Charles Kemp, Terry Regier, and Naftali Tishby. 2018. Efficient compression in color naming and its evolution. Proceedings of the National Academy of Sciences, 115(31):7937–7942. https://doi.org/10 .1073/pnas.1800521115, PubMed: 30021851 [find](https://scholar.google.com/scholar?q=Noga+Zaslavsky%2C+Charles+Kemp%2C+Terry+Regier%2C+and+Naftali+Tishby%2E+2018%2E+Efficient+compression+in+color+naming+and+its+evolution%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+115%2831%29%3A7937%E2%80%937942%2E+https)
56. Hugh Zhang, Daniel Duckworth, Daphne Ippolito, and Arvind Neelakantan. 2021. Trading off diversity and quality in natural language generation. In Proceedings of the Workshop on Human Evaluation of NLP Systems (Hum-Eval), pages 25–33, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Hugh+Zhang%2C+Daniel+Duckworth%2C+Daphne+Ippolito%2C+and+Arvind+Neelakantan%2E+2021%2E+Trading+off+diversity+and+quality+in+natural+language+generation%2E+In+Proceedings+of+the+Workshop+on+Human+Evaluation+of+NLP)
57. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V%2E+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+Jeff+Klingner%2C+Apurva+Shah%2C+Melvin+Johnson%2C+Xiaobing+Liu%2C+Luk)