---
title: "Enabling Language Models to Fill in the Blanks — Donahue 2020"
description: "Donahue, Lee and Liang propose infilling by language modeling (ILM). 28 references, cited by 82 papers here."
authors: ["Chris Donahue", "Mina Lee", "Percy Liang"]
year: 2020
venue: "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/2020.acl-main.225/"
doi: "10.18653/v1/2020.acl-main.225"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/donahue-2020-enabling-language-models-fill-blanks/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/donahue-2020-enabling-language-models-fill-blanks/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/) / Enabling Language Models to Fill in the Blanks

# Enabling Language Models to Fill in the Blanks

**Authors:** Chris Donahue, Mina Lee and [Percy Liang](https://md.research.elamur.ai/free/en/authors/percy-liang/) · **Year:** 2020 · **Venue:** Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.acl-main.225/) · [PDF](https://aclanthology.org/2020.acl-main.225.pdf) · [DOI 10.18653/v1/2020.acl-main.225](https://doi.org/10.18653/v1/2020.acl-main.225) · [OpenAlex](https://openalex.org/W3024131638)

**Citations:** 125 citations (OpenAlex, 2026-10-14) · [82](https://md.research.elamur.ai/free/en/papers/donahue-2020-enabling-language-models-fill-blanks/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
Standard left-to-right language models generate text well but can only use preceding context, so they cannot fill gaps in the middle of a document. Bidirectional models such as BERT and SpanBERT see both sides but need the span length in advance. Donahue, Lee and Liang propose infilling by language modeling (ILM). Spans of a complete text are randomly replaced with blank tokens. The model is then trained to read this masked text, followed by a separator and only the removed spans, each ending in an answer token. Any off-the-shelf LM can learn this format; the authors fine-tune GPT-2 on three domains: short STORIES, CS paper ABSTRACTS and song LYRICS. ILM matches the perplexity of a much costlier baseline that regenerates the whole document, while keeping sequences almost as short as ordinary language modeling. Starting from pre-trained weights consistently beat training from scratch. In a human study on stories, ILM-infilled sentences were the hardest to tell apart from human writing. This makes the method a practical, architecture-agnostic basis for writing-assistance tools.

## Key points
- **Low overhead:** For k blanks, an ILM training example is only 2k + 1 tokens longer than the original text, with k averaging about 2. The approach also adds just three special tokens to the vocabulary. The LM-All baseline, by contrast, makes sequences about 1.81× as long, versus about 1.01× for ILM.
- **Masking scheme:** The mask function walks a granularity hierarchy (document, paragraph, sentence, n-gram, word) and masks whole subtrees with 3% probability. This yields roughly 15% of tokens masked, echoing BERT. Each granularity has its own blank token, which gives users coarse control over span length.
- **Experimental comparison:** All strategies use GPT-2 "small" with identical hyperparameters. ILM is compared with three baselines:
  - LM (past context only);
  - LM-Rev (future context only);
  - LM-All (masked plus full text, following Fedus et al., 2018).
- **Human evaluation:** Annotators saw five-sentence STORIES with one machine-infilled sentence and had to identify it. Each system received 100 responses, and the maximum achievable score is effectively 80%. ILM outperformed BERT, the self-attention model of Zhu et al. (2019) (SA) and plain LM. Unexpectedly, plain LM also beat both BERT and SA.
- **Released resources:** Code, data and trained models are on GitHub (chrisdonahue/ilm) and CodaLab, and an interactive web demo is also available.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/)

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 248 works · 214,804 citations

## Cite

Cite as: Chris Donahue, Mina Lee and Percy Liang. 2020. Enabling Language Models to Fill in the Blanks. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{donahue2020enabling,
  title = {{Enabling Language Models to Fill in the Blanks}},
  author = {Chris Donahue, Mina Lee and Percy Liang},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.225},
  url = {https://doi.org/10.18653/v1/2020.acl-main.225}
}
```

- [← previous Logical Natural Language Generation from Open-Domain Tables](https://md.research.elamur.ai/free/en/papers/chen-2020-logical-natural-language-generation-open/)
- [next → Reasoning with Multimodal Sarcastic Tweets via Modeling Cross-Modality Contrast and Semantic Association](https://md.research.elamur.ai/free/en/papers/xu-2020-reasoning-multimodal-sarcastic-tweets-modeling/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/)

## References

28 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. AI21. 2019. HAIM: A modest step towards controllable text generation. AI21 Labs Blog. [find](https://scholar.google.com/scholar?q=AI21%2E+2019%2E+HAIM%3A+A+modest+step+towards+controllable+text+generation%2E+AI21+Labs+Blog%2E)
2. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8). [find](https://scholar.google.com/scholar?q=A%2E+Radford%2C+J%2E+Wu%2C+R%2E+Child%2C+D%2E+Luan%2C+D%2E+Amodei%2C+and+I%2E+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+Blog%2C+1%288%29%2E)
3. Yannis Assael, Thea Sommerschield, and Jonathan Prag. 2019. Restoring ancient text using deep learning: a case study on greek epigraphy. arXiv:1910.06262. [find](https://scholar.google.com/scholar?q=Yannis+Assael%2C+Thea+Sommerschield%2C+and+Jonathan+Prag%2E+2019%2E+Restoring+ancient+text+using+deep+learning%3A+a+case+study+on+greek+epigraphy%2E+arXiv%3A1910%2E06262%2E)
4. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. [our card](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/)
5. N. Chambers and D. Jurafsky. 2008. Unsupervised learning of narrative event chains. In Human Language Technology and Association for Computational Linguistics (HLT/ACL). [find](https://scholar.google.com/scholar?q=N%2E+Chambers+and+D%2E+Jurafsky%2E+2008%2E+Unsupervised+learning+of+narrative+event+chains%2E+In+Human+Language+Technology+and+Association+for+Computational+Linguistics+%28HLT%2FACL%29%2E)
6. R. Rudinger, P. Rastogi, F. Ferraro, and B. V. Durme. 2015. Script induction as language modeling. In Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=R%2E+Rudinger%2C+P%2E+Rastogi%2C+F%2E+Ferraro%2C+and+B%2E+V%2E+Durme%2E+2015%2E+Script+induction+as+language+modeling%2E+In+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2E)
7. Elizabeth Clark, Yangfeng Ji, and Noah A Smith. 2018. Neural text generation in stories using entity representations as context. In Association for Computational Linguistics: Human Language Technologies. [find](https://scholar.google.com/scholar?q=Elizabeth+Clark%2C+Yangfeng+Ji%2C+and+Noah+A+Smith%2E+2018%2E+Neural+text+generation+in+stories+using+entity+representations+as+context%2E+In+Association+for+Computational+Linguistics%3A+Human+Language+Technologi)
8. Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D Manning. 2019. Do massively pretrained language models make better storytellers? arXiv:1909.10705. [find](https://scholar.google.com/scholar?q=Abigail+See%2C+Aneesh+Pappu%2C+Rohun+Saxena%2C+Akhila+Yerukola%2C+and+Christopher+D+Manning%2E+2019%2E+Do+massively+pretrained+language+models+make+better+storytellers%3F+arXiv%3A1909%2E10705%2E)
9. J. Devlin, M. Chang, K. Lee, and K. Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Association for Computational Linguistics (ACL), pages 4171– 4186. [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
10. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909. [our card](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/)
11. T. Deyes. 1984. Towards an authentic ‘discourse cloze’. Applied Linguistics, 5(2):128–137. [find](https://scholar.google.com/scholar?q=T%2E+Deyes%2E+1984%2E+Towards+an+authentic+%E2%80%98discourse+cloze%E2%80%99%2E+Applied+Linguistics%2C+5%282%29%3A128%E2%80%93137%2E)
12. A. Fan, M. Lewis, and Y. Dauphin. 2018. Hierarchical neural story generation. arXiv preprint arXiv:1805.04833. [find](https://scholar.google.com/scholar?q=A%2E+Fan%2C+M%2E+Lewis%2C+and+Y%2E+Dauphin%2E+2018%2E+Hierarchical+neural+story+generation%2E+arXiv+preprint+arXiv%3A1805%2E04833%2E)
13. W. Fedus, I. Goodfellow, and A. M. Dai. 2018. Maskgan: Better text generation via filling in the. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=W%2E+Fedus%2C+I%2E+Goodfellow%2C+and+A%2E+M%2E+Dai%2E+2018%2E+Maskgan%3A+Better+text+generation+via+filling+in+the%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2E)
14. J. Gu, Q. Liu, and K. Cho. 2019. Insertion-based decoding with automatically inferred generation order. arXiv preprint arXiv:1902.01370. [find](https://scholar.google.com/scholar?q=J%2E+Gu%2C+Q%2E+Liu%2C+and+K%2E+Cho%2E+2019%2E+Insertion%2Dbased+decoding+with+automatically+inferred+generation+order%2E+arXiv+preprint+arXiv%3A1902%2E01370%2E)
15. D. Ippolito, D. Grangier, C. Callison-Burch, and D. Eck. 2019. Unsupervised hierarchical story infilling. In NAACL Workshop on Narrative Understanding, pages 37–43. [find](https://scholar.google.com/scholar?q=D%2E+Ippolito%2C+D%2E+Grangier%2C+C%2E+Callison%2DBurch%2C+and+D%2E+Eck%2E+2019%2E+Unsupervised+hierarchical+story+infilling%2E+In+NAACL+Workshop+on+Narrative+Understanding%2C+pages+37%E2%80%9343%2E)
16. M. Joshi, D. Chen, Y. Liu, D. S. Weld, L. Zettlemoyer, and O. Levy. 2019. SpanBERT: Improving pre-training by representing and predicting spans. arXiv preprint arXiv:1907.10529. [our card](https://md.research.elamur.ai/free/en/papers/joshi-2019-spanbert-improving-pre-training-representing/)
17. D. Liu, J. Fu, P. Liu, and J. Lv. 2019. TIGS: An inference algorithm for text infilling with gradient search. arXiv preprint arXiv:1905.10752. [find](https://scholar.google.com/scholar?q=D%2E+Liu%2C+J%2E+Fu%2C+P%2E+Liu%2C+and+J%2E+Lv%2E+2019%2E+TIGS%3A+An+inference+algorithm+for+text+infilling+with+gradient+search%2E+arXiv+preprint+arXiv%3A1905%2E10752%2E)
18. N. Mostafazadeh, N. Chambers, X. He, D. Parikh, D. Batra, L. Vanderwende, P. Kohli, and J. Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In North American Association for Computational Linguistics (NAACL). [find](https://scholar.google.com/scholar?q=N%2E+Mostafazadeh%2C+N%2E+Chambers%2C+X%2E+He%2C+D%2E+Parikh%2C+D%2E+Batra%2C+L%2E+Vanderwende%2C+P%2E+Kohli%2C+and+J%2E+Allen%2E+2016%2E+A+corpus+and+cloze+evaluation+for+deeper+understanding+of+commonsense+stories%2E+In+North+American)
19. Tianxiao Shen, Victor Quach, Regina Barzilay, and Tommi Jaakkola. 2020. Blank language models. arXiv:2002.03079. [find](https://scholar.google.com/scholar?q=Tianxiao+Shen%2C+Victor+Quach%2C+Regina+Barzilay%2C+and+Tommi+Jaakkola%2E+2020%2E+Blank+language+models%2E+arXiv%3A2002%2E03079%2E)
20. Y. Shih, W. Chang, and Y. Yang. 2019. XL-Editor: Post-editing sentences with xlnet. arXiv preprint arXiv:1910.10479. — [arXiv:1910.10479](https://arxiv.org/abs/1910.10479) [find](https://scholar.google.com/scholar?q=Y%2E+Shih%2C+W%2E+Chang%2C+and+Y%2E+Yang%2E+2019%2E+XL%2DEditor%3A+Post%2Dediting+sentences+with+xlnet%2E+arXiv+preprint+arXiv%3A1910%2E10479%2E+%E2%80%94+%5BarXiv%3A1910%2E10479%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1910%2E10479%29)
21. W. L. Taylor. 1953. “Cloze procedure”: A new tool for measuring readability. Journalism Bulletin, 30(4):415–433. [find](https://scholar.google.com/scholar?q=W%2E+L%2E+Taylor%2E+1953%2E+%E2%80%9CCloze+procedure%E2%80%9D%3A+A+new+tool+for+measuring+readability%2E+Journalism+Bulletin%2C+30%284%29%3A415%E2%80%93433%2E)
22. A. Wang and K. Cho. 2019. BERT has a mouth, and it must speak: BERT as a Markov random field language model. arXiv preprint arXiv:1902.04094. [our card](https://md.research.elamur.ai/free/en/papers/wang-2019-bert-has-mouth-must-speak/)
23. T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, and J. Brew. 2019. HuggingFace’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771. [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
24. Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, and Q. V. Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237. [our card](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/)
25. L. Yao, N. Peng, R. Weischedel, K. Knight, D. Zhao, and R. Yan. 2019. Plan-and-write: Towards better automatic storytelling. In Association for the Advancement of Artificial Intelligence (AAAI). [find](https://scholar.google.com/scholar?q=L%2E+Yao%2C+N%2E+Peng%2C+R%2E+Weischedel%2C+K%2E+Knight%2C+D%2E+Zhao%2C+and+R%2E+Yan%2E+2019%2E+Plan%2Dand%2Dwrite%3A+Towards+better+automatic+storytelling%2E+In+Association+for+the+Advancement+of+Artificial+Intelligence+%28AAAI%29%2E)
26. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In NeurIPS. [our card](https://md.research.elamur.ai/free/en/papers/zellers-2019-defending-against-neural-fake-news/)
27. W. Zhu, Z. Hu, and E. Xing. 2019. Text infilling. arXiv preprint arXiv:1901.00158. — [arXiv:1901.00158](https://arxiv.org/abs/1901.00158) [find](https://scholar.google.com/scholar?q=W%2E+Zhu%2C+Z%2E+Hu%2C+and+E%2E+Xing%2E+2019%2E+Text+infilling%2E+arXiv+preprint+arXiv%3A1901%2E00158%2E+%E2%80%94+%5BarXiv%3A1901%2E00158%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1901%2E00158%29)
28. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In ACL. [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)