---
title: "DiffusionBERT — paper by He et al. (2023) — summary and ideas"
description: "They add two design changes: Spindle noise schedule: tokens are masked according to how much… 32 references, cited by 49 papers here."
authors: ["Zhengfu He", "Tianxiang Sun", "Qiong Tang", "Kuanning Wang", "Xuanjing Huang", "Xipeng Qiu"]
year: 2023
venue: "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2023.acl-long.248/"
doi: "10.18653/v1/2023.acl-long.248"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/he-2023-diffusionbert-improving-generative-masked-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/he-2023-diffusionbert-improving-generative-masked-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/) / DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models

# DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models

**Authors:** Zhengfu He, [Tianxiang Sun](https://md.research.elamur.ai/free/en/authors/tianxiang-sun/), [Qiong Tang](https://md.research.elamur.ai/free/en/authors/qiong-tang/), Kuanning Wang, [Xuanjing Huang](https://md.research.elamur.ai/free/en/authors/xuanjing-huang/) and [Xipeng Qiu](https://md.research.elamur.ai/free/en/authors/xipeng-qiu/) · **Year:** 2023 · **Venue:** [Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2023](https://md.research.elamur.ai/free/en/venues/acl/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.acl-long.248/) · [PDF](https://aclanthology.org/2023.acl-long.248.pdf) · [DOI 10.18653/v1/2023.acl-long.248](https://doi.org/10.18653/v1/2023.acl-long.248) · [OpenAlex](https://openalex.org/W4385569744)

**Citations:** 85 citations (OpenAlex, 2026-10-14) · [49](https://md.research.elamur.ai/free/en/papers/he-2023-diffusionbert-improving-generative-masked-language/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Text is discrete, which makes diffusion models hard to apply to it. Earlier diffusion approaches for text also ignored pre-trained language models. DiffusionBERT addresses this by noting that BERT and diffusion models both learn to denoise. The authors use BERT as the network that reverses a discrete diffusion process, in which tokens are progressively replaced by an absorbing [MASK] state.

They add two design changes:
- **Spindle noise schedule:** tokens are masked according to how much information they carry. The most informative tokens are corrupted first, and the corruption is spread evenly across steps. As a result, the reverse process produces easy, frequent tokens first.
- **Time-agnostic decoding (TAD):** after comparing several ways to feed the time step into BERT, the authors found that leaving it out entirely worked best.

In unconditional generation, DiffusionBERT beats discrete and continuous text diffusion models and earlier generative masked LMs on perplexity and BLEU. In conditional tasks, its quality is competitive and its output is clearly more diverse. Starting from pre-trained weights also speeds up convergence substantially. This makes non-autoregressive generation built on existing PLMs more practical.

## Key points
- DiffusionBERT is presented as the first combination of diffusion models with pre-trained LMs. It outperforms D3PM, Diffusion-LM and BERT-Mouth in unconditional generation, measured by perplexity and BLEU.
- The spindle schedule makes the forward process non-Markovian, since each noising step depends on x0 as well as the previous step. The original training objective stays unchanged.
- All parameters come from pre-trained models, with no new time-step parameters. Training reaches convergence in about 40% of the steps D3PM needs and 20% of those DiffuSeq needs.
- Conditional generation was evaluated with BLEU and Rouge-L for quality and with Self-BLEU and Div-4 for diversity. DiffusionBERT outperforms 7 baselines on diversity, and MBR decoding further improves quality.
- Limitations:
  - Sampling is slower than with fine-tuned GPT.
  - Some long outputs (length 128) lack coherence.
  - MBR decoding raises inference cost several times over.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Fudan University](https://md.research.elamur.ai/free/en/universities/fudan-university/) — University, Shanghai, China · 65 works · 19,402 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works

## Cite

Cite as: Zhengfu He, Tianxiang Sun, Qiong Tang, Kuanning Wang, Xuanjing Huang and Xipeng Qiu. 2023. DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{he2023diffusionbert,
  title = {{DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models}},
  author = {Zhengfu He, Tianxiang Sun, Qiong Tang, Kuanning Wang, Xuanjing Huang and Xipeng Qiu},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.248},
  url = {https://doi.org/10.18653/v1/2023.acl-long.248}
}
```

- [← previous Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models](https://md.research.elamur.ai/free/en/papers/singh-2023-human-data-scaling-self-training/)
- [next → Theory of Mind for Multi-Agent Collaboration via Large Language Models](https://md.research.elamur.ai/free/en/papers/li-2023-theory-mind-multi-agent-collaboration/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/)

## References

32 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. 2021. [Structured denoising diffusion models in discrete state-spaces](https://md.research.elamur.ai/free/en/papers/austin-2021-structured-denoising-diffusion-models-discrete/). In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December…
2. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… [find](https://scholar.google.com/scholar?q=Tom+B%2E+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretche)
3. Ciprian Chelba, Tomás Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2014. [One billion word benchmark for measuring progress in statistical language modeling](https://md.research.elamur.ai/free/en/papers/chelba-2013-one-billion-word-benchmark-measuring/). In INTER-SPEECH 2014, 15th Annual Conference of the International Speech Communication Association, Singa…
4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
5. Bhuwan Dhingra, Kathryn Mazaitis, and William W. Cohen. 2017. Quasar: Datasets for question answering by search and reading. CoRR, abs/1707.03904. [find](https://scholar.google.com/scholar?q=Bhuwan+Dhingra%2C+Kathryn+Mazaitis%2C+and+William+W%2E+Cohen%2E+2017%2E+Quasar%3A+Datasets+for+question+answering+by+search+and+reading%2E+CoRR%2C+abs%2F1707%2E03904%2E)
6. Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. [Mask-predict: Parallel decoding of conditional masked language models](https://md.research.elamur.ai/free/en/papers/ghazvininejad-2019-mask-predict-parallel-decoding-conditional/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Pro…
7. Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. 2022. Diffuseq: Sequence to sequence text generation with diffusion models. CoRR, abs/2210.08933. [find](https://scholar.google.com/scholar?q=Shansan+Gong%2C+Mukai+Li%2C+Jiangtao+Feng%2C+Zhiyong+Wu%2C+and+Lingpeng+Kong%2E+2022%2E+Diffuseq%3A+Sequence+to+sequence+text+generation+with+diffusion+models%2E+CoRR%2C+abs%2F2210%2E08933%2E)
8. Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. K. Li, and Richard Socher. 2018. [Non-autoregressive neural machine translation](https://md.research.elamur.ai/free/en/papers/gu-2017-non-autoregressive-neural-machine-translation/). In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
9. Jiatao Gu, Changhan Wang, and Junbo Zhao. 2019. Levenshtein transformer. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 11179–11189. [find](https://scholar.google.com/scholar?q=Jiatao+Gu%2C+Changhan+Wang%2C+and+Junbo+Zhao%2E+2019%2E+Levenshtein+transformer%2E+In+Advances+in+Neural+Information+Processing+Systems+32%3A+Annual+Conference+on+Neural+Information+Processing+Systems+2019%2C+NeurI)
10. Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. [find](https://scholar.google.com/scholar?q=Jonathan+Ho%2C+Ajay+Jain%2C+and+Pieter+Abbeel%2E+2020%2E+Denoising+diffusion+probabilistic+models%2E+In+Advances+in+Neural+Information+Processing+Systems+33%3A+Annual+Conference+on+Neural+Information+Processing+S)
11. Emiel Hoogeboom, Alexey A. Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. 2022. Autoregressive diffusion models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. [find](https://scholar.google.com/scholar?q=Emiel+Hoogeboom%2C+Alexey+A%2E+Gritsenko%2C+Jasmijn+Bastings%2C+Ben+Poole%2C+Rianne+van+den+Berg%2C+and+Tim+Salimans%2E+2022%2E+Autoregressive+diffusion+models%2E+In+The+Tenth+International+Conference+on+Learning+Repre)
12. Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. 2021. Argmax flows and multinomial diffusion: Towards non-autoregressive language models. CoRR, abs/2102.05379. — [arXiv:2102.05379](https://arxiv.org/abs/2102.05379) [find](https://scholar.google.com/scholar?q=Emiel+Hoogeboom%2C+Didrik+Nielsen%2C+Priyank+Jaini%2C+Patrick+Forr%C3%A9%2C+and+Max+Welling%2E+2021%2E+Argmax+flows+and+multinomial+diffusion%3A+Towards+non%2Dautoregressive+language+models%2E+CoRR%2C+abs%2F2102%2E05379%2E+%E2%80%94+%5BarXiv)
13. Philipp Koehn. 2004. [Statistical significance tests for machine translation evaluation](https://md.research.elamur.ai/free/en/papers/koehn-2004-statistical-significance-tests-machine-translation/). In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing , EMNLP 2004, A meeting of SIGDAT, a Special Interest Group of the ACL, held in conjunction with ACL 2004, 25-26 July 2004…
14. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 3045– 3059.… [find](https://scholar.google.com/scholar?q=Brian+Lester%2C+Rami+Al%2DRfou%2C+and+Noah+Constant%2E+2021%2E+The+power+of+scale+for+parameter%2Defficient+prompt+tuning%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language+Processing)
15. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. [BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). In Proceedings of the 58th Annual Meeting of the…
16. Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B. Hashimoto. 2022. [Diffusion-lm improves controllable text generation](https://md.research.elamur.ai/free/en/papers/li-2022-diffusion-lm-improves-controllable-text/). CoRR, abs/2205.14217.
17. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2019%2E+Decoupled+weight+decay+regularization%2E+In+7th+International+Conference+on+Learning+Representations%2C+ICLR+2019%2C+New+Orleans%2C+LA%2C+USA%2C+May+6%2D9%2C+2019%2E+OpenReview%2En)
18. Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. 2022. Dpm-solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. CoRR, abs/2206.00927. — [arXiv:2206.00927](https://arxiv.org/abs/2206.00927) [find](https://scholar.google.com/scholar?q=Cheng+Lu%2C+Yuhao+Zhou%2C+Fan+Bao%2C+Jianfei+Chen%2C+Chongxuan+Li%2C+and+Jun+Zhu%2E+2022%2E+Dpm%2Dsolver%3A+A+fast+ODE+solver+for+diffusion+probabilistic+model+sampling+in+around+10+steps%2E+CoRR%2C+abs%2F2206%2E00927%2E+%E2%80%94+%5BarXi)
19. Alexander Quinn Nichol and Prafulla Dhariwal. 2021. Improved denoising diffusion probabilistic models. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8162–8171. PMLR. [find](https://scholar.google.com/scholar?q=Alexander+Quinn+Nichol+and+Prafulla+Dhariwal%2E+2021%2E+Improved+denoising+diffusion+probabilistic+models%2E+In+Proceedings+of+the+38th+International+Conference+on+Machine+Learning%2C+ICML+2021%2C+18%2D24+July+20)
20. Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. 2020. [Pre-trained models for natural language processing: A survey](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/). SCIENCE CHINA Technological Sciences.
21. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language understanding by generative pre-training. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+Ilya+Sutskever%2C+et+al%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E)
22. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). J. Mach. Learn. Res., 21:140:1–140:67.
23. Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. [Hierarchical text-conditional image generation with CLIP latents](https://md.research.elamur.ai/free/en/papers/ramesh-2022-hierarchical-text-conditional-image-generation/). CoRR, abs/2204.06125.
24. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 10674–10685. IEEE. [find](https://scholar.google.com/scholar?q=Robin+Rombach%2C+Andreas+Blattmann%2C+Dominik+Lorenz%2C+Patrick+Esser%2C+and+Bj%C3%B6rn+Ommer%2E+2022%2E+High%2Dresolution+image+synthesis+with+latent+diffusion+models%2E+In+IEEE%2FCVF+Conference+on+Computer+Vision+and+Patt)
25. Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. 2022. [Photorealistic text-to-image diffusion models with deep la…](https://md.research.elamur.ai/free/en/papers/saharia-2022-photorealistic-text-image-diffusion-models/)
26. Chitwan Saharia, William Chan, Saurabh Saxena, and Mohammad Norouzi. 2020. [Non-autoregressive machine translation with latent alignments](https://md.research.elamur.ai/free/en/papers/saharia-2020-non-autoregressive-machine-translation-latent/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 1098–1108. Association…
27. Nikolay Savinov, Junyoung Chung, Mikolaj Binkowski, Erich Elsen, and Aäron van den Oord. 2022. Step-unrolled denoising autoencoders for text generation. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. [find](https://scholar.google.com/scholar?q=Nikolay+Savinov%2C+Junyoung+Chung%2C+Mikolaj+Binkowski%2C+Erich+Elsen%2C+and+A%C3%A4ron+van+den+Oord%2E+2022%2E+Step%2Dunrolled+denoising+autoencoders+for+text+generation%2E+In+The+Tenth+International+Conference+on+Learni)
28. Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. 2015. [Deep unsupervised learning using nonequilibrium thermodynamics](https://md.research.elamur.ai/free/en/papers/sohl-dickstein-2015-deep-unsupervised-learning-using-nonequilibrium/). In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Confe…
29. Jiaming Song, Chenlin Meng, and Stefano Ermon. 2021. [Denoising diffusion implicit models](https://md.research.elamur.ai/free/en/papers/song-2020-denoising-diffusion-implicit-models/). In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
30. Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. 2022. Black-box tuning for language-model-as-a-service. In Proceedings of the 39th International Conference on Machine Learning, ICML 2022, Baltimore, Maryland, USA. [find](https://scholar.google.com/scholar?q=Tianxiang+Sun%2C+Yunfan+Shao%2C+Hong+Qian%2C+Xuanjing+Huang%2C+and+Xipeng+Qiu%2E+2022%2E+Black%2Dbox+tuning+for+language%2Dmodel%2Das%2Da%2Dservice%2E+In+Proceedings+of+the+39th+International+Conference+on+Machine+Learning%2C+)
31. Alex Wang and Kyunghyun Cho. 2019. [BERT has a mouth, and it must speak: BERT as a markov random field language model](https://md.research.elamur.ai/free/en/papers/wang-2019-bert-has-mouth-must-speak/). CoRR, abs/1902.04094.
32. Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. [Texygen: A benchmarking platform for text generation models](https://md.research.elamur.ai/free/en/papers/zhu-2018-texygen-benchmarking-platform-text-generation/). In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018, p…

*Updated 24 October 2026*