---
title: "Simplified and Generalized Masked Diffusion for Discrete Data"
description: "Masked (absorbing-state) diffusion is a possible alternative to autoregressive generation for discrete… 75 references, cited by 69 papers here."
authors: ["Jiaxin Shi", "Kehang Han", "Zhe Wang", "Arnaud Doucet", "Michalis K. Titsias"]
year: 2024
venue: "arXiv:2406.04329"
source: "https://arxiv.org/abs/2406.04329"
doi: "10.48550/arxiv.2406.04329"
arxiv: "2406.04329"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shi-2024-simplified-generalized-masked-diffusion-discrete/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shi-2024-simplified-generalized-masked-diffusion-discrete/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / Simplified and Generalized Masked Diffusion for Discrete Data

# Simplified and Generalized Masked Diffusion for Discrete Data

**Authors:** Jiaxin Shi, Kehang Han, [Zhe Wang](https://md.research.elamur.ai/free/en/authors/zhe-wang/), [Arnaud Doucet](https://md.research.elamur.ai/free/en/authors/arnaud-doucet/) and Michalis K. Titsias · **Year:** 2024 · **Venue:** [arXiv:2406.04329](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/) ([NeurIPS 2024](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/2024/)) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2406.04329) · [PDF](https://arxiv.org/pdf/2406.04329) · [DOI 10.48550/arxiv.2406.04329](https://doi.org/10.48550/arxiv.2406.04329) · [arXiv 2406.04329](https://arxiv.org/abs/2406.04329) · [OpenAlex](https://openalex.org/W4415796024)

**Citations:** 21 citations (OpenAlex, 2026-10-14) · [69](https://md.research.elamur.ai/free/en/papers/shi-2024-simplified-generalized-masked-diffusion-discrete/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Masked (absorbing-state) diffusion is a possible alternative to autoregressive generation for discrete data such as text and pixels. Earlier formulations, however, were convoluted, relied on awkward parameterizations and objectives, and needed ad hoc fixes. The Google DeepMind authors rebuild the approach in continuous time. They show that its variational bound reduces to a time-weighted integral of ordinary cross-entropy terms. As in continuous diffusion, the bound depends on the masking schedule only through the SNR at the endpoints, although the schedule still affects estimator variance. This view also explains how earlier continuous-time discrete diffusion methods depart from the bound, causing costly evaluation, high variance, or forward/reverse inconsistency. The resulting model, MD4, and a variant with state-dependent masking schedules, GenMD4, were trained on OpenWebText, text8, FineWeb-Edu, CIFAR-10 and ImageNet 64×64. They set new likelihood records for discrete diffusion and beat AR models of similar size on CIFAR-10 pixels. The work makes masked diffusion easier to implement and considerably more competitive.

## Key points
- The ELBO of masked diffusion becomes a weighted integral of cross-entropy losses. GenMD4 extends this to masking schedules that depend on the token state and gives further likelihood gains, for example a clearly better BPC on text8.
- At GPT-2 scale on OpenWebText, MD4 beats prior diffusion language models in perplexity and wins 4 of 5 zero-shot language modeling tasks.
- On pixel-level modeling, MD4 reaches 2.75 bits/dim on CIFAR-10 and 3.40 on ImageNet 64×64. This beats prior discrete diffusion models (D3PM, Campbell et al.) and is on par with VDM and with AR Transformers about 4× larger.
- On FineWeb-Edu, MD4 trails same-size AR models on Hellaswag but the gap does not widen with scale. AR-small hits 30% after 50k steps versus 200k for MD4; at medium scale, AR reaches 37% after 270k steps versus about 1M.
- Limitations: MD4 still lags AR models on text8, and GenMD4 tends to overfit and transfers less well zero-shot. Code is released at github.com/google-deepmind/md4 (NeurIPS 2024).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

## Cite

Cite as: Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet and Michalis K. Titsias. 2024. Simplified and Generalized Masked Diffusion for Discrete Data. arXiv:2406.04329.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{shi2024simplified,
  title = {{Simplified and Generalized Masked Diffusion for Discrete Data}},
  author = {Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet and Michalis K. Titsias},
  year = {2024},
  doi = {10.48550/arxiv.2406.04329},
  eprint = {2406.04329},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.04329}
}
```

- [← previous RQ-RAG: Learning to Refine Queries for Retrieval Augmented Generation](https://md.research.elamur.ai/free/en/papers/chan-2024-rq-rag-learning-refine-queries/)
- [next → Vidur: A Large-Scale Simulation Framework For LLM Inference](https://md.research.elamur.ai/free/en/papers/agrawal-2024-vidur-large-scale-simulation-framework/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/)

## References

75 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. [Deep unsupervised learning using nonequilibrium thermodynamics](https://md.research.elamur.ai/free/en/papers/sohl-dickstein-2015-deep-unsupervised-learning-using-nonequilibrium/). In International Conference on Machine Learning, 2015.
2. Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020. [find](https://scholar.google.com/scholar?q=Jonathan+Ho%2C+Ajay+Jain%2C+and+Pieter+Abbeel%2E+Denoising+diffusion+probabilistic+models%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2020%2E)
3. Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. [Score-based generative modeling through stochastic differential equations](https://md.research.elamur.ai/free/en/papers/song-2020-score-based-generative-modeling-through/). In International Conference on Learning Representations, 2020.
4. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. [High-resolution image synthesis with latent diffusion models](https://md.research.elamur.ai/free/en/papers/rombach-2021-high-resolution-image-synthesis-latent/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022.
5. Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. [Hierarchical text-conditional image generation with clip latents](https://md.research.elamur.ai/free/en/papers/ramesh-2022-hierarchical-text-conditional-image-generation/). arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
6. Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. [Photorealistic text-to-image diffusion models with deep language understanding](https://md.research.elamur.ai/free/en/papers/saharia-2022-photorealistic-text-image-diffusion-models/). In Advances in Neural Information Processing Systems,…
7. Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. In International Conference on Learning Representations, 2021. [find](https://scholar.google.com/scholar?q=Nanxin+Chen%2C+Yu+Zhang%2C+Heiga+Zen%2C+Ron+J+Weiss%2C+Mohammad+Norouzi%2C+and+William+Chan%2E+Wavegrad%3A+Estimating+gradients+for+waveform+generation%2E+In+International+Conference+on+Learning+Representations%2C+2021)
8. Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021. [find](https://scholar.google.com/scholar?q=Zhifeng+Kong%2C+Wei+Ping%2C+Jiaji+Huang%2C+Kexin+Zhao%2C+and+Bryan+Catanzaro%2E+Diffwave%3A+A+versatile+diffusion+model+for+audio+synthesis%2E+In+International+Conference+on+Learning+Representations%2C+2021%2E)
9. Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. In Advances in Neural Information Processing Systems, 2022. [find](https://scholar.google.com/scholar?q=Jonathan+Ho%2C+Tim+Salimans%2C+Alexey+Gritsenko%2C+William+Chan%2C+Mohammad+Norouzi%2C+and+David+J+Fleet%2E+Video+diffusion+models%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2022%2E)
10. Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference on Learning Representations, 2… [find](https://scholar.google.com/scholar?q=Ruben+Villegas%2C+Mohammad+Babaeizadeh%2C+Pieter%2DJan+Kindermans%2C+Hernan+Moraldo%2C+Han+Zhang%2C+Mohammad+Taghi+Saffar%2C+Santiago+Castro%2C+Julius+Kunze%2C+and+Dumitru+Erhan%2E+Phenaki%3A+Variable+length+video+generati)
11. Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli, et al. Lumiere: A space-time diffusion model for video generation. arXiv preprint arXiv:2401.12945, 2024. — [arXiv:2401.12945](https://arxiv.org/abs/2401.12945) [find](https://scholar.google.com/scholar?q=Omer+Bar%2DTal%2C+Hila+Chefer%2C+Omer+Tov%2C+Charles+Herrmann%2C+Roni+Paiss%2C+Shiran+Zada%2C+Ariel+Ephrat%2C+Junhwa+Hur%2C+Yuanzhen+Li%2C+Tomer+Michaeli%2C+et+al%2E+Lumiere%3A+A+space%2Dtime+diffusion+model+for+video+generation)
12. OpenAI. Sora. [https://openai.com/index/sora/](https://openai.com/index/sora/), 2024. [↗](https://openai.com/index/sora/)
13. Fan Bao, Chendong Xiang, Gang Yue, Guande He, Hongzhou Zhu, Kaiwen Zheng, Min Zhao, Shilong Liu, Yaole Wang, and Jun Zhu. Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models. arXiv preprint arXiv:2405.04233, 2024. — [arXiv:2405.04233](https://arxiv.org/abs/2405.04233) [find](https://scholar.google.com/scholar?q=Fan+Bao%2C+Chendong+Xiang%2C+Gang+Yue%2C+Guande+He%2C+Hongzhou+Zhu%2C+Kaiwen+Zheng%2C+Min+Zhao%2C+Shilong+Liu%2C+Yaole+Wang%2C+and+Jun+Zhu%2E+Vidu%3A+a+highly+consistent%2C+dynamic+and+skilled+text%2Dto%2Dvideo+generator+with+di)
14. Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. [Structured denoising diffusion models in discrete state-spaces](https://md.research.elamur.ai/free/en/papers/austin-2021-structured-denoising-diffusion-models-discrete/). In Advances in Neural Information Processing Systems, 2021.
15. Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. In Advances in Neural Information Processing Systems, 2021. [find](https://scholar.google.com/scholar?q=Emiel+Hoogeboom%2C+Didrik+Nielsen%2C+Priyank+Jaini%2C+Patrick+Forr%C3%A9%2C+and+Max+Welling%2E+Argmax+flows+and+multinomial+diffusion%3A+Learning+categorical+distributions%2E+In+Advances+in+Neural+Information+Processing)
16. Clément Vignac, Igor Krawczuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pascal Frossard. DiGress: Discrete denoising diffusion for graph generation. In International Conference on Learning Representations, 2023. [find](https://scholar.google.com/scholar?q=Cl%C3%A9ment+Vignac%2C+Igor+Krawczuk%2C+Antoine+Siraudin%2C+Bohan+Wang%2C+Volkan+Cevher%2C+and+Pascal+Frossard%2E+DiGress%3A+Discrete+denoising+diffusion+for+graph+generation%2E+In+International+Conference+on+Learning+Rep)
17. Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. Diffsound: Discrete diffusion model for text-to-sound generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023. [find](https://scholar.google.com/scholar?q=Dongchao+Yang%2C+Jianwei+Yu%2C+Helin+Wang%2C+Wen+Wang%2C+Chao+Weng%2C+Yuexian+Zou%2C+and+Dong+Yu%2E+Diffsound%3A+Discrete+diffusion+model+for+text%2Dto%2Dsound+generation%2E+IEEE%2FACM+Transactions+on+Audio%2C+Speech%2C+and+Lang)
18. Nate Gruver, Samuel Stanton, Nathan Frey, Tim GJ Rudner, Isidro Hotzel, Julien Lafrance-Vanasse, Arvind Rajpal, Kyunghyun Cho, and Andrew G Wilson. Protein design with guided discrete diffusion. In Advances in Neural Information Processing Systems, 2023. [find](https://scholar.google.com/scholar?q=Nate+Gruver%2C+Samuel+Stanton%2C+Nathan+Frey%2C+Tim+GJ+Rudner%2C+Isidro+Hotzel%2C+Julien+Lafrance%2DVanasse%2C+Arvind+Rajpal%2C+Kyunghyun+Cho%2C+and+Andrew+G+Wilson%2E+Protein+design+with+guided+discrete+diffusion%2E+In+Ad)
19. Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, et al. Continuous diffusion for categorical data. arXiv preprint arXiv:2211.15089, 2022. — [arXiv:2211.15089](https://arxiv.org/abs/2211.15089) [find](https://scholar.google.com/scholar?q=Sander+Dieleman%2C+Laurent+Sartran%2C+Arman+Roshannai%2C+Nikolay+Savinov%2C+Yaroslav+Ganin%2C+Pierre+H+Richemond%2C+Arnaud+Doucet%2C+Robin+Strudel%2C+Chris+Dyer%2C+Conor+Durkan%2C+et+al%2E+Continuous+diffusion+for+categori)
20. Ting Chen, Ruixiang ZHANG, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. In International Conference on Learning Representations, 2022. [find](https://scholar.google.com/scholar?q=Ting+Chen%2C+Ruixiang+ZHANG%2C+and+Geoffrey+Hinton%2E+Analog+bits%3A+Generating+discrete+data+using+diffusion+models+with+self%2Dconditioning%2E+In+International+Conference+on+Learning+Representations%2C+2022%2E)
21. Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. [Diffusion-LM improves controllable text generation](https://md.research.elamur.ai/free/en/papers/li-2022-diffusion-lm-improves-controllable-text/). In Advances in Neural Information Processing Systems, 2022.
22. Ishaan Gulrajani and Tatsunori B Hashimoto. Likelihood-based diffusion language models. In Advances in Neural Information Processing Systems, 2023. [find](https://scholar.google.com/scholar?q=Ishaan+Gulrajani+and+Tatsunori+B+Hashimoto%2E+Likelihood%2Dbased+diffusion+language+models%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2023%2E)
23. Justin Lovelace, Varsha Kishore, Chao Wan, Eliot Shekhtman, and Kilian Q Weinberger. Latent diffusion for language generation. In Advances in Neural Information Processing Systems, 2024. [find](https://scholar.google.com/scholar?q=Justin+Lovelace%2C+Varsha+Kishore%2C+Chao+Wan%2C+Eliot+Shekhtman%2C+and+Kilian+Q+Weinberger%2E+Latent+diffusion+for+language+generation%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2024%2E)
24. Pierre H Richemond, Sander Dieleman, and Arnaud Doucet. Categorical SDEs with simplex diffusion. arXiv preprint arXiv:2210.14784, 2022. — [arXiv:2210.14784](https://arxiv.org/abs/2210.14784) [find](https://scholar.google.com/scholar?q=Pierre+H+Richemond%2C+Sander+Dieleman%2C+and+Arnaud+Doucet%2E+Categorical+SDEs+with+simplex+diffusion%2E+arXiv+preprint+arXiv%3A2210%2E14784%2C+2022%2E+%E2%80%94+%5BarXiv%3A2210%2E14784%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2210%2E14784%29)
25. Pavel Avdeyev, Chenlai Shi, Yuhao Tan, Kseniia Dudnyk, and Jian Zhou. Dirichlet diffusion score model for biological sequence generation. In International Conference on Machine Learning, 2023. [find](https://scholar.google.com/scholar?q=Pavel+Avdeyev%2C+Chenlai+Shi%2C+Yuhao+Tan%2C+Kseniia+Dudnyk%2C+and+Jian+Zhou%2E+Dirichlet+diffusion+score+model+for+biological+sequence+generation%2E+In+International+Conference+on+Machine+Learning%2C+2023%2E)
26. Alex Graves, Rupesh Kumar Srivastava, Timothy Atkinson, and Faustino Gomez. Bayesian flow networks. arXiv preprint arXiv:2308.07037, 2023. [find](https://scholar.google.com/scholar?q=Alex+Graves%2C+Rupesh+Kumar+Srivastava%2C+Timothy+Atkinson%2C+and+Faustino+Gomez%2E+Bayesian+flow+networks%2E+arXiv+preprint+arXiv%3A2308%2E07037%2C+2023%2E)
27. Kaiwen Xue, Yuhao Zhou, Shen Nie, Xu Min, Xiaolu Zhang, Jun Zhou, and Chongxuan Li. Unifying Bayesian flow networks and diffusion models through stochastic differential equations. arXiv preprint arXiv:2404.15766, 2024. — [arXiv:2404.15766](https://arxiv.org/abs/2404.15766) [find](https://scholar.google.com/scholar?q=Kaiwen+Xue%2C+Yuhao+Zhou%2C+Shen+Nie%2C+Xu+Min%2C+Xiaolu+Zhang%2C+Jun+Zhou%2C+and+Chongxuan+Li%2E+Unifying+Bayesian+flow+networks+and+diffusion+models+through+stochastic+differential+equations%2E+arXiv+preprint+arXiv)
28. Guan-Horng Liu, Tianrong Chen, Evangelos Theodorou, and Molei Tao. Mirror diffusion models for constrained and watermarked generation. In Advances in Neural Information Processing Systems, 2024. [find](https://scholar.google.com/scholar?q=Guan%2DHorng+Liu%2C+Tianrong+Chen%2C+Evangelos+Theodorou%2C+and+Molei+Tao%2E+Mirror+diffusion+models+for+constrained+and+watermarked+generation%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2024%2E)
29. Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. In Advances in Neural Information Processing Systems, 2022. [find](https://scholar.google.com/scholar?q=Andrew+Campbell%2C+Joe+Benton%2C+Valentin+De+Bortoli%2C+Thomas+Rainforth%2C+George+Deligiannidis%2C+and+Arnaud+Doucet%2E+A+continuous+time+framework+for+discrete+denoising+models%2E+In+Advances+in+Neural+Informatio)
30. Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based continuous-time discrete diffusion models. In International Conference on Learning Representations, 2022. [find](https://scholar.google.com/scholar?q=Haoran+Sun%2C+Lijun+Yu%2C+Bo+Dai%2C+Dale+Schuurmans%2C+and+Hanjun+Dai%2E+Score%2Dbased+continuous%2Dtime+discrete+diffusion+models%2E+In+International+Conference+on+Learning+Representations%2C+2022%2E)
31. Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. A reparameterized discrete diffusion model for text generation. arXiv preprint arXiv:2302.05737, 2023. — [arXiv:2302.05737](https://arxiv.org/abs/2302.05737) [find](https://scholar.google.com/scholar?q=Lin+Zheng%2C+Jianbo+Yuan%2C+Lei+Yu%2C+and+Lingpeng+Kong%2E+A+reparameterized+discrete+diffusion+model+for+text+generation%2E+arXiv+preprint+arXiv%3A2302%2E05737%2C+2023%2E+%E2%80%94+%5BarXiv%3A2302%2E05737%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F230)
32. Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion language modeling by estimating the ratios of the data distribution. In International Conference on Machine Learning, 2024. [find](https://scholar.google.com/scholar?q=Aaron+Lou%2C+Chenlin+Meng%2C+and+Stefano+Ermon%2E+Discrete+diffusion+language+modeling+by+estimating+the+ratios+of+the+data+distribution%2E+In+International+Conference+on+Machine+Learning%2C+2024%2E)
33. Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In Advances in Neural Information Processing Systems, 2021. [find](https://scholar.google.com/scholar?q=Diederik+Kingma%2C+Tim+Salimans%2C+Ben+Poole%2C+and+Jonathan+Ho%2E+Variational+diffusion+models%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2021%2E)
34. Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Advances in Neural Information Processing Systems, 2022. [find](https://scholar.google.com/scholar?q=Tero+Karras%2C+Miika+Aittala%2C+Timo+Aila%2C+and+Samuli+Laine%2E+Elucidating+the+design+space+of+diffusion%2Dbased+generative+models%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2022%2E)
35. Joe Benton, Yuyang Shi, Valentin De Bortoli, George Deligiannidis, and Arnaud Doucet. From denoising diffusions to denoising Markov models. arXiv preprint arXiv:2211.03595, 2022. — [arXiv:2211.03595](https://arxiv.org/abs/2211.03595) [find](https://scholar.google.com/scholar?q=Joe+Benton%2C+Yuyang+Shi%2C+Valentin+De+Bortoli%2C+George+Deligiannidis%2C+and+Arnaud+Doucet%2E+From+denoising+diffusions+to+denoising+Markov+models%2E+arXiv+preprint+arXiv%3A2211%2E03595%2C+2022%2E+%E2%80%94+%5BarXiv%3A2211%2E03595%5D%28)
36. Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. arXiv preprint arXiv:2406.03736, 2024. — [arXiv:2406.03736](https://arxiv.org/abs/2406.03736) [find](https://scholar.google.com/scholar?q=Jingyang+Ou%2C+Shen+Nie%2C+Kaiwen+Xue%2C+Fengqi+Zhu%2C+Jiacheng+Sun%2C+Zhenguo+Li%2C+and+Chongxuan+Li%2E+Your+absorbing+discrete+diffusion+secretly+models+the+conditional+distributions+of+clean+data%2E+arXiv+preprint)
37. Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and Volodymyr Kuleshov. [Simple and effective masked diffusion language models](https://md.research.elamur.ai/free/en/papers/sahoo-2024-simple-effective-masked-diffusion-language/). arXiv preprint arXiv:2406.07524, 2024.
38. Lingxiao Zhao, Xueying Ding, Lijun Yu, and Leman Akoglu. Improving and unifying discrete and continuous-time discrete denoising diffusion. arXiv preprint arXiv:2402.03701, 2024. — [arXiv:2402.03701](https://arxiv.org/abs/2402.03701) [find](https://scholar.google.com/scholar?q=Lingxiao+Zhao%2C+Xueying+Ding%2C+Lijun+Yu%2C+and+Leman+Akoglu%2E+Improving+and+unifying+discrete+and+continuous%2Dtime+discrete+denoising+diffusion%2E+arXiv+preprint+arXiv%3A2402%2E03701%2C+2024%2E+%E2%80%94+%5BarXiv%3A2402%2E03701%5D%28h)
39. Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. [find](https://scholar.google.com/scholar?q=Huiwen+Chang%2C+Han+Zhang%2C+Lu+Jiang%2C+Ce+Liu%2C+and+William+T+Freeman%2E+Maskgit%3A+Masked+generative+image+transformer%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+20)
40. Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. Advances in Neural Information Processing Systems, 30, 2017. [find](https://scholar.google.com/scholar?q=Martin+Heusel%2C+Hubert+Ramsauer%2C+Thomas+Unterthiner%2C+Bernhard+Nessler%2C+and+Sepp+Hochreiter%2E+GANs+trained+by+a+two+time%2Dscale+update+rule+converge+to+a+local+Nash+equilibrium%2E+Advances+in+Neural+Informa)
41. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. [The curious case of neural text degeneration](https://md.research.elamur.ai/free/en/papers/holtzman-2019-curious-case-neural-text-degeneration/). In International Conference on Learning Representations, 2019.
42. Jonathan Ho and Tim Salimans. [Classifier-free diffusion guidance](https://md.research.elamur.ai/free/en/papers/ho-2022-classifier-free-diffusion-guidance/). arXiv preprint arXiv:2207.12598, 2022.
43. Hunter Nisonoff, Junhao Xiong, Stephan Allenspach, and Jennifer Listgarten. Unlocking guidance for discrete state-space diffusion and flow models. arXiv preprint arXiv:2406.01572, 2024. — [arXiv:2406.01572](https://arxiv.org/abs/2406.01572) [find](https://scholar.google.com/scholar?q=Hunter+Nisonoff%2C+Junhao+Xiong%2C+Stephan+Allenspach%2C+and+Jennifer+Listgarten%2E+Unlocking+guidance+for+discrete+state%2Dspace+diffusion+and+flow+models%2E+arXiv+preprint+arXiv%3A2406%2E01572%2C+2024%2E+%E2%80%94+%5BarXiv%3A2406%2E)
44. Yixiu Zhao, Jiaxin Shi, Lester Mackey, and Scott Linderman. Informed correctors for discrete diffusion models. arXiv preprint arXiv:2407.21243, 2024. — [arXiv:2407.21243](https://arxiv.org/abs/2407.21243) [find](https://scholar.google.com/scholar?q=Yixiu+Zhao%2C+Jiaxin+Shi%2C+Lester+Mackey%2C+and+Scott+Linderman%2E+Informed+correctors+for+discrete+diffusion+models%2E+arXiv+preprint+arXiv%3A2407%2E21243%2C+2024%2E+%E2%80%94+%5BarXiv%3A2407%2E21243%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2407%2E21)
45. James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL [http://github.com/jax-ml/jax](http://github.com/jax-ml/jax). [↗](http://github.com/jax-ml/jax)
46. Ouail Kitouni, Niklas Nolte, James Hensman, and Bhaskar Mitra. Disk: A diffusion model for structured knowledge. arXiv preprint arXiv:2312.05253, 2023. — [arXiv:2312.05253](https://arxiv.org/abs/2312.05253) [find](https://scholar.google.com/scholar?q=Ouail+Kitouni%2C+Niklas+Nolte%2C+James+Hensman%2C+and+Bhaskar+Mitra%2E+Disk%3A+A+diffusion+model+for+structured+knowledge%2E+arXiv+preprint+arXiv%3A2312%2E05253%2C+2023%2E+%E2%80%94+%5BarXiv%3A2312%2E05253%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2312%2E)
47. Benigno Uria, Iain Murray, and Hugo Larochelle. A deep and tractable density estimator. In International Conference on Machine Learning, pages 467–475. PMLR, 2014. [find](https://scholar.google.com/scholar?q=Benigno+Uria%2C+Iain+Murray%2C+and+Hugo+Larochelle%2E+A+deep+and+tractable+density+estimator%2E+In+International+Conference+on+Machine+Learning%2C+pages+467%E2%80%93475%2E+PMLR%2C+2014%2E)
48. Emiel Hoogeboom, Alexey A Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. In International Conference on Learning Representations, 2021. [find](https://scholar.google.com/scholar?q=Emiel+Hoogeboom%2C+Alexey+A+Gritsenko%2C+Jasmijn+Bastings%2C+Ben+Poole%2C+Rianne+van+den+Berg%2C+and+Tim+Salimans%2E+Autoregressive+diffusion+models%2E+In+International+Conference+on+Learning+Representations%2C+2021%2E)
49. Andrew Campbell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. In International Conference on Machine Learning, 2024. [find](https://scholar.google.com/scholar?q=Andrew+Campbell%2C+Jason+Yim%2C+Regina+Barzilay%2C+Tom+Rainforth%2C+and+Tommi+Jaakkola%2E+Generative+flows+on+discrete+state%2Dspaces%3A+Enabling+multimodal+flows+with+applications+to+protein+co%2Ddesign%2E+In+Internat)
50. Javier E Santos, Zachary R Fox, Nicholas Lubbers, and Yen Ting Lin. Blackout diffusion: generative diffusion models in discrete-state spaces. In International Conference on Machine Learning, pages 9034–9059. PMLR, 2023. [find](https://scholar.google.com/scholar?q=Javier+E+Santos%2C+Zachary+R+Fox%2C+Nicholas+Lubbers%2C+and+Yen+Ting+Lin%2E+Blackout+diffusion%3A+generative+diffusion+models+in+discrete%2Dstate+spaces%2E+In+International+Conference+on+Machine+Learning%2C+pages+903)
51. Nikolay Savinov, Junyoung Chung, Mikolaj Binkowski, Erich Elsen, and Aaron van den Oord. Step-unrolled denoising autoencoders for text generation. In International Conference on Learning Representations, 2022. [find](https://scholar.google.com/scholar?q=Nikolay+Savinov%2C+Junyoung+Chung%2C+Mikolaj+Binkowski%2C+Erich+Elsen%2C+and+Aaron+van+den+Oord%2E+Step%2Dunrolled+denoising+autoencoders+for+text+generation%2E+In+International+Conference+on+Learning+Representatio)
52. Jiaxin Shi, Yuhao Zhou, Jessica Hwang, Michalis Titsias, and Lester Mackey. Gradient estimation with discrete Stein operators. In Advances in Neural Information Processing Systems, 2022. [find](https://scholar.google.com/scholar?q=Jiaxin+Shi%2C+Yuhao+Zhou%2C+Jessica+Hwang%2C+Michalis+Titsias%2C+and+Lester+Mackey%2E+Gradient+estimation+with+discrete+Stein+operators%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2022%2E)
53. Tim Salimans and David A Knowles. On using control variates with stochastic approximation for variational bayes and its connection to stochastic linear regression. arXiv preprint arXiv:1401.1022, 2014. — [arXiv:1401.1022](https://arxiv.org/abs/1401.1022) [find](https://scholar.google.com/scholar?q=Tim+Salimans+and+David+A+Knowles%2E+On+using+control+variates+with+stochastic+approximation+for+variational+bayes+and+its+connection+to+stochastic+linear+regression%2E+arXiv+preprint+arXiv%3A1401%2E1022%2C+2014)
54. W. Kool, H. V. Hoof, and M. Welling. Buy 4 REINFORCE samples, get a baseline for free! DeepRLStructPred@ICLR, 2019. In [find](https://scholar.google.com/scholar?q=W%2E+Kool%2C+H%2E+V%2E+Hoof%2C+and+M%2E+Welling%2E+Buy+4+REINFORCE+samples%2C+get+a+baseline+for+free%21+DeepRLStructPred%40ICLR%2C+2019%2E+In)
55. Matt Mahoney. Text8. [https://mattmahoney.net/dc/textdata.html](https://mattmahoney.net/dc/textdata.html). Accessed: 2024-05-14. [↗](https://mattmahoney.net/dc/textdata.html)
56. Aaron Gokaslan and Vanya Cohen. OpenWebTextCorpus, 2019. Openwebtext corpus. [http://Skylion007.github.io/](http://Skylion007.github.io/) [↗](http://Skylion007.github.io/)
57. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2C+2019%2E)
58. Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. [The fineweb datasets: Decanting the web for the finest text data at scale](https://md.research.elamur.ai/free/en/papers/penedo-2024-fineweb-datasets-decanting-web-finest/). arXiv preprint arXiv:2406.17557, 2024.
59. Dustin Tran, Keyon Vafa, Kumar Agrawal, Laurent Dinh, and Ben Poole. Discrete flows: Invertible generative models of discrete data. In Advances in Neural Information Processing Systems, 2019. [find](https://scholar.google.com/scholar?q=Dustin+Tran%2C+Keyon+Vafa%2C+Kumar+Agrawal%2C+Laurent+Dinh%2C+and+Ben+Poole%2E+Discrete+flows%3A+Invertible+generative+models+of+discrete+data%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2019%2E)
60. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+arXiv+preprint+arXiv%3A1905%2E07830%2C+2019%2E)
61. Andy Shih, Dorsa Sadigh, and Stefano Ermon. Training and inference on any-order autoregressive models the right way. In Advances in Neural Information Processing Systems, 2022. [find](https://scholar.google.com/scholar?q=Andy+Shih%2C+Dorsa+Sadigh%2C+and+Stefano+Ermon%2E+Training+and+inference+on+any%2Dorder+autoregressive+models+the+right+way%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2022%2E)
62. Zachary Ziegler and Alexander Rush. Latent normalizing flows for discrete sequences. In International Conference on Machine Learning, 2019. [find](https://scholar.google.com/scholar?q=Zachary+Ziegler+and+Alexander+Rush%2E+Latent+normalizing+flows+for+discrete+sequences%2E+In+International+Conference+on+Machine+Learning%2C+2019%2E)
63. Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International Conference on Machine Learning, 2016. [find](https://scholar.google.com/scholar?q=A%C3%A4ron+Van+Den+Oord%2C+Nal+Kalchbrenner%2C+and+Koray+Kavukcuoglu%2E+Pixel+recurrent+neural+networks%2E+In+International+Conference+on+Machine+Learning%2C+2016%2E)
64. Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, and Alex Graves. Conditional image generation with pixelcnn decoders. In Advances in Neural Information Processing systems, 2016. [find](https://scholar.google.com/scholar?q=Aaron+Van+den+Oord%2C+Nal+Kalchbrenner%2C+Lasse+Espeholt%2C+Oriol+Vinyals%2C+and+Alex+Graves%2E+Conditional+image+generation+with+pixelcnn+decoders%2E+In+Advances+in+Neural+Information+Processing+systems%2C+2016%2E)
65. Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. In International Conference on Learning Representations, 2016. [find](https://scholar.google.com/scholar?q=Tim+Salimans%2C+Andrej+Karpathy%2C+Xi+Chen%2C+and+Diederik+P+Kingma%2E+Pixelcnn%2B%2B%3A+Improving+the+pixelcnn+with+discretized+logistic+mixture+likelihood+and+other+modifications%2E+In+International+Conference+on+L)
66. Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. Pixelsnail: An improved autoregressive generative model. In International Conference on Machine Learning, 2018. [find](https://scholar.google.com/scholar?q=Xi+Chen%2C+Nikhil+Mishra%2C+Mostafa+Rohaninejad%2C+and+Pieter+Abbeel%2E+Pixelsnail%3A+An+improved+autoregressive+generative+model%2E+In+International+Conference+on+Machine+Learning%2C+2018%2E)
67. Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning, 2018. [find](https://scholar.google.com/scholar?q=Niki+Parmar%2C+Ashish+Vaswani%2C+Jakob+Uszkoreit%2C+Lukasz+Kaiser%2C+Noam+Shazeer%2C+Alexander+Ku%2C+and+Dustin+Tran%2E+Image+transformer%2E+In+International+Conference+on+Machine+Learning%2C+2018%2E)
68. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. [Generating long sequences with sparse transformers](https://md.research.elamur.ai/free/en/papers/child-2019-generating-long-sequences-sparse-transformers/). arXiv preprint arXiv:1904.10509, 2019.
69. Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. [Efficient content-based sparse attention with routing transformers](https://md.research.elamur.ai/free/en/papers/roy-2020-efficient-content-based-sparse-attention/). Transactions of the Association for Computational Linguistics, 9: 53–68, 2021.
70. Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in Neural Information Processing Systems, 30, 2017. [find](https://scholar.google.com/scholar?q=Aaron+Van+Den+Oord%2C+Oriol+Vinyals%2C+et+al%2E+Neural+discrete+representation+learning%2E+Advances+in+Neural+Information+Processing+Systems%2C+30%2C+2017%2E)
71. Kehang Han, Kathleen Kenealy, Aditya Barua, Noah Fiedel, and Noah Constant. Transfer learning for text diffusion models. arXiv preprint arXiv:2401.17181, 2024. — [arXiv:2401.17181](https://arxiv.org/abs/2401.17181) [find](https://scholar.google.com/scholar?q=Kehang+Han%2C+Kathleen+Kenealy%2C+Aditya+Barua%2C+Noah+Fiedel%2C+and+Noah+Constant%2E+Transfer+learning+for+text+diffusion+models%2E+arXiv+preprint+arXiv%3A2401%2E17181%2C+2024%2E+%E2%80%94+%5BarXiv%3A2401%2E17181%5D%28https%3A%2F%2Farxiv%2Eorg%2Fa)
72. Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems, 2015. [find](https://scholar.google.com/scholar?q=Samy+Bengio%2C+Oriol+Vinyals%2C+Navdeep+Jaitly%2C+and+Noam+Shazeer%2E+Scheduled+sampling+for+sequence+prediction+with+recurrent+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2015%2E)
73. Peter W. Glynn. Likelihood ratio gradient estimation for stochastic systems. Communications of the ACM, 33(10):75–84, 1990. [find](https://scholar.google.com/scholar?q=Peter+W%2E+Glynn%2E+Likelihood+ratio+gradient+estimation+for+stochastic+systems%2E+Communications+of+the+ACM%2C+33%2810%29%3A75%E2%80%9384%2C+1990%2E)
74. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3-4):229–256, 1992. [find](https://scholar.google.com/scholar?q=Ronald+J+Williams%2E+Simple+statistical+gradient%2Dfollowing+algorithms+for+connectionist+reinforcement+learning%2E+Machine+Learning%2C+8%283%2D4%29%3A229%E2%80%93256%2C+1992%2E)
75. William Peebles and Saining Xie. [Scalable diffusion models with transformers](https://md.research.elamur.ai/free/en/papers/peebles-2022-scalable-diffusion-models-transformers/). In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023. *Table 4: Masking schedule formulas.* α t Linear 1 − t 1 − t Polynomial

*Updated 2 November 2026*