---
title: "Discrete Diffusion Modeling by Estimating the Ratios of the Data"
description: "Autoregressive models have long dominated generative modeling of discrete data such as text. 49 references, cited by 91 papers here."
authors: ["Aaron Lou", "Chenlin Meng", "Stefano Ermon"]
year: 2023
venue: "arXiv:2310.16834"
source: "https://arxiv.org/abs/2310.16834"
doi: "10.48550/arxiv.2310.16834"
arxiv: "2310.16834"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lou-2023-discrete-diffusion-modeling-estimating-ratios/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lou-2023-discrete-diffusion-modeling-estimating-ratios/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/) / Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution

# Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution

**Authors:** Aaron Lou, [Chenlin Meng](https://md.research.elamur.ai/free/en/authors/chenlin-meng/) and [Stefano Ermon](https://md.research.elamur.ai/free/en/authors/stefano-ermon/) · **Year:** 2023 · **Venue:** arXiv:2310.16834 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2310.16834) · [PDF](https://arxiv.org/pdf/2310.16834) · [DOI 10.48550/arxiv.2310.16834](https://doi.org/10.48550/arxiv.2310.16834) · [arXiv 2310.16834](https://arxiv.org/abs/2310.16834) · [OpenAlex](https://openalex.org/W4387964173)

**Citations:** 5 citations (OpenAlex, 2026-10-14) · [91](https://md.research.elamur.ai/free/en/papers/lou-2023-discrete-diffusion-modeling-estimating-ratios/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

Autoregressive models have long dominated generative modeling of discrete data such as text. Diffusion models excel on images but have lagged on language, partly because score matching, the theory behind continuous diffusion, has no well-performing discrete counterpart. Lou, Meng and Ermon (Stanford; ICML 2024) propose **score entropy**, a loss built on a Bregman divergence. It trains a network to estimate ratios between probabilities of neighbouring states (the "concrete score") and keeps those estimates positive through a log-barrier effect. A denoising variant makes training scalable, and a diffusion-weighted version yields an ELBO for likelihood training and evaluation. The resulting **Score Entropy Discrete Diffusion (SEDD)** models outperform earlier language diffusion approaches (D3PM-style mean prediction, ratio matching, concrete score matching, and continuous embedding diffusion). At similar sizes they are competitive with GPT-2. SEDD also produces good samples without temperature or nucleus tricks, lets users trade sampling steps for quality, and supports prompting at arbitrary positions. This makes non-autoregressive text generation a credible alternative to the left-to-right paradigm.

## Key points

- Compared with existing language diffusion models of comparable size, SEDD lowers perplexity by roughly 25–75% and beats GPT-2 on its zero-shot perplexity tasks.
- Without annealing, SEDD samples reach about 6–8× better generative perplexity than un-annealed GPT-2. Similar quality is obtained with 32× fewer network evaluations.
- For both standard prompting and infilling, SEDD matches the MAUVE scores of autoregressive models using nucleus sampling, with no specialized training for infilling.
- With about 100 sampling steps, inference time matches autoregressive decoding (Hugging Face transformers baseline). Because no KV-cache is needed, batch sizes can be roughly 4–6× larger.
- Code is released at github.com/louaaron/Score-Entropy-Discrete-Diffusion.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

## Cite

Cite as: Aaron Lou, Chenlin Meng and Stefano Ermon. 2023. Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution. arXiv:2310.16834.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{lou2023discrete,
  title = {{Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution}},
  author = {Aaron Lou, Chenlin Meng and Stefano Ermon},
  year = {2023},
  doi = {10.48550/arxiv.2310.16834},
  eprint = {2310.16834},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.16834}
}
```

- [← previous Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptation](https://md.research.elamur.ai/free/en/papers/li-2023-bactrian-x-multilingual-replicable-instruction/)
- [next → Faithfulness Tests for Natural Language Explanations](https://md.research.elamur.ai/free/en/papers/atanasova-2023-faithfulness-tests-natural-language-explanations/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/)

## References

49 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anderson, W. J. Continuous-time Markov chains: An applications-oriented approach. Springer Science & Business Media, 2012. [find](https://scholar.google.com/scholar?q=Anderson%2C+W%2E+J%2E+Continuous%2Dtime+Markov+chains%3A+An+applications%2Doriented+approach%2E+Springer+Science+%26+Business+Media%2C+2012%2E)
2. Efron, B. Tweedie’s formula and selection bias. Journal of the American Statistical Association, 106:1602 – 1614, 2011. URL [https://api.semanticscholar.org/CorpusID:23284154](https://api.semanticscholar.org/CorpusID:23284154). [↗](https://api.semanticscholar.org/CorpusID:23284154)
3. Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and van den Berg, R. [Structured denoising diffusion models in discrete state-spaces](https://md.research.elamur.ai/free/en/papers/austin-2021-structured-denoising-diffusion-models-discrete/). In Advances in Neural Information Processing Systems, 2021.
4. Gillespie, D. T. Approximate accelerated stochastic simulation of chemically reacting systems. Journal of Chemical Physics, 115:1716–1733, 2001. URL [https://api.semanticscholar.org/CorpusID:5109777](https://api.semanticscholar.org/CorpusID:5109777). [↗](https://api.semanticscholar.org/CorpusID:5109777)
5. Benton, J., Shi, Y., Bortoli, V. D., Deligiannidis, G., and Doucet, A. From denoising diffusions to denoising markov models. ArXiv, abs/2211.03595, 2022. URL [https://api.semanticscholar.org/CorpusID:253384277](https://api.semanticscholar.org/CorpusID:253384277). — [arXiv:2211.03595](https://arxiv.org/abs/2211.03595) [↗](https://api.semanticscholar.org/CorpusID:253384277)
6. Gokaslan, A. and Cohen, V. Openwebtext corpus. [http://Skylion007.github.io/](http://Skylion007.github.io/) OpenWebTextCorpus, 2019. [↗](http://Skylion007.github.io/)
7. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Gong, S., Li, M., Feng, J., Wu, Z., and Kong, L. [Diffuseq: Sequence to sequence text generation with diffusion models](https://md.research.elamur.ai/free/en/papers/gong-2022-diffuseq-sequence-sequence-text-generation/). In The Eleventh International Conference on Learning Representations, 2023.
8. Hyvärinen, A. Some extensions of score matching. Comput. Stat. Data Anal., 51:2499–2512, 2007. URL [https://api.semanticscholar.org/CorpusID:2352990](https://api.semanticscholar.org/CorpusID:2352990). [↗](https://api.semanticscholar.org/CorpusID:2352990)
9. Graves, A., Srivastava, R. K., Atkinson, T., and Gomez, F. Bayesian flow networks. arXiv preprint arXiv:2308.07037, 2023. [find](https://scholar.google.com/scholar?q=Graves%2C+A%2E%2C+Srivastava%2C+R%2E+K%2E%2C+Atkinson%2C+T%2E%2C+and+Gomez%2C+F%2E+Bayesian+flow+networks%2E+arXiv+preprint+arXiv%3A2308%2E07037%2C+2023%2E)
10. Kelly, F. Reversibility and stochastic networks. 1980. URL [https://api.semanticscholar.org/CorpusID:125211322](https://api.semanticscholar.org/CorpusID:125211322). [↗](https://api.semanticscholar.org/CorpusID:125211322)
11. Gulrajani, I. and Hashimoto, T. Likelihood-based diffusion language models. In Advances in Neural Information Processing Systems, 2023. [find](https://scholar.google.com/scholar?q=Gulrajani%2C+I%2E+and+Hashimoto%2C+T%2E+Likelihood%2Dbased+diffusion+language+models%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2023%2E)
12. Li, X., Thickstun, J., Gulrajani, I., Liang, P. S., and Hashimoto, T. B. [Diffusion-lm improves controllable text generation](https://md.research.elamur.ai/free/en/papers/li-2022-diffusion-lm-improves-controllable-text/). In Advances in Neural Information Processing Systems, 2022.
13. Han, X., Kumar, S., and Tsvetkov, Y. Ssd-lm: Semi-autoregressive simplex-based diffusion language model for text generation and modular control. arXiv preprint arXiv:2210.17432, 2022. [find](https://scholar.google.com/scholar?q=Han%2C+X%2E%2C+Kumar%2C+S%2E%2C+and+Tsvetkov%2C+Y%2E+Ssd%2Dlm%3A+Semi%2Dautoregressive+simplex%2Dbased+diffusion+language+model+for+text+generation+and+modular+control%2E+arXiv+preprint+arXiv%3A2210%2E17432%2C+2022%2E)
14. Lou, A. and Ermon, S. Reflected diffusion models. In International Conference on Machine Learning. PMLR, 2023. [find](https://scholar.google.com/scholar?q=Lou%2C+A%2E+and+Ermon%2C+S%2E+Reflected+diffusion+models%2E+In+International+Conference+on+Machine+Learning%2E+PMLR%2C+2023%2E)
15. Hanson, F. B. Applied Stochastic Processes and Control for Jump-Diffusions: Modeling, Analysis and Computation. Society for Industrial and Applied Mathematics, Philadelphia, PA, 2007. doi: 10.1137/1.9780898718638. URL [https://epubs.siam.org/doi/abs/10](https://epubs.siam.org/doi/abs/10). 1137/1.9780898718638. — [doi:10.1137/1.9780898718638](https://doi.org/10.1137/1.9780898718638) [↗](https://epubs.siam.org/doi/abs/10)
16. Mahabadi, R. K., Tae, J., Ivison, H., Henderson, J., Beltagy, I., Peters, M. E., and Cohan, A. Tess: Text-to-text self-conditioned simplex diffusion. arXiv preprint arXiv:2305.08379, 2023. [find](https://scholar.google.com/scholar?q=Mahabadi%2C+R%2E+K%2E%2C+Tae%2C+J%2E%2C+Ivison%2C+H%2E%2C+Henderson%2C+J%2E%2C+Beltagy%2C+I%2E%2C+Peters%2C+M%2E+E%2E%2C+and+Cohan%2C+A%2E+Tess%3A+Text%2Dto%2Dtext+self%2Dconditioned+simplex+diffusion%2E+arXiv+preprint+arXiv%3A2305%2E08379%2C+2023%2E)
17. He, Z., Sun, T., Wang, K., Huang, X., and Qiu, X. [Diffusionbert: Improving generative masked language models with diffusion models](https://md.research.elamur.ai/free/en/papers/he-2023-diffusionbert-improving-generative-masked-language/). In Annual Meeting of the Association for Computational Linguistics, 2022.
18. Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.-Y., and Ermon, S. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2021. URL [https://api.semanticscholar.org/CorpusID:245704504](https://api.semanticscholar.org/CorpusID:245704504). [↗](https://api.semanticscholar.org/CorpusID:245704504)
19. Ho, J. [Classifier-free diffusion guidance](https://md.research.elamur.ai/free/en/papers/ho-2022-classifier-free-diffusion-guidance/). ArXiv, abs/2207.12598, 2022. URL [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 249145348. [↗](https://api.semanticscholar.org/CorpusID:)
20. Meng, C., Choi, K., Song, J., and Ermon, S. Concrete score matching: Generalized score matching for discrete data. In Advances in Neural Information Processing Systems, 2022. [find](https://scholar.google.com/scholar?q=Meng%2C+C%2E%2C+Choi%2C+K%2E%2C+Song%2C+J%2E%2C+and+Ermon%2C+S%2E+Concrete+score+matching%3A+Generalized+score+matching+for+discrete+data%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2022%2E)
21. Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020. [find](https://scholar.google.com/scholar?q=Ho%2C+J%2E%2C+Jain%2C+A%2E%2C+and+Abbeel%2C+P%2E+Denoising+diffusion+probabilistic+models%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2020%2E)
22. Øksendal, B. Stochastic differential equations : an introduction with applications. Journal of the American Statistical Association, 82:948, 1987. [find](https://scholar.google.com/scholar?q=%C3%98ksendal%2C+B%2E+Stochastic+differential+equations+%3A+an+introduction+with+applications%2E+Journal+of+the+American+Statistical+Association%2C+82%3A948%2C+1987%2E)
23. Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. [The curious case of neural text degeneration](https://md.research.elamur.ai/free/en/papers/holtzman-2019-curious-case-neural-text-degeneration/). In International Conference on Learning Representations, 2019.
24. Peebles, W. S. and Xie, S. [Scalable diffusion models with transformers](https://md.research.elamur.ai/free/en/papers/peebles-2022-scalable-diffusion-models-transformers/). In International Conference on Computer Vision, 2023.
25. Hoogeboom, E., Nielsen, D., Jaini, P., Forré, P., and Welling, M. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems, 34:12454–12465, 2021. [find](https://scholar.google.com/scholar?q=Hoogeboom%2C+E%2E%2C+Nielsen%2C+D%2E%2C+Jaini%2C+P%2E%2C+Forre%CC%81%2C+P%2E%2C+and+Welling%2C+M%2E+Argmax+flows+and+multinomial+diffusion%3A+Learning+categorical+distributions%2E+Advances+in+Neural+Information+Processing+Systems%2C+34%3A124)
26. Pillutla, K., Swayamdipta, S., Zellers, R., Thickstun, J., Welleck, S., Choi, Y., and Harchaoui, Z. [Mauve: Measuring the gap between neural text and human text using divergence frontiers](https://md.research.elamur.ai/free/en/papers/pillutla-2021-mauve-measuring-gap-between-neural/). Advances in Neural Information Processing Systems, 34:4816–4828, 2021.
27. Hutchinson, M. F. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Statistics - Simulation and Computation, 18:1059–1076, 1989. URL [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 120969358. [↗](https://api.semanticscholar.org/CorpusID:)
28. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. 2019. URL [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 160025533. [↗](https://api.semanticscholar.org/CorpusID:)
29. Hyvärinen, A. Estimation of non-normalized statistical models by score matching. J. Mach. Learn. Res., 6:695– 709, 2005. [find](https://scholar.google.com/scholar?q=Hyva%CC%88rinen%2C+A%2E+Estimation+of+non%2Dnormalized+statistical+models+by+score+matching%2E+J%2E+Mach%2E+Learn%2E+Res%2E%2C+6%3A695%E2%80%93+709%2C+2005%2E)
30. Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. [Hierarchical text-conditional image generation with clip latents](https://md.research.elamur.ai/free/en/papers/ramesh-2022-hierarchical-text-conditional-image-generation/). ArXiv, abs/2204.06125, 2022. URL [https://api.semanticscholar.org/CorpusID:248097655](https://api.semanticscholar.org/CorpusID:248097655). [↗](https://api.semanticscholar.org/CorpusID:248097655)
31. Shih, A., Sadigh, D., and Ermon, S. Training and inference on any-order autoregressive models the right way. Advances in Neural Information Processing Systems, 35: 2762–2775, 2022. [find](https://scholar.google.com/scholar?q=Shih%2C+A%2E%2C+Sadigh%2C+D%2E%2C+and+Ermon%2C+S%2E+Training+and+inference+on+any%2Dorder+autoregressive+models+the+right+way%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A+2762%E2%80%932775%2C+2022%2E)
32. Vaswani, A., Shazeer, N. M., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In NIPS, 2017.
33. Vincent, P. A connection between score matching and denoising autoencoders. Neural Computation, 23:1661– 1674, 2011. [find](https://scholar.google.com/scholar?q=Vincent%2C+P%2E+A+connection+between+score+matching+and+denoising+autoencoders%2E+Neural+Computation%2C+23%3A1661%E2%80%93+1674%2C+2011%2E)
34. Wang, A. and Cho, K. [Bert has a mouth, and it must speak: Bert as a markov random field language model](https://md.research.elamur.ai/free/en/papers/wang-2019-bert-has-mouth-must-speak/). arXiv preprint arXiv:1902.04094, 2019.
35. Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. [Deep unsupervised learning using nonequilibrium thermodynamics](https://md.research.elamur.ai/free/en/papers/sohl-dickstein-2015-deep-unsupervised-learning-using-nonequilibrium/). In Proceedings of the 32nd International Conference on Machine Learning, 2015.
36. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. Transformers: State-of-the-art natural lan… — [Transformers: State-of-the-Art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
37. Song, J., Meng, C., and Ermon, S. [Denoising diffusion implicit models](https://md.research.elamur.ai/free/en/papers/song-2020-denoising-diffusion-implicit-models/). In International Conference on Learning Representations, 2021a. URL [https://openreview.net/forum?id=St1giarCHLP](https://openreview.net/forum?id=St1giarCHLP). [↗](https://openreview.net/forum?id=St1giarCHLP)
38. Song, Y. and Ermon, S. [Generative modeling by estimating gradients of the data distribution](https://md.research.elamur.ai/free/en/papers/song-2019-generative-modeling-estimating-gradients-data/). In Neural Information Processing Systems, 2019.
39. Song, Y., Garg, S., Shi, J., and Ermon, S. Sliced score matching: A scalable approach to density and score estimation. In Conference on Uncertainty in Artificial Intelligence, 2019. [find](https://scholar.google.com/scholar?q=Song%2C+Y%2E%2C+Garg%2C+S%2E%2C+Shi%2C+J%2E%2C+and+Ermon%2C+S%2E+Sliced+score+matching%3A+A+scalable+approach+to+density+and+score+estimation%2E+In+Conference+on+Uncertainty+in+Artificial+Intelligence%2C+2019%2E)
40. Ye, J., Zheng, Z., Bao, Y., Qian, L., and Wang, M. Dinoiser: Diffused conditional sequence learning by manipulating noises. arXiv preprint arXiv:2302.10025, 2023. — [arXiv:2302.10025](https://arxiv.org/abs/2302.10025) [find](https://scholar.google.com/scholar?q=Ye%2C+J%2E%2C+Zheng%2C+Z%2E%2C+Bao%2C+Y%2E%2C+Qian%2C+L%2E%2C+and+Wang%2C+M%2E+Dinoiser%3A+Diffused+conditional+sequence+learning+by+manipulating+noises%2E+arXiv+preprint+arXiv%3A2302%2E10025%2C+2023%2E+%E2%80%94+%5BarXiv%3A2302%2E10025%5D%28https%3A%2F%2Farxiv%2Eor)
41. Song, Y., Durkan, C., Murray, I., and Ermon, S. Maximum likelihood training of score-based diffusion models. In Neural Information Processing Systems, 2021b. [find](https://scholar.google.com/scholar?q=Song%2C+Y%2E%2C+Durkan%2C+C%2E%2C+Murray%2C+I%2E%2C+and+Ermon%2C+S%2E+Maximum+likelihood+training+of+score%2Dbased+diffusion+models%2E+In+Neural+Information+Processing+Systems%2C+2021b%2E)
42. Yule, G. U. On a method of investigating periodicities in disturbed series with special reference to wolfer’s sunspot numbers. Statistical Papers of George Udny Yule, pp. 389–420, 1971. [find](https://scholar.google.com/scholar?q=Yule%2C+G%2E+U%2E+On+a+method+of+investigating+periodicities+in+disturbed+series+with+special+reference+to+wolfer%E2%80%99s+sunspot+numbers%2E+Statistical+Papers+of+George+Udny+Yule%2C+pp%2E+389%E2%80%93420%2C+1971%2E)
43. Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. [Score-based generative modeling through stochastic differential equations](https://md.research.elamur.ai/free/en/papers/song-2020-score-based-generative-modeling-through/). In International Conference on Learning Representations, 2021c. URL [https://openreview.net/forum?](https://openreview.net/forum?) id=PxTIG12RRHS. [↗](https://openreview.net/forum?)
44. Zheng, L., Yuan, J., Yu, L., and Kong, L. A reparameterized discrete diffusion model for text generation. ArXiv, abs/2302.05737, 2023. — [arXiv:2302.05737](https://arxiv.org/abs/2302.05737) [find](https://scholar.google.com/scholar?q=Zheng%2C+L%2E%2C+Yuan%2C+J%2E%2C+Yu%2C+L%2E%2C+and+Kong%2C+L%2E+A+reparameterized+discrete+diffusion+model+for+text+generation%2E+ArXiv%2C+abs%2F2302%2E05737%2C+2023%2E+%E2%80%94+%5BarXiv%3A2302%2E05737%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2302%2E05737%29)
45. Ziegler, Z. and Rush, A. Latent normalizing flows for discrete sequences. In International Conference on Machine Learning, pp. 7673–7682. PMLR, 2019. [find](https://scholar.google.com/scholar?q=Ziegler%2C+Z%2E+and+Rush%2C+A%2E+Latent+normalizing+flows+for+discrete+sequences%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+7673%E2%80%937682%2E+PMLR%2C+2019%2E)
46. Strudel, R., Tallec, C., Altché, F., Du, Y., Ganin, Y., Mensch, A., Grathwohl, W. S., Savinov, N., Dieleman, S., Sifre, L., et al. Self-conditioned embedding diffusion for text generation. 2022. [find](https://scholar.google.com/scholar?q=Strudel%2C+R%2E%2C+Tallec%2C+C%2E%2C+Altche%CC%81%2C+F%2E%2C+Du%2C+Y%2E%2C+Ganin%2C+Y%2E%2C+Mensch%2C+A%2E%2C+Grathwohl%2C+W%2E+S%2E%2C+Savinov%2C+N%2E%2C+Dieleman%2C+S%2E%2C+Sifre%2C+L%2E%2C+et+al%2E+Self%2Dconditioned+embedding+diffusion+for+text+generation%2E+2022%2E)
47. Su, J., Lu, Y., Pan, S., Wen, B., and Liu, Y. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). ArXiv, abs/2104.09864, 2021.
48. Sun, H., Yu, L., Dai, B., Schuurmans, D., and Dai, H. Score-based continuous-time discrete diffusion models. In The Eleventh International Conference on Learning Representations, 2023. [find](https://scholar.google.com/scholar?q=Sun%2C+H%2E%2C+Yu%2C+L%2E%2C+Dai%2C+B%2E%2C+Schuurmans%2C+D%2E%2C+and+Dai%2C+H%2E+Score%2Dbased+continuous%2Dtime+discrete+diffusion+models%2E+In+The+Eleventh+International+Conference+on+Learning+Representations%2C+2023%2E)
49. Tran, D., Vafa, K., Agrawal, K., Dinh, L., and Poole, B. Discrete flows: Invertible generative models of discrete data. Advances in Neural Information Processing Systems, 32, 2019. [find](https://scholar.google.com/scholar?q=Tran%2C+D%2E%2C+Vafa%2C+K%2E%2C+Agrawal%2C+K%2E%2C+Dinh%2C+L%2E%2C+and+Poole%2C+B%2E+Discrete+flows%3A+Invertible+generative+models+of+discrete+data%2E+Advances+in+Neural+Information+Processing+Systems%2C+32%2C+2019%2E)

*Updated 2 November 2026*