---
title: "Beyond Autoregression: Fast LLMs via Self-Distillation Through"
description: "The authors propose discrete diffusion language models as a faster alternative. 87 references, cited by 10 papers here."
authors: ["Justin Deschenaux", "Caglar Gulcehre"]
year: 2024
venue: "arXiv:2410.21035"
source: "https://arxiv.org/abs/2410.21035"
doi: "10.48550/arxiv.2410.21035"
arxiv: "2410.21035"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/deschenaux-2024-autoregression-fast-llms-self-distillation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/deschenaux-2024-autoregression-fast-llms-self-distillation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/14/) / Beyond Autoregression: Fast LLMs via Self-Distillation Through Time

# Beyond Autoregression: Fast LLMs via Self-Distillation Through Time

**Authors:** Justin Deschenaux and [Caglar Gulcehre](https://md.research.elamur.ai/free/en/authors/caglar-gulcehre/) · **Year:** 2024 · **Venue:** arXiv:2410.21035 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2410.21035) · [PDF](https://arxiv.org/pdf/2410.21035) · [DOI 10.48550/arxiv.2410.21035](https://doi.org/10.48550/arxiv.2410.21035) · [arXiv 2410.21035](https://arxiv.org/abs/2410.21035) · [OpenAlex](https://openalex.org/W4404317990)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [10](https://md.research.elamur.ai/free/en/papers/deschenaux-2024-autoregression-fast-llms-self-distillation/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Autoregressive LLMs emit one token per forward pass, which makes inference slow and limits techniques that rely on sampling many completions, such as search and reranking. The authors propose discrete diffusion language models as a faster alternative. These models start from fully masked sequences and denoise many positions in parallel. Normally they need very many denoising steps. Existing step-reduction tricks for continuous diffusion depend on deterministic noise-to-data maps like DDIM, and such maps cannot exist when every sample begins from the same all-mask input. To get around this, the paper introduces Self-Distillation Through Time (SDTT). A student initialized from a masked diffusion (MDLM) teacher learns to reproduce, in a single step, the token log-probabilities the teacher reaches over several steps. Repeating this over multiple rounds shrinks the step count by 32–64×. The distilled students produce at least 32 tokens per step, beat GPT-2 with nucleus sampling on generative perplexity and keep the teacher's language understanding. This makes the approach attractive for inference-heavy reasoning pipelines.

## Key points
- SDTT is applied over successive rounds (7 in the main experiments). Each round halves the work by matching the teacher's multi-step predictions with a divergence loss. The paper compares KLD, TVD and MSE, and KLD retains performance best.
- On LAMBADA, a student distilled with KLD exceeds its teacher in last-word accuracy. Even when sampling with 16 rather than 1024 steps, it stays on par with AR baselines in perplexity.
- At the 1.3B-parameter scale with 16 decoding steps, the diffusion model can be as much as 8× quicker than autoregressive models equipped with KV caches. It achieves this without any activation caching, and the speedups are larger for smaller models.
- Distillation is shown for models up to 860M parameters, which the authors describe as the largest publicly released discrete diffusion LM.
- Evaluation covers LAMBADA plus 6 multiple-choice benchmarks from the Gao et al. (2021) harness, along with generative perplexity measured against GPT-2 (nucleus sampling, p=0.95).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [École Polytechnique Fédérale de Lausanne](https://md.research.elamur.ai/free/en/universities/ecole-polytechnique-federale-de-lausanne/) — University, Lausanne, Switzerland · 56 works · 8,189 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,621 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 4,145 works

## Cite

Cite as: Justin Deschenaux and Caglar Gulcehre. 2024. Beyond Autoregression: Fast LLMs via Self-Distillation Through Time. arXiv:2410.21035.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{deschenaux2024beyond,
  title = {{Beyond Autoregression: Fast LLMs via Self-Distillation Through Time}},
  author = {Justin Deschenaux and Caglar Gulcehre},
  year = {2024},
  doi = {10.48550/arxiv.2410.21035},
  eprint = {2410.21035},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.21035}
}
```

- [← previous Arabic Synonym BERT-based Adversarial Examples for Text Classification](https://md.research.elamur.ai/free/en/papers/alshahrani-2024-arabic-synonym-bert-based-adversarial/)
- [next → Bridging Context Gaps: Leveraging Coreference Resolution for Long Contextual Understanding](https://md.research.elamur.ai/free/en/papers/liu-2024-bridging-context-gaps-leveraging-coreference/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/14/)

## References

87 references, 44 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. [On-policy distillation of language models: Learning from self-generated mistakes](https://md.research.elamur.ai/free/en/papers/agarwal-2023-policy-distillation-language-models-learning/), 2024. URL https://arxiv.org/abs/2306.13649. [↗](https://arxiv.org/abs/2306.13649)
2. Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan, 2017. URL https://arxiv.org/abs/1701.07875. [↗](https://arxiv.org/abs/1701.07875)
3. Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. [Structured denoising diffusion models in discrete state-spaces](https://md.research.elamur.ai/free/en/papers/austin-2021-structured-denoising-diffusion-models-discrete/), 2023. URL https://arxiv.org/abs/ 2107.03006. [↗](https://arxiv.org/abs/)
4. Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. [Large language monkeys: Scaling inference compute with repeated sampling](https://md.research.elamur.ai/free/en/papers/brown-2024-large-language-monkeys-scaling-inference/), 2024. URL https://arxiv.org/abs/2407.21787. [↗](https://arxiv.org/abs/2407.21787)
5. Noam Brown, Anton Bakhtin, Adam Lerer, and Qucheng Gong. Combining deep reinforcement learning and search for imperfect-information games, 2020a. URL [https://arxiv.org/](https://arxiv.org/) abs/2007.13544. — [arXiv:2007.13544](https://arxiv.org/abs/2007.13544) [↗](https://arxiv.org/)
6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020b.
7. Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Knowledge Discovery and Data Mining, 2006. URL [https://api.semanticscholar.org/](https://api.semanticscholar.org/) CorpusID:11253972. [↗](https://api.semanticscholar.org/)
8. Andrew Campbell, Joe Benton, Valentin De Bortoli, Tom Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models, 2022. [find](https://scholar.google.com/scholar?q=Andrew+Campbell%2C+Joe+Benton%2C+Valentin+De+Bortoli%2C+Tom+Rainforth%2C+George+Deligiannidis%2C+and+Arnaud+Doucet%2E+A+continuous+time+framework+for+discrete+denoising+models%2C+2022%2E)
9. Murray Campbell, A.Joseph Hoane, and Feng hsiung Hsu. Deep blue. Artificial Intelligence, 134(1):57–83, 2002. ISSN 0004-3702. doi: https://doi.org/10.1016/S0004-3702(01) 00129-1. URL https://www.sciencedirect.com/science/article/pii/ S0004370201001291. — [doi:10.1016/s0004-3702(01)](https://doi.org/10.1016/s0004-3702%2801%29) [↗](https://doi.org/10.1016/S0004-3702(01)
10. Lingjiao Chen, Jared Quincy Davis, Boris Hanin, Peter Bailis, Ion Stoica, Matei Zaharia, and James Zou. Are more llm calls all you need? towards scaling laws of compound inference systems, 2024. URL https://arxiv.org/abs/2403.02419. [↗](https://arxiv.org/abs/2403.02419)
11. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374, 2021.
12. Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning, 2023. [find](https://scholar.google.com/scholar?q=Ting+Chen%2C+Ruixiang+Zhang%2C+and+Geoffrey+Hinton%2E+Analog+bits%3A+Generating+discrete+data+using+diffusion+models+with+self%2Dconditioning%2C+2023%2E)
13. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). Journal of Machine Learning Research, 24(240): 1–113, 2023.
14. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022. URL https://arxiv.org/abs/ 2205.14135. [↗](https://arxiv.org/abs/)
15. Justin Deschenaux and Caglar Gulcehre. Promises, outlooks and challenges of diffusion language modeling, 2024. URL https://arxiv.org/abs/2406.11473. — [arXiv:2406.11473](https://arxiv.org/abs/2406.11473) [↗](https://arxiv.org/abs/2406.11473)
16. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805, 2018.
17. Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H. Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, Curtis Hawthorne, Rémi Leblond, Will Grathwohl, and Jonas Adler. Continuous diffusion for categorical data, 2022. [find](https://scholar.google.com/scholar?q=Sander+Dieleman%2C+Laurent+Sartran%2C+Arman+Roshannai%2C+Nikolay+Savinov%2C+Yaroslav+Ganin%2C+Pierre+H%2E+Richemond%2C+Arnaud+Doucet%2C+Robin+Strudel%2C+Chris+Dyer%2C+Conor+Durkan%2C+Curtis+Hawthorne%2C+Re%CC%81mi+Leblond%2C+Will+G)
18. William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022. [find](https://scholar.google.com/scholar?q=William+Fedus%2C+Barret+Zoph%2C+and+Noam+Shazeer%2E+Switch+transformers%3A+Scaling+to+trillion+parameter+models+with+simple+and+efficient+sparsity%2E+Journal+of+Machine+Learning+Research%2C+23%28120%29%3A1%E2%80%9339%2C+2022%2E)
19. Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September… — [doi:10.5281/zenodo](https://doi.org/10.5281/zenodo) [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Jason+Phang%2C+Laria+Reynolds%2C+Eric+Tang%2C+Anish+Thite)
20. Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. [http://Skylion007.github.io/](http://Skylion007.github.io/) OpenWebTextCorpus, 2019. [↗](http://Skylion007.github.io/)
21. Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014. [URL https://arxiv.org/abs/1406.2661](https://md.research.elamur.ai/free/en/papers/goodfellow-2014-generative-adversarial-nets/). [↗](https://arxiv.org/abs/1406.2661)
22. Google. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805, 2023.
23. Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens, 2024. URL https://arxiv.org/abs/2310.02226. [↗](https://arxiv.org/abs/2310.02226)
24. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models, 2024. [URL https://arxiv.org/abs/2306.08543](https://md.research.elamur.ai/free/en/papers/gu-2023-minillm-policy-distillation-large-language/). [↗](https://arxiv.org/abs/2306.08543)
25. Ishaan Gulrajani and Tatsunori B Hashimoto. Likelihood-based diffusion language models. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Ishaan+Gulrajani+and+Tatsunori+B+Hashimoto%2E+Likelihood%2Dbased+diffusion+language+models%2E+Advances+in+Neural+Information+Processing+Systems%2C+36%2C+2024%2E)
26. Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. Ssd-lm: Semi-autoregressive simplex-based diffusion language model for text generation and modular control, 2023. URL https://arxiv.org/abs/2210.17432. [↗](https://arxiv.org/abs/2210.17432)
27. Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Info… [find](https://scholar.google.com/scholar?q=Martin+Heusel%2C+Hubert+Ramsauer%2C+Thomas+Unterthiner%2C+Bernhard+Nessler%2C+and+Sepp+Hochreiter%2E+Gans+trained+by+a+two+time%2Dscale+update+rule+converge+to+a+local+nash+equilibrium%2E+In+I%2E+Guyon%2C+U%2E+Von+Luxbur)
28. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. [Distilling the knowledge in a neural network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/), 2015. URL https://arxiv.org/abs/1503.02531. [↗](https://arxiv.org/abs/1503.02531)
29. Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, 2020. [find](https://scholar.google.com/scholar?q=Jonathan+Ho%2C+Ajay+Jain%2C+and+Pieter+Abbeel%2E+Denoising+diffusion+probabilistic+models%2C+2020%2E)
30. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. [Training compute-optimal large language models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/). arXiv preprint arXiv:2203.15556, 2022.
31. Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions, 2021. URL https://arxiv.org/abs/2102.05379. [↗](https://arxiv.org/abs/2102.05379)
32. Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. [V-star: Training verifiers for self-taught reasoners](https://md.research.elamur.ai/free/en/papers/hosseini-2024-v-star-training-verifiers-self/). arXiv preprint arXiv:2402.06457, 2024.
33. Andy L. Jones. Scaling scaling laws with board games, 2021. URL [https://arxiv.org/](https://arxiv.org/) abs/2104.03113. — [arXiv:2104.03113](https://arxiv.org/abs/2104.03113) [↗](https://arxiv.org/)
34. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). arXiv preprint arXiv:2001.08361, 2020.
35. Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models, 2023. URL https://arxiv.org/abs/2107.00630. — [arXiv:2107.00630](https://arxiv.org/abs/2107.00630) [↗](https://arxiv.org/abs/2107.00630)
36. Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. [Understanding the effects of rlhf on llm generalisation and diversity](https://md.research.elamur.ai/free/en/papers/kirk-2023-understanding-effects-rlhf-llm-generalisation/), 2024. URL https://arxiv.org/abs/2310.06452. [↗](https://arxiv.org/abs/2310.06452)
37. Adam Lerer, Hengyuan Hu, Jakob Foerster, and Noam Brown. Improving policies via search in cooperative partially observable games, 2019. URL https://arxiv.org/abs/1912. 02318. [↗](https://arxiv.org/abs/1912)
38. Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B. Hashimoto. [Diffusion-lm improves controllable text generation](https://md.research.elamur.ai/free/en/papers/li-2022-diffusion-lm-improves-controllable-text/), 2022. URL https://arxiv.org/abs/ 2205.14217. [↗](https://arxiv.org/abs/)
39. Ziniu Li, Congliang Chen, Tian Xu, Zeyu Qin, Jiancong Xiao, Ruoyu Sun, and Zhi-Quan Luo. Entropic distribution matching in supervised fine-tuning of llms: Less overfitting and better diversity, 2024. URL https://arxiv.org/abs/2408.16673. — [arXiv:2408.16673](https://arxiv.org/abs/2408.16673) [↗](https://arxiv.org/abs/2408.16673)
40. Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion language modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310.[16834, 2023](https://md.research.elamur.ai/free/en/papers/lou-2023-discrete-diffusion-modeling-estimating-ratios/).
41. Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed, 2021. URL https://arxiv.org/abs/2101.02388. — [arXiv:2101.02388](https://arxiv.org/abs/2101.02388) [↗](https://arxiv.org/abs/2101.02388)
42. Weijian Luo. A comprehensive survey on knowledge distillation of diffusion models, 2023. URL https://arxiv.org/abs/2304.04262. [↗](https://arxiv.org/abs/2304.04262)
43. Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang. Diff-instruct: A universal approach for transferring knowledge from pre-trained diffusion models, 2024. URL https://arxiv.org/abs/2305.18455. — [arXiv:2305.18455](https://arxiv.org/abs/2305.18455) [↗](https://arxiv.org/abs/2305.18455)
44. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-… [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Prakhar+Gupta%2C+Skyler+Hallinan%2C+Luyu+Gao%2C+Sarah+Wiegreffe%2C+Uri+Alon%2C+Nouha+Dziri%2C+Shrimai+Prabhumoye%2C+Yiming+Yang%2C+Shashank+Gupta%2C+Bodhisattwa+Prasad+Majumder%2C+Katherine+Her)
45. Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models, 2021. [find](https://scholar.google.com/scholar?q=Alex+Nichol+and+Prafulla+Dhariwal%2E+Improved+denoising+diffusion+probabilistic+models%2C+2021%2E)
46. Sajad Norouzi, Rasa Hosseinzadeh, Felipe Perez, and Maksims Volkovs. DiMS: Distilling multiple steps of iterative non-autoregressive transformers for machine translation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 202… [find](https://scholar.google.com/scholar?q=Sajad+Norouzi%2C+Rasa+Hosseinzadeh%2C+Felipe+Perez%2C+and+Maksims+Volkovs%2E+DiMS%3A+Distilling+multiple+steps+of+iterative+non%2Dautoregressive+transformers+for+machine+translation%2E+In+Anna+Rogers%2C+Jordan+Boyd%2DG)
47. OpenAI. Gpt-2 output dataset. [https://github.com/openai/](https://github.com/openai/) gpt-2-output-dataset, 2019. Accessed: 2024-09-30. [↗](https://github.com/openai/)
48. Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data, 2024. URL https://arxiv.org/abs/2406.03736. — [arXiv:2406.03736](https://arxiv.org/abs/2406.03736) [↗](https://arxiv.org/abs/2406.03736)
49. Arnaud Pannatier, Evann Courdier, and Francois Fleuret. Sigma-gpts: A new approach to autoregressive models, 2024. URL https://arxiv.org/abs/2404.09562. — [arXiv:2404.09562](https://arxiv.org/abs/2404.09562) [↗](https://arxiv.org/abs/2404.09562)
50. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. [The lambada dataset: Word prediction requiring a broad discourse context](https://md.research.elamur.ai/free/en/papers/paperno-2016-lambada-dataset-word-prediction-requiring/), 2016. URL [https://arxiv.org/](https://arxiv.org/) abs/1606.06031. [↗](https://arxiv.org/)
51. William Peebles and Saining Xie. [Scalable diffusion models with transformers](https://md.research.elamur.ai/free/en/papers/peebles-2022-scalable-diffusion-models-transformers/). In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195–4205, 2023.
52. Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. [Mauve: Measuring the gap between neural text and human text using divergence frontiers](https://md.research.elamur.ai/free/en/papers/pillutla-2021-mauve-measuring-gap-between-neural/), 2021.
53. Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference, 2022. URL https://arxiv.org/abs/2211.05102. [↗](https://arxiv.org/abs/2211.05102)
54. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2C+2019%2E)
55. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of machine learning research, 21(140):1–67, 2020.
56. Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. [Hierarchical text-conditional image generation with clip latents](https://md.research.elamur.ai/free/en/papers/ramesh-2022-hierarchical-text-conditional-image-generation/), 2022. URL https://arxiv.org/abs/ 2204.06125. [↗](https://arxiv.org/abs/)
57. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. [High-resolution image synthesis with latent diffusion models](https://md.research.elamur.ai/free/en/papers/rombach-2021-high-resolution-image-synthesis-latent/), 2022. URL [https://arxiv.org/](https://arxiv.org/) abs/2112.10752. [↗](https://arxiv.org/)
58. Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. [Mathematical discoveries from program search with large language models](https://md.research.elamur.ai/free/en/papers/paredes-2023-mathematical-discoveries-program-search-large/). Nature, 625(7995):468–475, 2024.
59. Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language… — [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://md.research.elamur.ai/free/en/papers/saharia-2022-photorealistic-text-image-diffusion-models/)
60. Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and Volodymyr Kuleshov. [Simple and effective masked diffusion language models](https://md.research.elamur.ai/free/en/papers/sahoo-2024-simple-effective-masked-diffusion-language/), 2024. URL https://arxiv.org/abs/2406.07524. [↗](https://arxiv.org/abs/2406.07524)
61. Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models, 2022. URL https://arxiv.org/abs/2202.00512. — [arXiv:2202.00512](https://arxiv.org/abs/2202.00512) [↗](https://arxiv.org/abs/2202.00512)
62. Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation, 2023. URL https://arxiv.org/abs/2311.17042. — [arXiv:2311.17042](https://arxiv.org/abs/2311.17042) [↗](https://arxiv.org/abs/2311.17042)
63. Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K. Titsias. [Simplified and generalized masked diffusion for discrete data](https://md.research.elamur.ai/free/en/papers/shi-2024-simplified-generalized-masked-diffusion-discrete/), 2024. URL https://arxiv.org/abs/2406. 04329. [↗](https://arxiv.org/abs/2406)
64. David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukc… [find](https://scholar.google.com/scholar?q=David+Silver%2C+Aja+Huang%2C+Chris+J%2E+Maddison%2C+Arthur+Guez%2C+Laurent+Sifre%2C+George+van+den+Driessche%2C+Julian+Schrittwieser%2C+Ioannis+Antonoglou%2C+Veda+Panneershelvam%2C+Marc+Lanctot%2C+Sander+Dieleman%2C+Dominik+)
65. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. [Scaling llm test-time compute optimally can be more effective than scaling model parameters](https://md.research.elamur.ai/free/en/papers/snell-2024-scaling-llm-test-time-compute/), 2024. URL [https://arxiv.org/](https://arxiv.org/) abs/2408.03314. [↗](https://arxiv.org/)
66. Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. [Deep unsupervised learning using nonequilibrium thermodynamics](https://md.research.elamur.ai/free/en/papers/sohl-dickstein-2015-deep-unsupervised-learning-using-nonequilibrium/). In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Resear…
67. Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. [Deep unsupervised learning using nonequilibrium thermodynamics](https://md.research.elamur.ai/free/en/papers/sohl-dickstein-2015-deep-unsupervised-learning-using-nonequilibrium/), 2015b. URL [https://arxiv.org/](https://arxiv.org/) abs/1503.03585. [↗](https://arxiv.org/)
68. Jiaming Song, Chenlin Meng, and Stefano Ermon. [Denoising diffusion implicit models](https://md.research.elamur.ai/free/en/papers/song-2020-denoising-diffusion-implicit-models/), 2022. URL https://arxiv.org/abs/2010.02502. [↗](https://arxiv.org/abs/2010.02502)
69. Yang Song and Stefano Ermon. [Generative modeling by estimating gradients of the data distribution](https://md.research.elamur.ai/free/en/papers/song-2019-generative-modeling-estimating-gradients-data/), 2020. URL https://arxiv.org/abs/1907.05600. [↗](https://arxiv.org/abs/1907.05600)
70. Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. [Score-based generative modeling through stochastic differential equations](https://md.research.elamur.ai/free/en/papers/song-2020-score-based-generative-modeling-through/), 2021.
71. Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models, 2023. URL https://arxiv.org/abs/2303.01469. [↗](https://arxiv.org/abs/2303.01469)
72. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/), 2023. URL https://arxiv.org/abs/ 2104.09864. [↗](https://arxiv.org/abs/)
73. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023.
74. Trieu H. Trinh, Yuhuai Wu, Quoc V. Le, He He, and Thang Luong. [Solving olympiad geometry without human demonstrations](https://md.research.elamur.ai/free/en/papers/trinh-2024-solving-olympiad-geometry-without-human/). Nature, 625(7995):476–482, Jan 2024a. ISSN 1476-4687. doi: 10.1038/s41586-023-06747-5. URL https://doi.org/10.1038/ s41586-023-06747-5.
75. Trieu H Trinh, Yuhuai Wu, Quoc V Le, He He, and Thang Luong. [Solving olympiad geometry without human demonstrations](https://md.research.elamur.ai/free/en/papers/trinh-2024-solving-olympiad-geometry-without-human/). Nature, 625(7995):476–482, 2024b.
76. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30, 2017.
77. Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang Sui. [Math-shepherd: Verify and reinforce llms step-by-step without human annotations](https://md.research.elamur.ai/free/en/papers/wang-2024-math-shepherd-verify-reinforce-llms/), 2024. URL https://arxiv.org/abs/2312.08935. [↗](https://arxiv.org/abs/2312.08935)
78. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/), 2023. URL https://arxiv.org/abs/2201.11903. [↗](https://arxiv.org/abs/2201.11903)
79. Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. [f-divergence minimization for sequence-level knowledge distillation](https://md.research.elamur.ai/free/en/papers/wen-2023-f-divergence-minimization-sequence-level/), 2023. URL https://arxiv.org/abs/2307.15190. [↗](https://arxiv.org/abs/2307.15190)
80. Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. An empirical analysis of compute-optimal inference for problem-solving with language models, 2024. [URL https://arxiv.org/abs/2408.00724](https://md.research.elamur.ai/free/en/papers/wu-2024-inference-scaling-laws-empirical-analysis/). [↗](https://arxiv.org/abs/2408.00724)
81. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. [Xlnet: Generalized autoregressive pretraining for language understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/), 2020. URL https://arxiv.org/abs/1906.08237. [↗](https://arxiv.org/abs/1906.08237)
82. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023. URL https://arxiv.org/abs/2305.10601. [↗](https://arxiv.org/abs/2305.10601)
83. Jiacheng Ye, Shansan Gong, Liheng Chen, Lin Zheng, Jiahui Gao, Han Shi, Chuan Wu, Xin Jiang, Zhenguo Li, Wei Bi, and Lingpeng Kong. Diffusion of thoughts: Chain-of-thought reasoning in diffusion language models, 2024. URL https://arxiv.org/abs/2402.07754. — [arXiv:2402.07754](https://arxiv.org/abs/2402.07754) [↗](https://arxiv.org/abs/2402.07754)
84. Lingxiao Zhao, Xueying Ding, Lijun Yu, and Leman Akoglu. Unified discrete diffusion for categorical data, 2024. URL https://arxiv.org/abs/2402.03701. — [arXiv:2402.03701](https://arxiv.org/abs/2402.03701) [↗](https://arxiv.org/abs/2402.03701)
85. Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming-Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. arXiv preprint arXiv:2409.02908, 2024a. — [arXiv:2409.02908](https://arxiv.org/abs/2409.02908) [find](https://scholar.google.com/scholar?q=Kaiwen+Zheng%2C+Yongxin+Chen%2C+Hanzi+Mao%2C+Ming%2DYu+Liu%2C+Jun+Zhu%2C+and+Qinsheng+Zhang%2E+Masked+diffusion+models+are+secretly+time%2Dagnostic+masked+models+and+exploit+inaccurate+categorical+sampling%2E+arXiv+pre)
86. Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. A reparameterized discrete diffusion model for text generation, 2024b. URL https://arxiv.org/abs/2302.05737. — [arXiv:2302.05737](https://arxiv.org/abs/2302.05737) [↗](https://arxiv.org/abs/2302.05737)
87. Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. [Texygen: A benchmarking platform for text generation models](https://md.research.elamur.ai/free/en/papers/zhu-2018-texygen-benchmarking-platform-text-generation/), 2018. URL [https://arxiv.org/](https://arxiv.org/) abs/1802.01886. *Table 1: Downstream evaluation results. We report the accuracy of GPT-2, the teacher and students after 7 roun… [↗](https://arxiv.org/)

*Updated 3 November 2026*