---
title: "PixArt-α — paper by Chen et al. (2023) — summary and ideas"
description: "Richer captions: they replace sparse web captions with dense descriptions that LLaVA writes for… 76 references, cited by 79 papers here."
authors: ["Junsong Chen", "Jincheng Yu", "Chongjian Ge", "Lewei Yao", "Enze Xie", "Yue Wu", "Zhongdao Wang", "James Kwok", "Ping Luo", "Huchuan Lu", "Zhenguo Li"]
year: 2023
venue: "arXiv:2310.00426"
source: "https://arxiv.org/abs/2310.00426"
doi: "10.48550/arxiv.2310.00426"
arxiv: "2310.00426"
licence: "CC BY-NC-SA 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-nc-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chen-2023-pixart-alpha-fast-training-diffusion/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chen-2023-pixart-alpha-fast-training-diffusion/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/) / PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis

# PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis

**Authors:** Junsong Chen, [Jincheng Yu](https://md.research.elamur.ai/free/en/authors/jincheng-yu/), Chongjian Ge, [Lewei Yao](https://md.research.elamur.ai/free/en/authors/lewei-yao/), [Enze Xie](https://md.research.elamur.ai/free/en/authors/enze-xie/), [Yue Wu](https://md.research.elamur.ai/free/en/authors/yue-wu/), Zhongdao Wang, [James Kwok](https://md.research.elamur.ai/free/en/authors/james-tin-yau-kwok/), [Ping Luo](https://md.research.elamur.ai/free/en/authors/ping-luo/), Huchuan Lu and [Zhenguo Li](https://md.research.elamur.ai/free/en/authors/zhenguo-li/) · **Year:** 2023 · **Venue:** arXiv:2310.00426 · **Licence:** [CC BY-NC-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-nc-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2310.00426) · [PDF](https://arxiv.org/pdf/2310.00426) · [DOI 10.48550/arxiv.2310.00426](https://doi.org/10.48550/arxiv.2310.00426) · [arXiv 2310.00426](https://arxiv.org/abs/2310.00426) · [OpenAlex](https://openalex.org/W7051896053)

**Citations:** 8 citations (OpenAlex, 2026-10-14) · [79](https://md.research.elamur.ai/free/en/papers/chen-2023-pixart-alpha-fast-training-diffusion/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Training state-of-the-art text-to-image (T2I) diffusion models takes thousands of GPU days and hundreds of thousands of dollars. This puts them out of reach for most labs and startups and adds a large carbon footprint. PixArt-α is a Transformer-based T2I diffusion model that aims for near-commercial image quality, up to 1024×1024 resolution, at a much lower cost. The authors combine three ideas:

- **Staged training:** they split training into three stages, learning pixel structure, then text-image alignment, then aesthetics and high resolution.
- **Lighter architecture:** they adapt DiT with cross-attention for text and a simpler conditioning branch, so it can start from ImageNet-pretrained weights.
- **Richer captions:** they replace sparse web captions with dense descriptions that LLaVA writes for images from SAM.

Training costs about 753 A100 GPU days (~$28,400), roughly 12% of Stable Diffusion v1.5's compute and about 1% of RAPHAEL's. In user studies, the model matches or beats systems such as DALL·E 2 and Stable Diffusion, and it scores well on T2I-CompBench. The work shows that careful training design and better data can replace much of the brute-force compute.

## Key points
- **Cost comparison:**
  - PixArt-α: ~753 A100 GPU days (~$28,400).
  - SDv1.5: ~6,250 GPU days (~$320,000).
  - RAPHAEL: ~60K GPU days (~$3.08M) and about 35 tons of CO₂.
- **Relative savings:**
  - Versus SDv1.5, PixArt-α uses under 1.25% of the training data and cuts CO₂ emissions by about 90%.
  - Versus RAPHAEL, its CO₂ emissions are about 1.2% and its training cost about 0.91%.
- **Architecture:** it builds on Diffusion Transformer (DiT) with three changes:
  - multi-head cross-attention that injects text features from a T5 encoder;
  - a streamlined conditioning branch for timestep and class, shared across blocks;
  - a reparameterization trick that lets class-conditional ImageNet weights load directly.
- **Data:** the authors identify several problems with LAION captions:
  - text that does not match the image;
  - descriptions that are incomplete;
  - a long tail of rare nouns.

  To fix this, LLaVA auto-captions SAM images, which contain many diverse objects.
- **Evaluation:** a human user study (image quality and semantic alignment) plus T2I-CompBench for compositional control.

## Interesting ideas
- **Decompose a hard task into stages with different data.** Learn the "easy" distributional structure first from cheap data, then alignment, then quality polishing on a small curated set. A similar curriculum could cut training costs for NLP models, for example general language first, then task alignment, then style.
- **Design new architectures to inherit old checkpoints.** Reparameterizing the modified model lets an existing pretrained checkpoint load directly, rather than training from random initialization. This is a reusable engineering pattern whenever you extend a model with a new conditioning input.
- **Concept density matters more than dataset size.** Captions that mention more objects per example teach more per iteration. Measuring and raising information density, for example with how many distinct nouns appear and how often, is a useful lens for any paired-data corpus, including Ukrainian text resources.
- **Regenerate weak labels with a strong model.** Using a vision-language model like LLaVA to rewrite noisy web captions is a cheap form of data repair. The same idea applies to re-annotating noisy NLP datasets with an LLM before training a smaller model.
- **Report cost and CO₂ alongside quality.** Treating GPU days, dollars and emissions as headline metrics makes efficiency claims concrete. This is a good reporting habit for any project with limited compute.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (1): [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 811 works

## Cite

Cite as: Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu and Zhenguo Li. 2023. PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis. arXiv:2310.00426.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{chen2023pixart,
  title = {{PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis}},
  author = {Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu and Zhenguo Li},
  year = {2023},
  doi = {10.48550/arxiv.2310.00426},
  eprint = {2310.00426},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.00426}
}
```

- [← previous InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition](https://md.research.elamur.ai/free/en/papers/zhang-2023-internlm-xcomposer-vision-language/)
- [next → REST: Retrieval-Based Speculative Decoding](https://md.research.elamur.ai/free/en/papers/he-2023-rest-retrieval-based-speculative-decoding/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/)

## References

76 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anne-Laure Ligozat Alexandra Sasha Luccioni, Sylvain Viguier. Estimating the carbon footprint of bloom, a 176b parameter language model. In arXiv preprint arXiv:2211.02001, 2022. — [arXiv:2211.02001](https://arxiv.org/abs/2211.02001) [find](https://scholar.google.com/scholar?q=Anne%2DLaure+Ligozat+Alexandra+Sasha+Luccioni%2C+Sylvain+Viguier%2E+Estimating+the+carbon+footprint+of+bloom%2C+a+176b+parameter+language+model%2E+In+arXiv+preprint+arXiv%3A2211%2E02001%2C+2022%2E+%E2%80%94+%5BarXiv%3A2211%2E02001%5D%28)
2. Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. In arXiv, 2022. [find](https://scholar.google.com/scholar?q=Yogesh+Balaji%2C+Seungjun+Nah%2C+Xun+Huang%2C+Arash+Vahdat%2C+Jiaming+Song%2C+Karsten+Kreis%2C+Miika+Aittala%2C+Timo+Aila%2C+Samuli+Laine%2C+Bryan+Catanzaro%2C+et+al%2E+ediffi%3A+Text%2Dto%2Dimage+diffusion+models+with+an+ensemb)
3. Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In CVPR, 2023. [find](https://scholar.google.com/scholar?q=Fan+Bao%2C+Shen+Nie%2C+Kaiwen+Xue%2C+Yue+Cao%2C+Chongxuan+Li%2C+Hang+Su%2C+and+Jun+Zhu%2E+All+are+worth+words%3A+A+vit+backbone+for+diffusion+models%2E+In+CVPR%2C+2023%2E)
4. Eyal Betzalel, Coby Penso, Aviv Navon, and Ethan Fetaya. A study on the evaluation of generative models. In arXiv, 2022. [find](https://scholar.google.com/scholar?q=Eyal+Betzalel%2C+Coby+Penso%2C+Aviv+Navon%2C+and+Ethan+Fetaya%2E+A+study+on+the+evaluation+of+generative+models%2E+In+arXiv%2C+2022%2E)
5. Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. [End-to-end object detection with transformers](https://md.research.elamur.ai/free/en/papers/carion-2020-end-end-object-detection-transformers/). In ECCV, 2020.
6. DeepFloyd. Deepfloyd, 2023. URL [https://www.deepfloyd.ai/](https://www.deepfloyd.ai/). [↗](https://www.deepfloyd.ai/)
7. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. [find](https://scholar.google.com/scholar?q=Jia+Deng%2C+Wei+Dong%2C+Richard+Socher%2C+Li%2DJia+Li%2C+Kai+Li%2C+and+Li+Fei%2DFei%2E+Imagenet%3A+A+large%2Dscale+hierarchical+image+database%2E+In+2009+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+pp%2E+248%E2%80%93)
8. Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780–8794, 2021. [find](https://scholar.google.com/scholar?q=Prafulla+Dhariwal+and+Alexander+Nichol%2E+Diffusion+models+beat+gans+on+image+synthesis%2E+Advances+in+neural+information+processing+systems%2C+34%3A8780%E2%80%938794%2C+2021%2E)
9. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. [An image is worth 16x16 words: Transformers for image recognition at scale](https://md.research.elamur.ai/free/en/papers/dosovitskiy-2020-image-worth-16x16-words-transformers/). In ICLR, 2020a.
10. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. [An image is worth 16x16 words: Transformers for image recognition at scale](https://md.research.elamur.ai/free/en/papers/dosovitskiy-2020-image-worth-16x16-words-transformers/). In arXiv, 2020b.
11. Zhida Feng, Zhenyu Zhang, Xintong Yu, Yewei Fang, Lanxin Li, Xuyi Chen, Yuxiang Lu, Jiaxiang Liu, Weichong Yin, Shikun Feng, et al. Ernie-vilg 2.0: Improving text-to-image diffusion model with knowledge-enhanced mixture-of-denoising-experts. In CVPR, 2023. [find](https://scholar.google.com/scholar?q=Zhida+Feng%2C+Zhenyu+Zhang%2C+Xintong+Yu%2C+Yewei+Fang%2C+Lanxin+Li%2C+Xuyi+Chen%2C+Yuxiang+Lu%2C+Jiaxiang+Liu%2C+Weichong+Yin%2C+Shikun+Feng%2C+et+al%2E+Ernie%2Dvilg+2%2E0%3A+Improving+text%2Dto%2Dimage+diffusion+model+with+knowled)
12. Chongjian Ge, Junsong Chen, Enze Xie, Zhongdao Wang, Lanqing Hong, Huchuan Lu, Zhenguo Li, and Ping Luo. Metabev: Solving sensor failures for 3d detection and map segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8721–8731, 2023. [find](https://scholar.google.com/scholar?q=Chongjian+Ge%2C+Junsong+Chen%2C+Enze+Xie%2C+Zhongdao+Wang%2C+Lanqing+Hong%2C+Huchuan+Lu%2C+Zhenguo+Li%2C+and+Ping+Luo%2E+Metabev%3A+Solving+sensor+failures+for+3d+detection+and+map+segmentation%2E+In+Proceedings+of+the+I)
13. Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. [Imagebind: One embedding space to bind them all](https://md.research.elamur.ai/free/en/papers/girdhar-2023-imagebind-one-embedding-space-bind/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15180–15190, 2023.
14. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. [Generative adversarial nets](https://md.research.elamur.ai/free/en/papers/goodfellow-2014-generative-adversarial-nets/). In NeurIPS, 2014.
15. Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. NeurIPS, 2021. [find](https://scholar.google.com/scholar?q=Kai+Han%2C+An+Xiao%2C+Enhua+Wu%2C+Jianyuan+Guo%2C+Chunjing+Xu%2C+and+Yunhe+Wang%2E+Transformer+in+transformer%2E+NeurIPS%2C+2021%2E)
16. Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. [Masked autoencoders are scalable vision learners](https://md.research.elamur.ai/free/en/papers/he-2021-masked-autoencoders-scalable-vision-learners/). In CVPR, 2022.
17. Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NeurIPS, 2017. [find](https://scholar.google.com/scholar?q=Martin+Heusel%2C+Hubert+Ramsauer%2C+Thomas+Unterthiner%2C+Bernhard+Nessler%2C+and+Sepp+Hochreiter%2E+Gans+trained+by+a+two+time%2Dscale+update+rule+converge+to+a+local+nash+equilibrium%2E+In+NeurIPS%2C+2017%2E)
18. Jonathan Ho and Tim Salimans. arXiv:2207.[12598, 2022](https://md.research.elamur.ai/free/en/papers/ho-2022-classifier-free-diffusion-guidance/).
19. [Classifier-free diffusion guidance](https://md.research.elamur.ai/free/en/papers/ho-2022-classifier-free-diffusion-guidance/). arXiv preprint Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
20. Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In ICLR, 2021.
21. Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. In ICCV, 2023. [find](https://scholar.google.com/scholar?q=Kaiyi+Huang%2C+Kaiyue+Sun%2C+Enze+Xie%2C+Zhenguo+Li%2C+and+Xihui+Liu%2E+T2i%2Dcompbench%3A+A+comprehensive+benchmark+for+open%2Dworld+compositional+text%2Dto%2Dimage+generation%2E+In+ICCV%2C+2023%2E)
22. Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In CVPR, 2023. [find](https://scholar.google.com/scholar?q=Minguk+Kang%2C+Jun%2DYan+Zhu%2C+Richard+Zhang%2C+Jaesik+Park%2C+Eli+Shechtman%2C+Sylvain+Paris%2C+and+Taesung+Park%2E+Scaling+up+gans+for+text%2Dto%2Dimage+synthesis%2E+In+CVPR%2C+2023%2E)
23. Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2426–2435, June 2022. [find](https://scholar.google.com/scholar?q=Gwanghyun+Kim%2C+Taesung+Kwon%2C+and+Jong+Chul+Ye%2E+Diffusionclip%3A+Text%2Dguided+diffusion+models+for+robust+image+manipulation%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recog)
24. Diederik P Kingma and Max Welling. [Auto-encoding variational bayes](https://md.research.elamur.ai/free/en/papers/kingma-2013-auto-encoding-variational-bayes/). In arXiv, 2013.
25. Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. [Segment anything](https://md.research.elamur.ai/free/en/papers/kirillov-2023-segment-anything/). In ICCV, 2023.
26. Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. In arXiv, 2023. [find](https://scholar.google.com/scholar?q=Yuval+Kirstain%2C+Adam+Polyak%2C+Uriel+Singer%2C+Shahbuland+Matiana%2C+Joe+Penna%2C+and+Omer+Levy%2E+Pick%2Da%2Dpic%3A+An+open+dataset+of+user+preferences+for+text%2Dto%2Dimage+generation%2E+In+arXiv%2C+2023%2E)
27. Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers. In ECCV, 2022a. [find](https://scholar.google.com/scholar?q=Zhiqi+Li%2C+Wenhai+Wang%2C+Hongyang+Li%2C+Enze+Xie%2C+Chonghao+Sima%2C+Tong+Lu%2C+Yu+Qiao%2C+and+Jifeng+Dai%2E+Bevformer%3A+Learning+bird%E2%80%99s%2Deye%2Dview+representation+from+multi%2Dcamera+images+via+spatiotemporal+transforme)
28. Zhiqi Li, Wenhai Wang, Enze Xie, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, Ping Luo, and Tong Lu. Panoptic segformer: Delving deeper into panoptic segmentation with transformers. In CVPR, 2022b. [find](https://scholar.google.com/scholar?q=Zhiqi+Li%2C+Wenhai+Wang%2C+Enze+Xie%2C+Zhiding+Yu%2C+Anima+Anandkumar%2C+Jose+M+Alvarez%2C+Ping+Luo%2C+and+Tong+Lu%2E+Panoptic+segformer%3A+Delving+deeper+into+panoptic+segmentation+with+transformers%2E+In+CVPR%2C+2022b%2E)
29. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. [Microsoft coco: Common objects in context](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/). In ECCV, 2014.
30. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). In arXiv, 2023.
31. Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. [Swin transformer: Hierarchical vision transformer using shifted windows](https://md.research.elamur.ai/free/en/papers/liu-2021-swin-transformer-hierarchical-vision-transformer/). In ICCV, 2021.
32. Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In CVPR, 2022. [find](https://scholar.google.com/scholar?q=Ze+Liu%2C+Jia+Ning%2C+Yue+Cao%2C+Yixuan+Wei%2C+Zheng+Zhang%2C+Stephen+Lin%2C+and+Han+Hu%2E+Video+swin+transformer%2E+In+CVPR%2C+2022%2E)
33. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In arXiv, 2017. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+Decoupled+weight+decay+regularization%2E+In+arXiv%2C+2017%2E)
34. Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775–5787, 2022. [find](https://scholar.google.com/scholar?q=Cheng+Lu%2C+Yuhao+Zhou%2C+Fan+Bao%2C+Jianfei+Chen%2C+Chongxuan+Li%2C+and+Jun+Zhu%2E+Dpm%2Dsolver%3A+A+fast+ode+solver+for+diffusion+probabilistic+model+sampling+in+around+10+steps%2E+Advances+in+Neural+Information+Proc)
35. Microsoft. Gpu selling, 2023. URL [https://www.leadergpu.com/](https://www.leadergpu.com/). [↗](https://www.leadergpu.com/)
36. Midjourney. Midjourney, 2023. URL [https://www.midjourney.com](https://www.midjourney.com). [↗](https://www.midjourney.com)
37. Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In arXiv, 2023. [find](https://scholar.google.com/scholar?q=Chong+Mou%2C+Xintao+Wang%2C+Liangbin+Xie%2C+Jian+Zhang%2C+Zhongang+Qi%2C+Ying+Shan%2C+and+Xiaohu+Qie%2E+T2i%2Dadapter%3A+Learning+adapters+to+dig+out+more+controllable+ability+for+text%2Dto%2Dimage+diffusion+models%2E+In+arX)
38. Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162–8171. PMLR, 2021. [find](https://scholar.google.com/scholar?q=Alexander+Quinn+Nichol+and+Prafulla+Dhariwal%2E+Improved+denoising+diffusion+probabilistic+models%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+8162%E2%80%938171%2E+PMLR%2C+2021%2E)
39. NLTK. Nltk, 2023. URL [https://www.nltk.org/](https://www.nltk.org/). [↗](https://www.nltk.org/)
40. NVIDIA. Getting immediate speedups with a100 and tf32, 2023. URL [https://developer.nvidia.com/blog/getting-immediate-speedups-with-a100-tf32](https://developer.nvidia.com/blog/getting-immediate-speedups-with-a100-tf32). [↗](https://developer.nvidia.com/blog/getting-immediate-speedups-with-a100-tf32)
41. OpenAI. Dalle-2, 2023. URL [https://openai.com/dall-e-2](https://openai.com/dall-e-2). [↗](https://openai.com/dall-e-2)
42. Junting Pan, Keqiang Sun, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. Journeydb: A benchmark for generative image understanding. In arXiv, 2023. [find](https://scholar.google.com/scholar?q=Junting+Pan%2C+Keqiang+Sun%2C+Yuying+Ge%2C+Hao+Li%2C+Haodong+Duan%2C+Xiaoshi+Wu%2C+Renrui+Zhang%2C+Aojun+Zhou%2C+Zipeng+Qin%2C+Yi+Wang%2C+Jifeng+Dai%2C+Yu+Qiao%2C+and+Hongsheng+Li%2E+Journeydb%3A+A+benchmark+for+generative+image)
43. William Peebles and Saining Xie. [Scalable diffusion models with transformers](https://md.research.elamur.ai/free/en/papers/peebles-2022-scalable-diffusion-models-transformers/). In ICCV, 2023.
44. Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. [Film: Visual reasoning with a general conditioning layer](https://md.research.elamur.ai/free/en/papers/perez-2017-film-visual-reasoning-general-conditioning/). In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
45. Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. [Sdxl: Improving latent diffusion models for high-resolution image synthesis](https://md.research.elamur.ai/free/en/papers/podell-2023-sdxl-improving-latent-diffusion-models/). In arXiv, 2023.
46. Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv, 2022. [find](https://scholar.google.com/scholar?q=Ben+Poole%2C+Ajay+Jain%2C+Jonathan+T%2E+Barron%2C+and+Ben+Mildenhall%2E+Dreamfusion%3A+Text%2Dto%2D3d+using+2d+diffusion%2E+arXiv%2C+2022%2E)
47. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. OpenAI blog, 2018. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+Ilya+Sutskever%2C+et+al%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+OpenAI+blog%2C+2018%2E)
48. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+2019%2E)
49. Danilo Rezende and Shakir Mohamed. [Variational inference with normalizing flows](https://md.research.elamur.ai/free/en/papers/rezende-2015-variational-inference-normalizing-flows/). In ICML, 2015.
50. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. [High-resolution image synthesis with latent diffusion models](https://md.research.elamur.ai/free/en/papers/rombach-2021-high-resolution-image-synthesis-latent/). In CVPR, 2022.
51. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. [U-net: Convolutional networks for biomedical image segmentation](https://md.research.elamur.ai/free/en/papers/ronneberger-2015-u-net-convolutional-networks-biomedical/). In MICCAI, 2015.
52. Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In arXiv, 2022. [find](https://scholar.google.com/scholar?q=Nataniel+Ruiz%2C+Yuanzhen+Li%2C+Varun+Jampani%2C+Yael+Pritch%2C+Michael+Rubinstein%2C+and+Kfir+Aberman%2E+Dreambooth%3A+Fine+tuning+text%2Dto%2Dimage+diffusion+models+for+subject%2Ddriven+generation%2E+In+arXiv%2C+2022%2E)
53. Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. [Photorealistic text-to-image diffusion models with deep language understanding](https://md.research.elamur.ai/free/en/papers/saharia-2022-photorealistic-text-image-diffusion-models/). In NeurIPS, 2022.
54. Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. [Laion-400m: Open dataset of clip-filtered 400 million image-text pairs](https://md.research.elamur.ai/free/en/papers/schuhmann-2021-laion-400m-open-dataset-clip/). In arXiv, 2021.
55. Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. [Deep unsupervised learning using nonequilibrium thermodynamics](https://md.research.elamur.ai/free/en/papers/sohl-dickstein-2015-deep-unsupervised-learning-using-nonequilibrium/). In ICML, 2015.
56. Yang Song and Stefano Ermon. [Generative modeling by estimating gradients of the data distribution](https://md.research.elamur.ai/free/en/papers/song-2019-generative-modeling-estimating-gradients-data/). In NeurIPS, 2019.
57. Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. [Score-based generative modeling through stochastic differential equations](https://md.research.elamur.ai/free/en/papers/song-2020-score-based-generative-modeling-through/). In ICLR, 2021.
58. Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In ICCV, 2021. [find](https://scholar.google.com/scholar?q=Robin+Strudel%2C+Ricardo+Garcia%2C+Ivan+Laptev%2C+and+Cordelia+Schmid%2E+Segmenter%3A+Transformer+for+semantic+segmentation%2E+In+ICCV%2C+2021%2E)
59. Peize Sun, Jinkun Cao, Yi Jiang, Rufeng Zhang, Enze Xie, Zehuan Yuan, Changhu Wang, and Ping Luo. Transtrack: Multiple object tracking with transformer. In arXiv, 2020. [find](https://scholar.google.com/scholar?q=Peize+Sun%2C+Jinkun+Cao%2C+Yi+Jiang%2C+Rufeng+Zhang%2C+Enze+Xie%2C+Zehuan+Yuan%2C+Changhu+Wang%2C+and+Ping+Luo%2E+Transtrack%3A+Multiple+object+tracking+with+transformer%2E+In+arXiv%2C+2020%2E)
60. Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. [Training data-efficient image transformers & distillation through attention](https://md.research.elamur.ai/free/en/papers/touvron-2020-training-data-efficient-image-transformers/). In ICML, 2021.
61. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In NeurIPS, 2017.
62. Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, 2021. [find](https://scholar.google.com/scholar?q=Wenhai+Wang%2C+Enze+Xie%2C+Xiang+Li%2C+Deng%2DPing+Fan%2C+Kaitao+Song%2C+Ding+Liang%2C+Tong+Lu%2C+Ping+Luo%2C+and+Ling+Shao%2E+Pyramid+vision+transformer%3A+A+versatile+backbone+for+dense+prediction+without+convolutions%2E+I)
63. Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 2022. [find](https://scholar.google.com/scholar?q=Wenhai+Wang%2C+Enze+Xie%2C+Xiang+Li%2C+Deng%2DPing+Fan%2C+Kaitao+Song%2C+Ding+Liang%2C+Tong+Lu%2C+Ping+Luo%2C+and+Ling+Shao%2E+Pvt+v2%3A+Improved+baselines+with+pyramid+vision+transformer%2E+Computational+Visual+Media%2C+2022%2E)
64. Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. arXiv preprint arXiv:2212.11565, 2022. [find](https://scholar.google.com/scholar?q=Jay+Zhangjie+Wu%2C+Yixiao+Ge%2C+Xintao+Wang%2C+Stan+Weixian+Lei%2C+Yuchao+Gu%2C+Wynne+Hsu%2C+Ying+Shan%2C+Xiaohu+Qie%2C+and+Mike+Zheng+Shou%2E+Tune%2Da%2Dvideo%3A+One%2Dshot+tuning+of+image+diffusion+models+for+text%2Dto%2Dvideo+g)
65. Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34:12077–12090, 2021. [find](https://scholar.google.com/scholar?q=Enze+Xie%2C+Wenhai+Wang%2C+Zhiding+Yu%2C+Anima+Anandkumar%2C+Jose+M+Alvarez%2C+and+Ping+Luo%2E+Segformer%3A+Simple+and+efficient+design+for+semantic+segmentation+with+transformers%2E+Advances+in+Neural+Information+Pr)
66. Enze Xie, Lewei Yao, Han Shi, Zhili Liu, Daquan Zhou, Zhaoqiang Liu, Jiawei Li, and Zhenguo Li. Difffit: Unlocking transferability of large diffusion models via simple parameter-efficient fine-tuning. In ICCV, 2023. [find](https://scholar.google.com/scholar?q=Enze+Xie%2C+Lewei+Yao%2C+Han+Shi%2C+Zhili+Liu%2C+Daquan+Zhou%2C+Zhaoqiang+Liu%2C+Jiawei+Li%2C+and+Zhenguo+Li%2E+Difffit%3A+Unlocking+transferability+of+large+diffusion+models+via+simple+parameter%2Defficient+fine%2Dtuning%2E)
67. Saining Xie and Zhuowen Tu. Holistically-nested edge detection. In ICCV, 2015. [find](https://scholar.google.com/scholar?q=Saining+Xie+and+Zhuowen+Tu%2E+Holistically%2Dnested+edge+detection%2E+In+ICCV%2C+2015%2E)
68. Shuchen Xue, Mingyang Yi, Weijian Luo, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhi-Ming Ma. Sa-solver: Stochastic adams solver for fast sampling of diffusion models. arXiv preprint arXiv:2309.05019, 2023a. — [arXiv:2309.05019](https://arxiv.org/abs/2309.05019) [find](https://scholar.google.com/scholar?q=Shuchen+Xue%2C+Mingyang+Yi%2C+Weijian+Luo%2C+Shifeng+Zhang%2C+Jiacheng+Sun%2C+Zhenguo+Li%2C+and+Zhi%2DMing+Ma%2E+Sa%2Dsolver%3A+Stochastic+adams+solver+for+fast+sampling+of+diffusion+models%2E+arXiv+preprint+arXiv%3A2309%2E050)
69. Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiao Liu, Zhuofan Zong, Yu Liu, and Ping Luo. Raphael: Text-to-image generation via large mixture of diffusion paths. In arXiv, 2023b. [find](https://scholar.google.com/scholar?q=Zeyue+Xue%2C+Guanglu+Song%2C+Qiushan+Guo%2C+Boxiao+Liu%2C+Zhuofan+Zong%2C+Yu+Liu%2C+and+Ping+Luo%2E+Raphael%3A+Text%2Dto%2Dimage+generation+via+large+mixture+of+diffusion+paths%2E+In+arXiv%2C+2023b%2E)
70. Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In ICCV, 2021. [find](https://scholar.google.com/scholar?q=Li+Yuan%2C+Yunpeng+Chen%2C+Tao+Wang%2C+Weihao+Yu%2C+Yujun+Shi%2C+Zi%2DHang+Jiang%2C+Francis+EH+Tay%2C+Jiashi+Feng%2C+and+Shuicheng+Yan%2E+Tokens%2Dto%2Dtoken+vit%3A+Training+vision+transformers+from+scratch+on+imagenet%2E+In+ICC)
71. Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023. [find](https://scholar.google.com/scholar?q=Lvmin+Zhang%2C+Anyi+Rao%2C+and+Maneesh+Agrawala%2E+Adding+conditional+control+to+text%2Dto%2Dimage+diffusion+models%2E+In+ICCV%2C+2023%2E)
72. Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In ICCV, 2021. [find](https://scholar.google.com/scholar?q=Hengshuang+Zhao%2C+Li+Jiang%2C+Jiaya+Jia%2C+Philip+HS+Torr%2C+and+Vladlen+Koltun%2E+Point+transformer%2E+In+ICCV%2C+2021%2E)
73. Hongkai Zheng, Weili Nie, Arash Vahdat, and Anima Anandkumar. Fast training of diffusion models with masked transformers. In arXiv, 2023. [find](https://scholar.google.com/scholar?q=Hongkai+Zheng%2C+Weili+Nie%2C+Arash+Vahdat%2C+and+Anima+Anandkumar%2E+Fast+training+of+diffusion+models+with+masked+transformers%2E+In+arXiv%2C+2023%2E)
74. Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In CVPR, 2021. [find](https://scholar.google.com/scholar?q=Sixiao+Zheng%2C+Jiachen+Lu%2C+Hengshuang+Zhao%2C+Xiatian+Zhu%2C+Zekun+Luo%2C+Yabiao+Wang%2C+Yanwei+Fu%2C+Jianfeng+Feng%2C+Tao+Xiang%2C+Philip+HS+Torr%2C+et+al%2E+Rethinking+semantic+segmentation+from+a+sequence%2Dto%2Dsequence)
75. Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Zihang Jiang, Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. In arXiv, 2021. [find](https://scholar.google.com/scholar?q=Daquan+Zhou%2C+Bingyi+Kang%2C+Xiaojie+Jin%2C+Linjie+Yang%2C+Xiaochen+Lian%2C+Zihang+Jiang%2C+Qibin+Hou%2C+and+Jiashi+Feng%2E+Deepvit%3A+Towards+deeper+vision+transformer%2E+In+arXiv%2C+2021%2E)
76. Daquan Zhou, Zhiding Yu, Enze Xie, Chaowei Xiao, Animashree Anandkumar, Jiashi Feng, and Jose M Alvarez. Understanding the robustness in vision transformers. In International Conference on Machine Learning, pp. 27378–27394. PMLR, 2022. [find](https://scholar.google.com/scholar?q=Daquan+Zhou%2C+Zhiding+Yu%2C+Enze+Xie%2C+Chaowei+Xiao%2C+Animashree+Anandkumar%2C+Jiashi+Feng%2C+and+Jose+M+Alvarez%2E+Understanding+the+robustness+in+vision+transformers%2E+In+International+Conference+on+Machine+Lea)

*Updated 2 November 2026*