--- title: "Zero-Shot Text-to-Image Generation — Ramesh 2021" description: "The paper asks whether text-to-image synthesis can advance through scale rather than through specialized architectures, auxiliary losses or extra…" authors: ["Aditya Ramesh", "Mikhail Pavlov", "Gabriel Goh", "Scott Gray", "Chelsea Voss", "Alec Radford", "Mark Chen", "Ilya Sutskever"] year: 2021 venue: "arXiv:2102.12092" source: "https://arxiv.org/abs/2102.12092" doi: "10.48550/arxiv.2102.12092" arxiv: "2102.12092" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/ramesh-2021-zero-shot-text-image-generation/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ramesh-2021-zero-shot-text-image-generation/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/6/) / Zero-Shot Text-to-Image Generation # Zero-Shot Text-to-Image Generation **Authors:** [Aditya Ramesh](https://md.research.elamur.ai/free/en/authors/aditya-ramesh/), Mikhail Pavlov, [Gabriel Goh](https://md.research.elamur.ai/free/en/authors/gabriel-goh/), Scott Gray, Chelsea Voss, [Alec Radford](https://md.research.elamur.ai/free/en/authors/alec-radford/), [Mark Chen](https://md.research.elamur.ai/free/en/authors/mark-chen/), [Ilya Sutskever](https://md.research.elamur.ai/free/en/authors/ilya-sutskever/) · **Year:** 2021 · **Venue:** arXiv:2102.12092 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2102.12092) · [PDF](https://arxiv.org/pdf/2102.12092) · [DOI 10.48550/arxiv.2102.12092](https://doi.org/10.48550/arxiv.2102.12092) · [arXiv 2102.12092](https://arxiv.org/abs/2102.12092) · [OpenAlex](https://openalex.org/W3129576130) **Citations:** 1067 citations (openalex-snapshot · 2026-10-14) · [356](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary The paper asks whether text-to-image synthesis can advance through scale rather than through specialized architectures, auxiliary losses or extra annotations such as segmentation masks. The authors (OpenAI) use a two-stage design. First, a discrete VAE (dVAE) compresses each 256×256 picture into a 32×32 grid of codes. Second, a 12-billion-parameter sparse transformer learns BPE caption tokens and those image codes together as one autoregressive sequence. Training uses roughly 250 million image–text pairs gathered from the web, including a filtered portion of YFCC100M. At sampling time, many candidates are generated and a contrastive model reranks them. Without ever training on MS-COCO captions, the system reaches an FID close to the best dataset-specific models. Human raters prefer its outputs over DF-GAN by a wide margin. It also shows unplanned abilities: combining unusual concepts, rendering text, and simple language-guided image-to-image translation. On narrow domains such as CUB it performs much worse. The work suggests that data and model scale can substitute for task-specific engineering in generative modeling. ## Key points - **dVAE tokenizer:** a codebook of 8192 entries turns images into 1024 tokens, which shrinks the transformer context about 192-fold. It is trained with a gumbel-softmax relaxation and a KL weight of β = 6.6. - **Transformer:** - 64 self-attention layers. - Captions are capped at 256 BPE tokens from a 16,384-word vocabulary. - The loss weights the image part at 7/8 and the text part at 1/8. - **Human evaluation against DF-GAN on MS-COCO:** its samples won the majority vote for caption fit in 93% of cases and for realism in 90%. - **FID results:** - On MS-COCO, it is within about 2 points of the leading prior method, and about 6 points better when a slight blur (radius 1) is applied. - On CUB, it trails by nearly 40 points. - Reranking gains level off beyond about 32 samples. - **Code:** released at github.com/openai/DALL-E. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/) ## Cited by Cited by 356 papers of our library cite it (of the 140673 whose bibliographies we read); 30 of them have a page here, 326 not yet. - [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://md.research.elamur.ai/free/en/papers/bang-2023-multitask-multilingual-multimodal-evaluation-chatgpt/) - [Acquired TASTE: Multimodal Stance Detection with Textual and Structural Embeddings](https://md.research.elamur.ai/free/en/papers/barel-2025-acquired-taste-multimodal-stance-detection/) - [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/) - [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/) - [8-bit Optimizers via Block-wise Quantization](https://md.research.elamur.ai/free/en/papers/dettmers-2021-8-bit-optimizers-block-wise/) - [CogView: Mastering Text-to-Image Generation via Transformers](https://md.research.elamur.ai/free/en/papers/ding-2021-cogview-mastering-text-image/) - [Gemini: A Family of Highly Capable Multimodal Models](https://md.research.elamur.ai/free/en/papers/gemini-2023-team-gemini-family-highly-capable-multimodal/) - [Bridging the Gap between Synthetic and Authentic Images for Multimodal Machine Translation](https://md.research.elamur.ai/free/en/papers/guo-2023-bridging-gap-between-synthetic-authentic/) - [Masked Autoencoders Are Scalable Vision Learners](https://md.research.elamur.ai/free/en/papers/he-2021-masked-autoencoders-scalable-vision-learners/) - [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://md.research.elamur.ai/free/en/papers/huang-2022-layoutlmv3-pre-training-document-ai/) - [ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education](https://md.research.elamur.ai/free/en/papers/kasneci-2023-chatgpt-good-opportunities-challenges-large/) - [Segment Anything](https://md.research.elamur.ai/free/en/papers/kirillov-2023-segment-anything/) - [Text and Code Embeddings by Contrastive Pre-Training](https://md.research.elamur.ai/free/en/papers/neelakantan-2022-text-code-embeddings-contrastive-pre/) - [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://md.research.elamur.ai/free/en/papers/podell-2023-sdxl-improving-latent-diffusion-models/) - [Formal Mathematics Statement Curriculum Learning](https://md.research.elamur.ai/free/en/papers/polu-2022-formal-mathematics-statement-curriculum-learning/) - [Assessing the Persuasive Effect of AI-Generated Image Support of Arguments](https://md.research.elamur.ai/free/en/papers/quadras-2026-assessing-persuasive-effect-ai-generated/) - [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://md.research.elamur.ai/free/en/papers/ramesh-2022-hierarchical-text-conditional-image-generation/) - [Exploring the Relationship between LLM Hallucinations and Prompt Linguistic Nuances: Readability, Formality, and Concreteness](https://md.research.elamur.ai/free/en/papers/rawte-2023-exploring-relationship-between-llm-hallucinations/) - [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://md.research.elamur.ai/free/en/papers/saharia-2022-photorealistic-text-image-diffusion-models/) - [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://md.research.elamur.ai/free/en/papers/schuhmann-2021-laion-400m-open-dataset-clip/) - [LAION-5B: An open large-scale dataset for training next generation image-text models](https://md.research.elamur.ai/free/en/papers/schuhmann-2022-laion-5b-open-large-scale/) - [ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation](https://md.research.elamur.ai/free/en/papers/sun-2021-ernie-3-0-large/) - [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://md.research.elamur.ai/free/en/papers/team-2024-chameleon-mixed-modal-early-fusion/) - [MUSIA: Multilingual Story Illustration Corpus for Cross-Cultural Alignment and Generation](https://md.research.elamur.ai/free/en/papers/tewari-2026-musia-multilingual-story-illustration-corpus/) - [Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models](https://md.research.elamur.ai/free/en/papers/tirumala-2022-memorization-overfitting-analyzing-training-dynamics/) - [SimVLM: Simple Visual Language Model Pretraining with Weak Supervision](https://md.research.elamur.ai/free/en/papers/wang-2021-simvlm-simple-visual-language-model/) - [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](https://md.research.elamur.ai/free/en/papers/wang-2022-ofa-unifying-architectures-tasks-modalities/) - [视觉语言预训练综述](https://md.research.elamur.ai/free/en/papers/yin-2022-vision-language-pre/) - [GLM-130B: An Open Bilingual Pre-trained Model](https://md.research.elamur.ai/free/en/papers/zeng-2022-glm-130b-open-bilingual-pre/) - [JudgeLM: Fine-tuned Large Language Models are Scalable Judges](https://md.research.elamur.ai/free/en/papers/zhu-2023-judgelm-fine-tuned-large-language/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{ramesh2021zero, title = {{Zero-Shot Text-to-Image Generation}}, author = {Ramesh, Aditya and Pavlov, Mikhail and Goh, Gabriel and Gray, Scott and Voss, Chelsea and Radford, Alec and Chen, Mark and Sutskever, Ilya}, year = {2021}, doi = {10.48550/arxiv.2102.12092}, eprint = {2102.12092}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2102.12092} } ``` - [← previous ZeRO-Offload: Democratizing Billion-Scale Model Training](https://md.research.elamur.ai/free/en/papers/ren-2021-zero-offload-democratizing-billion-scale/) - [next → “We Don’t Understand this Present”: Shimon Adaf’s Shadrach and the Possibilities of Hebrew Language](https://md.research.elamur.ai/free/en/papers/baroukh-2021-dont-understand-this-present-shimon/) - [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/6/) ## References 57 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pp. 265–283, 2016. [find](https://scholar.google.com/scholar?q=Abadi%2C+M%2E%2C+Barham%2C+P%2E%2C+Chen%2C+J%2E%2C+Chen%2C+Z%2E%2C+Davis%2C+A%2E%2C+Dean%2C+J%2E%2C+Devin%2C+M%2E%2C+Ghemawat%2C+S%2E%2C+Irving%2C+G%2E%2C+Isard%2C+M%2E%2C+et+al%2E+Tensorflow%3A+A+system+for+large%2Dscale+machine+learning%2E+In+12th+%7BUSENIX%7D+symposium) 2. Andreas, J., Klein, D., and Levine, S. Learning with latent language. arXiv preprint arXiv:1711.00482, 2017. [find](https://scholar.google.com/scholar?q=Andreas%2C+J%2E%2C+Klein%2C+D%2E%2C+and+Levine%2C+S%2E+Learning+with+latent+language%2E+arXiv+preprint+arXiv%3A1711%2E00482%2C+2017%2E) 3. Bengio, Y., Léonard, N., and Courville, A. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013. [find](https://scholar.google.com/scholar?q=Bengio%2C+Y%2E%2C+L%C3%A9onard%2C+N%2E%2C+and+Courville%2C+A%2E+Estimating+or+propagating+gradients+through+stochastic+neurons+for+conditional+computation%2E+arXiv+preprint+arXiv%3A1308%2E3432%2C+2013%2E) 4. Bowman, S. R., Vilnis, L., Vinyals, O., Dai, A. M., Jozefowicz, R., and Bengio, S. Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349, 2015. [find](https://scholar.google.com/scholar?q=Bowman%2C+S%2E+R%2E%2C+Vilnis%2C+L%2E%2C+Vinyals%2C+O%2E%2C+Dai%2C+A%2E+M%2E%2C+Jozefowicz%2C+R%2E%2C+and+Bengio%2C+S%2E+Generating+sentences+from+a+continuous+space%2E+arXiv+preprint+arXiv%3A1511%2E06349%2C+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/bowman-2016-generating-sentences-continuous-space/) 5. Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D., and Sutskever, I. Generative pretraining from pixels. In International Conference on Machine Learning, pp. 1691–1703. PMLR, 2020. [find](https://scholar.google.com/scholar?q=Chen%2C+M%2E%2C+Radford%2C+A%2E%2C+Child%2C+R%2E%2C+Wu%2C+J%2E%2C+Jun%2C+H%2E%2C+Luan%2C+D%2E%2C+and+Sutskever%2C+I%2E+Generative+pretraining+from+pixels%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+1691%E2%80%931703%2E+PMLR%2C+2020%2E) 6. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. [find](https://scholar.google.com/scholar?q=Deng%2C+J%2E%2C+Dong%2C+W%2E%2C+Socher%2C+R%2E%2C+Li%2C+L%2E%2DJ%2E%2C+Li%2C+K%2E%2C+and+Fei%2DFei%2C+L%2E+Imagenet%3A+A+large%2Dscale+hierarchical+image+database%2E+In+2009+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+pp%2E+248%E2%80%93255%2E) 7. Dhariwal, P., Jun, H., Payne, C., Kim, J. W., Radford, A., and Sutskever, I. Jukebox: A generative model for music. arXiv preprint arXiv:2005.00341, 2020. — [arXiv:2005.00341](https://arxiv.org/abs/2005.00341) [find](https://scholar.google.com/scholar?q=Dhariwal%2C+P%2E%2C+Jun%2C+H%2E%2C+Payne%2C+C%2E%2C+Kim%2C+J%2E+W%2E%2C+Radford%2C+A%2E%2C+and+Sutskever%2C+I%2E+Jukebox%3A+A+generative+model+for+music%2E+arXiv+preprint+arXiv%3A2005%2E00341%2C+2020%2E+%E2%80%94+%5BarXiv%3A2005%2E00341%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F20) 8. Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial networks. arXiv preprint arXiv:1406.2661, 2014. [find](https://scholar.google.com/scholar?q=Goodfellow%2C+I%2E+J%2E%2C+Pouget%2DAbadie%2C+J%2E%2C+Mirza%2C+M%2E%2C+Xu%2C+B%2E%2C+Warde%2DFarley%2C+D%2E%2C+Ozair%2C+S%2E%2C+Courville%2C+A%2E%2C+and+Bengio%2C+Y%2E+Generative+adversarial+networks%2E+arXiv+preprint+arXiv%3A1406%2E2661%2C+2014%2E) 9. Greff, K., van Steenkiste, S., and Schmidhuber, J. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05208, 2020. — [arXiv:2012.05208](https://arxiv.org/abs/2012.05208) [find](https://scholar.google.com/scholar?q=Greff%2C+K%2E%2C+van+Steenkiste%2C+S%2E%2C+and+Schmidhuber%2C+J%2E+On+the+binding+problem+in+artificial+neural+networks%2E+arXiv+preprint+arXiv%3A2012%2E05208%2C+2020%2E+%E2%80%94+%5BarXiv%3A2012%2E05208%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2012%2E05208%29) 10. Gregor, K., Danihelka, I., Graves, A., Rezende, D., and Wierstra, D. Draw: A recurrent neural network for image generation. In International Conference on Machine Learning, pp. 1462–1471. PMLR, 2015. [find](https://scholar.google.com/scholar?q=Gregor%2C+K%2E%2C+Danihelka%2C+I%2E%2C+Graves%2C+A%2E%2C+Rezende%2C+D%2E%2C+and+Wierstra%2C+D%2E+Draw%3A+A+recurrent+neural+network+for+image+generation%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+1462%E2%80%931471%2E+PMLR%2C+2015%2E) 11. He, K., Zhang, X., Ren, S., and Sun, J. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630–645. Springer, 2016. [find](https://scholar.google.com/scholar?q=He%2C+K%2E%2C+Zhang%2C+X%2E%2C+Ren%2C+S%2E%2C+and+Sun%2C+J%2E+Identity+mappings+in+deep+residual+networks%2E+In+European+conference+on+computer+vision%2C+pp%2E+630%E2%80%93645%2E+Springer%2C+2016%2E) 12. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. arXiv preprint arXiv:1706.08500, 2017. — [arXiv:1706.08500](https://arxiv.org/abs/1706.08500) [find](https://scholar.google.com/scholar?q=Heusel%2C+M%2E%2C+Ramsauer%2C+H%2E%2C+Unterthiner%2C+T%2E%2C+Nessler%2C+B%2E%2C+and+Hochreiter%2C+S%2E+Gans+trained+by+a+two+time%2Dscale+update+rule+converge+to+a+local+nash+equilibrium%2E+arXiv+preprint+arXiv%3A1706%2E08500%2C+2017%2E+%E2%80%94+%5B) 13. Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. beta-vae: Learning basic visual concepts with a constrained variational framework. 2016. [find](https://scholar.google.com/scholar?q=Higgins%2C+I%2E%2C+Matthey%2C+L%2E%2C+Pal%2C+A%2E%2C+Burgess%2C+C%2E%2C+Glorot%2C+X%2E%2C+Botvinick%2C+M%2E%2C+Mohamed%2C+S%2E%2C+and+Lerchner%2C+A%2E+beta%2Dvae%3A+Learning+basic+visual+concepts+with+a+constrained+variational+framework%2E+2016%2E) 14. Isola, P., Zhu, J.-Y., Zhou, T., and Efros, A. A. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125–1134, 2017. [find](https://scholar.google.com/scholar?q=Isola%2C+P%2E%2C+Zhu%2C+J%2E%2DY%2E%2C+Zhou%2C+T%2E%2C+and+Efros%2C+A%2E+A%2E+Image%2Dto%2Dimage+translation+with+conditional+adversarial+networks%2E+In+Proceedings+of+the+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+pp) 15. Jang, E., Gu, S., and Poole, B. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144, 2016. [find](https://scholar.google.com/scholar?q=Jang%2C+E%2E%2C+Gu%2C+S%2E%2C+and+Poole%2C+B%2E+Categorical+reparameterization+with+gumbel%2Dsoftmax%2E+arXiv+preprint+arXiv%3A1611%2E01144%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/jang-2016-categorical-reparameterization-gumbel-softmax/) 16. Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. [find](https://scholar.google.com/scholar?q=Kingma%2C+D%2E+P%2E+and+Ba%2C+J%2E+Adam%3A+A+method+for+stochastic+optimization%2E+arXiv+preprint+arXiv%3A1412%2E6980%2C+2014%2E) 17. Child, R., Gray, S., Radford, A., and Sutskever, I. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019. [find](https://scholar.google.com/scholar?q=Child%2C+R%2E%2C+Gray%2C+S%2E%2C+Radford%2C+A%2E%2C+and+Sutskever%2C+I%2E+Generating+long+sequences+with+sparse+transformers%2E+arXiv+preprint+arXiv%3A1904%2E10509%2C+2019%2E) 18. Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013. [find](https://scholar.google.com/scholar?q=Kingma%2C+D%2E+P%2E+and+Welling%2C+M%2E+Auto%2Dencoding+variational+bayes%2E+arXiv+preprint+arXiv%3A1312%2E6114%2C+2013%2E) [our card](https://md.research.elamur.ai/free/en/papers/kingma-2013-auto-encoding-variational-bayes/) 19. Cho, J., Lu, J., Schwenk, D., Hajishirzi, H., and Kembhavi, A. X-lxmert: Paint, caption and answer questions with multi-modal transformers. arXiv preprint arXiv:2009.11278, 2020. [find](https://scholar.google.com/scholar?q=Cho%2C+J%2E%2C+Lu%2C+J%2E%2C+Schwenk%2C+D%2E%2C+Hajishirzi%2C+H%2E%2C+and+Kembhavi%2C+A%2E+X%2Dlxmert%3A+Paint%2C+caption+and+answer+questions+with+multi%2Dmodal+transformers%2E+arXiv+preprint+arXiv%3A2009%2E11278%2C+2020%2E) 20. Koh, J. Y., Baldridge, J., Lee, H., and Yang, Y. Text-to-image generation grounded by fine-grained user attention. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 237–246, 2021. [find](https://scholar.google.com/scholar?q=Koh%2C+J%2E+Y%2E%2C+Baldridge%2C+J%2E%2C+Lee%2C+H%2E%2C+and+Yang%2C+Y%2E+Text%2Dto%2Dimage+generation+grounded+by+fine%2Dgrained+user+attention%2E+In+Proceedings+of+the+IEEE%2FCVF+Winter+Conference+on+Applications+of+Computer+Vision%2C+) 21. Köster, U., Webb, T. J., Wang, X., Nassar, M., Bansal, A. K., Constable, W. H., Elibol, O. H., Gray, S., Hall, S., Hornof, L., et al. Flexpoint: An adaptive numerical format for efficient training of deep neural networks. arXiv preprint arXiv:1711.02213, 2017. — [arXiv:1711.02213](https://arxiv.org/abs/1711.02213) [find](https://scholar.google.com/scholar?q=K%C3%B6ster%2C+U%2E%2C+Webb%2C+T%2E+J%2E%2C+Wang%2C+X%2E%2C+Nassar%2C+M%2E%2C+Bansal%2C+A%2E+K%2E%2C+Constable%2C+W%2E+H%2E%2C+Elibol%2C+O%2E+H%2E%2C+Gray%2C+S%2E%2C+Hall%2C+S%2E%2C+Hornof%2C+L%2E%2C+et+al%2E+Flexpoint%3A+An+adaptive+numerical+format+for+efficient+training+of+) 22. LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. [find](https://scholar.google.com/scholar?q=LeCun%2C+Y%2E%2C+Bottou%2C+L%2E%2C+Bengio%2C+Y%2E%2C+and+Haffner%2C+P%2E+Gradient%2Dbased+learning+applied+to+document+recognition%2E+Proceedings+of+the+IEEE%2C+86%2811%29%3A2278%E2%80%932324%2C+1998%2E) 23. Li, W., Zhang, P., Zhang, L., Huang, Q., He, X., Lyu, S., and Gao, J. Object-driven text-to-image synthesis via adversarial training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12174–12182, 2019. [find](https://scholar.google.com/scholar?q=Li%2C+W%2E%2C+Zhang%2C+P%2E%2C+Zhang%2C+L%2E%2C+Huang%2C+Q%2E%2C+He%2C+X%2E%2C+Lyu%2C+S%2E%2C+and+Gao%2C+J%2E+Object%2Ddriven+text%2Dto%2Dimage+synthesis+via+adversarial+training%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+P) 24. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740–755. Springer, 2014. [find](https://scholar.google.com/scholar?q=Lin%2C+T%2E%2DY%2E%2C+Maire%2C+M%2E%2C+Belongie%2C+S%2E%2C+Hays%2C+J%2E%2C+Perona%2C+P%2E%2C+Ramanan%2C+D%2E%2C+Doll%C3%A1r%2C+P%2E%2C+and+Zitnick%2C+C%2E+L%2E+Microsoft+coco%3A+Common+objects+in+context%2E+In+European+conference+on+computer+vision%2C+pp%2E+740%E2%80%93755) [our card](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/) 25. Liu, L., Liu, X., Gao, J., Chen, W., and Han, J. Understanding the difficulty of training transformers. arXiv preprint arXiv:2004.08249, 2020. [find](https://scholar.google.com/scholar?q=Liu%2C+L%2E%2C+Liu%2C+X%2E%2C+Gao%2C+J%2E%2C+Chen%2C+W%2E%2C+and+Han%2C+J%2E+Understanding+the+difficulty+of+training+transformers%2E+arXiv+preprint+arXiv%3A2004%2E08249%2C+2020%2E) 26. Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. [find](https://scholar.google.com/scholar?q=Loshchilov%2C+I%2E+and+Hutter%2C+F%2E+Decoupled+weight+decay+regularization%2E+arXiv+preprint+arXiv%3A1711%2E05101%2C+2017%2E) 27. Maddison, C. J., Mnih, A., and Teh, Y. W. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00712, 2016. [find](https://scholar.google.com/scholar?q=Maddison%2C+C%2E+J%2E%2C+Mnih%2C+A%2E%2C+and+Teh%2C+Y%2E+W%2E+The+concrete+distribution%3A+A+continuous+relaxation+of+discrete+random+variables%2E+arXiv+preprint+arXiv%3A1611%2E00712%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/maddison-2016-concrete-distribution-continuous-relaxation-discrete/) 28. Mansimov, E., Parisotto, E., Ba, J. L., and Salakhutdinov, R. Generating images from captions with attention. arXiv preprint arXiv:1511.02793, 2015. — [arXiv:1511.02793](https://arxiv.org/abs/1511.02793) [find](https://scholar.google.com/scholar?q=Mansimov%2C+E%2E%2C+Parisotto%2C+E%2E%2C+Ba%2C+J%2E+L%2E%2C+and+Salakhutdinov%2C+R%2E+Generating+images+from+captions+with+attention%2E+arXiv+preprint+arXiv%3A1511%2E02793%2C+2015%2E+%E2%80%94+%5BarXiv%3A1511%2E02793%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1511%2E027) 29. Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., Ginsburg, B., Houston, M., Kuchaiev, O., Venkatesh, G., et al. Mixed precision training. arXiv preprint arXiv:1710.03740, 2017. [find](https://scholar.google.com/scholar?q=Micikevicius%2C+P%2E%2C+Narang%2C+S%2E%2C+Alben%2C+J%2E%2C+Diamos%2C+G%2E%2C+Elsen%2C+E%2E%2C+Garcia%2C+D%2E%2C+Ginsburg%2C+B%2E%2C+Houston%2C+M%2E%2C+Kuchaiev%2C+O%2E%2C+Venkatesh%2C+G%2E%2C+et+al%2E+Mixed+precision+training%2E+arXiv+preprint+arXiv%3A1710%2E03740%2C+20) 30. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. 2021. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Kim%2C+J%2E+W%2E%2C+Hallacy%2C+C%2E%2C+Ramesh%2C+A%2E%2C+Goh%2C+G%2E%2C+Agarwal%2C+S%2E%2C+Sastry%2C+G%2E%2C+Askell%2C+A%2E%2C+Mishkin%2C+P%2E%2C+Clark%2C+J%2E%2C+Krueger%2C+G%2E%2C+and+Sutskever%2C+I%2E+Learning+transferable+visual+models+from+natural+) [our card](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/) 31. Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. Zero: Memory optimization towards training a trillion parameter models. arXiv preprint arXiv:1910.02054, 2019. [find](https://scholar.google.com/scholar?q=Rajbhandari%2C+S%2E%2C+Rasley%2C+J%2E%2C+Ruwase%2C+O%2E%2C+and+He%2C+Y%2E+Zero%3A+Memory+optimization+towards+training+a+trillion+parameter+models%2E+arXiv+preprint+arXiv%3A1910%2E02054%2C+2019%2E) 32. Razavi, A., Oord, A. v. d., and Vinyals, O. Generating diverse high-fidelity images with vq-vae-2. arXiv preprint arXiv:1906.00446, 2019. — [arXiv:1906.00446](https://arxiv.org/abs/1906.00446) [find](https://scholar.google.com/scholar?q=Razavi%2C+A%2E%2C+Oord%2C+A%2E+v%2E+d%2E%2C+and+Vinyals%2C+O%2E+Generating+diverse+high%2Dfidelity+images+with+vq%2Dvae%2D2%2E+arXiv+preprint+arXiv%3A1906%2E00446%2C+2019%2E+%E2%80%94+%5BarXiv%3A1906%2E00446%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1906%2E00446%29) 33. Reed, S., Akata, Z., Mohan, S., Tenka, S., Schiele, B., and Lee, H. Learning what and where to draw. arXiv preprint arXiv:1610.02454, 2016a. — [arXiv:1610.02454](https://arxiv.org/abs/1610.02454) [find](https://scholar.google.com/scholar?q=Reed%2C+S%2E%2C+Akata%2C+Z%2E%2C+Mohan%2C+S%2E%2C+Tenka%2C+S%2E%2C+Schiele%2C+B%2E%2C+and+Lee%2C+H%2E+Learning+what+and+where+to+draw%2E+arXiv+preprint+arXiv%3A1610%2E02454%2C+2016a%2E+%E2%80%94+%5BarXiv%3A1610%2E02454%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1610%2E02454%29) 34. Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., and Lee, H. Generative adversarial text to image synthesis. In International Conference on Machine Learning, pp. 1060–1069. PMLR, 2016b. [find](https://scholar.google.com/scholar?q=Reed%2C+S%2E%2C+Akata%2C+Z%2E%2C+Yan%2C+X%2E%2C+Logeswaran%2C+L%2E%2C+Schiele%2C+B%2E%2C+and+Lee%2C+H%2E+Generative+adversarial+text+to+image+synthesis%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+1060%E2%80%931069%2E+PMLR%2C+2016b%2E) 35. Rezende, D. J., Mohamed, S., and Wierstra, D. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pp. 1278–1286. PMLR, 2014. [find](https://scholar.google.com/scholar?q=Rezende%2C+D%2E+J%2E%2C+Mohamed%2C+S%2E%2C+and+Wierstra%2C+D%2E+Stochastic+backpropagation+and+approximate+inference+in+deep+generative+models%2E+In+International+conference+on+machine+learning%2C+pp%2E+1278%E2%80%931286%2E+PMLR%2C+2014) [our card](https://md.research.elamur.ai/free/en/papers/rezende-2014-stochastic-backpropagation-approximate-inference-deep/) 36. Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for training gans. arXiv preprint arXiv:1606.03498, 2016. [find](https://scholar.google.com/scholar?q=Salimans%2C+T%2E%2C+Goodfellow%2C+I%2E%2C+Zaremba%2C+W%2E%2C+Cheung%2C+V%2E%2C+Radford%2C+A%2E%2C+and+Chen%2C+X%2E+Improved+techniques+for+training+gans%2E+arXiv+preprint+arXiv%3A1606%2E03498%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/salimans-2016-improved-techniques-training-gans/) 37. Salimans, T., Karpathy, A., Chen, X., and Kingma, D. P. Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517, 2017. — [arXiv:1701.05517](https://arxiv.org/abs/1701.05517) [find](https://scholar.google.com/scholar?q=Salimans%2C+T%2E%2C+Karpathy%2C+A%2E%2C+Chen%2C+X%2E%2C+and+Kingma%2C+D%2E+P%2E+Pixelcnn%2B%2B%3A+Improving+the+pixelcnn+with+discretized+logistic+mixture+likelihood+and+other+modifications%2E+arXiv+preprint+arXiv%3A1701%2E05517%2C+2017%2E+) 38. Sennrich, R., Haddow, B., and Birch, A. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015. [find](https://scholar.google.com/scholar?q=Sennrich%2C+R%2E%2C+Haddow%2C+B%2E%2C+and+Birch%2C+A%2E+Neural+machine+translation+of+rare+words+with+subword+units%2E+arXiv+preprint+arXiv%3A1508%2E07909%2C+2015%2E) 39. Nguyen, A., Clune, J., Bengio, Y., Dosovitskiy, A., and Yosinski, J. Plug & play generative networks: Conditional iterative generation of images in latent space. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4467–4477, 2017. [find](https://scholar.google.com/scholar?q=Nguyen%2C+A%2E%2C+Clune%2C+J%2E%2C+Bengio%2C+Y%2E%2C+Dosovitskiy%2C+A%2E%2C+and+Yosinski%2C+J%2E+Plug+%26+play+generative+networks%3A+Conditional+iterative+generation+of+images+in+latent+space%2E+In+Proceedings+of+the+IEEE+Conference+) 40. Sharma, P., Ding, N., Goodman, S., and Soricut, R. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556– 2565, 2018. [find](https://scholar.google.com/scholar?q=Sharma%2C+P%2E%2C+Ding%2C+N%2E%2C+Goodman%2C+S%2E%2C+and+Soricut%2C+R%2E+Conceptual+captions%3A+A+cleaned%2C+hypernymed%2C+image+alt%2Dtext+dataset+for+automatic+image+captioning%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+A) [our card](https://md.research.elamur.ai/free/en/papers/sharma-2018-conceptual-captions-cleaned-hypernymed-image/) 41. Oord, A. v. d., Vinyals, O., and Kavukcuoglu, K. Neural discrete representation learning. arXiv preprint arXiv:1711.00937, 2017. — [arXiv:1711.00937](https://arxiv.org/abs/1711.00937) [find](https://scholar.google.com/scholar?q=Oord%2C+A%2E+v%2E+d%2E%2C+Vinyals%2C+O%2E%2C+and+Kavukcuoglu%2C+K%2E+Neural+discrete+representation+learning%2E+arXiv+preprint+arXiv%3A1711%2E00937%2C+2017%2E+%E2%80%94+%5BarXiv%3A1711%2E00937%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1711%2E00937%29) 42. Smolensky, P. Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial intelligence, 46(1-2):159–216, 1990. [find](https://scholar.google.com/scholar?q=Smolensky%2C+P%2E+Tensor+product+variable+binding+and+the+representation+of+symbolic+structures+in+connectionist+systems%2E+Artificial+intelligence%2C+46%281%2D2%29%3A159%E2%80%93216%2C+1990%2E) 43. Provilkov, I., Emelianenko, D., and Voita, E. Bpe-dropout: Simple and effective subword regularization. arXiv preprint arXiv:1910.13267, 2019. [find](https://scholar.google.com/scholar?q=Provilkov%2C+I%2E%2C+Emelianenko%2C+D%2E%2C+and+Voita%2C+E%2E+Bpe%2Ddropout%3A+Simple+and+effective+subword+regularization%2E+arXiv+preprint+arXiv%3A1910%2E13267%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/provilkov-2020-bpe-dropout-simple-effective-subword/) 44. Sun, C., Shrivastava, A., Singh, S., and Gupta, A. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pp. 843–852, 2017. [find](https://scholar.google.com/scholar?q=Sun%2C+C%2E%2C+Shrivastava%2C+A%2E%2C+Singh%2C+S%2E%2C+and+Gupta%2C+A%2E+Revisiting+unreasonable+effectiveness+of+data+in+deep+learning+era%2E+In+Proceedings+of+the+IEEE+international+conference+on+computer+vision%2C+pp%2E+843%E2%80%938) 45. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. 2019. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Wu%2C+J%2E%2C+Child%2C+R%2E%2C+Luan%2C+D%2E%2C+Amodei%2C+D%2E%2C+and+Sutskever%2C+I%2E+Language+models+are+unsupervised+multitask+learners%2E+2019%2E) 46. Sun, X., Wang, N., Chen, C.-Y., Ni, J., Agrawal, A., Cui, X., Venkataramani, S., El Maghraoui, K., Srinivasan, V. V., and Gopalakrishnan, K. Ultra-low precision 4-bit training of deep neural networks. Advances in Neural Information Processing Systems, 33, 2020. [find](https://scholar.google.com/scholar?q=Sun%2C+X%2E%2C+Wang%2C+N%2E%2C+Chen%2C+C%2E%2DY%2E%2C+Ni%2C+J%2E%2C+Agrawal%2C+A%2E%2C+Cui%2C+X%2E%2C+Venkataramani%2C+S%2E%2C+El+Maghraoui%2C+K%2E%2C+Srinivasan%2C+V%2E+V%2E%2C+and+Gopalakrishnan%2C+K%2E+Ultra%2Dlow+precision+4%2Dbit+training+of+deep+neural+networks%2E) 47. Tao, M., Tang, H., Wu, S., Sebe, N., Wu, F., and Jing, X.-Y. Df-gan: Deep fusion generative adversarial networks for text-to-image synthesis. arXiv preprint arXiv:2008.05865, 2020. — [arXiv:2008.05865](https://arxiv.org/abs/2008.05865) [find](https://scholar.google.com/scholar?q=Tao%2C+M%2E%2C+Tang%2C+H%2E%2C+Wu%2C+S%2E%2C+Sebe%2C+N%2E%2C+Wu%2C+F%2E%2C+and+Jing%2C+X%2E%2DY%2E+Df%2Dgan%3A+Deep+fusion+generative+adversarial+networks+for+text%2Dto%2Dimage+synthesis%2E+arXiv+preprint+arXiv%3A2008%2E05865%2C+2020%2E+%E2%80%94+%5BarXiv%3A2008%2E05865) 48. Thomee, B., Shamma, D. A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., and Li, L.-J. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):64–73, 2016. [find](https://scholar.google.com/scholar?q=Thomee%2C+B%2E%2C+Shamma%2C+D%2E+A%2E%2C+Friedland%2C+G%2E%2C+Elizalde%2C+B%2E%2C+Ni%2C+K%2E%2C+Poland%2C+D%2E%2C+Borth%2C+D%2E%2C+and+Li%2C+L%2E%2DJ%2E+Yfcc100m%3A+The+new+data+in+multimedia+research%2E+Communications+of+the+ACM%2C+59%282%29%3A64%E2%80%9373%2C+2016%2E) 49. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. arXiv preprint arXiv:1706.03762, 2017. [find](https://scholar.google.com/scholar?q=Vaswani%2C+A%2E%2C+Shazeer%2C+N%2E%2C+Parmar%2C+N%2E%2C+Uszkoreit%2C+J%2E%2C+Jones%2C+L%2E%2C+Gomez%2C+A%2E+N%2E%2C+Kaiser%2C+L%2E%2C+and+Polosukhin%2C+I%2E+Attention+is+all+you+need%2E+arXiv+preprint+arXiv%3A1706%2E03762%2C+2017%2E) 50. Vogels, T., Karimireddy, S. P., and Jaggi, M. Powersgd: Practical low-rank gradient compression for distributed optimization. arXiv preprint arXiv:1905.13727, 2019. — [arXiv:1905.13727](https://arxiv.org/abs/1905.13727) [find](https://scholar.google.com/scholar?q=Vogels%2C+T%2E%2C+Karimireddy%2C+S%2E+P%2E%2C+and+Jaggi%2C+M%2E+Powersgd%3A+Practical+low%2Drank+gradient+compression+for+distributed+optimization%2E+arXiv+preprint+arXiv%3A1905%2E13727%2C+2019%2E+%E2%80%94+%5BarXiv%3A1905%2E13727%5D%28https%3A%2F%2Farxiv%2E) 51. Welinder, P., Branson, S., Mita, T., Wah, C., Schroff, F., Belongie, S., and Perona, P. Caltech-ucsd birds 200. 2010. [find](https://scholar.google.com/scholar?q=Welinder%2C+P%2E%2C+Branson%2C+S%2E%2C+Mita%2C+T%2E%2C+Wah%2C+C%2E%2C+Schroff%2C+F%2E%2C+Belongie%2C+S%2E%2C+and+Perona%2C+P%2E+Caltech%2Ducsd+birds+200%2E+2010%2E) 52. Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., and He, X. Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1316–1324, 2018. [find](https://scholar.google.com/scholar?q=Xu%2C+T%2E%2C+Zhang%2C+P%2E%2C+Huang%2C+Q%2E%2C+Zhang%2C+H%2E%2C+Gan%2C+Z%2E%2C+Huang%2C+X%2E%2C+and+He%2C+X%2E+Attngan%3A+Fine%2Dgrained+text+to+image+generation+with+attentional+generative+adversarial+networks%2E+In+Proceedings+of+the+IEEE+conf) 53. Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., and Metaxas, D. N. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 5907–5915, 2017. [find](https://scholar.google.com/scholar?q=Zhang%2C+H%2E%2C+Xu%2C+T%2E%2C+Li%2C+H%2E%2C+Zhang%2C+S%2E%2C+Wang%2C+X%2E%2C+Huang%2C+X%2E%2C+and+Metaxas%2C+D%2E+N%2E+Stackgan%3A+Text+to+photo%2Drealistic+image+synthesis+with+stacked+generative+adversarial+networks%2E+In+Proceedings+of+the+IEEE) 54. Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., and Metaxas, D. N. Stackgan++: Realistic image synthesis with stacked generative adversarial networks. IEEE transactions on pattern analysis and machine intelligence, 41(8):1947–1962, 2018. [find](https://scholar.google.com/scholar?q=Zhang%2C+H%2E%2C+Xu%2C+T%2E%2C+Li%2C+H%2E%2C+Zhang%2C+S%2E%2C+Wang%2C+X%2E%2C+Huang%2C+X%2E%2C+and+Metaxas%2C+D%2E+N%2E+Stackgan%2B%2B%3A+Realistic+image+synthesis+with+stacked+generative+adversarial+networks%2E+IEEE+transactions+on+pattern+analysis+) 55. Zhu, M., Pan, P., Chen, W., and Yang, Y. Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5802–5810, 2019. def preprocess_image(img, target_res): h, w = tf.shape(img)[0], tf.s… [find](https://scholar.google.com/scholar?q=Zhu%2C+M%2E%2C+Pan%2C+P%2E%2C+Chen%2C+W%2E%2C+and+Yang%2C+Y%2E+Dm%2Dgan%3A+Dynamic+memory+generative+adversarial+networks+for+text%2Dto%2Dimage+synthesis%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Re) 56. Listing 1. TensorFlow (Abadi et al., 2016) image preprocessing code for training dVAE. We use target_res = 256 and channel_count = 3. [find](https://scholar.google.com/scholar?q=Listing+1%2E+TensorFlow+%28Abadi+et+al%2E%2C+2016%29+image+preprocessing+code+for+training+dVAE%2E+We+use+target%5Fres+%3D+256+and+channel%5Fcount+%3D+3%2E) 57. A. Details for Discrete VAE A.1. Architecture [find](https://scholar.google.com/scholar?q=A%2E+Details+for+Discrete+VAE+A%2E1%2E+Architecture)