---
title: "Improved Techniques for Training GANs — Salimans 2016"
description: "Training a GAN means looking for a Nash equilibrium between generator and discriminator."
authors: ["Tim Salimans", "Ian Goodfellow", "Wojciech Zaremba", "Vicki Cheung", "Alec Radford", "Xi Chen"]
year: 2016
venue: "arXiv:1606.03498"
source: "https://arxiv.org/abs/1606.03498"
doi: "10.48550/arxiv.1606.03498"
arxiv: "1606.03498"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/salimans-2016-improved-techniques-training-gans/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/salimans-2016-improved-techniques-training-gans/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/2/) / Improved Techniques for Training GANs

# Improved Techniques for Training GANs

**Authors:** [Tim Salimans](https://md.research.elamur.ai/free/en/authors/tim-salimans/), [Ian Goodfellow](https://md.research.elamur.ai/free/en/authors/ian-goodfellow/), [Wojciech Zaremba](https://md.research.elamur.ai/free/en/authors/wojciech-zaremba/), Vicki Cheung, [Alec Radford](https://md.research.elamur.ai/free/en/authors/alec-radford/), [Xi Chen](https://md.research.elamur.ai/free/en/authors/xi-chen/) · **Year:** 2016 · **Venue:** arXiv:1606.03498 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1606.03498) · [PDF](https://arxiv.org/pdf/1606.03498) · [DOI 10.48550/arxiv.1606.03498](https://doi.org/10.48550/arxiv.1606.03498) · [arXiv 1606.03498](https://arxiv.org/abs/1606.03498) · [OpenAlex](https://openalex.org/W2432004435)

**Citations:** 1231 citations (openalex-snapshot · 2026-10-14) · [189](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Training a GAN means looking for a Nash equilibrium between generator and discriminator. Ordinary gradient descent offers no guarantee of finding one, so training often oscillates or the generator collapses onto a single output. Salimans et al. (OpenAI) propose a set of heuristic fixes:
- **Feature matching:** the generator matches the average intermediate discriminator features of real data.
- **Minibatch discrimination:** the discriminator compares samples within a batch to detect collapse.
- **Historical averaging:** parameters are penalized for drifting from their running mean.
- **One-sided label smoothing.**
- **Virtual batch normalization:** a fixed reference batch is used for normalization.

The authors also propose two ways to measure sample quality: a human "visual Turing test" on Amazon Mechanical Turk and an automatic metric, the Inception score. They evaluate on MNIST, CIFAR-10, SVHN and 128×128 ImageNet (ILSVRC2012). The approach gives state-of-the-art semi-supervised classification on the three smaller datasets. Humans cannot tell MNIST samples from real digits. The ImageNet samples show recognizable animal parts, though with broken anatomy. The work matters because these practical tricks made earlier untrainable GANs workable and gave the field a standard evaluation metric. Code is released at github.com/openai/improved_gan.

## Key points
- **Semi-supervised CIFAR-10 (4000 labels):** the model reaches 18.63% test error. Ladder network gets 20.40% and CatGAN 19.58%. An ensemble of 10 models lowers this to 15.59%.
- **SVHN (1000 labels):** a single model reaches 8.11% error, and the ensemble reaches 5.88%.
- **MTurk test:** annotators correctly labeled 78.7% of real-vs-generated CIFAR-10 images, a 21.3% human error rate. Accuracy fell to 71.4% when only samples in the top 1% by Inception score were shown.
- **Inception score on CIFAR-10:** the full method scores 8.09 against 11.24 for real data. Removing minibatch features causes the largest drop in the ablations, down to 3.87.
- **Which technique to use:** feature matching works best for semi-supervised classification. Minibatch discrimination produces the most visually convincing samples.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

189 papers of our library cite it (of the 140673 whose bibliographies we read); 14 of them have a page here, 175 not yet.

- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://md.research.elamur.ai/free/en/papers/austin-2021-structured-denoising-diffusion-models-discrete/)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://md.research.elamur.ai/free/en/papers/ding-2021-cogview-mastering-text-image/)
- [Classifier-Free Diffusion Guidance](https://md.research.elamur.ai/free/en/papers/ho-2022-classifier-free-diffusion-guidance/)
- [Adversarial Learning for Neural Dialogue Generation](https://md.research.elamur.ai/free/en/papers/li-2017-adversarial-learning-neural-dialogue-generation/)
- [Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfer](https://md.research.elamur.ai/free/en/papers/li-2018-delete-retrieve-generate-simple-approach/)
- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://md.research.elamur.ai/free/en/papers/pillutla-2021-mauve-measuring-gap-between-neural/)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://md.research.elamur.ai/free/en/papers/podell-2023-sdxl-improving-latent-diffusion-models/)
- [Adversarial Connective-exploiting Networks for Implicit Discourse Relation Classification](https://md.research.elamur.ai/free/en/papers/qin-2017-adversarial-connective-exploiting-networks-implicit/)
- [Zero-Shot Text-to-Image Generation](https://md.research.elamur.ai/free/en/papers/ramesh-2021-zero-shot-text-image-generation/)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://md.research.elamur.ai/free/en/papers/song-2019-generative-modeling-estimating-gradients-data/)
- [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](https://md.research.elamur.ai/free/en/papers/wang-2022-ofa-unifying-architectures-tasks-modalities/)
- [Unsupervised Data Augmentation for Consistency Training](https://md.research.elamur.ai/free/en/papers/xie-2019-unsupervised-data-augmentation-consistency-training/)
- [Unsupervised Text Style Transfer using Language Models as Discriminators](https://md.research.elamur.ai/free/en/papers/yang-2018-unsupervised-text-style-transfer-using/)
- [Adversarial Training for Unsupervised Bilingual Lexicon Induction](https://md.research.elamur.ai/free/en/papers/zhang-2017-adversarial-training-unsupervised-bilingual-lexicon/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{salimans2016improved,
  title = {{Improved Techniques for Training GANs}},
  author = {Salimans, Tim and Goodfellow, Ian and Zaremba, Wojciech and Cheung, Vicki and Radford, Alec and Chen, Xi},
  year = {2016},
  doi = {10.48550/arxiv.1606.03498},
  eprint = {1606.03498},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1606.03498}
}
```

- [← previous Implicit Discourse Relation Recognition with Context-aware Character-enhanced Embeddings](https://md.research.elamur.ai/free/en/papers/qin-2016-implicit-discourse-relation-recognition-context/)
- [next → Improving Coreference Resolution by Learning Entity-Level Distributed Representations](https://md.research.elamur.ai/free/en/papers/clark-2016-improving-coreference-resolution-learning-entity/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/2/)

## References

26 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, et al. Generative adversarial nets. In NIPS, 2014. [find](https://scholar.google.com/scholar?q=Ian+Goodfellow%2C+Jean+Pouget%2DAbadie%2C+Mehdi+Mirza%2C+et+al%2E+Generative+adversarial+nets%2E+In+NIPS%2C+2014%2E)
2. Emily Denton, Soumith Chintala, Arthur Szlam, and Rob Fergus. Deep generative image models using a laplacian pyramid of adversarial networks. arXiv preprint arXiv:1506.05751, 2015. — [arXiv:1506.05751](https://arxiv.org/abs/1506.05751) [find](https://scholar.google.com/scholar?q=Emily+Denton%2C+Soumith+Chintala%2C+Arthur+Szlam%2C+and+Rob+Fergus%2E+Deep+generative+image+models+using+a+laplacian+pyramid+of+adversarial+networks%2E+arXiv+preprint+arXiv%3A1506%2E05751%2C+2015%2E+%E2%80%94+%5BarXiv%3A1506%2E05751)
3. Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Luke+Metz%2C+and+Soumith+Chintala%2E+Unsupervised+representation+learning+with+deep+convolutional+generative+adversarial+networks%2E+arXiv+preprint+arXiv%3A1511%2E06434%2C+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/radford-2015-unsupervised-representation-learning-deep-convolutional/)
4. Ian J Goodfellow. On distinguishability criteria for estimating generative models. arXiv:1412.6515, 2014. — [arXiv:1412.6515](https://arxiv.org/abs/1412.6515) [find](https://scholar.google.com/scholar?q=Ian+J+Goodfellow%2E+On+distinguishability+criteria+for+estimating+generative+models%2E+arXiv%3A1412%2E6515%2C+2014%2E+%E2%80%94+%5BarXiv%3A1412%2E6515%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1412%2E6515%29)
5. Daniel Jiwoong Im, Chris Dongjoo Kim, Hui Jiang, and Roland Memisevic. Generating images with recurrent adversarial networks. arXiv preprint arXiv:1602.05110, 2016. — [arXiv:1602.05110](https://arxiv.org/abs/1602.05110) [find](https://scholar.google.com/scholar?q=Daniel+Jiwoong+Im%2C+Chris+Dongjoo+Kim%2C+Hui+Jiang%2C+and+Roland+Memisevic%2E+Generating+images+with+recurrent+adversarial+networks%2E+arXiv+preprint+arXiv%3A1602%2E05110%2C+2016%2E+%E2%80%94+%5BarXiv%3A1602%2E05110%5D%28https%3A%2F%2Farxiv%2E)
6. Donggeun Yoo, Namil Kim, Sunggyun Park, Anthony S Paek, and In So Kweon. Pixel-level domain transfer. arXiv preprint arXiv:1603.07442, 2016. — [arXiv:1603.07442](https://arxiv.org/abs/1603.07442) [find](https://scholar.google.com/scholar?q=Donggeun+Yoo%2C+Namil+Kim%2C+Sunggyun+Park%2C+Anthony+S+Paek%2C+and+In+So+Kweon%2E+Pixel%2Dlevel+domain+transfer%2E+arXiv+preprint+arXiv%3A1603%2E07442%2C+2016%2E+%E2%80%94+%5BarXiv%3A1603%2E07442%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1603%2E07442%29)
7. Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilbert-schmidt norms. In Algorithmic learning theory, pages 63–77. Springer, 2005. [find](https://scholar.google.com/scholar?q=Arthur+Gretton%2C+Olivier+Bousquet%2C+Alex+Smola%2C+and+Bernhard+Scho%CC%88lkopf%2E+Measuring+statistical+dependence+with+hilbert%2Dschmidt+norms%2E+In+Algorithmic+learning+theory%2C+pages+63%E2%80%9377%2E+Springer%2C+2005%2E)
8. Kenji Fukumizu, Arthur Gretton, Xiaohai Sun, and Bernhard Schölkopf. Kernel measures of conditional dependence. In NIPS, volume 20, pages 489–496, 2007. [find](https://scholar.google.com/scholar?q=Kenji+Fukumizu%2C+Arthur+Gretton%2C+Xiaohai+Sun%2C+and+Bernhard+Scho%CC%88lkopf%2E+Kernel+measures+of+conditional+dependence%2E+In+NIPS%2C+volume+20%2C+pages+489%E2%80%93496%2C+2007%2E)
9. Alex Smola, Arthur Gretton, Le Song, and Bernhard Schölkopf. A hilbert space embedding for distributions. In Algorithmic learning theory, pages 13–31. Springer, 2007. [find](https://scholar.google.com/scholar?q=Alex+Smola%2C+Arthur+Gretton%2C+Le+Song%2C+and+Bernhard+Scho%CC%88lkopf%2E+A+hilbert+space+embedding+for+distributions%2E+In+Algorithmic+learning+theory%2C+pages+13%E2%80%9331%2E+Springer%2C+2007%2E)
10. Yujia Li, Kevin Swersky, and Richard S. Zemel. Generative moment matching networks. CoRR, abs/1502.02761, 2015. — [arXiv:1502.02761](https://arxiv.org/abs/1502.02761) [find](https://scholar.google.com/scholar?q=Yujia+Li%2C+Kevin+Swersky%2C+and+Richard+S%2E+Zemel%2E+Generative+moment+matching+networks%2E+CoRR%2C+abs%2F1502%2E02761%2C+2015%2E+%E2%80%94+%5BarXiv%3A1502%2E02761%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1502%2E02761%29)
11. Gintare Karolina Dziugaite, Daniel M Roy, and Zoubin Ghahramani. Training generative neural networks via maximum mean discrepancy optimization. arXiv preprint arXiv:1505.03906, 2015. — [arXiv:1505.03906](https://arxiv.org/abs/1505.03906) [find](https://scholar.google.com/scholar?q=Gintare+Karolina+Dziugaite%2C+Daniel+M+Roy%2C+and+Zoubin+Ghahramani%2E+Training+generative+neural+networks+via+maximum+mean+discrepancy+optimization%2E+arXiv+preprint+arXiv%3A1505%2E03906%2C+2015%2E+%E2%80%94+%5BarXiv%3A1505%2E039)
12. Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015. [find](https://scholar.google.com/scholar?q=Sergey+Ioffe+and+Christian+Szegedy%2E+Batch+normalization%3A+Accelerating+deep+network+training+by+reducing+internal+covariate+shift%2E+arXiv+preprint+arXiv%3A1502%2E03167%2C+2015%2E)
13. Ilya Sutskever, Rafal Jozefowicz, Karol Gregor, et al. Towards principled unsupervised learning. arXiv preprint arXiv:1511.06440, 2015. — [arXiv:1511.06440](https://arxiv.org/abs/1511.06440) [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Rafal+Jozefowicz%2C+Karol+Gregor%2C+et+al%2E+Towards+principled+unsupervised+learning%2E+arXiv+preprint+arXiv%3A1511%2E06440%2C+2015%2E+%E2%80%94+%5BarXiv%3A1511%2E06440%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1511%2E06440%29)
14. Jost Tobias Springenberg. Unsupervised and semi-supervised learning with categorical generative adversarial networks. arXiv preprint arXiv:1511.06390, 2015. — [arXiv:1511.06390](https://arxiv.org/abs/1511.06390) [find](https://scholar.google.com/scholar?q=Jost+Tobias+Springenberg%2E+Unsupervised+and+semi%2Dsupervised+learning+with+categorical+generative+adversarial+networks%2E+arXiv+preprint+arXiv%3A1511%2E06390%2C+2015%2E+%E2%80%94+%5BarXiv%3A1511%2E06390%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F)
15. Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. 2016. MIT Press. [find](https://scholar.google.com/scholar?q=Ian+Goodfellow%2C+Yoshua+Bengio%2C+and+Aaron+Courville%2E+Deep+Learning%2E+2016%2E+MIT+Press%2E)
16. George W Brown. Iterative solution of games by fictitious play. Activity analysis of production and allocation, 13(1):374–376, 1951. [find](https://scholar.google.com/scholar?q=George+W+Brown%2E+Iterative+solution+of+games+by+fictitious+play%2E+Activity+analysis+of+production+and+allocation%2C+13%281%29%3A374%E2%80%93376%2C+1951%2E)
17. C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the Inception Architecture for Computer Vision. ArXiv e-prints, December 2015. [find](https://scholar.google.com/scholar?q=C%2E+Szegedy%2C+V%2E+Vanhoucke%2C+S%2E+Ioffe%2C+J%2E+Shlens%2C+and+Z%2E+Wojna%2E+Rethinking+the+Inception+Architecture+for+Computer+Vision%2E+ArXiv+e%2Dprints%2C+December+2015%2E)
18. David Warde-Farley and Ian Goodfellow. Adversarial perturbations of deep neural networks. In Tamir Hazan, George Papandreou, and Daniel Tarlow, editors, Perturbations, Optimization, and Statistics, chapter 11. 2016. Book in preparation for MIT Press. [find](https://scholar.google.com/scholar?q=David+Warde%2DFarley+and+Ian+Goodfellow%2E+Adversarial+perturbations+of+deep+neural+networks%2E+In+Tamir+Hazan%2C+George+Papandreou%2C+and+Daniel+Tarlow%2C+editors%2C+Perturbations%2C+Optimization%2C+and+Statistics%2C+ch)
19. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. arXiv preprint arXiv:1512.00567, 2015. [find](https://scholar.google.com/scholar?q=Christian+Szegedy%2C+Vincent+Vanhoucke%2C+Sergey+Ioffe%2C+Jonathon+Shlens%2C+and+Zbigniew+Wojna%2E+Rethinking+the+inception+architecture+for+computer+vision%2E+arXiv+preprint+arXiv%3A1512%2E00567%2C+2015%2E)
20. Tim Salimans and Diederik P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. arXiv preprint arXiv:1602.07868, 2016. — [arXiv:1602.07868](https://arxiv.org/abs/1602.07868) [find](https://scholar.google.com/scholar?q=Tim+Salimans+and+Diederik+P+Kingma%2E+Weight+normalization%3A+A+simple+reparameterization+to+accelerate+training+of+deep+neural+networks%2E+arXiv+preprint+arXiv%3A1602%2E07868%2C+2016%2E+%E2%80%94+%5BarXiv%3A1602%2E07868%5D%28https%3A)
21. Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Neural Information Processing Systems, 2014. arXiv preprint [find](https://scholar.google.com/scholar?q=Diederik+P+Kingma%2C+Shakir+Mohamed%2C+Danilo+Jimenez+Rezende%2C+and+Max+Welling%2E+Semi%2Dsupervised+learning+with+deep+generative+models%2E+In+Neural+Information+Processing+Systems%2C+2014%2E+arXiv+preprint)
22. Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Ken Nakae, and Shin Ishii. Distributional smoothing by virtual adversarial examples. arXiv preprint arXiv:1507.00677, 2015. — [arXiv:1507.00677](https://arxiv.org/abs/1507.00677) [find](https://scholar.google.com/scholar?q=Takeru+Miyato%2C+Shin%2Dichi+Maeda%2C+Masanori+Koyama%2C+Ken+Nakae%2C+and+Shin+Ishii%2E+Distributional+smoothing+by+virtual+adversarial+examples%2E+arXiv+preprint+arXiv%3A1507%2E00677%2C+2015%2E+%E2%80%94+%5BarXiv%3A1507%2E00677%5D%28https%3A)
23. Lars Maaløe, Casper Kaae Sønderby, Søren Kaae Sønderby, and Ole Winther. Auxiliary deep generative models. arXiv preprint arXiv:1602.05473, 2016. — [arXiv:1602.05473](https://arxiv.org/abs/1602.05473) [find](https://scholar.google.com/scholar?q=Lars+Maal%C3%B8e%2C+Casper+Kaae+S%C3%B8nderby%2C+S%C3%B8ren+Kaae+S%C3%B8nderby%2C+and+Ole+Winther%2E+Auxiliary+deep+generative+models%2E+arXiv+preprint+arXiv%3A1602%2E05473%2C+2016%2E+%E2%80%94+%5BarXiv%3A1602%2E05473%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1602%2E05473%29)
24. Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi-supervised learning with ladder networks. In Advances in Neural Information Processing Systems, 2015. [find](https://scholar.google.com/scholar?q=Antti+Rasmus%2C+Mathias+Berglund%2C+Mikko+Honkala%2C+Harri+Valpola%2C+and+Tapani+Raiko%2E+Semi%2Dsupervised+learning+with+ladder+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2015%2E)
25. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, et al. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013. [find](https://scholar.google.com/scholar?q=Christian+Szegedy%2C+Wojciech+Zaremba%2C+Ilya+Sutskever%2C+et+al%2E+Intriguing+properties+of+neural+networks%2E+arXiv+preprint+arXiv%3A1312%2E6199%2C+2013%2E) [our card](https://md.research.elamur.ai/free/en/papers/szegedy-2013-intriguing-properties-neural-networks/)
26. Martı́n Abadi, Ashish Agarwal, Paul Barham, et al. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. Software available from tensorflow.org. [find](https://scholar.google.com/scholar?q=Mart%C4%B1%CC%81n+Abadi%2C+Ashish+Agarwal%2C+Paul+Barham%2C+et+al%2E+TensorFlow%3A+Large%2Dscale+machine+learning+on+heterogeneous+systems%2C+2015%2E+Software+available+from+tensorflow%2Eorg%2E)