---
title: "HiFi-GAN — paper by Kong et al. (2020) — summary and ideas"
description: "The paper addresses vocoding: turning mel-spectrograms into raw speech waveforms that are both… 27 references, cited by 247 papers here."
authors: ["Jungil Kong", "Jaehyeon Kim", "Jaekyoung Bae"]
year: 2020
venue: "arXiv:2010.05646"
source: "https://arxiv.org/abs/2010.05646"
doi: "10.48550/arxiv.2010.05646"
arxiv: "2010.05646"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kong-2020-hifi-gan-generative-adversarial-networks/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kong-2020-hifi-gan-generative-adversarial-networks/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/) / HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis

# HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis

**Authors:** Jungil Kong, Jaehyeon Kim and Jaekyoung Bae · **Year:** 2020 · **Venue:** arXiv:2010.05646 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2010.05646) · [PDF](https://arxiv.org/pdf/2010.05646) · [DOI 10.48550/arxiv.2010.05646](https://doi.org/10.48550/arxiv.2010.05646) · [arXiv 2010.05646](https://arxiv.org/abs/2010.05646) · [OpenAlex](https://openalex.org/W3092028330)

**Citations:** 722 citations (OpenAlex, 2026-10-14) · [247](https://md.research.elamur.ai/free/en/papers/kong-2020-hifi-gan-generative-adversarial-networks/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper addresses vocoding: turning mel-spectrograms into raw speech waveforms that are both fast to produce and as natural as those from slower autoregressive or flow-based vocoders. Earlier GAN vocoders were efficient but sounded noticeably worse than models such as WaveNet and WaveGlow. HiFi-GAN pairs a fully convolutional generator with two discriminators. The generator upsamples the spectrogram and uses multi-receptive field fusion (MRF) blocks. The first discriminator is new: it judges audio by looking at samples spaced at fixed periods. The second is a multi-scale discriminator inherited from MelGAN. Training combines least-squares adversarial objectives with auxiliary terms, including a mel-spectrogram loss.

On a single-speaker benchmark, the model's MOS approaches that of real recordings. It generates 22.05 kHz audio far faster than real time on a GPU, and a compact variant runs faster than real time on a CPU. It also handles unseen VCTK speakers well and works with Tacotron2 outputs, especially after fine-tuning. The implementation is open source, and the result made high-quality, on-device-friendly neural vocoding realistic.

## Key points
- **Architecture:**
  - The generator upsamples with transposed convolutions, each followed by an MRF module.
  - The multi-period discriminator (MPD) uses the prime periods [2, 3, 5, 7, 11].
  - The multi-scale discriminator (MSD) inspects raw, ×2 and ×4 average-pooled audio.
  - Training uses LS-GAN losses.
- **Speed and size:**
  - The model runs 167.9× faster than real time on one V100 at 22.05 kHz.
  - The smallest variant has only 0.92M parameters.
  - The fastest variant reaches about 13.44× real time on CPU and 1,186× on a V100.
- **Ablation (baseline HiFi-GAN V3, MOS 4.10; ground truth 4.57):**
  - Removing MPD drops MOS to 2.28.
  - Without MSD it falls to 3.74, without MRF to 3.92, and without the mel-spectrogram loss to 3.25.
  - Using power-of-2 periods [2, 4, 8, 16, 32] gives 3.90.
  - Adding MPD to MelGAN lifts it from 2.88 to 3.35.
- **Unseen speakers (VCTK; 9 held-out speakers, 50 utterances):**
  - HiFi-GAN V1/V2/V3 score 3.77/3.69/3.61.
  - WaveNet (MoL) and WaveGlow both score 3.52, and MelGAN scores 3.50.
  - Ground truth scores 3.79.
- **End-to-end with Tacotron2 (ground truth 4.23):**
  - Without fine-tuning, HiFi-GAN scores 3.88–3.91 versus 3.69 for WaveGlow.
  - After fine-tuning on Tacotron2-predicted spectrograms, HiFi-GAN reaches 4.02–4.18, while WaveGlow does not improve (3.66).

## Interesting ideas
- **Period-based reshaping:** A 1D signal is folded into a 2D grid by a period, and the convolutions use width-1 kernels, so each phase of the cycle is processed separately. Folding, rather than subsampling, lets gradients reach every time step. Prime periods keep the views from overlapping. This trick could carry over to any sequence with hidden periodic structure.
- **Complementary critics:** One discriminator sees disjoint, spaced-out samples, while the other sees smoothed, continuous audio. Combining critics with deliberately different "views" of the data is a general recipe for catching different kinds of artifacts.
- **Parallel receptive fields:** MRF sums residual blocks that use different kernel sizes and dilations. This gives the generator several pattern lengths at once and exposes clear knobs for trading quality against speed.
- **Fine-tuning on upstream predictions:** The vocoder is fine-tuned on the noisy outputs of the upstream model (Tacotron2 in teacher-forcing mode) to absorb the mismatch in an end-to-end pipeline. Notably, fine-tuning raised the spectrogram-level error yet improved perceived quality. This is a reminder that reconstruction metrics and human judgments can diverge.
- **Reusable discriminators across generator sizes:** The same discriminators and training setup worked for generators of different sizes. One can therefore pick a generator to fit a deployment budget without re-tuning the adversarial side, which saves costly hyper-parameter searches.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [🔊 Signal Processing](https://md.research.elamur.ai/free/en/shelves/computer-science/signal-processing/) · 493 works

🧭 Topics (1): [🎙️ Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) · 519 works

## Cite

Cite as: Jungil Kong, Jaehyeon Kim and Jaekyoung Bae. 2020. HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis. arXiv:2010.05646.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{kong2020hifi,
  title = {{HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis}},
  author = {Jungil Kong, Jaehyeon Kim and Jaekyoung Bae},
  year = {2020},
  doi = {10.48550/arxiv.2010.05646},
  eprint = {2010.05646},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2010.05646}
}
```

- [← previous MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://md.research.elamur.ai/free/en/papers/sun-2020-mobilebert-compact-task-agnostic-bert/)
- [next → AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-adapterfusion-non-destructive-task-composition/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/)

## References

27 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mikołaj Bińkowski, Jeff Donahue, Sander Dieleman, Aidan Clark, Erich Elsen, Norman Casagrande, Luis C Cobo, and Karen Simonyan. High fidelity speech synthesis with adversarial networks. arXiv preprint arXiv:1909.11646, 2019. — [arXiv:1909.11646](https://arxiv.org/abs/1909.11646) [find](https://scholar.google.com/scholar?q=Miko%C5%82aj+Bin%CC%81kowski%2C+Jeff+Donahue%2C+Sander+Dieleman%2C+Aidan+Clark%2C+Erich+Elsen%2C+Norman+Casagrande%2C+Luis+C+Cobo%2C+and+Karen+Simonyan%2E+High+fidelity+speech+synthesis+with+adversarial+networks%2E+arXiv+preprin)
2. Chris Donahue, Julian McAuley, and Miller Puckette. Adversarial audio synthesis. arXiv preprint arXiv:1802.04208, 2018. — [arXiv:1802.04208](https://arxiv.org/abs/1802.04208) [find](https://scholar.google.com/scholar?q=Chris+Donahue%2C+Julian+McAuley%2C+and+Miller+Puckette%2E+Adversarial+audio+synthesis%2E+arXiv+preprint+arXiv%3A1802%2E04208%2C+2018%2E+%E2%80%94+%5BarXiv%3A1802%2E04208%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1802%2E04208%29)
3. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. [Generative adversarial nets](https://md.research.elamur.ai/free/en/papers/goodfellow-2014-generative-adversarial-nets/). In Advances in neural information processing systems, pages 2672–2680, 2014.
4. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017. [find](https://scholar.google.com/scholar?q=Phillip+Isola%2C+Jun%2DYan+Zhu%2C+Tinghui+Zhou%2C+and+Alexei+A+Efros%2E+Image%2Dto%2Dimage+translation+with+conditional+adversarial+networks%2E+In+Proceedings+of+the+IEEE+conference+on+computer+vision+and+pattern+rec)
5. Keith Ito. The lj speech dataset. [https://keithito.com/LJ-Speech-Dataset/](https://keithito.com/LJ-Speech-Dataset/), 2017. [↗](https://keithito.com/LJ-Speech-Dataset/)
6. Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems, pages 10215–10224, 2018. [find](https://scholar.google.com/scholar?q=Durk+P+Kingma+and+Prafulla+Dhariwal%2E+Glow%3A+Generative+flow+with+invertible+1x1+convolutions%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+10215%E2%80%9310224%2C+2018%2E)
7. Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio, and Aaron C Courville. Melgan: Generative adversarial networks for conditional waveform synthesis. In Advances in Neural Information Processing Systems 32, pages 14910… [find](https://scholar.google.com/scholar?q=Kundan+Kumar%2C+Rithesh+Kumar%2C+Thibault+de+Boissiere%2C+Lucas+Gestin%2C+Wei+Zhen+Teoh%2C+Jose+Sotelo%2C+Alexandre+de+Br%C3%A9bisson%2C+Yoshua+Bengio%2C+and+Aaron+C+Courville%2E+Melgan%3A+Generative+adversarial+networks+for+)
8. Rithesh Kumar. descriptinc/melgan-neurips. melgan-neurips, 2019. [find](https://scholar.google.com/scholar?q=Rithesh+Kumar%2E+descriptinc%2Fmelgan%2Dneurips%2E+melgan%2Dneurips%2C+2019%2E)
9. Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoencoding beyond pixels using a learned similarity metric. In International conference on machine learning, pages 1558–1566. PMLR, 2016. [find](https://scholar.google.com/scholar?q=Anders+Boesen+Lindbo+Larsen%2C+S%C3%B8ren+Kaae+S%C3%B8nderby%2C+Hugo+Larochelle%2C+and+Ole+Winther%2E+Autoencoding+beyond+pixels+using+a+learned+similarity+metric%2E+In+International+conference+on+machine+learning%2C+pages)
10. Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. Neural speech synthesis with transformer network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6706–6713, 2019. [find](https://scholar.google.com/scholar?q=Naihan+Li%2C+Shujie+Liu%2C+Yanqing+Liu%2C+Sheng+Zhao%2C+and+Ming+Liu%2E+Neural+speech+synthesis+with+transformer+network%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+volume+33%2C+pages+6706%E2%80%93)
11. Ilya Loshchilov and Frank Hutter. arXiv:1711.05101, 2017. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+arXiv%3A1711%2E05101%2C+2017%2E)
12. Decoupled weight decay regularization. Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 2794–2802, 2017. [find](https://scholar.google.com/scholar?q=Decoupled+weight+decay+regularization%2E+Xudong+Mao%2C+Qing+Li%2C+Haoran+Xie%2C+Raymond+YK+Lau%2C+Zhen+Wang%2C+and+Stephen+Paul+Smolley%2E+Least+squares+generative+adversarial+networks%2E+In+Proceedings+of+the+IEEE+I)
13. Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. arXiv preprint arXiv:1511.05440, 2015. — [arXiv:1511.05440](https://arxiv.org/abs/1511.05440) [find](https://scholar.google.com/scholar?q=Michael+Mathieu%2C+Camille+Couprie%2C+and+Yann+LeCun%2E+Deep+multi%2Dscale+video+prediction+beyond+mean+square+error%2E+arXiv+preprint+arXiv%3A1511%2E05440%2C+2015%2E+%E2%80%94+%5BarXiv%3A1511%2E05440%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1511%2E054)
14. Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957, 2018. — [arXiv:1802.05957](https://arxiv.org/abs/1802.05957) [find](https://scholar.google.com/scholar?q=Takeru+Miyato%2C+Toshiki+Kataoka%2C+Masanori+Koyama%2C+and+Yuichi+Yoshida%2E+Spectral+normalization+for+generative+adversarial+networks%2E+arXiv+preprint+arXiv%3A1802%2E05957%2C+2018%2E+%E2%80%94+%5BarXiv%3A1802%2E05957%5D%28https%3A%2F%2Farx)
15. Aaron Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George Driessche, Edward Lockhart, Luis Cobo, Florian Stimberg, et al. Parallel wavenet: Fast high-fidelity speech synthesis. In International conference on machine learning, pages 3918–3926. PMLR, 2018. [find](https://scholar.google.com/scholar?q=Aaron+Oord%2C+Yazhe+Li%2C+Igor+Babuschkin%2C+Karen+Simonyan%2C+Oriol+Vinyals%2C+Koray+Kavukcuoglu%2C+George+Driessche%2C+Edward+Lockhart%2C+Luis+Cobo%2C+Florian+Stimberg%2C+et+al%2E+Parallel+wavenet%3A+Fast+high%2Dfidelity+spe)
16. Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. [Wavenet: A generative model for raw audio](https://md.research.elamur.ai/free/en/papers/oord-2016-wavenet-generative-model-raw-audio/). arXiv preprint arXiv:1609.03499, 2016.
17. Wei Ping, Kainan Peng, Andrew Gibiansky, Sercan O Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller. Deep voice 3: Scaling text-to-speech with convolutional sequence learning. arXiv preprint arXiv:1710.07654, 2017. [https://github.com/descriptinc/arXiv](https://github.com/descriptinc/arXiv) preprint Wei Ping, Kainan Peng,… [↗](https://github.com/descriptinc/arXiv)
18. Ryan Prenger, Rafael Valle, and Bryan Catanzaro. [Waveglow: A flow-based generative network for speech synthesis](https://md.research.elamur.ai/free/en/papers/prenger-2018-waveglow-flow-based-generative-network/). In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3617–3621.
19. Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in neural information processing systems, pages 901–909, 2016. [find](https://scholar.google.com/scholar?q=Tim+Salimans+and+Durk+P+Kingma%2E+Weight+normalization%3A+A+simple+reparameterization+to+accelerate+training+of+deep+neural+networks%2E+In+Advances+in+neural+information+processing+systems%2C+pages+901%E2%80%93909%2C+2)
20. Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE International Conference on Acoustics, Speech and Signal Proc… [find](https://scholar.google.com/scholar?q=Jonathan+Shen%2C+Ruoming+Pang%2C+Ron+J+Weiss%2C+Mike+Schuster%2C+Navdeep+Jaitly%2C+Zongheng+Yang%2C+Zhifeng+Chen%2C+Yu+Zhang%2C+Yuxuan+Wang%2C+Rj+Skerrv%2DRyan%2C+et+al%2E+Natural+tts+synthesis+by+conditioning+wavenet+on+mel)
21. Mingxing Tan, Ruoming Pang, and Quoc V Le. Efficientdet: Scalable and efficient object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10781–10790, 2020. [find](https://scholar.google.com/scholar?q=Mingxing+Tan%2C+Ruoming+Pang%2C+and+Quoc+V+Le%2E+Efficientdet%3A+Scalable+and+efficient+object+detection%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+pages+10781%E2%80%931079)
22. Rafael Valle. Nvidia/tacotron2. [https://github.com/NVIDIA/tacotron2](https://github.com/NVIDIA/tacotron2), 2018a. [↗](https://github.com/NVIDIA/tacotron2)
23. Rafael Valle. Nvidia/waveglow. [https://github.com/NVIDIA/waveglow](https://github.com/NVIDIA/waveglow), 2018b. [↗](https://github.com/NVIDIA/waveglow)
24. Christophe Veaux, Junichi Yamagishi, Kirsten MacDonald, et al. Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit. University of Edinburgh. The Centre for Speech Technology Research (CSTR), 2017. [find](https://scholar.google.com/scholar?q=Christophe+Veaux%2C+Junichi+Yamagishi%2C+Kirsten+MacDonald%2C+et+al%2E+Cstr+vctk+corpus%3A+English+multi%2Dspeaker+corpus+for+cstr+voice+cloning+toolkit%2E+University+of+Edinburgh%2E+The+Centre+for+Speech+Technology+)
25. Ryuichi Yamamoto. wavenet vocoder. [https://github.com/r9y9/wavenet_vocoder/](https://github.com/r9y9/wavenet_vocoder/), 2018. [↗](https://github.com/r9y9/wavenet_vocoder/)
26. Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel wavegan: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6199–6203. [find](https://scholar.google.com/scholar?q=Ryuichi+Yamamoto%2C+Eunwoo+Song%2C+and+Jae%2DMin+Kim%2E+Parallel+wavegan%3A+A+fast+waveform+generation+model+based+on+generative+adversarial+networks+with+multi%2Dresolution+spectrogram%2E+In+ICASSP+2020%2D2020+IEEE+)
27. Bohan Zhai, Tianren Gao, Flora Xue, Daniel Rothchild, Bichen Wu, Joseph E Gonzalez, and Kurt Keutzer. Squeezewave: Extremely lightweight vocoders for on-device speech synthesis. arXiv preprint arXiv:2001.05685, 2020. — [arXiv:2001.05685](https://arxiv.org/abs/2001.05685) [find](https://scholar.google.com/scholar?q=Bohan+Zhai%2C+Tianren+Gao%2C+Flora+Xue%2C+Daniel+Rothchild%2C+Bichen+Wu%2C+Joseph+E+Gonzalez%2C+and+Kurt+Keutzer%2E+Squeezewave%3A+Extremely+lightweight+vocoders+for+on%2Ddevice+speech+synthesis%2E+arXiv+preprint+arXiv%3A2)

*Updated 2 November 2026*