---
title: "HuBERT: Self-Supervised Speech Representation — Hsu 2021"
description: "Unsupervised speech representation learning is harder than in text or vision for three reasons."
authors: ["Wei-Ning Hsu", "Benjamin Bolte", "Yao-Hung Hubert Tsai", "Kushal Lakhotia", "Ruslan Salakhutdinov", "Abdelrahman Mohamed"]
year: 2021
venue: "arXiv:2106.07447"
source: "https://arxiv.org/abs/2106.07447"
doi: "10.48550/arxiv.2106.07447"
arxiv: "2106.07447"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/hsu-2021-hubert-self-supervised-speech-representation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hsu-2021-hubert-self-supervised-speech-representation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units

# HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units

**Authors:** [Wei-Ning Hsu](https://md.research.elamur.ai/free/en/authors/wei-ning-hsu/), [Benjamin Bolte](https://md.research.elamur.ai/free/en/authors/benjamin-bolte/), [Yao-Hung Hubert Tsai](https://md.research.elamur.ai/free/en/authors/yao-hung-hubert-tsai/), [Kushal Lakhotia](https://md.research.elamur.ai/free/en/authors/kushal-lakhotia/), [Ruslan Salakhutdinov](https://md.research.elamur.ai/free/en/authors/ruslan-salakhutdinov/), [Abdelrahman Mohamed](https://md.research.elamur.ai/free/en/authors/abdelrahman-mohamed/) · **Year:** 2021 · **Venue:** arXiv:2106.07447 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2106.07447) · [PDF](https://arxiv.org/pdf/2106.07447) · [DOI 10.48550/arxiv.2106.07447](https://doi.org/10.48550/arxiv.2106.07447) · [arXiv 2106.07447](https://arxiv.org/abs/2106.07447) · [OpenAlex](https://openalex.org/W3169320628)

**Citations:** 2919 citations (openalex-snapshot · 2026-10-14) · [522](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Unsupervised speech representation learning is harder than in text or vision for three reasons. Each utterance contains many sounds, there is no ready-made inventory of discrete units, and unit boundaries are unknown. HuBERT (Hidden-Unit BERT) handles this by first clustering acoustic frames offline, starting with k-means over MFCC features, and treating the cluster IDs as frame-level pseudo-targets. A Transformer encoder based on the wav2vec 2.0 design receives audio with spans masked out. It must infer the cluster IDs of the hidden frames from context, so it learns acoustic and long-range sequential structure together. The clusters are then rebuilt from the model's own learned features, and pre-training repeats. The authors stress that consistent targets matter more than accurate ones. The models are pre-trained on Librispeech 960h or Libri-Light 60k hours and fine-tuned on labeled subsets from 10 minutes up to 960 hours. Across all of these settings, HuBERT equals or beats wav2vec 2.0. This is useful for languages and dialects with little transcribed audio, and the released checkpoints make reuse straightforward.

## Key points
- **Training recipe:** the initial teacher is k-means with 100 clusters, followed by two clustering iterations. The cross-entropy loss is computed only on masked frames (α = 1), which makes training less sensitive to noisy cluster labels. Mask spans follow the SpanBERT / wav2vec 2.0 scheme.
- **Model sizes:** BASE (~95M parameters, 12 layers), LARGE (~317M, 24 layers) and X-LARGE (~964M, 48 layers). All three share a CNN waveform encoder made of seven 512-channel convolution layers.
- **Benchmarks:** pre-training uses Librispeech (960h) and Libri-Light (60,000h). Fine-tuning splits are 10min, 1h, 10h, 100h and 960h. Results match or improve on wav2vec 2.0 in every split.
- **Scaling result:** pre-trained on Libri-Light 60k hours, X-LARGE cuts WER relative to LARGE by up to 19% on dev-other and 13% on test-other.
- **Availability:** code plus pre-trained and fine-tuned models are released in fairseq (`examples/hubert`). The method draws on DeepCluster from computer vision.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

522 papers of our library cite it (of the 140884 whose bibliographies we read); 32 of them have a page here, 490 not yet.

- [Beyond Orthography: Automatic Recovery of Short Vowels and Dialectal Sounds in Arabic](https://md.research.elamur.ai/free/en/papers/acl-2024-kheir-beyond-orthography-automatic-recovery-short/)
- [FINDINGS OF THE IWSLT 2023 EVALUATION CAMPAIGN](https://md.research.elamur.ai/free/en/papers/agarwal-2023-findings-iwslt-2023-evaluation-campaign/)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://md.research.elamur.ai/free/en/papers/borsos-2022-audiolm-language-modeling-approach-audio/)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://md.research.elamur.ai/free/en/papers/chu-2023-qwen-audio-advancing-universal/)
- [Seamless: Multilingual Expressive and Streaming Speech Translation](https://md.research.elamur.ai/free/en/papers/communication-2023-seamless-multilingual-expressive-streaming-speech/)
- [SeamlessM4T: Massively Multilingual & Multimodal Machine Translation](https://md.research.elamur.ai/free/en/papers/communication-2023-seamlessm4t-massively-multilingual-multimodal-machine/)
- [High Fidelity Neural Audio Compression](https://md.research.elamur.ai/free/en/papers/defossez-2022-high-fidelity-neural-audio-compression/)
- [Moshi: a speech-text foundation model for real-time dialogue](https://md.research.elamur.ai/free/en/papers/defossez-2024-moshi-speech-text-foundation-model/)
- [ADI-20: Arabic Dialect Identification dataset and models](https://md.research.elamur.ai/free/en/papers/elleuch-2025-adi-20-arabic-dialect-identification/)
- [Ara-Best-RQ: Multi Dialectal Arabic SSL](https://md.research.elamur.ai/free/en/papers/elleuch-2026-ara-best-rq-multi-dialectal/)
- [Arabic Multimodal Machine Learning: Datasets, Applications, Approaches, and Challenges](https://md.research.elamur.ai/free/en/papers/haouhat-2025-arabic-multimodal-machine-learning-datasets/)
- [Towards a Unified Benchmark for Arabic Pronunciation Assessment: Quranic Recitation as Case Study](https://md.research.elamur.ai/free/en/papers/kheir-2025-towards-unified-benchmark-arabic-pronunciation/)
- [IQRA 2026: Interspeech Challenge on Automatic Pronunciation Assessment for Modern Standard Arabic (MSA)](https://md.research.elamur.ai/free/en/papers/kheir-2026-iqra-2026-interspeech-challenge-automatic/)
- [HebDB: a Weakly Supervised Dataset for Hebrew Speech Processing](https://md.research.elamur.ai/free/en/papers/langstream-2024-turetzky-hebdb-weakly-supervised-dataset-hebrew/)
- [Enhancing TTS Stability in Hebrew using Discrete Semantic Units](https://md.research.elamur.ai/free/en/papers/langstream-2024-zeldes-enhancing-tts-stability-hebrew-using/)
- [Do Factual Recall Mechanisms Carry over from Text to Speech in Multimodal Language Models?](https://md.research.elamur.ai/free/en/papers/modica-2026-do-factual-recall-mechanisms-carry/)
- [Probing Discrete Speech Tokens of Spoken Language Models](https://md.research.elamur.ai/free/en/papers/naber-2026-probing-discrete-speech-tokens-spoken/)
- [J-CHAT: Japanese Large-scale Spoken Dialogue Corpus for Spoken Dialogue Language Modeling](https://md.research.elamur.ai/free/en/papers/nakata-2026-j-chat-japanese-large-scale/)
- [LinTO Audio and Textual Datasets to Train and Evaluate Automatic Speech Recognition in Tunisian Arabic Dialect](https://md.research.elamur.ai/free/en/papers/naouara-2025-linto-audio-textual-datasets-train/)
- [A framework for analyzing concept representations in neural models](https://md.research.elamur.ai/free/en/papers/naowarat-2026-framework-analyzing-concept-representations-neural/)
- [Dynamic Layer Selection for Efficient Tone Recognition in Self-Supervised Speech Models](https://md.research.elamur.ai/free/en/papers/obiang-2026-dynamic-layer-selection-efficient-tone/)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://md.research.elamur.ai/free/en/papers/radford-2022-robust-speech-recognition-large-scale/)
- [AudioPaLM: A Large Language Model That Can Speak and Listen](https://md.research.elamur.ai/free/en/papers/rubenstein-2023-audiopalm-large-language-model-that/)
- [On the Robustness of Arabic Speech Dialect Identification](https://md.research.elamur.ai/free/en/papers/sullivan-2023-robustness-arabic-speech-dialect-identification/)
- [N-Shot Benchmarking of Whisper on Diverse Arabic Speech Recognition](https://md.research.elamur.ai/free/en/papers/talafha-2023-n-shot-benchmarking-whisper-diverse/)
- [大模型参数高效微调方法综述: 技术、趋势与挑战](https://md.research.elamur.ai/free/en/papers/tang-2026-parameter-efficient-fine/)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://md.research.elamur.ai/free/en/papers/wang-2023-neural-codec-language-models-zero/)
- [Open Universal Arabic ASR Leaderboard](https://md.research.elamur.ai/free/en/papers/wang-2024-open-universal-arabic-asr-leaderboard/)
- [Tools for Estimating the Perceived Level of Phonetic Reduction](https://md.research.elamur.ai/free/en/papers/ward-2026-tools-estimating-perceived-level-phonetic/)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://md.research.elamur.ai/free/en/papers/xi-2023-rise-potential-large-language/)
- [A Fusion-Aware Two-Stage Framework for Mispronunciation Detection and Diagnosis in Low-Resource Modern Standard Arabic](https://md.research.elamur.ai/free/en/papers/yang-2026-fusion-aware-two-stage-framework/)
- [SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities](https://md.research.elamur.ai/free/en/papers/zhang-2023-speechgpt-empowering-large-language-models/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{hsu2021hubert,
  title = {{HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units}},
  author = {Hsu, Wei-Ning and Bolte, Benjamin and Tsai, Yao-Hung Hubert and Lakhotia, Kushal and Salakhutdinov, Ruslan and Mohamed, Abdelrahman},
  year = {2021},
  doi = {10.48550/arxiv.2106.07447},
  eprint = {2106.07447},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2106.07447}
}
```

- [← previous How useful are Enhanced Universal Dependencies for semantic interpretation?](https://md.research.elamur.ai/free/en/papers/ud-2021-findlay-how-useful-are-enhanced-universal/)
- [next → Human Rationales as Attribution Priors for Explainable Stance Detection](https://md.research.elamur.ai/free/en/papers/jayaram-2021-human-rationales-as-attribution-priors/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/)

## References

64 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748, 2018. [find](https://scholar.google.com/scholar?q=A%2E+v%2E+d%2E+Oord%2C+Y%2E+Li%2C+and+O%2E+Vinyals%2C+%E2%80%9CRepresentation+learning+with+contrastive+predictive+coding%2C%E2%80%9D+arXiv+preprint+arXiv%3A1807%2E03748%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/oord-2018-representation-learning-contrastive-predictive-coding/)
2. S. Schneider, A. Baevski, R. Collobert, and M. Auli, “wav2vec: Unsupervised pre-training for speech recognition,” arXiv preprint arXiv:1904.05862, 2019. — [arXiv:1904.05862](https://arxiv.org/abs/1904.05862) [find](https://scholar.google.com/scholar?q=S%2E+Schneider%2C+A%2E+Baevski%2C+R%2E+Collobert%2C+and+M%2E+Auli%2C+%E2%80%9Cwav2vec%3A+Unsupervised+pre%2Dtraining+for+speech+recognition%2C%E2%80%9D+arXiv+preprint+arXiv%3A1904%2E05862%2C+2019%2E+%E2%80%94+%5BarXiv%3A1904%2E05862%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1904)
3. E. Kharitonov, M. Rivière, G. Synnaeve, L. Wolf, P.-E. Mazaré, M. Douze, and E. Dupoux, “Data augmenting contrastive learning of speech representations in the time domain,” arXiv preprint arXiv:2007.00991, 2020. — [arXiv:2007.00991](https://arxiv.org/abs/2007.00991) [find](https://scholar.google.com/scholar?q=E%2E+Kharitonov%2C+M%2E+Rivie%CC%80re%2C+G%2E+Synnaeve%2C+L%2E+Wolf%2C+P%2E%2DE%2E+Mazare%CC%81%2C+M%2E+Douze%2C+and+E%2E+Dupoux%2C+%E2%80%9CData+augmenting+contrastive+learning+of+speech+representations+in+the+time+domain%2C%E2%80%9D+arXiv+preprint+arXiv%3A2007)
4. Y.-A. Chung, W.-N. Hsu, H. Tang, and J. Glass, “An unsupervised autoregressive model for speech representation learning,” arXiv preprint arXiv:1904.03240, 2019. — [arXiv:1904.03240](https://arxiv.org/abs/1904.03240) [find](https://scholar.google.com/scholar?q=Y%2E%2DA%2E+Chung%2C+W%2E%2DN%2E+Hsu%2C+H%2E+Tang%2C+and+J%2E+Glass%2C+%E2%80%9CAn+unsupervised+autoregressive+model+for+speech+representation+learning%2C%E2%80%9D+arXiv+preprint+arXiv%3A1904%2E03240%2C+2019%2E+%E2%80%94+%5BarXiv%3A1904%2E03240%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
5. A. Baevski, S. Schneider, and M. Auli, “vq-wav2vec: Self-supervised learning of discrete speech representations,” arXiv preprint arXiv:1910.05453, 2019. — [arXiv:1910.05453](https://arxiv.org/abs/1910.05453) [find](https://scholar.google.com/scholar?q=A%2E+Baevski%2C+S%2E+Schneider%2C+and+M%2E+Auli%2C+%E2%80%9Cvq%2Dwav2vec%3A+Self%2Dsupervised+learning+of+discrete+speech+representations%2C%E2%80%9D+arXiv+preprint+arXiv%3A1910%2E05453%2C+2019%2E+%E2%80%94+%5BarXiv%3A1910%2E05453%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1910)
6. A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” arXiv preprint arXiv:2006.11477, 2020. [find](https://scholar.google.com/scholar?q=A%2E+Baevski%2C+H%2E+Zhou%2C+A%2E+Mohamed%2C+and+M%2E+Auli%2C+%E2%80%9Cwav2vec+2%2E0%3A+A+framework+for+self%2Dsupervised+learning+of+speech+representations%2C%E2%80%9D+arXiv+preprint+arXiv%3A2006%2E11477%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/baevski-2020-wav2vec-2-0-framework-self/)
7. G. Zavaliagkos and T. Colthurst, “Utilizing untranscribed training data to improve performance,” in DARPA Broadcast News Transcription and Understanding Workshop, 1998. [find](https://scholar.google.com/scholar?q=G%2E+Zavaliagkos+and+T%2E+Colthurst%2C+%E2%80%9CUtilizing+untranscribed+training+data+to+improve+performance%2C%E2%80%9D+in+DARPA+Broadcast+News+Transcription+and+Understanding+Workshop%2C+1998%2E)
8. J. Ma, S. Matsoukas, O. Kimball, and R. Schwartz, “Unsupervised training on large amounts of broadcast news data,” in ICASSP, 2006. [find](https://scholar.google.com/scholar?q=J%2E+Ma%2C+S%2E+Matsoukas%2C+O%2E+Kimball%2C+and+R%2E+Schwartz%2C+%E2%80%9CUnsupervised+training+on+large+amounts+of+broadcast+news+data%2C%E2%80%9D+in+ICASSP%2C+2006%2E)
9. J. Kahn, A. Lee, and A. Hannun, “Self-training for end-to-end speech recognition,” in ICASSP, 2020. [find](https://scholar.google.com/scholar?q=J%2E+Kahn%2C+A%2E+Lee%2C+and+A%2E+Hannun%2C+%E2%80%9CSelf%2Dtraining+for+end%2Dto%2Dend+speech+recognition%2C%E2%80%9D+in+ICASSP%2C+2020%2E)
10. W.-N. Hsu, A. Lee, G. Synnaeve, and A. Hannun, “Semi-supervised speech recognition via local prior matching,” arXiv preprint arXiv:2002.10336, 2020. — [arXiv:2002.10336](https://arxiv.org/abs/2002.10336) [find](https://scholar.google.com/scholar?q=W%2E%2DN%2E+Hsu%2C+A%2E+Lee%2C+G%2E+Synnaeve%2C+and+A%2E+Hannun%2C+%E2%80%9CSemi%2Dsupervised+speech+recognition+via+local+prior+matching%2C%E2%80%9D+arXiv+preprint+arXiv%3A2002%2E10336%2C+2020%2E+%E2%80%94+%5BarXiv%3A2002%2E10336%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2002%2E103)
11. A. Xiao, C. Fuegen, and A. Mohamed, “Contrastive semi-supervised learning for asr,” arXiv preprint arXiv:2103.05149, 2021. — [arXiv:2103.05149](https://arxiv.org/abs/2103.05149) [find](https://scholar.google.com/scholar?q=A%2E+Xiao%2C+C%2E+Fuegen%2C+and+A%2E+Mohamed%2C+%E2%80%9CContrastive+semi%2Dsupervised+learning+for+asr%2C%E2%80%9D+arXiv+preprint+arXiv%3A2103%2E05149%2C+2021%2E+%E2%80%94+%5BarXiv%3A2103%2E05149%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2103%2E05149%29)
12. Q. Xu, T. Likhomanenko, J. Kahn, A. Hannun, G. Synnaeve, and R. Collobert, “Iterative pseudo-labeling for speech recognition,” arXiv preprint arXiv:2005.09267, 2020. — [arXiv:2005.09267](https://arxiv.org/abs/2005.09267) [find](https://scholar.google.com/scholar?q=Q%2E+Xu%2C+T%2E+Likhomanenko%2C+J%2E+Kahn%2C+A%2E+Hannun%2C+G%2E+Synnaeve%2C+and+R%2E+Collobert%2C+%E2%80%9CIterative+pseudo%2Dlabeling+for+speech+recognition%2C%E2%80%9D+arXiv+preprint+arXiv%3A2005%2E09267%2C+2020%2E+%E2%80%94+%5BarXiv%3A2005%2E09267%5D%28https%3A%2F%2Farxiv)
13. M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, “Unsupervised learning of visual features by contrasting cluster assignments,” CoRR, vol. abs/2006.09882, 2020. — [arXiv:2006.09882](https://arxiv.org/abs/2006.09882) [find](https://scholar.google.com/scholar?q=M%2E+Caron%2C+I%2E+Misra%2C+J%2E+Mairal%2C+P%2E+Goyal%2C+P%2E+Bojanowski%2C+and+A%2E+Joulin%2C+%E2%80%9CUnsupervised+learning+of+visual+features+by+contrasting+cluster+assignments%2C%E2%80%9D+CoRR%2C+vol%2E+abs%2F2006%2E09882%2C+2020%2E+%E2%80%94+%5BarXiv%3A2006%2E098)
14. X. Chen and K. He, “Exploring simple siamese representation learning,” CoRR, vol. abs/2011.10566, 2020. — [arXiv:2011.10566](https://arxiv.org/abs/2011.10566) [find](https://scholar.google.com/scholar?q=X%2E+Chen+and+K%2E+He%2C+%E2%80%9CExploring+simple+siamese+representation+learning%2C%E2%80%9D+CoRR%2C+vol%2E+abs%2F2011%2E10566%2C+2020%2E+%E2%80%94+%5BarXiv%3A2011%2E10566%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2011%2E10566%29)
15. J. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. Á. Pires, Z. D. Guo, M. G. Azar, B. Piot, K. Kavukcuoglu, R. Munos, and M. Valko, “Bootstrap your own latent: A new approach to self-supervised learning,” CoRR, vol. abs/2006.07733, 2020. — [arXiv:2006.07733](https://arxiv.org/abs/2006.07733) [find](https://scholar.google.com/scholar?q=J%2E+Grill%2C+F%2E+Strub%2C+F%2E+Altche%CC%81%2C+C%2E+Tallec%2C+P%2E+H%2E+Richemond%2C+E%2E+Buchatskaya%2C+C%2E+Doersch%2C+B%2E+A%CC%81%2E+Pires%2C+Z%2E+D%2E+Guo%2C+M%2E+G%2E+Azar%2C+B%2E+Piot%2C+K%2E+Kavukcuoglu%2C+R%2E+Munos%2C+and+M%2E+Valko%2C+%E2%80%9CBootstrap+your+own+latent)
16. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Ber… [find](https://scholar.google.com/scholar?q=T%2E+B%2E+Brown%2C+B%2E+Mann%2C+N%2E+Ryder%2C+M%2E+Subbiah%2C+J%2E+Kaplan%2C+P%2E+Dhariwal%2C+A%2E+Neelakantan%2C+P%2E+Shyam%2C+G%2E+Sastry%2C+A%2E+Askell%2C+S%2E+Agarwal%2C+A%2E+Herbert%2DVoss%2C+G%2E+Krueger%2C+T%2E+Henighan%2C+R%2E+Child%2C+A%2E+Ramesh%2C+D%2E+M%2E+Zie)
17. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692, 2019. [find](https://scholar.google.com/scholar?q=Y%2E+Liu%2C+M%2E+Ott%2C+N%2E+Goyal%2C+J%2E+Du%2C+M%2E+Joshi%2C+D%2E+Chen%2C+O%2E+Levy%2C+M%2E+Lewis%2C+L%2E+Zettlemoyer%2C+and+V%2E+Stoyanov%2C+%E2%80%9CRoberta%3A+A+robustly+optimized+bert+pretraining+approach%2C%E2%80%9D+arXiv+preprint+arXiv%3A1907%2E11692%2C+2019)
18. M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” arXiv preprint arXiv:1910.13461, 2019. [find](https://scholar.google.com/scholar?q=M%2E+Lewis%2C+Y%2E+Liu%2C+N%2E+Goyal%2C+M%2E+Ghazvininejad%2C+A%2E+Mohamed%2C+O%2E+Levy%2C+V%2E+Stoyanov%2C+and+L%2E+Zettlemoyer%2C+%E2%80%9CBart%3A+Denoising+sequence%2Dto%2Dsequence+pre%2Dtraining+for+natural+language+generation%2C+translation%2C+and) [our card](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/)
19. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018. [find](https://scholar.google.com/scholar?q=J%2E+Devlin%2C+M%2E%2DW%2E+Chang%2C+K%2E+Lee%2C+and+K%2E+Toutanova%2C+%E2%80%9CBert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2C%E2%80%9D+arXiv+preprint+arXiv%3A1810%2E04805%2C+2018%2E)
20. K. Clark, M.-T. Luong, Q. V. Le, and C. D. Manning, “Electra: Pre-training text encoders as discriminators rather than generators,” arXiv preprint arXiv:2003.10555, 2020. [find](https://scholar.google.com/scholar?q=K%2E+Clark%2C+M%2E%2DT%2E+Luong%2C+Q%2E+V%2E+Le%2C+and+C%2E+D%2E+Manning%2C+%E2%80%9CElectra%3A+Pre%2Dtraining+text+encoders+as+discriminators+rather+than+generators%2C%E2%80%9D+arXiv+preprint+arXiv%3A2003%2E10555%2C+2020%2E)
21. M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, “Deep contextualized word representations,” in NAACL, 2018. [find](https://scholar.google.com/scholar?q=M%2E+E%2E+Peters%2C+M%2E+Neumann%2C+M%2E+Iyyer%2C+M%2E+Gardner%2C+C%2E+Clark%2C+K%2E+Lee%2C+and+L%2E+Zettlemoyer%2C+%E2%80%9CDeep+contextualized+word+representations%2C%E2%80%9D+in+NAACL%2C+2018%2E)
22. K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in CVPR, 2020. [find](https://scholar.google.com/scholar?q=K%2E+He%2C+H%2E+Fan%2C+Y%2E+Wu%2C+S%2E+Xie%2C+and+R%2E+Girshick%2C+%E2%80%9CMomentum+contrast+for+unsupervised+visual+representation+learning%2C%E2%80%9D+in+CVPR%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/he-2019-momentum-contrast-unsupervised-visual-representation/)
23. M. Caron, P. Bojanowski, A. Joulin, and M. Douze, “Deep clustering for unsupervised learning of visual features,” in ECCV, 2018. [find](https://scholar.google.com/scholar?q=M%2E+Caron%2C+P%2E+Bojanowski%2C+A%2E+Joulin%2C+and+M%2E+Douze%2C+%E2%80%9CDeep+clustering+for+unsupervised+learning+of+visual+features%2C%E2%80%9D+in+ECCV%2C+2018%2E)
24. V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in ICASSP, 2015. [find](https://scholar.google.com/scholar?q=V%2E+Panayotov%2C+G%2E+Chen%2C+D%2E+Povey%2C+and+S%2E+Khudanpur%2C+%E2%80%9CLibrispeech%3A+an+asr+corpus+based+on+public+domain+audio+books%2C%E2%80%9D+in+ICASSP%2C+2015%2E)
25. J. Kahn et al., “Libri-light: A benchmark for asr with limited or no supervision,” in ICASSP, 2020. [find](https://scholar.google.com/scholar?q=J%2E+Kahn+et+al%2E%2C+%E2%80%9CLibri%2Dlight%3A+A+benchmark+for+asr+with+limited+or+no+supervision%2C%E2%80%9D+in+ICASSP%2C+2020%2E)
26. C.-y. Lee and J. Glass, “A nonparametric bayesian approach to acoustic model discovery,” in ACL, 2012. [find](https://scholar.google.com/scholar?q=C%2E%2Dy%2E+Lee+and+J%2E+Glass%2C+%E2%80%9CA+nonparametric+bayesian+approach+to+acoustic+model+discovery%2C%E2%80%9D+in+ACL%2C+2012%2E)
27. L. Ondel, L. Burget, and J. Černockỳ, “Variational inference for acoustic unit discovery,” Procedia Computer Science, vol. 81, pp. 80–86, 2016. [find](https://scholar.google.com/scholar?q=L%2E+Ondel%2C+L%2E+Burget%2C+and+J%2E+C%CC%8Cernocky%CC%80%2C+%E2%80%9CVariational+inference+for+acoustic+unit+discovery%2C%E2%80%9D+Procedia+Computer+Science%2C+vol%2E+81%2C+pp%2E+80%E2%80%9386%2C+2016%2E)
28. J. Ebbers, J. Heymann, L. Drude, T. Glarner, R. Haeb-Umbach, and B. Raj, “Hidden markov model variational autoencoder for acoustic unit discovery.” in INTERSPEECH, 2017. [find](https://scholar.google.com/scholar?q=J%2E+Ebbers%2C+J%2E+Heymann%2C+L%2E+Drude%2C+T%2E+Glarner%2C+R%2E+Haeb%2DUmbach%2C+and+B%2E+Raj%2C+%E2%80%9CHidden+markov+model+variational+autoencoder+for+acoustic+unit+discovery%2E%E2%80%9D+in+INTERSPEECH%2C+2017%2E)
29. W.-N. Hsu, Y. Zhang, and J. Glass, “Learning latent representations for speech generation and transformation,” in INTERSPEECH, 2017. [find](https://scholar.google.com/scholar?q=W%2E%2DN%2E+Hsu%2C+Y%2E+Zhang%2C+and+J%2E+Glass%2C+%E2%80%9CLearning+latent+representations+for+speech+generation+and+transformation%2C%E2%80%9D+in+INTERSPEECH%2C+2017%2E)
30. ——, “Unsupervised learning of disentangled and interpretable representations from sequential data,” in NeurIPS, 2017. [find](https://scholar.google.com/scholar?q=%E2%80%94%E2%80%94%2C+%E2%80%9CUnsupervised+learning+of+disentangled+and+interpretable+representations+from+sequential+data%2C%E2%80%9D+in+NeurIPS%2C+2017%2E)
31. J. Chorowski, R. J. Weiss, S. Bengio, and A. van den Oord, “Unsupervised speech representation learning using wavenet autoencoders,” IEEE/ACM transactions on audio, speech, and language processing, vol. 27, no. 12, pp. 2041–2053, 2019. [find](https://scholar.google.com/scholar?q=J%2E+Chorowski%2C+R%2E+J%2E+Weiss%2C+S%2E+Bengio%2C+and+A%2E+van+den+Oord%2C+%E2%80%9CUnsupervised+speech+representation+learning+using+wavenet+autoencoders%2C%E2%80%9D+IEEE%2FACM+transactions+on+audio%2C+speech%2C+and+language+processing%2C+vo)
32. S. Khurana, S. R. Joty, A. Ali, and J. Glass, “A factorial deep markov model for unsupervised disentangled representation learning from speech,” in ICASSP, 2019. [find](https://scholar.google.com/scholar?q=S%2E+Khurana%2C+S%2E+R%2E+Joty%2C+A%2E+Ali%2C+and+J%2E+Glass%2C+%E2%80%9CA+factorial+deep+markov+model+for+unsupervised+disentangled+representation+learning+from+speech%2C%E2%80%9D+in+ICASSP%2C+2019%2E)
33. S. Khurana, A. Laurent, W.-N. Hsu, J. Chorowski, A. Lancucki, R. Marxer, and J. Glass, “A convolutional deep markov model for unsupervised speech representation learning,” arXiv preprint arXiv:2006.02547, 2020. — [arXiv:2006.02547](https://arxiv.org/abs/2006.02547) [find](https://scholar.google.com/scholar?q=S%2E+Khurana%2C+A%2E+Laurent%2C+W%2E%2DN%2E+Hsu%2C+J%2E+Chorowski%2C+A%2E+Lancucki%2C+R%2E+Marxer%2C+and+J%2E+Glass%2C+%E2%80%9CA+convolutional+deep+markov+model+for+unsupervised+speech+representation+learning%2C%E2%80%9D+arXiv+preprint+arXiv%3A2006%2E02)
34. M. Joshi, D. Chen, Y. Liu, D. S. Weld, L. Zettlemoyer, and O. Levy, “Spanbert: Improving pre-training by representing and predicting spans,” Transactions of the Association for Computational Linguistics, 2020. [find](https://scholar.google.com/scholar?q=M%2E+Joshi%2C+D%2E+Chen%2C+Y%2E+Liu%2C+D%2E+S%2E+Weld%2C+L%2E+Zettlemoyer%2C+and+O%2E+Levy%2C+%E2%80%9CSpanbert%3A+Improving+pre%2Dtraining+by+representing+and+predicting+spans%2C%E2%80%9D+Transactions+of+the+Association+for+Computational+Linguisti) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2019-spanbert-improving-pre-training-representing/)
35. S. Young, “Large vocabulary continuous speech recognition: A review,” IEEE Signal Processing Magazine, vol. 13, no. 5, pp. 45–57, 1996. [find](https://scholar.google.com/scholar?q=S%2E+Young%2C+%E2%80%9CLarge+vocabulary+continuous+speech+recognition%3A+A+review%2C%E2%80%9D+IEEE+Signal+Processing+Magazine%2C+vol%2E+13%2C+no%2E+5%2C+pp%2E+45%E2%80%9357%2C+1996%2E)
36. O. Abdel-Hamid, A.-r. Mohamed, H. Jiang, and G. Penn, “Applying convolutional neural networks concepts to hybrid nn-hmm model for speech recognition,” in 2012 IEEE international conference on Acoustics, speech and signal processing (ICASSP). IEEE, 2012, pp. 4277– 4280. [find](https://scholar.google.com/scholar?q=O%2E+Abdel%2DHamid%2C+A%2E%2Dr%2E+Mohamed%2C+H%2E+Jiang%2C+and+G%2E+Penn%2C+%E2%80%9CApplying+convolutional+neural+networks+concepts+to+hybrid+nn%2Dhmm+model+for+speech+recognition%2C%E2%80%9D+in+2012+IEEE+international+conference+on+Acoustic)
37. D. Povey, “Discriminative training for large vocabulary speech recognition,” Ph.D. dissertation, University of Cambridge, 2005. [find](https://scholar.google.com/scholar?q=D%2E+Povey%2C+%E2%80%9CDiscriminative+training+for+large+vocabulary+speech+recognition%2C%E2%80%9D+Ph%2ED%2E+dissertation%2C+University+of+Cambridge%2C+2005%2E)
38. H. A. Bourlard and N. Morgan, Connectionist speech recognition: a hybrid approach. Springer Science & Business Media, 2012, vol. 247. [find](https://scholar.google.com/scholar?q=H%2E+A%2E+Bourlard+and+N%2E+Morgan%2C+Connectionist+speech+recognition%3A+a+hybrid+approach%2E+Springer+Science+%26+Business+Media%2C+2012%2C+vol%2E+247%2E)
39. R. M. Gray and D. L. Neuhoff, “Quantization,” IEEE transactions on information theory, vol. 44, no. 6, pp. 2325–2383, 1998. [find](https://scholar.google.com/scholar?q=R%2E+M%2E+Gray+and+D%2E+L%2E+Neuhoff%2C+%E2%80%9CQuantization%2C%E2%80%9D+IEEE+transactions+on+information+theory%2C+vol%2E+44%2C+no%2E+6%2C+pp%2E+2325%E2%80%932383%2C+1998%2E)
40. Y. Zhang, J. Qin, D. S. Park, W. Han, C.-C. Chiu, R. Pang, Q. V. Le, and Y. Wu, “Pushing the limits of semi-supervised learning for automatic speech recognition,” arXiv preprint arXiv:2010.10504, 2020. — [arXiv:2010.10504](https://arxiv.org/abs/2010.10504) [find](https://scholar.google.com/scholar?q=Y%2E+Zhang%2C+J%2E+Qin%2C+D%2E+S%2E+Park%2C+W%2E+Han%2C+C%2E%2DC%2E+Chiu%2C+R%2E+Pang%2C+Q%2E+V%2E+Le%2C+and+Y%2E+Wu%2C+%E2%80%9CPushing+the+limits+of+semi%2Dsupervised+learning+for+automatic+speech+recognition%2C%E2%80%9D+arXiv+preprint+arXiv%3A2010%2E10504%2C+2020)
41. A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in ICML, 2006. [find](https://scholar.google.com/scholar?q=A%2E+Graves%2C+S%2E+Ferna%CC%81ndez%2C+F%2E+Gomez%2C+and+J%2E+Schmidhuber%2C+%E2%80%9CConnectionist+temporal+classification%3A+labelling+unsegmented+sequence+data+with+recurrent+neural+networks%2C%E2%80%9D+in+ICML%2C+2006%2E)
42. A. van den Oord, O. Vinyals et al., “Neural discrete representation learning,” in NeurIPS, 2017. [find](https://scholar.google.com/scholar?q=A%2E+van+den+Oord%2C+O%2E+Vinyals+et+al%2E%2C+%E2%80%9CNeural+discrete+representation+learning%2C%E2%80%9D+in+NeurIPS%2C+2017%2E)
43. T. Glarner, P. Hanebrink, J. Ebbers, and R. Haeb-Umbach, “Full bayesian hidden markov model variational autoencoder for acoustic unit discovery.” in INTERSPEECH, 2018. [find](https://scholar.google.com/scholar?q=T%2E+Glarner%2C+P%2E+Hanebrink%2C+J%2E+Ebbers%2C+and+R%2E+Haeb%2DUmbach%2C+%E2%80%9CFull+bayesian+hidden+markov+model+variational+autoencoder+for+acoustic+unit+discovery%2E%E2%80%9D+in+INTERSPEECH%2C+2018%2E)
44. Y.-A. Chung and J. Glass, “Generative pre-training for speech with autoregressive predictive coding,” in ICASSP, 2020. [find](https://scholar.google.com/scholar?q=Y%2E%2DA%2E+Chung+and+J%2E+Glass%2C+%E2%80%9CGenerative+pre%2Dtraining+for+speech+with+autoregressive+predictive+coding%2C%E2%80%9D+in+ICASSP%2C+2020%2E)
45. ——, “Improved speech representations with multi-target autoregressive predictive coding,” arXiv preprint arXiv:2004.05274, 2020. [find](https://scholar.google.com/scholar?q=%E2%80%94%E2%80%94%2C+%E2%80%9CImproved+speech+representations+with+multi%2Dtarget+autoregressive+predictive+coding%2C%E2%80%9D+arXiv+preprint+arXiv%3A2004%2E05274%2C+2020%2E)
46. S. Ling, Y. Liu, J. Salazar, and K. Kirchhoff, “Deep contextualized acoustic representations for semi-supervised speech recognition,” in ICASSP, 2020. [find](https://scholar.google.com/scholar?q=S%2E+Ling%2C+Y%2E+Liu%2C+J%2E+Salazar%2C+and+K%2E+Kirchhoff%2C+%E2%80%9CDeep+contextualized+acoustic+representations+for+semi%2Dsupervised+speech+recognition%2C%E2%80%9D+in+ICASSP%2C+2020%2E)
47. W. Wang, Q. Tang, and K. Livescu, “Unsupervised pre-training of bidirectional speech encoders via masked reconstruction,” in ICASSP, 2020. [find](https://scholar.google.com/scholar?q=W%2E+Wang%2C+Q%2E+Tang%2C+and+K%2E+Livescu%2C+%E2%80%9CUnsupervised+pre%2Dtraining+of+bidirectional+speech+encoders+via+masked+reconstruction%2C%E2%80%9D+in+ICASSP%2C+2020%2E)
48. A. T. Liu, S.-w. Yang, P.-H. Chi, P.-c. Hsu, and H.-y. Lee, “Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders,” in ICASSP, 2020. [find](https://scholar.google.com/scholar?q=A%2E+T%2E+Liu%2C+S%2E%2Dw%2E+Yang%2C+P%2E%2DH%2E+Chi%2C+P%2E%2Dc%2E+Hsu%2C+and+H%2E%2Dy%2E+Lee%2C+%E2%80%9CMockingjay%3A+Unsupervised+speech+representation+learning+with+deep+bidirectional+transformer+encoders%2C%E2%80%9D+in+ICASSP%2C+2020%2E)
49. P.-H. Chi, P.-H. Chung, T.-H. Wu, C.-C. Hsieh, S.-W. Li, and H.-y. Lee, “Audio albert: A lite bert for self-supervised learning of audio representation,” arXiv preprint arXiv:2005.08575, 2020. — [arXiv:2005.08575](https://arxiv.org/abs/2005.08575) [find](https://scholar.google.com/scholar?q=P%2E%2DH%2E+Chi%2C+P%2E%2DH%2E+Chung%2C+T%2E%2DH%2E+Wu%2C+C%2E%2DC%2E+Hsieh%2C+S%2E%2DW%2E+Li%2C+and+H%2E%2Dy%2E+Lee%2C+%E2%80%9CAudio+albert%3A+A+lite+bert+for+self%2Dsupervised+learning+of+audio+representation%2C%E2%80%9D+arXiv+preprint+arXiv%3A2005%2E08575%2C+2020%2E+%E2%80%94+%5BarXi)
50. S. Ling and Y. Liu, “Decoar 2.0: Deep contextualized acoustic representations with vector quantization,” arXiv preprint arXiv:2012.06659, 2020. — [arXiv:2012.06659](https://arxiv.org/abs/2012.06659) [find](https://scholar.google.com/scholar?q=S%2E+Ling+and+Y%2E+Liu%2C+%E2%80%9CDecoar+2%2E0%3A+Deep+contextualized+acoustic+representations+with+vector+quantization%2C%E2%80%9D+arXiv+preprint+arXiv%3A2012%2E06659%2C+2020%2E+%E2%80%94+%5BarXiv%3A2012%2E06659%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2012%2E06659%29)
51. A. Baevski, M. Auli, and A. Mohamed, “Effectiveness of self-supervised pre-training for speech recognition,” arXiv preprint arXiv:1911.03912, 2019. — [arXiv:1911.03912](https://arxiv.org/abs/1911.03912) [find](https://scholar.google.com/scholar?q=A%2E+Baevski%2C+M%2E+Auli%2C+and+A%2E+Mohamed%2C+%E2%80%9CEffectiveness+of+self%2Dsupervised+pre%2Dtraining+for+speech+recognition%2C%E2%80%9D+arXiv+preprint+arXiv%3A1911%2E03912%2C+2019%2E+%E2%80%94+%5BarXiv%3A1911%2E03912%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1911%2E0391)
52. Y.-H. H. Tsai, Y. Wu, R. Salakhutdinov, and L.-P. Morency, “Self-supervised learning from a multi-view perspective,” arXiv preprint arXiv:2006.05576, 2020. — [arXiv:2006.05576](https://arxiv.org/abs/2006.05576) [find](https://scholar.google.com/scholar?q=Y%2E%2DH%2E+H%2E+Tsai%2C+Y%2E+Wu%2C+R%2E+Salakhutdinov%2C+and+L%2E%2DP%2E+Morency%2C+%E2%80%9CSelf%2Dsupervised+learning+from+a+multi%2Dview+perspective%2C%E2%80%9D+arXiv+preprint+arXiv%3A2006%2E05576%2C+2020%2E+%E2%80%94+%5BarXiv%3A2006%2E05576%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2)
53. S. Pascual, M. Ravanelli, J. Serrà, A. Bonafonte, and Y. Bengio, “Learning problem-agnostic speech representations from multiple self-supervised tasks,” in INTERSPEECH, 2019. [find](https://scholar.google.com/scholar?q=S%2E+Pascual%2C+M%2E+Ravanelli%2C+J%2E+Serra%CC%80%2C+A%2E+Bonafonte%2C+and+Y%2E+Bengio%2C+%E2%80%9CLearning+problem%2Dagnostic+speech+representations+from+multiple+self%2Dsupervised+tasks%2C%E2%80%9D+in+INTERSPEECH%2C+2019%2E)
54. T. Likhomanenko, Q. Xu, J. Kahn, G. Synnaeve, and R. Collobert, “slimipl: Language-model-free iterative pseudo-labeling,” arXiv preprint arXiv:2010.11524, 2020. — [arXiv:2010.11524](https://arxiv.org/abs/2010.11524) [find](https://scholar.google.com/scholar?q=T%2E+Likhomanenko%2C+Q%2E+Xu%2C+J%2E+Kahn%2C+G%2E+Synnaeve%2C+and+R%2E+Collobert%2C+%E2%80%9Cslimipl%3A+Language%2Dmodel%2Dfree+iterative+pseudo%2Dlabeling%2C%E2%80%9D+arXiv+preprint+arXiv%3A2010%2E11524%2C+2020%2E+%E2%80%94+%5BarXiv%3A2010%2E11524%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
55. S. Lloyd, “Least squares quantization in pcm,” IEEE transactions on information theory, vol. 28, no. 2, pp. 129–137, 1982. [find](https://scholar.google.com/scholar?q=S%2E+Lloyd%2C+%E2%80%9CLeast+squares+quantization+in+pcm%2C%E2%80%9D+IEEE+transactions+on+information+theory%2C+vol%2E+28%2C+no%2E+2%2C+pp%2E+129%E2%80%93137%2C+1982%2E)
56. F. Pedregosa et al., “Scikit-learn: Machine learning in python,” the Journal of machine Learning research, 2011. [find](https://scholar.google.com/scholar?q=F%2E+Pedregosa+et+al%2E%2C+%E2%80%9CScikit%2Dlearn%3A+Machine+learning+in+python%2C%E2%80%9D+the+Journal+of+machine+Learning+research%2C+2011%2E)
57. D. Arthur and S. Vassilvitskii, “k-means++: The advantages of careful seeding,” Stanford, Tech. Rep., 2006. [find](https://scholar.google.com/scholar?q=D%2E+Arthur+and+S%2E+Vassilvitskii%2C+%E2%80%9Ck%2Dmeans%2B%2B%3A+The+advantages+of+careful+seeding%2C%E2%80%9D+Stanford%2C+Tech%2E+Rep%2E%2C+2006%2E)
58. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014. [find](https://scholar.google.com/scholar?q=D%2E+P%2E+Kingma+and+J%2E+Ba%2C+%E2%80%9CAdam%3A+A+method+for+stochastic+optimization%2C%E2%80%9D+arXiv+preprint+arXiv%3A1412%2E6980%2C+2014%2E)
59. V. Pratap et al., “wav2letter++: The fastest open-source speech recognition system,” arXiv preprint arXiv:1812.07625, 2018. [find](https://scholar.google.com/scholar?q=V%2E+Pratap+et+al%2E%2C+%E2%80%9Cwav2letter%2B%2B%3A+The+fastest+open%2Dsource+speech+recognition+system%2C%E2%80%9D+arXiv+preprint+arXiv%3A1812%2E07625%2C+2018%2E)
60. M. Ott et al., “fairseq: A fast, extensible toolkit for sequence modeling,” in NAACL, 2019. [find](https://scholar.google.com/scholar?q=M%2E+Ott+et+al%2E%2C+%E2%80%9Cfairseq%3A+A+fast%2C+extensible+toolkit+for+sequence+modeling%2C%E2%80%9D+in+NAACL%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/ott-2019-fairseq-fast-extensible-toolkit-sequence/)
61. D. S. Park, Y. Zhang, Y. Jia, W. Han, C.-C. Chiu, B. Li, Y. Wu, and Q. V. Le, “Improved noisy student training for automatic speech recognition,” arXiv preprint arXiv:2005.09629, 2020. — [arXiv:2005.09629](https://arxiv.org/abs/2005.09629) [find](https://scholar.google.com/scholar?q=D%2E+S%2E+Park%2C+Y%2E+Zhang%2C+Y%2E+Jia%2C+W%2E+Han%2C+C%2E%2DC%2E+Chiu%2C+B%2E+Li%2C+Y%2E+Wu%2C+and+Q%2E+V%2E+Le%2C+%E2%80%9CImproved+noisy+student+training+for+automatic+speech+recognition%2C%E2%80%9D+arXiv+preprint+arXiv%3A2005%2E09629%2C+2020%2E+%E2%80%94+%5BarXiv%3A2005%2E0)
62. A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu et al., “Conformer: Convolution-augmented transformer for speech recognition,” arXiv preprint arXiv:2005.08100, 2020. [find](https://scholar.google.com/scholar?q=A%2E+Gulati%2C+J%2E+Qin%2C+C%2E%2DC%2E+Chiu%2C+N%2E+Parmar%2C+Y%2E+Zhang%2C+J%2E+Yu%2C+W%2E+Han%2C+S%2E+Wang%2C+Z%2E+Zhang%2C+Y%2E+Wu+et+al%2E%2C+%E2%80%9CConformer%3A+Convolution%2Daugmented+transformer+for+speech+recognition%2C%E2%80%9D+arXiv+preprint+arXiv%3A2005%2E081) [our card](https://md.research.elamur.ai/free/en/papers/gulati-2020-conformer-convolution-augmented-transformer-speech/)
63. Q. Xu, A. Baevski, T. Likhomanenko, P. Tomasello, A. Conneau, R. Collobert, G. Synnaeve, and M. Auli, “Self-training and pre-training are complementary for speech recognition,” arXiv preprint arXiv:2010.11430, 2020. — [arXiv:2010.11430](https://arxiv.org/abs/2010.11430) [find](https://scholar.google.com/scholar?q=Q%2E+Xu%2C+A%2E+Baevski%2C+T%2E+Likhomanenko%2C+P%2E+Tomasello%2C+A%2E+Conneau%2C+R%2E+Collobert%2C+G%2E+Synnaeve%2C+and+M%2E+Auli%2C+%E2%80%9CSelf%2Dtraining+and+pre%2Dtraining+are+complementary+for+speech+recognition%2C%E2%80%9D+arXiv+preprint+arXiv%3A20)
64. D. Le, X. Zhang, W. Zheng, C. Fügen, G. Zweig, and M. L. Seltzer, “From senones to chenones: Tied context-dependent graphemes for hybrid speech recognition,” in ASRU, 2019. [find](https://scholar.google.com/scholar?q=D%2E+Le%2C+X%2E+Zhang%2C+W%2E+Zheng%2C+C%2E+Fu%CC%88gen%2C+G%2E+Zweig%2C+and+M%2E+L%2E+Seltzer%2C+%E2%80%9CFrom+senones+to+chenones%3A+Tied+context%2Ddependent+graphemes+for+hybrid+speech+recognition%2C%E2%80%9D+in+ASRU%2C+2019%2E)