---
title: "Conformer: Convolution-augmented Transformer — Gulati 2020"
description: "Speech recognition encoders built purely on self-attention handle long-range context well but struggle with fine local acoustic patterns."
authors: ["Anmol Gulati", "James Qin", "Chung-Cheng Chiu", "Niki Parmar", "Yu Zhang", "Jiahui Yu", "Wei Han", "Shibo Wang", "Zhengdong Zhang", "Yonghui Wu", "Ruoming Pang"]
year: 2020
venue: "arXiv:2005.08100"
source: "https://arxiv.org/abs/2005.08100"
doi: "10.48550/arxiv.2005.08100"
arxiv: "2005.08100"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/gulati-2020-conformer-convolution-augmented-transformer-speech/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gulati-2020-conformer-convolution-augmented-transformer-speech/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/2/) / Conformer: Convolution-augmented Transformer for Speech Recognition

# Conformer: Convolution-augmented Transformer for Speech Recognition

**Authors:** [Anmol Gulati](https://md.research.elamur.ai/free/en/authors/anmol-gulati/), James Qin, [Chung-Cheng Chiu](https://md.research.elamur.ai/free/en/authors/chung-cheng-chiu/), [Niki Parmar](https://md.research.elamur.ai/free/en/authors/niki-parmar/), [Yu Zhang](https://md.research.elamur.ai/free/en/authors/yu-zhang/), [Jiahui Yu](https://md.research.elamur.ai/free/en/authors/jiahui-yu/), [Wei Han](https://md.research.elamur.ai/free/en/authors/wei-han/), [Shibo Wang](https://md.research.elamur.ai/free/en/authors/shibo-wang/), Zhengdong Zhang, [Yonghui Wu](https://md.research.elamur.ai/free/en/authors/yonghui-wu/), [Ruoming Pang](https://md.research.elamur.ai/free/en/authors/ruoming-pang/) · **Year:** 2020 · **Venue:** arXiv:2005.08100 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2005.08100) · [PDF](https://arxiv.org/pdf/2005.08100) · [DOI 10.48550/arxiv.2005.08100](https://doi.org/10.48550/arxiv.2005.08100) · [arXiv 2005.08100](https://arxiv.org/abs/2005.08100) · [OpenAlex](https://openalex.org/W3097777922)

**Citations:** 3070 citations (openalex-snapshot · 2026-10-14) · [363](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Speech recognition encoders built purely on self-attention handle long-range context well but struggle with fine local acoustic patterns. CNN-based encoders show the opposite trade-off. This Google paper introduces Conformer, an encoder block that pairs both mechanisms. Each block places a multi-head self-attention module and a convolution module between two feed-forward layers, following the Macaron-Net design with half-step residual weights, and ends with a layernorm. The attention module uses relative sinusoidal positional encoding borrowed from Transformer-XL. The convolution module chains a pointwise convolution with GLU gating, a 1-D depthwise convolution, batchnorm and Swish.

The encoder is trained as a transducer with a single-layer LSTM decoder on the LibriSpeech corpus with SpecAugment. The authors release no new dataset. Three model sizes (S, M, L) beat earlier Transformer and CNN systems at every parameter budget. The largest one set a new state of the art on LibriSpeech. Ablations indicate that the macaron feed-forward pair and placing convolution after attention both matter. The result shows that local and global modeling complement each other in a parameter-efficient way.

## Key points
- Data: LibriSpeech, with 970 hours of transcribed audio and an 800M-token text corpus used for the language model. Inputs are 80-channel filterbanks (25 ms window, 10 ms stride), augmented with SpecAugment.
- Conformer (L) has 118.8M parameters, 17 layers and dimension 512. It reaches 2.1%/4.3% WER on test/testother without an LM and 1.9%/3.9% with shallow fusion of a 3-layer LSTM LM.
- Conformer (S) has 10.3M parameters and still scores 2.7%/6.3%, which is better than ContextNet(S) on testother.
- Conformer (M) has 30.7M parameters and already surpasses the 139M-parameter Transformer Transducer. The large model improves on that baseline by about 15% relative on testother with an LM.
- All models were implemented in the Lingvo toolkit and trained with Adam using a Transformer-style learning-rate schedule.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

363 papers of our library cite it (of the 140884 whose bibliographies we read); 20 of them have a page here, 343 not yet.

- [FINDINGS OF THE IWSLT 2023 EVALUATION CAMPAIGN](https://md.research.elamur.ai/free/en/papers/agarwal-2023-findings-iwslt-2023-evaluation-campaign/)
- [Findings of the IWSLT 2022 Evaluation Campaign](https://md.research.elamur.ai/free/en/papers/anastasopoulos-2022-findings-iwslt-2022-evaluation-campaign/)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://md.research.elamur.ai/free/en/papers/baevski-2020-wav2vec-2-0-framework-self/)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://md.research.elamur.ai/free/en/papers/borsos-2022-audiolm-language-modeling-approach-audio/)
- [Towards One Model to Rule All: Multilingual Strategy for Dialectal Code-Switching Arabic ASR](https://md.research.elamur.ai/free/en/papers/chowdhury-2021-towards-one-model-rule-all/)
- [Seamless: Multilingual Expressive and Streaming Speech Translation](https://md.research.elamur.ai/free/en/papers/communication-2023-seamless-multilingual-expressive-streaming-speech/)
- [SeamlessM4T: Massively Multilingual & Multimodal Machine Translation](https://md.research.elamur.ai/free/en/papers/communication-2023-seamlessm4t-massively-multilingual-multimodal-machine/)
- [Open Automatic Speech Recognition Models for Classical and Modern Standard Arabic](https://md.research.elamur.ai/free/en/papers/grigoryan-2025-open-automatic-speech-recognition-models/)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://md.research.elamur.ai/free/en/papers/hsu-2021-hubert-self-supervised-speech-representation/)
- [ALADAN at IWSLT24 Low-resource Arabic Dialectal Speech Translation Task](https://md.research.elamur.ai/free/en/papers/kheder-2024-aladan-iwslt24-low-resource-arabic/)
- [HebDB: a Weakly Supervised Dataset for Hebrew Speech Processing](https://md.research.elamur.ai/free/en/papers/langstream-2024-turetzky-hebdb-weakly-supervised-dataset-hebrew/)
- [Mapudungun-Spanish Speech Translation: A Low-Resource End-to-End System for the IWSLT 2026 Shared Task](https://md.research.elamur.ai/free/en/papers/martinez-2026-mapudungun-spanish-speech-translation-low/)
- [Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs](https://md.research.elamur.ai/free/en/papers/microsoft-2025-phi-4-mini-technical-report/)
- [Team QUESPA System Submission for the IWSLT 2026 Dialectal and Low-resource Speech Translation Task](https://md.research.elamur.ai/free/en/papers/ortega-2026-team-quespa-system-submission-iwslt/)
- [Transformers learn in-context by gradient descent](https://md.research.elamur.ai/free/en/papers/oswald-2022-transformers-learn-context-gradient-descent/)
- [Supplementary Resources and Analysis for Automatic Speech Recognition Systems Trained on the Loquacious Dataset](https://md.research.elamur.ai/free/en/papers/rossenbach-2026-supplementary-resources-analysis-automatic-speech/)
- [AraSpot: Arabic Spoken Command Spotting](https://md.research.elamur.ai/free/en/papers/salhab-2023-araspot-arabic-spoken-command-spotting/)
- [AraSpell: A Deep Learning Approach for Arabic Spelling Correction](https://md.research.elamur.ai/free/en/papers/salhab-2024-araspell-deep-learning-approach-arabic/)
- [FalAR: A Large-scale Speaker-Annotated European Portuguese Speech Corpus of Parliamentary Sessions](https://md.research.elamur.ai/free/en/papers/teixeira-2026-falar-large-scale-speaker-annotated/)
- [Open Universal Arabic ASR Leaderboard](https://md.research.elamur.ai/free/en/papers/wang-2024-open-universal-arabic-asr-leaderboard/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{gulati2020conformer,
  title = {{Conformer: Convolution-augmented Transformer for Speech Recognition}},
  author = {Gulati, Anmol and Qin, James and Chiu, Chung-Cheng and Parmar, Niki and Zhang, Yu and Yu, Jiahui and Han, Wei and Wang, Shibo and Zhang, Zhengdong and Wu, Yonghui and Pang, Ruoming},
  year = {2020},
  doi = {10.48550/arxiv.2005.08100},
  eprint = {2005.08100},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2005.08100}
}
```

- [← previous Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning](https://md.research.elamur.ai/free/en/papers/gordon-2020-compressing-bert-studying-effects-weight/)
- [next → Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://md.research.elamur.ai/free/en/papers/ho-2020-constructing-multi-hop-qa-dataset/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/2/)

## References

35 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. C.-C. Chiu, T. N. Sainath, Y. Wu, R. Prabhavalkar, P. Nguyen, Z. Chen, A. Kannan, R. J. Weiss, K. Rao, E. Gonina et al., “State-of-the-art speech recognition with sequence-to-sequence models,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp.… [find](https://scholar.google.com/scholar?q=C%2E%2DC%2E+Chiu%2C+T%2E+N%2E+Sainath%2C+Y%2E+Wu%2C+R%2E+Prabhavalkar%2C+P%2E+Nguyen%2C+Z%2E+Chen%2C+A%2E+Kannan%2C+R%2E+J%2E+Weiss%2C+K%2E+Rao%2C+E%2E+Gonina+et+al%2E%2C+%E2%80%9CState%2Dof%2Dthe%2Dart+speech+recognition+with+sequence%2Dto%2Dsequence+models%2C%E2%80%9D+in+2018)
2. K. Rao, H. Sak, and R. Prabhavalkar, “Exploring architectures, data and units for streaming end-to-end speech recognition with rnn-transducer,” in 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2017, pp. 193–199. [find](https://scholar.google.com/scholar?q=K%2E+Rao%2C+H%2E+Sak%2C+and+R%2E+Prabhavalkar%2C+%E2%80%9CExploring+architectures%2C+data+and+units+for+streaming+end%2Dto%2Dend+speech+recognition+with+rnn%2Dtransducer%2C%E2%80%9D+in+2017+IEEE+Automatic+Speech+Recognition+and+Understand)
3. Y. He, T. N. Sainath, R. Prabhavalkar, I. McGraw, R. Alvarez, D. Zhao, D. Rybach, A. Kannan, Y. Wu, R. Pang, Q. Liang, D. Bhatia, Y. Shangguan, B. Li, G. Pundak, K. C. Sim, T. Bagby, S.-Y. Chang, K. Rao, and A. Gruenstein, “Streaming End-to-end Speech Recognition For Mobile Devices,” in Proc. ICASSP… [find](https://scholar.google.com/scholar?q=Y%2E+He%2C+T%2E+N%2E+Sainath%2C+R%2E+Prabhavalkar%2C+I%2E+McGraw%2C+R%2E+Alvarez%2C+D%2E+Zhao%2C+D%2E+Rybach%2C+A%2E+Kannan%2C+Y%2E+Wu%2C+R%2E+Pang%2C+Q%2E+Liang%2C+D%2E+Bhatia%2C+Y%2E+Shangguan%2C+B%2E+Li%2C+G%2E+Pundak%2C+K%2E+C%2E+Sim%2C+T%2E+Bagby%2C+S%2E%2DY%2E+Chang%2C+K%2E+R)
4. T. N. Sainath, Y. He, B. Li, A. Narayanan, R. Pang, A. Bruguier, S.-y. Chang, W. Li, R. Alvarez, Z. Chen, and et al., “A streaming on-device end-to-end model surpassing server-side conventional model quality and latency,” in ICASSP, 2020. [find](https://scholar.google.com/scholar?q=T%2E+N%2E+Sainath%2C+Y%2E+He%2C+B%2E+Li%2C+A%2E+Narayanan%2C+R%2E+Pang%2C+A%2E+Bruguier%2C+S%2E%2Dy%2E+Chang%2C+W%2E+Li%2C+R%2E+Alvarez%2C+Z%2E+Chen%2C+and+et+al%2E%2C+%E2%80%9CA+streaming+on%2Ddevice+end%2Dto%2Dend+model+surpassing+server%2Dside+conventional+model+)
5. A. Graves, “Sequence transduction with recurrent neural networks,” arXiv preprint arXiv:1211.3711, 2012. [find](https://scholar.google.com/scholar?q=A%2E+Graves%2C+%E2%80%9CSequence+transduction+with+recurrent+neural+networks%2C%E2%80%9D+arXiv+preprint+arXiv%3A1211%2E3711%2C+2012%2E) [our card](https://md.research.elamur.ai/free/en/papers/graves-2012-sequence-transduction-recurrent-neural-networks/)
6. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” 2017. [find](https://scholar.google.com/scholar?q=A%2E+Vaswani%2C+N%2E+Shazeer%2C+N%2E+Parmar%2C+J%2E+Uszkoreit%2C+L%2E+Jones%2C+A%2E+N%2E+Gomez%2C+L%2E+Kaiser%2C+and+I%2E+Polosukhin%2C+%E2%80%9CAttention+is+all+you+need%2C%E2%80%9D+2017%2E)
7. Q. Zhang, H. Lu, H. Sak, A. Tripathi, E. McDermott, S. Koo, and S. Kumar, “Transformer transducer: A streamable speech recognition model with transformer encoders and rnn-t loss,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 7… [find](https://scholar.google.com/scholar?q=Q%2E+Zhang%2C+H%2E+Lu%2C+H%2E+Sak%2C+A%2E+Tripathi%2C+E%2E+McDermott%2C+S%2E+Koo%2C+and+S%2E+Kumar%2C+%E2%80%9CTransformer+transducer%3A+A+streamable+speech+recognition+model+with+transformer+encoders+and+rnn%2Dt+loss%2C%E2%80%9D+in+ICASSP+2020%2D2020+)
8. J. Li, V. Lavrukhin, B. Ginsburg, R. Leary, O. Kuchaiev, J. M. Cohen, H. Nguyen, and R. T. Gadde, “Jasper: An end-to-end convolutional neural acoustic model,” arXiv preprint arXiv:1904.03288, 2019. — [arXiv:1904.03288](https://arxiv.org/abs/1904.03288) [find](https://scholar.google.com/scholar?q=J%2E+Li%2C+V%2E+Lavrukhin%2C+B%2E+Ginsburg%2C+R%2E+Leary%2C+O%2E+Kuchaiev%2C+J%2E+M%2E+Cohen%2C+H%2E+Nguyen%2C+and+R%2E+T%2E+Gadde%2C+%E2%80%9CJasper%3A+An+end%2Dto%2Dend+convolutional+neural+acoustic+model%2C%E2%80%9D+arXiv+preprint+arXiv%3A1904%2E03288%2C+2019%2E+%E2%80%94+)
9. S. Kriman, S. Beliaev, B. Ginsburg, J. Huang, O. Kuchaiev, V. Lavrukhin, R. Leary, J. Li, and Y. Zhang, “Quartznet: Deep automatic speech recognition with 1d time-channel separable convolutions,” arXiv preprint arXiv:1910.10261, 2019. — [arXiv:1910.10261](https://arxiv.org/abs/1910.10261) [find](https://scholar.google.com/scholar?q=S%2E+Kriman%2C+S%2E+Beliaev%2C+B%2E+Ginsburg%2C+J%2E+Huang%2C+O%2E+Kuchaiev%2C+V%2E+Lavrukhin%2C+R%2E+Leary%2C+J%2E+Li%2C+and+Y%2E+Zhang%2C+%E2%80%9CQuartznet%3A+Deep+automatic+speech+recognition+with+1d+time%2Dchannel+separable+convolutions%2C%E2%80%9D+arXi)
10. W. Han, Z. Zhang, Y. Zhang, J. Yu, C.-C. Chiu, J. Qin, A. Gulati, R. Pang, and Y. Wu, “Contextnet: Improving convolutional neural networks for automatic speech recognition with global context,” arXiv preprint arXiv:2005.03191, 2020. — [arXiv:2005.03191](https://arxiv.org/abs/2005.03191) [find](https://scholar.google.com/scholar?q=W%2E+Han%2C+Z%2E+Zhang%2C+Y%2E+Zhang%2C+J%2E+Yu%2C+C%2E%2DC%2E+Chiu%2C+J%2E+Qin%2C+A%2E+Gulati%2C+R%2E+Pang%2C+and+Y%2E+Wu%2C+%E2%80%9CContextnet%3A+Improving+convolutional+neural+networks+for+automatic+speech+recognition+with+global+context%2C%E2%80%9D+arXiv+)
11. T. N. Sainath, A.-r. Mohamed, B. Kingsbury, and B. Ramabhadran, “Deep convolutional neural networks for lvcsr,” in 2013 IEEE international conference on acoustics, speech and signal processing. IEEE, 2013, pp. 8614–8618. [find](https://scholar.google.com/scholar?q=T%2E+N%2E+Sainath%2C+A%2E%2Dr%2E+Mohamed%2C+B%2E+Kingsbury%2C+and+B%2E+Ramabhadran%2C+%E2%80%9CDeep+convolutional+neural+networks+for+lvcsr%2C%E2%80%9D+in+2013+IEEE+international+conference+on+acoustics%2C+speech+and+signal+processing%2E+IEEE%2C+)
12. O. Abdel-Hamid, A.-r. Mohamed, H. Jiang, L. Deng, G. Penn, and D. Yu, “Convolutional neural networks for speech recognition,” IEEE/ACM Transactions on audio, speech, and language processing, vol. 22, no. 10, pp. 1533–1545, 2014. [find](https://scholar.google.com/scholar?q=O%2E+Abdel%2DHamid%2C+A%2E%2Dr%2E+Mohamed%2C+H%2E+Jiang%2C+L%2E+Deng%2C+G%2E+Penn%2C+and+D%2E+Yu%2C+%E2%80%9CConvolutional+neural+networks+for+speech+recognition%2C%E2%80%9D+IEEE%2FACM+Transactions+on+audio%2C+speech%2C+and+language+processing%2C+vol%2E+22%2C+)
13. J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7132–7141. [find](https://scholar.google.com/scholar?q=J%2E+Hu%2C+L%2E+Shen%2C+and+G%2E+Sun%2C+%E2%80%9CSqueeze%2Dand%2Dexcitation+networks%2C%E2%80%9D+in+Proceedings+of+the+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+2018%2C+pp%2E+7132%E2%80%937141%2E)
14. I. Bello, B. Zoph, A. Vaswani, J. Shlens, and Q. V. Le, “Attention augmented convolutional networks,” in Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 3286– 3295. [find](https://scholar.google.com/scholar?q=I%2E+Bello%2C+B%2E+Zoph%2C+A%2E+Vaswani%2C+J%2E+Shlens%2C+and+Q%2E+V%2E+Le%2C+%E2%80%9CAttention+augmented+convolutional+networks%2C%E2%80%9D+in+Proceedings+of+the+IEEE+International+Conference+on+Computer+Vision%2C+2019%2C+pp%2E+3286%E2%80%93+3295%2E)
15. B. Yang, L. Wang, D. Wong, L. S. Chao, and Z. Tu, “Convolutional self-attention networks,” arXiv preprint arXiv:1904.03107, 2019. [find](https://scholar.google.com/scholar?q=B%2E+Yang%2C+L%2E+Wang%2C+D%2E+Wong%2C+L%2E+S%2E+Chao%2C+and+Z%2E+Tu%2C+%E2%80%9CConvolutional+self%2Dattention+networks%2C%E2%80%9D+arXiv+preprint+arXiv%3A1904%2E03107%2C+2019%2E)
16. A. W. Yu, D. Dohan, M.-T. Luong, R. Zhao, K. Chen, M. Norouzi, and Q. V. Le, “Qanet: Combining local convolution with global self-attention for reading comprehension,” arXiv preprint arXiv:1804.09541, 2018. [find](https://scholar.google.com/scholar?q=A%2E+W%2E+Yu%2C+D%2E+Dohan%2C+M%2E%2DT%2E+Luong%2C+R%2E+Zhao%2C+K%2E+Chen%2C+M%2E+Norouzi%2C+and+Q%2E+V%2E+Le%2C+%E2%80%9CQanet%3A+Combining+local+convolution+with+global+self%2Dattention+for+reading+comprehension%2C%E2%80%9D+arXiv+preprint+arXiv%3A1804%2E09541%2C) [our card](https://md.research.elamur.ai/free/en/papers/yu-2018-qanet-combining-local-convolution-global/)
17. Z. Wu, Z. Liu, J. Lin, Y. Lin, and S. Han, “Lite transformer with long-short range attention,” arXiv preprint arXiv:2004.11886, 2020. [find](https://scholar.google.com/scholar?q=Z%2E+Wu%2C+Z%2E+Liu%2C+J%2E+Lin%2C+Y%2E+Lin%2C+and+S%2E+Han%2C+%E2%80%9CLite+transformer+with+long%2Dshort+range+attention%2C%E2%80%9D+arXiv+preprint+arXiv%3A2004%2E11886%2C+2020%2E)
18. Y. Lu, Z. Li, D. He, Z. Sun, B. Dong, T. Qin, L. Wang, and T.-Y. Liu, “Understanding and improving transformer from a multi-particle dynamic system point of view,” arXiv preprint arXiv:1906.02762, 2019. — [arXiv:1906.02762](https://arxiv.org/abs/1906.02762) [find](https://scholar.google.com/scholar?q=Y%2E+Lu%2C+Z%2E+Li%2C+D%2E+He%2C+Z%2E+Sun%2C+B%2E+Dong%2C+T%2E+Qin%2C+L%2E+Wang%2C+and+T%2E%2DY%2E+Liu%2C+%E2%80%9CUnderstanding+and+improving+transformer+from+a+multi%2Dparticle+dynamic+system+point+of+view%2C%E2%80%9D+arXiv+preprint+arXiv%3A1906%2E02762%2C+201)
19. S. Karita, N. Chen, T. Hayashi, T. Hori, H. Inaguma, Z. Jiang, M. Someki, N. E. Y. Soplin, R. Yamamoto, X. Wang et al., “A comparative study on transformer vs rnn in speech applications,” arXiv preprint arXiv:1909.06317, 2019. — [arXiv:1909.06317](https://arxiv.org/abs/1909.06317) [find](https://scholar.google.com/scholar?q=S%2E+Karita%2C+N%2E+Chen%2C+T%2E+Hayashi%2C+T%2E+Hori%2C+H%2E+Inaguma%2C+Z%2E+Jiang%2C+M%2E+Someki%2C+N%2E+E%2E+Y%2E+Soplin%2C+R%2E+Yamamoto%2C+X%2E+Wang+et+al%2E%2C+%E2%80%9CA+comparative+study+on+transformer+vs+rnn+in+speech+applications%2C%E2%80%9D+arXiv+prepri)
20. Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. V. Le, and R. Salakhutdinov, “Transformer-xl: Attentive language models beyond a fixed-length context,” 2019. [find](https://scholar.google.com/scholar?q=Z%2E+Dai%2C+Z%2E+Yang%2C+Y%2E+Yang%2C+J%2E+Carbonell%2C+Q%2E+V%2E+Le%2C+and+R%2E+Salakhutdinov%2C+%E2%80%9CTransformer%2Dxl%3A+Attentive+language+models+beyond+a+fixed%2Dlength+context%2C%E2%80%9D+2019%2E)
21. Q. Wang, B. Li, T. Xiao, J. Zhu, C. Li, D. F. Wong, and L. S. Chao, “Learning deep transformer models for machine translation,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Jul. 2019, pp. 1810–1822. [find](https://scholar.google.com/scholar?q=Q%2E+Wang%2C+B%2E+Li%2C+T%2E+Xiao%2C+J%2E+Zhu%2C+C%2E+Li%2C+D%2E+F%2E+Wong%2C+and+L%2E+S%2E+Chao%2C+%E2%80%9CLearning+deep+transformer+models+for+machine+translation%2C%E2%80%9D+in+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computa)
22. T. Q. Nguyen and J. Salazar, “Transformers without tears: Improving the normalization of self-attention,” arXiv preprint arXiv:1910.05895, 2019. [find](https://scholar.google.com/scholar?q=T%2E+Q%2E+Nguyen+and+J%2E+Salazar%2C+%E2%80%9CTransformers+without+tears%3A+Improving+the+normalization+of+self%2Dattention%2C%E2%80%9D+arXiv+preprint+arXiv%3A1910%2E05895%2C+2019%2E)
23. Y. N. Dauphin, A. Fan, M. Auli, and D. Grangier, “Language modeling with gated convolutional networks,” in Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017, pp. 933–941. [find](https://scholar.google.com/scholar?q=Y%2E+N%2E+Dauphin%2C+A%2E+Fan%2C+M%2E+Auli%2C+and+D%2E+Grangier%2C+%E2%80%9CLanguage+modeling+with+gated+convolutional+networks%2C%E2%80%9D+in+Proceedings+of+the+34th+International+Conference+on+Machine+Learning%2DVolume+70%2E+JMLR%2E+org%2C+20) [our card](https://md.research.elamur.ai/free/en/papers/dauphin-2016-language-modeling-gated-convolutional-networks/)
24. L. Dong, S. Xu, and B. Xu, “Speech-transformer: a no-recurrence sequence-to-sequence model for speech recognition,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 5884–5888. [find](https://scholar.google.com/scholar?q=L%2E+Dong%2C+S%2E+Xu%2C+and+B%2E+Xu%2C+%E2%80%9CSpeech%2Dtransformer%3A+a+no%2Drecurrence+sequence%2Dto%2Dsequence+model+for+speech+recognition%2C%E2%80%9D+in+2018+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+%28IC)
25. P. Ramachandran, B. Zoph, and Q. V. Le, “Searching for activation functions,” arXiv preprint arXiv:1710.05941, 2017. [find](https://scholar.google.com/scholar?q=P%2E+Ramachandran%2C+B%2E+Zoph%2C+and+Q%2E+V%2E+Le%2C+%E2%80%9CSearching+for+activation+functions%2C%E2%80%9D+arXiv+preprint+arXiv%3A1710%2E05941%2C+2017%2E)
26. V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2015, pp. 5206–5210. [find](https://scholar.google.com/scholar?q=V%2E+Panayotov%2C+G%2E+Chen%2C+D%2E+Povey%2C+and+S%2E+Khudanpur%2C+%E2%80%9CLibrispeech%3A+an+asr+corpus+based+on+public+domain+audio+books%2C%E2%80%9D+in+2015+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+%28IC)
27. D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “Specaugment: A simple data augmentation method for automatic speech recognition,” arXiv preprint arXiv:1904.08779, 2019. [find](https://scholar.google.com/scholar?q=D%2E+S%2E+Park%2C+W%2E+Chan%2C+Y%2E+Zhang%2C+C%2E%2DC%2E+Chiu%2C+B%2E+Zoph%2C+E%2E+D%2E+Cubuk%2C+and+Q%2E+V%2E+Le%2C+%E2%80%9CSpecaugment%3A+A+simple+data+augmentation+method+for+automatic+speech+recognition%2C%E2%80%9D+arXiv+preprint+arXiv%3A1904%2E08779%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/park-2019-specaugment-simple-data-augmentation-method/)
28. D. S. Park, Y. Zhang, C.-C. Chiu, Y. Chen, B. Li, W. Chan, Q. V. Le, and Y. Wu, “Specaugment on large scale datasets,” arXiv preprint arXiv:1912.05533, 2019. — [arXiv:1912.05533](https://arxiv.org/abs/1912.05533) [find](https://scholar.google.com/scholar?q=D%2E+S%2E+Park%2C+Y%2E+Zhang%2C+C%2E%2DC%2E+Chiu%2C+Y%2E+Chen%2C+B%2E+Li%2C+W%2E+Chan%2C+Q%2E+V%2E+Le%2C+and+Y%2E+Wu%2C+%E2%80%9CSpecaugment+on+large+scale+datasets%2C%E2%80%9D+arXiv+preprint+arXiv%3A1912%2E05533%2C+2019%2E+%E2%80%94+%5BarXiv%3A1912%2E05533%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs)
29. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” Journal of Machine Learning Research, vol. 15, no. 56, pp. 1929–1958, 2014. [find](https://scholar.google.com/scholar?q=N%2E+Srivastava%2C+G%2E+Hinton%2C+A%2E+Krizhevsky%2C+I%2E+Sutskever%2C+and+R%2E+Salakhutdinov%2C+%E2%80%9CDropout%3A+A+simple+way+to+prevent+neural+networks+from+overfitting%2C%E2%80%9D+Journal+of+Machine+Learning+Research%2C+vol%2E+15%2C+no%2E+56%2C)
30. K.-C. Jim, C. L. Giles, and B. G. Horne, “An analysis of noise in recurrent neural networks: convergence and generalization,” IEEE Transactions on neural networks, vol. 7, no. 6, pp. 1424– 1438, 1996. [find](https://scholar.google.com/scholar?q=K%2E%2DC%2E+Jim%2C+C%2E+L%2E+Giles%2C+and+B%2E+G%2E+Horne%2C+%E2%80%9CAn+analysis+of+noise+in+recurrent+neural+networks%3A+convergence+and+generalization%2C%E2%80%9D+IEEE+Transactions+on+neural+networks%2C+vol%2E+7%2C+no%2E+6%2C+pp%2E+1424%E2%80%93+1438%2C+1996%2E)
31. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014. [find](https://scholar.google.com/scholar?q=D%2E+P%2E+Kingma+and+J%2E+Ba%2C+%E2%80%9CAdam%3A+A+method+for+stochastic+optimization%2C%E2%80%9D+arXiv+preprint+arXiv%3A1412%2E6980%2C+2014%2E)
32. J. Shen, P. Nguyen, Y. Wu, Z. Chen, and et al., “Lingvo: a modular and scalable framework for sequence-to-sequence modeling,” 2019. [find](https://scholar.google.com/scholar?q=J%2E+Shen%2C+P%2E+Nguyen%2C+Y%2E+Wu%2C+Z%2E+Chen%2C+and+et+al%2E%2C+%E2%80%9CLingvo%3A+a+modular+and+scalable+framework+for+sequence%2Dto%2Dsequence+modeling%2C%E2%80%9D+2019%2E)
33. Y. Wang, A. Mohamed, D. Le, C. Liu, A. Xiao, J. Mahadeokar, H. Huang, A. Tjandra, X. Zhang, F. Zhang et al., “Transformer-based acoustic modeling for hybrid speech recognition,” arXiv preprint arXiv:1910.09799, 2019. — [arXiv:1910.09799](https://arxiv.org/abs/1910.09799) [find](https://scholar.google.com/scholar?q=Y%2E+Wang%2C+A%2E+Mohamed%2C+D%2E+Le%2C+C%2E+Liu%2C+A%2E+Xiao%2C+J%2E+Mahadeokar%2C+H%2E+Huang%2C+A%2E+Tjandra%2C+X%2E+Zhang%2C+F%2E+Zhang+et+al%2E%2C+%E2%80%9CTransformer%2Dbased+acoustic+modeling+for+hybrid+speech+recognition%2C%E2%80%9D+arXiv+preprint+arXiv%3A1)
34. G. Synnaeve, Q. Xu, J. Kahn, T. Likhomanenko, E. Grave, V. Pratap, A. Sriram, V. Liptchinsky, and R. Collobert, “End-to-end asr: from supervised to semi-supervised learning with modern architectures,” 2019. [find](https://scholar.google.com/scholar?q=G%2E+Synnaeve%2C+Q%2E+Xu%2C+J%2E+Kahn%2C+T%2E+Likhomanenko%2C+E%2E+Grave%2C+V%2E+Pratap%2C+A%2E+Sriram%2C+V%2E+Liptchinsky%2C+and+R%2E+Collobert%2C+%E2%80%9CEnd%2Dto%2Dend+asr%3A+from+supervised+to+semi%2Dsupervised+learning+with+modern+architectures%2C%E2%80%9D)
35. F. Wu, A. Fan, A. Baevski, Y. N. Dauphin, and M. Auli, “Pay less attention with lightweight and dynamic convolutions,” arXiv preprint arXiv:1901.10430, 2019. [find](https://scholar.google.com/scholar?q=F%2E+Wu%2C+A%2E+Fan%2C+A%2E+Baevski%2C+Y%2E+N%2E+Dauphin%2C+and+M%2E+Auli%2C+%E2%80%9CPay+less+attention+with+lightweight+and+dynamic+convolutions%2C%E2%80%9D+arXiv+preprint+arXiv%3A1901%2E10430%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/wu-2019-pay-less-attention-lightweight-dynamic/)