---
title: "Robust Speech Recognition via Large-Scale — Radford 2022"
description: "Self-supervised speech encoders such as Wav2Vec 2.0 learn strong audio representations, but they need fine-tuning on each target dataset before they can…"
authors: ["Alec Radford", "Jong Wook Kim", "Tao Xu", "Greg Brockman", "Christine McLeavey", "Ilya Sutskever"]
year: 2022
venue: "arXiv:2212.04356"
source: "https://arxiv.org/abs/2212.04356"
doi: "10.48550/arxiv.2212.04356"
arxiv: "2212.04356"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/radford-2022-robust-speech-recognition-large-scale/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/radford-2022-robust-speech-recognition-large-scale/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) / Robust Speech Recognition via Large-Scale Weak Supervision

# Robust Speech Recognition via Large-Scale Weak Supervision

**Authors:** [Alec Radford](https://md.research.elamur.ai/free/en/authors/alec-radford/), [Jong Wook Kim](https://md.research.elamur.ai/free/en/authors/jong-wook-kim/), [Tao Xu](https://md.research.elamur.ai/free/en/authors/tao-xu/), [Greg Brockman](https://md.research.elamur.ai/free/en/authors/greg-brockman/), [Christine McLeavey](https://md.research.elamur.ai/free/en/authors/christine-mcleavey/), [Ilya Sutskever](https://md.research.elamur.ai/free/en/authors/ilya-sutskever/) · **Year:** 2022 · **Venue:** arXiv:2212.04356 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2212.04356) · [PDF](https://arxiv.org/pdf/2212.04356) · [DOI 10.48550/arxiv.2212.04356](https://doi.org/10.48550/arxiv.2212.04356) · [arXiv 2212.04356](https://arxiv.org/abs/2212.04356) · [OpenAlex](https://openalex.org/W4311000453)

**Citations:** 1134 citations (openalex-snapshot · 2026-10-14) · [1354](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Self-supervised speech encoders such as Wav2Vec 2.0 learn strong audio representations, but they need fine-tuning on each target dataset before they can transcribe. That step is laborious, and it can make models overfit to dataset-specific quirks. Whisper takes the opposite route. The authors collected 680,000 hours of web audio paired with existing transcripts, which is far more labeled speech than earlier supervised or weakly supervised collections. They cleaned the data mostly automatically: heuristics remove captions that were likely produced by other ASR systems, an audio language detector checks that speech and text languages agree, near-duplicate transcripts are removed, and a manual audit targets poor data sources. An ordinary encoder-decoder Transformer is then trained on many tasks at once (multilingual transcription, translation into English, language identification and voice activity detection), with the task signalled by special decoder tokens. Without any fine-tuning, the resulting models often rival fully supervised systems on standard benchmarks and come close to human accuracy and robustness. This shows that scaling weak supervision is an underused path to dependable speech recognition, and the released models give a ready foundation for further work.

## Key points
- The training corpus has 680,000 hours in total. Of these, 117,000 hours cover 96 non-English languages and 125,000 hours are X→en speech translation pairs. For comparison, SpeechStew combined 5,140 hours, and earlier weakly supervised sets reached 10,000–30,000 hours.
- The model is an off-the-shelf encoder-decoder Transformer:
  - Input is 16 kHz audio cut into 30-second chunks and converted to 80-channel log-Mel spectrograms.
  - Text uses the byte-level BPE tokenizer from GPT-2, with the vocabulary refit for the multilingual models.
- Data filtering includes:
  - a language detector fine-tuned on VoxLingua107 and checked against CLD2;
  - fuzzy de-duplication of transcripts;
  - removal of training transcripts that overlap with TED-LIUM 3, to avoid evaluation contamination.
- Whisper uses no self-supervised pre-training and no self-training. Its models and inference code are public at github.com/openai/whisper.
- The authors report several limitations:
  - Most languages have under 1,000 hours of training data, and accuracy on them is still weak.
  - Long-form transcription can get stuck in repetition loops or produce hallucinated text.
  - Fine-tuning has not yet been studied.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

1354 papers of our library cite it (of the 140673 whose bibliographies we read); 103 of them have a page here, the first 100 below, 1251 not yet.

- [Leveraging Data Collection and Unsupervised Learning for Code-switched Tunisian Arabic Automatic Speech Recognition](https://md.research.elamur.ai/free/en/papers/abdallah-2023-leveraging-data-collection-unsupervised-learning/)
- [Benchmarking Commercial ASR Systems on Code-Switching Speech: Arabic, Persian, and German](https://md.research.elamur.ai/free/en/papers/abdoli-2026-benchmarking-commercial-asr-systems-code/)
- [Beyond Orthography: Automatic Recovery of Short Vowels and Dialectal Sounds in Arabic](https://md.research.elamur.ai/free/en/papers/acl-2024-kheir-beyond-orthography-automatic-recovery-short/)
- [FINDINGS OF THE IWSLT 2023 EVALUATION CAMPAIGN](https://md.research.elamur.ai/free/en/papers/agarwal-2023-findings-iwslt-2023-evaluation-campaign/)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://md.research.elamur.ai/free/en/papers/agarwal-2023-policy-distillation-language-models-learning/)
- [Bridging Attribution and Open-Set Detection using Graph-Augmented Instance Learning in Synthetic Speech](https://md.research.elamur.ai/free/en/papers/akhtar-2026-bridging-attribution-open-set-detection/)
- [Poem Meter Classification of Recited Arabic Poetry: Integrating High-Resource Systems for a Low-Resource Task](https://md.research.elamur.ai/free/en/papers/al-2025-shaibani-poem-meter-classification-recited-arabic/)
- [OASIS: A Multilingual and Multimodal Dataset for Culturally Grounded Spoken Visual QA](https://md.research.elamur.ai/free/en/papers/alam-2025-oasis-multilingual-multimodal-dataset-culturally/)
- [Diacritic Recognition Performance in Arabic ASR](https://md.research.elamur.ai/free/en/papers/aldarmaki-2023-diacritic-recognition-performance-arabic-asr/)
- [A Bilingual Bimodal Benchmark for Arabic-English NLP across Grammatical Correction, Essay Scoring, Morphological Tagging, and Speech Recognition](https://md.research.elamur.ai/free/en/papers/alhafni-2026-bilingual-bimodal-benchmark-arabic-english/)
- [Arabic Little STT: Arabic Children Speech Recognition Dataset](https://md.research.elamur.ai/free/en/papers/alkadri-2025-arabic-little-stt-arabic-children/)
- [Prediction of Arabic Legal Rulings using Large Language Models](https://md.research.elamur.ai/free/en/papers/ammar-2023-prediction-arabic-legal-rulings-using/)
- [Bulbul: A Dataset for Dialectal Arabic Speech Recognition](https://md.research.elamur.ai/free/en/papers/ashraf-2026-bulbul-dataset-dialectal-arabic-speech/)
- [Potential of ASR for the study of L2 learner corpora](https://md.research.elamur.ai/free/en/papers/ayari-2024-potential-asr-study-l2-learner/)
- [Multi-Task Instruction Tuning via Data Scheduling for Low-Resource Arabic SpeechLLMs](https://md.research.elamur.ai/free/en/papers/bhatti-2026-multi-task-instruction-tuning-data/)
- [TutlAit v1: a crowdsourced Moroccan Tamazight speech dataset with Arabic transcriptions and regional accent labels](https://md.research.elamur.ai/free/en/papers/chadi-2026-tutlait-v1-crowdsourced-moroccan-tamazight/)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://md.research.elamur.ai/free/en/papers/chu-2023-qwen-audio-advancing-universal/)
- [Qwen2-Audio Technical Report](https://md.research.elamur.ai/free/en/papers/chu-2024-qwen2-audio-technical-report/)
- [Seamless: Multilingual Expressive and Streaming Speech Translation](https://md.research.elamur.ai/free/en/papers/communication-2023-seamless-multilingual-expressive-streaming-speech/)
- [SeamlessM4T: Massively Multilingual & Multimodal Machine Translation](https://md.research.elamur.ai/free/en/papers/communication-2023-seamlessm4t-massively-multilingual-multimodal-machine/)
- [Moshi: a speech-text foundation model for real-time dialogue](https://md.research.elamur.ai/free/en/papers/defossez-2024-moshi-speech-text-foundation-model/)
- [ADI-20: Arabic Dialect Identification dataset and models](https://md.research.elamur.ai/free/en/papers/elleuch-2025-adi-20-arabic-dialect-identification/)
- [ArFake: A Robust Framework for Multi-Dialect Arabic Speech Spoofing Detection Benchmark](https://md.research.elamur.ai/free/en/papers/elsetohy-2025-arfake-robust-framework-multi-dialect/)
- [CTC-DID: CTC-Based Arabic dialect identification for streaming applications](https://md.research.elamur.ai/free/en/papers/farooq-2026-ctc-did-ctc-based-arabic/)
- [Gemini: A Family of Highly Capable Multimodal Models](https://md.research.elamur.ai/free/en/papers/gemini-2023-team-gemini-family-highly-capable-multimodal/)
- [Almieyar: A Culturally Grounded Benchmark for Multi-Dialect Arabic Speech Recognition](https://md.research.elamur.ai/free/en/papers/ghahroodi-2026-almieyar-culturally-grounded-benchmark-multi/)
- [Open Automatic Speech Recognition Models for Classical and Modern Standard Arabic](https://md.research.elamur.ai/free/en/papers/grigoryan-2025-open-automatic-speech-recognition-models/)
- [Hallucinations in Large Multilingual Translation Models](https://md.research.elamur.ai/free/en/papers/guerreiro-2023-hallucinations-large-multilingual-translation-models/)
- [Arabic Multimodal Machine Learning: Datasets, Applications, Approaches, and Challenges](https://md.research.elamur.ai/free/en/papers/haouhat-2025-arabic-multimodal-machine-learning-datasets/)
- [Challenges and Applications of Large Language Models](https://md.research.elamur.ai/free/en/papers/kaddour-2023-challenges-applications-large-language-models/)
- [Learning the Arabic Dialect Continuum as a Continuous Space: A Regression Approach to Speaker Origin Prediction](https://md.research.elamur.ai/free/en/papers/khadraoui-2026-learning-arabic-dialect-continuum-continuous/)
- [Findings of the WMT24 General Machine Translation Shared Task: The LLM Era Is Here but MT Is Not Solved Yet](https://md.research.elamur.ai/free/en/papers/kocmi-2024-findings-wmt24-general-machine-translation/)
- [Findings of the WMT25 General Machine Translation Shared Task: Time to Stop Evaluating on Easy Test Sets](https://md.research.elamur.ai/free/en/papers/kocmi-2025-findings-wmt25-general-machine-translation/)
- [Enhancing Quranic Learning: A Multimodal Deep Learning Approach for Arabic Phoneme Recognition](https://md.research.elamur.ai/free/en/papers/kucukmanisa-2025-enhancing-quranic-learning-multimodal-deep/)
- [ivrit.ai: A Comprehensive Dataset of Hebrew Speech for AI Research and Development](https://md.research.elamur.ai/free/en/papers/langstream-2023-marmor-ivrit-ai-comprehensive-dataset-hebrew/)
- [A Language Modeling Approach to Diacritic-Free Hebrew TTS](https://md.research.elamur.ai/free/en/papers/langstream-2024-roth-language-modeling-approach-diacritic-free/)
- [HebDB: a Weakly Supervised Dataset for Hebrew Speech Processing](https://md.research.elamur.ai/free/en/papers/langstream-2024-turetzky-hebdb-weakly-supervised-dataset-hebrew/)
- [Enhancing TTS Stability in Hebrew using Discrete Semantic Units](https://md.research.elamur.ai/free/en/papers/langstream-2024-zeldes-enhancing-tts-stability-hebrew-using/)
- [VoxKnesset: A Large-Scale Longitudinal Hebrew Speech Dataset for Aging Speaker Modeling](https://md.research.elamur.ai/free/en/papers/langstream-2026-marmor-voxknesset-large-scale-longitudinal-hebrew/)
- [VideoChat: Chat-Centric Video Understanding](https://md.research.elamur.ai/free/en/papers/li-2023-videochat-chat-centric-video-understanding/)
- [Assessing Suicide Risk in Arabic Crisis Helpline Calls: A Comparison of Arabic and English Large Language Models](https://md.research.elamur.ai/free/en/papers/ma-2026-assessing-suicide-risk-arabic-crisis/)
- [Adaptive Emotion Management in Human-robot Dialogue using Online Group Relative Policy Optimization](https://md.research.elamur.ai/free/en/papers/manaseryan-2026-adaptive-emotion-management-human-robot/)
- [Automatic Prediction of Prominence and Boundary Strength from Text](https://md.research.elamur.ai/free/en/papers/mas-2026-automatic-prediction-prominence-boundary-strength/)
- [Phonetic-based Ranking for Improved Pseudo-Labeling in Low-Resource ASR](https://md.research.elamur.ai/free/en/papers/matassoni-2026-phonetic-based-ranking-improved-pseudo/)
- [Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs](https://md.research.elamur.ai/free/en/papers/microsoft-2025-phi-4-mini-technical-report/)
- [From LLM Prompts to Acoustic Baselines: A Scalable Pipeline for Under-Resourced Disfluent Code-Mixed Speech](https://md.research.elamur.ai/free/en/papers/mitra-2026-llm-prompts-acoustic-baselines-scalable/)
- [Fleurs-Badini: Translation and Recording Fleurs Dataset for Badini Variant of Northern Kurdish](https://md.research.elamur.ai/free/en/papers/mohammadamini-2026-fleurs-badini-translation-recording-fleurs/)
- [LIUM Submission for IWSLT 2026 Low-resource Speech Translation Track](https://md.research.elamur.ai/free/en/papers/mohammadamini-2026-lium-submission-iwslt-2026-low/)
- [Southern Kurdish Speech Recognition Resources and Benchmarking](https://md.research.elamur.ai/free/en/papers/mohammadamini-2026-southern-kurdish-speech-recognition-resources/)
- [DialogueSidon: Recovering Full-Duplex Dialogue Tracks from In-the-Wild Dialogue Audio](https://md.research.elamur.ai/free/en/papers/nakata-2026-dialoguesidon-recovering-full-duplex-dialogue/)
- [Fine-tuning Whisper with Spontaneous Persian Speech (SPS)](https://md.research.elamur.ai/free/en/papers/namdarzadeh-2026-fine-tuning-whisper-spontaneous-persian/)
- [LinTO Audio and Textual Datasets to Train and Evaluate Automatic Speech Recognition in Tunisian Arabic Dialect](https://md.research.elamur.ai/free/en/papers/naouara-2025-linto-audio-textual-datasets-train/)
- [A Comprehensive Overview of Large Language Models](https://md.research.elamur.ai/free/en/papers/naveed-2023-comprehensive-overview-large-language-models/)
- [ViMedCSS: A Vietnamese Medical Code-Switching Speech Dataset & Benchmark](https://md.research.elamur.ai/free/en/papers/nguyen-2026-vimedcss-vietnamese-medical-code-switching/)
- [A New Benchmark for Evaluating Automatic Speech Recognition in the Arabic Call Domain](https://md.research.elamur.ai/free/en/papers/obaidah-2024-new-benchmark-evaluating-automatic-speech/)
- [Amulwe Kimün: A Community-Grounded Demo, Resource, and ASR Baseline for Mapuzugun](https://md.research.elamur.ai/free/en/papers/oliva-2026-amulwe-kimun-community-grounded-demo/)
- [TELÓ: AI-Driven Automatic Subtitling for the Promotion of the Performing Arts](https://md.research.elamur.ai/free/en/papers/oliver-2026-telo-ai-driven-automatic-subtitling/)
- [A Pocket Offline Model for Simultaneous Speech Translation as CUNI Submission to IWSLT 2026](https://md.research.elamur.ai/free/en/papers/ortega-2026-pocket-offline-model-simultaneous-speech/)
- [Team QUESPA System Submission for the IWSLT 2026 Dialectal and Low-resource Speech Translation Task](https://md.research.elamur.ai/free/en/papers/ortega-2026-team-quespa-system-submission-iwslt/)
- [Overcoming Data Scarcity in Multi-Dialectal Arabic ASR via Whisper Fine-Tuning](https://md.research.elamur.ai/free/en/papers/ozyilmaz-2025-overcoming-data-scarcity-multi-dialectal/)
- [Toward Equitable Machine Translation for Atypical Speech: An LLM Post-Correction Approach](https://md.research.elamur.ai/free/en/papers/pasion-2026-equitable-machine-translation-atypical-speech/)
- [SagaQA: A Multi-hop Reasoning Benchmark for Long-form Narrative Understanding in TV Series](https://md.research.elamur.ai/free/en/papers/pennec-2026-sagaqa-multi-hop-reasoning-benchmark/)
- [Scaling Speech Technology to 1,000+ Languages](https://md.research.elamur.ai/free/en/papers/pratap-2023-scaling-speech-technology-1-000/)
- [A Parameter-Efficient Learning Approach to Arabic Dialect Identification with Pre-Trained General-Purpose Speech Model](https://md.research.elamur.ai/free/en/papers/radhakrishnan-2023-parameter-efficient-learning-approach-arabic/)
- [Leveraging Semi-Supervised Learning for Multimodal Hate Speech Data Annotation and Detection](https://md.research.elamur.ai/free/en/papers/rammohan-2026-leveraging-semi-supervised-learning-multimodal/)
- [AudioPaLM: A Large Language Model That Can Speak and Listen](https://md.research.elamur.ai/free/en/papers/rubenstein-2023-audiopalm-large-language-model-that/)
- [Listening for Ideology: Automatic Analysis of Character Speech in Historical Nazi Propaganda Films](https://md.research.elamur.ai/free/en/papers/ruth-2026-listening-ideology-automatic-analysis-character/)
- [Ramsa: A Large Sociolinguistically Rich Emirati Arabic Speech Corpus for ASR and TTS](https://md.research.elamur.ai/free/en/papers/sabbagh-2026-ramsa-large-sociolinguistically-rich-emirati/)
- [Regional Variation in the Performance of ASR Models on Croatian and Serbian](https://md.research.elamur.ai/free/en/papers/samardzic-2026-regional-variation-performance-asr-models/)
- [MLLP-VRAIN UPV System for the IWSLT 2026 Simultaneous Speech Translation Task](https://md.research.elamur.ai/free/en/papers/sanchez-2026-mllp-vrain-upv-system-iwslt/)
- [A Benchmark Dataset and Comparative Evaluation of Phonemized and Romanized Urdu for Text-to-Speech](https://md.research.elamur.ai/free/en/papers/shahid-2026-benchmark-dataset-comparative-evaluation-phonemized/)
- [Is Semi-Automatic Transcription Useful in Corpus Creation? Preliminary Considerations on the KIParla Corpus](https://md.research.elamur.ai/free/en/papers/simonotti-2026-is-semi-automatic-transcription-useful/)
- [CrossOracle: An Agentic Framework for Real-Time Expert Personas with Parallelized Acoustic Synthesis](https://md.research.elamur.ai/free/en/papers/singh-2026-crossoracle-agentic-framework-real-time/)
- [Fine-tuning Whisper Across 81 Languages](https://md.research.elamur.ai/free/en/papers/singh-2026-fine-tuning-whisper-across-81/)
- [IIIT-BGP IWSLT 2026 Systems for Low-resource ST](https://md.research.elamur.ai/free/en/papers/singh-2026-iiit-bgp-iwslt-2026-systems/)
- [MUSCAT: MUltilingual, SCientific ConversATion Benchmark](https://md.research.elamur.ai/free/en/papers/sinhamahapatra-2026-muscat-multilingual-scientific-conversation-benchmark/)
- [Voice Activation Detection for Transcription of Indigenous Languages](https://md.research.elamur.ai/free/en/papers/solano-2026-voice-activation-detection-transcription-indigenous/)
- [Automatic Speech Recognition for Documenting Endangered Languages: Case Study of Ikema Miyakoan](https://md.research.elamur.ai/free/en/papers/taguchi-2026-automatic-speech-recognition-documenting-endangered/)
- [N-Shot Benchmarking of Whisper on Diverse Arabic Speech Recognition](https://md.research.elamur.ai/free/en/papers/talafha-2023-n-shot-benchmarking-whisper-diverse/)
- [SALMONN: Towards Generic Hearing Abilities for Large Language Models](https://md.research.elamur.ai/free/en/papers/tang-2023-salmonn-towards-generic-hearing-abilities/)
- [Conversational Assistants to Support Patients with Heart Failure: Comparing a Neurosymbolic Architecture with GPT](https://md.research.elamur.ai/free/en/papers/tayal-2026-conversational-assistants-support-patients-heart/)
- [FalAR: A Large-scale Speaker-Annotated European Portuguese Speech Corpus of Parliamentary Sessions](https://md.research.elamur.ai/free/en/papers/teixeira-2026-falar-large-scale-speaker-annotated/)
- [ArVoice: A Multi-Speaker Dataset for Arabic Speech Synthesis](https://md.research.elamur.ai/free/en/papers/toyin-2025-arvoice-multi-speaker-dataset-arabic/)
- [Extending ASR Evaluation Resources for Modern Greek Dialects](https://md.research.elamur.ai/free/en/papers/tsoukala-2026-extending-asr-evaluation-resources-modern/)
- [From Felt Sense to Words: Construction and Analysis of a Focusing Dialogue Dataset for Verbalization](https://md.research.elamur.ai/free/en/papers/tsunomori-2026-felt-sense-words-construction-analysis/)
- [Multilingual Long-Form Speech Instruction Following: KIT's Submission to IWSLT 2026](https://md.research.elamur.ai/free/en/papers/ugan-2026-multilingual-long-form-speech-instruction/)
- [Open Universal Arabic ASR Leaderboard](https://md.research.elamur.ai/free/en/papers/wang-2024-open-universal-arabic-asr-leaderboard/)
- [Nuha-Speech: Building General-Purpose Arabic Speech-LLMs](https://md.research.elamur.ai/free/en/papers/wang-2026-nuha-speech-building-general-purpose/)
- [When Consistency Becomes Bias: Interviewer Effects in Semi-Structured Clinical Interviews](https://md.research.elamur.ai/free/en/papers/watawana-2026-when-consistency-becomes-bias-interviewer/)
- [PAIR: A Pilot Dataset for Dual Perspective-based Video-Grounded Dialogue and Reconciliation](https://md.research.elamur.ai/free/en/papers/watson-2026-pair-pilot-dataset-dual-perspective/)
- [Chronological Thinking in Full-Duplex Spoken Dialogue Language Models](https://md.research.elamur.ai/free/en/papers/wu-2026-chronological-thinking-full-duplex-spoken/)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://md.research.elamur.ai/free/en/papers/xi-2023-rise-potential-large-language/)
- [Qwen2.5-Omni Technical Report](https://md.research.elamur.ai/free/en/papers/xu-2025-qwen2-5-omni-technical/)
- [CS-YODAS: A Mined Dataset of In-the-Wild Code-Switched Speech](https://md.research.elamur.ai/free/en/papers/yan-2026-cs-yodas-mined-dataset-wild/)
- [CUHKSZ Simultaneous Speech Translation System for IWSLT 2026](https://md.research.elamur.ai/free/en/papers/yang-2026-cuhksz-simultaneous-speech-translation-system/)
- [A Fusion-Aware Two-Stage Framework for Mispronunciation Detection and Diagnosis in Low-Resource Modern Standard Arabic](https://md.research.elamur.ai/free/en/papers/yang-2026-fusion-aware-two-stage-framework/)
- [TG-ASR: Translation-Guided Learning with Parallel Gated Cross Attention for Low-Resource Automatic Speech Recognition](https://md.research.elamur.ai/free/en/papers/yang-2026-tg-asr-translation-guided-learning/)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://md.research.elamur.ai/free/en/papers/yu-2023-mm-vet-evaluating-large-multimodal/)
- [Hurdles of Automatic Metric for Speech Translation Evaluation](https://md.research.elamur.ai/free/en/papers/zarzu-2026-hurdles-automatic-metric-speech-translation/)
- [CATENG Submission for the IWSLT 2026: Dialectal and Low-resource Speech Translation Task](https://md.research.elamur.ai/free/en/papers/zevallos-2026-cateng-submission-iwslt-2026-dialectal/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{radford2022robust,
  title = {{Robust Speech Recognition via Large-Scale Weak Supervision}},
  author = {Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
  year = {2022},
  doi = {10.48550/arxiv.2212.04356},
  eprint = {2212.04356},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2212.04356}
}
```

- [← previous RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/deng-2022-rlprompt-optimizing-discrete-text-prompts/)
- [next → ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning](https://md.research.elamur.ai/free/en/papers/golovneva-2022-roscoe-suite-metrics-scoring-step/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/)

## References

86 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alcorn, M. A., Li, Q., Gong, Z., Wang, C., Mai, L., Ku, W.-S., and Nguyen, A. Strike (with) a pose: Neural networks are easily fooled by strange poses of familiar objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4845–4854, 2019. [find](https://scholar.google.com/scholar?q=Alcorn%2C+M%2E+A%2E%2C+Li%2C+Q%2E%2C+Gong%2C+Z%2E%2C+Wang%2C+C%2E%2C+Mai%2C+L%2E%2C+Ku%2C+W%2E%2DS%2E%2C+and+Nguyen%2C+A%2E+Strike+%28with%29+a+pose%3A+Neural+networks+are+easily+fooled+by+strange+poses+of+familiar+objects%2E+In+Proceedings+of+the+IEEE%2FC)
2. Amodei, D., Anubhai, R., Battenberg, E., Case, C., Casper, J., Catanzaro, B., Chen, J., Chrzanowski, M., Coates, A., Diamos, G., et al. Deep speech 2: end-to-end speech recognition in english and mandarin. arxiv. arXiv preprint arXiv:1512.02595, 2015. [find](https://scholar.google.com/scholar?q=Amodei%2C+D%2E%2C+Anubhai%2C+R%2E%2C+Battenberg%2C+E%2E%2C+Case%2C+C%2E%2C+Casper%2C+J%2E%2C+Catanzaro%2C+B%2E%2C+Chen%2C+J%2E%2C+Chrzanowski%2C+M%2E%2C+Coates%2C+A%2E%2C+Diamos%2C+G%2E%2C+et+al%2E+Deep+speech+2%3A+end%2Dto%2Dend+speech+recognition+in+english+and+mand) [our card](https://md.research.elamur.ai/free/en/papers/amodei-2015-deep-speech-2-end-end/)
3. Ardila, R., Branson, M., Davis, K., Henretty, M., Kohler, M., Meyer, J., Morais, R., Saunders, L., Tyers, F. M., and Weber, G. Common voice: A massively-multilingual speech corpus. arXiv preprint arXiv:1912.06670, 2019. [find](https://scholar.google.com/scholar?q=Ardila%2C+R%2E%2C+Branson%2C+M%2E%2C+Davis%2C+K%2E%2C+Henretty%2C+M%2E%2C+Kohler%2C+M%2E%2C+Meyer%2C+J%2E%2C+Morais%2C+R%2E%2C+Saunders%2C+L%2E%2C+Tyers%2C+F%2E+M%2E%2C+and+Weber%2C+G%2E+Common+voice%3A+A+massively%2Dmultilingual+speech+corpus%2E+arXiv+preprint+arXi) [our card](https://md.research.elamur.ai/free/en/papers/ardila-2019-common-voice-massively-multilingual-speech/)
4. Babu, A., Wang, C., Tjandra, A., Lakhotia, K., Xu, Q., Goyal, N., Singh, K., von Platen, P., Saraf, Y., Pino, J., et al. XLS-R: Self-supervised cross-lingual speech representation learning at scale. arXiv preprint arXiv:2111.09296, 2021. [find](https://scholar.google.com/scholar?q=Babu%2C+A%2E%2C+Wang%2C+C%2E%2C+Tjandra%2C+A%2E%2C+Lakhotia%2C+K%2E%2C+Xu%2C+Q%2E%2C+Goyal%2C+N%2E%2C+Singh%2C+K%2E%2C+von+Platen%2C+P%2E%2C+Saraf%2C+Y%2E%2C+Pino%2C+J%2E%2C+et+al%2E+XLS%2DR%3A+Self%2Dsupervised+cross%2Dlingual+speech+representation+learning+at+scale%2E+a) [our card](https://md.research.elamur.ai/free/en/papers/babu-2021-xls-r-self-supervised-cross/)
5. Baevski, A., Zhou, H., Mohamed, A., and Auli, M. wav2vec 2.0: A framework for self-supervised learning of speech representations. arXiv preprint arXiv:2006.11477, 2020. [find](https://scholar.google.com/scholar?q=Baevski%2C+A%2E%2C+Zhou%2C+H%2E%2C+Mohamed%2C+A%2E%2C+and+Auli%2C+M%2E+wav2vec+2%2E0%3A+A+framework+for+self%2Dsupervised+learning+of+speech+representations%2E+arXiv+preprint+arXiv%3A2006%2E11477%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/baevski-2020-wav2vec-2-0-framework-self/)
6. Baevski, A., Hsu, W.-N., Conneau, A., and Auli, M. Unsupervised speech recognition. Advances in Neural Information Processing Systems, 34:27826–27839, 2021. [find](https://scholar.google.com/scholar?q=Baevski%2C+A%2E%2C+Hsu%2C+W%2E%2DN%2E%2C+Conneau%2C+A%2E%2C+and+Auli%2C+M%2E+Unsupervised+speech+recognition%2E+Advances+in+Neural+Information+Processing+Systems%2C+34%3A27826%E2%80%9327839%2C+2021%2E)
7. Bapna, A., Cherry, C., Zhang, Y., Jia, Y., Johnson, M., Cheng, Y., Khanuja, S., Riesa, J., and Conneau, A. mslam: Massively multilingual joint pre-training for speech and text. arXiv preprint arXiv:2202.01374, 2022. — [arXiv:2202.01374](https://arxiv.org/abs/2202.01374) [find](https://scholar.google.com/scholar?q=Bapna%2C+A%2E%2C+Cherry%2C+C%2E%2C+Zhang%2C+Y%2E%2C+Jia%2C+Y%2E%2C+Johnson%2C+M%2E%2C+Cheng%2C+Y%2E%2C+Khanuja%2C+S%2E%2C+Riesa%2C+J%2E%2C+and+Conneau%2C+A%2E+mslam%3A+Massively+multilingual+joint+pre%2Dtraining+for+speech+and+text%2E+arXiv+preprint+arXiv%3A22)
8. Barbu, A., Mayo, D., Alverio, J., Luo, W., Wang, C., Gutfreund, D., Tenenbaum, J., and Katz, B. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019. [find](https://scholar.google.com/scholar?q=Barbu%2C+A%2E%2C+Mayo%2C+D%2E%2C+Alverio%2C+J%2E%2C+Luo%2C+W%2E%2C+Wang%2C+C%2E%2C+Gutfreund%2C+D%2E%2C+Tenenbaum%2C+J%2E%2C+and+Katz%2C+B%2E+Objectnet%3A+A+large%2Dscale+bias%2Dcontrolled+dataset+for+pushing+the+limits+of+object+recognition+models%2E+Ad)
9. Caruana, R. Multitask learning. Machine learning, 28(1): 41–75, 1997. [find](https://scholar.google.com/scholar?q=Caruana%2C+R%2E+Multitask+learning%2E+Machine+learning%2C+28%281%29%3A+41%E2%80%9375%2C+1997%2E)
10. Chan, W., Park, D., Lee, C., Zhang, Y., Le, Q., and Norouzi, M. SpeechStew: Simply mix all available speech recognition data to train one large neural network. arXiv preprint arXiv:2104.02133, 2021. — [arXiv:2104.02133](https://arxiv.org/abs/2104.02133) [find](https://scholar.google.com/scholar?q=Chan%2C+W%2E%2C+Park%2C+D%2E%2C+Lee%2C+C%2E%2C+Zhang%2C+Y%2E%2C+Le%2C+Q%2E%2C+and+Norouzi%2C+M%2E+SpeechStew%3A+Simply+mix+all+available+speech+recognition+data+to+train+one+large+neural+network%2E+arXiv+preprint+arXiv%3A2104%2E02133%2C+2021%2E+%E2%80%94)
11. Chen, G., Chai, S., Wang, G., Du, J., Zhang, W.-Q., Weng, C., Su, D., Povey, D., Trmal, J., Zhang, J., et al. Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio. arXiv preprint arXiv:2106.06909, 2021. — [arXiv:2106.06909](https://arxiv.org/abs/2106.06909) [find](https://scholar.google.com/scholar?q=Chen%2C+G%2E%2C+Chai%2C+S%2E%2C+Wang%2C+G%2E%2C+Du%2C+J%2E%2C+Zhang%2C+W%2E%2DQ%2E%2C+Weng%2C+C%2E%2C+Su%2C+D%2E%2C+Povey%2C+D%2E%2C+Trmal%2C+J%2E%2C+Zhang%2C+J%2E%2C+et+al%2E+Gigaspeech%3A+An+evolving%2C+multi%2Ddomain+asr+corpus+with+10%2C000+hours+of+transcribed+audio%2E+a)
12. Chen, S., Wu, Y., Wang, C., Chen, Z., Chen, Z., Liu, S., Wu, J., Qian, Y., Wei, F., Li, J., et al. Unispeech-sat: Universal speech representation learning with speaker aware pre-training. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6152–… [find](https://scholar.google.com/scholar?q=Chen%2C+S%2E%2C+Wu%2C+Y%2E%2C+Wang%2C+C%2E%2C+Chen%2C+Z%2E%2C+Chen%2C+Z%2E%2C+Liu%2C+S%2E%2C+Wu%2C+J%2E%2C+Qian%2C+Y%2E%2C+Wei%2C+F%2E%2C+Li%2C+J%2E%2C+et+al%2E+Unispeech%2Dsat%3A+Universal+speech+representation+learning+with+speaker+aware+pre%2Dtraining%2E+In+ICASSP+20)
13. Chen, T., Xu, B., Zhang, C., and Guestrin, C. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174, 2016. [find](https://scholar.google.com/scholar?q=Chen%2C+T%2E%2C+Xu%2C+B%2E%2C+Zhang%2C+C%2E%2C+and+Guestrin%2C+C%2E+Training+deep+nets+with+sublinear+memory+cost%2E+arXiv+preprint+arXiv%3A1604%2E06174%2C+2016%2E)
14. Chen, Z., Zhang, Y., Rosenberg, A., Ramabhadran, B., Moreno, P., Bapna, A., and Zen, H. Maestro: Matched speech text representations through modality matching. arXiv preprint arXiv:2204.03409, 2022b. — [arXiv:2204.03409](https://arxiv.org/abs/2204.03409) [find](https://scholar.google.com/scholar?q=Chen%2C+Z%2E%2C+Zhang%2C+Y%2E%2C+Rosenberg%2C+A%2E%2C+Ramabhadran%2C+B%2E%2C+Moreno%2C+P%2E%2C+Bapna%2C+A%2E%2C+and+Zen%2C+H%2E+Maestro%3A+Matched+speech+text+representations+through+modality+matching%2E+arXiv+preprint+arXiv%3A2204%2E03409%2C+2022b%2E+)
15. Child, R., Gray, S., Radford, A., and Sutskever, I. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019. [find](https://scholar.google.com/scholar?q=Child%2C+R%2E%2C+Gray%2C+S%2E%2C+Radford%2C+A%2E%2C+and+Sutskever%2C+I%2E+Generating+long+sequences+with+sparse+transformers%2E+arXiv+preprint+arXiv%3A1904%2E10509%2C+2019%2E)
16. Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., and Kuksa, P. Natural language processing (almost) from scratch. Journal of machine learning research, 12(ARTICLE):2493–2537, 2011. [find](https://scholar.google.com/scholar?q=Collobert%2C+R%2E%2C+Weston%2C+J%2E%2C+Bottou%2C+L%2E%2C+Karlen%2C+M%2E%2C+Kavukcuoglu%2C+K%2E%2C+and+Kuksa%2C+P%2E+Natural+language+processing+%28almost%29+from+scratch%2E+Journal+of+machine+learning+research%2C+12%28ARTICLE%29%3A2493%E2%80%932537%2C+2011%2E) [our card](https://md.research.elamur.ai/free/en/papers/collobert-2011-natural-language-processing-almost-scratch/)
17. Conneau, A., Ma, M., Khanuja, S., Zhang, Y., Axelrod, V., Dalmia, S., Riesa, J., Rivera, C., and Bapna, A. Fleurs: Few-shot learning evaluation of universal representations of speech. arXiv preprint arXiv:2205.12446, 2022. [find](https://scholar.google.com/scholar?q=Conneau%2C+A%2E%2C+Ma%2C+M%2E%2C+Khanuja%2C+S%2E%2C+Zhang%2C+Y%2E%2C+Axelrod%2C+V%2E%2C+Dalmia%2C+S%2E%2C+Riesa%2C+J%2E%2C+Rivera%2C+C%2E%2C+and+Bapna%2C+A%2E+Fleurs%3A+Few%2Dshot+learning+evaluation+of+universal+representations+of+speech%2E+arXiv+preprint+a) [our card](https://md.research.elamur.ai/free/en/papers/conneau-2022-fleurs-few-shot-learning-evaluation/)
18. Del Rio, M., Delworth, N., Westerman, R., Huang, M., Bhandari, N., Palakapilly, J., McNamara, Q., Dong, J., Zelasko, P., and Jetté, M. Earnings-21: a practical benchmark for asr in the wild. arXiv preprint arXiv:2104.11348, 2021. — [arXiv:2104.11348](https://arxiv.org/abs/2104.11348) [find](https://scholar.google.com/scholar?q=Del+Rio%2C+M%2E%2C+Delworth%2C+N%2E%2C+Westerman%2C+R%2E%2C+Huang%2C+M%2E%2C+Bhandari%2C+N%2E%2C+Palakapilly%2C+J%2E%2C+McNamara%2C+Q%2E%2C+Dong%2C+J%2E%2C+Zelasko%2C+P%2E%2C+and+Jette%CC%81%2C+M%2E+Earnings%2D21%3A+a+practical+benchmark+for+asr+in+the+wild%2E+arXiv+pr)
19. Galvez, D., Diamos, G., Torres, J. M. C., Achorn, K., Gopi, A., Kanter, D., Lam, M., Mazumder, M., and Reddi, V. J. The people’s speech: A large-scale diverse english speech recognition dataset for commercial usage. arXiv preprint arXiv:2111.09344, 2021. — [arXiv:2111.09344](https://arxiv.org/abs/2111.09344) [find](https://scholar.google.com/scholar?q=Galvez%2C+D%2E%2C+Diamos%2C+G%2E%2C+Torres%2C+J%2E+M%2E+C%2E%2C+Achorn%2C+K%2E%2C+Gopi%2C+A%2E%2C+Kanter%2C+D%2E%2C+Lam%2C+M%2E%2C+Mazumder%2C+M%2E%2C+and+Reddi%2C+V%2E+J%2E+The+people%E2%80%99s+speech%3A+A+large%2Dscale+diverse+english+speech+recognition+dataset+for+co)
20. Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., and Wichmann, F. A. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673, 2020. [find](https://scholar.google.com/scholar?q=Geirhos%2C+R%2E%2C+Jacobsen%2C+J%2E%2DH%2E%2C+Michaelis%2C+C%2E%2C+Zemel%2C+R%2E%2C+Brendel%2C+W%2E%2C+Bethge%2C+M%2E%2C+and+Wichmann%2C+F%2E+A%2E+Shortcut+learning+in+deep+neural+networks%2E+Nature+Machine+Intelligence%2C+2%2811%29%3A665%E2%80%93673%2C+2020%2E)
21. Ghorbani, B., Firat, O., Freitag, M., Bapna, A., Krikun, M., Garcia, X., Chelba, C., and Cherry, C. Scaling laws for neural machine translation. arXiv preprint arXiv:2109.07740, 2021. — [arXiv:2109.07740](https://arxiv.org/abs/2109.07740) [find](https://scholar.google.com/scholar?q=Ghorbani%2C+B%2E%2C+Firat%2C+O%2E%2C+Freitag%2C+M%2E%2C+Bapna%2C+A%2E%2C+Krikun%2C+M%2E%2C+Garcia%2C+X%2E%2C+Chelba%2C+C%2E%2C+and+Cherry%2C+C%2E+Scaling+laws+for+neural+machine+translation%2E+arXiv+preprint+arXiv%3A2109%2E07740%2C+2021%2E+%E2%80%94+%5BarXiv%3A2109%2E07)
22. Griewank, A. and Walther, A. Algorithm 799: revolve: an implementation of checkpointing for the reverse or adjoint mode of computational differentiation. ACM Transactions on Mathematical Software (TOMS), 26(1):19–45, 2000. [find](https://scholar.google.com/scholar?q=Griewank%2C+A%2E+and+Walther%2C+A%2E+Algorithm+799%3A+revolve%3A+an+implementation+of+checkpointing+for+the+reverse+or+adjoint+mode+of+computational+differentiation%2E+ACM+Transactions+on+Mathematical+Software+%28TOM)
23. Gunter, K., Vaughn, C., and Kendall, T. Contextualizing/s/retraction: Sibilant variation and change in washington dc african american language. Language Variation and Change, 33(3):331–357, 2021. [find](https://scholar.google.com/scholar?q=Gunter%2C+K%2E%2C+Vaughn%2C+C%2E%2C+and+Kendall%2C+T%2E+Contextualizing%2Fs%2Fretraction%3A+Sibilant+variation+and+change+in+washington+dc+african+american+language%2E+Language+Variation+and+Change%2C+33%283%29%3A331%E2%80%93357%2C+2021%2E)
24. Harris, C. R., Millman, K. J., van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., Kern, R., Picus, M., Hoyer, S., van Kerkwijk, M. H., Brett, M., Haldane, A., Fernández del Rı́o, J., Wiebe, M., Peterson, P., Gérard-Marchant, P., Sheppard… [find](https://scholar.google.com/scholar?q=Harris%2C+C%2E+R%2E%2C+Millman%2C+K%2E+J%2E%2C+van+der+Walt%2C+S%2E+J%2E%2C+Gommers%2C+R%2E%2C+Virtanen%2C+P%2E%2C+Cournapeau%2C+D%2E%2C+Wieser%2C+E%2E%2C+Taylor%2C+J%2E%2C+Berg%2C+S%2E%2C+Smith%2C+N%2E+J%2E%2C+Kern%2C+R%2E%2C+Picus%2C+M%2E%2C+Hoyer%2C+S%2E%2C+van+Kerkwijk%2C+M%2E+H%2E%2C+Bret) [our card](https://md.research.elamur.ai/free/en/papers/harris-2020-array-programming-numpy/)
25. Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016. [find](https://scholar.google.com/scholar?q=Hendrycks%2C+D%2E+and+Gimpel%2C+K%2E+Gaussian+error+linear+units+%28gelus%29%2E+arXiv+preprint+arXiv%3A1606%2E08415%2C+2016%2E)
26. Hendrycks, D., Liu, X., Wallace, E., Dziedzic, A., Krishnan, R., and Song, D. Pretrained transformers improve out-of-distribution robustness. arXiv preprint arXiv:2004.06100, 2020. [find](https://scholar.google.com/scholar?q=Hendrycks%2C+D%2E%2C+Liu%2C+X%2E%2C+Wallace%2C+E%2E%2C+Dziedzic%2C+A%2E%2C+Krishnan%2C+R%2E%2C+and+Song%2C+D%2E+Pretrained+transformers+improve+out%2Dof%2Ddistribution+robustness%2E+arXiv+preprint+arXiv%3A2004%2E06100%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-pretrained-transformers-improve-out-distribution/)
27. Hernandez, F., Nguyen, V., Ghannay, S., Tomashenko, N. A., and Estève, Y. Ted-lium 3: twice as much data and corpus repartition for experiments on speaker adaptation. In SPECOM, 2018. [find](https://scholar.google.com/scholar?q=Hernandez%2C+F%2E%2C+Nguyen%2C+V%2E%2C+Ghannay%2C+S%2E%2C+Tomashenko%2C+N%2E+A%2E%2C+and+Este%CC%80ve%2C+Y%2E+Ted%2Dlium+3%3A+twice+as+much+data+and+corpus+repartition+for+experiments+on+speaker+adaptation%2E+In+SPECOM%2C+2018%2E)
28. Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460, 2021a. [find](https://scholar.google.com/scholar?q=Hsu%2C+W%2E%2DN%2E%2C+Bolte%2C+B%2E%2C+Tsai%2C+Y%2E%2DH%2E+H%2E%2C+Lakhotia%2C+K%2E%2C+Salakhutdinov%2C+R%2E%2C+and+Mohamed%2C+A%2E+Hubert%3A+Self%2Dsupervised+speech+representation+learning+by+masked+prediction+of+hidden+units%2E+IEEE%2FACM+Transactio) [our card](https://md.research.elamur.ai/free/en/papers/hsu-2021-hubert-self-supervised-speech-representation/)
29. Hsu, W.-N., Sriram, A., Baevski, A., Likhomanenko, T., Xu, Q., Pratap, V., Kahn, J., Lee, A., Collobert, R., Synnaeve, G., et al. Robust wav2vec 2.0: Analyzing domain shift in self-supervised pre-training. arXiv preprint arXiv:2104.01027, 2021b. — [arXiv:2104.01027](https://arxiv.org/abs/2104.01027) [find](https://scholar.google.com/scholar?q=Hsu%2C+W%2E%2DN%2E%2C+Sriram%2C+A%2E%2C+Baevski%2C+A%2E%2C+Likhomanenko%2C+T%2E%2C+Xu%2C+Q%2E%2C+Pratap%2C+V%2E%2C+Kahn%2C+J%2E%2C+Lee%2C+A%2E%2C+Collobert%2C+R%2E%2C+Synnaeve%2C+G%2E%2C+et+al%2E+Robust+wav2vec+2%2E0%3A+Analyzing+domain+shift+in+self%2Dsupervised+pre%2Dtrai)
30. Huang, G., Sun, Y., Liu, Z., Sedra, D., and Weinberger, K. Q. Deep networks with stochastic depth. In European conference on computer vision, pp. 646–661. Springer, 2016. [find](https://scholar.google.com/scholar?q=Huang%2C+G%2E%2C+Sun%2C+Y%2E%2C+Liu%2C+Z%2E%2C+Sedra%2C+D%2E%2C+and+Weinberger%2C+K%2E+Q%2E+Deep+networks+with+stochastic+depth%2E+In+European+conference+on+computer+vision%2C+pp%2E+646%E2%80%93661%2E+Springer%2C+2016%2E)
31. Jia, R. and Liang, P. Adversarial examples for evaluating reading comprehension systems. arXiv preprint arXiv:1707.07328, 2017. [find](https://scholar.google.com/scholar?q=Jia%2C+R%2E+and+Liang%2C+P%2E+Adversarial+examples+for+evaluating+reading+comprehension+systems%2E+arXiv+preprint+arXiv%3A1707%2E07328%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/)
32. Johnson, M., Schuster, M., Le, Q. V., Krikun, M., Wu, Y., Chen, Z., Thorat, N., Viégas, F., Wattenberg, M., Corrado, G., et al. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339– 351, 2017. [find](https://scholar.google.com/scholar?q=Johnson%2C+M%2E%2C+Schuster%2C+M%2E%2C+Le%2C+Q%2E+V%2E%2C+Krikun%2C+M%2E%2C+Wu%2C+Y%2E%2C+Chen%2C+Z%2E%2C+Thorat%2C+N%2E%2C+Vie%CC%81gas%2C+F%2E%2C+Wattenberg%2C+M%2E%2C+Corrado%2C+G%2E%2C+et+al%2E+Google%E2%80%99s+multilingual+neural+machine+translation+system%3A+Enabling+zero%2D)
33. Kendall, T. and Farrington, C. The corpus of regional african american language. Version 2021.07. Eugene, OR: The Online Resources for African American Language Project. [http://oraal.uoregon.edu/coraal](http://oraal.uoregon.edu/coraal), 2021. Accessed: 2022-09-01. [↗](http://oraal.uoregon.edu/coraal)
34. Koenecke, A., Nam, A., Lake, E., Nudell, J., Quartey, M., Mengesha, Z., Toups, C., Rickford, J. R., Jurafsky, D., and Goel, S. Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences, 117(14):7684–7689, 2020. [find](https://scholar.google.com/scholar?q=Koenecke%2C+A%2E%2C+Nam%2C+A%2E%2C+Lake%2C+E%2E%2C+Nudell%2C+J%2E%2C+Quartey%2C+M%2E%2C+Mengesha%2C+Z%2E%2C+Toups%2C+C%2E%2C+Rickford%2C+J%2E+R%2E%2C+Jurafsky%2C+D%2E%2C+and+Goel%2C+S%2E+Racial+disparities+in+automated+speech+recognition%2E+Proceedings+of+the+Na)
35. Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., and Houlsby, N. Big transfer (bit): General visual representation learning. In European conference on computer vision, pp. 491–507. Springer, 2020. [find](https://scholar.google.com/scholar?q=Kolesnikov%2C+A%2E%2C+Beyer%2C+L%2E%2C+Zhai%2C+X%2E%2C+Puigcerver%2C+J%2E%2C+Yung%2C+J%2E%2C+Gelly%2C+S%2E%2C+and+Houlsby%2C+N%2E+Big+transfer+%28bit%29%3A+General+visual+representation+learning%2E+In+European+conference+on+computer+vision%2C+pp%2E+491)
36. Kuchaiev, O., Li, J., Nguyen, H., Hrinchuk, O., Leary, R., Ginsburg, B., Kriman, S., Beliaev, S., Lavrukhin, V., Cook, J., et al. Nemo: a toolkit for building ai applications using neural modules. arXiv preprint arXiv:1909.09577, 2019. [find](https://scholar.google.com/scholar?q=Kuchaiev%2C+O%2E%2C+Li%2C+J%2E%2C+Nguyen%2C+H%2E%2C+Hrinchuk%2C+O%2E%2C+Leary%2C+R%2E%2C+Ginsburg%2C+B%2E%2C+Kriman%2C+S%2E%2C+Beliaev%2C+S%2E%2C+Lavrukhin%2C+V%2E%2C+Cook%2C+J%2E%2C+et+al%2E+Nemo%3A+a+toolkit+for+building+ai+applications+using+neural+modules%2E+arX)
37. Lake, B. M., Ullman, T. D., Tenenbaum, J. B., and Gershman, S. J. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017. [find](https://scholar.google.com/scholar?q=Lake%2C+B%2E+M%2E%2C+Ullman%2C+T%2E+D%2E%2C+Tenenbaum%2C+J%2E+B%2E%2C+and+Gershman%2C+S%2E+J%2E+Building+machines+that+learn+and+think+like+people%2E+Behavioral+and+brain+sciences%2C+40%2C+2017%2E)
38. Liao, H., McDermott, E., and Senior, A. Large scale deep neural network acoustic modeling with semi-supervised training data for youtube video transcription. In 2013 IEEE Workshop on Automatic Speech Recognition and Understanding, pp. 368–373. [find](https://scholar.google.com/scholar?q=Liao%2C+H%2E%2C+McDermott%2C+E%2E%2C+and+Senior%2C+A%2E+Large+scale+deep+neural+network+acoustic+modeling+with+semi%2Dsupervised+training+data+for+youtube+video+transcription%2E+In+2013+IEEE+Workshop+on+Automatic+Speech+)
39. Likhomanenko, T., Xu, Q., Pratap, V., Tomasello, P., Kahn, J., Avidov, G., Collobert, R., and Synnaeve, G. Rethinking evaluation in asr: Are our models robust enough? arXiv preprint arXiv:2010.11745, 2020. — [arXiv:2010.11745](https://arxiv.org/abs/2010.11745) [find](https://scholar.google.com/scholar?q=Likhomanenko%2C+T%2E%2C+Xu%2C+Q%2E%2C+Pratap%2C+V%2E%2C+Tomasello%2C+P%2E%2C+Kahn%2C+J%2E%2C+Avidov%2C+G%2E%2C+Collobert%2C+R%2E%2C+and+Synnaeve%2C+G%2E+Rethinking+evaluation+in+asr%3A+Are+our+models+robust+enough%3F+arXiv+preprint+arXiv%3A2010%2E11745%2C+)
40. Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. [find](https://scholar.google.com/scholar?q=Loshchilov%2C+I%2E+and+Hutter%2C+F%2E+Decoupled+weight+decay+regularization%2E+arXiv+preprint+arXiv%3A1711%2E05101%2C+2017%2E)
41. Luong, M.-T., Le, Q. V., Sutskever, I., Vinyals, O., and Kaiser, L. Multi-task sequence to sequence learning. arXiv preprint arXiv:1511.06114, 2015. [find](https://scholar.google.com/scholar?q=Luong%2C+M%2E%2DT%2E%2C+Le%2C+Q%2E+V%2E%2C+Sutskever%2C+I%2E%2C+Vinyals%2C+O%2E%2C+and+Kaiser%2C+L%2E+Multi%2Dtask+sequence+to+sequence+learning%2E+arXiv+preprint+arXiv%3A1511%2E06114%2C+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/luong-2015-multi-task-sequence-sequence-learning/)
42. Mahajan, D., Girshick, R., Ramanathan, V., He, K., Paluri, M., Li, Y., Bharambe, A., and Van Der Maaten, L. Exploring the limits of weakly supervised pretraining. In Proceedings of the European conference on computer vision (ECCV), pp. 181–196, 2018. [find](https://scholar.google.com/scholar?q=Mahajan%2C+D%2E%2C+Girshick%2C+R%2E%2C+Ramanathan%2C+V%2E%2C+He%2C+K%2E%2C+Paluri%2C+M%2E%2C+Li%2C+Y%2E%2C+Bharambe%2C+A%2E%2C+and+Van+Der+Maaten%2C+L%2E+Exploring+the+limits+of+weakly+supervised+pretraining%2E+In+Proceedings+of+the+European+confer)
43. Mauch, M. and Ewert, S. The audio degradation toolbox and its application to robustness evaluation. In Proceedings of the 14th International Society for Music Information Retrieval Conference (ISMIR 2013), Curitiba, Brazil, 2013. accepted. [find](https://scholar.google.com/scholar?q=Mauch%2C+M%2E+and+Ewert%2C+S%2E+The+audio+degradation+toolbox+and+its+application+to+robustness+evaluation%2E+In+Proceedings+of+the+14th+International+Society+for+Music+Information+Retrieval+Conference+%28ISMIR+2)
44. McCann, B., Keskar, N. S., Xiong, C., and Socher, R. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730, 2018. [find](https://scholar.google.com/scholar?q=McCann%2C+B%2E%2C+Keskar%2C+N%2E+S%2E%2C+Xiong%2C+C%2E%2C+and+Socher%2C+R%2E+The+natural+language+decathlon%3A+Multitask+learning+as+question+answering%2E+arXiv+preprint+arXiv%3A1806%2E08730%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/mccann-2018-natural-language-decathlon-multitask-learning/)
45. Meyer, J., Rauchenstein, L., Eisenberg, J. D., and Howell, N. Artie bias corpus: An open dataset for detecting demographic bias in speech applications. In Proceedings of the 12th Language Resources and Evaluation Conference, pp. 6462–6468, Marseille, France, May 2020. European Language Resources Ass… [find](https://scholar.google.com/scholar?q=Meyer%2C+J%2E%2C+Rauchenstein%2C+L%2E%2C+Eisenberg%2C+J%2E+D%2E%2C+and+Howell%2C+N%2E+Artie+bias+corpus%3A+An+open+dataset+for+detecting+demographic+bias+in+speech+applications%2E+In+Proceedings+of+the+12th+Language+Resources+an)
46. Miller, J., Krauth, K., Recht, B., and Schmidt, L. The effect of natural distribution shift on question answering models. In ICML, 2020. [find](https://scholar.google.com/scholar?q=Miller%2C+J%2E%2C+Krauth%2C+K%2E%2C+Recht%2C+B%2E%2C+and+Schmidt%2C+L%2E+The+effect+of+natural+distribution+shift+on+question+answering+models%2E+In+ICML%2C+2020%2E)
47. Mohamed, A.-r., Dahl, G., Hinton, G., et al. Deep belief networks for phone recognition. In Nips workshop on deep learning for speech recognition and related applications, volume 1, pp. 39, 2009. [find](https://scholar.google.com/scholar?q=Mohamed%2C+A%2E%2Dr%2E%2C+Dahl%2C+G%2E%2C+Hinton%2C+G%2E%2C+et+al%2E+Deep+belief+networks+for+phone+recognition%2E+In+Nips+workshop+on+deep+learning+for+speech+recognition+and+related+applications%2C+volume+1%2C+pp%2E+39%2C+2009%2E)
48. Narayanan, A., Misra, A., Sim, K. C., Pundak, G., Tripathi, A., Elfeky, M., Haghani, P., Strohman, T., and Bacchiani, M. Toward domain-invariant speech recognition via large scale training. In 2018 IEEE Spoken Language Technology Workshop (SLT), pp. 441–447. [find](https://scholar.google.com/scholar?q=Narayanan%2C+A%2E%2C+Misra%2C+A%2E%2C+Sim%2C+K%2E+C%2E%2C+Pundak%2C+G%2E%2C+Tripathi%2C+A%2E%2C+Elfeky%2C+M%2E%2C+Haghani%2C+P%2E%2C+Strohman%2C+T%2E%2C+and+Bacchiani%2C+M%2E+Toward+domain%2Dinvariant+speech+recognition+via+large+scale+training%2E+In+2018+IE)
49. Panayotov, V., Chen, G., Povey, D., and Khudanpur, S. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206–5210. [find](https://scholar.google.com/scholar?q=Panayotov%2C+V%2E%2C+Chen%2C+G%2E%2C+Povey%2C+D%2E%2C+and+Khudanpur%2C+S%2E+Librispeech%3A+an+asr+corpus+based+on+public+domain+audio+books%2E+In+2015+IEEE+international+conference+on+acoustics%2C+speech+and+signal+processing+%28I)
50. IEEE, 2015. pandas development team, T. pandas-dev/pandas: Pandas, February 2020. URL https://doi.org/10.5281/zenodo.3509134. — [doi:10.5281/zenodo.3509134](https://doi.org/10.5281/zenodo.3509134) [↗](https://doi.org/10.5281/zenodo.3509134)
51. Park, D. S., Chan, W., Zhang, Y., Chiu, C.-C., Zoph, B., Cubuk, E. D., and Le, Q. V. SpecAugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv:1904.08779, 2019. [find](https://scholar.google.com/scholar?q=Park%2C+D%2E+S%2E%2C+Chan%2C+W%2E%2C+Zhang%2C+Y%2E%2C+Chiu%2C+C%2E%2DC%2E%2C+Zoph%2C+B%2E%2C+Cubuk%2C+E%2E+D%2E%2C+and+Le%2C+Q%2E+V%2E+SpecAugment%3A+A+simple+data+augmentation+method+for+automatic+speech+recognition%2E+arXiv+preprint+arXiv%3A1904%2E08779%2C+2) [our card](https://md.research.elamur.ai/free/en/papers/park-2019-specaugment-simple-data-augmentation-method/)
52. Pascanu, R., Mikolov, T., and Bengio, Y. On the difficulty of training recurrent neural networks. In International conference on machine learning, pp. 1310–1318. PMLR, 2013. [find](https://scholar.google.com/scholar?q=Pascanu%2C+R%2E%2C+Mikolov%2C+T%2E%2C+and+Bengio%2C+Y%2E+On+the+difficulty+of+training+recurrent+neural+networks%2E+In+International+conference+on+machine+learning%2C+pp%2E+1310%E2%80%931318%2E+PMLR%2C+2013%2E)
53. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. Pytorch: An imperative style, high-perfor… [find](https://scholar.google.com/scholar?q=Paszke%2C+A%2E%2C+Gross%2C+S%2E%2C+Massa%2C+F%2E%2C+Lerer%2C+A%2E%2C+Bradbury%2C+J%2E%2C+Chanan%2C+G%2E%2C+Killeen%2C+T%2E%2C+Lin%2C+Z%2E%2C+Gimelshein%2C+N%2E%2C+Antiga%2C+L%2E%2C+Desmaison%2C+A%2E%2C+Kopf%2C+A%2E%2C+Yang%2C+E%2E%2C+DeVito%2C+Z%2E%2C+Raison%2C+M%2E%2C+Tejani%2C+A%2E%2C+Chilamku) [our card](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/)
54. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E. Scikit-learn: Machine learning in Python. Journal of Machine Learning Researc… [find](https://scholar.google.com/scholar?q=Pedregosa%2C+F%2E%2C+Varoquaux%2C+G%2E%2C+Gramfort%2C+A%2E%2C+Michel%2C+V%2E%2C+Thirion%2C+B%2E%2C+Grisel%2C+O%2E%2C+Blondel%2C+M%2E%2C+Prettenhofer%2C+P%2E%2C+Weiss%2C+R%2E%2C+Dubourg%2C+V%2E%2C+Vanderplas%2C+J%2E%2C+Passos%2C+A%2E%2C+Cournapeau%2C+D%2E%2C+Brucher%2C+M%2E%2C+Perrot%2C)
55. Polyak, B. T. and Juditsky, A. B. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4):838–855, 1992. [find](https://scholar.google.com/scholar?q=Polyak%2C+B%2E+T%2E+and+Juditsky%2C+A%2E+B%2E+Acceleration+of+stochastic+approximation+by+averaging%2E+SIAM+journal+on+control+and+optimization%2C+30%284%29%3A838%E2%80%93855%2C+1992%2E)
56. Pratap, V., Sriram, A., Tomasello, P., Hannun, A. Y., Liptchinsky, V., Synnaeve, G., and Collobert, R. Massively multilingual asr: 50 languages, 1 model, 1 billion parameters. ArXiv, abs/2007.03001, 2020a. — [arXiv:2007.03001](https://arxiv.org/abs/2007.03001) [find](https://scholar.google.com/scholar?q=Pratap%2C+V%2E%2C+Sriram%2C+A%2E%2C+Tomasello%2C+P%2E%2C+Hannun%2C+A%2E+Y%2E%2C+Liptchinsky%2C+V%2E%2C+Synnaeve%2C+G%2E%2C+and+Collobert%2C+R%2E+Massively+multilingual+asr%3A+50+languages%2C+1+model%2C+1+billion+parameters%2E+ArXiv%2C+abs%2F2007%2E03001%2C+2)
57. Pratap, V., Xu, Q., Sriram, A., Synnaeve, G., and Collobert, R. Mls: A large-scale multilingual dataset for speech research. arXiv preprint arXiv:2012.03411, 2020b. [find](https://scholar.google.com/scholar?q=Pratap%2C+V%2E%2C+Xu%2C+Q%2E%2C+Sriram%2C+A%2E%2C+Synnaeve%2C+G%2E%2C+and+Collobert%2C+R%2E+Mls%3A+A+large%2Dscale+multilingual+dataset+for+speech+research%2E+arXiv+preprint+arXiv%3A2012%2E03411%2C+2020b%2E) [our card](https://md.research.elamur.ai/free/en/papers/pratap-2020-mls-large-scale-multilingual-dataset/)
58. Press, O. and Wolf, L. Using the output embedding to improve language models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pp. 157–163, Valencia, Spain, April 2017. Association for Computational Linguistics. U… [find](https://scholar.google.com/scholar?q=Press%2C+O%2E+and+Wolf%2C+L%2E+Using+the+output+embedding+to+improve+language+models%2E+In+Proceedings+of+the+15th+Conference+of+the+European+Chapter+of+the+Association+for+Computational+Linguistics%3A+Volume+2%2C+) [our card](https://md.research.elamur.ai/free/en/papers/press-2017-output-embedding-improve-language-models/)
59. Provilkov, I., Emelianenko, D., and Voita, E. Bpe-dropout: Simple and effective subword regularization. arXiv preprint arXiv:1910.13267, 2019. [find](https://scholar.google.com/scholar?q=Provilkov%2C+I%2E%2C+Emelianenko%2C+D%2E%2C+and+Voita%2C+E%2E+Bpe%2Ddropout%3A+Simple+and+effective+subword+regularization%2E+arXiv+preprint+arXiv%3A1910%2E13267%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/provilkov-2020-bpe-dropout-simple-effective-subword/)
60. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. 2019. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Wu%2C+J%2E%2C+Child%2C+R%2E%2C+Luan%2C+D%2E%2C+Amodei%2C+D%2E%2C+and+Sutskever%2C+I%2E+Language+models+are+unsupervised+multitask+learners%2E+2019%2E)
61. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020, 2021. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Kim%2C+J%2E+W%2E%2C+Hallacy%2C+C%2E%2C+Ramesh%2C+A%2E%2C+Goh%2C+G%2E%2C+Agarwal%2C+S%2E%2C+Sastry%2C+G%2E%2C+Askell%2C+A%2E%2C+Mishkin%2C+P%2E%2C+Clark%2C+J%2E%2C+Krueger%2C+G%2E%2C+and+Sutskever%2C+I%2E+Learning+transferable+visual+models+from+natural+) [our card](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/)
62. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P. J., et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1–67, 2020. [find](https://scholar.google.com/scholar?q=Raffel%2C+C%2E%2C+Shazeer%2C+N%2E%2C+Roberts%2C+A%2E%2C+Lee%2C+K%2E%2C+Narang%2C+S%2E%2C+Matena%2C+M%2E%2C+Zhou%2C+Y%2E%2C+Li%2C+W%2E%2C+Liu%2C+P%2E+J%2E%2C+et+al%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtext+transformer%2E+J%2E+Mach%2E+)
63. Ravanelli, M., Parcollet, T., Plantinga, P., Rouhe, A., Cornell, S., Lugosch, L., Subakan, C., Dawalatabad, N., Heba, A., Zhong, J., Chou, J.-C., Yeh, S.-L., Fu, S.-W., Liao, C.-F., Rastorgueva, E., Grondin, F., Aris, W., Na, H., Gao, Y., Mori, R. D., and Bengio, Y. SpeechBrain: A general-purpose sp… — [arXiv:2106.04624](https://arxiv.org/abs/2106.04624) [find](https://scholar.google.com/scholar?q=Ravanelli%2C+M%2E%2C+Parcollet%2C+T%2E%2C+Plantinga%2C+P%2E%2C+Rouhe%2C+A%2E%2C+Cornell%2C+S%2E%2C+Lugosch%2C+L%2E%2C+Subakan%2C+C%2E%2C+Dawalatabad%2C+N%2E%2C+Heba%2C+A%2E%2C+Zhong%2C+J%2E%2C+Chou%2C+J%2E%2DC%2E%2C+Yeh%2C+S%2E%2DL%2E%2C+Fu%2C+S%2E%2DW%2E%2C+Liao%2C+C%2E%2DF%2E%2C+Rastorgueva%2C+E%2E%2C+G)
64. Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do ImageNet classifiers generalize to ImageNet? In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5389–5400. PMLR, 09–15… [find](https://scholar.google.com/scholar?q=Recht%2C+B%2E%2C+Roelofs%2C+R%2E%2C+Schmidt%2C+L%2E%2C+and+Shankar%2C+V%2E+Do+ImageNet+classifiers+generalize+to+ImageNet%3F+In+Chaudhuri%2C+K%2E+and+Salakhutdinov%2C+R%2E+%28eds%2E%29%2C+Proceedings+of+the+36th+International+Conference+on+)
65. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3): 211–252, 2015. [find](https://scholar.google.com/scholar?q=Russakovsky%2C+O%2E%2C+Deng%2C+J%2E%2C+Su%2C+H%2E%2C+Krause%2C+J%2E%2C+Satheesh%2C+S%2E%2C+Ma%2C+S%2E%2C+Huang%2C+Z%2E%2C+Karpathy%2C+A%2E%2C+Khosla%2C+A%2E%2C+Bernstein%2C+M%2E%2C+et+al%2E+Imagenet+large+scale+visual+recognition+challenge%2E+International+journal) [our card](https://md.research.elamur.ai/free/en/papers/russakovsky-2014-imagenet-large-scale-visual-recognition/)
66. Schultz, T. and Kirchhoff, K. Multilingual speech processing. Elsevier, 2006. [find](https://scholar.google.com/scholar?q=Schultz%2C+T%2E+and+Kirchhoff%2C+K%2E+Multilingual+speech+processing%2E+Elsevier%2C+2006%2E)
67. Seide, F., Li, G., Chen, X., and Yu, D. Feature engineering in context-dependent deep neural networks for conversational speech transcription. In 2011 IEEE Workshop on Automatic Speech Recognition & Understanding, pp. 24–29. [find](https://scholar.google.com/scholar?q=Seide%2C+F%2E%2C+Li%2C+G%2E%2C+Chen%2C+X%2E%2C+and+Yu%2C+D%2E+Feature+engineering+in+context%2Ddependent+deep+neural+networks+for+conversational+speech+transcription%2E+In+2011+IEEE+Workshop+on+Automatic+Speech+Recognition+%26+U)
68. Sennrich, R., Haddow, B., and Birch, A. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015. [find](https://scholar.google.com/scholar?q=Sennrich%2C+R%2E%2C+Haddow%2C+B%2E%2C+and+Birch%2C+A%2E+Neural+machine+translation+of+rare+words+with+subword+units%2E+arXiv+preprint+arXiv%3A1508%2E07909%2C+2015%2E)
69. Speer, R. ftfy. Zenodo, 2019. URL https://doi.org/10.5281/zenodo.2591652. Version 5.5. — [doi:10.5281/zenodo.2591652](https://doi.org/10.5281/zenodo.2591652) [↗](https://doi.org/10.5281/zenodo.2591652)
70. Sutskever, I., Vinyals, O., and Le, Q. V. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27, 2014. [find](https://scholar.google.com/scholar?q=Sutskever%2C+I%2E%2C+Vinyals%2C+O%2E%2C+and+Le%2C+Q%2E+V%2E+Sequence+to+sequence+learning+with+neural+networks%2E+Advances+in+neural+information+processing+systems%2C+27%2C+2014%2E)
71. Wang, P., Sainath, T. N., and Weiss, R. J. Multitask training with text data for end-to-end speech recognition. arXiv Taori, R., Dave, A., Shankar, V., Carlini, N., Recht, B., preprint arXiv:2010.14318, 2020c. and Schmidt, L. Measuring robustness to natural distribution shifts in image classificatio… — [arXiv:2010.14318](https://arxiv.org/abs/2010.14318) [find](https://scholar.google.com/scholar?q=Wang%2C+P%2E%2C+Sainath%2C+T%2E+N%2E%2C+and+Weiss%2C+R%2E+J%2E+Multitask+training+with+text+data+for+end%2Dto%2Dend+speech+recognition%2E+arXiv+Taori%2C+R%2E%2C+Dave%2C+A%2E%2C+Shankar%2C+V%2E%2C+Carlini%2C+N%2E%2C+Recht%2C+B%2E%2C+preprint+arXiv%3A2010%2E1431)
72. Toshniwal, S., Sainath, T. N., Weiss, R. J., Li, B., Moreno, P. J., Weinstein, E., and Rao, K. Multilingual speech Zhang, Y., Qin, J., Park, D. S., Han, W., Chiu, C.-C., Pang, recognition with a single end-to-end model. 2018 IEEE R., Le, Q. V., and Wu, Y. Pushing the limits of semi-International Con… — [arXiv:2010.10504](https://arxiv.org/abs/2010.10504) [find](https://scholar.google.com/scholar?q=Toshniwal%2C+S%2E%2C+Sainath%2C+T%2E+N%2E%2C+Weiss%2C+R%2E+J%2E%2C+Li%2C+B%2E%2C+Moreno%2C+P%2E+J%2E%2C+Weinstein%2C+E%2E%2C+and+Rao%2C+K%2E+Multilingual+speech+Zhang%2C+Y%2E%2C+Qin%2C+J%2E%2C+Park%2C+D%2E+S%2E%2C+Han%2C+W%2E%2C+Chiu%2C+C%2E%2DC%2E%2C+Pang%2C+recognition+with+a+singl)
73. Valk, J. and Alumäe, T. Voxlingua107: a dataset for spoken language recognition. In 2021 IEEE Spoken Language Technology Workshop (SLT), pp. 652–658. [find](https://scholar.google.com/scholar?q=Valk%2C+J%2E+and+Aluma%CC%88e%2C+T%2E+Voxlingua107%3A+a+dataset+for+spoken+language+recognition%2E+In+2021+IEEE+Spoken+Language+Technology+Workshop+%28SLT%29%2C+pp%2E+652%E2%80%93658%2E)
74. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. In Advances in neural information processing systems, pp. 5998–6008, 2017. [find](https://scholar.google.com/scholar?q=Vaswani%2C+A%2E%2C+Shazeer%2C+N%2E%2C+Parmar%2C+N%2E%2C+Uszkoreit%2C+J%2E%2C+Jones%2C+L%2E%2C+Gomez%2C+A%2E+N%2E%2C+Kaiser%2C+%C5%81%2E%2C+and+Polosukhin%2C+I%2E+Attention+is+all+you+need%2E+In+Advances+in+neural+information+processing+systems%2C+pp%2E+5998%E2%80%936)
75. Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt, S. J., Brett, M., Wilson, J., Millman, K. J., Mayorov, N., Nelson, A. R. J., Jones, E., Kern, R., Larson, E., Carey, C. J., Polat, İ., Feng, Y.,… [find](https://scholar.google.com/scholar?q=Virtanen%2C+P%2E%2C+Gommers%2C+R%2E%2C+Oliphant%2C+T%2E+E%2E%2C+Haberland%2C+M%2E%2C+Reddy%2C+T%2E%2C+Cournapeau%2C+D%2E%2C+Burovski%2C+E%2E%2C+Peterson%2C+P%2E%2C+Weckesser%2C+W%2E%2C+Bright%2C+J%2E%2C+van+der+Walt%2C+S%2E+J%2E%2C+Brett%2C+M%2E%2C+Wilson%2C+J%2E%2C+Millman%2C+K%2E+J%2E%2C) [our card](https://md.research.elamur.ai/free/en/papers/virtanen-2019-scipy-1-0-fundamental-algorithms/)
76. Wang, C., Tang, Y., Ma, X., Wu, A., Okhonko, D., and Pino, J. fairseq s2t: Fast speech-to-text modeling with fairseq. arXiv preprint arXiv:2010.05171, 2020a. [find](https://scholar.google.com/scholar?q=Wang%2C+C%2E%2C+Tang%2C+Y%2E%2C+Ma%2C+X%2E%2C+Wu%2C+A%2E%2C+Okhonko%2C+D%2E%2C+and+Pino%2C+J%2E+fairseq+s2t%3A+Fast+speech%2Dto%2Dtext+modeling+with+fairseq%2E+arXiv+preprint+arXiv%3A2010%2E05171%2C+2020a%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2020-fairseq-s2t-fast-speech-text/)
77. Wang, C., Wu, A., and Pino, J. Covost 2 and massively multilingual speech-to-text translation. arXiv preprint arXiv:2007.10310, 2020b. [find](https://scholar.google.com/scholar?q=Wang%2C+C%2E%2C+Wu%2C+A%2E%2C+and+Pino%2C+J%2E+Covost+2+and+massively+multilingual+speech%2Dto%2Dtext+translation%2E+arXiv+preprint+arXiv%3A2007%2E10310%2C+2020b%2E)
78. Wang, C., Riviere, M., Lee, A., Wu, A., Talnikar, C., Haziza, D., Williamson, M., Pino, J., and Dupoux, E. Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. arXiv preprint arXiv:2101.00390, 2021. [find](https://scholar.google.com/scholar?q=Wang%2C+C%2E%2C+Riviere%2C+M%2E%2C+Lee%2C+A%2E%2C+Wu%2C+A%2E%2C+Talnikar%2C+C%2E%2C+Haziza%2C+D%2E%2C+Williamson%2C+M%2E%2C+Pino%2C+J%2E%2C+and+Dupoux%2C+E%2E+Voxpopuli%3A+A+large%2Dscale+multilingual+speech+corpus+for+representation+learning%2C+semi%2Dsupervi) [our card](https://md.research.elamur.ai/free/en/papers/wang-2021-voxpopuli-large-scale-multilingual-speech/)
79. Zhang, Y., Park, D. S., Han, W., Qin, J., Gulati, A., Shor, J., Jansen, A., Xu, Y., Huang, Y., Wang, S., et al. BigSSL: Exploring the frontier of large-scale semi-supervised learning for automatic speech recognition. arXiv preprint arXiv:2109.13226, 2021. — [arXiv:2109.13226](https://arxiv.org/abs/2109.13226) [find](https://scholar.google.com/scholar?q=Zhang%2C+Y%2E%2C+Park%2C+D%2E+S%2E%2C+Han%2C+W%2E%2C+Qin%2C+J%2E%2C+Gulati%2C+A%2E%2C+Shor%2C+J%2E%2C+Jansen%2C+A%2E%2C+Xu%2C+Y%2E%2C+Huang%2C+Y%2E%2C+Wang%2C+S%2E%2C+et+al%2E+BigSSL%3A+Exploring+the+frontier+of+large%2Dscale+semi%2Dsupervised+learning+for+automatic+spe)
80. A. Evaluation Datasets. A.1. Short-form English-only datasets • LibriSpeech (Panayotov et al., 2015): We used the test-clean and test-other splits from the LibriSpeech ASR corpus. [find](https://scholar.google.com/scholar?q=A%2E+Evaluation+Datasets%2E+A%2E1%2E+Short%2Dform+English%2Donly+datasets+%E2%80%A2+LibriSpeech+%28Panayotov+et+al%2E%2C+2015%29%3A+We+used+the+test%2Dclean+and+test%2Dother+splits+from+the+LibriSpeech+ASR+corpus%2E)
81. TED-LIUM 3 (Hernandez et al., 2018): We used the test split of TED-LIUM Release 3, using the segmented manual transcripts included in the release. [find](https://scholar.google.com/scholar?q=TED%2DLIUM+3+%28Hernandez+et+al%2E%2C+2018%29%3A+We+used+the+test+split+of+TED%2DLIUM+Release+3%2C+using+the+segmented+manual+transcripts+included+in+the+release%2E)
82. Common Voice 5.1 (Ardila et al., 2019): We downloaded the English subset of Common Voice Corpus 5.1 from the official website. [find](https://scholar.google.com/scholar?q=Common+Voice+5%2E1+%28Ardila+et+al%2E%2C+2019%29%3A+We+downloaded+the+English+subset+of+Common+Voice+Corpus+5%2E1+from+the+official+website%2E)
83. Artie bias corpus (Meyer et al., 2020): We used the Artie bias corpus. This is a subset of the Common Voice dataset. [find](https://scholar.google.com/scholar?q=Artie+bias+corpus+%28Meyer+et+al%2E%2C+2020%29%3A+We+used+the+Artie+bias+corpus%2E+This+is+a+subset+of+the+Common+Voice+dataset%2E)
84. CallHome and Switchboard: We used the two corpora from LDC2002S09 and LDC2002T43. • WSJ: We used LDC93S6B and LDC94S13B and followed the s5 recipe to preprocess the dataset. • CORAAL: We used the 231 interviews from CORAAL (Kendall & Farrington, 2021) and used the preprocessing script from the FairS… [find](https://scholar.google.com/scholar?q=CallHome+and+Switchboard%3A+We+used+the+two+corpora+from+LDC2002S09+and+LDC2002T43%2E+%E2%80%A2+WSJ%3A+We+used+LDC93S6B+and+LDC94S13B+and+followed+the+s5+recipe+to+preprocess+the+dataset%2E+%E2%80%A2+CORAAL%3A+We+used+the+231+)
85. CHiME-6: For CHiME-6 (Watanabe et al., 2020), we downloaded the CHiME-5 dataset and followed the stage 0 of the s5 track1 recipe to create the CHiME-6 dataset which fixes synchronization. We then used the binaural recordings ( * P??.wav) and the corresponding transcripts. [find](https://scholar.google.com/scholar?q=CHiME%2D6%3A+For+CHiME%2D6+%28Watanabe+et+al%2E%2C+2020%29%2C+we+downloaded+the+CHiME%2D5+dataset+and+followed+the+stage+0+of+the+s5+track1+recipe+to+create+the+CHiME%2D6+dataset+which+fixes+synchronization%2E+We+then+used)
86. AMI-IHM and AMI-SDM1: We preprocessed the AMI Corpus by following the stage 0 ad 2 of the s5b recipe. A.2. Long-form English-only datasets [find](https://scholar.google.com/scholar?q=AMI%2DIHM+and+AMI%2DSDM1%3A+We+preprocessed+the+AMI+Corpus+by+following+the+stage+0+ad+2+of+the+s5b+recipe%2E+A%2E2%2E+Long%2Dform+English%2Donly+datasets)