---
title: "Speech & spoken language — papers — Elamur Research"
description: "Speech & spoken language: 233 papers of the Elamur Research library, the most cited first."
canonical: "https://research.elamur.ai/free/en/topics/speech/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/topics/speech/)

[home](https://md.research.elamur.ai/free/en/) / [topics](https://md.research.elamur.ai/free/en/topics/) / Speech & spoken language

# 🎙️ Speech & spoken language

233 papers · page 1 of 4 · 1–25 · the most cited first · [all topics](https://md.research.elamur.ai/free/en/topics/)

Spoken language: recognizing and synthesizing speech, prosody and the sounds of words.

A survey · about 20 minutes to read · every paper named here has its own page in this library

**In this article**

1. [What it is](#a-what-it-is)
2. [History and milestones](#a-history-and-milestones)
3. [Main problems and tasks](#a-main-problems-and-tasks)
4. [Methods and schools](#a-methods-and-schools)
5. [Datasets, resources and benchmarks](#a-datasets-resources-and-benchmarks)
6. [Landmark papers in this library](#a-landmark-papers-in-this-library)
7. [People](#a-people)
8. [Institutions and places](#a-institutions-and-places)
9. [Open questions](#a-open-questions)
10. [Neighbouring topics](#a-neighbouring-topics)
11. [Key works in this library](#a-key-works-in-this-library)

Speech and spoken language processing is the part of language technology that deals with language as sound rather than as writing. It covers machines that turn speech into text (automatic speech recognition, ASR), text into speech (speech synthesis, TTS), speech in one language into text or speech in another (speech translation), and spoken input into meaning or action (spoken language understanding and spoken dialogue). Around these tasks sit the study of prosody — rhythm, stress, pitch and timing — and of the paralinguistic layer that carries emotion, speaker identity and attitude.

The field is older than most of text-based natural language processing and was for decades a separate community with its own conferences, corpora and statistical habits. Hidden Markov models, weighted finite-state machines, n-gram language models and the habit of measuring progress by word error rate on shared test sets all matured in speech before they spread to the rest of computational linguistics. Since the mid-2010s the two communities have converged on shared neural architectures and pretraining, and speech is increasingly one more modality for a general model. The papers here range from classic foundations to recent work on Arabic, Hebrew, Amharic, Maltese, Kurdish and other languages long ignored by mainstream benchmarks.

## What it is

Speech processing starts from a continuous, noisy signal. A second of audio at telephone or wideband quality contains thousands of samples; words are not separated by silences; the same word sounds different from speaker to speaker, room to room and microphone to microphone. Every task in the field is in some way about mapping between this signal and discrete linguistic units — phones, characters, word pieces, words, intents or translations.

The core notions are shared across tasks:

- **Acoustic features.** Classic systems compute short-time spectra over overlapping windows (typically 25 ms every 10 ms) and summarise them as mel-filterbank energies or MFCCs. Modern systems often learn features directly from the waveform or from log-mel spectrograms.
- **Acoustic, pronunciation and language models.** The traditional recognizer factors the problem into a model of how sounds realise phones, a lexicon mapping words to phone sequences, and a language model giving the prior probability of word sequences.
- **Decoding.** A search over a large graph of hypotheses, usually beam search, finds the most probable transcription; output may be a single string, an n-best list or a lattice.
- **Evaluation by edit distance.** Word error rate (WER) and character error rate (CER) count substitutions, deletions and insertions against a reference transcript. Speech translation borrows BLEU, chrF and COMET from machine translation; synthesis relies on listening tests scored as mean opinion scores (MOS).

The topic differs from its neighbours in emphasis rather than in sharp boundaries. [Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) study conversation management whether typed or spoken; this topic covers the spoken front end and the problems peculiar to speech, such as recognition errors and turn-taking cues. [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) works on text; speech translation inherits its methods but must cope with missing punctuation, disfluency and automatic segmentation. [Speech act theory](https://md.research.elamur.ai/free/en/topics/speech-act-theory/) is a branch of pragmatics about what utterances do, whether spoken or written, and despite the name sits mostly outside this topic.

## History and milestones

Landmark papers on this topic: the 12 most cited, in time The 12 most cited papers on this topic span 1997–2022; the dot grows with the citations.

**The numbers**

| Paper | Citations |
| --- | --- |
| [1997 — Finite-State Transducers in Language and Speech Processing](https://md.research.elamur.ai/free/en/papers/mohri-1997-finite-state-transducers-language-speech/) | 1,023 |
| [2014 — Deep Speech: Scaling up end-to-end speech recognition](https://md.research.elamur.ai/free/en/papers/hannun-2014-deep-speech-scaling-up-end/) | 1,585 |
| [2015 — Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://md.research.elamur.ai/free/en/papers/amodei-2015-deep-speech-2-end-end/) | 2,314 |
| [2015 — Attention-Based Models for Speech Recognition](https://md.research.elamur.ai/free/en/papers/chorowski-2015-attention-based-models-speech-recognition/) | 1,787 |
| [2016 — WaveNet: A Generative Model for Raw Audio](https://md.research.elamur.ai/free/en/papers/oord-2016-wavenet-generative-model-raw-audio/) | 5,190 |
| [2018 — Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://md.research.elamur.ai/free/en/papers/coucke-2018-snips-voice-platform-embedded-spoken/) | 595 |
| [2019 — SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://md.research.elamur.ai/free/en/papers/park-2019-specaugment-simple-data-augmentation-method/) | 3,809 |
| [2019 — Common Voice: A Massively-Multilingual Speech Corpus](https://md.research.elamur.ai/free/en/papers/ardila-2019-common-voice-massively-multilingual-speech/) | 736 |
| [2020 — Conformer: Convolution-augmented Transformer for Speech Recognition](https://md.research.elamur.ai/free/en/papers/gulati-2020-conformer-convolution-augmented-transformer-speech/) | 3,439 |
| [2021 — SoundStream: An End-to-End Neural Audio Codec](https://md.research.elamur.ai/free/en/papers/zeghidour-2021-soundstream-end-end-neural-audio/) | 727 |
| [2022 — Robust Speech Recognition via Large-Scale Weak Supervision](https://md.research.elamur.ai/free/en/papers/radford-2022-robust-speech-recognition-large-scale/) | 2,501 |
| [2022 — AudioLM: a Language Modeling Approach to Audio Generation](https://md.research.elamur.ai/free/en/papers/borsos-2022-audiolm-language-modeling-approach-audio/) | 570 |

### From templates to statistics

Early recognizers of the 1950s and 1960s matched isolated digits or words against stored templates. Through the 1970s, large research programmes in the United States funded systems for connected speech, and two approaches competed: knowledge-based systems that encoded phonetic and linguistic rules, and statistical systems that learned from data. The statistical approach, championed at IBM and elsewhere, won decisively. By the late 1980s the standard architecture was a hidden Markov model whose emission densities were Gaussian mixtures, combined with an n-gram language model and trained on transcribed corpora such as TIMIT and, later, Switchboard.

The 1990s and 2000s refined this architecture rather than replacing it. Weighted finite-state transducers gave a clean algebra for combining the lexicon, context-dependency rules and language model into a single searchable graph; [Finite-State Transducers in Language and Speech Processing](https://md.research.elamur.ai/free/en/papers/mohri-1997-finite-state-transducers-language-speech/) is the classic statement of that framework. Government evaluation programmes pushed systems from read speech towards broadcast news and conversational telephone speech, and data collection grew to match, as with [The Fisher Corpus: a Resource for the Next Generations of Speech-to-Text](https://md.research.elamur.ai/free/en/papers/cieri-2004-fisher-corpus-resource-next-generations/). In parallel, work on dialogue acts and prosody, such as [Lexical, Prosodic, and Syntactic Cues for Dialog Acts](https://md.research.elamur.ai/free/en/papers/jurafsky-1998-lexical-prosodic-syntactic-cues-dialog/), looked at what conversational speech does beyond its words.

Speech translation emerged as a research field in the 1990s, with restricted travel and booking domains; [Toward a Broad-coverage Bilingual Corpus for Speech Translation of Travel Conversations in the Real World](https://md.research.elamur.ai/free/en/papers/takezawa-2002-broad-coverage-bilingual-corpus-speech/) belongs to that phase. The IWSLT workshop, begun in 2004, gave the field a yearly evaluation campaign, and early entries were statistical phrase-based systems such as the [Edinburgh System Description for the 2005 IWSLT Speech Translation Evaluation](https://md.research.elamur.ai/free/en/papers/koehn-2005-edinburgh-system-description-2005-iwslt/).

### The neural turn

Around 2010–2012, deep neural networks replaced Gaussian mixtures as the acoustic model inside the HMM framework, giving the "hybrid" DNN-HMM recognizer that dominated for several years and that toolkits such as Kaldi made widely available. The next step was to remove the HMM altogether. Connectionist temporal classification (CTC), introduced in 2006, let a network be trained on unaligned transcripts; [Deep Speech: Scaling up end-to-end speech recognition](https://md.research.elamur.ai/free/en/papers/hannun-2014-deep-speech-scaling-up-end/) and its successor [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://md.research.elamur.ai/free/en/papers/amodei-2015-deep-speech-2-end-end/) showed that a CTC network trained on enough data with enough compute could rival engineered pipelines. At the same time, [Attention-Based Models for Speech Recognition](https://md.research.elamur.ai/free/en/papers/chorowski-2015-attention-based-models-speech-recognition/) brought the encoder–decoder with attention from translation into recognition.

Synthesis changed just as sharply. [WaveNet: A Generative Model for Raw Audio](https://md.research.elamur.ai/free/en/papers/oord-2016-wavenet-generative-model-raw-audio/) generated the waveform sample by sample and was judged far more natural than the concatenative and parametric systems of the time.

Papers on this topic by year 233 dated papers from 1898 to 2026 (1 before 1966 are not drawn); the busiest year is 2026 with 70, and 83% of them appeared in 2018 or later.

**The numbers**

| Year | Papers |
| --- | --- |
| 2026 | 70 |
| 2025 | 15 |
| 2024 | 18 |
| 2023 | 21 |
| 2022 | 15 |
| 2021 | 23 |
| 2020 | 12 |
| 2019 | 8 |
| 2018 | 12 |
| 2017 | 6 |
| 2016 | 4 |
| 2015 | 4 |
| 2014 | 5 |
| 2013 | 3 |
| 2012 | 1 |
| 2011 | 1 |
| 2010 | 1 |
| 2008 | 1 |
| 2007 | 1 |
| 2006 | 2 |
| 2005 | 2 |
| 2004 | 2 |
| 2002 | 2 |
| 2000 | 1 |
| 1998 | 1 |
| 1997 | 1 |
| 1898 | 1 |

### Transformers, self-supervision and scale

From 2019 the pace accelerated. [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://md.research.elamur.ai/free/en/papers/park-2019-specaugment-simple-data-augmentation-method/) showed that masking parts of the spectrogram was enough to let end-to-end models overtake hybrid systems. [Conformer: Convolution-augmented Transformer for Speech Recognition](https://md.research.elamur.ai/free/en/papers/gulati-2020-conformer-convolution-augmented-transformer-speech/) became the default encoder. Self-supervised models such as wav2vec 2.0 and HuBERT learned representations from untranscribed audio, and large multilingual corpora — [Common Voice: A Massively-Multilingual Speech Corpus](https://md.research.elamur.ai/free/en/papers/ardila-2019-common-voice-massively-multilingual-speech/), [MLS: A Large-Scale Multilingual Dataset for Speech Research](https://md.research.elamur.ai/free/en/papers/pratap-2020-mls-large-scale-multilingual-dataset/), [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://md.research.elamur.ai/free/en/papers/wang-2021-voxpopuli-large-scale-multilingual-speech/) — supplied the data. [Robust Speech Recognition via Large-Scale Weak Supervision](https://md.research.elamur.ai/free/en/papers/radford-2022-robust-speech-recognition-large-scale/), the Whisper paper, took the opposite route to self-supervision and trained on a very large collection of loosely transcribed web audio.

The latest phase treats audio as a sequence of discrete tokens. Neural codecs such as [SoundStream: An End-to-End Neural Audio Codec](https://md.research.elamur.ai/free/en/papers/zeghidour-2021-soundstream-end-end-neural-audio/) and [High Fidelity Neural Audio Compression](https://md.research.elamur.ai/free/en/papers/defossez-2022-high-fidelity-neural-audio-compression/) turn sound into compact codes, and language models over those codes generate speech ([AudioLM: a Language Modeling Approach to Audio Generation](https://md.research.elamur.ai/free/en/papers/borsos-2022-audiolm-language-modeling-approach-audio/), [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://md.research.elamur.ai/free/en/papers/wang-2023-neural-codec-language-models-zero/)). Large language models with audio inputs and outputs followed, among them [SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities](https://md.research.elamur.ai/free/en/papers/zhang-2023-speechgpt-empowering-large-language-models/) and the Qwen audio models.

## Main problems and tasks

### Speech recognition

ASR remains the central task. Its difficulty depends on the setting: read speech in a quiet room is close to solved for major languages, while conversational, accented, far-field, overlapped or code-switched speech is not. Children's speech, disordered speech and dialects without a standard spelling are particularly hard; papers here address each of these, from [Arabic Dysarthric Speech Recognition Using Adversarial and Signal-Based Augmentation](https://md.research.elamur.ai/free/en/papers/baali-2023-arabic-dysarthric-speech-recognition-using/) to [Arabic Little STT: Arabic Children Speech Recognition Dataset](https://md.research.elamur.ai/free/en/papers/alkadri-2025-arabic-little-stt-arabic-children/).

### Speech synthesis

TTS aims at speech that is intelligible, natural and controllable. Modern work focuses on multi-speaker and zero-shot voice cloning, expressive prosody, and languages whose writing leaves out information needed for pronunciation — short vowels in Arabic and Hebrew, for example. [Phonikud: Overcoming Phonetic Underspecification for Hebrew Text-To-Speech](https://md.research.elamur.ai/free/en/papers/kolani-2025-phonikud-overcoming-phonetic-underspecification-hebrew/) and [More Data, Fewer Diacritics: Scaling Arabic TTS](https://md.research.elamur.ai/free/en/papers/musleh-2026-more-data-fewer-diacritics-scaling/) tackle that gap from two directions.

### Speech translation

Speech translation can be built as a cascade (ASR followed by MT, possibly followed by TTS) or as a single end-to-end model. Sub-problems include simultaneous translation with limited delay, segmentation of unpunctuated input, dubbing with matched timing and lip movement, formality and gender control, and translation from languages that have little or no written data.

### Spoken language understanding and dialogue

SLU maps an utterance to an intent and its slots (for example, *set an alarm* with *time = 7 am*). Pipelines run ASR and then a text classifier; end-to-end models go directly from audio. Dialogue state tracking keeps a belief about the user's goal across turns, and paralinguistic cues — backchannels, pauses, intonation — help decide what a turn means and when to speak.

### Paralinguistics and speaker analysis

Speech emotion recognition, speaker identification and diarization, language and dialect identification, and detection of synthetic speech all use the non-lexical side of the signal. [PhonemeDF: A Synthetic Speech Dataset for Audio Deepfake Detection and Naturalness Evaluation](https://md.research.elamur.ai/free/en/papers/nallaguntla-2026-phonemedf-synthetic-speech-dataset-audio/) represents the growing concern with spoofing.

### Spoken language as linguistic data

Finally, some papers treat speech as an object for linguistics: how to annotate syntax in transcribed talk ([Annotation guidelines of UD and SUD treebanks for spoken corpora: A proposal](https://md.research.elamur.ai/free/en/papers/kahane-2021-annotation-guidelines-ud-sud-treebanks/)), how prosody marks syntactic structure, or how to align audio, video and parse trees ([The Aligned Multimodal Movie Treebank: An audio, video, dependency-parse treebank](https://md.research.elamur.ai/free/en/papers/yaari-2022-aligned-multimodal-movie-treebank-audio/)).

## Methods and schools

### The statistical pipeline

For roughly three decades the dominant method was the HMM-based recognizer: context-dependent phone models, Gaussian mixture or later neural emission scores, a pronunciation lexicon, an n-gram language model, and a decoding graph compiled with weighted transducers. Its strengths are modularity and data efficiency: components train on different data, and words or domains can be added without retraining the whole. Several papers here still build on it, for instance the Kaldi-based baseline of [Speech Recognition Challenge in the Wild: Arabic MGB-3](https://md.research.elamur.ai/free/en/papers/ali-2017-speech-recognition-challenge-wild-arabic/), the private on-device assistant of [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://md.research.elamur.ai/free/en/papers/coucke-2018-snips-voice-platform-embedded-spoken/), and the union-of-graphs decoder for code-switching in [Arabic Code-Switching Speech Recognition using Monolingual Data](https://md.research.elamur.ai/free/en/papers/ali-2021-arabic-code-switching-speech-recognition/).

### End-to-end models

End-to-end recognizers map acoustics straight to characters or word pieces. Three loss families are in use: CTC, which assumes conditionally independent outputs and is simple to decode; attention-based encoder–decoders, which model output dependencies but can lose alignment on long input; and the transducer (RNN-T), which combines a streaming-friendly alignment with an output predictor. Many systems train with a joint CTC and attention objective. They remove hand-built components and scale with data, but need more transcribed audio and are harder to adapt to new words.

### Self-supervised and weakly supervised learning

Self-supervised encoders learn from untranscribed audio by predicting masked or future parts of the signal, and then need only a little labelled data for fine-tuning. Weak supervision, as in Whisper, instead gathers enormous quantities of imperfect transcripts. Both have changed what is possible for low-resource languages, though papers here repeatedly find that off-the-shelf multilingual models perform poorly on dialects and code-switched speech until they are adapted.

### Discrete units and "textless" processing

A distinct school represents speech as discrete units obtained by clustering self-supervised features or by quantizing a neural codec. Units let text-style language models operate directly on audio. [On Generative Spoken Language Modeling from Raw Audio](https://md.research.elamur.ai/free/en/papers/lakhotia-2021-generative-spoken-language-modeling-raw/) set out an evaluation framework for this "textless" approach; AudioLM, VALL-E and SpeechGPT build on it.

### Cascade versus end-to-end translation

In speech translation, the cascade-versus-direct debate has run for a decade. Cascades benefit from abundant ASR and MT data and are easy to inspect; direct models avoid error propagation and can use audio cues such as the speaker's voice. Work such as [Tied Multitask Learning for Neural Speech Translation](https://md.research.elamur.ai/free/en/papers/anastasopoulos-2018-tied-multitask-learning-neural-speech/) and [Understanding and Bridging the Modality Gap for Speech Translation](https://md.research.elamur.ai/free/en/papers/fang-2023-understanding-bridging-modality-gap-speech/) tries to obtain the advantages of both by sharing representations between transcription and translation.

## Datasets, resources and benchmarks

Shared corpora have driven the field since its statistical turn. Well-known resources include:

| Resource | Kind | Notes |
| --- | --- | --- |
| TIMIT | read English, phonetically labelled | the standard phone-recognition benchmark since the early 1990s |
| Switchboard and Fisher | conversational telephone English | Fisher scaled up collection with many speakers and short calls |
| LibriSpeech | read English audiobooks | the most common ASR benchmark since 2015 |
| LibriTTS | audiobooks re-segmented for synthesis | 24 kHz audio, sentence-level text |
| Common Voice | crowdsourced read speech | openly licensed, many languages |
| MLS, VoxPopuli | multilingual audiobooks; European Parliament | large multilingual ASR and pretraining data |
| MuST-C | TED talks with translations | the standard benchmark for English-to-X speech translation |
| SLURP | spoken commands to a home assistant | intent, scenario and entity labels |
| AudioCaps | sound clips with written captions | audio captioning beyond speech |
| MGB-2, MGB-3, MGB-5 | Arabic broadcast and web video | standard and dialectal Arabic benchmarks |

In this library the corpus papers include [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://md.research.elamur.ai/free/en/papers/zen-2019-libritts-corpus-derived-librispeech-text/), [MuST-C: a Multilingual Speech Translation Corpus](https://md.research.elamur.ai/free/en/papers/gangi-2019-must-c-multilingual-speech-translation/), [SLURP: A Spoken Language Understanding Resource Package](https://md.research.elamur.ai/free/en/papers/bastianelli-2020-slurp-spoken-language-understanding-resource/) and [AudioCaps: Generating Captions for Audios in The Wild](https://md.research.elamur.ai/free/en/papers/kim-2019-audiocaps-generating-captions-audios-wild/). Smaller-language resources are a strong theme, for example [MASRI-HEADSET: A Maltese Corpus for Speech Recognition](https://md.research.elamur.ai/free/en/papers/mena-2020-masri-headset-maltese-corpus-speech/), [HebDB: a Weakly Supervised Dataset for Hebrew Speech Processing](https://md.research.elamur.ai/free/en/papers/turetzky-2024-hebdb-weakly-supervised-dataset-hebrew/), [ClArTTS: An Open-Source Classical Arabic Text-to-Speech Corpus](https://md.research.elamur.ai/free/en/papers/kulkarni-2023-clartts-open-source-classical-arabic/), [ArVoice: A Multi-Speaker Dataset for Arabic Speech Synthesis](https://md.research.elamur.ai/free/en/papers/toyin-2025-arvoice-multi-speaker-dataset-arabic/), [A New Amharic Speech Emotion Dataset and Classification Benchmark](https://md.research.elamur.ai/free/en/papers/retta-2022-new-amharic-speech-emotion-dataset/), [ParlaSpeech 3.0: Richly Annotated Spoken Parliamentary Corpora of Croatian, Czech, Polish, and Serbian](https://md.research.elamur.ai/free/en/papers/ljubesic-2026-parlaspeech-3-0-richly-annotated/), [FalAR: A Large-scale Speaker-Annotated European Portuguese Speech Corpus of Parliamentary Sessions](https://md.research.elamur.ai/free/en/papers/teixeira-2026-falar-large-scale-speaker-annotated/) and [The Maaloula Aramaic Speech Corpus (MASC): From Printed Material to a Lemmatized and Time-Aligned Corpus](https://md.research.elamur.ai/free/en/papers/eid-2022-maaloula-aramaic-speech-corpus-masc/). For Arabic, [Masader: Metadata Sourcing for Arabic Text and Speech Data Resources](https://md.research.elamur.ai/free/en/papers/alyafeai-2022-masader-metadata-sourcing-arabic-text/) catalogues what exists, and the [Open Universal Arabic ASR Leaderboard](https://md.research.elamur.ai/free/en/papers/wang-2024-open-universal-arabic-asr-leaderboard/) compares models on common test sets.

Evaluation methods are themselves a research subject. [Evaluating Machine Translation Output with Automatic Sentence Segmentation](https://md.research.elamur.ai/free/en/papers/matusov-2005-evaluating-machine-translation-output-automatic/) solved the problem of scoring translations whose segment boundaries differ from the reference, a step every speech translation evaluation still uses in some form.

## Landmark papers in this library

Citations of the papers on this topic, by when they were published Papers published in 2015–2019 gather 46% of the 37,553 citations of the papers on this topic.

**The numbers**

| Published | Papers and their citations |
| --- | --- |
| 2025–2029 | 85 papers, 443 citations |
| 2020–2024 | 89 papers, 14,288 citations |
| 2015–2019 | 34 papers, 17,388 citations |
| 2010–2014 | 11 papers, 2,234 citations |
| 2005–2009 | 6 papers, 993 citations |
| 2000–2004 | 5 papers, 1,052 citations |
| 1995–1999 | 2 papers, 1,155 citations |
| 1895–1899 | 1 papers, 0 citations |

### WaveNet (2016)

[WaveNet: A Generative Model for Raw Audio](https://md.research.elamur.ai/free/en/papers/oord-2016-wavenet-generative-model-raw-audio/) predicted each audio sample from all previous ones with stacks of dilated causal convolutions, so that the receptive field grew exponentially with depth. Used as the back end of a TTS system it was preferred by listeners over the best concatenative and parametric systems, in both English and Mandarin.

### SpecAugment (2019)

[SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://md.research.elamur.ai/free/en/papers/park-2019-specaugment-simple-data-augmentation-method/) applied warping and masking of frequency bands and time spans to the input spectrogram during training. The trick costs almost nothing, and with longer training schedules it let attention-based recognizers beat tuned hybrid systems on LibriSpeech and Switchboard.

### Conformer (2020)

[Conformer: Convolution-augmented Transformer for Speech Recognition](https://md.research.elamur.ai/free/en/papers/gulati-2020-conformer-convolution-augmented-transformer-speech/) combined self-attention for long-range context with convolution for local acoustic detail inside a single block. It gave better accuracy than pure Transformer or pure CNN encoders at every model size tested and became the standard encoder for both research and production recognizers.

### Whisper (2022)

[Robust Speech Recognition via Large-Scale Weak Supervision](https://md.research.elamur.ai/free/en/papers/radford-2022-robust-speech-recognition-large-scale/) trained an ordinary encoder–decoder Transformer on a huge, automatically filtered set of web audio with existing transcripts, covering many languages and tasks at once. Used without fine-tuning, it approached supervised systems and showed unusual robustness across domains. Later papers here benchmark or adapt it, such as [N-Shot Benchmarking of Whisper on Diverse Arabic Speech Recognition](https://md.research.elamur.ai/free/en/papers/talafha-2023-n-shot-benchmarking-whisper-diverse/) and [Overcoming Data Scarcity in Multi-Dialectal Arabic ASR via Whisper Fine-Tuning](https://md.research.elamur.ai/free/en/papers/ozyilmaz-2025-overcoming-data-scarcity-multi-dialectal/).

### Deep Speech and Deep Speech 2 (2014–2015)

The two Baidu papers, [Deep Speech: Scaling up end-to-end speech recognition](https://md.research.elamur.ai/free/en/papers/hannun-2014-deep-speech-scaling-up-end/) and [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://md.research.elamur.ai/free/en/papers/amodei-2015-deep-speech-2-end-end/), replaced the recognizer's component chain with one recurrent network trained by CTC. The first relied on synthesized noisy data for robustness; the second added convolutional layers, batch normalisation and a high-performance multi-GPU training stack, and was deployed for Mandarin. Together they made the argument that data and compute could substitute for hand engineering.

### Attention-based recognition (2015)

[Attention-Based Models for Speech Recognition](https://md.research.elamur.ai/free/en/papers/chorowski-2015-attention-based-models-speech-recognition/) found that content-only attention, borrowed from translation, lost its place on long utterances, and added location awareness by feeding the previous alignment back into the scoring function.

### Weighted finite-state transducers (1997)

[Finite-State Transducers in Language and Speech Processing](https://md.research.elamur.ai/free/en/papers/mohri-1997-finite-state-transducers-language-speech/) gave determinization and minimization algorithms for weighted transducers and showed how they shrink recognition lattices. It provided the theoretical basis for the WFST decoders used in speech toolkits for the following two decades.

### Common Voice (2019)

[Common Voice: A Massively-Multilingual Speech Corpus](https://md.research.elamur.ai/free/en/papers/ardila-2019-common-voice-massively-multilingual-speech/) described Mozilla's volunteer platform, where people read sentences aloud and others validate the clips, released into the public domain. Any community can add its language, which made Common Voice the main open source of read speech for many small languages; [Building Collaborative Speech Corpora for Low-Resource Languages: The Galician Dataset in Mozilla Common Voice](https://md.research.elamur.ai/free/en/papers/vladu-2026-building-collaborative-speech-corpora-low/) is one later example.

### SoundStream and EnCodec (2021–2022)

[SoundStream: An End-to-End Neural Audio Codec](https://md.research.elamur.ai/free/en/papers/zeghidour-2021-soundstream-end-end-neural-audio/) and [High Fidelity Neural Audio Compression](https://md.research.elamur.ai/free/en/papers/defossez-2022-high-fidelity-neural-audio-compression/) trained encoder, residual vector quantizer and decoder jointly with adversarial losses, beating classical codecs at low bitrates. Their discrete codes became the vocabulary for a generation of audio language models.

### AudioLM and VALL-E (2022–2023)

[AudioLM: a Language Modeling Approach to Audio Generation](https://md.research.elamur.ai/free/en/papers/borsos-2022-audiolm-language-modeling-approach-audio/) combined coarse semantic tokens for long-term structure with fine codec tokens for acoustic detail and generated coherent continuations of speech and piano music without any transcripts. [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://md.research.elamur.ai/free/en/papers/wang-2023-neural-codec-language-models-zero/) applied the same idea to TTS: given phonemes and a three-second sample of an unseen voice, it continued the code sequence in that voice.

### MuST-C (2019)

[MuST-C: a Multilingual Speech Translation Corpus](https://md.research.elamur.ai/free/en/papers/gangi-2019-must-c-multilingual-speech-translation/) aligned English TED audio with transcripts and translations into eight languages, giving end-to-end speech translation its first large public benchmark. It underlies much of the later IWSLT work and the gender-bias benchmark [Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpus](https://md.research.elamur.ai/free/en/papers/bentivogli-2020-gender-danger-evaluating-speech-translation/).

### Dialogue state tracking from ASR (2014)

[Word-Based Dialog State Tracking with Recurrent Neural Networks](https://md.research.elamur.ai/free/en/papers/henderson-2014-word-based-dialog-state-tracking/) skipped the separate semantic parser and updated the dialogue state directly from recognition hypotheses with a recurrent network, using delexicalised features to cope with unseen slot values. It was among the strongest entries in the second Dialog State Tracking Challenge.

### Bias in automatic captions (2017)

[Gender and Dialect Bias in YouTube's Automatic Captions](https://md.research.elamur.ai/free/en/papers/tatman-2017-gender-dialect-bias-youtube-s/) measured caption errors for speakers of different genders and regions and found consistently worse results for women and for Scottish speakers. It was an early and much-cited demonstration that recognition quality is unevenly distributed, a theme continued in [A Sociophonetic Analysis of Racial Bias in Commercial ASR Systems Using the Pacific Northwest English Corpus](https://md.research.elamur.ai/free/en/papers/scott-2026-sociophonetic-analysis-racial-bias-commercial/) and [Regional Variation in the Performance of ASR Models on Croatian and Serbian](https://md.research.elamur.ai/free/en/papers/samardzic-2026-regional-variation-performance-asr-models/).

### SeamlessM4T and Seamless (2023)

[SeamlessM4T: Massively Multilingual & Multimodal Machine Translation](https://md.research.elamur.ai/free/en/papers/communication-2023-seamlessm4t-massively-multilingual-multimodal-machine/) put recognition, text translation and speech-to-speech and speech-to-text translation for around a hundred languages into one model. [Seamless: Multilingual Expressive and Streaming Speech Translation](https://md.research.elamur.ai/free/en/papers/communication-2023-seamless-multilingual-expressive-streaming-speech/) extended it to preserve the speaker's vocal style and to translate while the speaker is still talking.

## People

Authors with the most papers on this topic Luisa Bentivogli leads with 7 papers; the 12 authors shown wrote 64 of the papers on this topic between them, counting each paper once per author.

**The numbers**

| Author | Papers here · their citations |
| --- | --- |
| [Luisa Bentivogli](https://md.research.elamur.ai/free/en/authors/luisa-bentivogli/) | 7 papers · 1,184 citations |
| [Yossi Adi](https://md.research.elamur.ai/free/en/authors/yossi-adi/) | 7 papers · 979 citations |
| [George Saon](https://md.research.elamur.ai/free/en/authors/george-saon/) | 7 papers · 774 citations |
| [Ahmed Ali](https://md.research.elamur.ai/free/en/authors/ahmed-ali/) | 7 papers · 329 citations |
| [Eugene Kharitonov](https://md.research.elamur.ai/free/en/authors/eugene-kharitonov/) | 5 papers · 1,102 citations |
| [Michael Picheny](https://md.research.elamur.ai/free/en/authors/michael-picheny/) | 5 papers · 722 citations |
| [Claudia Borg](https://md.research.elamur.ai/free/en/authors/claudia-borg/) | 5 papers · 183 citations |
| [Shammur Absar Chowdhury](https://md.research.elamur.ai/free/en/authors/shammur-absar-chowdhury/) | 5 papers · 167 citations |
| [Changhan Wang](https://md.research.elamur.ai/free/en/authors/changhan-wang/) | 4 papers · 1,071 citations |
| [Juan Pino](https://md.research.elamur.ai/free/en/authors/juan-pino/) | 4 papers · 1,071 citations |
| [Emmanuel Dupoux](https://md.research.elamur.ai/free/en/authors/emmanuel-dupoux/) | 4 papers · 913 citations |
| [Roldano Cattoni](https://md.research.elamur.ai/free/en/authors/roldano-cattoni/) | 4 papers · 865 citations |

- **[Luisa Bentivogli](https://md.research.elamur.ai/free/en/authors/luisa-bentivogli/)** (Fondazione Bruno Kessler) is the most prolific author on this topic. Her work here spans the MuST-C corpus, the MuST-SHE gender benchmark and the organisation of several IWSLT evaluation campaigns, including the [Report on the 11th IWSLT evaluation campaign](https://md.research.elamur.ai/free/en/papers/cettolo-2014-report-11th-iwslt-evaluation-campaign/).
- **[Ahmed Ali](https://md.research.elamur.ai/free/en/authors/ahmed-ali/)** has driven much of the Arabic speech recognition work here: the MGB-3 challenge on dialectal web video, end-to-end Arabic ASR compared with human transcribers, and code-switching recognition from monolingual data.
- **[Roldano Cattoni](https://md.research.elamur.ai/free/en/authors/roldano-cattoni/)** and **[Matteo Negri](https://md.research.elamur.ai/free/en/authors/matteo-negri/)** (Fondazione Bruno Kessler) co-built MuST-C and MuST-SHE with Bentivogli and contributed to the IWSLT reports.
- **[Mauro Cettolo](https://md.research.elamur.ai/free/en/authors/mauro-cettolo/)** (Fondazione Bruno Kessler) led the 2014 IWSLT campaign report and remains a long-standing organiser of spoken translation evaluation.
- **[Antonios Anastasopoulos](https://md.research.elamur.ai/free/en/authors/antonios-anastasopoulos/)** (George Mason University) proposed the "triangle" multitask model for translating speech in endangered-language documentation and led the IWSLT 2022 findings report.
- **[Yossi Adi](https://md.research.elamur.ai/free/en/authors/yossi-adi/)** (Hebrew University of Jerusalem) co-authored the generative spoken language modelling benchmark and the EnCodec neural codec, and has applied this line of work to Hebrew speech data and synthesis.
- **[Shammur Absar Chowdhury](https://md.research.elamur.ai/free/en/authors/shammur-absar-chowdhury/)** (Hamad bin Khalifa University) works on multilingual and code-switching Arabic ASR and on spoken interaction with large language models, as in [WASIL: In-the-Wild Arabic Spoken Interactions with LLMs](https://md.research.elamur.ai/free/en/papers/ali-2026-wasil-wild-arabic-spoken-interactions/).
- **[Amir Hussein](https://md.research.elamur.ai/free/en/authors/amir-hussein/)** (Johns Hopkins University) built the first end-to-end Transformer recognizer for Arabic in ESPnet, with [Shinji Watanabe](https://md.research.elamur.ai/free/en/authors/shinji-watanabe/), and co-authored the multilingual code-switching work.
- **[Hanan Aldarmaki](https://md.research.elamur.ai/free/en/authors/hanan-aldarmaki/)** (Mohamed bin Zayed University of Artificial Intelligence) works on Arabic synthesis and diacritics, from the ClArTTS corpus to [Diacritic Recognition Performance in Arabic ASR](https://md.research.elamur.ai/free/en/papers/aldarmaki-2023-diacritic-recognition-performance-arabic-asr/).
- **[Marie Tahon](https://md.research.elamur.ai/free/en/authors/marie-tahon/)** and **[Mohammad Mohammadamini](https://md.research.elamur.ai/free/en/authors/mohammad-mohammadamini/)** (Le Mans Université) have built Kurdish speech resources, among them [Southern Kurdish Speech Recognition Resources and Benchmarking](https://md.research.elamur.ai/free/en/papers/mohammadamini-2026-southern-kurdish-speech-recognition-resources/) and [Central Kurdish Text-to-Speech and Its Application in Speech-to-Text Translation](https://md.research.elamur.ai/free/en/papers/mohammadamini-2026-central-kurdish-text-speech-its/).
- **[Yu Zhang](https://md.research.elamur.ai/free/en/authors/yu-zhang/)** and **[Chung-Cheng Chiu](https://md.research.elamur.ai/free/en/authors/chung-cheng-chiu/)** co-authored both SpecAugment and Conformer, two of the most cited works here; Zhang also co-authored LibriTTS.
- **[Heiga Zen](https://md.research.elamur.ai/free/en/authors/heiga-zen/)**, a central figure in statistical and neural speech synthesis, co-authored WaveNet and LibriTTS.
- **[Claudia Borg](https://md.research.elamur.ai/free/en/authors/claudia-borg/)** and **[Lonneke van der Plas](https://md.research.elamur.ai/free/en/authors/lonneke-van-der-plas/)** (University of Malta) produced the Maltese recognition and translation work, including [Analysis of Data Augmentation Methods for Low-Resource Maltese ASR](https://md.research.elamur.ai/free/en/papers/demarco-2021-analysis-data-augmentation-methods-low/) and [UM-DFKI Maltese Speech Translation](https://md.research.elamur.ai/free/en/papers/williams-2023-um-dfki-maltese-speech-translation/).
- **[Neil Zeghidour](https://md.research.elamur.ai/free/en/authors/neil-zeghidour/)** and **[Marco Tagliasacchi](https://md.research.elamur.ai/free/en/authors/marco-tagliasacchi/)** co-authored SoundStream and AudioLM, the codec-plus-language-model pairing behind much recent audio generation.
- **[Massa Baali](https://md.research.elamur.ai/free/en/authors/massa-baali/)** (Carnegie Mellon University) works on Arabic synthesis and disordered speech, including [Unsupervised Data Selection for TTS: Using Arabic Broadcast News as a Case Study](https://md.research.elamur.ai/free/en/papers/baali-2023-unsupervised-data-selection-tts-using/) and [FOOCTTS: Generating Arabic Speech with Acoustic Environment for Football Commentator](https://md.research.elamur.ai/free/en/papers/baali-2023-fooctts-generating-arabic-speech-acoustic/).

Earlier pioneers here include [Daniel Jurafsky](https://md.research.elamur.ai/free/en/authors/dan-jurafsky/), [Elizabeth Shriberg](https://md.research.elamur.ai/free/en/authors/elizabeth-shriberg/), [Steve Young](https://md.research.elamur.ai/free/en/authors/steve-young/), [Hermann Ney](https://md.research.elamur.ai/free/en/authors/hermann-ney/), [Steve Renals](https://md.research.elamur.ai/free/en/authors/steve-renals/), [Yoshua Bengio](https://md.research.elamur.ai/free/en/authors/yoshua-bengio/) and [Kyunghyun Cho](https://md.research.elamur.ai/free/en/authors/kyunghyun-cho/).

## Institutions and places

Where the authors on this topic work now Carnegie Mellon University is the current home of authors of 16 papers on this topic; counted are the authors with a page here, by their latest institution.

**The numbers**

| Institution | Papers |
| --- | --- |
| [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) | 16 papers |
| [Centre National de la Recherche Scientifique](https://md.research.elamur.ai/free/en/labs/centre-national-de-la-recherche-scientifique/) | 9 papers |
| [Chinese University of Hong Kong](https://md.research.elamur.ai/free/en/universities/chinese-university-of-hong-kong/) | 9 papers |
| [Meta](https://md.research.elamur.ai/free/en/companies/meta/) | 9 papers |
| [Fondazione Bruno Kessler](https://md.research.elamur.ai/free/en/labs/fondazione-bruno-kessler/) | 8 papers |
| [Google](https://md.research.elamur.ai/free/en/companies/google/) | 8 papers |
| [Hamad bin Khalifa University](https://md.research.elamur.ai/free/en/universities/hamad-bin-khalifa-university/) | 8 papers |
| [IBM](https://md.research.elamur.ai/free/en/companies/ibm/) | 8 papers |
| [New York University](https://md.research.elamur.ai/free/en/universities/new-york-university/) | 8 papers |
| [COMSATS University Islamabad](https://ror.org/00nqqvk19) | 7 papers |

Three groups stand out. The first is North American and European research centres: [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/), long a home of speech research, leads; [Johns Hopkins University](https://md.research.elamur.ai/free/en/universities/johns-hopkins-university/), [George Mason University](https://md.research.elamur.ai/free/en/universities/george-mason-university/) and [Meta](https://md.research.elamur.ai/free/en/companies/meta/) follow. In Europe, [Fondazione Bruno Kessler](https://md.research.elamur.ai/free/en/labs/fondazione-bruno-kessler/) in Trento is the centre of the IWSLT and MuST-C line of work, and [Le Mans Université](https://md.research.elamur.ai/free/en/universities/le-mans-universite/) with its [Laboratoire d’Informatique de l’Université du Mans](https://md.research.elamur.ai/free/en/labs/laboratoire-d-informatique-de-l-universite-du-mans/) hosts the Kurdish speech projects. The [Barcelona Supercomputing Center](https://md.research.elamur.ai/free/en/labs/barcelona-supercomputing-center/), the [Centre National de la Recherche Scientifique](https://md.research.elamur.ai/free/en/labs/centre-national-de-la-recherche-scientifique/) and the [Athena Research and Innovation Center In Information Communication & Knowledge Technologies](https://md.research.elamur.ai/free/en/labs/athena-research-and-innovation-center-in-information-communication-knowledge/) also appear. The [University of Cape Town](https://md.research.elamur.ai/free/en/universities/university-of-cape-town/) is listed for several authors of the translation and low-resource work.

The second group reflects the strong Arabic and Semitic presence: [Hamad bin Khalifa University](https://md.research.elamur.ai/free/en/universities/hamad-bin-khalifa-university/) in Qatar, [Mohamed bin Zayed University of Artificial Intelligence](https://md.research.elamur.ai/free/en/universities/mohamed-bin-zayed-university-of-artificial-intelligence/) in Abu Dhabi, [Zagazig University](https://md.research.elamur.ai/free/en/universities/zagazig-university/) in Egypt and the [Hebrew University of Jerusalem](https://md.research.elamur.ai/free/en/universities/hebrew-university-of-jerusalem/). The third is Chinese: [Zhejiang University](https://md.research.elamur.ai/free/en/universities/zhejiang-university/), the [Chinese University of Hong Kong](https://md.research.elamur.ai/free/en/universities/chinese-university-of-hong-kong/) and the [Chinese University of Hong Kong, Shenzhen](https://md.research.elamur.ai/free/en/universities/chinese-university-of-hong-kong-shenzhen/), the last of which has entered IWSLT shared tasks with systems such as [CUHKSZ Simultaneous Speech Translation System for IWSLT 2026](https://md.research.elamur.ai/free/en/papers/yang-2026-cuhksz-simultaneous-speech-translation-system/).

Countries of the authors on this topic United States comes first: authors working there now wrote 80 papers on this topic.

**The numbers**

| Country | Papers |
| --- | --- |
| [United States](https://md.research.elamur.ai/free/en/countries/united-states/) | 80 papers |
| [China](https://md.research.elamur.ai/free/en/countries/china/) | 38 papers |
| [United Kingdom](https://md.research.elamur.ai/free/en/countries/united-kingdom/) | 23 papers |
| [France](https://md.research.elamur.ai/free/en/countries/france/) | 22 papers |
| [Israel](https://md.research.elamur.ai/free/en/countries/israel/) | 11 papers |
| [Canada](https://md.research.elamur.ai/free/en/countries/canada/) | 10 papers |
| [Egypt](https://md.research.elamur.ai/free/en/countries/egypt/) | 10 papers |
| Hong Kong | 10 papers |
| [Spain](https://md.research.elamur.ai/free/en/countries/spain/) | 10 papers |
| [Italy](https://md.research.elamur.ai/free/en/countries/italy/) | 9 papers |

Where the papers on this topic were published Language Resources and Evaluation Conference (LREC) carries the most, 41 of 233.

**The numbers**

| Venue | Papers |
| --- | --- |
| [Language Resources and Evaluation Conference (LREC)](https://md.research.elamur.ai/free/en/venues/lrec/) | 41 papers |
| [International Conference on Spoken Language Translation (IWSLT)](https://md.research.elamur.ai/free/en/venues/iwslt/) | 27 papers |
| [Interspeech](https://md.research.elamur.ai/free/en/venues/interspeech-usb/) | 19 papers |
| [Annual Meeting of the Association for Computational Linguistics (ACL)](https://md.research.elamur.ai/free/en/venues/acl/) | 16 papers |
| [IEEE International Conference on Acoustics Speech and Signal Processing](https://md.research.elamur.ai/free/en/venues/ieee-international-conference-on-acoustics-speech-and-signal-processing/) | 7 papers |
| [Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)](https://md.research.elamur.ai/free/en/venues/naacl/) | 5 papers |
| [Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/) | 5 papers |
| [Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL)](https://md.research.elamur.ai/free/en/venues/sigdial/) | 4 papers |
| [Transactions of the Association for Computational Linguistics (TACL)](https://md.research.elamur.ai/free/en/venues/tacl/) | 4 papers |
| [Findings of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/findings/) | 3 papers |

The venues show where much of the recent work is published: the LREC conference, with its emphasis on language resources, and the IWSLT conference on spoken language translation account for a large share, followed by SIGDIAL for dialogue and the main ACL-family conferences.

## Open questions

- **Robustness and fairness.** Error rates still differ sharply across accents, dialects, genders, ages and recording conditions. Measuring these gaps reliably requires stratified test sets that most languages lack.
- **Languages without much data or without a standard spelling.** Dialectal Arabic, Kurdish, Tamasheq, Quechua, Mapuzugun and endangered languages such as Ikema Miyakoan raise problems of scoring as well as training, since several spellings of the same word may be valid. Papers such as [Automatic Speech Recognition for Documenting Endangered Languages: Case Study of Ikema Miyakoan](https://md.research.elamur.ai/free/en/papers/taguchi-2026-automatic-speech-recognition-documenting-endangered/) and [Amulwe Kimün: A Community-Grounded Demo, Resource, and ASR Baseline for Mapuzugun](https://md.research.elamur.ai/free/en/papers/oliva-2026-amulwe-kimun-community-grounded-demo/) explore how communities and researchers can share the work.
- **Code-switching.** Speakers mix languages within sentences, and most models trained on monolingual data handle switches poorly; see [Benchmarking Commercial ASR Systems on Code-Switching Speech: Arabic, Persian, and German](https://md.research.elamur.ai/free/en/papers/abdoli-2026-benchmarking-commercial-asr-systems-code/) and [ViMedCSS: A Vietnamese Medical Code-Switching Speech Dataset & Benchmark](https://md.research.elamur.ai/free/en/papers/nguyen-2026-vimedcss-vietnamese-medical-code-switching/).
- **Evaluation of speech output.** Listening tests are slow and inconsistent; automatic metrics for naturalness, expressiveness and translation quality of speech are still immature, as [Hurdles of Automatic Metric for Speech Translation Evaluation](https://md.research.elamur.ai/free/en/papers/zarzu-2026-hurdles-automatic-metric-speech-translation/) discusses.
- **Speech in large language models.** It is unclear how much of a text model's knowledge transfers when the input is speech; [Do Factual Recall Mechanisms Carry over from Text to Speech in Multimodal Language Models?](https://md.research.elamur.ai/free/en/papers/modica-2026-do-factual-recall-mechanisms-carry/) examines this directly. Full-duplex conversation, where a system listens and speaks at once, is another frontier ([DialogueSidon: Recovering Full-Duplex Dialogue Tracks from In-the-Wild Dialogue Audio](https://md.research.elamur.ai/free/en/papers/nakata-2026-dialoguesidon-recovering-full-duplex-dialogue/)).
- **Prosody and paralinguistics.** Intonation, emphasis and emotion are usually lost in recognition and translation; tools such as [ToneSwiper: Facilitating Manual ToDI-annotation of Dutch Prosody](https://md.research.elamur.ai/free/en/papers/westera-2026-toneswiper-facilitating-manual-todi-annotation/) are early steps.
- **Misuse.** Zero-shot voice cloning makes impersonation cheap; reliable detection and watermarking of synthetic speech remain open problems.

## Neighbouring topics

Neighbouring topics: papers shared Semitic languages shares the most: 77 of the 233 papers here (33%) stand on both.

**The numbers**

| Topic | Papers shared |
| --- | --- |
| [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) | 77 shared · 33% |
| [English](https://md.research.elamur.ai/free/en/topics/english/) | 39 shared · 17% |
| [Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) | 26 shared · 11% |
| [Low-resource languages](https://md.research.elamur.ai/free/en/topics/low-resource-languages/) | 22 shared · 9% |
| [Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) | 21 shared · 9% |
| [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) | 18 shared · 8% |
| [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) | 17 shared · 7% |
| [Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) | 16 shared · 7% |
| [Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) | 10 shared · 4% |
| [Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) | 9 shared · 4% |

Speech connects to many other parts of the library. The largest overlap is with [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) and [Hebrew & Aramaic](https://md.research.elamur.ai/free/en/topics/hebrew-aramaic/), because so much of the recent work concerns Arabic, Hebrew, Amharic, Maltese and Neo-Aramaic speech, and because prosody is an old tool of Hebrew grammar: [At the interface of syntax and prosody: Differentiating left dislocated and tripartite verbless clauses in Biblical Hebrew](https://md.research.elamur.ai/free/en/papers/naude-2018-interface-syntax-prosody-differentiating-left/) and [Towards a prosodic model for Tiberian Hebrew: An intonation-based analysis](https://md.research.elamur.ai/free/en/papers/pitcher-2021-towards-prosodic-model-tiberian-hebrew/) use the Masoretic accents as evidence of intonation.

[Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) and [Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) share the speech translation work and the multilingual models. [Low-resource languages](https://md.research.elamur.ai/free/en/topics/low-resource-languages/) shares the corpus-building papers. [Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) shares the shared-task reports and leaderboards. [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) and [Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) share the audio language models and audio-visual assistants such as [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://md.research.elamur.ai/free/en/papers/zhang-2023-video-llama-instruction-tuned-audio/). [Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) shares spoken language understanding and state tracking, and [Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) shares speech emotion recognition. Smaller overlaps link to [Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/), [Dependency parsing](https://md.research.elamur.ai/free/en/topics/dependency-parsing/), [Treebanks](https://md.research.elamur.ai/free/en/topics/treebanks/) and [Universal Dependencies](https://md.research.elamur.ai/free/en/topics/universal-dependencies/) (parsing transcribed speech), [Morphology](https://md.research.elamur.ai/free/en/topics/morphology/), [Tokenization & segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/) and [Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/).

The shelves this topic stands on Language and Linguistics: 226 of the 233 papers here.

**The numbers**

| Shelf | Papers |
| --- | --- |
| [Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) | 226 papers |
| [Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) | 222 papers |
| [Signal Processing](https://md.research.elamur.ai/free/en/shelves/computer-science/signal-processing/) | 135 papers |
| [Experimental and Cognitive Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/experimental-and-cognitive-psychology/) | 50 papers |
| [Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) | 17 papers |
| [Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) | 3 papers |
| [Cognitive Neuroscience](https://md.research.elamur.ai/free/en/shelves/neuroscience/cognitive-neuroscience/) | 2 papers |
| [Computational Theory and Mathematics](https://md.research.elamur.ai/free/en/shelves/computer-science/computational-theory-and-mathematics/) | 2 papers |
| [Human-Computer Interaction](https://md.research.elamur.ai/free/en/shelves/computer-science/human-computer-interaction/) | 2 papers |
| [Literature and Literary Theory](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/literature-and-literary-theory/) | 2 papers |
| [Physiology](https://md.research.elamur.ai/free/en/shelves/medicine/physiology/) | 2 papers |
| [Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) | 2 papers |

Most papers on this topic stand on the [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) shelf, with substantial parts on the [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) and [Natural language processing](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) shelves and a few on [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/).

## Key works in this library

- [WaveNet: A Generative Model for Raw Audio](https://md.research.elamur.ai/free/en/papers/oord-2016-wavenet-generative-model-raw-audio/) — autoregressive waveform generation that transformed synthesis.
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://md.research.elamur.ai/free/en/papers/park-2019-specaugment-simple-data-augmentation-method/) — spectrogram masking as a cheap, universal regulariser.
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://md.research.elamur.ai/free/en/papers/gulati-2020-conformer-convolution-augmented-transformer-speech/) — the standard modern recognition encoder.
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://md.research.elamur.ai/free/en/papers/radford-2022-robust-speech-recognition-large-scale/) — Whisper and the case for weak supervision at scale.
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://md.research.elamur.ai/free/en/papers/amodei-2015-deep-speech-2-end-end/) — end-to-end CTC recognition engineered for scale and deployment.
- [Attention-Based Models for Speech Recognition](https://md.research.elamur.ai/free/en/papers/chorowski-2015-attention-based-models-speech-recognition/) — location-aware attention for long utterances.
- [Finite-State Transducers in Language and Speech Processing](https://md.research.elamur.ai/free/en/papers/mohri-1997-finite-state-transducers-language-speech/) — the algebra of weighted transducers behind WFST decoders.
- [Common Voice: A Massively-Multilingual Speech Corpus](https://md.research.elamur.ai/free/en/papers/ardila-2019-common-voice-massively-multilingual-speech/) — crowdsourced, public-domain speech for many languages.
- [SoundStream: An End-to-End Neural Audio Codec](https://md.research.elamur.ai/free/en/papers/zeghidour-2021-soundstream-end-end-neural-audio/) — learned codec with residual vector quantization.
- [AudioLM: a Language Modeling Approach to Audio Generation](https://md.research.elamur.ai/free/en/papers/borsos-2022-audiolm-language-modeling-approach-audio/) — language modelling over semantic and acoustic tokens.
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://md.research.elamur.ai/free/en/papers/wang-2023-neural-codec-language-models-zero/) — VALL-E and voice cloning from a short prompt.
- [The Fisher Corpus: a Resource for the Next Generations of Speech-to-Text](https://md.research.elamur.ai/free/en/papers/cieri-2004-fisher-corpus-resource-next-generations/) — large-scale collection of conversational telephone speech.
- [MuST-C: a Multilingual Speech Translation Corpus](https://md.research.elamur.ai/free/en/papers/gangi-2019-must-c-multilingual-speech-translation/) — the benchmark corpus for end-to-end speech translation.
- [Word-Based Dialog State Tracking with Recurrent Neural Networks](https://md.research.elamur.ai/free/en/papers/henderson-2014-word-based-dialog-state-tracking/) — dialogue state from recognition output without a separate parser.
- [A Stack-Propagation Framework with Token-Level Intent Detection for Spoken Language Understanding](https://md.research.elamur.ai/free/en/papers/qin-2019-stack-propagation-framework-token-level/) — joint intent detection and slot filling.
- [Gender and Dialect Bias in YouTube's Automatic Captions](https://md.research.elamur.ai/free/en/papers/tatman-2017-gender-dialect-bias-youtube-s/) — unequal recognition accuracy across speaker groups.
- [SeamlessM4T: Massively Multilingual & Multimodal Machine Translation](https://md.research.elamur.ai/free/en/papers/communication-2023-seamlessm4t-massively-multilingual-multimodal-machine/) — one model for speech and text translation across many languages.
- [Qwen2-Audio Technical Report](https://md.research.elamur.ai/free/en/papers/chu-2024-qwen2-audio-technical-report/) — an open audio-language model for voice chat and audio analysis.
- [Findings of the IWSLT 2022 Evaluation Campaign](https://md.research.elamur.ai/free/en/papers/anastasopoulos-2022-findings-iwslt-2022-evaluation-campaign/) — the state of spoken translation across eight shared tasks.
- [Speech Recognition Challenge in the Wild: Arabic MGB-3](https://md.research.elamur.ai/free/en/papers/ali-2017-speech-recognition-challenge-wild-arabic/) — dialectal Arabic recognition scored with several references.
- [Arabic Speech Recognition by End-to-End, Modular Systems and Human](https://md.research.elamur.ai/free/en/papers/hussein-2021-arabic-speech-recognition-end-end/) — Arabic recognizers measured against expert and lay transcribers.

## Papers (233)

**1** [2](https://md.research.elamur.ai/free/en/topics/speech/page/2/) [3](https://md.research.elamur.ai/free/en/topics/speech/page/3/) [4](https://md.research.elamur.ai/free/en/topics/speech/page/4/) [next →](https://md.research.elamur.ai/free/en/topics/speech/page/2/)

1. [**WaveNet: A Generative Model for Raw Audio arXiv:1609.03499**Aaron van den Oord, Sander Dieleman, Heiga Zen, et al.*2016 · 5,190 citations*](https://md.research.elamur.ai/free/en/papers/oord-2016-wavenet-generative-model-raw-audio/)
2. **[SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://md.research.elamur.ai/free/en/papers/park-2019-specaugment-simple-data-augmentation-method/) [Interspeech](https://md.research.elamur.ai/free/en/venues/interspeech-usb/)**Daniel S. Park, William Chan, Yu Zhang, et al.*2019 · 3,809 citations*
3. **[Conformer: Convolution-augmented Transformer for Speech Recognition](https://md.research.elamur.ai/free/en/papers/gulati-2020-conformer-convolution-augmented-transformer-speech/) [Interspeech](https://md.research.elamur.ai/free/en/venues/interspeech-usb/)**Anmol Gulati, James Qin, Chung-Cheng Chiu, et al.*2020 · 3,439 citations*
4. [**Robust Speech Recognition via Large-Scale Weak Supervision arXiv:2212.04356**Alec Radford, Jong Wook Kim, Tao Xu, et al.*2022 · 2,501 citations*](https://md.research.elamur.ai/free/en/papers/radford-2022-robust-speech-recognition-large-scale/)
5. [**Deep Speech 2: End-to-End Speech Recognition in English and Mandarin arXiv:1512.02595**Dario Amodei, Rishita Anubhai, Eric Battenberg, et al.*2015 · 2,314 citations*](https://md.research.elamur.ai/free/en/papers/amodei-2015-deep-speech-2-end-end/)
6. [**Attention-Based Models for Speech Recognition arXiv:1506.07503**Jan Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, et al.*2015 · 1,787 citations*](https://md.research.elamur.ai/free/en/papers/chorowski-2015-attention-based-models-speech-recognition/)
7. [**Deep Speech: Scaling up end-to-end speech recognition arXiv:1412.5567**Awni Hannun, Carl Case, Jared Casper, et al.*2014 · 1,585 citations*](https://md.research.elamur.ai/free/en/papers/hannun-2014-deep-speech-scaling-up-end/)
8. **[Finite-State Transducers in Language and Speech Processing](https://md.research.elamur.ai/free/en/papers/mohri-1997-finite-state-transducers-language-speech/) [Computational Linguistics](https://md.research.elamur.ai/free/en/venues/cl/)**Mehryar Mohri*1997 · 1,023 citations*
9. [**Common Voice: A Massively-Multilingual Speech Corpus arXiv:1912.06670**Rosana Ardila, Megan Branson, Kelly Davis, et al.*2019 · 736 citations*](https://md.research.elamur.ai/free/en/papers/ardila-2019-common-voice-massively-multilingual-speech/)
10. **[SoundStream: An End-to-End Neural Audio Codec](https://md.research.elamur.ai/free/en/papers/zeghidour-2021-soundstream-end-end-neural-audio/) [IEEE/ACM Transactions on Audio Speech and Language Processing](https://md.research.elamur.ai/free/en/venues/ieee-acm-transactions-on-audio-speech-and-language-processing/)**Neil Zeghidour, Alejandro Luebs, Ahmed Omran, et al.*2021 · 727 citations*
11. [**Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces arXiv:1805.10190**Alice Coucke, Alaa Saade, Adrien Ball, et al.*2018 · 595 citations*](https://md.research.elamur.ai/free/en/papers/coucke-2018-snips-voice-platform-embedded-spoken/)
12. **[AudioLM: a Language Modeling Approach to Audio Generation](https://md.research.elamur.ai/free/en/papers/borsos-2022-audiolm-language-modeling-approach-audio/) [IEEE/ACM Transactions on Audio Speech and Language Processing](https://md.research.elamur.ai/free/en/venues/ieee-acm-transactions-on-audio-speech-and-language-processing/)**Zalan Borsos, Raphael Marinier, Damien Vincent, et al.*2022 · 570 citations*
13. **[Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis](https://md.research.elamur.ai/free/en/papers/han-2021-improving-multimodal-fusion-hierarchical-mutual/) [Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/)**Wei Han, Hui Chen, Soujanya Poria*2021 · 564 citations*
14. **[The Fisher Corpus: a Resource for the Next Generations of Speech-to-Text](https://md.research.elamur.ai/free/en/papers/cieri-2004-fisher-corpus-resource-next-generations/) [Language Resources and Evaluation Conference (LREC)](https://md.research.elamur.ai/free/en/venues/lrec/)**Christopher Cieri, David Miller, Kevin Walker*2004 · 560 citations*
15. **[Edinburgh System Description for the 2005 IWSLT Speech Translation Evaluation](https://md.research.elamur.ai/free/en/papers/koehn-2005-edinburgh-system-description-2005-iwslt/) [International Conference on Spoken Language Translation (IWSLT)](https://md.research.elamur.ai/free/en/venues/iwslt/)**Philipp Koehn, Amittai Axelrod, Alexandra Birch Mayne, et al.*2005 · 545 citations*
16. **[VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://md.research.elamur.ai/free/en/papers/wang-2021-voxpopuli-large-scale-multilingual-speech/) [Annual Meeting of the Association for Computational Linguistics (ACL)](https://md.research.elamur.ai/free/en/venues/acl/)**Changhan Wang, Morgane Riviere, Ann Lee, et al.*2021 · 524 citations*
17. **[MLS: A Large-Scale Multilingual Dataset for Speech Research](https://md.research.elamur.ai/free/en/papers/pratap-2020-mls-large-scale-multilingual-dataset/) [Interspeech](https://md.research.elamur.ai/free/en/venues/interspeech-usb/)**Vineel Pratap, Qiantong Xu, Anuroop Sriram, et al.*2020 · 512 citations*
18. **[MuST-C: a Multilingual Speech Translation Corpus](https://md.research.elamur.ai/free/en/papers/gangi-2019-must-c-multilingual-speech-translation/) [Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)](https://md.research.elamur.ai/free/en/venues/naacl/)**Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, et al.*2019 · 436 citations*
19. **[Word-Based Dialog State Tracking with Recurrent Neural Networks](https://md.research.elamur.ai/free/en/papers/henderson-2014-word-based-dialog-state-tracking/) [Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL)](https://md.research.elamur.ai/free/en/venues/sigdial/)**Matthew Henderson, Blaise Thomson, Steve Young*2014 · 432 citations*
20. **[Gender and Dialect Bias in YouTube's Automatic Captions](https://md.research.elamur.ai/free/en/papers/tatman-2017-gender-dialect-bias-youtube-s/) [ACL Workshop on Ethics in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/acl-workshop-on-ethics-in-natural-language-processing/)**Rachael Tatman*2017 · 414 citations*
21. [**High Fidelity Neural Audio Compression arXiv:2210.13438**Alexandre Defossez, Jade Copet, Gabriel Synnaeve, et al.*2022 · 412 citations*](https://md.research.elamur.ai/free/en/papers/defossez-2022-high-fidelity-neural-audio-compression/)
22. **[A Stack-Propagation Framework with Token-Level Intent Detection for Spoken Language Understanding](https://md.research.elamur.ai/free/en/papers/qin-2019-stack-propagation-framework-token-level/) [Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/)**Libo Qin, Wanxiang Che, Yangming Li, et al.*2019 · 385 citations*
23. **[English Conversational Telephone Speech Recognition by Humans and Machines](https://md.research.elamur.ai/free/en/papers/saon-2017-english-conversational-telephone-speech-recognition/) [Interspeech](https://md.research.elamur.ai/free/en/venues/interspeech-usb/)**George Saon, Gakuto Kurata, Tom Sercu, et al.*2017 · 382 citations*
24. **[SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities](https://md.research.elamur.ai/free/en/papers/zhang-2023-speechgpt-empowering-large-language-models/) [Findings of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/findings/)**Dong Zhang, Shimin Li, Xin Zhang, et al.*2023 · 346 citations*
25. [**Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers arXiv:2301.02111**Chengyi Wang, Sanyuan Chen, Yu Wu, et al.*2023 · 333 citations*](https://md.research.elamur.ai/free/en/papers/wang-2023-neural-codec-language-models-zero/)

**1** [2](https://md.research.elamur.ai/free/en/topics/speech/page/2/) [3](https://md.research.elamur.ai/free/en/topics/speech/page/3/) [4](https://md.research.elamur.ai/free/en/topics/speech/page/4/) [next →](https://md.research.elamur.ai/free/en/topics/speech/page/2/)

Updated 24 October 2026

- [← previous topic Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/)
- [next topic → Speech act theory](https://md.research.elamur.ai/free/en/topics/speech-act-theory/)
- [↑ back to topics](https://md.research.elamur.ai/free/en/topics/)