---
title: "The IBM 2016 English Conversational Telephone Speech"
description: "The authors chose a classical hybrid design over end-to-end… 6.6% WER on Hub5'00 Switchboard. 27 references, cited by 3 papers here."
authors: ["George Saon", "Tom Sercu", "Steven Rennie", "Hong-Kwang J. Kuo"]
year: 2016
venue: "arXiv:1604.08242"
source: "https://arxiv.org/abs/1604.08242"
doi: "10.48550/arxiv.1604.08242"
arxiv: "1604.08242"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/saon-2016-ibm-2016-english-conversational-telephone/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/saon-2016-ibm-2016-english-conversational-telephone/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/4/) / The IBM 2016 English Conversational Telephone Speech Recognition System

# The IBM 2016 English Conversational Telephone Speech Recognition System

**Authors:** [George Saon](https://md.research.elamur.ai/free/en/authors/george-saon/), [Tom Sercu](https://md.research.elamur.ai/free/en/authors/tom-sercu/), [Steven Rennie](https://md.research.elamur.ai/free/en/authors/steven-rennie/) and [Hong-Kwang J. Kuo](https://md.research.elamur.ai/free/en/authors/hong-kwang-j-kuo/) · **Year:** 2016 · **Venue:** arXiv:1604.08242 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1604.08242) · [PDF](https://arxiv.org/pdf/1604.08242) · [DOI 10.48550/arxiv.1604.08242](https://doi.org/10.48550/arxiv.1604.08242) · [arXiv 1604.08242](https://arxiv.org/abs/1604.08242) · [OpenAlex](https://openalex.org/W4300457772)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [3](https://md.research.elamur.ai/free/en/papers/saon-2016-ibm-2016-english-conversational-telephone/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper reports how IBM pushed English conversational telephone speech recognition to a record word error rate of 6.6% on the Switchboard portion of the Hub5 2000 evaluation. The authors chose a classical hybrid design over end-to-end systems: neural networks produce scores for context-dependent HMM states, and these scores are decoded with HMMs and decision trees. They argue that end-to-end systems still trail this approach in accuracy. All acoustic models were trained on 2000 hours of conversational speech. Three strong model families were improved and then fused at the score level:
- unfolded RNNs with maxout units and annealed dropout,
- VGG-style CNNs with ten small 3×3 convolutional layers,
- bidirectional LSTMs on FMLLR and i-vector inputs.

On the language side, the vocabulary was expanded from 30K to 85K words, and the authors also revised model "M" and added neural-net LMs with a hierarchical structure. The result matters because it narrows the gap to the estimated human level of about 4%. The authors also suggest that further gains will need sequence-to-sequence breakthroughs and more training data.

## Key points
- Final system: 6.6% WER on Hub5'00 Switchboard, using fusion of the RNN, VGG CNN and LSTM scores. Fusion cut WER by 0.4–0.7% compared with the LSTM alone.
- Maxout RNN: one recurrent layer unfolded over 6 frames, four further layers, and 32,000 output HMM states. With sequence training it reaches 9.3 / 15.4 (SWB / CH). The sigmoid version gets 10.8 / 16.9 after cross-entropy training.
- The 10-layer VGG CNN with pooling, built in Torch, reaches 9.4 WER on SWB after sequence training on 2000h. It uses balanced class sampling (γ = 0.8) and keeps cross-entropy as a regularizer during sequence training.
- The 4-layer bidirectional LSTM (512 units, state reset) was the best single model at 9.0 / 15.1 after sequence training. The 6-layer DNN scored 10.3 / 17.0.
- Carrying LSTM start states across subsequences gave lower held-out loss but worse WER than resetting them. The likely cause is a clash with the HMM and LM constraints.

## Interesting ideas
- A better intermediate loss can still produce a worse final result: the carried-over LSTM predicted states better yet recognized words worse, so judge models by the end metric.
- Fusing scores from architecturally different models pays off because their errors differ.
- Comparisons between variants were kept fair by sizing layers so that parameter counts roughly matched the baseline.
- The baseline improved simply by realigning the training data with the strongest model and retraining.
- Sampling training classes with a flattened frequency distribution (exponent below 1) helps rare targets get learned.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Gate every parser or tagger change on downstream frame and answer accuracy, not just on attachment or loss scores.
- **I2** (tree): Combine arc scores from the perceptron parser and gparse before decoding, rather than voting on finished trees.
- **I3** (learn): When testing a new parser feature set or architecture, match model size so that gains reflect design rather than capacity.
- **I4** (learn): Periodically regenerate silver data with the current best witness ensemble and retrain weaker parsers on it.
- **I5** (tokens): Apply tempered sampling to rare POS tags, morphological features or UD relations during tagger and parser training.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [IBM](https://md.research.elamur.ai/free/en/companies/ibm/) — Company, Armonk, United States · 85 works · 66,118 citations
- [IBM Research - Thomas J. Watson Research Center](https://md.research.elamur.ai/free/en/labs/ibm-research-thomas-j-watson-research-center/) — Research institute, Yorktown Heights, United States · 41 works · 46,197 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🔊 Signal Processing](https://md.research.elamur.ai/free/en/shelves/computer-science/signal-processing/) · 213 works

🧭 Topics (3): [English](https://md.research.elamur.ai/free/en/topics/english/) · 1,373 works · [🗨️ Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) · 401 works · [🎙️ Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) · 233 works

## Cite

Cite as: George Saon, Tom Sercu, Steven Rennie and Hong-Kwang J. Kuo. 2016. The IBM 2016 English Conversational Telephone Speech Recognition System. arXiv:1604.08242.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{saon2016ibm,
  title = {{The IBM 2016 English Conversational Telephone Speech Recognition System}},
  author = {George Saon, Tom Sercu, Steven Rennie and Hong-Kwang J. Kuo},
  year = {2016},
  doi = {10.48550/arxiv.1604.08242},
  eprint = {1604.08242},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1604.08242}
}
```

- [← previous SoNLP-DP System for ConLL-2016 English Shallow Discourse Parsing](https://md.research.elamur.ai/free/en/papers/kong-2016-sonlp-dp-system-conll-2016/)
- [next → Towards a QUD-Based Analysis of Gapping Constructions](https://md.research.elamur.ai/free/en/papers/park-2016-qud-based-analysis-gapping-constructions/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/4/)

## References

27 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. F. Seide, G. Li, X. Chien, and D. Yu, “Feature engineering in context-dependent deep neural networks for conversational speech transcription,” in Proc. ASRU, 2011. [find](https://scholar.google.com/scholar?q=F%2E+Seide%2C+G%2E+Li%2C+X%2E+Chien%2C+and+D%2E+Yu%2C+%E2%80%9CFeature+engineering+in+context%2Ddependent+deep+neural+networks+for+conversational+speech+transcription%2C%E2%80%9D+in+Proc%2E+ASRU%2C+2011%2E)
2. O. Abdel-Hamid, L. Deng, and D. Yu, “Exploring convolutional neural network structures and optimization techniques for speech recognition.” in INTERSPEECH, 2013, pp. 3366–3370. [find](https://scholar.google.com/scholar?q=O%2E+Abdel%2DHamid%2C+L%2E+Deng%2C+and+D%2E+Yu%2C+%E2%80%9CExploring+convolutional+neural+network+structures+and+optimization+techniques+for+speech+recognition%2E%E2%80%9D+in+INTERSPEECH%2C+2013%2C+pp%2E+3366%E2%80%933370%2E)
3. T. N. Sainath, A.-r. Mohamed, B. Kingsbury, and B. Ramabhadran, “Deep convolutional neural networks for lvcsr,” in Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on. IEEE, 2013, pp. 8614–8618. [find](https://scholar.google.com/scholar?q=T%2E+N%2E+Sainath%2C+A%2E%2Dr%2E+Mohamed%2C+B%2E+Kingsbury%2C+and+B%2E+Ramabhadran%2C+%E2%80%9CDeep+convolutional+neural+networks+for+lvcsr%2C%E2%80%9D+in+Acoustics%2C+Speech+and+Signal+Processing+%28ICASSP%29%2C+2013+IEEE+International+Conference+)
4. G. Saon, H. Soltau, A. Emami, and M. Picheny, “Unfolded recurrent neural networks for speech recognition,” in Fifteenth Annual Conference of the International Speech Communication Association, 2014. [find](https://scholar.google.com/scholar?q=G%2E+Saon%2C+H%2E+Soltau%2C+A%2E+Emami%2C+and+M%2E+Picheny%2C+%E2%80%9CUnfolded+recurrent+neural+networks+for+speech+recognition%2C%E2%80%9D+in+Fifteenth+Annual+Conference+of+the+International+Speech+Communication+Association%2C+2014%2E)
5. V. Peddinti, D. Povey, and S. Khudanpur, “A time delay neural network architecture for efficient modeling of long temporal contexts,” in Proceedings of INTERSPEECH, 2015. [find](https://scholar.google.com/scholar?q=V%2E+Peddinti%2C+D%2E+Povey%2C+and+S%2E+Khudanpur%2C+%E2%80%9CA+time+delay+neural+network+architecture+for+efficient+modeling+of+long+temporal+contexts%2C%E2%80%9D+in+Proceedings+of+INTERSPEECH%2C+2015%2E)
6. A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh, S. Sengupta, A. Coates et al., “Deepspeech: Scaling up end-to-end speech recognition,” arXiv preprint arXiv:1412.[5567, 2014](https://md.research.elamur.ai/free/en/papers/hannun-2014-deep-speech-scaling-up-end/). [↗](https://arxiv.org/abs/1412.5567)
7. H. Sak, A. Senior, K. Rao, O. Irsoy, A. Graves, F. Beaufays, and J. Schalkwyk, “Learning acoustic frame labeling for speech recognition with recurrent neural networks,” in Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on. IEEE, 2015, pp. 4280–4284. [find](https://scholar.google.com/scholar?q=H%2E+Sak%2C+A%2E+Senior%2C+K%2E+Rao%2C+O%2E+Irsoy%2C+A%2E+Graves%2C+F%2E+Beaufays%2C+and+J%2E+Schalkwyk%2C+%E2%80%9CLearning+acoustic+frame+labeling+for+speech+recognition+with+recurrent+neural+networks%2C%E2%80%9D+in+Acoustics%2C+Speech+and+Signal)
8. Y. Miao, M. Gowayyed, and F. Metze, “Eesen: End-to-end speech recognition using deep rnn models and wfst-based decoding,” arXiv preprint arXiv:1507.08240, 2015. [↗](https://arxiv.org/abs/1507.08240)
9. A.-r. Mohamed, F. Seide, D. Yu, J. Droppo, A. Stolcke, G. Zweig, and G. Penn, “Deep bi-directional recurrent networks over spectral windows,” in Automatic Speech Recognition and Understanding (ASRU), 2015 IEEE Workshop on. IEEE, 2015. [find](https://scholar.google.com/scholar?q=A%2E%2Dr%2E+Mohamed%2C+F%2E+Seide%2C+D%2E+Yu%2C+J%2E+Droppo%2C+A%2E+Stolcke%2C+G%2E+Zweig%2C+and+G%2E+Penn%2C+%E2%80%9CDeep+bi%2Ddirectional+recurrent+networks+over+spectral+windows%2C%E2%80%9D+in+Automatic+Speech+Recognition+and+Understanding+%28ASRU%29%2C+)
10. A. Graves, A.-r. Mohamed, and G. Hinton, “Speech recognition with deep recurrent neural networks,” in Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on. IEEE, 2013, pp. 6645–6649. [↗](https://doi.org/10.48550/arxiv.1303.5778)
11. G. Saon, H.-K. Kuo, S. Rennie, and M. Picheny, “The IBM 2015 English conversational speech recognition system,” in Sixteenth Annual Conference of the International Speech Communication Association, 2015. [find](https://scholar.google.com/scholar?q=G%2E+Saon%2C+H%2E%2DK%2E+Kuo%2C+S%2E+Rennie%2C+and+M%2E+Picheny%2C+%E2%80%9CThe+IBM+2015+English+conversational+speech+recognition+system%2C%E2%80%9D+in+Sixteenth+Annual+Conference+of+the+International+Speech+Communication+Association%2C+20)
12. I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio, “Maxout networks,” arXiv preprint arXiv:1302.4389, 2013. [↗](https://arxiv.org/abs/1302.4389)
13. B. Kingsbury, T. Sainath, and H. Soltau, “Scalable minimum Bayes risk training of deep neural network acoustic models using distributed Hessian-free optimization,” in Proc. Interspeech, 2012. [find](https://scholar.google.com/scholar?q=B%2E+Kingsbury%2C+T%2E+Sainath%2C+and+H%2E+Soltau%2C+%E2%80%9CScalable+minimum+Bayes+risk+training+of+deep+neural+network+acoustic+models+using+distributed+Hessian%2Dfree+optimization%2C%E2%80%9D+in+Proc%2E+Interspeech%2C+2012%2E)
14. T. Sercu, C. Puhrsch, B. Kingsbury, and Y. LeCun, “Very deep multilingual convolutional neural networks for lvcsr,” Proc. ICASSP, 2016. [↗](https://doi.org/10.48550/arxiv.1509.08967)
15. T. Sercu and V. Goel, “Advances in very deep convolutional neural networks for lvcsr,” arXiv, 2016. [find](https://scholar.google.com/scholar?q=T%2E+Sercu+and+V%2E+Goel%2C+%E2%80%9CAdvances+in+very+deep+convolutional+neural+networks+for+lvcsr%2C%E2%80%9D+arXiv%2C+2016%2E)
16. K. Simonyan and A. Zisserman, “[Very deep convolutional networks for large-scale image recognition](https://md.research.elamur.ai/free/en/papers/simonyan-2014-very-deep-convolutional-networks-large/),” CoRR arXiv:1409.1556, 2014. [↗](https://arxiv.org/abs/1409.1556)
17. H. Soltau, G. Saon, and T. N. Sainath, “Joint training of convolutional and non-convolutional neural networks,” to Proc. ICASSP, 2014. [find](https://scholar.google.com/scholar?q=H%2E+Soltau%2C+G%2E+Saon%2C+and+T%2E+N%2E+Sainath%2C+%E2%80%9CJoint+training+of+convolutional+and+non%2Dconvolutional+neural+networks%2C%E2%80%9D+to+Proc%2E+ICASSP%2C+2014%2E)
18. R. Collobert, K. Kavukcuoglu, and C. Farabet, “Torch7: A matlab-like environment for machine learning,” in BigLearn, NIPS Workshop, no. EPFL-CONF-192376, 2011. [find](https://scholar.google.com/scholar?q=R%2E+Collobert%2C+K%2E+Kavukcuoglu%2C+and+C%2E+Farabet%2C+%E2%80%9CTorch7%3A+A+matlab%2Dlike+environment+for+machine+learning%2C%E2%80%9D+in+BigLearn%2C+NIPS+Workshop%2C+no%2E+EPFL%2DCONF%2D192376%2C+2011%2E)
19. H. Su, G. Li, D. Yu, and F. Seide, “Error back propagation for sequence training of context-dependent deep networks for conversational speech transcription,” Proc. ICASSP, 2013. [find](https://scholar.google.com/scholar?q=H%2E+Su%2C+G%2E+Li%2C+D%2E+Yu%2C+and+F%2E+Seide%2C+%E2%80%9CError+back+propagation+for+sequence+training+of+context%2Ddependent+deep+networks+for+conversational+speech+transcription%2C%E2%80%9D+Proc%2E+ICASSP%2C+2013%2E)
20. S. F. Chen and J. Goodman, “An empirical study of smoothing techniques for language modeling,” Computer Speech & Language, vol. 13, no. 4, pp. 359–393, 1999. [↗](https://doi.org/10.1006/csla.1999.0128)
21. A. Stolcke, “Entropy-based pruning of backoff language models,” in Proc. DARPA Broadcast News Transcription and Understanding Workshop, 1998, pp. 270–274. [find](https://scholar.google.com/scholar?q=A%2E+Stolcke%2C+%E2%80%9CEntropy%2Dbased+pruning+of+backoff+language+models%2C%E2%80%9D+in+Proc%2E+DARPA+Broadcast+News+Transcription+and+Understanding+Workshop%2C+1998%2C+pp%2E+270%E2%80%93274%2E)
22. S. F. Chen, “Shrinking exponential language models,” in Proc. NAACL-HLT, 2009, pp. 468–476. [find](https://scholar.google.com/scholar?q=S%2E+F%2E+Chen%2C+%E2%80%9CShrinking+exponential+language+models%2C%E2%80%9D+in+Proc%2E+NAACL%2DHLT%2C+2009%2C+pp%2E+468%E2%80%93476%2E)
23. Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin, “[A neural probabilistic language model](https://md.research.elamur.ai/free/en/papers/bengio-2003-neural-probabilistic-language-model/),” Journal of Machine Learning Research, vol. 3, pp. 1137–1155, 2003.
24. A. Emami, “A neural syntactic language model,” Ph.D. dissertation, Johns Hopkins University, Baltimore, MD, USA, 2006. [↗](https://doi.org/10.1007/s10994-005-0916-y)
25. H. Schwenk, “Continuous space language models,” Computer Speech & Language, vol. 21, no. 3, pp. 492–518, 2007. [↗](https://doi.org/10.1016/j.csl.2006.09.003)
26. A. Emami and L. Mangu, “Empirical study of neural network language models for Arabic speech recognition,” in Proc. ASRU, 2007, pp. 147–152. [find](https://scholar.google.com/scholar?q=A%2E+Emami+and+L%2E+Mangu%2C+%E2%80%9CEmpirical+study+of+neural+network+language+models+for+Arabic+speech+recognition%2C%E2%80%9D+in+Proc%2E+ASRU%2C+2007%2C+pp%2E+147%E2%80%93152%2E)
27. H.-K. J. Kuo, E. Arısoy, A. Emami, and P. Vozila, “Large scale hierarchical neural network language models,” in Proc. Interspeech, 2012. [find](https://scholar.google.com/scholar?q=H%2E%2DK%2E+J%2E+Kuo%2C+E%2E+Ar%C4%B1soy%2C+A%2E+Emami%2C+and+P%2E+Vozila%2C+%E2%80%9CLarge+scale+hierarchical+neural+network+language+models%2C%E2%80%9D+in+Proc%2E+Interspeech%2C+2012%2E)

*Updated 24 October 2026*