---
title: "Direct Acoustics-to-Word Models for English Conversational"
description: "The authors train CTC models whose output units are whole words. 26 references, cited by 10 papers here."
authors: ["Kartik Audhkhasi", "Bhuvana Ramabhadran", "George Saon", "Michael Picheny", "David Nahamoo"]
year: 2017
venue: "arXiv:1703.07754"
source: "https://arxiv.org/abs/1703.07754"
doi: "10.48550/arxiv.1703.07754"
arxiv: "1703.07754"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/audhkhasi-2017-direct-acoustics-word-models-english/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/audhkhasi-2017-direct-acoustics-word-models-english/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/page/3/) / Direct Acoustics-to-Word Models for English Conversational Speech Recognition

# Direct Acoustics-to-Word Models for English Conversational Speech Recognition

**Authors:** [Kartik Audhkhasi](https://md.research.elamur.ai/free/en/authors/kartik-audhkhasi/), [Bhuvana Ramabhadran](https://md.research.elamur.ai/free/en/authors/bhuvana-ramabhadran/), [George Saon](https://md.research.elamur.ai/free/en/authors/george-saon/), [Michael Picheny](https://md.research.elamur.ai/free/en/authors/michael-picheny/) and David Nahamoo · **Year:** 2017 · **Venue:** [arXiv:1703.07754](https://md.research.elamur.ai/free/en/venues/interspeech-usb/) ([Interspeech 2017](https://md.research.elamur.ai/free/en/venues/interspeech-usb/2017/)) · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1703.07754) · [PDF](https://arxiv.org/pdf/1703.07754) · [DOI 10.48550/arxiv.1703.07754](https://doi.org/10.48550/arxiv.1703.07754) · [arXiv 1703.07754](https://arxiv.org/abs/1703.07754) · [OpenAlex](https://openalex.org/W2597757402)

**Citations:** 111 citations (OpenAlex, 2026-10-14) · [10](https://md.research.elamur.ai/free/en/papers/audhkhasi-2017-direct-acoustics-word-models-english/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper asks whether one neural network can turn speech directly into words, with no phone inventory, pronunciation dictionary, language model or decoder. It also asks whether this works on standard public benchmarks with moderate data, not only with very large private collections. The authors train CTC models whose output units are whole words. They use 300 hours of Switchboard and about 2000 hours of Switchboard+Fisher, with a 5-layer BLSTM over logMel and i-vector features. From random weights, the word model failed to converge. It became trainable once the BLSTM was initialised from a phone CTC model, and GloVe embeddings used to set the final layer helped further. A hierarchical multi-task CTC variant beat random initialisation but lost to plain phone-model initialisation. With no LM or decoder, the 2000-hour word model reaches 13.0%/18.8% WER on Hub5-2000 Switchboard/CallHome. A phone CTC system with a 4-gram LM scores 9.6%/16.0% on the same sets. The result shows that word-level end-to-end recognition is feasible on public data, and decoding becomes a simple per-frame pick followed by collapsing.

## Key points
- Word vocabulary keeps only words with at least 5 acoustic examples: about 10k words for 300 h and about 25k for 2000 h. Training OOV rates are 1.5% and 0.5%; one shared OOV token is used and mapped to silence at scoring time.
- The phone CTC baseline uses 44 phones plus blank. The 2000-hour version reaches 10.2/16.5 (small LM) and 9.6/16.0 (big LM). A hybrid BLSTM gets 7.7/14.0, but phone CTC decodes about 2x faster with 8x less memory.
- The input pipeline stacks two logMel frames and drops every other frame, halving sequence length at no accuracy cost. 100-dim i-vectors are appended, giving 180-dim inputs. Training uses Torch, cuDNN and Warp-CTC.
- Random initialisation of the word CTC model did not converge despite tuning. Copying the whole phone CTC BLSTM fixed this, and it beat hierarchical CTC at any tested phone/word loss weight α.
- The word CTC system makes no more errors than phone CTC on 72.1%/74.9% of test utterances. The paper also reports LM lattice rescoring of the word model.

## Interesting ideas
- Train a model on fine-grained, easy units first, then reuse its whole encoder for a coarse task with a huge label set that cannot be learned from scratch.
- Seed the rows of a very large output layer with pretrained embeddings, so rare labels inherit co-occurrence knowledge instead of starting as noise.
- Cap the label inventory by a minimum-frequency threshold, collapse everything else into one explicit OOV class, and report the resulting OOV rate.
- Compare two systems per item, by the share of utterances where one makes no more errors than the other, rather than only by an aggregate error rate.
- Test whether multi-task auxiliary losses actually beat simple initialisation; here a joint phone+word loss lost to plain weight copying.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tree): Initialise gparse's encoder from a tagger/morphology model trained on the same treebanks, so dependency training starts from features that already detect word-level structure.
- **I2** (tokens): A lemma or fine-tag classifier in crate en could have its output vectors initialised from embeddings, helping rare lemmas with few training examples.
- **I3** (world): When compiling verb classes or ontology labels from WordNet/FrameNet/Wikidata, an explicit "unknown" bucket with a measured coverage rate keeps the knowledge layer honest about what it cannot name.
- **I4** (learn): When comparing the perceptron parser and gparse, or rules before and after self-induced repair, a per-sentence "no worse than" share shows where a change helps or hurts.
- **I5** (learn): Before adding joint objectives to teacher-tier training, run the plain warm-start baseline as a control, since it may already be better.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [IBM Research - Thomas J. Watson Research Center](https://md.research.elamur.ai/free/en/labs/ibm-research-thomas-j-watson-research-center/) — Research institute, Yorktown Heights, United States · 41 works · 46,197 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🔊 Signal Processing](https://md.research.elamur.ai/free/en/shelves/computer-science/signal-processing/) · 213 works

🧭 Topics (3): [English](https://md.research.elamur.ai/free/en/topics/english/) · 1,373 works · [🗨️ Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) · 401 works · [🎙️ Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) · 233 works

## Cite

Cite as: Kartik Audhkhasi, Bhuvana Ramabhadran, George Saon, Michael Picheny and David Nahamoo. 2017. Direct Acoustics-to-Word Models for English Conversational Speech Recognition. arXiv:1703.07754.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{audhkhasi2017direct,
  title = {{Direct Acoustics-to-Word Models for English Conversational Speech Recognition}},
  author = {Kartik Audhkhasi, Bhuvana Ramabhadran, George Saon, Michael Picheny and David Nahamoo},
  year = {2017},
  doi = {10.48550/arxiv.1703.07754},
  eprint = {1703.07754},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1703.07754}
}
```

- [← previous FOIL it! Find One mismatch between Image and Language caption](https://md.research.elamur.ai/free/en/papers/shekhar-2017-foil-it-find-one-mismatch/)
- [next → An Incremental Parser for Abstract Meaning Representation](https://md.research.elamur.ai/free/en/papers/damonte-2017-incremental-parser-abstract-meaning-representation/)
- [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/page/3/)

## References

26 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. G. Hinton, L. Deng, D. Yu, G. E. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, and B. Kingsbury, “Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups,” IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 82–97, 2012. [↗](https://doi.org/10.1109/msp.2012.2205597)
2. G. Saon, G. Kurata, T. Sercu, K. Audhkhasi, S. Thomas, D. Dimitriadis, X. Cui, B. Ramabhadran, M. Picheny, L. Lim, B. Roomi, and P. Hall, “[English conversational telephone speech recognition by humans and machines](https://md.research.elamur.ai/free/en/papers/saon-2017-english-conversational-telephone-speech-recognition/),” arXiv preprint arXiv:1703.02136, 2017. [↗](https://arxiv.org/abs/1703.02136)
3. W. Xiong, J. Droppo, X. Huang, F. Seide, M. Seltzer, A. Stolcke, D. Yu, and G. Zweig, “The Microsoft 2016 conversational speech recognition system,” in Proc. ICASSP, 2017, pp. 5255–5259. [↗](https://doi.org/10.48550/arxiv.1609.03528)
4. T. Mikolov, A. Deoras, S. Kombrink, L. Burget, and J. Černockỳ, “Empirical evaluation and combination of advanced language modeling techniques,” in Proc. Interspeech, 2011. [find](https://scholar.google.com/scholar?q=T%2E+Mikolov%2C+A%2E+Deoras%2C+S%2E+Kombrink%2C+L%2E+Burget%2C+and+J%2E+C%CC%8Cernocky%CC%80%2C+%E2%80%9CEmpirical+evaluation+and+combination+of+advanced+language+modeling+techniques%2C%E2%80%9D+in+Proc%2E+Interspeech%2C+2011%2E)
5. C. Chelba, T. Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, and T. Robinson, “[One billion word benchmark for measuring progress in statistical language modeling](https://md.research.elamur.ai/free/en/papers/chelba-2013-one-billion-word-benchmark-measuring/),” in Proc. Interspeech, 2014.
6. S. F. Chen, “Shrinking exponential language models,” in Proc. HLT-NAACL, 2009, pp. 468–476. [find](https://scholar.google.com/scholar?q=S%2E+F%2E+Chen%2C+%E2%80%9CShrinking+exponential+language+models%2C%E2%80%9D+in+Proc%2E+HLT%2DNAACL%2C+2009%2C+pp%2E+468%E2%80%93476%2E)
7. F. Jelinek, Statistical methods for speech recognition. MIT press, 1997. [find](https://scholar.google.com/scholar?q=F%2E+Jelinek%2C+Statistical+methods+for+speech+recognition%2E+MIT+press%2C+1997%2E)
8. A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in Proc. ICML, 2006, pp. 369–376. [find](https://scholar.google.com/scholar?q=A%2E+Graves%2C+S%2E+Ferna%CC%81ndez%2C+F%2E+Gomez%2C+and+J%2E+Schmidhuber%2C+%E2%80%9CConnectionist+temporal+classification%3A+labelling+unsegmented+sequence+data+with+recurrent+neural+networks%2C%E2%80%9D+in+Proc%2E+ICML%2C+2006%2C+pp%2E+369%E2%80%93376%2E)
9. D. Bahdanau, K. Cho, and Y. Bengio, “[Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/),” arXiv preprint arXiv:1409.0473, 2014. [↗](https://arxiv.org/abs/1409.0473)
10. D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio, “End-to-end attention-based large vocabulary speech recognition,” in Proc. ICASSP, 2016, pp. 4945–4949. [find](https://scholar.google.com/scholar?q=D%2E+Bahdanau%2C+J%2E+Chorowski%2C+D%2E+Serdyuk%2C+P%2E+Brakel%2C+and+Y%2E+Bengio%2C+%E2%80%9CEnd%2Dto%2Dend+attention%2Dbased+large+vocabulary+speech+recognition%2C%E2%80%9D+in+Proc%2E+ICASSP%2C+2016%2C+pp%2E+4945%E2%80%934949%2E)
11. A. L. Maas, Z. Xie, D. Jurafsky, and A. Y. Ng, “Lexicon-free conversational speech recognition with neural networks,” in Proc. HLT-NAACL, 2015, pp. 345–354. [find](https://scholar.google.com/scholar?q=A%2E+L%2E+Maas%2C+Z%2E+Xie%2C+D%2E+Jurafsky%2C+and+A%2E+Y%2E+Ng%2C+%E2%80%9CLexicon%2Dfree+conversational+speech+recognition+with+neural+networks%2C%E2%80%9D+in+Proc%2E+HLT%2DNAACL%2C+2015%2C+pp%2E+345%E2%80%93354%2E)
12. A. Graves and N. Jaitly, “Towards end-to-end speech recognition with recurrent neural networks.” in ICML, vol. 14, 2014, pp. 1764–1772. [find](https://scholar.google.com/scholar?q=A%2E+Graves+and+N%2E+Jaitly%2C+%E2%80%9CTowards+end%2Dto%2Dend+speech+recognition+with+recurrent+neural+networks%2E%E2%80%9D+in+ICML%2C+vol%2E+14%2C+2014%2C+pp%2E+1764%E2%80%931772%2E)
13. A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh, S. Sengupta, A. Coates, and A. Y. Ng, “[Deep speech: Scaling up end-to-end speech recognition](https://md.research.elamur.ai/free/en/papers/hannun-2014-deep-speech-scaling-up-end/),” arXiv preprint arXiv:1412.5567, 2014. [↗](https://arxiv.org/abs/1412.5567)
14. Y. Miao, M. Gowayyed, and F. Metze, “EESEN: End-to-end speech recognition using deep RNN models and WFST-based decoding,” in Proc. ASRU, 2015, pp. 167–174. [find](https://scholar.google.com/scholar?q=Y%2E+Miao%2C+M%2E+Gowayyed%2C+and+F%2E+Metze%2C+%E2%80%9CEESEN%3A+End%2Dto%2Dend+speech+recognition+using+deep+RNN+models+and+WFST%2Dbased+decoding%2C%E2%80%9D+in+Proc%2E+ASRU%2C+2015%2C+pp%2E+167%E2%80%93174%2E)
15. Y. Miao, M. Gowayyed, X. Na, T. Ko, F. Metze, and A. Waibel, “An empirical exploration of CTC acoustic models,” in Proc. ICASSP, 2016, pp. 2623–2627. [↗](https://doi.org/10.5445/ir/1000166118)
16. H. Sak, A. Senior, K. Rao, and F. Beaufays, “Fast and accurate recurrent neural network acoustic models for speech recognition,” in Proc. Interspeech, 2015. [find](https://scholar.google.com/scholar?q=H%2E+Sak%2C+A%2E+Senior%2C+K%2E+Rao%2C+and+F%2E+Beaufays%2C+%E2%80%9CFast+and+accurate+recurrent+neural+network+acoustic+models+for+speech+recognition%2C%E2%80%9D+in+Proc%2E+Interspeech%2C+2015%2E)
17. G. Zweig, C. Yu, J. Droppo, and A. Stolcke, “Advances in all-neural speech recognition,” in Proc. ICASSP, 2017, pp. 4805– 4809. [find](https://scholar.google.com/scholar?q=G%2E+Zweig%2C+C%2E+Yu%2C+J%2E+Droppo%2C+and+A%2E+Stolcke%2C+%E2%80%9CAdvances+in+all%2Dneural+speech+recognition%2C%E2%80%9D+in+Proc%2E+ICASSP%2C+2017%2C+pp%2E+4805%E2%80%93+4809%2E)
18. H. Soltau, H. Liao, and H. Sak, “Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognition,” arXiv preprint arXiv:1610.09975, 2016. [↗](https://arxiv.org/abs/1610.09975)
19. S. Fernandez, A. Graves, and J. Schmidhuber, “Sequence labelling in structured domains with hierarchical recurrent neural networks,” in Proc. IJCAI, 2007, pp. 774–779. [find](https://scholar.google.com/scholar?q=S%2E+Fernandez%2C+A%2E+Graves%2C+and+J%2E+Schmidhuber%2C+%E2%80%9CSequence+labelling+in+structured+domains+with+hierarchical+recurrent+neural+networks%2C%E2%80%9D+in+Proc%2E+IJCAI%2C+2007%2C+pp%2E+774%E2%80%93779%2E)
20. J. Pennington, R. Socher, and C. D. Manning, “[Glove: Global Vectors for Word Representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/),” in Proc. EMNLP, vol. 14, 2014, pp. 1532–1543.
21. N. Dehak, P. J. Kenny, R. Dehak, P. Dumouchel, and P. Ouellet, “Front-end factor analysis for speaker verification,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 19, no. 4, pp. 788–798, 2011. [↗](https://doi.org/10.1109/tasl.2010.2064307)
22. R. Collobert, K. Kavukcuoglu, and C. Farabet, “Torch7: A Matlab-like Environment for Machine Learning,” in Proc. BigLearn, NIPS Workshop, 2011. [find](https://scholar.google.com/scholar?q=R%2E+Collobert%2C+K%2E+Kavukcuoglu%2C+and+C%2E+Farabet%2C+%E2%80%9CTorch7%3A+A+Matlab%2Dlike+Environment+for+Machine+Learning%2C%E2%80%9D+in+Proc%2E+BigLearn%2C+NIPS+Workshop%2C+2011%2E)
23. S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, B. Catanzaro, and E. Shelhamer, “CUDNN: Efficient primitives for deep learning,” arXiv preprint arXiv:1410.0759, 2014. [↗](https://arxiv.org/abs/1410.0759)
24. K. Rao and H. Sak, “Multi-accent speech recognition with hierarchical grapheme based models,” in Proc. ICASSP, 2017, pp. 4815–4819. [find](https://scholar.google.com/scholar?q=K%2E+Rao+and+H%2E+Sak%2C+%E2%80%9CMulti%2Daccent+speech+recognition+with+hierarchical+grapheme+based+models%2C%E2%80%9D+in+Proc%2E+ICASSP%2C+2017%2C+pp%2E+4815%E2%80%934819%2E)
25. K. Audhkhasi, A. Sethy, and B. Ramabhadran, “Semantic word embedding neural network language models for automatic speech recognition,” in Proc. ICASSP, 2016, pp. 5995–5999. [find](https://scholar.google.com/scholar?q=K%2E+Audhkhasi%2C+A%2E+Sethy%2C+and+B%2E+Ramabhadran%2C+%E2%80%9CSemantic+word+embedding+neural+network+language+models+for+automatic+speech+recognition%2C%E2%80%9D+in+Proc%2E+ICASSP%2C+2016%2C+pp%2E+5995%E2%80%935999%2E)
26. T. Mikolov, K. Chen, G. Corrado, and J. Dean, “[Efficient estimation of word representations in vector space](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/),” arXiv preprint arXiv:1301.3781, 2013. [↗](https://arxiv.org/abs/1301.3781)

*Updated 24 October 2026*