---
title: "Sequence Transduction with Recurrent Neural — Graves 2012"
description: "Ordinary recurrent networks can map one sequence to another only when the positions of inputs and outputs are matched in advance."
authors: ["Alex Graves"]
year: 2012
venue: "arXiv:1211.3711"
source: "https://arxiv.org/abs/1211.3711"
doi: "10.48550/arxiv.1211.3711"
arxiv: "1211.3711"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/graves-2012-sequence-transduction-recurrent-neural-networks/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/graves-2012-sequence-transduction-recurrent-neural-networks/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2012](https://md.research.elamur.ai/free/en/papers/2012/) / Sequence Transduction with Recurrent Neural Networks

# Sequence Transduction with Recurrent Neural Networks

**Authors:** [Alex Graves](https://md.research.elamur.ai/free/en/authors/alex-graves/) · **Institutions:** [University of Toronto](https://md.research.elamur.ai/free/en/universities/university-of-toronto/) · **Year:** 2012 · **Venue:** arXiv:1211.3711 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1211.3711) · [PDF](https://arxiv.org/pdf/1211.3711) · [DOI 10.48550/arxiv.1211.3711](https://doi.org/10.48550/arxiv.1211.3711) · [arXiv 1211.3711](https://arxiv.org/abs/1211.3711) · [OpenAlex](https://openalex.org/W1828163288)

**Citations:** 1170 citations (OpenAlex, 2026-10-14) · [226](https://md.research.elamur.ai/free/en/papers/graves-2012-sequence-transduction-recurrent-neural-networks/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
Ordinary recurrent networks can map one sequence to another only when the positions of inputs and outputs are matched in advance. In tasks such as speech recognition, finding that match, and even the output length, is often the hardest part. Graves proposes the RNN Transducer, a fully neural, probabilistic model that can map any input sequence to any finite discrete output sequence. It combines two parts: a bidirectional LSTM transcription network that reads the input, and an LSTM prediction network that works like a language model over earlier outputs. Their outputs are merged at every pair of input and output positions. A special null symbol lets the model emit nothing at a given step, so all alignments form a lattice that a forward-backward procedure sums efficiently. Decoding uses beam search. Unlike CTC, the model captures dependencies between output labels and permits outputs longer than inputs. On TIMIT phoneme recognition it reaches 23.2% error, beating a CTC baseline and, the author reports, the best RNN result at that time. The design later became a basis for streaming speech recognition.

## Key points
- **Data:** The TIMIT core sets were used, with 3696 training and 192 test utterances. A 184-utterance validation set was taken out of the training data. The targets were the reduced inventory of 39 phonemes.
- **Inputs:** The input features were 26-dimensional, consisting of 12 MFCCs plus energy together with their deltas, computed on 25 ms windows every 10 ms.
- **Architecture:** The transcription network has two 128-unit LSTM layers, and the prediction network has one 128-unit LSTM layer. Each network has 40 outputs, which are the 39 phonemes plus null. The full transducer has 261,328 weights.
- **Results (test phoneme error / log-loss in bits per phoneme):**
  - Transducer: 23.2% / 1.0
  - CTC: 25.5% / 1.3
  - Standalone prediction network: 72.9% / 4.0
  - The state-of-the-art benchmark at the time was 20.5% (Dahl et al., 2010).
- **Why the gain is small:** The author attributes the modest improvement over CTC to the small amount of label data, about 150K phonemes. He suggests pretraining the prediction network on large target-only corpora.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{graves2012sequence,
  title = {{Sequence Transduction with Recurrent Neural Networks}},
  author = {Graves, Alex},
  year = {2012},
  doi = {10.48550/arxiv.1211.3711},
  eprint = {1211.3711},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1211.3711}
}
```

- [← previous Semantic Compositionality through Recursive Matrix-Vector Spaces](https://md.research.elamur.ai/free/en/papers/socher-2012-semantic-compositionality-through-recursive-matrix/)
- [next → brat: a Web-based Tool for NLP-Assisted Text Annotation](https://md.research.elamur.ai/free/en/papers/stenetorp-2012-brat-web-based-tool-nlp/)
- [↑ back to papers of 2012](https://md.research.elamur.ai/free/en/papers/2012/)

## References

19 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bertolami, R, Zimmermann, M, and Bunke, H. Rejection strategies for offline handwritten text line recognition. Pattern Recognition Letters, 27(16): 2005–2012, 2006. [find](https://scholar.google.com/scholar?q=Bertolami%2C+R%2C+Zimmermann%2C+M%2C+and+Bunke%2C+H%2E+Rejection+strategies+for+offline+handwritten+text+line+recognition%2E+Pattern+Recognition+Letters%2C+27%2816%29%3A+2005%E2%80%932012%2C+2006%2E)
2. Bottou, Leon, Bengio, Yoshua, and Cun, Yann Le. Global training of document processing systems using graph transformer networks. In CVPR, pp. 489–, 1997. [find](https://scholar.google.com/scholar?q=Bottou%2C+Leon%2C+Bengio%2C+Yoshua%2C+and+Cun%2C+Yann+Le%2E+Global+training+of+document+processing+systems+using+graph+transformer+networks%2E+In+CVPR%2C+pp%2E+489%E2%80%93%2C+1997%2E)
3. Dahl, G., Ranzato, M., Mohamed, A., and Hinton, G. Phone recognition with the mean-covariance restricted boltzmann machine. In NIPS. 2010. [find](https://scholar.google.com/scholar?q=Dahl%2C+G%2E%2C+Ranzato%2C+M%2E%2C+Mohamed%2C+A%2E%2C+and+Hinton%2C+G%2E+Phone+recognition+with+the+mean%2Dcovariance+restricted+boltzmann+machine%2E+In+NIPS%2E+2010%2E)
4. The DARPA TIMIT Acoustic-Phonetic Continuous Speech Corpus (TIMIT). DARPA-ISTO, speech disc cd1-1.1 edition, 1990. [find](https://scholar.google.com/scholar?q=The+DARPA+TIMIT+Acoustic%2DPhonetic+Continuous+Speech+Corpus+%28TIMIT%29%2E+DARPA%2DISTO%2C+speech+disc+cd1%2D1%2E1+edition%2C+1990%2E)
5. Gers, F. Long Short-Term Memory in Recurrent Neural Networks. PhD thesis, Ecole Polytechnique Fédérale de Lausanne, 2001. [find](https://scholar.google.com/scholar?q=Gers%2C+F%2E+Long+Short%2DTerm+Memory+in+Recurrent+Neural+Networks%2E+PhD+thesis%2C+Ecole+Polytechnique+Fe%CC%81de%CC%81rale+de+Lausanne%2C+2001%2E)
6. Graves, A. and Schmidhuber, J. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks, 18: 602–610, 2005. [find](https://scholar.google.com/scholar?q=Graves%2C+A%2E+and+Schmidhuber%2C+J%2E+Framewise+phoneme+classification+with+bidirectional+LSTM+and+other+neural+network+architectures%2E+Neural+Networks%2C+18%3A+602%E2%80%93610%2C+2005%2E)
7. Graves, A. and Schmidhuber, J. Offline handwriting recognition with multidimensional recurrent neural networks. In NIPS, 2008. [find](https://scholar.google.com/scholar?q=Graves%2C+A%2E+and+Schmidhuber%2C+J%2E+Offline+handwriting+recognition+with+multidimensional+recurrent+neural+networks%2E+In+NIPS%2C+2008%2E)
8. Graves, A., Fernández, S., Gomez, F., and Schmidhuber, J. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In ICML, 2006. [find](https://scholar.google.com/scholar?q=Graves%2C+A%2E%2C+Ferna%CC%81ndez%2C+S%2E%2C+Gomez%2C+F%2E%2C+and+Schmidhuber%2C+J%2E+Connectionist+temporal+classification%3A+Labelling+unsegmented+sequence+data+with+recurrent+neural+networks%2E+In+ICML%2C+2006%2E)
9. Graves, A., Fernández, S., Liwicki, M., Bunke, H., and Schmidhuber, J. Unconstrained Online Handwriting Recognition with Recurrent Neural Networks. In NIPS. 2008. [find](https://scholar.google.com/scholar?q=Graves%2C+A%2E%2C+Ferna%CC%81ndez%2C+S%2E%2C+Liwicki%2C+M%2E%2C+Bunke%2C+H%2E%2C+and+Schmidhuber%2C+J%2E+Unconstrained+Online+Handwriting+Recognition+with+Recurrent+Neural+Networks%2E+In+NIPS%2E+2008%2E)
10. Hochreiter, S. and Schmidhuber, J. Long Short-Term Memory. Neural Computation, 9(8):1735– 1780, 1997. [find](https://scholar.google.com/scholar?q=Hochreiter%2C+S%2E+and+Schmidhuber%2C+J%2E+Long+Short%2DTerm+Memory%2E+Neural+Computation%2C+9%288%29%3A1735%E2%80%93+1780%2C+1997%2E)
11. Hochreiter, S., Bengio, Y., Frasconi, P., and Schmidhuber, J. Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-term Dependencies. In Kremer, S. C. and Kolen, J. F. (eds.), A Field Guide to Dynamical Recurrent Neural Networks. 2001. [find](https://scholar.google.com/scholar?q=Hochreiter%2C+S%2E%2C+Bengio%2C+Y%2E%2C+Frasconi%2C+P%2E%2C+and+Schmidhuber%2C+J%2E+Gradient+Flow+in+Recurrent+Nets%3A+the+Difficulty+of+Learning+Long%2Dterm+Dependencies%2E+In+Kremer%2C+S%2E+C%2E+and+Kolen%2C+J%2E+F%2E+%28eds%2E%29%2C+A+Field+Guid)
12. Jim, Kam-Chuen, Giles, C., and Horne, B. An analysis of noise in recurrent neural networks: convergence and generalization. Neural Networks, IEEE Transactions on, 7(6):1424 –1438, 1996. [find](https://scholar.google.com/scholar?q=Jim%2C+Kam%2DChuen%2C+Giles%2C+C%2E%2C+and+Horne%2C+B%2E+An+analysis+of+noise+in+recurrent+neural+networks%3A+convergence+and+generalization%2E+Neural+Networks%2C+IEEE+Transactions+on%2C+7%286%29%3A1424+%E2%80%931438%2C+1996%2E)
13. Lafferty, J. D., McCallum, A., and Pereira, F. C. N. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. In ICML, 2001. [find](https://scholar.google.com/scholar?q=Lafferty%2C+J%2E+D%2E%2C+McCallum%2C+A%2E%2C+and+Pereira%2C+F%2E+C%2E+N%2E+Conditional+Random+Fields%3A+Probabilistic+Models+for+Segmenting+and+Labeling+Sequence+Data%2E+In+ICML%2C+2001%2E)
14. LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. [find](https://scholar.google.com/scholar?q=LeCun%2C+Y%2E%2C+Bottou%2C+L%2E%2C+Bengio%2C+Y%2E%2C+and+Haffner%2C+P%2E+Gradient%2Dbased+learning+applied+to+document+recognition%2E+Proceedings+of+the+IEEE%2C+86%2811%29%3A2278%E2%80%932324%2C+1998%2E)
15. Lee, K. and Hon, H. Speaker-independent phone recognition using hidden markov models. IEEE Transactions on Acoustics, Speech, and Signal Processing, 1989. [find](https://scholar.google.com/scholar?q=Lee%2C+K%2E+and+Hon%2C+H%2E+Speaker%2Dindependent+phone+recognition+using+hidden+markov+models%2E+IEEE+Transactions+on+Acoustics%2C+Speech%2C+and+Signal+Processing%2C+1989%2E)
16. Mikolov, T., Karafit, M., Burget, L., Cernocky, J., and Khudanpur, S. Recurrent neural network based language model. In Eleventh Annual Conference of the International Speech Communication Association, 2010. [find](https://scholar.google.com/scholar?q=Mikolov%2C+T%2E%2C+Karafit%2C+M%2E%2C+Burget%2C+L%2E%2C+Cernocky%2C+J%2E%2C+and+Khudanpur%2C+S%2E+Recurrent+neural+network+based+language+model%2E+In+Eleventh+Annual+Conference+of+the+International+Speech+Communication+Association)
17. Schuster, M. and Paliwal, K. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45:2673–2681, 1997. [find](https://scholar.google.com/scholar?q=Schuster%2C+M%2E+and+Paliwal%2C+K%2E+Bidirectional+recurrent+neural+networks%2E+IEEE+Transactions+on+Signal+Processing%2C+45%3A2673%E2%80%932681%2C+1997%2E)
18. Sutskever, I., Martens, J., and Hinton, G. Generating text with recurrent neural networks. In ICML, 2011. [find](https://scholar.google.com/scholar?q=Sutskever%2C+I%2E%2C+Martens%2C+J%2E%2C+and+Hinton%2C+G%2E+Generating+text+with+recurrent+neural+networks%2E+In+ICML%2C+2011%2E)
19. Williams, R. and Zipser, D. Gradient-based learning algorithms for recurrent networks and their computational complexity. In Back-propagation: Theory, Architectures and Applications, pp. 433–486. 1995. [find](https://scholar.google.com/scholar?q=Williams%2C+R%2E+and+Zipser%2C+D%2E+Gradient%2Dbased+learning+algorithms+for+recurrent+networks+and+their+computational+complexity%2E+In+Back%2Dpropagation%3A+Theory%2C+Architectures+and+Applications%2C+pp%2E+433%E2%80%93486%2E+1995)