---
title: "LSTM: A Search Space Odyssey — paper by Greff et al. (2015)"
description: "They then build eight variants, each changing exactly one component. 44 references, cited by 105 papers here."
authors: ["Klaus Greff", "Rupesh Kumar Srivastava", "Jan Koutnik", "Bas R. Steunebrink", "Jurgen Schmidhuber"]
year: 2015
venue: "arXiv:1503.04069 · IEEE Transactions on Neural Networks and Learning Systems ( Volume: 28, Issue: 10, Oct. 2017 ) Pages: 2222 - 2232"
source: "https://arxiv.org/abs/1503.04069"
doi: "10.1109/tnnls.2016.2582924"
arxiv: "1503.04069"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/greff-2015-lstm-search-space-odyssey/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/greff-2015-lstm-search-space-odyssey/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/) / LSTM: A Search Space Odyssey

# LSTM: A Search Space Odyssey

**Authors:** [Klaus Greff](https://md.research.elamur.ai/free/en/authors/klaus-greff/), [Rupesh Kumar Srivastava](https://md.research.elamur.ai/free/en/authors/rupesh-kumar-srivastava/), [Jan Koutnik](https://md.research.elamur.ai/free/en/authors/jan-koutnik/), Bas R. Steunebrink and [Jurgen Schmidhuber](https://md.research.elamur.ai/free/en/authors/jurgen-schmidhuber/) · **Year:** 2015 · **Venue:** [arXiv:1503.04069 · IEEE Transactions on Neural Networks and Learning Systems ( Volume: 28, Issue: 10, Oct. 2017 ) Pages: 2222 - 2232](https://md.research.elamur.ai/free/en/venues/ieee-transactions-on-neural-networks-and-learning-systems/) ([IEEE Transactions on Neural Networks and Learning Systems 2015](https://md.research.elamur.ai/free/en/venues/ieee-transactions-on-neural-networks-and-learning-systems/2015/)) · **Publisher:** [Institute of Electrical and Electronics Engineers](https://md.research.elamur.ai/free/en/publishers/institute-of-electrical-and-electronics-engineers/) · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://doi.org/10.1109/tnnls.2016.2582924) · [PDF](https://arxiv.org/pdf/1503.04069) · [DOI 10.1109/tnnls.2016.2582924](https://doi.org/10.1109/tnnls.2016.2582924) · [arXiv 1503.04069](https://arxiv.org/abs/1503.04069) · [OpenAlex](https://openalex.org/W1689711448)

**Citations:** 6,903 citations (OpenAlex, 2026-10-14) · [105](https://md.research.elamur.ai/free/en/papers/greff-2015-lstm-search-space-odyssey/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

Since its introduction in the mid-1990s, LSTM has accumulated many architectural modifications, but no one had systematically tested which of its parts actually matter. Greff et al. address this gap with a large controlled comparison. They take the widely used "vanilla LSTM" as a baseline. It has input, forget and output gates, peephole connections, an output activation function, and is trained with full BPTT. They then build eight variants, each changing exactly one component. All nine models are tested on three tasks: speech recognition (acoustic modeling), online handwriting recognition, and polyphonic music modeling. Hyperparameters for every model–dataset pair are tuned separately with random search. The fANOVA framework then estimates how much each hyperparameter contributes to performance. No variant reliably beat the vanilla LSTM. Removing the forget gate or the output activation function caused the largest drops. The work matters because it gives empirical backing to the standard design and practical advice for tuning recurrent models.

## Key points

- **Scale:** 5400 experimental runs, roughly 15 years of CPU time, which the authors describe as the largest study of its kind on LSTM.
- **Variants:** eight single-change variants of vanilla LSTM, so the effect of each component can be isolated. The paper also places them in the history of LSTM, alongside GRU (Cho et al.) and Sak et al.'s projection layer.
- **Benchmarks:** the references point to TIMIT for speech, IAM-OnDB for handwriting, and the chorales data of Allan & Williams for music. The extract omits the setup section, so these mappings should be checked against the full paper.
- **Most critical components:** the forget gate, plus the activation function applied at the output.
- **Hyperparameters:** fANOVA shows they interact very little, so each can be tuned more or less independently. The authors turn this into tuning guidelines.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Università della Svizzera italiana](https://md.research.elamur.ai/free/en/universities/universita-della-svizzera-italiana/) — University, Lugano, Switzerland · 10 works · 8,004 citations
- [Dalle Molle Institute for Artificial Intelligence Research](https://md.research.elamur.ai/free/en/labs/dalle-molle-institute-for-artificial-intelligence-research/) — Research institute, Lugano, Switzerland · 10 works · 8,825 citations
- [University of Applied Sciences and Arts of Southern Switzerland](https://md.research.elamur.ai/free/en/universities/university-of-applied-sciences-and-arts-of-southern-switzerland/) — University, Manno, Switzerland · 8 works · 7,374 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

## Cite

Cite as: Klaus Greff, Rupesh Kumar Srivastava, Jan Koutnik, Bas R. Steunebrink and Jurgen Schmidhuber. 2015. LSTM: A Search Space Odyssey. arXiv:1503.04069 · IEEE Transactions on Neural Networks and Learning Systems ( Volume: 28, Issue: 10, Oct. 2017 ) Pages: 2222 - 2232.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{greff2015lstm,
  title = {{LSTM: A Search Space Odyssey}},
  author = {Klaus Greff, Rupesh Kumar Srivastava, Jan Koutnik, Bas R. Steunebrink and Jurgen Schmidhuber},
  year = {2015},
  doi = {10.1109/tnnls.2016.2582924},
  eprint = {1503.04069},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1109/tnnls.2016.2582924}
}
```

- [← previous Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://md.research.elamur.ai/free/en/papers/xu-2015-show-attend-tell-neural-image/)
- [next → Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://md.research.elamur.ai/free/en/papers/radford-2015-unsupervised-representation-learning-deep-convolutional/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/)

## References

44 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Sepp Hochreiter. Untersuchungen zu dynamischen neuronalen Netzen. Masters Thesis, Technische Universität München, München, 1991. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter%2E+Untersuchungen+zu+dynamischen+neuronalen+Netzen%2E+Masters+Thesis%2C+Technische+Universita%CC%88t+Mu%CC%88nchen%2C+Mu%CC%88nchen%2C+1991%2E)
2. S. Hochreiter, Y. Bengio, P. Frasconi, and J. Schmidhuber. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. In S. C. Kremer and J. F. Kolen, editors, A Field Guide to Dynamical Recurrent Neural Networks. IEEE Press, 2001. [find](https://scholar.google.com/scholar?q=S%2E+Hochreiter%2C+Y%2E+Bengio%2C+P%2E+Frasconi%2C+and+J%2E+Schmidhuber%2E+Gradient+flow+in+recurrent+nets%3A+the+difficulty+of+learning+long%2Dterm+dependencies%2E+In+S%2E+C%2E+Kremer+and+J%2E+F%2E+Kolen%2C+editors%2C+A+Field+Guide+t)
3. A Graves, M Liwicki, S Fernandez, R Bertolami, H Bunke, and J Schmidhuber. A Novel Connectionist System for Improved Unconstrained Handwriting Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(5), 2009. [find](https://scholar.google.com/scholar?q=A+Graves%2C+M+Liwicki%2C+S+Fernandez%2C+R+Bertolami%2C+H+Bunke%2C+and+J+Schmidhuber%2E+A+Novel+Connectionist+System+for+Improved+Unconstrained+Handwriting+Recognition%2E+IEEE+Transactions+on+Pattern+Analysis+and+Ma)
4. Vu Pham, Théodore Bluche, Christopher Kermorvant, and Jérôme Louradour. Dropout improves Recurrent Neural Networks for Handwriting Recognition. arXiv:1312.4569 [cs], November 2013. URL http://arxiv.org/abs/1312.4569. — [arXiv:1312.4569](https://arxiv.org/abs/1312.4569) [↗](http://arxiv.org/abs/1312.4569)
5. Patrick Doetsch, Michal Kozielski, and Hermann Ney. Fast and robust training of recurrent neural networks for offline handwriting recognition. In 14th International Conference on Frontiers in Handwriting Recognition, 2014. URL [http://people.sabanciuniv.edu/berrin/cs581/Papers/icfhr2014/data/4334a279…](http://people.sabanciuniv.edu/berrin/cs581/Papers/icfhr2014/data/4334a279…) [↗](http://people.sabanciuniv.edu/berrin/cs581/Papers/icfhr2014/data/4334a279…)
6. Alex Graves. Generating sequences with recurrent neural networks. arXiv:1308.0850 [cs], August 2013. URL http://arxiv.org/abs/1308.0850. [↗](http://arxiv.org/abs/1308.0850)
7. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. [Recurrent Neural Network Regularization](https://md.research.elamur.ai/free/en/papers/zaremba-2014-recurrent-neural-network-regularization/). arXiv:1409.2329 [cs], September 2014. URL http://arxiv.org/abs/1409.2329. [↗](http://arxiv.org/abs/1409.2329)
8. Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. [Addressing the Rare Word Problem in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/luong-2015-addressing-rare-word-problem-neural/). arXiv preprint arXiv:1410.8206, 2014. URL http://arxiv.org/abs/1410.8206. [↗](http://arxiv.org/abs/1410.8206)
9. Hasim Sak, Andrew Senior, and Françoise Beaufays. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In Proceedings of the Annual Conference of International Speech Communication Association (INTERSPEECH), 2014. URL [http://193.6.4.39/](http://193.6.4.39/) ∼ czap/letoltes/IS… [↗](http://193.6.4.39/)
10. Yuchen Fan, Yao Qian, Fenglong Xie, and Frank K. Soong. TTS synthesis with bidirectional LSTM based recurrent neural networks. In Proc. Interspeech, 2014. [find](https://scholar.google.com/scholar?q=Yuchen+Fan%2C+Yao+Qian%2C+Fenglong+Xie%2C+and+Frank+K%2E+Soong%2E+TTS+synthesis+with+bidirectional+LSTM+based+recurrent+neural+networks%2E+In+Proc%2E+Interspeech%2C+2014%2E)
11. Søren Kaae Sønderby and Ole Winther. Protein Secondary Structure Prediction with Long Short Term Memory Networks. arXiv:1412.7828 [cs, q-bio], December 2014. URL http://arxiv.org/abs/1412.7828. arXiv: 1412.7828. — [arXiv:1412.7828](https://arxiv.org/abs/1412.7828) [↗](http://arxiv.org/abs/1412.7828)
12. E. Marchi, G. Ferroni, F. Eyben, L. Gabrielli, S. Squartini, and B. Schuller. Multi-resolution linear prediction based features for audio onset detection with bidirectional LSTM neural networks. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2164–216… — [doi:10.1109/icassp.2014.6853982](https://doi.org/10.1109/icassp.2014.6853982) [find](https://scholar.google.com/scholar?q=E%2E+Marchi%2C+G%2E+Ferroni%2C+F%2E+Eyben%2C+L%2E+Gabrielli%2C+S%2E+Squartini%2C+and+B%2E+Schuller%2E+Multi%2Dresolution+linear+prediction+based+features+for+audio+onset+detection+with+bidirectional+LSTM+neural+networks%2E+In+20)
13. Jeff Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. [Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://md.research.elamur.ai/free/en/papers/donahue-2014-long-term-recurrent-convolutional-networks/). arXiv:1411.4389 [cs], November 2014. URL http://arxiv.org/abs/1411.4389. arXiv: 1411.4389… [↗](http://arxiv.org/abs/1411.4389)
14. Sepp Hochreiter and Jürgen Schmidhuber. Long Short Term Memory. Technical Report FKI-207-95, Technische Universität München, München, August 1995. URL [http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.51.3117](http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.51.3117). [↗](http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.51.3117)
15. Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8):1735–1780, November 1997. ISSN 0899-7667. doi: 10.1162/neco.1997.9.8.1735. URL [http://www.bioinf.jku.at/publications/older/2604.pdf](http://www.bioinf.jku.at/publications/older/2604.pdf). [↗](http://www.bioinf.jku.at/publications/older/2604.pdf)
16. R. L. Anderson. Recent Advances in Finding Best Operating Conditions. Journal of the American Statistical Association, 48(264):789–798, December 1953. ISSN 0162-1459. doi: 10.2307/2281072. URL [http://www.jstor.org/stable/2281072](http://www.jstor.org/stable/2281072). — [doi:10.2307/2281072](https://doi.org/10.2307/2281072) [↗](http://www.jstor.org/stable/2281072)
17. Francisco J. Solis and Roger J.-B. Wets. Minimization by Random Search Techniques. Mathematics of Operations Research, 6(1):19–30, February 1981. ISSN 0364-765X. doi: 10.1287/moor.6.1.19. URL [http://pubsonline.informs.org/doi/abs/10.1287/moor.6.1.19](http://pubsonline.informs.org/doi/abs/10.1287/moor.6.1.19). — [doi:10.1287/moor.6.1.19](https://doi.org/10.1287/moor.6.1.19) [↗](http://pubsonline.informs.org/doi/abs/10.1287/moor.6.1.19)
18. James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. The Journal of Machine Learning Research, 13(1):281–305, 2012. URL [http://dl.acm.org/citation.cfm?id=2188395](http://dl.acm.org/citation.cfm?id=2188395). [↗](http://dl.acm.org/citation.cfm?id=2188395)
19. Frank Hutter, Holger Hoos, and Kevin Leyton-Brown. An Efficient Approach for Assessing Hyperparameter Importance. pages 754–762, 2014. URL [http://jmlr.org/proceedings/papers/v32/hutter14.html](http://jmlr.org/proceedings/papers/v32/hutter14.html). [↗](http://jmlr.org/proceedings/papers/v32/hutter14.html)
20. Alex Graves and Jürgen Schmidhuber. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks, 18(5–6): 602–610, July 2005. ISSN 0893-6080. doi: 10.1016/j. neunet.2005.06.042. URL [http://www.sciencedirect.com/science/article/pii/S08936080050012…](http://www.sciencedirect.com/science/article/pii/S08936080050012…) [↗](http://www.sciencedirect.com/science/article/pii/S08936080050012…)
21. Felix A. Gers, Jürgen Schmidhuber, and Fred Cummins. Learning to forget: Continual prediction with LSTM. In Artificial Neural Networks, 1999. ICANN 99. Ninth International Conference on (Conf. Publ. No. 470), volume 2, pages 850–855, 1999. [find](https://scholar.google.com/scholar?q=Felix+A%2E+Gers%2C+Ju%CC%88rgen+Schmidhuber%2C+and+Fred+Cummins%2E+Learning+to+forget%3A+Continual+prediction+with+LSTM%2E+In+Artificial+Neural+Networks%2C+1999%2E+ICANN+99%2E+Ninth+International+Conference+on+%28Conf%2E+Publ%2E+)
22. Felix A. Gers and Jürgen Schmidhuber. Recurrent nets that time and count. In Neural Networks, 2000. IJCNN 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on, volume 3, pages 189–194. IEEE, 2000. ISBN 0769506194. [find](https://scholar.google.com/scholar?q=Felix+A%2E+Gers+and+Ju%CC%88rgen+Schmidhuber%2E+Recurrent+nets+that+time+and+count%2E+In+Neural+Networks%2C+2000%2E+IJCNN+2000%2C+Proceedings+of+the+IEEE%2DINNS%2DENNS+International+Joint+Conference+on%2C+volume+3%2C+pages+18)
23. AJ Robinson and Frank Fallside. The utility driven dynamic error propagation network. University of Cambridge Department of Engineering, 1987. [find](https://scholar.google.com/scholar?q=AJ+Robinson+and+Frank+Fallside%2E+The+utility+driven+dynamic+error+propagation+network%2E+University+of+Cambridge+Department+of+Engineering%2C+1987%2E)
24. R. J. Williams. Complexity of exact gradient computation algorithms for recurrent neural networks. Technical Report Technical Report NU-CCS-89-27, Boston: Northeastern University, College of Computer Science, 1989. [find](https://scholar.google.com/scholar?q=R%2E+J%2E+Williams%2E+Complexity+of+exact+gradient+computation+algorithms+for+recurrent+neural+networks%2E+Technical+Report+Technical+Report+NU%2DCCS%2D89%2D27%2C+Boston%3A+Northeastern+University%2C+College+of+Computer+)
25. P. J. Werbos. Generalization of backpropagation with application to a recurrent gas market model. Neural Networks, 1, 1988. [find](https://scholar.google.com/scholar?q=P%2E+J%2E+Werbos%2E+Generalization+of+backpropagation+with+application+to+a+recurrent+gas+market+model%2E+Neural+Networks%2C+1%2C+1988%2E)
26. JS Garofolo, LF Lamel, WM Fisher, JG Fiscus, DS Pallett, and NL Dahlgren. DARPA TIMIT Acoustic-Phonetic Continuous Speech Corpus CD-ROM. National Institute of Standards and Technology, NTIS Order No PB91-505065, 1993. [find](https://scholar.google.com/scholar?q=JS+Garofolo%2C+LF+Lamel%2C+WM+Fisher%2C+JG+Fiscus%2C+DS+Pallett%2C+and+NL+Dahlgren%2E+DARPA+TIMIT+Acoustic%2DPhonetic+Continuous+Speech+Corpus+CD%2DROM%2E+National+Institute+of+Standards+and+Technology%2C+NTIS+Order+No+P)
27. Felix A. Gers, Juan Antonio Pérez-Ortiz, Douglas Eck, and Jürgen Schmidhuber. DEFK-LSTM. In ESANN 2002, Proceedings of the 10th Eurorean Symposium on Artificial Neural Networks, 2002. [find](https://scholar.google.com/scholar?q=Felix+A%2E+Gers%2C+Juan+Antonio+Pe%CC%81rez%2DOrtiz%2C+Douglas+Eck%2C+and+Ju%CC%88rgen+Schmidhuber%2E+DEFK%2DLSTM%2E+In+ESANN+2002%2C+Proceedings+of+the+10th+Eurorean+Symposium+on+Artificial+Neural+Networks%2C+2002%2E)
28. J Schmidhuber, D Wierstra, M Gagliolo, and F J Gomez. Training Recurrent Networks by EVOLINO. Neural Computation, 19(3):757–779, 2007. [find](https://scholar.google.com/scholar?q=J+Schmidhuber%2C+D+Wierstra%2C+M+Gagliolo%2C+and+F+J+Gomez%2E+Training+Recurrent+Networks+by+EVOLINO%2E+Neural+Computation%2C+19%283%29%3A757%E2%80%93779%2C+2007%2E)
29. Justin Bayer, Daan Wierstra, Julian Togelius, and Jürgen Schmidhuber. Evolving memory cell structures for sequence learning. In Artificial Neural Networks–ICANN 2009, pages 755–764. Springer, 2009. URL [http://link.springer.com/chapter/10.1007/978-3-642-04277-5](http://link.springer.com/chapter/10.1007/978-3-642-04277-5) 76. — [doi:10.1007/978-3-642-04277-5](https://doi.org/10.1007/978-3-642-04277-5) [↗](http://link.springer.com/chapter/10.1007/978-3-642-04277-5)
30. Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever. An empirical exploration of recurrent network architectures. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pages 2342– 2350, 2015. [find](https://scholar.google.com/scholar?q=Rafal+Jozefowicz%2C+Wojciech+Zaremba%2C+and+Ilya+Sutskever%2E+An+empirical+exploration+of+recurrent+network+architectures%2E+In+Proceedings+of+the+32nd+International+Conference+on+Machine+Learning+%28ICML%2D15%29%2C+)
31. Sebastian Otte, Marcus Liwicki, and Andreas Zell. Dynamic Cortex Memory: Enhancing Recurrent Neural Networks for Gradient-Based Sequence Learning. In Artificial Neural Networks and Machine Learning – ICANN 2014, number 8681 in Lecture Notes in Computer Science, pages 1–8. Springer International Publ… — [doi:10.1007/978-3-319-11179-7](https://doi.org/10.1007/978-3-319-11179-7) [find](https://scholar.google.com/scholar?q=Sebastian+Otte%2C+Marcus+Liwicki%2C+and+Andreas+Zell%2E+Dynamic+Cortex+Memory%3A+Enhancing+Recurrent+Neural+Networks+for+Gradient%2DBased+Sequence+Learning%2E+In+Artificial+Neural+Networks+and+Machine+Learning+%E2%80%93+)
32. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. arXiv preprint arXiv:1406.[1078, 2014](https://md.research.elamur.ai/free/en/papers/cho-2014-learning-phrase-representations-rnn-encoder/). URL http://arxiv.org/abs/1406.1078. [↗](http://arxiv.org/abs/1406.1078)
33. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. arXiv:1412.3555 [cs], December 2014. URL http://arxiv.org/abs/1412.3555. [↗](http://arxiv.org/abs/1412.3555)
34. David Crystal. Dictionary of linguistics and phonetics, volume 30. John Wiley & Sons, 2011. [find](https://scholar.google.com/scholar?q=David+Crystal%2E+Dictionary+of+linguistics+and+phonetics%2C+volume+30%2E+John+Wiley+%26+Sons%2C+2011%2E)
35. P. Mermelstein. Distance measures for speech recognition: Psychological and instrumental. In C. H. Chen, editor, Pattern Recognition and Artificial Intelligence, pages 374– 388. Academic Press, New York, 1976. [find](https://scholar.google.com/scholar?q=P%2E+Mermelstein%2E+Distance+measures+for+speech+recognition%3A+Psychological+and+instrumental%2E+In+C%2E+H%2E+Chen%2C+editor%2C+Pattern+Recognition+and+Artificial+Intelligence%2C+pages+374%E2%80%93+388%2E+Academic+Press%2C+New+Yo)
36. Alexander Graves. Supervised Sequence Labelling with Recurrent Neural Networks. Ph.d., The Technical University of Munich, 2008. [find](https://scholar.google.com/scholar?q=Alexander+Graves%2E+Supervised+Sequence+Labelling+with+Recurrent+Neural+Networks%2E+Ph%2Ed%2E%2C+The+Technical+University+of+Munich%2C+2008%2E)
37. Andrew K. Halberstadt. Heterogeneous acoustic measurements and multiple classifiers for speech recognition. PhD thesis, Massachusetts Institute of Technology, 1998. [find](https://scholar.google.com/scholar?q=Andrew+K%2E+Halberstadt%2E+Heterogeneous+acoustic+measurements+and+multiple+classifiers+for+speech+recognition%2E+PhD+thesis%2C+Massachusetts+Institute+of+Technology%2C+1998%2E)
38. Marcus Liwicki and Horst Bunke. IAM-OnDB-an on-line English sentence database acquired from handwritten text on a whiteboard. In Document Analysis and Recognition, 2005. Proceedings. Eighth International Conference on, pages 956–961. IEEE, 2005. [find](https://scholar.google.com/scholar?q=Marcus+Liwicki+and+Horst+Bunke%2E+IAM%2DOnDB%2Dan+on%2Dline+English+sentence+database+acquired+from+handwritten+text+on+a+whiteboard%2E+In+Document+Analysis+and+Recognition%2C+2005%2E+Proceedings%2E+Eighth+Internatio)
39. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369–376, 2006. URL [http://dl.acm.org/ci…](http://dl.acm.org/ci…) [↗](http://dl.acm.org/ci…)
40. Moray Allan and Christopher KI Williams. Harmonising chorales by probabilistic inference. Advances in neural information processing systems, 17:25–32, 2005. [find](https://scholar.google.com/scholar?q=Moray+Allan+and+Christopher+KI+Williams%2E+Harmonising+chorales+by+probabilistic+inference%2E+Advances+in+neural+information+processing+systems%2C+17%3A25%E2%80%9332%2C+2005%2E)
41. Nicolas Boulanger-Lewandowski, Yoshua Bengio, and Pascal Vincent. Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription. pages 1159–1166, 2012. URL [http://icml.cc/discuss/2012/590.html](http://icml.cc/discuss/2012/590.html). [↗](http://icml.cc/discuss/2012/590.html)
42. Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In JMLR, pages 1139–1147, 2013. URL [http://jmlr.org/proceedings/papers/v28/sutskever13](http://jmlr.org/proceedings/papers/v28/sutskever13). html. [↗](http://jmlr.org/proceedings/papers/v28/sutskever13)
43. Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical Bayesian Optimization of Machine Learning Algorithms. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 25, pages 2951–2959. Curran Associates, Inc., 2012. [find](https://scholar.google.com/scholar?q=Jasper+Snoek%2C+Hugo+Larochelle%2C+and+Ryan+P+Adams%2E+Practical+Bayesian+Optimization+of+Machine+Learning+Algorithms%2E+In+F%2E+Pereira%2C+C%2E+J%2E+C%2E+Burges%2C+L%2E+Bottou%2C+and+K%2E+Q%2E+Weinberger%2C+editors%2C+Advances+in+N)
44. F. Hutter, H. H. Hoos, and K. Leyton-Brown. Sequential Model-Based Optimization for General Algorithm Klaus Greff received his Diploma in Computer Science from the University of Kaiserslautern, Germany in 2011. Currently he is pursuing his PhD at IDSIA in Lugano, Switzerland, under the supervision o… [find](https://scholar.google.com/scholar?q=F%2E+Hutter%2C+H%2E+H%2E+Hoos%2C+and+K%2E+Leyton%2DBrown%2E+Sequential+Model%2DBased+Optimization+for+General+Algorithm+Klaus+Greff+received+his+Diploma+in+Computer+Science+from+the+University+of+Kaiserslautern%2C+German)

*Updated 2 November 2026*