---
title: "Recurrent Highway Networks — paper by Zilly et al. (2016)"
description: "From this analysis the authors propose Recurrent Highway Networks (RHNs). 53 references, cited by 73 papers here."
authors: ["Julian Georg Zilly", "Rupesh Kumar Srivastava", "Jan Koutnik", "Jurgen Schmidhuber"]
year: 2016
venue: "arXiv:1607.03474"
source: "https://arxiv.org/abs/1607.03474"
doi: "10.48550/arxiv.1607.03474"
arxiv: "1607.03474"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zilly-2016-recurrent-highway-networks/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zilly-2016-recurrent-highway-networks/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/) / Recurrent Highway Networks

# Recurrent Highway Networks

**Authors:** Julian Georg Zilly, [Rupesh Kumar Srivastava](https://md.research.elamur.ai/free/en/authors/rupesh-kumar-srivastava/), [Jan Koutnik](https://md.research.elamur.ai/free/en/authors/jan-koutnik/) and [Jurgen Schmidhuber](https://md.research.elamur.ai/free/en/authors/jurgen-schmidhuber/) · **Year:** 2016 · **Venue:** arXiv:1607.03474 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1607.03474) · [PDF](https://arxiv.org/pdf/1607.03474) · [DOI 10.48550/arxiv.1607.03474](https://doi.org/10.48550/arxiv.1607.03474) · [arXiv 1607.03474](https://arxiv.org/abs/1607.03474) · [OpenAlex](https://openalex.org/W2473934411)

**Citations:** 74 citations (OpenAlex, 2026-10-14) · [73](https://md.research.elamur.ai/free/en/papers/zilly-2016-recurrent-highway-networks/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Recurrent networks usually update their hidden state with a single nonlinear step per time step. Making this step-to-step transition deeper should add modeling power, but such models have very long gradient paths and are hard to train, even when built from LSTM cells. Zilly et al. study this problem using Geršgorin's circle theorem. The theorem bounds where the eigenvalues of the recurrent Jacobian can lie. This shows how weight initialization and gating affect vanishing and exploding gradients, and why LSTM-style gates help keep training stable. From this analysis the authors propose Recurrent Highway Networks (RHNs). An RHN stacks several gated Highway layers inside each recurrent transition, so the recurrence depth can exceed one while training stays manageable. They test RHNs on language modeling with Penn Treebank (word level) and text8 and enwik8 (character level). On these benchmarks, deeper transitions give clear gains without adding parameters. RHNs matter because they show that depth within each time step is a practical and efficient way to make recurrent models stronger.

## Key points
- On Penn Treebank, increasing recurrence depth from 1 to 10 with a fixed parameter budget lowers word-level test perplexity from 90.6 to 65.4.
- A large RHN with recurrence depth 10 and 1500 units reaches 1.27 BPC on the test sets of both text8 (about 45M parameters) and enwik8 (about 46M parameters). This beat earlier published results that used fewer than 1.5 BPC and no dynamic evaluation.
- The Geršgorin analysis shows that large off-diagonal recurrent weights widen the possible spread of eigenvalues and risk exploding gradients. Identity initialization (Le et al., 2015) helps at first but can still drift during training.
- On JSB Chorales, gate inspection shows that a trained depth-6 RHN uses all of its layers. Lesioning experiments show the first transition layer contributes most to performance, but removing any layer hurts noticeably.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [ETH Zurich](https://md.research.elamur.ai/free/en/universities/eth-zurich/) — University, Zurich, Switzerland · 84 works · 9,181 citations
- [Università della Svizzera italiana](https://md.research.elamur.ai/free/en/universities/universita-della-svizzera-italiana/) — University, Lugano, Switzerland · 10 works · 8,004 citations
- [Dalle Molle Institute for Artificial Intelligence Research](https://md.research.elamur.ai/free/en/labs/dalle-molle-institute-for-artificial-intelligence-research/) — Research institute, Lugano, Switzerland · 10 works · 8,825 citations
- [University of Applied Sciences and Arts of Southern Switzerland](https://md.research.elamur.ai/free/en/universities/university-of-applied-sciences-and-arts-of-southern-switzerland/) — University, Manno, Switzerland · 8 works · 7,374 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

## Cite

Cite as: Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutnik and Jurgen Schmidhuber. 2016. Recurrent Highway Networks. arXiv:1607.03474.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zilly2016recurrent,
  title = {{Recurrent Highway Networks}},
  author = {Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutnik and Jurgen Schmidhuber},
  year = {2016},
  doi = {10.48550/arxiv.1607.03474},
  eprint = {1607.03474},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1607.03474}
}
```

- [← previous Sequence-to-Sequence Generation for Spoken Dialogue via Deep Syntax Trees and Strings](https://md.research.elamur.ai/free/en/papers/dusek-2016-sequence-sequence-generation-spoken-dialogue/)
- [next → Two negations for the price of one](https://md.research.elamur.ai/free/en/papers/thornton-2016-two-negations-price-one/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/)

## References

53 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bengio, Yoshua and LeCun, Yann. Scaling learning algorithms towards ai. Large-scale kernel machines, 34(5):1–41, 2007. [find](https://scholar.google.com/scholar?q=Bengio%2C+Yoshua+and+LeCun%2C+Yann%2E+Scaling+learning+algorithms+towards+ai%2E+Large%2Dscale+kernel+machines%2C+34%285%29%3A1%E2%80%9341%2C+2007%2E)
2. Bengio, Yoshua, Simard, Patrice, and Frasconi, Paolo. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157–166, 1994. [find](https://scholar.google.com/scholar?q=Bengio%2C+Yoshua%2C+Simard%2C+Patrice%2C+and+Frasconi%2C+Paolo%2E+Learning+long%2Dterm+dependencies+with+gradient+descent+is+difficult%2E+IEEE+transactions+on+neural+networks%2C+5%282%29%3A157%E2%80%93166%2C+1994%2E)
3. Bianchini, Monica and Scarselli, Franco. On the complexity of neural network classifiers: A comparison between shallow and deep architectures. IEEE Transactions on Neural Networks, 2014. [find](https://scholar.google.com/scholar?q=Bianchini%2C+Monica+and+Scarselli%2C+Franco%2E+On+the+complexity+of+neural+network+classifiers%3A+A+comparison+between+shallow+and+deep+architectures%2E+IEEE+Transactions+on+Neural+Networks%2C+2014%2E)
4. Boulanger-Lewandowski, N., Bengio, Y., and Vincent, P. Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription. ArXiv e-prints, June 2012. [find](https://scholar.google.com/scholar?q=Boulanger%2DLewandowski%2C+N%2E%2C+Bengio%2C+Y%2E%2C+and+Vincent%2C+P%2E+Modeling+temporal+dependencies+in+high%2Ddimensional+sequences%3A+Application+to+polyphonic+music+generation+and+transcription%2E+ArXiv+e%2Dprints%2C+June+)
5. Cho, Kyunghyun, Van Merriënboer, Bart, Gulcehre, Caglar, Bahdanau, Dzmitry, Bougares, Fethi, Schwenk, Holger, and Bengio, Yoshua. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.[1078, 2014](https://md.research.elamur.ai/free/en/papers/cho-2014-learning-phrase-representations-rnn-encoder/).
6. Chung, J., Ahn, S., and Bengio, Y. [Hierarchical Multiscale Recurrent Neural Networks](https://md.research.elamur.ai/free/en/papers/chung-2016-hierarchical-multiscale-recurrent-neural-networks/). ArXiv e-prints, September 2016.
7. Chung, Junyoung, Gulcehre, Caglar, Cho, Kyunghyun, and Bengio, Yoshua. Gated feedback recurrent neural networks. In International Conference on Machine Learning, 2015. [find](https://scholar.google.com/scholar?q=Chung%2C+Junyoung%2C+Gulcehre%2C+Caglar%2C+Cho%2C+Kyunghyun%2C+and+Bengio%2C+Yoshua%2E+Gated+feedback+recurrent+neural+networks%2E+In+International+Conference+on+Machine+Learning%2C+2015%2E)
8. Cooijmans, T., Ballas, N., Laurent, C., Gülçehre, Ç., and Courville, A. Recurrent Batch Normalization. ArXiv e-prints, March 2016. [find](https://scholar.google.com/scholar?q=Cooijmans%2C+T%2E%2C+Ballas%2C+N%2E%2C+Laurent%2C+C%2E%2C+G%C3%BCl%C3%A7ehre%2C+%C3%87%2E%2C+and+Courville%2C+A%2E+Recurrent+Batch+Normalization%2E+ArXiv+e%2Dprints%2C+March+2016%2E)
9. Gal, Yarin. [A theoretically grounded application of dropout in recurrent neural networks](https://md.research.elamur.ai/free/en/papers/gal-2015-theoretically-grounded-application-dropout-recurrent/). arXiv preprint arXiv:1512.05287, 2015.
10. Gers, Felix A., Schmidhuber, Jürgen, and Cummins, Fred. Learning to forget: Continual prediction with lstm. Neural Computation, 12(10):2451–2471, 2016/02/18 2000. [find](https://scholar.google.com/scholar?q=Gers%2C+Felix+A%2E%2C+Schmidhuber%2C+J%C3%BCrgen%2C+and+Cummins%2C+Fred%2E+Learning+to+forget%3A+Continual+prediction+with+lstm%2E+Neural+Computation%2C+12%2810%29%3A2451%E2%80%932471%2C+2016%2F02%2F18+2000%2E)
11. Geršgorin, S. Über die Abgrenzung der Eigenwerte einer Matrix. Bulletin de l’Acadèmie des Sciences de l’URSS. Classe des sciences mathèmatiques, no. 6:749–754, 1931. [find](https://scholar.google.com/scholar?q=Ger%C5%A1gorin%2C+S%2E+%C3%9Cber+die+Abgrenzung+der+Eigenwerte+einer+Matrix%2E+Bulletin+de+l%E2%80%99Acad%C3%A8mie+des+Sciences+de+l%E2%80%99URSS%2E+Classe+des+sciences+math%C3%A8matiques%2C+no%2E+6%3A749%E2%80%93754%2C+1931%2E)
12. Graves, A. Generating sequences with recurrent neural networks. ArXiv e-prints, August 2013. [find](https://scholar.google.com/scholar?q=Graves%2C+A%2E+Generating+sequences+with+recurrent+neural+networks%2E+ArXiv+e%2Dprints%2C+August+2013%2E)
13. Graves, A. Adaptive Computation Time for Recurrent Neural Networks. ArXiv e-prints, March 2016. [find](https://scholar.google.com/scholar?q=Graves%2C+A%2E+Adaptive+Computation+Time+for+Recurrent+Neural+Networks%2E+ArXiv+e%2Dprints%2C+March+2016%2E)
14. Greff, K., Srivastava, R. K., Koutník, J., Steunebrink, B. R, and Schmidhuber, J. [LSTM: A Search Space Odyssey](https://md.research.elamur.ai/free/en/papers/greff-2015-lstm-search-space-odyssey/). arXiv preprint arXiv:1503.04069, 2015.
15. Greff, Klaus, Srivastava, Rupesh K, and Schmidhuber, Jürgen. Highway and residual networks learn unrolled iterative estimation. arXiv preprint arXiv:1612.07771, 2016. — [arXiv:1612.07771](https://arxiv.org/abs/1612.07771) [find](https://scholar.google.com/scholar?q=Greff%2C+Klaus%2C+Srivastava%2C+Rupesh+K%2C+and+Schmidhuber%2C+J%C3%BCrgen%2E+Highway+and+residual+networks+learn+unrolled+iterative+estimation%2E+arXiv+preprint+arXiv%3A1612%2E07771%2C+2016%2E+%E2%80%94+%5BarXiv%3A1612%2E07771%5D%28https%3A%2F%2Farxi)
16. Ha, D., Dai, A., and Le, Q. V. HyperNetworks. ArXiv e-prints, September 2016. [find](https://scholar.google.com/scholar?q=Ha%2C+D%2E%2C+Dai%2C+A%2E%2C+and+Le%2C+Q%2E+V%2E+HyperNetworks%2E+ArXiv+e%2Dprints%2C+September+2016%2E)
17. He, K., Zhang, X., Ren, S., and Sun, J. [Deep residual learning for image recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/). arXiv preprint arXiv:1512.03385, 2015.
18. Hochreiter, S. Untersuchungen zu dynamischen neuronalen Netzen. Master’s thesis, Institut f. Informatik, Technische Univ. Munich, 1991. [find](https://scholar.google.com/scholar?q=Hochreiter%2C+S%2E+Untersuchungen+zu+dynamischen+neuronalen+Netzen%2E+Master%E2%80%99s+thesis%2C+Institut+f%2E+Informatik%2C+Technische+Univ%2E+Munich%2C+1991%2E)
19. Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural Computation, 9(8):1735–1780, 1997. [find](https://scholar.google.com/scholar?q=Hochreiter%2C+S%2E+and+Schmidhuber%2C+J%2E+Long+short%2Dterm+memory%2E+Neural+Computation%2C+9%288%29%3A1735%E2%80%931780%2C+1997%2E)
20. Hochreiter, S., Bengio, Y., Frasconi, P., and Schmidhuber, J. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. In Kremer, S. C. and Kolen, J. F. (eds.), A Field Guide to Dynamical Recurrent Neural Networks. IEEE Press, 2001. [find](https://scholar.google.com/scholar?q=Hochreiter%2C+S%2E%2C+Bengio%2C+Y%2E%2C+Frasconi%2C+P%2E%2C+and+Schmidhuber%2C+J%2E+Gradient+flow+in+recurrent+nets%3A+the+difficulty+of+learning+long%2Dterm+dependencies%2E+In+Kremer%2C+S%2E+C%2E+and+Kolen%2C+J%2E+F%2E+%28eds%2E%29%2C+A+Field+Guid)
21. Hutter, M. The human knowledge compression contest. [http://prize.hutter1.net/](http://prize.hutter1.net/), 2012. [↗](http://prize.hutter1.net/)
22. Inan, H., Khosravi, K., and Socher, R. [Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling](https://md.research.elamur.ai/free/en/papers/inan-2016-tying-word-vectors-word-classifiers/). ArXiv e-prints, November 2016.
23. Inan, Hakan and Khosravi, Khashayar. Improved learning through augmenting the loss, 2016. [find](https://scholar.google.com/scholar?q=Inan%2C+Hakan+and+Khosravi%2C+Khashayar%2E+Improved+learning+through+augmenting+the+loss%2C+2016%2E)
24. Jozefowicz, Rafal, Zaremba, Wojciech, and Sutskever, Ilya. An empirical exploration of recurrent network architectures. 2015. [find](https://scholar.google.com/scholar?q=Jozefowicz%2C+Rafal%2C+Zaremba%2C+Wojciech%2C+and+Sutskever%2C+Ilya%2E+An+empirical+exploration+of+recurrent+network+architectures%2E+2015%2E)
25. Jozefowicz, Rafal, Vinyals, Oriol, Schuster, Mike, Shazeer, Noam, and Wu, Yonghui. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016. [find](https://scholar.google.com/scholar?q=Jozefowicz%2C+Rafal%2C+Vinyals%2C+Oriol%2C+Schuster%2C+Mike%2C+Shazeer%2C+Noam%2C+and+Wu%2C+Yonghui%2E+Exploring+the+limits+of+language+modeling%2E+arXiv+preprint+arXiv%3A1602%2E02410%2C+2016%2E)
26. Kalchbrenner, Nal, Danihelka, Ivo, and Graves, Alex. Grid long short-term memory. CoRR, abs/1507.01526, 2015. URL http://arxiv.org/abs/1507.01526. [↗](http://arxiv.org/abs/1507.01526)
27. Kim, Yoon, Jernite, Yacine, Sontag, David, and Rush, Alexander M. [Character-aware neural language models](https://md.research.elamur.ai/free/en/papers/kim-2015-character-aware-neural-language-models/). arXiv preprint arXiv:1508.06615, 2015.
28. Krause, B., Lu, L., Murray, I., and Renals, S. Multiplicative LSTM for sequence modelling. ArXiv e-prints, September 2016. [find](https://scholar.google.com/scholar?q=Krause%2C+B%2E%2C+Lu%2C+L%2E%2C+Murray%2C+I%2E%2C+and+Renals%2C+S%2E+Multiplicative+LSTM+for+sequence+modelling%2E+ArXiv+e%2Dprints%2C+September+2016%2E)
29. Le, Q. V., Jaitly, N., and Hinton, G. E. A Simple Way to Initialize Recurrent Networks of Rectified Linear Units. ArXiv e-prints, April 2015. [find](https://scholar.google.com/scholar?q=Le%2C+Q%2E+V%2E%2C+Jaitly%2C+N%2E%2C+and+Hinton%2C+G%2E+E%2E+A+Simple+Way+to+Initialize+Recurrent+Networks+of+Rectified+Linear+Units%2E+ArXiv+e%2Dprints%2C+April+2015%2E)
30. Lei Ba, J., Kiros, J. R., and Hinton, G. E. [Layer Normalization](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/). ArXiv e-prints, July 2016.
31. Linnainmaa, S. The representation of the cumulative rounding error of an algorithm as a taylor expansion of the local rounding errors. Master’s thesis, Univ. Helsinki, 1970. [find](https://scholar.google.com/scholar?q=Linnainmaa%2C+S%2E+The+representation+of+the+cumulative+rounding+error+of+an+algorithm+as+a+taylor+expansion+of+the+local+rounding+errors%2E+Master%E2%80%99s+thesis%2C+Univ%2E+Helsinki%2C+1970%2E)
32. Linnainmaa, Seppo. Taylor expansion of the accumulated rounding error. BIT Numerical Mathematics, 16(2):146–160, 1976. ISSN 1572-9125. [find](https://scholar.google.com/scholar?q=Linnainmaa%2C+Seppo%2E+Taylor+expansion+of+the+accumulated+rounding+error%2E+BIT+Numerical+Mathematics%2C+16%282%29%3A146%E2%80%93160%2C+1976%2E+ISSN+1572%2D9125%2E)
33. Marcus, Mitchell P., Marcinkiewicz, Mary Ann, and Santorini, Beatrice. [Building a large annotated corpus of english: The penn treebank](https://md.research.elamur.ai/free/en/papers/marcus-1993-building-large-annotated-corpus-english/). Comput. Linguist., 19(2):313–330, June 1993. ISSN 0891-2017.
34. Merity, S., Xiong, C., Bradbury, J., and Socher, R. [Pointer Sentinel Mixture Models](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/). ArXiv e-prints, September 2016.
35. Mikolov, Tomas and Zweig, Geoffrey. Context dependent recurrent neural network language model. SLT, 12:234–239, 2012. [find](https://scholar.google.com/scholar?q=Mikolov%2C+Tomas+and+Zweig%2C+Geoffrey%2E+Context+dependent+recurrent+neural+network+language+model%2E+SLT%2C+12%3A234%E2%80%93239%2C+2012%2E)
36. Mikolov, Tomas, Karafiát, Martin, Burget, Lukas, Cernockỳ, Jan, and Khudanpur, Sanjeev. Recurrent neural network based language model. In Interspeech, volume 2, pp. 3, 2010. [find](https://scholar.google.com/scholar?q=Mikolov%2C+Tomas%2C+Karafi%C3%A1t%2C+Martin%2C+Burget%2C+Lukas%2C+Cernocky%CC%80%2C+Jan%2C+and+Khudanpur%2C+Sanjeev%2E+Recurrent+neural+network+based+language+model%2E+In+Interspeech%2C+volume+2%2C+pp%2E+3%2C+2010%2E)
37. Pascanu, R., Gulcehre, C., Cho, K., and Bengio, Y. [How to construct deep recurrent neural networks](https://md.research.elamur.ai/free/en/papers/pascanu-2013-construct-deep-recurrent-neural-networks/). ArXiv e-prints, December 2013.
38. Press, O. and Wolf, L. [Using the Output Embedding to Improve Language Models](https://md.research.elamur.ai/free/en/papers/press-2017-output-embedding-improve-language-models/). ArXiv e-prints, August 2016.
39. Robinson, A. J. and Fallside, F. The utility driven dynamic error propagation network. Technical Report CUED/F-INFENG/TR.1, Cambridge University Engineering Department, 1987. [find](https://scholar.google.com/scholar?q=Robinson%2C+A%2E+J%2E+and+Fallside%2C+F%2E+The+utility+driven+dynamic+error+propagation+network%2E+Technical+Report+CUED%2FF%2DINFENG%2FTR%2E1%2C+Cambridge+University+Engineering+Department%2C+1987%2E)
40. Schmidhuber, Jürgen. Reinforcement learning in markovian and non-markovian environments. In Advances in Neural Information Processing Systems 3. Morgan-Kaufmann, 1991. [find](https://scholar.google.com/scholar?q=Schmidhuber%2C+J%C3%BCrgen%2E+Reinforcement+learning+in+markovian+and+non%2Dmarkovian+environments%2E+In+Advances+in+Neural+Information+Processing+Systems+3%2E+Morgan%2DKaufmann%2C+1991%2E)
41. Schmidhuber, Jürgen. Learning complex, extended sequences using the principle of history compression. Neural Computation, 4(2):234–242, 1992. [find](https://scholar.google.com/scholar?q=Schmidhuber%2C+J%C3%BCrgen%2E+Learning+complex%2C+extended+sequences+using+the+principle+of+history+compression%2E+Neural+Computation%2C+4%282%29%3A234%E2%80%93242%2C+1992%2E)
42. Schmidhuber, Jürgen. Deep learning in neural networks: An overview. Neural Networks, 61:85–117, 2015. [find](https://scholar.google.com/scholar?q=Schmidhuber%2C+J%C3%BCrgen%2E+Deep+learning+in+neural+networks%3A+An+overview%2E+Neural+Networks%2C+61%3A85%E2%80%93117%2C+2015%2E)
43. Srivastava, Rupesh K, Greff, Klaus, and Schmidhuber, Juergen. Training very deep networks. In Advances in Neural Information Processing Systems 28, pp. 2368–2376. Curran Associates, Inc., 2015a. [find](https://scholar.google.com/scholar?q=Srivastava%2C+Rupesh+K%2C+Greff%2C+Klaus%2C+and+Schmidhuber%2C+Juergen%2E+Training+very+deep+networks%2E+In+Advances+in+Neural+Information+Processing+Systems+28%2C+pp%2E+2368%E2%80%932376%2E+Curran+Associates%2C+Inc%2E%2C+2015a%2E)
44. Srivastava, Rupesh Kumar, Steunebrink, Bas R, and Schmidhuber, Jürgen. First experiments with powerplay. Neural Networks, 41:130–136, 2013. [find](https://scholar.google.com/scholar?q=Srivastava%2C+Rupesh+Kumar%2C+Steunebrink%2C+Bas+R%2C+and+Schmidhuber%2C+J%C3%BCrgen%2E+First+experiments+with+powerplay%2E+Neural+Networks%2C+41%3A130%E2%80%93136%2C+2013%2E)
45. Srivastava, Rupesh Kumar, Greff, Klaus, and Schmidhuber, Jürgen. [Highway networks](https://md.research.elamur.ai/free/en/papers/srivastava-2015-highway-networks/). arXiv preprint arXiv:1505.00387, 2015b.
46. Werbos, Paul J. System Modeling and Optimization: Proceedings of the 10th IFIP Conference New York City, USA, August 31 – September 4, 1981, chapter Applications of advances in nonlinear sensitivity analysis, pp. 762–770. [find](https://scholar.google.com/scholar?q=Werbos%2C+Paul+J%2E+System+Modeling+and+Optimization%3A+Proceedings+of+the+10th+IFIP+Conference+New+York+City%2C+USA%2C+August+31+%E2%80%93+September+4%2C+1981%2C+chapter+Applications+of+advances+in+nonlinear+sensitivity+a)
47. Springer Berlin Heidelberg, 1982. [find](https://scholar.google.com/scholar?q=Springer+Berlin+Heidelberg%2C+1982%2E)
48. Werbos, Paul J. Generalization of backpropagation with application to a recurrent gas market model. Neural Networks, 1(4): 339–356, 1988. [find](https://scholar.google.com/scholar?q=Werbos%2C+Paul+J%2E+Generalization+of+backpropagation+with+application+to+a+recurrent+gas+market+model%2E+Neural+Networks%2C+1%284%29%3A+339%E2%80%93356%2C+1988%2E)
49. Williams, R. J. Complexity of exact gradient computation algorithms for recurrent neural networks. Technical Report NU-CCS-89-27, Boston: Northeastern University, College of Computer Science, 1989. [find](https://scholar.google.com/scholar?q=Williams%2C+R%2E+J%2E+Complexity+of+exact+gradient+computation+algorithms+for+recurrent+neural+networks%2E+Technical+Report+NU%2DCCS%2D89%2D27%2C+Boston%3A+Northeastern+University%2C+College+of+Computer+Science%2C+1989%2E)
50. Wu, Y., Zhang, S., Zhang, Y., Bengio, Y., and Salakhutdinov, R. On Multiplicative Integration with Recurrent Neural Networks. ArXiv e-prints, June 2016. [find](https://scholar.google.com/scholar?q=Wu%2C+Y%2E%2C+Zhang%2C+S%2E%2C+Zhang%2C+Y%2E%2C+Bengio%2C+Y%2E%2C+and+Salakhutdinov%2C+R%2E+On+Multiplicative+Integration+with+Recurrent+Neural+Networks%2E+ArXiv+e%2Dprints%2C+June+2016%2E)
51. Zaremba, W., Sutskever, I., and Vinyals, O. [Recurrent Neural Network Regularization](https://md.research.elamur.ai/free/en/papers/zaremba-2014-recurrent-neural-network-regularization/). ArXiv e-prints, September 2014.
52. Zhang, Yu, Chen, Guoguo, Yu, Dong, Yao, Kaisheng, Khudanpur, Sanjeev, and Glass, James. Highway long short-term memory RNNS for distant speech recognition. In 2016 IEEE, ICASSP, 2016. [find](https://scholar.google.com/scholar?q=Zhang%2C+Yu%2C+Chen%2C+Guoguo%2C+Yu%2C+Dong%2C+Yao%2C+Kaisheng%2C+Khudanpur%2C+Sanjeev%2C+and+Glass%2C+James%2E+Highway+long+short%2Dterm+memory+RNNS+for+distant+speech+recognition%2E+In+2016+IEEE%2C+ICASSP%2C+2016%2E)
53. Zoph, Barret and Le, Quoc V. [Neural architecture search with reinforcement learning](https://md.research.elamur.ai/free/en/papers/zoph-2016-neural-architecture-search-reinforcement-learning/). arXiv preprint arXiv:1611.01578, 2016. 8. Supplementary Material 8.1. Details of Experimental Setups The following paragraphs describe the precise experimental settings used to obtain results in this paper. The sour…

*Updated 2 November 2026*