---
title: "Visualizing and Understanding Recurrent — Karpathy 2015"
description: "LSTM networks perform very well on sequence tasks, but it has been unclear what they learn and why they beat simpler models."
authors: ["Andrej Karpathy", "Justin Johnson", "Li Fei-Fei"]
year: 2015
venue: "arXiv:1506.02078"
source: "https://arxiv.org/abs/1506.02078"
doi: "10.48550/arxiv.1506.02078"
arxiv: "1506.02078"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/karpathy-2015-visualizing-understanding-recurrent-networks/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/karpathy-2015-visualizing-understanding-recurrent-networks/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/) / Visualizing and Understanding Recurrent Networks

# Visualizing and Understanding Recurrent Networks

**Authors:** [Andrej Karpathy](https://md.research.elamur.ai/free/en/authors/andrej-karpathy/), [Justin Johnson](https://md.research.elamur.ai/free/en/authors/justin-johnson/), [Li Fei-Fei](https://md.research.elamur.ai/free/en/authors/li-fei-fei/) · **Year:** 2015 · **Venue:** arXiv:1506.02078 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1506.02078) · [PDF](https://arxiv.org/pdf/1506.02078) · [DOI 10.48550/arxiv.1506.02078](https://doi.org/10.48550/arxiv.1506.02078) · [arXiv 1506.02078](https://arxiv.org/abs/1506.02078) · [OpenAlex](https://openalex.org/W1951216520)

**Citations:** 817 citations (openalex-snapshot · 2026-10-14) · [122](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
LSTM networks perform very well on sequence tasks, but it has been unclear what they learn and why they beat simpler models. Earlier ablation studies judged architectures only by overall test perplexity. Karpathy, Johnson and Fei-Fei use character-level language modeling as a transparent setting for probing this question. They train vanilla RNNs, LSTMs and GRUs (1–3 layers, 64–512 units) on two contrasting corpora. One is Tolstoy's War and Peace, which is nearly plain English prose. The other is the Linux Kernel source code, which is highly structured. Beyond comparing test cross-entropy, they inspect individual memory cells, compare LSTM predictions against finite-context n-gram models, and dissect the remaining mistakes with a sequence of oracles. Some cells reliably track long-range properties such as position within a line, open quotes and bracket depth. Most of the LSTM's advantage over n-grams comes from characters that depend on distant structure. The study gives inspectable evidence of how recurrent memory works on real data and identifies error types worth targeting.

## Key points
- Two corpora were chosen to represent both ends of the structure spectrum:
  - War and Peace (WP) contains 3,258,246 characters.
  - Linux Kernel (LK) is a 6,206,996-character file made from randomly shuffled, concatenated header and source files.
  - WP is split 80/10/10 into train/val/test.
- Penn Treebank and the Hutter Prize Wikipedia data were set aside because they mix ordinary language with markup.
- On test cross-entropy, LSTM and GRU clearly beat the vanilla RNN.
  - The best WP result is about 1.077 (3-layer GRU, 512 units); a 3-layer LSTM reaches 1.082.
  - The best LK result is 0.829 (3-layer GRU); a 2-layer LSTM reaches 0.840.
- Training used the following settings:
  - RMSProp with learning rate 2×10⁻³ and batch size 100.
  - Unrolling over 100 time steps and up to 50 epochs.
  - Uniform initialization in [−0.08, 0.08], early stopping, and per-model tuned dropout.
- Interpretable LSTM cells emerge without supervision. They count line length and respond to quotes and brackets.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

122 papers of our library cite it (of the 140673 whose bibliographies we read); 21 of them have a page here, 101 not yet.

- [Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks](https://md.research.elamur.ai/free/en/papers/adi-2016-fine-grained-analysis-sentence-embeddings/)
- [Contextual String Embeddings for Sequence Labeling](https://md.research.elamur.ai/free/en/papers/akbik-2018-contextual-string-embeddings-sequence-labeling/)
- [Interpreting Neural Networks to Improve Politeness Comprehension](https://md.research.elamur.ai/free/en/papers/aubakirova-2016-interpreting-neural-networks-improve-politeness/)
- [What do Neural Machine Translation Models Learn about Morphology?](https://md.research.elamur.ai/free/en/papers/belinkov-2017-what-do-neural-machine-translation/)
- [Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks](https://md.research.elamur.ai/free/en/papers/belinkov-2018-evaluating-layers-representation-neural-machine/)
- [Analysis Methods in Neural Language Processing: A Survey](https://md.research.elamur.ai/free/en/papers/belinkov-2019-analysis-methods-neural-language-processing/)
- [Recurrent Attention Network on Memory for Aspect Sentiment Analysis](https://md.research.elamur.ai/free/en/papers/chen-2017-recurrent-attention-network-memory-aspect/)
- [The emergence of number and syntax units in LSTM language models](https://md.research.elamur.ai/free/en/papers/lakretz-2019-emergence-number-syntax-units-lstm/)
- [Visualizing and Understanding Neural Models in NLP](https://md.research.elamur.ai/free/en/papers/li-2016-visualizing-understanding-neural-models-nlp/)
- [Open Sesame: Getting inside BERT's Linguistic Knowledge](https://md.research.elamur.ai/free/en/papers/lin-2019-open-sesame-getting-inside-bert/)
- [Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies](https://md.research.elamur.ai/free/en/papers/linzen-2016-assessing-ability-lstms-learn-syntax/)
- [Linguistic Knowledge and Transferability of Contextual Representations](https://md.research.elamur.ai/free/en/papers/liu-2019-linguistic-knowledge-transferability-contextual-representations/)
- [Towards Faithful Model Explanation in NLP: A Survey](https://md.research.elamur.ai/free/en/papers/lyu-2024-faithful-model-explanation-nlp-survey/)
- [Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations](https://md.research.elamur.ai/free/en/papers/narrative-2016-kiperwasser-simple-accurate-dependency-parsing-bidirectional/)
- [Semi-supervised sequence tagging with bidirectional language models](https://md.research.elamur.ai/free/en/papers/peters-2017-semi-supervised-sequence-tagging-bidirectional/)
- [Dissecting Contextual Word Embeddings: Architecture and Representation](https://md.research.elamur.ai/free/en/papers/peters-2018-dissecting-contextual-word-embeddings-architecture/)
- [Learning to Generate Reviews and Discovering Sentiment](https://md.research.elamur.ai/free/en/papers/radford-2017-learning-generate-reviews-discovering-sentiment/)
- [An Analysis of Encoder Representations in Transformer-Based Machine Translation](https://md.research.elamur.ai/free/en/papers/raganato-2018-analysis-encoder-representations-transformer-based/)
- [Does String-Based Neural MT Learn Source Syntax?](https://md.research.elamur.ai/free/en/papers/shi-2016-does-string-based-neural-mt/)
- [Information-Theoretic Probing with Minimum Description Length](https://md.research.elamur.ai/free/en/papers/voita-2020-information-theoretic-probing-minimum-description/)
- [Inference is Everything: Recasting Semantic Resources into a Unified Evaluation Framework](https://md.research.elamur.ai/free/en/papers/white-2017-inference-is-everything-recasting-semantic/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{karpathy2015visualizing,
  title = {{Visualizing and Understanding Recurrent Networks}},
  author = {Karpathy, Andrej and Johnson, Justin and Fei-Fei, Li},
  year = {2015},
  doi = {10.48550/arxiv.1506.02078},
  eprint = {1506.02078},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1506.02078}
}
```

- [← previous Visual7W: Grounded Question Answering in Images](https://md.research.elamur.ai/free/en/papers/zhu-2015-visual7w-grounded-question-answering-images/)
- [next → WikiQA: A Challenge Dataset for Open-Domain Question Answering](https://md.research.elamur.ai/free/en/papers/yang-2015-wikiqa-challenge-dataset-open-domain/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/)

## References

35 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio, Yoshua. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014. [find](https://scholar.google.com/scholar?q=Bahdanau%2C+Dzmitry%2C+Cho%2C+Kyunghyun%2C+and+Bengio%2C+Yoshua%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+arXiv+preprint+arXiv%3A1409%2E0473%2C+2014%2E) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
2. Bengio, Yoshua, Simard, Patrice, and Frasconi, Paolo. Learning long-term dependencies with gradient descent is difficult. Neural Networks, IEEE Transactions on, 5(2):157–166, 1994. [find](https://scholar.google.com/scholar?q=Bengio%2C+Yoshua%2C+Simard%2C+Patrice%2C+and+Frasconi%2C+Paolo%2E+Learning+long%2Dterm+dependencies+with+gradient+descent+is+difficult%2E+Neural+Networks%2C+IEEE+Transactions+on%2C+5%282%29%3A157%E2%80%93166%2C+1994%2E)
3. Chen, Stanley F and Goodman, Joshua. An empirical study of smoothing techniques for language modeling. Computer Speech & Language, 13(4):359–393, 1999. [find](https://scholar.google.com/scholar?q=Chen%2C+Stanley+F+and+Goodman%2C+Joshua%2E+An+empirical+study+of+smoothing+techniques+for+language+modeling%2E+Computer+Speech+%26+Language%2C+13%284%29%3A359%E2%80%93393%2C+1999%2E)
4. Cho, Kyunghyun, van Merriënboer, Bart, Bahdanau, Dzmitry, and Bengio, Yoshua. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259, 2014. [find](https://scholar.google.com/scholar?q=Cho%2C+Kyunghyun%2C+van+Merrie%CC%88nboer%2C+Bart%2C+Bahdanau%2C+Dzmitry%2C+and+Bengio%2C+Yoshua%2E+On+the+properties+of+neural+machine+translation%3A+Encoder%2Ddecoder+approaches%2E+arXiv+preprint+arXiv%3A1409%2E1259%2C+2014%2E) [our card](https://md.research.elamur.ai/free/en/papers/cho-2014-properties-neural-machine-translation-encoder/)
5. Chung, Junyoung, Gulcehre, Caglar, Cho, KyungHyun, and Bengio, Yoshua. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014. [find](https://scholar.google.com/scholar?q=Chung%2C+Junyoung%2C+Gulcehre%2C+Caglar%2C+Cho%2C+KyungHyun%2C+and+Bengio%2C+Yoshua%2E+Empirical+evaluation+of+gated+recurrent+neural+networks+on+sequence+modeling%2E+arXiv+preprint+arXiv%3A1412%2E3555%2C+2014%2E)
6. Dai, Andrew M and Le, Quoc V. arXiv:1511.01432, 2015. — [arXiv:1511.01432](https://arxiv.org/abs/1511.01432) [find](https://scholar.google.com/scholar?q=Dai%2C+Andrew+M+and+Le%2C+Quoc+V%2E+arXiv%3A1511%2E01432%2C+2015%2E+%E2%80%94+%5BarXiv%3A1511%2E01432%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1511%2E01432%29)
7. Dauphin, Yann N, de Vries, Harm, Chung, Junyoung, and Bengio, Yoshua. Rmsprop and equi-librated adaptive learning rates for non-convex optimization. arXiv preprint arXiv:1502.04390, 2015. — [arXiv:1502.04390](https://arxiv.org/abs/1502.04390) [find](https://scholar.google.com/scholar?q=Dauphin%2C+Yann+N%2C+de+Vries%2C+Harm%2C+Chung%2C+Junyoung%2C+and+Bengio%2C+Yoshua%2E+Rmsprop+and+equi%2Dlibrated+adaptive+learning+rates+for+non%2Dconvex+optimization%2E+arXiv+preprint+arXiv%3A1502%2E04390%2C+2015%2E+%E2%80%94+%5BarXiv%3A150)
8. Donahue, Jeff, Hendricks, Lisa Anne, Guadarrama, Sergio, Rohrbach, Marcus, Venugopalan, Subhashini, Saenko, Kate, and Darrell, Trevor. Long-term recurrent convolutional networks for visual recognition and description. CVPR, 2015. [find](https://scholar.google.com/scholar?q=Donahue%2C+Jeff%2C+Hendricks%2C+Lisa+Anne%2C+Guadarrama%2C+Sergio%2C+Rohrbach%2C+Marcus%2C+Venugopalan%2C+Subhashini%2C+Saenko%2C+Kate%2C+and+Darrell%2C+Trevor%2E+Long%2Dterm+recurrent+convolutional+networks+for+visual+recognition) [our card](https://md.research.elamur.ai/free/en/papers/donahue-2014-long-term-recurrent-convolutional-networks/)
9. Graves, Alex. Generating sequences with recurrent neural networks. arXiv:1308.0850, 2013. [find](https://scholar.google.com/scholar?q=Graves%2C+Alex%2E+Generating+sequences+with+recurrent+neural+networks%2E+arXiv%3A1308%2E0850%2C+2013%2E)
10. Graves, Alex, Mohamed, A-R, and Hinton, Geoffrey. Speech recognition with deep recurrent neural networks. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pp. 6645–6649. [find](https://scholar.google.com/scholar?q=Graves%2C+Alex%2C+Mohamed%2C+A%2DR%2C+and+Hinton%2C+Geoffrey%2E+Speech+recognition+with+deep+recurrent+neural+networks%2E+In+Acoustics%2C+Speech+and+Signal+Processing+%28ICASSP%29%2C+2013+IEEE+International+Conference+on%2C+pp)
11. Graves, Alex, Wayne, Greg, and Danihelka, Ivo. arXiv:1410.5401, 2014. [find](https://scholar.google.com/scholar?q=Graves%2C+Alex%2C+Wayne%2C+Greg%2C+and+Danihelka%2C+Ivo%2E+arXiv%3A1410%2E5401%2C+2014%2E)
12. Greff, Klaus, Srivastava, Rupesh Kumar, Koutnı́k, Jan, Steunebrink, Bas R., and Schmidhuber, Jürgen. LSTM: A search space odyssey. CoRR, abs/1503.04069, 2015. URL http://arxiv.org/abs/1503.04069. [↗](http://arxiv.org/abs/1503.04069)
13. Heafield, Kenneth, Pouzyrevsky, Ivan, Clark, Jonathan H., and Koehn, Philipp. Scalable modified Kneser-Ney language model estimation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pp. 690–696, Sofia, Bulgaria, August 2013. URL [http://kheafield.com/profes…](http://kheafield.com/profes…) [↗](http://kheafield.com/profes…) [our card](https://md.research.elamur.ai/free/en/papers/heafield-2013-scalable-modified-kneser-ney-language/)
14. Hermans, Michiel and Schrauwen, Benjamin. Training and analysing deep recurrent neural networks. In Advances in Neural Information Processing Systems, pp. 190–198, 2013. [find](https://scholar.google.com/scholar?q=Hermans%2C+Michiel+and+Schrauwen%2C+Benjamin%2E+Training+and+analysing+deep+recurrent+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pp%2E+190%E2%80%93198%2C+2013%2E)
15. Hochreiter, Sepp and Schmidhuber, Jürgen. Long short-term memory. Neural computation, 9(8): 1735–1780, 1997. [find](https://scholar.google.com/scholar?q=Hochreiter%2C+Sepp+and+Schmidhuber%2C+Ju%CC%88rgen%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A+1735%E2%80%931780%2C+1997%2E)
16. Hoiem, Derek, Chodpathumwan, Yodsawalai, and Dai, Qieyun. Diagnosing error in object detectors. In Computer Vision–ECCV 2012, pp. 340–353. [find](https://scholar.google.com/scholar?q=Hoiem%2C+Derek%2C+Chodpathumwan%2C+Yodsawalai%2C+and+Dai%2C+Qieyun%2E+Diagnosing+error+in+object+detectors%2E+In+Computer+Vision%E2%80%93ECCV+2012%2C+pp%2E+340%E2%80%93353%2E)
17. Huang, Xuedong, Acero, Alex, Hon, Hsiao-Wuen, and Foreword By-Reddy, Raj. Spoken language processing: A guide to theory, algorithm, and system development. Prentice Hall PTR, 2001. [find](https://scholar.google.com/scholar?q=Huang%2C+Xuedong%2C+Acero%2C+Alex%2C+Hon%2C+Hsiao%2DWuen%2C+and+Foreword+By%2DReddy%2C+Raj%2E+Spoken+language+processing%3A+A+guide+to+theory%2C+algorithm%2C+and+system+development%2E+Prentice+Hall+PTR%2C+2001%2E)
18. Hutter, Marcus. The human knowledge compression contest. 2012. [find](https://scholar.google.com/scholar?q=Hutter%2C+Marcus%2E+The+human+knowledge+compression+contest%2E+2012%2E)
19. Jelinek, Frederick, Merialdo, Bernard, Roukos, Salim, and Strauss, Martin. A dynamic language model for speech recognition. In HLT, volume 91, pp. 293–295, 1991. [find](https://scholar.google.com/scholar?q=Jelinek%2C+Frederick%2C+Merialdo%2C+Bernard%2C+Roukos%2C+Salim%2C+and+Strauss%2C+Martin%2E+A+dynamic+language+model+for+speech+recognition%2E+In+HLT%2C+volume+91%2C+pp%2E+293%E2%80%93295%2C+1991%2E)
20. Semi-supervised sequence learning. arXiv preprint arXiv preprint Neural turing machines. arXiv preprint Joulin, Armand and Mikolov, Tomas. Inferring algorithmic patterns with stack-augmented recurrent nets. CoRR, abs/1503.01007, 2015. URL http://arxiv.org/abs/1503.01007. [↗](http://arxiv.org/abs/1503.01007)
21. Jozefowicz, Rafal, Zaremba, Wojciech, and Sutskever, Ilya. An empirical exploration of recurrent network architectures. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pp. 2342–2350, 2015. [find](https://scholar.google.com/scholar?q=Jozefowicz%2C+Rafal%2C+Zaremba%2C+Wojciech%2C+and+Sutskever%2C+Ilya%2E+An+empirical+exploration+of+recurrent+network+architectures%2E+In+Proceedings+of+the+32nd+International+Conference+on+Machine+Learning+%28ICML%2D15)
22. Karpathy, Andrej and Fei-Fei, Li. Deep visual-semantic alignments for generating image descriptions. CVPR, 2015. [find](https://scholar.google.com/scholar?q=Karpathy%2C+Andrej+and+Fei%2DFei%2C+Li%2E+Deep+visual%2Dsemantic+alignments+for+generating+image+descriptions%2E+CVPR%2C+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/karpathy-2014-deep-visual-semantic-alignments-generating/)
23. Marcus, Mitchell P, Marcinkiewicz, Mary Ann, and Santorini, Beatrice. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313–330, 1993. [find](https://scholar.google.com/scholar?q=Marcus%2C+Mitchell+P%2C+Marcinkiewicz%2C+Mary+Ann%2C+and+Santorini%2C+Beatrice%2E+Building+a+large+annotated+corpus+of+english%3A+The+penn+treebank%2E+Computational+linguistics%2C+19%282%29%3A313%E2%80%93330%2C+1993%2E) [our card](https://md.research.elamur.ai/free/en/papers/bank-1993-marcus-building-large-annotated-corpus-english/)
24. Mikolov, Tomáš. Statistical language models based on neural networks. Presentation at Google, Mountain View, 2nd April, 2012. [find](https://scholar.google.com/scholar?q=Mikolov%2C+Toma%CC%81s%CC%8C%2E+Statistical+language+models+based+on+neural+networks%2E+Presentation+at+Google%2C+Mountain+View%2C+2nd+April%2C+2012%2E)
25. Mikolov, Tomas, Karafiát, Martin, Burget, Lukas, Cernockỳ, Jan, and Khudanpur, Sanjeev. Recurrent neural network based language model. In INTERSPEECH 2010, 11th Annual Conference of the International Speech Communication Association, Makuhari, Chiba, Japan, September 26-30, 2010, pp. 1045–1048, 20… [find](https://scholar.google.com/scholar?q=Mikolov%2C+Tomas%2C+Karafia%CC%81t%2C+Martin%2C+Burget%2C+Lukas%2C+Cernocky%CC%80%2C+Jan%2C+and+Khudanpur%2C+Sanjeev%2E+Recurrent+neural+network+based+language+model%2E+In+INTERSPEECH+2010%2C+11th+Annual+Conference+of+the+Internationa)
26. Pascanu, Razvan, Mikolov, Tomas, and Bengio, Yoshua. On the difficulty of training recurrent neural networks. arXiv preprint arXiv:1211.5063, 2012. [find](https://scholar.google.com/scholar?q=Pascanu%2C+Razvan%2C+Mikolov%2C+Tomas%2C+and+Bengio%2C+Yoshua%2E+On+the+difficulty+of+training+recurrent+neural+networks%2E+arXiv+preprint+arXiv%3A1211%2E5063%2C+2012%2E)
27. Pascanu, Razvan, Gülçehre, Çaglar, Cho, Kyunghyun, and Bengio, Yoshua. How to construct deep recurrent neural networks. CoRR, abs/1312.6026, 2013. URL http://arxiv.org/abs/1312.6026. [↗](http://arxiv.org/abs/1312.6026)
28. Rumelhart, David E, Hinton, Geoffrey E, and Williams, Ronald J. Learning internal representations by error propagation. Technical report, DTIC Document, 1985. [find](https://scholar.google.com/scholar?q=Rumelhart%2C+David+E%2C+Hinton%2C+Geoffrey+E%2C+and+Williams%2C+Ronald+J%2E+Learning+internal+representations+by+error+propagation%2E+Technical+report%2C+DTIC+Document%2C+1985%2E)
29. Schmidhuber, J. Deep learning in neural networks: An overview. Neural Networks, 61:85–117, 2015. doi: 10.1016/j.neunet.2014.09.003. Published online 2014; based on TR arXiv:1404.7828 [cs.NE]. — [doi:10.1016/j.neunet.2014.09.003](https://doi.org/10.1016/j.neunet.2014.09.003) [find](https://scholar.google.com/scholar?q=Schmidhuber%2C+J%2E+Deep+learning+in+neural+networks%3A+An+overview%2E+Neural+Networks%2C+61%3A85%E2%80%93117%2C+2015%2E+doi%3A+10%2E1016%2Fj%2Eneunet%2E2014%2E09%2E003%2E+Published+online+2014%3B+based+on+TR+arXiv%3A1404%2E7828+%5Bcs%2ENE%5D%2E+%E2%80%94+%5Bdoi%3A1)
30. Sutskever, Ilya, Martens, James, and Hinton, Geoffrey E. Generating text with recurrent neural networks. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 1017–1024, 2011. [find](https://scholar.google.com/scholar?q=Sutskever%2C+Ilya%2C+Martens%2C+James%2C+and+Hinton%2C+Geoffrey+E%2E+Generating+text+with+recurrent+neural+networks%2E+In+Proceedings+of+the+28th+International+Conference+on+Machine+Learning+%28ICML%2D11%29%2C+pp%2E+1017%E2%80%93102)
31. Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc VV. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pp. 3104–3112, 2014. [find](https://scholar.google.com/scholar?q=Sutskever%2C+Ilya%2C+Vinyals%2C+Oriol%2C+and+Le%2C+Quoc+VV%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pp%2E+3104%E2%80%933112%2C+2014%2E)
32. Van der Maaten, Laurens and Hinton, Geoffrey. Visualizing data using t-sne. Journal of Machine Learning Research, 9(2579-2605):85, 2008. [find](https://scholar.google.com/scholar?q=Van+der+Maaten%2C+Laurens+and+Hinton%2C+Geoffrey%2E+Visualizing+data+using+t%2Dsne%2E+Journal+of+Machine+Learning+Research%2C+9%282579%2D2605%29%3A85%2C+2008%2E)
33. Vinyals, Oriol, Toshev, Alexander, Bengio, Samy, and Erhan, Dumitru. Show and tell: A neural image caption generator. CVPR, 2015. [find](https://scholar.google.com/scholar?q=Vinyals%2C+Oriol%2C+Toshev%2C+Alexander%2C+Bengio%2C+Samy%2C+and+Erhan%2C+Dumitru%2E+Show+and+tell%3A+A+neural+image+caption+generator%2E+CVPR%2C+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/vinyals-2014-show-tell-neural-image-caption/)
34. Werbos, Paul J. Generalization of backpropagation with application to a recurrent gas market model. Neural Networks, 1(4):339–356, 1988. [find](https://scholar.google.com/scholar?q=Werbos%2C+Paul+J%2E+Generalization+of+backpropagation+with+application+to+a+recurrent+gas+market+model%2E+Neural+Networks%2C+1%284%29%3A339%E2%80%93356%2C+1988%2E)
35. Weston, Jason, Chopra, Sumit, and Bordes, Antoine. Memory networks. CoRR, abs/1410.3916, 2014. URL http://arxiv.org/abs/1410.3916. [↗](http://arxiv.org/abs/1410.3916)