---
title: "Improving Neural Language Models with a Continuous Cache"
description: "Grave, Joulin and Usunier propose the Neural Cache Model, a continuous… 82.3 perplexity. 50 references, cited by 96 papers here."
authors: ["Edouard Grave", "Armand Joulin", "Nicolas Usunier"]
year: 2016
venue: "arXiv:1612.04426"
source: "https://arxiv.org/abs/1612.04426"
doi: "10.48550/arxiv.1612.04426"
arxiv: "1612.04426"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/grave-2016-improving-neural-language-models-continuous/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/grave-2016-improving-neural-language-models-continuous/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/) / Improving Neural Language Models with a Continuous Cache

# Improving Neural Language Models with a Continuous Cache

**Authors:** [Edouard Grave](https://md.research.elamur.ai/free/en/authors/edouard-grave/), [Armand Joulin](https://md.research.elamur.ai/free/en/authors/armand-joulin/) and [Nicolas Usunier](https://md.research.elamur.ai/free/en/authors/nicolas-usunier/) · **Year:** 2016 · **Venue:** arXiv:1612.04426 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1612.04426) · [PDF](https://arxiv.org/pdf/1612.04426) · [DOI 10.48550/arxiv.1612.04426](https://doi.org/10.48550/arxiv.1612.04426) · [arXiv 1612.04426](https://arxiv.org/abs/1612.04426) · [OpenAlex](https://openalex.org/W2571859396)

**Citations:** 83 citations (OpenAlex, 2026-10-14) · [96](https://md.research.elamur.ai/free/en/papers/grave-2016-improving-neural-language-models-continuous/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Standard neural language models such as LSTMs are poor at adjusting to the immediate context of a document. Memory-augmented networks address this, but they need learned read/write mechanisms that are costly and restrict how large the memory can be. Grave, Joulin and Usunier propose the Neural Cache Model, a continuous counterpart of the classic count-based cache. It keeps past hidden states, each paired with the word that followed it. At prediction time, it scores these stored states by their dot product with the current hidden state, producing a distribution over recently seen words. This distribution is merged with the base model's output, either through linear interpolation or a joint softmax. The cache has only a few hyperparameters, which are tuned on validation data. It involves no training and can be attached to any pretrained recurrent model. Because lookup is inexpensive, it handles caches with thousands of entries. On Penn Tree Bank, it substantially lowers the perplexity of an LSTM baseline. This makes it a cheap, practical tool for domain adaptation and for long-range coherence.

## Key points
- **Mechanism:** The memory stores pairs (h_i, x_{i+1}). A temperature-like parameter θ controls how peaked the retrieval distribution is. With θ = 0, the model reduces to a unigram cache (Kuhn & De Mori, 1990).
- **Combining with the base model:** There are two variants. One uses linear interpolation with weight λ. The other uses global normalization, with a bias α applied to cache entries.
- **Penn Tree Bank test results:** The authors' LSTM scores 82.3 perplexity, and adding the neural cache lowers it to 72.1. For comparison, the Pointer Sentinel LSTM reports 70.9 and the Recurrent Highway Network reports 66.0.
- **Penn Tree Bank validation results:** With a cache of size 500, perplexity falls from 86.9 to 74.6 with linear interpolation and to 74.9 with global normalization.
- **Motivation and practicality:** On its own Wikipedia page, "tiger" makes up 2.8% of words, versus 0.0037% across all of Wikipedia. The cache is applied only at test time, avoids backpropagation through long contexts, and can predict OOV words once they have appeared. The paper also evaluates on LAMBADA and other language modeling datasets.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Columbia University](https://md.research.elamur.ai/free/en/universities/columbia-university/) — University, New York, United States · 254 works · 56,747 citations
- [Meta (Israel)](https://md.research.elamur.ai/free/en/companies/meta-israel/) — Company, Tel Aviv, Israel · 46 works · 42,810 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Edouard Grave, Armand Joulin and Nicolas Usunier. 2016. Improving Neural Language Models with a Continuous Cache. arXiv:1612.04426.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{grave2016improving,
  title = {{Improving Neural Language Models with a Continuous Cache}},
  author = {Edouard Grave, Armand Joulin and Nicolas Usunier},
  year = {2016},
  doi = {10.48550/arxiv.1612.04426},
  eprint = {1612.04426},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1612.04426}
}
```

- [← previous Capturing Reliable Fine-Grained Sentiment Associations by Crowdsourcing and Best–Worst Scaling](https://md.research.elamur.ai/free/en/papers/kiritchenko-2016-capturing-reliable-fine-grained-sentiment/)
- [next → Theano: A Python framework for fast computation of mathematical expressions](https://md.research.elamur.ai/free/en/papers/the-theano-development-team-2016-theano-python-framework-fast-computation/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/)

## References

50 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). arXiv preprint arXiv:1409.0473, 2014.
2. Lalit R Bahl, Frederick Jelinek, and Robert L Mercer. A maximum likelihood approach to continuous speech recognition. PAMI, 1983. [find](https://scholar.google.com/scholar?q=Lalit+R+Bahl%2C+Frederick+Jelinek%2C+and+Robert+L+Mercer%2E+A+maximum+likelihood+approach+to+continuous+speech+recognition%2E+PAMI%2C+1983%2E)
3. Jerome R Bellegarda. Exploiting latent semantic information in statistical language modeling. Proceedings of the IEEE, 2000. [find](https://scholar.google.com/scholar?q=Jerome+R+Bellegarda%2E+Exploiting+latent+semantic+information+in+statistical+language+modeling%2E+Proceedings+of+the+IEEE%2C+2000%2E)
4. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. [A neural probabilistic language model](https://md.research.elamur.ai/free/en/papers/bengio-2003-neural-probabilistic-language-model/). JMLR, 2003.
5. Peter F Brown, Vincent J Della Pietra, Stephen A Della Pietra, and Robert L Mercer. The mathematics of statistical machine translation: Parameter estimation. Computational linguistics, 1993. [find](https://scholar.google.com/scholar?q=Peter+F+Brown%2C+Vincent+J+Della+Pietra%2C+Stephen+A+Della+Pietra%2C+and+Robert+L+Mercer%2E+The+mathematics+of+statistical+machine+translation%3A+Parameter+estimation%2E+Computational+linguistics%2C+1993%2E)
6. Danqi Chen, Jason Bolton, and Christopher D Manning. [A thorough examination of the cnn/daily mail reading comprehension task](https://md.research.elamur.ai/free/en/papers/chen-2016-thorough-examination-cnn-daily-mail/). arXiv preprint arXiv:1606.02858, 2016.
7. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014. [find](https://scholar.google.com/scholar?q=Junyoung+Chung%2C+Caglar+Gulcehre%2C+KyungHyun+Cho%2C+and+Yoshua+Bengio%2E+Empirical+evaluation+of+gated+recurrent+neural+networks+on+sequence+modeling%2E+arXiv+preprint+arXiv%3A1412%2E3555%2C+2014%2E)
8. Noah Coccaro and Daniel Jurafsky. Towards better integration of semantic predictors in statistical language modeling. In ICSLP. Citeseer, 1998. [find](https://scholar.google.com/scholar?q=Noah+Coccaro+and+Daniel+Jurafsky%2E+Towards+better+integration+of+semantic+predictors+in+statistical+language+modeling%2E+In+ICSLP%2E+Citeseer%2C+1998%2E)
9. Stephen Della Pietra, Vincent Della Pietra, Robert L Mercer, and Salim Roukos. Adaptive language modeling using minimum discriminant estimation. In Proceedings of the workshop on Speech and Natural Language, 1992. [find](https://scholar.google.com/scholar?q=Stephen+Della+Pietra%2C+Vincent+Della+Pietra%2C+Robert+L+Mercer%2C+and+Salim+Roukos%2E+Adaptive+language+modeling+using+minimum+discriminant+estimation%2E+In+Proceedings+of+the+workshop+on+Speech+and+Natural+La)
10. Jesse Dodge, Andreea Gane, Xiang Zhang, Antoine Bordes, Sumit Chopra, Alexander Miller, Arthur Szlam, and Jason Weston. [Evaluating prerequisite qualities for learning end-to-end dialog systems](https://md.research.elamur.ai/free/en/papers/dodge-2015-evaluating-prerequisite-qualities-learning-end/). arXiv preprint arXiv:1511.06931, 2015.
11. John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. JMLR, 2011. [find](https://scholar.google.com/scholar?q=John+Duchi%2C+Elad+Hazan%2C+and+Yoram+Singer%2E+Adaptive+subgradient+methods+for+online+learning+and+stochastic+optimization%2E+JMLR%2C+2011%2E)
12. Jeffrey L Elman. Finding structure in time. Cognitive science, 1990. [find](https://scholar.google.com/scholar?q=Jeffrey+L+Elman%2E+Finding+structure+in+time%2E+Cognitive+science%2C+1990%2E)
13. Yarin Gal and Zoubin Ghahramani. [A theoretically grounded application of dropout in recurrent neural networks](https://md.research.elamur.ai/free/en/papers/gal-2015-theoretically-grounded-application-dropout-recurrent/). arXiv preprint arXiv:1512.05287, 2015.
14. Joshua T Goodman. A bit of progress in language modeling. Computer Speech & Language, 2001. [find](https://scholar.google.com/scholar?q=Joshua+T+Goodman%2E+A+bit+of+progress+in+language+modeling%2E+Computer+Speech+%26+Language%2C+2001%2E)
15. Edouard Grave, Armand Joulin, Moustapha Cissé, David Grangier, and Hervé Jégou. [Efficient softmax approximation for gpus](https://md.research.elamur.ai/free/en/papers/grave-2016-efficient-softmax-approximation-gpus/). arXiv preprint arXiv:1609.04309, 2016.
16. A. Graves, A. Mohamed, and G. Hinton. Speech recognition with deep recurrent neural networks. In ICASSP, 2013. [find](https://scholar.google.com/scholar?q=A%2E+Graves%2C+A%2E+Mohamed%2C+and+G%2E+Hinton%2E+Speech+recognition+with+deep+recurrent+neural+networks%2E+In+ICASSP%2C+2013%2E)
17. Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014. [find](https://scholar.google.com/scholar?q=Alex+Graves%2C+Greg+Wayne%2C+and+Ivo+Danihelka%2E+Neural+turing+machines%2E+arXiv+preprint+arXiv%3A1410%2E5401%2C+2014%2E)
18. Edward Grefenstette, Karl Moritz Hermann, Mustafa Suleyman, and Phil Blunsom. Learning to transduce with unbounded memory. In Advances in Neural Information Processing Systems, pp. 1828–1836, 2015. [find](https://scholar.google.com/scholar?q=Edward+Grefenstette%2C+Karl+Moritz+Hermann%2C+Mustafa+Suleyman%2C+and+Phil+Blunsom%2E+Learning+to+transduce+with+unbounded+memory%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pp%2E+1828%E2%80%931836%2C+2015%2E)
19. Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, and Yoshua Bengio. [Pointing the unknown words](https://md.research.elamur.ai/free/en/papers/gulcehre-2016-pointing-unknown-words/). arXiv preprint arXiv:1603.08148, 2016.
20. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. [Teaching machines to read and comprehend](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/). In NIPS, 2015.
21. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 1997. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+J%C3%BCrgen+Schmidhuber%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+1997%2E)
22. Rukmini M Iyer and Mari Ostendorf. Modeling long distance dependence in language: Topic mixtures versus dynamic cache models. IEEE Transactions on speech and audio processing, 1999. [find](https://scholar.google.com/scholar?q=Rukmini+M+Iyer+and+Mari+Ostendorf%2E+Modeling+long+distance+dependence+in+language%3A+Topic+mixtures+versus+dynamic+cache+models%2E+IEEE+Transactions+on+speech+and+audio+processing%2C+1999%2E)
23. Frederick Jelinek, Bernard Merialdo, Salim Roukos, and Martin Strauss. A dynamic language model for speech recognition. In HLT, 1991. [find](https://scholar.google.com/scholar?q=Frederick+Jelinek%2C+Bernard+Merialdo%2C+Salim+Roukos%2C+and+Martin+Strauss%2E+A+dynamic+language+model+for+speech+recognition%2E+In+HLT%2C+1991%2E)
24. Armand Joulin and Tomas Mikolov. Inferring algorithmic patterns with stack-augmented recurrent nets. In Advances in Neural Information Processing Systems, pp. 190–198, 2015. [find](https://scholar.google.com/scholar?q=Armand+Joulin+and+Tomas+Mikolov%2E+Inferring+algorithmic+patterns+with+stack%2Daugmented+recurrent+nets%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pp%2E+190%E2%80%93198%2C+2015%2E)
25. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016. [find](https://scholar.google.com/scholar?q=Rafal+Jozefowicz%2C+Oriol+Vinyals%2C+Mike+Schuster%2C+Noam+Shazeer%2C+and+Yonghui+Wu%2E+Exploring+the+limits+of+language+modeling%2E+arXiv+preprint+arXiv%3A1602%2E02410%2C+2016%2E)
26. Rudolf Kadlec, Martin Schmid, Ondrej Bajgar, and Jan Kleindienst. [Text understanding with the attention sum reader network](https://md.research.elamur.ai/free/en/papers/kadlec-2016-text-understanding-attention-sum-reader/). arXiv preprint arXiv:1603.01547, 2016.
27. Slava M Katz. Estimation of probabilities from sparse data for the language model component of a speech recognizer. ICASSP, 1987. [find](https://scholar.google.com/scholar?q=Slava+M+Katz%2E+Estimation+of+probabilities+from+sparse+data+for+the+language+model+component+of+a+speech+recognizer%2E+ICASSP%2C+1987%2E)
28. Sanjeev Khudanpur and Jun Wu. Maximum entropy techniques for exploiting syntactic, semantic and collocational dependencies in language modeling. Computer Speech & Language, 2000. [find](https://scholar.google.com/scholar?q=Sanjeev+Khudanpur+and+Jun+Wu%2E+Maximum+entropy+techniques+for+exploiting+syntactic%2C+semantic+and+collocational+dependencies+in+language+modeling%2E+Computer+Speech+%26+Language%2C+2000%2E)
29. Reinhard Kneser and Hermann Ney. Improved backing-off for m-gram language modeling. In ICASSP, 1995. [find](https://scholar.google.com/scholar?q=Reinhard+Kneser+and+Hermann+Ney%2E+Improved+backing%2Doff+for+m%2Dgram+language+modeling%2E+In+ICASSP%2C+1995%2E)
30. Reinhard Kneser and Volker Steinbiss. On the dynamic adaptation of stochastic language models. In ICASSP, 1993. [find](https://scholar.google.com/scholar?q=Reinhard+Kneser+and+Volker+Steinbiss%2E+On+the+dynamic+adaptation+of+stochastic+language+models%2E+In+ICASSP%2C+1993%2E)
31. Roland Kuhn. Speech recognition and the frequency of recently used words: A modified markov model for natural language. In Proceedings of the 12th conference on Computational linguistics-Volume 1, 1988. [find](https://scholar.google.com/scholar?q=Roland+Kuhn%2E+Speech+recognition+and+the+frequency+of+recently+used+words%3A+A+modified+markov+model+for+natural+language%2E+In+Proceedings+of+the+12th+conference+on+Computational+linguistics%2DVolume+1%2C+198)
32. Roland Kuhn and Renato De Mori. A cache-based natural language model for speech recognition. PAMI, 1990. [find](https://scholar.google.com/scholar?q=Roland+Kuhn+and+Renato+De+Mori%2E+A+cache%2Dbased+natural+language+model+for+speech+recognition%2E+PAMI%2C+1990%2E)
33. Julien Kupiec. Probabilistic models of short and long distance word dependencies in running text. In Proceedings of the workshop on Speech and Natural Language, 1989. [find](https://scholar.google.com/scholar?q=Julien+Kupiec%2E+Probabilistic+models+of+short+and+long+distance+word+dependencies+in+running+text%2E+In+Proceedings+of+the+workshop+on+Speech+and+Natural+Language%2C+1989%2E)
34. Raymond Lau, Ronald Rosenfeld, and Salim Roukos. Trigger-based language models: A maximum entropy approach. In ICASSP, 1993. [find](https://scholar.google.com/scholar?q=Raymond+Lau%2C+Ronald+Rosenfeld%2C+and+Salim+Roukos%2E+Trigger%2Dbased+language+models%3A+A+maximum+entropy+approach%2E+In+ICASSP%2C+1993%2E)
35. Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. [Building a large annotated corpus of english: The penn treebank](https://md.research.elamur.ai/free/en/papers/marcus-1993-building-large-annotated-corpus-english/). Computational linguistics, 1993.
36. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. [Pointer sentinel mixture models](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/). arXiv preprint arXiv:1609.07843, 2016.
37. Tomas Mikolov and Geoffrey Zweig. Context dependent recurrent neural network language model. In SLT, 2012. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov+and+Geoffrey+Zweig%2E+Context+dependent+recurrent+neural+network+language+model%2E+In+SLT%2C+2012%2E)
38. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. Recurrent neural network based language model. In INTERSPEECH, 2010. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Martin+Karafi%C3%A1t%2C+Lukas+Burget%2C+Jan+Cernocky%CC%80%2C+and+Sanjeev+Khudanpur%2E+Recurrent+neural+network+based+language+model%2E+In+INTERSPEECH%2C+2010%2E)
39. Tomas Mikolov, Anoop Deoras, Stefan Kombrink, Lukas Burget, and Jan Cernockỳ. Empirical evaluation and combination of advanced language modeling techniques. In INTERSPEECH, 2011. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Anoop+Deoras%2C+Stefan+Kombrink%2C+Lukas+Burget%2C+and+Jan+Cernocky%CC%80%2E+Empirical+evaluation+and+combination+of+advanced+language+modeling+techniques%2E+In+INTERSPEECH%2C+2011%2E)
40. Tomas Mikolov, Armand Joulin, Sumit Chopra, Michael Mathieu, and Marc’Aurelio Ranzato. Learning longer memory in recurrent neural networks. arXiv preprint arXiv:1412.7753, 2014. — [arXiv:1412.7753](https://arxiv.org/abs/1412.7753) [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Armand+Joulin%2C+Sumit+Chopra%2C+Michael+Mathieu%2C+and+Marc%E2%80%99Aurelio+Ranzato%2E+Learning+longer+memory+in+recurrent+neural+networks%2E+arXiv+preprint+arXiv%3A1412%2E7753%2C+2014%2E+%E2%80%94+%5BarXiv%3A1412%2E7753%5D%28ht)
41. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. [The lambada dataset: Word prediction requiring a broad discourse context](https://md.research.elamur.ai/free/en/papers/paperno-2016-lambada-dataset-word-prediction-requiring/). arXiv preprint arXiv:1606.06031, 2016.
42. Ronald Rosenfeld. A maximum entropy approach to adaptive statistical language modeling. Computer, Speech and Language, 1996. [find](https://scholar.google.com/scholar?q=Ronald+Rosenfeld%2E+A+maximum+entropy+approach+to+adaptive+statistical+language+modeling%2E+Computer%2C+Speech+and+Language%2C+1996%2E)
43. Andreas Stolcke, Noah Coccaro, Rebecca Bates, Paul Taylor, Carol Van Ess-Dykema, Klaus Ries, Elizabeth Shriberg, Daniel Jurafsky, Rachel Martin, and Marie Meteer. [Dialogue act modeling for automatic tagging and recognition of conversational speech](https://md.research.elamur.ai/free/en/papers/stolcke-2000-dialogue-act-modeling-automatic-tagging/). Computational linguistics, 2000.
44. Sainbayar Sukhbaatar, Szlam Arthur, Jason Weston, and Rob Fergus. End-to-end memory networks. In NIPS, 2015. [find](https://scholar.google.com/scholar?q=Sainbayar+Sukhbaatar%2C+Szlam+Arthur%2C+Jason+Weston%2C+and+Rob+Fergus%2E+End%2Dto%2Dend+memory+networks%2E+In+NIPS%2C+2015%2E)
45. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In NIPS, 2015. [find](https://scholar.google.com/scholar?q=Oriol+Vinyals%2C+Meire+Fortunato%2C+and+Navdeep+Jaitly%2E+Pointer+networks%2E+In+NIPS%2C+2015%2E)
46. Tian Wang and Kyunghyun Cho. Larger-context language modelling. arXiv preprint arXiv:1511.03729, 2015. [find](https://scholar.google.com/scholar?q=Tian+Wang+and+Kyunghyun+Cho%2E+Larger%2Dcontext+language+modelling%2E+arXiv+preprint+arXiv%3A1511%2E03729%2C+2015%2E)
47. Paul J Werbos. Backpropagation through time: what it does and how to do it. 1990. [find](https://scholar.google.com/scholar?q=Paul+J+Werbos%2E+Backpropagation+through+time%3A+what+it+does+and+how+to+do+it%2E+1990%2E)
48. Ronald J Williams and Jing Peng. An efficient gradient-based algorithm for on-line training of recurrent network trajectories. Neural computation, 1990. [find](https://scholar.google.com/scholar?q=Ronald+J+Williams+and+Jing+Peng%2E+An+efficient+gradient%2Dbased+algorithm+for+on%2Dline+training+of+recurrent+network+trajectories%2E+Neural+computation%2C+1990%2E)
49. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. [Recurrent neural network regularization](https://md.research.elamur.ai/free/en/papers/zaremba-2014-recurrent-neural-network-regularization/). arXiv preprint arXiv:1409.2329, 2014.
50. Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, and Jürgen Schmidhuber. [Recurrent highway networks](https://md.research.elamur.ai/free/en/papers/zilly-2016-recurrent-highway-networks/). arXiv preprint arXiv:1607.03474, 2016.

*Updated 2 November 2026*