---
title: "Reinforcement Learning Neural Turing Machines - Revised (2015)"
description: "The authors present the RL–NTM, a controller connected to three discrete tapes: input, memory, and… 17 references, cited by 46 papers here."
authors: ["Wojciech Zaremba", "Ilya Sutskever"]
year: 2015
venue: "arXiv:1505.00521"
source: "https://arxiv.org/abs/1505.00521"
doi: "10.48550/arxiv.1505.00521"
arxiv: "1505.00521"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zaremba-2015-reinforcement-learning-neural-turing-machines/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zaremba-2015-reinforcement-learning-neural-turing-machines/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/) / Reinforcement Learning Neural Turing Machines - Revised

# Reinforcement Learning Neural Turing Machines - Revised

**Authors:** [Wojciech Zaremba](https://md.research.elamur.ai/free/en/authors/wojciech-zaremba/) and [Ilya Sutskever](https://md.research.elamur.ai/free/en/authors/ilya-sutskever/) · **Year:** 2015 · **Venue:** arXiv:1505.00521 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1505.00521) · [PDF](https://arxiv.org/pdf/1505.00521) · [DOI 10.48550/arxiv.1505.00521](https://doi.org/10.48550/arxiv.1505.00521) · [arXiv 1505.00521](https://arxiv.org/abs/1505.00521) · [OpenAlex](https://openalex.org/W2204302769)

**Citations:** 93 citations (OpenAlex, 2026-10-14) · [46](https://md.research.elamur.ai/free/en/papers/zaremba-2015-reinforcement-learning-neural-turing-machines/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Neural Turing Machines extend neural networks with external memory, but that memory is continuous and differentiable. Many useful external tools, such as databases or search engines, accept only discrete actions, and the cost of continuous memory access grows with memory size. This paper asks whether a network can learn to operate discrete interfaces. The authors present the RL–NTM, a controller connected to three discrete tapes: input, memory, and write-only output. Head movements and the decision to emit a symbol are trained with the Reinforce algorithm. Memory contents and output predictions are trained with ordinary backpropagation. With an unbounded memory and a self-timed output, the system is Turing complete. On simple algorithmic tasks (reversal, repeated copying, ForwardReverse), the model learns correct solutions. Success depends heavily on the controller architecture. The work also contributes practical tools for Reinforce, including a numerical gradient check and a learned baseline. These matter for future models that must interact with non-differentiable external systems.

## Key points
- The training split is hybrid: Reinforce learns *where* to move the heads on the input and memory tapes (moves of −1, 0 or +1) and *whether* to predict. Backpropagation learns *what* to write to memory and output.
- The total number of steps is capped. Once the remaining steps equal the remaining outputs, the model is forced to emit predictions.
- Controller variants include a plain LSTM, a "direct access" LSTM and a feed-forward network. Results are highly sensitive to this choice. For example, RepeatCopy fails when the input head can only move forward, because the memory pointer then wanders randomly.
- To reduce variance in Reinforce, a separate baseline LSTM first reads the entire input tape and then predicts a per-step baseline. This gives a biased estimator but works well in practice.
- The authors propose a general procedure for numerically checking Reinforce gradients. Code is available at github.com/ilyasu123/rlntm.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 492 works · 440,463 citations
- [Meta](https://md.research.elamur.ai/free/en/companies/meta/) — Company, Menlo Park, United States · 214 works · 241,492 citations
- [New York University](https://md.research.elamur.ai/free/en/universities/new-york-university/) — University, New York, United States · 219 works · 103,487 citations

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🧮 Computational Theory and Mathematics](https://md.research.elamur.ai/free/en/shelves/computer-science/computational-theory-and-mathematics/) · 321 works

## Cite

Cite as: Wojciech Zaremba and Ilya Sutskever. 2015. Reinforcement Learning Neural Turing Machines - Revised. arXiv:1505.00521.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zaremba2015reinforcement,
  title = {{Reinforcement Learning Neural Turing Machines - Revised}},
  author = {Wojciech Zaremba and Ilya Sutskever},
  year = {2015},
  doi = {10.48550/arxiv.1505.00521},
  eprint = {1505.00521},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1505.00521}
}
```

- [← previous Investigating the distribution of some (but not all) implicatures using corpora and web-based methods](https://md.research.elamur.ai/free/en/papers/degen-2015-investigating-distribution-some-but-not/)
- [next → Generating High Quality Proposition Banks for Multilingual Semantic Role Labeling](https://md.research.elamur.ai/free/en/papers/akbik-2015-generating-high-quality-proposition-banks/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/)

## References

17 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Aberdeen, Douglas and Baxter, Jonathan. Scaling internal-state policy-gradient methods for pomdps. In MACHINE LEARNING-INTERNATIONAL WORKSHOP THEN CONFERENCE-, pp. 3–10, 2002. [find](https://scholar.google.com/scholar?q=Aberdeen%2C+Douglas+and+Baxter%2C+Jonathan%2E+Scaling+internal%2Dstate+policy%2Dgradient+methods+for+pomdps%2E+In+MACHINE+LEARNING%2DINTERNATIONAL+WORKSHOP+THEN+CONFERENCE%2D%2C+pp%2E+3%E2%80%9310%2C+2002%2E)
2. Ba, Jimmy, Mnih, Volodymyr, and Kavukcuoglu, Koray. Multiple object recognition with visual attention. arXiv preprint arXiv:1412.7755, 2014. [find](https://scholar.google.com/scholar?q=Ba%2C+Jimmy%2C+Mnih%2C+Volodymyr%2C+and+Kavukcuoglu%2C+Koray%2E+Multiple+object+recognition+with+visual+attention%2E+arXiv+preprint+arXiv%3A1412%2E7755%2C+2014%2E)
3. Bengio, Yoshua, Louradour, Jérôme, Collobert, Ronan, and Weston, Jason. [Curriculum learning](https://md.research.elamur.ai/free/en/papers/bengio-2009-curriculum-learning/). In Proceedings of the 26th annual international conference on machine learning, pp. 41–48. ACM, 2009.
4. Graves, Alex, Wayne, Greg, and Danihelka, Ivo. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014a. [find](https://scholar.google.com/scholar?q=Graves%2C+Alex%2C+Wayne%2C+Greg%2C+and+Danihelka%2C+Ivo%2E+Neural+turing+machines%2E+arXiv+preprint+arXiv%3A1410%2E5401%2C+2014a%2E)
5. Graves, Alex, Wayne, Greg, and Danihelka, Ivo. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014b. [find](https://scholar.google.com/scholar?q=Graves%2C+Alex%2C+Wayne%2C+Greg%2C+and+Danihelka%2C+Ivo%2E+Neural+turing+machines%2E+arXiv+preprint+arXiv%3A1410%2E5401%2C+2014b%2E)
6. Grefenstette, Edward, Hermann, Karl Moritz, Suleyman, Mustafa, and Blunsom, Phil. Learning to transduce with unbounded memory. arXiv preprint arXiv:1506.02516, 2015. [find](https://scholar.google.com/scholar?q=Grefenstette%2C+Edward%2C+Hermann%2C+Karl+Moritz%2C+Suleyman%2C+Mustafa%2C+and+Blunsom%2C+Phil%2E+Learning+to+transduce+with+unbounded+memory%2E+arXiv+preprint+arXiv%3A1506%2E02516%2C+2015%2E)
7. Joulin, Armand and Mikolov, Tomas. Inferring algorithmic patterns with stack-augmented recurrent nets. arXiv preprint arXiv:1503.01007, 2015. [find](https://scholar.google.com/scholar?q=Joulin%2C+Armand+and+Mikolov%2C+Tomas%2E+Inferring+algorithmic+patterns+with+stack%2Daugmented+recurrent+nets%2E+arXiv+preprint+arXiv%3A1503%2E01007%2C+2015%2E)
8. Kohl, Nate and Stone, Peter. Policy gradient reinforcement learning for fast quadrupedal locomotion. In Robotics and Automation, 2004. Proceedings. ICRA’04. 2004 IEEE International Conference on, volume 3, pp. 2619– 2624. [find](https://scholar.google.com/scholar?q=Kohl%2C+Nate+and+Stone%2C+Peter%2E+Policy+gradient+reinforcement+learning+for+fast+quadrupedal+locomotion%2E+In+Robotics+and+Automation%2C+2004%2E+Proceedings%2E+ICRA%E2%80%9904%2E+2004+IEEE+International+Conference+on%2C+volu)
9. Levine, Sergey, Finn, Chelsea, Darrell, Trevor, and Abbeel, Pieter. End-to-end training of deep visuomotor policies. arXiv preprint arXiv:1504.00702, 2015. [find](https://scholar.google.com/scholar?q=Levine%2C+Sergey%2C+Finn%2C+Chelsea%2C+Darrell%2C+Trevor%2C+and+Abbeel%2C+Pieter%2E+End%2Dto%2Dend+training+of+deep+visuomotor+policies%2E+arXiv+preprint+arXiv%3A1504%2E00702%2C+2015%2E)
10. Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Graves, Alex, Antonoglou, Ioannis, Wierstra, Daan, and Riedmiller, Martin. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013. [find](https://scholar.google.com/scholar?q=Mnih%2C+Volodymyr%2C+Kavukcuoglu%2C+Koray%2C+Silver%2C+David%2C+Graves%2C+Alex%2C+Antonoglou%2C+Ioannis%2C+Wierstra%2C+Daan%2C+and+Riedmiller%2C+Martin%2E+Playing+atari+with+deep+reinforcement+learning%2E+arXiv+preprint+arXiv%3A1312)
11. Mnih, Volodymyr, Heess, Nicolas, Graves, Alex, et al. Recurrent models of visual attention. In Advances in Neural Information Processing Systems, pp. 2204–2212, 2014. [find](https://scholar.google.com/scholar?q=Mnih%2C+Volodymyr%2C+Heess%2C+Nicolas%2C+Graves%2C+Alex%2C+et+al%2E+Recurrent+models+of+visual+attention%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pp%2E+2204%E2%80%932212%2C+2014%2E)
12. Peters, Jan and Schaal, Stefan. Policy gradient methods for robotics. In Intelligent Robots and Systems, 2006 IEEE/RSJ International Conference on, pp. 2219–2225. [find](https://scholar.google.com/scholar?q=Peters%2C+Jan+and+Schaal%2C+Stefan%2E+Policy+gradient+methods+for+robotics%2E+In+Intelligent+Robots+and+Systems%2C+2006+IEEE%2FRSJ+International+Conference+on%2C+pp%2E+2219%E2%80%932225%2E)
13. Schmidhuber, Juergen. Self-delimiting neural networks. arXiv preprint arXiv:1210.0118, 2012. — [arXiv:1210.0118](https://arxiv.org/abs/1210.0118) [find](https://scholar.google.com/scholar?q=Schmidhuber%2C+Juergen%2E+Self%2Ddelimiting+neural+networks%2E+arXiv+preprint+arXiv%3A1210%2E0118%2C+2012%2E+%E2%80%94+%5BarXiv%3A1210%2E0118%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1210%2E0118%29)
14. Schmidhuber, Jürgen. Optimal ordered problem solver. Machine Learning, 54(3):211–254, 2004. [find](https://scholar.google.com/scholar?q=Schmidhuber%2C+Ju%CC%88rgen%2E+Optimal+ordered+problem+solver%2E+Machine+Learning%2C+54%283%29%3A211%E2%80%93254%2C+2004%2E)
15. Sukhbaatar, Sainbayar, Szlam, Arthur, Weston, Jason, and Fergus, Rob. Weakly supervised memory networks. arXiv preprint arXiv:1503.08895, 2015. [find](https://scholar.google.com/scholar?q=Sukhbaatar%2C+Sainbayar%2C+Szlam%2C+Arthur%2C+Weston%2C+Jason%2C+and+Fergus%2C+Rob%2E+Weakly+supervised+memory+networks%2E+arXiv+preprint+arXiv%3A1503%2E08895%2C+2015%2E)
16. Williams, Ronald J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229–256, 1992. [find](https://scholar.google.com/scholar?q=Williams%2C+Ronald+J%2E+Simple+statistical+gradient%2Dfollowing+algorithms+for+connectionist+reinforcement+learning%2E+Machine+learning%2C+8%283%2D4%29%3A229%E2%80%93256%2C+1992%2E)
17. Zaremba, Wojciech and Sutskever, Ilya. Learning to execute. arXiv preprint arXiv:1410.4615, 2014. [find](https://scholar.google.com/scholar?q=Zaremba%2C+Wojciech+and+Sutskever%2C+Ilya%2E+Learning+to+execute%2E+arXiv+preprint+arXiv%3A1410%2E4615%2C+2014%2E)

*Updated 2 November 2026*