---
title: "Thinking Fast and Slow with Deep Learning and Tree Search"
description: "Anthony, Tian and Barber propose Expert Iteration (ExIt), which splits these two jobs. 21 references, cited by 95 papers here."
authors: ["Thomas Anthony", "Zheng Tian", "David Barber"]
year: 2017
venue: "arXiv:1705.08439"
source: "https://arxiv.org/abs/1705.08439"
doi: "10.48550/arxiv.1705.08439"
arxiv: "1705.08439"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/anthony-2017-thinking-fast-slow-deep-learning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/anthony-2017-thinking-fast-slow-deep-learning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/page/2/) / Thinking Fast and Slow with Deep Learning and Tree Search

# Thinking Fast and Slow with Deep Learning and Tree Search

**Authors:** Thomas Anthony, Zheng Tian and David Barber · **Year:** 2017 · **Venue:** arXiv:1705.08439 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1705.08439) · [PDF](https://arxiv.org/pdf/1705.08439) · [DOI 10.48550/arxiv.1705.08439](https://doi.org/10.48550/arxiv.1705.08439) · [arXiv 1705.08439](https://arxiv.org/abs/1705.08439) · [OpenAlex](https://openalex.org/W2618097077)

**Citations:** 186 citations (OpenAlex, 2026-10-14) · [95](https://md.research.elamur.ai/free/en/papers/anthony-2017-thinking-fast-slow-deep-learning/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Deep reinforcement learning methods like REINFORCE and DQN train a network that picks moves with no lookahead. That network has to find good strategies and also generalise them. Anthony, Tian and Barber propose Expert Iteration (ExIt), which splits these two jobs. A tree-search "expert" based on Monte Carlo Tree Search plans strong moves one position at a time. A deep neural network "apprentice" learns to imitate those choices through imitation learning on states from self-play. The improved apprentice then guides the search and helps score positions inside it, which makes the expert stronger, and the cycle repeats. The authors compare this to fast intuition and slow deliberation in dual-process theory. In online and distributed versions, datasets are pooled across iterations and the costly search runs in parallel. The authors trained the system on 9×9 Hex from scratch. ExIt trained a policy network better than a REINFORCE baseline, and its final searcher beat MoHex 1.0 with no hand-coded game knowledge. The work shows that a system can keep improving its own expert instead of needing a strong human or hand-built teacher.

## Key points
- Each ExIt round has the same steps: generate self-play states, label them with expert targets, train a new apprentice, then rebuild the expert around that apprentice. If the expert-improvement step is removed, online ExIt becomes DAgger.
- Search is far slower than network training, so most of the compute goes into generating expert moves. Each plan fits in under 1 KB, so the work is easy to spread across machines even with little bandwidth.
- A combined policy-and-value apprentice clearly beat a policy-only one. That run produced about 7.4 million additional expert move choices after an initial dataset of about 550,000 positions.
- Against MoHex 1.0 (MoHex versions have won every Olympiad Hex event since 2009), ExIt at 10,000 iterations won 75.3% of games against MoHex at 10,000 iterations, 59.3% against MoHex at 100,000 iterations, and 55.6% against MoHex at 4 seconds per move. According to the authors, the training curves had not yet converged.
- The method was developed independently of AlphaGo Zero and is closely related to it. The paper appeared at NIPS 2017.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works

## Cite

Cite as: Thomas Anthony, Zheng Tian and David Barber. 2017. Thinking Fast and Slow with Deep Learning and Tree Search. arXiv:1705.08439.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{anthony2017thinking,
  title = {{Thinking Fast and Slow with Deep Learning and Tree Search}},
  author = {Thomas Anthony, Zheng Tian and David Barber},
  year = {2017},
  doi = {10.48550/arxiv.1705.08439},
  eprint = {1705.08439},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1705.08439}
}
```

- [← previous Best-Worst Scaling More Reliable than Rating Scales: A Case Study on Sentiment Intensity Annotation](https://md.research.elamur.ai/free/en/papers/kiritchenko-2017-best-worst-scaling-more-reliable/)
- [next → Reinforced Mnemonic Reader for Machine Reading Comprehension](https://md.research.elamur.ai/free/en/papers/hu-2017-reinforced-mnemonic-reader-machine-reading/)
- [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/page/2/)

## References

21 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. J. St B. T. Evans. Heuristic and Analytic Processes in Reasoning. British Journal of Psychology, 75(4):451–468, 1984. [find](https://scholar.google.com/scholar?q=J%2E+St+B%2E+T%2E+Evans%2E+Heuristic+and+Analytic+Processes+in+Reasoning%2E+British+Journal+of+Psychology%2C+75%284%29%3A451%E2%80%93468%2C+1984%2E)
2. Daniel Kahneman. Maps of Bounded Rationality: Psychology for Behavioral Economics. The American Economic Review, 93(5):1449–1475, 2003. [find](https://scholar.google.com/scholar?q=Daniel+Kahneman%2E+Maps+of+Bounded+Rationality%3A+Psychology+for+Behavioral+Economics%2E+The+American+Economic+Review%2C+93%285%29%3A1449%E2%80%931475%2C+2003%2E)
3. R. J. Williams. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. Machine Learning, 8(3-4):229–256, 1992. [find](https://scholar.google.com/scholar?q=R%2E+J%2E+Williams%2E+Simple+Statistical+Gradient%2DFollowing+Algorithms+for+Connectionist+Reinforcement+Learning%2E+Machine+Learning%2C+8%283%2D4%29%3A229%E2%80%93256%2C+1992%2E)
4. V. Mnih et al. Human-Level Control through Deep Reinforcement Learning. Nature, 518(7540):529–533, 2015. [find](https://scholar.google.com/scholar?q=V%2E+Mnih+et+al%2E+Human%2DLevel+Control+through+Deep+Reinforcement+Learning%2E+Nature%2C+518%287540%29%3A529%E2%80%93533%2C+2015%2E)
5. S. Ross, G. J. Gordon, and J. A. Bagnell. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. AISTATS, 2011. [find](https://scholar.google.com/scholar?q=S%2E+Ross%2C+G%2E+J%2E+Gordon%2C+and+J%2E+A%2E+Bagnell%2E+A+Reduction+of+Imitation+Learning+and+Structured+Prediction+to+No%2DRegret+Online+Learning%2E+AISTATS%2C+2011%2E)
6. H. Daumé III, J. Langford, and D. Marcu. Search-based Structured Prediction. Machine Learning, 2009. [find](https://scholar.google.com/scholar?q=H%2E+Daum%C3%A9+III%2C+J%2E+Langford%2C+and+D%2E+Marcu%2E+Search%2Dbased+Structured+Prediction%2E+Machine+Learning%2C+2009%2E)
7. L. Kocsis and C. Szepesvári. Bandit Based Monte-Carlo Planning. In European Conference on Machine Learning, pages 282–293. Springer, 2006. [find](https://scholar.google.com/scholar?q=L%2E+Kocsis+and+C%2E+Szepesv%C3%A1ri%2E+Bandit+Based+Monte%2DCarlo+Planning%2E+In+European+Conference+on+Machine+Learning%2C+pages+282%E2%80%93293%2E+Springer%2C+2006%2E)
8. M. Genesereth, N. Love, and B. Pell. General Game Playing: Overview of the AAAI Competition. AI Magazine, 26(2):62, 2005. [find](https://scholar.google.com/scholar?q=M%2E+Genesereth%2C+N%2E+Love%2C+and+B%2E+Pell%2E+General+Game+Playing%3A+Overview+of+the+AAAI+Competition%2E+AI+Magazine%2C+26%282%29%3A62%2C+2005%2E)
9. S. Gelly and D. Silver. Combining Online and Offline Knowledge in UCT. In Proceedings of the 24th International Conference on Machine learning, pages 273–280. ACM, 2007. [find](https://scholar.google.com/scholar?q=S%2E+Gelly+and+D%2E+Silver%2E+Combining+Online+and+Offline+Knowledge+in+UCT%2E+In+Proceedings+of+the+24th+International+Conference+on+Machine+learning%2C+pages+273%E2%80%93280%2E+ACM%2C+2007%2E)
10. D. Kingma and J. Ba. [Adam: A Method for Stochastic Optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). arXiv preprint arXiv:1412.6980, 2014.
11. X. Guo, S. Singh, H. Lee, R. L. Lewis, and X. Wang. Deep Learning for Real-Time Atari Game Play Using Offline Monte-Carlo Rree Search Planning. In Advances in Neural Information Processing Systems, pages 3338–3346, 2014. [find](https://scholar.google.com/scholar?q=X%2E+Guo%2C+S%2E+Singh%2C+H%2E+Lee%2C+R%2E+L%2E+Lewis%2C+and+X%2E+Wang%2E+Deep+Learning+for+Real%2DTime+Atari+Game+Play+Using+Offline+Monte%2DCarlo+Rree+Search+Planning%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pa)
12. D. Silver et al. Mastering the Game of Go with Deep Neural Networks and Tree Search. Nature, 529(7587):484–489, 2016. [find](https://scholar.google.com/scholar?q=D%2E+Silver+et+al%2E+Mastering+the+Game+of+Go+with+Deep+Neural+Networks+and+Tree+Search%2E+Nature%2C+529%287587%29%3A484%E2%80%93489%2C+2016%2E)
13. R Coulom. Bayeselo. [http://remi.coulom.free.fr/Bayesian-Elo/](http://remi.coulom.free.fr/Bayesian-Elo/), 2005. [↗](http://remi.coulom.free.fr/Bayesian-Elo/)
14. B. Arneson, R. Hayward, and P. Hednerson. Monte Carlo Tree Search in Hex. In IEEE Transactions on Computational Intelligence and AI in Games, pages 251–258. IEEE, 2010. [find](https://scholar.google.com/scholar?q=B%2E+Arneson%2C+R%2E+Hayward%2C+and+P%2E+Hednerson%2E+Monte+Carlo+Tree+Search+in+Hex%2E+In+IEEE+Transactions+on+Computational+Intelligence+and+AI+in+Games%2C+pages+251%E2%80%93258%2E+IEEE%2C+2010%2E)
15. S. Ross and J. A. Bagnell. Reinforcement and Imitation Learning via Interactive No-Regret Learning. ArXiv e-prints, 2014. [find](https://scholar.google.com/scholar?q=S%2E+Ross+and+J%2E+A%2E+Bagnell%2E+Reinforcement+and+Imitation+Learning+via+Interactive+No%2DRegret+Learning%2E+ArXiv+e%2Dprints%2C+2014%2E)
16. K. Chang, A. Krishnamurthy, A. Agarwal, H. Daumé III, and J. Langford. Learning to Search Better Than Your Teacher. CoRR, abs/1502.02206, 2015. [find](https://scholar.google.com/scholar?q=K%2E+Chang%2C+A%2E+Krishnamurthy%2C+A%2E+Agarwal%2C+H%2E+Daum%C3%A9+III%2C+and+J%2E+Langford%2E+Learning+to+Search+Better+Than+Your+Teacher%2E+CoRR%2C+abs%2F1502%2E02206%2C+2015%2E)
17. D. Silver et al. Mastering the Game of Go without Human Knowledge. Nature, 550(7676):354– 359, 2017. [find](https://scholar.google.com/scholar?q=D%2E+Silver+et+al%2E+Mastering+the+Game+of+Go+without+Human+Knowledge%2E+Nature%2C+550%287676%29%3A354%E2%80%93+359%2C+2017%2E)
18. K. Young, R. Hayward, and G. Vasan. NeuroHex: A Deep Q-learning Hex Agent. arXiv preprint arXiv:1604.07097, 2016. — [arXiv:1604.07097](https://arxiv.org/abs/1604.07097) [find](https://scholar.google.com/scholar?q=K%2E+Young%2C+R%2E+Hayward%2C+and+G%2E+Vasan%2E+NeuroHex%3A+A+Deep+Q%2Dlearning+Hex+Agent%2E+arXiv+preprint+arXiv%3A1604%2E07097%2C+2016%2E+%E2%80%94+%5BarXiv%3A1604%2E07097%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1604%2E07097%29)
19. Y. Goldberg and J. Nivre. Training Deterministic Parsers with Non-Deterministic Oracles. Transactions of the Association for Computational Linguistics, 1:403–414, 2013. [find](https://scholar.google.com/scholar?q=Y%2E+Goldberg+and+J%2E+Nivre%2E+Training+Deterministic+Parsers+with+Non%2DDeterministic+Oracles%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+1%3A403%E2%80%93414%2C+2013%2E)
20. D. Arpit, Y. Zhou, B. U. Kota, and V. Govindaraju. Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks. arXiv preprint arXiv:1603.01431, 2016. — [arXiv:1603.01431](https://arxiv.org/abs/1603.01431) [find](https://scholar.google.com/scholar?q=D%2E+Arpit%2C+Y%2E+Zhou%2C+B%2E+U%2E+Kota%2C+and+V%2E+Govindaraju%2E+Normalization+Propagation%3A+A+Parametric+Technique+for+Removing+Internal+Covariate+Shift+in+Deep+Networks%2E+arXiv+preprint+arXiv%3A1603%2E01431%2C+2016%2E+%E2%80%94+%5Ba)
21. D.-A. Clevert, T. Unterthiner, and S. Hochreiter. Fast and Accurate Deep Network Learning by Exponential Linear Units(ELUs). CoRR, abs/1511.[07289, 2015](https://md.research.elamur.ai/free/en/papers/clevert-2015-fast-accurate-deep-network-learning/).

*Updated 2 November 2026*