---
title: "AI safety via debate — paper by Irving et al. (2018)"
description: "Irving, Christiano and Amodei (OpenAI) propose debate as a remedy. 33 references, cited by 81 papers here."
authors: ["Geoffrey Irving", "Paul Christiano", "Dario Amodei"]
year: 2018
venue: "arXiv:1805.00899"
source: "https://arxiv.org/abs/1805.00899"
doi: "10.48550/arxiv.1805.00899"
arxiv: "1805.00899"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/irving-2018-ai-safety-debate/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/irving-2018-ai-safety-debate/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/5/) / AI safety via debate

# AI safety via debate

**Authors:** [Geoffrey Irving](https://md.research.elamur.ai/free/en/authors/geoffrey-irving/), [Paul Christiano](https://md.research.elamur.ai/free/en/authors/paul-christiano/) and [Dario Amodei](https://md.research.elamur.ai/free/en/authors/dario-amodei/) · **Year:** 2018 · **Venue:** arXiv:1805.00899 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1805.00899) · [PDF](https://arxiv.org/pdf/1805.00899) · [DOI 10.48550/arxiv.1805.00899](https://doi.org/10.48550/arxiv.1805.00899) · [arXiv 1805.00899](https://arxiv.org/abs/1805.00899) · [OpenAlex](https://openalex.org/W2798877128)

**Citations:** 29 citations (OpenAlex, 2026-10-14) · [81](https://md.research.elamur.ai/free/en/papers/irving-2018-ai-safety-debate/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Teaching AI systems complex human goals often depends on people rating the system's behavior. That breaks down once a task is too hard for a person to evaluate directly. Irving, Christiano and Amodei (OpenAI) propose debate as a remedy. Two agents see a question and each commits to an answer. They then trade short statements up to a fixed limit, and a human decides which side supplied the most truthful, useful information. Because the game is zero-sum, agents can be trained through self play, as AlphaZero is in Go. The approach rests on one bet: exposing a falsehood should be easier than defending it. Using complexity theory, the authors argue that ideal debate lets a polynomial-time judge resolve PSPACE questions, whereas direct judging reaches only NP. A first MNIST experiment had debaters compete to convince a sparse classifier, and this substantially raised its accuracy. The paper matters as a framework for overseeing systems that may outgrow their supervisors. Most empirical validation, especially in natural language, is left for future work.

## Key points
- Debate protocol: a question goes to both agents, they give answers, take turns making statements, and a judge picks the winner. At test time, only the answers are needed.
- Theory: with optimal play, debate corresponds to PSPACE, versus NP for direct human judging. The model assumes error-free judges and agents with unlimited compute.
- MNIST result: the sparse classifier's accuracy rose from 59.4% to 88.9% with 6 pixels and from 48.2% to 85.2% with 4 pixels.
- Debate is closely related to amplification (Christiano et al., 2018). Debate copes better with high branching factors, and the two approaches can be hybridized.
- Open directions include human experiments on value judgments and bias, natural-language debate, and combining debate with other safety methods.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,954 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 626 works

🧭 Topics (1): [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 587 works

## Cite

Cite as: Geoffrey Irving, Paul Christiano and Dario Amodei. 2018. AI safety via debate. arXiv:1805.00899.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{irving2018ai,
  title = {{AI safety via debate}},
  author = {Geoffrey Irving, Paul Christiano and Dario Amodei},
  year = {2018},
  doi = {10.48550/arxiv.1805.00899},
  eprint = {1805.00899},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1805.00899}
}
```

- [← previous You Tweet What You Speak: A City-Level Dataset of Arabic Dialects](https://md.research.elamur.ai/free/en/papers/mageed-2018-you-tweet-what-you-speak/)
- [next → Pragmatically Informative Image Captioning with Character-Level Inference](https://md.research.elamur.ai/free/en/papers/gordon-2018-pragmatically-informative-image-captioning-character/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/5/)

## References

33 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Stuart J. Russell, Daniel Dewey, and Max Tegmark. Research priorities for robust and beneficial artificial intelligence. CoRR, abs/1602.03506, 2016. URL https://arxiv.org/abs/1602.03506. — [arXiv:1602.03506](https://arxiv.org/abs/1602.03506) [↗](https://arxiv.org/abs/1602.03506)
2. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dandelion Mané. [Concrete problems in AI safety](https://md.research.elamur.ai/free/en/papers/amodei-2016-concrete-problems-ai-safety/). CoRR, abs/1606.06565, 2016. URL https://arxiv.org/abs/1606.06565. [↗](https://arxiv.org/abs/1606.06565)
3. Shira Mitchell and Jackie Shadlen. Mirror mirror: Reflections on quantitative fairness. [https://speak-statistics-to-power.github.io/fairness](https://speak-statistics-to-power.github.io/fairness), 2018. [↗](https://speak-statistics-to-power.github.io/fairness)
4. Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, pages 4302–4310, 2017. [find](https://scholar.google.com/scholar?q=Paul+Christiano%2C+Jan+Leike%2C+Tom+Brown%2C+Miljan+Martic%2C+Shane+Legg%2C+and+Dario+Amodei%2E+Deep+reinforcement+learning+from+human+preferences%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+4302)
5. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484–489, 2016. [find](https://scholar.google.com/scholar?q=David+Silver%2C+Aja+Huang%2C+Chris+J+Maddison%2C+Arthur+Guez%2C+Laurent+Sifre%2C+George+Van+Den+Driessche%2C+Julian+Schrittwieser%2C+Ioannis+Antonoglou%2C+Veda+Panneershelvam%2C+Marc+Lanctot%2C+et+al%2E+Mastering+the+game+)
6. David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of Go without human knowledge. Nature, 550(7676):354, 2017a. [find](https://scholar.google.com/scholar?q=David+Silver%2C+Julian+Schrittwieser%2C+Karen+Simonyan%2C+Ioannis+Antonoglou%2C+Aja+Huang%2C+Arthur+Guez%2C+Thomas+Hubert%2C+Lucas+Baker%2C+Matthew+Lai%2C+Adrian+Bolton%2C+et+al%2E+Mastering+the+game+of+Go+without+human+kn)
7. David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. [Mastering chess and shogi by self-play with a general reinforcement learning algorithm](https://md.research.elamur.ai/free/en/papers/al-2017-mastering-chess-shogi-self-play/). arXiv preprint arXiv:1712.01815, 2017b.
8. OpenAI. More on Dota 2. [https://blog.openai.com/more-on-dota-2](https://blog.openai.com/more-on-dota-2), 2017. [↗](https://blog.openai.com/more-on-dota-2)
9. Paul Christiano, Buck Shlegeris, and Dario Amodei. Supervising strong learners by amplifying weak experts. arXiv preprint arXiv:1810.08575, 2018. [find](https://scholar.google.com/scholar?q=Paul+Christiano%2C+Buck+Shlegeris%2C+and+Dario+Amodei%2E+Supervising+strong+learners+by+amplifying+weak+experts%2E+arXiv+preprint+arXiv%3A1810%2E08575%2C+2018%2E)
10. Ryan Lowe, Michael Noseworthy, Iulian V Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. [Towards an automatic Turing test: Learning to evaluate dialogue responses](https://md.research.elamur.ai/free/en/papers/lowe-2017-towards-automatic-turing-test-learning/). arXiv preprint arXiv:1708.07149, 2017a.
11. Michael Sipser. Introduction to the Theory of Computation. Course Technology, Boston, MA, third edition, 2013. ISBN 113318779X. [find](https://scholar.google.com/scholar?q=Michael+Sipser%2E+Introduction+to+the+Theory+of+Computation%2E+Course+Technology%2C+Boston%2C+MA%2C+third+edition%2C+2013%2E+ISBN+113318779X%2E)
12. Jeffrey C Lagarias and Andrew M. Odlyzko. Computing π(x): An analytic method. Journal of Algorithms, 8(2):173–191, 1987. [find](https://scholar.google.com/scholar?q=Jeffrey+C+Lagarias+and+Andrew+M%2E+Odlyzko%2E+Computing+%CF%80%28x%29%3A+An+analytic+method%2E+Journal+of+Algorithms%2C+8%282%29%3A173%E2%80%93191%2C+1987%2E)
13. Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. In NIPS 2017 Workshop on Meta-Learning, 2017. [find](https://scholar.google.com/scholar?q=Nikhil+Mishra%2C+Mostafa+Rohaninejad%2C+Xi+Chen%2C+and+Pieter+Abbeel%2E+A+simple+neural+attentive+meta%2Dlearner%2E+In+NIPS+2017+Workshop+on+Meta%2DLearning%2C+2017%2E)
14. Smitha Milli, Pieter Abbeel, and Igor Mordatch. Interpretable and pedagogical examples. arXiv preprint arXiv:1711.00694, 2017. — [arXiv:1711.00694](https://arxiv.org/abs/1711.00694) [find](https://scholar.google.com/scholar?q=Smitha+Milli%2C+Pieter+Abbeel%2C+and+Igor+Mordatch%2E+Interpretable+and+pedagogical+examples%2E+arXiv+preprint+arXiv%3A1711%2E00694%2C+2017%2E+%E2%80%94+%5BarXiv%3A1711%2E00694%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1711%2E00694%29)
15. Rémi Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In International conference on computers and games, pages 72–83. Springer, 2006. [find](https://scholar.google.com/scholar?q=R%C3%A9mi+Coulom%2E+Efficient+selectivity+and+backup+operators+in+monte%2Dcarlo+tree+search%2E+In+International+conference+on+computers+and+games%2C+pages+72%E2%80%9383%2E+Springer%2C+2006%2E)
16. John Tromp and Gunnar Farnebäck. Combinatorics of Go. In International Conference on Computers and Games, pages 84–99. Springer, 2006. [find](https://scholar.google.com/scholar?q=John+Tromp+and+Gunnar+Farneb%C3%A4ck%2E+Combinatorics+of+Go%2E+In+International+Conference+on+Computers+and+Games%2C+pages+84%E2%80%9399%2E+Springer%2C+2006%2E)
17. Radiolab. Debatable. [https://www.radiolab.org/story/debatable](https://www.radiolab.org/story/debatable), March 2016. [↗](https://www.radiolab.org/story/debatable)
18. Trapit Bansal, Jakub Pachocki, Szymon Sidor, Ilya Sutskever, and Igor Mordatch. Emergent complexity via multi-agent competition. arXiv preprint arXiv:1710.03748, 2017. — [arXiv:1710.03748](https://arxiv.org/abs/1710.03748) [find](https://scholar.google.com/scholar?q=Trapit+Bansal%2C+Jakub+Pachocki%2C+Szymon+Sidor%2C+Ilya+Sutskever%2C+and+Igor+Mordatch%2E+Emergent+complexity+via+multi%2Dagent+competition%2E+arXiv+preprint+arXiv%3A1710%2E03748%2C+2017%2E+%E2%80%94+%5BarXiv%3A1710%2E03748%5D%28https%3A%2F%2Farx)
19. Marc Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Julien Perolat, David Silver, Thore Graepel, et al. A unified game-theoretic approach to multiagent reinforcement learning. In Advances in Neural Information Processing Systems, pages 4193–4206, 2017. [find](https://scholar.google.com/scholar?q=Marc+Lanctot%2C+Vinicius+Zambaldi%2C+Audrunas+Gruslys%2C+Angeliki+Lazaridou%2C+Julien+Perolat%2C+David+Silver%2C+Thore+Graepel%2C+et+al%2E+A+unified+game%2Dtheoretic+approach+to+multiagent+reinforcement+learning%2E+In+Ad)
20. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. In Advances in Neural Information Processing Systems, pages 2672–2680, 2014. [find](https://scholar.google.com/scholar?q=Ian+Goodfellow%2C+Jean+Pouget%2DAbadie%2C+Mehdi+Mirza%2C+Bing+Xu%2C+David+Warde%2DFarley%2C+Sherjil+Ozair%2C+Aaron+Courville%2C+and+Yoshua+Bengio%2E+Generative+adversarial+networks%2E+In+Advances+in+Neural+Information+Proc)
21. Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017. — [arXiv:1710.10196](https://arxiv.org/abs/1710.10196) [find](https://scholar.google.com/scholar?q=Tero+Karras%2C+Timo+Aila%2C+Samuli+Laine%2C+and+Jaakko+Lehtinen%2E+Progressive+growing+of+GANs+for+improved+quality%2C+stability%2C+and+variation%2E+arXiv+preprint+arXiv%3A1710%2E10196%2C+2017%2E+%E2%80%94+%5BarXiv%3A1710%2E10196%5D%28https)
22. Keith E Stanovich and Richard F West. Reasoning independently of prior belief and individual differences in actively open-minded thinking. Journal of Educational Psychology, 89(2):342, 1997. [find](https://scholar.google.com/scholar?q=Keith+E+Stanovich+and+Richard+F+West%2E+Reasoning+independently+of+prior+belief+and+individual+differences+in+actively+open%2Dminded+thinking%2E+Journal+of+Educational+Psychology%2C+89%282%29%3A342%2C+1997%2E)
23. Donna Torrens. Individual differences and the belief bias effect: Mental models, logical necessity, and abstract reasoning. Thinking & Reasoning, 5(1):1–28, 1999. [find](https://scholar.google.com/scholar?q=Donna+Torrens%2E+Individual+differences+and+the+belief+bias+effect%3A+Mental+models%2C+logical+necessity%2C+and+abstract+reasoning%2E+Thinking+%26+Reasoning%2C+5%281%29%3A1%E2%80%9328%2C+1999%2E)
24. Dan Kahan. Weekend update: You’d have to be science illiterate to think “belief in evolution” measures science literacy. [http://www.culturalcognition.net/blog/2014/5/24/weekend-update-youd-have-to-be-science-illiterate-to-think-b.html](http://www.culturalcognition.net/blog/2014/5/24/weekend-update-youd-have-to-be-science-illiterate-to-think-b.html), May 2014. [↗](http://www.culturalcognition.net/blog/2014/5/24/weekend-update-youd-have-to-be-science-illiterate-to-think-b.html)
25. Jonathan St. B. T. Evans and Jodie Curtis-Holmes. Rapid responding increases belief bias: Evidence for the dual-process theory of reasoning. Thinking & Reasoning, 11(4):382–389, 2005. [find](https://scholar.google.com/scholar?q=Jonathan+St%2E+B%2E+T%2E+Evans+and+Jodie+Curtis%2DHolmes%2E+Rapid+responding+increases+belief+bias%3A+Evidence+for+the+dual%2Dprocess+theory+of+reasoning%2E+Thinking+%26+Reasoning%2C+11%284%29%3A382%E2%80%93389%2C+2005%2E)
26. Glenda Andrews. Belief-based and analytic processing in transitive inference depends on premise integration difficulty. Memory & cognition, 38(7):928–940, 2010. [find](https://scholar.google.com/scholar?q=Glenda+Andrews%2E+Belief%2Dbased+and+analytic+processing+in+transitive+inference+depends+on+premise+integration+difficulty%2E+Memory+%26+cognition%2C+38%287%29%3A928%E2%80%93940%2C+2010%2E)
27. Jonathan St BT Evans, Simon J Handley, and Alison M Bacon. Reasoning under time pressure: A study of causal conditional inference. Experimental Psychology, 56(2):77, 2009. [find](https://scholar.google.com/scholar?q=Jonathan+St+BT+Evans%2C+Simon+J+Handley%2C+and+Alison+M+Bacon%2E+Reasoning+under+time+pressure%3A+A+study+of+causal+conditional+inference%2E+Experimental+Psychology%2C+56%282%29%3A77%2C+2009%2E)
28. Vinod Goel and Oshin Vartanian. Negative emotions can attenuate the influence of beliefs on logical reasoning. Cognition and Emotion, 25(1):121–131, 2011. [find](https://scholar.google.com/scholar?q=Vinod+Goel+and+Oshin+Vartanian%2E+Negative+emotions+can+attenuate+the+influence+of+beliefs+on+logical+reasoning%2E+Cognition+and+Emotion%2C+25%281%29%3A121%E2%80%93131%2C+2011%2E)
29. Nick Bostrom. The superintelligent will: Motivation and instrumental rationality in advanced artificial agents. Minds and Machines, 22(2):71–85, 2012. [find](https://scholar.google.com/scholar?q=Nick+Bostrom%2E+The+superintelligent+will%3A+Motivation+and+instrumental+rationality+in+advanced+artificial+agents%2E+Minds+and+Machines%2C+22%282%29%3A71%E2%80%9385%2C+2012%2E)
30. Eliezer Yudkowsky. The AI-box experiment. [http://yudkowsky.net/singularity/aibox](http://yudkowsky.net/singularity/aibox), 2002. [↗](http://yudkowsky.net/singularity/aibox)
31. Nick Bostrom. Superintelligence. Dunod, 2017. [find](https://scholar.google.com/scholar?q=Nick+Bostrom%2E+Superintelligence%2E+Dunod%2C+2017%2E)
32. OpenAI. Faulty reward functions faulty-reward-functions, 2016. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Faulty+reward+functions+faulty%2Dreward%2Dfunctions%2C+2016%2E)
33. Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. In Advances in Neural Information Processing Systems, pages 6382–6393, 2017b. in the wild. [https://blog.openai.com/](https://blog.openai.com/) [↗](https://blog.openai.com/)

*Updated 2 November 2026*