---
title: "Can Neural Networks Understand Logical Entailment? (2018)"
description: "The DeepMind authors build a new dataset of triples (A, B, label), with labels computed by the minisat… 42 references, cited by 34 papers here."
authors: ["Richard Evans", "David Saxton", "David Amos", "Pushmeet Kohli", "Edward Grefenstette"]
year: 2018
venue: "arXiv:1802.08535"
source: "https://arxiv.org/abs/1802.08535"
doi: "10.48550/arxiv.1802.08535"
arxiv: "1802.08535"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "https://arxiv.org/licenses/nonexclusive-distrib/1.0/license.html"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/evans-2018-can-neural-networks-understand-logical/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/evans-2018-can-neural-networks-understand-logical/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/5/) / Can Neural Networks Understand Logical Entailment?

# Can Neural Networks Understand Logical Entailment?

**Authors:** [Richard Evans](https://md.research.elamur.ai/free/en/authors/richard-evans/), [David Saxton](https://md.research.elamur.ai/free/en/authors/david-saxton/), David Amos, [Pushmeet Kohli](https://md.research.elamur.ai/free/en/authors/pushmeet-kohli/) and [Edward Grefenstette](https://md.research.elamur.ai/free/en/authors/edward-grefenstette/) · **Year:** 2018 · **Venue:** arXiv:1802.08535 · **Licence:** [arXiv — non-exclusive licence to distribute](https://arxiv.org/licenses/nonexclusive-distrib/1.0/license.html)

**Links:** [Publisher's page](https://arxiv.org/abs/1802.08535) · [PDF](https://arxiv.org/pdf/1802.08535v1) · [DOI 10.48550/arxiv.1802.08535](https://doi.org/10.48550/arxiv.1802.08535) · [arXiv 1802.08535](https://arxiv.org/abs/1802.08535) · [OpenAlex](https://openalex.org/W2787525746)

**Citations:** 10 citations (OpenAlex, 2026-10-14) · [34](https://md.research.elamur.ai/free/en/papers/evans-2018-can-neural-networks-understand-logical/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks whether neural networks can decide if one propositional formula logically entails another. The task works as a clean probe of how well architectures capture purely structural properties of sequences, because variable names carry no meaning and only the connectives matter. The DeepMind authors build a new dataset of triples (A, B, label), with labels computed by the minisat solver. To remove trivial cues, they generate formulas in 4-tuples so that every formula appears in exactly one positive and one negative pair. This equalises length and operator statistics across the two classes. The test splits range from small formulas to very large ones, plus a set of textbook examples. Test and validation formulas that are α-equivalent to training formulas are filtered out. The authors compare bag-of-words baselines, convolutional networks, LSTMs and tree-structured networks. They also introduce PossibleWorldNet, a TreeNet variant inspired by model-theoretic semantics that evaluates formulas across many "possible worlds". Convolutional models show an unsuitable inductive bias, tree models beat LSTMs by exploiting syntax, and PossibleWorldNet outperforms every benchmark. The work provides a testbed for studying structural reasoning and inductive biases.

## Key points
- **Dataset sizes:** 100,000 train, 5,000 validation, and 5,000 each for test (easy), (hard) and (big). Test (massive) has 2,230 items and test (exam) has 100 items drawn from logic textbooks.
- **Variables and difficulty:** formulas use up to 26 variables (a–z). Mean 2^#vars, a proxy for difficulty, grows from about 52 in train to about 848,570 in test (massive).
- **Debiasing:** naive sampling leaks shortcuts. For example, positive premises are satisfied by about 3.7 truth assignments on average versus 10.3 for negatives. The 4-tuple scheme brings length and operator-count differences to zero, leaving only a small gap in new variables (1.25 vs 1.60).
- **Anti-memorisation and augmentation:** de-Bruijn normal forms are used to drop α-equivalent duplicates. During training, variables are randomly permuted within each example, similar to image crops or flips.
- **Models compared:** Linear BoW, MLP BoW, CNNs, LSTMs, TreeNets and the new PossibleWorldNet. Random or majority-class guessing scores 50% because the classes are balanced.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 533 works · 443,884 citations

Also on the byline: [University of Westminster](https://ror.org/04ycpbx82).

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works

## Cite

Cite as: Richard Evans, David Saxton, David Amos, Pushmeet Kohli and Edward Grefenstette. 2018. Can Neural Networks Understand Logical Entailment?. arXiv:1802.08535.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{evans2018can,
  title = {{Can Neural Networks Understand Logical Entailment?}},
  author = {Richard Evans, David Saxton, David Amos, Pushmeet Kohli and Edward Grefenstette},
  year = {2018},
  doi = {10.48550/arxiv.1802.08535},
  eprint = {1802.08535},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1802.08535}
}
```

- [← previous A Corpus of Non-Native Written English Annotated for Metaphor](https://md.research.elamur.ai/free/en/papers/klebanov-2018-corpus-non-native-written-english/)
- [next → Zero Pronoun Resolution with Attention-based Neural Network](https://md.research.elamur.ai/free/en/papers/yin-2018-zero-pronoun-resolution-attention-based/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/5/)

## References

42 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Martı́n Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. [Tensorflow: Large-scale machine learning on heterogeneous distributed systems](https://md.research.elamur.ai/free/en/papers/abadi-2016-tensorflow-large-scale-machine-learning/). arXiv preprint arXiv:1603.04467, 2016.
2. Miltiadis Allamanis, Pankajan Chanthirasegaran, Pushmeet Kohli, and Charles Sutton. [Learning continuous semantic representations of symbolic expressions](https://md.research.elamur.ai/free/en/papers/allamanis-2016-learning-continuous-semantic-representations-symbolic/). arXiv preprint arXiv:1611.01423, 2016.
3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). arXiv preprint arXiv:1409.0473, 2014.
4. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. [A large annotated corpus for learning natural language inference](https://md.research.elamur.ai/free/en/papers/bowman-2015-large-annotated-corpus-learning-natural/). arXiv preprint arXiv:1508.05326, 2015.
5. Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment, pp. 177–190. Springer, 2006. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+The+pascal+recognising+textual+entailment+challenge%2E+In+Machine+learning+challenges%2E+evaluating+predictive+uncertainty%2C+visual+object+classification%2C+an)
6. Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. Robustfill: Neural program learning under noisy i/o. arXiv preprint arXiv:1703.07469, 2017. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Jonathan+Uesato%2C+Surya+Bhupatiraju%2C+Rishabh+Singh%2C+Abdel%2Drahman+Mohamed%2C+and+Pushmeet+Kohli%2E+Robustfill%3A+Neural+program+learning+under+noisy+i%2Fo%2E+arXiv+preprint+arXiv%3A1703%2E07469%2C+2017%2E)
7. Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538 (7626):471–476, 2016. [find](https://scholar.google.com/scholar?q=Alex+Graves%2C+Greg+Wayne%2C+Malcolm+Reynolds%2C+Tim+Harley%2C+Ivo+Danihelka%2C+Agnieszka+Grabska%2DBarwin%CC%81ska%2C+Sergio+Go%CC%81mez+Colmenarejo%2C+Edward+Grefenstette%2C+Tiago+Ramalho%2C+John+Agapiou%2C+et+al%2E+Hybrid+computing)
8. Edward Grefenstette, Karl Moritz Hermann, Mustafa Suleyman, and Phil Blunsom. Learning to transduce with unbounded memory. In Advances in Neural Information Processing Systems, pp. 1828–1836, 2015. [find](https://scholar.google.com/scholar?q=Edward+Grefenstette%2C+Karl+Moritz+Hermann%2C+Mustafa+Suleyman%2C+and+Phil+Blunsom%2E+Learning+to+transduce+with+unbounded+memory%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pp%2E+1828%E2%80%931836%2C+2015%2E)
9. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. [Teaching machines to read and comprehend](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/). In Advances in Neural Information Processing Systems, pp. 1693–1701, 2015.
10. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. The goldilocks principle: Reading children’s books with explicit memory representations. arXiv preprint arXiv:1511.[02301, 2015](https://md.research.elamur.ai/free/en/papers/hill-2015-goldilocks-principle-reading-children-s/).
11. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735–1780, 1997. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A+1735%E2%80%931780%2C+1997%2E)
12. Armand Joulin and Tomas Mikolov. Inferring algorithmic patterns with stack-augmented recurrent nets. In Advances in neural information processing systems, pp. 190–198, 2015. [find](https://scholar.google.com/scholar?q=Armand+Joulin+and+Tomas+Mikolov%2E+Inferring+algorithmic+patterns+with+stack%2Daugmented+recurrent+nets%2E+In+Advances+in+neural+information+processing+systems%2C+pp%2E+190%E2%80%93198%2C+2015%2E)
13. Łukasz Kaiser and Ilya Sutskever. Neural gpus learn algorithms. arXiv preprint arXiv:1511.08228, 2015. [find](https://scholar.google.com/scholar?q=%C5%81ukasz+Kaiser+and+Ilya+Sutskever%2E+Neural+gpus+learn+algorithms%2E+arXiv+preprint+arXiv%3A1511%2E08228%2C+2015%2E)
14. Nal Kalchbrenner and Phil Blunsom. [Recurrent continuous translation models](https://md.research.elamur.ai/free/en/papers/kalchbrenner-2013-recurrent-continuous-translation-models/). In EMNLP, volume 3, pp. 413, 2013.
15. Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. [A convolutional neural network for modelling sentences](https://md.research.elamur.ai/free/en/papers/kalchbrenner-2014-convolutional-neural-network-modelling-sentences/). arXiv preprint arXiv:1404.2188, 2014.
16. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. [Character-aware neural language models](https://md.research.elamur.ai/free/en/papers/kim-2015-character-aware-neural-language-models/). In AAAI, pp. 2741–2749, 2016.
17. Diederik Kingma and Jimmy Ba. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). arXiv preprint arXiv:1412.6980, 2014.
18. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. [Imagenet classification with deep convolutional neural networks](https://md.research.elamur.ai/free/en/papers/krizhevsky-2012-imagenet-classification-deep-convolutional/). In Advances in neural information processing systems, pp. 1097–1105, 2012.
19. Phong Le and Willem Zuidema. Compositional distributional semantics with long short term memory. arXiv preprint arXiv:1503.02510, 2015. [find](https://scholar.google.com/scholar?q=Phong+Le+and+Willem+Zuidema%2E+Compositional+distributional+semantics+with+long+short+term+memory%2E+arXiv+preprint+arXiv%3A1503%2E02510%2C+2015%2E)
20. Yann LeCun, Yoshua Bengio, et al. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361(10):1995, 1995. [find](https://scholar.google.com/scholar?q=Yann+LeCun%2C+Yoshua+Bengio%2C+et+al%2E+Convolutional+networks+for+images%2C+speech%2C+and+time+series%2E+The+handbook+of+brain+theory+and+neural+networks%2C+3361%2810%29%3A1995%2C+1995%2E)
21. Wang Ling, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiskỳ, Andrew Senior, Fumin Wang, and Phil Blunsom. [Latent predictor networks for code generation](https://md.research.elamur.ai/free/en/papers/ling-2016-latent-predictor-networks-code-generation/). arXiv preprint arXiv:1603.06744, 2016.
22. Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andy Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, et al. Learning to navigate in complex environments. arXiv preprint arXiv:1611.03673, 2016. [find](https://scholar.google.com/scholar?q=Piotr+Mirowski%2C+Razvan+Pascanu%2C+Fabio+Viola%2C+Hubert+Soyer%2C+Andy+Ballard%2C+Andrea+Banino%2C+Misha+Denil%2C+Ross+Goroshin%2C+Laurent+Sifre%2C+Koray+Kavukcuoglu%2C+et+al%2E+Learning+to+navigate+in+complex+environment)
23. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015. [find](https://scholar.google.com/scholar?q=Volodymyr+Mnih%2C+Koray+Kavukcuoglu%2C+David+Silver%2C+Andrei+A+Rusu%2C+Joel+Veness%2C+Marc+G+Bellemare%2C+Alex+Graves%2C+Martin+Riedmiller%2C+Andreas+K+Fidjeland%2C+Georg+Ostrovski%2C+et+al%2E+Human%2Dlevel+control+through+)
24. Emilio Parisotto, Abdel-rahman Mohamed, Rishabh Singh, Lihong Li, Dengyong Zhou, and Pushmeet Kohli. [Neuro-symbolic program synthesis](https://md.research.elamur.ai/free/en/papers/parisotto-2016-neuro-symbolic-program-synthesis/). arXiv preprint arXiv:1611.01855, 2016.
25. Benjamin C Pierce. Types and programming languages. MIT press, 2002. [find](https://scholar.google.com/scholar?q=Benjamin+C+Pierce%2E+Types+and+programming+languages%2E+MIT+press%2C+2002%2E)
26. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. [Squad: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). arXiv preprint arXiv:1606.05250, 2016.
27. Scott Reed and Nando De Freitas. arXiv:1511.06279, 2015. [find](https://scholar.google.com/scholar?q=Scott+Reed+and+Nando+De+Freitas%2E+arXiv%3A1511%2E06279%2C+2015%2E)
28. Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiskỳ, and Phil Blunsom. [Reasoning about entailment with neural attention](https://md.research.elamur.ai/free/en/papers/rocktaschel-2015-reasoning-about-entailment-neural-attention/). arXiv preprint arXiv:1509.06664, 2015.
29. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484–489, 2016. [find](https://scholar.google.com/scholar?q=David+Silver%2C+Aja+Huang%2C+Chris+J+Maddison%2C+Arthur+Guez%2C+Laurent+Sifre%2C+George+Van+Den+Driessche%2C+Julian+Schrittwieser%2C+Ioannis+Antonoglou%2C+Veda+Panneershelvam%2C+Marc+Lanctot%2C+et+al%2E+Mastering+the+game+)
30. Karen Simonyan and Andrew Zisserman. [Very deep convolutional networks for large-scale image recognition](https://md.research.elamur.ai/free/en/papers/simonyan-2014-very-deep-convolutional-networks-large/). arXiv preprint arXiv:1409.1556, 2014.
31. Richard Socher, Brody Huval, Christopher D Manning, and Andrew Y Ng. [Semantic compositionality through recursive matrix-vector spaces](https://md.research.elamur.ai/free/en/papers/socher-2012-semantic-compositionality-through-recursive-matrix/). In Proceedings of the 2012 joint conference on empirical methods in natural language processing and computational natural language learning, pp. 1201–1211. Associati…
32. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. [Recursive deep models for semantic compositionality over a sentiment treebank](https://md.research.elamur.ai/free/en/papers/socher-2013-recursive-deep-models-semantic-compositionality/). In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631–1642, 201…
33. Niklas Sorensson and Niklas Een. Minisat v1. 13-a sat solver with conflict-clause minimization. SAT, 2005(53):1–2, 2005. [find](https://scholar.google.com/scholar?q=Niklas+Sorensson+and+Niklas+Een%2E+Minisat+v1%2E+13%2Da+sat+solver+with+conflict%2Dclause+minimization%2E+SAT%2C+2005%2853%29%3A1%E2%80%932%2C+2005%2E)
34. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. [Sequence to sequence learning with neural networks](https://md.research.elamur.ai/free/en/papers/sutskever-2014-sequence-sequence-learning-neural-networks/). In Advances in neural information processing systems, pp. 3104–3112, 2014.
35. Kai Sheng Tai, Richard Socher, and Christopher D Manning. Improved semantic representations from tree-structured long short-term memory networks. arXiv preprint arXiv:1503.00075, 2015. [find](https://scholar.google.com/scholar?q=Kai+Sheng+Tai%2C+Richard+Socher%2C+and+Christopher+D+Manning%2E+Improved+semantic+representations+from+tree%2Dstructured+long+short%2Dterm+memory+networks%2E+arXiv+preprint+arXiv%3A1503%2E00075%2C+2015%2E)
36. Yuandong Tian and Yan Zhu. Better computer go player with neural network and long-term prediction. arXiv preprint arXiv:1511.06410, 2015. — [arXiv:1511.06410](https://arxiv.org/abs/1511.06410) [find](https://scholar.google.com/scholar?q=Yuandong+Tian+and+Yan+Zhu%2E+Better+computer+go+player+with+neural+network+and+long%2Dterm+prediction%2E+arXiv+preprint+arXiv%3A1511%2E06410%2C+2015%2E+%E2%80%94+%5BarXiv%3A1511%2E06410%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1511%2E06410%29)
37. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, pp. 6000–6010, 2017.
38. Shuohang Wang and Jing Jiang. [Learning natural language inference with lstm](https://md.research.elamur.ai/free/en/papers/wang-2016-learning-natural-language-inference-lstm/). arXiv preprint arXiv:1512.08849, 2015.
39. Wenpeng Yin, Hinrich Schütze, Bing Xiang, and Bowen Zhou. [Abcnn: Attention-based convolutional neural network for modeling sentence pairs](https://md.research.elamur.ai/free/en/papers/yin-2016-abcnn-attention-based-convolutional-neural/). arXiv preprint arXiv:1512.05193, 2015.
40. Wojciech Zaremba, Karol Kurach, and Rob Fergus. Learning to discover efficient mathematical identities. In Advances in Neural Information Processing Systems, pp. 1278–1286, 2014. [find](https://scholar.google.com/scholar?q=Wojciech+Zaremba%2C+Karol+Kurach%2C+and+Rob+Fergus%2E+Learning+to+discover+efficient+mathematical+identities%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pp%2E+1278%E2%80%931286%2C+2014%2E)
41. Neural programmer-interpreters. arXiv preprint Xiang Zhang, Junbo Zhao, and Yann LeCun. [Character-level convolutional networks for text classification](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/). In Advances in neural information processing systems, pp. 649–657, 2015.
42. Xiaodan Zhu, Parinaz Sobihani, and Hongyu Guo. Long short-term memory over recursive structures. In International Conference on Machine Learning, pp. 1604–1612, 2015. [find](https://scholar.google.com/scholar?q=Xiaodan+Zhu%2C+Parinaz+Sobihani%2C+and+Hongyu+Guo%2E+Long+short%2Dterm+memory+over+recursive+structures%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+1604%E2%80%931612%2C+2015%2E)

*Updated 3 November 2026*