---
title: "Measuring abstract reasoning in neural networks — Barrett 2018"
description: "The proposed Relation Network variant called WReN, designed to compare parts of the image with… 33 references, cited by 38 papers here."
authors: ["David G. T. Barrett", "Felix Hill", "Adam Santoro", "Ari S. Morcos", "Timothy Lillicrap"]
year: 2018
venue: "arXiv:1807.04225"
source: "https://arxiv.org/abs/1807.04225"
doi: "10.48550/arxiv.1807.04225"
arxiv: "1807.04225"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/barrett-2018-measuring-abstract-reasoning-neural-networks/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/barrett-2018-measuring-abstract-reasoning-neural-networks/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/) / Measuring abstract reasoning in neural networks

# Measuring abstract reasoning in neural networks

**Authors:** [David G. T. Barrett](https://md.research.elamur.ai/free/en/authors/david-g-t-barrett/), [Felix Hill](https://md.research.elamur.ai/free/en/authors/felix-hill/), [Adam Santoro](https://md.research.elamur.ai/free/en/authors/adam-santoro/), [Ari S. Morcos](https://md.research.elamur.ai/free/en/authors/ari-s-morcos/) and [Timothy Lillicrap](https://md.research.elamur.ai/free/en/authors/timothy-lillicrap/) · **Year:** 2018 · **Venue:** arXiv:1807.04225 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1807.04225) · [PDF](https://arxiv.org/pdf/1807.04225) · [DOI 10.48550/arxiv.1807.04225](https://doi.org/10.48550/arxiv.1807.04225) · [arXiv 1807.04225](https://arxiv.org/abs/1807.04225) · [OpenAlex](https://openalex.org/W2804180102)

**Citations:** 135 citations (OpenAlex, 2026-10-14) · [38](https://md.research.elamur.ai/free/en/papers/barrett-2018-measuring-abstract-reasoning-neural-networks/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

The authors from DeepMind examine whether neural networks truly learn to reason abstractly or merely exploit superficial statistical regularities in the data. Taking the Raven's Progressive Matrices IQ test as a model, they built a puzzle generator and a large open dataset, PGM (Procedurally Generated Matrices). Each task is a 3×3 matrix of images with one cell missing, which must be chosen from several candidates. The abstract structure of each problem is specified explicitly, so the training and test sets can be separated in a controlled way, through so-called generalization regimes. Purely convolutional models, including ResNet-50, performed poorly. The proposed Relation Network variant called WReN, designed to compare parts of the image with one another, performs considerably better. It interpolates attribute values well and applies known rules in novel combinations. At the same time, the model weakens markedly when it must extrapolate beyond the values it has seen or deal with unfamiliar attributes. Generalization improves noticeably if the model is additionally trained to predict a symbolic explanation of its answer. The work offers a tool for honestly measuring the abstract reasoning of models and for developing it.

## Key points

- The structure of each PGM is a set of 1–4 triples [relation, object, attribute]. Relations: progression, XOR, OR, AND, consistent union. Objects: shape, line. Attributes: size, type, colour, position, number. Separately, there is a regime with distracting attributes that vary randomly.
- Several generalization regimes are studied: neutral, interpolation, extrapolation, held-out attribute (shape-colour, line-type), held-out triples and held-out pairs of triples.
- There are 29 possible unique triples, 7 of which are held out for testing only. Of the 400 permissible pairs of triples, 360 are assigned to training and 40 to testing.
- WReN substantially outperforms CNN and ResNet-50. However, the model does not transfer a known relation to a different object type, whereas for a human this would be trivial.
- An auxiliary task of predicting a symbolic explanation of the answer (meta-targets) improves both accuracy and generalization ability.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 492 works · 440,463 citations
- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 402 works · 212,763 citations
- [University of Toronto](https://md.research.elamur.ai/free/en/universities/university-of-toronto/) — University, Toronto, Canada · 110 works · 179,062 citations
- [Meta (Israel)](https://md.research.elamur.ai/free/en/companies/meta-israel/) — Company, Tel Aviv, Israel · 46 works · 42,810 citations

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works

## Cite

Cite as: David G. T. Barrett, Felix Hill, Adam Santoro, Ari S. Morcos and Timothy Lillicrap. 2018. Measuring abstract reasoning in neural networks. arXiv:1807.04225.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{barrett2018measuring,
  title = {{Measuring abstract reasoning in neural networks}},
  author = {David G. T. Barrett, Felix Hill, Adam Santoro, Ari S. Morcos and Timothy Lillicrap},
  year = {2018},
  doi = {10.48550/arxiv.1807.04225},
  eprint = {1807.04225},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1807.04225}
}
```

- [← previous The IIT Bombay English-Hindi Parallel Corpus](https://md.research.elamur.ai/free/en/papers/kunchukuttan-2018-iit-bombay-english-hindi-parallel/)
- [next → DeepProbLog: Neural Probabilistic Logic Programming](https://md.research.elamur.ai/free/en/papers/manhaeve-2018-deepproblog-neural-probabilistic-logic/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/)

## References

33 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Andreas, J., Klein, D., and Levine, S. Modular multitask reinforcement learning with policy sketches. arXiv preprint arXiv:1611.01796, 2016. [find](https://scholar.google.com/scholar?q=Andreas%2C+J%2E%2C+Klein%2C+D%2E%2C+and+Levine%2C+S%2E+Modular+multitask+reinforcement+learning+with+policy+sketches%2E+arXiv+preprint+arXiv%3A1611%2E01796%2C+2016%2E)
2. Botvinick, M., Barrett, D., Battaglia, P., de Freitas, N., Kumaran, D., Leibo, J., Lillicrap, T., Modayil, J., Mohamed, S., Rabinowitz, N., et al. Building machines that learn and think for themselves: Commentary on lake et al., behavioral and brain sciences, 2017. arXiv preprint arXiv:1711.08378, 2… — [arXiv:1711.08378](https://arxiv.org/abs/1711.08378) [find](https://scholar.google.com/scholar?q=Botvinick%2C+M%2E%2C+Barrett%2C+D%2E%2C+Battaglia%2C+P%2E%2C+de+Freitas%2C+N%2E%2C+Kumaran%2C+D%2E%2C+Leibo%2C+J%2E%2C+Lillicrap%2C+T%2E%2C+Modayil%2C+J%2E%2C+Mohamed%2C+S%2E%2C+Rabinowitz%2C+N%2E%2C+et+al%2E+Building+machines+that+learn+and+think+for+themselves)
3. Carpenter, P. A., Just, M. A., and Shell, P. What one intelligence test measures: a theoretical account of the processing in the raven progressive matrices test. Psychological review, 97(3):404, 1990. [find](https://scholar.google.com/scholar?q=Carpenter%2C+P%2E+A%2E%2C+Just%2C+M%2E+A%2E%2C+and+Shell%2C+P%2E+What+one+intelligence+test+measures%3A+a+theoretical+account+of+the+processing+in+the+raven+progressive+matrices+test%2E+Psychological+review%2C+97%283%29%3A404%2C+1990%2E)
4. Clark, P. and Etzioni, O. My computer is an honor student-but how intelligent is it? standardized tests as a measure of ai. AI Magazine, 37(1):5–12, 2016. [find](https://scholar.google.com/scholar?q=Clark%2C+P%2E+and+Etzioni%2C+O%2E+My+computer+is+an+honor+student%2Dbut+how+intelligent+is+it%3F+standardized+tests+as+a+measure+of+ai%2E+AI+Magazine%2C+37%281%29%3A5%E2%80%9312%2C+2016%2E)
5. Davis, E. The limitations of standardized science tests as benchmarks for artificial intelligence research: Position paper. arXiv preprint arXiv:1411.1629, 2014. — [arXiv:1411.1629](https://arxiv.org/abs/1411.1629) [find](https://scholar.google.com/scholar?q=Davis%2C+E%2E+The+limitations+of+standardized+science+tests+as+benchmarks+for+artificial+intelligence+research%3A+Position+paper%2E+arXiv+preprint+arXiv%3A1411%2E1629%2C+2014%2E+%E2%80%94+%5BarXiv%3A1411%2E1629%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
6. Fleuret, F., Li, T., Dubout, C., Wampler, E. K., Yantis, S., and Geman, D. Comparing machines and humans on a visual categorization test. Proceedings of the National Academy of Sciences, 108(43):17621–17625, 2011. [find](https://scholar.google.com/scholar?q=Fleuret%2C+F%2E%2C+Li%2C+T%2E%2C+Dubout%2C+C%2E%2C+Wampler%2C+E%2E+K%2E%2C+Yantis%2C+S%2E%2C+and+Geman%2C+D%2E+Comparing+machines+and+humans+on+a+visual+categorization+test%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+108%2843%29%3A17621)
7. Flynn, J. R. Massive iq gains in 14 nations: What iq tests really measure. Psychological bulletin, 101(2):171, 1987. [find](https://scholar.google.com/scholar?q=Flynn%2C+J%2E+R%2E+Massive+iq+gains+in+14+nations%3A+What+iq+tests+really+measure%2E+Psychological+bulletin%2C+101%282%29%3A171%2C+1987%2E)
8. Garnelo, M., Arulkumaran, K., and Shanahan, M. Towards deep symbolic reinforcement learning. arXiv preprint arXiv:1609.05518, 2016. — [arXiv:1609.05518](https://arxiv.org/abs/1609.05518) [find](https://scholar.google.com/scholar?q=Garnelo%2C+M%2E%2C+Arulkumaran%2C+K%2E%2C+and+Shanahan%2C+M%2E+Towards+deep+symbolic+reinforcement+learning%2E+arXiv+preprint+arXiv%3A1609%2E05518%2C+2016%2E+%E2%80%94+%5BarXiv%3A1609%2E05518%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1609%2E05518%29)
9. He, K., Zhang, X., Ren, S., and Sun, J. [Deep residual learning for image recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/). In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
10. Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. beta-vae: Learning basic visual concepts with a constrained variational framework. 2016. [find](https://scholar.google.com/scholar?q=Higgins%2C+I%2E%2C+Matthey%2C+L%2E%2C+Pal%2C+A%2E%2C+Burgess%2C+C%2E%2C+Glorot%2C+X%2E%2C+Botvinick%2C+M%2E%2C+Mohamed%2C+S%2E%2C+and+Lerchner%2C+A%2E+beta%2Dvae%3A+Learning+basic+visual+concepts+with+a+constrained+variational+framework%2E+2016%2E)
11. Higgins, I., Sonnerat, N., Matthey, L., Pal, A., Burgess, C. P., Botvinick, M., Hassabis, D., and Lerchner, A. Scan: learning abstract hierarchical compositional visual concepts. arXiv preprint arXiv:1707.03389, 2017. — [arXiv:1707.03389](https://arxiv.org/abs/1707.03389) [find](https://scholar.google.com/scholar?q=Higgins%2C+I%2E%2C+Sonnerat%2C+N%2E%2C+Matthey%2C+L%2E%2C+Pal%2C+A%2E%2C+Burgess%2C+C%2E+P%2E%2C+Botvinick%2C+M%2E%2C+Hassabis%2C+D%2E%2C+and+Lerchner%2C+A%2E+Scan%3A+learning+abstract+hierarchical+compositional+visual+concepts%2E+arXiv+preprint+arXiv%3A)
12. Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural Comput., 9(8):1735–1780, November 1997. ISSN 0899-7667. doi: 10.1162/neco.1997.9.8.1735. [find](https://scholar.google.com/scholar?q=Hochreiter%2C+S%2E+and+Schmidhuber%2C+J%2E+Long+short%2Dterm+memory%2E+Neural+Comput%2E%2C+9%288%29%3A1735%E2%80%931780%2C+November+1997%2E+ISSN+0899%2D7667%2E+doi%3A+10%2E1162%2Fneco%2E1997%2E9%2E8%2E1735%2E)
13. Hoshen, D. and Werman, M. Iq of neural networks. arXiv preprint arXiv:1710.01692, 2017. — [arXiv:1710.01692](https://arxiv.org/abs/1710.01692) [find](https://scholar.google.com/scholar?q=Hoshen%2C+D%2E+and+Werman%2C+M%2E+Iq+of+neural+networks%2E+arXiv+preprint+arXiv%3A1710%2E01692%2C+2017%2E+%E2%80%94+%5BarXiv%3A1710%2E01692%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1710%2E01692%29)
14. Jaeggi, S. M., Buschkuehl, M., Jonides, J., and Perrig, W. J. Improving fluid intelligence with training on working memory. Proceedings of the National Academy of Sciences, 105(19):6829–6833, 2008. [find](https://scholar.google.com/scholar?q=Jaeggi%2C+S%2E+M%2E%2C+Buschkuehl%2C+M%2E%2C+Jonides%2C+J%2E%2C+and+Perrig%2C+W%2E+J%2E+Improving+fluid+intelligence+with+training+on+working+memory%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+105%2819%29%3A6829%E2%80%936833%2C+2008%2E)
15. Jo, J. and Bengio, Y. Measuring the tendency of cnns to learn surface statistical regularities. arXiv preprint arXiv:1711.11561, 2017. — [arXiv:1711.11561](https://arxiv.org/abs/1711.11561) [find](https://scholar.google.com/scholar?q=Jo%2C+J%2E+and+Bengio%2C+Y%2E+Measuring+the+tendency+of+cnns+to+learn+surface+statistical+regularities%2E+arXiv+preprint+arXiv%3A1711%2E11561%2C+2017%2E+%E2%80%94+%5BarXiv%3A1711%2E11561%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1711%2E11561%29)
16. Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., and Girshick, R. [Clevr: A diagnostic dataset for compositional language and elementary visual reasoning](https://md.research.elamur.ai/free/en/papers/johnson-2016-clevr-diagnostic-dataset-compositional-language/). In Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on, pp. 1988–1997.
17. Kingma, D. P. and Ba, J. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). arXiv preprint arXiv:1412.6980, 2014.
18. Lake, B. M. and Baroni, M. Still not systematic after all these years: On the compositional skills of sequence-to-sequence recurrent networks. arXiv preprint arXiv:1711.00350, 2017. [find](https://scholar.google.com/scholar?q=Lake%2C+B%2E+M%2E+and+Baroni%2C+M%2E+Still+not+systematic+after+all+these+years%3A+On+the+compositional+skills+of+sequence%2Dto%2Dsequence+recurrent+networks%2E+arXiv+preprint+arXiv%3A1711%2E00350%2C+2017%2E)
19. LeCun, Y., Bengio, Y., and Hinton, G. Deep learning. nature, 521(7553):436, 2015. [find](https://scholar.google.com/scholar?q=LeCun%2C+Y%2E%2C+Bengio%2C+Y%2E%2C+and+Hinton%2C+G%2E+Deep+learning%2E+nature%2C+521%287553%29%3A436%2C+2015%2E)
20. Ling, W., Yogatama, D., Dyer, C., and Blunsom, P. Program induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv preprint arXiv:1705.04146, 2017. [find](https://scholar.google.com/scholar?q=Ling%2C+W%2E%2C+Yogatama%2C+D%2E%2C+Dyer%2C+C%2E%2C+and+Blunsom%2C+P%2E+Program+induction+by+rationale+generation%3A+Learning+to+solve+and+explain+algebraic+word+problems%2E+arXiv+preprint+arXiv%3A1705%2E04146%2C+2017%2E)
21. Lovett, A. and Forbus, K. Modeling visual problem solving as analogical reasoning. Psychological review, 124(1):60, 2017. [find](https://scholar.google.com/scholar?q=Lovett%2C+A%2E+and+Forbus%2C+K%2E+Modeling+visual+problem+solving+as+analogical+reasoning%2E+Psychological+review%2C+124%281%29%3A60%2C+2017%2E)
22. Marcus, G. [Deep learning: A critical appraisal](https://md.research.elamur.ai/free/en/papers/marcus-2018-deep-learning-critical-appraisal/). arXiv preprint arXiv:1801.00631, 2018.
23. Marcus, G., Rossi, F., and Veloso, M. Beyond the turing test. Ai Magazine, 37(1):3–4, 2016. [find](https://scholar.google.com/scholar?q=Marcus%2C+G%2E%2C+Rossi%2C+F%2E%2C+and+Veloso%2C+M%2E+Beyond+the+turing+test%2E+Ai+Magazine%2C+37%281%29%3A3%E2%80%934%2C+2016%2E)
24. Ramakrishnan, S. K., Pal, A., Sharma, G., and Mittal, A. An empirical evaluation of visual question answering for novel objects. arXiv preprint arXiv:1704.02516, 2017. — [arXiv:1704.02516](https://arxiv.org/abs/1704.02516) [find](https://scholar.google.com/scholar?q=Ramakrishnan%2C+S%2E+K%2E%2C+Pal%2C+A%2E%2C+Sharma%2C+G%2E%2C+and+Mittal%2C+A%2E+An+empirical+evaluation+of+visual+question+answering+for+novel+objects%2E+arXiv+preprint+arXiv%3A1704%2E02516%2C+2017%2E+%E2%80%94+%5BarXiv%3A1704%2E02516%5D%28https%3A%2F%2Farx)
25. Raposo, D., Santoro, A., Barrett, D., Pascanu, R., Lillicrap, T., and Battaglia, P. Discovering objects and their relations from entangled scene representations. 2017. [find](https://scholar.google.com/scholar?q=Raposo%2C+D%2E%2C+Santoro%2C+A%2E%2C+Barrett%2C+D%2E%2C+Pascanu%2C+R%2E%2C+Lillicrap%2C+T%2E%2C+and+Battaglia%2C+P%2E+Discovering+objects+and+their+relations+from+entangled+scene+representations%2E+2017%2E)
26. Raven, J. C. et al. Raven’s progressive matrices. Western Psychological Services, 1938. [find](https://scholar.google.com/scholar?q=Raven%2C+J%2E+C%2E+et+al%2E+Raven%E2%80%99s+progressive+matrices%2E+Western+Psychological+Services%2C+1938%2E)
27. Santoro, A., Raposo, D., Barrett, D. G., Malinowski, M., Pascanu, R., Battaglia, P., and Lillicrap, T. [A simple neural network module for relational reasoning](https://md.research.elamur.ai/free/en/papers/santoro-2017-simple-neural-network-module-relational/). In Advances in neural information processing systems, pp. 4974–4983, 2017.
28. Snow, R. E., Kyllonen, P. C., and Marshalek, B. The topography of ability and learning correlations. Advances in the psychology of human intelligence, 2(S 47):103, 1984. [find](https://scholar.google.com/scholar?q=Snow%2C+R%2E+E%2E%2C+Kyllonen%2C+P%2E+C%2E%2C+and+Marshalek%2C+B%2E+The+topography+of+ability+and+learning+correlations%2E+Advances+in+the+psychology+of+human+intelligence%2C+2%28S+47%29%3A103%2C+1984%2E)
29. Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. [Intriguing properties of neural networks](https://md.research.elamur.ai/free/en/papers/szegedy-2013-intriguing-properties-neural-networks/). arXiv preprint arXiv:1312.6199, 2013.
30. Te Nijenhuis, J., Voskuijl, O. F., and Schijve, N. B. Practice and coaching on iq tests: Quite a lot of g. International Journal of Selection and Assessment, 9(4):302–308, 2001. [find](https://scholar.google.com/scholar?q=Te+Nijenhuis%2C+J%2E%2C+Voskuijl%2C+O%2E+F%2E%2C+and+Schijve%2C+N%2E+B%2E+Practice+and+coaching+on+iq+tests%3A+Quite+a+lot+of+g%2E+International+Journal+of+Selection+and+Assessment%2C+9%284%29%3A302%E2%80%93308%2C+2001%2E)
31. Wang, K. and Su, Z. Automatic generation of ravens progressive matrices. In Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015. [find](https://scholar.google.com/scholar?q=Wang%2C+K%2E+and+Su%2C+Z%2E+Automatic+generation+of+ravens+progressive+matrices%2E+In+Twenty%2DFourth+International+Joint+Conference+on+Artificial+Intelligence%2C+2015%2E)
32. Zaremba, W., Sutskever, I., and Vinyals, O. [Recurrent neural network regularization](https://md.research.elamur.ai/free/en/papers/zaremba-2014-recurrent-neural-network-regularization/). arXiv preprint arXiv:1409.2329, 2014.
33. Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016. [find](https://scholar.google.com/scholar?q=Zhang%2C+C%2E%2C+Bengio%2C+S%2E%2C+Hardt%2C+M%2E%2C+Recht%2C+B%2E%2C+and+Vinyals%2C+O%2E+Understanding+deep+learning+requires+rethinking+generalization%2E+arXiv+preprint+arXiv%3A1611%2E03530%2C+2016%2E)

*Updated 2 November 2026*