---
title: "Learning and Evaluating General Linguistic Intelligence (2019)"
description: "They propose a notion of \"general linguistic intelligence\": a system's capacity to reuse what it… 36 references, cited by 104 papers here."
authors: ["Dani Yogatama", "Cyprien de Masson d'Autume", "Jerome Connor", "Tomas Kocisky", "Mike Chrzanowski", "Lingpeng Kong", "Angeliki Lazaridou", "Wang Ling", "Lei Yu", "Chris Dyer", "Phil Blunsom"]
year: 2019
venue: "arXiv:1901.11373"
source: "https://arxiv.org/abs/1901.11373"
doi: "10.48550/arxiv.1901.11373"
arxiv: "1901.11373"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yogatama-2019-learning-evaluating-general-linguistic-intelligence/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yogatama-2019-learning-evaluating-general-linguistic-intelligence/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/) / Learning and Evaluating General Linguistic Intelligence

# Learning and Evaluating General Linguistic Intelligence

**Authors:** [Dani Yogatama](https://md.research.elamur.ai/free/en/authors/dani-yogatama/), [Cyprien de Masson d'Autume](https://md.research.elamur.ai/free/en/authors/cyprien-de-masson-d-autume/), Jerome Connor, [Tomas Kocisky](https://md.research.elamur.ai/free/en/authors/tomas-kocisky/), [Mike Chrzanowski](https://md.research.elamur.ai/free/en/authors/mike-chrzanowski/), [Lingpeng Kong](https://md.research.elamur.ai/free/en/authors/lingpeng-kong/), [Angeliki Lazaridou](https://md.research.elamur.ai/free/en/authors/angeliki-lazaridou/), [Wang Ling](https://md.research.elamur.ai/free/en/authors/wang-ling/), [Lei Yu](https://md.research.elamur.ai/free/en/authors/lei-yu/), [Chris Dyer](https://md.research.elamur.ai/free/en/authors/chris-dyer/) and [Phil Blunsom](https://md.research.elamur.ai/free/en/authors/phil-blunsom/) · **Year:** 2019 · **Venue:** arXiv:1901.11373 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1901.11373) · [PDF](https://arxiv.org/pdf/1901.11373) · [DOI 10.48550/arxiv.1901.11373](https://doi.org/10.48550/arxiv.1901.11373) · [arXiv 1901.11373](https://arxiv.org/abs/1901.11373) · [OpenAlex](https://openalex.org/W2911681509)

**Citations:** 144 citations (OpenAlex, 2026-10-14) · [104](https://md.research.elamur.ai/free/en/papers/yogatama-2019-learning-evaluating-general-linguistic-intelligence/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

The authors argue that standard leaderboard scores overstate progress in NLP. They propose a notion of "general linguistic intelligence": a system's capacity to reuse what it already knows about a language's words, grammar, meaning and usage so it can master unfamiliar tasks from few examples. To measure this, they add a metric based on online (prequential) coding, following Blier & Ollivier (2018). It scores how many task-specific examples a model needs before it performs well, rather than only its final accuracy.

They test pretrained BERT and an ELMo-based BiDAF model on several reading-comprehension, semantic role labeling, relation extraction and inference datasets, under four setups:
- fine-tuning,
- transfer between related tasks,
- cross-dataset generalization,
- continual learning.

Both models still need substantial in-domain data and gain clearly from related-task pretraining. However, without fine-tuning they transfer poorly between datasets of the same task, which suggests overfitting to quirks of particular datasets such as SQuAD. They also forget earlier tasks when trained sequentially. The study matters because it reframes evaluation around sample efficiency and robustness. It also points to remedies such as memory modules, meta-learning and better curricula.

## Key points

- **Models compared:**
  - BERT base: 12 Transformer layers, 12 attention heads, 768 hidden units, about 110M parameters.
  - ELMo: about 100M parameters, with a 300-dimensional BiLSTM and a BiDAF network on top.
- **Datasets:** SQuAD 1.1, TriviaQA (Web section), QuAC, QA-SRL, the zero-shot relation extraction set (QA-ZRE), MNLI and SNLI. Most tasks are cast as span prediction.
- **New metric:** an online codelength measure that captures how fast a model adapts, rewarding sample efficiency on top of conventional end-of-training scores.
- **Training order matters:** sequential (continual) training causes catastrophic forgetting. In contrast, a random curriculum over 200,000 iterations lets BERT keep competence across all tasks, with these results:

| Model | SQuAD | MNLI | SNLI | QA-ZRE |
|---|---|---|---|---|
| BERT | 85.4 | 81.1 | 88.0 | 88.2 |
| ELMo | 78.3 | 69.1 | 77.9 | 88.5 |

- **Proposed directions:** elastic weight consolidation, progress and compress, neural cache and dynamic evaluation, meta-learning (with the codelength as a possible objective), and Population Based Training for designing curricula.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 492 works · 440,463 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works

## Cite

Cite as: Dani Yogatama, Cyprien de Masson d'Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer and Phil Blunsom. 2019. Learning and Evaluating General Linguistic Intelligence. arXiv:1901.11373.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{yogatama2019learning,
  title = {{Learning and Evaluating General Linguistic Intelligence}},
  author = {Dani Yogatama, Cyprien {de Masson d'Autume}, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer and Phil Blunsom},
  year = {2019},
  doi = {10.48550/arxiv.1901.11373},
  eprint = {1901.11373},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1901.11373}
}
```

- [← previous Improved Knowledge Distillation via Teacher Assistant](https://md.research.elamur.ai/free/en/papers/mirzadeh-2019-improved-knowledge-distillation-teacher-assistant/)
- [next → E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT](https://md.research.elamur.ai/free/en/papers/poerner-2019-e-bert-efficient-yet-effective/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/)

## References

36 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Kuttler, Andrew Lefrancq, Simon Green, Víctor Valdes, Amir Sadik, Julian Schrittwieser, Keith Anderson, Sarah York, Max Cant, Adam Cain, Adrian Bolton, Stephen Gaffney, Helen King, Demis Hassabis, Shane Legg, and… — [DeepMind Lab](https://md.research.elamur.ai/free/en/papers/beattie-2016-deepmind-lab/)
2. Leonard Blier and Yann Ollivier. The description length of deep learning models. In Proc. of NIPS, 2018. [find](https://scholar.google.com/scholar?q=Leonard+Blier+and+Yann+Ollivier%2E+The+description+length+of+deep+learning+models%2E+In+Proc%2E+of+NIPS%2C+2018%2E)
3. Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. [A large annotated corpus for learning natural language inference](https://md.research.elamur.ai/free/en/papers/bowman-2015-large-annotated-corpus-learning-natural/). In Proc. of EMNLP, 2015.
4. Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym. arXiv preprint, 2016. [find](https://scholar.google.com/scholar?q=Greg+Brockman%2C+Vicki+Cheung%2C+Ludwig+Pettersson%2C+Jonas+Schneider%2C+John+Schulman%2C+Jie+Tang%2C+and+Wojciech+Zaremba%2E+OpenAI+Gym%2E+arXiv+preprint%2C+2016%2E)
5. Gregory J. Chaitin. On the intelligibility of the universe and the notions of simplicity, complexity and irreducibility. In Thinking about Godel and Turing: Essays on Complexity, 1970–2007. [find](https://scholar.google.com/scholar?q=Gregory+J%2E+Chaitin%2E+On+the+intelligibility+of+the+universe+and+the+notions+of+simplicity%2C+complexity+and+irreducibility%2E+In+Thinking+about+Godel+and+Turing%3A+Essays+on+Complexity%2C+1970%E2%80%932007%2E)
6. World Scientific, 2007. [find](https://scholar.google.com/scholar?q=World+Scientific%2C+2007%2E)
7. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. [QuAC: Question answering in context](https://md.research.elamur.ai/free/en/papers/choi-2018-quac-question-answering-context/). In Proc. of EMNLP, 2018.
8. Andrew M. Dai and Quoc V. Le. Semi-supervised sequence learning. In Proc. of NIPS, 2015. [find](https://scholar.google.com/scholar?q=Andrew+M%2E+Dai+and+Quoc+V%2E+Le%2E+Semi%2Dsupervised+sequence+learning%2E+In+Proc%2E+of+NIPS%2C+2015%2E)
9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint, 2018.
10. Tim Van Erven, Peter Grunwald, and Steven De Rooij. Catching up faster by switching sooner: A predictive approach to adaptive estimation with an application to the AIC-BIC dilemma. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 74(3):361––417, 2012. [find](https://scholar.google.com/scholar?q=Tim+Van+Erven%2C+Peter+Grunwald%2C+and+Steven+De+Rooij%2E+Catching+up+faster+by+switching+sooner%3A+A+predictive+approach+to+adaptive+estimation+with+an+application+to+the+AIC%2DBIC+dilemma%2E+Journal+of+the+Roya)
11. Chelsea Finn, Pieter Abbeel, and Sergey Levine. [Model-agnostic meta-learning for fast adaptation of deep networks](https://md.research.elamur.ai/free/en/papers/finn-2017-model-agnostic-meta-learning-fast/). In Proc. of ICML, 2017.
12. Nicholas FitzGerald, Julian Michael, Luheng He, and Luke Zettlemoyer. [Large-scale QA-SRL parsing](https://md.research.elamur.ai/free/en/papers/fitzgerald-2018-large-scale-qa-srl-parsing/). In Proc. of ACL, 2018.
13. Robert M French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3(4):128–135, 1999. [find](https://scholar.google.com/scholar?q=Robert+M+French%2E+Catastrophic+forgetting+in+connectionist+networks%2E+Trends+in+Cognitive+Sciences%2C+3%284%29%3A128%E2%80%93135%2C+1999%2E)
14. Edouard Grave, Armand Joulin, and Nicolas Usunier. [Improving neural language models with a continuous cache](https://md.research.elamur.ai/free/en/papers/grave-2016-improving-neural-language-models-continuous/). In Proc. of ICLR, 2017.
15. Isabelle Guyon, Gavin Cawley, Gideon Dror, and Vincent Lemaire. Results of the active learning challenge. In Proc. of Workshop on Active Learning and Experimental Design, 2011. [find](https://scholar.google.com/scholar?q=Isabelle+Guyon%2C+Gavin+Cawley%2C+Gideon+Dror%2C+and+Vincent+Lemaire%2E+Results+of+the+active+learning+challenge%2E+In+Proc%2E+of+Workshop+on+Active+Learning+and+Experimental+Design%2C+2011%2E)
16. Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zh… — [Achieving Human Parity on Automatic Chinese to English News Translation](https://md.research.elamur.ai/free/en/papers/hassan-2018-achieving-human-parity-automatic-chinese/)
17. Sepp Hochreiter and Jurgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735–1780, 1997. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Jurgen+Schmidhuber%2E+Long+short%2Dterm+memory%2E+Neural+Computation%2C+9%288%29%3A+1735%E2%80%931780%2C+1997%2E)
18. Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu. Population based training of neural networks. arXiv preprint, 2017. [find](https://scholar.google.com/scholar?q=Max+Jaderberg%2C+Valentin+Dalibard%2C+Simon+Osindero%2C+Wojciech+M%2E+Czarnecki%2C+Jeff+Donahue%2C+Ali+Razavi%2C+Oriol+Vinyals%2C+Tim+Green%2C+Iain+Dunning%2C+Karen+Simonyan%2C+Chrisantha+Fernando%2C+and+Koray+Kavukcuoglu%2E+P)
19. Robin Jia and Percy Liang. [Adversarial examples for evaluating reading comprehension systems](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/). In Proc. of EMNLP, 2017.
20. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. [Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). In Proc. of ACL, 2017.
21. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proc… [find](https://scholar.google.com/scholar?q=James+Kirkpatrick%2C+Razvan+Pascanu%2C+Neil+Rabinowitz%2C+Joel+Veness%2C+Guillaume+Desjardins%2C+Andrei+A%2E+Rusu%2C+Kieran+Milan%2C+John+Quan%2C+Tiago+Ramalho%2C+Agnieszka+Grabska%2DBarwinska%2C+Demis+Hassabis%2C+Claudia+Clop)
22. Ben Krause, Emmanuel Kahembwe, Iain Murray, and Steve Renals. Dynamic evaluation of sequence models. In Proc. of ICML, 2018. [find](https://scholar.google.com/scholar?q=Ben+Krause%2C+Emmanuel+Kahembwe%2C+Iain+Murray%2C+and+Steve+Renals%2E+Dynamic+evaluation+of+sequence+models%2E+In+Proc%2E+of+ICML%2C+2018%2E)
23. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Rhinehart, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural Quest… — [Natural Questions: A Benchmark for Question Answering Research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
24. Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. [Zero-shot relation extraction via reading comprehension](https://md.research.elamur.ai/free/en/papers/levy-2017-zero-shot-relation-extraction-reading/). In Proc. of CoNLL, 2017.
25. Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. [The natural language decathlon: Multitask learning as question answering](https://md.research.elamur.ai/free/en/papers/mccann-2018-natural-language-decathlon-multitask-learning/). arXiv preprint, 2018.
26. Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. The Psychology of Learning and Motivation, 24(92):109–165, 1989. [find](https://scholar.google.com/scholar?q=Michael+McCloskey+and+Neal+J+Cohen%2E+Catastrophic+interference+in+connectionist+networks%3A+The+sequential+learning+problem%2E+The+Psychology+of+Learning+and+Motivation%2C+24%2892%29%3A109%E2%80%93165%2C+1989%2E)
27. Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint, 2018. [find](https://scholar.google.com/scholar?q=Alex+Nichol%2C+Joshua+Achiam%2C+and+John+Schulman%2E+On+first%2Dorder+meta%2Dlearning+algorithms%2E+arXiv+preprint%2C+2018%2E)
28. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. [Deep contextualized word representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/). In Proc. of NAACL, 2018.
29. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. arXiv preprint, 2018. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+arXiv+preprint%2C+2018%2E)
30. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. [SQuAD: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In Proc. of EMNLP, 2016.
31. Jonathan Schwarz, Jelena Luketina, Wojciech M. Czarnecki, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress and compress: A scalable framework for continual learning. In Proc. of ICML, 2018. [find](https://scholar.google.com/scholar?q=Jonathan+Schwarz%2C+Jelena+Luketina%2C+Wojciech+M%2E+Czarnecki%2C+Agnieszka+Grabska%2DBarwinska%2C+Yee+Whye+Teh%2C+Razvan+Pascanu%2C+and+Raia+Hadsell%2E+Progress+and+compress%3A+A+scalable+framework+for+continual+learnin)
32. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hananneh Hajishirzi. Bi-directional attention flow for machine comprehension. In Proc. of ICLR, 2017. [find](https://scholar.google.com/scholar?q=Minjoon+Seo%2C+Aniruddha+Kembhavi%2C+Ali+Farhadi%2C+and+Hananneh+Hajishirzi%2E+Bi%2Ddirectional+attention+flow+for+machine+comprehension%2E+In+Proc%2E+of+ICLR%2C+2017%2E)
33. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Proc. of NIPS, 2017.
34. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). arXiv preprint, 2018.
35. Adina Williams, Nikita Nangia, and Samuel Bowman. [A broad-coverage challenge corpus for sentence understanding through inference](https://md.research.elamur.ai/free/en/papers/williams-2018-broad-coverage-challenge-corpus-sentence/). In Proc. of NAACL-HLT, 2018.
36. J. Gerard Wolff. Language acquisition, data compression and generalization. Language & Communication, 2(1):57–89, 1982. [find](https://scholar.google.com/scholar?q=J%2E+Gerard+Wolff%2E+Language+acquisition%2C+data+compression+and+generalization%2E+Language+%26+Communication%2C+2%281%29%3A57%E2%80%9389%2C+1982%2E)

*Updated 2 November 2026*