---
title: "Natural language understanding — definition and key papers"
description: "Natural language understanding (NLU) is the branch of language technology that maps text or speech onto representations of what it means…"
canonical: "https://research.elamur.ai/free/en/concepts/natural-language-understanding/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/concepts/natural-language-understanding/)

[home](https://md.research.elamur.ai/free/en/) / [concepts](https://md.research.elamur.ai/free/en/concepts/) / Natural language understanding

# 💬 Natural language understanding

Field · 41 key papers · [all concepts](https://md.research.elamur.ai/free/en/concepts/)

Part of: [🗨️ Natural language processing](https://md.research.elamur.ai/free/en/concepts/natural-language-processing/) · [❓ Question answering](https://md.research.elamur.ai/free/en/concepts/question-answering/) · [💭 Chatbots](https://md.research.elamur.ai/free/en/concepts/chatbot/)

Natural language understanding (NLU) is the branch of language technology that maps text or speech onto representations of what it means, such as intents, entities, relations, logical forms or inferences. A system then uses these representations to answer, act or reason.

Also: NLU, language understanding, natural language comprehension, machine reading comprehension (MRC), spoken language understanding (SLU)

About 17 minutes to read · every paper named here has its own page in this library

**On this page**

1. [What it is](#a-what-it-is)
2. [How it works](#a-how-it-works)
3. [History](#a-history)
4. [Methods](#a-methods)
5. [Datasets and benchmarks](#a-datasets-and-benchmarks)
6. [People and institutions](#a-people-and-institutions)
7. [Open questions](#a-open-questions)
8. [The library in figures](#a-in-figures)

## What it is

When someone tells a phone assistant "wake me at seven tomorrow", the software has to recognise the request (setting an alarm) and pull out its details (the time, the day). When a student asks a search engine who won a battle, a system has to find a passage and pick out the answer. Both jobs belong to natural language understanding: getting from a string of words to something a program can act on.

NLU is usually placed inside [natural language processing](https://md.research.elamur.ai/free/en/concepts/natural-language-processing/), the wider field that covers every computational treatment of language, from tokenizing to translating. It is often contrasted with natural language generation, which goes the other way, from data or meaning to text. The split is useful but not sharp. Many tasks need both directions, and some modern systems are trained to do both with a single network ([Dong et al. 2019](https://md.research.elamur.ai/free/en/papers/dong-2019-unified-language-model-pre-training/)).

"Understanding" in NLU is an engineering label. It names a family of tasks on which a system can be scored. It is not a claim that the system grasps language the way a person does. Whether such a claim could ever be justified is a live research question (see Open questions).

The component tasks fall into a few groups:

- Intent detection and slot filling. This is the core of voice assistants and task-oriented [chatbots](https://md.research.elamur.ai/free/en/concepts/chatbot/): classify the user's goal and fill a frame of arguments ([Coucke et al. 2018](https://md.research.elamur.ai/free/en/papers/coucke-2018-snips-voice-platform-embedded-spoken/)).
- Textual entailment, also called natural language inference (NLI). The system decides whether a hypothesis follows from a premise, contradicts it, or is neither ([Parikh et al. 2016](https://md.research.elamur.ai/free/en/papers/parikh-2016-decomposable-attention-model-natural-language/)).
- Reading comprehension and [question answering](https://md.research.elamur.ai/free/en/concepts/question-answering/). The system answers questions about a passage ([Rajpurkar et al. 2016](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)).
- [Semantic parsing](https://md.research.elamur.ai/free/en/concepts/semantic-parsing/). The system translates a sentence into a logical form or an executable query.
- [Coreference resolution](https://md.research.elamur.ai/free/en/concepts/coreference-resolution/). The system links the expressions that refer to the same entity.
- Commonsense inference. The system fills in what a text leaves unsaid about people, events and the physical world ([Sap et al. 2019](https://md.research.elamur.ai/free/en/papers/sap-2019-socialiqa-commonsense-reasoning-about-social/)).

Supporting analyses feed these tasks: [named entity recognition](https://md.research.elamur.ai/free/en/concepts/named-entity-recognition/), [word sense disambiguation](https://md.research.elamur.ai/free/en/concepts/word-sense-disambiguation/), [semantic role labeling](https://md.research.elamur.ai/free/en/concepts/semantic-role-labeling/) and [dependency parsing](https://md.research.elamur.ai/free/en/concepts/dependency-parsing/).

## How it works

### From words to representations

Every NLU system first turns text into units, typically words or subword tokens (see [subword tokenization](https://md.research.elamur.ai/free/en/concepts/subword-tokenization/)). Each unit is then mapped to a vector.

Early neural systems used fixed [word embeddings](https://md.research.elamur.ai/free/en/concepts/word-embeddings/). These rest on [distributional semantics](https://md.research.elamur.ai/free/en/concepts/distributional-semantics/), an idea usually traced to Harris (1954) and Firth (1957). Contemporary systems instead compute contextual vectors with a pretrained [Transformer](https://md.research.elamur.ai/free/en/concepts/transformer/) encoder, the architecture introduced by Vaswani et al. (2017).

BERT pre-trains such an encoder by hiding some input tokens and asking the network to restore them from both sides of the context. A second objective predicts whether two sentences follow each other. The same network is then fine-tuned for a target task by adding a small output layer ([Devlin et al. 2019](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)).

### Task heads

On top of the encoder, each task needs its own output head.

- Classification tasks such as sentiment, entailment or intent read a pooled sentence vector and apply a softmax over labels.
- Sequence labelling tasks such as slots or entities predict a tag for every token.
- Extractive reading comprehension predicts where an answer starts and ends inside the passage. BiDAF does this ([Seo et al. 2016](https://md.research.elamur.ai/free/en/papers/seo-2016-bidirectional-attention-flow-machine-comprehension/)), and so does fine-tuned BERT on SQuAD ([Devlin et al. 2019](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)).

The standard span model, as in BiDAF, scores each passage position i ([Seo et al. 2016](https://md.research.elamur.ai/free/en/papers/seo-2016-bidirectional-attention-flow-machine-comprehension/)):

```
P(start = i) = exp(wₛ · hᵢ) / Σⱼ exp(wₛ · hⱼ)
P(end = k)   = exp(wₑ · hₖ) / Σⱼ exp(wₑ · hⱼ)
```

Here hᵢ is the contextual vector of the i-th passage token, and wₛ and wₑ are learned weight vectors for the start and the end. The sum over j runs over all passage tokens. The predicted answer is the span (i, k), with k not before i, that maximises the product of the two probabilities.

Training minimises the negative log-probability of the gold start and end positions. Gated self-matching networks use the same loss, but they produce the two positions with a pointer network rather than with two independent softmaxes ([Wang et al. 2017](https://md.research.elamur.ai/free/en/papers/wang-2017-gated-self-matching-networks-reading/)).

### Pairs of texts

Inference and question answering compare two texts. Before pretraining took over, much of the design effort went into how the two texts interact.

- The decomposable attention model softly aligned words across premise and hypothesis, compared the aligned pieces and summed the evidence. This proved enough to rival large recurrent encoders on entailment ([Parikh et al. 2016](https://md.research.elamur.ai/free/en/papers/parikh-2016-decomposable-attention-model-natural-language/)).
- BiDAF let attention run from passage to question and from question to passage at every position, without collapsing the passage into a single vector ([Seo et al. 2016](https://md.research.elamur.ai/free/en/papers/seo-2016-bidirectional-attention-flow-machine-comprehension/)).
- In a BERT-style encoder, both texts are simply concatenated with a separator, and self-attention handles the interaction ([Devlin et al. 2019](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)).

### Spoken input

In a voice assistant, NLU sits after speech recognition. The Snips platform trained its recognizer's language model jointly with the intent-and-slot engine, so that the two are consistent and specialised to each assistant's domain. The whole pipeline ran offline, in real time, on a Raspberry Pi 3 ([Coucke et al. 2018](https://md.research.elamur.ai/free/en/papers/coucke-2018-snips-voice-platform-embedded-spoken/)).

## History

Rule-based understanding goes back to a few early programs: Weizenbaum's ELIZA (1966), Woods's LUNAR question-answering system over a lunar-rock database (1973), and Winograd's SHRDLU (1972). ELIZA relied on pattern matching rather than any analysis of meaning. LUNAR and SHRDLU worked inside narrow, hand-built worlds.

From 2005, the PASCAL Recognizing Textual Entailment challenges moved evaluation to open text by framing inference as a shared task. The third round appeared in 2007 (Giampiccolo et al. 2007). MCTest, released in 2013, offered fictional children's stories with multiple-choice questions. On its MC500 set a lexical baseline answered about 58% of test questions, and an off-the-shelf entailment system did worse than that baseline ([Richardson et al. 2013](https://md.research.elamur.ai/free/en/papers/richardson-2013-mctest-challenge-dataset-open-domain/)).

The decisive change was scale. In 2015 SNLI supplied entailment data large enough to train neural networks (Bowman et al. 2015), with about 570k labelled sentence pairs ([Conneau et al. 2017](https://md.research.elamur.ai/free/en/papers/conneau-2017-supervised-learning-universal-sentence-representations/)). SQuAD followed in 2016 for span-based reading, with 107,785 question–answer pairs over 536 Wikipedia articles ([Rajpurkar et al. 2016](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)).

A burst of neural readers came next:

- BiDAF ([Seo et al. 2016](https://md.research.elamur.ai/free/en/papers/seo-2016-bidirectional-attention-flow-machine-comprehension/));
- gated self-matching networks ([Wang et al. 2017](https://md.research.elamur.ai/free/en/papers/wang-2017-gated-self-matching-networks-reading/));
- the recurrence-free QANet ([Yu et al. 2018](https://md.research.elamur.ai/free/en/papers/yu-2018-qanet-combining-local-convolution-global/)).

In the same period, sentence encoders trained on entailment data were shown to transfer to many downstream tasks ([Conneau et al. 2017](https://md.research.elamur.ai/free/en/papers/conneau-2017-supervised-learning-universal-sentence-representations/)).

GLUE, released in 2018, gathered nine English sentence-level tasks under one score and one leaderboard ([Wang et al. 2018](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)). A few months later BERT raised the best GLUE score by 7.7 points, to 80.5, and set new results on eleven tasks ([Devlin et al. 2019](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)). The previous best had been OpenAI GPT's 72.8 ([Dong et al. 2019](https://md.research.elamur.ai/free/en/papers/dong-2019-unified-language-model-pre-training/)).

A wave of variants followed:

- multi-task fine-tuning ([Liu et al. 2019](https://md.research.elamur.ai/free/en/papers/liu-2019-multi-task-deep-neural-networks/));
- permutation objectives ([Yang et al. 2019](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/));
- continual multi-task pretraining ([Sun et al. 2019](https://md.research.elamur.ai/free/en/papers/sun-2019-ernie-2-0-continual/));
- hybrids of the masked and permuted objectives ([Song et al. 2020](https://md.research.elamur.ai/free/en/papers/song-2020-mpnet-masked-permuted-pre-training/)).

As models approached GLUE's own estimate of non-expert human performance, SuperGLUE was first released in May 2019 as a harder successor. By July 2019 the top GLUE entry had passed that estimate ([Wang et al. 2019](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)).

The same years produced a critical literature. It showed that high scores could rest on artefacts of how datasets were built (Gururangan et al. 2018; Poliak et al. 2018; [McCoy et al. 2019](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/)). A related, more fundamental argument, made in 2020, held that training on form alone cannot yield meaning ([Bender & Koller 2020](https://md.research.elamur.ai/free/en/papers/bender-2020-climbing-nlu-meaning-form-understanding/)).

With [large language models](https://md.research.elamur.ai/free/en/concepts/large-language-models/), evaluation moved again. The focus shifted to broad knowledge exams taken without fine-tuning ([Hendrycks et al. 2020](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/)) and to their harder successors ([Wang et al. 2024](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/)).

Papers on this subject by year 343 dated papers from 1987 to 2026; the busiest year is 2019 with 56, and 76% of them appeared in 2018 or later.

**The numbers**

| Year | Papers |
| --- | --- |
| 2026 | 14 |
| 2025 | 5 |
| 2024 | 10 |
| 2023 | 25 |
| 2022 | 27 |
| 2021 | 36 |
| 2020 | 56 |
| 2019 | 56 |
| 2018 | 32 |
| 2017 | 20 |
| 2016 | 14 |
| 2015 | 9 |
| 2014 | 8 |
| 2013 | 6 |
| 2012 | 3 |
| 2011 | 6 |
| 2010 | 4 |
| 2009 | 2 |
| 2008 | 4 |
| 2007 | 2 |
| 2006 | 1 |
| 2005 | 1 |
| 2004 | 1 |
| 1987 | 1 |

Landmark papers on this subject: the 12 most cited, in time The 12 most cited papers on this subject span 2016–2022; the dot grows with the citations.

**The numbers**

| Paper | Citations |
| --- | --- |
| [2016 — SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/) | 9,513 |
| [2016 — Bidirectional Attention Flow for Machine Comprehension](https://md.research.elamur.ai/free/en/papers/seo-2016-bidirectional-attention-flow-machine-comprehension/) | 1,968 |
| [2017 — TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/) | 3,213 |
| [2017 — Adversarial Examples for Evaluating Reading Comprehension Systems](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/) | 1,942 |
| [2018 — GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/) | 7,169 |
| [2019 — BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/) | 55,571 |
| [2019 — XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/) | 3,789 |
| [2019 — SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/) | 2,190 |
| [2019 — Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/) | 1,635 |
| [2020 — Measuring Massive Multitask Language Understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/) | 3,495 |
| [2021 — Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/) | 1,803 |
| [2022 — Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://md.research.elamur.ai/free/en/papers/saharia-2022-photorealistic-text-image-diffusion-models/) | 2,423 |

## Methods

### Neural readers and matching networks

Before pretraining, architecture engineering dominated.

BiDAF stacked six layers: character embedding, word embedding, a contextual recurrent layer, bidirectional attention flow, a modelling layer and a span output layer. When it was submitted it led the SQuAD leaderboard ([Seo et al. 2016](https://md.research.elamur.ai/free/en/papers/seo-2016-bidirectional-attention-flow-machine-comprehension/)).

Gated self-matching networks added a relevance gate on the passage and let the passage attend to itself to collect distant evidence. A single model reached 71.3% exact match on the hidden SQuAD test set ([Wang et al. 2017](https://md.research.elamur.ai/free/en/papers/wang-2017-gated-self-matching-networks-reading/)).

QANet replaced recurrence with convolution and self-attention, which made it train several times faster. It spent the saved compute on back-translation augmentation. A single model scored 84.6 F1 on the SQuAD test set ([Yu et al. 2018](https://md.research.elamur.ai/free/en/papers/yu-2018-qanet-combining-local-convolution-global/)).

DiSAN showed that direction-masked self-attention alone could encode sentences competitively, without recurrent or convolutional layers (Shen et al. 2017). For entailment, the decomposable attention model matched the best published SNLI accuracy, 86.3%, with roughly a tenth of the parameters of the strongest recurrent model ([Parikh et al. 2016](https://md.research.elamur.ai/free/en/papers/parikh-2016-decomposable-attention-model-natural-language/)).

Moving from paragraphs to documents raised a further problem: scores from different paragraphs have to be comparable. Clark and Gardner trained on paragraphs both with and without the answer, and normalised the softmax jointly across paragraphs. This made confidence-based selection over many paragraphs work on TriviaQA ([Clark & Gardner 2017](https://md.research.elamur.ai/free/en/papers/clark-2017-simple-effective-multi-paragraph-reading/)).

### Pretrained encoders

Pretraining changed the recipe to "pretrain once, fine-tune everywhere". BERT's masked-token objective, described under How it works, gave bidirectional context ([Devlin et al. 2019](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)).

- XLNet kept an autoregressive model but sampled random factorisation orders. On average, each token learns from both sides without seeing artificial mask symbols. In comparable settings it beat BERT on 20 tasks ([Yang et al. 2019](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/)).
- MPNet folded the masked and permuted objectives into one framework and added position information for the tokens being predicted ([Song et al. 2020](https://md.research.elamur.ai/free/en/papers/song-2020-mpnet-masked-permuted-pre-training/)).
- DeBERTa kept separate content and position vectors for each token and built attention from their cross-terms. Its 1.5B-parameter version was the first single model to pass the SuperGLUE human baseline ([He et al. 2020](https://md.research.elamur.ai/free/en/papers/he-2020-deberta-decoding-enhanced-bert-disentangled/)).
- ERNIE 2.0 added seven pretraining tasks in sequence, each trained jointly with those before it. The tasks ranged from knowledge masking to discourse relations ([Sun et al. 2019](https://md.research.elamur.ai/free/en/papers/sun-2019-ernie-2-0-continual/)).

The approach spread to other languages. AraBERT, for example, trained a BERT-base model on about 24GB of Arabic text that had been morphologically segmented before subword tokenization ([Antoun et al. 2020](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/)).

Domain adaptation could be done cheaply. Post-training BERT on in-domain reviews and on general reading-comprehension data improved review question answering and aspect-based sentiment analysis ([Xu et al. 2019](https://md.research.elamur.ai/free/en/papers/xu-2019-bert-post-training-review-reading/)). For deployment, knowledge distillation compresses large encoders into small students (Jiao et al. 2020).

### Multi-task learning, prompts and few-shot learning

MT-DNN refined a shared BERT encoder on all GLUE tasks at once, with one head per task type. Its representations adapted to new domains with very little data. With 0.1% of the SNLI training set it reached 82.1% dev accuracy, against 52.5% for plain BERT ([Liu et al. 2019](https://md.research.elamur.ai/free/en/papers/liu-2019-multi-task-deep-neural-networks/)).

Intermediate training on a related labelled task before the target task also helps; Phang et al. (2018) called this approach STILTs. In the SuperGLUE baselines, an extra MultiNLI stage gained 2–5 points on BoolQ, CB and RTE, and an extra SWAG stage gained 8 points on COPA ([Wang et al. 2019](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)).

Pattern-Exploiting Training turns inputs into cloze questions and maps each label to a single word. It fine-tunes one masked model per pattern, then distils the ensemble on unlabelled text. This makes classification and inference with [few-shot learning](https://md.research.elamur.ai/free/en/concepts/few-shot-learning/) practical ([Schick & Schütze 2021](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/)). It also anticipates the prompt engineering of later large models.

UniLM showed that one network can serve both understanding and generation by switching only its self-attention mask ([Dong et al. 2019](https://md.research.elamur.ai/free/en/papers/dong-2019-unified-language-model-pre-training/)).

### Explanations and knowledge

Another line of work asks systems to justify their answers. e-SNLI added free-text human explanations to every SNLI pair. A model seeing only the hypothesis could guess labels from artefacts, but those artefacts helped much less when the model also had to produce a correct explanation ([Camburu et al. 2018](https://md.research.elamur.ai/free/en/papers/camburu-2018-e-snli-natural-language-inference/)).

On CommonsenseQA, one system first fine-tuned a language model to generate explanations and then fed its output to a classifier. It reached about 65% accuracy on version 1.0 ([Rajani et al. 2019](https://md.research.elamur.ai/free/en/papers/rajani-2019-explain-yourself-leveraging-language-models/)).

Links to [knowledge representation](https://md.research.elamur.ai/free/en/concepts/knowledge-representation/) appear in resources such as SocialIQA. Its questions follow the inference dimensions of a commonsense [knowledge graph](https://md.research.elamur.ai/free/en/concepts/knowledge-graph/) ([Sap et al. 2019](https://md.research.elamur.ai/free/en/papers/sap-2019-socialiqa-commonsense-reasoning-about-social/)).

### Beyond text

Language understanding also matters in multimodal models. Imagen, a text-to-image generator, found that enlarging its frozen text encoder improved image fidelity and prompt alignment more than enlarging the image generator did ([Saharia et al. 2022](https://md.research.elamur.ai/free/en/papers/saharia-2022-photorealistic-text-image-diffusion-models/)). BLIP ([Li et al. 2022](https://md.research.elamur.ai/free/en/papers/li-2022-blip-bootstrapping-language-image-pre/)) and MiniGPT-4 ([Zhu et al. 2023](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/)) treat vision-language understanding as aligning images with a language model's representations.

## Datasets and benchmarks

Arguably, [evaluation benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) have shaped NLU more than any single method. The GLUE, SuperGLUE and MMLU timelines below suggest a cycle:

1. A new dataset exposes a gap between models and people.
2. Models close much of the gap, often within a year or two.
3. Analysis shows that part of the closing came from shortcuts.
4. A harder dataset follows.

Citations of the papers on this subject, by when they were published Papers published in 2015–2019 gather 76% of the 177,118 citations of the papers on this subject.

**The numbers**

| Published | Papers and their citations |
| --- | --- |
| 2025–2029 | 19 papers, 177 citations |
| 2020–2024 | 154 papers, 36,859 citations |
| 2015–2019 | 131 papers, 134,114 citations |
| 2010–2014 | 27 papers, 4,051 citations |
| 2005–2009 | 10 papers, 1,803 citations |
| 2000–2004 | 1 papers, 30 citations |
| 1985–1989 | 1 papers, 84 citations |

### Entailment

The RTE challenges set the task format (Giampiccolo et al. 2007). SNLI then supplied the training scale described under History (Bowman et al. 2015). Adversarial NLI later collected examples over several rounds, with annotators writing items specifically to fool strong models (Nie et al. 2020).

### Reading comprehension

The reading datasets differ in how answers are formed and in which skill each is meant to test:

| Benchmark | Answer format | What it adds |
| --- | --- | --- |
| SQuAD | span of a Wikipedia paragraph | scale; feature baseline 51.0% F1 against human 86.8% ([Rajpurkar et al. 2016](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)) |
| NewsQA | span in CNN articles, some unanswerable | questions written from the headline and summary highlights only; human 69.4 F1 ([Trischler et al. 2017](https://md.research.elamur.ai/free/en/papers/trischler-2017-newsqa-machine-comprehension-dataset/)) |
| TriviaQA | span in web or Wikipedia evidence | questions written independently of evidence; BiDAF about 40% against humans near 80% ([Joshi et al. 2017](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/)) |
| RACE | four-option multiple choice from school exams | neural readers near 43–44% against a ceiling near 95% ([Lai et al. 2017](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/)) |
| MS MARCO | free-form answers to real search queries | 1,010,916 questions ([Bajaj et al. 2016](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/)) |
| MultiRC | multiple correct options, multi-sentence | human 86.4 F1m on dev ([Khashabi et al. 2018](https://md.research.elamur.ai/free/en/papers/khashabi-2018-looking-beyond-surface-challenge-set/)) |
| DROP | numbers, dates or spans needing arithmetic | BERT 32.7% F1 against expert humans at 96.4% ([Dua et al. 2019](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/)) |

NarrativeQA asks questions about whole books and film scripts ([Kočiský et al. 2018](https://md.research.elamur.ai/free/en/papers/kocisky-2018-narrativeqa-reading-comprehension-challenge/)). Cosmos QA requires commonsense inference about what everyday passages leave unstated ([Huang et al. 2019](https://md.research.elamur.ai/free/en/papers/huang-2019-cosmos-qa-machine-reading-comprehension/)). DROP also tests the boundary with semantic parsing: pipelined parsers found logical forms for only about a third of its training questions ([Dua et al. 2019](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/)).

### Commonsense

SocialIQA has about 38k multiple-choice questions about motives, reactions and consequences in social situations. BERT-large scored 64.5% on it, while humans scored near 90% ([Sap et al. 2019](https://md.research.elamur.ai/free/en/papers/sap-2019-socialiqa-commonsense-reasoning-about-social/)).

Its wrong options were built carefully. Some are correct answers to a related question about the same situation. This design was meant to remove the stylistic cues that otherwise give away distractors written by crowdworkers.

### Aggregate suites and their saturation

GLUE grouped single-sentence, similarity and inference tasks. It added a diagnostic set of 550 expert-built NLI pairs, each tagged by linguistic phenomenon. Its best initial baseline averaged 69.0 ([Wang et al. 2018](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)).

Saturation came fast. In July 2019 XLNet's 88.4 exceeded GLUE's human estimate of 87.1 ([Wang et al. 2019](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)).

SuperGLUE answered with eight harder tasks, including BoolQ, COPA, MultiRC, ReCoRD, WiC and the Winograd schemas (after Levesque, 2011). Coreference and word sense thus entered the suite. At launch, BERT-based systems trailed humans by close to 20 points on average ([Wang et al. 2019](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)). That margin also closed: DeBERTa's single 1.5B model scored 89.9, against a human baseline of 89.8 ([He et al. 2020](https://md.research.elamur.ai/free/en/papers/he-2020-deberta-decoding-enhanced-bert-disentangled/)).

The MMLU authors noted that GLUE, SuperGLUE and commonsense suites had each saturated within about a year. They proposed a 57-subject exam taken zero-shot or few-shot. On it, the 175B GPT-3 reached 43.9%, while expert-level accuracy was estimated near 89.8% ([Hendrycks et al. 2020](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/)).

By 2024 frontier models scored around 86–87% on MMLU. MMLU-Pro then filtered out easy items, raised the options per question from four to ten and cleaned label errors. This lowered accuracy by 16 to 33 percentage points across models ([Wang et al. 2024](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/)).

MS MARCO had followed the same pattern earlier. Machines passed its first human baseline within about 15 months, and the authors had to build a stronger one ([Bajaj et al. 2016](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/)).

Where the papers on this subject were published Annual Meeting of the Association for Computational Linguistics (ACL) carries the most, 67 of 343.

**The numbers**

| Venue | Papers |
| --- | --- |
| [Annual Meeting of the Association for Computational Linguistics (ACL)](https://md.research.elamur.ai/free/en/venues/acl/) | 67 papers |
| [Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/) | 54 papers |
| [Findings of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/findings/) | 24 papers |
| [Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)](https://md.research.elamur.ai/free/en/venues/naacl/) | 20 papers |
| [Language Resources and Evaluation Conference (LREC)](https://md.research.elamur.ai/free/en/venues/lrec/) | 19 papers |
| [Transactions of the Association for Computational Linguistics (TACL)](https://md.research.elamur.ai/free/en/venues/tacl/) | 10 papers |
| [International Conference on Computational Linguistics (COLING)](https://md.research.elamur.ai/free/en/venues/coling/) | 7 papers |
| [Conference of the European Chapter of the Association for Computational Linguistics (EACL)](https://md.research.elamur.ai/free/en/venues/eacl/) | 5 papers |
| [Joint Conference on Lexical and Computational Semantics (*SEM)](https://md.research.elamur.ai/free/en/venues/starsem/) | 5 papers |
| [International Workshop on Semantic Evaluation (SemEval)](https://md.research.elamur.ai/free/en/venues/semeval/) | 4 papers |

### Shortcuts

Many results show that benchmark gains can come from the dataset rather than from understanding.

- Crowd-written entailment hypotheses carry annotation artefacts, such as words that correlate with a label (Gururangan et al. 2018). A classifier that never sees the premise can therefore beat the majority baseline (Poliak et al. 2018).
- Models trained on MNLI adopt syntactic heuristics such as lexical overlap. HANS is a test set built to break these heuristics. On its items whose hypothesis is not entailed, BERT falls well below chance ([McCoy et al. 2019](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/); [Bender & Koller 2020](https://md.research.elamur.ai/free/en/papers/bender-2020-climbing-nlu-meaning-form-understanding/)).
- Niven and Kao (2019) found a similar collapse for BERT on the Argument Reasoning Comprehension Task. Its results dropped to chance once adversarial negated examples were added ([Bender & Koller 2020](https://md.research.elamur.ai/free/en/papers/bender-2020-climbing-nlu-meaning-form-understanding/)).
- Appending distracting but irrelevant sentences to SQuAD passages sharply misleads reading systems ([Jia & Liang 2017](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/)).
- A hand analysis of the CNN/Daily Mail cloze task found that most answerable questions need only one matched sentence. About a quarter are damaged by coreference or anonymisation errors ([Chen et al. 2016](https://md.research.elamur.ai/free/en/papers/chen-2016-thorough-examination-cnn-daily-mail/)).

Dataset designers responded in several ways:

- they wrote questions without seeing the evidence ([Joshi et al. 2017](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/); [Trischler et al. 2017](https://md.research.elamur.ai/free/en/papers/trischler-2017-newsqa-machine-comprehension-dataset/));
- they kept only questions that a running model failed ([Dua et al. 2019](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/));
- they ran human-and-model adversarial loops (Nie et al. 2020);
- they required evidence from several sentences ([Khashabi et al. 2018](https://md.research.elamur.ai/free/en/papers/khashabi-2018-looking-beyond-surface-challenge-set/)).

## People and institutions

A few groups built much of the evaluation infrastructure.

[Samuel R. Bowman](https://md.research.elamur.ai/free/en/authors/samuel-r-bowman/) co-led SNLI (Bowman et al. 2015), GLUE ([Wang et al. 2018](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)) and SuperGLUE ([Wang et al. 2019](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)). He also co-authored the study of annotation artefacts (Gururangan et al. 2018).

[Christopher Manning](https://md.research.elamur.ai/free/en/authors/christopher-d-manning/), at [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/), co-authored SNLI (Bowman et al. 2015) and the critical examination of the CNN/Daily Mail task ([Chen et al. 2016](https://md.research.elamur.ai/free/en/papers/chen-2016-thorough-examination-cnn-daily-mail/)). Also at Stanford, Percy Liang's group produced SQuAD ([Rajpurkar et al. 2016](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)) and its adversarial stress tests ([Jia & Liang 2017](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/)).

[Ido Dagan](https://md.research.elamur.ai/free/en/authors/ido-dagan/), at Bar-Ilan University, helped organise the textual entailment challenges (Giampiccolo et al. 2007).

Commonsense and reading work is strongly associated with [Yejin Choi](https://md.research.elamur.ai/free/en/authors/yejin-choi/), now at Stanford, through SocialIQA ([Sap et al. 2019](https://md.research.elamur.ai/free/en/papers/sap-2019-socialiqa-commonsense-reasoning-about-social/)) and Cosmos QA ([Huang et al. 2019](https://md.research.elamur.ai/free/en/papers/huang-2019-cosmos-qa-machine-reading-comprehension/)).

[Matt Gardner](https://md.research.elamur.ai/free/en/authors/matt-gardner/) co-authored DROP ([Dua et al. 2019](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/)) and the shared-normalisation method for multi-paragraph reading ([Clark & Gardner 2017](https://md.research.elamur.ai/free/en/papers/clark-2017-simple-effective-multi-paragraph-reading/)). Both were built in the ecosystem of the [Allen Institute for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/allen-institute-for-artificial-intelligence/).

Several other authors each co-created a major dataset:

- [Luke Zettlemoyer](https://md.research.elamur.ai/free/en/authors/luke-zettlemoyer/), at the [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) and [Meta](https://md.research.elamur.ai/free/en/companies/meta/), co-created TriviaQA ([Joshi et al. 2017](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/)).
- [Eduard Hovy](https://md.research.elamur.ai/free/en/authors/eduard-hovy/), now at the University of Melbourne, co-authored RACE ([Lai et al. 2017](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/)).
- [Dan Roth](https://md.research.elamur.ai/free/en/authors/dan-roth/), at the University of Pennsylvania, co-authored MultiRC ([Khashabi et al. 2018](https://md.research.elamur.ai/free/en/papers/khashabi-2018-looking-beyond-surface-challenge-set/)).
- [Mohit Bansal](https://md.research.elamur.ai/free/en/authors/mohit-bansal/), at the University of North Carolina at Chapel Hill, co-authored Adversarial NLI (Nie et al. 2020).

The diagnosis of entailment shortcuts owes much to three researchers who together introduced hypothesis-only baselines (Poliak et al. 2018):

- [Adam Poliak](https://md.research.elamur.ai/free/en/authors/adam-poliak/), now at Bryn Mawr College;
- [Rachel Rudinger](https://md.research.elamur.ai/free/en/authors/rachel-rudinger/), now at the University of Maryland, College Park;
- [Benjamin Van Durme](https://md.research.elamur.ai/free/en/authors/benjamin-van-durme/), at [Johns Hopkins University](https://md.research.elamur.ai/free/en/universities/johns-hopkins-university/) and [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/).

Industrial labs, often working with universities, drove much of the modelling. [Google](https://md.research.elamur.ai/free/en/companies/google/) produced BERT ([Devlin et al. 2019](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)). [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) and Google Brain together produced XLNet ([Yang et al. 2019](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/)) and QANet ([Yu et al. 2018](https://md.research.elamur.ai/free/en/papers/yu-2018-qanet-combining-local-convolution-global/)).

At Microsoft, [Jianfeng Gao](https://md.research.elamur.ai/free/en/authors/jianfeng-gao/) and [Xiaodong Liu](https://md.research.elamur.ai/free/en/authors/xiaodong-liu/) co-authored MT-DNN ([Liu et al. 2019](https://md.research.elamur.ai/free/en/papers/liu-2019-multi-task-deep-neural-networks/)), DeBERTa ([He et al. 2020](https://md.research.elamur.ai/free/en/papers/he-2020-deberta-decoding-enhanced-bert-disentangled/)) and MS MARCO ([Bajaj et al. 2016](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/)). [Furu Wei](https://md.research.elamur.ai/free/en/authors/furu-wei/), also at Microsoft, co-authored gated self-matching networks ([Wang et al. 2017](https://md.research.elamur.ai/free/en/papers/wang-2017-gated-self-matching-networks-reading/)) and UniLM ([Dong et al. 2019](https://md.research.elamur.ai/free/en/papers/dong-2019-unified-language-model-pre-training/)).

[Caiming Xiong](https://md.research.elamur.ai/free/en/authors/caiming-xiong/), at Salesforce, co-authored the explanation-based commonsense work ([Rajani et al. 2019](https://md.research.elamur.ai/free/en/papers/rajani-2019-explain-yourself-leveraging-language-models/)) and BLIP ([Li et al. 2022](https://md.research.elamur.ai/free/en/papers/li-2022-blip-bootstrapping-language-image-pre/)).

Authors with the most papers on this subject Matt Gardner leads with 10 papers; the 12 authors shown wrote 82 of the papers on this subject between them, counting each paper once per author.

**The numbers**

| Author | Papers here · their citations |
| --- | --- |
| [Matt Gardner](https://md.research.elamur.ai/free/en/authors/matt-gardner/) | 10 papers · 3,495 citations |
| [Yejin Choi](https://md.research.elamur.ai/free/en/authors/yejin-choi/) | 10 papers · 3,377 citations |
| [Benjamin Van Durme](https://md.research.elamur.ai/free/en/authors/benjamin-van-durme/) | 8 papers · 1,307 citations |
| [Caiming Xiong](https://md.research.elamur.ai/free/en/authors/caiming-xiong/) | 7 papers · 4,242 citations |
| [Christopher D. Manning](https://md.research.elamur.ai/free/en/authors/christopher-d-manning/) | 7 papers · 3,619 citations |
| [Adam Poliak](https://md.research.elamur.ai/free/en/authors/adam-poliak/) | 7 papers · 705 citations |
| [Samuel R. Bowman](https://md.research.elamur.ai/free/en/authors/samuel-r-bowman/) | 6 papers · 9,956 citations |
| [Dan Roth](https://md.research.elamur.ai/free/en/authors/dan-roth/) | 6 papers · 1,226 citations |
| [Ting Liu](https://md.research.elamur.ai/free/en/authors/ting-liu/) | 6 papers · 902 citations |
| [Luke Zettlemoyer](https://md.research.elamur.ai/free/en/authors/luke-zettlemoyer/) | 5 papers · 5,071 citations |
| [Jianfeng Gao](https://md.research.elamur.ai/free/en/authors/jianfeng-gao/) | 5 papers · 4,760 citations |
| [Richard Socher](https://md.research.elamur.ai/free/en/authors/richard-socher/) | 5 papers · 2,652 citations |

Where the authors on this subject work now Stanford University is the current home of authors of 37 papers on this subject; counted are the authors with a page here, by their latest institution.

**The numbers**

| Institution | Papers |
| --- | --- |
| [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) | 37 papers |
| [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) | 28 papers |
| [Google](https://md.research.elamur.ai/free/en/companies/google/) | 26 papers |
| [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) | 25 papers |
| [Allen Institute for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/allen-institute-for-artificial-intelligence/) | 21 papers |
| [Google DeepMind](https://md.research.elamur.ai/free/en/companies/google-deepmind/) | 15 papers |
| [New York University](https://md.research.elamur.ai/free/en/universities/new-york-university/) | 15 papers |
| [Amazon](https://md.research.elamur.ai/free/en/companies/amazon/) | 14 papers |
| [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) | 14 papers |
| [Shanghai Jiao Tong University](https://md.research.elamur.ai/free/en/universities/shanghai-jiao-tong-university/) | 14 papers |

Countries of the authors on this subject United States comes first: authors working there now wrote 211 papers on this subject.

**The numbers**

| Country | Papers |
| --- | --- |
| [United States](https://md.research.elamur.ai/free/en/countries/united-states/) | 211 papers |
| [China](https://md.research.elamur.ai/free/en/countries/china/) | 98 papers |
| [United Kingdom](https://md.research.elamur.ai/free/en/countries/united-kingdom/) | 47 papers |
| [Germany](https://md.research.elamur.ai/free/en/countries/germany/) | 34 papers |
| [Canada](https://md.research.elamur.ai/free/en/countries/canada/) | 25 papers |
| [Israel](https://md.research.elamur.ai/free/en/countries/israel/) | 23 papers |
| Hong Kong | 17 papers |
| [Japan](https://md.research.elamur.ai/free/en/countries/japan/) | 14 papers |
| [Singapore](https://md.research.elamur.ai/free/en/countries/singapore/) | 12 papers |
| [United Arab Emirates](https://md.research.elamur.ai/free/en/countries/united-arab-emirates/) | 12 papers |

## Open questions

### Meaning versus form

The deepest dispute concerns what NLU systems learn. Bender and Koller define meaning as a relation between expressions and something outside language:

```
M ⊆ E × I
C ⊆ E × S
```

Here E is the set of linguistic expressions and I the set of communicative intents, so M pairs each expression with what a speaker could mean by it. S is the set of standing, conventional meanings that can be evaluated against a world, and C pairs expressions with those.

On this view, understanding means recovering an intent i from an expression e. A system trained only to predict strings sees E and nothing else, so it receives no signal about I or S ([Bender & Koller 2020](https://md.research.elamur.ai/free/en/papers/bender-2020-climbing-nlu-meaning-form-understanding/)).

Their thought experiment makes the point with an octopus that eavesdrops on two people. The octopus learns the pair's message statistics but cannot respond sensibly when something new happens. The argument echoes Searle (1980) and the grounding problem in cognitive science. It also draws on pragmatics, where meaning depends on intent, as in Grice's (1975) [implicature](https://md.research.elamur.ai/free/en/concepts/implicature/).

The paper itself takes up grounding in distributional semantics. It also reviews probing studies, such as Tenney et al. (2019) and Hewitt and Manning (2019), which show that language models encode formal structure like agreement and dependency structure ([Bender & Koller 2020](https://md.research.elamur.ai/free/en/papers/bender-2020-climbing-nlu-meaning-form-understanding/)). Whether that structure, or richer grounded input, amounts to meaning is exactly what remains contested.

### Evaluation that lasts

GLUE, SuperGLUE and MMLU each lost most of their headroom within a few years of release ([Wang et al. 2019](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/); [He et al. 2020](https://md.research.elamur.ai/free/en/papers/he-2020-deberta-decoding-enhanced-bert-disentangled/); [Wang et al. 2024](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/)). Several fixes are in use:

- adversarial collection (Nie et al. 2020);
- harder formats ([Wang et al. 2024](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/));
- diagnostic sets tagged by phenomenon ([Wang et al. 2018](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)).

All of them buy time without settling what a fixed test set can certify.

Prompt sensitivity adds a further problem. Across 24 prompt wordings, MMLU scores move by 4–5 points, against about 2 points on MMLU-Pro ([Wang et al. 2024](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/)).

Human baselines are themselves unstable. They depend on who the annotators are and on how the baseline is assembled. Crowdworkers score 34.5% on MMLU, against an estimated 89.8% for experts ([Hendrycks et al. 2020](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/)). MS MARCO's baseline had to be rebuilt from its five best editors ([Bajaj et al. 2016](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/)).

### Robustness and reasoning

Shortcut learning remains unsolved. Models that rely on overlap, on artefacts in hypotheses, or on distracting sentences can still post high in-distribution scores ([McCoy et al. 2019](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/); Gururangan et al. 2018; [Jia & Liang 2017](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/)).

Discrete reasoning (counting, arithmetic, comparison) exposed a wide gap for span-extraction readers ([Dua et al. 2019](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/)). Social and contextual commonsense also lagged far behind humans when those datasets were introduced ([Sap et al. 2019](https://md.research.elamur.ai/free/en/papers/sap-2019-socialiqa-commonsense-reasoning-about-social/); [Huang et al. 2019](https://md.research.elamur.ai/free/en/papers/huang-2019-cosmos-qa-machine-reading-comprehension/)).

It is unresolved whether explanation-based training yields more faithful [reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) or only more fluent rationales. Generated explanations transferred to new datasets in fluent form, yet accuracy there fell slightly ([Rajani et al. 2019](https://md.research.elamur.ai/free/en/papers/rajani-2019-explain-yourself-leveraging-language-models/)).

Social bias carried over from training data is a related concern. The explanation work on CommonsenseQA found gender bias inherited from that dataset ([Rajani et al. 2019](https://md.research.elamur.ai/free/en/papers/rajani-2019-explain-yourself-leveraging-language-models/)). In the multimodal setting, Imagen's authors reported skews toward lighter skin tones and Western gender stereotypes, and they withheld the model partly for that reason ([Saharia et al. 2022](https://md.research.elamur.ai/free/en/papers/saharia-2022-photorealistic-text-image-diffusion-models/)).

### Coverage and deployment

Most benchmarks are English. Monolingual models for languages such as Arabic show that a dedicated model can do better than a multilingual one. AraBERT beat multilingual BERT on most of its sentiment, named-entity and question-answering benchmarks ([Antoun et al. 2020](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/)). The broader agenda lives in [multilingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) and [low-resource languages](https://md.research.elamur.ai/free/en/topics/low-resource-languages/).

Practical systems also need small, private and fast models. This motivates distillation (Jiao et al. 2020) and on-device understanding ([Coucke et al. 2018](https://md.research.elamur.ai/free/en/papers/coucke-2018-snips-voice-platform-embedded-spoken/)).

What remains open is how to keep the accuracy of large pretrained encoders while meeting these constraints. A second open problem is making NLU interpretable enough to tell whether a correct answer was reached for the right reasons. That is the concern of research on [interpretability](https://md.research.elamur.ai/free/en/topics/interpretability/).

The shelves its papers stand on Artificial Intelligence: 338 of the 343 papers here.

**The numbers**

| Shelf | Papers |
| --- | --- |
| [Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) | 338 papers |
| [Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) | 338 papers |
| [Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) | 93 papers |
| [Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) | 24 papers |
| [Developmental and Educational Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/developmental-and-educational-psychology/) | 17 papers |
| [Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/molecular-biology/) | 5 papers |
| [Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) | 5 papers |
| [Experimental and Cognitive Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/experimental-and-cognitive-psychology/) | 3 papers |
| [Political Science and International Relations](https://md.research.elamur.ai/free/en/shelves/social-sciences/political-science-and-international-relations/) | 3 papers |
| [Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) | 2 papers |
| [Computer Science Applications](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-science-applications/) | 1 paper |
| [Education](https://md.research.elamur.ai/free/en/shelves/social-sciences/education/) | 1 paper |

## The library in figures

Neighbouring subjects: papers shared Natural language processing shares the most: 301 of the 343 papers here (88%) stand on both.

**The numbers**

| Shelf | Papers shared |
| --- | --- |
| [Natural language processing](https://md.research.elamur.ai/free/en/concepts/natural-language-processing/) | 301 shared · 88% |
| [Question answering](https://md.research.elamur.ai/free/en/concepts/question-answering/) | 93 shared · 27% |
| [Large language models](https://md.research.elamur.ai/free/en/concepts/large-language-models/) | 29 shared · 8% |
| [Few-shot learning](https://md.research.elamur.ai/free/en/concepts/few-shot-learning/) | 16 shared · 5% |
| [Parsing](https://md.research.elamur.ai/free/en/concepts/parsing/) | 16 shared · 5% |
| [Information extraction](https://md.research.elamur.ai/free/en/concepts/information-extraction/) | 14 shared · 4% |
| [Semantics](https://md.research.elamur.ai/free/en/concepts/semantics/) | 14 shared · 4% |
| [Chatbots](https://md.research.elamur.ai/free/en/concepts/chatbot/) | 6 shared · 2% |
| [Knowledge graph](https://md.research.elamur.ai/free/en/concepts/knowledge-graph/) | 5 shared · 1% |
| [Text summarization](https://md.research.elamur.ai/free/en/concepts/text-summarization/) | 5 shared · 1% |

## Key papers

The works this page cites, the most cited first; citations as counted by OpenAlex plus the papers of this library that cite the work, read 2026-10-24.

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/) — Jacob Devlin, Ming-Wei Chang, Kenton Lee et al. · 2019 · 55,571 citations
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/) — Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev et al. · 2016 · 9,513 citations
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/) — Alex Wang, Amanpreet Singh, Julian Michael et al. · 2018 · 7,169 citations
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/) — Zhilin Yang, Zihang Dai, Yiming Yang et al. · 2019 · 3,789 citations
- [Measuring Massive Multitask Language Understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/) — Dan Hendrycks, Collin Burns, Steven Basart et al. · 2020 · 3,495 citations
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/) — Mandar Joshi, Eunsol Choi, Daniel Weld et al. · 2017 · 3,213 citations
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://md.research.elamur.ai/free/en/papers/saharia-2022-photorealistic-text-image-diffusion-models/) — Chitwan Saharia, William Chan, Saurabh Saxena et al. · 2022 · 2,423 citations
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/) — Alex Wang, Yada Pruksachatkun, Nikita Nangia et al. · 2019 · 2,190 citations
- [Bidirectional Attention Flow for Machine Comprehension](https://md.research.elamur.ai/free/en/papers/seo-2016-bidirectional-attention-flow-machine-comprehension/) — Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi et al. · 2016 · 1,968 citations
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/) — Robin Jia, Percy Liang · 2017 · 1,942 citations
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://md.research.elamur.ai/free/en/papers/he-2020-deberta-decoding-enhanced-bert-disentangled/) — Pengcheng He, Xiaodong Liu, Jianfeng Gao et al. · 2020 · 1,808 citations
- [Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/) — Timo Schick, Hinrich Schütze · 2021 · 1,803 citations
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/) — R. Thomas McCoy, Ellie Pavlick, Tal Linzen · 2019 · 1,635 citations
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/) — Guokun Lai, Qizhe Xie, Hanxiao Liu et al. · 2017 · 1,629 citations
- [A Decomposable Attention Model for Natural Language Inference](https://md.research.elamur.ai/free/en/papers/parikh-2016-decomposable-attention-model-natural-language/) — Ankur Parikh, Oscar Täckström, Dipanjan Das et al. · 2016 · 1,601 citations
- [Multi-Task Deep Neural Networks for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/liu-2019-multi-task-deep-neural-networks/) — Xiaodong Liu, Pengcheng He, Weizhu Chen et al. · 2019 · 1,511 citations
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://md.research.elamur.ai/free/en/papers/li-2022-blip-bootstrapping-language-image-pre/) — Junnan Li, Dongxu Li, Caiming Xiong et al. · 2022 · 1,469 citations
- [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://md.research.elamur.ai/free/en/papers/dong-2019-unified-language-model-pre-training/) — Li Dong, Nan Yang, Wenhui Wang et al. · 2019 · 1,406 citations
- [Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data](https://md.research.elamur.ai/free/en/papers/bender-2020-climbing-nlu-meaning-form-understanding/) — Emily M. Bender, Alexander Koller · 2020 · 1,375 citations
- [AraBERT: Transformer-based Model for Arabic Language Understanding](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/) — Wissam Antoun, Fady Baly, Hazem Hajj · 2020 · 1,302 citations
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/) — Payal Bajaj, Daniel Campos, Nick Craswell et al. · 2016 · 1,112 citations
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/) — Deyao Zhu, Jun Chen, Xiaoqian Shen et al. · 2023 · 1,106 citations
- [NewsQA: A Machine Comprehension Dataset](https://md.research.elamur.ai/free/en/papers/trischler-2017-newsqa-machine-comprehension-dataset/) — Adam Trischler, Tong Wang, Xingdi Yuan et al. · 2017 · 1,093 citations
- [The NarrativeQA Reading Comprehension Challenge](https://md.research.elamur.ai/free/en/papers/kocisky-2018-narrativeqa-reading-comprehension-challenge/) — Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom et al. · 2018 · 1,031 citations
- [ERNIE 2.0: A Continual Pre-training Framework for Language Understanding](https://md.research.elamur.ai/free/en/papers/sun-2019-ernie-2-0-continual/) — Yu Sun, Shuohuan Wang, Yukun Li et al. · 2019 · 984 citations
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://md.research.elamur.ai/free/en/papers/song-2020-mpnet-masked-permuted-pre-training/) — Kaitao Song, Xu Tan, Tao Qin et al. · 2020 · 948 citations
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/) — Dheeru Dua, Yizhong Wang, Pradeep Dasigi et al. · 2019 · 943 citations
- [MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text](https://md.research.elamur.ai/free/en/papers/richardson-2013-mctest-challenge-dataset-open-domain/) — Matthew Richardson, Christopher J.C. Burges, Erin Renshaw · 2013 · 881 citations
- [Gated Self-Matching Networks for Reading Comprehension and Question Answering](https://md.research.elamur.ai/free/en/papers/wang-2017-gated-self-matching-networks-reading/) — Wenhui Wang, Nan Yang, Furu Wei et al. · 2017 · 827 citations
- [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://md.research.elamur.ai/free/en/papers/conneau-2017-supervised-learning-universal-sentence-representations/) — Alexis Conneau, Douwe Kiela, Holger Schwenk et al. · 2017 · 784 citations
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/) — Yubo Wang, Xueguang Ma, Ge Zhang et al. · 2024 · 731 citations
- [Explain Yourself! Leveraging Language Models for Commonsense Reasoning](https://md.research.elamur.ai/free/en/papers/rajani-2019-explain-yourself-leveraging-language-models/) — Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong et al. · 2019 · 693 citations
- [A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task](https://md.research.elamur.ai/free/en/papers/chen-2016-thorough-examination-cnn-daily-mail/) — Danqi Chen, Jason Bolton, Christopher D. Manning · 2016 · 691 citations
- [Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences](https://md.research.elamur.ai/free/en/papers/khashabi-2018-looking-beyond-surface-challenge-set/) — Daniel Khashabi, Snigdha Chaturvedi, Michael Roth et al. · 2018 · 671 citations
- [e-SNLI: Natural Language Inference with Natural Language Explanations](https://md.research.elamur.ai/free/en/papers/camburu-2018-e-snli-natural-language-inference/) — Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz et al. · 2018 · 613 citations
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://md.research.elamur.ai/free/en/papers/coucke-2018-snips-voice-platform-embedded-spoken/) — Alice Coucke, Alaa Saade, Adrien Ball et al. · 2018 · 595 citations
- [QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension](https://md.research.elamur.ai/free/en/papers/yu-2018-qanet-combining-local-convolution-global/) — Adams Wei Yu, David Dohan, Minh-Thang Luong et al. · 2018 · 586 citations
- [Simple and Effective Multi-Paragraph Reading Comprehension](https://md.research.elamur.ai/free/en/papers/clark-2017-simple-effective-multi-paragraph-reading/) — Christopher Clark, Matt Gardner · 2017 · 558 citations
- [BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis](https://md.research.elamur.ai/free/en/papers/xu-2019-bert-post-training-review-reading/) — Hu Xu, Bing Liu, Lei Shu et al. · 2019 · 546 citations
- [Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning](https://md.research.elamur.ai/free/en/papers/huang-2019-cosmos-qa-machine-reading-comprehension/) — Lifu Huang, Ronan Le Bras, Chandra Bhagavatula et al. · 2019 · 539 citations
- [SocialIQA: Commonsense Reasoning about Social Interactions](https://md.research.elamur.ai/free/en/papers/sap-2019-socialiqa-commonsense-reasoning-about-social/) — Maarten Sap, Hannah Rashkin, Derek Chen et al. · 2019 · 246 citations

## Related concepts

- [Question answering](https://md.research.elamur.ai/free/en/concepts/question-answering/) — Question answering (QA) is the task of building systems that reply to a question in natural language with a correct answer. The answer may…
- [Semantic parsing](https://md.research.elamur.ai/free/en/concepts/semantic-parsing/) — Semantic parsing is the task of mapping a natural-language utterance to a formal, machine-interpretable representation of its meaning, such…

## More papers

Topics: [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · [🧾 Semantic parsing & meaning representation](https://md.research.elamur.ai/free/en/topics/semantic-parsing/) · [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · [🗨️ Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/)

Shelves: [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/)

Updated 24 October 2026

- [← previous concept Natural language processing](https://md.research.elamur.ai/free/en/concepts/natural-language-processing/)
- [next concept → nmod: nominal modifier](https://md.research.elamur.ai/free/en/concepts/ud-nmod/)
- [↑ back to concepts](https://md.research.elamur.ai/free/en/concepts/)