---
title: "Question answering — methods, history and key papers"
description: "Question answering (QA) is the task of building systems that reply to a question in natural language with a correct answer."
canonical: "https://research.elamur.ai/free/en/concepts/question-answering/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/concepts/question-answering/)

[home](https://md.research.elamur.ai/free/en/) / [concepts](https://md.research.elamur.ai/free/en/concepts/) / Question answering

# ❓ Question answering

Field · 45 key papers · [all concepts](https://md.research.elamur.ai/free/en/concepts/)

Part of: [🗨️ Natural language processing](https://md.research.elamur.ai/free/en/concepts/natural-language-processing/) · [🧶 Knowledge graph](https://md.research.elamur.ai/free/en/concepts/knowledge-graph/) · [💬 Natural language understanding](https://md.research.elamur.ai/free/en/concepts/natural-language-understanding/) · [📝 Text summarization](https://md.research.elamur.ai/free/en/concepts/text-summarization/) · [📎 Retrieval-augmented generation](https://md.research.elamur.ai/free/en/concepts/retrieval-augmented-generation/)

Question answering (QA) is the task of building systems that reply to a question in natural language with a correct answer. The answer may come from a given passage, a large document collection, a structured knowledge base, an image or video, or knowledge stored inside a trained model.

Also: machine reading comprehension (MRC), reading comprehension

About 18 minutes to read · every paper named here has its own page in this library

**On this page**

1. [What it is](#a-what-it-is)
2. [How it works](#a-how-it-works)
3. [History](#a-history)
4. [Methods](#a-methods)
5. [Datasets and benchmarks](#a-datasets-and-benchmarks)
6. [People and institutions](#a-people-and-institutions)
7. [Open questions](#a-open-questions)
8. [The library in figures](#a-in-figures)

## What it is

A question answering system takes a question such as "When was Mozart born?" and returns "1756". It does not return a list of documents to read. This is what separates QA from search. A search engine ranks pages, while a QA system commits to an answer and, ideally, points to the evidence behind it. QA is one of the oldest goals of [natural language processing](https://md.research.elamur.ai/free/en/concepts/natural-language-processing/). It also became one of its most heavily benchmarked tasks, because a question with a known answer is an easy way to probe whether a machine has understood a text.

The field is usually divided by where the answer comes from and what form it takes.

- Extractive reading comprehension. The system gets a question and a passage, and the answer is a contiguous span of that passage. SQuAD made this format standard ([Rajpurkar et al. 2016](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)).
- Open-domain QA. No passage is given. The system must first find relevant text in a large corpus such as Wikipedia, then read it ([Karpukhin et al. 2020](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/); [Lee et al. 2019](https://md.research.elamur.ai/free/en/papers/lee-2019-latent-retrieval-weakly-supervised-open/)).
- Knowledge-base question answering (KBQA). The answer is looked up in a structured [knowledge graph](https://md.research.elamur.ai/free/en/concepts/knowledge-graph/) like Freebase, often by translating the question into an executable query ([Yih et al. 2015](https://md.research.elamur.ai/free/en/papers/yih-2015-semantic-parsing-staged-query-graph/)).
- Multi-hop QA. Answering requires combining facts from two or more documents, as in "Which river flows through the birthplace of X?" ([Yang et al. 2018](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/)).
- Visual QA. The question is about an image, a video or a chart ([Goyal et al. 2016](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/); [Lei et al. 2018](https://md.research.elamur.ai/free/en/papers/lei-2018-tvqa-localized-compositional-video-question/); [Masry et al. 2022](https://md.research.elamur.ai/free/en/papers/masry-2022-chartqa-benchmark-question-answering-about/)).
- Conversational QA. Questions arrive in a dialogue, and later ones depend on earlier turns ([Reddy et al. 2019](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/); [Choi et al. 2018](https://md.research.elamur.ai/free/en/papers/choi-2018-quac-question-answering-context/)).

Answers also differ in form. They can be a span, a yes/no verdict, a choice among options, a number computed from the text, or a free-form paragraph. Long-form QA, which asks for explanatory answers to "why" and "how" questions, sits close to [text generation](https://md.research.elamur.ai/free/en/topics/text-generation/) ([Fan et al. 2019](https://md.research.elamur.ai/free/en/papers/fan-2019-eli5-long-form-question-answering/)).

The shelves its papers stand on Artificial Intelligence: 338 of the 350 papers here.

**The numbers**

| Shelf | Papers |
| --- | --- |
| [Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) | 338 papers |
| [Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) | 329 papers |
| [Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) | 158 papers |
| [Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) | 114 papers |
| [Developmental and Educational Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/developmental-and-educational-psychology/) | 16 papers |
| [Health Informatics](https://md.research.elamur.ai/free/en/shelves/medicine/health-informatics/) | 10 papers |
| [Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/molecular-biology/) | 5 papers |
| [Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) | 4 papers |
| [Political Science and International Relations](https://md.research.elamur.ai/free/en/shelves/social-sciences/political-science-and-international-relations/) | 3 papers |
| [Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) | 3 papers |
| [Computational Theory and Mathematics](https://md.research.elamur.ai/free/en/shelves/computer-science/computational-theory-and-mathematics/) | 2 papers |
| [Applied Mathematics](https://md.research.elamur.ai/free/en/shelves/mathematics/applied-mathematics/) | 1 paper |

## How it works

### The retriever–reader pipeline

Most modern open-domain systems split the work into two stages. A retriever narrows a corpus of millions of passages down to a handful of candidates. A reader then examines those candidates and produces the answer. The design echoes the TREC question answering evaluations of the late 1990s and 2000s. Chen and colleagues (2017) recast it in neural form in DrQA, described in [Reading Wikipedia to Answer Open-Domain Questions](https://md.research.elamur.ai/free/en/papers/chen-2017-reading-wikipedia-answer-open-domain/), with Wikipedia as the knowledge source ([Clark & Gardner 2017](https://md.research.elamur.ai/free/en/papers/clark-2017-simple-effective-multi-paragraph-reading/)).

For years the retriever was a sparse keyword method: term frequency–inverse document frequency (TF-IDF) or BM25, term-weighting schemes from [information retrieval](https://md.research.elamur.ai/free/en/topics/information-retrieval/). Dense retrieval replaces keyword overlap with learned vectors. Two encoders map the question and each passage into the same vector space, and relevance is measured by the inner product:

```
sim(q, p) = E_Q(q) · E_P(p)
```

Here q is the question and p a passage. E_Q and E_P are the question and passage encoders, each a neural network that outputs a fixed-length vector. The dot "·" is the dot product. A higher score means the passage is judged more likely to contain the answer.

Because passage vectors do not depend on the question, they can be computed once, stored, and searched with an approximate nearest-neighbour index. In the Dense Passage Retriever, both encoders are BERT-base models, and the representation is the 768-dimensional vector of the first token ([Karpukhin et al. 2020](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/)). The earlier ORQA system used 128-dimensional encodings of more than 13 million Wikipedia blocks ([Lee et al. 2019](https://md.research.elamur.ai/free/en/papers/lee-2019-latent-retrieval-weakly-supervised-open/)).

Training pushes the score of the correct passage above the scores of incorrect ones. A common objective is a softmax over one positive and several negatives:

```
L = − log [ exp(sim(q, p⁺)) / ( exp(sim(q, p⁺)) + Σⱼ exp(sim(q, pⱼ⁻)) ) ]
```

p⁺ is a passage that contains the answer, the pⱼ⁻ are negative passages, and the sum runs over all negatives. L is the loss to be minimised.

DPR's central finding was that cheap negatives are enough. It reused the gold passages of the other questions in the same mini-batch and added one BM25 hard negative per question. No extra pretraining was needed ([Karpukhin et al. 2020](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/)). On Natural Questions, top-5 retrieval accuracy rose from 42.9% with BM25 to 65.2% with DPR ([Karpukhin et al. 2020](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/)).

### Span prediction

An extractive reader encodes the question and passage together, producing a vector hᵢ for every passage token i. Two learned vectors then score each token as a possible start or end of the answer:

```
P(start = i) = exp(wₛ · hᵢ) / Σⱼ exp(wₛ · hⱼ)
P(end = k)   = exp(wₑ · hₖ) / Σⱼ exp(wₑ · hⱼ)
```

wₛ and wₑ are trained parameter vectors, hᵢ is the contextual representation of token i, and the sums run over all tokens j of the passage. The predicted answer is the span (i, k) with i ≤ k, usually under a maximum length, that maximises the product of the two probabilities. Training minimises the negative log probability of the gold start and end positions ([Wang et al. 2017](https://md.research.elamur.ai/free/en/papers/wang-2017-gated-self-matching-networks-reading/)).

Single-pass prediction can lock in an early mistake. The Dynamic Coattention Network addressed this by re-estimating start and end positions over several iterations ([Xiong et al. 2016](https://md.research.elamur.ai/free/en/papers/xiong-2016-dynamic-coattention-networks-question-answering/)).

When a reader sees many passages, the raw scores from different passages are not comparable. Clark and Gardner fixed this with a shared-normalisation objective that applies the softmax jointly across all paragraphs from one document ([Clark & Gardner 2017](https://md.research.elamur.ai/free/en/papers/clark-2017-simple-effective-multi-paragraph-reading/)).

### Generative readers and language models

A generative reader writes the answer token by token instead of pointing to it. [Retrieval-augmented generation](https://md.research.elamur.ai/free/en/concepts/retrieval-augmented-generation/) couples a retriever with a sequence-to-sequence [language model](https://md.research.elamur.ai/free/en/concepts/language-model/) for knowledge-intensive tasks (Lewis et al. 2020).

Fusion-in-Decoder takes a different route. It encodes each retrieved passage separately with the question, and lets a T5 decoder attend over all of the encodings at once. Cost therefore grows linearly with the number of passages, so the reader can take 100 of them ([Izacard & Grave 2021](https://md.research.elamur.ai/free/en/papers/izacard-2021-leveraging-passage-retrieval-generative-models/)). The same work contrasts this with closed-book generation, where a model answers from its weights alone. A 770M-parameter reader with BM25 retrieval outscored an 11B-parameter closed-book T5 on Natural Questions ([Izacard & Grave 2021](https://md.research.elamur.ai/free/en/papers/izacard-2021-leveraging-passage-retrieval-generative-models/)).

At the far end of this line, WebGPT fine-tuned GPT-3 to drive a text-only browser. The model learned to search, click and quote, and was trained from human demonstrations and preference comparisons ([Nakano et al. 2021](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/)). These systems connect QA to retrieval-augmented generation and to [large language models](https://md.research.elamur.ai/free/en/concepts/large-language-models/) more broadly.

### Measuring answers

Extractive QA is usually scored with two measures introduced with SQuAD ([Rajpurkar et al. 2016](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)). Exact match (EM) is 1 if the normalised prediction equals a reference answer and 0 otherwise. Normalisation typically lowercases the text and strips punctuation and articles. Token-level F1 treats the prediction and the reference as bags of tokens:

```
F1 = 2 · P · R / (P + R)
```

P (precision) is the share of predicted tokens that appear in the reference. R (recall) is the share of reference tokens that appear in the prediction. When several references exist, the maximum over them is taken. Both measures are averaged over questions.

Free-form answers are harder to score. MS MARCO reported ROUGE-L ([Bajaj et al. 2016](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/)), and its free-form answers were also scored with [bilingual evaluation understudy (BLEU)](https://md.research.elamur.ai/free/en/concepts/bleu/) ([Kwiatkowski et al. 2019](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)). Long-form QA relies heavily on human preference judgements ([Fan et al. 2019](https://md.research.elamur.ai/free/en/papers/fan-2019-eli5-long-form-question-answering/); [Nakano et al. 2021](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/)).

## History

### Patterns and the TREC era

Early systems such as BASEBALL (Green and colleagues, 1961) and LUNAR (Woods, 1973) answered questions over small structured databases. They were forerunners of today's knowledge-base QA. Open-domain QA over text became a shared evaluation at TREC, where answer extraction often depended on surface patterns.

Ravichandran and Hovy learned such patterns automatically. They started from a few seed question–answer pairs, mined web sentences with suffix trees, and estimated each pattern's precision. For instance, a pattern made of the person's name followed by an opening parenthesis, the answer and a dash proved fully reliable for birth dates ([Ravichandran & Hovy 2002](https://md.research.elamur.ai/free/en/papers/ravichandran-2002-learning-surface-text-patterns-question/)). They found that answers mined from the web beat answers drawn from the TREC corpus, and credited the redundancy of the web ([Ravichandran & Hovy 2002](https://md.research.elamur.ai/free/en/papers/ravichandran-2002-learning-surface-text-patterns-question/)).

Answer sentence selection was benchmarked on WikiQA, built from real Bing queries. On the older QASent set, derived from TREC-QA, lexical word-matching models such as LCLR did well. On WikiQA, sentence-level neural models did better ([Yang et al. 2015](https://md.research.elamur.ai/free/en/papers/yang-2015-wikiqa-challenge-dataset-open-domain/)).

### Large datasets and neural readers

The shift to deep learning came with large datasets. Hermann and colleagues (2015) turned news summaries into cloze questions. A careful re-examination showed that a simple attentive model reached 73.6% on CNN and 76.6% on Daily Mail. The same study found that about a quarter of the questions were damaged by coreference errors or anonymisation, so the benchmark said little about multi-sentence reasoning ([Chen et al. 2016](https://md.research.elamur.ai/free/en/papers/chen-2016-thorough-examination-cnn-daily-mail/)).

SQuAD, released in 2016, offered 107,785 crowd-written questions over Wikipedia paragraphs with span answers. Its feature-based baseline reached 51.0% F1 against 86.8% for humans ([Rajpurkar et al. 2016](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)). A rapid sequence of neural readers followed, each built from recurrent encoders and new forms of attention between question and passage:

- BiDAF ([Seo et al. 2016](https://md.research.elamur.ai/free/en/papers/seo-2016-bidirectional-attention-flow-machine-comprehension/));
- the Dynamic Coattention Network ([Xiong et al. 2016](https://md.research.elamur.ai/free/en/papers/xiong-2016-dynamic-coattention-networks-question-answering/));
- gated self-matching networks ([Wang et al. 2017](https://md.research.elamur.ai/free/en/papers/wang-2017-gated-self-matching-networks-reading/)).

Intra-attention inside recurrent readers had been explored for machine reading just before ([Cheng et al. 2016](https://md.research.elamur.ai/free/en/papers/cheng-2016-long-short-term-memory-networks/)). QANet, from Carnegie Mellon University and Google Brain, then removed recurrence altogether. It paired convolutions with self-attention in the style of the [Transformer](https://md.research.elamur.ai/free/en/concepts/transformer/) and trained several times faster at similar accuracy ([Yu et al. 2018](https://md.research.elamur.ai/free/en/papers/yu-2018-qanet-combining-local-convolution-global/)). Pretrained encoders, notably BERT (Devlin and colleagues, 2018), soon made a fine-tuned span head on a large pretrained model the default extractive reader.

### From reading to retrieval and generation

After reading came finding. ORQA trained retriever and reader together from question–answer pairs alone, treating the evidence passage as a latent variable ([Lee et al. 2019](https://md.research.elamur.ai/free/en/papers/lee-2019-latent-retrieval-weakly-supervised-open/)). It pretrained the retriever with an Inverse Cloze Task, in which the model is given one sentence and must identify the passage that originally contained it ([Lee et al. 2019](https://md.research.elamur.ai/free/en/papers/lee-2019-latent-retrieval-weakly-supervised-open/)). DPR showed that ordinary supervised training of a dual encoder was enough ([Karpukhin et al. 2020](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/)). RocketQA improved that training with cross-batch negatives, denoised hard negatives and pseudo-labelled data ([Qu et al. 2021](https://md.research.elamur.ai/free/en/papers/qu-2021-rocketqa-optimized-training-approach-dense/)).

Papers on this subject by year 350 dated papers from 1983 to 2026; the busiest year is 2019 with 59, and 81% of them appeared in 2018 or later.

**The numbers**

| Year | Papers |
| --- | --- |
| 2026 | 26 |
| 2025 | 11 |
| 2024 | 17 |
| 2023 | 39 |
| 2022 | 31 |
| 2021 | 35 |
| 2020 | 39 |
| 2019 | 59 |
| 2018 | 25 |
| 2017 | 18 |
| 2016 | 21 |
| 2015 | 7 |
| 2014 | 4 |
| 2013 | 2 |
| 2010 | 1 |
| 2008 | 1 |
| 2007 | 1 |
| 2006 | 1 |
| 2005 | 2 |
| 2003 | 3 |
| 2002 | 2 |
| 2001 | 1 |
| 2000 | 2 |
| 1999 | 1 |
| 1983 | 1 |

Citations of the papers on this subject, by when they were published Papers published in 2015–2019 gather 75% of the 201,993 citations of the papers on this subject.

**The numbers**

| Published | Papers and their citations |
| --- | --- |
| 2025–2029 | 37 papers, 503 citations |
| 2020–2024 | 161 papers, 42,510 citations |
| 2015–2019 | 130 papers, 150,959 citations |
| 2010–2014 | 7 papers, 2,852 citations |
| 2005–2009 | 5 papers, 989 citations |
| 2000–2004 | 8 papers, 4,071 citations |
| 1995–1999 | 1 papers, 35 citations |
| 1980–1984 | 1 papers, 74 citations |

Landmark papers on this subject: the 12 most cited, in time The 12 most cited papers on this subject span 2015–2023; the dot grows with the citations.

**The numbers**

| Paper | Citations |
| --- | --- |
| [2015 — Teaching Machines to Read and Comprehend](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/) | 3,829 |
| [2016 — SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/) | 9,513 |
| [2017 — TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/) | 3,213 |
| [2017 — Reading Wikipedia to Answer Open-Domain Questions](https://md.research.elamur.ai/free/en/papers/chen-2017-reading-wikipedia-answer-open-domain/) | 2,260 |
| [2018 — HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/) | 3,495 |
| [2018 — Know What You Don't Know: Unanswerable Questions for SQuAD](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/) | 3,373 |
| [2019 — BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/) | 55,571 |
| [2019 — Natural Questions: A Benchmark for Question Answering Research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/) | 4,035 |
| [2019 — Language Models as Knowledge Bases?](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/) | 2,992 |
| [2019 — GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering](https://md.research.elamur.ai/free/en/papers/hudson-2019-gqa-new-dataset-real-world/) | 2,006 |
| [2020 — Dense Passage Retrieval for Open-Domain Question Answering](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/) | 5,375 |
| [2023 — Lost in the Middle: How Language Models Use Long Contexts](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/) | 2,541 |

## Methods

### Extractive reading comprehension

The neural readers of 2016–2018 shared one recipe:

1. Embed words, often with GloVe [word embeddings](https://md.research.elamur.ai/free/en/concepts/word-embeddings/) plus character-level features.
2. Encode question and passage in context.
3. Fuse the two with attention.
4. Predict a span.

BiDAF computed one similarity matrix and used it for attention in both directions. Its attention was computed at every passage position rather than squeezed into a single vector ([Seo et al. 2016](https://md.research.elamur.ai/free/en/papers/seo-2016-bidirectional-attention-flow-machine-comprehension/)). Gated self-matching added a relevance gate and let the passage attend over itself to gather distant evidence ([Wang et al. 2017](https://md.research.elamur.ai/free/en/papers/wang-2017-gated-self-matching-networks-reading/)). These readers became standard baselines on many later datasets, including TriviaQA ([Joshi et al. 2017](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/)), DROP ([Dua et al. 2019](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/)) and QuAC ([Choi et al. 2018](https://md.research.elamur.ai/free/en/papers/choi-2018-quac-question-answering-context/)).

### Dense, sparse and hybrid retrieval

Dense retrievers capture paraphrase that keywords miss, but they have blind spots. ORQA beat BM25 on datasets whose questions came from people who actually needed the answer. BM25 stayed stronger where the question writers already knew the answer, as in SQuAD and TriviaQA ([Lee et al. 2019](https://md.research.elamur.ai/free/en/papers/lee-2019-latent-retrieval-weakly-supervised-open/)). The same work noted that compact vectors struggle with very specific concepts and suggested hybrid sparse–dense retrieval ([Lee et al. 2019](https://md.research.elamur.ai/free/en/papers/lee-2019-latent-retrieval-weakly-supervised-open/)).

Hard negatives are a delicate part of training. Manual inspection of MS MARCO found that about 70% of top-ranked but unlabelled passages were in fact relevant, so many "negatives" are false ones ([Qu et al. 2021](https://md.research.elamur.ai/free/en/papers/qu-2021-rocketqa-optimized-training-approach-dense/)).

### Generation with retrieval

Generative readers can combine evidence from many passages and can produce answers that are not literal spans. Fusion-in-Decoder kept improving as passages were added. Going from 10 to 100 passages added about 6 points on TriviaQA and 3.5 on Natural Questions, whereas extractive readers usually stopped gaining at 10–20 passages ([Izacard & Grave 2021](https://md.research.elamur.ai/free/en/papers/izacard-2021-leveraging-passage-retrieval-generative-models/)). For long-form answers, the evidence pipeline itself is a bottleneck. In ELI5, fewer than 9% of the best-matching sentences came from the top 20 retrieved passages ([Fan et al. 2019](https://md.research.elamur.ai/free/en/papers/fan-2019-eli5-long-form-question-answering/)).

### Knowledge-base QA and semantic parsing

KBQA usually maps a question onto a query over the graph, which links it to [semantic parsing](https://md.research.elamur.ai/free/en/topics/semantic-parsing/). STAGG built a query graph in stages: it chose a topic entity, then a chain of relations, then constraints and aggregations. Because the knowledge base guided every step, the search space shrank. The system reached 52.5% F1 on WebQuestions ([Yih et al. 2015](https://md.research.elamur.ai/free/en/papers/yih-2015-semantic-parsing-staged-query-graph/)).

Labelling questions with full SPARQL parses, rather than answers only, raised average F1 from 66.8% to 71.7%. Per question, it also took experts less time than finding the answers by hand ([Yih et al. 2016](https://md.research.elamur.ai/free/en/papers/yih-2016-value-semantic-parse-labeling-knowledge/)).

Real knowledge graphs are incomplete. EmbedKGQA scored candidate answers with ComplEx embeddings, which represent the graph's entities and relations as complex-valued vectors. This let it recover answers even when the direct link had been deleted ([Saxena et al. 2020](https://md.research.elamur.ai/free/en/papers/saxena-2020-improving-multi-hop-question-answering/)). Neural module networks took a compositional route. They assembled a network per question from small modules such as find, relate and exists, and worked over both images and a geographic database ([Andreas et al. 2016](https://md.research.elamur.ai/free/en/papers/andreas-2016-learning-compose-neural-networks-question/)).

### Visual QA

Visual readers fuse an image representation with the question. Stacked attention networks let the question vector attend over a 14×14 grid of image regions in successive steps, narrowing focus each time (Yang et al. 2015). Multimodal Compact Bilinear pooling approximated the full outer product of image and text vectors cheaply ([Fukui et al. 2016](https://md.research.elamur.ai/free/en/papers/fukui-2016-multimodal-compact-bilinear-pooling-visual/)). It hashed each vector into a larger space with Count Sketch, a random projection. It then combined the two sketches by multiplying them element-wise after a fast Fourier transform (FFT), which is equivalent to convolving them ([Fukui et al. 2016](https://md.research.elamur.ai/free/en/papers/fukui-2016-multimodal-compact-bilinear-pooling-visual/)). Grounding answers to boxes in the image made it possible to inspect where a model looks ([Zhu et al. 2015](https://md.research.elamur.ai/free/en/papers/zhu-2015-visual7w-grounded-question-answering-images/)). This line of work belongs to [multimodal learning](https://md.research.elamur.ai/free/en/topics/multimodal/).

### QA as a general interface and as a data source

Questions can also serve as a format for other tasks. Event extraction was recast as extractive QA: one model is queried for triggers and another is asked role-specific questions. This removed the need for entity annotation and allowed zero-shot extraction of unseen roles ([Du & Cardie 2020](https://md.research.elamur.ai/free/en/papers/du-2020-event-extraction-answering-almost-natural/)). It ties QA to [information extraction](https://md.research.elamur.ai/free/en/concepts/information-extraction/).

In the opposite direction, neural question generation trained a sequence-to-sequence model on SQuAD sentence–question pairs. Its questions beat a rule-based system in human judgements ([Du et al. 2017](https://md.research.elamur.ai/free/en/papers/du-2017-learning-ask-neural-question-generation/)). Generated questions can then be used to build new datasets ([Du et al. 2017](https://md.research.elamur.ai/free/en/papers/du-2017-learning-ask-neural-question-generation/)).

## Datasets and benchmarks

Benchmarks have driven the field, and each new one was usually built to expose what the previous one let models skip. The main families are summarised below. Sizes are as reported by the creators, and the unit is given in each row.

| Benchmark | Kind | Scale (unit per row) | Answer form |
| --- | --- | --- | --- |
| SQuAD ([Rajpurkar et al. 2016](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)) | extractive | 107,785 question–answer pairs | span |
| SQuAD 2.0 ([Rajpurkar et al. 2018](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/)) | extractive, unanswerable | +53,775 unanswerable questions added to SQuAD | span or abstain |
| TriviaQA ([Joshi et al. 2017](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/)) | reading / open-domain | 95,956 question–answer pairs | span (distant supervision) |
| Natural Questions ([Kwiatkowski et al. 2019](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)) | reading over a whole Wikipedia page, real queries | 307,373 training examples (query plus page) | long and short answer, or NULL |
| HotpotQA ([Yang et al. 2018](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/)) | multi-hop | 112,779 question–answer pairs | span, yes/no, supporting facts |
| CoQA ([Reddy et al. 2019](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/)) | conversational | about 127k question–answer turns in about 8k conversations | free-form with rationale |

### Reading comprehension

SQuAD's paragraphs come from 536 high-PageRank Wikipedia articles. Its analysis showed that the difficulty lay mostly in finding the exact span, not the right sentence ([Rajpurkar et al. 2016](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)).

TriviaQA reversed the order of construction. Trivia enthusiasts wrote the questions first, and evidence documents were gathered afterwards. As a result, questions and evidence share less wording ([Joshi et al. 2017](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/)).

RACE drew multiple-choice questions from English exams for Chinese schoolchildren. Its options are free-form sentences rather than spans ([Lai et al. 2017](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/)).

Several datasets targeted reasoning that span matching cannot do:

- MultiRC required evidence from several sentences, and a question could have any number of correct options ([Khashabi et al. 2018](https://md.research.elamur.ai/free/en/papers/khashabi-2018-looking-beyond-surface-challenge-set/)).
- DROP demanded counting, sorting and arithmetic over paragraphs, and was collected adversarially against BiDAF ([Dua et al. 2019](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/)).
- NarrativeQA posed questions over whole stories ([Kočiský et al. 2018](https://md.research.elamur.ai/free/en/papers/kocisky-2018-narrativeqa-reading-comprehension-challenge/)).

### Open-domain and real information needs

MS MARCO paired anonymised Bing queries with retrieved web passages and human-written answers. It became a standard for neural ranking as well as for reading ([Bajaj et al. 2016](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/)). Natural Questions used real Google queries of eight or more words, each paired with a Wikipedia page from the top results. Annotators marked a long answer, a short answer, or NULL ([Kwiatkowski et al. 2019](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)). In this original form a system reads the given page, and open versions without the page were used later to evaluate retrievers ([Lee et al. 2019](https://md.research.elamur.ai/free/en/papers/lee-2019-latent-retrieval-weakly-supervised-open/)). Only about 49% of its examples have a non-NULL long answer, and repeated annotation showed that many real questions have several acceptable answers ([Kwiatkowski et al. 2019](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)).

### Multi-hop, knowledge and commonsense

HotpotQA linked Wikipedia paragraphs through bridge entities: an entity described in one paragraph and mentioned in the other, through which the reasoning must pass. It also built comparison pairs. Workers marked the sentences each answer depends on, so systems can be scored on evidence as well as on answers ([Yang et al. 2018](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/)). 2WikiMultiHopQA added Wikidata triples that trace the reasoning path, and removed templates known to yield single-hop shortcuts ([Ho et al. 2020](https://md.research.elamur.ai/free/en/papers/ho-2020-constructing-multi-hop-qa-dataset/)). WebQuestionsSP supplies executable SPARQL parses for KBQA ([Yih et al. 2016](https://md.research.elamur.ai/free/en/papers/yih-2016-value-semantic-parse-labeling-knowledge/)).

Several benchmarks test knowledge that is not stated in any passage, and they form a large part of research on [reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/):

- CommonsenseQA built five-way questions from ConceptNet relations ([Talmor et al. 2019](https://md.research.elamur.ai/free/en/papers/talmor-2019-commonsenseqa-question-answering-challenge-targeting/)).
- ARC collected grade-school science questions (Clark et al. 2018).
- OpenBookQA pairs questions with a small book of science facts (Mihaylov et al. 2018).

### Conversational, multilingual, visual and domain QA

CoQA has free-form answers backed by span rationales, and nearly half its questions contain references back to earlier turns ([Reddy et al. 2019](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/)). QuAC hid the evidence from the asker, which produced more exploratory questions. About 20% of its questions have no answer ([Choi et al. 2018](https://md.research.elamur.ai/free/en/papers/choi-2018-quac-question-answering-context/)). Both link QA to [dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/).

For [multilingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/), MLQA aligned Wikipedia sentences across seven languages and translated the questions by hand ([Lewis et al. 2019](https://md.research.elamur.ai/free/en/papers/lewis-2019-mlqa-evaluating-cross-lingual-extractive/)). TyDi QA instead collected native questions in 11 typologically diverse languages, with no translation at any stage ([Clark et al. 2020](https://md.research.elamur.ai/free/en/papers/clark-2020-tydi-qa-benchmark-information-seeking/)).

Visual QA moved through several datasets:

- VQA v2.0, which pairs nearly every question with two similar images that need different answers ([Goyal et al. 2016](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/));
- Visual7W, with answers grounded to image regions ([Zhu et al. 2015](https://md.research.elamur.ai/free/en/papers/zhu-2015-visual7w-grounded-question-answering-images/));
- GQA, with compositional questions generated from scene graphs ([Hudson & Manning 2019](https://md.research.elamur.ai/free/en/papers/hudson-2019-gqa-new-dataset-real-world/));
- TVQA, on television clips with subtitles ([Lei et al. 2018](https://md.research.elamur.ai/free/en/papers/lei-2018-tvqa-localized-compositional-video-question/));
- ChartQA, on real charts ([Masry et al. 2022](https://md.research.elamur.ai/free/en/papers/masry-2022-chartqa-benchmark-question-answering-about/)).

PubMedQA asks yes/no/maybe questions about biomedical abstracts ([Jin et al. 2019](https://md.research.elamur.ai/free/en/papers/jin-2019-pubmedqa-dataset-biomedical-research-question/)).

### Unanswerable questions

A system that must always return a span will guess even when the text is silent. WikiQA kept questions with no correct sentence and defined answer triggering, the decision of whether any candidate answers at all. Most of its questions have no correct candidate ([Yang et al. 2015](https://md.research.elamur.ai/free/en/papers/yang-2015-wikiqa-challenge-dataset-open-domain/)).

SQuAD 2.0 made abstention central. Crowdworkers wrote on-topic unanswerable questions, and each paragraph contained a plausible decoy span of the expected answer type ([Rajpurkar et al. 2018](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/)). Baselines predicted a no-answer probability and abstained above a threshold tuned on the dev set ([Rajpurkar et al. 2018](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/)). Automatically generated negatives proved much easier than these written ones ([Rajpurkar et al. 2018](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/)). Natural Questions, TyDi QA and MS MARCO all contain unanswerable items that arise naturally from real queries ([Kwiatkowski et al. 2019](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/); [Clark et al. 2020](https://md.research.elamur.ai/free/en/papers/clark-2020-tydi-qa-benchmark-information-seeking/); [Bajaj et al. 2016](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/)).

Where the papers on this subject were published Annual Meeting of the Association for Computational Linguistics (ACL) carries the most, 74 of 350.

**The numbers**

| Venue | Papers |
| --- | --- |
| [Annual Meeting of the Association for Computational Linguistics (ACL)](https://md.research.elamur.ai/free/en/venues/acl/) | 74 papers |
| [Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/) | 63 papers |
| [Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)](https://md.research.elamur.ai/free/en/venues/naacl/) | 24 papers |
| [Transactions of the Association for Computational Linguistics (TACL)](https://md.research.elamur.ai/free/en/venues/tacl/) | 21 papers |
| [Language Resources and Evaluation Conference (LREC)](https://md.research.elamur.ai/free/en/venues/lrec/) | 17 papers |
| [Findings of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/findings/) | 15 papers |
| [International Conference on Computational Linguistics (COLING)](https://md.research.elamur.ai/free/en/venues/coling/) | 5 papers |
| [Journal of Chinese Information Processing (中文信息学报)](https://md.research.elamur.ai/free/en/venues/journal-of-chinese-information-processing/) | 5 papers |
| [Conference on Neural Information Processing Systems (NeurIPS)](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/) | 4 papers |
| [IEEE Computer Society Conference on Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/venues/ieee-computer-society-conference-on-computer-vision-and-pattern-recognition/) | 4 papers |

For how such numbers are produced and read, see [evaluation and benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/).

## People and institutions

Much of the modern field traces to a few groups that built both datasets and models.

At Stanford, [Percy Liang](https://md.research.elamur.ai/free/en/authors/percy-liang/) and his co-authors created SQuAD and its unanswerable successor ([Rajpurkar et al. 2016](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/); [Rajpurkar et al. 2018](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/)). Together with Robin Jia, Liang also built adversarial tests showing that reading systems can be distracted ([Jia & Liang 2017](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/)).

[Christopher D. Manning](https://md.research.elamur.ai/free/en/authors/christopher-d-manning/), at Stanford University, co-authored three benchmarks ([Yang et al. 2018](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/); [Hudson & Manning 2019](https://md.research.elamur.ai/free/en/papers/hudson-2019-gqa-new-dataset-real-world/); [Reddy et al. 2019](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/)):

- HotpotQA, for multi-hop QA;
- GQA, for visual reasoning;
- CoQA, for conversational QA.

[Danqi Chen](https://md.research.elamur.ai/free/en/authors/danqi-chen/), now at Princeton University, examined the CNN/Daily Mail task ([Chen et al. 2016](https://md.research.elamur.ai/free/en/papers/chen-2016-thorough-examination-cnn-daily-mail/)) and co-authored CoQA ([Reddy et al. 2019](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/)) and DPR ([Karpukhin et al. 2020](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/)).

The University of Washington is another centre. [Luke Zettlemoyer](https://md.research.elamur.ai/free/en/authors/luke-zettlemoyer/) and [Eunsol Choi](https://md.research.elamur.ai/free/en/authors/eunsol-choi/) built TriviaQA ([Joshi et al. 2017](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/)). Choi also led QuAC ([Choi et al. 2018](https://md.research.elamur.ai/free/en/papers/choi-2018-quac-question-answering-context/)) and contributed to TyDi QA ([Clark et al. 2020](https://md.research.elamur.ai/free/en/papers/clark-2020-tydi-qa-benchmark-information-seeking/)). [Hannaneh Hajishirzi](https://md.research.elamur.ai/free/en/authors/hannaneh-hajishirzi/), at the University of Washington and the Allen Institute for Artificial Intelligence, co-authored BiDAF ([Seo et al. 2016](https://md.research.elamur.ai/free/en/papers/seo-2016-bidirectional-attention-flow-machine-comprehension/)). [Yejin Choi](https://md.research.elamur.ai/free/en/authors/yejin-choi/), now at Stanford University, and [Mohit Iyyer](https://md.research.elamur.ai/free/en/authors/mohit-iyyer/) were co-authors of QuAC ([Choi et al. 2018](https://md.research.elamur.ai/free/en/papers/choi-2018-quac-question-answering-context/)).

Researchers at Google built several of the main benchmarks of real information needs. [Tom Kwiatkowski](https://md.research.elamur.ai/free/en/authors/tom-kwiatkowski/) and [Michael Collins](https://md.research.elamur.ai/free/en/authors/michael-collins/) were among the authors of Natural Questions ([Kwiatkowski et al. 2019](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)) and TyDi QA ([Clark et al. 2020](https://md.research.elamur.ai/free/en/papers/clark-2020-tydi-qa-benchmark-information-seeking/)). [Ming-Wei Chang](https://md.research.elamur.ai/free/en/authors/ming-wei-chang/) co-developed ORQA ([Lee et al. 2019](https://md.research.elamur.ai/free/en/papers/lee-2019-latent-retrieval-weakly-supervised-open/)) and, earlier, the query-graph KBQA work ([Yih et al. 2015](https://md.research.elamur.ai/free/en/papers/yih-2015-semantic-parsing-staged-query-graph/)). MS MARCO came from Microsoft ([Bajaj et al. 2016](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/)).

Researchers at Meta (then Facebook AI Research) produced DPR, retrieval-augmented generation and MLQA. [Patrick Lewis](https://md.research.elamur.ai/free/en/authors/patrick-lewis/), now at Cohere, co-authored MLQA ([Lewis et al. 2019](https://md.research.elamur.ai/free/en/papers/lewis-2019-mlqa-evaluating-cross-lingual-extractive/)), DPR ([Karpukhin et al. 2020](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/)) and retrieval-augmented generation (Lewis et al. 2020). [Sewon Min](https://md.research.elamur.ai/free/en/authors/sewon-min/) was also a DPR author ([Karpukhin et al. 2020](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/)). [Wen-tau Yih](https://md.research.elamur.ai/free/en/authors/wen-tau-yih/), now at Meta's FAIR lab and earlier at Microsoft Research, has worked across the whole range of the field:

- WikiQA, for answer sentence selection ([Yang et al. 2015](https://md.research.elamur.ai/free/en/papers/yang-2015-wikiqa-challenge-dataset-open-domain/));
- the staged query-graph approach to KBQA ([Yih et al. 2015](https://md.research.elamur.ai/free/en/papers/yih-2015-semantic-parsing-staged-query-graph/));
- WebQuestionsSP ([Yih et al. 2016](https://md.research.elamur.ai/free/en/papers/yih-2016-value-semantic-parse-labeling-knowledge/));
- QuAC ([Choi et al. 2018](https://md.research.elamur.ai/free/en/papers/choi-2018-quac-question-answering-context/));
- DPR ([Karpukhin et al. 2020](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/)).

Others extended QA into new forms:

- [Matt Gardner](https://md.research.elamur.ai/free/en/authors/matt-gardner/) co-authored DROP ([Dua et al. 2019](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/)) and multi-paragraph reading ([Clark & Gardner 2017](https://md.research.elamur.ai/free/en/papers/clark-2017-simple-effective-multi-paragraph-reading/)).
- [Jonathan Berant](https://md.research.elamur.ai/free/en/authors/jonathan-berant/), now at Google and Tel Aviv University, co-authored CommonsenseQA ([Talmor et al. 2019](https://md.research.elamur.ai/free/en/papers/talmor-2019-commonsenseqa-question-answering-challenge-targeting/)).
- [Claire Cardie](https://md.research.elamur.ai/free/en/authors/claire-cardie/), at Cornell University, worked on neural question generation ([Du et al. 2017](https://md.research.elamur.ai/free/en/papers/du-2017-learning-ask-neural-question-generation/)) and QA-style event extraction ([Du & Cardie 2020](https://md.research.elamur.ai/free/en/papers/du-2020-event-extraction-answering-almost-natural/)).
- [Mohit Bansal](https://md.research.elamur.ai/free/en/authors/mohit-bansal/), at the University of North Carolina at Chapel Hill, co-authored TVQA ([Lei et al. 2018](https://md.research.elamur.ai/free/en/papers/lei-2018-tvqa-localized-compositional-video-question/)).

Among institutions, [Google](https://md.research.elamur.ai/free/en/companies/google/), [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/), the [Allen Institute for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/allen-institute-for-artificial-intelligence/), [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/), the [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/), [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) and [Meta](https://md.research.elamur.ai/free/en/companies/meta/) are prominent research homes for the field.

Authors with the most papers on this subject Wen-tau Yih leads with 10 papers; the 12 authors shown wrote 97 of the papers on this subject between them, counting each paper once per author.

**The numbers**

| Author | Papers here · their citations |
| --- | --- |
| [Wen-tau Yih](https://md.research.elamur.ai/free/en/authors/wen-tau-yih/) | 10 papers · 10,603 citations |
| [Hannaneh Hajishirzi](https://md.research.elamur.ai/free/en/authors/hannaneh-hajishirzi/) | 10 papers · 1,820 citations |
| [Luke Zettlemoyer](https://md.research.elamur.ai/free/en/authors/luke-zettlemoyer/) | 9 papers · 6,011 citations |
| [Matt Gardner](https://md.research.elamur.ai/free/en/authors/matt-gardner/) | 9 papers · 3,397 citations |
| [Ming-Wei Chang](https://md.research.elamur.ai/free/en/authors/ming-wei-chang/) | 8 papers · 60,522 citations |
| [Danqi Chen](https://md.research.elamur.ai/free/en/authors/danqi-chen/) | 8 papers · 10,732 citations |
| [Jonathan Berant](https://md.research.elamur.ai/free/en/authors/jonathan-berant/) | 8 papers · 3,788 citations |
| [Percy Liang](https://md.research.elamur.ai/free/en/authors/percy-liang/) | 7 papers · 19,324 citations |
| [Patrick Lewis](https://md.research.elamur.ai/free/en/authors/patrick-lewis/) | 7 papers · 9,973 citations |
| [Christopher D. Manning](https://md.research.elamur.ai/free/en/authors/christopher-d-manning/) | 7 papers · 8,187 citations |
| [Eunsol Choi](https://md.research.elamur.ai/free/en/authors/eunsol-choi/) | 7 papers · 5,752 citations |
| [Claire Cardie](https://md.research.elamur.ai/free/en/authors/claire-cardie/) | 7 papers · 2,466 citations |

Where the authors on this subject work now Google is the current home of authors of 54 papers on this subject; counted are the authors with a page here, by their latest institution.

**The numbers**

| Institution | Papers |
| --- | --- |
| [Google](https://md.research.elamur.ai/free/en/companies/google/) | 54 papers |
| [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) | 44 papers |
| [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) | 36 papers |
| [Allen Institute for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/allen-institute-for-artificial-intelligence/) | 34 papers |
| [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) | 32 papers |
| [Google DeepMind](https://md.research.elamur.ai/free/en/companies/google-deepmind/) | 26 papers |
| [New York University](https://md.research.elamur.ai/free/en/universities/new-york-university/) | 24 papers |
| [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) | 22 papers |
| [Meta](https://md.research.elamur.ai/free/en/companies/meta/) | 18 papers |
| [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) | 17 papers |

Countries of the authors on this subject United States comes first: authors working there now wrote 240 papers on this subject.

**The numbers**

| Country | Papers |
| --- | --- |
| [United States](https://md.research.elamur.ai/free/en/countries/united-states/) | 240 papers |
| [China](https://md.research.elamur.ai/free/en/countries/china/) | 130 papers |
| [United Kingdom](https://md.research.elamur.ai/free/en/countries/united-kingdom/) | 55 papers |
| [Canada](https://md.research.elamur.ai/free/en/countries/canada/) | 31 papers |
| [Israel](https://md.research.elamur.ai/free/en/countries/israel/) | 24 papers |
| Hong Kong | 19 papers |
| [Singapore](https://md.research.elamur.ai/free/en/countries/singapore/) | 18 papers |
| [Germany](https://md.research.elamur.ai/free/en/countries/germany/) | 17 papers |
| [Japan](https://md.research.elamur.ai/free/en/countries/japan/) | 13 papers |
| [United Arab Emirates](https://md.research.elamur.ai/free/en/countries/united-arab-emirates/) | 13 papers |

## Open questions

### Shortcuts and robustness

High scores have repeatedly turned out to rest on surface cues. Adversarially inserted sentences exposed how fragile SQuAD-trained readers are ([Jia & Liang 2017](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/)). In the original VQA data, answering "yes" to every "Do you see a…" question scored 87%. Every model tested lost accuracy once images were balanced ([Goyal et al. 2016](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/)). In Visual7W, a question-only model beat humans who could not see the image. It did so by learning answer priors, that is, which answers tend to go with which questions in the training data ([Zhu et al. 2015](https://md.research.elamur.ai/free/en/papers/zhu-2015-visual7w-grounded-question-answering-images/)).

Whether a benchmark truly needs the skill it names, whether careful reading, multi-hop reasoning or looking at the image, remains a design problem for each new dataset ([Jia & Liang 2017](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/); [Goyal et al. 2016](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/); [Ho et al. 2020](https://md.research.elamur.ai/free/en/papers/ho-2020-constructing-multi-hop-qa-dataset/)). Detecting which cues a trained model actually relies on is a goal it shares with [interpretability](https://md.research.elamur.ai/free/en/topics/interpretability/) research.

### Reasoning beyond spans

Discrete reasoning is still hard. On DROP, BERT reached 32.7% F1 against 96.4% for expert humans, and a numerically aware model reached 47.0% ([Dua et al. 2019](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/)). CommonsenseQA's learning curve suggested that even 100K training examples would bring BERT-large only to about 75% accuracy ([Talmor et al. 2019](https://md.research.elamur.ai/free/en/papers/talmor-2019-commonsenseqa-question-answering-challenge-targeting/)). In ChartQA, errors in nested operations and in extracting data from the chart kept accuracy low on human-written questions ([Masry et al. 2022](https://md.research.elamur.ai/free/en/papers/masry-2022-chartqa-benchmark-question-answering-about/)). Biomedical reasoning over quantitative findings also left a gap: 68.1% for the best system versus 78.0% for a single human on PubMedQA ([Jin et al. 2019](https://md.research.elamur.ai/free/en/papers/jin-2019-pubmedqa-dataset-biomedical-research-question/)).

### Knowing when not to answer

Abstention is far from solved. On SQuAD 2.0, the best tested system trailed humans by 23.2 F1 points, and a system that always abstains already scored 48.9 F1 ([Rajpurkar et al. 2018](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/)). With large generative models the problem changes shape. A fluent answer may be wrong, and WebGPT's citations were meant partly to make fact-checking easier ([Nakano et al. 2021](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/)).

### Retrieval quality and incomplete knowledge

Retrievers trained on incompletely labelled data learn from false negatives ([Qu et al. 2021](https://md.research.elamur.ai/free/en/papers/qu-2021-rocketqa-optimized-training-approach-dense/)). Dense vectors can miss rare entities ([Lee et al. 2019](https://md.research.elamur.ai/free/en/papers/lee-2019-latent-retrieval-weakly-supervised-open/)). Knowledge graphs lack links that questions need ([Saxena et al. 2020](https://md.research.elamur.ai/free/en/papers/saxena-2020-improving-multi-hop-question-answering/)). Combining sparse, dense and structured evidence in one system is still an open engineering and modelling problem.

### Evaluation of free-form and ambiguous answers

Exact match punishes correct paraphrases, and F1 rewards partial overlap that may be wrong. Natural Questions showed that real questions are often ambiguous or have several valid answers, which forced multi-reference metrics and a lower human ceiling ([Kwiatkowski et al. 2019](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)). For long-form answers, automatic metrics track quality poorly. In ELI5, human raters preferred the gold answer over every model in at least 85.5% of comparisons ([Fan et al. 2019](https://md.research.elamur.ai/free/en/papers/fan-2019-eli5-long-form-question-answering/)). Truthfulness is a separate axis from fluency. On the adversarial TruthfulQA benchmark, 75% of WebGPT's answers were true, but only 54% were both true and informative ([Nakano et al. 2021](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/)).

### Languages beyond English

Cross-lingual transfer still drops sharply away from English. With XLM on MLQA, F1 fell from 74.9 in English to 48.8 in Hindi ([Lewis et al. 2019](https://md.research.elamur.ai/free/en/papers/lewis-2019-mlqa-evaluating-cross-lingual-extractive/)). On TyDi QA, a SQuAD-trained model transferred zero-shot reached 56.4 F1 on the gold-passage task, against 79.0 after training on TyDi QA itself ([Clark et al. 2020](https://md.research.elamur.ai/free/en/papers/clark-2020-tydi-qa-benchmark-information-seeking/)). Building native, information-seeking data for under-served languages remains a priority shared with work on [low-resource languages](https://md.research.elamur.ai/free/en/topics/low-resource-languages/).

## The library in figures

Neighbouring subjects: papers shared Natural language processing shares the most: 311 of the 350 papers here (89%) stand on both.

**The numbers**

| Shelf | Papers shared |
| --- | --- |
| [Natural language processing](https://md.research.elamur.ai/free/en/concepts/natural-language-processing/) | 311 shared · 89% |
| [Natural language understanding](https://md.research.elamur.ai/free/en/concepts/natural-language-understanding/) | 93 shared · 27% |
| [Large language models](https://md.research.elamur.ai/free/en/concepts/large-language-models/) | 54 shared · 15% |
| [Knowledge graph](https://md.research.elamur.ai/free/en/concepts/knowledge-graph/) | 29 shared · 8% |
| [Retrieval-augmented generation](https://md.research.elamur.ai/free/en/concepts/retrieval-augmented-generation/) | 29 shared · 8% |
| [Few-shot learning](https://md.research.elamur.ai/free/en/concepts/few-shot-learning/) | 20 shared · 6% |
| [Information extraction](https://md.research.elamur.ai/free/en/concepts/information-extraction/) | 17 shared · 5% |
| [Text summarization](https://md.research.elamur.ai/free/en/concepts/text-summarization/) | 16 shared · 5% |
| [Parsing](https://md.research.elamur.ai/free/en/concepts/parsing/) | 14 shared · 4% |
| [Semantics](https://md.research.elamur.ai/free/en/concepts/semantics/) | 6 shared · 2% |

## Key papers

The works this page cites, the most cited first; citations as counted by OpenAlex plus the papers of this library that cite the work, read 2026-10-24.

- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/) — Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev et al. · 2016 · 9,513 citations
- [Dense Passage Retrieval for Open-Domain Question Answering](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/) — Vladimir Karpukhin, Barlas Oguz, Sewon Min et al. · 2020 · 5,375 citations
- [Natural Questions: A Benchmark for Question Answering Research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/) — Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield et al. · 2019 · 4,035 citations
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/) — Zhilin Yang, Peng Qi, Saizheng Zhang et al. · 2018 · 3,495 citations
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/) — Pranav Rajpurkar, Robin Jia, Percy Liang · 2018 · 3,373 citations
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/) — Mandar Joshi, Eunsol Choi, Daniel Weld et al. · 2017 · 3,213 citations
- [GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering](https://md.research.elamur.ai/free/en/papers/hudson-2019-gqa-new-dataset-real-world/) — Drew A. Hudson, Christopher D. Manning · 2019 · 2,006 citations
- [Bidirectional Attention Flow for Machine Comprehension](https://md.research.elamur.ai/free/en/papers/seo-2016-bidirectional-attention-flow-machine-comprehension/) — Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi et al. · 2016 · 1,968 citations
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/) — Robin Jia, Percy Liang · 2017 · 1,942 citations
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://md.research.elamur.ai/free/en/papers/izacard-2021-leveraging-passage-retrieval-generative-models/) — Gautier Izacard, Edouard Grave · 2021 · 1,675 citations
- [Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding](https://md.research.elamur.ai/free/en/papers/fukui-2016-multimodal-compact-bilinear-pooling-visual/) — Akira Fukui, Dong Huk Park, Daylen Yang et al. · 2016 · 1,653 citations
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://md.research.elamur.ai/free/en/papers/talmor-2019-commonsenseqa-question-answering-challenge-targeting/) — Alon Talmor, Jonathan Herzig, Nicholas Lourie et al. · 2019 · 1,639 citations
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/) — Guokun Lai, Qizhe Xie, Hanxiao Liu et al. · 2017 · 1,629 citations
- [CoQA: A Conversational Question Answering Challenge](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/) — Siva Reddy, Danqi Chen, Christopher D. Manning · 2019 · 1,545 citations
- [Long Short-Term Memory-Networks for Machine Reading](https://md.research.elamur.ai/free/en/papers/cheng-2016-long-short-term-memory-networks/) — Jianpeng Cheng, Li Dong, Mirella Lapata · 2016 · 1,307 citations
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://md.research.elamur.ai/free/en/papers/lee-2019-latent-retrieval-weakly-supervised-open/) — Kenton Lee, Ming-Wei Chang, Kristina Toutanova · 2019 · 1,236 citations
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/) — Yash Goyal, Tejas Khot, Douglas Summers-Stay et al. · 2016 · 1,176 citations
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://md.research.elamur.ai/free/en/papers/jin-2019-pubmedqa-dataset-biomedical-research-question/) — Qiao Jin, Bhuwan Dhingra, Zhengping Liu et al. · 2019 · 1,160 citations
- [WikiQA: A Challenge Dataset for Open-Domain Question Answering](https://md.research.elamur.ai/free/en/papers/yang-2015-wikiqa-challenge-dataset-open-domain/) — Yi Yang, Wen-tau Yih, Christopher Meek · 2015 · 1,129 citations
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/) — Payal Bajaj, Daniel Campos, Nick Craswell et al. · 2016 · 1,112 citations
- [QuAC: Question Answering in Context](https://md.research.elamur.ai/free/en/papers/choi-2018-quac-question-answering-context/) — Eunsol Choi, He He, Mohit Iyyer et al. · 2018 · 1,062 citations
- [The NarrativeQA Reading Comprehension Challenge](https://md.research.elamur.ai/free/en/papers/kocisky-2018-narrativeqa-reading-comprehension-challenge/) — Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom et al. · 2018 · 1,031 citations
- [Learning surface text patterns for a Question Answering System](https://md.research.elamur.ai/free/en/papers/ravichandran-2002-learning-surface-text-patterns-question/) — Deepak Ravichandran, Eduard Hovy · 2002 · 972 citations
- [Semantic Parsing via Staged Query Graph Generation: Question Answering with Knowledge Base](https://md.research.elamur.ai/free/en/papers/yih-2015-semantic-parsing-staged-query-graph/) — Wen-tau Yih, Ming-Wei Chang, Xiaodong He et al. · 2015 · 962 citations
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/) — Dheeru Dua, Yizhong Wang, Pradeep Dasigi et al. · 2019 · 943 citations
- [Visual7W: Grounded Question Answering in Images](https://md.research.elamur.ai/free/en/papers/zhu-2015-visual7w-grounded-question-answering-images/) — Yuke Zhu, Oliver Groth, Michael Bernstein et al. · 2015 · 921 citations
- [Learning to Ask: Neural Question Generation for Reading Comprehension](https://md.research.elamur.ai/free/en/papers/du-2017-learning-ask-neural-question-generation/) — Xinya Du, Junru Shao, Claire Cardie · 2017 · 916 citations
- [Gated Self-Matching Networks for Reading Comprehension and Question Answering](https://md.research.elamur.ai/free/en/papers/wang-2017-gated-self-matching-networks-reading/) — Wenhui Wang, Nan Yang, Furu Wei et al. · 2017 · 827 citations
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://md.research.elamur.ai/free/en/papers/masry-2022-chartqa-benchmark-question-answering-about/) — Ahmed Masry, Do Xuan Long, Jia Qing Tan et al. · 2022 · 745 citations
- [MLQA: Evaluating Cross-lingual Extractive Question Answering](https://md.research.elamur.ai/free/en/papers/lewis-2019-mlqa-evaluating-cross-lingual-extractive/) — Patrick Lewis, Barlas Oğuz, Ruty Rinott et al. · 2019 · 738 citations
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://md.research.elamur.ai/free/en/papers/ho-2020-constructing-multi-hop-qa-dataset/) — Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara et al. · 2020 · 735 citations
- [The Value of Semantic Parse Labeling for Knowledge Base Question Answering](https://md.research.elamur.ai/free/en/papers/yih-2016-value-semantic-parse-labeling-knowledge/) — Wen-tau Yih, Matthew Richardson, Chris Meek et al. · 2016 · 716 citations
- [WebGPT: Browser-assisted question-answering with human feedback](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/) — Reiichiro Nakano, Jacob Hilton, Suchir Balaji et al. · 2021 · 695 citations
- [TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://md.research.elamur.ai/free/en/papers/clark-2020-tydi-qa-benchmark-information-seeking/) — Jonathan H. Clark, Eunsol Choi, Michael Collins et al. · 2020 · 692 citations
- [A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task](https://md.research.elamur.ai/free/en/papers/chen-2016-thorough-examination-cnn-daily-mail/) — Danqi Chen, Jason Bolton, Christopher D. Manning · 2016 · 691 citations
- [ELI5: Long Form Question Answering](https://md.research.elamur.ai/free/en/papers/fan-2019-eli5-long-form-question-answering/) — Angela Fan, Yacine Jernite, Ethan Perez et al. · 2019 · 685 citations
- [TVQA: Localized, Compositional Video Question Answering](https://md.research.elamur.ai/free/en/papers/lei-2018-tvqa-localized-compositional-video-question/) — Jie Lei, Licheng Yu, Mohit Bansal et al. · 2018 · 673 citations
- [Improving Multi-hop Question Answering over Knowledge Graphs using Knowledge Base Embeddings](https://md.research.elamur.ai/free/en/papers/saxena-2020-improving-multi-hop-question-answering/) — Apoorv Saxena, Aditay Tripathi, Partha Talukdar · 2020 · 671 citations
- [Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences](https://md.research.elamur.ai/free/en/papers/khashabi-2018-looking-beyond-surface-challenge-set/) — Daniel Khashabi, Snigdha Chaturvedi, Michael Roth et al. · 2018 · 671 citations
- [RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering](https://md.research.elamur.ai/free/en/papers/qu-2021-rocketqa-optimized-training-approach-dense/) — Yingqi Qu, Yuchen Ding, Jing Liu et al. · 2021 · 602 citations
- [Event Extraction by Answering (Almost) Natural Questions](https://md.research.elamur.ai/free/en/papers/du-2020-event-extraction-answering-almost-natural/) — Xinya Du, Claire Cardie · 2020 · 602 citations
- [QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension](https://md.research.elamur.ai/free/en/papers/yu-2018-qanet-combining-local-convolution-global/) — Adams Wei Yu, David Dohan, Minh-Thang Luong et al. · 2018 · 586 citations
- [Dynamic Coattention Networks For Question Answering](https://md.research.elamur.ai/free/en/papers/xiong-2016-dynamic-coattention-networks-question-answering/) — Caiming Xiong, Victor Zhong, Richard Socher · 2016 · 581 citations
- [Simple and Effective Multi-Paragraph Reading Comprehension](https://md.research.elamur.ai/free/en/papers/clark-2017-simple-effective-multi-paragraph-reading/) — Christopher Clark, Matt Gardner · 2017 · 558 citations
- [Learning to Compose Neural Networks for Question Answering](https://md.research.elamur.ai/free/en/papers/andreas-2016-learning-compose-neural-networks-question/) — Jacob Andreas, Marcus Rohrbach, Trevor Darrell et al. · 2016 · 552 citations

## Related concepts

- [Information extraction](https://md.research.elamur.ai/free/en/concepts/information-extraction/) — Information extraction is the automatic conversion of unstructured or semi-structured text into structured records. It finds the entities a…
- [Natural language understanding](https://md.research.elamur.ai/free/en/concepts/natural-language-understanding/) — Natural language understanding (NLU) is the branch of language technology that maps text or speech onto representations of what it means…
- [Knowledge graph](https://md.research.elamur.ai/free/en/concepts/knowledge-graph/) — A knowledge graph stores facts about the world in graph form. Its nodes stand for entities such as people, places, works and concepts, and…

## More papers

Topics: [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/)

Shelves: [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/)

Updated 24 October 2026

- [← previous concept PUNCT: punctuation](https://md.research.elamur.ai/free/en/concepts/upos-punct/)
- [next concept → reparandum: overridden disfluency](https://md.research.elamur.ai/free/en/concepts/ud-reparandum/)
- [↑ back to concepts](https://md.research.elamur.ai/free/en/concepts/)