---
title: "BoolQ: Exploring the Surprising Difficulty of — Clark 2019"
description: "Most entailment benchmarks depend on crowd workers writing hypotheses, which tends to produce examples that shallow cues can solve."
authors: ["Christopher Clark", "Kenton Lee", "Ming-Wei Chang", "Tom Kwiatkowski", "Michael Collins", "Kristina Toutanova"]
year: 2019
venue: "Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)"
source: "https://aclanthology.org/N19-1300/"
doi: "10.18653/v1/n19-1300"
arxiv: "1905.10044"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/) / BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions

# BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions

**Authors:** [Christopher Clark](https://md.research.elamur.ai/free/en/authors/christopher-clark/), [Kenton Lee](https://md.research.elamur.ai/free/en/authors/kenton-lee/), [Ming-Wei Chang](https://md.research.elamur.ai/free/en/authors/ming-wei-chang/), [Tom Kwiatkowski](https://md.research.elamur.ai/free/en/authors/tom-kwiatkowski/), [Michael Collins](https://md.research.elamur.ai/free/en/authors/michael-collins/), [Kristina Toutanova](https://md.research.elamur.ai/free/en/authors/kristina-toutanova/) · **Year:** 2019 · **Venue:** Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/N19-1300/) · [PDF](https://aclanthology.org/N19-1300.pdf) · [DOI 10.18653/v1/n19-1300](https://doi.org/10.18653/v1/n19-1300) · [arXiv 1905.10044](https://arxiv.org/abs/1905.10044) · [OpenAlex](https://openalex.org/W2946659172)

**Citations:** 312 citations (openalex-snapshot · 2026-10-14) · [985](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Most entailment benchmarks depend on crowd workers writing hypotheses, which tends to produce examples that shallow cues can solve. This paper tests inference with yes/no questions instead: questions real users typed into Google search without being prompted and without knowing the answer. The authors reuse the Natural Questions (NQ) annotation pipeline and add a filter for yes/no queries. The result is BoolQ, about 16k questions. Each is paired with a Wikipedia paragraph that an annotator judged sufficient to answer it, and each is labeled yes or no. Their analysis shows that many questions ask for non-factoid information and need entailment-style reasoning across a whole paragraph. They then compare transfer-learning baselines built on entailment, paraphrase, extractive QA and unsupervised pre-training. Crowd-sourced NLI data, especially MultiNLI, transfers best, and it still helps on top of BERT. Fine-tuning BERT-large on MultiNLI and then on BoolQ reaches 80.4% test accuracy, compared with 62% for the majority class and 90% for humans. BoolQ is therefore a hard benchmark tied to a real task, and supervised entailment data appears to add something that language-model pre-training lacks.

## Key points
- **Data source:** questions come from anonymized Google search queries that begin with one of 13 indicator words ("is", "does", "can" and so on) and that return a Wikipedia page among the top five results. Each question is paired with an annotator-selected passage rather than the full article, and the article title is also provided.
- **Test accuracy:** majority class 62.31%; recurrent model + MultiNLI 67.52%; BERT-large + MultiNLI 80.43%. Human annotators reach about 90%.
- **MultiNLI on top of BERT:** pre-training BERT-large on MultiNLI before BoolQ fine-tuning adds about 3.5 points. Removing the contradiction class from MultiNLI drops dev accuracy to 78.43%, close to plain BERT-large. The authors conclude that contradiction examples are the key ingredient.
- **Datasets in a similar format help little:** QA-style data with multi-sentence passages (SQuAD 2.0, RACE, and a yes/no subset of MS Marco) transferred poorly. NQ, QNLI, QQP and SNLI gave smaller gains than MultiNLI.
- **Small training sets:** with only 1,000 training examples, MultiNLI pre-training improves BERT-large by nearly 10 points. At that size, the recurrent model with MultiNLI even beats BERT-large without it. Data and code were released at goo.gl/boolq.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

985 papers of our library cite it (of the 140673 whose bibliographies we read); 71 of them have a page here, 914 not yet.

- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-3-technical-report-highly/)
- [Muppet: Massive Multi-task Representations with Pre-Finetuning](https://md.research.elamur.ai/free/en/papers/aghajanyan-2021-muppet-massive-multi-task-representations/)
- [Yi: Open Foundation Models by 01.AI](https://md.research.elamur.ai/free/en/papers/ai-2024-yi-open-foundation-models-01/)
- [The Falcon Series of Open Language Models](https://md.research.elamur.ai/free/en/papers/almazrouei-2023-falcon-series-open-language-models/)
- [Detecting Language Model Attacks with Perplexity](https://md.research.elamur.ai/free/en/papers/alon-2023-detecting-language-model-attacks-perplexity/)
- [Qwen Technical Report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/)
- [ALLaM: Large Language Models for Arabic and English](https://md.research.elamur.ai/free/en/papers/bari-2024-allam-large-language-models-arabic/)
- [Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions](https://md.research.elamur.ai/free/en/papers/bianchi-2023-safety-tuned-llamas-lessons-improving/)
- [Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://md.research.elamur.ai/free/en/papers/burns-2023-weak-strong-generalization-eliciting-strong/)
- [Adapting Language Models to Compress Contexts](https://md.research.elamur.ai/free/en/papers/chevalier-2023-adapting-language-models-compress-contexts/)
- ["I'll be there for you": The One with Understanding Indirect Answers](https://md.research.elamur.ai/free/en/papers/damgaard-2021-i-ll-be-there-you/)
- [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://md.research.elamur.ai/free/en/papers/du-2021-glam-efficient-scaling-language-models/)
- [English Machine Reading Comprehension Datasets: A Survey](https://md.research.elamur.ai/free/en/papers/dzendzik-2021-english-machine-reading-comprehension-datasets/)
- [LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding](https://md.research.elamur.ai/free/en/papers/elhoushi-2024-layerskip-enabling-early-exit-inference/)
- [Evaluating Models' Local Decision Boundaries via Contrast Sets](https://md.research.elamur.ai/free/en/papers/gardner-2020-evaluating-models-local-decision-boundaries/)
- [Gemini: A Family of Highly Capable Multimodal Models](https://md.research.elamur.ai/free/en/papers/gemini-2023-team-gemini-family-highly-capable-multimodal/)
- [PPT: Pre-trained Prompt Tuning for Few-shot Learning](https://md.research.elamur.ai/free/en/papers/gu-2022-ppt-pre-trained-prompt-tuning/)
- [MiniLLM: On-Policy Distillation of Large Language Models](https://md.research.elamur.ai/free/en/papers/gu-2023-minillm-policy-distillation-large-language/)
- [LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models](https://md.research.elamur.ai/free/en/papers/guha-2023-legalbench-collaboratively-built-benchmark-measuring/)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://md.research.elamur.ai/free/en/papers/han-2024-parameter-efficient-fine-tuning-large/)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://md.research.elamur.ai/free/en/papers/he-2020-deberta-decoding-enhanced-bert-disentangled/)
- [AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators](https://md.research.elamur.ai/free/en/papers/he-2023-annollm-making-large-language-models/)
- [Surface Form Competition: Why the Highest Probability Answer Isn't Always Right](https://md.research.elamur.ai/free/en/papers/holtzman-2021-surface-form-competition-why-highest/)
- [LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models](https://md.research.elamur.ai/free/en/papers/hu-2023-llm-adapters-adapter-family-parameter/)
- [Language Is Not All You Need: Aligning Perception with Language Models](https://md.research.elamur.ai/free/en/papers/huang-2023-language-all-need-aligning-perception/)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://md.research.elamur.ai/free/en/papers/jin-2019-pubmedqa-dataset-biomedical-research-question/)
- [UNIFIEDQA: Crossing Format Boundaries with a Single QA System](https://md.research.elamur.ai/free/en/papers/khashabi-2020-unifiedqa-crossing-format-boundaries-single/)
- [Textbooks Are All You Need II: phi-1.5 technical report](https://md.research.elamur.ai/free/en/papers/li-2023-textbooks-all-need-ii-phi/)
- [DataComp-LM: In search of the next generation of training sets for language models](https://md.research.elamur.ai/free/en/papers/li-2024-datacomp-lm-search-next-generation/)
- [RA-DIT: Retrieval-Augmented Dual Instruction Tuning](https://md.research.elamur.ai/free/en/papers/lin-2023-ra-dit-retrieval-augmented-dual/)
- [GPT Understands, Too](https://md.research.elamur.ai/free/en/papers/liu-2021-gpt-understands-too/)
- [WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation](https://md.research.elamur.ai/free/en/papers/liu-2022-wanli-worker-ai-collaboration-natural/)
- [大语言模型的幻觉问题研究综述](https://md.research.elamur.ai/free/en/papers/liu-2024-hallucinations-large-language/)
- [LLM-QAT: Data-Free Quantization Aware Training for Large Language Models](https://md.research.elamur.ai/free/en/papers/liu-2024-llm-qat-data-free-quantization/)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://md.research.elamur.ai/free/en/papers/longpre-2023-flan-collection-designing-data-methods/)
- ["I'd rather just go to bed": Understanding Indirect Answers](https://md.research.elamur.ai/free/en/papers/louis-2020-i-d-rather-just-go/)
- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://md.research.elamur.ai/free/en/papers/ma-2023-llm-pruner-structural-pruning-large/)
- [ShortGPT: Layers in Large Language Models are More Redundant Than You Expect](https://md.research.elamur.ai/free/en/papers/men-2024-shortgpt-layers-large-language-models/)
- [Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs](https://md.research.elamur.ai/free/en/papers/microsoft-2025-phi-4-mini-technical-report/)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/min-2021-recent-advances-natural-language-processing/)
- [Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/minaee-2024-large-language-models-survey/)
- [SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning](https://md.research.elamur.ai/free/en/papers/mirzaee-2021-spartqa-textual-question-answering-benchmark/)
- [KCIF: Knowledge-Conditioned Instruction Following](https://md.research.elamur.ai/free/en/papers/murthy-2026-kcif-knowledge-conditioned-instruction-following/)
- [A Comprehensive Overview of Large Language Models](https://md.research.elamur.ai/free/en/papers/naveed-2023-comprehensive-overview-large-language-models/)
- [True Few-Shot Learning with Language Models](https://md.research.elamur.ai/free/en/papers/perez-2021-true-few-shot-learning-language/)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-adapterfusion-non-destructive-task-composition/)
- [Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work?](https://md.research.elamur.ai/free/en/papers/pruksachatkun-2020-intermediate-task-transfer-learning-pretrained/)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://md.research.elamur.ai/free/en/papers/qin-2023-chatgpt-general-purpose-natural-language/)
- [Disentangling Indirect Answers to Yes-No Questions in Real Conversations](https://md.research.elamur.ai/free/en/papers/sanagavarapu-2022-disentangling-indirect-answers-yes-no/)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/)
- [How many data points is a prompt worth?](https://md.research.elamur.ai/free/en/papers/scao-2021-how-many-data-points-is/)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/schick-2021-it-s-not-just-size/)
- [Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models](https://md.research.elamur.ai/free/en/papers/sengupta-2023-jais-jais-chat-arabic-centric/)
- [OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models](https://md.research.elamur.ai/free/en/papers/shao-2023-omniquant-omnidirectionally-calibrated-quantization-large/)
- [When Your Language Model Cannot Even Do Determiners Right: Probing for Anti-Presuppositions and the Maximize Presupposition! Principle](https://md.research.elamur.ai/free/en/papers/sieker-2023-when-your-language-model-cannot/)
- [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://md.research.elamur.ai/free/en/papers/smith-2022-using-deepspeed-megatron-train-megatron/)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/)
- [大模型参数高效微调方法综述: 技术、趋势与挑战](https://md.research.elamur.ai/free/en/papers/tang-2026-parameter-efficient-fine/)
- [Galactica: A Large Language Model for Science](https://md.research.elamur.ai/free/en/papers/taylor-2022-galactica-large-language-model-science/)
- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://md.research.elamur.ai/free/en/papers/ustun-2024-aya-model-instruction-finetuned-open/)
- [SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer](https://md.research.elamur.ai/free/en/papers/vu-2022-spot-better-frozen-model-adaptation/)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/)
- [BloombergGPT: A Large Language Model for Finance](https://md.research.elamur.ai/free/en/papers/wu-2023-bloomberggpt-large-language-model-finance/)
- [ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers](https://md.research.elamur.ai/free/en/papers/yao-2022-zeroquant-efficient-affordable-post-training/)
- [CREPE: Open-Domain Question Answering with False Presuppositions](https://md.research.elamur.ai/free/en/papers/yu-2023-crepe-open-domain-question-answering/)
- [Instruction Tuning for Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/zhang-2023-instruction-tuning-large-language-models/)
- [TinyLlama: An Open-Source Small Language Model](https://md.research.elamur.ai/free/en/papers/zhang-2024-tinyllama-open-source-small-language/)
- [Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning](https://md.research.elamur.ai/free/en/papers/zhang-2026-beyond-random-sampling-efficient-language/)
- [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://md.research.elamur.ai/free/en/papers/zou-2023-representation-engineering-top-down-approach/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{clark2019boolq,
  title = {{BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions}},
  author = {Clark, Christopher and Lee, Kenton and Chang, Ming-Wei and Kwiatkowski, Tom and Collins, Michael and Toutanova, Kristina},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1300},
  eprint = {1905.10044},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/n19-1300}
}
```

- [← previous BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://md.research.elamur.ai/free/en/papers/lee-2019-biobert-pre-trained-biomedical-language/)
- [next → Bridging the Gap between Training and Inference for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/zhang-2019-bridging-gap-between-training-inference/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/)

## References

41 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual Question Answering. In Proceedings of the IEEE international conference on computer vision. [find](https://scholar.google.com/scholar?q=Stanislaw+Antol%2C+Aishwarya+Agrawal%2C+Jiasen+Lu%2C+Margaret+Mitchell%2C+Dhruv+Batra%2C+C+Lawrence+Zitnick%2C+and+Devi+Parikh%2E+2015%2E+VQA%3A+Visual+Question+Answering%2E+In+Proceedings+of+the+IEEE+international+confe)
2. Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. SciTail: A Textual Entailment Dataset from Science Question Answering. In Proceedings of AAAI. [find](https://scholar.google.com/scholar?q=Tushar+Khot%2C+Ashish+Sabharwal%2C+and+Peter+Clark%2E+2018%2E+SciTail%3A+A+Textual+Entailment+Dataset+from+Science+Question+Answering%2E+In+Proceedings+of+AAAI%2E)
3. Diederik P Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. ICLR. [find](https://scholar.google.com/scholar?q=Diederik+P+Kingma+and+Jimmy+Ba%2E+2014%2E+Adam%3A+A+Method+for+Stochastic+Optimization%2E+ICLR%2E)
4. Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2011. The Seventh PASCAL Recognizing Textual Entailment Challenge. In TAC. [find](https://scholar.google.com/scholar?q=Luisa+Bentivogli%2C+Peter+Clark%2C+Ido+Dagan%2C+and+Danilo+Giampiccolo%2E+2011%2E+The+Seventh+PASCAL+Recognizing+Textual+Entailment+Challenge%2E+In+TAC%2E)
5. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red-field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural… [find](https://scholar.google.com/scholar?q=Tom+Kwiatkowski%2C+Jennimaria+Palomaki%2C+Olivia+Red%2Dfield%2C+Michael+Collins%2C+Ankur+Parikh%2C+Chris+Alberti%2C+Danielle+Epstein%2C+Illia+Polosukhin%2C+Matthew+Kelcey%2C+Jacob+Devlin%2C+Kenton+Lee%2C+Kristina+N%2E+Toutanov) [our card](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
6. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A Large Annotated Corpus for Learning Natural Language Inference. EMNLP. [find](https://scholar.google.com/scholar?q=Samuel+R+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D+Manning%2E+2015%2E+A+Large+Annotated+Corpus+for+Learning+Natural+Language+Inference%2E+EMNLP%2E)
7. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-Scale Reading Comprehension Dataset from Examinations. EMNLP. [find](https://scholar.google.com/scholar?q=Guokun+Lai%2C+Qizhe+Xie%2C+Hanxiao+Liu%2C+Yiming+Yang%2C+and+Eduard+Hovy%2E+2017%2E+Race%3A+Large%2DScale+Reading+Comprehension+Dataset+from+Examinations%2E+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/)
8. Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009. The Sixth PASCAL Recognizing Textual Entailment Challenge. In TAC. [find](https://scholar.google.com/scholar?q=Luisa+Bentivogli%2C+Peter+Clark%2C+Ido+Dagan%2C+and+Danilo+Giampiccolo%2E+2009%2E+The+Sixth+PASCAL+Recognizing+Textual+Entailment+Challenge%2E+In+TAC%2E)
9. Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for Natural Language Inference. In ACL. [find](https://scholar.google.com/scholar?q=Qian+Chen%2C+Xiaodan+Zhu%2C+Zhenhua+Ling%2C+Si+Wei%2C+Hui+Jiang%2C+and+Diana+Inkpen%2E+2017%2E+Enhanced+LSTM+for+Natural+Language+Inference%2E+In+ACL%2E)
10. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuaC: Question Answering in Context. EMNLP. [find](https://scholar.google.com/scholar?q=Eunsol+Choi%2C+He+He%2C+Mohit+Iyyer%2C+Mark+Yatskar%2C+Wen%2Dtau+Yih%2C+Yejin+Choi%2C+Percy+Liang%2C+and+Luke+Zettlemoyer%2E+2018%2E+QuaC%3A+Question+Answering+in+Context%2E+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/choi-2018-quac-question-answering-context/)
11. Alexis Conneau and Douwe Kiela. 2018. Senteval: An Evaluation Toolkit for Universal Sentence Representations. In LREC. [find](https://scholar.google.com/scholar?q=Alexis+Conneau+and+Douwe+Kiela%2E+2018%2E+Senteval%3A+An+Evaluation+Toolkit+for+Universal+Sentence+Representations%2E+In+LREC%2E) [our card](https://md.research.elamur.ai/free/en/papers/conneau-2018-senteval-evaluation-toolkit-universal-sentence/)
12. Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018. Transforming Question Answering Datasets Into Natural Language Inference Datasets. arXiv:1809.02922. Version 2. [find](https://scholar.google.com/scholar?q=Dorottya+Demszky%2C+Kelvin+Guu%2C+and+Percy+Liang%2E+2018%2E+Transforming+Question+Answering+Datasets+Into+Natural+Language+Inference+Datasets%2E+arXiv%3A1809%2E02922%2E+Version+2%2E)
13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805. Version 1. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2018%2E+BERT%3A+Pre%2Dtraining+of+Deep+Bidirectional+Transformers+for+Language+Understanding%2E+arXiv%3A1810%2E04805%2E+Version+1%2E)
14. Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking NLI Systems with Sentences that Require Simple Lexical Inferences. ACL. [find](https://scholar.google.com/scholar?q=Max+Glockner%2C+Vered+Shwartz%2C+and+Yoav+Goldberg%2E+2018%2E+Breaking+NLI+Systems+with+Sentences+that+Require+Simple+Lexical+Inferences%2E+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/glockner-2018-breaking-nli-systems-sentences-that/)
15. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A Smith. 2018. Annotation Artifacts in Natural Language Inference Data. NAACL. [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Swabha+Swayamdipta%2C+Omer+Levy%2C+Roy+Schwartz%2C+Samuel+R+Bowman%2C+and+Noah+A+Smith%2E+2018%2E+Annotation+Artifacts+in+Natural+Language+Inference+Data%2E+NAACL%2E)
16. Minghao Hu, Yuxing Peng, Zhen Huang, Nan Yang, Ming Zhou, et al. 2018. Read+ Verify: Machine Reading Comprehension with Unanswerable Questions. CoRR. [find](https://scholar.google.com/scholar?q=Minghao+Hu%2C+Yuxing+Peng%2C+Zhen+Huang%2C+Nan+Yang%2C+Ming+Zhou%2C+et+al%2E+2018%2E+Read%2B+Verify%3A+Machine+Reading+Comprehension+with+Unanswerable+Questions%2E+CoRR%2E)
17. Robin Jia and Percy Liang. 2017. Adversarial Examples for Evaluating Reading Comprehension Systems. EMNLP. [find](https://scholar.google.com/scholar?q=Robin+Jia+and+Percy+Liang%2E+2017%2E+Adversarial+Examples+for+Evaluating+Reading+Comprehension+Systems%2E+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/)
18. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. ACL. [find](https://scholar.google.com/scholar?q=Mandar+Joshi%2C+Eunsol+Choi%2C+Daniel+S+Weld%2C+and+Luke+Zettlemoyer%2E+2017%2E+Triviaqa%3A+A+Large+Scale+Distantly+Supervised+Challenge+Dataset+for+Reading+Comprehension%2E+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/)
19. R Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference. Computing Research Repository, arXiv:1902.01007. Version 1. [find](https://scholar.google.com/scholar?q=R+Thomas+McCoy%2C+Ellie+Pavlick%2C+and+Tal+Linzen%2E+2019%2E+Right+for+the+Wrong+Reasons%3A+Diagnosing+Syntactic+Heuristics+in+Natural+Language+Inference%2E+Computing+Research+Repository%2C+arXiv%3A1902%2E01007%2E+Versio) [our card](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/)
20. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. In EMNLP. [find](https://scholar.google.com/scholar?q=Todor+Mihaylov%2C+Peter+Clark%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+2018%2E+Can+a+Suit+of+Armor+Conduct+Electricity%3F+A+New+Dataset+for+Open+Book+Question+Answering%2E+In+EMNLP%2E)
21. Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in Pre-Training Distributed Word Representations. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018). [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Edouard+Grave%2C+Piotr+Bojanowski%2C+Christian+Puhrsch%2C+and+Armand+Joulin%2E+2018%2E+Advances+in+Pre%2DTraining+Distributed+Word+Representations%2E+In+Proceedings+of+the+International+Conference+on) [our card](https://md.research.elamur.ai/free/en/papers/mikolov-2017-advances-pre-training-distributed-word/)
22. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A Human Generated Machine Reading Comprehension Dataset. arXiv:1611.09268. Version 3. [find](https://scholar.google.com/scholar?q=Tri+Nguyen%2C+Mir+Rosenberg%2C+Xia+Song%2C+Jianfeng+Gao%2C+Saurabh+Tiwary%2C+Rangan+Majumder%2C+and+Li+Deng%2E+2016%2E+MS+MARCO%3A+A+Human+Generated+Machine+Reading+Comprehension+Dataset%2E+arXiv%3A1611%2E09268%2E+Version+3%2E) [our card](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/)
23. Ankur P Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A Decomposable Attention Model for Natural Language Inference. In EMNLP. [find](https://scholar.google.com/scholar?q=Ankur+P+Parikh%2C+Oscar+Ta%CC%88ckstro%CC%88m%2C+Dipanjan+Das%2C+and+Jakob+Uszkoreit%2E+2016%2E+A+Decomposable+Attention+Model+for+Natural+Language+Inference%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/parikh-2016-decomposable-attention-model-natural-language/)
24. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations. In NAACL. [find](https://scholar.google.com/scholar?q=Matthew+E+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+2018%2E+Deep+Contextualized+Word+Representations%2E+In+NAACL%2E)
25. Jason Phang, Thibault Févry, and Samuel R Bowman. 2018. Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks. Computing Research Repository, arXiv:1811.01088. Version 2. [find](https://scholar.google.com/scholar?q=Jason+Phang%2C+Thibault+Fe%CC%81vry%2C+and+Samuel+R+Bowman%2E+2018%2E+Sentence+Encoders+on+STILTs%3A+Supplementary+Training+on+Intermediate+Labeled%2Ddata+Tasks%2E+Computing+Research+Repository%2C+arXiv%3A1811%2E01088%2E+Versio) [our card](https://md.research.elamur.ai/free/en/papers/phang-2018-sentence-encoders-stilts-supplementary-training/)
26. Adam Poliak, Aparajita Haldar, Rachel Rudinger, J Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. 2018. Collecting Diverse Natural Language Inference Problems for Sentence Representation Evaluation. In EMNLP. [find](https://scholar.google.com/scholar?q=Adam+Poliak%2C+Aparajita+Haldar%2C+Rachel+Rudinger%2C+J+Edward+Hu%2C+Ellie+Pavlick%2C+Aaron+Steven+White%2C+and+Benjamin+Van+Durme%2E+2018%2E+Collecting+Diverse+Natural+Language+Inference+Problems+for+Sentence+Repres) [our card](https://md.research.elamur.ai/free/en/papers/poliak-2018-collecting-diverse-natural-language-inference/)
27. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving Language Understanding by Generative Pre-training. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+2018%2E+Improving+Language+Understanding+by+Generative+Pre%2Dtraining%2E)
28. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know What You Don’t Know: Unanswerable Questions for SQuAD. ACL. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Robin+Jia%2C+and+Percy+Liang%2E+2018%2E+Know+What+You+Don%E2%80%99t+Know%3A+Unanswerable+Questions+for+SQuAD%2E+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/)
29. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ Questions for Machine Comprehension of Text. EMNLP. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+2016%2E+Squad%3A+100%2C000%2B+Questions+for+Machine+Comprehension+of+Text%2E+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
30. Siva Reddy, Danqi Chen, and Christopher D Manning. 2018. CoQA: A Conversational Question Answering Challenge. In TACL. [find](https://scholar.google.com/scholar?q=Siva+Reddy%2C+Danqi+Chen%2C+and+Christopher+D+Manning%2E+2018%2E+CoQA%3A+A+Conversational+Question+Answering+Challenge%2E+In+TACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/)
31. Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Rocktäschel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. 2018. Interpretation of Natural Language Rules in Conversational Machine Reading. In EMNLP. [find](https://scholar.google.com/scholar?q=Marzieh+Saeidi%2C+Max+Bartolo%2C+Patrick+Lewis%2C+Sameer+Singh%2C+Tim+Rockta%CC%88schel%2C+Mike+Sheldon%2C+Guillaume+Bouchard%2C+and+Sebastian+Riedel%2E+2018%2E+Interpretation+of+Natural+Language+Rules+in+Conversational+Mac)
32. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional Attention Flow for Machine Comprehension. In ICLR. [find](https://scholar.google.com/scholar?q=Minjoon+Seo%2C+Aniruddha+Kembhavi%2C+Ali+Farhadi%2C+and+Hannaneh+Hajishirzi%2E+2017%2E+Bidirectional+Attention+Flow+for+Machine+Comprehension%2E+In+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/seo-2016-bidirectional-attention-flow-machine-comprehension/)
33. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+Bowman%2E+2018%2E+GLUE%3A+A+Multi%2DTask+Benchmark+and+Analysis+Platform+for+Natural+Language+Understanding%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
34. Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing Datasets for Multi-hop Reading Comprehension Across Documents. In ACL. [find](https://scholar.google.com/scholar?q=Johannes+Welbl%2C+Pontus+Stenetorp%2C+and+Sebastian+Riedel%2E+2018%2E+Constructing+Datasets+for+Multi%2Dhop+Reading+Comprehension+Across+Documents%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/welbl-2018-constructing-datasets-multi-hop-reading/)
35. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks. In ICLR. [find](https://scholar.google.com/scholar?q=Jason+Weston%2C+Antoine+Bordes%2C+Sumit+Chopra%2C+Alexander+M+Rush%2C+Bart+van+Merrie%CC%88nboer%2C+Armand+Joulin%2C+and+Tomas+Mikolov%2E+Towards+AI%2DComplete+Question+Answering%3A+A+Set+of+Prerequisite+Toy+Tasks%2E+In+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/weston-2015-towards-ai-complete-question-answering/)
36. Adina Williams, Nikita Nangia, and Samuel R Bowman. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. NACL. [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+R+Bowman%2E+2018%2E+A+Broad%2DCoverage+Challenge+Corpus+for+Sentence+Understanding+through+Inference%2E+NACL%2E)
37. Qi Wu, Damien Teney, Peng Wang, Chunhua Shen, Anthony Dick, and Anton van den Hengel. 2017. Visual Question Answering: A Survey of Methods and Datasets. In Computer Vision and Image Understanding. Elsevier. [find](https://scholar.google.com/scholar?q=Qi+Wu%2C+Damien+Teney%2C+Peng+Wang%2C+Chunhua+Shen%2C+Anthony+Dick%2C+and+Anton+van+den+Hengel%2E+2017%2E+Visual+Question+Answering%3A+A+Survey+of+Methods+and+Datasets%2E+In+Computer+Vision+and+Image+Understanding%2E+Els)
38. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Peng+Qi%2C+Saizheng+Zhang%2C+Yoshua+Bengio%2C+William+W+Cohen%2C+Ruslan+Salakhutdinov%2C+and+Christopher+D+Manning%2E+2018%2E+Hotpotqa%3A+A+Dataset+for+Diverse%2C+Explainable+Multi%2Dhop+Question+Answering%2E+) [our card](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/)
39. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. Swag: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference. In EMNLP. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Yonatan+Bisk%2C+Roy+Schwartz%2C+and+Yejin+Choi%2E+2018%2E+Swag%3A+A+Large%2DScale+Adversarial+Dataset+for+Grounded+Commonsense+Inference%2E+In+EMNLP%2E)
40. Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension. Computing Research Repository, arXiv:1810.12885. Version 1. [find](https://scholar.google.com/scholar?q=Sheng+Zhang%2C+Xiaodong+Liu%2C+Jingjing+Liu%2C+Jianfeng+Gao%2C+Kevin+Duh%2C+and+Benjamin+Van+Durme%2E+2018%2E+ReCoRD%3A+Bridging+the+Gap+between+Human+and+Machine+Commonsense+Reading+Comprehension%2E+Computing+Research) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2018-record-bridging-gap-between-human/)
41. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books. In Proceedings of the IEEE international conference on computer vision, pages 19–… [find](https://scholar.google.com/scholar?q=Yukun+Zhu%2C+Ryan+Kiros%2C+Rich+Zemel%2C+Ruslan+Salakhutdinov%2C+Raquel+Urtasun%2C+Antonio+Torralba%2C+and+Sanja+Fidler%2E+2015%2E+Aligning+Books+and+Movies%3A+Towards+Story%2DLike+Visual+Explanations+by+Watching+Movies+) [our card](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/)