--- title: "Annotation Artifacts in Natural Language Inference Data (2018)" description: "The paper examines whether large corpora for natural language inference (NLI) actually require a… 1 reference, cited by 721 papers here." authors: ["Suchin Gururangan", "Swabha Swayamdipta", "Omer Levy", "Roy Schwartz", "Samuel R. Bowman", "Noah A. Smith"] year: 2018 venue: "Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)" source: "https://aclanthology.org/N18-2017/" doi: "10.18653/v1/n18-2017" arxiv: "1803.02324" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/gururangan-2018-annotation-artifacts-natural-language-inference/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gururangan-2018-annotation-artifacts-natural-language-inference/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/) / Annotation Artifacts in Natural Language Inference Data # Annotation Artifacts in Natural Language Inference Data **Authors:** [Suchin Gururangan](https://md.research.elamur.ai/free/en/authors/suchin-gururangan/), [Swabha Swayamdipta](https://md.research.elamur.ai/free/en/authors/swabha-swayamdipta/), [Omer Levy](https://md.research.elamur.ai/free/en/authors/omer-levy/), [Roy Schwartz](https://md.research.elamur.ai/free/en/authors/roy-schwartz/), [Samuel R. Bowman](https://md.research.elamur.ai/free/en/authors/samuel-r-bowman/) and [Noah A. Smith](https://md.research.elamur.ai/free/en/authors/noah-a-smith/) · **Year:** 2018 · **Venue:** [Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2018](https://md.research.elamur.ai/free/en/venues/naacl/2018/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/N18-2017/) · [PDF](https://aclanthology.org/N18-2017.pdf) · [DOI 10.18653/v1/n18-2017](https://doi.org/10.18653/v1/n18-2017) · [arXiv 1803.02324](https://arxiv.org/abs/1803.02324) · [OpenAlex](https://openalex.org/W2790415926) **Citations:** 894 citations (OpenAlex, 2026-10-14) · [721](https://md.research.elamur.ai/free/en/papers/gururangan-2018-annotation-artifacts-natural-language-inference/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary The paper examines whether large corpora for natural language inference (NLI) actually require a model to perform logical inference. The SNLI and MultiNLI corpora were created by crowdworkers. Each worker received a premise sentence and wrote three hypotheses for it: one that follows from it (entailment), one neutral (neutral), and one that contradicts it (contradiction). The authors show that this collection method leaves systematic traces in the hypotheses, called annotation artifacts. These make it possible to guess the label without seeing the premise. A fastText classifier trained only on hypotheses correctly identifies the class in about two thirds of SNLI examples and in slightly more than half of MultiNLI examples. An analysis of words via PMI and of sentence length links these traces to annotators' strategies. For entailment, annotators generalize and remove gender or exact quantities; for neutral, they add a purpose or a modifier; for contradiction, they use negation. The test sets were then split into Easy and Hard: Easy contains the examples this classifier recognized, and Hard contains the rest. Strong NLI models perform considerably worse on Hard. Thus, progress in NLI has been overestimated, and the task remains open. This is an important lesson for anyone who collects their own datasets through crowdsourcing. ## Key points - fastText (bag of words and bigrams, premise not taken into account): 67.0% on SNLI, 53.9% on MultiNLI matched, and 52.3% on mismatched. The majority-class baseline yields only about 34–35%. Logistic regression and DAM without the premise gave similar results. - Words with the highest PMI: for entailment, outdoors, animal, instrument, some, various; for neutral, tall, sad, first, favorite, because; for contradiction, nobody, no, never, nothing, sleeping. Some of these patterns were probably prompted by the example in the SNLI instructions, where dogs turn into "animals outdoors". - Length in SNLI: the median neutral hypothesis is 9 tokens long, and 60% of entailment hypotheses have at most 7 tokens. When sentences are treated as bags of words, 8.8% of entailment hypotheses are fully contained in the premise. Among neutral and contradiction hypotheses, only 0.2% are. - The DAM, ESIM, and DIIN models on SNLI: full test set, 84.7, 85.8, and 86.5; Hard, 69.4, 71.3, and 72.7; Easy, 92.4, 92.6, and 93.4. On MultiNLI matched, the Hard portion yields 55.8–64.1, and Easy yields 85.3–87.6. - The authors recommend not removing artifacts but distributing them more evenly across labels. They propose using the Hard subsets as an additional test for evaluating NLI models. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 492 works · 440,462 citations - [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 598 works · 191,225 citations - [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 402 works · 212,762 citations - [Allen Institute for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/allen-institute-for-artificial-intelligence/) — Research institute, Seattle, United States · 304 works · 106,138 citations - [New York University](https://md.research.elamur.ai/free/en/universities/new-york-university/) — University, New York, United States · 219 works · 103,487 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works 🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works 💡 Concepts (2): [💡 Natural language understanding](https://md.research.elamur.ai/free/en/concepts/natural-language-understanding/) · 52 works · [💡 Linguistic annotation](https://md.research.elamur.ai/free/en/concepts/annotation/) · 26 works ## Cite Cite as: Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman and Noah A. Smith. 2018. Annotation Artifacts in Natural Language Inference Data. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{gururangan2018annotation, title = {{Annotation Artifacts in Natural Language Inference Data}}, author = {Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman and Noah A. Smith}, booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)}, year = {2018}, doi = {10.18653/v1/n18-2017}, eprint = {1803.02324}, archivePrefix = {arXiv}, url = {https://doi.org/10.18653/v1/n18-2017} } ``` - [← previous Universal Sentence Encoder](https://md.research.elamur.ai/free/en/papers/cer-2018-universal-sentence-encoder/) - [next → Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://md.research.elamur.ai/free/en/papers/yu-2018-spider-large-scale-human-labeled/) - [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/) ## References 1 references, 1 of them with a card of ours. ↗ leads to the work. 1. Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. 2016. [Analyzing the behavior of visual question answering models](https://md.research.elamur.ai/free/en/papers/agrawal-2016-analyzing-behavior-visual-question-answering/). In Proc. of EMNLP. https://aclweb.org/anthology/D16-1203. [↗](https://aclanthology.org/D16-1203/) *Updated 30 October 2026*