---
title: "A large annotated corpus for learning natural language inference"
description: "All sentences describe the same scene, so coreference problems become far less frequent. 35 references, cited by 2,008 papers here."
authors: ["Samuel R. Bowman", "Gabor Angeli", "Christopher Potts", "Christopher D. Manning"]
year: 2015
venue: "Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/D15-1075/"
doi: "10.18653/v1/d15-1075"
arxiv: "1508.05326"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/bowman-2015-large-annotated-corpus-learning-natural/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bowman-2015-large-annotated-corpus-learning-natural/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/) / A large annotated corpus for learning natural language inference

# A large annotated corpus for learning natural language inference

**Authors:** [Samuel R. Bowman](https://md.research.elamur.ai/free/en/authors/samuel-r-bowman/), [Gabor Angeli](https://md.research.elamur.ai/free/en/authors/gabor-angeli/), [Christopher Potts](https://md.research.elamur.ai/free/en/authors/christopher-e-potts/) and [Christopher D. Manning](https://md.research.elamur.ai/free/en/authors/christopher-d-manning/) · **Year:** 2015 · **Venue:** [Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2015](https://md.research.elamur.ai/free/en/venues/emnlp/2015/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/D15-1075/) · [PDF](https://aclanthology.org/D15-1075.pdf) · [DOI 10.18653/v1/d15-1075](https://doi.org/10.18653/v1/d15-1075) · [arXiv 1508.05326](https://arxiv.org/abs/1508.05326) · [OpenAlex](https://openalex.org/W1840435438)

**Citations:** 3,291 citations (OpenAlex, 2026-10-14) · [2008](https://md.research.elamur.ai/free/en/papers/bowman-2015-large-annotated-corpus-learning-natural/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper addresses the task of natural language inference (NLI). The goal is to determine whether the second sentence follows from the first, contradicts it, or is unrelated to it. The authors note that existing corpora for this task suffer from three shortcomings. They are too small for modern models, some of them were generated automatically, and their labels are often ambiguous because of problems with event and entity coreference. To address this, they created the SNLI corpus. About 2.5 thousand workers on Amazon Mechanical Turk were given a caption of a Flickr30k photo without the image itself. For each caption they wrote three new sentences: one definitely true, one possibly true, and one definitely false. All sentences describe the same scene, so coreference problems become far less frequent. On this corpus the authors compared rule-based systems, a lexicalized classifier with many features, and an LSTM-based model. The latter two achieved nearly identical accuracy. The LSTM trained on SNLI was then transferred to SICK, yielding the best result among neural networks at the time. The significance of the work is that it provided, for the first time, a large NLI corpus in which all sentences and labels were produced by humans. Its size is sufficient for training neural network models.

## Key points
- SNLI contains 570,152 sentence pairs. This is two orders of magnitude more than other comparable resources. The labels are entailment, neutral, and contradiction, and the classes are balanced, since each worker wrote one sentence for each label.
- For 56,941 pairs, four additional independent judgments were collected. In 98% of cases at least three annotators assigned the same label, and in 58% all five agreed.
- The premise averages 14.1 tokens, the hypothesis 8.3. The corpus contains 37,026 distinct words. About 4 thousand additional pilot pairs from VisualGenome appear only in train.
- The LSTM's accuracy on the SICK test set depends on training: SICK only — 71.3%, SNLI only — 46.7%, with SNLI→SICK transfer — 80.8%. The best overall result at the time was 84.6% (Lai and Hockenmaier, 2014), and inter-annotator agreement is about 84%.
- The same transfer to RTE-3 did not yield competitive results: 800 training examples were not enough for the model to adapt to an unfamiliar text genre.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 434 works · 314,419 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works

💡 Concepts (2): [💡 Natural language understanding](https://md.research.elamur.ai/free/en/concepts/natural-language-understanding/) · 51 works · [💡 Linguistic annotation](https://md.research.elamur.ai/free/en/concepts/annotation/) · 26 works

## Cite

Cite as: Samuel R. Bowman, Gabor Angeli, Christopher Potts and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{bowman2015large,
  title = {{A large annotated corpus for learning natural language inference}},
  author = {Samuel R. Bowman, Gabor Angeli, Christopher Potts and Christopher D. Manning},
  booktitle = {Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1075},
  eprint = {1508.05326},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d15-1075}
}
```

- [← previous Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://md.research.elamur.ai/free/en/papers/radford-2015-unsupervised-representation-learning-deep-convolutional/)
- [next → Character-level Convolutional Networks for Text Classification](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/)

## References

35 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Johan Bos and Katja Markert. 2005. [Recognising textual entailment with logical inference](https://md.research.elamur.ai/free/en/papers/bos-2005-recognising-textual-entailment-logical-inference/). In Proc. EMNLP.
2. Samuel R. Bowman, Christopher Potts, and Christopher D. Manning. 2015. Recursive neural networks can learn logical semantics. In Proc. of the 3rd Workshop on Continuous Vector Space Models and their Compositionality. [find](https://scholar.google.com/scholar?q=Samuel+R%2E+Bowman%2C+Christopher+Potts%2C+and+Christopher+D%2E+Manning%2E+2015%2E+Recursive+neural+networks+can+learn+logical+semantics%2E+In+Proc%2E+of+the+3rd+Workshop+on+Continuous+Vector+Space+Models+and+their+C)
3. Timothy Chklovski and Patrick Pantel. 2004. Verb-Ocean: Mining the web for fine-grained semantic verb relations. In Proc. EMNLP. [find](https://scholar.google.com/scholar?q=Timothy+Chklovski+and+Patrick+Pantel%2E+2004%2E+Verb%2DOcean%3A+Mining+the+web+for+fine%2Dgrained+semantic+verb+relations%2E+In+Proc%2E+EMNLP%2E)
4. Cleo Condoravdi, Dick Crouch, Valeria de Paiva, Reinhard Stolle, and Daniel G. Bobrow. 2003. Entailment, intensionality and text understanding. In Proc. of the HLT-NAACL 2003 Workshop on Text Meaning. [find](https://scholar.google.com/scholar?q=Cleo+Condoravdi%2C+Dick+Crouch%2C+Valeria+de+Paiva%2C+Reinhard+Stolle%2C+and+Daniel+G%2E+Bobrow%2E+2003%2E+Entailment%2C+intensionality+and+text+understanding%2E+In+Proc%2E+of+the+HLT%2DNAACL+2003+Workshop+on+Text+Meaning%2E)
5. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In Machine learning challenges. Evaluating predictive uncertainty, visual object classification, and recognising tectual entailment, pages 177– 190. Springer. [↗](https://doi.org/10.1007/11736790_9)
6. Marie-Catherine de Marneffe, Anna N. Rafferty, and Christopher D. Manning. 2008. [Finding contradictions in text](https://md.research.elamur.ai/free/en/papers/marneffe-2008-finding-contradictions-text/). In Proc. ACL.
7. W. Nelson Francis and Henry Kucera. 1979. Brown corpus manual. Brown University. [find](https://scholar.google.com/scholar?q=W%2E+Nelson+Francis+and+Henry+Kucera%2E+1979%2E+Brown+corpus+manual%2E+Brown+University%2E)
8. Yaroslav Fyodorov, Yoad Winter, and Nissim Francez. 2000. A natural logic inference system. In Proc. of the 2nd Workshop on Inference in Computational Semantics. [find](https://scholar.google.com/scholar?q=Yaroslav+Fyodorov%2C+Yoad+Winter%2C+and+Nissim+Francez%2E+2000%2E+A+natural+logic+inference+system%2E+In+Proc%2E+of+the+2nd+Workshop+on+Inference+in+Computational+Semantics%2E)
9. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. [The third PASCAL recognizing textual entailment challenge](https://md.research.elamur.ai/free/en/papers/giampiccolo-2007-third-pascal-recognizing-textual-entailment/). In Proc. of the ACL-PASCAL workshop on textual entailment and paraphrasing.
10. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. [↗](https://doi.org/10.1162/neco.1997.9.8.1735)
11. Jerrold J. Katz. 1972. Semantic Theory. Harper & Row, New York. [find](https://scholar.google.com/scholar?q=Jerrold+J%2E+Katz%2E+1972%2E+Semantic+Theory%2E+Harper+%26+Row%2C+New+York%2E)
12. Dan Klein and Christopher D. Manning. 2003. [Accurate unlexicalized parsing](https://md.research.elamur.ai/free/en/papers/klein-2003-accurate-unlexicalized-parsing/). In Proc. ACL.
13. Alice Lai and Julia Hockenmaier. 2014. Illinois-LH: A denotational and distributional approach to semantics. In Proc. SemEval. [find](https://scholar.google.com/scholar?q=Alice+Lai+and+Julia+Hockenmaier%2E+2014%2E+Illinois%2DLH%3A+A+denotational+and+distributional+approach+to+semantics%2E+In+Proc%2E+SemEval%2E)
14. Hector J. Levesque. 2013. On our best behaviour. In Proc. AAAI. [↗](https://doi.org/10.1016/j.artint.2014.03.007)
15. Omer Levy, Ido Dagan, and Jacob Goldberger. 2014. Focused entailment graphs for open IE propositions. In Proc. CoNLL. [find](https://scholar.google.com/scholar?q=Omer+Levy%2C+Ido+Dagan%2C+and+Jacob+Goldberger%2E+2014%2E+Focused+entailment+graphs+for+open+IE+propositions%2E+In+Proc%2E+CoNLL%2E)
16. Bill MacCartney and Christopher D Manning. 2009. An extended model of natural logic. In Proc. of the Eighth International Conference on Computational Semantics. [find](https://scholar.google.com/scholar?q=Bill+MacCartney+and+Christopher+D+Manning%2E+2009%2E+An+extended+model+of+natural+logic%2E+In+Proc%2E+of+the+Eighth+International+Conference+on+Computational+Semantics%2E)
17. Bernardo Magnini, Roberto Zanoli, Ido Dagan, Kathrin Eichler, Günter Neumann, Tae-Gil Noh, Sebastian Pado, Asher Stern, and Omer Levy. 2014. The Excitement Open Platform for textual inferences. Proc. ACL. [find](https://scholar.google.com/scholar?q=Bernardo+Magnini%2C+Roberto+Zanoli%2C+Ido+Dagan%2C+Kathrin+Eichler%2C+Gu%CC%88nter+Neumann%2C+Tae%2DGil+Noh%2C+Sebastian+Pado%2C+Asher+Stern%2C+and+Omer+Levy%2E+2014%2E+The+Excitement+Open+Platform+for+textual+inferences%2E+Proc%2E)
18. Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. 2014a. [SemEval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment](https://md.research.elamur.ai/free/en/papers/marelli-2014-semeval-2014-task-1-evaluation/). In Proc. SemEval.
19. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014b. [A SICK cure for the evaluation of compositional distributional semantic models](https://md.research.elamur.ai/free/en/papers/marelli-2014-sick-cure-evaluation-compositional-distributional/). In Proc. LREC.
20. George A Miller. 1995. [WordNet: a lexical database for english](https://md.research.elamur.ai/free/en/papers/miller-1994-wordnet/). Communications of the ACM, 38(11):39–41. [↗](https://doi.org/10.1145/219717.219748)
21. Sebastian Padó, Tae-Gil Noh, Asher Stern, Rui Wang, and Roberto Zanoli. 2014. Design and realization of a modular architecture for textual entailment. Journal of Natural Language Engineering. [↗](https://doi.org/10.1017/s1351324913000351)
22. Ellie Pavlick, Johan Bos, Malvina Nissim, Charley Beller, Ben Van Durme, and Chris Callison-Burch. 2015. [PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification](https://md.research.elamur.ai/free/en/papers/pavlick-2015-ppdb-2-0-better-paraphrase/). In Proc. ACL.
23. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. [GloVe: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In Proc. EMNLP.
24. Lorien Y Pratt, Jack Mostow, Candace A Kamm, and Ace A Kamm. 1991. Direct transfer of learned information among neural networks. In Proc. AAAI. [find](https://scholar.google.com/scholar?q=Lorien+Y+Pratt%2C+Jack+Mostow%2C+Candace+A+Kamm%2C+and+Ace+A+Kamm%2E+1991%2E+Direct+transfer+of+learned+information+among+neural+networks%2E+In+Proc%2E+AAAI%2E)
25. Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. [Recursive deep models for semantic compositionality over a sentiment treebank](https://md.research.elamur.ai/free/en/papers/socher-2013-recursive-deep-models-semantic-compositionality/). In Proc. EMNLP.
26. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. JMLR. [find](https://scholar.google.com/scholar?q=Nitish+Srivastava%2C+Geoffrey+Hinton%2C+Alex+Krizhevsky%2C+Ilya+Sutskever%2C+and+Ruslan+Salakhutdinov%2E+2014%2E+Dropout%3A+A+simple+way+to+prevent+neural+networks+from+overfitting%2E+JMLR%2E)
27. Johan van Benthem. 2008. A brief history of natural logic. In M. Chakraborty, B. Löwe, M. Nath Mitra, and S. Sarukki, editors, Logic, Navya-Nyaya and Applications: Homage to Bimal Matilal. College Publications. [find](https://scholar.google.com/scholar?q=Johan+van+Benthem%2E+2008%2E+A+brief+history+of+natural+logic%2E+In+M%2E+Chakraborty%2C+B%2E+Lo%CC%88we%2C+M%2E+Nath+Mitra%2C+and+S%2E+Sarukki%2C+editors%2C+Logic%2C+Navya%2DNyaya+and+Applications%3A+Homage+to+Bimal+Matilal%2E+College+Pu)
28. Sida I. Wang and Christopher D. Manning. 2012. [Baselines and bigrams: Simple, good sentiment and topic classification](https://md.research.elamur.ai/free/en/papers/wang-2012-baselines-bigrams-simple-good-sentiment/). In Proc. ACL.
29. Rui Wang and Günter Neumann. 2007. [Recognizing textual entailment using sentence similarity based on dependency tree skeletons](https://md.research.elamur.ai/free/en/papers/wang-2007-recognizing-textual-entailment-sentence-similarity/). In ACL-PASCAL Workshop on Textual Entailment and Paraphrasing.
30. Gregory Ward and Betty Birner. 2004. Information structure and non-canonical syntax. In Laurence R. Horn and Gregory Ward, editors, Handbook of Pragmatics, pages 153–174. Blackwell, Oxford. [find](https://scholar.google.com/scholar?q=Gregory+Ward+and+Betty+Birner%2E+2004%2E+Information+structure+and+non%2Dcanonical+syntax%2E+In+Laurence+R%2E+Horn+and+Gregory+Ward%2C+editors%2C+Handbook+of+Pragmatics%2C+pages+153%E2%80%93174%2E+Blackwell%2C+Oxford%2E)
31. Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. 2015a. [Towards AI-complete question answering: A set of prerequisite toy tasks](https://md.research.elamur.ai/free/en/papers/weston-2015-towards-ai-complete-question-answering/). arXiv:1502.05698. [↗](https://arxiv.org/abs/1502.05698)
32. Jason Weston, Sumit Chopra, and Antoine Bordes. 2015b. Memory networks. In Proc. ICLR. [find](https://scholar.google.com/scholar?q=Jason+Weston%2C+Sumit+Chopra%2C+and+Antoine+Bordes%2E+2015b%2E+Memory+networks%2E+In+Proc%2E+ICLR%2E)
33. Terry Winograd. 1972. Understanding natural language. Cognitive Psychology, 3(1):1–191. [↗](https://doi.org/10.1016/0010-0285(72)90002-3)
34. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, 2:67– 78. [↗](https://doi.org/10.1162/tacl_a_00166)
35. Matthew D. Zeiler. 2012. [ADADELTA: an adaptive learning rate method](https://md.research.elamur.ai/free/en/papers/zeiler-2012-adadelta-adaptive-learning-rate-method/). arXiv preprint arXiv:1212.5701. [↗](https://arxiv.org/abs/1212.5701)

*Updated 1 November 2026*