--- title: "Hypothesis Only Baselines in Natural Language Inference (2018)" description: "The authors test whether Natural Language Inference (NLI) datasets actually force a model to… 2 references, cited by 377 papers here." authors: ["Adam Poliak", "Jason Naradowsky", "Aparajita Haldar", "Rachel Rudinger", "Benjamin Van Durme"] year: 2018 venue: "Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics" source: "https://aclanthology.org/S18-2023/" doi: "10.18653/v1/s18-2023" arxiv: "1805.01042" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/poliak-2018-hypothesis-only-baselines-natural-language/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/poliak-2018-hypothesis-only-baselines-natural-language/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/) / Hypothesis Only Baselines in Natural Language Inference # Hypothesis Only Baselines in Natural Language Inference **Authors:** [Adam Poliak](https://md.research.elamur.ai/free/en/authors/adam-poliak/), [Jason Naradowsky](https://md.research.elamur.ai/free/en/authors/jason-naradowsky/), [Aparajita Haldar](https://md.research.elamur.ai/free/en/authors/aparajita-haldar/), [Rachel Rudinger](https://md.research.elamur.ai/free/en/authors/rachel-rudinger/) and [Benjamin Van Durme](https://md.research.elamur.ai/free/en/authors/benjamin-van-durme/) · **Year:** 2018 · **Venue:** [Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics](https://md.research.elamur.ai/free/en/venues/starsem/) ([\*SEM 2018](https://md.research.elamur.ai/free/en/venues/starsem/2018/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/S18-2023/) · [PDF](https://aclanthology.org/S18-2023.pdf) · [DOI 10.18653/v1/s18-2023](https://doi.org/10.18653/v1/s18-2023) · [arXiv 1805.01042](https://arxiv.org/abs/1805.01042) · [OpenAlex](https://openalex.org/W2798358706) **Citations:** 480 citations (OpenAlex, 2026-10-14) · [377](https://md.research.elamur.ai/free/en/papers/poliak-2018-hypothesis-only-baselines-natural-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary The authors test whether Natural Language Inference (NLI) datasets actually force a model to analyze the relationship between the premise and the hypothesis. To do so, they train a classifier that sees only the hypothesis and ignores the context entirely. It is a modified InferSent: a BiLSTM encoder with max-pooling and an MLP classifier over 300-dimensional GloVe vectors. This "degenerate" model is compared with a majority-class baseline on ten corpora. The corpora are divided into three groups by how they were created: recast (DPR, SPR, FN+), judged (ADD-1, SciTail, SICK, MPE, JOCI), and elicited (SNLI, Multi-NLI). On many of them, the premise-free model clearly outperforms the majority class. According to the authors' analysis, it exploits statistical artifacts in the data, such as characteristic words or the grammaticality of sentences, as well as background knowledge about the world. Two conclusions follow. High accuracy on an NLI benchmark does not by itself prove that a model understands language. The majority baseline is a poor indicator of the true difficulty of the task. The authors therefore recommend including a hypothesis-only baseline in the description of every new NLI corpus. ## Key points - Ten NLI corpora of varying size are studied: from DPR (~3.4K pairs) to SNLI (~550K) and Multi-NLI (~425K). These corpora have 2 or 3 label classes. - On FN+, the premise-free model reaches 76.31% for the NOT-ENTAILED class versus 56.79% for the majority baseline (+34.37%). For ENTAILED, the result barely changes: 44.18% versus 43.20%. - The authors manually examined 200 sentences from FN+. Among the hypotheses the model correctly labeled as ENTAILED, 88% are grammatical, whereas among those labeled incorrectly only 38% are. The model thus likely picks up on a correlation between a sentence's grammaticality and its label. - On SPR (956 hypotheses per property), the largest gain comes from the property "aware": 88.70% versus 59.94% (+47.99%). The model guesses the answer from the lexical semantics of the argument, for example whether it is sentient; that is, it relies on world knowledge rather than on the context. - The code and data are openly available on GitHub (azpoliak/hypothesis-only-NLI). Independently of the authors, Tsuchiya (2018) and Gururangan et al. (2018) obtained similar results on three of these ten corpora. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Johns Hopkins University](https://md.research.elamur.ai/free/en/universities/johns-hopkins-university/) — University, Baltimore, United States · 248 works · 106,397 citations - [Birla Institute of Technology and Science, Pilani](https://md.research.elamur.ai/free/en/universities/birla-institute-of-technology-and-science-pilani/) — University, Pilāni, India · 16 works · 41,879 citations - [Birla Institute of Technology and Science, Pilani - Goa Campus](https://md.research.elamur.ai/free/en/universities/birla-institute-of-technology-and-science-pilani-goa-campus/) — University, Sancoale, India · 4 works · 1,114 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works 🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works 💡 Concepts (1): [💡 Natural language understanding](https://md.research.elamur.ai/free/en/concepts/natural-language-understanding/) · 52 works ## Cite Cite as: Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger and Benjamin Van Durme. 2018. Hypothesis Only Baselines in Natural Language Inference. Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{poliak2018hypothesis, title = {{Hypothesis Only Baselines in Natural Language Inference}}, author = {Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger and Benjamin {Van Durme}}, booktitle = {Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics}, year = {2018}, doi = {10.18653/v1/s18-2023}, eprint = {1805.01042}, archivePrefix = {arXiv}, url = {https://doi.org/10.18653/v1/s18-2023} } ``` - [← previous Gender Bias in Coreference Resolution](https://md.research.elamur.ai/free/en/papers/rudinger-2018-gender-bias-coreference-resolution/) - [next → A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings](https://md.research.elamur.ai/free/en/papers/artetxe-2018-vecmap-unsupervised/) - [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/) ## References 2 references, 1 of them with a card of ours. ↗ leads to the work. 1. Ion Androutsopoulos and Prodromos Malakasiotis. 2010. [A survey of paraphrasing and textual entailment methods](https://md.research.elamur.ai/free/en/papers/androutsopoulos-2009-survey-paraphrasing-textual-entailment-methods/). Journal of Artificial Intelligence Research, 38:135–187. [↗](https://doi.org/10.1613/jair.2985) 2. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, pages 2425–2433. [↗](https://doi.org/10.1007/s11263-016-0966-6) *Updated 1 November 2026*