---
title: "Partial-input baselines show that NLI models can ignore context,"
description: "Second, they build a 600-item evaluation set. 31 references, cited by 9 papers here. Among them: 4."
authors: ["Neha Srikanth", "Rachel Rudinger"]
year: 2022
venue: "Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies"
source: "https://aclanthology.org/2022.naacl-main.350/"
doi: "10.18653/v1/2022.naacl-main.350"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/srikanth-2022-partial-input-baselines-show-that/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/srikanth-2022-partial-input-baselines-show-that/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/5/) / Partial-input baselines show that NLI models can ignore context, but they don't.

# Partial-input baselines show that NLI models can ignore context, but they don't.

**Authors:** [Neha Srikanth](https://md.research.elamur.ai/free/en/authors/neha-srikanth/) and [Rachel Rudinger](https://md.research.elamur.ai/free/en/authors/rachel-rudinger/) · **Year:** 2022 · **Venue:** [Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2022](https://md.research.elamur.ai/free/en/venues/naacl/2022/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.naacl-main.350/) · [PDF](https://aclanthology.org/2022.naacl-main.350.pdf) · [DOI 10.18653/v1/2022.naacl-main.350](https://doi.org/10.18653/v1/2022.naacl-main.350) · [OpenAlex](https://openalex.org/W4287889150)

**Citations:** 4 citations (OpenAlex, 2026-10-14) · [9](https://md.research.elamur.ai/free/en/papers/srikanth-2022-partial-input-baselines-show-that/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
When hypothesis-only or update-only models score well on NLI benchmarks, people often conclude that full-input models trained on the same data are just exploiting annotation artifacts. Srikanth and Rudinger test that conclusion directly. They fine-tune roberta-base in two versions, one that sees only the target and one that sees context plus target, on SNLI and δ-NLI. First, they compare calibrated confidence in the gold label between the two versions. Second, they build a 600-item evaluation set. It contains test items that a target-only model, or a bag-of-words model, already gets right. For each item, the context is rewritten by hand so that the gold label changes while the target stays the same. The full-input RoBERTa model mostly follows the edited context, while a fasttext BoW model fails badly on the same items. The authors conclude that strong partial-input baselines are a warning sign that a model *could* ignore context. They do not prove that it *does*. The necessary condition of using the whole input can still be met despite artifact-laden training data.

## Key points
- Test accuracy, target-only vs. full-input RoBERTa: SNLI 0.70 vs. 0.91; δ-SNLI 0.65 vs. 0.82.
- Confidence was calibrated with temperature scaling (calibration changed little). Full-input models placed more confidence on the correct label than partial-input ones, with most points falling above the diagonal.
- The evaluation set has 600 edited items:
  - SNLI: 300 items with edited premises, 50 for each of the six label-to-label directions.
  - δ-SNLI: 300 items with edited hypotheses, 150 strengthener→weakener and 150 the reverse.
- One author edited the items and another validated them blind to the labels, giving Cohen's κ of 0.78 (SNLI) and 0.76 (δ-SNLI).
- On edited δ-SNLI items the full-input model reached about 0.75–0.76 accuracy, and most SNLI direction pairs were similar or higher. A full-input fasttext BoW model scored only 16% (SNLI) and 24.3% (δ-SNLI), which shows the edits cannot be solved lexically.

## Interesting ideas
- Pick test items where a cheap partial-input or bag-of-words model is already right, then minimally edit the *other* part of the input to flip the answer, so that only real use of context can succeed.
- Plot a model's confidence in the gold label with and without context, or before and after an edit, and read the shift relative to the diagonal as evidence of how much the extra input is used.
- Use a lexical BoW model on the edited set as a difficulty check: if it fails, the edits did not introduce trivial word cues.
- Treat a strong partial-input baseline as a necessary-but-not-sufficient warning, not a verdict, and test the specific behavior directly.
- Hold the target fixed and edit the context to swap strengthener and weakener, which yields minimal contrast pairs for defeasible reasoning.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (answer): Fable/QA tests could include items where the question alone suggests an answer, with the story edited so the true answer differs, proving the readers actually consult the world of the text.
- **I2** (learn): For each pipeline stage we can ablate an upstream input (e.g., drop coreference or frames) and chart answer confidence shifts as a negative-control diagnostic.
- **I3** (learn): Any hand-built or self-play evaluation set should be screened by a bag-of-words baseline so that high scores reflect structure, not word overlap.
- **I4** (frames): When frame/role assignments look predictable from the verb alone, contrast pairs where arguments change the roles can show whether the frame builder truly reads the dependents.
- **I5** (world): Pairs of stories differing in one state fact could check that the world model updates beliefs about events (more or less likely) rather than reacting to the hypothesis wording.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Maryland, College Park](https://md.research.elamur.ai/free/en/universities/university-of-maryland-college-park/) — University, College Park, United States · 83 works · 23,784 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works

## Cite

Cite as: Neha Srikanth and Rachel Rudinger. 2022. Partial-input baselines show that NLI models can ignore context, but they don't. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{srikanth2022partial,
  title = {{Partial-input baselines show that NLI models can ignore context, but they don't.}},
  author = {Neha Srikanth and Rachel Rudinger},
  booktitle = {Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-main.350},
  url = {https://doi.org/10.18653/v1/2022.naacl-main.350}
}
```

- [← previous Looking at the Overlooked: An Analysis on the Word-Overlap Bias in Natural Language Inference](https://md.research.elamur.ai/free/en/papers/rajaee-2022-looking-overlooked-analysis-word-overlap/)
- [next → Sarcasm Detection is Way Too Easy! An Empirical Comparison of Human and Machine Sarcasm Detection](https://md.research.elamur.ai/free/en/papers/farha-2022-sarcasm-detection-is-way-too/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/5/)

## References

31 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ron Artstein and Massimo Poesio. 2008. [Inter-coder agreement for computational linguistics](https://md.research.elamur.ai/free/en/papers/artstein-2008-survey-article-inter-coder-agreement/). Computational Linguistics, 34(4):555–596. [↗](https://doi.org/10.1162/coli.07-034-r2)
2. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d15-1075)
3. Zheng Cai, Lifu Tu, and Kevin Gimpel. 2017. Pay attention to the ending:strong neural baselines for the ROC story cloze task. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 616–622, Vancouver, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zheng+Cai%2C+Lifu+Tu%2C+and+Kevin+Gimpel%2E+2017%2E+Pay+attention+to+the+ending%3Astrong+neural+baselines+for+the+ROC+story+cloze+task%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computati)
4. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. [↗](https://doi.org/10.1177/001316446002000104)
5. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017. [Supervised learning of universal sentence representations from natural language inference data](https://md.research.elamur.ai/free/en/papers/conneau-2017-supervised-learning-universal-sentence-representations/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670–680, Copenhagen, Denmark. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d17-1070)
6. Shrey Desai and Greg Durrett. 2020. Calibration of pre-trained transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 295–302, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.emnlp-main.21)
7. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In International Conference on Machine Learning, pages 1321–1330. PMLR. [↗](https://doi.org/10.48550/arxiv.1706.04599)
8. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n18-2017)
9. Marc Harper. 2015. python-ternary: Ternary plots in python. Zenodo 10.5281/zenodo.594435. [↗](https://doi.org/10.5281/zenodo.594435)
10. Christine Herlihy and Rachel Rudinger. 2021. [MedNLI is not immune: Natural language inference artifacts in the clinical domain](https://md.research.elamur.ai/free/en/papers/herlihy-2021-mednli-is-not-immune-natural/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 1020–1027, Online. Association for Computational Linguistics. [↗](https://doi.org/10.48448/cj47-vp53)
11. Robin Jia and Percy Liang. 2017. [Adversarial examples for evaluating reading comprehension systems](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021–2031, Copenhagen, Denmark. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d17-1215)
12. Shi Feng, Eric Wallace, and Jordan Boyd-Graber. 2019. Misleading failures of partial-input baselines. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5533– 5538, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shi+Feng%2C+Eric+Wallace%2C+and+Jordan+Boyd%2DGraber%2E+2019%2E+Misleading+failures+of+partial%2Dinput+baselines%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+)
13. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427–431, Valencia, Spain. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Armand+Joulin%2C+Edouard+Grave%2C+Piotr+Bojanowski%2C+and+Tomas+Mikolov%2E+2017%2E+Bag+of+tricks+for+efficient+text+classification%2E+In+Proceedings+of+the+15th+Conference+of+the+European+Chapter+of+the+Associati)
14. Maxwell Forbes, Jena D. Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi. 2020. Social chemistry 101: Learning to reason about social and moral norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 653–670, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Maxwell+Forbes%2C+Jena+D%2E+Hwang%2C+Vered+Shwartz%2C+Maarten+Sap%2C+and+Yejin+Choi%2E+2020%2E+Social+chemistry+101%3A+Learning+to+reason+about+social+and+moral+norms%2E+In+Proceedings+of+the+2020+Conference+on+Empiric)
15. Divyansh Kaushik and Zachary C. Lipton. 2018. [How much reading does reading comprehension require? a critical investigation of popular benchmarks](https://md.research.elamur.ai/free/en/papers/kaushik-2018-how-much-reading-does-reading/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5010– 5015, Brussels, Belgium. Association for Computational Linguistics.
16. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 1–9, Prague. Association for Computational Linguistics. [↗](https://doi.org/10.3115/1654536.1654538)
17. Ronan Le Bras, Swabha Swayamdipta, Chandra Bhagavatula, Rowan Zellers, Matthew Peters, Ashish Sabharwal, and Yejin Choi. 2020. Adversarial filters of dataset biases. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1078–1088. PMLR. [↗](https://doi.org/10.48550/arxiv.2002.04108)
18. Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. [Breaking NLI systems with sentences that require simple lexical inferences](https://md.research.elamur.ai/free/en/papers/glockner-2018-breaking-nli-systems-sentences-that/). In Proceedings Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [↗](https://arxiv.org/abs/1907.11692)
19. Bill MacCartney. 2009. Natural language inference. Stanford University. [find](https://scholar.google.com/scholar?q=Bill+MacCartney%2E+2009%2E+Natural+language+inference%2E+Stanford+University%2E)
20. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. [Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/p19-1334)
21. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180–191, New Orleans, Louisiana. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/s18-2023)
22. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging NLP models. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 856–865, Melbourne, Australia. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/p18-1079)
23. Alexey Romanov and Chaitanya Shivade. 2018. [Lessons from natural language inference in the clinical domain](https://md.research.elamur.ai/free/en/papers/romanov-2018-lessons-natural-language-inference-clinical/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1586–1596, Brussels, Belgium. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1808.06752)
24. Rachel Rudinger, Vered Shwartz, Jena D Hwang, Chandra Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A Smith, and Yejin Choi. 2020. Thinking like a skeptic: Defeasible inference in natural language. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 4661–4675. Processing (Volume 1: Long Papers), pages 996– 1005. [find](https://scholar.google.com/scholar?q=Rachel+Rudinger%2C+Vered+Shwartz%2C+Jena+D+Hwang%2C+Chandra+Bhagavatula%2C+Maxwell+Forbes%2C+Ronan+Le+Bras%2C+Noah+A+Smith%2C+and+Yejin+Choi%2E+2020%2E+Thinking+like+a+skeptic%3A+Defeasible+inference+in+natural+language%2E)
25. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n18-1101)
26. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. [Transformers: State-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.emnlp-demos.6)
27. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67–78. [↗](https://doi.org/10.1162/tacl_a_00166)
28. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93–104, Brussels, Belgium. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d18-1009)
29. Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019. Atomic: An atlas of machine commonsense for if-then reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3027–3035. [↗](https://doi.org/10.1609/aaai.v33i01.33013027)
30. Masatoshi Tsuchiya. 2018. [Performance impact caused by hidden bias of training data for recognizing textual entailment](https://md.research.elamur.ai/free/en/papers/tsuchiya-2018-performance-impact-caused-hidden-bias/). In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
31. Aaron Steven White, Pushpendre Rastogi, Kevin Duh, and Benjamin Van Durme. 2017. [Inference is everything: Recasting semantic resources into a unified evaluation framework](https://md.research.elamur.ai/free/en/papers/white-2017-inference-is-everything-recasting-semantic/). In Proceedings of the Eighth International Joint Conference on Natural Language Dataset SNLI δ-NLI δ-SNLI Split train valid test train valid test train valid test Size 550,152 10,000 10,000 200,694 14,968 15,414 88,676 1,785 1,837 *Table 4: Split sizes across SNLI, δ-NLI, and δ-SNLI.*

*Updated 24 October 2026*