---
title: "MedNLI Is Not Immune — paper by Herlihy and Rudinger (2021)"
description: "Herlihy and Rudinger test whether MedNLI, written by physicians over premises from MIMIC-III… 17 references, cited by 10 papers here."
authors: ["Christine Herlihy", "Rachel Rudinger"]
year: 2021
venue: "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)"
source: "https://aclanthology.org/2021.acl-short.129/"
doi: "10.18653/v1/2021.acl-short.129"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/herlihy-2021-mednli-is-not-immune-natural/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/herlihy-2021-mednli-is-not-immune-natural/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/5/) / MedNLI Is Not Immune: Natural Language Inference Artifacts in the Clinical Domain

# MedNLI Is Not Immune: Natural Language Inference Artifacts in the Clinical Domain

**Authors:** Christine Herlihy and [Rachel Rudinger](https://md.research.elamur.ai/free/en/authors/rachel-rudinger/) · **Year:** 2021 · **Venue:** [Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2021](https://md.research.elamur.ai/free/en/venues/acl/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.acl-short.129/) · [PDF](https://aclanthology.org/2021.acl-short.129.pdf) · [DOI 10.18653/v1/2021.acl-short.129](https://doi.org/10.18653/v1/2021.acl-short.129) · [OpenAlex](https://openalex.org/W3168928090)

**Citations:** 20 citations (OpenAlex, 2026-10-14) · [10](https://md.research.elamur.ai/free/en/papers/herlihy-2021-mednli-is-not-immune-natural/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Large crowd-built inference benchmarks such as SNLI and MultiNLI contain annotation artifacts: label-correlated cues that let a model guess the label from the hypothesis alone. Herlihy and Rudinger test whether MedNLI, written by physicians over premises from MIMIC-III clinical notes, avoids this problem. It does not. A premise-blind fastText classifier scores about 63–65 micro-F1, against 33.3 for chance on this class-balanced set. The authors then examine which words drive each label, using PMI over scispaCy entity-merged tokens. They test three suspected physician writing habits with χ² tests over MeSH-linked entities:
- hypernyms for entailment,
- plausible causes or co-occurring conditions for neutral,
- negation or assertions of good health for contradiction.

Next, they apply AFLite adversarial filtering to split MedNLI into easy and difficult partitions, and they release the partition ids. On the difficult part, both hypothesis-only and full-input models lose substantial accuracy. The takeaway is that expert annotation alone does not remove shortcuts. Knowledge-heavy domains need different dataset designs, and reported scores should be discounted for artifacts.

## Key points
- MedNLI has 11,232 train, 1395 dev and 1422 test pairs. Each premise received one hypothesis per class (entailment, neutral, contradiction), so the label distribution is balanced.
- The hypothesis-only fastText model reaches 64.8 (dev) and 62.6 (test) micro-F1, against a 33.3 baseline. It does best on contradiction (P 73.2 / R 72.8) and worst on entailment (P 56.7 / R 53.8), and it often confuses entailment with neutral.
- The top PMI tokens (add-50 smoothing, minimum count 5) differ by class:
  - Entailment: generic terms and modifiers such as *comorbidities*, *pressors*, *high_risk*.
  - Neutral: common conditions and behaviors such as *smoking*, *obesity*.
  - Contradiction: tokens like *no_history_of_cancer* and *normal_heart_rhythm*.
- Unlike in SNLI, hypothesis length does not separate the classes (mean about 5 tokens for every label).
- AFLite settings were 64 logistic regression models, m = 5620, k = 500, over averaged MIMIC-III fastText embeddings. On the difficult partition:
  - The hypothesis-only model falls to 0.46 dev / 0.40 test.
  - The premise-plus-hypothesis model falls from 0.53/0.51 to 0.43/0.40.

## Interesting ideas
- A premise-blind classifier works as a cheap probe for dataset shortcuts, because any score above chance shows that labels leak through surface wording.
- Collapsing multi-token entities into single underscore-joined units before counting statistics keeps meaningful spans like *normal_heart_rhythm* intact.
- Class-specific PMI with heavy smoothing and a frequency floor turns a vague suspicion of bias into a ranked, inspectable list of cue words per label.
- The authors write explicit annotator heuristics (hypernym, probable cause, assertion of good health) as premise-hypothesis patterns and test them with χ² over linked ontology entities, which makes artifacts explainable instead of merely detected.
- Adversarial filtering into easy and difficult partitions, with partition ids published, gives a harder evaluation slice without new annotation.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Train a deliberately crippled model on every benchmark Mova uses (e.g. answer-only or question-only) as a negative control, and report Mova's margin over it rather than raw accuracy.
- **I2** (tokens): Add an optional entity-merge view of tokens for corpus statistics, so PMI or frequency analyses over Mova's data treat named spans as one unit.
- **I3** (learn): Run the same PMI audit on silver data from the witness parsers or LLM teachers, to spot words that a teacher tier systematically ties to one relation or label.
- **I4** (world): Using WordNet hypernymy in the knowledge layer, flag entailment items where the "inference" is just a super-type substitution, so the answer readers can be scored separately on trivial versus genuine inferences.
- **I5** (answer): Split school-test and fable-QA sets with a simple-model ensemble filter, and track Mova's accuracy on the difficult slice as the headline number.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Maryland, College Park](https://md.research.elamur.ai/free/en/universities/university-of-maryland-college-park/) — University, College Park, United States · 83 works · 23,784 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🧬 Biochemistry, Genetics and Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/) · 116 works · [🧬 Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/molecular-biology/) · 114 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works

## Cite

Cite as: Christine Herlihy and Rachel Rudinger. 2021. MedNLI Is Not Immune: Natural Language Inference Artifacts in the Clinical Domain. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{herlihy2021mednli,
  title = {{MedNLI Is Not Immune: Natural Language Inference Artifacts in the Clinical Domain}},
  author = {Christine Herlihy and Rachel Rudinger},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-short.129},
  url = {https://doi.org/10.18653/v1/2021.acl-short.129}
}
```

- [← previous Human Rationales as Attribution Priors for Explainable Stance Detection](https://md.research.elamur.ai/free/en/papers/jayaram-2021-human-rationales-as-attribution-priors/)
- [next → Not Just Classification: Recognizing Implicit Discourse Relation on Joint Modeling of Classification and Generation](https://md.research.elamur.ai/free/en/papers/jiang-2021-not-just-classification-recognizing-implicit/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/5/)

## References

17 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d15-1075)
2. Ronan Le Bras, Swabha Swayamdipta, Chandra Bhagavatula, Rowan Zellers, Matthew E. Peters, Ashish Sabharwal, and Yejin Choi. 2020. Adversarial filters of dataset biases. [↗](https://doi.org/10.48550/arxiv.2002.04108)
3. A. L. Goldberger, L. A. N. Amaral, L. Glass, J. M. Hausdorff, P. Ch. Ivanov, R. G. Mark, J. E. Mietus, G. B. Moody, C.-K. Peng, and H. E. Stanley. 2000 (June 13). PhysioBank, PhysioToolkit, and PhysioNet: Components of a new research resource for complex physiologic signals. Circulation, 101(23):e215–e220. Circulation Electronic Pages: http://circ.ahajournals.org/content/101/23/e215.full PMID:1085218; doi: 10.1161/01.CIR.101.23.e215. [↗](https://doi.org/10.1161/01.cir.101.23.e215)
4. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n18-2017)
5. Chien-Ju Ho, Aleksandrs Slivkins, Siddharth Suri, and Jennifer Wortman Vaughan. 2015. Incentivizing high quality crowdwork. In Proceedings of the 24th International Conference on World Wide Web, WWW ’15, page 419–429, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee. [↗](https://doi.org/10.1145/2904104.2904108)
6. Alistair EW Johnson, Tom J Pollard, Lu Shen, Liwei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016. [MIMIC-III, a freely accessible critical care database](https://md.research.elamur.ai/free/en/papers/johnson-2016-mimic-iii-freely-accessible-critical/). Scientific data, 3:160035.
7. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759. [↗](https://arxiv.org/abs/1607.01759)
8. Yang Liu and Yiling Chen. 2017. Machine-learning aided peer prediction. In Proceedings of the 2017 ACM Conference on Economics and Computation, EC ’17, page 63–80, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Yang+Liu+and+Yiling+Chen%2E+2017%2E+Machine%2Dlearning+aided+peer+prediction%2E+In+Proceedings+of+the+2017+ACM+Conference+on+Economics+and+Computation%2C+EC+%E2%80%9917%2C+page+63%E2%80%9380%2C+New+York%2C+NY%2C+USA%2E+Association+for+C)
9. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. [Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/p19-1334)
10. Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar. 2019a. [ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing](https://md.research.elamur.ai/free/en/papers/neumann-2019-scispacy-fast-robust-models-biomedical/). In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 319–327, Florence, Italy. Association for Computational Linguistics.
11. Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar. 2019b. [ScispaCy: Fast and robust models for biomedical natural language processing](https://md.research.elamur.ai/free/en/papers/neumann-2019-scispacy-fast-robust-models-biomedical/). In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 319–327, Florence, Italy. Association for Computational Linguistics.
12. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180–191, New Orleans, Louisiana. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/s18-2023)
13. Alexey Romanov and Chaitanya Shivade. 2018. [Lessons from Natural Language Inference in the Clinical Domain](https://md.research.elamur.ai/free/en/papers/romanov-2018-lessons-natural-language-inference-clinical/). CoRR, abs/1808.06752. [↗](https://arxiv.org/abs/1808.06752)
14. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8732–8740. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2020%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligenc)
15. Masatoshi Tsuchiya. 2018. [Performance impact caused by hidden bias of training data for recognizing textual entailment](https://md.research.elamur.ai/free/en/papers/tsuchiya-2018-performance-impact-caused-hidden-bias/). In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
16. Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2017. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. CoRR, abs/1704.05426. [↗](https://arxiv.org/abs/1704.05426)
17. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791– 4800, Florence, Italy. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/p19-1472)

*Updated 24 October 2026*