---
title: "Analyzing Compositionality-Sensitivity of NLI Models — Nie 2018"
description: "They build two rule-based adversaries that keep the vocabulary of the premise but change its meaning. 23 references, cited by 43 papers here."
authors: ["Yixin Nie", "Yicheng Wang", "Mohit Bansal"]
year: 2018
venue: "arXiv:1811.07033"
source: "https://arxiv.org/abs/1811.07033"
doi: "10.48550/arxiv.1811.07033"
arxiv: "1811.07033"
licence: "arXiv non-exclusive distribution licence (read only, no redistribution)"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/nie-2018-analyzing-compositionality-sensitivity-nli-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nie-2018-analyzing-compositionality-sensitivity-nli-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/) / Analyzing Compositionality-Sensitivity of NLI Models

# Analyzing Compositionality-Sensitivity of NLI Models

**Authors:** [Yixin Nie](https://md.research.elamur.ai/free/en/authors/yixin-nie/), [Yicheng Wang](https://md.research.elamur.ai/free/en/authors/yicheng-wang/) and [Mohit Bansal](https://md.research.elamur.ai/free/en/authors/mohit-bansal/) · **Year:** 2018 · **Venue:** [arXiv:1811.07033](https://md.research.elamur.ai/free/en/venues/proceedings-of-the-aaai-conference-on-artificial-intelligence/) ([Proceedings of the AAAI Conference on Artificial Intelligence 2018](https://md.research.elamur.ai/free/en/venues/proceedings-of-the-aaai-conference-on-artificial-intelligence/2018/)) · **Publisher:** [Association for the Advancement of Artificial Intelligence](https://md.research.elamur.ai/free/en/publishers/association-for-the-advancement-of-artificial-intelligence/) · **Licence:** [arXiv non-exclusive distribution licence (read only, no redistribution)](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1811.07033) · [PDF](https://arxiv.org/pdf/1811.07033) · [DOI 10.48550/arxiv.1811.07033](https://doi.org/10.48550/arxiv.1811.07033) · [arXiv 1811.07033](https://arxiv.org/abs/1811.07033) · [OpenAlex](https://openalex.org/W2963542100)

**Citations:** 76 citations (OpenAlex, 2026-10-14) · [43](https://md.research.elamur.ai/free/en/papers/nie-2018-analyzing-compositionality-sensitivity-nli-models/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
NLI systems reach near-human accuracy on SNLI and MNLI. This paper asks whether they use sentence structure or mostly match words. The authors test seven strong models, both sentence encoders and co-attention models. They build two rule-based adversaries that keep the vocabulary of the premise but change its meaning. On these, every model fails badly. Two more experiments support this: models whose RNNs are replaced by fully-connected layers, or that are trained on shuffled words, still score well on the standard test sets. Adversarial training is also too narrow, because learning to handle one adversary type does not carry over to another.

As a broader tool, the authors propose compositionality-sensitivity (CS) testing. A bag-of-words regression model scores each natural test example with a lexically-misleading score (LMS), and evaluation keeps only examples where that model confidently prefers a wrong label. On these subsets, models that look equal on standard benchmarks separate by architecture: bag-of-words models do worst, sequential models do better, and syntax-tree models do best. The authors release the code and data.

## Key points
- Seven models were evaluated: RSE, G-TLSTM, DAM, ESIM, S-TLSTM, DIIN and DR-BiLSTM. Their SNLI accuracies range from about 85% to 88.3%.
- **SOSWAP adversary:** it swaps subject and object in SNLI premises with subject-verb-object structure, producing contradictions. It yielded 971 examples, built using dependency parses from the Stanford Parser (CoreNLP 3.8.0).
- **ADDAMOD adversary:** it adds the same adjective to different nouns in the premise and hypothesis, producing neutral pairs.
- On these adversaries, models overwhelmingly predict entailment. DAM, for example, labels about 99.7% of SOSWAP pairs as entailment.
- Removing composition costs little on standard data. ESIM falls only from 88.17 to 86.79 on SNLI with shuffled training words, and RSE (BoW) still reaches 85.02. On the CS subsets, models drop sharply while humans stay at roughly 80–89%.

## Interesting ideas
- Build a hard test set by keeping only examples where a bag-of-words model is confidently wrong; this filters natural data, so no synthetic sentences are needed, and it isolates cases that require structure.
- Adversaries that keep the words unchanged but alter meaning, such as swapping subject and object or moving a modifier to another noun, expose models that ignore who did what to whom.
- Ablate structure deliberately, by shuffling input words or deleting the sequence model, and check whether the score drops; if it does not, the benchmark cannot measure structural understanding.
- A model can solve one targeted phenomenon, such as double negation, through a lexical shortcut, so success on a diagnostic item should be checked against what a bag-of-words model could do.
- Training on one adversary type causes overfitting rather than general robustness, so evaluation should span several unrelated perturbation families.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Use a bag-of-words baseline as a negative control, scoring Mova's accuracy separately on items where that baseline is confidently wrong, to show the tree-based pipeline earns its keep.
- **I2** (frames): Generate role-swapped and modifier-moved variants of parsed sentences and check that the semantic-role frames flip correctly, giving cheap regression tests for the role assigner.
- **I3** (answer): Feed word-shuffled school tests and fable questions to the readers; if accuracy barely changes, the test items are lexically solvable and need replacing.
- **I4** (learn): Attach a lexical-shortcut score to each diagnostic or self-play item so that passing easy items is not taken as proof that a rule was learned.
- **I5** (learn): When inducing repair rules from one error type, hold out a different perturbation family to confirm the rules generalize rather than memorize the pattern.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of North Carolina at Chapel Hill](https://md.research.elamur.ai/free/en/universities/university-of-north-carolina-at-chapel-hill/) — University, Chapel Hill, United States · 37 works · 14,332 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works

## Cite

Cite as: Yixin Nie, Yicheng Wang and Mohit Bansal. 2018. Analyzing Compositionality-Sensitivity of NLI Models. arXiv:1811.07033.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{nie2018analyzing,
  title = {{Analyzing Compositionality-Sensitivity of NLI Models}},
  author = {Yixin Nie, Yicheng Wang and Mohit Bansal},
  year = {2018},
  doi = {10.48550/arxiv.1811.07033},
  eprint = {1811.07033},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1811.07033}
}
```

- [← previous Parsing Tweets into Universal Dependencies](https://md.research.elamur.ai/free/en/papers/liu-2018-parsing-tweets-into-universal-dependencies/)
- [next → Automatic Detection of Fake News](https://md.research.elamur.ai/free/en/papers/rosas-2018-automatic-detection-fake-news/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/)

## References

23 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bowman, S. R.; Angeli, G.; Potts, C.; and Manning, C. D. 2015. A large annotated corpus for learning natural language inference. In EMNLP. [↗](https://doi.org/10.48550/arxiv.1508.05326)
2. Carmona, V. I. S.; Mitchell, J.; and Riedel, S. 2018. [Behavior analysis of nli models: Uncovering the influence of three factors on robustness](https://md.research.elamur.ai/free/en/papers/sanchez-2018-behavior-analysis-nli-models-uncovering/). NAACL. [↗](https://doi.org/10.48550/arxiv.1805.04212)
3. Chen, D., and Manning, C. 2014. [A fast and accurate dependency parser using neural networks](https://md.research.elamur.ai/free/en/papers/chen-2014-fast-accurate-dependency-parser-neural/). In EMNLP.
4. Chen, Q.; Zhu, X.; Ling, Z.-H.; Wei, S.; Jiang, H.; and Inkpen, D. 2017. Enhanced lstm for natural language inference. In ACL. [find](https://scholar.google.com/scholar?q=Chen%2C+Q%2E%3B+Zhu%2C+X%2E%3B+Ling%2C+Z%2E%2DH%2E%3B+Wei%2C+S%2E%3B+Jiang%2C+H%2E%3B+and+Inkpen%2C+D%2E+2017%2E+Enhanced+lstm+for+natural+language+inference%2E+In+ACL%2E)
5. Choi, J.; Yoo, K. M.; and Lee, S.-g. 2017. Learning to compose task-specific tree structures. In AAAI. [↗](https://doi.org/10.1609/aaai.v32i1.11975)
6. Dasgupta, I.; Guo, D.; Stuhlmüller, A.; Gershman, S. J.; and Goodman, N. D. 2018. Evaluating compositionality in sentence embeddings. arXiv preprint arXiv:1802.04302. [↗](https://arxiv.org/abs/1802.04302)
7. Evans, R.; Saxton, D.; Amos, D.; Kohli, P.; and Grefenstette, E. 2018. Can neural networks understand logical entailment? ICLR. [find](https://scholar.google.com/scholar?q=Evans%2C+R%2E%3B+Saxton%2C+D%2E%3B+Amos%2C+D%2E%3B+Kohli%2C+P%2E%3B+and+Grefenstette%2C+E%2E+2018%2E+Can+neural+networks+understand+logical+entailment%3F+ICLR%2E)
8. Ghaeini, R.; Hasan, S. A.; Datla, V.; Liu, J.; Lee, K.; Qadir, A.; Ling, Y.; Prakash, A.; Fern, X. Z.; and Farri, O. 2018. Dr-bilstm: Dependent reading bidirectional lstm for natural language inference. NAACL. [find](https://scholar.google.com/scholar?q=Ghaeini%2C+R%2E%3B+Hasan%2C+S%2E+A%2E%3B+Datla%2C+V%2E%3B+Liu%2C+J%2E%3B+Lee%2C+K%2E%3B+Qadir%2C+A%2E%3B+Ling%2C+Y%2E%3B+Prakash%2C+A%2E%3B+Fern%2C+X%2E+Z%2E%3B+and+Farri%2C+O%2E+2018%2E+Dr%2Dbilstm%3A+Dependent+reading+bidirectional+lstm+for+natural+language+inferenc)
9. Glockner, M.; Shwartz, V.; and Goldberg, Y. 2018. [Breaking nli systems with sentences that require simple lexical inferences](https://md.research.elamur.ai/free/en/papers/glockner-2018-breaking-nli-systems-sentences-that/). NAACL.
10. Gong, Y.; Luo, H.; and Zhang, J. 2017. Natural language inference over interaction space. ICLR. [↗](https://doi.org/10.48550/arxiv.1709.04348)
11. Gururangan, S.; Swayamdipta, S.; Levy, O.; Schwartz, R.; Bowman, S. R.; and Smith, N. A. 2018. Annotation artifacts in natural language inference data. NAACL. [find](https://scholar.google.com/scholar?q=Gururangan%2C+S%2E%3B+Swayamdipta%2C+S%2E%3B+Levy%2C+O%2E%3B+Schwartz%2C+R%2E%3B+Bowman%2C+S%2E+R%2E%3B+and+Smith%2C+N%2E+A%2E+2018%2E+Annotation+artifacts+in+natural+language+inference+data%2E+NAACL%2E)
12. Jia, R., and Liang, P. 2017. [Adversarial examples for evaluating reading comprehension systems](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/). EMNLP.
13. Nangia, N., and Bowman, S. R. 2018. Listops: A diagnostic dataset for latent tree learning. ACL-SRW. [find](https://scholar.google.com/scholar?q=Nangia%2C+N%2E%2C+and+Bowman%2C+S%2E+R%2E+2018%2E+Listops%3A+A+diagnostic+dataset+for+latent+tree+learning%2E+ACL%2DSRW%2E)
14. Nangia, N.; Williams, A.; Lazaridou, A.; and Bowman, S. R. 2017. [The repeval 2017 shared task: Multi-genre natural language inference with sentence representations](https://md.research.elamur.ai/free/en/papers/nangia-2017-repeval-2017-shared-task-multi/). RepEval.
15. Nie, Y., and Bansal, M. 2017. Shortcut-stacked sentence encoders for multi-domain inference. RepEval. [find](https://scholar.google.com/scholar?q=Nie%2C+Y%2E%2C+and+Bansal%2C+M%2E+2017%2E+Shortcut%2Dstacked+sentence+encoders+for+multi%2Ddomain+inference%2E+RepEval%2E)
16. Parikh, A. P.; Täckström, O.; Das, D.; and Uszkoreit, J. 2016. [A decomposable attention model for natural language inference](https://md.research.elamur.ai/free/en/papers/parikh-2016-decomposable-attention-model-natural-language/). In EMNLP. [↗](https://doi.org/10.48550/arxiv.1606.01933)
17. Poliak, A.; Haldar, A.; Rudinger, R.; Hu, J. E.; Pavlick, E.; White, A. S.; and Van Durme, B. 2018a. [Towards a unified natural language inference framework to evaluate sentence representations](https://md.research.elamur.ai/free/en/papers/poliak-2018-collecting-diverse-natural-language-inference/). arXiv preprint arXiv:1804.08207. [↗](https://arxiv.org/abs/1804.08207)
18. Poliak, A.; Naradowsky, J.; Haldar, A.; Rudinger, R.; and Van Durme, B. 2018b. Hypothesis only baselines in natural language inference. *SEM. [find](https://scholar.google.com/scholar?q=Poliak%2C+A%2E%3B+Naradowsky%2C+J%2E%3B+Haldar%2C+A%2E%3B+Rudinger%2C+R%2E%3B+and+Van+Durme%2C+B%2E+2018b%2E+Hypothesis+only+baselines+in+natural+language+inference%2E+%2ASEM%2E)
19. Rajpurkar, P.; Zhang, J.; Lopyrev, K.; and Liang, P. 2016. [Squad: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). EMNLP. [↗](https://doi.org/10.48550/arxiv.1606.05250)
20. Tsuchiya, M. 2018. [Performance impact caused by hidden bias of training data for recognizing textual entailment](https://md.research.elamur.ai/free/en/papers/tsuchiya-2018-performance-impact-caused-hidden-bias/). LREC.
21. Wang, A.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S. R. 2018. [Glue: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). EMNLP. [↗](https://doi.org/10.48550/arxiv.1804.07461)
22. Wang, Z.; Hamza, W.; and Florian, R. 2017. [Bilateral multi-perspective matching for natural language sentences](https://md.research.elamur.ai/free/en/papers/wang-2017-bilateral-multi-perspective-matching-natural/). IJCAI.
23. Williams, A.; Nangia, N.; and Bowman, S. R. 2018. A broad-coverage challenge corpus for sentence understanding through inference. NAACL. [find](https://scholar.google.com/scholar?q=Williams%2C+A%2E%3B+Nangia%2C+N%2E%3B+and+Bowman%2C+S%2E+R%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+NAACL%2E)

*Updated 24 October 2026*