---
title: "HypoNLI — paper by Liu et al. (2020) — summary and ideas"
description: "NLI datasets such as SNLI and MultiNLI let models guess the label from the hypothesis alone… 50 references, cited by 15 papers here."
authors: ["Tianyu Liu", "Xin Zheng", "Baobao Chang", "Zhifang Sui"]
year: 2020
venue: "Proceedings of the Twelfth Language Resources and Evaluation Conference"
source: "https://aclanthology.org/2020.lrec-1.846/"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2020-hyponli-exploring-artificial-patterns-hypothesis/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2020-hyponli-exploring-artificial-patterns-hypothesis/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/6/) / HypoNLI: Exploring the Artificial Patterns of Hypothesis-only Bias in Natural Language Inference

# HypoNLI: Exploring the Artificial Patterns of Hypothesis-only Bias in Natural Language Inference

**Authors:** [Tianyu Liu](https://md.research.elamur.ai/free/en/authors/tianyu-liu/), [Xin Zheng](https://md.research.elamur.ai/free/en/authors/xin-zheng/), [Baobao Chang](https://md.research.elamur.ai/free/en/authors/baobao-chang/) and [Zhifang Sui](https://md.research.elamur.ai/free/en/authors/zhifang-sui/) · **Year:** 2020 · **Venue:** [Proceedings of the Twelfth Language Resources and Evaluation Conference](https://md.research.elamur.ai/free/en/venues/lrec/) ([LREC 2020](https://md.research.elamur.ai/free/en/venues/lrec/2020/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.lrec-1.846/) · [PDF](https://aclanthology.org/2020.lrec-1.846.pdf) · [OpenAlex](https://openalex.org/W3009380369)

**Citations:** 12 citations (OpenAlex, 2026-10-14) · [15](https://md.research.elamur.ai/free/en/papers/liu-2020-hyponli-exploring-artificial-patterns-hypothesis/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
NLI datasets such as SNLI and MultiNLI let models guess the label from the hypothesis alone, because crowd workers who wrote hypotheses for a given label left recurring lexical traces. The authors make this bias explicit. They mine short, possibly gapped word patterns from training hypotheses that predict one label with high conditional probability. They show that a classifier fed only these patterns comes close to a fastText hypothesis-only baseline. Using the patterns, they divide the test sets into "easy" items, where every pattern agrees with the gold label, and "hard" items, where every pattern points elsewhere. Items with mixed signals are left out of both. They benchmark InferSent, DAM, ESIM, BERT, XLNet and RoBERTa on these subsets and find a large gap: InferSent exceeds 95% on easy items but stays below 60% on hard ones. They also test two pattern-guided debiasing baselines, down-sampling of cue-revealing training items and adversarial training of sentence encoders. The work matters because it shows that headline NLI accuracy partly reflects shortcut learning, and it releases the subsets as a diagnostic benchmark.

## Key points
- Patterns are defined as H(M, t, λ): at most M words, at most t positions between neighbouring words, and p(label | pattern) > λ. The defaults are M = 3 and t = 3, and each pattern must occur in at least 50 training instances. One example from the paper: 85.2% of hypotheses containing "video games" are labelled contradiction.
- Hypothesis-only accuracy on the SNLI test set is 64.4 for the pattern-feature MLP, 60.2 for unigram features, 67.2 for fastText and 34.3 for the majority class. On MultiNLI matched the figures are 52.9 for patterns and 53.7 for fastText.
- Subsets use λ = 0.8 for SNLI and λ = 0.7 for MultiNLI. The reported easy/hard sizes are 327/1760 for the SNLI test set, 410/1032 for MultiNLI-matched dev and 371/1085 for MultiNLI-mismatched dev.
- Results for models trained on SNLI, given as full / easy / hard accuracy:
  - InferSent: 84.5 / 97.2 / 58.9
  - ESIM: 87.6 / 97.7 / 68.2
  - BERT: 90.5 / 98.2 / 71.2
  - RoBERTa: 91.7 / 98.9 / 75.8
- Pretraining narrows the easy–hard gap but does not close it. The two proposed debiasing baselines are down-sampling and adversarial debiasing.

## Interesting ideas
- Mining gapped n-grams with a minimum frequency and a label-probability threshold gives a readable list of shortcuts, unlike an opaque hypothesis-only neural classifier.
- Splitting a test set by whether the shortcut agrees or disagrees with gold, and discarding mixed cases, turns any existing benchmark into a bias probe without new annotation.
- A partial-input classifier serves as a negative control: if it scores well above the majority class, the task leaks its answer through one side of the input.
- Removing training items whose label is fully revealed by known cues pushes a learner toward the harder signal.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Run the same mining on silver data and LLM-teacher labels to list surface cues that predict an annotation regardless of the actual sentence.
- **I2** (answer): Split school-course tests and fable QA into cue-consistent and cue-contradicting items, so that a reader which guesses from question wording shows a visible accuracy gap.
- **I3** (learn): Add question-only and text-only baselines to evaluation, so that a gain from the world model can be credited only when it beats these controls.
- **I4** (learn): When training the perceptron parser or repair rules on silver trees, down-sample sentences where a trivial heuristic already fixes the attachment, keeping the cases that need real syntax.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 100 works · 35,618 citations
- [Beijing University of Posts and Telecommunications](https://md.research.elamur.ai/free/en/universities/beijing-university-of-posts-and-telecommunications/) — University, Beijing, China · 22 works · 32,364 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works

## Cite

Cite as: Tianyu Liu, Xin Zheng, Baobao Chang and Zhifang Sui. 2020. HypoNLI: Exploring the Artificial Patterns of Hypothesis-only Bias in Natural Language Inference. Proceedings of the Twelfth Language Resources and Evaluation Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{liu2020hyponli,
  title = {{HypoNLI: Exploring the Artificial Patterns of Hypothesis-only Bias in Natural Language Inference}},
  author = {Tianyu Liu, Xin Zheng, Baobao Chang and Zhifang Sui},
  booktitle = {Proceedings of the Twelfth Language Resources and Evaluation Conference},
  year = {2020},
  url = {https://aclanthology.org/2020.lrec-1.846/}
}
```

- [← previous Code-Switching Patterns Can Be an Effective Route to Improve Performance of Downstream NLP Applications: A Case Study of Humour, Sarcasm and Hate Speech Detection](https://md.research.elamur.ai/free/en/papers/bansal-2020-code-switching-patterns-can-be/)
- [next → New Protocols and Negative Results for Textual Entailment Data Collection](https://md.research.elamur.ai/free/en/papers/bowman-2020-new-protocols-negative-results-textual/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/6/)

## References

50 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Belinkov, Y., Poliak, A., Shieber, S. M., and Van Durme, B. (2018). Mitigating bias in natural language inference using adversarial learning. [find](https://scholar.google.com/scholar?q=Belinkov%2C+Y%2E%2C+Poliak%2C+A%2E%2C+Shieber%2C+S%2E+M%2E%2C+and+Van+Durme%2C+B%2E+%282018%29%2E+Mitigating+bias+in+natural+language+inference+using+adversarial+learning%2E)
2. Belinkov, Y., Poliak, A., Shieber, S. M., Van Durme, B., and Rush, A. M. (2019). [Don’t take the premise for granted: Mitigating artifacts in natural language inference](https://md.research.elamur.ai/free/en/papers/belinkov-2019-don-t-take-premise-granted/). arXiv preprint arXiv:1907.04380. [↗](https://arxiv.org/abs/1907.04380)
3. Cai, Z., Tu, L., and Gimpel, K. (2017). Pay attention to the ending: Strong neural baselines for the ROC story cloze task. In ACL. [find](https://scholar.google.com/scholar?q=Cai%2C+Z%2E%2C+Tu%2C+L%2E%2C+and+Gimpel%2C+K%2E+%282017%29%2E+Pay+attention+to+the+ending%3A+Strong+neural+baselines+for+the+ROC+story+cloze+task%2E+In+ACL%2E)
4. Chen, Q., Zhu, X., Ling, Z.-H., Inkpen, D., and Wei, S. (2017a). [Neural natural language inference models enhanced with external knowledge](https://md.research.elamur.ai/free/en/papers/chen-2018-neural-natural-language-inference-models/). arXiv preprint arXiv:1711.04289. [↗](https://arxiv.org/abs/1711.04289)
5. Chen, Q., Zhu, X., Ling, Z., Wei, S., Jiang, H., and Inkpen, D. (2017b). Enhanced LSTM for natural language inference. In ACL. [find](https://scholar.google.com/scholar?q=Chen%2C+Q%2E%2C+Zhu%2C+X%2E%2C+Ling%2C+Z%2E%2C+Wei%2C+S%2E%2C+Jiang%2C+H%2E%2C+and+Inkpen%2C+D%2E+%282017b%29%2E+Enhanced+LSTM+for+natural+language+inference%2E+In+ACL%2E)
6. Chen, X., Sun, Y., Athiwaratkun, B., Cardie, C., and Weinberger, K. (2018). [Adversarial deep averaging networks for cross-lingual sentiment classification](https://md.research.elamur.ai/free/en/papers/chen-2018-adversarial-deep-averaging-networks-cross/). TACL, 6:557– 570. [↗](https://doi.org/10.1162/tacl_a_00039)
7. Clark, C., Yatskar, M., and Zettlemoyer, L. (2019). Don’t take the easy way out: Ensemble based methods for avoiding known dataset biases. arXiv preprint arXiv:1909.03683. [↗](https://arxiv.org/abs/1909.03683)
8. Conneau, A., Kiela, D., Schwenk, H., Barrault, L., and Bordes, A. (2017). [Supervised learning of universal sentence representations from natural language inference data](https://md.research.elamur.ai/free/en/papers/conneau-2017-supervised-learning-universal-sentence-representations/). In EMNLP, pages 670–680.
9. Dagan, I., Glickman, O., and Magnini, B. (2006). The pascal recognising textual entailment challenge. pages 177– 190. Springer. [find](https://scholar.google.com/scholar?q=Dagan%2C+I%2E%2C+Glickman%2C+O%2E%2C+and+Magnini%2C+B%2E+%282006%29%2E+The+pascal+recognising+textual+entailment+challenge%2E+pages+177%E2%80%93+190%2E+Springer%2E)
10. Dagan, I., Roth, D., Sammons, M., and Zanzotto, F. M. (2013). Recognizing Textual Entailment: Models and Applications. Synthesis Lectures on Human Language Technologies. [↗](https://doi.org/10.2200/s00509ed1v01y201305hlt023)
11. Dasgupta, I., Guo, D., Stuhlmüller, A., Gershman, S. J., and Goodman, N. D. (2018). Evaluating compositionality in sentence embeddings. arXiv preprint arXiv:1802.04302. [↗](https://arxiv.org/abs/1802.04302)
12. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805. [↗](https://arxiv.org/abs/1810.04805)
13. Feng, S., Wallace, E., and Boyd-Graber, J. (2019). Misleading failures of partial-input baselines. arXiv preprint arXiv:1905.05778. [↗](https://arxiv.org/abs/1905.05778)
14. Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., and Lempitsky, V. (2016). Domain-adversarial training of neural networks. JMLR, 17(1):2096–2030. [↗](https://doi.org/10.1007/978-3-319-58347-1_10)
15. Geiger, A., Cases, I., Karttunen, L., and Potts, C. (2018). [Stress-testing neural models of natural language inference with multiply-quantified sentences](https://md.research.elamur.ai/free/en/papers/geiger-2018-stress-testing-neural-models-natural/). arXiv preprint arXiv:1810.13033. [↗](https://arxiv.org/abs/1810.13033)
16. Glockner, M., Shwartz, V., and Goldberg, Y. (2018). [Breaking NLI systems with sentences that require simple lexical inferences](https://md.research.elamur.ai/free/en/papers/glockner-2018-breaking-nli-systems-sentences-that/). In ACL, pages 650–655.
17. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. In NIPS, pages 2672–2680. [find](https://scholar.google.com/scholar?q=Goodfellow%2C+I%2E%2C+Pouget%2DAbadie%2C+J%2E%2C+Mirza%2C+M%2E%2C+Xu%2C+B%2E%2C+Warde%2DFarley%2C+D%2E%2C+Ozair%2C+S%2E%2C+Courville%2C+A%2E%2C+and+Bengio%2C+Y%2E+%282014%29%2E+Generative+adversarial+nets%2E+In+NIPS%2C+pages+2672%E2%80%932680%2E)
18. Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. (2017). [Making the V in VQA matter: Elevating the role of image understanding in visual question answering](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/). In CVPR. [↗](https://doi.org/10.1007/s11263-018-1116-0)
19. Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S. R., and Smith, N. A. (2018). Annotation artifacts in natural language inference data. In NAACL, pages 107–112. [find](https://scholar.google.com/scholar?q=Gururangan%2C+S%2E%2C+Swayamdipta%2C+S%2E%2C+Levy%2C+O%2E%2C+Schwartz%2C+R%2E%2C+Bowman%2C+S%2E+R%2E%2C+and+Smith%2C+N%2E+A%2E+%282018%29%2E+Annotation+artifacts+in+natural+language+inference+data%2E+In+NAACL%2C+pages+107%E2%80%93112%2E)
20. He, H., Zha, S., and Wang, H. (2019). [Unlearn dataset bias in natural language inference by fitting the residual](https://md.research.elamur.ai/free/en/papers/he-2019-unlearn-dataset-bias-natural-language/). arXiv preprint arXiv:1908.10763. [↗](https://arxiv.org/abs/1908.10763)
21. Iyyer, M., Wieting, J., Gimpel, K., and Zettlemoyer, L. (2018). [Adversarial example generation with syntactically controlled paraphrase networks](https://md.research.elamur.ai/free/en/papers/iyyer-2018-adversarial-example-generation-syntactically-controlled/). In NAACL, pages 1875–1885.
22. Jaiswal, A., Wu, R. Y., Abd-Almageed, W., and Natarajan, P. (2018). Unsupervised adversarial invariance. In NIPS, pages 5097–5107. [↗](https://doi.org/10.48550/arxiv.1809.10083)
23. Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H., and Mikolov, T. (2016). [Fasttext](https://md.research.elamur.ai/free/en/papers/joulin-2016-fasttext-zip-compressing-text-classification/). zip: Compressing text classification models. arXiv preprint arXiv:1612.03651. [↗](https://arxiv.org/abs/1612.03651)
24. Levy, O., Remus, S., Biemann, C., and Dagan, I. (2015). [Do supervised distributional methods really learn lexical inference relations?](https://md.research.elamur.ai/free/en/papers/levy-2015-do-supervised-distributional-methods-really/) In NAACL, pages 970–976.
25. Lin, Z., Feng, M., Santos, C. N. d., Yu, M., Xiang, B., Zhou, B., and Bengio, Y. (2017). [A structured self-attentive sentence embedding](https://md.research.elamur.ai/free/en/papers/lin-2017-structured-self-attentive-sentence-embedding/). arXiv preprint arXiv:1703.03130. [↗](https://arxiv.org/abs/1703.03130)
26. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [↗](https://arxiv.org/abs/1907.11692)
27. Luo, F., Liu, T., He, Z., Xia, Q., Sui, Z., and Chang, B. (2018). Leveraging gloss knowledge in neural word sense disambiguation by hierarchical co-attention. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1402–1411. [↗](https://doi.org/10.60692/21hzt-91623)
28. MacCartney, B. and Manning, C. D. (2009). Natural language inference. Citeseer. [find](https://scholar.google.com/scholar?q=MacCartney%2C+B%2E+and+Manning%2C+C%2E+D%2E+%282009%29%2E+Natural+language+inference%2E+Citeseer%2E)
29. McCoy, R. T., Pavlick, E., and Linzen, T. (2019). [Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/). arXiv preprint arXiv:1902.01007. [↗](https://arxiv.org/abs/1902.01007)
30. Minervini, P. and Riedel, S. (2018). Adversarially regularising neural NLI models to integrate logical background knowledge. In CoNLL, pages 65–74. [↗](https://doi.org/10.48550/arxiv.1808.08609)
31. Moyer, D., Gao, S., Brekelmans, R., Steeg, G. V., and Galstyan, A. (2018). Evading the adversary in invariant representation. arXiv preprint arXiv:1805.09458. [↗](https://arxiv.org/abs/1805.09458)
32. Naik, A., Ravichander, A., Sadeh, N., Rosé, C. P., and Neubig, G. (2018). Stress test evaluation for natural language inference. In COLING, pages 2340–2353. [↗](https://doi.org/10.48550/arxiv.1806.00692)
33. Nie, Y., Wang, Y., and Bansal, M. (2019a). [Analyzing compositionality-sensitivity of NLI models](https://md.research.elamur.ai/free/en/papers/nie-2018-analyzing-compositionality-sensitivity-nli-models/). AAAI.
34. Nie, Y., Williams, A., Dinan, E., Bansal, M., Weston, J., and Kiela, D. (2019b). Adversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599. [↗](https://arxiv.org/abs/1910.14599)
35. Parikh, A. P., Täckström, O., Das, D., and Uszkoreit, J. (2016). [A decomposable attention model for natural language inference](https://md.research.elamur.ai/free/en/papers/parikh-2016-decomposable-attention-model-natural-language/). In EMNLP, pages 2249–2255. [↗](https://doi.org/10.48550/arxiv.1606.01933)
36. Poliak, A., Naradowsky, J., Haldar, A., Rudinger, R., and Durme, B. V. (2018). Hypothesis only baselines in natural language inference. In *SEM@NAACL-HLT, pages 180–191. [find](https://scholar.google.com/scholar?q=Poliak%2C+A%2E%2C+Naradowsky%2C+J%2E%2C+Haldar%2C+A%2E%2C+Rudinger%2C+R%2E%2C+and+Durme%2C+B%2E+V%2E+%282018%29%2E+Hypothesis+only+baselines+in+natural+language+inference%2E+In+%2ASEM%40NAACL%2DHLT%2C+pages+180%E2%80%93191%2E)
37. Rudinger, R., May, C., and Durme, B. V. (2017). [Social bias in elicited natural language inferences](https://md.research.elamur.ai/free/en/papers/rudinger-2017-social-bias-elicited-natural-language/). In EthNLP@EACL, pages 74–79.
38. Sanchez, I., Mitchell, J., and Riedel, S. (2018). [Behavior analysis of nli models: Uncovering the influence of three factors on robustness](https://md.research.elamur.ai/free/en/papers/sanchez-2018-behavior-analysis-nli-models-uncovering/). In EMNLP, volume 1, pages 1975–1985.
39. Schluter, N. and Varab, D. (2018). [When data permutations are pathological: the case of neural natural language inference](https://md.research.elamur.ai/free/en/papers/schluter-2018-when-data-permutations-are-pathological/). In EMNLP, pages 4935–4939.
40. Tsuchiya, M. (2018). [Performance impact caused by hidden bias of training data for recognizing textual entailment](https://md.research.elamur.ai/free/en/papers/tsuchiya-2018-performance-impact-caused-hidden-bias/). In LREC.
41. Wang, H., Sun, D., and Xing, E. P. (2018). What if we simply swap the two text fragments? a straightforward yet effective way to test the robustness of methods to confounding signals in nature language inference tasks. arXiv preprint arXiv:1809.02719. [↗](https://arxiv.org/abs/1809.02719)
42. Wu, W., Wang, H., Liu, T., and Ma, S. (2018). Phrase-level self-attention networks for universal sentence encoding. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3729–3738. [↗](https://doi.org/10.60692/69za1-kts24)
43. Xie, Q., Dai, Z., Du, Y., Hovy, E., and Neubig, G. (2017). Controllable invariance through adversarial feature learning. In NIPS. [↗](https://doi.org/10.48550/arxiv.1705.11122)
44. Yang, Z., Yang, D., Dyer, C., He, X., Smola, A., and Hovy, E. (2016). [Hierarchical attention networks for document classification](https://md.research.elamur.ai/free/en/papers/yang-2016-hierarchical-attention-networks-document-classification/). In NAACL 2016, pages 1480–1489.
45. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., and Le, Q. V. (2019). [Xlnet: Generalized autoregressive pretraining for language understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/). arXiv preprint arXiv:1906.08237. [↗](https://arxiv.org/abs/1906.08237)
46. Zellers, R., Bisk, Y., Schwartz, R., and Choi, Y. (2018). Swag: A large-scale adversarial dataset for grounded commonsense inference. arXiv preprint arXiv:1808.05326. [↗](https://arxiv.org/abs/1808.05326)
47. Zhang, Y., Barzilay, R., and Jaakkola, T. (2017). Aspect-augmented adversarial networks for domain adaptation. TACL, 5:515–528. [↗](https://doi.org/10.1162/tacl_a_00077)
48. Zhang, G., Bai, B., Liang, J., Bai, K., Chang, S., Yu, M., Zhu, C., and Zhao, T. (2019). Selection bias explorations and debias methods for natural language sentence matching datasets. arXiv preprint arXiv:1905.06221. 7. [↗](https://arxiv.org/abs/1905.06221)
49. Language Resource References Bowman, S. R., Angeli, G., Potts, C., and Manning, C. D. (2015). A large annotated corpus for learning natural language inference. In EMNLP. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1508.05326)
50. Williams, A., Nangia, N., and Bowman, S. (2018). A broad-coverage challenge corpus for sentence understanding through inference. In NAACL, pages 1112– 1122. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Williams%2C+A%2E%2C+Nangia%2C+N%2E%2C+and+Bowman%2C+S%2E+%282018%29%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+NAACL%2C+pages+1112%E2%80%93+1122%2E+Association+for+Computational+Linguistics%2E)

*Updated 24 October 2026*