---
title: "Looking at the Overlooked — paper by Rajaee et al. (2022)"
description: "NLI models are known to treat heavy word overlap between premise and hypothesis as a shortcut to… 37 references, cited by 7 papers here."
authors: ["Sara Rajaee", "Yadollah Yaghoobzadeh", "Mohammad Taher Pilehvar"]
year: 2022
venue: "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2022.emnlp-main.725/"
doi: "10.18653/v1/2022.emnlp-main.725"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/rajaee-2022-looking-overlooked-analysis-word-overlap/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/rajaee-2022-looking-overlooked-analysis-word-overlap/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/5/) / Looking at the Overlooked: An Analysis on the Word-Overlap Bias in Natural Language Inference

# Looking at the Overlooked: An Analysis on the Word-Overlap Bias in Natural Language Inference

**Authors:** Sara Rajaee, [Yadollah Yaghoobzadeh](https://md.research.elamur.ai/free/en/authors/yadollah-yaghoobzadeh/) and [Mohammad Taher Pilehvar](https://md.research.elamur.ai/free/en/authors/mohammad-taher-pilehvar/) · **Year:** 2022 · **Venue:** [Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2022](https://md.research.elamur.ai/free/en/venues/emnlp/2022/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.emnlp-main.725/) · [PDF](https://aclanthology.org/2022.emnlp-main.725.pdf) · [DOI 10.18653/v1/2022.emnlp-main.725](https://doi.org/10.18653/v1/2022.emnlp-main.725) · [OpenAlex](https://openalex.org/W4385573497)

**Citations:** 6 citations (OpenAlex, 2026-10-14) · 7 in our library, of the 141,840 papers whose bibliographies we read

## Summary
NLI models are known to treat heavy word overlap between premise and hypothesis as a shortcut to "entailment." This paper examines the other end of the same scale. Overlap is measured as the share of hypothesis words that also occur in the premise. When it is low or zero, models lean strongly toward "non-entailment." The authors call this the reverse word-overlap bias. BERT-base and RoBERTa-base were fine-tuned on MNLI with labels collapsed to two classes, then probed on SNLI split into overlap bins and evaluated on HANS and WANLI. Three debiasing methods were compared: long-tuning, forgettable examples, and Product of Experts. All three help on the HANS non-entailment subset but do little for the reverse bias. Two further findings challenge common assumptions. Prompt-based zero-shot tests show no overlap bias coming from pre-training, though a handful of fine-tuning examples is enough to create it. Removing minority examples does not noticeably weaken the forgettable-examples method. The study shows that a bias should be examined over its full value range, not only at the extreme that existing benchmarks happen to cover.

## Key points
- Word overlap is defined as |h∩p|/|h|. SNLI instances are grouped into overlap bins, from none (0.0) to full (1.0). In the low and no-overlap bins, BERT's confidence clearly favours non-entailment, even for gold-entailment items.
- Setup:
  - MNLI training data (433k examples), results reported on MNLI-dev matched.
  - Neutral and contradiction merged into non-entailment.
  - Training: 3 epochs, learning rate 2e-5, batch size 32, three random seeds.
- Debiasing methods tested:
  - long-tuning for 20 epochs;
  - forgettable examples, mined with a Siamese BoW model;
  - PoE with TinyBERT as the weak learner.
- BERT results on HANS− and WANLI:
  - HANS− rises from 29.3 (baseline) to 55.9 (forgettable) and 62.2 (PoE).
  - WANLI rises from 56.9 to 71.6 with PoE.
  - These gains do not carry over to the low-overlap reverse case.
- Prompt-based BERT:
  - Zero-shot is roughly balanced: HANS+ 57.5 vs HANS− 53.1.
  - With K=32 training examples, HANS+ jumps to 98.3 and HANS− falls to 3.3.
  - A training set balanced by overlap reduces the skew only partly.

## Interesting ideas
- Treat a suspected shortcut feature as a continuous scale, bin the evaluation data along it, and compare per-label confidence in each bin, so that biases at both ends become visible.
- A debiasing fix validated only on a benchmark built around one extreme (like HANS) can leave the opposite extreme untouched, so tests should cover the whole range.
- Test whether a bias comes from pre-training or from fine-tuning by using zero-shot prompts, then adding a few labelled examples and watching when the skew appears.
- Balancing training data on the suspected feature helps only partly, so data balancing alone is not a complete remedy.
- Examples like Table 1 show that entailment can hold with no shared words, through synonyms and generalisations ("couple sits in the grass" → "people are outside").

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Mova's evaluation could bin test items by lexical overlap between question and source text, checking that answers are not driven by surface word matches.
- **I2** (answer): School-test and fable QA suites could include paraphrased correct answers with near-zero word overlap, as a negative control against matching instead of reasoning.
- **I3** (learn): When training gparse or the LLM teacher tiers on small silver sets, track label or attachment skew as data grows, to catch shortcuts that small samples introduce.
- **I4** (learn): Silver data agreed by witness parsers could be stratified by surface features such as sentence length or overlap, while still keeping separate negative-control checks.
- **I5** (world): Entailment checks over the text world should rely on the WordNet/ontology links and frames, not on lexical identity, which is exactly the explainable path Mova already favours.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Amsterdam](https://md.research.elamur.ai/free/en/universities/university-of-amsterdam/) — University, Amsterdam, Netherlands · 89 works · 81,337 citations
- [University of Tehran](https://md.research.elamur.ai/free/en/universities/university-of-tehran/) — University, Tehran, Iran · 4 works · 163 citations
- [Khatam University](https://md.research.elamur.ai/free/en/universities/khatam-university/) — University, Tehran, Iran · 2 works · 27 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works

## Cite

Cite as: Sara Rajaee, Yadollah Yaghoobzadeh and Mohammad Taher Pilehvar. 2022. Looking at the Overlooked: An Analysis on the Word-Overlap Bias in Natural Language Inference. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{rajaee2022looking,
  title = {{Looking at the Overlooked: An Analysis on the Word-Overlap Bias in Natural Language Inference}},
  author = {Sara Rajaee, Yadollah Yaghoobzadeh and Mohammad Taher Pilehvar},
  booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.725},
  url = {https://doi.org/10.18653/v1/2022.emnlp-main.725}
}
```

- [← previous Introducing BEREL: BERT Embeddings for Rabbinic-Encoded Language](https://md.research.elamur.ai/free/en/papers/shmidman-2022-introducing-berel-bert-embeddings-rabbinic/)
- [next → Partial-input baselines show that NLI models can ignore context, but they don't.](https://md.research.elamur.ai/free/en/papers/srikanth-2022-partial-input-baselines-show-that/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/5/)

## References

37 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dimion Asael, Zachary Ziegler, and Yonatan Belinkov. 2022. A generative approach for mitigating structural biases in natural language inference. In Proceedings of the 11th Joint Conference on Lexical and Computational Semantics, pages 186–199, Seattle, Washington. Association for Computational Linguistics. Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Dimion+Asael%2C+Zachary+Ziegler%2C+and+Yonatan+Belinkov%2E+2022%2E+A+generative+approach+for+mitigating+structural+biases+in+natural+language+inference%2E+In+Proceedings+of+the+11th+Joint+Conference+on+Lexical+)
2. Yonatan Belinkov, Adam Poliak, Stuart Shieber, Benjamin Van Durme, and Alexander Rush. 2019. [Don’t take the premise for granted: Mitigating artifacts in natural language inference](https://md.research.elamur.ai/free/en/papers/belinkov-2019-don-t-take-premise-granted/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 877–891, Florence, Italy. Association for Computational Linguistics.
3. Matt Gardner, William Merrill, Jesse Dodge, Matthew E Peters, Alexis Ross, Sameer Singh, and Noah A Smith. 2021b. Competency problems: On finding and removing artifacts in language data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1801–1813. [↗](https://doi.org/10.48550/arxiv.2104.08646)
4. Prajjwal Bhargava, Aleksandr Drozd, and Anna Rogers. 2021. [Generalization in NLI: Ways (not) to go beyond simple heuristics](https://md.research.elamur.ai/free/en/papers/bhargava-2021-generalization-nli-ways-not-go/). In Proceedings of the Second Workshop on Insights from Negative Results in NLP, pages 125–135, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2110.01518)
5. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d15-1075)
6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.
7. Kai-Wei Chang, He He, Robin Jia, and Sameer Singh. 2021. Robustness and adversarial examples in natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts, pages 22–26, Punta Cana, Dominican Republic & Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kai%2DWei+Chang%2C+He+He%2C+Robin+Jia%2C+and+Sameer+Singh%2E+2021%2E+Robustness+and+adversarial+examples+in+natural+language+processing%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Langu)
8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1423)
9. Matt Gardner, William Merrill, Jesse Dodge, Matthew Peters, Alexis Ross, Sameer Singh, and Noah A. Smith. 2021a. Competency problems: On finding and removing artifacts in language data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1801–1813, Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2104.08646)
10. He He, Sheng Zha, and Haohan Wang. 2019. [Unlearn dataset bias in natural language inference by fitting the residual](https://md.research.elamur.ai/free/en/papers/he-2019-unlearn-dataset-bias-natural-language/). In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 132–142, Hong Kong, China. Association for Computational Linguistics.
11. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. TinyBERT: Distilling BERT for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4163– 4174, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xiaoqi+Jiao%2C+Yichun+Yin%2C+Lifeng+Shang%2C+Xin+Jiang%2C+Xiao+Chen%2C+Linlin+Li%2C+Fang+Wang%2C+and+Qun+Liu%2E+2020%2E+TinyBERT%3A+Distilling+BERT+for+natural+language+understanding%2E+In+Findings+of+the+Association+for+C)
12. Nitish Joshi, Xiang Pan, and He He. 2022. Are all spurious features in natural language alike? an analysis through a causal lens. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Nitish+Joshi%2C+Xiang+Pan%2C+and+He+He%2E+2022%2E+Are+all+spurious+features+in+natural+language+alike%3F+an+analysis+through+a+causal+lens%2E+In+Proceedings+of+EMNLP%2E)
13. Rabeeh Karimi Mahabadi, Yonatan Belinkov, and James Henderson. 2020. [End-to-end bias mitigation by modelling biases in corpora](https://md.research.elamur.ai/free/en/papers/mahabadi-2020-end-end-bias-mitigation-modelling/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8706–8716, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.acl-main.769)
14. Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2022. [Wanli: Worker and ai collaboration for natural language inference dataset creation](https://md.research.elamur.ai/free/en/papers/liu-2022-wanli-worker-ai-collaboration-natural/).
15. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Ro{bert}a: A robustly optimized {bert} pretraining approach. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Ro%7Bbert%7Da%3A+A+robustly+optimized+%7Bbert%7D+pretraining+approac)
16. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. [Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/p19-1334)
17. Michael Mendelson and Yonatan Belinkov. 2021. Debiasing methods in natural language understanding make bias more accessible. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1545–1557, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. [↗](https://doi.org/10.48448/n22w-e721)
18. Junghyun Min, R. Thomas McCoy, Dipanjan Das, Emily Pitler, and Tal Linzen. 2020. [Syntactic data augmentation increases robustness to inference heuristics](https://md.research.elamur.ai/free/en/papers/min-2020-syntactic-data-augmentation-increases-robustness/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2339–2352, Online. Association for Computational Linguistics.
19. Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340–2353, Santa Fe, New Mexico, USA. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1806.00692)
20. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180–191, New Orleans, Louisiana. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/s18-2023)
21. Victor Sanh, Thomas Wolf, Yonatan Belinkov, and Alexander M. Rush. 2021. Learning from others’ mistakes: Avoiding dataset biases without modeling them. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. [↗](https://doi.org/10.48550/arxiv.2012.01300)
22. Timo Schick and Hinrich Schütze. 2021. [Exploiting cloze-questions for few-shot text classification and natural language inference](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/). In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 255–269, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2021.eacl-main.20)
23. Tal Schuster, Darsh Shah, Yun Jie Serene Yeo, Daniel Roberto Filizzola Ortiz, Enrico Santus, and Regina Barzilay. 2019. [Towards debiasing fact verification models](https://md.research.elamur.ai/free/en/papers/schuster-2019-debiasing-fact-verification-models/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3419–3425, Hong Kong, China. Association for Computational Linguistics.
24. Roy Schwartz and Gabriel Stanovsky. 2022a. On the limitations of dataset balancing: The lost battle against spurious correlations. arXiv preprint arXiv:2204.12708. [↗](https://arxiv.org/abs/2204.12708)
25. Roy Schwartz and Gabriel Stanovsky. 2022b. On the limitations of dataset balancing: The lost battle against spurious correlations. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2182–2194, Seattle, United States. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Roy+Schwartz+and+Gabriel+Stanovsky%2E+2022b%2E+On+the+limitations+of+dataset+balancing%3A+The+lost+battle+against+spurious+correlations%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+NAACL+2)
26. Joe Stacey, Pasquale Minervini, Haim Dubossarsky, Sebastian Riedel, and Tim Rocktäschel. 2020. [Avoiding the Hypothesis-Only Bias in Natural Language Inference via Ensemble Adversarial Training](https://md.research.elamur.ai/free/en/papers/stacey-2020-avoiding-hypothesis-only-bias-natural/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8281–8291, Online. Association for Computational Linguistics.
27. Lifu Tu, Garima Lalwani, Spandana Gella, and He He. 2020. An empirical study on robustness to spurious correlations using pre-trained language models. Transactions of the Association for Computational Linguistics, 8:621–633. [↗](https://doi.org/10.1162/tacl_a_00335)
28. Prasetya Utama, Nafise Sadat Moosavi, Victor Sanh, and Iryna Gurevych. 2021. Avoiding inference heuristics in few-shot prompt-based finetuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9063–9074, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Prasetya+Utama%2C+Nafise+Sadat+Moosavi%2C+Victor+Sanh%2C+and+Iryna+Gurevych%2E+2021%2E+Avoiding+inference+heuristics+in+few%2Dshot+prompt%2Dbased+finetuning%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Metho)
29. Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. 2020a. [Mind the trade-off: Debiasing NLU models without degrading the in-distribution performance](https://md.research.elamur.ai/free/en/papers/utama-2020-mind-trade-off-debiasing-nlu/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8717–8729, Online. Association for Computational Linguistics.
30. Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. 2020b. Towards debiasing NLU models from unknown biases. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7597–7610, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Prasetya+Ajie+Utama%2C+Nafise+Sadat+Moosavi%2C+and+Iryna+Gurevych%2E+2020b%2E+Towards+debiasing+NLU+models+from+unknown+biases%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+P)
31. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n18-1101)
32. Adina Williams, Tristan Thrush, and Douwe Kiela. 2022. [ANLIzing the adversarial natural language inference dataset](https://md.research.elamur.ai/free/en/papers/williams-2022-anlizing-adversarial-natural-language-inference/). In Proceedings of the Society for Computation in Linguistics 2022, pages 23–54, online. Association for Computational Linguistics.
33. Yuxiang Wu, Matt Gardner, Pontus Stenetorp, and Pradeep Dasigi. 2022. [Generating data to mitigate spurious correlations in natural language inference datasets](https://md.research.elamur.ai/free/en/papers/wu-2022-generating-data-mitigate-spurious-correlations/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2660–2676, Dublin, Ireland. Association for Computational Linguistics.
34. Yadollah Yaghoobzadeh, Soroush Mehri, Remi Tachet des Combes, T. J. Hazen, and Alessandro Sordoni. 2021. Increasing robustness to spurious correlations using forgettable examples. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3319–3332, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yadollah+Yaghoobzadeh%2C+Soroush+Mehri%2C+Remi+Tachet+des+Combes%2C+T%2E+J%2E+Hazen%2C+and+Alessandro+Sordoni%2E+2021%2E+Increasing+robustness+to+spurious+correlations+using+forgettable+examples%2E+In+Proceedings+of+th)
35. Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. [PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification](https://md.research.elamur.ai/free/en/papers/yang-2019-paws-x-cross-lingual-adversarial/). In Proc. of EMNLP. [↗](https://doi.org/10.48550/arxiv.1908.11828)
36. Yuan Zhang, Jason Baldridge, and Luheng He. 2019. [PAWS: Paraphrase adversaries from word scrambling](https://md.research.elamur.ai/free/en/papers/zhang-2019-paws-paraphrase-adversaries-word-scrambling/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1298–1308, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1131)
37. Xiang Zhou and Mohit Bansal. 2020. [Towards robustifying NLI models against lexical dataset biases](https://md.research.elamur.ai/free/en/papers/zhou-2020-robustifying-nli-models-against-lexical/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8759– 8771, Online. Association for Computational Linguistics.

*Updated 24 October 2026*