---
title: "PECO — paper by Saxon et al. (2023) — summary and ideas"
description: "NLI is meant to be a pairwise task: the relation between premise and hypothesis should depend on… 50 references, cited by 7 papers here."
authors: ["Michael Saxon", "Xinyi Wang", "Wenda Xu", "William Yang Wang"]
year: 2023
venue: "Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics"
source: "https://aclanthology.org/2023.eacl-main.223/"
doi: "10.18653/v1/2023.eacl-main.223"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/saxon-2023-peco-examining-single-sentence-label/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/saxon-2023-peco-examining-single-sentence-label/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/11/) / PECO: Examining Single Sentence Label Leakage in Natural Language Inference Datasets through Progressive Evaluation of Cluster Outliers

# PECO: Examining Single Sentence Label Leakage in Natural Language Inference Datasets through Progressive Evaluation of Cluster Outliers

**Authors:** [Michael Saxon](https://md.research.elamur.ai/free/en/authors/michael-saxon/), [Xinyi Wang](https://md.research.elamur.ai/free/en/authors/xinyi-wang/), [Wenda Xu](https://md.research.elamur.ai/free/en/authors/wenda-xu/) and [William Yang Wang](https://md.research.elamur.ai/free/en/authors/william-yang-wang/) · **Year:** 2023 · **Venue:** [Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2023](https://md.research.elamur.ai/free/en/venues/eacl/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.eacl-main.223/) · [PDF](https://aclanthology.org/2023.eacl-main.223.pdf) · [DOI 10.18653/v1/2023.eacl-main.223](https://doi.org/10.18653/v1/2023.eacl-main.223) · [OpenAlex](https://openalex.org/W4386566791)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [7](https://md.research.elamur.ai/free/en/papers/saxon-2023-peco-examining-single-sentence-label/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
NLI is meant to be a pairwise task: the relation between premise and hypothesis should depend on both sentences together. Many datasets, however, let a model guess the label from only one sentence. This happens because crowdworkers who write the elicited sentence leave systematic traces of the label they were given. Saxon et al. test whether newer datasets, built with adversarial, counterfactual or debiasing methods, actually removed this single sentence label leakage.

They fine-tune plain transformer classifiers (roberta-large, xlm-roberta, bert-base-chinese) on 10 NLI datasets. Each model is trained either on both sentences (PSC) or on one sentence only (SSC). They score leakage by comparing SSC accuracy with majority guessing and with PSC accuracy.

They also introduce PECO, a model-driven tool. PECO clusters samples in a learned single-sentence embedding space and flags clusters whose label distribution is heavily skewed. These clusters are the subpopulations where shortcuts live.

Leakage persists in every recent dataset tested, though in different forms. PECO provides both a dataset-level measure and concrete sample groups to target in future data collection.

## Key points
- The study covers 10 datasets with 14 test/validation sets and 17 single-sentence conditions (premise-only or hypothesis-only). The datasets are SICK, SNLI, MNLI (matched/unmatched), XNLI, FEVER NLI, ANLI (A1–A3), OCNLI, CAugNLI, SNLI debiased and MNLI debiased.
- The replication procedure is the same for all datasets: a HuggingFace sequence classifier with 3 classes and no architectural changes. Its PSC accuracy tracks published SOTA closely (PCC = 0.97).
- Two relative leakage scores are used:
  - SSC gain over the majority-label baseline, which corrects for skewed label priors such as in SICK.
  - The share of PSC accuracy that the SSC model recovers.
- SNLI shows large contiguous label-biased regions in hypothesis-only space. CAugNLI shows much smaller ones and leaks less.
- On the SNLI validation set, the authors label PECO clusters with L2 > 0.25 as "biased". About one sample in three falls into these clusters, and this ratio is similar across all dataset-cartography confidence bins. PECO therefore captures something that cartography does not.

## Interesting ideas
- Train a copy of the model with part of the input deliberately hidden; if it still beats chance by a lot, the data leaks the label through that part alone
- Cluster the model's representations and look for clusters where one label dominates, which turns vague "dataset bias" into a list of concrete sample groups to inspect or rebalance
- Report leakage relative to both a majority baseline and full-input accuracy, so that datasets with uneven label priors or different difficulty stay comparable
- High model confidence can mean "simple example" or "the model used a shortcut", and these two cases are independent, so confidence alone is a poor filter
- Compare decisions and token importance between a full-input model and a reduced-input model to see whether the full model actually relies on the same shortcuts

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Add "question-only" and "text-only" negative-control runs for every evaluation set, so that a high score cannot come from surface cues instead of the built world.
- **I2** (answer): Cluster school-test and fable QA items by question features and flag groups whose answers are predictable without the story, then rewrite or drop those items.
- **I3** (learn): Express each reader's score as gain over majority and as a fraction of the full-pipeline score, which makes component ablations honest.
- **I4** (learn): Do not keep silver parses or teacher labels just because witnesses agree confidently; also check whether they fall into shortcut-prone clusters.
- **I5** (frames): Compare a role labeler that sees the whole UD tree with one that sees only the verb lemma; high agreement would reveal role assignments driven by verb priors rather than by the arguments.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of California, Santa Barbara](https://md.research.elamur.ai/free/en/universities/university-of-california-santa-barbara/) — University, Santa Barbara, United States · 57 works · 9,752 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works

## Cite

Cite as: Michael Saxon, Xinyi Wang, Wenda Xu and William Yang Wang. 2023. PECO: Examining Single Sentence Label Leakage in Natural Language Inference Datasets through Progressive Evaluation of Cluster Outliers. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{saxon2023peco,
  title = {{PECO: Examining Single Sentence Label Leakage in Natural Language Inference Datasets through Progressive Evaluation of Cluster Outliers}},
  author = {Michael Saxon, Xinyi Wang, Wenda Xu and William Yang Wang},
  booktitle = {Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.eacl-main.223},
  url = {https://doi.org/10.18653/v1/2023.eacl-main.223}
}
```

- [← previous Open-source Frame Semantic Parsing](https://md.research.elamur.ai/free/en/papers/chanin-2023-open-source-frame-semantic-parsing/)
- [next → PEEP-Talk: A Situational Dialogue-based Chatbot for English Education](https://md.research.elamur.ai/free/en/papers/lee-2023-peep-talk-situational-dialogue-based/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/11/)

## References

50 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. [SemEval-2012 task 6: A pilot on semantic textual similarity](https://md.research.elamur.ai/free/en/papers/agirre-2012-semeval-2012-task-6-pilot/). In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 385– 393, Montréal, Canada. Association for Computational Linguistics.
2. Yonatan Belinkov, Adam Poliak, Stuart Shieber, Benjamin Van Durme, and Alexander Rush. 2019. [Don’t take the premise for granted: Mitigating artifacts in natural language inference](https://md.research.elamur.ai/free/en/papers/belinkov-2019-don-t-take-premise-granted/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 877–891, Florence, Italy. Association for Computational Linguistics.
3. Prajjwal Bhargava, Aleksandr Drozd, and Anna Rogers. 2021. [Generalization in NLI: Ways (not) to go beyond simple heuristics](https://md.research.elamur.ai/free/en/papers/bhargava-2021-generalization-nli-ways-not-go/). In Proceedings of the Second Workshop on Insights from Negative Results in NLP, pages 125–135, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2110.01518)
4. Samuel Bowman and George Dahl. 2021. What will it take to fix benchmarking in natural language understanding? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4843–4855. [↗](https://doi.org/10.48448/q9gr-k852)
5. Samuel R Bowman. 2021. When combating hype, proceed with caution. arXiv preprint arXiv:2110.08300. [↗](https://arxiv.org/abs/2110.08300)
6. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. The snli corpus. [find](https://scholar.google.com/scholar?q=Samuel+R+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D+Manning%2E+2015%2E+The+snli+corpus%2E)
7. Zeming Chen, Qiyue Gao, and Lawrence S. Moss. 2021. NeuralLog: Natural language inference with joint neural and logical reasoning. In Proceedings of *SEM 2021: The Tenth Joint Conference on Lexical and Computational Semantics, pages 78–88, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zeming+Chen%2C+Qiyue+Gao%2C+and+Lawrence+S%2E+Moss%2E+2021%2E+NeuralLog%3A+Natural+language+inference+with+joint+neural+and+logical+reasoning%2E+In+Proceedings+of+%2ASEM+2021%3A+The+Tenth+Joint+Conference+on+Lexical+an)
8. Matt Gardner, William Merrill, Jesse Dodge, Matthew Peters, Alexis Ross, Sameer Singh, and Noah A. Smith. 2021. Competency problems: On finding and removing artifacts in language data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1801–1813, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2104.08646)
9. Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1161–1166, Hong Kong, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Yoav+Goldberg%2C+and+Jonathan+Berant%2E+2019%2E+Are+we+modeling+the+task+or+the+annotator%3F+an+investigation+of+annotator+bias+in+natural+language+understanding+datasets%2E+In+Proceedings+of+the+2019)
10. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. [Annotation artifacts in natural language inference data](https://md.research.elamur.ai/free/en/papers/gururangan-2018-annotation-artifacts-natural-language-inference/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n18-2017)
11. Hai Hu, Kyle Richardson, Liang Xu, Lu Li, Sandra Kuebler, and Larry Moss. 2020. [Ocnli: Original chinese natural language inference](https://md.research.elamur.ai/free/en/papers/hu-2020-ocnli-original-chinese-natural/). In Findings of EMNLP.
12. William Huang, Haokun Liu, and Samuel R Bowman. 2020. [Counterfactually-augmented snli training data does not yield better generalization than unaugmented data](https://md.research.elamur.ai/free/en/papers/huang-2020-counterfactually-augmented-snli-training-data/). arXiv preprint arXiv:2010.04762. [↗](https://arxiv.org/abs/2010.04762)
13. Divyansh Kaushik, Amrith Setlur, Eduard Hovy, and Zachary C Lipton. 2020. Explaining the efficacy of counterfactually augmented data. International Conference on Learning Representations (ICLR). [↗](https://doi.org/10.48550/arxiv.2010.02114)
14. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. CoRR, abs/1911.02116. [↗](https://arxiv.org/abs/1911.02116)
15. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. 2017. [Visual genome: Connecting language and vision using crowdsourced dense image annotations](https://md.research.elamur.ai/free/en/papers/krishna-2016-visual-genome-connecting-language-vision/). Int. J. Comput. Vision, 123(1):32–73. [↗](https://doi.org/10.1007/s11263-016-0981-7)
16. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1809.05053)
17. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. [Albert: A lite bert for self-supervised learning of language representations](https://md.research.elamur.ai/free/en/papers/lan-2019-albert-lite-bert-self-supervised/). In International Conference on Learning Representations.
18. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805. [↗](https://arxiv.org/abs/1810.04805)
19. Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2022. [Wanli: Worker and ai collaboration for natural language inference dataset creation](https://md.research.elamur.ai/free/en/papers/liu-2022-wanli-worker-ai-collaboration-natural/). In Conference on Empirical Methods in Natural Language Processing.
20. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint arXiv:1907.11692. [↗](https://arxiv.org/abs/1907.11692)
21. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. [Learning word vectors for sentiment analysis](https://md.research.elamur.ai/free/en/papers/maas-2011-learning-word-vectors-sentiment-analysis/). In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon, USA. Association for Computational Linguistics.
22. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. [A sick cure for the evaluation of compositional distributional semantic models](https://md.research.elamur.ai/free/en/papers/marelli-2014-sick-cure-evaluation-compositional-distributional/). In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 216– 223. [↗](https://doi.org/10.63317/39qdhuevzbqa)
23. R Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2020. [Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/). In 57th Annual Meeting of the Association for Computational Linguistics, ACL 2019, pages 3428–3448. Association for Computational Linguistics (ACL).
24. Joseph P. McKenna, Samridhi Choudhary, Michael Saxon, Grant P. Strimel, and Athanasios Mouchtaris. 2020. Semantic Complexity in End-to-End Spoken Language Understanding. In Proc. Interspeech 2020, pages 4273–4277. [find](https://scholar.google.com/scholar?q=Joseph+P%2E+McKenna%2C+Samridhi+Choudhary%2C+Michael+Saxon%2C+Grant+P%2E+Strimel%2C+and+Athanasios+Mouchtaris%2E+2020%2E+Semantic+Complexity+in+End%2Dto%2DEnd+Spoken+Language+Understanding%2E+In+Proc%2E+Interspeech+2020%2C+pag)
25. Junghyun Min, R. Thomas McCoy, Dipanjan Das, Emily Pitler, and Tal Linzen. 2020. [Syntactic data augmentation increases robustness to inference heuristics](https://md.research.elamur.ai/free/en/papers/min-2020-syntactic-data-augmentation-increases-robustness/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2339–2352, Online. Association for Computational Linguistics.
26. Nafise Sadat Moosavi, Marcel de Boer, Prasetya Ajie Utama, and Iryna Gurevych. 2020. Improving robustness by augmenting training sentences with predicate-argument structures. arXiv preprint arXiv:2010.12510. [↗](https://arxiv.org/abs/2010.12510)
27. Yixin Nie, Haonan Chen, and Mohit Bansal. 2019. Combining fact extraction and verification with neural semantic matching networks. In Association for the Advancement of Artificial Intelligence (AAAI). [find](https://scholar.google.com/scholar?q=Yixin+Nie%2C+Haonan+Chen%2C+and+Mohit+Bansal%2E+2019%2E+Combining+fact+extraction+and+verification+with+neural+semantic+matching+networks%2E+In+Association+for+the+Advancement+of+Artificial+Intelligence+%28AAAI%29%2E)
28. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/nie-2020-adversarial-nli-new-benchmark-natural/). arXiv:1910.14599 [cs]. ArXiv: 1910.14599. [↗](https://arxiv.org/abs/1910.14599)
29. Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021. [KILT: a benchmark for knowledge intensive language tasks](https://md.research.elamur.ai/free/en/papers/petroni-2020-kilt-benchmark-knowledge-intensive-language/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2523–2544, Online. Association for Computational Linguistics.
30. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. [Hypothesis only baselines in natural language inference](https://md.research.elamur.ai/free/en/papers/poliak-2018-hypothesis-only-baselines-natural-language/). In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180– 191. [↗](https://doi.org/10.18653/v1/s18-2023)
31. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
32. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). arXiv preprint arXiv:1910.10683. [↗](https://arxiv.org/abs/1910.10683)
33. Michael Saxon, Samridhi Choudhary, Joseph P. McKenna, and Athanasios Mouchtaris. 2021. End-to-End Spoken Language Understanding for Generalized Voice Assistants. In Proc. Interspeech 2021, pages 4738–4742. [find](https://scholar.google.com/scholar?q=Michael+Saxon%2C+Samridhi+Choudhary%2C+Joseph+P%2E+McKenna%2C+and+Athanasios+Mouchtaris%2E+2021%2E+End%2Dto%2DEnd+Spoken+Language+Understanding+for+Generalized+Voice+Assistants%2E+In+Proc%2E+Interspeech+2021%2C+pages+4738%E2%80%93)
34. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In International conference on machine learning, pages 3319– 3328. PMLR. [↗](https://doi.org/10.48550/arxiv.1703.01365)
35. Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A Smith, and Yejin Choi. 2020. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9275–9293. [find](https://scholar.google.com/scholar?q=Swabha+Swayamdipta%2C+Roy+Schwartz%2C+Nicholas+Lourie%2C+Yizhong+Wang%2C+Hannaneh+Hajishirzi%2C+Noah+A+Smith%2C+and+Yejin+Choi%2E+2020%2E+Dataset+cartography%3A+Mapping+and+diagnosing+datasets+with+training+dynamics%2E+I)
36. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. [Fever: a large-scale dataset for fact extraction and verification](https://md.research.elamur.ai/free/en/papers/thorne-2018-fever-large-scale-dataset-fact/). In NAACL-HLT.
37. Lifu Tu, Garima Lalwani, Spandana Gella, and He He. 2020. An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models. Transactions of the Association for Computational Linguistics, 8:621–633. [↗](https://doi.org/10.1162/tacl_a_00335)
38. Neeraj Varshney, Pratyay Banerjee, Tejas Gokhale, and Chitta Baral. 2022. Unsupervised natural language inference using PHL triplet generation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2003–2016, Dublin, Ireland. Association for Computational Linguistics. [↗](https://doi.org/10.13016/m2qz56-fwst)
39. Boxin Wang, Shuohang Wang, Yu Cheng, Zhe Gan, Ruoxi Jia, Bo Li, and Jingjing Liu. 2021a. Info{bert}: Improving robustness of language models from an information theoretic perspective. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Boxin+Wang%2C+Shuohang+Wang%2C+Yu+Cheng%2C+Zhe+Gan%2C+Ruoxi+Jia%2C+Bo+Li%2C+and+Jingjing+Liu%2E+2021a%2E+Info%7Bbert%7D%3A+Improving+robustness+of+language+models+from+an+information+theoretic+perspective%2E+In+International)
40. Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, and Hao Ma. 2021b. Entailment as few-shot learner. arXiv preprint arXiv:2104.14690. [↗](https://arxiv.org/abs/2104.14690)
41. Xinyi Wang, Wenhu Chen, Michael Saxon, and William Yang Wang. 2021c. Counterfactual maximum likelihood estimation for training deep networks. arXiv preprint arXiv:2106.03831. [↗](https://arxiv.org/abs/2106.03831)
42. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. [A broad-coverage challenge corpus for sentence understanding through inference](https://md.research.elamur.ai/free/en/papers/williams-2018-broad-coverage-challenge-corpus-sentence/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n18-1101)
43. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. [Huggingface’s transformers: State-ofthe-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). arXiv preprint arXiv:1910.03771. [↗](https://arxiv.org/abs/1910.03771)
44. Yuxiang Wu, Matt Gardner, Pontus Stenetorp, and Pradeep Dasigi. 2022. [Generating data to mitigate spurious correlations in natural language inference datasets](https://md.research.elamur.ai/free/en/papers/wu-2022-generating-data-mitigate-spurious-correlations/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics.
45. Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, et al. 2020. [Clue: A chinese language understanding evaluation benchmark](https://md.research.elamur.ai/free/en/papers/xu-2020-clue-chinese-language-understanding/). In Proceedings of the 28th International Conference on Computational Linguistics, pages 4762–4772. [↗](https://doi.org/10.48550/arxiv.2004.05986)
46. Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2021. Byt5: Towards a token-free future with pre-trained byte-to-byte models. CoRR, abs/2105.13626. [↗](https://arxiv.org/abs/2105.13626)
47. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. [HotpotQA: A dataset for diverse, explainable multi-hop question answering](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380, Brussels, Belgium. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d18-1259)
48. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67–78. [↗](https://doi.org/10.1162/tacl_a_00166)
49. Guanhua Zhang, Bing Bai, Jian Liang, Kun Bai, Shiyu Chang, Mo Yu, Conghui Zhu, and Tiejun Zhao. 2019. Selection Bias Explorations and Debias Methods for Natural Language Sentence Matching Datasets. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4418– 4429, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Guanhua+Zhang%2C+Bing+Bai%2C+Jian+Liang%2C+Kun+Bai%2C+Shiyu+Chang%2C+Mo+Yu%2C+Conghui+Zhu%2C+and+Tiejun+Zhao%2E+2019%2E+Selection+Bias+Explorations+and+Debias+Methods+for+Natural+Language+Sentence+Matching+Datasets%2E+In)
50. Xiang Zhou and Mohit Bansal. 2020. [Towards robustifying NLI models against lexical dataset biases](https://md.research.elamur.ai/free/en/papers/zhou-2020-robustifying-nli-models-against-lexical/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8759– 8771, Online. Association for Computational Linguistics.

*Updated 1 November 2026*