---
title: "How Often Are Errors in Natural Language — Srikanth 2024"
description: "Benchmark test sets for natural language reasoning usually present each problem in one wording."
authors: ["Neha Srikanth", "Marine Carpuat", "Rachel Rudinger"]
year: 2024
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/2024.tacl-1.63/"
doi: "10.1162/tacl_a_00692"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/srikanth-2024-how-often-are-errors-natural/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/srikanth-2024-how-often-are-errors-natural/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2024](https://md.research.elamur.ai/free/papers/2024/page/2/) / How Often Are Errors in Natural Language Reasoning Due to Paraphrastic Variability?

# How Often Are Errors in Natural Language Reasoning Due to Paraphrastic Variability?

**Authors:** [Neha Srikanth](https://md.research.elamur.ai/free/authors/neha-srikanth/), [Marine Carpuat](https://md.research.elamur.ai/free/authors/marine-carpuat/), [Rachel Rudinger](https://md.research.elamur.ai/free/authors/rachel-rudinger/) · **Year:** 2024 · **Venue:** Transactions of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.tacl-1.63/) · [PDF](https://aclanthology.org/2024.tacl-1.63.pdf) · [DOI 10.1162/tacl\_a\_00692](https://doi.org/10.1162/tacl_a_00692) · [OpenAlex](https://openalex.org/W4402640336)

**Citations:** 4 citations (openalex-snapshot · 2026-10-14) · 6 in our library, of the 140673 papers whose bibliographies we read

## Summary
Benchmark test sets for natural language reasoning usually present each problem in one wording. A wrong answer may therefore come from misreading the phrasing or from faulty reasoning, and accuracy alone cannot tell these apart. Srikanth, Carpuat and Rudinger propose a metric, paraphrastic consistency (PC). It is the chance that a model is equally right or equally wrong on two rewordings of the same problem, and it links formally to the share of correctness variance caused by paraphrasing. To estimate PC, they built ParaNLU on top of Abductive NLI and Defeasible NLI (δ-NLI). ParaNLU covers 1,000 problems of varied difficulty, rewritten by humans and by models under a label-preserving constraint, and every paraphrase was checked manually. Across several model families, consistency rises sharply with more pretraining but not with fine-tuning, and every model leaves room for improvement. No model combines high accuracy with high consistency. Accuracy-based claims about reasoning are thus partly confounded by sensitivity to wording, so the authors recommend reporting PC next to accuracy.

## Key points
- ParaNLU contains 7,782 human-written and 7,295 model-generated paraphrases, all manually validated, over 1,000 problems drawn from Abductive NLI and δ-NLI. Data and code are public (github.com/nehasrikn/paraphrase-nlu).
- PC is computed from per-bucket accuracy θ as E[θ²] + E[(1−θ)²]. Total correctness variance splits into a within-bucket part (paraphrasing) and a between-bucket part (problem difficulty).
- Larger pretraining volume improves consistency. Adding fine-tuning data tends to lower it: models trained on ≤5% of the data look consistent only because they predict one label everywhere (about 50% accuracy).
- Human paraphrases bring more semantic and pragmatic diversity than automatic ones. Bidirectional-entailment filtering of generated paraphrases is too strict to test harder label-preserving rewrites.
- The method only needs binary correct/incorrect scoring, so it can transfer to other paraphrase-invariant tasks.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/shelves/nlp/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{srikanth2024how,
  title = {{How Often Are Errors in Natural Language Reasoning Due to Paraphrastic Variability?}},
  author = {Srikanth, Neha and Carpuat, Marine and Rudinger, Rachel},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.1162/tacl\_a\_00692},
  url = {https://doi.org/10.1162/tacl_a_00692}
}
```

- [← previous How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://md.research.elamur.ai/free/papers/chen-2024-far-gpt-4v-closing/)
- [next → Human Temporal Inferences Go Beyond Aspectual Class](https://md.research.elamur.ai/free/papers/prus-2024-human-temporal-inferences-go-beyond/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/papers/2024/page/2/)

## References

49 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Erik Arakelyan, Zhaoqi Liu, and Isabelle Augenstein. 2024. Semantic sensitivities and inconsistent predictions: Measuring the fragility of NLI models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 432–444, St. Julian’s, Malta. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Erik+Arakelyan%2C+Zhaoqi+Liu%2C+and+Isabelle+Augenstein%2E+2024%2E+Semantic+sensitivities+and+inconsistent+predictions%3A+Measuring+the+fragility+of+NLI+models%2E+In+Proceedings+of+the+18th+Conference+of+the+Euro)
2. Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The Berkeley Framenet Project. In COLING 1998 Volume 1: The 17th International Conference on Computational Linguistics. https://doi.org/10.3115/980451.980860 [↗](https://doi.org/10.3115/980451.980860) [our card](https://md.research.elamur.ai/free/papers/lex-1998-baker-berkeley-framenet/)
3. Elron Bandel, Ranit Aharonov, Michal Shmueli-Scheuer, Ilya Shnayderman, Noam Slonim, and Liat Ein-Dor. 2022. Quality controlled paraphrase generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 596–609, Dublin, Ireland. Association for Computational Linguistics. https://doi.org/10 .18653/v1/2022.acl-long.45 [↗](https://aclanthology.org/2022.acl-long.45/)
4. Rahul Bhagat and Eduard Hovy. 2013. What is a paraphrase? Computational Linguistics, 39(3):463–472. https://doi.org/10.1162/COLI a 00166 [↗](https://doi.org/10.1162/coli)
5. Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Wen-tau Yih, and Yejin Choi. 2019. Abductive commonsense reasoning. arXiv preprint arXiv:1908.05739. [↗](https://arxiv.org/abs/1908.05739) [our card](https://md.research.elamur.ai/free/papers/bhagavatula-2019-abductive-commonsense-reasoning/)
6. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics. https://doi.org/10 .18653/v1/D15-1075 [↗](https://aclanthology.org/D15-1075/)
7. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D%2E+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretche)
8. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. [↗](https://arxiv.org/abs/2210.11416)
9. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loı̈c Barrault, and Antoine Bordes. 2017. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Douwe+Kiela%2C+Holger+Schwenk%2C+Lo%C4%B1%CC%88c+Barrault%2C+and+Antoine+Bordes%2E+2017%2E)
10. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670–680, Copenhagen, Denmark. Association for Computational Linguistics. https://doi.org/10.18653/v1/D17-1070 [↗](https://aclanthology.org/D17-1070/) [our card](https://md.research.elamur.ai/free/papers/conneau-2017-supervised-learning-universal-sentence-representations/)
11. Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021. Measuring and improving consistency in pretrained language models. Transactions of the Association for Computational Linguistics, 9:1012–1031. https://doi.org/10.1162/tacl a 00410 [↗](https://doi.org/10.1162/tacl)
12. Shi Feng, Eric Wallace, and Jordan Boyd-Graber. 2019. Misleading failures of partial-input baselines. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5533–5538, Florence, Italy. Association for Computational Linguistics. https://doi.org/10.18653/v1/P19 -1554 [↗](https://doi.org/10.18653/v1/p19)
13. Joseph L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378. https://doi.org/10 .1037/h0031619 [↗](https://doi.org/10.1037/h0031619)
14. Maxwell Forbes, Jena D. Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi. 2020. Social chemistry 101: Learning to reason about social and moral norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 653–670, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.emnlp-main.48 [↗](https://doi.org/10.18653/v1/2020.emnlp-main.48)
15. Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020. Evaluating models’ local decision boundaries via contrast sets. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1307–1323, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.findings-emnlp.117 [↗](https://aclanthology.org/2020.findings-emnlp.117/) [our card](https://md.research.elamur.ai/free/papers/gardner-2020-evaluating-models-local-decision-boundaries/)
16. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 1–9, Prague. Association for Computational Linguistics. https://doi.org/10 .3115/1654536.1654538 [↗](https://doi.org/10)
17. Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking nli systems with sentences that require simple lexical inferences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 650–655. https://doi.org/10 .18653/v1/P18-2103 [↗](https://aclanthology.org/P18-2103/) [our card](https://md.research.elamur.ai/free/papers/glockner-2018-breaking-nli-systems-sentences-that/)
18. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics. https://doi.org/10.18653/v1/N18-2017 [↗](https://doi.org/10.18653/v1/n18-2017)
19. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2022. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. In The Eleventh International Conference on Learning Representations. [↗](https://doi.org/10.48550/arxiv.2111.09543) [our card](https://md.research.elamur.ai/free/papers/he-2021-debertav3-improving-deberta-using-electra/)
20. Christine Herlihy and Rachel Rudinger. 2021. MedNLI is not immune: Natural language inference artifacts in the clinical domain. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 1020–1027, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.acl-short.129 [↗](https://aclanthology.org/2021.acl-short.129/)
21. J. Edward Hu, Huda Khayrallah, Ryan Culkin, Patrick Xia, Tongfei Chen, Matt Post, and Benjamin Van Durme. 2019. Improved lexically constrained decoding for translation and monolingual rewriting. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 839–850, Minneapolis, Minnesota. Association for Computational Linguistics. https://doi.org/10.18653/v1/N19 -1090 [↗](https://doi.org/10.18653/v1/n19)
22. Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1875–1885, New Orleans, Louisiana. Association for Computational Linguistics. https://doi.org/10.18653/v1/N18-1170 [↗](https://doi.org/10.18653/v1/n18-1170) [our card](https://md.research.elamur.ai/free/papers/iyyer-2018-adversarial-example-generation-syntactically-controlled/)
23. Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know when language models know? On the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 9:962–977. https://doi.org/10.1162/tacl a 00407 [↗](https://doi.org/10.1162/tacl)
24. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427–431, Valencia, Spain. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Armand+Joulin%2C+Edouard+Grave%2C+Piotr+Bojanowski%2C+and+Tomas+Mikolov%2E+2017%2E+Bag+of+tricks+for+efficient+text+classification%2E+In+Proceedings+of+the+15th+Conference+of+the+European+Chapter+of+the+Associati)
25. Divyansh Kaushik, Eduard Hovy, and Zachary C. Lipton. 2020. Learning the difference that makes a difference with counterfactually-augmented data. [↗](https://doi.org/10.48550/arxiv.1909.12434)
26. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy. Association for Computational Linguistics. https://doi.org/10 .18653/v1/P19-1334 [↗](https://aclanthology.org/P19-1334/) [our card](https://md.research.elamur.ai/free/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/)
27. Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340–2353, Santa Fe, New Mexico, USA. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1806.00692)
28. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885–4901, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.441 [↗](https://aclanthology.org/2020.acl-main.441/)
29. Joakim Nivre, Marie-Catherine De Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal dependencies v1: A multilingual treebank collection. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 1659–1666. [find](https://scholar.google.com/scholar?q=Joakim+Nivre%2C+Marie%2DCatherine+De+Marneffe%2C+Filip+Ginter%2C+Yoav+Goldberg%2C+Jan+Hajic%2C+Christopher+D%2E+Manning%2C+Ryan+McDonald%2C+Slav+Petrov%2C+Sampo+Pyysalo%2C+Natalia+Silveira%2C+Reut+Tsarfaty%2C+and+Daniel+Zeman%2E) [our card](https://md.research.elamur.ai/free/papers/ud-2016-nivre-universal-dependencies-v1-multilingual-treebank/)
30. Charles Sanders Peirce. 1974. Collected Papers of Charles Sanders Peirce, volume 5. Harvard University Press. [find](https://scholar.google.com/scholar?q=Charles+Sanders+Peirce%2E+1974%2E+Collected+Papers+of+Charles+Sanders+Peirce%2C+volume+5%2E+Harvard+University+Press%2E)
31. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics. https://doi.org/10.3115/v1/D14-1162 [↗](https://doi.org/10.3115/v1/d14-1162)
32. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180–191, New Orleans, Louisiana. Association for Computational Linguistics. https://doi.org/10.18653/v1/S18-2023 [↗](https://doi.org/10.18653/v1/s18-2023)
33. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(1):5485–5551. [↗](https://doi.org/10.48550/arxiv.1910.10683)
34. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. https://doi.org/10 .18653/v1/D16-1264 [↗](https://aclanthology.org/D16-1264/) [our card](https://md.research.elamur.ai/free/papers/rajpurkar-2016-squad-100-000-questions-machine/)
35. Raymond Reiter. 1980. A logic for default reasoning. Artificial Intelligence, 13(1-2):81–132. https://doi.org/10.1016/0004 -3702(80)90014-4 [↗](https://doi.org/10.1016/0004)
36. Rachel Rudinger, Vered Shwartz, Jena D. Hwang, Chandra Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A. Smith, and Yejin Choi. 2020. Thinking like a skeptic: Defeasible inference in natural language. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4661–4675, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.findings -emnlp.418 [↗](https://doi.org/10.18653/v1/2020.findings)
37. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106. https://doi.org/10 .1145/3474381 [↗](https://doi.org/10)
38. Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. 2019. Atomic: An atlas of machine commonsense for if-then reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3027–3035. https://doi.org/10 .1609/aaai.v33i01.33013027 [↗](https://doi.org/10.1609/aaai.v33i01.33013027)
39. Lenhart Schubert. 2015. Semantic representation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29. https://doi.org/10.1609/aaai.v29i1.9759 [↗](https://doi.org/10.1609/aaai.v29i1.9759)
40. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.704 [↗](https://aclanthology.org/2020.acl-main.704/)
41. Neha Srikanth and Rachel Rudinger. 2022. Partial-input baselines show that NLI models can ignore context, but they don’t. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4753–4763, Seattle, United States. Association for Computational Linguistics. https://doi.org/10 .18653/v1/2022.naacl-main.350 [↗](https://aclanthology.org/2022.naacl-main.350/)
42. Alessandro Stolfo, Zhijing Jin, Kumar Shridhar, Bernhard Schoelkopf, and Mrinmaya Sachan. 2023. A causal framework to quantify the robustness of mathematical reasoning with language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 545–561, Toronto, Canada. Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.acl-long.32 [↗](https://doi.org/10.18653/v1/2023.acl-long.32)
43. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: A large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809–819, New Orleans, Louisiana. Association for Computational Linguistics. https://doi.org/10.18653/v1/N18-1074 [↗](https://aclanthology.org/N18-1074/) [our card](https://md.research.elamur.ai/free/papers/thorne-2018-fever-large-scale-dataset-fact/)
44. Victor Veitch, Alexander D’Amour, Steve Yadlowsky, and Jacob Eisenstein. 2021. Counterfactual invariance to spurious correlations in text classification. Advances in Neural Information Processing Systems, 34:16196–16208. [find](https://scholar.google.com/scholar?q=Victor+Veitch%2C+Alexander+D%E2%80%99Amour%2C+Steve+Yadlowsky%2C+and+Jacob+Eisenstein%2E+2021%2E+Counterfactual+invariance+to+spurious+correlations+in+text+classification%2E+Advances+in+Neural+Information+Processing+Syst)
45. Dhruv Verma, Yash Kumar Lal, Shreyashee Sinha, Benjamin Van Durme, and Adam Poliak. 2023. Evaluating paraphrastic robustness in textual entailment models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 880–892, Toronto, Canada. Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.acl-short.76 [↗](https://aclanthology.org/2023.acl-short.76/)
46. Alex Warstadt, Yian Zhang, Xiaocheng Li, Haokun Liu, and Samuel R. Bowman. 2020. Learning which features matter: RoBERTa acquires a preference for linguistic generalizations (eventually). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 217–235, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.emnlp-main.16 [↗](https://doi.org/10.18653/v1/2020.emnlp-main.16)
47. N. A. Weiss, P. T. Holmes, and M. Hardy. 2006. A Course in Probability. Pearson Addison Wesley. [find](https://scholar.google.com/scholar?q=N%2E+A%2E+Weiss%2C+P%2E+T%2E+Holmes%2C+and+M%2E+Hardy%2E+2006%2E+A+Course+in+Probability%2E+Pearson+Addison+Wesley%2E)
48. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics. https://doi.org/10 .18653/v1/N18-1101 [↗](https://aclanthology.org/N18-1101/)
49. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93–104, Brussels, Belgium. Association for Computational Linguistics. https://doi.org/10.18653/v1/D18-1009 [↗](https://aclanthology.org/D18-1009/)