---
title: "ANLIzing the Adversarial Natural Language Inference Dataset"
description: "The authors therefore design a hierarchical, genre-independent scheme of 40… 70% accuracy on ANLI. 85 references, cited by 34 papers here."
authors: ["Adina Williams", "Tristan Thrush", "Douwe Kiela"]
year: 2022
venue: "Proceedings of the Society for Computation in Linguistics 2022"
source: "https://aclanthology.org/2022.scil-1.3/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/williams-2022-anlizing-adversarial-natural-language-inference/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/williams-2022-anlizing-adversarial-natural-language-inference/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) / ANLIzing the Adversarial Natural Language Inference Dataset

# ANLIzing the Adversarial Natural Language Inference Dataset

**Authors:** [Adina Williams](https://md.research.elamur.ai/free/en/authors/adina-williams/), [Tristan Thrush](https://md.research.elamur.ai/free/en/authors/tristan-thrush/) and [Douwe Kiela](https://md.research.elamur.ai/free/en/authors/douwe-kiela/) · **Year:** 2022 · **Venue:** [Proceedings of the Society for Computation in Linguistics 2022](https://md.research.elamur.ai/free/en/venues/scil/) ([SCiL 2022](https://md.research.elamur.ai/free/en/venues/scil/2022/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.scil-1.3/) · [PDF](https://aclanthology.org/2022.scil-1.3.pdf) · [OpenAlex](https://openalex.org/W3093582159)

**Citations:** 14 citations (OpenAlex, 2026-10-14) · [34](https://md.research.elamur.ai/free/en/papers/williams-2022-anlizing-adversarial-natural-language-inference/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The authors study why transformer models still struggle with ANLI, an adversarial NLI benchmark built over three rounds in which crowdworkers on Dynabench wrote hypotheses meant to fool a target model. Accuracy numbers alone do not show which reasoning skills fail. The authors therefore design a hierarchical, genre-independent scheme of 40 inference-type tags, up to four levels deep, under five top-level classes. Three NLP researchers derived the scheme inductively from round-1 data. Experts then used it to label all 3,200 development pairs (A1–A3), and the labels are released publicly. Eight finetuned models, including the original round adversaries, are scored per tag and per genre.

ALBERT-XXLarge performs best overall, but much headroom remains. Numerical and commonsense-reasoning inferences are the most frequent. Facts, coreference, negation and syntax stay hardest even though they are common. Different models make highly similar predictions, which suggests that incremental tweaks to pretrain-finetune are unlikely to close the gap. The annotations allow future NLI models to be diagnosed by phenomenon rather than by a single score.

## Key points
- The scheme has 40 tags, nested up to four levels. Its top level is fixed by the original ANLI paper: NUMERICAL, BASIC, REFERENCE, TRICKY and REASONING. An extra imperfection branch marks gold-label errors, ambiguity and spelling issues.
- The released resource consists of expert hand-annotations of the full ANLI dev sets (3,200 pairs), hosted on the facebookresearch/anli GitHub repository.
- Numerical and commonsense-reasoning inferences appear in roughly 40–60% of examples. Their share shifts across rounds as later adversaries learn earlier tricks; for example, there are fewer NUMERICAL items after A1.
- ALBERT-XXLarge correlates best with the gold labels (about 0.5), while the round-1 BERT adversary is slightly negatively correlated. RoBERTa, BART, XLNet and ALBERT agree with each other above 0.6.
- In A3, Wikipedia is the easiest genre, followed by Procedural, News, Fiction, Legal and RTE. Neither GPT-3 nor DeBERTa reaches 70% accuracy on ANLI.

## Interesting ideas
- A layered taxonomy of phenomena lets the same benchmark be scored coarsely or finely, so a weak overall score can be traced to one specific skill.
- The authors keep a separate tag for items whose gold label looks wrong or ambiguous, so dataset noise is not counted as model failure.
- Measuring how strongly different models' outputs correlate shows whether they share blind spots, which matters more than their individual accuracies.
- Watching which phenomena fade across adversarial rounds shows what each successive model has actually learned.
- A genre-by-phenomenon breakdown reveals that some text sources train certain skills better, which supports targeted upsampling.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (answer): Tag school-test and fable-QA items by the reasoning they require (counting, negation, coreference, kinship), and report accuracy per tag so a failing reader points to the responsible module.
- **I2** (learn): When building silver data and running evaluations, flag disputed or ambiguous items explicitly and exclude them from error statistics, much like a negative control.
- **I3** (learn): Before trusting agreement among witness parsers, check how correlated their errors are; agreement between near-identical witnesses is weak evidence for silver labels.
- **I4** (learn): In self-play or repair-rule loops, log the phenomenon tags of the examples that still break the system each round, so you can see which weaknesses are being fixed and which persist.
- **I5** (tree): Break parser errors down by treebank genre and construction type, then upsample the genres that best cover the weak constructions.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 310 works · 284,480 citations
- [Meta](https://md.research.elamur.ai/free/en/companies/meta/) — Company, Menlo Park, United States · 158 works · 211,459 citations
- [Meta (United Kingdom)](https://md.research.elamur.ai/free/en/companies/meta-united-kingdom/) — Company, London, United Kingdom · 39 works · 11,541 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works

## Cite

Cite as: Adina Williams, Tristan Thrush and Douwe Kiela. 2022. ANLIzing the Adversarial Natural Language Inference Dataset. Proceedings of the Society for Computation in Linguistics 2022.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{williams2022anlizing,
  title = {{ANLIzing the Adversarial Natural Language Inference Dataset}},
  author = {Adina Williams, Tristan Thrush and Douwe Kiela},
  booktitle = {Proceedings of the Society for Computation in Linguistics 2022},
  year = {2022},
  url = {https://aclanthology.org/2022.scil-1.3/}
}
```

- [← previous ID10M: Idiom Identification in 10 Languages](https://md.research.elamur.ai/free/en/papers/tedeschi-2022-id10m-idiom-identification-10-languages/)
- [next → CorefUD 1.0: Coreference Meets Universal Dependencies](https://md.research.elamur.ai/free/en/papers/nedoluzhko-2022-corefud-1-0-coreference-meets/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/)

## References

85 references, 39 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam Mc-Candlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). [↗](https://doi.org/10.48550/arxiv.2005.14165)
2. Afra Alishahi, Grzegorz Chrupała, and Tal Linzen. 2019. Analyzing and interpreting neural networks for NLP: A report on the first BlackboxNLP workshop. Natural Language Engineering, 25(4):543– 557. [↗](https://doi.org/10.1017/s135132491900024x)
3. Valerio Basile, Johan Bos, Kilian Evang, and Noortje Venhuizen. 2012. [Developing a large semantically annotated corpus](https://md.research.elamur.ai/free/en/papers/basile-2012-developing-large-semantically-annotated-corpus/). In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012), pages 3196–3200, Istanbul, Turkey. European Languages Resources Association (ELRA).
4. Tongfei Chen, Zhengping Jiang, Adam Poliak, Keisuke Sakaguchi, and Benjamin Van Durme. 2020. [Uncertain natural language inference](https://md.research.elamur.ai/free/en/papers/chen-2020-uncertain-natural-language-inference/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8772–8779, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.acl-main.774)
5. Isaac I Bejar, Roger Chaffin, and Susan Embretson. 2012. Cognitive and psychometric analysis of analogical problem solving. Springer Science & Business Media. [find](https://scholar.google.com/scholar?q=Isaac+I+Bejar%2C+Roger+Chaffin%2C+and+Susan+Embretson%2E+2012%2E+Cognitive+and+psychometric+analysis+of+analogical+problem+solving%2E+Springer+Science+%26+Business+Media%2E)
6. Jean-Philippe Bernardy and Stergios Chatzikyriakidis. 2019. What kind of natural language inference are nlp systems learning: Is this enough? In International Conference on Agents and Artificial Intelligence (ICAART2), pages 919–931. [find](https://scholar.google.com/scholar?q=Jean%2DPhilippe+Bernardy+and+Stergios+Chatzikyriakidis%2E+2019%2E+What+kind+of+natural+language+inference+are+nlp+systems+learning%3A+Is+this+enough%3F+In+International+Conference+on+Agents+and+Artificial+Intel)
7. Gennaro Chierchia et al. 2004. Scalar implicatures, polarity phenomena, and the syntax/pragmatics interface. Structures and beyond, 3:39–103. [↗](https://doi.org/10.1093/oso/9780195171976.003.0003)
8. Noam Chomsky. 1980. On cognitive structures and their development: A reply to Piaget. Harvard University Press. [find](https://scholar.google.com/scholar?q=Noam+Chomsky%2E+1980%2E+On+cognitive+structures+and+their+development%3A+A+reply+to+Piaget%2E+Harvard+University+Press%2E)
9. Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. 2020. [Abductive commonsense reasoning](https://md.research.elamur.ai/free/en/papers/bhagavatula-2019-abductive-commonsense-reasoning/). In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. [↗](https://doi.org/10.48550/arxiv.1908.05739)
10. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: pre-training text encoders as discriminators rather than generators. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Minh%2DThang+Luong%2C+Quoc+V%2E+Le%2C+and+Christopher+D%2E+Manning%2E+2020%2E+ELECTRA%3A+pre%2Dtraining+text+encoders+as+discriminators+rather+than+generators%2E+In+8th+International+Conference+on+Learning+R)
11. Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. 2021. Stereotyping Norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1004–1015, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2021.acl-long.81)
12. Peter Clark. 2018. [What knowledge is needed to solve the RTE5 textual entailment challenge?](https://md.research.elamur.ai/free/en/papers/clark-2018-knowledge-needed-solve-rte5-textual/) [↗](https://doi.org/10.48550/arxiv.1806.03561)
13. Robin Cooper, Crouch Dick, Jan van Eijck, Chris Fox, Joseph van Genabith, Han Jaspars, Hans Kamp, David Milward, Manfred Pinkal, Massimo Poesio, Steve Pulman, Ted Brisco, Holger Maier, and Karsten Konrad. 1996. Using the framework. technical report lre 62-051r. The FraCaS consortium. [find](https://scholar.google.com/scholar?q=Robin+Cooper%2C+Crouch+Dick%2C+Jan+van+Eijck%2C+Chris+Fox%2C+Joseph+van+Genabith%2C+Han+Jaspars%2C+Hans+Kamp%2C+David+Milward%2C+Manfred+Pinkal%2C+Massimo+Poesio%2C+Steve+Pulman%2C+Ted+Brisco%2C+Holger+Maier%2C+and+Karsten+Kon)
14. Johan Bos, Valerio Basile, Kilian Evang, Noortje J Venhuizen, and Johannes Bjerva. 2017. The Groningen Meaning Bank. In Handbook of linguistic annotation, pages 463–496. Springer. [↗](https://doi.org/10.1007/978-94-024-0881-2_18)
15. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 1161–1166. Association for Computational Linguistics. challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment, pages 177– 190. Springer. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+2006%2E+The+PASCAL+recognising+textual+entailment+Mor+Geva%2C+Yoav+Goldberg%2C+and+Jonathan+Berant%2E+2019%2E+Are+we+modeling+the+task+or+the+annotator%3F+an+invest)
16. DataPerf Working Group. 2021. DataPerf: Benchmarking data for better ML. Technical report. [find](https://scholar.google.com/scholar?q=DataPerf+Working+Group%2E+2021%2E+DataPerf%3A+Benchmarking+data+for+better+ML%2E+Technical+report%2E)
17. Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018. Transforming question answering datasets into natural language inference datasets. arXiv preprint arXiv:1809.02922. [↗](https://arxiv.org/abs/1809.02922)
18. Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. [Breaking NLI systems with sentences that require simple lexical inferences](https://md.research.elamur.ai/free/en/papers/glockner-2018-breaking-nli-systems-sentences-that/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 650–655, Melbourne, Australia. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/p18-2103)
19. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1423)
20. H Paul Grice. 1975. Logic and conversation. 1975, pages 41–58. [↗](https://doi.org/10.1163/9789004368811_003)
21. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Swabha+Swayamdipta%2C+Omer+Levy%2C+Roy+Schwartz%2C+Samuel+Bowman%2C+and+Noah+A%2E+Smith%2E+2018%2E+Annotation+artifacts+in+natural+language+inference+data%2E+In+Proceedings+of+the+2018+Conference+o)
22. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. [DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368–2378, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1246)
23. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. [DeBERTa: decoding-enhanced BERT with disentangled attention](https://md.research.elamur.ai/free/en/papers/he-2020-deberta-decoding-enhanced-bert-disentangled/). In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. [↗](https://doi.org/10.48550/arxiv.2006.03654)
24. Yaroslav Fyodorov, Yoad Winter, and Nissim Francez. 2000. A natural logic inference system. In Proceedings of the 2nd Workshop on Inference in Computational Semantics (ICoS-2). Citeseer. [find](https://scholar.google.com/scholar?q=Yaroslav+Fyodorov%2C+Yoad+Winter%2C+and+Nissim+Francez%2E+2000%2E+A+natural+logic+inference+system%2E+In+Proceedings+of+the+2nd+Workshop+on+Inference+in+Computational+Semantics+%28ICoS%2D2%29%2E+Citeseer%2E)
25. Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020. [Evaluating models’ local decision boundaries via contrast sets](https://md.research.elamur.ai/free/en/papers/gardner-2020-evaluating-models-local-decision-boundaries/). In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1307–1323, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.findings-emnlp.117)
26. Md Mosharaf Hossain, Venelin Kovatchev, Pranoy Dutta, Tiffany Kao, Elizabeth Wei, and Eduardo Blanco. 2020. [An analysis of natural language inference benchmarks through the lens of negation](https://md.research.elamur.ai/free/en/papers/hossain-2020-analysis-natural-language-inference-benchmarks/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9106–9118, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.emnlp-main.732)
27. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. [Are natural language inference models IMPPRESsive? Learning IMPlicature and PRESupposition](https://md.research.elamur.ai/free/en/papers/jeretic-2020-are-natural-language-inference-models/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8690–8705, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.acl-main.768)
28. Atticus Geiger, Ignacio Cases, Lauri Karttunen, and Christopher Potts. 2018. [Stress-testing neural models of natural language inference with multiply-quantified sentences](https://md.research.elamur.ai/free/en/papers/geiger-2018-stress-testing-neural-models-natural/). arXiv preprint arXiv:1810.13033. [↗](https://arxiv.org/abs/1810.13033)
29. Robin Jia and Percy Liang. 2017. [Adversarial examples for evaluating reading comprehension systems](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021–2031, Copenhagen, Denmark. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d17-1215)
30. Atticus Geiger, Kyle Richardson, and Christopher Potts. 2020. [Neural natural language inference models partially embed theories of lexical entailment and negation](https://md.research.elamur.ai/free/en/papers/geiger-2020-neural-natural-language-inference-models/). In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 163–173, Online. Association for Computational Linguistics.
31. Pratik Joshi, Somak Aditya, Aalok Sathe, and Monojit Choudhury. 2020. TaxiNLI: Taking a ride up the Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. NLU hill. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 41–55, Online. Association for Computational Linguistics. [↗](https://arxiv.org/abs/1907.11692)
32. Peter LoBue and Alexander Yates. 2011. [Types of common-sense knowledge needed for recognizing textual entailment](https://md.research.elamur.ai/free/en/papers/lobue-2011-types-common-sense-knowledge-needed/). In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 329–334, Portland, Oregon, USA. Association for Computational Linguistics.
33. Alain Joubert, Mathieu Lafourcade, and Nathalie Le Brun. 2018. The jeuxdemots project is 10 years old: What we have learned. Games and Gamification for Natural Language Processing, 22. [find](https://scholar.google.com/scholar?q=Alain+Joubert%2C+Mathieu+Lafourcade%2C+and+Nathalie+Le+Brun%2E+2018%2E+The+jeuxdemots+project+is+10+years+old%3A+What+we+have+learned%2E+Games+and+Gamification+for+Natural+Language+Processing%2C+22%2E)
34. David Jurgens, Saif Mohammad, Peter Turney, and Keith Holyoak. 2012. [SemEval-2012 task 2: Measuring degrees of relational similarity](https://md.research.elamur.ai/free/en/papers/jurgens-2012-semeval-2012-task-2-measuring/). In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 356–364, Montréal, Canada. Association for Computational Linguistics.
35. Zhiyi Ma, Kawin Ethayarajh, Tristan Thrush, Somya Jain, Ledell Wu, Robin Jia, Christopher Potts, Adina Williams, and Douwe Kiela. 2021. Dynaboard: An evaluation-as-a-service platform for holistic next-generation benchmarking. Advances in Neural Information Processing Systems. [↗](https://doi.org/10.48550/arxiv.2106.06052)
36. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. [Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/p19-1334)
37. Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021. Dynabench: Rethinking benchmarking in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4110–4124, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Douwe+Kiela%2C+Max+Bartolo%2C+Yixin+Nie%2C+Divyansh+Kaushik%2C+Atticus+Geiger%2C+Zhengxuan+Wu%2C+Bertie+Vidgen%2C+Grusha+Prasad%2C+Amanpreet+Singh%2C+Pratik+Ringshia%2C+Zhiyi+Ma%2C+Tristan+Thrush%2C+Sebastian+Riedel%2C+Zeerak+)
38. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering ambiguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Julian+Michael%2C+Hannaneh+Hajishirzi%2C+and+Luke+Zettlemoyer%2E+2020%2E+AmbigQA%3A+Answering+ambiguous+open%2Ddomain+questions%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+La)
39. Richard Montague. 1970. Universal grammar. Theoria, 36(3):373–398. [find](https://scholar.google.com/scholar?q=Richard+Montague%2E+1970%2E+Universal+grammar%2E+Theoria%2C+36%283%29%3A373%E2%80%93398%2E)
40. Najoung Kim and Tal Linzen. 2020. COGS: A compositional generalization challenge based on semantic interpretation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9087–9105, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Najoung+Kim+and+Tal+Linzen%2E+2020%2E+COGS%3A+A+compositional+generalization+challenge+based+on+semantic+interpretation%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Proces)
41. Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340–2353, Santa Fe, New Mexico, USA. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1806.00692)
42. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. [find](https://scholar.google.com/scholar?q=Zhenzhong+Lan%2C+Mingda+Chen%2C+Sebastian+Goodman%2C+Kevin+Gimpel%2C+Piyush+Sharma%2C+and+Radu+Soricut%2E+2020%2E+ALBERT%3A+A+lite+BERT+for+self%2Dsupervised+learning+of+language+representations%2E+In+8th+International+C)
43. Nikita Nangia and Samuel R. Bowman. 2019. Human vs. muppet: A conservative estimate of human performance on the GLUE benchmark. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4566–4575, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nikita+Nangia+and+Samuel+R%2E+Bowman%2E+2019%2E+Human+vs%2E+muppet%3A+A+conservative+estimate+of+human+performance+on+the+GLUE+benchmark%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computa)
44. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019. [BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). CoRR, abs/1910.13461. [↗](https://arxiv.org/abs/1910.13461)
45. Nikita Nangia, Saku Sugawara, Harsh Trivedi, Alex Warstadt, Clara Vania, and Samuel R. Bowman. 2021. What ingredients make for an effective crowdsourcing protocol for difficult NLU data collection tasks? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1221–1235, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nikita+Nangia%2C+Saku+Sugawara%2C+Harsh+Trivedi%2C+Alex+Warstadt%2C+Clara+Vania%2C+and+Samuel+R%2E+Bowman%2E+2021%2E+What+ingredients+make+for+an+effective+crowdsourcing+protocol+for+difficult+NLU+data+collection+tas)
46. Tal Linzen, Grzegorz Chrupała, Yonatan Belinkov, Dieuwke Hupkes, and et al., editors. 2019. Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. Association for Computational Linguistics, Florence, Italy. [find](https://scholar.google.com/scholar?q=Tal+Linzen%2C+Grzegorz+Chrupa%C5%82a%2C+Yonatan+Belinkov%2C+Dieuwke+Hupkes%2C+and+et+al%2E%2C+editors%2E+2019%2E+Proceedings+of+the+2019+ACL+Workshop+BlackboxNLP%3A+Analyzing+and+Interpreting+Neural+Networks+for+NLP%2E+Associ)
47. Yixin Nie, Yicheng Wang, and Mohit Bansal. 2019. [Analyzing compositionality-sensitivity of NLI models](https://md.research.elamur.ai/free/en/papers/nie-2018-analyzing-compositionality-sensitivity-nli-models/). Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):6867–6874. [↗](https://doi.org/10.1609/aaai.v33i01.33016867)
48. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Pedro Rodriguez, Joe Barrow, Alexander Miserlis Hoyle, John P. Lalor, Robin Jia, and Jordan Boyd-Graber. 2021. Evaluation examples are not equally informative: How should that change NLP leaderboards? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4486–4503, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Pedro+Rodriguez%2C+Joe+Barrow%2C+Alexander+Miserlis+Hoyle%2C+John+P%2E+Lalor%2C+Robin+Jia%2C+and+Jordan+Boyd%2DGraber%2E)
49. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020a. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885–4901, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yixin+Nie%2C+Adina+Williams%2C+Emily+Dinan%2C+Mohit+Bansal%2C+Jason+Weston%2C+and+Douwe+Kiela%2E+2020a%2E+Adversarial+NLI%3A+A+new+benchmark+for+natural+language+understanding%2E+In+Proceedings+of+the+58th+Annual+Meeti)
50. Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020b. [What can we learn from collective human opinions on natural language inference data?](https://md.research.elamur.ai/free/en/papers/nie-2020-what-can-we-learn-collective/) In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9131–9143, Online. Association for Computational Linguistics.
51. Ohad Rozen, Vered Shwartz, Roee Aharoni, and Ido Dagan. 2019. Diversify your datasets: Analyzing generalization via controlled variance in adversarial datasets. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 196–205, Hong Kong, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ohad+Rozen%2C+Vered+Shwartz%2C+Roee+Aharoni%2C+and+Ido+Dagan%2E+2019%2E+Diversify+your+datasets%3A+Analyzing+generalization+via+controlled+variance+in+adversarial+datasets%2E+In+Proceedings+of+the+23rd+Conference+o)
52. Alicia Parrish, William Huang, Omar Agha, Soo-Hwan Lee, Nikita Nangia, Alex Warstadt, Karmanya Aggarwal, Emily Allaway, Tal Linzen, and Samuel R Bowman. 2021. Does putting a linguist in the loop improve nlu data collection? arXiv preprint arXiv:2104.07179. [↗](https://arxiv.org/abs/2104.07179)
53. Rachel Rudinger, Vered Shwartz, Jena D. Hwang, Chandra Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A. Smith, and Yejin Choi. 2020. Thinking like a skeptic: Defeasible inference in natural language. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4661–4675, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rachel+Rudinger%2C+Vered+Shwartz%2C+Jena+D%2E+Hwang%2C+Chandra+Bhagavatula%2C+Maxwell+Forbes%2C+Ronan+Le+Bras%2C+Noah+A%2E+Smith%2C+and+Yejin+Choi%2E+2020%2E+Thinking+like+a+skeptic%3A+Defeasible+inference+in+natural+languag)
54. Ellie Pavlick and Tom Kwiatkowski. 2019. [Inherent disagreements in human textual inferences](https://md.research.elamur.ai/free/en/papers/pavlick-2019-inherent-disagreements-human-textual-inferences/). Trans. Assoc. Comput. Linguistics, 7:677–694. [↗](https://doi.org/10.1162/tacl_a_00293)
55. Adam Poliak. 2020. [A survey on recognizing textual entailment as an NLP evaluation](https://md.research.elamur.ai/free/en/papers/poliak-2020-survey-recognizing-textual-entailment-as/). In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, pages 92–109, Online. Association for Computational Linguistics.
56. Swarnadeep Saha, Yixin Nie, and Mohit Bansal. 2020. [ConjNLI: Natural language inference over conjunctive sentences](https://md.research.elamur.ai/free/en/papers/saha-2020-conjnli-natural-language-inference-over/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8240–8252, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.emnlp-main.661)
57. Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. 2018a. [Collecting diverse natural language inference problems for sentence representation evaluation](https://md.research.elamur.ai/free/en/papers/poliak-2018-collecting-diverse-natural-language-inference/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 67–81, Brussels, Belgium. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1804.08207)
58. Mark Sammons, V.G.Vinod Vydiswaran, and Dan Roth. 2010. [“ask not what textual entailment can do for you...”](https://md.research.elamur.ai/free/en/papers/sammons-2010-ask-not-what-textual-entailment/). In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1199–1208, Uppsala, Sweden. Association for Computational Linguistics.
59. Maarten Sap, Vered Shwartz, Antoine Bosselut, Yejin Choi, and Dan Roth. 2020. Commonsense reasoning for natural language processing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pages 27–33, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Maarten+Sap%2C+Vered+Shwartz%2C+Antoine+Bosselut%2C+Yejin+Choi%2C+and+Dan+Roth%2E+2020%2E+Commonsense+reasoning+for+natural+language+processing%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Co)
60. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018b. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180–191, New Orleans, Louisiana. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Adam+Poliak%2C+Jason+Naradowsky%2C+Aparajita+Haldar%2C+Rachel+Rudinger%2C+and+Benjamin+Van+Durme%2E+2018b%2E+Hypothesis+only+baselines+in+natural+language+inference%2E+In+Proceedings+of+the+Seventh+Joint+Conference)
61. Karin Kipper Schuler. 2005. VerbNet: A broad-coverage, comprehensive verb lexicon. Ph.D. thesis, University of Colorado. [↗](https://doi.org/10.5281/zenodo.8100881)
62. Abhilasha Ravichander, Aakanksha Naik, Carolyn Rose, and Eduard Hovy. 2019. [EQUATE: A benchmark evaluation framework for quantitative reasoning in natural language inference](https://md.research.elamur.ai/free/en/papers/ravichander-2019-equate-benchmark-evaluation-framework-quantitative/). In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 349–361, Hong Kong, China. Association for Computational Linguistics.
63. Sebastian Schuster, Yuxing Chen, and Judith Degen. 2020. [Harnessing the linguistic signal to predict scalar inferences](https://md.research.elamur.ai/free/en/papers/schuster-2020-harnessing-linguistic-signal-predict-scalar/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5387–5403, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.acl-main.479)
64. Kyle Richardson, Hai Hu, Lawrence S Moss, and Ashish Sabharwal. 2019. [Probing natural language inference models through semantic fragments](https://md.research.elamur.ai/free/en/papers/richardson-2019-probing-natural-language-inference-models/). arXiv preprint arXiv:1909.07521. [↗](https://arxiv.org/abs/1909.07521)
65. Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. 2021a. Masked language modeling and the distributional Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium. Association for Computational Linguistics. hypothesis: Order word matters pre-training for little. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2888–2913, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
66. Koustuv Sinha, Prasanna Parthasarathi, Joelle Pineau, and Adina Williams. 2021b. [UnNatural Language Inference](https://md.research.elamur.ai/free/en/papers/sinha-2021-unnatural-language-inference/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7329–7346, Online. Association for Computational Linguistics.
67. Alex Warstadt, Yu Cao, Ioana Grosu, Wei Peng, Hagen Blix, Yining Nie, Anna Alsop, Shikha Bordia, Haokun Liu, Alicia Parrish, Sheng-Fu Wang, Jason Phang, Anhad Mohananey, Phu Mon Htut, Paloma Jeretic, and Samuel R. Bowman. 2019a. [Investigating BERT’s knowledge of language: Five analysis methods with NPIs](https://md.research.elamur.ai/free/en/papers/warstadt-2019-investigating-bert-s-knowledge-language/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2877–2887, Hong Kong, China. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1909.02597)
68. Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9275–9293, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Swabha+Swayamdipta%2C+Roy+Schwartz%2C+Nicholas+Lourie%2C+Yizhong+Wang%2C+Hannaneh+Hajishirzi%2C+Noah+A%2E+Smith%2C+and+Yejin+Choi%2E+2020%2E+Dataset+cartography%3A+Mapping+and+diagnosing+datasets+with+training+dynamics%2E+)
69. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. [BLiMP: The benchmark of linguistic minimal pairs for English](https://md.research.elamur.ai/free/en/papers/warstadt-2020-blimp-benchmark-linguistic-minimal-pairs/). Transactions of the Association for Computational Linguistics, 8:377– 392. [↗](https://doi.org/10.1162/tacl_a_00321)
70. David R Thomas. 2006. A general inductive approach for analyzing qualitative evaluation data. American journal of evaluation, 27(2):237–246. [↗](https://doi.org/10.1177/1098214005283748)
71. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019b. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625–641. [↗](https://doi.org/10.1162/tacl_a_00290)
72. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. [FEVER: a large-scale dataset for fact extraction and VERification](https://md.research.elamur.ai/free/en/papers/thorne-2018-fever-large-scale-dataset-fact/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809–819, New Orleans, Louisiana. Association for Computational Linguistics.
73. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. 2016. [Towards ai-complete question answering: A set of prerequisite toy tasks](https://md.research.elamur.ai/free/en/papers/weston-2015-towards-ai-complete-question-answering/). International Conference on Learning Representations.
74. Assaf Toledo, Sophia Katrenko, Stavroula Alexandropoulou, Heidi Klockmann, Asher Stern, Ido Dagan, and Yoad Winter. 2012. Semantic annotation for textual entailment recognition. In Mexican International Conference on Artificial Intelligence, pages 12–25. Springer. [find](https://scholar.google.com/scholar?q=Assaf+Toledo%2C+Sophia+Katrenko%2C+Stavroula+Alexandropoulou%2C+Heidi+Klockmann%2C+Asher+Stern%2C+Ido+Dagan%2C+and+Yoad+Winter%2E+2012%2E+Semantic+annotation+for+textual+entailment+recognition%2E+In+Mexican+Internation)
75. Aaron Steven White, Pushpendre Rastogi, Kevin Duh, and Benjamin Van Durme. 2017. [Inference is everything: Recasting semantic resources into a unified evaluation framework](https://md.research.elamur.ai/free/en/papers/white-2017-inference-is-everything-recasting-semantic/). In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 996–1005, Taipei, Taiwan. Asian Federation of Natural Language Processing.
76. Masatoshi Tsuchiya. 2018. [Performance impact caused by hidden bias of training data for recognizing textual entailment](https://md.research.elamur.ai/free/en/papers/tsuchiya-2018-performance-impact-caused-hidden-bias/). In Proceedings of LREC.
77. Aaron Steven White and Kyle Rawlins. 2018. The role of veridicality and factivity in clause selection. In Proceedings of the 48th Annual Meeting of the North East Linguistic Society. [find](https://scholar.google.com/scholar?q=Aaron+Steven+White+and+Kyle+Rawlins%2E+2018%2E+The+role+of+veridicality+and+factivity+in+clause+selection%2E+In+Proceedings+of+the+48th+Annual+Meeting+of+the+North+East+Linguistic+Society%2E)
78. Clara Vania, Ruijie Chen, and Samuel R. Bowman. 2020. Asking Crowdworkers to Write Entailment Examples: The Best of Bad options. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 672–686, Suzhou, China. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2010.06122)
79. Aaron Steven White, Elias Stengel-Eskin, Siddharth Vashishtha, Venkata Subrahmanyan Govindarajan, Dee Ann Reisinger, Tim Vieira, Keisuke Sakaguchi, Sheng Zhang, Francis Ferraro, Rachel Rudinger, Kyle Rawlins, and Benjamin Van Durme. 2020. [The universal decompositional semantics dataset and decomp toolkit](https://md.research.elamur.ai/free/en/papers/white-2020-universal-decompositional-semantics-dataset-decomp/). In Proceedings of the 12th Language Resources and Evaluation Conference, pages 5698– 5707, Marseille, France. European Language Resources Association. [↗](https://doi.org/10.13016/m2fhz1-knec)
80. Eric Wallace, Adina Williams, Robin Jia, and Douwe Kiela. 2021. Analyzing dynamic adversarial training data in the limit. arXiv preprint arXiv:2110.08514. [↗](https://arxiv.org/abs/2110.08514)
81. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n18-1101)
82. Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abzianidze, and Johan Bos. 2019. HELP: A dataset for identifying shortcomings of neural models in monotonicity reasoning. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019), pages 250–255, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1904.12166)
83. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. [Xlnet: Generalized autoregressive pretraining for language understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/). In Advances in neural information processing systems, pages 5754–5764. [↗](https://doi.org/10.48550/arxiv.1906.08237)
84. Lang Yu and Allyson Ettinger. 2020. Assessing phrasal representation and composition in transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4896–4907, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Lang+Yu+and+Allyson+Ettinger%2E+2020%2E+Assessing+phrasal+representation+and+composition+in+transformers%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C)
85. Xiang Zhou, Yixin Nie, Hao Tan, and Mohit Bansal. 2020. The curse of performance instability in analysis datasets: Consequences, source, and suggestions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8215–8228, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xiang+Zhou%2C+Yixin+Nie%2C+Hao+Tan%2C+and+Mohit+Bansal%2E+2020%2E+The+curse+of+performance+instability+in+analysis+datasets%3A+Consequences%2C+source%2C+and+suggestions%2E+In+Proceedings+of+the+2020+Conference+on+Empir)

*Updated 24 October 2026*