---
title: "Adversarial NLI — paper by Nie et al. (2020) — summary and ideas"
description: "The authors propose HAMLET, a procedure in which data are collected by humans with a model in… 59 references, cited by 659 papers here."
authors: ["Yixin Nie", "Adina Williams", "Emily Dinan", "Mohit Bansal", "Jason Weston", "Douwe Kiela"]
year: 2020
venue: "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/2020.acl-main.441/"
doi: "10.18653/v1/2020.acl-main.441"
arxiv: "1910.14599"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/nie-2020-adversarial-nli-new-benchmark-natural/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nie-2020-adversarial-nli-new-benchmark-natural/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/) / Adversarial NLI: A New Benchmark for Natural Language Understanding

# Adversarial NLI: A New Benchmark for Natural Language Understanding

**Authors:** [Yixin Nie](https://md.research.elamur.ai/free/en/authors/yixin-nie/), [Adina Williams](https://md.research.elamur.ai/free/en/authors/adina-williams/), [Emily Dinan](https://md.research.elamur.ai/free/en/authors/emily-dinan/), [Mohit Bansal](https://md.research.elamur.ai/free/en/authors/mohit-bansal/), [Jason Weston](https://md.research.elamur.ai/free/en/authors/jason-weston/) and [Douwe Kiela](https://md.research.elamur.ai/free/en/authors/douwe-kiela/) · **Year:** 2020 · **Venue:** [Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2020](https://md.research.elamur.ai/free/en/venues/acl/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.acl-main.441/) · [PDF](https://aclanthology.org/2020.acl-main.441.pdf) · [DOI 10.18653/v1/2020.acl-main.441](https://doi.org/10.18653/v1/2020.acl-main.441) · [arXiv 1910.14599](https://arxiv.org/abs/1910.14599) · [OpenAlex](https://openalex.org/W3034850762)

**Citations:** 637 citations (OpenAlex, 2026-10-14) · [659](https://md.research.elamur.ai/free/en/papers/nie-2020-adversarial-nli-new-benchmark-natural/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

Modern NLU benchmarks are quickly saturated: models such as BERT achieve high scores on them. At the same time, these models often rely on superficial statistical regularities in the data rather than on genuine language understanding. The authors propose HAMLET, a procedure in which data are collected by humans with a model in the loop. An annotator receives a context and a target label (entailment, contradiction or neutral) and writes a hypothesis intended to mislead the model. Examples on which the model made a mistake are verified by other people. After each round, the model is retrained on the new data, and the next round is conducted against a stronger system. Over three rounds, the Adversarial NLI (ANLI) corpus was collected in this way. Models trained on it achieved state-of-the-art results at the time on a number of well-known NLI benchmarks. By contrast, the test set of ANLI itself remained difficult for leading models. The value of the work lies in the fact that the benchmark is designed as a moving target: once it becomes saturated, a new round can be collected.

## Key points

- **Three rounds, increasingly difficult.** In round A1, the adversary model is BERT-Large trained on SNLI and MNLI. In A2, it is RoBERTa trained on SNLI, MNLI, the NLI version of FEVER and the A1 data. Rounds 2 and 3 used ensembles of models. The contexts for A1 and A2 are Wikipedia passages from HotpotQA; for A3, texts from various genres, including news.
- **Size of ANLI.** In total, 162,865 examples in train and 3,200 each in dev and test. Of these, A1 has 16,946 / 1,000 / 1,000, A2 has 45,460 / 1,000 / 1,000, and A3 has 100,459 / 1,200 / 1,200.
- **Model error rate.** After human verification, the model was wrong on 18.33% of examples in A1, 8.07% in A2 and 8.60% in A3; 9.52% across the corpus as a whole. Annotators needed 5.7 attempts on average to fool the model.
- **Quality control.** Each example is checked by two verifiers and, if they disagree, by a third. Dev and test consist only of verified examples on which the model made a mistake. The test set is balanced by label and contains only examples from annotators whose data do not appear in train. For each model error, the author of the example explains why, in their view, it occurred.
- **Access.** The corpus is available at github.com/facebookresearch/anli. The authors consider the method applicable to other classification tasks, but not to generative tasks.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Meta](https://md.research.elamur.ai/free/en/companies/meta/) — Company, Menlo Park, United States · 183 works · 228,558 citations
- [University of North Carolina at Chapel Hill](https://md.research.elamur.ai/free/en/universities/university-of-north-carolina-at-chapel-hill/) — University, Chapel Hill, United States · 67 works · 16,474 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

💡 Concepts (1): [💡 Natural language understanding](https://md.research.elamur.ai/free/en/concepts/natural-language-understanding/) · 51 works

## Cite

Cite as: Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston and Douwe Kiela. 2020. Adversarial NLI: A New Benchmark for Natural Language Understanding. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{nie2020adversarial,
  title = {{Adversarial NLI: A New Benchmark for Natural Language Understanding}},
  author = {Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston and Douwe Kiela},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.441},
  eprint = {1910.14599},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.acl-main.441}
}
```

- [← previous AraBERT: Transformer-based Model for Arabic Language Understanding](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/)
- [next → On the Cross-lingual Transferability of Monolingual Representations](https://md.research.elamur.ai/free/en/papers/artetxe-2020-cross-lingual-transferability-monolingual-representations/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/)

## References

59 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425–2433. [↗](https://doi.org/10.1007/s11263-016-0966-6)
2. Yonatan Belinkov and Yonatan Bisk. 2017. Synthetic and natural noise both break neural machine translation. arXiv preprint arXiv:1711.02173. [↗](https://arxiv.org/abs/1711.02173)
3. Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. 2009. The Fifth PASCAL Recognizing Textual Entailment Challenge. TAC. [find](https://scholar.google.com/scholar?q=Luisa+Bentivogli%2C+Ido+Dagan%2C+Hoa+Trang+Dang%2C+Danilo+Giampiccolo%2C+and+Bernardo+Magnini%2E+2009%2E+The+Fifth+PASCAL+Recognizing+Textual+Entailment+Challenge%2E+TAC%2E)
4. Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Wen-tau Yih, and Yejin Choi. 2019. [Abductive commonsense reasoning](https://md.research.elamur.ai/free/en/papers/bhagavatula-2019-abductive-commonsense-reasoning/). arXiv preprint arXiv:1908.05739. [↗](https://arxiv.org/abs/1908.05739)
5. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. [A large annotated corpus for learning natural language inference](https://md.research.elamur.ai/free/en/papers/bowman-2015-large-annotated-corpus-learning-natural/). arXiv preprint arXiv:1508.05326. [↗](https://arxiv.org/abs/1508.05326)
6. Ronan Le Bras, Swabha Swayamdipta, Chandra Bhagavatula, Rowan Zellers, Matthew E Peters, Ashish Sabharwal, and Yejin Choi. 2020. Adversarial filters of dataset biases. arXiv preprint arXiv:2002.04108. [↗](https://arxiv.org/abs/2002.04108)
7. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. [Reading Wikipedia to answer open-domain questions](https://md.research.elamur.ai/free/en/papers/chen-2017-reading-wikipedia-answer-open-domain/). In Association for Computational Linguistics (ACL).
8. Michael Chen, Mike D’Arcy, Alisa Liu, Jared Fernandez, and Doug Downey. 2019. CODAH: an adversarially authored question-answer dataset for common sense. CoRR, abs/1904.04365. [↗](https://arxiv.org/abs/1904.04365)
9. Dan Cireşan, Ueli Meier, and Jürgen Schmidhuber. 2012. Multi-column deep neural networks for image classification. arXiv preprint arXiv:1202.2745. [↗](https://arxiv.org/abs/1202.2745)
10. Alexis Conneau and Douwe Kiela. 2018. [Senteval: An evaluation toolkit for universal sentence representations](https://md.research.elamur.ai/free/en/papers/conneau-2018-senteval-evaluation-toolkit-universal-sentence/). arXiv preprint arXiv:1803.05449. [↗](https://arxiv.org/abs/1803.05449)
11. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loı̈c Barrault, and Antoine Bordes. 2017. [Supervised learning of universal sentence representations from natural language inference data](https://md.research.elamur.ai/free/en/papers/conneau-2017-supervised-learning-universal-sentence-representations/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670–680. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d17-1070)
12. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee. [find](https://scholar.google.com/scholar?q=Jia+Deng%2C+Wei+Dong%2C+Richard+Socher%2C+Li%2DJia+Li%2C+Kai+Li%2C+and+Li+Fei%2DFei%2E+2009%2E+Imagenet%3A+A+large%2Dscale+hierarchical+image+database%2E+In+2009+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+pa)
13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805. [↗](https://arxiv.org/abs/1810.04805)
14. Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. 2019. [Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack](https://md.research.elamur.ai/free/en/papers/dinan-2019-build-it-break-it-fix/). In Proceedings of EMNLP.
15. Allyson Ettinger, Sudha Rao, Hal Daumé III, and Emily M Bender. 2017. Towards linguistically generalizable nlp systems: A workshop and shared task. arXiv preprint arXiv:1711.01505. [↗](https://arxiv.org/abs/1711.01505)
16. Atticus Geiger, Ignacio Cases, Lauri Karttunen, and Christopher Potts. 2018. [Stress-testing neural models of natural language inference with multiply-quantified sentences](https://md.research.elamur.ai/free/en/papers/geiger-2018-stress-testing-neural-models-natural/). arXiv preprint arXiv:1810.13033. [↗](https://arxiv.org/abs/1810.13033)
17. Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. arXiv preprint arXiv:1908.07898. [↗](https://arxiv.org/abs/1908.07898)
18. Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. [Breaking nli systems with sentences that require simple lexical inferences](https://md.research.elamur.ai/free/en/papers/glockner-2018-breaking-nli-systems-sentences-that/). In Proceedings of ACL.
19. Yichen Gong, Heng Luo, and Jian Zhang. 2018. Natural language inference over interaction space. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. [↗](https://doi.org/10.48550/arxiv.1709.04348)
20. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. [Making the v in vqa matter: Elevating the role of image understanding in visual question answering](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/). In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6904–6913. [↗](https://doi.org/10.1007/s11263-018-1116-0)
21. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A Smith. 2018. [Annotation artifacts in natural language inference data](https://md.research.elamur.ai/free/en/papers/gururangan-2018-annotation-artifacts-natural-language-inference/). In Proceedings of NAACL.
22. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. [Deep residual learning for image recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/). In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778. [↗](https://doi.org/10.48550/arxiv.1512.03385)
23. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015. [The goldilocks principle: Reading children’s books with explicit memory representations](https://md.research.elamur.ai/free/en/papers/hill-2015-goldilocks-principle-reading-children-s/). arXiv preprint arXiv:1511.02301. [↗](https://arxiv.org/abs/1511.02301)
24. Allan Jabri, Armand Joulin, and Laurens Van Der Maaten. 2016. Revisiting visual question answering baselines. In European conference on computer vision, pages 727–739. Springer. [↗](https://doi.org/10.48550/arxiv.1606.08390)
25. Pushmeet Kohli, and James Allen. 2016. A corpus and evaluation framework for deeper understanding of commonsense stories. arXiv preprint arXiv:1604.01696. [↗](https://arxiv.org/abs/1604.01696)
26. Robin Jia and Percy Liang. 2017. [Adversarial examples for evaluating reading comprehension systems](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/). In Proceedings of EMNLP.
27. Divyansh Kaushik, Eduard Hovy, and Zachary C Lipton. 2019. Learning the difference that makes a difference with counterfactually-augmented data. arXiv preprint arXiv:1909.12434. [↗](https://arxiv.org/abs/1909.12434)
28. Divyansh Kaushik and Zachary C Lipton. 2018. [How much reading does reading comprehension require? a critical investigation of popular benchmarks](https://md.research.elamur.ai/free/en/papers/kaushik-2018-how-much-reading-does-reading/). arXiv preprint arXiv:1808.04926. [↗](https://arxiv.org/abs/1808.04926)
29. Bernhard Kratzwald and Stefan Feuerriegel. 2019. Learning from on-line user feedback in neural question answering on the web. In The World Wide Web Conference, pages 906–916. ACM. [find](https://scholar.google.com/scholar?q=Bernhard+Kratzwald+and+Stefan+Feuerriegel%2E+2019%2E+Learning+from+on%2Dline+user+feedback+in+neural+question+answering+on+the+web%2E+In+The+World+Wide+Web+Conference%2C+pages+906%E2%80%93916%2E+ACM%2E)
30. Wuwei Lan, Siyu Qiu, Hua He, and Wei Xu. 2017. [A continuously growing dataset of sentential paraphrases](https://md.research.elamur.ai/free/en/papers/lan-2017-continuously-growing-dataset-sentential-paraphrases/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1224–1234, Copenhagen, Denmark. Association for Computational Linguistics.
31. Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324. [↗](https://doi.org/10.1109/5.726791)
32. Huan Ling and Sanja Fidler. 2017. Teaching machines to describe images via natural language feedback. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 5075–5085. [↗](https://doi.org/10.48550/arxiv.1706.00130)
33. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019a. [Multi-task deep neural networks for natural language understanding](https://md.research.elamur.ai/free/en/papers/liu-2019-multi-task-deep-neural-networks/). arXiv preprint arXiv:1901.11504. [↗](https://arxiv.org/abs/1901.11504)
34. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint arXiv:1907.11692. [↗](https://arxiv.org/abs/1907.11692)
35. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. [Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/p19-1334)
36. Tom Mitchell, William Cohen, Estevam Hruschka, Partha Talukdar, Bo Yang, Justin Betteridge, Andrew Carlson, B Dalvi, Matt Gardner, Bryan Kisiel, et al. 2018. Never-ending learning. Communications of the ACM, 61(5):103–115. [↗](https://doi.org/10.1145/3191513)
37. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340–2353, Santa Fe, New Mexico, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nasrin+Mostafazadeh%2C+Nathanael+Chambers%2C+Xiaodong+He%2C+Devi+Parikh%2C+Dhruv+Batra%2C+Lucy+Vanderwende%2C+Aakanksha+Naik%2C+Abhilasha+Ravichander%2C+Norman+Sadeh%2C+Carolyn+Rose%2C+and+Graham+Neubig%2E+2018%2E+Stress+tes)
38. Yixin Nie and Mohit Bansal. 2017. Shortcut-stacked sentence encoders for multi-domain inference. arXiv preprint arXiv:1708.02312. [↗](https://arxiv.org/abs/1708.02312)
39. Yixin Nie, Haonan Chen, and Mohit Bansal. 2019. Combining fact extraction and verification with neural semantic matching networks. In Association for the Advancement of Artificial Intelligence (AAAI). [find](https://scholar.google.com/scholar?q=Yixin+Nie%2C+Haonan+Chen%2C+and+Mohit+Bansal%2E+2019%2E+Combining+fact+extraction+and+verification+with+neural+semantic+matching+networks%2E+In+Association+for+the+Advancement+of+Artificial+Intelligence+%28AAAI%29%2E)
40. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. [The lambada dataset: Word prediction requiring a broad discourse context](https://md.research.elamur.ai/free/en/papers/paperno-2016-lambada-dataset-word-prediction-requiring/). arXiv preprint arXiv:1606.06031. [↗](https://arxiv.org/abs/1606.06031)
41. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. [Hypothesis only baselines in natural language inference](https://md.research.elamur.ai/free/en/papers/poliak-2018-hypothesis-only-baselines-natural-language/). In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics.
42. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [Squad: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). arXiv preprint arXiv:1606.05250. [↗](https://arxiv.org/abs/1606.05250)
43. Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. 2018. Do cifar-10 classifiers generalize to cifar-10? arXiv preprint arXiv:1806.00451. [↗](https://arxiv.org/abs/1806.00451)
44. Andrew Ruef, Michael Hicks, James Parker, Dave Levin, Michelle L Mazurek, and Piotr Mardziel. 2016. Build it, break it, fix it: Contesting secure development. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pages 690–703. ACM. [↗](https://doi.org/10.48550/arxiv.1606.01881)
45. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015. [Imagenet large scale visual recognition challenge](https://md.research.elamur.ai/free/en/papers/russakovsky-2014-imagenet-large-scale-visual-recognition/). International journal of computer vision, 115(3):211–252. [↗](https://doi.org/10.1007/s11263-015-0816-y)
46. Chinnadhurai Sankar, Sandeep Subramanian, Christopher Pal, Sarath Chandar, and Yoshua Bengio. 2019. Do neural dialog systems use the conversation history effectively? an empirical study. arXiv preprint arXiv:1906.01603. William Shakespeare. 1603. The Tragedy of Hamlet, Prince of Denmark. [↗](https://arxiv.org/abs/1906.01603)
47. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. [Fever: a large-scale dataset for fact extraction and verification](https://md.research.elamur.ai/free/en/papers/thorne-2018-fever-large-scale-dataset-fact/). arXiv preprint arXiv:1803.05355. [↗](https://arxiv.org/abs/1803.05355)
48. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of ACL. [↗](https://doi.org/10.48550/arxiv.1905.07830)
49. Masatoshi Tsuchiya. 2018. [Performance impact caused by hidden bias of training data for recognizing textual entailment](https://md.research.elamur.ai/free/en/papers/tsuchiya-2018-performance-impact-caused-hidden-bias/). In Proceedings of LREC.
50. Eric Wallace, Pedro Rodriguez, Shi Feng, Ikuya Yamada, and Jordan Boyd-Graber. 2019. Trick me if you can: Human-in-the-loop generation of adversarial question answering examples. In Transactions of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Eric+Wallace%2C+Pedro+Rodriguez%2C+Shi+Feng%2C+Ikuya+Yamada%2C+and+Jordan+Boyd%2DGraber%2E+2019%2E+Trick+me+if+you+can%3A+Human%2Din%2Dthe%2Dloop+generation+of+adversarial+question+answering+examples%2E+In+Transactions+of+th)
51. Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. 2013. Regularization of neural networks using dropconnect. In International conference on machine learning, pages 1058–1066. [find](https://scholar.google.com/scholar?q=Li+Wan%2C+Matthew+Zeiler%2C+Sixin+Zhang%2C+Yann+Le+Cun%2C+and+Rob+Fergus%2E+2013%2E+Regularization+of+neural+networks+using+dropconnect%2E+In+International+conference+on+machine+learning%2C+pages+1058%E2%80%931066%2E)
52. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. [Superglue: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). arXiv preprint arXiv:1905.00537. [↗](https://arxiv.org/abs/1905.00537)
53. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. [Glue: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). arXiv preprint arXiv:1804.07461. [↗](https://arxiv.org/abs/1804.07461)
54. John Wieting and Douwe Kiela. 2019. No training required: Exploring random encoders for sentence classification. arXiv preprint arXiv:1901.10444. [↗](https://arxiv.org/abs/1901.10444)
55. Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017. [A broad-coverage challenge corpus for sentence understanding through inference](https://md.research.elamur.ai/free/en/papers/williams-2018-broad-coverage-challenge-corpus-sentence/). arXiv preprint arXiv:1704.05426. [↗](https://arxiv.org/abs/1704.05426)
56. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. [Xlnet: Generalized autoregressive pretraining for language understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/). arXiv preprint arXiv:1906.08237. [↗](https://arxiv.org/abs/1906.08237)
57. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. [Hotpotqa: A dataset for diverse, explainable multi-hop question answering](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/). arXiv preprint arXiv:1809.09600. [↗](https://arxiv.org/abs/1809.09600)
58. Zhilin Yang, Saizheng Zhang, Jack Urbanek, Will Feng, Alexander H Miller, Arthur Szlam, Douwe Kiela, and Jason Weston. 2017. Mastering the dungeon: Grounded language learning by mechanical turker descent. arXiv preprint arXiv:1711.07950. [↗](https://arxiv.org/abs/1711.07950)
59. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. Swag: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Yonatan+Bisk%2C+Roy+Schwartz%2C+and+Yejin+Choi%2E+2018%2E+Swag%3A+A+large%2Dscale+adversarial+dataset+for+grounded+commonsense+inference%2E+In+Proceedings+of+EMNLP%2E)

*Updated 1 November 2026*