---
title: "UnNatural Language Inference — paper by Sinha et al. (2021)"
description: "To quantify this, they introduce a family of Permutation Acceptance metrics and build… 0.38 accuracy. 107 references, cited by 61 papers here."
authors: ["Koustuv Sinha", "Prasanna Parthasarathi", "Joelle Pineau", "Adina Williams"]
year: 2021
venue: "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)"
source: "https://aclanthology.org/2021.acl-long.569/"
doi: "10.18653/v1/2021.acl-long.569"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sinha-2021-unnatural-language-inference/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sinha-2021-unnatural-language-inference/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / UnNatural Language Inference

# UnNatural Language Inference

**Authors:** [Koustuv Sinha](https://md.research.elamur.ai/free/en/authors/koustuv-sinha/), Prasanna Parthasarathi, [Joelle Pineau](https://md.research.elamur.ai/free/en/authors/joelle-pineau/) and [Adina Williams](https://md.research.elamur.ai/free/en/authors/adina-williams/) · **Year:** 2021 · **Venue:** [Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2021](https://md.research.elamur.ai/free/en/venues/acl/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.acl-long.569/) · [PDF](https://aclanthology.org/2021.acl-long.569.pdf) · [DOI 10.18653/v1/2021.acl-long.569](https://doi.org/10.18653/v1/2021.acl-long.569) · [OpenAlex](https://openalex.org/W3119636502)

**Citations:** 71 citations (OpenAlex, 2026-10-14) · [61](https://md.research.elamur.ai/free/en/papers/sinha-2021-unnatural-language-inference/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
This paper tests whether high-performing Natural Language Inference (NLI) models actually depend on word order, which is a basic requirement for "knowing syntax." The authors randomly shuffle the words of premise and hypothesis sentences and check whether models still output the original gold label. To quantify this, they introduce a family of Permutation Acceptance metrics and build large permuted test sets from SNLI, MultiNLI (MNLI), ANLI, and the Chinese OCNLI. They evaluate pre-trained Transformers (BERT, RoBERTa, BART, DistilBERT) and older encoders (InferSent, ConvNet, BiLSTM). All of these models largely ignore word order. Scrambled inputs often receive the same label as the originals, and shuffling sometimes even fixes a wrong prediction. Human experts, by contrast, perform poorly on such scrambled pairs. The authors also propose a training fix that pushes models toward uncertainty on permuted inputs. The study matters because it questions claims that strong benchmark scores reflect humanlike syntactic understanding.

## Key points
- For 98.7% of MNLI examples, at least one random word permutation leads the model to the gold label. This exceeds the roughly 90.5% state-of-the-art accuracy on unpermuted data.
- The effect appears across English (SNLI, MNLI, ANLI) and Mandarin Chinese (OCNLI), and for both Transformer and non-Transformer encoders.
- Transformer models (RoBERTa, BART, DistilBERT) accept more permutations when local POS-tag neighborhoods are preserved. Non-Transformer models do not show this pattern.
- Two NLI experts labeled 200 permuted MNLI pairs, all of which RoBERTa-large had labeled correctly. The experts reached only about 0.58 and 0.38 accuracy.
- Fine-tuning RoBERTa on MNLI with an extra term that maximizes entropy on permuted examples lowers Ω_max from about 0.98 to about 0.33 on MNLI matched dev. Accuracy stays the same (0.905 vs. 0.908). Code and data are released at github.com/facebookresearch/unlu.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Meta](https://md.research.elamur.ai/free/en/companies/meta/) — Company, Menlo Park, United States · 158 works · 211,459 citations
- [Meta (Israel)](https://md.research.elamur.ai/free/en/companies/meta-israel/) — Company, Tel Aviv, Israel · 36 works · 31,506 citations
- [McGill University](https://md.research.elamur.ai/free/en/universities/mcgill-university/) — University, Montreal, Canada · 51 works · 10,476 citations
- [Mila - Quebec Artificial Intelligence Institute](https://md.research.elamur.ai/free/en/labs/mila-quebec-artificial-intelligence-institute/) — Research institute, Montreal, Canada · 31 works · 13,061 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Koustuv Sinha, Prasanna Parthasarathi, Joelle Pineau and Adina Williams. 2021. UnNatural Language Inference. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{sinha2021unnatural,
  title = {{UnNatural Language Inference}},
  author = {Koustuv Sinha, Prasanna Parthasarathi, Joelle Pineau and Adina Williams},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.569},
  url = {https://doi.org/10.18653/v1/2021.acl-long.569}
}
```

- [← previous Text2Mol: Cross-Modal Molecule Retrieval with Natural Language Queries](https://md.research.elamur.ai/free/en/papers/edwards-2021-text2mol-cross-modal-molecule-retrieval/)
- [next → What to Pre-Train on? Efficient Intermediate Task Selection](https://md.research.elamur.ai/free/en/papers/poth-2021-what-pre-train-efficient-intermediate/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/)

## References

107 references, 35 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anne Abeille. 1990. Lexical and syntactic rules in a Tree Adjoining Grammar. In 28th Annual Meeting of the Association for Computational Linguistics, pages 292–298, Pittsburgh, Pennsylvania, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Anne+Abeille%2E+1990%2E+Lexical+and+syntactic+rules+in+a+Tree+Adjoining+Grammar%2E+In+28th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+292%E2%80%93298%2C+Pittsburgh%2C+Pennsylvania%2C+USA%2E+Asso)
2. Matteo Alleman, Jonathan Mamou, Miguel A Del Rio, Hanlin Tang, Yoon Kim, and SueYeon Chung. 2021. Syntactic perturbations reveal representational correlates of hierarchical phrase structure in pretrained language models. arXiv preprint arXiv:2104.07578. [find](https://scholar.google.com/scholar?q=Matteo+Alleman%2C+Jonathan+Mamou%2C+Miguel+A+Del+Rio%2C+Hanlin+Tang%2C+Yoon+Kim%2C+and+SueYeon+Chung%2E+2021%2E+Syntactic+perturbations+reveal+representational+correlates+of+hierarchical+phrase+structure+in+pretrai)
3. Aixiu An, Peng Qian, Ethan Wilcox, and Roger Levy. 2019. Representation of constituents in neural language models: Coordination phrase as a case study. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lang… [find](https://scholar.google.com/scholar?q=Aixiu+An%2C+Peng+Qian%2C+Ethan+Wilcox%2C+and+Roger+Levy%2E+2019%2E+Representation+of+constituents+in+neural+language+models%3A+Coordination+phrase+as+a+case+study%2E+In+Proceedings+of+the+2019+Conference+on+Empiric)
4. Alan D Baddeley, Graham J Hitch, and Richard J Allen. 2009. Working memory and binding in sentence recall. Journal of Memory and Language. [find](https://scholar.google.com/scholar?q=Alan+D+Baddeley%2C+Graham+J+Hitch%2C+and+Richard+J+Allen%2E+2009%2E+Working+memory+and+binding+in+sentence+recall%2E+Journal+of+Memory+and+Language%2E)
5. Emily M. Bender and Alexander Koller. 2020. [Climbing towards NLU: On meaning, form, and understanding in the age of data](https://md.research.elamur.ai/free/en/papers/bender-2020-climbing-nlu-meaning-form-understanding/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185–5198, Online. Association for Computational Linguistics.
6. Jean-Phillipe Bernardy and Shalom Lappin. 2017. Using deep neural networks to learn syntactic agreement. In Linguistic Issues in Language Technology, Volume 15, 2017. CSLI Publications. [find](https://scholar.google.com/scholar?q=Jean%2DPhillipe+Bernardy+and+Shalom+Lappin%2E+2017%2E+Using+deep+neural+networks+to+learn+syntactic+agreement%2E+In+Linguistic+Issues+in+Language+Technology%2C+Volume+15%2C+2017%2E+CSLI+Publications%2E)
7. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. [Latent dirichlet allocation](https://md.research.elamur.ai/free/en/papers/blei-2003-latent-dirichlet-allocation/). JMLR.
8. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computati… [find](https://scholar.google.com/scholar?q=Samuel+R%2E+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D%2E+Manning%2E+2015%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+Proceedings+of+the+2015+Conference+on+Empirica)
9. Joan Bresnan, Ash Asudeh, Ida Toivonen, and Stephen Wechsler. 2015. Lexical-functional syntax. John Wiley & Sons. [find](https://scholar.google.com/scholar?q=Joan+Bresnan%2C+Ash+Asudeh%2C+Ida+Toivonen%2C+and+Stephen+Wechsler%2E+2015%2E+Lexical%2Dfunctional+syntax%2E+John+Wiley+%26+Sons%2E)
10. [Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri…](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
11. James McKeen Cattell. 1886. The time it takes to see and name objects. Mind, os-XI(41):63–65. [find](https://scholar.google.com/scholar?q=James+McKeen+Cattell%2E+1886%2E+The+time+it+takes+to+see+and+name+objects%2E+Mind%2C+os%2DXI%2841%29%3A63%E2%80%9365%2E)
12. Rui Chaves. 2020. What don’t RNN language models learn about filler-gap dependencies? In Proceedings of the Society for Computation in Linguistics 2020, pages 1–11, New York, New York. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rui+Chaves%2E+2020%2E+What+don%E2%80%99t+RNN+language+models+learn+about+filler%2Dgap+dependencies%3F+In+Proceedings+of+the+Society+for+Computation+in+Linguistics+2020%2C+pages+1%E2%80%9311%2C+New+York%2C+New+York%2E+Association+for)
13. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1657–1668, Vancouver, Canada. Association for Comp… [find](https://scholar.google.com/scholar?q=Qian+Chen%2C+Xiaodan+Zhu%2C+Zhen%2DHua+Ling%2C+Si+Wei%2C+Hui+Jiang%2C+and+Diana+Inkpen%2E+2017%2E+Enhanced+LSTM+for+natural+language+inference%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computa)
14. Gennaro Chierchia and Sally McConnell-Ginet. 1990. Meaning and grammar: An Introduction to Semantics. Cambridge, Ma: MIT Press. [find](https://scholar.google.com/scholar?q=Gennaro+Chierchia+and+Sally+McConnell%2DGinet%2E+1990%2E+Meaning+and+grammar%3A+An+Introduction+to+Semantics%2E+Cambridge%2C+Ma%3A+MIT+Press%2E)
15. Grzegorz Chrupała and Afra Alishahi. 2019. Correlating neural and symbolic representations of language. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2952–2962, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Grzegorz+Chrupa%C5%82a+and+Afra+Alishahi%2E+2019%2E+Correlating+neural+and+symbolic+representations+of+language%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+page)
16. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286, Florence, Italy. Association for Computationa… [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Urvashi+Khandelwal%2C+Omer+Levy%2C+and+Christopher+D%2E+Manning%2E+2019%2E+What+does+BERT+look+at%3F+an+analysis+of+BERT%E2%80%99s+attention%2E+In+Proceedings+of+the+2019+ACL+Workshop+BlackboxNLP%3A+Analyzing+an)
17. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In ICML. [find](https://scholar.google.com/scholar?q=Ronan+Collobert+and+Jason+Weston%2E+2008%2E+A+unified+architecture+for+natural+language+processing%3A+Deep+neural+networks+with+multitask+learning%2E+In+ICML%2E)
18. Cleo Condoravdi, Dick Crouch, Valeria de Paiva, Reinhard Stolle, and Daniel G. Bobrow. 2003. Entailment, intensionality and text understanding. In Proceedings of the HLT-NAACL 2003 Workshop on Text Meaning, pages 38–45. [find](https://scholar.google.com/scholar?q=Cleo+Condoravdi%2C+Dick+Crouch%2C+Valeria+de+Paiva%2C+Reinhard+Stolle%2C+and+Daniel+G%2E+Bobrow%2E+2003%2E+Entailment%2C+intensionality+and+text+understanding%2E+In+Proceedings+of+the+HLT%2DNAACL+2003+Workshop+on+Text+Me)
19. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loı̈c Barrault, and Antoine Bordes. 2017. [Supervised learning of universal sentence representations from natural language inference data](https://md.research.elamur.ai/free/en/papers/conneau-2017-supervised-learning-universal-sentence-representations/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670–680, Copenhagen,…
20. Jillian Da Costa and Rui Chaves. 2020. Assessing the ability of transformer-based neural models to represent structurally unbounded dependencies. In Proceedings of the Society for Computation in Linguistics 2020, pages 12–21, New York, New York. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jillian+Da+Costa+and+Rui+Chaves%2E+2020%2E+Assessing+the+ability+of+transformer%2Dbased+neural+models+to+represent+structurally+unbounded+dependencies%2E+In+Proceedings+of+the+Society+for+Computation+in+Lingu)
21. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112, New Orleans, Louisiana. Association for Com… [find](https://scholar.google.com/scholar?q=Smith%2E+2018%2E+Annotation+artifacts+in+natural+language+inference+data%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics%3A+Human+Langua)
22. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop. Springer. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+2005%2E+The+pascal+recognising+textual+entailment+challenge%2E+In+Machine+Learning+Challenges+Workshop%2E+Springer%2E)
23. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment. Springer. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+2006%2E+The+PASCAL+recognising+textual+entailment+challenge%2E+In+Machine+learning+challenges%2E+evaluating+predictive+uncertainty%2C+visual+object+classificati)
24. Ishita Dasgupta, Demi Guo, Andreas Stuhlmüller, Samuel J Gershman, and Noah D Goodman. 2018. Evaluating compositionality in sentence embeddings. In Proceedings of Annual Meeting of the Cognitive Science Society. [find](https://scholar.google.com/scholar?q=Ishita+Dasgupta%2C+Demi+Guo%2C+Andreas+Stuhlmu%CC%88ller%2C+Samuel+J+Gershman%2C+and+Noah+D+Goodman%2E+2018%2E+Evaluating+compositionality+in+sentence+embeddings%2E+In+Proceedings+of+Annual+Meeting+of+the+Cognitive+Scie)
25. Forrest Davis and Marten van Schijndel. 2020. Recurrent neural network language models always learn English-like relative clause attachment. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1979–1990, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Forrest+Davis+and+Marten+van+Schijndel%2E+2020%2E+Recurrent+neural+network+language+models+always+learn+English%2Dlike+relative+clause+attachment%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Associatio)
26. Allyson Ettinger. 2020. [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://md.research.elamur.ai/free/en/papers/ettinger-2020-what-bert-is-not-lessons/). Transactions of the Association for Computational Linguistics, 8:34–48.
27. Gottlob Frege. 1948. Sense and reference. The philosophical review. [find](https://scholar.google.com/scholar?q=Gottlob+Frege%2E+1948%2E+Sense+and+reference%2E+The+philosophical+review%2E)
28. Kanishk Gandhi and Brenden M Lake. 2020. Mutual exclusivity as a challenge for deep neural networks. In Advances in Neural Information Processing Systems, volume 33, pages 14182–14192. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Kanishk+Gandhi+and+Brenden+M+Lake%2E+2020%2E+Mutual+exclusivity+as+a+challenge+for+deep+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+33%2C+pages+14182%E2%80%9314192%2E+Curran+Associat)
29. Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy. 2020. [SyntaxGym: An online platform for targeted evaluation of language models](https://md.research.elamur.ai/free/en/papers/gauthier-2020-syntaxgym-online-platform-targeted-evaluation/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 70–76, Online. Association for…
30. Yoav Goldberg. 2019. [Assessing BERT’s syntactic abilities](https://md.research.elamur.ai/free/en/papers/goldberg-2019-assessing-bert-s-syntactic-abilities/). arXiv preprint arXiv:1901.05287.
31. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. [Colorless green recurrent networks dream hierarchically](https://md.research.elamur.ai/free/en/papers/gulordava-2018-colorless-green-recurrent-networks-dream/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume…
32. Ashim Gupta, Giorgi Kvernadze, and Vivek Srikumar. 2021. BERT & family eat word salad: Experiments with text understanding. AAAI. [find](https://scholar.google.com/scholar?q=Ashim+Gupta%2C+Giorgi+Kvernadze%2C+and+Vivek+Srikumar%2E+2021%2E+BERT+%26+family+eat+word+salad%3A+Experiments+with+text+understanding%2E+AAAI%2E)
33. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Swabha+Swayamdipta%2C+Omer+Levy%2C+Roy+Schwartz%2C+Samuel+Bowman%2C+and+Noah+A%2E)
34. Zellig S Harris. 1954. Distributional structure. Word. [find](https://scholar.google.com/scholar?q=Zellig+S+Harris%2E+1954%2E+Distributional+structure%2E+Word%2E)
35. Robert Hawkins, Takateru Yamakoshi, Thomas Griffiths, and Adele Goldberg. 2020. Investigating representations of verb bias in neural language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4653–4663, Online. Association for Computatio… [find](https://scholar.google.com/scholar?q=Robert+Hawkins%2C+Takateru+Yamakoshi%2C+Thomas+Griffiths%2C+and+Adele+Goldberg%2E+2020%2E+Investigating+representations+of+verb+bias+in+neural+language+models%2E+In+Proceedings+of+the+2020+Conference+on+Empirical)
36. Irene Heim and Angelika Kratzer. 1998. Semantics in generative grammar. Blackwell Oxford. [find](https://scholar.google.com/scholar?q=Irene+Heim+and+Angelika+Kratzer%2E+1998%2E+Semantics+in+generative+grammar%2E+Blackwell+Oxford%2E)
37. John Hewitt and Christopher D. Manning. 2019. [A structural probe for finding syntax in word representations](https://md.research.elamur.ai/free/en/papers/hewitt-2019-structural-probe-finding-syntax-word/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129…
38. Matthew Honnibal, Ines Montani, Sofie Van Lan-deghem, and Adriane Boyd. 2020. spaCy: Industrial-strength Natural Language Processing in Python. [find](https://scholar.google.com/scholar?q=Matthew+Honnibal%2C+Ines+Montani%2C+Sofie+Van+Lan%2Ddeghem%2C+and+Adriane+Boyd%2E+2020%2E+spaCy%3A+Industrial%2Dstrength+Natural+Language+Processing+in+Python%2E)
39. Hai Hu, Kyle Richardson, Liang Xu, Lu Li, Sandra Kübler, and Lawrence Moss. 2020a. [OCNLI: Original Chinese Natural Language Inference](https://md.research.elamur.ai/free/en/papers/hu-2020-ocnli-original-chinese-natural/). In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3512–3526, Online. Association for Computational Linguistics.
40. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020b. A systematic assessment of syntactic generalization in neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1725–1744, Online. Association for Computational… [find](https://scholar.google.com/scholar?q=Jennifer+Hu%2C+Jon+Gauthier%2C+Peng+Qian%2C+Ethan+Wilcox%2C+and+Roger+Levy%2E+2020b%2E+A+systematic+assessment+of+syntactic+generalization+in+neural+language+models%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+t)
41. Ganesh Jawahar, Benoı̂t Sagot, and Djamé Seddah. 2019. [What does BERT learn about the structure of language?](https://md.research.elamur.ai/free/en/papers/jawahar-2019-what-does-bert-learn-about/) In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651–3657, Florence, Italy. Association for Computational Linguistics.
42. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. [Are natural language inference models IMPPRESsive? Learning IMPlicature and PRESupposition](https://md.research.elamur.ai/free/en/papers/jeretic-2020-are-natural-language-inference-models/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8690–8705, Online. Association for C…
43. Ronald M Kaplan and Joan Bresnan. 1995. Formal system for grammatical representation. Formal Issues in Lexical-Functional Grammar. [find](https://scholar.google.com/scholar?q=Ronald+M+Kaplan+and+Joan+Bresnan%2E+1995%2E+Formal+system+for+grammatical+representation%2E+Formal+Issues+in+Lexical%2DFunctional+Grammar%2E)
44. Yiğitcan Kaya, Sanghyun Hong, and Tudor Dumitras. 2019. Shallow-Deep Networks: Understanding and mitigating network overthinking. In Proceedings of the 2019 International Conference on Machine Learning (ICML), Long Beach, CA. [find](https://scholar.google.com/scholar?q=Yig%CC%86itcan+Kaya%2C+Sanghyun+Hong%2C+and+Tudor+Dumitras%2E+2019%2E+Shallow%2DDeep+Networks%3A+Understanding+and+mitigating+network+overthinking%2E+In+Proceedings+of+the+2019+International+Conference+on+Machine+Learni)
45. Jordan Kodner and Nitish Gupta. 2020. Overestimation of syntactic representation in neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1757–1762, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jordan+Kodner+and+Nitish+Gupta%2E+2020%2E+Overestimation+of+syntactic+representation+in+neural+language+models%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+)
46. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. [BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). In Proceedings of the 58th Annual Meeting of the…
47. Peiguang Li, Hongfeng Yu, Wenkai Zhang, Guangluan Xu, and Xian Sun. 2020. SA-NLI: A supervised attention based framework for natural language inference. Neurocomputing. [find](https://scholar.google.com/scholar?q=Peiguang+Li%2C+Hongfeng+Yu%2C+Wenkai+Zhang%2C+Guangluan+Xu%2C+and+Xian+Sun%2E+2020%2E+SA%2DNLI%3A+A+supervised+attention+based+framework+for+natural+language+inference%2E+Neurocomputing%2E)
48. Yongjie Lin, Yi Chern Tan, and Robert Frank. 2019. [Open sesame: Getting inside BERT’s linguistic knowledge](https://md.research.elamur.ai/free/en/papers/lin-2019-open-sesame-getting-inside-bert/). In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 241–253, Florence, Italy. Association for Computational Linguistics.
49. Tal Linzen and Marco Baroni. 2021. Syntactic structure from deep learning. Annual Review of Linguistics. [find](https://scholar.google.com/scholar?q=Tal+Linzen+and+Marco+Baroni%2E+2021%2E+Syntactic+structure+from+deep+learning%2E+Annual+Review+of+Linguistics%2E)
50. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. [Assessing the ability of LSTMs to learn syntax-sensitive dependencies](https://md.research.elamur.ai/free/en/papers/linzen-2016-assessing-ability-lstms-learn-syntax/). Transactions of the Association for Computational Linguistics, 4:521– 535.
51. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. arXiv. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+BERT+pretraining+approach%2E+a)
52. Christopher D Manning. 2015. Computational linguistics and deep learning. Computational Linguistics. [find](https://scholar.google.com/scholar?q=Christopher+D+Manning%2E+2015%2E+Computational+linguistics+and+deep+learning%2E+Computational+Linguistics%2E)
53. Christopher D. Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy. 2020. Emergent linguistic structure in artificial neural networks trained by self-supervision. Proceedings of the National Academy of Sciences, 117(48):30046–30054. [find](https://scholar.google.com/scholar?q=Christopher+D%2E+Manning%2C+Kevin+Clark%2C+John+Hewitt%2C+Urvashi+Khandelwal%2C+and+Omer+Levy%2E+2020%2E+Emergent+linguistic+structure+in+artificial+neural+networks+trained+by+self%2Dsupervision%2E+Proceedings+of+the+N)
54. Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rebecca+Marvin+and+Tal+Linzen%2E+2018%2E+Targeted+syntactic+evaluation+of+language+models%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+1192%E2%80%931202%2C+Bruss)
55. R. Thomas McCoy, Junghyun Min, and Tal Linzen. 2020. BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 217–2… [find](https://scholar.google.com/scholar?q=R%2E+Thomas+McCoy%2C+Junghyun+Min%2C+and+Tal+Linzen%2E+2020%2E+BERTs+of+a+feather+do+not+generalize+together%3A+Large+variability+in+generalization+across+models+with+similar+test+set+performance%2E+In+Proceedings+)
56. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. [Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy. Association for Computational Ling…
57. Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. [Efficient estimation of word representations in vector space](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/). In 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings.
58. Francis Mollica, Matthew Siegelman, Evgeniia Diachek, Steven T Piantadosi, Zachary Mineroff, Richard Futrell, Hope Kean, Peng Qian, and Evelina Fedorenko. 2020. Composition is the core driver of the language-selective network. Neurobiology of Language, 1(1):104–134. [find](https://scholar.google.com/scholar?q=Francis+Mollica%2C+Matthew+Siegelman%2C+Evgeniia+Diachek%2C+Steven+T+Piantadosi%2C+Zachary+Mineroff%2C+Richard+Futrell%2C+Hope+Kean%2C+Peng+Qian%2C+and+Evelina+Fedorenko%2E+2020%2E+Composition+is+the+core+driver+of+the+l)
59. Richard Montague. 1970. Universal grammar. Theoria. [find](https://scholar.google.com/scholar?q=Richard+Montague%2E+1970%2E+Universal+grammar%2E+Theoria%2E)
60. Aakanksha Naik, Abhilasha Ravichander, Carolyn Rose, and Eduard Hovy. 2019. Exploring numeracy in word embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3374–3380, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Aakanksha+Naik%2C+Abhilasha+Ravichander%2C+Carolyn+Rose%2C+and+Eduard+Hovy%2E+2019%2E+Exploring+numeracy+in+word+embeddings%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Lingui)
61. Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340–2353, Santa Fe, New Mexico, USA. Association for Computation… [find](https://scholar.google.com/scholar?q=Aakanksha+Naik%2C+Abhilasha+Ravichander%2C+Norman+Sadeh%2C+Carolyn+Rose%2C+and+Graham+Neubig%2E+2018%2E+Stress+test+evaluation+for+natural+language+inference%2E+In+Proceedings+of+the+27th+International+Conference+o)
62. Nikita Nangia and Samuel R. Bowman. 2019. Human vs. muppet: A conservative estimate of human performance on the GLUE benchmark. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4566–4575, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nikita+Nangia+and+Samuel+R%2E+Bowman%2E+2019%2E+Human+vs%2E+muppet%3A+A+conservative+estimate+of+human+performance+on+the+GLUE+benchmark%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computa)
63. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885–4901, Online. Association for Computa… [find](https://scholar.google.com/scholar?q=Yixin+Nie%2C+Adina+Williams%2C+Emily+Dinan%2C+Mohit+Bansal%2C+Jason+Weston%2C+and+Douwe+Kiela%2E+2020%2E+Adversarial+NLI%3A+A+new+benchmark+for+natural+language+understanding%2E+In+Proceedings+of+the+58th+Annual+Meetin)
64. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. [fairseq: A fast, extensible toolkit for sequence modeling](https://md.research.elamur.ai/free/en/papers/ott-2019-fairseq-fast-extensible-toolkit-sequence/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstr…
65. Prasanna Parthasarathi, Koustuv Sinha, Joelle Pineau, and Adina Williams. 2021. Sometimes we want translationese. arXiv preprint arXiv:2104.07623. — [arXiv:2104.07623](https://arxiv.org/abs/2104.07623) [find](https://scholar.google.com/scholar?q=Prasanna+Parthasarathi%2C+Koustuv+Sinha%2C+Joelle+Pineau%2C+and+Adina+Williams%2E+2021%2E+Sometimes+we+want+translationese%2E+arXiv+preprint+arXiv%3A2104%2E07623%2E+%E2%80%94+%5BarXiv%3A2104%2E07623%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2104%2E07623)
66. Thang M Pham, Trung Bui, Long Mai, and Anh Nguyen. 2020. [Out of order: How important is the sequential order of words in a sentence in natural language understanding tasks?](https://md.research.elamur.ai/free/en/papers/pham-2021-out-order-how-important-is/) arXiv preprint arXiv:2012.15180.
67. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180–191, New Orleans, Louisiana. Association for Compu… [find](https://scholar.google.com/scholar?q=Adam+Poliak%2C+Jason+Naradowsky%2C+Aparajita+Haldar%2C+Rachel+Rudinger%2C+and+Benjamin+Van+Durme%2E+2018%2E+Hypothesis+only+baselines+in+natural+language+inference%2E+In+Proceedings+of+the+Seventh+Joint+Conference+)
68. Carl Pollard and Ivan A Sag. 1994. Head-driven phrase structure grammar. University of Chicago Press. [find](https://scholar.google.com/scholar?q=Carl+Pollard+and+Ivan+A+Sag%2E+1994%2E+Head%2Ddriven+phrase+structure+grammar%2E+University+of+Chicago+Press%2E)
69. Grusha Prasad, Marten van Schijndel, and Tal Linzen. 2019. Using priming to uncover the organization of syntactic representations in neural language models. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 66–76, Hong Kong, China. Association for Comput… [find](https://scholar.google.com/scholar?q=Grusha+Prasad%2C+Marten+van+Schijndel%2C+and+Tal+Linzen%2E+2019%2E+Using+priming+to+uncover+the+organization+of+syntactic+representations+in+neural+language+models%2E+In+Proceedings+of+the+23rd+Conference+on+Co)
70. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2E)
71. Shauli Ravfogel, Yoav Goldberg, and Tal Linzen. 2019. Studying the inductive biases of RNNs with synthetic variations of natural languages. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Lon… [find](https://scholar.google.com/scholar?q=Shauli+Ravfogel%2C+Yoav+Goldberg%2C+and+Tal+Linzen%2E+2019%2E+Studying+the+inductive+biases+of+RNNs+with+synthetic+variations+of+natural+languages%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+)
72. Shauli Ravfogel, Yoav Goldberg, and Francis Tyers. 2018. Can LSTM learn to capture agreement? the case of Basque. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 98–107, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shauli+Ravfogel%2C+Yoav+Goldberg%2C+and+Francis+Tyers%2E+2018%2E+Can+LSTM+learn+to+capture+agreement%3F+the+case+of+Basque%2E+In+Proceedings+of+the+2018+EMNLP+Workshop+BlackboxNLP%3A+Analyzing+and+Interpreting+Neur)
73. Abhilasha Ravichander, Aakanksha Naik, Carolyn Rose, and Eduard Hovy. 2019. [EQUATE: A benchmark evaluation framework for quantitative reasoning in natural language inference](https://md.research.elamur.ai/free/en/papers/ravichander-2019-equate-benchmark-evaluation-framework-quantitative/). In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 349–361, Hong Kong, China. As…
74. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. [A primer in BERTology: What we know about how BERT works](https://md.research.elamur.ai/free/en/papers/rogers-2020-primer-bertology-what-we-know/). Transactions of the Association for Computational Linguistics, 8:842–866.
75. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. [find](https://scholar.google.com/scholar?q=Victor+Sanh%2C+Lysandre+Debut%2C+Julien+Chaumond%2C+and+Thomas+Wolf%2E+2020%2E+Distilbert%2C+a+distilled+version+of+bert%3A+smaller%2C+faster%2C+cheaper+and+lighter%2E)
76. Yves Schabes, Anne Abeille, and Aravind K. Joshi. 1988. [Parsing strategies with ‘lexicalized’ grammars: Application to Tree Adjoining Grammars](https://md.research.elamur.ai/free/en/papers/schabes-1988-parsing-strategies-lexicalized-grammars-application/). In Coling Budapest 1988 Volume 2: International Conference on Computational Linguistics.
77. Eckart Scheerer. 1981. Early german approaches to experimental reading research: The contributions of wilhelm wundt and ernst meumann. Psychological Research. [find](https://scholar.google.com/scholar?q=Eckart+Scheerer%2E+1981%2E+Early+german+approaches+to+experimental+reading+research%3A+The+contributions+of+wilhelm+wundt+and+ernst+meumann%2E+Psychological+Research%2E)
78. Marten van Schijndel and Tal Linzen. 2018. A neural model of adaptation in reading. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4704–4710, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Marten+van+Schijndel+and+Tal+Linzen%2E+2018%2E+A+neural+model+of+adaptation+in+reading%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+4704%E2%80%934710%2C+Brussels)
79. Marten van Schijndel, Aaron Mueller, and Tal Linzen. 2019. Quantity doesn’t buy quality syntax with neural language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNL… [find](https://scholar.google.com/scholar?q=Marten+van+Schijndel%2C+Aaron+Mueller%2C+and+Tal+Linzen%2E+2019%2E+Quantity+doesn%E2%80%99t+buy+quality+syntax+with+neural+language+models%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Langua)
80. Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. 2021. [Masked language modeling and the distributional hypothesis: Order word matters pre-training for little](https://md.research.elamur.ai/free/en/papers/sinha-2021-masked-language-modeling-distributional-hypothesis/). arXiv preprint arXiv:2104.06644.
81. Joshua Snell and Jonathan Grainger. 2017. The sentence superiority effect revisited. Cognition. [find](https://scholar.google.com/scholar?q=Joshua+Snell+and+Jonathan+Grainger%2E+2017%2E+The+sentence+superiority+effect+revisited%2E+Cognition%2E)
82. Joshua Snell and Jonathan Grainger. 2019. Word position coding in reading is noisy. Psychonomic bulletin & review, 26(2):609–615. [find](https://scholar.google.com/scholar?q=Joshua+Snell+and+Jonathan+Grainger%2E+2019%2E+Word+position+coding+in+reading+is+noisy%2E+Psychonomic+bulletin+%26+review%2C+26%282%29%3A609%E2%80%93615%2E)
83. Whitney Tabor. 1994. Syntactic innovation: A connectionist model. Ph.D. thesis. [find](https://scholar.google.com/scholar?q=Whitney+Tabor%2E+1994%2E+Syntactic+innovation%3A+A+connectionist+model%2E+Ph%2ED%2E+thesis%2E)
84. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. [BERT rediscovers the classical NLP pipeline](https://md.research.elamur.ai/free/en/papers/tenney-2019-bert-rediscovers-classical-nlp-pipeline/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593– 4601, Florence, Italy. Association for Computational Linguistics.
85. Hiroshi Toyota. 2001. Changes in the constraints of semantic and syntactic congruity on memory across three age groups. Perceptual and Motor Skills. [find](https://scholar.google.com/scholar?q=Hiroshi+Toyota%2E+2001%2E+Changes+in+the+constraints+of+semantic+and+syntactic+congruity+on+memory+across+three+age+groups%2E+Perceptual+and+Motor+Skills%2E)
86. Masatoshi Tsuchiya. 2018. [Performance impact caused by hidden bias of training data for recognizing textual entailment](https://md.research.elamur.ai/free/en/papers/tsuchiya-2018-performance-impact-caused-hidden-bias/). In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
87. Lucy Vanderwende and William B Dolan. 2005. What syntax can contribute in the entailment task. In Machine Learning Challenges Workshop. Springer. [find](https://scholar.google.com/scholar?q=Lucy+Vanderwende+and+William+B+Dolan%2E+2005%2E+What+syntax+can+contribute+in+the+entailment+task%2E+In+Machine+Learning+Challenges+Workshop%2E+Springer%2E)
88. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
89. Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing NLP. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proce… [find](https://scholar.google.com/scholar?q=Eric+Wallace%2C+Shi+Feng%2C+Nikhil+Kandpal%2C+Matt+Gardner%2C+and+Sameer+Singh%2E+2019%2E+Universal+adversarial+triggers+for+attacking+and+analyzing+NLP%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods)
90. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. [Superglue: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). In NeurIPS.
91. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,…
92. Alex Warstadt and Samuel R Bowman. 2020. Can neural networks acquire a structural bias from raw linguistic data? In Proceedings of the 42nd Annual Virtual Meeting of the Cognitive Science Society. [find](https://scholar.google.com/scholar?q=Alex+Warstadt+and+Samuel+R+Bowman%2E+2020%2E+Can+neural+networks+acquire+a+structural+bias+from+raw+linguistic+data%3F+In+Proceedings+of+the+42nd+Annual+Virtual+Meeting+of+the+Cognitive+Science+Society%2E)
93. Alex Warstadt, Yu Cao, Ioana Grosu, Wei Peng, Hagen Blix, Yining Nie, Anna Alsop, Shikha Bordia, Haokun Liu, Alicia Parrish, Sheng-Fu Wang, Jason Phang, Anhad Mohananey, Phu Mon Htut, Paloma Jeretic, and Samuel R. Bowman. 2019a. [Investigating BERT’s knowledge of language: Five analysis methods with…](https://md.research.elamur.ai/free/en/papers/warstadt-2019-investigating-bert-s-knowledge-language/)
94. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. [BLiMP: The benchmark of linguistic minimal pairs for English](https://md.research.elamur.ai/free/en/papers/warstadt-2020-blimp-benchmark-linguistic-minimal-pairs/). Transactions of the Association for Computational Linguistics, 8:377–392.
95. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019b. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625–641. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Amanpreet+Singh%2C+and+Samuel+R%2E+Bowman%2E+2019b%2E+Neural+network+acceptability+judgments%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+7%3A625%E2%80%93641%2E)
96. Yun Wen, Joshua Snell, and Jonathan Grainger. 2019. Parallel, cascaded, interactive processing of words during sentence reading. Cognition. [find](https://scholar.google.com/scholar?q=Yun+Wen%2C+Joshua+Snell%2C+and+Jonathan+Grainger%2E+2019%2E+Parallel%2C+cascaded%2C+interactive+processing+of+words+during+sentence+reading%2E+Cognition%2E)
97. Aaron Steven White, Rachel Rudinger, Kyle Rawlins, and Benjamin Van Durme. 2018. Lexicosyntactic inference in neural models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4717–4724, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Aaron+Steven+White%2C+Rachel+Rudinger%2C+Kyle+Rawlins%2C+and+Benjamin+Van+Durme%2E+2018%2E+Lexicosyntactic+inference+in+neural+models%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Langu)
98. Ethan Wilcox, Roger Levy, Takashi Morita, and Richard Futrell. 2018. [What do RNN language models learn about filler–gap dependencies?](https://md.research.elamur.ai/free/en/papers/wilcox-2018-rnn-language-models-learn-about/) In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 211–221, Brussels, Belgium. Association for Computati…
99. Adina Williams, Andrew Drozdov, and Samuel R. Bowman. 2018a. [Do latent tree learning models identify meaningful structure in sentences?](https://md.research.elamur.ai/free/en/papers/williams-2018-do-latent-tree-learning-models/) Transactions of the Association for Computational Linguistics, 6:253–267.
100. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018b. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long P… [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018b%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Cha)
101. Ludwig Wittgenstein. 1922. Tractatus Logico-Philosophicus. Harcourt, Brace & Company, Inc. [find](https://scholar.google.com/scholar?q=Ludwig+Wittgenstein%2E+1922%2E+Tractatus+Logico%2DPhilosophicus%2E+Harcourt%2C+Brace+%26+Company%2C+Inc%2E)
102. Thomas Wolf. 2019. [Some additional experiments extending the tech report” assessing berts syntactic abilities” by yoav goldberg](https://md.research.elamur.ai/free/en/papers/goldberg-2019-assessing-bert-s-syntactic-abilities/). Technical report, HuggingFace.
103. Thomas Wolf, Julien Chaumond, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pier-ric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, et al. 2020. [Transformers: State-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). In EMNLP: System Demonstrations.
104. Zhiyong Wu, Yun Chen, Ben Kao, and Qun Liu. 2020. [Perturbed masking: Parameter-free probing for analyzing and interpreting BERT](https://md.research.elamur.ai/free/en/papers/wu-2020-perturbed-masking-parameter-free-probing/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4166–4176, Online. Association for Computational Linguistics.
105. Kelly Zhang and Samuel Bowman. 2018. Language modeling teaches you more than translation does: Lessons learned through auxiliary syntactic task analysis. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 359–361, Brussels, Belgium. Assoc… [find](https://scholar.google.com/scholar?q=Kelly+Zhang+and+Samuel+Bowman%2E+2018%2E+Language+modeling+teaches+you+more+than+translation+does%3A+Lessons+learned+through+auxiliary+syntactic+task+analysis%2E+In+Proceedings+of+the+2018+EMNLP+Workshop+Blac)
106. Han Zhao, Zhengdong Lu, and Pascal Poupart. 2015. Self-adaptive hierarchical sentence model. In Proceedings of the 24th International Conference on Artificial Intelligence, pages 4069–4076. [find](https://scholar.google.com/scholar?q=Han+Zhao%2C+Zhengdong+Lu%2C+and+Pascal+Poupart%2E+2015%2E+Self%2Dadaptive+hierarchical+sentence+model%2E+In+Proceedings+of+the+24th+International+Conference+on+Artificial+Intelligence%2C+pages+4069%E2%80%934076%2E)
107. Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. 2020. Bert loses patience: Fast and robust inference with early exit. In Advances in Neural Information Processing Systems, volume 33, pages 18330–18341. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Wangchunshu+Zhou%2C+Canwen+Xu%2C+Tao+Ge%2C+Julian+McAuley%2C+Ke+Xu%2C+and+Furu+Wei%2E+2020%2E+Bert+loses+patience%3A+Fast+and+robust+inference+with+early+exit%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+vo)

*Updated 24 October 2026*