---
title: "On the Systematicity of Probing Contextualized Word"
description: "Earlier probing work, notably Ettinger (2020), reported that BERT retrieves correct noun hypernyms in… 86 references, cited by 55 papers here."
authors: ["Abhilasha Ravichander", "Eduard Hovy", "Kaheer Suleman", "Adam Trischler", "Jackie Chi Kit Cheung"]
year: 2020
venue: "Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics"
source: "https://aclanthology.org/2020.starsem-1.10/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ravichander-2020-systematicity-probing-contextualized-word-representations/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ravichander-2020-systematicity-probing-contextualized-word-representations/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/6/) / On the Systematicity of Probing Contextualized Word Representations: The Case of Hypernymy in BERT

# On the Systematicity of Probing Contextualized Word Representations: The Case of Hypernymy in BERT

**Authors:** [Abhilasha Ravichander](https://md.research.elamur.ai/free/en/authors/abhilasha-ravichander/), [Eduard Hovy](https://md.research.elamur.ai/free/en/authors/eduard-hovy/), [Kaheer Suleman](https://md.research.elamur.ai/free/en/authors/kaheer-suleman/), [Adam Trischler](https://md.research.elamur.ai/free/en/authors/adam-trischler/) and [Jackie Chi Kit Cheung](https://md.research.elamur.ai/free/en/authors/jackie-chi-kit-cheung/) · **Year:** 2020 · **Venue:** [Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics](https://md.research.elamur.ai/free/en/venues/starsem/) ([\*SEM 2020](https://md.research.elamur.ai/free/en/venues/starsem/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.starsem-1.10/) · [PDF](https://aclanthology.org/2020.starsem-1.10.pdf) · [OpenAlex](https://openalex.org/W6948006532)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [55](https://md.research.elamur.ai/free/en/papers/ravichander-2020-systematicity-probing-contextualized-word-representations/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

Earlier probing work, notably Ettinger (2020), reported that BERT retrieves correct noun hypernyms in cloze prompts such as "A robin is a [MASK]". This was taken as evidence that BERT encodes hypernymy. Ravichander et al. ask whether such probe success reflects systematic knowledge or only template-specific behaviour. They test two kinds of generalization:

- **Syntagmatic:** the same fact should hold across different contexts. Grammatical-number consistency probes recast singular prompts in plural form, and contextual probes mask the hyponym inside Wikipedia sentences.
- **Paradigmatic:** knowledge should transfer to novel items. Classifiers trained on contextualized representations are evaluated on hyponym–hypernym pairs not seen in training.

The authors use the NEG-136 affirmative items and a larger WordNet-derived extension. BERT's hypernym predictions turn out to be fragile. Correct singular answers often fail to carry over to plural or in-context versions. Success appears tied to word frequency and to examples that occur in the training data, which suggests memorization rather than reasoning. The paper's main message is a warning: high probing accuracy does not show that a model understands a concept. Probing studies therefore need consistency checks and controls.

## Key points

- **LM DIAGNOSTIC:** 18 prompts from the affirmative part of Ettinger's NEG-136, built on Fischler et al. (1983) items. The hypernyms span nine Battig and Montague categories: three animal groups (insects, birds and fish), three plant groups (flowers, trees and vegetables), plus tools, vehicles and buildings.
- **LM DIAGNOSTIC EXTENDED:** 576 prompts. Hyponyms were drawn from WordNet and restricted to single-sense common nouns that are not named entities.
- **Contextual probes:** manually selected Wikipedia sentences that contain the hyponym but not the hypernym. Each sentence must permit the abstraction, select the correct category, and be upward-entailing.
- **Paradigmatic probing:** classifiers on frozen contextual representations, tested in SEEN (known hypernyms, new hyponyms) and UNSEEN (both new) settings with three-fold cross-validation.
- **Framing and resource:** the authors separate an "instrumentative" view (PLMs as knowledge stores) from an "agentive" view (PLMs as reasoners), arguing that systematicity matters for the agentive view. The diagnostic framework is released on GitHub as `probe-generalization`.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🔬 Interpretability](https://md.research.elamur.ai/free/en/topics/interpretability/) · 130 works · [📖 Lexical semantics & word embeddings](https://md.research.elamur.ai/free/en/topics/lexical-semantics/) · 63 works

## Cite

Cite as: Abhilasha Ravichander, Eduard Hovy, Kaheer Suleman, Adam Trischler and Jackie Chi Kit Cheung. 2020. On the Systematicity of Probing Contextualized Word Representations: The Case of Hypernymy in BERT. Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ravichander2020systematicity,
  title = {{On the Systematicity of Probing Contextualized Word Representations: The Case of Hypernymy in BERT}},
  author = {Abhilasha Ravichander, Eduard Hovy, Kaheer Suleman, Adam Trischler and Jackie Chi Kit Cheung},
  booktitle = {Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics},
  year = {2020},
  doi = {10.48448/sa1q-9g43},
  url = {https://doi.org/10.48448/sa1q-9g43}
}
```

- [← previous Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task](https://md.research.elamur.ai/free/en/papers/sellam-2020-learning-evaluate-translation-beyond-english/)
- [next → Paraphrase Generation as Zero-Shot Multilingual Translation: Disentangling Semantic Similarity from Lexical and Syntactic Diversity](https://md.research.elamur.ai/free/en/papers/thompson-2020-paraphrase-generation-as-zero-shot/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/6/)

## References

86 references, 43 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. [Fine-grained analysis of sentence embeddings using auxiliary prediction tasks](https://md.research.elamur.ai/free/en/papers/adi-2016-fine-grained-analysis-sentence-embeddings/). In International Conference on Learning Representations.
2. Elena Akhmatova and Mark Dras. 2009. Using hypernymy acquisition to tackle (part of) textual entailment. In Proceedings of the 2009 Workshop on Applied Textual Inference (TextInfer), pages 52–60, Suntec, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Elena+Akhmatova+and+Mark+Dras%2E+2009%2E+Using+hypernymy+acquisition+to+tackle+%28part+of%29+textual+entailment%2E+In+Proceedings+of+the+2009+Workshop+on+Applied+Textual+Inference+%28TextInfer%29%2C+pages+52%E2%80%9360%2C+Sunt)
3. Marco Baroni, Raffaella Bernardi, Ngoc-Quynh Do, and Chung-chieh Shan. 2012. [Entailment above the word level in distributional semantics](https://md.research.elamur.ai/free/en/papers/baroni-2012-entailment-above-word-level-distributional/). In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 23–32. Association for Computational Linguis…
4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
5. Marco Baroni and Alessandro Lenci. 2011. [How we blessed distributional semantic evaluation](https://md.research.elamur.ai/free/en/papers/baroni-2011-how-we-blessed-distributional-semantic/). In Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language Semantics, pages 1–10. Association for Computational Linguistics.
6. William F Battig and William E Montague. 1969. Category norms of verbal items in 56 categories a replication and extension of the connecticut category norms. Journal of experimental Psychology, 80(3p2):1. [find](https://scholar.google.com/scholar?q=William+F+Battig+and+William+E+Montague%2E+1969%2E+Category+norms+of+verbal+items+in+56+categories+a+replication+and+extension+of+the+connecticut+category+norms%2E+Journal+of+experimental+Psychology%2C+80%283p2)
7. Allyson Ettinger. 2020. [What bert is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://md.research.elamur.ai/free/en/papers/ettinger-2020-what-bert-is-not-lessons/). Transactions of the Association for Computational Linguistics, 8:34–48.
8. Or Biran and Kathleen McKeown. 2013. Classifying taxonomic relations between pairs of Wikipedia articles. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 788–794, Nagoya, Japan. Asian Federation of Natural Language Processing. [find](https://scholar.google.com/scholar?q=Or+Biran+and+Kathleen+McKeown%2E+2013%2E+Classifying+taxonomic+relations+between+pairs+of+Wikipedia+articles%2E+In+Proceedings+of+the+Sixth+International+Joint+Conference+on+Natural+Language+Processing%2C+pag)
9. Allyson Ettinger, Ahmed Elgohary, Colin Phillips, and Philip Resnik. 2018. Assessing composition in sentence vector representations. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1790–1801, Santa Fe, New Mexico, USA. Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Allyson+Ettinger%2C+Ahmed+Elgohary%2C+Colin+Phillips%2C+and+Philip+Resnik%2E+2018%2E+Assessing+composition+in+sentence+vector+representations%2E+In+Proceedings+of+the+27th+International+Conference+on+Computationa)
10. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. [Enriching word vectors with subword information](https://md.research.elamur.ai/free/en/papers/bojanowski-2017-enriching-word-vectors-subword-information/). Transactions of the Association for Computational Linguistics, 5:135–146.
11. Federico Fancellu, Adam Lopez, and Bonnie Webber. 2016. Neural networks for negation scope detection. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 495–504, Berlin, Germany. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Federico+Fancellu%2C+Adam+Lopez%2C+and+Bonnie+Webber%2E+2016%2E+Neural+networks+for+negation+scope+detection%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume)
12. Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Çelikyilmaz, and Yejin Choi. 2019. [Comet: Commonsense transformers for automatic knowledge graph construction](https://md.research.elamur.ai/free/en/papers/bosselut-2019-comet-commonsense-transformers-automatic-knowledge/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL).
13. Christiane Fellbaum. 1998a. Towards a representation of idioms in WordNet. In Usage of WordNet in Natural Language Processing Systems. [find](https://scholar.google.com/scholar?q=Christiane+Fellbaum%2E+1998a%2E+Towards+a+representation+of+idioms+in+WordNet%2E+In+Usage+of+WordNet+in+Natural+Language+Processing+Systems%2E)
14. Zied Bouraoui, Jose Camacho-Collados, and Steven Schockaert. 2019. Inducing relational knowledge from bert. [find](https://scholar.google.com/scholar?q=Zied+Bouraoui%2C+Jose+Camacho%2DCollados%2C+and+Steven+Schockaert%2E+2019%2E+Inducing+relational+knowledge+from+bert%2E)
15. Christiane Fellbaum. 1998b. WordNet: An Electronic Lexical Database. Bradford Books. [find](https://scholar.google.com/scholar?q=Christiane+Fellbaum%2E+1998b%2E+WordNet%3A+An+Electronic+Lexical+Database%2E+Bradford+Books%2E)
16. Andy Coenen, Emily Reif, Ann Yuan, Been Kim, Adam Pearce, Fernanda Viégas, and Martin Wattenberg. 2019. [Visualizing and measuring the geometry of bert](https://md.research.elamur.ai/free/en/papers/coenen-2019-visualizing-measuring-geometry-bert/).
17. Ira Fischler, Paul A Bloom, Donald G Childers, Salim E Roucos, and Nathan W Perry Jr. 1983. Brain potentials related to stages of sentence verification. Psychophysiology, 20(4):400–409. [find](https://scholar.google.com/scholar?q=Ira+Fischler%2C+Paul+A+Bloom%2C+Donald+G+Childers%2C+Salim+E+Roucos%2C+and+Nathan+W+Perry+Jr%2E+1983%2E+Brain+potentials+related+to+stages+of+sentence+verification%2E+Psychophysiology%2C+20%284%29%3A400%E2%80%93409%2E)
18. Alexis Conneau, German Kruszewski, Guillaume Lample, Loı̈c Barrault, and Marco Baroni. 2018. [What you can cram into a single $&!\#* vector: Probing sentence embeddings for linguistic properties](https://md.research.elamur.ai/free/en/papers/conneau-2018-what-you-can-cram-into/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long…
19. Maxwell Forbes, Ari Holtzman, and Yejin Choi. 2019. Do neural language representations learn physical commonsense? arXiv preprint arXiv:1908.02899. [find](https://scholar.google.com/scholar?q=Maxwell+Forbes%2C+Ari+Holtzman%2C+and+Yejin+Choi%2E+2019%2E+Do+neural+language+representations+learn+physical+commonsense%3F+arXiv+preprint+arXiv%3A1908%2E02899%2E)
20. Roxana Girju, Manju Putcha, and Dan Moldovan. 2003. Discovery of manner relations and their applicability to question answering. In Proceedings of the ACL 2003 Workshop on Multilingual Summarization and Question Answering, pages 54–60, Sapporo, Japan. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Roxana+Girju%2C+Manju+Putcha%2C+and+Dan+Moldovan%2E+2003%2E+Discovery+of+manner+relations+and+their+applicability+to+question+answering%2E+In+Proceedings+of+the+ACL+2003+Workshop+on+Multilingual+Summarization+a)
21. Jeff Da and Jungo Kasai. 2019. Cracking the contextual commonsense code: Understanding commonsense reasoning aptitude of deep contextual representations. In Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing, pages 1–12, Hong Kong, China. Association for Comput… [find](https://scholar.google.com/scholar?q=Jeff+Da+and+Jungo+Kasai%2E+2019%2E+Cracking+the+contextual+commonsense+code%3A+Understanding+commonsense+reasoning+aptitude+of+deep+contextual+representations%2E+In+Proceedings+of+the+First+Workshop+on+Common)
22. Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. [Breaking NLI systems with sentences that require simple lexical inferences](https://md.research.elamur.ai/free/en/papers/glockner-2018-breaking-nli-systems-sentences-that/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 650–655, Melbourne, Australia. Association for…
23. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment, pages 177– 190. Springer. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+2006%2E+The+pascal+recognising+textual+entailment+challenge%2E+In+Machine+learning+challenges%2E+evaluating+predictive+uncertainty%2C+visual+object+classificati)
24. Yoav Goldberg. 2019. [Assessing bert’s syntactic abilities](https://md.research.elamur.ai/free/en/papers/goldberg-2019-assessing-bert-s-syntactic-abilities/). arXiv preprint arXiv:1901.05287.
25. Omer Levy, Steffen Remus, Chris Biemann, and Ido Dagan. 2015. [Do supervised distributional methods really learn lexical inference relations?](https://md.research.elamur.ai/free/en/papers/levy-2015-do-supervised-distributional-methods-really/) In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 970–97…
26. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A Smith. 2018. [Annotation artifacts in natural language inference data](https://md.research.elamur.ai/free/en/papers/gururangan-2018-annotation-artifacts-natural-language-inference/). arXiv preprint arXiv:1803.02324.
27. Marti A Hearst. 1992. [Automatic acquisition of hyponyms from large text corpora](https://md.research.elamur.ai/free/en/papers/hearst-1992-automatic-acquisition-hyponyms-large-text/). In Proceedings of the 14th conference on Computational linguistics-Volume 2, pages 539–545. Association for Computational Linguistics.
28. Dekang Lin. 1998. An information-theoretic definition of similarity. In Proceedings of the Fifteenth International Conference on Machine Learning, pages 296–304. [find](https://scholar.google.com/scholar?q=Dekang+Lin%2E+1998%2E+An+information%2Dtheoretic+definition+of+similarity%2E+In+Proceedings+of+the+Fifteenth+International+Conference+on+Machine+Learning%2C+pages+296%E2%80%93304%2E)
29. John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733–2743. [find](https://scholar.google.com/scholar?q=John+Hewitt+and+Percy+Liang%2E+2019%2E+Designing+and+interpreting+probes+with+control+tasks%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+the+9th+Internati)
30. Nelson F Liu, Matt Gardner, Yonatan Belinkov, Matthew E Peters, and Noah A Smith. 2019. [Linguistic knowledge and transferability of contextual representations](https://md.research.elamur.ai/free/en/papers/liu-2019-linguistic-knowledge-transferability-contextual-representations/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technol…
31. John Hewitt and Christopher D Manning. 2019. [A structural probe for finding syntax in word representations](https://md.research.elamur.ai/free/en/papers/hewitt-2019-structural-probe-finding-syntax-word/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129–…
32. Aman Madaan, Dheeraj Rajagopal, Yiming Yang, Abhilasha Ravichander, Eduard Hovy, and Shrimai Prabhumoye. 2020. Eigen: Event influence generation using pre-trained language models. [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Dheeraj+Rajagopal%2C+Yiming+Yang%2C+Abhilasha+Ravichander%2C+Eduard+Hovy%2C+and+Shrimai+Prabhumoye%2E+2020%2E+Eigen%3A+Event+influence+generation+using+pre%2Dtrained+language+models%2E)
33. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. [A SICK cure for the evaluation of compositional distributional semantic models](https://md.research.elamur.ai/free/en/papers/marelli-2014-sick-cure-evaluation-compositional-distributional/). In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014), page…
34. Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. 2018. Visualisation and’diagnostic classifiers’ reveal how recurrent and recursive neural networks process hierarchical structure. Journal of Artificial Intelligence Research, 61:907–926. [find](https://scholar.google.com/scholar?q=Dieuwke+Hupkes%2C+Sara+Veldhoen%2C+and+Willem+Zuidema%2E+2018%2E+Visualisation+and%E2%80%99diagnostic+classifiers%E2%80%99+reveal+how+recurrent+and+recursive+neural+networks+process+hierarchical+structure%2E+Journal+of+Artific)
35. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. [How can we know what language models know?](https://md.research.elamur.ai/free/en/papers/jiang-2020-how-can-we-know-what/) Transactions of the Association for Computational Linguistics, 8:423–438.
36. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. [Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy. Association for Computational Ling…
37. Nora Kassner and Hinrich Schütze. 2020. [Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly](https://md.research.elamur.ai/free/en/papers/kassner-2020-negated-misprimed-probes-pretrained-language/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7811–7818, Online. Association for Computational Linguistics.
38. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. [Distributed representations of words and phrases and their compositionality](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/). In Advances in neural information processing systems, pages 3111–3119.
39. Divyansh Kaushik and Zachary C. Lipton. 2018. [How much reading does reading comprehension require? a critical investigation of popular benchmarks](https://md.research.elamur.ai/free/en/papers/kaushik-2018-how-much-reading-does-reading/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5010–5015, Brussels, Belgium. Association for Computatio…
40. Shachar Mirkin, Lucia Specia, Nicola Cancedda, Ido Dagan, Marc Dymetman, and Idan Szpektor. 2009. Source-language entailment modeling for translating unknown terms. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Lang… [find](https://scholar.google.com/scholar?q=Shachar+Mirkin%2C+Lucia+Specia%2C+Nicola+Cancedda%2C+Ido+Dagan%2C+Marc+Dymetman%2C+and+Idan+Szpektor%2E+2009%2E+Source%2Dlanguage+entailment+modeling+for+translating+unknown+terms%2E+In+Proceedings+of+the+Joint+Confere)
41. Najoung Kim, Roma Patel, Adam Poliak, Patrick Xia, Alex Wang, Tom McCoy, Ian Tenney, Alexis Ross, Tal Linzen, Benjamin Van Durme, Samuel R. Bowman, and Ellie Pavlick. 2019. [Probing what different NLP tasks teach machines about function word comprehension](https://md.research.elamur.ai/free/en/papers/kim-2019-probing-what-different-nlp-tasks/). In Proceedings of the Eighth Joint Conferenc…
42. Michael Mohler, David Bracewell, Marc Tomlinson, and David Hinote. 2013. Semantic signatures for example-based linguistic metaphor detection. In Proceedings of the First Workshop on Metaphor in Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral t… [find](https://scholar.google.com/scholar?q=Michael+Mohler%2C+David+Bracewell%2C+Marc+Tomlinson%2C+and+David+Hinote%2E+2013%2E+Semantic+signatures+for+example%2Dbased+linguistic+metaphor+detection%2E+In+Proceedings+of+the+First+Workshop+on+Metaphor+in+Marco+)
43. NLP, pages 27–35, Atlanta, Georgia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=NLP%2C+pages+27%E2%80%9335%2C+Atlanta%2C+Georgia%2E+Association+for+Computational+Linguistics%2E)
44. Roser Morante and Caroline Sporleder. 2012. [Modality and negation: An introduction to the special issue](https://md.research.elamur.ai/free/en/papers/morante-2012-modality-negation-introduction-special-issue/). Computational Linguistics, 38(2):223–260.
45. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Associati… [find](https://scholar.google.com/scholar?q=Nasrin+Mostafazadeh%2C+Nathanael+Chambers%2C+Xiaodong+He%2C+Devi+Parikh%2C+Dhruv+Batra%2C+Lucy+Vanderwende%2C+Pushmeet+Kohli%2C+and+James+Allen%2E+2016%2E+A+corpus+and+cloze+evaluation+for+deeper+understanding+of+commo)
46. Laura Rimell. 2014. Distributional lexical entailment by topic coherence. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 511–519. [find](https://scholar.google.com/scholar?q=Laura+Rimell%2E+2014%2E+Distributional+lexical+entailment+by+topic+coherence%2E+In+Proceedings+of+the+14th+Conference+of+the+European+Chapter+of+the+Association+for+Computational+Linguistics%2C+pages+511%E2%80%93519%2E)
47. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. [A primer in bertology: What we know about how bert works](https://md.research.elamur.ai/free/en/papers/rogers-2020-primer-bertology-what-we-know/). arXiv preprint arXiv:2002.12327.
48. Stephen Roller, Katrin Erk, and Gemma Boleda. 2014. [Inclusive yet selective: Supervised distributional hypernymy detection](https://md.research.elamur.ai/free/en/papers/roller-2014-inclusive-yet-selective-supervised-distributional/). In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1025– 1036, Dublin, Ireland. Dublin City University and A…
49. Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340–2353, Santa Fe, New Mexico, USA. Association for Computation… [find](https://scholar.google.com/scholar?q=Aakanksha+Naik%2C+Abhilasha+Ravichander%2C+Norman+Sadeh%2C+Carolyn+Rose%2C+and+Graham+Neubig%2E+2018%2E+Stress+test+evaluation+for+natural+language+inference%2E+In+Proceedings+of+the+27th+International+Conference+o)
50. Eleanor Rosch and Barbara Bloom Lloyd. 1978. Cognition and categorization. [find](https://scholar.google.com/scholar?q=Eleanor+Rosch+and+Barbara+Bloom+Lloyd%2E+1978%2E+Cognition+and+categorization%2E)
51. Silvia Necsulescu. 2011. Automatic acquisition of possible contexts for low-frequent words. In Proceedings of the Second Student Research Workshop associated with RANLP 2011, pages 121–126, Hissar, Bulgaria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Silvia+Necsulescu%2E+2011%2E+Automatic+acquisition+of+possible+contexts+for+low%2Dfrequent+words%2E+In+Proceedings+of+the+Second+Student+Research+Workshop+associated+with+RANLP+2011%2C+pages+121%E2%80%93126%2C+Hissar%2C+Bu)
52. Enrico Santus, Alessandro Lenci, Tin-Shing Chiu, Qin Lu, and Chu-Ren Huang. 2016. Nine features in a random forest to learn taxonomical semantic relations. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 4557–4564, Portorož, Slovenia. Europ… [find](https://scholar.google.com/scholar?q=Enrico+Santus%2C+Alessandro+Lenci%2C+Tin%2DShing+Chiu%2C+Qin+Lu%2C+and+Chu%2DRen+Huang%2E+2016%2E+Nine+features+in+a+random+forest+to+learn+taxonomical+semantic+relations%2E+In+Proceedings+of+the+Tenth+International+Co)
53. Alessandro Oltramari, Aldo Gangemi, Nicola Guarino, and Claudio Masolo. Restructuring wordnet’s top-level: The ontoclean approach. [find](https://scholar.google.com/scholar?q=Alessandro+Oltramari%2C+Aldo+Gangemi%2C+Nicola+Guarino%2C+and+Claudio+Masolo%2E+Restructuring+wordnet%E2%80%99s+top%2Dlevel%3A+The+ontoclean+approach%2E)
54. Enrico Santus, Frances Yung, Alessandro Lenci, and Chu-Ren Huang. 2015. EVALution 1.0: an evolving semantic dataset for training and evaluation of distributional semantic models. In Proceedings of the 4th Workshop on Linked Data in Linguistics: Resources and Applications, pages 64–69, Beijing, China… [find](https://scholar.google.com/scholar?q=Enrico+Santus%2C+Frances+Yung%2C+Alessandro+Lenci%2C+and+Chu%2DRen+Huang%2E+2015%2E+EVALution+1%2E0%3A+an+evolving+semantic+dataset+for+training+and+evaluation+of+distributional+semantic+models%2E+In+Proceedings+of+the)
55. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. [Glove: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543.
56. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. [Language models as knowledge bases?](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/) In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural…
57. Ferdinand de Saussure. 1916. Course in general linguistics (trans. wade baskin). London: Fontana/Collins, page 74. [find](https://scholar.google.com/scholar?q=Ferdinand+de+Saussure%2E+1916%2E+Course+in+general+linguistics+%28trans%2E+wade+baskin%29%2E+London%3A+Fontana%2FCollins%2C+page+74%2E)
58. Xing Shi, Inkit Padhi, and Kevin Knight. 2016. [Does string-based neural mt learn source syntax?](https://md.research.elamur.ai/free/en/papers/shi-2016-does-string-based-neural-mt/) In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1526– 1534.
59. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. [Hypothesis only baselines in natural language inference](https://md.research.elamur.ai/free/en/papers/poliak-2018-hypothesis-only-baselines-natural-language/). In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180–191, New Orleans, Louisiana. Association for Compu…
60. Vered Shwartz and Ido Dagan. 2019. [Still a pain in the neck: Evaluating text representations on lexical composition](https://md.research.elamur.ai/free/en/papers/shwartz-2019-still-pain-neck-evaluating-text/). Transactions of the Association for Computational Linguistics, 7:403–419.
61. Vered Shwartz, Omer Levy, Ido Dagan, and Jacob Goldberger. 2015. Learning to exploit structured resources for lexical inference. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning, pages 175–184, Beijing, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Vered+Shwartz%2C+Omer+Levy%2C+Ido+Dagan%2C+and+Jacob+Goldberger%2E+2015%2E+Learning+to+exploit+structured+resources+for+lexical+inference%2E+In+Proceedings+of+the+Nineteenth+Conference+on+Computational+Natural+La)
62. John Prager, Jennifer Chu-Carroll, Eric W Brown, and Krzysztof Czuba. 2008. Question answering by predictive annotation. In Advances in Open Domain Question Answering, pages 307–347. Springer. [find](https://scholar.google.com/scholar?q=John+Prager%2C+Jennifer+Chu%2DCarroll%2C+Eric+W+Brown%2C+and+Krzysztof+Czuba%2E+2008%2E+Question+answering+by+predictive+annotation%2E+In+Advances+in+Open+Domain+Question+Answering%2C+pages+307%E2%80%93347%2E+Springer%2E)
63. Rion Snow, Daniel Jurafsky, and Andrew Y Ng. 2005. Learning syntactic patterns for automatic hypernym discovery. In Advances in neural information processing systems, pages 1297–1304. [find](https://scholar.google.com/scholar?q=Rion+Snow%2C+Daniel+Jurafsky%2C+and+Andrew+Y+Ng%2E+2005%2E+Learning+syntactic+patterns+for+automatic+hypernym+discovery%2E+In+Advances+in+neural+information+processing+systems%2C+pages+1297%E2%80%931304%2E)
64. Rion Snow, Daniel Jurafsky, and Andrew Y. Ng. 2006. Semantic taxonomy induction from heterogenous evidence. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 801–808, Sydney, Australia. Ass… [find](https://scholar.google.com/scholar?q=Rion+Snow%2C+Daniel+Jurafsky%2C+and+Andrew+Y%2E+Ng%2E+2006%2E+Semantic+taxonomy+induction+from+heterogenous+evidence%2E+In+Proceedings+of+the+21st+International+Conference+on+Computational+Linguistics+and+44th+An)
65. Swabha Swayamdipta, Matthew Peters, Brendan Roof, Chris Dyer, and Noah A Smith. 2019. Shallow syntax in deep water. arXiv preprint arXiv:1908.11047. — [arXiv:1908.11047](https://arxiv.org/abs/1908.11047) [find](https://scholar.google.com/scholar?q=Swabha+Swayamdipta%2C+Matthew+Peters%2C+Brendan+Roof%2C+Chris+Dyer%2C+and+Noah+A+Smith%2E+2019%2E+Shallow+syntax+in+deep+water%2E+arXiv+preprint+arXiv%3A1908%2E11047%2E+%E2%80%94+%5BarXiv%3A1908%2E11047%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1908%2E110)
66. Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2019. [olmpics – on what language model pre-training captures](https://md.research.elamur.ai/free/en/papers/talmor-2019-olmpics-language-model-pre-training/).
67. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. 2019. [What do you learn from context? probing for sentence structure in contextualized word representations](https://md.research.elamur.ai/free/en/papers/tenney-2019-learn-context-probing-sentence-structure/). arXiv preprint arXiv:1905.06316.
68. Masatoshi Tsuchiya. 2018. [Performance impact caused by hidden bias of training data for recognizing textual entailment](https://md.research.elamur.ai/free/en/papers/tsuchiya-2018-performance-impact-caused-hidden-bias/). In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018), Miyazaki, Japan. European Languages Resources Association (ELRA).
69. Amos Tversky. 1977. Features of similarity. Psychological review, 84(4):327. [find](https://scholar.google.com/scholar?q=Amos+Tversky%2E+1977%2E+Features+of+similarity%2E+Psychological+review%2C+84%284%29%3A327%2E)
70. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, pages 5998–6008.
71. Tu Vu and Vered Shwartz. 2018. Integrating multiplicative features into supervised distributional methods for lexical entailment. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 160–166, New Orleans, Louisiana. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tu+Vu+and+Vered+Shwartz%2E+2018%2E+Integrating+multiplicative+features+into+supervised+distributional+methods+for+lexical+entailment%2E+In+Proceedings+of+the+Seventh+Joint+Conference+on+Lexical+and+Computat)
72. Ivan Vulić and Nikola Mrkšić. 2018. [Specialising word vectors for lexical entailment](https://md.research.elamur.ai/free/en/papers/vulic-2018-specialising-word-vectors-lexical-entailment/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1134–1145, New Orleans, Louisiana.…
73. Yogarshi Vyas and Marine Carpuat. 2017. Detecting asymmetric semantic relations in context: A case-study on hypernymy detection. In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (* SEM 2017), pages 33–43. [find](https://scholar.google.com/scholar?q=Yogarshi+Vyas+and+Marine+Carpuat%2E+2017%2E+Detecting+asymmetric+semantic+relations+in+context%3A+A+case%2Dstudy+on+hypernymy+detection%2E+In+Proceedings+of+the+6th+Joint+Conference+on+Lexical+and+Computational)
74. Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019. [Do NLP models know numbers? probing numeracy in embeddings](https://md.research.elamur.ai/free/en/papers/wallace-2019-do-nlp-models-know-numbers/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing…
75. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,…
76. Alex Warstadt, Yu Cao, Ioana Grosu, Wei Peng, Hagen Blix, Yining Nie, Anna Alsop, Shikha Bordia, Haokun Liu, Alicia Parrish, Sheng-Fu Wang, Jason Phang, Anhad Mohananey, Phu Mon Htut, Paloma Jeretic, and Samuel R. Bowman. 2019. [Investigating BERT’s knowledge of language: Five analysis methods with N…](https://md.research.elamur.ai/free/en/papers/warstadt-2019-investigating-bert-s-knowledge-language/)
77. Julie Weeds, Daoud Clarke, Jeremy Reffin, David Weir, and Bill Keller. 2014. [Learning to distinguish hypernyms and co-hyponyms](https://md.research.elamur.ai/free/en/papers/weeds-2014-learning-distinguish-hypernyms-co-hyponyms/). In Proceedings of COL-ING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 2249–2259. Dublin City University and Association fo…
78. Julie Weeds and David Weir. 2003. [A general framework for distributional similarity](https://md.research.elamur.ai/free/en/papers/weeds-2003-general-framework-distributional-similarity/). In Proceedings of the 2003 conference on Empirical methods in natural language processing, pages 81–88.
79. Henry M Wellman and Susan A Gelman. 1992. Cognitive development: Foundational theories of core domains. Annual review of psychology, 43(1):337– 375. [find](https://scholar.google.com/scholar?q=Henry+M+Wellman+and+Susan+A+Gelman%2E+1992%2E+Cognitive+development%3A+Foundational+theories+of+core+domains%2E+Annual+review+of+psychology%2C+43%281%29%3A337%E2%80%93+375%2E)
80. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. [A broad-coverage challenge corpus for sentence understanding through inference](https://md.research.elamur.ai/free/en/papers/williams-2018-broad-coverage-challenge-corpus-sentence/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa…
81. Edwin Williams. 1981. On the notions” lexically related” and” head of a word”. Linguistic inquiry, 12(2):245–274. [find](https://scholar.google.com/scholar?q=Edwin+Williams%2E+1981%2E+On+the+notions%E2%80%9D+lexically+related%E2%80%9D+and%E2%80%9D+head+of+a+word%E2%80%9D%2E+Linguistic+inquiry%2C+12%282%29%3A245%E2%80%93274%2E)
82. Thomas Wolf. 2019. [Some additional experiments extending the tech report” assessing berts syntactic abilities” by yoav goldberg](https://md.research.elamur.ai/free/en/papers/goldberg-2019-assessing-bert-s-syntactic-abilities/). Technical report.
83. Mohamed Yahya, Klaus Berberich, Shady Elbassuoni, and Gerhard Weikum. 2013. Robust question answering over the web of linked data. In Proceedings of the 22nd ACM international conference on Conference on information & knowledge management, pages 1107–1116. ACM. [find](https://scholar.google.com/scholar?q=Mohamed+Yahya%2C+Klaus+Berberich%2C+Shady+Elbassuoni%2C+and+Gerhard+Weikum%2E+2013%2E+Robust+question+answering+over+the+web+of+linked+data%2E+In+Proceedings+of+the+22nd+ACM+international+conference+on+Conference)
84. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93– 104, Brussels, Belgium. Association for Computational Ling… [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Yonatan+Bisk%2C+Roy+Schwartz%2C+and+Yejin+Choi%2E+2018%2E+SWAG%3A+A+large%2Dscale+adversarial+dataset+for+grounded+commonsense+inference%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+)
85. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791– 4800, Florence, Italy. Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+HellaSwag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Com)
86. Fold \\\#Train \\\#Dev \\\#Test 1 2 3 6164 7936 5892 206 92 222 232 112 219 *Table 5: Statistics of UNSEEN dataset to study paradigmatic generalization.* Fold \\\#Train \\\#Dev \\\#Test 1 2 3 6682 6556 6582 176 182 164 148 144 154 *Table 6: Statistics of SEEN dataset to examine paradigmatic generalization.* [find](https://scholar.google.com/scholar?q=Fold+%5C%5C%5C%23Train+%5C%5C%5C%23Dev+%5C%5C%5C%23Test+1+2+3+6164+7936+5892+206+92+222+232+112+219+%2ATable+5%3A+Statistics+of+UNSEEN+dataset+to+study+paradigmatic+generalization%2E%2A+Fold+%5C%5C%5C%23Train+%5C%5C%5C%23Dev+%5C%5C%5C%23Test+1+2+3+6682+655)

*Updated 29 October 2026*