---
title: "WiC: the Word-in-Context Dataset for — Pilehvar 2019"
description: "Standard word embeddings give each word a single vector, so they cannot show that one word form can carry different meanings in different contexts."
authors: ["Mohammad Taher Pilehvar", "Jose Camacho-Collados"]
year: 2019
venue: "Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)"
source: "https://aclanthology.org/N19-1128/"
doi: "10.18653/v1/n19-1128"
arxiv: "1808.09121"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/pilehvar-2019-wic-word-context-dataset-evaluating/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pilehvar-2019-wic-word-context-dataset-evaluating/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/2/) / WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations

# WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations

**Authors:** [Mohammad Taher Pilehvar](https://md.research.elamur.ai/free/en/authors/mohammad-taher-pilehvar/), [Jose Camacho-Collados](https://md.research.elamur.ai/free/en/authors/jose-camacho-collados/) · **Year:** 2019 · **Venue:** Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/N19-1128/) · [PDF](https://aclanthology.org/N19-1128.pdf) · [DOI 10.18653/v1/n19-1128](https://doi.org/10.18653/v1/n19-1128) · [arXiv 1808.09121](https://arxiv.org/abs/1808.09121) · [OpenAlex](https://openalex.org/W2956105246)

**Citations:** 192 citations (OpenAlex, 2026-10-14) · [400](https://md.research.elamur.ai/free/en/papers/pilehvar-2019-wic-word-context-dataset-evaluating/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
Standard word embeddings give each word a single vector, so they cannot show that one word form can carry different meanings in different contexts. Sense embeddings and contextualized models such as ELMo and BERT were built to fix this, but there was almost no benchmark that tests context sensitivity directly. The only established option, Stanford Contextual Word Similarity (SCWS), has low inter-annotator agreement. Even a model that ignores context can beat its estimated human ceiling. The authors present WiC, a binary classification task. Each item gives one target noun or verb in two sentences, and the system must decide whether the word has the same sense in both. The sentences come from lexicographer examples in WordNet, VerbNet and Wiktionary, linked through BabelNet. Item pairs whose senses were too close to tell apart were removed automatically. Because both sentences contain the same word, a static embedding cannot beat chance. In tests, BERT large scored best, but only about 14 points above a random baseline and well below human accuracy. This makes WiC a demanding, freely available benchmark for context-sensitive meaning representations.

## Key points
- **Dataset size:** WiC has 5,428 training, 638 development and 1,400 test instances, each split evenly between positive and negative labels. No example sentence appears in more than one split, and only 36% of test target words also occur in training.
- **Sense pruning:** To limit WordNet's very fine sense distinctions, the authors dropped pairs whose senses were directly linked in the WordNet graph (including sister senses) or shared one of the 44 lexicographer supersenses.
- **Human performance:** Annotators who were not shown any sense inventory reached about 80% accuracy on the final data. On a sample of the pruned items they managed only 57%, which supports the pruning step.
- **Model results:** The best model, BERT large, scored roughly 64%, about 16 points below the human estimate. A simple cosine-threshold classifier beat an MLP. ELMo and Context2Vec did not beat a sentence bag-of-words baseline, and DeConf was the strongest multi-prototype model.
- **Problems with SCWS:** Pairwise annotator agreement on SCWS is only 0.35 Spearman ρ, and the mean agreement (the human ceiling) is 0.52. Google News Word2vec, which ignores context, reaches 0.65.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

Topics: [Semantic parsing & meaning representation](https://md.research.elamur.ai/free/en/topics/semantic-parsing/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{pilehvar2019wic,
  title = {{WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations}},
  author = {Pilehvar, Mohammad Taher and Camacho-Collados, Jose},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1128},
  eprint = {1808.09121},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/n19-1128}
}
```

- [← previous Attention Guided Graph Convolutional Networks for Relation Extraction](https://md.research.elamur.ai/free/en/papers/guo-2019-attention-guided-graph-convolutional-networks/)
- [next → Evaluating Gender Bias in Machine Translation](https://md.research.elamur.ai/free/en/papers/stanovsky-2019-evaluating-gender-bias-machine-translation/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/2/)

## References

29 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jose Camacho-Collados and Taher Pilehvar. 2018. From word to sense embeddings: A survey on vector representations of meaning. Journal of Artificial Intelligence Research, 63:743–788. [find](https://scholar.google.com/scholar?q=Jose+Camacho%2DCollados+and+Taher+Pilehvar%2E+2018%2E+From+word+to+sense+embeddings%3A+A+survey+on+vector+representations+of+meaning%2E+Journal+of+Artificial+Intelligence+Research%2C+63%3A743%E2%80%93788%2E)
2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL, Minneapolis, United States. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+NAACL%2C+Minneapolis%2C+United+S)
3. Haim Dubossarsky, Eitan Grossman, and Daphna Weinshall. 2018. Coming to your senses: on controls and evaluation sets in polysemy research. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1732–1740, Brussels, Belgium. [find](https://scholar.google.com/scholar?q=Haim+Dubossarsky%2C+Eitan+Grossman%2C+and+Daphna+Weinshall%2E+2018%2E+Coming+to+your+senses%3A+on+controls+and+evaluation+sets+in+polysemy+research%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in)
4. Christiane Fellbaum, editor. 1998. WordNet: An Electronic Database. MIT Press, Cambridge, MA. [find](https://scholar.google.com/scholar?q=Christiane+Fellbaum%2C+editor%2E+1998%2E+WordNet%3A+An+Electronic+Database%2E+MIT+Press%2C+Cambridge%2C+MA%2E)
5. Lucie Flekova and Iryna Gurevych. 2016. Supersense embeddings: A unified model for supersense interpretation, prediction, and utilization. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Lucie+Flekova+and+Iryna+Gurevych%2E+2016%2E+Supersense+embeddings%3A+A+unified+model+for+supersense+interpretation%2C+prediction%2C+and+utilization%2E+In+Proceedings+of+ACL%2E)
6. Felix Hill, Roi Reichart, and Anna Korhonen. 2015. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics. [find](https://scholar.google.com/scholar?q=Felix+Hill%2C+Roi+Reichart%2C+and+Anna+Korhonen%2E+2015%2E+Simlex%2D999%3A+Evaluating+semantic+models+with+%28genuine%29+similarity+estimation%2E+Computational+Linguistics%2E) [our card](https://md.research.elamur.ai/free/en/papers/hill-2015-simlex-999-evaluating-semantic-models/)
7. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A1735%E2%80%931780%2E)
8. Eric H. Huang, Richard Socher, Christopher D. Manning, and Andrew Y. Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of ACL, pages 873–882, Jeju Island, Korea. [find](https://scholar.google.com/scholar?q=Eric+H%2E+Huang%2C+Richard+Socher%2C+Christopher+D%2E+Manning%2C+and+Andrew+Y%2E+Ng%2E+2012%2E+Improving+word+representations+via+global+context+and+multiple+word+prototypes%2E+In+Proceedings+of+ACL%2C+pages+873%E2%80%93882%2C+Jej) [our card](https://md.research.elamur.ai/free/en/papers/huang-2012-improving-word-representations-global-context/)
9. Karin Kipper-Schuler. 2005. VerbNet: A broad-coverage, comprehensive verb lexicon. Ph.D. thesis. [find](https://scholar.google.com/scholar?q=Karin+Kipper%2DSchuler%2E+2005%2E+VerbNet%3A+A+broad%2Dcoverage%2C+comprehensive+verb+lexicon%2E+Ph%2ED%2E+thesis%2E)
10. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Ilya+Sutskever%2C+Kai+Chen%2C+Greg+S+Corrado%2C+and+Jeff+Dean%2E+2013b%2E+Distributed+representations+of+words+and+phrases+and+their+compositionality%2E+In+Advances+in+neural+information+processing)
11. Roberto Navigli. 2006. Meaningful clustering of senses helps boost Word Sense Disambiguation performance. In Proceedings of COLING-ACL, pages 105–112, Sydney, Australia. [find](https://scholar.google.com/scholar?q=Roberto+Navigli%2E+2006%2E+Meaningful+clustering+of+senses+helps+boost+Word+Sense+Disambiguation+performance%2E+In+Proceedings+of+COLING%2DACL%2C+pages+105%E2%80%93112%2C+Sydney%2C+Australia%2E)
12. Roberto Navigli and Simone Paolo Ponzetto. 2012. BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artificial Intelligence, 193:217– 250. [find](https://scholar.google.com/scholar?q=Roberto+Navigli+and+Simone+Paolo+Ponzetto%2E+2012%2E+BabelNet%3A+The+automatic+construction%2C+evaluation+and+application+of+a+wide%2Dcoverage+multilingual+semantic+network%2E+Artificial+Intelligence%2C+193%3A217%E2%80%93+25)
13. Arvind Neelakantan, Jeevan Shankar, Alexandre Passos, and Andrew McCallum. 2014. Efficient non-parametric estimation of multiple embeddings per word in vector space. In Proceedings of EMNLP, pages 1059–1069, Doha, Qatar. [find](https://scholar.google.com/scholar?q=Arvind+Neelakantan%2C+Jeevan+Shankar%2C+Alexandre+Passos%2C+and+Andrew+McCallum%2E+2014%2E+Efficient+non%2Dparametric+estimation+of+multiple+embeddings+per+word+in+vector+space%2E+In+Proceedings+of+EMNLP%2C+pages+105) [our card](https://md.research.elamur.ai/free/en/papers/neelakantan-2014-efficient-non-parametric-estimation-multiple/)
14. Maria Pelevina, Nikolay Arefyev, Chris Biemann, and Alexander Panchenko. 2016. Making sense of word embeddings. In Proceedings of the 1st Workshop on Representation Learning for NLP, pages 174–183. [find](https://scholar.google.com/scholar?q=Maria+Pelevina%2C+Nikolay+Arefyev%2C+Chris+Biemann%2C+and+Alexander+Panchenko%2E+2016%2E+Making+sense+of+word+embeddings%2E+In+Proceedings+of+the+1st+Workshop+on+Representation+Learning+for+NLP%2C+pages+174%E2%80%93183%2E)
15. M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL, New Orleans, LA, USA. [find](https://scholar.google.com/scholar?q=M%2E+E%2E+Peters%2C+M%2E+Neumann%2C+M%2E+Iyyer%2C+M%2E+Gardner%2C+C%2E+Clark%2C+K%2E+Lee%2C+and+L%2E+Zettlemoyer%2E+2018%2E+Deep+contextualized+word+representations%2E+In+Proceedings+of+NAACL%2C+New+Orleans%2C+LA%2C+USA%2E)
16. Mohammad Taher Pilehvar, Jose Camacho-Collados, Roberto Navigli, and Nigel Collier. 2017. Towards a Seamless Integration of Word Senses into Downstream NLP Applications. In Proceedings of ACL, Vancouver, Canada. [find](https://scholar.google.com/scholar?q=Mohammad+Taher+Pilehvar%2C+Jose+Camacho%2DCollados%2C+Roberto+Navigli%2C+and+Nigel+Collier%2E+2017%2E+Towards+a+Seamless+Integration+of+Word+Senses+into+Downstream+NLP+Applications%2E+In+Proceedings+of+ACL%2C+Vancouv)
17. Mohammad Taher Pilehvar and Nigel Collier. 2016. De-conflated semantic representations. In Proceedings of EMNLP, pages 1680–1690, Austin, TX. [find](https://scholar.google.com/scholar?q=Mohammad+Taher+Pilehvar+and+Nigel+Collier%2E+2016%2E+De%2Dconflated+semantic+representations%2E+In+Proceedings+of+EMNLP%2C+pages+1680%E2%80%931690%2C+Austin%2C+TX%2E)
18. Massimiliano Mancini, Jose Camacho-Collados, Ignacio Iacobacci, and Roberto Navigli. 2017. Embedding words and senses together via joint knowledge-enhanced training. In Proceedings of CoNLL, pages 100–111, Vancouver, Canada. [find](https://scholar.google.com/scholar?q=Massimiliano+Mancini%2C+Jose+Camacho%2DCollados%2C+Ignacio+Iacobacci%2C+and+Roberto+Navigli%2E+2017%2E+Embedding+words+and+senses+together+via+joint+knowledge%2Denhanced+training%2E+In+Proceedings+of+CoNLL%2C+pages+100)
19. Mohammad Taher Pilehvar, David Jurgens, and Roberto Navigli. 2013. Align, Disambiguate and Walk: a Unified Approach for Measuring Semantic Similarity. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 1341–1351, Sofia, Bulgaria. [find](https://scholar.google.com/scholar?q=Mohammad+Taher+Pilehvar%2C+David+Jurgens%2C+and+Roberto+Navigli%2E+2013%2E+Align%2C+Disambiguate+and+Walk%3A+a+Unified+Approach+for+Measuring+Semantic+Similarity%2E+In+Proceedings+of+the+51st+Annual+Meeting+of+the+)
20. Diana McCarthy, Marianna Apidianaki, and Katrin Erk. 2016. Word sense clustering and clusterability. Computational Linguistics. [find](https://scholar.google.com/scholar?q=Diana+McCarthy%2C+Marianna+Apidianaki%2C+and+Katrin+Erk%2E+2016%2E+Word+sense+clustering+and+clusterability%2E+Computational+Linguistics%2E)
21. Joseph Reisinger and Raymond J. Mooney. 2010. Multi-prototype vector-space models of word meaning. In Proceedings of ACL, pages 109–117. [find](https://scholar.google.com/scholar?q=Joseph+Reisinger+and+Raymond+J%2E+Mooney%2E+2010%2E+Multi%2Dprototype+vector%2Dspace+models+of+word+meaning%2E+In+Proceedings+of+ACL%2C+pages+109%E2%80%93117%2E) [our card](https://md.research.elamur.ai/free/en/papers/reisinger-2010-multi-prototype-vector-space-models/)
22. Oren Melamud, Jacob Goldberger, and Ido Dagan. 2016. context2vec: Learning generic context embedding with bidirectional lstm. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 51–61, Berlin, Germany. [find](https://scholar.google.com/scholar?q=Oren+Melamud%2C+Jacob+Goldberger%2C+and+Ido+Dagan%2E+2016%2E+context2vec%3A+Learning+generic+context+embedding+with+bidirectional+lstm%2E+In+Proceedings+of+The+20th+SIGNLL+Conference+on+Computational+Natural+Lang) [our card](https://md.research.elamur.ai/free/en/papers/melamud-2016-context2vec-learning-generic-context-embedding/)
23. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. CoRR, abs/1301.3781. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Kai+Chen%2C+Greg+Corrado%2C+and+Jeffrey+Dean%2E+2013a%2E+Efficient+estimation+of+word+representations+in+vector+space%2E+CoRR%2C+abs%2F1301%2E3781%2E)
24. Herbert Rubenstein and John B. Goodenough. 1965. Contextual correlates of synonymy. Communications of the ACM, 8(10):627–633. [find](https://scholar.google.com/scholar?q=Herbert+Rubenstein+and+John+B%2E+Goodenough%2E+1965%2E+Contextual+correlates+of+synonymy%2E+Communications+of+the+ACM%2C+8%2810%29%3A627%E2%80%93633%2E)
25. Stefan Rüd, Massimiliano Ciaramita, Jens Müller, and Hinrich Schütze. 2011. Piggyback: Using search engines for robust cross-domain named entity recognition. In Proceedings of ACL-HLT, pages 965–975, Portland, Oregon, USA. [find](https://scholar.google.com/scholar?q=Stefan+Ru%CC%88d%2C+Massimiliano+Ciaramita%2C+Jens+Mu%CC%88ller%2C+and+Hinrich+Schu%CC%88tze%2E+2011%2E+Piggyback%3A+Using+search+engines+for+robust+cross%2Ddomain+named+entity+recognition%2E+In+Proceedings+of+ACL%2DHLT%2C+pages+965%E2%80%9397)
26. Aliaksei Severyn, Massimo Nicosia, and Alessandro Moschitti. 2013. Learning semantic textual similarity with structural representations. In Proceedings of ACL (2), pages 714–718, Sofia, Bulgaria. [find](https://scholar.google.com/scholar?q=Aliaksei+Severyn%2C+Massimo+Nicosia%2C+and+Alessandro+Moschitti%2E+2013%2E+Learning+semantic+textual+similarity+with+structural+representations%2E+In+Proceedings+of+ACL+%282%29%2C+pages+714%E2%80%93718%2C+Sofia%2C+Bulgaria%2E)
27. Rion Snow, Sushant Prakash, Daniel Jurafsky, and Andrew Y. Ng. 2007. Learning to merge word senses. In Proceedings of EMNLP, pages 1005– 1014, Prague, Czech Republic. [find](https://scholar.google.com/scholar?q=Rion+Snow%2C+Sushant+Prakash%2C+Daniel+Jurafsky%2C+and+Andrew+Y%2E+Ng%2E+2007%2E+Learning+to+merge+word+senses%2E+In+Proceedings+of+EMNLP%2C+pages+1005%E2%80%93+1014%2C+Prague%2C+Czech+Republic%2E)
28. Yogarshi Vyas and Marine Carpuat. 2017. Detecting asymmetric semantic relations in context: A case-study on hypernymy detection. In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017), pages 33–43. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yogarshi+Vyas+and+Marine+Carpuat%2E+2017%2E+Detecting+asymmetric+semantic+relations+in+context%3A+A+case%2Dstudy+on+hypernymy+detection%2E+In+Proceedings+of+the+6th+Joint+Conference+on+Lexical+and+Computational)
29. George K. Zipf. 1949. Human Behaviour and the Principle of Least-Effort. Addison-Wesley, Cambridge, MA. [find](https://scholar.google.com/scholar?q=George+K%2E+Zipf%2E+1949%2E+Human+Behaviour+and+the+Principle+of+Least%2DEffort%2E+Addison%2DWesley%2C+Cambridge%2C+MA%2E)