---
title: "How we BLESSed distributional semantic — Baroni 2011"
description: "Benchmarks commonly used to test distributional semantic models (DSMs), such as TOEFL, WordSim 353 and concept categorization sets like AP, were mostly…"
authors: ["Marco Baroni", "Alessandro Lenci"]
year: 2011
venue: "Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language Semantics"
source: "https://aclanthology.org/W11-2501/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/baroni-2011-how-we-blessed-distributional-semantic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/baroni-2011-how-we-blessed-distributional-semantic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2011](https://md.research.elamur.ai/free/en/papers/2011/) / How we BLESSed distributional semantic evaluation

# How we BLESSed distributional semantic evaluation

**Authors:** [Marco Baroni](https://md.research.elamur.ai/free/en/authors/marco-baroni/), [Alessandro Lenci](https://md.research.elamur.ai/free/en/authors/alessandro-lenci/) · **Year:** 2011 · **Venue:** Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language Semantics · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W11-2501/) · [PDF](https://aclanthology.org/W11-2501.pdf) · [OpenAlex](https://openalex.org/W2135964261)

**Citations:** 271 citations (openalex-snapshot · 2026-10-14) · [154](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Benchmarks commonly used to test distributional semantic models (DSMs), such as TOEFL, WordSim 353 and concept categorization sets like AP, were mostly not built for this purpose. Each yields a single score that blends different semantic relations, can depend on outside components such as clustering algorithms, and gives little insight into how models differ. Baroni and Lenci propose BLESS, a data set for intrinsic evaluation. It consists of tuples linking a concrete, basic-level nominal concept to a relatum through an explicitly typed relation: coordinate, hypernym, meronym, attribute or event. It also includes controlled random relata (nouns, adjectives, verbs) for each concept. Models are compared by looking at the distributions of concept-relatum cosines, z-normalized concept by concept, for each relation type. Applied to several DSMs, the method shows clear profiles. Narrow content-word windows rank coordinates highest. A window over all words drifts toward syntactic similarity. Document-based co-occurrence separates related from random words but distinguishes relation types poorly. The approach exposes model behaviour that a single-number benchmark would hide.

## Key points
- The critique of existing benchmarks rests on several observations:
  - TOEFL has 80 multiple-choice items, so one extra correct answer shifts accuracy by 1.25%.
  - In WordSim 353, an annotation by Agirre et al. found that 92 of the 174 above-median pairs are only topically related.
  - AP has 402 concepts in 21 classes and includes rare items like casuarina.
- BLESS tuples have the form concept–relation–relatum (e.g., coyote-hyper-animal). The relation labels are COORD, HYPER, MERO, ATTRI and EVENT, plus RAN.N, RAN.J and RAN.V for random relata.
- Attribute and event relations are intentionally left underspecified, so the authors avoid committing to finer, debatable distinctions.
- The compared models are ContentWindow2, ContentWindow20, AllWindow2 and Document, plus two baselines, RelatumFrequency and RelationCardinality. For practical reasons, the Document model keeps 1 million dimensions selected by Local Mutual Information mass.
- Neither baseline reproduces the meaningful patterns, so those patterns are not artifacts of relatum frequency or relation cardinality.
- In AllWindow2, random nouns are statistically indistinguishable from hypernyms and meronyms. The authors attribute this to function-word contexts.

## Interesting ideas
- **Typed relations plus matched random distractors in one resource.** This lets you measure two things separately: whether a model finds related words at all, and which kinds of relations it prefers. The idea transfers to evaluating embeddings for any language, Ukrainian included.
- **Per-target z-normalization of similarity scores before pooling.** This removes differences in cosine scale between target words, so distributions over relation types become comparable across concepts and models.
- **Reporting full score distributions per relation, with significance tests between relation types.** This gives a behavioural profile of a model instead of one quantized number. It suits any similarity-based system.
- **Trivial baselines built from data-set properties (frequency, relation cardinality).** These check that observed patterns are not just artifacts of how the benchmark was built, which is a sanity check worth adding to any new evaluation set.
- **Underspecifying hard-to-define relation types.** Using broad labels like "event" or "attribute" avoids fragile fine-grained labels and keeps gold-standard annotation less controversial.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

154 papers of our library cite it (of the 140884 whose bibliographies we read); 8 of them have a page here, 146 not yet.

- [Entailment above the word level in distributional semantics](https://md.research.elamur.ai/free/en/papers/baroni-2012-entailment-above-word-level-distributional/)
- [Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn't.](https://md.research.elamur.ai/free/en/papers/gladkova-2016-analogy-based-detection-morphological-semantic/)
- [Do Supervised Distributional Methods Really Learn Lexical Inference Relations?](https://md.research.elamur.ai/free/en/papers/levy-2015-do-supervised-distributional-methods-really/)
- [On Quick Kisses and How to Make Them Count: A Study on Event Construal in Light Verb Constructions with BERT](https://md.research.elamur.ai/free/en/papers/liu-2023-quick-kisses-how-make-them/)
- [One Word Is Not Enough: Simple Prompts Improve Word Embeddings](https://md.research.elamur.ai/free/en/papers/ranjan-2026-one-word-is-not-enough/)
- [Integrating Predictions from Neural-Network Relation Classifiers into Coreference and Bridging Resolution](https://md.research.elamur.ai/free/en/papers/roesiger-2018-integrating-predictions-neural-network-relation/)
- [Improving Hypernymy Detection with an Integrated Path-based and Distributional Method](https://md.research.elamur.ai/free/en/papers/shwartz-2016-improving-hypernymy-detection-integrated-path/)
- [Learning to Distinguish Hypernyms and Co-Hyponyms](https://md.research.elamur.ai/free/en/papers/weeds-2014-learning-distinguish-hypernyms-co-hyponyms/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{baroni2011how,
  title = {{How we BLESSed distributional semantic evaluation}},
  author = {Baroni, Marco and Lenci, Alessandro},
  booktitle = {Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language Semantics},
  year = {2011},
  url = {https://aclanthology.org/W11-2501/}
}
```

- [← previous Helping Our Own: The HOO 2011 Pilot Shared Task](https://md.research.elamur.ai/free/en/papers/dale-2011-helping-our-own-hoo-2011/)
- [next → Identifying Sarcasm in Twitter: A Closer Look](https://md.research.elamur.ai/free/en/papers/ibanez-2011-identifying-sarcasm-twitter-closer-look/)
- [↑ back to papers of 2011](https://md.research.elamur.ai/free/en/papers/2011/)

## References

23 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Herv Abdi and Lynne Williams. 2010. Newman-Keuls and Tukey test. In N.J. Salkind, D.M. Dougherty, and B. Frey, editors, Encyclopedia of Research Design. Sage, Thousand Oaks, CA. [find](https://scholar.google.com/scholar?q=Herv+Abdi+and+Lynne+Williams%2E+2010%2E+Newman%2DKeuls+and+Tukey+test%2E+In+N%2EJ%2E+Salkind%2C+D%2EM%2E+Dougherty%2C+and+B%2E+Frey%2C+editors%2C+Encyclopedia+of+Research+Design%2E+Sage%2C+Thousand+Oaks%2C+CA%2E)
2. Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Pasça, and Aitor Soroa. 2009. A study on similarity and relatedness using distributional and WordNet-based approaches. In Proceedings of HLT-NAACL, pages 19–27, Boulder, CO. [find](https://scholar.google.com/scholar?q=Eneko+Agirre%2C+Enrique+Alfonseca%2C+Keith+Hall%2C+Jana+Kravalova%2C+Marius+Pasc%CC%A7a%2C+and+Aitor+Soroa%2E+2009%2E+A+study+on+similarity+and+relatedness+using+distributional+and+WordNet%2Dbased+approaches%2E+In+Proceedin) [our card](https://md.research.elamur.ai/free/en/papers/agirre-2009-study-similarity-relatedness-distributional-wordnet/)
3. Abdulrahman Almuhareb. 2006. Attributes in Lexical Acquisition. Phd thesis, University of Essex. [find](https://scholar.google.com/scholar?q=Abdulrahman+Almuhareb%2E+2006%2E+Attributes+in+Lexical+Acquisition%2E+Phd+thesis%2C+University+of+Essex%2E)
4. Marco Baroni and Alessandro Lenci. 2010. Distributional Memory: A general framework for corpus-based semantics. Computational Linguistics, 36(4):673–721. [find](https://scholar.google.com/scholar?q=Marco+Baroni+and+Alessandro+Lenci%2E+2010%2E+Distributional+Memory%3A+A+general+framework+for+corpus%2Dbased+semantics%2E+Computational+Linguistics%2C+36%284%29%3A673%E2%80%93721%2E) [our card](https://md.research.elamur.ai/free/en/papers/baroni-2010-distributional-memory-general-framework-corpus/)
5. Marco Baroni, Stefan Evert, and Alessandro Lenci, editors. 2008. Bridging the Gap between Semantic Theory and Computational Simulations: Proceedings of the ESSLLI Workshop on Distributional Lexical Semantic. FOLLI, Hamburg. [find](https://scholar.google.com/scholar?q=Marco+Baroni%2C+Stefan+Evert%2C+and+Alessandro+Lenci%2C+editors%2E+2008%2E+Bridging+the+Gap+between+Semantic+Theory+and+Computational+Simulations%3A+Proceedings+of+the+ESSLLI+Workshop+on+Distributional+Lexical+Se)
6. Marco Baroni, Eduard Barbu, Brian Murphy, and Massimo Poesio. 2010. Strudel: A distributional semantic model based on properties and types. Cognitive Science, 34(2):222–254. [find](https://scholar.google.com/scholar?q=Marco+Baroni%2C+Eduard+Barbu%2C+Brian+Murphy%2C+and+Massimo+Poesio%2E+2010%2E+Strudel%3A+A+distributional+semantic+model+based+on+properties+and+types%2E+Cognitive+Science%2C+34%282%29%3A222%E2%80%93254%2E)
7. Alexander Budanitsky and Graeme Hirst. 2006. Evaluating wordnet-based measures of lexical semantic relatedness. Computational Linguistics, 32:13–47. [find](https://scholar.google.com/scholar?q=Alexander+Budanitsky+and+Graeme+Hirst%2E+2006%2E+Evaluating+wordnet%2Dbased+measures+of+lexical+semantic+relatedness%2E+Computational+Linguistics%2C+32%3A13%E2%80%9347%2E) [our card](https://md.research.elamur.ai/free/en/papers/budanitsky-2006-evaluating-wordnet-based-measures-lexical/)
8. D. A. Cruse. 1986. Lexical Semantics. Cambridge University Press, Cambridge. [find](https://scholar.google.com/scholar?q=D%2E+A%2E+Cruse%2E+1986%2E+Lexical+Semantics%2E+Cambridge+University+Press%2C+Cambridge%2E)
9. Stefan Evert. 2005. The Statistics of Word Cooccurrences. Dissertation, Stuttgart University. [find](https://scholar.google.com/scholar?q=Stefan+Evert%2E+2005%2E+The+Statistics+of+Word+Cooccurrences%2E+Dissertation%2C+Stuttgart+University%2E)
10. Christiane Fellbaum, editor. 1998. WordNet: An Electronic Lexical Database. MIT Press, Cambridge, MA. [find](https://scholar.google.com/scholar?q=Christiane+Fellbaum%2C+editor%2E+1998%2E+WordNet%3A+An+Electronic+Lexical+Database%2E+MIT+Press%2C+Cambridge%2C+MA%2E)
11. Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2002. Placing search in context: The concept revisited. ACM Transactions on Information Systems, 20(1):116–131. [find](https://scholar.google.com/scholar?q=Lev+Finkelstein%2C+Evgeniy+Gabrilovich%2C+Yossi+Matias%2C+Ehud+Rivlin%2C+Zach+Solan%2C+Gadi+Wolfman%2C+and+Eytan+Ruppin%2E+2002%2E+Placing+search+in+context%3A+The+concept+revisited%2E+ACM+Transactions+on+Information+Sys)
12. Thomas Landauer and Susan Dumais. 1997. A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104(2):211–240. [find](https://scholar.google.com/scholar?q=Thomas+Landauer+and+Susan+Dumais%2E+1997%2E+A+solution+to+Plato%E2%80%99s+problem%3A+The+latent+semantic+analysis+theory+of+acquisition%2C+induction%2C+and+representation+of+knowledge%2E+Psychological+Review%2C+104%282%29%3A211%E2%80%93)
13. Hugo Liu and Push Singh. 2004. ConceptNet: A practical commonsense reasoning toolkit. BT Technology Journal, pages 211–226. [find](https://scholar.google.com/scholar?q=Hugo+Liu+and+Push+Singh%2E+2004%2E+ConceptNet%3A+A+practical+commonsense+reasoning+toolkit%2E+BT+Technology+Journal%2C+pages+211%E2%80%93226%2E)
14. Kevin Lund and Curt Burgess. 1996. Producing high-dimensional semantic spaces from lexical co-occurrence. Behavior Research Methods, 28:203–208. [find](https://scholar.google.com/scholar?q=Kevin+Lund+and+Curt+Burgess%2E+1996%2E+Producing+high%2Ddimensional+semantic+spaces+from+lexical+co%2Doccurrence%2E+Behavior+Research+Methods%2C+28%3A203%E2%80%93208%2E)
15. Ken McRae, George Cree, Mark Seidenberg, and Chris McNorgan. 2005. Semantic feature production norms for a large set of living and nonliving things. Behavior Research Methods, 37(4):547–559. [find](https://scholar.google.com/scholar?q=Ken+McRae%2C+George+Cree%2C+Mark+Seidenberg%2C+and+Chris+McNorgan%2E+2005%2E+Semantic+feature+production+norms+for+a+large+set+of+living+and+nonliving+things%2E+Behavior+Research+Methods%2C+37%284%29%3A547%E2%80%93559%2E)
16. George Miller and Walter Charles. 1991. Contextual correlates of semantic similarity. Language and Cognitive Processes, 6(1):1–28. [find](https://scholar.google.com/scholar?q=George+Miller+and+Walter+Charles%2E+1991%2E+Contextual+correlates+of+semantic+similarity%2E+Language+and+Cognitive+Processes%2C+6%281%29%3A1%E2%80%9328%2E)
17. Gregory Murphy. 2002. The Big Book of Concepts. MIT Press, Cambridge, MA. [find](https://scholar.google.com/scholar?q=Gregory+Murphy%2E+2002%2E+The+Big+Book+of+Concepts%2E+MIT+Press%2C+Cambridge%2C+MA%2E)
18. Timothy Rogers and James McClelland. 2004. Semantic Cognition: A Parallel Distributed Processing Approach. MIT Press, Cambridge, MA. [find](https://scholar.google.com/scholar?q=Timothy+Rogers+and+James+McClelland%2E+2004%2E+Semantic+Cognition%3A+A+Parallel+Distributed+Processing+Approach%2E+MIT+Press%2C+Cambridge%2C+MA%2E)
19. Herbert Rubenstein and John Goodenough. 1965. Contextual correlates of synonymy. Communications of the ACM, 8(10):627–633. [find](https://scholar.google.com/scholar?q=Herbert+Rubenstein+and+John+Goodenough%2E+1965%2E+Contextual+correlates+of+synonymy%2E+Communications+of+the+ACM%2C+8%2810%29%3A627%E2%80%93633%2E)
20. Karen Sparck Jones and Julia R. Galliers. 1996. Evaluating Natural Language Processing Systems: An Analysis and Review. Springer Verlag, Berlin. [find](https://scholar.google.com/scholar?q=Karen+Sparck+Jones+and+Julia+R%2E+Galliers%2E+1996%2E+Evaluating+Natural+Language+Processing+Systems%3A+An+Analysis+and+Review%2E+Springer+Verlag%2C+Berlin%2E)
21. Peter Turney and Patrick Pantel. 2010. From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37:141–188. [find](https://scholar.google.com/scholar?q=Peter+Turney+and+Patrick+Pantel%2E+2010%2E+From+frequency+to+meaning%3A+Vector+space+models+of+semantics%2E+Journal+of+Artificial+Intelligence+Research%2C+37%3A141%E2%80%93188%2E)
22. Peter Turney. 2006. Similarity of semantic relations. Computational Linguistics, 32(3):379–416. [find](https://scholar.google.com/scholar?q=Peter+Turney%2E+2006%2E+Similarity+of+semantic+relations%2E+Computational+Linguistics%2C+32%283%29%3A379%E2%80%93416%2E)
23. Morton E. Winston, Roger Chaffin, and Douglas Herrmann. 1987. A taxonomy of part-whole relations. Cognitive Science, 11:417–444. [find](https://scholar.google.com/scholar?q=Morton+E%2E+Winston%2C+Roger+Chaffin%2C+and+Douglas+Herrmann%2E+1987%2E+A+taxonomy+of+part%2Dwhole+relations%2E+Cognitive+Science%2C+11%3A417%E2%80%93444%2E)