---
title: "BUAP — paper by León et al. (2014) — summary and ideas"
description: "The BUAP team took part in SemEval 2014 Task 1, which asks systems to judge… 0.697 Spearman. 10 references, cited by 2 papers here."
authors: ["Saúl León", "Darnes Vilariño", "David Pinto", "Mireya Tovar", "Beatriz Beltrán"]
year: 2014
venue: "Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)"
source: "https://aclanthology.org/S14-2021/"
doi: "10.3115/v1/s14-2021"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/leon-2014-buap-evaluating-compositional-distributional-semantic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/leon-2014-buap-evaluating-compositional-distributional-semantic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2014](https://md.research.elamur.ai/free/en/papers/2014/page/3/) / BUAP: Evaluating Compositional Distributional Semantic Models on Full Sentences through Semantic Relatedness and Textual Entailment

# BUAP: Evaluating Compositional Distributional Semantic Models on Full Sentences through Semantic Relatedness and Textual Entailment

**Authors:** Saúl León, Darnes Vilariño, David Pinto, Mireya Tovar and Beatriz Beltrán · **Year:** 2014 · **Venue:** [Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)](https://md.research.elamur.ai/free/en/venues/semeval/) ([SemEval 2014](https://md.research.elamur.ai/free/en/venues/semeval/2014/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/S14-2021/) · [PDF](https://aclanthology.org/S14-2021.pdf) · [DOI 10.3115/v1/s14-2021](https://doi.org/10.3115/v1/s14-2021) · [OpenAlex](https://openalex.org/W2139404310)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [2](https://md.research.elamur.ai/free/en/papers/leon-2014-buap-evaluating-compositional-distributional-semantic/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The BUAP team took part in SemEval 2014 Task 1, which asks systems to judge pairs of full sentences in two ways. The first is a graded relatedness score. The second is an entailment label. Their system turns each word into several kinds of evidence:
- dependency partners (object, subject, property);
- character n-grams and skip-grams;
- an LSA vector trained in R on a home-made corpus built from Europarl, Project Gutenberg and the Open Office Thesaurus.

For each of nine word-level similarity measures, every word of the first sentence is matched to its most similar word in the second sentence, and these best scores are summed. This yields a nine-dimensional feature vector per pair. Weka's logistic regression predicts relatedness. For entailment, the vector is extended with synonym, antonym and negation-cue flags and fed to an SVM.

Relatedness came out slightly below the task average, at rank 12 of 17. Entailment did better, at rank 7 of 18. The authors credit this gain to the lexical-relation and negation features. The work shows that cheap, hand-built similarity features can be competitive on sentence-pair tasks.

## Key points
- Features f1–f3 compare dependency partners (object/subject/property). Each takes the best of six NLTK WordNet metrics: Leacock & Chodorow, Lesk, Wu & Palmer, Resnik, Lin and Jiang & Conrath.
- Features f4–f5 apply cosine similarity to character n-grams (n=2–5) and skip-grams (n=2–5, skip r=2–5). Features f6–f9 compare LSA vectors using the Mihalcea measure, cosine, Euclidean distance and Chebyshev distance.
- Relatedness results with logistic regression (Weka defaults) were Pearson 0.697, Spearman 0.645 and MSE 0.528. The averages across all runs were 0.719, 0.672 and 0.639.
- For entailment, the SVM (Weka defaults) added several binary features:
  - WordNet synonyms;
  - antonyms collected from Wikipedia;
  - the negation cues "no", "not", "nobody" and "none".
- The entailment run placed 7th of 18 teams. The authors plan to reuse the entailment-only features for relatedness.

## Interesting ideas
- Scoring a sentence pair by greedily aligning each word to its best partner and summing gives an easy alignment-based similarity that can work with any word-level metric.
- Instead of committing to one WordNet metric, take the maximum over six of them, so whichever metric fits a given word pair wins.
- Comparing words through their dependency partners (subject, object, property) brings sentence structure into the similarity, beyond what lexical overlap captures.
- A handful of negation-cue and antonym flags noticeably improved entailment over purely similarity-based features, because related sentences can still contradict each other.
- The gap between the two subtasks acts as an informal ablation, isolating what the lexical-relation and negation features contribute.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (answer): Comparing a question with candidate answer sentences could use this greedy alignment, but over UD tree nodes rather than raw words, so each matched pair stays inspectable as an explanation.
- **I2** (world): When the WordNet-derived ontology is queried for concept similarity, an ensemble-max over several path- and information-content-based measures could be cached alongside the record of which measure produced the value.
- **I3** (frames): Frame matching between two texts could score role fillers (agent, patient, attribute) against each other separately, which mirrors these f1–f3 features in a frame-level form.
- **I4** (answer): Readers judging whether a statement follows from a fable should explicitly check polarity and antonymy between aligned frames before trusting a high similarity score.
- **I5** (learn): Evaluations could routinely run the same pipeline with and without a feature group on paired tasks, which treats the gap as evidence of what that group contributes.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

Also on the byline: [Benemérita Universidad Autónoma de Puebla](https://ror.org/03p2z7827).

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works

## Cite

Cite as: Saúl León, Darnes Vilariño, David Pinto, Mireya Tovar and Beatriz Beltrán. 2014. BUAP: Evaluating Compositional Distributional Semantic Models on Full Sentences through Semantic Relatedness and Textual Entailment. Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{leon2014buap,
  title = {{BUAP: Evaluating Compositional Distributional Semantic Models on Full Sentences through Semantic Relatedness and Textual Entailment}},
  author = {Saúl León, Darnes Vilariño, David Pinto, Mireya Tovar and Beatriz Beltrán},
  booktitle = {Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)},
  year = {2014},
  doi = {10.3115/v1/s14-2021},
  url = {https://doi.org/10.3115/v1/s14-2021}
}
```

- [← previous Bimorphisms and synchronous grammars](https://md.research.elamur.ai/free/en/papers/shieber-2014-bimorphisms-synchronous-grammars/)
- [next → MUHIT: A Multilingual Harmonized Dictionary](https://md.research.elamur.ai/free/en/papers/alansary-2014-muhit-multilingual-harmonized-dictionary/)
- [↑ back to papers of 2014](https://md.research.elamur.ai/free/en/papers/2014/page/3/)

## References

10 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jay J. Jiang and David W. Conrath. Semantic similarity based on corpus statistics and lexical taxonomy. In Proc of 10th International Conference on Research in Computational Linguistics, RO-CLING’97, pages 19–33, 1997. [↗](https://doi.org/10.48550/arxiv.cmp-lg/9709008)
2. Claudia Leacock and Martin Chodorow. Combining local context and wordnet similarity for word sense identification. In Christiane Fellfaum, editor, MIT Press, pages 265–283, 1998. [↗](https://doi.org/10.7551/mitpress/7287.003.0018)
3. Table 2: Results obtained at the substask “Textual Entailment” of the Semeval 2014 Task 1 [find](https://scholar.google.com/scholar?q=Table+2%3A+Results+obtained+at+the+substask+%E2%80%9CTextual+Entailment%E2%80%9D+of+the+Semeval+2014+Task+1)
4. TEAM ID Illinois-LH run1 ECNU run1 UNAL-NLP run1 SemantiKLUE run1 The Meaning Factory run1 CECL ALL run1 BUAP run1 UoW run1 CDT run1 UIO-Lien run1 FBK-TR run3 StanfordNLP run5 UTexas run1 Yamraj run1 asjai run5 haLF run2 CNGL run1 UANLPCourse run2 Overall average Our difference against the overall average Michael Lesk. Automatic sense disambiguation using machine readable dictionaries: How to tell a pine cone from an ice cream cone. In Proceedings of the 5th Annual International Conference on Systems Documentation, pages 24–26. ACM, 1986. [find](https://scholar.google.com/scholar?q=TEAM+ID+Illinois%2DLH+run1+ECNU+run1+UNAL%2DNLP+run1+SemantiKLUE+run1+The+Meaning+Factory+run1+CECL+ALL+run1+BUAP+run1+UoW+run1+CDT+run1+UIO%2DLien+run1+FBK%2DTR+run3+StanfordNLP+run5+UTexas+run1+Yamraj+run1+)
5. Dekang Lin. An information-theoretic definition of similarity. In Proceedings of the Fifteenth International Conference on Machine Learning, ICML ’98, pages 296–304, San Francisco, CA, USA, 1998. Morgan Kaufmann Publishers Inc. [find](https://scholar.google.com/scholar?q=Dekang+Lin%2E+An+information%2Dtheoretic+definition+of+similarity%2E+In+Proceedings+of+the+Fifteenth+International+Conference+on+Machine+Learning%2C+ICML+%E2%80%9998%2C+pages+296%E2%80%93304%2C+San+Francisco%2C+CA%2C+USA%2C+1998%2E+Morg)
6. Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. [Semeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment](https://md.research.elamur.ai/free/en/papers/marelli-2014-semeval-2014-task-1-evaluation/). In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval-2014), Dublin, Ireland, 2014a.
7. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. [A sick cure for the evaluation of compositional distributional semantic models](https://md.research.elamur.ai/free/en/papers/marelli-2014-sick-cure-evaluation-compositional-distributional/). In ACCURACY 84.575 83.641 83.053 82.322 81.591 79.988 79.663 78.526 77.106 77.004 75.401 74.488 73.229 70.753 69.758 69.413 67.201 48.731 75.358 4.31% Rank 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 11-12 - Proceedings of LREC 2014, Reykjavik, Iceland, 2014b.
8. Rada Mihalcea, Courtney Corley, and Carlo Strapparava. Corpus-based and knowledge-based measures of text semantic similarity. In Proceedings of the 21st National Conference on Artificial Intelligence, pages 775–780, 2006. [find](https://scholar.google.com/scholar?q=Rada+Mihalcea%2C+Courtney+Corley%2C+and+Carlo+Strapparava%2E+Corpus%2Dbased+and+knowledge%2Dbased+measures+of+text+semantic+similarity%2E+In+Proceedings+of+the+21st+National+Conference+on+Artificial+Intelligence%2C)
9. Philip Resnik. [Using information content to evaluate semantic similarity in a taxonomy](https://md.research.elamur.ai/free/en/papers/resnik-1995-information-content-evaluate-semantic-similarity/). In Proceedings of the 14th International Joint Conference on Artificial Intelligence, IJCAI’95, pages 448–453, San Francisco, CA, USA, 1995. [↗](https://doi.org/10.48550/arxiv.cmp-lg/9511007)
10. Zhibiao Wu and Martha Stone Palmer. [Verb semantics and lexical selection](https://md.research.elamur.ai/free/en/papers/wu-1994-verb-semantics-lexical-selection/). In Proceedings of the 32nd Annual Meeting of the Association for Computational Linguistics, pages 133–138, 1994. [↗](https://doi.org/10.48550/arxiv.cmp-lg/9406033)

*Updated 24 October 2026*