---
title: "UdL at SemEval-2017 Task 1 — paper by Al-Natsheh et al. (2017)"
description: "The paper describes UdL, a system for Track 5 of SemEval-2017 Task 1. 7 references, cited by 1 paper here."
authors: ["Hussein T. Al-Natsheh", "Lucie Martinet", "Fabrice Muhlenbach", "Djamel Abdelkader Zighed"]
year: 2017
venue: "Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)"
source: "https://aclanthology.org/S17-2013/"
doi: "10.18653/v1/s17-2013"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/natsheh-2017-udl-semeval-2017-task-1/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/natsheh-2017-udl-semeval-2017-task-1/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/page/5/) / UdL at SemEval-2017 Task 1: Semantic Textual Similarity Estimation of English Sentence Pairs Using Regression Model over Pairwise Features

# UdL at SemEval-2017 Task 1: Semantic Textual Similarity Estimation of English Sentence Pairs Using Regression Model over Pairwise Features

**Authors:** [Hussein T. Al-Natsheh](https://md.research.elamur.ai/free/en/authors/hussein-t-al-natsheh/), Lucie Martinet, Fabrice Muhlenbach and Djamel Abdelkader Zighed · **Year:** 2017 · **Venue:** [Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)](https://md.research.elamur.ai/free/en/venues/semeval/) ([SemEval 2017](https://md.research.elamur.ai/free/en/venues/semeval/2017/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/S17-2013/) · [PDF](https://aclanthology.org/S17-2013.pdf) · [DOI 10.18653/v1/s17-2013](https://doi.org/10.18653/v1/s17-2013) · [OpenAlex](https://openalex.org/W2752127475)

**Citations:** 12 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/natsheh-2017-udl-semeval-2017-task-1/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper describes UdL, a system for Track 5 of SemEval-2017 Task 1. The track asks for a 0–5 semantic similarity score for each English sentence pair, and systems are scored by Pearson correlation against hidden human ratings on 250 test pairs. The authors trained on the STS datasets released since 2012, plus aggregated data shared by an earlier team. The pipeline has two stages. A pairwise transformer first turns each sentence pair into a small set of comparison features: word-vector alignment of tokens sharing the same PoS or NE tag, cosine similarity of tf–idf vectors built from character n-grams, a numeric-content difference, a length difference, and an optional domain label. A random forest regressor then maps these features to a score. The best run reached 0.8004, above the track baseline (0.7278) and median (0.7775). The authors present the comparator-style setup as a first step toward learning sentence representations from pairs. They also report that the shallow character n-gram feature carried most of the model's predictive weight.

## Key points
- **Features and their importance in the random forest:**
  - Aligned PoS tags (17 tags): 0.113
  - Aligned NE tags (10 tags): 0.003
  - tf–idf char 2–3-gram BoW cosine: 0.847
  - Number-sum difference: 0.006
  - Character-length difference: 0.032
- **Regressor choice:** Random forest (1024 trees, max depth 8) beat Lasso CV (512 iterations) in development, with a mean of 0.81 versus 0.79 on images, answers-students and headlines 2016.
- **Official runs:**
  - Run1 (all features except domain class): 0.8004
  - Run2 (with the domain feature, all test pairs labeled "Image-captions"): 0.7805
  - Run3 (hashtags removed from tweet-news training pairs): 0.7901
  - Best system in the track: 0.8547
- **Tooling:** polyglot and spaCy for tagging, 300-dimensional GloVe and spaCy vectors, a scikit-learn pipeline, and code released at github.com/natsheh/sensim.
- **Data cleaning:** Tab separators that also appeared inside sentences required manual repair. A spell-correction step with Enchant was tested but not used in the submissions.

## Interesting ideas
- Tokens are aligned only when they carry the same PoS or NE tag and their vector similarity passes a threshold, with competing matches settled by the higher score; restricting matches by category keeps the comparison structured rather than a loose overlap of words.
- Spelled-out numbers are first converted to numeric values (e.g. "sixty-five" to 65), and the pair is then compared on numeric content as its own signal.
- The feature-importance analysis revealed that a shallow character n-gram cosine dominated the more linguistic features, which shows how easily surface overlap can explain most of a benchmark score.
- Learning a comparator over pairs is proposed as a route to good single-item representations, much as face verification is trained on same/different pairs, because pairs are far more plentiful than labeled examples.
- Adding a domain feature whose test value had to be guessed by hand lowered the score (0.7805 vs 0.8004), a reminder that metadata unavailable or unreliable at test time can hurt more than help.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (frames): Two frames could be compared by aligning their arguments only within matching semantic roles, with conflicts resolved greedily by similarity, to give an explainable paraphrase check.
- **I2** (world): Quantities in the world of a text could be normalized to values before comparison, which helps consistency checks in maths word problems and fable QA.
- **I3** (learn): A char n-gram similarity baseline could serve as a negative control, so that structured tree/frame comparisons must show gains beyond what surface overlap already predicts.
- **I4** (learn): Self-play could generate many sentence pairs (paraphrase versus perturbed variant) to tune which tree and frame features matter for meaning equivalence.
- **I5** (answer): Readers should not depend on genre or source tags that will be missing or guessed at inference, or should be evaluated with those tags withheld.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Centre National de la Recherche Scientifique](https://md.research.elamur.ai/free/en/labs/centre-national-de-la-recherche-scientifique/) — Research institute, Paris, France · 169 works · 62,488 citations
- [Université de Lyon](https://md.research.elamur.ai/free/en/universities/universite-de-lyon/) — University, Lyon, France · 4 works · 372 citations
- [Laboratoire Hubert Curien](https://md.research.elamur.ai/free/en/labs/laboratoire-hubert-curien/) — Research institute, Saint-Etienne, France · 3 works · 364 citations
- [Université Lumière Lyon 2](https://md.research.elamur.ai/free/en/universities/universite-lumiere-lyon-2/) — University, Lyon, France · 3 works · 46 citations
- [Université Jean Monnet](https://md.research.elamur.ai/free/en/universities/universite-jean-monnet/) — University, Saint-Etienne, France · 2 works · 14 citations

Also on the byline: [Entrepôts, Représentation et Ingénierie des Connaissances](https://ror.org/009frb846) · [Centre d'Etudes Superieures Industrielles](https://ror.org/04f87d812).

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,379 works

## Cite

Cite as: Hussein T. Al-Natsheh, Lucie Martinet, Fabrice Muhlenbach and Djamel Abdelkader Zighed. 2017. UdL at SemEval-2017 Task 1: Semantic Textual Similarity Estimation of English Sentence Pairs Using Regression Model over Pairwise Features. Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{alnatsheh2017udl,
  title = {{UdL at SemEval-2017 Task 1: Semantic Textual Similarity Estimation of English Sentence Pairs Using Regression Model over Pairwise Features}},
  author = {Hussein T. Al-Natsheh, Lucie Martinet, Fabrice Muhlenbach and Djamel Abdelkader Zighed},
  booktitle = {Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)},
  year = {2017},
  doi = {10.18653/v1/s17-2013},
  url = {https://doi.org/10.18653/v1/s17-2013}
}
```

- [← previous The lexical pragmatics of count-mass polysemy](https://md.research.elamur.ai/free/en/papers/falkum-2017-lexical-pragmatics-count-mass-polysemy/)
- [next → A non-projective greedy dependency parser with bidirectional LSTMs](https://md.research.elamur.ai/free/en/papers/vilares-2017-non-projective-greedy-dependency-parser/)
- [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/page/5/)

## References

7 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. [Semeval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation](https://md.research.elamur.ai/free/en/papers/agirre-2016-semeval-2016-task-1-semantic/). In Steven Bethard, Daniel Cer, Marine Carpuat, David Jurgens, Preslav Nakov, and Torsten Zesch, editors, Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016). Association for Computational Linguistics, San Diego, California, pages 497–511.
2. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. [Glove: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In Alessandro Moschitti, Bo Pang, and Walter Daelemans, editors, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIG-DAT, a Special Interest Group of the ACL. ACL, pages 1532–1543.
3. Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2013. [Polyglot: Distributed word representations for multilingual NLP](https://md.research.elamur.ai/free/en/papers/rfou-2013-polyglot-distributed-word-representations-multilingual/). In Julia Hockenmaier and Sebastian Riedel, editors, Proceedings of the Seventeenth Conference on Computational Natural Language Learning, CoNLL 2013, Sofia, Bulgaria, August 8-9, 2013. ACL, pages 183–192. [↗](https://doi.org/10.48550/arxiv.1307.1662)
4. Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mesnil. 2014. A latent semantic model with convolutional-pooling structure for information retrieval. In Jianzhong Li, Xiaoyang Sean Wang, Minos N. Garofalakis, Ian Soboroff, Torsten Suel, and Min Wang, editors, Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, CIKM 2014, Shanghai, China, November 3-7, 2014. ACM, pages 101–110. [find](https://scholar.google.com/scholar?q=Yelong+Shen%2C+Xiaodong+He%2C+Jianfeng+Gao%2C+Li+Deng%2C+and+Gre%CC%81goire+Mesnil%2E+2014%2E+A+latent+semantic+model+with+convolutional%2Dpooling+structure+for+information+retrieval%2E+In+Jianzhong+Li%2C+Xiaoyang+Sean+Wang)
5. Léon Bottou. 2014. From machine learning to machine reasoning - an essay. Machine Learning 94(2):133– 149. [↗](https://doi.org/10.1007/s10994-013-5335-x)
6. Liling Tan, Carolina Scarton, Lucia Specia, and Josef van Genabith. 2015. USAAR-SHEFFIELD: semantic textual similarity with deep regression and machine translation evaluation metrics. In Proceedings of the 9th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2015, Denver, Colorado, USA, June 4-5, 2015. The Association for Computer Linguistics, pages 85–89. [find](https://scholar.google.com/scholar?q=Liling+Tan%2C+Carolina+Scarton%2C+Lucia+Specia%2C+and+Josef+van+Genabith%2E+2015%2E+USAAR%2DSHEFFIELD%3A+semantic+textual+similarity+with+deep+regression+and+machine+translation+evaluation+metrics%2E+In+Proceedings+o)
7. Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. [Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation](https://md.research.elamur.ai/free/en/papers/cer-2017-semeval-2017-task-1-semantic/). In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). Association for Computational Linguistics, Vancouver, Canada, pages 1–14. http://www.aclweb.org/anthology/S17-2001. [↗](https://aclanthology.org/S17-2001/)

*Updated 1 November 2026*