---
title: "NSURL-2019 Task 8: Semantic Question Similarity in Arabic (2019)"
description: "This overview paper describes the shared task on Arabic question-to-question (Q2Q) semantic… 27 references, cited by 13 papers here."
authors: ["Haitham Seelawi", "Ahmad Mustafa", "Hesham Al-Bataineh", "Wael Farhan", "Hussein T. Al-Natsheh"]
year: 2019
venue: "Proceedings of the First International Workshop on NLP Solutions for Under Resourced Languages (NSURL 2019) co-located with ICNLSP 2019 - Short Papers"
source: "https://aclanthology.org/2019.nsurl-1.1/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/seelawi-2019-nsurl-2019-task-8-semantic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/seelawi-2019-nsurl-2019-task-8-semantic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/7/) / NSURL-2019 Task 8: Semantic Question Similarity in Arabic

# NSURL-2019 Task 8: Semantic Question Similarity in Arabic

**Authors:** [Haitham Seelawi](https://md.research.elamur.ai/free/en/authors/haitham-seelawi/), [Ahmad Mustafa](https://md.research.elamur.ai/free/en/authors/ahmad-mustafa/), [Hesham Al-Bataineh](https://md.research.elamur.ai/free/en/authors/hesham-al-bataineh/), [Wael Farhan](https://md.research.elamur.ai/free/en/authors/wael-farhan/) and [Hussein T. Al-Natsheh](https://md.research.elamur.ai/free/en/authors/hussein-t-al-natsheh/) · **Year:** 2019 · **Venue:** [Proceedings of the First International Workshop on NLP Solutions for Under Resourced Languages (NSURL 2019) co-located with ICNLSP 2019 - Short Papers](https://md.research.elamur.ai/free/en/venues/nsurl/) ([International Workshop on NLP Solutions for Under Resourced Languages co-located with ICNLSP - Short Papers 2019](https://md.research.elamur.ai/free/en/venues/nsurl/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2019.nsurl-1.1/) · [PDF](https://aclanthology.org/2019.nsurl-1.1.pdf)

**Citations:** [13](https://md.research.elamur.ai/free/en/papers/seelawi-2019-nsurl-2019-task-8-semantic/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
This overview paper describes the shared task on Arabic question-to-question (Q2Q) semantic similarity, held at the NSURL 2019 workshop. Systems must decide whether two Arabic questions are semantically equivalent, even when they are worded differently. The organizers define equivalence through answers: two questions count as similar only when their answers fully coincide. Mawdoo3 annotators built a new labelled set of 15,712 question pairs for the task. The pairs are binary-labelled, roughly balanced between classes, and mix factoid and non-factoid questions. Nine teams took part, and four of them submitted system descriptions. The approaches ranged from character and word n-gram features with linear classifiers to Siamese recurrent networks over Word2Vec, ELMo and BERT. A BERT-based system won. The organizers argue that subword and character-aware contextual representations suit morphologically rich Arabic much better than whole-word embeddings. The released dataset fills a gap in Arabic semantic similarity resources.

## Key points
- The Mawdoo3 Q2Q dataset has 15,712 pairs. Of these, 11,997 are for training (5,397 similar, 6,600 not) and 3,715 for testing (1,718 similar, 1,997 not). No question is longer than 15 words.
- The labelling rule is answer-based. A pair gets label 1 only when both questions share the same answer, and pairs whose answers differ even partly get label 0.
- Top accuracies on the test set:
  - The Inception (BERT): 0.9592
  - Tha3aroon (ELMo with ON-LSTM and attention, trained on an augmented set of 45,514 samples): 0.9485
  - onekaggler (3 stacked BiGRUs over AraVec-initialized Word2Vec): 0.9481
  - Speech Translation (n-gram features with LSVM) came 7th at 0.8270.
- The onekaggler system scored 99% on validation but stayed below 94% on test, a clear generalization gap.
- Tha3aroon's design has three stages. A shared encoder produces a vector for each question. The two vectors are merged with a pairwise squared distance. Four dense layers then output a sigmoid prediction.

## Interesting ideas
- Defining similarity as "same answer" gives annotators a test they can actually check, which beats relying on intuition about paraphrase.
- A weight-shared encoder turns each question into a standalone vector that can be compared with a simple distance, so stored questions can be pre-encoded and searched at scale.
- The organizers attribute BERT and ELMo beating Word2Vec to morphology: whole-word embeddings must learn every inflected form separately and cannot share knowledge across variants of one root.
- The dataset deliberately mixes factoid and non-factoid questions and varies how lexically and semantically close the pairs are, so word overlap alone cannot solve the task.
- The large gap between validation and test accuracy for one strong team shows how unreliable a random validation split can be.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (answer): QA on fables could judge two questions equivalent when they resolve to the same entity or state in the world of the text, which also explains why they match.
- **I2** (frames): Frames could likewise be normalized once into a canonical who-did-what-to-whom form, so incoming questions are matched against cached frames by structural comparison rather than by re-reading the text.
- **I3** (tokens): This supports keeping lemma and morphological features explicit in crate en, so that later stages generalize over inflections instead of over surface forms.
- **I4** (learn): Evaluation sets could include pairs with high word overlap but different answers as negative controls, exposing any hidden bag-of-words behaviour.
- **I5** (learn): Mova should report results on a held-out set built separately from its silver data, so that apparent progress is not merely overfitting to the training distribution.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works

🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works

## Cite

Cite as: Haitham Seelawi, Ahmad Mustafa, Hesham Al-Bataineh, Wael Farhan and Hussein T. Al-Natsheh. 2019. NSURL-2019 Task 8: Semantic Question Similarity in Arabic. Proceedings of the First International Workshop on NLP Solutions for Under Resourced Languages (NSURL 2019) co-located with ICNLSP 2019 - Short Papers.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{seelawi2019nsurl,
  title = {{NSURL-2019 Task 8: Semantic Question Similarity in Arabic}},
  author = {Haitham Seelawi, Ahmad Mustafa, Hesham Al-Bataineh, Wael Farhan and Hussein T. Al-Natsheh},
  booktitle = {Proceedings of the First International Workshop on NLP Solutions for Under Resourced Languages (NSURL 2019) co-located with ICNLSP 2019 - Short Papers},
  year = {2019},
  url = {https://aclanthology.org/2019.nsurl-1.1/}
}
```

- [← previous Hateful People or Hateful Bots? Detection and Characterization of Bots Spreading Religious Hatred in Arabic Social Media](https://md.research.elamur.ai/free/en/papers/albadi-2019-hateful-people-hateful-bots-detection/)
- [next → QC-GO Submission for MADAR Shared Task: Arabic Fine-Grained Dialect Identification](https://md.research.elamur.ai/free/en/papers/samih-2019-qc-go-submission-madar-shared/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/7/)

## References

27 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Hesham Al-Bataineh, Wael Farhan, Ahmad Mustafa, Haitham Seelawi, and Hussein T Al-Natsheh. 2019. [Deep contextualized pairwise semantic similarity for arabic language questions](https://md.research.elamur.ai/free/en/papers/al-bataineh-2019-deep-contextualized-pairwise-semantic-similarity/). arXiv preprint arXiv:1909.09490. [↗](https://arxiv.org/abs/1909.09490)
2. Marwah Alian and Arafat Awajan. 2018. Arabic semantic similarity approaches–review. In The 19th International Arab Conference on Information Technology (ACIT’ 2018). [find](https://scholar.google.com/scholar?q=Marwah+Alian+and+Arafat+Awajan%2E+2018%2E+Arabic+semantic+similarity+approaches%E2%80%93review%2E+In+The+19th+International+Arab+Conference+on+Information+Technology+%28ACIT%E2%80%99+2018%29%2E)
3. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. [Universal sentence encoder](https://md.research.elamur.ai/free/en/papers/cer-2018-universal-sentence-encoder/). CoRR, abs/1803.11175. [↗](https://arxiv.org/abs/1803.11175)
4. Daniel M. Cer, Mona T. Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. [Semeval-2017 task 1: Semantic textual similarity - multilingual and cross-lingual focused evaluation](https://md.research.elamur.ai/free/en/papers/cer-2017-semeval-2017-task-1-semantic/). CoRR, abs/1708.00055. [↗](https://arxiv.org/abs/1708.00055)
5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805. [↗](https://arxiv.org/abs/1810.04805)
6. Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. 2017. [Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm](https://md.research.elamur.ai/free/en/papers/felbo-2017-millions-emoji-occurrences-learn-any/). arXiv preprint arXiv:1708.00524. [↗](https://arxiv.org/abs/1708.00524)
7. Hua He and Jimmy Lin. 2016. [Pairwise word interaction modeling with deep neural networks for semantic similarity measurement](https://md.research.elamur.ai/free/en/papers/he-2016-pairwise-word-interaction-modeling-deep/). In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 937–948.
8. Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 3294– 3302. [↗](https://doi.org/10.48550/arxiv.1506.06726)
9. Zhouhan Lin, Minwei Feng, Cı́cero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. [A structured self-attentive sentence embedding](https://md.research.elamur.ai/free/en/papers/lin-2017-structured-self-attentive-sentence-embedding/). CoRR, abs/1703.03130. ai.mawdoo3.com Goutam Majumder, Partha Pakray, Alexander F. Gelbukh, and David Pinto. 2016. Semantic textual similarity methods, tools, and applications: A survey. Computación y Sistemas, 20(4). [↗](https://arxiv.org/abs/1703.03130)
10. Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. [Distributed representations of words and phrases and their compositionality](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/). In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States., pages 3111– 3119. [↗](https://doi.org/10.48550/arxiv.1310.4546)
11. George A. Miller and Christiane Fellbaum. 2007. Wordnet then and now. Language Resources and Evaluation, 41(2):209–214. [↗](https://doi.org/10.1007/s10579-007-9044-6)
12. Jonas Mueller and Aditya Thyagarajan. 2016. Siamese recurrent architectures for learning sentence similarity. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA., pages 2786–2792. [find](https://scholar.google.com/scholar?q=Jonas+Mueller+and+Aditya+Thyagarajan%2E+2016%2E+Siamese+recurrent+architectures+for+learning+sentence+similarity%2E+In+Proceedings+of+the+Thirtieth+AAAI+Conference+on+Artificial+Intelligence%2C+February+12%2D17)
13. Preslav Nakov, Lluı́s Màrquez, Alessandro Moschitti, Walid Magdy, Hamdy Mubarak, Abed Alhakim Freihat, Jim Glass, and Bilal Randeree. 2016. [SemEval-2016 task 3: Community question answering](https://md.research.elamur.ai/free/en/papers/nakov-2016-semeval-2016-task-3-community/). In Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval ’16, San Diego, California. Association for Computational Linguistics.
14. Paul Neculoiu, Maarten Versteegh, and Mihai Rotaru. 2016. Learning text similarity with siamese recurrent networks. In Proceedings of the 1st Workshop on Representation Learning for NLP, Rep4NLP@ACL 2016, Berlin, Germany, August 11, 2016, pages 148–157. [find](https://scholar.google.com/scholar?q=Paul+Neculoiu%2C+Maarten+Versteegh%2C+and+Mihai+Rotaru%2E+2016%2E+Learning+text+similarity+with+siamese+recurrent+networks%2E+In+Proceedings+of+the+1st+Workshop+on+Representation+Learning+for+NLP%2C+Rep4NLP%40ACL+2)
15. Chris Olah and Shan Carter. 2016. Attention and augmented recurrent neural networks. Distill. [↗](https://doi.org/10.23915/distill.00001)
16. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. [Glove: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. [↗](https://doi.org/10.3115/v1/d14-1162)
17. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. [Deep contextualized word representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/). arXiv preprint arXiv:1802.05365. [↗](https://arxiv.org/abs/1802.05365)
18. Mohammad Taher Pilehvar, David Jurgens, and Roberto Navigli. 2013. Align, disambiguate and walk: A unified approach for measuring semantic similarity. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, ACL 2013, 4-9 August 2013, Sofia, Bulgaria, Volume 1: Long Papers, pages 1341–1351. [find](https://scholar.google.com/scholar?q=Mohammad+Taher+Pilehvar%2C+David+Jurgens%2C+and+Roberto+Navigli%2E+2013%2E+Align%2C+disambiguate+and+walk%3A+A+unified+approach+for+measuring+semantic+similarity%2E+In+Proceedings+of+the+51st+Annual+Meeting+of+the+)
19. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [Squad: 100, 000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). CoRR, abs/1606.05250. [↗](https://arxiv.org/abs/1606.05250)
20. Siva Reddy, Danqi Chen, and Christopher D. Manning. 2018. [Coqa: A conversational question answering challenge](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/). CoRR, abs/1808.07042. [↗](https://arxiv.org/abs/1808.07042)
21. Frane Saric, Goran Glavas, Mladen Karan, Jan Snajder, and Bojana Dalbelo Basic. 2012. [Takelab: Systems for measuring semantic text similarity](https://md.research.elamur.ai/free/en/papers/saric-2012-takelab-systems-measuring-semantic-text/). In Proceedings of the 6th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2012, Montréal, Canada, June 7-8, 2012, pages 441–448.
22. Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. 2018. Ordered neurons: Integrating tree structures into recurrent neural networks. arXiv preprint arXiv:1810.09536. [↗](https://arxiv.org/abs/1810.09536)
23. Gizem Sogancioglu, Hakime Öztürk, and Arzucan Özgür. 2017. BIOSSES: a semantic sentence similarity estimation system for the biomedical domain. Bioinformatics, 33(14):i49–i58. [find](https://scholar.google.com/scholar?q=Gizem+Sogancioglu%2C+Hakime+O%CC%88ztu%CC%88rk%2C+and+Arzucan+O%CC%88zgu%CC%88r%2E+2017%2E+BIOSSES%3A+a+semantic+sentence+similarity+estimation+system+for+the+biomedical+domain%2E+Bioinformatics%2C+33%2814%29%3Ai49%E2%80%93i58%2E)
24. Abu Bakr Soliman, Kareem Eissa, and Samhaa R. El-Beltagy. 2017. Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science, 117:256 – 265. Arabic Computational Linguistics. [find](https://scholar.google.com/scholar?q=Abu+Bakr+Soliman%2C+Kareem+Eissa%2C+and+Samhaa+R%2E+El%2DBeltagy%2E+2017%2E+Aravec%3A+A+set+of+arabic+word+embedding+models+for+use+in+arabic+nlp%2E+Procedia+Computer+Science%2C+117%3A256+%E2%80%93+265%2E+Arabic+Computational+Ling)
25. Radu Soricut and Eric Brill. 2004. Automatic question answering: Beyond the factoid. In Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, HLT-NAACL 2004, Boston, Massachusetts, USA, May 2-7, 2004, pages 57–64. [find](https://scholar.google.com/scholar?q=Radu+Soricut+and+Eric+Brill%2E+2004%2E+Automatic+question+answering%3A+Beyond+the+factoid%2E+In+Human+Language+Technology+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguist)
26. John Wieting and Douwe Kiela. 2019. No training required: Exploring random encoders for sentence classification. CoRR, abs/1901.10444. [↗](https://arxiv.org/abs/1901.10444)
27. Yinfei Yang, Steve Yuan, Daniel Cer, Sheng-yi Kong, Noah Constant, Petr Pilar, Heming Ge, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Learning semantic textual similarity from conversations. In Proceedings of The Third Workshop on Representation Learning for NLP, Rep4NLP@ACL 2018, Melbourne, Australia, July 20, 2018, pages 164–174. [↗](https://doi.org/10.48550/arxiv.1804.07754)

*Updated 1 November 2026*