---
title: "QU-BIGIR at SemEval 2017 Task 3 — paper by Torki et al. (2017)"
description: "The paper addresses SemEval 2017 Task 3, Subtask D, which concerns Arabic community question… 7 references, cited by 1 paper here."
authors: ["Marwan Torki", "Maram Hasanain", "Tamer Elsayed"]
year: 2017
venue: "Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)"
source: "https://aclanthology.org/S17-2059/"
doi: "10.18653/v1/s17-2059"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/torki-2017-qu-bigir-semeval-2017-task/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/torki-2017-qu-bigir-semeval-2017-task/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/page/5/) / QU-BIGIR at SemEval 2017 Task 3: Using Similarity Features for Arabic Community Question Answering Forums

# QU-BIGIR at SemEval 2017 Task 3: Using Similarity Features for Arabic Community Question Answering Forums

**Authors:** [Marwan Torki](https://md.research.elamur.ai/free/en/authors/marwan-torki/), [Maram Hasanain](https://md.research.elamur.ai/free/en/authors/maram-hasanain/) and [Tamer Elsayed](https://md.research.elamur.ai/free/en/authors/tamer-elsayed/) · **Year:** 2017 · **Venue:** [Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)](https://md.research.elamur.ai/free/en/venues/semeval/) ([SemEval 2017](https://md.research.elamur.ai/free/en/venues/semeval/2017/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/S17-2059/) · [PDF](https://aclanthology.org/S17-2059.pdf) · [DOI 10.18653/v1/s17-2059](https://doi.org/10.18653/v1/s17-2059) · [OpenAlex](https://openalex.org/W2751124516)

**Citations:** 8 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper addresses SemEval 2017 Task 3, Subtask D, which concerns Arabic community question answering. A search engine retrieves 30 earlier question-answer pairs for each new question. The system must re-rank these pairs by relevance and label each one as relevant or irrelevant.

The Qatar University team, QU-BIGIR, frames this as supervised learning. A linear SVM is trained over a small set of pairwise similarity features. The lexical features are:
- synonym-aware overlap using Arabic WordNet,
- a Dirichlet-smoothed language-model score,
- TF-IDF cosine,
- three Jaccard variants.

The semantic features compare texts through the average of their word2vec vectors and through the covariance matrix of those vectors. Every feature is computed three ways: the new question against the old question, against the answer, and against both together. The SVM decision value serves as the ranking score, and its sign serves as the label.

The best run combined lexical and embedding features and reached MAP 59.48, placing second in the subtask. This shows that cheap, transparent similarity signals can compete in ranking tasks with limited resources.

## Key points
- Each new question comes with 30 candidate QApairs, and the system must both rank them and classify them as true or false.
- The feature set grows across runs:
  - Contrastive-1 uses 6 term-based features for each of 3 setups (QQ, QA, QQA), giving 18 features.
  - Contrastive-2 adds average-embedding cosine and covariance-embedding cosine, giving 24 features.
  - Primary additionally fuses scores with a separate SVM trained on covariance vectors.
- Background statistics come from ArabicWeb16, and the LM score uses µ = 2000, which is Lucene's default.
- The word2vec model was trained with Gensim (d = 100) on the task's own Arabic data, with test data held out.
- Official test MAP was 59.48 for Contrastive-2, 59.13 for Contrastive-1 and 56.69 for Primary. The IR baseline scored 60.55, though it does no labeling. Contrastive-2 also had the best accuracy (66.15).
- On the development set the Primary run was best (MAP 43.41 vs 42.87 and 42.54). The authors attribute its weaker test result to the test set containing only medical questions.

## Interesting ideas
- A text can be summarized by the covariance of its word vectors, not only by their mean, which gives a fixed-length second-order descriptor that is cheap to compute.
- The same similarity can be measured against three targets (the old question, the answer, and both joined) to reveal where the match actually lives.
- Asymmetric Jaccard variants divide the overlap by the size of one side only, turning similarity into directional coverage.
- Overlap counts a term as matched if the other text has it or a WordNet synonym of it, which is a simple and inspectable form of soft matching.
- A configuration that won on the development set lost on a test set that had shifted toward medical topics, which warns against tuning fusion weights on one domain.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Mova could use it as a strong bag-of-vectors negative control, so that structure-based answering must demonstrably beat it on the same items.
- **I2** (answer): when picking supporting passages for a fable question, Mova could score the question frame separately against the passage's events and against its stated outcomes, then report which one matched.
- **I3** (frames): Mova could measure what share of a question's frame slots are filled by a candidate frame, independently of how much extra the candidate contains.
- **I4** (world): when aligning question entities and predicates to the text's world, Mova could accept a lemma match through a shared WordNet synset and record that synset as the explanation.
- **I5** (learn): Mova's evaluation could hold out an entire genre, such as tales versus school tests, before accepting any tuned combination of components.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Qatar University](https://md.research.elamur.ai/free/en/universities/qatar-university/) — University, Doha, Qatar · 48 works · 1,026 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

🧭 Topics (2): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works · [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 441 works

## Cite

Cite as: Marwan Torki, Maram Hasanain and Tamer Elsayed. 2017. QU-BIGIR at SemEval 2017 Task 3: Using Similarity Features for Arabic Community Question Answering Forums. Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{torki2017qu,
  title = {{QU-BIGIR at SemEval 2017 Task 3: Using Similarity Features for Arabic Community Question Answering Forums}},
  author = {Marwan Torki, Maram Hasanain and Tamer Elsayed},
  booktitle = {Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)},
  year = {2017},
  doi = {10.18653/v1/s17-2059},
  url = {https://doi.org/10.18653/v1/s17-2059}
}
```

- [← previous Morphological Analysis for the Maltese Language: The challenges of a hybrid system](https://md.research.elamur.ai/free/en/papers/borg-2017-morphological-analysis-maltese-language-challenges/)
- [next → A discourse account of intervention phenomena: An investigation of interrogatives](https://md.research.elamur.ai/free/en/papers/riches-2017-discourse-account-intervention-phenomena-investigation/)
- [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/page/5/)

## References

7 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Our results justify the usage of SVM model for labeling and re-ranking question-answer Rana Malhas, Marwan Torki, and Tamer Elsayed. 2016. [QU-IR at SemEval 2016 Task 3: Learning to Rank on Arabic Community Question Answering Forums with Word Embedding](https://md.research.elamur.ai/free/en/papers/malhas-2016-qu-ir-semeval-2016-task/). In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016). Association for Computational Linguistics, San Diego, California, pages 866–871.
2. Donald Metzler and Tapas Kanungo. 2008. Machine learned sentence selection strategies for query-biased summarization. In SIGIR Learning to Rank Workshop. pages 40–47. [find](https://scholar.google.com/scholar?q=Donald+Metzler+and+Tapas+Kanungo%2E+2008%2E+Machine+learned+sentence+selection+strategies+for+query%2Dbiased+summarization%2E+In+SIGIR+Learning+to+Rank+Workshop%2E+pages+40%E2%80%9347%2E)
3. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. [Efficient estimation of word representations in vector space](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/). arXiv preprint arXiv:1301.3781 . [↗](https://arxiv.org/abs/1301.3781)
4. Preslav Nakov, Doris Hoogeveen, Lluı́s Màrquez, Alessandro Moschitti, Hamdy Mubarak, Timothy Baldwin, and Karin Verspoor. 2017. [SemEval-2017 task 3: Community question answering](https://md.research.elamur.ai/free/en/papers/nakov-2017-semeval-2017-task-3-community/). In Proceedings of the 11th International Workshop on Semantic Evaluation. Association for Computational Linguistics, Vancouver, Canada, SemEval ’17.
5. Anna Shtok, Gideon Dror, Yoelle Maarek, and Idan Szpektor. 2012. Learning from the past: answering new questions with past answers. In Proceedings of the 21st international conference on World Wide Web. ACM, pages 759–768. [find](https://scholar.google.com/scholar?q=Anna+Shtok%2C+Gideon+Dror%2C+Yoelle+Maarek%2C+and+Idan+Szpektor%2E+2012%2E+Learning+from+the+past%3A+answering+new+questions+with+past+answers%2E+In+Proceedings+of+the+21st+international+conference+on+World+Wide+We)
6. Reem Suwaileh, Mucahid Kutlu, Nihal Fathima, Tamer Elsayed, and Matthew Lease. 2016. Arabicweb16: A new crawl for today’s arabic web. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ’16, pages 673–676. [find](https://scholar.google.com/scholar?q=Reem+Suwaileh%2C+Mucahid+Kutlu%2C+Nihal+Fathima%2C+Tamer+Elsayed%2C+and+Matthew+Lease%2E+2016%2E+Arabicweb16%3A+A+new+crawl+for+today%E2%80%99s+arabic+web%2E+In+Proceedings+of+the+39th+International+ACM+SIGIR+Conference+on+R)
7. Liu Yang, Qingyao Ai, Damiano Spina, Ruey-Cheng Chen, Liang Pang, W. Bruce Croft, Jiafeng Guo, and Falk Scholer. 2016. Beyond factoid qa: Effective methods for non-factoid answer sentence retrieval. In Advances in Information Retrieval: 38th European Conference on IR Research, ECIR 2016, Padua, Italy, March 20–23, 2016. Proceedings, Springer International Publishing, pages 115– 128. [find](https://scholar.google.com/scholar?q=Liu+Yang%2C+Qingyao+Ai%2C+Damiano+Spina%2C+Ruey%2DCheng+Chen%2C+Liang+Pang%2C+W%2E+Bruce+Croft%2C+Jiafeng+Guo%2C+and+Falk+Scholer%2E+2016%2E+Beyond+factoid+qa%3A+Effective+methods+for+non%2Dfactoid+answer+sentence+retrieval%2E+I)

*Updated 31 October 2026*