---
title: "QU-IR at SemEval 2016 Task 3 — paper by Malhas et al. (2016)"
description: "This system paper addresses answer re-ranking for Arabic community question answering in SemEval… 6 references, cited by 4 papers here."
authors: ["Rana Malhas", "Marwan Torki", "Tamer Elsayed"]
year: 2016
venue: "Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)"
source: "https://aclanthology.org/S16-1134/"
doi: "10.18653/v1/s16-1134"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/malhas-2016-qu-ir-semeval-2016-task/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/malhas-2016-qu-ir-semeval-2016-task/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/5/) / QU-IR at SemEval 2016 Task 3: Learning to Rank on Arabic Community Question Answering Forums with Word Embedding

# QU-IR at SemEval 2016 Task 3: Learning to Rank on Arabic Community Question Answering Forums with Word Embedding

**Authors:** [Rana Malhas](https://md.research.elamur.ai/free/en/authors/rana-malhas/), [Marwan Torki](https://md.research.elamur.ai/free/en/authors/marwan-torki/) and [Tamer Elsayed](https://md.research.elamur.ai/free/en/authors/tamer-elsayed/) · **Year:** 2016 · **Venue:** [Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)](https://md.research.elamur.ai/free/en/venues/semeval/) ([SemEval 2016](https://md.research.elamur.ai/free/en/venues/semeval/2016/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/S16-1134/) · [PDF](https://aclanthology.org/S16-1134.pdf) · [DOI 10.18653/v1/s16-1134](https://doi.org/10.18653/v1/s16-1134) · [OpenAlex](https://openalex.org/W2467216768)

**Citations:** 15 citations (OpenAlex, 2026-10-14) · [4](https://md.research.elamur.ai/free/en/papers/malhas-2016-qu-ir-semeval-2016-task/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
This system paper addresses answer re-ranking for Arabic community question answering in SemEval 2016 Task 3, Subtask D. For each newly posted question, a search engine returns 30 earlier question–answer pairs, and the task is to order them by relevance. The data were collected from Arabic medical forums (webteb, altibbi) and Consult Islamweb.

The authors pair the new question with each retrieved candidate to form a document. They train a word2vec model with Gensim on the training collection and describe each document by a dense vector. That vector is either the mean of its word embeddings or the flattened triangular half of the covariance matrix of those embeddings. Pointwise learning-to-rank models from RankLib (MART and Random Forests) then score the candidates.

The primary run reached MAP 38.63, placing fourth among primary entries and 29.7% above the 29.79 baseline. After the deadline, covariance features raised MAP to 41.07, a 37.9% gain. The work suggests that second-order statistics of embeddings hold relevance signal that plain averaging misses, at very low engineering cost.

## Key points
- The training data contain 1,031 questions with 30,411 candidate QApairs labelled Direct, Relevant or Irrelevant (about 3% / 57% / 40%). Development data contain 7,355 question–QApair items, after one malformed question was dropped.
- Three document setups were compared:
  - QQA: new question plus the full candidate pair.
  - QA: new question plus the answer only.
  - QQ: new question plus the retrieved question only.
- QQA worked better than QA. The contrastive-1 run used QA with Random Forests and scored the lowest MAP, 37.80.
- Pointwise L2Rank methods beat the pairwise and listwise ones the authors tried. The official runs scored 38.63 (MART, QQA), 37.80 and 39.07 (Random Forests, QQA, 80/20 train/validation split).
- CovWE-50 reached 41.07 and CovWE-100 reached 40.68. Training took about 44.5 minutes for CovWE-50 versus 5.27 hours for CovWE-100.
- Adding tf-idf vectors over the 5000 most frequent unigrams did not help. Feature normalization (z-score or linear) tended to raise MAP.
- To produce binary labels, the 10 top-ranked candidates were marked true and the rest false.

## Interesting ideas
- Pooling a text's word vectors through their covariance captures how dimensions vary together, not just their centre, and it beat the mean in this setting.
- Systematically varying which parts of a candidate are fed in (QQA vs QA vs QQ) reveals where the useful matching signal lives.
- With modest labelled data, a simple per-item (pointwise) scorer outperformed pairwise and listwise rankers.
- Halving the embedding size kept accuracy while cutting training time several-fold, so dimensionality should be tested rather than assumed.
- Turning a ranking into true/false labels with a fixed top-k cutoff is crude but transparent; the authors note a score threshold or a classifier as alternatives.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): such an orderless covariance descriptor makes a strong negative-control baseline, showing how far a structure-free reader gets before crediting the UD-tree-and-frames pipeline with any gain.
- **I2** (answer): when choosing supporting sentences for fable or school-test QA, compare matching against the question part versus the answer part to learn which evidence source the reader should weight.
- **I3** (answer): ranking candidate answers for test questions could start from an explainable per-candidate score before trying more complex comparative ranking.
- **I4** (tree): before scaling gparse's hidden or embedding sizes, run a smaller-dimension variant to check whether the extra cost actually buys LAS.
- **I5** (answer): when ranked options must become yes/no verdicts, an explicit, logged cutoff rule keeps the decision explainable and easy to audit.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Qatar University](https://md.research.elamur.ai/free/en/universities/qatar-university/) — University, Doha, Qatar · 53 works · 1,027 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works

🧭 Topics (3): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 3,066 works · [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 487 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 552 works

## Cite

Cite as: Rana Malhas, Marwan Torki and Tamer Elsayed. 2016. QU-IR at SemEval 2016 Task 3: Learning to Rank on Arabic Community Question Answering Forums with Word Embedding. Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{malhas2016qu,
  title = {{QU-IR at SemEval 2016 Task 3: Learning to Rank on Arabic Community Question Answering Forums with Word Embedding}},
  author = {Rana Malhas, Marwan Torki and Tamer Elsayed},
  booktitle = {Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)},
  year = {2016},
  doi = {10.18653/v1/s16-1134},
  url = {https://doi.org/10.18653/v1/s16-1134}
}
```

- [← previous MWEs in Treebanks: From Survey to Guidelines](https://md.research.elamur.ai/free/en/papers/rosen-2016-mwes-treebanks-survey-guidelines/)
- [next → The Denoised Web Treebank: Evaluating Dependency Parsing under Noisy Input Conditions](https://md.research.elamur.ai/free/en/papers/daiber-2016-denoised-web-treebank-evaluating-dependency/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/5/)

## References

6 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ruey-Cheng Chen, Damiano Spina, W Bruce Croft, Mark Sanderson, and Falk Scholer. 2015. Harnessing semantics for answer sentence retrieval. In Proceedings of the Eighth Workshop on Exploiting Semantic Annotations in Information Retrieval, pages 21–27. ACM. [find](https://scholar.google.com/scholar?q=Ruey%2DCheng+Chen%2C+Damiano+Spina%2C+W+Bruce+Croft%2C+Mark+Sanderson%2C+and+Falk+Scholer%2E+2015%2E+Harnessing+semantics+for+answer+sentence+retrieval%2E+In+Proceedings+of+the+Eighth+Workshop+on+Exploiting+Semantic+)
2. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. [Efficient estimation of word representations in vector space](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/). arXiv preprint arXiv:1301.3781. [↗](https://arxiv.org/abs/1301.3781)
3. Preslav Nakov, Lluı́s Màrquez, Walid Magdy, Alessandro Moschitti, Jim Glass, and Bilal Randeree. 2015. SemEval-2015 task 3: Answer selection in community question answering. In Proceedings of the 9th International Workshop on Semantic Evaluation, SemEval ’2015, pages 269–281, Denver, Colorado, June. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Preslav+Nakov%2C+Llu%C4%B1%CC%81s+Ma%CC%80rquez%2C+Walid+Magdy%2C+Alessandro+Moschitti%2C+Jim+Glass%2C+and+Bilal+Randeree%2E+2015%2E+SemEval%2D2015+task+3%3A+Answer+selection+in+community+question+answering%2E+In+Proceedings+of+the+9th)
4. Preslav Nakov, Lluı́s Màrquez, Alessandro Moschitti, Walid Magdy, Hamdy Mubarak, Abed Alhakim Freihat, Jim Glass, and Bilal Randeree. 2016. [SemEval-2016 task 3: Community question answering](https://md.research.elamur.ai/free/en/papers/nakov-2016-semeval-2016-task-3-community/). In Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval ’16, San Diego, California, June. Association for Computational Linguistics.
5. Anna Shtok, Gideon Dror, Yoelle Maarek, and Idan Szpektor. 2012. Learning from the past: answering new questions with past answers. In Proceedings of the 21st international conference on World Wide Web, pages 759–768. ACM. [find](https://scholar.google.com/scholar?q=Anna+Shtok%2C+Gideon+Dror%2C+Yoelle+Maarek%2C+and+Idan+Szpektor%2E+2012%2E+Learning+from+the+past%3A+answering+new+questions+with+past+answers%2E+In+Proceedings+of+the+21st+international+conference+on+World+Wide+We)
6. Mihai Surdeanu, Massimiliano Ciaramita, and Hugo Zaragoza. 2008. Learning to rank answers on large online qa collections. In ACL, volume 8, pages 719– 727. [find](https://scholar.google.com/scholar?q=Mihai+Surdeanu%2C+Massimiliano+Ciaramita%2C+and+Hugo+Zaragoza%2E+2008%2E+Learning+to+rank+answers+on+large+online+qa+collections%2E+In+ACL%2C+volume+8%2C+pages+719%E2%80%93+727%2E)

*Updated 31 October 2026*