---
title: "A Benchmark and Scoring Algorithm for Enriching Arabic"
description: "The paper tackles the problem of growing an existing synset with extra synonyms when synonymy is… 35 references, cited by 10 papers here."
authors: ["Sana Ghanem", "Mustafa Jarrar", "Radi Jarrar", "Ibrahim Bounhas"]
year: 2023
venue: "Proceedings of the 12th Global Wordnet Conference"
source: "https://aclanthology.org/2023.gwc-1.34/"
doi: "10.18653/v1/2023.gwc-1.34"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ghanem-2023-benchmark-scoring-algorithm-enriching-arabic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ghanem-2023-benchmark-scoring-algorithm-enriching-arabic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/) / A Benchmark and Scoring Algorithm for Enriching Arabic Synonyms

# A Benchmark and Scoring Algorithm for Enriching Arabic Synonyms

**Authors:** [Sana Ghanem](https://md.research.elamur.ai/free/en/authors/sana-ghanem/), [Mustafa Jarrar](https://md.research.elamur.ai/free/en/authors/mustafa-jarrar/), [Radi Jarrar](https://md.research.elamur.ai/free/en/authors/radi-jarrar/) and Ibrahim Bounhas · **Year:** 2023 · **Venue:** [Proceedings of the 12th Global Wordnet Conference](https://md.research.elamur.ai/free/en/venues/gwc/) ([Global Wordnet Conference 2023](https://md.research.elamur.ai/free/en/venues/gwc/2023/)) · **Publisher:** [Global Wordnet Association](https://md.research.elamur.ai/free/en/publishers/global-wordnet-association/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.gwc-1.34/) · [PDF](https://aclanthology.org/2023.gwc-1.34.pdf) · [DOI 10.18653/v1/2023.gwc-1.34](https://doi.org/10.18653/v1/2023.gwc-1.34) · [OpenAlex](https://openalex.org/W7103892228)

**Citations:** 8 citations (OpenAlex, 2026-10-14) · [10](https://md.research.elamur.ai/free/en/papers/ghanem-2023-benchmark-scoring-algorithm-enriching-arabic/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper tackles the problem of growing an existing synset with extra synonyms when synonymy is treated as graded rather than binary. Arabic synonym resources are small, and judgments of substitutability are subjective, so the authors attach a fuzzy strength in [0–1] to each candidate. They make two contributions. The first is a benchmark: 500 synsets from Arabic WordNet with 3K candidate synonyms, each scored by four linguists. The second is an algorithm that expands a mono- or multilingual synset into a directed synonymy graph up to depth k. It treats words lying on cyclic paths as candidates and scores each one with a fuzzy model tuned on the benchmark, using graph connectivity as the signal. Against the linguists' average, the algorithm's error is RMSE 0.32 and MAE 0.27. An ANOVA finds it statistically indistinguishable from most annotators. In a masking test on AWN, the masked term is usually recovered at top rank. The work supplies a missing gold standard for evaluating synonym extraction.

## Key points
- The benchmark has 500 Arabic WordNet synsets, 3,000 candidate synonyms and 4 linguist annotators per candidate. It is public at portal.sina.birzeit.edu/synonyms.
- Linguists disagree with each other by RMSE 0.16–0.22 and MAE 0.12–0.16. Against the linguists' average, the algorithm scores RMSE 0.32 and MAE 0.27.
- A one-way ANOVA finds the algorithm not significantly different from the annotators and their average. Only linguist L1 stands out from the rest.
- The masking evaluation runs on the ~10K AWN synsets, using Arabic and English terms, at graph depths 3 and 4:
  - **Depth 3:** top-1 recovery is 99.1% for the highest-frequency term, 98.7% for average, 98.1% for random and 88.4% for lowest.
  - **Depth 4:** recovery drops to 95.2%, 92.0%, 89.3% and 62.0% respectively.
- Stated limitations:
  - Diacritic variants of the same Arabic word are counted as distinct words.
  - Dialects are not yet handled.
  - Accuracy depends strongly on the structure of the lexicon used.

## Interesting ideas
- Treat membership of a word in a synset as a fuzzy score, so that different consumers can pick their own cutoff: a strict one for building lexicons, a loose one for retrieval.
- Generate candidates by finding cycles in a graph of synonym or translation links, since words that loop back to the seed synset are probably close in meaning.
- Measure annotator-vs-annotator disagreement first and use it as the reference band for judging a system, rather than demanding perfect agreement with a single gold label.
- Use masking as self-supervised evaluation: delete a known member, then check whether the method ranks it first, with the test stratified by how frequent the masked item is.
- Graph depth trades recall against ranking precision, because a deeper expansion admits many more distractor candidates.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (world): Ontology links compiled from WordNet could carry a strength value, letting the world model accept only strong synonyms when merging entities but weaker ones when matching question wording.
- **I2** (world): Cycle detection over WordNet/Wikidata/translation edges could propose missing equivalences in the knowledge graph, each with an explainable path as evidence.
- **I3** (learn): When evaluating silver data from witness parsers or LLM teachers, compare system-vs-human error against the human-vs-human spread to know what counts as "good enough".
- **I4** (learn): Mask known lexicon entries or frame-role fillers and test recovery, reporting results separately for rare items, which this paper shows are where accuracy collapses.
- **I5** (world): Cap expansion depth when inferring relations through the knowledge graph, and report the depth used as part of the explanation.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Birzeit University](https://md.research.elamur.ai/free/en/universities/birzeit-university/) — University, Birzeit, Palestine · 36 works · 798 citations
- [University of Carthage](https://md.research.elamur.ai/free/en/universities/university-of-carthage/) — University, Tunis, Tunisia · 3 works · 275 citations
- [National Institute of Applied Science and Technology](https://md.research.elamur.ai/free/en/labs/national-institute-of-applied-science-and-technology/) — Research institute, Tunis, Tunisia · 2 works · 274 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

## Cite

Cite as: Sana Ghanem, Mustafa Jarrar, Radi Jarrar and Ibrahim Bounhas. 2023. A Benchmark and Scoring Algorithm for Enriching Arabic Synonyms. Proceedings of the 12th Global Wordnet Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ghanem2023benchmark,
  title = {{A Benchmark and Scoring Algorithm for Enriching Arabic Synonyms}},
  author = {Sana Ghanem, Mustafa Jarrar, Radi Jarrar and Ibrahim Bounhas},
  booktitle = {Proceedings of the 12th Global Wordnet Conference},
  year = {2023},
  doi = {10.18653/v1/2023.gwc-1.34},
  url = {https://doi.org/10.18653/v1/2023.gwc-1.34}
}
```

- [← previous Topic-Guided Sampling For Data-Efficient Multi-Domain Stance Detection](https://md.research.elamur.ai/free/en/papers/arakelyan-2023-topic-guided-sampling-data-efficient/)
- [next → Can Diffusion Model Achieve Better Performance in Text Generation ? Bridging the Gap between Training and Inference !](https://md.research.elamur.ai/free/en/papers/tang-2023-can-diffusion-model-achieve-better/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/)

## References

35 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Moustafa Al-Hajj and Mustafa Jarrar. 2021. [Arabglossbert: Fine-tuning bert on context-gloss pairs for wsd](https://md.research.elamur.ai/free/en/papers/hajj-2021-arabglossbert-fine-tuning-bert-context/). In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 40–48, Online. INCOMA Ltd.
2. Rawan N Al-Matham and Hend S Al-Khalifa. 2021. Synoextractor: a novel pipeline for arabic synonym extraction using word2vec word embeddings. [↗](https://doi.org/10.1155/2021/6627434)
3. Complexity, 2021. Yousef Alizadeh-Q, Behrouz Minaei-Bidgoli, Sayyed-Ali Hossayni, Mohammad-R. [find](https://scholar.google.com/scholar?q=Complexity%2C+2021%2E+Yousef+Alizadeh%2DQ%2C+Behrouz+Minaei%2DBidgoli%2C+Sayyed%2DAli+Hossayni%2C+Mohammad%2DR%2E)
4. Akbarzadeh-T., Diego Reforgiato Recupero, Mohammad Reza Rajati, and Aldo Gangemi. 2021. Interval probabilistic fuzzy wordnet. CoRR, abs/2104.10660. [↗](https://arxiv.org/abs/2104.10660)
5. Tiziano Flati and Roberto Navigli. 2012. The cqc algorithm: Cycling in graphs to semantically enrich and enhance a bilingual dictionary. Journal of Artificial Intelligence Research, 43:135–171. [↗](https://doi.org/10.1613/jair.3456)
6. Karim El Haff, Mustafa Jarrar, Tymaa Hammouda, and Fadi Zaraket. 2022. Curras + baladi: Towards a levantine corpus. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2022), Marseille, France. [↗](https://doi.org/10.48550/arxiv.2205.09692)
7. Mamoun Abu Helou, Matteo Palmonari, and Mustafa Jarrar. 2016. Effectiveness of automatic translations for cross-lingual ontology mapping. Journal of Artificial Intelligence Research, 55(1):165–208. [↗](https://doi.org/10.1613/jair.4789)
8. Mamoun Abu Helou, Matteo Palmonari, Mustafa Jarrar, and Christiane Fellbaum. 2014. Towards building lexical ontology via cross-language matching. In Proceedings of the 7th Conference on Global WordNet, pages 346–354. Global WordNet Association. [find](https://scholar.google.com/scholar?q=Mamoun+Abu+Helou%2C+Matteo+Palmonari%2C+Mustafa+Jarrar%2C+and+Christiane+Fellbaum%2E+2014%2E+Towards+building+lexical+ontology+via+cross%2Dlanguage+matching%2E+In+Proceedings+of+the+7th+Conference+on+Global+WordNet)
9. Sayyed-Ali Hossayni, Mohammad-R. Akbarzadeh-T., Diego Reforgiato Recupero, Aldo Gangemi, Esteve del Acebo, and Josep Lluís de la Rosa i Esteva. 2020. An algorithm for fuzzification of wordnets, supported by a mathematical proof. CoRR, abs/2006.04042. [↗](https://arxiv.org/abs/2006.04042)
10. Mustafa Jarrar. 2005. Towards Methodological Principles for Ontology Engineering. Ph.D. thesis, Vrije Universiteit Brussel. [find](https://scholar.google.com/scholar?q=Mustafa+Jarrar%2E+2005%2E+Towards+Methodological+Principles+for+Ontology+Engineering%2E+Ph%2ED%2E+thesis%2C+Vrije+Universiteit+Brussel%2E)
11. Kareem Darwish, Nizar Habash, Mourad Abbas, Hend Al-Khalifa, Huseein T. Al-Natsheh, Houda Bouamor, Karim Bouzoubaa, Violetta Cavalli-Sforza, Samhaa R. El-Beltagy, Wassim El-Hajj, Mustafa Jarrar, and Hamdy Mubarak. 2021. A panoramic survey of natural language processing in the arab worlds. Commun. ACM, 64(4):72–81. [find](https://scholar.google.com/scholar?q=Kareem+Darwish%2C+Nizar+Habash%2C+Mourad+Abbas%2C+Hend+Al%2DKhalifa%2C+Huseein+T%2E+Al%2DNatsheh%2C+Houda+Bouamor%2C+Karim+Bouzoubaa%2C+Violetta+Cavalli%2DSforza%2C+Samhaa+R%2E+El%2DBeltagy%2C+Wassim+El%2DHajj%2C+Mustafa+Jarrar%2C+and+H)
12. Mustafa Jarrar. 2011. Building a formal arabic ontology (invited paper). In Proceedings of the Experts Meeting on Arabic Ontologies and Semantic Networks. ALECSO, Arab League. [find](https://scholar.google.com/scholar?q=Mustafa+Jarrar%2E+2011%2E+Building+a+formal+arabic+ontology+%28invited+paper%29%2E+In+Proceedings+of+the+Experts+Meeting+on+Arabic+Ontologies+and+Semantic+Networks%2E+ALECSO%2C+Arab+League%2E)
13. Sabry Elkateb, William Black, Piek Vossen, David Farwell, H Rodríguez, A Pease, and M Alkhalifa. 2006. [Arabic wordnet and the challenges of arabic](https://md.research.elamur.ai/free/en/papers/elkateb-2006-arabic-wordnet-challenges-arabic/). In Proceedings of Arabic NLP/MT Conference, London, UK, pages 665–670.
14. Mustafa Jarrar. 2021. [The arabic ontology - an arabic wordnet with ontologically clean content](https://md.research.elamur.ai/free/en/papers/jarrar-2022-arabic-ontology-arabic-wordnet-ontologically/). Applied Ontology Journal, 16(1):1–26. [↗](https://doi.org/10.3233/ao-200241)
15. Gonenc Ercan and Farid Haziyev. 2019. Synset expansion on translation graph for automatic wordnet construction. Information Processing & Management, 56(1):130–150. [↗](https://doi.org/10.1016/j.ipm.2018.10.002)
16. Mustafa Jarrar. 2020. Digitization of Arabic Lexicons, pages 214–217. UAE Ministry of Culture and Youth. [find](https://scholar.google.com/scholar?q=Mustafa+Jarrar%2E+2020%2E+Digitization+of+Arabic+Lexicons%2C+pages+214%E2%80%93217%2E+UAE+Ministry+of+Culture+and+Youth%2E)
17. Mustafa Jarrar and Hamzeh Amayreh. 2019. An arabic-multilingual database with a lexicographic search engine. In The 24th International Conference on Applications of Natural Language to Information Systems (NLDB 2019), volume 11608 of LNCS, pages 234–246. Springer. [find](https://scholar.google.com/scholar?q=Mustafa+Jarrar+and+Hamzeh+Amayreh%2E+2019%2E+An+arabic%2Dmultilingual+database+with+a+lexicographic+search+engine%2E+In+The+24th+International+Conference+on+Applications+of+Natural+Language+to+Information+Sys)
18. Mustafa Jarrar, Hamzeh Amayreh, and John P. Mc-Crae. 2019. Representing arabic lexicons in lemon - a preliminary study. In The 2nd Conference on Language, Data and Knowledge (LDK 2019), volume 2402, pages 29–33. CEUR Workshop Proceedings. [↗](https://doi.org/10.5281/zenodo.3266885)
19. Eman Naser-Karajah, Nabil Arman, and Mustafa Jarrar. 2021. [Current trends and approaches in synonyms extraction: Potential adaptation to arabic](https://md.research.elamur.ai/free/en/papers/naser-karajah-2022-current-trends-approaches-synonyms-extraction/). In Proceedings of the 2021 International Conference on Information Technology (ICIT), pages 428–434, Amman, Jordan. IEEE.
20. Mustafa Jarrar, Nizar Habash, Faeq Alrimawi, Diyam Akra, and Nasser Zalmout. 2017. Curras: An annotated corpus for the palestinian arabic dialect. Journal Language Resources and Evaluation, 51(3):745–775. [↗](https://doi.org/10.1007/s10579-016-9370-7)
21. Hugo Gonçalo Oliveira and Paulo Gomes. 2014. Eco and onto.pt: A flexible approach for creating a portuguese wordnet automatically. Language Resources and Evaluation, 48. [↗](https://doi.org/10.1007/s10579-013-9249-9)
22. Mustafa Jarrar, Eman Karajah, Muhammad Khalifa, and Khaled Shaalan. 2021. Extracting synonyms from bilingual dictionaries. In Proceedings of the 11th International Global Wordnet Conference (GWC2021), pages 215–222. Global Wordnet Association. [find](https://scholar.google.com/scholar?q=Mustafa+Jarrar%2C+Eman+Karajah%2C+Muhammad+Khalifa%2C+and+Khaled+Shaalan%2E+2021%2E+Extracting+synonyms+from+bilingual+dictionaries%2E+In+Proceedings+of+the+11th+International+Global+Wordnet+Conference+%28GWC2021%29%2C)
23. Mustafa Jarrar, Fadi Zaraket, Rami Asia, and Hamzeh Amayreh. 2018. Diacritic-based matching of arabic words. ACM Asian and Low-Resource Language Information Processing, 18(2):10:1–10:21. [↗](https://doi.org/10.1145/3242177)
24. Mustafa Jarrar, Fadi A Zaraket, Tymaa Hammouda, Daanish Masood Alavi, and Martin Waahlisch. 2022. Lisan: Yemeni, irqi, libyan, and sudanese arabic dialect copora with morphological annotations. [find](https://scholar.google.com/scholar?q=Mustafa+Jarrar%2C+Fadi+A+Zaraket%2C+Tymaa+Hammouda%2C+Daanish+Masood+Alavi%2C+and+Martin+Waahlisch%2E+2022%2E+Lisan%3A+Yemeni%2C+irqi%2C+libyan%2C+and+sudanese+arabic+dialect+copora+with+morphological+annotations%2E)
25. Mikhail Khodak, Andrej Risteski, Christiane Fellbaum, and Sanjeev Arora. 2017. Automated wordnet construction using word embeddings. In Proceedings of the 1st Workshop on Sense, Concept and Entity Representations and their Applications, pages 12–23. [find](https://scholar.google.com/scholar?q=Mikhail+Khodak%2C+Andrej+Risteski%2C+Christiane+Fellbaum%2C+and+Sanjeev+Arora%2E+2017%2E+Automated+wordnet+construction+using+word+embeddings%2E+In+Proceedings+of+the+1st+Workshop+on+Sense%2C+Concept+and+Entity+Rep)
26. Marco Maru, Simone Conia, Michele Bevilacqua, and Roberto Navigli. 2022. Nibbling at the hard core of Word Sense Disambiguation. In Proceedings of the ACL2022 (Vol.1), pages 4724–4737, Dublin, Ireland. ACL. [find](https://scholar.google.com/scholar?q=Marco+Maru%2C+Simone+Conia%2C+Michele+Bevilacqua%2C+and+Roberto+Navigli%2E+2022%2E+Nibbling+at+the+hard+core+of+Word+Sense+Disambiguation%2E+In+Proceedings+of+the+ACL2022+%28Vol%2E1%29%2C+pages+4724%E2%80%934737%2C+Dublin%2C+Ireland)
27. George A Miller, Richard Beckwith, Christiane Fellbaum, Derek Gross, and Katherine J Miller. 1990. Introduction to wordnet: An on-line lexical database. International journal of lexicography, 3(4):235–244. [↗](https://doi.org/10.1093/ijl/3.4.235)
28. Nora Mohammed. 2020. Extracting word synonyms from text using neural approaches. International Arab Journal of Information Technology, 17. [↗](https://doi.org/10.34028/iajit/17/1/6)
29. Hugo Gonçalo Oliveira and Fábio Santos. 2016. Discovering fuzzy synsets from the redundancy in different lexical-semantic resources. In Proceedings of LREC 2016, Paris, France. ELRA. [find](https://scholar.google.com/scholar?q=Hugo+Gon%C3%A7alo+Oliveira+and+F%C3%A1bio+Santos%2E+2016%2E+Discovering+fuzzy+synsets+from+the+redundancy+in+different+lexical%2Dsemantic+resources%2E+In+Proceedings+of+LREC+2016%2C+Paris%2C+France%2E+ELRA%2E)
30. Barbara G Tabachnick and LS Fidell. 2001. Using multivariate statististics. Allyn &Bacon A Pearson Education Company: Boston. [find](https://scholar.google.com/scholar?q=Barbara+G+Tabachnick+and+LS+Fidell%2E+2001%2E+Using+multivariate+statististics%2E+Allyn+%26Bacon+A+Pearson+Education+Company%3A+Boston%2E)
31. Feras Al Tarouti and Jugal Kalita. 2016. Enhancing automatic wordnet construction using word embeddings. [find](https://scholar.google.com/scholar?q=Feras+Al+Tarouti+and+Jugal+Kalita%2E+2016%2E+Enhancing+automatic+wordnet+construction+using+word+embeddings%2E)
32. Daniel Torregrosa, Mihael Arcan, Sina Ahmadi, and John P McCrae. 2019. Tiad 2019 shared task: Leveraging knowledge graphs with neural machine translation for automatic multilingual dictionary generation. Translation Inference Across Dictionaries. [find](https://scholar.google.com/scholar?q=Daniel+Torregrosa%2C+Mihael+Arcan%2C+Sina+Ahmadi%2C+and+John+P+McCrae%2E+2019%2E+Tiad+2019+shared+task%3A+Leveraging+knowledge+graphs+with+neural+machine+translation+for+automatic+multilingual+dictionary+generati)
33. Marta Villegas, Maite Melero, Núria Bel, and Jorge Gracia. 2016. Leveraging rdf graphs for crossing multiple bilingual dictionaries. In Proceedings of LREC2016, pages 868–876. [find](https://scholar.google.com/scholar?q=Marta+Villegas%2C+Maite+Melero%2C+N%C3%BAria+Bel%2C+and+Jorge+Gracia%2E+2016%2E+Leveraging+rdf+graphs+for+crossing+multiple+bilingual+dictionaries%2E+In+Proceedings+of+LREC2016%2C+pages+868%E2%80%93876%2E)
34. Weijie Wang and Yanmin Lu. 2018. Analysis of the mean absolute error (mae) and the root mean square error (rmse) in assessing rounding model. In IOP conference series: materials science and engineering, volume 324. IOP Publishing. [find](https://scholar.google.com/scholar?q=Weijie+Wang+and+Yanmin+Lu%2E+2018%2E+Analysis+of+the+mean+absolute+error+%28mae%29+and+the+root+mean+square+error+%28rmse%29+in+assessing+rounding+model%2E+In+IOP+conference+series%3A+materials+science+and+engineerin)
35. Hua Wu and Ming Zhou. 2003. Optimizing synonym extraction using monolingual and bilingual resources. In Proceedings of the second international workshop on Paraphrasing, pages 72–79. [find](https://scholar.google.com/scholar?q=Hua+Wu+and+Ming+Zhou%2E+2003%2E+Optimizing+synonym+extraction+using+monolingual+and+bilingual+resources%2E+In+Proceedings+of+the+second+international+workshop+on+Paraphrasing%2C+pages+72%E2%80%9379%2E)

*Updated 1 November 2026*