---
title: "Zero-Shot and Fine-Tuned Evaluation of Generative LLMs for"
description: "Arabic is hard for word sense disambiguation (WSD). 79% accuracy. 29 references. Among them: 2."
authors: ["Yossra Noureldien", "Abdelrazig Mohamed", "Farah Attallah"]
year: 2025
venue: "Proceedings of The Third Arabic Natural Language Processing Conference"
source: "https://aclanthology.org/2025.arabicnlp-main.24/"
doi: "10.18653/v1/2025.arabicnlp-main.24"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/noureldien-2025-zero-shot-fine-tuned-evaluation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/noureldien-2025-zero-shot-fine-tuned-evaluation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/14/) / Zero-Shot and Fine-Tuned Evaluation of Generative LLMs for Arabic Word Sense Disambiguation

# Zero-Shot and Fine-Tuned Evaluation of Generative LLMs for Arabic Word Sense Disambiguation

**Authors:** [Yossra Noureldien](https://md.research.elamur.ai/free/en/authors/yossra-noureldien/), [Abdelrazig Mohamed](https://md.research.elamur.ai/free/en/authors/abdelrazig-mohamed/) and [Farah Attallah](https://md.research.elamur.ai/free/en/authors/farah-attallah/) · **Year:** 2025 · **Venue:** [Proceedings of The Third Arabic Natural Language Processing Conference](https://md.research.elamur.ai/free/en/venues/arabicnlp/) ([ArabicNLP 2025](https://md.research.elamur.ai/free/en/venues/arabicnlp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.arabicnlp-main.24/) · [PDF](https://aclanthology.org/2025.arabicnlp-main.24.pdf) · [DOI 10.18653/v1/2025.arabicnlp-main.24](https://doi.org/10.18653/v1/2025.arabicnlp-main.24) · [OpenAlex](https://openalex.org/W4416037657)

**Citations:** 2 citations (OpenAlex, 2026-10-14)

## Summary
Arabic is hard for word sense disambiguation (WSD). Its morphology is rich, many words are polysemous, and diacritics are usually dropped in writing. This work asks how well generative LLMs choose the intended sense of an Arabic word. The authors first survey six Arabic WSD resources. They then pick two public ones: a dictionary-derived gloss benchmark by El-Razzaz et al. and the SALMA corpus. Both are converted into one format: a sentence, a target word, and a numbered list of candidate glosses. Four models are tested. GPT-4o is used zero-shot only. LLaMA 3.1-8B, Qwen 2.5-7B and Gemma 2-9B are tested zero-shot and again after LoRA fine-tuning in 4-bit precision on a single GPU. Zero-shot, GPT-4o leads with roughly 79% accuracy on both datasets. After fine-tuning, every open model beats that score, reaching about 88–91% accuracy. An error analysis links the remaining difficulty to the number of candidate senses and to the type of mistake. The result matters because mid-sized open models with cheap adaptation give strong, reproducible Arabic WSD baselines, and the code is released.

## Key points
- Two datasets were used. Dataset A (El-Razzaz et al., 2021) has 15,549 senses for 5,347 words and is framed as binary gloss classification. Dataset B (SALMA, Jarrar et al., 2023) has about 34K tokens with graded relatedness scores across two lexicons; the top-scoring sense was taken as the gold label.
- No official splits exist, so the authors built their own 64/16/20 splits. Dataset A was split by sentence. Dataset B was stratified by token, with no sentence shared between train and test.
- Zero-shot results:
  - GPT-4o reached 79.16% accuracy and 67.92 macro-F1 on A, and 79.55% / 64.23 on B.
  - Gemma 2-9B was the best open model, especially on B (72.46%).
  - LLaMA 3.1-8B was weakest.
- After LoRA fine-tuning (NVIDIA L4 GPU, 4-bit loading, AdamW_8bit):
  - Qwen 2.5-7B was best on A with 90.77% accuracy and 83.98 macro-F1.
  - LLaMA 3.1-8B was best on B with 88.51% and 69.41.
- Fine-tuning changed the error profile:
  - Invalid outputs (refusals or non-existent sense IDs) disappeared for LLaMA, which had 638 refusals on A zero-shot.
  - LLaMA's semantically distant errors on A fell from 938 to 291.
  - Accuracy drops as the candidate set grows, so SALMA, with many items having 5+ senses, is harder.

## Interesting ideas
- Wrong answers are sorted into three bins: invalid (refusal or hallucinated ID), close (a near-paraphrase gloss), and distant (a semantically unrelated gloss). This separates harmless misses from real misunderstandings.
- Accuracy is reported per number of candidate senses rather than as one overall figure. This shows that difficulty grows with inventory size and keeps easy and hard datasets comparable.
- The model must answer with a sense ID from a closed list, and any ID outside that list is counted as an explicit failure. This turns hallucination into a measurable quantity.
- A graded resource (relatedness scores from 1–100%) is reduced to one gold label by taking the highest-scoring sense, while the other scores remain available for later analysis.
- Sentences with several targets are split at the token level, but whole sentences stay on one side of the split to prevent leakage.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Grade LLM teacher outputs with the same three bins, so that near-synonym slips are not punished as hard as truly wrong sense picks.
- **I2** (world): When choosing WordNet senses for entities and verbs, track accuracy by polysemy level to see where sense choice in the text world breaks down.
- **I3** (learn): Check every LLM teacher answer against the allowed inventory (VerbNet class, frame or sense ID) and reject or re-ask on invalid IDs before the answer becomes silver data.
- **I4** (learn): When several witness parsers or teachers give weighted votes, keep the top choice as the silver label and store the margin as a confidence signal.
- **I5** (tree): Apply the same rule when building train/test partitions from treebanks or multi-annotation silver sets, so evaluation is not inflated by shared sentences.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Khartoum](https://md.research.elamur.ai/free/en/universities/university-of-khartoum/) — University, Khartoum, Sudan · 4 works · 71 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (4): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [🔀 Word sense disambiguation](https://md.research.elamur.ai/free/en/topics/word-sense-disambiguation/) · 83 works

## Cite

Cite as: Yossra Noureldien, Abdelrazig Mohamed and Farah Attallah. 2025. Zero-Shot and Fine-Tuned Evaluation of Generative LLMs for Arabic Word Sense Disambiguation. Proceedings of The Third Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{noureldien2025zero,
  title = {{Zero-Shot and Fine-Tuned Evaluation of Generative LLMs for Arabic Word Sense Disambiguation}},
  author = {Yossra Noureldien, Abdelrazig Mohamed and Farah Attallah},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-main.24},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-main.24}
}
```

- [← previous WikiGap: Promoting Epistemic Equity by Surfacing Knowledge Gaps Between English Wikipedia and other Language Editions](https://md.research.elamur.ai/free/en/papers/wang-2025-wikigap-promoting-epistemic-equity-surfacing/)
- [next → !MSA at AraHealthQA 2025 Shared Task: Enhancing LLM Performance for Arabic Clinical Question Answering through Prompt Engineering and Ensemble Learning](https://md.research.elamur.ai/free/en/papers/younes-2025-msa-arahealthqa-2025-shared-task/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/14/)

## References

29 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Miuru Abeysiriwardana and Deshan Sumanathilaka. 2024. A survey on lexical ambiguity detection and word sense disambiguation. Preprint, arXiv:2403.16129. [↗](https://arxiv.org/abs/2403.16129)
2. Moustafa Al-Hajj and Mustafa Jarrar. 2021. ArabGloss-BERT: Fine-tuning BERT on context-gloss pairs for WSD. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 35–43, Held Online. INCOMA Ltd. [find](https://scholar.google.com/scholar?q=Moustafa+Al%2DHajj+and+Mustafa+Jarrar%2E+2021%2E+ArabGloss%2DBERT%3A+Fine%2Dtuning+BERT+on+context%2Dgloss+pairs+for+WSD%2E+In+Proceedings+of+the+International+Conference+on+Recent+Advances+in+Natural+Language+Proces)
3. Sawsan Alqahtani, Hanan Aldarmaki, and Mona Diab. 2019. Homograph disambiguation through selective diacritic restoration. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 49–59, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sawsan+Alqahtani%2C+Hanan+Aldarmaki%2C+and+Mona+Diab%2E+2019%2E+Homograph+disambiguation+through+selective+diacritic+restoration%2E+In+Proceedings+of+the+Fourth+Arabic+Natural+Language+Processing+Workshop%2C+page)
4. Waad Alshammari, Amal Almazrua, Asma Al Wazrah, Rawan Almatham, Muneera Alhoshan, and Abdulrahman Alosaimy. 2024. [KSAA-CAD shared task: Contemporary Arabic dictionary for reverse dictionary and word sense disambiguation](https://md.research.elamur.ai/free/en/papers/alshammari-2024-ksaa-cad-shared-task-contemporary/). In Proceedings of the Second Arabic Natural Language Processing Conference, pages 677–685, Bangkok, Thailand. Association for Computational Linguistics.
5. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. [AraBERT: Transformer-based model for Arabic language understanding](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/). In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. [↗](https://doi.org/10.48550/arxiv.2003.00104)
6. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, and 29 others. 2023. [Qwen technical report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/). Preprint, arXiv:2309.16609. [↗](https://arxiv.org/abs/2309.16609)
7. M Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham A. Alyahya, Sultan Al-Rashed, Faisal A. Mirza, Shaykhah Z. Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef Almushayqih, Raneem Alnajim, Salman Alsubaihi, Maryam Al Mansour, Majed Alrubaian, Ali Alammari, Zaki Alawami, Abdulmohsen Al-Thubaity, and 6 others. 2024. [Allam: Large language models for arabic and english](https://md.research.elamur.ai/free/en/papers/bari-2024-allam-large-language-models-arabic/). Preprint, arXiv:2407.15390. [↗](https://arxiv.org/abs/2407.15390)
8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1423)
9. M Soha Eid, Almoataz B Al-Said, Nayer M Wanas, Mohsen A Rashwan, and Nadia H Hegazy. 2010. Comparative study of rocchio classifier applied to supervised wsd using arabic lexical samples. In Proceedings of the tenth conference of language engeneering (SEOLEC’2010), Cairo, Egypt. [find](https://scholar.google.com/scholar?q=M+Soha+Eid%2C+Almoataz+B+Al%2DSaid%2C+Nayer+M+Wanas%2C+Mohsen+A+Rashwan%2C+and+Nadia+H+Hegazy%2E+2010%2E+Comparative+study+of+rocchio+classifier+applied+to+supervised+wsd+using+arabic+lexical+samples%2E+In+Proceeding)
10. Mohammed El-Razzaz, Mohamed Waleed Fakhr, and Fahima A. Maghraby. 2021. Arabic gloss wsd using bert. Applied Sciences, 11(6). [↗](https://doi.org/10.3390/app11062567)
11. AbdelRahim Elmadany, ElMoatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. [ORCA: A challenging benchmark for Arabic language understanding](https://md.research.elamur.ai/free/en/papers/elmadany-2023-orca-challenging-benchmark-arabic-language/). In Findings of the Association for Computational Linguistics: ACL 2023, pages 9559–9586, Toronto, Canada. Association for Computational Linguistics.
12. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). Preprint, arXiv:2407.21783. [↗](https://arxiv.org/abs/2407.21783)
13. Ahmed Hasanaath, Aisha Alansari, Ahmed Ashraf, Chafik Salmane, Hamzah Luqman, and Saad Ezzini. 2025. [Arareasoner: Evaluating reasoning-based llms for arabic nlp](https://md.research.elamur.ai/free/en/papers/hasanaath-2025-arareasoner-evaluating-reasoning-based-llms/). Preprint, arXiv:2506.08768. [↗](https://arxiv.org/abs/2506.08768)
14. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, and 1 others. 2018. Improving language understanding by generative pre-training. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+Ilya+Sutskever%2C+and+1+others%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E)
15. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). Preprint, arXiv:2106.09685. [↗](https://arxiv.org/abs/2106.09685)
16. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. [The interplay of variant, size, and task type in Arabic pre-trained language models](https://md.research.elamur.ai/free/en/papers/inoue-2021-interplay-variant-size-task-type/). In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92– 104, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
17. Mustafa Jarrar, Sanad Malaysha, Tymaa Hammouda, and Mohammed Khalilia. 2023. [SALMA: Arabic sense-annotated corpus and WSD benchmarks](https://md.research.elamur.ai/free/en/papers/jarrar-2023-salma-arabic-sense-annotated-corpus/). In Proceedings of ArabicNLP 2023, pages 359–369, Singapore (Hybrid). Association for Computational Linguistics.
18. Sanaa Kaddoura and Reem Nassar. 2024a. A comprehensive dataset for arabic word sense disambiguation. Data in Brief, 55:110591. [find](https://scholar.google.com/scholar?q=Sanaa+Kaddoura+and+Reem+Nassar%2E+2024a%2E+A+comprehensive+dataset+for+arabic+word+sense+disambiguation%2E+Data+in+Brief%2C+55%3A110591%2E)
19. Sanaa Kaddoura and Reem Nassar. 2024b. Enhancedbert: A feature-rich ensemble model for arabic word sense disambiguation with statistical analysis and optimized data collection. Journal of King Saud University - Computer and Information Sciences, 36(1):101911. [↗](https://doi.org/10.1016/j.jksuci.2023.101911)
20. Mohammed Khalilia, Sanad Malaysha, Reem Suwaileh, Mustafa Jarrar, Alaa Aljabari, Tamer Elsayed, and Imed Zitouni. 2024. [ArabicNLU 2024: The first Arabic natural language understanding shared task](https://md.research.elamur.ai/free/en/papers/khalilia-2024-arabicnlu-2024-first-arabic-natural/). In Proceedings of the Second Arabic Natural Language Processing Conference, pages 361–371, Bangkok, Thailand. Association for Computational Linguistics.
21. Md Tawkat Islam Khondaker, Abdul Waheed, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. [Gptaraeval: A comprehensive evaluation of chatgpt on arabic nlp](https://md.research.elamur.ai/free/en/papers/khondaker-2023-gptaraeval-comprehensive-evaluation-chatgpt-arabic/). Preprint, arXiv:2305.14976. [↗](https://arxiv.org/abs/2305.14976)
22. OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mądry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, and 401 others. 2024. [Gpt-4o system card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/). Preprint, arXiv:2410.21276. [↗](https://arxiv.org/abs/2410.21276)
23. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. [↗](https://arxiv.org/abs/2412.15115)
24. Rakia Saidi, Fethi Jarray, Asma Akacha, and Wissem Aribi. 2023. Wsdtn a novel dataset for arabic word sense disambiguation. In Advances in Computational Collective Intelligence, pages 203–212, Cham. Springer Nature Switzerland. [find](https://scholar.google.com/scholar?q=Rakia+Saidi%2C+Fethi+Jarray%2C+Asma+Akacha%2C+and+Wissem+Aribi%2E+2023%2E+Wsdtn+a+novel+dataset+for+arabic+word+sense+disambiguation%2E+In+Advances+in+Computational+Collective+Intelligence%2C+pages+203%E2%80%93212%2C+Cham%2E+S)
25. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Xudong Han, Sondos Mahmoud Bsharat, and 13 others. 2023. [Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models](https://md.research.elamur.ai/free/en/papers/sengupta-2023-jais-jais-chat-arabic-centric/). Preprint, arXiv:2308.16149. [↗](https://arxiv.org/abs/2308.16149)
26. Reem Suwaileh, Tamer Elsayed, and Muhammad Imran. 2023a. [IDRISI-D: Arabic and English datasets and benchmarks for location mention disambiguation over disaster microblogs](https://md.research.elamur.ai/free/en/papers/suwaileh-2023-idrisi-d-arabic-english-datasets/). In Proceedings of ArabicNLP 2023, pages 158–169, Singapore (Hybrid). Association for Computational Linguistics.
27. Reem Suwaileh, Muhammad Imran, and Tamer Elsayed. 2023b. [IDRISI-RA: The first Arabic location mention recognition dataset of disaster tweets](https://md.research.elamur.ai/free/en/papers/suwaileh-2023-idrisi-ra-first-arabic-location/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16298–16317, Toronto, Canada. Association for Computational Linguistics.
28. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, and 179 others. 2024. Gemma 2: Improving open language models at a practical size. Preprint, arXiv:2408.00118. [↗](https://arxiv.org/abs/2408.00118)
29. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). Preprint, arXiv:2302.13971. [↗](https://arxiv.org/abs/2302.13971)

*Updated 1 November 2026*