---
title: "Hamyaria at PalmX2025: Leveraging Large Language Models to"
description: "The paper reports a system for the PalmX 2025 shared task, which benchmarks how well LLMs answer… 10 references, cited by 1 paper here."
authors: ["Walid Al-Dhabyani", "Hamzah A. Alsayadi"]
year: 2025
venue: "Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks"
source: "https://aclanthology.org/2025.arabicnlp-sharedtasks.108/"
doi: "10.18653/v1/2025.arabicnlp-sharedtasks.108"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/dhabyani-2025-hamyaria-palmx2025-leveraging-large-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/dhabyani-2025-hamyaria-palmx2025-leveraging-large-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/15/) / Hamyaria at PalmX2025: Leveraging Large Language Models to Improve Arabic Multiple-Choice Questions in Cultural and Islamic Domains

# Hamyaria at PalmX2025: Leveraging Large Language Models to Improve Arabic Multiple-Choice Questions in Cultural and Islamic Domains

**Authors:** Walid Al-Dhabyani and [Hamzah A. Alsayadi](https://md.research.elamur.ai/free/en/authors/hamzah-a-alsayadi/) · **Year:** 2025 · **Venue:** [Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks](https://md.research.elamur.ai/free/en/venues/arabicnlp/) ([ArabicNLP 2025](https://md.research.elamur.ai/free/en/venues/arabicnlp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.arabicnlp-sharedtasks.108/) · [PDF](https://aclanthology.org/2025.arabicnlp-sharedtasks.108.pdf) · [DOI 10.18653/v1/2025.arabicnlp-sharedtasks.108](https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.108) · [OpenAlex](https://openalex.org/W4416037608)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/dhabyani-2025-hamyaria-palmx2025-leveraging-large-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper reports a system for the PalmX 2025 shared task, which benchmarks how well LLMs answer Arabic multiple-choice questions about general culture and Islamic knowledge. The authors fine-tune the small Qwen2.5-3B-Instruct model separately for each domain. They treat the two domains differently. Cultural questions are expanded with paraphrases generated by QCRI/Fanar-1-9B-Instruct and with reorderings of the answer options. Islamic questions are left untouched so that religious wording is not distorted.

The fine-tuned model scores 65.90% on the cultural test set, slightly below the 70% baseline, and 70.83% on the Islamic test set, above its 65% baseline. Augmentation added only about half a point over training on the original cultural data. The authors call this gain negligible and suggest that preserving content may matter more than adding data in sensitive domains. The work matters mainly as an honest small-compute data point: cheap augmentation of MCQ data gives little benefit, and per-letter error analysis exposes answer-position biases.

## Key points
- PalmX is derived from the Palm instruction dataset and has two splits:
  - Culture: 2000 train / 500 dev / 2000 test.
  - Islamic: 1000 train / 300 dev / 1001 test.
- Cultural augmentation pipeline:
  - Fanar-1-9B-Instruct generated two paraphrases per question.
  - Each item then received three shuffled answer orders.
  - The paraphraser sometimes slipped English words into the output, which had to be fixed.
- Training setup for Qwen2.5-3B-Instruct:
  - Learning rate 1e-5, batch size 1, 8 gradient-accumulation steps.
  - Max sequence length 512, AdamW with cosine schedule.
  - Run on a Colab A100 40GB.
  - The Islamic runs were allowed more epochs (up to 10 vs. 5), which confounds the domain comparison.
- Results:
  - Cultural: 65.90% with augmentation vs. 65.40% without.
  - Islamic: 70.83%, weighted F1 0.72 but macro F1 only 0.51, because a rare "OTHER" label (16 items) was never predicted correctly.
- Model comparisons:
  - Llama-3.2-3B-Instruct consistently trailed Qwen as the trained model.
  - ALLaM paraphrases often changed the syntactic form of questions into hard-to-read outputs.

## Interesting ideas
- Apply augmentation per domain: rewrite only where meaning is robust to rewording, and freeze text where exact phrasing carries the answer.
- Shuffling answer options multiplies training items and discourages the model from learning which letter tends to be correct, rather than the content.
- A per-letter confusion matrix reveals positional preferences, such as over-predicting A, that a single accuracy number hides.
- Comparing training on original-only data against augmented data showed the augmentation was nearly worthless, a cheap check worth running before trusting any augmentation.
- Generated paraphrases were checked for leakage into another language (English tokens inside Arabic text), a simple filter on teacher output.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): silver-data generation could tag sources as paraphrase-safe or verbatim-only, so that legal, religious or numeric texts are never reworded before training.
- **I2** (answer): school-test readers could be evaluated on several permutations of the options, counting a question as solved only if the answer survives every reordering.
- **I3** (learn): add option-position confusion tables to evaluation reports as a negative control showing that answers follow from the world model, not from letter priors.
- **I4** (learn): every LLM-teacher or synthetic-data tier should be paired with an ablation run on gold-only data, to show its contribution is real.
- **I5** (back): output of the text → graph → text compressor could be screened for off-language or off-register tokens before being accepted as a valid round trip.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Cairo University](https://md.research.elamur.ai/free/en/universities/cairo-university/) — University, Giza, Egypt · 43 works · 1,352 citations
- [Ibb University](https://md.research.elamur.ai/free/en/universities/ibb-university/) — University, Ibb, Yemen · 6 works · 13 citations

Also on the byline: [Hadhramout University](https://ror.org/02kv0px94).

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 3,066 works

## Cite

Cite as: Walid Al-Dhabyani and Hamzah A. Alsayadi. 2025. Hamyaria at PalmX2025: Leveraging Large Language Models to Improve Arabic Multiple-Choice Questions in Cultural and Islamic Domains. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{aldhabyani2025hamyaria,
  title = {{Hamyaria at PalmX2025: Leveraging Large Language Models to Improve Arabic Multiple-Choice Questions in Cultural and Islamic Domains}},
  author = {Walid Al-Dhabyani and Hamzah A. Alsayadi},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.108},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.108}
}
```

- [← previous HAMSA: Hijacking Aligned Compact Models via Stealthy Automation](https://md.research.elamur.ai/free/en/papers/krylov-2025-hamsa-hijacking-aligned-compact-models/)
- [next → Harmonizing Annotation of Turkic Postverbial Constructions: A Comparative Study of UD Treebanks](https://md.research.elamur.ai/free/en/papers/akhundjanova-2025-harmonizing-annotation-turkic-postverbial-constructions/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/15/)

## References

10 references, 9 of them with a card of ours. ↗ leads to the work.

1. Abdelrahman Abdallah, Mahmoud Kasem, Mahmoud Abdalla, Mohamed Mahmoud, Mohamed Elkasaby, Yasser Elbendary, and Adam Jatowt. 2024. [Arabicaqa: A comprehensive dataset for arabic question answering](https://md.research.elamur.ai/free/en/papers/abdallah-2024-arabicaqa-comprehensive-dataset-arabic-question/). In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2049–2059.
2. Fakhraddin Alwajih, Abdellah El Mekki, Samar Mohamed Magdy, AbdelRahim A. Elmadany, Omer Nacar, El Moatez Billah Nagoudi, Reem Abdel-Salam, Hanin Atwany, Youssef Nafea, Abdulfattah Mohammed Yahya, Rahaf Alhamouri, Hamzah A. Alsayadi, Hiba Zayed, Sara Shatnawi, Serry Sibaee, Yasir Ech-chammakhy, Walid Al-Dhabyani, Marwa Mohamed Ali, Imen Jarraya, Ahmed Oumar El-Shangiti, Aisha Alraeesi, Mohammed Anwar AL-Ghrawi, Abdulrahman S. Al-Batati, Elgizouli Mohamed, Noha Taha Elgindi, Muhammed Saeed, Houdaifa Atou, Issam Ait Yahia, Abdelhak Bouayad, Mohammed Machrouh, Amal Makouar, Dania Alkawi, Mukhtar Mohamed, Safaa Taher Abdelfadil, Amine Ziad Ounnoughene, Anfel Rouabhia, Rwaa Assi, Ahmed Sorkatti, Mohamedou Cheikh Tourad, Anis Koubaa, Ismail Berrada, Mustafa Jarrar, Shady Shehata, and Muhammad Abdul-Mageed. 2025a. [Palm: A culturally inclusive and linguistically diverse dataset for Arabic LLMs](https://md.research.elamur.ai/free/en/papers/alwajih-2025-palm-culturally-inclusive-linguistically-diverse/). In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 32871–32894, Vienna, Austria. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2503.00151)
3. Fakhraddin Alwajih, Abdellah El Mekki, Hamdy Mubarak, Majd Hawasly, Abubakr Mohamed, and Muhammad Abdul-Mageed. 2025b. [PalmX 2025: The First Shared Task on Benchmarking LLMs on Arabic and Islamic Culture](https://md.research.elamur.ai/free/en/papers/alwajih-2025-palmx-2025-first-shared-task/). In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), Suzhou, China. Association for Computational Linguistics. Co-located with EMNLP 2025, November 5–9.
4. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. [A survey on evaluation of large language models](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/). ACM transactions on intelligent systems and technology, 15(3):1–45. [↗](https://doi.org/10.1145/3641289)
5. Malak Mashaabi, Shahad Al-Khalifa, and Hend Al-Khalifa. 2024. [A survey of large language models for arabic language and its dialects](https://md.research.elamur.ai/free/en/papers/mashaabi-2024-survey-large-language-models-arabic/). arXiv preprint arXiv:2410.20238. [↗](https://arxiv.org/abs/2410.20238)
6. Fanar Team, Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, et al. 2025. [Fanar: An arabic-centric multimodal generative ai platform](https://md.research.elamur.ai/free/en/papers/abbas-2025-fanar-arabic-centric-multimodal-generative/). arXiv preprint arXiv:2501.13944. [↗](https://arxiv.org/abs/2501.13944)
7. Qwen Team. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671. [↗](https://arxiv.org/abs/2407.10671)
8. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30. A PalmX Dataset Examples Examples of PalmX dataset for Cultural and Islamic Domains. Input (Cultural Domain):  • Question: ?Qå”Ó éÖޕ A« ù ë AÓ ñë (What is the capital of Egypt?)    • Options: éK PY JºƒB @ (H . èQëA ®Ë@ ( @ à@ñƒ @ (X è Q m.Ì '@ (h.   • Expected Output: èQëA ®Ë@ ( @ Input (Islamic Domain): Fakhraddin Alwajih, Abdellah El Mekki, Samar Mohamed Magdy, Abdelrahim A Elmadany, Omer Nacar, El Moatez Billah Nagoudi, Reem Abdel-Salam, Hanin Atwany, Youssef Nafea, Abdulfattah Mohammed Yahya, et al. 2025c. [Palm: A culturally inclusive and linguistically diverse dataset for arabic llms](https://md.research.elamur.ai/free/en/papers/alwajih-2025-palm-culturally-inclusive-linguistically-diverse/). arXiv preprint arXiv:2503.00151. [↗](https://arxiv.org/abs/2503.00151)
9. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. [Arabert: Transformer-based model for arabic language understanding](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/). arXiv preprint arXiv:2003.00104. [↗](https://arxiv.org/abs/2003.00104)
10. M Saiful Bari, Yazeed Alnumay, Norah A Alzahrani, Nouf M Alotaibi, Hisham A Alyahya, Sultan Al-Rashed, Faisal A Mirza, Shaykhah Z Alsubaie, Hassan A Alahmed, Ghadah Alabduljabbar, et al. 2024. [Allam: Large language models for arabic and english](https://md.research.elamur.ai/free/en/papers/bari-2024-allam-large-language-models-arabic/). arXiv preprint arXiv:2407.15390. [↗](https://arxiv.org/abs/2407.15390)

*Updated 1 November 2026*