--- title: "Star at PalmX 2025 — paper by Elrefai et al. (2025)" description: "The work offers a cheap recipe for adding regional knowledge to compact models. 8 references, cited by 1 paper here." authors: ["Eman Elrefai", "Esraa Khaled", "Alhassan Ehab"] year: 2025 venue: "Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks" source: "https://aclanthology.org/2025.arabicnlp-sharedtasks.113/" doi: "10.18653/v1/2025.arabicnlp-sharedtasks.113" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/elrefai-2025-star-palmx-2025-arabic-cultural/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/elrefai-2025-star-palmx-2025-arabic-cultural/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/14/) / Star at PalmX 2025: Arabic Cultural Understanding via Targeted Pretraining and Lightweight Fine-tuning # Star at PalmX 2025: Arabic Cultural Understanding via Targeted Pretraining and Lightweight Fine-tuning **Authors:** [Eman Elrefai](https://md.research.elamur.ai/free/en/authors/eman-elrefai/), Esraa Khaled and [Alhassan Ehab](https://md.research.elamur.ai/free/en/authors/alhassan-ehab/) · **Year:** 2025 · **Venue:** [Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks](https://md.research.elamur.ai/free/en/venues/arabicnlp/) ([ArabicNLP 2025](https://md.research.elamur.ai/free/en/venues/arabicnlp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2025.arabicnlp-sharedtasks.113/) · [PDF](https://aclanthology.org/2025.arabicnlp-sharedtasks.113.pdf) · [DOI 10.18653/v1/2025.arabicnlp-sharedtasks.113](https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.113) · [OpenAlex](https://openalex.org/W4416037747) **Citations:** 1 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/elrefai-2025-star-palmx-2025-arabic-cultural/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary The paper addresses PalmX 2025 Subtask 1, a four-option multiple-choice benchmark that tests knowledge of Arab culture through questions in Modern Standard Arabic. The authors ask whether a small model can gain cultural competence at low compute cost. They start from the Qwen3-4B base model and train it in two steps, both using LoRA adapters through Unsloth. In the first step, they continue causal-LM pretraining on a corpus built from Arabic Wikipedia. The corpus covers 10 Arab countries and ten domains, including history, religion, festivals and tourism. Articles are cleaned, filtered for Arabic, deduplicated by hashing, cut into chunks of up to 4096 tokens, and labelled with their article titles. In the second step, they instruction-tune the model on the 2,000 official QA items, computing the loss only on the answer tokens. The system reaches 74% on dev and 64% when evaluated on the blind test set, finishing ninth. The untouched base model scores 64% on dev. The work offers a cheap recipe for adding regional knowledge to compact models. Its error analysis shows that coverage gaps and multi-step reasoning remain the main obstacles. ## Key points - Setup: Qwen3-4B with LoRA. - Rank and alpha are both 128, with dropout 0.05 and a maximum length of 4096. - Adapters cover q/k/v/o_proj, gate/up/down_proj, lm_head and embed_tokens. - Training uses AdamW with a cosine schedule and weight decay 0.01. - Schedule: - Pretraining runs 3 epochs at LR 2e-5, with an embedding LR of 5e-6 and batch size 16. About 3% of the corpus is held out to monitor it (reported as 4480 train and 139 dev examples). - QA fine-tuning runs 2 epochs at LR 2e-5 with batch size 16. - Results: 74.0% on dev, 64.0% on the blind test, ninth place. The ablation compares only two configurations, base model (64.0 dev) versus the full system (74.0 dev), so the separate effect of each stage is not isolated. - Error types from manual inspection: - Overlapping plausible answers, such as confusing a media outlet's chief editor with its founder. - Failure to chain knowledge across history and religion. - Missing coverage, such as Mauritania, Yemen and folk traditions. - Stated limitations: no dialectal data, a focus on factual rather than implicit cultural knowledge, and a 10-point dev–test drop that hints at overfitting. ## Interesting ideas - The corpus is gathered on a country × domain grid, and cells with thin Wikipedia content are dropped rather than left sparse, which keeps the resource balanced. - Each text chunk keeps its source article title as metadata, which anchors fragments to their topic after chunking. - Training loss is computed only on response tokens, so the model learns to produce the answer instead of reproducing the prompt. - The error analysis found that answers fail when two people hold different roles toward the same organization, as with founder versus chief editor. - The paper reports an untrained-base control and treats the gap between the dev set and the blind test as an overfitting signal. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (world): Compile knowledge slices from Wikidata/WordNet on an entity-type × domain grid and report coverage per cell, so gaps in the ontology are explicit rather than silent. - **I2** (learn): When producing silver data from long documents, store the document title and id with every sentence so witness parsers, LLM teachers and repair rules can be traced back and reference-resolved in context. - **I3** (learn): When training LLM teacher tiers or gparse auxiliary heads on prompt+target formats, mask the prompt part of the loss to focus capacity on the actual output. - **I4** (frames): Add test items where several agents relate differently to one entity, checking that frames keep distinct VerbNet-style roles instead of merging them. - **I5** (answer): On school-course tests and word problems, always report a baseline run, and keep a blind split that is never used to tune self-induced repair rules. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Cairo University](https://md.research.elamur.ai/free/en/universities/cairo-university/) — University, Giza, Egypt · 43 works · 1,352 citations - [Alexandria University](https://md.research.elamur.ai/free/en/universities/alexandria-university/) — University, Alexandria, Egypt · 42 works · 287 citations - [Minia University](https://md.research.elamur.ai/free/en/universities/minia-university/) — University, Minya, Egypt · 3 works · 134 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works 🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 3,066 works ## Cite Cite as: Eman Elrefai, Esraa Khaled and Alhassan Ehab. 2025. Star at PalmX 2025: Arabic Cultural Understanding via Targeted Pretraining and Lightweight Fine-tuning. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{elrefai2025star, title = {{Star at PalmX 2025: Arabic Cultural Understanding via Targeted Pretraining and Lightweight Fine-tuning}}, author = {Eman Elrefai, Esraa Khaled and Alhassan Ehab}, booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks}, year = {2025}, doi = {10.18653/v1/2025.arabicnlp-sharedtasks.113}, url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.113} } ``` - [← previous SmolLab_SEU at MAHED Shared Task: Do Arabic-Native Encoders Surpass Multilingual Models in Detecting the Nuances of Hope, Hate, and Emotion?](https://md.research.elamur.ai/free/en/papers/rahman-2025-smollab-seu-mahed-shared-task/) - [next → Step-by-step Instructions and a Simple Tabular Output Format Improve the Dependency Parsing Accuracy of LLMs](https://md.research.elamur.ai/free/en/papers/matsuda-2025-step-step-instructions-simple-tabular/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/14/) ## References 8 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Fakhraddin Alwajih, Abdellah El Mekki, Hamdy Mubarak, Majd Hawasly, Abubakr Mohamed, and Muhammad Abdul-Mageed. 2025. [PalmX 2025: The First Shared Task on Benchmarking LLMs on Arabic and Islamic Culture](https://md.research.elamur.ai/free/en/papers/alwajih-2025-palmx-2025-first-shared-task/). In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), Suzhou, China. Association for Computational Linguistics. Co-located with EMNLP 2025, November 5–9. 2. M Saiful Bari, Yazeed Alnumay, Norah A Alzahrani, Nouf M Alotaibi, Hisham A Alyahya, Sultan Al-Rashed, Faisal A Mirza, Shaykhah Z Alsubaie, Hassan A Alahmed, Ghadah Alabduljabbar, and 1 others. 2024. [Allam: Large language models for arabic and english](https://md.research.elamur.ai/free/en/papers/bari-2024-allam-large-language-models-arabic/). arXiv preprint arXiv:2407.15390. [↗](https://arxiv.org/abs/2407.15390) 3. Siddhesh Pawar, Junyeong Park, Jiho Jin, Arnav Arora, Junho Myung, Srishti Yadav, Faiz Ghifari Haznitrama, Inhwa Song, Alice Oh, and Isabelle Augenstein. 2025. [Survey of cultural awareness in language models: Text and beyond](https://md.research.elamur.ai/free/en/papers/pawar-2025-survey-cultural-awareness-language-models/). Computational Linguistics, pages 1–96. [↗](https://doi.org/10.1162/coli.a.14) 4. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, and 1 others. 2023. [Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models](https://md.research.elamur.ai/free/en/papers/sengupta-2023-jais-jais-chat-arabic-centric/). arXiv preprint arXiv:2308.16149. [↗](https://arxiv.org/abs/2308.16149) 5. Kritsada Singhapoo, Akarachai Inthanil, and Attapon Pillai. 2025. Fine-tuning ai models with limited resources. In 2025 11th International Conference on Engineering, Applied Sciences, and Technology (ICEAST), pages 148–151. IEEE. [find](https://scholar.google.com/scholar?q=Kritsada+Singhapoo%2C+Akarachai+Inthanil%2C+and+Attapon+Pillai%2E+2025%2E+Fine%2Dtuning+ai+models+with+limited+resources%2E+In+2025+11th+International+Conference+on+Engineering%2C+Applied+Sciences%2C+and+Technology+%28) 6. Jihoon Tack, Jack Lanchantin, Jane Yu, Andrew Cohen, Ilia Kulikov, Janice Lan, Shibo Hao, Yuandong Tian, Jason Weston, and Xian Li. 2025. Llm pre-training with continuous concepts. arXiv preprint arXiv:2502.08524. [↗](https://arxiv.org/abs/2502.08524) 7. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388. [↗](https://arxiv.org/abs/2505.09388) 8. Dan Zhang, Tao Feng, Lilong Xue, Yuandong Wang, Yuxiao Dong, and Jie Tang. 2025. Parameter-efficient fine-tuning for foundation models. arXiv preprint arXiv:2501.13787. [↗](https://arxiv.org/abs/2501.13787) *Updated 1 November 2026*