--- title: "Arabic Train at NADI 2024 shared task: LLMs' Ability to Translate" description: "This paper describes a team entry for Subtask 3 of the NADI 2024 shared task: translating sentences… 9 references, cited by 7 papers here." authors: ["Anastasiia Demidova", "Hanin Atwany", "Nour Rabih", "Sanad Sha'ban"] year: 2024 venue: "Proceedings of the Second Arabic Natural Language Processing Conference" source: "https://aclanthology.org/2024.arabicnlp-1.80/" doi: "10.18653/v1/2024.arabicnlp-1.80" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/demidova-2024-arabic-train-nadi-2024-shared/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/demidova-2024-arabic-train-nadi-2024-shared/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/) / Arabic Train at NADI 2024 shared task: LLMs' Ability to Translate Arabic Dialects into Modern Standard Arabic # Arabic Train at NADI 2024 shared task: LLMs' Ability to Translate Arabic Dialects into Modern Standard Arabic **Authors:** [Anastasiia Demidova](https://md.research.elamur.ai/free/en/authors/anastasiia-demidova/), [Hanin Atwany](https://md.research.elamur.ai/free/en/authors/hanin-atwany/), [Nour Rabih](https://md.research.elamur.ai/free/en/authors/nour-rabih/) and [Sanad Sha'ban](https://md.research.elamur.ai/free/en/authors/sanad-sha-ban/) · **Year:** 2024 · **Venue:** [Proceedings of the Second Arabic Natural Language Processing Conference](https://md.research.elamur.ai/free/en/venues/arabicnlp/) ([ArabicNLP 2024](https://md.research.elamur.ai/free/en/venues/arabicnlp/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2024.arabicnlp-1.80/) · [PDF](https://aclanthology.org/2024.arabicnlp-1.80.pdf) · [DOI 10.18653/v1/2024.arabicnlp-1.80](https://doi.org/10.18653/v1/2024.arabicnlp-1.80) · [OpenAlex](https://openalex.org/W4402671393) **Citations:** 4 citations (OpenAlex, 2026-10-14) · [7](https://md.research.elamur.ai/free/en/papers/demidova-2024-arabic-train-nadi-2024-shared/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary This paper describes a team entry for Subtask 3 of the NADI 2024 shared task: translating sentences from four Arabic dialects (Egyptian, Emirati, Jordanian, Palestinian) into Modern Standard Arabic (MSA). The authors compared several systems: Jais, NLLB, GPT-3.5, GPT-4 and a Google Translate round-trip baseline. They tested prompting in zero-shot and few-shot modes. They also fine-tuned two models on a custom dataset. To build it, they sent Arabic sentences through Google Translate into English or French and back, then paired these noisy outputs with gold MSA. This turns translation into a correction problem. NLLB was fine-tuned this way, and Jais with LoRA. MADAR was also used as a resource. The best setup was Jais with a mixed prompting scheme, one-shot for some dialects and zero-shot for others. It reached 19.48 average BLEU on validation and 20.44 on test, which placed the team second. Fine-tuning hurt rather than helped. The study suggests that a model pre-trained on Arabic beats larger general-purpose LLMs on this task. ## Key points - The validation set has 400 sentences (100 per dialect) and the test set 2,000 (500 per dialect). Some sentences are long (one has 58 words) or metaphorical. - Validation average BLEU by system: - Jais combination approach: 19.48 - Jais 0-shot: 18.72 - GPT-3.5 2-shot: 16.67 - GPT-4 2-shot: 16.53 - Google Translate: 11.93 - Jais fine-tuned on custom data: 9.58 - NLLB fine-tuned on custom data: 8.77 - In the combination approach, Egyptian and Emirati inputs received one-shot prompts, while Jordanian and Palestinian inputs were given no examples at all. Jais already appeared to handle the last two well without examples. - Generation settings were temperature 0.1 and top_p 0.4. The few-shot examples stayed fixed within each dialect: 2-shot for GPT and 1-shot for Jais. - The submitted GPT-3.5 2-shot run scored 0 BLEU on the test set, which the authors attribute to hallucination. Most remaining errors came from named entities and culture-specific words such as dish names. ## Interesting ideas - Choose the prompting regime separately for each language variety from validation scores, since examples help on some varieties and hurt on others. - Recast translation as correction: give the model a rough machine draft and train it to repair the draft toward the gold target. - Push sentences through a pivot language and back to get paraphrased, noisy variants of the same content. - Report scores for each variety alongside the overall average, so a strong mean cannot hide a weak subgroup. - The error analysis found that culture-bound items such as local dishes and named entities cause most mistakes. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (learn): LLM teacher tiers could use per-genre or per-register prompt configurations selected on held-out data, instead of one global prompt. - **I2** (back): text generation from the meaning graph could produce a crude draft first and then apply explainable repair steps, logging each fix. - **I3** (learn): round-trip variants could serve as robustness tests and negative controls, checking that parses and frames stay stable when only the surface wording changes. - **I4** (tree): parser evaluation could always break down LAS per treebank and per UD dialect conversion, not only give a pooled number. - **I5** (world): the system could flag such entities and resolve them through the Wikidata/WordNet ontology before downstream reasoning, rather than treating them as ordinary words. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works 🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works ## Cite Cite as: Anastasiia Demidova, Hanin Atwany, Nour Rabih and Sanad Sha'ban. 2024. Arabic Train at NADI 2024 shared task: LLMs' Ability to Translate Arabic Dialects into Modern Standard Arabic. Proceedings of the Second Arabic Natural Language Processing Conference. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{demidova2024arabic, title = {{Arabic Train at NADI 2024 shared task: LLMs' Ability to Translate Arabic Dialects into Modern Standard Arabic}}, author = {Anastasiia Demidova, Hanin Atwany, Nour Rabih and Sanad Sha'ban}, booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference}, year = {2024}, doi = {10.18653/v1/2024.arabicnlp-1.80}, url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.80} } ``` - [← previous WorryWords: Norms of Anxiety Association for over 44k English Words](https://md.research.elamur.ai/free/en/papers/mohammad-2024-worrywords-norms-anxiety-association-over/) - [next → AraPoemBERT: A Pretrained Language Model for Arabic Poetry Analysis](https://md.research.elamur.ai/free/en/papers/qarah-2024-arapoembert-pretrained-language-model-arabic/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/) ## References 9 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Muhammad Abdul-Mageed, Amr Keleg, AbdelRahim Elmadany, Chiyu Zhang, Injy Hamed, Walid Magdy, Houda Bouamor, and Nizar Habash. 2024. [NADI 2024: The Fifth Nuanced Arabic Dialect Identification Shared Task](https://md.research.elamur.ai/free/en/papers/mageed-2024-nadi-2024-fifth-nuanced-arabic/). In Proceedings of The Second Arabic Natural Language Processing Conference (ArabicNLP 2024). 2. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The MADAR Arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Houda+Bouamor%2C+Nizar+Habash%2C+Mohammad+Salameh%2C+Wajdi+Zaghouani%2C+Owen+Rambow%2C+Dana+Abdulrahim%2C+Ossama+Obeid%2C+Salam+Khalifa%2C+Fadhl+Eryani%2C+Alexander+Erdmann%2C+and+Kemal+Oflazer%2E+2018%2E+The+MADAR+Arabic+di) 3. Wael Farhan, Bashar Talafha, Analle Abuammar, Ruba Jaikat, Mahmoud Al-Ayyoub, Ahmad Bisher Tarakji, and Anas Toma. 2020. Unsupervised dialectal neural machine translation. Information Processing & Management, 57(3):102181. [↗](https://doi.org/10.1016/j.ipm.2019.102181) 4. G42. 2023. Jais. [https://inceptioniai.org/](https://inceptioniai.org/) jais/. Accessed: [24/04/2024]. [↗](https://inceptioniai.org/) 5. Karima Kadaoui, Samar M. Magdy, Abdul Waheed, Md Tawkat Islam Khondaker, Ahmed Oumar El-Shangiti, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. [Tarjamat: Evaluation of bard and chatgpt on machine translation of ten arabic varieties](https://md.research.elamur.ai/free/en/papers/kadaoui-2023-tarjamat-evaluation-bard-chatgpt-machine/). 6. Amr Keleg and Walid Magdy. 2023. [Arabic dialect identification under scrutiny: Limitations of single-label classification](https://md.research.elamur.ai/free/en/papers/keleg-2023-arabic-dialect-identification-under-scrutiny/). arXiv preprint arXiv:2310.13661. [↗](https://arxiv.org/abs/2310.13661) 7. OpenAI. 2023. GPT-4 Model. [https://www.openai.com](https://www.openai.com). Accessed: [24/04/2024]. [↗](https://www.openai.com) 8. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Preprint, arXiv:2203.02155. [↗](https://arxiv.org/abs/2203.02155) 9. Hassan Sawaf. 2010. [Arabic dialect handling in hybrid machine translation](https://md.research.elamur.ai/free/en/papers/sawaf-2010-arabic-dialect-handling-hybrid-machine/). In Proceedings of the 9th Conference of the Association for Machine Translation in the Americas: Research Papers, Denver, Colorado, USA. Association for Machine Translation in the Americas. *Updated 1 November 2026*