--- title: "Creating Arabic LLM Prompts at Scale — El-Sheikh 2024" description: "Arabic has very few instruction-style training examples (prompts paired with gold answers)." authors: ["Abdelrahman El-Sheikh", "Ahmed Elmogtaba", "Kareem Darwish", "Muhammad Elmallah", "Ashraf Elneima", "Hassan Sawaf"] year: 2024 venue: "arXiv:2408.05882" source: "https://arxiv.org/abs/2408.05882" doi: "10.48550/arxiv.2408.05882" arxiv: "2408.05882" licence: "CC BY-NC-ND 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by-nc-nd/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/el-2024-sheikh-creating-arabic-llm-prompts-scale/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/el-2024-sheikh-creating-arabic-llm-prompts-scale/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / Creating Arabic LLM Prompts at Scale # Creating Arabic LLM Prompts at Scale **Authors:** [Abdelrahman El-Sheikh](https://md.research.elamur.ai/free/en/authors/abdelrahman-el-sheikh/), Ahmed Elmogtaba, [Kareem Darwish](https://md.research.elamur.ai/free/en/authors/kareem-darwish/), Muhammad Elmallah, Ashraf Elneima, Hassan Sawaf · **Year:** 2024 · **Venue:** arXiv:2408.05882 · **Licence:** [CC BY-NC-ND 4.0 (arXiv)](http://creativecommons.org/licenses/by-nc-nd/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2408.05882) · [PDF](https://arxiv.org/pdf/2408.05882) · [DOI 10.48550/arxiv.2408.05882](https://doi.org/10.48550/arxiv.2408.05882) · [arXiv 2408.05882](https://arxiv.org/abs/2408.05882) · [OpenAlex](https://openalex.org/W4402387169) **Citations:** 0 citations (openalex-snapshot · 2026-10-14) · 1 in our library, of the 140884 papers whose bibliographies we read ## Summary Arabic has very few instruction-style training examples (prompts paired with gold answers). Such data is what turns a base LLM into an assistant that follows requests. The authors build this data cheaply in two ways. First, they write PromptSource templates over about 78 public Arabic NLP datasets, which yields roughly 67.5 million prompts. Second, they machine-translate the English PromptSource and Super-NaturalInstructions collections. Each translated prompt is scored sentence by sentence with the reference-free COMET-QE estimator, only high-scoring prompts are kept, and the results are spot-checked by hand. This adds close to 20 million prompts, for more than 87 million in total, spanning 67 task types. Using LoRA, they tune the base Qwen2 7B on random samples of 800K and 8M prompts. They then compare it with existing Arabic and multilingual chat models on a held-out test set, scored with ROUGE-L. The 8M-tuned 7B model matches, and slightly exceeds, Llama3-Instruct 70B. This suggests that targeted instruction data can matter more than model size for a lower-resource language. Both methods can be applied to other languages. ## Key points - Template route: 2–8 templates per dataset, each set taking about 30–90 minutes to write, with varied wording and field order. This produced 67,488,303 prompts. - Translation route: - Pipeline steps: Moses-style sentence splitting, a Helsinki opus-mt Marian model chosen over paid APIs, and COMET-QE with a 0.7 threshold applied to every sentence. About 20% of prompts survived. - Kept output: 19,555,120 prompts from 43 PromptSource datasets and 345,168 from 320 Super-NaturalInstructions tasks. - Excluded: English-specific tasks and tasks left with fewer than 10 prompts. - Test set: at most 50 items per dataset, giving 13,462 prompts from 643 datasets. - Fine-tuning: LoRA on Qwen2 7B for one epoch, learning rate 2e-4, warmup ratio 0.1, weight decay 0.01, with a fixed Arabic system prompt. - Average ROUGE-L scores: | Model | ROUGE-L | |---|---| | AceGPT 7B | 0.102 | | Jais 13B | 0.138 | | Qwen2-Instruct 7B | 0.143 | | Qwen2 7B, 800K tuned | 0.184 | | Llama3 70B | 0.221 | | Qwen2 7B, 8M tuned | 0.224 | Significance was checked with a paired t-test over per-dataset scores. The 8M model's lead over Llama3 70B is not significant. ## Interesting ideas - A translated example is rejected if even one of its sentences falls below the quality threshold, so a single bad sentence cannot hide behind a good average. — For Mova (learn): Silver trees could be gated the same way, dropping a sentence whenever any single clause's arcs lack agreement among the witness parsers, rather than averaging agreement over the whole sentence. - Writing several templates per source dataset, with different phrasing and different field order, multiplies data and discourages the model from latching onto one surface form. — For Mova (answer): School-test and word-problem items could be reworded through several templates to check that the readers answer from the meaning graph rather than from familiar phrasing. - Hand-inspecting a small random sample from each dataset exposed systematic faults that can be fixed once. Examples were "passage" being rendered as a walkway, and numeric lists that never needed translating. — For Mova (back): A per-domain sample review of generated or translated output could catch recurring sense errors, which could then become fixed lexical choices in the meta-world translation path. - Before converting anything, tasks that depend on English-only phenomena are filtered out, since their labels cannot survive a change of language. — For Mova (tree): When converting treebanks between UD dialects, constructions that the target scheme cannot represent could be flagged and set aside rather than forced into the conversion. - The test set is capped at 50 items per dataset, and significance is computed with paired tests over per-dataset scores, so no huge dataset dominates and claims carry statistical backing. — For Mova (learn): Evaluation across many small test suites could use per-suite caps and paired tests between system versions, alongside the negative controls. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) Topics: [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{elsheikh2024creating, title = {{Creating Arabic LLM Prompts at Scale}}, author = {El-Sheikh, Abdelrahman and Elmogtaba, Ahmed and Darwish, Kareem and Elmallah, Muhammad and Elneima, Ashraf and Sawaf, Hassan}, year = {2024}, doi = {10.48550/arxiv.2408.05882}, eprint = {2408.05882}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2408.05882} } ``` - [← previous Computational Methods for the Analysis of Complementizer Variability in Language and Literature: The Case of Hebrew "she-" and "ki"](https://md.research.elamur.ai/free/en/papers/aclbulk-2024-shmidman-computational-methods-analysis-complementizer-variability/) - [next → D-Nikud: Enhancing Hebrew Diacritization with LSTM and Pretrained Models](https://md.research.elamur.ai/free/en/papers/hbo-2024-rosenthal-d-nikud-enhancing-hebrew-diacritization/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) ## References 3 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Stephen H. Bach, Victor Sanh, Zheng-Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-David, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Alan Fries, Maged S. Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Xiangru Tang, Mike Tian-Jian Jiang, and Alexander M. Rush. 2022. Promptsource: An integrated development environment and repository for natural language prompts. [find](https://scholar.google.com/scholar?q=Stephen+H%2E+Bach%2C+Victor+Sanh%2C+Zheng%2DXin+Yong%2C+Albert+Webson%2C+Colin+Raffel%2C+Nihal+V%2E+Nayak%2C+Abheesht+Sharma%2C+Taewoon+Kim%2C+M+Saiful+Bari%2C+Thibault+Fevry%2C+Zaid+Alyafeai%2C+Manan+Dey%2C+Andrea+Santilli%2C+Zhiqi) [our card](https://md.research.elamur.ai/free/en/papers/bach-2022-promptsource-integrated-development-environment-repository/) 2. Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world’s first truly open instruction-tuned llm. [find](https://scholar.google.com/scholar?q=Mike+Conover%2C+Matt+Hayes%2C+Ankit+Mathur%2C+Jianwei+Xie%2C+Jun+Wan%2C+Sam+Shah%2C+Ali+Ghodsi%2C+Patrick+Wendell%2C+Matei+Zaharia%2C+and+Reynold+Xin%2E+2023%2E+Free+dolly%3A+Introducing+the+world%E2%80%99s+first+truly+open+instruct) 3. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Task No. of Datasets AceGPT 7B Answer Verification Answerability Classification Cause Effect Classification Classification Coherence Classification Command Interpretation Commonsense Validation Coreference Resolution Data to Text Definition Generation Diacritization Dialect Classification Dialogue Generation Disease Mention Identification Duplicate Question Identification Emotion Detection Ethics Classification Explanation Fact Verification Fill in The Blank Grammar Correction Inference Detection Information Extraction Information Retrieval Instruction Generation Intent Identification Keyword Extraction Logical Reasoning Misc. Named Entity Recognition Natural Language Inference News Article Generation Next Action Prediction Offensive Language Detection Paraphrasing Poem Generation Problem Identification Program Execution Query Classification Question Answering Question Categorization Question Decomposition Question Generation Question Understanding Reading Comprehension Review Rating Prediction Riddle Solving Sarcasm Detection Semantic Similarity Sentence Composition Sentence Ordering Sentiment Analysis Spam Detection Stereotype Detection Story Composition Style Transfer Summarization Text Categorization Text Classification Text Completion Text Generation Text Simplification Title Generation Topic Identification Translation Transliteration Wrong Candidate Generation [↗](https://arxiv.org/abs/2407.21783)