---
title: "Resource-Aware Arabic LLM Creation — paper by Aryan (2024)"
description: "The paper asks whether a mid-sized multilingual LLM can be specialised for Arabic on very modest… 23 references, cited by 1 paper here."
authors: ["Prakash Aryan"]
year: 2024
venue: "arXiv:2412.17548"
source: "https://arxiv.org/abs/2412.17548"
doi: "10.48550/arxiv.2412.17548"
arxiv: "2412.17548"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/aryan-2024-resource-aware-arabic-llm-creation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/aryan-2024-resource-aware-arabic-llm-creation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/12/) / Resource-Aware Arabic LLM Creation: Model Adaptation, Integration, and Multi-Domain Testing

# Resource-Aware Arabic LLM Creation: Model Adaptation, Integration, and Multi-Domain Testing

**Authors:** Prakash Aryan · **Year:** 2024 · **Venue:** [arXiv:2412.17548](https://md.research.elamur.ai/free/en/venues/communications-in-computer-and-information-science/) ([Communications in computer and information science 2024](https://md.research.elamur.ai/free/en/venues/communications-in-computer-and-information-science/2024/)) · **Publisher:** [Springer Science+Business Media](https://md.research.elamur.ai/free/en/publishers/springer-science-business-media/) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2412.17548) · [PDF](https://arxiv.org/pdf/2412.17548) · [DOI 10.48550/arxiv.2412.17548](https://doi.org/10.48550/arxiv.2412.17548) · [arXiv 2412.17548](https://arxiv.org/abs/2412.17548) · [OpenAlex](https://openalex.org/W4408237521)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/aryan-2024-resource-aware-arabic-llm-creation/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks whether a mid-sized multilingual LLM can be specialised for Arabic on very modest hardware. The author fine-tunes Qwen2-1.5B with QLoRA, which combines 4-bit weight quantization with trainable low-rank adapters, on a machine with just 4GB of VRAM. Gradient accumulation and mixed-precision training help keep memory within that limit. The training mix draws on three sources: the Bactrian corpus, the Arabic part of OpenAssistant and selected Arabic Wikipedia articles. Together they cover mainly Modern Standard Arabic, with some dialectal material.

Over 10,000 steps the training loss falls to 0.1083. Evaluation covers several tasks:
- **Question answering:** small gains.
- **Dialects:** scores broken down by variety.
- **Noisy input:** a clear improvement in robustness.

Gaps remain. No exact-match answers are produced, and the Maghrebi dialects are handled worst. The broader point is practical: adapting a multilingual base model to one language with parameter-efficient methods can be done on consumer GPUs, which opens this work to researchers without large compute budgets.

## Key points
- Setup: Qwen2-1.5B adapted with QLoRA on 4GB VRAM, using gradient accumulation and mixed precision. Final training loss was 0.1083 after 10,000 steps.
- Data: the Bactrian corpus, the Arabic subset of OpenAssistant, and curated Arabic Wikipedia content.
- Question answering: neither the base nor the tuned model produced any exact match. Average F1 rose from 0.25 (base) to 0.3 (tuned).
- Per-dialect results (perplexity / BLEU / F1):

  | Variety | Perplexity | BLEU | F1 |
  |---|---|---|---|
  | MSA (best) | 4.72 | 42.3 | 0.83 |
  | EGY | 6.31 | 36.8 | 0.76 |
  | LEV | 6.54 | 35.9 | 0.75 |
  | GLF | 6.85 | 35.2 | 0.74 |
  | MGR (worst) | 7.12 | 33.7 | 0.71 |

  The discussion also notes that base and tuned models had similar perplexity on dialectal text.
- Noise robustness: at the highest noise level (0.5), the base model's semantic similarity falls to 0, whereas the tuned model still scores 3.5.

## Interesting ideas
- The paper plots base and adapted models side by side across a rising scale of input corruption, which shows where each one breaks rather than giving a single clean-data score.
- Results are split by language variety (MSA vs. four dialects) instead of one averaged number, which exposes the weakest subpopulation.
- Reporting token-level F1 next to exact match reveals partial correctness even when no answer is fully right.
- Combining 4-bit quantization, low-rank adapters, gradient accumulation and mixed precision makes adapting a 1.5B model feasible on a 4GB card.
- The training mix pairs conversational instruction data with encyclopedic prose to widen coverage of genre and style.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Inject typos and character noise into evaluation sentences at graded levels, then chart how LAS and frame accuracy decay; this works as a negative-control stress test for each new parser version.
- **I2** (tree): Report parser scores separately per register, treebank and genre, so that a weak domain is not hidden inside an overall LAS average.
- **I3** (answer): Score answers to fable and school-test questions on both an exact and an overlap metric, so that "almost right" answers are told apart from complete misses.
- **I4** (tree): The same memory-saving recipe could let gparse, or a local LLM teacher tier, be fine-tuned on cheap GPUs without shrinking the effective batch size.
- **I5** (learn): When building silver training data, mix dialogue-like and expository sources on purpose so that parsers and frame extractors see both registers.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

Also on the byline: [Birla Institute of Technology and Science, Pilani - Dubai Campus](https://ror.org/05jnbme07).

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works

## Cite

Cite as: Prakash Aryan. 2024. Resource-Aware Arabic LLM Creation: Model Adaptation, Integration, and Multi-Domain Testing. arXiv:2412.17548.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{aryan2024resource,
  title = {{Resource-Aware Arabic LLM Creation: Model Adaptation, Integration, and Multi-Domain Testing}},
  author = {Prakash Aryan},
  year = {2024},
  doi = {10.48550/arxiv.2412.17548},
  eprint = {2412.17548},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2412.17548}
}
```

- [← previous Quranic Audio Dataset: Crowdsourced and Labeled Recitation from Non-Arabic Speakers](https://md.research.elamur.ai/free/en/papers/salameh-2024-quranic-audio-dataset-crowdsourced-labeled/)
- [next → SemanticCuetSync at AraFinNLP2024: Classification of Cross-Dialect Intent in the Banking Domain using Transformers](https://md.research.elamur.ai/free/en/papers/paran-2024-semanticcuetsync-arafinnlp2024-classification-cross-dialect/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/12/)

## References

23 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abdul-Mageed, M., Elmadany, A.R., Nagoudi, E.M.B.: ARBERT & MAR-BERT: Deep bidirectional transformers for Arabic. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics. pp. 7088–7105. Association for Computational Linguistics (2021) [find](https://scholar.google.com/scholar?q=Abdul%2DMageed%2C+M%2E%2C+Elmadany%2C+A%2ER%2E%2C+Nagoudi%2C+E%2EM%2EB%2E%3A+ARBERT+%26+MAR%2DBERT%3A+Deep+bidirectional+transformers+for+Arabic%2E+In%3A+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Lingui)
2. Alduailej, A., Alothaim, A.: AraXLNet: pre-trained language model for sentiment analysis of Arabic. Journal of Big Data 9(1), 72 (2022) [find](https://scholar.google.com/scholar?q=Alduailej%2C+A%2E%2C+Alothaim%2C+A%2E%3A+AraXLNet%3A+pre%2Dtrained+language+model+for+sentiment+analysis+of+Arabic%2E+Journal+of+Big+Data+9%281%29%2C+72+%282022%29)
3. Alshaikh, K.A., Almatrafi, O., Abushark, Y.B.: BERT-based model for aspect-based sentiment analysis for analyzing Arabic open-ended survey responses: A case study. IEEE Access (2023) [find](https://scholar.google.com/scholar?q=Alshaikh%2C+K%2EA%2E%2C+Almatrafi%2C+O%2E%2C+Abushark%2C+Y%2EB%2E%3A+BERT%2Dbased+model+for+aspect%2Dbased+sentiment+analysis+for+analyzing+Arabic+open%2Dended+survey+responses%3A+A+case+study%2E+IEEE+Access+%282023%29)
4. Antoun, W., Baly, F., Hajj, H.: [AraGPT2: Pre-trained transformer for Arabic language generation](https://md.research.elamur.ai/free/en/papers/antoun-2021-aragpt2-pre-trained-transformer-arabic/). arXiv preprint arXiv:2012.15520 (2020) [↗](https://arxiv.org/abs/2012.15520)
5. Bai, J., et al.: [Qwen technical report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/). arXiv preprint arXiv:2309.16609 (2023) [↗](https://arxiv.org/abs/2309.16609)
6. Brown, T.B., Mann, B., Ryder, N., et al.: [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). In: Advances in Neural Information Processing Systems. vol. 2020-December. Neural information processing systems foundation (2020)
7. Conneau, A., Khandelwal, K., Goyal, N., et al.: Unsupervised cross-lingual representation learning at scale. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 8440–8451. Association for Computational Linguistics (2020) [find](https://scholar.google.com/scholar?q=Conneau%2C+A%2E%2C+Khandelwal%2C+K%2E%2C+Goyal%2C+N%2E%2C+et+al%2E%3A+Unsupervised+cross%2Dlingual+representation+learning+at+scale%2E+In%3A+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics)
8. Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L.: [QLoRA: Efficient finetuning of quantized LLMs](https://md.research.elamur.ai/free/en/papers/dettmers-2023-qlora-efficient-finetuning-quantized-llms/). In: Advances in Neural Information Processing Systems. vol. 36. Neural information processing systems foundation (2023)
9. Dighe, P., et al.: Leveraging large language models for exploiting ASR uncertainty. In: ICASSP. pp. 12231–12235. IEEE (2024) [find](https://scholar.google.com/scholar?q=Dighe%2C+P%2E%2C+et+al%2E%3A+Leveraging+large+language+models+for+exploiting+ASR+uncertainty%2E+In%3A+ICASSP%2E+pp%2E+12231%E2%80%9312235%2E+IEEE+%282024%29)
10. Gubelmann, R.: Large language models, agency, and why speech acts are beyond them (For now) – A Kantian-Cum-Pragmatist case. Philosophy and Technology 37(1), 1–24 (2024) [find](https://scholar.google.com/scholar?q=Gubelmann%2C+R%2E%3A+Large+language+models%2C+agency%2C+and+why+speech+acts+are+beyond+them+%28For+now%29+%E2%80%93+A+Kantian%2DCum%2DPragmatist+case%2E+Philosophy+and+Technology+37%281%29%2C+1%E2%80%9324+%282024%29)
11. Guellil, I., Saâdane, H., Azouaou, F., Gueni, B., Nouvel, D.: [Arabic natural language processing: An overview](https://md.research.elamur.ai/free/en/papers/guellil-2019-arabic-natural-language-processing-overview/). Journal of King Saud University -Computer and Information Sciences 33(5), 497–507 (2021)
12. Habash, N.Y.: Introduction to Arabic Natural Language Processing. Springer International Publishing, Cham (2010) [find](https://scholar.google.com/scholar?q=Habash%2C+N%2EY%2E%3A+Introduction+to+Arabic+Natural+Language+Processing%2E+Springer+International+Publishing%2C+Cham+%282010%29)
13. Hashmi, E., et al.: Enhancing multilingual hate speech detection: From language-specific insights to cross-linguistic integration. IEEE Access (2024) [find](https://scholar.google.com/scholar?q=Hashmi%2C+E%2E%2C+et+al%2E%3A+Enhancing+multilingual+hate+speech+detection%3A+From+language%2Dspecific+insights+to+cross%2Dlinguistic+integration%2E+IEEE+Access+%282024%29)
14. Hu, E., Shen, Y., Wallis, P., et al.: [LoRA: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In: ICLR 2022 - 10th International Conference on Learning Representations. International Conference on Learning Representations (2021)
15. Jo, W., Park, J.: An empirical study of korean cross-lingual summarization based on large language models (2024) [find](https://scholar.google.com/scholar?q=Jo%2C+W%2E%2C+Park%2C+J%2E%3A+An+empirical+study+of+korean+cross%2Dlingual+summarization+based+on+large+language+models+%282024%29)
16. Joshi, P., Santy, S., Budhiraja, A., Bali, K., Choudhury, M.: [The state and fate of linguistic diversity and inclusion in the NLP world](https://md.research.elamur.ai/free/en/papers/joshi-2020-state-fate-linguistic-diversity-inclusion/). In: Proceedings of the Annual Meeting of the Association for Computational Linguistics. pp. 6282–6293. Association for Computational Linguistics (2020)
17. Köpf, A., et al.: OpenAssistant conversations – democratizing large language model alignment. arXiv preprint arXiv:2304.[07327 (2023)](https://md.research.elamur.ai/free/en/papers/kopf-2023-openassistant-conversations-democratizing-large-language/) [↗](https://arxiv.org/abs/2304.07327)
18. Min, Z., Wang, J.: Exploring the integration of large language models into automatic speech recognition systems: An empirical study. Communications in Computer and Information Science 1968 CCIS, 69–84 (2024) [find](https://scholar.google.com/scholar?q=Min%2C+Z%2E%2C+Wang%2C+J%2E%3A+Exploring+the+integration+of+large+language+models+into+automatic+speech+recognition+systems%3A+An+empirical+study%2E+Communications+in+Computer+and+Information+Science+1968+CCIS%2C+69%E2%80%9384)
19. Pfeiffer, J., Rücklé, A., Poth, C., Kamath, A., Vulić, I., Ruder, S., Cho, K., Gurevych, I.: [AdapterHub: A framework for adapting transformers](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-adapterhub-framework-adapting-transformers/). arXiv preprint arXiv:2007.07779 (2020) [↗](https://arxiv.org/abs/2007.07779)
20. Rust, P., Pfeiffer, J., Vulic, I., Ruder, S., Gurevych, I.: [How good is your tokenizer? On the monolingual performance of multilingual language models](https://md.research.elamur.ai/free/en/papers/rust-2021-how-good-is-your-tokenizer/). In: ACL-IJCNLP 2021 - 59th Annual Meeting of the Association for Computational Linguistics. pp. 3118–3135. Association for Computational Linguistics (2021)
21. Soularidis, A., Litaina, T., Bouchouras, G., Kotis, K., Kavakli, E.: Towards LLM-based semantic analysis of historical legal documents (2024) [find](https://scholar.google.com/scholar?q=Soularidis%2C+A%2E%2C+Litaina%2C+T%2E%2C+Bouchouras%2C+G%2E%2C+Kotis%2C+K%2E%2C+Kavakli%2C+E%2E%3A+Towards+LLM%2Dbased+semantic+analysis+of+historical+legal+documents+%282024%29)
22. Tiedemann, J.: Parallel data, tools and interfaces in OPUS. In: Proceedings of the Eighth International Conference on Language Resources and Evaluation. pp. 2214–2218 (2012) [find](https://scholar.google.com/scholar?q=Tiedemann%2C+J%2E%3A+Parallel+data%2C+tools+and+interfaces+in+OPUS%2E+In%3A+Proceedings+of+the+Eighth+International+Conference+on+Language+Resources+and+Evaluation%2E+pp%2E+2214%E2%80%932218+%282012%29)
23. Tunstall, L., Reimers, N., Jo, U.E.S., Bates, L., Korat, D., Wasserblat, M., Pereg, O.: Efficient few-shot learning without prompts. arXiv preprint arXiv:2209.11055 (2022) [↗](https://arxiv.org/abs/2209.11055)

*Updated 31 October 2026*