---
title: "RightNow-Arabic-0.5B-Turbo — paper by Jaber and Jaber (2026)"
description: "The 4-bit build is small enough for phones. 0.44 points. 19 references. Open Arabic LLMs fall into two groups."
authors: ["Jaber Jaber", "Osama Jaber"]
year: 2026
venue: "arXiv:2605.28827"
source: "https://arxiv.org/abs/2605.28827"
doi: "10.48550/arxiv.2605.28827"
arxiv: "2605.28827"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/jaber-2026-rightnow-arabic-0-5b-turbo/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jaber-2026-rightnow-arabic-0-5b-turbo/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/41/) / RightNow-Arabic-0.5B-Turbo: An Open Sub-1B Arabic Language Model via Vocabulary Injection and Edge-First Deployment

# RightNow-Arabic-0.5B-Turbo: An Open Sub-1B Arabic Language Model via Vocabulary Injection and Edge-First Deployment

**Authors:** Jaber Jaber and Osama Jaber · **Year:** 2026 · **Venue:** arXiv:2605.28827 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2605.28827) · [PDF](https://arxiv.org/pdf/2605.28827) · [DOI 10.48550/arxiv.2605.28827](https://doi.org/10.48550/arxiv.2605.28827) · [arXiv 2605.28827](https://arxiv.org/abs/2605.28827) · [OpenAlex](https://openalex.org/W7162755643)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Open Arabic LLMs fall into two groups. Small multilingual models give Arabic little dedicated capacity, and specialized Arabic models (Jais, AceGPT, ALLaM, SILMA) are 7B or larger and need a server. The only sub-2B Arabic-specific attempt, Kuwain-1.5B, never released its weights. The authors fill this gap with a 518M-parameter decoder built on Qwen2.5-0.5B (base). They add 27,032 Arabic tokens, initialized by averaging the embeddings of each token's old sub-pieces, and continue pretraining on Arabic Wikipedia. They then run SFT with the loss computed only on responses, a DPO pass, and a weight-averaging "soup" of three checkpoints. The model is exported to GGUF through llama.cpp. On COPA-ar, Arabic HellaSwag and ArabicMMLU it averages 35.9%, which beats other open models of its size. It also matches a model three times larger on COPA-ar. The 4-bit build is small enough for phones. The paper also documents where the approach falls short: knowledge-heavy tasks, dialects and preference tuning. Code, weights and benchmark scripts are all released.

## Key points
- Tokenizer surgery: a 32k SentencePiece unigram model was trained on 12.54 GB of normalized Arabic and merged into Qwen's BPE. Pieces already covered by an existing token were dropped, leaving 27,032 new tokens and a vocabulary of 178,697. Fertility fell from 2.18 to 1.80 tokens/word, a 17.3% reduction.
- Training data:
  - Continued pretraining used 504M Arabic Wikipedia tokens over 2,500 steps (about 10.5B tokens seen) on 8×H100 with FSDP, FlashAttention varlen and Liger kernels.
  - SFT used 129,116 deduplicated pairs from five datasets, including CIDAR and the aya_dataset Arabic subset.
  - DPO used 6,750 argilla machine-translated pairs.
- Results:
  - The model averages 35.9% across the three lm-evaluation-harness tasks.
  - It scores 58.4% on COPA-ar, tying Falcon-H1-1.5B.
  - It reaches about 67% of SILMA-9B's mean score.
  - It still trails 7B+ models on ArabicMMLU by more than 29 points.
- Post-training: DPO had essentially no effect, with the loss staying at ln 2 and the reward margin near zero. The best of seven soup variants added 0.44 points over the DPO endpoint.
- Deployment:
  - The q4_k_m build is 398 MB and runs at about 635 tok/s at batch size 1 on an H100 via llama.cpp, compared with 82 tok/s through HF generate().
  - However, 144 tensors fall back to higher-bit formats because the odd vocabulary size does not fit k-quant block sizes, so q4_k_m is effectively 6.45 bits per weight.

## Interesting ideas
- **Vocabulary extension without disturbing the original.** A new vocabulary item is accepted only if the old tokenizer cannot already produce it as a single unit. Its starting representation is then the average of the pieces it used to split into, and the original rows are left untouched.
- **Loss only on the part you care about.** Masking prompt tokens during SFT spends the whole training signal on assistant responses.
- **Averaging checkpoints to recover forgotten skills.** Taking a weighted mean of pretrain, SFT and DPO weights brought back capability that later stages had partly erased, with no extra objective needed.
- **Treat a loss stuck at chance as a verdict.** The DPO loss sitting at ln 2 with zero reward margin showed the stage taught nothing, and the authors reported this openly.
- **Separate skill limits from knowledge limits.** Reasoning-style tasks improved, while the knowledge benchmark stayed capped by model size, so the authors read the two separately.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tokens): When adding multiword expressions or new lemmas to crate en, keep only those that are genuinely new and seed their tagger features from their constituent words. This avoids an unseen-word cold start.
- **I2** (learn): When training on silver trees, compute updates only on arcs where the witness parsers agree. Disputed arcs would contribute no gradient instead of injecting noise.
- **I3** (tree): gparse checkpoints fine-tuned on different treebanks or UD dialect conversions could be souped. This could regain general accuracy lost during specialization, with a simple ablation over mixing weights.
- **I4** (learn): Each training stage should log its distance from the chance-level loss and be checked against a negative control. A stage that leaves these unchanged should be flagged as a no-op rather than counted as progress.
- **I5** (answer): School-test and QA scores could be split into questions needing compiled world knowledge versus questions needing text-internal reasoning. This shows which layer, world or frames, is the real bottleneck.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works

## Cite

Cite as: Jaber Jaber and Osama Jaber. 2026. RightNow-Arabic-0.5B-Turbo: An Open Sub-1B Arabic Language Model via Vocabulary Injection and Edge-First Deployment. arXiv:2605.28827.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{jaber2026rightnow,
  title = {{RightNow-Arabic-0.5B-Turbo: An Open Sub-1B Arabic Language Model via Vocabulary Injection and Edge-First Deployment}},
  author = {Jaber Jaber and Osama Jaber},
  year = {2026},
  doi = {10.48550/arxiv.2605.28827},
  eprint = {2605.28827},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2605.28827}
}
```

- [← previous RIFT: Repurposing Negative Samples via Reward-Informed Fine-Tuning](https://md.research.elamur.ai/free/en/papers/liu-2026-rift-repurposing-negative-samples-reward/)
- [next → Risk-Controlled Event-Driven Cascading Updates for Knowledge Graph Consistency Restoration](https://md.research.elamur.ai/free/en/papers/ni-2026-risk-controlled-event-driven-cascading/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/41/)

## References

19 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. M. Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham A. Alyahya, Sultan AlRashed, et al. [ALLaM: Large Language Models for Arabic and English](https://md.research.elamur.ai/free/en/papers/bari-2024-allam-large-language-models-arabic/). arXiv preprint arXiv:2407.15390, 2024. [↗](https://arxiv.org/abs/2407.15390)
2. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems (NeurIPS), 2022. [find](https://scholar.google.com/scholar?q=Tri+Dao%2C+Daniel+Y%2E+Fu%2C+Stefano+Ermon%2C+Atri+Rudra%2C+and+Christopher+R%C3%A9%2E+FlashAttention%3A+Fast+and+memory%2Defficient+exact+attention+with+IO%2Dawareness%2E+In+Advances+in+Neural+Information+Processing+Systems+)
3. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and Eric Tang. A framework for few-shot language model evaluation. EleutherAI open-source software, 2021. [https://github.com/EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness). [↗](https://github.com/EleutherAI/lm-evaluation-harness)
4. Khalil Hennara, Sara Chrouf, Mohamed Motaism Hamed, Zeina Aldallal, Omar Hadid, and Safwan AlModhayan. [Kuwain 1.5B: An Arabic SLM via language injection](https://md.research.elamur.ai/free/en/papers/hennara-2025-kuwain-1-5b-arabic-slm/). arXiv preprint arXiv:2504.15120, 2025. [↗](https://arxiv.org/abs/2504.15120)
5. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. [Training compute-optimal large language models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/). In Advances in Neural Information Processing Systems (NeurIPS), 2022.
6. Pin-Lun Hsu, Yun Dai, Vignesh Kothapalli, Qingquan Song, Shao Tang, Siyu Zhu, Steven Shimizu, Shivam Sahni, Haowen Ning, and Yanning Chen. Liger Kernel: Efficient Triton kernels for LLM training. arXiv preprint arXiv:2410.10989, 2024. [↗](https://arxiv.org/abs/2410.10989)
7. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Dingjie Song, Zhihong Chen, Abdulmohsen Alharthi, Bang An, Juncai He, Ziche Liu, Zhiyi Zhang, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, and Jinchao Xu. [AceGPT, Localizing Large Language Models in Arabic](https://md.research.elamur.ai/free/en/papers/huang-2024-acegpt-localizing-large-language-models/). arXiv preprint arXiv:2309.12053, 2023. [↗](https://arxiv.org/abs/2309.12053)
8. Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Boda Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. [ArabicMMLU: Assessing massive multitask language understanding in Arabic](https://md.research.elamur.ai/free/en/papers/koto-2024-arabicmmlu-assessing-massive-multitask-language/). arXiv preprint arXiv:2402.12840, 2024. [↗](https://arxiv.org/abs/2402.12840)
9. Taku Kudo and John Richardson. [SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/). In Proceedings of EMNLP: System Demonstrations, 2018.
10. Khalil Hennara, Muhammad Hreden, Mohamed Motaism Hamed, Zeina Aldallal, Sara Chrouf, and Safwan AlModhayan. Mutarjim: Advancing bidirectional Arabic-English translation with a small language model. arXiv preprint arXiv:2505.17894, 2025. [↗](https://arxiv.org/abs/2505.17894)
11. Benjamin Minixhofer, Fabian Paischer, and Navid Rekabsaz. [WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models](https://md.research.elamur.ai/free/en/papers/minixhofer-2022-wechsel-effective-initialization-subword-embeddings/). In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2022. [↗](https://doi.org/10.18653/v1/2022.naacl-main.293)
12. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). In Advances in Neural Information Processing Systems (NeurIPS), 2022.
13. Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro von Werra, and Thomas Wolf. [The FineWeb datasets: Decanting the web for the finest text data at scale](https://md.research.elamur.ai/free/en/papers/penedo-2024-fineweb-datasets-decanting-web-finest/). In Advances in Neural Information Processing Systems (NeurIPS), 2024.
14. Qwen Team: An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024. [↗](https://arxiv.org/abs/2412.15115)
15. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). In Advances in Neural Information Processing Systems (NeurIPS), 2023. [↗](https://doi.org/10.48550/arxiv.2305.18290)
16. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Xudong Han, Sondos Mahmoud Bsharat, Alham Fikri Aji, Zhiqiang Shen, Zhengzhong Liu, Natalia Vassilieva, Joel Hestness, Andy Hock, Andrew Feldman, Jonathan Lee, Andrew Jackson, Hector Xuguang Ren, Preslav Nakov, Timothy Baldwin, and Eric Xing. [Jais and Jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models](https://md.research.elamur.ai/free/en/papers/sengupta-2023-jais-jais-chat-arabic-centric/). arXiv preprint arXiv:2308.16149, 2023. [↗](https://arxiv.org/abs/2308.16149)
17. Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. [Model soups: Averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://md.research.elamur.ai/free/en/papers/wortsman-2022-model-soups-averaging-weights-multiple/). In International Conference on Machine Learning (ICML), 2022. [↗](https://doi.org/10.48550/arxiv.2203.05482)
18. Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. [TinyLlama: An open-source small language model](https://md.research.elamur.ai/free/en/papers/zhang-2024-tinyllama-open-source-small-language/). arXiv preprint arXiv:2401.02385, 2024. [↗](https://arxiv.org/abs/2401.02385)
19. Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. PyTorch FSDP: Experiences on scaling fully sharded data parallel. Proceedings of the VLDB Endowment, 16(12):3848–3860, 2023. [↗](https://doi.org/10.14778/3611540.3611569)

*Updated 1 November 2026*