---
title: "Benchmarking Arabic--Russian Machine — Arabov 2026"
description: "Arabic–Russian translation has received little research attention."
authors: ["Mullosharaf K. Arabov"]
year: 2026
venue: "arXiv:2609.29559"
source: "https://arxiv.org/abs/2609.29559"
doi: "10.48550/arxiv.2609.29559"
arxiv: "2609.29559"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/arabov-2026-benchmarking-arabic-russian-machine-translation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/arabov-2026-benchmarking-arabic-russian-machine-translation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/4/) / Benchmarking Arabic--Russian Machine Translation: A Comparison of Fine-tuned NMT and Few-shot LLMs under Rich Morphology and Low Lexical Overlap

# Benchmarking Arabic--Russian Machine Translation: A Comparison of Fine-tuned NMT and Few-shot LLMs under Rich Morphology and Low Lexical Overlap

**Authors:** [Mullosharaf K. Arabov](https://md.research.elamur.ai/free/en/authors/mullosharaf-k-arabov/) · **Year:** 2026 · **Venue:** arXiv:2609.29559 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2609.29559) · [PDF](https://arxiv.org/pdf/2609.29559) · [DOI 10.48550/arxiv.2609.29559](https://doi.org/10.48550/arxiv.2609.29559) · [arXiv 2609.29559](https://arxiv.org/abs/2609.29559) · [OpenAlex](https://openalex.org/W7214292957)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Arabic–Russian translation has received little research attention. Arabic combines root-and-pattern morphology with heavy cliticization, and the two languages share almost no surface forms. The paper builds a controlled benchmark from the Arabic–Russian Translation Corpus. That corpus has 15.8M raw pairs, 15.47M after filtering, mostly from OPUS. The author samples a 20k/5k/5k train/validation/test split to mimic a low-resource setting.

Seven NMT systems are adapted on one 24 GB L4 GPU, using full fine-tuning, LoRA or QLoRA:
- Marian
- mT5-small, mT5-base and mT5-large
- NLLB-600M and NLLB-1.3B
- M2M100

These are compared against four instruction-tuned LLMs run through Ollama in zero-shot and five-shot mode. Scoring uses BLEU, chrF, BERTScore, COMET and TER; uncertainty is estimated by bootstrapping, and systems are compared pairwise for statistical significance. A rule-based error taxonomy is applied to the worst outputs.

NLLB-1.3B with QLoRA comes out on top (BLEU 16.3, COMET 0.738). The strongest LLM, Aya-Expanse 8B, lags well behind. The study gives this language pair its first systematic baseline. It also shows that when parallel data is scarce, a dedicated fine-tuned NMT model is still the better choice than a general-purpose LLM.

## Key points
- **Data preparation:** filtering removed empty segments, duplicates and overlong segments (>2,000 chars / 250 words). Arabic diacritics were stripped. A manual check of 500 random pairs found 96% correctly aligned. The split was 20k/5k/5k with seed 42.
- **Quality versus training cost:**
  - NLLB-1.3B (QLoRA) reached BLEU 16.3 but needed 373.5 minutes of training.
  - Marian with full fine-tuning reached 15.48 BLEU in 10 minutes, using 3.7 GB of peak memory.
  - Full fine-tuning of mT5-small used the most memory (14.1 GB).
- **Best LLM:** Aya-Expanse 8B, five-shot on 500 sentences, scored BLEU 1.7, chrF 25.7, BERTScore 0.675 and COMET 0.654. On the full 5,000 sentences it scored BLEU 5.6 and COMET 0.612. The other LLMs were llama3.1:8b, qwen2.5:7b-instruct and mistral:7b-instruct.
- **Error analysis:** low lexical overlap accounts for 84–96% of the 50 worst translations of each NMT model. Among mT5-small's worst outputs, 32% are too short. LLM outputs read fluently but pick synonyms that differ from the reference. LLMs also lose up to 4 BLEU on the longest sentence quantiles.
- **Metric agreement:**
  - chrF, BERTScore and COMET are strongly correlated (Spearman ρ ≥ 0.81).
  - The 500-sentence subset tracks the full test set closely (NLLB-1.3B: 16.40 vs 16.30 BLEU).
  - mT5-base beats mT5-small on BLEU (6.54 vs 1.51), yet their TER is similar (106.7 vs 103.7).

## Interesting ideas
- The worst outputs are sorted into transparent rule-based buckets (too short, too long, repetition, character-Jaccard below 0.2, other), with thresholds set from the training length distribution, giving a cheap and explainable failure profile per system. — For Mova (back): The same thresholded buckets could label failures of the text → graph → text compressor, separating dropped content, runaway repetition and wrong-word outputs without an opaque judge.
- When BLEU and TER disagree, BLEU is shown to flatter weak systems by crediting scattered n-gram hits, so ranking on one metric alone can mislead. — For Mova (learn): Evaluation reports could always pair an overlap metric with an edit-distance one, and flag runs where they diverge as candidates for inspection.
- Before trusting a cheap 500-item evaluation subset, the author checks that it reproduces the full-set score within bootstrap confidence intervals. — For Mova (learn): Expensive LLM-teacher or parser comparisons could run on a validated small slice once that slice is shown to match the full set within its CIs.
- LLMs are screened in three stages: zero-shot, then few-shot ranking on the same subset, then full-scale testing only for the winner, which saves compute while keeping a fair head-to-head. — For Mova (learn): Choosing which LLM teacher tier to use for silver data could follow this funnel instead of running every candidate on the full set.
- The paper warns that a single reference turns legitimate paraphrases into apparent low-overlap errors, which is exactly how fluent LLM output with different synonyms gets penalised. — For Mova (answer): Grading free-text answers should compare frames and roles rather than surface word overlap, so correct paraphrased answers are not marked wrong.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/)

Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · [Morphology](https://md.research.elamur.ai/free/en/topics/morphology/) · [Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{arabov2026benchmarking,
  title = {{Benchmarking Arabic--Russian Machine Translation: A Comparison of Fine-tuned NMT and Few-shot LLMs under Rich Morphology and Low Lexical Overlap}},
  author = {Arabov, Mullosharaf K.},
  year = {2026},
  doi = {10.48550/arxiv.2609.29559},
  eprint = {2609.29559},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2609.29559}
}
```

- [← previous Benchmarking Arabic Authorship Attribution and Style Transfer with Large Language Models](https://md.research.elamur.ai/free/en/papers/hamed-2026-benchmarking-arabic-authorship-attribution-style/)
- [next → Benchmarking Large Language Models for Game Localization Quality Assurance: A Cross-Model, Cross-Lingual Analysis](https://md.research.elamur.ai/free/en/papers/tian-2026-benchmarking-large-language-models-game/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/4/)

## References

20 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mohammed Alrekabee. Arabic NLP: A survey of pre-processing and representation techniques. Journal of Computer Networks, Architecture and High Performance Computing, 7(4), 2025. URL [https://itscience-indexing.com/jurnal/index.php/CNAPC/article/view/6383](https://itscience-indexing.com/jurnal/index.php/CNAPC/article/view/6383). [↗](https://itscience-indexing.com/jurnal/index.php/CNAPC/article/view/6383)
2. ArabicNLPWorld. Arabic-russian parallel corpus, 2026. URL [https://huggingface.co/datasets/ArabicNLPWorld/arabic-russian-parallel-corpus](https://huggingface.co/datasets/ArabicNLPWorld/arabic-russian-parallel-corpus). 116,393 curated pairs, 6 sources: religion, dictionary, Bible, Tatoeba, news, conversation. [↗](https://huggingface.co/datasets/ArabicNLPWorld/arabic-russian-parallel-corpus)
3. M. K. Arabov. Bridging scientific heritage: An Arabic–Russian parallel corpus and LLM benchmark for sustainable knowledge transfer. arXiv preprint, arXiv:2606.30943, 2026a. URL https://arxiv.org/abs/2606.30943. [↗](https://arxiv.org/abs/2606.30943) [our card](https://md.research.elamur.ai/free/en/papers/arabov-2026-bridging-scientific-heritage-arabic-russian/)
4. M. K. Arabov. A systematic benchmark of machine transliteration models for the Tajik-Farsi language pair: A comparative study from rule-based to transformer architectures. arXiv preprint, arXiv:2605.02270, 2026b. URL https://arxiv.org/abs/2605.02270. [↗](https://arxiv.org/abs/2605.02270)
5. M. K. Arabov and S. S. Khaybullina. Adapting large language models to a low-resource agglutinative language: A comparative study of LoRA and QLoRA for Bashkir. arXiv preprint, arXiv:2605.04948, 2026. URL https://arxiv.org/abs/2605.04948. [↗](https://arxiv.org/abs/2605.04948)
6. Mullosharaf Kurbonovich Arabov. Arabic-russian translation corpus. [https://huggingface.co/datasets/ArabicNLPWorld/arabic-russian-translation-corpus](https://huggingface.co/datasets/ArabicNLPWorld/arabic-russian-translation-corpus), 2026c. 15,801,992 sentence pairs, MIT license. [↗](https://huggingface.co/datasets/ArabicNLPWorld/arabic-russian-translation-corpus)
7. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. The MADAR Arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, 2018. European Language Resources Association (ELRA). URL https://aclanthology.org/L18-1535/. [↗](https://aclanthology.org/L18-1535/)
8. Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loïc Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. No language left behind: Scaling human-centered machine translation. arXiv preprint, 2022. URL https://arxiv.org/abs/2207.04672. [↗](https://arxiv.org/abs/2207.04672)
9. Kareem Darwish and Hamdy Mubarak. Farasa: A new fast and accurate Arabic word segmenter. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 1070–1074, Portorož, Slovenia, 2016. European Language Resources Association (ELRA). URL https://aclanthology.org/L16-1170/. [↗](https://aclanthology.org/L16-1170/)
10. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: efficient finetuning of quantized llms. In Proceedings of the 37th International Conference on Neural Information Processing Systems, New Orleans, LA, USA, 2023. Curran Associates Inc. [find](https://scholar.google.com/scholar?q=Tim+Dettmers%2C+Artidoro+Pagnoni%2C+Ari+Holtzman%2C+and+Luke+Zettlemoyer%2E+Qlora%3A+efficient+finetuning+of+quantized+llms%2E+In+Proceedings+of+the+37th+International+Conference+on+Neural+Information+Processing+)
11. Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48, 2021. URL [https://jmlr.org/papers/v22/20-1307.html](https://jmlr.org/papers/v22/20-1307.html). [↗](https://jmlr.org/papers/v22/20-1307.html)
12. Nizar Habash and Owen Rambow. Arabic tokenization, part-of-speech tagging and morphological disambiguation in one fell swoop. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 573–580, Ann Arbor, Michigan, 2005. Association for Computational Linguistics. doi:10.3115/1219840.1219911. URL https://aclanthology.org/P05-1071/. [↗](https://doi.org/10.3115/1219840.1219911) [our card](https://md.research.elamur.ai/free/en/papers/aclbulk-2005-habash-arabic-tokenization-part-speech-tagging/)
13. Clive Holes and Roger Allen. Modern Arabic: Structures, Functions, and Varieties. Georgetown University Press, Washington, DC, revised edition, 2004. [find](https://scholar.google.com/scholar?q=Clive+Holes+and+Roger+Allen%2E+Modern+Arabic%3A+Structures%2C+Functions%2C+and+Varieties%2E+Georgetown+University+Press%2C+Washington%2C+DC%2C+revised+edition%2C+2004%2E)
14. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint, 2021. URL https://arxiv.org/abs/2106.09685. [↗](https://arxiv.org/abs/2106.09685)
15. Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. Marian: Fast neural machine translation in C++. In Proceedings of ACL 2018, System Demonstrations, pages 116–121, Melbourne, Australia, 2018. Association for Computational Linguistics. doi:10.18653/v1/P18-4020. URL https://aclanthology.org/P18-4020/. [↗](https://doi.org/10.18653/v1/p18-4020) [our card](https://md.research.elamur.ai/free/en/papers/junczys-2018-dowmunt-marian-fast-neural-machine-translation/)
16. Arfath Pasha, Mohamed Al-Badrashiny, Mona Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. MADAMIRA: A fast, comprehensive tool for morphological analysis and disambiguation of Arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 1094–1101, Reykjavik, Iceland, 2014. European Language Resources Association (ELRA). URL https://aclanthology.org/L14-1479/. [↗](https://aclanthology.org/L14-1479/) [our card](https://md.research.elamur.ai/free/en/papers/lingbulk-2014-pasha-madamira-fast-comprehensive-tool-morphological/)
17. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online, 2020. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ricardo+Rei%2C+Craig+Stewart%2C+Ana+C+Farinha%2C+and+Alon+Lavie%2E+COMET%3A+A+neural+framework+for+MT+evaluation%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP) [our card](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/)
18. Janet C. E. Watson. The Phonology and Morphology of Arabic. Oxford University Press, Oxford, 2007. doi:10.1093/oso/9780199257591.001.0001. [↗](https://doi.org/10.1093/oso/9780199257591.001.0001)
19. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online, 2021. Association for Computational Linguistics. doi:10.18653/v1/2021.naacl-main.41. URL https://aclanthology.org/2021.naacl-main.41/. [↗](https://doi.org/10.18653/v1/2021.naacl-main.41)
20. Wajdi Zaghouani. Critical survey of the freely available arabic corpora. arXiv preprint, arXiv:1702.07835, 2014. URL https://arxiv.org/abs/1702.07835. [↗](https://arxiv.org/abs/1702.07835)