--- title: "ArabicNumBench — Alhumud 2026" description: "Arabic text writes digits in two scripts: Eastern Arabic-Indic and Western Arabic." authors: ["Anas Alhumud", "Abdulaziz Alhammadi", "Muhammad Badruddin Khan"] year: 2026 venue: "arXiv:2602.18776" source: "https://arxiv.org/abs/2602.18776" doi: "10.48550/arxiv.2602.18776" arxiv: "2602.18776" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/alhumud-2026-arabicnumbench-evaluating-arabic-number-reading/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/alhumud-2026-arabicnumbench-evaluating-arabic-number-reading/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/2/) / ArabicNumBench: Evaluating Arabic Number Reading in Large Language Models # ArabicNumBench: Evaluating Arabic Number Reading in Large Language Models **Authors:** Anas Alhumud, Abdulaziz Alhammadi, Muhammad Badruddin Khan · **Year:** 2026 · **Venue:** arXiv:2602.18776 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2602.18776) · [PDF](https://arxiv.org/pdf/2602.18776) · [DOI 10.48550/arxiv.2602.18776](https://doi.org/10.48550/arxiv.2602.18776) · [arXiv 2602.18776](https://arxiv.org/abs/2602.18776) · [OpenAlex](https://openalex.org/W7131382857) **Citations:** 0 citations (openalex-snapshot · 2026-10-14) ## Summary Arabic text writes digits in two scripts: Eastern Arabic-Indic and Western Arabic. Existing Arabic LLM benchmarks check whether answers are right but not how they are produced. ArabicNumBench fills this gap with 210 number-reading items in six categories: pure Eastern numerals, pure Western numerals, addresses, dates, quantities and prices. The authors ran 71 models from 10 providers through OpenRouter at temperature 0. Each model was tested under four prompting regimes: zero-shot, zero-shot CoT, few-shot and few-shot CoT. This gives 281 model–strategy runs and 59,010 scored cases. Answers are pulled out by a ranked cascade of extractors, and the benchmark records which extractor fired. This yields a "structured output" rate alongside accuracy and a format-preservation score. Few-shot CoT works best, but the key result is a split between two skills. Several models reach 98–99% accuracy while seldom using the demonstrated answer format. Only a handful follow the format reliably. Accuracy and instruction-following should therefore be measured, and chosen for, separately. ## Key points - Scale: 71 models (Qwen, Mistral, OpenAI, Meta, Google, Anthropic, Cohere, DeepCogito, Microsoft, MoonshotAI), 4 strategies, 210 items with 35 per category, and 59,010 evaluations in total. - Accuracy spans 14.29% to 99.05%. Few-shot CoT averages 80.06%, against 74.45% for plain few-shot and 28.76% for zero-shot. - Structured output appears in 34.12% of few-shot CoT answers versus 7.14% for plain few-shot. The Arabic "final answer" marker shows up in 29.45% of few-shot CoT outputs and at most 1.24% elsewhere. - Only 6 models reach ≥90% structured output (Qwen3 Max, Llama 3.1 405B, Command A, GPT-4o-mini, Claude 3 Opus, Qwen3 235B), while 44 stay under 10%. - Gemini 2.5 Pro scores 99.05% accuracy and Gemma 2 27B scores 98.57%, yet about 93% of their answers come only through fallback extraction. ## Interesting ideas - Log which extraction tier produced each answer, so a correct result reached by a lucky last-number fallback is not counted as equal to one given in the requested format. — For Mova (answer): Each school-test or word-problem answer could carry a tag saying whether it came from the world graph or from a heuristic fallback, and the two rates could be reported separately. - Parse outputs with an ordered cascade from strict delimiters down to loose heuristics, and keep the tier as metadata rather than discarding it. — For Mova (learn): LLM teacher outputs could pass through this cascade, with silver labels from lower tiers down-weighted or sent to the witness parsers for extra agreement. - Choose models for format reliability as well as accuracy, since the paper shows the two skills come apart and prompting cannot fully fix a weak format-follower. — For Mova (learn): Teacher tiers could be ranked on a small format-compliance probe before any of their annotations are trusted at scale. - Present the same values in two surface scripts and inside different contexts (addresses, dates, prices, quantities), and also score whether the original notation is kept. — For Mova (tokens): A paired test set of digits, spelled-out numbers and separator variants could check that the tokenizer and lemmas give one value while keeping the surface form. - Embed one skill, reading a number, in several semantic settings to see whether context changes what the number means. — For Mova (world): The same split could test whether the measures/values layer types a number correctly as a date, price, quantity or address part. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{alhumud2026arabicnumbench, title = {{ArabicNumBench: Evaluating Arabic Number Reading in Large Language Models}}, author = {Alhumud, Anas and Alhammadi, Abdulaziz and Khan, Muhammad Badruddin}, year = {2026}, doi = {10.48550/arxiv.2602.18776}, eprint = {2602.18776}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2602.18776} } ``` - [← previous ArabicDialectSafety: A Dialect-Aware Benchmark for Arabic Content Safety Classification](https://md.research.elamur.ai/free/en/papers/zaghouani-2026-arabicdialectsafety-dialect-aware-benchmark-arabic/) - [next → ArabiGEE: A Hierarchical Taxonomy for Arabic Grammatical Error Explanation](https://md.research.elamur.ai/free/en/papers/elhady-2026-arabigee-hierarchical-taxonomy-arabic-grammatical/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/2/) ## References 25 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano et al., “Training verifiers to solve math word problems,” arXiv preprint arXiv:2110.14168, 2021. [Online]. Available: https://arxiv.org/abs/2110.14168 [↗](https://arxiv.org/abs/2110.14168) [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/) 2. N. Y. Habash, “Introduction to Arabic natural language processing,” Synthesis Lectures on Human Language Technologies, vol. 3, no. 1, pp. 1–187, 2010. [find](https://scholar.google.com/scholar?q=N%2E+Y%2E+Habash%2C+%E2%80%9CIntroduction+to+Arabic+natural+language+processing%2C%E2%80%9D+Synthesis+Lectures+on+Human+Language+Technologies%2C+vol%2E+3%2C+no%2E+1%2C+pp%2E+1%E2%80%93187%2C+2010%2E) 3. E. Kheir Eddine, I. Guellil, S. Khilji, T. Ait Baha, and G. J. Jones, “ArabicMMLU: Assessing massive multitask language understanding in Arabic,” in Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 2024, pp. 4521–4533. [Online]. Available: https://aclanthology.org/2024.lrec-main.397/ [↗](https://aclanthology.org/2024.lrec-main.397/) [our card](https://md.research.elamur.ai/free/en/papers/koto-2024-arabicmmlu-assessing-massive-multitask-language/) 4. R. Al-Matham et al., “BALSAM: A platform for benchmarking arabic large language models,” arXiv preprint arXiv:2507.22603, 2025. [Online]. Available: https://arxiv.org/abs/2507.22603 [↗](https://arxiv.org/abs/2507.22603) 5. E. A. Almubarak, D. Alhamed, S. Alqahtani et al., “AraTrust: An evaluation of trustworthiness for LLMs in Arabic,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024, pp. 8912–8931. [Online]. Available: https://aclanthology.org/2024.emnlp-main.521/ [↗](https://aclanthology.org/2024.emnlp-main.521/) 6. J. Team, “CamelEval: Advancing LLM-as-judge for Arabic language tasks,” arXiv preprint arXiv:2410.03267, 2024. [Online]. Available: https://arxiv.org/abs/2410.03267 [↗](https://arxiv.org/abs/2410.03267) 7. X. Yue, J. Ni, K. Zhang et al., “MMATH: A massively multilingual math word problem benchmark,” arXiv preprint arXiv:2402.10893, 2024. [Online]. Available: https://arxiv.org/abs/2402.10893 [↗](https://arxiv.org/abs/2402.10893) 8. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal et al., “Language models are few-shot learners,” Advances in Neural Information Processing Systems (NeurIPS), vol. 33, pp. 1877–1901, 2020. [Online]. Available: https://arxiv.org/abs/2005.14165 [↗](https://arxiv.org/abs/2005.14165) 9. J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” Advances in Neural Information Processing Systems (NeurIPS), vol. 35, pp. 24 824–24 837, 2022. [Online]. Available: https://arxiv.org/abs/2201.11903 [↗](https://arxiv.org/abs/2201.11903) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/) 10. T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, “Large language models are zero-shot reasoners,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 35, 2022, pp. 22 199–22 213. [Online]. Available: https://arxiv.org/abs/2205.11916 [↗](https://arxiv.org/abs/2205.11916) [our card](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/) 11. X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,” in Proceedings of the International Conference on Learning Representations (ICLR), 2023. [Online]. Available: https://arxiv.org/abs/2203.11171 [↗](https://arxiv.org/abs/2203.11171) [our card](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/) 12. F. Shi, M. Suzgun, M. Freitag, X. Wang, S. Srivats, S. Vosoughi, H. W. Chung, Y. Tay, S. Ruder, D. Zhou et al., “Language models are multilingual chain-of-thought reasoners,” in Proceedings of the International Conference on Learning Representations (ICLR), 2023. [Online]. Available: https://arxiv.org/abs/2210.03057 [↗](https://arxiv.org/abs/2210.03057) [our card](https://md.research.elamur.ai/free/en/papers/shi-2022-language-models-multilingual-chain-thought/) 13. J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou, “Instruction-following evaluation for large language models,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024, pp. 5498–5516. [Online]. Available: https://aclanthology.org/2024.acl-long.299/ [↗](https://aclanthology.org/2024.acl-long.299/) [our card](https://md.research.elamur.ai/free/en/papers/zhou-2023-instruction-following-evaluation-large-language/) 14. C. Chen, C. Tao, J. Zhang, W. Ouyang, T. Chen, M. Yang, and F. Huang, “FoFo: A benchmark to evaluate LLMs’ format-following capability,” arXiv preprint arXiv:2402.18667, 2024. [Online]. Available: https://arxiv.org/abs/2402.18667 [↗](https://arxiv.org/abs/2402.18667) 15. A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou et al., “Qwen2 technical report,” arXiv preprint arXiv:2407.10671, 2024. [Online]. Available: https://arxiv.org/abs/2407.10671 [↗](https://arxiv.org/abs/2407.10671) 16. A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas et al., “Mistral 7B,” arXiv preprint arXiv:2310.06825, 2023. [Online]. Available: https://arxiv.org/abs/2310.06825 [↗](https://arxiv.org/abs/2310.06825) 17. OpenAI, “GPT-4 technical report,” arXiv preprint arXiv:2303.08774, 2024. [Online]. Available: https://arxiv.org/abs/2303.08774 [↗](https://arxiv.org/abs/2303.08774) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/) 18. A. Dubey, A. Jauhri, A. Pandey et al., “The Llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024. [Online]. Available: https://arxiv.org/abs/2407.21783 [↗](https://arxiv.org/abs/2407.21783) 19. R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut et al., “Gemini: A family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805, 2024. [Online]. Available: https://arxiv.org/abs/2312. 11805 [↗](https://arxiv.org/abs/2312.11805) [our card](https://md.research.elamur.ai/free/en/papers/gemini-2023-team-gemini-family-highly-capable-multimodal/) 20. Anthropic, “The Claude 3 model family: Opus, Sonnet, Haiku,” Technical Report, 2024. [Online]. Available: [https://www.anthropic.com/claude](https://www.anthropic.com/claude) [↗](https://www.anthropic.com/claude) 21. Cohere For AI, “Command R: A scalable generative model with multilinguality and RAG optimizations,” Technical Report, 2024. [Online]. Available: [https://txt.cohere.com/command-r/](https://txt.cohere.com/command-r/) [↗](https://txt.cohere.com/command-r/) 22. M. Abdin, S. A. Jacobs, A. A. Awan et al., “Phi-3 technical report: A highly capable language model locally on your phone,” arXiv preprint arXiv:2404.14219, 2024. [Online]. Available: https://arxiv.org/abs/2404.14219 [↗](https://arxiv.org/abs/2404.14219) [our card](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-3-technical-report-highly/) 23. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., “Training language models to follow instructions with human feedback,” Advances in Neural Information Processing Systems (NeurIPS), vol. 35, pp. 27 730–27 744, 2022. [Online]. Available: https://arxiv.org/abs/2203.02155 [↗](https://arxiv.org/abs/2203.02155) 24. J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le, “Finetuned language models are zero-shot learners,” Proceedings of the International Conference on Learning Representations (ICLR), 2022. [Online]. Available: https://arxiv.org/abs/2109.01652 [↗](https://arxiv.org/abs/2109.01652) 25. P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga et al., “Holistic evaluation of language models,” in Transactions on Machine Learning Research, 2023. [Online]. Available: https://arxiv.org/abs/2211.09110 [↗](https://arxiv.org/abs/2211.09110)