---
title: "AraSTEM: A Native Arabic Multiple Choice — Al-Sakkaf; 2024"
description: "Most knowledge benchmarks for LLMs are in English, though many models claim multilingual skill."
authors: ["Ahmad Mustapha; Hadi Al-Khansa; Hadi Al-Mubasher; Aya Mourad; Ranam Hamoud; Hasan El-Husseini; Marwah Al-Sakkaf;", "Mariette Awad"]
year: 2024
venue: "arXiv:2501.00559"
source: "https://arxiv.org/abs/2501.00559"
doi: "10.48550/arxiv.2501.00559"
arxiv: "2501.00559"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mustapha-2024-arastem-native-arabic-multiple-choice/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mustapha-2024-arastem-native-arabic-multiple-choice/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / AraSTEM: A Native Arabic Multiple Choice Question Benchmark for Evaluating LLMs Knowledge In STEM Subjects

# AraSTEM: A Native Arabic Multiple Choice Question Benchmark for Evaluating LLMs Knowledge In STEM Subjects

**Authors:** Ahmad Mustapha; Hadi Al-Khansa; Hadi Al-Mubasher; Aya Mourad; Ranam Hamoud; Hasan El-Husseini; Marwah Al-Sakkaf;, [Mariette Awad](https://md.research.elamur.ai/free/en/authors/mariette-awad/) · **Year:** 2024 · **Venue:** arXiv:2501.00559 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2501.00559) · [PDF](https://arxiv.org/pdf/2501.00559) · [DOI 10.48550/arxiv.2501.00559](https://doi.org/10.48550/arxiv.2501.00559) · [arXiv 2501.00559](https://arxiv.org/abs/2501.00559) · [OpenAlex](https://openalex.org/W4406031801)

**Citations:** 1 citations (openalex-snapshot · 2026-10-14) · [14](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Most knowledge benchmarks for LLMs are in English, though many models claim multilingual skill. Arabic has very few evaluation sets written natively in the language. The authors build AraSTEM, a collection of 11,637 Arabic multiple-choice questions that were originally written in Arabic, not translated. It covers primary, secondary and college levels in math, science, physics, chemistry, biology, information technology, medicine, pharmacy and dentistry. They gathered items in three ways: scraping MCQ websites, pulling and reshaping questions from reference books by hand, and running PDFs through ChatGPT 4 as an OCR tool before human proofreading. Every item keeps a link to its origin. The authors profile the data by subject mix, option counts, question length, embedding clusters and tokenizer coverage. They then test many open models zero-shot. The best model, Jais 30B Chat V3, averages only about 0.56 accuracy. Most top models are overconfident, and different models solve different questions. AraSTEM gives Arabic a traceable, STEM-focused yardstick and supports training on more local-language text.

## Key points
- AraSTEM has 11,637 MCQs. The biggest sources are beadaya.com (7,721 scraped items) and www.mehe.gov.lb (1,806 items obtained via LLM extraction). Primary level dominates, with 4,250 primary science and 3,220 primary math questions.
- Questions have between two and four options. Most are 10–15 words long, with a maximum of about 170 words, and medicine has the longest questions. E5 multilingual embeddings projected with UMAP show math clearly separated from other subjects, and the medical subjects clustering together.
- Zero-shot average accuracy by model:
  - Jais 30B Chat V3: about 0.56.
  - Jais 13B Chat and Llama 3.1 8B Instruct: 0.51.
  - Falcon and Llama 2 variants: about 0.29–0.32.
  - Bloomz 7B1 performs roughly on par with AceGPT 13B.
  - Instruction tuning helps Jais much more than the other models.
- Calibration plots show overconfidence on high-confidence predictions for all top models except Llama 3.1.
- No tested model answered 193 questions correctly, mostly in dentistry and medicine. Some questions were solved by only one model: about 600 by Bloomz 7B1 alone and over 550 by Jais alone.

## Interesting ideas
- Every question carries a link to its exact source, so any item can be audited and traced back for data-governance reasons. — For Mova (learn): Tag each silver sentence and test item with its origin, so that bad parses or contaminated sources can be found and removed in bulk.
- Definition and fill-in-the-blank questions from textbooks were turned into multiple-choice items by supplying alternative answers alongside the correct one. — For Mova (answer): Convert school-course reading material into many graded multiple-choice checks for the readers, at little authoring cost.
- Before evaluating, they ran the whole dataset through each model's tokenizer and counted unknown words. This separates vocabulary gaps from knowledge or reasoning gaps. — For Mova (tokens): Run every test set through the en tokenizer and lexicon first, and report out-of-vocabulary lemmas, so answer errors are not wrongly blamed on later stages.
- Calibration plots compare a model's confidence with its real accuracy and reveal overconfidence that plain accuracy hides. — For Mova (answer): Have readers output a confidence score and check its calibration, so a well-calibrated system can say "unknown" instead of guessing.
- An upset plot of which models solved which items shows that the systems are complementary, with large sets of questions only one model gets right. — For Mova (learn): Run the same overlap analysis on the witness parsers, to see where a single parser is uniquely right and tune the agreement rules for silver data.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · [Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/)

## Cited by

Cited by

14 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 11 not yet.

- [MedArabiQ: Benchmarking Large Language Models on Arabic Medical Tasks](https://md.research.elamur.ai/free/en/papers/daoud-2025-medarabiq-benchmarking-large-language-models/)
- [From Guidelines to Practice: A New Paradigm for Arabic Language Model Evaluation](https://md.research.elamur.ai/free/en/papers/sibaee-2025-guidelines-practice-new-paradigm-arabic/)
- [Benchmarking the Medical Understanding and Reasoning of Large Language Models in Arabic Healthcare Tasks](https://md.research.elamur.ai/free/en/papers/aldahoul-2025-benchmarking-medical-understanding-reasoning-large/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{alsakkaf2024arastem,
  title = {{AraSTEM: A Native Arabic Multiple Choice Question Benchmark for Evaluating LLMs Knowledge In STEM Subjects}},
  author = {Al-Sakkaf;, Ahmad Mustapha; Hadi Al-Khansa; Hadi Al-Mubasher; Aya Mourad; Ranam Hamoud; Hasan El-Husseini; Marwah and Awad, Mariette},
  year = {2024},
  doi = {10.48550/arxiv.2501.00559},
  eprint = {2501.00559},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2501.00559}
}
```

- [← previous AraSpell: A Deep Learning Approach for Arabic Spelling Correction](https://md.research.elamur.ai/free/en/papers/salhab-2024-araspell-deep-learning-approach-arabic/)
- [next → Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics](https://md.research.elamur.ai/free/en/papers/perrella-2024-beyond-correlation-interpretable-evaluation-machine/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

5 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. N. Muennighoff, T. Wang, L. Sutawika, A. Roberts, S. Biderman, T. L. Scao, M. S. Bari, S. Shen, Z.-X. Yong, H. Schoelkopf, X. Tang, D. Radev, A. F. Aji, K. Almubarak, S. Albanie, Z. Alyafeai, A. Webson, E. Raff, and C. Raffel, “Crosslingual generalization through multitask finetuning,” 11 2022. [find](https://scholar.google.com/scholar?q=N%2E+Muennighoff%2C+T%2E+Wang%2C+L%2E+Sutawika%2C+A%2E+Roberts%2C+S%2E+Biderman%2C+T%2E+L%2E+Scao%2C+M%2E+S%2E+Bari%2C+S%2E+Shen%2C+Z%2E%2DX%2E+Yong%2C+H%2E+Schoelkopf%2C+X%2E+Tang%2C+D%2E+Radev%2C+A%2E+F%2E+Aji%2C+K%2E+Almubarak%2C+S%2E+Albanie%2C+Z%2E+Alyafeai%2C+A%2E+Webso) [our card](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-crosslingual-generalization-through-multitask-finetuning/)
2. S. Singh, F. Vargus, D. Dsouza, B. F. Karlsson, A. Mahendiran, W.-Y. Ko, H. Shandilya, J. Patel, D. Mataciunas, L. OMahony, M. Zhang, R. Hettiarachchi, J. Wilson, M. Machado, L. S. Moura, D. Krzeminski, H. Fadaei, I. Ergün, I. Okoh, A. Alaagib, O. Mudannayake, Z. Alyafeai, V. M. Chien, S. Ruder, S. Guthikonda, E. A. Alghamdi, S. Gehrmann, N. Muennighoff, M. Bartolo, J. Kreutzer, A. Ustun, M. Fadaee, and S. Hooker, “Aya dataset: An open-access collection for multilingual instruction tuning,” 2 2024. “Measuring mathematical problem solving with the math dataset,” 3 2021. [9] M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, and J. Wei, “Challenging big-bench tasks and whether chain-of-thought can solve them,” 10 2022. [find](https://scholar.google.com/scholar?q=S%2E+Singh%2C+F%2E+Vargus%2C+D%2E+Dsouza%2C+B%2E+F%2E+Karlsson%2C+A%2E+Mahendiran%2C+W%2E%2DY%2E+Ko%2C+H%2E+Shandilya%2C+J%2E+Patel%2C+D%2E+Mataciunas%2C+L%2E+OMahony%2C+M%2E+Zhang%2C+R%2E+Hettiarachchi%2C+J%2E+Wilson%2C+M%2E+Machado%2C+L%2E+S%2E+Moura%2C+D%2E+Krzeminsk) [our card](https://md.research.elamur.ai/free/en/papers/singh-2024-aya-dataset-open-access-collection/)
3. N. Sengupta, S. K. Sahu, B. Jia, S. Katipomu, H. Li, F. Koto, W. Marshall, G. Gosal, C. Liu, [10] D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner, “Drop: A read-Z. Chen, O. M. Afzal, S. Kamboj, O. Pandit, ing comprehension benchmark requiring discrete R. Pal, L. Pradhan, Z. M. Mujahid, M. Baali, reasoning over paragraphs,” 3 2019. X. Han, S. M. Bsharat, A. F. Aji, Z. Shen, Z. Liu, N. Vassilieva, J. Hestness, A. Hock, A. Feld-[11] Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, man, J. Lee, A. Jackson, H. X. Ren, P. Nakov, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, T. Baldwin, and E. Xing, “Jais and jais-chat: T. Li, M. Ku, K. Wang, A. Zhuang, R. Fan, Arabic-centric foundation and instruction-tuned X. Yue, and W. Chen, “Mmlu-pro: A more ro-open generative large language models,” 8 2023. bust and challenging multi-task language understanding benchmark,” 6 2024. [find](https://scholar.google.com/scholar?q=N%2E+Sengupta%2C+S%2E+K%2E+Sahu%2C+B%2E+Jia%2C+S%2E+Katipomu%2C+H%2E+Li%2C+F%2E+Koto%2C+W%2E+Marshall%2C+G%2E+Gosal%2C+C%2E+Liu%2C+%5B10%5D+D%2E+Dua%2C+Y%2E+Wang%2C+P%2E+Dasigi%2C+G%2E+Stanovsky%2C+S%2E+Singh%2C+and+M%2E+Gardner%2C+%E2%80%9CDrop%3A+A+read%2DZ%2E+Chen%2C+O%2E+M%2E+Afzal)
4. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Mea-[12] D. Rein, B. L. Hou, A. C. Stickland, J. Petty, suring massive multitask language understand-R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowing,” 9 2020. man, “Gpqa: A graduate-level google-proof q&a [find](https://scholar.google.com/scholar?q=D%2E+Hendrycks%2C+C%2E+Burns%2C+S%2E+Basart%2C+A%2E+Zou%2C+M%2E+Mazeika%2C+D%2E+Song%2C+and+J%2E+Steinhardt%2C+%E2%80%9CMea%2D%5B12%5D+D%2E+Rein%2C+B%2E+L%2E+Hou%2C+A%2E+C%2E+Stickland%2C+J%2E+Petty%2C+suring+massive+multitask+language+understand%2DR%2E+Y%2E+Pang%2C+J%2E+)
5. R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, benchmark,” 11 2023. and Y. Choi, “Hellaswag: Can a machine really [13] J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, finish your sentence?,” 5 2019. Y. Luan, D. Zhou, and L. Hou, “Instruction-[6] K. Sakaguchi, R. L. Bras, C. Bhagavatula, and following evaluation for large language models,” Y. Choi, “Winogrande: An adversarial winograd 11 2023. schema challenge at scale,” 7 2019. [14] “Multilingual massive multitask language under-[7] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, standing (mmmlu).” https://huggingface. Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. co/datasets/openai/MMMLU. Accessed: 2024-Xing, H. Zhang, J. E. Gonzalez, and I. Sto-10-08. ica, “Judging llm-as-a-judge with mt-bench and chatbot arena,” 6 2023. The aim is to develop [15] F. Koto, H. Li, S. Shatnawi, J. Doughman, A. B. Sadallah, A. Alraeesi, K. Almubarak, a way to evaluate LMMs. Paradoxly enough Z. Alyafeai, N. Sengupta, S. Shehata, N. Habash, the authors claimes GPT-4 can be use to judge P. Nakov, and T. Baldwin, “Arabicmmlu: As-LLMs.¡br/¿While doesnt make sense the authors sessing massive multitask language understand-create MTBench dataset that contain 80 ques-ing in arabic,” 2 2024. tions ans asked both humans and GPT-4 to judge performance of two different models on the [16] O. Alharbi, A. Hamed, and M. Alzoubi, “A questions. GPT-4 tended to agree with human deep neural network optimized by a genetic al-judgment. gorithm to improve arabic sentiment classifica-[8] D. Hendrycks, C. Burns, S. Kadavath, A. Arora, tion,” Ingénierie Des Systèmes D Informa-S. Basart, E. Tang, D. Song, and J. Steinhardt, tion, vol. 28, pp. 67–75, 2023. [17] N. Hajj and M. Awad, “Weighted entropy corti- [24] L. McInnes, J. Healy, and J. Melville, “Umap: cal algorithms for isolated arabic speech recogni-Uniform manifold approximation and projection tion,” in Proceedings of the International Joint for dimension reduction,” 2020. Conference on Neural Networks, Institute of Electrical and Electronics Engineers Inc., 2013. [25] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in [18] H. Alsayadi, A. Abdelhamid, I. Hegazy, and large language models,” 2023. Z. Fayed, “Non-diacritized arabic speech recognition based on cnn-lstm and attention-based [26] H. Huang, F. Yu, J. Zhu, X. Sun, H. Cheng, models,” Journal of Intelligent & Fuzzy Systems, S. Dingjie, Z. Chen, M. Alharthi, B. An, J. He, vol. 41, pp. 6207–6219, 2021. Z. Liu, J. Chen, J. Li, B. Wang, L. Zhang, R. Sun, X. Wan, H. Li, and J. Xu, “AceGPT, [19] G. Bhatia, E. M. B. Nagoudi, F. Alwajih, and localizing large language models in Arabic,” in M. Abdul-Mageed, “Qalam : A multimodal Proceedings of the 2024 Conference of the North llm for arabic optical character and handwriting American Chapter of the Association for Com-recognition,” 7 2024. putational Linguistics: Human Language Technologies (Volume 1: Long Papers) (K. Duh, [20] J. El Zini, Y. Rizk, M. Awad, and J. Antoun, H. Gomez, and S. Bethard, eds.), (Mexico City, “Towards a deep learning question-answering Mexico), pp. 8139–8163, Association for Compu-specialized chatbot for objective structured clin-tational Linguistics, June 2024. ical examinations,” in 2019 International Joint Conference on Neural Networks (IJCNN), pp. 1– [27] Meta-AI-Llama-team, “The llama 3 herd of models,” 2024. 9, 2019. [21] Z. Qiu, X. Wu, and W. Fan, “Question difficulty prediction for multiple choice problems in medical exams,” in Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM19, (New York, NY, USA), pp. 139–148, Association for Computing Machinery, 2019. [28] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre, “Training compute-optimal large language models,” 3 2022. [22] S. Longpre, â. R. Mahari, A. Chen, N. Obeng-Marnu, D. Sileo, W. Brannon, N. Muennighoff, [29] J. E. ZINI and M. AWAD, “On the explain-N. Khazam, J. Kabbara, K. P. Xinyi, A. Wu, ability of natural language processing deep mod-E. Shippole, K. Bollacker, T. Wu, L. Villa, els,” in IEEE International Conference on Pro-S. Pentland, S. Hooker, H. L. School, and U. C. gram Comprehension, vol. 2022-March, pp. 36– Irvine, “The data provenance initiative: A large 47, IEEE Computer Society, 2022. scale audit of dataset licensing & attribution in ai.” [23] L. Wang, N. Yang, X. Huang, L. Yang, R. Majumder, and F. Wei, “Multilingual e5 text embeddings: A technical report,” arXiv preprint arXiv:2402.05672, 2024. [↗](https://arxiv.org/abs/2402.05672) [our card](https://md.research.elamur.ai/free/en/papers/wang-2024-multilingual-e5-text-embeddings-technical/)