--- title: "ATHAR — paper by Mohammed and Khalil (2025)" description: "The public release gives heritage-language translation a shared test bed. 5 references, cited by 4 papers here." authors: ["Mohammed Sabry Mohammed", "Mohammed Khalil"] year: 2025 venue: "Proceedings of The Third Arabic Natural Language Processing Conference" source: "https://aclanthology.org/2025.arabicnlp-main.8/" doi: "10.18653/v1/2025.arabicnlp-main.8" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/mohammed-2025-athar-high-quality-diverse-dataset/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mohammed-2025-athar-high-quality-diverse-dataset/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/12/) / ATHAR: A High-Quality and Diverse Dataset for Classical Arabic to English Translation # ATHAR: A High-Quality and Diverse Dataset for Classical Arabic to English Translation **Authors:** Mohammed Sabry Mohammed and Mohammed Khalil · **Year:** 2025 · **Venue:** [Proceedings of The Third Arabic Natural Language Processing Conference](https://md.research.elamur.ai/free/en/venues/arabicnlp/) ([ArabicNLP 2025](https://md.research.elamur.ai/free/en/venues/arabicnlp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2025.arabicnlp-main.8/) · [PDF](https://aclanthology.org/2025.arabicnlp-main.8.pdf) · [DOI 10.18653/v1/2025.arabicnlp-main.8](https://doi.org/10.18653/v1/2025.arabicnlp-main.8) · [OpenAlex](https://openalex.org/W4416037680) **Citations:** 0 citations (OpenAlex, 2026-10-14) · [4](https://md.research.elamur.ai/free/en/papers/mohammed-2025-athar-high-quality-diverse-dataset/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Classical Arabic is the language of medieval scholarship and literature, and machine translation handles it poorly. Existing parallel resources focus on MSA, dialects, religious texts (Tanzil, Authentic Hadith) or poetry (PCD). The authors built ATHAR, about 66k Classical Arabic–English sentence pairs. They scraped it from rasaif.com, a public library of volunteer translations, covering 18 works from the 8th–14th centuries: history, travel, philosophy, science, medicine, poetry and adab. To clean it, they fixed entries with swapped languages using a script-majority heuristic, removed noise, duplicates and the few Quranic verses, and checked alignment by hand. They compared ATHAR with other corpora on MTLD, stopword ratio and sentence-length distribution, and it comes out lexically varied with a balanced mix of short and long sentences. In benchmarks, GPT-4o and Llama-3 70B lead in zero-shot mode, while small Llama models nearly fail. Fine-tuning Llama-3 8B on the training split, either fully or with LoRA, raises it well above its zero-shot level, though it stays below GPT-4o. The public release gives heritage-language translation a shared test bed. ## Key points - 66k pairs split into 65k train and 1k test, from 18 works of the 8th–14th centuries; released as `mohamed-khalil/ATHAR` on HuggingFace. Language-swapped entries were estimated at 15–20% of the raw scrape. - Table 1 profile of ATHAR: - 138,944 unique words, MTLD 55.63, average sentence length 20.78 words. - 24.06% very short sentences (≤10 words) and 23.11% very long ones (≥30 words). - By contrast, Tanzil has MTLD 101.31 but 47.53% long sentences, and OPUS-100-ar-en is 79.81% short sentences. - Zero-shot scores (METEOR / ROUGE-L / SacreBLEU): - GPT-4o: 0.357 / 0.441 / 14.7 - Llama-3 70B Instruct: 0.342 / 0.413 / 13.0 - Llama-3 8B Instruct: 0.115 / 0.068 / 0.3 - Llama-3 8B with adaptation: - LoRA (default PEFT settings, r=8, α=8): 0.279 / 0.339 / 8.8, slightly ahead of full fine-tuning at 0.275 / 0.336 / 6.1. - Three-shot prompting: 0.174 / 0.167 / 1.0. - For Llama-2 7B, METEOR and ROUGE-L stayed below its zero-shot level for every k in {1, 2, 3, 5}. The authors read this as a model-specific weakness, not a shortage of examples. ## Interesting ideas - Detecting which side of a pair holds which language by counting characters from each alphabet is a cheap, transparent way to repair swapped fields in scraped bilingual data. - Profiling candidate corpora on lexical diversity (MTLD), stopword share and the proportions of very short versus very long sentences shows imbalances that raw size hides. - Sweeping the number of in-context examples separates "the model is weak at this" from "it was given too few demonstrations." - Training examples were built from consecutive sentences of one source document, capped by length, so document boundaries stay intact instead of shuffling isolated sentences. - Reporting several metrics exposes disagreements: LoRA and full tuning are nearly tied on METEOR and ROUGE-L but differ clearly on SacreBLEU. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (learn): Run a similar script/character-majority check on scraped training text so that stray non-English or mislabeled segments never reach the silver-data pipeline. - **I2** (tree): Profile treebanks and silver corpora this way before training the perceptron parser or gparse, so that long, hard sentences are adequately represented. - **I3** (learn): When an LLM teacher tier underperforms with few-shot prompts, a k-sweep tells whether to add examples or replace that teacher. - **I4** (frames): Coreference training and evaluation should use document-contiguous windows so that antecedents remain inside the context. - **I5** (back): Evaluate text → graph → text output with more than one surface metric, plus a meaning-graph comparison, so that no single score misleads. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Dublin City University](https://md.research.elamur.ai/free/en/universities/dublin-city-university/) — University, Dublin, Ireland · 72 works · 9,746 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works 🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 3,066 works ## Cite Cite as: Mohammed Sabry Mohammed and Mohammed Khalil. 2025. ATHAR: A High-Quality and Diverse Dataset for Classical Arabic to English Translation. Proceedings of The Third Arabic Natural Language Processing Conference. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{mohammed2025athar, title = {{ATHAR: A High-Quality and Diverse Dataset for Classical Arabic to English Translation}}, author = {Mohammed Sabry Mohammed and Mohammed Khalil}, booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference}, year = {2025}, doi = {10.18653/v1/2025.arabicnlp-main.8}, url = {https://doi.org/10.18653/v1/2025.arabicnlp-main.8} } ``` - [← previous Are BabyLMs Deaf to Gricean Maxims? A Pragmatic Evaluation of Sample-efficient Language Models](https://md.research.elamur.ai/free/en/papers/askari-2025-are-babylms-deaf-gricean-maxims/) - [next → Automatic Speech Recognition of African American English: Lexical and Contextual Effects](https://md.research.elamur.ai/free/en/papers/mojarad-2025-automatic-speech-recognition-african-american/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/12/) ## References 5 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Manel Aloui, Hasna Chouikhi, Ghaith Chaabane, Haithem Kchaou, and Chehir Dhaouadi. 2024. [101 billion arabic words dataset](https://md.research.elamur.ai/free/en/papers/aloui-2024-101-billion-arabic-words-dataset/). Preprint, arXiv:2405.01590. [↗](https://arxiv.org/abs/2405.01590) 2. M Alrabiah, A Al-Salman, ES Atwell, and N Alhelewh. 2014. Ksucca: a key to exploring arabic historical linguistics. International Journal of Computational Linguistics (IJCL), 5(2):27 – 36. (c) 2014, Alrabiah, M, Al-Salman, A, Atwell, ES and Alhelewh, N. This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial ShareAlike (CC BY-NC-SA 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, provided the original work is properly cited, the use is non-commercial and any derivative works are licensed under the same terms. [find](https://scholar.google.com/scholar?q=M+Alrabiah%2C+A+Al%2DSalman%2C+ES+Atwell%2C+and+N+Alhelewh%2E+2014%2E+Ksucca%3A+a+key+to+exploring+arabic+historical+linguistics%2E+International+Journal+of+Computational+Linguistics+%28IJCL%29%2C+5%282%29%3A27+%E2%80%93+36%2E+%28c%29+2014%2C+A) 3. Shatha Altammami, Eric Atwell, and Ammar Alsalka. 2020. The arabic–english parallel corpus of authentic hadith. International Journal on Islamic Applications in Computer Science And Technology, 8(2). [find](https://scholar.google.com/scholar?q=Shatha+Altammami%2C+Eric+Atwell%2C+and+Ammar+Alsalka%2E+2020%2E+The+arabic%E2%80%93english+parallel+corpus+of+authentic+hadith%2E+International+Journal+on+Islamic+Applications+in+Computer+Science+And+Technology%2C+8%282%29%2E) 4. Satanjeev Banerjee and Alon Lavie. 2005. [METEOR: An automatic metric for MT evaluation with improved correlation with human judgments](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/). In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics. 5. Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Jan Niehues, Sebastian Stüker, Katsuhito Sudoh, Koichiro Yoshino, and Christian Federmann. 2017. [Overview of the IWSLT 2017 evaluation campaign](https://md.research.elamur.ai/free/en/papers/cettolo-2017-overview-iwslt-2017-evaluation-campaign/). In Proceedings of the 14th International Conference on Spoken Language Translation, pages 2–14, Tokyo, Japan. International Workshop on Spoken Language Translation. *Table 2: Performance of State-of-the-Art LLMs on the Classical Arabic to English Translation Task. The table displays METEOR, ROUGE-L, and SacreBLEU scores for various models under different settings: zero-shot (ZS), few-shot with three samples (FS3), and fine-tuning (Full-Tuning & LoRA) on a 1k test set.* systems in this domain. Future work will aim to expand the ATHAR dataset to include an even wider array of texts and topics, thus further enhancing translation quality. *Updated 31 October 2026*