--- title: "ARxHYOKA at TAQEEM2025 — paper by Alnajjar et al. (2025)" description: "This paper describes the ARxHYOKA entry to TAQEEM2025 Task B. The task is to score Arabic… 2 references, cited by 3 papers here." authors: ["Mohamad Alnajjar", "Ahmad Almoustafa", "Tomohiro Nishiyama", "Shoko Wakamiya", "Eiji Aramaki", "Takuya Matsuzaki"] year: 2025 venue: "Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks" source: "https://aclanthology.org/2025.arabicnlp-sharedtasks.135/" doi: "10.18653/v1/2025.arabicnlp-sharedtasks.135" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/alnajjar-2025-arxhyoka-taqeem2025-comparative-approaches-arabic/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/alnajjar-2025-arxhyoka-taqeem2025-comparative-approaches-arabic/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/12/) / ARxHYOKA at TAQEEM2025: Comparative Approaches to Arabic Essay Trait Scoring # ARxHYOKA at TAQEEM2025: Comparative Approaches to Arabic Essay Trait Scoring **Authors:** Mohamad Alnajjar, Ahmad Almoustafa, [Tomohiro Nishiyama](https://md.research.elamur.ai/free/en/authors/tomohiro-nishiyama/), [Shoko Wakamiya](https://md.research.elamur.ai/free/en/authors/shoko-wakamiya/), [Eiji Aramaki](https://md.research.elamur.ai/free/en/authors/eiji-aramaki/) and [Takuya Matsuzaki](https://md.research.elamur.ai/free/en/authors/takuya-matsuzaki/) · **Year:** 2025 · **Venue:** [Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks](https://md.research.elamur.ai/free/en/venues/arabicnlp/) ([ArabicNLP 2025](https://md.research.elamur.ai/free/en/venues/arabicnlp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2025.arabicnlp-sharedtasks.135/) · [PDF](https://aclanthology.org/2025.arabicnlp-sharedtasks.135.pdf) · [DOI 10.18653/v1/2025.arabicnlp-sharedtasks.135](https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.135) · [OpenAlex](https://openalex.org/W4416037581) **Citations:** 0 citations (OpenAlex, 2026-10-14) · [3](https://md.research.elamur.ai/free/en/papers/alnajjar-2025-arxhyoka-taqeem2025-comparative-approaches-arabic/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary This paper describes the ARxHYOKA entry to TAQEEM2025 Task B. The task is to score Arabic student essays on seven traits from the CAST rubric: Relevance is rated from 0 to 2, while the remaining six (Vocabulary, Style, Organization, Grammar, Development and Mechanics) are each rated from 0 to 5. The hard part is that test essays answer writing prompts that never appear in training. The team compared four system families: - GPT-4.1 scoring with a structured chain-of-thought prompt and in-context examples; - encoder fine-tuning (mDeBERTa-v3-base, XLM-R-large, CAMeLBERT-mix) with a joint seven-output regression head; - classical regressors on CAMeLBERT-mix embeddings, optionally with 14 handcrafted linguistic features; - LoRA adaptation of the Arabic generator Saka-14B on hand-written rubric reasoning chains. All tuning used a cross-genre protocol (explanatory essays for training, persuasive for evaluation), followed by retraining on both genres. On the official test, GPT-4.1 with 10 examples gave the best mean QWK (0.612 against a 0.472 baseline) and also among the smallest errors. Every family beat the baseline on the test data. The work offers an inexpensive, side-by-side picture of which modelling choices carry over to unseen prompts in Arabic AES. ## Key points - **Data:** development phase used prompts 1 (explanatory, 215 essays) and 2 (persuasive, 210). Testing used prompts 9 and 10 (420 essays each). Essays are about 300 words and were rated by experts against the CAST rubrics. - **GPT-4.1 prompting:** - Switching rubric and prompts to English (translated by GPT-4.1) raised QWK from 0.539 to 0.579. - During experiments, 0/1/5/10 shots gave 0.579/0.597/0.603/0.631. - All API use cost roughly USD 21 for about 21.3M tokens. - **Development phase (mean QWK):** - baseline 0.582 - mDeBERTa-v3-base (lr 2e−5, last 6 layers trainable) 0.575 - GPT 10-shot 0.564 - Ridge with AVG pooling plus linguistic features 0.539 - Saka-14B with LoRA 0.480 - **Official test (mean QWK):** - GPT 10-shot 0.612, GPT 1-shot 0.610 (lowest MSE 0.758 and RMSE 0.845) - classical ML 0.582 - fine-tuned BERT 0.554 - Saka-14B 0.538 - baseline 0.472 - **Error tendencies:** - GPT had the most exact matches. - BERT and Saka-14B tended to score too high, especially on Vocabulary and Style. - Ridge tended to score too low, most clearly on Organization and Development. ## Interesting ideas - Choose every hyperparameter and prompt under the same kind of shift expected at test time (one genre in, another out), and only then retrain on everything; this makes model selection honest about transfer. - Alongside the agreement metric, report per category the share of exact, too-high and too-low predictions, which exposes systematic bias that a single score hides. - Appending a small set of shallow, interpretable linguistic counts to dense embeddings gave a consistent lift for simple linear regressors. - The language in which the rubric and instructions are written measurably changed the LLM's scoring quality, so instruction language is a tunable variable worth testing on held-out data. - Force a strict output format, then parse it mechanically, and snap continuous predictions to the valid range of each field by rounding and clamping. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (learn): when picking repair rules or parser settings, train on one text genre (say, tales) and score on another (say, school tests) before merging the silver data, so gains that only hold in-genre are filtered out. - **I2** (learn): evaluation reports for witness parsers and students could split errors by direction per label (e.g. attaching too high vs. too low, overcounting vs. undercounting quantities), turning a flat accuracy into actionable repair targets. - **I3** (tokens): the tagger and morphology layer could expose explicit, explainable features (affix, length, frequency cues) next to learned representations, keeping decisions traceable while gaining accuracy. - **I4** (learn): for LLM teacher tiers, systematically compare instruction wordings and formats on a small gold set before trusting a teacher's silver labels. - **I5** (answer): readers answering tests or word problems could emit structured, machine-checkable answers and project them onto the set of admissible values (e.g. integer counts, allowed options), catching malformed outputs early. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Nara Institute of Science and Technology](https://md.research.elamur.ai/free/en/universities/nara-institute-of-science-and-technology/) — University, Ikoma, Japan · 46 works · 4,565 citations - [Tokyo University of Science](https://md.research.elamur.ai/free/en/universities/tokyo-university-of-science/) — University, Tokyo, Japan · 2 works · 694 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works 🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works ## Cite Cite as: Mohamad Alnajjar, Ahmad Almoustafa, Tomohiro Nishiyama, Shoko Wakamiya, Eiji Aramaki and Takuya Matsuzaki. 2025. ARxHYOKA at TAQEEM2025: Comparative Approaches to Arabic Essay Trait Scoring. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{alnajjar2025arxhyoka, title = {{ARxHYOKA at TAQEEM2025: Comparative Approaches to Arabic Essay Trait Scoring}}, author = {Mohamad Alnajjar, Ahmad Almoustafa, Tomohiro Nishiyama, Shoko Wakamiya, Eiji Aramaki and Takuya Matsuzaki}, booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks}, year = {2025}, doi = {10.18653/v1/2025.arabicnlp-sharedtasks.135}, url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.135} } ``` - [← previous Are Dialects Better Prompters? A Case Study on Arabic Subjective Text Classification](https://md.research.elamur.ai/free/en/papers/moudjari-2025-are-dialects-better-prompters-case/) - [next → Athar at QIAS2025: LLM-based Question Answering Systems for Islamic Inheritance and Classical Islamic Knowledge](https://md.research.elamur.ai/free/en/papers/noureldien-2025-athar-qias2025-llm-based-question/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/12/) ## References 2 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. May Bashendy, Salam Albatarni, Sohaila Eltanbouly, Walid Massoud, Houda Bouamor, and Tamer Elsayed. 2025. TAQEEM 2025: Overview of the First Shared Task for Arabic Quality Evaluation of Essays in Multi-dimensions. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), China. [find](https://scholar.google.com/scholar?q=May+Bashendy%2C+Salam+Albatarni%2C+Sohaila+Eltanbouly%2C+Walid+Massoud%2C+Houda+Bouamor%2C+and+Tamer+Elsayed%2E+2025%2E+TAQEEM+2025%3A+Overview+of+the+First+Shared+Task+for+Arabic+Quality+Evaluation+of+Essays+in+Mult) 2. May Bashendy, Salam Albatarni, Sohaila Eltanbouly, Eslam Zahran, Hager Elhuseyin, Tamer Elsayed, Walid Massoud, and Houda Bouamor. 2024. [QAES: First Publicly-Available Trait-Specific Annotations for Automated Scoring of Arabic Essays](https://md.research.elamur.ai/free/en/papers/bashendy-2024-qaes-first-publicly-available-trait/). In Proceedings of the Second Arabic Natural Language Processing Conference (ArabicNLP 2024). *Figure 2: Testing-phase calibration across traits and models. Bars show the proportion of predictions that were exact matches (Same), overestimates (Over), or underestimates (Under); stacks sum to 100%.* *Updated 31 October 2026*