---
title: "Taibah at TAQEEM 2025 — paper by Almarwani et al. (2025)"
description: "The paper describes the Taibah team's entry to the TAQEEM 2025 shared task on… 0.73 points. 7 references, cited by 3 papers here."
authors: ["Nada Almarwani", "Alaa Alharbi", "Samah Aloufi"]
year: 2025
venue: "Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks"
source: "https://aclanthology.org/2025.arabicnlp-sharedtasks.137/"
doi: "10.18653/v1/2025.arabicnlp-sharedtasks.137"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/almarwani-2025-taibah-taqeem-2025-leveraging-gpt/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/almarwani-2025-taibah-taqeem-2025-leveraging-gpt/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/12/) / Taibah at TAQEEM 2025: Leveraging GPT-4o for Arabic Essay Scoring

# Taibah at TAQEEM 2025: Leveraging GPT-4o for Arabic Essay Scoring

**Authors:** [Nada Almarwani](https://md.research.elamur.ai/free/en/authors/nada-almarwani/), [Alaa Alharbi](https://md.research.elamur.ai/free/en/authors/alaa-alharbi/) and [Samah Aloufi](https://md.research.elamur.ai/free/en/authors/samah-aloufi/) · **Year:** 2025 · **Venue:** [Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks](https://md.research.elamur.ai/free/en/venues/arabicnlp/) ([ArabicNLP 2025](https://md.research.elamur.ai/free/en/venues/arabicnlp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.arabicnlp-sharedtasks.137/) · [PDF](https://aclanthology.org/2025.arabicnlp-sharedtasks.137.pdf) · [DOI 10.18653/v1/2025.arabicnlp-sharedtasks.137](https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.137) · [OpenAlex](https://openalex.org/W4416037572)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [3](https://md.research.elamur.ai/free/en/papers/almarwani-2025-taibah-taqeem-2025-leveraging-gpt/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper describes the Taibah team's entry to the TAQEEM 2025 shared task on automated scoring of Arabic essays. Annotated Arabic essay data is scarce, and the task is cross-prompt: systems must score essays on topics they did not see in training. There are two subtasks. Task A gives one holistic score per essay. Task B gives separate scores for seven rubric traits. The authors fine-tune nothing. They prompt GPT-4o with a natural-language rubric written in Arabic and 20 human-scored training essays as examples, all taken from one training prompt. For Task A, the rubric's dimensions come from the task description. For Task B, each trait has its own maximum score and definition. On the two test prompts the system beats the fine-tuned AraBERTv02 baseline in every reported metric. It reaches a holistic QWK of about 0.75 and an average trait QWK of about 0.65. An error analysis shows the model drifts toward middle scores. This matters because it shows that a well-prompted general LLM can be a competitive grader in a low-resource language without any task-specific training.

## Key points
- **Data:** The TAQEEM 2025 data has 1,265 Arabic essays, split 425 for training and 840 for testing. The seven traits are Relevance (0–2), Organization, Vocabulary, Style, Development, Mechanics and Grammar (each up to 5).
- **Setup:** GPT-4o runs through the API at temperature 0.1. The prompt includes 20 exemplars covering the score range. The Task A rubric has six criteria: content clarity, correctness, structure, argument strength, style and word count. It does not use the CAST rubric.
- **Task A:** Average QWK is 0.751 against the baseline's 0.639 (0.717 on prompt 9, 0.784 on prompt 10). Average MSE is 25.438 against 29.005.
- **Task B:** The system wins on every trait. The largest gains are in Relevance (0.562 vs 0.155) and Development (0.703 vs 0.458).
- **Errors:**
  - Holistic predictions cluster on a few values (14, 18, 24, 28), so extreme scores are rarely predicted.
  - The prompt stated a 1–32 score range instead of 0–32, which pushes predictions away from 0.
  - Correlation with human scores is high (Pearson r = 0.87), but the model scores about 0.73 points lower on average, and this gap is statistically significant.
  - Performance is best for essays of 150–200 words.
  - In Task B, Development and Style are scored too low, Vocabulary is scored slightly too high, and Mechanics shows no significant bias.

## Interesting ideas
- Measure signed bias (model minus human) separately for each gold score level, not just overall agreement; this reveals the model's habit of pulling scores toward the middle.
- The label range written in the instruction (1–32 instead of 0–32) visibly changed which outputs the model produced, so how the output space is stated is itself a source of error.
- Take all few-shot examples from a single source prompt and evaluate on different prompts, which directly tests transfer rather than memorization.
- Break evaluation down by input length and compare a ranking metric (QWK) with an absolute error metric (MAE); the two disagree on long essays.
- Split one overall judgment into explicitly defined criteria, each with its own scale and a plain-language definition, which makes the final score easier to interpret.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Silver labels from LLM teacher tiers can be audited by signed bias for each gold bucket. This would show whether a teacher systematically flattens extreme labels before its output becomes training data.
- **I2** (learn): Prompts for the teacher tiers should list the exact label inventory, checked against the treebank or frame schema, so that valid labels are never ruled out by the wording.
- **I3** (answer): Test-taking readers could be given worked examples from one test type or fable collection and evaluated on another, to check that their reasoning carries across domains.
- **I4** (tree): Reporting gparse and perceptron LAS by sentence-length bucket, along with a ranking-style score, could expose length-dependent failures that one average number hides.
- **I5** (answer): Grading answers to school-course tests could produce per-criterion scores linked to the frames and world facts behind them, keeping each step explainable.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Taibah University](https://md.research.elamur.ai/free/en/universities/taibah-university/) — University, Medina, Saudi Arabia · 12 works · 64 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works · [🖍️ Grammatical error correction & learner language](https://md.research.elamur.ai/free/en/topics/grammatical-error-correction/) · 156 works

## Cite

Cite as: Nada Almarwani, Alaa Alharbi and Samah Aloufi. 2025. Taibah at TAQEEM 2025: Leveraging GPT-4o for Arabic Essay Scoring. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{almarwani2025taibah,
  title = {{Taibah at TAQEEM 2025: Leveraging GPT-4o for Arabic Essay Scoring}},
  author = {Nada Almarwani, Alaa Alharbi and Samah Aloufi},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.137},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.137}
}
```

- [← previous TafBERTa: Learning Grammatical Rules from Small-Scale Language Acquisition Data in Hebrew](https://md.research.elamur.ai/free/en/papers/gelboim-2025-tafberta-learning-grammatical-rules-small/)
- [next → Targeted Syntactic Evaluation for Grammatical Error Correction](https://md.research.elamur.ai/free/en/papers/koyama-2025-targeted-syntactic-evaluation-grammatical-error/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/12/)

## References

7 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. [Arabert: Transformer-based model for arabic language understanding](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/). In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. [↗](https://doi.org/10.48550/arxiv.2003.00104)
2. May Bashendy, Salam Albatarni, Sohaila Eltanbouly, Walid Massoud, Houda Bouamor, and Tamer Elsayed. 2025. TAQEEM 2025: Overview of the First Shared Task for Arabic Quality Evaluation of Essays in Multi-dimensions. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), China. [find](https://scholar.google.com/scholar?q=May+Bashendy%2C+Salam+Albatarni%2C+Sohaila+Eltanbouly%2C+Walid+Massoud%2C+Houda+Bouamor%2C+and+Tamer+Elsayed%2E+2025%2E+TAQEEM+2025%3A+Overview+of+the+First+Shared+Task+for+Arabic+Quality+Evaluation+of+Essays+in+Mult)
3. Haiyue Feng, Sixuan Du, Gaoxia Zhu, Yan Zou, Poh Boon Phua, Yuhong Feng, Haoming Zhong, Zhiqi Shen, and Siyuan Liu. 2024. Leveraging large language models for automated chinese essay scoring. In International Conference on Artificial Intelligence in Education, pages 454–467. Springer. [find](https://scholar.google.com/scholar?q=Haiyue+Feng%2C+Sixuan+Du%2C+Gaoxia+Zhu%2C+Yan+Zou%2C+Poh+Boon+Phua%2C+Yuhong+Feng%2C+Haoming+Zhong%2C+Zhiqi+Shen%2C+and+Siyuan+Liu%2E+2024%2E+Leveraging+large+language+models+for+automated+chinese+essay+scoring%2E+In+Inter)
4. Jonas Flodén. 2025. Grading exams using large language models: A comparison between human and ai grading of exams in higher education using chatgpt. British educational research journal, 51(1):201–224. [↗](https://doi.org/10.1002/berj.4069)
5. Masaki Uto and Masashi Okano. 2020. Robust neural automated essay scoring using item response theory. In International conference on artificial intelligence in education, pages 549–561. Springer. [find](https://scholar.google.com/scholar?q=Masaki+Uto+and+Masashi+Okano%2E+2020%2E+Robust+neural+automated+essay+scoring+using+item+response+theory%2E+In+International+conference+on+artificial+intelligence+in+education%2C+pages+549%E2%80%93561%2E+Springer%2E)
6. Yang Yang. 2024. The reliability of using chatgpt in rating efl writings. Shanlax International Journal of Education, 12(4):49–59. [↗](https://doi.org/10.34293/education.v12i4.7855)
7. Fatih Yavuz, Özgür Çelik, and Gamze Yavaş Çelik. 2025. Utilizing large language models for efl essay grading: An examination of reliability and validity in rubric-based assessments. British Journal of Educational Technology, 56(1):150–166. [↗](https://doi.org/10.1111/bjet.13494)

*Updated 1 November 2026*