---
title: "AR-ASAG An ARabic Dataset for Automatic — Ouahrani 2020"
description: "Automatic grading of short free-text answers needs gold-standard test collections, and before this work Arabic had none that were publicly available."
authors: ["Leila Ouahrani", "Djamal Bennouar"]
year: 2020
venue: "Proceedings of the Twelfth Language Resources and Evaluation Conference"
source: "https://aclanthology.org/2020.lrec-1.321/"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ouahrani-2020-ar-asag-arabic-dataset-automatic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ouahrani-2020-ar-asag-arabic-dataset-automatic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/) / AR-ASAG An ARabic Dataset for Automatic Short Answer Grading Evaluation

# AR-ASAG An ARabic Dataset for Automatic Short Answer Grading Evaluation

**Authors:** Leila Ouahrani, Djamal Bennouar · **Year:** 2020 · **Venue:** Proceedings of the Twelfth Language Resources and Evaluation Conference · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.lrec-1.321/) · [PDF](https://aclanthology.org/2020.lrec-1.321.pdf) · [OpenAlex](https://openalex.org/W3031374736)

**Citations:** 13 citations (openalex-snapshot · 2026-10-14) · [5](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Automatic grading of short free-text answers needs gold-standard test collections, and before this work Arabic had none that were publicly available. The authors build AR-ASAG from real exams in a cybercrime course taught in Arabic to master students. Each student answer is paired with a teacher's model answer and scored by two human experts. They also propose an unsupervised grader for low-resource settings, which needs only a plain text corpus and a stemmer. The grader learns word vectors with COALS, a co-occurrence-based distributional method. It builds answer vectors by summing word vectors and adds term weighting and a shared-word component, then compares the student answer with the model answer. Experiments vary three factors: how domain-specific the training corpus is, the dimension of the semantic space, and the stemming technique. The approach is tested on AR-ASAG and on the Arabic-Arabic STS 250 set from SemEval-2017. The authors call the results promising and intend them as a baseline. The resource matters because earlier Arabic grading work used private or ad hoc examples, so methods could not be compared.

## Key points
- AR-ASAG has 2133 (Model Answer, Student Answer) pairs and is distributed as txt, xml, Moodle xml and .db (ISLRN 529-005-230-448-6). A companion Arabic Cyber Text Corpus is also released.
- The data comes from three exams of 16 questions each (48 questions in total), taken under normal exam conditions by about 170 native Arabic-speaking students. Duplicate answers to a question are kept only once.
- There are five question types: define, explain, what consequences, justify, and what is the difference. Each question has one model answer.
- Two computer science teachers graded every answer from 0 to 5. The average of their two grades is the gold score.
- For comparison, the only earlier Arabic dataset (Cairo University) has 610 answers and is not public. The English Texas dataset has 630 answers. Evaluation also uses STS 250 SemEval-2017 track 1.

## Interesting ideas
- Identical student answers are stored only once, so frequent stock answers cannot inflate agreement scores or bias the evaluation toward easy cases. — For Mova (learn): Before computing accuracy, deduplicate test and silver items by their normalized surface form, so that repeated easy sentences do not mask failures on rarer ones.
- Grading is framed as comparing a student answer with a single teacher reference answer, so no per-question training or hand-written patterns are needed. — For Mova (answer): A student reader could check an answer by aligning its frames and world entities with those of a reference answer, producing an explainable partial score instead of right/wrong.
- The gold score averages two independent annotators, and related datasets report the correlation between human graders. That human-human agreement gives a natural ceiling for any automatic grader. — For Mova (answer): Report Mova's school-test scores next to inter-annotator agreement, so a gap shows how much is left to gain and whether the gold itself is noisy.
- The experiments vary one language-dependent component at a time (stemming technique, corpus domain, space dimension) to see how much each affects final grading quality. — For Mova (tokens): Run the same downstream QA with lemmas, crude stems and raw forms to measure how much the morphology crate actually contributes.
- The dataset labels each question by type (define, explain, consequences, justify, difference), so results can be broken down by type of reasoning required. — For Mova (answer): Tag Mova's test questions by type (definition, cause, contrast, consequence) and report accuracy per type, so it is clear which reasoning paths in the world model are weak.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

## Cited by

Cited by

5 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 4 not yet.

- [Automated Scoring of Arabic Text Using Large Language Models: A Literature Review](https://md.research.elamur.ai/free/en/papers/dahimi-2026-automated-scoring-arabic-text-using/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ouahrani2020ar,
  title = {{AR-ASAG An ARabic Dataset for Automatic Short Answer Grading Evaluation}},
  author = {Ouahrani, Leila and Bennouar, Djamal},
  booktitle = {Proceedings of the Twelfth Language Resources and Evaluation Conference},
  year = {2020},
  url = {https://aclanthology.org/2020.lrec-1.321/}
}
```

- [← previous Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://md.research.elamur.ai/free/en/papers/xiong-2020-approximate-nearest-neighbor-negative-contrastive/)
- [next → AraBERT: Transformer-based Model for Arabic Language Understanding](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/)

## References

30 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Al-Thubaity, A. O. (2015). A 700M+ Arabic corpus: KACST Arabic corpus design and construction. Language Resources and Evaluation, 49(3), 721– 751. https://doi.org/10.1007/s10579-014-9284-1 [↗](https://doi.org/10.1007/s10579-014-9284-1)
2. Anderson, Lorin W.; Krathwohl, David R.; Bloom, B. S. (2001). A taxonomy for learning, teaching, and assessing : a revision of Bloom’s taxonomy of educational objectives (omplete ed., Ed.). New York : Longman. [find](https://scholar.google.com/scholar?q=Anderson%2C+Lorin+W%2E%3B+Krathwohl%2C+David+R%2E%3B+Bloom%2C+B%2E+S%2E+%282001%29%2E+A+taxonomy+for+learning%2C+teaching%2C+and+assessing+%3A+a+revision+of+Bloom%E2%80%99s+taxonomy+of+educational+objectives+%28omplete+ed%2E%2C+Ed%2E%29%2E+New+York+%3A)
3. Bennouar, D. (2017). An Automatic Grading System Based on Dynamic Corpora. The International Arab Journal of Information Technology, 14(4A). [find](https://scholar.google.com/scholar?q=Bennouar%2C+D%2E+%282017%29%2E+An+Automatic+Grading+System+Based+on+Dynamic+Corpora%2E+The+International+Arab+Journal+of+Information+Technology%2C+14%284A%29%2E)
4. Brown, Sally, Ed.; Glasner, Angela, E. (1999). Assessment Matters in Higher Education: Choosing and Using Diverse Approaches., 1999. [find](https://scholar.google.com/scholar?q=Brown%2C+Sally%2C+Ed%2E%3B+Glasner%2C+Angela%2C+E%2E+%281999%29%2E+Assessment+Matters+in+Higher+Education%3A+Choosing+and+Using+Diverse+Approaches%2E%2C+1999%2E)
5. Burrows, S., Gurevych, I., & Stein, B. (2015). The eras and trends of automatic short answer grading. In International Journal of Artificial Intelligence in Education (Vol. 25). https://doi.org/10.1007/s40593-014-0026-8 [↗](https://doi.org/10.1007/s40593-014-0026-8)
6. Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., & Specia, L. (2017). SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation. In Association for Computational Linguistics (Ed.), Proceedings ofthe 11th International Workshop on Semantic Evaluations (SemEval-2017) (pp. 1–14). https://doi.org/10.18653/v1/S17-2001 [↗](https://doi.org/10.18653/v1/s17-2001)
7. El Moatez Billah Nagoudi, Jérémy Ferrero, D. S. (2017). LIM-LIG at SemEval-2017 Task1: Enhancing the Semantic Similarity for Arabic Sentences with Vectors Weighting. In Association for Computational Linguistics (Ed.), Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) (pp. 134–138). [find](https://scholar.google.com/scholar?q=El+Moatez+Billah+Nagoudi%2C+J%C3%A9r%C3%A9my+Ferrero%2C+D%2E+S%2E+%282017%29%2E+LIM%2DLIG+at+SemEval%2D2017+Task1%3A+Enhancing+the+Semantic+Similarity+for+Arabic+Sentences+with+Vectors+Weighting%2E+In+Association+for+Computational+L)
8. Gomaa, W. H., & Fahmy, A. A. (2014a). Arabic Short Answer Scoring with Effective Feedback for Students. International Journal of Computer Applications, 86(2), 35–41. https://doi.org/10.5120/14961-3177 [↗](https://doi.org/10.5120/14961-3177)
9. Gomaa, W. H., & Fahmy, A. A. (2014b). Automatic scoring for answers to Arabic test questions. Computer Speech and Language, 28(4), 833–857. https://doi.org/10.1016/j.csl.2013.10.005 [↗](https://doi.org/10.1016/j.csl.2013.10.005)
10. Gratta, R. del, Frontini, F., Fahad, K., Mariani, J., & Soria, C. (2014). The LREMap for under-resourced languages. [find](https://scholar.google.com/scholar?q=Gratta%2C+R%2E+del%2C+Frontini%2C+F%2E%2C+Fahad%2C+K%2E%2C+Mariani%2C+J%2E%2C+%26+Soria%2C+C%2E+%282014%29%2E+The+LREMap+for+under%2Dresourced+languages%2E)
11. J. B. MacQueen. (1967). Some Methods for classification and Analysis of Multivariate Observations. Proceedings of 5th Berkeley Symposium on Mathematical Statistics and Probability, 281–297. [find](https://scholar.google.com/scholar?q=J%2E+B%2E+MacQueen%2E+%281967%29%2E+Some+Methods+for+classification+and+Analysis+of+Multivariate+Observations%2E+Proceedings+of+5th+Berkeley+Symposium+on+Mathematical+Statistics+and+Probability%2C+281%E2%80%93297%2E)
12. Khoja, S., & Garside, R. (1999). Stemming arabic text. Lancaster, UK, Computing Department, Lancaster University. [find](https://scholar.google.com/scholar?q=Khoja%2C+S%2E%2C+%26+Garside%2C+R%2E+%281999%29%2E+Stemming+arabic+text%2E+Lancaster%2C+UK%2C+Computing+Department%2C+Lancaster+University%2E)
13. LEE R. DICE. (2012). Measures of the Amount of Ecologic Association Between Species Author ( s ): Lee R . Dice Reviewed work ( s ): Published by : Ecological Society of America Stable URL : [http://www.jstor.org/stable/1932409](http://www.jstor.org/stable/1932409) . 26(3), 297– 302. [↗](http://www.jstor.org/stable/1932409)
14. Liu, O. L., Brew, C., Blackmore, J., Gerard, L., Madhok, J., & Linn, M. C. (2014). Automated scoring of constructed-response science items: Prospects and obstacles. Educational Measurement: Issues and Practice. https://doi.org/10.1111/emip.12028 [↗](https://doi.org/10.1111/emip.12028)
15. Lovins, J. (1963). Development of a stemming algorithm. Cambridge: M.I.T. Information Processing Group Electronic Systems Laboratory. [find](https://scholar.google.com/scholar?q=Lovins%2C+J%2E+%281963%29%2E+Development+of+a+stemming+algorithm%2E+Cambridge%3A+M%2EI%2ET%2E+Information+Processing+Group+Electronic+Systems+Laboratory%2E)
16. Lund, K., & Burgess, C. (1996). Producing high-dimensional semantic spaces from lexical co-occurrence. Behavior Research Methods, Instruments, & Computers, 28(2), 203–208. https://doi.org/10.3758/BF03204766 [↗](https://doi.org/10.3758/bf03204766)
17. Mag`ooda, A., Zahran, M. A., Rashwan, M., Raafat, H., & Fayek, M. B. (2016). Vector Based Techniques for Short Answer Grading. FLAIRS Conference, 238– 243. [find](https://scholar.google.com/scholar?q=Mag%60ooda%2C+A%2E%2C+Zahran%2C+M%2E+A%2E%2C+Rashwan%2C+M%2E%2C+Raafat%2C+H%2E%2C+%26+Fayek%2C+M%2E+B%2E+%282016%29%2E+Vector+Based+Techniques+for+Short+Answer+Grading%2E+FLAIRS+Conference%2C+238%E2%80%93+243%2E)
18. Mahmoud El-Haj, Udo Kruschwitz, Colchester, & Fox, C. (2015). Creating Language Resources for Under-resourced Languages : Methodologies , and experiments with Arabic. Language Resources and Evaluation, 49(3), 549–580. https://doi.org/10.1007/s10579-014-9274-3 [↗](https://doi.org/10.1007/s10579-014-9274-3)
19. Mihalcea, R., Corley, C., & Strapparava, C. (2006). Corpus-based and knowledge-based measures of text semantic similarity. Proceedings of the 21st National Conference on Artificial Intelligence, 1, 775–780. https://doi.org/10.1.1.65.3690 [↗](https://doi.org/10.1.1.65.3690)
20. Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. Retrieved from http://arxiv.org/abs/1301.3781 [↗](https://arxiv.org/abs/1301.3781)
21. Mohler, M., Bunescu, R., & Mihalcea, R. (2011). Learning to Grade Short Answer Questions using Semantic Similarity Measures and Dependency Graph Alignments. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, 752–762. [find](https://scholar.google.com/scholar?q=Mohler%2C+M%2E%2C+Bunescu%2C+R%2E%2C+%26+Mihalcea%2C+R%2E+%282011%29%2E+Learning+to+Grade+Short+Answer+Questions+using+Semantic+Similarity+Measures+and+Dependency+Graph+Alignments%2E+Proceedings+of+the+49th+Annual+Meeting+of+t)
22. Mohler, M., & Mihalcea, R. (2009). Text-to-text semantic similarity for automatic short answer grading. Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics on - EACL ’09, 567–575. https://doi.org/10.3115/1609067.1609130 [↗](https://doi.org/10.3115/1609067.1609130)
23. Rohde, D. L. T., Gonnerman, L. M., & Plaut, D. C. (2004). An Improved Method for Deriving Word Meaning from Lexical. Cognitive Psychology, 7, 573–605. [find](https://scholar.google.com/scholar?q=Rohde%2C+D%2E+L%2E+T%2E%2C+Gonnerman%2C+L%2E+M%2E%2C+%26+Plaut%2C+D%2E+C%2E+%282004%29%2E+An+Improved+Method+for+Deriving+Word+Meaning+from+Lexical%2E+Cognitive+Psychology%2C+7%2C+573%E2%80%93605%2E)
24. Sahlgren, M., (2005). An introduction to random indexing. In Methods and Applications of Semantic Indexing Worshop at the 7th International Conference on Terminology and Knowldge Engineering, TKE 2005. [find](https://scholar.google.com/scholar?q=Sahlgren%2C+M%2E%2C+%282005%29%2E+An+introduction+to+random+indexing%2E+In+Methods+and+Applications+of+Semantic+Indexing+Worshop+at+the+7th+International+Conference+on+Terminology+and+Knowldge+Engineering%2C+TKE+2005)
25. Salton, G., & Buckley, C. (1988). Term-weighting approaches in automatic text retrieval. Information Processing & Management, 24(5), 513–523. https://doi.org/10.1016/0306-4573(88)90021-0 [↗](https://doi.org/10.1016/0306-4573(88)90021-0)
26. Shourya Roy, Y. Narahari, and O. D. D. (2015). A Perspective on Computer Assisted Assessment Techniques for Short Free-text Answers. Communications in Computer and Information Science, 571(June 2015). https://doi.org/10.1007/978-3-319-27704-2 [↗](https://doi.org/10.1007/978-3-319-27704-2)
27. Wu, H., Huang, H., Jian, P., Guo, Y., & Su, C. (2017). BIT at SemEval-2017 Task 1: Using Semantic Information Space to Evaluate Semantic Textual Similarity. In Association for Computational Linguistics (Ed.), Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017): Vol. L (pp. 77–84). [find](https://scholar.google.com/scholar?q=Wu%2C+H%2E%2C+Huang%2C+H%2E%2C+Jian%2C+P%2E%2C+Guo%2C+Y%2E%2C+%26+Su%2C+C%2E+%282017%29%2E+BIT+at+SemEval%2D2017+Task+1%3A+Using+Semantic+Information+Space+to+Evaluate+Semantic+Textual+Similarity%2E+In+Association+for+Computational+Linguistic)
28. Zahran, M. A., Magooda, A., Mahgoub, A. Y., Raafat, H., Rashwan, M., & Atyia, A. (2015). Word Representations in Vector Space and their Applications for Arabic. In A. Gelbukh (Ed.) (Ed.), 16th international conference, CICLing 2015 Cairo, Egypt, april 14 (Vol. 9041, pp. 430–443). https://doi.org/10.1007/978-3-319-18111-0_32 9. [↗](https://doi.org/10.1007/978-3-319-18111-0_32)
29. Language Resource References (AR-ASAG Dataset, 2020). The Arabic Dataset for Automatic Short Answer Grading Evaluation, V. 1.0, ISLRN 529-005-230-448-6. [find](https://scholar.google.com/scholar?q=Language+Resource+References+%28AR%2DASAG+Dataset%2C+2020%29%2E+The+Arabic+Dataset+for+Automatic+Short+Answer+Grading+Evaluation%2C+V%2E+1%2E0%2C+ISLRN+529%2D005%2D230%2D448%2D6%2E)
30. (Arabic Cyber Text Corpus, 2020). The Arabic In-Domain Cyber Text Corpus, V. 1.0, ISLRN 798-080-268-332-8. [find](https://scholar.google.com/scholar?q=%28Arabic+Cyber+Text+Corpus%2C+2020%29%2E+The+Arabic+In%2DDomain+Cyber+Text+Corpus%2C+V%2E+1%2E0%2C+ISLRN+798%2D080%2D268%2D332%2D8%2E)