---
title: "Large Scale Arabic Error Annotation — Zaghouani 2014"
description: "Arabic error correction has lagged behind other languages, partly because no large, varied, manually corrected corpus existed."
authors: ["Wajdi Zaghouani", "Behrang Mohit", "Nizar Habash", "Ossama Obeid", "Nadi Tomeh", "Alla Rozovskaya", "Noura Farra", "Sarah Alkuhlani", "Kemal Oflazer"]
year: 2014
venue: "Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)"
source: "https://aclanthology.org/L14-1721/"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/zaghouani-2014-large-scale-arabic-error-annotation/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/zaghouani-2014-large-scale-arabic-error-annotation/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2014](https://md.research.elamur.ai/free/papers/2014/) / Large Scale Arabic Error Annotation: Guidelines and Framework

# Large Scale Arabic Error Annotation: Guidelines and Framework

**Authors:** [Wajdi Zaghouani](https://md.research.elamur.ai/free/authors/wajdi-zaghouani/), [Behrang Mohit](https://md.research.elamur.ai/free/authors/behrang-mohit/), [Nizar Habash](https://md.research.elamur.ai/free/authors/nizar-habash/), [Ossama Obeid](https://md.research.elamur.ai/free/authors/ossama-obeid/), Nadi Tomeh, [Alla Rozovskaya](https://md.research.elamur.ai/free/authors/alla-rozovskaya/), [Noura Farra](https://md.research.elamur.ai/free/authors/noura-farra/), [Sarah Alkuhlani](https://md.research.elamur.ai/free/authors/sarah-alkuhlani/), [Kemal Oflazer](https://md.research.elamur.ai/free/authors/kemal-oflazer/) · **Year:** 2014 · **Venue:** Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/L14-1721/) · [PDF](https://aclanthology.org/L14-1721.pdf) · [OpenAlex](https://openalex.org/W2251944986)

**Citations:** 87 citations (openalex-snapshot · 2026-10-14) · [73](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Arabic error correction has lagged behind other languages, partly because no large, varied, manually corrected corpus existed. Earlier Arabic error corpora were small (9K–65K words) and mostly designed for learner studies rather than for training correction systems. This paper describes the Qatar Arabic Language Bank (QALB) project, which targets 2 million words of corrected Arabic. It draws on genres with many errors: Al Jazeera reader comments, native and non-native student essays, and machine translation output. The authors present several components:
- annotation guidelines, refined over several trial rounds;
- a web-based tool built for large, distributed annotation teams;
- a pipeline in which MADA automatically fixes simple spelling errors before humans edit the text.

When the paper was written, 1.2M words had been annotated. On a test sample, the MADA step cut word error rate substantially, and annotators agreed closely with each other. The corpus is meant both as training data and as a gold standard, and much of it was planned for release in the EMNLP 2014 Arabic error correction shared task.

## Key points
- **Annotated so far:** 1.2M of a planned 2M words. This includes about 1,170,000 words of Al Jazeera comments and 31,600 words of native student essays. Non-native essays (Arabic Learners Written Corpus) and Google Translate output of Wikipedia text were planned next.
- **Team and categories:** 10 junior annotators and a lead annotator fixed errors falling into seven groups: misspellings, faulty punctuation, poorly chosen words, morphological mistakes, syntactic mistakes, errors in proper names, and dialectal words. They did not have to label error types.
- **Effect of MADA pre-correction:** input WER dropped from 31.8% to 23.0% (about 27% relative). With punctuation ignored, it dropped from 23.3% to 13.4% (about 42% relative).
- **Inter-annotator agreement:** measured on 200 files (10,288 words), each corrected by at least three annotators. AWER was 11.3% on all words and 3.8% without punctuation. APP was 93.0% and 98.1%, respectively.
- **Annotation history:** the full sequence of correction actions is stored in XML exports, together with token alignments.

## Interesting ideas
- **Choose high-error source texts.** Picking genres where errors are dense, such as user comments and learner essays, makes each annotation hour more productive. This helps any error-correction or normalization dataset where errors are otherwise rare.
- **Pre-correct automatically before humans edit.** Running a tool that fixes routine mistakes leaves annotators to handle the harder cases. Measuring the tool's WER against the gold corrections also shows how much labor it actually saved.
- **Ask for corrections only, not error labels.** Skipping error-type tagging and telling annotators to leave style alone reduces over-correction and inconsistency. Error categories could be derived automatically later.
- **Measure agreement as per-word edit labels.** Each annotator's correction is aligned to the source text by minimum edit distance, and each word receives an action label: keep, delete, or replace-with-w. Comparing these labels across annotators gives an agreement score that suits free-form correction tasks, where kappa on fixed categories does not apply.
- **Report metrics with and without punctuation.** Punctuation norms in Arabic are loose and inconsistent, so separating them shows where disagreement and remaining errors really come from. This is worth copying for any language with unstable punctuation conventions.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

## Cited by

Cited by

73 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 64 not yet.

- [DAICT: A Dialectal Arabic Irony Corpus Extracted from Twitter](https://md.research.elamur.ai/free/papers/abbes-2020-daict-dialectal-arabic-irony-corpus/)
- [Advancements in Arabic Grammatical Error Detection and Correction: An Empirical Investigation](https://md.research.elamur.ai/free/papers/alhafni-2023-advancements-arabic-grammatical-error-detection/)
- [A Bilingual Bimodal Benchmark for Arabic-English NLP across Grammatical Correction, Essay Scoring, Morphological Tagging, and Speech Recognition](https://md.research.elamur.ai/free/papers/alhafni-2026-bilingual-bimodal-benchmark-arabic-english/)
- [Classifying Syntactic Errors in Learner Language](https://md.research.elamur.ai/free/papers/choshen-2020-classifying-syntactic-errors-learner-language/)
- [Unified Guidelines and Resources for Arabic Dialect Orthography](https://md.research.elamur.ai/free/papers/habash-2018-unified-guidelines-resources-arabic-dialect/)
- [Arabic Sentence Segmentation Across Genres and Punctuation Conditions](https://md.research.elamur.ai/free/papers/langstream-2026-elkholy-arabic-sentence-segmentation-across-genres/)
- [Camel Treebank: An Open Multi-genre Arabic Dependency Treebank](https://md.research.elamur.ai/free/papers/lingbulk-2022-habash-camel-treebank-open-multi-genre/)
- [Morphotactic Modeling in an Open-source Multi-dialectal Arabic Morphological Analyzer and Generator](https://md.research.elamur.ai/free/papers/lingbulk-2022-habash-morphotactic-modeling-open-source-multi/)
- [CAMeL Tools: An Open Source Python Toolkit for Arabic Natural Language Processing](https://md.research.elamur.ai/free/papers/obeid-2020-camel-tools-open-source-python/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zaghouani2014large,
  title = {{Large Scale Arabic Error Annotation: Guidelines and Framework}},
  author = {Zaghouani, Wajdi and Mohit, Behrang and Habash, Nizar and Obeid, Ossama and Tomeh, Nadi and Rozovskaya, Alla and Farra, Noura and Alkuhlani, Sarah and Oflazer, Kemal},
  booktitle = {Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)},
  year = {2014},
  doi = {10.63317/5mvsos89i9i9},
  url = {https://doi.org/10.63317/5mvsos89i9i9}
}
```

- [← previous LAF-Fabric: a data analysis tool for Linguistic Annotation Framework with an application to the Hebrew Bible](https://md.research.elamur.ai/free/papers/hbo-2014-roorda-laf-fabric-data-analysis-tool/)
- [next → Latent Trees for Coreference Resolution](https://md.research.elamur.ai/free/papers/fernandes-2014-latent-trees-coreference-resolution/)
- [↑ back to papers of 2014](https://md.research.elamur.ai/free/papers/2014/)

## References

26 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abuhakema, Ghazi, Faraj, Reem, Feldman, Anna, and Fitzpatrick, Eileen. (2008). Annotating an Arabic learner corpus for error. In Proceedings of LREC. [find](https://scholar.google.com/scholar?q=Abuhakema%2C+Ghazi%2C+Faraj%2C+Reem%2C+Feldman%2C+Anna%2C+and+Fitzpatrick%2C+Eileen%2E+%282008%29%2E+Annotating+an+Arabic+learner+corpus+for+error%2E+In+Proceedings+of+LREC%2E)
2. Alfaifi, Abdullah and Atwell, Eric. (2012). Arabic Learner Corpora (ALC): A taxonomy of coding errors. In The 8th International Computing Conference in Arabic. [↗](https://doi.org/10.13140/2.1.3126.8809)
3. Alkanhal, Mohamed I, Al-Badrashiny, Mohamed A, Alghamdi, Mansour M, and Al-Qabbany, Abdulaziz O. (2012). Automatic Stochastic Arabic Spelling Correction With Emphasis on Space Insertions and Deletions. IEEE Transactions on Audio, Speech, and Language Processing, 20(7):2111–2122. [↗](https://doi.org/10.1109/tasl.2012.2197612)
4. Awad, Dana. (2013). La ponctuation arabe : histoire et règles. [find](https://scholar.google.com/scholar?q=Awad%2C+Dana%2E+%282013%29%2E+La+ponctuation+arabe+%3A+histoire+et+r%C3%A8gles%2E)
5. Bušta, Jan, Hlavácková, Dana, Jakubícek, Miloš, and Pala, Karel. (2009). Classification of Errors in Text. RASLAN 2009, page 109. [find](https://scholar.google.com/scholar?q=Bu%C5%A1ta%2C+Jan%2C+Hlav%C3%A1ckov%C3%A1%2C+Dana%2C+Jakub%C3%ADcek%2C+Milo%C5%A1%2C+and+Pala%2C+Karel%2E+%282009%29%2E+Classification+of+Errors+in+Text%2E+RASLAN+2009%2C+page+109%2E)
6. Dahlmeier, D., H.T. Ng, Hwee, and Wu, S.M. (2013). Building a large annotated corpus of learner english: The NUS corpus of learner english. In Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications, pages 22–31, Atlanta, Georgia, June. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Dahlmeier%2C+D%2E%2C+H%2ET%2E+Ng%2C+Hwee%2C+and+Wu%2C+S%2EM%2E+%282013%29%2E+Building+a+large+annotated+corpus+of+learner+english%3A+The+NUS+corpus+of+learner+english%2E+In+Proceedings+of+the+Eighth+Workshop+on+Innovative+Use+of+N)
7. Dickinson, Markus and Ledbetter, Scott. (2012). Annotating errors in a Hungarian learner corpus. In Proceedings of LREC. [find](https://scholar.google.com/scholar?q=Dickinson%2C+Markus+and+Ledbetter%2C+Scott%2E+%282012%29%2E+Annotating+errors+in+a+Hungarian+learner+corpus%2E+In+Proceedings+of+LREC%2E)
8. Eskander, Ramy, Habash, Nizar, Rambow, Owen, and Tomeh, Nadi. (2013). Processing Spontaneous Orthography. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), Atlanta, GA. [find](https://scholar.google.com/scholar?q=Eskander%2C+Ramy%2C+Habash%2C+Nizar%2C+Rambow%2C+Owen%2C+and+Tomeh%2C+Nadi%2E+%282013%29%2E+Processing+Spontaneous+Orthography%2E+In+Proceedings+of+the+2013+Conference+of+the+North+American+Chapter+of+the+Association+for+Com)
9. Farwaneh, Samira and Tamimi, Mohammed. (2012). Arabic Learners Written Corpus: A Resource for Research and Learning. The Center for Educational Resources in Culture, Language and Literacy. [find](https://scholar.google.com/scholar?q=Farwaneh%2C+Samira+and+Tamimi%2C+Mohammed%2E+%282012%29%2E+Arabic+Learners+Written+Corpus%3A+A+Resource+for+Research+and+Learning%2E+The+Center+for+Educational+Resources+in+Culture%2C+Language+and+Literacy%2E)
10. Habash, Nizar and Rambow, Owen. (2005). Arabic Tokenization, Part-of-Speech Tagging and Morphological Disambiguation in One Fell Swoop. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 573–580, Ann Arbor, Michigan. [find](https://scholar.google.com/scholar?q=Habash%2C+Nizar+and+Rambow%2C+Owen%2E+%282005%29%2E+Arabic+Tokenization%2C+Part%2Dof%2DSpeech+Tagging+and+Morphological+Disambiguation+in+One+Fell+Swoop%2E+In+Proceedings+of+the+43rd+Annual+Meeting+of+the+Association+for) [our card](https://md.research.elamur.ai/free/papers/aclbulk-2005-habash-arabic-tokenization-part-speech-tagging/)
11. Habash, Nizar, Soudi, Abdelhadi, and Buckwalter, Tim. (2007). On Arabic Transliteration. In van den Bosch, A. and Soudi, A., editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods. Springer. [↗](https://doi.org/10.1007/978-1-4020-6046-5_2)
12. Habash, N., Rambow, O., Diab, M., and Kanjawi-Faraj, R. (2008). Guidelines for Annotation of Arabic Dialectness. In Proceedings of the LREC Workshop on HLT & NLP within the Arabic world. [find](https://scholar.google.com/scholar?q=Habash%2C+N%2E%2C+Rambow%2C+O%2E%2C+Diab%2C+M%2E%2C+and+Kanjawi%2DFaraj%2C+R%2E+%282008%29%2E+Guidelines+for+Annotation+of+Arabic+Dialectness%2E+In+Proceedings+of+the+LREC+Workshop+on+HLT+%26+NLP+within+the+Arabic+world%2E)
13. Habash, Nizar, Rambow, Owen, and Roth, Ryan. (2009). Mada+tokan: A toolkit for Arabic tokenization, diacritization, morphological disambiguation, pos tagging, stemming and lemmatization. In Proceedings of the 2nd International Conference on Arabic Language Resources and Tools (MEDAR), Cairo, Egypt, pages 102–109. [find](https://scholar.google.com/scholar?q=Habash%2C+Nizar%2C+Rambow%2C+Owen%2C+and+Roth%2C+Ryan%2E+%282009%29%2E+Mada%2Btokan%3A+A+toolkit+for+Arabic+tokenization%2C+diacritization%2C+morphological+disambiguation%2C+pos+tagging%2C+stemming+and+lemmatization%2E+In+Proceeding)
14. Habash, Nizar. (2008). Four Techniques for Online Handling of Out-of-Vocabulary Words in Arabic-English Statistical Machine Translation. In Proceedings of ACL-08: HLT, Short Papers, pages 57–60, Columbus, Ohio. [find](https://scholar.google.com/scholar?q=Habash%2C+Nizar%2E+%282008%29%2E+Four+Techniques+for+Online+Handling+of+Out%2Dof%2DVocabulary+Words+in+Arabic%2DEnglish+Statistical+Machine+Translation%2E+In+Proceedings+of+ACL%2D08%3A+HLT%2C+Short+Papers%2C+pages+57%E2%80%9360%2C+Colum)
15. Habash, Nizar. (2010). Introduction to Arabic Natural Language Processing. Morgan & Claypool Publishers. [↗](https://doi.org/10.1007/978-3-031-02139-8)
16. Obeid, Ossama, Zaghouani, Wajdi, Mohit, Behrang, Habash, Nizar, Oflazer, Kemal, and Tomeh, Nadi. (2013). A web-based annotation framework for large-scale text correction. In The Companion Volume of the Proceedings of IJCNLP 2013: System Demonstrations, Nagoya, Japan, October. [find](https://scholar.google.com/scholar?q=Obeid%2C+Ossama%2C+Zaghouani%2C+Wajdi%2C+Mohit%2C+Behrang%2C+Habash%2C+Nizar%2C+Oflazer%2C+Kemal%2C+and+Tomeh%2C+Nadi%2E+%282013%29%2E+A+web%2Dbased+annotation+framework+for+large%2Dscale+text+correction%2E+In+The+Companion+Volume+of+th)
17. Pala, Karel, Rychlỳ, Pavel, and Smrž, Pavel. (2003). Text corpus with errors. In Text, Speech and Dialogue, pages 90–97. Springer. [find](https://scholar.google.com/scholar?q=Pala%2C+Karel%2C+Rychly%CC%80%2C+Pavel%2C+and+Smr%C5%BE%2C+Pavel%2E+%282003%29%2E+Text+corpus+with+errors%2E+In+Text%2C+Speech+and+Dialogue%2C+pages+90%E2%80%9397%2E+Springer%2E)
18. Pasha, Arfath, Al-Badrashiny, Mohamed, Kholy, Ahmed El, Eskander, Ramy, Diab, Mona, Habash, Nizar, Pooleery, Manoj, Rambow, Owen, and Roth, Ryan. (2014). Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of arabic. In In Proceedings of the 9th International Conference on Language Resources and Evaluation, Reykjavik, Iceland. [find](https://scholar.google.com/scholar?q=Pasha%2C+Arfath%2C+Al%2DBadrashiny%2C+Mohamed%2C+Kholy%2C+Ahmed+El%2C+Eskander%2C+Ramy%2C+Diab%2C+Mona%2C+Habash%2C+Nizar%2C+Pooleery%2C+Manoj%2C+Rambow%2C+Owen%2C+and+Roth%2C+Ryan%2E+%282014%29%2E+Madamira%3A+A+fast%2C+comprehensive+tool+for+morph) [our card](https://md.research.elamur.ai/free/papers/lingbulk-2014-pasha-madamira-fast-comprehensive-tool-morphological/)
19. Rozovskaya, A. and Roth, D. (2010). Annotating esl errors: Challenges and rewards. In Proceedings of the NAACL Workshop on Innovative Use of NLP for Building Educational Applications, 6. [find](https://scholar.google.com/scholar?q=Rozovskaya%2C+A%2E+and+Roth%2C+D%2E+%282010%29%2E+Annotating+esl+errors%3A+Challenges+and+rewards%2E+In+Proceedings+of+the+NAACL+Workshop+on+Innovative+Use+of+NLP+for+Building+Educational+Applications%2C+6%2E)
20. Rytting, C., Buckwalter, T., Wayland, S., Hettick, C., Rodrigues, P., and Zajic, D. (2011). Spelling correction for dialectal Arabic dictionary lookup. ACM Transactions of Asian Language Information Processing, 10(1):1–15. [↗](https://doi.org/10.1145/1929908.1929911)
21. Salloum, Wael and Habash, Nizar. (2011). Dialectal to Standard Arabic Paraphrasing to Improve Arabic-English Statistical Machine Translation. In Proceedings of the First Workshop on Algorithms and Resources for Modelling of Dialects and Language Varieties, pages 10– 21, Edinburgh, Scotland. [↗](https://doi.org/10.7916/d8697bg2)
22. Salloum, Wael and Habash, Nizar. (2013). Dialectal Arabic to English Machine Translation: Pivoting through Modern Standard Arabic. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), Atlanta, GA. [find](https://scholar.google.com/scholar?q=Salloum%2C+Wael+and+Habash%2C+Nizar%2E+%282013%29%2E+Dialectal+Arabic+to+English+Machine+Translation%3A+Pivoting+through+Modern+Standard+Arabic%2E+In+Proceedings+of+the+2013+Conference+of+the+North+American+Chapter+o)
23. Schmidt, Thomas. (2010). Linguistic tool development between community practices and technology standards. In Proceedings of the LREC Workshop Language Resource and Language Technology Standards state of the art, emerging needs, and future developments. [find](https://scholar.google.com/scholar?q=Schmidt%2C+Thomas%2E+%282010%29%2E+Linguistic+tool+development+between+community+practices+and+technology+standards%2E+In+Proceedings+of+the+LREC+Workshop+Language+Resource+and+Language+Technology+Standards+state)
24. Shaalan, Khaled F., Attia, Mohammed, Pecina, Pavel, Samih, Younes, and van Genabith, Josef. (2012). Arabic word generation and modelling for spell checking. In LREC, pages 719–725. [find](https://scholar.google.com/scholar?q=Shaalan%2C+Khaled+F%2E%2C+Attia%2C+Mohammed%2C+Pecina%2C+Pavel%2C+Samih%2C+Younes%2C+and+van+Genabith%2C+Josef%2E+%282012%29%2E+Arabic+word+generation+and+modelling+for+spell+checking%2E+In+LREC%2C+pages+719%E2%80%93725%2E)
25. Yannakoudakis, H., Briscoe, T., and Medlock, B. (2011). A new dataset and method for automatically grading esol texts. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 180– 189, Portland, Oregon, USA, June. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yannakoudakis%2C+H%2E%2C+Briscoe%2C+T%2E%2C+and+Medlock%2C+B%2E+%282011%29%2E+A+new+dataset+and+method+for+automatically+grading+esol+texts%2E+In+Proceedings+of+the+49th+Annual+Meeting+of+the+Association+for+Computational+Li)
26. Zaki, Ahmad. (1995). at-tarqim wa alamatu-hu fi al-lugati al-arabiyya, maktabat al-matbuat al islamiyya bi halab. [find](https://scholar.google.com/scholar?q=Zaki%2C+Ahmad%2E+%281995%29%2E+at%2Dtarqim+wa+alamatu%2Dhu+fi+al%2Dlugati+al%2Darabiyya%2C+maktabat+al%2Dmatbuat+al+islamiyya+bi+halab%2E)