---
title: "A Hybrid Approach for Building Arabic Diacritizer — Shaalan 2009"
description: "The authors build a diacritizer that splits the task in two. 11.795% WER. 19 references, cited by 6 papers here."
authors: ["Khaled Shaalan", "Hitham M. Abo Bakr", "Ibrahim Ziedan"]
year: 2009
venue: "Proceedings of the EACL 2009 Workshop on Computational Approaches to Semitic Languages"
source: "https://aclanthology.org/W09-0804/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shaalan-2009-hybrid-approach-building-arabic-diacritizer/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shaalan-2009-hybrid-approach-building-arabic-diacritizer/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2009](https://md.research.elamur.ai/free/en/papers/2009/) / A Hybrid Approach for Building Arabic Diacritizer

# A Hybrid Approach for Building Arabic Diacritizer

**Authors:** [Khaled Shaalan](https://md.research.elamur.ai/free/en/authors/khaled-shaalan/), Hitham M. Abo Bakr and Ibrahim Ziedan · **Year:** 2009 · **Venue:** [Proceedings of the EACL 2009 Workshop on Computational Approaches to Semitic Languages](https://md.research.elamur.ai/free/en/venues/eacl-workshop-on-computational-approaches-to-semitic-languages/) ([EACL Workshop on Computational Approaches to Semitic Languages 2009](https://md.research.elamur.ai/free/en/venues/eacl-workshop-on-computational-approaches-to-semitic-languages/2009/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W09-0804/) · [PDF](https://aclanthology.org/W09-0804.pdf) · [OpenAlex](https://openalex.org/W2065400286)

**Citations:** 44 citations (OpenAlex, 2026-10-14) · [6](https://md.research.elamur.ai/free/en/papers/shaalan-2009-hybrid-approach-building-arabic-diacritizer/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Arabic is normally written without short vowels and other diacritic marks, so many written forms map to several readings. This complicates speech synthesis, recognition and text analysis. The authors build a diacritizer that splits the task in two. Word-internal marks are treated as a morphological problem. The final-letter case ending is treated as a syntactic one.

For internal marks, three components vote:
- an unambiguous-entry lookup in an Arabic lexicon;
- a database of frequent diacritized two-word expressions;
- an SVM tokenizer/POS tagger whose tags pick among candidates from a modified Buckwalter analyzer.

Case endings are then predicted by a separate SVM. It uses POS, base-phrase chunk and positional features, trained on tags extracted from the Penn Arabic Treebank (ATB). After feature and rule refinements, the full system reaches 11.795% WER and about 3.245% DER. That compares favourably with the figures reported for MADA-D. The result suggests that a fairly simple hybrid design can rival heavier statistical systems.

## Key points
- The final pipeline LR+DB+SVM+CE+SHR scores 11.795% WER and roughly 3.245% DER. The previously reported MADA-D numbers are 14.9% WER and 4.8% DER.
- Case-ending accuracy rose from 95.35% (CE1, window −3/+3) to 96.57% (CE2). The gain came from adding last-character and last-two-character features.
- In ablations (Table 9), WER falls from 35.81% for LR+DB to 17.31% once the CE model is added. With the "accepts any" (AA) fallback it reaches 16.66%.
- ArabicSVMTools gave a better internal-only result (12.79% WER / 9.94% DER) than the authors' own tagger (16.28% / 11.36%). However, their tagger runs about 4× faster because it uses fewer features.
- Buckwalter's analyzer was modified by deleting case-ending diacritics from its suffix table. This shrinks overgeneration: one word that had 7 analyses differing only by case ending now has a single internal candidate.

## Interesting ideas
- Separate the ambiguity that belongs to the word from the ambiguity that belongs to its sentence role, and strip the latter out of the morphological analyzer so it stops multiplying candidates.
- Combine several solvers by agreement first, and fall back to a fixed trust order (exact lookup > collocation memory > statistical model) only when they disagree.
- Treat a lexicon hit with exactly one candidate as final and skip the other components entirely.
- Cheap orthographic cues such as the last one or two characters noticeably improved both tagging and case prediction. They also removed grammatically impossible outputs.
- Report error rates with and without the hardest sub-problem (case endings) to show where the errors actually come from.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tokens): the en morphology could withhold context-dependent readings (e.g. syntactic function of an -ing form) and leave them to the parser, keeping the lemma/feature lattice small and its choices easier to explain.
- **I2** (learn): witness parsers producing silver data could use the same agree-then-rank rule, with a recorded reason for every chosen label.
- **I3** (tokens): unambiguous lexicon entries could short-circuit the tagger, saving compute and giving a trivially explainable decision path.
- **I4** (tokens): suffix features in the en tagger, and a check rejecting agreement-impossible combinations, could cut absurd morphological tags at little cost.
- **I5** (learn): evaluations could be split by component, e.g. scoring with and without attachment-heavy relations, so regressions are traced to the stage that caused them.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [British University in Dubai](https://md.research.elamur.ai/free/en/universities/british-university-in-dubai/) — University, Dubai, United Arab Emirates · 11 works · 589 citations
- [Zagazig University](https://md.research.elamur.ai/free/en/universities/zagazig-university/) — University, Zagazig, Egypt · 4 works · 51 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 3,066 works

## Cite

Cite as: Khaled Shaalan, Hitham M. Abo Bakr and Ibrahim Ziedan. 2009. A Hybrid Approach for Building Arabic Diacritizer. Proceedings of the EACL 2009 Workshop on Computational Approaches to Semitic Languages.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{shaalan2009hybrid,
  title = {{A Hybrid Approach for Building Arabic Diacritizer}},
  author = {Khaled Shaalan, Hitham M. Abo Bakr and Ibrahim Ziedan},
  booktitle = {Proceedings of the EACL 2009 Workshop on Computational Approaches to Semitic Languages},
  year = {2009},
  doi = {10.3115/1621774.1621780},
  url = {https://doi.org/10.3115/1621774.1621780}
}
```

- [← previous Unsupervised Classification of Dialogue Acts using a Dirichlet Process Mixture Model](https://md.research.elamur.ai/free/en/papers/crook-2009-unsupervised-classification-dialogue-acts-dirichlet/)
- [next → Automatic Treebank-Based Acquisition of Arabic LFG Dependency Structures](https://md.research.elamur.ai/free/en/papers/tounsi-2009-automatic-treebank-based-acquisition-arabic/)
- [↑ back to papers of 2009](https://md.research.elamur.ai/free/en/papers/2009/)

## References

19 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abo Bakr H. M. , Shaalan K., Ziedan I., 2008, "A Statistical Method for Adding Case Ending Diacritics for Arabic Text", The Eighth Conference on Language Engineering, ESOLEC’2008, Page 225-234, Cairo, Egypt,Deceber 17-18 2008. [find](https://scholar.google.com/scholar?q=Abo+Bakr+H%2E+M%2E+%2C+Shaalan+K%2E%2C+Ziedan+I%2E%2C+2008%2C+%22A+Statistical+Method+for+Adding+Case+Ending+Diacritics+for+Arabic+Text%22%2C+The+Eighth+Conference+on+Language+Engineering%2C+ESOLEC%E2%80%992008%2C+Page+225%2D234%2C+Cairo%2C)
2. Ananthakrishnan, Narayanan S., and Bangalore S., (2005), “Automatic diacritization of arabic transcripts for asr”. In Proceedings of ICON-05, Kanpur, India. [find](https://scholar.google.com/scholar?q=Ananthakrishnan%2C+Narayanan+S%2E%2C+and+Bangalore+S%2E%2C+%282005%29%2C+%E2%80%9CAutomatic+diacritization+of+arabic+transcripts+for+asr%E2%80%9D%2E+In+Proceedings+of+ICON%2D05%2C+Kanpur%2C+India%2E)
3. Buckwalter T., (2002). Buckwalter Arabic morphological analyzer version 1.0. Technical report, Linguistic Data Consortium, LDC2002L49 and ISBN 1-58563- 257-0. [find](https://scholar.google.com/scholar?q=Buckwalter+T%2E%2C+%282002%29%2E+Buckwalter+Arabic+morphological+analyzer+version+1%2E0%2E+Technical+report%2C+Linguistic+Data+Consortium%2C+LDC2002L49+and+ISBN+1%2D58563%2D+257%2D0%2E)
4. Cristianini N. and Taylor J.S., (2000), “An Introduction to Support Vector Machines and Other Kernel-based Learning Methods”, The Press Syndicate of the University of Cambridge, Cambridge, United Kingdom. [find](https://scholar.google.com/scholar?q=Cristianini+N%2E+and+Taylor+J%2ES%2E%2C+%282000%29%2C+%E2%80%9CAn+Introduction+to+Support+Vector+Machines+and+Other+Kernel%2Dbased+Learning+Methods%E2%80%9D%2C+The+Press+Syndicate+of+the+University+of+Cambridge%2C+Cambridge%2C+United+King)
5. Diab M., Hacioglu K., and Jurafsky D., (2004), "[Automatic Tagging of Arabic Text: From Raw Text to Base Phrase Chunks](https://md.research.elamur.ai/free/en/papers/diab-2004-automatic-tagging-arabic-text-raw/)," In Proc. of HLT/NAACL 2004, Boston.
6. Diab M., Hacioglu K., and Jurafsky D.,(2007), “Arabic Computational Morphology Knowledge-based and Empirical Methods” - Chapter 7 “Automatic Processing of Modern Standard Arabic Text”,ISBN: 978-1-4020-6046-5, SpringerLink. [find](https://scholar.google.com/scholar?q=Diab+M%2E%2C+Hacioglu+K%2E%2C+and+Jurafsky+D%2E%2C%282007%29%2C+%E2%80%9CArabic+Computational+Morphology+Knowledge%2Dbased+and+Empirical+Methods%E2%80%9D+%2D+Chapter+7+%E2%80%9CAutomatic+Processing+of+Modern+Standard+Arabic+Text%E2%80%9D%2CISBN%3A+978%2D1%2D4020)
7. El-Imam Y., (2003). Phonetization of Arabic: rules and algorithms. Computer Speech and Language, 18:339– 373. [find](https://scholar.google.com/scholar?q=El%2DImam+Y%2E%2C+%282003%29%2E+Phonetization+of+Arabic%3A+rules+and+algorithms%2E+Computer+Speech+and+Language%2C+18%3A339%E2%80%93+373%2E)
8. El-Sadany T. and Hashish M., (1988). Semi-automatic vowelization of Arabic verbs. In 10th NC Conference, Jeddah, Saudi Arabia. [find](https://scholar.google.com/scholar?q=El%2DSadany+T%2E+and+Hashish+M%2E%2C+%281988%29%2E+Semi%2Dautomatic+vowelization+of+Arabic+verbs%2E+In+10th+NC+Conference%2C+Jeddah%2C+Saudi+Arabia%2E)
9. Elshafei M., Al-Muhtaseb H., and Alghamdi M., (2006), “Statistical Methods for Automatic Diacritization of Arabic Text”. The Saudi 18th National Computer Conference. Riyadh. 18: 301-306. [find](https://scholar.google.com/scholar?q=Elshafei+M%2E%2C+Al%2DMuhtaseb+H%2E%2C+and+Alghamdi+M%2E%2C+%282006%29%2C+%E2%80%9CStatistical+Methods+for+Automatic+Diacritization+of+Arabic+Text%E2%80%9D%2E+The+Saudi+18th+National+Computer+Conference%2E+Riyadh%2E+18%3A+301%2D306%2E)
10. Emam O. and Fisher V. (2004)., A hierarchical approach for the statistical vowelization of Arabic text. Technical report, IBM patent filed, DE9-2004-0006, US patent application US2005/0192809 A1. [find](https://scholar.google.com/scholar?q=Emam+O%2E+and+Fisher+V%2E+%282004%29%2E%2C+A+hierarchical+approach+for+the+statistical+vowelization+of+Arabic+text%2E+Technical+report%2C+IBM+patent+filed%2C+DE9%2D2004%2D0006%2C+US+patent+application+US2005%2F0192809+A1%2E)
11. Gal Y., (2002). [An HMM approach to vowel restoration in Arabic and Hebrew](https://md.research.elamur.ai/free/en/papers/gal-2002-hmm-approach-vowel-restoration-arabic/). In ACL-02 Workshop on Computational Approaches to Semitic Languages.
12. Habash N. and Rambow O., (2007), “[Arabic Diacritization through Full Morphological Tagging](https://md.research.elamur.ai/free/en/papers/habash-2007-arabic-diacritization-through-full-morphological/)”, In Proceedings of the North American chapter of the Association for Computational Linguistics (NAACL), Rochester, New York.
13. Hearst M. A., (1998), "Support Vector Machines," IEEE Intelligent Systems, vol. 13, no. 4, pp. 18-28, Jul/Aug, 1998. [find](https://scholar.google.com/scholar?q=Hearst+M%2E+A%2E%2C+%281998%29%2C+%22Support+Vector+Machines%2C%22+IEEE+Intelligent+Systems%2C+vol%2E+13%2C+no%2E+4%2C+pp%2E+18%2D28%2C+Jul%2FAug%2C+1998%2E)
14. Kirchhoff K. and Vergyri D., (2005). Cross-dialectal data sharing for acoustic modeling in Arabic speech recognition. Speech Communication, 46(1):37–51, May. [find](https://scholar.google.com/scholar?q=Kirchhoff+K%2E+and+Vergyri+D%2E%2C+%282005%29%2E+Cross%2Ddialectal+data+sharing+for+acoustic+modeling+in+Arabic+speech+recognition%2E+Speech+Communication%2C+46%281%29%3A37%E2%80%9351%2C+May%2E)
15. Kudo T. and Matsumoto Y., (2003), " [Fast methods for kernel-based text analysis](https://md.research.elamur.ai/free/en/papers/kudo-2003-fast-methods-kernel-based-text/)," In Proceedings of the 41st Annual Meeting on Association For Computational Linguistics - Volume 1 (Sapporo, Japan, July 07 - 12, 2003). Annual Meeting of the ACL. Association for Computational Linguistics, Morristown.
16. Maamouri, M., Bies, A. & Buckwalter, T. (2004). The Penn Arabic treebank: Building a largescale annotated Arabic corpus. In NEMLAR Conference on Arabic Language Resources and Tools, Cairo, Egypt. [find](https://scholar.google.com/scholar?q=Maamouri%2C+M%2E%2C+Bies%2C+A%2E+%26+Buckwalter%2C+T%2E+%282004%29%2E+The+Penn+Arabic+treebank%3A+Building+a+largescale+annotated+Arabic+corpus%2E+In+NEMLAR+Conference+on+Arabic+Language+Resources+and+Tools%2C+Cairo%2C+Egypt%2E)
17. Maamouri M., Bies A., Kulick S., (2008), "Enhanced Annotation and Parsing of the Arabic Treebank"; INFOS 2008, Cairo, Egypt, March 27-29, 2008. [find](https://scholar.google.com/scholar?q=Maamouri+M%2E%2C+Bies+A%2E%2C+Kulick+S%2E%2C+%282008%29%2C+%22Enhanced+Annotation+and+Parsing+of+the+Arabic+Treebank%22%3B+INFOS+2008%2C+Cairo%2C+Egypt%2C+March+27%2D29%2C+2008%2E)
18. Sang E. and Buchholz S., (2000), [” Introduction to the CoNLL-2000 Shared Task: Chunking”, Proceeding of CoNLL-2000 and LLL-2000,Page 127-132, Lisbon,Portugal](https://md.research.elamur.ai/free/en/papers/sang-2000-introduction-conll-2000-shared-task/).
19. Zitouni I., Sorensen J. S., and Sarikaya R., (2006), “[Maximum entropy based restoration of Arabic diacritics](https://md.research.elamur.ai/free/en/papers/zitouni-2006-maximum-entropy-based-restoration-arabic/)”. In Proceedings of ACL’06.

*Updated 2 November 2026*