--- title: "Automatic Arabic diacritics restoration based on deep nets (2014)" description: "Modern Standard Arabic is usually written without diacritics, yet speech synthesis, translation and… 6 references, cited by 2 papers here." authors: ["Ahmad Al Sallab", "Mohsen Rashwan", "Hazem M. Raafat", "Ahmed Rafea"] year: 2014 venue: "Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP)" source: "https://aclanthology.org/W14-3608/" doi: "10.3115/v1/w14-3608" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/sallab-2014-automatic-arabic-diacritics-restoration-based/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sallab-2014-automatic-arabic-diacritics-restoration-based/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2014](https://md.research.elamur.ai/free/en/papers/2014/page/3/) / Automatic Arabic diacritics restoration based on deep nets # Automatic Arabic diacritics restoration based on deep nets **Authors:** [Ahmad Al Sallab](https://md.research.elamur.ai/free/en/authors/ahmad-al-sallab/), [Mohsen Rashwan](https://md.research.elamur.ai/free/en/authors/mohsen-rashwan/), [Hazem M. Raafat](https://md.research.elamur.ai/free/en/authors/hazem-raafat/) and [Ahmed Rafea](https://md.research.elamur.ai/free/en/authors/ahmed-rafea/) · **Year:** 2014 · **Venue:** [Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP)](https://md.research.elamur.ai/free/en/venues/emnlp-workshop-on-arabic-natural-language-processing-anlp/) ([ANLP 2014](https://md.research.elamur.ai/free/en/venues/emnlp-workshop-on-arabic-natural-language-processing-anlp/2014/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/W14-3608/) · [PDF](https://aclanthology.org/W14-3608.pdf) · [DOI 10.3115/v1/w14-3608](https://doi.org/10.3115/v1/w14-3608) · [OpenAlex](https://openalex.org/W2251265054) **Citations:** 21 citations (OpenAlex, 2026-10-14) · [2](https://md.research.elamur.ai/free/en/papers/sallab-2014-automatic-arabic-diacritics-restoration-based/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Modern Standard Arabic is usually written without diacritics, yet speech synthesis, translation and similar applications need them restored. The authors treat restoration as per-character classification over diacritic classes. They separate morphological diacritization (word-internal marks) from syntactic diacritization (case endings), and the syntactic part is their main target because earlier systems seemed to plateau there. They train Deep Neural Networks with DBN pre-training. Input features include PoS, morphological quadruples, the last character, a character-level encoding of the word, and context from neighbouring words, including the previously predicted diacritic. Standalone DNN PoS taggers are also built so the system can run on raw text. Their Confused Subset Resolution (CSR) method reads the training confusion matrix and adds sub-classifiers for class groups that get mixed up. On custom RDI data and LDC Arabic Tree Bank Part 3, syntactic error falls to 9.9% and morphological error to 3%, compared with 12.7% and 3.8% for the best previous systems. This is roughly a 22% relative error reduction. ## Key points - Data: custom TRN_DB_I and TRN_DB_II for training and TST_DB for testing (RDI, available online, drawn from varied sources), plus the standard LDC Arabic Tree Bank Part 3 (LDC2005T20) for benchmarking. - Baselines: the hybrid A*/n-gram system of Rashwan et al. (2009, 2011), MaxEnt by Zitouni et al. (2006), and MADA+SVM by Habash and Rambow (2007). - Context window: three preceding words and one following word gave the best results among the tested configurations. - Network setup: - The classifier uses three equal-width layers (20-20-20) and the PoS taggers use 60-60-60. - Pre-training is DBN for 20 epochs per layer, followed by 30 epochs of conjugate-gradient fine-tuning. - Reported gains: - The last-character feature cuts error by about 4%. - Autoregressive label context cuts it by about 1.1%. - CSR cuts it by about 2%. ## Interesting ideas - Turn the confusion matrix into architecture: find class pairs or subsets the main model mixes up, train a dedicated sub-classifier for each, and nest subsets that sit inside larger ones deeper in a tree. This gives targeted fixes without retraining the whole model. - Encode a word as its sequence of characters rather than as a vocabulary identity, so an unseen word lands near similar known words. This reduces out-of-vocabulary failures and keeps input vectors short. - Feed the model's own previous output label back in as a context feature, in an autoregressive way. It is a cheap way to capture dependencies between labels in a sequence. - Report errors separately for the hard sub-task (case endings) and the easy one (word-internal marks), and run per-feature ablations. This shows where the real ceiling lies instead of hiding it in one averaged score. - Include a small tagger trained on the gold-annotated part of the data so the full pipeline can work on raw text, and use that tagger to label the remaining unannotated portion. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (tokens): Mova's tagger could get small arbiters for its known confusion sets, such as VBD/VBN or IN/RP/RB, triggered only when the main tagger outputs one of those tags. - **I2** (tokens): Character-based word features could back the morphology and lemma guesser for rare or novel English words, with the nearest known spelling given as the explanation. - **I3** (tree): The perceptron parser could add the already-predicted relation of the previous token or head as a feature, which helps keep chains like conj/cc labelling consistent. - **I4** (learn): Evaluation could split parser or tagger accuracy into difficulty strata and log an ablation for each feature, so that gains on the hard slice stay visible. - **I5** (learn): Partial gold annotations could train an auxiliary tagger that silver-labels the rest of a corpus, then cross-checked against the witness parsers. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works 🧭 Topics (2): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works · [🧩 Morphology](https://md.research.elamur.ai/free/en/topics/morphology/) · 405 works ## Cite Cite as: Ahmad Al Sallab, Mohsen Rashwan, Hazem M. Raafat and Ahmed Rafea. 2014. Automatic Arabic diacritics restoration based on deep nets. Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{sallab2014automatic, title = {{Automatic Arabic diacritics restoration based on deep nets}}, author = {Ahmad {Al Sallab}, Mohsen Rashwan, Hazem M. Raafat and Ahmed Rafea}, booktitle = {Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP)}, year = {2014}, doi = {10.3115/v1/w14-3608}, url = {https://doi.org/10.3115/v1/w14-3608} } ``` - [← previous AusTalk: an audio-visual corpus of Australian English](https://md.research.elamur.ai/free/en/papers/estival-2014-austalk-audio-visual-corpus-australian/) - [next → Coreference Resolution Evaluation for Higher Level Applications](https://md.research.elamur.ai/free/en/papers/tuggener-2014-coreference-resolution-evaluation-higher-level/) - [↑ back to papers of 2014](https://md.research.elamur.ai/free/en/papers/2014/page/3/) ## References 6 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Rashwan, Mohsen A A; Attia, Mohamed; Abdou, Sherif M.; Abdou, S.; Rafea, Ahmed A. 2009. “A Hybrid System for Automatic Arabic Diacritization”, International Conference on Natural Language Processing and Knowledge Engineering, pp 1-8, 24-27. [find](https://scholar.google.com/scholar?q=Rashwan%2C+Mohsen+A+A%3B+Attia%2C+Mohamed%3B+Abdou%2C+Sherif+M%2E%3B+Abdou%2C+S%2E%3B+Rafea%2C+Ahmed+A%2E+2009%2E+%E2%80%9CA+Hybrid+System+for+Automatic+Arabic+Diacritization%E2%80%9D%2C+International+Conference+on+Natural+Language+Processing+a) 2. Rashwan, Mohsen A A , Al-Badrashiny, Mohamed A S A A; Attia, Mohamed; Abdou, Sherif M.; Rafea, Ahmed A. 2011. “A Stochastic Arabic Diacritizer Based on a Hybrid of Factorized and Unfactorized Textual Features”, IEEE Transactions on Audio, Speech, and Language Processing, vol. 19, issue 1, pp 166-175. [↗](https://doi.org/10.1109/tasl.2010.2045240) 3. I. Zitouni; J. S. Sorensen; R. Sarikaya, 2006. “[Maximum Entropy Based Restoration of Arabic Diacritics](https://md.research.elamur.ai/free/en/papers/zitouni-2006-maximum-entropy-based-restoration-arabic/)”, Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics (ACL); Workshop on Computational Approaches to Semitic Languages; Sydney-Australia N. Habash; O. Rambo. 2007. “Arabic Diacritization through Full Morphological Tagging”, Proceedings of the 8th Meeting of the North American Chapter of the Association for Computational Linguistics (ACL); Human Language Technologies Conference (HLT-NAACL). 4. G. E. Hinton; S. Osindero; Y. Teh, “A fast learning algorithm for deep belief nets” Neural Computation, vol. 18, pp. 1527–1554, 2006. [↗](https://doi.org/10.1162/neco.2006.18.7.1527) 5. Ruslan Salakhutdinov. 2009.“Learning Deep Generative Models” PhD thesis, Graduate Department of Computer Science, University of Toronto, Raafat, H.; Rashwan, M.A.A. 1993. “A tree structured neural network, Proceedings of the Second International Conference on Document Analysis and Recognition” , pp. 939 – 941, ISBN: 0-8186-4960-7 [find](https://scholar.google.com/scholar?q=Ruslan+Salakhutdinov%2E+2009%2E%E2%80%9CLearning+Deep+Generative+Models%E2%80%9D+PhD+thesis%2C+Graduate+Department+of+Computer+Science%2C+University+of+Toronto%2C+Raafat%2C+H%2E%3B+Rashwan%2C+M%2EA%2EA%2E+1993%2E+%E2%80%9CA+tree+structured+neural+net) 6. Hai-Son Le ; Oparin, I. ; Allauzen, A. ; Gauvain, J., 2011. “Structured Output Layer neural network language model” 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5524 - 5527, ISBN: 978-1-4577-0537-3 [http://www.RDI-eg.com/RDI/TrainingData](http://www.RDI-eg.com/RDI/TrainingData) is where to download TRN_DB_II. http://www.RDI-eg.com/RDI/TestData is where to download TST_DB LDC Arabic Tree Bank Part 3, http://www.ldc.upenn.edu/Catalog/catalogEntry.jsp?c atalogId=LDC2005T20 [↗](http://www.RDI-eg.com/RDI/TrainingData) *Updated 1 November 2026*