---
title: "Smoothing methods for a morpho-statistical approach of"
description: "The authors present AlKhalil Diacritizer, a three-stage hybrid system. 21 references. Among them: 1."
authors: ["Amine Chennoufi", "Azzeddine Mazroui"]
year: 2014
venue: "Proceedings of TALN 2014 (Volume 2: Short Papers)"
source: "https://aclanthology.org/F14-2010/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chennoufi-2014-smoothing-methods-morpho-statistical-approach/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chennoufi-2014-smoothing-methods-morpho-statistical-approach/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2014](https://md.research.elamur.ai/free/en/papers/2014/page/4/) / Smoothing methods for a morpho-statistical approach of automatic diacritization Arabic texts (Méthodes de lissage d'une approche morpho-statistique pour la voyellation automatique des textes arabes) \[in French\]

# Smoothing methods for a morpho-statistical approach of automatic diacritization Arabic texts (Méthodes de lissage d'une approche morpho-statistique pour la voyellation automatique des textes arabes) \[in French\]

**Authors:** Amine Chennoufi and [Azzeddine Mazroui](https://md.research.elamur.ai/free/en/authors/azzeddine-mazroui/) · **Year:** 2014 · **Venue:** [Proceedings of TALN 2014 (Volume 2: Short Papers)](https://md.research.elamur.ai/free/en/venues/jeptalnrecital/) ([JEP-TALN-RECITAL (the French-speaking conferences on speech and natural language processing) 2014](https://md.research.elamur.ai/free/en/venues/jeptalnrecital/2014/)) · **Publisher:** [Association pour le Traitement Automatique des Langues](https://md.research.elamur.ai/free/en/publishers/association-pour-le-traitement-automatique-des-langues/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/F14-2010/) · [PDF](https://aclanthology.org/F14-2010.pdf) · [OpenAlex](https://openalex.org/W2250769825)

**Citations:** 1 citations (OpenAlex, 2026-10-14)

## Summary
Most Arabic text is written without short vowels and other diacritics. This leaves the large majority of words ambiguous, and it hurts parsing, translation, treebank building and speech technology. The authors present AlKhalil Diacritizer, a three-stage hybrid system. In the first stage, the open-source analyzer AlKhalil Morpho Sys proposes every possible vowelized form of each word. A dictionary of frequent words, mined from about 250 million words in eight online corpora, supplements the analyzer. In the second stage, a word-level first-order HMM picks one candidate per word using Viterbi decoding. Its observations are unvowelized words and its hidden states are vowelized words. Its parameters are maximum-likelihood estimates from roughly 57 million vowelized words of Arabic books, and smoothing deals with word transitions never seen in training. In the third stage, a character-level HMM handles words the analyzer cannot process. The system is compared with published diacritizers. It gets the lowest full-word error and diacritic error rates, but it is not the best once case endings are ignored. Errors on the final, syntax-dependent character make up about half of its word errors.

## Key points
- Pipeline:
  - AlKhalil Morpho Sys plus a frequent-word dictionary of 16,369 vowelized entries with their alternative vowelizations.
  - A word-level HMM with smoothing and Viterbi decoding.
  - A character-level HMM fallback for unanalyzed words.
- Training corpus: about 57 million vowelized words from classical and contemporary Arabic books (the Tashkeela resource on SourceForge is cited).
- The emission matrix is effectively binary. An unvowelized word either matches a vowelized form once its diacritics are stripped, or it does not.
- Full-word scores against the strongest compared system (Rashwan et al., 2011):

  | Metric | AlKhalil Diacritizer | Rashwan et al. |
  |---|---|---|
  | WER1 | 10.06% | 12.50% |
  | DER1 | 3.72% | 3.80% |

- On the variants that ignore the final character, Rashwan et al. score better:

  | Metric | AlKhalil Diacritizer | Rashwan et al. |
  |---|---|---|
  | WER2 | 5.40% | 3.10% |
  | DER2 | 2.09% | 1.20% |

- The authors blame part of the remaining error on spelling mistakes in the corpora, which degrade the morphological analyzer. Planned work: clean and extend the corpora, complete partially vowelized dictionary entries, and use the analyzer's syntactic information for word-final diacritics.

## Interesting ideas
- A precomputed lexicon of the most frequent words sits in front of the rule-based analyzer, which both speeds analysis and lets hand-checked forms override analyzer output for the commonest tokens.
- The analyzer generates the candidate set and the statistical model only chooses among those candidates, so the 0/1 emission matrix turns the HMM into a pure contextual selector over linguistically licensed options.
- A finer-grained character-level model serves as a fallback only for words the analyzer cannot handle, rather than replacing the analyzer everywhere.
- Reporting error rates both with and without the final character separates morphological errors from syntax-driven case-ending errors, which shows where the remaining errors come from.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tokens): A frequent-word table with verified lemma/morphology entries could short-circuit the en analyzer on high-frequency forms and pin down their preferred readings explainably.
- **I2** (tokens): Lemma and morphology disambiguation could keep the rule analyzer as the candidate generator and use a sequence model only to rank, so every choice traces back to a valid analysis.
- **I3** (tokens): Unknown or misspelled English words could go to a character-level guesser for POS and lemma, flagged as low-confidence fallback output.
- **I4** (learn): Evaluations could add paired metrics that drop one error class, such as attachment versus label or inflection versus lemma, to locate which component is responsible.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (2): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 3,066 works · [French](https://md.research.elamur.ai/free/en/topics/french/) · 83 works

## Cite

Cite as: Amine Chennoufi and Azzeddine Mazroui. 2014. Smoothing methods for a morpho-statistical approach of automatic diacritization Arabic texts (Méthodes de lissage d'une approche morpho-statistique pour la voyellation automatique des textes arabes) \[in French\]. Proceedings of TALN 2014 (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{chennoufi2014smoothing,
  title = {{Smoothing methods for a morpho-statistical approach of automatic diacritization Arabic texts (Méthodes de lissage d'une approche morpho-statistique pour la voyellation automatique des textes arabes) [in French]}},
  author = {Amine Chennoufi and Azzeddine Mazroui},
  booktitle = {Proceedings of TALN 2014 (Volume 2: Short Papers)},
  year = {2014},
  url = {https://aclanthology.org/F14-2010/}
}
```

- [← previous Innovation of Verbs in Hebrew](https://md.research.elamur.ai/free/en/papers/ornan-2014-innovation-verbs-hebrew/)
- [next → The Biblical Hebrew Feminine Singular Qal Participle: A Historical Reconstruction](https://md.research.elamur.ai/free/en/papers/price-2014-biblical-hebrew-feminine-singular-qal/)
- [↑ back to papers of 2014](https://md.research.elamur.ai/free/en/papers/2014/page/4/)

## References

21 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alghamdi M., Muzaffar Z., Alhakami H. (2010), "Automatic Restoration of Arabic Diacritics: A Simple, Purely Statistical Approach," The Arabian Journal for Science and Engineering, vol. 35, 2010. [find](https://scholar.google.com/scholar?q=Alghamdi+M%2E%2C+Muzaffar+Z%2E%2C+Alhakami+H%2E+%282010%29%2C+%22Automatic+Restoration+of+Arabic+Diacritics%3A+A+Simple%2C+Purely+Statistical+Approach%2C%22+The+Arabian+Journal+for+Science+and+Engineering%2C+vol%2E+35%2C+2010%2E)
2. Buckwalter T. (2004). Arabic Morphological Analyzer version 2.0. LDC2004L02. [find](https://scholar.google.com/scholar?q=Buckwalter+T%2E+%282004%29%2E+Arabic+Morphological+Analyzer+version+2%2E0%2E+LDC2004L02%2E)
3. Chen S. F., Goodman J. (1998). An empirical study of smoothing techniques for language modeling, Harvard Univ.,Computer Science Group, Cambridge, MA, Tech. Rep. TR-10-98, August 1998. [find](https://scholar.google.com/scholar?q=Chen+S%2E+F%2E%2C+Goodman+J%2E+%281998%29%2E+An+empirical+study+of+smoothing+techniques+for+language+modeling%2C+Harvard+Univ%2E%2CComputer+Science+Group%2C+Cambridge%2C+MA%2C+Tech%2E+Rep%2E+TR%2D10%2D98%2C+August+1998%2E)
4. Chennoufi A., Mazroui A. (2014). Vers un Voyaliseur Automatique des Textes Classiques de la Langue Arabe. 1ère Journée Doctorale Nationale sur L'ingénierie de la Langue Arabe, (JDILA'14), 8 Février 2014, Rabat, Maroc. [find](https://scholar.google.com/scholar?q=Chennoufi+A%2E%2C+Mazroui+A%2E+%282014%29%2E+Vers+un+Voyaliseur+Automatique+des+Textes+Classiques+de+la+Langue+Arabe%2E+1%C3%A8re+Journ%C3%A9e+Doctorale+Nationale+sur+L%27ing%C3%A9nierie+de+la+Langue+Arabe%2C+%28JDILA%2714%29%2C+8+F%C3%A9vrier+20)
5. Debili F., Achour H. (1998). Voyellation automatique de l'arabe. In Proceedings of the workshop on Computation approaches to Semitic languages, COLING-ACL '98. Montréal. [find](https://scholar.google.com/scholar?q=Debili+F%2E%2C+Achour+H%2E+%281998%29%2E+Voyellation+automatique+de+l%27arabe%2E+In+Proceedings+of+the+workshop+on+Computation+approaches+to+Semitic+languages%2C+COLING%2DACL+%2798%2E+Montr%C3%A9al%2E)
6. El-Sadany T., Hashish M. (1988). Semi-automatic vowelization of arabic verbs. In 10th NC Conference, Saudi Arabia. [find](https://scholar.google.com/scholar?q=El%2DSadany+T%2E%2C+Hashish+M%2E+%281988%29%2E+Semi%2Dautomatic+vowelization+of+arabic+verbs%2E+In+10th+NC+Conference%2C+Saudi+Arabia%2E)
7. Gale W., Church K.(1994). What is wrong with adding one? In Corpus-Based Research into Language, N. Oostdijk and P. D. Haan, Eds. Amsterdam, The Netherlands: Rodopi, 1994, pp. 189–198. [find](https://scholar.google.com/scholar?q=Gale+W%2E%2C+Church+K%2E%281994%29%2E+What+is+wrong+with+adding+one%3F+In+Corpus%2DBased+Research+into+Language%2C+N%2E+Oostdijk+and+P%2E+D%2E+Haan%2C+Eds%2E+Amsterdam%2C+The+Netherlands%3A+Rodopi%2C+1994%2C+pp%2E+189%E2%80%93198%2E)
8. Gal Y. (2002). [An HMM Approach to Vowel Restoration in Arabic and Hebrew](https://md.research.elamur.ai/free/en/papers/gal-2002-hmm-approach-vowel-restoration-arabic/). In ACL-02 Workshop on Computational Approaches to Semitic Languages.
9. Habash N., Rambow O.(2007), “[Arabic diacritization through full morphological tagging](https://md.research.elamur.ai/free/en/papers/habash-2007-arabic-diacritization-through-full-morphological/),” in Proceedings of NAACL/HLT 2007. Companion Volume, Short Papers, Rochester, New York, USA. April. 2007.pages53-56
10. Hifny Y.(2013). Restoration of arabic diacritics using dynamic programming. In The 8th International Conference on Computer Engineering & Systems (ICCES’2013), 26-28 Nov. 2013, Cairo, Egypt, pages 3-8 ISBN:978-1-4799-0078-7. [find](https://scholar.google.com/scholar?q=Hifny+Y%2E%282013%29%2E+Restoration+of+arabic+diacritics+using+dynamic+programming%2E+In+The+8th+International+Conference+on+Computer+Engineering+%26+Systems+%28ICCES%E2%80%992013%29%2C+26%2D28+Nov%2E+2013%2C+Cairo%2C+Egypt%2C+pages+3%2D8)
11. Maamouri M., Bies A., Kulick S. (2006). Diacritization: A Challenge to Arabic Treebank Annotation and Parsing. In Proceedings of the British Computer Society Arabic NLP/MT Conference. [find](https://scholar.google.com/scholar?q=Maamouri+M%2E%2C+Bies+A%2E%2C+Kulick+S%2E+%282006%29%2E+Diacritization%3A+A+Challenge+to+Arabic+Treebank+Annotation+and+Parsing%2E+In+Proceedings+of+the+British+Computer+Society+Arabic+NLP%2FMT+Conference%2E)
12. Manning C. , Schutze H. (1999). Foundations of Statistical Natural Language Processing, MIT Press. Cambridge, MA. [find](https://scholar.google.com/scholar?q=Manning+C%2E+%2C+Schutze+H%2E+%281999%29%2E+Foundations+of+Statistical+Natural+Language+Processing%2C+MIT+Press%2E+Cambridge%2C+MA%2E)
13. Messaoudi A., Lamel L., Gauvain J. (2004). The LIMSI RT04 BN Arabic system. In Proc. Darpa RT04, Palisades NY. [find](https://scholar.google.com/scholar?q=Messaoudi+A%2E%2C+Lamel+L%2E%2C+Gauvain+J%2E+%282004%29%2E+The+LIMSI+RT04+BN+Arabic+system%2E+In+Proc%2E+Darpa+RT04%2C+Palisades+NY%2E)
14. Neuhoff D.L. (1975). The Viterbi Algorithm as an Aid in Text Recognition. In IEEE Transaction on Information Theory. Pages 222-226. [find](https://scholar.google.com/scholar?q=Neuhoff+D%2EL%2E+%281975%29%2E+The+Viterbi+Algorithm+as+an+Aid+in+Text+Recognition%2E+In+IEEE+Transaction+on+Information+Theory%2E+Pages+222%2D226%2E)
15. Ney H., Essen U., Kneser R. (1994). On structuring probabilistic dependences in stochastic language modeling. Computer, Speech, and Language, vol. 8, pp. 1{38, 1994. Ould Bebah M. O. A, Mazroui. [find](https://scholar.google.com/scholar?q=Ney+H%2E%2C+Essen+U%2E%2C+Kneser+R%2E+%281994%29%2E+On+structuring+probabilistic+dependences+in+stochastic+language+modeling%2E+Computer%2C+Speech%2C+and+Language%2C+vol%2E+8%2C+pp%2E+1%7B38%2C+1994%2E+Ould+Bebah+M%2E+O%2E+A%2C+Mazroui%2E)
16. A, Meziane A., Lakhouaja A. (2011). Alkhali Morpho Sys. In International Computing Conference in Arabic. Riadh, Arabie Saoudite. [find](https://scholar.google.com/scholar?q=A%2C+Meziane+A%2E%2C+Lakhouaja+A%2E+%282011%29%2E+Alkhali+Morpho+Sys%2E+In+International+Computing+Conference+in+Arabic%2E+Riadh%2C+Arabie+Saoudite%2E)
17. Ould Bebah M. O. A., Mazroui A ., Lakhouaja A., Chennoufi A.(2013). Approche morpho-statistique pour la voyellation automatique des texts arabes. In 8th International Conference on intelligent system: theories and applications (SITA’13), May 08-09, 2013, EMI, Rabat , Morocco. [find](https://scholar.google.com/scholar?q=Ould+Bebah+M%2E+O%2E+A%2E%2C+Mazroui+A+%2E%2C+Lakhouaja+A%2E%2C+Chennoufi+A%2E%282013%29%2E+Approche+morpho%2Dstatistique+pour+la+voyellation+automatique+des+texts+arabes%2E+In+8th+International+Conference+on+intelligent+system%3A)
18. Vergyri D., Kirchoff K. (2004). [Automatic diacritization of Arabic for Acoustic Modeling in Speech Recognition](https://md.research.elamur.ai/free/en/papers/vergyri-2004-automatic-diacritization-arabic-acoustic-modeling/). In Proceedings of the Workshop on Computational Approaches to Arabic Script-based Languages. Geneva. Pages 66-73. [↗](https://doi.org/10.21236/ada457846)
19. Rashwan M., Al-Badrashiny M. , Attia M., Abdou S.M., Rafea A. (2011), "A Stochastic Arabic Diacritizer Based on a Hybrid of Factorized and Unfactorized Textual Features," Audio, Speech, and Language Processing, pages. 166-175. [↗](https://doi.org/10.1109/tasl.2010.2045240)
20. Schlippe T., Nguyen T. , Vogel S. (2008). Diacritization as a Machine Translation Problem and as a Sequence Labeling Problem. In 8th AMTA conference, Hawaii. Pages 21-25. [find](https://scholar.google.com/scholar?q=Schlippe+T%2E%2C+Nguyen+T%2E+%2C+Vogel+S%2E+%282008%29%2E+Diacritization+as+a+Machine+Translation+Problem+and+as+a+Sequence+Labeling+Problem%2E+In+8th+AMTA+conference%2C+Hawaii%2E+Pages+21%2D25%2E)
21. Zitouni I., Sarikaya R. (2009), "Arabic Diacritic Restoration Approach Based on Maximum Entropy Models," Computer Speech & Language, vol. 23, no. 3, pp. 257-276, 2009. [↗](https://doi.org/10.1016/j.csl.2008.06.001)

*Updated 2 November 2026*