--- title: "Smoothing a Lexicon-based POS Tagger for — Manour 2007" description: "The authors port Bar-Haim et al.'s Hebrew HMM tagger (MorphTagger) to Modern Standard Arabic by swapping in Buckwalter's analyser and training on the…" authors: ["Saib Manour", "Khalil Sima'an", "Yoad Winter"] year: 2007 venue: "Proceedings of the 2007 Workshop on Computational Approaches to Semitic Languages: Common Issues and Resources" source: "https://aclanthology.org/W07-0813/" doi: "10.3115/1654576.1654593" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/aclbulk-2007-manour-smoothing-lexicon-based-pos-tagger/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/aclbulk-2007-manour-smoothing-lexicon-based-pos-tagger/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2007](https://md.research.elamur.ai/free/en/papers/2007/) / Smoothing a Lexicon-based POS Tagger for Arabic and Hebrew # Smoothing a Lexicon-based POS Tagger for Arabic and Hebrew **Authors:** Saib Mansour; [Khalil Sima'an](https://md.research.elamur.ai/free/en/authors/khalil-sima-an/); [Yoad Winter](https://md.research.elamur.ai/free/en/authors/yoad-winter/) · **Year:** 2007 · **Venue:** Proceedings of the 2007 Workshop on Computational Approaches to Semitic Languages: Common Issues and Resources **Link:** [aclanthology.org/W07-0813](https://aclanthology.org/W07-0813/) · library `aclbulk-2007-manour-smoothing-lexicon-based-pos-tagger` · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) (ACL Anthology, before 2016) — summary only [**24** citations](https://openalex.org/works?filter=cites:W2033830177) OpenAlex · 2026-10-14 · [**10** citations in our library](https://md.research.elamur.ai/free/en/papers/aclbulk-2007-manour-smoothing-lexicon-based-pos-tagger/cited-by/) of the 140673 papers whose bibliographies we read · [Proceedings of the 2007 Workshop on Computational Approaches to Semitic Languages: Common Issues and Resources](https://aclanthology.org/volumes/W07-08/) · [licence](https://creativecommons.org/licenses/by-nc-sa/3.0/) - [Publisher's page](https://aclanthology.org/W07-0813/) - [PDF](https://aclanthology.org/W07-0813.pdf) - [DOI 10.3115/1654576.1654593](https://doi.org/10.3115/1654576.1654593) - [OpenAlex](https://openalex.org/W2033830177) **Authors (3)** 1. Saib Manour [Semantic Scholar](https://www.semanticscholar.org/author/1405826362) 2. [Khalil Sima'an](https://openalex.org/A5034829413) [Semantic Scholar](https://www.semanticscholar.org/author/3540477) · h 23 · 2396 citations · 168 works 3. [Yoad Winter](https://openalex.org/A5070230184) [ORCID](https://orcid.org/0000-0002-7209-710X) · [Semantic Scholar](https://www.semanticscholar.org/author/2021738) · h 28 · 3517 citations · 194 works Citations and author metrics: OpenAlex, 2026-10-14; authors with the same name are not always told apart. ## What it is about The authors port Bar-Haim et al.'s Hebrew HMM tagger (MorphTagger) to Modern Standard Arabic by swapping in Buckwalter's analyser and training on the Arabic Treebank. This gives 96.12% tagging accuracy on the test setting used by Diab et al. (95.81%), close to Habash and Rambow (97.5%). The main limit is coverage, not unknown words: 2.8% of ATB1 words, 4.5% of ATB2 and 5.3% of ATB3 have a correct analysis missing from the analyser. Missing proper nouns, nouns and adjectives make up most of this. Adding character-based SVM analyses to every sentence reaches 96.18%, but 60% of the added analyses are wrong. Adding them only when the sentence's HMM probability falls below a threshold gives 96.28% and raises analysis coverage from 97% to 98%. Hebrew has about 2.17 analyses per word and about 4% of words with a missing correct analysis. ## Ideas - I1. Coverage audit for Hebrew: count the tokens whose correct analysis (from the tagged corpus) is absent from the analyser, and split the count by POS class, as Table 3 does. Run it on `tools/hbo` lexical tables before any model is added. - I2. Gate any fallback analyses on the sentence probability: use an extra candidate source (for example a character model for unpointed forms) only when the best path under the lexicon model has low probability. Calibrate the threshold on a dev part and report accuracy by part, as the paper's ATB1, ATB2 and ATB3 split does. - I3. For unknown or missing words, test a proper-name fallback first: a rule that gives every unanalysed form a proper-noun reading is almost enough on the newer ATB parts. Test the same fallback on Hebrew OOV words in `tools/hbo`. ## For Mova I1 is a measure for `tools/hbo` lexical tables and the dictionary builder; it goes into the coverage notes of [languages/hbo/10-lexicon.md](https://mova.elamur.ai/free/en/docs/languages/hbo/10-lexicon/). I2 is a gate in the segmenter of `tools/hbo` (the fallback candidates for unseen forms). I3 is a test on [languages/he/08-lexicon-register.md](https://mova.elamur.ai/free/en/docs/languages/he/08-lexicon-register/) and on the unknown-word path of `tools/hbo`. The paper is a 2007 HMM; Mova needs the coverage measure and the gate, not the tagger itself. Topics: [Part-of-speech tagging](https://md.research.elamur.ai/free/en/topics/part-of-speech-tagging/) ## Cite The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.3115/1654576.1654593](https://doi.org/10.3115/1654576.1654593) ```bibtex @inproceedings{manour2007smoothing, title = {{Smoothing a Lexicon-based POS Tagger for Arabic and Hebrew}}, author = {Manour, Saib and Sima'an, Khalil and Winter, Yoad}, booktitle = {Proceedings of the 2007 Workshop on Computational Approaches to Semitic Languages: Common Issues and Resources}, year = {2007}, doi = {10.3115/1654576.1654593}, url = {https://doi.org/10.3115/1654576.1654593} } ``` - [← previous Semi-automatic error analysis for large-scale statistical machine translation](https://md.research.elamur.ai/free/en/papers/kirchhoff-2007-semi-automatic-error-analysis-large/) - [next → SVM Model Tampering and Anchored Learning: A Case Study in Hebrew NP Chunking](https://md.research.elamur.ai/free/en/papers/aclbulk-2007-goldberg-svm-model-tampering-anchored-learning/) - [↑ back to papers of 2007](https://md.research.elamur.ai/free/en/papers/2007/) ## References 12 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Roy Bar Haim, Khalil Sima’an and Yoad Winter. 2005. Choosing an Optimal Architecture for Segmentation and POS-Tagging of Modern Hebrew. ACL Workshop on Computational Approaches to Semitic Languages. A revised and extended version to appear in Journal of Natural Language Engineering. [find](https://scholar.google.com/scholar?q=Roy+Bar+Haim%2C+Khalil+Sima%E2%80%99an+and+Yoad+Winter%2E+2005%2E+Choosing+an+Optimal+Architecture+for+Segmentation+and+POS%2DTagging+of+Modern+Hebrew%2E+ACL+Workshop+on+Computational+Approaches+to+Semitic+Languages%2E+A) [our card](https://md.research.elamur.ai/free/en/papers/aclbulk-2005-haim-choosing-optimal-architecture-segmentation-pos/) 2. Eric Brill. 1995. Transformation-based error-driven learning and natural language processing: a case study in part of speech tagging. In Computational Linguistics 21, pages 543-565. [find](https://scholar.google.com/scholar?q=Eric+Brill%2E+1995%2E+Transformation%2Dbased+error%2Ddriven+learning+and+natural+language+processing%3A+a+case+study+in+part+of+speech+tagging%2E+In+Computational+Linguistics+21%2C+pages+543%2D565%2E) [our card](https://md.research.elamur.ai/free/en/papers/brill-1995-transformation-based-error-driven-learning/) 3. Tim Buckwalter. 2002. Arabic Morphological Analyzer Version 1.0. Linguistic Data Consortium, University of Pennsylvania. [find](https://scholar.google.com/scholar?q=Tim+Buckwalter%2E+2002%2E+Arabic+Morphological+Analyzer+Version+1%2E0%2E+Linguistic+Data+Consortium%2C+University+of+Pennsylvania%2E) 4. Kenneth W. Church. 1988. A stochastic parts program and noun phrase parser for unrestricted text. Proceedings of the second conference on Applied natural language processing, Pages 136-143. [find](https://scholar.google.com/scholar?q=Kenneth+W%2E+Church%2E+1988%2E+A+stochastic+parts+program+and+noun+phrase+parser+for+unrestricted+text%2E+Proceedings+of+the+second+conference+on+Applied+natural+language+processing%2C+Pages+136%2D143%2E) 5. Mona Diab, Kadri Hacioglu, and Daniel Jurafsky. 2004. Automatic Tagging of Arabic Text: From Raw Text to Base Phras e Chunks. In HLT-NAACL: Short Papers, pages 149-152. [find](https://scholar.google.com/scholar?q=Mona+Diab%2C+Kadri+Hacioglu%2C+and+Daniel+Jurafsky%2E+2004%2E+Automatic+Tagging+of+Arabic+Text%3A+From+Raw+Text+to+Base+Phras+e+Chunks%2E+In+HLT%2DNAACL%3A+Short+Papers%2C+pages+149%2D152%2E) 6. Nizar Habash and Owen Rambow. 2005. Arabic Tokenization, Part-of-Speech Tagging and Morphological Disambiguation in One Fell Swoop. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics, pages 573-580, Ann Arbor. [find](https://scholar.google.com/scholar?q=Nizar+Habash+and+Owen+Rambow%2E+2005%2E+Arabic+Tokenization%2C+Part%2Dof%2DSpeech+Tagging+and+Morphological+Disambiguation+in+One+Fell+Swoop%2E+In+Proceedings+of+the+43rd+Annual+Meeting+of+the+Association+for+Com) [our card](https://md.research.elamur.ai/free/en/papers/aclbulk-2005-habash-arabic-tokenization-part-speech-tagging/) 7. Young-Suk Lee, Kishore Papineni, Salim Roukos, Ossama Emam, and Hany Hassan. 2003. Language model based Arabic word segmentation. In ACL, pages 399-406. [find](https://scholar.google.com/scholar?q=Young%2DSuk+Lee%2C+Kishore+Papineni%2C+Salim+Roukos%2C+Ossama+Emam%2C+and+Hany+Hassan%2E+2003%2E+Language+model+based+Arabic+word+segmentation%2E+In+ACL%2C+pages+399%2D406%2E) 8. Mohamed Maamouri and Ann Bies. 2004. Developing an Arabic treebank: Methods, guidelines, procedures, and tools. In Proceedings of the Workshop on Computational Approaches to Arabic Script-based Languages (COLING), Geneva. [find](https://scholar.google.com/scholar?q=Mohamed+Maamouri+and+Ann+Bies%2E+2004%2E+Developing+an+Arabic+treebank%3A+Methods%2C+guidelines%2C+procedures%2C+and+tools%2E+In+Proceedings+of+the+Workshop+on+Computational+Approaches+to+Arabic+Script%2Dbased+Langua) [our card](https://md.research.elamur.ai/free/en/papers/bank-2004-maamouri-developing-arabic-treebank-methods-guidelines/) 9. Christopher D. Manning and Hinrich Schütze. 1999. Foundations of Statistical Natural Language Processing. The MIT press, Cambridge, Massachusetts. [find](https://scholar.google.com/scholar?q=Christopher+D%2E+Manning+and+Hinrich+Sch%C3%BCtze%2E+1999%2E+Foundations+of+Statistical+Natural+Language+Processing%2E+The+MIT+press%2C+Cambridge%2C+Massachusetts%2E) 10. Tetsuji Nakagawa. 2004. Chinese and Japanese word segmentation using word-level and character-level information. In Proceedings of the 20th International Conference on Computational Linguistics, pages 466-472, Geneva. [find](https://scholar.google.com/scholar?q=Tetsuji+Nakagawa%2E+2004%2E+Chinese+and+Japanese+word+segmentation+using+word%2Dlevel+and+character%2Dlevel+information%2E+In+Proceedings+of+the+20th+International+Conference+on+Computational+Linguistics%2C+pages) 11. Erel Segal. 2001. Hebrew morphological analyzer for Hebrew undotted texts. Master's thesis, Computer Science Department, Technion, Haifa, Israel. [find](https://scholar.google.com/scholar?q=Erel+Segal%2E+2001%2E+Hebrew+morphological+analyzer+for+Hebrew+undotted+texts%2E+Master%27s+thesis%2C+Computer+Science+Department%2C+Technion%2C+Haifa%2C+Israel%2E) 12. Ralph Weischedel, Marie Meteer, Richard Schwartz, Lance Ramshaw and Jeff Palmucci. 1993. Coping with Ambiguity and Unknown Words through Probabilistic Models. Computational Linguistics (Special issue on using large corpora: II) volume 19, pages 361-382. [find](https://scholar.google.com/scholar?q=Ralph+Weischedel%2C+Marie+Meteer%2C+Richard+Schwartz%2C+Lance+Ramshaw+and+Jeff+Palmucci%2E+1993%2E+Coping+with+Ambiguity+and+Unknown+Words+through+Probabilistic+Models%2E+Computational+Linguistics+%28Special+issue+)