--- title: "Unsupervised Concept Discovery In Hebrew Using Simple Unsupervised Word Prefix — Dinur 2009" description: "Hebrew and Arabic attach function words (and, in, the, that) to the next word, so word-based pattern methods for… 22 references, cited by 2 papers here." authors: ["Elad Dinur", "Dmitry Davidov", "Ari Rappoport"] year: 2009 venue: "Proceedings of the EACL 2009 Workshop on Computational Approaches to Semitic Languages" source: "https://aclanthology.org/W09-0805/" doi: "10.3115/1621774.1621782" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/dinur-2009-unsupervised-concept-discovery-hebrew-simple/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/dinur-2009-unsupervised-concept-discovery-hebrew-simple/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2009](https://md.research.elamur.ai/free/en/papers/2009/) / Unsupervised Concept Discovery In Hebrew Using Simple Unsupervised Word Prefix Segmentation for Hebrew and Arabic # Unsupervised Concept Discovery In Hebrew Using Simple Unsupervised Word Prefix Segmentation for Hebrew and Arabic **Authors:** Elad Dinur; [Dmitry Davidov](https://md.research.elamur.ai/free/en/authors/dmitry-davidov/); [Ari Rappoport](https://md.research.elamur.ai/free/en/authors/ari-rappoport/) · **Year:** 2009 · **Venue:** Proceedings of the EACL 2009 Workshop on Computational Approaches to Semitic Languages **Link:** [aclanthology.org/W09-0805](https://aclanthology.org/W09-0805/) · library `aclbulk-2009-dinur-unsupervised-concept-discovery-hebrew-simple` · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) (ACL Anthology, before 2016) — summary only [**2** citations](https://openalex.org/works?filter=cites:W1977618669) OpenAlex, 2026-10-14 · **2** citations in our library of the 140,884 papers whose bibliographies we read · [Proceedings of the EACL 2009 Workshop on Computational Approaches to Semitic Languages](https://aclanthology.org/volumes/W09-08/) · [licence](https://creativecommons.org/licenses/by-nc-sa/3.0/) - [Publisher's page](https://aclanthology.org/W09-0805/) - [PDF](https://aclanthology.org/W09-0805.pdf) - [DOI 10.3115/1621774.1621782](https://doi.org/10.3115/1621774.1621782) - [OpenAlex](https://openalex.org/W1977618669) **Authors (3)** 1. [Elad Dinur](https://openalex.org/A5007657377) [Semantic Scholar](https://www.semanticscholar.org/author/2895721) · h 2 · 117 citations · 3 works 2. [Dmitry Davidov](https://openalex.org/A5034956621) [Semantic Scholar](https://www.semanticscholar.org/author/33406957) · h 11 · 1,937 citations · 24 works 3. [Ari Rappoport](https://openalex.org/A5038467328) [Semantic Scholar](https://www.semanticscholar.org/author/145009917) · h 40 · 5,585 citations · 146 works Citations and author metrics: OpenAlex, 2026-10-14; authors with the same name are not always told apart. ## What it is about Hebrew and Arabic attach function words (and, in, the, that) to the next word, so word-based pattern methods for concept discovery miss the patterns that use "and" or "or". The authors split off the closed set of prefixes without any annotation. A word is split only when a rank, which compares the frequency of the prefixed form with the frequency of the stem and a short-word factor, is below a threshold; a trigram model over morphemes, found by dynamic programming, picks the best split of each sentence. On a 24,000-word hand-segmented Hebrew sample the method reaches F 0.851 and accuracy 0.893, against 0.523 for a baseline and 0.658 for Morfessor. On a 130 MB corpus, segmentation raises the number of Hebrew concepts from 36 to 148 and their average size from 2.9 to 4.1, with one junk word; without it, about ten times as much data is needed for similar results. ## For Mova - I1. A rank-and-threshold split for unseen prefixed forms in `tools/hbo`: keep the closed prefix list the segmenter already uses, and score each candidate split by the frequency ratio of the prefixed form to its stem, with a length factor for short words. Test it on the unseen-word path against a hand-segmented sample, and report F and accuracy. - I2. Use the prefix "w" (and) as a coordination signal: a pattern "x w y" with the content words on both sides is a symmetric relation. Test whether the coordination reading helps the event chain or the coordinated-NP attachment in [languages/hbo/08-syntax.md](https://mova.elamur.ai/free/en/docs/languages/hbo/08-syntax/). - I3. Measure how much corpus Mova's lexicon needs with and without prefix splitting, on the same corpus size, by counting distinct lexemes. This is a cheap check of the segmenter's value for `world`. I1 is a fallback for unseen forms in `tools/hbo` (the segmenter; see [languages/hbo/13-mova-tools.md](https://mova.elamur.ai/free/en/docs/languages/hbo/13-mova-tools/)). I2 belongs to [languages/hbo/08-syntax.md](https://mova.elamur.ai/free/en/docs/languages/hbo/08-syntax/) (coordination) and [languages/he/06-prepositions-particles.md](https://mova.elamur.ai/free/en/docs/languages/he/06-prepositions-particles/) (the prefix "w"). I3 is a measurement for [languages/he/12-mova.md](https://mova.elamur.ai/free/en/docs/languages/he/12-mova/), run on the corpus outside the repo. The method is unsupervised and needs only the prefix list, which fits Mova's rules. The paper notes that the "bbyt" form has two readings; keep it as a test case for `tools/hbo`. ## Institutions [🎓**Hebrew University of Jerusalem** Jerusalem · Israel 42 works · 8,354 citations](https://md.research.elamur.ai/free/en/universities/hebrew-university-of-jerusalem/) Topics[Hebrew & Aramaic](https://md.research.elamur.ai/free/en/topics/hebrew-aramaic/) · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) ## Cite Cite as Elad Dinur, Dmitry Davidov and Ari Rappoport. 2009. Unsupervised Concept Discovery In Hebrew Using Simple Unsupervised Word Prefix Segmentation for Hebrew and Arabic. Proceedings of the EACL 2009 Workshop on Computational Approaches to Semitic Languages. The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.3115/1621774.1621782](https://doi.org/10.3115/1621774.1621782) ```bibtex @inproceedings{dinur2009unsupervised, title = {{Unsupervised Concept Discovery In Hebrew Using Simple Unsupervised Word Prefix Segmentation for Hebrew and Arabic}}, author = {Elad Dinur, Dmitry Davidov and Ari Rappoport}, booktitle = {Proceedings of the EACL 2009 Workshop on Computational Approaches to Semitic Languages}, year = {2009}, doi = {10.3115/1621774.1621782}, url = {https://doi.org/10.3115/1621774.1621782} } ``` - [← previous The Karamel System and Semitic Languages: Structured Multi-Tiered Morphology](https://md.research.elamur.ai/free/en/papers/barthelemy-2009-karamel-system-semitic-languages-structured/) - [next → Effective Analysis of Causes and Inter-dependencies of Parsing Errors](https://md.research.elamur.ai/free/en/papers/hara-2009-effective-analysis-causes-inter-dependencies/) - [↑ back to papers of 2009](https://md.research.elamur.ai/free/en/papers/2009/) ## References 22 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Meni Adler, Michael Elhadad, 2006. An Unsupervised Morpheme-Based HMM for Hebrew Morphological Disambiguation. ACL ’06. [our card](https://md.research.elamur.ai/free/en/papers/adler-2006-unsupervised-morpheme-based-hmm-hebrew/) 2. Roy Bar-Haim, Khalil Simaan, Yoad Winter, 2005. Choosing an Optimal Architecture for Segmentation and POS-Tagging of Modern Hebrew. ACL Workshop on Computational Approaches to Semitic Languages ’05. [our card](https://md.research.elamur.ai/free/en/papers/haim-2005-choosing-optimal-architecture-segmentation-pos/) 3. Mathias Creutz, 2003. Unsupervised Segmentation of Words Using Prior Distributions of Morph Length and Frequency. ACL ’03. [find](https://scholar.google.com/scholar?q=Mathias+Creutz%2C+2003%2E+Unsupervised+Segmentation+of+Words+Using+Prior+Distributions+of+Morph+Length+and+Frequency%2E+ACL+%E2%80%9903%2E) 4. Mathias Creutz and Krista Lagus, 2005. Unsupervised Morpheme Segmentation and Morphology Induction from Text Corpora Using Morfessor 1.0. In Computer and Information Science, Report A81, Helsinki University of Technology. [find](https://scholar.google.com/scholar?q=Mathias+Creutz+and+Krista+Lagus%2C+2005%2E+Unsupervised+Morpheme+Segmentation+and+Morphology+Induction+from+Text+Corpora+Using+Morfessor+1%2E0%2E+In+Computer+and+Information+Science%2C+Report+A81%2C+Helsinki+Univ) 5. Sajib Dasgupta and Vincent Ng, 2007. High Performance, Language-Independent Morphological Segmentation. NAACL/HLT ’07. [find](https://scholar.google.com/scholar?q=Sajib+Dasgupta+and+Vincent+Ng%2C+2007%2E+High+Performance%2C+Language%2DIndependent+Morphological+Segmentation%2E+NAACL%2FHLT+%E2%80%9907%2E) 6. Dmitry Davidov, Ari Rappoport, 2006. Efficient Unsupervised Discovery of Word Categories Using Symmetric Patterns and High Frequency Words. ACL ’06. [find](https://scholar.google.com/scholar?q=Dmitry+Davidov%2C+Ari+Rappoport%2C+2006%2E+Efficient+Unsupervised+Discovery+of+Word+Categories+Using+Symmetric+Patterns+and+High+Frequency+Words%2E+ACL+%E2%80%9906%2E) 7. Dmitry Davidov, Ari Rappoport, Moshe Koppel, 2007. Fully Unsupervised Discovery of Concept-Specific Relationships by Web Mining. ACL ’07. [find](https://scholar.google.com/scholar?q=Dmitry+Davidov%2C+Ari+Rappoport%2C+Moshe+Koppel%2C+2007%2E+Fully+Unsupervised+Discovery+of+Concept%2DSpecific+Relationships+by+Web+Mining%2E+ACL+%E2%80%9907%2E) 8. Dmitry Davidov, Ari Rappoport, 2008a. Classification of Semantic Relationships between Nominals Using Pattern Clusters. ACL ’08. [find](https://scholar.google.com/scholar?q=Dmitry+Davidov%2C+Ari+Rappoport%2C+2008a%2E+Classification+of+Semantic+Relationships+between+Nominals+Using+Pattern+Clusters%2E+ACL+%E2%80%9908%2E) 9. Dmitry Davidov, Ari Rappoport, 2008b. Unsupervised Discovery of Generic Relationships Using Pattern Clusters and its Evaluation by Automatically Generated SAT Analogy Questions. ACL ’08. [find](https://scholar.google.com/scholar?q=Dmitry+Davidov%2C+Ari+Rappoport%2C+2008b%2E+Unsupervised+Discovery+of+Generic+Relationships+Using+Pattern+Clusters+and+its+Evaluation+by+Automatically+Generated+SAT+Analogy+Questions%2E+ACL+%E2%80%9908%2E) 10. Vera Demberg, 2007. A Language-Independent Unsupervised Model for Morphological Segmentation. ACL ’07. [find](https://scholar.google.com/scholar?q=Vera+Demberg%2C+2007%2E+A+Language%2DIndependent+Unsupervised+Model+for+Morphological+Segmentation%2E+ACL+%E2%80%9907%2E) 11. Beate Dorow, Dominic Widdows, Katarina Ling, Jean-Pierre Eckmann, Danilo Sergi, Elisha Moses, 2005. Using Curvature and Markov Clustering in Graphs for Lexical Acquisition and Word Sense Discrimination. MEANING ’05. [find](https://scholar.google.com/scholar?q=Beate+Dorow%2C+Dominic+Widdows%2C+Katarina+Ling%2C+Jean%2DPierre+Eckmann%2C+Danilo+Sergi%2C+Elisha+Moses%2C+2005%2E+Using+Curvature+and+Markov+Clustering+in+Graphs+for+Lexical+Acquisition+and+Word+Sense+Discriminatio) 12. Samarth Keshava, Emily Pitler, 2006. A Simpler, Intuitive Approach to Morpheme Induction. In Proceedings of 2nd Pascal Challenges Workshop, Venice, Italy. [find](https://scholar.google.com/scholar?q=Samarth+Keshava%2C+Emily+Pitler%2C+2006%2E+A+Simpler%2C+Intuitive+Approach+to+Morpheme+Induction%2E+In+Proceedings+of+2nd+Pascal+Challenges+Workshop%2C+Venice%2C+Italy%2E) 13. Young-Suk Lee, Kishore Papineni, Salim Roukos, Ossama Emam, Hany Hassan, 2003. Language Model Based Arabic Word Segmentation. ACL ’03. [find](https://scholar.google.com/scholar?q=Young%2DSuk+Lee%2C+Kishore+Papineni%2C+Salim+Roukos%2C+Ossama+Emam%2C+Hany+Hassan%2C+2003%2E+Language+Model+Based+Arabic+Word+Segmentation%2E+ACL+%E2%80%9903%2E) 14. Moshe Levinger, Uzzi Ornan, Alon Itai, 1995. Learning Morpho-Lexical Probabilities from an Untagged Corpus with an Application to Hebrew. Comput. Linguistics, 21:383:404. [our card](https://md.research.elamur.ai/free/en/papers/levinger-1995-learning-morpho-lexical-probabilities-untagged/) 15. Dekang Lin, 1998. Automatic Retrieval and Clustering of Similar Words. COLING ’98. [our card](https://md.research.elamur.ai/free/en/papers/lin-1998-automatic-retrieval-clustering-similar-words/) 16. Uzzi Ornan, 2005. Hebrew in Latin Script. Lesonenu, LXIV:137:151 (in Hebrew). [find](https://scholar.google.com/scholar?q=Uzzi+Ornan%2C+2005%2E+Hebrew+in+Latin+Script%2E+Lesonenu%2C+LXIV%3A137%3A151+%28in+Hebrew%29%2E) 17. Patrick Pantel, Dekang Lin, 2002. Discovering Word Senses from Text. SIGKDD ’02. [find](https://scholar.google.com/scholar?q=Patrick+Pantel%2C+Dekang+Lin%2C+2002%2E+Discovering+Word+Senses+from+Text%2E+SIGKDD+%E2%80%9902%2E) 18. Patrick Pantel, Deepak Ravichandran, Eduard Hovy, 2004. Towards Terascale Knowledge Acquisition. COLING ’04. [find](https://scholar.google.com/scholar?q=Patrick+Pantel%2C+Deepak+Ravichandran%2C+Eduard+Hovy%2C+2004%2E+Towards+Terascale+Knowledge+Acquisition%2E+COLING+%E2%80%9904%2E) 19. Fernando Pereira, Naftali Tishby, Lillian Lee, 1993. Distributional Clustering of English Words. ACL ’93. [our card](https://md.research.elamur.ai/free/en/papers/pereira-1993-distributional-clustering-english-words/) 20. Benjamin Snyder, Regina Bazilay, 2008. Unsupervised Multilingual Learning for Morphological Segmentation. ACL/HLT ’08. [our card](https://md.research.elamur.ai/free/en/papers/snyder-2008-unsupervised-multilingual-learning-morphological-segmentation/) 21. Andreas Stolcke, 2002. SRILM – an Extensible Language Modeling Toolkit. ICSLP, pages 901-904, Denver, Colorado. [find](https://scholar.google.com/scholar?q=Andreas+Stolcke%2C+2002%2E+SRILM+%E2%80%93+an+Extensible+Language+Modeling+Toolkit%2E+ICSLP%2C+pages+901%2D904%2C+Denver%2C+Colorado%2E) 22. Dominic Widdows, Beate Dorow, 2002. A Graph Model for Unsupervised Lexical Acquisition. COL-ING ’02. [find](https://scholar.google.com/scholar?q=Dominic+Widdows%2C+Beate+Dorow%2C+2002%2E+A+Graph+Model+for+Unsupervised+Lexical+Acquisition%2E+COL%2DING+%E2%80%9902%2E)