--- title: "Language Classification and Segmentation of Noisy Documents" description: "Jewish manuscripts from the Cairo Genizah mix Hebrew, Aramaic, Judeo-Arabic and other languages written in the Hebrew… 6 references." authors: ["Alex Zhicharevich", "Nachum Dershowitz"] year: 2012 venue: "Proceedings of the 6th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities" source: "https://aclanthology.org/W12-1016/" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/zhicharevich-2012-language-classification-segmentation-noisy-documents/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhicharevich-2012-language-classification-segmentation-noisy-documents/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2012](https://md.research.elamur.ai/free/en/papers/2012/page/3/) / Language Classification and Segmentation of Noisy Documents in Hebrew Scripts # Language Classification and Segmentation of Noisy Documents in Hebrew Scripts Paper · 2012 · Alex Zhicharevich and Nachum Dershowitz **Authors:** Alex Zhicharevich; [Nachum Dershowitz](https://md.research.elamur.ai/free/en/authors/nachum-dershowitz/) · **Year:** 2012 · **Venue:** Proceedings of the 6th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities **Link:** [aclanthology.org/W12-1016](https://aclanthology.org/W12-1016/) · library `lingbulk-2012-zhicharevich-language-classification-segmentation-noisy-documents` · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) — summary only [**0** citations OpenAlex, 2026-10-14](https://openalex.org/works?filter=cites:W2139097105) · [SIGHUM Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities](https://md.research.elamur.ai/free/en/venues/sighum-workshop-on-language-technology-for-cultural-heritage-social-sciences-and/) · [SIGHUM Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities 2012](https://md.research.elamur.ai/free/en/venues/sighum-workshop-on-language-technology-for-cultural-heritage-social-sciences-and/2012/) — Proceedings of the 6th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities · published by [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · [licence](https://creativecommons.org/licenses/by-nc-sa/3.0/) - [Publisher's page](https://aclanthology.org/W12-1016/) - [PDF](https://aclanthology.org/W12-1016.pdf) - [OpenAlex](https://openalex.org/W2139097105) **Authors (2)** 1. [Alex Zhicharevich](https://openalex.org/A5047605219) [Semantic Scholar](https://www.semanticscholar.org/author/3459405) · h 1 · 3 citations · 4 works 2. [Nachum Dershowitz](https://openalex.org/A5039993352) [ORCID](https://orcid.org/0000-0003-0363-2735) · [Semantic Scholar](https://www.semanticscholar.org/author/1759551) · h 43 · 8,852 citations · 511 works Citations and author metrics: OpenAlex, 2026-10-14; authors with the same name are not always told apart. ## What it is about Jewish manuscripts from the Cairo Genizah mix Hebrew, Aramaic, Judeo-Arabic and other languages written in the Hebrew script, often switching language several times on one page, and the text they come with is the output of noisy OCR on handwriting. The authors cut a text into arbitrary fixed-size segments and classify each one by the cosine distance between its character bigram distributions; bigrams beat unigrams and usually trigrams. Classification is then refined with neighbouring segments as features, a boundary search after each language switch, and an "unknown character" symbol for OCR gaps. The test documents are synthetic: segments from pairs of languages, with a controlled average segment length k. Hebrew against Aramaic is the hard pair; Hebrew against Judeo-Arabic is easy to classify but hard to segment. Neighbour features help short segments, farther neighbours add nothing, and post-processing improves word accuracy at every segment length. For the noise, simply ignoring unknown bigrams does about as well as estimating them, while error correction and restricting to the most common n-grams do not help. The figures carry the exact numbers; they were not readable in the extracted text, so none are quoted here. ## Ideas - I2. Boundary refinement as an exact search: after each detected switch, try a few split points in the joined two windows and keep the split with the best combined score. It is cheap, and the paper's post-processing result says it helps at every length. ## For Mova - I1. A segmenter for mixed-language text in `tools/langseg`: cut at word boundaries into windows, score each window against bigram profiles of hbo, arc, tmr and jpa, then merge neighbours with a penalty for each language switch. Keep the profiles as count tables built from existing corpora, so the whole thing is deterministic and each decision can say which profile won. Measure on synthetic mixes built as in the paper, with k from about 50 to 250 characters. - I3. An unknown-character symbol for OCR input in `tools/ocrfix`: a marker for an unreadable letter, with every bigram that contains it ignored in the score. This is the simplest variant the paper found as good as the others; start with it. I1 and I2 belong to `tools/langseg`, where the language classes already exist (hbo, arc, tmr, jpa), and their test is a set of synthetic mixed documents shared with `tools/hbo`. I3 belongs to `tools/ocrfix`. The Aramaic side is covered by `languages/arc/` and `languages/tmr/`, which the classes must match. ## Institutions [🎓**Tel Aviv University** Tel Aviv · Israel 67 works · 10,006 citations](https://md.research.elamur.ai/free/en/universities/tel-aviv-university/) ## Fields, topics and concepts 📚 Shelves 6 The field of science this work belongs to (OpenAlex fields).[Computer Science · 9,085 works](https://md.research.elamur.ai/free/en/shelves/computer-science/) · [Artificial Intelligence · 8,888 works](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · [Arts and Humanities · 9,296 works](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · [Language and Linguistics · 9,206 works](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · [Computer Vision and Pattern Recognition · 1,282 works](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · [History · 66 works](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/history/) 🧭 Topics 3 Research subjects and languages the work is about.[Semitic languages · 1,485 works](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · [Hebrew & Aramaic · 482 works](https://md.research.elamur.ai/free/en/topics/hebrew-aramaic/) · [Modern Hebrew · 153 works](https://md.research.elamur.ai/free/en/topics/modern-hebrew/) ## Cite Cite as Alex Zhicharevich and Nachum Dershowitz. 2012. Language Classification and Segmentation of Noisy Documents in Hebrew Scripts. Proceedings of the 6th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities. The paper itself, not this card. BibTeX from the library's record of the work. ```bibtex @inproceedings{zhicharevich2012language, title = {{Language Classification and Segmentation of Noisy Documents in Hebrew Scripts}}, author = {Alex Zhicharevich and Nachum Dershowitz}, booktitle = {Proceedings of the 6th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities}, year = {2012}, url = {https://aclanthology.org/W12-1016/} } ``` - [← previous Labile Verbs and Word Order in Early Middle English: an Initial Study](https://md.research.elamur.ai/free/en/papers/garcia-2019-labile-verbs-word-order-early/) - [next → Reflections on Abu Rumi’s Amharic Translation of the Book of Ruth](https://md.research.elamur.ai/free/en/papers/mulugetta-2012-reflections-abu-rumis-amharic-translation/) - [↑ back to papers of 2012](https://md.research.elamur.ai/free/en/papers/2012/page/3/) ## References 6 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Gökhan Bakır. 2007. Predicting Structured Data. MIT Press, Cambridge, MA. [find](https://scholar.google.com/scholar?q=G%C3%B6khan+Bak%C4%B1r%2E+2007%2E+Predicting+Structured+Data%2E+MIT+Press%2C+Cambridge%2C+MA%2E) 2. Freddy Y. Y. Choi. 2000. [Advances in domain independent linear text segmentation](https://md.research.elamur.ai/free/en/papers/choi-2000-advances-domain-independent-linear-text/). Proc. 1st North American Chapter of the Association for Computational Linguistics Conference, pp. 26-33. [↗](https://doi.org/10.48550/arxiv.cs/0003083) 3. Juha Häkkinen and Jilei Tian. 2001. N-gram and decision tree based language identification for written words. Proc. Automatic Speech Recognition and Understanding (ASRU '01), Italy, pp. 335-338. [find](https://scholar.google.com/scholar?q=Juha+H%C3%A4kkinen+and+Jilei+Tian%2E+2001%2E+N%2Dgram+and+decision+tree+based+language+identification+for+written+words%2E+Proc%2E+Automatic+Speech+Recognition+and+Understanding+%28ASRU+%2701%29%2C+Italy%2C+pp%2E+335%2D338%2E) 4. Marti A. Hearst. 1993. TextTiling: A quantitative approach to discourse segmentation. Technical Report, Sequoia 93/24, Computer Science Division. [find](https://scholar.google.com/scholar?q=Marti+A%2E+Hearst%2E+1993%2E+TextTiling%3A+A+quantitative+approach+to+discourse+segmentation%2E+Technical+Report%2C+Sequoia+93%2F24%2C+Computer+Science+Division%2E) 5. Sylvain Lamprier, Tassadit Amghar, Bernard Levrat, and Frédéric Saubion. 2007. On evaluation methodologies for text segmentation algorithms. Proc. 9th IEEE International Conference on Tools with Artificial Intelligence (ICTAI), volume 2, pp. 19-26. [find](https://scholar.google.com/scholar?q=Sylvain+Lamprier%2C+Tassadit+Amghar%2C+Bernard+Levrat%2C+and+Fr%C3%A9d%C3%A9ric+Saubion%2E+2007%2E+On+evaluation+methodologies+for+text+segmentation+algorithms%2E+Proc%2E+9th+IEEE+International+Conference+on+Tools+with+Artif) 6. Karen Kukich. 1992. Techniques for automatically correcting words in text. ACM Computing Surveys 24(4): 377-439. [↗](https://doi.org/10.1145/146370.146380) Updated 24 October 2026