---
title: "Identification of Parallel Passages Across a Large Hebrew/Aramaic Corpus — Shmidman 2016"
description: "A fast method for finding all approximate parallel passages (20+ words) in a large Hebrew/Aramaic corpus, where… 14 references, cited by 7 papers here."
authors: ["Avi Shmidman", "Moshe Koppel", "Ely Porat"]
year: 2016
venue: "Journal of Data Mining & Digital Humanities, Special Issue on Computer-Aided Processing of Intertextuality in Ancient Languages, Towards a Digital Ecosystem: NLP. Corpus infrastructure. Methods for Retrieving Texts and Computing Text Similarities (March 11, 2018) jdmdh:1388"
source: "https://doi.org/10.46298/jdmdh.1388"
doi: "10.46298/jdmdh.1388"
arxiv: "1602.08715"
licence: "arXiv non-exclusive distribution licence (read only, no redistribution)"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shmidman-2016-identification-parallel-passages-across-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shmidman-2016-identification-parallel-passages-across-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/) / Identification of Parallel Passages Across a Large Hebrew/Aramaic Corpus

# Identification of Parallel Passages Across a Large Hebrew/Aramaic Corpus

**Authors:** [Avi Shmidman](https://md.research.elamur.ai/free/en/authors/avi-shmidman/), [Moshe Koppel](https://md.research.elamur.ai/free/en/authors/moshe-koppel/) and Ely Porat · **Year:** 2016 · **Venue:** Journal of Data Mining and Digital Humanities; arXiv 1602.08715 **Link:** [arxiv.org/abs/1602.08715](https://arxiv.org/abs/1602.08715) · **Licence:** arXiv non-exclusive licence — copyrighted, summary only

[**6** citations](https://openalex.org/works?filter=cites:W2291996448) OpenAlex, 2026-10-14 · [**7** citations in our library](https://md.research.elamur.ai/free/en/papers/shmidman-2016-identification-parallel-passages-across-large/cited-by/) of the 140,884 papers whose bibliographies we read · [Journal of Data Mining & Digital Humanities, Special Issue on Computer-Aided Processing of Intertextuality in Ancient Languages, Towards a Digital Ecosystem: NLP. Corpus infrastructure. Methods for Retrieving Texts and Computing Text Similarities (March 11, 2018) jdmdh:1388](https://openalex.org/S2736708624) · [licence](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

- [Publisher's page](https://doi.org/10.46298/jdmdh.1388)
- [PDF](https://arxiv.org/pdf/1602.08715)
- [DOI 10.46298/jdmdh.1388](https://doi.org/10.46298/jdmdh.1388)
- [arXiv 1602.08715](https://arxiv.org/abs/1602.08715)
- [OpenAlex](https://openalex.org/W2291996448)

**Authors (3)**

1. Avi Shmidman [Semantic Scholar](https://www.semanticscholar.org/author/3361477)
2. Moshe Koppel [Semantic Scholar](https://www.semanticscholar.org/author/145200175)
3. Ely Porat [Semantic Scholar](https://www.semanticscholar.org/author/2167957)

Citations and author metrics: OpenAlex, 2026-10-14; authors with the same name are not always told apart.

## What it is about

A fast method for finding all approximate parallel passages (20+ words) in a large Hebrew/Aramaic corpus, where spelling (plene vs defective, final letters, א/י), particles and synonyms vary. Each word is reduced to its two rarest letters (by corpus letter frequency), keeping their order; this strips most prefixes and matres lectionis, so דילמא and דלמא both become דל. From every five-word window four "skip-grams" of four words are taken (each leaving one word out), packed into collision-free integers and indexed. A parallel is a cluster of at least 3 matching skip-grams, gaps of at most 8 words, spanning at least 20 words, found by binning matches along the diagonal of a dot-plot. On the Babylonian Talmud (1.8 million words) it finds 4,602 parallel pairs in about 11 seconds on one thread; on a test pair of tractates it finds all 41 parallels that exhaustive Levenshtein finds, plus 5 more, with no false positives seen.

## Ideas

- I1. The two-rarest-letters key is a cheap spelling-robust word signature for unvocalised Hebrew: use it to match Torah words against other witnesses (Samaritan Pentateuch, Qumran texts, Mishnah quotations of verses) where plene/defective spelling differs, before any morphological analysis.
- I2. Skip-gram clusters as the parallel finder for quotation detection: find every Torah verse quoted or paraphrased in the Mishnah corpus, giving aligned Biblical–Mishnaic pairs for the comparison chapter.

## For Mova

- I3. The whole method is integer-only and deterministic; a Rust implementation is small and fits Mova's rule of BTreeMap order and two identical runs. Measure recall on known inner-Torah parallels (Exod 20 / Deut 5) against a plain lemma-Jaccard method.

I1 is a helper for [languages/hbo/01-script.md](https://mova.elamur.ai/free/en/docs/languages/hbo/01-script/) (matres lectionis, plene/defective) and for any OCR or witness-alignment work. I2 feeds `11-mishnaic.md` and `12-biblical-mishnaic-modern.md` with aligned quotations. I3 is a tool to write beside `tools/macula`; the paper is read-only, so the method is reimplemented from its description.

## Institutions

[🎓**Bar-Ilan University** Ramat Gan · Israel 102 works · 17,587 citations](https://md.research.elamur.ai/free/en/universities/bar-ilan-university/)

📚 Shelves 5[Computer Science · 7,204 works](https://md.research.elamur.ai/free/en/shelves/computer-science/) · [Artificial Intelligence · 7,042 works](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · [Arts and Humanities · 7,381 works](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · [Language and Linguistics · 7,325 works](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · [Literature and Literary Theory · 223 works](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/literature-and-literary-theory/)

🧭 Topics 4[Semitic languages · 1,336 works](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · [Hebrew & Aramaic · 474 works](https://md.research.elamur.ai/free/en/topics/hebrew-aramaic/) · [Biblical Hebrew · 254 works](https://md.research.elamur.ai/free/en/topics/biblical-hebrew/) · [Aramaic · 60 works](https://md.research.elamur.ai/free/en/topics/aramaic/)

## Cite

Cite as Avi Shmidman, Moshe Koppel and Ely Porat. 2016. Identification of Parallel Passages Across a Large Hebrew/Aramaic Corpus. Journal of Data Mining & Digital Humanities, Special Issue on Computer-Aided Processing of Intertextuality in Ancient Languages, Towards a Digital Ecosystem: NLP. Corpus infrastructure. Methods for Retrieving Texts and Computing Text Similarities (March 11, 2018) jdmdh:1388.

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.46298/jdmdh.1388](https://doi.org/10.46298/jdmdh.1388) · [arXiv 1602.08715](https://arxiv.org/abs/1602.08715)

```bibtex
@article{shmidman2016identification,
  title = {{Identification of Parallel Passages Across a Large Hebrew/Aramaic Corpus}},
  author = {Avi Shmidman, Moshe Koppel and Ely Porat},
  journal = {Journal of Data Mining \& Digital Humanities, Special Issue on Computer-Aided Processing of Intertextuality in Ancient Languages, Towards a Digital Ecosystem: NLP. Corpus infrastructure. Methods for Retrieving Texts and Computing Text Similarities (March 11, 2018) jdmdh:1388},
  year = {2016},
  doi = {10.46298/jdmdh.1388},
  eprint = {1602.08715},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.46298/jdmdh.1388}
}
```

- [← previous ARRAU: Linguistically-Motivated Annotation of Anaphoric Descriptions](https://md.research.elamur.ai/free/en/papers/uryupina-2016-arrau-linguistically-motivated-annotation-anaphoric/)
- [next → Apertium: a free/open source platform for machine translation and basic language technology](https://md.research.elamur.ai/free/en/papers/forcada-2016-apertium-free-open-source-platform/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/)

## References

14 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Basile C., Benedetto D., Caglioti G., and Degli Esposti M. A Plagiarism Detection Procedure in Three Steps: Selection, Matches and Squares, PAN 2009:19-23. [find](https://scholar.google.com/scholar?q=Basile+C%2E%2C+Benedetto+D%2E%2C+Caglioti+G%2E%2C+and+Degli+Esposti+M%2E+A+Plagiarism+Detection+Procedure+in+Three+Steps%3A+Selection%2C+Matches+and+Squares%2C+PAN+2009%3A19%2D23%2E)
2. Barron-Cedeno A., Basile C., Degli Esposti M. and Rosso P. Word length n-grams for text re-use detection, in Proceedings of the 11th international conference on Computational Linguistics and Intelligent Text Processing, ser. CICLing’10. 2010:687-699. [find](https://scholar.google.com/scholar?q=Barron%2DCedeno+A%2E%2C+Basile+C%2E%2C+Degli+Esposti+M%2E+and+Rosso+P%2E+Word+length+n%2Dgrams+for+text+re%2Duse+detection%2C+in+Proceedings+of+the+11th+international+conference+on+Computational+Linguistics+and+Intellige)
3. Büchler M. Informationstechnische Aspekte des Historischen Text Re-use (Engl. Computational Aspects of Historical Text Re-use Detection), Ph.D. dissertation, Universitat Leipzig, 2013. [find](https://scholar.google.com/scholar?q=B%C3%BCchler+M%2E+Informationstechnische+Aspekte+des+Historischen+Text+Re%2Duse+%28Engl%2E+Computational+Aspects+of+Historical+Text+Re%2Duse+Detection%29%2C+Ph%2ED%2E+dissertation%2C+Universitat+Leipzig%2C+2013%2E)
4. Büchler, M., Burns, P. R., Müller, M., Franzini, E., and Franzini, G. Towards a Historical Text Re-use Detection, in: Biemann, C. and Mehler, A. (eds.) Text Mining, Theory and Applications of Natural Language Processing. 2014. [↗](https://doi.org/10.1007/978-3-319-12655-5_11)
5. Skolnik F. (ed.) Encyclopedia Judaica, second edition. Keter Publishing House (Jerusalem), 2007. [find](https://scholar.google.com/scholar?q=Skolnik+F%2E+%28ed%2E%29+Encyclopedia+Judaica%2C+second+edition%2E+Keter+Publishing+House+%28Jerusalem%29%2C+2007%2E)
6. Franzini G., Büchler M., Franzini E., and Moritz M. Scaling Historical Text Re-use. The IEEE International Conference on Big Data 2014 (IEEE BigData 2014), 2014:27-30. [find](https://scholar.google.com/scholar?q=Franzini+G%2E%2C+B%C3%BCchler+M%2E%2C+Franzini+E%2E%2C+and+Moritz+M%2E+Scaling+Historical+Text+Re%2Duse%2E+The+IEEE+International+Conference+on+Big+Data+2014+%28IEEE+BigData+2014%29%2C+2014%3A27%2D30%2E)
7. Grozea C., Gehl C., and Popescu M. ENCOPLOT: Pairwise Sequence Matching in Linear Time Applied to Plagiarism Detection. 3rd PAN Workshop. Uncovering Plagiarism, Authorship and Social Software Misuse. 2009:10. [find](https://scholar.google.com/scholar?q=Grozea+C%2E%2C+Gehl+C%2E%2C+and+Popescu+M%2E+ENCOPLOT%3A+Pairwise+Sequence+Matching+in+Linear+Time+Applied+to+Plagiarism+Detection%2E+3rd+PAN+Workshop%2E+Uncovering+Plagiarism%2C+Authorship+and+Social+Software+Misuse%2E+)
8. Grozea C., and Popescu M. The Encoplot Similarity Measure for Automatic Detection of Plagiarism Notebook for PAN at CLEF 2011. [find](https://scholar.google.com/scholar?q=Grozea+C%2E%2C+and+Popescu+M%2E+The+Encoplot+Similarity+Measure+for+Automatic+Detection+of+Plagiarism+Notebook+for+PAN+at+CLEF+2011%2E)
9. Guthrie D., Allison B., Liu W., Guthrie L. and Wilks Y. A Closer Look at Skip-gram Modelling. Proceedings of the 5th International Conference on Language Resources and Evaluation. 2006:1-4. [find](https://scholar.google.com/scholar?q=Guthrie+D%2E%2C+Allison+B%2E%2C+Liu+W%2E%2C+Guthrie+L%2E+and+Wilks+Y%2E+A+Closer+Look+at+Skip%2Dgram+Modelling%2E+Proceedings+of+the+5th+International+Conference+on+Language+Resources+and+Evaluation%2E+2006%3A1%2D4%2E)
10. Harris Z. Distributional structure. Word. 1954;10(23):146-162. http1 [https://www.dropbox.com/s/e3syo8vhncptruj/FindAllSimilarPassages.exe](https://www.dropbox.com/s/e3syo8vhncptruj/FindAllSimilarPassages.exe) [↗](https://www.dropbox.com/s/e3syo8vhncptruj/FindAllSimilarPassages.exe)
11. Karp R. and Rabin M. O. Efficient randomized pattern-matching algorithms. IBM Journal of Research and Development 1987; 31(2):249-260. [find](https://scholar.google.com/scholar?q=Karp+R%2E+and+Rabin+M%2E+O%2E+Efficient+randomized+pattern%2Dmatching+algorithms%2E+IBM+Journal+of+Research+and+Development+1987%3B+31%282%29%3A249%2D260%2E)
12. Klein E., Dershowitz N., Wolf L., Almogi O., and Wangchuk D. Finding Inexact Quotations Within a Tibetan Buddhist Corpus. Digital Humanities. 2014:486-488. [find](https://scholar.google.com/scholar?q=Klein+E%2E%2C+Dershowitz+N%2E%2C+Wolf+L%2E%2C+Almogi+O%2E%2C+and+Wangchuk+D%2E+Finding+Inexact+Quotations+Within+a+Tibetan+Buddhist+Corpus%2E+Digital+Humanities%2E+2014%3A486%2D488%2E)
13. Lee, J. A Computational Model of Text Reuse in Ancient Literary Texts. Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics. 2007:472-479 [find](https://scholar.google.com/scholar?q=Lee%2C+J%2E+A+Computational+Model+of+Text+Reuse+in+Ancient+Literary+Texts%2E+Proceedings+of+the+45th+Annual+Meeting+of+the+Association+of+Computational+Linguistics%2E+2007%3A472%2D479)
14. Lee L. Measures of distributional similarity. 37th Annual Meeting of the Association for Computational Linguistics. 1999:25-32. [our card](https://md.research.elamur.ai/free/en/papers/lee-1999-measures-distributional-similarity/)

Updated 23 October 2026