--- title: "An Open-Source Package for Recognizing Textual Entailment" description: "The paper presents EDITS (Edit Distance Textual Entailment Suite), an open-source… 60.17% accuracy. 7 references, cited by 19 papers here." authors: ["Milen Kouylekov", "Matteo Negri"] year: 2010 venue: "Proceedings of the ACL 2010 System Demonstrations" source: "https://aclanthology.org/P10-4008/" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/kouylekov-2010-open-source-package-recognizing-textual/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kouylekov-2010-open-source-package-recognizing-textual/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2010](https://md.research.elamur.ai/free/en/papers/2010/) / An Open-Source Package for Recognizing Textual Entailment # An Open-Source Package for Recognizing Textual Entailment **Authors:** [Milen Kouylekov](https://md.research.elamur.ai/free/en/authors/milen-kouylekov/) and [Matteo Negri](https://md.research.elamur.ai/free/en/authors/matteo-negri/) · **Year:** 2010 · **Venue:** [Proceedings of the ACL 2010 System Demonstrations](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2010](https://md.research.elamur.ai/free/en/venues/acl/2010/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/P10-4008/) · [PDF](https://aclanthology.org/P10-4008.pdf) · [OpenAlex](https://openalex.org/W2144187685) **Citations:** 51 citations (OpenAlex, 2026-10-14) · [19](https://md.research.elamur.ai/free/en/papers/kouylekov-2010-open-source-package-recognizing-textual/cited-by/) in our library, of the 141,840 papers whose bibliographies we read ## Summary The paper presents EDITS (Edit Distance Textual Entailment Suite), an open-source Java toolkit for Recognizing Textual Entailment (RTE). RTE asks whether a Hypothesis (H) follows from a Text (T). The authors argue that RTE research lacks freely available systems, which makes experiments and prototyping slow and discourages newcomers. EDITS treats entailment as a matter of distance: the closer T and H are, the more likely entailment holds. Each engine computes a normalized score between 0 and 1. The score comes from an edit-distance or similarity algorithm paired with a user-defined XML cost scheme, plus optional entailment and contradiction rules. During training, the system learns a threshold that separates positive from negative pairs. Costs can be tuned with a genetic-algorithm optimizer, and several engines can be merged either linearly or through a Weka classifier. Earlier versions scored about average on RTE-5, ranked first at EVALITA 2009 for Italian, and served as the core of a question-answering system. The release matters because it offers a shared, reproducible baseline environment for the field. ## Key points - Algorithms: Token Edit Distance (Levenshtein over tokens) and Tree Edit Distance (Zhang and Shasha, 1990). There are also five similarity measures turned into distances: Word Overlap, Jaro-Winkler, Cosine Similarity, Longest Common Subsequence and Jaccard Coefficient. - Cost schemes: - Costs are XML files that use a lisp-like condition language. - The Simple scheme charges 10 for insertion, 10 for deletion, 0 for an identical substitution and 20 for a different one. - The IDF scheme uses IDF values for the 100K most frequent English words, computed on the Brown Corpus. - Input and models: - Input uses the ETAF XML format, at either the token level or the tree level. - Plug-ins exist for TreeTagger, Stanford Parser and OpenNLP. - RTE 1-3 and EVALITA 2009 corpora come pre-annotated with the package. - A trained model is a zip file holding the threshold, configuration, cost scheme and rules. - Results: - On the RTE-5 2-way task, EDITS reached 60.17% accuracy, against a participant average of 60.36%. A newer model reaches 61.83%. - On the RTE-5 Search task it reached 33.44% F1, ranking 3rd of 8. - At EVALITA 2009 it ranked first with 71.0% accuracy. - In the QALL-ME project, EDITS was used for entailment-based relation extraction over 14 CINEMA relations. It reached 72.9% F1, which let the system answer 83% of 400 test questions correctly. ## Interesting ideas - At test time EDITS outputs a decision and a confidence score, plus the exact sequence of edit operations behind the score, so every verdict can be inspected. - Edit costs can be asymmetric: when texts are much longer than hypotheses, cheap deletion stops short hypotheses that are fully contained in T from being penalized. - Tree Edit Distance runs over dependency nodes, with costs defined by conditions on the two nodes being compared, so the distance reflects syntactic structure rather than a bag of words. - Relation extraction is recast as entailment: the input is tested against a small set of textual realizations for each target relation. - A trained model is packaged together with everything that produced it: threshold, configuration, cost scheme and rules. This makes experiments easy to share and reproduce. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (answer): When a reader checks whether a candidate answer follows from a fable, it could log the alignment steps (kept, dropped and substituted nodes) as the human-readable justification. - **I2** (answer): Matching a short question frame against a long story could make unmatched story material nearly free, while making content missing from the question expensive. - **I3** (learn): A cost-weighted tree edit distance between the outputs of witness parsers could give a graded disagreement score for filtering silver data, rather than a plain match-or-mismatch. - **I4** (frames): Each frame could carry a few canonical template sentences, and a sentence would be linked to the frame whose templates it entails with the lowest structural cost. - **I5** (learn): Bundling each trained parser or repair-rule set with its config and evaluation controls would make regressions traceable and runs replicable. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Fondazione Bruno Kessler](https://md.research.elamur.ai/free/en/labs/fondazione-bruno-kessler/) — Research institute, Trento, Italy · 25 works · 43,498 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works 🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works ## Cite Cite as: Milen Kouylekov and Matteo Negri. 2010. An Open-Source Package for Recognizing Textual Entailment. Proceedings of the ACL 2010 System Demonstrations. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{kouylekov2010open, title = {{An Open-Source Package for Recognizing Textual Entailment}}, author = {Milen Kouylekov and Matteo Negri}, booktitle = {Proceedings of the ACL 2010 System Demonstrations}, year = {2010}, url = {https://aclanthology.org/P10-4008/} } ``` - [← previous Generating Entailment Rules from FrameNet](https://md.research.elamur.ai/free/en/papers/aharon-2010-generating-entailment-rules-framenet/) - [next → Multiword Expressions in the wild? The mwetoolkit comes in handy](https://md.research.elamur.ai/free/en/papers/ramisch-2010-multiword-expressions-wild-mwetoolkit-comes/) - [↑ back to papers of 2010](https://md.research.elamur.ai/free/en/papers/2010/) ## References 7 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Prodromos Malakasiotis and Ion Androutsopoulos 2007. Learning Textual Entailment using SVMs and String Similarity Measures. Proc. of the ACL ’07 Workshop on Textual Entailment and Paraphrasing. [find](https://scholar.google.com/scholar?q=Prodromos+Malakasiotis+and+Ion+Androutsopoulos+2007%2E+Learning+Textual+Entailment+using+SVMs+and+String+Similarity+Measures%2E+Proc%2E+of+the+ACL+%E2%80%9907+Workshop+on+Textual+Entailment+and+Paraphrasing%2E) 2. Ido Dagan and 0ren Glickman 2004. Probabilistic Textual Entailment: Generic Applied Modeling of Language Variability. Proc. of the PASCAL Workshop on Learning Methods for Text Understanding and Mining. [find](https://scholar.google.com/scholar?q=Ido+Dagan+and+0ren+Glickman+2004%2E+Probabilistic+Textual+Entailment%3A+Generic+Applied+Modeling+of+Language+Variability%2E+Proc%2E+of+the+PASCAL+Workshop+on+Learning+Methods+for+Text+Understanding+and+Mining) 3. Kaizhong Zhang and Dennis Shasha 1990. Fast Algorithm for the Unit Cost Editing Distance Between Trees. Journal of Algorithms. vol.11. [find](https://scholar.google.com/scholar?q=Kaizhong+Zhang+and+Dennis+Shasha+1990%2E+Fast+Algorithm+for+the+Unit+Cost+Editing+Distance+Between+Trees%2E+Journal+of+Algorithms%2E+vol%2E11%2E) 4. Yashar Mehdad 2009. Automatic Cost Estimation for Tree Edit Distance Using Particle Swarm Optimization. Proc. of ACL-IJCNLP 2009. [find](https://scholar.google.com/scholar?q=Yashar+Mehdad+2009%2E+Automatic+Cost+Estimation+for+Tree+Edit+Distance+Using+Particle+Swarm+Optimization%2E+Proc%2E+of+ACL%2DIJCNLP+2009%2E) 5. Matteo Negri and Milen Kouylekov 2009. Question Answering over Structured Data: an Entailment-Based Approach to Question Analysis. Proc. of RANLP-2009. [find](https://scholar.google.com/scholar?q=Matteo+Negri+and+Milen+Kouylekov+2009%2E+Question+Answering+over+Structured+Data%3A+an+Entailment%2DBased+Approach+to+Question+Analysis%2E+Proc%2E+of+RANLP%2D2009%2E) 6. Elena Cabrio, Yashar Mehdad, Matteo Negri, Milen Kouylekov, and Bernardo Magnini 2009. Recognizing Textual Entailment for Italian EDITS @ EVALITA 2009 Proc. of EVALITA 2009. [find](https://scholar.google.com/scholar?q=Elena+Cabrio%2C+Yashar+Mehdad%2C+Matteo+Negri%2C+Milen+Kouylekov%2C+and+Bernardo+Magnini+2009%2E+Recognizing+Textual+Entailment+for+Italian+EDITS+%40+EVALITA+2009+Proc%2E+of+EVALITA+2009%2E) 7. Yashar Mehdad, Matteo Negri, Elena Cabrio, Milen Kouylekov, and Bernardo Magnini 2009. Recognizing Textual Entailment for English EDITS @ TAC 2009 To appear in Proceedings of TAC 2009. [find](https://scholar.google.com/scholar?q=Yashar+Mehdad%2C+Matteo+Negri%2C+Elena+Cabrio%2C+Milen+Kouylekov%2C+and+Bernardo+Magnini+2009%2E+Recognizing+Textual+Entailment+for+English+EDITS+%40+TAC+2009+To+appear+in+Proceedings+of+TAC+2009%2E) *Updated 24 October 2026*