--- title: "A Study of Machine Learning Algorithms for Recognizing Textual" description: "The paper addresses Recognizing Textual Entailment (RTE): deciding whether a hypothesis H follows from a text T. It covers… 11 references." authors: ["Julio Javier Castillo"] year: 2009 venue: "Proceedings of the Student Research Workshop" source: "https://aclanthology.org/R09-2003/" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/castillo-2009-study-machine-learning-algorithms-recognizing/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/castillo-2009-study-machine-learning-algorithms-recognizing/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2009](https://md.research.elamur.ai/free/en/papers/2009/page/2/) / A Study of Machine Learning Algorithms for Recognizing Textual Entailment # A Study of Machine Learning Algorithms for Recognizing Textual Entailment **Authors:** Julio Javier Castillo · **Year:** 2009 · **Venue:** [Proceedings of the Student Research Workshop](https://md.research.elamur.ai/free/en/venues/ranlp/) ([RANLP 2009](https://md.research.elamur.ai/free/en/venues/ranlp/2009/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/R09-2003/) · [PDF](https://aclanthology.org/R09-2003.pdf) · [OpenAlex](https://openalex.org/W2250430763) **Citations:** 3 citations (OpenAlex, 2026-10-14) ## Summary The paper addresses Recognizing Textual Entailment (RTE): deciding whether a hypothesis H follows from a text T. It covers the classic two-way setting and the three-way setting (entailment / contradiction / unknown) used in RTE4. The author fixes a small set of four similarity features and compares five Weka classifiers on equal terms: SVM, AdaBoost, BayesNet, Multilayer Perceptron and Decision Trees. The features are: - character-level Levenshtein distance; - a token-level lexical distance; - a WordNet-based semantic similarity; - a longest common substring score. The classifiers are trained on every combination of the RTE1–RTE3 development sets, and BPI is added for two-way. All models are tested on the RTE4 (TAC 2008) gold set. A named-entity heuristic filter flags pairs that cannot be entailments. A separate SVM then sorts those pairs into contradiction or unknown. The best results come from MLP trained on RTE3 alone: 58.4% for two-way and 55.4% for three-way. Both beat the baseline. Merging more RTE corpora did not help. This questions the common belief that more training data always improves shallow entailment classifiers. ## Key points - The feature vector is deliberately small to limit overfitting. Its four features are stemmed-string Levenshtein, a token-level Levenshtein distance, WordNet similarity and longest common substring. - The lexical and WordNet features align tokens of T and H with the Hungarian algorithm. - Word senses are chosen with Lesk. - Word-to-word similarity is computed from the depth of the least common subsumer. - The NER filter rule: if H contains a named entity of some type that is absent from T, the pair is labelled non-entailment. - It fired on about 10% of RTE4 pairs. - It was correct in 66 of 92 cases (0.71). - On RTE4 test, the best scores were MLP trained on RTE3: 58.4% two-way and 55.4% three-way. - Ten-fold cross-validation gave much higher scores than the test set, which points to overfitting or dataset shift: - 67.85% two-way (RTE3, BayesNet); - 65.5% three-way (RTE3, MLP). - Kappa was low: on average 0.138 two-way and 0.168 three-way, with a best of 0.35. The Levenshtein and lexical-distance features correlate at about 0.86, and dropping either one cost only around 1%. ## Interesting ideas - Hold the feature set fixed and vary only the classifier and the training-corpus combination; this cleanly separates the effect of the learner from the effect of the data. - A cheap, high-precision veto rule says a hypothesis mentioning an entity of some type missing from the text cannot be entailed; it handles a slice of cases before any statistical model runs. - Alignment through optimal bipartite matching (Hungarian algorithm) pairs each H token with its best partner in T, which beats a bag-of-words overlap because every token is matched at most once. - The error analysis shows the filter failing on "fifty years ago" versus "fiftieth birthday"; surface entity matching misses equivalent temporal and numeric expressions. - Comparing cross-validation scores with scores on a separate test set exposed a large gap between them, a reminder that in-distribution scores flatter shallow models. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (learn): Our evaluation runs could freeze inputs and swap one component at a time, such as the parser or the data mix, so that gains can be attributed honestly. - **I2** (answer): Readers answering yes/no questions could first check whether the question names entities absent from the world of the text, and then explain a "not stated" answer by pointing to that missing entity. - **I3** (world): Optimal matching could link entities or frame arguments in a question to those in the text world, giving an explicit, inspectable alignment. - **I4** (world): Normalizing dates and quantities into values in the world layer would avoid such false contradictions and keep the reasoning explainable. - **I5** (learn): Evaluations should always include a held-out corpus from a different source alongside cross-validation, to catch overfitting to one treebank or test collection. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works 🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works ## Cite Cite as: Julio Javier Castillo. 2009. A Study of Machine Learning Algorithms for Recognizing Textual Entailment. Proceedings of the Student Research Workshop. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{castillo2009study, title = {{A Study of Machine Learning Algorithms for Recognizing Textual Entailment}}, author = {Julio Javier Castillo}, booktitle = {Proceedings of the Student Research Workshop}, year = {2009}, url = {https://aclanthology.org/R09-2003/} } ``` - [← previous Unsupervised Concept Discovery In Hebrew Using Simple Unsupervised Word Prefix Segmentation for Hebrew and Arabic](https://md.research.elamur.ai/free/en/papers/dinur-2009-unsupervised-concept-discovery-hebrew-simple/) - [next → Arabic Cross-Document Coreference Resolution](https://md.research.elamur.ai/free/en/papers/sayeed-2009-arabic-cross-document-coreference-resolution/) - [↑ back to papers of 2009](https://md.research.elamur.ai/free/en/papers/2009/page/2/) ## References 11 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Prodromos Malakasiotis and Ion Androutsopoulos. Learning Textual Entailment using SVMs and String Similarity Measures. ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, (ACL 2007), Prague, Czech Republic, 2007. [find](https://scholar.google.com/scholar?q=Prodromos+Malakasiotis+and+Ion+Androutsopoulos%2E+Learning+Textual+Entailment+using+SVMs+and+String+Similarity+Measures%2E+ACL%2DPASCAL+Workshop+on+Textual+Entailment+and+Paraphrasing%2C+%28ACL+2007%29%2C+Prague%2C+C) 2. Julio Javier Castillo, and Laura Alonso i Alemany. An approach using Named Entities for Recognizing Textual Entailment. TAC 2008, Gaithersburg, Maryland, USA, November 2008. [find](https://scholar.google.com/scholar?q=Julio+Javier+Castillo%2C+and+Laura+Alonso+i+Alemany%2E+An+approach+using+Named+Entities+for+Recognizing+Textual+Entailment%2E+TAC+2008%2C+Gaithersburg%2C+Maryland%2C+USA%2C+November+2008%2E) 3. M. Lesk. Automatic sense disambiguation using machine readable dictionaries: How to tell a pine cone from a ice cream cone. In SIGDOC ’86, 1986. [find](https://scholar.google.com/scholar?q=M%2E+Lesk%2E+Automatic+sense+disambiguation+using+machine+readable+dictionaries%3A+How+to+tell+a+pine+cone+from+a+ice+cream+cone%2E+In+SIGDOC+%E2%80%9986%2C+1986%2E) 4. Gusfield, Dan. Algorithms on Strings, Trees and Sequences: Computer Science and Computational Biology. CUP, 1999. [find](https://scholar.google.com/scholar?q=Gusfield%2C+Dan%2E+Algorithms+on+Strings%2C+Trees+and+Sequences%3A+Computer+Science+and+Computational+Biology%2E+CUP%2C+1999%2E) 5. V. Levenshtein. Binary Codes Capable of Correcting Deletions, Insertions and Reversals. Soviet Physics Doklady, 10:707, 1966. [find](https://scholar.google.com/scholar?q=V%2E+Levenshtein%2E+Binary+Codes+Capable+of+Correcting+Deletions%2C+Insertions+and+Reversals%2E+Soviet+Physics+Doklady%2C+10%3A707%2C+1966%2E) 6. Ian H. Witten and Eibe Frank (2005). Data Mining: Practical machine learning tools and techniques, 2nd Edition, Morgan Kaufmann, San Francisco, 2005. [find](https://scholar.google.com/scholar?q=Ian+H%2E+Witten+and+Eibe+Frank+%282005%29%2E+Data+Mining%3A+Practical+machine+learning+tools+and+techniques%2C+2nd+Edition%2C+Morgan+Kaufmann%2C+San+Francisco%2C+2005%2E) 7. Alvaro Rodrigo, Anselmo Peñas, Jesus Herrera, Felisa Verdejo. Experiments of UNED at the Third RTE Challenge. Proceedings of the ACL-PASCAL 2007. [find](https://scholar.google.com/scholar?q=Alvaro+Rodrigo%2C+Anselmo+Pe%C3%B1as%2C+Jesus+Herrera%2C+Felisa+Verdejo%2E+Experiments+of+UNED+at+the+Third+RTE+Challenge%2E+Proceedings+of+the+ACL%2DPASCAL+2007%2E) 8. D. Inkpen, D. Kipp and V. Nastase. Machine Learning Experiments for Textual Entailment. Proceedings of the second RTE Challenge, Venice-Italy, 2006. [find](https://scholar.google.com/scholar?q=D%2E+Inkpen%2C+D%2E+Kipp+and+V%2E+Nastase%2E+Machine+Learning+Experiments+for+Textual+Entailment%2E+Proceedings+of+the+second+RTE+Challenge%2C+Venice%2DItaly%2C+2006%2E) 9. Bill Dolan, Chris Quirk, and Chris Brockett. 2004. [Unsupervised construction of large paraphrase corpora: exploiting massively parallel news sources](https://md.research.elamur.ai/free/en/papers/dolan-2004-unsupervised-construction-large-paraphrase-corpora/). In COLING ’04: Proceedings of the 20th international conference on Computational Linguistics, page 350, Morristown, NJ, USA. Association for Computational Linguistics. 10. F. Zanzotto, Marco Pennacchiotti and Alessandro Moschitti. Shallow Semantics in Fast Textual Entailment Rule Learners. In Proceedings of the Third Recognizing Textual Entailment Challenge, Prague, 2007. [find](https://scholar.google.com/scholar?q=F%2E+Zanzotto%2C+Marco+Pennacchiotti+and+Alessandro+Moschitti%2E+Shallow+Semantics+in+Fast+Textual+Entailment+Rule+Learners%2E+In+Proceedings+of+the+Third+Recognizing+Textual+Entailment+Challenge%2C+Prague%2C+200) 11. Marie-Catherine de Marneffe, et al. Manning.Learning to distinguish valid textual entailments. In Proceedings of the Third Recognizing Textual Entailment Challenge, Italy, 2006. [find](https://scholar.google.com/scholar?q=Marie%2DCatherine+de+Marneffe%2C+et+al%2E+Manning%2ELearning+to+distinguish+valid+textual+entailments%2E+In+Proceedings+of+the+Third+Recognizing+Textual+Entailment+Challenge%2C+Italy%2C+2006%2E) *Updated 30 October 2026*