---
title: "Is it Worth Submitting this Run? Assess your RTE System with a"
description: "Building systems for Recognizing Textual Entailment (RTE) is hampered by two things. 13 references, cited by 3 papers here."
authors: ["Milen Kouylekov", "Yashar Mehdad", "Matteo Negri"]
year: 2011
venue: "Proceedings of the TextInfer 2011 Workshop on Textual Entailment"
source: "https://aclanthology.org/W11-2404/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kouylekov-2011-is-it-worth-submitting-this/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kouylekov-2011-is-it-worth-submitting-this/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2011](https://md.research.elamur.ai/free/en/papers/2011/page/2/) / Is it Worth Submitting this Run? Assess your RTE System with a Good Sparring Partner

# Is it Worth Submitting this Run? Assess your RTE System with a Good Sparring Partner

**Authors:** [Milen Kouylekov](https://md.research.elamur.ai/free/en/authors/milen-kouylekov/), [Yashar Mehdad](https://md.research.elamur.ai/free/en/authors/yashar-mehdad/) and [Matteo Negri](https://md.research.elamur.ai/free/en/authors/matteo-negri/) · **Year:** 2011 · **Venue:** Proceedings of the TextInfer 2011 Workshop on Textual Entailment · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W11-2404/) · [PDF](https://aclanthology.org/W11-2404.pdf) · [OpenAlex](https://openalex.org/W93043330)

**Citations:** 5 citations (OpenAlex, 2026-10-14) · 3 in our library, of the 141,840 papers whose bibliographies we read

## Summary
Building systems for Recognizing Textual Entailment (RTE) is hampered by two things. The evaluation conditions change from one RTE Challenge to the next, so experience does not carry over well. Developers also have no easy way to judge how good their training-set results really are before submitting. The authors extend the open-source EDITS package with EDITS-GA. It treats system configuration as a search problem and uses a genetic algorithm (via JGAP) to explore combinations of components: algorithms, cost schemes, rules and optimizers. Each candidate is trained and scored on the development data with a user-chosen fitness measure. They ran it on the "Main" task data of RTE1–RTE6. The automatically found configurations beat EDITS's own hand-picked official submissions by up to 4.51% and always scored above the participants' average. Each search took minutes for most datasets and under two hours for the largest. The result is a cheap, fast baseline, or "sparring partner", that developers can use to place their own system's numbers in context.

## Key points
- **Search space:** configurations are encoded as 18 boolean switches, giving 2^18 = 262,144 combinations.
  - Exhaustive training was estimated at about 4 days even on the small datasets.
  - The genetic algorithm evaluates around 400 configurations per dataset, with a maximum of 20 iterations.
- **Fitness measure:** accuracy for RTE1–RTE5 and F-measure for RTE6, following the official metric of each challenge.
- **Preprocessing:**
  - Lucene's tokenizer and stemmer are applied automatically to raw data.
  - The Stanford parser plugin supplies trees for the syntax-based algorithms, such as tree edit distance.
- **EDITS-GA results (rank among participants):**

  | Dataset | Score | Rank |
  |---|---|---|
  | RTE1 | 0.5787 | 3rd |
  | RTE2 | 0.6225 | 5th |
  | RTE3 | 0.6875 | 4th |
  | RTE4 | 0.595 | 10th |
  | RTE5 | 0.6233 | 9th |
  | RTE6 | 0.4673 | 3rd |

  - Relative gains over the official EDITS runs were +2.89% to +4.51%; RTE3 has no official EDITS entry.
- **What changed on RTE6:** the hand-picked run used lexical overlap with a Levenshtein-based cost for unmatched hypothesis words. The search instead chose an IDF-based cost scheme and turned on stop-word filtering, which gave the 4.51% gain.

## Interesting ideas
- Represent every optional component and parameter of a pipeline as an on/off gene, then let a genetic search over training-set scores pick the combination; this beats both exhaustive search and expert guessing when the space is large.
- An automatically tuned, freely available system serves as a "sparring partner": a quick reference score telling you whether your own sophisticated approach is actually better than a well-configured simple one.
- Small, seemingly minor configuration choices (cost scheme, stop-word filtering) caused gaps of several points, so hand-chosen "reasonable" settings should not be trusted without checking.
- The fitness function is left to the user and should follow the target evaluation metric (accuracy vs. F-measure), so the search optimizes what will actually be judged.
- Making the search cheap (lightweight default preprocessing, bounded iterations) keeps tuning feasible under deadlines.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Mova's many optional repair rules, witness-parser votes and feature toggles could be encoded the same way and tuned automatically against a dev set instead of by intuition.
- **I2** (learn): every new reader or rule set could be compared against an auto-tuned shallow baseline on the same data, as a sanity check and a kind of negative control.
- **I3** (frames): role-assignment and coreference heuristics with thresholds or cost choices should be swept systematically, since tiny switches may move accuracy noticeably.
- **I4** (answer): when tuning the school-test or fable-QA readers, the search objective should match each test's real scoring, not a generic proxy.
- **I5** (tree): parser hyperparameter or dialect-conversion option searches could use a fast tokenize-and-lemmatize path and a fixed generation budget so that tuning runs finish in minutes.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Trento](https://md.research.elamur.ai/free/en/universities/university-of-trento/) — University, Trento, Italy · 62 works · 14,868 citations
- [FBK Center for Information and Communication Technology](https://md.research.elamur.ai/free/en/labs/fbk-center-for-information-and-communication-technology/) — Research institute, Trento, Italy · 2 works · 110 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 523 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works

## Cite

Cite as: Milen Kouylekov, Yashar Mehdad and Matteo Negri. 2011. Is it Worth Submitting this Run? Assess your RTE System with a Good Sparring Partner. Proceedings of the TextInfer 2011 Workshop on Textual Entailment.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{kouylekov2011is,
  title = {{Is it Worth Submitting this Run? Assess your RTE System with a Good Sparring Partner}},
  author = {Milen Kouylekov, Yashar Mehdad and Matteo Negri},
  booktitle = {Proceedings of the TextInfer 2011 Workshop on Textual Entailment},
  year = {2011},
  url = {https://aclanthology.org/W11-2404/}
}
```

- [← previous Design of Arabic Diacritical Marks](https://md.research.elamur.ai/free/en/papers/hssini-2011-design-arabic-diacritical-marks/)
- [next → Measuring Language Development in Early Childhood Education: A Case Study of Grammar Checking in Child Language Transcripts](https://md.research.elamur.ai/free/en/papers/hassanali-2011-measuring-language-development-early-childhood/)
- [↑ back to papers of 2011](https://md.research.elamur.ai/free/en/papers/2011/page/2/)

## References

13 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Margaret Aycinena, Mykel J. Kochenderfer, and David Carl Mulford. 2003. An Evolutionary Approach to Natural Language Grammar Induction. Stanford CS 224N Natural Language Processing. [find](https://scholar.google.com/scholar?q=Margaret+Aycinena%2C+Mykel+J%2E+Kochenderfer%2C+and+David+Carl+Mulford%2E+2003%2E+An+Evolutionary+Approach+to+Natural+Language+Grammar+Induction%2E+Stanford+CS+224N+Natural+Language+Processing%2E)
2. Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, Bernardo Magnini. 2009. The Fifth PASCAL Recognizing Textual Entailment Challenge. Proceedings of the TAC 2009 Workshop. [find](https://scholar.google.com/scholar?q=Luisa+Bentivogli%2C+Ido+Dagan%2C+Hoa+Trang+Dang%2C+Danilo+Giampiccolo%2C+Bernardo+Magnini%2E+2009%2E+The+Fifth+PASCAL+Recognizing+Textual+Entailment+Challenge%2E+Proceedings+of+the+TAC+2009+Workshop%2E)
3. Ido Dagan and Oren Glickman. 2004. Probabilistic textual entailment: Generic applied modeling of language variability. Proceedings of the PASCAL Workshop of Learning Methods for Text Understanding and Mining. [find](https://scholar.google.com/scholar?q=Ido+Dagan+and+Oren+Glickman%2E+2004%2E+Probabilistic+textual+entailment%3A+Generic+applied+modeling+of+language+variability%2E+Proceedings+of+the+PASCAL+Workshop+of+Learning+Methods+for+Text+Understanding+and)
4. Alejandro G. Figueroa and Günter Neumann. 2008. Genetic Algorithms for Data-driven Web Question Answering. Evolutionary Computation 16(1) (2008) pp. 89-125. [↗](https://doi.org/10.1162/evco.2008.16.1.89)
5. Andrew Hickl, John Williams, Jeremy Bensley, Kirk Roberts, Bryan Rink, and Ying Shi. 2006. Recognizing Textual Entailment with LCCs Groundhog System. Proceedings of the Second PASCAL Challenges Workshop. [find](https://scholar.google.com/scholar?q=Andrew+Hickl%2C+John+Williams%2C+Jeremy+Bensley%2C+Kirk+Roberts%2C+Bryan+Rink%2C+and+Ying+Shi%2E+2006%2E+Recognizing+Textual+Entailment+with+LCCs+Groundhog+System%2E+Proceedings+of+the+Second+PASCAL+Challenges+Worksh)
6. Andrew Hickl and Jeremy Bensley. 2007. A discourse commitment-based framework for recognizing textual entailment. Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing. [find](https://scholar.google.com/scholar?q=Andrew+Hickl+and+Jeremy+Bensley%2E+2007%2E+A+discourse+commitment%2Dbased+framework+for+recognizing+textual+entailment%2E+Proceedings+of+the+ACL%2DPASCAL+Workshop+on+Textual+Entailment+and+Paraphrasing%2E)
7. Houping Jia, Xiaojiang Huang, Tengfei Ma, Xiaojun Wan, and Jianguo Xiao. 2010. PKUTM Participation at TAC 2010 RTE and Summarization Track. Proceedings of the Sixth Recognizing Textual Entailment Challenge. [find](https://scholar.google.com/scholar?q=Houping+Jia%2C+Xiaojiang+Huang%2C+Tengfei+Ma%2C+Xiaojun+Wan%2C+and+Jianguo+Xiao%2E+2010%2E+PKUTM+Participation+at+TAC+2010+RTE+and+Summarization+Track%2E+Proceedings+of+the+Sixth+Recognizing+Textual+Entailment+Chal)
8. Milen Kouylekov and Matteo Negri. 2010. [An Open-source Package for Recognizing Textual Entailment](https://md.research.elamur.ai/free/en/papers/kouylekov-2010-open-source-package-recognizing-textual/). Proceedings of ACL 2010 Demo session.
9. Milen Kouylekov, Yashar Mehdad, Matteo Negri, and Elena Cabrio. 2010. FBK Participation in RTE6: Main and KBP Validation Task. Proceedings of the Sixth Recognizing Textual Entailment Challenge. [find](https://scholar.google.com/scholar?q=Milen+Kouylekov%2C+Yashar+Mehdad%2C+Matteo+Negri%2C+and+Elena+Cabrio%2E+2010%2E+FBK+Participation+in+RTE6%3A+Main+and+KBP+Validation+Task%2E+Proceedings+of+the+Sixth+Recognizing+Textual+Entailment+Challenge%2E)
10. Yashar Mehdad 2009. Automatic Cost Estimation for Tree Edit Distance Using Particle Swarm Optimization. Proceedings of ACL-IJCNLP 2009. [find](https://scholar.google.com/scholar?q=Yashar+Mehdad+2009%2E+Automatic+Cost+Estimation+for+Tree+Edit+Distance+Using+Particle+Swarm+Optimization%2E+Proceedings+of+ACL%2DIJCNLP+2009%2E)
11. Eridan Otto and Marı́a Cristina Riff 2004. Towards an efficient evolutionary decoding algorithm for statistical machine translation. LNAI, 2972:438447.. [find](https://scholar.google.com/scholar?q=Eridan+Otto+and+Mar%C4%B1%CC%81a+Cristina+Riff+2004%2E+Towards+an+efficient+evolutionary+decoding+algorithm+for+statistical+machine+translation%2E+LNAI%2C+2972%3A438447%2E%2E)
12. Marta Tatu and Dan Moldovan. 2007. COGEX at RTE3. Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing. [find](https://scholar.google.com/scholar?q=Marta+Tatu+and+Dan+Moldovan%2E+2007%2E+COGEX+at+RTE3%2E+Proceedings+of+the+ACL%2DPASCAL+Workshop+on+Textual+Entailment+and+Paraphrasing%2E)
13. Fabio Massimo Zanzotto, Marco Pennacchiotti and Alessandro Moschitti. 2007. Shallow Semantics in Fast Textual Entailment Rule Learners. Proceedings of the Third Recognizing Textual Entailment Challenge. [find](https://scholar.google.com/scholar?q=Fabio+Massimo+Zanzotto%2C+Marco+Pennacchiotti+and+Alessandro+Moschitti%2E+2007%2E+Shallow+Semantics+in+Fast+Textual+Entailment+Rule+Learners%2E+Proceedings+of+the+Third+Recognizing+Textual+Entailment+Challeng)

*Updated 24 October 2026*