--- title: "Syntactic Testsuites and Textual Entailment Recognition (2010)" description: "The authors isolate entailments that depend only on syntax and propose an evaluation method built on two parts. 73.2% accuracy. 7 references." authors: ["Paul Bedaride", "Claire Gardent"] year: 2010 venue: "Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)" source: "https://aclanthology.org/L10-1259/" licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)" licence_url: "https://creativecommons.org/licenses/by-nc/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/bedaride-2010-syntactic-testsuites-textual-entailment-recognition/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bedaride-2010-syntactic-testsuites-textual-entailment-recognition/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2010](https://md.research.elamur.ai/free/en/papers/2010/page/2/) / Syntactic Testsuites and Textual Entailment Recognition # Syntactic Testsuites and Textual Entailment Recognition **Authors:** Paul Bedaride and [Claire Gardent](https://md.research.elamur.ai/free/en/authors/claire-gardent/) · **Year:** 2010 · **Venue:** [Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)](https://md.research.elamur.ai/free/en/venues/lrec/) ([LREC 2010](https://md.research.elamur.ai/free/en/venues/lrec/2010/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/) **Links:** [Publisher's page](https://aclanthology.org/L10-1259/) · [PDF](https://aclanthology.org/L10-1259.pdf) · [OpenAlex](https://openalex.org/W1553679662) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary Standard RTE benchmarks mix many phenomena: anaphora, temporal reasoning, world knowledge and more. Because of this, they cannot show how well a system handles any single one. The authors isolate entailments that depend only on syntax and propose an evaluation method built on two parts. The first part is a testsuite generated automatically. Hand-written XML syntactic patterns and a small base lexicon produce sentence variants such as active, passive, relativised, nominalised and coordinated forms. Each sentence carries predicate–argument triples, and the entailment label is computed by comparing these triples. Every sentence also keeps tags recording the constructions used to build it. The second part applies error mining, a technique borrowed from grammar debugging. It ranks the tag pairs that most often co-occur with the system's mistakes. Applied to the Afazio system on 10,000 pairs, the method gives 73.2% accuracy and reveals many missed true entailments. The mining points to noun/verb alternations and coordination as the main weaknesses. The broader value is a diagnostic tool that tells developers what to fix, not just a single score. ## Key points - **Testsuite design:** - 10 sub-suites of 1,000 items each, with true and false entailments balanced in each. - Sentences have up to three clauses. - Predicates cover 10 subcategorisation types, combined differently in each sub-suite. - **Afazio pipeline:** the Stanford parser, then cascaded rewrite modules that normalise paraphrases (voice changes, redistributions, noun/verb variants) into one semantic form, then automated first-order reasoning. - **Overall results:** - 7,317 correct, 2,515 incorrect, and 168 cases where the reasoner gave no answer. - On gold-true pairs, 2,482 were wrong, roughly half. On gold-false pairs, only 33 were wrong. - **Scoring method:** errors are ranked with the suspicion rate of Sagot and de La Clergerie (2006), weighted by the log of the number of error pairs. Failures on true pairs and on false pairs are mined separately. - **Top finding:** a passive relativised verb in H paired with an active coordinated verb in T fails in about 61% of its 65 cases, an expected gain of about 37 pairs if fixed. The rare false-pair failures cluster around relativised and PP arguments, pointing to PP attachment and relative pronoun resolution. ## Interesting ideas - Generate entailment pairs from a pattern grammar plus a lexicon, and derive gold labels by comparing predicate–argument triples, so that labels are computed rather than annotated by hand. - Have every generated item carry tags naming the constructions it contains (voice, relativisation, nominal vs verbal predicate, coordination), so that failures can later be traced to specific phenomena. - Apply grammar-debugging error mining (suspicion rate) to a downstream semantic task, ranking feature pairs from text and hypothesis that co-occur with failures. - Rank suspects by suspicion times the log of the error count, which balances how strongly a feature predicts failure against how much fixing it would gain. - Mine the two error directions (missed entailments vs spurious ones) separately, because system biases make them qualitatively different. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (answer): A similar generator could build controlled "who did what to whom" question sets in which only the syntactic form varies, testing whether readers answer the same way across forms. - **I2** (learn): Evaluation sets could carry such construction tags, letting regressions be explained by phenomenon rather than by a single aggregate number. - **I3** (frames): Mining mismatched frame outputs by the pair of UD constructions involved would show which alternations, such as passive vs active or nominalisation vs verb, the frame builder mishandles. - **I4** (learn): When choosing which self-induced repair rule to write next, this score prioritises problems that are both likely real and worth fixing. - **I5** (world): Checks on world-of-text inference could count missed facts and hallucinated facts separately, since each points to different broken components. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Centre National de la Recherche Scientifique](https://md.research.elamur.ai/free/en/labs/centre-national-de-la-recherche-scientifique/) — Research institute, Paris, France · 104 works · 60,877 citations - [Institut national de recherche en sciences et technologies du numérique](https://md.research.elamur.ai/free/en/labs/institut-national-de-recherche-en-sciences-et-technologies-du-numerique/) — Research institute, Le Chesnay-Rocquencourt, France · 62 works · 17,271 citations - [Laboratoire Lorrain de Recherche en Informatique et ses Applications](https://md.research.elamur.ai/free/en/labs/laboratoire-lorrain-de-recherche-en-informatique-et-ses-applications/) — Research institute, Nancy, France · 23 works · 3,750 citations Also on the byline: [Université Henri Poincaré](https://ror.org/019wrwh65). 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works 🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works ## Cite Cite as: Paul Bedaride and Claire Gardent. 2010. Syntactic Testsuites and Textual Entailment Recognition. Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{bedaride2010syntactic, title = {{Syntactic Testsuites and Textual Entailment Recognition}}, author = {Paul Bedaride and Claire Gardent}, booktitle = {Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)}, year = {2010}, doi = {10.63317/5gzp4dr7c9w3}, url = {https://doi.org/10.63317/5gzp4dr7c9w3} } ``` - [← previous Peters, Pam, Peter Collins & Adam Smith (editors). Comparative Studies in Australian and New Zealand English: Grammar and Beyond. Varieties of English Around the World G39. Amsterdam/Philadelphia: John Benjamins 2009. 406 pp. ISBN 978 90,272 4899 2](https://md.research.elamur.ai/free/en/papers/gustawsson-2010-peters-pam-peter-collins-adam/) - [next → Topicalization and Truth Conditions: A Categorial Account](https://md.research.elamur.ai/free/en/papers/nakamura-2010-topicalization-truth-conditions-categorial-account/) - [↑ back to papers of 2010](https://md.research.elamur.ai/free/en/papers/2010/page/2/) ## References 7 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. P. Bedaride and C. Gardent. 2009a. Normalising semantics : a framework and an experiment. In IWCS 2009 (International Conference on Computational Semantics), Tilburg, The Netherlands. [find](https://scholar.google.com/scholar?q=P%2E+Bedaride+and+C%2E+Gardent%2E+2009a%2E+Normalising+semantics+%3A+a+framework+and+an+experiment%2E+In+IWCS+2009+%28International+Conference+on+Computational+Semantics%29%2C+Tilburg%2C+The+Netherlands%2E) 2. P. Bedaride and C. Gardent. 2009b. Noun/verb entailment. In 4th Language and Technology Conference, Poznan, Poland. [find](https://scholar.google.com/scholar?q=P%2E+Bedaride+and+C%2E+Gardent%2E+2009b%2E+Noun%2Fverb+entailment%2E+In+4th+Language+and+Technology+Conference%2C+Poznan%2C+Poland%2E) 3. J.R. Curran, S. Clark, and J. Bos. 2007. [Linguistically motivated large-scale nlp with c&c and boxer](https://md.research.elamur.ai/free/en/papers/curran-2007-linguistically-motivated-large-scale-nlp/). In Proceedings of the ACL 2007 Demo and Poster Sessions, pages 33–36, Prague, Czech Republic. 4. C. Gardent and E. Kow. 2005. Generating and selecting grammatical paraphrases. ENLG, Aug. [find](https://scholar.google.com/scholar?q=C%2E+Gardent+and+E%2E+Kow%2E+2005%2E+Generating+and+selecting+grammatical+paraphrases%2E+ENLG%2C+Aug%2E) 5. W. Lewis Johnson, P. Rizzo, W. Bosma, S. Kole, and M. Ghijsen. 2004. Generating socially appropriate tutorial dialog. In Workshop on Affective dialog systems. [find](https://scholar.google.com/scholar?q=W%2E+Lewis+Johnson%2C+P%2E+Rizzo%2C+W%2E+Bosma%2C+S%2E+Kole%2C+and+M%2E+Ghijsen%2E+2004%2E+Generating+socially+appropriate+tutorial+dialog%2E+In+Workshop+on+Affective+dialog+systems%2E) 6. D. Klein and C. D. Manning. 2003. [Accurate unlexicalized parsing](https://md.research.elamur.ai/free/en/papers/klein-2003-accurate-unlexicalized-parsing/). In ACL ’03: Proceedings of the 41st Annual Meeting on Association for Computational Linguistics, pages 423–430, Morristown, NJ, USA. Association for Computational Linguistics. 7. B. Sagot and E. de La Clergerie. 2006. Error mining in parsing results. In Proceedings of ACL-CoLing 06, pages 329–336, Sydney, Australie. [find](https://scholar.google.com/scholar?q=B%2E+Sagot+and+E%2E+de+La+Clergerie%2E+2006%2E+Error+mining+in+parsing+results%2E+In+Proceedings+of+ACL%2DCoLing+06%2C+pages+329%E2%80%93336%2C+Sydney%2C+Australie%2E) *Updated 24 October 2026*