---
title: "The Role of Sentence Structure in Recognizing Textual Entailment"
description: "From these elements she builds 11 derived matching features. 65.87% accuracy. 11 references, cited by 4 papers here."
authors: ["Catherine Blake"]
year: 2007
venue: "Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing"
source: "https://aclanthology.org/W07-1417/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/blake-2007-role-sentence-structure-recognizing-textual/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/blake-2007-role-sentence-structure-recognizing-textual/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2007](https://md.research.elamur.ai/free/en/papers/2007/page/2/) / The Role of Sentence Structure in Recognizing Textual Entailment

# The Role of Sentence Structure in Recognizing Textual Entailment

**Authors:** Catherine Blake · **Year:** 2007 · **Venue:** [Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing](https://md.research.elamur.ai/free/en/venues/acl-pascal-workshop-on-textual-entailment-and-paraphrasing/) ([ACL-PASCAL Workshop on Textual Entailment and Paraphrasing 2007](https://md.research.elamur.ai/free/en/venues/acl-pascal-workshop-on-textual-entailment-and-paraphrasing/2007/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W07-1417/) · [PDF](https://aclanthology.org/W07-1417.pdf) · [OpenAlex](https://openalex.org/W2139596890)

**Citations:** 16 citations (OpenAlex, 2026-10-14) · [4](https://md.research.elamur.ai/free/en/papers/blake-2007-role-sentence-structure-recognizing-textual/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks how much syntax alone, without background resources such as WordNet, gazetteers or custom knowledge bases, can help decide whether a text entails a hypothesis in the RTE3 challenge. Blake parses both sentences into a typed dependency representation and extracts four base elements: subject (nsubj, nsubjpass, plus heuristics), object (dobj), verb (any word linked to a subject or object) and collapsed prepositional paths. Subject terms also absorb noun compounds and modifiers. From these elements she builds 11 derived matching features. They range from strict subject-verb-object triple overlap, through variants with swapped roles that roughly capture passive/active alternation, down to subject words appearing anywhere in the text. Decision trees and earlier RTE results guided two final rules: a summed-score threshold and a hand-built disjunction of high-precision features with tuned thresholds. On RTE3 Test the rules reached 60.50% and 65.87% accuracy, which is 9.25–14.62 points over the majority baseline. This supports the view that structure carries real entailment signal that bag-of-words methods miss.

## Key points
- Data: the revised RTE3 development set (800 pairs, 412 entailments, majority baseline 51.50%), the RTE3 Test set (baseline 51.52%) and all of RTE2 for comparison.
- Coverage differed sharply between base features: a subject was found for 789 of 800 dev hypotheses, but an object for only 480.
- Strict features were accurate but rare. On dev, full triple matches were right 74.29% of the time across 35 pairs, and subject+object matches 78.79% across 66 pairs. The loose Subject-Word feature covered 771 pairs at only about 53%.
- Rule 1 sums the match proportions of all features against a threshold of 2.4, scoring 62.38% on dev and 60.50% on test, with average precision 58.97%.
- Rule 2 uses features 1, 2, 3, 5, 10 and 11 with thresholds of 80% and 70%, scoring 71.50% on dev and 65.87% on test, with average precision 60.96%.
- Results varied strongly by pair source. Information extraction stayed near chance (about 50%), while information retrieval reached 70–75.5%. Question answering gained the most from Rule 2 (65.5% → 77%).

## Interesting ideas
- The method uses a ladder of matching strictness, from a full predicate-argument triple down to a subject word appearing anywhere, and trusts each rung according to its measured precision; this gives a principled fallback when a strict structural match is unavailable.
- The "opposite" features align the hypothesis subject with the text object, a cheap approximation of voice alternation that recovers entailments a role-blind matcher would miss.
- The author notes that greedy decision trees overlook features that are highly precise but cover few cases, so she built a disjunctive rule from such features by hand; this is a useful caution whenever rare but reliable signals compete with broad, noisy ones.
- Thresholds are chosen by plotting accuracy against the degree of overlap (for example, the share of subject terms matched) and placing the cut where accuracy visibly rises, rather than by tuning blindly.
- Each evaluation is broken down by data source (IE, IR, QA, SUM), and outputs are ranked by confidence and scored with average precision; this exposes uneven behaviour that a single accuracy number hides.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (answer): A reader could first try to match the full frame of the question against the world, then fall back to argument-only or head-only matches, reporting which rung produced the answer as part of the explanation.
- **I2** (frames): Swapped-role matches can serve as a diagnostic: when they fire but role-normalized frame matching does not, it signals that passive or causative normalization failed.
- **I3** (learn): Self-induced repair rules could be ranked by precision with coverage reported alongside, instead of by impurity reduction, so that rare but nearly certain rules are kept rather than absorbed by broad heuristics.
- **I4** (learn): Plotting accuracy against degree of match for each matching feature would make every threshold in the pipeline explainable and easy to audit against negative controls.
- **I5** (answer): Results on school tests, word problems and fable QA should be reported per genre and as confidence-ranked lists, so that weak task types show up early.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of North Carolina at Chapel Hill](https://md.research.elamur.ai/free/en/universities/university-of-north-carolina-at-chapel-hill/) — University, Chapel Hill, United States · 67 works · 16,474 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works

## Cite

Cite as: Catherine Blake. 2007. The Role of Sentence Structure in Recognizing Textual Entailment. Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{blake2007role,
  title = {{The Role of Sentence Structure in Recognizing Textual Entailment}},
  author = {Catherine Blake},
  booktitle = {Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing},
  year = {2007},
  doi = {10.3115/1654536.1654557},
  url = {https://doi.org/10.3115/1654536.1654557}
}
```

- [← previous HIT: Web based Scoring Method for English Lexical Substitution](https://md.research.elamur.ai/free/en/papers/zhao-2007-hit-web-based-scoring-method/)
- [next → Semi-automatic error analysis for large-scale statistical machine translation](https://md.research.elamur.ai/free/en/papers/kirchhoff-2007-semi-automatic-error-analysis-large/)
- [↑ back to papers of 2007](https://md.research.elamur.ai/free/en/papers/2007/page/2/)

## References

11 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bar-Haim, R., Dagan, I., Dolan, B., Ferro, L., Giampiccolo, D., Magnini, B., et al. (2006). The Second PASCAL Recognising Textual Entailment Challenge. Venice, Italy. [find](https://scholar.google.com/scholar?q=Bar%2DHaim%2C+R%2E%2C+Dagan%2C+I%2E%2C+Dolan%2C+B%2E%2C+Ferro%2C+L%2E%2C+Giampiccolo%2C+D%2E%2C+Magnini%2C+B%2E%2C+et+al%2E+%282006%29%2E+The+Second+PASCAL+Recognising+Textual+Entailment+Challenge%2E+Venice%2C+Italy%2E)
2. Dagan, I., Glickman, O., & Magnini, B. (2005). The PASCAL Recognising Textual Entailment Challenge. Southampton, U.K. [find](https://scholar.google.com/scholar?q=Dagan%2C+I%2E%2C+Glickman%2C+O%2E%2C+%26+Magnini%2C+B%2E+%282005%29%2E+The+PASCAL+Recognising+Textual+Entailment+Challenge%2E+Southampton%2C+U%2EK%2E)
3. de Marneffe, M.-C., MacCartney, B., Grenager, T., Cer, D., Rafferty, A., & Manning, C. D. (2006). Learning to distinguish valid textual entailments, In The Second PASCAL Challenges Workshop on Recognising Textual Entailment, Venice, Italy. [find](https://scholar.google.com/scholar?q=de+Marneffe%2C+M%2E%2DC%2E%2C+MacCartney%2C+B%2E%2C+Grenager%2C+T%2E%2C+Cer%2C+D%2E%2C+Rafferty%2C+A%2E%2C+%26+Manning%2C+C%2E+D%2E+%282006%29%2E+Learning+to+distinguish+valid+textual+entailments%2C+In+The+Second+PASCAL+Challenges+Workshop+on+Recogni)
4. Hickl, A., Williams, J., Bensley, J., Roberts, K., Rink, B., & Shi, Y. (2006). Recognizing Textual Entailment with LCC’s GROUNDHOG System In The Second PASCAL Recognising Textual Entailment Challenge, Venice, Italy. [find](https://scholar.google.com/scholar?q=Hickl%2C+A%2E%2C+Williams%2C+J%2E%2C+Bensley%2C+J%2E%2C+Roberts%2C+K%2E%2C+Rink%2C+B%2E%2C+%26+Shi%2C+Y%2E+%282006%29%2E+Recognizing+Textual+Entailment+with+LCC%E2%80%99s+GROUNDHOG+System+In+The+Second+PASCAL+Recognising+Textual+Entailment+Challenge%2C)
5. Klein, D., & Manning, C. D. (2002). Fast Exact Inference with a Factored Model for Natural Language Parsing. Paper presented at the Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Klein%2C+D%2E%2C+%26+Manning%2C+C%2E+D%2E+%282002%29%2E+Fast+Exact+Inference+with+a+Factored+Model+for+Natural+Language+Parsing%2E+Paper+presented+at+the+Advances+in+Neural+Information+Processing+Systems%2E)
6. Lin, D. (1998). Dependency-based Evaluation of MINIPAR. In Workshop on the Evaluation of Parsing Systems, First International Conference on Language Resources and Evaluation, Granada,Spain. [find](https://scholar.google.com/scholar?q=Lin%2C+D%2E+%281998%29%2E+Dependency%2Dbased+Evaluation+of+MINIPAR%2E+In+Workshop+on+the+Evaluation+of+Parsing+Systems%2C+First+International+Conference+on+Language+Resources+and+Evaluation%2C+Granada%2CSpain%2E)
7. Lin, D., & Pantel, P. (2001). Induction of semantic classes from natural language text. In The 7 th International conference on Knowledge discovery and Data Mining, San Francisco, CA. [find](https://scholar.google.com/scholar?q=Lin%2C+D%2E%2C+%26+Pantel%2C+P%2E+%282001%29%2E+Induction+of+semantic+classes+from+natural+language+text%2E+In+The+7+th+International+conference+on+Knowledge+discovery+and+Data+Mining%2C+San+Francisco%2C+CA%2E)
8. Litkowski, K. (2006). Componential Analysis for Recognizing Textual Entailment. In The Second PASCAL Challenges Workshop on Recognising Textual Entailment, Venice, Italy. [find](https://scholar.google.com/scholar?q=Litkowski%2C+K%2E+%282006%29%2E+Componential+Analysis+for+Recognizing+Textual+Entailment%2E+In+The+Second+PASCAL+Challenges+Workshop+on+Recognising+Textual+Entailment%2C+Venice%2C+Italy%2E)
9. Miller, G. (1995). [WordNet: A Lexical Database for English](https://md.research.elamur.ai/free/en/papers/miller-1994-wordnet/). Communications of the ACM, 38(11), 39-41.
10. National Library of Medicine. (2000). The SPECIALIST Lexicon, from www.nlm.nih.gov/pubs/factsheets/umlslex.html Vanderwende, L., Coughlin, D., & Dolan, B. (2005). What Syntax can Contribute in Entailment Task. The PASCAL Challenges Workshop on Recognising Textual Entailment. Southhampton, UK. [find](https://scholar.google.com/scholar?q=National+Library+of+Medicine%2E+%282000%29%2E+The+SPECIALIST+Lexicon%2C+from+www%2Enlm%2Enih%2Egov%2Fpubs%2Ffactsheets%2Fumlslex%2Ehtml+Vanderwende%2C+L%2E%2C+Coughlin%2C+D%2E%2C+%26+Dolan%2C+B%2E+%282005%29%2E+What+Syntax+can+Contribute+in+Entailm)
11. Voorhees, E. M., & Harman., D. (1999). Overview of the seventh text retrieval conference. In The Seventh Text REtrieval Conference (TREC-7). [find](https://scholar.google.com/scholar?q=Voorhees%2C+E%2E+M%2E%2C+%26+Harman%2E%2C+D%2E+%281999%29%2E+Overview+of+the+seventh+text+retrieval+conference%2E+In+The+Seventh+Text+REtrieval+Conference+%28TREC%2D7%29%2E)

*Updated 30 October 2026*