---
title: "Semantic Annotation of Textual Entailment — Toledo 2013"
description: "The authors propose a model-theoretic framework for this. 76% precision. 16 references, cited by 1 paper here."
authors: ["Assaf Toledo", "Stavroula Alexandropoulou", "Sophia Katrenko", "Heidi Klockmann", "Pepijn Kokke", "Yoad Winter"]
year: 2013
venue: "Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013) – Long Papers"
source: "https://aclanthology.org/W13-0121/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/toledo-2013-semantic-annotation-textual-entailment/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/toledo-2013-semantic-annotation-textual-entailment/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2013](https://md.research.elamur.ai/free/en/papers/2013/page/2/) / Semantic Annotation of Textual Entailment

# Semantic Annotation of Textual Entailment

**Authors:** [Assaf Toledo](https://md.research.elamur.ai/free/en/authors/assaf-toledo/), [Stavroula Alexandropoulou](https://md.research.elamur.ai/free/en/authors/stavroula-alexandropoulou/), [Sophia Katrenko](https://md.research.elamur.ai/free/en/authors/sophia-katrenko/), Heidi Klockmann, Pepijn Kokke and [Yoad Winter](https://md.research.elamur.ai/free/en/authors/yoad-winter/) · **Year:** 2013 · **Venue:** [Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013) – Long Papers](https://md.research.elamur.ai/free/en/venues/international-conference-on-computational-semantics-long-papers/) ([International Conference on Computational Semantics – Long Papers 2013](https://md.research.elamur.ai/free/en/venues/international-conference-on-computational-semantics-long-papers/2013/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W13-0121/) · [PDF](https://aclanthology.org/W13-0121.pdf) · [OpenAlex](https://openalex.org/W2252051222)

**Citations:** 6 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/toledo-2013-semantic-annotation-textual-entailment/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
RTE corpora only label sentence pairs as entailing or not. They give no account of the reasoning that links a premise (T) to a hypothesis (H), so systems can be ranked by accuracy, but their inference steps cannot be checked against a gold standard. The authors propose a model-theoretic framework for this. Its core is a formally defined interpreted lexicon: words receive semantic types over e and t, plus denotations, some of them fixed logical operators. An informal annotation scheme then tells annotators how to bind T–H material to that lexicon. The work targets restrictive, intersective and appositive modification, because these are frequent in RTE and can be annotated consistently. As a first stage, a narrower scheme was applied to all valid pairs in RTE 1–4. Annotators marked restrictive modification (RMOD), appositives (APP) and conjunction (CONJ), without spelling out the full proof. These phenomena appear in 80.65% of entailments, and annotators agreed 68% of the time on average. The annotated corpus is public and is a step toward a benchmark whose entailments a theorem prover could verify.

## Key points
- The lexicon covers proper names, nouns, verbs, copula (both predicative and equality readings), articles (with *the* as iota), *some*, and restrictive versus appositive *who*. It also covers intersective and restrictive adjectives and adverbs, which are built from helper functions such as AND, I_m, R_m and P_r.
- Three annotation categories were used: RMOD, APP and CONJ. They were applied only to valid entailment pairs in RTE 1–4, which were found in 80.65% of the entailments.
- Consistency was measured in four checks of 50–70 jointly annotated pairs each, giving roughly 66.67%, 74.11%, 66.67% and 64.66% identical annotations (about 68% on average).
- Annotation was done in GATE Developer in two steps:
  - mark the relevant span in T;
  - mark the corresponding span in H that results from the inference.
- Related logic-based systems showed high precision and very low recall. Bos & Markert reached 76% precision but 5.8% recall on RTE 2. The MacCartney & Manning natural-logic system had a similar profile on RTE 3.

## Interesting ideas
- The lexicon separates content words, which get arbitrary constants, from logical words whose meanings are fixed (iota, existential, conjunction, identity), so only the fixed part drives inference.
- Intersective modifiers license two inferences ("Dutch man" gives both man and Dutch), while restrictive ones license only the head noun ("short man" is a man, but not short in absolute terms).
- Appositive relative clauses add a side fact about an entity that is already identified, and they leave the noun's reference unchanged, unlike restrictive clauses.
- Each annotation links a span in the premise to the span in the hypothesis that it justifies, which gives a traceable evidence chain for every entailment.
- Annotator disagreements were sorted into missing, wrong scheme or scope, structural ambiguity, inference ambiguity and scheme unclarity, rather than reported only as a single agreement number.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (world): Compiled knowledge could keep a small, hard-coded set of operator words apart from open-class constants, so that inference rules are defined only over the fixed operators.
- **I2** (world): When asserting entity states, the adjective class could decide whether the property is stored as a standalone fact about the entity or only relative to its noun.
- **I3** (frames): Non-restrictive clauses and appositions could be split off as separate assertion frames about the head entity, instead of being folded into its identifying description.
- **I4** (answer): Yes/no and QA readers could output premise-span → hypothesis-span links as their explanation, making each answer checkable step by step.
- **I5** (learn): When witness parsers or teacher tiers disagree on silver data, tagging the cause (attachment ambiguity versus guideline gap) would show whether to fix rules or flag the case as genuinely ambiguous.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works

## Cite

Cite as: Assaf Toledo, Stavroula Alexandropoulou, Sophia Katrenko, Heidi Klockmann, Pepijn Kokke and Yoad Winter. 2013. Semantic Annotation of Textual Entailment. Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013) – Long Papers.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{toledo2013semantic,
  title = {{Semantic Annotation of Textual Entailment}},
  author = {Assaf Toledo, Stavroula Alexandropoulou, Sophia Katrenko, Heidi Klockmann, Pepijn Kokke and Yoad Winter},
  booktitle = {Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013) – Long Papers},
  year = {2013},
  url = {https://aclanthology.org/W13-0121/}
}
```

- [← previous Exploration of Speech enabled System for English](https://md.research.elamur.ai/free/en/papers/sharma-2013-exploration-speech-enabled-system-english/)
- [next → Automatic Generation of English Respellings](https://md.research.elamur.ai/free/en/papers/hauer-2013-automatic-generation-english-respellings/)
- [↑ back to papers of 2013](https://md.research.elamur.ai/free/en/papers/2013/page/2/)

## References

16 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bar Haim, R., I. Dagan, B. Dolan, L. Ferro, D. Giampiccolo, B. Magnini, and I. Szpektor (2006). The second pascal recognising textual entailment challenge. In In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment. [find](https://scholar.google.com/scholar?q=Bar+Haim%2C+R%2E%2C+I%2E+Dagan%2C+B%2E+Dolan%2C+L%2E+Ferro%2C+D%2E+Giampiccolo%2C+B%2E+Magnini%2C+and+I%2E+Szpektor+%282006%29%2E+The+second+pascal+recognising+textual+entailment+challenge%2E+In+In+Proceedings+of+the+Second+PASCAL+Chall)
2. Bentivogli, L., I. Dagan, H. T. Dang, D. Giampiccolo, and B. Magnini (2009). The fifth pascal recognizing textual entailment challenge. Proceedings of TAC 9, 14–24. [find](https://scholar.google.com/scholar?q=Bentivogli%2C+L%2E%2C+I%2E+Dagan%2C+H%2E+T%2E+Dang%2C+D%2E+Giampiccolo%2C+and+B%2E+Magnini+%282009%29%2E+The+fifth+pascal+recognizing+textual+entailment+challenge%2E+Proceedings+of+TAC+9%2C+14%E2%80%9324%2E)
3. Bos, J., S. Clark, M. Steedman, J. R. Curran, and J. Hockenmaier (2004). Wide-coverage semantic representations from a CCG parser. In Proceedings of the 20th international conference on Computational Linguistics, pp. 12–40. [find](https://scholar.google.com/scholar?q=Bos%2C+J%2E%2C+S%2E+Clark%2C+M%2E+Steedman%2C+J%2E+R%2E+Curran%2C+and+J%2E+Hockenmaier+%282004%29%2E+Wide%2Dcoverage+semantic+representations+from+a+CCG+parser%2E+In+Proceedings+of+the+20th+international+conference+on+Computational+)
4. Bos, J. and K. Markert (2005). [Recognising textual entailment with logical inference](https://md.research.elamur.ai/free/en/papers/bos-2005-recognising-textual-entailment-logical-inference/). In Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, pp. 628–635.
5. Cooper, R., D. Crouch, J. Van Eijck, C. Fox, J. Van Genabith, J. Jaspars, H. Kamp, D. Milward, M. Pinkal, M. Poesio, S. Pulman, T. Briscoe, H. Maier, and K. Konrad (1996). Using the Framework. The Fracas Consortium. [find](https://scholar.google.com/scholar?q=Cooper%2C+R%2E%2C+D%2E+Crouch%2C+J%2E+Van+Eijck%2C+C%2E+Fox%2C+J%2E+Van+Genabith%2C+J%2E+Jaspars%2C+H%2E+Kamp%2C+D%2E+Milward%2C+M%2E+Pinkal%2C+M%2E+Poesio%2C+S%2E+Pulman%2C+T%2E+Briscoe%2C+H%2E+Maier%2C+and+K%2E+Konrad+%281996%29%2E+Using+the+Framework%2E+The+Fra)
6. Cunningham, H., D. Maynard, K. Bontcheva, V. Tablan, N. Aswani, I. Roberts, G. Gorrell, A. Funk, A. Roberts, D. Damljanovic, T. Heitz, M. A. Greenwood, H. Saggion, J. Petrak, Y. Li, and W. Peters (2011). Text Processing with GATE (Version 6). [find](https://scholar.google.com/scholar?q=Cunningham%2C+H%2E%2C+D%2E+Maynard%2C+K%2E+Bontcheva%2C+V%2E+Tablan%2C+N%2E+Aswani%2C+I%2E+Roberts%2C+G%2E+Gorrell%2C+A%2E+Funk%2C+A%2E+Roberts%2C+D%2E+Damljanovic%2C+T%2E+Heitz%2C+M%2E+A%2E+Greenwood%2C+H%2E+Saggion%2C+J%2E+Petrak%2C+Y%2E+Li%2C+and+W%2E+Peters+%28201)
7. Dagan, I., O. Glickman, and B. Magnini (2006). The pascal recognising textual entailment challenge. Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, 177–190. [find](https://scholar.google.com/scholar?q=Dagan%2C+I%2E%2C+O%2E+Glickman%2C+and+B%2E+Magnini+%282006%29%2E+The+pascal+recognising+textual+entailment+challenge%2E+Machine+Learning+Challenges%2E+Evaluating+Predictive+Uncertainty%2C+Visual+Object+Classification%2C+and+Re)
8. Garoufi, K. (2007). Towards a better understanding of applied textual entailment: Annotation and evaluation of the rte-2 dataset. Master’s thesis, Saarland University. [find](https://scholar.google.com/scholar?q=Garoufi%2C+K%2E+%282007%29%2E+Towards+a+better+understanding+of+applied+textual+entailment%3A+Annotation+and+evaluation+of+the+rte%2D2+dataset%2E+Master%E2%80%99s+thesis%2C+Saarland+University%2E)
9. Giampiccolo, D., H. T. Dang, B. Magnini, I. Dagan, and E. Cabrio (2008). The fourth pascal recognising textual entailment challenge. In In TAC 2008 Proceedings. [find](https://scholar.google.com/scholar?q=Giampiccolo%2C+D%2E%2C+H%2E+T%2E+Dang%2C+B%2E+Magnini%2C+I%2E+Dagan%2C+and+E%2E+Cabrio+%282008%29%2E+The+fourth+pascal+recognising+textual+entailment+challenge%2E+In+In+TAC+2008+Proceedings%2E)
10. Giampiccolo, D., B. Magnini, I. Dagan, and B. Dolan (2007). The third pascal recognizing textual entailment challenge. In In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, RTE ’07, Stroudsburg, PA, USA, pp. 1–9. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Giampiccolo%2C+D%2E%2C+B%2E+Magnini%2C+I%2E+Dagan%2C+and+B%2E+Dolan+%282007%29%2E+The+third+pascal+recognizing+textual+entailment+challenge%2E+In+In+Proceedings+of+the+ACL%2DPASCAL+Workshop+on+Textual+Entailment+and+Paraphrasi)
11. Kamp, H. and U. Reyle (1993). From discourse to logic: Introduction to model-theoretic semantics of natural language, formal logic and discourse representation theory, Volume 42. Kluwer Academic Dordrecht,, The Netherlands. [find](https://scholar.google.com/scholar?q=Kamp%2C+H%2E+and+U%2E+Reyle+%281993%29%2E+From+discourse+to+logic%3A+Introduction+to+model%2Dtheoretic+semantics+of+natural+language%2C+formal+logic+and+discourse+representation+theory%2C+Volume+42%2E+Kluwer+Academic+Dordr)
12. MacCartney, B. and C. D. Manning (2007). Natural logic for textual inference. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pp. 193–200. [find](https://scholar.google.com/scholar?q=MacCartney%2C+B%2E+and+C%2E+D%2E+Manning+%282007%29%2E+Natural+logic+for+textual+inference%2E+In+Proceedings+of+the+ACL%2DPASCAL+Workshop+on+Textual+Entailment+and+Paraphrasing%2C+pp%2E+193%E2%80%93200%2E)
13. Pratt-Hartmann, I. and L. S. Moss (2009). Logics for the relational syllogistic. Review of Symbolic Logic, 647–683. [↗](https://doi.org/10.1017/s1755020309990086)
14. Stern, A. and I. Dagan (2011). A confidence model for syntactically-motivated entailment proofs. In Proceedings of RANLP 2011. [find](https://scholar.google.com/scholar?q=Stern%2C+A%2E+and+I%2E+Dagan+%282011%29%2E+A+confidence+model+for+syntactically%2Dmotivated+entailment+proofs%2E+In+Proceedings+of+RANLP+2011%2E)
15. Toledo, A., S. Katrenko, S. Alexandropoulou, H. Klockmann, A. Stern, I. Dagan, and Y. Winter (2012). Semantic annotation for textual entailment recognition. In Proceedings of the Eleventh Mexican International Conference on Artificial Intelligence (MICAI). [find](https://scholar.google.com/scholar?q=Toledo%2C+A%2E%2C+S%2E+Katrenko%2C+S%2E+Alexandropoulou%2C+H%2E+Klockmann%2C+A%2E+Stern%2C+I%2E+Dagan%2C+and+Y%2E+Winter+%282012%29%2E+Semantic+annotation+for+textual+entailment+recognition%2E+In+Proceedings+of+the+Eleventh+Mexican+Inte)
16. Valencia, V. S. (1991). Studies on natural logic and categorial grammar. Ph. D. thesis, University of Amsterdam. [find](https://scholar.google.com/scholar?q=Valencia%2C+V%2E+S%2E+%281991%29%2E+Studies+on+natural+logic+and+categorial+grammar%2E+Ph%2E+D%2E+thesis%2C+University+of+Amsterdam%2E)

*Updated 24 October 2026*