---
title: "FATE: a FrameNet-Annotated Corpus for — Burchardt 2008"
description: "Earlier studies suggested that predicate-argument structure captures many phenomena in textual entailment data, such as paraphrases and argument…"
authors: ["Aljoscha Burchardt", "Marco Pennacchiotti"]
year: 2008
venue: "Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)"
source: "https://aclanthology.org/L08-1038/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/burchardt-2008-fate-framenet-annotated-corpus-textual/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/burchardt-2008-fate-framenet-annotated-corpus-textual/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2008](https://md.research.elamur.ai/free/papers/2008/) / FATE: a FrameNet-Annotated Corpus for Textual Entailment

# FATE: a FrameNet-Annotated Corpus for Textual Entailment

**Authors:** [Aljoscha Burchardt](https://md.research.elamur.ai/free/authors/aljoscha-burchardt/), [Marco Pennacchiotti](https://md.research.elamur.ai/free/authors/marco-pennacchiotti/) · **Year:** 2008 · **Venue:** Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/L08-1038/) · [PDF](https://aclanthology.org/L08-1038.pdf) · [OpenAlex](https://openalex.org/W159676160)

**Citations:** 24 citations (openalex-snapshot · 2026-10-14) · [11](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Earlier studies suggested that predicate-argument structure captures many phenomena in textual entailment data, such as paraphrases and argument alternations. However, the only FrameNet-based RTE system had performed only moderately. The authors give three possible reasons: FrameNet may not cover enough of the data, frame semantic parsers may be unreliable, or systems may use frame information poorly. To help tell these explanations apart, they built FATE, a manually produced gold-standard corpus. It covers all 800 text–hypothesis pairs of the RTE-2 test set, annotated with frames and semantic roles. The annotation uses a new scheme. It is full-text annotation, but only of the "relevant" frame evoking elements rather than every word that could evoke a frame. Consistency was checked through inter- and intra-annotator agreement and weekly supervision meetings. The resulting statistics show that very few instances needed an Unknown frame. This suggests FrameNet coverage is probably not what holds back frame-based RTE. The corpus can also serve as a benchmark for parsers such as Shalmaneser and as a clean testbed for designing entailment systems.

## Key points
- FATE annotates the 800 (T, H) pairs of the RTE-2 test set with FrameNet frames and roles. It is distributed in SALSA/TIGER XML format.
- Annotation took 230 hours in total. Positive pairs averaged about 13 minutes each and negative pairs about 21 minutes.
- The corpus contains 4,489 frames (about 5.6 per pair) and 9,518 roles (about 2.1 per frame).
- Unknown frames make up only 8% of frame instances (373 cases) and unknown roles only 1%. This contrasts with SALSA's German annotation, where about a third of verb occurrences lacked a suitable frame.
- Intra-annotator agreement was measured on 109 repeated sentence pairs. It reached 97% for FEEs, 98% for frames and 88% for roles.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

Topics: [Treebanks](https://md.research.elamur.ai/free/topics/treebanks/) · [Semantic role labeling](https://md.research.elamur.ai/free/topics/semantic-role-labeling/)

## Cited by

Cited by

11 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 10 not yet.

- [Developing a French FrameNet: Methodology and First results](https://md.research.elamur.ai/free/papers/candito-2014-developing-french-framenet-methodology-first/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{burchardt2008fate,
  title = {{FATE: a FrameNet-Annotated Corpus for Textual Entailment}},
  author = {Burchardt, Aljoscha and Pennacchiotti, Marco},
  booktitle = {Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)},
  year = {2008},
  doi = {10.63317/37cq5v73ysjz},
  url = {https://doi.org/10.63317/37cq5v73ysjz}
}
```

- [← previous Error Analysis for Learning-based Coreference Resolution](https://md.research.elamur.ai/free/papers/uryupina-2008-error-analysis-learning-based-coreference/)
- [next → Feature Forest Models for Probabilistic HPSG Parsing](https://md.research.elamur.ai/free/papers/cl-2008-miyao-feature-forest-models-probabilistic-hpsg/)
- [↑ back to papers of 2008](https://md.research.elamur.ai/free/papers/2008/)

## References

18 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The Berkeley FrameNet project. In Proceedings of COLING-ACL, Montreal, Canada. [find](https://scholar.google.com/scholar?q=Collin+F%2E+Baker%2C+Charles+J%2E+Fillmore%2C+and+John+B%2E+Lowe%2E+1998%2E+The+Berkeley+FrameNet+project%2E+In+Proceedings+of+COLING%2DACL%2C+Montreal%2C+Canada%2E) [our card](https://md.research.elamur.ai/free/papers/lex-1998-baker-berkeley-framenet/)
2. Roy Bar-Haim, Idan Szpektor, and Oren Glickman. 2005. Definition and analysis of intermediate entailment levels. In Definition and Analysis of Intermediate Entailment Levels, Ann Arbor, Michigan. [find](https://scholar.google.com/scholar?q=Roy+Bar%2DHaim%2C+Idan+Szpektor%2C+and+Oren+Glickman%2E+2005%2E+Definition+and+analysis+of+intermediate+entailment+levels%2E+In+Definition+and+Analysis+of+Intermediate+Entailment+Levels%2C+Ann+Arbor%2C+Michigan%2E)
3. Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor, editors. 2006. Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, Venice, Italy. [find](https://scholar.google.com/scholar?q=Roy+Bar%2DHaim%2C+Ido+Dagan%2C+Bill+Dolan%2C+Lisa+Ferro%2C+Danilo+Giampiccolo%2C+Bernardo+Magnini%2C+and+Idan+Szpektor%2C+editors%2E+2006%2E+Proceedings+of+the+Second+PASCAL+Challenges+Workshop+on+Recognising+Textual+Ent)
4. Johan Bos and Katja Markert. 2005. Combining shallow and deep nlp methods for recognizing textual entailment. In Pascal, Proceedings of the First Challenge Workshop, Recognizing Textual Entailment., Southampton, UK. [find](https://scholar.google.com/scholar?q=Johan+Bos+and+Katja+Markert%2E+2005%2E+Combining+shallow+and+deep+nlp+methods+for+recognizing+textual+entailment%2E+In+Pascal%2C+Proceedings+of+the+First+Challenge+Workshop%2C+Recognizing+Textual+Entailment%2E%2C+S)
5. Aljoscha Burchardt and Anette Frank. 2006. Approximating Textual Entailment with LFG and FrameNet Frames. In Proceedings of PASCAL RTE2 Workshop. [find](https://scholar.google.com/scholar?q=Aljoscha+Burchardt+and+Anette+Frank%2E+2006%2E+Approximating+Textual+Entailment+with+LFG+and+FrameNet+Frames%2E+In+Proceedings+of+PASCAL+RTE2+Workshop%2E)
6. Aljoscha Burchardt, Katrin Erk, and Anette Frank. 2005. A WordNet Detour to FrameNet. In B. Fisseni, H.-C. Schmitz, B. Schröder, and P. Wagner, editors, Sprachtechnologie, mobile Kommunikation und linguistische Resourcen, volume 8 of Computer Studies in Language and Speech. Peter Lang, Frankfurt/Main. [find](https://scholar.google.com/scholar?q=Aljoscha+Burchardt%2C+Katrin+Erk%2C+and+Anette+Frank%2E+2005%2E+A+WordNet+Detour+to+FrameNet%2E+In+B%2E+Fisseni%2C+H%2E%2DC%2E+Schmitz%2C+B%2E+Schro%CC%88der%2C+and+P%2E+Wagner%2C+editors%2C+Sprachtechnologie%2C+mobile+Kommunikation+und+li)
7. Aljoscha Burchardt, Katrin Erk, Anette Frank, Andrea Kowalski, Sebastian Pado, and Manfred Pinkal. 2006a. The salsa corpus: a german corpus resource for lexical semantics. In Proceedings of LREC 2006, Genoa, Italy. [find](https://scholar.google.com/scholar?q=Aljoscha+Burchardt%2C+Katrin+Erk%2C+Anette+Frank%2C+Andrea+Kowalski%2C+Sebastian+Pado%2C+and+Manfred+Pinkal%2E+2006a%2E+The+salsa+corpus%3A+a+german+corpus+resource+for+lexical+semantics%2E+In+Proceedings+of+LREC+2006%2C) [our card](https://md.research.elamur.ai/free/papers/burchardt-2006-salsa-corpus-german-corpus-resource/)
8. Aljoscha Burchardt, Katrin Erk, Anette Frank, Andrea Kowalski, Sebastian Pado, and Manfred Pinkal. 2006b. Salto – a versatile multi-level annotation tool. In Proceedings of LREC 2006, Genoa, Italy. [find](https://scholar.google.com/scholar?q=Aljoscha+Burchardt%2C+Katrin+Erk%2C+Anette+Frank%2C+Andrea+Kowalski%2C+Sebastian+Pado%2C+and+Manfred+Pinkal%2E+2006b%2E+Salto+%E2%80%93+a+versatile+multi%2Dlevel+annotation+tool%2E+In+Proceedings+of+LREC+2006%2C+Genoa%2C+Italy%2E)
9. Aljoscha Burchardt, Nils Reiter, Stefan Thater, and Anette Frank. 2007. A semantic approach to textual entailment: System evaluation and task analysis. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, Prague. [find](https://scholar.google.com/scholar?q=Aljoscha+Burchardt%2C+Nils+Reiter%2C+Stefan+Thater%2C+and+Anette+Frank%2E+2007%2E+A+semantic+approach+to+textual+entailment%3A+System+evaluation+and+task+analysis%2E+In+Proceedings+of+the+ACL%2DPASCAL+Workshop+on+Tex)
10. M. Collins. 1999. Head-driven Statistical Models for Natural Language Parsing. Ph.D. thesis, University of Pennsylvania, Philadelphia. [find](https://scholar.google.com/scholar?q=M%2E+Collins%2E+1999%2E+Head%2Ddriven+Statistical+Models+for+Natural+Language+Parsing%2E+Ph%2ED%2E+thesis%2C+University+of+Pennsylvania%2C+Philadelphia%2E) [our card](https://md.research.elamur.ai/free/papers/cl-2003-collins-head-driven-statistical-models-natural/)
11. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL Recognising Textual Entailment Challenge. In Joaquin Quiñonero-Candela, Ido Dagan, Bernardo Magnini, and Florence D’Alché-Buc, editors, Evaluating Predictive Uncertainty, Visual Object Categorization and Textual Entailment, volume 3944 of Lecture Notes in Computer Science, pages 1–27, Heidelberg, Germany. Springer. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+2006%2E+The+PASCAL+Recognising+Textual+Entailment+Challenge%2E+In+Joaquin+Quin%CC%83onero%2DCandela%2C+Ido+Dagan%2C+Bernardo+Magnini%2C+and+Florence+D%E2%80%99Alche%CC%81%2DBuc%2C+editor)
12. Katrin Erk and Sebastian Pado. 2006. Shalmaneser - a flexible toolbox for semantic role assignment. In Proceedings of LREC 2006, Genoa, Italy. [find](https://scholar.google.com/scholar?q=Katrin+Erk+and+Sebastian+Pado%2E+2006%2E+Shalmaneser+%2D+a+flexible+toolbox+for+semantic+role+assignment%2E+In+Proceedings+of+LREC+2006%2C+Genoa%2C+Italy%2E)
13. K. Garoufi. 2007. Towards a better understanding of applied textual entailment: Annotation and evaluation of the rte-2 dataset. M.Sc. thesis, Saarland University. [find](https://scholar.google.com/scholar?q=K%2E+Garoufi%2E+2007%2E+Towards+a+better+understanding+of+applied+textual+entailment%3A+Annotation+and+evaluation+of+the+rte%2D2+dataset%2E+M%2ESc%2E+thesis%2C+Saarland+University%2E)
14. Paul Kingsbury, Martha Palmer, and Mitch Marcus. 2002. Adding semantic annotation to the Penn TreeBank. In Proceedings of the Human Language Technology Conference, San Diego. [find](https://scholar.google.com/scholar?q=Paul+Kingsbury%2C+Martha+Palmer%2C+and+Mitch+Marcus%2E+2002%2E+Adding+semantic+annotation+to+the+Penn+TreeBank%2E+In+Proceedings+of+the+Human+Language+Technology+Conference%2C+San+Diego%2E)
15. Ken Litkowski. 2006. Componential analysis for recognizing textual entailment. In Proceedings of PASCAL RTE2 Workshop. [find](https://scholar.google.com/scholar?q=Ken+Litkowski%2E+2006%2E+Componential+analysis+for+recognizing+textual+entailment%2E+In+Proceedings+of+PASCAL+RTE2+Workshop%2E)
16. S. Pado. 2007. Cross-Lingual Annotation Projection Models for Role-Semantic Information. Ph.D. thesis, Saarland University, Germany. [find](https://scholar.google.com/scholar?q=S%2E+Pado%2E+2007%2E+Cross%2DLingual+Annotation+Projection+Models+for+Role%2DSemantic+Information%2E+Ph%2ED%2E+thesis%2C+Saarland+University%2C+Germany%2E)
17. Josef Ruppenhofer, Michael Ellsworth, Miriam R. L. Petruck, and Christopher R. Johnson. 2007. Framenet: Theory and practice. Available at [http://framenet.icsi.berkeley.edu/](http://framenet.icsi.berkeley.edu/). [↗](http://framenet.icsi.berkeley.edu/)
18. Satoshi Sekine, Kentaro Inui, Ido Dagan, Bill Dolan, Danilo Giampiccolo, and Bernardo Magnini, editors. 2007. Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing. Association for Computational Linguistics, Prague, June. [find](https://scholar.google.com/scholar?q=Satoshi+Sekine%2C+Kentaro+Inui%2C+Ido+Dagan%2C+Bill+Dolan%2C+Danilo+Giampiccolo%2C+and+Bernardo+Magnini%2C+editors%2E+2007%2E+Proceedings+of+the+ACL%2DPASCAL+Workshop+on+Textual+Entailment+and+Paraphrasing%2E+Association)