---
title: "Evaluating FrameNet-style semantic parsing — Palmer 2010"
description: "Supervised FrameNet-style semantic role labeling (SRL) systems often report accuracies around 80%."
authors: ["Alexis Palmer", "Caroline Sporleder"]
year: 2010
venue: "Coling 2010: Posters"
source: "https://aclanthology.org/C10-2107/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/palmer-2010-evaluating-framenet-style-semantic-parsing/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/palmer-2010-evaluating-framenet-style-semantic-parsing/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2010](https://md.research.elamur.ai/free/papers/2010/) / Evaluating FrameNet-style semantic parsing: the role of coverage gaps in FrameNet

# Evaluating FrameNet-style semantic parsing: the role of coverage gaps in FrameNet

**Authors:** [Alexis Palmer](https://md.research.elamur.ai/free/authors/alexis-palmer/), [Caroline Sporleder](https://md.research.elamur.ai/free/authors/caroline-sporleder/) · **Year:** 2010 · **Venue:** Coling 2010: Posters · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/C10-2107/) · [PDF](https://aclanthology.org/C10-2107.pdf) · [OpenAlex](https://openalex.org/W2008011355)

**Citations:** 25 citations (openalex-snapshot · 2026-10-14) · [16](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Supervised FrameNet-style semantic role labeling (SRL) systems often report accuracies around 80%. Those figures, however, come from lemma-level evaluations on hand-picked predicates that have training data. Palmer and Sporleder examine what happens when such systems are evaluated on whole documents, where FrameNet's incomplete coverage becomes the main obstacle.

They propose a typology of five kinds of coverage gaps in FrameNet. They then measure coverage and run the Shalmaneser frame-assignment system on three full-text data sets: the FN1.3 lexicographic corpus (FNL), the FN1.3 full-text corpus (FNF), and texts annotated for SemEval 2010 Task-10.

Frame-assignment accuracy was only 37–47%. Few of these errors were genuine classifier mistakes. Most arose because FrameNet lacked the needed lemma, frame, or training examples. The authors conclude that tuning features or learning algorithms will gain little, and that research should focus on handling unseen data. As one such direction, they explore generalizing across lemmas and frames that overlap in FrameNet.

## Key points
- On average, 46.8% of targets (41–48% per data set) are "no-chance" cases, where FrameNet offers nothing for the target or the frame. The abstract calls this figure an oracle upper bound.
- The analysis covers more than 150,000 frame-assignment targets across FNL, FNF, and SemEval 2010 Task-10. Shalmaneser's frame-assignment accuracy ranges from 37% to 47%.
- Over 15% of lexical units in the SemEval gold annotations are absent from FrameNet. This matches Baker et al. (2007), who had to add 40 new frames for a SemEval 2007 test set containing 272 frame types.
- Overlap is a possible remedy:
  - Lemma overlap: 8,358 of FrameNet's 8,370 lemmas share at least one frame with another lemma.
  - Frame overlap: the 502 frames that have annotated examples average 278 training instances each, ranging from 1 to 6,233 (SELF_MOTION).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

Topics: [Semantic role labeling](https://md.research.elamur.ai/free/topics/semantic-role-labeling/)

## Cited by

Cited by

16 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 14 not yet.

- [Out-of-domain FrameNet Semantic Role Labeling](https://md.research.elamur.ai/free/papers/hartmann-2017-out-domain-framenet-semantic-role/)
- [FrameNet+: Fast Paraphrastic Tripling of FrameNet](https://md.research.elamur.ai/free/papers/pavlick-2015-framenet-fast-paraphrastic-tripling-framenet/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{palmer2010evaluating,
  title = {{Evaluating FrameNet-style semantic parsing: the role of coverage gaps in FrameNet}},
  author = {Palmer, Alexis and Sporleder, Caroline},
  booktitle = {Coling 2010: Posters},
  year = {2010},
  url = {https://aclanthology.org/C10-2107/}
}
```

- [← previous End-to-End Coreference Resolution via Hypergraph Partitioning](https://md.research.elamur.ai/free/papers/cai-2010-end-end-coreference-resolution-hypergraph/)
- [next → Evaluation Metrics For End-to-End Coreference Resolution Systems](https://md.research.elamur.ai/free/papers/cai-2010-evaluation-metrics-end-end-coreference/)
- [↑ back to papers of 2010](https://md.research.elamur.ai/free/papers/2010/)

## References

28 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. C. Baker, M. Ellsworth, K. Erk. 2007. Semeval-2007 task 19: Frame semantic structure extraction. In Proceedings of SemEval-2007. [find](https://scholar.google.com/scholar?q=C%2E+Baker%2C+M%2E+Ellsworth%2C+K%2E+Erk%2E+2007%2E+Semeval%2D2007+task+19%3A+Frame+semantic+structure+extraction%2E+In+Proceedings+of+SemEval%2D2007%2E) [our card](https://md.research.elamur.ai/free/papers/baker-2007-semeval-2007-task-19-frame/)
2. A. Burchardt, K. Erk, A. Frank. 2005. A WordNet Detour to FrameNet. In Proceedings of the GLDV-05 Workshop GermaNet II. [find](https://scholar.google.com/scholar?q=A%2E+Burchardt%2C+K%2E+Erk%2C+A%2E+Frank%2E+2005%2E+A+WordNet+Detour+to+FrameNet%2E+In+Proceedings+of+the+GLDV%2D05+Workshop+GermaNet+II%2E)
3. A. Burchardt, M. Pennacchiotti, S. Thater, M. Pinkal. 2009. Assessing the impact of frame semantics on textual entailment. Journal of Natural Language Engineering, Special Issue on Textual Entailment, 15(4):527–550. [find](https://scholar.google.com/scholar?q=A%2E+Burchardt%2C+M%2E+Pennacchiotti%2C+S%2E+Thater%2C+M%2E+Pinkal%2E+2009%2E+Assessing+the+impact+of+frame+semantics+on+textual+entailment%2E+Journal+of+Natural+Language+Engineering%2C+Special+Issue+on+Textual+Entailment%2C)
4. D. D. Cao, D. Croce, M. Pennacchiotti, R. Basili. 2008. Combining word sense and usage for modeling frame semantics. In Proceedings of STEP-08. [find](https://scholar.google.com/scholar?q=D%2E+D%2E+Cao%2C+D%2E+Croce%2C+M%2E+Pennacchiotti%2C+R%2E+Basili%2E+2008%2E+Combining+word+sense+and+usage+for+modeling+frame+semantics%2E+In+Proceedings+of+STEP%2D08%2E) [our card](https://md.research.elamur.ai/free/papers/cao-2008-combining-word-sense-usage-modeling/)
5. K. Deschacht, M.-F. Moens. 2009. Semi-supervised Semantic Role Labeling Using the Latent Words Language Model. In Proceedings of EMNLP-09. [find](https://scholar.google.com/scholar?q=K%2E+Deschacht%2C+M%2E%2DF%2E+Moens%2E+2009%2E+Semi%2Dsupervised+Semantic+Role+Labeling+Using+the+Latent+Words+Language+Model%2E+In+Proceedings+of+EMNLP%2D09%2E)
6. M. Ellsworth, K. Erk, P. Kingsbury, S. Padó. 2004. [find](https://scholar.google.com/scholar?q=M%2E+Ellsworth%2C+K%2E+Erk%2C+P%2E+Kingsbury%2C+S%2E+Pado%CC%81%2E+2004%2E)
7. PropBank, SALSA, and FrameNet: How Design Determines Product. In Proceedings LREC 2004 Workshop on Building Lexical Resources from Semantically Annotated Corpora. [find](https://scholar.google.com/scholar?q=PropBank%2C+SALSA%2C+and+FrameNet%3A+How+Design+Determines+Product%2E+In+Proceedings+LREC+2004+Workshop+on+Building+Lexical+Resources+from+Semantically+Annotated+Corpora%2E)
8. K. Erk, S. Padó. 2006. Shalmaneser – a toolchain for shallow semantic parsing. In Proceedings of LREC-06. [find](https://scholar.google.com/scholar?q=K%2E+Erk%2C+S%2E+Pado%CC%81%2E+2006%2E+Shalmaneser+%E2%80%93+a+toolchain+for+shallow+semantic+parsing%2E+In+Proceedings+of+LREC%2D06%2E)
9. K. Erk. 2005. Frame assignment as word sense disambiguation. In Proceedings of IWCS 6. [find](https://scholar.google.com/scholar?q=K%2E+Erk%2E+2005%2E+Frame+assignment+as+word+sense+disambiguation%2E+In+Proceedings+of+IWCS+6%2E)
10. H. Fürstenau, M. Lapata. 2009a. Graph alignment for semi-supervised semantic role labeling. In Proceedings of EMNLP 2009. [find](https://scholar.google.com/scholar?q=H%2E+Fu%CC%88rstenau%2C+M%2E+Lapata%2E+2009a%2E+Graph+alignment+for+semi%2Dsupervised+semantic+role+labeling%2E+In+Proceedings+of+EMNLP+2009%2E)
11. H. Fürstenau, M. Lapata. 2009b. Semi-supervised semantic role labeling. In Proceedings of EACL 2009. [find](https://scholar.google.com/scholar?q=H%2E+Fu%CC%88rstenau%2C+M%2E+Lapata%2E+2009b%2E+Semi%2Dsupervised+semantic+role+labeling%2E+In+Proceedings+of+EACL+2009%2E)
12. D. Gildea, D. Jurafsky. 2002. Automatic labeling of semantic roles. Computational Linguistics, 28(3):245–288. [find](https://scholar.google.com/scholar?q=D%2E+Gildea%2C+D%2E+Jurafsky%2E+2002%2E+Automatic+labeling+of+semantic+roles%2E+Computational+Linguistics%2C+28%283%29%3A245%E2%80%93288%2E) [our card](https://md.research.elamur.ai/free/papers/gildea-2002-automatic-labeling-semantic-roles/)
13. A. Gordon, R. Swanson. 2007. Generalizing semantic role annotations across syntactically similar verbs. In Proceedings of ACL 2007. [find](https://scholar.google.com/scholar?q=A%2E+Gordon%2C+R%2E+Swanson%2E+2007%2E+Generalizing+semantic+role+annotations+across+syntactically+similar+verbs%2E+In+Proceedings+of+ACL+2007%2E)
14. R. Johansson, P. Nugues. 2007. Using WordNet to extend FrameNet coverage. In Proceedings of the Workshop on Building Frame-semantic Resources for Scandinavian and Baltic Languages, NODAL-IDA. [find](https://scholar.google.com/scholar?q=R%2E+Johansson%2C+P%2E+Nugues%2E+2007%2E+Using+WordNet+to+extend+FrameNet+coverage%2E+In+Proceedings+of+the+Workshop+on+Building+Frame%2Dsemantic+Resources+for+Scandinavian+and+Baltic+Languages%2C+NODAL%2DIDA%2E)
15. M. Marcus, G. Kim, M. A. Marcinkiewicz, R. MacIntyre, A. Bies, M. Ferguson, K. Katz, B. Schasberger. 1994. The Penn Treebank: Annotating predicate argument structure. In ARPA Human Language Technology Workshop. [find](https://scholar.google.com/scholar?q=M%2E+Marcus%2C+G%2E+Kim%2C+M%2E+A%2E+Marcinkiewicz%2C+R%2E+MacIntyre%2C+A%2E+Bies%2C+M%2E+Ferguson%2C+K%2E+Katz%2C+B%2E+Schasberger%2E+1994%2E+The+Penn+Treebank%3A+Annotating+predicate+argument+structure%2E+In+ARPA+Human+Language+Technology) [our card](https://md.research.elamur.ai/free/papers/bank-1994-marcus-penn-treebank-annotating-predicate-argument/)
16. D. McCarthy. 2009. Word Sense Disambiguation: An Overview. Language and Linguistics Compass, 3(2):537–558. [find](https://scholar.google.com/scholar?q=D%2E+McCarthy%2E+2009%2E+Word+Sense+Disambiguation%3A+An+Overview%2E+Language+and+Linguistics+Compass%2C+3%282%29%3A537%E2%80%93558%2E)
17. A. Moschitti, P. Morarescu, S. Harabagiu. 2003. Open-domain information extraction via automatic semantic labeling. In Proceedings of FLAIRS. [find](https://scholar.google.com/scholar?q=A%2E+Moschitti%2C+P%2E+Morarescu%2C+S%2E+Harabagiu%2E+2003%2E+Open%2Ddomain+information+extraction+via+automatic+semantic+labeling%2E+In+Proceedings+of+FLAIRS%2E)
18. A. Moschitti, D. Pighin, R. Basili. 2008. Tree Kernels for Semantic Role Labeling. Computational Linguistics, 34(2). [find](https://scholar.google.com/scholar?q=A%2E+Moschitti%2C+D%2E+Pighin%2C+R%2E+Basili%2E+2008%2E+Tree+Kernels+for+Semantic+Role+Labeling%2E+Computational+Linguistics%2C+34%282%29%2E)
19. S. Padó, M. Pennacchiotti, C. Sporleder. 2008. Semantic role assignment for event nominalisations by leveraging verbal data. In Proceedings of Coling 2008. [find](https://scholar.google.com/scholar?q=S%2E+Pado%CC%81%2C+M%2E+Pennacchiotti%2C+C%2E+Sporleder%2E+2008%2E+Semantic+role+assignment+for+event+nominalisations+by+leveraging+verbal+data%2E+In+Proceedings+of+Coling+2008%2E)
20. M. Palmer, D. Gildea, P. Kingsbury. 2005. The Proposition Bank: An Annotated Corpus of Semantic Roles. Computational Linguistics, 31(1):71–105. [find](https://scholar.google.com/scholar?q=M%2E+Palmer%2C+D%2E+Gildea%2C+P%2E+Kingsbury%2E+2005%2E+The+Proposition+Bank%3A+An+Annotated+Corpus+of+Semantic+Roles%2E+Computational+Linguistics%2C+31%281%29%3A71%E2%80%93105%2E) [our card](https://md.research.elamur.ai/free/papers/palmer-2005-proposition-bank-annotated-corpus-semantic/)
21. M. Pennacchiotti, D. D. Cao, R. Basili, D. Croce, M. Roth. 2008a. Automatic induction of FrameNet lexical units. In Proceedings of EMNLP-08. [find](https://scholar.google.com/scholar?q=M%2E+Pennacchiotti%2C+D%2E+D%2E+Cao%2C+R%2E+Basili%2C+D%2E+Croce%2C+M%2E+Roth%2E+2008a%2E+Automatic+induction+of+FrameNet+lexical+units%2E+In+Proceedings+of+EMNLP%2D08%2E) [our card](https://md.research.elamur.ai/free/papers/pennacchiotti-2008-automatic-induction-framenet-lexical-units/)
22. M. Pennacchiotti, D. D. Cao, P. Marocco, R. Basili. 2008b. Towards a Vector Space Model for FrameNet-like Resources. In Proceedings of LREC-08. [find](https://scholar.google.com/scholar?q=M%2E+Pennacchiotti%2C+D%2E+D%2E+Cao%2C+P%2E+Marocco%2C+R%2E+Basili%2E+2008b%2E+Towards+a+Vector+Space+Model+for+FrameNet%2Dlike+Resources%2E+In+Proceedings+of+LREC%2D08%2E)
23. J. Ruppenhofer, M. Ellsworth, M. R. L. Petruck, C. R. Johnson, J. Scheffczyk. 2006. FrameNet II: Extended Theory and Practice. [find](https://scholar.google.com/scholar?q=J%2E+Ruppenhofer%2C+M%2E+Ellsworth%2C+M%2E+R%2E+L%2E+Petruck%2C+C%2E+R%2E+Johnson%2C+J%2E+Scheffczyk%2E+2006%2E+FrameNet+II%3A+Extended+Theory+and+Practice%2E)
24. J. Ruppenhofer, C. Sporleder, R. Morante, C. Baker, M. Palmer. 2009. SemEval-2010 Task 10: Linking Events and Their Participants in Discourse. In Proceedings of SEW-2009. [find](https://scholar.google.com/scholar?q=J%2E+Ruppenhofer%2C+C%2E+Sporleder%2C+R%2E+Morante%2C+C%2E+Baker%2C+M%2E+Palmer%2E+2009%2E+SemEval%2D2010+Task+10%3A+Linking+Events+and+Their+Participants+in+Discourse%2E+In+Proceedings+of+SEW%2D2009%2E)
25. J. Ruppenhofer, M. Pinkal, J. Sunde. 2010. Generating FrameNets of various granularities. In Proceedings of LREC 2010. [find](https://scholar.google.com/scholar?q=J%2E+Ruppenhofer%2C+M%2E+Pinkal%2C+J%2E+Sunde%2E+2010%2E+Generating+FrameNets+of+various+granularities%2E+In+Proceedings+of+LREC+2010%2E)
26. D. Shen, M. Lapata. 2007. Using semantic roles to improve question answering. In Proceedings of EMNLP-2007. [find](https://scholar.google.com/scholar?q=D%2E+Shen%2C+M%2E+Lapata%2E+2007%2E+Using+semantic+roles+to+improve+question+answering%2E+In+Proceedings+of+EMNLP%2D2007%2E)
27. M. Surdeanu, S. Harabagiu, J. Williams, P. Aarseth. 2003. Using predicate-argument structures for information extraction. In Proceedings of ACL 2003. [find](https://scholar.google.com/scholar?q=M%2E+Surdeanu%2C+S%2E+Harabagiu%2C+J%2E+Williams%2C+P%2E+Aarseth%2E+2003%2E+Using+predicate%2Dargument+structures+for+information+extraction%2E+In+Proceedings+of+ACL+2003%2E)
28. K. Toutanova, A. Haghighi, C. D. Manning. 2008. A Global Joint Model for Semantic Role Labeling. Computational Linguistics, 34(2). [find](https://scholar.google.com/scholar?q=K%2E+Toutanova%2C+A%2E+Haghighi%2C+C%2E+D%2E+Manning%2E+2008%2E+A+Global+Joint+Model+for+Semantic+Role+Labeling%2E+Computational+Linguistics%2C+34%282%29%2E)