---
title: "A Unified Model of Phrasal and Sentential Evidence for"
description: "Patwardhan and Riloff propose GLACIER, a probabilistic model for noun phrase extraction. 5% F-score. 21 references, cited by 63 papers here."
authors: ["Siddharth Patwardhan", "Ellen Riloff"]
year: 2009
venue: "Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/D09-1016/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/patwardhan-2009-unified-model-phrasal-sentential-evidence/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/patwardhan-2009-unified-model-phrasal-sentential-evidence/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2009](https://md.research.elamur.ai/free/en/papers/2009/) / A Unified Model of Phrasal and Sentential Evidence for Information Extraction

# A Unified Model of Phrasal and Sentential Evidence for Information Extraction

**Authors:** [Siddharth Patwardhan](https://md.research.elamur.ai/free/en/authors/siddharth-patwardhan/) and [Ellen Riloff](https://md.research.elamur.ai/free/en/authors/ellen-riloff/) · **Year:** 2009 · **Venue:** [Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2009](https://md.research.elamur.ai/free/en/venues/emnlp/2009/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/D09-1016/) · [PDF](https://aclanthology.org/D09-1016.pdf) · [OpenAlex](https://openalex.org/W2134486566)

**Citations:** 117 citations (OpenAlex, 2026-10-14) · [63](https://md.research.elamur.ai/free/en/papers/patwardhan-2009-unified-model-phrasal-sentential-evidence/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Event-oriented information extraction systems usually judge a candidate noun phrase by its immediate surroundings. This misses role fillers that sit in clauses with no event word, such as "people died" or "bridges were destroyed," which could follow many kinds of incidents. Patwardhan and Riloff propose GLACIER, a probabilistic model for noun phrase extraction. It treats the decision as a joint probability: the sentence describes a relevant event, and the phrase is a plausible filler for a given role. The product rule splits this into two components:

- a **sentential event recognizer**, built with Naïve Bayes or SVM classifiers;
- a **plausible role-filler recognizer**, a Naïve Bayes classifier over local lexical, semantic and syntactic features, conditioned on the sentence being an event sentence.

Neither component makes a hard keep-or-drop decision about a sentence. The authors evaluate on MUC-4 (terrorism) and ProMed (disease outbreaks) against pattern-based and classifier baselines. GLACIER wins clearly on most roles, with gains in recall and in precision. The work shows that sentence-level context can rescue facts that local evidence alone treats as too vague.

## Key points
- **MUC-4, NB/SVM variant (threshold 0.40):**
  - It beats all three baselines (AutoSlog-TS, a Naïve Bayes baseline, Sem Affinity) on PerpInd, Victim and Weapon.
  - PerpInd F-score rose from 0.43 (Sem Affinity) to 0.54.
  - Victim and Weapon each gained about 5% F-score over the best baseline.
- **Weak spot:** PerpOrg results were worse than the baselines. Responsible organizations tend to be named later in a document, away from the event description, so their sentences receive low event probabilities.
- **ProMed:**
  - The NB/NB variant worked better than NB/SVM in this domain.
  - Disease F-score went from 0.43 to 0.49, mainly through higher precision (up to 0.41).
  - Victim F-score tied the best baseline at 0.44, with 5% more recall and 3% less precision.
- **Implementation:** Weka for Naïve Bayes and SVMLight for the SVM. Extraction thresholds were 0.90 for NB/NB, and 0.50 or 0.40 for NB/SVM. Training required event sentences to be annotated.
- **Shared component:** One sentential event recognizer serves all roles, while each role gets its own role-filler recognizer.

## Interesting ideas
- **Product-rule factorization instead of a pipeline.** Multiplying a context-level probability by a local probability conditioned on that context replaces a filter-then-extract cascade. Strong local evidence can still win in a weak sentence, and weak local evidence can pass in a strong sentence. This suits any task where a span's relevance depends on a larger unit, such as relation extraction or document-level tagging.
- **Conditioning the local classifier on the context class.** The role-filler model is trained to assume it is looking at an event sentence. It can therefore accept generic phrases that would be too risky in isolation, because the sentence component supplies the topical evidence.
- **Using a better-calibrated classifier where probabilities matter.** Naïve Bayes pushes its estimates toward 0 or 1, so the authors tried an SVM for the sentence component. Score calibration deserves attention whenever classifier outputs are combined rather than just used as labels.
- **Tuning the extraction threshold per configuration.** The SVM-based variant needed a threshold below 0.5 to recover recall. Reporting results at several thresholds is a cheap way to avoid underselling a combined model.
- **Reusing one context model across many targets.** A single sentence-relevance model feeds every role-specific extractor, which reduces annotation and training cost when a schema has many slots.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Utah](https://md.research.elamur.ai/free/en/universities/university-of-utah/) — University, Salt Lake City, United States · 29 works · 5,781 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🔎 Information extraction](https://md.research.elamur.ai/free/en/topics/information-extraction/) · 109 works

## Cite

Cite as: Siddharth Patwardhan and Ellen Riloff. 2009. A Unified Model of Phrasal and Sentential Evidence for Information Extraction. Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{patwardhan2009unified,
  title = {{A Unified Model of Phrasal and Sentential Evidence for Information Extraction}},
  author = {Siddharth Patwardhan and Ellen Riloff},
  booktitle = {Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing},
  year = {2009},
  doi = {10.3115/1699510.1699530},
  url = {https://doi.org/10.3115/1699510.1699530}
}
```

- [← previous Application-driven Statistical Paraphrase Generation](https://md.research.elamur.ai/free/en/papers/zhao-2009-application-driven-statistical-paraphrase-generation/)
- [next → Using a Dependency Parser to Improve SMT for Subject-Object-Verb Languages](https://md.research.elamur.ai/free/en/papers/xu-2009-dependency-parser-improve-smt-subject/)
- [↑ back to papers of 2009](https://md.research.elamur.ai/free/en/papers/2009/)

## References

21 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. R. Bunescu and R. Mooney. 2004. Collective Information Extraction with Relational Markov Networks. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, pages 438–445, Barcelona, Spain, July. [find](https://scholar.google.com/scholar?q=R%2E+Bunescu+and+R%2E+Mooney%2E+2004%2E+Collective+Information+Extraction+with+Relational+Markov+Networks%2E+In+Proceedings+of+the+42nd+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+438)
2. M. Califf and R. Mooney. 2003. Bottom-Up Relational Learning of Pattern Matching Rules for Information Extraction. Journal of Machine Learning Research, 4:177–210, December. [find](https://scholar.google.com/scholar?q=M%2E+Califf+and+R%2E+Mooney%2E+2003%2E+Bottom%2DUp+Relational+Learning+of+Pattern+Matching+Rules+for+Information+Extraction%2E+Journal+of+Machine+Learning+Research%2C+4%3A177%E2%80%93210%2C+December%2E)
3. H. Chieu, H. Ng, and Y. Lee. 2003. Closing the Gap: Learning-Based Information Extraction Rivaling Knowledge-Engineering Methods. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 216–223, Sapporo, Japan, July. [find](https://scholar.google.com/scholar?q=H%2E+Chieu%2C+H%2E+Ng%2C+and+Y%2E+Lee%2E+2003%2E+Closing+the+Gap%3A+Learning%2DBased+Information+Extraction+Rivaling+Knowledge%2DEngineering+Methods%2E+In+Proceedings+of+the+41st+Annual+Meeting+of+the+Association+for+Compu)
4. P. Domingos and M. Pazzani. 1996. Beyond Independence: Conditions for the Optimality of the Simple Bayesian Classifier. In Proceedings of the Thirteenth International Conference on Machine Learning, pages 105–112, Bari, Italy, July. [find](https://scholar.google.com/scholar?q=P%2E+Domingos+and+M%2E+Pazzani%2E+1996%2E+Beyond+Independence%3A+Conditions+for+the+Optimality+of+the+Simple+Bayesian+Classifier%2E+In+Proceedings+of+the+Thirteenth+International+Conference+on+Machine+Learning%2C+p)
5. J. Finkel, T. Grenager, and C. Manning. 2005. [Incorporating Non-local Information into Information Extraction Systems by Gibbs Sampling](https://md.research.elamur.ai/free/en/papers/finkel-2005-incorporating-non-local-information-into/). In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics, pages 363–370, Ann Arbor, MI, June.
6. D. Freitag and A. McCallum. 2000. Information Extraction with HMM Structures Learned by Stochastic Optimization. In Proceedings of the Seventeenth National Conference on Artificial Intelligence, pages 584–589, Austin, TX, August. [find](https://scholar.google.com/scholar?q=D%2E+Freitag+and+A%2E+McCallum%2E+2000%2E+Information+Extraction+with+HMM+Structures+Learned+by+Stochastic+Optimization%2E+In+Proceedings+of+the+Seventeenth+National+Conference+on+Artificial+Intelligence%2C+pages)
7. D. Freitag. 1998. Toward General-Purpose Learning for Information Extraction. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, pages 404–408, Montreal, Quebec, August. [find](https://scholar.google.com/scholar?q=D%2E+Freitag%2E+1998%2E+Toward+General%2DPurpose+Learning+for+Information+Extraction%2E+In+Proceedings+of+the+36th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+17th+International+Conferen)
8. H. Ji and R. Grishman. 2008. [Refining Event Extraction through Cross-Document Inference](https://md.research.elamur.ai/free/en/papers/ji-2008-refining-event-extraction-through-cross/). In Proceedings of ACL-08: HLT, pages 254–262, Columbus, OH, June.
9. T. Joachims. 1998. [Text Categorization with Support Vector Machines: Learning with Many Relevant Features](https://md.research.elamur.ai/free/en/papers/joachims-1998-text-categorization-support-vector-machines/). In Proceedings of the Tenth European Conference on Machine Learning, pages 137–142, April.
10. C. Manning, P. Raghavan, and H Schütze. 2008. Introduction to Information Retrieval. Cambridge University Press, New York, NY. [find](https://scholar.google.com/scholar?q=C%2E+Manning%2C+P%2E+Raghavan%2C+and+H+Schu%CC%88tze%2E+2008%2E+Introduction+to+Information+Retrieval%2E+Cambridge+University+Press%2C+New+York%2C+NY%2E)
11. M. Maslennikov and T. Chua. 2007. A Multi-resolution Framework for Information Extraction from Free Text. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 592–599, Prague, Czech Republic, June. [find](https://scholar.google.com/scholar?q=M%2E+Maslennikov+and+T%2E+Chua%2E+2007%2E+A+Multi%2Dresolution+Framework+for+Information+Extraction+from+Free+Text%2E+In+Proceedings+of+the+45th+Annual+Meeting+of+the+Association+of+Computational+Linguistics%2C+pag)
12. S. Patwardhan and E. Riloff. 2007. Effective Information Extraction with Semantic Affinity Patterns and Relevant Regions. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 717–727, Prague, Czech Republic… [find](https://scholar.google.com/scholar?q=S%2E+Patwardhan+and+E%2E+Riloff%2E+2007%2E+Effective+Information+Extraction+with+Semantic+Affinity+Patterns+and+Relevant+Regions%2E+In+Proceedings+of+the+2007+Joint+Conference+on+Empirical+Methods+in+Natural+La)
13. W. Phillips and E. Riloff. 2007. Exploiting Role-Identifying Nouns and Expressions for Information Extraction. In Proceedings of International Conference on Recent Advances in Natural Language Processing, pages 165–172, Borovets, Bulgaria, September. [find](https://scholar.google.com/scholar?q=W%2E+Phillips+and+E%2E+Riloff%2E+2007%2E+Exploiting+Role%2DIdentifying+Nouns+and+Expressions+for+Information+Extraction%2E+In+Proceedings+of+International+Conference+on+Recent+Advances+in+Natural+Language+Process)
14. E. Riloff and W. Phillips. 2004. An Introduction to the Sundance and AutoSlog Systems. Technical Report UUCS-04-015, School of Computing, University of Utah. [find](https://scholar.google.com/scholar?q=E%2E+Riloff+and+W%2E+Phillips%2E+2004%2E+An+Introduction+to+the+Sundance+and+AutoSlog+Systems%2E+Technical+Report+UUCS%2D04%2D015%2C+School+of+Computing%2C+University+of+Utah%2E)
15. E. Riloff. 1996. Automatically Generating Extraction Patterns from Untagged Text. In Proceedings of the Thirteenth National Conference on Articial Intelligence, pages 1044–1049, Portland, OR, August. [find](https://scholar.google.com/scholar?q=E%2E+Riloff%2E+1996%2E+Automatically+Generating+Extraction+Patterns+from+Untagged+Text%2E+In+Proceedings+of+the+Thirteenth+National+Conference+on+Articial+Intelligence%2C+pages+1044%E2%80%931049%2C+Portland%2C+OR%2C+August%2E)
16. S. Soderland, D. Fisher, J. Aseltine, and W. Lehnert. 1995. CRYSTAL: Inducing a Conceptual Dictionary. In Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, pages 1314–1319, Montreal, Canada, August. [find](https://scholar.google.com/scholar?q=S%2E+Soderland%2C+D%2E+Fisher%2C+J%2E+Aseltine%2C+and+W%2E+Lehnert%2E+1995%2E+CRYSTAL%3A+Inducing+a+Conceptual+Dictionary%2E+In+Proceedings+of+the+Fourteenth+International+Joint+Conference+on+Artificial+Intelligence%2C+pages)
17. B. Sundheim. 1992. Overview of the Fourth Message Understanding Evaluation and Conference. In Proceedings of the Fourth Message Understanding Conference (MUC-4), pages 3–21, McLean, VA, June. [find](https://scholar.google.com/scholar?q=B%2E+Sundheim%2E+1992%2E+Overview+of+the+Fourth+Message+Understanding+Evaluation+and+Conference%2E+In+Proceedings+of+the+Fourth+Message+Understanding+Conference+%28MUC%2D4%29%2C+pages+3%E2%80%9321%2C+McLean%2C+VA%2C+June%2E)
18. V. Vapnik. 1995. The Nature of Statistical Learning Theory. Springer, New York, NY. [find](https://scholar.google.com/scholar?q=V%2E+Vapnik%2E+1995%2E+The+Nature+of+Statistical+Learning+Theory%2E+Springer%2C+New+York%2C+NY%2E)
19. I. Witten and E. Frank. 2005. Data Mining - Practical Machine Learning Tools and Techniques. Morgan– Kaufmann, San Francisco, CA. [find](https://scholar.google.com/scholar?q=I%2E+Witten+and+E%2E+Frank%2E+2005%2E+Data+Mining+%2D+Practical+Machine+Learning+Tools+and+Techniques%2E+Morgan%E2%80%93+Kaufmann%2C+San+Francisco%2C+CA%2E)
20. R. Yangarber, R. Grishman, P. Tapanainen, and S. Huttunen. 2000. [Automatic Acquisition of Domain Knowledge for Information Extraction](https://md.research.elamur.ai/free/en/papers/yangarber-2000-automatic-acquisition-domain-knowledge-information/). In Proceedings of the 18th International Conference on Computational Linguistics, pages 940–946, Saarbrücken, Germany, August.
21. B. Zadrozny and C. Elkan. 2001. Obtaining Calibrated Probability Estimates from Decision Trees and Naı̈ve Bayesian Classiers. In Proceedings of the Eighteenth International Conference on Machine Learning, pages 609–616, Williamstown, MA, June. [find](https://scholar.google.com/scholar?q=B%2E+Zadrozny+and+C%2E+Elkan%2E+2001%2E+Obtaining+Calibrated+Probability+Estimates+from+Decision+Trees+and+Na%C4%B1%CC%88ve+Bayesian+Classiers%2E+In+Proceedings+of+the+Eighteenth+International+Conference+on+Machine+Learn)

*Updated 24 October 2026*