---
title: "Learning subjective nouns using extraction — Riloff 2003"
description: "The paper addresses sentence-level subjectivity classification: separating sentences that express opinions, emotions, evaluations or speculation…"
authors: ["Ellen Riloff", "Janyce Wiebe", "Theresa Wilson"]
year: 2003
venue: "Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003"
source: "https://aclanthology.org/W03-0404/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/riloff-2003-learning-subjective-nouns-extraction-pattern/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/riloff-2003-learning-subjective-nouns-extraction-pattern/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2003](https://md.research.elamur.ai/free/papers/2003/) / Learning subjective nouns using extraction pattern bootstrapping

# Learning subjective nouns using extraction pattern bootstrapping

**Authors:** [Ellen Riloff](https://md.research.elamur.ai/free/authors/ellen-riloff/), [Janyce Wiebe](https://md.research.elamur.ai/free/authors/janyce-wiebe/), Theresa Wilson · **Year:** 2003 · **Venue:** Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003 · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W03-0404/) · [PDF](https://aclanthology.org/W03-0404.pdf) · [OpenAlex](https://openalex.org/W2098136027)

**Citations:** 494 citations (openalex-snapshot · 2026-10-14) · [85](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
The paper addresses sentence-level subjectivity classification: separating sentences that express opinions, emotions, evaluations or speculation ("private states") from factual ones in news text. The authors repurpose two extraction-pattern bootstrapping algorithms originally built for semantic lexicon induction, Meta-Bootstrapping and Basilisk. Starting from a few subjective seed nouns and a large unannotated FBIS news collection, the algorithms grow lists of subjective nouns. The hypothesis is that subjective words, like members of a semantic class, tend to appear in similar pattern contexts (e.g., as the object of "expressed").

The learned nouns, together with discourse features and subjectivity clues from earlier research, feed a Naive Bayes classifier. It is trained and tested on a small manually annotated FBIS corpus, where a sentence counts as subjective only if it contains a private state of at least medium strength. The bootstrappers produced over 1000 subjective nouns. The classifier reached 77% recall at 81% precision, improving on the earlier state of the art (Wiebe et al., 1999). The work shows that weakly supervised lexicon learning from raw text can strengthen supervised subjectivity models.

## Key points
- Data comes from English versions of foreign news collected by FBIS:
  - an experiment corpus of 109 documents (2197 sentences);
  - a tuning corpus of 33 documents (698 sentences);
  - a separate unannotated bootstrapping corpus of 950 texts.
- Inter-annotator agreement on 178 sentences was 88% (κ = 0.71). Removing the 11% of sentences that were borderline (only low-strength private states) raised agreement to 94% (κ = 0.87).
- Seed candidates were 850 nouns correlated with subjective sentences in another dataset. They were filtered by how often they occur in the bootstrapping corpus, because rare seeds cannot drive the bootstrapping.
- The best Naive Bayes classifier combines the learned nouns, discourse features and prior subjectivity clues. It reaches 76.1% accuracy, significantly higher than the other classifiers tested.
- A simple rule, labelling a sentence subjective if it contains any StrongSubjective noun, gives 87% precision at 26% recall.

## Interesting ideas
- **Reuse a semantic-class bootstrapper for a pragmatic category.** Seeding with subjective words instead of words like LOCATION or VEHICLE turns a lexicon learner into a subjectivity-lexicon learner. Any category whose members share syntactic contexts could be learned the same way, such as hedges, threats or trade-related terms.
- **Build the gold standard from annotation strength.** The authors noticed that disagreements clustered on low-strength cases. They therefore folded weak subjectivity into the objective class, which gives a cleaner and more reliable target for the classifier.
- **Choose seeds by their frequency in the target corpus.** Ranking candidate seeds by how often they occur in the corpus being bootstrapped avoids a stalled start. This check is cheap and applies to any seed-based method.
- **Combine unannotated and annotated resources.** A large lexicon is learned without labels, then used as features in a classifier trained on a small annotated set. This suits low-resource settings, such as Ukrainian, where labelled data is scarce.
- **Offer a high-precision lexicon-only mode.** A plain "any strong noun present" rule trades recall for 87% precision. This is useful when false positives are costly, or for pre-filtering candidates before annotation.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

## Cited by

Cited by

85 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 79 not yet.

- [SENTIWORDNET: A Publicly Available Lexical Resource for Opinion Mining](https://md.research.elamur.ai/free/papers/esuli-2006-sentiwordnet-publicly-available-lexical-resource/)
- [Finding Hedges by Chasing Weasels: Hedge Detection Using Wikipedia Tags and Shallow Linguistic Features](https://md.research.elamur.ai/free/papers/ganter-2009-finding-hedges-chasing-weasels-hedge/)
- [Weakly Supervised Learning for Hedge Classification in Scientific Literature](https://md.research.elamur.ai/free/papers/medlock-2007-weakly-supervised-learning-hedge-classification/)
- [Classifying Sentences as Speech Acts in Message Board Posts](https://md.research.elamur.ai/free/papers/qadir-2011-classifying-sentences-as-speech-acts/)
- [Hedge Classification in Biomedical Texts with a Weakly Supervised Selection of Keywords](https://md.research.elamur.ai/free/papers/szarvas-2008-hedge-classification-biomedical-texts-weakly/)
- [Unsupervised Discovery of Discourse Relations for Eliminating Intra-sentence Polarity Ambiguities](https://md.research.elamur.ai/free/papers/zhou-2011-unsupervised-discovery-discourse-relations-eliminating/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{riloff2003learning,
  title = {{Learning subjective nouns using extraction pattern bootstrapping}},
  author = {Riloff, Ellen and Wiebe, Janyce and Wilson, Theresa},
  booktitle = {Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003},
  year = {2003},
  doi = {10.3115/1119176.1119180},
  url = {https://doi.org/10.3115/1119176.1119180}
}
```

- [← previous Learning Non-Isomorphic Tree Mappings for Machine Translation](https://md.research.elamur.ai/free/papers/eisner-2003-learning-non-isomorphic-tree-mappings/)
- [next → Learning to Paraphrase: An Unsupervised Approach Using Multiple-Sequence Alignment](https://md.research.elamur.ai/free/papers/barzilay-2003-learning-paraphrase-unsupervised-approach-multiple/)
- [↑ back to papers of 2003](https://md.research.elamur.ai/free/papers/2003/)

## References

23 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. C. Baker, C. Fillmore, and J. Lowe. 1998. The berkeley framenet project. In Proceedings of the COLING-ACL. [find](https://scholar.google.com/scholar?q=C%2E+Baker%2C+C%2E+Fillmore%2C+and+J%2E+Lowe%2E+1998%2E+The+berkeley+framenet+project%2E+In+Proceedings+of+the+COLING%2DACL%2E) [our card](https://md.research.elamur.ai/free/papers/lex-1998-baker-berkeley-framenet/)
2. T. Ballmer and W. Brennenstuhl. 1981. Speech Act Classification: A Study in the Lexical Analysis of English Speech Activity Verbs. Springer-Verlag. [find](https://scholar.google.com/scholar?q=T%2E+Ballmer+and+W%2E+Brennenstuhl%2E+1981%2E+Speech+Act+Classification%3A+A+Study+in+the+Lexical+Analysis+of+English+Speech+Activity+Verbs%2E+Springer%2DVerlag%2E)
3. A. Banfield. 1982. Unspeakable Sentences. Routledge and Kegan Paul, Boston. [find](https://scholar.google.com/scholar?q=A%2E+Banfield%2E+1982%2E+Unspeakable+Sentences%2E+Routledge+and+Kegan+Paul%2C+Boston%2E)
4. S. Caraballo. 1999. Automatic Acquisition of a Hypernym-Labeled Noun Hierarchy from Text. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, pages 120–126. [find](https://scholar.google.com/scholar?q=S%2E+Caraballo%2E+1999%2E+Automatic+Acquisition+of+a+Hypernym%2DLabeled+Noun+Hierarchy+from+Text%2E+In+Proceedings+of+the+37th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+120%E2%80%93126%2E)
5. C. Cardie. 1997. Empirical Methods in Information Extraction. AI Magazine, 18(4):65–79. [find](https://scholar.google.com/scholar?q=C%2E+Cardie%2E+1997%2E+Empirical+Methods+in+Information+Extraction%2E+AI+Magazine%2C+18%284%29%3A65%E2%80%9379%2E)
6. V. Hatzivassiloglou and K. McKeown. 1997. Predicting the semantic orientation of adjectives. In ACL-EACL 1997. [find](https://scholar.google.com/scholar?q=V%2E+Hatzivassiloglou+and+K%2E+McKeown%2E+1997%2E+Predicting+the+semantic+orientation+of+adjectives%2E+In+ACL%2DEACL+1997%2E)
7. M. Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In Proc. of the 14th International Conference on Computational Linguistics. [find](https://scholar.google.com/scholar?q=M%2E+Hearst%2E+1992%2E+Automatic+acquisition+of+hyponyms+from+large+text+corpora%2E+In+Proc%2E+of+the+14th+International+Conference+on+Computational+Linguistics%2E)
8. J. Karlgren and D. Cutting. 1994. Recognizing text genres with simple metrics using discriminant analysis. In COLING-94. [find](https://scholar.google.com/scholar?q=J%2E+Karlgren+and+D%2E+Cutting%2E+1994%2E+Recognizing+text+genres+with+simple+metrics+using+discriminant+analysis%2E+In+COLING%2D94%2E)
9. B. Kessler, G. Nunberg, and H. Schutze. 1997. Automatic detection of text genre. In Proc. ACL-EACL-97. [find](https://scholar.google.com/scholar?q=B%2E+Kessler%2C+G%2E+Nunberg%2C+and+H%2E+Schutze%2E+1997%2E+Automatic+detection+of+text+genre%2E+In+Proc%2E+ACL%2DEACL%2D97%2E)
10. Beth Levin. 1993. English Verb Classes and Alternations: A Preliminary Investigation. University of Chicago Press, Chicago. [find](https://scholar.google.com/scholar?q=Beth+Levin%2E+1993%2E+English+Verb+Classes+and+Alternations%3A+A+Preliminary+Investigation%2E+University+of+Chicago+Press%2C+Chicago%2E)
11. B. Pang, L. Lee, and S. Vaithyanathan. 2002. Thumbs up? sentiment classification using machine learning techniques. In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=B%2E+Pang%2C+L%2E+Lee%2C+and+S%2E+Vaithyanathan%2E+2002%2E+Thumbs+up%3F+sentiment+classification+using+machine+learning+techniques%2E+In+Proceedings+of+the+2002+Conference+on+Empirical+Methods+in+Natural+Language+Proce)
12. R. Quirk, S. Greenbaum, G. Leech, and J. Svartvik. 1985. A Comprehensive Grammar of the English Language. Longman, New York. [find](https://scholar.google.com/scholar?q=R%2E+Quirk%2C+S%2E+Greenbaum%2C+G%2E+Leech%2C+and+J%2E+Svartvik%2E+1985%2E+A+Comprehensive+Grammar+of+the+English+Language%2E+Longman%2C+New+York%2E)
13. E. Riloff and R. Jones. 1999. Learning Dictionaries for Information Extraction by Multi-Level Bootstrapping. In Proceedings of the 16th National Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=E%2E+Riloff+and+R%2E+Jones%2E+1999%2E+Learning+Dictionaries+for+Information+Extraction+by+Multi%2DLevel+Bootstrapping%2E+In+Proceedings+of+the+16th+National+Conference+on+Artificial+Intelligence%2E)
14. E. Riloff and J. Shepherd. 1997. A Corpus-Based Approach for Building Semantic Lexicons. In Proceedings of the Second Conference on Empirical Methods in Natural Language Processing, pages 117–124. [find](https://scholar.google.com/scholar?q=E%2E+Riloff+and+J%2E+Shepherd%2E+1997%2E+A+Corpus%2DBased+Approach+for+Building+Semantic+Lexicons%2E+In+Proceedings+of+the+Second+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+117%E2%80%93124%2E)
15. B. Roark and E. Charniak. 1998. Noun-phrase Co-occurrence Statistics for Semi-automatic Semantic Lexicon Construction. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics, pages 1110–1116. [find](https://scholar.google.com/scholar?q=B%2E+Roark+and+E%2E+Charniak%2E+1998%2E+Noun%2Dphrase+Co%2Doccurrence+Statistics+for+Semi%2Dautomatic+Semantic+Lexicon+Construction%2E+In+Proceedings+of+the+36th+Annual+Meeting+of+the+Association+for+Computational+Li)
16. E. Spertus. 1997. Smokey: Automatic recognition of hostile messages. In Proc. IAAI. [find](https://scholar.google.com/scholar?q=E%2E+Spertus%2E+1997%2E+Smokey%3A+Automatic+recognition+of+hostile+messages%2E+In+Proc%2E+IAAI%2E)
17. M. Thelen and E. Riloff. 2002. A Bootstrapping Method for Learning Semantic Lexicons Using Extraction Pa ttern Contexts. In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=M%2E+Thelen+and+E%2E+Riloff%2E+2002%2E+A+Bootstrapping+Method+for+Learning+Semantic+Lexicons+Using+Extraction+Pa+ttern+Contexts%2E+In+Proceedings+of+the+2002+Conference+on+Empirical+Methods+in+Natural+Language+)
18. R. Tong. 2001. An operational system for detecting and tracking opinions in on-line discussion. In SIGIR 2001 Workshop on Operational Text Classification. [find](https://scholar.google.com/scholar?q=R%2E+Tong%2E+2001%2E+An+operational+system+for+detecting+and+tracking+opinions+in+on%2Dline+discussion%2E+In+SIGIR+2001+Workshop+on+Operational+Text+Classification%2E)
19. P. Turney. 2002. Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=P%2E+Turney%2E+2002%2E+Thumbs+Up+or+Thumbs+Down%3F+Semantic+Orientation+Applied+to+Unsupervised+Classification+of+Reviews%2E+In+Proceedings+of+the+40th+Annual+Meeting+of+the+Association+for+Computational+Lingui)
20. J. Wiebe, R. Bruce, and T. O’Hara. 1999. Development and use of a gold standard data set for subjectivity classifications. In Proc. 37th Annual Meeting of the Assoc. for Computational Linguistics (ACL-99). [find](https://scholar.google.com/scholar?q=J%2E+Wiebe%2C+R%2E+Bruce%2C+and+T%2E+O%E2%80%99Hara%2E+1999%2E+Development+and+use+of+a+gold+standard+data+set+for+subjectivity+classifications%2E+In+Proc%2E+37th+Annual+Meeting+of+the+Assoc%2E+for+Computational+Linguistics+%28ACL)
21. J. Wiebe, T. Wilson, and M. Bell. 2001. Identifying collocations for recognizing opinions. In Proc. ACL-01 Workshop on Collocation: Computational Extraction, Analysis, and Exploitation, July. [find](https://scholar.google.com/scholar?q=J%2E+Wiebe%2C+T%2E+Wilson%2C+and+M%2E+Bell%2E+2001%2E+Identifying+collocations+for+recognizing+opinions%2E+In+Proc%2E+ACL%2D01+Workshop+on+Collocation%3A+Computational+Extraction%2C+Analysis%2C+and+Exploitation%2C+July%2E)
22. J. Wiebe. 1990. Recognizing Subjective Sentences: A Computational Investigation of Narrative Text. Ph.D. thesis, State University of New York at Buffalo. [find](https://scholar.google.com/scholar?q=J%2E+Wiebe%2E+1990%2E+Recognizing+Subjective+Sentences%3A+A+Computational+Investigation+of+Narrative+Text%2E+Ph%2ED%2E+thesis%2C+State+University+of+New+York+at+Buffalo%2E)
23. J. Wiebe. 2000. Learning subjective adjectives from corpora. In 17th National Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=J%2E+Wiebe%2E+2000%2E+Learning+subjective+adjectives+from+corpora%2E+In+17th+National+Conference+on+Artificial+Intelligence%2E)