---
title: "A Corpus Study of Evaluative and Speculative Language (2001)"
description: "The paper looks at how to find subjective language in text. 27 references, cited by 12 papers here."
authors: ["Janyce Wiebe", "Rebecca Bruce", "Matthew Bell", "Melanie Martin", "Theresa Wilson"]
year: 2001
venue: "Proceedings of the Second SIGdial Workshop on Discourse and Dialogue"
source: "https://aclanthology.org/W01-1626/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wiebe-2001-corpus-study-evaluative-speculative-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wiebe-2001-corpus-study-evaluative-speculative-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2001](https://md.research.elamur.ai/free/en/papers/2001/) / A Corpus Study of Evaluative and Speculative Language

# A Corpus Study of Evaluative and Speculative Language

**Authors:** [Janyce Wiebe](https://md.research.elamur.ai/free/en/authors/janyce-wiebe/), [Rebecca Bruce](https://md.research.elamur.ai/free/en/authors/rebecca-bruce/), [Matthew Bell](https://md.research.elamur.ai/free/en/authors/matthew-bell/), [Melanie Martin](https://md.research.elamur.ai/free/en/authors/melanie-martin/) and [Theresa Wilson](https://md.research.elamur.ai/free/en/authors/theresa-wilson/) · **Year:** 2001 · **Venue:** [Proceedings of the Second SIGdial Workshop on Discourse and Dialogue](https://md.research.elamur.ai/free/en/venues/sigdial/) ([SIGDIAL 2001](https://md.research.elamur.ai/free/en/venues/sigdial/2001/)) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W01-1626/) · [PDF](https://aclanthology.org/W01-1626.pdf) · [OpenAlex](https://openalex.org/W2074523230)

**Citations:** 59 citations (OpenAlex, 2026-10-14) · [12](https://md.research.elamur.ai/free/en/papers/wiebe-2001-corpus-study-evaluative-speculative-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper looks at how to find subjective language in text. This covers evaluation (emotions, judgements, opinions) and speculation (wording that removes the presupposition that an event occurred or a state holds). Standard lexical resources do not mark these clues, so they must be learned from corpora. The authors annotate subjectivity at three granularities:
- individual expressions inside sentences,
- whole sentences,
- whole documents, using editorials, reviews and hostile "flame" messages from Usenet and the Wall Street Journal.

They measure annotator agreement at each level and use the expression-level tags to describe what subjective language looks like. They then mine potential subjective elements (PSEs) from cheap, pre-existing document labels. Adjectives and verbs are clustered by distributional similarity, and the clustering parameters are tuned with a train–validate–test protocol.

The main results:
- Document-level subjectivity is annotated reliably. Expression-level tagging is harder but feasible.
- Words occurring only once are themselves a subjectivity clue.
- Mined PSEs raise precision over baseline consistently across datasets.

This matters for extraction, retrieval and review mining, which must tell opinion from presented fact.

## Key points
- Flame corpus: 1140 Usenet messages, balanced across the alt, sci, comp and rec hierarchies, split into 778 training and 362 test messages.
- Earlier sentence-level work on the Wall Street Journal Treebank Corpus:
  - Three judges reached an average pairwise Cohen's κ of .69 after two rounds.
  - Consensus tags were derived with EM (Dawid and Skene style).
  - A probabilistic classifier scored 72.17% in 10-fold cross validation, more than 20 points above the majority-class baseline.
- Opinion-piece labels are "impure", and the authors work out how much this limits precision:
  - Assume 15% of sentences sit in opinion pieces, with 70% of those subjective versus 44% elsewhere.
  - Then the expected precision of subjective sentences for the opinion class is only about .22.
- PSE mining procedure:
  - Lin's (1998) clustering, seeded with all training adjectives or verbs.
  - Cluster size C and filtering threshold FT are chosen on a validation set, keeping only sets with precision ≥ .25 and frequency ≥ 100.
  - Training and validation use existing WSJ opinion labels; testing uses manual annotations.
  - Test sets are W9-10, W9-22, W9-33 and W9-04.
- Precision gains over baseline:
  - adjectives +.09 to +.21,
  - verbs +.07 to +.16,
  - unique words +.06 to +.10.
  - All three feature types rise and fall together across datasets.
  - Agreement is higher for adjectives than for determiners.

## Interesting ideas
- Every subjective element carries a source (writer or a mentioned person) and a target, so a reported complaint is attributed to the complainer instead of being stored as fact, unlike a neutral "a judge said" report.
- The paper separates a *potential* subjective element (a lexicon entry such as *eroding*) from an actual subjective use, which depends on context.
- Words that appear only once in a text are treated as a positive clue for subjectivity, rather than discarded as noise.
- With noisy labels, the authors first compute the precision that is achievable at all from the class mixture, then judge features by their gain over a word-frequency baseline.
- EM-based consensus (Dawid–Skene style) merges several annotators into corrected tags, weighting each by estimated reliability instead of taking a simple vote.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (frames): prag could attach source/target slots to evaluative verbs like *complain*, so quoted content under them is marked as an attributed attitude rather than an asserted event.
- **I2** (world): the knowledge layer could store words as potentially evaluative, and the world builder would mark a state as opinion only when the context in the tree confirms it, keeping the decision explainable.
- **I3** (tokens): crate en could flag hapax tokens per document, giving prag's stance detection a cheap, interpretable signal.
- **I4** (learn): evaluation of silver-trained components could report an expected ceiling and baseline-relative gains, so impure supervision does not make good rules look bad.
- **I5** (learn): silver data from multiple witness parsers could be combined the same way, learning per-parser reliability for each relation type.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Pittsburgh](https://md.research.elamur.ai/free/en/universities/university-of-pittsburgh/) — University, Pittsburgh, United States · 59 works · 22,324 citations
- [New Mexico State University](https://md.research.elamur.ai/free/en/universities/new-mexico-state-university/) — University, Las Cruces, United States · 7 works · 5,672 citations
- [University of North Carolina at Asheville](https://md.research.elamur.ai/free/en/universities/university-of-north-carolina-at-asheville/) — University, Asheville, United States · 4 works · 1,502 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [📚 Literature and Literary Theory](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/literature-and-literary-theory/) · 519 works

## Cite

Cite as: Janyce Wiebe, Rebecca Bruce, Matthew Bell, Melanie Martin and Theresa Wilson. 2001. A Corpus Study of Evaluative and Speculative Language. Proceedings of the Second SIGdial Workshop on Discourse and Dialogue.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wiebe2001corpus,
  title = {{A Corpus Study of Evaluative and Speculative Language}},
  author = {Janyce Wiebe, Rebecca Bruce, Matthew Bell, Melanie Martin and Theresa Wilson},
  booktitle = {Proceedings of the Second SIGdial Workshop on Discourse and Dialogue},
  year = {2001},
  doi = {10.3115/1118078.1118104},
  url = {https://doi.org/10.3115/1118078.1118104}
}
```

- [← previous An Algorithm for Anaphora Resolution in Spanish Texts](https://md.research.elamur.ai/free/en/papers/palomar-2001-algorithm-anaphora-resolution-spanish-texts/)
- [next → DATE: A Dialogue Act Tagging Scheme for Evaluation of Spoken Dialogue Systems](https://md.research.elamur.ai/free/en/papers/walker-2001-date-dialogue-act-tagging-scheme/)
- [↑ back to papers of 2001](https://md.research.elamur.ai/free/en/papers/2001/)

## References

27 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. C. Aone, M. Ramos-Santacruz, and W. Niehaus. 2000. Assentor: An nlp-based solution to e-mail monitoring. In Proc. IAAI-2000 , pages 945{ 950. [find](https://scholar.google.com/scholar?q=C%2E+Aone%2C+M%2E+Ramos%2DSantacruz%2C+and+W%2E+Niehaus%2E+2000%2E+Assentor%3A+An+nlp%2Dbased+solution+to+e%2Dmail+monitoring%2E+In+Proc%2E+IAAI%2D2000+%2C+pages+945%7B+950%2E)
2. A. Ban eld. 1982. Unspeakable Sentences . Routledge and Kegan Paul, Boston. [find](https://scholar.google.com/scholar?q=A%2E+Ban+eld%2E+1982%2E+Unspeakable+Sentences+%2E+Routledge+and+Kegan+Paul%2C+Boston%2E)
3. R. Barzilay, M. Collins, J. Hirschberg, and S. Whittaker. 2000. The rules behind roles: Identifying speaker role in radio broadcasts. In Proc. AAAI . [↗](https://doi.org/10.7916/d8pc39q9)
4. E. Brill. 1992. [A simple rule-based part of speech tagger](https://md.research.elamur.ai/free/en/papers/brill-1992-simple-rule-based-part-speech/). In Proc. of the 3rd Conference on Applied Natural Language Processing (ANLP-92) , pages 152{155.
5. R. Bruce and J. Wiebe. 1999. Recognizing subjectivity: A case study of manual tagging. Natural Language Engineering , 5(2). [find](https://scholar.google.com/scholar?q=R%2E+Bruce+and+J%2E+Wiebe%2E+1999%2E+Recognizing+subjectivity%3A+A+case+study+of+manual+tagging%2E+Natural+Language+Engineering+%2C+5%282%29%2E)
6. J. Cohen. 1960. A coecient of agreement for nominal scales. Educational and Psychological Meas. , 20:37{46. [find](https://scholar.google.com/scholar?q=J%2E+Cohen%2E+1960%2E+A+coe%0Ecient+of+agreement+for+nominal+scales%2E+Educational+and+Psychological+Meas%2E+%2C+20%3A37%7B46%2E)
7. A. P. Dawid and A. M. Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics , 28:20{28. [↗](https://doi.org/10.2307/2346806)
8. M. Fludernik. 1993. The Fictions of Language and the Languages of Fiction . Routledge, London. [find](https://scholar.google.com/scholar?q=M%2E+Fludernik%2E+1993%2E+The+Fictions+of+Language+and+the+Languages+of+Fiction+%2E+Routledge%2C+London%2E)
9. L. Goodman. 1974. Exploratory latent structure analysis using both identi able and unidenti - able models. Biometrika , 61:2:215{231. [find](https://scholar.google.com/scholar?q=L%2E+Goodman%2E+1974%2E+Exploratory+latent+structure+analysis+using+both+identi+able+and+unidenti+%2D+able+models%2E+Biometrika+%2C+61%3A2%3A215%7B231%2E)
10. E. Hovy. 1987. Generating Natural Language under Pragmatic Constraints . Ph.D. thesis, Yale University. [find](https://scholar.google.com/scholar?q=E%2E+Hovy%2E+1987%2E+Generating+Natural+Language+under+Pragmatic+Constraints+%2E+Ph%2ED%2E+thesis%2C+Yale+University%2E)
11. D. Karp, Y. Schabes, M. Zaidel, and D. Egedi. 1992. A freely available wide coverage morphological analyzer for English. In Proc. of the 14th International Conference on Computational Linguistics (COLING-92) . [find](https://scholar.google.com/scholar?q=D%2E+Karp%2C+Y%2E+Schabes%2C+M%2E+Zaidel%2C+and+D%2E+Egedi%2E+1992%2E+A+freely+available+wide+coverage+morphological+analyzer+for+English%2E+In+Proc%2E+of+the+14th+International+Conference+on+Computational+Linguistics+%28COL)
12. D. Kaufer. 2000. Flaming: A White Paper . www.eudora.com. [find](https://scholar.google.com/scholar?q=D%2E+Kaufer%2E+2000%2E+Flaming%3A+A+White+Paper+%2E+www%2Eeudora%2Ecom%2E)
13. B. Kessler, G. Nunberg, and H. Schutze. 1997. [Automatic detection of text genre](https://md.research.elamur.ai/free/en/papers/kessler-1997-automatic-detection-text-genre/). In Proc. ACL-EACL-97 . [↗](https://doi.org/10.48550/arxiv.cmp-lg/9707002)
14. D. Lin. 1998. [Automatic retrieval and clustering of similar words](https://md.research.elamur.ai/free/en/papers/lin-1998-automatic-retrieval-clustering-similar-words/). In Proc. COLING-ACL '98 , pages 768{773.
15. Diane J. Litman and R. J. Passonneau. 1995. Combining multiple knowledge sources for discourse segmentation. In Proc. 33rd Annual Meeting of the Association for Computational Linguistics (ACL-95) , pages 108{115. Association for Computational Linguistics, june. [find](https://scholar.google.com/scholar?q=Diane+J%2E+Litman+and+R%2E+J%2E+Passonneau%2E+1995%2E+Combining+multiple+knowledge+sources+for+discourse+segmentation%2E+In+Proc%2E+33rd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28ACL%2D95%29+%2C+pa)
16. D. Marcu, M. Romera, and E. Amorrortu. 1999. Experiments in constructing a corpus of discourse trees: Problems, annotation choices, issues. In The Workshop on Levels of Representation in Discourse , pages 71{78. [find](https://scholar.google.com/scholar?q=D%2E+Marcu%2C+M%2E+Romera%2C+and+E%2E+Amorrortu%2E+1999%2E+Experiments+in+constructing+a+corpus+of+discourse+trees%3A+Problems%2C+annotation+choices%2C+issues%2E+In+The+Workshop+on+Levels+of+Representation+in+Discourse+%2C+p)
17. M. Marcus, Santorini, B., and M. Marcinkiewicz. 1993. [Building a large annotated corpus of English: The penn treebank](https://md.research.elamur.ai/free/en/papers/marcus-1993-building-large-annotated-corpus-english/). Computational Linguistics , 19(2):313{330. [↗](https://doi.org/10.21236/ada273556)
18. G. Nunberg, I. Sag, and T. Wasow. 1994. Idioms. Language , 70:491{538. [find](https://scholar.google.com/scholar?q=G%2E+Nunberg%2C+I%2E+Sag%2C+and+T%2E+Wasow%2E+1994%2E+Idioms%2E+Language+%2C+70%3A491%7B538%2E)
19. W. Sack. 1995. Representing and recognizing point of view. In Proc. AAAI Fall Symposium on AI Applications in Knowledge Navigation and Retrieval . [find](https://scholar.google.com/scholar?q=W%2E+Sack%2E+1995%2E+Representing+and+recognizing+point+of+view%2E+In+Proc%2E+AAAI+Fall+Symposium+on+AI+Applications+in+Knowledge+Navigation+and+Retrieval+%2E)
20. E. Spertus. 1997. Smokey: Automatic recognition of hostile messages. In Proc. IAAI . [find](https://scholar.google.com/scholar?q=E%2E+Spertus%2E+1997%2E+Smokey%3A+Automatic+recognition+of+hostile+messages%2E+In+Proc%2E+IAAI+%2E)
21. D. Stein and S. Wright, editors. 1995. Subjectivity and Subjectivisation . Cambridge University Press, Cambridge. [find](https://scholar.google.com/scholar?q=D%2E+Stein+and+S%2E+Wright%2C+editors%2E+1995%2E+Subjectivity+and+Subjectivisation+%2E+Cambridge+University+Press%2C+Cambridge%2E)
22. L. Terveen, W. Hill, B. Amento, D. McDonald, and J. Creter. 1997. Building task-speci c interfaces to high volume conversational data. In Proc. CHI 97 , pages 226{233. [find](https://scholar.google.com/scholar?q=L%2E+Terveen%2C+W%2E+Hill%2C+B%2E+Amento%2C+D%2E+McDonald%2C+and+J%2E+Creter%2E+1997%2E+Building+task%2Dspeci+c+interfaces+to+high+volume+conversational+data%2E+In+Proc%2E+CHI+97+%2C+pages+226%7B233%2E)
23. S. Teufel and M. Moens. 2000. What's yours and what's mine: Determining intellectual attribution in scienti c texts. In Proc. Joint SIGDAT Converence on EMNLP and VLC . [find](https://scholar.google.com/scholar?q=S%2E+Teufel+and+M%2E+Moens%2E+2000%2E+What%27s+yours+and+what%27s+mine%3A+Determining+intellectual+attribution+in+scienti+c+texts%2E+In+Proc%2E+Joint+SIGDAT+Converence+on+EMNLP+and+VLC+%2E)
24. J. Wiebe, K. McKeever, and R. Bruce. 1998. Mapping collocational properties into machine learning features. In Proc. 6th Workshop on Very Large Corpora (WVLC-98) , pages 225{ 233, Montreal, Canada, August. ACL SIGDAT. [find](https://scholar.google.com/scholar?q=J%2E+Wiebe%2C+K%2E+McKeever%2C+and+R%2E+Bruce%2E+1998%2E+Mapping+collocational+properties+into+machine+learning+features%2E+In+Proc%2E+6th+Workshop+on+Very+Large+Corpora+%28WVLC%2D98%29+%2C+pages+225%7B+233%2C+Montreal%2C+Canada%2C+Au)
25. J. Wiebe, R. Bruce, and T. O'Hara. 1999. Development and use of a gold standard data set for subjectivity classi cations. In Proc. 37th Annual Meeting of the Assoc. for Computational Linguistics (ACL-99) , pages 246{253, University of Maryland, June. ACL. [find](https://scholar.google.com/scholar?q=J%2E+Wiebe%2C+R%2E+Bruce%2C+and+T%2E+O%27Hara%2E+1999%2E+Development+and+use+of+a+gold+standard+data+set+for+subjectivity+classi+cations%2E+In+Proc%2E+37th+Annual+Meeting+of+the+Assoc%2E+for+Computational+Linguistics+%28ACL%2D)
26. J. Wiebe. 1994. [Tracking point of view in narrative](https://md.research.elamur.ai/free/en/papers/wiebe-1994-tracking-point-view-narrative/). Computational Linguistics , 20(2):233{287. [↗](https://doi.org/10.5555/972525.972529)
27. J. Wiebe. 2000. Learning subjective adjectives from corpora. In 17th National Conference on Arti cial Intelligence (AAAI-2000) . [find](https://scholar.google.com/scholar?q=J%2E+Wiebe%2E+2000%2E+Learning+subjective+adjectives+from+corpora%2E+In+17th+National+Conference+on+Arti+cial+Intelligence+%28AAAI%2D2000%29+%2E)

*Updated 1 November 2026*