---
title: "The Benefits of a Model of Annotation — Passonneau 2014"
description: "Annotated corpora in computational linguistics are usually judged by whether a few trained annotators agree, measured by raw agreement or by…"
authors: ["Rebecca J. Passonneau", "Bob Carpenter"]
year: 2014
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/Q14-1025/"
doi: "10.1162/tacl_a_00185"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/passonneau-2014-benefits-model-annotation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/passonneau-2014-benefits-model-annotation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2014](https://md.research.elamur.ai/free/en/papers/2014/page/2/) / The Benefits of a Model of Annotation

# The Benefits of a Model of Annotation

**Authors:** Rebecca J. Passonneau, Bob Carpenter · **Year:** 2014 · **Venue:** Transactions of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/Q14-1025/) · [PDF](https://aclanthology.org/Q14-1025.pdf) · [DOI 10.1162/tacl\_a\_00185](https://doi.org/10.1162/tacl_a_00185) · [OpenAlex](https://openalex.org/W2251089262)

**Citations:** 151 citations (openalex-snapshot · 2026-10-14) · [87](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Annotated corpora in computational linguistics are usually judged by whether a few trained annotators agree, measured by raw agreement or by chance-adjusted coefficients such as Cohen's κ or Krippendorff's α. Passonneau and Carpenter argue that these numbers reveal nothing about whether individual labels are correct, and that they can mislead. Accurate annotators can score low on skewed data, and annotators who share biases or errors can score high. As an alternative, the authors apply the probabilistic model of Dawid and Skene (1979). It jointly estimates how often each category occurs and how accurate and biased each annotator is for each category, which yields a probability for every possible label of every item. The case study revisits the MASC word sense sentence corpus, where trained annotators' agreement on WordNet senses ranged from very high to chance level. For 45 randomly chosen MASC words, crowdworkers labeled each item and the model was fitted to their labels. Most items received highly certain labels, even for words with poor expert agreement, at under half the cost.

## Key points
- MASC's word sense sentence corpus covers 116 words with roughly 1,000 sentences each. The crowdsourced re-annotation covered a random subset of 45 of these words, with up to 25 Mechanical Turk annotators per item.
- The authors give a worked example of the κ problem on skewed data. With 95% prevalence of one class, two annotators who are each 90% accurate get a chance-adjusted agreement of about −0.053.
- Agreement statistics are seldom reported with confidence intervals. Even if the true labels were known, scoring 80 correct out of 100 gives a 95% interval of roughly 74–86%. The authors recommend bootstrapping to compute such intervals.
- The value of collecting more labels is quantified with information gain and mutual information. Results are illustrated for add.v, ask.v, date.n and help.v, comparing trained annotators with Turkers.
- Related work includes Snow et al. (2008), Hovy et al. (2013) and the Troia project. The authors suggest that low-confidence words such as meet (verb) and date (noun) could be improved by revising their sense inventories.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

87 papers of our library cite it (of the 140884 whose bibliographies we read); 13 of them have a page here, 74 not yet.

- [We Need to Consider Disagreement in Evaluation](https://md.research.elamur.ai/free/en/papers/basile-2021-we-need-consider-disagreement-evaluation/)
- [The Media Frames Corpus: Annotations of Frames Across Issues](https://md.research.elamur.ai/free/en/papers/card-2015-media-frames-corpus-annotations-frames/)
- [Which argument is more convincing? Analyzing and predicting convincingness of Web arguments using bidirectional LSTM](https://md.research.elamur.ai/free/en/papers/habernal-2016-which-argument-is-more-convincing/)
- [The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrants](https://md.research.elamur.ai/free/en/papers/habernal-2018-argument-reasoning-comprehension-task-identification/)
- [Onomasiological Sense Alignment Across Dialect Dictionaries. A Taxonomy-Constrained LLM Classification](https://md.research.elamur.ai/free/en/papers/mederake-2026-onomasiological-sense-alignment-across-dialect/)
- [A Probabilistic Annotation Model for Crowdsourcing Coreference](https://md.research.elamur.ai/free/en/papers/paun-2018-probabilistic-annotation-model-crowdsourcing-coreference/)
- [A Crowdsourced Corpus of Multiple Judgments and Disagreement on Anaphoric Interpretation](https://md.research.elamur.ai/free/en/papers/poesio-2019-crowdsourced-corpus-multiple-judgments-disagreement/)
- [Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design](https://md.research.elamur.ai/free/en/papers/pyatkin-2023-design-choices-crowdsourcing-implicit-discourse/)
- [Recovering discourse relations: Varying influence of discourse adverbials](https://md.research.elamur.ai/free/en/papers/rohde-2015-recovering-discourse-relations-varying-influence/)
- [DiscoGeM: A Crowdsourced Corpus of Genre-Mixed Implicit Discourse Relations](https://md.research.elamur.ai/free/en/papers/scholman-2022-discogem-crowdsourced-corpus-genre-mixed/)
- [Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview](https://md.research.elamur.ai/free/en/papers/shah-2019-predictive-biases-natural-language-processing/)
- [Expert, Crowdsourced, and Machine Assessment of Suicide Risk via Online Postings](https://md.research.elamur.ai/free/en/papers/shing-2018-expert-crowdsourced-machine-assessment-suicide/)
- [Disentangling Annotator Skill from Verifier Strictness in Cross-Verified Dialogue Annotation](https://md.research.elamur.ai/free/en/papers/tao-2026-disentangling-annotator-skill-verifier-strictness/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{passonneau2014benefits,
  title = {{The Benefits of a Model of Annotation}},
  author = {Passonneau, Rebecca J. and Carpenter, Bob},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2014},
  doi = {10.1162/tacl\_a\_00185},
  url = {https://doi.org/10.1162/tacl_a_00185}
}
```

- [← previous The Ashkar-Gilson Manuscript: Remnant of a Proto-Masoretic Model Scroll of the Torah](https://md.research.elamur.ai/free/en/papers/hbo-2014-sanders-ashkar-gilson-manuscript-remnant-proto/)
- [next → The Biblical Hebrew Feminine Singular Qal Participle: A Historical Reconstruction](https://md.research.elamur.ai/free/en/papers/hbo-2014-price-biblical-hebrew-feminine-singular-qal/)
- [↑ back to papers of 2014](https://md.research.elamur.ai/free/en/papers/2014/page/2/)

## References

35 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Paul S. Albert and Lori E. Dodd. 2008. On estimating diagnostic accuracy from studies with multiple raters and partial gold standard evaluation. Journal of the American Statistical Association, 103(481):61–73. [find](https://scholar.google.com/scholar?q=Paul+S%2E+Albert+and+Lori+E%2E+Dodd%2E+2008%2E+On+estimating+diagnostic+accuracy+from+studies+with+multiple+raters+and+partial+gold+standard+evaluation%2E+Journal+of+the+American+Statistical+Association%2C+103%2848)
2. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555–596. [find](https://scholar.google.com/scholar?q=Ron+Artstein+and+Massimo+Poesio%2E+2008%2E+Inter%2Dcoder+agreement+for+computational+linguistics%2E+Computational+Linguistics%2C+34%284%29%3A555%E2%80%93596%2E)
3. Rebecca F. Bruce and Janyce M. Wiebe. 1998. Word-sense distinguishability and inter-coder agreement. In Proceedings of Empirical Methods in Natural Language Processing, pages 53–60. [find](https://scholar.google.com/scholar?q=Rebecca+F%2E+Bruce+and+Janyce+M%2E+Wiebe%2E+1998%2E+Word%2Dsense+distinguishability+and+inter%2Dcoder+agreement%2E+In+Proceedings+of+Empirical+Methods+in+Natural+Language+Processing%2C+pages+53%E2%80%9360%2E)
4. Rebecca F. Bruce and Janyce M. Wiebe. 1999. Recognizing subjectivity: a case study of manual tagging. Natural Language Engineering, 1(1):1–16. [find](https://scholar.google.com/scholar?q=Rebecca+F%2E+Bruce+and+Janyce+M%2E+Wiebe%2E+1999%2E+Recognizing+subjectivity%3A+a+case+study+of+manual+tagging%2E+Natural+Language+Engineering%2C+1%281%29%3A1%E2%80%9316%2E)
5. Chris Callison-Burch and Mark Dredze. 2010. Creating speech and language data with Amazon’s Mechanical Turk. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, pages 1–12. [find](https://scholar.google.com/scholar?q=Chris+Callison%2DBurch+and+Mark+Dredze%2E+2010%2E+Creating+speech+and+language+data+with+Amazon%E2%80%99s+Mechanical+Turk%2E+In+Proceedings+of+the+NAACL+HLT+2010+Workshop+on+Creating+Speech+and+Language+Data+with+Ama)
6. Bob Carpenter. 2008. Multilevel Bayesian models of categorical data annotation. Technical report, Alias-i, Inc. [find](https://scholar.google.com/scholar?q=Bob+Carpenter%2E+2008%2E+Multilevel+Bayesian+models+of+categorical+data+annotation%2E+Technical+report%2C+Alias%2Di%2C+Inc%2E)
7. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20:37–46. [find](https://scholar.google.com/scholar?q=Jacob+Cohen%2E+1960%2E+A+coefficient+of+agreement+for+nominal+scales%2E+Educational+and+Psychological+Measurement%2C+20%3A37%E2%80%9346%2E)
8. Thomas M. Cover and Joy A. Thomas. 1991. Elements of Information Theory. Wiley-Interscience, New York, NY, USA. [find](https://scholar.google.com/scholar?q=Thomas+M%2E+Cover+and+Joy+A%2E+Thomas%2E+1991%2E+Elements+of+Information+Theory%2E+Wiley%2DInterscience%2C+New+York%2C+NY%2C+USA%2E)
9. A. P. Dawid and A. M. Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics), 28(1):20–28. [find](https://scholar.google.com/scholar?q=A%2E+P%2E+Dawid+and+A%2E+M%2E+Skene%2E+1979%2E+Maximum+likelihood+estimation+of+observer+error%2Drates+using+the+EM+algorithm%2E+Journal+of+the+Royal+Statistical+Society%2E+Series+C+%28Applied+Statistics%29%2C+28%281%29%3A20%E2%80%9328%2E)
10. Barbara di Eugenio and Michael Glass. 2004. The Kappa statistic: A second look. Computational Linguistics, 30(1):95–101. [find](https://scholar.google.com/scholar?q=Barbara+di+Eugenio+and+Michael+Glass%2E+2004%2E+The+Kappa+statistic%3A+A+second+look%2E+Computational+Linguistics%2C+30%281%29%3A95%E2%80%93101%2E)
11. Barbara di Eugenio. 2000. On the usage of Kappa to evaluate agreement on coding tasks. In Proceedings of the Second International Conference on Language Resources and Evaluation (LREC). [find](https://scholar.google.com/scholar?q=Barbara+di+Eugenio%2E+2000%2E+On+the+usage+of+Kappa+to+evaluate+agreement+on+coding+tasks%2E+In+Proceedings+of+the+Second+International+Conference+on+Language+Resources+and+Evaluation+%28LREC%29%2E)
12. B. Efron and R. Tibshirani. 1986. Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy. Statistical Science, 1(1):54–77. [find](https://scholar.google.com/scholar?q=B%2E+Efron+and+R%2E+Tibshirani%2E+1986%2E+Bootstrap+methods+for+standard+errors%2C+confidence+intervals%2C+and+other+measures+of+statistical+accuracy%2E+Statistical+Science%2C+1%281%29%3A54%E2%80%9377%2E)
13. Alvan R. Feinstein and Domenic V. Cicchetti. 1990. High agreement but low Kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6):543 – 549. [find](https://scholar.google.com/scholar?q=Alvan+R%2E+Feinstein+and+Domenic+V%2E+Cicchetti%2E+1990%2E+High+agreement+but+low+Kappa%3A+I%2E+The+problems+of+two+paradoxes%2E+Journal+of+Clinical+Epidemiology%2C+43%286%29%3A543+%E2%80%93+549%2E)
14. George Forman. 2003. An extensive empirical study of feature selection metrics for text classification. Journal of Machine Learning Research, 3:1289–1305. [find](https://scholar.google.com/scholar?q=George+Forman%2E+2003%2E+An+extensive+empirical+study+of+feature+selection+metrics+for+text+classification%2E+Journal+of+Machine+Learning+Research%2C+3%3A1289%E2%80%931305%2E)
15. Kilem Li Gwet. 2008. Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1):29–48. [find](https://scholar.google.com/scholar?q=Kilem+Li+Gwet%2E+2008%2E+Computing+inter%2Drater+reliability+and+its+variance+in+the+presence+of+high+agreement%2E+British+Journal+of+Mathematical+and+Statistical+Psychology%2C+61%281%29%3A29%E2%80%9348%2E)
16. J. W. Holley and J. P. Guildford. 1964. A note on the G index of agreement. Educational and Psychological Measurement, 24:749–753. [find](https://scholar.google.com/scholar?q=J%2E+W%2E+Holley+and+J%2E+P%2E+Guildford%2E+1964%2E+A+note+on+the+G+index+of+agreement%2E+Educational+and+Psychological+Measurement%2C+24%3A749%E2%80%93753%2E)
17. Dirk Hovy, Tayler Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom to trust with MACE. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1120–1130, Atlanta, Georgia, June. Ass… [find](https://scholar.google.com/scholar?q=Dirk+Hovy%2C+Tayler+Berg%2DKirkpatrick%2C+Ashish+Vaswani%2C+and+Eduard+Hovy%2E+2013%2E+Learning+whom+to+trust+with+MACE%2E+In+Proceedings+of+the+2013+Conference+of+the+North+American+Chapter+of+the+Association+for+) [our card](https://md.research.elamur.ai/free/en/papers/hovy-2013-learning-whom-trust-mace/)
18. Panagiotis G. Ipeirotis, Foster Provost, and Jing Wang. 2010. Quality management on Amazon Mechanical Turk. In Proceedings of the ACM SIGKDD Workshop on Human Computation, HCOMP 2010, pages 64–67, New York, NY, USA. [find](https://scholar.google.com/scholar?q=Panagiotis+G%2E+Ipeirotis%2C+Foster+Provost%2C+and+Jing+Wang%2E+2010%2E+Quality+management+on+Amazon+Mechanical+Turk%2E+In+Proceedings+of+the+ACM+SIGKDD+Workshop+on+Human+Computation%2C+HCOMP+2010%2C+pages+64%E2%80%9367%2C+New)
19. ACM. Beata Beigman Klebanov and Eyal Beigman. 2009. From annotator agreement to noise models. Computational Linguistics, 35(4):495–503. [find](https://scholar.google.com/scholar?q=ACM%2E+Beata+Beigman+Klebanov+and+Eyal+Beigman%2E+2009%2E+From+annotator+agreement+to+noise+models%2E+Computational+Linguistics%2C+35%284%29%3A495%E2%80%93503%2E)
20. Klaus Krippendorff. 1980. Content analysis: An introduction to its methodology. [find](https://scholar.google.com/scholar?q=Klaus+Krippendorff%2E+1980%2E+Content+analysis%3A+An+introduction+to+its+methodology%2E)
21. Sage Publications, Beverly Hills, CA. David J. C. MacKay. 1992. Information-based objective functions for active data selection. Neural Computation, 4:590–604. [find](https://scholar.google.com/scholar?q=Sage+Publications%2C+Beverly+Hills%2C+CA%2E+David+J%2E+C%2E+MacKay%2E+1992%2E+Information%2Dbased+objective+functions+for+active+data+selection%2E+Neural+Computation%2C+4%3A590%E2%80%93604%2E)
22. B. W. Matthews. 1975. Comparison of the predicted and observed secondary structure of t4 phage lysozyme. Biochimica et Biophysica Acta (BBA) - Protein Structure, 405(2):442–451. [find](https://scholar.google.com/scholar?q=B%2E+W%2E+Matthews%2E+1975%2E+Comparison+of+the+predicted+and+observed+secondary+structure+of+t4+phage+lysozyme%2E+Biochimica+et+Biophysica+Acta+%28BBA%29+%2D+Protein+Structure%2C+405%282%29%3A442%E2%80%93451%2E)
23. George A. Miller. 1995. A lexical database for English. Communications of the ACM, 38(11):39–41. [find](https://scholar.google.com/scholar?q=George+A%2E+Miller%2E+1995%2E+A+lexical+database+for+English%2E+Communications+of+the+ACM%2C+38%2811%29%3A39%E2%80%9341%2E)
24. Rebecca J. Passonneau, Collin F. Baker, Christiane Fellbaum, and Nancy Ide. 2012a. The MASC word sense corpus. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 3025–3030, Istanbul, Turkey. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Rebecca+J%2E+Passonneau%2C+Collin+F%2E+Baker%2C+Christiane+Fellbaum%2C+and+Nancy+Ide%2E+2012a%2E+The+MASC+word+sense+corpus%2E+In+Proceedings+of+the+Eighth+International+Conference+on+Language+Resources+and+Evaluatio)
25. Rebecca J. Passonneau, Vikas Bhardwaj, Ansaf Salleb-Aouissi, and Nancy Ide. 2012b. Multiplicity and word sense: Evaluating and learning from multiply labeled word sense annotations. Language Resources and Evaluation, 46(2):219–252. [find](https://scholar.google.com/scholar?q=Rebecca+J%2E+Passonneau%2C+Vikas+Bhardwaj%2C+Ansaf+Salleb%2DAouissi%2C+and+Nancy+Ide%2E+2012b%2E+Multiplicity+and+word+sense%3A+Evaluating+and+learning+from+multiply+labeled+word+sense+annotations%2E+Language+Resources)
26. Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. 2010. Learning from crowds. Journal of Machine Learning Research, 11:1297– 1322. [find](https://scholar.google.com/scholar?q=Vikas+C%2E+Raykar%2C+Shipeng+Yu%2C+Linda+H%2E+Zhao%2C+Gerardo+Hermosillo+Valadez%2C+Charles+Florin%2C+Luca+Bogoni%2C+and+Linda+Moy%2E+2010%2E+Learning+from+crowds%2E+Journal+of+Machine+Learning+Research%2C+11%3A1297%E2%80%93+1322%2E)
27. Simon Rogers, Mark Girolami, and Tamara Polajnar. 2010. Semi-parametric analysis of multi-rater data. Statistical Computing, 20:317–334. [find](https://scholar.google.com/scholar?q=Simon+Rogers%2C+Mark+Girolami%2C+and+Tamara+Polajnar%2E+2010%2E+Semi%2Dparametric+analysis+of+multi%2Drater+data%2E+Statistical+Computing%2C+20%3A317%E2%80%93334%2E)
28. Andrey Rzhetsky, Hagit Shatkay, and W. John Wilbur. 2009. How to get the most out of your curation effort. PLoS Computational Biology., 5(5):1–13. [find](https://scholar.google.com/scholar?q=Andrey+Rzhetsky%2C+Hagit+Shatkay%2C+and+W%2E+John+Wilbur%2E+2009%2E+How+to+get+the+most+out+of+your+curation+effort%2E+PLoS+Computational+Biology%2E%2C+5%285%29%3A1%E2%80%9313%2E)
29. Victor S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis. 2008. Get another label? Improving data quality and data mining using multiple, noisy labelers. In Proceedings of the Fourteenth ACM International Conference on Knowledge Discovery and Data Mining (KDD). [find](https://scholar.google.com/scholar?q=Victor+S%2E+Sheng%2C+Foster+Provost%2C+and+Panagiotis+G%2E+Ipeirotis%2E+2008%2E+Get+another+label%3F+Improving+data+quality+and+data+mining+using+multiple%2C+noisy+labelers%2E+In+Proceedings+of+the+Fourteenth+ACM+Inter)
30. Padhraic Smyth, Usama Fayyad, Michael Burl, Pietro Perona, and Pierre Baldi. 1995. Inferring ground truth from subjectively-labeled images of Venus. In Advances in Neural Information Processing Systems 7, pages 1085–1092. [find](https://scholar.google.com/scholar?q=Padhraic+Smyth%2C+Usama+Fayyad%2C+Michael+Burl%2C+Pietro+Perona%2C+and+Pierre+Baldi%2E+1995%2E+Inferring+ground+truth+from+subjectively%2Dlabeled+images+of+Venus%2E+In+Advances+in+Neural+Information+Processing+System)
31. MIT Press. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast - but is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of Empirical Methods in Natural Language Processing (EMNLP), pages 254–263, Honolulu. [find](https://scholar.google.com/scholar?q=MIT+Press%2E+Rion+Snow%2C+Brendan+O%E2%80%99Connor%2C+Daniel+Jurafsky%2C+and+Andrew+Y%2E+Ng%2E+2008%2E+Cheap+and+fast+%2D+but+is+it+good%3F+Evaluating+non%2Dexpert+annotations+for+natural+language+tasks%2E+In+Proceedings+of+Empiri) [our card](https://md.research.elamur.ai/free/en/papers/snow-2008-cheap-fast-but-is-it/)
32. Jan Vegelius. 1981. Significance tests for the G-index. Educational and Psychological Measurement, 41:99– 108. [find](https://scholar.google.com/scholar?q=Jan+Vegelius%2E+1981%2E+Significance+tests+for+the+G%2Dindex%2E+Educational+and+Psychological+Measurement%2C+41%3A99%E2%80%93+108%2E)
33. Jacob Whitehill, Paul Ruvolo, Tingfan Wu, Jacob Bergsma, and Javier Movellan. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Y. Bengio, D. Schuurmans, J. Lafferty, C. K. I. Williams, and A. Culotta, editors, Advances in Neural Information Pro… [find](https://scholar.google.com/scholar?q=Jacob+Whitehill%2C+Paul+Ruvolo%2C+Tingfan+Wu%2C+Jacob+Bergsma%2C+and+Javier+Movellan%2E+2009%2E+Whose+vote+should+count+more%3A+Optimal+integration+of+labels+from+labelers+of+unknown+expertise%2E+In+Y%2E+Bengio%2C+D%2E+Sch)
34. Yiming Yang and Jan O. Pedersen. 1997. A comparative study on feature selection in text categorization. [find](https://scholar.google.com/scholar?q=Yiming+Yang+and+Jan+O%2E+Pedersen%2E+1997%2E+A+comparative+study+on+feature+selection+in+text+categorization%2E)
35. In Proceedings of the Fourteenth International Conference on Machine Learning, ICML ’97, pages 412– 420, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. [find](https://scholar.google.com/scholar?q=In+Proceedings+of+the+Fourteenth+International+Conference+on+Machine+Learning%2C+ICML+%E2%80%9997%2C+pages+412%E2%80%93+420%2C+San+Francisco%2C+CA%2C+USA%2E+Morgan+Kaufmann+Publishers+Inc%2E)