--- title: "Cheap and Fast – But is it Good? Evaluating — Snow 2008" description: "Expert linguistic annotation is slow and costly. This paper asks whether crowds of paid non-experts on Amazon Mechanical Turk (AMT) can stand in for…" authors: ["Rion Snow", "Brendan O'Connor", "Daniel Jurafsky", "Andrew Ng"] year: 2008 venue: "Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing" source: "https://aclanthology.org/D08-1027/" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/snow-2008-cheap-fast-but-is-it/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/snow-2008-cheap-fast-but-is-it/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2008](https://md.research.elamur.ai/free/en/papers/2008/) / Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks # Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks **Authors:** Rion Snow, [Brendan O'Connor](https://md.research.elamur.ai/free/en/authors/brendan-o-connor/), [Daniel Jurafsky](https://md.research.elamur.ai/free/en/authors/daniel-jurafsky/), [Andrew Ng](https://md.research.elamur.ai/free/en/authors/andrew-ng/) · **Year:** 2008 · **Venue:** Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/D08-1027/) · [PDF](https://aclanthology.org/D08-1027.pdf) · [OpenAlex](https://openalex.org/W7097455967) **Citations:** 0 citations (openalex-snapshot · 2026-10-14) · [390](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary Expert linguistic annotation is slow and costly. This paper asks whether crowds of paid non-experts on Amazon Mechanical Turk (AMT) can stand in for specialists. The authors picked five tasks for which expert gold labels already existed: judging emotion in text (affect recognition), rating how similar pairs of words are, deciding textual entailment (RTE), placing events in temporal order, and word sense disambiguation. They used short task descriptions with an example for each class and restricted responses to multiple-choice or bounded numeric input. They gathered ten independent annotations per item, which let them measure how label quality grows as more workers are added. On all five tasks, crowd labels agreed well with expert labels. A small number of averaged non-expert judgments typically matched one expert. For affect recognition, a classifier trained on crowd labels performed as well as, or better than, one trained on a single expert's labels. The authors also introduced a bias-correction scheme that weights workers using a small set of gold-labelled items. The work matters because it showed that large annotation projects can be run at a small fraction of the usual price, and it gave a practical recipe for checking quality. ## Key points - **Affect data:** 100 headlines were sampled from the SemEval affective text test set. Each was rated for six emotions plus valence, with 10 annotations per label type, for 7000 labels in total. The whole collection cost US$2.00, which is 3500 labels per dollar or at least 875 expert-equivalent labels per dollar. - **Workers needed per expert:** On average, about 4 non-expert labels per item matched the agreement of one expert. For most emotions, expert level was reached with 9 or fewer workers, often with just 2. "Fear" was the exception and never reached expert level. - **Better gold labels:** Adding crowd labels to the expert pool raised agreement for experts. Average Pearson correlation across all tasks went from 0.580 to 0.607. - **Bias correction:** Per-worker models were calibrated on gold items, using Laplace smoothing with 1 pseudocount, uniform priors, and 20-fold cross-validation. This beat naive majority voting on RTE-1 (about +4.0% accuracy) and on event annotation. Word sense disambiguation was already at maximum accuracy, so it could not improve. - **Training a classifier:** A simple bag-of-words unigram affect model similar to SWAT was trained on 100 headlines and tested on 900. For five of the seven label types, a model trained on one set of crowd labels beat the average model trained on one expert's labels. ## Interesting ideas - **Express crowd cost in expert units.** Convert cost into "expert-equivalent labels per dollar" by finding how many cheap labels match one expert. This makes annotation budgets comparable across vendors or labelling strategies, for example when planning a Ukrainian corpus. - **Collect deliberate redundancy.** Gather many labels per item, such as ten, then resample subsets of size k to plot quality against annotator count. This tells you the cheapest redundancy level for production before you commit the full budget. - **Calibrate workers on a gold subset.** Seed a few expert-labelled items and use them to estimate each worker's reliability, then use these estimates to weight votes. This is a cheap way to beat plain majority voting and applies to any categorical labelling job. - **Mix many annotators to reduce bias.** A label set pooled from several people can train a better model than one consistent expert, possibly because individual biases average out. When hiring annotators, it may be worth spreading items across several people rather than relying on one. - **Score annotators with leave-one-out agreement.** Correlate one annotator with the average of the others, and compare expert-vs-expert with crowd-vs-expert scores. This gives a fair reference point for numeric rating tasks where no single "true" label exists. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) Topics: [Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) ## Cited by Cited by 390 papers of our library cite it (of the 140884 whose bibliographies we read); 40 of them have a page here, 350 not yet. - [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://md.research.elamur.ai/free/en/papers/krishna-2016-visual-genome-connecting-language-vision/) - [Lexicon-Based Methods for Sentiment Analysis](https://md.research.elamur.ai/free/en/papers/taboada-2011-lexicon-based-methods-sentiment-analysis/) - [Reading Tea Leaves: How Humans Interpret Topic Models](https://md.research.elamur.ai/free/en/papers/chang-2009-reading-tea-leaves-humans-interpret-topic/) - [Improving Word Representations via Global Context and Multiple Word Prototypes](https://md.research.elamur.ai/free/en/papers/huang-2012-improving-word-representations-global-context/) - [Better Word Representations with Recursive Neural Networks for Morphology](https://md.research.elamur.ai/free/en/papers/luong-2013-better-word-representations-recursive-neural/) - [Can Large Language Models Transform Computational Social Science?](https://md.research.elamur.ai/free/en/papers/ziems-2023-large-language-models-transform-computational/) - [Data-Driven Response Generation in Social Media](https://md.research.elamur.ai/free/en/papers/ritter-2011-data-driven-response-generation-social/) - [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://md.research.elamur.ai/free/en/papers/coucke-2018-snips-voice-platform-embedded-spoken/) - [Collecting Image Annotations Using Amazon's Mechanical Turk](https://md.research.elamur.ai/free/en/papers/rashtchian-2010-collecting-image-annotations-amazon-s/) - [Multi-Prototype Vector-Space Models of Word Meaning](https://md.research.elamur.ai/free/en/papers/reisinger-2010-multi-prototype-vector-space-models/) - [Learning Whom to Trust with MACE](https://md.research.elamur.ai/free/en/papers/hovy-2013-learning-whom-trust-mace/) - [Inherent Disagreements in Human Textual Inferences](https://md.research.elamur.ai/free/en/papers/pavlick-2019-inherent-disagreements-human-textual-inferences/) - [Crowdsourcing Translation: Professional Quality from Non-Professionals](https://md.research.elamur.ai/free/en/papers/zaidan-2011-crowdsourcing-translation-professional-quality-non/) - [The “Problem” of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation](https://md.research.elamur.ai/free/en/papers/plank-2022-problem-human-label-variation-ground/) - [A Corpus for Research on Deliberation and Debate](https://md.research.elamur.ai/free/en/papers/walker-2012-corpus-research-deliberation-debate/) - [Expert, Crowdsourced, and Machine Assessment of Suicide Risk via Online Postings](https://md.research.elamur.ai/free/en/papers/shing-2018-expert-crowdsourced-machine-assessment-suicide/) - [The Benefits of a Model of Annotation](https://md.research.elamur.ai/free/en/papers/passonneau-2014-benefits-model-annotation/) - [An Empirical Study on Compositionality in Compound Nouns](https://md.research.elamur.ai/free/en/papers/reddy-2011-empirical-study-compositionality-compound-nouns/) - [Interpretation of Natural Language Rules in Conversational Machine Reading](https://md.research.elamur.ai/free/en/papers/saeidi-2018-interpretation-natural-language-rules-conversational/) - [Universal Decompositional Semantics on Universal Dependencies](https://md.research.elamur.ai/free/en/papers/white-2016-universal-decompositional-semantics-universal-dependencies/) - [Did It Happen? The Pragmatic Complexity of Veridicality Assessment](https://md.research.elamur.ai/free/en/papers/marneffe-2012-did-it-happen-pragmatic-complexity/) - [Paraphrasing for Style](https://md.research.elamur.ai/free/en/papers/xu-2012-paraphrasing-style/) - [Generating Coherent Event Schemas at Scale](https://md.research.elamur.ai/free/en/papers/balasubramanian-2013-generating-coherent-event-schemas-scale/) - [Fast, Cheap, and Creative: Evaluating Translation Quality Using Amazon's Mechanical Turk](https://md.research.elamur.ai/free/en/papers/burch-2009-fast-cheap-creative-evaluating-translation/) - [A Crowdsourced Corpus of Multiple Judgments and Disagreement on Anaphoric Interpretation](https://md.research.elamur.ai/free/en/papers/poesio-2019-crowdsourced-corpus-multiple-judgments-disagreement/) - [Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation](https://md.research.elamur.ai/free/en/papers/fernandes-2023-bridging-gap-survey-integrating-human/) - [Was It Good? It Was Provocative.” Learning the Meaning of Scalar Adjectives](https://md.research.elamur.ai/free/en/papers/marneffe-2010-was-it-good-it-was/) - [Outsourcing FrameNet to the Crowd](https://md.research.elamur.ai/free/en/papers/fossati-2013-outsourcing-framenet-crowd/) - [Examples and Specifications that Prove a Point: Identifying Elaborative and Argumentative Discourse Relations](https://md.research.elamur.ai/free/en/papers/scholman-2017-examples-specifications-that-prove-point/) - [Phrase Detectives Corpus 1.0 Crowdsourced Anaphoric Coreference.](https://md.research.elamur.ai/free/en/papers/chamberlain-2016-phrase-detectives-corpus-1-0/) - [Really? Well. Apparently Bootstrapping Improves the Performance of Sarcasm and Nastiness Classifiers for Online Dialogue](https://md.research.elamur.ai/free/en/papers/lukin-2013-really-well-apparently-bootstrapping-improves/) - [TyDiP: A Dataset for Politeness Classification in Nine Typologically Diverse Languages](https://md.research.elamur.ai/free/en/papers/srinivasan-2022-tydip-dataset-politeness-classification-nine/) - [Statistical Parsing by Machine Learning from a Classical Arabic Treebank](https://md.research.elamur.ai/free/en/papers/langstream-2015-dukes-statistical-parsing-machine-learning-classical/) - [A Probabilistic Annotation Model for Crowdsourcing Coreference](https://md.research.elamur.ai/free/en/papers/paun-2018-probabilistic-annotation-model-crowdsourcing-coreference/) - [Getting Reliable Annotations for Sarcasm in Online Dialogues](https://md.research.elamur.ai/free/en/papers/swanson-2014-getting-reliable-annotations-sarcasm-online/) - [Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design](https://md.research.elamur.ai/free/en/papers/pyatkin-2023-design-choices-crowdsourcing-implicit-discourse/) - [Improving Crowdsourcing-Based Annotation of Japanese Discourse Relations](https://md.research.elamur.ai/free/en/papers/kishimoto-2018-improving-crowdsourcing-based-annotation-japanese/) - [Design Choices in Crowdsourcing Discourse Relation Annotations: The Effect of Worker Selection and Training](https://md.research.elamur.ai/free/en/papers/scholman-2022-design-choices-crowdsourcing-discourse-relation/) - [Towards Semi-Automated Annotation for Prepositional Phrase Attachment](https://md.research.elamur.ai/free/en/papers/lingbulk-2010-rosenthal-semi-automated-annotation-prepositional-phrase/) - [Coreference Annotation of an Arabic Corpus using a Virtual World Game](https://md.research.elamur.ai/free/en/papers/aliady-2022-coreference-annotation-arabic-corpus-virtual/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{snow2008cheap, title = {{Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks}}, author = {Snow, Rion and O'Connor, Brendan and Jurafsky, Daniel and Ng, Andrew}, booktitle = {Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing}, year = {2008}, url = {https://aclanthology.org/D08-1027/} } ``` - [← previous Lattice-based Minimum Error Rate Training for Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/macherey-2008-lattice-based-minimum-error-rate/) - [next → A New String-to-Dependency Machine Translation Algorithm with a Target Dependency Language Model](https://md.research.elamur.ai/free/en/papers/shen-2008-new-string-dependency-machine-translation/) - [↑ back to papers of 2008](https://md.research.elamur.ai/free/en/papers/2008/) ## References 33 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Paul S. Albert and Lori E. Dodd. 2004. A Cautionary Note on the Robustness of Latent Class Models for Estimating Diagnostic Error without a Gold Standard. Biometrics, Vol. 60 (2004), pp. 427-435. [find](https://scholar.google.com/scholar?q=Paul+S%2E+Albert+and+Lori+E%2E+Dodd%2E+2004%2E+A+Cautionary+Note+on+the+Robustness+of+Latent+Class+Models+for+Estimating+Diagnostic+Error+without+a+Gold+Standard%2E+Biometrics%2C+Vol%2E+60+%282004%29%2C+pp%2E+427%2D435%2E) 2. Collin F. Baker, Charles J. Fillmore and John B. Lowe. 1998. The Berkeley FrameNet project. In Proc. of COLING-ACL 1998. [find](https://scholar.google.com/scholar?q=Collin+F%2E+Baker%2C+Charles+J%2E+Fillmore+and+John+B%2E+Lowe%2E+1998%2E+The+Berkeley+FrameNet+project%2E+In+Proc%2E+of+COLING%2DACL+1998%2E) [our card](https://md.research.elamur.ai/free/en/papers/lex-1998-baker-berkeley-framenet/) 3. Michele Banko and Eric Brill. 2001. Scaling to Very Very Large Corpora for Natural Language Disambiguation. In Proc. of ACL-2001. [find](https://scholar.google.com/scholar?q=Michele+Banko+and+Eric+Brill%2E+2001%2E+Scaling+to+Very+Very+Large+Corpora+for+Natural+Language+Disambiguation%2E+In+Proc%2E+of+ACL%2D2001%2E) [our card](https://md.research.elamur.ai/free/en/papers/banko-2001-scaling-very-very-large-corpora/) 4. Junfu Cai, Wee Sun Lee and Yee Whye Teh. 2007. Improving Word Sense Disambiguation Using Topic Features. In Proc. of EMNLP-2007 . [find](https://scholar.google.com/scholar?q=Junfu+Cai%2C+Wee+Sun+Lee+and+Yee+Whye+Teh%2E+2007%2E+Improving+Word+Sense+Disambiguation+Using+Topic+Features%2E+In+Proc%2E+of+EMNLP%2D2007+%2E) 5. Timothy Chklovski and Rada Mihalcea. 2002. Building a sense tagged corpus with Open Mind Word Expert. In Proc. of the Workshop on ”Word Sense Disambiguation: Recent Successes and Future Directions”, ACL 2002. [find](https://scholar.google.com/scholar?q=Timothy+Chklovski+and+Rada+Mihalcea%2E+2002%2E+Building+a+sense+tagged+corpus+with+Open+Mind+Word+Expert%2E+In+Proc%2E+of+the+Workshop+on+%E2%80%9DWord+Sense+Disambiguation%3A+Recent+Successes+and+Future+Directions%E2%80%9D%2C+A) 6. Timothy Chklovski and Yolanda Gil. 2005. Towards Managing Knowledge Collection from Volunteer Contributors. Proceedings of AAAI Spring Symposium on Knowledge Collection from Volunteer Contributors (KCVC05). [find](https://scholar.google.com/scholar?q=Timothy+Chklovski+and+Yolanda+Gil%2E+2005%2E+Towards+Managing+Knowledge+Collection+from+Volunteer+Contributors%2E+Proceedings+of+AAAI+Spring+Symposium+on+Knowledge+Collection+from+Volunteer+Contributors+%28KC) 7. Ido Dagan, Oren Glickman and Bernardo Magnini. 2006. The PASCAL Recognising Textual Entailment Challenge. Machine Learning Challenges. Lecture Notes in Computer Science, Vol. 3944, pp. 177-190, Springer, 2006. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman+and+Bernardo+Magnini%2E+2006%2E+The+PASCAL+Recognising+Textual+Entailment+Challenge%2E+Machine+Learning+Challenges%2E+Lecture+Notes+in+Computer+Science%2C+Vol%2E+3944%2C+pp%2E+177%2D190%2C+Spring) 8. Wisam Dakka and Panagiotis G. Ipeirotis. 2008. Automatic Extraction of Useful Facet Terms from Text Documents. In Proc. of ICDE-2008. [find](https://scholar.google.com/scholar?q=Wisam+Dakka+and+Panagiotis+G%2E+Ipeirotis%2E+2008%2E+Automatic+Extraction+of+Useful+Facet+Terms+from+Text+Documents%2E+In+Proc%2E+of+ICDE%2D2008%2E) 9. A. P. Dawid and A. M. Skene. 1979. Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. Applied Statistics, Vol. 28, No. 1 (1979), pp. 20-28. [find](https://scholar.google.com/scholar?q=A%2E+P%2E+Dawid+and+A%2E+M%2E+Skene%2E+1979%2E+Maximum+Likelihood+Estimation+of+Observer+Error%2DRates+Using+the+EM+Algorithm%2E+Applied+Statistics%2C+Vol%2E+28%2C+No%2E+1+%281979%29%2C+pp%2E+20%2D28%2E) 10. Michael Kaisser and John B. Lowe. 2008. A Research Collection of QuestionAnswer Sentence Pairs. In Proc. of LREC-2008. [find](https://scholar.google.com/scholar?q=Michael+Kaisser+and+John+B%2E+Lowe%2E+2008%2E+A+Research+Collection+of+QuestionAnswer+Sentence+Pairs%2E+In+Proc%2E+of+LREC%2D2008%2E) 11. Michael Kaisser, Marti Hearst, and John B. Lowe. 2008. Evidence for Varying Search Results Summary Lengths. In Proc. of ACL-2008. [find](https://scholar.google.com/scholar?q=Michael+Kaisser%2C+Marti+Hearst%2C+and+John+B%2E+Lowe%2E+2008%2E+Evidence+for+Varying+Search+Results+Summary+Lengths%2E+In+Proc%2E+of+ACL%2D2008%2E) 12. Phil Katz, Matthew Singleton, Richard Wicentowski. 2007. SWAT-MP: The SemEval-2007 Systems for Task 5 and Task 14. In Proc. of SemEval-2007. [find](https://scholar.google.com/scholar?q=Phil+Katz%2C+Matthew+Singleton%2C+Richard+Wicentowski%2E+2007%2E+SWAT%2DMP%3A+The+SemEval%2D2007+Systems+for+Task+5+and+Task+14%2E+In+Proc%2E+of+SemEval%2D2007%2E) 13. Aniket Kittur, Ed H. Chi, and Bongwon Suh. 2008. Crowdsourcing user studies with Mechanical Turk. In Proc. of CHI-2008. [find](https://scholar.google.com/scholar?q=Aniket+Kittur%2C+Ed+H%2E+Chi%2C+and+Bongwon+Suh%2E+2008%2E+Crowdsourcing+user+studies+with+Mechanical+Turk%2E+In+Proc%2E+of+CHI%2D2008%2E) 14. Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: the Penn Treebank. Computational Linguistics 19:2, June 1993. [find](https://scholar.google.com/scholar?q=Mitchell+P%2E+Marcus%2C+Mary+Ann+Marcinkiewicz%2C+and+Beatrice+Santorini%2E+1993%2E+Building+a+large+annotated+corpus+of+English%3A+the+Penn+Treebank%2E+Computational+Linguistics+19%3A2%2C+June+1993%2E) [our card](https://md.research.elamur.ai/free/en/papers/bank-1993-marcus-building-large-annotated-corpus-english/) 15. George A. Miller and William G. Charles. 1991. Contextual Correlates of Semantic Similarity. Language and Cognitive Processes, vol. 6, no. 1, pp. 1-28, 1991. [find](https://scholar.google.com/scholar?q=George+A%2E+Miller+and+William+G%2E+Charles%2E+1991%2E+Contextual+Correlates+of+Semantic+Similarity%2E+Language+and+Cognitive+Processes%2C+vol%2E+6%2C+no%2E+1%2C+pp%2E+1%2D28%2C+1991%2E) 16. George A. Miller, Claudia Leacock, Randee Tengi, and Ross T. Bunke. 1993. A semantic concordance. In Proc. of HLT-1993. [find](https://scholar.google.com/scholar?q=George+A%2E+Miller%2C+Claudia+Leacock%2C+Randee+Tengi%2C+and+Ross+T%2E+Bunke%2E+1993%2E+A+semantic+concordance%2E+In+Proc%2E+of+HLT%2D1993%2E) 17. Preslav Nakov. 2008. Paraphrasing Verbs for Noun Compound Interpretation. In Proc. of the Workshop on Multiword Expressions, LREC-2008. [find](https://scholar.google.com/scholar?q=Preslav+Nakov%2E+2008%2E+Paraphrasing+Verbs+for+Noun+Compound+Interpretation%2E+In+Proc%2E+of+the+Workshop+on+Multiword+Expressions%2C+LREC%2D2008%2E) 18. Martha Palmer, Dan Gildea, and Paul Kingsbury. 2005. The Proposition Bank: A Corpus Annotated with Semantic Roles. Computational Linguistics, 31:1. [find](https://scholar.google.com/scholar?q=Martha+Palmer%2C+Dan+Gildea%2C+and+Paul+Kingsbury%2E+2005%2E+The+Proposition+Bank%3A+A+Corpus+Annotated+with+Semantic+Roles%2E+Computational+Linguistics%2C+31%3A1%2E) 19. Sameer Pradhan, Edward Loper, Dmitriy Dligach and Martha Palmer. 2007. SemEval-2007 Task-17: English Lexical Sample, SRL and All Words. In Proc. of SemEval-2007 . [find](https://scholar.google.com/scholar?q=Sameer+Pradhan%2C+Edward+Loper%2C+Dmitriy+Dligach+and+Martha+Palmer%2E+2007%2E+SemEval%2D2007+Task%2D17%3A+English+Lexical+Sample%2C+SRL+and+All+Words%2E+In+Proc%2E+of+SemEval%2D2007+%2E) [our card](https://md.research.elamur.ai/free/en/papers/pradhan-2007-semeval-2007-task-17-english/) 20. James Pustejovsky, Patrick Hanks, Roser Saur, Andrew See, Robert Gaizauskas, Andrea Setzer, Dragomir Radev, Beth Sundheim, David Day, Lisa Ferro and Marcia Lazo. 2003. The TIMEBANK Corpus. In Proc. of Corpus Linguistics 2003, 647-656. [find](https://scholar.google.com/scholar?q=James+Pustejovsky%2C+Patrick+Hanks%2C+Roser+Saur%2C+Andrew+See%2C+Robert+Gaizauskas%2C+Andrea+Setzer%2C+Dragomir+Radev%2C+Beth+Sundheim%2C+David+Day%2C+Lisa+Ferro+and+Marcia+Lazo%2E+2003%2E+The+TIMEBANK+Corpus%2E+In+Proc%2E+of) 21. Philip Resnik. 1999. Semantic Similarity in a Taxonomy: An Information-Based Measure and its Application to Problems of Ambiguity in Natural Language. JAIR, Volume 11, pages 95-130. [find](https://scholar.google.com/scholar?q=Philip+Resnik%2E+1999%2E+Semantic+Similarity+in+a+Taxonomy%3A+An+Information%2DBased+Measure+and+its+Application+to+Problems+of+Ambiguity+in+Natural+Language%2E+JAIR%2C+Volume+11%2C+pages+95%2D130%2E) 22. Herbert Rubenstein and John B. Goodenough. 1965. Contextual Correlates of Synonymy. Communications of the ACM, 8(10):627–633. [find](https://scholar.google.com/scholar?q=Herbert+Rubenstein+and+John+B%2E+Goodenough%2E+1965%2E+Contextual+Correlates+of+Synonymy%2E+Communications+of+the+ACM%2C+8%2810%29%3A627%E2%80%93633%2E) 23. Victor S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis. 2008. Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers. In Proc. of KDD-2008. [find](https://scholar.google.com/scholar?q=Victor+S%2E+Sheng%2C+Foster+Provost%2C+and+Panagiotis+G%2E+Ipeirotis%2E+2008%2E+Get+Another+Label%3F+Improving+Data+Quality+and+Data+Mining+Using+Multiple%2C+Noisy+Labelers%2E+In+Proc%2E+of+KDD%2D2008%2E) 24. Push Singh. 2002. The public acquisition of commonsense knowledge. In Proc. of AAAI Spring Symposium on Acquiring (and Using) Linguistic (and World) Knowledge for Information Access, 2002. [find](https://scholar.google.com/scholar?q=Push+Singh%2E+2002%2E+The+public+acquisition+of+commonsense+knowledge%2E+In+Proc%2E+of+AAAI+Spring+Symposium+on+Acquiring+%28and+Using%29+Linguistic+%28and+World%29+Knowledge+for+Information+Access%2C+2002%2E) 25. Alexander Sorokin and David Forsyth. 2008. Utility data annotation with Amazon Mechanical Turk. To appear in Proc. of First IEEE Workshop on Internet Vision at CVPR, 2008. See also: [http://vision.cs.uiuc.edu/annotation/](http://vision.cs.uiuc.edu/annotation/) [↗](http://vision.cs.uiuc.edu/annotation/) 26. David G. Stork. 1999. The Open Mind Initiative. IEEE Expert Systems and Their Applications pp. 16-20, May/June 1999. [find](https://scholar.google.com/scholar?q=David+G%2E+Stork%2E+1999%2E+The+Open+Mind+Initiative%2E+IEEE+Expert+Systems+and+Their+Applications+pp%2E+16%2D20%2C+May%2FJune+1999%2E) 27. Carlo Strapparava and Rada Mihalcea. 2007. SemEval-2007 Task 14: Affective Text In Proc. of SemEval-2007. [find](https://scholar.google.com/scholar?q=Carlo+Strapparava+and+Rada+Mihalcea%2E+2007%2E+SemEval%2D2007+Task+14%3A+Affective+Text+In+Proc%2E+of+SemEval%2D2007%2E) [our card](https://md.research.elamur.ai/free/en/papers/strapparava-2007-semeval-2007-task-14-affective/) 28. Qi Su, Dmitry Pavlov, Jyh-Herng Chow, and Wendell C. Baker. 2007. Internet-Scale Collection of Human-Reviewed Data. In Proc. of WWW-2007. [find](https://scholar.google.com/scholar?q=Qi+Su%2C+Dmitry+Pavlov%2C+Jyh%2DHerng+Chow%2C+and+Wendell+C%2E+Baker%2E+2007%2E+Internet%2DScale+Collection+of+Human%2DReviewed+Data%2E+In+Proc%2E+of+WWW%2D2007%2E) 29. Luis von Ahn and Laura Dabbish. 2004. Labeling Images with a Computer Game. In ACM Conference on Human Factors in Computing Systems, CHI 2004. [find](https://scholar.google.com/scholar?q=Luis+von+Ahn+and+Laura+Dabbish%2E+2004%2E+Labeling+Images+with+a+Computer+Game%2E+In+ACM+Conference+on+Human+Factors+in+Computing+Systems%2C+CHI+2004%2E) 30. Luis von Ahn, Mihir Kedia and Manuel Blum. 2006. Verbosity: A Game for Collecting Common-Sense Knowledge. In ACM Conference on Human Factors in Computing Systems, CHI Notes 2006. [find](https://scholar.google.com/scholar?q=Luis+von+Ahn%2C+Mihir+Kedia+and+Manuel+Blum%2E+2006%2E+Verbosity%3A+A+Game+for+Collecting+Common%2DSense+Knowledge%2E+In+ACM+Conference+on+Human+Factors+in+Computing+Systems%2C+CHI+Notes+2006%2E) 31. Ellen Voorhees and Hoa Trang Dang. 2006. Overview of the TREC 2005 question answering track. In Proc. of TREC-2005. [find](https://scholar.google.com/scholar?q=Ellen+Voorhees+and+Hoa+Trang+Dang%2E+2006%2E+Overview+of+the+TREC+2005+question+answering+track%2E+In+Proc%2E+of+TREC%2D2005%2E) 32. Janyce M. Wiebe, Rebecca F. Bruce and Thomas P. O’Hara. 1999. Development and use of a gold-standard data set for subjectivity classifications. In Proc. of ACL-1999. [find](https://scholar.google.com/scholar?q=Janyce+M%2E+Wiebe%2C+Rebecca+F%2E+Bruce+and+Thomas+P%2E+O%E2%80%99Hara%2E+1999%2E+Development+and+use+of+a+gold%2Dstandard+data+set+for+subjectivity+classifications%2E+In+Proc%2E+of+ACL%2D1999%2E) 33. Annie Zaenen. Submitted. Do give a penny for their thoughts. International Journal of Natural Language Engineering (submitted). [find](https://scholar.google.com/scholar?q=Annie+Zaenen%2E+Submitted%2E+Do+give+a+penny+for+their+thoughts%2E+International+Journal+of+Natural+Language+Engineering+%28submitted%29%2E)