---
title: "Creating Speech and Language Data With Amazon's Mechanical"
description: "This paper introduces the NAACL-2010 workshop on using Amazon's Mechanical Turk (MTurk) to… 62 references, cited by 86 papers here."
authors: ["Chris Callison-Burch", "Mark Dredze"]
year: 2010
venue: "Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk"
source: "https://aclanthology.org/W10-0701/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2010](https://md.research.elamur.ai/free/en/papers/2010/) / Creating Speech and Language Data With Amazon's Mechanical Turk

# Creating Speech and Language Data With Amazon's Mechanical Turk

**Authors:** [Chris Callison-Burch](https://md.research.elamur.ai/free/en/authors/chris-callison-burch/) and [Mark Dredze](https://md.research.elamur.ai/free/en/authors/mark-dredze/) · **Year:** 2010 · **Venue:** [Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk](https://md.research.elamur.ai/free/en/venues/naacl-hlt-workshop-on-creating-speech-and-language-data-with-amazon-s-mechanical/) ([NAACL HLT Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk 2010](https://md.research.elamur.ai/free/en/venues/naacl-hlt-workshop-on-creating-speech-and-language-data-with-amazon-s-mechanical/2010/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W10-0701/) · [PDF](https://aclanthology.org/W10-0701.pdf)

**Citations:** [86](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
This paper introduces the NAACL-2010 workshop on using Amazon's Mechanical Turk (MTurk) to build data for speech and language technology, and it also serves as a practical guide to the platform. The organizers ran an open-ended shared task. Each accepted researcher received a $100 MTurk credit, chose their own annotation task, reported on the experience and publicly released the collected data. The authors explain how MTurk works: Requesters post small paid Human Intelligence Tasks (HITs), and Turkers choose which ones to complete. They summarize a survey of who Turkers are and why they take part. They then cover quality control, including Amazon's built-in mechanisms and HIT designs that discourage or expose cheating. Finally, they argue that cheap crowdsourcing could reshape NLP research by enabling:
- new large corpora,
- data for low-resource languages and new domains,
- learning from multiple noisy annotators,
- active learning evaluated with real users.

The paper matters because it gathers early community experience on whether non-experts can produce usable annotations cheaply.

## Key points
- **Shared task numbers:** 35 credits of $100 were awarded, and participants had 10 days to run experiments. Of 30 shared-task submissions, 24 were accepted. Of 14 general-track submissions, 10 were accepted. This gives 34 papers and a 77% acceptance rate, and all shared-task data was released on the workshop website.
- **Earlier evidence (Snow et al., 2008):** combining votes from several non-experts can match expert annotations closely. Weighting each worker's vote using a small expert gold set helps further. Gathering 21,000 labels cost them only slightly more than $25.
- **Turker survey (Ipeirotis, 2010; 1,000 Turkers):**
  - 47% were from the US, 34% from India, and 19% from 66 other countries.
  - Most worked under 8 hours a week and earned under $10 a week.
  - 30% of US respondents reported being unemployed or underemployed.
- **Amazon's built-in quality mechanisms:** Requesters can reject work, assign each HIT to several Turkers, and require qualifications. The defaults are HIT Approval Rate and Location, and custom qualification tests can be added. Language proficiency is not a default qualification.
- **Payment and quality:** Mason and Watts (2009) found that paying more does not reliably improve quality. In an Urdu–English translation HIT paying $1, some workers cheated by pasting the text into online MT.

## Interesting ideas
- **Show text as images.** Displaying source sentences as pictures blocks copy-pasting into machine translation. Any task where automatic tools could stand in for real human work could use this trick.
- **Hide reference data inside the HIT.** Uploading hidden reference material, such as MT output, alongside each item lets reviewers automatically flag answers that copy it or have nothing to do with the source. This is a cheap automatic check for crowd-sourced translation or paraphrase collection.
- **Mix in known-answer control items.** Gold items with known labels, both positive and negative, reveal unreliable workers, whose labels can then be discarded. This works as a general quality filter for any annotation campaign.
- **Screen workers by language and location.** A custom qualification test can check language ability before annotation in a given language, and IP geolocation can decide how closely to review work. Both are relevant when recruiting annotators for less widely spoken languages.
- **Build complex labels from simple tasks.** Breaking a complex annotation into simple, intuitive micro-tasks makes it feasible for untrained workers. Keeping all workers' labels rather than a single gold answer also opens the door to learning from several annotators at once.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works

## Cite

Cite as: Chris Callison-Burch and Mark Dredze. 2010. Creating Speech and Language Data With Amazon's Mechanical Turk. Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{callisonburch2010creating,
  title = {{Creating Speech and Language Data With Amazon's Mechanical Turk}},
  author = {Chris Callison-Burch and Mark Dredze},
  booktitle = {Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk},
  year = {2010},
  url = {https://aclanthology.org/W10-0701/}
}
```

- [← previous Probabilistic Frame-Semantic Parsing](https://md.research.elamur.ai/free/en/papers/das-2010-probabilistic-frame-semantic-parsing/)
- [next → Tree Edit Models for Recognizing Textual Entailments, Paraphrases, and Answers to Questions](https://md.research.elamur.ai/free/en/papers/heilman-2010-tree-edit-models-recognizing-textual/)
- [↑ back to papers of 2010](https://md.research.elamur.ai/free/en/papers/2010/)

## References

62 references, 40 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Cem Akkaya, Alexander Conrad, Janyce Wiebe, and Rada Mihalcea. 2010. Amazon Mechanical Turk for subjectivity word sense disambiguation. In NAACL [http://wiki.github.com/callison-burch/mechanical_turk_workshop/](http://wiki.github.com/callison-burch/mechanical_turk_workshop/) [↗](http://wiki.github.com/callison-burch/mechanical_turk_workshop/)
2. Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk. [find](https://scholar.google.com/scholar?q=Workshop+on+Creating+Speech+and+Language+Data+With+Amazon%E2%80%99s+Mechanical+Turk%2E)
3. Vamshi Ambati and Stephan Vogel. 2010. [Can crowds build parallel corpora for machine translation systems?](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/) In NAACL Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk.
4. Vamshi Ambati, Stephan Vogel, and Jamie Carbonell. 2010. Active learning and crowd-sourcing for machine translation. Language Resources and Evaluation (LREC). [find](https://scholar.google.com/scholar?q=Vamshi+Ambati%2C+Stephan+Vogel%2C+and+Jamie+Carbonell%2E+2010%2E+Active+learning+and+crowd%2Dsourcing+for+machine+translation%2E+Language+Resources+and+Evaluation+%28LREC%29%2E)
5. Kathy Baker, Steven Bethard, Michael Bloodgood, Ralf Brown, Chris Callison-Burch, Glen Coppersmith, Bonnie Dorr, Wes Filardo, Kendall Giles, Ann Irvine, Mike Kayser, Lori Levin, Justin Martineau, Jim Mayfield, Scott Miller, Aaron Phillips, Andrew Philpot, Christine Piatko, Lane Schwartz, and David Z… [find](https://scholar.google.com/scholar?q=Kathy+Baker%2C+Steven+Bethard%2C+Michael+Bloodgood%2C+Ralf+Brown%2C+Chris+Callison%2DBurch%2C+Glen+Coppersmith%2C+Bonnie+Dorr%2C+Wes+Filardo%2C+Kendall+Giles%2C+Ann+Irvine%2C+Mike+Kayser%2C+Lori+Levin%2C+Justin+Martineau%2C+Jim+)
6. John Blitzer, Mark Dredze, and Fernando Pereira. 2007. [Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification](https://md.research.elamur.ai/free/en/papers/blitzer-2007-biographies-bollywood-boom-boxes-blenders/). In Association for Computational Linguistics (ACL).
7. Michael Bloodgood and Chris Callison-Burch. 2010a. Bucking the trend: Large-scale cost-focused active learning for statistical machine translation. In 48th Annual Meeting of the Association for Computational Linguistics, Uppsala, Sweden. [find](https://scholar.google.com/scholar?q=Michael+Bloodgood+and+Chris+Callison%2DBurch%2E+2010a%2E+Bucking+the+trend%3A+Large%2Dscale+cost%2Dfocused+active+learning+for+statistical+machine+translation%2E+In+48th+Annual+Meeting+of+the+Association+for+Comput)
8. Michael Bloodgood and Chris Callison-Burch. 2010b. [Using Mechanical Turk to build machine translation evaluation sets](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In NAACL Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk.
9. Cristina Butnariu, Su Nam Kim, Preslav Nakov, Diarmuid Ó Séaghdha, Stan Szpakowicz, and Tony Veale. 2009. Semeval-2010 Task 9: The interpretation of noun compounds using paraphrasing verbs and prepositions. In Workshop on Semantic Evaluations. [find](https://scholar.google.com/scholar?q=Cristina+Butnariu%2C+Su+Nam+Kim%2C+Preslav+Nakov%2C+Diarmuid+O%CC%81+Se%CC%81aghdha%2C+Stan+Szpakowicz%2C+and+Tony+Veale%2E+2009%2E+Semeval%2D2010+Task+9%3A+The+interpretation+of+noun+compounds+using+paraphrasing+verbs+and+prepo)
10. Olivia Buzek, Philip Resnik, and Ben Bederson. 2010. [Error driven paraphrase annotation using Mechanical Turk](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In NAACL Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk.
11. Chris Callison-Burch. 2009. [Fast, cheap, and creative: Evaluating translation quality using amazon’s mechanical turk](https://md.research.elamur.ai/free/en/papers/burch-2009-fast-cheap-creative-evaluating-translation/). In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing (EMNLP-2009), Singapore.
12. Jonathan Chang, Jordan Boyd-Graber, Chong Wang, Sean Gerrish, and David M. Blei. 2009. [Reading tea leaves: How humans interpret topic models](https://md.research.elamur.ai/free/en/papers/chang-2009-reading-tea-leaves-humans-interpret-topic/). In Neural Information Processing Systems.
13. Jonathan Chang. 2010. [Not-so-Latent Dirichlet Allocation: Collapsed Gibbs sampling using human judgments](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In NAACL Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk.
14. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR, Miami Beach, Floriday. [find](https://scholar.google.com/scholar?q=Jia+Deng%2C+Wei+Dong%2C+Richard+Socher%2C+Li%2DJia+Li%2C+Kai+Li%2C+and+Li+Fei%2DFei%2E+2009%2E+Imagenet%3A+A+large%2Dscale+hierarchical+image+database%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+R)
15. Michael Denkowski and Alon Lavie. 2010. [Exploring normalization techniques for human judgments of machine translation adequacy collected using Amazon Mechanical Turk](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In NAACL Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk.
16. Michael Denkowski, Hassan Al-Haj, and Alon Lavie. 2010. [Turker-assisted paraphrasing for English-Arabic machine translation](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In NAACL Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk.
17. Mark Dredze, Partha Pratim Talukdar, and Koby Crammer. 2009. Sequence learning from data with multiple labels. In ECML/PKDD Workshop on Learning from Multi-Label Data (MLD). [find](https://scholar.google.com/scholar?q=Mark+Dredze%2C+Partha+Pratim+Talukdar%2C+and+Koby+Crammer%2E+2009%2E+Sequence+learning+from+data+with+multiple+labels%2E+In+ECML%2FPKDD+Workshop+on+Learning+from+Multi%2DLabel+Data+%28MLD%29%2E)
18. Keelan Evanini, Derrick Higgins, and Klaus Zechner. 2010. [Using Amazon Mechanical Turk for transcription of non-native speech](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In NAACL Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk.
19. Christiane Fellbaum. 1998. WordNet: An Electronic Lexical Database. Bradford Books. [find](https://scholar.google.com/scholar?q=Christiane+Fellbaum%2E+1998%2E+WordNet%3A+An+Electronic+Lexical+Database%2E+Bradford+Books%2E)
20. Tim Finin, William Murnane, Anand Karandikar, Nicholas Keller, Justin Martineau, and Mark Dredze. 2010. [Touring PER, ORG and LOC on $100 a day](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In NAACL Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk.
21. Qin Gao and Stephan Vogel. 2010. [Semi-supervised word alignment with Mechanical Turk](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In NAACL Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk.
22. Dan Gillick and Yang Liu. 2010. [Non-expert evaluation of summarization systems is risky](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In NAACL Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk.
23. Jonathan Gordon, Benjamin Van Durme, and Lenhart Schubert. 2010. [Evaluation of commonsense knowledge with Mechanical Turk](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In NAACL Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk.
24. Matthew R. Gormley, Adam Gerber, Mary Harper, and Mark Dredze. 2010. [Non-expert correction of automatically generated relation annotations](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In NAACL Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk.
25. Catherine Grady and Matthew Lease. 2010. [Crowdsourcing document relevance assessment with Mechanical Turk](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In NAACL Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk.
26. Michael Heilman and Noah A. Smith. 2010. [Rating computer-generated questions with Mechanical Turk](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In NAACL Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk.
27. Chiara Higgins, Elizabeth McGrath, and Laila Moretto. 2010. [AMT crowdsourcing: A viable method for rapid discovery of Arabic nicknames?](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/) In NAACL Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk.
28. Paul J. Hopper and Sandra A. Thompson. 1980. Transitivity in grammar and discourse. Language, 56:251– 299. [find](https://scholar.google.com/scholar?q=Paul+J%2E+Hopper+and+Sandra+A%2E+Thompson%2E+1980%2E+Transitivity+in+grammar+and+discourse%2E+Language%2C+56%3A251%E2%80%93+299%2E)
29. Pei-Yun Hsueh, Prem Melville, and Vikas Sindhwani. 2009. Data quality from crowdsourcing: A study of annotation selection criteria. In Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing, pages 27–35, Boulder, Colorado, June. Association for Computational Li… [find](https://scholar.google.com/scholar?q=Pei%2DYun+Hsueh%2C+Prem+Melville%2C+and+Vikas+Sindhwani%2E+2009%2E+Data+quality+from+crowdsourcing%3A+A+study+of+annotation+selection+criteria%2E+In+Proceedings+of+the+NAACL+HLT+2009+Workshop+on+Active+Learning+for)
30. Panos Ipeirotis. 2010. New demographics of Mechanical Turk. [http://behind-the-enemy-lines.blogspot.com/2010/03/](http://behind-the-enemy-lines.blogspot.com/2010/03/) new-demographics-of-mechanical-turk. html. [↗](http://behind-the-enemy-lines.blogspot.com/2010/03/)
31. Ann Irvine and Alexandre Klementiev. 2010. [Using Mechanical Turk to annotate lexicons for less commonly used languages](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In NAACL Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk.
32. Mukund Jha, Jacob Andreas, Kapil Thadani, Sara Rosenthal, and Kathleen McKeown. 2010. [Corpus creation for new genres: A crowdsourced approach to PP attachment](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In NAACL Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk.
33. Michael Kaisser and John Lowe. 2008. Creating a research collection of question answer sentence pairs with Amazons Mechanical Turk. In Proceedings of the Sixth International Language Resources and Evaluation (LREC’08), Marrakech, Morocco. [find](https://scholar.google.com/scholar?q=Michael+Kaisser+and+John+Lowe%2E+2008%2E+Creating+a+research+collection+of+question+answer+sentence+pairs+with+Amazons+Mechanical+Turk%2E+In+Proceedings+of+the+Sixth+International+Language+Resources+and+Eva)
34. Stephen Kunath and Steven Weinberger. 2010. [The wisdom of the crowd’s ear: Speech accent rating and annotation with Amazon Mechanical Turk](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In NAACL Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk.
35. Ian Lane, Matthias Eck, Kay Rottmann, and Alex Waibel. 2010. [Tools for collecting speech corpora via Mechanical-Turk](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In NAACL Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk.
36. Nolan Lawson, Kevin Eustice, Mike Perkowitz, and Meliha Yetisgen Yildiz. 2010. [Annotating large email datasets for named entity recognition with Mechanical Turk](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In NAACL Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk.
37. Greg Little, Lydia B. Chilton, Rob Miller, and Max Goldman. 2009. Turkit: Tools for iterative tasks on mechanical turk. In Proceedings of the Workshop on Human Computation at the International Conference on Knowledge Discovery and Data Mining (KDD-HCOMP ’09), Paris. [find](https://scholar.google.com/scholar?q=Greg+Little%2C+Lydia+B%2E+Chilton%2C+Rob+Miller%2C+and+Max+Goldman%2E+2009%2E+Turkit%3A+Tools+for+iterative+tasks+on+mechanical+turk%2E+In+Proceedings+of+the+Workshop+on+Human+Computation+at+the+International+Confere)
38. Nitin Madnani and Jordan Boyd-Graber. 2010. [Measuring transitivity using untrained annotators](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In NAACL Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk.
39. Mitch Marcus., Beatrice Santorini, and Mary Ann Marcinkiewicz. 1994. [Building a large annotated corpus of english: The penn treebank](https://md.research.elamur.ai/free/en/papers/marcus-1993-building-large-annotated-corpus-english/). Computational linguistics, 19(2):313–330.
40. Matthew Marge, Satanjeev Banerjee, and Alexander Rudnicky. 2010a. Using the Amazon Mechanical Turk for transcription of spoken language. ICASSP, March. [find](https://scholar.google.com/scholar?q=Matthew+Marge%2C+Satanjeev+Banerjee%2C+and+Alexander+Rudnicky%2E+2010a%2E+Using+the+Amazon+Mechanical+Turk+for+transcription+of+spoken+language%2E+ICASSP%2C+March%2E)
41. Matthew Marge, Satanjeev Banerjee, and Alexander Rudnicky. 2010b. [Using the Amazon Mechanical Turk to transcribe and annotate meeting speech for extractive summarization](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In NAACL Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk.
42. Winter Mason and Duncan J. Watts. 2009. Financial incentives and the “performance of crowds”. In Proceedings of the Workshop on Human Computation at the International Conference on Knowledge Discovery and Data Mining (KDD-HCOMP ’09), Paris. [find](https://scholar.google.com/scholar?q=Winter+Mason+and+Duncan+J%2E+Watts%2E+2009%2E+Financial+incentives+and+the+%E2%80%9Cperformance+of+crowds%E2%80%9D%2E+In+Proceedings+of+the+Workshop+on+Human+Computation+at+the+International+Conference+on+Knowledge+Discovery)
43. Ian McGraw, Chia ying Lee, Lee Hetherington, and Jim Glass. 2010. Collecting voices from the crowd. LREC, May. [find](https://scholar.google.com/scholar?q=Ian+McGraw%2C+Chia+ying+Lee%2C+Lee+Hetherington%2C+and+Jim+Glass%2E+2010%2E+Collecting+voices+from+the+crowd%2E+LREC%2C+May%2E)
44. Bart Mellebeek, Francesc Benavent, Jens Grivolla, Joan Codina, Marta R. Costa-Jussà, and Rafael Banchs. 2010. [Opinion mining of spanish customer comments with non-expert annotations on Mechanical Turk](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In NAACL Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk.
45. Robert Munro, Steven Bethard, Victor Kuperman, Vicky Tzuyin Lai, Robin Melnick, Christopher Potts, Tyler Schnoebelen, and Harry Tily. 2010. [Crowdsourcing and language studies: the new generation of linguistic data](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In NAACL Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk.
46. Matteo Negri and Yashar Mehdad. 2010. [Creating a bi-lingual entailment corpus through translations with Mechanical Turk: $100 for a 10 days rush](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In NAACL Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk.
47. Scott Novotney and Chris Callison-Burch. 2010a. Cheap, fast and good enough: Automatic speech recognition with non-expert transcription. NAACL, June. [find](https://scholar.google.com/scholar?q=Scott+Novotney+and+Chris+Callison%2DBurch%2E+2010a%2E+Cheap%2C+fast+and+good+enough%3A+Automatic+speech+recognition+with+non%2Dexpert+transcription%2E+NAACL%2C+June%2E)
48. Scott Novotney and Chris Callison-Burch. 2010b. [Crowdsourced accessibility: Elicitation of Wikipedia articles](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In NAACL Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk.
49. Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. [Thumbs up?: sentiment classification using machine learning techniques](https://md.research.elamur.ai/free/en/papers/pang-2002-thumbs-up-sentiment-classification-machine/). In Empirical Methods in Natural Language Processing (EMNLP).
50. Gabriel Parent and Maxine Eskenazi. 2010. [Clustering dictionary definitions using Amazon Mechanical Turk](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In NAACL Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk.
51. Dragomir R. Radev, Pradeep Muthukrishnan, and Vahed Qazvinian. 2009. The acl anthology network. In Proceedings of the 2009 Workshop on Text and Citation Analysis for Scholarly Digital Libraries, pages 54–61, Suntec City, Singapore, August. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Dragomir+R%2E+Radev%2C+Pradeep+Muthukrishnan%2C+and+Vahed+Qazvinian%2E+2009%2E+The+acl+anthology+network%2E+In+Proceedings+of+the+2009+Workshop+on+Text+and+Citation+Analysis+for+Scholarly+Digital+Libraries%2C+pages)
52. Cyrus Rashtchian, Peter Young, Micah Hodosh, and Julia Hockenmaier. 2010. [Collecting image annotations using Amazon’s Mechanical Turk](https://md.research.elamur.ai/free/en/papers/rashtchian-2010-collecting-image-annotations-amazon-s/). In NAACL Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk.
53. Joel Rose. 2010. Some turn to ‘Mechanical’ job search. [http://marketplace.publicradio.org/display/web/2009/06/30/pm_turking/](http://marketplace.publicradio.org/display/web/2009/06/30/pm_turking/). Marketplace public radio. Air date: June 30, 2009. [↗](http://marketplace.publicradio.org/display/web/2009/06/30/pm_turking/)
54. Erik Tjong Kim Sang and Fien De Meulder. 2003. [Introduction to the conll-2003 shared task: Language-independent named entity recognition](https://md.research.elamur.ai/free/en/papers/sang-2003-introduction-conll-2003-shared-task/). In CoNLL-2003.
55. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. [Cheap and fast - but is it good?](https://md.research.elamur.ai/free/en/papers/snow-2008-cheap-fast-but-is-it/) Evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing (EMNLP-2008), Honolulu, Hawaii.
56. Audrey Tong, Jerome Ajot, Mark Przybocki, and Stephanie Strassel. 2010. [Document image collection using Amazon’s Mechanical Turk](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In NAACL Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk.
57. Luis von Ahn and Laura Dabbish. 2008. General techniques for designing games with a purpose. Communications of the ACM. [find](https://scholar.google.com/scholar?q=Luis+von+Ahn+and+Laura+Dabbish%2E+2008%2E+General+techniques+for+designing+games+with+a+purpose%2E+Communications+of+the+ACM%2E)
58. Rui Wang and Chris Callison-Burch. 2010. [Cheap facts and counter-facts](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In NAACL Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk.
59. Tae Yano, Philip Resnik, and Noah A Smith. 2010. [Shedding (a thousand points of) light on biased language](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In NAACL Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk.
60. Meliha Yetisgen-Yildiz, Imre Solti, Scott Halgrim, and Fei Xia. 2010. [Preliminary experiments with Amazon’s Mechanical Turk for annotating medical named entities](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In NAACL Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk.
61. Omar F. Zaidan and Chris Callison-Burch. 2009. Feasibility of human-in-the-loop minimum error rate training. In Proceedings of EMNLP 2009, pages 52–61, August. [find](https://scholar.google.com/scholar?q=Omar+F%2E+Zaidan+and+Chris+Callison%2DBurch%2E+2009%2E+Feasibility+of+human%2Din%2Dthe%2Dloop+minimum+error+rate+training%2E+In+Proceedings+of+EMNLP+2009%2C+pages+52%E2%80%9361%2C+August%2E)
62. Omar Zaidan and Juri Ganitkevitch. 2010. [An enriched MT grammar for under $100](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In NAACL Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk.

*Updated 24 October 2026*