---
title: "It's All Fun and Games until Someone Annotates: Video Games"
description: "They present two games: Puzzle Racer , a racing-plus-puzzle game in which players pick images that… 40 references, cited by 17 papers here."
authors: ["David Jurgens", "Roberto Navigli"]
year: 2014
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/Q14-1035/"
doi: "10.1162/tacl_a_00195"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/jurgens-2014-it-s-all-fun-games/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jurgens-2014-it-s-all-fun-games/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2014](https://md.research.elamur.ai/free/en/papers/2014/page/2/) / It's All Fun and Games until Someone Annotates: Video Games with a Purpose for Linguistic Annotation

# It's All Fun and Games until Someone Annotates: Video Games with a Purpose for Linguistic Annotation

**Authors:** [David Jurgens](https://md.research.elamur.ai/free/en/authors/david-jurgens/) and [Roberto Navigli](https://md.research.elamur.ai/free/en/authors/roberto-navigli/) · **Year:** 2014 · **Venue:** [Transactions of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/tacl/) ([Transactions of the Association for Computational Linguistics (TACL) 2014](https://md.research.elamur.ai/free/en/venues/tacl/2014/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/Q14-1035/) · [PDF](https://aclanthology.org/Q14-1035.pdf) · [DOI 10.1162/tacl\_a\_00195](https://doi.org/10.1162/tacl_a_00195) · [OpenAlex](https://openalex.org/W1918413956)

**Citations:** 51 citations (OpenAlex, 2026-10-14) · [17](https://md.research.elamur.ai/free/en/papers/jurgens-2014-it-s-all-fun-games/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Building annotated corpora is slow and expensive. Crowdsourcing lowers the cost, but tasks must be simplified for non-experts, and they can be boring enough to drive workers away. Earlier games with a purpose (GWAPs) for NLP mostly wrapped text-based annotation in scores and leaderboards. Jurgens and Navigli reverse this approach. They start from a playable graphical video game and make the annotation task a core game mechanic, with no linguistic terminology shown to players.

They present two games:
- **Puzzle Racer**, a racing-plus-puzzle game in which players pick images that match a theme. This links WordNet senses of nouns, verbs and adjectives, both concrete and abstract, to graded images.
- **Ka-boom!**, which performs Word Sense Disambiguation as players interact with pictures.

Puzzle Racer matched expert annotation quality at substantially lower cost than equivalent crowdsourcing. Ka-boom! clearly beat Wordrobe, a state-of-the-art text-based WSD game. The work shows that real video games can serve as a channel for high-quality semantic annotation. It also supplies reusable image-sense resources for further visual and multilingual games.

## Key points
- Puzzle Racer produced an image library of 19,073 images covering 443 WordNet senses across nouns, verbs and adjectives. Its quality equaled expert ratings at a cost at least 73% lower than crowdsourcing the same resource.
- Ka-boom! improved WSD accuracy by 16.3% over Wordrobe (p \< 0.01). Its gain over the MFS baseline was not statistically significant (p = 0.14).
- Each game yields several sense judgments per context. Ka-boom! surpassed MFS on nouns after only two games, whereas Wordrobe's accuracy fell as more players disagreed.
- Earlier image libraries cover only concrete nouns:
  - ImageNet has 3.2M images for 5,247 noun synsets.
  - BabelNet has images for 38.6% of WordNet senses.
- The released resources include sense-image mappings, image word labels, sense-related Web queries and image-based sense annotations (lcl.uniroma1.it/videogames). The game software is planned for open-source release.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Sapienza University of Rome](https://md.research.elamur.ai/free/en/universities/sapienza-university-of-rome/) — University, Rome, Italy · 57 works · 9,564 citations

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 1,552 works · [💭 Developmental and Educational Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/developmental-and-educational-psychology/) · 499 works · [🖥️ Computer Science Applications](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-science-applications/) · 102 works

## Cite

Cite as: David Jurgens and Roberto Navigli. 2014. It's All Fun and Games until Someone Annotates: Video Games with a Purpose for Linguistic Annotation. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{jurgens2014it,
  title = {{It's All Fun and Games until Someone Annotates: Video Games with a Purpose for Linguistic Annotation}},
  author = {David Jurgens and Roberto Navigli},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2014},
  doi = {10.1162/tacl\_a\_00195},
  url = {https://doi.org/10.1162/tacl_a_00195}
}
```

- [← previous Foreign Words and the Automatic Processing of Arabic Social Media Text Written in Roman Script](https://md.research.elamur.ai/free/en/papers/eskander-2014-foreign-words-automatic-processing-arabic/)
- [next → AraNLP: a Java-based Library for the Processing of Arabic Text.](https://md.research.elamur.ai/free/en/papers/althobaiti-2014-aranlp-java-based-library-processing/)
- [↑ back to papers of 2014](https://md.research.elamur.ai/free/en/papers/2014/page/2/)

## References

40 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Guillaume Artignan, Mountaz Hascoët, and Mathieu Lafourcade. 2009. Multiscale visual analysis of lexical networks. In Proceedings of the International Conference on Information Visualisation, pages 685–690. [find](https://scholar.google.com/scholar?q=Guillaume+Artignan%2C+Mountaz+Hascoe%CC%88t%2C+and+Mathieu+Lafourcade%2E+2009%2E+Multiscale+visual+analysis+of+lexical+networks%2E+In+Proceedings+of+the+International+Conference+on+Information+Visualisation%2C+pages+6)
2. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555–596. [find](https://scholar.google.com/scholar?q=Ron+Artstein+and+Massimo+Poesio%2E+2008%2E+Inter%2Dcoder+agreement+for+computational+linguistics%2E+Computational+Linguistics%2C+34%284%29%3A555%E2%80%93596%2E)
3. Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. [The Berkeley FrameNet project](https://md.research.elamur.ai/free/en/papers/baker-1998-berkeley-framenet/). In Proceedings of the 17th International Conference on Computational Linguistics and 36th Annual Meeting of the Association for Computational Linguistics, Montréal, Québec, Canada, 10–14 August 1998, Montr…
4. Kobus Barnard, Pinar Duygulu, David Forsyth, Nando De Freitas, David M. Blei, and Michael I. Jordan. 2003. Matching words and pictures. The Journal of Machine Learning Research, 3:1107–1135. [find](https://scholar.google.com/scholar?q=Kobus+Barnard%2C+Pinar+Duygulu%2C+David+Forsyth%2C+Nando+De+Freitas%2C+David+M%2E+Blei%2C+and+Michael+I%2E+Jordan%2E+2003%2E+Matching+words+and+pictures%2E+The+Journal+of+Machine+Learning+Research%2C+3%3A1107%E2%80%931135%2E)
5. Chris Biemann and Valerie Nygaard. 2010. Crowdsourcing WordNet. In The 5th International Conference of the Global WordNet Association (GWC-2010). Jon Chamberlain, Karën Fort, Udo Kruschwitz, Mathieu Lafourcade, and Massimo Poesio. 2013. Using games to create language resources: Successes and limita… [find](https://scholar.google.com/scholar?q=Chris+Biemann+and+Valerie+Nygaard%2E+2010%2E+Crowdsourcing+WordNet%2E+In+The+5th+International+Conference+of+the+Global+WordNet+Association+%28GWC%2D2010%29%2E+Jon+Chamberlain%2C+Kare%CC%88n+Fort%2C+Udo+Kruschwitz%2C+Mathieu+)
6. Springer. Tim Chklovski and Yolanda Gil. 2005. Improving the design of intelligent acquisition interfaces for collecting world knowledge from web contributors. In Proceedings of the International Conference on Knowledge Capture, pages 35–42. [find](https://scholar.google.com/scholar?q=Springer%2E+Tim+Chklovski+and+Yolanda+Gil%2E+2005%2E+Improving+the+design+of+intelligent+acquisition+interfaces+for+collecting+world+knowledge+from+web+contributors%2E+In+Proceedings+of+the+International+Conf)
7. ACM. Tim Chklovski and Rada Mihalcea. 2002. Building a Sense Tagged Corpus with Open Mind Word Expert. In Proceedings of ACL 2002 Workshop on WSD: Recent Successes and Future Directions, Philadelphia, PA, USA. Seth Cooper, Firas Khatib, Adrien Treuille, Janos Barbero, Jeehyung Lee, Michael Beenen, A… [find](https://scholar.google.com/scholar?q=ACM%2E+Tim+Chklovski+and+Rada+Mihalcea%2E+2002%2E+Building+a+Sense+Tagged+Corpus+with+Open+Mind+Word+Expert%2E+In+Proceedings+of+ACL+2002+Workshop+on+WSD%3A+Recent+Successes+and+Future+Directions%2C+Philadelphia%2C)
8. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), pages 248–255. [find](https://scholar.google.com/scholar?q=Jia+Deng%2C+Wei+Dong%2C+Richard+Socher%2C+Li%2DJia+Li%2C+Kai+Li%2C+and+Li+Fei%2DFei%2E+2009%2E+ImageNet%3A+A+large%2Dscale+hierarchical+image+database%2E+In+Proceedings+of+the+Conference+on+Computer+Vision+and+Pattern+Recogn)
9. Philip Edmonds and Adam Kilgarriff. 2002. Introduction to the special issue on evaluating word sense disambiguation systems. Natural Language Engineering, 8(4):279–291. [find](https://scholar.google.com/scholar?q=Philip+Edmonds+and+Adam+Kilgarriff%2E+2002%2E+Introduction+to+the+special+issue+on+evaluating+word+sense+disambiguation+systems%2E+Natural+Language+Engineering%2C+8%284%29%3A279%E2%80%93291%2E)
10. Katrin Erk and Diana McCarthy. 2009. Graded word sense assignment. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 440–449, Singapore. [find](https://scholar.google.com/scholar?q=Katrin+Erk+and+Diana+McCarthy%2E+2009%2E+Graded+word+sense+assignment%2E+In+Proceedings+of+the+2009+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+pages+440%E2%80%93449%2C+Singapore%2E)
11. Katrin Erk, Diana McCarthy, and Nicholas Gaylord. 2012. [Measuring word meaning in context](https://md.research.elamur.ai/free/en/papers/erk-2013-measuring-word-meaning-context/). Computational Linguistics, 39(3):511–554.
12. Christiane Fellbaum, Joachim Grabowski, and Shari Landes. 1998. Performance and confidence in a semantic annotation task. In Christiane Fellbaum, editor, Word-Net: An electronic lexical database, pages 217–237. MIT Press. Christiane Fellbaum, editor. 1998. [find](https://scholar.google.com/scholar?q=Christiane+Fellbaum%2C+Joachim+Grabowski%2C+and+Shari+Landes%2E+1998%2E+Performance+and+confidence+in+a+semantic+annotation+task%2E+In+Christiane+Fellbaum%2C+editor%2C+Word%2DNet%3A+An+electronic+lexical+database%2C+page)
13. WordNet: An Electronic Database. MIT Press, Cambridge, MA. Yansong Feng and Mirella Lapata. 2013. Automatic caption generation for news images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(4):797–812. [find](https://scholar.google.com/scholar?q=WordNet%3A+An+Electronic+Database%2E+MIT+Press%2C+Cambridge%2C+MA%2E+Yansong+Feng+and+Mirella+Lapata%2E+2013%2E+Automatic+caption+generation+for+news+images%2E+IEEE+Transactions+on+Pattern+Analysis+and+Machine+Intell)
14. Tiziano Flati and Roberto Navigli. 2013. SPred: Large-scale Harvesting of Semantic Predicates. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL), pages 1222–1232, Sofia, Bulgaria. Amaç Herdağdelen and Marco Baroni. 2012. Bootstrapping a game with a pu… [find](https://scholar.google.com/scholar?q=Tiziano+Flati+and+Roberto+Navigli%2E+2013%2E+SPred%3A+Large%2Dscale+Harvesting+of+Semantic+Predicates%2E+In+Proceedings+of+the+51st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28ACL%29%2C+pages+1)
15. Jisup Hong and Collin F. Baker. 2011. How Good is the Crowd at “real” WSD? In Proceedings of the Fifth Linguistic Annotation Workshop (LAW V), pages 30– 37. [find](https://scholar.google.com/scholar?q=Jisup+Hong+and+Collin+F%2E+Baker%2E+2011%2E+How+Good+is+the+Crowd+at+%E2%80%9Creal%E2%80%9D+WSD%3F+In+Proceedings+of+the+Fifth+Linguistic+Annotation+Workshop+%28LAW+V%29%2C+pages+30%E2%80%93+37%2E)
16. ACL. Eduard H. Hovy, Roberto Navigli, and Simone Paolo Ponzetto. 2013. Collaboratively built semi-structured content and Artificial Intelligence: The story so far. Artificial Intelligence, 194:2–27. [find](https://scholar.google.com/scholar?q=ACL%2E+Eduard+H%2E+Hovy%2C+Roberto+Navigli%2C+and+Simone+Paolo+Ponzetto%2E+2013%2E+Collaboratively+built+semi%2Dstructured+content+and+Artificial+Intelligence%3A+The+story+so+far%2E+Artificial+Intelligence%2C+194%3A2%E2%80%9327%2E)
17. David Jurgens. 2013. Embracing ambiguity: A comparison of annotation methodologies for crowdsourcing word sense labels. In Proceedings of the Conference of the North American Chapter of the Association of Computational Linguistics (NAACL), pages 556–562. [find](https://scholar.google.com/scholar?q=David+Jurgens%2E+2013%2E+Embracing+ambiguity%3A+A+comparison+of+annotation+methodologies+for+crowdsourcing+word+sense+labels%2E+In+Proceedings+of+the+Conference+of+the+North+American+Chapter+of+the+Associatio)
18. Ramesh Krishnamurthy and Diane Nicholls. 2000. Peeling an onion: The lexicographer’s experience of manual sense-tagging. Computers and the Humanities, 34(1-2):85–97. [find](https://scholar.google.com/scholar?q=Ramesh+Krishnamurthy+and+Diane+Nicholls%2E+2000%2E+Peeling+an+onion%3A+The+lexicographer%E2%80%99s+experience+of+manual+sense%2Dtagging%2E+Computers+and+the+Humanities%2C+34%281%2D2%29%3A85%E2%80%9397%2E)
19. Girish Kulkarni, Visruth Premraj, Vicente Ordonez, Sagnik Dhar, Siming Li, Yejin Choi, Alexander C. Berg, and Tamara L. Berg. 2013. Babytalk: Understanding and generating simple image descriptions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(12):2891–2903. [find](https://scholar.google.com/scholar?q=Girish+Kulkarni%2C+Visruth+Premraj%2C+Vicente+Ordonez%2C+Sagnik+Dhar%2C+Siming+Li%2C+Yejin+Choi%2C+Alexander+C%2E+Berg%2C+and+Tamara+L%2E+Berg%2E+2013%2E+Babytalk%3A+Understanding+and+generating+simple+image+descriptions%2E+IE)
20. Yen-ling Kuo, Jong-Chuan Lee, Kai-yang Chiang, Rex Wang, Edward Shen, Cheng-wei Chan, and Jane Yung-jen Hsu. 2009. Community-based game design: experiments on social games for commonsense data collection. In Proceedings of the ACM SIGKDD Workshop on Human Computation, pages 15–22. [find](https://scholar.google.com/scholar?q=Yen%2Dling+Kuo%2C+Jong%2DChuan+Lee%2C+Kai%2Dyang+Chiang%2C+Rex+Wang%2C+Edward+Shen%2C+Cheng%2Dwei+Chan%2C+and+Jane+Yung%2Djen+Hsu%2E+2009%2E+Community%2Dbased+game+design%3A+experiments+on+social+games+for+commonsense+data+collect)
21. Mathieu Lafourcade and Alain Joubert. 2010. Computing trees of named word usages from a crowdsourced lexical network. In Proceedings of the International Multiconference on Computer Science and Information Technology (IMCSIT), pages 439–446, Wisla, Poland. [find](https://scholar.google.com/scholar?q=Mathieu+Lafourcade+and+Alain+Joubert%2E+2010%2E+Computing+trees+of+named+word+usages+from+a+crowdsourced+lexical+network%2E+In+Proceedings+of+the+International+Multiconference+on+Computer+Science+and+Inform)
22. Tak Yeon Lee, Casey Dugan, Werner Geyer, Tristan Ratchford, Jamie Rasmussen, N. Sadat Shami, and Stela Lupushor. 2013. Experiments on motivational feedback for crowdsourced workers. In Seventh International AAAI Conference on Weblogs and Social Media (ICWSM), pages 341–350. [find](https://scholar.google.com/scholar?q=Tak+Yeon+Lee%2C+Casey+Dugan%2C+Werner+Geyer%2C+Tristan+Ratchford%2C+Jamie+Rasmussen%2C+N%2E+Sadat+Shami%2C+and+Stela+Lupushor%2E+2013%2E+Experiments+on+motivational+feedback+for+crowdsourced+workers%2E+In+Seventh+Interna)
23. Thomas Mensink, Jakob J. Verbeek, and Gabriela Csurka. 2013. Tree-structured crf models for interactive image labeling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(2):476–489. [find](https://scholar.google.com/scholar?q=Thomas+Mensink%2C+Jakob+J%2E+Verbeek%2C+and+Gabriela+Csurka%2E+2013%2E+Tree%2Dstructured+crf+models+for+interactive+image+labeling%2E+IEEE+Transactions+on+Pattern+Analysis+and+Machine+Intelligence%2C+35%282%29%3A476%E2%80%93489%2E)
24. George A. Miller, Claudia Leacock, Randee Tengi, and Ross Bunker. 1993. [A semantic concordance](https://md.research.elamur.ai/free/en/papers/miller-1993-semantic-concordance/). In Proceedings of the 3rd DARPA Workshop on Human Language Technology, pages 303–308, Plainsboro, N.J.
25. Roberto Navigli and Simone Paolo Ponzetto. 2012. BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artificial Intelligence, 193:217–250. [find](https://scholar.google.com/scholar?q=Roberto+Navigli+and+Simone+Paolo+Ponzetto%2E+2012%2E+BabelNet%3A+The+automatic+construction%2C+evaluation+and+application+of+a+wide%2Dcoverage+multilingual+semantic+network%2E+Artificial+Intelligence%2C+193%3A217%E2%80%93250)
26. Roberto Navigli. 2009. Word Sense Disambiguation: A survey. ACM Computing Surveys (CSUR), 41(2):1–69. [find](https://scholar.google.com/scholar?q=Roberto+Navigli%2E+2009%2E+Word+Sense+Disambiguation%3A+A+survey%2E+ACM+Computing+Surveys+%28CSUR%29%2C+41%282%29%3A1%E2%80%9369%2E)
27. Martha Palmer, Hoa Dang, and Christiane Fellbaum. 2007. Making fine-grained and coarse-grained sense distinctions, both manually and automatically. Natural Language Engineering, 13(2):137–163. [find](https://scholar.google.com/scholar?q=Martha+Palmer%2C+Hoa+Dang%2C+and+Christiane+Fellbaum%2E+2007%2E+Making+fine%2Dgrained+and+coarse%2Dgrained+sense+distinctions%2C+both+manually+and+automatically%2E+Natural+Language+Engineering%2C+13%282%29%3A137%E2%80%93163%2E)
28. Rebecca J. Passonneau and Bob Carpenter. 2013. [The benefits of a model of annotation](https://md.research.elamur.ai/free/en/papers/passonneau-2014-benefits-model-annotation/). In 7th Linguistic Annotation Workshop and Interoperability with Discourse, August 8–9.
29. Rebecca J. Passonneau, Vikas Bhardwaj, Ansaf Salleb-Aouissi, and Nancy Ide. 2012. Multiplicity and word sense: evaluating and learning from multiply labeled word sense annotations. Language Resources and Evaluation, 46(2):209–252. [find](https://scholar.google.com/scholar?q=Rebecca+J%2E+Passonneau%2C+Vikas+Bhardwaj%2C+Ansaf+Salleb%2DAouissi%2C+and+Nancy+Ide%2E+2012%2E+Multiplicity+and+word+sense%3A+evaluating+and+learning+from+multiply+labeled+word+sense+annotations%2E+Language+Resources+)
30. Ei Pa Pa Pe-Than, DH-L Goh, and Chei Sian Lee. 2012. A survey and typology of human computation games. In Information Technology: New Generations (ITNG), 2012 Ninth International Conference on, pages 720– 725. [find](https://scholar.google.com/scholar?q=Ei+Pa+Pa+Pe%2DThan%2C+DH%2DL+Goh%2C+and+Chei+Sian+Lee%2E+2012%2E+A+survey+and+typology+of+human+computation+games%2E+In+Information+Technology%3A+New+Generations+%28ITNG%29%2C+2012+Ninth+International+Conference+on%2C+pages+)
31. IEEE. Mohammad Taher Pilehvar and Roberto Navigli. 2014. [A Large-scale Pseudoword-based Evaluation Framework for State-of-the-Art Word Sense Disambiguation](https://md.research.elamur.ai/free/en/papers/pilehvar-2014-large-scale-pseudoword-based-evaluation/). Computational Linguistics, 40(4).
32. Massimo Poesio, Jon Chamberlain, Udo Kruschwitz, Livio Robaldo, and Luca Ducceschi. 2013. Phrase detectives: Utilizing collective intelligence for internet-scale language resource creation. ACM Transactions on Interactive Intelligent Systems, 3(1):3:1–3:44, April. [find](https://scholar.google.com/scholar?q=Massimo+Poesio%2C+Jon+Chamberlain%2C+Udo+Kruschwitz%2C+Livio+Robaldo%2C+and+Luca+Ducceschi%2E+2013%2E+Phrase+detectives%3A+Utilizing+collective+intelligence+for+internet%2Dscale+language+resource+creation%2E+ACM+Transa)
33. Nitin Seemakurty, Jonathan Chu, Luis Von Ahn, and Anthony Tomasic. 2010. Word sense disambiguation via human computation. In Proceedings of the ACM SIGKDD Workshop on Human Computation, pages 60–63. [find](https://scholar.google.com/scholar?q=Nitin+Seemakurty%2C+Jonathan+Chu%2C+Luis+Von+Ahn%2C+and+Anthony+Tomasic%2E+2010%2E+Word+sense+disambiguation+via+human+computation%2E+In+Proceedings+of+the+ACM+SIGKDD+Workshop+on+Human+Computation%2C+pages+60%E2%80%9363%2E)
34. ACM. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Ng. 2008. [Cheap and fast – but is it good? evaluating non-expert annotations for natural language tasks](https://md.research.elamur.ai/free/en/papers/snow-2008-cheap-fast-but-is-it/). In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, Waikiki, Honolulu, Hawaii, 25-27 October,…
35. Antonio Torralba, Robert Fergus, and William T Freeman. 2008. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(11):1958–1970. [find](https://scholar.google.com/scholar?q=Antonio+Torralba%2C+Robert+Fergus%2C+and+William+T+Freeman%2E+2008%2E+80+million+tiny+images%3A+A+large+data+set+for+nonparametric+object+and+scene+recognition%2E+IEEE+Transactions+on+Pattern+Analysis+and+Machine)
36. Daniele Vannella, David Jurgens, Daniele Scarfini, Domenico Toscani, and Roberto Navigli. 2014. Validating and extending semantic knowledge bases using video games with a purpose. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 1294–1304. [find](https://scholar.google.com/scholar?q=Daniele+Vannella%2C+David+Jurgens%2C+Daniele+Scarfini%2C+Domenico+Toscani%2C+and+Roberto+Navigli%2E+2014%2E+Validating+and+extending+semantic+knowledge+bases+using+video+games+with+a+purpose%2E+In+Proceedings+of+th)
37. Noortje J. Venhuizen, Valerio Basile, Kilian Evang, and Johan Bos. 2013. Gamification for word sense labeling. In Proceedings of the International Conference on Computational Semantics (IWCS), pages 397–403. [find](https://scholar.google.com/scholar?q=Noortje+J%2E+Venhuizen%2C+Valerio+Basile%2C+Kilian+Evang%2C+and+Johan+Bos%2E+2013%2E+Gamification+for+word+sense+labeling%2E+In+Proceedings+of+the+International+Conference+on+Computational+Semantics+%28IWCS%29%2C+pages+3)
38. Luis von Ahn and Laura Dabbish. 2004. Labeling images with a computer game. In Proceedings of the Conference on Human Factors in Computing Systems (CHI), pages 319–326. [find](https://scholar.google.com/scholar?q=Luis+von+Ahn+and+Laura+Dabbish%2E+2004%2E+Labeling+images+with+a+computer+game%2E+In+Proceedings+of+the+Conference+on+Human+Factors+in+Computing+Systems+%28CHI%29%2C+pages+319%E2%80%93326%2E)
39. Luis von Ahn and Laura Dabbish. 2008. Designing games with a purpose. Communications of the ACM, 51(8):58–67. [find](https://scholar.google.com/scholar?q=Luis+von+Ahn+and+Laura+Dabbish%2E+2008%2E+Designing+games+with+a+purpose%2E+Communications+of+the+ACM%2C+51%288%29%3A58%E2%80%9367%2E)
40. Aobo Wang, Cong Duy Vu Hoang, and Min-Yen Kan. 2013. Perspectives on crowdsourcing annotations for natural language processing. Language Resources and Evaluation, 47(1):9–31. [find](https://scholar.google.com/scholar?q=Aobo+Wang%2C+Cong+Duy+Vu+Hoang%2C+and+Min%2DYen+Kan%2E+2013%2E+Perspectives+on+crowdsourcing+annotations+for+natural+language+processing%2E+Language+Resources+and+Evaluation%2C+47%281%29%3A9%E2%80%9331%2E)

*Updated 30 October 2026*