---
title: "Phrase Detectives Corpus 1.0 Crowdsourced — Chamberlain 2016"
description: "Building coreference resolvers requires large, richly annotated data."
authors: ["Jon Chamberlain", "Massimo Poesio", "Udo Kruschwitz"]
year: 2016
venue: "Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)"
source: "https://aclanthology.org/L16-1323/"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/chamberlain-2016-phrase-detectives-corpus-1-0/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/chamberlain-2016-phrase-detectives-corpus-1-0/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2016](https://md.research.elamur.ai/free/papers/2016/page/2/) / Phrase Detectives Corpus 1.0 Crowdsourced Anaphoric Coreference.

# Phrase Detectives Corpus 1.0 Crowdsourced Anaphoric Coreference.

**Authors:** [Jon Chamberlain](https://md.research.elamur.ai/free/authors/jon-chamberlain/), [Massimo Poesio](https://md.research.elamur.ai/free/authors/massimo-poesio/), [Udo Kruschwitz](https://md.research.elamur.ai/free/authors/udo-kruschwitz/) · **Year:** 2016 · **Venue:** Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/L16-1323/) · [PDF](https://aclanthology.org/L16-1323.pdf) · [OpenAlex](https://openalex.org/W2575233155)

**Citations:** 20 citations (openalex-snapshot · 2026-10-14) · [11](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Building coreference resolvers requires large, richly annotated data. Expert annotation is slow and expensive, so the authors explore whether a crowd of non-experts can do the job. Their data come from Phrase Detectives, an online game-with-a-purpose (GWAP). Players either propose an interpretation for a highlighted markable or vote on interpretations that other players have entered. The resulting release, Phrase Detectives Corpus 1.0, covers copyright-free Wikipedia articles and Project Gutenberg fiction, two genres missing from earlier anaphora resources. Each document carries both expert gold annotations and multiple crowd interpretations. Markables were extracted automatically with an openNLP-based tokeniser/splitter and the Berkeley Parser, then corrected by an administrator. The descriptive analysis shows that narrative texts read more easily and needed fewer markable corrections than encyclopedic ones. Experts agree with each other very closely. Crowd agreement is high on some texts and much weaker on others, and the encyclopedic material proves harder to annotate. Because the release keeps the disagreements between annotators, it supports work on probabilistic coreference models and on learning from crowds.

## Key points
- Corpus 1.0 has 45 documents, 19,886 words and 6,452 markables. It is split into five subsets:
  - GN: GNOME, with an existing gold standard plus one extra expert
  - W2 and G2: gold by two experts
  - W1 and G1: gold by one expert
- Agreement among experts is κ = .88–.93, while the crowd baseline varies from κ = .52 to .96. Encyclopedic texts show lower agreement than narrative texts.
- Activity in the first six years of the game:
  - over 38,000 players registered, and 2,746 passed the training phase
  - 524 documents (302,224 words) were fully annotated
  - this is about 25% of the 1.2M words uploaded for annotation
- Gutenberg (G1) and Wikipedia (W1) differ in readability and correction rates:
  - Flesch Reading Ease is 84.3 for G1 and 50.7 for W1
  - G1 needed fewer markable edits (4.5% vs 9.1%)
  - readability correlates only weakly with parser-induced markable errors
- The data are distributed in MAS-XML via anawiki.essex.ac.uk (ISLRN 955-898-221-547-9).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

Topics: [Coreference resolution](https://md.research.elamur.ai/free/topics/coreference-resolution/)

## Cited by

Cited by

11 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 10 not yet.

- [A Probabilistic Annotation Model for Crowdsourcing Coreference](https://md.research.elamur.ai/free/papers/paun-2018-probabilistic-annotation-model-crowdsourcing-coreference/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{chamberlain2016phrase,
  title = {{Phrase Detectives Corpus 1.0 Crowdsourced Anaphoric Coreference.}},
  author = {Chamberlain, Jon and Poesio, Massimo and Kruschwitz, Udo},
  booktitle = {Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)},
  year = {2016},
  doi = {10.63317/3ha6azwmf87e},
  url = {https://doi.org/10.63317/3ha6azwmf87e}
}
```

- [← previous Participant-Reference Shifts in Zech. 1:1-6. An Assessment of Diachronic and Synchronic Approaches](https://md.research.elamur.ai/free/papers/hbo-2016-jensen-participant-reference-shifts-zech-1/)
- [next → pkudblab at SemEval-2016 Task 6 : A Specific Convolutional Neural Network System for Effective Stance Detection](https://md.research.elamur.ai/free/papers/wei-2016-pkudblab-semeval-2016-task-6/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/papers/2016/page/2/)

## References

23 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Artstein, R. and Poesio, M. (2008). Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555–596. Attardi, G. and the Galoap Team. (2010). Phratris. Demo presented at the 9th International Semantic Web Conference (ISWC’10) tutorial INSEMTIVES’10. [find](https://scholar.google.com/scholar?q=Artstein%2C+R%2E+and+Poesio%2C+M%2E+%282008%29%2E+Inter%2Dcoder+agreement+for+computational+linguistics%2E+Computational+Linguistics%2C+34%284%29%3A555%E2%80%93596%2E+Attardi%2C+G%2E+and+the+Galoap+Team%2E+%282010%29%2E+Phratris%2E+Demo+presented+at+)
2. Chamberlain, J., Poesio, M., and Kruschwitz, U. (2008). Phrase Detectives: A web-based collaborative annotation game. In Proceedings of the 2008 International Conference on Semantic Systems (I-Semantics’08). [find](https://scholar.google.com/scholar?q=Chamberlain%2C+J%2E%2C+Poesio%2C+M%2E%2C+and+Kruschwitz%2C+U%2E+%282008%29%2E+Phrase+Detectives%3A+A+web%2Dbased+collaborative+annotation+game%2E+In+Proceedings+of+the+2008+International+Conference+on+Semantic+Systems+%28I%2DSemanti)
3. Chamberlain, J., Kruschwitz, U., and Poesio, M. (2009). Constructing an anaphorically annotated corpus with non-experts: Assessing the quality of collaborative annotations. In Proceedings of the 4th International Joint Conference on Natural Language Processing (IJC-NLP’09) Workshop on The People’s W… [find](https://scholar.google.com/scholar?q=Chamberlain%2C+J%2E%2C+Kruschwitz%2C+U%2E%2C+and+Poesio%2C+M%2E+%282009%29%2E+Constructing+an+anaphorically+annotated+corpus+with+non%2Dexperts%3A+Assessing+the+quality+of+collaborative+annotations%2E+In+Proceedings+of+the+4th+I)
4. Chamberlain, J., Fort, K., Kruschwitz, U., Mathieu, L., and Poesio, M. (2013). Using games to create language resources: Successes and limitations of the approach. In ACM Transactions on Interactive Intelligent Systems, volume The People’s Web Meets NLP: Collaboratively Constructed Language Resource… [find](https://scholar.google.com/scholar?q=Chamberlain%2C+J%2E%2C+Fort%2C+K%2E%2C+Kruschwitz%2C+U%2E%2C+Mathieu%2C+L%2E%2C+and+Poesio%2C+M%2E+%282013%29%2E+Using+games+to+create+language+resources%3A+Successes+and+limitations+of+the+approach%2E+In+ACM+Transactions+on+Interactive+I)
5. Springer. Chamberlain, J. (2014). The Annotation-Validation (AV) model: Rewarding contribution using retrospective agreement. In Proceedings of the 1st International Workshop on Gamification for Information Retrieval (Gami-fIR’14). [find](https://scholar.google.com/scholar?q=Springer%2E+Chamberlain%2C+J%2E+%282014%29%2E+The+Annotation%2DValidation+%28AV%29+model%3A+Rewarding+contribution+using+retrospective+agreement%2E+In+Proceedings+of+the+1st+International+Workshop+on+Gamification+for+Infor)
6. Chklovski, T. and Gil, Y. (2005). Improving the design of intelligent acquisition interfaces for collecting world knowledge from web contributors. In Proceedings of the 3rd International Conference on Knowledge Capture (K-CAP’05). [find](https://scholar.google.com/scholar?q=Chklovski%2C+T%2E+and+Gil%2C+Y%2E+%282005%29%2E+Improving+the+design+of+intelligent+acquisition+interfaces+for+collecting+world+knowledge+from+web+contributors%2E+In+Proceedings+of+the+3rd+International+Conference+on)
7. Chklovski, T. (2005). Collecting paraphrase corpora from volunteer contributors. In Proceedings of the 3rd International Conference on Knowledge Capture (K-CAP’05). [find](https://scholar.google.com/scholar?q=Chklovski%2C+T%2E+%282005%29%2E+Collecting+paraphrase+corpora+from+volunteer+contributors%2E+In+Proceedings+of+the+3rd+International+Conference+on+Knowledge+Capture+%28K%2DCAP%E2%80%9905%29%2E)
8. Green, N., Breimyer, P., Kumar, V., and Samatova, N. F. (2010). Packplay: Mining semantic data in collaborative games. In Proceedings of the 4th Linguistic Annotation Workshop (LAW IV). Hladká, B., Mı́rovskỳ, J., and Schlesinger, P. (2009). Play the language: Play coreference. In Proceedings of th… [find](https://scholar.google.com/scholar?q=Green%2C+N%2E%2C+Breimyer%2C+P%2E%2C+Kumar%2C+V%2E%2C+and+Samatova%2C+N%2E+F%2E+%282010%29%2E+Packplay%3A+Mining+semantic+data+in+collaborative+games%2E+In+Proceedings+of+the+4th+Linguistic+Annotation+Workshop+%28LAW+IV%29%2E+Hladka%CC%81%2C+B%2E%2C+M)
9. Howe, J. (2008). Crowdsourcing: Why the power of the crowd is driving the future of business. [find](https://scholar.google.com/scholar?q=Howe%2C+J%2E+%282008%29%2E+Crowdsourcing%3A+Why+the+power+of+the+crowd+is+driving+the+future+of+business%2E)
10. Crown Publishing Group. Jovian, L. T. and Amprimo, O. (2011). OCR correction via human computational game. In Proceedings of the 44th Hawaii International Conference on System Sciences (HICSS’11). [find](https://scholar.google.com/scholar?q=Crown+Publishing+Group%2E+Jovian%2C+L%2E+T%2E+and+Amprimo%2C+O%2E+%282011%29%2E+OCR+correction+via+human+computational+game%2E+In+Proceedings+of+the+44th+Hawaii+International+Conference+on+System+Sciences+%28HICSS%E2%80%9911%29%2E)
11. Kincaid, J. P., Fishburne, R. P., Rogers, R. L., and Chissom, B. S. (1975). Derivation of new readability formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy enlisted personnel. Technical report. [find](https://scholar.google.com/scholar?q=Kincaid%2C+J%2E+P%2E%2C+Fishburne%2C+R%2E+P%2E%2C+Rogers%2C+R%2E+L%2E%2C+and+Chissom%2C+B%2E+S%2E+%281975%29%2E+Derivation+of+new+readability+formulas+%28Automated+Readability+Index%2C+Fog+Count+and+Flesch+Reading+Ease+Formula%29+for+Navy+enl)
12. Petrov, S., Barrett, L., Thibaux, R., and Klein, D. (2006). [find](https://scholar.google.com/scholar?q=Petrov%2C+S%2E%2C+Barrett%2C+L%2E%2C+Thibaux%2C+R%2E%2C+and+Klein%2C+D%2E+%282006%29%2E)
13. Learning accurate, compact, and interpretable tree annotation. In Proceedings of the 21st International Conference on Computational Linguistics. Poesio, M. and Artstein, R. (2008). Anaphoric annotation in the ARRAU corpus. In Proceedings of the 6th International Conference on Language Resources and… [find](https://scholar.google.com/scholar?q=Learning+accurate%2C+compact%2C+and+interpretable+tree+annotation%2E+In+Proceedings+of+the+21st+International+Conference+on+Computational+Linguistics%2E+Poesio%2C+M%2E+and+Artstein%2C+R%2E+%282008%29%2E+Anaphoric+annotatio) [our card](https://md.research.elamur.ai/free/papers/aclbulk-2006-petrov-learning-accurate-compact-interpretable-tree/)
14. Poesio, M., Chamberlain, J., Kruschwitz, U., Robaldo, L., and Ducceschi, L. (2012). The Phrase Detective multilingual corpus, release 0.1. In Proceedings of LREC’12 Workshop on Collaborative Resource Development and Delivery. [find](https://scholar.google.com/scholar?q=Poesio%2C+M%2E%2C+Chamberlain%2C+J%2E%2C+Kruschwitz%2C+U%2E%2C+Robaldo%2C+L%2E%2C+and+Ducceschi%2C+L%2E+%282012%29%2E+The+Phrase+Detective+multilingual+corpus%2C+release+0%2E1%2E+In+Proceedings+of+LREC%E2%80%9912+Workshop+on+Collaborative+Resource+)
15. Poesio, M., Chamberlain, J., Kruschwitz, U., Robaldo, L., and Ducceschi, L. (2013). Phrase Detectives: Utilizing collective intelligence for internet-scale language resource creation. ACM Transactions on Interactive Intelligent Systems. [find](https://scholar.google.com/scholar?q=Poesio%2C+M%2E%2C+Chamberlain%2C+J%2E%2C+Kruschwitz%2C+U%2E%2C+Robaldo%2C+L%2E%2C+and+Ducceschi%2C+L%2E+%282013%29%2E+Phrase+Detectives%3A+Utilizing+collective+intelligence+for+internet%2Dscale+language+resource+creation%2E+ACM+Transactions)
16. Poesio, M. (2004a). Discourse annotation and semantic annotation in the GNOME corpus. In Proceedings of the 2004 ACL Workshop on Discourse Annotation. [find](https://scholar.google.com/scholar?q=Poesio%2C+M%2E+%282004a%29%2E+Discourse+annotation+and+semantic+annotation+in+the+GNOME+corpus%2E+In+Proceedings+of+the+2004+ACL+Workshop+on+Discourse+Annotation%2E)
17. Poesio, M. (2004b). The MATE/GNOME scheme for anaphoric annotation, revisited. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue (SIG-DIAL’04). [find](https://scholar.google.com/scholar?q=Poesio%2C+M%2E+%282004b%29%2E+The+MATE%2FGNOME+scheme+for+anaphoric+annotation%2C+revisited%2E+In+Proceedings+of+the+5th+SIGdial+Workshop+on+Discourse+and+Dialogue+%28SIG%2DDIAL%E2%80%9904%29%2E) [our card](https://md.research.elamur.ai/free/papers/poesio-2004-mate-gnome-proposals-anaphoric-annotation/)
18. Pradhan, S. S., Ramshaw, L., Weischedel, R., MacBride, J., and Micciulla, L. (2007). Unrestricted coreference: Identifying entities and events in OntoNotes. In Proceedings of the 1st IEEE International Conference on Semantic Computing (ICSC’07). [find](https://scholar.google.com/scholar?q=Pradhan%2C+S%2E+S%2E%2C+Ramshaw%2C+L%2E%2C+Weischedel%2C+R%2E%2C+MacBride%2C+J%2E%2C+and+Micciulla%2C+L%2E+%282007%29%2E+Unrestricted+coreference%3A+Identifying+entities+and+events+in+OntoNotes%2E+In+Proceedings+of+the+1st+IEEE+Internationa)
19. Rafelsberger, W. and Scharl, A. (2009). Games with a purpose for social networking platforms. In Proceedings of the 20th ACM Conference on Hypertext and Hypermedia. [find](https://scholar.google.com/scholar?q=Rafelsberger%2C+W%2E+and+Scharl%2C+A%2E+%282009%29%2E+Games+with+a+purpose+for+social+networking+platforms%2E+In+Proceedings+of+the+20th+ACM+Conference+on+Hypertext+and+Hypermedia%2E)
20. Raykar, V. C., Yu, S., Zhao, L. H., Valadez, G. H., Florin, C., Bogoni, L., and Moy, L. (2010). Learning from crowds. Journal of Machine Learning Research, 11:1297–1322. Recasens, M., Màrquez, L., Sapena, E., Martı́, M. A., Taulé, M., Hoste, V., Poesio, M., and Versley, Y. (2010). Semeval-2010 tas… [find](https://scholar.google.com/scholar?q=Raykar%2C+V%2E+C%2E%2C+Yu%2C+S%2E%2C+Zhao%2C+L%2E+H%2E%2C+Valadez%2C+G%2E+H%2E%2C+Florin%2C+C%2E%2C+Bogoni%2C+L%2E%2C+and+Moy%2C+L%2E+%282010%29%2E+Learning+from+crowds%2E+Journal+of+Machine+Learning+Research%2C+11%3A1297%E2%80%931322%2E+Recasens%2C+M%2E%2C+Ma%CC%80rquez%2C+L%2E%2C+Sa) [our card](https://md.research.elamur.ai/free/papers/recasens-2009-semeval-2010-task-1-coreference/)
21. Snow, R., O’Connor, B., Jurafsky, D., and Ng, A. Y. (2008). Cheap and fast - but is it good?: Evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing (EMNLP’08). [find](https://scholar.google.com/scholar?q=Snow%2C+R%2E%2C+O%E2%80%99Connor%2C+B%2E%2C+Jurafsky%2C+D%2E%2C+and+Ng%2C+A%2E+Y%2E+%282008%29%2E+Cheap+and+fast+%2D+but+is+it+good%3F%3A+Evaluating+non%2Dexpert+annotations+for+natural+language+tasks%2E+In+Proceedings+of+the+2008+Conference+on+Emp)
22. Venhuizen, N., Basile, V., Evang, K., and Bos, J. (2013). Gamification for word sense labeling. In Proceedings of the 10th International Conference on Computational Semantics (IWCS’13). von Ahn, L. (2006). Games with a purpose. Computer, 39(6):92–94. 11. [find](https://scholar.google.com/scholar?q=Venhuizen%2C+N%2E%2C+Basile%2C+V%2E%2C+Evang%2C+K%2E%2C+and+Bos%2C+J%2E+%282013%29%2E+Gamification+for+word+sense+labeling%2E+In+Proceedings+of+the+10th+International+Conference+on+Computational+Semantics+%28IWCS%E2%80%9913%29%2E+von+Ahn%2C+L%2E+%282)
23. Language Resource References Chamberlain, J., and Poesio, M., and Kruschwitz, U. (2016). Phrase Detectives Corpus 1.0. AnaWiki, University of Essex, ISLRN 955-898-221-547-9. [find](https://scholar.google.com/scholar?q=Language+Resource+References+Chamberlain%2C+J%2E%2C+and+Poesio%2C+M%2E%2C+and+Kruschwitz%2C+U%2E+%282016%29%2E+Phrase+Detectives+Corpus+1%2E0%2E+AnaWiki%2C+University+of+Essex%2C+ISLRN+955%2D898%2D221%2D547%2D9%2E)