---
title: "A Probabilistic Annotation Model for — Paun 2018"
description: "Large coreference corpora are costly to build with expert annotators."
authors: ["Silviu Paun", "Jon Chamberlain", "Udo Kruschwitz", "Juntao Yu", "Massimo Poesio"]
year: 2018
venue: "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/D18-1218/"
doi: "10.18653/v1/d18-1218"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/paun-2018-probabilistic-annotation-model-crowdsourcing-coreference/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/paun-2018-probabilistic-annotation-model-crowdsourcing-coreference/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2018](https://md.research.elamur.ai/free/papers/2018/) / A Probabilistic Annotation Model for Crowdsourcing Coreference

# A Probabilistic Annotation Model for Crowdsourcing Coreference

**Authors:** [Silviu Paun](https://md.research.elamur.ai/free/authors/silviu-paun/), [Jon Chamberlain](https://md.research.elamur.ai/free/authors/jon-chamberlain/), [Udo Kruschwitz](https://md.research.elamur.ai/free/authors/udo-kruschwitz/), [Juntao Yu](https://md.research.elamur.ai/free/authors/juntao-yu/), [Massimo Poesio](https://md.research.elamur.ai/free/authors/massimo-poesio/) · **Year:** 2018 · **Venue:** Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D18-1218/) · [PDF](https://aclanthology.org/D18-1218.pdf) · [DOI 10.18653/v1/d18-1218](https://doi.org/10.18653/v1/d18-1218) · [OpenAlex](https://openalex.org/W2891045077)

**Citations:** 15 citations (openalex-snapshot · 2026-10-14) · [10](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Large coreference corpora are costly to build with expert annotators. Crowdsourcing is a cheaper option, but existing aggregation models assume a fixed label set shared across all items. In anaphoric annotation, the possible labels (antecedents and chains) differ in every document, so those models do not apply. The authors present MPA (Mention Pair Annotations), a Bayesian model for aggregating crowd judgments of anaphora. Each mention's annotations become yes/no decisions for every candidate label, using the binary relevance method. Each annotator gets a sensitivity and specificity per general class (e.g., DO, DN), and parameters are estimated with Variational Inference. The model picks the most probable label for each mention, and chains are formed by following the inferred links. Experiments used Phrase Detectives data and simulated datasets. MPA beat majority voting and the Stanford deterministic resolver. When used to train the Lee et al. (2017) system, its silver chains came closest to gold-chain training. This suggests crowd-built coreference corpora can approach expert quality.

## Key points
- Phrase Detectives contains 843 documents, about 1.2 million tokens and 392,741 markables. Of these, 563 documents (roughly 360,000 tokens) are currently annotated.
- MPA follows the two-step mention pair framework: an unsupervised step infers coreferent pairs, and the pairs are then grouped into chains.
- When Lee et al. (2017) was trained on MPA silver chains, the best score was only about 1 percentage point below training on gold chains.
- Even plain majority-vote chains produced better training data than chains from the Stanford system (Lee et al., 2011).
- On the standard Snow et al. (2008) benchmarks, MPA matched Bayesian Dawid–Skene and MACE: 93 on RTE, 94 on TEMP and 99 on WSD (accuracy, %). Majority vote scored 90, 93 and 99.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

Topics: [Coreference resolution](https://md.research.elamur.ai/free/topics/coreference-resolution/)

## Cited by

Cited by

10 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 8 not yet.

- [A Crowdsourced Corpus of Multiple Judgments and Disagreement on Anaphoric Interpretation](https://md.research.elamur.ai/free/papers/poesio-2019-crowdsourced-corpus-multiple-judgments-disagreement/)
- [Free the Plural: Unrestricted Split-Antecedent Anaphora Resolution](https://md.research.elamur.ai/free/papers/yu-2020-free-plural-unrestricted-split-antecedent/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{paun2018probabilistic,
  title = {{A Probabilistic Annotation Model for Crowdsourcing Coreference}},
  author = {Paun, Silviu and Chamberlain, Jon and Kruschwitz, Udo and Yu, Juntao and Poesio, Massimo},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1218},
  url = {https://doi.org/10.18653/v1/d18-1218}
}
```

- [← previous A Predictive Model for Notional Anaphora in English](https://md.research.elamur.ai/free/papers/zeldes-2018-predictive-model-notional-anaphora-english/)
- [next → A Retrospective Analysis of the Fake News Challenge Stance-Detection Task](https://md.research.elamur.ai/free/papers/hanselowski-2018-retrospective-analysis-fake-news-challenge/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/papers/2018/)

## References

52 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555–596. An early version of this paper has been circulating since 2005 as “Kappa 3 = Alpha (or Beta)”. This version is still available from the ARRAU website. [find](https://scholar.google.com/scholar?q=Ron+Artstein+and+Massimo+Poesio%2E+2008%2E+Inter%2Dcoder+agreement+for+computational+linguistics%2E+Computational+Linguistics%2C+34%284%29%3A555%E2%80%93596%2E+An+early+version+of+this+paper+has+been+circulating+since+2005+as+)
2. Breck Baldwin and Thomas S. Morton. 1998. Dynamic coreference-based summarization. In Proceedings of the Third Conference on Empirical Methods for Natural Language Processing, pages 1– 6. [find](https://scholar.google.com/scholar?q=Breck+Baldwin+and+Thomas+S%2E+Morton%2E+1998%2E+Dynamic+coreference%2Dbased+summarization%2E+In+Proceedings+of+the+Third+Conference+on+Empirical+Methods+for+Natural+Language+Processing%2C+pages+1%E2%80%93+6%2E) [our card](https://md.research.elamur.ai/free/papers/baldwin-1998-dynamic-coreference-based-summarization/)
3. Anders Björkelund and Jonas Kuhn. 2014. Learning structured perceptrons for coreference resolution with latent antecedents and non-local features. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 47–57. [find](https://scholar.google.com/scholar?q=Anders+Bjo%CC%88rkelund+and+Jonas+Kuhn%2E+2014%2E+Learning+structured+perceptrons+for+coreference+resolution+with+latent+antecedents+and+non%2Dlocal+features%2E+In+Proceedings+of+the+52nd+Annual+Meeting+of+the+Ass) [our card](https://md.research.elamur.ai/free/papers/bjorkelund-2014-learning-structured-perceptrons-coreference-resolution/)
4. David M. Blei and Michael I. Jordan. 2006. Variational inference for Dirichlet process mixtures. Bayesian Analysis, 1(1):121–143. [find](https://scholar.google.com/scholar?q=David+M%2E+Blei+and+Michael+I%2E+Jordan%2E+2006%2E+Variational+inference+for+Dirichlet+process+mixtures%2E+Bayesian+Analysis%2C+1%281%29%3A121%E2%80%93143%2E)
5. David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. 2017. Variational inference: A review for statisticians. Journal of the American Statistical Association, 112(518):859–877. [find](https://scholar.google.com/scholar?q=David+M%2E+Blei%2C+Alp+Kucukelbir%2C+and+Jon+D%2E+McAuliffe%2E+2017%2E+Variational+inference%3A+A+review+for+statisticians%2E+Journal+of+the+American+Statistical+Association%2C+112%28518%29%3A859%E2%80%93877%2E)
6. Matthew D. Hoffman, David M. Blei, Chong Wang, and John Paisley. 2013. Stochastic variational inference. The Journal of Machine Learning Research, 14(1):1303–1347. [find](https://scholar.google.com/scholar?q=Matthew+D%2E+Hoffman%2C+David+M%2E+Blei%2C+Chong+Wang%2C+and+John+Paisley%2E+2013%2E+Stochastic+variational+inference%2E+The+Journal+of+Machine+Learning+Research%2C+14%281%29%3A1303%E2%80%931347%2E)
7. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet allocation. The Journal of Machine Learning Research, 3:993–1022. [find](https://scholar.google.com/scholar?q=David+M%2E+Blei%2C+Andrew+Y%2E+Ng%2C+and+Michael+I%2E+Jordan%2E+2003%2E+Latent+Dirichlet+allocation%2E+The+Journal+of+Machine+Learning+Research%2C+3%3A993%E2%80%931022%2E) [our card](https://md.research.elamur.ai/free/papers/blei-2003-latent-dirichlet-allocation/)
8. Veronique Hoste. 2016. The mention-pair model. In Anaphora Resolution: Algorithms, Resources and Applications. Springer. [find](https://scholar.google.com/scholar?q=Veronique+Hoste%2E+2016%2E+The+mention%2Dpair+model%2E+In+Anaphora+Resolution%3A+Algorithms%2C+Resources+and+Applications%2E+Springer%2E)
9. Johan Bos, Valerio Basile, Kilian Evang, Noortje J. Venhuizen, and Johannes Bjerva. 2017. The Groningen Meaning Bank. In The Handbook of Linguistic Annotation, chapter 18, pages 463–496. Springer. [find](https://scholar.google.com/scholar?q=Johan+Bos%2C+Valerio+Basile%2C+Kilian+Evang%2C+Noortje+J%2E+Venhuizen%2C+and+Johannes+Bjerva%2E+2017%2E+The+Groningen+Meaning+Bank%2E+In+The+Handbook+of+Linguistic+Annotation%2C+chapter+18%2C+pages+463%E2%80%93496%2E+Springer%2E)
10. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom to trust with MACE. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1120–1130. [find](https://scholar.google.com/scholar?q=Dirk+Hovy%2C+Taylor+Berg%2DKirkpatrick%2C+Ashish+Vaswani%2C+and+Eduard+Hovy%2E+2013%2E+Learning+whom+to+trust+with+MACE%2E+In+Proceedings+of+the+2013+Conference+of+the+North+American+Chapter+of+the+Association+for+) [our card](https://md.research.elamur.ai/free/papers/hovy-2013-learning-whom-trust-mace/)
11. Bob Carpenter. 2008. Multilevel Bayesian models of categorical data annotation. Available at [http://lingpipe-blog.com/lingpipe-white-papers](http://lingpipe-blog.com/lingpipe-white-papers). [↗](http://lingpipe-blog.com/lingpipe-white-papers)
12. Jon Chamberlain. 2016. Harnessing Collective Intelligence on Social Networks. Ph.D. thesis, University of Essex, School of Computer Science and Electronic Engineering. [find](https://scholar.google.com/scholar?q=Jon+Chamberlain%2E+2016%2E+Harnessing+Collective+Intelligence+on+Social+Networks%2E+Ph%2ED%2E+thesis%2C+University+of+Essex%2C+School+of+Computer+Science+and+Electronic+Engineering%2E)
13. Jon Chamberlain, Massimo Poesio, and Udo Kruschwitz. 2016. Phrase detectives corpus 1.0: Crowdsourced anaphoric coreference. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2016), Portoroz, Slovenia. [find](https://scholar.google.com/scholar?q=Jon+Chamberlain%2C+Massimo+Poesio%2C+and+Udo+Kruschwitz%2E+2016%2E+Phrase+detectives+corpus+1%2E0%3A+Crowdsourced+anaphoric+coreference%2E+In+Proceedings+of+the+International+Conference+on+Language+Resources+and+Ev) [our card](https://md.research.elamur.ai/free/papers/chamberlain-2016-phrase-detectives-corpus-1-0/)
14. Nancy A. Chinchor. 1998. Overview of MUC-7/MET-2. In Proceedings of the 7 th Message Understanding Conference (MUC7). [find](https://scholar.google.com/scholar?q=Nancy+A%2E+Chinchor%2E+1998%2E+Overview+of+MUC%2D7%2FMET%2D2%2E+In+Proceedings+of+the+7+th+Message+Understanding+Conference+%28MUC7%29%2E)
15. Kevin Clark and Christopher D. Manning. 2016. Improving coreference resolution by learning entity-level distributed representations. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). [find](https://scholar.google.com/scholar?q=Kevin+Clark+and+Christopher+D%2E+Manning%2E+2016%2E+Improving+coreference+resolution+by+learning+entity%2Dlevel+distributed+representations%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Co) [our card](https://md.research.elamur.ai/free/papers/clark-2016-improving-coreference-resolution-learning-entity/)
16. Alexander Philip Dawid and Allan M. Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics, 28(1):20–28. [find](https://scholar.google.com/scholar?q=Alexander+Philip+Dawid+and+Allan+M%2E+Skene%2E+1979%2E+Maximum+likelihood+estimation+of+observer+error%2Drates+using+the+EM+algorithm%2E+Applied+Statistics%2C+28%281%29%3A20%E2%80%9328%2E)
17. Paul Felt, Robbie Haertel, Eric K. Ringger, and Kevin D. Seppi. 2014. MOMRESP: A Bayesian model for multi-annotator document labeling. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2014), Reykjavik. [find](https://scholar.google.com/scholar?q=Paul+Felt%2C+Robbie+Haertel%2C+Eric+K%2E+Ringger%2C+and+Kevin+D%2E+Seppi%2E+2014%2E+MOMRESP%3A+A+Bayesian+model+for+multi%2Dannotator+document+labeling%2E+In+Proceedings+of+the+International+Conference+on+Language+Resour)
18. Ralph Grishman and Beth Sundheim. 1995. Design of the MUC-6 evaluation. In Proceedings of the 6th Conference on Message Understanding, MUC6 ’95, pages 1–11, Stroudsburg, PA, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ralph+Grishman+and+Beth+Sundheim%2E+1995%2E+Design+of+the+MUC%2D6+evaluation%2E+In+Proceedings+of+the+6th+Conference+on+Message+Understanding%2C+MUC6+%E2%80%9995%2C+pages+1%E2%80%9311%2C+Stroudsburg%2C+PA%2C+USA%2E+Association+for+Compu)
19. Anupam Guha, Mohit Iyyer, Danny Bouman, and Jordan Boyd-Graber. 2015. Removing the training wheels: A coreference dataset that entertains humans and challenges computers. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu… [find](https://scholar.google.com/scholar?q=Anupam+Guha%2C+Mohit+Iyyer%2C+Danny+Bouman%2C+and+Jordan+Boyd%2DGraber%2E+2015%2E+Removing+the+training+wheels%3A+A+coreference+dataset+that+entertains+humans+and+challenges+computers%2E+In+Proceedings+of+the+2015+Co) [our card](https://md.research.elamur.ai/free/papers/guha-2015-removing-training-wheels-coreference-dataset/)
20. Kevin Humphreys, Robert Gaizauskas, and Saliha Azzam. Event coreference for information extraction. In Proceedings of a Workshop on Operational Factors in Practical, Robust Anaphora Resolution for Unrestricted Texts. [find](https://scholar.google.com/scholar?q=Kevin+Humphreys%2C+Robert+Gaizauskas%2C+and+Saliha+Azzam%2E+Event+coreference+for+information+extraction%2E+In+Proceedings+of+a+Workshop+on+Operational+Factors+in+Practical%2C+Robust+Anaphora+Resolution+for+Unr) [our card](https://md.research.elamur.ai/free/papers/humphreys-1997-event-coreference-information-extraction/)
21. Heeyoung Lee, Yves Peirsman, Angel Chang, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2011. Stanford’s multi-pass sieve coreference resolution system at the CoNLL-2011 shared task. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, CONLL Sha… [find](https://scholar.google.com/scholar?q=Heeyoung+Lee%2C+Yves+Peirsman%2C+Angel+Chang%2C+Nathanael+Chambers%2C+Mihai+Surdeanu%2C+and+Dan+Jurafsky%2E+2011%2E+Stanford%E2%80%99s+multi%2Dpass+sieve+coreference+resolution+system+at+the+CoNLL%2D2011+shared+task%2E+In+Procee) [our card](https://md.research.elamur.ai/free/papers/lee-2011-stanford-s-multi-pass-sieve/)
22. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188–197. [find](https://scholar.google.com/scholar?q=Kenton+Lee%2C+Luheng+He%2C+Mike+Lewis%2C+and+Luke+Zettlemoyer%2E+2017%2E+End%2Dto%2Dend+neural+coreference+resolution%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+page) [our card](https://md.research.elamur.ai/free/papers/lee-2017-end-end-neural-coreference-resolution/)
23. Gjorgji Madjarov, Dragi Kocev, Dejan Gjorgjevikj, and SašO Deroski. 2012. An extensive experimental comparison of methods for multi-label learning. Pattern Recognition, 45(9):3084–3104. [find](https://scholar.google.com/scholar?q=Gjorgji+Madjarov%2C+Dragi+Kocev%2C+Dejan+Gjorgjevikj%2C+and+Sas%CC%8CO+Deroski%2E+2012%2E+An+extensive+experimental+comparison+of+methods+for+multi%2Dlabel+learning%2E+Pattern+Recognition%2C+45%289%29%3A3084%E2%80%933104%2E)
24. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55–60. [find](https://scholar.google.com/scholar?q=Christopher+D%2E+Manning%2C+Mihai+Surdeanu%2C+John+Bauer%2C+Jenny+Finkel%2C+Steven+J%2E+Bethard%2C+and+David+Mc%2DClosky%2E+2014%2E+The+Stanford+CoreNLP+natural+language+processing+toolkit%2E+In+Association+for+Computation) [our card](https://md.research.elamur.ai/free/papers/manning-2014-stanford-corenlp-natural-language-processing/)
25. Sebastian Martschat and Michael Strube. 2015. Latent structures for coreference resolution. Transactions of the Association of Computational Linguistics, 3(1):405–418. [find](https://scholar.google.com/scholar?q=Sebastian+Martschat+and+Michael+Strube%2E+2015%2E+Latent+structures+for+coreference+resolution%2E+Transactions+of+the+Association+of+Computational+Linguistics%2C+3%281%29%3A405%E2%80%93418%2E) [our card](https://md.research.elamur.ai/free/papers/martschat-2015-latent-structures-coreference-resolution/)
26. Thomas S. Morton. 1999. Using coreference for question answering. In Proceedings of the Workshop on Coreference and Its Applications, CorefApp ’99, pages 85–89, Stroudsburg, PA, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Thomas+S%2E+Morton%2E+1999%2E+Using+coreference+for+question+answering%2E+In+Proceedings+of+the+Workshop+on+Coreference+and+Its+Applications%2C+CorefApp+%E2%80%9999%2C+pages+85%E2%80%9389%2C+Stroudsburg%2C+PA%2C+USA%2E+Association+for+C) [our card](https://md.research.elamur.ai/free/papers/morton-1999-coreference-question-answering/)
27. Malvina Nissim, Shipra Dingare, Jean Carletta, and Mark Steedman. 2004. An annotation scheme for information status in dialogue. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2004). [find](https://scholar.google.com/scholar?q=Malvina+Nissim%2C+Shipra+Dingare%2C+Jean+Carletta%2C+and+Mark+Steedman%2E+2004%2E+An+annotation+scheme+for+information+status+in+dialogue%2E+In+Proceedings+of+the+International+Conference+on+Language+Resources+an)
28. Rebecca J. Passonneau. 2004. Computing reliability for coreference annotation. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2004), Lisbon. [find](https://scholar.google.com/scholar?q=Rebecca+J%2E+Passonneau%2E+2004%2E+Computing+reliability+for+coreference+annotation%2E+In+Proceedings+of+the+International+Conference+on+Language+Resources+and+Evaluation+%28LREC+2004%29%2C+Lisbon%2E)
29. Rebecca J. Passonneau and Bob Carpenter. 2014. The benefits of a model of annotation. Transactions of the Association for Computational Linguistics, 2:311–326. [find](https://scholar.google.com/scholar?q=Rebecca+J%2E+Passonneau+and+Bob+Carpenter%2E+2014%2E+The+benefits+of+a+model+of+annotation%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+2%3A311%E2%80%93326%2E)
30. Barbara Plank, Dirk Hovy, and Anders Sogaard. 2014. Linguistically debatable or just plain wrong? In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). [find](https://scholar.google.com/scholar?q=Barbara+Plank%2C+Dirk+Hovy%2C+and+Anders+Sogaard%2E+2014%2E+Linguistically+debatable+or+just+plain+wrong%3F+In+Proceedings+of+the+52nd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+2%3A+)
31. Massimo Poesio and Ron Artstein. 2005. The reliability of anaphoric annotation, reconsidered: Taking ambiguity into account. In Proceedings of ACL Workshop on Frontiers in Corpus Annotation, pages 76–83. [find](https://scholar.google.com/scholar?q=Massimo+Poesio+and+Ron+Artstein%2E+2005%2E+The+reliability+of+anaphoric+annotation%2C+reconsidered%3A+Taking+ambiguity+into+account%2E+In+Proceedings+of+ACL+Workshop+on+Frontiers+in+Corpus+Annotation%2C+pages+76%E2%80%93) [our card](https://md.research.elamur.ai/free/papers/poesio-2005-reliability-anaphoric-annotation-reconsidered-taking/)
32. Massimo Poesio, Jon Chamberlain, and Udo Kruschwitz. 2017. Crowdsourcing. In Handbook of Linguistic Annotation. Springer. [find](https://scholar.google.com/scholar?q=Massimo+Poesio%2C+Jon+Chamberlain%2C+and+Udo+Kruschwitz%2E+2017%2E+Crowdsourcing%2E+In+Handbook+of+Linguistic+Annotation%2E+Springer%2E)
33. Massimo Poesio, Jon Chamberlain, Udo Kruschwitz, Livio Robaldo, and Luca Ducceschi. 2013. Phrase Detectives: Utilizing collective intelligence for internet-scale language resource creation. ACM Transactions on Intelligent Interactive Systems, 3(1). [find](https://scholar.google.com/scholar?q=Massimo+Poesio%2C+Jon+Chamberlain%2C+Udo+Kruschwitz%2C+Livio+Robaldo%2C+and+Luca+Ducceschi%2E+2013%2E+Phrase+Detectives%3A+Utilizing+collective+intelligence+for+internet%2Dscale+language+resource+creation%2E+ACM+Transa)
34. Massimo Poesio, Yulia Grishina, Varada Kolhatkar, Nafise Moosavi, Ina Roesiger, Adam Roussel, Fabian Simonjetz, Alexandra Uma, Olga Uryupina, Juntao Yu, and Heike Zinsmeister. 2018. Anaphora resolution with the ARRAU corpus. In Proceedings of the NAACL Workshop on Computational Models of Reference,… [find](https://scholar.google.com/scholar?q=Massimo+Poesio%2C+Yulia+Grishina%2C+Varada+Kolhatkar%2C+Nafise+Moosavi%2C+Ina+Roesiger%2C+Adam+Roussel%2C+Fabian+Simonjetz%2C+Alexandra+Uma%2C+Olga+Uryupina%2C+Juntao+Yu%2C+and+Heike+Zinsmeister%2E+2018%2E+Anaphora+resolutio) [our card](https://md.research.elamur.ai/free/papers/poesio-2018-anaphora-resolution-arrau-corpus/)
35. Massimo Poesio, Sameer Pradhan, Marta Recasens, Kepa Rodriguez, and Yannick Versley. 2016a. Annotated corpora and annotation tools. In Anaphora Resolution: Algorithms, Resources and Applications. Springer. [find](https://scholar.google.com/scholar?q=Massimo+Poesio%2C+Sameer+Pradhan%2C+Marta+Recasens%2C+Kepa+Rodriguez%2C+and+Yannick+Versley%2E+2016a%2E+Annotated+corpora+and+annotation+tools%2E+In+Anaphora+Resolution%3A+Algorithms%2C+Resources+and+Applications%2E+Spri)
36. Massimo Poesio, Roland Stuckardt, and Yannick Versley. 2016b. Anaphora Resolution: Algorithms, Resources and Applications. Springer, Berlin. [find](https://scholar.google.com/scholar?q=Massimo+Poesio%2C+Roland+Stuckardt%2C+and+Yannick+Versley%2E+2016b%2E+Anaphora+Resolution%3A+Algorithms%2C+Resources+and+Applications%2E+Springer%2C+Berlin%2E)
37. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Proceedings of the Sixteenth Conference on Computational Natural Language Learning (CoNLL 2012), Jeju, Korea. [find](https://scholar.google.com/scholar?q=Sameer+Pradhan%2C+Alessandro+Moschitti%2C+Nianwen+Xue%2C+Olga+Uryupina%2C+and+Yuchen+Zhang%2E+2012%2E+CoNLL%2D2012+shared+task%3A+Modeling+multilingual+unrestricted+coreference+in+OntoNotes%2E+In+Proceedings+of+the+Six) [our card](https://md.research.elamur.ai/free/papers/pradhan-2012-conll-2012-shared-task-modeling/)
38. Sameer S. Pradhan, Eduard Hovy, Mitch Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2007. Ontonotes: A unified relational semantic representation. In International Conference on Semantic Computing (ICSC 2007), pages 517–526. [find](https://scholar.google.com/scholar?q=Sameer+S%2E+Pradhan%2C+Eduard+Hovy%2C+Mitch+Marcus%2C+Martha+Palmer%2C+Lance+Ramshaw%2C+and+Ralph+Weischedel%2E+2007%2E+Ontonotes%3A+A+unified+relational+semantic+representation%2E+In+International+Conference+on+Semantic)
39. Nguyen Quoc Viet Hung, Nguyen Thanh Tam, Lam Ngoc Tran, and Karl Aberer. 2013. An evaluation of aggregation techniques in crowdsourcing. In Web Information Systems Engineering – WISE 2013, pages 1–15, Berlin, Heidelberg. Springer Berlin Heidelberg. [find](https://scholar.google.com/scholar?q=Nguyen+Quoc+Viet+Hung%2C+Nguyen+Thanh+Tam%2C+Lam+Ngoc+Tran%2C+and+Karl+Aberer%2E+2013%2E+An+evaluation+of+aggregation+techniques+in+crowdsourcing%2E+In+Web+Information+Systems+Engineering+%E2%80%93+WISE+2013%2C+pages+1%E2%80%9315%2C)
40. Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bo-goni, and Linda Moy. 2010. Learning from crowds. Journal of Machine Learning Research, 11:1297– 1322. [find](https://scholar.google.com/scholar?q=Vikas+C%2E+Raykar%2C+Shipeng+Yu%2C+Linda+H%2E+Zhao%2C+Gerardo+Hermosillo+Valadez%2C+Charles+Florin%2C+Luca+Bo%2Dgoni%2C+and+Linda+Moy%2E+2010%2E+Learning+from+crowds%2E+Journal+of+Machine+Learning+Research%2C+11%3A1297%E2%80%93+1322%2E)
41. Arndt Riester, David Lorenz, and Nina Seeman. 2010. A recursive annotation scheme for referential information status. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2010), pages 717–722. [find](https://scholar.google.com/scholar?q=Arndt+Riester%2C+David+Lorenz%2C+and+Nina+Seeman%2E+2010%2E+A+recursive+annotation+scheme+for+referential+information+status%2E+In+Proceedings+of+the+International+Conference+on+Language+Resources+and+Evaluatio)
42. Aashish Sheshadri and Matthew Lease. 2013. SQUARE: A Benchmark for Research on Computing Crowd Consensus. In Proceedings of the 1st AAAI Conference on Human Computation (HCOMP), pages 156–164. [find](https://scholar.google.com/scholar?q=Aashish+Sheshadri+and+Matthew+Lease%2E+2013%2E+SQUARE%3A+A+Benchmark+for+Research+on+Computing+Crowd+Consensus%2E+In+Proceedings+of+the+1st+AAAI+Conference+on+Human+Computation+%28HCOMP%29%2C+pages+156%E2%80%93164%2E)
43. Padhraic Smyth, Usama Fayyad, Michael Burl, Pietro Perona, and Pierre Baldi. 1994. Inferring ground truth from subjective labelling of Venus images. In Proceedings of the 7th International Conference on Neural Information Processing Systems, NIPS’94, pages 1085–1092, Cambridge, MA, USA. MIT Press. [find](https://scholar.google.com/scholar?q=Padhraic+Smyth%2C+Usama+Fayyad%2C+Michael+Burl%2C+Pietro+Perona%2C+and+Pierre+Baldi%2E+1994%2E+Inferring+ground+truth+from+subjective+labelling+of+Venus+images%2E+In+Proceedings+of+the+7th+International+Conference+)
44. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast - but is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 254–263. [find](https://scholar.google.com/scholar?q=Rion+Snow%2C+Brendan+O%E2%80%99Connor%2C+Daniel+Jurafsky%2C+and+Andrew+Y%2E+Ng%2E+2008%2E+Cheap+and+fast+%2D+but+is+it+good%3F+Evaluating+non%2Dexpert+annotations+for+natural+language+tasks%2E+In+Proceedings+of+the+Conference+on)
45. Wee Meng Soon, Hwee Tou Ng, and Daniel Chung Yong Lim. 2001. A machine learning approach to coreference resolution of noun phrases. Computational Linguistics, 27(4):521–544. [find](https://scholar.google.com/scholar?q=Wee+Meng+Soon%2C+Hwee+Tou+Ng%2C+and+Daniel+Chung+Yong+Lim%2E+2001%2E+A+machine+learning+approach+to+coreference+resolution+of+noun+phrases%2E+Computational+Linguistics%2C+27%284%29%3A521%E2%80%93544%2E) [our card](https://md.research.elamur.ai/free/papers/soon-2001-machine-learning-approach-coreference-resolution/)
46. Josef Steinberger, Massimo Poesio, Mijail Kabadjov, and Karel Jezek. 2007. Two uses of anaphora resolution in summarization. Information Processing and Management, 43(6):1663–1680. Special issue on Summarization. [find](https://scholar.google.com/scholar?q=Josef+Steinberger%2C+Massimo+Poesio%2C+Mijail+Kabadjov%2C+and+Karel+Jezek%2E+2007%2E+Two+uses+of+anaphora+resolution+in+summarization%2E+Information+Processing+and+Management%2C+43%286%29%3A1663%E2%80%931680%2E+Special+issue+on+Su)
47. Grigorios Tsoumakas and Ioannis Katakis. 2007. Multi-label classification: An overview. International Journal of Data Warehousing and Mining (IJDWM), 2007:1–13. [find](https://scholar.google.com/scholar?q=Grigorios+Tsoumakas+and+Ioannis+Katakis%2E+2007%2E+Multi%2Dlabel+classification%3A+An+overview%2E+International+Journal+of+Data+Warehousing+and+Mining+%28IJDWM%29%2C+2007%3A1%E2%80%9313%2E)
48. Ralph Weischedel, Eduard Hovy, Mitchell Marcus, Martha Palmer, Robert Belvin, Sameer Pradhan, Lance Ramshaw, and Nianwen Xue. 2011. Ontonotes: A large training corpus for enhanced processing. Handbook of Natural Language Processing and Machine Translation. Springer. [find](https://scholar.google.com/scholar?q=Ralph+Weischedel%2C+Eduard+Hovy%2C+Mitchell+Marcus%2C+Martha+Palmer%2C+Robert+Belvin%2C+Sameer+Pradhan%2C+Lance+Ramshaw%2C+and+Nianwen+Xue%2E+2011%2E+Ontonotes%3A+A+large+training+corpus+for+enhanced+processing%2E+Handbook)
49. Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier R. Movellan, and Paul L. Ruvolo. 2009. [find](https://scholar.google.com/scholar?q=Jacob+Whitehill%2C+Ting%2Dfan+Wu%2C+Jacob+Bergsma%2C+Javier+R%2E+Movellan%2C+and+Paul+L%2E+Ruvolo%2E+2009%2E)
50. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems 22, pages 2035–2043. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Whose+vote+should+count+more%3A+Optimal+integration+of+labels+from+labelers+of+unknown+expertise%2E+In+Advances+in+Neural+Information+Processing+Systems+22%2C+pages+2035%E2%80%932043%2E+Curran+Associates%2C+Inc%2E)
51. Zhiping Zheng. 2002. Answerbus question answering system. In Proceedings of the Second International Conference on Human Language Technology Research, HLT ’02, pages 399–404, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. [find](https://scholar.google.com/scholar?q=Zhiping+Zheng%2E+2002%2E+Answerbus+question+answering+system%2E+In+Proceedings+of+the+Second+International+Conference+on+Human+Language+Technology+Research%2C+HLT+%E2%80%9902%2C+pages+399%E2%80%93404%2C+San+Francisco%2C+CA%2C+USA%2E+M)
52. A Similar steps were taken to derive the variational parameters associated with the sensitivity α and specificity β. In Equation (12) we derive the complete conditional associated with the positive outcome of the true label indicator: p(c i,m = 1\|...) ∝ p(c i,m = 1\|π z i,m )× × [find](https://scholar.google.com/scholar?q=A+Similar+steps+were+taken+to+derive+the+variational+parameters+associated+with+the+sensitivity+%CE%B1+and+specificity+%CE%B2%2E+In+Equation+%2812%29+we+derive+the+complete+conditional+associated+with+the+positive+ou)