---
title: "Beyond Black & White — paper by Fornaciari et al. (2021)"
description: "Fornaciari et al. propose a different use of annotator disagreement. 85.73% accuracy. 39 references, cited by 78 papers here."
authors: ["Tommaso Fornaciari", "Alexandra Uma", "Silviu Paun", "Barbara Plank", "Dirk Hovy", "Massimo Poesio"]
year: 2021
venue: "Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies"
source: "https://aclanthology.org/2021.naacl-main.204/"
doi: "10.18653/v1/2021.naacl-main.204"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/fornaciari-2021-beyond-black-white-leveraging-annotator/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/fornaciari-2021-beyond-black-white-leveraging-annotator/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / Beyond Black & White: Leveraging Annotator Disagreement via Soft-Label Multi-Task Learning

# Beyond Black & White: Leveraging Annotator Disagreement via Soft-Label Multi-Task Learning

**Authors:** [Tommaso Fornaciari](https://md.research.elamur.ai/free/en/authors/tommaso-fornaciari/), [Alexandra Uma](https://md.research.elamur.ai/free/en/authors/alexandra-uma/), [Silviu Paun](https://md.research.elamur.ai/free/en/authors/silviu-paun/), [Barbara Plank](https://md.research.elamur.ai/free/en/authors/barbara-plank/), [Dirk Hovy](https://md.research.elamur.ai/free/en/authors/dirk-hovy/) and [Massimo Poesio](https://md.research.elamur.ai/free/en/authors/massimo-poesio/) · **Year:** 2021 · **Venue:** [Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2021](https://md.research.elamur.ai/free/en/venues/naacl/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.naacl-main.204/) · [PDF](https://aclanthology.org/2021.naacl-main.204.pdf) · [DOI 10.18653/v1/2021.naacl-main.204](https://doi.org/10.18653/v1/2021.naacl-main.204) · [OpenAlex](https://openalex.org/W3166716308)

**Citations:** 65 citations (OpenAlex, 2026-10-14) · [78](https://md.research.elamur.ai/free/en/papers/fornaciari-2021-beyond-black-white-leveraging-annotator/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Supervised NLP models usually train against a single "correct" label per item, even though human annotators often disagree. This hides genuine ambiguity and can cause overfitting. Fornaciari et al. propose a different use of annotator disagreement. Alongside the usual gold-label classification, a multi-task network learns an auxiliary task: predicting the soft label, meaning the distribution of labels the annotators gave. The authors compare three losses for this auxiliary output: standard KL-divergence, inverse KL-divergence and Cross-Entropy. They test on two tasks with both gold and crowd annotations. The first is Twitter POS tagging (Gimpel et al. data with crowd labels from Hovy et al.). The second is binary morphological stemming (Carpenter et al. data). Across conditions, the MTL models significantly beat matching single-task baselines and mostly exceed earlier results. The gains hold even when the main task is trained on majority-vote silver labels. This shows the benefit comes from modelling disagreement itself, not from extra information. The work suggests that annotator ambiguity is a useful training signal and that it can soften penalties on inherently ambiguous items.

## Key points
- **Data:**
  - POS tagging uses Universal POS tags, 177 crowd annotators and at least 5 annotations per token. It is evaluated with 5-fold CV and on a separate test set from Plank et al. (2014).
  - Stemming uses 26 annotators with at least 4 labels per (word, stem) pair.
- **POS, 5-fold CV:** STL reaches 85.73% accuracy, while MTL + Cross-Entropy reaches 86.76%. The earlier result by Jamison and Gurevych was 78.9%.
- **POS, independent test set:** MTL + inverse KL reaches 86.29%. This compares with 85.84% for STL and 83.6% for Plank et al.
- **Stemming:** MTL + inverse KL achieves 77.09% accuracy and 58.41 F1. STL scores 73.59% and 57.57, and Jamison and Gurevych reported 76.6%.
- **Silver-label control and setup:**
  - With majority-vote training labels, MTL still significantly beats STL (e.g., POS CV 76.00% vs 75.22%).
  - Significance was tested by bootstrap over 10 runs per condition.
  - The architectures were bi-RNN plus attention for POS and CNN plus attention for stemming.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Queen Mary University of London](https://md.research.elamur.ai/free/en/universities/queen-mary-university-of-london/) — University, London, United Kingdom · 39 works · 3,307 citations
- [Bocconi University](https://md.research.elamur.ai/free/en/universities/bocconi-university/) — University, Milan, Italy · 16 works · 2,850 citations
- [IT University of Copenhagen](https://md.research.elamur.ai/free/en/universities/it-university-of-copenhagen/) — University, Copenhagen, Denmark · 16 works · 2,225 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Tommaso Fornaciari, Alexandra Uma, Silviu Paun, Barbara Plank, Dirk Hovy and Massimo Poesio. 2021. Beyond Black & White: Leveraging Annotator Disagreement via Soft-Label Multi-Task Learning. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{fornaciari2021beyond,
  title = {{Beyond Black \& White: Leveraging Annotator Disagreement via Soft-Label Multi-Task Learning}},
  author = {Tommaso Fornaciari, Alexandra Uma, Silviu Paun, Barbara Plank, Dirk Hovy and Massimo Poesio},
  booktitle = {Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.204},
  url = {https://doi.org/10.18653/v1/2021.naacl-main.204}
}
```

- [← previous Zero-shot Neural Passage Retrieval via Domain-targeted Synthetic Question Generation](https://md.research.elamur.ai/free/en/papers/ma-2021-zero-shot-neural-passage-retrieval/)
- [next → Towards Robustness of Text-to-SQL Models against Synonym Substitution](https://md.research.elamur.ai/free/en/papers/gan-2021-robustness-text-sql-models-against/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/)

## References

39 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. In a broad sense, our study belongs to the research area of regularization methods for neural networks. Among them, label smoothing (Pereyra et al., 2017) penalizes the cases of over-confident network predictions. Both label smoothing and soft labels reduce overfitting regulating the loss size. Howe… [find](https://scholar.google.com/scholar?q=In+a+broad+sense%2C+our+study+belongs+to+the+research+area+of+regularization+methods+for+neural+networks%2E+Among+them%2C+label+smoothing+%28Pereyra+et+al%2E%2C+2017%29+penalizes+the+cases+of+over%2Dconfident+network)
2. Several different lines of research use annotation disagreement. One line focuses on the aggregation of multiple annotations before model training. Seminal work includes the proposal by Dawid and Skene (1979), who proposed an Expectation-Maximization (EM) based aggregation model. This model has sinc… [find](https://scholar.google.com/scholar?q=Several+different+lines+of+research+use+annotation+disagreement%2E+One+line+focuses+on+the+aggregation+of+multiple+annotations+before+model+training%2E+Seminal+work+includes+the+proposal+by+Dawid+and+Sken)
3. Beata Beigman Klebanov and Eyal Beigman. 2014. Difficult cases: From data to learning, and back. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 390–396, Baltimore, Maryland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Beata+Beigman+Klebanov+and+Eyal+Beigman%2E+2014%2E+Difficult+cases%3A+From+data+to+learning%2C+and+back%2E+In+Proceedings+of+the+52nd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+2%3A+S)
4. Taylor Berg-Kirkpatrick, David Burkett, and Dan Klein. 2012. An empirical investigation of statistical significance in NLP. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 995–1005, Jeju Island, Korea.… [find](https://scholar.google.com/scholar?q=Taylor+Berg%2DKirkpatrick%2C+David+Burkett%2C+and+Dan+Klein%2E+2012%2E+An+empirical+investigation+of+statistical+significance+in+NLP%2E+In+Proceedings+of+the+2012+Joint+Conference+on+Empirical+Methods+in+Natural+)
5. Bob Carpenter. 2008. Multilevel bayesian models of categorical data annotation. Available as [http://lingpipe.files.wordpress.com/2008/](http://lingpipe.files.wordpress.com/2008/) 11/carp-bayesian-multilevel-annotation. pdf . [↗](http://lingpipe.files.wordpress.com/2008/)
6. Bob Carpenter, Emily Jamison, and Breck Baldwin. 2009. Building a stemming corpus: Coding standards. [https://lingpipe-blog.com/2009/](https://lingpipe-blog.com/2009/) 02/25/ . [↗](https://lingpipe-blog.com/2009/)
7. Trevor Cohn and Lucia Specia. 2013. Modelling annotator bias with multi-task Gaussian processes: An application to machine translation quality estimation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 32–42, Sofia, Bulgaria.… [find](https://scholar.google.com/scholar?q=Trevor+Cohn+and+Lucia+Specia%2E+2013%2E+Modelling+annotator+bias+with+multi%2Dtask+Gaussian+processes%3A+An+application+to+machine+translation+quality+estimation%2E+In+Proceedings+of+the+51st+Annual+Meeting+of+)
8. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. [Learning whom to trust with MACE](https://md.research.elamur.ai/free/en/papers/hovy-2013-learning-whom-trust-mace/). In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1120–1130, Atlanta, Georgia. Associati…
9. Dirk Hovy, Barbara Plank, and Anders Søgaard. 2014. Experiments with crowdsourced re-annotation of a POS tagging data set. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 377–382, Baltimore, Maryland. Association for Computat… [find](https://scholar.google.com/scholar?q=Dirk+Hovy%2C+Barbara+Plank%2C+and+Anders+S%C3%B8gaard%2E+2014%2E+Experiments+with+crowdsourced+re%2Dannotation+of+a+POS+tagging+data+set%2E+In+Proceedings+of+the+52nd+Annual+Meeting+of+the+Association+for+Computationa)
10. Dirk Hovy and Diyi Yang. 2021. [The importance of modeling social factors of language: Theory and practice](https://md.research.elamur.ai/free/en/papers/hovy-2021-importance-modeling-social-factors-language/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics, Minneapolis, Minnesota. Association for Computational Linguistics.
11. Emily Jamison and Iryna Gurevych. 2015. Noise or additional information? leveraging crowdsource annotation item agreement for natural language tasks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 291–297. [find](https://scholar.google.com/scholar?q=Emily+Jamison+and+Iryna+Gurevych%2E+2015%2E+Noise+or+additional+information%3F+leveraging+crowdsource+annotation+item+agreement+for+natural+language+tasks%2E+In+Proceedings+of+the+2015+Conference+on+Empirical)
12. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. [find](https://scholar.google.com/scholar?q=Diederik+P+Kingma+and+Jimmy+Ba%2E+2014%2E+Adam%3A+A+method+for+stochastic+optimization%2E+arXiv+preprint+arXiv%3A1412%2E6980%2E)
13. S. Kullback and R. A. Leibler. 1951. On information and sufficiency. Ann. Math. Statist., 22(1):79–86. [find](https://scholar.google.com/scholar?q=S%2E+Kullback+and+R%2E+A%2E+Leibler%2E+1951%2E+On+information+and+sufficiency%2E+Ann%2E+Math%2E+Statist%2E%2C+22%281%29%3A79%E2%80%9386%2E)
14. A. P. Dawid and A. M. Skene. 1979. Maximum likelihood estimation of observer error-rates using the em algorithm. Applied Statistics, 28(1):20–28. [find](https://scholar.google.com/scholar?q=A%2E+P%2E+Dawid+and+A%2E+M%2E+Skene%2E+1979%2E+Maximum+likelihood+estimation+of+observer+error%2Drates+using+the+em+algorithm%2E+Applied+Statistics%2C+28%281%29%3A20%E2%80%9328%2E)
15. John P Lalor, Hao Wu, and Hong Yu. 2017. Soft label memorization-generalization for natural language inference. arXiv preprint arXiv:1702.08563. — [arXiv:1702.08563](https://arxiv.org/abs/1702.08563) [find](https://scholar.google.com/scholar?q=John+P+Lalor%2C+Hao+Wu%2C+and+Hong+Yu%2E+2017%2E+Soft+label+memorization%2Dgeneralization+for+natural+language+inference%2E+arXiv+preprint+arXiv%3A1702%2E08563%2E+%E2%80%94+%5BarXiv%3A1702%2E08563%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1702%2E08563%29)
16. Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A Smith. 2010. [Part-of-speech tagging for twitter: Annotation, features, and experiments](https://md.research.elamur.ai/free/en/papers/gimpel-2011-part-speech-tagging-twitter-annotation/). Technical report, Carnegie-Mellon Univ Pittsburgh Pa Sc…
17. Jianhua Lin. 1991. Divergence measures based on the shannon entropy. IEEE Transactions on Information theory, 37(1):145–151. [find](https://scholar.google.com/scholar?q=Jianhua+Lin%2E+1991%2E+Divergence+measures+based+on+the+shannon+entropy%2E+IEEE+Transactions+on+Information+theory%2C+37%281%29%3A145%E2%80%93151%2E)
18. Jacob Goldberger and Ehud Ben-Reuven. 2016. Training deep neural-networks using a noise adaptation layer. [find](https://scholar.google.com/scholar?q=Jacob+Goldberger+and+Ehud+Ben%2DReuven%2E+2016%2E+Training+deep+neural%2Dnetworks+using+a+noise+adaptation+layer%2E)
19. Bo Han, Jiangchao Yao, Gang Niu, Mingyuan Zhou, Ivor Tsang, Ya Zhang, and Masashi Sugiyama. 2018a. Masking: A new perspective of noisy supervision. In Advances in Neural Information Processing Systems, pages 5836–5846. [find](https://scholar.google.com/scholar?q=Bo+Han%2C+Jiangchao+Yao%2C+Gang+Niu%2C+Mingyuan+Zhou%2C+Ivor+Tsang%2C+Ya+Zhang%2C+and+Masashi+Sugiyama%2E+2018a%2E+Masking%3A+A+new+perspective+of+noisy+supervision%2E+In+Advances+in+Neural+Information+Processing+Systems)
20. Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. 2018b. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in neural information processing systems, pages 8527–8537. [find](https://scholar.google.com/scholar?q=Bo+Han%2C+Quanming+Yao%2C+Xingrui+Yu%2C+Gang+Niu%2C+Miao+Xu%2C+Weihua+Hu%2C+Ivor+Tsang%2C+and+Masashi+Sugiyama%2E+2018b%2E+Co%2Dteaching%3A+Robust+training+of+deep+neural+networks+with+extremely+noisy+labels%2E+In+Advances+i)
21. Henry B Mann and Donald R Whitney. 1947. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics, pages 50–60. [find](https://scholar.google.com/scholar?q=Henry+B+Mann+and+Donald+R+Whitney%2E+1947%2E+On+a+test+of+whether+one+of+two+random+variables+is+stochastically+larger+than+the+other%2E+The+annals+of+mathematical+statistics%2C+pages+50%E2%80%9360%2E)
22. Boyuan Pan, Yazheng Yang, Zhou Zhao, Yueting Zhuang, Deng Cai, and Xiaofei He. 2019. Discourse marker augmented network with reinforcement learning for natural language inference. arXiv preprint arXiv:1907.09692. [find](https://scholar.google.com/scholar?q=Boyuan+Pan%2C+Yazheng+Yang%2C+Zhou+Zhao%2C+Yueting+Zhuang%2C+Deng+Cai%2C+and+Xiaofei+He%2E+2019%2E+Discourse+marker+augmented+network+with+reinforcement+learning+for+natural+language+inference%2E+arXiv+preprint+arXiv)
23. Silviu Paun, Bob Carpenter, Jon Chamberlain, Dirk Hovy, Udo Kruschwitz, and Massimo Poesio. 2018. [Comparing Bayesian models of annotation](https://md.research.elamur.ai/free/en/papers/paun-2018-comparing-bayesian-models-annotation/). Transactions of the Association for Computational Linguistics, 6:571–585.
24. Ellie Pavlick and Tom Kwiatkowski. 2019. [Inherent disagreements in human textual inferences](https://md.research.elamur.ai/free/en/papers/pavlick-2019-inherent-disagreements-human-textual-inferences/). Transactions of the Association for Computational Linguistics, 7:677–694.
25. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. [Glove: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. pages 5248–5264, Online. Association for Computational Linguistics.
26. Gabriel Pereyra, George Tucker, Jan Chorowski, Łukasz Kaiser, and Geoffrey Hinton. 2017. [Regularizing neural networks by penalizing confident output distributions](https://md.research.elamur.ai/free/en/papers/pereyra-2017-regularizing-neural-networks-penalizing-confident/). arXiv preprint arXiv:1701.06548.
27. Victor S Sheng, Foster Provost, and Panagiotis G Ipeirotis. 2008. Get another label? improving data quality and data mining using multiple, noisy labelers. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 614–622. ACM. [find](https://scholar.google.com/scholar?q=Victor+S+Sheng%2C+Foster+Provost%2C+and+Panagiotis+G+Ipeirotis%2E+2008%2E+Get+another+label%3F+improving+data+quality+and+data+mining+using+multiple%2C+noisy+labelers%2E+In+Proceedings+of+the+14th+ACM+SIGKDD+intern)
28. Slav Petrov, Dipanjan Das, and Ryan McDonald. 2012. [A universal part-of-speech tagset](https://md.research.elamur.ai/free/en/papers/petrov-2012-universal-part-speech-tagset/). In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 2089– 2096, Istanbul, Turkey. European Language Resources Association (ELRA).
29. Anders Søgaard, Anders Johannsen, Barbara Plank, Dirk Hovy, and Hector Martı́nez Alonso. 2014. What’s in a p-value in NLP? In Proceedings of the Eighteenth Conference on Computational Natural Language Learning, pages 1–10, Ann Arbor, Michigan. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Anders+S%C3%B8gaard%2C+Anders+Johannsen%2C+Barbara+Plank%2C+Dirk+Hovy%2C+and+Hector+Mart%C4%B1%CC%81nez+Alonso%2E+2014%2E+What%E2%80%99s+in+a+p%2Dvalue+in+NLP%3F+In+Proceedings+of+the+Eighteenth+Conference+on+Computational+Natural+Language)
30. Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014. [Learning part-of-speech taggers with inter-annotator agreement loss](https://md.research.elamur.ai/free/en/papers/plank-2014-learning-part-speech-taggers-inter/). In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 742–751, Gothenburg, Sweden. Association for Computational Li…
31. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in neural information processing systems, pages 5998–6008.
32. Barbara Plank, Anders Søgaard, and Yoav Goldberg. 2016. [Multilingual part-of-speech tagging with bidirectional long short-term memory models and auxiliary loss](https://md.research.elamur.ai/free/en/papers/plank-2016-multilingual-part-speech-tagging-bidirectional/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 412–418, Berlin,…
33. Daniel Zeman. 2010. Hard problems of tagset conversion. In Proceedings of the Second International Conference on Global Interoperability for Language Resources, pages 181–185. [find](https://scholar.google.com/scholar?q=Daniel+Zeman%2E+2010%2E+Hard+problems+of+tagset+conversion%2E+In+Proceedings+of+the+Second+International+Conference+on+Global+Interoperability+for+Language+Resources%2C+pages+181%E2%80%93185%2E)
34. Massimo Poesio and Ron Artstein. 2005. [The reliability of anaphoric annotation, reconsidered: Taking ambiguity into account](https://md.research.elamur.ai/free/en/papers/poesio-2005-reliability-anaphoric-annotation-reconsidered-taking/). In Proc. of ACL Workshop on Frontiers in Corpus Annotation, pages 76–83.
35. Vikas C Raykar, Shipeng Yu, Linda H Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. 2010. Learning from crowds. Journal of Machine Learning Research, 11(Apr):1297–1322. [find](https://scholar.google.com/scholar?q=Vikas+C+Raykar%2C+Shipeng+Yu%2C+Linda+H+Zhao%2C+Gerardo+Hermosillo+Valadez%2C+Charles+Florin%2C+Luca+Bogoni%2C+and+Linda+Moy%2E+2010%2E+Learning+from+crowds%2E+Journal+of+Machine+Learning+Research%2C+11%28Apr%29%3A1297%E2%80%931322%2E)
36. Filipe Rodrigues and Francisco C Pereira. 2018. Deep learning from crowds. In Thirty-Second AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Filipe+Rodrigues+and+Francisco+C+Pereira%2E+2018%2E+Deep+learning+from+crowds%2E+In+Thirty%2DSecond+AAAI+Conference+on+Artificial+Intelligence%2E)
37. Pablo Ruiz, Pablo Morales-Álvarez, Rafael Molina, and Aggelos K Katsaggelos. 2019. Learning from crowds with variational gaussian processes. Pattern Recognition, 88:298–311. [find](https://scholar.google.com/scholar?q=Pablo+Ruiz%2C+Pablo+Morales%2DA%CC%81lvarez%2C+Rafael+Molina%2C+and+Aggelos+K+Katsaggelos%2E+2019%2E+Learning+from+crowds+with+variational+gaussian+processes%2E+Pattern+Recognition%2C+88%3A298%E2%80%93311%2E)
38. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. [The risk of racial bias in hate speech detection](https://md.research.elamur.ai/free/en/papers/sap-2019-risk-racial-bias-hate-speech/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668–1678, Florence, Italy. Association for Computational Linguistics.
39. Deven Santosh Shah, H. Andrew Schwartz, and Dirk Hovy. 2020. [Predictive biases in natural language processing models: A conceptual framework and overview](https://md.research.elamur.ai/free/en/papers/shah-2019-predictive-biases-natural-language-processing/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,

*Updated 24 October 2026*