---
title: "Toward a Perspectivist Turn in Ground Truthing for Predictive Computing — Basile 2021"
description: "This position paper presents no new experiments. 13 references, cited by 140 papers here."
authors: ["Valerio Basile", "Federico Cabitza", "Andrea Campagner", "Michael Fell"]
year: 2021
venue: "arXiv:2109.04270 · Proceedings of the AAAI Conference on Artificial Intelligence, 37(6), 6860-6868 (2023)"
source: "https://arxiv.org/abs/2109.04270"
doi: "10.1609/aaai.v37i6.25840"
arxiv: "2109.04270"
licence: "CC BY-SA 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/basile-2021-toward-perspectivist-turn-ground-truthing/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/basile-2021-toward-perspectivist-turn-ground-truthing/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/2/) / Toward a Perspectivist Turn in Ground Truthing for Predictive Computing

# Toward a Perspectivist Turn in Ground Truthing for Predictive Computing

**Authors:** [Valerio Basile](https://md.research.elamur.ai/free/en/authors/valerio-basile/), Federico Cabitza, Andrea Campagner and [Michael Fell](https://md.research.elamur.ai/free/en/authors/michael-fell/) · **Year:** 2021 · **Venue:** arXiv:2109.04270 · Proceedings of the AAAI Conference on Artificial Intelligence, 37(6), 6860-6868 (2023) · **Licence:** [CC BY-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-sa/4.0/)

**Links:** [Publisher's page](https://doi.org/10.1609/aaai.v37i6.25840) · [PDF](https://arxiv.org/pdf/2109.04270) · [DOI 10.1609/aaai.v37i6.25840](https://doi.org/10.1609/aaai.v37i6.25840) · [arXiv 2109.04270](https://arxiv.org/abs/2109.04270) · [OpenAlex](https://openalex.org/W3197587259)

**Citations:** 85 citations (OpenAlex, 2026-10-14) · [140](https://md.research.elamur.ai/free/en/papers/basile-2021-toward-perspectivist-turn-ground-truthing/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
Supervised machine learning usually relies on datasets in which several annotators' labels are merged, typically by majority vote, into a single "gold standard." The authors argue that this aggregation discards useful information. It can silence minority experts in hard cases and erase legitimate disagreement on ambiguous items, which are common in NLP. They contend the problem also affects tasks usually treated as objective, such as clinical decision support. This position paper presents no new experiments. Instead, it proposes **data perspectivism**, a paradigm that keeps raters' differing judgments in the ML pipeline rather than collapsing them. It defines a weak and a strong form of the idea and models the ground-truthing workflow as a BPMN diagram. The authors survey ways to train and evaluate models on non-aggregated labels and weigh the benefits against the drawbacks. They close with recommendations and a ten-point research agenda. The paper matters because it shifts attention from data quality alone to representativeness, fairness and explainability. It invites practitioners to ask whose opinions a model actually encodes.

## Key points
- **Weak perspectivism** collects many labels per object, producing a "diamond standard," but then reduces them to one label, for example by weighted voting ("perspective reduction"). **Strong perspectivism** keeps all labels through both training and benchmarking.
- Raters may disagree for reasons other than error, including:
  - ambiguity of the phenomenon
  - instability of the phenomenon over time
  - task complexity
  - cognitive biases
  - "decision noise" in Kahneman's sense
- Proposed implementation strategies include:
  - replicating training objects once for each label they received (the most backward-compatible option)
  - ensemble methods
  - soft-loss training
- Suggested evaluation approaches include testing on items with "overwhelming majority" agreement and using soft metrics. The authors call for unified metrics that balance richness of perspectives against reliability, for instance by combining inter-rater agreement with entropy-based measures.
- The research agenda also covers:
  - fairness, including detecting biases introduced by groups of raters
  - explainable AI
  - simulating observer variability
  - richer rating formats, such as rankings and confidence levels, drawing on computational social choice

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

## Institutions

- [University of Turin](https://md.research.elamur.ai/free/en/universities/university-of-turin/) — University, Turin, Italy · 21 works · 6,124 citations
- [University of Milano-Bicocca](https://md.research.elamur.ai/free/en/universities/university-of-milano-bicocca/) — University, Milan, Italy · 9 works · 4,698 citations

Also on the byline: [IRCCS Ospedale Galeazzi - Sant'Ambrogio](https://ror.org/01vyrje42).

## Cite

Cite as: Valerio Basile, Federico Cabitza, Andrea Campagner and Michael Fell. 2021. Toward a Perspectivist Turn in Ground Truthing for Predictive Computing. arXiv:2109.04270 · Proceedings of the AAAI Conference on Artificial Intelligence, 37(6), 6860-6868 (2023).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{basile2021toward,
  title = {{Toward a Perspectivist Turn in Ground Truthing for Predictive Computing}},
  author = {Valerio Basile, Federico Cabitza, Andrea Campagner and Michael Fell},
  year = {2021},
  doi = {10.1609/aaai.v37i6.25840},
  eprint = {2109.04270},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1609/aaai.v37i6.25840}
}
```

- [← previous Open-Domain Question Answering Goes Conversational via Question Rewriting](https://md.research.elamur.ai/free/en/papers/anantha-2021-open-domain-question-answering-goes/)
- [next → Multilingual Translation from Denoising Pre-Training](https://md.research.elamur.ai/free/en/papers/tang-2021-multilingual-translation-denoising-pre-training/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/2/)

## References

13 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. 1. Akhtar, S., Basile, V., Patti, V.: A new measure of polarization in the annotation of hate speech. In: Alviano, M., Greco, G., Scarcello, F. (eds.) AI*IA 2019 – Advances in Artificial Intelligence. pp. 588–603. [find](https://scholar.google.com/scholar?q=1%2E+Akhtar%2C+S%2E%2C+Basile%2C+V%2E%2C+Patti%2C+V%2E%3A+A+new+measure+of+polarization+in+the+annotation+of+hate+speech%2E+In%3A+Alviano%2C+M%2E%2C+Greco%2C+G%2E%2C+Scarcello%2C+F%2E+%28eds%2E%29+AI%2AIA+2019+%E2%80%93+Advances+in+Artificial+Intelligence%2E)
2. Springer International Publishing, Cham (2019) 2. Akhtar, S., Basile, V., Patti, V.: Modeling annotator perspective and polarized opinions to improve hate speech detection. In: Proceedings of the AAAI Conference on Human Computation and Crowdsourcing. vol. 8, pp. 151–154 (2020) 3. Aroyo, L., Welty,… [find](https://scholar.google.com/scholar?q=Springer+International+Publishing%2C+Cham+%282019%29+2%2E+Akhtar%2C+S%2E%2C+Basile%2C+V%2E%2C+Patti%2C+V%2E%3A+Modeling+annotator+perspective+and+polarized+opinions+to+improve+hate+speech+detection%2E+In%3A+Proceedings+of+the+AAAI)
3. Newnes (2014) 6. Barnard, M.E., Pyden, A., Rice, M.S., Linares, M., Tworoger, S.S., Howitt, B.E., Meserve, E.E., Hecht, J.L.: Inter-pathologist and pathology report agreement for ovarian tumor characteristics in the nurses’ health studies. Gynecologic oncology 150(3), 521–526 (2018) 7. Basile, V.: I… [find](https://scholar.google.com/scholar?q=Newnes+%282014%29+6%2E+Barnard%2C+M%2EE%2E%2C+Pyden%2C+A%2E%2C+Rice%2C+M%2ES%2E%2C+Linares%2C+M%2E%2C+Tworoger%2C+S%2ES%2E%2C+Howitt%2C+B%2EE%2E%2C+Meserve%2C+E%2EE%2E%2C+Hecht%2C+J%2EL%2E%3A+Inter%2Dpathologist+and+pathology+report+agreement+for+ovarian+tumor+charact)
4. Springer Nature Switzerland AG (2021) 8. Bechmann, A., Bowker, G.C.: Unsupervised by any other name: Hidden layers of knowledge production in artificial intelligence on social media. Big Data & Society 6(1), 2053951718819569 (2019) 9. Bender, E.M., Friedman, B.: Data statements for natural language… — [arXiv:2006.07159](https://arxiv.org/abs/2006.07159) [find](https://scholar.google.com/scholar?q=Springer+Nature+Switzerland+AG+%282021%29+8%2E+Bechmann%2C+A%2E%2C+Bowker%2C+G%2EC%2E%3A+Unsupervised+by+any+other+name%3A+Hidden+layers+of+knowledge+production+in+artificial+intelligence+on+social+media%2E+Big+Data+%26+Societ)
5. Cambridge University Press (2016) 12. Cabitza, F., Campagner, A.: The need to separate the wheat from the chaff in medical informatics. International Journal of Medical Informatics 152 (2021) 13. Cabitza, F., Campagner, A., Sconfienza, L.M.: As if sand were stone. new concepts and metrics to probe t… [find](https://scholar.google.com/scholar?q=Cambridge+University+Press+%282016%29+12%2E+Cabitza%2C+F%2E%2C+Campagner%2C+A%2E%3A+The+need+to+separate+the+wheat+from+the+chaff+in+medical+informatics%2E+International+Journal+of+Medical+Informatics+152+%282021%29+13%2E+Cabi)
6. Data Perspectivist turn 15. Campagner, A., Ciucci, D., Svensson, C.M., Figge, M.T., Cabitza, F.: Ground truthing from multi-rater labeling with three-way decision and possibility theory. Information Sciences 545, 771–790 (2021) 16. Chang, J.C., Amershi, S., Kamar, E.: Revolt: Collaborative crowdsour… [find](https://scholar.google.com/scholar?q=Data+Perspectivist+turn+15%2E+Campagner%2C+A%2E%2C+Ciucci%2C+D%2E%2C+Svensson%2C+C%2EM%2E%2C+Figge%2C+M%2ET%2E%2C+Cabitza%2C+F%2E%3A+Ground+truthing+from+multi%2Drater+labeling+with+three%2Dway+decision+and+possibility+theory%2E+Information+S)
7. AAAI (2019) 26. Hildebrandt, M.: The issue of bias. the framing powers of ml. In: editor, T. (ed.) Machine Learning and Society: Impact, Trust, Transparency. [find](https://scholar.google.com/scholar?q=AAAI+%282019%29+26%2E+Hildebrandt%2C+M%2E%3A+The+issue+of+bias%2E+the+framing+powers+of+ml%2E+In%3A+editor%2C+T%2E+%28ed%2E%29+Machine+Learning+and+Society%3A+Impact%2C+Trust%2C+Transparency%2E)
8. MIT Press (2021) 27. Jagabathula, S., Subramanian, L., Venkataraman, A.: Identifying unreliable and adversarial workers in crowdsourced labeling tasks. The Journal of Machine Learning Research 18(1), 3233–3299 (2017) 28. Jewett, M.A., Bombardier, C., Caron, D., Ryan, M.R., Gray, R.R., Louis, E.L.S.,… [find](https://scholar.google.com/scholar?q=MIT+Press+%282021%29+27%2E+Jagabathula%2C+S%2E%2C+Subramanian%2C+L%2E%2C+Venkataraman%2C+A%2E%3A+Identifying+unreliable+and+adversarial+workers+in+crowdsourced+labeling+tasks%2E+The+Journal+of+Machine+Learning+Research+18%281%29%2C+)
9. V. Basile et al. 33. Linstone, H.A., Turoff, M.: The delphi method: Techniques and applications. In: The Delphi method: Techniques and applications. Addison-Wesley Publishing (1975) 34. Lok, C.E., Morgan, C.D., Ranganathan, N.: The accuracy and interobserver agreement in detecting the ‘gallop sounds… [our card](https://md.research.elamur.ai/free/en/papers/plank-2014-learning-part-speech-taggers-inter/)
10. Springer (2020) 41. Salminen, J.O., Al-Merekhi, H.A., Dey, P., Jansen, B.J.: Inter-rater agreement for social computing studies. In: 2018 Fifth International Conference on Social Networks Analysis, Management and Security (SNAMS). pp. 80–87. [find](https://scholar.google.com/scholar?q=Springer+%282020%29+41%2E+Salminen%2C+J%2EO%2E%2C+Al%2DMerekhi%2C+H%2EA%2E%2C+Dey%2C+P%2E%2C+Jansen%2C+B%2EJ%2E%3A+Inter%2Drater+agreement+for+social+computing+studies%2E+In%3A+2018+Fifth+International+Conference+on+Social+Networks+Analysis%2C+Ma)
11. IEEE (2018) 42. Schaekermann, M., Goh, J., Larson, K., Law, E.: Resolvable vs. irresolvable disagreement: A study on worker deliberation in crowd work. Proceedings of the ACM on Human-Computer Interaction 2(CSCW), 1–19 (2018) 43. Sharmanska, V., Hernández-Lobato, D., Miguel Hernandez-Lobato, J., Qu… [find](https://scholar.google.com/scholar?q=IEEE+%282018%29+42%2E+Schaekermann%2C+M%2E%2C+Goh%2C+J%2E%2C+Larson%2C+K%2E%2C+Law%2C+E%2E%3A+Resolvable+vs%2E+irresolvable+disagreement%3A+A+study+on+worker+deliberation+in+crowd+work%2E+Proceedings+of+the+ACM+on+Human%2DComputer+Interac)
12. Springer (2019) 45. Svensson, C.M., Figge, M.T., Hübler, R.: Automated classification of circulating tumor cells and the impact of interobsever variability on classifier training and performance. Journal of Immunology Research 2015 (2015) 46. Tao, F., Jiang, L., Li, C.: Label similarity-based weigh… [find](https://scholar.google.com/scholar?q=Springer+%282019%29+45%2E+Svensson%2C+C%2EM%2E%2C+Figge%2C+M%2ET%2E%2C+Hu%CC%88bler%2C+R%2E%3A+Automated+classification+of+circulating+tumor+cells+and+the+impact+of+interobsever+variability+on+classifier+training+and+performance%2E+Jou)
13. Data Perspectivist turn 50. Yager, R.R.: Interval valued entropies for dempster–shafer structures. Knowledge-Based Systems 161, 390–397 (2018) 51. Yun, S., Oh, S.J., Heo, B., Han, D., Choe, J., Chun, S.: Re-labeling imagenet: from single to multi-labels, from global to localized labels. arXiv prepri… — [arXiv:2101.05022](https://arxiv.org/abs/2101.05022) [find](https://scholar.google.com/scholar?q=Data+Perspectivist+turn+50%2E+Yager%2C+R%2ER%2E%3A+Interval+valued+entropies+for+dempster%E2%80%93shafer+structures%2E+Knowledge%2DBased+Systems+161%2C+390%E2%80%93397+%282018%29+51%2E+Yun%2C+S%2E%2C+Oh%2C+S%2EJ%2E%2C+Heo%2C+B%2E%2C+Han%2C+D%2E%2C+Choe%2C+J%2E%2C+Chun%2C+)