---
title: "Linguistically debatable or just plain wrong? — Plank 2014"
description: "This position paper asks whether annotation projects should keep suppressing disagreement between annotators or treat it as useful signal."
authors: ["Barbara Plank", "Dirk Hovy", "Anders Søgaard"]
year: 2014
venue: "Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)"
source: "https://aclanthology.org/P14-2083/"
doi: "10.3115/v1/p14-2083"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/plank-2014-linguistically-debatable-or-just-plain/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/plank-2014-linguistically-debatable-or-just-plain/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2014](https://md.research.elamur.ai/free/en/papers/2014/page/2/) / Linguistically debatable or just plain wrong?

# Linguistically debatable or just plain wrong?

**Authors:** [Barbara Plank](https://md.research.elamur.ai/free/en/authors/barbara-plank/), [Dirk Hovy](https://md.research.elamur.ai/free/en/authors/dirk-hovy/), [Anders Søgaard](https://md.research.elamur.ai/free/en/authors/anders-sogaard/) · **Year:** 2014 · **Venue:** Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/P14-2083/) · [PDF](https://aclanthology.org/P14-2083.pdf) · [DOI 10.3115/v1/p14-2083](https://doi.org/10.3115/v1/p14-2083) · [OpenAlex](https://openalex.org/W2252039660)

**Citations:** 85 citations (openalex-snapshot · 2026-10-14) · [87](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
This position paper asks whether annotation projects should keep suppressing disagreement between annotators or treat it as useful signal. The authors study part-of-speech annotation using the 12 universal POS tags. Linguists without guidelines annotated English newswire (PTB WSJ), spoken transcripts (Talkbank) and Twitter. They also used crowdsourced lay annotations and French social media data. Confusion patterns between tags turn out to be strongly correlated across domains, weaker but still significant across languages, and similar for lay people. This indicates systematic ambiguity rather than random noise. Typical cases are ADP vs. PRT, ADJ vs. NOUN and PRON vs. DET. Next, the authors mined inconsistencies with variation n-grams, and experts judged whether each tag was plausible in context. Only about a fifth of disagreements were real mistakes, which is under 2% of all annotations. A classifier using surface features separated errors from hard cases only modestly. The authors conclude that most disagreements are defensible analyses. They argue that forcing agreement through guidelines biases models toward one linguistic theory.

## Key points
- English data covered three domains, with 500 doubly-annotated sentences or tweets each: WSJ Section 00, Talkbank and Twitter. Cross-domain disagreement correlations reached Spearman 0.644–0.869 and Kendall's τ 0.498–0.659.
- In the lay study, 177 Crowdflower workers re-annotated the Gimpel et al. (2011) Twitter training set, totalling 14,619 items. Majority vote matched experts 79.54% of the time. Wiktionary filtering plus MACE raised this to 80.58%.
- The French Social Media Bank was compared with English Twitter and showed Spearman 0.288 (p \< 0.001).
- Three PhD linguists judged 440 tag-confusion pairs from WSJ-00 and agreed 86% of the time. Only about 20% of conflicts were errors, and NOUN–PRON confusions were always hard cases.
- An L2-regularized logistic regression model detected errors with an F1 of about 70%. Separately, nine linguistics faculty members were each shown 10 hard cases, and they disagreed on 8 of them.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Part-of-speech tagging](https://md.research.elamur.ai/free/en/topics/part-of-speech-tagging/)

## Cited by

Cited by

87 papers of our library cite it (of the 140884 whose bibliographies we read); 12 of them have a page here, 75 not yet.

- [ArMIS - The Arabic Misogyny and Sexism Corpus with Annotator Subjective Disagreements](https://md.research.elamur.ai/free/en/papers/almanea-2022-armis-arabic-misogyny-sexism-corpus/)
- [We Need to Consider Disagreement in Evaluation](https://md.research.elamur.ai/free/en/papers/basile-2021-we-need-consider-disagreement-evaluation/)
- [Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future](https://md.research.elamur.ai/free/en/papers/cl-2023-klie-annotation-error-detection-analyzing-past/)
- [VariErr NLI: Separating Annotation Error from Human Label Variation](https://md.research.elamur.ai/free/en/papers/genzel-2024-varierr-nli-separating-annotation-error/)
- [Investigating Reasons for Disagreement in Natural Language Inference](https://md.research.elamur.ai/free/en/papers/jiang-2022-investigating-reasons-disagreement-natural-language/)
- [Subjectivity in the Annotation of Bridging Anaphora](https://md.research.elamur.ai/free/en/papers/levine-2025-subjectivity-annotation-bridging-anaphora/)
- [A Probabilistic Annotation Model for Crowdsourcing Coreference](https://md.research.elamur.ai/free/en/papers/paun-2018-probabilistic-annotation-model-crowdsourcing-coreference/)
- [The “Problem” of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation](https://md.research.elamur.ai/free/en/papers/plank-2022-problem-human-label-variation-ground/)
- [A Crowdsourced Corpus of Multiple Judgments and Disagreement on Anaphoric Interpretation](https://md.research.elamur.ai/free/en/papers/poesio-2019-crowdsourced-corpus-multiple-judgments-disagreement/)
- [Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design](https://md.research.elamur.ai/free/en/papers/pyatkin-2023-design-choices-crowdsourcing-implicit-discourse/)
- [Neural Unsupervised Domain Adaptation in NLP–-A Survey](https://md.research.elamur.ai/free/en/papers/ramponi-2020-neural-unsupervised-domain-adaptation-nlp/)
- [Two Contrasting Data Annotation Paradigms for Subjective NLP Tasks](https://md.research.elamur.ai/free/en/papers/rottger-2022-two-contrasting-data-annotation-paradigms/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{plank2014linguistically,
  title = {{Linguistically debatable or just plain wrong?}},
  author = {Plank, Barbara and Hovy, Dirk and Søgaard, Anders},
  booktitle = {Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2014},
  doi = {10.3115/v1/p14-2083},
  url = {https://doi.org/10.3115/v1/p14-2083}
}
```

- [← previous Linguistic Regularities in Sparse and Explicit Word Representations](https://md.research.elamur.ai/free/en/papers/levy-2014-linguistic-regularities-sparse-explicit-word/)
- [next → Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://md.research.elamur.ai/free/en/papers/donahue-2014-long-term-recurrent-convolutional-networks/)
- [↑ back to papers of 2014](https://md.research.elamur.ai/free/en/papers/2014/page/2/)

## References

18 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bob Carpenter. 2008. Multilevel Bayesian models of categorical data annotation. Technical report, Ling-Pipe. [find](https://scholar.google.com/scholar?q=Bob+Carpenter%2E+2008%2E+Multilevel+Bayesian+models+of+categorical+data+annotation%2E+Technical+report%2C+Ling%2DPipe%2E)
2. A. Philip Dawid and Allan M. Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics, pages 20–28. [find](https://scholar.google.com/scholar?q=A%2E+Philip+Dawid+and+Allan+M%2E+Skene%2E+1979%2E+Maximum+likelihood+estimation+of+observer+error%2Drates+using+the+EM+algorithm%2E+Applied+Statistics%2C+pages+20%E2%80%9328%2E)
3. Markus Dickinson and Detmar Meurers. 2003. Detecting errors in part-of-speech annotation. In EACL. [find](https://scholar.google.com/scholar?q=Markus+Dickinson+and+Detmar+Meurers%2E+2003%2E+Detecting+errors+in+part%2Dof%2Dspeech+annotation%2E+In+EACL%2E)
4. Cecily Duffield, Jena Hwang, Susan Brown, Dmitriy Dligach, Sarah Vieweg, Jenny Davis, and Martha Palmer. 2007. Criteria for the manual grouping of verb senses. In LAW. [find](https://scholar.google.com/scholar?q=Cecily+Duffield%2C+Jena+Hwang%2C+Susan+Brown%2C+Dmitriy+Dligach%2C+Sarah+Vieweg%2C+Jenny+Davis%2C+and+Martha+Palmer%2E+2007%2E+Criteria+for+the+manual+grouping+of+verb+senses%2E+In+LAW%2E)
5. Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A. Smith. 2011. Part-of-Speech Tagging for Twitter: Annotation, Features, and Experiments. In ACL. [find](https://scholar.google.com/scholar?q=Kevin+Gimpel%2C+Nathan+Schneider%2C+Brendan+O%E2%80%99Connor%2C+Dipanjan+Das%2C+Daniel+Mills%2C+Jacob+Eisenstein%2C+Michael+Heilman%2C+Dani+Yogatama%2C+Jeffrey+Flanigan%2C+and+Noah+A%2E+Smith%2E+2011%2E+Part%2Dof%2DSpeech+Tagging+for+Tw) [our card](https://md.research.elamur.ai/free/en/papers/gimpel-2011-part-speech-tagging-twitter-annotation/)
6. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom to trust with MACE. In NAACL. [find](https://scholar.google.com/scholar?q=Dirk+Hovy%2C+Taylor+Berg%2DKirkpatrick%2C+Ashish+Vaswani%2C+and+Eduard+Hovy%2E+2013%2E+Learning+whom+to+trust+with+MACE%2E+In+NAACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/hovy-2013-learning-whom-trust-mace/)
7. Dirk Hovy, Barbara Plank, and Anders Søgaard. 2014. Experiments with crowdsourced re-annotation of a POS tagging data set. In ACL. [find](https://scholar.google.com/scholar?q=Dirk+Hovy%2C+Barbara+Plank%2C+and+Anders+S%C3%B8gaard%2E+2014%2E+Experiments+with+crowdsourced+re%2Dannotation+of+a+POS+tagging+data+set%2E+In+ACL%2E)
8. David Juergens. 2014. An analysis of ambiguity in word sense annotations. In LREC. [find](https://scholar.google.com/scholar?q=David+Juergens%2E+2014%2E+An+analysis+of+ambiguity+in+word+sense+annotations%2E+In+LREC%2E)
9. Mitchell Marcus, Mary Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: the Penn Treebank. Computational Linguistics, 19(2):313–330. [find](https://scholar.google.com/scholar?q=Mitchell+Marcus%2C+Mary+Marcinkiewicz%2C+and+Beatrice+Santorini%2E+1993%2E+Building+a+large+annotated+corpus+of+English%3A+the+Penn+Treebank%2E+Computational+Linguistics%2C+19%282%29%3A313%E2%80%93330%2E) [our card](https://md.research.elamur.ai/free/en/papers/bank-1993-marcus-building-large-annotated-corpus-english/)
10. Slav Petrov, Dipanjan Das, and Ryan McDonald. 2012. A universal part-of-speech tagset. In LREC. [find](https://scholar.google.com/scholar?q=Slav+Petrov%2C+Dipanjan+Das%2C+and+Ryan+McDonald%2E+2012%2E+A+universal+part%2Dof%2Dspeech+tagset%2E+In+LREC%2E)
11. Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014. Learning part-of-speech taggers with inter-annotator agreement loss. In EACL. [find](https://scholar.google.com/scholar?q=Barbara+Plank%2C+Dirk+Hovy%2C+and+Anders+S%C3%B8gaard%2E+2014%2E+Learning+part%2Dof%2Dspeech+taggers+with+inter%2Dannotator+agreement+loss%2E+In+EACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/plank-2014-learning-part-speech-taggers-inter/)
12. Vikas C. Raykar and Shipeng Yu. 2012. Eliminating Spammers and Ranking Annotators for Crowdsourced Labeling Tasks. Journal of Machine Learning Research, 13:491–518. [find](https://scholar.google.com/scholar?q=Vikas+C%2E+Raykar+and+Shipeng+Yu%2E+2012%2E+Eliminating+Spammers+and+Ranking+Annotators+for+Crowdsourced+Labeling+Tasks%2E+Journal+of+Machine+Learning+Research%2C+13%3A491%E2%80%93518%2E)
13. Djamé Seddah, Benoit Sagot, Marie Candito, Virginie Mouilleron, and Vanessa Combet. 2012. The French Social Media Bank: a treebank of noisy user generated content. In COLING. [find](https://scholar.google.com/scholar?q=Djame%CC%81+Seddah%2C+Benoit+Sagot%2C+Marie+Candito%2C+Virginie+Mouilleron%2C+and+Vanessa+Combet%2E+2012%2E+The+French+Social+Media+Bank%3A+a+treebank+of+noisy+user+generated+content%2E+In+COLING%2E)
14. Padhraic Smyth, Usama Fayyad, Mike Burl, Pietro Perona, and Pierre Baldi. 1995. Inferring ground truth from subjective labelling of Venus images. In NIPS. [find](https://scholar.google.com/scholar?q=Padhraic+Smyth%2C+Usama+Fayyad%2C+Mike+Burl%2C+Pietro+Perona%2C+and+Pierre+Baldi%2E+1995%2E+Inferring+ground+truth+from+subjective+labelling+of+Venus+images%2E+In+NIPS%2E)
15. Peter Welinder, Steve Branson, Serge Belongie, and Pietro Perona. 2010. The multidimensional wisdom of crowds. In NIPS. [find](https://scholar.google.com/scholar?q=Peter+Welinder%2C+Steve+Branson%2C+Serge+Belongie%2C+and+Pietro+Perona%2E+2010%2E+The+multidimensional+wisdom+of+crowds%2E+In+NIPS%2E)
16. Jacob Whitehill, Paul Ruvolo, Tingfan Wu, Jacob Bergsma, and Javier Movellan. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In NIPS. [find](https://scholar.google.com/scholar?q=Jacob+Whitehill%2C+Paul+Ruvolo%2C+Tingfan+Wu%2C+Jacob+Bergsma%2C+and+Javier+Movellan%2E+2009%2E+Whose+vote+should+count+more%3A+Optimal+integration+of+labels+from+labelers+of+unknown+expertise%2E+In+NIPS%2E)
17. Yan Yan, Rómer Rosales, Glenn Fung, Mark Schmidt, Gerardo Hermosillo, Luca Bogoni, Linda Moy, and Jennifer Dy. 2010. Modeling annotator expertise: Learning when everybody knows a bit of something. In AIStats. [find](https://scholar.google.com/scholar?q=Yan+Yan%2C+Ro%CC%81mer+Rosales%2C+Glenn+Fung%2C+Mark+Schmidt%2C+Gerardo+Hermosillo%2C+Luca+Bogoni%2C+Linda+Moy%2C+and+Jennifer+Dy%2E+2010%2E+Modeling+annotator+expertise%3A+Learning+when+everybody+knows+a+bit+of+something%2E+In)
18. Daniel Zeman. 2010. Hard problems of tagset conversion. In Proceedings of the Second International Conference on Global Interoperability for Language Resources. [find](https://scholar.google.com/scholar?q=Daniel+Zeman%2E+2010%2E+Hard+problems+of+tagset+conversion%2E+In+Proceedings+of+the+Second+International+Conference+on+Global+Interoperability+for+Language+Resources%2E)