--- title: "Evaluating Inter-Annotator Agreement on — Bollmann 2016" description: "Normalisation labels are open and sparse, and most tokens are close to their surface form, so the usual agreement measures do not fit the task well. The…" authors: ["Marcel Bollmann", "Stefanie Dipper", "Florian Petran"] year: 2016 venue: "Proceedings of the 10th Linguistic Annotation Workshop held in conjunction with ACL 2016 (LAW-X 2016)" source: "https://aclanthology.org/W16-1711/" doi: "10.18653/v1/w16-1711" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/aclbulk-2016-bollmann-evaluating-inter-annotator-agreement-historical/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/aclbulk-2016-bollmann-evaluating-inter-annotator-agreement-historical/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/2/) / Evaluating Inter-Annotator Agreement on Historical Spelling Normalization # Evaluating Inter-Annotator Agreement on Historical Spelling Normalization **Authors:** [Marcel Bollmann](https://md.research.elamur.ai/free/en/authors/marcel-bollmann/), [Stefanie Dipper](https://md.research.elamur.ai/free/en/authors/stefanie-dipper/) and [Florian Petran](https://md.research.elamur.ai/free/en/authors/florian-petran/) · **Year:** 2016 · **Venue:** Proceedings of the 10th Linguistic Annotation Workshop held in conjunction with ACL 2016 (LAW-X 2016) **Link:** [aclanthology.org/W16-1711](https://aclanthology.org/W16-1711/) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) [**3** citations](https://openalex.org/works?filter=cites:W2515887364) OpenAlex · 2026-10-14 · [**1** citations in our library](https://md.research.elamur.ai/free/en/papers/aclbulk-2016-bollmann-evaluating-inter-annotator-agreement-historical/cited-by/) of the 140673 papers whose bibliographies we read · [Proceedings of the 10th Linguistic Annotation Workshop held in conjunction with ACL 2016 (LAW-X 2016)](https://aclanthology.org/volumes/W16-17/) · [licence](https://creativecommons.org/licenses/by/4.0/) - [Publisher's page](https://aclanthology.org/W16-1711/) - [PDF](https://aclanthology.org/W16-1711.pdf) - [DOI 10.18653/v1/w16-1711](https://doi.org/10.18653/v1/w16-1711) - [OpenAlex](https://openalex.org/W2515887364) **Authors (3)** 1. [Marcel Bollmann](https://openalex.org/A5005438563) [ORCID](https://orcid.org/0000-0003-2598-8150) · [Semantic Scholar](https://www.semanticscholar.org/author/34887843) · h 11 · 318 citations · 42 works 2. [Stefanie Dipper](https://openalex.org/A5063444213) [ORCID](https://orcid.org/0000-0003-4357-9078) · [Semantic Scholar](https://www.semanticscholar.org/author/1889046) · h 18 · 1104 citations · 146 works 3. [Florian Petran](https://openalex.org/A5031866050) [Semantic Scholar](https://www.semanticscholar.org/author/2964686) · h 5 · 101 citations · 13 works Citations and author metrics: OpenAlex, 2026-10-14; authors with the same name are not always told apart. ## What it is about Normalisation labels are open and sparse, and most tokens are close to their surface form, so the usual agreement measures do not fit the task well. The paper proposes a way to use chance-corrected agreement (Krippendorff's alpha) with a character-level edit distance, so that near-identical normalisations count as partial agreement. Five annotators labelled a German dataset. Word-based percentage agreement over all tokens is 92.62%. The character-based alpha is higher than the word-based one, because partial matches count. The paper points out that the word-based figure of 96.9% reported for another historical German corpus ignores partial agreement and chance. ## Ideas - I1. `en morph-eval`: add an edit-distance agreement score next to exact accuracy, so that a one-letter miss is shown apart from a wrong lemma. Measure both on the lemma set, where accuracy is 99.3–99.5%, and report the split. - I2. Distance-aware error report: in `en/src/morph.rs`, group the errors by edit distance to the gold form, and give each group its count per lesson. - I3. Double annotation for archaic tokens: two annotators label about 200 archaic tokens from the lessons, and the chance-corrected agreement is computed before any lemma accuracy for them is claimed. This is a test set only. ## For Mova The chapter is [languages/en/06-spelling-rules.md](https://mova.elamur.ai/free/en/docs/languages/en/06-spelling-rules/), with [languages/en/09-early-modern-and-archaic.md](https://mova.elamur.ai/free/en/docs/languages/en/09-early-modern-and-archaic/) for the forms. The code is `en/src/morph.rs` and `en morph-eval`. The paper is about German. Its method is for evaluation only, and it adds no rule to the build. ## Cite The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/w16-1711](https://doi.org/10.18653/v1/w16-1711) ```bibtex @inproceedings{bollmann2016evaluating, title = {{Evaluating Inter-Annotator Agreement on Historical Spelling Normalization}}, author = {Bollmann, Marcel and Dipper, Stefanie and Petran, Florian}, booktitle = {Proceedings of the 10th Linguistic Annotation Workshop held in conjunction with ACL 2016 (LAW-X 2016)}, year = {2016}, doi = {10.18653/v1/w16-1711}, url = {https://doi.org/10.18653/v1/w16-1711} } ``` - [← previous Equality of Opportunity in Supervised Learning](https://md.research.elamur.ai/free/en/papers/hardt-2016-equality-opportunity-supervised-learning/) - [next → Event Detection and Co-reference with Minimal Supervision](https://md.research.elamur.ai/free/en/papers/peng-2016-event-detection-co-reference-minimal/) - [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/2/) ## References 12 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555–596. [↗](https://doi.org/10.1162/coli.07-034-r2) [our card](https://md.research.elamur.ai/free/en/papers/artstein-2008-survey-article-inter-coder-agreement/) 2. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. [↗](https://doi.org/10.1177/001316446002000104) 3. Stefanie Dipper and Simone Schultz-Balluff. 2013. The Anselm corpus: Methods and perspectives of a parallel aligned corpus. In Proceedings of the NODALIDA Workshop on Computational Historical Linguistics, Oslo, Norway. [find](https://scholar.google.com/scholar?q=Stefanie+Dipper+and+Simone+Schultz%2DBalluff%2E+2013%2E+The+Anselm+corpus%3A+Methods+and+perspectives+of+a+parallel+aligned+corpus%2E+In+Proceedings+of+the+NODALIDA+Workshop+on+Computational+Historical+Linguist) 4. Joseph L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378–382. [↗](https://doi.org/10.1037/h0031619) 5. Julia Krasselt, Marcel Bollmann, Stefanie Dipper, and Florian Petran. 2015. Guidelines for normalizing historical German texts. Bochumer Linguistische Arbeitsberichte, 15. [find](https://scholar.google.com/scholar?q=Julia+Krasselt%2C+Marcel+Bollmann%2C+Stefanie+Dipper%2C+and+Florian+Petran%2E+2015%2E+Guidelines+for+normalizing+historical+German+texts%2E+Bochumer+Linguistische+Arbeitsberichte%2C+15%2E) 6. Klaus Krippendorff. 1980. Content Analysis: An Introduction to Its Methodology. SAGE, Beverly Hills, CA. [find](https://scholar.google.com/scholar?q=Klaus+Krippendorff%2E+1980%2E+Content+Analysis%3A+An+Introduction+to+Its+Methodology%2E+SAGE%2C+Beverly+Hills%2C+CA%2E) 7. Vladimir I. Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady, 10(8):707–710. [find](https://scholar.google.com/scholar?q=Vladimir+I%2E+Levenshtein%2E+1966%2E+Binary+codes+capable+of+correcting+deletions%2C+insertions%2C+and+reversals%2E+Soviet+Physics+Doklady%2C+10%288%29%3A707%E2%80%93710%2E) 8. Johann-Mattis List and Robert Forkel. 2016. LingPy. A Python library for historical linguistics. Version 2.4. [http://lingpy.org](http://lingpy.org). With collaborations by Steven Moran, Peter Bouda, Johannes Dellert, Taraka Rama, Frank Nagel, and Simon Greenhill: Max Planck Institute for the Science of Human History. [↗](http://lingpy.org) 9. Saul B. Needleman and Christian D. Wunsch. 1970. A gene method applicable to the search for similarities in the amino acid sequence of two proteins. Journal of Molecular Biology, 48:443–453. [find](https://scholar.google.com/scholar?q=Saul+B%2E+Needleman+and+Christian+D%2E+Wunsch%2E+1970%2E+A+gene+method+applicable+to+the+search+for+similarities+in+the+amino+acid+sequence+of+two+proteins%2E+Journal+of+Molecular+Biology%2C+48%3A443%E2%80%93453%2E) 10. Rebecca J. Passonneau. 2004. Computing reliability for coreference annotation. In Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC), volume 4, pages 1503–1506, Lisbon, Portugal. [find](https://scholar.google.com/scholar?q=Rebecca+J%2E+Passonneau%2E+2004%2E+Computing+reliability+for+coreference+annotation%2E+In+Proceedings+of+the+4th+International+Conference+on+Language+Resources+and+Evaluation+%28LREC%29%2C+volume+4%2C+pages+1503%E2%80%931506) 11. Silke Scheible, Richard J. Whitt, Martin Durrell, and Paul Bennett. 2011. A gold standard corpus of Early Modern German. In Proceedings of the ACL-HLT 2011 Linguistic Annotation Workshop (LAW V), pages 124–128, Portland, Oregon, USA. [find](https://scholar.google.com/scholar?q=Silke+Scheible%2C+Richard+J%2E+Whitt%2C+Martin+Durrell%2C+and+Paul+Bennett%2E+2011%2E+A+gold+standard+corpus+of+Early+Modern+German%2E+In+Proceedings+of+the+ACL%2DHLT+2011+Linguistic+Annotation+Workshop+%28LAW+V%29%2C+page) [our card](https://md.research.elamur.ai/free/en/papers/aclbulk-2011-scheible-gold-standard-corpus-early-modern/) 12. William A. Scott. 1955. Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly, 19(3):321–325. [↗](https://doi.org/10.1086/266577)