--- title: "Evaluating Inter-Annotator Agreement on Historical Spelling" description: "The paper proposes a way to use chance-corrected agreement (Krippendorff's alpha) with a… 12 references, cited by 1 paper here." authors: ["Marcel Bollmann", "Stefanie Dipper", "Florian Petran"] year: 2016 venue: "Proceedings of the 10th Linguistic Annotation Workshop held in conjunction with ACL 2016 (LAW-X 2016)" source: "https://aclanthology.org/W16-1711/" doi: "10.18653/v1/w16-1711" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/bollmann-2016-evaluating-inter-annotator-agreement-historical/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bollmann-2016-evaluating-inter-annotator-agreement-historical/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/4/) / Evaluating Inter-Annotator Agreement on Historical Spelling Normalization # Evaluating Inter-Annotator Agreement on Historical Spelling Normalization Paper · 2016 · Marcel Bollmann, Stefanie Dipper and Florian Petran **Authors:** [Marcel Bollmann](https://md.research.elamur.ai/free/en/authors/marcel-bollmann/), [Stefanie Dipper](https://md.research.elamur.ai/free/en/authors/stefanie-dipper/) and [Florian Petran](https://md.research.elamur.ai/free/en/authors/florian-petran/) · **Year:** 2016 · **Venue:** Proceedings of the 10th Linguistic Annotation Workshop held in conjunction with ACL 2016 (LAW-X 2016) **Link:** [aclanthology.org/W16-1711](https://aclanthology.org/W16-1711/) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) [**3** citations OpenAlex, 2026-10-14](https://openalex.org/works?filter=cites:W2515887364) · Cited by **1** paper in our library of the 141,840 papers whose bibliographies we read · [Linguistic Annotation Workshop held in conjunction with ACL](https://md.research.elamur.ai/free/en/venues/linguistic-annotation-workshop-held-in-conjunction-with-acl/) · [Linguistic Annotation Workshop held in conjunction with ACL 2016](https://md.research.elamur.ai/free/en/venues/linguistic-annotation-workshop-held-in-conjunction-with-acl/2016/) · published by [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · [licence](https://creativecommons.org/licenses/by/4.0/) - [Publisher's page](https://aclanthology.org/W16-1711/) - [PDF](https://aclanthology.org/W16-1711.pdf) - [DOI 10.18653/v1/w16-1711](https://doi.org/10.18653/v1/w16-1711) - [OpenAlex](https://openalex.org/W2515887364) **Authors (3)** 1. [Marcel Bollmann](https://openalex.org/A5005438563) [ORCID](https://orcid.org/0000-0003-2598-8150) · [Semantic Scholar](https://www.semanticscholar.org/author/34887843) · h 11 · 318 citations · 42 works 2. [Stefanie Dipper](https://openalex.org/A5063444213) [ORCID](https://orcid.org/0000-0003-4357-9078) · [Semantic Scholar](https://www.semanticscholar.org/author/1889046) · h 18 · 1,104 citations · 146 works 3. [Florian Petran](https://openalex.org/A5031866050) [Semantic Scholar](https://www.semanticscholar.org/author/2964686) · h 5 · 101 citations · 13 works Citations and author metrics: OpenAlex, 2026-10-14; authors with the same name are not always told apart. ## What it is about Normalisation labels are open and sparse, and most tokens are close to their surface form, so the usual agreement measures do not fit the task well. The paper proposes a way to use chance-corrected agreement (Krippendorff's alpha) with a character-level edit distance, so that near-identical normalisations count as partial agreement. Five annotators labelled a German dataset. Word-based percentage agreement over all tokens is 92.62%. The character-based alpha is higher than the word-based one, because partial matches count. The paper points out that the word-based figure of 96.9% reported for another historical German corpus ignores partial agreement and chance. ## Ideas - I3. Double annotation for archaic tokens: two annotators label about 200 archaic tokens from the lessons, and the chance-corrected agreement is computed before any lemma accuracy for them is claimed. This is a test set only. ## For Mova - I1. `en morph-eval`: add an edit-distance agreement score next to exact accuracy, so that a one-letter miss is shown apart from a wrong lemma. Measure both on the lemma set, where accuracy is 99.3–99.5%, and report the split. - I2. Distance-aware error report: in `en/src/morph.rs`, group the errors by edit distance to the gold form, and give each group its count per lesson. The chapter is [languages/en/06-spelling-rules.md](https://mova.elamur.ai/free/en/docs/languages/en/06-spelling-rules/), with [languages/en/09-early-modern-and-archaic.md](https://mova.elamur.ai/free/en/docs/languages/en/09-early-modern-and-archaic/) for the forms. The code is `en/src/morph.rs` and `en morph-eval`. The paper is about German. Its method is for evaluation only, and it adds no rule to the build. ## Institutions [🎓**Ruhr University Bochum** Bochum · Germany 20 works · 1,151 citations](https://md.research.elamur.ai/free/en/universities/ruhr-university-bochum/) ## Fields, topics and concepts 📚 Shelves 7 The field of science this work belongs to (OpenAlex fields).[Computer Science · 9,085 works](https://md.research.elamur.ai/free/en/shelves/computer-science/) · [Artificial Intelligence · 8,888 works](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · [Arts and Humanities · 9,296 works](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · [Language and Linguistics · 9,206 works](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · [Decision Sciences · 85 works](https://md.research.elamur.ai/free/en/shelves/decision-sciences/) · [Literature and Literary Theory · 340 works](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/literature-and-literary-theory/) · [Statistics, Probability and Uncertainty · 32 works](https://md.research.elamur.ai/free/en/shelves/decision-sciences/statistics-probability-and-uncertainty/) 🧭 Topics 2 Research subjects and languages the work is about.[English · 1,373 works](https://md.research.elamur.ai/free/en/topics/english/) · [Evaluation & benchmarks · 685 works](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) ## Cite Cite as Marcel Bollmann, Stefanie Dipper and Florian Petran. 2016. Evaluating Inter-Annotator Agreement on Historical Spelling Normalization. Proceedings of the 10th Linguistic Annotation Workshop held in conjunction with ACL 2016 (LAW-X 2016). The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/w16-1711](https://doi.org/10.18653/v1/w16-1711) ```bibtex @inproceedings{bollmann2016evaluating, title = {{Evaluating Inter-Annotator Agreement on Historical Spelling Normalization}}, author = {Marcel Bollmann, Stefanie Dipper and Florian Petran}, booktitle = {Proceedings of the 10th Linguistic Annotation Workshop held in conjunction with ACL 2016 (LAW-X 2016)}, year = {2016}, doi = {10.18653/v1/w16-1711}, url = {https://doi.org/10.18653/v1/w16-1711} } ``` - [← previous ‘Who would have thought of that!': A Hierarchical Topic Model for Extraction of Sarcasm-prevalent Topics and Sarcasm Detection](https://md.research.elamur.ai/free/en/papers/joshi-2016-who-would-have-thought-that/) - [next → Interrogative Slifting: More syntactic, less parenthetical](https://md.research.elamur.ai/free/en/papers/stepanov-2016-interrogative-slifting-more-syntactic-less/) - [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/4/) ## References 12 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Ron Artstein and Massimo Poesio. 2008. [Inter-coder agreement for computational linguistics](https://md.research.elamur.ai/free/en/papers/artstein-2008-survey-article-inter-coder-agreement/). Computational Linguistics, 34(4):555–596. [↗](https://doi.org/10.1162/coli.07-034-r2) 2. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. [↗](https://doi.org/10.1177/001316446002000104) 3. Stefanie Dipper and Simone Schultz-Balluff. 2013. The Anselm corpus: Methods and perspectives of a parallel aligned corpus. In Proceedings of the NODALIDA Workshop on Computational Historical Linguistics, Oslo, Norway. [find](https://scholar.google.com/scholar?q=Stefanie+Dipper+and+Simone+Schultz%2DBalluff%2E+2013%2E+The+Anselm+corpus%3A+Methods+and+perspectives+of+a+parallel+aligned+corpus%2E+In+Proceedings+of+the+NODALIDA+Workshop+on+Computational+Historical+Linguist) 4. Joseph L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378–382. [↗](https://doi.org/10.1037/h0031619) 5. Julia Krasselt, Marcel Bollmann, Stefanie Dipper, and Florian Petran. 2015. Guidelines for normalizing historical German texts. Bochumer Linguistische Arbeitsberichte, 15. [find](https://scholar.google.com/scholar?q=Julia+Krasselt%2C+Marcel+Bollmann%2C+Stefanie+Dipper%2C+and+Florian+Petran%2E+2015%2E+Guidelines+for+normalizing+historical+German+texts%2E+Bochumer+Linguistische+Arbeitsberichte%2C+15%2E) 6. Klaus Krippendorff. 1980. Content Analysis: An Introduction to Its Methodology. SAGE, Beverly Hills, CA. [find](https://scholar.google.com/scholar?q=Klaus+Krippendorff%2E+1980%2E+Content+Analysis%3A+An+Introduction+to+Its+Methodology%2E+SAGE%2C+Beverly+Hills%2C+CA%2E) 7. Vladimir I. Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady, 10(8):707–710. [find](https://scholar.google.com/scholar?q=Vladimir+I%2E+Levenshtein%2E+1966%2E+Binary+codes+capable+of+correcting+deletions%2C+insertions%2C+and+reversals%2E+Soviet+Physics+Doklady%2C+10%288%29%3A707%E2%80%93710%2E) 8. Johann-Mattis List and Robert Forkel. 2016. LingPy. A Python library for historical linguistics. Version 2.4. [http://lingpy.org](http://lingpy.org). With collaborations by Steven Moran, Peter Bouda, Johannes Dellert, Taraka Rama, Frank Nagel, and Simon Greenhill: Max Planck Institute for the Science of Human History. [↗](http://lingpy.org) 9. Saul B. Needleman and Christian D. Wunsch. 1970. A gene method applicable to the search for similarities in the amino acid sequence of two proteins. Journal of Molecular Biology, 48:443–453. [find](https://scholar.google.com/scholar?q=Saul+B%2E+Needleman+and+Christian+D%2E+Wunsch%2E+1970%2E+A+gene+method+applicable+to+the+search+for+similarities+in+the+amino+acid+sequence+of+two+proteins%2E+Journal+of+Molecular+Biology%2C+48%3A443%E2%80%93453%2E) 10. Rebecca J. Passonneau. 2004. Computing reliability for coreference annotation. In Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC), volume 4, pages 1503–1506, Lisbon, Portugal. [find](https://scholar.google.com/scholar?q=Rebecca+J%2E+Passonneau%2E+2004%2E+Computing+reliability+for+coreference+annotation%2E+In+Proceedings+of+the+4th+International+Conference+on+Language+Resources+and+Evaluation+%28LREC%29%2C+volume+4%2C+pages+1503%E2%80%931506) 11. Silke Scheible, Richard J. Whitt, Martin Durrell, and Paul Bennett. 2011. [A gold standard corpus of Early Modern German](https://md.research.elamur.ai/free/en/papers/scheible-2011-gold-standard-corpus-early-modern/). In Proceedings of the ACL-HLT 2011 Linguistic Annotation Workshop (LAW V), pages 124–128, Portland, Oregon, USA. 12. William A. Scott. 1955. Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly, 19(3):321–325. [↗](https://doi.org/10.1086/266577) Updated 24 October 2026