--- title: "Evaluating the Quality of a Corpus Annotation — Akkurt 2024" description: "The authors propose a new way to compare alternative corpus annotation schemes." authors: ["Furkan Akkurt", "Onur Gungor", "Büşra Marşan", "Tunga Gungor", "Balkiz Ozturk Basaran", "Arzucan Özgür", "Susan Uskudarli"] year: 2024 venue: "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)" source: "https://aclanthology.org/2024.lrec-main.577/" licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)" licence_url: "https://creativecommons.org/licenses/by-nc/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/papers/akkurt-2024-evaluating-quality-corpus-annotation-scheme/" --- [Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/akkurt-2024-evaluating-quality-corpus-annotation-scheme/) [home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2024](https://md.research.elamur.ai/free/papers/2024/page/2/) / Evaluating the Quality of a Corpus Annotation Scheme Using Pretrained Language Models # Evaluating the Quality of a Corpus Annotation Scheme Using Pretrained Language Models **Authors:** Furkan Akkurt, [Onur Gungor](https://md.research.elamur.ai/free/authors/onur-gungor/), [Büşra Marşan](https://md.research.elamur.ai/free/authors/busra-marsan/), [Tunga Gungor](https://md.research.elamur.ai/free/authors/tunga-gungor/), [Balkiz Ozturk Basaran](https://md.research.elamur.ai/free/authors/balkiz-ozturk-basaran/), [Arzucan Özgür](https://md.research.elamur.ai/free/authors/arzucan-ozgur/), Susan Uskudarli · **Year:** 2024 · **Venue:** Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2024.lrec-main.577/) · [PDF](https://aclanthology.org/2024.lrec-main.577.pdf) · [OpenAlex](https://openalex.org/W7131797478) **Citations:** 0 citations (openalex-snapshot · 2026-10-14) · 2 in our library, of the 140673 papers whose bibliographies we read ## Summary The authors propose a new way to compare alternative corpus annotation schemes. Conventional approaches demand a lot of expert time to measure inter-annotator agreement or performance on relevant downstream tasks. Here, large language models (LLMs) are used instead. The idea is that a model with a good command of the language should find it easier to reconstruct the original sentence from a more complete and accurate annotation. The method was tested on two versions of the Turkish UD treebank BOUN. Version 2.8 was annotated semi-automatically and then manually proofread. In version 2.11, native-speaker linguists fully re-annotated the treebank by hand and took a new approach to null morphemes, lemmatization, the conflation of derivation with inflection, and syncretism. For each sentence, the prompt gives a natural-language description of each token's lemma, part of speech and morphological features. The model has to reconstruct the word forms in the correct order, and the output is compared with the original. On a subset of the treebank, version 2.11 yielded a success rate about 2 points higher. The approach is inexpensive and, thanks to multilingual LLMs, works for any language. It can therefore also be applied to morphologically rich Ukrainian UD resources. ## Key points - The BOUN treebank contains 9,761 sentences from various genres, randomly sampled from the Turkish National Corpus. Word order is free: the most frequent constructions are OV (~37%) and SV (~35%). - Version 2.8 was first annotated automatically: syntax with the Kanerva et al. pipeline and morphology with the Sak et al. disambiguator. Native speakers then checked the annotation. In version 2.11, the whole treebank was re-annotated manually. - Only token-level annotation is evaluated: lemmas, parts of speech and morphological features. Dependencies between tokens are not part of the comparison. - Main result: on a subset of sentences, version 2.11 lets the model reproduce the sentence exactly about 2 points more often. The authors also analyze reconstruction accuracy for individual tokens and the contribution of individual morphological features. - The code is published at github.com/boun-tabi/eval-ud so that the method can be applied to other languages and treebanks. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) Topics: [Treebanks](https://md.research.elamur.ai/free/topics/treebanks/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{akkurt2024evaluating, title = {{Evaluating the Quality of a Corpus Annotation Scheme Using Pretrained Language Models}}, author = {Akkurt, Furkan and Gungor, Onur and Marşan, Büşra and Gungor, Tunga and Basaran, Balkiz Ozturk and Özgür, Arzucan and Uskudarli, Susan}, booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)}, year = {2024}, doi = {10.63317/2zy52xp7m2v5}, url = {https://doi.org/10.63317/2zy52xp7m2v5} } ``` - [← previous Evaluating large language models for the tasks of PoS tagging within the Universal Dependency framework](https://md.research.elamur.ai/free/papers/machado-2024-evaluating-large-language-models-tasks/) - [next → Executable Code Actions Elicit Better LLM Agents](https://md.research.elamur.ai/free/papers/wang-2024-executable-code-actions-elicit-better/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/papers/2024/page/2/) ## References 2 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Wasi Uddin Ahmad, Zhisong Zhang, Xuezhe Ma, Kai-Wei Chang, and Nanyun Peng. 2019. Cross-lingual dependency parsing with unlabeled auxiliary languages. arXiv preprint arXiv:1909.09265. [find](https://scholar.google.com/scholar?q=Wasi+Uddin+Ahmad%2C+Zhisong+Zhang%2C+Xuezhe+Ma%2C+Kai%2DWei+Chang%2C+and+Nanyun+Peng%2E+2019%2E+Cross%2Dlingual+dependency+parsing+with+unlabeled+auxiliary+languages%2E+arXiv+preprint+arXiv%3A1909%2E09265%2E) 2. Yesim Aksan, Mustafa Aksan, Ahmet Koltuksuz, Taner Sezer, Ümit Mersinli, Umut Ufuk Demirhan, Hakan Yilmazer, Gülsüm Atasoy, Seda Öz, Ipek Yildiz, et al. 2012. Construction of the Turkish national corpus (TNC). In LREC, pages 3223– 3227. [find](https://scholar.google.com/scholar?q=Yesim+Aksan%2C+Mustafa+Aksan%2C+Ahmet+Koltuksuz%2C+Taner+Sezer%2C+%C3%9Cmit+Mersinli%2C+Umut+Ufuk+Demirhan%2C+Hakan+Yilmazer%2C+G%C3%BCls%C3%BCm+Atasoy%2C+Seda+%C3%96z%2C+Ipek+Yildiz%2C+et+al%2E+2012%2E+Construction+of+the+Turkish+national+corp)