--- title: "Different Flavors of GUM — Zeldes 2016" description: "Tagging and parsing accuracy is usually reported on newswire, and genre is the usual explanation when it falls… 3 references, cited by 10 papers here." authors: ["Amir Zeldes", "Dan Simonson"] year: 2016 venue: "Proceedings of the 10th Linguistic Annotation Workshop held in conjunction with ACL 2016 (LAW-X 2016)" source: "https://aclanthology.org/W16-1709/" doi: "10.18653/v1/w16-1709" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/zeldes-2016-different-flavors-gum-evaluating-genre/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zeldes-2016-different-flavors-gum-evaluating-genre/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/) / Different Flavors of GUM: Evaluating Genre and Sentence Type Effects on Multilayer Corpus Annotation Quality # Different Flavors of GUM: Evaluating Genre and Sentence Type Effects on Multilayer Corpus Annotation Quality **Authors:** [Amir Zeldes](https://md.research.elamur.ai/free/en/authors/amir-zeldes/) and Dan Simonson · **Year:** 2016 · **Venue:** Proceedings of the 10th Linguistic Annotation Workshop held in conjunction with ACL 2016 (LAW-X 2016) **Link:** [aclanthology.org/W16-1709](https://aclanthology.org/W16-1709/) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) [**7** citations](https://openalex.org/works?filter=cites:W2513006088) OpenAlex, 2026-10-14 · [**10** citations in our library](https://md.research.elamur.ai/free/en/papers/zeldes-2016-different-flavors-gum-evaluating-genre/cited-by/) of the 140,884 papers whose bibliographies we read · [Proceedings of the 10th Linguistic Annotation Workshop held in conjunction with ACL 2016 (LAW-X 2016)](https://aclanthology.org/volumes/W16-17/) · [licence](https://creativecommons.org/licenses/by/4.0/) - [Publisher's page](https://aclanthology.org/W16-1709/) - [PDF](https://aclanthology.org/W16-1709.pdf) - [DOI 10.18653/v1/w16-1709](https://doi.org/10.18653/v1/w16-1709) - [OpenAlex](https://openalex.org/W2513006088) **Authors (2)** 1. [Amir Zeldes](https://openalex.org/A5089212858) [ORCID](https://orcid.org/0000-0001-8016-6753) · [Semantic Scholar](https://www.semanticscholar.org/author/2548420) · h 16 · 1,524 citations · 238 works 2. [Dan Simonson](https://openalex.org/A5017060959) [Semantic Scholar](https://www.semanticscholar.org/author/48531944) · h 4 · 28 citations · 8 works Citations and author metrics: OpenAlex, 2026-10-14; authors with the same name are not always told apart. ## What it is about Tagging and parsing accuracy is usually reported on newswire, and genre is the usual explanation when it falls elsewhere. The authors ask whether sentence type (declarative, imperative, question, fragment, and so on) explains annotation quality better than genre. Using the GUM corpus (54 documents, about 44,000 tokens, four genres: news, interviews, travel guides, how-to guides), they compare uncorrected student annotations, the corrected gold, and three automatic taggers and one parser, with mixed-effects models that include document as a random effect and sentence length as a control. Sentence type predicts errors across the three tasks (POS tagging, dependency correction, coreference), and genre mostly disappears once document is in the model. Fragments and non-declarative types are the hardest for the taggers and the parser, and imperatives, which the EWT paper singles out, are not a strong problem. ## For Mova - I1. `en tag-eval` and `en ud-eval` broken down by sentence type (decl, imp, q, wh, frag, sub, inf) and by genre, with a mixed-effects check that sentence type stays significant when document is a random effect; measure UPOS and LAS per sentence type on GUM test (test only). - I2. `en/src/ptag.rs`: a sentence-type feature from the first word and the presence of a finite verb, to help the tagger on fragments and headings; measure UPOS on fragments and gerund headings in GUM test before and after. - I3. `annot` routing: send fragment and other-type sentences to a second teacher pass with a stricter gate, since the paper shows they carry most errors; measure gate violations per 1k silver sentences by sentence type with `treeagree check`. - I4. `en/src/ctx.rs`: document-level features (genre tag or a heading flag) only where they add to the sentence-type model; measure with the mixed-effects check, not by accuracy alone, since the paper finds genre adds nothing once sentence type is in. Touches `languages/en/37-register-and-variation`, `10-word-classes` and `38-dialogue-and-discourse-markers`, and the `en` crate (`ptag.rs`, `ctx.rs`, `parse.rs`), `annot` and `tools/treeagree`. GUM is non-commercial: test only, never in a shipped model. Measure first the per-sentence-type breakdown of the current `en` scores on GUM test, which costs nothing and shows whether fragments are the weak spot. ## Institutions [🎓**Georgetown University** Washington · United States 62 works · 4,229 citations](https://md.research.elamur.ai/free/en/universities/georgetown-university/) ## Cite Cite as Amir Zeldes and Dan Simonson. 2016. Different Flavors of GUM: Evaluating Genre and Sentence Type Effects on Multilayer Corpus Annotation Quality. Proceedings of the 10th Linguistic Annotation Workshop held in conjunction with ACL 2016 (LAW-X 2016). The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/w16-1709](https://doi.org/10.18653/v1/w16-1709) ```bibtex @inproceedings{zeldes2016different, title = {{Different Flavors of GUM: Evaluating Genre and Sentence Type Effects on Multilayer Corpus Annotation Quality}}, author = {Amir Zeldes and Dan Simonson}, booktitle = {Proceedings of the 10th Linguistic Annotation Workshop held in conjunction with ACL 2016 (LAW-X 2016)}, year = {2016}, doi = {10.18653/v1/w16-1709}, url = {https://doi.org/10.18653/v1/w16-1709} } ``` - [← previous Antecedent Selection for Sluicing: Structure and Content](https://md.research.elamur.ai/free/en/papers/anand-2016-antecedent-selection-sluicing-structure-content/) - [next → The Hebrew FrameNet Project](https://md.research.elamur.ai/free/en/papers/hayoun-2016-hebrew-framenet-project/) - [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/) ## References 3 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Emilia Apostolova, Noriko Tomuro, Pattanasak Mongkolwat and Dina Demner-Fushman. 2012. Domain Adaptation of Coreference Resolution for Radiology Reports. In Proceedings of the 2012 Workshop on Biomedical Natural Language Processing (BioNLP 2012). Montreal, 118–121. R. Harald. [our card](https://md.research.elamur.ai/free/en/papers/apostolova-2012-domain-adaptation-coreference-resolution-radiology/) 2. Baayen. 2008. Analyzing Linguistic Data. A Practical Introduction to Statistics using R. Cambridge: Cambridge University Press. [find](https://scholar.google.com/scholar?q=Baayen%2E+2008%2E+Analyzing+Linguistic+Data%2E+A+Practical+Introduction+to+Statistics+using+R%2E+Cambridge%3A+Cambridge+University+Press%2E) 3. Douglas Biber. 2009. Multi-Dimensional Approaches. In Anke Lüdeling & Merja Kytö (eds.), Corpus [find](https://scholar.google.com/scholar?q=Douglas+Biber%2E+2009%2E+Multi%2DDimensional+Approaches%2E+In+Anke+L%C3%BCdeling+%26+Merja+Kyt%C3%B6+%28eds%2E%29%2C+Corpus)