--- title: "Corpus Annotation for Parser Evaluation — Carroll 1999" description: "The authors propose an annotation scheme that records grammatical relations between heads and dependents… 24 references, cited by 28 papers here." authors: ["John Carroll", "Guido Minnen", "Ted Briscoe"] year: 1999 venue: "Proceedings of the EACL99 workshop on Linguistically Interpreted Corpora (LINC), Bergen, Norway, June 12" source: "https://arxiv.org/abs/cs/9907013" arxiv: "cs/9907013" licence: "arXiv non-exclusive distribution licence (read only, no redistribution)" licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/carroll-1999-corpus-annotation-parser-evaluation/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/carroll-1999-corpus-annotation-parser-evaluation/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [1999](https://md.research.elamur.ai/free/en/papers/1999/) / Corpus Annotation for Parser Evaluation # Corpus Annotation for Parser Evaluation **Authors:** [Ted Briscoe](https://md.research.elamur.ai/free/en/authors/ted-briscoe/), [John Carroll](https://md.research.elamur.ai/free/en/authors/john-carroll/) and [Guido Minnen](https://md.research.elamur.ai/free/en/authors/guido-minnen/) · **Year:** 1999 · **Venue:** Proceedings of the EACL Workshop on Linguistically Interpreted Corpora (LINC), Bergen, Norway **Link:** [arxiv.org/abs/cs/9907013](https://arxiv.org/abs/cs/9907013) · **Licence:** arXiv non-exclusive licence — summary only [**83** citations](https://openalex.org/works?filter=cites:W1625716732) OpenAlex, 2026-10-14 · [**28** citations in our library](https://md.research.elamur.ai/free/en/papers/carroll-1999-corpus-annotation-parser-evaluation/cited-by/) of the 140,884 papers whose bibliographies we read · [Proceedings of the EACL99 workshop on Linguistically Interpreted Corpora (LINC), Bergen, Norway, June 12](https://openalex.org/S4306400194) · [licence](http://arxiv.org/licenses/nonexclusive-distrib/1.0/) - [Publisher's page](https://arxiv.org/abs/cs/9907013) - [PDF](https://arxiv.org/pdf/cs/9907013) - [arXiv cs/9907013](https://arxiv.org/abs/cs/9907013) - [OpenAlex](https://openalex.org/W1625716732) **Authors (3)** 1. [John Carroll](https://openalex.org/A5101474712) [ORCID](https://orcid.org/0000-0003-3744-4919) · h 23 · 2,418 citations · 118 works 2. [Guido Minnen](https://openalex.org/A5073035830) h 8 · 861 citations · 28 works 3. [Ted Briscoe](https://openalex.org/A5030189297) h 42 · 7,511 citations · 222 works Citations and author metrics: OpenAlex, 2026-10-14; authors with the same name are not always told apart. ## What it is about Bracket-based parser scores (PARSEVAL) are hard to compare across grammars, because the bracketing of a treebank and the output of a parser differ in ways that are not errors. The authors propose an annotation scheme that records grammatical relations between heads and dependents (subject, object, modifier and so on), organised in a hierarchy so that a shallow parser may leave a label underspecified. They mark up a corpus of 500 sentences (10K words) taken from SUSANNE, with 9.72 relations per sentence and around 95% inter-annotator agreement. A robust lexicalised parser scores an F-score of 75.1 on the general dependent relation, 86.5 on non-clausal subjects and 50.7 on indirect objects. ## For Mova - I1. `en parse-eval`: add a per-label table (precision, recall, F) for the UD labels on EWT test, in the paper's style, with the general `dep` score first. Measure which labels lower LAS most, and start with the lowest-F label. - I2. Hierarchical diagnostic: count a predicted label as a near miss when it shares a parent with the gold label (`nsubj` and `csubj`, `obl` and `nmod`). Report this beside LAS as a diagnostic only, never as a tuning target. Measure: the gap between LAS and the near-miss score on EWT test. - I3. Control in `en/src/expert.rs`: a rule that adds a second, enhanced edge when one nominal is the subject of two verbs (the paper's example of control). Trees stay single-headed; the extra edge is reported only. Measure: `nsubj` F on EWT test with and without the enhanced edges, and the number of such cases. The chapters are [languages/en/20-syntax-overview.md](https://mova.elamur.ai/free/en/docs/languages/en/20-syntax-overview/) and `23-complementation.md` (control). The code is `en/src/parse.rs` with the `en parse-eval` command, and `en/src/expert.rs` for the control rule. The paper's corpus is not in Mova's data, and its numbers belong to a 1999 parser, so they are not comparable. Every figure must come from EWT test, which is measured once per release. ## Institutions [🎓**University of Cambridge** Cambridge · United Kingdom 119 works · 110,950 citations](https://md.research.elamur.ai/free/en/universities/university-of-cambridge/) · [🎓**University of Sussex** Brighton · United Kingdom 15 works · 3,064 citations](https://md.research.elamur.ai/free/en/universities/university-of-sussex/) Topics[Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) ## Cite Cite as John Carroll, Guido Minnen and Ted Briscoe. 1999. Corpus Annotation for Parser Evaluation. Proceedings of the EACL99 workshop on Linguistically Interpreted Corpora (LINC), Bergen, Norway, June 12. The paper itself, not this card. BibTeX from the library's record of the work. [arXiv cs/9907013](https://arxiv.org/abs/cs/9907013) ```bibtex @inproceedings{carroll1999corpus, title = {{Corpus Annotation for Parser Evaluation}}, author = {John Carroll, Guido Minnen and Ted Briscoe}, booktitle = {Proceedings of the EACL99 workshop on Linguistically Interpreted Corpora (LINC), Bergen, Norway, June 12}, year = {1999}, eprint = {cs/9907013}, archivePrefix = {arXiv}, url = {https://arxiv.org/abs/cs/9907013} } ``` - [← previous Statistical Properties of Probabilistic Context-Free Grammars](https://md.research.elamur.ai/free/en/papers/chi-1999-statistical-properties-probabilistic-context-free/) - [next → The MATE meta-scheme for coreference in dialogues in multiple languages](https://md.research.elamur.ai/free/en/papers/poesio-1999-mate-meta-scheme-coreference-dialogues/) - [↑ back to papers of 1999](https://md.research.elamur.ai/free/en/papers/1999/) ## References 24 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Atwell, E. (1996) Comparative evaluation of grammatical annotation models. In R. Sutcliffe, H. Koch & A. McElligott (Eds.), Industrial Parsing of Software Manuals, 25–46. Amsterdam: Rodopi. [find](https://scholar.google.com/scholar?q=Atwell%2C+E%2E+%281996%29+Comparative+evaluation+of+grammatical+annotation+models%2E+In+R%2E+Sutcliffe%2C+H%2E+Koch+%26+A%2E+McElligott+%28Eds%2E%29%2C+Industrial+Parsing+of+Software+Manuals%2C+25%E2%80%9346%2E+Amsterdam%3A+Rodopi%2E) 2. Bies, A., Ferguson, M., Katz, K., MacIntyre, R., Tredinnick, V., Kim, G., Marcinkiewicz, M., Schasberger, B. (1995) Bracketing guidelines for Treebank II style Penn Treebank Project. Technical Report, CIS, University of Pennsylvania, Philadelphia, PA. [find](https://scholar.google.com/scholar?q=Bies%2C+A%2E%2C+Ferguson%2C+M%2E%2C+Katz%2C+K%2E%2C+MacIntyre%2C+R%2E%2C+Tredinnick%2C+V%2E%2C+Kim%2C+G%2E%2C+Marcinkiewicz%2C+M%2E%2C+Schasberger%2C+B%2E+%281995%29+Bracketing+guidelines+for+Treebank+II+style+Penn+Treebank+Project%2E+Technical+Report%2C) 3. Carpenter, B. & Manning, C. (1997) Probabilistic parsing using left corner language models. In Proceedings of the 5th ACL/SIGPARSE International Workshop on Parsing Technologies. MIT, Cambridge, MA. [find](https://scholar.google.com/scholar?q=Carpenter%2C+B%2E+%26+Manning%2C+C%2E+%281997%29+Probabilistic+parsing+using+left+corner+language+models%2E+In+Proceedings+of+the+5th+ACL%2FSIGPARSE+International+Workshop+on+Parsing+Technologies%2E+MIT%2C+Cambridge%2C+MA%2E) 4. Carroll, J., Briscoe E. & Sanfilippo, A. (1998) Parser evaluation: a survey and a new proposal. In Proceedings of the International Conference on Language Resources and Evaluation, 447–454. Granada, Spain. Available online at \. [find](https://scholar.google.com/scholar?q=Carroll%2C+J%2E%2C+Briscoe+E%2E+%26+Sanfilippo%2C+A%2E+%281998%29+Parser+evaluation%3A+a+survey+and+a+new+proposal%2E+In+Proceedings+of+the+International+Conference+on+Language+Resources+and+Evaluation%2C+447%E2%80%93454%2E+Granada%2C+S) 5. Carroll, J., Minnen, G. & Briscoe E. (1998) Can subcategorisation probabilities help a statistical parser?. In Proceedings of the 6th ACL/SIGDAT Workshop on Very Large Corpora. Montreal, Canada. Available online at \. [↗](https://arxiv.org/abs/cmp-lg/9806013) 6. Charniak, E. (1996) Tree-bank grammars. In Proceedings of the 13th National Conference on Artificial Intelligence, AAAI-96, 1031–1036. . [find](https://scholar.google.com/scholar?q=Charniak%2C+E%2E+%281996%29+Tree%2Dbank+grammars%2E+In+Proceedings+of+the+13th+National+Conference+on+Artificial+Intelligence%2C+AAAI%2D96%2C+1031%E2%80%931036%2E+%2E) 7. Collins, M. (1996) A new statistical parser based on bigram lexical dependencies. In Proceedings of the 34th Meeting of the Association for Computational Linguistics, 184–191. Santa Cruz, CA. [find](https://scholar.google.com/scholar?q=Collins%2C+M%2E+%281996%29+A+new+statistical+parser+based+on+bigram+lexical+dependencies%2E+In+Proceedings+of+the+34th+Meeting+of+the+Association+for+Computational+Linguistics%2C+184%E2%80%93191%2E+Santa+Cruz%2C+CA%2E) 8. Gaizauskas, R. (1998) Special issue on evaluation. Computer Speech & Language. [find](https://scholar.google.com/scholar?q=Gaizauskas%2C+R%2E+%281998%29+Special+issue+on+evaluation%2E+Computer+Speech+%26+Language%2E) 9. Gaizauskas, R., Hepple M. & Huyck, C. (1998) Modifying existing annotated corpora for general comparative evaluation of parsing. In Proceedings of the LRE Workshop on Evaluation of Parsing Systems. Granada, Spain. [find](https://scholar.google.com/scholar?q=Gaizauskas%2C+R%2E%2C+Hepple+M%2E+%26+Huyck%2C+C%2E+%281998%29+Modifying+existing+annotated+corpora+for+general+comparative+evaluation+of+parsing%2E+In+Proceedings+of+the+LRE+Workshop+on+Evaluation+of+Parsing+Systems%2E+Gr) 10. Grishman, R., Macleod, C. & Sterling, J. (1992) Evaluating parsing strategies using standardized parse files. In Proceedings of the 3rd ACL Conference on Applied Natural Language Processing, 156–161. Trento, Italy. [find](https://scholar.google.com/scholar?q=Grishman%2C+R%2E%2C+Macleod%2C+C%2E+%26+Sterling%2C+J%2E+%281992%29+Evaluating+parsing+strategies+using+standardized+parse+files%2E+In+Proceedings+of+the+3rd+ACL+Conference+on+Applied+Natural+Language+Processing%2C+156%E2%80%93161%2E+) 11. Harrison, P., Abney, S., Black, E., Flickinger, D., Gdaniec, C., Grishman, R., Hindle, D., Ingria, B., Marcus, M., Santorini, B. & Strzalkowski, T. (1991) Evaluating syntax performance of parser/grammars of English. In Proceedings of the Workshop on Evaluating Natural Language Processing Systems. ACL. [find](https://scholar.google.com/scholar?q=Harrison%2C+P%2E%2C+Abney%2C+S%2E%2C+Black%2C+E%2E%2C+Flickinger%2C+D%2E%2C+Gdaniec%2C+C%2E%2C+Grishman%2C+R%2E%2C+Hindle%2C+D%2E%2C+Ingria%2C+B%2E%2C+Marcus%2C+M%2E%2C+Santorini%2C+B%2E+%26+Strzalkowski%2C+T%2E+%281991%29+Evaluating+syntax+performance+of+parser%2Fgramm) 12. Leech, G. & Garside, R. (1991) Running a grammar factory: the production of syntactically analysed corpora or ‘treebanks’. In S. Johansson & A. Stenstrom (Eds.), English Computer Corpora: Selected Papers and Bibliography, Mouton de Gruyter, Berlin. [find](https://scholar.google.com/scholar?q=Leech%2C+G%2E+%26+Garside%2C+R%2E+%281991%29+Running+a+grammar+factory%3A+the+production+of+syntactically+analysed+corpora+or+%E2%80%98treebanks%E2%80%99%2E+In+S%2E+Johansson+%26+A%2E+Stenstrom+%28Eds%2E%29%2C+English+Computer+Corpora%3A+Selected+Pap) 13. Lehmann, S., Oepen, S., Regnier-Prost, S., Netter, K., Lux, V., Klein, J., Falkedal, K., Fouvry, F., Estival, D., Dauphin, E., Compagnion, H., Baur, J., Balkan, L. & Arnold, D. (1996) tsnlp — test suites for natural language processing. In Proceedings of the International Conference on Computational Linguistics, COLING-96, 711–716. Copenhagen, Denmark. [find](https://scholar.google.com/scholar?q=Lehmann%2C+S%2E%2C+Oepen%2C+S%2E%2C+Regnier%2DProst%2C+S%2E%2C+Netter%2C+K%2E%2C+Lux%2C+V%2E%2C+Klein%2C+J%2E%2C+Falkedal%2C+K%2E%2C+Fouvry%2C+F%2E%2C+Estival%2C+D%2E%2C+Dauphin%2C+E%2E%2C+Compagnion%2C+H%2E%2C+Baur%2C+J%2E%2C+Balkan%2C+L%2E+%26+Arnold%2C+D%2E+%281996%29+tsnlp+%E2%80%94+test+sui) 14. Lin, D. (1995) A dependency-based method for evaluating broad-coverage parsers. In Proceedings of the 14th International Joint Conference on Artificial Intelligence, 1420–1425. Montreal, Canada. [find](https://scholar.google.com/scholar?q=Lin%2C+D%2E+%281995%29+A+dependency%2Dbased+method+for+evaluating+broad%2Dcoverage+parsers%2E+In+Proceedings+of+the+14th+International+Joint+Conference+on+Artificial+Intelligence%2C+1420%E2%80%931425%2E+Montreal%2C+Canada%2E) 15. Lin, D. (1996) Dependency-based parser evaluation: a study with a software manual corpus. In R. Sutcliffe, H-D. Koch & A. McElligott (Eds.), Industrial Parsing of Software Manuals, 13–24. Amsterdam, The Netherlands: Rodopi. [find](https://scholar.google.com/scholar?q=Lin%2C+D%2E+%281996%29+Dependency%2Dbased+parser+evaluation%3A+a+study+with+a+software+manual+corpus%2E+In+R%2E+Sutcliffe%2C+H%2DD%2E+Koch+%26+A%2E+McElligott+%28Eds%2E%29%2C+Industrial+Parsing+of+Software+Manuals%2C+13%E2%80%9324%2E+Amsterdam%2C+T) 16. Marcus, M., Santorini, B. & Marcinkiewicz (1993) Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2), 313–330. [our card](https://md.research.elamur.ai/free/en/papers/marcus-1993-building-large-annotated-corpus-english/) 17. Roland, D. & Jurafsky, D. (1998) How verb subcategorization frequencies are affected by corpus choice. In Proceedings of the 17th International Conference on Computational Linguistics (COLING-ACL’98), 1122–1128. Montreal, Canada. [find](https://scholar.google.com/scholar?q=Roland%2C+D%2E+%26+Jurafsky%2C+D%2E+%281998%29+How+verb+subcategorization+frequencies+are+affected+by+corpus+choice%2E+In+Proceedings+of+the+17th+International+Conference+on+Computational+Linguistics+%28COLING%2DACL%E2%80%9998%29%2C) 18. Rubio, A. (Ed.) (1998) International Conference on Language Resources and Evaluation. Granada, Spain. [find](https://scholar.google.com/scholar?q=Rubio%2C+A%2E+%28Ed%2E%29+%281998%29+International+Conference+on+Language+Resources+and+Evaluation%2E+Granada%2C+Spain%2E) 19. Sampson, G. (1995) English for the Computer. Oxford, UK: Oxford University Press. [find](https://scholar.google.com/scholar?q=Sampson%2C+G%2E+%281995%29+English+for+the+Computer%2E+Oxford%2C+UK%3A+Oxford+University+Press%2E) 20. Sanfilippo, A., Barnett, R., Calzolari, N., Flores, S., Hellwig, P., Leech, P., Melero, M., Montemagni, S., Odijk, J., Pirrelli, V., Teufel, S., Villegas M. & Zaysser, L. (1996) Subcategorization Standards. Report of the EAGLES Lexicon/Syntax Interest Group. Available through eagles@ilc.pi.cnr.it. [find](https://scholar.google.com/scholar?q=Sanfilippo%2C+A%2E%2C+Barnett%2C+R%2E%2C+Calzolari%2C+N%2E%2C+Flores%2C+S%2E%2C+Hellwig%2C+P%2E%2C+Leech%2C+P%2E%2C+Melero%2C+M%2E%2C+Montemagni%2C+S%2E%2C+Odijk%2C+J%2E%2C+Pirrelli%2C+V%2E%2C+Teufel%2C+S%2E%2C+Villegas+M%2E+%26+Zaysser%2C+L%2E+%281996%29+Subcategorization+Stan) 21. Sekine, S. (1997) The domain dependence of parsing. In Proceedings of the 5th ACL Conference on Applied Natural Language Processing, 96–102. Washington, DC. [find](https://scholar.google.com/scholar?q=Sekine%2C+S%2E+%281997%29+The+domain+dependence+of+parsing%2E+In+Proceedings+of+the+5th+ACL+Conference+on+Applied+Natural+Language+Processing%2C+96%E2%80%93102%2E+Washington%2C+DC%2E) 22. Srinivas, B., Doran, C., Hockey B. & Joshi A. (1996) An approach to robust partial parsing and evaluation metrics. In Proceedings of the ESSLLI’96 Workshop on Robust Parsing. Prague, Czech Republic. [find](https://scholar.google.com/scholar?q=Srinivas%2C+B%2E%2C+Doran%2C+C%2E%2C+Hockey+B%2E+%26+Joshi+A%2E+%281996%29+An+approach+to+robust+partial+parsing+and+evaluation+metrics%2E+In+Proceedings+of+the+ESSLLI%E2%80%9996+Workshop+on+Robust+Parsing%2E+Prague%2C+Czech+Republic%2E) 23. Srinivas, B., Doran, C. & Kulick, S. (1995) Heuristics and parse ranking. In Proceedings of the 4th ACL/SIGPARSE International Workshop on Parsing Technologies. Prague, Czech Republic. [find](https://scholar.google.com/scholar?q=Srinivas%2C+B%2E%2C+Doran%2C+C%2E+%26+Kulick%2C+S%2E+%281995%29+Heuristics+and+parse+ranking%2E+In+Proceedings+of+the+4th+ACL%2FSIGPARSE+International+Workshop+on+Parsing+Technologies%2E+Prague%2C+Czech+Republic%2E) 24. van Rijsbergen, C. (1979) Information Retrieval. Butterworth, London. [find](https://scholar.google.com/scholar?q=van+Rijsbergen%2C+C%2E+%281979%29+Information+Retrieval%2E+Butterworth%2C+London%2E)