---
title: "Characterizing the Errors of Data-Driven — McDonald 2007"
description: "The paper compares the two top systems of the CoNLL-X shared task. MSTParser is a global, exhaustive graph-based model with maximum spanning tree decoding…"
authors: ["Ryan McDonald", "Joakim Nivre"]
year: 2007
venue: "Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)"
source: "https://aclanthology.org/D07-1013/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/missing4-2007-mcdonald-characterizing-errors-data-driven-dependency/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/missing4-2007-mcdonald-characterizing-errors-data-driven-dependency/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2007](https://md.research.elamur.ai/free/en/papers/2007/) / Characterizing the Errors of Data-Driven Dependency Parsing Models

# Characterizing the Errors of Data-Driven Dependency Parsing Models

**Authors:** [Ryan McDonald](https://md.research.elamur.ai/free/en/authors/ryan-mcdonald/) and [Joakim Nivre](https://md.research.elamur.ai/free/en/authors/joakim-nivre/) · **Year:** 2007 · **Venue:** Proceedings of the 2007 Joint Conference on EMNLP and CoNLL (EMNLP-CoNLL) **Link:** [aclanthology.org/D07-1013](https://aclanthology.org/D07-1013/) · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) — summary only

[**262** citations](https://openalex.org/works?filter=cites:W1491975949) OpenAlex · 2026-10-14 · [**141** citations in our library](https://md.research.elamur.ai/free/en/papers/missing4-2007-mcdonald-characterizing-errors-data-driven-dependency/cited-by/) of the 140673 papers whose bibliographies we read · [Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)](https://aclanthology.org/volumes/D07-1/) · [licence](https://creativecommons.org/licenses/by-nc-sa/3.0/)

- [Publisher's page](https://aclanthology.org/D07-1013/)
- [PDF](https://aclanthology.org/D07-1013.pdf)
- [OpenAlex](https://openalex.org/W1491975949)

**Authors (2)**

1. [Ryan McDonald](https://openalex.org/A5022489655) [Semantic Scholar](https://www.semanticscholar.org/author/143957226) · h 48 · 16067 citations · 113 works
2. [Joakim Nivre](https://openalex.org/A5077063699) [Semantic Scholar](https://www.semanticscholar.org/author/1720988) · h 60 · 16139 citations · 333 works

Citations and author metrics: OpenAlex, 2026-10-14; authors with the same name are not always told apart.

## What it is about

The paper compares the two top systems of the CoNLL-X shared task. MSTParser is a global, exhaustive graph-based model with maximum spanning tree decoding and arc-pair features; MaltParser is a local, greedy transition-based model with an SVM classifier. Both reach almost the same labeled accuracy over 13 languages (about 80.8 and 80.8 LAS when the output files are pooled), but they make different kinds of errors. The authors break the errors down by arc length, distance to the root, number of siblings, degree of non-projectivity, part of speech and dependency type. MSTParser is more precise for arcs far from the root and for long arcs; MaltParser is better on short arcs and on nouns and pronouns. A simple oracle over the two outputs reaches 84.5 to 86.9% against about 81% for each system.

## Ideas

- I1. Error profile by distance to the root in `en parse-eval` and in `gparse` (Eisner decoding, `gparse/src/eisner.rs`): precision and recall of arcs at distance 1, 2, 3–4 and 5 or more, on EWT test; the question is whether the arc-standard parser fragments the tree near the root as the paper suggests for greedy parsers.
- I2. Root-modifier check: count predicted root arcs that are not verbs or auxiliaries in `en explain` output over EWT test; the paper reads low root-modifier precision with stable recall as a sign of fragmented parses.
- I3. Oracle diagnostic (test only, not shipped): take the `en` and `gparse` outputs on EWT test and measure the oracle LAS and the share of sentences where one of them is fully correct; if the gap is large, a combiner through `tools/parsecmp` is worth a later decision.
- I4. Non-projectivity split: report LAS separately for sentences with a gold arc of degree 1 or more; the arc-standard parser in `en/src/parse.rs` cannot build those arcs, so the number is its ceiling on EWT and GUM test (GUM is test only).

## For Mova

The relevant chapters are [languages/en/20-syntax-overview.md](https://mova.elamur.ai/free/en/docs/languages/en/20-syntax-overview/) (the main clause and root) and `21-clause-structure.md`. The code is `en/src/parse.rs` (arc-standard, beam 8), `gparse/src/eisner.rs`, and the `en parse-eval` and `tools/parsecmp` tools. Measure first: EWT test LAS by distance to the root for both parsers, against the current baseline LAS 80.24. The paper's lesson is that an error analysis by structure says more than one accuracy number.

Topics: [Dependency parsing](https://md.research.elamur.ai/free/en/topics/dependency-parsing/)

## Cite

The paper itself, not this card. BibTeX from the library's record of the work.

```bibtex
@inproceedings{mcdonald2007characterizing,
  title = {{Characterizing the Errors of Data-Driven Dependency Parsing Models}},
  author = {McDonald, Ryan and Nivre, Joakim},
  booktitle = {Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)},
  year = {2007},
  url = {https://aclanthology.org/D07-1013/}
}
```

- [← previous CCGbank: A Corpus of CCG Derivations and Dependency Structures Extracted from the Penn Treebank](https://md.research.elamur.ai/free/en/papers/bank-2007-hockenmaier-ccgbank-corpus-ccg-derivations-dependency/)
- [next → Chinese Syntactic Reordering for Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/wang-2007-chinese-syntactic-reordering-statistical-machine/)
- [↑ back to papers of 2007](https://md.research.elamur.ai/free/en/papers/2007/)

## References

17 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. A. Böhmová, J. Hajič, E. Hajičová, and B. Hladká. 2003. The PDT: A 3-level annotation scenario. In A. Abeillé, editor, Treebanks: Building and Using Parsed Corpora, chapter 7. Kluwer Academic Publishers. [find](https://scholar.google.com/scholar?q=A%2E+Bo%CC%88hmova%CC%81%2C+J%2E+Hajic%CC%8C%2C+E%2E+Hajic%CC%8Cova%CC%81%2C+and+B%2E+Hladka%CC%81%2E+2003%2E+The+PDT%3A+A+3%2Dlevel+annotation+scenario%2E+In+A%2E+Abeille%CC%81%2C+editor%2C+Treebanks%3A+Building+and+Using+Parsed+Corpora%2C+chapter+7%2E+Kluwer+Academic+P)
2. S. Buchholz and E. Marsi. 2006. CoNLL-X shared task on multilingual dependency parsing. In Proc. CoNLL. [find](https://scholar.google.com/scholar?q=S%2E+Buchholz+and+E%2E+Marsi%2E+2006%2E+CoNLL%2DX+shared+task+on+multilingual+dependency+parsing%2E+In+Proc%2E+CoNLL%2E) [our card](https://md.research.elamur.ai/free/en/papers/parsers-2006-buchholz-conll-x-shared-task-multilingual/)
3. Hal Daumé III, John Langford, and Daniel Marcu. 2006. Search-based structured prediction. In Submission. [find](https://scholar.google.com/scholar?q=Hal+Daume%CC%81+III%2C+John+Langford%2C+and+Daniel+Marcu%2E+2006%2E+Search%2Dbased+structured+prediction%2E+In+Submission%2E)
4. Y. Ding and M. Palmer. 2004. Synchronous dependency insertion grammars: A grammar formalism for syntax based statistical MT. In Workshop on Recent Advances in Dependency Grammars (COLING). [find](https://scholar.google.com/scholar?q=Y%2E+Ding+and+M%2E+Palmer%2E+2004%2E+Synchronous+dependency+insertion+grammars%3A+A+grammar+formalism+for+syntax+based+statistical+MT%2E+In+Workshop+on+Recent+Advances+in+Dependency+Grammars+%28COLING%29%2E)
5. R. Hudson. 1984. Word Grammar. Blackwell. [find](https://scholar.google.com/scholar?q=R%2E+Hudson%2E+1984%2E+Word+Grammar%2E+Blackwell%2E)
6. H. Maruyama. 1990. Structural disambiguation with constraint propagation. In Proc. ACL. [find](https://scholar.google.com/scholar?q=H%2E+Maruyama%2E+1990%2E+Structural+disambiguation+with+constraint+propagation%2E+In+Proc%2E+ACL%2E)
7. R. McDonald, K. Crammer, and F. Pereira. 2005a. Online large-margin training of dependency parsers. In Proc. ACL. [find](https://scholar.google.com/scholar?q=R%2E+McDonald%2C+K%2E+Crammer%2C+and+F%2E+Pereira%2E+2005a%2E+Online+large%2Dmargin+training+of+dependency+parsers%2E+In+Proc%2E+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/narrative-2005-mcdonald-online-large-margin-training-dependency/)
8. R. McDonald, F. Pereira, K. Ribarov, and J. Hajič. 2005b. Non-projective dependency parsing using spanning tree algorithms. In Proc. HLT/EMNLP. [find](https://scholar.google.com/scholar?q=R%2E+McDonald%2C+F%2E+Pereira%2C+K%2E+Ribarov%2C+and+J%2E+Hajic%CC%8C%2E+2005b%2E+Non%2Dprojective+dependency+parsing+using+spanning+tree+algorithms%2E+In+Proc%2E+HLT%2FEMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/narrative-2005-mcdonald-non-projective-dependency-parsing-spanning/)
9. R. McDonald, K. Lerman, and F. Pereira. 2006. Multilingual dependency analysis with a two-stage discriminative parser. In Proc. CoNLL. [find](https://scholar.google.com/scholar?q=R%2E+McDonald%2C+K%2E+Lerman%2C+and+F%2E+Pereira%2E+2006%2E+Multilingual+dependency+analysis+with+a+two%2Dstage+discriminative+parser%2E+In+Proc%2E+CoNLL%2E) [our card](https://md.research.elamur.ai/free/en/papers/mcdonald-2006-multilingual-dependency-analysis-two-stage/)
10. I.A. Mel’čuk. 1988. Dependency Syntax: Theory and Practice. State University of New York Press. [find](https://scholar.google.com/scholar?q=I%2EA%2E+Mel%E2%80%99c%CC%8Cuk%2E+1988%2E+Dependency+Syntax%3A+Theory+and+Practice%2E+State+University+of+New+York+Press%2E)
11. J. Nivre and J. Nilsson. 2005. Pseudo-projective dependency parsing. In Proc. ACL. [find](https://scholar.google.com/scholar?q=J%2E+Nivre+and+J%2E+Nilsson%2E+2005%2E+Pseudo%2Dprojective+dependency+parsing%2E+In+Proc%2E+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/nivre-2005-pseudo-projective-dependency-parsing/)
12. J. Nivre, J. Hall, J. Nilsson, G. Eryigit, and S. Marinov. 2006. Labeled pseudo-projective dependency parsing with support vector machines. In Proc. CoNLL. [find](https://scholar.google.com/scholar?q=J%2E+Nivre%2C+J%2E+Hall%2C+J%2E+Nilsson%2C+G%2E+Eryigit%2C+and+S%2E+Marinov%2E+2006%2E+Labeled+pseudo%2Dprojective+dependency+parsing+with+support+vector+machines%2E+In+Proc%2E+CoNLL%2E) [our card](https://md.research.elamur.ai/free/en/papers/nivre-2006-labeled-pseudo-projective-dependency-parsing/)
13. J. Nivre. 2003. An efficient algorithm for projective dependency parsing. In Proc. IWPT. [find](https://scholar.google.com/scholar?q=J%2E+Nivre%2E+2003%2E+An+efficient+algorithm+for+projective+dependency+parsing%2E+In+Proc%2E+IWPT%2E) [our card](https://md.research.elamur.ai/free/en/papers/parsers-2003-nivre-efficient-algorithm-projective-dependency-parsing/)
14. J. Nivre. 2006. Constraints on non-projective dependency parsing. In Proc. EACL. [find](https://scholar.google.com/scholar?q=J%2E+Nivre%2E+2006%2E+Constraints+on+non%2Dprojective+dependency+parsing%2E+In+Proc%2E+EACL%2E)
15. K. Sagae and A. Lavie. 2006. Parser combination by reparsing. In Proc. HLT/NAACL. [find](https://scholar.google.com/scholar?q=K%2E+Sagae+and+A%2E+Lavie%2E+2006%2E+Parser+combination+by+reparsing%2E+In+Proc%2E+HLT%2FNAACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/sagae-2006-parser-combination-reparsing/)
16. P. Sgall, E. Hajičová, and J. Panevová. 1986. The Meaning of the Sentence in Its Pragmatic Aspects. Reidel. [find](https://scholar.google.com/scholar?q=P%2E+Sgall%2C+E%2E+Hajic%CC%8Cova%CC%81%2C+and+J%2E+Panevova%CC%81%2E+1986%2E+The+Meaning+of+the+Sentence+in+Its+Pragmatic+Aspects%2E+Reidel%2E)
17. R. Snow, D. Jurafsky, and A. Y. Ng. 2004. Learning syntactic patterns for automatic hypernym discovery. In Proc. NIPS. [find](https://scholar.google.com/scholar?q=R%2E+Snow%2C+D%2E+Jurafsky%2C+and+A%2E+Y%2E+Ng%2E+2004%2E+Learning+syntactic+patterns+for+automatic+hypernym+discovery%2E+In+Proc%2E+NIPS%2E)