--- title: "Formalism-Independent Parser Evaluation with CCG and DepBank — Clark 2007" description: "The paper's message is to evaluate on a resource that was not used to develop the parser, and to report the… 22 references, cited by 24 papers here." authors: ["Stephen Clark", "James Curran"] year: 2007 venue: "Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics" source: "https://aclanthology.org/P07-1032/" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/clark-2007-formalism-independent-parser-evaluation-ccg/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/clark-2007-formalism-independent-parser-evaluation-ccg/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2007](https://md.research.elamur.ai/free/en/papers/2007/) / Formalism-Independent Parser Evaluation with CCG and DepBank # Formalism-Independent Parser Evaluation with CCG and DepBank **Authors:** [Stephen Clark](https://md.research.elamur.ai/free/en/authors/stephen-clark/) and [James R. Curran](https://md.research.elamur.ai/free/en/authors/james-r-curran/) · **Year:** 2007 · **Venue:** Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics **Link:** [aclanthology.org/P07-1032](https://aclanthology.org/P07-1032/) · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) — summary only [**55** citations](https://openalex.org/works?filter=cites:W2139117344) OpenAlex, 2026-10-14 · [**24** citations in our library](https://md.research.elamur.ai/free/en/papers/clark-2007-formalism-independent-parser-evaluation-ccg/cited-by/) of the 140,884 papers whose bibliographies we read · [Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics](https://aclanthology.org/volumes/P07-1/) · [licence](https://creativecommons.org/licenses/by-nc-sa/3.0/) - [Publisher's page](https://aclanthology.org/P07-1032/) - [PDF](https://aclanthology.org/P07-1032.pdf) - [OpenAlex](https://openalex.org/W2139117344) **Authors (2)** 1. [Stephen Clark](https://openalex.org/A5057392864) [ORCID](https://orcid.org/0000-0002-2193-4705) · [Semantic Scholar](https://www.semanticscholar.org/author/144523372) · h 64 · 17,686 citations · 441 works 2. James Curran [ORCID](https://orcid.org/0000-0001-9637-0484) · [Semantic Scholar](https://www.semanticscholar.org/author/1733593) Citations and author metrics: OpenAlex, 2026-10-14; authors with the same name are not always told apart. ## What it is about Parsers that use different formalisms produce different output, so their scores cannot be compared directly. The authors evaluate a CCG parser on DepBank, a set of grammatical relations that does not match the parser's own output. The main difficulty is the conversion from CCG output to DepBank relations, so they measure the conversion itself: parse the gold analyses, convert them, and compare. This gives an upper bound on what the parser can score. The CCG parser reaches an F-score of 81.9% on labelled dependencies against an upper bound of 84.8%, and beats the RASP parser by over 5% overall. The paper's message is to evaluate on a resource that was not used to develop the parser, and to report the conversion ceiling next to the score. ## For Mova - I1. `en/src/convert.rs`: measure Mova's conversion ceiling by converting gold EWT trees to the `mova` dialect and back to UD and scoring against the original; the gap from 100 is conversion loss, and should be reported next to `en ud-eval` (two runs, byte-identical). - I2. `tools/parsecmp`: report each witness score as a fraction of its own conversion ceiling, so that a converter loss is not counted as a parser error. - I3. Independent resource: score `en` against ERG-derived dependencies from the ACE setup listed in [docs/parsers.md](https://mova.elamur.ai/free/en/docs/docs/parsers/) on 200 held-out sentences (test only), and compare the agreement with the EWT LAS. Touches [languages/en/20-syntax-overview.md](https://mova.elamur.ai/free/en/docs/languages/en/20-syntax-overview/). The conversion ceiling is a cheap check that the `mova` dialect does not lose information, which the converters must keep. Measure I1 first; it is the quickest of the three and needs no new data. ## Institutions [🎓**University of Oxford** Oxford · United Kingdom 39 works · 153,213 citations](https://md.research.elamur.ai/free/en/universities/university-of-oxford/) · [🎓**The University of Sydney** Sydney · Australia 29 works · 67,547 citations](https://md.research.elamur.ai/free/en/universities/the-university-of-sydney/) Topics[Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · [Syntax & grammar formalisms](https://md.research.elamur.ai/free/en/topics/syntax-grammar/) ## Cite Cite as Stephen Clark and James Curran. 2007. Formalism-Independent Parser Evaluation with CCG and DepBank. Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics. The paper itself, not this card. BibTeX from the library's record of the work. ```bibtex @inproceedings{clark2007formalism, title = {{Formalism-Independent Parser Evaluation with CCG and DepBank}}, author = {Stephen Clark and James Curran}, booktitle = {Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics}, year = {2007}, url = {https://aclanthology.org/P07-1032/} } ``` - [← previous A Unified Local and Global Model for Discourse Coherence](https://md.research.elamur.ai/free/en/papers/elsner-2007-unified-local-global-model-discourse/) - [next → LinES: An English-Swedish Parallel Treebank](https://md.research.elamur.ai/free/en/papers/ahrenberg-2007-lines-english-swedish-parallel-treebank/) - [↑ back to papers of 2007](https://md.research.elamur.ai/free/en/papers/2007/) ## References 22 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Rens Bod. 2003. An efficient implementation of a new DOP model. In Proceedings of the 10th Meeting of the EACL, pages 19–26, Budapest, Hungary. [find](https://scholar.google.com/scholar?q=Rens+Bod%2E+2003%2E+An+efficient+implementation+of+a+new+DOP+model%2E+In+Proceedings+of+the+10th+Meeting+of+the+EACL%2C+pages+19%E2%80%9326%2C+Budapest%2C+Hungary%2E) 2. Ted Briscoe and John Carroll. 2006. Evaluating the accuracy of an unlexicalized statistical parser on the PARC DepBank. In Proceedings of the Poster Session of COLING/ACL-06, Sydney, Australia. [find](https://scholar.google.com/scholar?q=Ted+Briscoe+and+John+Carroll%2E+2006%2E+Evaluating+the+accuracy+of+an+unlexicalized+statistical+parser+on+the+PARC+DepBank%2E+In+Proceedings+of+the+Poster+Session+of+COLING%2FACL%2D06%2C+Sydney%2C+Australia%2E) 3. Ted Briscoe, John Carroll, and Rebecca Watson. 2006. The second release of the RASP system. In Proceedings of the Interactive Demo Session of COLING/ACL-06, Sydney, Australia. [our card](https://md.research.elamur.ai/free/en/papers/briscoe-2006-second-release-rasp-system/) 4. Sabine Buchholz, Jorn Veenstra, and Walter Daelemans. 1999. Cascaded grammatical relation assignment. In Proceedings of EMNLP/VLC-99, pages 239–246, University of Maryland, June 21-22. [find](https://scholar.google.com/scholar?q=Sabine+Buchholz%2C+Jorn+Veenstra%2C+and+Walter+Daelemans%2E+1999%2E+Cascaded+grammatical+relation+assignment%2E+In+Proceedings+of+EMNLP%2FVLC%2D99%2C+pages+239%E2%80%93246%2C+University+of+Maryland%2C+June+21%2D22%2E) 5. A. Cahill, M. Burke, R. O’Donovan, J. van Genabith, and A. Way. 2004. Long-distance dependency resolution in automatically acquired wide-coverage PCFG-based LFG approximations. In Proceedings of the 42nd Meeting of the ACL, pages 320–327, Barcelona, Spain. [find](https://scholar.google.com/scholar?q=A%2E+Cahill%2C+M%2E+Burke%2C+R%2E+O%E2%80%99Donovan%2C+J%2E+van+Genabith%2C+and+A%2E+Way%2E+2004%2E+Long%2Ddistance+dependency+resolution+in+automatically+acquired+wide%2Dcoverage+PCFG%2Dbased+LFG+approximations%2E+In+Proceedings+of+the+4) 6. John Carroll, Ted Briscoe, and Antonio Sanfilippo. 1998. Parser evaluation: a survey and a new proposal. In Proceedings of the 1st LREC Conference, pages 447–454, Granada, Spain. [find](https://scholar.google.com/scholar?q=John+Carroll%2C+Ted+Briscoe%2C+and+Antonio+Sanfilippo%2E+1998%2E+Parser+evaluation%3A+a+survey+and+a+new+proposal%2E+In+Proceedings+of+the+1st+LREC+Conference%2C+pages+447%E2%80%93454%2C+Granada%2C+Spain%2E) 7. Eugene Charniak and Mark Johnson. 2005. Coarse-to-fine nbest parsing and maxent discriminative reranking. In Proceedings of the 43rd Annual Meeting of the ACL, University of Michigan, Ann Arbor. [find](https://scholar.google.com/scholar?q=Eugene+Charniak+and+Mark+Johnson%2E+2005%2E+Coarse%2Dto%2Dfine+nbest+parsing+and+maxent+discriminative+reranking%2E+In+Proceedings+of+the+43rd+Annual+Meeting+of+the+ACL%2C+University+of+Michigan%2C+Ann+Arbor%2E) 8. Eugene Charniak. 2000. A maximum-entropy-inspired parser. In Proceedings of the 1st Meeting of the NAACL, pages 132– 139, Seattle, WA. [our card](https://md.research.elamur.ai/free/en/papers/charniak-2000-maximum-entropy-inspired-parser/) 9. Stephen Clark and James R. Curran. 2004a. The importance of supertagging for wide-coverage CCG parsing. In Proceedings of COLING-04, pages 282–288, Geneva, Switzerland. [our card](https://md.research.elamur.ai/free/en/papers/clark-2004-importance-supertagging-wide-coverage-ccg/) 10. Stephen Clark and James R. Curran. 2004b. Parsing the WSJ using CCG and log-linear models. In Proceedings of the 42nd Meeting of the ACL, pages 104–111, Barcelona, Spain. [our card](https://md.research.elamur.ai/free/en/papers/clark-2004-parsing-wsj-ccg-log-linear/) 11. Michael Collins. 2003. Head-driven statistical models for natural language parsing. Computational Linguistics, 29(4):589–637. [our card](https://md.research.elamur.ai/free/en/papers/collins-2003-head-driven-statistical-models-natural/) 12. James R. Curran and Stephen Clark. 2003. Investigating GIS and smoothing for maximum entropy taggers. In Proceedings of the 10th Meeting of the EACL, pages 91–98, Budapest, Hungary. [find](https://scholar.google.com/scholar?q=James+R%2E+Curran+and+Stephen+Clark%2E+2003%2E+Investigating+GIS+and+smoothing+for+maximum+entropy+taggers%2E+In+Proceedings+of+the+10th+Meeting+of+the+EACL%2C+pages+91%E2%80%9398%2C+Budapest%2C+Hungary%2E) 13. Julia Hockenmaier and Mark Steedman. 2002. Generative models for statistical parsing with Combinatory Categorial Grammar. In Proceedings of the 40th Meeting of the ACL, pages 335–342, Philadelphia, PA. [find](https://scholar.google.com/scholar?q=Julia+Hockenmaier+and+Mark+Steedman%2E+2002%2E+Generative+models+for+statistical+parsing+with+Combinatory+Categorial+Grammar%2E+In+Proceedings+of+the+40th+Meeting+of+the+ACL%2C+pages+335%E2%80%93342%2C+Philadelphia%2C+PA) 14. Julia Hockenmaier. 2003. Data and Models for Statistical Parsing with Combinatory Categorial Grammar. Ph.D. thesis, University of Edinburgh. [find](https://scholar.google.com/scholar?q=Julia+Hockenmaier%2E+2003%2E+Data+and+Models+for+Statistical+Parsing+with+Combinatory+Categorial+Grammar%2E+Ph%2ED%2E+thesis%2C+University+of+Edinburgh%2E) 15. Ron Kaplan, Stefan Riezler, Tracy H. King, John T. Maxwell III, Alexander Vasserman, and Richard Crouch. 2004. Speed and accuracy in shallow and deep stochastic parsing. In Proceedings of the HLT Conference and the 4th NAACL Meeting (HLT-NAACL’04), Boston, MA. [find](https://scholar.google.com/scholar?q=Ron+Kaplan%2C+Stefan+Riezler%2C+Tracy+H%2E+King%2C+John+T%2E+Maxwell+III%2C+Alexander+Vasserman%2C+and+Richard+Crouch%2E+2004%2E+Speed+and+accuracy+in+shallow+and+deep+stochastic+parsing%2E+In+Proceedings+of+the+HLT+Conf) 16. Tracy H. King, Richard Crouch, Stefan Riezler, Mary Dalrymple, and Ronald M. Kaplan. 2003. The PARC 700 Dependency Bank. In Proceedings of the LINC-03 Workshop, Budapest, Hungary. [find](https://scholar.google.com/scholar?q=Tracy+H%2E+King%2C+Richard+Crouch%2C+Stefan+Riezler%2C+Mary+Dalrymple%2C+and+Ronald+M%2E+Kaplan%2E+2003%2E+The+PARC+700+Dependency+Bank%2E+In+Proceedings+of+the+LINC%2D03+Workshop%2C+Budapest%2C+Hungary%2E) 17. Robert Malouf and Gertjan van Noord. 2004. Wide coverage parsing with stochastic attribute value grammars. In Proceedings of the IJCNLP-04 Workshop: Beyond shallow analyses - Formalisms and statistical modeling for deep analyses, Hainan Island, China. [find](https://scholar.google.com/scholar?q=Robert+Malouf+and+Gertjan+van+Noord%2E+2004%2E+Wide+coverage+parsing+with+stochastic+attribute+value+grammars%2E+In+Proceedings+of+the+IJCNLP%2D04+Workshop%3A+Beyond+shallow+analyses+%2D+Formalisms+and+statistica) 18. Joakim Nivre and Mario Scholz. 2004. Deterministic dependency parsing of English text. In Proceedings of COLING-2004, pages 64–70, Geneva, Switzerland. [our card](https://md.research.elamur.ai/free/en/papers/nivre-2004-deterministic-dependency-parsing-english-text/) 19. Judita Preiss. 2003. Using grammatical relations to compare parsers. In Proceedings of the 10th Meeting of the EACL, pages 291–298, Budapest, Hungary. [find](https://scholar.google.com/scholar?q=Judita+Preiss%2E+2003%2E+Using+grammatical+relations+to+compare+parsers%2E+In+Proceedings+of+the+10th+Meeting+of+the+EACL%2C+pages+291%E2%80%93298%2C+Budapest%2C+Hungary%2E) 20. Anoop Sarkar and Aravind Joshi. 2003. Tree-adjoining grammars and its application to statistical parsing. In Rens Bod, Remko Scha, and Khalil Sima’an, editors, Data-oriented parsing. CSLI. [find](https://scholar.google.com/scholar?q=Anoop+Sarkar+and+Aravind+Joshi%2E+2003%2E+Tree%2Dadjoining+grammars+and+its+application+to+statistical+parsing%2E+In+Rens+Bod%2C+Remko+Scha%2C+and+Khalil+Sima%E2%80%99an%2C+editors%2C+Data%2Doriented+parsing%2E+CSLI%2E) 21. Mark Steedman. 2000. The Syntactic Process. The MIT Press, Cambridge, MA. [find](https://scholar.google.com/scholar?q=Mark+Steedman%2E+2000%2E+The+Syntactic+Process%2E+The+MIT+Press%2C+Cambridge%2C+MA%2E) 22. Kristina Toutanova, Christopher Manning, Stuart Shieber, Dan Flickinger, and Stephan Oepen. 2002. Parse disambiguation for a rich HPSG grammar. In Proceedings of the First Workshop on Treebanks and Linguistic Theories, pages 253–263, Sozopol, Bulgaria. [find](https://scholar.google.com/scholar?q=Kristina+Toutanova%2C+Christopher+Manning%2C+Stuart+Shieber%2C+Dan+Flickinger%2C+and+Stephan+Oepen%2E+2002%2E+Parse+disambiguation+for+a+rich+HPSG+grammar%2E+In+Proceedings+of+the+First+Workshop+on+Treebanks+and+Li)