---
title: "Parsing the WSJ Using CCG and Log-Linear Models — Clark 2004"
description: "Combinatory Categorial Grammar (CCG) allows many different derivations to produce the same final structure, a phenomenon called spurious ambiguity."
authors: ["Stephen Clark", "James R. Curran"]
year: 2004
venue: "Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04)"
source: "https://aclanthology.org/P04-1014/"
doi: "10.3115/1218955.1218969"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/clark-2004-parsing-wsj-ccg-log-linear/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/clark-2004-parsing-wsj-ccg-log-linear/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2004](https://md.research.elamur.ai/free/en/papers/2004/) / Parsing the WSJ Using CCG and Log-Linear Models

# Parsing the WSJ Using CCG and Log-Linear Models

**Authors:** [Stephen Clark](https://md.research.elamur.ai/free/en/authors/stephen-clark/), [James R. Curran](https://md.research.elamur.ai/free/en/authors/james-r-curran/) · **Year:** 2004 · **Venue:** Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/P04-1014/) · [PDF](https://aclanthology.org/P04-1014.pdf) · [DOI 10.3115/1218955.1218969](https://doi.org/10.3115/1218955.1218969) · [OpenAlex](https://openalex.org/W2149660837)

**Citations:** 278 citations (openalex-snapshot · 2026-10-14) · [110](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Combinatory Categorial Grammar (CCG) allows many different derivations to produce the same final structure, a phenomenon called spurious ambiguity. This makes statistical modelling and parsing harder. Clark and Curran build discriminative log-linear models for wide-coverage CCG parsing of the WSJ Penn Treebank, trained on CCGbank. They compare two designs:
- a **dependency model**, which sums probability over every derivation that yields a given dependency structure;
- a **normal-form model**, which scores only canonical derivations.

Both are trained by maximising conditional log-likelihood with a Gaussian prior. Expectations are computed with an inside-outside procedure over packed charts, treated as feature forests. The authors found Generalised Iterative Scaling (GIS) unsuitable at this scale and replaced it with a parallel L-BFGS running on a Beowulf cluster, so the entire treebank could be used for training. They also propose a parsing algorithm that maximises the expected recall of dependencies.

The two models reach similar accuracy. The normal-form model is cheaper to train and much faster at parsing. It clearly beats Clark et al. (2002) and matches or exceeds the state of the art for CCG parsing.

## Key points
- **Test-set results** (gold POS tags), for the best normal-form log-linear model:
  - labelled P/R: 86.6/86.3
  - unlabelled P/R: 92.5/92.1
  - category accuracy: 93.6
- **Comparison systems** (labelled P/R): Hockenmaier 2003 reached 84.3/84.6, and Clark et al. 2002 reached 81.9/81.8.
- **Automatic POS tags:** with tags from the C&C POS tagger, the log-linear model drops to 84.8/84.5 labelled P/R, because the supertagger relies heavily on POS features.
- **Parsing speed:** the 2,401 sentences of section 23 take 1.6 minutes with the normal-form model and 10.5 minutes with the dependency model. The gap comes mainly from the normal-form constraints on rule application.
- **Dependency representation:** each dependency is a 5-tuple of head word, lexical category, argument slot, argument head and a long-range marker.
- **Larger training charts:** the best model was trained on charts with about 2.9 supertagger categories per word, estimated on 15 machines. This gave an accuracy gain.
- **Oracle experiment:** with gold CCGbank lexical categories on section 00, 7% of sentences could not be parsed. On the rest, labelled F-score was almost 98%.

## Interesting ideas
- **Training against a structure, not a derivation.** When many hidden derivations lead to the same gold output, the "empirical" side of the gradient can be an expectation over all derivations consistent with the gold structure. This applies to any task with latent structure and only partial supervision.
- **Packed charts as feature forests.** Grouping equivalent chart entries (same category, head and unfilled dependencies) lets estimation and decoding sum over exponentially many analyses without listing them. The features chosen decide which entries may be merged, a design trade-off worth keeping in mind for any chart-based model.
- **Decode for the evaluation metric.** Choosing the output that maximises expected dependency recall, rather than the single most probable derivation, aligns decoding with how the parser is scored. This idea transfers to other structured prediction tasks.
- **Parallelise the optimiser, not just the data.** Running L-BFGS across a cluster, and dropping GIS when it proved unsuitable, made training on the full corpus feasible. This is a reminder that the choice of optimiser can decide what data scale is reachable.
- **Oracle experiments to locate bottlenecks.** Feeding the parser gold lexical categories shows how much accuracy is lost upstream in supertagging. That points future effort at the right component, and the same check suits any pipeline system.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

110 papers of our library cite it (of the 140884 whose bibliographies we read); 11 of them have a page here, 99 not yet.

- [The Importance of Supertagging for Wide-Coverage CCG Parsing](https://md.research.elamur.ai/free/en/papers/aclbulk-2004-clark-importance-supertagging-wide-coverage-ccg/)
- [Probabilistic Disambiguation Models for Wide-Coverage HPSG Parsing](https://md.research.elamur.ai/free/en/papers/aclbulk-2005-miyao-probabilistic-disambiguation-models-wide-coverage/)
- [Formalism-Independent Parser Evaluation with CCG and DepBank](https://md.research.elamur.ai/free/en/papers/aclbulk-2007-clark-formalism-independent-parser-evaluation-ccg/)
- [Task-oriented Evaluation of Syntactic Parsers and Their Representations](https://md.research.elamur.ai/free/en/papers/aclbulk-2008-miyao-task-oriented-evaluation-syntactic-parsers/)
- [CCGbank: A Corpus of CCG Derivations and Dependency Structures Extracted from the Penn Treebank](https://md.research.elamur.ai/free/en/papers/bank-2007-hockenmaier-ccgbank-corpus-ccg-derivations-dependency/)
- [Towards Generating Text from Discourse Representation Structures](https://md.research.elamur.ai/free/en/papers/basile-2011-generating-text-discourse-representation-structures/)
- [Feature Forest Models for Probabilistic HPSG Parsing](https://md.research.elamur.ai/free/en/papers/cl-2008-miyao-feature-forest-models-probabilistic-hpsg/)
- [Probabilistic CFG with Latent Annotations](https://md.research.elamur.ai/free/en/papers/matsuzaki-2005-probabilistic-cfg-latent-annotations/)
- [Multi-Source Transfer of Delexicalized Dependency Parsers](https://md.research.elamur.ai/free/en/papers/mcdonald-2011-multi-source-transfer-delexicalized-dependency/)
- [Online Large-Margin Training of Dependency Parsers](https://md.research.elamur.ai/free/en/papers/narrative-2005-mcdonald-online-large-margin-training-dependency/)
- [Wide-Coverage Efficient Statistical Parsing with CCG and Log-Linear Models](https://md.research.elamur.ai/free/en/papers/nlpgap-2007-clark-wide-coverage-efficient-statistical-parsing/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{clark2004parsing,
  title = {{Parsing the WSJ Using CCG and Log-Linear Models}},
  author = {Clark, Stephen and Curran, James R.},
  booktitle = {Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04)},
  year = {2004},
  doi = {10.3115/1218955.1218969},
  url = {https://doi.org/10.3115/1218955.1218969}
}
```

- [← previous ORANGE: a Method for Evaluating Automatic Evaluation Metrics for Machine Translation](https://md.research.elamur.ai/free/en/papers/lin-2004-orange-method-evaluating-automatic-evaluation/)
- [next → Parsing Ungrammatical Input: An Evaluation Procedure](https://md.research.elamur.ai/free/en/papers/lingbulk-2004-foster-parsing-ungrammatical-input-evaluation-procedure/)
- [↑ back to papers of 2004](https://md.research.elamur.ai/free/en/papers/2004/)

## References

19 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Adam Berger, Stephen Della Pietra, and Vincent Della Pietra. 1996. A maximum entropy approach to natural language processing. Computational Linguistics, 22(1):39–71. [find](https://scholar.google.com/scholar?q=Adam+Berger%2C+Stephen+Della+Pietra%2C+and+Vincent+Della+Pietra%2E+1996%2E+A+maximum+entropy+approach+to+natural+language+processing%2E+Computational+Linguistics%2C+22%281%29%3A39%E2%80%9371%2E) [our card](https://md.research.elamur.ai/free/en/papers/berger-1996-maximum-entropy-approach-natural-language/)
2. Stanley Chen and Ronald Rosenfeld. 1999. A Gaussian prior for smoothing maximum entropy models. Technical report, Carnegie Mellon University, Pittsburgh, PA. [find](https://scholar.google.com/scholar?q=Stanley+Chen+and+Ronald+Rosenfeld%2E+1999%2E+A+Gaussian+prior+for+smoothing+maximum+entropy+models%2E+Technical+report%2C+Carnegie+Mellon+University%2C+Pittsburgh%2C+PA%2E)
3. Stephen Clark and James R. Curran. 2003. Log-linear models for wide-coverage CCG parsing. In Proceedings of the EMNLP Conference, pages 97–104, Sapporo, Japan. [find](https://scholar.google.com/scholar?q=Stephen+Clark+and+James+R%2E+Curran%2E+2003%2E+Log%2Dlinear+models+for+wide%2Dcoverage+CCG+parsing%2E+In+Proceedings+of+the+EMNLP+Conference%2C+pages+97%E2%80%93104%2C+Sapporo%2C+Japan%2E)
4. Stephen Clark and James R. Curran. 2004. The importance of supertagging for wide-coverage CCG parsing. In Proceedings of COLING-04, Geneva, Switzerland. [find](https://scholar.google.com/scholar?q=Stephen+Clark+and+James+R%2E+Curran%2E+2004%2E+The+importance+of+supertagging+for+wide%2Dcoverage+CCG+parsing%2E+In+Proceedings+of+COLING%2D04%2C+Geneva%2C+Switzerland%2E) [our card](https://md.research.elamur.ai/free/en/papers/aclbulk-2004-clark-importance-supertagging-wide-coverage-ccg/)
5. Stephen Clark, Julia Hockenmaier, and Mark Steedman. 2002. Building deep dependency structures with a wide-coverage CCG parser. In Proceedings of the 40th Meeting of the ACL, pages 327–334, Philadelphia, PA. [find](https://scholar.google.com/scholar?q=Stephen+Clark%2C+Julia+Hockenmaier%2C+and+Mark+Steedman%2E+2002%2E+Building+deep+dependency+structures+with+a+wide%2Dcoverage+CCG+parser%2E+In+Proceedings+of+the+40th+Meeting+of+the+ACL%2C+pages+327%E2%80%93334%2C+Philadelph)
6. Michael Collins. 1996. A new statistical parser based on bigram lexical dependencies. In Proceedings of the 34th Meeting of the ACL, pages 184–191, Santa Cruz, CA. [find](https://scholar.google.com/scholar?q=Michael+Collins%2E+1996%2E+A+new+statistical+parser+based+on+bigram+lexical+dependencies%2E+In+Proceedings+of+the+34th+Meeting+of+the+ACL%2C+pages+184%E2%80%93191%2C+Santa+Cruz%2C+CA%2E)
7. James R. Curran and Stephen Clark. 2003. Investigating GIS and smoothing for maximum entropy taggers. In Proceedings of the 10th Meeting of the EACL, pages 91–98, Budapest, Hungary. [find](https://scholar.google.com/scholar?q=James+R%2E+Curran+and+Stephen+Clark%2E+2003%2E+Investigating+GIS+and+smoothing+for+maximum+entropy+taggers%2E+In+Proceedings+of+the+10th+Meeting+of+the+EACL%2C+pages+91%E2%80%9398%2C+Budapest%2C+Hungary%2E)
8. Jason Eisner. 1996. Efficient normal-form parsing for Combinatory Categorial Grammar. In Proceedings of the 34th Meeting of the ACL, pages 79–86, Santa Cruz, CA. [find](https://scholar.google.com/scholar?q=Jason+Eisner%2E+1996%2E+Efficient+normal%2Dform+parsing+for+Combinatory+Categorial+Grammar%2E+In+Proceedings+of+the+34th+Meeting+of+the+ACL%2C+pages+79%E2%80%9386%2C+Santa+Cruz%2C+CA%2E)
9. Daniel Gildea. 2001. Corpus variation and parser performance. In Proceedings of the EMNLP Conference, pages 167–202, Pittsburgh, PA. [find](https://scholar.google.com/scholar?q=Daniel+Gildea%2E+2001%2E+Corpus+variation+and+parser+performance%2E+In+Proceedings+of+the+EMNLP+Conference%2C+pages+167%E2%80%93202%2C+Pittsburgh%2C+PA%2E) [our card](https://md.research.elamur.ai/free/en/papers/aclbulk-2001-gildea-corpus-variation-parser-performance/)
10. Joshua Goodman. 1996. Parsing algorithms and metrics. In Proceedings of the 34th Meeting of the ACL, pages 177–183, Santa Cruz, CA. [find](https://scholar.google.com/scholar?q=Joshua+Goodman%2E+1996%2E+Parsing+algorithms+and+metrics%2E+In+Proceedings+of+the+34th+Meeting+of+the+ACL%2C+pages+177%E2%80%93183%2C+Santa+Cruz%2C+CA%2E)
11. Julia Hockenmaier and Mark Steedman. 2002. Generative models for statistical parsing with Combinatory Categorial Grammar. In Proceedings of the 40th Meeting of the ACL, pages 335–342, Philadelphia, PA. [find](https://scholar.google.com/scholar?q=Julia+Hockenmaier+and+Mark+Steedman%2E+2002%2E+Generative+models+for+statistical+parsing+with+Combinatory+Categorial+Grammar%2E+In+Proceedings+of+the+40th+Meeting+of+the+ACL%2C+pages+335%E2%80%93342%2C+Philadelphia%2C+PA)
12. Julia Hockenmaier. 2003a. Data and Models for Statistical Parsing with Combinatory Categorial Grammar. Ph.D. thesis, University of Edinburgh. [find](https://scholar.google.com/scholar?q=Julia+Hockenmaier%2E+2003a%2E+Data+and+Models+for+Statistical+Parsing+with+Combinatory+Categorial+Grammar%2E+Ph%2ED%2E+thesis%2C+University+of+Edinburgh%2E)
13. Julia Hockenmaier. 2003b. Parsing with generative models of predicate-argument structure. In Proceedings of the 41st Meeting of the ACL, pages 359–366, Sapporo, Japan. [find](https://scholar.google.com/scholar?q=Julia+Hockenmaier%2E+2003b%2E+Parsing+with+generative+models+of+predicate%2Dargument+structure%2E+In+Proceedings+of+the+41st+Meeting+of+the+ACL%2C+pages+359%E2%80%93366%2C+Sapporo%2C+Japan%2E)
14. Yusuke Miyao and Jun’ichi Tsujii. 2002. Maximum entropy estimation for feature forests. In Proceedings of the Human Language Technology Conference, San Diego, CA. [find](https://scholar.google.com/scholar?q=Yusuke+Miyao+and+Jun%E2%80%99ichi+Tsujii%2E+2002%2E+Maximum+entropy+estimation+for+feature+forests%2E+In+Proceedings+of+the+Human+Language+Technology+Conference%2C+San+Diego%2C+CA%2E)
15. Jorge Nocedal and Stephen J. Wright. 1999. Numerical Optimization. Springer, New York, USA. [find](https://scholar.google.com/scholar?q=Jorge+Nocedal+and+Stephen+J%2E+Wright%2E+1999%2E+Numerical+Optimization%2E+Springer%2C+New+York%2C+USA%2E)
16. Stefan Riezler, Tracy H. King, Ronald M. Kaplan, Richard Crouch, John T. Maxwell III, and Mark Johnson. 2002. Parsing the Wall Street Journal using a Lexical-Functional Grammar and discriminative estimation techniques. In Proceedings of the 40th Meeting of the ACL, pages 271–278, Philadelphia, PA. [find](https://scholar.google.com/scholar?q=Stefan+Riezler%2C+Tracy+H%2E+King%2C+Ronald+M%2E+Kaplan%2C+Richard+Crouch%2C+John+T%2E+Maxwell+III%2C+and+Mark+Johnson%2E+2002%2E+Parsing+the+Wall+Street+Journal+using+a+Lexical%2DFunctional+Grammar+and+discriminative+esti) [our card](https://md.research.elamur.ai/free/en/papers/riezler-2002-parsing-wall-street-journal-lexical/)
17. Fei Sha and Fernando Pereira. 2003. Shallow parsing with conditional random fields. In Proceedings of the HLT/NAACL Conference, pages 213–220, Edmonton, Canada. [find](https://scholar.google.com/scholar?q=Fei+Sha+and+Fernando+Pereira%2E+2003%2E+Shallow+parsing+with+conditional+random+fields%2E+In+Proceedings+of+the+HLT%2FNAACL+Conference%2C+pages+213%E2%80%93220%2C+Edmonton%2C+Canada%2E)
18. Mark Steedman. 2000. The Syntactic Process. The MIT Press, Cambridge, MA. [find](https://scholar.google.com/scholar?q=Mark+Steedman%2E+2000%2E+The+Syntactic+Process%2E+The+MIT+Press%2C+Cambridge%2C+MA%2E)
19. Kristina Toutanova, Christopher Manning, Stuart Shieber, Dan Flickinger, and Stephan Oepen. 2002. Parse disambiguation for a rich HPSG grammar. In Proceedings of the First Workshop on Treebanks and Linguistic Theories, pages 253–263, Sozopol, Bulgaria. [find](https://scholar.google.com/scholar?q=Kristina+Toutanova%2C+Christopher+Manning%2C+Stuart+Shieber%2C+Dan+Flickinger%2C+and+Stephan+Oepen%2E+2002%2E+Parse+disambiguation+for+a+rich+HPSG+grammar%2E+In+Proceedings+of+the+First+Workshop+on+Treebanks+and+Li)