---
title: "Road-testing the English Resource Grammar — Baldwin 2004"
description: "The authors test how a large \"deep\" grammar, built for narrow domains, performs on open text."
authors: ["Timothy Baldwin", "Emily M. Bender", "Dan Flickinger", "Ara Kim", "Stephan Oepen"]
year: 2004
venue: "Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC'04)"
source: "https://aclanthology.org/L04-1488/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/baldwin-2004-road-testing-english-resource-grammar/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/baldwin-2004-road-testing-english-resource-grammar/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2004](https://md.research.elamur.ai/free/papers/2004/) / Road-testing the English Resource Grammar Over the British National Corpus

# Road-testing the English Resource Grammar Over the British National Corpus

**Authors:** [Timothy Baldwin](https://md.research.elamur.ai/free/authors/timothy-baldwin/), [Emily M. Bender](https://md.research.elamur.ai/free/authors/emily-m-bender/), [Dan Flickinger](https://md.research.elamur.ai/free/authors/dan-flickinger/), Ara Kim, [Stephan Oepen](https://md.research.elamur.ai/free/authors/stephan-oepen/) · **Year:** 2004 · **Venue:** Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC'04) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/L04-1488/) · [PDF](https://aclanthology.org/L04-1488.pdf) · [OpenAlex](https://openalex.org/W2187452446)

**Citations:** 64 citations (openalex-snapshot · 2026-10-14) · 31 in our library, of the 140673 papers whose bibliographies we read

## Summary
The authors test how a large "deep" grammar, built for narrow domains, performs on open text. They also look for the cheapest ways to close gaps in its coverage. They study the English Resource Grammar (ERG), an open-source HPSG grammar with MRS semantics. It was developed mainly on informal data: dialogues about scheduling meetings and e-commerce correspondence. The ERG was run on a random sample of 20,000 lines from the written portion of the British National Corpus (BNC). Beforehand, the text was POS-tagged, names and numbers were tokenized, and British spelling was converted to American spelling. Only lines whose words were all already known to the grammar were analyzed. For each unparsed sentence, the cause of failure was determined manually: minimal paraphrases were proposed, and the authors checked whether the grammar could parse them. The most frequent cause of failures turned out to be gaps in the lexicon rather than missing syntactic constructions. Extending the lexicon therefore yields the greatest gain, including for the rest of the corpus. The authors propose adding vocabulary semi-automatically, starting with nouns. Candidate entries would be validated against test sentences generated by the grammar itself. The paper shows the real cost of porting a precise hand-built grammar to a new domain.

## Key points
- The ERG (July 2003 version) has about 10,500 lexical items, of which roughly 3,000 are multiword expressions (MWEs). It also contains 77 phrase structure rules and about 4,000 types in its hierarchy.
- Only 32% of BNC lines had full lexical coverage. Of these, the ERG parsed 57%, and 83% of those parses were correct. For sentences 10–20 words long, the grammar produced an average of 64 analyses.
- Causes of failure were distributed as follows: lexical gaps 40% (incomplete categorization of words 32%, missing MWEs 8%), missing constructions 39%, "garbage" lines 11%. Preprocessing errors, fragments and parser failures accounted for 4% each.
- Among unknown words, 61% are nouns, 22% adjectives and 13% verbs. Verbs have 131 leaf lexical types, and the 10 most frequent cover only ~77% of verbal items. For nouns this share is 96%, so they are the easiest to add to the lexicon.
- The PET parser processes the vast majority of sentences in under a second. On its native domains, the ERG typically covers over 80% of sentences.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{baldwin2004road,
  title = {{Road-testing the English Resource Grammar Over the British National Corpus}},
  author = {Baldwin, Timothy and Bender, Emily M. and Flickinger, Dan and Kim, Ara and Oepen, Stephan},
  booktitle = {Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC'04)},
  year = {2004},
  doi = {10.63317/49266tjgdk8s},
  url = {https://doi.org/10.63317/49266tjgdk8s}
}
```

- [← previous Representing discourse coherence: A corpus-based analysis](https://md.research.elamur.ai/free/papers/wolf-2004-representing-discourse-coherence-corpus-based/)
- [next → Robust VPE Detection using Automatically Parsed Text](https://md.research.elamur.ai/free/papers/aclbulk-2004-nielsen-robust-vpe-detection-automatically-parsed/)
- [↑ back to papers of 2004](https://md.research.elamur.ai/free/papers/2004/)

## References

10 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Timothy Baldwin and Francis Bond. 2003. Learning the countability of English nouns from corpus data. In Proc. of the 41st Annual Meeting of the ACL, pages 463–70, Sapporo, Japan. [find](https://scholar.google.com/scholar?q=Timothy+Baldwin+and+Francis+Bond%2E+2003%2E+Learning+the+countability+of+English+nouns+from+corpus+data%2E+In+Proc%2E+of+the+41st+Annual+Meeting+of+the+ACL%2C+pages+463%E2%80%9370%2C+Sapporo%2C+Japan%2E)
2. Lou Burnard. 2000. User Reference Guide for the British National Corpus. Technical report, Oxford University Computing Services. [find](https://scholar.google.com/scholar?q=Lou+Burnard%2E+2000%2E+User+Reference+Guide+for+the+British+National+Corpus%2E+Technical+report%2C+Oxford+University+Computing+Services%2E)
3. Ulrich Callmeier. 2002. PET – a platform for experimentation with efficient HPSG processing techniques. In (Oepen et al., 2002a). [find](https://scholar.google.com/scholar?q=Ulrich+Callmeier%2E+2002%2E+PET+%E2%80%93+a+platform+for+experimentation+with+efficient+HPSG+processing+techniques%2E+In+%28Oepen+et+al%2E%2C+2002a%29%2E)
4. Ann Copestake and Dan Flickinger. 2000. An open-source grammar development environment and broad-coverage english grammar using HPSG. In Proc. of the 2nd International Conference on Language Resources and Evaluation (LREC 2000), Athens, Greece. [find](https://scholar.google.com/scholar?q=Ann+Copestake+and+Dan+Flickinger%2E+2000%2E+An+open%2Dsource+grammar+development+environment+and+broad%2Dcoverage+english+grammar+using+HPSG%2E+In+Proc%2E+of+the+2nd+International+Conference+on+Language+Resources)
5. Ann Copestake, Dan Flickinger, Ivan A. Sag, and Carl Pollard. 2003. Minimal recursion semantics: An introduction. Under review. [find](https://scholar.google.com/scholar?q=Ann+Copestake%2C+Dan+Flickinger%2C+Ivan+A%2E+Sag%2C+and+Carl+Pollard%2E+2003%2E+Minimal+recursion+semantics%3A+An+introduction%2E+Under+review%2E)
6. Dan Flickinger. 2002. On building a more efficient grammar by exploiting types. In (Oepen et al., 2002a). [find](https://scholar.google.com/scholar?q=Dan+Flickinger%2E+2002%2E+On+building+a+more+efficient+grammar+by+exploiting+types%2E+In+%28Oepen+et+al%2E%2C+2002a%29%2E) [our card](https://md.research.elamur.ai/free/papers/parsers-2000-flickinger-building-more-efficient-grammar/)
7. Stephan Oepen, Dan Flickinger, Jun’ichi Tsujii, and Hans Uszkoreit, editors. 2002a. Collaborative Language Engineering. CSLI Publications, Stanford, USA. [find](https://scholar.google.com/scholar?q=Stephan+Oepen%2C+Dan+Flickinger%2C+Jun%E2%80%99ichi+Tsujii%2C+and+Hans+Uszkoreit%2C+editors%2E+2002a%2E+Collaborative+Language+Engineering%2E+CSLI+Publications%2C+Stanford%2C+USA%2E)
8. Stephan Oepen, Kristina Toutanova, Stuart Shieber, Christopher Manning, Dan Flickinger, and Thorsten Brants. 2002b. The LinGO Redwoods Treebank: Motivation and preliminary applications. In Proc. of the 19th International Conference on Computational Linguistics (COLING 2002), pages 1253–7, Taipei, Taiwan. [find](https://scholar.google.com/scholar?q=Stephan+Oepen%2C+Kristina+Toutanova%2C+Stuart+Shieber%2C+Christopher+Manning%2C+Dan+Flickinger%2C+and+Thorsten+Brants%2E+2002b%2E+The+LinGO+Redwoods+Treebank%3A+Motivation+and+preliminary+applications%2E+In+Proc%2E+of+th) [our card](https://md.research.elamur.ai/free/papers/bank-2002-oepen-lingo-redwoods-treebank-motivation-preliminary/)
9. F. Jeffrey Pelletier. 1979. Non-singular reference: Some preliminaries. In F. Jeffrey Pelletier, editor, Mass Terms: Some Philosophical Problems, pages 1–14. Dordrecht, Reidel. [find](https://scholar.google.com/scholar?q=F%2E+Jeffrey+Pelletier%2E+1979%2E+Non%2Dsingular+reference%3A+Some+preliminaries%2E+In+F%2E+Jeffrey+Pelletier%2C+editor%2C+Mass+Terms%3A+Some+Philosophical+Problems%2C+pages+1%E2%80%9314%2E+Dordrecht%2C+Reidel%2E)
10. Hans Uszkoreit. 2002. New chances for deep linguistic processing. In Proc. of the 19th International Conference on Computational Linguistics (COLING 2002), Taipei, Taiwan. [find](https://scholar.google.com/scholar?q=Hans+Uszkoreit%2E+2002%2E+New+chances+for+deep+linguistic+processing%2E+In+Proc%2E+of+the+19th+International+Conference+on+Computational+Linguistics+%28COLING+2002%29%2C+Taipei%2C+Taiwan%2E)