---
title: "Parallel Distributed Grammar Engineering for Practical Applications"
description: "The paper examines how to control the quality of a large constraint-based grammar when it is… 18 references, cited by 2 papers here."
authors: ["Stephan Oepen", "Emily M. Bender", "Uli Callmeier", "Dan Flickinger", "Melanie Siegel"]
year: 2002
venue: "COLING-02: Grammar Engineering and Evaluation"
source: "https://aclanthology.org/W02-1508/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/oepen-2002-parallel-distributed-grammar-engineering-practical/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/oepen-2002-parallel-distributed-grammar-engineering-practical/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2002](https://md.research.elamur.ai/free/en/papers/2002/) / Parallel Distributed Grammar Engineering for Practical Applications

# Parallel Distributed Grammar Engineering for Practical Applications

**Authors:** [Stephan Oepen](https://md.research.elamur.ai/free/en/authors/stephan-oepen/), [Emily M. Bender](https://md.research.elamur.ai/free/en/authors/emily-m-bender/), Uli Callmeier, [Dan Flickinger](https://md.research.elamur.ai/free/en/authors/dan-flickinger/) and Melanie Siegel · **Year:** 2002 · **Venue:** COLING-02: Grammar Engineering and Evaluation · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W02-1508/) · [PDF](https://aclanthology.org/W02-1508.pdf) · [OpenAlex](https://openalex.org/W2016427405)

**Citations:** 17 citations (OpenAlex, 2026-10-14) · 2 in our library, of the 149,439 papers whose bibliographies we read

## Summary

The paper examines how to control the quality of a large constraint-based grammar when it is developed simultaneously by teams in different countries. The example is an open-source HPSG grammar of Japanese. Over eight months it was improved in parallel by DFKI (Saarbrücken) and the company YY Technologies (California), while the grammar was already running in a commercial product. The authors show that even a minor fix in one lexical class can unpredictably break neighboring constructions, the semantics, or the parse-ranking heuristics. From this they derive requirements for regression testing. The solution was the [incr tsdb()] environment for competence and performance profiling. It stores annotated test suites and runs them in batch and distributed mode through different parsers, including LKB and PET. It then collects metrics for coverage, ambiguity, time, and memory, and compares profiles of different grammar versions. Fast daily checks made it possible to port the grammar from spoken language to email and to a new domain, substantially extending coverage. The work demonstrates that, given a proper engineering methodology, broad-coverage HPSG grammars are already suitable for real-world applications.

## Key points

- On representative customer corpora, the coverage of the Japanese grammar rose from roughly 40% to over 90%. The grammar was ported from the appointment-scheduling domain (VerbMobil) to customer-support requests submitted by email.
- Over five months, coverage on the "banking" data increased by more than 70% in relative terms. The grammar itself grew only moderately: the number of types by about 15%, the number of rules by about 20%.
- As of June 2002, [incr tsdb()] had been integrated with nine grammar development and parsing systems, among them LKB, PET, LiLFeS, TRALE, and PAGE. The environment has three components: a DBMS, a batch control and statistics kernel with an API for C and Lisp, and a graphical interface in Tcl/Tk.
- ChaSen was used for tokenization. The average input length was about 50 lexical items for the "banking" set and 80 for "trading". The grammar typically produced fewer than ten parses per sentence.
- The grammar in the product was updated once or twice a week. At the start of each working day, the developers spent 10–20 minutes obtaining fresh profiles on the shared test suites.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 434 works · 314,419 citations
- [German Research Centre for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/german-research-centre-for-artificial-intelligence/) — Research institute, Kaiserslautern, Germany · 77 works · 29,479 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

🧭 Topics (1): [📐 Syntax & grammar formalisms](https://md.research.elamur.ai/free/en/topics/syntax-grammar/) · 214 works

## Cite

Cite as: Stephan Oepen, Emily M. Bender, Uli Callmeier, Dan Flickinger and Melanie Siegel. 2002. Parallel Distributed Grammar Engineering for Practical Applications. COLING-02: Grammar Engineering and Evaluation.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{oepen2002parallel,
  title = {{Parallel Distributed Grammar Engineering for Practical Applications}},
  author = {Stephan Oepen, Emily M. Bender, Uli Callmeier, Dan Flickinger and Melanie Siegel},
  booktitle = {COLING-02: Grammar Engineering and Evaluation},
  year = {2002},
  doi = {10.3115/1118783.1118791},
  url = {https://doi.org/10.3115/1118783.1118791}
}
```

- [← previous A Comprehensive NLP System for Modern Standard Arabic and Modern Hebrew](https://md.research.elamur.ai/free/en/papers/kamir-2002-comprehensive-nlp-system-modern-standard/)
- [next → Acquisition System for Arabic Noun Morphology](https://md.research.elamur.ai/free/en/papers/abuleil-2002-acquisition-system-arabic-noun-morphology/)
- [↑ back to papers of 2002](https://md.research.elamur.ai/free/en/papers/2002/)

## References

18 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Asahara, M., & Matsumoto, Y. (2000). Extended models and tools for high-performance part-of-speech tagger. In Proceedings of the 18th International Conference on Computational Linguistics (pp. 21 – 27). Saarbrücken, Germany. [find](https://scholar.google.com/scholar?q=Asahara%2C+M%2E%2C+%26+Matsumoto%2C+Y%2E+%282000%29%2E+Extended+models+and+tools+for+high%2Dperformance+part%2Dof%2Dspeech+tagger%2E+In+Proceedings+of+the+18th+International+Conference+on+Computational+Linguistics+%28pp%2E+21+%E2%80%93+27)
2. Butt, M., King, T. H., Niño, M.-E., & Segond, F. (1999). A grammar writer’s cookbook. Stanford, CA: CSLI Publications. [find](https://scholar.google.com/scholar?q=Butt%2C+M%2E%2C+King%2C+T%2E+H%2E%2C+Nin%CC%83o%2C+M%2E%2DE%2E%2C+%26+Segond%2C+F%2E+%281999%29%2E+A+grammar+writer%E2%80%99s+cookbook%2E+Stanford%2C+CA%3A+CSLI+Publications%2E)
3. Callmeier, U. (2002). Preprocessing and encoding techniques in PET. In S. Oepen, D. Flickinger, J. Tsujii, & H. Uszkoreit (Eds.), Collaborative language engineering. A case study in efficient grammar-based processing. Stanford, CA: CSLI Publications. (forthcoming) [find](https://scholar.google.com/scholar?q=Callmeier%2C+U%2E+%282002%29%2E+Preprocessing+and+encoding+techniques+in+PET%2E+In+S%2E+Oepen%2C+D%2E+Flickinger%2C+J%2E+Tsujii%2C+%26+H%2E+Uszkoreit+%28Eds%2E%29%2C+Collaborative+language+engineering%2E+A+case+study+in+efficient+grammar%2D)
4. Carroll, J. (1994). Relating complexity to practical performance in parsing with wide-coverage unification grammars. In Proceedings of the 32nd Meeting of the Association for Computational Linguistics (pp. 287 – 294). Las Cruces, NM. [find](https://scholar.google.com/scholar?q=Carroll%2C+J%2E+%281994%29%2E+Relating+complexity+to+practical+performance+in+parsing+with+wide%2Dcoverage+unification+grammars%2E+In+Proceedings+of+the+32nd+Meeting+of+the+Association+for+Computational+Linguistics)
5. Copestake, A. (2002). Implementing typed feature structure grammars. Stanford, CA: CSLI Publications. [find](https://scholar.google.com/scholar?q=Copestake%2C+A%2E+%282002%29%2E+Implementing+typed+feature+structure+grammars%2E+Stanford%2C+CA%3A+CSLI+Publications%2E)
6. Erbach, G. (1991). An environment for experimenting with parsing strategies. In J. Mylopoulos & R. Reiter (Eds.), Proceedings of IJCAI 1991 (pp. 931 – 937). San Mateo, CA: Morgan Kaufmann Publishers. [find](https://scholar.google.com/scholar?q=Erbach%2C+G%2E+%281991%29%2E+An+environment+for+experimenting+with+parsing+strategies%2E+In+J%2E+Mylopoulos+%26+R%2E+Reiter+%28Eds%2E%29%2C+Proceedings+of+IJCAI+1991+%28pp%2E+931+%E2%80%93+937%29%2E+San+Mateo%2C+CA%3A+Morgan+Kaufmann+Publishers%2E)
7. Erbach, G., & Uszkoreit, H. (1990). Grammar engineering. Problems and prospects (CLAUS Report \# 1). Saarbrücken, Germany: Computational Linguistics, Saarland University. [find](https://scholar.google.com/scholar?q=Erbach%2C+G%2E%2C+%26+Uszkoreit%2C+H%2E+%281990%29%2E+Grammar+engineering%2E+Problems+and+prospects+%28CLAUS+Report+%5C%23+1%29%2E+Saarbru%CC%88cken%2C+Germany%3A+Computational+Linguistics%2C+Saarland+University%2E)
8. Flickinger, D., Copestake, A., & Sag, I. A. (2000). HPSG analysis of English. In W. Wahlster (Ed.), Verbmobil. Foundations of speech-to-speech translation (Artificial Intelligence ed., pp. 321 – 330). Berlin, Germany: Springer. [find](https://scholar.google.com/scholar?q=Flickinger%2C+D%2E%2C+Copestake%2C+A%2E%2C+%26+Sag%2C+I%2E+A%2E+%282000%29%2E+HPSG+analysis+of+English%2E+In+W%2E+Wahlster+%28Ed%2E%29%2C+Verbmobil%2E+Foundations+of+speech%2Dto%2Dspeech+translation+%28Artificial+Intelligence+ed%2E%2C+pp%2E+321+%E2%80%93+330%29%2E)
9. Geist, A., Bequelin, A., Dongarra, J., Manchek, W. J. R., & Sunderam, V. (Eds.). (1994). PVM — parallel virtual machine. A users’ guide and tutorial for networked parallel computing. Cambridge, MA: The MIT Press. [find](https://scholar.google.com/scholar?q=Geist%2C+A%2E%2C+Bequelin%2C+A%2E%2C+Dongarra%2C+J%2E%2C+Manchek%2C+W%2E+J%2E+R%2E%2C+%26+Sunderam%2C+V%2E+%28Eds%2E%29%2E+%281994%29%2E+PVM+%E2%80%94+parallel+virtual+machine%2E+A+users%E2%80%99+guide+and+tutorial+for+networked+parallel+computing%2E+Cambridge%2C+MA%3A+Th)
10. Makino, T., Yoshida, M., Torisawa, K., & Tsujii, J. (1998). LiLFeS — towards a practical HPSG parser. In Proceedings of the 17th International Conference on Computational Linguistics and the 36th Annual Meeting of the Association for Computational Linguistics (pp. 807 – 11). Montreal, Canada. [find](https://scholar.google.com/scholar?q=Makino%2C+T%2E%2C+Yoshida%2C+M%2E%2C+Torisawa%2C+K%2E%2C+%26+Tsujii%2C+J%2E+%281998%29%2E+LiLFeS+%E2%80%94+towards+a+practical+HPSG+parser%2E+In+Proceedings+of+the+17th+International+Conference+on+Computational+Linguistics+and+the+36th+Annu)
11. Müller, S., & Kasper, W. (2000). HPSG analysis of German. In W. Wahlster (Ed.), Verbmobil. Foundations of speech-to-speech translation (Artificial Intelligence ed., pp. 238 – 253). Berlin, Germany: Springer. [find](https://scholar.google.com/scholar?q=Mu%CC%88ller%2C+S%2E%2C+%26+Kasper%2C+W%2E+%282000%29%2E+HPSG+analysis+of+German%2E+In+W%2E+Wahlster+%28Ed%2E%29%2C+Verbmobil%2E+Foundations+of+speech%2Dto%2Dspeech+translation+%28Artificial+Intelligence+ed%2E%2C+pp%2E+238+%E2%80%93+253%29%2E+Berlin%2C+Germany%3A+S)
12. Oepen, S., & Callmeier, U. (2000). Measure for measure: Parser cross-fertilization. Towards increased component comparability and exchange. In Proceedings of the 6th International Workshop on Parsing Technologies (pp. 183 – 194). Trento, Italy. [find](https://scholar.google.com/scholar?q=Oepen%2C+S%2E%2C+%26+Callmeier%2C+U%2E+%282000%29%2E+Measure+for+measure%3A+Parser+cross%2Dfertilization%2E+Towards+increased+component+comparability+and+exchange%2E+In+Proceedings+of+the+6th+International+Workshop+on+Parsing+)
13. Oepen, S., & Flickinger, D. P. (1998). Towards systematic grammar profiling. Test suite technology ten years after. Journal of Computer Speech and Language, 12 (4) (Special Issue on Evaluation), 411 – 436. [find](https://scholar.google.com/scholar?q=Oepen%2C+S%2E%2C+%26+Flickinger%2C+D%2E+P%2E+%281998%29%2E+Towards+systematic+grammar+profiling%2E+Test+suite+technology+ten+years+after%2E+Journal+of+Computer+Speech+and+Language%2C+12+%284%29+%28Special+Issue+on+Evaluation%29%2C+411+%E2%80%93)
14. Oepen, S., Netter, K., & Klein, J. (1997). TSNLP — Test Suites for Natural Language Processing. In J. Nerbonne (Ed.), Linguistic Databases (pp. 13 – 36). Stanford, CA: CSLI Publications. [find](https://scholar.google.com/scholar?q=Oepen%2C+S%2E%2C+Netter%2C+K%2E%2C+%26+Klein%2C+J%2E+%281997%29%2E+TSNLP+%E2%80%94+Test+Suites+for+Natural+Language+Processing%2E+In+J%2E+Nerbonne+%28Ed%2E%29%2C+Linguistic+Databases+%28pp%2E+13+%E2%80%93+36%29%2E+Stanford%2C+CA%3A+CSLI+Publications%2E)
15. Penn, G. (2000). Applying constraint handling rules to HPSG. In Proceedings of the first international conference on computational logic (pp. 51 – 68). London, UK. [find](https://scholar.google.com/scholar?q=Penn%2C+G%2E+%282000%29%2E+Applying+constraint+handling+rules+to+HPSG%2E+In+Proceedings+of+the+first+international+conference+on+computational+logic+%28pp%2E+51+%E2%80%93+68%29%2E+London%2C+UK%2E)
16. Siegel, M. (2000). HPSG analysis of Japanese. In W. Wahlster (Ed.), Verbmobil. Foundations of speech-to-speech translation (Artificial Intelligence ed., pp. 265 – 280). Berlin, Germany: Springer. [find](https://scholar.google.com/scholar?q=Siegel%2C+M%2E+%282000%29%2E+HPSG+analysis+of+Japanese%2E+In+W%2E+Wahlster+%28Ed%2E%29%2C+Verbmobil%2E+Foundations+of+speech%2Dto%2Dspeech+translation+%28Artificial+Intelligence+ed%2E%2C+pp%2E+265+%E2%80%93+280%29%2E+Berlin%2C+Germany%3A+Springer%2E)
17. Siegel, M., & Bender, E. M. (2002). Efficient deep processing of japanese. In Proceedings of the 19th International Conference on Computational Linguistics. Taipei, Taiwan. [find](https://scholar.google.com/scholar?q=Siegel%2C+M%2E%2C+%26+Bender%2C+E%2E+M%2E+%282002%29%2E+Efficient+deep+processing+of+japanese%2E+In+Proceedings+of+the+19th+International+Conference+on+Computational+Linguistics%2E+Taipei%2C+Taiwan%2E)
18. Uszkoreit, H., Backofen, R., Busemann, S., Diagne, A. K., Hinkelman, E. A., Kasper, W., Kiefer, B., Krieger, H.-U., Netter, K., Neumann, G., Oepen, S., & Spackman, S. P. (1994). DISCO — an HPSG-based NLP system and its application for appointment scheduling. In Proceedings of the 15th International… [find](https://scholar.google.com/scholar?q=Uszkoreit%2C+H%2E%2C+Backofen%2C+R%2E%2C+Busemann%2C+S%2E%2C+Diagne%2C+A%2E+K%2E%2C+Hinkelman%2C+E%2E+A%2E%2C+Kasper%2C+W%2E%2C+Kiefer%2C+B%2E%2C+Krieger%2C+H%2E%2DU%2E%2C+Netter%2C+K%2E%2C+Neumann%2C+G%2E%2C+Oepen%2C+S%2E%2C+%26+Spackman%2C+S%2E+P%2E+%281994%29%2E+DISCO+%E2%80%94+an+HPSG%2Dbased+)

*Updated 31 October 2026*