---
title: "Holistic Regression Testing for High-Quality MT — Oepen 2005"
description: "The authors argue that regression testing needs annotated inputs and a database of results, not log files, and that… 15 references, cited by 1 paper here."
authors: ["Stephan Oepen", "Helge Dyvik", "Dan Flickinger", "Jan Tore Lønning", "Paul Meurer", "Victoria Rosén"]
year: 2005
venue: "Proceedings of the 10th EAMT Conference: Practical applications of machine translation"
source: "https://aclanthology.org/2005.eamt-1.27/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/oepen-2005-holistic-regression-testing-high-quality/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/oepen-2005-holistic-regression-testing-high-quality/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2005](https://md.research.elamur.ai/free/en/papers/2005/) / Holistic Regression Testing for High-Quality MT: Some Methodological and Technological Reflections

# Holistic Regression Testing for High-Quality MT: Some Methodological and Technological Reflections

**Authors:** [Stephan Oepen](https://md.research.elamur.ai/free/en/authors/stephan-oepen/), [Helge Dyvik](https://md.research.elamur.ai/free/en/authors/helge-dyvik/), [Dan Flickinger](https://md.research.elamur.ai/free/en/authors/dan-flickinger/) et al. · **Year:** 2005 · **Venue:** Proceedings of the 10th EAMT Conference: Practical applications of machine translation **Link:** [aclanthology.org/2005.eamt-1.27](https://aclanthology.org/2005.eamt-1.27/) · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) — summary only

[**0** citations](https://openalex.org/works?filter=cites:W3103732163) OpenAlex, 2026-10-14 · **1** citations in our library of the 140,884 papers whose bibliographies we read · [Proceedings of the 10th EAMT Conference: Practical applications of machine translation](https://aclanthology.org/volumes/2005.eamt-1/) · [licence](https://creativecommons.org/licenses/by-nc-sa/3.0/)

- [Publisher's page](https://aclanthology.org/2005.eamt-1.27/)
- [PDF](https://aclanthology.org/2005.eamt-1.27.pdf)
- [OpenAlex](https://openalex.org/W3103732163)

**Authors (6)**

1. [Stephan Oepen](https://openalex.org/A5058412010) [Semantic Scholar](https://www.semanticscholar.org/author/2949607) · h 35 · 3,542 citations · 215 works
2. [Helge Dyvik](https://openalex.org/A5024530852) [Semantic Scholar](https://www.semanticscholar.org/author/2957905) · h 10 · 635 citations · 40 works
3. [Dan Flickinger](https://openalex.org/A5089106912) [ORCID](https://orcid.org/0000-0001-7959-999X) · [Semantic Scholar](https://www.semanticscholar.org/author/3209288) · h 31 · 5,030 citations · 90 works
4. [Jan Tore Lønning](https://openalex.org/A5030107252) [Semantic Scholar](https://www.semanticscholar.org/author/1403877043) · h 11 · 676 citations · 25 works
5. [Paul Meurer](https://openalex.org/A5083323467) [Semantic Scholar](https://www.semanticscholar.org/author/2481294) · h 10 · 291 citations · 41 works
6. [Victoria Rosén](https://openalex.org/A5070082766) [ORCID](https://orcid.org/0000-0002-1059-1049) · [Semantic Scholar](https://www.semanticscholar.org/author/38897341) · h 11 · 308 citations · 34 works

Citations and author metrics: OpenAlex, 2026-10-14; authors with the same name are not always told apart.

## What it is about

The LOGON project, a Norwegian-English MT system for tourism texts, uses regression testing as its main quality measure. It combines hand-built test suites, a development corpus of real text and tools that profile each run. The closed-class suite ('base') has close to 300 items over about 15 top-level phenomena, 66 in total. A 110-item sample of tourism text (average 13 words, up to 30) is the development set. The profiles report coverage and ambiguity for analysis, transfer and generation per phenomenon. The authors argue that regression testing needs annotated inputs and a database of results, not log files, and that stable results across releases keep the gold data cheap to update.

## Ideas

- I2. Per-phenomenon profile: group EWT test errors by the phenomenon tags of the suite and report LAS per group.

## For Mova

- I1. Build a regression suite under `treebanks/en/`: a hand-written closed-class set of about 300 items, each tagged with its phenomenon. Run `en parse-eval` on it at each release and store the results as a table keyed by item and release.
- I3. Stability gate: count the analyses that change between two `en` builds on the suite and on the 110-item sample. Every change must be listed. Negative control: a deliberately broken rule must turn the gate red.

The release gate is `en/src/udreg.rs`, with `en ud-check` and `en folds` for decisions. The chapter for the closed-class items is [languages/en/19-lexicon-mwe-register.md](https://mova.elamur.ai/free/en/docs/languages/en/19-lexicon-mwe-register/). The LOGON data are not in our repo and are MT-specific; the suite has to be written by us.

Topics[Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/)

## Cite

Cite as Stephan Oepen, Helge Dyvik, Dan Flickinger, Jan Tore Lønning, Paul Meurer and Victoria Rosén. 2005. Holistic Regression Testing for High-Quality MT: Some Methodological and Technological Reflections. Proceedings of the 10th EAMT Conference: Practical applications of machine translation.

The paper itself, not this card. BibTeX from the library's record of the work.

```bibtex
@inproceedings{oepen2005holistic,
  title = {{Holistic Regression Testing for High-Quality MT: Some Methodological and Technological Reflections}},
  author = {Stephan Oepen, Helge Dyvik, Dan Flickinger, Jan Tore Lønning, Paul Meurer and Victoria Rosén},
  booktitle = {Proceedings of the 10th EAMT Conference: Practical applications of machine translation},
  year = {2005},
  url = {https://aclanthology.org/2005.eamt-1.27/}
}
```

- [← previous The Types and Distributions of Errors in a Wide Coverage Surface Realizer Evaluation](https://md.research.elamur.ai/free/en/papers/callaway-2005-types-distributions-errors-wide-coverage/)
- [next → Dimensions of Deep Grammar Validation](https://md.research.elamur.ai/free/en/papers/flickinger-2005-dimensions-deep-grammar-validation/)
- [↑ back to papers of 2005](https://md.research.elamur.ai/free/en/papers/2005/)

## References

15 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Carroll, J., Copestake, A., Flickinger, D., & Poznanski, V. (1999). An efficient chart generator for (semi-)lexicalist grammars. In Proceedings of the 7th European Workshop on Natural Language Generation (pp. 86 – 95). Toulouse, France. [find](https://scholar.google.com/scholar?q=Carroll%2C+J%2E%2C+Copestake%2C+A%2E%2C+Flickinger%2C+D%2E%2C+%26+Poznanski%2C+V%2E+%281999%29%2E+An+efficient+chart+generator+for+%28semi%2D%29lexicalist+grammars%2E+In+Proceedings+of+the+7th+European+Workshop+on+Natural+Language+Generat)
2. Copestake, A. (2002). Implementing typed feature structure grammars. Stanford, CA: CSLI Publications. [find](https://scholar.google.com/scholar?q=Copestake%2C+A%2E+%282002%29%2E+Implementing+typed+feature+structure+grammars%2E+Stanford%2C+CA%3A+CSLI+Publications%2E)
3. Copestake, A., Flickinger, D., Sag, I. A., & Pollard, C. (2003). Minimal Recursion Semantics. An introduction. In preparation, CSLI Stanford, Stanford, CA. [find](https://scholar.google.com/scholar?q=Copestake%2C+A%2E%2C+Flickinger%2C+D%2E%2C+Sag%2C+I%2E+A%2E%2C+%26+Pollard%2C+C%2E+%282003%29%2E+Minimal+Recursion+Semantics%2E+An+introduction%2E+In+preparation%2C+CSLI+Stanford%2C+Stanford%2C+CA%2E)
4. Dyvik, H. (1999). The universality of f-structure. Discovery or stipulation? The case of modals. In Proceedings of the 4th International Lexical Functional Grammar Conference. Manchester, UK. [find](https://scholar.google.com/scholar?q=Dyvik%2C+H%2E+%281999%29%2E+The+universality+of+f%2Dstructure%2E+Discovery+or+stipulation%3F+The+case+of+modals%2E+In+Proceedings+of+the+4th+International+Lexical+Functional+Grammar+Conference%2E+Manchester%2C+UK%2E)
5. Flickinger, D. (2000). On building a more efficient grammar by exploiting types. Natural Language Engineering, 6 (1) (Special Issue on Efficient Processing with HPSG), 15 – 28. [our card](https://md.research.elamur.ai/free/en/papers/flickinger-2000-building-more-efficient-grammar/)
6. Flickinger, D., Nerbonne, J., Sag, I. A., & Wasow, T. (1987). Toward evaluation of NLP systems (Technical Report). Hewlett-Packard Laboratories. (Distributed at the 24th Annual Meeting of the Association for Computational Linguistics) [find](https://scholar.google.com/scholar?q=Flickinger%2C+D%2E%2C+Nerbonne%2C+J%2E%2C+Sag%2C+I%2E+A%2E%2C+%26+Wasow%2C+T%2E+%281987%29%2E+Toward+evaluation+of+NLP+systems+%28Technical+Report%29%2E+Hewlett%2DPackard+Laboratories%2E+%28Distributed+at+the+24th+Annual+Meeting+of+the+Associat)
7. Geist, A., Bequelin, A., Dongarra, J., Manchek, W. J. R., & Sunderam, V. (Eds.). (1994). PVM. Parallel Virtual Machine. A users’ guide and tutorial for networked parallel computing. Cambridge, MA: The MIT Press. [find](https://scholar.google.com/scholar?q=Geist%2C+A%2E%2C+Bequelin%2C+A%2E%2C+Dongarra%2C+J%2E%2C+Manchek%2C+W%2E+J%2E+R%2E%2C+%26+Sunderam%2C+V%2E+%28Eds%2E%29%2E+%281994%29%2E+PVM%2E+Parallel+Virtual+Machine%2E+A+users%E2%80%99+guide+and+tutorial+for+networked+parallel+computing%2E+Cambridge%2C+MA%3A+The)
8. Oepen, S., & Callmeier, U. (2004). Measure for measure. Towards increased component comparability and exchange. In H. Bunt, J. Carroll, & G. Satta (Eds.), New developments in parsing technology. Dordrecht, The Netherlands: Kluwer Academic Publishers. [find](https://scholar.google.com/scholar?q=Oepen%2C+S%2E%2C+%26+Callmeier%2C+U%2E+%282004%29%2E+Measure+for+measure%2E+Towards+increased+component+comparability+and+exchange%2E+In+H%2E+Bunt%2C+J%2E+Carroll%2C+%26+G%2E+Satta+%28Eds%2E%29%2C+New+developments+in+parsing+technology%2E+Dordr)
9. Oepen, S., Dyvik, H., Lønning, J. T., Velldal, E., Beermann, D., Carroll, J., Flickinger, D., Hellan, L., Johannessen, J. B., Meurer, P., Nordgård, T., & Rosén, V. (2004). Som å kapp-ete med trollet? Towards MRS-based Norwegian – English Machine Translation. In Proceedings of the 10th Internation… [find](https://scholar.google.com/scholar?q=Oepen%2C+S%2E%2C+Dyvik%2C+H%2E%2C+L%C3%B8nning%2C+J%2E+T%2E%2C+Velldal%2C+E%2E%2C+Beermann%2C+D%2E%2C+Carroll%2C+J%2E%2C+Flickinger%2C+D%2E%2C+Hellan%2C+L%2E%2C+Johannessen%2C+J%2E+B%2E%2C+Meurer%2C+P%2E%2C+Nordga%CC%8Ard%2C+T%2E%2C+%26+Rose%CC%81n%2C+V%2E+%282004%29%2E+Som+a%CC%8A+kapp%2Dete+med+trolle)
10. Oepen, S., & Flickinger, D. P. (1998). Towards systematic grammar profiling. Test suite technology ten years after. Journal of Computer Speech and Language, 12 (4) (Special Issue on Evaluation), 411 – 436. [find](https://scholar.google.com/scholar?q=Oepen%2C+S%2E%2C+%26+Flickinger%2C+D%2E+P%2E+%281998%29%2E+Towards+systematic+grammar+profiling%2E+Test+suite+technology+ten+years+after%2E+Journal+of+Computer+Speech+and+Language%2C+12+%284%29+%28Special+Issue+on+Evaluation%29%2C+411+%E2%80%93)
11. Oepen, S., Netter, K., & Klein, J. (1997). tsnlp — Test Suites for Natural Language Processing. In J. Nerbonne (Ed.), Linguistic Databases (pp. 13 – 36). Stanford, CA: CSLI Publications. [find](https://scholar.google.com/scholar?q=Oepen%2C+S%2E%2C+Netter%2C+K%2E%2C+%26+Klein%2C+J%2E+%281997%29%2E+tsnlp+%E2%80%94+Test+Suites+for+Natural+Language+Processing%2E+In+J%2E+Nerbonne+%28Ed%2E%29%2C+Linguistic+Databases+%28pp%2E+13+%E2%80%93+36%29%2E+Stanford%2C+CA%3A+CSLI+Publications%2E)
12. Oepen, S., Toutanova, K., Shieber, S., Manning, C., Flickinger, D., & Brants, T. (2002). The LinGO Redwoods treebank. Motivation and preliminary applications. In Proceedings of the 19th International Conference on Computational Linguistics. Taipei, Taiwan. [our card](https://md.research.elamur.ai/free/en/papers/oepen-2002-lingo-redwoods-treebank-motivation-preliminary/)
13. Sågvall Hein, A., Forsbom, E., Weijnitz, P., Gustavii, E., & Tiedemann, J. (2003). MATS. A glass box machine translation system. In Proceedings of the 9th Machine Translation Summit (pp. 24 – ). New Orleans, LO. [find](https://scholar.google.com/scholar?q=Sa%CC%8Agvall+Hein%2C+A%2E%2C+Forsbom%2C+E%2E%2C+Weijnitz%2C+P%2E%2C+Gustavii%2C+E%2E%2C+%26+Tiedemann%2C+J%2E+%282003%29%2E+MATS%2E+A+glass+box+machine+translation+system%2E+In+Proceedings+of+the+9th+Machine+Translation+Summit+%28pp%2E+24+%E2%80%93+%29%2E+New+)
14. Wahlster, W. (Ed.). (2000). Verbmobil. Foundations of speech-to-speech translation. Berlin, Germany: Springer. [find](https://scholar.google.com/scholar?q=Wahlster%2C+W%2E+%28Ed%2E%29%2E+%282000%29%2E+Verbmobil%2E+Foundations+of+speech%2Dto%2Dspeech+translation%2E+Berlin%2C+Germany%3A+Springer%2E)
15. Wilson, P. R., Johnstone, M. S., Neely, M., & Boles, D. (1995). Dynamic storage allocation: A survey and critical review. In Proceedings of the International Workshop on Memory Management. Kinross, Scotland: Springer. [find](https://scholar.google.com/scholar?q=Wilson%2C+P%2E+R%2E%2C+Johnstone%2C+M%2E+S%2E%2C+Neely%2C+M%2E%2C+%26+Boles%2C+D%2E+%281995%29%2E+Dynamic+storage+allocation%3A+A+survey+and+critical+review%2E+In+Proceedings+of+the+International+Workshop+on+Memory+Management%2E+Kinross%2C+Sc)