---
title: "Formal v. Informal — paper by Miller and Vanni (2006)"
description: "The paper asks whether a set of fine-grained linguistic quality measures can show how suitable machine translation output is… 25 references."
authors: ["Keith J. Miller", "Michelle Vanni"]
year: 2006
venue: "Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC'06)"
source: "https://aclanthology.org/L06-1454/"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/miller-2006-formal-v-informal-register-differentiated/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/miller-2006-formal-v-informal-register-differentiated/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2006](https://md.research.elamur.ai/free/en/papers/2006/page/2/) / Formal v. Informal: Register-Differentiated Arabic MT Evaluation in the PLATO Paradigm

# Formal v. Informal: Register-Differentiated Arabic MT Evaluation in the PLATO Paradigm

**Authors:** Keith J. Miller and Michelle Vanni · **Year:** 2006 · **Venue:** [Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC'06)](https://md.research.elamur.ai/free/en/venues/lrec/) ([LREC 2006](https://md.research.elamur.ai/free/en/venues/lrec/2006/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/L06-1454/) · [PDF](https://aclanthology.org/L06-1454.pdf) · [OpenAlex](https://openalex.org/W2756988793)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
The paper asks whether a set of fine-grained linguistic quality measures can show how suitable machine translation output is for particular analyst tasks, and whether that suitability depends on the register of the source text. Within the PLATO program, the authors compared five Arabic-English MT engines on 10 input texts. Half were formal Modern Standard Arabic newswire. The other half were informal web writing, a mixture of dialect and genre features drawn from discussion lists, weblogs and online editorials. Ten assessors applied eight tests covering clarity, coherence, morphology, syntax, untranslated words, name rendering, domain terminology and adequacy, which produced 800 score sets. The engines handled vocabulary coverage about equally well on both registers. Their scores split apart on clarity, name rendering and domain terms. The authors link each test to a downstream task. On that basis, the systems look usable for triage on either kind of input, but relevance judgment, information extraction and topic identification need register taken into account. The study also shows that per-dimension score profiles can tell apart text a system was presumably tuned for from text it was not.

## Key points
- Setup: 5 Arabic-English MT systems, 10 source texts in two registers (formal MSA newswire vs. informal web text), 8 tests and 10 assessors, giving 800 data sets.
- Clarity is judged without any source or reference text, on a 0–4 scale. Adequacy follows the 1994 DARPA test and compares output with a reference on a 1–5 scale.
- Several scores are computed as one minus an error ratio:
  - Morphology: word-formation fixes divided by the number of inflectable words.
  - Syntax: minimal edits needed to reach grammaticality, divided by sentence length (similar in spirit to TER).
  - Coherence: a binary judgment of whether some RST function can plausibly be assigned to the sentence in context.
- Dictionary Update, Name Rendering and Domain Terminology are checked against a human reference using exact match, which keeps rater agreement near maximal and makes automation easier.
- Untranslated-word rates were similar across registers. Clarity, name rendering and domain terms diverged. Earlier refinement had already brought Clarity inter-rater reliability to about .8.

## Interesting ideas
- The evaluation output is a profile of separate linguistic scores, a "signature", and each score is tied to the downstream task it predicts (lexical coverage for triage, names for extraction). A single number would hide which uses the output is fit for.
- Test sets are split by register so that each system faces input close to its training data and input far from it. The gap between the two profiles then exposes brittleness that a pooled average would blur.
- Raters give a quick reference-free clarity judgment first and only later run the analytic tests, an ordering meant to prevent bias and training effects. Fixing the order of human judgments protects the cheap intuitive signal from being shaped by the detailed ones.
- Coherence is reduced to a yes/no question: can any RST relation reasonably link this sentence to its context? This turns a vague discourse notion into a constrained, checkable decision.
- Name and term accuracy is scored by exact match against a list extracted from a human reference. The authors chose this to make scoring deterministic and easy to automate.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Mova's evaluation could report separate scores for tokens, tree, frames and world, each mapped to the reader tasks that depend on it, so a regression points to the specific capability it breaks.
- **I2** (tree): Parser accuracy could be reported separately for edited prose and informal web or forum text, so we can see where the UD dialect or training data stops generalizing.
- **I3** (back): When people rate generated or round-tripped text, a fluency-only judgment could be collected before they see the source graph, keeping the readability signal independent of the fidelity check.
- **I4** (frames): The same test could flag sentences whose frames attach to nothing in the preceding discourse, giving an explainable sign of a failed coreference or relation analysis.
- **I5** (world): Entity lists built from gold annotations could be matched exactly against the entities in Mova's text world, giving a cheap, unambiguous check on entity extraction and naming.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Mitre](https://md.research.elamur.ai/free/en/companies/mitre/) — Company, Bedford, United States · 10 works · 2,631 citations
- [DEVCOM Army Research Laboratory](https://md.research.elamur.ai/free/en/labs/devcom-army-research-laboratory/) — Research institute, Adelphi, United States · 14 works · 2,292 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🔁 Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · 1,087 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works

## Cite

Cite as: Keith J. Miller and Michelle Vanni. 2006. Formal v. Informal: Register-Differentiated Arabic MT Evaluation in the PLATO Paradigm. Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC'06).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{miller2006formal,
  title = {{Formal v. Informal: Register-Differentiated Arabic MT Evaluation in the PLATO Paradigm}},
  author = {Keith J. Miller and Michelle Vanni},
  booktitle = {Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC'06)},
  year = {2006},
  doi = {10.63317/5m5ph6wf35d7},
  url = {https://doi.org/10.63317/5m5ph6wf35d7}
}
```

- [← previous Dypedahl, Magne, Hilde Hasselgård and Berit Løken. 2006. Introducing English Grammar. Bergen: Fagbok-forlaget. pp. 165.](https://md.research.elamur.ai/free/en/papers/fredriksson-2006-dypedahl-magne-hilde-hasselgard-berit/)
- [next → Identifying Complex Phenomena in a Corpus via a Treebank Lens](https://md.research.elamur.ai/free/en/papers/flickinger-2006-identifying-complex-phenomena-corpus-treebank/)
- [↑ back to papers of 2006](https://md.research.elamur.ai/free/en/papers/2006/page/2/)

## References

25 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Aikmajian, A., R. A. Demers and R. M. Harnish. (1988). Linguistics: An Introduction to Language and Communication, Second Edition. Cambridge, MA: MIT Press. 521. [find](https://scholar.google.com/scholar?q=Aikmajian%2C+A%2E%2C+R%2E+A%2E+Demers+and+R%2E+M%2E+Harnish%2E+%281988%29%2E+Linguistics%3A+An+Introduction+to+Language+and+Communication%2C+Second+Edition%2E+Cambridge%2C+MA%3A+MIT+Press%2E+521%2E)
2. Church, K. and E. Hovy. (1993). Good Applications for Crummy Machine Translation. Machine Translation 8: 239-258. [↗](https://doi.org/10.1007/bf00981759)
3. Crystal, D. (2001). Language and the Internet. Cambridge: CUP. [↗](https://doi.org/10.1017/cbo9781139164771)
4. Freeman, A. (2002). In Search of a koine in Sana’a. Unpublished Ph.D. Dissertation. Departments of Linguistics and Near Eastern Studies. University of Michigan. [find](https://scholar.google.com/scholar?q=Freeman%2C+A%2E+%282002%29%2E+In+Search+of+a+koine+in+Sana%E2%80%99a%2E+Unpublished+Ph%2ED%2E+Dissertation%2E+Departments+of+Linguistics+and+Near+Eastern+Studies%2E+University+of+Michigan%2E)
5. Henderson, J. (2004). The Text Interpretability Rating Scale. Presentation at Panel on Trends in MT Evaluation. Conference of the Association for Machine Translation in the Americas (AMTA) Washington, DC. [find](https://scholar.google.com/scholar?q=Henderson%2C+J%2E+%282004%29%2E+The+Text+Interpretability+Rating+Scale%2E+Presentation+at+Panel+on+Trends+in+MT+Evaluation%2E+Conference+of+the+Association+for+Machine+Translation+in+the+Americas+%28AMTA%29+Washington%2C)
6. Hovy, E., M. King and A. Popescu-Belis. (2003). Principles of Context-Based Machine Translation Evaluation. Machine Translation 17:1. 43–75. [↗](https://doi.org/10.1023/a:1025510524115)
7. International Standards for Language Engineering. (2000). ISLE Classification MT Evaluations. [http://www.isi.edu/natural-language/mteval](http://www.isi.edu/natural-language/mteval). In Proceedings of Hands-on Workshop on Machine Translation Evaluation. Association for Machine Translation in the Americas, Cuernavaca, Mexico. [↗](http://www.isi.edu/natural-language/mteval)
8. Jones, D. and G. Rusk. 2000. Toward a Scoring Function for Quality-Driven Machine Translation. In Proceedings of COLING-2000. [find](https://scholar.google.com/scholar?q=Jones%2C+D%2E+and+G%2E+Rusk%2E+2000%2E+Toward+a+Scoring+Function+for+Quality%2DDriven+Machine+Translation%2E+In+Proceedings+of+COLING%2D2000%2E)
9. Lavie, L., K. Sagae and S. Jayaraman. (2004). The Significance of Recall in Automatic Metrics for MT Evaluation. In Machine Translation: From Real Users to Research, Proceedings of the Sixth Conference of the Association for Machine Translation in the Americas (AMTA-2004), Washington, D.C. [find](https://scholar.google.com/scholar?q=Lavie%2C+L%2E%2C+K%2E+Sagae+and+S%2E+Jayaraman%2E+%282004%29%2E+The+Significance+of+Recall+in+Automatic+Metrics+for+MT+Evaluation%2E+In+Machine+Translation%3A+From+Real+Users+to+Research%2C+Proceedings+of+the+Sixth+Conferenc)
10. Lyons, J. (1981). Language and Linguistics: An Introduction. Cambridge:CUP. 292. [find](https://scholar.google.com/scholar?q=Lyons%2C+J%2E+%281981%29%2E+Language+and+Linguistics%3A+An+Introduction%2E+Cambridge%3ACUP%2E+292%2E)
11. Mann, W.C. and S.A. Thompson. (1988). [Rhetorical Structure Theory: Toward a functional theory of text organization](https://md.research.elamur.ai/free/en/papers/mann-1988-rhetorical-structure-theory-functional/). Text 8:3. pp. 243-281. [↗](https://doi.org/10.1515/text.1.1988.8.3.243)
12. Melamed, I, R. Green, and J. Turian (2003) [Precision and Recall of Machine Translation](https://md.research.elamur.ai/free/en/papers/melamed-2003-precision-recall-machine-translation/). In Proceedings on NAACL/HLT, Edmonton, Canada.
13. Miller, K.J. and M. Vanni (2002). Scaling the ISLE Taxonomy: Development of Metrics for Multi-Dimensional Characterization of Machine Translation Quality. In Proceedings of MT Summit VIII. Santiago de Campostela, Spain. [find](https://scholar.google.com/scholar?q=Miller%2C+K%2EJ%2E+and+M%2E+Vanni+%282002%29%2E+Scaling+the+ISLE+Taxonomy%3A+Development+of+Metrics+for+Multi%2DDimensional+Characterization+of+Machine+Translation+Quality%2E+In+Proceedings+of+MT+Summit+VIII%2E+Santiago+de)
14. Miller, K. J. and M. Vanni. (2005). Inter-rater Agreement Measures and the Refinement of Metrics in the PLATO MT Evaluation Paradigm. In Proceedings of MT Summit X, Phuket, Thailand. [↗](https://doi.org/10.21236/ada456393)
15. Papineni, K., S. Roukos, T. Ward and W. Zhu. (2001). [BLEU: A Method for Automatic Evaluation of Machine Translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/), IBM Research Report, RC22176, September 2001.
16. Papineni, K., S. Roukos, T. Ward, J. Henderson, and F. Reeder. (2002). Corpus-Based Comprehensive and Diagnostic MT Evaluation: Initial Arabic, Chinese, French, and Spanish Results. Proceedings of the Human Language Technology Conference. [find](https://scholar.google.com/scholar?q=Papineni%2C+K%2E%2C+S%2E+Roukos%2C+T%2E+Ward%2C+J%2E+Henderson%2C+and+F%2E+Reeder%2E+%282002%29%2E+Corpus%2DBased+Comprehensive+and+Diagnostic+MT+Evaluation%3A+Initial+Arabic%2C+Chinese%2C+French%2C+and+Spanish+Results%2E+Proceedings+of+the)
17. Ryding, K.C. and D.J. Mehall. (2005). Formal Spoken Arabic. Washington, DC: Georgetown University Press. [find](https://scholar.google.com/scholar?q=Ryding%2C+K%2EC%2E+and+D%2EJ%2E+Mehall%2E+%282005%29%2E+Formal+Spoken+Arabic%2E+Washington%2C+DC%3A+Georgetown+University+Press%2E)
18. Snover, M., B.J. Dorr, R. Schwartz, J Makhoul, L. Micciula and R. Weischedel. (2005). A Study of Translation Error Rate with Targeted Human Annotation. Technical Report LAMP-TR-126, CS-TR-4755, UMIACS-TR-2005-58, University of Maryland, College Park. [find](https://scholar.google.com/scholar?q=Snover%2C+M%2E%2C+B%2EJ%2E+Dorr%2C+R%2E+Schwartz%2C+J+Makhoul%2C+L%2E+Micciula+and+R%2E+Weischedel%2E+%282005%29%2E+A+Study+of+Translation+Error+Rate+with+Targeted+Human+Annotation%2E+Technical+Report+LAMP%2DTR%2D126%2C+CS%2DTR%2D4755%2C+UMIACS)
19. Tate, C., C. Voss, B. J. Dorr and E. Slud. (2005). Toward a Predictive Statistical Model of Task-based Performance Using Automatic MT Evaluation Metrics. In Proceedings of the Association for Computational Linguistics Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization. Ann Arbor, MI. [find](https://scholar.google.com/scholar?q=Tate%2C+C%2E%2C+C%2E+Voss%2C+B%2E+J%2E+Dorr+and+E%2E+Slud%2E+%282005%29%2E+Toward+a+Predictive+Statistical+Model+of+Task%2Dbased+Performance+Using+Automatic+MT+Evaluation+Metrics%2E+In+Proceedings+of+the+Association+for+Computat)
20. Taylor, K. and J. White. (1998). Predicting What MT is Good for: User Judgments and Task Performance. In: Proceedings of the 1998 conference of the Association of Machine Translation in the Americas. 364-373. [find](https://scholar.google.com/scholar?q=Taylor%2C+K%2E+and+J%2E+White%2E+%281998%29%2E+Predicting+What+MT+is+Good+for%3A+User+Judgments+and+Task+Performance%2E+In%3A+Proceedings+of+the+1998+conference+of+the+Association+of+Machine+Translation+in+the+Americas%2E+)
21. Vanni, M. (1998). Evaluating MT Systems: Testing the Feasibility of a Task-Diagnostic. Proceedings of the Association for Information Management (ASLIB): Translating and the Computer 20. [find](https://scholar.google.com/scholar?q=Vanni%2C+M%2E+%281998%29%2E+Evaluating+MT+Systems%3A+Testing+the+Feasibility+of+a+Task%2DDiagnostic%2E+Proceedings+of+the+Association+for+Information+Management+%28ASLIB%29%3A+Translating+and+the+Computer+20%2E)
22. Vanni, M. & Miller, K.J. (2002). Scaling the ISLE Framework: Use of Existing Corpus Resources for Validation of MT Metrics across Languages. In: Proceedings of the Third International Conference on Language Resources and Evaluation (LREC). Las Palmas de Gran Canaria, Spain. 1254-1262. [find](https://scholar.google.com/scholar?q=Vanni%2C+M%2E+%26+Miller%2C+K%2EJ%2E+%282002%29%2E+Scaling+the+ISLE+Framework%3A+Use+of+Existing+Corpus+Resources+for+Validation+of+MT+Metrics+across+Languages%2E+In%3A+Proceedings+of+the+Third+International+Conference+on+La)
23. "Genre." Merriam-Webster On-Line Dictionary. (2006). [http://www.m-w.com/dictionary/genre](http://www.m-w.com/dictionary/genre). (27 Feb 2006). [↗](http://www.m-w.com/dictionary/genre)
24. Weinberg, A. (2005). Machine Translation Evaluation at the Center for Advanced Study of Language. Workshop on Intelligence Analysis, Mitre Corporation. [find](https://scholar.google.com/scholar?q=Weinberg%2C+A%2E+%282005%29%2E+Machine+Translation+Evaluation+at+the+Center+for+Advanced+Study+of+Language%2E+Workshop+on+Intelligence+Analysis%2C+Mitre+Corporation%2E)
25. White, J. and O’Connell T.A. (1996). [The ARPA MT evaluation methodologies: evolution, lessons and future approaches](https://md.research.elamur.ai/free/en/papers/white-1994-arpa-mt-evaluation-methodologies-evolution/). Proceedings of the 1994 Conference, of the Association for Machine Translation in the Americas.

*Updated 1 November 2026*