--- title: "An Evaluation Exercise for Word Alignment — Mihalcea 2003" description: "This paper reports on a shared word alignment task run as part of a 2003 HLT/NAACL workshop dedicated to creating and applying parallel texts." authors: ["Rada Mihalcea", "Ted Pedersen"] year: 2003 venue: "Proceedings of the HLT-NAACL 2003 Workshop on Building and Using Parallel Texts: Data Driven Machine Translation and Beyond" source: "https://aclanthology.org/W03-0301/" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/papers/mihalcea-2003-evaluation-exercise-word-alignment/" --- [Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/mihalcea-2003-evaluation-exercise-word-alignment/) [home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2003](https://md.research.elamur.ai/free/papers/2003/) / An Evaluation Exercise for Word Alignment # An Evaluation Exercise for Word Alignment **Authors:** [Rada Mihalcea](https://md.research.elamur.ai/free/authors/rada-mihalcea/), Ted Pedersen · **Year:** 2003 · **Venue:** Proceedings of the HLT-NAACL 2003 Workshop on Building and Using Parallel Texts: Data Driven Machine Translation and Beyond · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/W03-0301/) · [PDF](https://aclanthology.org/W03-0301.pdf) · [OpenAlex](https://openalex.org/W2141532438) **Citations:** 207 citations (openalex-snapshot · 2026-10-14) · [119](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary This paper reports on a shared word alignment task run as part of a 2003 HLT/NAACL workshop dedicated to creating and applying parallel texts. It covers how the task was set up, the data, the participating systems and the comparative results. Teams received sentence-aligned training corpora for two language pairs and had to link every token of a test set to its counterpart in the other language. - **English-French** stood for a resource-rich pair, with roughly 20M words of Canadian Hansards. - **Romanian-English** stood for a resource-poor pair, with about 1M words. Following the NIST MT evaluations, the organizers split participation into Limited and Unlimited Resources tracks. They also defined a shared output format with Sure/Probable labels and confidence scores, and reused annotation conventions from the Blinker project. Submissions were scored with precision, recall, F-measure and AER, both with and without NULL links. Seven teams sent 27 runs. Systems did clearly better on English-French than on Romanian-English, which supports the importance of training data size. Systems restricted to the provided data often matched or beat those that drew on extra resources. The organizers released the data and scoring tools, making this an early public benchmark for word alignment. ## Key points - **Training data:** - Romanian-English: about 850K/900K words of web-mined newspaper text, plus Orwell's 1984 from MULTEXT-EAST (~130K words) and the Romanian Constitution (~13K words). - English-French: a ~20M-word Hansards subset prepared by Ulrich Germann. - Tokenization used EGYPT Toolkit tokenizers for English and French and the organizers' own tokenizer for Romanian. - **Trial and test data:** trial sets had 37 English-French and 17 Romanian-English sentences. Test sets had 447 English-French sentences (gold alignments by Och and Ney) and 248 Romanian-English sentences. - **Participation:** seven teams took part: BiBr, Fourday, Ralign, RACAI, ProAlign, UMD and XRCE. They submitted 27 runs in total: - 14 English-French and 13 Romanian-English; - 17 Limited and 10 Unlimited. - Four systems were built on IBM Models 1, 2 or 4. - ProAlign and BiBr used syntactic constraints. - **Best systems:** XRCE (GIZA++-based) led Romanian-English Limited, ProAlign led both Unlimited tracks, and Ralign led English-French Limited. - **Best scores on Sure alignments:** | Language pair | F-measure | AER | |---|---|---| | English-French | 80.54% | about 5.71% | | Romanian-English | 71.14% | about 28.86% | Across all evaluation figures, Limited-resource systems took 20 of the 28 top spots. ## Interesting ideas - **Separate Limited and Unlimited tracks.** This separates algorithmic improvements from the benefit of extra dictionaries or corpora. It is useful for any benchmark in a low-resource language, where outside resources can otherwise blur comparisons. - **Two-level Sure/Probable labels plus an optional confidence score.** This lets each system decide for itself where a "certain" link ends and a "possible" one begins. It also allows ambiguous gold annotations without forcing a hard choice. - **Scoring with and without NULL alignments.** Doing both shows whether a system's strength comes from correctly leaving untranslated words unlinked or from linking real word pairs. This dual-view scoring carries over to other tasks with an "empty" class. - **Gold-standard construction affects comparability.** The English-French gold set used an intersection/union of two annotators, while the Romanian-English set used third-annotator arbitration. Because of this, the authors chose not to compare Probable scores across the two pairs. Recording how gold labels were built is a cheap safeguard against misleading cross-dataset claims. - **Phrase correspondences as all-pairs links.** Representing a phrase match by linking every word on one side to every word on the other keeps a simple token-level format while still capturing non-compositional translations. The cost is that such links count heavily in the scores. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/) ## Cited by Cited by 119 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 112 not yet. - [Advancements in Arabic Grammatical Error Detection and Correction: An Empirical Investigation](https://md.research.elamur.ai/free/papers/alhafni-2023-advancements-arabic-grammatical-error-detection/) - [Constructing Corpora for the Development and Evaluation of Paraphrase Systems](https://md.research.elamur.ai/free/papers/cohn-2008-constructing-corpora-development-evaluation-paraphrase/) - [Unsupervised Construction of Large Paraphrase Corpora: Exploiting Massively Parallel News Sources](https://md.research.elamur.ai/free/papers/dolan-2004-unsupervised-construction-large-paraphrase-corpora/) - [Better Word Alignments with Supervised ITG Models](https://md.research.elamur.ai/free/papers/haghighi-2009-better-word-alignments-supervised-itg/) - [Binarization of Synchronous Context-Free Grammars](https://md.research.elamur.ai/free/papers/huang-2009-binarization-synchronous-context-free-grammars/) - [HELFI: a Hebrew-Greek-Finnish Parallel Bible Corpus with Cross-Lingual Morpheme Alignment](https://md.research.elamur.ai/free/papers/lingbulk-2020-jyra-helfi-hebrew-greek-finnish-parallel/) - [Monolingual Machine Translation for Paraphrase Generation](https://md.research.elamur.ai/free/papers/quirk-2004-monolingual-machine-translation-paraphrase-generation/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{mihalcea2003evaluation, title = {{An Evaluation Exercise for Word Alignment}}, author = {Mihalcea, Rada and Pedersen, Ted}, booktitle = {Proceedings of the HLT-NAACL 2003 Workshop on Building and Using Parallel Texts: Data Driven Machine Translation and Beyond}, year = {2003}, doi = {10.3115/1118905.1118906}, url = {https://doi.org/10.3115/1118905.1118906} } ``` - [← previous An Empirical Model of Multiword Expression Decomposability](https://md.research.elamur.ai/free/papers/baldwin-2003-empirical-model-multiword-expression-decomposability/) - [next → An Expert Lexicon Approach to Identifying English Phrasal Verbs](https://md.research.elamur.ai/free/papers/li-2003-expert-lexicon-approach-identifying-english/) - [↑ back to papers of 2003](https://md.research.elamur.ai/free/papers/2003/) ## References 15 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. M. Banko and E. Brill. 2001. Scaling to very very large corpora for natural language disambiguation. In Proceedings of the 39th Annual Meeting of the Association for Computational Lingusitics (ACL-2001), Toulouse, France, July. [find](https://scholar.google.com/scholar?q=M%2E+Banko+and+E%2E+Brill%2E+2001%2E+Scaling+to+very+very+large+corpora+for+natural+language+disambiguation%2E+In+Proceedings+of+the+39th+Annual+Meeting+of+the+Association+for+Computational+Lingusitics+%28ACL%2D200) 2. P. Brown, S. Della Pietra, V. Della Pietra, and R. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2):263–311. [find](https://scholar.google.com/scholar?q=P%2E+Brown%2C+S%2E+Della+Pietra%2C+V%2E+Della+Pietra%2C+and+R%2E+Mercer%2E+1993%2E+The+mathematics+of+statistical+machine+translation%3A+Parameter+estimation%2E+Computational+Linguistics%2C+19%282%29%3A263%E2%80%93311%2E) 3. Herve Dejean, Eric Gaussier, Cyril Goutte, and Kenji Yamada. 2003. Reducing parameter space for word alignment. In HLT-NAACL 2003 Workshop: Building and Using Parallel Texts: Data Driven Machine Translation and Beyond, pages 23–26, Edmonton, Alberta, Canada, May 31. Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Herve+Dejean%2C+Eric+Gaussier%2C+Cyril+Goutte%2C+and+Kenji+Yamada%2E+2003%2E+Reducing+parameter+space+for+word+alignment%2E+In+HLT%2DNAACL+2003+Workshop%3A+Building+and+Using+Parallel+Texts%3A+Data+Driven+Machine+Trans) 4. T. Erjavec, N. Ide, and D. Tufiş. 1997. Encoding and parallel alignment of linguistic corpora in six central and Eastern European languages. In Proceedings of the Joint ACH/ALL Conference, Queen’s University, Kingston, Ontario, June. [find](https://scholar.google.com/scholar?q=T%2E+Erjavec%2C+N%2E+Ide%2C+and+D%2E+Tufis%CC%A7%2E+1997%2E+Encoding+and+parallel+alignment+of+linguistic+corpora+in+six+central+and+Eastern+European+languages%2E+In+Proceedings+of+the+Joint+ACH%2FALL+Conference%2C+Queen%E2%80%99s+Un) 5. U. Germann. 2001. Aligned hansards of the 36th parliament of canada. [http://www.isi.edu/natural-language/download/hansard/](http://www.isi.edu/natural-language/download/hansard/). [↗](http://www.isi.edu/natural-language/download/hansard/) 6. John C. Henderson. 2003. Word alignment baselines. In HLT-NAACL 2003 Workshop: Building and Using Parallel Texts: Data Driven Machine Translation and Beyond, pages 27–30, Edmonton, Alberta, Canada, May 31. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=John+C%2E+Henderson%2E+2003%2E+Word+alignment+baselines%2E+In+HLT%2DNAACL+2003+Workshop%3A+Building+and+Using+Parallel+Texts%3A+Data+Driven+Machine+Translation+and+Beyond%2C+pages+27%E2%80%9330%2C+Edmonton%2C+Alberta%2C+Canada%2C+Ma) 7. Dekang Lin and Colin Cherry. 2003. Proalign: Shared task system description. In HLT-NAACL 2003 Workshop: Building and Using Parallel Texts: Data Driven Machine Translation and Beyond, pages 11–14, Edmonton, Alberta, Canada, May 31. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Dekang+Lin+and+Colin+Cherry%2E+2003%2E+Proalign%3A+Shared+task+system+description%2E+In+HLT%2DNAACL+2003+Workshop%3A+Building+and+Using+Parallel+Texts%3A+Data+Driven+Machine+Translation+and+Beyond%2C+pages+11%E2%80%9314%2C+Edm) 8. D.I. Melamed. 1999. Empirical Methods for Exploiting Parallel Texts. MIT Press. [find](https://scholar.google.com/scholar?q=D%2EI%2E+Melamed%2E+1999%2E+Empirical+Methods+for+Exploiting+Parallel+Texts%2E+MIT+Press%2E) 9. F. Och and H. Ney. 2000. A comparison of alignment models for statistical machine translation. In Proceedings of the 18th International Conference on Computational Linguistics (COLING-ACL 2000), Saarbrucken, Germany, August. [find](https://scholar.google.com/scholar?q=F%2E+Och+and+H%2E+Ney%2E+2000%2E+A+comparison+of+alignment+models+for+statistical+machine+translation%2E+In+Proceedings+of+the+18th+International+Conference+on+Computational+Linguistics+%28COLING%2DACL+2000%29%2C+Saarb) 10. Michel Simard and Philippe Langlais. 2003. Statistical translation alignment with compositionality constraints. In HLT-NAACL 2003 Workshop: Building and Using Parallel Texts: Data Driven Machine Translation and Beyond, pages 19–22, Edmonton, Alberta, Canada, May 31. Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Michel+Simard+and+Philippe+Langlais%2E+2003%2E+Statistical+translation+alignment+with+compositionality+constraints%2E+In+HLT%2DNAACL+2003+Workshop%3A+Building+and+Using+Parallel+Texts%3A+Data+Driven+Machine+Trans) 11. Bridget Thomson McInnes and Ted Pedersen. 2003. The duluth word alignment system. In HLT-NAACL 2003 Workshop: Building and Using Parallel Texts: Data Driven Machine Translation and Beyond, pages 40– 43, Edmonton, Alberta, Canada, May 31. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Bridget+Thomson+McInnes+and+Ted+Pedersen%2E+2003%2E+The+duluth+word+alignment+system%2E+In+HLT%2DNAACL+2003+Workshop%3A+Building+and+Using+Parallel+Texts%3A+Data+Driven+Machine+Translation+and+Beyond%2C+pages+40%E2%80%93+4) 12. Dan Tufiş, Ana-Maria Barbu, and Radu Ion. 2003. Treq-al: A word alignment system with limited language resources. In HLT-NAACL 2003 Workshop: Building and Using Parallel Texts: Data Driven Machine Translation and Beyond, pages 36–39, Edmonton, Alberta, Canada, May 31. Association for Computational… [find](https://scholar.google.com/scholar?q=Dan+Tufis%CC%A7%2C+Ana%2DMaria+Barbu%2C+and+Radu+Ion%2E+2003%2E+Treq%2Dal%3A+A+word+alignment+system+with+limited+language+resources%2E+In+HLT%2DNAACL+2003+Workshop%3A+Building+and+Using+Parallel+Texts%3A+Data+Driven+Machine+Tr) 13. D Tufiş. 2002. A cheap and fast way to build useful translation lexicons. In Proceedings of the 19th International Conference on Computational Linguistics, pages 1030–1036, Taipei, August. [find](https://scholar.google.com/scholar?q=D+Tufis%CC%A7%2E+2002%2E+A+cheap+and+fast+way+to+build+useful+translation+lexicons%2E+In+Proceedings+of+the+19th+International+Conference+on+Computational+Linguistics%2C+pages+1030%E2%80%931036%2C+Taipei%2C+August%2E) 14. Bing Zhao and Stephan Vogel. 2003. Word alignment based on bilingual bracketing. In HLT-NAACL 2003 Workshop: Building and Using Parallel Texts: Data Driven Machine Translation and Beyond, pages 15– 18, Edmonton, Alberta, Canada, May 31. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Bing+Zhao+and+Stephan+Vogel%2E+2003%2E+Word+alignment+based+on+bilingual+bracketing%2E+In+HLT%2DNAACL+2003+Workshop%3A+Building+and+Using+Parallel+Texts%3A+Data+Driven+Machine+Translation+and+Beyond%2C+pages+15%E2%80%93+18) 15. System BiBr.EF.1 BiBr.EF.2 BiBr.EF.3 BiBr.EF.4 BiBr.EF.5 BiBr.EF.6 BiBr.EF.7 BiBr.EF.8 ProAlign.EF.1 Resources Limited Unlimited Unlimited Limited Unlimited Unlimited Limited Unlimited Unlimited [find](https://scholar.google.com/scholar?q=System+BiBr%2EEF%2E1+BiBr%2EEF%2E2+BiBr%2EEF%2E3+BiBr%2EEF%2E4+BiBr%2EEF%2E5+BiBr%2EEF%2E6+BiBr%2EEF%2E7+BiBr%2EEF%2E8+ProAlign%2EEF%2E1+Resources+Limited+Unlimited+Unlimited+Limited+Unlimited+Unlimited+Limited+Unlimited+Unlimited)