--- title: "Overview of the IWSLT 2017 Evaluation — Cettolo 2017" description: "This report describes the 2017 shared evaluation campaign of the International Workshop on Spoken Language Translation (IWSLT)." authors: ["Mauro Cettolo", "Marcello Federico", "Luisa Bentivogli", "Jan Niehues", "Sebastian Stüker", "Katsuhito Sudoh", "Koichiro Yoshino", "Christian Federmann"] year: 2017 venue: "Proceedings of the 14th International Conference on Spoken Language Translation" source: "https://aclanthology.org/2017.iwslt-1.1/" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/cettolo-2017-overview-iwslt-2017-evaluation-campaign/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/cettolo-2017-overview-iwslt-2017-evaluation-campaign/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/page/2/) / Overview of the IWSLT 2017 Evaluation Campaign # Overview of the IWSLT 2017 Evaluation Campaign **Authors:** [Mauro Cettolo](https://md.research.elamur.ai/free/en/authors/mauro-cettolo/), [Marcello Federico](https://md.research.elamur.ai/free/en/authors/marcello-federico/), [Luisa Bentivogli](https://md.research.elamur.ai/free/en/authors/luisa-bentivogli/), [Jan Niehues](https://md.research.elamur.ai/free/en/authors/jan-niehues/), [Sebastian Stüker](https://md.research.elamur.ai/free/en/authors/sebastian-stuker/), [Katsuhito Sudoh](https://md.research.elamur.ai/free/en/authors/katsuhito-sudoh/), Koichiro Yoshino, [Christian Federmann](https://md.research.elamur.ai/free/en/authors/christian-federmann/) · **Year:** 2017 · **Venue:** Proceedings of the 14th International Conference on Spoken Language Translation · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2017.iwslt-1.1/) · [PDF](https://aclanthology.org/2017.iwslt-1.1.pdf) · [OpenAlex](https://openalex.org/W6950163006) **Citations:** 0 citations (openalex-snapshot · 2026-10-14) · [148](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary This report describes the 2017 shared evaluation campaign of the International Workshop on Spoken Language Translation (IWSLT). The organisers fixed the training data, test conditions and metrics ahead of time so that systems could be compared fairly. There were three tasks: - **Multilingual task:** a single engine translates TED Talks among Dutch, English, German, Italian and Romanian. This includes zero-shot directions, where the language pair never appears together in the training data. - **Dialogue task:** translating Japanese–English conversations, using context to handle pronouns and other omitted elements. - **Lecture task:** recognising and translating spontaneous university lectures rather than polished talks. Outputs were scored automatically with BLEU, NIST and TER through an evaluation server, and the multilingual task also received human assessment. The main finding is that zero-shot systems can come close to systems that saw the tested pair during training. Multilingual training also appears to reduce data sparsity almost as well as adding large out-of-domain corpora. The organisers also report a noticeable drop in participation and plan to discuss its causes with the community. ## Key points - **Multilingual setup:** 20 directions in total. Zero-shot pairs were Dutch↔German and Italian↔Romanian, trained only on the other 16 pairs. Pivoting and synthetic training data were not allowed. In-domain data came from WIT 3, averaging about 3.99M tokens and 160k sentences per pair, with tst2017 as the test set. - **Submissions:** 7 teams took part overall (FBK, GTCT, KIT, KYOTO, RWTH, UEDIN, UDS-DFKI). The multilingual task received 9 primary runs from 5 sites. - **Zero-shot results (average BLEU):** KYOTO's zero-shot system scored 20.55, against 20.60 for its small-data system. KIT reached 21.13 with large data and 20.97 with small data. - **Dialogue corpus:** the development set has 1,476 utterances and the test set 1,510. Words were tokenised with KyTea for Japanese and the Moses tokenizer for English. - **Lecture task:** - KIT's ASR achieved an overall WER of 8.5 on English and 22.8 on German. - For German→English translation, UEDIN scored 13.99 BLEU and KIT 12.50. - For English→German translation, KIT scored 18.59 BLEU and UEDIN 15.98. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) ## Cited by Cited by 148 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 135 not yet. - [A unified approach to sentence segmentation of punctuated text in many languages](https://md.research.elamur.ai/free/en/papers/acl-2021-wicks-unified-approach-sentence-segmentation-punctuated/) - [FINDINGS OF THE IWSLT 2023 EVALUATION CAMPAIGN](https://md.research.elamur.ai/free/en/papers/agarwal-2023-findings-iwslt-2023-evaluation-campaign/) - [Massively Multilingual Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/aharoni-2019-massively-multilingual-neural-machine-translation/) - [Findings of the 2021 Conference on Machine Translation (WMT21)](https://md.research.elamur.ai/free/en/papers/akhbardeh-2021-findings-2021-conference-machine-translation/) - [Findings of the IWSLT 2022 Evaluation Campaign](https://md.research.elamur.ai/free/en/papers/anastasopoulos-2022-findings-iwslt-2022-evaluation-campaign/) - [Automatic Evaluation and Analysis of Idioms in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/baziotis-2023-automatic-evaluation-analysis-idioms-neural/) - [NTREX-128 – News Test References for MT Evaluation of 128 Languages](https://md.research.elamur.ai/free/en/papers/federmann-2022-ntrex-128-news-test-references/) - [Achieving Human Parity on Automatic Chinese to English News Translation](https://md.research.elamur.ai/free/en/papers/hassan-2018-achieving-human-parity-automatic-chinese/) - [DistiLLM: Towards Streamlined Distillation for Large Language Models](https://md.research.elamur.ai/free/en/papers/ko-2024-distillm-towards-streamlined-distillation-large/) - [Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation](https://md.research.elamur.ai/free/en/papers/laubli-2018-has-machine-translation-achieved-human/) - [A Call for Clarity in Reporting BLEU Scores](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/) - [Multilingual Translation from Denoising Pre-Training](https://md.research.elamur.ai/free/en/papers/tang-2021-multilingual-translation-denoising-pre-training/) - [PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts](https://md.research.elamur.ai/free/en/papers/zhu-2023-promptrobust-towards-evaluating-robustness-large/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{cettolo2017overview, title = {{Overview of the IWSLT 2017 Evaluation Campaign}}, author = {Cettolo, Mauro and Federico, Marcello and Bentivogli, Luisa and Niehues, Jan and Stüker, Sebastian and Sudoh, Katsuhito and Yoshino, Koichiro and Federmann, Christian}, booktitle = {Proceedings of the 14th International Conference on Spoken Language Translation}, year = {2017}, doi = {10.5445/ir/1000166204}, url = {https://doi.org/10.5445/ir/1000166204} } ``` - [← previous Out-of-domain FrameNet Semantic Role Labeling](https://md.research.elamur.ai/free/en/papers/hartmann-2017-out-domain-framenet-semantic-role/) - [next → Paraphrasing Revisited with Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/mallinson-2017-paraphrasing-revisited-neural-machine-translation/) - [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/page/2/) ## References 4 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. N. Ruiz and M. Federico, “Complexity of spoken versus written language for machine translation,” in Proc. of EAMT, Dubrovnik, Croatia, 2014, pp. 173–180. [find](https://scholar.google.com/scholar?q=N%2E+Ruiz+and+M%2E+Federico%2C+%E2%80%9CComplexity+of+spoken+versus+written+language+for+machine+translation%2C%E2%80%9D+in+Proc%2E+of+EAMT%2C+Dubrovnik%2C+Croatia%2C+2014%2C+pp%2E+173%E2%80%93180%2E) 2. S. M. Lakew, Q. F. Lotito, M. Turchi, M. Negri, and M. Federico, “FBK’s multilingual neural machine translation system for IWSLT 2017,” in Proc. of IWSLT, Tokyo, Japan, 2017. [find](https://scholar.google.com/scholar?q=S%2E+M%2E+Lakew%2C+Q%2E+F%2E+Lotito%2C+M%2E+Turchi%2C+M%2E+Negri%2C+and+M%2E+Federico%2C+%E2%80%9CFBK%E2%80%99s+multilingual+neural+machine+translation+system+for+IWSLT+2017%2C%E2%80%9D+in+Proc%2E+of+IWSLT%2C+Tokyo%2C+Japan%2C+2017%2E) 3. C. Bei and H. Zong, “Towards better translation performance on spoken language,” in Proc. of IWSLT, Tokyo, Japan, 2017. [find](https://scholar.google.com/scholar?q=C%2E+Bei+and+H%2E+Zong%2C+%E2%80%9CTowards+better+translation+performance+on+spoken+language%2C%E2%80%9D+in+Proc%2E+of+IWSLT%2C+Tokyo%2C+Japan%2C+2017%2E) 4. N.-Q. Pham, M. Sperber, E. Salesky, T.-L. Ha, J. Niehues, and A. Waibel, “KIT’s multilingual neural machine translation systems for IWSLT 2017,” in Proc. of IWSLT, Tokyo, Japan, 2017. \#\#\# 5.3. Future Work [find](https://scholar.google.com/scholar?q=N%2E%2DQ%2E+Pham%2C+M%2E+Sperber%2C+E%2E+Salesky%2C+T%2E%2DL%2E+Ha%2C+J%2E+Niehues%2C+and+A%2E+Waibel%2C+%E2%80%9CKIT%E2%80%99s+multilingual+neural+machine+translation+systems+for+IWSLT+2017%2C%E2%80%9D+in+Proc%2E+of+IWSLT%2C+Tokyo%2C+Japan%2C+2017%2E+%5C%23%5C%23%5C%23+5%2E3%2E+Futu)