---
title: "Multi30K: Multilingual English-German Image — Elliott 2016"
description: "Research on automatic image captioning has relied almost entirely on English data, leaving other languages without resources for tasks such as caption…"
authors: ["Desmond Elliott", "Stella Frank", "Khalil Sima'an", "Lucia Specia"]
year: 2016
venue: "arXiv:1605.00459"
source: "https://arxiv.org/abs/1605.00459"
doi: "10.48550/arxiv.1605.00459"
arxiv: "1605.00459"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/elliott-2016-multi30k-multilingual-english-german-image/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/elliott-2016-multi30k-multilingual-english-german-image/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/2/) / Multi30K: Multilingual English-German Image Descriptions

# Multi30K: Multilingual English-German Image Descriptions

**Authors:** [Desmond Elliott](https://md.research.elamur.ai/free/en/authors/desmond-elliott/), [Stella Frank](https://md.research.elamur.ai/free/en/authors/stella-frank/), [Khalil Sima'an](https://md.research.elamur.ai/free/en/authors/khalil-sima-an/), [Lucia Specia](https://md.research.elamur.ai/free/en/authors/lucia-specia/) · **Year:** 2016 · **Venue:** arXiv:1605.00459 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1605.00459) · [PDF](https://arxiv.org/pdf/1605.00459) · [DOI 10.48550/arxiv.1605.00459](https://doi.org/10.48550/arxiv.1605.00459) · [arXiv 1605.00459](https://arxiv.org/abs/1605.00459) · [OpenAlex](https://openalex.org/W2345720230)

**Citations:** 126 citations (OpenAlex, 2026-10-14) · [257](https://md.research.elamur.ai/free/en/papers/elliott-2016-multi30k-multilingual-english-german-image/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
Research on automatic image captioning has relied almost entirely on English data, leaving other languages without resources for tasks such as caption generation or natural-language image search. Elliott, Frank, Sima'an and Specia address this gap by adding German text to the Flickr30K images, creating Multi30K. The resource has two parts with different cross-lingual relationships. In the first, professional translators converted one English caption per image into German without seeing the picture, which yields sentence-aligned pairs. In the second, German-speaking crowdworkers wrote five new captions per image using a German version of the original collection interface. These captions describe the same image as the English ones but are not aligned with them sentence by sentence. Quality was checked by manual inspection, length and repetition filters, and a character-level 6-gram LM that flagged high-perplexity output. The paper reports corpus statistics, not model experiments. Translated pairs are similar in length, while independent German captions are shorter than English ones and have a larger vocabulary. Multi30K is larger than earlier multilingual image–text sets and supports multimodal MT and multilingual captioning.

## Key points
- Multi30K extends Flickr30K (31,014 images) with 31,014 professional German translations and 155,070 independently crowdsourced German descriptions.
- Translators worked without the images, at a total cost of about €23,000 (€0.06 per word). The English sources were chosen in equal shares of short, median and long captions.
- 185 Crowdflower workers wrote the descriptions over 31 days and were paid $0.05 per caption. Total spending, including platform fees, was $9,591.24.
- The German texts are lexically richer than the English ones:
  - Average length is 11.1 words for German translations versus 9.6 for German descriptions.
  - German vocabularies are more than twice the size of the English ones, with two to three times as many singletons, likely due to morphology and compounding.
- Earlier multilingual image datasets were smaller and translation-only: IAPR-TC12 has 20,000 English-German images, and Pascal Sentences has 1,000 Japanese-English images.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

Topics: [Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{elliott2016multi30k,
  title = {{Multi30K: Multilingual English-German Image Descriptions}},
  author = {Elliott, Desmond and Frank, Stella and Sima'an, Khalil and Specia, Lucia},
  year = {2016},
  doi = {10.48550/arxiv.1605.00459},
  eprint = {1605.00459},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1605.00459}
}
```

- [← previous CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://md.research.elamur.ai/free/en/papers/johnson-2016-clevr-diagnostic-dataset-compositional-language/)
- [next → Many Languages, One Parser](https://md.research.elamur.ai/free/en/papers/ammar-2016-many-languages-one-parser/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/2/)

## References

17 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2015%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+In+Proceedings+of+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
2. Raffaella Bernardi, Ruken Cakici, Desmond Elliott, Aykut Erdem, Erkut Erdem, Nazli Ikizler-Cinbis, Frank Keller, Adrian Muscat, and Barbara Plank. 2016. Automatic description generation from images: A survey of models, datasets, and evaluation measures. Journal of Artificial Intelligence Research, 5… [find](https://scholar.google.com/scholar?q=Raffaella+Bernardi%2C+Ruken+Cakici%2C+Desmond+Elliott%2C+Aykut+Erdem%2C+Erkut+Erdem%2C+Nazli+Ikizler%2DCinbis%2C+Frank+Keller%2C+Adrian+Muscat%2C+and+Barbara+Plank%2E+2016%2E+Automatic+description+generation+from+images%3A+A)
3. Iacer Calixto, Teo E. de Campos, and Lucia Specia. 2012. Images as context in statistical machine translation. In Second Annual Meeting of the EPSRC Network on Vision & Language, Sheffield. [find](https://scholar.google.com/scholar?q=Iacer+Calixto%2C+Teo+E%2E+de+Campos%2C+and+Lucia+Specia%2E+2012%2E+Images+as+context+in+statistical+machine+translation%2E+In+Second+Annual+Meeting+of+the+EPSRC+Network+on+Vision+%26+Language%2C+Sheffield%2E)
4. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. 2015. Microsoft COCO captions: Data collection and evaluation server. CoRR, abs/1504.00325. [find](https://scholar.google.com/scholar?q=Xinlei+Chen%2C+Hao+Fang%2C+Tsung%2DYi+Lin%2C+Ramakrishna+Vedantam%2C+Saurabh+Gupta%2C+Piotr+Dolla%CC%81r%2C+and+C%2E+Lawrence+Zitnick%2E+2015%2E+Microsoft+COCO+captions%3A+Data+collection+and+evaluation+server%2E+CoRR%2C+abs%2F1504%2E0) [our card](https://md.research.elamur.ai/free/en/papers/chen-2015-microsoft-coco-captions-data-collection/)
5. David Chiang. 2005. A hierarchical phrase-based model for statistical machine translation. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, pages 263–270. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=David+Chiang%2E+2005%2E+A+hierarchical+phrase%2Dbased+model+for+statistical+machine+translation%2E+In+Proceedings+of+the+43rd+Annual+Meeting+on+Association+for+Computational+Linguistics%2C+pages+263%E2%80%93270%2E+Associ) [our card](https://md.research.elamur.ai/free/en/papers/chiang-2005-hierarchical-phrase-based-model-statistical/)
6. Desmond Elliott and Frank Keller. 2013. Image Description using Visual Dependency Representations. In EMNLP, pages 1292– 1302. [find](https://scholar.google.com/scholar?q=Desmond+Elliott+and+Frank+Keller%2E+2013%2E+Image+Description+using+Visual+Dependency+Representations%2E+In+EMNLP%2C+pages+1292%E2%80%93+1302%2E)
7. Desmond Elliott, Stella Frank, and Eva Hasler. 2015. Multi-language image description with neural sequence models. CoRR, abs/1510.04709. — [arXiv:1510.04709](https://arxiv.org/abs/1510.04709) [find](https://scholar.google.com/scholar?q=Desmond+Elliott%2C+Stella+Frank%2C+and+Eva+Hasler%2E+2015%2E+Multi%2Dlanguage+image+description+with+neural+sequence+models%2E+CoRR%2C+abs%2F1510%2E04709%2E+%E2%80%94+%5BarXiv%3A1510%2E04709%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1510%2E04709%29)
8. Ruka Funaki and Hideki Nakayama. 2015. Image-mediated learning for zero-shot cross-lingual document retrieval. In Empirical Methods in Natural Language Processing 2015, pages 585–590. [find](https://scholar.google.com/scholar?q=Ruka+Funaki+and+Hideki+Nakayama%2E+2015%2E+Image%2Dmediated+learning+for+zero%2Dshot+cross%2Dlingual+document+retrieval%2E+In+Empirical+Methods+in+Natural+Language+Processing+2015%2C+pages+585%E2%80%93590%2E)
9. Michael Grubinger, Paul D. Clough, Henning Muller, and Thomas Desealers. 2006. The IAPR TC-12 benchmark: A new evaluation resource for visual information systems. In LREC. [find](https://scholar.google.com/scholar?q=Michael+Grubinger%2C+Paul+D%2E+Clough%2C+Henning+Muller%2C+and+Thomas+Desealers%2E+2006%2E+The+IAPR+TC%2D12+benchmark%3A+A+new+evaluation+resource+for+visual+information+systems%2E+In+LREC%2E)
10. Julian Hitschler and Stefan Riezler. 2016. Multimodal pivots for image caption translation. CoRR, abs/1601.03916. [find](https://scholar.google.com/scholar?q=Julian+Hitschler+and+Stefan+Riezler%2E+2016%2E+Multimodal+pivots+for+image+caption+translation%2E+CoRR%2C+abs%2F1601%2E03916%2E)
11. Micah Hodosh, Peter Young, and Julia Hockenmaier. 2013. Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics. Journal of Artificial Intelligence Research, 47:853–899. [find](https://scholar.google.com/scholar?q=Micah+Hodosh%2C+Peter+Young%2C+and+Julia+Hockenmaier%2E+2013%2E+Framing+Image+Description+as+a+Ranking+Task%3A+Data%2C+Models+and+Evaluation+Metrics%2E+Journal+of+Artificial+Intelligence+Research%2C+47%3A853%E2%80%93899%2E)
12. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1, NAACL ’03, pages 48–54, Stroudsburg, PA, USA. As… [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2C+Franz+Josef+Och%2C+and+Daniel+Marcu%2E+2003%2E+Statistical+phrase%2Dbased+translation%2E+In+Proceedings+of+the+2003+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+L)
13. Cyrus Rashtchian, Peter Young, Micha Hodosh, and Julia Hockenmaier. 2010. Collecting image annotations using Amazon’s Mechanical Turk. In NAACLHLT Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk. [find](https://scholar.google.com/scholar?q=Cyrus+Rashtchian%2C+Peter+Young%2C+Micha+Hodosh%2C+and+Julia+Hockenmaier%2E+2010%2E+Collecting+image+annotations+using+Amazon%E2%80%99s+Mechanical+Turk%2E+In+NAACLHLT+Workshop+on+Creating+Speech+and+Language+Data+with+Am) [our card](https://md.research.elamur.ai/free/en/papers/rashtchian-2010-collecting-image-annotations-amazon-s/)
14. Ilya Sutskever, Oriol Vinyals, and Quoc V. V Le. 2014. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N.D. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 3104–3112. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V%2E+V+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Z%2E+Ghahramani%2C+M%2E+Welling%2C+C%2E+Cortes%2C+N%2ED%2E+Lawrence%2C+and+K%2EQ%2E+Weinberger%2C+editors%2C+Advance)
15. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June. [find](https://scholar.google.com/scholar?q=Oriol+Vinyals%2C+Alexander+Toshev%2C+Samy+Bengio%2C+and+Dumitru+Erhan%2E+2015%2E+Show+and+tell%3A+A+neural+image+caption+generator%2E+In+The+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition+%28CVPR%29%2C+June%2E) [our card](https://md.research.elamur.ai/free/en/papers/vinyals-2014-show-tell-neural-image-caption/)
16. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. CoRR, abs/1502.03044. [find](https://scholar.google.com/scholar?q=Kelvin+Xu%2C+Jimmy+Ba%2C+Ryan+Kiros%2C+Kyunghyun+Cho%2C+Aaron+C%2E+Courville%2C+Ruslan+Salakhutdinov%2C+Richard+S%2E+Zemel%2C+and+Yoshua+Bengio%2E+2015%2E+Show%2C+attend+and+tell%3A+Neural+image+caption+generation+with+visual+) [our card](https://md.research.elamur.ai/free/en/papers/xu-2015-show-attend-tell-neural-image/)
17. Peter Young, Alice Lai, Micha Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Peter+Young%2C+Alice+Lai%2C+Micha+Hodosh%2C+and+Julia+Hockenmaier%2E+2014%2E+From+image+descriptions+to+visual+denotations%3A+New+similarity+metrics+for+semantic+inference+over+event+descriptions%2E+Transactions+of)