--- title: "Attention Strategies for Multi-Source Sequence-to-Sequence" description: "The authors propose two alternatives. 1.5 BLEU. 1 reference, cited by 79 papers here. Among them: 30." authors: ["Jindřich Libovický", "Jindřich Helcl"] year: 2017 venue: "Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)" source: "https://aclanthology.org/P17-2031/" doi: "10.18653/v1/p17-2031" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/libovicky-2017-attention-strategies-multi-source-sequence/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/libovicky-2017-attention-strategies-multi-source-sequence/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/page/3/) / Attention Strategies for Multi-Source Sequence-to-Sequence Learning # Attention Strategies for Multi-Source Sequence-to-Sequence Learning **Authors:** [Jindřich Libovický](https://md.research.elamur.ai/free/en/authors/jindrich-libovicky/) and [Jindřich Helcl](https://md.research.elamur.ai/free/en/authors/jindrich-helcl/) · **Year:** 2017 · **Venue:** [Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2017](https://md.research.elamur.ai/free/en/venues/acl/2017/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/P17-2031/) · [PDF](https://aclanthology.org/P17-2031.pdf) · [DOI 10.18653/v1/p17-2031](https://doi.org/10.18653/v1/p17-2031) · [OpenAlex](https://openalex.org/W2609160544) **Citations:** 30 citations (OpenAlex, 2026-10-14) · [79](https://md.research.elamur.ai/free/en/papers/libovicky-2017-attention-strategies-multi-source-sequence/cited-by/) in our library, of the 141,840 papers whose bibliographies we read ## Summary The paper looks at how a single recurrent decoder should combine attention when it reads several encoders at once, possibly of different modalities. The usual practice concatenates per-encoder context vectors. That leaves the weighting of the sources to later layers and gives no explicit view of how much each input is used. The authors propose two alternatives. **Flat** combination computes one softmax over the hidden states of all encoders after projecting them into a shared space. **Hierarchical** combination first builds a context vector for each encoder, then applies a second attention over those vectors. Both can optionally include a sentinel adapted to GRUs, which lets the decoder rely on its own state. The strategies were tested on WMT16 multimodal translation (Multi30k) and WMT16 automatic post-editing, implemented in Neural Monkey. Hierarchical attention without the sentinel gave the best multimodal scores and the fastest convergence. In APE, all variants slightly beat the do-nothing baseline but did not differ significantly from each other. The work matters because it makes the contribution of each source interpretable and works with conditional GRU decoders, where concatenation breaks down. ## Key points - **Data:** - Multi30k has 29,000 training, 1,014 validation and 1,000 test instances (image, English caption, German translation). - The WMT16 APE data has 12,000 / 2,000 / 1,000 English–MT German–post-edited German triplets from the IT domain. - **Multimodal translation results:** - Hierarchical attention (no sentinel, unshared projections) reached BLEU 32.1 and METEOR 49.1, against 31.4 / 48.0 for concatenation. - The text-only baseline (BLEU 33.0) was still not surpassed. - **Conditional GRU effect:** on average, conditional GRU units added about 1.5 BLEU to the new strategies. With concatenation, however, they cut about 5 BLEU. - **APE results:** - All models reached HTER of roughly 24.1–24.4, versus 24.8 for leaving the MT output unchanged. - The differences between combination strategies were not significant. - **Setup:** - Bidirectional GRU encoders with 300 units per direction and 300-dimensional embeddings. - A 500-unit attention space and Adam with learning rate 10⁻⁴. - A frozen VGG 16 for images and a 20,000-unit shared BPE vocabulary. ## Interesting ideas - **Explicit source weights:** making the decoder produce weights over sources, rather than mixing them implicitly, yields per-step weights you can inspect. In the paper's visualization, the image encoder lights up mainly on content words. This is handy for diagnosing whether an auxiliary input, such as metadata or a second language, is actually used. - **Two-level attention:** attending first inside each source and then across sources is a cheap, modular way to fuse heterogeneous inputs of different dimensionality. Each source is only projected into a common space at the top level. - **Sentinel option:** a sentinel entry lets the model "attend to nothing" and lean on its own language-model-like state. This is useful when some inputs are often uninformative for the current output step. - **Edit-operation targets for post-editing:** the APE targets are encoded as edit operations on the MT output rather than as full sentences. This stops the model from freely paraphrasing, which suits any correction task where the input is mostly right. - **Sharing projections is not free:** the paper tested whether energy and context computations can share one projection. Sharing brought no gain and made training dramatically slower, a reminder to test such parameter-saving shortcuts empirically. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Charles University](https://md.research.elamur.ai/free/en/universities/charles-university/) — University, Prague, Czechia · 108 works · 20,329 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works ## Cite Cite as: Jindřich Libovický and Jindřich Helcl. 2017. Attention Strategies for Multi-Source Sequence-to-Sequence Learning. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{libovicky2017attention, title = {{Attention Strategies for Multi-Source Sequence-to-Sequence Learning}}, author = {Jindřich Libovický and Jindřich Helcl}, booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)}, year = {2017}, doi = {10.18653/v1/p17-2031}, url = {https://doi.org/10.18653/v1/p17-2031} } ``` - [← previous AMR-to-text Generation with Synchronous Node Replacement Grammar](https://md.research.elamur.ai/free/en/papers/song-2017-amr-text-generation-synchronous-node/) - [next → Learning How to Simplify From Explicit Labeling of Complex-Simplified Text Pairs](https://md.research.elamur.ai/free/en/papers/manchego-2017-learning-how-simplify-explicit-labeling/) - [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/page/3/) ## References 1 references, 1 of them with a card of ours. ↗ leads to the work. 1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). CoRR abs/1409.0473. http://arxiv.org/abs/1409.0473. [↗](http://arxiv.org/abs/1409.0473) *Updated 24 October 2026*