--- title: "Improving the Transformer Translation Model — Zhang 2018" description: "Standard Transformer translation handles each sentence in isolation, so it cannot resolve discourse-dependent issues such as ambiguous word senses…" authors: ["Jiacheng Zhang", "Huanbo Luan", "Maosong Sun", "Feifei Zhai", "Jingfang Xu", "Min Zhang", "Yang Liu"] year: 2018 venue: "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing" source: "https://aclanthology.org/D18-1049/" doi: "10.18653/v1/d18-1049" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/zhang-2018-improving-transformer-translation-model-document/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2018-improving-transformer-translation-model-document/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/2/) / Improving the Transformer Translation Model with Document-Level Context # Improving the Transformer Translation Model with Document-Level Context **Authors:** Jiacheng Zhang, [Huanbo Luan](https://md.research.elamur.ai/free/en/authors/huanbo-luan/), [Maosong Sun](https://md.research.elamur.ai/free/en/authors/maosong-sun/), [Feifei Zhai](https://md.research.elamur.ai/free/en/authors/feifei-zhai/), Jingfang Xu, [Min Zhang](https://md.research.elamur.ai/free/en/authors/min-zhang/), [Yang Liu](https://md.research.elamur.ai/free/en/authors/yang-liu/) · **Year:** 2018 · **Venue:** Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/D18-1049/) · [PDF](https://aclanthology.org/D18-1049.pdf) · [DOI 10.18653/v1/d18-1049](https://doi.org/10.18653/v1/d18-1049) · [OpenAlex](https://openalex.org/W2962712961) **Citations:** 240 citations (openalex-snapshot · 2026-10-14) · [128](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Standard Transformer translation handles each sentence in isolation, so it cannot resolve discourse-dependent issues such as ambiguous word senses, coreference or lexical cohesion. The authors add a separate context encoder that uses multi-head self-attention to represent the preceding source sentences of a document. Target-side history is deliberately left out to avoid propagating errors. Extra context-attention sub-layers then feed this representation into every encoder and decoder layer, with a gating mechanism in place of plain residual connections. Document-aligned parallel data is scarce, so training happens in two stages. First, the ordinary sentence-level Transformer is fitted on large sentence corpora. Then those weights are frozen and only the new document-level components are learned on the smaller document corpus. Tested on NIST Chinese-English and IWSLT French-English, the model beats the Transformer baseline and an adapted cache-based approach, with little extra computational cost. The work shows that document awareness can be added to Transformer cheaply and is useful even with limited document-level data. ## Key points - Reported gains over the Transformer baseline are +1.96 BLEU on Chinese-English (NIST) and +0.89 BLEU on French-English (IWSLT). The model also outperforms the cache-based method of Kuang et al. (2017) adapted to Transformer. - Two-step training is essential: - Training only on the 940K-sentence document corpus gives 36.52 BLEU, below the sentence-level baseline's 39.53. - Combining 2M sentence pairs with the 940K document corpus reaches 47.93 BLEU, versus 45.97 for the baseline. - Context helps in either location, with the best result when integrated into both: - encoder only: 47.51 BLEU - decoder only: 47.48 BLEU - both: 47.93 BLEU - Context gating contributes about +0.38 BLEU compared with residual connections. - Source code is released as Document-Transformer on GitHub (Glaceon31). *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) ## Cited by Cited by 128 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 115 not yet. - [Findings of the WMT 2019 Shared Task on Automatic Post-Editing](https://md.research.elamur.ai/free/en/papers/chatterjee-2019-findings-wmt-2019-shared-task/) - [Findings of the WMT 2020 Shared Task on Chat Translation](https://md.research.elamur.ai/free/en/papers/farajian-2020-findings-wmt-2020-shared-task/) - [BlonDe: An Automatic Evaluation Metric for Document-level Machine Translation](https://md.research.elamur.ai/free/en/papers/jiang-2022-blonde-automatic-evaluation-metric-document/) - [Large Language Models Effectively Leverage Document-level Context for Literary Translation, but Critical Errors Persist](https://md.research.elamur.ai/free/en/papers/karpinska-2023-large-language-models-effectively-leverage/) - [Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense](https://md.research.elamur.ai/free/en/papers/krishna-2023-paraphrasing-evades-detectors-ai-generated/) - [Document-level Neural MT: A Systematic Comparison](https://md.research.elamur.ai/free/en/papers/lopes-2020-document-level-neural-mt-systematic/) - [文档级神经机器翻译综述](https://md.research.elamur.ai/free/en/papers/lv-2024-document-level-neural/) - [Selective Attention for Context-aware Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/maruf-2019-selective-attention-context-aware-neural/) - [Context-aware Neural Machine Translation with Coreference Information](https://md.research.elamur.ai/free/en/papers/ohtani-2019-context-aware-neural-machine-translation/) - [When a Good Translation is Wrong in Context: Context-Aware Machine Translation Improves on Deixis, Ellipsis, and Lexical Cohesion](https://md.research.elamur.ai/free/en/papers/voita-2019-when-good-translation-is-wrong/) - [AMR-To-Text Generation with Graph Transformer](https://md.research.elamur.ai/free/en/papers/wang-2020-amr-text-generation-graph-transformer/) - [GuoFeng: A Benchmark for Zero Pronoun Recovery and Translation](https://md.research.elamur.ai/free/en/papers/xu-2022-guofeng-benchmark-zero-pronoun-recovery/) - [Knowledge-Enriched Transformer for Emotion Detection in Textual Conversations](https://md.research.elamur.ai/free/en/papers/zhong-2019-knowledge-enriched-transformer-emotion-detection/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{zhang2018improving, title = {{Improving the Transformer Translation Model with Document-Level Context}}, author = {Zhang, Jiacheng and Luan, Huanbo and Sun, Maosong and Zhai, Feifei and Xu, Jingfang and Zhang, Min and Liu, Yang}, booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing}, year = {2018}, doi = {10.18653/v1/d18-1049}, url = {https://doi.org/10.18653/v1/d18-1049} } ``` - [← previous Improving Text-to-SQL Evaluation Methodology](https://md.research.elamur.ai/free/en/papers/dollak-2018-improving-text-sql-evaluation-methodology/) - [next → Integrating Predictions from Neural-Network Relation Classifiers into Coreference and Bridging Resolution](https://md.research.elamur.ai/free/en/papers/roesiger-2018-integrating-predictions-neural-network-relation/) - [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/2/) ## References 4 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Dzmitry Bahdanau, KyungHyun Cho, and Yoshua Bengio. 2015. Sequence to sequence learning with neural networks. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+KyungHyun+Cho%2C+and+Yoshua+Bengio%2E+2015%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Proceedings+of+ICLR%2E) 2. Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. 2018. Evaluating discourse phenomena in neural machine translation. In Proceedings of NAACL. [find](https://scholar.google.com/scholar?q=Rachel+Bawden%2C+Rico+Sennrich%2C+Alexandra+Birch%2C+and+Barry+Haddow%2E+2018%2E+Evaluating+discourse+phenomena+in+neural+machine+translation%2E+In+Proceedings+of+NAACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/bawden-2017-evaluating-discourse-phenomena-neural-machine/) 3. Eva Martı́nez Garcia, Cristina Espãna Bonet, and Lluı́z Màrquez. 2014. Document-level machine translation with word vector models. In Proceedings of EACL. [find](https://scholar.google.com/scholar?q=Eva+Mart%C4%B1%CC%81nez+Garcia%2C+Cristina+Espa%CC%83na+Bonet%2C+and+Llu%C4%B1%CC%81z+Ma%CC%80rquez%2E+2014%2E+Document%2Dlevel+machine+translation+with+word+vector+models%2E+In+Proceedings+of+EACL%2E) 4. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. CoRR, abs/1705.03122. \#\# Conclusion Zhengxian Gong, Min Zhang, and Guodong Zhou. 2011. Cache-based document-level statistical machine translation. In Proceedings of EMNLP… [find](https://scholar.google.com/scholar?q=Jonas+Gehring%2C+Michael+Auli%2C+David+Grangier%2C+Denis+Yarats%2C+and+Yann+N+Dauphin%2E+2017%2E+Convolutional+sequence+to+sequence+learning%2E+CoRR%2C+abs%2F1705%2E03122%2E+%5C%23%5C%23+Conclusion+Zhengxian+Gong%2C+Min+Zhang%2C+and+G) [our card](https://md.research.elamur.ai/free/en/papers/gehring-2017-convolutional-sequence-sequence-learning/)