---
title: "When and Why is Document-level Context Useful in Neural"
description: "Kim, Tran and Ney instead study general, non-targeted test sets. 31.4 BLEU. 29 references, cited by 64 papers here."
authors: ["Yunsu Kim", "Duc Thanh Tran", "Hermann Ney"]
year: 2019
venue: "Proceedings of the Fourth Workshop on Discourse in Machine Translation (DiscoMT 2019)"
source: "https://aclanthology.org/D19-6503/"
doi: "10.18653/v1/d19-6503"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kim-2019-when-why-is-document-level/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kim-2019-when-why-is-document-level/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/) / When and Why is Document-level Context Useful in Neural Machine Translation?

# When and Why is Document-level Context Useful in Neural Machine Translation?

**Authors:** [Yunsu Kim](https://md.research.elamur.ai/free/en/authors/yunsu-kim/), Duc Thanh Tran and [Hermann Ney](https://md.research.elamur.ai/free/en/authors/hermann-ney/) · **Year:** 2019 · **Venue:** [Proceedings of the Fourth Workshop on Discourse in Machine Translation (DiscoMT 2019)](https://md.research.elamur.ai/free/en/venues/workshop-on-discourse-in-machine-translation/) ([Workshop on Discourse in Machine Translation 2019](https://md.research.elamur.ai/free/en/venues/workshop-on-discourse-in-machine-translation/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D19-6503/) · [PDF](https://aclanthology.org/D19-6503.pdf) · [DOI 10.18653/v1/d19-6503](https://doi.org/10.18653/v1/d19-6503) · [OpenAlex](https://openalex.org/W2983108239)

**Citations:** 60 citations (OpenAlex, 2026-10-14) · [64](https://md.research.elamur.ai/free/en/papers/kim-2019-when-why-is-document-level/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
This paper asks whether the gains reported for document-level neural machine translation really come from using discourse context. Earlier work often backed such claims with a few hand-picked examples or with test sets built specifically for pronoun phenomena. Kim, Tran and Ney instead study general, non-targeted test sets. They use IWSLT 2017 English→Italian (TED) and WMT 2018 English→German (News Commentary v14), with one preceding source sentence as context. They compare single-encoder concatenation with several multi-encoder variants built in Sockeye on a base Transformer. They also try reducing the context by filtering its words, for example by dropping stopwords or keeping only named entities or selected POS tags.

A qualitative analysis shows that only a small share of improvements can be explained by things like coreference or topic-aware word choice. The authors suggest that most of the BLEU gain is actually a regularization side effect. Once the sentence-level baseline is properly regularized, or trained on much more data, the document-level model no longer beats it. They also find that long context adds little and that word embeddings alone are enough to encode the context.

## Key points
- Training data: En→It TED has 4.3M words, 227k sentences and 2,045 documents (about 111 sentences each). En→De News Commentary v14 has 8.1M words, 329k sentences and 8,891 documents (about 37 sentences each).
- Setup: BPE with 32k joint merges, a 6-layer base Transformer in Sockeye, and source embeddings shared between the context and current-sentence encoders. Initializing from a pre-trained sentence-level model proved crucial.
- Context filtering options:
  - spaCy stopword removal;
  - removing the n = 150 most frequent words;
  - keeping only FLAIR named entities or only words with selected FLAIR POS tags.
  Pruning context words this way did not hurt performance.
- With dropout 0.1, the document-level model wins (32.5 vs 31.4 BLEU En→It; 30.3 vs 28.9 En→De). With dropout 0.3, the sentence-level baseline is better or equal (33.7 vs 33.5; 32.3 vs 32.0).
- In a large-scale test, a sentence-level model trained on all WMT 2019 parallel data reached 40.2 BLEU. Fine-tuning it on document-level data gave 39.9.

## Interesting ideas
- **Strengthen the baseline before crediting a new component.** Extra inputs can help simply by acting as a regularizer. Comparing against a well-regularized baseline separates genuine gains from such side effects, a check that applies to any "add more input" method.
- **Categorize improvements manually on general test sets.** The authors label each improved output as coreference, topic-aware lexical choice, or not interpretable. This shows how often a claimed mechanism actually operates in realistic data, which a targeted test set cannot reveal.
- **Shrink the context with cheap linguistic filters.** Keeping only named entities, content POS tags, or non-frequent words cuts memory use and input length while preserving topic cues. The same trick could work for long-context or retrieval-augmented models.
- **Use a minimal context encoder.** Plain word embeddings matched heavier encoders. Before building a full encoder stack for auxiliary inputs, it is worth testing how little modeling power is actually needed.
- **Fix document-level errors with a separate lightweight post-editor.** Such errors are rare in practice, so the authors propose a small post-editing model rather than making the main translation model more complex. This keeps the core system simple.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [RWTH Aachen University](https://md.research.elamur.ai/free/en/universities/rwth-aachen-university/) — University, Aachen, Germany · 18 works · 4,791 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🔁 Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · 958 works

## Cite

Cite as: Yunsu Kim, Duc Thanh Tran and Hermann Ney. 2019. When and Why is Document-level Context Useful in Neural Machine Translation?. Proceedings of the Fourth Workshop on Discourse in Machine Translation (DiscoMT 2019).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{kim2019when,
  title = {{When and Why is Document-level Context Useful in Neural Machine Translation?}},
  author = {Yunsu Kim, Duc Thanh Tran and Hermann Ney},
  booktitle = {Proceedings of the Fourth Workshop on Discourse in Machine Translation (DiscoMT 2019)},
  year = {2019},
  doi = {10.18653/v1/d19-6503},
  url = {https://doi.org/10.18653/v1/d19-6503}
}
```

- [← previous Broad-Coverage Semantic Parsing as Transduction](https://md.research.elamur.ai/free/en/papers/zhang-2019-broad-coverage-semantic-parsing-as/)
- [next → Better Rewards Yield Better Summaries: Learning to Summarise Without References](https://md.research.elamur.ai/free/en/papers/bohm-2019-better-rewards-yield-better-summaries/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/)

## References

29 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Fourth Conference on Machine Translation (WMT), Volume 2: Shared Task Papers, pages 424–432, Florence, Italy. [find](https://scholar.google.com/scholar?q=Fourth+Conference+on+Machine+Translation+%28WMT%29%2C+Volume+2%3A+Shared+Task+Papers%2C+pages+424%E2%80%93432%2C+Florence%2C+Italy%2E)
2. Ruchit Rajeshkumar Agrawal, Marco Turchi, and Matteo Negri. 2018. Contextual handling in neural machine translation: Look behind, ahead and on both sides. In 21st Annual Conference of the European Association for Machine Translation, pages 11–20. [find](https://scholar.google.com/scholar?q=Ruchit+Rajeshkumar+Agrawal%2C+Marco+Turchi%2C+and+Matteo+Negri%2E+2018%2E+Contextual+handling+in+neural+machine+translation%3A+Look+behind%2C+ahead+and+on+both+sides%2E+In+21st+Annual+Conference+of+the+European+Ass)
3. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. [find](https://scholar.google.com/scholar?q=Diederik+P%2E+Kingma+and+Jimmy+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+3rd+International+Conference+on+Learning+Representations%2C+ICLR+2015%2C+San+Diego%2C+CA%2C+USA%2C+May+7%2D9%2C+2015%2C+Conference)
4. Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. [Contextual string embeddings for sequence labeling](https://md.research.elamur.ai/free/en/papers/akbik-2018-contextual-string-embeddings-sequence-labeling/). In Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26, 2018, pages 1638–1649.
5. Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28–39. [find](https://scholar.google.com/scholar?q=Philipp+Koehn+and+Rebecca+Knowles%2E+2017%2E+Six+challenges+for+neural+machine+translation%2E+In+Proceedings+of+the+First+Workshop+on+Neural+Machine+Translation%2C+pages+28%E2%80%9339%2E)
6. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
7. Shaohui Kuang and Deyi Xiong. 2018. Fusing recency into neural machine translation with an inter-sentence gate model. In Proceedings of the 27th International Conference on Computational Linguistics, pages 607–617. [find](https://scholar.google.com/scholar?q=Shaohui+Kuang+and+Deyi+Xiong%2E+2018%2E+Fusing+recency+into+neural+machine+translation+with+an+inter%2Dsentence+gate+model%2E+In+Proceedings+of+the+27th+International+Conference+on+Computational+Linguistics%2C+)
8. Shaohui Kuang, Deyi Xiong, Weihua Luo, and Guodong Zhou. 2018. Modeling coherence for neural machine translation with dynamic and topic caches. In Proceedings of the 27th International Conference on Computational Linguistics, pages 596–606. [find](https://scholar.google.com/scholar?q=Shaohui+Kuang%2C+Deyi+Xiong%2C+Weihua+Luo%2C+and+Guodong+Zhou%2E+2018%2E+Modeling+coherence+for+neural+machine+translation+with+dynamic+and+topic+caches%2E+In+Proceedings+of+the+27th+International+Conference+on+C)
9. Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. 2018. [Evaluating discourse phenomena in neural machine translation](https://md.research.elamur.ai/free/en/papers/bawden-2017-evaluating-discourse-phenomena-neural-machine/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Paper…
10. Samuel Läubli, Rico Sennrich, and Martin Volk. 2018. [Has machine translation achieved human parity? a case for document-level evaluation](https://md.research.elamur.ai/free/en/papers/laubli-2018-has-machine-translation-achieved-human/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4791–4796.
11. Qian Cao and Deyi Xiong. 2018. Encoding gated translation memory into neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3042–3047. [find](https://scholar.google.com/scholar?q=Qian+Cao+and+Deyi+Xiong%2E+2018%2E+Encoding+gated+translation+memory+into+neural+machine+translation%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+3042%E2%80%93)
12. Sameen Maruf and Gholamreza Haffari. 2018. [Document context neural machine translation with memory networks](https://md.research.elamur.ai/free/en/papers/maruf-2018-document-context-neural-machine-translation/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1275–1284.
13. Annette Rios Gonzales, Laura Mascarell, and Rico Sennrich. 2017. [Improving word sense disambiguation in neural machine translation with sense embeddings](https://md.research.elamur.ai/free/en/papers/gonzales-2017-improving-word-sense-disambiguation-neural/). In Proceedings of the Second Conference on Machine Translation, pages 11–19.
14. Sameen Maruf, André FT Martins, and Gholamreza Haffari. 2019. [Selective attention for context-aware neural machine translation](https://md.research.elamur.ai/free/en/papers/maruf-2019-selective-attention-context-aware-neural/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short…
15. Liane Guillou, Christian Hardmeier, Ekaterina Lapshinova-Koltunski, and Sharid Loáiciga. 2018. A pronoun test suite evaluation of the english– german mt systems at wmt 2018. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 570–577. [find](https://scholar.google.com/scholar?q=Liane+Guillou%2C+Christian+Hardmeier%2C+Ekaterina+Lapshinova%2DKoltunski%2C+and+Sharid+Loa%CC%81iciga%2E+2018%2E+A+pronoun+test+suite+evaluation+of+the+english%E2%80%93+german+mt+systems+at+wmt+2018%2E+In+Proceedings+of+the+Thi)
16. Rebecca Marvin and Philipp Koehn. 2018. Exploring word sense disambiguation abilities of neural machine translation systems (non-archival extended abstract). In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Papers), pages 125–131. [find](https://scholar.google.com/scholar?q=Rebecca+Marvin+and+Philipp+Koehn%2E+2018%2E+Exploring+word+sense+disambiguation+abilities+of+neural+machine+translation+systems+%28non%2Darchival+extended+abstract%29%2E+In+Proceedings+of+the+13th+Conference+of+t)
17. Felix Hieber, Tobias Domhan, Michael Denkowski, David Vilar, Artem Sokolov, Ann Clifton, and Matt Post. 2018. The sockeye neural machine translation toolkit at AMTA 2018. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas, AMTA 2018, Boston, MA, USA, Mar… [find](https://scholar.google.com/scholar?q=Felix+Hieber%2C+Tobias+Domhan%2C+Michael+Denkowski%2C+David+Vilar%2C+Artem+Sokolov%2C+Ann+Clifton%2C+and+Matt+Post%2E+2018%2E+The+sockeye+neural+machine+translation+toolkit+at+AMTA+2018%2E+In+Proceedings+of+the+13th+Co)
18. Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, and James Henderson. 2018. [Document-level neural machine translation with hierarchical attention networks](https://md.research.elamur.ai/free/en/papers/miculicich-2018-document-level-neural-machine-translation/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2947–2954.
19. Sebastien Jean, Stanislas Lauly, Orhan Firat, and Kyunghyun Cho. 2017. Does neural machine translation benefit from larger context? arXiv preprint arXiv:1704.05135. [find](https://scholar.google.com/scholar?q=Sebastien+Jean%2C+Stanislas+Lauly%2C+Orhan+Firat%2C+and+Kyunghyun+Cho%2E+2017%2E+Does+neural+machine+translation+benefit+from+larger+context%3F+arXiv+preprint+arXiv%3A1704%2E05135%2E)
20. Xiao Pu, Nikolaos Pappas, James Henderson, and Andrei Popescu-Belis. 2018. Integrating weakly supervised word sense disambiguation into neural machine translation. Transactions of the Association for Computational Linguistics, 6:635–649. [find](https://scholar.google.com/scholar?q=Xiao+Pu%2C+Nikolaos+Pappas%2C+James+Henderson%2C+and+Andrei+Popescu%2DBelis%2E+2018%2E+Integrating+weakly+supervised+word+sense+disambiguation+into+neural+machine+translation%2E+Transactions+of+the+Association+for+)
21. Marcin Junczys-Dowmunt. 2019. [Microsoft translator at wmt 2019: Towards large-scale document-level neural machine translation](https://md.research.elamur.ai/free/en/papers/dowmunt-2019-microsoft-translator-wmt-2019-large/). In Proceedings of the Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual…
22. Dario Stojanovski and Alexander Fraser. 2018. [Coreference and coherence in neural machine translation: A study using oracle experiments](https://md.research.elamur.ai/free/en/papers/stojanovski-2018-coreference-coherence-neural-machine-translation/). In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 49–60.
23. Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. 2019. [Adaptive attention span in transformers](https://md.research.elamur.ai/free/en/papers/sukhbaatar-2019-adaptive-attention-span-transformers/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.
24. Jörg Tiedemann and Yves Scherrer. 2017. [Neural machine translation with extended context](https://md.research.elamur.ai/free/en/papers/tiedemann-2017-neural-machine-translation-extended-context/). In Proceedings of the Third Workshop on Discourse in Machine Translation, pages 82–92.
25. Zhaopeng Tu, Yang Liu, Shuming Shi, and Tong Zhang. 2018. Learning to remember translation history with a continuous cache. Transactions of the Association for Computational Linguistics, 6:407– 420. [find](https://scholar.google.com/scholar?q=Zhaopeng+Tu%2C+Yang+Liu%2C+Shuming+Shi%2C+and+Tong+Zhang%2E+2018%2E+Learning+to+remember+translation+history+with+a+continuous+cache%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+6%3A407%E2%80%93+420%2E)
26. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017,…
27. Elena Voita, Rico Sennrich, and Ivan Titov. 2019. [When a good translation is wrong in context: Context-aware machine translation improves on deixis, ellipsis, and lexical cohesion](https://md.research.elamur.ai/free/en/papers/voita-2019-when-good-translation-is-wrong/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL 2019), pages 1198– 1212…
28. Elena Voita, Pavel Serdyukov, Rico Sennrich, and Ivan Titov. 2018. [Context-aware neural machine translation learns anaphora resolution](https://md.research.elamur.ai/free/en/papers/voita-2018-context-aware-neural-machine-translation/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1264–1274.
29. Jiacheng Zhang, Huanbo Luan, Maosong Sun, Feifei Zhai, Jingfang Xu, Min Zhang, and Yang Liu. 2018. [Improving the transformer translation model with document-level context](https://md.research.elamur.ai/free/en/papers/zhang-2018-improving-transformer-translation-model-document/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 533–542.

*Updated 24 October 2026*