---
title: "Context-aware Neural Machine Translation with — Ohtani 2019"
description: "Document-level NMT systems usually take context from neighbouring sentences through attention over every token."
authors: ["Takumi Ohtani", "Hidetaka Kamigaito", "Masaaki Nagata", "Manabu Okumura"]
year: 2019
venue: "Proceedings of the Fourth Workshop on Discourse in Machine Translation (DiscoMT 2019)"
source: "https://aclanthology.org/D19-6505/"
doi: "10.18653/v1/d19-6505"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/ohtani-2019-context-aware-neural-machine-translation/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/ohtani-2019-context-aware-neural-machine-translation/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2019](https://md.research.elamur.ai/free/papers/2019/) / Context-aware Neural Machine Translation with Coreference Information

# Context-aware Neural Machine Translation with Coreference Information

**Authors:** Takumi Ohtani, [Hidetaka Kamigaito](https://md.research.elamur.ai/free/authors/hidetaka-kamigaito/), [Masaaki Nagata](https://md.research.elamur.ai/free/authors/masaaki-nagata/), [Manabu Okumura](https://md.research.elamur.ai/free/authors/manabu-okumura/) · **Year:** 2019 · **Venue:** Proceedings of the Fourth Workshop on Discourse in Machine Translation (DiscoMT 2019) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D19-6505/) · [PDF](https://aclanthology.org/D19-6505.pdf) · [DOI 10.18653/v1/d19-6505](https://doi.org/10.18653/v1/d19-6505) · [OpenAlex](https://openalex.org/W2985010700)

**Citations:** 8 citations (openalex-snapshot · 2026-10-14) · [11](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Document-level NMT systems usually take context from neighbouring sentences through attention over every token. That cost grows quadratically with input length, which makes long contexts impractical. This matters especially for pro-drop target languages such as Japanese, where natural, concise output depends on wider context. Ohtani et al. propose a different encoder. They run NeuralCoref over the concatenated source sentences and use the resulting coreference links to wire a graph-structured Bi-LSTM encoder. Hidden states of antecedent words are merged into the states of the words that refer to them. Anaphoric links feed the forward pass and cataphoric links feed the backward pass. Attention is then applied only to the sentence currently being translated.

The paper tests two merge functions: simple averaging (Coref-mean) and a sigmoid-gated weighted sum (Coref-gate). On English-to-Japanese OpenSubtitles2018, both beat the concatenation baseline of Bawden et al. (2018) at every context size. The gated variant performs best and keeps improving as more sentences are added, whereas the baseline stalls and eventually runs out of GPU memory. The approach therefore makes context-aware translation of longer texts feasible.

## Key points
- **Data and setup:** About 1.87M training units and 2,000 test units were cut from OpenSubtitles2018 En→Ja, with n ∈ {1, 2, 3, 5, 7} consecutive sentences per unit. Japanese was tokenized with MeCab + NEologd. The models are 2-layer LSTMs with a 32k vocabulary, trained on one NVIDIA Tesla P100.
- **Best results:** Cor-g reached 8.79 BLEU at n=7. The best Concat score was 7.91 at n=3, and Concat could not run at n=7 because of GPU memory limits. The largest significant gain was about 0.9 BLEU.
- **Gate vs. mean:** Cor-g consistently outscored Cor-m. It also produced output lengths closer to the references, while the other two models under-generated.
- **Succeeding sentences:** Centering the target sentence with following context (Cor-g-c) scored lower than Cor-g at the same n, because most links in the data are anaphoric. Even so, it beat Cor-g at comparable amounts of preceding context, which suggests cataphora helps.
- **Coreference coverage:** The share of test sentences containing coreferences rose from 12.8% (n=1) to 15.4% (n=7). This also explains why the model gains even at n=1.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

Topics: [Coreference resolution](https://md.research.elamur.ai/free/topics/coreference-resolution/) · [Machine translation](https://md.research.elamur.ai/free/topics/machine-translation/)

## Cited by

Cited by

11 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 8 not yet.

- [Word-Level Coreference Resolution](https://md.research.elamur.ai/free/papers/dobrovolskii-2021-word-level-coreference-resolution/)
- [TED-CDB: A Large-Scale Chinese Discourse Relation Dataset on TED Talks](https://md.research.elamur.ai/free/papers/long-2020-ted-cdb-large-scale-chinese/)
- [Maverick: Efficient and Accurate Coreference Resolution Defying Recent Trends](https://md.research.elamur.ai/free/papers/martinelli-2024-maverick-efficient-accurate-coreference-resolution/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ohtani2019context,
  title = {{Context-aware Neural Machine Translation with Coreference Information}},
  author = {Ohtani, Takumi and Kamigaito, Hidetaka and Nagata, Masaaki and Okumura, Manabu},
  booktitle = {Proceedings of the Fourth Workshop on Discourse in Machine Translation (DiscoMT 2019)},
  year = {2019},
  doi = {10.18653/v1/d19-6505},
  url = {https://doi.org/10.18653/v1/d19-6505}
}
```

- [← previous Constrained Sequence-to-sequence Semitic Root Extraction for Enriching Word Embeddings](https://md.research.elamur.ai/free/papers/aclbulk-2019-kishky-constrained-sequence-sequence-semitic-root/)
- [next → Controllable Paraphrase Generation with a Syntactic Exemplar](https://md.research.elamur.ai/free/papers/chen-2019-controllable-paraphrase-generation-syntactic-exemplar/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/papers/2019/)

## References

13 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, and James Henderson. 2018. Document-level neural machine translation with hierarchical attention networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2947–2954, Brussels, Belgium. Association for… [find](https://scholar.google.com/scholar?q=Lesly+Miculicich%2C+Dhananjay+Ram%2C+Nikolaos+Pappas%2C+and+James+Henderson%2E+2018%2E+Document%2Dlevel+neural+machine+translation+with+hierarchical+attention+networks%2E+In+Proceedings+of+the+2018+Conference+on+Em) [our card](https://md.research.elamur.ai/free/papers/miculicich-2018-document-level-neural-machine-translation/)
2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv e-prints, abs/1409.0473. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2014%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+arXiv+e%2Dprints%2C+abs%2F1409%2E0473%2E) [our card](https://md.research.elamur.ai/free/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
3. Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. 2018. Evaluating discourse phenomena in neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Paper… [find](https://scholar.google.com/scholar?q=Rachel+Bawden%2C+Rico+Sennrich%2C+Alexandra+Birch%2C+and+Barry+Haddow%2E+2018%2E+Evaluating+discourse+phenomena+in+neural+machine+translation%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter) [our card](https://md.research.elamur.ai/free/papers/bawden-2017-evaluating-discourse-phenomena-neural-machine/)
4. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+Bleu%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+40th+Annual+Meeting+of+the+Association+for+Computat) [our card](https://md.research.elamur.ai/free/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
5. Jonathan H. Clark, Chris Dyer, Alon Lavie, and Noah A. Smith. 2011. Better hypothesis testing for statistical machine translation: Controlling for optimizer instability. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 176… [find](https://scholar.google.com/scholar?q=Jonathan+H%2E+Clark%2C+Chris+Dyer%2C+Alon+Lavie%2C+and+Noah+A%2E+Smith%2E+2011%2E+Better+hypothesis+testing+for+statistical+machine+translation%3A+Controlling+for+optimizer+instability%2E+In+Proceedings+of+the+49th+Ann)
6. Reid Pryzant, Youngjoo Chung, Dan Jurafsky, and Denny Britz. 2018. JESC: Japanese-English subtitle corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018), Miyazaki, Japan. European Languages Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Reid+Pryzant%2C+Youngjoo+Chung%2C+Dan+Jurafsky%2C+and+Denny+Britz%2E+2018%2E+JESC%3A+Japanese%2DEnglish+subtitle+corpus%2E+In+Proceedings+of+the+Eleventh+International+Conference+on+Language+Resources+and+Evaluation+)
7. Kevin Clark and Christopher D. Manning. 2016. Deep reinforcement learning for mention-ranking coreference models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2256–2262, Austin, Texas. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kevin+Clark+and+Christopher+D%2E+Manning%2E+2016%2E+Deep+reinforcement+learning+for+mention%2Dranking+coreference+models%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Language+Process) [our card](https://md.research.elamur.ai/free/papers/clark-2016-deep-reinforcement-learning-mention-ranking/)
8. Toshinori Sato, Taiichi Hashimoto, and Manabu Okumura. 2017. Implementation of a word segmentation dictionary called mecab-ipadic-neologd and study on how to use it effectively for information retrieval (in japanese). In NLP, pages NLP2017– B6–1. The Association for Natural Language Processing. [find](https://scholar.google.com/scholar?q=Toshinori+Sato%2C+Taiichi+Hashimoto%2C+and+Manabu+Okumura%2E+2017%2E+Implementation+of+a+word+segmentation+dictionary+called+mecab%2Dipadic%2Dneologd+and+study+on+how+to+use+it+effectively+for+information+retriev)
9. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the International Dario Stojanovski and Alexander Fraser. 2018. Coreference and coherence in neural machine translation: A study using oracle experiments. In Proceeding… [find](https://scholar.google.com/scholar?q=Xavier+Glorot+and+Yoshua+Bengio%2E+2010%2E+Understanding+the+difficulty+of+training+deep+feedforward+neural+networks%2E+In+Proceedings+of+the+International+Dario+Stojanovski+and+Alexander+Fraser%2E+2018%2E+Core)
10. Jörg Tiedemann and Yves Scherrer. 2017. Neural machine translation with extended context. In Proceedings of the Third Workshop on Discourse in Machine Translation, pages 82–92, Copenhagen, Denmark. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jo%CC%88rg+Tiedemann+and+Yves+Scherrer%2E+2017%2E+Neural+machine+translation+with+extended+context%2E+In+Proceedings+of+the+Third+Workshop+on+Discourse+in+Machine+Translation%2C+pages+82%E2%80%9392%2C+Copenhagen%2C+Denmark%2E+A) [our card](https://md.research.elamur.ai/free/papers/tiedemann-2017-neural-machine-translation-extended-context/)
11. Elena Voita, Pavel Serdyukov, Rico Sennrich, and Ivan Titov. 2018. Context-aware neural machine translation learns anaphora resolution. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1264–1274. Association for Computational L… [find](https://scholar.google.com/scholar?q=Elena+Voita%2C+Pavel+Serdyukov%2C+Rico+Sennrich%2C+and+Ivan+Titov%2E+2018%2E+Context%2Daware+neural+machine+translation+learns+anaphora+resolution%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for) [our card](https://md.research.elamur.ai/free/papers/voita-2018-context-aware-neural-machine-translation/)
12. Longyue Wang, Zhaopeng Tu, Andy Way, and Qun Liu. 2017. Exploiting cross-sentence context for neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2826–2831, Copenhagen, Denmark. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Longyue+Wang%2C+Zhaopeng+Tu%2C+Andy+Way%2C+and+Qun+Liu%2E+2017%2E+Exploiting+cross%2Dsentence+context+for+neural+machine+translation%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Natural+Language)
13. Jiacheng Zhang, Huanbo Luan, Maosong Sun, Feifei Zhai, Jingfang Xu, Min Zhang, and Yang Liu. 2018. Improving the transformer translation model with document-level context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 533–542, Brussels, Belgium. Ass… [find](https://scholar.google.com/scholar?q=Jiacheng+Zhang%2C+Huanbo+Luan%2C+Maosong+Sun%2C+Feifei+Zhai%2C+Jingfang+Xu%2C+Min+Zhang%2C+and+Yang+Liu%2E+2018%2E+Improving+the+transformer+translation+model+with+document%2Dlevel+context%2E+In+Proceedings+of+the+2018+C) [our card](https://md.research.elamur.ai/free/papers/zhang-2018-improving-transformer-translation-model-document/)