---
title: "Exploiting Cross-Sentence Context for Neural — Wang 2017"
description: "Standard neural machine translation handles each sentence in isolation."
authors: ["Longyue Wang", "Zhaopeng Tu", "Andy Way", "Qun Liu"]
year: 2017
venue: "Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/D17-1301/"
doi: "10.18653/v1/d17-1301"
arxiv: "1704.04347"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2017-exploiting-cross-sentence-context-neural/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2017-exploiting-cross-sentence-context-neural/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/page/2/) / Exploiting Cross-Sentence Context for Neural Machine Translation

# Exploiting Cross-Sentence Context for Neural Machine Translation

**Authors:** [Longyue Wang](https://md.research.elamur.ai/free/en/authors/longyue-wang/), [Zhaopeng Tu](https://md.research.elamur.ai/free/en/authors/zhaopeng-tu/), [Andy Way](https://md.research.elamur.ai/free/en/authors/andy-way/), [Qun Liu](https://md.research.elamur.ai/free/en/authors/qun-liu/) · **Year:** 2017 · **Venue:** Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D17-1301/) · [PDF](https://aclanthology.org/D17-1301.pdf) · [DOI 10.18653/v1/d17-1301](https://doi.org/10.18653/v1/d17-1301) · [arXiv 1704.04347](https://arxiv.org/abs/1704.04347) · [OpenAlex](https://openalex.org/W2964093087)

**Citations:** 190 citations (openalex-snapshot · 2026-10-14) · [101](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Standard neural machine translation handles each sentence in isolation. This makes it hard to resolve ambiguous words or to keep terminology consistent across a document. The authors address this by feeding information from preceding source sentences into an attention-based NMT system. A hierarchical RNN builds the context: a sentence-level RNN encodes each of the K previous sentences, and a document-level RNN combines those encodings into one history vector. The vector enters the translation model in two ways. First, it serves as the starting state of the encoder, the decoder, or both. Second, it acts as an extra input to the decoder at every step, optionally controlled by a context gate that decides how much history each target word needs. Experiments used Nematus on about 1M Chinese–English sentence pairs drawn from document-aligned LDC corpora. Every variant beat the baseline, and the initialization and gated-context methods together gained +2.1 BLEU on average. The work shows that inter-sentence context can be added cheaply and helps document-level translation.

## Key points
- **Data and evaluation:** Training used 1M sentence pairs (25.4M Chinese and 32.4M English words) from LDC corpora that have document boundaries. NIST05 (MT05) was the dev set, and NIST06 (MT06) and NIST08 (MT08) were the test sets. Scores are case-insensitive BLEU.
- **Setup:** The system was built on Nematus with the three previous sentences as context (K = 3), 35K-word vocabularies, 600-dimensional embeddings, and a hidden size of 1000.
- **Main results:** Average BLEU rose from 30.57 (Nematus) to 32.67 with Init enc+dec plus Gating Auxi. The Moses phrase-based system scored 28.24.
- **Effect of the gate:** Adding the context gate to auxiliary context gave roughly +1 BLEU over the ungated version (31.30 → 32.24 average).
- **Manual error analysis:** On 15 test documents (~60 sentences), the model fixed 29 of 38 ambiguity errors and 24 of 32 inconsistency errors. It also introduced 15 new errors. The code is released as LC-NMT.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/)

## Cited by

Cited by

101 papers of our library cite it (of the 140884 whose bibliographies we read); 14 of them have a page here, 87 not yet.

- [Evaluating Discourse Phenomena in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/bawden-2017-evaluating-discourse-phenomena-neural-machine/)
- [Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense](https://md.research.elamur.ai/free/en/papers/krishna-2023-paraphrasing-evades-detectors-ai-generated/)
- [文档级神经机器翻译综述](https://md.research.elamur.ai/free/en/papers/lv-2024-document-level-neural/)
- [Document Context Neural Machine Translation with Memory Networks](https://md.research.elamur.ai/free/en/papers/maruf-2018-document-context-neural-machine-translation/)
- [Selective Attention for Context-aware Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/maruf-2019-selective-attention-context-aware-neural/)
- [Document-Level Neural Machine Translation with Hierarchical Attention Networks](https://md.research.elamur.ai/free/en/papers/miculicich-2018-document-level-neural-machine-translation/)
- [Context-aware Neural Machine Translation with Coreference Information](https://md.research.elamur.ai/free/en/papers/ohtani-2019-context-aware-neural-machine-translation/)
- [Coreference and Coherence in Neural Machine Translation: A Study Using Oracle Experiments](https://md.research.elamur.ai/free/en/papers/stojanovski-2018-coreference-coherence-neural-machine-translation/)
- [Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/toral-2018-attaining-unattainable-reassessing-claims-human/)
- [Context-Aware Neural Machine Translation Learns Anaphora Resolution](https://md.research.elamur.ai/free/en/papers/voita-2018-context-aware-neural-machine-translation/)
- [When a Good Translation is Wrong in Context: Context-Aware Machine Translation Improves on Deixis, Ellipsis, and Lexical Cohesion](https://md.research.elamur.ai/free/en/papers/voita-2019-when-good-translation-is-wrong/)
- [One Model to Learn Both: Zero Pronoun Prediction and Translation](https://md.research.elamur.ai/free/en/papers/wang-2019-one-model-learn-both-zero/)
- [Document-Level Machine Translation with Large Language Models](https://md.research.elamur.ai/free/en/papers/wang-2023-document-level-machine-translation-large/)
- [GuoFeng: A Benchmark for Zero Pronoun Recovery and Translation](https://md.research.elamur.ai/free/en/papers/xu-2022-guofeng-benchmark-zero-pronoun-recovery/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wang2017exploiting,
  title = {{Exploiting Cross-Sentence Context for Neural Machine Translation}},
  author = {Wang, Longyue and Tu, Zhaopeng and Way, Andy and Liu, Qun},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1301},
  eprint = {1704.04347},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d17-1301}
}
```

- [← previous Exploiting Argument Information to Improve Event Detection via Supervised Attention Mechanisms](https://md.research.elamur.ai/free/en/papers/liu-2017-exploiting-argument-information-improve-event/)
- [next → Exploring Joint Neural Model for Sentence Level Discourse Parsing and Sentiment Analysis](https://md.research.elamur.ai/free/en/papers/nejat-2017-exploring-joint-neural-model-sentence/)
- [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/page/2/)

## References

22 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the 3rd International Conference on Learning Representations. San Diego, USA, pages 1–15. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2015%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+In+Proceedings+of+the+3rd+International+Conference+on+Learning+Represen) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
2. Iacer Calixto, Qun Liu, and Nick Campbell. 2017. Incorporating global visual features into attention-based neural machine translation. arXiv preprint arXiv:1701.06521 . [find](https://scholar.google.com/scholar?q=Iacer+Calixto%2C+Qun+Liu%2C+and+Nick+Campbell%2E+2017%2E+Incorporating+global+visual+features+into+attention%2Dbased+neural+machine+translation%2E+arXiv+preprint+arXiv%3A1701%2E06521+%2E)
3. Marine Carpuat and Michel Simard. 2012. The trouble with smt consistency. In Proceedings of the 7th Workshop on Statistical Machine Translation. Montreal, Quebec, Canada, pages 442–449. [find](https://scholar.google.com/scholar?q=Marine+Carpuat+and+Michel+Simard%2E+2012%2E+The+trouble+with+smt+consistency%2E+In+Proceedings+of+the+7th+Workshop+on+Statistical+Machine+Translation%2E+Montreal%2C+Quebec%2C+Canada%2C+pages+442%E2%80%93449%2E)
4. Heeyoul Choi, Kyunghyun Cho, and Yoshua Bengio. 2016. Context-dependent word representation for neural machine translation. arXiv preprint arXiv:1607.00578 . — [arXiv:1607.00578](https://arxiv.org/abs/1607.00578) [find](https://scholar.google.com/scholar?q=Heeyoul+Choi%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2016%2E+Context%2Ddependent+word+representation+for+neural+machine+translation%2E+arXiv+preprint+arXiv%3A1607%2E00578+%2E+%E2%80%94+%5BarXiv%3A1607%2E00578%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs)
5. Michael Collins, Philipp Koehn, and Ivona Kucerova. 2005. Clause restructuring for statistical machine translation. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics. Ann Arbor, Michigan, pages 531–540. [find](https://scholar.google.com/scholar?q=Michael+Collins%2C+Philipp+Koehn%2C+and+Ivona+Kucerova%2E+2005%2E+Clause+restructuring+for+statistical+machine+translation%2E+In+Proceedings+of+the+43rd+Annual+Meeting+of+the+Association+for+Computational+Lingu) [our card](https://md.research.elamur.ai/free/en/papers/collins-2005-clause-restructuring-statistical-machine-translation/)
6. Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015. Multi-task learning for multiple language translation. In Proceedings of the 53rd Annual Meeting of the Assocaition for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing. Beijing, Chin… [find](https://scholar.google.com/scholar?q=Daxiang+Dong%2C+Hua+Wu%2C+Wei+He%2C+Dianhai+Yu%2C+and+Haifeng+Wang%2E+2015%2E+Multi%2Dtask+learning+for+multiple+language+translation%2E+In+Proceedings+of+the+53rd+Annual+Meeting+of+the+Assocaition+for+Computational+) [our card](https://md.research.elamur.ai/free/en/papers/dong-2015-multi-task-learning-multiple-language/)
7. Sebastien Jean, Stanislas Lauly, Orhan Firat, and Kyunghyun Cho. 2017. Does neural machine translation benefit from larger context? arXiv preprint arXiv:1704.05135 . [find](https://scholar.google.com/scholar?q=Sebastien+Jean%2C+Stanislas+Lauly%2C+Orhan+Firat%2C+and+Kyunghyun+Cho%2E+2017%2E+Does+neural+machine+translation+benefit+from+larger+context%3F+arXiv+preprint+arXiv%3A1704%2E05135+%2E)
8. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Seattle, Washington, USA, pages 1700–1709. [find](https://scholar.google.com/scholar?q=Nal+Kalchbrenner+and+Phil+Blunsom%2E+2013%2E+Recurrent+continuous+translation+models%2E+In+Proceedings+of+the+2013+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2E+Association+for+Computation) [our card](https://md.research.elamur.ai/free/en/papers/kalchbrenner-2013-recurrent-continuous-translation-models/)
9. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Pro… [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2C+Hieu+Hoang%2C+Alexandra+Birch%2C+Chris+Callison%2DBurch%2C+Marcello+Federico%2C+Nicola+Bertoldi%2C+Brooke+Cowan%2C+Wade+Shen%2C+Christine+Moran%2C+Richard+Zens%2C+Chris+Dyer%2C+Ondr%CC%8Cej+Bojar%2C+Alexandra+Const)
10. Thang Luong, Hieu Pham, and D. Christopher Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Lisbon, Portugal, pages 1412–1421. [find](https://scholar.google.com/scholar?q=Thang+Luong%2C+Hieu+Pham%2C+and+D%2E+Christopher+Manning%2E+2015%2E+Effective+approaches+to+attention%2Dbased+neural+machine+translation%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Methods+in+Natural+Lang)
11. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. Philadelphia, Pennsylvania, USA, pages 311–318. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+Bleu%3A+A+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+the+40th+Annual+Meeting+on+Association+for+Computat) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
12. Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel Läubli, Antonio Valerio Miceli Barone, Jozef Mokry, et al. 2017. Nematus: a toolkit for neural machine translation. arXiv preprint arXiv:1703.04357 . [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Orhan+Firat%2C+Kyunghyun+Cho%2C+Alexandra+Birch%2C+Barry+Haddow%2C+Julian+Hitschler%2C+Marcin+Junczys%2DDowmunt%2C+Samuel+La%CC%88ubli%2C+Antonio+Valerio+Miceli+Barone%2C+Jozef+Mokry%2C+et+al%2E+2017%2E+Nematus%3A+a+) [our card](https://md.research.elamur.ai/free/en/papers/sennrich-2017-nematus-toolkit-neural-machine-translation/)
13. Rico Sennrich and Barry Haddow. 2016. Proceedings of the First Conference on Machine Translation: Volume 1, Research Papers, Berlin, Germany, chapter Linguistic Input Features Improve Neural Machine Translation, pages 83–91. [find](https://scholar.google.com/scholar?q=Rico+Sennrich+and+Barry+Haddow%2E+2016%2E+Proceedings+of+the+First+Conference+on+Machine+Translation%3A+Volume+1%2C+Research+Papers%2C+Berlin%2C+Germany%2C+chapter+Linguistic+Input+Features+Improve+Neural+Machine+T) [our card](https://md.research.elamur.ai/free/en/papers/sennrich-2016-linguistic-input-features-improve-neural/)
14. Iulian V. Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence. Phoenix, Arizona, pages 3776–3783. [find](https://scholar.google.com/scholar?q=Iulian+V%2E+Serban%2C+Alessandro+Sordoni%2C+Yoshua+Bengio%2C+Aaron+Courville%2C+and+Joelle+Pineau%2E+2016%2E+Building+end%2Dto%2Dend+dialogue+systems+using+generative+hierarchical+neural+network+models%2E+In+Proceedings+) [our card](https://md.research.elamur.ai/free/en/papers/serban-2015-building-end-end-dialogue-systems/)
15. Alessandro Sordoni, Yoshua Bengio, Hossein Vahabi, Christina Lioma, Jakob Grue Simonsen, and Jian-Yun Nie. 2015. A hierarchical recurrent encoder-decoder for generative context-aware query suggestion. In Proceedings of the 24th ACM International Conference on Information and Knowledge Management. Me… [find](https://scholar.google.com/scholar?q=Alessandro+Sordoni%2C+Yoshua+Bengio%2C+Hossein+Vahabi%2C+Christina+Lioma%2C+Jakob+Grue+Simonsen%2C+and+Jian%2DYun+Nie%2E+2015%2E+A+hierarchical+recurrent+encoder%2Ddecoder+for+generative+context%2Daware+query+suggestion%2E)
16. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of the 2014 Neural Information Processing Systems. Montreal, Canada, pages 3104–3112. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Proceedings+of+the+2014+Neural+Information+Processing+Systems%2E+Montreal%2C+Canada%2C+pages+3104%E2%80%933)
17. Zhaopeng Tu, Yang Liu, Zhengdong Lu, Xiaohua Liu, and Hang Li. 2017a. Context gates for neural machine translation. Transactions of the Association for Computational Linguistics . [find](https://scholar.google.com/scholar?q=Zhaopeng+Tu%2C+Yang+Liu%2C+Zhengdong+Lu%2C+Xiaohua+Liu%2C+and+Hang+Li%2E+2017a%2E+Context+gates+for+neural+machine+translation%2E+Transactions+of+the+Association+for+Computational+Linguistics+%2E)
18. Zhaopeng Tu, Yang Liu, Lifeng Shang, Xiaohua Liu, and Hang Li. 2017b. Neural machine translation with reconstruction. In Proceedings of the 31th AAAI Conference on Artificial Intelligence (AAAI-17). San Francisco, California, USA, pages 3097– 3103. [find](https://scholar.google.com/scholar?q=Zhaopeng+Tu%2C+Yang+Liu%2C+Lifeng+Shang%2C+Xiaohua+Liu%2C+and+Hang+Li%2E+2017b%2E+Neural+machine+translation+with+reconstruction%2E+In+Proceedings+of+the+31th+AAAI+Conference+on+Artificial+Intelligence+%28AAAI%2D17%29%2E+S)
19. Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Modeling coverage for neural machine translation. In Proceedings of the 54th annual meeting of the Association for Computational Linguistics. Berlin, Germany, pages 76–85. [find](https://scholar.google.com/scholar?q=Zhaopeng+Tu%2C+Zhengdong+Lu%2C+Yang+Liu%2C+Xiaohua+Liu%2C+and+Hang+Li%2E+2016%2E+Modeling+coverage+for+neural+machine+translation%2E+In+Proceedings+of+the+54th+annual+meeting+of+the+Association+for+Computational+Li) [our card](https://md.research.elamur.ai/free/en/papers/tu-2016-modeling-coverage-neural-machine-translation/)
20. Oriol Vinyals and Quoc Le. 2015. A neural conversational model. In Proceedings of the International Conference on Machine Learning, Deep Learning Workshop. pages 1–8. [find](https://scholar.google.com/scholar?q=Oriol+Vinyals+and+Quoc+Le%2E+2015%2E+A+neural+conversational+model%2E+In+Proceedings+of+the+International+Conference+on+Machine+Learning%2C+Deep+Learning+Workshop%2E+pages+1%E2%80%938%2E) [our card](https://md.research.elamur.ai/free/en/papers/vinyals-2015-neural-conversational-model/)
21. Tong Xiao, Jingbo Zhu, Shujie Yao, and Hao Zhang. 2011. Document-level consistency verification in machine translation. In Machine Translation Summit. Xiamen, China, volume 13, pages 131–138. [find](https://scholar.google.com/scholar?q=Tong+Xiao%2C+Jingbo+Zhu%2C+Shujie+Yao%2C+and+Hao+Zhang%2E+2011%2E+Document%2Dlevel+consistency+verification+in+machine+translation%2E+In+Machine+Translation+Summit%2E+Xiamen%2C+China%2C+volume+13%2C+pages+131%E2%80%93138%2E)
22. Barret Zoph and Kevin Knight. 2016. Multi-source neural translation. arXiv preprint arXiv:1601.00710 . [find](https://scholar.google.com/scholar?q=Barret+Zoph+and+Kevin+Knight%2E+2016%2E+Multi%2Dsource+neural+translation%2E+arXiv+preprint+arXiv%3A1601%2E00710+%2E) [our card](https://md.research.elamur.ai/free/en/papers/zoph-2016-multi-source-neural-translation/)