---
title: "Coreference and Coherence in Neural — Stojanovski 2018"
description: "Standard NMT systems translate each sentence in isolation."
authors: ["Dario Stojanovski", "Alexander Fraser"]
year: 2018
venue: "Proceedings of the Third Conference on Machine Translation: Research Papers"
source: "https://aclanthology.org/W18-6306/"
doi: "10.18653/v1/w18-6306"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/stojanovski-2018-coreference-coherence-neural-machine-translation/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/stojanovski-2018-coreference-coherence-neural-machine-translation/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2018](https://md.research.elamur.ai/free/papers/2018/) / Coreference and Coherence in Neural Machine Translation: A Study Using Oracle Experiments

# Coreference and Coherence in Neural Machine Translation: A Study Using Oracle Experiments

**Authors:** Dario Stojanovski, [Alexander Fraser](https://md.research.elamur.ai/free/authors/alexander-fraser/) · **Year:** 2018 · **Venue:** Proceedings of the Third Conference on Machine Translation: Research Papers · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/W18-6306/) · [PDF](https://aclanthology.org/W18-6306.pdf) · [DOI 10.18653/v1/w18-6306](https://doi.org/10.18653/v1/w18-6306) · [OpenAlex](https://openalex.org/W2902537726)

**Citations:** 30 citations (openalex-snapshot · 2026-10-14) · [16](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Standard NMT systems translate each sentence in isolation. They therefore struggle with discourse phenomena such as anaphoric pronouns (coreference) and consistent word choice across a document (coherence). These effects are also hard to measure with BLEU. Stojanovski and Fraser address both problems with automatically built oracle setups for English→German subtitle translation. In these setups the model receives an explicit contextual hint: the gold previous target sentence, the target-side pronouns (marked in the source with XPRONOUN), or the target translation of a word repeated from the prior sentence (marked with XREP). Each oracle also has a fair variant, which uses no target-side knowledge, and a noisy variant, which adds the previous target sentence to test robustness. The authors propose an RNN model whose context attention operates only over word embeddings, plus a context-aware Transformer. They compare both against concatenation baselines. Given clear signals, context-aware models gain strongly in BLEU, and concatenation models score highest but run slower. Because the oracles need no hand-built challenge sets, the method transfers easily to other domains and language pairs.

## Key points
- On English→German subtitles, the coreference oracle yields BLEU improvements of as much as 7.02, while the coherence oracle yields gains of at most 1.89 BLEU.
- The fair coreference variant obtains antecedents with CorefAnnotator from Stanford CoreNLP. The fair previous-sentence variant uses a baseline system's translation, or the previous source sentence, as context.
- The models compared are an NMT(RNN) with embedding-level context attention, a context-aware NMT(Transformer), and concatenation models following Tiedemann and Scherrer (2017). The paper also checks its oracles against the challenge sets of Bawden et al. (2018).
- In decoding the development set, the proposed RNN context model took 1233 s and the concatenation model 2063 s. Concatenation cost roughly 900 s more for each additional context sentence.
- Attention analysis on noisy oracles shows that the concatenation model locates XREP signals reliably. The embedding-attention model often attends to the correct word but still fails to generate it.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

Topics: [Coreference resolution](https://md.research.elamur.ai/free/topics/coreference-resolution/) · [Machine translation](https://md.research.elamur.ai/free/topics/machine-translation/)

## Cited by

Cited by

16 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 12 not yet.

- [CorefInst: Leveraging LLMs for Multilingual Coreference Resolution](https://md.research.elamur.ai/free/papers/arslan-2026-corefinst-leveraging-llms-multilingual-coreference/)
- [Incorporating Syntax and Semantics in Coreference Resolution with Heterogeneous Graph Attention Network](https://md.research.elamur.ai/free/papers/jiang-2021-incorporating-syntax-semantics-coreference-resolution/)
- [Maverick: Efficient and Accurate Coreference Resolution Defying Recent Trends](https://md.research.elamur.ai/free/papers/martinelli-2024-maverick-efficient-accurate-coreference-resolution/)
- [F-coref: Fast, Accurate and Easy to Use Coreference Resolution](https://md.research.elamur.ai/free/papers/otmazgin-2022-f-coref-fast-accurate-easy/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{stojanovski2018coreference,
  title = {{Coreference and Coherence in Neural Machine Translation: A Study Using Oracle Experiments}},
  author = {Stojanovski, Dario and Fraser, Alexander},
  booktitle = {Proceedings of the Third Conference on Machine Translation: Research Papers},
  year = {2018},
  doi = {10.18653/v1/w18-6306},
  url = {https://doi.org/10.18653/v1/w18-6306}
}
```

- [← previous Conversations Gone Awry: Detecting Early Signs of Conversational Failure](https://md.research.elamur.ai/free/papers/zhang-2018-conversations-gone-awry-detecting-early/)
- [next → Creating a Translation Matrix of the Bible's Names Across 591 Languages](https://md.research.elamur.ai/free/papers/wu-2018-creating-translation-matrix-bible-s/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/papers/2018/)

## References

30 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Christian Hardmeier, Sara Stymne, Jörg Tiedemann, and Joakim Nivre. 2013. Docent: A document-level decoder for phrase-based statistical machine translation. In ACL 2013 (51st Annual Meeting of the Association for Computational Linguistics); 4-9 August 2013; Sofia, Bulgaria, pages 193–198. [find](https://scholar.google.com/scholar?q=Christian+Hardmeier%2C+Sara+Stymne%2C+Jo%CC%88rg+Tiedemann%2C+and+Joakim+Nivre%2E+2013%2E+Docent%3A+A+document%2Dlevel+decoder+for+phrase%2Dbased+statistical+machine+translation%2E+In+ACL+2013+%2851st+Annual+Meeting+of+the+As)
2. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450. [find](https://scholar.google.com/scholar?q=Jimmy+Lei+Ba%2C+Jamie+Ryan+Kiros%2C+and+Geoffrey+E+Hinton%2E+2016%2E+Layer+normalization%2E+arXiv+preprint+arXiv%3A1607%2E06450%2E) [our card](https://md.research.elamur.ai/free/papers/ba-2016-layer-normalization/)
3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the 3rd International Conference on Learning Representations, ICLR ’15. ArXiv: 1409.0473. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2015%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+In+Proceedings+of+the+3rd+International+Conference+on+Learning+Represen) [our card](https://md.research.elamur.ai/free/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
4. Felix Hieber, Tobias Domhan, Michael Denkowski, David Vilar, Artem Sokolov, Ann Clifton, and Matt Post. 2017. Sockeye: A Toolkit for Neural Machine Translation. ArXiv e-prints. [find](https://scholar.google.com/scholar?q=Felix+Hieber%2C+Tobias+Domhan%2C+Michael+Denkowski%2C+David+Vilar%2C+Artem+Sokolov%2C+Ann+Clifton%2C+and+Matt+Post%2E+2017%2E+Sockeye%3A+A+Toolkit+for+Neural+Machine+Translation%2E+ArXiv+e%2Dprints%2E)
5. Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. 2018. Evaluating Discourse Phenomena in Neural Machine Translation. In NAACL 2018, New Orleans, USA. [find](https://scholar.google.com/scholar?q=Rachel+Bawden%2C+Rico+Sennrich%2C+Alexandra+Birch%2C+and+Barry+Haddow%2E+2018%2E+Evaluating+Discourse+Phenomena+in+Neural+Machine+Translation%2E+In+NAACL+2018%2C+New+Orleans%2C+USA%2E) [our card](https://md.research.elamur.ai/free/papers/bawden-2017-evaluating-discourse-phenomena-neural-machine/)
6. Sebastien Jean, Stanislas Lauly, Orhan Firat, and Kyunghyun Cho. 2017. Does neural machine translation benefit from larger context? arXiv preprint arXiv:1704.05135. [find](https://scholar.google.com/scholar?q=Sebastien+Jean%2C+Stanislas+Lauly%2C+Orhan+Firat%2C+and+Kyunghyun+Cho%2E+2017%2E+Does+neural+machine+translation+benefit+from+larger+context%3F+arXiv+preprint+arXiv%3A1704%2E05135%2E)
7. Marine Carpuat. 2009. One translation per discourse. In Proceedings of the Workshop on Semantic Evaluations: Recent Achievements and Future Directions, pages 19–27. [find](https://scholar.google.com/scholar?q=Marine+Carpuat%2E+2009%2E+One+translation+per+discourse%2E+In+Proceedings+of+the+Workshop+on+Semantic+Evaluations%3A+Recent+Achievements+and+Future+Directions%2C+pages+19%E2%80%9327%2E)
8. Yangfeng Ji, Trevor Cohn, Lingpeng Kong, Chris Dyer, and Jacob Eisenstein. 2015. Document context language models. arXiv preprint arXiv:1511.03962. — [arXiv:1511.03962](https://arxiv.org/abs/1511.03962) [find](https://scholar.google.com/scholar?q=Yangfeng+Ji%2C+Trevor+Cohn%2C+Lingpeng+Kong%2C+Chris+Dyer%2C+and+Jacob+Eisenstein%2E+2015%2E+Document+context+language+models%2E+arXiv+preprint+arXiv%3A1511%2E03962%2E+%E2%80%94+%5BarXiv%3A1511%2E03962%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1511%2E0396)
9. Marine Carpuat and Michel Simard. 2012. The trouble with smt consistency. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 442–449. [find](https://scholar.google.com/scholar?q=Marine+Carpuat+and+Michel+Simard%2E+2012%2E+The+trouble+with+smt+consistency%2E+In+Proceedings+of+the+Seventh+Workshop+on+Statistical+Machine+Translation%2C+pages+442%E2%80%93449%2E)
10. Ronan Le Nagard and Philipp Koehn. 2010. Aiding pronoun translation with co-reference resolution. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, pages 252–261. [find](https://scholar.google.com/scholar?q=Ronan+Le+Nagard+and+Philipp+Koehn%2E+2010%2E+Aiding+pronoun+translation+with+co%2Dreference+resolution%2E+In+Proceedings+of+the+Joint+Fifth+Workshop+on+Statistical+Machine+Translation+and+MetricsMATR%2C+pages+2) [our card](https://md.research.elamur.ai/free/papers/nagard-2010-aiding-pronoun-translation-co-reference/)
11. Kevin Clark and Christopher D. Manning. 2016a. Deep reinforcement learning for mention-ranking coreference models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2256–2262. [find](https://scholar.google.com/scholar?q=Kevin+Clark+and+Christopher+D%2E+Manning%2E+2016a%2E+Deep+reinforcement+learning+for+mention%2Dranking+coreference+models%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Language+Proces) [our card](https://md.research.elamur.ai/free/papers/clark-2016-deep-reinforcement-learning-mention-ranking/)
12. Sharid Loáiciga, Sara Stymne, Preslav Nakov, Christian Hardmeier, Jörg Tiedemann, Mauro Cettolo, and Yannick Versley. 2017. Findings of the 2017 discomt shared task on cross-lingual pronoun prediction. In The Third Workshop on Discourse in Machine Translation. [find](https://scholar.google.com/scholar?q=Sharid+Loa%CC%81iciga%2C+Sara+Stymne%2C+Preslav+Nakov%2C+Christian+Hardmeier%2C+Jo%CC%88rg+Tiedemann%2C+Mauro+Cettolo%2C+and+Yannick+Versley%2E+2017%2E+Findings+of+the+2017+discomt+shared+task+on+cross%2Dlingual+pronoun+predicti)
13. Kevin Clark and Christopher D. Manning. 2016b. Improving coreference resolution by learning entity-level distributed representations. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 643–653. [find](https://scholar.google.com/scholar?q=Kevin+Clark+and+Christopher+D%2E+Manning%2E+2016b%2E+Improving+coreference+resolution+by+learning+entity%2Dlevel+distributed+representations%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+C) [our card](https://md.research.elamur.ai/free/papers/clark-2016-improving-coreference-resolution-learning-entity/)
14. Thomas Mueller, Helmut Schmid, and Hinrich Schütze. 2013. Efficient higher-order CRFs for morphological tagging. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 322–332, Seattle, Washington, USA. [find](https://scholar.google.com/scholar?q=Thomas+Mueller%2C+Helmut+Schmid%2C+and+Hinrich+Schu%CC%88tze%2E+2013%2E+Efficient+higher%2Dorder+CRFs+for+morphological+tagging%2E+In+Proceedings+of+the+2013+Conference+on+Empirical+Methods+in+Natural+Language+Process)
15. Yarin Gal and Zoubin Ghahramani. 2016. A theoretically grounded application of dropout in recurrent neural networks. In Advances in neural information processing systems, pages 1019–1027. [find](https://scholar.google.com/scholar?q=Yarin+Gal+and+Zoubin+Ghahramani%2E+2016%2E+A+theoretically+grounded+application+of+dropout+in+recurrent+neural+networks%2E+In+Advances+in+neural+information+processing+systems%2C+pages+1019%E2%80%931027%2E) [our card](https://md.research.elamur.ai/free/papers/gal-2015-theoretically-grounded-application-dropout-recurrent/)
16. Ofir Press and Lior Wolf. 2017. Using the output embedding to improve language models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 157–163. [find](https://scholar.google.com/scholar?q=Ofir+Press+and+Lior+Wolf%2E+2017%2E+Using+the+output+embedding+to+improve+language+models%2E+In+Proceedings+of+the+15th+Conference+of+the+European+Chapter+of+the+Association+for+Computational+Linguistics%3A+V) [our card](https://md.research.elamur.ai/free/papers/press-2017-output-embedding-improve-language-models/)
17. Annette Rios Gonzales, Laura Mascarell, and Rico Sennrich. 2017. Improving word sense disambiguation in neural machine translation with sense embeddings. In Proceedings of the Second Conference on Machine Translation, pages 11–19. [find](https://scholar.google.com/scholar?q=Annette+Rios+Gonzales%2C+Laura+Mascarell%2C+and+Rico+Sennrich%2E+2017%2E+Improving+word+sense+disambiguation+in+neural+machine+translation+with+sense+embeddings%2E+In+Proceedings+of+the+Second+Conference+on+Mac) [our card](https://md.research.elamur.ai/free/papers/gonzales-2017-improving-word-sense-disambiguation-neural/)
18. Tim Salimans and Diederik P Kingma. 2016. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems, pages 901–909. [find](https://scholar.google.com/scholar?q=Tim+Salimans+and+Diederik+P+Kingma%2E+2016%2E+Weight+normalization%3A+A+simple+reparameterization+to+accelerate+training+of+deep+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+)
19. Liane Kirsten Guillou. 2016. Incorporating pronoun function into statistical machine translation. Ph.D. thesis, The University of Edinburgh, UK. [find](https://scholar.google.com/scholar?q=Liane+Kirsten+Guillou%2E+2016%2E+Incorporating+pronoun+function+into+statistical+machine+translation%2E+Ph%2ED%2E+thesis%2C+The+University+of+Edinburgh%2C+UK%2E)
20. Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel Läubli, Antonio Valerio Miceli Barone, Jozef Mokry, and Maria Nadejde. 2017. Nematus: a toolkit for neural machine translation. In Proceedings of the Software Demonstrations of… [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Orhan+Firat%2C+Kyunghyun+Cho%2C+Alexandra+Birch%2C+Barry+Haddow%2C+Julian+Hitschler%2C+Marcin+Junczys%2DDowmunt%2C+Samuel+La%CC%88ubli%2C+Antonio+Valerio+Miceli+Barone%2C+Jozef+Mokry%2C+and+Maria+Nadejde%2E+2017%2E) [our card](https://md.research.elamur.ai/free/papers/sennrich-2017-nematus-toolkit-neural-machine-translation/)
21. Christian Hardmeier. 2012. Discourse in statistical machine translation. a survey and a case study. Discours. Revue de linguistique, psycholinguistique et informatique. A journal of linguistics, psycholinguistics and computational linguistics, (11). [find](https://scholar.google.com/scholar?q=Christian+Hardmeier%2E+2012%2E+Discourse+in+statistical+machine+translation%2E+a+survey+and+a+case+study%2E+Discours%2E+Revue+de+linguistique%2C+psycholinguistique+et+informatique%2E+A+journal+of+linguistics%2C+psych)
22. Christian Hardmeier and Marcello Federico. 2010. Modelling pronominal anaphora in statistical machine translation. In IWSLT (International Workshop on Spoken Language Translation); Paris, France; December 2nd and 3rd, 2010., pages 283– 289. [find](https://scholar.google.com/scholar?q=Christian+Hardmeier+and+Marcello+Federico%2E+2010%2E+Modelling+pronominal+anaphora+in+statistical+machine+translation%2E+In+IWSLT+%28International+Workshop+on+Spoken+Language+Translation%29%3B+Paris%2C+France%3B+Dece) [our card](https://md.research.elamur.ai/free/papers/hardmeier-2010-modelling-pronominal-anaphora-statistical-machine/)
23. Iulian Vlad Serban, Alessandro Sordoni, Yoshua Bengio, Aaron C Courville, and Joelle Pineau. 2016. [find](https://scholar.google.com/scholar?q=Iulian+Vlad+Serban%2C+Alessandro+Sordoni%2C+Yoshua+Bengio%2C+Aaron+C+Courville%2C+and+Joelle+Pineau%2E+2016%2E)
24. Building end-to-end dialogue systems using generative hierarchical neural network models. In AAAI, pages 3776–3784. Jörg Tiedemann and Yves Scherrer. 2017. Neural machine translation with extended context. In Proceedings of the Third Workshop on Discourse in Machine Translation, pages 82–92. [find](https://scholar.google.com/scholar?q=Building+end%2Dto%2Dend+dialogue+systems+using+generative+hierarchical+neural+network+models%2E+In+AAAI%2C+pages+3776%E2%80%933784%2E+Jo%CC%88rg+Tiedemann+and+Yves+Scherrer%2E+2017%2E+Neural+machine+translation+with+extended+co) [our card](https://md.research.elamur.ai/free/papers/serban-2015-building-end-end-dialogue-systems/)
25. Zhaopeng Tu, Yang Liu, Shuming Shi, and Tong Zhang. 2017. Learning to remember translation history with a continuous cache. arXiv preprint arXiv:1711.09367. [find](https://scholar.google.com/scholar?q=Zhaopeng+Tu%2C+Yang+Liu%2C+Shuming+Shi%2C+and+Tong+Zhang%2E+2017%2E+Learning+to+remember+translation+history+with+a+continuous+cache%2E+arXiv+preprint+arXiv%3A1711%2E09367%2E)
26. Ferhan Ture, Douglas W Oard, and Philip Resnik. 2012. Encouraging consistent translation choices. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 417–426. [find](https://scholar.google.com/scholar?q=Ferhan+Ture%2C+Douglas+W+Oard%2C+and+Philip+Resnik%2E+2012%2E+Encouraging+consistent+translation+choices%2E+In+Proceedings+of+the+2012+Conference+of+the+North+American+Chapter+of+the+Association+for+Computation)
27. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 6000–6010. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+Advances+in+Neural+Information+Processi)
28. Elena Voita, Pavel Serdyukov, Rico Sennrich, and Ivan Titov. 2018. Context-Aware Neural Machine Translation Learns Anaphora Resolution. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1264–1274, Melbourne, Australia. [find](https://scholar.google.com/scholar?q=Elena+Voita%2C+Pavel+Serdyukov%2C+Rico+Sennrich%2C+and+Ivan+Titov%2E+2018%2E+Context%2DAware+Neural+Machine+Translation+Learns+Anaphora+Resolution%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for) [our card](https://md.research.elamur.ai/free/papers/voita-2018-context-aware-neural-machine-translation/)
29. Longyue Wang, Zhaopeng Tu, Andy Way, and Qun Liu. 2017. Exploiting cross-sentence context for neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2826–2831. [find](https://scholar.google.com/scholar?q=Longyue+Wang%2C+Zhaopeng+Tu%2C+Andy+Way%2C+and+Qun+Liu%2E+2017%2E+Exploiting+cross%2Dsentence+context+for+neural+machine+translation%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Natural+Language)
30. Tian Wang and Kyunghyun Cho. 2016. Larger-context language modelling with recurrent neural network. In 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016 - Long Papers, volume 3, pages 1319–1329. [find](https://scholar.google.com/scholar?q=Tian+Wang+and+Kyunghyun+Cho%2E+2016%2E+Larger%2Dcontext+language+modelling+with+recurrent+neural+network%2E+In+54th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+ACL+2016+%2D+Long+Papers%2C+vol)