---
title: "Overcoming Catastrophic Forgetting During Domain Adaptation"
description: "When a general-domain neural machine translation model is fine-tuned on a small… 28.7 BLEU. 27 references, cited by 71 papers here."
authors: ["Brian Thompson", "Jeremy Gwinnup", "Huda Khayrallah", "Kevin Duh", "Philipp Koehn"]
year: 2019
venue: "Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)"
source: "https://aclanthology.org/N19-1209/"
doi: "10.18653/v1/n19-1209"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/thompson-2019-overcoming-catastrophic-forgetting-during-domain/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/thompson-2019-overcoming-catastrophic-forgetting-during-domain/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation

# Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation

**Authors:** [Brian Thompson](https://md.research.elamur.ai/free/en/authors/brian-thompson/), Jeremy Gwinnup, [Huda Khayrallah](https://md.research.elamur.ai/free/en/authors/huda-khayrallah/), [Kevin Duh](https://md.research.elamur.ai/free/en/authors/kevin-duh/) and [Philipp Koehn](https://md.research.elamur.ai/free/en/authors/philipp-koehn/) · **Year:** 2019 · **Venue:** [Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2019](https://md.research.elamur.ai/free/en/venues/naacl/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/N19-1209/) · [PDF](https://aclanthology.org/N19-1209.pdf) · [DOI 10.18653/v1/n19-1209](https://doi.org/10.18653/v1/n19-1209) · [OpenAlex](https://openalex.org/W2945383715)

**Citations:** 113 citations (OpenAlex, 2026-10-14) · [71](https://md.research.elamur.ai/free/en/papers/thompson-2019-overcoming-catastrophic-forgetting-during-domain/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
When a general-domain neural machine translation model is fine-tuned on a small in-domain corpus (continued training), it improves on the target domain but translates generic text much worse. The authors treat this as catastrophic forgetting. They adapt Elastic Weight Consolidation (EWC) to NMT and give a Bayesian derivation that does not require the two translation tasks to be independent. The diagonal of the empirical Fisher matrix, computed from back-propagation gradients, measures how important each parameter is for general-domain translation. A penalty then keeps those important parameters near their general-domain values, while less important ones adapt freely to the new domain.

The experiments use six directions (En↔De, En↔Ru, En↔Zh). General-domain data come from WMT17 and OpenSubtitles18, and in-domain patent data come from WIPO COPPA-V2. Across the full trade-off between in-domain and general-domain BLEU, EWC beats the earlier MCL distillation baseline in five of six directions. At equal in-domain quality, it recovers most of the general-domain BLEU lost by plain fine-tuning. Translation still uses a single model, with no added memory or compute.

## Key points
- **Setup:** Systems are two-layer LSTMs (512 hidden units) built in Sockeye, with separate source/target BPE vocabularies of about 30k. Code is released at github.com/thompsonb/sockeye_ewc.
- **Data:** The general-domain corpora are large (323M, 730M and 584M English words for De, Ru and Zh). The in-domain WIPO sets are small (3.2M, 0.81M and 0.80M words), and WIPO De↔En was cut to 100k lines.
- **Fisher estimation:** The Fisher diagonal is estimated on 500k training sentences in under an hour on a GTX 1080 Ti. Using more data or held-out data made no meaningful difference.
- **Results without in-domain loss:** EWC raises general-domain BLEU over continued training by roughly 8–19 points. For example, Zh→En goes from 9.9 to 28.7 BLEU, against 43.5 for the original model. MCL gives no gain for En→Zh or Zh→En.
- **Exception and sensitivity:** The only case where MCL wins is Ru→En, where it gives a small in-domain edge at lower general-domain quality. Both learning rate and regularization strength strongly shape the outcome for either method.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Johns Hopkins University](https://md.research.elamur.ai/free/en/universities/johns-hopkins-university/) — University, Baltimore, United States · 187 works · 92,194 citations

Also on the byline: [United States Air Force Research Laboratory](https://ror.org/02e2egq70).

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🔁 Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · 958 works

## Cite

Cite as: Brian Thompson, Jeremy Gwinnup, Huda Khayrallah, Kevin Duh and Philipp Koehn. 2019. Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{thompson2019overcoming,
  title = {{Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation}},
  author = {Brian Thompson, Jeremy Gwinnup, Huda Khayrallah, Kevin Duh and Philipp Koehn},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1209},
  url = {https://doi.org/10.18653/v1/n19-1209}
}
```

- [← previous compare-mt: A Tool for Holistic Comparison of Language Generation Systems](https://md.research.elamur.ai/free/en/papers/neubig-2019-compare-mt-tool-holistic-comparison/)
- [next → Trick Me If You Can: Human-in-the-Loop Generation of Adversarial Examples for Question Answering](https://md.research.elamur.ai/free/en/papers/wallace-2019-trick-me-if-you-can/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/)

## References

27 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. [Findings of the 2017 conference on machine transla…](https://md.research.elamur.ai/free/en/papers/bojar-2017-findings-2017-conference-machine-translation/)
2. Patrick Cadwell, Sharon OBrien, and Carlos SC Teixeira. 2018. Resistance and accommodation: factors for the (non-) adoption of machine translation among professional translators. Perspectives, 26(3):301–321. [find](https://scholar.google.com/scholar?q=Patrick+Cadwell%2C+Sharon+OBrien%2C+and+Carlos+SC+Teixeira%2E+2018%2E+Resistance+and+accommodation%3A+factors+for+the+%28non%2D%29+adoption+of+machine+translation+among+professional+translators%2E+Perspectives%2C+26%283%29%3A3)
3. Chenhui Chu, Raj Dabre, and Sadao Kurohashi. 2017. [An empirical comparison of domain adaptation methods for neural machine translation](https://md.research.elamur.ai/free/en/papers/chu-2017-empirical-comparison-domain-adaptation-methods/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 385–391. Association for Computational Li…
4. Huda Khayrallah, Brian Thompson, Kevin Duh, and Philipp Koehn. 2018. Regularized training objective for continued training for domain adaptation in neural machine translation. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 36–44. Association for Computational… [find](https://scholar.google.com/scholar?q=Huda+Khayrallah%2C+Brian+Thompson%2C+Kevin+Duh%2C+and+Philipp+Koehn%2E+2018%2E+Regularized+training+objective+for+continued+training+for+domain+adaptation+in+neural+machine+translation%2E+In+Proceedings+of+the+2n)
5. Yoon Kim and Alexander M. Rush. 2016. [Sequence-level knowledge distillation](https://md.research.elamur.ai/free/en/papers/kim-2016-sequence-level-knowledge-distillation/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317–1327. Association for Computational Linguistics.
6. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. [find](https://scholar.google.com/scholar?q=Diederik+P+Kingma+and+Jimmy+Ba%2E+2014%2E+Adam%3A+A+method+for+stochastic+optimization%2E+arXiv+preprint+arXiv%3A1412%2E6980%2E)
7. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, page 2016118… [find](https://scholar.google.com/scholar?q=James+Kirkpatrick%2C+Razvan+Pascanu%2C+Neil+Rabinowitz%2C+Joel+Veness%2C+Guillaume+Desjardins%2C+Andrei+A+Rusu%2C+Kieran+Milan%2C+John+Quan%2C+Tiago+Ramalho%2C+Agnieszka+Grabska%2DBarwinska%2C+et+al%2E+2017%2E+Overcoming+catas)
8. Markus Freitag and Yaser Al-Onaizan. 2016. Fast Domain Adaptation for Neural Machine Translation. CoRR, abs/1612.06897. [find](https://scholar.google.com/scholar?q=Markus+Freitag+and+Yaser+Al%2DOnaizan%2E+2016%2E+Fast+Domain+Adaptation+for+Neural+Machine+Translation%2E+CoRR%2C+abs%2F1612%2E06897%2E)
9. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open Source Toolkit for Statistical Machine Translation. In Proc… [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2C+Hieu+Hoang%2C+Alexandra+Birch%2C+Chris+Callison%2DBurch%2C+Marcello+Federico%2C+Nicola+Bertoldi%2C+Brooke+Cowan%2C+Wade+Shen%2C+Christine+Moran%2C+Richard+Zens%2C+Chris+Dyer%2C+Ondrej+Bojar%2C+Alexandra+Consta)
10. Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. 2013. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211. [find](https://scholar.google.com/scholar?q=Ian+J+Goodfellow%2C+Mehdi+Mirza%2C+Da+Xiao%2C+Aaron+Courville%2C+and+Yoshua+Bengio%2E+2013%2E+An+empirical+investigation+of+catastrophic+forgetting+in+gradient%2Dbased+neural+networks%2E+arXiv+preprint+arXiv%3A1312%2E621)
11. Philipp Koehn and Rebecca Knowles. 2017. Six Challenges for Neural Machine Translation. In Proceedings of the 1st Workshop on Neural Machine Translation (and Generation) at ACL. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Philipp+Koehn+and+Rebecca+Knowles%2E+2017%2E+Six+Challenges+for+Neural+Machine+Translation%2E+In+Proceedings+of+the+1st+Workshop+on+Neural+Machine+Translation+%28and+Generation%29+at+ACL%2E+Association+for+Comput)
12. Felix Hieber, Tobias Domhan, Michael Denkowski, David Vilar, Artem Sokolov, Ann Clifton, and Matt Post. 2017. Sockeye: A toolkit for neural machine translation. arXiv preprint arXiv:1712.05690. [find](https://scholar.google.com/scholar?q=Felix+Hieber%2C+Tobias+Domhan%2C+Michael+Denkowski%2C+David+Vilar%2C+Artem+Sokolov%2C+Ann+Clifton%2C+and+Matt+Post%2E+2017%2E+Sockeye%3A+A+toolkit+for+neural+machine+translation%2E+arXiv+preprint+arXiv%3A1712%2E05690%2E)
13. Pierre Lison, Jörg Tiedemann, and Milen Kouylekov. 2018. [OpenSubtitles2018: Statistical Rescoring of Sentence Alignments in Large, Noisy Parallel Corpora](https://md.research.elamur.ai/free/en/papers/lison-2018-opensubtitles2018-statistical-rescoring-sentence-alignments/). In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Res…
14. Praveen Dakwale and Christof Monz. 2017. Fine-tuning for neural machine translation with limited degradation across in-and out-of-domain data. Proceedings of the XVI Machine Translation Summit, page 117. [find](https://scholar.google.com/scholar?q=Praveen+Dakwale+and+Christof+Monz%2E+2017%2E+Fine%2Dtuning+for+neural+machine+translation+with+limited+degradation+across+in%2Dand+out%2Dof%2Ddomain+data%2E+Proceedings+of+the+XVI+Machine+Translation+Summit%2C+page+1)
15. Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. [Distilling the knowledge in a neural network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/). In NeurIPS Deep Learning and Representation Learning Workshop.
16. Marcin Junczys-Dowmunt, Bruno Pouliquen, and Christophe Mazenc. 2016. COPPA V2.0: Corpus of Parallel Patent Applications building large parallel corpora with GNU make. In 4th Workshop on Challenges in the Management of Large Corpora Workshop Programme. [find](https://scholar.google.com/scholar?q=Marcin+Junczys%2DDowmunt%2C+Bruno+Pouliquen%2C+and+Christophe+Mazenc%2E+2016%2E+COPPA+V2%2E0%3A+Corpus+of+Parallel+Patent+Applications+building+large+parallel+corpora+with+GNU+make%2E+In+4th+Workshop+on+Challenges+in)
17. Opinions, interpretations, conclusions and recommendations are those of the authors and are not necessarily endorsed by the United States Government. Cleared for public release on 14 Jan 2019. Originator reference number RH-19-119318. Case number 88ABW-2019-0136. [find](https://scholar.google.com/scholar?q=Opinions%2C+interpretations%2C+conclusions+and+recommendations+are+those+of+the+authors+and+are+not+necessarily+endorsed+by+the+United+States+Government%2E+Cleared+for+public+release+on+14+Jan+2019%2E+Origina)
18. Minh-Thang Luong and Christopher D. Manning. 2015. Stanford Neural Machine Translation Systems for Spoken Language Domain. In International Workshop on Spoken Language Translation, Da Nang, Vietnam. [find](https://scholar.google.com/scholar?q=Minh%2DThang+Luong+and+Christopher+D%2E+Manning%2E+2015%2E+Stanford+Neural+Machine+Translation+Systems+for+Spoken+Language+Domain%2E+In+International+Workshop+on+Spoken+Language+Translation%2C+Da+Nang%2C+Vietnam%2E)
19. David J. C. MacKay. 1992. A practical Bayesian framework for backpropagation networks. Neural Computation, 4(3):448–472. [find](https://scholar.google.com/scholar?q=David+J%2E+C%2E+MacKay%2E+1992%2E+A+practical+Bayesian+framework+for+backpropagation+networks%2E+Neural+Computation%2C+4%283%29%3A448%E2%80%93472%2E)
20. James Martens. 2014. New insights and perspectives on the natural gradient method. arXiv preprint arXiv:1412.1193. — [arXiv:1412.1193](https://arxiv.org/abs/1412.1193) [find](https://scholar.google.com/scholar?q=James+Martens%2E+2014%2E+New+insights+and+perspectives+on+the+natural+gradient+method%2E+arXiv+preprint+arXiv%3A1412%2E1193%2E+%E2%80%94+%5BarXiv%3A1412%2E1193%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1412%2E1193%29)
21. Antonio Valerio Miceli Barone, Barry Haddow, Ulrich Germann, and Rico Sennrich. 2017. [Regularization techniques for fine-tuning in neural machine translation](https://md.research.elamur.ai/free/en/papers/barone-2017-regularization-techniques-fine-tuning-neural/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1489–1494, Copenhagen, Denmark. Association…
22. Razvan Pascanu and Yoshua Bengio. 2013. Revisiting natural gradient for deep networks. arXiv preprint arXiv:1301.3584. — [arXiv:1301.3584](https://arxiv.org/abs/1301.3584) [find](https://scholar.google.com/scholar?q=Razvan+Pascanu+and+Yoshua+Bengio%2E+2013%2E+Revisiting+natural+gradient+for+deep+networks%2E+arXiv+preprint+arXiv%3A1301%2E3584%2E+%E2%80%94+%5BarXiv%3A1301%2E3584%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1301%2E3584%29)
23. Victoria Porro Rodriguez, Lucia Morado Vazquez, and Pierrette Bouillon. 2017. Study on the use of machine translation and post-editing in swiss-based language service providers. Parallèles, (29 (2)). [find](https://scholar.google.com/scholar?q=Victoria+Porro+Rodriguez%2C+Lucia+Morado+Vazquez%2C+and+Pierrette+Bouillon%2E+2017%2E+Study+on+the+use+of+machine+translation+and+post%2Dediting+in+swiss%2Dbased+language+service+providers%2E+Paralle%CC%80les%2C+%2829+%282%29%29%2E)
24. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2015%2E+Improving+neural+machine+translation+models+with+monolingual+data%2E+arXiv+preprint+arXiv%3A1511%2E06709%2E)
25. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. [Neural Machine Translation of Rare Words with Subword Units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Berlin, Germany. Association for Computational Linguistics.
26. Brian Thompson, Huda Khayrallah, Antonios Anastasopoulos, Arya D. McCarthy, Kevin Duh, Rebecca Marvin, Paul McNamee, Jeremy Gwinnup, Tim Anderson, and Philipp Koehn. 2018. Freezing subnetworks to analyze domain adaptation in neural machine translation. In Proceedings of the Third Conference on Machi… [find](https://scholar.google.com/scholar?q=Brian+Thompson%2C+Huda+Khayrallah%2C+Antonios+Anastasopoulos%2C+Arya+D%2E+McCarthy%2C+Kevin+Duh%2C+Rebecca+Marvin%2C+Paul+McNamee%2C+Jeremy+Gwinnup%2C+Tim+Anderson%2C+and+Philipp+Koehn%2E+2018%2E+Freezing+subnetworks+to+anal)
27. David Vilar. 2018. Learning hidden unit contribution for adapting neural machine translation models. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), volume 2, pages 500–505. [find](https://scholar.google.com/scholar?q=David+Vilar%2E+2018%2E+Learning+hidden+unit+contribution+for+adapting+neural+machine+translation+models%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+Association+for+Computat)

*Updated 24 October 2026*