---
title: "Jointly Learning to Align and Translate with — Garg 2019"
description: "Neural models now lead machine translation, yet word alignment is still usually handled by statistical tools such as GIZA++."
authors: ["Sarthak Garg", "Stephan Peitz", "Udhyakumar Nallasamy", "Matthias Paulik"]
year: 2019
venue: "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)"
source: "https://aclanthology.org/D19-1453/"
doi: "10.18653/v1/d19-1453"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/garg-2019-jointly-learning-align-translate-transformer/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/garg-2019-jointly-learning-align-translate-transformer/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/) / Jointly Learning to Align and Translate with Transformer Models

# Jointly Learning to Align and Translate with Transformer Models

**Authors:** Sarthak Garg, Stephan Peitz, Udhyakumar Nallasamy, Matthias Paulik · **Year:** 2019 · **Venue:** Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D19-1453/) · [PDF](https://aclanthology.org/D19-1453.pdf) · [DOI 10.18653/v1/d19-1453](https://doi.org/10.18653/v1/d19-1453) · [OpenAlex](https://openalex.org/W2970045405)

**Citations:** 121 citations (openalex-snapshot · 2026-10-14) · [78](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Neural models now lead machine translation, yet word alignment is still usually handled by statistical tools such as GIZA++. Interpreting attention in the Transformer is also hard because it spreads across many layers and heads. Garg et al. (Apple) train a single Transformer to output both translations and alignments. They first show that averaging attention over the heads of the penultimate layer gives far better alignments than averaging over the whole network. They use these unsupervised alignments, or alternatively GIZA++ output, as labels. A cross-entropy alignment loss on one dedicated attention head is added to the usual NLL translation loss. Translation stays auto-regressive, but the alignment term sees the entire target sentence. Without relying on external aligners, the method is competitive with GIZA++ and surpasses prior Transformer-based alignment approaches, namely the work of Zenkel and colleagues (2019) and of Peter and colleagues (2017). With GIZA++ supervision, it beats GIZA++ on three public alignment sets, and translation quality does not drop. This matters for lexicon induction, dictionary-constrained translation and transferring markup into translated text.

## Key points
- **Layer average baseline:** heads are averaged within the penultimate layer, which aligns best on its own. On WMT'18 English→German, its test AER reaches its lowest point of 42.7% as early as epoch 5. This suggests that earlier checkpoints could give better labels.
- **Training objective:** the loss is L = L_t + λL_a. Only one alignment head in the penultimate layer is supervised, which leaves the other heads free for translation.
- **Target context:** the alignment loss is conditioned on the full target sentence, while the translation loss uses only past tokens.
- **Qualitative analysis:** on German→English, the model correctly links pronouns to their nouns (e.g., *them* to *Voraussetzungen*), links that GIZA++ misses. The authors attribute this to contextual encoding.
- **Code and limitation:** the code is released in fairseq (pull request #1095). The current self-training setup still requires two training runs.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

78 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 77 not yet.

- [Survey of Hallucination in Natural Language Generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{garg2019jointly,
  title = {{Jointly Learning to Align and Translate with Transformer Models}},
  author = {Garg, Sarthak and Peitz, Stephan and Nallasamy, Udhyakumar and Paulik, Matthias},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1453},
  url = {https://doi.org/10.18653/v1/d19-1453}
}
```

- [← previous Joint Transition-Based Models for Morpho-Syntactic Parsing: Parsing Strategies for MRLs and a Case Study from Modern Hebrew](https://md.research.elamur.ai/free/en/papers/tacl-2019-more-joint-transition-based-models-morpho/)
- [next → Justifying Recommendations using Distantly-Labeled Reviews and Fine-Grained Aspects](https://md.research.elamur.ai/free/en/papers/ni-2019-justifying-recommendations-distantly-labeled-reviews/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/)

## References

34 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Tamer Alkhouli, Gabriel Bretschner, and Hermann Ney. 2018. On the alignment problem in multi-head attention-based neural machine translation. In Conference on Empirical Methods in Natural Language Processing, pages 177–185, Brussels, Belgium. [find](https://scholar.google.com/scholar?q=Tamer+Alkhouli%2C+Gabriel+Bretschner%2C+and+Hermann+Ney%2E+2018%2E+On+the+alignment+problem+in+multi%2Dhead+attention%2Dbased+neural+machine+translation%2E+In+Conference+on+Empirical+Methods+in+Natural+Language+Pro)
2. Tamer Alkhouli and Hermann Ney. 2017. Biasing attention-based recurrent neural networks using external alignment information. In Conference on Empirical Methods in Natural Language Processing, pages 108–117, Copenhagen, Denmark. [find](https://scholar.google.com/scholar?q=Tamer+Alkhouli+and+Hermann+Ney%2E+2017%2E+Biasing+attention%2Dbased+recurrent+neural+networks+using+external+alignment+information%2E+In+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+1)
3. Philip Arthur, Graham Neubig, and Satoshi Nakamura. 2016. Incorporating discrete translation lexicons into neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1557–1567. [find](https://scholar.google.com/scholar?q=Philip+Arthur%2C+Graham+Neubig%2C+and+Satoshi+Nakamura%2E+2016%2E+Incorporating+discrete+translation+lexicons+into+neural+machine+translation%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Nat) [our card](https://md.research.elamur.ai/free/en/papers/arthur-2016-incorporating-discrete-translation-lexicons-into/)
4. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations, San Diego, CA, USA. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2015%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+In+International+Conference+on+Learning+Representations%2C+San+Diego%2C+CA%2C) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
5. Ondřej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 conference on machine translation (wmt18). In Conference on Statistical Machine Translation, pages 272– 303, Belgium, Brussels. [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Bojar%2C+Christian+Federmann%2C+Mark+Fishel%2C+Yvette+Graham%2C+Barry+Haddow%2C+Philipp+Koehn%2C+and+Christof+Monz%2E+2018%2E+Findings+of+the+2018+conference+on+machine+translation+%28wmt18%29%2E+In+Conference+on+S) [our card](https://md.research.elamur.ai/free/en/papers/bojar-2018-findings-2018-conference-machine-translation/)
6. Peter F. Brown, Vincent J. Della Pietra, Stephen A. Della Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2):263–311. [find](https://scholar.google.com/scholar?q=Peter+F%2E+Brown%2C+Vincent+J%2E+Della+Pietra%2C+Stephen+A%2E+Della+Pietra%2C+and+Robert+L%2E+Mercer%2E+1993%2E+The+mathematics+of+statistical+machine+translation%3A+Parameter+estimation%2E+Computational+Linguistics%2C+19%282%29)
7. Rajen Chatterjee, Matteo Negri, Marco Turchi, Marcello Federico, Lucia Specia, and Frédéric Blain. 2017. Guiding neural machine translation decoding with external knowledge. In Proceedings of the Second Conference on Machine Translation, pages 157–168. [find](https://scholar.google.com/scholar?q=Rajen+Chatterjee%2C+Matteo+Negri%2C+Marco+Turchi%2C+Marcello+Federico%2C+Lucia+Specia%2C+and+Fre%CC%81de%CC%81ric+Blain%2E+2017%2E+Guiding+neural+machine+translation+decoding+with+external+knowledge%2E+In+Proceedings+of+the+Se)
8. Wenhu Chen, Evgeny Matusov, Shahram Khadivi, and Jan-Thorsten Peter. 2016. Guided alignment training for topic-aware neural machine translation. In Association for Machine Translation in the Americas, pages 121–134, Austin, TX, USA. [find](https://scholar.google.com/scholar?q=Wenhu+Chen%2C+Evgeny+Matusov%2C+Shahram+Khadivi%2C+and+Jan%2DThorsten+Peter%2E+2016%2E+Guided+alignment+training+for+topic%2Daware+neural+machine+translation%2E+In+Association+for+Machine+Translation+in+the+Americas%2C)
9. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Conference on Empirical Methods in Natural Language Processing, pages 489–500, Brussels, Belgium. [find](https://scholar.google.com/scholar?q=Sergey+Edunov%2C+Myle+Ott%2C+Michael+Auli%2C+and+David+Grangier%2E+2018%2E+Understanding+back%2Dtranslation+at+scale%2E+In+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+489%E2%80%93500%2C+Brussels%2C+Be)
10. Qin Gao and Stephan Vogel. 2008. Parallel implementations of word alignment tool. In Software Engineering, Testing, and Quality Assurance for Natural Language Processing, pages 49–57, Columbus, OH, USA. [find](https://scholar.google.com/scholar?q=Qin+Gao+and+Stephan+Vogel%2E+2008%2E+Parallel+implementations+of+word+alignment+tool%2E+In+Software+Engineering%2C+Testing%2C+and+Quality+Assurance+for+Natural+Language+Processing%2C+pages+49%E2%80%9357%2C+Columbus%2C+OH%2C+US) [our card](https://md.research.elamur.ai/free/en/papers/gao-2008-parallel-implementations-word-alignment-tool/)
11. Hamidreza Ghader and Christof Monz. 2017. What does attention in neural machine translation pay attention to? In International Joint Conference on Natural Language Processing, pages 30–39, Taipei, Taiwan. [find](https://scholar.google.com/scholar?q=Hamidreza+Ghader+and+Christof+Monz%2E+2017%2E+What+does+attention+in+neural+machine+translation+pay+attention+to%3F+In+International+Joint+Conference+on+Natural+Language+Processing%2C+pages+30%E2%80%9339%2C+Taipei%2C+Tai)
12. Philipp Koehn, Amittai Axelrod, Ra Birch Mayne, Chris Callison-burch, Miles Osborne, and David Talbot. 2005. Edinburgh system description for the 2005 iwslt speech translation evaluation. In International Workshop on Spoken Language Translation, pages 68–75, Pittsburgh, PA, USA. [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2C+Amittai+Axelrod%2C+Ra+Birch+Mayne%2C+Chris+Callison%2Dburch%2C+Miles+Osborne%2C+and+David+Talbot%2E+2005%2E+Edinburgh+system+description+for+the+2005+iwslt+speech+translation+evaluation%2E+In+Internati) [our card](https://md.research.elamur.ai/free/en/papers/koehn-2005-edinburgh-system-description-2005-iwslt/)
13. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Annu… [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2C+Hieu+Hoang%2C+Alexandra+Birch%2C+Chris+Callison%2DBurch%2C+Marcello+Federico%2C+Nicola+Bertoldi%2C+Brooke+Cowan%2C+Wade+Shen%2C+Christine+Moran%2C+Richard+Zens%2C+Chris+Dyer%2C+Ondrej+Bojar%2C+Alexandra+Consta)
14. Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28–39. [find](https://scholar.google.com/scholar?q=Philipp+Koehn+and+Rebecca+Knowles%2E+2017%2E+Six+challenges+for+neural+machine+translation%2E+In+Proceedings+of+the+First+Workshop+on+Neural+Machine+Translation%2C+pages+28%E2%80%9339%2E)
15. Xintong Li, Lemao Liu, Zhaopeng Tu, Shuming Shi, and Max Meng. 2018. Target foresight based attention for neural machine translation. In Conference of the North American Chapter of the Association for Computational Linguistics, pages 1380–1390, New Orleans, LA, USA. [find](https://scholar.google.com/scholar?q=Xintong+Li%2C+Lemao+Liu%2C+Zhaopeng+Tu%2C+Shuming+Shi%2C+and+Max+Meng%2E+2018%2E+Target+foresight+based+attention+for+neural+machine+translation%2E+In+Conference+of+the+North+American+Chapter+of+the+Association+for)
16. Lemao Liu, Masao Utiyama, Andrew Finch, and Eiichiro Sumita. 2016. Neural machine translation with supervised attention. In International Conference on Computational Linguistics, pages 3093– 3102, Osaka, Japan. [find](https://scholar.google.com/scholar?q=Lemao+Liu%2C+Masao+Utiyama%2C+Andrew+Finch%2C+and+Eiichiro+Sumita%2E+2016%2E+Neural+machine+translation+with+supervised+attention%2E+In+International+Conference+on+Computational+Linguistics%2C+pages+3093%E2%80%93+3102%2C+Osa)
17. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations, New Orleans, LA, USA. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2019%2E+Decoupled+weight+decay+regularization%2E+In+International+Conference+on+Learning+Representations%2C+New+Orleans%2C+LA%2C+USA%2E)
18. Haitao Mi, Zhiguo Wang, and Abe Ittycheriah. 2016. Supervised attentions for neural machine translation. In Conference on Empirical Methods in Natural Language Processing, pages 2283–2288, Austin, TX, USA. [find](https://scholar.google.com/scholar?q=Haitao+Mi%2C+Zhiguo+Wang%2C+and+Abe+Ittycheriah%2E+2016%2E+Supervised+attentions+for+neural+machine+translation%2E+In+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+2283%E2%80%932288%2C+Austin%2C+TX%2C)
19. Rada Mihalcea and Ted Pedersen. 2003. An evaluation exercise for word alignment. In Proceedings of the HLT-NAACL 2003 Workshop on Building and Using Parallel Texts: Data Driven Machine Translation and Beyond - Volume 3, pages 1–10, Edmonton, Canada. [find](https://scholar.google.com/scholar?q=Rada+Mihalcea+and+Ted+Pedersen%2E+2003%2E+An+evaluation+exercise+for+word+alignment%2E+In+Proceedings+of+the+HLT%2DNAACL+2003+Workshop+on+Building+and+Using+Parallel+Texts%3A+Data+Driven+Machine+Translation+and) [our card](https://md.research.elamur.ai/free/en/papers/mihalcea-2003-evaluation-exercise-word-alignment/)
20. Franz Josef Och and Hermann Ney. 2000. Improved statistical alignment models. In Annual Meeting of the Association for Computational Linguistics, pages 440–447, Hong Kong. [find](https://scholar.google.com/scholar?q=Franz+Josef+Och+and+Hermann+Ney%2E+2000%2E+Improved+statistical+alignment+models%2E+In+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+440%E2%80%93447%2C+Hong+Kong%2E) [our card](https://md.research.elamur.ai/free/en/papers/och-2000-improved-statistical-alignment-models/)
21. Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51. [find](https://scholar.google.com/scholar?q=Franz+Josef+Och+and+Hermann+Ney%2E+2003%2E+A+systematic+comparison+of+various+statistical+alignment+models%2E+Computational+Linguistics%2C+29%281%29%3A19%E2%80%9351%2E)
22. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, PA, USA. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+BLEU%3A+A+method+for+automatic+evaluation+of+machine+translation%2E+In+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+p) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
23. Jan-Thorsten Peter, Arne Nix Nix, and Hermann Ney. 2017. Generating alignments using target foresight in attention-based neural machine translation. In Conference of the European Association for Machine Translation, pages 27–36, Prague, Czech Republic. in Natural Language Processing, pages 5027–5038… [find](https://scholar.google.com/scholar?q=Jan%2DThorsten+Peter%2C+Arne+Nix+Nix%2C+and+Hermann+Ney%2E+2017%2E+Generating+alignments+using+target+foresight+in+attention%2Dbased+neural+machine+translation%2E+In+Conference+of+the+European+Association+for+Machi)
24. Gongbo Tang, Rico Sennrich, and Joakim Nivre. 2018. An analysis of attention mechanisms: The case of word sense disambiguation in neural machine translation. In Conference on Statistical Machine Translation, pages 26–35, Brussels, Belgium. [find](https://scholar.google.com/scholar?q=Gongbo+Tang%2C+Rico+Sennrich%2C+and+Joakim+Nivre%2E+2018%2E+An+analysis+of+attention+mechanisms%3A+The+case+of+word+sense+disambiguation+in+neural+machine+translation%2E+In+Conference+on+Statistical+Machine+Trans)
25. Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Modeling coverage for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 76–85. [find](https://scholar.google.com/scholar?q=Zhaopeng+Tu%2C+Zhengdong+Lu%2C+Yang+Liu%2C+Xiaohua+Liu%2C+and+Hang+Li%2E+2016%2E+Modeling+coverage+for+neural+machine+translation%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computational+Li) [our card](https://md.research.elamur.ai/free/en/papers/tu-2016-modeling-coverage-neural-machine-translation/)
26. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 1–11, Long Beach, CA, USA. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N%2E+Gomez%2C+Lukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+Advances+in+Neural+Information+Process)
27. David Vilar, Maja Popović, and Hermann Ney. 2006. AER: Do we need to “improve” our alignments? In International Workshop on Spoken Language Translation, pages 205–212, Kyoto, Japan. [find](https://scholar.google.com/scholar?q=David+Vilar%2C+Maja+Popovic%CC%81%2C+and+Hermann+Ney%2E+2006%2E+AER%3A+Do+we+need+to+%E2%80%9Cimprove%E2%80%9D+our+alignments%3F+In+International+Workshop+on+Spoken+Language+Translation%2C+pages+205%E2%80%93212%2C+Kyoto%2C+Japan%2E)
28. Stephan Vogel, Hermann Ney, and Christoph Tillmann. 1996. HMM-based word alignment in statistical translation. In International Conference on Computational Linguistics, pages 836–841, Copenhagen, Denmark. [find](https://scholar.google.com/scholar?q=Stephan+Vogel%2C+Hermann+Ney%2C+and+Christoph+Tillmann%2E+1996%2E+HMM%2Dbased+word+alignment+in+statistical+translation%2E+In+International+Conference+on+Computational+Linguistics%2C+pages+836%E2%80%93841%2C+Copenhagen%2C+Denm)
29. Thomas Zenkel, Joern Wuebker, and John DeNero. 2019. Adding Interpretable Attention to Neural Translation Models Improves Word Alignment. arXiv e-prints. [find](https://scholar.google.com/scholar?q=Thomas+Zenkel%2C+Joern+Wuebker%2C+and+John+DeNero%2E+2019%2E+Adding+Interpretable+Attention+to+Neural+Translation+Models+Improves+Word+Alignment%2E+arXiv+e%2Dprints%2E)
30. Matt Post. 2018. A call for clarity in reporting bleu scores. In Conference on Statistical Machine Translation, pages 186–191, Belgium, Brussels. [find](https://scholar.google.com/scholar?q=Matt+Post%2E+2018%2E+A+call+for+clarity+in+reporting+bleu+scores%2E+In+Conference+on+Statistical+Machine+Translation%2C+pages+186%E2%80%93191%2C+Belgium%2C+Brussels%2E) [our card](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/)
31. Ofir Press and Lior Wolf. 2017. Using the output embedding to improve language models. In Conference of the European Chapter of the Association for Computational Linguistics, pages 157–163, Valencia, Spain. [find](https://scholar.google.com/scholar?q=Ofir+Press+and+Lior+Wolf%2E+2017%2E+Using+the+output+embedding+to+improve+language+models%2E+In+Conference+of+the+European+Chapter+of+the+Association+for+Computational+Linguistics%2C+pages+157%E2%80%93163%2C+Valencia%2C+) [our card](https://md.research.elamur.ai/free/en/papers/press-2017-output-embedding-improve-language-models/)
32. Alessandro Raganato and Jörg Tiedemann. 2018. An analysis of encoder representations in transformer-based machine translation. In BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 287–297, Brussels, Belgium. [find](https://scholar.google.com/scholar?q=Alessandro+Raganato+and+Jo%CC%88rg+Tiedemann%2E+2018%2E+An+analysis+of+encoder+representations+in+transformer%2Dbased+machine+translation%2E+In+BlackboxNLP%3A+Analyzing+and+Interpreting+Neural+Networks+for+NLP%2C+page) [our card](https://md.research.elamur.ai/free/en/papers/raganato-2018-analysis-encoder-representations-transformer-based/)
33. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Annual Meeting of the Association for Computational Linguistics, pages 1715– 1725, Berlin, Germany. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016%2E+Neural+machine+translation+of+rare+words+with+subword+units%2E+In+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+1715%E2%80%93+172)
34. Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018. Linguistically-informed self-attention for semantic role labeling. In Conference on Empirical Methods [find](https://scholar.google.com/scholar?q=Emma+Strubell%2C+Patrick+Verga%2C+Daniel+Andor%2C+David+Weiss%2C+and+Andrew+McCallum%2E+2018%2E+Linguistically%2Dinformed+self%2Dattention+for+semantic+role+labeling%2E+In+Conference+on+Empirical+Methods) [our card](https://md.research.elamur.ai/free/en/papers/strubell-2018-linguistically-informed-self-attention-semantic/)