--- title: "Tensor2Tensor for Neural Machine Translation — Vaswani 2018" description: "This paper presents Tensor2Tensor (T2T), an open-source TensorFlow library from Google." authors: ["Ashish Vaswani", "Samy Bengio", "Eugene Brevdo", "Francois Chollet", "Aidan Gomez", "Stephan Gouws", "Llion Jones", "Łukasz Kaiser", "Nal Kalchbrenner", "Niki Parmar", "Ryan Sepassi", "Noam Shazeer", "Jakob Uszkoreit"] year: 2018 venue: "Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track)" source: "https://aclanthology.org/W18-1819/" arxiv: "1803.07416" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/vaswani-2018-tensor2tensor-neural-machine-translation/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/vaswani-2018-tensor2tensor-neural-machine-translation/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/) / Tensor2Tensor for Neural Machine Translation # Tensor2Tensor for Neural Machine Translation **Authors:** [Ashish Vaswani](https://md.research.elamur.ai/free/en/authors/ashish-vaswani/), [Samy Bengio](https://md.research.elamur.ai/free/en/authors/samy-bengio/), [Eugene Brevdo](https://md.research.elamur.ai/free/en/authors/eugene-brevdo/), Francois Chollet, Aidan Gomez, Stephan Gouws, [Llion Jones](https://md.research.elamur.ai/free/en/authors/llion-jones/), [Łukasz Kaiser](https://md.research.elamur.ai/free/en/authors/lukasz-kaiser/), [Nal Kalchbrenner](https://md.research.elamur.ai/free/en/authors/nal-kalchbrenner/), [Niki Parmar](https://md.research.elamur.ai/free/en/authors/niki-parmar/), [Ryan Sepassi](https://md.research.elamur.ai/free/en/authors/ryan-sepassi/), [Noam Shazeer](https://md.research.elamur.ai/free/en/authors/noam-shazeer/) et al. (13 authors) · **Year:** 2018 · **Venue:** Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/W18-1819/) · [PDF](https://aclanthology.org/W18-1819.pdf) · [arXiv 1803.07416](https://arxiv.org/abs/1803.07416) · [OpenAlex](https://openalex.org/W2793348910) **Citations:** 14 citations (openalex-snapshot · 2026-10-14) · [160](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary This paper presents Tensor2Tensor (T2T), an open-source TensorFlow library from Google. It aims to make deep learning research, especially neural machine translation, faster to run and easier to reproduce. The authors first trace NMT from recurrent sequence-to-sequence models with attention, through convolutional systems such as ByteNet and the Extended Neural GPU, to the self-attention-based Transformer. T2T contains the reference implementation of the Transformer. Each training run is broken into interchangeable parts: dataset definitions, device setup, named hyperparameter sets, models, and TensorFlow's training machinery. As a result, researchers can apply one model to many tasks, or many models to one task, while changing only the part they care about. The same code runs on CPU, GPU or TPU, either locally or in the cloud. Several building blocks first developed in T2T later moved into TensorFlow or supported follow-up work. The paper reports no new benchmark scores. Its value is practical: shared infrastructure that speeds up Transformer-style research and keeps results reproducible as the code changes. ## Key points - A training run in T2T is defined by five components: Problem (dataset), device configuration, hyperparameters, Model, and Estimator/Experiment. - T2T supports CPU, GPU and TPU, single or multi-device, with synchronous or asynchronous data-parallel training. It started with NMT and later added image, video and audio tasks. - Research components first built or tested in T2T include the Image Transformer, `tf.contrib.layers.rev_block` (reversible layers), the memory-saving Adafactor optimizer, and `tf.contrib.data.bucket_by_sequence_length`. - Reproducibility relies on three mechanisms: versioned named hyperparameter sets, regular end-to-end regression tests on key model-problem pairs, and random seeds fixed in Python, numpy and TensorFlow. - The code is developed openly at github.com/tensorflow/tensor2tensor, with contributors from inside and outside Google. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) ## Cited by Cited by 160 papers of our library cite it (of the 140673 whose bibliographies we read); 20 of them have a page here, 140 not yet. - [Towards a Human-like Open-Domain Chatbot](https://md.research.elamur.ai/free/en/papers/adiwardana-2020-towards-human-like-open-domain/) - [Findings of the WMT 2019 Biomedical Translation Shared Task: Evaluation for MEDLINE Abstracts and Biomedical Terminologies](https://md.research.elamur.ai/free/en/papers/bawden-2019-findings-wmt-2019-biomedical-translation/) - [Findings of the 2018 Conference on Machine Translation (WMT18)](https://md.research.elamur.ai/free/en/papers/bojar-2018-findings-2018-conference-machine-translation/) - [Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset](https://md.research.elamur.ai/free/en/papers/byrne-2019-taskmaster-1-realistic-diverse-dialog/) - [Representation Degeneration Problem in Training Natural Language Generation Models](https://md.research.elamur.ai/free/en/papers/gao-2019-representation-degeneration-problem-training-natural/) - [ConveRT: Efficient and Accurate Conversational Representations from Transformers](https://md.research.elamur.ai/free/en/papers/henderson-2020-convert-efficient-accurate-conversational-representations/) - [Trivial Transfer Learning for Low-Resource Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/kocmi-2018-trivial-transfer-learning-low-resource/) - [Similarity of Neural Network Representations Revisited](https://md.research.elamur.ai/free/en/papers/kornblith-2019-similarity-neural-network-representations-revisited/) - [Hurdles to Progress in Long-form Question Answering](https://md.research.elamur.ai/free/en/papers/krishna-2021-hurdles-progress-long-form-question/) - [FlauBERT: Unsupervised Language Model Pre-training for French](https://md.research.elamur.ai/free/en/papers/le-2019-flaubert-unsupervised-language-model-pre/) - [The NiuTrans Machine Translation Systems for WMT19](https://md.research.elamur.ai/free/en/papers/li-2019-niutrans-machine-translation-systems-wmt19/) - [End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classification](https://md.research.elamur.ai/free/en/papers/libovicky-2018-end-end-non-autoregressive-neural/) - [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://md.research.elamur.ai/free/en/papers/ott-2019-fairseq-fast-extensible-toolkit-sequence/) - [Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problem](https://md.research.elamur.ai/free/en/papers/saunders-2020-reducing-gender-bias-neural-machine/) - [On NMT Search Errors and Model Errors: Cat Got Your Tongue?](https://md.research.elamur.ai/free/en/papers/stahlberg-2019-nmt-search-errors-model-errors/) - [Analyzing the Structure of Attention in a Transformer Language Model](https://md.research.elamur.ai/free/en/papers/vig-2019-analyzing-structure-attention-transformer-language/) - [A Multiscale Visualization of Attention in the Transformer Model](https://md.research.elamur.ai/free/en/papers/vig-2019-multiscale-visualization-attention-transformer-model/) - [Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq](https://md.research.elamur.ai/free/en/papers/wang-2020-fairseq-s2t-fast-speech-text/) - [Transformers: State-of-the-Art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/) - [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{vaswani2018tensor2tensor, title = {{Tensor2Tensor for Neural Machine Translation}}, author = {Vaswani, Ashish and Bengio, Samy and Brevdo, Eugene and Chollet, Francois and Gomez, Aidan and Gouws, Stephan and Jones, Llion and Kaiser, Łukasz and Kalchbrenner, Nal and Parmar, Niki and Sepassi, Ryan and Shazeer, Noam and Uszkoreit, Jakob}, booktitle = {Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track)}, year = {2018}, eprint = {1803.07416}, archivePrefix = {arXiv}, url = {https://arxiv.org/abs/1803.07416} } ``` - [← previous SzegedKoref: A Hungarian Coreference Corpus](https://md.research.elamur.ai/free/en/papers/vincze-2018-szegedkoref-hungarian-coreference-corpus/) - [next → TextBugger: Generating Adversarial Text Against Real-world Applications](https://md.research.elamur.ai/free/en/papers/li-2018-textbugger-generating-adversarial-text-against/) - [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/) ## References 14 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., Kudlur, M., Levenberg, J., Monga, R., Moore, S., Murray, D. G., Steiner, B., Tucker, P., Vasudevan, V., Warden, P., Wicke, M., Yu, Y., and Zheng, X. (2016). Tensorflow: A system for large-… [find](https://scholar.google.com/scholar?q=Abadi%2C+M%2E%2C+Barham%2C+P%2E%2C+Chen%2C+J%2E%2C+Chen%2C+Z%2E%2C+Davis%2C+A%2E%2C+Dean%2C+J%2E%2C+Devin%2C+M%2E%2C+Ghemawat%2C+S%2E%2C+Irving%2C+G%2E%2C+Isard%2C+M%2E%2C+Kudlur%2C+M%2E%2C+Levenberg%2C+J%2E%2C+Monga%2C+R%2E%2C+Moore%2C+S%2E%2C+Murray%2C+D%2E+G%2E%2C+Steiner%2C+B%2E%2C+Tucker%2C+P%2E%2C) 2. Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473. [find](https://scholar.google.com/scholar?q=Bahdanau%2C+D%2E%2C+Cho%2C+K%2E%2C+and+Bengio%2C+Y%2E+%282014%29%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+CoRR%2C+abs%2F1409%2E0473%2E) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/) 3. Cho, K., van Merrienboer, B., Gulcehre, C., Bougares, F., Schwenk, H., and Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. CoRR, abs/1406.1078. [find](https://scholar.google.com/scholar?q=Cho%2C+K%2E%2C+van+Merrienboer%2C+B%2E%2C+Gulcehre%2C+C%2E%2C+Bougares%2C+F%2E%2C+Schwenk%2C+H%2E%2C+and+Bengio%2C+Y%2E+%282014%29%2E+Learning+phrase+representations+using+RNN+encoder%2Ddecoder+for+statistical+machine+translation%2E+CoRR%2C+abs%2F1) 4. Gehring, J., Auli, M., Grangier, D., Yarats, D., and Dauphin, Y. N. (2017). Convolutional sequence to sequence learning. CoRR, abs/1705.03122. [find](https://scholar.google.com/scholar?q=Gehring%2C+J%2E%2C+Auli%2C+M%2E%2C+Grangier%2C+D%2E%2C+Yarats%2C+D%2E%2C+and+Dauphin%2C+Y%2E+N%2E+%282017%29%2E+Convolutional+sequence+to+sequence+learning%2E+CoRR%2C+abs%2F1705%2E03122%2E) [our card](https://md.research.elamur.ai/free/en/papers/gehring-2017-convolutional-sequence-sequence-learning/) 5. Gomez, A. N., Ren, M., Urtasun, R., and Grosse, R. B. (2017). The reversible residual network: Backpropagation without storing activations. CoRR, abs/1707.04585. — [arXiv:1707.04585](https://arxiv.org/abs/1707.04585) [find](https://scholar.google.com/scholar?q=Gomez%2C+A%2E+N%2E%2C+Ren%2C+M%2E%2C+Urtasun%2C+R%2E%2C+and+Grosse%2C+R%2E+B%2E+%282017%29%2E+The+reversible+residual+network%3A+Backpropagation+without+storing+activations%2E+CoRR%2C+abs%2F1707%2E04585%2E+%E2%80%94+%5BarXiv%3A1707%2E04585%5D%28https%3A%2F%2Farxiv%2Eorg) 6. Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8):1735– 1780. [find](https://scholar.google.com/scholar?q=Hochreiter%2C+S%2E+and+Schmidhuber%2C+J%2E+%281997%29%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A1735%E2%80%93+1780%2E) 7. Kaiser, Ł. and Bengio, S. (2016). Can active memory replace attention? In Advances in Neural Information Processing Systems, pages 3781–3789. [find](https://scholar.google.com/scholar?q=Kaiser%2C+%C5%81%2E+and+Bengio%2C+S%2E+%282016%29%2E+Can+active+memory+replace+attention%3F+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+3781%E2%80%933789%2E) 8. Kaiser, L., Gomez, A. N., and Chollet, F. (2017). Depthwise separable convolutions for neural machine translation. CoRR, abs/1706.03059. — [arXiv:1706.03059](https://arxiv.org/abs/1706.03059) [find](https://scholar.google.com/scholar?q=Kaiser%2C+L%2E%2C+Gomez%2C+A%2E+N%2E%2C+and+Chollet%2C+F%2E+%282017%29%2E+Depthwise+separable+convolutions+for+neural+machine+translation%2E+CoRR%2C+abs%2F1706%2E03059%2E+%E2%80%94+%5BarXiv%3A1706%2E03059%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1706%2E03059%29) 9. Kalchbrenner, N. and Blunsom, P. (2013). Recurrent continuous translation models. In Proceedings EMNLP 2013, pages 1700–1709. [find](https://scholar.google.com/scholar?q=Kalchbrenner%2C+N%2E+and+Blunsom%2C+P%2E+%282013%29%2E+Recurrent+continuous+translation+models%2E+In+Proceedings+EMNLP+2013%2C+pages+1700%E2%80%931709%2E) [our card](https://md.research.elamur.ai/free/en/papers/kalchbrenner-2013-recurrent-continuous-translation-models/) 10. Kalchbrenner, N., Espeholt, L., Simonyan, K., van den Oord, A., Graves, A., and Kavukcuoglu, K. (2016). Neural machine translation in linear time. CoRR, abs/1610.10099. — [arXiv:1610.10099](https://arxiv.org/abs/1610.10099) [find](https://scholar.google.com/scholar?q=Kalchbrenner%2C+N%2E%2C+Espeholt%2C+L%2E%2C+Simonyan%2C+K%2E%2C+van+den+Oord%2C+A%2E%2C+Graves%2C+A%2E%2C+and+Kavukcuoglu%2C+K%2E+%282016%29%2E+Neural+machine+translation+in+linear+time%2E+CoRR%2C+abs%2F1610%2E10099%2E+%E2%80%94+%5BarXiv%3A1610%2E10099%5D%28https%3A%2F%2Far) 11. Meng, F., Lu, Z., Wang, M., Li, H., Jiang, W., and Liu, Q. (2015). Encoding source language with convolutional neural network for machine translation. In ACL, pages 20–30. [find](https://scholar.google.com/scholar?q=Meng%2C+F%2E%2C+Lu%2C+Z%2E%2C+Wang%2C+M%2E%2C+Li%2C+H%2E%2C+Jiang%2C+W%2E%2C+and+Liu%2C+Q%2E+%282015%29%2E+Encoding+source+language+with+convolutional+neural+network+for+machine+translation%2E+In+ACL%2C+pages+20%E2%80%9330%2E) 12. Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, Ł., Shazeer, N., and Ku, A. (2018). Image Transformer. ArXiv e-prints. [find](https://scholar.google.com/scholar?q=Parmar%2C+N%2E%2C+Vaswani%2C+A%2E%2C+Uszkoreit%2C+J%2E%2C+Kaiser%2C+%C5%81%2E%2C+Shazeer%2C+N%2E%2C+and+Ku%2C+A%2E+%282018%29%2E+Image+Transformer%2E+ArXiv+e%2Dprints%2E) 13. Sutskever, I., Vinyals, O., and Le, Q. V. (2014). Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pages 3104–3112. van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu,… [find](https://scholar.google.com/scholar?q=Sutskever%2C+I%2E%2C+Vinyals%2C+O%2E%2C+and+Le%2C+Q%2E+V%2E+%282014%29%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+3104%E2%80%933112%2E+van+den+Oord%2C+A%2E%2C+Dieleman%2C) [our card](https://md.research.elamur.ai/free/en/papers/oord-2016-wavenet-generative-model-raw-audio/) 14. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. CoRR. [find](https://scholar.google.com/scholar?q=Vaswani%2C+A%2E%2C+Shazeer%2C+N%2E%2C+Parmar%2C+N%2E%2C+Uszkoreit%2C+J%2E%2C+Jones%2C+L%2E%2C+Gomez%2C+A%2E+N%2E%2C+Kaiser%2C+L%2E%2C+and+Polosukhin%2C+I%2E+%282017%29%2E+Attention+is+all+you+need%2E+CoRR%2E)