--- title: "Tagged Back-Translation — Caswell 2019" description: "Back-translation (BT) creates synthetic parallel data by translating target-language monolingual text into the source language." authors: ["Isaac Caswell", "Ciprian Chelba", "David Grangier"] year: 2019 venue: "Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers)" source: "https://aclanthology.org/W19-5206/" doi: "10.18653/v1/w19-5206" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/caswell-2019-tagged-back-translation/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/caswell-2019-tagged-back-translation/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / Tagged Back-Translation # Tagged Back-Translation **Authors:** [Isaac Caswell](https://md.research.elamur.ai/free/en/authors/isaac-caswell/), [Ciprian Chelba](https://md.research.elamur.ai/free/en/authors/ciprian-chelba/), [David Grangier](https://md.research.elamur.ai/free/en/authors/david-grangier/) · **Year:** 2019 · **Venue:** Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/W19-5206/) · [PDF](https://aclanthology.org/W19-5206.pdf) · [DOI 10.18653/v1/w19-5206](https://doi.org/10.18653/v1/w19-5206) · [OpenAlex](https://openalex.org/W2950169745) **Citations:** 179 citations (openalex-snapshot · 2026-10-14) · [144](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Back-translation (BT) creates synthetic parallel data by translating target-language monolingual text into the source language. Edunov et al. (2018) found that adding noise to beam-search BT output (NoisedBT) improves NMT, and attributed this to greater source-side diversity. Caswell, Chelba and Grangier challenge that explanation. They argue that noise mainly lets the model recognize which training sources are synthetic. This allows it to learn from the useful target-side signal while discounting the artifacts of machine translation. To test this, they replace noise with a single reserved `` token placed before each back-translated source sentence (TaggedBT). They train Transformer models in lingvo on WMT EnDe and WMT16 EnRo bitext plus large Newscrawl back-translations, and evaluate with SacreBLEU. TaggedBT equals or slightly beats NoisedBT on EnDe. On EnRo it beats both standard and noised BT, and it makes iterative BT work where untagged BT degraded. Attention and tagged-decoding analyses support the domain-marker interpretation. The method is simpler and more robust than noising and gives a new state of the art on EnRo. ## Key points - **Data:** about 5.0M filtered EnDe bitext pairs with 216.5M back-translated sentences, and about 0.6M EnRo pairs. EnFr was also explored. Models used transformer-base/big, 32k subwords and EMA weights. - **EnRo results:** iterative TaggedBT reached 33.4 SacreBLEU on WMT16 EnRo, about +3.5 over the best previously published figure the authors knew of. Plain TaggedBT gained about +0.4 over vanilla BT. - **EnDe results:** TaggedBT was on par with or above the best Edunov et al. (2018) model on five of seven test sets. The largest gain was +1.4 on Newstest2018. - **Evidence against the diversity explanation:** noising bitext sources barely hurt quality. Training only on NoisedBT data scored about 1 BLEU below training on clean BT alone. - **Tagged decoding:** feeding the `` tag at test time raised the source-target copy rate from 8.9% to 10.7%, close to the 11.4% seen in BT data. This suggests the tag acts as a domain marker. Tagging forward-translation brought no benefit. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/) Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) ## Cited by Cited by 144 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 139 not yet. - [On The Evaluation of Machine Translation Systems Trained With Back-Translation](https://md.research.elamur.ai/free/en/papers/edunov-2020-evaluation-machine-translation-systems-trained/) - [IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages](https://md.research.elamur.ai/free/en/papers/gala-2023-indictrans2-towards-high-quality-accessible/) - [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://md.research.elamur.ai/free/en/papers/keskar-2019-ctrl-conditional-transformer-language-model/) - [Self-Alignment with Instruction Backtranslation](https://md.research.elamur.ai/free/en/papers/li-2023-self-alignment-instruction-backtranslation/) - [Prompting Large Language Model for Machine Translation: A Case Study](https://md.research.elamur.ai/free/en/papers/zhang-2023-prompting-large-language-model-machine/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{caswell2019tagged, title = {{Tagged Back-Translation}}, author = {Caswell, Isaac and Chelba, Ciprian and Grangier, David}, booktitle = {Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers)}, year = {2019}, doi = {10.18653/v1/w19-5206}, url = {https://doi.org/10.18653/v1/w19-5206} } ``` - [← previous TabFact: A Large-scale Dataset for Table-based Fact Verification](https://md.research.elamur.ai/free/en/papers/chen-2019-tabfact-large-scale-dataset-table/) - [next → Taking MT Evaluation Metrics to Extremes: Beyond Correlation with Human Judgments](https://md.research.elamur.ai/free/en/papers/fomicheva-2019-taking-mt-evaluation-metrics-extremes/) - [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) ## References 4 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018. Unsupervised Statistical Machine Translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3632–3642. [find](https://scholar.google.com/scholar?q=Mikel+Artetxe%2C+Gorka+Labaka%2C+and+Eneko+Agirre%2E+2018%2E+Unsupervised+Statistical+Machine+Translation%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+3632) [our card](https://md.research.elamur.ai/free/en/papers/artetxe-2018-unsupervised-statistical-machine-translation/) 2. Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011. Domain adaptation via pseudo in-domain data selection. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 355–362. [find](https://scholar.google.com/scholar?q=Amittai+Axelrod%2C+Xiaodong+He%2C+and+Jianfeng+Gao%2E+2011%2E+Domain+adaptation+via+pseudo+in%2Ddomain+data+selection%2E+In+Proceedings+of+the+2011+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+) [our card](https://md.research.elamur.ai/free/en/papers/axelrod-2011-domain-adaptation-pseudo-domain-data/) 3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In 3rd International Conference on Learning Representations, ICLR 2015. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2015%2E+Neural+Machine+Translation+by+Jointly+Learning+to+Align+and+Translate%2E+In+3rd+International+Conference+on+Learning+Representations%2C+ICLR+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/) 4. Nicola Bertoldi and Marcello Federico. 2009. Domain adaptation for statistical machine translation with monolingual resources. In Proceedings of the fourth workshop on statistical machine translation, pages 182–189. Association for Computational Linguistics. \#\# Future Work [find](https://scholar.google.com/scholar?q=Nicola+Bertoldi+and+Marcello+Federico%2E+2009%2E+Domain+adaptation+for+statistical+machine+translation+with+monolingual+resources%2E+In+Proceedings+of+the+fourth+workshop+on+statistical+machine+translation%2C) [our card](https://md.research.elamur.ai/free/en/papers/bertoldi-2009-domain-adaptation-statistical-machine-translation/)