---
title: "Improving Speech Translation by Understanding and Learning"
description: "Building on this, they propose three changes: share the whole decoder and upper encoder layers… 27 references, cited by 47 papers here."
authors: ["Yun Tang", "Juan Pino", "Xian Li", "Changhan Wang", "Dmitriy Genzel"]
year: 2021
venue: "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)"
source: "https://aclanthology.org/2021.acl-long.328/"
doi: "10.18653/v1/2021.acl-long.328"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/tang-2021-improving-speech-translation-understanding-learning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tang-2021-improving-speech-translation-understanding-learning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task

# Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task

**Authors:** [Yun Tang](https://md.research.elamur.ai/free/en/authors/yun-tang/), [Juan Pino](https://md.research.elamur.ai/free/en/authors/juan-pino/), [Xian Li](https://md.research.elamur.ai/free/en/authors/xian-li/), [Changhan Wang](https://md.research.elamur.ai/free/en/authors/changhan-wang/) and [Dmitriy Genzel](https://md.research.elamur.ai/free/en/authors/dmitriy-genzel/) · **Year:** 2021 · **Venue:** [Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2021](https://md.research.elamur.ai/free/en/venues/acl/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.acl-long.328/) · [PDF](https://aclanthology.org/2021.acl-long.328.pdf) · [DOI 10.18653/v1/2021.acl-long.328](https://doi.org/10.18653/v1/2021.acl-long.328) · [OpenAlex](https://openalex.org/W3176711365)

**Citations:** 70 citations (OpenAlex, 2026-10-14) · [47](https://md.research.elamur.ai/free/en/papers/tang-2021-improving-speech-translation-understanding-learning/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
End-to-end speech translation (ST) depends on paired speech–translation data, which is expensive and scarce. Systems are therefore often built from pretrained ASR and MT modules and trained jointly with related tasks. This Facebook AI paper examines how an auxiliary text-to-text MT task actually affects the primary ST task in such a multitask setup. Starting from models initialized with pretrained ASR and MT components, the authors run two analyses. One measures how far the fine-tuned modules move away from their pretrained versions. The other compares internal representations produced from speech versus text input. They find little negative transfer and clear benefits from sharing more parameters. A noticeable modality gap remains in the top decoder layers, which are also the most important for quality. Building on this, they propose three changes: share the whole decoder and upper encoder layers, initialized from MT; add a cross-attentive regularization (CAR) that aligns speech and text encoder outputs; and distill knowledge online from the co-trained MT branch. On MuST-C EN-DE, EN-FR and EN-ES, ST quality improves by over 2 BLEU against a strong baseline and reaches state-of-the-art results.

## Key points
- **Setup:** A speech encoder and a text encoder feed one shared decoder, and the uppermost encoder layers are likewise common to both paths. The shared parts are initialized from an MT model pretrained on WMT parallel data. Only the ST path is used at inference.
- **CAR:** Speech and text encoder outputs have different lengths, so both are reconstructed to text length. The speech outputs are rebuilt through a cosine-similarity attention over the text outputs, and the text outputs through self-attention. The L2 gap between the two is then penalized, with a stop-gradient applied on the text side.
- **Training objective:** The loss combines ST NLL, online KD toward the MT branch's output distribution (weighted by α), CAR (weighted by λ), and MT NLL.
- **Final ST BLEU on MuST-C tst-COMMON:** 26.8 for EN-DE, 31.0 for EN-ES and 37.4 for EN-FR.
- **Task-specific components and the MT task:** Adding an adapter left BLEU unchanged at 24.7, and dedicated decoder attention lowered it to 24.2, so extra task-specific parts seem unnecessary. The co-trained MT models, which take phoneme input, also gained 0.7–1.6 BLEU from the proposed techniques.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🎙️ Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) · 233 works

## Cite

Cite as: Yun Tang, Juan Pino, Xian Li, Changhan Wang and Dmitriy Genzel. 2021. Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{tang2021improving,
  title = {{Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task}},
  author = {Yun Tang, Juan Pino, Xian Li, Changhan Wang and Dmitriy Genzel},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.328},
  url = {https://doi.org/10.18653/v1/2021.acl-long.328}
}
```

- [← previous The Impact of Multiple Parallel Phrase Suggestions on Email Input and Composition Behaviour of Native and Non-Native English Writers](https://md.research.elamur.ai/free/en/papers/buschek-2021-impact-multiple-parallel-phrase-suggestions/)
- [next → Relevance-guided Supervision for OpenQA with ColBERT](https://md.research.elamur.ai/free/en/papers/khattab-2021-relevance-guided-supervision-openqa-colbert/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/)

## References

27 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Antonios Anastasopoulos and David Chiang. 2018. [Tied multitask learning for neural speech translation](https://md.research.elamur.ai/free/en/papers/anastasopoulos-2018-tied-multitask-learning-neural-speech/). In NAACL-HLT.
2. Parnia Bahar, Tobias Bieschke, and Hermann Ney. 2019. A comparative study on end-to-end speech to text translation. In ASRU. [find](https://scholar.google.com/scholar?q=Parnia+Bahar%2C+Tobias+Bieschke%2C+and+Hermann+Ney%2E+2019%2E+A+comparative+study+on+end%2Dto%2Dend+speech+to+text+translation%2E+In+ASRU%2E)
3. Ankur Bapna, N. Arivazhagan, and Orhan Firat. 2019. [Simple, scalable adaptation for neural machine translation](https://md.research.elamur.ai/free/en/papers/bapna-2019-simple-scalable-adaptation-neural-machine/). In EMNLP/IJCNLP.
4. G. Blackwood, Miguel Ballesteros, and T. Ward. 2018. Multilingual neural machine translation with task-specific attention. In COLING. [find](https://scholar.google.com/scholar?q=G%2E+Blackwood%2C+Miguel+Ballesteros%2C+and+T%2E+Ward%2E+2018%2E+Multilingual+neural+machine+translation+with+task%2Dspecific+attention%2E+In+COLING%2E)
5. Niladri S. Chatterji, Behnam Neyshabur, and H. Sedghi. 2020. The intriguing role of module criticality in the generalization of deep networks. In ICLR. [find](https://scholar.google.com/scholar?q=Niladri+S%2E+Chatterji%2C+Behnam+Neyshabur%2C+and+H%2E+Sedghi%2E+2020%2E+The+intriguing+role+of+module+criticality+in+the+generalization+of+deep+networks%2E+In+ICLR%2E)
6. Z. Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. 2018. Gradnorm: Gradient normalization for adaptive loss balancing in deep multi-task networks. In ICML. [find](https://scholar.google.com/scholar?q=Z%2E+Chen%2C+Vijay+Badrinarayanan%2C+Chen%2DYu+Lee%2C+and+Andrew+Rabinovich%2E+2018%2E+Gradnorm%3A+Gradient+normalization+for+adaptive+loss+balancing+in+deep+multi%2Dtask+networks%2E+In+ICML%2E)
7. Marco Gaido, Mattia Antonino Di Gangi, Matteo Negri, and Marco Turchi. 2020. End-to-end speech-translation with knowledge distillation: Fbk@iwslt2020. [find](https://scholar.google.com/scholar?q=Marco+Gaido%2C+Mattia+Antonino+Di+Gangi%2C+Matteo+Negri%2C+and+Marco+Turchi%2E+2020%2E+End%2Dto%2Dend+speech%2Dtranslation+with+knowledge+distillation%3A+Fbk%40iwslt2020%2E)
8. Mattia Antonino Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019a. [MuST-C: a multilingual speech translation corpus](https://md.research.elamur.ai/free/en/papers/gangi-2019-must-c-multilingual-speech-translation/). In NAACL-HLT.
9. Mattia Antonino Di Gangi, Matteo Negri, and Marco Turchi. 2019b. One-to-many multilingual end-to-end speech translation. In ASRU. [find](https://scholar.google.com/scholar?q=Mattia+Antonino+Di+Gangi%2C+Matteo+Negri%2C+and+Marco+Turchi%2E+2019b%2E+One%2Dto%2Dmany+multilingual+end%2Dto%2Dend+speech+translation%2E+In+ASRU%2E)
10. Geoffrey E. Hinton, Oriol Vinyals, and J. Dean. 2015. [Distilling the knowledge in a neural network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/). ArXiv, abs/1503.02531.
11. Xian Li, Changhan Wang, Yun Tang, Chau Tran, Yuqing Tang, Juan Pino, Alexei Baevski, Alexis Conneau, and Michael Auli. 2020. Multilingual speech translation with efficient finetuning of pre-trained models. arXiv: Computation and Language. [find](https://scholar.google.com/scholar?q=Xian+Li%2C+Changhan+Wang%2C+Yun+Tang%2C+Chau+Tran%2C+Yuqing+Tang%2C+Juan+Pino%2C+Alexei+Baevski%2C+Alexis+Conneau%2C+and+Michael+Auli%2E+2020%2E+Multilingual+speech+translation+with+efficient+finetuning+of+pre%2Dtrained+mo)
12. Shikun Liu, Edward Johns, and A. Davison. 2019a. End-to-end multi-task learning with attention. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1871–1880. [find](https://scholar.google.com/scholar?q=Shikun+Liu%2C+Edward+Johns%2C+and+A%2E+Davison%2E+2019a%2E+End%2Dto%2Dend+multi%2Dtask+learning+with+attention%2E+2019+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition+%28CVPR%29%2C+pages+1871%E2%80%931880%2E)
13. Yuchen Liu, Hao Xiong, Zhongjun He, Jiajun Zhang, Hua Wu, Haifeng Wang, and Chengqing Zong. 2019b. End-to-end speech translation with knowledge distillation. In Interspeech. [find](https://scholar.google.com/scholar?q=Yuchen+Liu%2C+Hao+Xiong%2C+Zhongjun+He%2C+Jiajun+Zhang%2C+Hua+Wu%2C+Haifeng+Wang%2C+and+Chengqing+Zong%2E+2019b%2E+End%2Dto%2Dend+speech+translation+with+knowledge+distillation%2E+In+Interspeech%2E)
14. K. Maninis, Ilija Radosavovic, and I. Kokkinos. 2019. Attentive single-tasking of multiple tasks. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1851–1860. [find](https://scholar.google.com/scholar?q=K%2E+Maninis%2C+Ilija+Radosavovic%2C+and+I%2E+Kokkinos%2E+2019%2E+Attentive+single%2Dtasking+of+multiple+tasks%2E+2019+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition+%28CVPR%29%2C+pages+1851%E2%80%931860%2E)
15. M. McCloskey and N. J. Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation, 24:109–165. [find](https://scholar.google.com/scholar?q=M%2E+McCloskey+and+N%2E+J%2E+Cohen%2E+1989%2E+Catastrophic+interference+in+connectionist+networks%3A+The+sequential+learning+problem%2E+Psychology+of+Learning+and+Motivation%2C+24%3A109%E2%80%93165%2E)
16. Ari S. Morcos, M. Raghu, and S. Bengio. 2018. Insights on representational similarity in neural networks with canonical correlation. In NeurIPS. [find](https://scholar.google.com/scholar?q=Ari+S%2E+Morcos%2C+M%2E+Raghu%2C+and+S%2E+Bengio%2E+2018%2E+Insights+on+representational+similarity+in+neural+networks+with+canonical+correlation%2E+In+NeurIPS%2E)
17. Jan Niehues, R. Cattoni, Sebastian Stüker, Matteo Negri, Marco Turchi, Elizabeth Salesky, Ramon Sanabria, Loı̈c Barrault, Lucia Specia, and Marcello Federico. 2019. [find](https://scholar.google.com/scholar?q=Jan+Niehues%2C+R%2E+Cattoni%2C+Sebastian+Stu%CC%88ker%2C+Matteo+Negri%2C+Marco+Turchi%2C+Elizabeth+Salesky%2C+Ramon+Sanabria%2C+Lo%C4%B1%CC%88c+Barrault%2C+Lucia+Specia%2C+and+Marcello+Federico%2E+2019%2E)
18. The IWSLT 2019 evaluation campaign. [find](https://scholar.google.com/scholar?q=The+IWSLT+2019+evaluation+campaign%2E)
19. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, S. Gross, Nathan Ng, David Grangier, and M. Auli. 2019. [fairseq: A fast, extensible toolkit for sequence modeling](https://md.research.elamur.ai/free/en/papers/ott-2019-fairseq-fast-extensible-toolkit-sequence/). In NAACL.
20. D. Park, W. Chan, Y. Zhang, C. Chiu, B. Zoph, E. Cubuk, and Q. Le. 2019. [Specaugment: A simple data augmentation method for automatic speech recognition](https://md.research.elamur.ai/free/en/papers/park-2019-specaugment-simple-data-augmentation-method/). Interspeech.
21. Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, and Sebastian Ruder. 2020. [MAD-X: An adapter-based framework for multi-task cross-lingual transfer](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-mad-x-adapter-based-framework/). In EMNLP.
22. J. Pino, Q. Xu, X. Ma, M. Dousti, and Y. Tang. 2020. Self-training for end-to-end speech translation. In Interspeech. [find](https://scholar.google.com/scholar?q=J%2E+Pino%2C+Q%2E+Xu%2C+X%2E+Ma%2C+M%2E+Dousti%2C+and+Y%2E+Tang%2E+2020%2E+Self%2Dtraining+for+end%2Dto%2Dend+speech+translation%2E+In+Interspeech%2E)
23. Matt Post. 2018. [A call for clarity in reporting BLEU scores](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/). In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Brussels, Belgium. Association for Computational Linguistics.
24. M. Raghu, J. Gilmer, J. Yosinski, and Jascha Sohl-Dickstein. 2017. Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In NIPS. [find](https://scholar.google.com/scholar?q=M%2E+Raghu%2C+J%2E+Gilmer%2C+J%2E+Yosinski%2C+and+Jascha+Sohl%2DDickstein%2E+2017%2E+Svcca%3A+Singular+vector+canonical+correlation+analysis+for+deep+learning+dynamics+and+interpretability%2E+In+NIPS%2E)
25. A. Renduchintala, S. Ding, M. Wiesner, and S. Watanabe. 2018. Multi-modal data augmentation for end-to-end asr. In INTERSPEECH. [find](https://scholar.google.com/scholar?q=A%2E+Renduchintala%2C+S%2E+Ding%2C+M%2E+Wiesner%2C+and+S%2E+Watanabe%2E+2018%2E+Multi%2Dmodal+data+augmentation+for+end%2Dto%2Dend+asr%2E+In+INTERSPEECH%2E)
26. Elizabeth Salesky and Alan W Black. 2020. Phone features improve speech translation. In ACL. [find](https://scholar.google.com/scholar?q=Elizabeth+Salesky+and+Alan+W+Black%2E+2020%2E+Phone+features+improve+speech+translation%2E+In+ACL%2E)
27. T. Standley, A. Zamir, Dawn Chen, L. Guibas, Jitendra Malik, and S. Savarese. 2020. Which tasks should be learned together in multi-task learning? In ICML. [find](https://scholar.google.com/scholar?q=T%2E+Standley%2C+A%2E+Zamir%2C+Dawn+Chen%2C+L%2E+Guibas%2C+Jitendra+Malik%2C+and+S%2E+Savarese%2E+2020%2E+Which+tasks+should+be+learned+together+in+multi%2Dtask+learning%3F+In+ICML%2E)

*Updated 24 October 2026*