--- title: "Understanding and Bridging the Modality Gap for — Fang 2023" description: "End-to-end speech translation (ST) has little parallel training data, so systems often borrow knowledge from text machine translation (MT) through…" authors: ["Qingkai Fang", "Yang Feng"] year: 2023 venue: "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)" source: "https://aclanthology.org/2023.acl-long.884/" doi: "10.18653/v1/2023.acl-long.884" arxiv: "2305.08706" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/papers/fang-2023-understanding-bridging-modality-gap-speech/" --- [Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/fang-2023-understanding-bridging-modality-gap-speech/) [home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2023](https://md.research.elamur.ai/free/papers/2023/page/6/) / Understanding and Bridging the Modality Gap for Speech Translation # Understanding and Bridging the Modality Gap for Speech Translation **Authors:** [Qingkai Fang](https://md.research.elamur.ai/free/authors/qingkai-fang/), [Yang Feng](https://md.research.elamur.ai/free/authors/yang-feng/) · **Year:** 2023 · **Venue:** Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2023.acl-long.884/) · [PDF](https://aclanthology.org/2023.acl-long.884.pdf) · [DOI 10.18653/v1/2023.acl-long.884](https://doi.org/10.18653/v1/2023.acl-long.884) · [arXiv 2305.08706](https://arxiv.org/abs/2305.08706) · [OpenAlex](https://openalex.org/W4385571004) **Citations:** 15 citations (openalex-snapshot · 2026-10-14) · [16](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary End-to-end speech translation (ST) has little parallel training data, so systems often borrow knowledge from text machine translation (MT) through multi-task learning with a shared encoder-decoder. Fang and Feng ask why ST still trails MT. They quantify a "modality gap" as the cosine distance between final decoder states computed from speech and from its transcript. On MuST-C En→De, the gap is mostly small under teacher forcing, apart from a long tail of hard tokens. At inference, however, it widens with every decoding step, because ST and MT condition on diverging self-generated prefixes. The authors frame this compounding effect as a form of exposure bias. Their method, CRESS, applies scheduled sampling to both tasks, so training contexts mix gold and model-predicted tokens. It also adds a KL-divergence term that pulls the ST and MT output distributions together. A token-level adaptive weighting further emphasizes tokens with large gaps. CRESS outperforms a strong multi-task baseline in all eight MuST-C directions, with the largest benefit on long sentences. The work offers both a diagnostic tool and a practical training recipe for speech-to-text transfer under data scarcity. ## Key points - Model: a pre-trained HuBERT encoder plus two convolutional layers that shorten the speech sequence. These feed a Transformer translation model, pre-trained on MT data, whose layers are fully shared between speech and text inputs. - The modality gap is defined as 1 − cosine similarity of last-decoder-layer representations. On the MuST-C En→De dev set it is below 10% in most cases. - Under greedy search the gap grows steadily with decoding steps. Beam search (beam size 8) slows this growth, and teacher forcing keeps the gap flat across steps. - Average improvements over the multi-task learning baseline across eight MuST-C directions: +1.8 BLEU (base setting) and +1.3 BLEU (expanded setting). - Code is released at github.com/ictnlp/CRESS (ACL 2023, Long Papers). *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) Topics: [Machine translation](https://md.research.elamur.ai/free/topics/machine-translation/) ## Cited by Cited by 16 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 15 not yet. - [Bridging the Gap between Synthetic and Authentic Images for Multimodal Machine Translation](https://md.research.elamur.ai/free/papers/guo-2023-bridging-gap-between-synthetic-authentic/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{fang2023understanding, title = {{Understanding and Bridging the Modality Gap for Speech Translation}}, author = {Fang, Qingkai and Feng, Yang}, booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, year = {2023}, doi = {10.18653/v1/2023.acl-long.884}, eprint = {2305.08706}, archivePrefix = {arXiv}, url = {https://doi.org/10.18653/v1/2023.acl-long.884} } ``` - [← previous UMR Annotation of Multiword Expressions](https://md.research.elamur.ai/free/papers/bonn-2023-umr-annotation-multiword-expressions/) - [next → Understanding the Effects of RLHF on LLM Generalisation and Diversity](https://md.research.elamur.ai/free/papers/kirk-2023-understanding-effects-rlhf-llm-generalisation/) - [↑ back to papers of 2023](https://md.research.elamur.ai/free/papers/2023/page/6/)