---
title: "Towards Dynamic Attention Masking for — Pong 2026"
description: "This IWSLT 2026 system paper asks whether adaptive lookahead, an idea from streaming ASR (ANCAT), can carry over to simultaneous speech translation."
authors: ["Benjamin Pong"]
year: 2026
venue: "Proceedings of the 23rd International Conference on Spoken Language Translation (IWSLT 2026)"
source: "https://aclanthology.org/2026.iwslt-1.20/"
doi: "10.18653/v1/2026.iwslt-1.20"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/pong-2026-dynamic-attention-masking-simultaneous-speech/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/pong-2026-dynamic-attention-masking-simultaneous-speech/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2026](https://md.research.elamur.ai/free/papers/2026/page/5/) / Towards Dynamic Attention Masking for Simultaneous Speech Translation

# Towards Dynamic Attention Masking for Simultaneous Speech Translation

**Authors:** Benjamin Pong · **Year:** 2026 · **Venue:** Proceedings of the 23rd International Conference on Spoken Language Translation (IWSLT 2026) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.iwslt-1.20/) · [PDF](https://aclanthology.org/2026.iwslt-1.20.pdf) · [DOI 10.18653/v1/2026.iwslt-1.20](https://doi.org/10.18653/v1/2026.iwslt-1.20) · [OpenAlex](https://openalex.org/W7166838643)

**Citations:** 0 citations (openalex-snapshot · 2026-10-14) · 1 in our library, of the 140673 papers whose bibliographies we read

## Summary
This IWSLT 2026 system paper asks whether adaptive lookahead, an idea from streaming ASR (ANCAT), can carry over to simultaneous speech translation. The author extends SeamlessM4T-medium by adding a small scheduler network to each of its 12 conformer encoder layers. Each scheduler estimates, frame by frame, how much future audio is needed and produces a soft additive mask on self-attention, capped by a lookahead budget K_max. The schedulers are fine-tuned together with LoRA adapters while the rest of the model stays frozen. Training covers English to German, Italian and Chinese on Europarl and CoVoST2. Evaluation uses the long-form MCIF dev set (ACL conference talks) under two policies: sliding window retranslation, and a modified StreamAtt whose fixed cutoff is replaced by a threshold based on scheduler scores. Translation quality falls well below the unmodified baseline, likely because of the masking constraints and a domain mismatch. However, the score-based emission cutoff often lowers latency, which suggests the schedulers learn a usable timing signal. The work is presented as a first step toward training-time lookahead control in this setting.

## Key points
- **Architecture:** Each scheduler is a two-layer feedforward network over a low-dimensional projection of hidden states.
  - K_max is 24 for En→De/It and 16 for En→Zh.
  - LoRA is applied to the self-attention output projections in both the encoder and decoder.
  - Training ran on a single A100 with HuggingFace Transformers; inference was built on the SimulStream toolkit.
- **Metrics:** chrF, COMET (Unbabel/XCOMET-XL) and LongYAAL (computation-unaware).
- **Sliding window (12 s window), quality trend:** Moving from 4 s to 8 s chunks, the dynamic model's COMET rises (De +0.024, Zh +0.012, It +0.003), while the baseline's falls (−0.008, −0.004, −0.031).
- **Sliding window, absolute results:** Absolute quality stays lower than the baseline, and the dynamic model shows no latency advantage.
- **Modified StreamAtt (En→De/It only), latency:**
  - En→De at 4 s reaches 1976 ms, 817 ms faster than the baseline's 2793 ms, but German results are inconsistent across chunk sizes.
  - En→It has lower latency than the baseline at every chunk size, with negative LongYAAL at 4 s and 6 s (−557 and −1748 ms).
- **Modified StreamAtt, quality cost:** Quality drops sharply, for example En→It chrF at 4 s is 25.6 versus 57.40 for the baseline.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{pong2026dynamic,
  title = {{Towards Dynamic Attention Masking for Simultaneous Speech Translation}},
  author = {Pong, Benjamin},
  booktitle = {Proceedings of the 23rd International Conference on Spoken Language Translation (IWSLT 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.iwslt-1.20},
  url = {https://doi.org/10.18653/v1/2026.iwslt-1.20}
}
```

- [← previous Towards Consistent Detection of Cognitive Distortions: LLM-Based Annotation and Dataset-Agnostic Evaluation](https://md.research.elamur.ai/free/papers/sharma-2026-consistent-detection-cognitive-distortions-llm/)
- [next → Towards Reliable AI Fairness: Challenges in Steering Features within Bias-Implicated Neurons](https://md.research.elamur.ai/free/papers/munoz-2026-reliable-ai-fairness-challenges-steering/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/papers/2026/page/5/)

## References

20 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. David Ifeoluwa Adelani, Victor Agostinelli, Antonios Anastasopoulos, Luisa Bentivogli, Ondřej Bojar, Sebastien Bratières, Marine Carpuat, Roldano Cattoni, Mauro Cettolo, Lizhong Chen, Marcello Federico, Marco Gaido, Mahendra Gupta, HyoJung Han, Ali Hatami, David Javorský, Yejin Jeon, Marek Kaszteln… [find](https://scholar.google.com/scholar?q=David+Ifeoluwa+Adelani%2C+Victor+Agostinelli%2C+Antonios+Anastasopoulos%2C+Luisa+Bentivogli%2C+Ondr%CC%8Cej+Bojar%2C+Sebastien+Brati%C3%A8res%2C+Marine+Carpuat%2C+Roldano+Cattoni%2C+Mauro+Cettolo%2C+Lizhong+Chen%2C+Marcello+Federi)
2. Xie Chen, Yu Wu, Zhenghao Wang, Shujie Liu, and Jinyu Li. 2021. Developing real-time streaming transformer transducer for speech recognition on large-scale dataset. Preprint, arXiv:2010.11395. — [arXiv:2010.11395](https://arxiv.org/abs/2010.11395) [find](https://scholar.google.com/scholar?q=Xie+Chen%2C+Yu+Wu%2C+Zhenghao+Wang%2C+Shujie+Liu%2C+and+Jinyu+Li%2E+2021%2E+Developing+real%2Dtime+streaming+transformer+transducer+for+speech+recognition+on+large%2Dscale+dataset%2E+Preprint%2C+arXiv%3A2010%2E11395%2E+%E2%80%94+%5BarXi)
3. Seamless Communication, Loïc Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, Christopher Klaiber, Pengwei Li, Daniel Licht, Jean Maillard, Alice Rakotoarison, Kaushik Ram Sadagopan, Guillaume Wenze… [find](https://scholar.google.com/scholar?q=Seamless+Communication%2C+Lo%C3%AFc+Barrault%2C+Yu%2DAn+Chung%2C+Mariano+Cora+Meglioli%2C+David+Dale%2C+Ning+Dong%2C+Paul%2DAmbroise+Duquenne%2C+Hady+Elsahar%2C+Hongyu+Gong%2C+Kevin+Heffernan%2C+John+Hoffman%2C+Christopher+Klaiber%2C) [our card](https://md.research.elamur.ai/free/papers/communication-2023-seamlessm4t-massively-multilingual-multimodal-machine/)
4. Marco Gaido, Mattia A Di Gangi, Matteo Negri, and Marco Turchi. 2020. End-to-end speech-translation with knowledge distillation: Fbk@ iwslt2020. In Proceedings of the 17th International Conference on Spoken Language Translation, pages 80–88. [find](https://scholar.google.com/scholar?q=Marco+Gaido%2C+Mattia+A+Di+Gangi%2C+Matteo+Negri%2C+and+Marco+Turchi%2E+2020%2E+End%2Dto%2Dend+speech%2Dtranslation+with+knowledge+distillation%3A+Fbk%40+iwslt2020%2E+In+Proceedings+of+the+17th+International+Conference+on+)
5. Marco Gaido, Sara Papi, Mauro Cettolo, Matteo Negri, and Luisa Bentivogli. 2025. simulstream: Open-Source Toolkit for Evaluation and Demonstration of Streaming Speech-to-Text Translation Systems. arXiv. [find](https://scholar.google.com/scholar?q=Marco+Gaido%2C+Sara+Papi%2C+Mauro+Cettolo%2C+Matteo+Negri%2C+and+Luisa+Bentivogli%2E+2025%2E+simulstream%3A+Open%2DSource+Toolkit+for+Evaluation+and+Demonstration+of+Streaming+Speech%2Dto%2DText+Translation+Systems%2E+arXi)
6. Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André F. T. Martins. 2023. xcomet: Transparent machine translation evaluation through fine-grained error detection. Preprint, arXiv:2310.10482. [find](https://scholar.google.com/scholar?q=Nuno+M%2E+Guerreiro%2C+Ricardo+Rei%2C+Daan+van+Stigt%2C+Luisa+Coheur%2C+Pierre+Colombo%2C+and+Andr%C3%A9+F%2E+T%2E+Martins%2E+2023%2E+xcomet%3A+Transparent+machine+translation+evaluation+through+fine%2Dgrained+error+detection%2E+Pr) [our card](https://md.research.elamur.ai/free/papers/guerreiro-2024-xcomet-transparent-machine-translation-evaluation/)
7. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685. [find](https://scholar.google.com/scholar?q=Edward+J%2E+Hu%2C+Yelong+Shen%2C+Phillip+Wallis%2C+Zeyuan+Allen%2DZhu%2C+Yuanzhi+Li%2C+Shean+Wang%2C+Lu+Wang%2C+and+Weizhu+Chen%2E+2021%2E+Lora%3A+Low%2Drank+adaptation+of+large+language+models%2E+Preprint%2C+arXiv%3A2106%2E09685%2E)
8. Hirofumi Inaguma, Tatsuya Kawahara, and Shinji Watanabe. 2021. Source and target bidirectional knowledge distillation for end-to-end speech translation. In Proceedings of the 2021 conference of the North American chapter of the association for computational linguistics: human language technologies,… [find](https://scholar.google.com/scholar?q=Hirofumi+Inaguma%2C+Tatsuya+Kawahara%2C+and+Shinji+Watanabe%2E+2021%2E+Source+and+target+bidirectional+knowledge+distillation+for+end%2Dto%2Dend+speech+translation%2E+In+Proceedings+of+the+2021+conference+of+the+No)
9. Javier Iranzo-Sánchez, Joan Albert Silvestre-Cerdà, Javier Jorge, Nahuel Roselló, Adrià Giménez, Albert Sanchis, Jorge Civera, and Alfons Juan. 2020. Europarl-st: A multilingual corpus for speech translation of parliamentary debates. In ICASSP 2020 -2020 IEEE International Conference on Acoustics, S… [find](https://scholar.google.com/scholar?q=Javier+Iranzo%2DS%C3%A1nchez%2C+Joan+Albert+Silvestre%2DCerd%C3%A0%2C+Javier+Jorge%2C+Nahuel+Rosell%C3%B3%2C+Adri%C3%A0+Gim%C3%A9nez%2C+Albert+Sanchis%2C+Jorge+Civera%2C+and+Alfons+Juan%2E+2020%2E+Europarl%2Dst%3A+A+multilingual+corpus+for+speech+tran)
10. Siqi Ouyang, Xi Xu, and Lei Li. 2025. [find](https://scholar.google.com/scholar?q=Siqi+Ouyang%2C+Xi+Xu%2C+and+Lei+Li%2E+2025%2E)
11. CMU’s IWSLT 2025 simultaneous speech translation system. In Proceedings of the 22nd International Conference on Spoken Language Translation (IWSLT 2025), pages 309–314, Vienna, Austria (in-person and online). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=CMU%E2%80%99s+IWSLT+2025+simultaneous+speech+translation+system%2E+In+Proceedings+of+the+22nd+International+Conference+on+Spoken+Language+Translation+%28IWSLT+2025%29%2C+pages+309%E2%80%93314%2C+Vienna%2C+Austria+%28in%2Dperson+and+)
12. Sara Papi, Marco Gaido, Matteo Negri, and Luisa Bentivogli. 2024. Streamatt: Direct streaming speech-to-text translation with attention-based audio history selection. Preprint, arXiv:2406.06097. [find](https://scholar.google.com/scholar?q=Sara+Papi%2C+Marco+Gaido%2C+Matteo+Negri%2C+and+Luisa+Bentivogli%2E+2024%2E+Streamatt%3A+Direct+streaming+speech%2Dto%2Dtext+translation+with+attention%2Dbased+audio+history+selection%2E+Preprint%2C+arXiv%3A2406%2E06097%2E)
13. A Sara Papi, Maike Züfle, Marco Gaido, Beatrice Savoldi, Danni Liu, Ioannis Douros, Luisa Bentivogli, and Jan Niehues. 2026. Mcif: Multimodal crosslingual instruction-following benchmark from scientific talks. Preprint, arXiv:2507.19634. — [arXiv:2507.19634](https://arxiv.org/abs/2507.19634) [find](https://scholar.google.com/scholar?q=A+Sara+Papi%2C+Maike+Z%C3%BCfle%2C+Marco+Gaido%2C+Beatrice+Savoldi%2C+Danni+Liu%2C+Ioannis+Douros%2C+Luisa+Bentivogli%2C+and+Jan+Niehues%2E+2026%2E+Mcif%3A+Multimodal+crosslingual+instruction%2Dfollowing+benchmark+from+scientif)
14. Peter Polák, Sara Papi, Luisa Bentivogli, and Ondřej Bojar. 2026. Better late than never: Meta-evaluation of latency metrics for simultaneous speech-to-text translation. Preprint, arXiv:2509.17349. [find](https://scholar.google.com/scholar?q=Peter+Pol%C3%A1k%2C+Sara+Papi%2C+Luisa+Bentivogli%2C+and+Ondr%CC%8Cej+Bojar%2E+2026%2E+Better+late+than+never%3A+Meta%2Devaluation+of+latency+metrics+for+simultaneous+speech%2Dto%2Dtext+translation%2E+Preprint%2C+arXiv%3A2509%2E17349%2E)
15. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Maja+Popovic%CC%81%2E+2015%2E+chrF%3A+character+n%2Dgram+F%2Dscore+for+automatic+MT+evaluation%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+Translation%2C+pages+392%E2%80%93395%2C+Lisbon%2C+Portugal%2E+Association+f) [our card](https://md.research.elamur.ai/free/papers/popovic-2015-chrf-character-n-gram-f/)
16. Sukanta Sen, Ondřej Bojar, and Barry Haddow. 2022. Simultaneous translation for unsegmented input: A sliding window approach. Preprint, arXiv:2210.09754. — [arXiv:2210.09754](https://arxiv.org/abs/2210.09754) [find](https://scholar.google.com/scholar?q=Sukanta+Sen%2C+Ondr%CC%8Cej+Bojar%2C+and+Barry+Haddow%2E+2022%2E+Simultaneous+translation+for+unsegmented+input%3A+A+sliding+window+approach%2E+Preprint%2C+arXiv%3A2210%2E09754%2E+%E2%80%94+%5BarXiv%3A2210%2E09754%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F22)
17. Yangyang Shi, Yongqiang Wang, Chunyang Wu, Ching-Feng Yeh, Julian Chan, Frank Zhang, Duc Le, and Mike Seltzer. 2020. Emformer: Efficient memory transformer based acoustic model for low latency streaming speech recognition. Preprint, arXiv:2010.10759. — [arXiv:2010.10759](https://arxiv.org/abs/2010.10759) [find](https://scholar.google.com/scholar?q=Yangyang+Shi%2C+Yongqiang+Wang%2C+Chunyang+Wu%2C+Ching%2DFeng+Yeh%2C+Julian+Chan%2C+Frank+Zhang%2C+Duc+Le%2C+and+Mike+Seltzer%2E+2020%2E+Emformer%3A+Efficient+memory+transformer+based+acoustic+model+for+low+latency+streami)
18. Grant Strimel, Yi Xie, Brian King, Martin Radfar, Ariya Rastrow, and Thanasis Mouchtaris. 2023. Lookahead when it matters: Adaptive non-causal transformers for streaming neural transducers. [find](https://scholar.google.com/scholar?q=Grant+Strimel%2C+Yi+Xie%2C+Brian+King%2C+Martin+Radfar%2C+Ariya+Rastrow%2C+and+Thanasis+Mouchtaris%2E+2023%2E+Lookahead+when+it+matters%3A+Adaptive+non%2Dcausal+transformers+for+streaming+neural+transducers%2E)
19. Changhan Wang, Anne Wu, Jiatao Gu, and Juan Pino. 2021. CoVoST 2 and Massively Multilingual Speech Translation. In Interspeech 2021, pages 2247–2251. [find](https://scholar.google.com/scholar?q=Changhan+Wang%2C+Anne+Wu%2C+Jiatao+Gu%2C+and+Juan+Pino%2E+2021%2E+CoVoST+2+and+Massively+Multilingual+Speech+Translation%2E+In+Interspeech+2021%2C+pages+2247%E2%80%932251%2E)
20. Training Parameters Hyperparameter Value LoRA Rank r α Dropout Target modules 8 16 0.15 q,k,v linear modules Optimizer Optimizer Scheduler MLP learning rate LoRA learning rate Weight decay Gradient clipping AdamW 5 × 10 −6 3 × 10 −6 0.01 1.0 Training Batch size (per device) Gradient accumulation ste… [find](https://scholar.google.com/scholar?q=Training+Parameters+Hyperparameter+Value+LoRA+Rank+r+%CE%B1+Dropout+Target+modules+8+16+0%2E15+q%2Ck%2Cv+linear+modules+Optimizer+Optimizer+Scheduler+MLP+learning+rate+LoRA+learning+rate+Weight+decay+Gradient+cl)