---
title: "S2ST-Omni — paper by Pan et al. (2026) — summary and ideas"
description: "S2ST-Omni targets multilingual speech-to-speech translation (S2ST). 34 references. Among them: 0."
authors: ["Yu Pan", "Xiongfei Wu", "Yang Yuguang", "Jixun Yao", "Maxime Cordy", "Lei Ma", "Jianjun Zhao"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.1004/"
doi: "10.18653/v1/2026.findings-acl.1004"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/pan-2026-s2st-omni-hierarchical-language-aware/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pan-2026-s2st-omni-hierarchical-language-aware/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/42/) / S2ST-Omni: Hierarchical Language-Aware SpeechLLM Adaptation for Multilingual Speech-to-Speech Translation

# S2ST-Omni: Hierarchical Language-Aware SpeechLLM Adaptation for Multilingual Speech-to-Speech Translation

**Authors:** [Yu Pan](https://md.research.elamur.ai/free/en/authors/yu-pan/), Xiongfei Wu, Yang Yuguang, Jixun Yao, Maxime Cordy, Lei Ma and Jianjun Zhao · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1004/) · [PDF](https://aclanthology.org/2026.findings-acl.1004.pdf) · [DOI 10.18653/v1/2026.findings-acl.1004](https://doi.org/10.18653/v1/2026.findings-acl.1004) · [OpenAlex](https://openalex.org/W4416059734)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
S2ST-Omni targets multilingual speech-to-speech translation (S2ST). End-to-end systems in this area face hard trade-offs between sub-tasks, depend on scarce speech-to-speech parallel data and give no inspectable text. Earlier compositional pipelines use shallow interfaces and ignore the identity of the source language. The authors split the task into a SpeechLLM speech-to-text translation (S2TT) frontend and an interchangeable TTS backend. The frontend connects a Whisper Large-v3 encoder to a Qwen3-4B LLM through a hybrid adapter. Convolution blocks first capture local acoustic patterns, strided downsampling then shortens the sequence, and Transformer layers finally model long-range context. Source-language cues enter at two levels. LA-Dual-CTC applies gated, FiLM-style modulation to intermediate adapter features, and Language-Aware Prompting conditions the LLM prompt on the predicted language. Training has two stages: speech-text alignment comes first, followed by LoRA tuning for translation. On CVSS-C (French, Spanish and German into English), the system outperforms eight E2E and compositional baselines in BLEU, COMET and ASR-BLEU. Because the text frontend is strong and language-aware, any modern synthesizer can be attached without retraining.

## Key points
- Architecture: Whisper Large-v3 encoder, a "local-then-global" hybrid adapter and Qwen3-4B. Hierarchical language-aware (HLA) augmentation is built from LA-Dual-CTC and LAP. Training uses progressive fine-tuning (PFT) with LoRA.
- Language ID reuses Whisper itself instead of a separate classifier. Its accuracy is 99.67% (Fr), 99.76% (De) and 99.80% (Es), and it adds only 22.8 ms of latency.
- Results on CVSS-C: average BLEU is 35.67 and ASR-BLEU is 33.45 (with IndexTTS2). Removing PFT lowers these to 34.37 and 31.81.
- Five TTS backends were tested (IndexTTS2, FireRedTTS2, CosyVoice3, ZipVoice, VoxCPM1.5). Their average ASR-BLEU stays between 32.09 and 33.45, and even the weakest outperforms StreamSpeech.
- The S2TT frontend takes about 704 ms per utterance on an RTX A5000, with roughly 79% of that spent in LLM decoding. Limitations: the study covers only three high-resource source languages, English as the only target, a single benchmark and no human evaluation.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [The University of Tokyo](https://md.research.elamur.ai/free/en/universities/the-university-of-tokyo/) — University, Tokyo, Japan · 90 works · 9,156 citations
- [ByteDance](https://md.research.elamur.ai/free/en/companies/bytedance/) — Company, Beijing, China · 41 works · 4,180 citations
- [University of Luxembourg](https://md.research.elamur.ai/free/en/universities/university-of-luxembourg/) — University, Luxembourg, Luxembourg · 27 works · 337 citations
- [Kyushu University](https://md.research.elamur.ai/free/en/universities/kyushu-university/) — University, Fukuoka, Japan · 4 works · 25 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🔊 Signal Processing](https://md.research.elamur.ai/free/en/shelves/computer-science/signal-processing/) · 452 works

🧭 Topics (2): [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 828 works · [🎙️ Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) · 480 works

## Cite

Cite as: Yu Pan, Xiongfei Wu, Yang Yuguang, Jixun Yao, Maxime Cordy, Lei Ma and Jianjun Zhao. 2026. S2ST-Omni: Hierarchical Language-Aware SpeechLLM Adaptation for Multilingual Speech-to-Speech Translation. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{pan2026s2st,
  title = {{S2ST-Omni: Hierarchical Language-Aware SpeechLLM Adaptation for Multilingual Speech-to-Speech Translation}},
  author = {Yu Pan, Xiongfei Wu, Yang Yuguang, Jixun Yao, Maxime Cordy, Lei Ma and Jianjun Zhao},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1004},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1004}
}
```

- [← previous S2H-DPO: Hardness-Aware Preference Optimization for Vision–Language Models](https://md.research.elamur.ai/free/en/papers/shukla-2026-s2h-dpo-hardness-aware-preference/)
- [next → Saar-Voice: A Multi-Speaker Saarbrücken Dialect Speech Corpus](https://md.research.elamur.ai/free/en/papers/oberkircher-2026-saar-voice-multi-speaker-saarbrucken/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/42/)

## References

34 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Loïc Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, and 1 others. 2023. [Seamlessm4t: massively multilingual & multimodal machine translation](https://md.research.elamur.ai/free/en/papers/communication-2023-seamlessm4t-massively-multilingual-multimodal-machine/). arXiv preprint arXiv:2308.11596.
2. Sijing Chen, Yuan Feng, Laipeng He, Tianwei He, Wendi He, Yanni Hu, Bin Lin, Yiting Lin, Yu Pan, Pengfei Tan, and 1 others. 2024. Takin: A cohort of superior quality zero-shot speech generation models. arXiv preprint arXiv:2409.12139. — [arXiv:2409.12139](https://arxiv.org/abs/2409.12139) [find](https://scholar.google.com/scholar?q=Sijing+Chen%2C+Yuan+Feng%2C+Laipeng+He%2C+Tianwei+He%2C+Wendi+He%2C+Yanni+Hu%2C+Bin+Lin%2C+Yiting+Lin%2C+Yu+Pan%2C+Pengfei+Tan%2C+and+1+others%2E+2024%2E+Takin%3A+A+cohort+of+superior+quality+zero%2Dshot+speech+generation+models)
3. Yunfei Chu and 1 others. 2023. [Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models](https://md.research.elamur.ai/free/en/papers/chu-2023-qwen-audio-advancing-universal/). arXiv preprint arXiv:2311.07919.
4. Yunfei Chu and 1 others. 2024. [Qwen2-audio technical report](https://md.research.elamur.ai/free/en/papers/chu-2024-qwen2-audio-technical-report/). arXiv preprint arXiv:2407.10759.
5. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
6. Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. 2025. Llama-omni: Seamless speech interaction with large language models. In International Conference on Learning Representations, volume 2025, pages 57607–57624. [find](https://scholar.google.com/scholar?q=Qingkai+Fang%2C+Shoutao+Guo%2C+Yan+Zhou%2C+Zhengrui+Ma%2C+Shaolei+Zhang%2C+and+Yang+Feng%2E+2025%2E+Llama%2Domni%3A+Seamless+speech+interaction+with+large+language+models%2E+In+International+Conference+on+Learning+Repres)
7. Qingkai Fang, Shaolei Zhang, Zhengrui Ma, Min Zhang, and Yang Feng. 2024. Can we achieve high-quality direct speech-to-speech translation without parallel speech data? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7264–7277,… [find](https://scholar.google.com/scholar?q=Qingkai+Fang%2C+Shaolei+Zhang%2C+Zhengrui+Ma%2C+Min+Zhang%2C+and+Yang+Feng%2E+2024%2E+Can+we+achieve+high%2Dquality+direct+speech%2Dto%2Dspeech+translation+without+parallel+speech+data%3F+In+Proceedings+of+the+62nd+Annua)
8. Qingkai Fang, Yan Zhou, and Yang Feng. 2023. Daspeech: Directed acyclic transformer for fast and high-quality speech-to-speech translation. Advances in Neural Information Processing Systems, 36:72604–72623. [find](https://scholar.google.com/scholar?q=Qingkai+Fang%2C+Yan+Zhou%2C+and+Yang+Feng%2E+2023%2E+Daspeech%3A+Directed+acyclic+transformer+for+fast+and+high%2Dquality+speech%2Dto%2Dspeech+translation%2E+Advances+in+Neural+Information+Processing+Systems%2C+36%3A72604%E2%80%93)
9. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In ICML. [find](https://scholar.google.com/scholar?q=Alex+Graves%2C+Santiago+Fern%C3%A1ndez%2C+Faustino+Gomez%2C+and+J%C3%BCrgen+Schmidhuber%2E+2006%2E+Connectionist+temporal+classification%3A+Labelling+unsegmented+sequence+data+with+recurrent+neural+networks%2E+In+ICML%2E)
10. Yingxu He, Zhuohan Liu, Geyu Lin, Shuo Sun, Bin Wang, Wenyu Zhang, Xunlong Zou, Nancy Chen, and Aiti Aw. 2025. Meralion-audiollm: Advancing speech and language understanding for singapore. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 3: System De… [find](https://scholar.google.com/scholar?q=Yingxu+He%2C+Zhuohan+Liu%2C+Geyu+Lin%2C+Shuo+Sun%2C+Bin+Wang%2C+Wenyu+Zhang%2C+Xunlong+Zou%2C+Nancy+Chen%2C+and+Aiti+Aw%2E+2025%2E+Meralion%2Daudiollm%3A+Advancing+speech+and+language+understanding+for+singapore%2E+In+Proceedi)
11. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). ICLR, 1(2):3.
12. Hirofumi Inaguma, Sravya Popuri, Ilia Kulikov, Peng-Jen Chen, Changhan Wang, Yu-An Chung, Yun Tang, Ann Lee, Shinji Watanabe, and Juan Pino. 2023. UnitY: Two-pass [direct speech-to-speech translation with discrete units](https://md.research.elamur.ai/free/en/papers/lee-2022-direct-speech-speech-translation-discrete/). In Proceedings of the 61st Annual Meeting of the Association for Computational L…
13. Ye Jia, Michelle Tadmor Ramanovich, Tal Remez, and Roi Pomerantz. 2022a. Translatotron 2: High-quality direct speech-to-speech translation with voice preservation. In International Conference on Machine Learning, pages 10120–10134. PMLR. [find](https://scholar.google.com/scholar?q=Ye+Jia%2C+Michelle+Tadmor+Ramanovich%2C+Tal+Remez%2C+and+Roi+Pomerantz%2E+2022a%2E+Translatotron+2%3A+High%2Dquality+direct+speech%2Dto%2Dspeech+translation+with+voice+preservation%2E+In+International+Conference+on+Machi)
14. Ye Jia, Michelle Tadmor Ramanovich, Quan Wang, and Heiga Zen. 2022b. CVSS corpus and massively multilingual speech-to-speech translation. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6691–6703, Marseille, France. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Ye+Jia%2C+Michelle+Tadmor+Ramanovich%2C+Quan+Wang%2C+and+Heiga+Zen%2E+2022b%2E+CVSS+corpus+and+massively+multilingual+speech%2Dto%2Dspeech+translation%2E+In+Proceedings+of+the+Thirteenth+Language+Resources+and+Evalua)
15. Ye Jia, Ron J Weiss, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Zhifeng Chen, and Yonghui Wu. 2019. Direct speech-to-speech translation with a sequence-to-sequence model. arXiv preprint arXiv:1904.06037. [find](https://scholar.google.com/scholar?q=Ye+Jia%2C+Ron+J+Weiss%2C+Fadi+Biadsy%2C+Wolfgang+Macherey%2C+Melvin+Johnson%2C+Zhifeng+Chen%2C+and+Yonghui+Wu%2E+2019%2E+Direct+speech%2Dto%2Dspeech+translation+with+a+sequence%2Dto%2Dsequence+model%2E+arXiv+preprint+arXiv%3A190)
16. Gen-ichiro Kikui, Eiichiro Sumita, Toshiyuki Takezawa, and Seiichi Yamamoto. 2003. Creating corpora for speech-to-speech translation. In INTERSPEECH, pages 381–384. [find](https://scholar.google.com/scholar?q=Gen%2Dichiro+Kikui%2C+Eiichiro+Sumita%2C+Toshiyuki+Takezawa%2C+and+Seiichi+Yamamoto%2E+2003%2E+Creating+corpora+for+speech%2Dto%2Dspeech+translation%2E+In+INTERSPEECH%2C+pages+381%E2%80%93384%2E)
17. Suyoun Kim, Takaaki Hori, and Shinji Watanabe. 2017. Joint ctc-attention based end-to-end speech recognition using multi-task learning. In ICASSP. [find](https://scholar.google.com/scholar?q=Suyoun+Kim%2C+Takaaki+Hori%2C+and+Shinji+Watanabe%2E+2017%2E+Joint+ctc%2Dattention+based+end%2Dto%2Dend+speech+recognition+using+multi%2Dtask+learning%2E+In+ICASSP%2E)
18. Tom Labiausse, Laurent Mazaré, Edouard Grave, Alexandre Défossez, and Neil Zeghidour. 2025. High-fidelity simultaneous speech-to-speech translation. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 32116–32129. PML… [find](https://scholar.google.com/scholar?q=Tom+Labiausse%2C+Laurent+Mazar%C3%A9%2C+Edouard+Grave%2C+Alexandre+D%C3%A9fossez%2C+and+Neil+Zeghidour%2E+2025%2E+High%2Dfidelity+simultaneous+speech%2Dto%2Dspeech+translation%2E+In+Proceedings+of+the+42nd+International+Conference)
19. Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Sravya Popuri, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, Juan Pino, and Wei-Ning Hsu. 2022a. [Direct speech-to-speech translation with discrete units](https://md.research.elamur.ai/free/en/papers/lee-2022-direct-speech-speech-translation-discrete/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistic…
20. Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Yossi Adi, Juan Pino, Jiatao Gu, and Wei-Ning Hsu. 2022b. Textless speech-to-speech translation on real data. In Proceedings of the 2022 Conference of the North American Chapter of the Associat… [find](https://scholar.google.com/scholar?q=Ann+Lee%2C+Hongyu+Gong%2C+Paul%2DAmbroise+Duquenne%2C+Holger+Schwenk%2C+Peng%2DJen+Chen%2C+Changhan+Wang%2C+Sravya+Popuri%2C+Yossi+Adi%2C+Juan+Pino%2C+Jiatao+Gu%2C+and+Wei%2DNing+Hsu%2E+2022b%2E+Textless+speech%2Dto%2Dspeech+translati)
21. Zhiliang Peng, Jianwei Yu, Wenhui Wang, Yaoyao Chang, Yutao Sun, Li Dong, Yi Zhu, Weijiang Xu, Hangbo Bao, Zehua Wang, and 1 others. 2025. Vibevoice technical report. arXiv preprint arXiv:2508.19205. — [arXiv:2508.19205](https://arxiv.org/abs/2508.19205) [find](https://scholar.google.com/scholar?q=Zhiliang+Peng%2C+Jianwei+Yu%2C+Wenhui+Wang%2C+Yaoyao+Chang%2C+Yutao+Sun%2C+Li+Dong%2C+Yi+Zhu%2C+Weijiang+Xu%2C+Hangbo+Bao%2C+Zehua+Wang%2C+and+1+others%2E+2025%2E+Vibevoice+technical+report%2E+arXiv+preprint+arXiv%3A2508%2E19205%2E+)
22. Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. 2018. [Film: Visual reasoning with a general conditioning layer](https://md.research.elamur.ai/free/en/papers/perez-2017-film-visual-reasoning-general-conditioning/). In AAAI.
23. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. [Robust speech recognition via large-scale weak supervision](https://md.research.elamur.ai/free/en/papers/radford-2022-robust-speech-recognition-large-scale/). In International conference on machine learning, pages 28492–28518. PMLR.
24. Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. 2020. Fastspeech 2: Fast and high-quality end-to-end text to speech. arXiv preprint arXiv:2006.04558. [find](https://scholar.google.com/scholar?q=Yi+Ren%2C+Chenxu+Hu%2C+Xu+Tan%2C+Tao+Qin%2C+Sheng+Zhao%2C+Zhou+Zhao%2C+and+Tie%2DYan+Liu%2E+2020%2E+Fastspeech+2%3A+Fast+and+high%2Dquality+end%2Dto%2Dend+text+to+speech%2E+arXiv+preprint+arXiv%3A2006%2E04558%2E)
25. Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, and 1 others. 2023. [Audiopalm: A large language model that can speak and listen](https://md.research.elamur.ai/free/en/papers/rubenstein-2023-audiopalm-large-language-model-that/). arXiv preprint arXiv:2306.12925.
26. Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang. 2024. [SALMONN: Towards generic hearing abilities for large language models](https://md.research.elamur.ai/free/en/papers/tang-2023-salmonn-towards-generic-hearing-abilities/). In The Twelfth International Conference on Learning Representations.
27. Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R Hershey, and Tomoki Hayashi. 2017. Hybrid ctc/attention architecture for end-to-end speech recognition. IEEE Journal of Selected Topics in Signal Processing. [find](https://scholar.google.com/scholar?q=Shinji+Watanabe%2C+Takaaki+Hori%2C+Suyoun+Kim%2C+John+R+Hershey%2C+and+Tomoki+Hayashi%2E+2017%2E+Hybrid+ctc%2Fattention+architecture+for+end%2Dto%2Dend+speech+recognition%2E+IEEE+Journal+of+Selected+Topics+in+Signal+Proc)
28. Kai-Tuo Xu, Feng-Long Xie, Xu Tang, and Yao Hu. 2025. Fireredasr: Open-source industrial-grade mandarin speech recognition models from encoder-decoder to llm integration. arXiv preprint arXiv:2501.14350. — [arXiv:2501.14350](https://arxiv.org/abs/2501.14350) [find](https://scholar.google.com/scholar?q=Kai%2DTuo+Xu%2C+Feng%2DLong+Xie%2C+Xu+Tang%2C+and+Yao+Hu%2E+2025%2E+Fireredasr%3A+Open%2Dsource+industrial%2Dgrade+mandarin+speech+recognition+models+from+encoder%2Ddecoder+to+llm+integration%2E+arXiv+preprint+arXiv%3A2501%2E143)
29. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.
30. Yuguang Yang, Yu Pan, Jingjing Yin, Jiangyu Han, Lei Ma, and Heng Lu. 2023. Hybridformer: Improving squeezeformer with hybrid attention and nsr mechanism. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. [find](https://scholar.google.com/scholar?q=Yuguang+Yang%2C+Yu+Pan%2C+Jingjing+Yin%2C+Jiangyu+Han%2C+Lei+Ma%2C+and+Heng+Lu%2E+2023%2E+Hybridformer%3A+Improving+squeezeformer+with+hybrid+attention+and+nsr+mechanism%2E+In+ICASSP+2023%2D2023+IEEE+International+Confer)
31. Dong Zhang and 1 others. 2023. [Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities](https://md.research.elamur.ai/free/en/papers/zhang-2023-speechgpt-empowering-large-language-models/). arXiv preprint arXiv:2305.11000.
32. Shaolei Zhang, Qingkai Fang, Shoutao Guo, Zhengrui Ma, Min Zhang, and Yang Feng. 2024. Stream-Speech: Simultaneous speech-to-speech translation with multi-task learning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8964–898… [find](https://scholar.google.com/scholar?q=Shaolei+Zhang%2C+Qingkai+Fang%2C+Shoutao+Guo%2C+Zhengrui+Ma%2C+Min+Zhang%2C+and+Yang+Feng%2E+2024%2E+Stream%2DSpeech%3A+Simultaneous+speech%2Dto%2Dspeech+translation+with+multi%2Dtask+learning%2E+In+Proceedings+of+the+62nd+Ann)
33. Siyi Zhou, Yiquan Zhou, Yi He, Xun Zhou, Jinchao Wang, Wei Deng, and Jingchen Shu. 2025a. Indextts2: A breakthrough in emotionally expressive and duration-controlled auto-regressive zero-shot text-to-speech. arXiv preprint arXiv:2506.21619. — [arXiv:2506.21619](https://arxiv.org/abs/2506.21619) [find](https://scholar.google.com/scholar?q=Siyi+Zhou%2C+Yiquan+Zhou%2C+Yi+He%2C+Xun+Zhou%2C+Jinchao+Wang%2C+Wei+Deng%2C+and+Jingchen+Shu%2E+2025a%2E+Indextts2%3A+A+breakthrough+in+emotionally+expressive+and+duration%2Dcontrolled+auto%2Dregressive+zero%2Dshot+text%2Dto%2D)
34. Yixuan Zhou, Guoyang Zeng, Xin Liu, Xiang Li, Renjie Yu, Ziyang Wang, Runchuan Ye, Weiyue Sun, Jiancheng Gui, Kehan Li, and 1 others. 2025b. Voxcpm: Tokenizer-free tts for context-aware speech generation and true-to-life voice cloning. arXiv preprint arXiv:2509.24650. — [arXiv:2509.24650](https://arxiv.org/abs/2509.24650) [find](https://scholar.google.com/scholar?q=Yixuan+Zhou%2C+Guoyang+Zeng%2C+Xin+Liu%2C+Xiang+Li%2C+Renjie+Yu%2C+Ziyang+Wang%2C+Runchuan+Ye%2C+Weiyue+Sun%2C+Jiancheng+Gui%2C+Kehan+Li%2C+and+1+others%2E+2025b%2E+Voxcpm%3A+Tokenizer%2Dfree+tts+for+context%2Daware+speech+generat)

*Updated 1 November 2026*