---
title: "Efficient Training for Cross-lingual Speech Language Models"
description: "The authors present CSLM (Cross-lingual Speech Language Model), a recipe for training a… 46 references, cited by 2 papers here."
authors: ["Yan Zhou", "Qingkai Fang", "Yun Hong", "Yang Feng"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.642/"
doi: "10.18653/v1/2026.findings-acl.642"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhou-2026-efficient-training-cross-lingual-speech/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhou-2026-efficient-training-cross-lingual-speech/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/7/) / Efficient Training for Cross-lingual Speech Language Models

# Efficient Training for Cross-lingual Speech Language Models

**Authors:** [Yan Zhou](https://md.research.elamur.ai/free/en/authors/yan-zhou/), [Qingkai Fang](https://md.research.elamur.ai/free/en/authors/qingkai-fang/), Yun Hong and [Yang Feng](https://md.research.elamur.ai/free/en/authors/yang-feng/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.642/) · [PDF](https://aclanthology.org/2026.findings-acl.642.pdf) · [DOI 10.18653/v1/2026.findings-acl.642](https://doi.org/10.18653/v1/2026.findings-acl.642) · [OpenAlex](https://openalex.org/W7166799440)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [2](https://md.research.elamur.ai/free/en/papers/zhou-2026-efficient-training-cross-lingual-speech/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Building end-to-end speech LLMs is difficult because speech data is scarce, particularly outside a few major languages. Aligning both modalities and languages usually requires very large corpora. The authors present CSLM (Cross-lingual Speech Language Model), a recipe for training a Chinese–English speech LLM on modest data.

CSLM adds discrete speech tokens to the vocabulary of an instruction-tuned text LLM. It then continues pre-training on four kinds of data:
- ASR and TTS pairs, which link speech and text within each language;
- Chinese–English MT data, which connects the two languages through text;
- monolingual instruction data.

Supervised fine-tuning follows a chain-of-modality pattern. The model first transcribes the spoken question, then answers in alternating text and speech segments, so audio can begin before the full text answer is written.

In evaluations on cross-modal tasks and on monolingual and cross-lingual spoken dialogue, CSLM aligns speech and text more closely than SpeechGPT and GLM-4-Voice. It also cuts response latency considerably. Because the method relies on text as the bridge between languages, it suggests a low-cost route to adding languages to speech LLMs, including lower-resource ones.

## Key points
- **Architecture:**
  - The tokenizer is from CosyVoice-300M-25hz: 4096 speech tokens at 25 Hz, built on a Conformer encoder with vector quantization. Consecutive repeated tokens are merged before they enter the LLM.
  - The decoder combines CosyVoice's flow matching model, a HiFi-GAN vocoder and a new duration predictor that restores the merged repeats.
- **Speech–text alignment:** on LibriSpeech test-clean, last-layer similarity between parallel speech and text reaches 72.5% for CSLM, versus 39.4% for GLM-4-Voice and 1.2% for SpeechGPT. Random speech–text pairs score 56.7% for CSLM.
- **Latency:** interleaved generation runs about ×2.93 faster on average than producing the full text answer before any speech (full CoM). The largest gain, roughly ×4.6, appears on En→Zh InstructS2S-Eval. Quality is generally equal to or better than full CoM.
- **Ablations:**
  - Removing MT data hurts Zh→En spoken dialogue: GPT scores drop from 3.53 to 3.08 and from 3.20 to 2.74, and ASR-ER rises from 7.4 to 9.2.
  - Skipping the transcription of the spoken question sharply lowers results, especially in cross-lingual tasks.
  - Coarse chunks of size 4 also degrade quality.
- **Evaluation and code:**
  - Benchmarks are InstructS2S-Eval (English) and BELLE-eval-S2S (Chinese), scored with GPT-based scores, ASR-ER and UTMOS.
  - Human C-MOS and A-MOS ratings follow the same trends as the automatic metrics.
  - Code is available at github.com/ictnlp/CSLM.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 882 works

## Cite

Cite as: Yan Zhou, Qingkai Fang, Yun Hong and Yang Feng. 2026. Efficient Training for Cross-lingual Speech Language Models. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhou2026efficient,
  title = {{Efficient Training for Cross-lingual Speech Language Models}},
  author = {Yan Zhou, Qingkai Fang, Yun Hong and Yang Feng},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.642},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.642}
}
```

- [← previous Efficient Multi-Agent System Training with Data Influence-Oriented Tree Search](https://md.research.elamur.ai/free/en/papers/shi-2026-efficient-multi-agent-system-training/)
- [next → Evaluating Customized vs. Generalist Transformer-based Models for Legal Contract Classification](https://md.research.elamur.ai/free/en/papers/singh-2026-evaluating-customized-vs-generalist-transformer/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/7/)

## References

46 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Keyu An, Qian Chen, Chong Deng, Zhihao Du, Changfeng Gao, Zhifu Gao, Yue Gu, Ting He, Hangrui Hu, Kai Hu, Shengpeng Ji, Yabin Li, Zerui Li, Heng Lu, Haoneng Luo, Xiang Lv, Bin Ma, Ziyang Ma, Chongjia Ni, Changhe Song, Jiaqi Shi, Xian Shi, Hao Wang, Wen Wang, Yuxuan Wang, Zhangyu Xiao, Zhijie Yan, Ye… — [arXiv:2407.04051](https://arxiv.org/abs/2407.04051) [find](https://scholar.google.com/scholar?q=Keyu+An%2C+Qian+Chen%2C+Chong+Deng%2C+Zhihao+Du%2C+Changfeng+Gao%2C+Zhifu+Gao%2C+Yue+Gu%2C+Ting+He%2C+Hangrui+Hu%2C+Kai+Hu%2C+Shengpeng+Ji%2C+Yabin+Li%2C+Zerui+Li%2C+Heng+Lu%2C+Haoneng+Luo%2C+Xiang+Lv%2C+Bin+Ma%2C+Ziyang+Ma%2C+Chongjia+)
2. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. [wav2vec 2.0: A framework for self-supervised learning of speech representations](https://md.research.elamur.ai/free/en/papers/baevski-2020-wav2vec-2-0-framework-self/). In Advances in Neural Information Processing Systems, volume 33, pages 12449–12460. Curran Associates, Inc.
3. BELLEGroup. 2023. Belle: Be everyone’s large language model engine. [https://github.com/](https://github.com/) LianjiaTech/BELLE. [↗](https://github.com/)
4. Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng. 2017. Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline. In 2017 20th Conference of the Oriental Chapter of the International Coordinating Committee on Speech Databases and Speech I/O Systems and Assessment (O-COC… [find](https://scholar.google.com/scholar?q=Hui+Bu%2C+Jiayu+Du%2C+Xingyu+Na%2C+Bengu+Wu%2C+and+Hao+Zheng%2E+2017%2E+Aishell%2D1%3A+An+open%2Dsource+mandarin+speech+corpus+and+a+speech+recognition+baseline%2E+In+2017+20th+Conference+of+the+Oriental+Chapter+of+the+I)
5. Guoguo Chen, Shuzhou Chai, Guan-Bo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, Mingjie Jin, Sanjeev Khudanpur, Shinji Watanabe, Shuaijiang Zhao, Wei Zou, Xiangang Li, Xuchen Yao, Yongqing Wang, Zhao You, and Zhiyong Yan. 2021. Gigaspeech: An evolving, mu… [find](https://scholar.google.com/scholar?q=Guoguo+Chen%2C+Shuzhou+Chai%2C+Guan%2DBo+Wang%2C+Jiayu+Du%2C+Wei%2DQiang+Zhang%2C+Chao+Weng%2C+Dan+Su%2C+Daniel+Povey%2C+Jan+Trmal%2C+Junbo+Zhang%2C+Mingjie+Jin%2C+Sanjeev+Khudanpur%2C+Shinji+Watanabe%2C+Shuaijiang+Zhao%2C+Wei+Zou%2C+)
6. Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. [Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models](https://md.research.elamur.ai/free/en/papers/chu-2023-qwen-audio-advancing-universal/). Preprint, arXiv:2311.07919.
7. Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. 2021. W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 244–250.… [find](https://scholar.google.com/scholar?q=Yu%2DAn+Chung%2C+Yu+Zhang%2C+Wei+Han%2C+Chung%2DCheng+Chiu%2C+James+Qin%2C+Ruoming+Pang%2C+and+Yonghui+Wu%2E+2021%2E+W2v%2Dbert%3A+Combining+contrastive+learning+and+masked+language+modeling+for+self%2Dsupervised+speech+pre%2Dtr)
8. Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. 2022. [High fidelity neural audio compression](https://md.research.elamur.ai/free/en/papers/defossez-2022-high-fidelity-neural-audio-compression/). arXiv preprint arXiv:2210.13438.
9. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. [Enhancing chat language models by scaling high-quality instructional conversations](https://md.research.elamur.ai/free/en/papers/ding-2023-enhancing-chat-language-models-scaling/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3029–3051, Si…
10. Jiayu Du, Xingyu Na, Xuechen Liu, and Hui Bu. 2018. Aishell-2: Transforming mandarin asr research into industrial scale. Preprint, arXiv:1808.10583. [find](https://scholar.google.com/scholar?q=Jiayu+Du%2C+Xingyu+Na%2C+Xuechen+Liu%2C+and+Hui+Bu%2E+2018%2E+Aishell%2D2%3A+Transforming+mandarin+asr+research+into+industrial+scale%2E+Preprint%2C+arXiv%3A1808%2E10583%2E)
11. Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, Zhifu Gao, and Zhijie Yan. 2024. [Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens](https://md.research.elamur.ai/free/en/papers/du-2024-cosyvoice-scalable-multilingual-zero-shot/). Preprint, arXiv:2407.05407.
12. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
13. Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. 2024. [Moshi: a speech-text foundation model for real-time dialogue](https://md.research.elamur.ai/free/en/papers/defossez-2024-moshi-speech-text-foundation-model/). Preprint, arXiv:2410.00037.
14. Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. 2024. [Llama-omni: Seamless speech interaction with large language models](https://md.research.elamur.ai/free/en/papers/fang-2024-llama-omni-seamless-speech-interaction/). Preprint, arXiv:2409.06666.
15. Zhifu Gao, ShiLiang Zhang, Ian McLoughlin, and Zhijie Yan. 2022. Paraformer: Fast and accurate parallel transformer for non-autoregressive end-to-end speech recognition. In Interspeech 2022, pages 2063–2067. [find](https://scholar.google.com/scholar?q=Zhifu+Gao%2C+ShiLiang+Zhang%2C+Ian+McLoughlin%2C+and+Zhijie+Yan%2E+2022%2E+Paraformer%3A+Fast+and+accurate+parallel+transformer+for+non%2Dautoregressive+end%2Dto%2Dend+speech+recognition%2E+In+Interspeech+2022%2C+pages+206)
16. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd International Conference on Machine Learning, ICML ’06, page 369–376, New York, NY, USA… [find](https://scholar.google.com/scholar?q=Alex+Graves%2C+Santiago+Fern%C3%A1ndez%2C+Faustino+Gomez%2C+and+J%C3%BCrgen+Schmidhuber%2E+2006%2E+Connectionist+temporal+classification%3A+labelling+unsegmented+sequence+data+with+recurrent+neural+networks%2E+In+Proceedings)
17. Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. 2020. [Conformer: Convolution-augmented transformer for speech recognition](https://md.research.elamur.ai/free/en/papers/gulati-2020-conformer-convolution-augmented-transformer-speech/). In Interspeech 2020, pages 5036–5040.
18. Michael Hassid, Tal Remez, Tu Anh Nguyen, Itai Gat, Alexis CONNEAU, Felix Kreuk, Jade Copet, Alexandre Defossez, Gabriel Synnaeve, Emmanuel Dupoux, Roy Schwartz, and Yossi Adi. 2023. Textually pre-trained speech language models. In Advances in Neural Information Processing Systems, volume 36, pages… [find](https://scholar.google.com/scholar?q=Michael+Hassid%2C+Tal+Remez%2C+Tu+Anh+Nguyen%2C+Itai+Gat%2C+Alexis+CONNEAU%2C+Felix+Kreuk%2C+Jade+Copet%2C+Alexandre+Defossez%2C+Gabriel+Synnaeve%2C+Emmanuel+Dupoux%2C+Roy+Schwartz%2C+and+Yossi+Adi%2E+2023%2E+Textually+pre%2Dtra)
19. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. [Hubert: Self-supervised speech representation learning by masked prediction of hidden units](https://md.research.elamur.ai/free/en/papers/hsu-2021-hubert-self-supervised-speech-representation/). IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460.
20. Zhichao Huang, Chutong Meng, and Tom Ko. 2023. Repcodec: A speech representation codec for speech tokenization. arXiv preprint arXiv:2309.00169. [find](https://scholar.google.com/scholar?q=Zhichao+Huang%2C+Chutong+Meng%2C+and+Tom+Ko%2E+2023%2E+Repcodec%3A+A+speech+representation+codec+for+speech+tokenization%2E+arXiv+preprint+arXiv%3A2309%2E00169%2E)
21. Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020. [Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis](https://md.research.elamur.ai/free/en/papers/kong-2020-hifi-gan-generative-adversarial-networks/). In Advances in Neural Information Processing Systems, volume 33, pages 17022–17033. Curran Associates, Inc.
22. Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, and Emmanuel Dupoux. 2021. [On generative spoken language modeling from raw audio](https://md.research.elamur.ai/free/en/papers/lakhotia-2021-generative-spoken-language-modeling-raw/). Transactions of the Association for Computational Linguistics, 9…
23. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. [https://github.com/tatsu-lab/alpaca_eval](https://github.com/tatsu-lab/alpaca_eval). [↗](https://github.com/tatsu-lab/alpaca_eval)
24. Shijia Liao, Yuxuan Wang, Tianyu Li, Yifan Cheng, Ruoyi Zhang, Rongzhi Zhou, and Yijin Xing. 2024. Fish-speech: Leveraging large language models for advanced multilingual text-to-speech synthesis. Preprint, arXiv:2411.01156. — [arXiv:2411.01156](https://arxiv.org/abs/2411.01156) [find](https://scholar.google.com/scholar?q=Shijia+Liao%2C+Yuxuan+Wang%2C+Tianyu+Li%2C+Yifan+Cheng%2C+Ruoyi+Zhang%2C+Rongzhi+Zhou%2C+and+Yijin+Xing%2E+2024%2E+Fish%2Dspeech%3A+Leveraging+large+language+models+for+advanced+multilingual+text%2Dto%2Dspeech+synthesis%2E+Pre)
25. Linhan Ma, Dake Guo, Kun Song, Yuepeng Jiang, Shuai Wang, Liumeng Xue, Weiming Xu, Huan Zhao, Binbin Zhang, and Lei Xie. 2024. Wenetspeech4tts: A 12,800-hour mandarin tts corpus for large speech generation model benchmark. Preprint, arXiv:2406.05763. — [arXiv:2406.05763](https://arxiv.org/abs/2406.05763) [find](https://scholar.google.com/scholar?q=Linhan+Ma%2C+Dake+Guo%2C+Kun+Song%2C+Yuepeng+Jiang%2C+Shuai+Wang%2C+Liumeng+Xue%2C+Weiming+Xu%2C+Huan+Zhao%2C+Binbin+Zhang%2C+and+Lei+Xie%2E+2024%2E+Wenetspeech4tts%3A+A+12%2C800%2Dhour+mandarin+tts+corpus+for+large+speech+gener)
26. Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R. Costa-jussa, Maha Elbayad, Sravya Popuri, Christophe Ropers, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, Itai Gat, Mary Williamson, Gabriel Synnaeve, Juan Pino, Benoît Sagot, and Emmanuel Dupoux. 2025. [Spirit-lm: Interleaved spoken and…](https://md.research.elamur.ai/free/en/papers/nguyen-2025-spirit-lm-interleaved-spoken-written/)
27. OpenAI. 2022. Introducing chatgpt. [https://openai.com/index/chatgpt/](https://openai.com/index/chatgpt/). [↗](https://openai.com/index/chatgpt/)
28. OpenAI. 2024. Hello gpt-4o. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2024%2E+Hello+gpt%2D4o%2E)
29. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: An asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206–5210. [find](https://scholar.google.com/scholar?q=Vassil+Panayotov%2C+Guoguo+Chen%2C+Daniel+Povey%2C+and+Sanjeev+Khudanpur%2E+2015%2E+Librispeech%3A+An+asr+corpus+based+on+public+domain+audio+books%2E+In+2015+IEEE+International+Conference+on+Acoustics%2C+Speech+and+)
30. Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. 2020. [Mls: A large-scale multilingual dataset for speech research](https://md.research.elamur.ai/free/en/papers/pratap-2020-mls-large-scale-multilingual-dataset/). In Interspeech 2020, pages 2757–2761.
31. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng… [find](https://scholar.google.com/scholar?q=Qwen%2C+%3A%2C+An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+Huan+Lin%2C+Jian+Yang%2C+Jianhong+Tu%2C+Jianwei+Zhang%2C+Jianxin+Yang%2C+Jiaxi+)
32. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. [Robust speech recognition via large-scale weak supervision](https://md.research.elamur.ai/free/en/papers/radford-2022-robust-speech-recognition-large-scale/). In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org.
33. Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. 2022. [Utmos: Utokyo-sarulab system for voicemos challenge 2022](https://md.research.elamur.ai/free/en/papers/saeki-2022-utmos-utokyo-sarulab-system-voicemos/). In Interspeech 2022, pages 4521–4525.
34. Yao Shi, Hui Bu, Xin Xu, Shaoji Zhang, and Ming Li. 2021. Aishell-3: A multi-speaker mandarin tts corpus. In Interspeech 2021, pages 2756–2760. [find](https://scholar.google.com/scholar?q=Yao+Shi%2C+Hui+Bu%2C+Xin+Xu%2C+Shaoji+Zhang%2C+and+Ming+Li%2E+2021%2E+Aishell%2D3%3A+A+multi%2Dspeaker+mandarin+tts+corpus%2E+In+Interspeech+2021%2C+pages+2756%E2%80%932760%2E)
35. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [find](https://scholar.google.com/scholar?q=Rohan+Taori%2C+Ishaan+Gulrajani%2C+Tianyi+Zhang%2C+Yann+Dubois%2C+Xuechen+Li%2C+Carlos+Guestrin%2C+Percy+Liang%2C+and+Tatsunori+B+Hashimoto%2E+2023%2E+Stanford+alpaca%3A+An+instruction%2Dfollowing+llama+model%2E)
36. Xiong Wang, Yangze Li, Chaoyou Fu, Yunhang Shen, Lei Xie, Ke Li, Xing Sun, and Long Ma. 2024. Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen llm. Preprint, arXiv:2411.00774. [find](https://scholar.google.com/scholar?q=Xiong+Wang%2C+Yangze+Li%2C+Chaoyou+Fu%2C+Yunhang+Shen%2C+Lei+Xie%2C+Ke+Li%2C+Xing+Sun%2C+and+Long+Ma%2E+2024%2E+Freeze%2Domni%3A+A+smart+and+low+latency+speech%2Dto%2Dspeech+dialogue+model+with+frozen+llm%2E+Preprint%2C+arXiv%3A2411)
37. Zhifei Xie and Changqiao Wu. 2024. [Mini-omni: Language models can hear, talk while thinking in streaming](https://md.research.elamur.ai/free/en/papers/xie-2024-mini-omni-language-models-hear/). Preprint, arXiv:2408.16725.
38. Junichi Yamagishi, Christophe Veaux, and Kirsten Mac-Donald. 2019. Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92). [find](https://scholar.google.com/scholar?q=Junichi+Yamagishi%2C+Christophe+Veaux%2C+and+Kirsten+Mac%2DDonald%2E+2019%2E+Cstr+vctk+corpus%3A+English+multi%2Dspeaker+corpus+for+cstr+voice+cloning+toolkit+%28version+0%2E92%29%2E)
39. Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. 2022. [Soundstream: An end-to-end neural audio codec](https://md.research.elamur.ai/free/en/papers/zeghidour-2021-soundstream-end-end-neural-audio/). IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495–507.
40. Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J. Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. 2019. [Libritts: A corpus derived from librispeech for text-to-speech](https://md.research.elamur.ai/free/en/papers/zen-2019-libritts-corpus-derived-librispeech-text/). In Interspeech 2019, pages 1526–1530.
41. Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. 2024. [Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot](https://md.research.elamur.ai/free/en/papers/zeng-2024-glm-4-voice-intelligent/). Preprint, arXiv:2412.02612.
42. Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, Yugang Jiang, and Xipeng Qiu. 2024. [Anygpt: Unified multimodal llm with discrete sequence modeling](https://md.research.elamur.ai/free/en/papers/zhan-2024-anygpt-unified-multimodal-llm-discrete/). Preprint, arXiv:2402.12226.
43. Binbin Zhang, Hang Lv, Pengcheng Guo, Qijie Shao, Chao Yang, Lei Xie, Xin Xu, Hui Bu, Xiaoyu Chen, Chenchen Zeng, Di Wu, and Zhendong Peng. 2022. Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech… [find](https://scholar.google.com/scholar?q=Binbin+Zhang%2C+Hang+Lv%2C+Pengcheng+Guo%2C+Qijie+Shao%2C+Chao+Yang%2C+Lei+Xie%2C+Xin+Xu%2C+Hui+Bu%2C+Xiaoyu+Chen%2C+Chenchen+Zeng%2C+Di+Wu%2C+and+Zhendong+Peng%2E+2022%2E+Wenetspeech%3A+A+10000%2B+hours+multi%2Ddomain+mandarin+corp)
44. Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. 2023. [SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abilities](https://md.research.elamur.ai/free/en/papers/zhang-2023-speechgpt-empowering-large-language-models/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15757–15773, Singapore. Associa…
45. Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. 2024. [Speechtokenizer: Unified speech tokenizer for speech large language models](https://md.research.elamur.ai/free/en/papers/zhang-2023-speechtokenizer-unified-speech-tokenizer-speech/). Preprint, arXiv:2308.16692.
46. Wenhao Zhu, Yunzhe Lv, Qingxiu Dong, Fei Yuan, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2023. [Extrapolating large language models to non-english by aligning languages](https://md.research.elamur.ai/free/en/papers/zhu-2023-extrapolating-large-language-models-non/). arXiv preprint arXiv:2308.04948.

*Updated 2 November 2026*