---
title: "AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling"
description: "AnyGPT addresses the challenge of building one language model that can both understand and… 55 references, cited by 44 papers here."
authors: ["Jun Zhan", "Junqi Dai", "Jiasheng Ye", "Yunhua Zhou", "Dong Zhang", "Zhigeng Liu", "Xin Zhang", "Ruibin Yuan", "Ge Zhang", "Linyang Li", "Hang Yan", "Jie Fu", "Tao Gui", "Tianxiang Sun", "Yu-Gang Jiang", "Xipeng Qiu"]
year: 2024
venue: "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.acl-long.521/"
doi: "10.18653/v1/2024.acl-long.521"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhan-2024-anygpt-unified-multimodal-llm-discrete/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhan-2024-anygpt-unified-multimodal-llm-discrete/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling

# AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling

**Authors:** [Jun Zhan](https://md.research.elamur.ai/free/en/authors/jun-zhan/), [Junqi Dai](https://md.research.elamur.ai/free/en/authors/junqi-dai/), Jiasheng Ye, Yunhua Zhou, [Dong Zhang](https://md.research.elamur.ai/free/en/authors/dong-zhang/), Zhigeng Liu, [Xin Zhang](https://md.research.elamur.ai/free/en/authors/xin-zhang/), Ruibin Yuan, [Ge Zhang](https://md.research.elamur.ai/free/en/authors/ge-zhang/), [Linyang Li](https://md.research.elamur.ai/free/en/authors/linyang-li/), [Hang Yan](https://md.research.elamur.ai/free/en/authors/hang-yan/), [Jie Fu](https://md.research.elamur.ai/free/en/authors/jie-fu/), [Tao Gui](https://md.research.elamur.ai/free/en/authors/tao-gui/), [Tianxiang Sun](https://md.research.elamur.ai/free/en/authors/tianxiang-sun/), Yu-Gang Jiang and [Xipeng Qiu](https://md.research.elamur.ai/free/en/authors/xipeng-qiu/) · **Year:** 2024 · **Venue:** [Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2024](https://md.research.elamur.ai/free/en/venues/acl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.acl-long.521/) · [PDF](https://aclanthology.org/2024.acl-long.521.pdf) · [DOI 10.18653/v1/2024.acl-long.521](https://doi.org/10.18653/v1/2024.acl-long.521) · [OpenAlex](https://openalex.org/W4402672034)

**Citations:** 50 citations (OpenAlex, 2026-10-14) · [44](https://md.research.elamur.ai/free/en/papers/zhan-2024-anygpt-unified-multimodal-llm-discrete/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
AnyGPT addresses the challenge of building one language model that can both understand and generate several non-text modalities (speech, images, music) in any combination. Earlier systems either covered a single extra modality or attached separately trained encoders and decoders, which makes inputs and outputs inconsistent. AnyGPT converts every modality into discrete tokens using dedicated tokenizers:

- SEED for images,
- SpeechTokenizer for speech,
- Encodec for music.

An LLM then models these mixed token sequences with ordinary next-token prediction. De-tokenizers turn the generated tokens back into images, audio or music. The LLM architecture and training objective stay unchanged, so adding a modality works much like adding a new language.

Pre-training aligns each modality with text, using text as the bridge between modalities. The authors also synthesized AnyInstruct-108k, a dataset of multi-turn conversations with interleaved modalities, using generative models. Zero-shot results on ASR, TTS, and music captioning and generation are competitive with specialized models. This suggests that discrete representations are a practical way to unify many modalities in a single LLM.

## Key points
- **Image tokenizer:** SEED turns a 224×224 image into 32 tokens drawn from an 8192-entry codebook. Images are reconstructed through an unCLIP Stable Diffusion-aligned decoder.
- **Speech tokenizer:** SpeechTokenizer uses 8 RVQ layers, each with 1024 entries, at 50 Hz. The LLM models only the first (semantic) layer. A voice-cloning stage restores acoustic detail and can mimic a 3-second voice prompt.
- **Music tokenizer:** Encodec (32 kHz, 50 Hz) handles music, but music output is restricted to 5-second clips.
- **Instruction data:** AnyInstruct-108k contains 108k synthetic any-to-any dialogues.
- **Evaluation:** ASR is tested on LibriSpeech test-clean against Wav2vec 2.0 and Whisper Large V2. TTS is compared with VALL-E and USLM. Music is evaluated on MusicCaps using CLAP score.
- **Limitations noted by the authors:** multimodal training has higher loss than unimodal training, tokenizer quality caps performance, and context length is limited.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Fudan University](https://md.research.elamur.ai/free/en/universities/fudan-university/) — University, Shanghai, China · 65 works · 19,402 citations
- [Shanghai Artificial Intelligence Laboratory](https://md.research.elamur.ai/free/en/labs/shanghai-artificial-intelligence-laboratory/) — Research institute, Shanghai, China · 44 works · 10,162 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 269 works

## Cite

Cite as: Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, Yu-Gang Jiang and Xipeng Qiu. 2024. AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhan2024anygpt,
  title = {{AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling}},
  author = {Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, Yu-Gang Jiang and Xipeng Qiu},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.521},
  url = {https://doi.org/10.18653/v1/2024.acl-long.521}
}
```

- [← previous WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild](https://md.research.elamur.ai/free/en/papers/lin-2024-wildbench-benchmarking-llms-challenging-tasks/)
- [next → Boosting Language Models Reasoning with Chain-of-Knowledge Prompting](https://md.research.elamur.ai/free/en/papers/wang-2024-boosting-language-models-reasoning-chain/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

55 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. 2022a. Laion-aesthetics. [https://laion.ai/blog/](https://laion.ai/blog/) laion-aesthetics/. 2022b. Laion coco: from laion2b-en. laion-coco/. 600m synthetic captions https://laion.ai/blog/ [↗](https://laion.ai/blog/)
2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). ArXiv preprint, abs/2303.08774.
3. Andrea Agostinelli, Timo I. Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matthew Sharifi, Neil Zeghidour, and Christian Havnø Frank. 2023. Musiclm: Generating music from text. ArXiv preprint, abs/2301.11325. — [arXiv:2301.11325](https://arxiv.org/abs/2301.11325) [find](https://scholar.google.com/scholar?q=Andrea+Agostinelli%2C+Timo+I%2E+Denk%2C+Zal%C3%A1n+Borsos%2C+Jesse+Engel%2C+Mauro+Verzetti%2C+Antoine+Caillon%2C+Qingqing+Huang%2C+Aren+Jansen%2C+Adam+Roberts%2C+Marco+Tagliasacchi%2C+Matthew+Sharifi%2C+Neil+Zeghidour%2C+and+Christ)
4. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock,… — [Flamingo: a Visual Language Model for Few-Shot Learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/)
5. Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. 2016. [Deep speech 2: End-to-end speech recognition in english and mandarin](https://md.research.elamur.ai/free/en/papers/amodei-2015-deep-speech-2-end-end/). In International conference on machine learning, pages 17…
6. Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. 2020. [Common voice: A massively-multilingual speech corpus](https://md.research.elamur.ai/free/en/papers/ardila-2019-common-voice-massively-multilingual-speech/). In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 421…
7. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. [wav2vec 2.0: A framework for self-supervised learning of speech representations](https://md.research.elamur.ai/free/en/papers/baevski-2020-wav2vec-2-0-framework-self/). In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6…
8. James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. 2023. Improving image generation with better captions. Computer Science. [https://cdn.openai](https://cdn.openai). com/papers/dall-e-3. pdf, 2(3):8. [↗](https://cdn.openai)
9. Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. 2023a. [Audiolm: a language modeling approach to audio generation](https://md.research.elamur.ai/free/en/papers/borsos-2022-audiolm-language-modeling-approach-audio/). IEEE/ACM Transactions on Audio, Speech, and Language Process…
10. Zalán Borsos, Matt Sharifi, Damien Vincent, Eugene Kharitonov, Neil Zeghidour, and Marco Tagliasacchi. 2023b. Soundstorm: Efficient parallel audio generation. ArXiv preprint, abs/2305.09636. — [arXiv:2305.09636](https://arxiv.org/abs/2305.09636) [find](https://scholar.google.com/scholar?q=Zal%C3%A1n+Borsos%2C+Matt+Sharifi%2C+Damien+Vincent%2C+Eugene+Kharitonov%2C+Neil+Zeghidour%2C+and+Marco+Tagliasacchi%2E+2023b%2E+Soundstorm%3A+Efficient+parallel+audio+generation%2E+ArXiv+preprint%2C+abs%2F2305%2E09636%2E+%E2%80%94+%5BarXiv%3A)
11. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
12. Guoguo Chen, Shuzhou Chai, Guan-Bo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, Mingjie Jin, Sanjeev Khudanpur, Shinji Watanabe, Shuaijiang Zhao, Wei Zou, Xiangang Li, Xuchen Yao, Yongqing Wang, Zhao You, and Zhiyong Yan. 2021. Gigaspeech: An evolving, mu… [find](https://scholar.google.com/scholar?q=Guoguo+Chen%2C+Shuzhou+Chai%2C+Guan%2DBo+Wang%2C+Jiayu+Du%2C+Wei%2DQiang+Zhang%2C+Chao+Weng%2C+Dan+Su%2C+Daniel+Povey%2C+Jan+Trmal%2C+Junbo+Zhang%2C+Mingjie+Jin%2C+Sanjeev+Khudanpur%2C+Shinji+Watanabe%2C+Shuaijiang+Zhao%2C+Wei+Zou%2C+)
13. Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. 2023. Simple and controllable music generation. ArXiv preprint, abs/2306.05284. — [arXiv:2306.05284](https://arxiv.org/abs/2306.05284) [find](https://scholar.google.com/scholar?q=Jade+Copet%2C+Felix+Kreuk%2C+Itai+Gat%2C+Tal+Remez%2C+David+Kant%2C+Gabriel+Synnaeve%2C+Yossi+Adi%2C+and+Alexandre+D%C3%A9fossez%2E+2023%2E+Simple+and+controllable+music+generation%2E+ArXiv+preprint%2C+abs%2F2306%2E05284%2E+%E2%80%94+%5BarXiv%3A)
14. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Albert Li, Pascale Fung, and Steven C. H. Hoi. 2023. [Instructblip: Towards general-purpose vision-language models with instruction tuning](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/). ArXiv preprint, abs/2305.06500.
15. Alexandre D’efossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. 2022. [High fidelity neural audio compression](https://md.research.elamur.ai/free/en/papers/defossez-2022-high-fidelity-neural-audio-compression/). ArXiv preprint, abs/2210.13438.
16. Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jian-Yuan Sun, Hongyu Zhou, Hao-Ran Wei, Xiangwen Kong, Xiangyu Zhang, Kaisheng Ma, and Li Yi. 2023. Dreamllm: Synergistic multimodal comprehension and creation. ArXiv preprint, abs/2309.11499. — [arXiv:2309.11499](https://arxiv.org/abs/2309.11499) [find](https://scholar.google.com/scholar?q=Runpei+Dong%2C+Chunrui+Han%2C+Yuang+Peng%2C+Zekun+Qi%2C+Zheng+Ge%2C+Jinrong+Yang%2C+Liang+Zhao%2C+Jian%2DYuan+Sun%2C+Hongyu+Zhou%2C+Hao%2DRan+Wei%2C+Xiangwen+Kong%2C+Xiangyu+Zhang%2C+Kaisheng+Ma%2C+and+Li+Yi%2E+2023%2E+Dreamllm%3A+Syner)
17. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. [An image is worth 16x16 words: Transformers for image recognition at scale](https://md.research.elamur.ai/free/en/papers/dosovitskiy-2020-image-worth-16x16-words-transformers/). In 9th In…
18. Seth* Forsgren and Hayk* Martiros. 2022. Riffusion -Stable diffusion for real-time music generation. [find](https://scholar.google.com/scholar?q=Seth%2A+Forsgren+and+Hayk%2A+Martiros%2E+2022%2E+Riffusion+%2DStable+diffusion+for+real%2Dtime+music+generation%2E)
19. Josh Gardner, Simon Durand, Daniel Stoller, and Rachel M. Bittner. 2023. Llark: A multimodal instruction-following language model for music. [find](https://scholar.google.com/scholar?q=Josh+Gardner%2C+Simon+Durand%2C+Daniel+Stoller%2C+and+Rachel+M%2E+Bittner%2E+2023%2E+Llark%3A+A+multimodal+instruction%2Dfollowing+language+model+for+music%2E)
20. Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan. 2023a. Planting a seed of vision in large language model. ArXiv preprint, abs/2307.08041. [find](https://scholar.google.com/scholar?q=Yuying+Ge%2C+Yixiao+Ge%2C+Ziyun+Zeng%2C+Xintao+Wang%2C+and+Ying+Shan%2E+2023a%2E+Planting+a+seed+of+vision+in+large+language+model%2E+ArXiv+preprint%2C+abs%2F2307%2E08041%2E)
21. Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. 2023b. Making llama see and draw with seed tokenizer. ArXiv preprint, abs/2310.01218. — [arXiv:2310.01218](https://arxiv.org/abs/2310.01218) [find](https://scholar.google.com/scholar?q=Yuying+Ge%2C+Sijie+Zhao%2C+Ziyun+Zeng%2C+Yixiao+Ge%2C+Chen+Li%2C+Xintao+Wang%2C+and+Ying+Shan%2E+2023b%2E+Making+llama+see+and+draw+with+seed+tokenizer%2E+ArXiv+preprint%2C+abs%2F2310%2E01218%2E+%E2%80%94+%5BarXiv%3A2310%2E01218%5D%28https%3A%2F%2Far)
22. Rongjie Huang, Jia-Bin Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiaoyue Yin, and Zhou Zhao. 2023. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. ArXiv preprint, abs/2301.12661. — [arXiv:2301.12661](https://arxiv.org/abs/2301.12661) [find](https://scholar.google.com/scholar?q=Rongjie+Huang%2C+Jia%2DBin+Huang%2C+Dongchao+Yang%2C+Yi+Ren%2C+Luping+Liu%2C+Mingze+Li%2C+Zhenhui+Ye%2C+Jinglin+Liu%2C+Xiaoyue+Yin%2C+and+Zhou+Zhao%2E+2023%2E+Make%2Dan%2Daudio%3A+Text%2Dto%2Daudio+generation+with+prompt%2Denhanced+diff)
23. Jing Yu Koh, Daniel Fried, and Ruslan Salakhutdinov. 2023. Generating images with multimodal language models. ArXiv preprint, abs/2305.17216. — [arXiv:2305.17216](https://arxiv.org/abs/2305.17216) [find](https://scholar.google.com/scholar?q=Jing+Yu+Koh%2C+Daniel+Fried%2C+and+Ruslan+Salakhutdinov%2E+2023%2E+Generating+images+with+multimodal+language+models%2E+ArXiv+preprint%2C+abs%2F2305%2E17216%2E+%E2%80%94+%5BarXiv%3A2305%2E17216%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2305%2E17216%29)
24. Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. 2023. [Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/). In International Conference on Machine Learning.
25. Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. [Microsoft coco: Common objects in context](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/). In European Conference on Computer Vision.
26. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). Advances in neural information processing systems, 36.
27. Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. 2023. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action. ArXiv preprint, abs/2312.17172. — [arXiv:2312.17172](https://arxiv.org/abs/2312.17172) [find](https://scholar.google.com/scholar?q=Jiasen+Lu%2C+Christopher+Clark%2C+Sangho+Lee%2C+Zichen+Zhang%2C+Savya+Khosla%2C+Ryan+Marten%2C+Derek+Hoiem%2C+and+Aniruddha+Kembhavi%2E+2023%2E+Unified%2Dio+2%3A+Scaling+autoregressive+multimodal+models+with+vision%2C+langua)
28. Microsoft. Microsoft azure text-to-speech api. [https://azure.microsoft.com/en-us/](https://azure.microsoft.com/en-us/) products/ai-services/ai-speech. [↗](https://azure.microsoft.com/en-us/)
29. Junting Pan, Keqiang Sun, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, et al. 2023. Journeydb: A benchmark for generative image understanding. ArXiv preprint, abs/2307.00716. — [arXiv:2307.00716](https://arxiv.org/abs/2307.00716) [find](https://scholar.google.com/scholar?q=Junting+Pan%2C+Keqiang+Sun%2C+Yuying+Ge%2C+Hao+Li%2C+Haodong+Duan%2C+Xiaoshi+Wu%2C+Renrui+Zhang%2C+Aojun+Zhou%2C+Zipeng+Qin%2C+Yi+Wang%2C+et+al%2E+2023%2E+Journeydb%3A+A+benchmark+for+generative+image+understanding%2E+ArXiv+prep)
30. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: An ASR corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2015, South Brisbane, Queensland, Australia, April 19-24, 2015, pages 5206… [find](https://scholar.google.com/scholar?q=Vassil+Panayotov%2C+Guoguo+Chen%2C+Daniel+Povey%2C+and+Sanjeev+Khudanpur%2E+2015%2E+Librispeech%3A+An+ASR+corpus+based+on+public+domain+audio+books%2E+In+2015+IEEE+International+Conference+on+Acoustics%2C+Speech+and+)
31. Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. 2020. [MLS: A large-scale multilingual dataset for speech research](https://md.research.elamur.ai/free/en/papers/pratap-2020-mls-large-scale-multilingual-dataset/). In Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020,…
32. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). In Proceedings of the 38th Internatio…
33. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. [Robust speech recognition via large-scale weak supervision](https://md.research.elamur.ai/free/en/papers/radford-2022-robust-speech-recognition-large-scale/). ArXiv preprint, abs/2212.04356. 2023a. Generative multimodal models are in-context learners. ArXiv preprint, abs/2312.13286.
34. Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. 2023b. Generative pretraining in multimodality. ArXiv preprint, abs/2307.05222. — [arXiv:2307.05222](https://arxiv.org/abs/2307.05222) [find](https://scholar.google.com/scholar?q=Quan+Sun%2C+Qiying+Yu%2C+Yufeng+Cui%2C+Fan+Zhang%2C+Xiaosong+Zhang%2C+Yueze+Wang%2C+Hongcheng+Gao%2C+Jingjing+Liu%2C+Tiejun+Huang%2C+and+Xinlong+Wang%2E+2023b%2E+Generative+pretraining+in+multimodality%2E+ArXiv+preprint%2C+abs)
35. Tianxiang Sun, Xiaotian Zhang, Zhengfu He, Peng Li, Qinyuan Cheng, Xiangyang Liu, Hang Yan, Yunfan Shao, Qiong Tang, Shiduo Zhang, Xingjian Zhao, Ke Chen, Yining Zheng, Zhejian Zhou, Ruixiao Li, Jun Zhan, Yunhua Zhou, Linyang Li, Xiaogui Yang, Lingling Wu, Zhangyue Yin, Xuanjing Huang, Yu-Gang Jiang… [find](https://scholar.google.com/scholar?q=Tianxiang+Sun%2C+Xiaotian+Zhang%2C+Zhengfu+He%2C+Peng+Li%2C+Qinyuan+Cheng%2C+Xiangyang+Liu%2C+Hang+Yan%2C+Yunfan+Shao%2C+Qiong+Tang%2C+Shiduo+Zhang%2C+Xingjian+Zhao%2C+Ke+Chen%2C+Yining+Zheng%2C+Zhejian+Zhou%2C+Ruixiao+Li%2C+Jun+Z)
36. Zineng Tang, Ziyi Yang, Mahmoud Khademi, Yang Liu, Chenguang Zhu, and Mohit Bansal. 2023a. Codi-2: In-context, interleaved, and interactive any-to-any generation. ArXiv preprint, abs/2311.18775. — [arXiv:2311.18775](https://arxiv.org/abs/2311.18775) [find](https://scholar.google.com/scholar?q=Zineng+Tang%2C+Ziyi+Yang%2C+Mahmoud+Khademi%2C+Yang+Liu%2C+Chenguang+Zhu%2C+and+Mohit+Bansal%2E+2023a%2E+Codi%2D2%3A+In%2Dcontext%2C+interleaved%2C+and+interactive+any%2Dto%2Dany+generation%2E+ArXiv+preprint%2C+abs%2F2311%2E18775%2E+%E2%80%94+%5Bar)
37. Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. 2023b. Any-to-any generation via composable diffusion. ArXiv preprint, abs/2305.11846. — [arXiv:2305.11846](https://arxiv.org/abs/2305.11846) [find](https://scholar.google.com/scholar?q=Zineng+Tang%2C+Ziyi+Yang%2C+Chenguang+Zhu%2C+Michael+Zeng%2C+and+Mohit+Bansal%2E+2023b%2E+Any%2Dto%2Dany+generation+via+composable+diffusion%2E+ArXiv+preprint%2C+abs%2F2305%2E11846%2E+%E2%80%94+%5BarXiv%3A2305%2E11846%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs)
38. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). J. Mach. Learn. Res., 21:140:1–140:67.
39. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. ArXiv preprint, abs/2307.09288. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+2023%2E+Llama+2%3A+Open+foundation+and+fi)
40. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695. [find](https://scholar.google.com/scholar?q=Robin+Rombach%2C+Andreas+Blattmann%2C+Dominik+Lorenz%2C+Patrick+Esser%2C+and+Bj%C3%B6rn+Ommer%2E+2022%2E+High%2Dresolution+image+synthesis+with+latent+diffusion+models%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+compu)
41. Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. 2017. Neural discrete representation learning. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 6306–6315. [find](https://scholar.google.com/scholar?q=A%C3%A4ron+van+den+Oord%2C+Oriol+Vinyals%2C+and+Koray+Kavukcuoglu%2E+2017%2E+Neural+discrete+representation+learning%2E+In+Advances+in+Neural+Information+Processing+Systems+30%3A+Annual+Conference+on+Neural+Informatio)
42. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. [U-net: Convolutional networks for biomedical image segmentation](https://md.research.elamur.ai/free/en/papers/ronneberger-2015-u-net-convolutional-networks-biomedical/). In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234…
43. Chengyi Wang, Sanyuan Chen, Yu Wu, Zi-Hua Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. 2023. [Neural codec language models are zero-shot text to speech synthesizers](https://md.research.elamur.ai/free/en/papers/wang-2023-neural-codec-language-models-zero/). ArXiv preprint, abs/2301.02111.
44. Flavio Schneider, Zhijing Jin, and Bernhard Schölkopf. 2023. Moûsai: Text-to-music generation with long-context latent diffusion. ArXiv preprint, abs/2301.11757. [find](https://scholar.google.com/scholar?q=Flavio+Schneider%2C+Zhijing+Jin%2C+and+Bernhard+Sch%C3%B6lkopf%2E+2023%2E+Mo%C3%BBsai%3A+Text%2Dto%2Dmusic+generation+with+long%2Dcontext+latent+diffusion%2E+ArXiv+preprint%2C+abs%2F2301%2E11757%2E)
45. Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. 2022. [Laion-5b: An open large-scale dataset for training next generation image-text models](https://md.research.elamur.ai/free/en/papers/schuhmann-2022-laion-5b-open-large-scale/). Advances in Neural Information Processing S…
46. Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. [find](https://scholar.google.com/scholar?q=Quan+Sun%2C+Yufeng+Cui%2C+Xiaosong+Zhang%2C+Fan+Zhang%2C+Qiying+Yu%2C+Zhengxiong+Luo%2C+Yueze+Wang%2C+Yongming+Rao%2C+Jingjing+Liu%2C+Tiejun+Huang%2C+and+Xinlong+Wang%2E)
47. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language model with self generated instructions. ArXiv preprint, abs/2212.10560. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+2022%2E+Self%2Dinstruct%3A+Aligning+language+model+with+self+generated+instructions%2E+ArXiv+pre)
48. Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. 2023. Next-gpt: Any-to-any multimodal llm. ArXiv preprint, abs/2309.05519. [find](https://scholar.google.com/scholar?q=Shengqiong+Wu%2C+Hao+Fei%2C+Leigang+Qu%2C+Wei+Ji%2C+and+Tat%2DSeng+Chua%2E+2023%2E+Next%2Dgpt%3A+Any%2Dto%2Dany+multimodal+llm%2E+ArXiv+preprint%2C+abs%2F2309%2E05519%2E)
49. Yusong Wu, K. Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. 2022. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. [find](https://scholar.google.com/scholar?q=Yusong+Wu%2C+K%2E+Chen%2C+Tianyu+Zhang%2C+Yuchen+Hui%2C+Taylor+Berg%2DKirkpatrick%2C+and+Shlomo+Dubnov%2E+2022%2E+Large%2Dscale+contrastive+language%2Daudio+pretraining+with+feature+fusion+and+keyword%2Dto%2Dcaption+augmentati)
50. ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. [find](https://scholar.google.com/scholar?q=ICASSP+2023+%2D+2023+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+%28ICASSP%29%2C+pages+1%E2%80%935%2E)
51. Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. 2021. [Soundstream: An end-to-end neural audio codec](https://md.research.elamur.ai/free/en/papers/zeghidour-2021-soundstream-end-end-neural-audio/). IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495–507.
52. Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. 2023a. [Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities](https://md.research.elamur.ai/free/en/papers/zhang-2023-speechgpt-empowering-large-language-models/). In Conference on Empirical Methods in Natural Language Processing.
53. Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. 2023b. [Speechtokenizer: Unified speech tokenizer for speech large language models](https://md.research.elamur.ai/free/en/papers/zhang-2023-speechtokenizer-unified-speech-tokenizer-speech/). ArXiv preprint, abs/2308.16692.
54. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023a. [Minigpt-4: Enhancing vision-language understanding with advanced large language models](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/). arXiv preprint arXiv:2304.10592.
55. Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. 2023b. Multimodal c4: An open, billion-scale corpus of images interleaved with text. ArXiv preprint, abs/2304.06939. — [arXiv:2304.06939](https://arxiv.org/abs/2304.06939) [find](https://scholar.google.com/scholar?q=Wanrong+Zhu%2C+Jack+Hessel%2C+Anas+Awadalla%2C+Samir+Yitzhak+Gadre%2C+Jesse+Dodge%2C+Alex+Fang%2C+Youngjae+Yu%2C+Ludwig+Schmidt%2C+William+Yang+Wang%2C+and+Yejin+Choi%2E+2023b%2E+Multimodal+c4%3A+An+open%2C+billion%2Dscale+corpu)

*Updated 24 October 2026*