---
title: "Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling — Chen 2025"
description: "Unified models that handle both image understanding and image generation often route both tasks through one visual… 57 references, cited by 85 papers here."
authors: ["Xiaokang Chen", "Zhiyu Wu", "Xingchao Liu", "Zizheng Pan", "Wen Liu", "Zhenda Xie", "Xingkai Yu", "Chong Ruan"]
year: 2025
venue: "arXiv:2501.17811"
source: "https://arxiv.org/abs/2501.17811"
doi: "10.48550/arxiv.2501.17811"
arxiv: "2501.17811"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chen-2025-janus-pro-unified-multimodal/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chen-2025-janus-pro-unified-multimodal/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/) / Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling

# Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling

**Authors:** [Xiaokang Chen](https://md.research.elamur.ai/free/en/authors/xiaokang-chen/), [Zhiyu Wu](https://md.research.elamur.ai/free/en/authors/zhiyu-wu/), [Xingchao Liu](https://md.research.elamur.ai/free/en/authors/xingchao-liu/), [Zizheng Pan](https://md.research.elamur.ai/free/en/authors/zizheng-pan/), [Wen Liu](https://md.research.elamur.ai/free/en/authors/wen-liu/), [Zhenda Xie](https://md.research.elamur.ai/free/en/authors/zhenda-xie/), Xingkai Yu and [Chong Ruan](https://md.research.elamur.ai/free/en/authors/chong-ruan/) · **Year:** 2025 · **Venue:** arXiv:2501.17811 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2501.17811) · [PDF](https://arxiv.org/pdf/2501.17811) · [DOI 10.48550/arxiv.2501.17811](https://doi.org/10.48550/arxiv.2501.17811) · [arXiv 2501.17811](https://arxiv.org/abs/2501.17811) · [OpenAlex](https://openalex.org/W4406975803)

**Citations:** 12 citations (OpenAlex, 2026-10-14) · [85](https://md.research.elamur.ai/free/en/papers/chen-2025-janus-pro-unified-multimodal/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
Unified models that handle both image understanding and image generation often route both tasks through one visual encoder. Because the two tasks need different representations, understanding quality tends to suffer. Janus addressed this by giving each task its own visual encoder, but it was only tested at about 1B parameters. It also produced weak and unstable images, especially from short prompts.

Janus-Pro, from DeepSeek-AI, keeps the same decoupled architecture and improves three things:
- **Training schedule:** a longer Stage I on ImageNet and a Stage II that drops ImageNet in favor of dense text-to-image data.
- **Training data:** more understanding data plus a large share of synthetic aesthetic images.
- **Model size:** scaling from 1B up to 7B parameters.

The 7B model surpasses earlier unified models on MMBench and beats DALL-E 3 and SD3-Medium on GenEval and DPG-Bench. This shows that the decoupled-encoder approach scales well. Code and weights are openly released. The main remaining weakness is the 384×384 resolution, which limits OCR and fine visual detail.

## Key points
- **Architecture:** SigLIP-Large-Patch16-384 handles understanding, and a VQ tokenizer (codebook of 16,384 entries, 16× downsampling) handles generation. Both feed one autoregressive DeepSeek-LLM backbone (1.5B or 7B) through two-layer MLP adaptors.
- **Data:** about 90M additional understanding samples, drawn from the DeepSeek-VL2 pipeline (e.g., YFCC, Docmatix). About 72M synthetic aesthetic samples bring the real-to-synthetic ratio to 1:1. The fine-tuning mix of understanding, text-only and generation data changes from 7:3:10 to 5:1:4.
- **Understanding:** Janus-Pro-7B scores 79.2 on MMBench, ahead of Janus (69.4), TokenFlow (68.9) and MetaMorph (75.2).
- **Generation:** Janus-Pro-7B scores 0.80 on GenEval, versus Janus 0.61, DALL-E 3 0.67 and SD3-Medium 0.74. It reaches 84.19 on DPG-Bench.
- **Compute:** training took about 9 days for the 1.5B model on 16 nodes and about 14 days for the 7B model on 32 nodes. Each node had 8× A100 (40GB) GPUs, and training used the HAI-LLM framework.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/)

## Cite

Cite as: Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu and Chong Ruan. 2025. Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling. arXiv:2501.17811.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{chen2025janus,
  title = {{Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling}},
  author = {Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu and Chong Ruan},
  year = {2025},
  doi = {10.48550/arxiv.2501.17811},
  eprint = {2501.17811},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2501.17811}
}
```

- [← previous Qwen2.5-1M Technical Report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen2-5-1m-technical/)
- [next → Findings of the WMT25 General Machine Translation Shared Task: Time to Stop Evaluating on Easy Test Sets](https://md.research.elamur.ai/free/en/papers/kocmi-2025-findings-wmt25-general-machine-translation/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/)

## References

57 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. [our card](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/)
2. J. Betker, G. Goh, L. Jing, T. Brooks, J. Wang, L. Li, L. Ouyang, J. Zhuang, J. Lee, Y. Guo, et al. Improving image generation with better captions. Computer Science. [https://cdn.openai](https://cdn.openai). com/papers/dall-e-3. pdf, 2(3):8, 2023. [↗](https://cdn.openai)
3. X. Bi, D. Chen, G. Chen, S. Chen, D. Dai, C. Deng, H. Ding, K. Dong, Q. Du, Z. Fu, et al. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024. [find](https://scholar.google.com/scholar?q=X%2E+Bi%2C+D%2E+Chen%2C+G%2E+Chen%2C+S%2E+Chen%2C+D%2E+Dai%2C+C%2E+Deng%2C+H%2E+Ding%2C+K%2E+Dong%2C+Q%2E+Du%2C+Z%2E+Fu%2C+et+al%2E+Deepseek+llm%3A+Scaling+open%2Dsource+language+models+with+longtermism%2E+arXiv+preprint+arXiv%3A2401%2E02954%2C+2024%2E)
4. J. Chen, J. Yu, C. Ge, L. Yao, E. Xie, Y. Wu, Z. Wang, J. Kwok, P. Luo, H. Lu, et al. Pixart-𝑎𝑙 𝑝ℎ𝑎 : Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023. [find](https://scholar.google.com/scholar?q=J%2E+Chen%2C+J%2E+Yu%2C+C%2E+Ge%2C+L%2E+Yao%2C+E%2E+Xie%2C+Y%2E+Wu%2C+Z%2E+Wang%2C+J%2E+Kwok%2C+P%2E+Luo%2C+H%2E+Lu%2C+et+al%2E+Pixart%2D%F0%9D%91%8E%F0%9D%91%99+%F0%9D%91%9D%E2%84%8E%F0%9D%91%8E+%3A+Fast+training+of+diffusion+transformer+for+photorealistic+text%2Dto%2Dimage+synthesis%2E+arXiv+preprint+)
5. J. Chen, C. Ge, E. Xie, Y. Wu, L. Yao, X. Ren, Z. Wang, P. Luo, H. Lu, and Z. Li. PixArt-Sigma: Weak-to-strong training of diffusion transformer for 4K text-to-image generation. arXiv preprint arXiv:2403.04692, 2024. — [arXiv:2403.04692](https://arxiv.org/abs/2403.04692) [find](https://scholar.google.com/scholar?q=J%2E+Chen%2C+C%2E+Ge%2C+E%2E+Xie%2C+Y%2E+Wu%2C+L%2E+Yao%2C+X%2E+Ren%2C+Z%2E+Wang%2C+P%2E+Luo%2C+H%2E+Lu%2C+and+Z%2E+Li%2E+PixArt%2DSigma%3A+Weak%2Dto%2Dstrong+training+of+diffusion+transformer+for+4K+text%2Dto%2Dimage+generation%2E+arXiv+preprint+arXiv%3A2)
6. X. Chu, L. Qiao, X. Lin, S. Xu, Y. Yang, Y. Hu, F. Wei, X. Zhang, B. Zhang, X. Wei, et al. Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint arXiv:2312.16886, 2023. [find](https://scholar.google.com/scholar?q=X%2E+Chu%2C+L%2E+Qiao%2C+X%2E+Lin%2C+S%2E+Xu%2C+Y%2E+Yang%2C+Y%2E+Hu%2C+F%2E+Wei%2C+X%2E+Zhang%2C+B%2E+Zhang%2C+X%2E+Wei%2C+et+al%2E+Mobilevlm%3A+A+fast%2C+reproducible+and+strong+vision+language+assistant+for+mobile+devices%2E+arXiv+preprint+arXiv)
7. X. Chu, L. Qiao, X. Zhang, S. Xu, F. Wei, Y. Yang, X. Sun, Y. Hu, X. Lin, B. Zhang, et al. Mobilevlm v2: Faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.03766, 2024. — [arXiv:2402.03766](https://arxiv.org/abs/2402.03766) [find](https://scholar.google.com/scholar?q=X%2E+Chu%2C+L%2E+Qiao%2C+X%2E+Zhang%2C+S%2E+Xu%2C+F%2E+Wei%2C+Y%2E+Yang%2C+X%2E+Sun%2C+Y%2E+Hu%2C+X%2E+Lin%2C+B%2E+Zhang%2C+et+al%2E+Mobilevlm+v2%3A+Faster+and+stronger+baseline+for+vision+language+model%2E+arXiv+preprint+arXiv%3A2402%2E03766%2C+2024%2E+)
8. W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023. [our card](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/)
9. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. [find](https://scholar.google.com/scholar?q=J%2E+Deng%2C+W%2E+Dong%2C+R%2E+Socher%2C+L%2E%2DJ%2E+Li%2C+K%2E+Li%2C+and+L%2E+Fei%2DFei%2E+Imagenet%3A+A+large%2Dscale+hierarchical+image+database%2E+In+2009+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+pages+248%E2%80%93255%2E+Ie)
10. R. Dong, C. Han, Y. Peng, Z. Qi, Z. Ge, J. Yang, L. Zhao, J. Sun, H. Zhou, H. Wei, et al. Dreamllm: Synergistic multimodal comprehension and creation. arXiv preprint arXiv:2309.11499, 2023. — [arXiv:2309.11499](https://arxiv.org/abs/2309.11499) [find](https://scholar.google.com/scholar?q=R%2E+Dong%2C+C%2E+Han%2C+Y%2E+Peng%2C+Z%2E+Qi%2C+Z%2E+Ge%2C+J%2E+Yang%2C+L%2E+Zhao%2C+J%2E+Sun%2C+H%2E+Zhou%2C+H%2E+Wei%2C+et+al%2E+Dreamllm%3A+Synergistic+multimodal+comprehension+and+creation%2E+arXiv+preprint+arXiv%3A2309%2E11499%2C+2023%2E+%E2%80%94+%5BarXiv%3A2)
11. P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel, D. Podell, T. Dockhorn, Z. English, K. Lacey, A. Goodwin, Y. Marek, and R. Rombach. Scaling rectified flow transformers for high-resolution image synthesis, 2024. URL https://arxiv.org/abs/24… — [arXiv:2403.03206](https://arxiv.org/abs/2403.03206) [↗](https://arxiv.org/abs/24…)
12. C. Fu, P. Chen, Y. Shen, Y. Qin, M. Zhang, X. Lin, J. Yang, X. Zheng, K. Li, X. Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023. [our card](https://md.research.elamur.ai/free/en/papers/fu-2023-mme-comprehensive-evaluation-benchmark-multimodal/)
13. Y. Ge, S. Zhao, J. Zhu, Y. Ge, K. Yi, L. Song, C. Li, X. Ding, and Y. Shan. Seed-x: Multimodal models with unified multi-granularity comprehension and generation. arXiv preprint arXiv:2404.14396, 2024. — [arXiv:2404.14396](https://arxiv.org/abs/2404.14396) [find](https://scholar.google.com/scholar?q=Y%2E+Ge%2C+S%2E+Zhao%2C+J%2E+Zhu%2C+Y%2E+Ge%2C+K%2E+Yi%2C+L%2E+Song%2C+C%2E+Li%2C+X%2E+Ding%2C+and+Y%2E+Shan%2E+Seed%2Dx%3A+Multimodal+models+with+unified+multi%2Dgranularity+comprehension+and+generation%2E+arXiv+preprint+arXiv%3A2404%2E14396%2C+2024)
14. D. Ghosh, H. Hajishirzi, and L. Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=D%2E+Ghosh%2C+H%2E+Hajishirzi%2C+and+L%2E+Schmidt%2E+Geneval%3A+An+object%2Dfocused+framework+for+evaluating+text%2Dto%2Dimage+alignment%2E+Advances+in+Neural+Information+Processing+Systems%2C+36%2C+2024%2E)
15. High-flyer. Hai-llm: Efficient and lightweight training tool for large models, 2023. URL [https://www.high-flyer.cn/en/blog/hai-llm](https://www.high-flyer.cn/en/blog/hai-llm) . [↗](https://www.high-flyer.cn/en/blog/hai-llm)
16. X. Hu, R. Wang, Y. Fang, B. Fu, P. Cheng, and G. Yu. Ella: Equip diffusion models with llm for enhanced semantic alignment. arXiv preprint arXiv:2403.05135, 2024. — [arXiv:2403.05135](https://arxiv.org/abs/2403.05135) [find](https://scholar.google.com/scholar?q=X%2E+Hu%2C+R%2E+Wang%2C+Y%2E+Fang%2C+B%2E+Fu%2C+P%2E+Cheng%2C+and+G%2E+Yu%2E+Ella%3A+Equip+diffusion+models+with+llm+for+enhanced+semantic+alignment%2E+arXiv+preprint+arXiv%3A2403%2E05135%2C+2024%2E+%E2%80%94+%5BarXiv%3A2403%2E05135%5D%28https%3A%2F%2Farxiv%2Eor)
17. D. A. Hudson and C. D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019. [our card](https://md.research.elamur.ai/free/en/papers/hudson-2019-gqa-new-dataset-real-world/)
18. Y. Jin, K. Xu, L. Chen, C. Liao, J. Tan, B. Chen, C. Lei, A. Liu, C. Song, X. Lei, et al. Unified language-vision pretraining with dynamic discrete visual tokenization. arXiv preprint arXiv:2309.04669, 2023. — [arXiv:2309.04669](https://arxiv.org/abs/2309.04669) [find](https://scholar.google.com/scholar?q=Y%2E+Jin%2C+K%2E+Xu%2C+L%2E+Chen%2C+C%2E+Liao%2C+J%2E+Tan%2C+B%2E+Chen%2C+C%2E+Lei%2C+A%2E+Liu%2C+C%2E+Song%2C+X%2E+Lei%2C+et+al%2E+Unified+language%2Dvision+pretraining+with+dynamic+discrete+visual+tokenization%2E+arXiv+preprint+arXiv%3A2309%2E04669)
19. H. Laurençon, D. van Strien, S. Bekman, L. Tronchon, L. Saulnier, T. Wang, S. Karamcheti, A. Singh, G. Pistilli, Y. Jernite, and et al. Introducing idefics: An open reproduction of state-of-the-art visual language model, 2023. URL [https://huggingface.co/blog/id](https://huggingface.co/blog/id) efics . [↗](https://huggingface.co/blog/id)
20. H. Laurençon, A. Marafioti, V. Sanh, and L. Tronchon. Building and better understanding vision-language models: insights and future directions., 2024. [find](https://scholar.google.com/scholar?q=H%2E+Lauren%C3%A7on%2C+A%2E+Marafioti%2C+V%2E+Sanh%2C+and+L%2E+Tronchon%2E+Building+and+better+understanding+vision%2Dlanguage+models%3A+insights+and+future+directions%2E%2C+2024%2E)
21. B. Li, R. Wang, G. Wang, Y. Ge, Y. Ge, and Y. Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023. [our card](https://md.research.elamur.ai/free/en/papers/li-2023-seed-bench-benchmarking-multimodal-llms/)
22. D. Li, A. Kamko, E. Akhgari, A. Sabet, L. Xu, and S. Doshi. Playground v2.5: Three insights towards enhancing aesthetic quality in text-to-image generation. arXiv preprint arXiv:2402.17245, 2024. — [arXiv:2402.17245](https://arxiv.org/abs/2402.17245) [find](https://scholar.google.com/scholar?q=D%2E+Li%2C+A%2E+Kamko%2C+E%2E+Akhgari%2C+A%2E+Sabet%2C+L%2E+Xu%2C+and+S%2E+Doshi%2E+Playground+v2%2E5%3A+Three+insights+towards+enhancing+aesthetic+quality+in+text%2Dto%2Dimage+generation%2E+arXiv+preprint+arXiv%3A2402%2E17245%2C+2024%2E+%E2%80%94+%5Ba)
23. Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J.-R. Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355, 2023. [our card](https://md.research.elamur.ai/free/en/papers/li-2023-evaluating-object-hallucination-large-vision/)
24. Z. Li, H. Li, Y. Shi, A. B. Farimani, Y. Kluger, L. Yang, and P. Wang. Dual diffusion for unified image generation and understanding. arXiv preprint arXiv:2501.00289, 2024. — [arXiv:2501.00289](https://arxiv.org/abs/2501.00289) [find](https://scholar.google.com/scholar?q=Z%2E+Li%2C+H%2E+Li%2C+Y%2E+Shi%2C+A%2E+B%2E+Farimani%2C+Y%2E+Kluger%2C+L%2E+Yang%2C+and+P%2E+Wang%2E+Dual+diffusion+for+unified+image+generation+and+understanding%2E+arXiv+preprint+arXiv%3A2501%2E00289%2C+2024%2E+%E2%80%94+%5BarXiv%3A2501%2E00289%5D%28https%3A)
25. Z. Li, J. Zhang, Q. Lin, J. Xiong, Y. Long, X. Deng, Y. Zhang, X. Liu, M. Huang, Z. Xiao, et al. Hunyuan-DiT: A powerful multi-resolution diffusion transformer with fine-grained chinese understanding. arXiv preprint arXiv:2405.08748, 2024. [find](https://scholar.google.com/scholar?q=Z%2E+Li%2C+J%2E+Zhang%2C+Q%2E+Lin%2C+J%2E+Xiong%2C+Y%2E+Long%2C+X%2E+Deng%2C+Y%2E+Zhang%2C+X%2E+Liu%2C+M%2E+Huang%2C+Z%2E+Xiao%2C+et+al%2E+Hunyuan%2DDiT%3A+A+powerful+multi%2Dresolution+diffusion+transformer+with+fine%2Dgrained+chinese+understanding%2E)
26. H. Liu, C. Li, Y. Li, and Y. J. Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/)
27. H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/)
28. H. Liu, W. Yan, M. Zaharia, and P. Abbeel. World model on million-length video and language with ringattention. arXiv preprint arXiv:2402.08268, 2024. [find](https://scholar.google.com/scholar?q=H%2E+Liu%2C+W%2E+Yan%2C+M%2E+Zaharia%2C+and+P%2E+Abbeel%2E+World+model+on+million%2Dlength+video+and+language+with+ringattention%2E+arXiv+preprint+arXiv%3A2402%2E08268%2C+2024%2E)
29. Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023. [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-mmbench-multi-modal-model-all/)
30. Y. Ma, X. Liu, X. Chen, W. Liu, C. Wu, Z. Wu, Z. Pan, Z. Xie, H. Zhang, X. yu, L. Zhao, Y. Wang, J. Liu, and C. Ruan. Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation, 2024. [find](https://scholar.google.com/scholar?q=Y%2E+Ma%2C+X%2E+Liu%2C+X%2E+Chen%2C+W%2E+Liu%2C+C%2E+Wu%2C+Z%2E+Wu%2C+Z%2E+Pan%2C+Z%2E+Xie%2C+H%2E+Zhang%2C+X%2E+yu%2C+L%2E+Zhao%2C+Y%2E+Wang%2C+J%2E+Liu%2C+and+C%2E+Ruan%2E+Janusflow%3A+Harmonizing+autoregression+and+rectified+flow+for+unified+multimodal+un)
31. mehdidc. Yfcc-huggingface. [https://huggingface.co/datasets/mehdidc/yfcc15](https://huggingface.co/datasets/mehdidc/yfcc15) m , 2024. [↗](https://huggingface.co/datasets/mehdidc/yfcc15)
32. D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. [our card](https://md.research.elamur.ai/free/en/papers/podell-2023-sdxl-improving-latent-diffusion-models/)
33. D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. 2024. [our card](https://md.research.elamur.ai/free/en/papers/podell-2023-sdxl-improving-latent-diffusion-models/)
34. L. Qu, H. Zhang, Y. Liu, X. Wang, Y. Jiang, Y. Gao, H. Ye, D. K. Du, Z. Yuan, and X. Wu. Tokenflow: Unified image tokenizer for multimodal understanding and generation. arXiv preprint arXiv:2412.03069, 2024. — [arXiv:2412.03069](https://arxiv.org/abs/2412.03069) [find](https://scholar.google.com/scholar?q=L%2E+Qu%2C+H%2E+Zhang%2C+Y%2E+Liu%2C+X%2E+Wang%2C+Y%2E+Jiang%2C+Y%2E+Gao%2C+H%2E+Ye%2C+D%2E+K%2E+Du%2C+Z%2E+Yuan%2C+and+X%2E+Wu%2E+Tokenflow%3A+Unified+image+tokenizer+for+multimodal+understanding+and+generation%2E+arXiv+preprint+arXiv%3A2412%2E03069)
35. A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022. [our card](https://md.research.elamur.ai/free/en/papers/ramesh-2022-hierarchical-text-conditional-image-generation/)
36. R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. 2022. [find](https://scholar.google.com/scholar?q=R%2E+Rombach%2C+A%2E+Blattmann%2C+D%2E+Lorenz%2C+P%2E+Esser%2C+and+B%2E+Ommer%2E+High%2Dresolution+image+synthesis+with+latent+diffusion+models%2E+2022%2E)
37. R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. [find](https://scholar.google.com/scholar?q=R%2E+Rombach%2C+A%2E+Blattmann%2C+D%2E+Lorenz%2C+P%2E+Esser%2C+and+B%2E+Ommer%2E+High%2Dresolution+image+synthesis+with+latent+diffusion+models%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+computer+vision+and+pattern+reco)
38. P. Sun, Y. Jiang, S. Chen, S. Zhang, B. Peng, P. Luo, and Z. Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525, 2024. — [arXiv:2406.06525](https://arxiv.org/abs/2406.06525) [find](https://scholar.google.com/scholar?q=P%2E+Sun%2C+Y%2E+Jiang%2C+S%2E+Chen%2C+S%2E+Zhang%2C+B%2E+Peng%2C+P%2E+Luo%2C+and+Z%2E+Yuan%2E+Autoregressive+model+beats+diffusion%3A+Llama+for+scalable+image+generation%2E+arXiv+preprint+arXiv%3A2406%2E06525%2C+2024%2E+%E2%80%94+%5BarXiv%3A2406%2E06525)
39. Q. Sun, Q. Yu, Y. Cui, F. Zhang, X. Zhang, Y. Wang, H. Gao, J. Liu, T. Huang, and X. Wang. Generative pretraining in multimodality. arXiv preprint arXiv:2307.05222, 2023. — [arXiv:2307.05222](https://arxiv.org/abs/2307.05222) [find](https://scholar.google.com/scholar?q=Q%2E+Sun%2C+Q%2E+Yu%2C+Y%2E+Cui%2C+F%2E+Zhang%2C+X%2E+Zhang%2C+Y%2E+Wang%2C+H%2E+Gao%2C+J%2E+Liu%2C+T%2E+Huang%2C+and+X%2E+Wang%2E+Generative+pretraining+in+multimodality%2E+arXiv+preprint+arXiv%3A2307%2E05222%2C+2023%2E+%E2%80%94+%5BarXiv%3A2307%2E05222%5D%28https%3A%2F%2F)
40. C. Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024. [our card](https://md.research.elamur.ai/free/en/papers/team-2024-chameleon-mixed-modal-early-fusion/)
41. G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. [our card](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/)
42. S. Tong, D. Fan, J. Zhu, Y. Xiong, X. Chen, K. Sinha, M. Rabbat, Y. LeCun, S. Xie, and Z. Liu. Metamorph: Multimodal understanding and generation via instruction tuning. arXiv preprint arXiv:2412.14164, 2024. — [arXiv:2412.14164](https://arxiv.org/abs/2412.14164) [find](https://scholar.google.com/scholar?q=S%2E+Tong%2C+D%2E+Fan%2C+J%2E+Zhu%2C+Y%2E+Xiong%2C+X%2E+Chen%2C+K%2E+Sinha%2C+M%2E+Rabbat%2C+Y%2E+LeCun%2C+S%2E+Xie%2C+and+Z%2E+Liu%2E+Metamorph%3A+Multimodal+understanding+and+generation+via+instruction+tuning%2E+arXiv+preprint+arXiv%3A2412%2E1416)
43. Vivym. Midjourney prompts dataset. [https://huggingface.co/datasets/vivym/](https://huggingface.co/datasets/vivym/) midjourney-prompts , 2023. Accessed: [Insert Date of Access, e.g., 2023-10-15]. [↗](https://huggingface.co/datasets/vivym/)
44. C. Wang, G. Lu, J. Yang, R. Huang, J. Han, L. Hou, W. Zhang, and H. Xu. Illume: Illuminating your llms to see, draw, and self-enhance. arXiv preprint arXiv:2412.06673, 2024. — [arXiv:2412.06673](https://arxiv.org/abs/2412.06673) [find](https://scholar.google.com/scholar?q=C%2E+Wang%2C+G%2E+Lu%2C+J%2E+Yang%2C+R%2E+Huang%2C+J%2E+Han%2C+L%2E+Hou%2C+W%2E+Zhang%2C+and+H%2E+Xu%2E+Illume%3A+Illuminating+your+llms+to+see%2C+draw%2C+and+self%2Denhance%2E+arXiv+preprint+arXiv%3A2412%2E06673%2C+2024%2E+%E2%80%94+%5BarXiv%3A2412%2E06673%5D%28https)
45. X. Wang, X. Zhang, Z. Luo, Q. Sun, Y. Cui, J. Wang, F. Zhang, Y. Wang, Z. Li, Q. Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024. — [arXiv:2409.18869](https://arxiv.org/abs/2409.18869) [find](https://scholar.google.com/scholar?q=X%2E+Wang%2C+X%2E+Zhang%2C+Z%2E+Luo%2C+Q%2E+Sun%2C+Y%2E+Cui%2C+J%2E+Wang%2C+F%2E+Zhang%2C+Y%2E+Wang%2C+Z%2E+Li%2C+Q%2E+Yu%2C+et+al%2E+Emu3%3A+Next%2Dtoken+prediction+is+all+you+need%2E+arXiv+preprint+arXiv%3A2409%2E18869%2C+2024%2E+%E2%80%94+%5BarXiv%3A2409%2E18869%5D%28htt)
46. C. Wu, X. Chen, Z. Wu, Y. Ma, X. Liu, Z. Pan, W. Liu, Z. Xie, X. Yu, C. Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint arXiv:2410.13848, 2024. [find](https://scholar.google.com/scholar?q=C%2E+Wu%2C+X%2E+Chen%2C+Z%2E+Wu%2C+Y%2E+Ma%2C+X%2E+Liu%2C+Z%2E+Pan%2C+W%2E+Liu%2C+Z%2E+Xie%2C+X%2E+Yu%2C+C%2E+Ruan%2C+et+al%2E+Janus%3A+Decoupling+visual+encoding+for+unified+multimodal+understanding+and+generation%2E+arXiv+preprint+arXiv%3A2410%2E13)
47. S. Wu, H. Fei, L. Qu, W. Ji, and T.-S. Chua. Next-gpt: Any-to-any multimodal llm. arXiv preprint arXiv:2309.05519, 2023. [find](https://scholar.google.com/scholar?q=S%2E+Wu%2C+H%2E+Fei%2C+L%2E+Qu%2C+W%2E+Ji%2C+and+T%2E%2DS%2E+Chua%2E+Next%2Dgpt%3A+Any%2Dto%2Dany+multimodal+llm%2E+arXiv+preprint+arXiv%3A2309%2E05519%2C+2023%2E)
48. Y. Wu, Z. Zhang, J. Chen, H. Tang, D. Li, Y. Fang, L. Zhu, E. Xie, H. Yin, L. Yi, et al. Vila-u: a unified foundation model integrating visual understanding and generation. arXiv preprint arXiv:2409.04429, 2024. — [arXiv:2409.04429](https://arxiv.org/abs/2409.04429) [find](https://scholar.google.com/scholar?q=Y%2E+Wu%2C+Z%2E+Zhang%2C+J%2E+Chen%2C+H%2E+Tang%2C+D%2E+Li%2C+Y%2E+Fang%2C+L%2E+Zhu%2C+E%2E+Xie%2C+H%2E+Yin%2C+L%2E+Yi%2C+et+al%2E+Vila%2Du%3A+a+unified+foundation+model+integrating+visual+understanding+and+generation%2E+arXiv+preprint+arXiv%3A2409%2E0)
49. Z. Wu, X. Chen, Z. Pan, X. Liu, W. Liu, D. Dai, H. Gao, Y. Ma, C. Wu, B. Wang, et al. Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302, 2024. [our card](https://md.research.elamur.ai/free/en/papers/wu-2024-deepseek-vl2-mixture-experts/)
50. J. Xie, W. Mao, Z. Bai, D. J. Zhang, W. Wang, K. Q. Lin, Y. Gu, Z. Chen, Z. Yang, and M. Z. Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528, 2024. — [arXiv:2408.12528](https://arxiv.org/abs/2408.12528) [find](https://scholar.google.com/scholar?q=J%2E+Xie%2C+W%2E+Mao%2C+Z%2E+Bai%2C+D%2E+J%2E+Zhang%2C+W%2E+Wang%2C+K%2E+Q%2E+Lin%2C+Y%2E+Gu%2C+Z%2E+Chen%2C+Z%2E+Yang%2C+and+M%2E+Z%2E+Shou%2E+Show%2Do%3A+One+single+transformer+to+unify+multimodal+understanding+and+generation%2E+arXiv+preprint+arXiv%3A)
51. W. Yu, Z. Yang, L. Li, J. Wang, K. Lin, Z. Liu, X. Wang, and L. Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023. [our card](https://md.research.elamur.ai/free/en/papers/yu-2023-mm-vet-evaluating-large-multimodal/)
52. X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556–9567, 2… [our card](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/)
53. X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023. [our card](https://md.research.elamur.ai/free/en/papers/zhai-2023-sigmoid-loss-language-image-pre/)
54. C. Zhao, Y. Song, W. Wang, H. Feng, E. Ding, Y. Sun, X. Xiao, and J. Wang. Monoformer: One transformer for both diffusion and autoregression. arXiv preprint arXiv:2409.16280, 2024. — [arXiv:2409.16280](https://arxiv.org/abs/2409.16280) [find](https://scholar.google.com/scholar?q=C%2E+Zhao%2C+Y%2E+Song%2C+W%2E+Wang%2C+H%2E+Feng%2C+E%2E+Ding%2C+Y%2E+Sun%2C+X%2E+Xiao%2C+and+J%2E+Wang%2E+Monoformer%3A+One+transformer+for+both+diffusion+and+autoregression%2E+arXiv+preprint+arXiv%3A2409%2E16280%2C+2024%2E+%E2%80%94+%5BarXiv%3A2409%2E16280)
55. C. Zhou, L. Yu, A. Babu, K. Tirumala, M. Yasunaga, L. Shamis, J. Kahn, X. Ma, L. Zettlemoyer, and O. Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.11039, 2024. [find](https://scholar.google.com/scholar?q=C%2E+Zhou%2C+L%2E+Yu%2C+A%2E+Babu%2C+K%2E+Tirumala%2C+M%2E+Yasunaga%2C+L%2E+Shamis%2C+J%2E+Kahn%2C+X%2E+Ma%2C+L%2E+Zettlemoyer%2C+and+O%2E+Levy%2E+Transfusion%3A+Predict+the+next+token+and+diffuse+images+with+one+multi%2Dmodal+model%2E+arXiv+prep)
56. Y. Zhu, M. Zhu, N. Liu, Z. Ou, X. Mou, and J. Tang. Llava-phi: Efficient multi-modal assistant with small language model. arXiv preprint arXiv:2401.02330, 2024. — [arXiv:2401.02330](https://arxiv.org/abs/2401.02330) [find](https://scholar.google.com/scholar?q=Y%2E+Zhu%2C+M%2E+Zhu%2C+N%2E+Liu%2C+Z%2E+Ou%2C+X%2E+Mou%2C+and+J%2E+Tang%2E+Llava%2Dphi%3A+Efficient+multi%2Dmodal+assistant+with+small+language+model%2E+arXiv+preprint+arXiv%3A2401%2E02330%2C+2024%2E+%E2%80%94+%5BarXiv%3A2401%2E02330%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
57. L. Zhuo, R. Du, H. Xiao, Y. Li, D. Liu, R. Huang, W. Liu, L. Zhao, F.-Y. Wang, Z. Ma, et al. Lumina-Next: Making Lumina-T2X stronger and faster with Next-DiT. arXiv preprint arXiv:2406.18583, 2024. — [arXiv:2406.18583](https://arxiv.org/abs/2406.18583) [find](https://scholar.google.com/scholar?q=L%2E+Zhuo%2C+R%2E+Du%2C+H%2E+Xiao%2C+Y%2E+Li%2C+D%2E+Liu%2C+R%2E+Huang%2C+W%2E+Liu%2C+L%2E+Zhao%2C+F%2E%2DY%2E+Wang%2C+Z%2E+Ma%2C+et+al%2E+Lumina%2DNext%3A+Making+Lumina%2DT2X+stronger+and+faster+with+Next%2DDiT%2E+arXiv+preprint+arXiv%3A2406%2E18583%2C+2024%2E+%E2%80%94+)