---
title: "JanusFlow — paper by Ma et al. (2024) — summary and ideas"
description: "Two design choices improve quality. 110 references, cited by 26 papers here. Among them: 26."
authors: ["Yiyang Ma", "Xingchao Liu", "Xiaokang Chen", "Wen Liu", "Chengyue Wu", "Zhiyu Wu", "Zizheng Pan", "Zhenda Xie", "Haowei Zhang", "Xingkai yu", "Liang Zhao", "Yisong Wang", "Jiaying Liu", "Chong Ruan"]
year: 2024
venue: "arXiv:2411.07975"
source: "https://arxiv.org/abs/2411.07975"
doi: "10.48550/arxiv.2411.07975"
arxiv: "2411.07975"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ma-2024-janusflow-harmonizing-autoregression-rectified/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ma-2024-janusflow-harmonizing-autoregression-rectified/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation

# JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation

**Authors:** [Yiyang Ma](https://md.research.elamur.ai/free/en/authors/yiyang-ma/), [Xingchao Liu](https://md.research.elamur.ai/free/en/authors/xingchao-liu/), [Xiaokang Chen](https://md.research.elamur.ai/free/en/authors/xiaokang-chen/), [Wen Liu](https://md.research.elamur.ai/free/en/authors/wen-liu/), [Chengyue Wu](https://md.research.elamur.ai/free/en/authors/chengyue-wu/), [Zhiyu Wu](https://md.research.elamur.ai/free/en/authors/zhiyu-wu/), [Zizheng Pan](https://md.research.elamur.ai/free/en/authors/zizheng-pan/), [Zhenda Xie](https://md.research.elamur.ai/free/en/authors/zhenda-xie/), [Haowei Zhang](https://md.research.elamur.ai/free/en/authors/haowei-zhang/), [Xingkai yu](https://md.research.elamur.ai/free/en/authors/xingkai-yu/), [Liang Zhao](https://md.research.elamur.ai/free/en/authors/liang-zhao/), Yisong Wang, Jiaying Liu and [Chong Ruan](https://md.research.elamur.ai/free/en/authors/chong-ruan/) · **Year:** 2024 · **Venue:** [arXiv:2411.07975](https://md.research.elamur.ai/free/en/venues/ieee-computer-society-conference-on-computer-vision-and-pattern-recognition/) ([IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2024](https://md.research.elamur.ai/free/en/venues/ieee-computer-society-conference-on-computer-vision-and-pattern-recognition/2024/)) · **Publisher:** [Institute of Electrical and Electronics Engineers](https://md.research.elamur.ai/free/en/publishers/institute-of-electrical-and-electronics-engineers/) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2411.07975) · [PDF](https://arxiv.org/pdf/2411.07975) · [DOI 10.48550/arxiv.2411.07975](https://doi.org/10.48550/arxiv.2411.07975) · [arXiv 2411.07975](https://arxiv.org/abs/2411.07975) · [OpenAlex](https://openalex.org/W4413147805)

**Citations:** 26 citations (OpenAlex, 2026-10-14) · [26](https://md.research.elamur.ai/free/en/papers/ma-2024-janusflow-harmonizing-autoregression-rectified/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Unified vision-language systems usually either attach an external text-to-image generator to an LLM or turn images into discrete vector-quantized tokens. The first adds architectural complexity, and the second is limited by tokenizer quality. JanusFlow, from DeepSeek-AI and collaborators, shows that rectified flow can be trained directly inside an autoregressive LLM. Only a small generation encoder and decoder are needed, operating in the SDXL-VAE latent space. Text answers are produced by next-token prediction. Images are produced by having the LLM predict velocity vectors, which an Euler solver integrates from Gaussian noise, with classifier-free guidance applied. Two design choices improve quality. Understanding and generation use separate visual encoders, which avoids interference between the tasks. The model is also trained with a representation alignment regularizer that ties the two pathways together. With only a 1.3B-parameter LLM, JanusFlow outperforms earlier unified models and matches or beats several specialized systems on both understanding and generation. This makes it a compact, simple baseline for single-model multimodal work.

## Key points
- **Generation benchmarks:** MJHQ FID-30k 9.51, GenEval 0.63 and DPG-Bench 80.09%, ahead of SDv1.5 and SDXL.
- **Understanding benchmarks:** MMBench 74.9, SeedBench 70.5 and GQA 60.3, above LLaVA-v1.5 and Qwen-VL-Chat.
- **Architecture and output:** the backbone LLM has 1.3B parameters, and generated images are 384 × 384. Plain causal attention proved sufficient, since alternative masking schemes brought no gains.
- **Ablations:** these ran at 256 × 256 for 50,000 iterations with a SigLIP-Large-Patch/16 understanding encoder. Removing representation alignment worsened FID and CLIP scores. A shared ConvNeXt encoder (Transfusion-style) underperformed decoupled encoders. Task-specific models trained on the same data were only marginally better.
- **Code:** released at github.com/deepseek-ai/Janus.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 244 works · 39,359 citations
- [DeepSeek](https://md.research.elamur.ai/free/en/companies/deepseek/) — Company, Hangzhou, China · 9 works · 7,942 citations

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 657 works

## Cite

Cite as: Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Xingkai yu, Liang Zhao, Yisong Wang, Jiaying Liu and Chong Ruan. 2024. JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation. arXiv:2411.07975.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{ma2024janusflow,
  title = {{JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation}},
  author = {Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Xingkai yu, Liang Zhao, Yisong Wang, Jiaying Liu and Chong Ruan},
  year = {2024},
  doi = {10.48550/arxiv.2411.07975},
  eprint = {2411.07975},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2411.07975}
}
```

- [← previous CogVLM2: Visual Language Models for Image and Video Understanding](https://md.research.elamur.ai/free/en/papers/hong-2024-cogvlm2-visual-language-models/)
- [next → LLM Internal States Reveal Hallucination Risk Faced With a Query](https://md.research.elamur.ai/free/en/papers/ji-2024-llm-internal-states-reveal-hallucination/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/)

## References

110 references, 40 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774, 2023.
2. J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. [Flamingo: a visual language model for few-shot learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/). In Proc. Annu. Conf. Neural Inf. Process. Systems, 2022.
3. M. Albergo and E. Vanden-Eijnden. Building normalizing flows with stochastic interpolants. In Proc. Int’l Conf. Learning Representations, 2023. [find](https://scholar.google.com/scholar?q=M%2E+Albergo+and+E%2E+Vanden%2DEijnden%2E+Building+normalizing+flows+with+stochastic+interpolants%2E+In+Proc%2E+Int%E2%80%99l+Conf%2E+Learning+Representations%2C+2023%2E)
4. J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou. Qwen-VL: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.[12966, 2023](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/).
5. F. Bao, S. Nie, K. Xue, Y. Cao, C. Li, H. Su, and J. Zhu. All are worth words: A ViT backbone for diffusion models. In Proc. IEEE Int’l Conf. Computer Vision and Pattern Recognition, 2023. [find](https://scholar.google.com/scholar?q=F%2E+Bao%2C+S%2E+Nie%2C+K%2E+Xue%2C+Y%2E+Cao%2C+C%2E+Li%2C+H%2E+Su%2C+and+J%2E+Zhu%2E+All+are+worth+words%3A+A+ViT+backbone+for+diffusion+models%2E+In+Proc%2E+IEEE+Int%E2%80%99l+Conf%2E+Computer+Vision+and+Pattern+Recognition%2C+2023%2E)
6. J. Betker, G. Goh, L. Jing, T. Brooks, J. Wang, L. Li, L. Ouyang, J. Zhuang, J. Lee, Y. Guo, et al. Improving image generation with better captions. Computer Science, 2023. [find](https://scholar.google.com/scholar?q=J%2E+Betker%2C+G%2E+Goh%2C+L%2E+Jing%2C+T%2E+Brooks%2C+J%2E+Wang%2C+L%2E+Li%2C+L%2E+Ouyang%2C+J%2E+Zhuang%2C+J%2E+Lee%2C+Y%2E+Guo%2C+et+al%2E+Improving+image+generation+with+better+captions%2E+Computer+Science%2C+2023%2E)
7. X. Bi, D. Chen, G. Chen, S. Chen, D. Dai, C. Deng, H. Ding, K. Dong, Q. Du, Z. Fu, et al. DeepSeek LLM: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024. [find](https://scholar.google.com/scholar?q=X%2E+Bi%2C+D%2E+Chen%2C+G%2E+Chen%2C+S%2E+Chen%2C+D%2E+Dai%2C+C%2E+Deng%2C+H%2E+Ding%2C+K%2E+Dong%2C+Q%2E+Du%2C+Z%2E+Fu%2C+et+al%2E+DeepSeek+LLM%3A+Scaling+open%2Dsource+language+models+with+longtermism%2E+arXiv+preprint+arXiv%3A2401%2E02954%2C+2024%2E)
8. S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. [Sparks of artificial general intelligence: Early experiments with GPT-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/). arXiv preprint arXiv:2303.12712, 2023.
9. J. Chen, J. Yu, C. Ge, L. Yao, E. Xie, Y. Wu, Z. Wang, J. Kwok, P. Luo, H. Lu, et al. PixArt-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023. [find](https://scholar.google.com/scholar?q=J%2E+Chen%2C+J%2E+Yu%2C+C%2E+Ge%2C+L%2E+Yao%2C+E%2E+Xie%2C+Y%2E+Wu%2C+Z%2E+Wang%2C+J%2E+Kwok%2C+P%2E+Luo%2C+H%2E+Lu%2C+et+al%2E+PixArt%2Dalpha%3A+Fast+training+of+diffusion+transformer+for+photorealistic+text%2Dto%2Dimage+synthesis%2E+arXiv+preprint+ar)
10. J. Chen, C. Ge, E. Xie, Y. Wu, L. Yao, X. Ren, Z. Wang, P. Luo, H. Lu, and Z. Li. PixArt-Sigma: Weak-to-strong training of diffusion transformer for 4K text-to-image generation. arXiv preprint arXiv:2403.04692, 2024. — [arXiv:2403.04692](https://arxiv.org/abs/2403.04692) [find](https://scholar.google.com/scholar?q=J%2E+Chen%2C+C%2E+Ge%2C+E%2E+Xie%2C+Y%2E+Wu%2C+L%2E+Yao%2C+X%2E+Ren%2C+Z%2E+Wang%2C+P%2E+Luo%2C+H%2E+Lu%2C+and+Z%2E+Li%2E+PixArt%2DSigma%3A+Weak%2Dto%2Dstrong+training+of+diffusion+transformer+for+4K+text%2Dto%2Dimage+generation%2E+arXiv+preprint+arXiv%3A2)
11. L. Chen, J. Li, X. Dong, P. Zhang, C. He, J. Wang, F. Zhao, and D. Lin. [ShareGPT4V: Improving large multi-modal models with better captions](https://md.research.elamur.ai/free/en/papers/chen-2023-sharegpt4v-improving-large-multi-modal/). arXiv preprint arXiv:2311.12793, 2023.
12. X. Chu, L. Qiao, X. Lin, S. Xu, Y. Yang, Y. Hu, F. Wei, X. Zhang, B. Zhang, X. Wei, et al. MobileVLM: A fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint arXiv:2312.16886, 2023. [find](https://scholar.google.com/scholar?q=X%2E+Chu%2C+L%2E+Qiao%2C+X%2E+Lin%2C+S%2E+Xu%2C+Y%2E+Yang%2C+Y%2E+Hu%2C+F%2E+Wei%2C+X%2E+Zhang%2C+B%2E+Zhang%2C+X%2E+Wei%2C+et+al%2E+MobileVLM%3A+A+fast%2C+reproducible+and+strong+vision+language+assistant+for+mobile+devices%2E+arXiv+preprint+arXiv)
13. X. Chu, L. Qiao, X. Zhang, S. Xu, F. Wei, Y. Yang, X. Sun, Y. Hu, X. Lin, B. Zhang, et al. MobileVLM V2: Faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.03766, 2024. — [arXiv:2402.03766](https://arxiv.org/abs/2402.03766) [find](https://scholar.google.com/scholar?q=X%2E+Chu%2C+L%2E+Qiao%2C+X%2E+Zhang%2C+S%2E+Xu%2C+F%2E+Wei%2C+Y%2E+Yang%2C+X%2E+Sun%2C+Y%2E+Hu%2C+X%2E+Lin%2C+B%2E+Zhang%2C+et+al%2E+MobileVLM+V2%3A+Faster+and+stronger+baseline+for+vision+language+model%2E+arXiv+preprint+arXiv%3A2402%2E03766%2C+2024%2E+)
14. K. Crowson, S. A. Baumann, A. Birch, T. M. Abraham, D. Z. Kaplan, and E. Shippole. Scalable high-resolution pixel-space image synthesis with hourglass diffusion transformers. In Proc. Int’l Conf. Machine Learning, 2024. [find](https://scholar.google.com/scholar?q=K%2E+Crowson%2C+S%2E+A%2E+Baumann%2C+A%2E+Birch%2C+T%2E+M%2E+Abraham%2C+D%2E+Z%2E+Kaplan%2C+and+E%2E+Shippole%2E+Scalable+high%2Dresolution+pixel%2Dspace+image+synthesis+with+hourglass+diffusion+transformers%2E+In+Proc%2E+Int%E2%80%99l+Conf%2E+Mach)
15. W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi. [InstructBLIP: Towards general-purpose vision-language models with instruction tuning](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/). In Proc. Annu. Conf. Neural Inf. Process. Systems, 2023.
16. dclure. LAION-Aesthetics-UMAP, 2022. URL [https://huggingface.co/datasets/](https://huggingface.co/datasets/) dclure/laion-aesthetics-12m-umap . [↗](https://huggingface.co/datasets/)
17. DeepFloyd. DeepFloyd IF, 2023. URL [https://huggingface.co/DeepFloyd/IF-I](https://huggingface.co/DeepFloyd/IF-I) -XL-v1.0 . [↗](https://huggingface.co/DeepFloyd/IF-I)
18. P. Dhariwal and A. Nichol. Diffusion models beat GANs on image synthesis. In Proc. Annu. Conf. Neural Inf. Process. Systems, 2021. [find](https://scholar.google.com/scholar?q=P%2E+Dhariwal+and+A%2E+Nichol%2E+Diffusion+models+beat+GANs+on+image+synthesis%2E+In+Proc%2E+Annu%2E+Conf%2E+Neural+Inf%2E+Process%2E+Systems%2C+2021%2E)
19. R. Dong, C. Han, Y. Peng, Z. Qi, Z. Ge, J. Yang, L. Zhao, J. Sun, H. Zhou, H. Wei, et al. DreamLLM: Synergistic multimodal comprehension and creation. In Proc. Int’l Conf. Learning Representations, 2024. [find](https://scholar.google.com/scholar?q=R%2E+Dong%2C+C%2E+Han%2C+Y%2E+Peng%2C+Z%2E+Qi%2C+Z%2E+Ge%2C+J%2E+Yang%2C+L%2E+Zhao%2C+J%2E+Sun%2C+H%2E+Zhou%2C+H%2E+Wei%2C+et+al%2E+DreamLLM%3A+Synergistic+multimodal+comprehension+and+creation%2E+In+Proc%2E+Int%E2%80%99l+Conf%2E+Learning+Representations%2C+20)
20. echo840. Detailed caption, 2023. URL [https://huggingface.co/datasets/echo84](https://huggingface.co/datasets/echo84) 0/Detailed_Caption . [↗](https://huggingface.co/datasets/echo84)
21. B. Egan, A. Redden, XWAVE, and SilentAntagonist. DALLE-3 1 million+ high quality captions, 2024. URL [https://huggingface.co/datasets/ProGamerGov/synthe](https://huggingface.co/datasets/ProGamerGov/synthe) tic-dataset-1m-dalle3-high-quality-captions . [↗](https://huggingface.co/datasets/ProGamerGov/synthe)
22. P. Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis. In Proc. IEEE Int’l Conf. Computer Vision and Pattern Recognition, 2021. [find](https://scholar.google.com/scholar?q=P%2E+Esser%2C+R%2E+Rombach%2C+and+B%2E+Ommer%2E+Taming+transformers+for+high%2Dresolution+image+synthesis%2E+In+Proc%2E+IEEE+Int%E2%80%99l+Conf%2E+Computer+Vision+and+Pattern+Recognition%2C+2021%2E)
23. P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Proc. Int’l Conf. Machine Learning, 2024. [find](https://scholar.google.com/scholar?q=P%2E+Esser%2C+S%2E+Kulal%2C+A%2E+Blattmann%2C+R%2E+Entezari%2C+J%2E+M%C3%BCller%2C+H%2E+Saini%2C+Y%2E+Levi%2C+D%2E+Lorenz%2C+A%2E+Sauer%2C+F%2E+Boesel%2C+et+al%2E+Scaling+rectified+flow+transformers+for+high%2Dresolution+image+synthesis%2E+In+Proc%2E+In)
24. C. Fu, P. Chen, Y. Shen, Y. Qin, M. Zhang, X. Lin, J. Yang, X. Zheng, K. Li, X. Sun, Y. Wu, and R. Ji. [MME: A comprehensive evaluation benchmark for multimodal large language models](https://md.research.elamur.ai/free/en/papers/fu-2023-mme-comprehensive-evaluation-benchmark-multimodal/). arXiv preprint arXiv:2306.13394, 2024.
25. Y. Ge, Y. Ge, Z. Zeng, X. Wang, and Y. Shan. Planting a SEED of vision in large language model. arXiv preprint arXiv:2307.08041, 2023. [find](https://scholar.google.com/scholar?q=Y%2E+Ge%2C+Y%2E+Ge%2C+Z%2E+Zeng%2C+X%2E+Wang%2C+and+Y%2E+Shan%2E+Planting+a+SEED+of+vision+in+large+language+model%2E+arXiv+preprint+arXiv%3A2307%2E08041%2C+2023%2E)
26. Y. Ge, S. Zhao, Z. Zeng, Y. Ge, C. Li, X. Wang, and Y. Shan. Making LLaMA SEE and draw with SEED tokenizer. arXiv preprint arXiv:2310.01218, 2023. — [arXiv:2310.01218](https://arxiv.org/abs/2310.01218) [find](https://scholar.google.com/scholar?q=Y%2E+Ge%2C+S%2E+Zhao%2C+Z%2E+Zeng%2C+Y%2E+Ge%2C+C%2E+Li%2C+X%2E+Wang%2C+and+Y%2E+Shan%2E+Making+LLaMA+SEE+and+draw+with+SEED+tokenizer%2E+arXiv+preprint+arXiv%3A2310%2E01218%2C+2023%2E+%E2%80%94+%5BarXiv%3A2310%2E01218%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2310%2E01218)
27. Y. Ge, S. Zhao, J. Zhu, Y. Ge, K. Yi, L. Song, C. Li, X. Ding, and Y. Shan. SEED-X: Multimodal models with unified multi-granularity comprehension and generation. arXiv preprint arXiv:2404.14396, 2024. — [arXiv:2404.14396](https://arxiv.org/abs/2404.14396) [find](https://scholar.google.com/scholar?q=Y%2E+Ge%2C+S%2E+Zhao%2C+J%2E+Zhu%2C+Y%2E+Ge%2C+K%2E+Yi%2C+L%2E+Song%2C+C%2E+Li%2C+X%2E+Ding%2C+and+Y%2E+Shan%2E+SEED%2DX%3A+Multimodal+models+with+unified+multi%2Dgranularity+comprehension+and+generation%2E+arXiv+preprint+arXiv%3A2404%2E14396%2C+2024)
28. D. Ghosh, H. Hajishirzi, and L. Schmidt. GenEval: An object-focused framework for evaluating text-to-image alignment. In Proc. Annu. Conf. Neural Inf. Process. Systems, 2024. [find](https://scholar.google.com/scholar?q=D%2E+Ghosh%2C+H%2E+Hajishirzi%2C+and+L%2E+Schmidt%2E+GenEval%3A+An+object%2Dfocused+framework+for+evaluating+text%2Dto%2Dimage+alignment%2E+In+Proc%2E+Annu%2E+Conf%2E+Neural+Inf%2E+Process%2E+Systems%2C+2024%2E)
29. Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh. [Making the v in VQA matter: Elevating the role of image understanding in visual question answering](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/). In Proc. IEEE Int’l Conf. Computer Vision and Pattern Recognition, 2017.
30. M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs trained by a two time-scale update rule converge to a local nash equilibrium. Proc. Annu. Conf. Neural Inf. Process. Systems, 2017. [find](https://scholar.google.com/scholar?q=M%2E+Heusel%2C+H%2E+Ramsauer%2C+T%2E+Unterthiner%2C+B%2E+Nessler%2C+and+S%2E+Hochreiter%2E+GANs+trained+by+a+two+time%2Dscale+update+rule+converge+to+a+local+nash+equilibrium%2E+Proc%2E+Annu%2E+Conf%2E+Neural+Inf%2E+Process%2E+Systems)
31. High-flyer. HAI-LLM: Efficient and lightweight training tool for large models, 2023. URL [https://www.high-flyer.cn/en/blog/hai-llm](https://www.high-flyer.cn/en/blog/hai-llm) . [↗](https://www.high-flyer.cn/en/blog/hai-llm)
32. J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In Proc. Annu. Conf. Neural Inf. Process. Systems, 2020. [find](https://scholar.google.com/scholar?q=J%2E+Ho%2C+A%2E+Jain%2C+and+P%2E+Abbeel%2E+Denoising+diffusion+probabilistic+models%2E+In+Proc%2E+Annu%2E+Conf%2E+Neural+Inf%2E+Process%2E+Systems%2C+2020%2E)
33. Y.-C. Hsiao, F. Zubach, G. Baechler, V. Carbune, J. Lin, M. Wang, S. Sunkara, Y. Zhu, and J. Chen. ScreenQA: Large-scale question-answer pairs over mobile app screenshots. arXiv preprint arXiv:2209.08199, 2022. — [arXiv:2209.08199](https://arxiv.org/abs/2209.08199) [find](https://scholar.google.com/scholar?q=Y%2E%2DC%2E+Hsiao%2C+F%2E+Zubach%2C+G%2E+Baechler%2C+V%2E+Carbune%2C+J%2E+Lin%2C+M%2E+Wang%2C+S%2E+Sunkara%2C+Y%2E+Zhu%2C+and+J%2E+Chen%2E+ScreenQA%3A+Large%2Dscale+question%2Danswer+pairs+over+mobile+app+screenshots%2E+arXiv+preprint+arXiv%3A2209%2E08)
34. X. Hu, R. Wang, Y. Fang, B. Fu, P. Cheng, and G. Yu. ELLA: Equip diffusion models with llm for enhanced semantic alignment. arXiv preprint arXiv:2403.05135, 2024. — [arXiv:2403.05135](https://arxiv.org/abs/2403.05135) [find](https://scholar.google.com/scholar?q=X%2E+Hu%2C+R%2E+Wang%2C+Y%2E+Fang%2C+B%2E+Fu%2C+P%2E+Cheng%2C+and+G%2E+Yu%2E+ELLA%3A+Equip+diffusion+models+with+llm+for+enhanced+semantic+alignment%2E+arXiv+preprint+arXiv%3A2403%2E05135%2C+2024%2E+%E2%80%94+%5BarXiv%3A2403%2E05135%5D%28https%3A%2F%2Farxiv%2Eor)
35. D. A. Hudson and C. D. Manning. [GQA: A new dataset for real-world visual reasoning and compositional question answering](https://md.research.elamur.ai/free/en/papers/hudson-2019-gqa-new-dataset-real-world/). In Proc. IEEE Int’l Conf. Computer Vision and Pattern Recognition, 2019.
36. Y. Jin, Z. Sun, N. Li, K. Xu, H. Jiang, N. Zhuang, Q. Huang, Y. Song, Y. Mu, and Z. Lin. Pyramidal flow matching for efficient video generative modeling. arXiv preprint arXiv:2410.05954, 2024. [find](https://scholar.google.com/scholar?q=Y%2E+Jin%2C+Z%2E+Sun%2C+N%2E+Li%2C+K%2E+Xu%2C+H%2E+Jiang%2C+N%2E+Zhuang%2C+Q%2E+Huang%2C+Y%2E+Song%2C+Y%2E+Mu%2C+and+Z%2E+Lin%2E+Pyramidal+flow+matching+for+efficient+video+generative+modeling%2E+arXiv+preprint+arXiv%3A2410%2E05954%2C+2024%2E)
37. Y. Jin, K. Xu, L. Chen, C. Liao, J. Tan, Q. Huang, C. Bin, C. Song, D. ZHANG, W. Ou, et al. Unified language-vision pretraining in llm with dynamic discrete visual tokenization. In Proc. Int’l Conf. Learning Representations, 2024. [find](https://scholar.google.com/scholar?q=Y%2E+Jin%2C+K%2E+Xu%2C+L%2E+Chen%2C+C%2E+Liao%2C+J%2E+Tan%2C+Q%2E+Huang%2C+C%2E+Bin%2C+C%2E+Song%2C+D%2E+ZHANG%2C+W%2E+Ou%2C+et+al%2E+Unified+language%2Dvision+pretraining+in+llm+with+dynamic+discrete+visual+tokenization%2E+In+Proc%2E+Int%E2%80%99l+Conf%2E+L)
38. B. Jing, B. Berger, and T. Jaakkola. AlphaFold meets flow matching for generating protein ensembles. In Proc. Int’l Conf. Machine Learning, 2024. [find](https://scholar.google.com/scholar?q=B%2E+Jing%2C+B%2E+Berger%2C+and+T%2E+Jaakkola%2E+AlphaFold+meets+flow+matching+for+generating+protein+ensembles%2E+In+Proc%2E+Int%E2%80%99l+Conf%2E+Machine+Learning%2C+2024%2E)
39. M. Kang, J.-Y. Zhu, R. Zhang, J. Park, E. Shechtman, S. Paris, and T. Park. Scaling up GANs for text-to-image synthesis. In Proc. IEEE Int’l Conf. Computer Vision and Pattern Recognition, 2023. [find](https://scholar.google.com/scholar?q=M%2E+Kang%2C+J%2E%2DY%2E+Zhu%2C+R%2E+Zhang%2C+J%2E+Park%2C+E%2E+Shechtman%2C+S%2E+Paris%2C+and+T%2E+Park%2E+Scaling+up+GANs+for+text%2Dto%2Dimage+synthesis%2E+In+Proc%2E+IEEE+Int%E2%80%99l+Conf%2E+Computer+Vision+and+Pattern+Recognition%2C+2023%2E)
40. S. Kim, K. Shih, J. F. Santos, E. Bakhturina, M. Desta, R. Valle, S. Yoon, B. Catanzaro, et al. P-Flow: a fast and data-efficient zero-shot tts through speech prompting. In Proc. Annu. Conf. Neural Inf. Process. Systems, 2024. [find](https://scholar.google.com/scholar?q=S%2E+Kim%2C+K%2E+Shih%2C+J%2E+F%2E+Santos%2C+E%2E+Bakhturina%2C+M%2E+Desta%2C+R%2E+Valle%2C+S%2E+Yoon%2C+B%2E+Catanzaro%2C+et+al%2E+P%2DFlow%3A+a+fast+and+data%2Defficient+zero%2Dshot+tts+through+speech+prompting%2E+In+Proc%2E+Annu%2E+Conf%2E+Neural+In)
41. A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al. [Segment anything](https://md.research.elamur.ai/free/en/papers/kirillov-2023-segment-anything/). In Proc. IEEE Int. Conf. Comput. Vision, 2023.
42. M. Koupaee and W. Y. Wang. WikiHow: A large scale text summarization dataset. arXiv preprint arXiv:1810.09305, 2018. [find](https://scholar.google.com/scholar?q=M%2E+Koupaee+and+W%2E+Y%2E+Wang%2E+WikiHow%3A+A+large+scale+text+summarization+dataset%2E+arXiv+preprint+arXiv%3A1810%2E09305%2C+2018%2E)
43. A. Kuznetsova, H. Rom, N. Alldrin, J. Uijlings, I. Krasin, J. Pont-Tuset, S. Kamali, S. Popov, M. Malloci, A. Kolesnikov, et al. The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale. Int’l Journal of Computer Vision, 2020. [find](https://scholar.google.com/scholar?q=A%2E+Kuznetsova%2C+H%2E+Rom%2C+N%2E+Alldrin%2C+J%2E+Uijlings%2C+I%2E+Krasin%2C+J%2E+Pont%2DTuset%2C+S%2E+Kamali%2C+S%2E+Popov%2C+M%2E+Malloci%2C+A%2E+Kolesnikov%2C+et+al%2E+The+Open+Images+Dataset+V4%3A+Unified+image+classification%2C+object+detect)
44. H. Laurençon, D. van Strien, S. Bekman, L. Tronchon, L. Saulnier, T. Wang, S. Karamcheti, A. Singh, G. Pistilli, Y. Jernite, et al. Introducing IDEFICS: An open reproduction of state-of-the-art visual language model, 2023, 2023. URL [https://huggingface.co/b](https://huggingface.co/b) log/idefics . [↗](https://huggingface.co/b)
45. M. Le, A. Vyas, B. Shi, B. Karrer, L. Sari, R. Moritz, M. Williamson, V. Manohar, Y. Adi, J. Mahadeokar, et al. VoiceBox: Text-guided multilingual universal speech generation at scale. In Proc. Annu. Conf. Neural Inf. Process. Systems, 2024. [find](https://scholar.google.com/scholar?q=M%2E+Le%2C+A%2E+Vyas%2C+B%2E+Shi%2C+B%2E+Karrer%2C+L%2E+Sari%2C+R%2E+Moritz%2C+M%2E+Williamson%2C+V%2E+Manohar%2C+Y%2E+Adi%2C+J%2E+Mahadeokar%2C+et+al%2E+VoiceBox%3A+Text%2Dguided+multilingual+universal+speech+generation+at+scale%2E+In+Proc%2E+Annu%2E+)
46. B. Li, R. Wang, G. Wang, Y. Ge, Y. Ge, and Y. Shan. [SEED-Bench: Benchmarking multimodal llms with generative comprehension](https://md.research.elamur.ai/free/en/papers/li-2023-seed-bench-benchmarking-multimodal-llms/). arXiv preprint arXiv:2307.16125, 2023.
47. B. Li, Y. Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, Y. Li, Z. Liu, and C. Li. [LLaVA-OneVision: Easy visual task transfer](https://md.research.elamur.ai/free/en/papers/li-2024-llava-onevision-easy-visual-task/). arXiv preprint arXiv:2408.03326, 2024.
48. D. Li, A. Kamko, E. Akhgari, A. Sabet, L. Xu, and S. Doshi. Playground v2.5: Three insights towards enhancing aesthetic quality in text-to-image generation. arXiv preprint arXiv:2402.17245, 2024. — [arXiv:2402.17245](https://arxiv.org/abs/2402.17245) [find](https://scholar.google.com/scholar?q=D%2E+Li%2C+A%2E+Kamko%2C+E%2E+Akhgari%2C+A%2E+Sabet%2C+L%2E+Xu%2C+and+S%2E+Doshi%2E+Playground+v2%2E5%3A+Three+insights+towards+enhancing+aesthetic+quality+in+text%2Dto%2Dimage+generation%2E+arXiv+preprint+arXiv%3A2402%2E17245%2C+2024%2E+%E2%80%94+%5Ba)
49. J. Li, D. Li, S. Savarese, and S. Hoi. [BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/). In Proc. Int’l Conf. Machine Learning, 2023.
50. L. Li, Y. Wang, R. Xu, P. Wang, X. Feng, L. Kong, and Q. Liu. [Multimodal arXiv: A dataset for improving scientific comprehension of large vision-language models](https://md.research.elamur.ai/free/en/papers/li-2024-multimodal-arxiv-dataset-improving-scientific/). In Annual Meeting of the Association for Computational Linguistics, 2024.
51. X. Li, F. Zhang, H. Diao, Y. Wang, X. Wang, and L.-Y. Duan. DenseFusion-1M: Merging vision experts for comprehensive multimodal perception. In Proc. Annu. Conf. Neural Inf. Process. Systems, 2024. [find](https://scholar.google.com/scholar?q=X%2E+Li%2C+F%2E+Zhang%2C+H%2E+Diao%2C+Y%2E+Wang%2C+X%2E+Wang%2C+and+L%2E%2DY%2E+Duan%2E+DenseFusion%2D1M%3A+Merging+vision+experts+for+comprehensive+multimodal+perception%2E+In+Proc%2E+Annu%2E+Conf%2E+Neural+Inf%2E+Process%2E+Systems%2C+2024%2E)
52. Y. Li, Y. Du, K. Zhou, J. Wang, X. Zhao, and J.-R. Wen. [Evaluating object hallucination in large vision-language models](https://md.research.elamur.ai/free/en/papers/li-2023-evaluating-object-hallucination-large-vision/). In Proc. Conf. on Empirical Methods in Natural Language Process., 2023.
53. Z. Li, X. Yang, K. Choi, W. Zhu, R. Hsieh, H. Kim, J. H. Lim, S. Ji, B. Lee, X. Yan, et al. MMSci: A multimodal multi-discipline dataset for phd-level scientific comprehension. In AI for Accelerated Materials Design, 2024. [find](https://scholar.google.com/scholar?q=Z%2E+Li%2C+X%2E+Yang%2C+K%2E+Choi%2C+W%2E+Zhu%2C+R%2E+Hsieh%2C+H%2E+Kim%2C+J%2E+H%2E+Lim%2C+S%2E+Ji%2C+B%2E+Lee%2C+X%2E+Yan%2C+et+al%2E+MMSci%3A+A+multimodal+multi%2Ddiscipline+dataset+for+phd%2Dlevel+scientific+comprehension%2E+In+AI+for+Accelerated+M)
54. Z. Li, J. Zhang, Q. Lin, J. Xiong, Y. Long, X. Deng, Y. Zhang, X. Liu, M. Huang, Z. Xiao, et al. [Hunyuan-DiT: A powerful multi-resolution diffusion transformer with fine-grained chinese understanding](https://md.research.elamur.ai/free/en/papers/li-2024-hunyuan-dit-powerful-multi/). arXiv preprint arXiv:2405.08748, 2024.
55. Y. Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le. [Flow matching for generative modeling](https://md.research.elamur.ai/free/en/papers/lipman-2022-flow-matching-generative-modeling/). In Proc. Int’l Conf. Learning Representations, 2023.
56. H. Liu, C. Li, Y. Li, and Y. J. Lee. [Improved baselines with visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/). In Proc. IEEE Int’l Conf. Computer Vision and Pattern Recognition, 2024.
57. H. Liu, C. Li, Y. Li, B. Li, Y. Zhang, S. Shen, and Y. J. Lee. LLaVA-NeXT: Improved reasoning, OCR, and world knowledge, 2024. URL [https://llava-vl.github.io/](https://llava-vl.github.io/) blog/2024-01-30-llava-next/ . [↗](https://llava-vl.github.io/)
58. H. Liu, C. Li, Q. Wu, and Y. J. Lee. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). In Proc. Annu. Conf. Neural Inf. Process. Systems, 2024.
59. H. Liu, W. Yan, M. Zaharia, and P. Abbeel. World model on million-length video and language with ringattention. arXiv preprint arXiv:2402.08268, 2024. [find](https://scholar.google.com/scholar?q=H%2E+Liu%2C+W%2E+Yan%2C+M%2E+Zaharia%2C+and+P%2E+Abbeel%2E+World+model+on+million%2Dlength+video+and+language+with+ringattention%2E+arXiv+preprint+arXiv%3A2402%2E08268%2C+2024%2E)
60. Q. Liu. Rectified flow: A marginal preserving approach to optimal transport. arXiv preprint arXiv:2209.14577, 2022. — [arXiv:2209.14577](https://arxiv.org/abs/2209.14577) [find](https://scholar.google.com/scholar?q=Q%2E+Liu%2E+Rectified+flow%3A+A+marginal+preserving+approach+to+optimal+transport%2E+arXiv+preprint+arXiv%3A2209%2E14577%2C+2022%2E+%E2%80%94+%5BarXiv%3A2209%2E14577%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2209%2E14577%29)
61. X. Liu, C. Gong, and Q. Liu. [Flow straight and fast: Learning to generate and transfer data with rectified flow](https://md.research.elamur.ai/free/en/papers/liu-2022-flow-straight-fast-learning-generate/). In Proc. Int’l Conf. Learning Representations, 2023.
62. X. Liu, X. Zhang, J. Ma, J. Peng, et al. InstaFlow: One step is enough for high-quality diffusion-based text-to-image generation. In Proc. Int’l Conf. Learning Representations, 2024. [find](https://scholar.google.com/scholar?q=X%2E+Liu%2C+X%2E+Zhang%2C+J%2E+Ma%2C+J%2E+Peng%2C+et+al%2E+InstaFlow%3A+One+step+is+enough+for+high%2Dquality+diffusion%2Dbased+text%2Dto%2Dimage+generation%2E+In+Proc%2E+Int%E2%80%99l+Conf%2E+Learning+Representations%2C+2024%2E)
63. Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu, et al. [MMBench: Is your multi-modal model an all-around player?](https://md.research.elamur.ai/free/en/papers/liu-2023-mmbench-multi-modal-model-all/) In Proc. European Conf. Computer Vision, 2024.
64. H. Lu, W. Liu, B. Zhang, B. Wang, K. Dong, B. Liu, J. Sun, T. Ren, Z. Li, H. Yang, et al. [DeepSeek-VL: towards real-world vision-language understanding](https://md.research.elamur.ai/free/en/papers/lu-2024-deepseek-vl-real-world/). arXiv preprint arXiv:2403.05525, 2024.
65. P. Lu, L. Qiu, J. Chen, T. Xia, Y. Zhao, W. Zhang, Z. Yu, X. Liang, and S.-C. Zhu. IconQA: A new benchmark for abstract diagram understanding and visual language reasoning. In Proc. Annu. Conf. Neural Inf. Process. Systems, 2021. [find](https://scholar.google.com/scholar?q=P%2E+Lu%2C+L%2E+Qiu%2C+J%2E+Chen%2C+T%2E+Xia%2C+Y%2E+Zhao%2C+W%2E+Zhang%2C+Z%2E+Yu%2C+X%2E+Liang%2C+and+S%2E%2DC%2E+Zhu%2E+IconQA%3A+A+new+benchmark+for+abstract+diagram+understanding+and+visual+language+reasoning%2E+In+Proc%2E+Annu%2E+Conf%2E+Neural)
66. N. Ma, M. Goldstein, M. S. Albergo, N. M. Boffi, E. Vanden-Eijnden, and S. Xie. SiT: Exploring flow and diffusion-based generative models with scalable interpolant transformers. arXiv preprint arXiv:2401.08740, 2024. [find](https://scholar.google.com/scholar?q=N%2E+Ma%2C+M%2E+Goldstein%2C+M%2E+S%2E+Albergo%2C+N%2E+M%2E+Boffi%2C+E%2E+Vanden%2DEijnden%2C+and+S%2E+Xie%2E+SiT%3A+Exploring+flow+and+diffusion%2Dbased+generative+models+with+scalable+interpolant+transformers%2E+arXiv+preprint+arXiv%3A2)
67. Y. Ma, H. Yang, W. Wang, J. Fu, and J. Liu. Unified multi-modal latent diffusion for joint subject and text conditional image generation. arXiv preprint arXiv:2303.09319, 2023. — [arXiv:2303.09319](https://arxiv.org/abs/2303.09319) [find](https://scholar.google.com/scholar?q=Y%2E+Ma%2C+H%2E+Yang%2C+W%2E+Wang%2C+J%2E+Fu%2C+and+J%2E+Liu%2E+Unified+multi%2Dmodal+latent+diffusion+for+joint+subject+and+text+conditional+image+generation%2E+arXiv+preprint+arXiv%3A2303%2E09319%2C+2023%2E+%E2%80%94+%5BarXiv%3A2303%2E09319%5D%28ht)
68. madebyollin. Megalith-10M, 2024. URL [https://huggingface.co/datasets/made](https://huggingface.co/datasets/made) byollin/megalith-10m . [↗](https://huggingface.co/datasets/made)
69. B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). arXiv preprint arXiv:2005.14165, 2020.
70. A. Masry, X. L. Do, J. Q. Tan, S. Joty, and E. Hoque. [ChartQA: A benchmark for question answering about charts with visual and logical reasoning](https://md.research.elamur.ai/free/en/papers/masry-2022-chartqa-benchmark-question-answering-about/). In Annual Meeting of the Association for Computational Linguistics, 2022.
71. mehdidc. YFCC-15M, 2024. URL [https://huggingface.co/datasets/mehdidc/yf](https://huggingface.co/datasets/mehdidc/yf) cc15m . [↗](https://huggingface.co/datasets/mehdidc/yf)
72. W. Peebles and S. Xie. [Scalable diffusion models with transformers](https://md.research.elamur.ai/free/en/papers/peebles-2022-scalable-diffusion-models-transformers/). In Proc. IEEE Int. Conf. Comput. Vision, 2023.
73. D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach. [SDXL: Improving latent diffusion models for high-resolution image synthesis](https://md.research.elamur.ai/free/en/papers/podell-2023-sdxl-improving-latent-diffusion-models/). In Proc. Int’l Conf. Learning Representations, 2024.
74. PyTorch-Contributors. PyTorch, 2024. URL [https://pytorch.org](https://pytorch.org) . [↗](https://pytorch.org)
75. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). In Proc. Int’l Conf. Machine Learning, 2021.
76. A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. [Hierarchical text-conditional image generation with CLIP latents](https://md.research.elamur.ai/free/en/papers/ramesh-2022-hierarchical-text-conditional-image-generation/). arXiv preprint arXiv:2204.06125, 2022.
77. R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proc. IEEE Int’l Conf. Computer Vision and Pattern Recognition, 2022. [find](https://scholar.google.com/scholar?q=R%2E+Rombach%2C+A%2E+Blattmann%2C+D%2E+Lorenz%2C+P%2E+Esser%2C+and+B%2E+Ommer%2E+High%2Dresolution+image+synthesis+with+latent+diffusion+models%2E+In+Proc%2E+IEEE+Int%E2%80%99l+Conf%2E+Computer+Vision+and+Pattern+Recognition%2C+2022%2E)
78. L. Ruan, Y. Ma, H. Yang, H. He, B. Liu, J. Fu, N. J. Yuan, Q. Jin, and B. Guo. MM-Diffusion: Learning multi-modal diffusion models for joint audio and video generation. In Proc. IEEE Int’l Conf. Computer Vision and Pattern Recognition, 2022. [find](https://scholar.google.com/scholar?q=L%2E+Ruan%2C+Y%2E+Ma%2C+H%2E+Yang%2C+H%2E+He%2C+B%2E+Liu%2C+J%2E+Fu%2C+N%2E+J%2E+Yuan%2C+Q%2E+Jin%2C+and+B%2E+Guo%2E+MM%2DDiffusion%3A+Learning+multi%2Dmodal+diffusion+models+for+joint+audio+and+video+generation%2E+In+Proc%2E+IEEE+Int%E2%80%99l+Conf%2E+Compu)
79. C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans, et al. [Photorealistic text-to-image diffusion models with deep language understanding](https://md.research.elamur.ai/free/en/papers/saharia-2022-photorealistic-text-image-diffusion-models/). In Proc. Annu. Conf. Neural Inf. Process. Systems, 2022.
80. S. Shah, A. Mishra, N. Yadati, and P. P. Talukdar. KVQA: Knowledge-aware visual question answering. In Proc. AAAI Conf. on Artificial Intelligence, 2019. [find](https://scholar.google.com/scholar?q=S%2E+Shah%2C+A%2E+Mishra%2C+N%2E+Yadati%2C+and+P%2E+P%2E+Talukdar%2E+KVQA%3A+Knowledge%2Daware+visual+question+answering%2E+In+Proc%2E+AAAI+Conf%2E+on+Artificial+Intelligence%2C+2019%2E)
81. A. Singh, V. Natarajan, M. Shah, Y. Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach. Towards VQA models that can read. In Proc. IEEE Int’l Conf. Computer Vision and Pattern Recognition, 2019. [find](https://scholar.google.com/scholar?q=A%2E+Singh%2C+V%2E+Natarajan%2C+M%2E+Shah%2C+Y%2E+Jiang%2C+X%2E+Chen%2C+D%2E+Batra%2C+D%2E+Parikh%2C+and+M%2E+Rohrbach%2E+Towards+VQA+models+that+can+read%2E+In+Proc%2E+IEEE+Int%E2%80%99l+Conf%2E+Computer+Vision+and+Pattern+Recognition%2C+2019%2E)
82. V. Singla, K. Yue, S. Paul, R. Shirkavand, M. Jayawardhana, A. Ganjdanesh, H. Huang, A. Bhatele, G. Somepalli, and T. Goldstein. From pixels to prose: A large dataset of dense image captions. arXiv preprint arXiv:2406.10328, 2024. — [arXiv:2406.10328](https://arxiv.org/abs/2406.10328) [find](https://scholar.google.com/scholar?q=V%2E+Singla%2C+K%2E+Yue%2C+S%2E+Paul%2C+R%2E+Shirkavand%2C+M%2E+Jayawardhana%2C+A%2E+Ganjdanesh%2C+H%2E+Huang%2C+A%2E+Bhatele%2C+G%2E+Somepalli%2C+and+T%2E+Goldstein%2E+From+pixels+to+prose%3A+A+large+dataset+of+dense+image+captions%2E+arXiv+pr)
83. Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. [Score-based generative modeling through stochastic differential equations](https://md.research.elamur.ai/free/en/papers/song-2020-score-based-generative-modeling-through/). In Proc. Int’l Conf. Learning Representations, 2021.
84. K. Srinivasan, K. Raman, J. Chen, M. Bendersky, and M. Najork. WIT: Wikipedia-based image text dataset for multimodal multilingual machine learning. In Proc. ACM SIGIR Conf. Research and Develop. in Info. Retrieval, 2021. [find](https://scholar.google.com/scholar?q=K%2E+Srinivasan%2C+K%2E+Raman%2C+J%2E+Chen%2C+M%2E+Bendersky%2C+and+M%2E+Najork%2E+WIT%3A+Wikipedia%2Dbased+image+text+dataset+for+multimodal+multilingual+machine+learning%2E+In+Proc%2E+ACM+SIGIR+Conf%2E+Research+and+Develop%2E+in+I)
85. K. Sun, J. Pan, Y. Ge, H. Li, H. Duan, X. Wu, R. Zhang, A. Zhou, Z. Qin, Y. Wang, et al. JourneyDB: A benchmark for generative image understanding. In Proc. Annu. Conf. Neural Inf. Process. Systems, 2024. [find](https://scholar.google.com/scholar?q=K%2E+Sun%2C+J%2E+Pan%2C+Y%2E+Ge%2C+H%2E+Li%2C+H%2E+Duan%2C+X%2E+Wu%2C+R%2E+Zhang%2C+A%2E+Zhou%2C+Z%2E+Qin%2C+Y%2E+Wang%2C+et+al%2E+JourneyDB%3A+A+benchmark+for+generative+image+understanding%2E+In+Proc%2E+Annu%2E+Conf%2E+Neural+Inf%2E+Process%2E+Systems%2C+2)
86. P. Sun, Y. Jiang, S. Chen, S. Zhang, B. Peng, P. Luo, and Z. Yuan. Autoregressive model beats diffusion: LLaMA for scalable image generation. arXiv preprint arXiv:2406.06525, 2024. — [arXiv:2406.06525](https://arxiv.org/abs/2406.06525) [find](https://scholar.google.com/scholar?q=P%2E+Sun%2C+Y%2E+Jiang%2C+S%2E+Chen%2C+S%2E+Zhang%2C+B%2E+Peng%2C+P%2E+Luo%2C+and+Z%2E+Yuan%2E+Autoregressive+model+beats+diffusion%3A+LLaMA+for+scalable+image+generation%2E+arXiv+preprint+arXiv%3A2406%2E06525%2C+2024%2E+%E2%80%94+%5BarXiv%3A2406%2E06525)
87. Q. Sun, Q. Yu, Y. Cui, F. Zhang, X. Zhang, Y. Wang, H. Gao, J. Liu, T. Huang, and X. Wang. Generative pretraining in multimodality. In Proc. Int’l Conf. Learning Representations, 2024. [find](https://scholar.google.com/scholar?q=Q%2E+Sun%2C+Q%2E+Yu%2C+Y%2E+Cui%2C+F%2E+Zhang%2C+X%2E+Zhang%2C+Y%2E+Wang%2C+H%2E+Gao%2C+J%2E+Liu%2C+T%2E+Huang%2C+and+X%2E+Wang%2E+Generative+pretraining+in+multimodality%2E+In+Proc%2E+Int%E2%80%99l+Conf%2E+Learning+Representations%2C+2024%2E)
88. C. Team. [Chameleon: Mixed-modal early-fusion foundation models](https://md.research.elamur.ai/free/en/papers/team-2024-chameleon-mixed-modal-early-fusion/). arXiv preprint arXiv:2405.09818, 2024.
89. G. Team. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805, 2023.
90. S. Tong, E. Brown, P. Wu, S. Woo, M. Middepogu, S. C. Akula, J. Yang, S. Yang, A. Iyer, X. Pan, et al. [Cambrian-1: A fully open, vision-centric exploration of multimodal llms](https://md.research.elamur.ai/free/en/papers/tong-2024-cambrian-1-fully-open-vision/). arXiv preprint arXiv:2406.16860, 2024.
91. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. [LLaMA: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023.
92. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. [LLaMA 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023.
93. C. N. Vasconcelos, A. Rashwan, A. Waters, T. Walker, K. Xu, J. Yan, R. Qian, Y. Li, S. LUO, Y. Onoe, et al. Greedy growing enables high-resolution pixel-based diffusion models. Transactions on Machine Learning Research, 2024. [find](https://scholar.google.com/scholar?q=C%2E+N%2E+Vasconcelos%2C+A%2E+Rashwan%2C+A%2E+Waters%2C+T%2E+Walker%2C+K%2E+Xu%2C+J%2E+Yan%2C+R%2E+Qian%2C+Y%2E+Li%2C+S%2E+LUO%2C+Y%2E+Onoe%2C+et+al%2E+Greedy+growing+enables+high%2Dresolution+pixel%2Dbased+diffusion+models%2E+Transactions+on+Machine)
94. P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, et al. Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.[12191, 2024](https://md.research.elamur.ai/free/en/papers/wang-2024-qwen2-vl-enhancing-vision/).
95. X. Wang, X. Zhang, Z. Luo, Q. Sun, Y. Cui, J. Wang, F. Zhang, Y. Wang, Z. Li, Q. Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024. — [arXiv:2409.18869](https://arxiv.org/abs/2409.18869) [find](https://scholar.google.com/scholar?q=X%2E+Wang%2C+X%2E+Zhang%2C+Z%2E+Luo%2C+Q%2E+Sun%2C+Y%2E+Cui%2C+J%2E+Wang%2C+F%2E+Zhang%2C+Y%2E+Wang%2C+Z%2E+Li%2C+Q%2E+Yu%2C+et+al%2E+Emu3%3A+Next%2Dtoken+prediction+is+all+you+need%2E+arXiv+preprint+arXiv%3A2409%2E18869%2C+2024%2E+%E2%80%94+%5BarXiv%3A2409%2E18869%5D%28htt)
96. S. Woo, S. Debnath, R. Hu, X. Chen, Z. Liu, I. S. Kweon, and S. Xie. ConvNeXt v2: Co-designing and scaling ConvNets with masked autoencoders. In Proc. IEEE Int’l Conf. Computer Vision and Pattern Recognition, 2023. [find](https://scholar.google.com/scholar?q=S%2E+Woo%2C+S%2E+Debnath%2C+R%2E+Hu%2C+X%2E+Chen%2C+Z%2E+Liu%2C+I%2E+S%2E+Kweon%2C+and+S%2E+Xie%2E+ConvNeXt+v2%3A+Co%2Ddesigning+and+scaling+ConvNets+with+masked+autoencoders%2E+In+Proc%2E+IEEE+Int%E2%80%99l+Conf%2E+Computer+Vision+and+Pattern+Reco)
97. C. Wu, X. Chen, Z. Wu, Y. Ma, X. Liu, Z. Pan, W. Liu, Z. Xie, X. Yu, C. Ruan, et al. [Janus: Decoupling visual encoding for unified multimodal understanding and generation](https://md.research.elamur.ai/free/en/papers/wu-2024-janus-decoupling-visual-encoding/). arXiv preprint arXiv:2410.13848, 2024.
98. S. Wu, H. Fei, L. Qu, W. Ji, and T.-S. Chua. NExT-GPT: Any-to-any multimodal LLM. In Proc. Int’l Conf. Machine Learning, 2024. [find](https://scholar.google.com/scholar?q=S%2E+Wu%2C+H%2E+Fei%2C+L%2E+Qu%2C+W%2E+Ji%2C+and+T%2E%2DS%2E+Chua%2E+NExT%2DGPT%3A+Any%2Dto%2Dany+multimodal+LLM%2E+In+Proc%2E+Int%E2%80%99l+Conf%2E+Machine+Learning%2C+2024%2E)
99. Y. Wu, Z. Zhang, J. Chen, H. Tang, D. Li, Y. Fang, L. Zhu, E. Xie, H. Yin, L. Yi, et al. VILA-U: A unified foundation model integrating visual understanding and generation. arXiv preprint arXiv:2409.04429, 2024. — [arXiv:2409.04429](https://arxiv.org/abs/2409.04429) [find](https://scholar.google.com/scholar?q=Y%2E+Wu%2C+Z%2E+Zhang%2C+J%2E+Chen%2C+H%2E+Tang%2C+D%2E+Li%2C+Y%2E+Fang%2C+L%2E+Zhu%2C+E%2E+Xie%2C+H%2E+Yin%2C+L%2E+Yi%2C+et+al%2E+VILA%2DU%3A+A+unified+foundation+model+integrating+visual+understanding+and+generation%2E+arXiv+preprint+arXiv%3A2409%2E0)
100. J. Xie, W. Mao, Z. Bai, D. J. Zhang, W. Wang, K. Q. Lin, Y. Gu, Z. Chen, Z. Yang, and M. Z. Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528, 2024. — [arXiv:2408.12528](https://arxiv.org/abs/2408.12528) [find](https://scholar.google.com/scholar?q=J%2E+Xie%2C+W%2E+Mao%2C+Z%2E+Bai%2C+D%2E+J%2E+Zhang%2C+W%2E+Wang%2C+K%2E+Q%2E+Lin%2C+Y%2E+Gu%2C+Z%2E+Chen%2C+Z%2E+Yang%2C+and+M%2E+Z%2E+Shou%2E+Show%2Do%3A+One+single+transformer+to+unify+multimodal+understanding+and+generation%2E+arXiv+preprint+arXiv%3A)
101. H. Ye, D.-A. Huang, Y. Lu, Z. Yu, W. Ping, A. Tao, J. Kautz, S. Han, D. Xu, P. Molchanov, et al. X-VILA: Cross-modality alignment for large language model. arXiv preprint arXiv:2405.19335, 2024. — [arXiv:2405.19335](https://arxiv.org/abs/2405.19335) [find](https://scholar.google.com/scholar?q=H%2E+Ye%2C+D%2E%2DA%2E+Huang%2C+Y%2E+Lu%2C+Z%2E+Yu%2C+W%2E+Ping%2C+A%2E+Tao%2C+J%2E+Kautz%2C+S%2E+Han%2C+D%2E+Xu%2C+P%2E+Molchanov%2C+et+al%2E+X%2DVILA%3A+Cross%2Dmodality+alignment+for+large+language+model%2E+arXiv+preprint+arXiv%3A2405%2E19335%2C+2024%2E+%E2%80%94+%5Bar)
102. L. Yu, J. Lezama, N. B. Gundavarapu, L. Versari, K. Sohn, D. Minnen, Y. Cheng, A. Gupta, X. Gu, A. G. Hauptmann, et al. Language model beats diffusion-tokenizer is key to visual generation. In Proc. Int’l Conf. Learning Representations, 2024. [find](https://scholar.google.com/scholar?q=L%2E+Yu%2C+J%2E+Lezama%2C+N%2E+B%2E+Gundavarapu%2C+L%2E+Versari%2C+K%2E+Sohn%2C+D%2E+Minnen%2C+Y%2E+Cheng%2C+A%2E+Gupta%2C+X%2E+Gu%2C+A%2E+G%2E+Hauptmann%2C+et+al%2E+Language+model+beats+diffusion%2Dtokenizer+is+key+to+visual+generation%2E+In+Proc%2E+I)
103. S. Yu, S. Kwak, H. Jang, J. Jeong, J. Huang, J. Shin, and S. Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. arXiv preprint arXiv:2410.06940, 2024. [find](https://scholar.google.com/scholar?q=S%2E+Yu%2C+S%2E+Kwak%2C+H%2E+Jang%2C+J%2E+Jeong%2C+J%2E+Huang%2C+J%2E+Shin%2C+and+S%2E+Xie%2E+Representation+alignment+for+generation%3A+Training+diffusion+transformers+is+easier+than+you+think%2E+arXiv+preprint+arXiv%3A2410%2E06940%2C+20)
104. W. Yu, Z. Yang, L. Li, J. Wang, K. Lin, Z. Liu, X. Wang, and L. Wang. [MM-Vet: Evaluating large multimodal models for integrated capabilities](https://md.research.elamur.ai/free/en/papers/yu-2023-mm-vet-evaluating-large-multimodal/). In Proc. Int’l Conf. Machine Learning, 2024.
105. X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, et al. [MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/). In Proc. IEEE Int’l Conf. Computer Vision and Pattern Recognition, 2024.
106. X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. [Sigmoid loss for language image pre-training](https://md.research.elamur.ai/free/en/papers/zhai-2023-sigmoid-loss-language-image-pre/). In Proc. IEEE Int. Conf. Comput. Vision, 2023.
107. C. Zhao, Y. Song, W. Wang, H. Feng, E. Ding, Y. Sun, X. Xiao, and J. Wang. MonoFormer: One transformer for both diffusion and autoregression. arXiv preprint arXiv:2409.16280, 2024. — [arXiv:2409.16280](https://arxiv.org/abs/2409.16280) [find](https://scholar.google.com/scholar?q=C%2E+Zhao%2C+Y%2E+Song%2C+W%2E+Wang%2C+H%2E+Feng%2C+E%2E+Ding%2C+Y%2E+Sun%2C+X%2E+Xiao%2C+and+J%2E+Wang%2E+MonoFormer%3A+One+transformer+for+both+diffusion+and+autoregression%2E+arXiv+preprint+arXiv%3A2409%2E16280%2C+2024%2E+%E2%80%94+%5BarXiv%3A2409%2E16280)
108. C. Zhou, L. Yu, A. Babu, K. Tirumala, M. Yasunaga, L. Shamis, J. Kahn, X. Ma, L. Zettlemoyer, and O. Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.11039, 2024. [find](https://scholar.google.com/scholar?q=C%2E+Zhou%2C+L%2E+Yu%2C+A%2E+Babu%2C+K%2E+Tirumala%2C+M%2E+Yasunaga%2C+L%2E+Shamis%2C+J%2E+Kahn%2C+X%2E+Ma%2C+L%2E+Zettlemoyer%2C+and+O%2E+Levy%2E+Transfusion%3A+Predict+the+next+token+and+diffuse+images+with+one+multi%2Dmodal+model%2E+arXiv+prep)
109. Y. Zhu, M. Zhu, N. Liu, Z. Ou, X. Mou, and J. Tang. LLaVA-Phi: Efficient multi-modal assistant with small language model. arXiv preprint arXiv:2401.02330, 2024. — [arXiv:2401.02330](https://arxiv.org/abs/2401.02330) [find](https://scholar.google.com/scholar?q=Y%2E+Zhu%2C+M%2E+Zhu%2C+N%2E+Liu%2C+Z%2E+Ou%2C+X%2E+Mou%2C+and+J%2E+Tang%2E+LLaVA%2DPhi%3A+Efficient+multi%2Dmodal+assistant+with+small+language+model%2E+arXiv+preprint+arXiv%3A2401%2E02330%2C+2024%2E+%E2%80%94+%5BarXiv%3A2401%2E02330%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
110. L. Zhuo, R. Du, H. Xiao, Y. Li, D. Liu, R. Huang, W. Liu, L. Zhao, F.-Y. Wang, Z. Ma, et al. Lumina-Next: Making Lumina-T2X stronger and faster with Next-DiT. arXiv preprint arXiv:2406.18583, 2024. — [arXiv:2406.18583](https://arxiv.org/abs/2406.18583) [find](https://scholar.google.com/scholar?q=L%2E+Zhuo%2C+R%2E+Du%2C+H%2E+Xiao%2C+Y%2E+Li%2C+D%2E+Liu%2C+R%2E+Huang%2C+W%2E+Liu%2C+L%2E+Zhao%2C+F%2E%2DY%2E+Wang%2C+Z%2E+Ma%2C+et+al%2E+Lumina%2DNext%3A+Making+Lumina%2DT2X+stronger+and+faster+with+Next%2DDiT%2E+arXiv+preprint+arXiv%3A2406%2E18583%2C+2024%2E+%E2%80%94+)

*Updated 29 October 2026*