---
title: "CogVideo — paper by Hong et al. (2022) — summary and ideas"
description: "The authors also introduce multi-frame-rate hierarchical training. 37 references, cited by 60 papers here."
authors: ["Wenyi Hong", "Ming Ding", "Wendi Zheng", "Xinghan Liu", "Jie Tang"]
year: 2022
venue: "arXiv:2205.15868"
source: "https://arxiv.org/abs/2205.15868"
doi: "10.48550/arxiv.2205.15868"
arxiv: "2205.15868"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/hong-2022-cogvideo-large-scale-pretraining/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hong-2022-cogvideo-large-scale-pretraining/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/2/) / CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers

# CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers

**Authors:** [Wenyi Hong](https://md.research.elamur.ai/free/en/authors/wenyi-hong/), [Ming Ding](https://md.research.elamur.ai/free/en/authors/ming-ding/), [Wendi Zheng](https://md.research.elamur.ai/free/en/authors/wendi-zheng/), Xinghan Liu and [Jie Tang](https://md.research.elamur.ai/free/en/authors/jie-tang/) · **Year:** 2022 · **Venue:** arXiv:2205.15868 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2205.15868) · [PDF](https://arxiv.org/pdf/2205.15868) · [DOI 10.48550/arxiv.2205.15868](https://doi.org/10.48550/arxiv.2205.15868) · [arXiv 2205.15868](https://arxiv.org/abs/2205.15868) · [OpenAlex](https://openalex.org/W4281632497)

**Citations:** 116 citations (OpenAlex, 2026-10-14) · [60](https://md.research.elamur.ai/free/en/papers/hong-2022-cogvideo-large-scale-pretraining/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Autoregressive transformers work well for text-to-image generation, but extending them to video is difficult. Training from scratch is very expensive, and paired text-video data is scarce and often only loosely matched. Fixed-length clipping also separates captions from the actions they describe. CogVideo addresses these problems with a 9.4B-parameter transformer that starts from the pretrained text-to-image model CogView2 instead of random weights. It uses dual-channel attention to add temporal modeling on top of the inherited image knowledge.

The authors also introduce multi-frame-rate hierarchical training. A frame-rate token is prepended to the text, and frames are sampled at the slowest rate that still yields five frames, so each training sample covers a whole action. One model generates key frames in order. A second model, sharing the same CogLM structure with a different attention mask, recursively fills in the frames between them. Swin-style attention, extended to autoregressive generation, reduces compute and memory costs.

CogVideo produces 480×480 clips and beats publicly available systems in both automatic and human evaluation. It is presented as the first open-source large-scale pretrained general-domain text-to-video model.

## Key points
- **Scale and output:** CogVideo has 9.4B parameters and was trained on 5.4M text-video pairs. It generates 4-second, 32-frame clips at 480×480, and its prompts are in Chinese.
- **Data scarcity:** The authors cite VATEX, the largest annotated text-video dataset, at only 41,250 videos. Retrieval-based sources such as Howto100M are only weakly relevant to their captions.
- **Tokenization:** A VQVAE tokenizer turns each frame into 20×20 = 400 tokens, and each training sample contains 5 frames.
- **Hierarchy ablation:** On a 5,000-sample Kinetics-600 test subset (FVD, lower is better), CogVideo scores 108.27. One-stage sliding-window generation scores 137.13 with N_overlap = 1 and 120.82 with N_overlap = 2.
- **Initialization ablation:** On the same subset, a model initialized from CogView2 but without video pretraining scores 124.92, and a randomly initialized model scores 166.13. CogView2 weights also make the training loss fall faster. Code and models are released at github.com/THUDM/CogVideo.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works

## Cite

Cite as: Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu and Jie Tang. 2022. CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers. arXiv:2205.15868.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{hong2022cogvideo,
  title = {{CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers}},
  author = {Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu and Jie Tang},
  year = {2022},
  doi = {10.48550/arxiv.2205.15868},
  eprint = {2205.15868},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2205.15868}
}
```

- [← previous LILA: A Unified Benchmark for Mathematical Reasoning](https://md.research.elamur.ai/free/en/papers/mishra-2022-lila-unified-benchmark-mathematical-reasoning/)
- [next → Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofs](https://md.research.elamur.ai/free/en/papers/jiang-2022-draft-sketch-prove-guiding-formal/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/2/)

## References

37 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). arXiv preprint arXiv:2005.14165, 2020.
2. J. Carreira and A. Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299–6308, 2017. [find](https://scholar.google.com/scholar?q=J%2E+Carreira+and+A%2E+Zisserman%2E+Quo+vadis%2C+action+recognition%3F+a+new+model+and+the+kinetics+dataset%2E+In+proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+pages+6299%E2%80%936308%2C+20)
3. J. Carreira, E. Noland, A. Banki-Horvath, C. Hillier, and A. Zisserman. A short note about kinetics-600. arXiv preprint arXiv:1808.01340, 2018. — [arXiv:1808.01340](https://arxiv.org/abs/1808.01340) [find](https://scholar.google.com/scholar?q=J%2E+Carreira%2C+E%2E+Noland%2C+A%2E+Banki%2DHorvath%2C+C%2E+Hillier%2C+and+A%2E+Zisserman%2E+A+short+note+about+kinetics%2D600%2E+arXiv+preprint+arXiv%3A1808%2E01340%2C+2018%2E+%E2%80%94+%5BarXiv%3A1808%2E01340%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1808%2E01340%29)
4. A. Clark, J. Donahue, and K. Simonyan. Adversarial video generation on complex datasets. arXiv preprint arXiv:1907.06571, 2019. — [arXiv:1907.06571](https://arxiv.org/abs/1907.06571) [find](https://scholar.google.com/scholar?q=A%2E+Clark%2C+J%2E+Donahue%2C+and+K%2E+Simonyan%2E+Adversarial+video+generation+on+complex+datasets%2E+arXiv+preprint+arXiv%3A1907%2E06571%2C+2019%2E+%E2%80%94+%5BarXiv%3A1907%2E06571%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1907%2E06571%29)
5. M. Ding, Z. Yang, W. Hong, W. Zheng, C. Zhou, D. Yin, J. Lin, X. Zou, Z. Shao, H. Yang, et al. [Cogview: Mastering text-to-image generation via transformers](https://md.research.elamur.ai/free/en/papers/ding-2021-cogview-mastering-text-image/). Advances in Neural Information Processing Systems, 34, 2021.
6. M. Ding, W. Zheng, W. Hong, and J. Tang. Cogview2: Faster and better text-to-image generation via hierarchical transformers. arXiv preprint arXiv:2204.14217, 2022. — [arXiv:2204.14217](https://arxiv.org/abs/2204.14217) [find](https://scholar.google.com/scholar?q=M%2E+Ding%2C+W%2E+Zheng%2C+W%2E+Hong%2C+and+J%2E+Tang%2E+Cogview2%3A+Faster+and+better+text%2Dto%2Dimage+generation+via+hierarchical+transformers%2E+arXiv+preprint+arXiv%3A2204%2E14217%2C+2022%2E+%E2%80%94+%5BarXiv%3A2204%2E14217%5D%28https%3A%2F%2Farxiv%2Eo)
7. P. Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis. arXiv preprint arXiv:2012.09841, 2020. — [arXiv:2012.09841](https://arxiv.org/abs/2012.09841) [find](https://scholar.google.com/scholar?q=P%2E+Esser%2C+R%2E+Rombach%2C+and+B%2E+Ommer%2E+Taming+transformers+for+high%2Dresolution+image+synthesis%2E+arXiv+preprint+arXiv%3A2012%2E09841%2C+2020%2E+%E2%80%94+%5BarXiv%3A2012%2E09841%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2012%2E09841%29)
8. C. Finn, I. Goodfellow, and S. Levine. Unsupervised learning for physical interaction through video prediction. Advances in neural information processing systems, 29, 2016. [find](https://scholar.google.com/scholar?q=C%2E+Finn%2C+I%2E+Goodfellow%2C+and+S%2E+Levine%2E+Unsupervised+learning+for+physical+interaction+through+video+prediction%2E+Advances+in+neural+information+processing+systems%2C+29%2C+2016%2E)
9. S. Ge, T. Hayes, H. Yang, X. Yin, G. Pang, D. Jacobs, J.-B. Huang, and D. Parikh. Long video generation with time-agnostic vqgan and time-sensitive transformer. arXiv preprint arXiv:2204.03638, 2022. — [arXiv:2204.03638](https://arxiv.org/abs/2204.03638) [find](https://scholar.google.com/scholar?q=S%2E+Ge%2C+T%2E+Hayes%2C+H%2E+Yang%2C+X%2E+Yin%2C+G%2E+Pang%2C+D%2E+Jacobs%2C+J%2E%2DB%2E+Huang%2C+and+D%2E+Parikh%2E+Long+video+generation+with+time%2Dagnostic+vqgan+and+time%2Dsensitive+transformer%2E+arXiv+preprint+arXiv%3A2204%2E03638%2C+2022%2E+)
10. I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. arXiv preprint arXiv:1406.2661, 2014. [find](https://scholar.google.com/scholar?q=I%2E+J%2E+Goodfellow%2C+J%2E+Pouget%2DAbadie%2C+M%2E+Mirza%2C+B%2E+Xu%2C+D%2E+Warde%2DFarley%2C+S%2E+Ozair%2C+A%2E+Courville%2C+and+Y%2E+Bengio%2E+Generative+adversarial+networks%2E+arXiv+preprint+arXiv%3A1406%2E2661%2C+2014%2E)
11. J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet. Video diffusion models. arXiv preprint arXiv:2204.03458, 2022. — [arXiv:2204.03458](https://arxiv.org/abs/2204.03458) [find](https://scholar.google.com/scholar?q=J%2E+Ho%2C+T%2E+Salimans%2C+A%2E+Gritsenko%2C+W%2E+Chan%2C+M%2E+Norouzi%2C+and+D%2E+J%2E+Fleet%2E+Video+diffusion+models%2E+arXiv+preprint+arXiv%3A2204%2E03458%2C+2022%2E+%E2%80%94+%5BarXiv%3A2204%2E03458%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2204%2E03458%29)
12. A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei. Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1725–1732, 2014. [find](https://scholar.google.com/scholar?q=A%2E+Karpathy%2C+G%2E+Toderici%2C+S%2E+Shetty%2C+T%2E+Leung%2C+R%2E+Sukthankar%2C+and+L%2E+Fei%2DFei%2E+Large%2Dscale+video+classification+with+convolutional+neural+networks%2E+In+Proceedings+of+the+IEEE+conference+on+Computer+Vis)
13. J. Lin, C. Gan, and S. Han. Tsm: Temporal shift module for efficient video understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7083–7093, 2019. [find](https://scholar.google.com/scholar?q=J%2E+Lin%2C+C%2E+Gan%2C+and+S%2E+Han%2E+Tsm%3A+Temporal+shift+module+for+efficient+video+understanding%2E+In+Proceedings+of+the+IEEE%2FCVF+International+Conference+on+Computer+Vision%2C+pages+7083%E2%80%937093%2C+2019%2E)
14. Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. [Swin transformer: Hierarchical vision transformer using shifted windows](https://md.research.elamur.ai/free/en/papers/liu-2021-swin-transformer-hierarchical-vision-transformer/). In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012–10022, 2021.
15. P. Luc, A. Clark, S. Dieleman, D. d. L. Casas, Y. Doron, A. Cassirer, and K. Simonyan. Transformation-based adversarial video prediction on large-scale data. arXiv preprint arXiv:2003.04035, 2020. — [arXiv:2003.04035](https://arxiv.org/abs/2003.04035) [find](https://scholar.google.com/scholar?q=P%2E+Luc%2C+A%2E+Clark%2C+S%2E+Dieleman%2C+D%2E+d%2E+L%2E+Casas%2C+Y%2E+Doron%2C+A%2E+Cassirer%2C+and+K%2E+Simonyan%2E+Transformation%2Dbased+adversarial+video+prediction+on+large%2Dscale+data%2E+arXiv+preprint+arXiv%3A2003%2E04035%2C+2020%2E+%E2%80%94+%5B)
16. A. Miech, D. Zhukov, J.-B. Alayrac, M. Tapaswi, I. Laptev, and J. Sivic. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2630–2640, 2019. [find](https://scholar.google.com/scholar?q=A%2E+Miech%2C+D%2E+Zhukov%2C+J%2E%2DB%2E+Alayrac%2C+M%2E+Tapaswi%2C+I%2E+Laptev%2C+and+J%2E+Sivic%2E+Howto100m%3A+Learning+a+text%2Dvideo+embedding+by+watching+hundred+million+narrated+video+clips%2E+In+Proceedings+of+the+IEEE%2FCVF+Int)
17. R. Rakhimov, D. Volkhonskiy, A. Artemov, D. Zorin, and E. Burnaev. Latent video transformer. arXiv preprint arXiv:2006.10704, 2020. — [arXiv:2006.10704](https://arxiv.org/abs/2006.10704) [find](https://scholar.google.com/scholar?q=R%2E+Rakhimov%2C+D%2E+Volkhonskiy%2C+A%2E+Artemov%2C+D%2E+Zorin%2C+and+E%2E+Burnaev%2E+Latent+video+transformer%2E+arXiv+preprint+arXiv%3A2006%2E10704%2C+2020%2E+%E2%80%94+%5BarXiv%3A2006%2E10704%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2006%2E10704%29)
18. A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. [Zero-shot text-to-image generation](https://md.research.elamur.ai/free/en/papers/ramesh-2021-zero-shot-text-image-generation/). arXiv preprint arXiv:2102.12092, 2021.
19. M. Saito, E. Matsumoto, and S. Saito. Temporal generative adversarial nets with singular value clipping. In Proceedings of the IEEE international conference on computer vision, pages 2830–2839, 2017. [find](https://scholar.google.com/scholar?q=M%2E+Saito%2C+E%2E+Matsumoto%2C+and+S%2E+Saito%2E+Temporal+generative+adversarial+nets+with+singular+value+clipping%2E+In+Proceedings+of+the+IEEE+international+conference+on+computer+vision%2C+pages+2830%E2%80%932839%2C+2017%2E)
20. M. Saito, S. Saito, M. Koyama, and S. Kobayashi. Train sparsely, generate densely: Memory-efficient unsupervised training of high-resolution temporal gan. International Journal of Computer Vision, 128(10):2586–2606, 2020. [find](https://scholar.google.com/scholar?q=M%2E+Saito%2C+S%2E+Saito%2C+M%2E+Koyama%2C+and+S%2E+Kobayashi%2E+Train+sparsely%2C+generate+densely%3A+Memory%2Defficient+unsupervised+training+of+high%2Dresolution+temporal+gan%2E+International+Journal+of+Computer+Vision%2C+128)
21. T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. [Improved techniques for training gans](https://md.research.elamur.ai/free/en/papers/salimans-2016-improved-techniques-training-gans/). In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 2234–2242, 2016.
22. K. Soomro, A. R. Zamir, and M. Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012. — [arXiv:1212.0402](https://arxiv.org/abs/1212.0402) [find](https://scholar.google.com/scholar?q=K%2E+Soomro%2C+A%2E+R%2E+Zamir%2C+and+M%2E+Shah%2E+Ucf101%3A+A+dataset+of+101+human+actions+classes+from+videos+in+the+wild%2E+arXiv+preprint+arXiv%3A1212%2E0402%2C+2012%2E+%E2%80%94+%5BarXiv%3A1212%2E0402%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1212%2E0402%29)
23. I. Sutskever, J. Martens, and G. Hinton. Generating text with recurrent neural networks. In ICML’11, page 1017–1024, 2011. [find](https://scholar.google.com/scholar?q=I%2E+Sutskever%2C+J%2E+Martens%2C+and+G%2E+Hinton%2E+Generating+text+with+recurrent+neural+networks%2E+In+ICML%E2%80%9911%2C+page+1017%E2%80%931024%2C+2011%2E)
24. Y. Tian, J. Ren, M. Chai, K. Olszewski, X. Peng, D. N. Metaxas, and S. Tulyakov. A good image generator is what you need for high-resolution video synthesis. arXiv preprint arXiv:2104.15069, 2021. — [arXiv:2104.15069](https://arxiv.org/abs/2104.15069) [find](https://scholar.google.com/scholar?q=Y%2E+Tian%2C+J%2E+Ren%2C+M%2E+Chai%2C+K%2E+Olszewski%2C+X%2E+Peng%2C+D%2E+N%2E+Metaxas%2C+and+S%2E+Tulyakov%2E+A+good+image+generator+is+what+you+need+for+high%2Dresolution+video+synthesis%2E+arXiv+preprint+arXiv%3A2104%2E15069%2C+2021%2E+%E2%80%94+%5B)
25. D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 4489–4497, 2015. [find](https://scholar.google.com/scholar?q=D%2E+Tran%2C+L%2E+Bourdev%2C+R%2E+Fergus%2C+L%2E+Torresani%2C+and+M%2E+Paluri%2E+Learning+spatiotemporal+features+with+3d+convolutional+networks%2E+In+Proceedings+of+the+IEEE+international+conference+on+computer+vision%2C+pa)
26. S. Tulyakov, M.-Y. Liu, X. Yang, and J. Kautz. Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1526–1535, 2018. [find](https://scholar.google.com/scholar?q=S%2E+Tulyakov%2C+M%2E%2DY%2E+Liu%2C+X%2E+Yang%2C+and+J%2E+Kautz%2E+Mocogan%3A+Decomposing+motion+and+content+for+video+generation%2E+In+Proceedings+of+the+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+pages+152)
27. T. Unterthiner, S. van Steenkiste, K. Kurach, R. Marinier, M. Michalski, and S. Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018. — [arXiv:1812.01717](https://arxiv.org/abs/1812.01717) [find](https://scholar.google.com/scholar?q=T%2E+Unterthiner%2C+S%2E+van+Steenkiste%2C+K%2E+Kurach%2C+R%2E+Marinier%2C+M%2E+Michalski%2C+and+S%2E+Gelly%2E+Towards+accurate+generative+models+of+video%3A+A+new+metric+%26+challenges%2E+arXiv+preprint+arXiv%3A1812%2E01717%2C+2018%2E+%E2%80%94+)
28. A. van den Oord, O. Vinyals, and K. Kavukcuoglu. Neural discrete representation learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 6309–6318, 2017. [find](https://scholar.google.com/scholar?q=A%2E+van+den+Oord%2C+O%2E+Vinyals%2C+and+K%2E+Kavukcuoglu%2E+Neural+discrete+representation+learning%2E+In+Proceedings+of+the+31st+International+Conference+on+Neural+Information+Processing+Systems%2C+pages+6309%E2%80%936318%2C)
29. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). arXiv preprint arXiv:1706.03762, 2017.
30. C. Vondrick, H. Pirsiavash, and A. Torralba. Generating videos with scene dynamics. Advances in neural information processing systems, 29, 2016. [find](https://scholar.google.com/scholar?q=C%2E+Vondrick%2C+H%2E+Pirsiavash%2C+and+A%2E+Torralba%2E+Generating+videos+with+scene+dynamics%2E+Advances+in+neural+information+processing+systems%2C+29%2C+2016%2E)
31. X. Wang, J. Wu, J. Chen, L. Li, Y.-F. Wang, and W. Y. Wang. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4581–4591, 2019. [find](https://scholar.google.com/scholar?q=X%2E+Wang%2C+J%2E+Wu%2C+J%2E+Chen%2C+L%2E+Li%2C+Y%2E%2DF%2E+Wang%2C+and+W%2E+Y%2E+Wang%2E+Vatex%3A+A+large%2Dscale%2C+high%2Dquality+multilingual+dataset+for+video%2Dand%2Dlanguage+research%2E+In+Proceedings+of+the+IEEE%2FCVF+International+Confer)
32. Y. Wang, M. Long, J. Wang, Z. Gao, and P. S. Yu. Predrnn: Recurrent neural networks for predictive learning using spatiotemporal lstms. Advances in neural information processing systems, 30, 2017. [find](https://scholar.google.com/scholar?q=Y%2E+Wang%2C+M%2E+Long%2C+J%2E+Wang%2C+Z%2E+Gao%2C+and+P%2E+S%2E+Yu%2E+Predrnn%3A+Recurrent+neural+networks+for+predictive+learning+using+spatiotemporal+lstms%2E+Advances+in+neural+information+processing+systems%2C+30%2C+2017%2E)
33. D. Weissenborn, O. Täckström, and J. Uszkoreit. Scaling autoregressive video models. arXiv preprint arXiv:1906.02634, 2019. — [arXiv:1906.02634](https://arxiv.org/abs/1906.02634) [find](https://scholar.google.com/scholar?q=D%2E+Weissenborn%2C+O%2E+T%C3%A4ckstr%C3%B6m%2C+and+J%2E+Uszkoreit%2E+Scaling+autoregressive+video+models%2E+arXiv+preprint+arXiv%3A1906%2E02634%2C+2019%2E+%E2%80%94+%5BarXiv%3A1906%2E02634%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1906%2E02634%29)
34. C. Wu, L. Huang, Q. Zhang, B. Li, L. Ji, F. Yang, G. Sapiro, and N. Duan. Godiva: Generating open-domain videos from natural descriptions. arXiv preprint arXiv:2104.14806, 2021. — [arXiv:2104.14806](https://arxiv.org/abs/2104.14806) [find](https://scholar.google.com/scholar?q=C%2E+Wu%2C+L%2E+Huang%2C+Q%2E+Zhang%2C+B%2E+Li%2C+L%2E+Ji%2C+F%2E+Yang%2C+G%2E+Sapiro%2C+and+N%2E+Duan%2E+Godiva%3A+Generating+open%2Ddomain+videos+from+natural+descriptions%2E+arXiv+preprint+arXiv%3A2104%2E14806%2C+2021%2E+%E2%80%94+%5BarXiv%3A2104%2E14806%5D%28h)
35. C. Wu, J. Liang, L. Ji, F. Yang, Y. Fang, D. Jiang, and N. Duan. N\\" uwa: Visual synthesis pre-training for neural visual world creation. arXiv preprint arXiv:2111.12417, 2021. — [arXiv:2111.12417](https://arxiv.org/abs/2111.12417) [find](https://scholar.google.com/scholar?q=C%2E+Wu%2C+J%2E+Liang%2C+L%2E+Ji%2C+F%2E+Yang%2C+Y%2E+Fang%2C+D%2E+Jiang%2C+and+N%2E+Duan%2E+N%5C%5C%22+uwa%3A+Visual+synthesis+pre%2Dtraining+for+neural+visual+world+creation%2E+arXiv+preprint+arXiv%3A2111%2E12417%2C+2021%2E+%E2%80%94+%5BarXiv%3A2111%2E12417%5D%28h)
36. W. Yan, Y. Zhang, P. Abbeel, and A. Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104.10157, 2021. — [arXiv:2104.10157](https://arxiv.org/abs/2104.10157) [find](https://scholar.google.com/scholar?q=W%2E+Yan%2C+Y%2E+Zhang%2C+P%2E+Abbeel%2C+and+A%2E+Srinivas%2E+Videogpt%3A+Video+generation+using+vq%2Dvae+and+transformers%2E+arXiv+preprint+arXiv%3A2104%2E10157%2C+2021%2E+%E2%80%94+%5BarXiv%3A2104%2E10157%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2104%2E10157%29)
37. S. Yu, J. Tack, S. Mo, H. Kim, J. Kim, J.-W. Ha, and J. Shin. Generating videos with dynamics-aware implicit generative adversarial networks. arXiv preprint arXiv:2202.10571, 2022. — [arXiv:2202.10571](https://arxiv.org/abs/2202.10571) [find](https://scholar.google.com/scholar?q=S%2E+Yu%2C+J%2E+Tack%2C+S%2E+Mo%2C+H%2E+Kim%2C+J%2E+Kim%2C+J%2E%2DW%2E+Ha%2C+and+J%2E+Shin%2E+Generating+videos+with+dynamics%2Daware+implicit+generative+adversarial+networks%2E+arXiv+preprint+arXiv%3A2202%2E10571%2C+2022%2E+%E2%80%94+%5BarXiv%3A2202%2E10571)

*Updated 24 October 2026*