---
title: "From Slow Bidirectional to Fast Autoregressive Video Diffusion"
description: "The authors present CausVid, which converts a pretrained bidirectional diffusion transformer into a… 10 references, cited by 102 papers here."
authors: ["Tianwei Yin", "Qiang Zhang", "Richard Zhang", "William T. Freeman", "Fredo Durand", "Eli Shechtman", "Xun Huang"]
year: 2024
venue: "arXiv:2412.07772"
source: "https://arxiv.org/abs/2412.07772"
doi: "10.48550/arxiv.2412.07772"
arxiv: "2412.07772"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yin-2024-slow-bidirectional-fast-autoregressive-video/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yin-2024-slow-bidirectional-fast-autoregressive-video/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / From Slow Bidirectional to Fast Autoregressive Video Diffusion Models

# From Slow Bidirectional to Fast Autoregressive Video Diffusion Models

**Authors:** Tianwei Yin, [Qiang Zhang](https://md.research.elamur.ai/free/en/authors/qiang-zhang/), Richard Zhang, William T. Freeman, Fredo Durand, [Eli Shechtman](https://md.research.elamur.ai/free/en/authors/eli-shechtman/) and [Xun Huang](https://md.research.elamur.ai/free/en/authors/xun-huang/) · **Year:** 2024 · **Venue:** [arXiv:2412.07772](https://md.research.elamur.ai/free/en/venues/ieee-computer-society-conference-on-computer-vision-and-pattern-recognition/) ([IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2024](https://md.research.elamur.ai/free/en/venues/ieee-computer-society-conference-on-computer-vision-and-pattern-recognition/2024/)) · **Publisher:** [Institute of Electrical and Electronics Engineers](https://md.research.elamur.ai/free/en/publishers/institute-of-electrical-and-electronics-engineers/) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2412.07772) · [PDF](https://arxiv.org/pdf/2412.07772) · [DOI 10.48550/arxiv.2412.07772](https://doi.org/10.48550/arxiv.2412.07772) · [arXiv 2412.07772](https://arxiv.org/abs/2412.07772) · [OpenAlex](https://openalex.org/W4413144436)

**Citations:** 46 citations (OpenAlex, 2026-10-14) · [102](https://md.research.elamur.ai/free/en/papers/yin-2024-slow-bidirectional-fast-autoregressive-video/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most strong text-to-video diffusion models use bidirectional attention across all frames. As a result, no frame can be shown until the whole clip is denoised, and cost grows quadratically with length. This rules out streaming or interactive use. The authors present CausVid, which converts a pretrained bidirectional diffusion transformer into a frame-by-frame autoregressive generator with causal attention and KV caching. To cut the number of denoising steps, they adapt distribution matching distillation (DMD) from images to video, compressing a 50-step model into a 4-step one. Two ingredients stabilize training. First, the student is initialized by regressing onto ODE trajectories produced by the teacher. Second, the supervising teacher stays bidirectional while the student is causal. This asymmetric setup reduces drift over long rollouts, so the model can generate videos far longer than its training clips. CausVid scores 84.27 on VBench-Long, ahead of earlier models, and streams video at about 9.4 FPS on one GPU after a 1.3-second start. This shows that autoregressive video generation can match bidirectional quality while becoming interactive.

## Key points
- Latency comparison: the bidirectional teacher needs 219 s to produce a 128-frame video, whereas CausVid shows its first frames after 1.3 s and then streams at about 9.4 FPS.
- Distillation reduces sampling from 50 steps to 4 via DMD. In the ablation, the setup with ODE initialization and a bidirectional teacher gave the best few-step scores (temporal quality 94.7, frame quality 64.4, text alignment 30.1).
- With no extra training, the same model handles image-to-video, streaming video-to-video translation and dynamic prompting.
  - **Image-to-video:** it beats CogVideoX and Pyramid Flow on VBench-I2V.
  - **Video-to-video:** it beats StreamV2V on 60 DAVIS videos.
- The model produces good quality up to about 30 s and has shown a 14-minute sample with only mild overexposure.
- The authors list several limitations:
  - quality still degrades on extremely long outputs;
  - latency is bounded by a VAE that needs five latent frames before decoding;
  - DMD's reverse-KL objective reduces output diversity;
  - throughput is around 10 FPS without engineering optimizations.

## Interesting ideas
- **Asymmetric teacher–student distillation:** a student with a restricted architecture (causal) learns from a teacher with full context (bidirectional). Because DMD matches distributions rather than trajectories, the two architectures need not agree. The same trick could transfer to other settings where a fast streaming model must learn from a slower full-context model, for example streaming speech or text models distilled from bidirectional ones.
- **Distribution-level supervision against error accumulation:** matching the output distribution of a strong teacher, rather than only next-step targets, kept long autoregressive rollouts from drifting. This is a useful lens for any sequential generator that suffers from exposure bias.
- **Warm-starting the student before the main objective:** the student is first fitted to deterministic noise-to-output pairs from the teacher's ODE solver, and only then trained with DMD. A cheap regression stage like this can stabilize harder distillation objectives.
- **Zero-shot task reuse through input manipulation:** image-to-video simply repeats the input image as the first chunk. Video-to-video adds noise to incoming chunks SDEdit-style and denoises them in one step. Both show how a causal generator can serve new tasks without retraining.
- **Reporting both initial latency and throughput:** separating time-to-first-frame from steady-state FPS is a clearer way to evaluate streaming systems than total generation time alone.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Massachusetts Institute of Technology](https://md.research.elamur.ai/free/en/universities/massachusetts-institute-of-technology/) — University, Cambridge, United States · 243 works · 76,506 citations
- [Adobe Systems](https://md.research.elamur.ai/free/en/companies/adobe-systems/) — Company, San Jose, United States · 40 works · 4,251 citations

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

## Cite

Cite as: Tianwei Yin, Qiang Zhang, Richard Zhang, William T. Freeman, Fredo Durand, Eli Shechtman and Xun Huang. 2024. From Slow Bidirectional to Fast Autoregressive Video Diffusion Models. arXiv:2412.07772.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{yin2024slow,
  title = {{From Slow Bidirectional to Fast Autoregressive Video Diffusion Models}},
  author = {Tianwei Yin, Qiang Zhang, Richard Zhang, William T. Freeman, Fredo Durand, Eli Shechtman and Xun Huang},
  year = {2024},
  doi = {10.48550/arxiv.2412.07772},
  eprint = {2412.07772},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2412.07772}
}
```

- [← previous Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding](https://md.research.elamur.ai/free/en/papers/zhang-2024-draft-verify-lossless-large-language/)
- [next → Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge](https://md.research.elamur.ai/free/en/papers/ye-2024-justice-prejudice-quantifying-biases-llm/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

10 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos Storkey, Tim Pearce, and François Fleuret. Diffusion for world modeling: Visual details matter in atari. In NeurIPS, 2024. 3 [find](https://scholar.google.com/scholar?q=Eloi+Alonso%2C+Adam+Jelley%2C+Vincent+Micheli%2C+Anssi+Kanervisto%2C+Amos+Storkey%2C+Tim+Pearce%2C+and+Franc%CC%A7ois+Fleuret%2E+Diffusion+for+world+modeling%3A+Visual+details+matter+in+atari%2E+In+NeurIPS%2C+2024%2E+3)
2. Fan Bao, Chongxuan Li, Yue Cao, and Jun Zhu. All are worth words: a vit backbone for score-based diffusion models. In NeurIPS 2022 Workshop on Score-Based Methods, 2022. 1 [find](https://scholar.google.com/scholar?q=Fan+Bao%2C+Chongxuan+Li%2C+Yue+Cao%2C+and+Jun+Zhu%2E+All+are+worth+words%3A+a+vit+backbone+for+score%2Dbased+diffusion+models%2E+In+NeurIPS+2022+Workshop+on+Score%2DBased+Methods%2C+2022%2E+1)
3. Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 1 — [arXiv:2311.15127](https://arxiv.org/abs/2311.15127) [find](https://scholar.google.com/scholar?q=Andreas+Blattmann%2C+Tim+Dockhorn%2C+Sumith+Kulal%2C+Daniel+Mendelevitch%2C+Maciej+Kilian%2C+Dominik+Lorenz%2C+Yam+Levi%2C+Zion+English%2C+Vikram+Voleti%2C+Adam+Letts%2C+et+al%2E+Stable+video+diffusion%3A+Scaling+latent+vide)
4. Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, 2023. 6 [find](https://scholar.google.com/scholar?q=Andreas+Blattmann%2C+Robin+Rombach%2C+Huan+Ling%2C+Tim+Dockhorn%2C+Seung+Wook+Kim%2C+Sanja+Fidler%2C+and+Karsten+Kreis%2E+Align+your+latents%3A+High%2Dresolution+video+synthesis+with+latent+diffusion+models%2E+In+CVPR%2C+2)
5. Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. 1, 3, 6, 8 [find](https://scholar.google.com/scholar?q=Tim+Brooks%2C+Bill+Peebles%2C+Connor+Holmes%2C+Will+DePue%2C+Yufei+Guo%2C+Li+Jing%2C+David+Schnurr%2C+Joe+Taylor%2C+Troy+Luhman%2C+Eric+Luhman%2C+Clarence+Ng%2C+Ricky+Wang%2C+and+Aditya+Ramesh%2E+Video+generation+models+as+wor)
6. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). In NeurIPS, 2020. 2, 3, 8
7. Haoxuan Che, Xuanhua He, Quande Liu, Cheng Jin, and Hao Chen. Gamegen-x: Interactive open-world game video generation. arXiv preprint arXiv:2411.00769, 2024. 2, 11 — [arXiv:2411.00769](https://arxiv.org/abs/2411.00769) [find](https://scholar.google.com/scholar?q=Haoxuan+Che%2C+Xuanhua+He%2C+Quande+Liu%2C+Cheng+Jin%2C+and+Hao+Chen%2E+Gamegen%2Dx%3A+Interactive+open%2Dworld+game+video+generation%2E+arXiv+preprint+arXiv%3A2411%2E00769%2C+2024%2E+2%2C+11+%E2%80%94+%5BarXiv%3A2411%2E00769%5D%28https%3A%2F%2Farxiv%2Eo)
8. Boyuan Chen, Diego Marti Monso, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. arXiv preprint arXiv:2407.01392, 2024. 3, 7, 10 — [arXiv:2407.01392](https://arxiv.org/abs/2407.01392) [find](https://scholar.google.com/scholar?q=Boyuan+Chen%2C+Diego+Marti+Monso%2C+Yilun+Du%2C+Max+Simchowitz%2C+Russ+Tedrake%2C+and+Vincent+Sitzmann%2E+Diffusion+forcing%3A+Next%2Dtoken+prediction+meets+full%2Dsequence+diffusion%2E+arXiv+preprint+arXiv%3A2407%2E01392%2C+2)
9. Feng Chen, Zhen Yang, Bohan Zhuang, and Qi Wu. Streaming video diffusion: Online video editing with diffusion models. arXiv preprint arXiv:2405.19726, 2024. 11 — [arXiv:2405.19726](https://arxiv.org/abs/2405.19726) [find](https://scholar.google.com/scholar?q=Feng+Chen%2C+Zhen+Yang%2C+Bohan+Zhuang%2C+and+Qi+Wu%2E+Streaming+video+diffusion%3A+Online+video+editing+with+diffusion+models%2E+arXiv+preprint+arXiv%3A2405%2E19726%2C+2024%2E+11+%E2%80%94+%5BarXiv%3A2405%2E19726%5D%28https%3A%2F%2Farxiv%2Eorg%2Fa)
10. Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafter1: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19512, 2023. 6 [37] Alex X Lee, Richard Zhang, Frederik Ebert, Pi… [find](https://scholar.google.com/scholar?q=Haoxin+Chen%2C+Menghan+Xia%2C+Yingqing+He%2C+Yong+Zhang%2C+Xiaodong+Cun%2C+Shaoshu+Yang%2C+Jinbo+Xing%2C+Yaofang+Liu%2C+Qifeng+Chen%2C+Xintao+Wang%2C+et+al%2E+Videocrafter1%3A+Open+diffusion+models+for+high%2Dquality+video+gen)

*Updated 2 November 2026*