---
title: "Can VLMs Predict Future States? Bootstrapping World Models"
description: "Inverse dynamics prediction (IDP), which describes the action between two frames, proves much easier to learn by fine-tuning. 57 references."
authors: ["Yifu Qiu", "Yftah Ziser", "Anna Korhonen", "Shay B Cohen", "Edoardo Ponti"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.1772/"
doi: "10.18653/v1/2026.findings-acl.1772"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/qiu-2026-can-vlms-predict-future-states/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/qiu-2026-can-vlms-predict-future-states/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/24/) / Can VLMs Predict Future States? Bootstrapping World Models from Inverse Dynamics

# Can VLMs Predict Future States? Bootstrapping World Models from Inverse Dynamics

**Authors:** [Yifu Qiu](https://md.research.elamur.ai/free/en/authors/yifu-qiu/), Yftah Ziser, [Anna Korhonen](https://md.research.elamur.ai/free/en/authors/anna-korhonen/), [Shay B Cohen](https://md.research.elamur.ai/free/en/authors/shay-b-cohen/) and [Edoardo Ponti](https://md.research.elamur.ai/free/en/authors/edoardo-maria-ponti/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1772/) · [PDF](https://aclanthology.org/2026.findings-acl.1772.pdf) · [DOI 10.18653/v1/2026.findings-acl.1772](https://doi.org/10.18653/v1/2026.findings-acl.1772) · [OpenAlex](https://openalex.org/W4417097554)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
The paper asks whether unified vision–language models, which can interleave text and images during generation, can act as world models. Specifically, it tests forward dynamics prediction (FDP): generating the next image frame from a previous frame and a textual action. The authors probe 9 VLMs on AURORA-Bench. When comparing real trajectories with manipulated ones, the models perform close to chance, so off-the-shelf models do not reliably favour real transitions. Inverse dynamics prediction (IDP), which describes the action between two frames, proves much easier to learn by fine-tuning. The authors exploit this asymmetry in two ways:
- **Weak supervision:** a VLM fine-tuned for IDP labels key-frame pairs mined from unlabelled videos, and these synthetic trajectories train FDP.
- **Verification:** the IDP model ranks several FDP samples at inference time.

Both strategies are applied to Chameleon-7B and Liquid-8B and evaluated as action-centric image editing. The best general-purpose model beats specialised diffusion editors by 7–13% under GPT4o-as-judge and achieves the top average human rating. This suggests VLMs could become a practical starting point for world models.

## Key points
- **Zero-shot probing.** Nine VLMs were compared on reference trajectories versus four negative types: Random Action, Random Observation, Copy Observation and Inverse Observation. Preference for the real trajectory hovered around 50%. The best IDP result, from Qwen2.5-VL-7B, was 60.08% against random actions and 67.34% against copied observations.
- **IDP training.** The inverse-dynamics model was fine-tuned on AURORA triplets and on first/last frames from the action recognition track of EPIC-Kitchen.
- **Synthetic data.** About 45 hours of video from Moments-in-Time, Kinetics-700 and UCF-101 were processed. Key frames were selected by optical flow (I_f = 20, K_f = 6), producing roughly 20K, 46K and 21K labelled triplets, respectively, before filtering. The loss is weighted toward image tokens in regions that change most between frames.
- **Evaluation.** Testing covered the AURORA-Bench subsets MagicBrush, Action-Genome, Something-Something, WhatsUp and Kubric. IDP-based verification, which needs no extra training, reached gains comparable to training on synthetic trajectories.
- **Limitations.** The model sometimes copies the source frame, and it handles fine spatial or quantitative instructions poorly. Chameleon results vary considerably between runs. Code and models are available at github.com/yfqiu-nlp/vlm-world-model.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Edinburgh](https://md.research.elamur.ai/free/en/universities/university-of-edinburgh/) — University, Edinburgh, United Kingdom · 359 works · 77,352 citations
- [University of Cambridge](https://md.research.elamur.ai/free/en/universities/university-of-cambridge/) — University, Cambridge, United Kingdom · 243 works · 121,396 citations
- [University of Groningen](https://md.research.elamur.ai/free/en/universities/university-of-groningen/) — University, Groningen, Netherlands · 70 works · 5,142 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works · [⚙️ Engineering](https://md.research.elamur.ai/free/en/shelves/engineering/) · 133 works · [⚙️ Control and Systems Engineering](https://md.research.elamur.ai/free/en/shelves/engineering/control-and-systems-engineering/) · 45 works

## Cite

Cite as: Yifu Qiu, Yftah Ziser, Anna Korhonen, Shay B Cohen and Edoardo Ponti. 2026. Can VLMs Predict Future States? Bootstrapping World Models from Inverse Dynamics. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{qiu2026can,
  title = {{Can VLMs Predict Future States? Bootstrapping World Models from Inverse Dynamics}},
  author = {Yifu Qiu, Yftah Ziser, Anna Korhonen, Shay B Cohen and Edoardo Ponti},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1772},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1772}
}
```

- [← previous Can Video LLMs See Through Illusions? Video-Illusion QA Benchmark Dataset](https://md.research.elamur.ai/free/en/papers/ohira-2026-can-video-llms-see-through/)
- [next → Can You Be More Explicit? A Task and Dataset on Explicitations of Implicit Meaning](https://md.research.elamur.ai/free/en/papers/zeidler-2026-can-you-be-more-explicit/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/24/)

## References

57 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders Søgaard. 2021. [Can language models encode perceptual structure without grounding? A case study in color](https://md.research.elamur.ai/free/en/papers/abdou-2021-can-language-models-encode-perceptual/). In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 109–132, Online.…
2. Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, and 1 others. 2025. Cosmos world foundation model platform for physical AI. arXiv preprint arXiv:2501.03575. — [arXiv:2501.03575](https://arxiv.org/abs/2501.03575) [find](https://scholar.google.com/scholar?q=Niket+Agarwal%2C+Arslan+Ali%2C+Maciej+Bala%2C+Yogesh+Balaji%2C+Erik+Barker%2C+Tiffany+Cai%2C+Prithvijit+Chattopadhyay%2C+Yongxin+Chen%2C+Yin+Cui%2C+Yifan+Ding%2C+and+1+others%2E+2025%2E+Cosmos+world+foundation+model+platform)
3. Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos J Storkey, Tim Pearce, and François Fleuret. 2024. Diffusion for world modeling: Visual details matter in atari. Advances in Neural Information Processing Systems, 37:58757–58791. [find](https://scholar.google.com/scholar?q=Eloi+Alonso%2C+Adam+Jelley%2C+Vincent+Micheli%2C+Anssi+Kanervisto%2C+Amos+J+Storkey%2C+Tim+Pearce%2C+and+Fran%C3%A7ois+Fleuret%2E+2024%2E+Diffusion+for+world+modeling%3A+Visual+details+matter+in+atari%2E+Advances+in+Neural+In)
4. Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. 2022. Video PreTraining (VPT): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639–24654. [find](https://scholar.google.com/scholar?q=Bowen+Baker%2C+Ilge+Akkaya%2C+Peter+Zhokov%2C+Joost+Huizinga%2C+Jie+Tang%2C+Adrien+Ecoffet%2C+Brandon+Houghton%2C+Raul+Sampedro%2C+and+Jeff+Clune%2E+2022%2E+Video+PreTraining+%28VPT%29%3A+Learning+to+act+by+watching+unlabeled+)
5. Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, and 1 others. 2023. [Do as I can, not as I say: Grounding language in robotic affordances](https://md.research.elamur.ai/free/en/papers/ahn-2022-i-i-say-grounding-language/). In Conference on robot learning, pages 287–318. PMLR.
6. Tim Brooks, Aleksander Holynski, and Alexei A Efros. 2023. InstructPix2Pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18392–18402. [find](https://scholar.google.com/scholar?q=Tim+Brooks%2C+Aleksander+Holynski%2C+and+Alexei+A+Efros%2E+2023%2E+InstructPix2Pix%3A+Learning+to+follow+image+editing+instructions%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+computer+vision+and+pattern+reco)
7. Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, and 1 others. 2024. Video generation models as world simulators. 2024. URL [https://openai.com/research/video-generation-models-as-world-simulators](https://openai.com/research/video-generation-models-as-world-simulators), 3:1. [↗](https://openai.com/research/video-generation-models-as-world-simulators)
8. Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, and 1 others. 2024. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Jake+Bruce%2C+Michael+D+Dennis%2C+Ashley+Edwards%2C+Jack+Parker%2DHolder%2C+Yuge+Shi%2C+Edward+Hughes%2C+Matthew+Lai%2C+Aditi+Mavalankar%2C+Richie+Steigerwald%2C+Chris+Apps%2C+and+1+others%2E+2024%2E+Genie%3A+Generative+interact)
9. Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. 2019. A short note on the kinetics-700 human action dataset. arXiv preprint arXiv:1907.06987. [find](https://scholar.google.com/scholar?q=Joao+Carreira%2C+Eric+Noland%2C+Chloe+Hillier%2C+and+Andrew+Zisserman%2E+2019%2E+A+short+note+on+the+kinetics%2D700+human+action+dataset%2E+arXiv+preprint+arXiv%3A1907%2E06987%2E)
10. Chameleon Team. 2024. [Chameleon: Mixed-modal early-fusion foundation models](https://md.research.elamur.ai/free/en/papers/team-2024-chameleon-mixed-modal-early-fusion/). arXiv preprint arXiv:2405.09818.
11. Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. 2021. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084–15097. [find](https://scholar.google.com/scholar?q=Lili+Chen%2C+Kevin+Lu%2C+Aravind+Rajeswaran%2C+Kimin+Lee%2C+Aditya+Grover%2C+Misha+Laskin%2C+Pieter+Abbeel%2C+Aravind+Srinivas%2C+and+Igor+Mordatch%2E+2021%2E+Decision+transformer%3A+Reinforcement+learning+via+sequence+mod)
12. William Chen, Oier Mees, Aviral Kumar, and Sergey Levine. 2024. Vision-language models provide promptable representations for reinforcement learning. In Automated Reinforcement Learning: Exploring Meta-Learning, AutoML, and LLMs. [find](https://scholar.google.com/scholar?q=William+Chen%2C+Oier+Mees%2C+Aviral+Kumar%2C+and+Sergey+Levine%2E+2024%2E+Vision%2Dlanguage+models+provide+promptable+representations+for+reinforcement+learning%2E+In+Automated+Reinforcement+Learning%3A+Exploring+Met)
13. Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. 2025. [Janus-Pro: Unified multimodal understanding and generation with data and model scaling](https://md.research.elamur.ai/free/en/papers/chen-2025-janus-pro-unified-multimodal/). arXiv preprint arXiv:2501.17811. 35588 10
14. Ethan Chern, Jiadi Su, Yan Ma, and Pengfei Liu. 2024. ANOLE: An open, autoregressive, native large multimodal models for interleaved image-text generation. arXiv preprint arXiv:2407.06135. — [arXiv:2407.06135](https://arxiv.org/abs/2407.06135) [find](https://scholar.google.com/scholar?q=Ethan+Chern%2C+Jiadi+Su%2C+Yan+Ma%2C+and+Pengfei+Liu%2E+2024%2E+ANOLE%3A+An+open%2C+autoregressive%2C+native+large+multimodal+models+for+interleaved+image%2Dtext+generation%2E+arXiv+preprint+arXiv%3A2407%2E06135%2E+%E2%80%94+%5BarXiv%3A24)
15. Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and 1 others. 2018. Scaling egocentric vision: The EPIC-KITCHENS dataset. In Proceedings of the European conference on computer vision… [find](https://scholar.google.com/scholar?q=Dima+Damen%2C+Hazel+Doughty%2C+Giovanni+Maria+Farinella%2C+Sanja+Fidler%2C+Antonino+Furnari%2C+Evangelos+Kazakos%2C+Davide+Moltisanti%2C+Jonathan+Munro%2C+Toby+Perrett%2C+Will+Price%2C+and+1+others%2E+2018%2E+Scaling+egocent)
16. Mengfei Du, Binhao Wu, Zejun Li, Xuan-Jing Huang, and Zhongyu Wei. 2024. Embspatial-bench: Benchmarking spatial understanding for embodied tasks with large vision-language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pag… [find](https://scholar.google.com/scholar?q=Mengfei+Du%2C+Binhao+Wu%2C+Zejun+Li%2C+Xuan%2DJing+Huang%2C+and+Zhongyu+Wei%2E+2024%2E+Embspatial%2Dbench%3A+Benchmarking+spatial+understanding+for+embodied+tasks+with+large+vision%2Dlanguage+models%2E+In+Proceedings+of+th)
17. Rongyao Fang, Chengqi Duan, Kun Wang, Linjiang Huang, Hao Li, Shilin Yan, Hao Tian, Xingyu Zeng, Rui Zhao, Jifeng Dai, and 1 others. 2025. GoT: Unleashing reasoning capability of multimodal large language model for visual generation and editing. arXiv preprint arXiv:2503.10639. — [arXiv:2503.10639](https://arxiv.org/abs/2503.10639) [find](https://scholar.google.com/scholar?q=Rongyao+Fang%2C+Chengqi+Duan%2C+Kun+Wang%2C+Linjiang+Huang%2C+Hao+Li%2C+Shilin+Yan%2C+Hao+Tian%2C+Xingyu+Zeng%2C+Rui+Zhao%2C+Jifeng+Dai%2C+and+1+others%2E+2025%2E+GoT%3A+Unleashing+reasoning+capability+of+multimodal+large+lang)
18. Jensen Gao, Bidipta Sarkar, Fei Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, and Dorsa Sadigh. 2024. Physically grounded vision-language models for robotic manipulation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 12462–12469. IEEE. [find](https://scholar.google.com/scholar?q=Jensen+Gao%2C+Bidipta+Sarkar%2C+Fei+Xia%2C+Ted+Xiao%2C+Jiajun+Wu%2C+Brian+Ichter%2C+Anirudha+Majumdar%2C+and+Dorsa+Sadigh%2E+2024%2E+Physically+grounded+vision%2Dlanguage+models+for+robotic+manipulation%2E+In+2024+IEEE+Int)
19. Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, and 1 others. 2022. Kubric: A scalable dataset generator. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p… [find](https://scholar.google.com/scholar?q=Klaus+Greff%2C+Francois+Belletti%2C+Lucas+Beyer%2C+Carl+Doersch%2C+Yilun+Du%2C+Daniel+Duckworth%2C+David+J+Fleet%2C+Dan+Gnanapragasam%2C+Florian+Golemo%2C+Charles+Herrmann%2C+and+1+others%2E+2022%2E+Kubric%3A+A+scalable+datase)
20. David Ha and Jürgen Schmidhuber. 2018. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=David+Ha+and+J%C3%BCrgen+Schmidhuber%2E+2018%2E+Recurrent+world+models+facilitate+policy+evolution%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+31%2E+Curran+Associates%2C+Inc%2E)
21. Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. 2019a. Learning latent dynamics for planning from pixels. In International conference on machine learning, pages 2555–2565. PMLR. [find](https://scholar.google.com/scholar?q=Danijar+Hafner%2C+Timothy+Lillicrap%2C+Ian+Fischer%2C+Ruben+Villegas%2C+David+Ha%2C+Honglak+Lee%2C+and+James+Davidson%2E+2019a%2E+Learning+latent+dynamics+for+planning+from+pixels%2E+In+International+conference+on+mach)
22. Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. 2019b. Learning latent dynamics for planning from pixels. In International conference on machine learning, pages 2555–2565. PMLR. [find](https://scholar.google.com/scholar?q=Danijar+Hafner%2C+Timothy+Lillicrap%2C+Ian+Fischer%2C+Ruben+Villegas%2C+David+Ha%2C+Honglak+Lee%2C+and+James+Davidson%2E+2019b%2E+Learning+latent+dynamics+for+planning+from+pixels%2E+In+International+conference+on+mach)
23. Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. 2025. Mastering diverse control tasks through world models. Nature, 640(8059):647–653. [find](https://scholar.google.com/scholar?q=Danijar+Hafner%2C+Jurgis+Pasukonis%2C+Jimmy+Ba%2C+and+Timothy+Lillicrap%2E+2025%2E+Mastering+diverse+control+tasks+through+world+models%2E+Nature%2C+640%288059%29%3A647%E2%80%93653%2E)
24. Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. [LoRA: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In International Conference on Learning Representations.
25. Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, and 1 others. 2022. Inner Monologue: Embodied reasoning through planning with language models. In 6th Annual Conference on Robot Learning. [find](https://scholar.google.com/scholar?q=Wenlong+Huang%2C+Fei+Xia%2C+Ted+Xiao%2C+Harris+Chan%2C+Jacky+Liang%2C+Pete+Florence%2C+Andy+Zeng%2C+Jonathan+Tompson%2C+Igor+Mordatch%2C+Yevgen+Chebotar%2C+and+1+others%2E+2022%2E+Inner+Monologue%3A+Embodied+reasoning+through+)
26. Yuzhou Huang, Liangbin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, and 1 others. 2024. SmartEdit: Exploring complex instruction-based image editing with multimodal large language models. In Proceedings of the IEEE/CVF Conference on Comput… [find](https://scholar.google.com/scholar?q=Yuzhou+Huang%2C+Liangbin+Xie%2C+Xintao+Wang%2C+Ziyang+Yuan%2C+Xiaodong+Cun%2C+Yixiao+Ge%2C+Jiantao+Zhou%2C+Chao+Dong%2C+Rui+Huang%2C+Ruimao+Zhang%2C+and+1+others%2E+2024%2E+SmartEdit%3A+Exploring+complex+instruction%2Dbased+imag)
27. Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, and 1 others. 2017. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950. [find](https://scholar.google.com/scholar?q=Will+Kay%2C+Joao+Carreira%2C+Karen+Simonyan%2C+Brian+Zhang%2C+Chloe+Hillier%2C+Sudheendra+Vijayanarasimhan%2C+Fabio+Viola%2C+Tim+Green%2C+Trevor+Back%2C+Paul+Natsev%2C+and+1+others%2E+2017%2E+The+kinetics+human+action+video+)
28. Benno Krojer, Dheeraj Vattikonda, Luis Lara, Varun Jampani, Eva Portelance, Christopher Pal, and Siva Reddy. 2024. Learning Action and Reasoning-Centric Image Editing from Videos and Simulations. In NeurIPS. Spotlight Paper. [find](https://scholar.google.com/scholar?q=Benno+Krojer%2C+Dheeraj+Vattikonda%2C+Luis+Lara%2C+Varun+Jampani%2C+Eva+Portelance%2C+Christopher+Pal%2C+and+Siva+Reddy%2E+2024%2E+Learning+Action+and+Reasoning%2DCentric+Image+Editing+from+Videos+and+Simulations%2E+In+N)
29. Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao, Li Dong, Ivan Vulić, and Furu Wei. 2025. Imagine while reasoning in space: Multimodal visualization-of-thought. arXiv preprint arXiv:2501.07542. — [arXiv:2501.07542](https://arxiv.org/abs/2501.07542) [find](https://scholar.google.com/scholar?q=Chengzu+Li%2C+Wenshan+Wu%2C+Huanyu+Zhang%2C+Yan+Xia%2C+Shaoguang+Mao%2C+Li+Dong%2C+Ivan+Vulic%CC%81%2C+and+Furu+Wei%2E+2025%2E+Imagine+while+reasoning+in+space%3A+Multimodal+visualization%2Dof%2Dthought%2E+arXiv+preprint+arXiv%3A2501)
30. Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. 2024. [Video-LLaVA: Learning united visual representation by alignment before projection](https://md.research.elamur.ai/free/en/papers/lin-2023-video-llava-learning-united-visual/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5971–5984.
31. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text summarization branches out, pages 74–81.
32. Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. 2024. World model on million-length video and language with blockwise RingAttention. arXiv preprint arXiv:2402.08268. [find](https://scholar.google.com/scholar?q=Hao+Liu%2C+Wilson+Yan%2C+Matei+Zaharia%2C+and+Pieter+Abbeel%2E+2024%2E+World+model+on+million%2Dlength+video+and+language+with+blockwise+RingAttention%2E+arXiv+preprint+arXiv%3A2402%2E08268%2E)
33. Jingping Liu, Ziyan Liu, Zhedong Cen, Yan Zhou, Yinan Zou, Weiyan Zhang, Haiyun Jiang, and Tong Ruan. 2025. [Can multimodal large language models understand spatial relations?](https://md.research.elamur.ai/free/en/papers/liu-2025-can-multimodal-large-language-models/) arXiv preprint arXiv:2505.19015.
34. Vincent Micheli, Eloi Alonso, and François Fleuret. 2022. Transformers are sample-efficient world models. In Deep Reinforcement Learning Workshop NeurIPS 2022. 35589 11 [find](https://scholar.google.com/scholar?q=Vincent+Micheli%2C+Eloi+Alonso%2C+and+Fran%C3%A7ois+Fleuret%2E+2022%2E+Transformers+are+sample%2Defficient+world+models%2E+In+Deep+Reinforcement+Learning+Workshop+NeurIPS+2022%2E+35589+11)
35. Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfruend, Carl Vondrick, and 1 others. 2019. Moments in time dataset: one million videos for event understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, p… [find](https://scholar.google.com/scholar?q=Mathew+Monfort%2C+Alex+Andonian%2C+Bolei+Zhou%2C+Kandan+Ramakrishnan%2C+Sarah+Adel+Bargal%2C+Tom+Yan%2C+Lisa+Brown%2C+Quanfu+Fan%2C+Dan+Gutfruend%2C+Carl+Vondrick%2C+and+1+others%2E+2019%2E+Moments+in+time+dataset%3A+one+milli)
36. Saman Motamed, Laura Culp, Kevin Swersky, Priyank Jaini, and Robert Geirhos. 2025. Do generative video models learn physical principles from watching videos? arXiv preprint arXiv:2501.09038. — [arXiv:2501.09038](https://arxiv.org/abs/2501.09038) [find](https://scholar.google.com/scholar?q=Saman+Motamed%2C+Laura+Culp%2C+Kevin+Swersky%2C+Priyank+Jaini%2C+and+Robert+Geirhos%2E+2025%2E+Do+generative+video+models+learn+physical+principles+from+watching+videos%3F+arXiv+preprint+arXiv%3A2501%2E09038%2E+%E2%80%94+%5BarXiv%3A)
37. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. [s1: Simple test-time scaling](https://md.research.elamur.ai/free/en/papers/muennighoff-2025-s1-simple-test-time-scaling/). arXiv preprint arXiv:2501.19393.
38. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [BLEU: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
39. Yiran Qin, Zhelun Shi, Jiwen Yu, Xijun Wang, Enshen Zhou, Lijun Li, Zhenfei Yin, Xihui Liu, Lu Sheng, Jing Shao, and 1 others. 2024. WorldSimBench: Towards video generation models as world simulators. arXiv preprint arXiv:2410.18072. — [arXiv:2410.18072](https://arxiv.org/abs/2410.18072) [find](https://scholar.google.com/scholar?q=Yiran+Qin%2C+Zhelun+Shi%2C+Jiwen+Yu%2C+Xijun+Wang%2C+Enshen+Zhou%2C+Lijun+Li%2C+Zhenfei+Yin%2C+Xihui+Liu%2C+Lu+Sheng%2C+Jing+Shao%2C+and+1+others%2E+2024%2E+WorldSimBench%3A+Towards+video+generation+models+as+world+simulators%2E)
40. Yifu Qiu, Zheng Zhao, Waylon Li, Yftah Ziser, Anna Korhonen, Shay B Cohen, and Edoardo M Ponti. 2026. Self-improving world modelling with latent actions. arXiv preprint arXiv:2602.06130. — [arXiv:2602.06130](https://arxiv.org/abs/2602.06130) [find](https://scholar.google.com/scholar?q=Yifu+Qiu%2C+Zheng+Zhao%2C+Waylon+Li%2C+Yftah+Ziser%2C+Anna+Korhonen%2C+Shay+B+Cohen%2C+and+Edoardo+M+Ponti%2E+2026%2E+Self%2Dimproving+world+modelling+with+latent+actions%2E+arXiv+preprint+arXiv%3A2602%2E06130%2E+%E2%80%94+%5BarXiv%3A2602)
41. Yifu Qiu, Zheng Zhao, Yftah Ziser, Anna Korhonen, Edoardo Ponti, and Shay B Cohen. 2024. Are large language model temporally grounded? In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pap… [find](https://scholar.google.com/scholar?q=Yifu+Qiu%2C+Zheng+Zhao%2C+Yftah+Ziser%2C+Anna+Korhonen%2C+Edoardo+Ponti%2C+and+Shay+B+Cohen%2E+2024%2E+Are+large+language+model+temporally+grounded%3F+In+Proceedings+of+the+2024+Conference+of+the+North+American+Chapt)
42. Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, and 1 others. 2022a. A generalist agent. arXiv preprint arXiv:2205.06175. [find](https://scholar.google.com/scholar?q=Scott+Reed%2C+Konrad+Zolna%2C+Emilio+Parisotto%2C+Sergio+Gomez+Colmenarejo%2C+Alexander+Novikov%2C+Gabriel+Barth%2DMaron%2C+Mai+Gimenez%2C+Yury+Sulsky%2C+Jackie+Kay%2C+Jost+Tobias+Springenberg%2C+and+1+others%2E+2022a%2E+A+gen)
43. Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, and 1 others. 2022b. A generalist agent. arXiv preprint arXiv:2205.06175. [find](https://scholar.google.com/scholar?q=Scott+Reed%2C+Konrad+Zolna%2C+Emilio+Parisotto%2C+Sergio+Gomez+Colmenarejo%2C+Alexander+Novikov%2C+Gabriel+Barth%2DMaron%2C+Mai+Gimenez%2C+Yury+Sulsky%2C+Jackie+Kay%2C+Jost+Tobias+Springenberg%2C+and+1+others%2E+2022b%2E+A+gen)
44. Jan Robine, Marc Höftmann, Tobias Uelwer, and Stefan Harmeling. 2023. Transformer-based world models are happy with 100k interactions. arXiv preprint arXiv:2303.07109. — [arXiv:2303.07109](https://arxiv.org/abs/2303.07109) [find](https://scholar.google.com/scholar?q=Jan+Robine%2C+Marc+H%C3%B6ftmann%2C+Tobias+Uelwer%2C+and+Stefan+Harmeling%2E+2023%2E+Transformer%2Dbased+world+models+are+happy+with+100k+interactions%2E+arXiv+preprint+arXiv%3A2303%2E07109%2E+%E2%80%94+%5BarXiv%3A2303%2E07109%5D%28https%3A%2F%2Farx)
45. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. [Scaling LLM test-time compute optimally can be more effective than scaling model parameters](https://md.research.elamur.ai/free/en/papers/snell-2024-scaling-llm-test-time-compute/). arXiv preprint arXiv:2408.03314.
46. Achint Soni, Sreyas Venkataraman, Abhranil Chandra, Sebastian Fischmeister, Percy Liang, Bo Dai, and Sherry Yang. 2024. VideoAgent: Self-improving video generation. arXiv preprint arXiv:2410.10076. [find](https://scholar.google.com/scholar?q=Achint+Soni%2C+Sreyas+Venkataraman%2C+Abhranil+Chandra%2C+Sebastian+Fischmeister%2C+Percy+Liang%2C+Bo+Dai%2C+and+Sherry+Yang%2E+2024%2E+VideoAgent%3A+Self%2Dimproving+video+generation%2E+arXiv+preprint+arXiv%3A2410%2E10076%2E)
47. Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. 2012. UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402. — [arXiv:1212.0402](https://arxiv.org/abs/1212.0402) [find](https://scholar.google.com/scholar?q=Khurram+Soomro%2C+Amir+Roshan+Zamir%2C+and+Mubarak+Shah%2E+2012%2E+UCF101%3A+A+dataset+of+101+human+actions+classes+from+videos+in+the+wild%2E+arXiv+preprint+arXiv%3A1212%2E0402%2E+%E2%80%94+%5BarXiv%3A1212%2E0402%5D%28https%3A%2F%2Farxiv%2Eorg)
48. Richard S Sutton. 1988. Learning to predict by the methods of temporal differences. Machine learning, 3:9–44. [find](https://scholar.google.com/scholar?q=Richard+S+Sutton%2E+1988%2E+Learning+to+predict+by+the+methods+of+temporal+differences%2E+Machine+learning%2C+3%3A9%E2%80%9344%2E)
49. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. [Transf…](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
50. Jialong Wu, Shaofeng Yin, Ningya Feng, Xu He, Dong Li, Jianye Hao, and Mingsheng Long. 2024a. iVideoGPT: Interactive VideoGPTs are scalable world models. In Advances in Neural Information Processing Systems, volume 37, pages 68082–68119. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Jialong+Wu%2C+Shaofeng+Yin%2C+Ningya+Feng%2C+Xu+He%2C+Dong+Li%2C+Jianye+Hao%2C+and+Mingsheng+Long%2E+2024a%2E+iVideoGPT%3A+Interactive+VideoGPTs+are+scalable+world+models%2E+In+Advances+in+Neural+Information+Processing+S)
51. Junfeng Wu, Yi Jiang, Chuofan Ma, Yuliang Liu, Hengshuang Zhao, Zehuan Yuan, Song Bai, and Xiang Bai. 2024b. Liquid: Language models are scalable and unified multi-modal generators. arXiv preprint arXiv:2412.04332. — [arXiv:2412.04332](https://arxiv.org/abs/2412.04332) [find](https://scholar.google.com/scholar?q=Junfeng+Wu%2C+Yi+Jiang%2C+Chuofan+Ma%2C+Yuliang+Liu%2C+Hengshuang+Zhao%2C+Zehuan+Yuan%2C+Song+Bai%2C+and+Xiang+Bai%2E+2024b%2E+Liquid%3A+Language+models+are+scalable+and+unified+multi%2Dmodal+generators%2E+arXiv+preprint+arX)
52. Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, and 1 others. 2024c. VILA-U: a unified foundation model integrating visual understanding and generation. arXiv preprint arXiv:2409.04429. — [arXiv:2409.04429](https://arxiv.org/abs/2409.04429) [find](https://scholar.google.com/scholar?q=Yecheng+Wu%2C+Zhuoyang+Zhang%2C+Junyu+Chen%2C+Haotian+Tang%2C+Dacheng+Li%2C+Yunhao+Fang%2C+Ligeng+Zhu%2C+Enze+Xie%2C+Hongxu+Yin%2C+Li+Yi%2C+and+1+others%2E+2024c%2E+VILA%2DU%3A+a+unified+foundation+model+integrating+visual+under)
53. Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. 2023. Learning interactive real-world simulators. arXiv preprint arXiv:2310.06114, 1(2):6. [find](https://scholar.google.com/scholar?q=Mengjiao+Yang%2C+Yilun+Du%2C+Kamyar+Ghasemipour%2C+Jonathan+Tompson%2C+Dale+Schuurmans%2C+and+Pieter+Abbeel%2E+2023%2E+Learning+interactive+real%2Dworld+simulators%2E+arXiv+preprint+arXiv%3A2310%2E06114%2C+1%282%29%3A6%2E)
54. Sherry Yang, Jacob Walker, Jack Parker-Holder, Yilun Du, Jake Bruce, Andre Barreto, Pieter Abbeel, and Dale Schuurmans. 2024. Video as the new language for real-world decision making. arXiv preprint arXiv:2402.17139. — [arXiv:2402.17139](https://arxiv.org/abs/2402.17139) [find](https://scholar.google.com/scholar?q=Sherry+Yang%2C+Jacob+Walker%2C+Jack+Parker%2DHolder%2C+Yilun+Du%2C+Jake+Bruce%2C+Andre+Barreto%2C+Pieter+Abbeel%2C+and+Dale+Schuurmans%2E+2024%2E+Video+as+the+new+language+for+real%2Dworld+decision+making%2E+arXiv+preprint+a)
55. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with bert. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+BERTScore%3A+Evaluating+text+generation+with+bert%2E+In+International+Conference+on+Learning+Representations%2E)
56. Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 2024. 3D-VLA: A 3D Vision-Language-Action 35590 12 [find](https://scholar.google.com/scholar?q=Haoyu+Zhen%2C+Xiaowen+Qiu%2C+Peihao+Chen%2C+Jincheng+Yang%2C+Xin+Yan%2C+Yilun+Du%2C+Yining+Hong%2C+and+Chuang+Gan%2E+2024%2E+3D%2DVLA%3A+A+3D+Vision%2DLanguage%2DAction+35590+12)
57. Generative World Model. In International Conference on Machine Learning, pages 61229–61245. PMLR. [find](https://scholar.google.com/scholar?q=Generative+World+Model%2E+In+International+Conference+on+Machine+Learning%2C+pages+61229%E2%80%9361245%2E+PMLR%2E)

*Updated 1 November 2026*