---
title: "Toward Consistent World Models with Multi-Token Prediction and"
description: "The paper asks whether models trained by predicting tokens really build coherent internal world models. 54 references."
authors: ["Qimin Zhong", "Hao Liao", "Haiming Qin", "Mingyang Zhou", "Rui Mao", "Wei Chen", "Naipeng Chao"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.618/"
doi: "10.18653/v1/2026.acl-long.618"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhong-2026-consistent-world-models-multi-token/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhong-2026-consistent-world-models-multi-token/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/53/) / Toward Consistent World Models with Multi-Token Prediction and Latent Semantic Enhancement

# Toward Consistent World Models with Multi-Token Prediction and Latent Semantic Enhancement

**Authors:** [Qimin Zhong](https://md.research.elamur.ai/free/en/authors/qimin-zhong/), [Hao Liao](https://md.research.elamur.ai/free/en/authors/hao-liao/), [Haiming Qin](https://md.research.elamur.ai/free/en/authors/haiming-qin/), [Mingyang Zhou](https://md.research.elamur.ai/free/en/authors/mingyang-zhou/), [Rui Mao](https://md.research.elamur.ai/free/en/authors/rui-mao/), [Wei Chen](https://md.research.elamur.ai/free/en/authors/wei-chen/) and [Naipeng Chao](https://md.research.elamur.ai/free/en/authors/naipeng-chao/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.618/) · [PDF](https://aclanthology.org/2026.acl-long.618.pdf) · [DOI 10.18653/v1/2026.acl-long.618](https://doi.org/10.18653/v1/2026.acl-long.618) · [OpenAlex](https://openalex.org/W7166880913)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
The paper asks whether models trained by predicting tokens really build coherent internal world models. Using a linearized gradient-flow (NTK) analysis, the authors argue that Multi-Token Prediction (MTP) couples gradients across histories that share a future. This pulls their hidden states toward common belief states. Because this pressure only considers outcomes, models can also learn latent "shortcuts" that skip intermediate steps and break environment rules. The authors call this failure structural hallucination. Their remedy, LSE-MTP, adds training losses that tie each multi-step predictive representation to the teacher-forced hidden state of the real future position and to semantic embedding anchors. The extra transition layers are removed at inference. The method is tested on synthetic graph navigation and the Manhattan taxi benchmark of Vafa et al. (2024). Compared with plain MTP, it gives better latent alignment and belief compression, fewer illegal paths, and stronger resistance to injected detours. The work shows that token-level accuracy is not evidence of a consistent world model, and it offers a training-time regularizer for long-horizon state tracking.

## Key points
- Theory: under NTP, contraction mainly affects states with the same next token (Theorem 1). Under MTP, states sharing a k-step-ahead target are pulled together even when their immediate targets differ (Theorem 2, Lemma 1). The analysis assumes the lazy-training/NTK regime.
- Architecture: a shared output head plus horizon-specific transition layers. LSE adds latent-consistency and semantic-anchoring losses, and inference uses standard autoregressive decoding.
- Evaluation metrics on Manhattan taxi data (Table 4) include:
  - state-wise similarity;
  - Compression Precision;
  - Distinction Precision and Distinction Recall, with a probability threshold of ϵ = 0.01;
  - Detour Robustness, with random legal non-Top-1 turns injected at p = 0.01.
- Findings: plain MTP raises similarity and compression but slightly lowers distinction precision. LSE-MTP keeps or improves distinction while compressing better and improving detour robustness.
- Limitations stated by the authors:
  - experiments cover only graph and path-planning tasks;
  - there is no comparison with RL, contrastive, or state-space approaches;
  - the theory rests on a linear approximation of training dynamics.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Microsoft Research Asia](https://md.research.elamur.ai/free/en/companies/microsoft-research-asia/) — Company, Beijing, China · 82 works · 61,484 citations
- [Shenzhen University](https://md.research.elamur.ai/free/en/universities/shenzhen-university/) — University, Shenzhen, China · 22 works · 95 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

## Cite

Cite as: Qimin Zhong, Hao Liao, Haiming Qin, Mingyang Zhou, Rui Mao, Wei Chen and Naipeng Chao. 2026. Toward Consistent World Models with Multi-Token Prediction and Latent Semantic Enhancement. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhong2026toward,
  title = {{Toward Consistent World Models with Multi-Token Prediction and Latent Semantic Enhancement}},
  author = {Qimin Zhong, Hao Liao, Haiming Qin, Mingyang Zhou, Rui Mao, Wei Chen and Naipeng Chao},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.618},
  url = {https://doi.org/10.18653/v1/2026.acl-long.618}
}
```

- [← previous Toward Cognitive Alignment in Large Language Models: Integrating Linguistic Theory and Human Data](https://md.research.elamur.ai/free/en/papers/zaghouani-2026-cognitive-alignment-large-language-models/)
- [next → Toward Dialect-Aware Safety Evaluation for Arabic Large Language Models](https://md.research.elamur.ai/free/en/papers/zaghouani-2026-dialect-aware-safety-evaluation-arabic/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/53/)

## References

54 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Gregor Bachmann and Vaishnavh Nagarajan. 2024. The pitfalls of next-token prediction. In Forty-first International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Gregor+Bachmann+and+Vaishnavh+Nagarajan%2E+2024%2E+The+pitfalls+of+next%2Dtoken+prediction%2E+In+Forty%2Dfirst+International+Conference+on+Machine+Learning%2E)
2. Marc Barthelemy and Geoff Boeing. 2025. Universal model of urban street networks. Physical Review Letters, 135:137401. [find](https://scholar.google.com/scholar?q=Marc+Barthelemy+and+Geoff+Boeing%2E+2025%2E+Universal+model+of+urban+street+networks%2E+Physical+Review+Letters%2C+135%3A137401%2E)
3. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In ACM Conference on Fairness, Accountability, and Transparency (FAccT), pages 610–623, New York, NY, USA. Association for Computing Machiner… [find](https://scholar.google.com/scholar?q=Emily+M%2E+Bender%2C+Timnit+Gebru%2C+Angelina+McMillan%2DMajor%2C+and+Shmargaret+Shmitchell%2E+2021%2E+On+the+dangers+of+stochastic+parrots%3A+Can+language+models+be+too+big%3F+In+ACM+Conference+on+Fairness%2C+Accountabi)
4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and… [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D%2E+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretche)
5. Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. 2024. Medusa: Simple large language model inference acceleration framework with multiple decoding heads. In Forty-first International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Tianle+Cai%2C+Yuhong+Li%2C+Zhengyang+Geng%2C+Hongwu+Peng%2C+Jason+D%2E+Lee%2C+Deming+Chen%2C+and+Tri+Dao%2E+2024%2E+Medusa%3A+Simple+large+language+model+inference+acceleration+framework+with+multiple+decoding+heads%2E+In+)
6. Lang Cao. 2024. GraphReason: Enhancing reasoning capabilities of large language models through a graph-based verification approach. In Proceedings of the 2nd Workshop on Natural Language Reasoning and Structured Explanations (@ACL 2024), pages 1–12, Bangkok, Thailand. Association for Computational L… [find](https://scholar.google.com/scholar?q=Lang+Cao%2E+2024%2E+GraphReason%3A+Enhancing+reasoning+capabilities+of+large+language+models+through+a+graph%2Dbased+verification+approach%2E+In+Proceedings+of+the+2nd+Workshop+on+Natural+Language+Reasoning+and)
7. Lénaïc Chizat, Edouard Oyallon, and Francis R. Bach. 2019. On lazy training in differentiable programming. In Conference on Neural Information Processing Systems, pages 2933–2943. [find](https://scholar.google.com/scholar?q=L%C3%A9na%C3%AFc+Chizat%2C+Edouard+Oyallon%2C+and+Francis+R%2E+Bach%2E+2019%2E+On+lazy+training+in+differentiable+programming%2E+In+Conference+on+Neural+Information+Processing+Systems%2C+pages+2933%E2%80%932943%2E)
8. Bahare Fatemi, Jonathan Halcrow, and Bryan Perozzi. 2024. [Talk like a graph: Encoding graphs for large language models](https://md.research.elamur.ai/free/en/papers/fatemi-2023-talk-like-graph-encoding-graphs/). In The Twelfth International Conference on Learning Representations.
9. Arvid Frydenlund. 2025. Language models, graph searching, and supervision adulteration: When more supervision is less and how to make more more. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 29011–29059, Vienna, Austria. Ass… [find](https://scholar.google.com/scholar?q=Arvid+Frydenlund%2E+2025%2E+Language+models%2C+graph+searching%2C+and+supervision+adulteration%3A+When+more+supervision+is+less+and+how+to+make+more+more%2E+In+Proceedings+of+the+63rd+Annual+Meeting+of+the+Associ)
10. Prakhar Gupta, Henry Conklin, Sarah-Jane Leslie, and Andrew Lee. 2025. Better World Models Can Lead to Better Post-Training Performance. arXiv e-prints, arXiv:2512.03400. — [arXiv:2512.03400](https://arxiv.org/abs/2512.03400) [find](https://scholar.google.com/scholar?q=Prakhar+Gupta%2C+Henry+Conklin%2C+Sarah%2DJane+Leslie%2C+and+Andrew+Lee%2E+2025%2E+Better+World+Models+Can+Lead+to+Better+Post%2DTraining+Performance%2E+arXiv+e%2Dprints%2C+arXiv%3A2512%2E03400%2E+%E2%80%94+%5BarXiv%3A2512%2E03400%5D%28https%3A%2F%2F)
11. Wes Gurnee and Max Tegmark. 2024. Language models represent space and time. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Wes+Gurnee+and+Max+Tegmark%2E+2024%2E+Language+models+represent+space+and+time%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2E)
12. David Ha and Jürgen Schmidhuber. 2018. World models. CoRR, abs/1803.10122. [find](https://scholar.google.com/scholar?q=David+Ha+and+J%C3%BCrgen+Schmidhuber%2E+2018%2E+World+models%2E+CoRR%2C+abs%2F1803%2E10122%2E)
13. Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. 2025. Mastering diverse control tasks through world models. Nature, 640:647–653. [find](https://scholar.google.com/scholar?q=Danijar+Hafner%2C+Jurgis+Pasukonis%2C+Jimmy+Ba%2C+and+Timothy+Lillicrap%2E+2025%2E+Mastering+diverse+control+tasks+through+world+models%2E+Nature%2C+640%3A647%E2%80%93653%2E)
14. Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong, Zhen Wang, Daisy Wang, and Zhiting Hu. 2023. Reasoning with language model is planning with world model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8154–8173, Singapore. Association for Computational Lingu… [find](https://scholar.google.com/scholar?q=Shibo+Hao%2C+Yi+Gu%2C+Haodi+Ma%2C+Joshua+Hong%2C+Zhen+Wang%2C+Daisy+Wang%2C+and+Zhiting+Hu%2E+2023%2E+Reasoning+with+language+model+is+planning+with+world+model%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+Met)
15. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. [Distilling the Knowledge in a Neural Network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/). arXiv e-prints, arXiv:1503.02531.
16. Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. 1998. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1):99–134. [find](https://scholar.google.com/scholar?q=Leslie+Pack+Kaelbling%2C+Michael+L%2E+Littman%2C+and+Anthony+R%2E+Cassandra%2E+1998%2E+Planning+and+acting+in+partially+observable+stochastic+domains%2E+Artificial+Intelligence%2C+101%281%29%3A99%E2%80%93134%2E)
17. Najoung Kim and Sebastian Schuster. 2023. [Entity tracking in language models](https://md.research.elamur.ai/free/en/papers/kim-2023-entity-tracking-language-models/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3835–3855, Toronto, Canada. Association for Computational Linguistics.
18. Quentin Garrido, Mahmoud Assran, Nicolas Ballas, Adrien Bardes, Laurent Najman, and Yann LeCun. 2024. Learning and Leveraging World Models in Visual Representation Learning. arXiv e-prints, arXiv:2403.00504. — [arXiv:2403.00504](https://arxiv.org/abs/2403.00504) [find](https://scholar.google.com/scholar?q=Quentin+Garrido%2C+Mahmoud+Assran%2C+Nicolas+Ballas%2C+Adrien+Bardes%2C+Laurent+Najman%2C+and+Yann+LeCun%2E+2024%2E+Learning+and+Leveraging+World+Models+in+Visual+Representation+Learning%2E+arXiv+e%2Dprints%2C+arXiv%3A2403)
19. Yann LeCun and Courant. 2022. A path towards autonomous machine intelligence version 0.9.2, 2022-06-27. In Proceedings of the International Conference on Machine Intelligence. [find](https://scholar.google.com/scholar?q=Yann+LeCun+and+Courant%2E+2022%2E+A+path+towards+autonomous+machine+intelligence+version+0%2E9%2E2%2C+2022%2D06%2D27%2E+In+Proceedings+of+the+International+Conference+on+Machine+Intelligence%2E)
20. Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard S. Zemel, Wieland Brendel, Matthias Bethge, and Felix Wichmann. 2020. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2:665–673. [find](https://scholar.google.com/scholar?q=Robert+Geirhos%2C+J%C3%B6rn%2DHenrik+Jacobsen%2C+Claudio+Michaelis%2C+Richard+S%2E+Zemel%2C+Wieland+Brendel%2C+Matthias+Bethge%2C+and+Felix+Wichmann%2E+2020%2E+Shortcut+learning+in+deep+neural+networks%2E+Nature+Machine+Intelli)
21. Anson Lei, Bernhard Schölkopf, and Ingmar Posner. 2023. Variational causal dynamics: Discovering modular world models from interventions. Transactions on Machine Learning Research. [find](https://scholar.google.com/scholar?q=Anson+Lei%2C+Bernhard+Sch%C3%B6lkopf%2C+and+Ingmar+Posner%2E+2023%2E+Variational+causal+dynamics%3A+Discovering+modular+world+models+from+interventions%2E+Transactions+on+Machine+Learning+Research%2E)
22. Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Roziere, David Lopez-Paz, and Gabriel Synnaeve. 2024. [Better & faster large language models via multi-token prediction](https://md.research.elamur.ai/free/en/papers/gloeckle-2024-better-faster-large-language-models/). In Forty-first International Conference on Machine Learning.
23. Anirudh Goyal, Alex Lamb, Ying Zhang, Saizheng Zhang, Aaron C. Courville, and Yoshua Bengio. 2016. Professor forcing: A new algorithm for training recurrent networks. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Anirudh+Goyal%2C+Alex+Lamb%2C+Ying+Zhang%2C+Saizheng+Zhang%2C+Aaron+C%2E+Courville%2C+and+Yoshua+Bengio%2E+2016%2E+Professor+forcing%3A+A+new+algorithm+for+training+recurrent+networks%2E+In+Advances+in+Neural+Information)
24. Belinda Z. Li, Zifan Carl Guo, and Jacob Andreas. 2025. (how) do language models track state? In Forty-second International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Belinda+Z%2E+Li%2C+Zifan+Carl+Guo%2C+and+Jacob+Andreas%2E+2025%2E+%28how%29+do+language+models+track+state%3F+In+Forty%2Dsecond+International+Conference+on+Machine+Learning%2E)
25. Belinda Z. Li, Maxwell Nye, and Jacob Andreas. 2021. Implicit representations of meaning in neural language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pa… [find](https://scholar.google.com/scholar?q=Belinda+Z%2E+Li%2C+Maxwell+Nye%2C+and+Jacob+Andreas%2E+2021%2E+Implicit+representations+of+meaning+in+neural+language+models%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Lingu)
26. Kenneth Li, Aspen K. Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2023a. Emergent world representations: Exploring a sequence model trained on a synthetic task. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Kenneth+Li%2C+Aspen+K%2E+Hopkins%2C+David+Bau%2C+Fernanda+Vi%C3%A9gas%2C+Hanspeter+Pfister%2C+and+Martin+Wattenberg%2E+2023a%2E+Emergent+world+representations%3A+Exploring+a+sequence+model+trained+on+a+synthetic+task%2E+In+Th)
27. Lei Li, Jingjing Xu, Qingxiu Dong, Ce Zheng, Xu Sun, Lingpeng Kong, and Qi Liu. 2023b. Can language models understand physical concepts? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 11843–11861, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Lei+Li%2C+Jingjing+Xu%2C+Qingxiu+Dong%2C+Ce+Zheng%2C+Xu+Sun%2C+Lingpeng+Kong%2C+and+Qi+Liu%2E+2023b%2E+Can+language+models+understand+physical+concepts%3F+In+Proceedings+of+the+2023+Conference+on+Empirical+Methods+in+N)
28. Michael L. Littman and Richard S. Sutton. 2001. Predictive representations of state. In Advances in Neural Information Processing Systems, volume 14. MIT Press. [find](https://scholar.google.com/scholar?q=Michael+L%2E+Littman+and+Richard+S%2E+Sutton%2E+2001%2E+Predictive+representations+of+state%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+14%2E+MIT+Press%2E)
29. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, and 7 others. 2022. [In-con…](https://md.research.elamur.ai/free/en/papers/olsson-2022-context-learning-induction-heads/)
30. Roma Patel and Ellie Pavlick. 2022. Mapping language models to grounded conceptual spaces. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Roma+Patel+and+Ellie+Pavlick%2E+2022%2E+Mapping+language+models+to+grounded+conceptual+spaces%2E+In+International+Conference+on+Learning+Representations%2E)
31. Jeffrey Pennington and Pratik Worah. 2017. Nonlinear random matrix theory for deep learning. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington+and+Pratik+Worah%2E+2017%2E+Nonlinear+random+matrix+theory+for+deep+learning%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+30%2E+Curran+Associates%2C+Inc%2E)
32. Weizhen Qi, Yu Yan, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. 2020. [ProphetNet: Predicting future n-gram for sequence-to-SequencePre-training](https://md.research.elamur.ai/free/en/papers/qi-2020-prophetnet-predicting-future-n-gram/). In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2401–2410, Online. Association for Comp…
33. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen… [find](https://scholar.google.com/scholar?q=Jack+W%2E+Rae%2C+Sebastian+Borgeaud%2C+Trevor+Cai%2C+Katie+Millican%2C+Jordan+Hoffmann%2C+Francis+Song%2C+John+Aslanides%2C+Sarah+Henderson%2C+Roman+Ring%2C+Susannah+Young%2C+Eliza+Rutherford%2C+Tom+Hennigan%2C+Jacob+Menick%2C+A)
34. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. [Sequence level training with recurrent neural networks](https://md.research.elamur.ai/free/en/papers/ranzato-2015-sequence-level-training-recurrent-neural/). In Fourth International Conference on Learning Representations (ICLR 2016), Conference Track Proceedings, San Juan, Puerto Rico.
35. Andrew M. Saxe, James L. McClelland, and Surya Ganguli. 2013. [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://md.research.elamur.ai/free/en/papers/saxe-2013-exact-solutions-nonlinear-dynamics-learning/). arXiv eprints, arXiv:1312.6120.
36. Jürgen Schmidhuber. 1990. Making the world differentiable: on using self-supervised fully recurrent neural networks for dynamic reinforcement learning and planning in non-stationary environments. Forschungsberichte, TU Munich, FKI 126 90:1–26. [find](https://scholar.google.com/scholar?q=J%C3%BCrgen+Schmidhuber%2E+1990%2E+Making+the+world+differentiable%3A+on+using+self%2Dsupervised+fully+recurrent+neural+networks+for+dynamic+reinforcement+learning+and+planning+in+non%2Dstationary+environments%2E+Fors)
37. Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, L. Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy P. Lillicrap, and David Silver. 2019. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588:604– 609. [find](https://scholar.google.com/scholar?q=Julian+Schrittwieser%2C+Ioannis+Antonoglou%2C+Thomas+Hubert%2C+Karen+Simonyan%2C+L%2E+Sifre%2C+Simon+Schmitt%2C+Arthur+Guez%2C+Edward+Lockhart%2C+Demis+Hassabis%2C+Thore+Graepel%2C+Timothy+P%2E+Lillicrap%2C+and+David+Silver%2E+2)
38. David Silver, Hado van Hasselt, Matteo Hessel, Tom Schaul, Arthur Guez, Tim Harley, Gabriel Dulac-Arnold, David P. Reichert, Neil C. Rabinowitz, André Barreto, and Thomas Degris. 2017. The predic-tron: End-to-end learning and planning. In Thirty-fourth International Conference on Machine Learning (I… [find](https://scholar.google.com/scholar?q=David+Silver%2C+Hado+van+Hasselt%2C+Matteo+Hessel%2C+Tom+Schaul%2C+Arthur+Guez%2C+Tim+Harley%2C+Gabriel+Dulac%2DArnold%2C+David+P%2E+Reichert%2C+Neil+C%2E+Rabinowitz%2C+Andr%C3%A9+Barreto%2C+and+Thomas+Degris%2E+2017%2E+The+predic%2Dtron)
39. Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. 2023. Consistency models. In International Conference on Machine Learning, pages 32211–32252. [find](https://scholar.google.com/scholar?q=Yang+Song%2C+Prafulla+Dhariwal%2C+Mark+Chen%2C+and+Ilya+Sutskever%2E+2023%2E+Consistency+models%2E+In+International+Conference+on+Machine+Learning%2C+pages+32211%E2%80%9332252%2E)
40. Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. 2025. On the self-verification limitations of large language models on reasoning and planning tasks. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Kaya+Stechly%2C+Karthik+Valmeekam%2C+and+Subbarao+Kambhampati%2E+2025%2E+On+the+self%2Dverification+limitations+of+large+language+models+on+reasoning+and+planning+tasks%2E+In+The+Thirteenth+International+Conferen)
41. Jayden Teoh, Manan Tomar, Kwangjun Ahn, Edward S. Hu, Pratyusha Sharma, Riashat Islam, Alex Lamb, and John Langford. 2025. Next-Latent Prediction Transformers Learn Compact World Models. arXiv e-prints, arXiv:2511.05963. — [arXiv:2511.05963](https://arxiv.org/abs/2511.05963) [find](https://scholar.google.com/scholar?q=Jayden+Teoh%2C+Manan+Tomar%2C+Kwangjun+Ahn%2C+Edward+S%2E+Hu%2C+Pratyusha+Sharma%2C+Riashat+Islam%2C+Alex+Lamb%2C+and+John+Langford%2E+2025%2E+Next%2DLatent+Prediction+Transformers+Learn+Compact+World+Models%2E+arXiv+e%2Dprint)
42. Keyon Vafa, Justin Y. Chen, Ashesh Rambachan, Jon Kleinberg, and Sendhil Mullainathan. 2024. Evaluating the world model implicit in a generative model. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Keyon+Vafa%2C+Justin+Y%2E+Chen%2C+Ashesh+Rambachan%2C+Jon+Kleinberg%2C+and+Sendhil+Mullainathan%2E+2024%2E+Evaluating+the+world+model+implicit+in+a+generative+model%2E+In+The+Thirty%2Deighth+Annual+Conference+on+Neural)
43. Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2023. [Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change](https://md.research.elamur.ai/free/en/papers/valmeekam-2022-planbench-extensible-benchmark-evaluating-large/). In The Thirty-seventh Annual Conference on Neural Information Processing Systems, Datas…
44. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. [Representation Learning with Contrastive Predictive Coding](https://md.research.elamur.ai/free/en/papers/oord-2018-representation-learning-contrastive-predictive-coding/). arXiv e-prints, arXiv:1807.03748.
45. Yilin Wen, Zifeng Wang, and Jimeng Sun. 2024. [MindMap: Knowledge graph prompting sparks graph of thoughts in large language models](https://md.research.elamur.ai/free/en/papers/wen-2024-mindmap-knowledge-graph-prompting-sparks/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10370–10388, Bangkok, Thailand. Association fo…
46. Xixi Wu, Yifei Shen, Caihua Shan, Kaitao Song, Siwei Wang, Bohang Zhang, Jiarui Feng, Hong Cheng, Wei Chen, Yun Xiong, and Dongsheng Li. 2024. Can graph learning improve planning in large language model-based agents? In The Thirty-eighth Annual Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Xixi+Wu%2C+Yifei+Shen%2C+Caihua+Shan%2C+Kaitao+Song%2C+Siwei+Wang%2C+Bohang+Zhang%2C+Jiarui+Feng%2C+Hong+Cheng%2C+Wei+Chen%2C+Yun+Xiong%2C+and+Dongsheng+Li%2E+2024%2E+Can+graph+learning+improve+planning+in+large+language+mod)
47. Charlie Wyatt, Aditya Joshi, and Flora Salim. 2025. Alternatives To Next Token Prediction In Text Generation – A Survey. arXiv e-prints, arXiv:2509.24435. — [arXiv:2509.24435](https://arxiv.org/abs/2509.24435) [find](https://scholar.google.com/scholar?q=Charlie+Wyatt%2C+Aditya+Joshi%2C+and+Flora+Salim%2E+2025%2E+Alternatives+To+Next+Token+Prediction+In+Text+Generation+%E2%80%93+A+Survey%2E+arXiv+e%2Dprints%2C+arXiv%3A2509%2E24435%2E+%E2%80%94+%5BarXiv%3A2509%2E24435%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F25)
48. Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie-Yan Liu. 2020. On Layer Normalization in the Transformer Architecture. arXiv e-prints, arXiv:2002.04745. [find](https://scholar.google.com/scholar?q=Ruibin+Xiong%2C+Yunchang+Yang%2C+Di+He%2C+Kai+Zheng%2C+Shuxin+Zheng%2C+Chen+Xing%2C+Huishuai+Zhang%2C+Yanyan+Lan%2C+Liwei+Wang%2C+and+Tie%2DYan+Liu%2E+2020%2E+On+Layer+Normalization+in+the+Transformer+Architecture%2E+arXiv+e%2Dp)
49. Yongjing Yin, Junran Ding, Kai Song, and Yue Zhang. 2024. Semformer: Transformer language models with semantic planning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18669–18680, Miami, Florida, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yongjing+Yin%2C+Junran+Ding%2C+Kai+Song%2C+and+Yue+Zhang%2E+2024%2E+Semformer%3A+Transformer+language+models+with+semantic+planning%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Natural+Language+)
50. Guangyao Zhai, Xingyuan Zhang, and Nassir Navab. 2025. Recurrent world model with tokenized latent states. In ICLR 2025 Workshop on World Models: Understanding, Modelling and Scaling. [find](https://scholar.google.com/scholar?q=Guangyao+Zhai%2C+Xingyuan+Zhang%2C+and+Nassir+Navab%2E+2025%2E+Recurrent+world+model+with+tokenized+latent+states%2E+In+ICLR+2025+Workshop+on+World+Models%3A+Understanding%2C+Modelling+and+Scaling%2E)
51. Fu Zhang, Jinghao Lin, and Jingwei Cheng. 2024. SALMON: A structure-aware language model with logicality and densification strategy for temporal knowledge graph reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 8761–8774, Miami, Florida, USA. Association for… [find](https://scholar.google.com/scholar?q=Fu+Zhang%2C+Jinghao+Lin%2C+and+Jingwei+Cheng%2E+2024%2E+SALMON%3A+A+structure%2Daware+language+model+with+logicality+and+densification+strategy+for+temporal+knowledge+graph+reasoning%2E+In+Findings+of+the+Associati)
52. Wen Zhang, Yang Feng, Fandong Meng, Di You, and Qun Liu. 2019. [Bridging the gap between training and inference for neural machine translation](https://md.research.elamur.ai/free/en/papers/zhang-2019-bridging-gap-between-training-inference/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4334– 4343, Florence, Italy. Association for Computational…
53. Qimin Zhong, Hao Liao, Siwei Wang, Mingyang Zhou, Xiaoqun Wu, Rui Mao, and Wei Chen. 2025. Understanding and Enhancing the Planning Capability of Language Models via Multi-Token Prediction. arXiv e-prints, arXiv:2509.23186. — [arXiv:2509.23186](https://arxiv.org/abs/2509.23186) [find](https://scholar.google.com/scholar?q=Qimin+Zhong%2C+Hao+Liao%2C+Siwei+Wang%2C+Mingyang+Zhou%2C+Xiaoqun+Wu%2C+Rui+Mao%2C+and+Wei+Chen%2E+2025%2E+Understanding+and+Enhancing+the+Planning+Capability+of+Language+Models+via+Multi%2DToken+Prediction%2E+arXiv+e%2Dpr)
54. Wentao Zhu, Zhining Zhang, and Yizhou Wang. 2024. Language models represent beliefs of self and others. In Forty-first International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Wentao+Zhu%2C+Zhining+Zhang%2C+and+Yizhou+Wang%2E+2024%2E+Language+models+represent+beliefs+of+self+and+others%2E+In+Forty%2Dfirst+International+Conference+on+Machine+Learning%2E)

*Updated 2 November 2026*