---
title: "LEDOM: Reverse Language Model — paper by Yin et al. (2026)"
description: "They build LEDOM, a purely right-to-left decoder-only model at 2B and 7B parameters. 41 references."
authors: ["Xunjian Yin", "Sitao Cheng", "Yuxi Xie", "Xinyu Hu", "Li Lin", "Xinyi Wang", "Liangming Pan", "William Yang Wang", "Xiaojun Wan"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.975/"
doi: "10.18653/v1/2026.acl-long.975"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yin-2026-ledom-reverse-language-model/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yin-2026-ledom-reverse-language-model/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/39/) / LEDOM: Reverse Language Model

# LEDOM: Reverse Language Model

**Authors:** [Xunjian Yin](https://md.research.elamur.ai/free/en/authors/xunjian-yin/), [Sitao Cheng](https://md.research.elamur.ai/free/en/authors/sitao-cheng/), [Yuxi Xie](https://md.research.elamur.ai/free/en/authors/yuxi-xie/), [Xinyu Hu](https://md.research.elamur.ai/free/en/authors/xinyu-hu/), [Li Lin](https://md.research.elamur.ai/free/en/authors/li-lin/), [Xinyi Wang](https://md.research.elamur.ai/free/en/authors/xinyi-wang/), [Liangming Pan](https://md.research.elamur.ai/free/en/authors/liangming-pan/), [William Yang Wang](https://md.research.elamur.ai/free/en/authors/william-yang-wang/) and [Xiaojun Wan](https://md.research.elamur.ai/free/en/authors/xiaojun-wan/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.975/) · [PDF](https://aclanthology.org/2026.acl-long.975.pdf) · [DOI 10.18653/v1/2026.acl-long.975](https://doi.org/10.18653/v1/2026.acl-long.975) · [OpenAlex](https://openalex.org/W4417182753)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Standard autoregressive language models are trained only in the left-to-right direction. The authors ask what happens if the opposite factorization is trained at scale, with each token predicted from the text that follows it. They build LEDOM, a purely right-to-left decoder-only model at 2B and 7B parameters. It is pretrained on 435B tokens of general, math and code data. Forward baselines share its architecture, tokenizer and data.

LEDOM shows several distinctive behaviours. It reconstructs plausible premises from a stated conclusion, derives questions from answers, works backward from mathematical results, and sidesteps the reversal curse by design. It performs on par with forward models on semantic understanding benchmarks but lags on forward-causal tasks such as code generation.

The authors then frame forward and reverse scoring as noisy-channel verification. They introduce Reverse Reward, which reranks forward-model candidates by how well the reverse model reconstructs the prompt, and prove that this penalizes hallucinated reasoning chains. On competition math, the method gives consistent gains. This suggests that generation direction is an underused resource for verifying model outputs without labeled data.

## Key points
- **Models:** LEDOM ("Model" spelled backwards) comes in 2B and 7B sizes, with matched forward LMs for comparison.
- **Pretraining data:** 435B tokens in total, made up of 284B from DCLM, 102B of math data and 48B of code from MAP-Neo.
- **Training setup:** 64 NVIDIA A100 GPUs. The reverse model converges more slowly and reaches a higher training loss than its forward counterpart.
- **Reverse Reward results:** up to +6.6% on AIME 2024 and +15% on AMC 2023 across three strong baselines, whose greedy accuracy ranges from about 42% to 96%.
- **Limitations and release:** the authors list weakness on forward-causal tasks and a possible bypass of left-to-right safety mechanisms. Code, models and data are released at github.com/Arvid-pku/LEDOM.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 315 works · 43,268 citations
- [Princeton University](https://md.research.elamur.ai/free/en/universities/princeton-university/) — University, Princeton, United States · 113 works · 48,612 citations
- [National University of Singapore](https://md.research.elamur.ai/free/en/universities/national-university-of-singapore/) — University, Singapore, Singapore · 207 works · 39,576 citations
- [University of Waterloo](https://md.research.elamur.ai/free/en/universities/university-of-waterloo/) — University, Waterloo, Canada · 80 works · 15,030 citations
- [University of California, Santa Barbara](https://md.research.elamur.ai/free/en/universities/university-of-california-santa-barbara/) — University, Santa Barbara, United States · 67 works · 10,550 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Xunjian Yin, Sitao Cheng, Yuxi Xie, Xinyu Hu, Li Lin, Xinyi Wang, Liangming Pan, William Yang Wang and Xiaojun Wan. 2026. LEDOM: Reverse Language Model. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yin2026ledom,
  title = {{LEDOM: Reverse Language Model}},
  author = {Xunjian Yin, Sitao Cheng, Yuxi Xie, Xinyu Hu, Li Lin, Xinyi Wang, Liangming Pan, William Yang Wang and Xiaojun Wan},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.975},
  url = {https://doi.org/10.18653/v1/2026.acl-long.975}
}
```

- [← previous LEDGER: Scaling Agentic Document Editing with Dependency-aware Graph Retrieval](https://md.research.elamur.ai/free/en/papers/wang-2026-ledger-scaling-agentic-document-editing/)
- [next → Legal Considerations in the Use of Synthetic Data for AI Development and Finetuning: The Case of LLMs4EU](https://md.research.elamur.ai/free/en/papers/talmoudi-2026-legal-considerations-use-synthetic-data/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/39/)

## References

41 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, and Luke Zettlemoyer. 2022. CM3: A causal masked multimodal model of the internet. CoRR, abs/2201.07520. [find](https://scholar.google.com/scholar?q=Armen+Aghajanyan%2C+Bernie+Huang%2C+Candace+Ross%2C+Vladimir+Karpukhin%2C+Hu+Xu%2C+Naman+Goyal%2C+Dmytro+Okhonko%2C+Mandar+Joshi%2C+Gargi+Ghosh%2C+Mike+Lewis%2C+and+Luke+Zettlemoyer%2E+2022%2E+CM3%3A+A+causal+masked+multimodal)
2. Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. 2022. [Efficient training of language models to fill in the middle](https://md.research.elamur.ai/free/en/papers/bavarian-2022-efficient-training-language-models-fill/). CoRR, abs/2207.14255.
3. Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. 2023. The reversal curse: Llms trained on" a is b" fail to learn" b is a". arXiv preprint arXiv:2309.12288. [find](https://scholar.google.com/scholar?q=Lukas+Berglund%2C+Meg+Tong%2C+Max+Kaufmann%2C+Mikita+Balesni%2C+Asa+Cooper+Stickland%2C+Tomasz+Korbak%2C+and+Owain+Evans%2E+2023%2E+The+reversal+curse%3A+Llms+trained+on%22+a+is+b%22+fail+to+learn%22+b+is+a%22%2E+arXiv+preprint+)
4. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
5. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374.
6. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. [Boolq: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). arXiv preprint arXiv:1905.10044.
7. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Minh%2DThang+Luong%2C+Quoc+V%2E+Le%2C+and+Christopher+D%2E+Manning%2E+2020%2E+ELECTRA%3A+Pre%2Dtraining+text+encoders+as+discriminators+rather+than+generators%2E+In+International+Conference+on+Learning+Repre)
8. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
9. OpenCompass Contributors. 2023. Opencompass: A universal evaluation platform for foundation models. [https://github.com/open-compass/](https://github.com/open-compass/) opencompass. [↗](https://github.com/open-compass/)
10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019a. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies…
11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019b. [BERT: pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies…
12. Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2022. Incoder: A generative model for code infilling and synthesis. CoRR, abs/2204.05999. [find](https://scholar.google.com/scholar?q=Daniel+Fried%2C+Armen+Aghajanyan%2C+Jessy+Lin%2C+Sida+Wang%2C+Eric+Wallace%2C+Freda+Shi%2C+Ruiqi+Zhong%2C+Wen%2Dtau+Yih%2C+Luke+Zettlemoyer%2C+and+Mike+Lewis%2E+2022%2E+Incoder%3A+A+generative+model+for+code+infilling+and+synt)
13. Olga Golovneva, Zeyuan Allen-Zhu, Jason Weston, and Sainbayar Sukhbaatar. 2024. Reverse training to nurse the reversal curse. arXiv preprint arXiv:2403.13799. — [arXiv:2403.13799](https://arxiv.org/abs/2403.13799) [find](https://scholar.google.com/scholar?q=Olga+Golovneva%2C+Zeyuan+Allen%2DZhu%2C+Jason+Weston%2C+and+Sainbayar+Sukhbaatar%2E+2024%2E+Reverse+training+to+nurse+the+reversal+curse%2E+arXiv+preprint+arXiv%3A2403%2E13799%2E+%E2%80%94+%5BarXiv%3A2403%2E13799%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
14. Qingyan Guo, Rui Wang, Junliang Guo, Xu Tan, Jiang Bian, and Yujiu Yang. 2024. Mitigating reversal curse via semantic-aware permutation training. arXiv preprint arXiv:2403.00758. — [arXiv:2403.00758](https://arxiv.org/abs/2403.00758) [find](https://scholar.google.com/scholar?q=Qingyan+Guo%2C+Rui+Wang%2C+Junliang+Guo%2C+Xu+Tan%2C+Jiang+Bian%2C+and+Yujiu+Yang%2E+2024%2E+Mitigating+reversal+curse+via+semantic%2Daware+permutation+training%2E+arXiv+preprint+arXiv%3A2403%2E00758%2E+%E2%80%94+%5BarXiv%3A2403%2E00758%5D%28)
15. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
16. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. [Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). arXiv preprint arXiv:1705.03551.
17. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. [Natural questions: a benchmark for question answering research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/). Transactions of the Association for Computational Linguistic…
18. Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhur… — [DataComp-LM: In search of the next generation of training sets for language models](https://md.research.elamur.ai/free/en/papers/li-2024-datacomp-lm-search-next-generation/)
19. John X. Morris, Wenting Zhao, Justin T. Chiu, Vitaly Shmatikov, and Alexander M. Rush. 2023. Language model inversion. Preprint, arXiv:2311.13647. — [arXiv:2311.13647](https://arxiv.org/abs/2311.13647) [find](https://scholar.google.com/scholar?q=John+X%2E+Morris%2C+Wenting+Zhao%2C+Justin+T%2E+Chiu%2C+Vitaly+Shmatikov%2C+and+Alexander+M%2E+Rush%2E+2023%2E+Language+model+inversion%2E+Preprint%2C+arXiv%3A2311%2E13647%2E+%E2%80%94+%5BarXiv%3A2311%2E13647%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2311%2E13647)
20. Jacob Pfau, Alex Infanger, Abhay Sheshadri, Ayush Panda, Julian Michael, and Curtis Huebner. 2023. Eliciting language model behaviors using reverse language models. In Socially Responsible Language Modelling Research. [find](https://scholar.google.com/scholar?q=Jacob+Pfau%2C+Alex+Infanger%2C+Abhay+Sheshadri%2C+Ayush+Panda%2C+Julian+Michael%2C+and+Curtis+Huebner%2E+2023%2E+Eliciting+language+model+behaviors+using+reverse+language+models%2E+In+Socially+Responsible+Language+Mo)
21. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Preprint, arXiv:1910.10683.
22. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2021%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+Communications+of+the+ACM%2C+64%289%29%3A99%E2%80%93106%2E)
23. Dmitriy Serdyuk, Nan Rosemary Ke, Alessandro Sordoni, Adam Trischler, Chris Pal, and Yoshua Bengio. 2017. Twin networks: Matching the future for sequence generation. arXiv preprint arXiv:1708.06742. — [arXiv:1708.06742](https://arxiv.org/abs/1708.06742) [find](https://scholar.google.com/scholar?q=Dmitriy+Serdyuk%2C+Nan+Rosemary+Ke%2C+Alessandro+Sordoni%2C+Adam+Trischler%2C+Chris+Pal%2C+and+Yoshua+Bengio%2E+2017%2E+Twin+networks%3A+Matching+the+future+for+sequence+generation%2E+arXiv+preprint+arXiv%3A1708%2E06742%2E+%E2%80%94)
24. Claude E. Shannon. 1948. A mathematical theory of communication. The Bell System Technical Journal, 27(3):379–423. [find](https://scholar.google.com/scholar?q=Claude+E%2E+Shannon%2E+1948%2E+A+mathematical+theory+of+communication%2E+The+Bell+System+Technical+Journal%2C+27%283%29%3A379%E2%80%93423%2E)
25. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). CoRR, abs/2402.03300.
26. Noam Shazeer. 2020. Glu variants improve transformer. Preprint, arXiv:2002.05202. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2E+2020%2E+Glu+variants+improve+transformer%2E+Preprint%2C+arXiv%3A2002%2E05202%2E)
27. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2023. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). Preprint, arXiv:2104.09864.
28. Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. 2025. [Openmathinstruct-2: Accelerating AI for math with massive open-source instruction data](https://md.research.elamur.ai/free/en/papers/toshniwal-2024-openmathinstruct-2-accelerating-ai-math/). In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2…
29. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. [Let’s verify step by step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/). arXiv preprint arXiv:2305.20050.
30. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). Preprint…
31. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789. [find](https://scholar.google.com/scholar?q=Todor+Mihaylov%2C+Peter+Clark%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+2018%2E+Can+a+suit+of+armor+conduct+electricity%3F+a+new+dataset+for+open+book+question+answering%2E+arXiv+preprint+arXiv%3A1809%2E02789%2E)
32. Yerram Varun, Rahul Madhavan, Sravanti Addepalli, Arun Suggala, Karthikeyan Shanmugam, and Prateek Jain. 2025. Time-reversal provides unsupervised feedback to llms. Preprint, arXiv:2412.02626. — [arXiv:2412.02626](https://arxiv.org/abs/2412.02626) [find](https://scholar.google.com/scholar?q=Yerram+Varun%2C+Rahul+Madhavan%2C+Sravanti+Addepalli%2C+Arun+Suggala%2C+Karthikeyan+Shanmugam%2C+and+Prateek+Jain%2E+2025%2E+Time%2Dreversal+provides+unsupervised+feedback+to+llms%2E+Preprint%2C+arXiv%3A2412%2E02626%2E+%E2%80%94+%5BarXi)
33. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Preprint, arXiv:1706.03762.
34. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. 2024. [Qwen2.5-math technical report: Toward mathematical expert model via self-improvement](https://md.research.elamur.ai/free/en/papers/yang-2024-qwen2-5-math-technical-report/). C…
35. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2020. [Xlnet: Generalized autoregressive pretraining for language understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/). Preprint, arXiv:1906.08237.
36. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. [Xlnet: Generalized autoregressive pretraining for language understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/). In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, Ne…
37. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+arXiv+preprint+arXiv%3A1905%2E07830%2E)
38. Biao Zhang and Rico Sennrich. 2019. Root mean square layer normalization. Preprint, arXiv:1910.07467. [find](https://scholar.google.com/scholar?q=Biao+Zhang+and+Rico+Sennrich%2E+2019%2E+Root+mean+square+layer+normalization%2E+Preprint%2C+arXiv%3A1910%2E07467%2E)
39. Ge Zhang, Scott Qu, Jiaheng Liu, Chenchen Zhang, Chenghua Lin, Chou Leuang Yu, Danny Pan, Esther Cheng, Jie Liu, Qunshu Lin, Raven Yuan, Tuney Zheng, Wei Pang, Xinrun Du, Yiming Liang, Yinghao Ma, Yizhi Li, Ziyang Ma, Bill Lin, Emmanouil Benetos, Huan Yang, Junting Zhou, Kaijing Ma, Minghao Liu, Mor… [find](https://scholar.google.com/scholar?q=Ge+Zhang%2C+Scott+Qu%2C+Jiaheng+Liu%2C+Chenchen+Zhang%2C+Chenghua+Lin%2C+Chou+Leuang+Yu%2C+Danny+Pan%2C+Esther+Cheng%2C+Jie+Liu%2C+Qunshu+Lin%2C+Raven+Yuan%2C+Tuney+Zheng%2C+Wei+Pang%2C+Xinrun+Du%2C+Yiming+Liang%2C+Yinghao+Ma%2C+Yiz)
40. Zhirui Zhang, Shuangzhi Wu, Shujie Liu, Mu Li, Ming Zhou, and Tong Xu. 2019. Regularizing neural machine translation by target-bidirectional agreement. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 443–450. [find](https://scholar.google.com/scholar?q=Zhirui+Zhang%2C+Shuangzhi+Wu%2C+Shujie+Liu%2C+Mu+Li%2C+Ming+Zhou%2C+and+Tong+Xu%2E+2019%2E+Regularizing+neural+machine+translation+by+target%2Dbidirectional+agreement%2E+In+Proceedings+of+the+AAAI+Conference+on+Artific)
41. Dataset Component General Math Code Token Count 284.16B 102.97B 48.24B *Table 5: Token distribution across the primary categories in our pre-training corpus. Total token count is approximately 435 billion.* [find](https://scholar.google.com/scholar?q=Dataset+Component+General+Math+Code+Token+Count+284%2E16B+102%2E97B+48%2E24B+%2ATable+5%3A+Token+distribution+across+the+primary+categories+in+our+pre%2Dtraining+corpus%2E+Total+token+count+is+approximately+435+bil)

*Updated 2 November 2026*