--- title: "Pseudo-Likelihood Training for Reasoning Diffusion Language" description: "The paper introduces PADRE, which treats KL-regularized RL as inference toward a reward-tilted Boltzmann distribution. 4 references." authors: ["Shiv Shankar"] year: 2026 venue: "Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)" source: "https://aclanthology.org/2026.eacl-long.257/" doi: "10.18653/v1/2026.eacl-long.257" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/shankar-2026-pseudo-likelihood-training-reasoning-diffusion/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shankar-2026-pseudo-likelihood-training-reasoning-diffusion/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/40/) / Pseudo-Likelihood Training for Reasoning Diffusion Language Models # Pseudo-Likelihood Training for Reasoning Diffusion Language Models **Authors:** [Shiv Shankar](https://md.research.elamur.ai/free/en/authors/shiv-shankar/) · **Year:** 2026 · **Venue:** [Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2026](https://md.research.elamur.ai/free/en/venues/eacl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.eacl-long.257/) · [PDF](https://aclanthology.org/2026.eacl-long.257.pdf) · [DOI 10.18653/v1/2026.eacl-long.257](https://doi.org/10.18653/v1/2026.eacl-long.257) · [OpenAlex](https://openalex.org/W7140081686) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary Policy-gradient RL methods such as PPO and GRPO are the standard way to post-train LLMs for reasoning. They need token-level likelihoods of sampled outputs, which masked diffusion language models (dLLMs) cannot provide cheaply. Existing workarounds either use expensive Monte-Carlo ELBO estimates or a one-step mean-field approximation (diffu-GRPO/d1), and the latter produces biased gradients. The paper introduces PADRE, which treats KL-regularized RL as inference toward a reward-tilted Boltzmann distribution. It replaces the exact sequence likelihood with a pseudo-likelihood objective: each output token is scored conditioned on all other output tokens and the prompt. The author argues this objective has the same optimum as standard KL-regularized RL. These conditionals are computed efficiently by appending a masked copy of the completion that shares position embeddings with it. Random masking of prompt tokens is added as a regularizer. The method fine-tunes LLaDA-8B with group-normalized rewards. It outperforms d1, wd1, VRPO and Dream 7B on GSM8K, MATH500 and MATH. This makes RL fine-tuning of reasoning-focused dLLMs more principled and practical. ## Key points - The base model is LLaDA-8B, trained with the code, hyperparameters, composite format-plus-correctness reward and 0-shot prompts from d1 (Zhao et al., 2025). - On a small test subset, the d1 one-step approximation diverges from the true likelihood with KL of 2.87 (GSM8K) and 3.15 (MATH500). PADRE's conditionals (PKL) show an average KL of roughly 1 or lower. - The best variant, PKL⁺ (with prompt perturbation), reaches 87.3% on GSM8K, 42.4% on MATH500 (+2.2 over d1-LLaDA) and 44.3% on MATH. Dream 7B scores 42.9% on MATH. - Two cheaper variants, PKL-U and PKL-T, are less accurate than PKL but still approximate the true likelihood better than d1. Both also beat d1 in accuracy. - Coding benchmarks (HumanEval, MBPP) were also evaluated; those results appear in the paper's appendix. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works 🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works ## Cite Cite as: Shiv Shankar. 2026. Pseudo-Likelihood Training for Reasoning Diffusion Language Models. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{shankar2026pseudo, title = {{Pseudo-Likelihood Training for Reasoning Diffusion Language Models}}, author = {Shiv Shankar}, booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)}, year = {2026}, doi = {10.18653/v1/2026.eacl-long.257}, url = {https://doi.org/10.18653/v1/2026.eacl-long.257} } ``` - [← previous Pruning Unsafe Tickets: A Resource-Efficient Framework for Safer and More Robust LLMs](https://md.research.elamur.ai/free/en/papers/si-2026-pruning-unsafe-tickets-resource-efficient/) - [next → PseudoSeer: a Search Engine for Pseudocode](https://md.research.elamur.ai/free/en/papers/toksoz-2026-pseudoseer-search-engine-pseudocode/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/40/) ## References 4 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021b. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732. 2. Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. 2024. [A general theoretical paradigm to understand learning from human preferences](https://md.research.elamur.ai/free/en/papers/azar-2023-general-theoretical-paradigm-understand-learning/). In International Conference on Artificial Intelligence and Statistics, pages 4447–4455. PMLR… 3. Leemon Baird. 1995. Residual algorithms: Reinforcement learning with function approximation. In Proceedings of the Twelfth International Conference on Machine Learning, pages 30–37. [find](https://scholar.google.com/scholar?q=Leemon+Baird%2E+1995%2E+Residual+algorithms%3A+Reinforcement+learning+with+function+approximation%2E+In+Proceedings+of+the+Twelfth+International+Conference+on+Machine+Learning%2C+pages+30%E2%80%9337%2E) 4. Julian Besag. 1975. Statistical analysis of non-lattice data. Journal of the Royal Statistical Society Series D: The Statistician, 24(3):179–195. \#\# Conclusion [find](https://scholar.google.com/scholar?q=Julian+Besag%2E+1975%2E+Statistical+analysis+of+non%2Dlattice+data%2E+Journal+of+the+Royal+Statistical+Society+Series+D%3A+The+Statistician%2C+24%283%29%3A179%E2%80%93195%2E+%5C%23%5C%23+Conclusion) *Updated 1 November 2026*