---
title: "Assessing the Belief Consistency of Large Language Models on"
description: "The authors build a synthetic 3-option MCQA game modelled on 20 Questions, in which any choice is valid. 17 references."
authors: ["Tomoki Tsujimura", "Matīss Rikters", "Masaki Asada", "Shusaku Egami", "Tatsuya Ishigaki", "Ken Yano", "Hiroya Takamura"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.1860/"
doi: "10.18653/v1/2026.acl-long.1860"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/tsujimura-2026-assessing-belief-consistency-large-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tsujimura-2026-assessing-belief-consistency-large-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/22/) / Assessing the Belief Consistency of Large Language Models on the Logical Conversation Process

# Assessing the Belief Consistency of Large Language Models on the Logical Conversation Process

**Authors:** Tomoki Tsujimura, Matīss Rikters, Masaki Asada, Shusaku Egami, Tatsuya Ishigaki, Ken Yano and [Hiroya Takamura](https://md.research.elamur.ai/free/en/authors/hiroya-takamura/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.1860/) · [PDF](https://aclanthology.org/2026.acl-long.1860.pdf) · [DOI 10.18653/v1/2026.acl-long.1860](https://doi.org/10.18653/v1/2026.acl-long.1860) · [OpenAlex](https://openalex.org/W7166866271)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
The paper asks whether an LLM's hidden preference stays logically coherent as a conversation grows. This matters if generated reasoning is to be read as a faithful trace of the model's state. Existing consistency benchmarks rely on questions with correct answers, so task skill confounds the measurement.

The authors build a synthetic 3-option MCQA game modelled on 20 Questions, in which any choice is valid. The model secretly picks one of three entities. In a "prior" conversation it reveals its pick immediately. In a "posterior" conversation, extra dialogue first rules out one option. Choice distributions are estimated from repeated free-form sampling and compared with Jensen–Shannon divergence, giving a score in [0, 1]. To limit bias, options have similar likelihood and results are averaged over all orderings.

Across open-weight model families, scaling alone does not raise belief consistency, while continual pre-training on code and math does. Yet these code/math models make more explicit logical errors, so surface-level and belief-level consistency are distinct properties.

## Key points
- The model's answer is mapped by string matching to a four-class distribution: the three options plus a verbalization error.
- The posterior context eliminates exactly one option. The expected update is therefore well defined, since the two remaining options should keep their relative preference.
- Qwen2.5 32B made almost no logical reasoning errors (4.44e-08%), whereas Qwen2.5-Coder 32B exceeded 10%. Qwen2.5-Coder models from 7B upward converge to similar invalid probabilities.
- Non-reasoning models were sampled 5,000 times per context, with 10 contexts per entity triple (50k generations per triple). This cost is why closed-weight models were not tested.
- Scores depend on prompt design: for example, Qwen3 results shift with thinking mode on or off. Only single-step inference was evaluated.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [National Institute of Advanced Industrial Science and Technology](https://md.research.elamur.ai/free/en/labs/national-institute-of-advanced-industrial-science-and-technology/) — Research institute, Tsukuba, Japan · 17 works · 2,760 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Tomoki Tsujimura, Matīss Rikters, Masaki Asada, Shusaku Egami, Tatsuya Ishigaki, Ken Yano and Hiroya Takamura. 2026. Assessing the Belief Consistency of Large Language Models on the Logical Conversation Process. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{tsujimura2026assessing,
  title = {{Assessing the Belief Consistency of Large Language Models on the Logical Conversation Process}},
  author = {Tomoki Tsujimura, Matīss Rikters, Masaki Asada, Shusaku Egami, Tatsuya Ishigaki, Ken Yano and Hiroya Takamura},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1860},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1860}
}
```

- [← previous Assessing Suicide Risk in Arabic Crisis Helpline Calls: A Comparison of Arabic and English Large Language Models](https://md.research.elamur.ai/free/en/papers/ma-2026-assessing-suicide-risk-arabic-crisis/)
- [next → Assessing the Difficulty of Inference Types in Natural Language Inference for Clinical Trials](https://md.research.elamur.ai/free/en/papers/aguiar-2026-assessing-difficulty-inference-types-natural/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/22/)

## References

17 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Leonardo Bertolazzi, Davide Mazzaccara, Filippo Merlo, and Raffaella Bernardi. 2023. ChatGPT’s information seeking strategy: Insights from the 20-questions game. In Proceedings of the 16th International Natural Language Generation Conference, pages 153–162, Prague, Czechia. Association for Computati… [find](https://scholar.google.com/scholar?q=Leonardo+Bertolazzi%2C+Davide+Mazzaccara%2C+Filippo+Merlo%2C+and+Raffaella+Bernardi%2E+2023%2E+ChatGPT%E2%80%99s+information+seeking+strategy%3A+Insights+from+the+20%2Dquestions+game%2E+In+Proceedings+of+the+16th+Internation)
2. Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, and 67 others. 2024. DeepSeek LLM: Scaling open-source language models with… [find](https://scholar.google.com/scholar?q=Xiao+Bi%2C+Deli+Chen%2C+Guanting+Chen%2C+Shanhuang+Chen%2C+Damai+Dai%2C+Chengqi+Deng%2C+Honghui+Ding%2C+Kai+Dong%2C+Qiushi+Du%2C+Zhe+Fu%2C+Huazuo+Gao%2C+Kaige+Gao%2C+Wenjun+Gao%2C+Ruiqi+Ge%2C+Kang+Guan%2C+Daya+Guo%2C+Jianzhong+Guo%2C+)
3. Fengxiang Cheng, Haoxuan Li, Fenrong Liu, Robert van Rooij, Kun Zhang, and Zhouchen Lin. 2025. Empowering LLMs with logical reasoning: A comprehensive survey. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25, pages 10400–10408. International Joi… [find](https://scholar.google.com/scholar?q=Fengxiang+Cheng%2C+Haoxuan+Li%2C+Fenrong+Liu%2C+Robert+van+Rooij%2C+Kun+Zhang%2C+and+Zhouchen+Lin%2E+2025%2E+Empowering+LLMs+with+logical+reasoning%3A+A+comprehensive+survey%2E+In+Proceedings+of+the+Thirty%2DFourth+Inter)
4. Zorik Gekhman, Eyal Ben-David, Hadas Orgad, Eran Ofek, Yonatan Belinkov, Idan Szpektor, Jonathan Herzig, and Roi Reichart. 2025. Inside-out: Hidden factual knowledge in LLMs. In Second Conference on Language Modeling. [find](https://scholar.google.com/scholar?q=Zorik+Gekhman%2C+Eyal+Ben%2DDavid%2C+Hadas+Orgad%2C+Eran+Ofek%2C+Yonatan+Belinkov%2C+Idan+Szpektor%2C+Jonathan+Herzig%2C+and+Roi+Reichart%2E+2025%2E+Inside%2Dout%3A+Hidden+factual+knowledge+in+LLMs%2E+In+Second+Conference+on+L)
5. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
6. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, and 175 others. 2025. DeepSeek-R1 incentivizes reasoning in LLMs through rein… [find](https://scholar.google.com/scholar?q=Daya+Guo%2C+Dejian+Yang%2C+Haowei+Zhang%2C+Junxiao+Song%2C+Peiyi+Wang%2C+Qihao+Zhu%2C+Runxin+Xu%2C+Ruoyu+Zhang%2C+Shirong+Ma%2C+Xiao+Bi%2C+Xiaokang+Zhang%2C+Xingkai+Yu%2C+Yu+Wu%2C+Z%2E+F%2E+Wu%2C+Zhibin+Gou%2C+Zhihong+Shao%2C+Zhuoshu+Li)
7. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, and 5 others. 2024. [Qwen2.5-Coder technical report](https://md.research.elamur.ai/free/en/papers/hui-2024-qwen2-5-coder-technical-report/). Preprint, arXiv:2409.12186.
8. Alex Irpan, Alexander Matt Turner, Mark Kurzeja, David K. Elson, and Rohin Shah. 2025. Consistency training helps stop sycophancy and jailbreaks. Preprint, arXiv:2510.27062. — [arXiv:2510.27062](https://arxiv.org/abs/2510.27062) [find](https://scholar.google.com/scholar?q=Alex+Irpan%2C+Alexander+Matt+Turner%2C+Mark+Kurzeja%2C+David+K%2E+Elson%2C+and+Rohin+Shah%2E+2025%2E+Consistency+training+helps+stop+sycophancy+and+jailbreaks%2E+Preprint%2C+arXiv%3A2510%2E27062%2E+%E2%80%94+%5BarXiv%3A2510%2E27062%5D%28https)
9. J. Lin. 1991. Divergence measures based on the shannon entropy. IEEE Transactions on Information Theory, 37(1):145–151. [find](https://scholar.google.com/scholar?q=J%2E+Lin%2E+1991%2E+Divergence+measures+based+on+the+shannon+entropy%2E+IEEE+Transactions+on+Information+Theory%2C+37%281%29%3A145%E2%80%93151%2E)
10. Dasha Metropolitansky and Jonathan Larson. 2025. [Towards effective extraction and evaluation of factual claims](https://md.research.elamur.ai/free/en/papers/metropolitansky-2025-effective-extraction-evaluation-factual-claims/). In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6996–7045, Vienna, Austria. Association for Computational Linguist…
11. Masanari Ohi, Masahiro Kaneko, Ryuto Koike, Mengsay Loem, and Naoaki Okazaki. 2024. Likelihood-based mitigation of evaluation bias in large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 3237–3245, Bangkok, Thailand. Association for Computational Lingu… [find](https://scholar.google.com/scholar?q=Masanari+Ohi%2C+Masahiro+Kaneko%2C+Ryuto+Koike%2C+Mengsay+Loem%2C+and+Naoaki+Okazaki%2E+2024%2E+Likelihood%2Dbased+mitigation+of+evaluation+bias+in+large+language+models%2E+In+Findings+of+the+Association+for+Computat)
12. Pouya Pezeshkpour and Estevam Hruschka. 2024. [Large language models sensitivity to the order of options in multiple-choice questions](https://md.research.elamur.ai/free/en/papers/pezeshkpour-2023-large-language-models-sensitivity-order/). In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2006–2017, Mexico City, Mexico. Association for Computational Linguistics.
13. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). In Advances in Neural Information Processing Systems, volume 36, pages 53728–53741. Curran Associates, Inc.
14. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, R…
15. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025a. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). Preprint, arXiv:2505.09388.
16. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 23 others. 2025b. Qwen2.5 technical report. Preprint, arXiv:24… [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+Huan+Lin%2C+Jian+Yang%2C+Jianhong+Tu%2C+Jianwei+Zhang%2C+Jianxin+Yang%2C+Jiaxi+Yang%2C+Jin)
17. Yizhe Zhang, Jiarui Lu, and Navdeep Jaitly. 2024. Probing the multi-turn planning capabilities of LLMs via 20 question games. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1495– 1516, Bangkok, Thailand. Association for Compu… [find](https://scholar.google.com/scholar?q=Yizhe+Zhang%2C+Jiarui+Lu%2C+and+Navdeep+Jaitly%2E+2024%2E+Probing+the+multi%2Dturn+planning+capabilities+of+LLMs+via+20+question+games%2E+In+Proceedings+of+the+62nd+Annual+Meeting+of+the+Association+for+Computati)

*Updated 1 November 2026*