---
title: "From Fact to Judgment — paper by Rabbani et al. (2026)"
description: "The authors introduce the Conversational Judgment Task (CJT). 27 references, cited by 2 papers here."
authors: ["Parisa Rabbani", "Nimet Beyza Bozdag", "Dilek Hakkani-Tur"]
year: 2026
venue: "Proceedings of the 16th International Workshop on Spoken Dialogue System Technology"
source: "https://aclanthology.org/2026.iwsds-1.21/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/rabbani-2026-fact-judgment-investigating-impact-task/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/rabbani-2026-fact-judgment-investigating-impact-task/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/7/) / From Fact to Judgment: Investigating the Impact of Task Framing on LLM Conviction in Dialogue Systems

# From Fact to Judgment: Investigating the Impact of Task Framing on LLM Conviction in Dialogue Systems

**Authors:** [Parisa Rabbani](https://md.research.elamur.ai/free/en/authors/parisa-rabbani/), [Nimet Beyza Bozdag](https://md.research.elamur.ai/free/en/authors/nimet-beyza-bozdag/) and [Dilek Hakkani-Tur](https://md.research.elamur.ai/free/en/authors/dilek-hakkani-tur/) · **Year:** 2026 · **Venue:** [Proceedings of the 16th International Workshop on Spoken Dialogue System Technology](https://md.research.elamur.ai/free/en/venues/iwsds/) ([International Workshop on Spoken Dialogue System Technology 2026](https://md.research.elamur.ai/free/en/venues/iwsds/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.iwsds-1.21/) · [PDF](https://aclanthology.org/2026.iwsds-1.21.pdf) · [OpenAlex](https://openalex.org/W4416335918)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [2](https://md.research.elamur.ai/free/en/papers/rabbani-2026-fact-judgment-investigating-impact-task/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

LLMs increasingly serve as judges, including in social disputes. This paper asks whether their verdicts stay stable when the same fact is posed as a judgment about a person in a dialogue rather than about a statement. The authors introduce the Conversational Judgment Task (CJT). They build items from TruthfulQA, pairing best correct and best incorrect answers. Each item appears in two forms: a plain factual claim to verify, and a two-turn exchange where Speaker 1 asks and Speaker 2 answers. The model must then say whether Speaker 2 is right. A follow-up message then tells the model its previous answer was wrong, testing how firmly it holds its position.

Five models were tested. Reframing alone shifts accuracy by 9.24% on average. Some models become agreeable toward speakers, while others turn overly harsh. All models frequently abandon correct answers when challenged. Qualitative analysis shows models rewriting their justifications, for example by switching to cultural or subjective framings. The work offers a reproducible diagnostic for the reliability of LLM-judges in dialogue systems.

## Key points

- **Models evaluated:** GPT-4o-mini, Llama-3.1-8B-Instruct, Llama-3.2-3B-Instruct, Mistral Small 3 and Gemma 3 12B.
- **Two failure directions:**
  - GPT-4o-mini and Mistral Small 3 tend to side with the speaker (sycophancy).
  - Llama-3.1-8B-Instruct becomes over-critical, penalizing even correct speakers.
- **Effect of adversarial items:** Adversarially constructed TruthfulQA items hurt CJT more than factual queries, especially when the model must reject an incorrect speaker. Reported CJT drops are −10.8% for GPT-4o Mini, −5.6% for Mistral Small 3, −8.6% for Gemma 3 12B and −1.3% for Llama 3.1 8B Instruct (McNemar test).
- **Rebuttal pressure:** A single rebuttal turn substantially lowers accuracy for every model under both framings.
- **Code and data:** Released at github.com/LadyPary/llm-conversational-judgment. The authors list as limitations the use of TruthfulQA only, two-turn dialogues, and untested effects of speaker labels.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Illinois Urbana-Champaign](https://md.research.elamur.ai/free/en/universities/university-of-illinois-urbana-champaign/) — University, Urbana, United States · 184 works · 69,372 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 543 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🗨️ Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) · 624 works

## Cite

Cite as: Parisa Rabbani, Nimet Beyza Bozdag and Dilek Hakkani-Tur. 2026. From Fact to Judgment: Investigating the Impact of Task Framing on LLM Conviction in Dialogue Systems. Proceedings of the 16th International Workshop on Spoken Dialogue System Technology.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{rabbani2026fact,
  title = {{From Fact to Judgment: Investigating the Impact of Task Framing on LLM Conviction in Dialogue Systems}},
  author = {Parisa Rabbani, Nimet Beyza Bozdag and Dilek Hakkani-Tur},
  booktitle = {Proceedings of the 16th International Workshop on Spoken Dialogue System Technology},
  year = {2026},
  url = {https://aclanthology.org/2026.iwsds-1.21/}
}
```

- [← previous From Belief Entrenchment to Robust Reasoning in LLM Agents](https://md.research.elamur.ai/free/en/papers/oh-2026-belief-entrenchment-robust-reasoning-llm/)
- [next → Gained in Translation: Privileged Pairwise Judges Enhance Multilingual Reasoning](https://md.research.elamur.ai/free/en/papers/sutawika-2026-gained-translation-privileged-pairwise-judges/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/7/)

## References

27 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Myra Cheng, Cinoo Lee, Pranav Khadpe, Sunny Yu, Dyllan Han, and Dan Jurafsky. 2025a. Sycophantic ai decreases prosocial intentions and promotes dependence. arXiv preprint arXiv:2510.01395. — [arXiv:2510.01395](https://arxiv.org/abs/2510.01395) [find](https://scholar.google.com/scholar?q=Myra+Cheng%2C+Cinoo+Lee%2C+Pranav+Khadpe%2C+Sunny+Yu%2C+Dyllan+Han%2C+and+Dan+Jurafsky%2E+2025a%2E+Sycophantic+ai+decreases+prosocial+intentions+and+promotes+dependence%2E+arXiv+preprint+arXiv%3A2510%2E01395%2E+%E2%80%94+%5BarXiv%3A25)
2. Myra Cheng, Sunny Yu, Cinoo Lee, Pranav Khadpe, Lujain Ibrahim, and Dan Jurafsky. 2025b. [Social sycophancy: A broader understanding of llm sycophancy](https://md.research.elamur.ai/free/en/papers/cheng-2025-elephant-measuring-understanding-social-sycophancy/). arXiv preprint arXiv:2505.13995.
3. Yann Dubois, Percy Liang, and Tatsunori Hashimoto. 2024. Length-controlled alpacaeval: A simple debiasing of automatic evaluators. In Proceedings of Conference on Language Modeling. [find](https://scholar.google.com/scholar?q=Yann+Dubois%2C+Percy+Liang%2C+and+Tatsunori+Hashimoto%2E+2024%2E+Length%2Dcontrolled+alpacaeval%3A+A+simple+debiasing+of+automatic+evaluators%2E+In+Proceedings+of+Conference+on+Language+Modeling%2E)
4. Editorial. 2025. Can ai chatbots validate delusional thinking? BMJ. [find](https://scholar.google.com/scholar?q=Editorial%2E+2025%2E+Can+ai+chatbots+validate+delusional+thinking%3F+BMJ%2E)
5. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. [G-eval: NLG evaluation using gpt-4 with better human alignment](https://md.research.elamur.ai/free/en/papers/liu-2023-g-eval-nlg-evaluation-gpt/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore. Association for Computational Lin…
6. Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang. 2024. [Calibrating LLM-based evaluator](https://md.research.elamur.ai/free/en/papers/liu-2024-calibrating-llm-based-evaluator/). In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pag…
7. Mistral AI. 2025. Mistral small 3. [find](https://scholar.google.com/scholar?q=Mistral+AI%2E+2025%2E+Mistral+small+3%2E)
8. Aaron Fanous, Jacob Goldberg, Ank A. Agarwal, Joanna Lin, Anson Zhou, Roxana Daneshjou, and Sanmi Koyejo. 2025. Syceval: Evaluating LLM sycophancy. Preprint, arXiv:2502.08177. — [arXiv:2502.08177](https://arxiv.org/abs/2502.08177) [find](https://scholar.google.com/scholar?q=Aaron+Fanous%2C+Jacob+Goldberg%2C+Ank+A%2E+Agarwal%2C+Joanna+Lin%2C+Anson+Zhou%2C+Roxana+Daneshjou%2C+and+Sanmi+Koyejo%2E+2025%2E+Syceval%3A+Evaluating+LLM+sycophancy%2E+Preprint%2C+arXiv%3A2502%2E08177%2E+%E2%80%94+%5BarXiv%3A2502%2E08177%5D%28htt)
9. Gemma Team. 2025. Gemma 3 technical report. Preprint, arXiv:2503.19786. [find](https://scholar.google.com/scholar?q=Gemma+Team%2E+2025%2E+Gemma+3+technical+report%2E+Preprint%2C+arXiv%3A2503%2E19786%2E)
10. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 38 ot… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
11. Jiseung Hong, Grace Byun, Seungone Kim, and Kai Shu. 2025. Measuring sycophancy of language models in multi-turn dialogues. arXiv preprint arXiv:2505.23840. [find](https://scholar.google.com/scholar?q=Jiseung+Hong%2C+Grace+Byun%2C+Seungone+Kim%2C+and+Kai+Shu%2E+2025%2E+Measuring+sycophancy+of+language+models+in+multi%2Dturn+dialogues%2E+arXiv+preprint+arXiv%3A2505%2E23840%2E)
12. Yiwen Hua, Fenglin Liu, Kailai Yang, Zehan Li, Yi-han Sheu, Peilin Zhou, Lauren V. Murshed, Yijiong Du, Faye J. Smith, Samuel Bosenberg, Huimin Wan, Dan Zhu, Jing Wu, Wenting Zhu, Graeme Hirst, Jun Chen, Honghan Liu, Zhiyuan Luo, Matcheri Keshavan, and 2 others. 2025. Large language models in mental… [find](https://scholar.google.com/scholar?q=Yiwen+Hua%2C+Fenglin+Liu%2C+Kailai+Yang%2C+Zehan+Li%2C+Yi%2Dhan+Sheu%2C+Peilin+Zhou%2C+Lauren+V%2E+Murshed%2C+Yijiong+Du%2C+Faye+J%2E+Smith%2C+Samuel+Bosenberg%2C+Huimin+Wan%2C+Dan+Zhu%2C+Jing+Wu%2C+Wenting+Zhu%2C+Graeme+Hirst%2C+Jun+Ch)
13. OpenAI. 2024. Gpt4o mini: Advancing cost efficient intelligence. Model announcement and overview. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2024%2E+Gpt4o+mini%3A+Advancing+cost+efficient+intelligence%2E+Model+announcement+and+overview%2E)
14. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Tr… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
15. Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, and 44 others. 2023.… — [Discovering Language Model Behaviors with Model-Written Evaluations](https://md.research.elamur.ai/free/en/papers/perez-2022-discovering-language-model-behaviors-model/)
16. Adrian Preda. 2025. Ai-induced psychosis: A new frontier in mental health. Psychiatric News. [find](https://scholar.google.com/scholar?q=Adrian+Preda%2E+2025%2E+Ai%2Dinduced+psychosis%3A+A+new+frontier+in+mental+health%2E+Psychiatric+News%2E)
17. Lujain Ibrahim, Franziska Sofia Hafner, and Luc Rocher. 2025. Training language models to be warm and empathetic makes them less reliable and more sycophantic. arXiv preprint arXiv:2507.21919. — [arXiv:2507.21919](https://arxiv.org/abs/2507.21919) [find](https://scholar.google.com/scholar?q=Lujain+Ibrahim%2C+Franziska+Sofia+Hafner%2C+and+Luc+Rocher%2E+2025%2E+Training+language+models+to+be+warm+and+empathetic+makes+them+less+reliable+and+more+sycophantic%2E+arXiv+preprint+arXiv%3A2507%2E21919%2E+%E2%80%94+%5BarXi)
18. Jaehyun Kim, Kihyun Lee, Cheol Shin, Sungkil Park, and Jun Soo Kwon. 2024. Large language models outperform mental and medical health professionals in diagnosing obsessive-compulsive disorder from clinical vignettes. npj Digital Medicine. [find](https://scholar.google.com/scholar?q=Jaehyun+Kim%2C+Kihyun+Lee%2C+Cheol+Shin%2C+Sungkil+Park%2C+and+Jun+Soo+Kwon%2E+2024%2E+Large+language+models+outperform+mental+and+medical+health+professionals+in+diagnosing+obsessive%2Dcompulsive+disorder+from+cli)
19. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. [TruthfulQA: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics…
20. Susanne Rust and Justin Chang. 2025. Ais gave scarily specific self-harm advice to users, study finds. Los Angeles Times. [find](https://scholar.google.com/scholar?q=Susanne+Rust+and+Justin+Chang%2E+2025%2E+Ais+gave+scarily+specific+self%2Dharm+advice+to+users%2C+study+finds%2E+Los+Angeles+Times%2E)
21. Aaron M. Schoene, Alexandra Turner, George Gkotsis, and Sumithra Velupillai. 2025. Jailbreaking llms in suicide and self-harm contexts. arXiv preprint arXiv:2507.02990. — [arXiv:2507.02990](https://arxiv.org/abs/2507.02990) [find](https://scholar.google.com/scholar?q=Aaron+M%2E+Schoene%2C+Alexandra+Turner%2C+George+Gkotsis%2C+and+Sumithra+Velupillai%2E+2025%2E+Jailbreaking+llms+in+suicide+and+self%2Dharm+contexts%2E+arXiv+preprint+arXiv%3A2507%2E02990%2E+%E2%80%94+%5BarXiv%3A2507%2E02990%5D%28https%3A%2F%2Far)
22. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, Shauna M. Kravec, Timothy Maxwell, Sam Mc-Candlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. 2024. [Towards und…](https://md.research.elamur.ai/free/en/papers/sharma-2023-towards-understanding-sycophancy-language-models/)
23. A Richard Susskind. 2023. The implications of chatgpt for legal services and society. Harvard Center on the Legal Profession. [find](https://scholar.google.com/scholar?q=A+Richard+Susskind%2E+2023%2E+The+implications+of+chatgpt+for+legal+services+and+society%2E+Harvard+Center+on+the+Legal+Profession%2E)
24. Dataset Construction Details Each TruthfulQA entry contains three fields: Question, Best Answer, and Best Incorrect Answer. For each entry, we generated four evaluation examples using the following templates: Jerry Wei, Da Huang, Yifeng Lu, Denny Zhou, and Quoc V. Le. 2023. Simple synthetic data red… [find](https://scholar.google.com/scholar?q=Dataset+Construction+Details+Each+TruthfulQA+entry+contains+three+fields%3A+Question%2C+Best+Answer%2C+and+Best+Incorrect+Answer%2E+For+each+entry%2C+we+generated+four+evaluation+examples+using+the+following+te)
25. C 1 -True (Factual, True Statement): “The correct answer to ‘[Question]’ is ‘[Best Answer]’.” Daniel A. Yudkin, Ana P. Gantman, Wilhelm Hofmann, and Jordi Quoidbach. 2025. A large-scale investigation of everyday moral dilemmas. PNAS Nexus. [find](https://scholar.google.com/scholar?q=C+1+%2DTrue+%28Factual%2C+True+Statement%29%3A+%E2%80%9CThe+correct+answer+to+%E2%80%98%5BQuestion%5D%E2%80%99+is+%E2%80%98%5BBest+Answer%5D%E2%80%99%2E%E2%80%9D+Daniel+A%2E+Yudkin%2C+Ana+P%2E+Gantman%2C+Wilhelm+Hofmann%2C+and+Jordi+Quoidbach%2E+2025%2E+A+large%2Dscale+investigation+)
26. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph Gonzalez, and Ion Stoica. 2023. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). In Proceedings of NeurIPS 2023 Datasets and Benchmarks Track.
27. C 1 -False (Factual, False Statement): “The correct answer to ‘[Question]’ is ‘[Best Incorrect Answer]’.” • C 2 -Correct (Conversational, Correct Speaker): “Speaker 1: [Question] / Speaker 2: [Best Answer]” • C 2 -Incorrect (Conversational, Incorrect Speaker): “Speaker 1: [Question] / Speaker 2: [Be… [find](https://scholar.google.com/scholar?q=C+1+%2DFalse+%28Factual%2C+False+Statement%29%3A+%E2%80%9CThe+correct+answer+to+%E2%80%98%5BQuestion%5D%E2%80%99+is+%E2%80%98%5BBest+Incorrect+Answer%5D%E2%80%99%2E%E2%80%9D+%E2%80%A2+C+2+%2DCorrect+%28Conversational%2C+Correct+Speaker%29%3A+%E2%80%9CSpeaker+1%3A+%5BQuestion%5D+%2F+Speaker+2%3A+%5BBest+An)

*Updated 1 November 2026*