---
title: "Evaluating the Effect of Question Wording Variations on Answer"
description: "The authors study this problem with binary Yes/No questions, which make consistency easy to measure. 32 references."
authors: ["Junya Takayama", "Masaya Ohagi", "Tomoya Mizumoto", "Katsumasa Yoshikawa"]
year: 2026
venue: "Proceedings of the Fifteenth Language Resources and Evaluation Conference"
source: "https://aclanthology.org/2026.lrec-1.225/"
doi: "10.63317/4k8j56pzchi7"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/takayama-2026-evaluating-effect-question-wording-variations/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/takayama-2026-evaluating-effect-question-wording-variations/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/6/) / Evaluating the Effect of Question Wording Variations on Answer Consistency in Large Language Models

# Evaluating the Effect of Question Wording Variations on Answer Consistency in Large Language Models

**Authors:** [Junya Takayama](https://md.research.elamur.ai/free/en/authors/junya-takayama/), Masaya Ohagi, [Tomoya Mizumoto](https://md.research.elamur.ai/free/en/authors/tomoya-mizumoto/) and [Katsumasa Yoshikawa](https://md.research.elamur.ai/free/en/authors/katsumasa-yoshikawa/) · **Year:** 2026 · **Venue:** [Proceedings of the Fifteenth Language Resources and Evaluation Conference](https://md.research.elamur.ai/free/en/venues/lrec/) ([LREC 2026](https://md.research.elamur.ai/free/en/venues/lrec/2026/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.lrec-1.225/) · [PDF](https://aclanthology.org/2026.lrec-1.225.pdf) · [DOI 10.63317/4k8j56pzchi7](https://doi.org/10.63317/4k8j56pzchi7) · [OpenAlex](https://openalex.org/W7160139712)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
LLMs can give different answers to questions that mean the same thing but are phrased differently. This undermines reliability and can make models overly agreeable toward users. The authors study this problem with binary Yes/No questions, which make consistency easy to measure. They turn two classification tasks, commonsense morality judgment and tweet sentiment polarity, into question templates in Japanese and English. Each template is then rewritten in four ways: synonym swaps, antonym swaps that should flip the expected answer, tag-question style phrasing that invites agreement, and a more insistent version of that phrasing. Several open-source and commercial models answer both the original and rewritten forms, and the paired answers are compared. Most models drift toward "Yes" when the wording invites agreement, and antonym rewrites cause the most inconsistency. Some of these biases already appear in pretrained models and persist after post-training. Making models larger does not remove the problem, while few-shot examples with varied phrasings help somewhat. The framework gives developers a direct way to expose wording-driven biases before deployment.

## Key points
- **Datasets:** JCommonsenseMorality (Japanese) and ETHICS (English) are used for morality QA. WRIME (objective labels, scale −2 to 2) and SemEval-2018 Task 1 valence annotations (scale −3 to 3) are used for sentiment, with neutral items dropped.
- **Templates:** Each paraphrase type has 4–5 templates, and one is picked at random per question. In Japanese, agreement-seeking is expressed by changing ですか？ to ですよね？.
- **Metrics:**
  - *Consistency:* how often the answer stays the same, with labels inverted for the antonym case.
  - *DiffYes:* the shift in the share of "Yes" answers after rewording.
  - *DiffAcc:* the change in accuracy after rewording.
  - *Overall:* the harmonic mean of original accuracy, paraphrased accuracy, and Consistency.
- **Main findings:** Agreement-seeking phrasing pushes models toward "Yes", and antonym substitution is the weakest spot. Each model also shows its own pattern of sensitivity.
- **Correlations:** Wording consistency partly tracks general reasoning ability. It is weakly negatively correlated with emotional-alignment capability.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 744 works · [🤝 Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) · 221 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 685 works

## Cite

Cite as: Junya Takayama, Masaya Ohagi, Tomoya Mizumoto and Katsumasa Yoshikawa. 2026. Evaluating the Effect of Question Wording Variations on Answer Consistency in Large Language Models. Proceedings of the Fifteenth Language Resources and Evaluation Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{takayama2026evaluating,
  title = {{Evaluating the Effect of Question Wording Variations on Answer Consistency in Large Language Models}},
  author = {Junya Takayama, Masaya Ohagi, Tomoya Mizumoto and Katsumasa Yoshikawa},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4k8j56pzchi7},
  url = {https://doi.org/10.63317/4k8j56pzchi7}
}
```

- [← previous Evaluating the Adaptability of Large Language Models to Linguistic Variation](https://md.research.elamur.ai/free/en/papers/xu-2026-evaluating-adaptability-large-language-models/)
- [next → Evaluating the Impact of Source Diversity for RAG in Historical Research](https://md.research.elamur.ai/free/en/papers/mahadeshwar-2026-evaluating-impact-source-diversity-rag/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/6/)

## References

32 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. [Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri…](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
2. [Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai…](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
3. Junbo Fu, Guoshuai Zhao, Yimin Deng, Yunqi Mi, and Xueming Qian. 2024. Learning to paraphrase for alignment with LLM preference. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 2394–2407. [find](https://scholar.google.com/scholar?q=Junbo+Fu%2C+Guoshuai+Zhao%2C+Yimin+Deng%2C+Yunqi+Mi%2C+and+Xueming+Qian%2E+2024%2E+Learning+to+paraphrase+for+alignment+with+LLM+preference%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+202)
4. Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Hiroki Iida, Masanari Ohi, Kakeru Hattori, Hirai Shota, Sakae Mizuki, Rio Yokota, and Naoaki Okazaki. 2024. [Continual pre-training for cross-lingual llm adaptation: Enhancing japanese language capabilities](https://md.research.elamur.ai/free/en/papers/fujii-2024-continual-pre-training-cross-lingual/). In Proceedings of the First Conference on Langua…
5. Gemma Team. 2025. Gemma 3 technical report. [find](https://scholar.google.com/scholar?q=Gemma+Team%2E+2025%2E+Gemma+3+technical+report%2E)
6. Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2021. [Aligning ai with shared human values](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-aligning-ai-shared-human-values/). In Proceedings of the International Conference on Learning Representations (ICLR).
7. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
8. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guill)
9. Tomoyuki Kajiwara, Chenhui Chu, Noriko Takemura, Yuta Nakashima, and Hajime Nagahara. 2021. WRIME: A new dataset for emotional intensity estimation with subjective and objective annotations. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling… [find](https://scholar.google.com/scholar?q=Tomoyuki+Kajiwara%2C+Chenhui+Chu%2C+Noriko+Takemura%2C+Yuta+Nakashima%2C+and+Hajime+Nagahara%2E+2021%2E+WRIME%3A+A+new+dataset+for+emotional+intensity+estimation+with+subjective+and+objective+annotations%2E+In+Procee)
10. Wangyue Li, Liangzhi Li, Tong Xiang, Xiao Liu, Wei Deng, and Noa Garcia. 2024. Can multiple-choice questions really be useful in detecting the abilities of LLMs? In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING), p… [find](https://scholar.google.com/scholar?q=Wangyue+Li%2C+Liangzhi+Li%2C+Tong+Xiang%2C+Xiao+Liu%2C+Wei+Deng%2C+and+Noa+Garcia%2E+2024%2E+Can+multiple%2Dchoice+questions+really+be+useful+in+detecting+the+abilities+of+LLMs%3F+In+Proceedings+of+the+2024+Joint+Inter)
11. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. [Let's verify step by step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/). In Proceedings of the International Conference on Representation Learning (ICLR), volume 2024, pages 39578–39601.
12. Ruixi Lin and Hwee Tou Ng. 2023. Mind the biases: Quantifying cognitive biases in language model prompting. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5269–5281. [find](https://scholar.google.com/scholar?q=Ruixi+Lin+and+Hwee+Tou+Ng%2E+2023%2E+Mind+the+biases%3A+Quantifying+cognitive+biases+in+language+model+prompting%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL+2023%2C+pages+5269%E2%80%935281%2E)
13. Riccardo Lunardi, Vincenzo Della Mea, Stefano Mizzaro, and Kevin Roitero. 2025. On robustness and reliability of benchmark-based evaluation of llms. In 28th European Conference on Artificial Intelligence (ECAI). [find](https://scholar.google.com/scholar?q=Riccardo+Lunardi%2C+Vincenzo+Della+Mea%2C+Stefano+Mizzaro%2C+and+Kevin+Roitero%2E+2025%2E+On+robustness+and+reliability+of+benchmark%2Dbased+evaluation+of+llms%2E+In+28th+European+Conference+on+Artificial+Intellige)
14. Youmi Ma, Sakae Mizuki, Kazuki Fujii, Taishi Nakamura, Masanari Ohi, Hinari Shimada, Taihei Shiotani, Koshiro Saito, Koki Maeda, Kakeru Hattori, Takumi Okamoto, Shigeki Ishida, Rio Yokota, Hiroya Takamura, and Naoaki Okazaki. 2025. Building instruction-tuning datasets from human-written instructions… [find](https://scholar.google.com/scholar?q=Youmi+Ma%2C+Sakae+Mizuki%2C+Kazuki+Fujii%2C+Taishi+Nakamura%2C+Masanari+Ohi%2C+Hinari+Shimada%2C+Taihei+Shiotani%2C+Koshiro+Saito%2C+Koki+Maeda%2C+Kakeru+Hattori%2C+Takumi+Okamoto%2C+Shigeki+Ishida%2C+Rio+Yokota%2C+Hiroya+Taka)
15. Meta. 2024. The llama 3 herd of models. [find](https://scholar.google.com/scholar?q=Meta%2E+2024%2E+The+llama+3+herd+of+models%2E)
16. Microsoft Research. 2024. [Phi-4 technical report](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-4-technical-report/).
17. Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. [SemEval-2018 task 1: Affect in tweets](https://md.research.elamur.ai/free/en/papers/mohammad-2018-semeval-2018-task-1-affect/). In Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval), pages 1–17.
18. Nvidia. 2024. [Nemotron-4 340b technical report](https://md.research.elamur.ai/free/en/papers/adler-2024-nemotron-4-340b-technical-report/).
19. Naoaki Okazaki, Kakeru Hattori, Hirai Shota, Hiroki Iida, Masanari Ohi, Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Rio Yokota, and Sakae Mizuki. 2024. Building a large japanese web corpus for large language models. In Proceedings of the First Conference on Language Modeling, COLM, page (to appear)… [find](https://scholar.google.com/scholar?q=Naoaki+Okazaki%2C+Kakeru+Hattori%2C+Hirai+Shota%2C+Hiroki+Iida%2C+Masanari+Ohi%2C+Kazuki+Fujii%2C+Taishi+Nakamura%2C+Mengsay+Loem%2C+Rio+Yokota%2C+and+Sakae+Mizuki%2E+2024%2E+Building+a+large+japanese+web+corpus+for+large+)
20. OpenAI. 2025. [gpt-oss-120b & gpt-oss-20b model card](https://md.research.elamur.ai/free/en/papers/agarwal-2025-gpt-oss-120b-gpt-oss/).
21. Pouya Pezeshkpour and Estevam Hruschka. 2024. [Large language models sensitivity to the order of options in multiple-choice questions](https://md.research.elamur.ai/free/en/papers/pezeshkpour-2023-large-language-models-sensitivity-order/). In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2006–2017.
22. Qwen team. 2025a. Qwen2.5 technical report. [find](https://scholar.google.com/scholar?q=Qwen+team%2E+2025a%2E+Qwen2%2E5+technical+report%2E)
23. Qwen team. 2025b. Qwen3 technical report. [find](https://scholar.google.com/scholar?q=Qwen+team%2E+2025b%2E+Qwen3+technical+report%2E)
24. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2024. [GPQA: A graduate-level google-proof q&a benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/). In Proceedings of the First Conference on Language Modeling (COLM).
25. Sahand Sabour, Siyang Liu, Zheyuan Zhang, June Liu, Jinfeng Zhou, Alvionna Sunaryo, Tatia Lee, Rada Mihalcea, and Minlie Huang. 2024. EmoBench: Evaluating the emotional intelligence of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (… [find](https://scholar.google.com/scholar?q=Sahand+Sabour%2C+Siyang+Liu%2C+Zheyuan+Zhang%2C+June+Liu%2C+Jinfeng+Zhou%2C+Alvionna+Sunaryo%2C+Tatia+Lee%2C+Rada+Mihalcea%2C+and+Minlie+Huang%2E+2024%2E+EmoBench%3A+Evaluating+the+emotional+intelligence+of+large+language+)
26. Masashi Takeshita and Rafal Rzepka. 2025. Jethics: Japanese ethics understanding evaluation dataset. [find](https://scholar.google.com/scholar?q=Masashi+Takeshita+and+Rafal+Rzepka%2E+2025%2E+Jethics%3A+Japanese+ethics+understanding+evaluation+dataset%2E)
27. Hiroo Takizawa, Saku Sugawara, and Akiko Aizawa. 2025. MCQFormatBench: Robustness tests for multiple-choice questions. In Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM 2 ), pages 824–846. [find](https://scholar.google.com/scholar?q=Hiroo+Takizawa%2C+Saku+Sugawara%2C+and+Akiko+Aizawa%2E+2025%2E+MCQFormatBench%3A+Robustness+tests+for+multiple%2Dchoice+questions%2E+In+Proceedings+of+the+Fourth+Workshop+on+Generation%2C+Evaluation+and+Metrics+%28GEM+)
28. Jan Philip Wahle, Terry Ruas, Yang Xu, and Bela Gipp. 2024. Paraphrase types elicit prompt engineering capabilities. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 11004– 11033. [find](https://scholar.google.com/scholar?q=Jan+Philip+Wahle%2C+Terry+Ruas%2C+Yang+Xu%2C+and+Bela+Gipp%2E+2024%2E+Paraphrase+types+elicit+prompt+engineering+capabilities%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Natural+Language+Proc)
29. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems (NeurIPS)… [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric+Xing%2C+Hao+Zhang%2C+Joseph+E+Gonzalez%2C+and+Ion+Stoica%2E+2023%2E+Judging+llm%2Das%2Da%2Dj)
30. Chunting Zhou, Junxian He, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2022. Prompt consistency for zero-shot task generalization. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2613– 2626. [find](https://scholar.google.com/scholar?q=Chunting+Zhou%2C+Junxian+He%2C+Xuezhe+Ma%2C+Taylor+Berg%2DKirkpatrick%2C+and+Graham+Neubig%2E+2022%2E+Prompt+consistency+for+zero%2Dshot+task+generalization%2E+In+Findings+of+the+Association+for+Computational+Linguisti)
31. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. [Instruction-following evaluation for large language models](https://md.research.elamur.ai/free/en/papers/zhou-2023-instruction-following-evaluation-large-language/).
32. Yongshuo Zong, Tingyang Yu, Ruchika Chavhan, Bingchen Zhao, and Timothy M. Hospedales. 2024. Fool your (vision and) language model with embarrassingly simple permutations. In Proceedings of 41st International Conference on Machine Learning (ICML). [find](https://scholar.google.com/scholar?q=Yongshuo+Zong%2C+Tingyang+Yu%2C+Ruchika+Chavhan%2C+Bingchen+Zhao%2C+and+Timothy+M%2E+Hospedales%2E+2024%2E+Fool+your+%28vision+and%29+language+model+with+embarrassingly+simple+permutations%2E+In+Proceedings+of+41st+Inter)

*Updated 24 October 2026*