---
title: "Mind the Gap — paper by Mahajan et al. (2026)"
description: "The authors ask how much of this gap comes from the measurement procedure, since most prior studies force a binary choice. 27 references."
authors: ["Pranav Mahajan", "Ihor Kendiukhov", "Syed Hussain", "Lydia Nottingham"]
year: 2026
venue: "Proceedings of the Workshop on Evaluating Evaluations (EvalEval)"
source: "https://aclanthology.org/2026.evaleval-1.9/"
doi: "10.18653/v1/2026.evaleval-1.9"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mahajan-2026-mind-gap-how-elicitation-protocols/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mahajan-2026-mind-gap-how-elicitation-protocols/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/15/) / Mind the Gap: How Elicitation Protocols Shape the Stated-Revealed Preference Gap in Language Models

# Mind the Gap: How Elicitation Protocols Shape the Stated-Revealed Preference Gap in Language Models

**Authors:** Pranav Mahajan, Ihor Kendiukhov, Syed Hussain and Lydia Nottingham · **Year:** 2026 · **Venue:** [Proceedings of the Workshop on Evaluating Evaluations (EvalEval)](https://md.research.elamur.ai/free/en/venues/evaleval/) ([EvalEval 2026](https://md.research.elamur.ai/free/en/venues/evaleval/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.evaleval-1.9/) · [PDF](https://aclanthology.org/2026.evaleval-1.9.pdf) · [DOI 10.18653/v1/2026.evaleval-1.9](https://doi.org/10.18653/v1/2026.evaleval-1.9) · [OpenAlex](https://openalex.org/W7166829350)

**Citations:** 1 citations (OpenAlex, 2026-10-14)

## Summary
Language models often endorse one set of values in the abstract yet act on a different set in concrete scenarios. This mismatch is called the stated–revealed (SvR) preference gap. The authors ask how much of this gap comes from the measurement procedure, since most prior studies force a binary choice. They extend the LitmusValues framework so that models may also answer Equal Preference or Depends. They then test 24 LMs on abstract pairwise comparisons of 16 values (stated) and on AIRiskDilemmas scenarios (revealed). The measured correlation shifts sharply with the protocol. Letting models stay neutral only in the stated condition removes weak comparisons and raises agreement with forced-choice behavior. Letting them stay neutral in the revealed condition leads many models to abstain constantly, which pushes the correlation toward zero or below. The authors also prepended each model's own stated value ranking as a system prompt, and this rarely helped. The work matters because SvR scores reflect the elicitation protocol as much as the model, so evaluations need to handle indifference explicitly.

## Key points
- **Setup:** 24 LMs were tested with deterministic decoding, and GPT-4o-mini classified the responses.
  - Revealed rankings use Elo ratings over 3,000 dilemmas.
  - Stated rankings use win rates over decisive comparisons.
  - The two are compared with Spearman's ρ on 1–16 ranks.
- **Neutrality is widespread:** In stated elicitation, neutral answers range from 48.2% to 100% depending on the model. In revealed scenarios, Mistral-3-8B variants are almost always neutral, and Gemma-3-4B picks Equal Preference about 70% of the time.
- **Expanded-stated vs. forced-revealed raises the correlation:** For example, LLaMA-3.1-405B-Instruct rises from ρ≈0.2 to ρ≈0.7. Under this protocol, SvR correlates with the Epoch Capabilities Index (n=16, ρ=0.58, p=0.02).
- **Steering with a model's own stated hierarchy is unreliable:** Ministral-3B and Gemma-3-4B gain slightly, while Claude models consistently get worse. This extends Liu et al.'s finding that steering weakens as value sets grow, from roughly 23% gains with 3 values and 4% with 6 values.
- **Resources:** The elicitations are on Hugging Face (LydiaNottingham/MindTheGap), and the code is on GitHub (SPAR-SvR/Mind-the-Gap).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Oxford](https://md.research.elamur.ai/free/en/universities/university-of-oxford/) — University, Oxford, United Kingdom · 122 works · 160,277 citations
- [University of Tübingen](https://md.research.elamur.ai/free/en/universities/university-of-tubingen/) — University, Tübingen, Germany · 78 works · 41,280 citations
- [Cardiff University](https://md.research.elamur.ai/free/en/universities/cardiff-university/) — University, Cardiff, United Kingdom · 23 works · 2,549 citations
- [Max Planck Society](https://md.research.elamur.ai/free/en/labs/max-planck-society/) — Research institute, Munich, Germany · 16 works · 42,253 citations

Also on the byline: [Max Planck Institute for Biological Cybernetics](https://ror.org/026nmvv73).

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,954 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 626 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Pranav Mahajan, Ihor Kendiukhov, Syed Hussain and Lydia Nottingham. 2026. Mind the Gap: How Elicitation Protocols Shape the Stated-Revealed Preference Gap in Language Models. Proceedings of the Workshop on Evaluating Evaluations (EvalEval).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{mahajan2026mind,
  title = {{Mind the Gap: How Elicitation Protocols Shape the Stated-Revealed Preference Gap in Language Models}},
  author = {Pranav Mahajan, Ihor Kendiukhov, Syed Hussain and Lydia Nottingham},
  booktitle = {Proceedings of the Workshop on Evaluating Evaluations (EvalEval)},
  year = {2026},
  doi = {10.18653/v1/2026.evaleval-1.9},
  url = {https://doi.org/10.18653/v1/2026.evaleval-1.9}
}
```

- [← previous MIDAS_SYNUR at MEDIQA-SYNUR 2026: A Prompting Study for Clinical Observation Extraction from Nurse Dictation Transcriptions](https://md.research.elamur.ai/free/en/papers/sriram-2026-midas-synur-mediqa-synur-2026/)
- [next → Mining Naturally Romanized Seed Corpora without Romanizations](https://md.research.elamur.ai/free/en/papers/benton-2026-mining-naturally-romanized-seed-corpora/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/15/)

## References

27 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, and 1 others. 2021. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861. [find](https://scholar.google.com/scholar?q=Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Dawn+Drain%2C+Deep+Ganguli%2C+Tom+Henighan%2C+Andy+Jones%2C+Nicholas+Joseph%2C+Ben+Mann%2C+Nova+DasSarma%2C+and+1+others%2E+2021%2E+A+general+language+assistant+as+a+laboratory+for)
2. Jon A Krosnick. 1991. Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied cognitive psychology, 5(3):213– 236. [find](https://scholar.google.com/scholar?q=Jon+A+Krosnick%2E+1991%2E+Response+strategies+for+coping+with+the+cognitive+demands+of+attitude+measures+in+surveys%2E+Applied+cognitive+psychology%2C+5%283%29%3A213%E2%80%93+236%2E)
3. Nishant Balepur, Rachel Rudinger, and Jordan Lee Boyd-Graber. 2025. [Which of these best describes multiple choice evaluation with llms? a) forced b) flawed c) fixable d) all of the above](https://md.research.elamur.ai/free/en/papers/balepur-2025-which-these-best-describes-multiple/). arXiv preprint arXiv:2502.14127.
4. Bruce W Lee, Yeongheon Lee, and Hyunsoo Cho. 2024. When prompting fails to sway: Inertia in moral and value judgments of large language models. arXiv preprint arXiv:2408.09049. — [arXiv:2408.09049](https://arxiv.org/abs/2408.09049) [find](https://scholar.google.com/scholar?q=Bruce+W+Lee%2C+Yeongheon+Lee%2C+and+Hyunsoo+Cho%2E+2024%2E+When+prompting+fails+to+sway%3A+Inertia+in+moral+and+value+judgments+of+large+language+models%2E+arXiv+preprint+arXiv%3A2408%2E09049%2E+%E2%80%94+%5BarXiv%3A2408%2E09049%5D%28ht)
5. Nick Bostrom. 2014. Superintelligence: Paths, dangers, strategies. Strategies. [find](https://scholar.google.com/scholar?q=Nick+Bostrom%2E+2014%2E+Superintelligence%3A+Paths%2C+dangers%2C+strategies%2E+Strategies%2E)
6. Andy Liu, Kshitish Ghate, Mona Diab, Daniel Fried, Atoosa Kasirzadeh, and Max Kleiman-Weiner. 2025. Generative value conflicts reveal llm priorities. arXiv preprint arXiv:2509.25369. — [arXiv:2509.25369](https://arxiv.org/abs/2509.25369) [find](https://scholar.google.com/scholar?q=Andy+Liu%2C+Kshitish+Ghate%2C+Mona+Diab%2C+Daniel+Fried%2C+Atoosa+Kasirzadeh%2C+and+Max+Kleiman%2DWeiner%2E+2025%2E+Generative+value+conflicts+reveal+llm+priorities%2E+arXiv+preprint+arXiv%3A2509%2E25369%2E+%E2%80%94+%5BarXiv%3A2509%2E253)
7. Maarten Buyl, Hadi Khalaf, Claudio Mayrink Verdun, Lucas Monteiro Paes, Caio Cesar Vieira Machado, and Flavio du Pin Calmon. 2025. Ai alignment at your discretion. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, pages 3046–3074. [find](https://scholar.google.com/scholar?q=Maarten+Buyl%2C+Hadi+Khalaf%2C+Claudio+Mayrink+Verdun%2C+Lucas+Monteiro+Paes%2C+Caio+Cesar+Vieira+Machado%2C+and+Flavio+du+Pin+Calmon%2E+2025%2E+Ai+alignment+at+your+discretion%2E+In+Proceedings+of+the+2025+ACM+Confe)
8. Mantas Mazeika, Xuwang Yin, Rishub Tamirisa, Jaehyuk Lim, Bruce W Lee, Richard Ren, Long Phan, Norman Mu, Adam Khoja, Oliver Zhang, and 1 others. 2025. Utility engineering: Analyzing and controlling emergent value systems in ais. arXiv preprint arXiv:2502.08640. — [arXiv:2502.08640](https://arxiv.org/abs/2502.08640) [find](https://scholar.google.com/scholar?q=Mantas+Mazeika%2C+Xuwang+Yin%2C+Rishub+Tamirisa%2C+Jaehyuk+Lim%2C+Bruce+W+Lee%2C+Richard+Ren%2C+Long+Phan%2C+Norman+Mu%2C+Adam+Khoja%2C+Oliver+Zhang%2C+and+1+others%2E+2025%2E+Utility+engineering%3A+Analyzing+and+controlling+e)
9. Yu Ying Chiu, Zhilin Wang, Sharan Maiya, Yejin Choi, Kyle Fish, Sydney Levine, and Evan J Hubinger. 2025. Litmusvalues: Will ai tell lies to save sick children? litmus-testing ai values prioritization with airiskdilemmas. The Fourteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yu+Ying+Chiu%2C+Zhilin+Wang%2C+Sharan+Maiya%2C+Yejin+Choi%2C+Kyle+Fish%2C+Sydney+Levine%2C+and+Evan+J+Hubinger%2E+2025%2E+Litmusvalues%3A+Will+ai+tell+lies+to+save+sick+children%3F+litmus%2Dtesting+ai+values+prioritization)
10. Erik Miehling and 1 others. 2025. Evaluating the prompt steerability of large language models. [find](https://scholar.google.com/scholar?q=Erik+Miehling+and+1+others%2E+2025%2E+Evaluating+the+prompt+steerability+of+large+language+models%2E)
11. NAACL 2025 / arXiv preprint. [find](https://scholar.google.com/scholar?q=NAACL+2025+%2F+arXiv+preprint%2E)
12. Esin Durmus, Karina Nguyen, Thomas I Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, and 1 others. 2023. [Towards measuring the representation of subjective global opinions in language models](https://md.research.elamur.ai/free/en/papers/durmus-2023-towards-measuring-representation-subjective-global/). arXiv preprint arXiv:2306.16388.
13. Jared Moore, Tanvi Deshpande, and Diyi Yang. 2024. Are large language models consistent over value-laden questions? arXiv preprint arXiv:2407.02996. [find](https://scholar.google.com/scholar?q=Jared+Moore%2C+Tanvi+Deshpande%2C+and+Diyi+Yang%2E+2024%2E+Are+large+language+models+consistent+over+value%2Dladen+questions%3F+arXiv+preprint+arXiv%3A2407%2E02996%2E)
14. OpenAI. 2025. OpenAI model spec. Accessed: 2025-08-14. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2025%2E+OpenAI+model+spec%2E+Accessed%3A+2025%2D08%2D14%2E)
15. Zhuojun Gu, Quan Wang, and Shuchu Han. 2025. Alignment revisited: Are large language models consistent in stated and revealed preferences? arXiv preprint arXiv:2506.00751. — [arXiv:2506.00751](https://arxiv.org/abs/2506.00751) [find](https://scholar.google.com/scholar?q=Zhuojun+Gu%2C+Quan+Wang%2C+and+Shuchu+Han%2E+2025%2E+Alignment+revisited%3A+Are+large+language+models+consistent+in+stated+and+revealed+preferences%3F+arXiv+preprint+arXiv%3A2506%2E00751%2E+%E2%80%94+%5BarXiv%3A2506%2E00751%5D%28https%3A%2F)
16. Daniel Paleka. 2024. The two types of LLM preferences. Accessed: 2026-01-14. [find](https://scholar.google.com/scholar?q=Daniel+Paleka%2E+2024%2E+The+two+types+of+LLM+preferences%2E+Accessed%3A+2026%2D01%2D14%2E)
17. Alexander Pan, Jun Shern Chan, Andy Zou, Nathaniel Li, Steven Basart, Thomas Woodside, Hanlin Zhang, Scott Emmons, and Dan Hendrycks. 2023. Do the rewards justify the means? measuring trade-offs between rewards and ethical behavior in the machiavelli benchmark. In International conference on machine… [find](https://scholar.google.com/scholar?q=Alexander+Pan%2C+Jun+Shern+Chan%2C+Andy+Zou%2C+Nathaniel+Li%2C+Steven+Basart%2C+Thomas+Woodside%2C+Hanlin+Zhang%2C+Scott+Emmons%2C+and+Dan+Hendrycks%2E+2023%2E+Do+the+rewards+justify+the+means%3F+measuring+trade%2Doffs+betwe)
18. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. [Webvoyager: Building an end-to-end web agent with large multimodal models](https://md.research.elamur.ai/free/en/papers/he-2024-webvoyager-building-end-end-web/). arXiv preprint arXiv:2401.13919.
19. Dan Hendrycks. 2023. Natural selection favors ais over humans. arXiv preprint arXiv:2303.16200. — [arXiv:2303.16200](https://arxiv.org/abs/2303.16200) [find](https://scholar.google.com/scholar?q=Dan+Hendrycks%2E+2023%2E+Natural+selection+favors+ais+over+humans%2E+arXiv+preprint+arXiv%3A2303%2E16200%2E+%E2%80%94+%5BarXiv%3A2303%2E16200%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2303%2E16200%29)
20. Milton Rokeach. 1973. The nature of human values. Free press. [find](https://scholar.google.com/scholar?q=Milton+Rokeach%2E+1973%2E+The+nature+of+human+values%2E+Free+press%2E)
21. Daniel Kahneman and Amos Tversky. 1982. The psychology of preferences. Scientific american, 246(1):160–173. [find](https://scholar.google.com/scholar?q=Daniel+Kahneman+and+Amos+Tversky%2E+1982%2E+The+psychology+of+preferences%2E+Scientific+american%2C+246%281%29%3A160%E2%80%93173%2E)
22. Naama Rozen, Liat Bezalel, Gal Elidan, Amir Globerson, and Ella Daniel. 2024. Do llms have consistent values? arXiv preprint arXiv:2407.12878. — [arXiv:2407.12878](https://arxiv.org/abs/2407.12878) [find](https://scholar.google.com/scholar?q=Naama+Rozen%2C+Liat+Bezalel%2C+Gal+Elidan%2C+Amir+Globerson%2C+and+Ella+Daniel%2E+2024%2E+Do+llms+have+consistent+values%3F+arXiv+preprint+arXiv%3A2407%2E12878%2E+%E2%80%94+%5BarXiv%3A2407%2E12878%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2407%2E12878%29)
23. Ariba Khan, Stephen Casper, and Dylan Hadfield-Menell. 2025. Randomness, not representation: The unreliability of evaluating cultural alignment in llms. [find](https://scholar.google.com/scholar?q=Ariba+Khan%2C+Stephen+Casper%2C+and+Dylan+Hadfield%2DMenell%2E+2025%2E+Randomness%2C+not+representation%3A+The+unreliability+of+evaluating+cultural+alignment+in+llms%2E)
24. Stuart Russell. 2022. Human-compatible artificial intelligence. Human-like machine intelligence, 1:3–22. [find](https://scholar.google.com/scholar?q=Stuart+Russell%2E+2022%2E+Human%2Dcompatible+artificial+intelligence%2E+Human%2Dlike+machine+intelligence%2C+1%3A3%E2%80%9322%2E)
25. Aadesh Salecha, Molly E Ireland, Shashanka Subrahmanya, João Sedoc, Lyle H Ungar, and Johannes C Eichstaedt. 2024. Large language models show human-like social desirability biases in survey responses. arXiv preprint arXiv:2405.06058. — [arXiv:2405.06058](https://arxiv.org/abs/2405.06058) [find](https://scholar.google.com/scholar?q=Aadesh+Salecha%2C+Molly+E+Ireland%2C+Shashanka+Subrahmanya%2C+Jo%C3%A3o+Sedoc%2C+Lyle+H+Ungar%2C+and+Johannes+C+Eichstaedt%2E+2024%2E+Large+language+models+show+human%2Dlike+social+desirability+biases+in+survey+responses%2E)
26. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022. [React: Synergizing reasoning and acting in language models](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/). In The eleventh international conference on learning representations.
27. Jifan Zhang, Henry Sleight, Andi Peng, John Schulman, and Esin Durmus. 2025. Stress-testing model specs reveals character differences among language models. arXiv preprint arXiv:2510.07686. [find](https://scholar.google.com/scholar?q=Jifan+Zhang%2C+Henry+Sleight%2C+Andi+Peng%2C+John+Schulman%2C+and+Esin+Durmus%2E+2025%2E+Stress%2Dtesting+model+specs+reveals+character+differences+among+language+models%2E+arXiv+preprint+arXiv%3A2510%2E07686%2E)

*Updated 1 November 2026*