---
title: "When Models Decide and When They Bind: A Two-Stage"
description: "Multiple-choice question answering (MCQA) is a popular evaluation format, but it hides a second task. 33 references."
authors: ["Hugh Mee Wong", "Rick Nouwen", "Albert Gatt"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.1144/"
doi: "10.18653/v1/2026.findings-acl.1144"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wong-2026-when-models-decide-when-they/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wong-2026-when-models-decide-when-they/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/56/) / When Models Decide and When They Bind: A Two-Stage Computation for Multiple-Choice Question Answering

# When Models Decide and When They Bind: A Two-Stage Computation for Multiple-Choice Question Answering

**Authors:** Hugh Mee Wong, [Rick Nouwen](https://md.research.elamur.ai/free/en/authors/rick-nouwen/) and [Albert Gatt](https://md.research.elamur.ai/free/en/authors/albert-gatt/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1144/) · [PDF](https://aclanthology.org/2026.findings-acl.1144.pdf) · [DOI 10.18653/v1/2026.findings-acl.1144](https://doi.org/10.18653/v1/2026.findings-acl.1144) · [OpenAlex](https://openalex.org/W7166862129)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Multiple-choice question answering (MCQA) is a popular evaluation format, but it hides a second task. A model must work out the right answer and also emit the letter attached to it. As a result, an error can come from faulty reasoning or from faulty symbol binding. The authors study how this unfolds inside two instruction-tuned models, Llama-3.2-Instruct-3B and Qwen3-8B. They use two datasets: a modified Colors copying task, with object names masked so that factual recall plays no role, and the four-option subset of ARC-Easy. Their methods are PCA on residual states, linear probes, and probe-aligned activation patching. The newline that ends each option often carries a linearly readable signal of whether that option is correct. Probes for the winning answer show a two-step pattern. The winning content position is decodable just after the final option is read, while the letter to output appears only near the answer position. Tests that shuffle symbols and contents support this split between selecting an answer and binding it to a symbol. The implication is that MCQA scores mix knowledge with format effects, which matters for benchmark design and for calibration.

## Key points
- **Data:** the Colors test set (100 items), rewritten to use "this object" instead of named objects, plus 2365 four-option ARC-Easy items with numeric labels converted to letters.
- **Models:** Llama-3.2-Instruct-3B (28 layers) and Qwen3-8B (36 layers). Code is released at github.com/hughmee/mcqa-binding.
- **PCA findings:** activations group by symbol position, and in middle layers they also separate by whether the preceding option was correct.
  - In Qwen3-8B, positional information is entangled with the alphabetical identity of the labels.
  - In Llama, list order dominates.
  - In both models, the first option behaves atypically.
- **Winner-identity results:** 4-way probes tested under symbol and content permutations show that the content position of the winner is encoded early, while the output symbol is encoded late. Causal patching backs up this dissociation.
- **Limitations noted by the authors:**
  - only small models from two families were tested;
  - a single fixed prompt template was used;
  - known MCQA biases were not examined;
  - patching for per-option correctness used only a one-dimensional probe direction.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works

🧭 Topics (1): [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 487 works

## Cite

Cite as: Hugh Mee Wong, Rick Nouwen and Albert Gatt. 2026. When Models Decide and When They Bind: A Two-Stage Computation for Multiple-Choice Question Answering. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wong2026when,
  title = {{When Models Decide and When They Bind: A Two-Stage Computation for Multiple-Choice Question Answering}},
  author = {Hugh Mee Wong, Rick Nouwen and Albert Gatt},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1144},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1144}
}
```

- [← previous When LLMs Read Tables Carelessly: Measuring and Reducing Data Referencing Errors](https://md.research.elamur.ai/free/en/papers/yang-2026-when-llms-read-tables-carelessly/)
- [next → When Numbers Tell Half the Story: Human-Metric Alignment in Topic Model Evaluation](https://md.research.elamur.ai/free/en/papers/prouteau-2026-when-numbers-tell-half-story/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/56/)

## References

33 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Guillaume Alain and Yoshua Bengio. 2017. Understanding intermediate layers using linear classifier probes. [find](https://scholar.google.com/scholar?q=Guillaume+Alain+and+Yoshua+Bengio%2E+2017%2E+Understanding+intermediate+layers+using+linear+classifier+probes%2E)
2. Thomas Burwick. 2014. The binding problem. WIREs Cognitive Science, 5(3):305–315. [find](https://scholar.google.com/scholar?q=Thomas+Burwick%2E+2014%2E+The+binding+problem%2E+WIREs+Cognitive+Science%2C+5%283%29%3A305%E2%80%93315%2E)
3. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv preprint arXiv:1803.05457.
4. Qin Dai, Benjamin Heinzerling, and Kentaro Inui. 2024. Representational analysis of binding in language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17468–17493, Miami, Florida, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Qin+Dai%2C+Benjamin+Heinzerling%2C+and+Kentaro+Inui%2E+2024%2E+Representational+analysis+of+binding+in+language+models%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Natural+Language+Processin)
5. Ricardo Dominguez-Olmedo, Moritz Hardt, and Celestine Mendler-Dünner. 2024. Questioning the survey responses of large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Ricardo+Dominguez%2DOlmedo%2C+Moritz+Hardt%2C+and+Celestine+Mendler%2DD%C3%BCnner%2E+2024%2E+Questioning+the+survey+responses+of+large+language+models%2E+In+The+Thirty%2Deighth+Annual+Conference+on+Neural+Information+Proc)
6. Jiahai Feng and Jacob Steinhardt. 2024. How do language models bind entities in context? In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Jiahai+Feng+and+Jacob+Steinhardt%2E+2024%2E+How+do+language+models+bind+entities+in+context%3F+In+The+Twelfth+International+Conference+on+Learning+Representations%2E)
7. Atticus Geiger, Kyle Richardson, and Christopher Potts. 2020. [Neural natural language inference models partially embed theories of lexical entailment and negation](https://md.research.elamur.ai/free/en/papers/geiger-2020-neural-natural-language-inference-models/). In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 163–173, Online. Associati…
8. Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah Goodman. 2024. [Finding alignments between interpretable causal variables and distributed neural representations](https://md.research.elamur.ai/free/en/papers/geiger-2023-finding-alignments-between-interpretable-causal/). In Proceedings of the Third Conference on Causal Learning and Reasoning, volume 236 of Proceedings of Machine Learn…
9. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
10. Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. 2018. Visualisation and ‘diagnostic classifiers’ reveal how recurrent and recursive neural networks process hierarchical structure. Journal of Artificial Intelligence Research, 61:907–926. [find](https://scholar.google.com/scholar?q=Dieuwke+Hupkes%2C+Sara+Veldhoen%2C+and+Willem+Zuidema%2E+2018%2E+Visualisation+and+%E2%80%98diagnostic+classifiers%E2%80%99+reveal+how+recurrent+and+recursive+neural+networks+process+hierarchical+structure%2E+Journal+of+Artifi)
11. Najoung Kim and Sebastian Schuster. 2023. [Entity tracking in language models](https://md.research.elamur.ai/free/en/papers/kim-2023-entity-tracking-language-models/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3835–3855, Toronto, Canada. Association for Computational Linguistics.
12. Vedang Lad, Wes Gurnee, and Max Tegmark. 2024. The remarkable robustness of LLMs: Stages of inference? In ICML 2024 Workshop on Mechanistic Interpretability. [find](https://scholar.google.com/scholar?q=Vedang+Lad%2C+Wes+Gurnee%2C+and+Max+Tegmark%2E+2024%2E+The+remarkable+robustness+of+LLMs%3A+Stages+of+inference%3F+In+ICML+2024+Workshop+on+Mechanistic+Interpretability%2E)
13. Ruizhe Li and Yanjun Gao. 2025. Anchored answers: Unravelling positional bias in GPT-2’s multiple-choice questions. In Findings of the Association for Computational Linguistics: ACL 2025, pages 2439–2465, Vienna, Austria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ruizhe+Li+and+Yanjun+Gao%2E+2025%2E+Anchored+answers%3A+Unravelling+positional+bias+in+GPT%2D2%E2%80%99s+multiple%2Dchoice+questions%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL+2025%2C+pages+2439%E2%80%932)
14. Tom Lieberum, Matthew Rahtz, János Kramár, Neel Nanda, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. 2023. [Does circuit analysis interpretability scale? Evidence from multiple choice capabilities in Chinchilla](https://md.research.elamur.ai/free/en/papers/lieberum-2023-circuit-analysis-interpretability-scale-evidence/). arXiv preprint arXiv:2307.09458.
15. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA. Curran Associates Inc. [find](https://scholar.google.com/scholar?q=Kevin+Meng%2C+David+Bau%2C+Alex+Andonian%2C+and+Yonatan+Belinkov%2E+2022%2E+Locating+and+editing+factual+associations+in+gpt%2E+In+Proceedings+of+the+36th+International+Conference+on+Neural+Information+Processing)
16. Meta AI. 2024. Llama 3.2: Revolutionizing edge AI and vision with open, customizable models. Meta AI Blog. [find](https://scholar.google.com/scholar?q=Meta+AI%2E+2024%2E+Llama+3%2E2%3A+Revolutionizing+edge+AI+and+vision+with+open%2C+customizable+models%2E+Meta+AI+Blog%2E)
17. Joseph Miller, Bilal Chughtai, and William Saunders. 2024. Transformer circuit evaluation metrics are not robust. In First Conference on Language Modeling. [find](https://scholar.google.com/scholar?q=Joseph+Miller%2C+Bilal+Chughtai%2C+and+William+Saunders%2E+2024%2E+Transformer+circuit+evaluation+metrics+are+not+robust%2E+In+First+Conference+on+Language+Modeling%2E)
18. Aaron Mueller, Atticus Geiger, Sarah Wiegreffe, Dana Arad, Iván Arcuschin, Adam Belfki, Yik Siu Chan, Jaden Fried Fiotto-Kaufman, Tal Haklay, Michael Hanna, Jing Huang, Rohan Gupta, Yaniv Nikankin, Hadas Orgad, Nikhil Prakash, Anja Reusch, Aruna Sankaranarayanan, Shun Shao, Alessandro Stolfo, and 4… [find](https://scholar.google.com/scholar?q=Aaron+Mueller%2C+Atticus+Geiger%2C+Sarah+Wiegreffe%2C+Dana+Arad%2C+Iv%C3%A1n+Arcuschin%2C+Adam+Belfki%2C+Yik+Siu+Chan%2C+Jaden+Fried+Fiotto%2DKaufman%2C+Tal+Haklay%2C+Michael+Hanna%2C+Jing+Huang%2C+Rohan+Gupta%2C+Yaniv+Nikankin%2C+Ha)
19. Tobias Norlund, Lovisa Hagström, and Richard Johansson. 2021. Transferring knowledge from vision to language: How to achieve it and how to measure it? In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 149–162, Punta Cana, Dominican Republi… [find](https://scholar.google.com/scholar?q=Tobias+Norlund%2C+Lovisa+Hagstr%C3%B6m%2C+and+Richard+Johansson%2E+2021%2E+Transferring+knowledge+from+vision+to+language%3A+How+to+achieve+it+and+how+to+measure+it%3F+In+Proceedings+of+the+Fourth+BlackboxNLP+Workshop)
20. Pouya Pezeshkpour and Estevam Hruschka. 2024. [Large language models sensitivity to the order of options in multiple-choice questions](https://md.research.elamur.ai/free/en/papers/pezeshkpour-2023-large-language-models-sensitivity-order/). In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2006–2017, Mexico City, Mexico. Association for Computational Linguistics.
21. Curt Tigges, Oskar J. Hollinsworth, Atticus Geiger, and Neel Nanda. 2024. Language models linearly represent sentiment. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 58–87, Miami, Florida, US. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Curt+Tigges%2C+Oskar+J%2E+Hollinsworth%2C+Atticus+Geiger%2C+and+Neel+Nanda%2E+2024%2E+Language+models+linearly+represent+sentiment%2E+In+Proceedings+of+the+7th+BlackboxNLP+Workshop%3A+Analyzing+and+Interpreting+Neura)
22. Polina Tsvilodub, Hening Wang, Sharon Grosch, and Michael Franke. 2024. Predictions from language models for multiple-choice tasks are not robust under variation of scoring methods. arXiv preprint arXiv:2403.00998. — [arXiv:2403.00998](https://arxiv.org/abs/2403.00998) [find](https://scholar.google.com/scholar?q=Polina+Tsvilodub%2C+Hening+Wang%2C+Sharon+Grosch%2C+and+Michael+Franke%2E+2024%2E+Predictions+from+language+models+for+multiple%2Dchoice+tasks+are+not+robust+under+variation+of+scoring+methods%2E+arXiv+preprint+arX)
23. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. In Advances in Neural Information Processing Systems, volume 33, pages 12388–12401. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Jesse+Vig%2C+Sebastian+Gehrmann%2C+Yonatan+Belinkov%2C+Sharon+Qian%2C+Daniel+Nevo%2C+Yaron+Singer%2C+and+Stuart+Shieber%2E+2020%2E+Investigating+gender+bias+in+language+models+using+causal+mediation+analysis%2E+In+Adva)
24. Christoph Von Der Malsburg. 1999. The What and Why of Binding. Neuron, 24(1):95–104. [find](https://scholar.google.com/scholar?q=Christoph+Von+Der+Malsburg%2E+1999%2E+The+What+and+Why+of+Binding%2E+Neuron%2C+24%281%29%3A95%E2%80%93104%2E)
25. Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2023. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Kevin+Ro+Wang%2C+Alexandre+Variengien%2C+Arthur+Conmy%2C+Buck+Shlegeris%2C+and+Jacob+Steinhardt%2E+2023%2E+Interpretability+in+the+wild%3A+a+circuit+for+indirect+object+identification+in+GPT%2D2+small%2E+In+The+Elevent)
26. Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul Röttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. 2024. [“my answer is C”: First-token probabilities do not match text answers in instruction-tuned language models](https://md.research.elamur.ai/free/en/papers/wang-2024-my-answer-is-c-first/). In Findings of the Association for Computational Linguistics: ACL 2024…
27. Sarah Wiegreffe, Oyvind Tafjord, Yonatan Belinkov, Hannaneh Hajishirzi, and Ashish Sabharwal. 2025. Answer, assemble, ace: Understanding how LMs answer multiple choice questions. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Sarah+Wiegreffe%2C+Oyvind+Tafjord%2C+Yonatan+Belinkov%2C+Hannaneh+Hajishirzi%2C+and+Ashish+Sabharwal%2E+2025%2E+Answer%2C+assemble%2C+ace%3A+Understanding+how+LMs+answer+multiple+choice+questions%2E+In+The+Thirteenth+Int)
28. Mengge Xue, Zhenyu Hu, Liqun Liu, Kuo Liao, Shuang Li, Honglin Han, Meng Zhao, and Chengguo Yin. 2024. Strengthened symbol binding makes large language models reliable multiple-choice selectors. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon… [find](https://scholar.google.com/scholar?q=Mengge+Xue%2C+Zhenyu+Hu%2C+Liqun+Liu%2C+Kuo+Liao%2C+Shuang+Li%2C+Honglin+Han%2C+Meng+Zhao%2C+and+Chengguo+Yin%2E+2024%2E+Strengthened+symbol+binding+makes+large+language+models+reliable+multiple%2Dchoice+selectors%2E+In+Pr)
29. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao 22814 11 Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025a. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). Preprint, arXiv:2505.093…
30. Zhen Yang, Ping Jian, and Chengzhi Li. 2025b. Option symbol matters: Investigating and mitigating multiple-choice option symbol bias of large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language… [find](https://scholar.google.com/scholar?q=Zhen+Yang%2C+Ping+Jian%2C+and+Chengzhi+Li%2E+2025b%2E+Option+symbol+matters%3A+Investigating+and+mitigating+multiple%2Dchoice+option+symbol+bias+of+large+language+models%2E+In+Proceedings+of+the+2025+Conference+of+)
31. Fred Zhang and Neel Nanda. 2024. [Towards best practices of activation patching in language models: Metrics and methods](https://md.research.elamur.ai/free/en/papers/zhang-2023-towards-best-practices-activation-patching/). In The Twelfth International Conference on Learning Representations.
32. Guangxiang Zhao, Saier Hu, Xiaoqi Jian, Wu Jinzhu, Yuhan Wu, Lin Sun, and Xiangzheng Zhang. 2025. Large language models badly generalize across option length, problem types, and irrelevant noun replacements. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p… [find](https://scholar.google.com/scholar?q=Guangxiang+Zhao%2C+Saier+Hu%2C+Xiaoqi+Jian%2C+Wu+Jinzhu%2C+Yuhan+Wu%2C+Lin+Sun%2C+and+Xiangzheng+Zhang%2E+2025%2E+Large+language+models+badly+generalize+across+option+length%2C+problem+types%2C+and+irrelevant+noun+replac)
33. Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024. Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations. 22815 12 *Figure 8: Performance on the color copying task of the multinomial probes trained on… [find](https://scholar.google.com/scholar?q=Chujie+Zheng%2C+Hao+Zhou%2C+Fandong+Meng%2C+Jie+Zhou%2C+and+Minlie+Huang%2E+2024%2E+Large+language+models+are+not+robust+multiple+choice+selectors%2E+In+The+Twelfth+International+Conference+on+Learning+Representati)

*Updated 2 November 2026*