--- title: "Referential ambiguity and clarification requests: comparing" description: "The authors build on the Minecraft Dialogue Corpus, in which an Architect directs a Builder in a voxel… 4 references, cited by 3 papers here." authors: ["Chris Madge", "Matthew Purver", "Massimo Poesio"] year: 2025 venue: "Proceedings of the Eighth Workshop on Computational Models of Reference, Anaphora and Coreference" source: "https://aclanthology.org/2025.crac-1.1/" doi: "10.18653/v1/2025.crac-1.1" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/madge-2025-referential-ambiguity-clarification-requests-comparing/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/madge-2025-referential-ambiguity-clarification-requests-comparing/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/12/) / Referential ambiguity and clarification requests: comparing human and LLM behaviour # Referential ambiguity and clarification requests: comparing human and LLM behaviour **Authors:** [Chris Madge](https://md.research.elamur.ai/free/en/authors/chris-madge/), [Matthew Purver](https://md.research.elamur.ai/free/en/authors/matthew-purver/) and [Massimo Poesio](https://md.research.elamur.ai/free/en/authors/massimo-poesio/) · **Year:** 2025 · **Venue:** [Proceedings of the Eighth Workshop on Computational Models of Reference, Anaphora and Coreference](https://md.research.elamur.ai/free/en/venues/crac/) ([CRAC 2025](https://md.research.elamur.ai/free/en/venues/crac/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2025.crac-1.1/) · [PDF](https://aclanthology.org/2025.crac-1.1.pdf) · [DOI 10.18653/v1/2025.crac-1.1](https://doi.org/10.18653/v1/2025.crac-1.1) · [OpenAlex](https://openalex.org/W4416037848) **Citations:** 1 citations (OpenAlex, 2026-10-14) · [3](https://md.research.elamur.ai/free/en/papers/madge-2025-referential-ambiguity-clarification-requests-comparing/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary This paper asks whether ambiguous references actually lead people to ask clarification questions, and whether LLMs behave the same way. The authors build on the Minecraft Dialogue Corpus, in which an Architect directs a Builder in a voxel world. They combine two existing annotation efforts into one aligned MMAX resource: MDC-R, which marks reference and referential ambiguity, and MSDC, which marks SDRT discourse relations, including clarification and confirmation requests. They then ask several LLMs to produce the Builder's turn at points of possible uncertainty and compare these turns with what the original human Builders said. The link between referential ambiguity and human clarification turns out to be weak. People rarely question an ambiguous reference. They usually act on a guess and later check task-level details. LLMs show the reverse pattern, and they agree poorly with humans. Models built for reasoning, and older models given Chain of Thought prompts, ask more questions and more relevant ones. This suggests that the ability to clarify depends on simulated reasoning, a link the authors connect to metacognition and Theory of Mind. ## Key points - **Source corpus:** MDC contains 509 dialogues set in an 11 × 9 × 11 Malmo-based world. - MDC-R covers 100 of these dialogues, with reference annotated by experts under the ARRAU guidelines. - MSDC provides exhaustive SDRT annotation. - **Merging the corpora:** a token-level script aligned the two corpora. It adds a new MMAX layer called SDRT. Each markable in this layer is an Elementary Discourse Unit, and a `to` attribute holds the ID of the related markable. - **Ambiguity types studied:** - discourse deixis, which points back into the conversation; - spatial deixis, which points into the voxel world. A clarification question appears in roughly 77% of dialogues, while a confirmation question appears in roughly 75%. - **Models compared:** - llama2:13b in three versions: plain, zero-shot CoT (COTZERO) and one-shot CoT (COTONE); - deepseek-r1:8b; - llama3.2:3b; - llama3.3:70b. - **Relevance result:** relevance was scored only on spatial-deixis ambiguity. - Human Builders asked no relevant questions there. - Plain llama2 also asked none. - CoT variants and newer models asked a few, with llama3.3:70b asking the most (5). ## Interesting ideas - Two independently made annotation layers on the same text (reference and discourse relations) are merged into one format through token-level alignment, so phenomena can be studied in relation to each other. - The paper separates uncertainty about what a phrase refers to from uncertainty about what the task requires, and finds that humans react to the second far more than the first. - Relevance is judged only on the most literal category, spatial deixis, because there it can be checked objectively whether a question targets the annotated ambiguity. - People often act first and then ask for confirmation, rather than asking before acting, so the reference behaviour is a strategy and not simply "ask when unsure". - Zero-shot or one-shot CoT prompting let an older model approach the questioning behaviour of reasoning-tuned models. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (frames): coreference chains and speech-act or discourse links could share one token-anchored store, which would let prag check whether an unresolved referent coincides with a clarification move. - **I2** (world): the world builder could record competing referent candidates as an explicit ambiguity state instead of silently choosing one, and say which kind of uncertainty it is. - **I3** (learn): when evaluating a fuzzy behaviour such as asking, abstaining or flagging, start with the subset where correctness is mechanically checkable and use it as a trustworthy anchor metric. - **I4** (answer): readers could commit to the best reading but attach a confirm/verify note naming the assumed referent, keeping the answer explainable. - **I5** (learn): when an LLM teacher labels referents, it could be prompted to reason step by step and to state when it cannot decide, giving silver data with honest "ambiguous" labels instead of forced guesses. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of London](https://md.research.elamur.ai/free/en/universities/university-of-london/) — University, London, United Kingdom · 12 works · 924 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works 🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works ## Cite Cite as: Chris Madge, Matthew Purver and Massimo Poesio. 2025. Referential ambiguity and clarification requests: comparing human and LLM behaviour. Proceedings of the Eighth Workshop on Computational Models of Reference, Anaphora and Coreference. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{madge2025referential, title = {{Referential ambiguity and clarification requests: comparing human and LLM behaviour}}, author = {Chris Madge, Matthew Purver and Massimo Poesio}, booktitle = {Proceedings of the Eighth Workshop on Computational Models of Reference, Anaphora and Coreference}, year = {2025}, doi = {10.18653/v1/2025.crac-1.1}, url = {https://doi.org/10.18653/v1/2025.crac-1.1} } ``` - [← previous Read Your Own Mind: Reasoning Helps Surface Self-Confidence Signals in LLMs](https://md.research.elamur.ai/free/en/papers/podolak-2025-read-own-mind-reasoning-helps/) - [next → SCI-IDEA: Context-Aware Scientific Ideation Using Token and Sentence Embeddings](https://md.research.elamur.ai/free/en/papers/keya-2025-sci-idea-context-aware-scientific/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/12/) ## References 4 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Mohammad Aliannejadi, Hamed Zamani, Fabio Crestani, and W Bruce Croft. 2019. Asking clarifying questions in open-domain information-seeking conversations. In Proceedings of the 42nd international acm sigir conference on research and development in information retrieval, pages 475–484. [find](https://scholar.google.com/scholar?q=Mohammad+Aliannejadi%2C+Hamed+Zamani%2C+Fabio+Crestani%2C+and+W+Bruce+Croft%2E+2019%2E+Asking+clarifying+questions+in+open%2Ddomain+information%2Dseeking+conversations%2E+In+Proceedings+of+the+42nd+international+acm+) 2. Chinmaya Andukuri, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah D Goodman. 2024. Star-gate: Teaching language models to ask clarifying questions. arXiv preprint arXiv:2403.19154. [↗](https://arxiv.org/abs/2403.19154) 3. Nicholas Asher and Alex Lascarides. 2003. Logics of conversation. Cambridge University Press. [find](https://scholar.google.com/scholar?q=Nicholas+Asher+and+Alex+Lascarides%2E+2003%2E+Logics+of+conversation%2E+Cambridge+University+Press%2E) 4. Cristian-Paul Bara, Sky CH-Wang, and Joyce Chai. 2021. Mindcraft: Theory of mind modeling for situated dialogue in collaborative tasks. arXiv preprint arXiv:2109.06275. [↗](https://arxiv.org/abs/2109.06275) *Updated 29 October 2026*