--- title: "Scope Ambiguities in Large Language Models — Kamath 2024" description: "The paper asks how autoregressive LLMs handle scope ambiguity. 3 references, cited by 13 papers here." authors: ["Gaurav Kamath", "Sebastian Schuster", "Sowmya Vajjala", "Siva Reddy"] year: 2024 venue: "Transactions of the Association for Computational Linguistics" source: "https://aclanthology.org/2024.tacl-1.41/" doi: "10.1162/tacl_a_00670" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/kamath-2024-scope-ambiguities-large-language-models/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kamath-2024-scope-ambiguities-large-language-models/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) / Scope Ambiguities in Large Language Models # Scope Ambiguities in Large Language Models **Authors:** Gaurav Kamath, [Sebastian Schuster](https://md.research.elamur.ai/free/en/authors/sebastian-schuster/), [Sowmya Vajjala](https://md.research.elamur.ai/free/en/authors/sowmya-vajjala/) and [Siva Reddy](https://md.research.elamur.ai/free/en/authors/siva-reddy/) · **Year:** 2024 · **Venue:** [Transactions of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/tacl/) ([Transactions of the Association for Computational Linguistics (TACL) 2024](https://md.research.elamur.ai/free/en/venues/tacl/2024/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2024.tacl-1.41/) · [PDF](https://aclanthology.org/2024.tacl-1.41.pdf) · [DOI 10.1162/tacl\_a\_00670](https://doi.org/10.1162/tacl_a_00670) · [OpenAlex](https://openalex.org/W4394647506) **Citations:** 22 citations (OpenAlex, 2026-10-14) · [13](https://md.research.elamur.ai/free/en/papers/kamath-2024-scope-ambiguities-large-language-models/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary The paper asks how autoregressive LLMs handle scope ambiguity. This occurs when a sentence contains several semantic operators, such as quantifiers, negation or adverbs like "usually", and their relative scope can be ordered in more than one way. Resolving such sentences depends on both semantic structure and world knowledge. The authors built new English datasets of nearly 1,000 distinct scope-ambiguous sentences, each annotated with human judgments, and tested GPT-2, GPT-3/3.5, Llama 2 and GPT-4. Experiment 1 checks whether models favour the same reading as humans, using a question-answering setup. Experiment 2 uses probabilities of acceptable versus unacceptable follow-up sentences to test whether models register that two readings exist. The stronger models matched human preferences with high accuracy, above 90% in some cases, and showed sensitivity to the ambiguity itself. These results run counter to earlier zero-shot studies. The authors argue that probability-based probing and metalinguistic prompting can lead to different conclusions about what LLMs know. ## Key points - **Data:** Roughly 1,000 unique scope-ambiguous sentences with human annotations, which the authors describe as the largest resource of its kind. The sentences cover quantifier–quantifier, quantifier–negation and quantifier–adverb interactions. Human judgments were collected via Prolific. Data and code are available at github.com/McGill-NLP/scope-ambiguity. - **Experiment 2B (110 items):** text-davinci-003 and Llama2-13b produced positive α-scores, indicating a preference for acceptable follow-ups, on more than 90% of items. Llama 2 chat variants correlated better with human proxy scores than the base models. - **Reformatted test:** The authors recast part of their own data into the True/False format of Liu et al. (2023). Most models then performed near chance, and even GPT-4 and GPT-3.5-turbo reached only 64%. - **Earlier datasets:** A manual review of the 579 ambiguous sentences in Liu et al.'s dataset found only about 20 genuine scope ambiguities. - **Limitations:** The study covers English only and does not examine how context shifts the preferred reading. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [National Research Council Canada](https://md.research.elamur.ai/free/en/government/national-research-council-canada/) — Government body, Ottawa, Canada · 59 works · 21,677 citations - [University College London](https://md.research.elamur.ai/free/en/universities/university-college-london/) — University, London, United Kingdom · 88 works · 19,432 citations - [McGill University](https://md.research.elamur.ai/free/en/universities/mcgill-university/) — University, Montreal, Canada · 87 works · 11,027 citations - [Mila - Quebec Artificial Intelligence Institute](https://md.research.elamur.ai/free/en/labs/mila-quebec-artificial-intelligence-institute/) — Research institute, Montreal, Canada · 54 works · 13,565 citations - [Canadian Institute for Advanced Research](https://md.research.elamur.ai/free/en/labs/canadian-institute-for-advanced-research/) — Research institute, Toronto, Canada · 28 works · 16,725 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works 🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works ## Cite Cite as: Gaurav Kamath, Sebastian Schuster, Sowmya Vajjala and Siva Reddy. 2024. Scope Ambiguities in Large Language Models. Transactions of the Association for Computational Linguistics. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @article{kamath2024scope, title = {{Scope Ambiguities in Large Language Models}}, author = {Gaurav Kamath, Sebastian Schuster, Sowmya Vajjala and Siva Reddy}, journal = {Transactions of the Association for Computational Linguistics}, year = {2024}, doi = {10.1162/tacl\_a\_00670}, url = {https://doi.org/10.1162/tacl_a_00670} } ``` - [← previous SarcasmBench: Towards Evaluating Large Language Models on Sarcasm Understanding](https://md.research.elamur.ai/free/en/papers/zhang-2024-sarcasmbench-towards-evaluating-large-language/) - [next → Self-Refine Instruction-Tuning for Aligning Reasoning in Language Models](https://md.research.elamur.ai/free/en/papers/ranaldi-2024-self-refine-instruction-tuning-aligning/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) ## References 3 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. a. Context: Ada finds it hard to study alone, so she generally invites others for joint study sessions. [find](https://scholar.google.com/scholar?q=a%2E+Context%3A+Ada+finds+it+hard+to+study+alone%2C+so+she+generally+invites+others+for+joint+study+sessions%2E) 2. Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona Diab, and Marjan Ghazvininejad. 2022. A review on language models as knowledge bases. arXiv preprint arXiv:2204.06031. b. Context: Ada, Rohan, and Jo are good friends in the same program, and prepare for exams together. [find](https://scholar.google.com/scholar?q=Badr+AlKhamissi%2C+Millicent+Li%2C+Asli+Celikyilmaz%2C+Mona+Diab%2C+and+Marjan+Ghazvininejad%2E+2022%2E+A+review+on+language+models+as+knowledge+bases%2E+arXiv+preprint+arXiv%3A2204%2E06031%2E+b%2E+Context%3A+Ada%2C+Rohan%2C+and) 3. Scott AnderBois, Adrian Brasoveanu, and Robert Henderson. 2012. The pragmatics of quantifier scope: A corpus study. In Proceedings of Sinn und Bedeutung, volume 16, pages 15–28. [find](https://scholar.google.com/scholar?q=Scott+AnderBois%2C+Adrian+Brasoveanu%2C+and+Robert+Henderson%2E+2012%2E+The+pragmatics+of+quantifier+scope%3A+A+corpus+study%2E+In+Proceedings+of+Sinn+und+Bedeutung%2C+volume+16%2C+pages+15%E2%80%9328%2E) *Updated 30 October 2026*