--- title: "A Comparative Evaluation of Semantic Ambiguity Detection in" description: "This pilot study asks whether two OpenAI models, GPT-4.1 and its smaller sibling GPT-4.1 mini, notice when a text has more… 3 references." authors: ["Lili Tamas"] year: 2026 venue: "Proceedings of the Workshop Neology and Large Language Models" source: "https://aclanthology.org/2026.neollm-1.7/" doi: "10.63317/49osn8x7xs7d" licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)" licence_url: "https://creativecommons.org/licenses/by-nc/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/tamas-2026-comparative-evaluation-semantic-ambiguity-detection/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tamas-2026-comparative-evaluation-semantic-ambiguity-detection/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/20/) / A Comparative Evaluation of Semantic Ambiguity Detection in Two LLMs # A Comparative Evaluation of Semantic Ambiguity Detection in Two LLMs **Authors:** Lili Tamas · **Year:** 2026 · **Venue:** [Proceedings of the Workshop Neology and Large Language Models](https://md.research.elamur.ai/free/en/venues/neollm/) ([Workshop Neology and Large Language Models 2026](https://md.research.elamur.ai/free/en/venues/neollm/2026/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.neollm-1.7/) · [PDF](https://aclanthology.org/2026.neollm-1.7.pdf) · [DOI 10.63317/49osn8x7xs7d](https://doi.org/10.63317/49osn8x7xs7d) · [OpenAlex](https://openalex.org/W7166525771) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary This pilot study asks whether two OpenAI models, GPT-4.1 and its smaller sibling GPT-4.1 mini, notice when a text has more than one meaning, a skill tied to metalinguistic awareness. Drawing on test materials originally designed for children by Zipke et al. (2009) and by Shultz and Pilon (1973), the author built 29 contexts: riddles, single sentences, and sentence pairs. These cover lexical ambiguity and two kinds of structural ambiguity (surface-structure and deep-structure). Each context was paired with instructions at four levels, ranging from no hint of ambiguity to naming it outright, which gave 116 tasks per model. Responses were collected through the API with default settings and scored as pass/fail. Left unprompted, both models usually settled on a single reading. Their performance rose sharply once instructions suggested that multiple meanings might exist. The larger model did better on lexical items, while the mini model led on structural ones, so size alone did not predict success. The work is meant as groundwork for ambiguity-annotated datasets and for studying how LLMs handle newly coined, often ambiguous words. ## Key points - **Task set:** 8 riddles and 7 single sentences from Zipke et al. (2009), plus 14 sentence pairs from Shultz and Pilon (1973). These 29 contexts were run under 4 instruction levels, giving 116 tasks per model with binary scoring. - **Lexical ambiguity at Level 1 (no hint):** GPT-4.1 solved 61% of tasks and GPT-4.1 mini only 11%. Neither model reached a perfect score even when ambiguity was stated directly. - **Deep-structure ambiguity:** GPT-4.1 mini scored 50% at Level 1 and 100% on Levels 2–4. GPT-4.1 started at 17%, peaked at Level 3, and dropped at Level 4. - **Overall trend:** GPT-4.1 mini improved with every level and overtook GPT-4.1 at Level 4. GPT-4.1 did not improve between Levels 3 and 4. - **Persistent failures and limitations:** Both models failed every task built on two contexts. In SP4, "tank" was never read as a vehicle; in SP11, the vision-related sense of "see" was missed. The study also has clear limitations: lexical items are over-represented, there is no human baseline, and each task was run only once. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Károli Gáspár University of the Reformed Church in Hungary](https://md.research.elamur.ai/free/en/universities/karoli-gaspar-university-of-the-reformed-church-in-hungary/) — University, Budapest, Hungary · 3 works 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works 🧭 Topics (1): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works ## Cite Cite as: Lili Tamas. 2026. A Comparative Evaluation of Semantic Ambiguity Detection in Two LLMs. Proceedings of the Workshop Neology and Large Language Models. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{tamas2026comparative, title = {{A Comparative Evaluation of Semantic Ambiguity Detection in Two LLMs}}, author = {Lili Tamas}, booktitle = {Proceedings of the Workshop Neology and Large Language Models}, year = {2026}, doi = {10.63317/49osn8x7xs7d}, url = {https://doi.org/10.63317/49osn8x7xs7d} } ``` - [← previous A Comparative Evaluation of Open-Source Models for Russian-Kazakh Translation](https://md.research.elamur.ai/free/en/papers/shanshin-2026-comparative-evaluation-open-source-models/) - [next → A Comparative Study in Corpus Linguistics Applied to Automatic Terminology Extraction](https://md.research.elamur.ai/free/en/papers/vazquez-2026-comparative-study-corpus-linguistics-applied/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/20/) ## References 3 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Zipke, M., Ehri, L. C., & Cairns, H. S. (2009). Using Semantic Ambiguity Instruction to Improve Third Graders’ Metalinguistic Awareness and Reading Comprehension: An Bowdle, B. F., & Gentner, D. (2005). The Career of Metaphor. Psychological Review, 112(1), Experimental Study. Reading Research Quarte… — [doi:10.1598/rrq.44.3.4](https://doi.org/10.1598/rrq.44.3.4) [find](https://scholar.google.com/scholar?q=Zipke%2C+M%2E%2C+Ehri%2C+L%2E+C%2E%2C+%26+Cairns%2C+H%2E+S%2E+%282009%29%2E+Using+Semantic+Ambiguity+Instruction+to+Improve+Third+Graders%E2%80%99+Metalinguistic+Awareness+and+Reading+Comprehension%3A+An+Bowdle%2C+B%2E+F%2E%2C+%26+Gentner%2C+D%2E+%282005) 2. Language Resource References OpenAI, Inc. (n.d.). Models. Platform.openai.com; OpenAI, Inc. Retrieved October 8, 2025, from [https://platform.openai.com/docs/models/](https://platform.openai.com/docs/models/) [↗](https://platform.openai.com/docs/models/) 3. Google. (n.d.). Books Ngram Viewer. Google.com. [https://books.google.com/ngrams/](https://books.google.com/ngrams/) [↗](https://books.google.com/ngrams/) *Updated 1 November 2026*