---
title: "CUCLASIC at SemEval-2026 Task 5 — paper by Riba et al. (2026)"
description: "The paper describes a system for SemEval-2026 Task 5. 14 references, cited by 1 paper here."
authors: ["Federico Ortega Riba", "Jasper Wilkerson", "Kelsey Lafreniere Adams"]
year: 2026
venue: "Proceedings of the 20th International Workshop on Semantic Evaluation (2026)"
source: "https://aclanthology.org/2026.semeval-1.121/"
doi: "10.18653/v1/2026.semeval-1.121"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/riba-2026-cuclasic-semeval-2026-task-5/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/riba-2026-cuclasic-semeval-2026-task-5/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/11/) / CUCLASIC at SemEval-2026 Task 5: LLM Prompting Strategies for Rating Ambiguous Word Senses

# CUCLASIC at SemEval-2026 Task 5: LLM Prompting Strategies for Rating Ambiguous Word Senses

**Authors:** Federico Ortega Riba, Jasper Wilkerson and Kelsey Lafreniere Adams · **Year:** 2026 · **Venue:** [Proceedings of the 20th International Workshop on Semantic Evaluation (2026)](https://md.research.elamur.ai/free/en/venues/semeval/) ([SemEval 2026](https://md.research.elamur.ai/free/en/venues/semeval/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.semeval-1.121/) · [PDF](https://aclanthology.org/2026.semeval-1.121.pdf) · [DOI 10.18653/v1/2026.semeval-1.121](https://doi.org/10.18653/v1/2026.semeval-1.121) · [OpenAlex](https://openalex.org/W7166856104)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper describes a system for SemEval-2026 Task 5. The task treats word sense disambiguation as graded plausibility rating rather than a single correct label. Systems receive short English narratives from the AmbiStory dataset, each built around a homonym with two plausible readings. They must predict how well a given sense fits, matching the averaged 1–5 Likert judgments of five human annotators. The authors tested six large language models, drawn from both the Gemini and Llama lines, using zero-shot, 3-shot and 5-shot prompting. The prompts used an expert-linguist persona and asked for strict JSON output. They compared two kinds of demonstrations: contentious items with high annotator disagreement, and items with near-unanimous ratings. The near-unanimous demonstrations worked better. Gemini 3-Flash performed best, reaching 0.75 on the official metric. Across models, the authors found a strong tendency to avoid middle ratings, especially 3, even though 3 is the most common human score. This points to overconfidence as the main obstacle when LLMs must model graded, subjective ambiguity.

## Key points
- **Data and metrics:** AmbiStory supplies 2,280 training items, each with a precontext, an ambiguous sentence and an optional ending. Systems are scored on Spearman correlation and on accuracy within one standard deviation of the human mean, and the two are averaged.
- **Models:** gemini-2.5-pro, gemini-3-pro-preview, gemini-3-flash-preview, Llama4 Maverick-17B-128E, Llama4 Scout-17B-16E and Llama 3.3-70B-Instruct. All were run via API at temperature 0.0.
- **Best result:** Gemini 3-Flash scored 0.75 overall, with Spearman 0.73 and accuracy 0.77. Its per-score accuracy was lowest for predictions of 3 (0.69).
- **Model behavior:** Llama models predicted 5 almost every time. Most Gemini models gave mostly 1s and 5s. The authors attribute the weaker result of the larger Pro model to inverse scaling and RLHF-induced overconfidence.
- **Open questions:** The authors did not run ablations of individual prompt fields. They also did not test in a controlled way why low-standard-deviation examples help. Code and prompts are available on GitHub (fede-ortega/SemEval-2026-SharedTask-5).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Colorado Boulder](https://md.research.elamur.ai/free/en/universities/university-of-colorado-boulder/) — University, Boulder, United States · 69 works · 51,069 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🔀 Word sense disambiguation](https://md.research.elamur.ai/free/en/topics/word-sense-disambiguation/) · 83 works

## Cite

Cite as: Federico Ortega Riba, Jasper Wilkerson and Kelsey Lafreniere Adams. 2026. CUCLASIC at SemEval-2026 Task 5: LLM Prompting Strategies for Rating Ambiguous Word Senses. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{riba2026cuclasic,
  title = {{CUCLASIC at SemEval-2026 Task 5: LLM Prompting Strategies for Rating Ambiguous Word Senses}},
  author = {Federico Ortega Riba, Jasper Wilkerson and Kelsey Lafreniere Adams},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.121},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.121}
}
```

- [← previous CTC-DID: CTC-Based Arabic dialect identification for streaming applications](https://md.research.elamur.ai/free/en/papers/farooq-2026-ctc-did-ctc-based-arabic/)
- [next → CUET-823 at SemEval-2026 Task 9: LoRA-Based Instruction Fine-Tuning of LLMs vs. Transformer Models for Bengali Polarization Detection](https://md.research.elamur.ai/free/en/papers/mallik-2026-cuet-823-semeval-2026-task/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/11/)

## References

14 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Pierpaolo Basile, Lucia Siciliani, Elio Musacchio, and Giovanni Semeraro. 2025. Exploring the word sense disambiguation capabilities of large language models. arXiv preprint arXiv:2503.08662. — [arXiv:2503.08662](https://arxiv.org/abs/2503.08662) [find](https://scholar.google.com/scholar?q=Pierpaolo+Basile%2C+Lucia+Siciliani%2C+Elio+Musacchio%2C+and+Giovanni+Semeraro%2E+2025%2E+Exploring+the+word+sense+disambiguation+capabilities+of+large+language+models%2E+arXiv+preprint+arXiv%3A2503%2E08662%2E+%E2%80%94+%5BarXiv)
2. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
3. Philip Edmonds and Scott Cotton. 2001. [SENSEVAL-2: Overview](https://md.research.elamur.ai/free/en/papers/edmonds-2001-senseval-2-overview/). In Proceedings of SENSEVAL-2 Second International Workshop on Evaluating Word Sense Disambiguation Systems, pages 1–5, Toulouse, France. Association for Computational Linguistics.
4. Janosch Gehring, Selina Meyer, and Michael Roth. 2026. SemEval-2026 task 5: Rating plausibility of word senses in ambiguous stories through narrative understanding. In Proceedings of the 20th International Workshop on Semantic Evaluation, San Diego, California. Association for Computational Linguist… [find](https://scholar.google.com/scholar?q=Janosch+Gehring%2C+Selina+Meyer%2C+and+Michael+Roth%2E+2026%2E+SemEval%2D2026+task+5%3A+Rating+plausibility+of+word+senses+in+ambiguous+stories+through+narrative+understanding%2E+In+Proceedings+of+the+20th+Internat)
5. Raihan Kibria, Sheikh Intiser Uddin Dipta, and Muhammad Abdullah Adnan. 2024. On functional competence of LLMs for linguistic disambiguation. In Proceedings of the 28th Conference on Computational Natural Language Learning, pages 143–160, Miami, FL, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Raihan+Kibria%2C+Sheikh+Intiser+Uddin+Dipta%2C+and+Muhammad+Abdullah+Adnan%2E+2024%2E+On+functional+competence+of+LLMs+for+linguistic+disambiguation%2E+In+Proceedings+of+the+28th+Conference+on+Computational+Nat)
6. Robert Krovetz and W Bruce Croft. 1989. Word sense disambiguation using machine-readable dictionaries. In Proceedings of the 12th annual international ACM SIGIR conference on Research and development in information retrieval, pages 127–136. [find](https://scholar.google.com/scholar?q=Robert+Krovetz+and+W+Bruce+Croft%2E+1989%2E+Word+sense+disambiguation+using+machine%2Dreadable+dictionaries%2E+In+Proceedings+of+the+12th+annual+international+ACM+SIGIR+conference+on+Research+and+development+)
7. Federico Martelli, Najla Kalach, Gabriele Tola, and Roberto Navigli. 2021. SemEval-2021 task 2: Multilingual and cross-lingual word-in-context disambiguation (MCL-WiC). In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 24–36, Online. Association for Compu… [find](https://scholar.google.com/scholar?q=Federico+Martelli%2C+Najla+Kalach%2C+Gabriele+Tola%2C+and+Roberto+Navigli%2E+2021%2E+SemEval%2D2021+task+2%3A+Multilingual+and+cross%2Dlingual+word%2Din%2Dcontext+disambiguation+%28MCL%2DWiC%29%2E+In+Proceedings+of+the+15th+Inte)
8. Rada Mihalcea, Timothy Chklovski, and Adam Kilgarriff. 2004. [The senseval-3 English lexical sample task](https://md.research.elamur.ai/free/en/papers/mihalcea-2004-senseval-3-english-lexical-sample/). In Proceedings of SENSEVAL-3, the Third International Workshop on the Evaluation of Systems for the Semantic Analysis of Text, pages 25–28, Barcelona, Spain. Association for Computational Linguis…
9. Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2025. [Large language models: A survey](https://md.research.elamur.ai/free/en/papers/minaee-2024-large-language-models-survey/). Preprint, arXiv:2402.06196.
10. Fengfei Sun, Ningke Li, Kailong Wang, and Lorenz Goette. 2025. Large language models are overconfident and amplify human bias. arXiv preprint arXiv:2505.02151. — [arXiv:2505.02151](https://arxiv.org/abs/2505.02151) [find](https://scholar.google.com/scholar?q=Fengfei+Sun%2C+Ningke+Li%2C+Kailong+Wang%2C+and+Lorenz+Goette%2E+2025%2E+Large+language+models+are+overconfident+and+amplify+human+bias%2E+arXiv+preprint+arXiv%3A2505%2E02151%2E+%E2%80%94+%5BarXiv%3A2505%2E02151%5D%28https%3A%2F%2Farxiv%2Eorg%2Fa)
11. Liqiang Ming, Sheng-hua Zhong, and Luncong Li. 2025. Towards general-domain word sense disambiguation: Distilling large language model into compact disambiguator. In Conference on Empirical Methods in Natural Language Processing, pages 884–897. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Liqiang+Ming%2C+Sheng%2Dhua+Zhong%2C+and+Luncong+Li%2E+2025%2E+Towards+general%2Ddomain+word+sense+disambiguation%3A+Distilling+large+language+model+into+compact+disambiguator%2E+In+Conference+on+Empirical+Methods+in)
12. Jean Veronis and Nancy Ide. 1990. Word sense disambiguation with very large neural networks extracted from machine readable dictionaries. In COLING 1990 Volume 2: Papers presented to the 13th International Conference on Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jean+Veronis+and+Nancy+Ide%2E+1990%2E+Word+sense+disambiguation+with+very+large+neural+networks+extracted+from+machine+readable+dictionaries%2E+In+COLING+1990+Volume+2%3A+Papers+presented+to+the+13th+Internat)
13. Andrea Moro and Roberto Navigli. 2015. SemEval-2015 task 13: Multilingual all-words sense disambiguation and entity linking. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 288–297, Denver, Colorado. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Andrea+Moro+and+Roberto+Navigli%2E+2015%2E+SemEval%2D2015+task+13%3A+Multilingual+all%2Dwords+sense+disambiguation+and+entity+linking%2E+In+Proceedings+of+the+9th+International+Workshop+on+Semantic+Evaluation+%28Se)
14. Roberto Navigli, David Jurgens, and Daniele Vannella. 2013. SemEval-2013 task 12: Multilingual word sense disambiguation. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages… [find](https://scholar.google.com/scholar?q=Roberto+Navigli%2C+David+Jurgens%2C+and+Daniele+Vannella%2E+2013%2E+SemEval%2D2013+task+12%3A+Multilingual+word+sense+disambiguation%2E+In+Second+Joint+Conference+on+Lexical+and+Computational+Semantics+%28%2ASEM%29%2C+Volu)

*Updated 31 October 2026*