---
title: "NCL-UoR at SemEval-2026 Task 5 — paper by Wu et al. (2026)"
description: "The authors compare three families of methods: Embedding-based… 0.731 accuracy. 20 references, cited by 1 paper here."
authors: ["Tong Wu", "Thanet Markchom", "Huizhi(elly) Liang"]
year: 2026
venue: "Proceedings of the 20th International Workshop on Semantic Evaluation (2026)"
source: "https://aclanthology.org/2026.semeval-1.242/"
doi: "10.18653/v1/2026.semeval-1.242"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wu-2026-ncl-uor-semeval-2026-task/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wu-2026-ncl-uor-semeval-2026-task/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/15/) / NCL-UoR at SemEval-2026 Task 5: Embedding-Based Methods, Fine-Tuning, and LLMs for Word Sense Plausibility Rating

# NCL-UoR at SemEval-2026 Task 5: Embedding-Based Methods, Fine-Tuning, and LLMs for Word Sense Plausibility Rating

**Authors:** [Tong Wu](https://md.research.elamur.ai/free/en/authors/tong-wu/), Thanet Markchom and [Huizhi(elly) Liang](https://md.research.elamur.ai/free/en/authors/huizhi-liang/) · **Year:** 2026 · **Venue:** [Proceedings of the 20th International Workshop on Semantic Evaluation (2026)](https://md.research.elamur.ai/free/en/venues/semeval/) ([SemEval 2026](https://md.research.elamur.ai/free/en/venues/semeval/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.semeval-1.242/) · [PDF](https://aclanthology.org/2026.semeval-1.242.pdf) · [DOI 10.18653/v1/2026.semeval-1.242](https://doi.org/10.18653/v1/2026.semeval-1.242) · [OpenAlex](https://openalex.org/W7166841505)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary
This SemEval-2026 Task 5 system paper tackles graded word sense plausibility. Given a five-sentence English story from the AmbiStory dataset with an ambiguous homonym and a candidate sense, a system must predict how plausible human annotators find that sense on a 1–5 scale. The authors compare three families of methods:

- **Embedding-based:** sentence-embedding similarity features fed into Ridge or XGBoost regressors.
- **Fine-tuned encoders:** ELECTRA and DeBERTa-large with LoRA. The DeBERTa variant adds a RankNet pairwise loss and a loss that tolerates errors within the annotator standard deviation.
- **LLM prompting:** few-shot prompting versus a structured prompt. The structured prompt scores precontext, target sentence and ending separately, then applies explicit calibration rules (e.g., a contradicting ending caps the rating at 1–2).

The structured prompt with GPT-4o performed best on the test set (Spearman ρ = 0.731, accuracy = 0.794). It beat both the fine-tuned and embedding systems. The work suggests that a task-specific reasoning scaffold matters more than model size. It also shows that the remaining errors stem mostly from calibration under mixed evidence.

## Key points
- **Data:** AmbiStory has 2,280 train, 588 dev and 930 test samples. Gold labels are the average of at least five annotators. Metrics are Spearman ρ and accuracy, where a prediction counts as correct if it falls within one standard deviation of the mean rating.
- **Development set:**
  - Embedding baselines were weak: MPNet + Ridge reached ρ = 0.174 and RoBERTa + XGBoost ρ = 0.114.
  - GPT-4o (P2) reached ρ = 0.749 and accuracy 0.818.
  - Llama-3.2-3B with the same prompt reached only ρ = 0.108.
- **Uncertainty loss:** it helped DeBERTa-large + LoRA on dev but hurt on test (ρ fell from 0.492 to 0.435). The authors attribute this to overfitting during dev-based model selection.
- **Error analysis:**
  - High-disagreement items (σ ≥ 1.0) are harder, with MAE 0.962 versus 0.765 for low-disagreement items.
  - Mid-range ratings are the hardest to predict.
  - Integer-only outputs introduce a discretization bias.
- **Prompt comparison and code:** GPT-4o (P2) had a lower MAE (0.702) than GPT-5.2 with few-shot prompting (0.791). Code is available at github.com/tongwu17/SemEval-2026-Task5.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Newcastle University](https://md.research.elamur.ai/free/en/universities/newcastle-university/) — University, Newcastle upon Tyne, United Kingdom · 18 works · 619 citations
- [University of Reading](https://md.research.elamur.ai/free/en/universities/university-of-reading/) — University, Reading, United Kingdom · 2 works · 5 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🔀 Word sense disambiguation](https://md.research.elamur.ai/free/en/topics/word-sense-disambiguation/) · 83 works

## Cite

Cite as: Tong Wu, Thanet Markchom and Huizhi(elly) Liang. 2026. NCL-UoR at SemEval-2026 Task 5: Embedding-Based Methods, Fine-Tuning, and LLMs for Word Sense Plausibility Rating. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wu2026ncl,
  title = {{NCL-UoR at SemEval-2026 Task 5: Embedding-Based Methods, Fine-Tuning, and LLMs for Word Sense Plausibility Rating}},
  author = {Tong Wu, Thanet Markchom and Huizhi(elly) Liang},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.242},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.242}
}
```

- [← previous NCL-BU at SemEval-2026 Task 3: Fine-tuning XLM-RoBERTa for Multilingual Dimensional Sentiment Regression](https://md.research.elamur.ai/free/en/papers/wu-2026-ncl-bu-semeval-2026-task/)
- [next → Nearest-Neighbor Retrieval for Indigenous Image Captioning](https://md.research.elamur.ai/free/en/papers/vasselli-2026-nearest-neighbor-retrieval-indigenous-image/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/15/)

## References

20 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and… [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretchen)
2. Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. 2005. [Learning to rank using gradient descent](https://md.research.elamur.ai/free/en/papers/burges-2005-learning-rank-using-gradient/). In Proceedings of the 22nd International Conference on Machine Learning, ICML ’05, page 89–96, New York, NY, USA. Association for Computing Machinery.
3. Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 785–794, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Tianqi+Chen+and+Carlos+Guestrin%2E+2016%2E+Xgboost%3A+A+scalable+tree+boosting+system%2E+In+Proceedings+of+the+22nd+ACM+SIGKDD+International+Conference+on+Knowledge+Discovery+and+Data+Mining%2C+KDD+%E2%80%9916%2C+page+78)
4. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Minh%2DThang+Luong%2C+Quoc+V%2E+Le%2C+and+Christopher+D%2E+Manning%2E+2020%2E+Electra%3A+Pre%2Dtraining+text+encoders+as+discriminators+rather+than+generators%2E+In+International+Conference+on+Learning+Repre)
5. Katrin Erk and Diana McCarthy. 2009. Graded word sense assignment. In Proceedings of EMNLP, pages 440–449, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Katrin+Erk+and+Diana+McCarthy%2E+2009%2E+Graded+word+sense+assignment%2E+In+Proceedings+of+EMNLP%2C+pages+440%E2%80%93449%2C+Singapore%2E+Association+for+Computational+Linguistics%2E)
6. Janosch Gehring, Selina Meyer, and Michael Roth. 2026. SemEval-2026 task 5: Rating plausibility of word senses in ambiguous stories through narrative understanding. In Proceedings of the 20th International Workshop on Semantic Evaluation, San Diego, California. Association for Computational Linguist… [find](https://scholar.google.com/scholar?q=Janosch+Gehring%2C+Selina+Meyer%2C+and+Michael+Roth%2E+2026%2E+SemEval%2D2026+task+5%3A+Rating+plausibility+of+word+senses+in+ambiguous+stories+through+narrative+understanding%2E+In+Proceedings+of+the+20th+Internat)
7. Janosch Gehring and Michael Roth. 2025. AmbiStory: A challenging dataset of lexically ambiguous short stories. In Proceedings of the 14th Joint Conference on Lexical and Computational Semantics (*SEM 2025), pages 152–171, Suzhou, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Janosch+Gehring+and+Michael+Roth%2E+2025%2E+AmbiStory%3A+A+challenging+dataset+of+lexically+ambiguous+short+stories%2E+In+Proceedings+of+the+14th+Joint+Conference+on+Lexical+and+Computational+Semantics+%28%2ASEM+)
8. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, and 1 others. 2024. [The Llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). Preprint, arXiv:2407.21783.
9. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. [DeBERTa: Decoding-Enhanced BERT with Disentangled Attention](https://md.research.elamur.ai/free/en/papers/he-2020-deberta-decoding-enhanced-bert-disentangled/). In International Conference on Learning Representations.
10. Arthur E. Hoerl and Robert W. Kennard. 2000. Ridge regression: biased estimation for nonorthogonal problems. Technometrics, 42(1):80–86. [find](https://scholar.google.com/scholar?q=Arthur+E%2E+Hoerl+and+Robert+W%2E+Kennard%2E+2000%2E+Ridge+regression%3A+biased+estimation+for+nonorthogonal+problems%2E+Technometrics%2C+42%281%29%3A80%E2%80%9386%2E)
11. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. [LoRA: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In International Conference on Learning Representations.
12. Peter J. Huber. 1964. Robust estimation of a location parameter. The Annals of Mathematical Statistics, 35(1):73–101. [find](https://scholar.google.com/scholar?q=Peter+J%2E+Huber%2E+1964%2E+Robust+estimation+of+a+location+parameter%2E+The+Annals+of+Mathematical+Statistics%2C+35%281%29%3A73%E2%80%93101%2E)
13. Alexander H. Liu, Kartik Khandelwal, Sandeep Subramanian, and 1 others. 2026. Ministral 3. Preprint, arXiv:2601.08584. [find](https://scholar.google.com/scholar?q=Alexander+H%2E+Liu%2C+Kartik+Khandelwal%2C+Sandeep+Subramanian%2C+and+1+others%2E+2026%2E+Ministral+3%2E+Preprint%2C+arXiv%3A2601%2E08584%2E)
14. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. [Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/). ACM Comput. Surv., 55(9).
15. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). Preprint, arXiv:1907.11692.
16. Roberto Navigli. 2009. Word sense disambiguation: A survey. ACM Comput. Surv., 41(2). [find](https://scholar.google.com/scholar?q=Roberto+Navigli%2E+2009%2E+Word+sense+disambiguation%3A+A+survey%2E+ACM+Comput%2E+Surv%2E%2C+41%282%29%2E)
17. Nils Reimers and Iryna Gurevych. 2019. [Sentence-BERT: Sentence embeddings using Siamese BERT-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,…
18. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. [Mpnet: masked and permuted pre-training for language understanding](https://md.research.elamur.ai/free/en/papers/song-2020-mpnet-masked-permuted-pre-training/). In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA. Curran Associates Inc.
19. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. [Transf…](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
20. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. [Judging LLM-as-a-judge with MT-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). In Proceedings of the 37th International Conference on Neu…

*Updated 31 October 2026*