--- title: "CiNet-Handai-Kyodai at SemEval-2026 Task 5: Combining LLM" description: "This system description addresses SemEval-2026 Task 5, which asks how plausible a… 79.19 Spearman. 4 references, cited by 1 paper here." authors: ["Lis Kanashiro Pereira", "Fei Cheng"] year: 2026 venue: "Proceedings of the 20th International Workshop on Semantic Evaluation (2026)" source: "https://aclanthology.org/2026.semeval-1.245/" doi: "10.18653/v1/2026.semeval-1.245" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/pereira-2026-cinet-handai-kyodai-semeval-2026/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pereira-2026-cinet-handai-kyodai-semeval-2026/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/11/) / CiNet-Handai-Kyodai at SemEval-2026 Task 5: Combining LLM Prompting, Semantic Similarity, and Synthetic Gaze for Graded Sense Plausibility # CiNet-Handai-Kyodai at SemEval-2026 Task 5: Combining LLM Prompting, Semantic Similarity, and Synthetic Gaze for Graded Sense Plausibility **Authors:** [Lis Kanashiro Pereira](https://md.research.elamur.ai/free/en/authors/lis-pereira/) and [Fei Cheng](https://md.research.elamur.ai/free/en/authors/fei-cheng/) · **Year:** 2026 · **Venue:** [Proceedings of the 20th International Workshop on Semantic Evaluation (2026)](https://md.research.elamur.ai/free/en/venues/semeval/) ([SemEval 2026](https://md.research.elamur.ai/free/en/venues/semeval/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.semeval-1.245/) · [PDF](https://aclanthology.org/2026.semeval-1.245.pdf) · [DOI 10.18653/v1/2026.semeval-1.245](https://doi.org/10.18653/v1/2026.semeval-1.245) · [OpenAlex](https://openalex.org/W7166869928) **Citations:** 0 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read ## Summary This system description addresses SemEval-2026 Task 5, which asks how plausible a given word sense is for a homonym inside a short English story. Several senses may fit at once, so systems output a 1–5 rating rather than a single label. The task uses the AmbiStory dataset. The authors observe that direct LLM prompting tends to favour dominant senses and miss subtle narrative cues. Their system starts with a GPT-4.1 zero-shot plausibility rating and adds three auxiliary signals: - embedding-based similarity features from OpenAI text-embedding-3-large; - a story-specific dictionary-style definition generated by the LLM and compared with the candidate sense; - a synthetic gaze value, namely predicted total fixation time on the target word. An ordinary least squares regressor trained on the training split combines these signals. The best configuration reaches 90.10 Acc±SD and 79.19 Spearman on the test set. Its Acc±SD exceeds the human reference, while its rank correlation stays below it. The work shows that cheap semantic and cognitively motivated features can improve LLM judgments on graded lexical-semantic tasks. ## Key points - AmbiStory has 3,798 instances (633 setups of six), split 2,280 / 588 / 930 for train / dev / test. Splits are stratified by homonym to prevent lexical overlap. - On test, scores rise as features are added: - Prompt + EmbSim: 89.46 / 76.09 - adding Def.Gen.: 89.89 / 78.87 - adding TFT: 90.10 / 79.19 - human reference: 89.20 / 83.40 - Synthetic gaze comes from the RoBERTa-based model of Li and Rudzicz (2021). Only total fixation time (TFT) on the target word is used. - Fine-tuned baselines performed much worse on dev: DeBERTa-v3-large scored 66.03 / 66.03 and gpt-oss-20b scored 73.00 / 65.00. - The TFT analysis shows where gaze helps and where it hurts: - It lowers over-prediction bias (mean error 0.06 → 0.02). - It improves low-plausibility items. - It hurts highly plausible idiomatic cases, such as "got to the point". *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Kyoto University](https://md.research.elamur.ai/free/en/universities/kyoto-university/) — University, Kyoto, Japan · 40 works · 8,235 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works 🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works ## Cite Cite as: Lis Kanashiro Pereira and Fei Cheng. 2026. CiNet-Handai-Kyodai at SemEval-2026 Task 5: Combining LLM Prompting, Semantic Similarity, and Synthetic Gaze for Graded Sense Plausibility. Proceedings of the 20th International Workshop on Semantic Evaluation (2026). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{pereira2026cinet, title = {{CiNet-Handai-Kyodai at SemEval-2026 Task 5: Combining LLM Prompting, Semantic Similarity, and Synthetic Gaze for Graded Sense Plausibility}}, author = {Lis Kanashiro Pereira and Fei Cheng}, booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)}, year = {2026}, doi = {10.18653/v1/2026.semeval-1.245}, url = {https://doi.org/10.18653/v1/2026.semeval-1.245} } ``` - [← previous CICL26 at SemEval-2026 Task 4: Narrative Story Similarity and Narrative Representation Learning](https://md.research.elamur.ai/free/en/papers/zhang-2026-cicl26-semeval-2026-task-4/) - [next → CITD@UIT at SemEval-2026 Task 2: Temporal Mixture-of-Experts for Longitudinal Valence and Arousal Prediction from Ecological Essays](https://md.research.elamur.ai/free/en/papers/phuong-2026-citd-uit-semeval-2026-task/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/11/) ## References 4 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. \# CiNet-Handai-Kyodai at SemEval-2026 Task 5: Combining LLM Prompting, Semantic Similarity, and Synthetic Gaze for Graded Sense Plausibility Lis Kanashiro Pereira, Fei Cheng · Proceedings of the 20th International Workshop on Semantic Evaluation (2026) · 2026 [find](https://scholar.google.com/scholar?q=%5C%23+CiNet%2DHandai%2DKyodai+at+SemEval%2D2026+Task+5%3A+Combining+LLM+Prompting%2C+Semantic+Similarity%2C+and+Synthetic+Gaze+for+Graded+Sense+Plausibility+Lis+Kanashiro+Pereira%2C+Fei+Cheng+%C2%B7+Proceedings+of+the+20th) 2. Source: https://aclanthology.org/2026.semeval-1.245/ · License: CC BY 4.0 (ACL Anthology). Text extracted automatically from the PDF (poppler): reading-order and hyphenation errors are possible; formulas are Unicode, not LaTeX. CiNet-Handai-Kyodai at SemEval-2026 Task 5: Combining LLM Prompting, Sem… — [ACL 2026.semeval-1.245](https://aclanthology.org/2026.semeval-1.245/) [↗](https://aclanthology.org/2026.semeval-1.245/) 3. Task Description SemEval-2026 Task 5 (Gehring et al., 2026) focuses on rating the plausibility of word senses in lexically ambiguous sentences through narrative understanding. Unlike traditional Word Sense Disambiguation (WSD), which typically assumes a single *Figure 1: Overview of SemEval-2026 Tas… [find](https://scholar.google.com/scholar?q=Task+Description+SemEval%2D2026+Task+5+%28Gehring+et+al%2E%2C+2026%29+focuses+on+rating+the+plausibility+of+word+senses+in+lexically+ambiguous+sentences+through+narrative+understanding%2E+Unlike+traditional+Word+) 4. Method Our system is a hybrid of prompt-based plausibility rating and auxiliary features derived from embeddings and synthetic gaze. Specifically, we: (i) query an LLM with a prompt based on the AmbiStory prompt format (Gehring and Roth, 2025) to obtain a plausibility score on the 1–5 scale (§3.1); [find](https://scholar.google.com/scholar?q=Method+Our+system+is+a+hybrid+of+prompt%2Dbased+plausibility+rating+and+auxiliary+features+derived+from+embeddings+and+synthetic+gaze%2E+Specifically%2C+we%3A+%28i%29+query+an+LLM+with+a+prompt+based+on+the+AmbiS) *Updated 31 October 2026*