--- title: "Comhis at SemEval-2026 Task 4 — paper by Shu et al. (2026)" description: "This paper describes the Comhis team's entry to SemEval-2026 Task 4. 0.68 accuracy. 6 references, cited by 1 paper here." authors: ["Ke Shu", "Eetu Mäkelä", "Mikko Tolonen"] year: 2026 venue: "Proceedings of the 20th International Workshop on Semantic Evaluation (2026)" source: "https://aclanthology.org/2026.semeval-1.118/" doi: "10.18653/v1/2026.semeval-1.118" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/shu-2026-comhis-semeval-2026-task-4/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shu-2026-comhis-semeval-2026-task-4/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/11/) / Comhis at SemEval-2026 Task 4: Embedding-Space Adaptation and LLM-Assisted Inference for Narrative Similarity # Comhis at SemEval-2026 Task 4: Embedding-Space Adaptation and LLM-Assisted Inference for Narrative Similarity **Authors:** [Ke Shu](https://md.research.elamur.ai/free/en/authors/ke-shu/), [Eetu Mäkelä](https://md.research.elamur.ai/free/en/authors/eetu-makela/) and [Mikko Tolonen](https://md.research.elamur.ai/free/en/authors/mikko-tolonen/) · **Year:** 2026 · **Venue:** [Proceedings of the 20th International Workshop on Semantic Evaluation (2026)](https://md.research.elamur.ai/free/en/venues/semeval/) ([SemEval 2026](https://md.research.elamur.ai/free/en/venues/semeval/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.semeval-1.118/) · [PDF](https://aclanthology.org/2026.semeval-1.118.pdf) · [DOI 10.18653/v1/2026.semeval-1.118](https://doi.org/10.18653/v1/2026.semeval-1.118) · [OpenAlex](https://openalex.org/W7166834492) **Citations:** 0 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read ## Summary This paper describes the Comhis team's entry to SemEval-2026 Task 4. The task asks which of two candidate stories is narratively closer to an anchor story, judged by theme, course of action and outcome rather than word overlap. The system has two stages. - **Track B (representation learning):** Qwen3-Embedding-8B, served with vLLM, stays frozen. Only a linear projection layer on top of it is trained. This layer starts close to the identity matrix and is optimised with a cosine-based triplet loss and AdamW. Training uses synthetic comparison data plus the official example data. Hard examples are upweighted through weighted sampling; these are training/dev items the base embeddings ranked wrongly or with small margins. - **Track A (pairwise judgment):** cosine similarities from the adapted space, and their difference, are added to the prompt of an LLM, which makes the final choice. If the LLM output cannot be parsed, the system falls back to the raw embedding comparison. Test-set accuracy came to 0.68 for Track A and 0.66 for Track B, both higher than the baselines. The work shows that adapting a frozen embedding model cheaply, then pairing it with an LLM, is an efficient route to modelling story-level similarity. ## Key points - Official test results: Track A 0.68 accuracy (20th of 44 teams); Track B 0.66 (10th of 27). - Development results: full system 0.78 (Track A) and 0.82 (Track B), against baselines of 0.48 and 0.58. - Ablation on test (A/B): without synthetic data 0.57/0.55; without hard example mining 0.54/0.53. Removing hard example mining causes the larger drop. - Errors cluster where the two candidates' similarity scores are close. Recurring causes are lexical-overlap bias, stories that share a theme but end differently, and loosely structured episodic plots. - The LLM usually follows the embedding scores but sometimes overrides them, especially when themes match but outcomes differ. The authors read Track A's slight test advantage (0.68 vs 0.66) as a sign of this added value, although on dev Track B scored higher. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of Helsinki](https://md.research.elamur.ai/free/en/universities/university-of-helsinki/) — University, Helsinki, Finland · 50 works · 5,318 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works 🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works ## Cite Cite as: Ke Shu, Eetu Mäkelä and Mikko Tolonen. 2026. Comhis at SemEval-2026 Task 4: Embedding-Space Adaptation and LLM-Assisted Inference for Narrative Similarity. Proceedings of the 20th International Workshop on Semantic Evaluation (2026). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{shu2026comhis, title = {{Comhis at SemEval-2026 Task 4: Embedding-Space Adaptation and LLM-Assisted Inference for Narrative Similarity}}, author = {Ke Shu, Eetu Mäkelä and Mikko Tolonen}, booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)}, year = {2026}, doi = {10.18653/v1/2026.semeval-1.118}, url = {https://doi.org/10.18653/v1/2026.semeval-1.118} } ``` - [← previous ComfyUI-R1: Exploring Reasoning Models for Workflow Generation](https://md.research.elamur.ai/free/en/papers/xu-2026-comfyui-r1-exploring-reasoning-models/) - [next → Command-Line Obfuscation Detection in Real-World Telemetry under Extreme Class Imbalance](https://md.research.elamur.ai/free/en/papers/outrata-2026-command-line-obfuscation-detection-real/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/11/) ## References 6 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Xi Chen, Ali Zeynali, Chico Camargo, Fabian Flöck, Devin Gaffney, Przemyslaw Grabowicz, Scott A. Hale, David Jurgens, and Mattia Samory. 2022. SemEval-2022 task 8: Multilingual news article similarity. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 109… [find](https://scholar.google.com/scholar?q=Xi+Chen%2C+Ali+Zeynali%2C+Chico+Camargo%2C+Fabian+Fl%C3%B6ck%2C+Devin+Gaffney%2C+Przemyslaw+Grabowicz%2C+Scott+A%2E+Hale%2C+David+Jurgens%2C+and+Mattia+Samory%2E+2022%2E+SemEval%2D2022+task+8%3A+Multilingual+news+article+similarity) 2. Gabriel de Souza P. Moreira, Radek Osmulski, Mengyao Xu, Ronay Ak, Benedikt Schifferer, and Even Oldridge. 2025. Nv-retriever: Improving text embedding models with effective hard-negative mining. Preprint, arXiv:2407.15831. — [arXiv:2407.15831](https://arxiv.org/abs/2407.15831) [find](https://scholar.google.com/scholar?q=Gabriel+de+Souza+P%2E+Moreira%2C+Radek+Osmulski%2C+Mengyao+Xu%2C+Ronay+Ak%2C+Benedikt+Schifferer%2C+and+Even+Oldridge%2E+2025%2E+Nv%2Dretriever%3A+Improving+text+embedding+models+with+effective+hard%2Dnegative+mining%2E+Prep) 3. Hans Ole Hatzel, Ekaterina Artemova, Haimo Stiemer, Evelyn Gius, and Chris Biemann. 2026. SemEval-2026 Task 4: Narrative similarity and narrative representation learning. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, CA, USA. Association for Comp… [find](https://scholar.google.com/scholar?q=Hans+Ole+Hatzel%2C+Ekaterina+Artemova%2C+Haimo+Stiemer%2C+Evelyn+Gius%2C+and+Chris+Biemann%2E+2026%2E+SemEval%2D2026+Task+4%3A+Narrative+similarity+and+narrative+representation+learning%2E+In+Proceedings+of+the+20th+In) 4. Hans Ole Hatzel and Chris Biemann. 2024. Story embeddings — narrative-focused representations of fictional stories. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5931–5943, Miami, Florida, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Hans+Ole+Hatzel+and+Chris+Biemann%2E+2024%2E+Story+embeddings+%E2%80%94+narrative%2Dfocused+representations+of+fictional+stories%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Natural+Language+Proce) 5. Anand Kumar and Hemanth Kumar. 2024. [scaLAR SemEval-2024 task 1: Semantic textual relatednes for English](https://md.research.elamur.ai/free/en/papers/kumar-2024-scalar-semeval-2024-task-1/). In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), pages 902–906, Mexico City, Mexico. Association for Computational Linguistics. 6. Arjumand Younus and Muhammad Atif Qureshi. 2025. nlptuducd at SemEval-2025 task 10: Narrative classification as a retrieval task through story embeddings. In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025), pages 1742–1746, Vienna, Austria. Association for Comput… [find](https://scholar.google.com/scholar?q=Arjumand+Younus+and+Muhammad+Atif+Qureshi%2E+2025%2E+nlptuducd+at+SemEval%2D2025+task+10%3A+Narrative+classification+as+a+retrieval+task+through+story+embeddings%2E+In+Proceedings+of+the+19th+International+Work) *Updated 31 October 2026*