--- title: "CuriosAI at SemEval-2026 Task 4 — paper by Shibata et al. (2026)" description: "The paper describes a SoftBank team's entry to SemEval-2026 Task 4, which deals with judging how… 10 references, cited by 1 paper here." authors: ["Yuki Shibata", "Hiroki Takushima", "Fumika Beppu", "Aiswariya Manoj Kumar", "Daichi Yamaga", "Takayuki Hori"] year: 2026 venue: "Proceedings of the 20th International Workshop on Semantic Evaluation (2026)" source: "https://aclanthology.org/2026.semeval-1.62/" doi: "10.18653/v1/2026.semeval-1.62" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/shibata-2026-curiosai-semeval-2026-task-4/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shibata-2026-curiosai-semeval-2026-task-4/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/11/) / CuriosAI at SemEval-2026 Task 4: A Comprehensive Study of Zero-Shot versus Fine-Tuned Approaches for Narrative Similarity # CuriosAI at SemEval-2026 Task 4: A Comprehensive Study of Zero-Shot versus Fine-Tuned Approaches for Narrative Similarity **Authors:** [Yuki Shibata](https://md.research.elamur.ai/free/en/authors/yuki-shibata/), [Hiroki Takushima](https://md.research.elamur.ai/free/en/authors/hiroki-takushima/), [Fumika Beppu](https://md.research.elamur.ai/free/en/authors/fumika-beppu/), [Aiswariya Manoj Kumar](https://md.research.elamur.ai/free/en/authors/aiswariya-manoj-kumar/), [Daichi Yamaga](https://md.research.elamur.ai/free/en/authors/daichi-yamaga/) and [Takayuki Hori](https://md.research.elamur.ai/free/en/authors/takayuki-hori/) · **Year:** 2026 · **Venue:** [Proceedings of the 20th International Workshop on Semantic Evaluation (2026)](https://md.research.elamur.ai/free/en/venues/semeval/) ([SemEval 2026](https://md.research.elamur.ai/free/en/venues/semeval/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.semeval-1.62/) · [PDF](https://aclanthology.org/2026.semeval-1.62.pdf) · [DOI 10.18653/v1/2026.semeval-1.62](https://doi.org/10.18653/v1/2026.semeval-1.62) · [OpenAlex](https://openalex.org/W7166840709) **Citations:** 0 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read ## Summary The paper describes a SoftBank team's entry to SemEval-2026 Task 4, which deals with judging how similar English short stories are at the narrative level. Track A asks which of two candidate stories is closer to an anchor. Track B asks for a fixed-size embedding per story. The authors compared three families of approaches: - off-the-shelf sentence encoders used without training, scoring candidates by cosine similarity; - multi-step prompting of GPT-4o and GPT-5; - fine-tuning bi-encoders and a cross-encoder on roughly 3,800 synthetic, LLM-generated training samples. The untrained Sentence-T5-XL turned out to be the strongest system, reaching 67.5% on the 200-item development set. Every fine-tuned variant performed worse, losing between 2 and 18 points, and longer training made results steadily worse even as training loss fell. The authors attribute this to noisy synthetic labels, a stylistic gap between synthetic and real stories, and a different notion of "similarity" in the generated data. The study is a useful negative result: zero-shot performance should be checked before investing in fine-tuning on synthetic data. ## Key points - **Data:** 200 development instances, 400 Track A test triplets and 849 Track B test narratives. Accuracy was the Track A metric. - **Zero-shot development accuracy:** - Sentence-T5-XL: 67.5% (95% CI 61.0–74.0) - mxbai-embed-large-v1: 66.0% - all-mpnet-base-v2: 63.5% - all-MiniLM-L6-v2: 59.5% - paraphrase-MiniLM-L3-v2: 57.0% - **Fine-tuning results:** - Sentence-T5-XL, fine-tuned with a triplet margin loss: fell from 67.5% to 61.5% after 1 epoch and to 49.5% after 10 epochs. - ms-marco-MiniLM-L-6-v2 cross-encoder: never beat its 52.0% zero-shot baseline. - **Official test results** (zero-shot Sentence-T5-XL): Track A reached 73.50% (rank 8) and Track B reached 63.00% (rank 15). - **LLM prompting and errors:** The GPT-4o/GPT-5 prompting could not be fully evaluated because of API rate limits and content filtering. Manual error analysis showed the model favours surface lexical overlap over deeper plot and theme alignment. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [SoftBank Corp.](https://md.research.elamur.ai/free/en/companies/softbank-corp/) — Company, Tokyo, Japan · 2 works · 1 citation 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works ## Cite Cite as: Yuki Shibata, Hiroki Takushima, Fumika Beppu, Aiswariya Manoj Kumar, Daichi Yamaga and Takayuki Hori. 2026. CuriosAI at SemEval-2026 Task 4: A Comprehensive Study of Zero-Shot versus Fine-Tuned Approaches for Narrative Similarity. Proceedings of the 20th International Workshop on Semantic Evaluation (2026). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{shibata2026curiosai, title = {{CuriosAI at SemEval-2026 Task 4: A Comprehensive Study of Zero-Shot versus Fine-Tuned Approaches for Narrative Similarity}}, author = {Yuki Shibata, Hiroki Takushima, Fumika Beppu, Aiswariya Manoj Kumar, Daichi Yamaga and Takayuki Hori}, booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)}, year = {2026}, doi = {10.18653/v1/2026.semeval-1.62}, url = {https://doi.org/10.18653/v1/2026.semeval-1.62} } ``` - [← previous CUNI at SemEval-2026 Task 4: Multi-Head Narrative Aspect Disentanglement via Entangled Synthetic Dataset](https://md.research.elamur.ai/free/en/papers/mitka-2026-cuni-semeval-2026-task-4/) - [next → Current Agents Fail to Leverage World Model as Tool for Foresight](https://md.research.elamur.ai/free/en/papers/qian-2026-current-agents-fail-leverage-world/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/11/) ## References 10 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901. 1. Negative results: W… 2. Hans Ole Hatzel, Ekaterina Artemova, Haimo Stiemer, Evelyn Gius, and Chris Biemann. 2026. SemEval-2026 Task 4: Narrative similarity and narrative representation learning. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, CA, USA. Association for Comp… [find](https://scholar.google.com/scholar?q=Hans+Ole+Hatzel%2C+Ekaterina+Artemova%2C+Haimo+Stiemer%2C+Evelyn+Gius%2C+and+Chris+Biemann%2E+2026%2E+SemEval%2D2026+Task+4%3A+Narrative+similarity+and+narrative+representation+learning%2E+In+Proceedings+of+the+20th+In) 3. Mixedbread.ai. 2024. mxbai-embed-large-v1: Mixedbread.ai embedding model. Model card. [find](https://scholar.google.com/scholar?q=Mixedbread%2Eai%2E+2024%2E+mxbai%2Dembed%2Dlarge%2Dv1%3A+Mixedbread%2Eai+embedding+model%2E+Model+card%2E) 4. Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. 2021. [Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models](https://md.research.elamur.ai/free/en/papers/ni-2021-sentence-t5-scalable-sentence-encoders/). arXiv preprint arXiv:2108.08877. 5. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, and 1 others. 2019. [Pytorch: An imperative style, high-performance deep learning library](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/). In Advances in Neural Information Processing Systems. 6. Andrew J Reagan, Lewis Mitchell, Dilan Kiley, Christopher M Danforth, and Peter Sheridan Dodds. 2016. Computational approaches to narrative similarity. PLOS ONE, 11(1):e0146279. [find](https://scholar.google.com/scholar?q=Andrew+J+Reagan%2C+Lewis+Mitchell%2C+Dilan+Kiley%2C+Christopher+M+Danforth%2C+and+Peter+Sheridan+Dodds%2E+2016%2E+Computational+approaches+to+narrative+similarity%2E+PLOS+ONE%2C+11%281%29%3Ae0146279%2E) 7. Nils Reimers and Iryna Gurevych. 2019. [Sentence-bert: Sentence embeddings using siamese bert-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982– 3992,… 8. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. [Mpnet: Masked and permuted pre-training for language understanding](https://md.research.elamur.ai/free/en/papers/song-2020-mpnet-masked-permuted-pre-training/). In Advances in Neural Information Processing Systems. 9. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Wenhui+Wang%2C+Furu+Wei%2C+Li+Dong%2C+Hangbo+Bao%2C+Nan+Yang%2C+and+Ming+Zhou%2E+2020%2E+Minilm%3A+Deep+self%2Dattention+distillation+for+task%2Dagnostic+compression+of+pre%2Dtrained+transformers%2E+In+Advances+in+Neural+Inf) 10. Lucia Zheng, Neel Guha, Brandon R Anderson, Peter Henderson, and Daniel E Ho. 2021. When does pretraining help? assessing self-supervised learning for law and the casehold dataset. arXiv preprint arXiv:2104.08671. [find](https://scholar.google.com/scholar?q=Lucia+Zheng%2C+Neel+Guha%2C+Brandon+R+Anderson%2C+Peter+Henderson%2C+and+Daniel+E+Ho%2E+2021%2E+When+does+pretraining+help%3F+assessing+self%2Dsupervised+learning+for+law+and+the+casehold+dataset%2E+arXiv+preprint+arXi) *Updated 1 November 2026*