--- title: "Cryptix at SemEval-2026 Task 4 — paper by M et al. (2026)" description: "The paper presents Team Cryptix's entry to SemEval-2026 Task 4, which asks systems to judge… 6 references, cited by 1 paper here." authors: ["Sushmitha M", "Sarath Kumar P", "Thanalaxmi S", "Beulah A"] year: 2026 venue: "Proceedings of the 20th International Workshop on Semantic Evaluation (2026)" source: "https://aclanthology.org/2026.semeval-1.168/" doi: "10.18653/v1/2026.semeval-1.168" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/m-2026-cryptix-semeval-2026-task-4/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/m-2026-cryptix-semeval-2026-task-4/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/11/) / Cryptix at SemEval-2026 Task 4: Zero-Shot Bi-Encoder Modeling for Narrative Story Similarity - A Sentence Transformer Approach # Cryptix at SemEval-2026 Task 4: Zero-Shot Bi-Encoder Modeling for Narrative Story Similarity - A Sentence Transformer Approach **Authors:** Sushmitha M, Sarath Kumar P, Thanalaxmi S and Beulah A · **Year:** 2026 · **Venue:** [Proceedings of the 20th International Workshop on Semantic Evaluation (2026)](https://md.research.elamur.ai/free/en/venues/semeval/) ([SemEval 2026](https://md.research.elamur.ai/free/en/venues/semeval/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.semeval-1.168/) · [PDF](https://aclanthology.org/2026.semeval-1.168.pdf) · [DOI 10.18653/v1/2026.semeval-1.168](https://doi.org/10.18653/v1/2026.semeval-1.168) · [OpenAlex](https://openalex.org/W7166806008) **Citations:** 0 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read ## Summary The paper presents Team Cryptix's entry to SemEval-2026 Task 4, which asks systems to judge narrative similarity between story summaries. The judgment rests on theme, course of action and outcome, not word overlap. Data come as triples: an anchor story and two candidates, one of which is narratively closer. The team entered Track A, which makes a direct choice between candidates, and Track B, which outputs story embeddings scored by cosine distance. They did no training at all. The pretrained sentence-transformers/all-mpnet-base-v2 bi-encoder encodes each summary separately, truncated to 2000 characters, using mean pooling and L2 normalization. Cosine similarity then picks the closer candidate. Track A accuracy was 59.50%, ranking 37th, while Track B accuracy was 57.50%, ranking 25th. This beats chance but falls well short of the GPT-4o-mini baseline on Track A. The authors back this up with ablations and margin and error analyses. The work matters as a reproducible zero-shot reference point. It shows that general sentence encoders mainly capture topical overlap and miss causal progression and outcome alignment. ## Key points - Official baselines were GPT-4o-mini at 73.00% (Track A) and 58.00% (Track B), and MiniLM-L6-v2 at 62.00% and 56.00%. Cryptix slightly beats MiniLM on Track B only. - Development-set ablations (MPNet baseline 61.5%): - Removing normalization had no effect, at 61.5%. - Sentence-level averaging dropped accuracy to 56.5%. - Truncating at 1000 characters gave 62.0%. - Swapping in RoBERTa-Large gave 57.5%. - The authors caution that 1–2 point differences may not be statistically significant, citing McNemar's test as one way to check. - Errors cluster near the decision boundary. The average cosine margin is 0.0606 for wrong predictions versus 0.1101 for correct ones. - The main failure mode is shared themes despite different event structure. Other failure types are: - missing structural parallels when wording differs; - diverging outcomes; - surface lexical bias. - Proposed next steps are: - fine-tuning on the training triples with triplet or MNR loss; - cross-encoders; - event-graph representations; - ensembles with generative LLM judgments. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works ## Cite Cite as: Sushmitha M, Sarath Kumar P, Thanalaxmi S and Beulah A. 2026. Cryptix at SemEval-2026 Task 4: Zero-Shot Bi-Encoder Modeling for Narrative Story Similarity - A Sentence Transformer Approach. Proceedings of the 20th International Workshop on Semantic Evaluation (2026). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{m2026cryptix, title = {{Cryptix at SemEval-2026 Task 4: Zero-Shot Bi-Encoder Modeling for Narrative Story Similarity - A Sentence Transformer Approach}}, author = {Sushmitha M, Sarath Kumar P, Thanalaxmi S and Beulah A}, booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)}, year = {2026}, doi = {10.18653/v1/2026.semeval-1.168}, url = {https://doi.org/10.18653/v1/2026.semeval-1.168} } ``` - [← previous CrowS-Pairs-NL: A Benchmark to Evaluate Dutch Stereotype Bias in LLMs](https://md.research.elamur.ai/free/en/papers/weide-2026-crows-pairs-nl-benchmark-evaluate/) - [next → CS_Metro at PsyDefDetect: Detecting Psychological Defense Mechanisms in Mental Health Dialogues with Summarization-Enhanced Transformer Ensembles](https://md.research.elamur.ai/free/en/papers/rebayet-2026-cs-metro-psydefdetect-detecting-psychological/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/11/) ## References 6 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. [Longformer: The long-document transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/). In Advances in Neural Information Processing Systems, volume 33, pages 16156–16170. 2. Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. [SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation](https://md.research.elamur.ai/free/en/papers/cer-2017-semeval-2017-task-1-semantic/). In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14, Vancouver, Canada.… 3. Nathanael Chambers and Dan Jurafsky. 2008. Unsupervised learning of narrative event chains. In Proceedings of ACL-08: HLT, pages 789–797, Columbus, Ohio. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nathanael+Chambers+and+Dan+Jurafsky%2E+2008%2E+Unsupervised+learning+of+narrative+event+chains%2E+In+Proceedings+of+ACL%2D08%3A+HLT%2C+pages+789%E2%80%93797%2C+Columbus%2C+Ohio%2E+Association+for+Computational+Linguistics%2E) 4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… 5. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [RoBERTa: A robustly optimized BERT pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 6. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT- [find](https://scholar.google.com/scholar?q=Nils+Reimers+and+Iryna+Gurevych%2E+2019%2E+Sentence%2DBERT%3A+Sentence+embeddings+using+Siamese+BERT%2D) *Updated 31 October 2026*