---
title: "AI-Monitors at SemEval-2026 Task 4: A Hybrid Embedding and"
description: "The paper addresses SemEval-2026 Task 4 (Track A). Given an anchor story, a system… 78% accuracy. 18 references, cited by 1 paper here."
authors: ["Vishnu Tripathi", "Prakhar Joshi", "Azad Singh", "Pragyanand Saho", "Gaurav Kumar", "Neel Mani", "Piyush Arora"]
year: 2026
venue: "Proceedings of the 20th International Workshop on Semantic Evaluation (2026)"
source: "https://aclanthology.org/2026.semeval-1.271/"
doi: "10.18653/v1/2026.semeval-1.271"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/tripathi-2026-ai-monitors-semeval-2026-task/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tripathi-2026-ai-monitors-semeval-2026-task/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/10/) / AI-Monitors at SemEval-2026 Task 4: A Hybrid Embedding and LLM Ensemble for Narrative Similarity

# AI-Monitors at SemEval-2026 Task 4: A Hybrid Embedding and LLM Ensemble for Narrative Similarity

**Authors:** Vishnu Tripathi, Prakhar Joshi, Azad Singh, Pragyanand Saho, [Gaurav Kumar](https://md.research.elamur.ai/free/en/authors/gaurav-kumar/), Neel Mani and Piyush Arora · **Year:** 2026 · **Venue:** [Proceedings of the 20th International Workshop on Semantic Evaluation (2026)](https://md.research.elamur.ai/free/en/venues/semeval/) ([SemEval 2026](https://md.research.elamur.ai/free/en/venues/semeval/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.semeval-1.271/) · [PDF](https://aclanthology.org/2026.semeval-1.271.pdf) · [DOI 10.18653/v1/2026.semeval-1.271](https://doi.org/10.18653/v1/2026.semeval-1.271) · [OpenAlex](https://openalex.org/W7166883562)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper addresses SemEval-2026 Task 4 (Track A). Given an anchor story, a system must decide which of two candidate stories is closer to it in narrative terms: Abstract Theme, Course of Action and Outcomes, not word overlap. The dataset uses Wikipedia synopses and keeps mainly ambiguous triples on which two commercial LLMs disagreed. Human annotators reach only about 78% accuracy.

The authors built their system in four stages, each tied to a hypothesis:
1. Cosine similarity over dense embeddings, including SimCSE-style projection fine-tuning.
2. Prompts grounded in the official annotation guidelines.
3. Curated k-shot demonstrations.
4. An ensemble whose members were chosen for how differently they err, not only for raw accuracy.

The final majority-vote ensemble combines t5-xxl embeddings, GPT-5-mini and DeepSeek-R1:32b. It reached 75% test accuracy, ranking 3rd of 47 systems and close to the human level. The work suggests three things for comparative semantic tasks: structured LLM prompting beats embedding similarity, demonstration framing matters, and diversity-aware ensembling helps.

## Key points
- **Data:** 200 development triples and 400 test instances. Organiser baselines on test were Jaccard similarity at 56.3% and zero-shot GPT-4o-mini at 67.0%.
- **Test accuracy by component:** t5-xxl 63.0%, DeepSeek-R1:32b with k-shot prompting 69.0%, GPT-5-mini 74.0%. The ensemble reached 81.0% on dev and 75.0% on test, 1 point above its best member on test.
- **Prompt families:** three were compared — a loose baseline, an aspect-grounded version using the guideline definitions with JSON output, and a k-shot version. Labelling demonstrations neutrally ("EXAMPLE") rather than as "HARD EXAMPLE" changed accuracy by about 3 points.
- **Ensemble selection:** members were chosen by their agreement rate with the embedding baseline. Low agreement combined with strong accuracy was taken to signal complementary errors.
- **Error analysis:** surface overlap (shared settings, roles or plot cues) was the main cause of mistakes. Majority voting fixes errors made by one model alone, but it amplifies errors that two models share. The authors propose confidence-based routing and weighted voting as future work.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

Also on the byline: [Swami Rama Himalayan University](https://ror.org/02nw97x94).

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Vishnu Tripathi, Prakhar Joshi, Azad Singh, Pragyanand Saho, Gaurav Kumar, Neel Mani and Piyush Arora. 2026. AI-Monitors at SemEval-2026 Task 4: A Hybrid Embedding and LLM Ensemble for Narrative Similarity. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{tripathi2026ai,
  title = {{AI-Monitors at SemEval-2026 Task 4: A Hybrid Embedding and LLM Ensemble for Narrative Similarity}},
  author = {Vishnu Tripathi, Prakhar Joshi, Azad Singh, Pragyanand Saho, Gaurav Kumar, Neel Mani and Piyush Arora},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.271},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.271}
}
```

- [← previous AHA-Memes: A Fine-Grained Multimodal Benchmark for Understanding Hate in Arabic Memes](https://md.research.elamur.ai/free/en/papers/kmainasi-2026-aha-memes-fine-grained-multimodal/)
- [next → AICOE-Tredence at SemEval-2026 Task 11: Mitigating Content Bias in Syllogisms via Symbolic Logic-Language Decoupling](https://md.research.elamur.ai/free/en/papers/r-2026-aicoe-tredence-semeval-2026-task/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/10/)

## References

18 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Snigdha Chaturvedi, Shashank Srivastava, and Dan Roth. 2018. Where have i heard this story before? identifying narrative similarity in movie remakes. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol… [find](https://scholar.google.com/scholar?q=Snigdha+Chaturvedi%2C+Shashank+Srivastava%2C+and+Dan+Roth%2E+2018%2E+Where+have+i+heard+this+story+before%3F+identifying+narrative+similarity+in+movie+remakes%2E+In+Proceedings+of+the+2018+Conference+of+the+North)
2. Jon Chun. 2024. Aistorysimilarity: Quantifying story similarity using narrative for search, ip infringement, and guided creativity. In Proceedings of the 28th Conference on Computational Natural Language Learning, pages 161–177. [find](https://scholar.google.com/scholar?q=Jon+Chun%2E+2024%2E+Aistorysimilarity%3A+Quantifying+story+similarity+using+narrative+for+search%2C+ip+infringement%2C+and+guided+creativity%2E+In+Proceedings+of+the+28th+Conference+on+Computational+Natural+Langu)
3. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. [Simcse: Simple contrastive learning of sentence embeddings](https://md.research.elamur.ai/free/en/papers/gao-2021-simcse-simple-contrastive-learning-sentence/). In Proceedings of the 2021 conference on empirical methods in natural language processing, pages 6894–6910.
4. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
5. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, and 1 others. 2025. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948.
6. Hans Ole Hatzel, Ekaterina Artemova, Haimo Stiemer, Evelyn Gius, and Chris Biemann. 2026. SemEval-2025 task 4: Narrative similarity and narrative representation learning. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, CA, USA. [find](https://scholar.google.com/scholar?q=Hans+Ole+Hatzel%2C+Ekaterina+Artemova%2C+Haimo+Stiemer%2C+Evelyn+Gius%2C+and+Chris+Biemann%2E+2026%2E+SemEval%2D2025+task+4%3A+Narrative+similarity+and+narrative+representation+learning%2E+In+Proceedings+of+the+20th+In)
7. Hans Ole Hatzel and Chris Biemann. 2024a. Story embeddings—narrative-focused representations of fictional stories. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5931–5943. [find](https://scholar.google.com/scholar?q=Hans+Ole+Hatzel+and+Chris+Biemann%2E+2024a%2E+Story+embeddings%E2%80%94narrative%2Dfocused+representations+of+fictional+stories%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Natural+Language+Proces)
8. Hans Ole Hatzel and Chris Biemann. 2024b. Tell me again! a large-scale dataset of multiple summaries for the same story. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 15732–15741. [find](https://scholar.google.com/scholar?q=Hans+Ole+Hatzel+and+Chris+Biemann%2E+2024b%2E+Tell+me+again%21+a+large%2Dscale+dataset+of+multiple+summaries+for+the+same+story%2E+In+Proceedings+of+the+2024+Joint+International+Conference+on+Computational+Ling)
9. Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, and 1 others. 2022. [Large dual encoders are generalizable retrievers](https://md.research.elamur.ai/free/en/papers/ni-2021-large-dual-encoders-generalizable-retrievers/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9844– 98…
10. OpenAI. 2025. GPT-5 mini: A faster, cost-efficient version of GPT-5. [https://platform.openai.com/](https://platform.openai.com/) docs/models/gpt-5-mini. Model snapshot: gpt-5-mini-2025-08-07. Accessed via Microsoft Azure OpenAI API. [↗](https://platform.openai.com/)
11. Andrew Piper, Richard Jean So, and David Bamman. 2021. Narrative theory for computational narrative understanding. In Proceedings of the 2021 conference on empirical methods in natural language processing, pages 298–311. [find](https://scholar.google.com/scholar?q=Andrew+Piper%2C+Richard+Jean+So%2C+and+David+Bamman%2E+2021%2E+Narrative+theory+for+computational+narrative+understanding%2E+In+Proceedings+of+the+2021+conference+on+empirical+methods+in+natural+language+proces)
12. Nils Reimers and Iryna Gurevych. 2019. [Sentence-bert: Sentence embeddings using siamese bert-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 3982–3992.
13. Abel Salinas and Fred Morstatter. 2024. The butterfly effect of altering prompts: How small changes and jailbreaks affect large language model performance. In Findings of the Association for Computational Linguistics: ACL 2024, pages 4629–4651. [find](https://scholar.google.com/scholar?q=Abel+Salinas+and+Fred+Morstatter%2E+2024%2E+The+butterfly+effect+of+altering+prompts%3A+How+small+changes+and+jailbreaks+affect+large+language+model+performance%2E+In+Findings+of+the+Association+for+Computati)
14. Igor Sterner, Alex Lascarides, and Frank Keller. 2026. [Contrastive learning with narrative twins for modeling story salience](https://md.research.elamur.ai/free/en/papers/sterner-2026-contrastive-learning-narrative-twins-modeling/). arXiv preprint arXiv:2601.07765.
15. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. [Text embeddings by weakly-supervised contrastive pre-training](https://md.research.elamur.ai/free/en/papers/wang-2022-text-embeddings-weakly-supervised-contrastive/). arXiv preprint arXiv:2212.03533.
16. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.
17. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. [Calibrate before use: Improving few-shot performance of language models](https://md.research.elamur.ai/free/en/papers/zhao-2021-calibrate-before-use-improving-few/). In International conference on machine learning, pages 12697–12706. Pmlr.
18. Louisa Lam and SY Suen. 1997. Application of majority voting to pattern recognition: an analysis of its behavior and performance. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 27(5):553–568. [find](https://scholar.google.com/scholar?q=Louisa+Lam+and+SY+Suen%2E+1997%2E+Application+of+majority+voting+to+pattern+recognition%3A+an+analysis+of+its+behavior+and+performance%2E+IEEE+Transactions+on+Systems%2C+Man%2C+and+Cybernetics%2DPart+A%3A+Systems+and)

*Updated 31 October 2026*