--- title: "IITKanBDone at SemEval-2026 Task 8: ELSER-Based Sparse" description: "The paper describes a team's entry to SemEval-2026 Task 8 (MT-RAG), a benchmark for… 8 references, cited by 1 paper here." authors: ["Soumendra Ray", "Garima Gupta"] year: 2026 venue: "Proceedings of the 20th International Workshop on Semantic Evaluation (2026)" source: "https://aclanthology.org/2026.semeval-1.145/" doi: "10.18653/v1/2026.semeval-1.145" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/ray-2026-iitkanbdone-semeval-2026-task-8/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ray-2026-iitkanbdone-semeval-2026-task-8/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/13/) / IITKanBDone at SemEval-2026 Task 8: ELSER-Based Sparse Retrieval with Mistral-7B for Multi-Turn RAG Evaluation # IITKanBDone at SemEval-2026 Task 8: ELSER-Based Sparse Retrieval with Mistral-7B for Multi-Turn RAG Evaluation **Authors:** Soumendra Ray and Garima Gupta · **Year:** 2026 · **Venue:** [Proceedings of the 20th International Workshop on Semantic Evaluation (2026)](https://md.research.elamur.ai/free/en/venues/semeval/) ([SemEval 2026](https://md.research.elamur.ai/free/en/venues/semeval/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.semeval-1.145/) · [PDF](https://aclanthology.org/2026.semeval-1.145.pdf) · [DOI 10.18653/v1/2026.semeval-1.145](https://doi.org/10.18653/v1/2026.semeval-1.145) · [OpenAlex](https://openalex.org/W7166861146) **Citations:** 0 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/ray-2026-iitkanbdone-semeval-2026-task-8/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary The paper describes a team's entry to SemEval-2026 Task 8 (MT-RAG), a benchmark for retrieval-augmented question answering in multi-turn conversations. The team took part in all three subtasks: passage retrieval (Task A), answer generation from gold passages (Task B), and a full retrieve-then-generate pipeline (Task C). For retrieval, they compared BM25, BGE-base-1.5 dense embeddings, a BM25+BGE hybrid and ELSER v2 running in Elasticsearch 8.10. They chose ELSER and queried with only the latest user turn. Generation used Mistral-7B-Instruct-v0.2 zero-shot through HuggingFace Transformers. The prompt held the dialogue history, ID-tagged passages and instructions against hallucination. Retrieval and end-to-end results landed in the lower part of the leaderboard. The generation run, however, placed 13th of 26 and beat the much larger gpt-oss-120b baseline. The sharp drop from Task B to Task C, along with case studies of coreference failures, shows that conversation-aware query rewriting is the main missing component. Overall, the work suggests a small open model paired with learned sparse retrieval is a cheap and reasonable baseline for conversational RAG. ## Key points - Official scores: Task A nDCG@5 0.4018 (rank 24/38; top 0.5776); Task B harmonic mean 0.6976 (rank 13/26; top 0.7827); Task C 0.4289 (rank 23/29; top 0.5861). - Task B components: R_Bagg 0.5753, R_LF 0.7865, R_Bllm 0.7746. It beat the gpt-oss-120b baseline (0.639). Mixtral-8x7B was barred because it was used to build the dataset. - Development comparison (R@10 / nDCG@10): BM25 0.263/0.205, BGE 0.385/0.308, ELSER 0.544/0.450. ELSER reached about 94% of the performance reported in the reference paper. - Settings: top-10 documents for Task A and top-5 passages as context for Task C, with temperature 0.3 and at most 512 tokens. The paper also mentions 0.7 for Task C, which is an internal inconsistency. No fine-tuning was done. Code is at github.com/guptagarima2/mtrag. - Error analysis: the dominant failures were missing coreference context in single-turn queries and unsupported synthesis or entity switches in generation. A query-rewriting baseline reached 0.4795 nDCG@5. Note that the paper calls the corpus UN documents, but its case examples (U2, Anggun, SAP Center) suggest more general-domain data. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Indian Institute of Technology Kanpur](https://md.research.elamur.ai/free/en/universities/indian-institute-of-technology-kanpur/) — University, Kanpur, India · 14 works · 619 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works 🧭 Topics (1): [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 473 works ## Cite Cite as: Soumendra Ray and Garima Gupta. 2026. IITKanBDone at SemEval-2026 Task 8: ELSER-Based Sparse Retrieval with Mistral-7B for Multi-Turn RAG Evaluation. Proceedings of the 20th International Workshop on Semantic Evaluation (2026). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{ray2026iitkanbdone, title = {{IITKanBDone at SemEval-2026 Task 8: ELSER-Based Sparse Retrieval with Mistral-7B for Multi-Turn RAG Evaluation}}, author = {Soumendra Ray and Garima Gupta}, booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)}, year = {2026}, doi = {10.18653/v1/2026.semeval-1.145}, url = {https://doi.org/10.18653/v1/2026.semeval-1.145} } ``` - [← previous IIMAS-RAG at SemEval-2026 Task 8: Hybrid Sparse-Dense Retrieval and Answerability-Conditioned Generation for Multi-Turn RAG](https://md.research.elamur.ai/free/en/papers/rios-2026-iimas-rag-semeval-2026-task/) - [next → Image Corruption-Inspired Membership Inference Attacks against Large Vision-Language Models](https://md.research.elamur.ai/free/en/papers/wu-2026-image-corruption-inspired-membership-inference/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/13/) ## References 8 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Elastic. 2023. Elastic learned sparse encoder (elser). Accessed: 2024. [find](https://scholar.google.com/scholar?q=Elastic%2E+2023%2E+Elastic+learned+sparse+encoder+%28elser%29%2E+Accessed%3A+2024%2E) 2. Kshitij P Fadnis, Siva Sankalp Patel, Odellia Boni, Yannis Katsis, Sara Rosenthal, Benjamin Sznajder, and Marina Danilevsky. 2025. InspectorRAGet: An introspection platform for RAG evaluation. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computa… [find](https://scholar.google.com/scholar?q=Kshitij+P+Fadnis%2C+Siva+Sankalp+Patel%2C+Odellia+Boni%2C+Yannis+Katsis%2C+Sara+Rosenthal%2C+Benjamin+Sznajder%2C+and+Marina+Danilevsky%2E+2025%2E+InspectorRAGet%3A+An+introspection+platform+for+RAG+evaluation%2E+In+Proc) 3. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, and 1 others. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+and+1+others%2E+2023%2E+Mistral+7b%2E+arXiv+preprint+arXiv%3A2310%2E06825%2E) 4. Yannis Katsis, Sara Rosenthal, Kshitij Fadnis, Chulaka Gunasekara, Young-Suk Lee, Lucian Popa, Vraj Shah, Huaiyu Zhu, Danish Contractor, and Marina Danilevsky. 2025. Mtrag: A multi-turn conversational benchmark for evaluating retrieval-augmented generation systems. Transactions of the Association fo… [find](https://scholar.google.com/scholar?q=Yannis+Katsis%2C+Sara+Rosenthal%2C+Kshitij+Fadnis%2C+Chulaka+Gunasekara%2C+Young%2DSuk+Lee%2C+Lucian+Popa%2C+Vraj+Shah%2C+Huaiyu+Zhu%2C+Danish+Contractor%2C+and+Marina+Danilevsky%2E+2025%2E+Mtrag%3A+A+multi%2Dturn+conversational) 5. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Bouza, Alexandros Konstantinidis, Zhengzhong Ren, Kevin Murray, Kevin Murphy, Walid Soubra, Jeremy Devers, Younes Belkada, Aman Acharya… — [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://md.research.elamur.ai/free/en/papers/lewis-2020-retrieval-augmented-generation-knowledge-intensive/) 6. Sara Rosenthal, Yannis Katsis, Vraj Shah, Lihong He, Lucian Popa, and Marina Danilevsky. 2026a. [Mtrag-un: A benchmark for open challenges in multi-turn rag conversations](https://md.research.elamur.ai/free/en/papers/rosenthal-2026-mtrag-un-benchmark-open-challenges/). Preprint, arXiv:2602.23184. 7. Sara Rosenthal, Vraj Shah, Yannis Katsis, and Marina Danilevsky. 2026b. [Semeval-2026 task 8: Mtrageval: Evaluating multi-turn rag conversations](https://md.research.elamur.ai/free/en/papers/rosenthal-2026-semeval-2026-task-8-mtrageval/). In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, California. Association for Computational Linguistics. 8. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R ’emi Louf, Morgan Funtowicz, and 1 others. 2020. [Transformers: State-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). In Proceedings of the 2020 Conference on Empirical Methods in Natural L… *Updated 31 October 2026*