--- title: "uircis at SemEval-2026 Task 8 — paper by Zhang et al. (2026)" description: "This work presents a team's entry to SemEval-2026 Task 8 (MTRAGEval), which evaluates retrieval-augmented generation in… 15 references." authors: ["Jiaqi Zhang", "Wenbin Duan", "Yingqi Zhang", "Yan Li", "Binyang Li"] year: 2026 venue: "Proceedings of the 20th International Workshop on Semantic Evaluation (2026)" source: "https://aclanthology.org/2026.semeval-1.394/" doi: "10.18653/v1/2026.semeval-1.394" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/zhang-2026-uircis-semeval-2026-task-8/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2026-uircis-semeval-2026-task-8/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/54/) / uircis at SemEval-2026 Task 8: A Unified Lightweight Pipeline for Multi-Turn RAG Evaluation # uircis at SemEval-2026 Task 8: A Unified Lightweight Pipeline for Multi-Turn RAG Evaluation **Authors:** Jiaqi Zhang, Wenbin Duan, Yingqi Zhang, [Yan Li](https://md.research.elamur.ai/free/en/authors/yan-li/) and [Binyang Li](https://md.research.elamur.ai/free/en/authors/binyang-li/) · **Year:** 2026 · **Venue:** [Proceedings of the 20th International Workshop on Semantic Evaluation (2026)](https://md.research.elamur.ai/free/en/venues/semeval/) ([SemEval 2026](https://md.research.elamur.ai/free/en/venues/semeval/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.semeval-1.394/) · [PDF](https://aclanthology.org/2026.semeval-1.394.pdf) · [DOI 10.18653/v1/2026.semeval-1.394](https://doi.org/10.18653/v1/2026.semeval-1.394) · [OpenAlex](https://openalex.org/W7166796110) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary This work presents a team's entry to SemEval-2026 Task 8 (MTRAGEval), which evaluates retrieval-augmented generation in multi-turn English dialogues over four domain collections: FIQA, CLOUD, CLAPNQ and GOVT. The team covered Subtask A (passage retrieval) and Subtask B (grounded answer generation). Their goal was a reproducible, low-cost design that uses only open-weight models on one GPU and no proprietary APIs. Qwen2.5-7B-Instruct first rewrites each user turn into a self-contained query. BGE-M3 then retrieves dense candidates, and BGE-Reranker-v2-M3 reorders them. The same Qwen model writes answers from the reference documents supplied by the organizers. It is instructed to stay strictly grounded and to say "I don't know" when the evidence is insufficient. The system scored nDCG@5 of 0.4422 on retrieval and a harmonic mean of 0.5664 on generation. Ablations show that reranking gives the most reliable improvement. Rewriting helps in some domains and hurts in others. Error analysis traces most generation failures back to evidence selection rather than hallucination. The work is a transparent, replicable baseline for compute-limited multi-turn RAG. ## Key points - Official ranks: 19/38 in Subtask A (nDCG@5 0.4422) and 21/26 in Subtask B (harmonic mean 0.5664, which combines RB_agg, RL_F and RB_llm). - Pipeline details: - BGE-M3 retrieves 100 candidates using an instruction-prefixed query. - The cross-encoder reranker keeps the top 10. - Reference documents are truncated to 1200 characters for generation. - Everything runs on a 20GB slice of one NVIDIA A800. - Local ablation (overall nDCG@5): - Dense-only: 0.2705 - Dense + reranking: 0.4210 - Rewriting + dense: 0.3374 - Full system: 0.4466 - Rewriting is domain-sensitive: - It lifts CLAPNQ from 0.3339 to 0.4730 (rewrite-only vs. dense-only). - It cuts FIQA from 0.2076 to 0.1399, probably because rewrites distort financial entities and numerical constraints. - Error analysis identifies two main failure modes: - Over-aggregation of loosely related multi-document evidence into verbose answers. - Correct "I don't know" abstentions caused by retrieved passages that are too generic. - The authors propose tighter evidence filtering and answer-length control as remedies. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of International Relations](https://md.research.elamur.ai/free/en/universities/university-of-international-relations/) — University, Beijing, China · 4 works · 1 citation 📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works ## Cite Cite as: Jiaqi Zhang, Wenbin Duan, Yingqi Zhang, Yan Li and Binyang Li. 2026. uircis at SemEval-2026 Task 8: A Unified Lightweight Pipeline for Multi-Turn RAG Evaluation. Proceedings of the 20th International Workshop on Semantic Evaluation (2026). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{zhang2026uircis, title = {{uircis at SemEval-2026 Task 8: A Unified Lightweight Pipeline for Multi-Turn RAG Evaluation}}, author = {Jiaqi Zhang, Wenbin Duan, Yingqi Zhang, Yan Li and Binyang Li}, booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)}, year = {2026}, doi = {10.18653/v1/2026.semeval-1.394}, url = {https://doi.org/10.18653/v1/2026.semeval-1.394} } ``` - [← previous UCSC NLP at SemEval-2026 Task 10: Boundary-Aware Span Extraction and RoBERTa Classification for Conspiracy Detection](https://md.research.elamur.ai/free/en/papers/marhoefer-2026-ucsc-nlp-semeval-2026-task/) - [next → Ultra-Low-Dimensional Prompt Tuning via Random Projection](https://md.research.elamur.ai/free/en/papers/wu-2026-ultra-low-dimensional-prompt-tuning/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/54/) ## References 15 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. [M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation](https://md.research.elamur.ai/free/en/papers/chen-2024-m3-embedding-multi-linguality/). In Findings of the association for computational linguistics: ACL 2024, pages 2318–2335. 2. Jeff Dalton et al. 2020. TREC Conversational Assistance Track (CAST). In Proceedings of the 29th Text REtrieval Conference (TREC). [find](https://scholar.google.com/scholar?q=Jeff+Dalton+et+al%2E+2020%2E+TREC+Conversational+Assistance+Track+%28CAST%29%2E+In+Proceedings+of+the+29th+Text+REtrieval+Conference+%28TREC%29%2E) 3. Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. [Ragas: Automated evaluation of retrieval augmented generation](https://md.research.elamur.ai/free/en/papers/es-2023-ragas-automated-evaluation-retrieval-augmented/). In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 150–158. 4. Kshitij Fadnis et al. 2025. An introspection platform for RAG evaluation. In Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Demo Track). [find](https://scholar.google.com/scholar?q=Kshitij+Fadnis+et+al%2E+2025%2E+An+introspection+platform+for+RAG+evaluation%2E+In+Proceedings+of+the+2025+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics%3A+Human+La) 5. FlagOpen. 2024. Flagembedding: Retrieval and reranking models (bge family). [https://github.com/](https://github.com/) FlagOpen/FlagEmbedding. [↗](https://github.com/) 6. Vladimir Karpukhin, Barlas Oguz, Sewon Min, et al. 2020. [Dense passage retrieval for open-domain question answering](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 7. Yannis Katsis, Sara Rosenthal, Kshitij Fadnis, Chulaka Gunasekara, Young-Suk Lee, Lucian Popa, Vraj Shah, Huaiyu Zhu, Danish Contractor, and Marina Danilevsky. 2025. Mtrag: A multi-turn conversational benchmark for evaluating retrieval-augmented generation systems. Transactions of the Association fo… [find](https://scholar.google.com/scholar?q=Yannis+Katsis%2C+Sara+Rosenthal%2C+Kshitij+Fadnis%2C+Chulaka+Gunasekara%2C+Young%2DSuk+Lee%2C+Lucian+Popa%2C+Vraj+Shah%2C+Huaiyu+Zhu%2C+Danish+Contractor%2C+and+Marina+Danilevsky%2E+2025%2E+Mtrag%3A+A+multi%2Dturn+conversational) 8. Patrick Lewis, Ethan Perez, Aleksandra Piktus, et al. 2020. [Retrieval-augmented generation for knowledge-intensive NLP tasks](https://md.research.elamur.ai/free/en/papers/lewis-2020-retrieval-augmented-generation-knowledge-intensive/). In Advances in Neural Information Processing Systems. 9. Jimmy Lin et al. 2021. Few-shot conversational dense retrieval. In Proceedings of the 30th ACM International Conference on Information and Knowledge Management. [find](https://scholar.google.com/scholar?q=Jimmy+Lin+et+al%2E+2021%2E+Few%2Dshot+conversational+dense+retrieval%2E+In+Proceedings+of+the+30th+ACM+International+Conference+on+Information+and+Knowledge+Management%2E) 10. Tri Nguyen, Mir Rosenberg, Xia Song, et al. 2016. [MS MARCO: A human generated machine reading comprehension dataset](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/). In Proceedings of the Workshop on Cognitive Computation: Integrating Neural and Symbolic Approaches. 11. Rodrigo Nogueira and Kyunghyun Cho. 2019. [Passage re-ranking with BERT](https://md.research.elamur.ai/free/en/papers/nogueira-2019-passage-re-ranking-bert/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 12. Sara Rosenthal, Yannis Katsis, Vraj Shah, Lihong He, Lucian Popa, and Marina Danilevsky. 2026a. [MTRAG-UN: A benchmark for open challenges in multi-turn RAG conversations](https://md.research.elamur.ai/free/en/papers/rosenthal-2026-mtrag-un-benchmark-open-challenges/). arXiv preprint arXiv:2602.23184. 13. Sara Rosenthal, Vraj Shah, Yannis Katsis, and Marina Danilevsky. 2026b. [SemEval-2026 task 8: MTRAGEval: Evaluating multi-turn RAG conversations](https://md.research.elamur.ai/free/en/papers/rosenthal-2026-semeval-2026-task-8-mtrageval/). In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, California. Association for Computational Linguistics. 14. James Thorne et al. 2018. The FEVER shared task: System evaluation of fact extraction and verification. In Proceedings of the First Workshop on Fact Extraction and Verification (FEVER). [find](https://scholar.google.com/scholar?q=James+Thorne+et+al%2E+2018%2E+The+FEVER+shared+task%3A+System+evaluation+of+fact+extraction+and+verification%2E+In+Proceedings+of+the+First+Workshop+on+Fact+Extraction+and+Verification+%28FEVER%29%2E) 15. Nikos Voskarides, Dan Li, Pengjie Ren, Evangelos Kanoulas, and Maarten de Rijke. 2020. Query resolution for conversational search with limited supervision. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. [find](https://scholar.google.com/scholar?q=Nikos+Voskarides%2C+Dan+Li%2C+Pengjie+Ren%2C+Evangelos+Kanoulas%2C+and+Maarten+de+Rijke%2E+2020%2E+Query+resolution+for+conversational+search+with+limited+supervision%2E+In+Proceedings+of+the+43rd+International+ACM) *Updated 2 November 2026*