--- title: "DUTIR at SemEval-2026 Task 8 — paper by Yang et al. (2026)" description: "The paper describes the DUTIR Lab (Dalian University of Technology) entry for Task C of SemEval-2026… 7 references, cited by 1 paper here." authors: ["Jin Yang", "Yichong Chen", "Liang Yang"] year: 2026 venue: "Proceedings of the 20th International Workshop on Semantic Evaluation (2026)" source: "https://aclanthology.org/2026.semeval-1.48/" doi: "10.18653/v1/2026.semeval-1.48" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/yang-2026-dutir-semeval-2026-task-8/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yang-2026-dutir-semeval-2026-task-8/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/12/) / DUTIR at SemEval-2026 Task 8: A Hybrid Retrieval and Faithfulness-Guarded Framework for Multi-Turn RAG # DUTIR at SemEval-2026 Task 8: A Hybrid Retrieval and Faithfulness-Guarded Framework for Multi-Turn RAG **Authors:** Jin Yang, Yichong Chen and [Liang Yang](https://md.research.elamur.ai/free/en/authors/liang-yang/) · **Year:** 2026 · **Venue:** [Proceedings of the 20th International Workshop on Semantic Evaluation (2026)](https://md.research.elamur.ai/free/en/venues/semeval/) ([SemEval 2026](https://md.research.elamur.ai/free/en/venues/semeval/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.semeval-1.48/) · [PDF](https://aclanthology.org/2026.semeval-1.48.pdf) · [DOI 10.18653/v1/2026.semeval-1.48](https://doi.org/10.18653/v1/2026.semeval-1.48) · [OpenAlex](https://openalex.org/W7166867393) **Citations:** 0 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/yang-2026-dutir-semeval-2026-task-8/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary The paper describes the DUTIR Lab (Dalian University of Technology) entry for Task C of SemEval-2026 Task 8 (MTRAGEval). Task C is end-to-end conversational RAG in English on the MTRAG-UN benchmark. Multi-turn questions often contain pronouns, ellipsis and topic shifts. They also use casual wording that differs from formal corpora in finance (FiQA), government (Govt), cloud documentation and Wikipedia (ClapNQ). The authors chain four stages: - GPT-5.2 rewrites each turn into a self-contained query. - Dense BGE-M3 and sparse BM25 results are merged with Reciprocal Rank Fusion, then rescored by bge-reranker-large. - A zero-shot answerability classifier outputs a confidence score with a threshold. - A second LLM pass removes unsupported claims and adds hedging to partial answers. On the blind test set the system reached a Composite Score of 0.5576, placing 4th of 29 teams. It was strongest on faithfulness. Error analysis shows the cost of strict gating: the system sometimes refuses partially answerable questions. Parametric knowledge also still leaks through on open-ended requests. This makes the work a useful reference for balancing grounding against helpfulness in dialogue RAG. ## Key points - **Leaderboard results:** - Composite 0.5576 (harmonic mean of RB_agg 0.4073, RL_F 0.7123, RB_llm 0.6575). - Top-1 system: 0.5861. - qwen-30b-a3b-thinking baseline: 0.5366. - **Retrieval setup:** - Dense and BM25 top-k of 200 each, fused with RRF (k = 60), then reranked by a cross-encoder. - Five passages are kept, compressed to about 1,400 characters. - Built with FAISS and PyTorch. - The paper reports both 120 and 80 as the rerank cutoff. - **Gating threshold:** - τ = 0.85, tuned on the dev set over the range 0.70–0.95. - UNANSWERABLE predictions below τ are downgraded to PARTIAL. - Gating and guarding prompts use 3-shot examples. - **Underspecified queries:** they made up about 28.4% of the test set, and the system's IDK rate was 0.26. - **Score gap:** the low RB_agg relative to RL_F is attributed to GPT-5.2's abstractive style versus extractive references. - **Author names:** the PDF lists the first author as Ruiyang Jin, which differs from the library metadata ("Jin Yang"). *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Dalian University of Technology](https://md.research.elamur.ai/free/en/universities/dalian-university-of-technology/) — University, Dalian, China · 12 works · 46 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works 🧭 Topics (1): [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 473 works ## Cite Cite as: Jin Yang, Yichong Chen and Liang Yang. 2026. DUTIR at SemEval-2026 Task 8: A Hybrid Retrieval and Faithfulness-Guarded Framework for Multi-Turn RAG. Proceedings of the 20th International Workshop on Semantic Evaluation (2026). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{yang2026dutir, title = {{DUTIR at SemEval-2026 Task 8: A Hybrid Retrieval and Faithfulness-Guarded Framework for Multi-Turn RAG}}, author = {Jin Yang, Yichong Chen and Liang Yang}, booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)}, year = {2026}, doi = {10.18653/v1/2026.semeval-1.48}, url = {https://doi.org/10.18653/v1/2026.semeval-1.48} } ``` - [← previous DuoNova@LTEDI 2026: Multilingual Span Detection and Counter-Narrative Generation on Homophobic and Transphobic Comments](https://md.research.elamur.ai/free/en/papers/s-2026-duonova-ltedi-2026-multilingual-span/) - [next → Easy Samples Are All You Need: Self-Evolving LLMs via Data-Efficient Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/yu-2026-easy-samples-are-all-you/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/12/) ## References 7 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Jianlv Chen, Shitao Wang, Jianlin Liu, Chen Gao, Zheng Liu, and 1 others. 2024. [BGE M3-Embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation](https://md.research.elamur.ai/free/en/papers/chen-2024-m3-embedding-multi-linguality/). arXiv preprint arXiv:2402.03216. 2. Error Analysis Cases Table 2 provides concrete examples of the failure modes discussed in Section 5.4. These cases highlight the trade-offs between rigorous faithfulness gating and helpfulness in complex multi-turn contexts. *Table 2: System failure modes identified during post-evaluation.* Type 1:… [find](https://scholar.google.com/scholar?q=Error+Analysis+Cases+Table+2+provides+concrete+examples+of+the+failure+modes+discussed+in+Section+5%2E4%2E+These+cases+highlight+the+trade%2Doffs+between+rigorous+faithfulness+gating+and+helpfulness+in+comp) 3. Gordon V Cormack, Charles LA Clarke, and Stefan Buettcher. 2009. [Reciprocal rank fusion outperforms condorcet and individual rank learning methods](https://md.research.elamur.ai/free/en/papers/cormack-2009-reciprocal-rank-fusion-outperforms/). In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, pages 758–759. 4. Query: "Which one in the US trades options?" Context: Mentions options trade on dedicated exchanges, but specific names were truncated. Output: "The specific names... are not provided..." Impact: Triggered full IDK rejection instead of a partial answer. IDK: 1.00 [find](https://scholar.google.com/scholar?q=Query%3A+%22Which+one+in+the+US+trades+options%3F%22+Context%3A+Mentions+options+trade+on+dedicated+exchanges%2C+but+specific+names+were+truncated%2E+Output%3A+%22The+specific+names%2E%2E%2E+are+not+provided%2E%2E%2E%22+Impact%3A+Trig) 5. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. [Billion-scale similarity search with GPUs](https://md.research.elamur.ai/free/en/papers/johnson-2017-billion-scale-similarity-search-gpus/). IEEE Transactions on Big Data, 7(3):535–547. 6. Type 2: Parametric Hallucination (Low Faithfulness) [find](https://scholar.google.com/scholar?q=Type+2%3A+Parametric+Hallucination+%28Low+Faithfulness%29) 7. Yannis Katsis, Sara Rosenthal, Kshitij Fadnis, Chulaka Gunasekara, Young-Suk Lee, Lucian Popa, Vraj Shah, Huaiyu Zhu, Danish Contractor, and Marina Danilevsky. 2025. Mtrag: A multi-turn conversational benchmark for evaluating retrieval-augmented Query: "Any other terminology you think I should know?… [find](https://scholar.google.com/scholar?q=Yannis+Katsis%2C+Sara+Rosenthal%2C+Kshitij+Fadnis%2C+Chulaka+Gunasekara%2C+Young%2DSuk+Lee%2C+Lucian+Popa%2C+Vraj+Shah%2C+Huaiyu+Zhu%2C+Danish+Contractor%2C+and+Marina+Danilevsky%2E+2025%2E+Mtrag%3A+A+multi%2Dturn+conversational) *Updated 31 October 2026*