---
title: "MindFlayer at SemEval-2026 Task — paper by Tuli et al. (2026)"
description: "This paper presents DUALRAG, which the Mind Flayer team submitted to Subtask B of SemEval-2026 Task 8 (MTRAGEval). 17 references."
authors: ["Jerin Romijah Tuli", "Md. Sartaj Alam Pritom", "Talukder Naemul Hasan Naem"]
year: 2026
venue: "Proceedings of the 20th International Workshop on Semantic Evaluation (2026)"
source: "https://aclanthology.org/2026.semeval-1.293/"
doi: "10.18653/v1/2026.semeval-1.293"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/tuli-2026-mindflayer-semeval-2026-task-8/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tuli-2026-mindflayer-semeval-2026-task-8/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/36/) / MindFlayer at SemEval-2026 Task 8:DUALRAG:Answerability-Aware Generation for Multi-Turn RAG Conversations

# MindFlayer at SemEval-2026 Task 8:DUALRAG:Answerability-Aware Generation for Multi-Turn RAG Conversations

**Authors:** [Jerin Romijah Tuli](https://md.research.elamur.ai/free/en/authors/jerin-romijah-tuli/), [Md. Sartaj Alam Pritom](https://md.research.elamur.ai/free/en/authors/md-sartaj-alam-pritom/) and [Talukder Naemul Hasan Naem](https://md.research.elamur.ai/free/en/authors/talukder-naemul-hasan-naem/) · **Year:** 2026 · **Venue:** [Proceedings of the 20th International Workshop on Semantic Evaluation (2026)](https://md.research.elamur.ai/free/en/venues/semeval/) ([SemEval 2026](https://md.research.elamur.ai/free/en/venues/semeval/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.semeval-1.293/) · [PDF](https://aclanthology.org/2026.semeval-1.293.pdf) · [DOI 10.18653/v1/2026.semeval-1.293](https://doi.org/10.18653/v1/2026.semeval-1.293) · [OpenAlex](https://openalex.org/W7166881052)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
This paper presents DUALRAG, which the Mind Flayer team submitted to Subtask B of SemEval-2026 Task 8 (MTRAGEval). The task asks a system to write a grounded reply to the latest user turn of a multi-turn conversation, given the dialogue history and gold reference passages. Some turns cannot be answered from the corpus, and the IDK-conditioned metrics penalize both hallucinated answers and unjustified refusals.

The authors do no training. They run meta-llama/llama-4-scout-17b-16e-instruct through the Groq API in a zero-shot, prompt-only setup. A simple check for passage availability routes each task. Answerable cases get a prompt with the history, numbered passages and domain-specific hints at temperature 0.3. Cases without passages get a strict refusal prompt at temperature 0.1. A rule-based post-processing layer then fixes false refusals and false answers.

The system reached a harmonic mean of 0.7492, ranking 8th of 26 teams and beating the strongest baseline. It also refused every unanswerable test item correctly. This suggests that answerability routing alone can yield competitive, faithful multi-turn RAG without fine-tuning.

## Key points
- **Test data:** The MTRAG-based test set has 507 tasks (377 answerable, 130 unanswerable). They span four domains: Wikipedia/CLAPNQ, Finance/FiQA, Government and IBM Cloud. Mean turn depth is 7.64, and 91.7% of tasks are at turn 2 or later.
- **Scores:** HM = 0.7492, with RB_agg = 0.6024, RL_F = 0.8782 and RB_llm = 0.8297. This is +0.110 over the GPT-OSS-120B baseline (0.639) and about 0.033 below the top system.
- **Answerability handling:** Refusal accuracy was 100% (130/130) across all domains, and accuracy on answerable tasks was 99.7%. Post-processing altered 7 of 507 outputs (6 false refusals, 1 false answer).
- **Error patterns:** Manual inspection found three problems: answers drawing only on early passages, overly long IBM Cloud responses, and a vague fallback string for corrected refusals. Proposed fixes include re-prompting, query rewriting and fine-tuning on MTRAG.
- **Reproducibility:** Code is public on GitHub, and generating all 507 tasks takes about 22 minutes using two rotated API keys.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

## Cite

Cite as: Jerin Romijah Tuli, Md. Sartaj Alam Pritom and Talukder Naemul Hasan Naem. 2026. MindFlayer at SemEval-2026 Task 8:DUALRAG:Answerability-Aware Generation for Multi-Turn RAG Conversations. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{tuli2026mindflayer,
  title = {{MindFlayer at SemEval-2026 Task 8:DUALRAG:Answerability-Aware Generation for Multi-Turn RAG Conversations}},
  author = {Jerin Romijah Tuli, Md. Sartaj Alam Pritom and Talukder Naemul Hasan Naem},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.293},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.293}
}
```

- [← previous MindFlayer at SemEval-2026 Task 13:LACR-ENS: Calibration-Aware Ensemble Routing for Cross-Language AI-Generated Code Detection](https://md.research.elamur.ai/free/en/papers/tuli-2026-mindflayer-semeval-2026-task-13/)
- [next → MiniGPT: Rebuilding GPT from First Principles](https://md.research.elamur.ai/free/en/papers/joseph-2026-minigpt-rebuilding-gpt-first-principles/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/36/)

## References

17 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Vaibhav Adlakha, Parishad BehnamGhader, Xing Han Lu, Nicholas Meade, and Siva Reddy. 2024. [Evaluating correctness and faithfulness of instruction-following models for question answering](https://md.research.elamur.ai/free/en/papers/adlakha-2024-evaluating-correctness-faithfulness-instruction-following/). Transactions of the Association for Computational Linguistics, 12:681–699.
2. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. [Self-RAG: Learning to retrieve, generate, and critique through self-reflection](https://md.research.elamur.ai/free/en/papers/asai-2023-self-rag-learning-retrieve-generate/). In The Twelfth International Conference on Learning Representations (ICLR).
3. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. [QuAC: Question answering in context](https://md.research.elamur.ai/free/en/papers/choi-2018-quac-question-answering-context/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174–2184.
4. Nouha Dziri, Ehsan Kamalloo, Sivan Milton, Osmar Zaiane, Mo Yu, Edoardo M. Ponti, and Siva Reddy. 2022. [FaithDial: A faithful benchmark for information-seeking dialogue](https://md.research.elamur.ai/free/en/papers/dziri-2022-faithdial-faithful-benchmark-information-seeking/). Transactions of the Association for Computational Linguistics, 10:1473– 1490.
5. Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. [RAGAS: Automated evaluation of retrieval augmented generation](https://md.research.elamur.ai/free/en/papers/es-2023-ragas-automated-evaluation-retrieval-augmented/). In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 150–158.
6. Groq Inc. 2024. Groq inference API. [find](https://scholar.google.com/scholar?q=Groq+Inc%2E+2024%2E+Groq+inference+API%2E)
7. Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. [Active retrieval augmented generation](https://md.research.elamur.ai/free/en/papers/jiang-2023-active-retrieval-augmented-generation/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7969–7992.
8. Yannis Katsis, Sara Rosenthal, Kshitij Fadnis, Chulaka Gunasekara, Young-Suk Lee, Lucian Popa, Vraj Shah, Huaiyu Zhu, Danish Contractor, and Marina Danilevsky. 2025. MTRAG: A multi-turn conversational benchmark for evaluating retrieval-augmented generation systems. Transactions of the Association fo… [find](https://scholar.google.com/scholar?q=Yannis+Katsis%2C+Sara+Rosenthal%2C+Kshitij+Fadnis%2C+Chulaka+Gunasekara%2C+Young%2DSuk+Lee%2C+Lucian+Popa%2C+Vraj+Shah%2C+Huaiyu+Zhu%2C+Danish+Contractor%2C+and+Marina+Danilevsky%2E+2025%2E+MTRAG%3A+A+multi%2Dturn+conversational)
9. Tzu-Lin Kuo, Feng-Ting Liao, Mu-Wei Hsieh, Fu-Chieh Chang, Po-Chun Hsu, and Da-Shan Shiu. 2024. RAD-Bench: Evaluating large language models capabilities in retrieval augmented dialogues. Preprint, arXiv:2409.12558. [find](https://scholar.google.com/scholar?q=Tzu%2DLin+Kuo%2C+Feng%2DTing+Liao%2C+Mu%2DWei+Hsieh%2C+Fu%2DChieh+Chang%2C+Po%2DChun+Hsu%2C+and+Da%2DShan+Shiu%2E+2024%2E+RAD%2DBench%3A+Evaluating+large+language+models+capabilities+in+retrieval+augmented+dialogues%2E+Preprint%2C+arX)
10. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. [Retrieval-augmented generation for knowledge-intensive NLP tasks](https://md.research.elamur.ai/free/en/papers/lewis-2020-retrieval-augmented-generation-knowledge-intensive/). In Advances in Neural Information Pr…
11. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. [Know what you don’t know: Unanswerable questions for SQuAD](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789.
12. Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. [CoQA: A conversational question answering challenge](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/). Transactions of the Association for Computational Linguistics, 7:249–266.
13. Sara Rosenthal, Yannis Katsis, Vraj Shah, Lihong He, Lucian Popa, and Marina Danilevsky. 2026a. [MTRAG-UN: A benchmark for open challenges in multi-turn RAG conversations](https://md.research.elamur.ai/free/en/papers/rosenthal-2026-mtrag-un-benchmark-open-challenges/). Preprint, arXiv:2602.23184.
14. Sara Rosenthal, Vraj Shah, Yannis Katsis, and Marina Danilevsky. 2026b. [SemEval-2026 task 8: MTRAGEval: Evaluating multi-turn RAG conversations](https://md.research.elamur.ai/free/en/papers/rosenthal-2026-semeval-2026-task-8-mtrageval/). In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, California. Association for Computational Linguistics.
15. Maojia Song, Shang Hong Sim, Rishabh Bhardwaj, Hai Leong Chieu, Navonil Majumder, and Soujanya Poria. 2024. Measuring and enhancing trustworthiness of LLMs in RAG through grounded attributions and learning to refuse. Preprint, arXiv:2409.11242. — [arXiv:2409.11242](https://arxiv.org/abs/2409.11242) [find](https://scholar.google.com/scholar?q=Maojia+Song%2C+Shang+Hong+Sim%2C+Rishabh+Bhardwaj%2C+Hai+Leong+Chieu%2C+Navonil+Majumder%2C+and+Soujanya+Poria%2E+2024%2E+Measuring+and+enhancing+trustworthiness+of+LLMs+in+RAG+through+grounded+attributions+and+lea)
16. Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. 2024. [Making retrieval-augmented language models robust to irrelevant context](https://md.research.elamur.ai/free/en/papers/yoran-2023-making-retrieval-augmented-language-models/). In The Twelfth International Conference on Learning Representations (ICLR).
17. Hanning Zhang, Shizhe Diao, Yong Lin, Yi R. Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. 2024. R-tuning: Instructing large language models to say ‘I Don’t Know’. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Hanning+Zhang%2C+Shizhe+Diao%2C+Yong+Lin%2C+Yi+R%2E+Fung%2C+Qing+Lian%2C+Xingyao+Wang%2C+Yangyi+Chen%2C+Heng+Ji%2C+and+Tong+Zhang%2E+2024%2E+R%2Dtuning%3A+Instructing+large+language+models+to+say+%E2%80%98I+Don%E2%80%99t+Know%E2%80%99%2E+In+Proceedings)

*Updated 1 November 2026*