--- title: "MT-Bench-101 — Bai 2024" description: "Existing LLM benchmarks mostly test single-turn instructions." authors: ["Ge Bai", "Jie Liu", "Xingyuan Bu", "Yancheng He", "Jiaheng Liu", "Zhanhui Zhou", "Zhuoran Lin", "Wenbo Su", "Tiezheng Ge", "Bo Zheng", "Wanli Ouyang"] year: 2024 venue: "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)" source: "https://aclanthology.org/2024.acl-long.401/" doi: "10.18653/v1/2024.acl-long.401" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/bai-2024-mt-bench-101-fine-grained/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bai-2024-mt-bench-101-fine-grained/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues # MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues **Authors:** Ge Bai, [Jie Liu](https://md.research.elamur.ai/free/en/authors/jie-liu/), Xingyuan Bu, Yancheng He, [Jiaheng Liu](https://md.research.elamur.ai/free/en/authors/jiaheng-liu/), Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, [Bo Zheng](https://md.research.elamur.ai/free/en/authors/bo-zheng/), Wanli Ouyang · **Year:** 2024 · **Venue:** Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2024.acl-long.401/) · [PDF](https://aclanthology.org/2024.acl-long.401.pdf) · [DOI 10.18653/v1/2024.acl-long.401](https://doi.org/10.18653/v1/2024.acl-long.401) · [OpenAlex](https://openalex.org/W4402671584) **Citations:** 60 citations (openalex-snapshot · 2026-10-14) · [111](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Existing LLM benchmarks mostly test single-turn instructions. Multi-turn ones such as MT-Bench cover only two turns and broad skill categories, so they miss much of what happens in real chatbot conversations. MT-Bench-101 fills this gap with a fine-grained multi-turn benchmark. The authors studied real user–chatbot logs from ShareGPT and RealChat alongside teaching taxonomies from educational psychology. From this, they built a three-level ability hierarchy: - three top-level abilities: perceptivity, adaptability and interactivity; - seven mid-level abilities; - 13 concrete tasks. They used GPT-4 with task-specific prompts to generate the dialogues. For scoring, GPT-4 acts as the judge, applying per-task guidelines. Models answer with golden context as dialogue history, and each dialogue receives its weakest turn's score. With this setup, 21 LLMs were evaluated. GPT-4 performed best overall. Adaptability and interactivity emerged as the main weaknesses across models. Larger models scored higher, but alignment methods such as RLHF and chat-oriented designs gave no clear multi-turn gains. The benchmark thus offers a validated, diagnostic tool for analysing conversational LLM behaviour. ## Key points - The dataset contains 4208 turns in 1388 multi-turn dialogues across 13 tasks. Examples include anaphora resolution, topic shift, self-correction, self-affirmation and proactive interaction. - 21 models were tested: 2 closed-source and 19 open-source. - GPT-4's judgments matched human experts 87% of the time, higher than the 80% agreement among the experts themselves. Removing the scoring guidelines lowered agreement to 77%. Averaging turn scores instead of taking the minimum lowered it to 75%. - Feeding models their own earlier answers as history let errors pile up over turns, while golden context raised scores. This motivated the choice of golden context in the evaluation protocol. - Data and code are released at github.com/mtbench101/mt-bench-101. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 111 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 107 not yet. - [Sacred or Synthetic? Evaluating LLM Reliability and Abstention for Religious Questions](https://md.research.elamur.ai/free/en/papers/atif-2025-sacred-synthetic-evaluating-llm-reliability/) - [Beyond MCQ: An Open-Ended Arabic Cultural QA Benchmark with Dialect Variants](https://md.research.elamur.ai/free/en/papers/bhatti-2026-beyond-mcq-open-ended-arabic/) - [XQ-MEval: A Dataset with Cross-lingual Parallel Quality for Benchmarking Translation Metrics](https://md.research.elamur.ai/free/en/papers/liu-2026-xq-meval-dataset-cross-lingual/) - [MT-Video-Bench: A Holistic Video Understanding Benchmark for Evaluating Multimodal LLMs in Multi-Turn Dialogues](https://md.research.elamur.ai/free/en/papers/pan-2026-mt-video-bench-holistic-video/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{bai2024mt, title = {{MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues}}, author = {Bai, Ge and Liu, Jie and Bu, Xingyuan and He, Yancheng and Liu, Jiaheng and Zhou, Zhanhui and Lin, Zhuoran and Su, Wenbo and Ge, Tiezheng and Zheng, Bo and Ouyang, Wanli}, booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, year = {2024}, doi = {10.18653/v1/2024.acl-long.401}, url = {https://doi.org/10.18653/v1/2024.acl-long.401} } ``` - [← previous MSCAW-coref: Multilingual, Singleton and Conjunction-Aware Word-Level Coreference Resolution](https://md.research.elamur.ai/free/en/papers/liu-2024-mscaw-coref-multilingual-singleton-conjunction/) - [next → MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models](https://md.research.elamur.ai/free/en/papers/kwan-2024-mt-eval-multi-turn-capabilities/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) ## References 1 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Robin Alexander. 2018. Developing dialogic teaching: Genesis, process, trial. Research papers in education, 33(5):561–598. [↗](https://doi.org/10.1080/02671522.2018.1481140)