--- title: "MdEval: Massively Multilingual Code Debugging (2026)" description: "The bug taxonomy includes generic errors and language-specific ones, such as a missing mut in Rust… 5 references, cited by 11 papers here." authors: ["Shukai Liu", "Linzheng Chai", "Jian Yang", "Jiajun Shi", "He Zhu", "Liran Wang", "Jin Ke", "Wei Zhang", "Hualei Zhu", "Shuyue Guo", "Tao Sun", "Jiaheng Liu", "Yunlong Duan", "Yu Hao", "Liqun Yang", "Guanglin Niu", "Ge Zhang", "Zhoujun Li"] year: 2026 venue: "Findings of the Association for Computational Linguistics: ACL 2026" source: "https://aclanthology.org/2026.findings-acl.1041/" doi: "10.18653/v1/2026.findings-acl.1041" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/liu-2026-mdeval-massively-multilingual-code-debugging/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2026-mdeval-massively-multilingual-code-debugging/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/) / MdEval: Massively Multilingual Code Debugging # MdEval: Massively Multilingual Code Debugging **Authors:** [Shukai Liu](https://md.research.elamur.ai/free/en/authors/shukai-liu/), [Linzheng Chai](https://md.research.elamur.ai/free/en/authors/linzheng-chai/), [Jian Yang](https://md.research.elamur.ai/free/en/authors/jian-yang/), [Jiajun Shi](https://md.research.elamur.ai/free/en/authors/jiajun-shi/), [He Zhu](https://md.research.elamur.ai/free/en/authors/he-zhu/), Liran Wang, [Jin Ke](https://md.research.elamur.ai/free/en/authors/jin-ke/), [Wei Zhang](https://md.research.elamur.ai/free/en/authors/wei-zhang/), [Hualei Zhu](https://md.research.elamur.ai/free/en/authors/hualei-zhu/), [Shuyue Guo](https://md.research.elamur.ai/free/en/authors/shuyue-guo/), [Tao Sun](https://md.research.elamur.ai/free/en/authors/tao-sun/), [Jiaheng Liu](https://md.research.elamur.ai/free/en/authors/jiaheng-liu/), Yunlong Duan, Yu Hao, Liqun Yang, Guanglin Niu, [Ge Zhang](https://md.research.elamur.ai/free/en/authors/ge-zhang/) and [Zhoujun Li](https://md.research.elamur.ai/free/en/authors/zhoujun-li/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1041/) · [PDF](https://aclanthology.org/2026.findings-acl.1041.pdf) · [DOI 10.18653/v1/2026.findings-acl.1041](https://doi.org/10.18653/v1/2026.findings-acl.1041) · [OpenAlex](https://openalex.org/W4404355202) **Citations:** 1 citations (OpenAlex, 2026-10-14) · [11](https://md.research.elamur.ai/free/en/papers/liu-2026-mdeval-massively-multilingual-code-debugging/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Most existing code-debugging benchmarks for LLMs, such as DebugBench and DebugEval, cover only one to three languages and are mostly Python-centred. They also ignore bugs that only occur in particular languages. MdEval addresses this gap with a human-annotated benchmark of about 3.9K problems in 20 programming languages. Its three tasks are bug identification (BI), bug localization (BL), and automated program repair (APR). The bug taxonomy includes generic errors and language-specific ones, such as a missing `mut` in Rust or a misused macro definition in C. The authors also built MdEval-Instruct, an instruction-tuning corpus produced by xDebugGen. xDebugGen injects bugs into LLM-generated questions and solutions, or creates them through round-trip code translation. Models fine-tuned on this corpus form the xDebugCoder baseline. The authors tested 40 models. Closed-source systems (GPT, Claude series) clearly outperform open-source ones such as DeepSeekCoder and Qwen-Coder. Fine-tuning base models on MdEval-Instruct improved APR results to roughly the level of the official instruct versions, sometimes higher. The resource enables realistic, multilingual and multitask evaluation of debugging ability. ## Key points - **Benchmark size:** MdEval contains 3,897 problems, split evenly into 1,299 each for APR, BL and BI, with more than 60 problems per language. Each problem includes a question, example tests, buggy code, correct code and unit tests. - **Data construction:** GitHub snippets were filtered following StarCoder and then annotated by 13 CS graduates. Annotators cross-refined each other's work, and three volunteers checked accuracy against a target above 90%. - **xDebugGen strategies:** bugs are injected in three ways: into the query, into the solution, or through round-trip code translation. The output is the MdEval-Instruct corpus. - **Fine-tuning comparison:** the authors fine-tuned Meta-Llama-3-8B, OpenCoder-8B and Qwen2.5-Coder-7B on MdEval-Instruct. Their APR pass@1 matched or exceeded that of the official instruct models. - **Inconsistent statistics:** the extracted text gives conflicting figures for difficulty splits, average lengths and total unit tests (7,133 vs 6,838). Check the original paper before citing these numbers. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Beihang University](https://md.research.elamur.ai/free/en/universities/beihang-university/) — University, Beijing, China · 65 works · 8,452 citations 📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works · [💻 Software](https://md.research.elamur.ai/free/en/shelves/computer-science/software/) · 92 works ## Cite Cite as: Shukai Liu, Linzheng Chai, Jian Yang, Jiajun Shi, He Zhu, Liran Wang, Jin Ke, Wei Zhang, Hualei Zhu, Shuyue Guo, Tao Sun, Jiaheng Liu, Yunlong Duan, Yu Hao, Liqun Yang, Guanglin Niu, Ge Zhang and Zhoujun Li. 2026. MdEval: Massively Multilingual Code Debugging. Findings of the Association for Computational Linguistics: ACL 2026. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{liu2026mdeval, title = {{MdEval: Massively Multilingual Code Debugging}}, author = {Shukai Liu, Linzheng Chai, Jian Yang, Jiajun Shi, He Zhu, Liran Wang, Jin Ke, Wei Zhang, Hualei Zhu, Shuyue Guo, Tao Sun, Jiaheng Liu, Yunlong Duan, Yu Hao, Liqun Yang, Guanglin Niu, Ge Zhang and Zhoujun Li}, booktitle = {Findings of the Association for Computational Linguistics: ACL 2026}, year = {2026}, doi = {10.18653/v1/2026.findings-acl.1041}, url = {https://doi.org/10.18653/v1/2026.findings-acl.1041} } ``` - [← previous LLM-Based Human-Agent Collaboration and Interaction Systems: A Survey](https://md.research.elamur.ai/free/en/papers/zou-2026-llm-based-human-agent-collaboration/) - [next → Multimodal Hate and Sentiment Understanding in Low-Resource Text-Embedded Images for Online Safety and Digital Well-being](https://md.research.elamur.ai/free/en/papers/thapa-2026-multimodal-hate-sentiment-understanding-low/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/) ## References 5 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Meta AI. 2024. Introducing meta llama 3: The most capable openly available llm to date. [https://ai.meta.com/blog/meta-llama-3/](https://ai.meta.com/blog/meta-llama-3/). [↗](https://ai.meta.com/blog/meta-llama-3/) 2. Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, and 1 others. 2023. [Santa-Coder: Don’t reach for the stars!](https://md.research.elamur.ai/free/en/papers/allal-2023-santacoder-don-t-reach-stars/) arXiv preprint arXiv:2301.03988. 3. Miltiadis Allamanis, Henry Jackson-Flux, and Marc Brockschmidt. 2021. Self-supervised bug detection and repair. Advances in Neural Information Processing Systems, 34:27865–27876. [find](https://scholar.google.com/scholar?q=Miltiadis+Allamanis%2C+Henry+Jackson%2DFlux%2C+and+Marc+Brockschmidt%2E+2021%2E+Self%2Dsupervised+bug+detection+and+repair%2E+Advances+in+Neural+Information+Processing+Systems%2C+34%3A27865%E2%80%9327876%2E) 4. Ethical Statement Anthropic. 2023. Introducing Claude. [find](https://scholar.google.com/scholar?q=Ethical+Statement+Anthropic%2E+2023%2E+Introducing+Claude%2E) 5. Potential Risks Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and 1 others. 2021. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732. *Updated 31 October 2026*