---
title: "SUAT-BMI at MEDIQA-EVAL 2026 — paper by Peng et al. (2026)"
description: "They build three types of judge: Few-shot judge: Qwen3-30B-A3B is prompted with the gold expert… 24 references, cited by 1 paper here."
authors: ["Xinzhe Peng", "Liyuan E", "Kun Feng", "Jielin Li", "Yuxuan Tang", "Zhao Li"]
year: 2026
venue: "Proceedings of the 8th Workshop on Clinical Natural Language Processing (Clinical NLP) @ LREC 2026"
source: "https://aclanthology.org/2026.clinicalnlp-1.2/"
doi: "10.63317/2kdt525sk8is"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/peng-2026-suat-bmi-mediqa-eval-2026/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/peng-2026-suat-bmi-mediqa-eval-2026/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/17/) / SUAT-BMI at MEDIQA-EVAL 2026: An Ensemble Approach to Language Models as Judges for Automatic Rating of Medical Responses

# SUAT-BMI at MEDIQA-EVAL 2026: An Ensemble Approach to Language Models as Judges for Automatic Rating of Medical Responses

**Authors:** Xinzhe Peng, Liyuan E, Kun Feng, Jielin Li, [Yuxuan Tang](https://md.research.elamur.ai/free/en/authors/yuxuan-tang/) and [Zhao Li](https://md.research.elamur.ai/free/en/authors/zhao-li/) · **Year:** 2026 · **Venue:** [Proceedings of the 8th Workshop on Clinical Natural Language Processing (Clinical NLP) @ LREC 2026](https://md.research.elamur.ai/free/en/venues/clinicalnlp/) ([Workshop on Clinical Natural Language Processing (Clinical NLP) @ LREC 2026](https://md.research.elamur.ai/free/en/venues/clinicalnlp/2026/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.clinicalnlp-1.2/) · [PDF](https://aclanthology.org/2026.clinicalnlp-1.2.pdf) · [DOI 10.63317/2kdt525sk8is](https://doi.org/10.63317/2kdt525sk8is) · [OpenAlex](https://openalex.org/W7166532601)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary
The authors tackle the automatic rating of LLM-generated answers to dermatology and wound-care questions. They predict the scores that clinical experts would assign along several quality dimensions, using data from the English sub-task of the MEDIQA-EVAL 2026 shared task, built on DermaVQA and WoundcareVQA.

They build three types of judge:
- **Few-shot judge:** Qwen3-30B-A3B is prompted with the gold expert answers, the official rating rules and bootstrapped labelled examples. Seven runs are averaged.
- **PubMedBERT regressor:** It works on text, with images turned into captions by MedGemma-27b-it. It uses inputs tailored to each metric and upsamples minority classes.
- **RAG judge:** It is grounded in retrieved clinical guideline passages.

For each metric, the configuration that works best on the development set feeds the final ensemble. Systems are scored by the mean of Pearson, Spearman and Kendall correlations with expert labels.

The ensemble scored 0.481 and placed first of 41 teams. This shows that LLM judges can track expert opinion reasonably well, although reliability differs noticeably across criteria.

## Key points
- The ensemble scored 0.481 officially and ranked 1st among 41 teams. The six rated metrics are Disagree_Flag, Factual-Accuracy, Relevance, Completeness, Writing-Style and Overall.
- The English data has 2,898 development items, used for training, and 3,474 test items. The development set covers 483 English cases.
- Bootstrapping 20 shots across 7 conversations and averaging the outputs beat a single few-shot pass. Its reliance on gold responses limits use beyond MEDIQA-EVAL.
- On the test set, the few-shot judge dominated the per-metric results, reaching 0.60 on Relevance, 0.55 on Overall and 0.54 on Factual-Accuracy. PubMedBERT was weakest overall, at 0.20–0.40.
- In error analysis, Disagree_Flag accounted for most large errors (above 0.5): 70 cases, against 12 for Factual-Accuracy and none for Writing-Style.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Shenzhen University of Advanced Technology](https://md.research.elamur.ai/free/en/universities/shenzhen-university-of-advanced-technology/) — University, Shenzhen, China · 2 works · 1 citation

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works · [🩺 Medicine](https://md.research.elamur.ai/free/en/shelves/medicine/) · 275 works · [🩺 Health Informatics](https://md.research.elamur.ai/free/en/shelves/medicine/health-informatics/) · 214 works

## Cite

Cite as: Xinzhe Peng, Liyuan E, Kun Feng, Jielin Li, Yuxuan Tang and Zhao Li. 2026. SUAT-BMI at MEDIQA-EVAL 2026: An Ensemble Approach to Language Models as Judges for Automatic Rating of Medical Responses. Proceedings of the 8th Workshop on Clinical Natural Language Processing (Clinical NLP) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{peng2026suat,
  title = {{SUAT-BMI at MEDIQA-EVAL 2026: An Ensemble Approach to Language Models as Judges for Automatic Rating of Medical Responses}},
  author = {Xinzhe Peng, Liyuan E, Kun Feng, Jielin Li, Yuxuan Tang and Zhao Li},
  booktitle = {Proceedings of the 8th Workshop on Clinical Natural Language Processing (Clinical NLP) @ LREC 2026},
  year = {2026},
  doi = {10.63317/2kdt525sk8is},
  url = {https://doi.org/10.63317/2kdt525sk8is}
}
```

- [← previous Stylometry at SemEval-2026 Task 13: Clustered Stylometric Modeling for Machine-Generated Code Detection](https://md.research.elamur.ai/free/en/papers/santhanam-2026-stylometry-semeval-2026-task-13/)
- [next → Subevent Structure as a Predictor of Entity Identity Change in Procedural Text](https://md.research.elamur.ai/free/en/papers/rim-2026-subevent-structure-as-predictor-entity/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/17/)

## References

24 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Pedram Hosseini, Jessica M Sin, Bing Ren, Bryceton G Thomas, Elnaz Nouri, Ali Farahanchi, and Saeed Hassanpour. 2024. A benchmark for long-form medical question answering. arXiv preprint arXiv:2411.09834. — [arXiv:2411.09834](https://arxiv.org/abs/2411.09834) [find](https://scholar.google.com/scholar?q=Pedram+Hosseini%2C+Jessica+M+Sin%2C+Bing+Ren%2C+Bryceton+G+Thomas%2C+Elnaz+Nouri%2C+Ali+Farahanchi%2C+and+Saeed+Hassanpour%2E+2024%2E+A+benchmark+for+long%2Dform+medical+question+answering%2E+arXiv+preprint+arXiv%3A2411%2E09)
2. American Academy of Dermatology. 2026. Aad guidelines of care. [EB/OL]. [2026-02-03]. [find](https://scholar.google.com/scholar?q=American+Academy+of+Dermatology%2E+2026%2E+Aad+guidelines+of+care%2E+%5BEB%2FOL%5D%2E+%5B2026%2D02%2D03%5D%2E)
3. American Physical Therapy Association. 2026. Apta clinical practice guidelines. [EB/OL]. [2026-02-03]. [find](https://scholar.google.com/scholar?q=American+Physical+Therapy+Association%2E+2026%2E+Apta+clinical+practice+guidelines%2E+%5BEB%2FOL%5D%2E+%5B2026%2D02%2D03%5D%2E)
4. Infectious Diseases Society of America. 2026. Idsa practice guidelines. [EB/OL]. [2026-02-03]. [find](https://scholar.google.com/scholar?q=Infectious+Diseases+Society+of+America%2E+2026%2E+Idsa+practice+guidelines%2E+%5BEB%2FOL%5D%2E+%5B2026%2D02%2D03%5D%2E)
5. Wen-wai Yim Asma Ben Abacha. 2026. Overview of the mediqa-eval 2026 shared task on evaluation metrics in medical multimodal question answering. In Proceedings of the 8th Clinical Natural Language Processing Workshop, ClinicalNLP@LREC 2026, Palma, Mallorca, Spain, May 16, 2026. Association for Comput… [find](https://scholar.google.com/scholar?q=Wen%2Dwai+Yim+Asma+Ben+Abacha%2E+2026%2E+Overview+of+the+mediqa%2Deval+2026+shared+task+on+evaluation+metrics+in+medical+multimodal+question+answering%2E+In+Proceedings+of+the+8th+Clinical+Natural+Language+Proc)
6. Veysel Kocaman, Murat Kaya, Adrian Feier, and David Talby. 2025. Clinical large language model evaluation by expert review (clever): Framework development and validation. JMIR AI, 4:e72153. [find](https://scholar.google.com/scholar?q=Veysel+Kocaman%2C+Murat+Kaya%2C+Adrian+Feier%2C+and+David+Talby%2E+2025%2E+Clinical+large+language+model+evaluation+by+expert+review+%28clever%29%3A+Framework+development+and+validation%2E+JMIR+AI%2C+4%3Ae72153%2E)
7. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
8. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024a. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. [find](https://scholar.google.com/scholar?q=Jianlv+Chen%2C+Shitao+Xiao%2C+Peitian+Zhang%2C+Kun+Luo%2C+Defu+Lian%2C+and+Zheng+Liu%2E+2024a%2E+Bge+m3%2Dembedding%3A+Multi%2Dlingual%2C+multi%2Dfunctionality%2C+multi%2Dgranularity+text+embeddings+through+self%2Dknowledge+distil)
9. Jerry Liu. 2022. LlamaIndex. [find](https://scholar.google.com/scholar?q=Jerry+Liu%2E+2022%2E+LlamaIndex%2E)
10. National Institute for Health and Care Excellence. 2026. Nice guidelines: Evidence-based recommendations for health and care. [EB/OL]. [2026-02-03]. [find](https://scholar.google.com/scholar?q=National+Institute+for+Health+and+Care+Excellence%2E+2026%2E+Nice+guidelines%3A+Evidence%2Dbased+recommendations+for+health+and+care%2E+%5BEB%2FOL%5D%2E+%5B2026%2D02%2D03%5D%2E)
11. M. L. Chen et al. 2024b. Performance and risk of harm of a large language model on dermatology NHS England. 2023. Wound care information standard. [EB/OL]. [2026-02-03]. [find](https://scholar.google.com/scholar?q=M%2E+L%2E+Chen+et+al%2E+2024b%2E+Performance+and+risk+of+harm+of+a+large+language+model+on+dermatology+NHS+England%2E+2023%2E+Wound+care+information+standard%2E+%5BEB%2FOL%5D%2E+%5B2026%2D02%2D03%5D%2E)
12. Chung, Yi Xing, Ming-Hsiu Tsai, Eric Schaffer, Yi Shi, Nan Liu, Zhi Liu, and Rong Zhang. 2025. Automating expert-level medical reasoning evaluation of large language models. NPJ Digital Medicine, 9(1):34. [find](https://scholar.google.com/scholar?q=Chung%2C+Yi+Xing%2C+Ming%2DHsiu+Tsai%2C+Eric+Schaffer%2C+Yi+Shi%2C+Nan+Liu%2C+Zhi+Liu%2C+and+Rong+Zhang%2E+2025%2E+Automating+expert%2Dlevel+medical+reasoning+evaluation+of+large+language+models%2E+NPJ+Digital+Medicine%2C+9%281%29)
13. QwenTeam. 2025. Qwen3 technical report. [find](https://scholar.google.com/scholar?q=QwenTeam%2E+2025%2E+Qwen3+technical+report%2E)
14. Daniel Reichenpfader, Peter Rösslhuemer, and Kerstin Denecke. 2024. Large language model-based evaluation of medical question answering systems: Algorithm development and case study. Studies in Health Technology and Informatics, 313:22–27. [find](https://scholar.google.com/scholar?q=Daniel+Reichenpfader%2C+Peter+R%C3%B6sslhuemer%2C+and+Kerstin+Denecke%2E+2024%2E+Large+language+model%2Dbased+evaluation+of+medical+question+answering+systems%3A+Algorithm+development+and+case+study%2E+Studies+in+Health)
15. World Health Organization. 2026. World health organization: Guidelines and technical documents. [EB/OL]. [2026-02-03]. [find](https://scholar.google.com/scholar?q=World+Health+Organization%2E+2026%2E+World+health+organization%3A+Guidelines+and+technical+documents%2E+%5BEB%2FOL%5D%2E+%5B2026%2D02%2D03%5D%2E)
16. Wound Healing Society. 2026. Wound healing society clinical resources and guidelines. [EB/OL]. [2026-02-03]. [find](https://scholar.google.com/scholar?q=Wound+Healing+Society%2E+2026%2E+Wound+healing+society+clinical+resources+and+guidelines%2E+%5BEB%2FOL%5D%2E+%5B2026%2D02%2D03%5D%2E)
17. Wound, Ostomy and Continence Nurses Society. 2026. Wocn society: Clinical guidelines and resources. [EB/OL]. [2026-02-03]. [find](https://scholar.google.com/scholar?q=Wound%2C+Ostomy+and+Continence+Nurses+Society%2E+2026%2E+Wocn+society%3A+Clinical+guidelines+and+resources%2E+%5BEB%2FOL%5D%2E+%5B2026%2D02%2D03%5D%2E)
18. Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023. C-pack: Packaged resources to advance general chinese embedding. [find](https://scholar.google.com/scholar?q=Shitao+Xiao%2C+Zheng+Liu%2C+Peitian+Zhang%2C+and+Niklas+Muennighoff%2E+2023%2E+C%2Dpack%3A+Packaged+resources+to+advance+general+chinese+embedding%2E)
19. Wen-wai Yim, Asma Ben Abacha, Robert Doerning, Chia-Yu Chen, Jiaying Xu, Anita Subbarao, Zixuan Yu, Fei Xia, M. Kennedy Hall, and Meliha Yetisgen. 2025a. Woundcarevqa: A multilingual visual question answering benchmark dataset for wound care. Journal of Biomedical Informatics, 170:104888. [find](https://scholar.google.com/scholar?q=Wen%2Dwai+Yim%2C+Asma+Ben+Abacha%2C+Robert+Doerning%2C+Chia%2DYu+Chen%2C+Jiaying+Xu%2C+Anita+Subbarao%2C+Zixuan+Yu%2C+Fei+Xia%2C+M%2E+Kennedy+Hall%2C+and+Meliha+Yetisgen%2E+2025a%2E+Woundcarevqa%3A+A+multilingual+visual+question+a)
20. Wen-wai Yim, Asma Ben Abacha, Zixuan Yu, Robert Doerning, Fei Xia, and Meliha Yetisgen. 2025b. [MORQA: benchmarking evaluation metrics for medical open-ended question answering](https://md.research.elamur.ai/free/en/papers/yim-2026-morqa-benchmarking-evaluation-metrics-medical/). volume abs/2509.12405.
21. Ww. Yim, Y. Fu, Z. Sun, A. B. Abacha, M. Yetisgen, and F. Xia. 2024. Dermavqa: A multilingual visual question answering dataset for dermatology. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2024, volume 15005 of Lecture Notes in Computer Science, Cham. Springer. [find](https://scholar.google.com/scholar?q=Ww%2E+Yim%2C+Y%2E+Fu%2C+Z%2E+Sun%2C+A%2E+B%2E+Abacha%2C+M%2E+Yetisgen%2C+and+F%2E+Xia%2E+2024%2E+Dermavqa%3A+A+multilingual+visual+question+answering+dataset+for+dermatology%2E+In+Medical+Image+Computing+and+Computer+Assisted+Interv)
22. E Yu, X Chu, W Zhang, X Meng, Y Yang, X Ji, and C Wu. 2025. Large language models in medicine: Applications, challenges, and future directions. International Journal of Medical Sciences, 22(11):2792–2801. [find](https://scholar.google.com/scholar?q=E+Yu%2C+X+Chu%2C+W+Zhang%2C+X+Meng%2C+Y+Yang%2C+X+Ji%2C+and+C+Wu%2E+2025%2E+Large+language+models+in+medicine%3A+Applications%2C+challenges%2C+and+future+directions%2E+International+Journal+of+Medical+Sciences%2C+22%2811%29%3A2792%E2%80%932)
23. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E)
24. Sheng Zhou, Wei Xie, Jian Li, Zhi Zhan, Ming Song, Hui Yang, Carlos Espinoza, Laura Welton, Xiang Mai, Yan Jin, Zhen Xu, Young-Hoon [find](https://scholar.google.com/scholar?q=Sheng+Zhou%2C+Wei+Xie%2C+Jian+Li%2C+Zhi+Zhan%2C+Ming+Song%2C+Hui+Yang%2C+Carlos+Espinoza%2C+Laura+Welton%2C+Xiang+Mai%2C+Yan+Jin%2C+Zhen+Xu%2C+Young%2DHoon)

*Updated 2 November 2026*