---
title: "MORQA: Benchmarking Evaluation Metrics for — Yim 2026"
description: "Automatic scores like BLEU, ROUGE and BERTScore are poor judges of free-text medical answers."
authors: ["Wen-wai Yim", "Asma Ben Abacha", "Zixuan Yu", "Robert Doerning", "Fei Xia", "Meliha Yetisgen"]
year: 2026
venue: "Proceedings of the Fifteenth Language Resources and Evaluation Conference"
source: "https://aclanthology.org/2026.lrec-1.396/"
doi: "10.63317/5e46vrbnwfwe"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/yim-2026-morqa-benchmarking-evaluation-metrics-medical/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/yim-2026-morqa-benchmarking-evaluation-metrics-medical/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2026](https://md.research.elamur.ai/free/papers/2026/page/3/) / MORQA: Benchmarking Evaluation Metrics for Medical Open-Ended Question Answering

# MORQA: Benchmarking Evaluation Metrics for Medical Open-Ended Question Answering

**Authors:** Wen-wai Yim, Asma Ben Abacha, Zixuan Yu, Robert Doerning, [Fei Xia](https://md.research.elamur.ai/free/authors/fei-xia/), Meliha Yetisgen · **Year:** 2026 · **Venue:** Proceedings of the Fifteenth Language Resources and Evaluation Conference · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.lrec-1.396/) · [PDF](https://aclanthology.org/2026.lrec-1.396.pdf) · [DOI 10.63317/5e46vrbnwfwe](https://doi.org/10.63317/5e46vrbnwfwe) · [OpenAlex](https://openalex.org/W4415315915)

**Citations:** 1 citations (openalex-snapshot · 2026-10-14) · 10 in our library, of the 140673 papers whose bibliographies we read

## Summary
Automatic scores like BLEU, ROUGE and BERTScore are poor judges of free-text medical answers. Many different answers can be equally correct, and a reply that shares wording with the reference can still be clinically wrong. The authors present MORQA (Medical Open-Response QA), a bilingual English–Chinese benchmark for testing how well evaluation metrics agree with expert opinion. It draws on visual and text-based medical QA collections: WoundcareVQA, DermaVQA (iiyi), LiveQA and MedDialog. Each query has between two and four or more gold answers written by medical professionals. For DermaVQA, two extra responses per instance were added by board-certified dermatologists. Domain experts rated system answers, and in the English clinical subsets two practicing physicians did the rating. The study compares n-gram metrics, embedding-based metrics, and LLM judges such as GPT-4o, Gemini-1.5-pro and DeepSeek-V3. The LLM judges correlate with expert ratings much more closely than traditional metrics. The authors link this to the LLMs' sensitivity to meaning and their tolerance of variation among reference answers. Code and annotations are publicly released.

## Key points
- Test splits contain 93 WoundcareVQA queries, 100 DermaVQA queries, 40 LiveQA queries and 259 MedDialog queries. Validation splits contain 105, 56 and 236 queries respectively; LiveQA has no validation split.
- The compared metrics are BLEU/deltaBLEU, ROUGE-1/2/L, BERTScore, BLEURT-20, Prometheus 7B v2, GPT-4o, Gemini-1.5-pro and DeepSeek-V3. Agreement with experts is measured with Kendall, Pearson and Spearman correlations plus accuracy.
- Ratings are broken down by quality aspects, including relevance, writing style, factual consistency with the gold answer, and overall quality.
- A survey of earlier work shows how inconsistent metrics can be. On WoundcareVQA, a ROUGE-L of 0.501 appears alongside a BLEU of only 0.123.
- Code is available at github.com/wyim/morqa-eval and data at osf.io/kcv2n. The paper appeared at LREC 2026.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

Topics: [Machine translation](https://md.research.elamur.ai/free/topics/machine-translation/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yim2026morqa,
  title = {{MORQA: Benchmarking Evaluation Metrics for Medical Open-Ended Question Answering}},
  author = {Yim, Wen-wai and Abacha, Asma Ben and Yu, Zixuan and Doerning, Robert and Xia, Fei and Yetisgen, Meliha},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/5e46vrbnwfwe},
  url = {https://doi.org/10.63317/5e46vrbnwfwe}
}
```

- [← previous MorphoFiltered-Gemini at MWE-2026 PARSEME 2.0 Subtask 1: Tackling LLM Overgeneration via Universal POS-based Constraints](https://md.research.elamur.ai/free/papers/moise-2026-morphofiltered-gemini-mwe-2026-parseme/)
- [next → MQM Re-Annotation: A Technique for Collaborative Evaluation of Machine Translation](https://md.research.elamur.ai/free/papers/riley-2026-mqm-re-annotation-technique-collaborative/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/papers/2026/page/3/)

## References

25 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Asma Ben Abacha, Eugene Agichtein, Yuval Pinter, and Dina Demner-Fushman. 2017. Overview of the medical question answering task at trec 2017 liveqa. In TREC 2017. [find](https://scholar.google.com/scholar?q=Asma+Ben+Abacha%2C+Eugene+Agichtein%2C+Yuval+Pinter%2C+and+Dina+Demner%2DFushman%2E+2017%2E+Overview+of+the+medical+question+answering+task+at+trec+2017+liveqa%2E+In+TREC+2017%2E)
2. Chongyan Chen, Mengchen Liu, Noel Codella, Yunsheng Li, Lu Yuan, and Danna Gurari. 2024. Fully authentic visual question answering dataset from online communities. In Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part XLVIII, volume… [find](https://scholar.google.com/scholar?q=Chongyan+Chen%2C+Mengchen+Liu%2C+Noel+Codella%2C+Yunsheng+Li%2C+Lu+Yuan%2C+and+Danna+Gurari%2E+2024%2E+Fully+authentic+visual+question+answering+dataset+from+online+communities%2E+In+Computer+Vision+%2D+ECCV+2024+%2D+18t)
3. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bing-Li Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dong-Li Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Ha… [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2C+Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+Bing%2DLi+Wang%2C+Bochao+Wu%2C+Chengda+Lu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chenyu+Zhang%2C+Chong+Ruan%2C+Damai+Dai%2C+Daya+Guo%2C+Dejian+Yang%2C+Deli+Chen%2C+Dong%2DLi+Ji%2C+Erhang)
4. Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. In North American Chapter of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jinlan+Fu%2C+See%2DKiong+Ng%2C+Zhengbao+Jiang%2C+and+Pengfei+Liu%2E+2023%2E+Gptscore%3A+Evaluate+as+you+desire%2E+In+North+American+Chapter+of+the+Association+for+Computational+Linguistics%2E)
5. Michel Galley, Chris Brockett, Alessandro Sordoni, Yangfeng Ji, Michael Auli, Chris Quirk, Margaret Mitchell, Jianfeng Gao, and Bill Dolan. 2015. deltaBLEU: A discriminative metric for generation tasks with intrinsically diverse targets. In Proceedings of the 53rd Annual Meeting of the Association f… [find](https://scholar.google.com/scholar?q=Michel+Galley%2C+Chris+Brockett%2C+Alessandro+Sordoni%2C+Yangfeng+Ji%2C+Michael+Auli%2C+Chris+Quirk%2C+Margaret+Mitchell%2C+Jianfeng+Gao%2C+and+Bill+Dolan%2E+2015%2E+deltaBLEU%3A+A+discriminative+metric+for+generation+task)
6. Taneesh Gupta, Shivam Shandilya, Xuchao Zhang, Rahul Madhavan, Supriyo Ghosh, Chetan Bansal, Huaxiu Yao, and Saravan Rajmohan. 2025. Carmo: Dynamic criteria generation for context-aware reward modelling. [find](https://scholar.google.com/scholar?q=Taneesh+Gupta%2C+Shivam+Shandilya%2C+Xuchao+Zhang%2C+Rahul+Madhavan%2C+Supriyo+Ghosh%2C+Chetan+Bansal%2C+Huaxiu+Yao%2C+and+Saravan+Rajmohan%2E+2025%2E+Carmo%3A+Dynamic+criteria+generation+for+context%2Daware+reward+modelli)
7. Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, S. Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. 2023. Prometheus: Inducing fine-grained evaluation capability in language models. ArXiv, abs/2310.08491. [find](https://scholar.google.com/scholar?q=Seungone+Kim%2C+Jamin+Shin%2C+Yejin+Cho%2C+Joel+Jang%2C+S%2E+Longpre%2C+Hwaran+Lee%2C+Sangdoo+Yun%2C+Seongjin+Shin%2C+Sungdong+Kim%2C+James+Thorne%2C+and+Minjoon+Seo%2E+2023%2E+Prometheus%3A+Inducing+fine%2Dgrained+evaluation+capa) [our card](https://md.research.elamur.ai/free/papers/kim-2023-prometheus-inducing-fine-grained-evaluation/)
8. Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. Prometheus 2: An open source language model specialized in evaluating other language models. ArXiv, abs/2405.01535. [find](https://scholar.google.com/scholar?q=Seungone+Kim%2C+Juyoung+Suk%2C+Shayne+Longpre%2C+Bill+Yuchen+Lin%2C+Jamin+Shin%2C+Sean+Welleck%2C+Graham+Neubig%2C+Moontae+Lee%2C+Kyungjae+Lee%2C+and+Minjoon+Seo%2E+2024%2E+Prometheus+2%3A+An+open+source+language+model+speci) [our card](https://md.research.elamur.ai/free/papers/kim-2024-prometheus-2-open-source-language/)
9. Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. 2018. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1–10. [find](https://scholar.google.com/scholar?q=Jason+J+Lau%2C+Soumya+Gayen%2C+Asma+Ben+Abacha%2C+and+Dina+Demner%2DFushman%2E+2018%2E+A+dataset+of+clinically+generated+visual+questions+and+answers+about+radiology+images%2E+Scientific+data%2C+5%281%29%3A1%E2%80%9310%2E)
10. Chin-Yew Lin. 2004a. Rouge: A package for automatic evaluation of summaries. In Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin%2E+2004a%2E+Rouge%3A+A+package+for+automatic+evaluation+of+summaries%2E+In+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2E)
11. Chin-Yew Lin. 2004b. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin%2E+2004b%2E+ROUGE%3A+A+package+for+automatic+evaluation+of+summaries%2E+In+Text+Summarization+Branches+Out%2C+pages+74%E2%80%9381%2C+Barcelona%2C+Spain%2E+Association+for+Computational+Linguistics%2E)
12. Wenge Liu, Jianheng Tang, Yi Cheng, Wenjie Li, Yefeng Zheng, and Xiaodan Liang. 2020. Meddg: An entity-centric medical consultation dataset for entity-aware medical dialogue generation. In Natural Language Processing and Chinese Computing. [find](https://scholar.google.com/scholar?q=Wenge+Liu%2C+Jianheng+Tang%2C+Yi+Cheng%2C+Wenjie+Li%2C+Yefeng+Zheng%2C+and+Xiaodan+Liang%2E+2020%2E+Meddg%3A+An+entity%2Dcentric+medical+consultation+dataset+for+entity%2Daware+medical+dialogue+generation%2E+In+Natural+Lan)
13. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+Bleu%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2E) [our card](https://md.research.elamur.ai/free/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
14. Thibault Sellam, Dipanjan Das, and Ankur P. Parikh. 2020. Bleurt: Learning robust metrics for text generation. In Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Thibault+Sellam%2C+Dipanjan+Das%2C+and+Ankur+P%2E+Parikh%2E+2020%2E+Bleurt%3A+Learning+robust+metrics+for+text+generation%2E+In+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2E)
15. Ivan Srba and Mária Bieliková. 2016. A comprehensive survey and classification of approaches for community question answering. ACM Transactions on the Web (TWEB), 10:1 – 63. [find](https://scholar.google.com/scholar?q=Ivan+Srba+and+M%C3%A1ria+Bielikov%C3%A1%2E+2016%2E+A+comprehensive+survey+and+classification+of+approaches+for+community+question+answering%2E+ACM+Transactions+on+the+Web+%28TWEB%29%2C+10%3A1+%E2%80%93+63%2E)
16. Zhiliang Tian, Rui Yan, Lili Mou, Yiping Song, Yansong Feng, and Dongyan Zhao. 2017. How to make context more useful? an empirical study on context-aware neural conversational models. In Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zhiliang+Tian%2C+Rui+Yan%2C+Lili+Mou%2C+Yiping+Song%2C+Yansong+Feng%2C+and+Dongyan+Zhao%2E+2017%2E+How+to+make+context+more+useful%3F+an+empirical+study+on+context%2Daware+neural+conversational+models%2E+In+Annual+Meetin)
17. Wen wai Yim, Asma Ben Abacha, Robert Doerning, Chia-Yu Chen, Jiaying Xu, Anita Subbarao, Zixuan Yu, Fei Xia, M. Kennedy Hall, and Meliha Yetisgen. 2025. Woundcarevqa: A multilingual visual question answering benchmark dataset for wound care. Journal of Biomedical Informatics, 170:104888. [find](https://scholar.google.com/scholar?q=Wen+wai+Yim%2C+Asma+Ben+Abacha%2C+Robert+Doerning%2C+Chia%2DYu+Chen%2C+Jiaying+Xu%2C+Anita+Subbarao%2C+Zixuan+Yu%2C+Fei+Xia%2C+M%2E+Kennedy+Hall%2C+and+Meliha+Yetisgen%2E+2025%2E+Woundcarevqa%3A+A+multilingual+visual+question+an)
18. Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. 2023. Flask: Fine-grained language model evaluation based on alignment skill sets. ArXiv, abs/2307.10928. [find](https://scholar.google.com/scholar?q=Seonghyeon+Ye%2C+Doyoung+Kim%2C+Sungdong+Kim%2C+Hyeonbin+Hwang%2C+Seungone+Kim%2C+Yongrae+Jo%2C+James+Thorne%2C+Juho+Kim%2C+and+Minjoon+Seo%2E+2023%2E+Flask%3A+Fine%2Dgrained+language+model+evaluation+based+on+alignment+skil)
19. Wen-wai Yim, Yujuan Fu, Zhaoyi Sun, Asma Ben Abacha, Meliha Yetisgen, and Fei Xia. 2024. Dermavqa: A multilingual visual question answering dataset for dermatology. In Medical Image Computing and Computer Assisted Intervention - MICCAI 2024 - 27th International Conference, Marrakesh, Morocco, Octobe… [find](https://scholar.google.com/scholar?q=Wen%2Dwai+Yim%2C+Yujuan+Fu%2C+Zhaoyi+Sun%2C+Asma+Ben+Abacha%2C+Meliha+Yetisgen%2C+and+Fei+Xia%2E+2024%2E+Dermavqa%3A+A+multilingual+visual+question+answering+dataset+for+dermatology%2E+In+Medical+Image+Computing+and+Comp)
20. Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. ArXiv, abs/2106.11520. [find](https://scholar.google.com/scholar?q=Weizhe+Yuan%2C+Graham+Neubig%2C+and+Pengfei+Liu%2E+2021%2E+Bartscore%3A+Evaluating+generated+text+as+text+generation%2E+ArXiv%2C+abs%2F2106%2E11520%2E) [our card](https://md.research.elamur.ai/free/papers/yuan-2021-bartscore-evaluating-generated-text-text/)
21. Guangtao Zeng, Wenmian Yang, Zeqian Ju, Yue Yang, Sicheng Wang, Ruisi Zhang, Meng Zhou, Jiaqi Zeng, Xiangyu Dong, Ruoyu Zhang, Hongchao Fang, Penghui Zhu, Shu Chen, and Pengtao Xie. 2020. MedDialog: Large-scale medical dialogue datasets. In Proceedings of the 2020 Conference on Empirical Methods in… [find](https://scholar.google.com/scholar?q=Guangtao+Zeng%2C+Wenmian+Yang%2C+Zeqian+Ju%2C+Yue+Yang%2C+Sicheng+Wang%2C+Ruisi+Zhang%2C+Meng+Zhou%2C+Jiaqi+Zeng%2C+Xiangyu+Dong%2C+Ruoyu+Zhang%2C+Hongchao+Fang%2C+Penghui+Zhu%2C+Shu+Chen%2C+and+Pengtao+Xie%2E+2020%2E+MedDialog%3A+L)
22. Hainan Zhang, Yanyan Lan, Liang Pang, J. Guo, and Xueqi Cheng. 2019a. Recosa: Detecting the relevant contexts with self-attention for multi-turn dialogue generation. In Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Hainan+Zhang%2C+Yanyan+Lan%2C+Liang+Pang%2C+J%2E+Guo%2C+and+Xueqi+Cheng%2E+2019a%2E+Recosa%3A+Detecting+the+relevant+contexts+with+self%2Dattention+for+multi%2Dturn+dialogue+generation%2E+In+Annual+Meeting+of+the+Associati)
23. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019b. Bertscore: Evaluating text generation with bert. ArXiv, abs/1904.09675. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2019b%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+ArXiv%2C+abs%2F1904%2E09675%2E)
24. Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance. In Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Wei+Zhao%2C+Maxime+Peyrard%2C+Fei+Liu%2C+Yang+Gao%2C+Christian+M%2E+Meyer%2C+and+Steffen+Eger%2E+2019%2E+Moverscore%3A+Text+generation+evaluating+with+contextualized+embeddings+and+earth+mover+distance%2E+In+Conference+o) [our card](https://md.research.elamur.ai/free/papers/zhao-2019-moverscore-text-generation-evaluating-contextualized/)
25. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Haotong Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. ArXiv, abs/2306.05685. [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric+P%2E+Xing%2C+Haotong+Zhang%2C+Joseph+E%2E+Gonzalez%2C+and+Ion+Stoica%2E+2023%2E+Judging+ll)