---
title: "Towards Unified Factuality Evaluation for Biomedical QA and"
description: "The authors propose an evaluation framework that adds evidence-based checking to these reference-based… 0.61 ROUGE-L. 16 references."
authors: ["Mahule Roy", "Subhas Roy"]
year: 2026
venue: "BioNLP 2026"
source: "https://aclanthology.org/2026.bionlp-1.2/"
doi: "10.18653/v1/2026.bionlp-1.2"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/roy-2026-unified-factuality-evaluation-biomedical-qa/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/roy-2026-unified-factuality-evaluation-biomedical-qa/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/54/) / Towards Unified Factuality Evaluation for Biomedical QA and Summarization: Aligning Metrics with Clinical Use-Cases

# Towards Unified Factuality Evaluation for Biomedical QA and Summarization: Aligning Metrics with Clinical Use-Cases

**Authors:** Mahule Roy and Subhas Roy · **Year:** 2026 · **Venue:** [BioNLP 2026](https://md.research.elamur.ai/free/en/venues/bionlp/) ([BioNLP 2026](https://md.research.elamur.ai/free/en/venues/bionlp/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.bionlp-1.2/) · [PDF](https://aclanthology.org/2026.bionlp-1.2.pdf) · [DOI 10.18653/v1/2026.bionlp-1.2](https://doi.org/10.18653/v1/2026.bionlp-1.2) · [OpenAlex](https://openalex.org/W7166822807)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Standard overlap metrics such as ROUGE, BLEU and BERTScore can give biomedical LLM outputs high marks while missing clinically important factual mistakes. The authors propose an evaluation framework that adds evidence-based checking to these reference-based scores. Each generated answer is split into atomic claims with spaCy. For every claim, the top five PubMed abstracts (2014–2024) are retrieved, and BioBERT-NLI, trained on MedNLI, labels the claim as supported, contradicted or unsupported. Separate rule-based checks against RxNorm and an FDA contraindication list flag dosage and medication hazards. The results are combined into a composite Fact-Aligned Score (FAS).

They tested Llama-2-13B-chat, Llama-2-7B-chat, Mistral-7B-Instruct-v0.2 and an encoder-only PubMedBERT baseline on BioASQ, PubMedQA and MedLFQA. Even the strongest model had a sizeable share of contradicted claims. On MedLFQA, FAS tracked expert claim-level labels much more closely than lexical metrics did. The work matters because rewarding surface overlap can hide errors that are unsafe in clinical use. An open-source toolkit with outputs and scripts is released.

## Key points
- **FAS definition:** FAS = 0.3·BERTScore + 0.4·Support@5 + 0.3·(1−Contra@5). It uses a 0.7 NLI confidence threshold, chosen through a sensitivity sweep from 0.5 to 0.9.
- **Benchmarks:** BioASQ Task B, 10th edition (492 questions), PubMedQA (500 instances) and a 200-item MedLFQA subset. MedLFQA gold labels were derived from its Must Have / Nice to Have annotations.
- **Contradiction rates on BioASQ:** Contra@5 ranged from 13.4% (Llama-2-13B, Support@5 71.2%) to 24.7% (PubMedBERT).
- **Correlation with gold labels on MedLFQA (Spearman ρ):** FAS 0.68, Support@5 alone 0.61, ROUGE-L 0.41, BLEU-4 0.36.
- **Ablations:** removing retrieval lowers ρ to 0.52. Using a general MNLI model instead of BioBERT-NLI lowers it to 0.45. Retrieving only a single abstract per claim lowers it to 0.56.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Oxford](https://md.research.elamur.ai/free/en/universities/university-of-oxford/) — University, Oxford, United Kingdom · 122 works · 160,277 citations

📚 Shelves (8): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🧬 Biochemistry, Genetics and Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/) · 263 works · [🧬 Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/molecular-biology/) · 260 works · [🩺 Medicine](https://md.research.elamur.ai/free/en/shelves/medicine/) · 275 works · [🩺 Health Informatics](https://md.research.elamur.ai/free/en/shelves/medicine/health-informatics/) · 214 works

🧭 Topics (3): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works · [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 487 works · [📝 Summarization](https://md.research.elamur.ai/free/en/topics/summarization/) · 243 works

## Cite

Cite as: Mahule Roy and Subhas Roy. 2026. Towards Unified Factuality Evaluation for Biomedical QA and Summarization: Aligning Metrics with Clinical Use-Cases. BioNLP 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{roy2026unified,
  title = {{Towards Unified Factuality Evaluation for Biomedical QA and Summarization: Aligning Metrics with Clinical Use-Cases}},
  author = {Mahule Roy and Subhas Roy},
  booktitle = {BioNLP 2026},
  year = {2026},
  doi = {10.18653/v1/2026.bionlp-1.2},
  url = {https://doi.org/10.18653/v1/2026.bionlp-1.2}
}
```

- [← previous Towards Understanding the Robustness of Sparse Autoencoders](https://md.research.elamur.ai/free/en/papers/saiyed-2026-understanding-robustness-sparse-autoencoders/)
- [next → Towards Unified Multimodal Large Language Models: A survey](https://md.research.elamur.ai/free/en/papers/ma-2026-unified-multimodal-large-language-models/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/54/)

## References

16 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Nentidis, A., Katsimpras, G., Krithara, A., Lima López, S., Farré-Maduell, E., Gasco, L., ... & Paliouras, G. (2023, September). Overview of bioasq 2023: The eleventh bioasq challenge on large-scale biomedical semantic indexing and question answering. In International Conference of the Cross-Languag… [find](https://scholar.google.com/scholar?q=Nentidis%2C+A%2E%2C+Katsimpras%2C+G%2E%2C+Krithara%2C+A%2E%2C+Lima+L%C3%B3pez%2C+S%2E%2C+Farr%C3%A9%2DMaduell%2C+E%2E%2C+Gasco%2C+L%2E%2C+%2E%2E%2E+%26+Paliouras%2C+G%2E+%282023%2C+September%29%2E+Overview+of+bioasq+2023%3A+The+eleventh+bioasq+challenge+on+large%2Dscale+b)
2. Bird, S., Klein, E., & Loper, E. (2009). Natural language processing with Python: analyzing text with the natural language toolkit. " O’Reilly Media, Inc.". [find](https://scholar.google.com/scholar?q=Bird%2C+S%2E%2C+Klein%2C+E%2E%2C+%26+Loper%2C+E%2E+%282009%29%2E+Natural+language+processing+with+Python%3A+analyzing+text+with+the+natural+language+toolkit%2E+%22+O%E2%80%99Reilly+Media%2C+Inc%2E%22%2E)
3. Cock, P. J., Antao, T., Chang, J. T., Chapman, B. A., Cox, C. J., Dalke, A., ... & De Hoon, M. J. (2009). Biopython: freely available Python tools for computational molecular biology and bioinformatics. Bioinformatics, 25(11), 1422. [find](https://scholar.google.com/scholar?q=Cock%2C+P%2E+J%2E%2C+Antao%2C+T%2E%2C+Chang%2C+J%2E+T%2E%2C+Chapman%2C+B%2E+A%2E%2C+Cox%2C+C%2E+J%2E%2C+Dalke%2C+A%2E%2C+%2E%2E%2E+%26+De+Hoon%2C+M%2E+J%2E+%282009%29%2E+Biopython%3A+freely+available+Python+tools+for+computational+molecular+biology+and+bioinformatic)
4. Jin, Q., Dhingra, B., Liu, Z., Cohen, W., & Lu, X. (2019, November). [Pubmedqa: A dataset for biomedical research question answering](https://md.research.elamur.ai/free/en/papers/jin-2019-pubmedqa-dataset-biomedical-research-question/). In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EM…
5. Jeong, M., Hwang, H., Yoon, C., Lee, T., & Kang, J. (2024). Olaph: Improving factuality in biomedical long-form question answering. arXiv preprint arXiv:2405.12701. — [arXiv:2405.12701](https://arxiv.org/abs/2405.12701) [find](https://scholar.google.com/scholar?q=Jeong%2C+M%2E%2C+Hwang%2C+H%2E%2C+Yoon%2C+C%2E%2C+Lee%2C+T%2E%2C+%26+Kang%2C+J%2E+%282024%29%2E+Olaph%3A+Improving+factuality+in+biomedical+long%2Dform+question+answering%2E+arXiv+preprint+arXiv%3A2405%2E12701%2E+%E2%80%94+%5BarXiv%3A2405%2E12701%5D%28https%3A%2F%2Farxiv%2E)
6. Shivade, C. (2017). Mednli-a natural language inference dataset for the clinical domain. [find](https://scholar.google.com/scholar?q=Shivade%2C+C%2E+%282017%29%2E+Mednli%2Da+natural+language+inference+dataset+for+the+clinical+domain%2E)
7. Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019). Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. [find](https://scholar.google.com/scholar?q=Zhang%2C+T%2E%2C+Kishore%2C+V%2E%2C+Wu%2C+F%2E%2C+Weinberger%2C+K%2E+Q%2E%2C+%26+Artzi%2C+Y%2E+%282019%29%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+arXiv+preprint+arXiv%3A1904%2E09675%2E)
8. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., ... & Scialom, T. (2023). [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288.
9. Jiang, Y., Li, X., Zhu, G., Li, H., Deng, J., Han, K., ... & Zhang, R. (2023). 6G non-terrestrial networks enabled low-altitude economy: Opportunities and challenges. arXiv preprint arXiv:2311.09047. — [arXiv:2311.09047](https://arxiv.org/abs/2311.09047) [find](https://scholar.google.com/scholar?q=Jiang%2C+Y%2E%2C+Li%2C+X%2E%2C+Zhu%2C+G%2E%2C+Li%2C+H%2E%2C+Deng%2C+J%2E%2C+Han%2C+K%2E%2C+%2E%2E%2E+%26+Zhang%2C+R%2E+%282023%29%2E+6G+non%2Dterrestrial+networks+enabled+low%2Daltitude+economy%3A+Opportunities+and+challenges%2E+arXiv+preprint+arXiv%3A2311%2E09047%2E+)
10. Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., ... & Poon, H. (2021). [Domain-specific language model pretraining for biomedical natural language processing](https://md.research.elamur.ai/free/en/papers/gu-2020-domain-specific-language-model-pretraining/). ACM Transactions on Computing for Healthcare (HEALTH), 3(1), 1-23.
11. Efron, B., & Tibshirani, R. (1986). Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy. Statistical science, 54-75. [find](https://scholar.google.com/scholar?q=Efron%2C+B%2E%2C+%26+Tibshirani%2C+R%2E+%281986%29%2E+Bootstrap+methods+for+standard+errors%2C+confidence+intervals%2C+and+other+measures+of+statistical+accuracy%2E+Statistical+science%2C+54%2D75%2E)
12. Honnibal, M., Montani, I., Van Landeghem, S., & Boyd, A. (2020). spaCy: Industrial-strength natural language processing in Python. [find](https://scholar.google.com/scholar?q=Honnibal%2C+M%2E%2C+Montani%2C+I%2E%2C+Van+Landeghem%2C+S%2E%2C+%26+Boyd%2C+A%2E+%282020%29%2E+spaCy%3A+Industrial%2Dstrength+natural+language+processing+in+Python%2E)
13. Liu, S., Ma, W., Moore, R., Ganesan, V., & Nelson, S. (2005). RxNorm: prescription for electronic drug information exchange. IT professional, 7(5), 17-23. [find](https://scholar.google.com/scholar?q=Liu%2C+S%2E%2C+Ma%2C+W%2E%2C+Moore%2C+R%2E%2C+Ganesan%2C+V%2E%2C+%26+Nelson%2C+S%2E+%282005%29%2E+RxNorm%3A+prescription+for+electronic+drug+information+exchange%2E+IT+professional%2C+7%285%29%2C+17%2D23%2E)
14. Papineni, K., Roukos, S., Ward, T., & Zhu, W. J. (2002, July). [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th annual meeting of the Association for Computational Linguistics (pp. 311-318).
15. Lin, C. Y. (2004, July). [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text summarization branches out (pp. 74-81).
16. FDA, U. (2023). FDA adverse event reporting system (FAERS) public dashboard. [find](https://scholar.google.com/scholar?q=FDA%2C+U%2E+%282023%29%2E+FDA+adverse+event+reporting+system+%28FAERS%29+public+dashboard%2E)

*Updated 1 November 2026*