---
title: "VERICITE — paper by Ma et al. (2026) · BioNLP 2026"
description: "The authors introduce VERICITE, a framework for measuring this citation faithfulness in biomedical QA. It… 99.9% recall. 25 references."
authors: ["Yixian Ma", "Bohao Chu", "Norbert Fuhr"]
year: 2026
venue: "BioNLP 2026"
source: "https://aclanthology.org/2026.bionlp-1.62/"
doi: "10.18653/v1/2026.bionlp-1.62"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ma-2026-vericite-evaluating-sentence-level-citation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ma-2026-vericite-evaluating-sentence-level-citation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/47/) / VERICITE: Evaluating Sentence-Level Citation Faithfulness in Retrieval-Augmented Medical Question Answering

# VERICITE: Evaluating Sentence-Level Citation Faithfulness in Retrieval-Augmented Medical Question Answering

**Authors:** Yixian Ma, [Bohao Chu](https://md.research.elamur.ai/free/en/authors/bohao-chu/) and [Norbert Fuhr](https://md.research.elamur.ai/free/en/authors/norbert-fuhr/) · **Year:** 2026 · **Venue:** [BioNLP 2026](https://md.research.elamur.ai/free/en/venues/bionlp/) ([BioNLP 2026](https://md.research.elamur.ai/free/en/venues/bionlp/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.bionlp-1.62/) · [PDF](https://aclanthology.org/2026.bionlp-1.62.pdf) · [DOI 10.18653/v1/2026.bionlp-1.62](https://doi.org/10.18653/v1/2026.bionlp-1.62) · [OpenAlex](https://openalex.org/W7166843029)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Retrieval-augmented generation lets LLMs attach references to their answers. However, a citation's presence says little about whether the source actually backs the statement, which is a serious risk in clinical settings. The authors introduce VERICITE, a framework for measuring this citation faithfulness in biomedical QA. It fetches PubMed abstracts through the NCBI E-utilities API and has an LLM write a short answer with inline document citations. It then checks every cited abstract sentence by sentence with a DeBERTa-v3-large NLI model. A citation counts as supported if at least one individual source sentence entails the generated claim. Four LLMs were tested on BioASQ Task 13b questions at several retrieval depths. An oracle condition supplied the gold PMIDs directly. At shallow depths, only about a quarter to two fifths of citation pairs were supported, and most were judged irrelevant. Supplying more documents lowered support further. Even gold documents improved answer quality without making citations more faithful. This suggests that the generation step, not just retrieval, drives unfaithful citing in medical RAG.

## Key points
- Setup: 500 non-yes/no BioASQ questions. 421 of them returned abstracts under standard retrieval. Models tested were Mistral-7B-Instruct-v0.3, Llama-3.1-70B-Instruct, GPT-4o-mini and Claude-3.5-Haiku. Each citation was labeled supported, contradicted or irrelevant.
- At k ∈ {3, 5}, supported citations ranged from 27.4% to 41.2%. Llama-3.1-70B scored best (41.2% at k=3) and Mistral-7B scored lowest. Contradictions made up roughly 6.5–9.6% of pairs.
- Retrieval was deliberately simple and weak (Recall@3 = 0.127, Recall@5 = 0.162). Raising k to 15 lifted recall to 0.226 but steadily cut support, to 22.7% for Mistral and 33.4% for Llama. The authors frame this as a precision–coverage trade-off.
- Oracle retrieval reached 99.9% recall. ROUGE-L rose from .171 to .212 for Mistral and from .185 to .256 for Llama. Support, however, dropped to 22.5% for Mistral and stayed at a similar level for Llama (38.1%).
- Stated limitations:
  - The NLI model is general-domain and was not trained on MedNLI.
  - No human validation was performed.
  - Single-sentence entailment likely underestimates faithfulness when models merge information across sentences.
  - The oracle comparison is not fully controlled.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Duisburg-Essen](https://md.research.elamur.ai/free/en/universities/university-of-duisburg-essen/) — University, Essen, Germany · 3 works · 48 citations

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works · [🧬 Biochemistry, Genetics and Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/) · 234 works · [🧬 Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/molecular-biology/) · 231 works

🧭 Topics (2): [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 441 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 473 works

## Cite

Cite as: Yixian Ma, Bohao Chu and Norbert Fuhr. 2026. VERICITE: Evaluating Sentence-Level Citation Faithfulness in Retrieval-Augmented Medical Question Answering. BioNLP 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ma2026vericite,
  title = {{VERICITE: Evaluating Sentence-Level Citation Faithfulness in Retrieval-Augmented Medical Question Answering}},
  author = {Yixian Ma, Bohao Chu and Norbert Fuhr},
  booktitle = {BioNLP 2026},
  year = {2026},
  doi = {10.18653/v1/2026.bionlp-1.62},
  url = {https://doi.org/10.18653/v1/2026.bionlp-1.62}
}
```

- [← previous VerbaNexAI at ClinicalSkillQA: From Visual Evidence to Procedural Order A Two-Stage Generative Vision-Language Framework for ClinSkillQA](https://md.research.elamur.ai/free/en/papers/tovar-2026-verbanexai-clinicalskillqa-visual-evidence-procedural/)
- [next → Verifying the Subjective: Structured Multilingual Rewards for Low-Resource Alignment](https://md.research.elamur.ai/free/en/papers/sha-2026-verifying-subjective-structured-multilingual-rewards/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/47/)

## References

25 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anthropic. 2024. Claude 3.5 model card. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2024%2E+Claude+3%2E5+model+card%2E)
2. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. [Self-RAG: Learning to retrieve, generate, and critique through self-reflection](https://md.research.elamur.ai/free/en/papers/asai-2023-self-rag-learning-retrieve-generate/). In International Conference on Learning Representations.
3. Jiawei Du et al. 2025. MedCite: Towards faithful and traceable citation in biomedical question answering. arXiv preprint arXiv:2501.07653. — [arXiv:2501.07653](https://arxiv.org/abs/2501.07653) [find](https://scholar.google.com/scholar?q=Jiawei+Du+et+al%2E+2025%2E+MedCite%3A+Towards+faithful+and+traceable+citation+in+biomedical+question+answering%2E+arXiv+preprint+arXiv%3A2501%2E07653%2E+%E2%80%94+%5BarXiv%3A2501%2E07653%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2501%2E07653%29)
4. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, et al. 2024. [The Llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
5. Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. [Enabling large language models to generate text with citations](https://md.research.elamur.ai/free/en/papers/gao-2023-enabling-large-language-models-generate/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6465–6488.
6. Deepak Gupta, Davis Bartels, and Dina Demner-Fushman. 2026a. BioACE: An automated framework for biomedical answer and citation evaluations. arXiv preprint arXiv:2602.04982. — [arXiv:2602.04982](https://arxiv.org/abs/2602.04982) [find](https://scholar.google.com/scholar?q=Deepak+Gupta%2C+Davis+Bartels%2C+and+Dina+Demner%2DFushman%2E+2026a%2E+BioACE%3A+An+automated+framework+for+biomedical+answer+and+citation+evaluations%2E+arXiv+preprint+arXiv%3A2602%2E04982%2E+%E2%80%94+%5BarXiv%3A2602%2E04982%5D%28https%3A)
7. Deepak Gupta, Dina Demner-Fushman, William Hersh, Steven Bedrick, and Kirk Roberts. 2024. Overview of TREC 2024 biomedical generative retrieval (BioGen) track. In Proceedings of the Thirty-Third Text REtrieval Conference (TREC 2024). ArXiv:2411.18069. — [arXiv:2411.18069](https://arxiv.org/abs/2411.18069) [find](https://scholar.google.com/scholar?q=Deepak+Gupta%2C+Dina+Demner%2DFushman%2C+William+Hersh%2C+Steven+Bedrick%2C+and+Kirk+Roberts%2E+2024%2E+Overview+of+TREC+2024+biomedical+generative+retrieval+%28BioGen%29+track%2E+In+Proceedings+of+the+Thirty%2DThird+Text+)
8. Deepak Gupta, Dina Demner-Fushman, William Hersh, Steven Bedrick, and Kirk Roberts. 2026b. Overview of TREC 2025 biomedical generative retrieval (Bio-Gen) track. arXiv preprint arXiv:2603.21582. — [arXiv:2603.21582](https://arxiv.org/abs/2603.21582) [find](https://scholar.google.com/scholar?q=Deepak+Gupta%2C+Dina+Demner%2DFushman%2C+William+Hersh%2C+Steven+Bedrick%2C+and+Kirk+Roberts%2E+2026b%2E+Overview+of+TREC+2025+biomedical+generative+retrieval+%28Bio%2DGen%29+track%2E+arXiv+preprint+arXiv%3A2603%2E21582%2E+%E2%80%94+%5Bar)
9. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. [DeBERTaV3: Improving DeBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing](https://md.research.elamur.ai/free/en/papers/he-2021-debertav3-improving-deberta-using-electra/). In International Conference on Learning Representations. ArXiv:2111.09543.
10. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+2023%2E+Mist)
11. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. [Retrieval-augmented generation for knowledge-intensive NLP tasks](https://md.research.elamur.ai/free/en/papers/lewis-2020-retrieval-augmented-generation-knowledge-intensive/). In Advances in Neural Information Pr…
12. Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR), pages 22–32. [find](https://scholar.google.com/scholar?q=Conference+on+Innovative+Concepts+and+Theories+in+Information+Retrieval+%28ICTIR%29%2C+pages+22%E2%80%9332%2E)
13. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out, pages 74–81.
14. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. [On faithfulness and factuality in abstractive summarization](https://md.research.elamur.ai/free/en/papers/maynez-2020-faithfulness-factuality-abstractive-summarization/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919.
15. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. [Adversarial NLI: A new benchmark for natural language understanding](https://md.research.elamur.ai/free/en/papers/nie-2020-adversarial-nli-new-benchmark-natural/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885–4901.
16. OpenAI. 2024. [GPT-4o system card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/).
17. Alexey Romanov and Chaitanya Shivade. 2018. [Lessons from natural language inference in the clinical domain](https://md.research.elamur.ai/free/en/papers/romanov-2018-lessons-natural-language-inference-clinical/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1586–1596.
18. David L Sackett, William MC Rosenberg, JA Muir Gray, R Brian Haynes, and W Scott Richardson. 1996. Evidence based medicine: What it is and what it isn’t. BMJ, 312(7023):71–72. [find](https://scholar.google.com/scholar?q=David+L+Sackett%2C+William+MC+Rosenberg%2C+JA+Muir+Gray%2C+R+Brian+Haynes%2C+and+W+Scott+Richardson%2E+1996%2E+Evidence+based+medicine%3A+What+it+is+and+what+it+isn%E2%80%99t%2E+BMJ%2C+312%287023%29%3A71%E2%80%9372%2E)
19. Eric W Sayers, Evan E Bolton, J Rodney Brister, Kathi Canese, Jessica Chan, Donald C Comeau, et al. 2022. Database resources of the national center for biotechnology information. Nucleic Acids Research, 50(D1):D206–D214. [find](https://scholar.google.com/scholar?q=Eric+W+Sayers%2C+Evan+E+Bolton%2C+J+Rodney+Brister%2C+Kathi+Canese%2C+Jessica+Chan%2C+Donald+C+Comeau%2C+et+al%2E+2022%2E+Database+resources+of+the+national+center+for+biotechnology+information%2E+Nucleic+Acids+Researc)
20. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. [FEVER: a large-scale dataset for fact extraction and VERification](https://md.research.elamur.ai/free/en/papers/thorne-2018-fever-large-scale-dataset-fact/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pa…
21. George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, et al. 2015. An overview of the BIOASQ large-scale biomedical semantic indexing and question answeri… [find](https://scholar.google.com/scholar?q=George+Tsatsaronis%2C+Georgios+Balikas%2C+Prodromos+Malakasiotis%2C+Ioannis+Partalas%2C+Matthias+Zschunke%2C+Michael+R+Alvers%2C+Dirk+Weissenborn%2C+Anastasia+Krithara%2C+Sergios+Petridis%2C+Dimitris+Polychronopoulos%2C+)
22. Juraj Vladika et al. 2025. Source-checkup: A framework for evaluating source attribution in LLM-generated medical responses. arXiv preprint arXiv:2501.09940. — [arXiv:2501.09940](https://arxiv.org/abs/2501.09940) [find](https://scholar.google.com/scholar?q=Juraj+Vladika+et+al%2E+2025%2E+Source%2Dcheckup%3A+A+framework+for+evaluating+source+attribution+in+LLM%2Dgenerated+medical+responses%2E+arXiv+preprint+arXiv%3A2501%2E09940%2E+%E2%80%94+%5BarXiv%3A2501%2E09940%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs)
23. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. [A broad-coverage challenge corpus for sentence understanding through inference](https://md.research.elamur.ai/free/en/papers/williams-2018-broad-coverage-challenge-corpus-sentence/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1112–1122.
24. Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. 2024. [Benchmarking retrieval-augmented generation for medicine](https://md.research.elamur.ai/free/en/papers/xiong-2024-benchmarking-retrieval-augmented-generation-medicine/). arXiv preprint arXiv:2402.13178.
25. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. BERTscore: Evaluating text generation with BERT. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+BERTscore%3A+Evaluating+text+generation+with+BERT%2E+In+International+Conference+on+Learning+Representations%2E)

*Updated 1 November 2026*