---
title: "PubMedQA: A Dataset for Biomedical Research — Jin 2019"
description: "Biomedical QA resources have been either small and expert-labeled (BioASQ has under 3k training items) or large but built automatically, with mostly…"
authors: ["Qiao Jin", "Bhuwan Dhingra", "Zhengping Liu", "William Cohen", "Xinghua Lu"]
year: 2019
venue: "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)"
source: "https://aclanthology.org/D19-1259/"
doi: "10.18653/v1/d19-1259"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/jin-2019-pubmedqa-dataset-biomedical-research-question/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jin-2019-pubmedqa-dataset-biomedical-research-question/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/) / PubMedQA: A Dataset for Biomedical Research Question Answering

# PubMedQA: A Dataset for Biomedical Research Question Answering

**Authors:** [Qiao Jin](https://md.research.elamur.ai/free/en/authors/qiao-jin/), [Bhuwan Dhingra](https://md.research.elamur.ai/free/en/authors/bhuwan-dhingra/), Zhengping Liu, [William Cohen](https://md.research.elamur.ai/free/en/authors/william-cohen/), Xinghua Lu · **Year:** 2019 · **Venue:** Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D19-1259/) · [PDF](https://aclanthology.org/D19-1259.pdf) · [DOI 10.18653/v1/d19-1259](https://doi.org/10.18653/v1/d19-1259) · [OpenAlex](https://openalex.org/W2970482702)

**Citations:** 657 citations (openalex-snapshot · 2026-10-14) · [495](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Biomedical QA resources have been either small and expert-labeled (BioASQ has under 3k training items) or large but built automatically, with mostly factoid questions that need little reasoning. The authors use a common pattern in PubMed: many articles have a question as their title and a structured abstract whose conclusion answers it. From these articles they build PubMedQA. A system reads the abstract with its conclusion removed and must answer yes, no or maybe. The removed conclusion serves as a "long answer" that can provide extra supervision. The resource has three parts: 1k items labeled by two M.D. candidates, 61.2k unlabeled items, and 211.3k artificial items made by turning declarative titles into questions with heuristic labels. As a baseline, the authors fine-tune BioBERT in several phases: first on the artificial data, then on pseudo-labeled unlabeled data, then on the labeled set. They also add an auxiliary objective that predicts bag-of-words statistics of the conclusion. The best system reaches 68.1% accuracy, compared with 78.0% for a single human and 55.2% for the majority class. The benchmark tests reasoning over quantitative findings, and it is still far from solved.

## Key points
- **Subsets:** PQA-L has 1k expert-labeled items (55.2% yes, 33.8% no, 11.0% maybe), split into 500 for 10-fold cross-validation and 500 for testing. PQA-U has 61.2k unlabeled items. PQA-A has 211.3k auto-labeled items (92.8% yes, 7.2% no).
- **Source:** About 760k PubMed articles have question titles, and about 120k of those have structured abstracts. More than half of such titles can be answered with yes/no/maybe, compared with about 1% in Natural Questions and 6% in HotpotQA.
- **PQA-A construction:** Statement titles with an NP-(VBP/VBZ) structure (found with Stanford CoreNLP) are rewritten as questions and labeled by whether they contain a negation. For PQA-U, a rule-based filter that picks out yes/no/maybe questions agrees with an expert annotator over 93% of the time.
- **Results (reasoning-required setting):** Multi-phase BioBERT with long-answer supervision gets 68.1% accuracy. A single human gets 78.0%, and the majority baseline gets 55.2%. Other baselines include shallow features, BiLSTM and ESIM with BioELMo.
- **Open problems:** About 21% of contexts give numbers without any text describing them. The authors suggest generating the long answer as a richer auxiliary task. The data is available at https://pubmedqa.github.io.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/)

## Cited by

Cited by

495 papers of our library cite it (of the 140884 whose bibliographies we read); 32 of them have a page here, 463 not yet.

- [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://md.research.elamur.ai/free/en/papers/gu-2020-domain-specific-language-model-pretraining/)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://md.research.elamur.ai/free/en/papers/chen-2024-m3-embedding-multi-linguality/)
- [Fact or Fiction: Verifying Scientific Claims](https://md.research.elamur.ai/free/en/papers/wadden-2020-fact-or-fiction-verifying-scientific/)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://md.research.elamur.ai/free/en/papers/nori-2023-capabilities-gpt-4-medical-challenge/)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://md.research.elamur.ai/free/en/papers/longpre-2023-flan-collection-designing-data-methods/)
- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://md.research.elamur.ai/free/en/papers/mitchell-2023-detectgpt-zero-shot-machine-generated/)
- [LinkBERT: Pretraining Language Models with Document Links](https://md.research.elamur.ai/free/en/papers/yasunaga-2022-linkbert-pretraining-language-models-document/)
- [Benchmarking Retrieval-Augmented Generation for Medicine](https://md.research.elamur.ai/free/en/papers/xiong-2024-benchmarking-retrieval-augmented-generation-medicine/)
- [BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains](https://md.research.elamur.ai/free/en/papers/labrak-2024-biomistral-collection-open-source-pretrained/)
- [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://md.research.elamur.ai/free/en/papers/dasigi-2021-dataset-information-seeking-questions-answers/)
- [Towards Mitigating LLM Hallucination via Self Reflection](https://md.research.elamur.ai/free/en/papers/ji-2023-mitigating-llm-hallucination-self-reflection/)
- [MEDITRON-70B: Scaling Medical Pretraining for Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2023-meditron-70b-scaling-medical-pretraining/)
- [Can AI-Generated Text be Reliably Detected?](https://md.research.elamur.ai/free/en/papers/sadasivan-2023-ai-generated-text-reliably-detected/)
- [RedPajama: an Open Dataset for Training Large Language Models](https://md.research.elamur.ai/free/en/papers/weber-2024-redpajama-open-dataset-training-large/)
- [MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning](https://md.research.elamur.ai/free/en/papers/tang-2024-medagents-large-language-models-as/)
- [Evaluating Large Language Models: A Comprehensive Survey](https://md.research.elamur.ai/free/en/papers/guo-2023-evaluating-large-language-models-comprehensive/)
- [Generative Representational Instruction Tuning](https://md.research.elamur.ai/free/en/papers/muennighoff-2024-generative-representational-instruction-tuning/)
- [DataComp-LM: In search of the next generation of training sets for language models](https://md.research.elamur.ai/free/en/papers/li-2024-datacomp-lm-search-next-generation/)
- [Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature](https://md.research.elamur.ai/free/en/papers/bao-2023-fast-detectgpt-efficient-zero-shot/)
- [RA-DIT: Retrieval-Augmented Dual Instruction Tuning](https://md.research.elamur.ai/free/en/papers/lin-2023-ra-dit-retrieval-augmented-dual/)
- [PMC-LLaMA: Towards Building Open-source Language Models for Medicine](https://md.research.elamur.ai/free/en/papers/wu-2023-pmc-llama-towards-building-open/)
- [RAFT: Adapting Language Model to Domain Specific RAG](https://md.research.elamur.ai/free/en/papers/zhang-2024-raft-adapting-language-model-domain/)
- [Task-aware Retrieval with Instructions](https://md.research.elamur.ai/free/en/papers/asai-2023-task-aware-retrieval-instructions/)
- [Knowledge Unlearning for Mitigating Privacy Risks in Language Models](https://md.research.elamur.ai/free/en/papers/jang-2022-knowledge-unlearning-mitigating-privacy-risks/)
- [HealthBench: Evaluating Large Language Models Towards Improved Human Health](https://md.research.elamur.ai/free/en/papers/arora-2025-healthbench-evaluating-large-language-models/)
- [ExpertQA: Expert-Curated Questions and Attributed Answers](https://md.research.elamur.ai/free/en/papers/malaviya-2024-expertqa-expert-curated-questions-attributed/)
- [English Machine Reading Comprehension Datasets: A Survey](https://md.research.elamur.ai/free/en/papers/dzendzik-2021-english-machine-reading-comprehension-datasets/)
- [MultiPragEval: Multilingual Pragmatic Evaluation of Large Language Models](https://md.research.elamur.ai/free/en/papers/park-2024-multiprageval-multilingual-pragmatic-evaluation-large/)
- [MedAraBench: Large-Scale Arabic Medical Question Answering Dataset and Benchmark](https://md.research.elamur.ai/free/en/papers/abu-2026-daoud-medarabench-large-scale-arabic-medical/)
- [Bridging Language Barriers in Healthcare: A Study on Arabic LLMs](https://md.research.elamur.ai/free/en/papers/saadi-2025-bridging-language-barriers-healthcare-study/)
- [AccurateRAG: A Framework for Building Accurate Retrieval-Augmented Question-Answering Applications](https://md.research.elamur.ai/free/en/papers/nguyen-2026-accuraterag-framework-building-accurate-retrieval/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{jin2019pubmedqa,
  title = {{PubMedQA: A Dataset for Biomedical Research Question Answering}},
  author = {Jin, Qiao and Dhingra, Bhuwan and Liu, Zhengping and Cohen, William and Lu, Xinghua},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1259},
  url = {https://doi.org/10.18653/v1/d19-1259}
}
```

- [← previous Generative Modeling by Estimating Gradients of the Data Distribution](https://md.research.elamur.ai/free/en/papers/song-2019-generative-modeling-estimating-gradients-data/)
- [next → The Risk of Racial Bias in Hate Speech Detection](https://md.research.elamur.ai/free/en/papers/sap-2019-risk-racial-bias-hate-speech/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/)

## References

15 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2016. Enhanced lstm for natural language inference. arXiv preprint arXiv:1609.06038. [find](https://scholar.google.com/scholar?q=Qian+Chen%2C+Xiaodan+Zhu%2C+Zhenhua+Ling%2C+Si+Wei%2C+Hui+Jiang%2C+and+Diana+Inkpen%2E+2016%2E+Enhanced+lstm+for+natural+language+inference%2E+arXiv+preprint+arXiv%3A1609%2E06038%2E)
2. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:… [find](https://scholar.google.com/scholar?q=Christopher+Clark%2C+Kenton+Lee%2C+Ming%2DWei+Chang%2C+Tom+Kwiatkowski%2C+Michael+Collins%2C+and+Kristina+Toutanova%2E+2019%2E+Boolq%3A+Exploring+the+surprising+difficulty+of+natural+yes%2Fno+questions%2E+In+Proceedings+of) [our card](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/)
3. Shuming Ma, Xu Sun, Yizhong Wang, and Junyang Lin. 2018. Bag-of-words as target for neural machine translation. arXiv preprint arXiv:1805.04871. [find](https://scholar.google.com/scholar?q=Shuming+Ma%2C+Xu+Sun%2C+Yizhong+Wang%2C+and+Junyang+Lin%2E+2018%2E+Bag%2Dof%2Dwords+as+target+for+neural+machine+translation%2E+arXiv+preprint+arXiv%3A1805%2E04871%2E)
4. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55–60. [find](https://scholar.google.com/scholar?q=Christopher+D%2E+Manning%2C+Mihai+Surdeanu%2C+John+Bauer%2C+Jenny+Finkel%2C+Steven+J%2E+Bethard%2C+and+David+Mc%2DClosky%2E+2014%2E+The+Stanford+CoreNLP+natural+language+processing+toolkit%2E+In+Association+for+Computation) [our card](https://md.research.elamur.ai/free/en/papers/manning-2014-stanford-corenlp-natural-language-processing/)
5. Roser Morante, Martin Krallinger, Alfonso Valencia, and Walter Daelemans. 2012. Machine reading of biomedical texts about alzheimers disease. In CLEF 2012 Conference and Labs of the Evaluation Forum-Question Answering For Machine Reading Evaluation (QA4MRE), Rome/Forner, J.[edit.]; ea, pages 1–14. [find](https://scholar.google.com/scholar?q=Roser+Morante%2C+Martin+Krallinger%2C+Alfonso+Valencia%2C+and+Walter+Daelemans%2E+2012%2E+Machine+reading+of+biomedical+texts+about+alzheimers+disease%2E+In+CLEF+2012+Conference+and+Labs+of+the+Evaluation+Forum%2DQ)
6. Anusri Pampari, Preethi Raghavan, Jennifer Liang, and Jian Peng. 2018. emrqa: A large corpus for question answering on electronic medical records. arXiv preprint arXiv:1809.00732. [find](https://scholar.google.com/scholar?q=Anusri+Pampari%2C+Preethi+Raghavan%2C+Jennifer+Liang%2C+and+Jian+Peng%2E+2018%2E+emrqa%3A+A+large+corpus+for+question+answering+on+electronic+medical+records%2E+arXiv+preprint+arXiv%3A1809%2E00732%2E) [our card](https://md.research.elamur.ai/free/en/papers/pampari-2018-emrqa-large-corpus-question-answering/)
7. Dimitris Pappas, Ion Androutsopoulos, and Haris Papageorgiou. 2018. Bioread: A new dataset for biomedical reading comprehension. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018). [find](https://scholar.google.com/scholar?q=Dimitris+Pappas%2C+Ion+Androutsopoulos%2C+and+Haris+Papageorgiou%2E+2018%2E+Bioread%3A+A+new+dataset+for+biomedical+reading+comprehension%2E+In+Proceedings+of+the+Eleventh+International+Conference+on+Language+Res)
8. Anselmo Peñas, Eduard Hovy, Pamela Forner, Álvaro Rodrigo, Richard Sutcliffe, and Roser Morante. 2013. Qa4mre 2011-2013: Overview of question answering for machine reading evaluation. In International Conference of the Cross-Language Evaluation Forum for European Languages, pages 303– 320. Springe… [find](https://scholar.google.com/scholar?q=Anselmo+Pen%CC%83as%2C+Eduard+Hovy%2C+Pamela+Forner%2C+A%CC%81lvaro+Rodrigo%2C+Richard+Sutcliffe%2C+and+Roser+Morante%2E+2013%2E+Qa4mre+2011%2D2013%3A+Overview+of+question+answering+for+machine+reading+evaluation%2E+In+Internation)
9. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365. [find](https://scholar.google.com/scholar?q=Matthew+E+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+2018%2E+Deep+contextualized+word+representations%2E+arXiv+preprint+arXiv%3A1802%2E05365%2E)
10. Sampo Pyysalo, Filip Ginter, Hans Moen, Tapio Salakoski, and Sophia Ananiadou. 2013. Distributional semantics resources for biomedical text processing. [find](https://scholar.google.com/scholar?q=Sampo+Pyysalo%2C+Filip+Ginter%2C+Hans+Moen%2C+Tapio+Salakoski%2C+and+Sophia+Ananiadou%2E+2013%2E+Distributional+semantics+resources+for+biomedical+text+processing%2E)
11. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+2016%2E+Squad%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+arXiv+preprint+arXiv%3A1606%2E05250%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
12. Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Rocktäschel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. 2018. Interpretation of natural language rules in conversational machine reading. arXiv preprint arXiv:1809.01494. [find](https://scholar.google.com/scholar?q=Marzieh+Saeidi%2C+Max+Bartolo%2C+Patrick+Lewis%2C+Sameer+Singh%2C+Tim+Rockta%CC%88schel%2C+Mike+Sheldon%2C+Guillaume+Bouchard%2C+and+Sebastian+Riedel%2E+2018%2E+Interpretation+of+natural+language+rules+in+conversational+mac) [our card](https://md.research.elamur.ai/free/en/papers/saeidi-2018-interpretation-natural-language-rules-conversational/)
13. Hiroaki Sakamoto, Yasunori Watanabe, and Masataka Satou. 2011. Do preoperative statins reduce atrial fibrillation after coronary artery bypass grafting? Annals of thoracic and cardiovascular surgery, 17(4):376–382. [find](https://scholar.google.com/scholar?q=Hiroaki+Sakamoto%2C+Yasunori+Watanabe%2C+and+Masataka+Satou%2E+2011%2E+Do+preoperative+statins+reduce+atrial+fibrillation+after+coronary+artery+bypass+grafting%3F+Annals+of+thoracic+and+cardiovascular+surgery%2C+)
14. George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, et al. 2015. An overview of the bioasq large-scale biomedical semantic indexing and question answeri… [find](https://scholar.google.com/scholar?q=George+Tsatsaronis%2C+Georgios+Balikas%2C+Prodromos+Malakasiotis%2C+Ioannis+Partalas%2C+Matthias+Zschunke%2C+Michael+R+Alvers%2C+Dirk+Weissenborn%2C+Anastasia+Krithara%2C+Sergios+Petridis%2C+Dimitris+Polychronopoulos%2C+)
15. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Peng+Qi%2C+Saizheng+Zhang%2C+Yoshua+Bengio%2C+William+W+Cohen%2C+Ruslan+Salakhutdinov%2C+and+Christopher+D+Manning%2E+2018%2E+Hotpotqa%3A+A+dataset+for+diverse%2C+explainable+multi%2Dhop+question+answering%2E+) [our card](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/)