---
title: "MedMCQA : A Large-scale Multi-Subject — Pal 2022"
description: "Automated answering of real medical exam questions has received little research attention."
authors: ["Ankit Pal", "Logesh Kumar Umapathi", "Malaikannan Sankarasubbu"]
year: 2022
venue: "arXiv:2203.14371 · ACM Conference on Health, Inference, and Learning (CHIL) 2022"
source: "https://arxiv.org/abs/2203.14371"
doi: "10.48550/arxiv.2203.14371"
arxiv: "2203.14371"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/pal-2022-medmcqa-large-scale-multi-subject/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pal-2022-medmcqa-large-scale-multi-subject/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/) / MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering

# MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering

**Authors:** [Ankit Pal](https://md.research.elamur.ai/free/en/authors/ankit-pal/), [Logesh Kumar Umapathi](https://md.research.elamur.ai/free/en/authors/logesh-kumar-umapathi/), [Malaikannan Sankarasubbu](https://md.research.elamur.ai/free/en/authors/malaikannan-sankarasubbu/) · **Year:** 2022 · **Venue:** arXiv:2203.14371 · ACM Conference on Health, Inference, and Learning (CHIL) 2022 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2203.14371) · [PDF](https://arxiv.org/pdf/2203.14371) · [DOI 10.48550/arxiv.2203.14371](https://doi.org/10.48550/arxiv.2203.14371) · [arXiv 2203.14371](https://arxiv.org/abs/2203.14371) · [OpenAlex](https://openalex.org/W4221154592)

**Citations:** 71 citations (openalex-snapshot · 2026-10-14) · [380](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Automated answering of real medical exam questions has received little research attention. One reason is that existing biomedical multiple-choice collections are small and cover few subjects. The authors address this with MedMCQA, a benchmark of roughly 194k four-option questions. The questions come from Indian postgraduate medical entrance exams (AIIMS PG and NEET PG) and from mock tests and online test series written by medical professionals. Each item has its correct answer, a written explanation, and a label for one of 21 subjects and about 2.4k topics. During cleaning, the authors removed items that depend on images, tables or outside information. They also fixed formatting and spelling and removed duplicates. The splits are based on the exam a question came from, not random sampling, and near-duplicate questions are kept out of the evaluation sets. The baselines are BERT-family encoders, tested with and without retrieved context passages. The best one, PubMedBERT with PubMed passages, reaches only 47% test accuracy, while top human candidates average about 90%. This gap makes MedMCQA a demanding benchmark for medical reasoning and retrieval-augmented QA.

## Key points
- The final dataset has 193,155 questions: about 183K for training (mock and online test series), 6K for development (NEET PG, 2001 onward) and 4K for testing (AIIMS PG, 1991 onward).
- To prevent leakage, any question with Levenshtein-based similarity above 0.9 to another question was excluded from the dev and test sets.
- The baselines compared are BERT Base, BioBERT, SciBERT and PubMedBERT, each in three settings: no context, Wikipedia passages and PubMed passages. PubMedBERT scored best in every setting.
- PubMed passages raised accuracy by about 4–7 points, while Wikipedia context gave almost no gain. This points to a need for domain-specific knowledge sources.
- The authors manually checked about 100 wrong predictions. The main causes were questions needing multi-hop reasoning (diagnosis and treatment), poor retrieved passages and arithmetic. The data and a leaderboard are at medmcqa.github.io.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

380 papers of our library cite it (of the 140884 whose bibliographies we read); 19 of them have a page here, 361 not yet.

- [MedAraBench: Large-Scale Arabic Medical Question Answering Dataset and Benchmark](https://md.research.elamur.ai/free/en/papers/abu-2026-daoud-medarabench-large-scale-arabic-medical/)
- [Task-aware Retrieval with Instructions](https://md.research.elamur.ai/free/en/papers/asai-2023-task-aware-retrieval-instructions/)
- [MAWARITH: A Dataset and Benchmark for Legal Inheritance Reasoning with LLMs](https://md.research.elamur.ai/free/en/papers/bouchekif-2026-mawarith-dataset-benchmark-legal-inheritance/)
- [MEDITRON-70B: Scaling Medical Pretraining for Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2023-meditron-70b-scaling-medical-pretraining/)
- [MedArabiQ: Benchmarking Large Language Models on Arabic Medical Tasks](https://md.research.elamur.ai/free/en/papers/daoud-2025-medarabiq-benchmarking-large-language-models/)
- [Olmo 3](https://md.research.elamur.ai/free/en/papers/ettinger-2025-olmo-3/)
- [Towards Mitigating LLM Hallucination via Self Reflection](https://md.research.elamur.ai/free/en/papers/ji-2023-mitigating-llm-hallucination-self-reflection/)
- [BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains](https://md.research.elamur.ai/free/en/papers/labrak-2024-biomistral-collection-open-source-pretrained/)
- [ExpertQA: Expert-Curated Questions and Attributed Answers](https://md.research.elamur.ai/free/en/papers/malaviya-2024-expertqa-expert-curated-questions-attributed/)
- [Generative Representational Instruction Tuning](https://md.research.elamur.ai/free/en/papers/muennighoff-2024-generative-representational-instruction-tuning/)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://md.research.elamur.ai/free/en/papers/nori-2023-capabilities-gpt-4-medical-challenge/)
- [Med-HALT: Medical Domain Hallucination Test for Large Language Models](https://md.research.elamur.ai/free/en/papers/pal-2023-med-halt-medical-domain-hallucination/)
- [Leveraging Large Language Models for Multiple Choice Question Answering](https://md.research.elamur.ai/free/en/papers/robinson-2022-leveraging-large-language-models-multiple/)
- [Bridging Language Barriers in Healthcare: A Study on Arabic LLMs](https://md.research.elamur.ai/free/en/papers/saadi-2025-bridging-language-barriers-healthcare-study/)
- [One Embedder, Any Task: Instruction-Finetuned Text Embeddings](https://md.research.elamur.ai/free/en/papers/su-2022-one-embedder-any-task-instruction/)
- [MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning](https://md.research.elamur.ai/free/en/papers/tang-2024-medagents-large-language-models-as/)
- [PMC-LLaMA: Towards Building Open-source Language Models for Medicine](https://md.research.elamur.ai/free/en/papers/wu-2023-pmc-llama-towards-building-open/)
- [Benchmarking Retrieval-Augmented Generation for Medicine](https://md.research.elamur.ai/free/en/papers/xiong-2024-benchmarking-retrieval-augmented-generation-medicine/)
- [Baichuan 2: Open Large-scale Language Models](https://md.research.elamur.ai/free/en/papers/yang-2023-baichuan-2-open-large-scale/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{pal2022medmcqa,
  title = {{MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering}},
  author = {Pal, Ankit and Umapathi, Logesh Kumar and Sankarasubbu, Malaikannan},
  year = {2022},
  doi = {10.48550/arxiv.2203.14371},
  eprint = {2203.14371},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2203.14371}
}
```

- [← previous MAVEN-ERE: A Unified Large-scale Dataset for Event Coreference, Temporal, Causal, and Subevent Relation Extraction](https://md.research.elamur.ai/free/en/papers/wang-2022-maven-ere-unified-large-scale/)
- [next → Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models](https://md.research.elamur.ai/free/en/papers/tirumala-2022-memorization-overfitting-analyzing-training-dynamics/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/)

## References

13 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Asma Ben Abacha, Chaitanya Shivade, and Dina Yuxian Gu, Robert Tinn, Hao Cheng, Michael R. Demner-Fushman. Overview of the mediqa 2019 Lucas, Naoto Usuyama, Xiaodong Liu, Tris-shared task on textual inference, question entail-tan Naumann, Jianfeng Gao, and Hoifung Poon. [find](https://scholar.google.com/scholar?q=Asma+Ben+Abacha%2C+Chaitanya+Shivade%2C+and+Dina+Yuxian+Gu%2C+Robert+Tinn%2C+Hao+Cheng%2C+Michael+R%2E+Demner%2DFushman%2E+Overview+of+the+mediqa+2019+Lucas%2C+Naoto+Usuyama%2C+Xiaodong+Liu%2C+Tris%2Dshared+task+on+textual+i)
2. Domain-specific language model pretraining for biomedical natural language processing. ArXiv, abs/2007.15779, 2022. [find](https://scholar.google.com/scholar?q=Domain%2Dspecific+language+model+pretraining+for+biomedical+natural+language+processing%2E+ArXiv%2C+abs%2F2007%2E15779%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/gu-2020-domain-specific-language-model-pretraining/)
3. Jaewoo Kang. Biobert: A pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240, February 2020. ISSN 1367-4803. doi: 10.1093/ bioinformatics/btz682. [find](https://scholar.google.com/scholar?q=Jaewoo+Kang%2E+Biobert%3A+A+pre%2Dtrained+biomedical+language+representation+model+for+biomedical+text+mining%2E+Bioinformatics%2C+36%284%29%3A1234%E2%80%931240%2C+February+2020%2E+ISSN+1367%2D4803%2E+doi%3A+10%2E1093%2F+bioinformatics%2Fbt) [our card](https://md.research.elamur.ai/free/en/papers/lee-2019-biobert-pre-trained-biomedical-language/)
4. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines Timo Möller, Anthony Reina, Raghavan Jayakumar, and Malte Pietsch. Covid-qa: A to read and comprehend. NIPS’15: Proceedings of question answering dataset for… [find](https://scholar.google.com/scholar?q=Karl+Moritz+Hermann%2C+Toma%CC%81s%CC%8C+Koc%CC%8Cisky%CC%81%2C+Edward+Grefenstette%2C+Lasse+Espeholt%2C+Will+Kay%2C+Mustafa+Suleyman%2C+and+Phil+Blunsom%2E+Teaching+machines+Timo+Mo%CC%88ller%2C+Anthony+Reina%2C+Raghavan+Jayakumar%2C+and+Malte+)
5. Siva Reddy, Danqi Chen, and Christopher D. Man-Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red-ning. Coqa: A conversational question answer-field, Michael Collins, Ankur Parikh, Chris Al-ing challenge. Transactions of the Association for berti, Danielle Epstein, Illia Polosukhin, Matthew Computatio… [find](https://scholar.google.com/scholar?q=Siva+Reddy%2C+Danqi+Chen%2C+and+Christopher+D%2E+Man%2DTom+Kwiatkowski%2C+Jennimaria+Palomaki%2C+Olivia+Red%2Dning%2E+Coqa%3A+A+conversational+question+answer%2Dfield%2C+Michael+Collins%2C+Ankur+Parikh%2C+Chris+Al%2Ding+challeng)
6. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz and Eduard Hovy. Race: Large-scale reading com-Kaiser, and Illia Polosukhin. Attention is all prehension dataset from examinations. Proceedings you need. I… [find](https://scholar.google.com/scholar?q=Guokun+Lai%2C+Qizhe+Xie%2C+Hanxiao+Liu%2C+Yiming+Yang%2C+Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+L+ukasz+and+Eduard+Hovy%2E+Race%3A+Large%2Dscale+reading+com%2DKaiser%2C+)
7. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Curran Associates, Inc., 2017. URL https: Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Xiao Zhang, Ji Wu, Zhiyang He, Xien Liu, and Ying Su. Medical exam question answering with large-scale reading comprehension. In Proceedings of the AAAI Conference on Artifici… [find](https://scholar.google.com/scholar?q=Jinhyuk+Lee%2C+Wonjin+Yoon%2C+Sungdong+Kim%2C+Curran+Associates%2C+Inc%2E%2C+2017%2E+URL+https%3A+Donghyeon+Kim%2C+Sunkyu+Kim%2C+Chan+Ho+So%2C+and+Xiao+Zhang%2C+Ji+Wu%2C+Zhiyang+He%2C+Xien+Liu%2C+and+Ying+Su%2E+Medical+exam+question)
8. David Vilares and Carlos Gomez-Rodr. Head-qa: A healthcare dataset for complex reasoning. In HEAD-QA: A Healthcare Dataset for Complex Reasoning, pages 960–966. Association for Computational Linguistics, 7 2019. doi: 10.18653/ v1/P19-1092. URL [https://www.aclweb.org/](https://www.aclweb.org/) anthology/P19-1092. [↗](https://www.aclweb.org/)
9. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Simon Šuster and Walter Daelemans. Clicr: A dataset of clinical case reports for machine reading Chaumond, Clement Delangue, Anthony Moi, comprehension. Proceedings of the 2018 Conference Pierric Cistac, Tim Rault, Remi Louf, Morgan of the North Amer… [find](https://scholar.google.com/scholar?q=Thomas+Wolf%2C+Lysandre+Debut%2C+Victor+Sanh%2C+Julien+Simon+S%CC%8Custer+and+Walter+Daelemans%2E+Clicr%3A+A+dataset+of+clinical+case+reports+for+machine+reading+Chaumond%2C+Clement+Delangue%2C+Anthony+Moi%2C+comprehensio) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
10. Semih Yagcioglu, Aykut Erdem, Erkut Erdem, and Nazli Ikizler-Cinbis. Recipeqa: A challenge dataset for multimodal comprehension of cooking recipes. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018. [find](https://scholar.google.com/scholar?q=Semih+Yagcioglu%2C+Aykut+Erdem%2C+Erkut+Erdem%2C+and+Nazli+Ikizler%2DCinbis%2E+Recipeqa%3A+A+challenge+dataset+for+multimodal+comprehension+of+cooking+recipes%2E+Proceedings+of+the+2018+Conference+on+Empirical+Meth)
11. Yi Yang, Wen tau Yih, and Christopher Meek. Wikiqa: A challenge dataset for open-domain question answering. In WikiQA: A Challenge Dataset for Open-Domain Question Answering, pages 2013– 2018. Association for Computational Linguistics, 9 2015. doi: 10.18653/v1/D15-1237. URL [https://www.aclweb.org/a…](https://www.aclweb.org/a…) [↗](https://www.aclweb.org/a…) [our card](https://md.research.elamur.ai/free/en/papers/yang-2015-wikiqa-challenge-dataset-open-domain/)
12. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Peng+Qi%2C+Saizheng+Zhang%2C+Yoshua+Bengio%2C+William+W+Cohen%2C+Ruslan+Salakhutdinov%2C+and+Christopher+D+Manning%2E+Hotpotqa%3A+A+dataset+for+diverse%2C+explainable+multi%2Dhop+question+answering%2E+Procee) [our card](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/)
13. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. Swag: A large-scale adversarial dataset for grounded commonsense inference. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018. *Figure 8: Distribution of topics per subject & Cumulative* Frequency G… [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Yonatan+Bisk%2C+Roy+Schwartz%2C+and+Yejin+Choi%2E+Swag%3A+A+large%2Dscale+adversarial+dataset+for+grounded+commonsense+inference%2E+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natura)