---
title: "What Disease does this Patient Have? A — Jin 2020"
description: "Most open-domain QA benchmarks have one of two weaknesses."
authors: ["Di Jin", "Eileen Pan", "Nassim Oufattole", "Wei-Hung Weng", "Hanyi Fang", "Peter Szolovits"]
year: 2020
venue: "arXiv:2009.13081"
source: "https://arxiv.org/abs/2009.13081"
doi: "10.48550/arxiv.2009.13081"
arxiv: "2009.13081"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/jin-2020-disease-this-patient-have-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jin-2020-disease-this-patient-have-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/6/) / What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams

# What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams

**Authors:** [Di Jin](https://md.research.elamur.ai/free/en/authors/di-jin/), Eileen Pan, Nassim Oufattole, [Wei-Hung Weng](https://md.research.elamur.ai/free/en/authors/wei-hung-weng/), Hanyi Fang, [Peter Szolovits](https://md.research.elamur.ai/free/en/authors/peter-szolovits/) · **Year:** 2020 · **Venue:** arXiv:2009.13081 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2009.13081) · [PDF](https://arxiv.org/pdf/2009.13081) · [DOI 10.48550/arxiv.2009.13081](https://doi.org/10.48550/arxiv.2009.13081) · [arXiv 2009.13081](https://arxiv.org/abs/2009.13081) · [OpenAlex](https://openalex.org/W3088056511)

**Citations:** 63 citations (openalex-snapshot · 2026-10-14) · [477](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Most open-domain QA benchmarks have one of two weaknesses. Some ask for answer spans that surface word matching can locate. Others test only school-level knowledge. Jin et al. address this with MedQA, a free-form multiple-choice OpenQA dataset taken from professional medical licensing exams in the US, Mainland China and Taiwan. It covers English, simplified Chinese and traditional Chinese. With the questions, the authors release a large corpus of medical textbook text that systems must search for supporting knowledge. Their baselines follow the standard two-stage design: a document retriever gathers evidence passages, then a rule-based or neural reader, including BERT-style pre-trained models, picks an option. Even the strongest system performs poorly. It reaches 36.7% test accuracy on English, 42.0% on traditional Chinese and 70.1% on simplified Chinese. Manual analysis shows retrieval is the main bottleneck, especially for questions that need two reasoning steps, such as diagnosing first and then choosing a treatment. MedQA is therefore a hard, cross-lingual testbed for reasoning that depends on domain knowledge.

## Key points
- Size: 12,723 English (USMLE), 34,251 simplified Chinese (MCMLE) and 14,123 traditional Chinese (TWMLE) questions, 61,097 in total, all manually annotated. Data and code are at github.com/jind11/MedQA.
- The authors compare it with LiveQA, Medication QA, BioASQ, emrQA, MEDIQA and MedQuAD and present it as the first open-domain medical QA set in multiple-choice format.
- Evidence check on 100 development samples using the top 25 retrieved paragraphs: no evidence was found for 68% of USMLE questions, compared with 4% for MCMLE and 23.3% for TWMLE.
- On USMLE, neural readers score below a plain IR baseline because the retrieved evidence is poor. Almost all USMLE items are clinical case questions.
- Retrieval tends to succeed when distinctive symptoms point to a single diagnosis. It fails when symptoms are common to many conditions or the question needs a second inference step, such as treatment, tests or mechanism.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

477 papers of our library cite it (of the 140884 whose bibliographies we read); 20 of them have a page here, 457 not yet.

- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-3-technical-report-highly/)
- [MedAraBench: Large-Scale Arabic Medical Question Answering Dataset and Benchmark](https://md.research.elamur.ai/free/en/papers/abu-2026-daoud-medarabench-large-scale-arabic-medical/)
- [MEDITRON-70B: Scaling Medical Pretraining for Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2023-meditron-70b-scaling-medical-pretraining/)
- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2025-towards-reasoning-era-survey-long/)
- [MedArabiQ: Benchmarking Large Language Models on Arabic Medical Tasks](https://md.research.elamur.ai/free/en/papers/daoud-2025-medarabiq-benchmarking-large-language-models/)
- [Olmo 3](https://md.research.elamur.ai/free/en/papers/ettinger-2025-olmo-3/)
- [MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data](https://md.research.elamur.ai/free/en/papers/han-2023-medalpaca-open-source-collection-medical/)
- [BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains](https://md.research.elamur.ai/free/en/papers/labrak-2024-biomistral-collection-open-source-pretrained/)
- [ExpertQA: Expert-Curated Questions and Attributed Answers](https://md.research.elamur.ai/free/en/papers/malaviya-2024-expertqa-expert-curated-questions-attributed/)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://md.research.elamur.ai/free/en/papers/nori-2023-capabilities-gpt-4-medical-challenge/)
- [Med-HALT: Medical Domain Hallucination Test for Large Language Models](https://md.research.elamur.ai/free/en/papers/pal-2023-med-halt-medical-domain-hallucination/)
- [QA-KGNet: 一种语言模型驱动的知识图谱问答模型](https://md.research.elamur.ai/free/en/papers/qiao-2023-qa-kgnet-language/)
- [Bridging Language Barriers in Healthcare: A Study on Arabic LLMs](https://md.research.elamur.ai/free/en/papers/saadi-2025-bridging-language-barriers-healthcare-study/)
- [Large Language Models Encode Clinical Knowledge](https://md.research.elamur.ai/free/en/papers/singhal-2022-large-language-models-encode-clinical/)
- [MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning](https://md.research.elamur.ai/free/en/papers/tang-2024-medagents-large-language-models-as/)
- [PMC-LLaMA: Towards Building Open-source Language Models for Medicine](https://md.research.elamur.ai/free/en/papers/wu-2023-pmc-llama-towards-building-open/)
- [Benchmarking Retrieval-Augmented Generation for Medicine](https://md.research.elamur.ai/free/en/papers/xiong-2024-benchmarking-retrieval-augmented-generation-medicine/)
- [Baichuan 2: Open Large-scale Language Models](https://md.research.elamur.ai/free/en/papers/yang-2023-baichuan-2-open-large-scale/)
- [LinkBERT: Pretraining Language Models with Document Links](https://md.research.elamur.ai/free/en/papers/yasunaga-2022-linkbert-pretraining-language-models-document/)
- [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{jin2020what,
  title = {{What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams}},
  author = {Jin, Di and Pan, Eileen and Oufattole, Nassim and Weng, Wei-Hung and Fang, Hanyi and Szolovits, Peter},
  year = {2020},
  doi = {10.48550/arxiv.2009.13081},
  eprint = {2009.13081},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2009.13081}
}
```

- [← previous What determines the order of adjectives in English? Comparing efficiency-based theories using dependency treebanks](https://md.research.elamur.ai/free/en/papers/bank-2020-futrell-what-determines-order-adjectives-english/)
- [next → What Happens To BERT Embeddings During Fine-tuning?](https://md.research.elamur.ai/free/en/papers/merchant-2020-what-happens-bert-embeddings-during/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/6/)

## References

39 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abacha, A. B.; Agichtein, E.; Pinter, Y.; and Demner-Fushman, D. 2017. Overview of the Medical Question Answering Task at TREC 2017 LiveQA. In TREC. [find](https://scholar.google.com/scholar?q=Abacha%2C+A%2E+B%2E%3B+Agichtein%2C+E%2E%3B+Pinter%2C+Y%2E%3B+and+Demner%2DFushman%2C+D%2E+2017%2E+Overview+of+the+Medical+Question+Answering+Task+at+TREC+2017+LiveQA%2E+In+TREC%2E)
2. Abacha, A. B.; and Demner-Fushman, D. 2019. A question-entailment approach to question answering. BMC bioinformatics 20(1): 511. [find](https://scholar.google.com/scholar?q=Abacha%2C+A%2E+B%2E%3B+and+Demner%2DFushman%2C+D%2E+2019%2E+A+question%2Dentailment+approach+to+question+answering%2E+BMC+bioinformatics+20%281%29%3A+511%2E)
3. Abacha, A. B.; Mrabet, Y.; Sharp, M.; Goodwin, T. R.; Shooshan, S. E.; and Demner-Fushman, D. 2019. Bridging the Gap Between Consumers’ Medication Questions and Trusted Answers. In MedInfo, 25–29. [find](https://scholar.google.com/scholar?q=Abacha%2C+A%2E+B%2E%3B+Mrabet%2C+Y%2E%3B+Sharp%2C+M%2E%3B+Goodwin%2C+T%2E+R%2E%3B+Shooshan%2C+S%2E+E%2E%3B+and+Demner%2DFushman%2C+D%2E+2019%2E+Bridging+the+Gap+Between+Consumers%E2%80%99+Medication+Questions+and+Trusted+Answers%2E+In+MedInfo%2C+25%E2%80%9329%2E)
4. Alsentzer, E.; Murphy, J.; Boag, W.; Weng, W.-H.; Jin, D.; Naumann, T.; and McDermott, M. 2019. Publicly Available Clinical BERT Embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, 72– 78. Minneapolis, Minnesota, USA: Association for Computational Linguistics. doi:10… [find](https://scholar.google.com/scholar?q=Alsentzer%2C+E%2E%3B+Murphy%2C+J%2E%3B+Boag%2C+W%2E%3B+Weng%2C+W%2E%2DH%2E%3B+Jin%2C+D%2E%3B+Naumann%2C+T%2E%3B+and+McDermott%2C+M%2E+2019%2E+Publicly+Available+Clinical+BERT+Embeddings%2E+In+Proceedings+of+the+2nd+Clinical+Natural+Language+Process) [our card](https://md.research.elamur.ai/free/en/papers/asai-2019-learning-retrieve-reasoning-paths-over/)
5. Ben Abacha, A.; Shivade, C.; and Demner-Fushman, D. 2019. Overview of the MEDIQA 2019 Shared Task on Textual Inference, Question Entailment and Question Answering. In Proceedings of the 18th BioNLP Workshop and Shared Task, 370–379. Florence, Italy: Association for Computational Linguistics. doi:10.… [find](https://scholar.google.com/scholar?q=Ben+Abacha%2C+A%2E%3B+Shivade%2C+C%2E%3B+and+Demner%2DFushman%2C+D%2E+2019%2E+Overview+of+the+MEDIQA+2019+Shared+Task+on+Textual+Inference%2C+Question+Entailment+and+Question+Answering%2E+In+Proceedings+of+the+18th+BioNLP+Wo)
6. Bojanowski, P.; Grave, E.; Joulin, A.; and Mikolov, T. 2017. Enriching Word Vectors with Subword Information. Transactions of the Association for Computational Linguistics 5: 135–146. doi:10.1162/tacl a 00051. URL [https://www.aclweb.org/anthology/Q17-1010](https://www.aclweb.org/anthology/Q17-1010) . [↗](https://www.aclweb.org/anthology/Q17-1010)
7. Chen, D.; Fisch, A.; Weston, J.; and Bordes, A. 2017. Reading Wikipedia to Answer Open-Domain Questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1870– 1879. Vancouver, Canada: Association for Computational Linguistics. doi:1… [find](https://scholar.google.com/scholar?q=Chen%2C+D%2E%3B+Fisch%2C+A%2E%3B+Weston%2C+J%2E%3B+and+Bordes%2C+A%2E+2017%2E+Reading+Wikipedia+to+Answer+Open%2DDomain+Questions%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Vol) [our card](https://md.research.elamur.ai/free/en/papers/chen-2017-reading-wikipedia-answer-open-domain/)
8. Clark, C.; and Gardner, M. 2018. Simple and Effective Multi-Paragraph Reading Comprehension. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 845–855. Melbourne, Australia: Association for Computational Linguistics. doi:10.18653/v1/P… [find](https://scholar.google.com/scholar?q=Clark%2C+C%2E%3B+and+Gardner%2C+M%2E+2018%2E+Simple+and+Effective+Multi%2DParagraph+Reading+Comprehension%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+)
9. Clark, P.; Etzioni, O.; Khot, T.; Sabharwal, A.; Tafjord, O.; Turney, P.; and Khashabi, D. 2016. Combining retrieval, statistics, and inference to answer elementary science questions. In Thirtieth AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Clark%2C+P%2E%3B+Etzioni%2C+O%2E%3B+Khot%2C+T%2E%3B+Sabharwal%2C+A%2E%3B+Tafjord%2C+O%2E%3B+Turney%2C+P%2E%3B+and+Khashabi%2C+D%2E+2016%2E+Combining+retrieval%2C+statistics%2C+and+inference+to+answer+elementary+science+questions%2E+In+Thirtieth+AAA)
10. Cui, Y.; Che, W.; Liu, T.; Qin, B.; Yang, Z.; Wang, S.; and Hu, G. 2019. Pre-Training with Whole Word Masking for Chinese BERT. arXiv preprint arXiv:1906.08101 . [find](https://scholar.google.com/scholar?q=Cui%2C+Y%2E%3B+Che%2C+W%2E%3B+Liu%2C+T%2E%3B+Qin%2C+B%2E%3B+Yang%2C+Z%2E%3B+Wang%2C+S%2E%3B+and+Hu%2C+G%2E+2019%2E+Pre%2DTraining+with+Whole+Word+Masking+for+Chinese+BERT%2E+arXiv+preprint+arXiv%3A1906%2E08101+%2E) [our card](https://md.research.elamur.ai/free/en/papers/cui-2019-pre-training-whole-word-masking/)
11. Das, R.; Dhuliawala, S.; Zaheer, M.; and McCallum, A. 2019. Multi-step Retriever-Reader Interaction for Scalable Open-domain Question Answering. ArXiv abs/1905.05733. — [arXiv:1905.05733](https://arxiv.org/abs/1905.05733) [find](https://scholar.google.com/scholar?q=Das%2C+R%2E%3B+Dhuliawala%2C+S%2E%3B+Zaheer%2C+M%2E%3B+and+McCallum%2C+A%2E+2019%2E+Multi%2Dstep+Retriever%2DReader+Interaction+for+Scalable+Open%2Ddomain+Question+Answering%2E+ArXiv+abs%2F1905%2E05733%2E+%E2%80%94+%5BarXiv%3A1905%2E05733%5D%28https%3A%2F%2Farxi)
12. Demner-Fushman, D.; Mrabet, Y.; and Ben Abacha, A. 2020. Consumer health information and question answering: helping consumers find answers to their health-related information needs. Journal of the American Medical Informatics Association 27(2): 194–201. [find](https://scholar.google.com/scholar?q=Demner%2DFushman%2C+D%2E%3B+Mrabet%2C+Y%2E%3B+and+Ben+Abacha%2C+A%2E+2020%2E+Consumer+health+information+and+question+answering%3A+helping+consumers+find+answers+to+their+health%2Drelated+information+needs%2E+Journal+of+the+Am)
13. Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Lo… [find](https://scholar.google.com/scholar?q=Devlin%2C+J%2E%3B+Chang%2C+M%2E%2DW%2E%3B+Lee%2C+K%2E%3B+and+Toutanova%2C+K%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+Deep+Bidirectional+Transformers+for+Language+Understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American)
14. Dhingra, B.; Mazaitis, K.; and Cohen, W. W. 2017. Quasar: Datasets for Question Answering by Search and Reading. ArXiv abs/1707.03904. — [arXiv:1707.03904](https://arxiv.org/abs/1707.03904) [find](https://scholar.google.com/scholar?q=Dhingra%2C+B%2E%3B+Mazaitis%2C+K%2E%3B+and+Cohen%2C+W%2E+W%2E+2017%2E+Quasar%3A+Datasets+for+Question+Answering+by+Search+and+Reading%2E+ArXiv+abs%2F1707%2E03904%2E+%E2%80%94+%5BarXiv%3A1707%2E03904%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1707%2E03904%29)
15. Dunn, M.; Sagun, L.; Higgins, M.; Güney, V. U.; Cirik, V.; and Cho, K. 2017. SearchQA: A New QA Dataset Augmented with Context from a Search Engine. ArXiv abs/1704.05179. [find](https://scholar.google.com/scholar?q=Dunn%2C+M%2E%3B+Sagun%2C+L%2E%3B+Higgins%2C+M%2E%3B+Gu%CC%88ney%2C+V%2E+U%2E%3B+Cirik%2C+V%2E%3B+and+Cho%2C+K%2E+2017%2E+SearchQA%3A+A+New+QA+Dataset+Augmented+with+Context+from+a+Search+Engine%2E+ArXiv+abs%2F1704%2E05179%2E) [our card](https://md.research.elamur.ai/free/en/papers/dunn-2017-searchqa-new-q-dataset-augmented/)
16. Gao, S.; Agarwal, S.; Jin, D.; Chung, T.; and Hakkani-Tur, D. 2020. From Machine Reading Comprehension to Dialogue State Tracking: Bridging the Gap. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, 79–89. Online: Association for Computational Linguistics. doi:… [find](https://scholar.google.com/scholar?q=Gao%2C+S%2E%3B+Agarwal%2C+S%2E%3B+Jin%2C+D%2E%3B+Chung%2C+T%2E%3B+and+Hakkani%2DTur%2C+D%2E+2020%2E+From+Machine+Reading+Comprehension+to+Dialogue+State+Tracking%3A+Bridging+the+Gap%2E+In+Proceedings+of+the+2nd+Workshop+on+Natural+Langu)
17. Gururangan, S.; Marasovi, A.; Swayamdipta, S.; Lo, K.; Beltagy, I.; Downey, D.; and Smith, N. A. 2020. Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Gururangan%2C+S%2E%3B+Marasovi%2C+A%2E%3B+Swayamdipta%2C+S%2E%3B+Lo%2C+K%2E%3B+Beltagy%2C+I%2E%3B+Downey%2C+D%2E%3B+and+Smith%2C+N%2E+A%2E+2020%2E+Don%E2%80%99t+Stop+Pretraining%3A+Adapt+Language+Models+to+Domains+and+Tasks%2E+In+Proceedings+of+ACL%2E)
18. Ha, L. A.; and Yaneva, V. 2019. Automatic Question Answering for Medical MCQs: Can It go Further than Information Retrieval? In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), 418–422. Varna, Bulgaria: INCOMA Ltd. doi:10.26615/978-954-452-0… [find](https://scholar.google.com/scholar?q=Ha%2C+L%2E+A%2E%3B+and+Yaneva%2C+V%2E+2019%2E+Automatic+Question+Answering+for+Medical+MCQs%3A+Can+It+go+Further+than+Information+Retrieval%3F+In+Proceedings+of+the+International+Conference+on+Recent+Advances+in+Natura)
19. Jin, D.; Gao, S.; Kao, J.-Y.; Chung, T.; and Hakkani-Tür, D. Z. 2019. MMM: Multi-stage Multi-task Learning for Multi-choice Reading Comprehension. ArXiv abs/1910.00458. — [arXiv:1910.00458](https://arxiv.org/abs/1910.00458) [find](https://scholar.google.com/scholar?q=Jin%2C+D%2E%3B+Gao%2C+S%2E%3B+Kao%2C+J%2E%2DY%2E%3B+Chung%2C+T%2E%3B+and+Hakkani%2DTu%CC%88r%2C+D%2E+Z%2E+2019%2E+MMM%3A+Multi%2Dstage+Multi%2Dtask+Learning+for+Multi%2Dchoice+Reading+Comprehension%2E+ArXiv+abs%2F1910%2E00458%2E+%E2%80%94+%5BarXiv%3A1910%2E00458%5D%28https%3A%2F%2Fa)
20. Joshi, M.; Choi, E.; Weld, D.; and Zettlemoyer, L. 2017. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1601–1611. Vancouver, Canada: Associa… [find](https://scholar.google.com/scholar?q=Joshi%2C+M%2E%3B+Choi%2C+E%2E%3B+Weld%2C+D%2E%3B+and+Zettlemoyer%2C+L%2E+2017%2E+TriviaQA%3A+A+Large+Scale+Distantly+Supervised+Challenge+Dataset+for+Reading+Comprehension%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Asso) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/)
21. Lai, G.; Xie, Q.; Liu, H.; Yang, Y.; and Hovy, E. 2017. RACE: Large-scale ReAding Comprehension Dataset From Examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 785– 794. Copenhagen, Denmark: Association for Computational Linguistics. doi:10.18653… [find](https://scholar.google.com/scholar?q=Lai%2C+G%2E%3B+Xie%2C+Q%2E%3B+Liu%2C+H%2E%3B+Yang%2C+Y%2E%3B+and+Hovy%2C+E%2E+2017%2E+RACE%3A+Large%2Dscale+ReAding+Comprehension+Dataset+From+Examinations%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Natural+Languag) [our card](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/)
22. Lan, Z.; Chen, M.; Goodman, S.; Gimpel, K.; Sharma, P.; and Soricut, R. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 . [find](https://scholar.google.com/scholar?q=Lan%2C+Z%2E%3B+Chen%2C+M%2E%3B+Goodman%2C+S%2E%3B+Gimpel%2C+K%2E%3B+Sharma%2C+P%2E%3B+and+Soricut%2C+R%2E+2019%2E+Albert%3A+A+lite+bert+for+self%2Dsupervised+learning+of+language+representations%2E+arXiv+preprint+arXiv%3A1909%2E11942+%2E)
23. Lee, J.; Yoon, W.; Kim, S.; Kim, D.; Kim, S.; So, C. H.; and Kang, J. 2019. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics ISSN 1367-4803. doi: 10.1093/bioinformatics/btz682. URL https://doi.org/10.1093/bioinformatics/btz682 . [↗](https://doi.org/10.1093/bioinformatics/btz682) [our card](https://md.research.elamur.ai/free/en/papers/lee-2019-biobert-pre-trained-biomedical-language/)
24. Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. ArXiv abs/1907.11692. [find](https://scholar.google.com/scholar?q=Liu%2C+Y%2E%3B+Ott%2C+M%2E%3B+Goyal%2C+N%2E%3B+Du%2C+J%2E%3B+Joshi%2C+M%2E%3B+Chen%2C+D%2E%3B+Levy%2C+O%2E%3B+Lewis%2C+M%2E%3B+Zettlemoyer%2C+L%2E%3B+and+Stoyanov%2C+V%2E+2019%2E+RoBERTa%3A+A+Robustly+Optimized+BERT+Pretraining+Approach%2E+ArXiv+abs%2F1907%2E11692%2E)
25. Mihaylov, T.; Clark, P.; Khot, T.; and Sabharwal, A. 2018. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2381–2391. Brussels, Belgium: Association for Computational L… [find](https://scholar.google.com/scholar?q=Mihaylov%2C+T%2E%3B+Clark%2C+P%2E%3B+Khot%2C+T%2E%3B+and+Sabharwal%2C+A%2E+2018%2E+Can+a+Suit+of+Armor+Conduct+Electricity%3F+A+New+Dataset+for+Open+Book+Question+Answering%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+M)
26. Moen, S.; and Ananiadou, T. S. S. 2013. Distributional semantics resources for biomedical text processing. Proceedings of LBM 39–44. [find](https://scholar.google.com/scholar?q=Moen%2C+S%2E%3B+and+Ananiadou%2C+T%2E+S%2E+S%2E+2013%2E+Distributional+semantics+resources+for+biomedical+text+processing%2E+Proceedings+of+LBM+39%E2%80%9344%2E)
27. Nentidis, A.; Krithara, A.; Bougiatiotis, K.; Krallinger, M.; Rodriguez-Penagos, C.; Villegas, M.; and Paliouras, G. 2020. Overview of BioASQ 2020: The Eighth BioASQ Challenge on Large-Scale Biomedical Semantic Indexing and Question Answering. In International Conference of the Cross-Language Evalua… [find](https://scholar.google.com/scholar?q=Nentidis%2C+A%2E%3B+Krithara%2C+A%2E%3B+Bougiatiotis%2C+K%2E%3B+Krallinger%2C+M%2E%3B+Rodriguez%2DPenagos%2C+C%2E%3B+Villegas%2C+M%2E%3B+and+Paliouras%2C+G%2E+2020%2E+Overview+of+BioASQ+2020%3A+The+Eighth+BioASQ+Challenge+on+Large%2DScale+Biomedica)
28. Pampari, A.; Raghavan, P.; Liang, J.; and Peng, J. 2018. emrQA: A Large Corpus for Question Answering on Electronic Medical Records. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2357–2368. Brussels, Belgium: Association for Computational Linguistics. doi… [find](https://scholar.google.com/scholar?q=Pampari%2C+A%2E%3B+Raghavan%2C+P%2E%3B+Liang%2C+J%2E%3B+and+Peng%2C+J%2E+2018%2E+emrQA%3A+A+Large+Corpus+for+Question+Answering+on+Electronic+Medical+Records%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natur) [our card](https://md.research.elamur.ai/free/en/papers/pampari-2018-emrqa-large-corpus-question-answering/)
29. Petroni, F.; Rocktäschel, T.; Lewis, P.; Bakhtin, A.; Wu, Y.; Miller, A. H.; and Riedel, S. 2019. Language Models as Knowledge Bases? In EMNLP/IJCNLP. [find](https://scholar.google.com/scholar?q=Petroni%2C+F%2E%3B+Rockta%CC%88schel%2C+T%2E%3B+Lewis%2C+P%2E%3B+Bakhtin%2C+A%2E%3B+Wu%2C+Y%2E%3B+Miller%2C+A%2E+H%2E%3B+and+Riedel%2C+S%2E+2019%2E+Language+Models+as+Knowledge+Bases%3F+In+EMNLP%2FIJCNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/)
30. Rajpurkar, P.; Jia, R.; and Liang, P. 2018. Know What You Don’t Know: Unanswerable Questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 784–789. Melbourne, Australia: Association for Computational Linguistics. doi:1… [find](https://scholar.google.com/scholar?q=Rajpurkar%2C+P%2E%3B+Jia%2C+R%2E%3B+and+Liang%2C+P%2E+2018%2E+Know+What+You+Don%E2%80%99t+Know%3A+Unanswerable+Questions+for+SQuAD%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volu) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/)
31. Rajpurkar, P.; Zhang, J.; Lopyrev, K.; and Liang, P. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2383–2392. Austin, Texas: Association for Computational Linguistics. doi:10.18653/v1/D16-… [find](https://scholar.google.com/scholar?q=Rajpurkar%2C+P%2E%3B+Zhang%2C+J%2E%3B+Lopyrev%2C+K%2E%3B+and+Liang%2C+P%2E+2016%2E+SQuAD%3A+100%2C000%2B+Questions+for+Machine+Comprehension+of+Text%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Language+P) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
32. Reddy, S.; Chen, D.; and Manning, C. D. 2019. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics 7: 249– 266. [find](https://scholar.google.com/scholar?q=Reddy%2C+S%2E%3B+Chen%2C+D%2E%3B+and+Manning%2C+C%2E+D%2E+2019%2E+Coqa%3A+A+conversational+question+answering+challenge%2E+Transactions+of+the+Association+for+Computational+Linguistics+7%3A+249%E2%80%93+266%2E) [our card](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/)
33. Sun, K.; Yu, D.; Yu, D.; and Cardie, C. 2019. Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehension. Transactions of the Association for Computational Linguistics 8: 141–155. [find](https://scholar.google.com/scholar?q=Sun%2C+K%2E%3B+Yu%2C+D%2E%3B+Yu%2C+D%2E%3B+and+Cardie%2C+C%2E+2019%2E+Investigating+Prior+Knowledge+for+Challenging+Chinese+Machine+Reading+Comprehension%2E+Transactions+of+the+Association+for+Computational+Linguistics+8%3A+141%E2%80%93)
34. Talmor, A.; Elazar, Y.; Goldberg, Y.; and Berant, J. 2019. oLMpics - On what Language Model Pre-training Captures. ArXiv abs/1912.13283. [find](https://scholar.google.com/scholar?q=Talmor%2C+A%2E%3B+Elazar%2C+Y%2E%3B+Goldberg%2C+Y%2E%3B+and+Berant%2C+J%2E+2019%2E+oLMpics+%2D+On+what+Language+Model+Pre%2Dtraining+Captures%2E+ArXiv+abs%2F1912%2E13283%2E) [our card](https://md.research.elamur.ai/free/en/papers/talmor-2019-olmpics-language-model-pre-training/)
35. Wang, S.; Yu, M.; Guo, X.; Wang, Z.; Klinger, T.; Zhang, W.; Chang, S.; Tesauro, G.; Zhou, B.; and Jiang, J. 2018. R3: Reinforced Ranker-Reader for Open-Domain Question Answering. In AAAI. [find](https://scholar.google.com/scholar?q=Wang%2C+S%2E%3B+Yu%2C+M%2E%3B+Guo%2C+X%2E%3B+Wang%2C+Z%2E%3B+Klinger%2C+T%2E%3B+Zhang%2C+W%2E%3B+Chang%2C+S%2E%3B+Tesauro%2C+G%2E%3B+Zhou%2C+B%2E%3B+and+Jiang%2C+J%2E+2018%2E+R3%3A+Reinforced+Ranker%2DReader+for+Open%2DDomain+Question+Answering%2E+In+AAAI%2E)
36. Wang, Z.; Ng, P.; Ma, X.; Nallapati, R.; and Xiang, B. 2019. Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering. In EMNLP/IJCNLP. [find](https://scholar.google.com/scholar?q=Wang%2C+Z%2E%3B+Ng%2C+P%2E%3B+Ma%2C+X%2E%3B+Nallapati%2C+R%2E%3B+and+Xiang%2C+B%2E+2019%2E+Multi%2Dpassage+BERT%3A+A+Globally+Normalized+BERT+Model+for+Open%2Ddomain+Question+Answering%2E+In+EMNLP%2FIJCNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2019-multi-passage-bert-globally-normalized/)
37. Yan, M.; Zhang, H.; Jin, D.; and Zhou, J. T. 2020. Multi-source Meta Transfer for Low Resource Multiple-Choice Question Answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 7331–7341. Online: Association for Computational Linguistics. doi:10.18653/v1… [find](https://scholar.google.com/scholar?q=Yan%2C+M%2E%3B+Zhang%2C+H%2E%3B+Jin%2C+D%2E%3B+and+Zhou%2C+J%2E+T%2E+2020%2E+Multi%2Dsource+Meta+Transfer+for+Low+Resource+Multiple%2DChoice+Question+Answering%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Comp)
38. Yu, W.; Jiang, Z.-H.; Dong, Y.; and Feng, J. 2020. ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning. ArXiv abs/2002.04326. [find](https://scholar.google.com/scholar?q=Yu%2C+W%2E%3B+Jiang%2C+Z%2E%2DH%2E%3B+Dong%2C+Y%2E%3B+and+Feng%2C+J%2E+2020%2E+ReClor%3A+A+Reading+Comprehension+Dataset+Requiring+Logical+Reasoning%2E+ArXiv+abs%2F2002%2E04326%2E) [our card](https://md.research.elamur.ai/free/en/papers/yu-2020-reclor-reading-comprehension-dataset-requiring/)
39. Zhang, X.; Wu, J.; He, Z.; Liu, X.; and Su, Y. 2018. Medical exam question answering with large-scale reading comprehension. In Thirty-Second AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Zhang%2C+X%2E%3B+Wu%2C+J%2E%3B+He%2C+Z%2E%3B+Liu%2C+X%2E%3B+and+Su%2C+Y%2E+2018%2E+Medical+exam+question+answering+with+large%2Dscale+reading+comprehension%2E+In+Thirty%2DSecond+AAAI+Conference+on+Artificial+Intelligence%2E)