---
title: "HEAD-QA: A Healthcare Dataset for Complex Reasoning (2019)"
description: "HEAD-QA addresses a gap in question answering resources. 37% accuracy. 27 references, cited by 58 papers here."
authors: ["David Vilares", "Carlos Gómez-Rodríguez"]
year: 2019
venue: "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/P19-1092/"
doi: "10.18653/v1/p19-1092"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/vilares-2019-head-qa-healthcare-dataset-complex/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/vilares-2019-head-qa-healthcare-dataset-complex/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/) / HEAD-QA: A Healthcare Dataset for Complex Reasoning

# HEAD-QA: A Healthcare Dataset for Complex Reasoning

**Authors:** [David Vilares](https://md.research.elamur.ai/free/en/authors/david-vilares/) and [Carlos Gómez-Rodríguez](https://md.research.elamur.ai/free/en/authors/carlos-gomez-rodriguez/) · **Year:** 2019 · **Venue:** [Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2019](https://md.research.elamur.ai/free/en/venues/acl/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P19-1092/) · [PDF](https://aclanthology.org/P19-1092.pdf) · [DOI 10.18653/v1/p19-1092](https://doi.org/10.18653/v1/p19-1092) · [OpenAlex](https://openalex.org/W2949818215)

**Citations:** 46 citations (OpenAlex, 2026-10-14) · [58](https://md.research.elamur.ai/free/en/papers/vilares-2019-head-qa-healthcare-dataset-complex/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
HEAD-QA addresses a gap in question answering resources. Popular benchmarks such as SQuAD and bAbI can often be solved with surface-level cues, and existing reasoning-oriented datasets are mostly limited to grade-school science. To provide a harder, specialist-level test, Vilares and Gómez-Rodríguez collected multiple-choice questions from the Spanish government's annual exams for specialization posts in public healthcare. Candidates sitting these exams already hold degrees and typically study for a year or more. The questions cover six areas: medicine, nursing, biology, chemistry, psychology and pharmacology. The dataset is in Spanish, and the authors add a machine-translated English version, HEAD-QA-EN, for cross-lingual experiments.

The authors evaluate several kinds of systems over Wikipedia:
- simple control heuristics;
- a TF-IDF retrieval baseline built on DrQA's Document Retriever;
- English neural systems adapted to multiple choice (DrQA, BiDAF, DGEM, Decompatt).

The non-neural retrieval approach performs best. Even so, it stays well below human examinees and under the official pass mark. This makes HEAD-QA a demanding benchmark for knowledge-intensive and multilingual QA.

## Key points
- **Size and splits:** the dataset has 6,765 questions from exams held from 2013 onward, covering MIR, EIR, BIR, QIR, PIR and FIR. The official supervised split is chronological: 2013–2014 exams for training (2,657), 2015 for development (1,366) and later exams for testing (2,742). Random splitting was avoided so that results stay comparable with official human scores.
- **Images:** about 14% of the medicine questions rely on accompanying images. These are included in the release but were not used in the experiments.
- **Translation quality:** the English version was produced with the Google API. Human raters checked 60 sampled items, giving adequacy scores of 4.35 and 4.71 and a fluency score of 4, all on 5-point scales. Molecular formulae were the main weakness.
- **Evaluation:** systems are scored by accuracy and by the exams' POINTS scheme (+3 for a correct answer, −1 for a wrong one). Cross-lingual IR achieved the best results, averaging about 37% accuracy on the supervised test split.
- **Difficulty and human gap:** medicine is the hardest category, which the authors link to its much longer questions and answers. On the 2016 exams, the best system scored far below the top-10 human average and below the pass mark in every area.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Universidade da Coruña](https://md.research.elamur.ai/free/en/universities/universidade-da-coruna/) — University, A Coruña, Spain · 10 works · 273 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 523 works

🧭 Topics (2): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works · [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 243 works

## Cite

Cite as: David Vilares and Carlos Gómez-Rodríguez. 2019. HEAD-QA: A Healthcare Dataset for Complex Reasoning. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{vilares2019head,
  title = {{HEAD-QA: A Healthcare Dataset for Complex Reasoning}},
  author = {David Vilares and Carlos Gómez-Rodríguez},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1092},
  url = {https://doi.org/10.18653/v1/p19-1092}
}
```

- [← previous APE at Scale and Its Implications on MT Evaluation Biases](https://md.research.elamur.ai/free/en/papers/freitag-2019-ape-scale-its-implications-mt/)
- [next → Simple and Effective Noisy Channel Modeling for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/yee-2019-simple-effective-noisy-channel-modeling/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/)

## References

27 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. 5010–5015, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=5010%E2%80%935015%2C+Brussels%2C+Belgium%2E+Association+for+Computational+Linguistics%2E)
2. Asma Ben Abacha and Demner-Fushman Dina. 2016. Recognizing question entailment for medical question answering. In AMIA Annual Symposium Proceedings, volume 2016, page 310. American Medical Informatics Association. [find](https://scholar.google.com/scholar?q=Asma+Ben+Abacha+and+Demner%2DFushman+Dina%2E+2016%2E+Recognizing+question+entailment+for+medical+question+answering%2E+In+AMIA+Annual+Symposium+Proceedings%2C+volume+2016%2C+page+310%2E+American+Medical+Informatics)
3. Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 4999–5007. [find](https://scholar.google.com/scholar?q=Aniruddha+Kembhavi%2C+Minjoon+Seo%2C+Dustin+Schwenk%2C+Jonghyun+Choi%2C+Ali+Farhadi%2C+and+Hannaneh+Hajishirzi%2E+2017%2E+Are+you+smarter+than+a+sixth+grader%3F+textbook+question+answering+for+multimodal+machine+comp)
4. Asma Ben Abacha, Duy Dinh, and Yassine Mrabet. 2015. Semantic analysis and automatic corpus construction for entailment recognition in medical texts. In Conference on Artificial Intelligence in Medicine in Europe, pages 238–242. Springer. [find](https://scholar.google.com/scholar?q=Asma+Ben+Abacha%2C+Duy+Dinh%2C+and+Yassine+Mrabet%2E+2015%2E+Semantic+analysis+and+automatic+corpus+construction+for+entailment+recognition+in+medical+texts%2E+In+Conference+on+Artificial+Intelligence+in+Medici)
5. Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. In Proceedings of AAAI. [find](https://scholar.google.com/scholar?q=Tushar+Khot%2C+Ashish+Sabharwal%2C+and+Peter+Clark%2E+2018%2E+Scitail%3A+A+textual+entailment+dataset+from+science+question+answering%2E+In+Proceedings+of+AAAI%2E)
6. Davide Buscaldi and Paolo Rosso. 2006. Mining knowledge from wikipedia for the question answering task. In Proceedings of the International Conference on Language Resources and Evaluation, pages 727–730. [find](https://scholar.google.com/scholar?q=Davide+Buscaldi+and+Paolo+Rosso%2E+2006%2E+Mining+knowledge+from+wikipedia+for+the+question+answering+task%2E+In+Proceedings+of+the+International+Conference+on+Language+Resources+and+Evaluation%2C+pages+727%E2%80%937)
7. Philipp Koehn and Christof Monz. 2006. [Manual and automatic evaluation of machine translation between European languages](https://md.research.elamur.ai/free/en/papers/koehn-2006-manual-automatic-evaluation-machine-translation/). In Proceedings on the Workshop on Statistical Machine Translation, pages 102– 121, New York City. Association for Computational Linguistics.
8. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. [Reading Wikipedia to answer open-domain questions](https://md.research.elamur.ai/free/en/papers/chen-2017-reading-wikipedia-answer-open-domain/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870– 1879, Vancouver, Canada. Association for Computational Li…
9. Souvik Kundu and Hwee Tou Ng. 2018. A nil-aware answer extraction framework for question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4243–4252, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Souvik+Kundu+and+Hwee+Tou+Ng%2E+2018%2E+A+nil%2Daware+answer+extraction+framework+for+question+answering%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+424)
10. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. [find](https://scholar.google.com/scholar?q=Peter+Clark%2C+Isaac+Cowhey%2C+Oren+Etzioni%2C+Tushar+Khot%2C+Ashish+Sabharwal%2C+Carissa+Schoenick%2C+and+Oyvind+Tafjord%2E+2018%2E+Think+you+have+solved+question+answering%3F+try+arc%2C+the+ai2+reasoning+challenge%2E+arX)
11. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. [RACE: Large-scale ReAding comprehension dataset from examinations](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785–794, Copenhagen, Denmark. Association for Computational Ling…
12. Peter Clark, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter D Turney, and Daniel Khashabi. 2016. Combining retrieval, statistics, and inference to answer elementary science questions. In AAAI, pages 2580–2586. [find](https://scholar.google.com/scholar?q=Peter+Clark%2C+Oren+Etzioni%2C+Tushar+Khot%2C+Ashish+Sabharwal%2C+Oyvind+Tafjord%2C+Peter+D+Turney%2C+and+Daniel+Khashabi%2E+2016%2E+Combining+retrieval%2C+statistics%2C+and+inference+to+answer+elementary+science+questio)
13. Bernardo Magnini, Simone Romagnoli, Alessandro Vallin, Jesús Herrera, Anselmo Penas, Vı́ctor Peinado, Felisa Verdejo, and Maarten de Rijke. 2003. The multiple language question answering track at clef 2003. In Workshop of the Cross-Language Evaluation Forum for European Languages, pages 471–486. Sp… [find](https://scholar.google.com/scholar?q=Bernardo+Magnini%2C+Simone+Romagnoli%2C+Alessandro+Vallin%2C+Jesu%CC%81s+Herrera%2C+Anselmo+Penas%2C+V%C4%B1%CC%81ctor+Peinado%2C+Felisa+Verdejo%2C+and+Maarten+de+Rijke%2E+2003%2E+The+multiple+language+question+answering+track+at+cle)
14. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. [Teaching machines to read and comprehend](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/). In Advances in Neural Information Processing Systems, pages 1693– 1701.
15. Anastasios Nentidis, Anastasia Krithara, Konstantinos Bougiatiotis, Georgios Paliouras, and Ioannis Kakadiaris. 2018. Results of the sixth edition of the bioasq challenge. In Proceedings of the 6th BioASQ Workshop A challenge on large-scale biomedical semantic indexing and question answering, pages… [find](https://scholar.google.com/scholar?q=Anastasios+Nentidis%2C+Anastasia+Krithara%2C+Konstantinos+Bougiatiotis%2C+Georgios+Paliouras%2C+and+Ioannis+Kakadiaris%2E+2018%2E+Results+of+the+sixth+edition+of+the+bioasq+challenge%2E+In+Proceedings+of+the+6th+Bi)
16. Minghao Hu, Yuxing Peng, Zhen Huang, Xipeng Qiu, Furu Wei, and Ming Zhou. 2018. Reinforced mnemonic reader for machine reading comprehension. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI-18). [find](https://scholar.google.com/scholar?q=Minghao+Hu%2C+Yuxing+Peng%2C+Zhen+Huang%2C+Xipeng+Qiu%2C+Furu+Wei%2C+and+Ming+Zhou%2E+2018%2E+Reinforced+mnemonic+reader+for+machine+reading+comprehension%2E+In+Proceedings+of+the+Twenty%2DSeventh+International+Joint+C)
17. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics.
18. Eleni Kamateri, Theodora Tsikrika, Spyridon Symeonidis, Stefanos Vrochidis, Wolfgang Minker, and Yiannis Kompatsiaris. 2019. A test collection for passage retrieval evaluation of spanish health-related resources. In European Conference on Information Retrieval, pages 148–154. Springer. [find](https://scholar.google.com/scholar?q=Eleni+Kamateri%2C+Theodora+Tsikrika%2C+Spyridon+Symeonidis%2C+Stefanos+Vrochidis%2C+Wolfgang+Minker%2C+and+Yiannis+Kompatsiaris%2E+2019%2E+A+test+collection+for+passage+retrieval+evaluation+of+spanish+health%2Drelate)
19. Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. [A decomposable attention model for natural language inference](https://md.research.elamur.ai/free/en/papers/parikh-2016-decomposable-attention-model-natural-language/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2249–2255. Association for Computational Linguistics.
20. Divyansh Kaushik and Zachary C. Lipton. 2018. [How much reading does reading comprehension require? a critical investigation of popular benchmarks](https://md.research.elamur.ai/free/en/papers/kaushik-2018-how-much-reading-does-reading/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages William Poundstone. 2014. Rock breaks scissors: a practi…
21. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. [Know what you don’t know: Unanswerable questions for SQuAD](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/). pages 784–789.
22. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [Squad: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392.
23. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. [Bidirectional attention flow for machine comprehension](https://md.research.elamur.ai/free/en/papers/seo-2016-bidirectional-attention-flow-machine-comprehension/). arXiv preprint arXiv:1611.01603.
24. José L Vicedo, Ruben Izquierdo, Fernando Llopis, and Rafael Munoz. 2003. Question answering in spanish. In Workshop of the Cross-Language Evaluation Forum for European Languages, pages 541– 548. Springer. [find](https://scholar.google.com/scholar?q=Jose%CC%81+L+Vicedo%2C+Ruben+Izquierdo%2C+Fernando+Llopis%2C+and+Rafael+Munoz%2E+2003%2E+Question+answering+in+spanish%2E+In+Workshop+of+the+Cross%2DLanguage+Evaluation+Forum+for+European+Languages%2C+pages+541%E2%80%93+548%2E+Spri)
25. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. 2015. [Towards AI-complete question answering: A set of prerequisite toy tasks](https://md.research.elamur.ai/free/en/papers/weston-2015-towards-ai-complete-question-answering/). arXiv preprint arXiv:1502.05698.
26. Caiming Xiong, Victor Zhong, and Richard Socher. 2017. Dcn+: Mixed objective and deep residual coattention for question answering. arXiv preprint arXiv:1711.00106. — [arXiv:1711.00106](https://arxiv.org/abs/1711.00106) [find](https://scholar.google.com/scholar?q=Caiming+Xiong%2C+Victor+Zhong%2C+and+Richard+Socher%2E+2017%2E+Dcn%2B%3A+Mixed+objective+and+deep+residual+coattention+for+question+answering%2E+arXiv+preprint+arXiv%3A1711%2E00106%2E+%E2%80%94+%5BarXiv%3A1711%2E00106%5D%28https%3A%2F%2Farxiv%2Eo)
27. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93–104, Brussels, Belgium. Association for Computational Lingu… [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Yonatan+Bisk%2C+Roy+Schwartz%2C+and+Yejin+Choi%2E+2018%2E+SWAG%3A+A+large%2Dscale+adversarial+dataset+for+grounded+commonsense+inference%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+)

*Updated 24 October 2026*