--- title: "Overview of the MEDIQA 2019 Shared Task on Textual" description: "Textual inference and question entailment have had relatively little attention in… 98% accuracy on NLI. 2 references, cited by 63 papers here." authors: ["Asma Ben Abacha", "Chaitanya Shivade", "Dina Demner-Fushman"] year: 2019 venue: "Proceedings of the 18th BioNLP Workshop and Shared Task" source: "https://aclanthology.org/W19-5039/" doi: "10.18653/v1/w19-5039" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/abacha-2019-overview-mediqa-2019-shared-task/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/abacha-2019-overview-mediqa-2019-shared-task/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / Overview of the MEDIQA 2019 Shared Task on Textual Inference, Question Entailment and Question Answering # Overview of the MEDIQA 2019 Shared Task on Textual Inference, Question Entailment and Question Answering **Authors:** [Asma Ben Abacha](https://md.research.elamur.ai/free/en/authors/asma-ben-abacha/), [Chaitanya Shivade](https://md.research.elamur.ai/free/en/authors/chaitanya-shivade/) and [Dina Demner-Fushman](https://md.research.elamur.ai/free/en/authors/dina-demner-fushman/) · **Year:** 2019 · **Venue:** [Proceedings of the 18th BioNLP Workshop and Shared Task](https://md.research.elamur.ai/free/en/venues/bionlp-workshop-and-shared-task/) ([BioNLP Shared Task Workshop 2019](https://md.research.elamur.ai/free/en/venues/bionlp-workshop-and-shared-task/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/W19-5039/) · [PDF](https://aclanthology.org/W19-5039.pdf) · [DOI 10.18653/v1/w19-5039](https://doi.org/10.18653/v1/w19-5039) · [OpenAlex](https://openalex.org/W2970986790) **Citations:** 116 citations (OpenAlex, 2026-10-14) · [63](https://md.research.elamur.ai/free/en/papers/abacha-2019-overview-mediqa-2019-shared-task/cited-by/) in our library, of the 141,840 papers whose bibliographies we read ## Summary Textual inference and question entailment have had relatively little attention in medicine, even though both could help health question-answering systems choose better answers. To address this, the MEDIQA 2019 shared task at the ACL-BioNLP workshop set three linked challenges on AIcrowd: inferring logical relations between medical sentences (NLI), deciding whether one question entails another (RQE), and question answering (QA), where systems had to filter and re-rank answers retrieved by the CHiQA system. The organizers released datasets for each task: MedNLI for training NLI, consumer health question/FAQ pairs for RQE, and expert-rated CHiQA answers for QA. They also strongly encouraged teams to reuse their NLI and RQE models inside the QA pipeline. Seventy-two teams took part. The best scores were 98% accuracy on NLI, 74.9% on RQE and 78.3% on QA. Most leading systems were BERT variants, often combined with multi-task learning, ensembling, domain knowledge from UMLS and external data. The shared task produced public benchmarks and a comparison of methods for medical inference and answer ranking. ## Key points - **NLI data.** Training used MedNLI, which has 14,049 clinical sentence pairs drawn from MIMIC-III. The test set had 405 pairs labeled Entailment, Neutral or Contradiction. The baseline was InferSent with fastText embeddings trained on MIMIC-III. - **RQE data.** The training set had 8,890 pairs built from doctors' clinical questions, and the validation set had 302 pairs. The test set had 230 pairs matching real consumer health questions sent to NLM with NIH FAQs, all validated by medical experts. The baseline was a feature-based SVM. - **QA data.** - Training: two sets, TREC-2017-LiveQA (104 questions, 839 answers) and Alexa (104 questions, 862 answers). - Validation: 25 questions with 234 answers. - Test: 150 questions with 1,107 answers. - Labels: experts gave each answer a score from 1 to 4 and a reference rank. - Evaluation: accuracy, MRR, Precision and Spearman's rho. - **Participation and results.** - Of 72 teams, 20 submitted working notes papers, and eight of those papers covered all three tasks. - NLI had 17 official teams out of 43 participants, and most built on BERT variants (BioBERT, ClinicalBERT, SciBERT). - QA had 10 official teams out of 23. The best MRR was 96.22%, and the top six systems reached a precision of 79.3–81.9%. - **Top QA systems.** - DoubleTransfer (MT-DNN plus ensembles) had the best QA accuracy and precision. - Pentagon had the best MRR and Spearman's rho. It used MedQuAD (47K QA pairs) and iCliniq to retrieve entailed questions. ## Interesting ideas - **Defining question entailment by answers rather than similarity.** One question counts as entailing another when answers to the second fully or partly answer the first. This makes the relation directly usable for reusing stored answers, which suits FAQ retrieval in any domain or language. - **Using inference models as feature extractors for re-ranking.** The best ranking system reused its NLI and RQE models to embed answers jointly with the top retrieved entailed questions. It also added simple metadata such as answer source, answer length and the original rank. A pairwise answer-to-answer comparison then outperformed stronger standalone classifiers on ranking metrics. - **Mixing external data into training batches (DoubleTransfer's Multi-Source).** Each training batch was topped up with randomly selected external examples at about a 50% ratio to curb overfitting on small in-domain sets. The final ensemble used majority voting and broke ties by summing class probabilities. Both tricks are cheap and work for any low-resource task. - **Testing on realistic data that differs from the training data.** The RQE test pairs were messy, multi-part consumer questions, while training used cleaner generated pairs. This exposes how well models generalize, a useful design choice for benchmarks in under-resourced languages. - **Reporting several metrics for ranking.** The teams that led on accuracy did not lead on MRR or Spearman's rho. Reporting classification and ranking metrics together therefore shows trade-offs that a single score would hide. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [National Institutes of Health](https://md.research.elamur.ai/free/en/government/national-institutes-of-health/) — Government body, Bethesda, United States · 9 works · 3,833 citations - [IBM Research - Almaden](https://md.research.elamur.ai/free/en/labs/ibm-research-almaden/) — Research institute, San Jose, United States · 5 works · 8,486 citations 📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🧬 Biochemistry, Genetics and Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/) · 116 works · [🧬 Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/molecular-biology/) · 114 works 🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 685 works · [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 242 works ## Cite Cite as: Asma Ben Abacha, Chaitanya Shivade and Dina Demner-Fushman. 2019. Overview of the MEDIQA 2019 Shared Task on Textual Inference, Question Entailment and Question Answering. Proceedings of the 18th BioNLP Workshop and Shared Task. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{abacha2019overview, title = {{Overview of the MEDIQA 2019 Shared Task on Textual Inference, Question Entailment and Question Answering}}, author = {Asma Ben Abacha, Chaitanya Shivade and Dina Demner-Fushman}, booktitle = {Proceedings of the 18th BioNLP Workshop and Shared Task}, year = {2019}, doi = {10.18653/v1/w19-5039}, url = {https://doi.org/10.18653/v1/w19-5039} } ``` - [← previous It's All in the Name: Mitigating Gender Bias with Name-Based Counterfactual Data Substitution](https://md.research.elamur.ai/free/en/papers/maudslay-2019-it-s-all-name-mitigating/) - [next → Generalized Data Augmentation for Low-Resource Translation](https://md.research.elamur.ai/free/en/papers/xia-2019-generalized-data-augmentation-low-resource/) - [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) ## References 2 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Meni Adler, Jonathan Berant, and Ido Dagan. 2012. Entailment-based text exploration with application to the health-care domain. In The 50th Annual Meeting of the Association for Computational Linguistics, System Demonstrations, 2012, Korea. [find](https://scholar.google.com/scholar?q=Meni+Adler%2C+Jonathan+Berant%2C+and+Ido+Dagan%2E+2012%2E+Entailment%2Dbased+text+exploration+with+application+to+the+health%2Dcare+domain%2E+In+The+50th+Annual+Meeting+of+the+Association+for+Computational+Linguist) 2. Anumeha Agrawal, Rosa Anil George, Selvan Sunitha Ravi, Sowmya Kamath, and Anand Kumar. 2019. Ars nitk at mediqa 2019:analysing various methods for natural language inference, recognising question entailment and medical question answering system. In Proceedings of the BioNLP 2019 workshop, Florence,… [find](https://scholar.google.com/scholar?q=Anumeha+Agrawal%2C+Rosa+Anil+George%2C+Selvan+Sunitha+Ravi%2C+Sowmya+Kamath%2C+and+Anand+Kumar%2E+2019%2E+Ars+nitk+at+mediqa+2019%3Aanalysing+various+methods+for+natural+language+inference%2C+recognising+question+ent) *Updated 24 October 2026*