--- title: "MRQA 2019 Shared Task — Fisch 2019" description: "Reading comprehension models are usually scored on test data that matches their training distribution, so it is unclear how well they cope with…" authors: ["Adam Fisch", "Alon Talmor", "Robin Jia", "Minjoon Seo", "Eunsol Choi", "Danqi Chen"] year: 2019 venue: "Proceedings of the 2nd Workshop on Machine Reading for Question Answering" source: "https://aclanthology.org/D19-5801/" doi: "10.18653/v1/d19-5801" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/fisch-2019-mrqa-2019-shared-task-evaluating/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/fisch-2019-mrqa-2019-shared-task-evaluating/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/) / MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension # MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension **Authors:** [Adam Fisch](https://md.research.elamur.ai/free/en/authors/adam-fisch/), [Alon Talmor](https://md.research.elamur.ai/free/en/authors/alon-talmor/), [Robin Jia](https://md.research.elamur.ai/free/en/authors/robin-jia/), [Minjoon Seo](https://md.research.elamur.ai/free/en/authors/minjoon-seo/), [Eunsol Choi](https://md.research.elamur.ai/free/en/authors/eunsol-choi/), [Danqi Chen](https://md.research.elamur.ai/free/en/authors/danqi-chen/) · **Year:** 2019 · **Venue:** Proceedings of the 2nd Workshop on Machine Reading for Question Answering · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/D19-5801/) · [PDF](https://aclanthology.org/D19-5801.pdf) · [DOI 10.18653/v1/d19-5801](https://doi.org/10.18653/v1/d19-5801) · [OpenAlex](https://openalex.org/W2988421999) **Citations:** 215 citations (openalex-snapshot · 2026-10-14) · [160](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Reading comprehension models are usually scored on test data that matches their training distribution, so it is unclear how well they cope with unfamiliar domains. The MRQA 2019 Shared Task measures this out-of-domain generalization for English extractive question answering. The organizers converted 18 existing QA datasets into a single span-extraction format. Contexts are capped at 800 tokens and marked with [DOC], [TLE] and [PAR] tokens. Multiple-choice and answer-string-only sources were adapted as needed. The datasets were split three ways: six for training, six for development and testing, and six fully hidden from participants. Systems were ranked by macro-averaged F1 over the 12 held-out test sets. Ten teams took part, trying approaches such as: - data sampling - multi-task learning - adversarial domain training - ensembling The winning system, D-Net, built on newer pre-trained language models and reached 72.5 F1, 10.7 points above the BERT-based baseline. The released benchmark gives the community a common, reusable way to test whether QA systems hold up beyond their training data. ## Key points - Training sets (Split I) are SQuAD, NewsQA, TriviaQA, SearchQA, HotpotQA and Natural Questions. Split II includes BioASQ, DROP, DuoRC, RACE, RelationExtraction and TextbookQA. Split III includes BioProcess, ComplexWebQ, MCTest, QAMR, QAST and TREC. - Test portions were rebalanced to about 1,500 examples per sub-domain, partitioned by context. Scoring uses EM and F1 based on answer-string match. - Adversarial training with a domain discriminator added +1.9 F1 over a BERT-Base model but only +0.4 F1 over BERT-Large. Negative sampling gave up to +1.5 F1 on Split II. - Fine-tuning the BERT-Large baseline on Split II development data raised its score by 4.6 F1. This closed the gap to D-Net only on RelationExtraction. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 160 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 148 not yet. - [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/) - [Generalizing Cross-Document Event Coreference Resolution Across Multiple Corpora](https://md.research.elamur.ai/free/en/papers/bugert-2021-generalizing-cross-document-event-coreference/) - [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://md.research.elamur.ai/free/en/papers/joshi-2019-spanbert-improving-pre-training-representing/) - [UNIFIEDQA: Crossing Format Boundaries with a Single QA System](https://md.research.elamur.ai/free/en/papers/khashabi-2020-unifiedqa-crossing-format-boundaries-single/) - [Entity-Based Knowledge Conflicts in Question Answering](https://md.research.elamur.ai/free/en/papers/longpre-2021-entity-based-knowledge-conflicts-question/) - [RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering](https://md.research.elamur.ai/free/en/papers/qu-2021-rocketqa-optimized-training-approach-dense/) - [A guide to the dataset explosion in QA, NLI, and commonsense reasoning](https://md.research.elamur.ai/free/en/papers/rogers-2020-guide-dataset-explosion-qa-nli/) - [SCROLLS: Standardized CompaRison Over Long Language Sequences](https://md.research.elamur.ai/free/en/papers/shaham-2022-scrolls-standardized-comparison-over-long/) - [Prompting GPT-3 To Be Reliable](https://md.research.elamur.ai/free/en/papers/si-2022-prompting-gpt-3-reliable/) - [SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer](https://md.research.elamur.ai/free/en/papers/vu-2022-spot-better-frozen-model-adaptation/) - [DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models](https://md.research.elamur.ai/free/en/papers/wang-2023-decodingtrust-comprehensive-assessment-trustworthiness-gpt/) - [LinkBERT: Pretraining Language Models with Document Links](https://md.research.elamur.ai/free/en/papers/yasunaga-2022-linkbert-pretraining-language-models-document/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{fisch2019mrqa, title = {{MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension}}, author = {Fisch, Adam and Talmor, Alon and Jia, Robin and Seo, Minjoon and Choi, Eunsol and Chen, Danqi}, booktitle = {Proceedings of the 2nd Workshop on Machine Reading for Question Answering}, year = {2019}, doi = {10.18653/v1/d19-5801}, url = {https://doi.org/10.18653/v1/d19-5801} } ``` - [← previous MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://md.research.elamur.ai/free/en/papers/zhao-2019-moverscore-text-generation-evaluating-contextualized/) - [next → Multi-Modal Sarcasm Detection in Twitter with Hierarchical Fusion Model](https://md.research.elamur.ai/free/en/papers/cai-2019-multi-modal-sarcasm-detection-twitter/) - [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/) ## References 4 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Petr Baudiš and Jan Šedivý. 2015. Modeling of the question answering task in the yodaqa system. In Proceedings of the 6th International Conference on Experimental IR Meets Multilinguality, Multimodality, and Interaction - Volume 9283. [find](https://scholar.google.com/scholar?q=Petr+Baudis%CC%8C+and+Jan+S%CC%8Cedivy%CC%81%2E+2015%2E+Modeling+of+the+question+answering+task+in+the+yodaqa+system%2E+In+Proceedings+of+the+6th+International+Conference+on+Experimental+IR+Meets+Multilinguality%2C+Multimod) 2. Jonathan Berant, Vivek Srikumar, Pei-Chun Chen, Abby Vander Linden, Brittany Harding, Brad Huang, Peter Clark, and Christopher D. Manning. 2014. Modeling biological processes for reading comprehension. In EMNLP. [find](https://scholar.google.com/scholar?q=Jonathan+Berant%2C+Vivek+Srikumar%2C+Pei%2DChun+Chen%2C+Abby+Vander+Linden%2C+Brittany+Harding%2C+Brad+Huang%2C+Peter+Clark%2C+and+Christopher+D%2E+Manning%2E+2014%2E+Modeling+biological+processes+for+reading+comprehension) 3. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In ACL. [find](https://scholar.google.com/scholar?q=Danqi+Chen%2C+Adam+Fisch%2C+Jason+Weston%2C+and+Antoine+Bordes%2E+2017%2E+Reading+Wikipedia+to+answer+open%2Ddomain+questions%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2017-reading-wikipedia-answer-open-domain/) 4. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuAC: Question answering in context. pages 2174–2184. \#\# Conclusions [find](https://scholar.google.com/scholar?q=Eunsol+Choi%2C+He+He%2C+Mohit+Iyyer%2C+Mark+Yatskar%2C+Wen%2Dtau+Yih%2C+Yejin+Choi%2C+Percy+Liang%2C+and+Luke+Zettlemoyer%2E+2018%2E+QuAC%3A+Question+answering+in+context%2E+pages+2174%E2%80%932184%2E+%5C%23%5C%23+Conclusions) [our card](https://md.research.elamur.ai/free/en/papers/choi-2018-quac-question-answering-context/)