--- title: "MultiQA: An Empirical Investigation of — Talmor 2019" description: "Many extractive reading comprehension (RC) benchmarks now exist, but it was unclear whether a model trained on one transfers to another, or whether…" authors: ["Alon Talmor", "Jonathan Berant"] year: 2019 venue: "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics" source: "https://aclanthology.org/P19-1485/" doi: "10.18653/v1/p19-1485" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/talmor-2019-multiqa-empirical-investigation-generalization-transfer/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/talmor-2019-multiqa-empirical-investigation-generalization-transfer/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/) / MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension # MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension **Authors:** [Alon Talmor](https://md.research.elamur.ai/free/en/authors/alon-talmor/), [Jonathan Berant](https://md.research.elamur.ai/free/en/authors/jonathan-berant/) · **Year:** 2019 · **Venue:** Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/P19-1485/) · [PDF](https://aclanthology.org/P19-1485.pdf) · [DOI 10.18653/v1/p19-1485](https://doi.org/10.18653/v1/p19-1485) · [OpenAlex](https://openalex.org/W2947567669) **Citations:** 152 citations (openalex-snapshot · 2026-10-14) · [82](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary Many extractive reading comprehension (RC) benchmarks now exist, but it was unclear whether a model trained on one transfers to another, or whether existing data can reduce the work of building a new dataset. Talmor and Berant study this across 10 RC datasets, all converted to a single-turn extractive format. They compare DocQA, which uses GloVe inputs, with a BERT-based reader called BertQA. They test generalization, where no target examples are used, and transfer, where the model is fine-tuned on target data. Models tend to overfit their training set. BERT improves generalization, mostly on Wikipedia and news text and much less on web snippets. Performance drops when source and target differ in document language, question style or the reasoning phenomenon they probe, and the losses grow as these mismatches combine. Pre-training on other RC data still helps even with BERT. Mixing several sources matches target-only training with about three times fewer target examples. The resulting model, MultiQA (BERT-large trained on multiple datasets), reaches state of the art on five datasets. For teams building new QA resources with limited annotation budgets, this is practically useful. ## Key points - The study covers five large datasets (SQuAD, NewsQA, SearchQA, TriviaQA, HotpotQA), each capped at 75K training examples to control for size. It also uses five small ones (CQ, CWQ, ComQA, WikiHop, DROP), and only DROP's 33,573 extractive examples are used for training. - A shared pre-processing pipeline splits documents into chunks, ranks them by tf-idf similarity to the question, and merges them up to a fixed length: 400 tokens for DocQA and 512 wordpieces for BertQA. - Three factors explain much of the generalization gap: differences in context language, differences in question language, and the targeted language phenomenon. - Multi-dataset pre-training improves both generalization and transfer. It saves roughly two thirds of target annotation, and a single BERT-large model can perform near or above the state of the art on all datasets without per-dataset fine-tuning. - The infrastructure is built on AllenNLP and released openly. Unified-format data is at www.tau-nlp.org/multiqa and code at github.com/alontalmor/multiqa. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 82 papers of our library cite it (of the 140884 whose bibliographies we read); 10 of them have a page here, 72 not yet. - [PPT: Pre-trained Prompt Tuning for Few-shot Learning](https://md.research.elamur.ai/free/en/papers/gu-2022-ppt-pre-trained-prompt-tuning/) - [UNIFIEDQA: Crossing Format Boundaries with a Single QA System](https://md.research.elamur.ai/free/en/papers/khashabi-2020-unifiedqa-crossing-format-boundaries-single/) - [A Discrete Hard EM Approach for Weakly Supervised Question Answering](https://md.research.elamur.ai/free/en/papers/min-2019-discrete-hard-em-approach-weakly/) - [Syntactic Data Augmentation Increases Robustness to Inference Heuristics](https://md.research.elamur.ai/free/en/papers/min-2020-syntactic-data-augmentation-increases-robustness/) - [Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work?](https://md.research.elamur.ai/free/en/papers/pruksachatkun-2020-intermediate-task-transfer-learning-pretrained/) - [Training Question Answering Models From Synthetic Data](https://md.research.elamur.ai/free/en/papers/puri-2020-training-question-answering-models-synthetic/) - [Prompting GPT-3 To Be Reliable](https://md.research.elamur.ai/free/en/papers/si-2022-prompting-gpt-3-reliable/) - [Exploring and Predicting Transferability across NLP Tasks](https://md.research.elamur.ai/free/en/papers/vu-2020-exploring-predicting-transferability-across-nlp/) - [SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer](https://md.research.elamur.ai/free/en/papers/vu-2022-spot-better-frozen-model-adaptation/) - [CorefQA: Coreference Resolution as Query-based Span Prediction](https://md.research.elamur.ai/free/en/papers/wu-2020-corefqa-coreference-resolution-as-query/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{talmor2019multiqa, title = {{MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension}}, author = {Talmor, Alon and Berant, Jonathan}, booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics}, year = {2019}, doi = {10.18653/v1/p19-1485}, url = {https://doi.org/10.18653/v1/p19-1485} } ``` - [← previous Multimodal Transformer for Unaligned Multimodal Language Sequences](https://md.research.elamur.ai/free/en/papers/tsai-2019-multimodal-transformer-unaligned-multimodal-language/) - [next → MuST-C: a Multilingual Speech Translation Corpus](https://md.research.elamur.ai/free/en/papers/gangi-2019-must-c-multilingual-speech-translation/) - [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/) ## References 47 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. dataset , even in the presence of BERT representa-text (A DD S ENT variant). M ULTI QA obtained tions. It also leads to substantial reduction in the 66.7 EM and 73.1 F 1 , outperforming BERT-large number of necessary training examples for a fixed (60.4EM, 66.3F1) by a significant margin, and perform… [find](https://scholar.google.com/scholar?q=dataset+%2C+even+in+the+presence+of+BERT+representa%2Dtext+%28A+DD+S+ENT+variant%29%2E+M+ULTI+QA+obtained+tions%2E+It+also+leads+to+substantial+reduction+in+the+66%2E7+EM+and+73%2E1+F+1+%2C+outperforming+BERT%2Dlarge+num) 2. Sun et al. (2018) • Performance over an RC dataset can be improved has recently shown this in a larger experiment for by retrieving web snippets for all questions and multi-choice questions, where they first fine-tuned adding them as examples (context augmentation). BERT on RACE (Lai et al., 2017) a… [find](https://scholar.google.com/scholar?q=Sun+et+al%2E+%282018%29+%E2%80%A2+Performance+over+an+RC+dataset+can+be+improved+has+recently+shown+this+in+a+larger+experiment+for+by+retrieving+web+snippets+for+all+questions+and+multi%2Dchoice+questions%2C+where+the) 3. J. Berant, V. Srikumar, P. Chen, A. V. Linden, B. Harding, B. Huang, P. Clark, and C. D. Manning. 2014. Modeling biological processes for reading comprehension. In Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=J%2E+Berant%2C+V%2E+Srikumar%2C+P%2E+Chen%2C+A%2E+V%2E+Linden%2C+B%2E+Harding%2C+B%2E+Huang%2C+P%2E+Clark%2C+and+C%2E+D%2E+Manning%2E+2014%2E+Modeling+biological+processes+for+reading+comprehension%2E+In+Empirical+Methods+in+Natural+Languag) [our card](https://md.research.elamur.ai/free/en/papers/berant-2014-modeling-biological-processes-reading-comprehension/) 4. K. Bollacker, C. Evans, P. Paritosh, T. Sturge, and J. Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In International Conference on Management of Data (SIGMOD), pages 1247–1250. [find](https://scholar.google.com/scholar?q=K%2E+Bollacker%2C+C%2E+Evans%2C+P%2E+Paritosh%2C+T%2E+Sturge%2C+and+J%2E+Taylor%2E+2008%2E+Freebase%3A+a+collaboratively+created+graph+database+for+structuring+human+knowledge%2E+In+International+Conference+on+Management+of+Da) 5. E. Choi, H. He, M. Iyyer, M. Yatskar, W. Yih, Y. Choi, P. Liang, and L. Zettlemoyer. 2018. Quac: Question answering in context. In Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=E%2E+Choi%2C+H%2E+He%2C+M%2E+Iyyer%2C+M%2E+Yatskar%2C+W%2E+Yih%2C+Y%2E+Choi%2C+P%2E+Liang%2C+and+L%2E+Zettlemoyer%2E+2018%2E+Quac%3A+Question+answering+in+context%2E+In+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/choi-2018-quac-question-answering-context/) 6. R. Jia and P. Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=R%2E+Jia+and+P%2E+Liang%2E+2017%2E+Adversarial+examples+for+evaluating+reading+comprehension+systems%2E+In+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/) 7. A. Chronopoulou, C. Baziotis, and A. Potamianos. 2019. An embarrassingly simple approach for transfer learning from pretrained language models. arXiv preprint arXiv:1902.10547. [find](https://scholar.google.com/scholar?q=A%2E+Chronopoulou%2C+C%2E+Baziotis%2C+and+A%2E+Potamianos%2E+2019%2E+An+embarrassingly+simple+approach+for+transfer+learning+from+pretrained+language+models%2E+arXiv+preprint+arXiv%3A1902%2E10547%2E) 8. M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=M%2E+Joshi%2C+E%2E+Choi%2C+D%2E+Weld%2C+and+L%2E+Zettlemoyer%2E+2017%2E+TriviaQA%3A+A+large+scale+distantly+supervised+challenge+dataset+for+reading+comprehension%2E+In+Association+for+Computational+Linguistics+%28ACL%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/) 9. Y. Chung, H. Lee, and J. Glass. 2018. Supervised and unsupervised transfer learning for question answering. In North American Association for Computational Linguistics (NAACL). [find](https://scholar.google.com/scholar?q=Y%2E+Chung%2C+H%2E+Lee%2C+and+J%2E+Glass%2E+2018%2E+Supervised+and+unsupervised+transfer+learning+for+question+answering%2E+In+North+American+Association+for+Computational+Linguistics+%28NAACL%29%2E) 10. G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683. [find](https://scholar.google.com/scholar?q=G%2E+Lai%2C+Q%2E+Xie%2C+H%2E+Liu%2C+Y%2E+Yang%2C+and+E%2E+Hovy%2E+2017%2E+Race%3A+Large%2Dscale+reading+comprehension+dataset+from+examinations%2E+arXiv+preprint+arXiv%3A1704%2E04683%2E) [our card](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/) 11. C. Clark and M. Gardner. 2018. Simple and effective multi-paragraph reading comprehension. In Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=C%2E+Clark+and+M%2E+Gardner%2E+2018%2E+Simple+and+effective+multi%2Dparagraph+reading+comprehension%2E+In+Association+for+Computational+Linguistics+%28ACL%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/clark-2017-simple-effective-multi-paragraph-reading/) 12. R. Das, S. Dhuliawala, M. Zaheer, and A. McCallum. 2019. Multi-step retriever-reader interaction for scalable open-domain question answering. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=R%2E+Das%2C+S%2E+Dhuliawala%2C+M%2E+Zaheer%2C+and+A%2E+McCallum%2E+2019%2E+Multi%2Dstep+retriever%2Dreader+interaction+for+scalable+open%2Ddomain+question+answering%2E+In+International+Conference+on+Learning+Representations+%28I) 13. Y. Lin, H. Ji, Z. Liu, and M. Sun. 2018. Denoising distantly supervised open-domain question answering. In Association for Computational Linguistics (ACL), volume 1, pages 1736–1745. [find](https://scholar.google.com/scholar?q=Y%2E+Lin%2C+H%2E+Ji%2C+Z%2E+Liu%2C+and+M%2E+Sun%2E+2018%2E+Denoising+distantly+supervised+open%2Ddomain+question+answering%2E+In+Association+for+Computational+Linguistics+%28ACL%29%2C+volume+1%2C+pages+1736%E2%80%931745%2E) 14. X. Liu, P. He, W. Chen, and J. Gao. 2019. Multi-task deep neural networks for natural language understanding. arXiv preprint arXiv:1901.11504. [find](https://scholar.google.com/scholar?q=X%2E+Liu%2C+P%2E+He%2C+W%2E+Chen%2C+and+J%2E+Gao%2E+2019%2E+Multi%2Dtask+deep+neural+networks+for+natural+language+understanding%2E+arXiv+preprint+arXiv%3A1901%2E11504%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2019-multi-task-deep-neural-networks/) 15. J. Devlin, M. Chang, K. Lee, and K. Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Association for Computational Linguistics (NAACL). [find](https://scholar.google.com/scholar?q=J%2E+Devlin%2C+M%2E+Chang%2C+K%2E+Lee%2C+and+K%2E+Toutanova%2E+2019%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+North+American+Association+for+Computational+Linguistics+%28NAAC) 16. K. Luo1, F. Lin1, X., L. Kenny, and Q.Zhu1. 2018. Knowledge base question answering via encoding of complex query graphs. In Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=K%2E+Luo1%2C+F%2E+Lin1%2C+X%2E%2C+L%2E+Kenny%2C+and+Q%2EZhu1%2E+2018%2E+Knowledge+base+question+answering+via+encoding+of+complex+query+graphs%2E+In+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2E) 17. M. Ding, C. Zhou, Q. Chen, H. Yang, and J. Tang. 2019. Cognitive graph for multi-hop reading comprehension at scale. In Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=M%2E+Ding%2C+C%2E+Zhou%2C+Q%2E+Chen%2C+H%2E+Yang%2C+and+J%2E+Tang%2E+2019%2E+Cognitive+graph+for+multi%2Dhop+reading+comprehension+at+scale%2E+In+Association+for+Computational+Linguistics+%28ACL%29%2E) 18. B. McCann, N. S. Keskar, C. Xiong, and R. Socher. 2018. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730. [find](https://scholar.google.com/scholar?q=B%2E+McCann%2C+N%2E+S%2E+Keskar%2C+C%2E+Xiong%2C+and+R%2E+Socher%2E+2018%2E+The+natural+language+decathlon%3A+Multitask+learning+as+question+answering%2E+arXiv+preprint+arXiv%3A1806%2E08730%2E) [our card](https://md.research.elamur.ai/free/en/papers/mccann-2018-natural-language-decathlon-multitask-learning/) 19. D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. 2019. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In North American Association for Computational Linguistics (NAACL). [find](https://scholar.google.com/scholar?q=D%2E+Dua%2C+Y%2E+Wang%2C+P%2E+Dasigi%2C+G%2E+Stanovsky%2C+S%2E+Singh%2C+and+M%2E+Gardner%2E+2019%2E+Drop%3A+A+reading+comprehension+benchmark+requiring+discrete+reasoning+over+paragraphs%2E+In+North+American+Association+for+Comput) [our card](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/) 20. S. Min, M. Seo, and H. Hajishirzi. 2017. Question answering through transfer learning from large fine-grained supervision data. In Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=S%2E+Min%2C+M%2E+Seo%2C+and+H%2E+Hajishirzi%2E+2017%2E+Question+answering+through+transfer+learning+from+large+fine%2Dgrained+supervision+data%2E+In+Association+for+Computational+Linguistics+%28ACL%29%2E) 21. M. Dunn, , L. Sagun, M. Higgins, U. Guney, V. Cirik, and K. Cho. 2017. SearchQA: A new Q&A dataset augmented with context from a search engine. arXiv. [find](https://scholar.google.com/scholar?q=M%2E+Dunn%2C+%2C+L%2E+Sagun%2C+M%2E+Higgins%2C+U%2E+Guney%2C+V%2E+Cirik%2C+and+K%2E+Cho%2E+2017%2E+SearchQA%3A+A+new+Q%26A+dataset+augmented+with+context+from+a+search+engine%2E+arXiv%2E) [our card](https://md.research.elamur.ai/free/en/papers/dunn-2017-searchqa-new-q-dataset-augmented/) 22. T. Nguyen, M. Rosenberg, X. Song, J. Gao, S. Tiwary, R. Majumder, and L. Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In Workshop on Cognitive Computing at NIPS. [find](https://scholar.google.com/scholar?q=T%2E+Nguyen%2C+M%2E+Rosenberg%2C+X%2E+Song%2C+J%2E+Gao%2C+S%2E+Tiwary%2C+R%2E+Majumder%2C+and+L%2E+Deng%2E+2016%2E+MS+MARCO%3A+A+human+generated+machine+reading+comprehension+dataset%2E+In+Workshop+on+Cognitive+Computing+at+NIPS%2E) [our card](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/) 23. T. M. Fruchterman and E. M. Reingold. 1991. Graph drawing by force-directed placement. Software: Practice and experience, 21(11):1129–1164. [find](https://scholar.google.com/scholar?q=T%2E+M%2E+Fruchterman+and+E%2E+M%2E+Reingold%2E+1991%2E+Graph+drawing+by+force%2Ddirected+placement%2E+Software%3A+Practice+and+experience%2C+21%2811%29%3A1129%E2%80%931164%2E) 24. M. Gardner, J. Grus, M. Neumann, O. Tafjord, P. Dasigi, N. Liu, M. Peters, M. Schmitz, and L. Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. arXiv preprint arXiv:1803.07640. [find](https://scholar.google.com/scholar?q=M%2E+Gardner%2C+J%2E+Grus%2C+M%2E+Neumann%2C+O%2E+Tafjord%2C+P%2E+Dasigi%2C+N%2E+Liu%2C+M%2E+Peters%2C+M%2E+Schmitz%2C+and+L%2E+Zettlemoyer%2E+2018%2E+AllenNLP%3A+A+deep+semantic+natural+language+processing+platform%2E+arXiv+preprint+arXiv%3A18) [our card](https://md.research.elamur.ai/free/en/papers/gardner-2018-allennlp-deep-semantic-natural-language/) 25. K. M. Hermann, T. Koisk, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems (NeurIPS). [find](https://scholar.google.com/scholar?q=K%2E+M%2E+Hermann%2C+T%2E+Koisk%2C+E%2E+Grefenstette%2C+L%2E+Espeholt%2C+W%2E+Kay%2C+M%2E+Suleyman%2C+and+P%2E+Blunsom%2E+2015%2E+Teaching+machines+to+read+and+comprehend%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NeurIP) [our card](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/) 26. M. Hu, Y. Peng, F. Wei, Z. Huang, D. Li, N. Yang, and M. Zhou. 2018. Attention-guided answer distillation for machine reading comprehension. In Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=M%2E+Hu%2C+Y%2E+Peng%2C+F%2E+Wei%2C+Z%2E+Huang%2C+D%2E+Li%2C+N%2E+Yang%2C+and+M%2E+Zhou%2E+2018%2E+Attention%2Dguided+answer+distillation+for+machine+reading+comprehension%2E+In+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29) 27. J. Pennington, R. Socher, and C. D. Manning. 2014. GloVe: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. [find](https://scholar.google.com/scholar?q=J%2E+Pennington%2C+R%2E+Socher%2C+and+C%2E+D%2E+Manning%2E+2014%2E+GloVe%3A+Global+vectors+for+word+representation%2E+In+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+pages+1532%E2%80%931543%2E) 28. M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. 2018. Deep contextualized word representations. In North American Association for Computational Linguistics (NAACL). [find](https://scholar.google.com/scholar?q=M%2E+E%2E+Peters%2C+M%2E+Neumann%2C+M%2E+Iyyer%2C+M%2E+Gardner%2C+C%2E+Clark%2C+K%2E+Lee%2C+and+L%2E+Zettlemoyer%2E+2018%2E+Deep+contextualized+word+representations%2E+In+North+American+Association+for+Computational+Linguistics+%28NAACL) 29. J. Phang, T. Fevry, and S. R. Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088. [find](https://scholar.google.com/scholar?q=J%2E+Phang%2C+T%2E+Fevry%2C+and+S%2E+R%2E+Bowman%2E+2018%2E+Sentence+encoders+on+stilts%3A+Supplementary+training+on+intermediate+labeled%2Ddata+tasks%2E+arXiv+preprint+arXiv%3A1811%2E01088%2E) [our card](https://md.research.elamur.ai/free/en/papers/phang-2018-sentence-encoders-stilts-supplementary-training/) 30. A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. 2018. Improving language understanding by generative pre-training. Technical report, OpenAI. [find](https://scholar.google.com/scholar?q=A%2E+Radford%2C+K%2E+Narasimhan%2C+T%2E+Salimans%2C+and+I%2E+Sutskever%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+Technical+report%2C+OpenAI%2E) 31. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=P%2E+Rajpurkar%2C+J%2E+Zhang%2C+K%2E+Lopyrev%2C+and+P%2E+Liang%2E+2016%2E+SQuAD%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/) 32. Y. Xu, X. Liu, Y. Shen, J. Liu, and J. Gao. 2019. Multi-task learning with sample re-weighting for machine reading comprehension. In North American Association for Computational Linguistics (NAACL). [find](https://scholar.google.com/scholar?q=Y%2E+Xu%2C+X%2E+Liu%2C+Y%2E+Shen%2C+J%2E+Liu%2C+and+J%2E+Gao%2E+2019%2E+Multi%2Dtask+learning+with+sample+re%2Dweighting+for+machine+reading+comprehension%2E+In+North+American+Association+for+Computational+Linguistics+%28NAACL%29%2E) 33. S. Reddy, D. Chen, and C. D. Manning. 2018. Coqa: A conversational question answering challenge. arXiv preprint arXiv:1808.07042. [find](https://scholar.google.com/scholar?q=S%2E+Reddy%2C+D%2E+Chen%2C+and+C%2E+D%2E+Manning%2E+2018%2E+Coqa%3A+A+conversational+question+answering+challenge%2E+arXiv+preprint+arXiv%3A1808%2E07042%2E) [our card](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/) 34. Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Z%2E+Yang%2C+P%2E+Qi%2C+S%2E+Zhang%2C+Y%2E+Bengio%2C+W%2E+W%2E+Cohen%2C+R%2E+Salakhutdinov%2C+and+C%2E+D%2E+Manning%2E+2018%2E+HotpotQA%3A+A+dataset+for+diverse%2C+explainable+multi%2Dhop+question+answering%2E+In+Empirical+Methods+in+Natural+) [our card](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/) 35. M. Richardson, C. J. Burges, and E. Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP), pages 193–203. [find](https://scholar.google.com/scholar?q=M%2E+Richardson%2C+C%2E+J%2E+Burges%2C+and+E%2E+Renshaw%2E+2013%2E+Mctest%3A+A+challenge+dataset+for+the+open%2Ddomain+machine+comprehension+of+text%2E+In+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+pages+193) [our card](https://md.research.elamur.ai/free/en/papers/missing-2013-richardson-mctest-challenge-dataset-open-domain/) 36. M. Seo, A. Kembhavi, A. Farhadi, and H. Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv. [find](https://scholar.google.com/scholar?q=M%2E+Seo%2C+A%2E+Kembhavi%2C+A%2E+Farhadi%2C+and+H%2E+Hajishirzi%2E+2016%2E+Bidirectional+attention+flow+for+machine+comprehension%2E+arXiv%2E) [our card](https://md.research.elamur.ai/free/en/papers/seo-2016-bidirectional-attention-flow-machine-comprehension/) 37. D. Yogatama, C. de M. d’Autume, J. Connor, T. Kocisky, M. Chrzanowski, L. Kong, A. Lazaridou, W. Ling, L. Yu, C. Dyer, et al. 2019. Learning and evaluating general linguistic intelligence. arXiv preprint arXiv:1901.11373. [find](https://scholar.google.com/scholar?q=D%2E+Yogatama%2C+C%2E+de+M%2E+d%E2%80%99Autume%2C+J%2E+Connor%2C+T%2E+Kocisky%2C+M%2E+Chrzanowski%2C+L%2E+Kong%2C+A%2E+Lazaridou%2C+W%2E+Ling%2C+L%2E+Yu%2C+C%2E+Dyer%2C+et+al%2E+2019%2E+Learning+and+evaluating+general+linguistic+intelligence%2E+arXiv+prepr) 38. K. Sun, D. Yu, D. Yu, and C. Cardie. 2018. Improving machine reading comprehension with general reading strategies. arXiv preprint arXiv:1810.13441. [find](https://scholar.google.com/scholar?q=K%2E+Sun%2C+D%2E+Yu%2C+D%2E+Yu%2C+and+C%2E+Cardie%2E+2018%2E+Improving+machine+reading+comprehension+with+general+reading+strategies%2E+arXiv+preprint+arXiv%3A1810%2E13441%2E) 39. A. Talmor and J. Berant. 2018a. Repartitioning of the complexwebquestions dataset. arXiv preprint arXiv:1807.09623. — [arXiv:1807.09623](https://arxiv.org/abs/1807.09623) [find](https://scholar.google.com/scholar?q=A%2E+Talmor+and+J%2E+Berant%2E+2018a%2E+Repartitioning+of+the+complexwebquestions+dataset%2E+arXiv+preprint+arXiv%3A1807%2E09623%2E+%E2%80%94+%5BarXiv%3A1807%2E09623%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1807%2E09623%29) 40. A. Talmor and J. Berant. 2018b. Repartitioning of the complexwebquestions dataset. arXiv preprint arXiv:1807.09623. — [arXiv:1807.09623](https://arxiv.org/abs/1807.09623) [find](https://scholar.google.com/scholar?q=A%2E+Talmor+and+J%2E+Berant%2E+2018b%2E+Repartitioning+of+the+complexwebquestions+dataset%2E+arXiv+preprint+arXiv%3A1807%2E09623%2E+%E2%80%94+%5BarXiv%3A1807%2E09623%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1807%2E09623%29) 41. A. Talmor and J. Berant. 2018c. The web as knowledge-base for answering complex questions. In North American Association for Computational Linguistics (NAACL). [find](https://scholar.google.com/scholar?q=A%2E+Talmor+and+J%2E+Berant%2E+2018c%2E+The+web+as+knowledge%2Dbase+for+answering+complex+questions%2E+In+North+American+Association+for+Computational+Linguistics+%28NAACL%29%2E) 42. A. Talmor, M. Geva, and J. Berant. 2017. Evaluating semantic parsing against a simple web-based question answering model. In *SEM. [find](https://scholar.google.com/scholar?q=A%2E+Talmor%2C+M%2E+Geva%2C+and+J%2E+Berant%2E+2017%2E+Evaluating+semantic+parsing+against+a+simple+web%2Dbased+question+answering+model%2E+In+%2ASEM%2E) 43. Y. Tay, L. Tuan, S. Hui, and J. Su. 2018. Densely connected attention propagation for reading comprehension. In Advances in Neural Information Processing Systems (NeurIPS). [find](https://scholar.google.com/scholar?q=Y%2E+Tay%2C+L%2E+Tuan%2C+S%2E+Hui%2C+and+J%2E+Su%2E+2018%2E+Densely+connected+attention+propagation+for+reading+comprehension%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NeurIPS%29%2E) 44. A. Trischler, T. Wang, X. Yuan, J. Harris, A. Sordoni, P. Bachman, and K. Suleman. 2017. NewsQA: A machine comprehension dataset. In Workshop on Representation Learning for NLP. [find](https://scholar.google.com/scholar?q=A%2E+Trischler%2C+T%2E+Wang%2C+X%2E+Yuan%2C+J%2E+Harris%2C+A%2E+Sordoni%2C+P%2E+Bachman%2C+and+K%2E+Suleman%2E+2017%2E+NewsQA%3A+A+machine+comprehension+dataset%2E+In+Workshop+on+Representation+Learning+for+NLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/trischler-2017-newsqa-machine-comprehension-dataset/) 45. A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. 2019. Glue: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=A%2E+Wang%2C+A%2E+Singh%2C+J%2E+Michael%2C+F%2E+Hill%2C+O%2E+Levy%2C+and+S%2E+R%2E+Bowman%2E+2019%2E+Glue%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+In+International+Conference+on+Learning+) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/) 46. W. Wang, M. Yan, and C. Wu. 2018. Multi-granularity hierarchical attention fusion networks for reading comprehension and question answering. In Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=W%2E+Wang%2C+M%2E+Yan%2C+and+C%2E+Wu%2E+2018%2E+Multi%2Dgranularity+hierarchical+attention+fusion+networks+for+reading+comprehension+and+question+answering%2E+In+Association+for+Computational+Linguistics+%28ACL%29%2E) 47. J. Welbl, P. Stenetorp, and S. Riedel. 2017. Constructing datasets for multi-hop reading comprehension across documents. arXiv preprint arXiv:1710.06481. [find](https://scholar.google.com/scholar?q=J%2E+Welbl%2C+P%2E+Stenetorp%2C+and+S%2E+Riedel%2E+2017%2E+Constructing+datasets+for+multi%2Dhop+reading+comprehension+across+documents%2E+arXiv+preprint+arXiv%3A1710%2E06481%2E) [our card](https://md.research.elamur.ai/free/en/papers/welbl-2018-constructing-datasets-multi-hop-reading/)