---
title: "Training Question Answering Models From — Puri 2020"
description: "Extractive QA systems depend on costly human-annotated data, and question-answer pairs produced automatically have traditionally been much weaker than…"
authors: ["Raul Puri", "Ryan Spring", "Mohammad Shoeybi", "Mostofa Patwary", "Bryan Catanzaro"]
year: 2020
venue: "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)"
source: "https://aclanthology.org/2020.emnlp-main.468/"
doi: "10.18653/v1/2020.emnlp-main.468"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/puri-2020-training-question-answering-models-synthetic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/puri-2020-training-question-answering-models-synthetic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/) / Training Question Answering Models From Synthetic Data

# Training Question Answering Models From Synthetic Data

**Authors:** [Raul Puri](https://md.research.elamur.ai/free/en/authors/raul-puri/), Ryan Spring, [Mohammad Shoeybi](https://md.research.elamur.ai/free/en/authors/mohammad-shoeybi/), [Mostofa Patwary](https://md.research.elamur.ai/free/en/authors/mostofa-patwary/), [Bryan Catanzaro](https://md.research.elamur.ai/free/en/authors/bryan-catanzaro/) · **Year:** 2020 · **Venue:** Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.emnlp-main.468/) · [PDF](https://aclanthology.org/2020.emnlp-main.468.pdf) · [DOI 10.18653/v1/2020.emnlp-main.468](https://doi.org/10.18653/v1/2020.emnlp-main.468) · [OpenAlex](https://openalex.org/W3007306981)

**Citations:** 125 citations (OpenAlex, 2026-10-14) · [81](https://md.research.elamur.ai/free/en/papers/puri-2020-training-question-answering-models-synthetic/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
Extractive QA systems depend on costly human-annotated data, and question-answer pairs produced automatically have traditionally been much weaker than those written by people. The authors tackle this gap with large pretrained transformers. Their pipeline has three stages:

1. A BERT-style model, fed only the passage, selects candidate answer spans.
2. A finetuned GPT-2 model writes a question conditioned on the passage and the chosen span.
3. A QA model applies roundtrip filtering, keeping only questions it answers with the original span.

They extend the filtering stage by sampling two questions per answer, which recovers valid pairs that a single attempt would lose. They also study how results depend on model scale (up to 8.3B parameters), pretraining quality, amount of generated data and design choices. Synthetic data generated from unlabeled Wikipedia lets a BERT-345M reader slightly exceed the same reader trained on human SQuAD1.1 annotations. Even when the source passages are themselves produced by GPT-2, performance holds. This suggests synthetic data can replace or supplement manual labeling, a promising direction for low-resource QA.

## Key points
- On the SQuAD1.1 dev set, BERT-345M trained only on synthetic pairs from Wikipedia scores 88.4 EM / 94.1 F1. The fully supervised baseline scores 87.7 EM / 94.0 F1. Further finetuning on real SQuAD data raises EM to 89.4.
- The fully synthetic setting uses passages generated by an 8.3B GPT-2 model (about 17.4M questions) and still reaches 88.4 EM / 93.9 F1.
- Ablations show that answer candidate quality is crucial: matching the dataset's answer distribution adds 19.8 EM points.
- Roundtrip filtering adds 7.2 EM over unfiltered synthetic data. Overgeneration (one top-k sample with k=40 and one nucleus sample with p=0.9) adds roughly one more point.
- Models were built with the Megatron-LM codebase and HuggingFace transformers. On SQuAD2.0 the method also beats earlier synthetic-data approaches in EM.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/)

## Institutions

- [Nvidia](https://md.research.elamur.ai/free/en/companies/nvidia/) — Company, Santa Clara, United States · 11 works · 4 351 citations
- [Rice University](https://md.research.elamur.ai/free/en/universities/rice-university/) — University, Houston, United States · 4 works · 1 595 citations

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{puri2020training,
  title = {{Training Question Answering Models From Synthetic Data}},
  author = {Puri, Raul and Spring, Ryan and Shoeybi, Mohammad and Patwary, Mostofa and Catanzaro, Bryan},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.468},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.468}
}
```

- [← previous Reasoning with Multimodal Sarcastic Tweets via Modeling Cross-Modality Contrast and Semantic Association](https://md.research.elamur.ai/free/en/papers/xu-2020-reasoning-multimodal-sarcastic-tweets-modeling/)
- [next → Entities as Experts: Sparse Memory Access with Entity Supervision](https://md.research.elamur.ai/free/en/papers/fevry-2020-entities-as-experts-sparse-memory/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/)

## References

43 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. 2019a. Synthetic qa corpora generation with roundtrip consistency. arXiv preprint arXiv:1906.05416. [find](https://scholar.google.com/scholar?q=Chris+Alberti%2C+Daniel+Andor%2C+Emily+Pitler%2C+Jacob+Devlin%2C+and+Michael+Collins%2E+2019a%2E+Synthetic+qa+corpora+generation+with+roundtrip+consistency%2E+arXiv+preprint+arXiv%3A1906%2E05416%2E) [our card](https://md.research.elamur.ai/free/en/papers/alberti-2019-synthetic-qa-corpora-generation-roundtrip/)
2. Chris Alberti, Kenton Lee, and Michael Collins. 2019b. [find](https://scholar.google.com/scholar?q=Chris+Alberti%2C+Kenton+Lee%2C+and+Michael+Collins%2E+2019b%2E)
3. A bert baseline for the natural questions. preprint arXiv:1901.08634. arXiv Alex Boyd, Raul Puri, Mohammad Shoeybi, Mostofa Patwary, and Bryan Catanzaro. 2020. Large scale multi-actor generative dialog modeling. arXiv preprint arXiv:2005.06114. [find](https://scholar.google.com/scholar?q=A+bert+baseline+for+the+natural+questions%2E+preprint+arXiv%3A1901%2E08634%2E+arXiv+Alex+Boyd%2C+Raul+Puri%2C+Mohammad+Shoeybi%2C+Mostofa+Patwary%2C+and+Bryan+Catanzaro%2E+2020%2E+Large+scale+multi%2Dactor+generative+dialo)
4. Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. 2009. Semi-supervised learning (chapelle, o. et al., eds.; 2006)[book reviews]. IEEE Transactions on Neural Networks, 20(3):542–542. [find](https://scholar.google.com/scholar?q=Olivier+Chapelle%2C+Bernhard+Scholkopf%2C+and+Alexander+Zien%2E+2009%2E+Semi%2Dsupervised+learning+%28chapelle%2C+o%2E+et+al%2E%2C+eds%2E%3B+2006%29%5Bbook+reviews%5D%2E+IEEE+Transactions+on+Neural+Networks%2C+20%283%29%3A542%E2%80%93542%2E)
5. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044. [find](https://scholar.google.com/scholar?q=Christopher+Clark%2C+Kenton+Lee%2C+Ming%2DWei+Chang%2C+Tom+Kwiatkowski%2C+Michael+Collins%2C+and+Kristina+Toutanova%2E+2019%2E+Boolq%3A+Exploring+the+surprising+difficulty+of+natural+yes%2Fno+questions%2E+arXiv+preprint+ar) [our card](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/)
6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2018%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+arXiv+preprint+arXiv%3A1810%2E04805%2E) [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
7. Bhuwan Dhingra, Danish Pruthi, and Dheeraj Rajagopal. 2018. Simple and effective semi-supervised question answering. arXiv preprint arXiv:1804.00720. [find](https://scholar.google.com/scholar?q=Bhuwan+Dhingra%2C+Danish+Pruthi%2C+and+Dheeraj+Rajagopal%2E+2018%2E+Simple+and+effective+semi%2Dsupervised+question+answering%2E+arXiv+preprint+arXiv%3A1804%2E00720%2E)
8. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. arXiv preprint arXiv:1905.03197. [find](https://scholar.google.com/scholar?q=Li+Dong%2C+Nan+Yang%2C+Wenhui+Wang%2C+Furu+Wei%2C+Xiaodong+Liu%2C+Yu+Wang%2C+Jianfeng+Gao%2C+Ming+Zhou%2C+and+Hsiao%2DWuen+Hon%2E+2019%2E+Unified+language+model+pre%2Dtraining+for+natural+language+understanding+and+generatio) [our card](https://md.research.elamur.ai/free/en/papers/dong-2019-unified-language-model-pre-training/)
9. Xinya Du, Junru Shao, and Claire Cardie. 2017. Learning to ask: Neural question generation for reading comprehension. arXiv preprint arXiv:1705.00106. [find](https://scholar.google.com/scholar?q=Xinya+Du%2C+Junru+Shao%2C+and+Claire+Cardie%2E+2017%2E+Learning+to+ask%3A+Neural+question+generation+for+reading+comprehension%2E+arXiv+preprint+arXiv%3A1705%2E00106%2E) [our card](https://md.research.elamur.ai/free/en/papers/du-2017-learning-ask-neural-question-generation/)
10. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv preprint arXiv:1903.00161. [find](https://scholar.google.com/scholar?q=Dheeru+Dua%2C+Yizhong+Wang%2C+Pradeep+Dasigi%2C+Gabriel+Stanovsky%2C+Sameer+Singh%2C+and+Matt+Gardner%2E+2019%2E+Drop%3A+A+reading+comprehension+benchmark+requiring+discrete+reasoning+over+paragraphs%2E+arXiv+preprint+) [our card](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/)
11. Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017. Searchqa: A new q&a dataset augmented with context from a search engine. arXiv preprint arXiv:1704.05179. [find](https://scholar.google.com/scholar?q=Matthew+Dunn%2C+Levent+Sagun%2C+Mike+Higgins%2C+V+Ugur+Guney%2C+Volkan+Cirik%2C+and+Kyunghyun+Cho%2E+2017%2E+Searchqa%3A+A+new+q%26a+dataset+augmented+with+context+from+a+search+engine%2E+arXiv+preprint+arXiv%3A1704%2E05179%2E) [our card](https://md.research.elamur.ai/free/en/papers/dunn-2017-searchqa-new-q-dataset-augmented/)
12. Aaron Gokaslan and Vanya Cohen. 2019. Openwebtext corpus. [find](https://scholar.google.com/scholar?q=Aaron+Gokaslan+and+Vanya+Cohen%2E+2019%2E+Openwebtext+corpus%2E)
13. Michael Heilman and Noah A Smith. 2010a. Good question! statistical ranking for question generation. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 609–617. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Michael+Heilman+and+Noah+A+Smith%2E+2010a%2E+Good+question%21+statistical+ranking+for+question+generation%2E+In+Human+Language+Technologies%3A+The+2010+Annual+Conference+of+the+North+American+Chapter+of+the+Ass)
14. Michael Heilman and Noah A Smith. 2010b. Good question! statistical ranking for question generation. In Human Language Technologies: The 2010 [find](https://scholar.google.com/scholar?q=Michael+Heilman+and+Noah+A+Smith%2E+2010b%2E+Good+question%21+statistical+ranking+for+question+generation%2E+In+Human+Language+Technologies%3A+The+2010)
15. Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 609–617. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Annual+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics%2C+pages+609%E2%80%93617%2E+Association+for+Computational+Linguistics%2E)
16. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J+Liu%2E+2019%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtext) [our card](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/)
17. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2019. Spanbert: Improving pre-training by representing and predicting spans. CoRR, abs/1907.10529. [find](https://scholar.google.com/scholar?q=Mandar+Joshi%2C+Danqi+Chen%2C+Yinhan+Liu%2C+Daniel+S%2E+Weld%2C+Luke+Zettlemoyer%2C+and+Omer+Levy%2E+2019%2E+Spanbert%3A+Improving+pre%2Dtraining+by+representing+and+predicting+spans%2E+CoRR%2C+abs%2F1907%2E10529%2E) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2019-spanbert-improving-pre-training-representing/)
18. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Robin+Jia%2C+and+Percy+Liang%2E+2018%2E+Know+what+you+don%E2%80%99t+know%3A+Unanswerable+questions+for+squad%2E+arXiv+preprint+arXiv%3A1806%2E03822%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/)
19. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551. [find](https://scholar.google.com/scholar?q=Mandar+Joshi%2C+Eunsol+Choi%2C+Daniel+S+Weld%2C+and+Luke+Zettlemoyer%2E+2017%2E+Triviaqa%3A+A+large+scale+distantly+supervised+challenge+dataset+for+reading+comprehension%2E+arXiv+preprint+arXiv%3A1705%2E03551%2E) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/)
20. Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266. [find](https://scholar.google.com/scholar?q=Siva+Reddy%2C+Danqi+Chen%2C+and+Christopher+D+Manning%2E+2019%2E+Coqa%3A+A+conversational+question+answering+challenge%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+7%3A249%E2%80%93266%2E) [our card](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/)
21. Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858. [find](https://scholar.google.com/scholar?q=Nitish+Shirish+Keskar%2C+Bryan+McCann%2C+Lav+R+Varshney%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+2019%2E+Ctrl%3A+A+conditional+transformer+language+model+for+controllable+generation%2E+arXiv+preprint+arXiv%3A1909%2E0585) [our card](https://md.research.elamur.ai/free/en/papers/keskar-2019-ctrl-conditional-transformer-language-model/)
22. Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. 2014. Semi-supervised learning with deep generative models. In Advances in neural information processing systems, pages 3581–3589. [find](https://scholar.google.com/scholar?q=Durk+P+Kingma%2C+Shakir+Mohamed%2C+Danilo+Jimenez+Rezende%2C+and+Max+Welling%2E+2014%2E+Semi%2Dsupervised+learning+with+deep+generative+models%2E+In+Advances+in+neural+information+processing+systems%2C+pages+3581%E2%80%93358)
23. Tassilo Klein and Moin Nabi. 2019. Learning to answer by learning to ask: Getting the best of gpt-2 and bert worlds. arXiv preprint arXiv:1911.02365. — [arXiv:1911.02365](https://arxiv.org/abs/1911.02365) [find](https://scholar.google.com/scholar?q=Tassilo+Klein+and+Moin+Nabi%2E+2019%2E+Learning+to+answer+by+learning+to+ask%3A+Getting+the+best+of+gpt%2D2+and+bert+worlds%2E+arXiv+preprint+arXiv%3A1911%2E02365%2E+%E2%80%94+%5BarXiv%3A1911%2E02365%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1911%2E02)
24. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942. [find](https://scholar.google.com/scholar?q=Zhenzhong+Lan%2C+Mingda+Chen%2C+Sebastian+Goodman%2C+Kevin+Gimpel%2C+Piyush+Sharma%2C+and+Radu+Soricut%2E+2019%2E+Albert%3A+A+lite+bert+for+self%2Dsupervised+learning+of+language+representations%2E+arXiv+preprint+arXiv%3A1)
25. Patrick Lewis, Ludovic Denoyer, and Sebastian Riedel. 2019. Unsupervised question answering by cloze translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4896–4910, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Patrick+Lewis%2C+Ludovic+Denoyer%2C+and+Sebastian+Riedel%2E+2019%2E+Unsupervised+question+answering+by+cloze+translation%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguis)
26. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+a)
27. Ilya Loshchilov and Frank Hutter. 2018. Fixing weight decay regularization in adam. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2018%2E+Fixing+weight+decay+regularization+in+adam%2E)
28. Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. 2017. Mixed precision training. arXiv preprint arXiv:1710.03740. [find](https://scholar.google.com/scholar?q=Paulius+Micikevicius%2C+Sharan+Narang%2C+Jonah+Alben%2C+Gregory+Diamos%2C+Erich+Elsen%2C+David+Garcia%2C+Boris+Ginsburg%2C+Michael+Houston%2C+Oleksii+Kuchaiev%2C+Ganesh+Venkatesh%2C+et+al%2E+2017%2E+Mixed+precision+training%2E)
29. Raul Puri and Bryan Catanzaro. 2019. Zero-shot text classification with generative language models. arXiv preprint arXiv:1912.10165. — [arXiv:1912.10165](https://arxiv.org/abs/1912.10165) [find](https://scholar.google.com/scholar?q=Raul+Puri+and+Bryan+Catanzaro%2E+2019%2E+Zero%2Dshot+text+classification+with+generative+language+models%2E+arXiv+preprint+arXiv%3A1912%2E10165%2E+%E2%80%94+%5BarXiv%3A1912%2E10165%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1912%2E10165%29)
30. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8). [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+Blog%2C+1%288%29%2E)
31. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053. [find](https://scholar.google.com/scholar?q=Mohammad+Shoeybi%2C+Mostofa+Patwary%2C+Raul+Puri%2C+Patrick+LeGresley%2C+Jared+Casper%2C+and+Bryan+Catanzaro%2E+2019%2E+Megatron%2Dlm%3A+Training+multi%2Dbillion+parameter+language+models+using+gpu+model+parallelism%2E+arX)
32. Sandeep Subramanian, Tong Wang, Xingdi Yuan, and Adam Trischler. 2017. Neural models for key phrase detection and question generation. CoRR, abs/1706.04560. — [arXiv:1706.04560](https://arxiv.org/abs/1706.04560) [find](https://scholar.google.com/scholar?q=Sandeep+Subramanian%2C+Tong+Wang%2C+Xingdi+Yuan%2C+and+Adam+Trischler%2E+2017%2E+Neural+models+for+key+phrase+detection+and+question+generation%2E+CoRR%2C+abs%2F1706%2E04560%2E+%E2%80%94+%5BarXiv%3A1706%2E04560%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F)
33. Alon Talmor and Jonathan Berant. 2018. Repartitioning of the complexwebquestions dataset. arXiv preprint arXiv:1807.09623. — [arXiv:1807.09623](https://arxiv.org/abs/1807.09623) [find](https://scholar.google.com/scholar?q=Alon+Talmor+and+Jonathan+Berant%2E+2018%2E+Repartitioning+of+the+complexwebquestions+dataset%2E+arXiv+preprint+arXiv%3A1807%2E09623%2E+%E2%80%94+%5BarXiv%3A1807%2E09623%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1807%2E09623%29)
34. Alon Talmor and Jonathan Berant. 2019. MultiQA: An empirical investigation of generalization and transfer in reading comprehension. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4911–4921, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alon+Talmor+and+Jonathan+Berant%2E+2019%2E+MultiQA%3A+An+empirical+investigation+of+generalization+and+transfer+in+reading+comprehension%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Com) [our card](https://md.research.elamur.ai/free/en/papers/talmor-2019-multiqa-empirical-investigation-generalization-transfer/)
35. Trieu H Trinh and Quoc V Le. 2018. A simple method for commonsense reasoning. arXiv preprint arXiv:1806.02847. [find](https://scholar.google.com/scholar?q=Trieu+H+Trinh+and+Quoc+V+Le%2E+2018%2E+A+simple+method+for+commonsense+reasoning%2E+arXiv+preprint+arXiv%3A1806%2E02847%2E)
36. Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association for Computational Linguistics, 6:287–302. [find](https://scholar.google.com/scholar?q=Johannes+Welbl%2C+Pontus+Stenetorp%2C+and+Sebastian+Riedel%2E+2018%2E+Constructing+datasets+for+multi%2Dhop+reading+comprehension+across+documents%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C) [our card](https://md.research.elamur.ai/free/en/papers/welbl-2018-constructing-datasets-multi-hop-reading/)
37. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. [find](https://scholar.google.com/scholar?q=Thomas+Wolf%2C+Lysandre+Debut%2C+Victor+Sanh%2C+Julien+Chaumond%2C+Clement+Delangue%2C+Anthony+Moi%2C+Pier%2Dric+Cistac%2C+Tim+Rault%2C+Re%CC%81mi+Louf%2C+Morgan+Funtowicz%2C+and+Jamie+Brew%2E+2019%2E+Huggingface%E2%80%99s+transformers%3A+St) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
38. Qizhe Xie, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. 2019. Self-training with noisy student improves imagenet classification. arXiv preprint arXiv:1911.04252. — [arXiv:1911.04252](https://arxiv.org/abs/1911.04252) [find](https://scholar.google.com/scholar?q=Qizhe+Xie%2C+Eduard+Hovy%2C+Minh%2DThang+Luong%2C+and+Quoc+V+Le%2E+2019%2E+Self%2Dtraining+with+noisy+student+improves+imagenet+classification%2E+arXiv+preprint+arXiv%3A1911%2E04252%2E+%E2%80%94+%5BarXiv%3A1911%2E04252%5D%28https%3A%2F%2Farxiv%2Eor)
39. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Peng+Qi%2C+Saizheng+Zhang%2C+Yoshua+Bengio%2C+William+W+Cohen%2C+Ruslan+Salakhutdinov%2C+and+Christopher+D+Manning%2E+2018%2E+Hotpotqa%3A+A+dataset+for+diverse%2C+explainable+multi%2Dhop+question+answering%2E+) [our card](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/)
40. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In Advances in Neural Information Processing Systems, pages 9051–9062. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Hannah+Rashkin%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+Franziska+Roesner%2C+and+Yejin+Choi%2E+2019%2E+Defending+against+neural+fake+news%2E+In+Advances+in+Neural+Information+Processing+System) [our card](https://md.research.elamur.ai/free/en/papers/zellers-2019-defending-against-neural-fake-news/)
41. Haichao Zhu, Li Dong, Furu Wei, Wenhui Wang, Bing Qin, and Ting Liu. 2019. Learning to ask unanswerable questions for machine reading comprehension. arXiv preprint arXiv:1906.06045. [find](https://scholar.google.com/scholar?q=Haichao+Zhu%2C+Li+Dong%2C+Furu+Wei%2C+Wenhui+Wang%2C+Bing+Qin%2C+and+Ting+Liu%2E+2019%2E+Learning+to+ask+unanswerable+questions+for+machine+reading+comprehension%2E+arXiv+preprint+arXiv%3A1906%2E06045%2E)
42. Xiaojin Zhu and Andrew B Goldberg. 2009. Introduction to semi-supervised learning. Synthesis lectures on artificial intelligence and machine learning, 3(1):1–130. [find](https://scholar.google.com/scholar?q=Xiaojin+Zhu+and+Andrew+B+Goldberg%2E+2009%2E+Introduction+to+semi%2Dsupervised+learning%2E+Synthesis+lectures+on+artificial+intelligence+and+machine+learning%2C+3%281%29%3A1%E2%80%93130%2E)
43. Xiaojin Jerry Zhu. 2005. Semi-supervised learning literature survey. Technical report, University of Wisconsin-Madison Department of Computer Sciences. [find](https://scholar.google.com/scholar?q=Xiaojin+Jerry+Zhu%2E+2005%2E+Semi%2Dsupervised+learning+literature+survey%2E+Technical+report%2C+University+of+Wisconsin%2DMadison+Department+of+Computer+Sciences%2E)