--- title: "Synthetic QA Corpora Generation with — Alberti 2019" description: "This paper asks whether QA systems can benefit from a pretraining task closer to question answering than masked language modeling." authors: ["Chris Alberti", "Daniel Andor", "Emily Pitler", "Jacob Devlin", "Michael Collins"] year: 2019 venue: "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics" source: "https://aclanthology.org/P19-1620/" doi: "10.18653/v1/p19-1620" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/alberti-2019-synthetic-qa-corpora-generation-roundtrip/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/alberti-2019-synthetic-qa-corpora-generation-roundtrip/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / Synthetic QA Corpora Generation with Roundtrip Consistency # Synthetic QA Corpora Generation with Roundtrip Consistency **Authors:** [Chris Alberti](https://md.research.elamur.ai/free/en/authors/chris-alberti/), [Daniel Andor](https://md.research.elamur.ai/free/en/authors/daniel-andor/), [Emily Pitler](https://md.research.elamur.ai/free/en/authors/emily-pitler/), [Jacob Devlin](https://md.research.elamur.ai/free/en/authors/jacob-devlin/), [Michael Collins](https://md.research.elamur.ai/free/en/authors/michael-collins/) · **Year:** 2019 · **Venue:** Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/P19-1620/) · [PDF](https://aclanthology.org/P19-1620.pdf) · [DOI 10.18653/v1/p19-1620](https://doi.org/10.18653/v1/p19-1620) · [OpenAlex](https://openalex.org/W2949849869) **Citations:** 209 citations (openalex-snapshot · 2026-10-14) · [146](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary This paper asks whether QA systems can benefit from a pretraining task closer to question answering than masked language modeling. The authors build millions of synthetic question–answer–context triples from unlabeled Wikipedia text using three BERT-based models trained on labeled data. The first picks a candidate answer span from a passage. The second writes a question about that span. The third answers the generated question. A triple is kept only when the predicted answer agrees with the original span, a filter they call roundtrip consistency, inspired by back-translation in MT. A QA model is first pretrained on this synthetic corpus and then fine-tuned on SQuAD2 or NQ. There are two variants. One only fine-tunes public BERT, so it is fully reproducible. The other uses a fully pretrained sequence-to-sequence Transformer for question generation. Both improve results. NQ reaches a new state of the art, and the strongest SQuAD2 ensemble nearly matches human scores. The work shows that cheap, filtered synthetic data can serve as an effective QA-specific pretraining signal. ## Key points - **Data sources:** 8M unlabeled 512-token windows from the NQ training text (about 1B words). After roundtrip filtering, these yielded 2.4M SQuAD2-style and 3.2M NQ-style answerable examples. Added unanswerable examples brought the totals to 3M and 4M. - **Fine-tuning-only setting on SQuAD2:** BERT-Large rose from 78.7/81.9 to 81.2/84.0 dev EM/F1 when pretrained on combined SQuAD2+NQ synthetic data. - **Full pretraining setting:** a 50M-example synthetic corpus plus a 6-model ensemble scored 86.7 EM / 89.1 F1 on the SQuAD2 test set. Human performance is 86.8 / 89.5, and the system ranked third on the leaderboard (May 2019). - **NQ results:** over the BERT joint baseline, short-answer test F1 rose from 52.1 to 53.9 and long-answer test F1 from 66.2 to 66.8. - **Value of the filter:** in a manual check, 39% of kept triples were correct versus 16% of discarded ones. Filtered data consistently beat unfiltered data in the learning curves. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 146 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 143 not yet. - [Promptagator: Few-shot Dense Retrieval From 8 Examples](https://md.research.elamur.ai/free/en/papers/dai-2022-promptagator-few-shot-dense-retrieval/) - [MLQA: Evaluating Cross-lingual Extractive Question Answering](https://md.research.elamur.ai/free/en/papers/lewis-2019-mlqa-evaluating-cross-lingual-extractive/) - [PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them](https://md.research.elamur.ai/free/en/papers/lewis-2021-paq-65-million-probably-asked/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{alberti2019synthetic, title = {{Synthetic QA Corpora Generation with Roundtrip Consistency}}, author = {Alberti, Chris and Andor, Daniel and Pitler, Emily and Devlin, Jacob and Collins, Michael}, booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics}, year = {2019}, doi = {10.18653/v1/p19-1620}, url = {https://doi.org/10.18653/v1/p19-1620} } ``` - [← previous SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/) - [next → TabFact: A Large-scale Dataset for Table-based Fact Verification](https://md.research.elamur.ai/free/en/papers/chen-2019-tabfact-large-scale-dataset-table/) - [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) ## References 12 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Chris Alberti, Kenton Lee, and Michael Collins. 2019. A bert baseline for the natural questions. arXiv Xinya Du and Claire Cardie. 2018. Harvesting paragraph-level question-answer pairs from wikipedia. arXiv preprint arXiv:1805.05942. [find](https://scholar.google.com/scholar?q=Chris+Alberti%2C+Kenton+Lee%2C+and+Michael+Collins%2E+2019%2E+A+bert+baseline+for+the+natural+questions%2E+arXiv+Xinya+Du+and+Claire+Cardie%2E+2018%2E+Harvesting+paragraph%2Dlevel+question%2Danswer+pairs+from+wikipedia) 2. Xinya Du, Junru Shao, and Claire Cardie. 2017. Learning to ask: Neural question generation for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1342– 1352. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xinya+Du%2C+Junru+Shao%2C+and+Claire+Cardie%2E+2017%2E+Learning+to+ask%3A+Neural+question+generation+for+reading+comprehension%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computational+Lin) [our card](https://md.research.elamur.ai/free/en/papers/du-2017-learning-ask-neural-question-generation/) 3. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489–500. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sergey+Edunov%2C+Myle+Ott%2C+Michael+Auli%2C+and+David+Grangier%2E+2018%2E+Understanding+back%2Dtranslation+at+scale%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pag) 4. Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. 2016. Dual learning for machine translation. In Advances in Neural Information Processing Systems, pages 820–828. [find](https://scholar.google.com/scholar?q=Di+He%2C+Yingce+Xia%2C+Tao+Qin%2C+Liwei+Wang%2C+Nenghai+Yu%2C+Tie%2DYan+Liu%2C+and+Wei%2DYing+Ma%2E+2016%2E+Dual+learning+for+machine+translation%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+820%E2%80%93828%2E) 5. Michael Heilman and Noah A Smith. 2010. Good question! statistical ranking for question generation. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 609–617. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Michael+Heilman+and+Noah+A+Smith%2E+2010%2E+Good+question%21+statistical+ranking+for+question+generation%2E+In+Human+Language+Technologies%3A+The+2010+Annual+Conference+of+the+North+American+Chapter+of+the+Asso) 6. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Rhinehart, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural… [find](https://scholar.google.com/scholar?q=Tom+Kwiatkowski%2C+Jennimaria+Palomaki%2C+Olivia+Rhinehart%2C+Michael+Collins%2C+Ankur+Parikh%2C+Chris+Alberti%2C+Danielle+Epstein%2C+Illia+Polosukhin%2C+Matthew+Kelcey%2C+Jacob+Devlin%2C+Kenton+Lee%2C+Kristina+N%2E+Toutanov) [our card](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/) 7. Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, et al. 2018. Phrase-based & neural unsupervised machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5039–5049. [find](https://scholar.google.com/scholar?q=Guillaume+Lample%2C+Myle+Ott%2C+Alexis+Conneau%2C+Ludovic+Denoyer%2C+et+al%2E+2018%2E+Phrase%2Dbased+%26+neural+unsupervised+machine+translation%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+) 8. Mike Lewis and Angela Fan. 2019. Generative question answering: Learning to answer the whole question. International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Mike+Lewis+and+Angela+Fan%2E+2019%2E+Generative+question+answering%3A+Learning+to+answer+the+whole+question%2E+International+Conference+on+Learning+Representations+%28ICLR%29%2E) 9. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you dont know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 784–789. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Robin+Jia%2C+and+Percy+Liang%2E+2018%2E+Know+what+you+dont+know%3A+Unanswerable+questions+for+squad%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistic) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/) 10. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 86–96. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016%2E+Improving+neural+machine+translation+models+with+monolingual+data%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computationa) 11. Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. Ernie: Enhanced representation through knowledge integration. CoRR, abs/1904.09223. [find](https://scholar.google.com/scholar?q=Yu+Sun%2C+Shuohuan+Wang%2C+Yukun+Li%2C+Shikun+Feng%2C+Xuyi+Chen%2C+Han+Zhang%2C+Xin+Tian%2C+Danxiang+Zhu%2C+Hao+Tian%2C+and+Hua+Wu%2E+2019%2E+Ernie%3A+Enhanced+representation+through+knowledge+integration%2E+CoRR%2C+abs%2F1904%2E092) [our card](https://md.research.elamur.ai/free/en/papers/sun-2019-ernie-enhanced-representation-through-knowledge/) 12. Zhilin Yang, Junjie Hu, Ruslan Salakhutdinov, and William Cohen. 2017. Semi-supervised qa with generative domain-adaptive nets. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1040–1050. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Junjie+Hu%2C+Ruslan+Salakhutdinov%2C+and+William+Cohen%2E+2017%2E+Semi%2Dsupervised+qa+with+generative+domain%2Dadaptive+nets%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computa)