--- title: "Multi-passage BERT — Wang 2019" description: "Open-domain question answering requires finding answers in large collections such as Wikipedia, not in a single given paragraph." authors: ["Zhiguo Wang", "Patrick Ng", "Xiaofei Ma", "Ramesh Nallapati", "Bing Xiang"] year: 2019 venue: "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)" source: "https://aclanthology.org/D19-1599/" doi: "10.18653/v1/d19-1599" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/wang-2019-multi-passage-bert-globally-normalized/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2019-multi-passage-bert-globally-normalized/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/) / Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering # Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering **Authors:** [Zhiguo Wang](https://md.research.elamur.ai/free/en/authors/zhiguo-wang/), Patrick Ng, Xiaofei Ma, [Ramesh Nallapati](https://md.research.elamur.ai/free/en/authors/ramesh-nallapati/), [Bing Xiang](https://md.research.elamur.ai/free/en/authors/bing-xiang/) · **Year:** 2019 · **Venue:** Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/D19-1599/) · [PDF](https://aclanthology.org/D19-1599.pdf) · [DOI 10.18653/v1/d19-1599](https://doi.org/10.18653/v1/d19-1599) · [OpenAlex](https://openalex.org/W2971105107) **Citations:** 224 citations (openalex-snapshot · 2026-10-14) · [80](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary Open-domain question answering requires finding answers in large collections such as Wikipedia, not in a single given paragraph. Earlier BERT-based systems scored each retrieved passage separately, so answer scores from different passages could not be reliably compared. The authors (AWS AI Labs) propose multi-passage BERT. It encodes each question–passage pair as BERT-RC does, but applies one softmax over the start and end positions of all passages retrieved for a question. They also study several design choices on OpenSQuAD: - passage granularity, - overlapping sliding-window splitting, - a BERT-based passage ranker whose scores are combined with span probabilities, - whether explicit question–passage matching (BERT+QANet) still helps. Global normalization lets accuracy keep rising as more passages are added, whereas per-passage normalization degrades. With all components, the model beats prior systems on Quasar-T, SearchQA, TriviaQA and OpenSQuAD. The work shows that cross-passage score calibration, rather than additional matching layers, is key when adapting reading-comprehension models to retrieval-based QA. ## Key points - **Passage length:** 100-word passages work better than single sentences or 50- and 200-word chunks. A sliding window (size 100, stride 50) adds 4.7 EM on the OpenSQuAD validation set (35.7 → 40.4). - **Passage ranker:** The ranker reranks the top-100 BM25/ElasticSearch passages and keeps the top 30. With its scores, validation results rise to 42.8 EM / 53.4 F1 (from 40.4 / 49.8). - **Explicit matching:** Adding QANet-style inter-sentence matching on top of BERT hurts performance. The best variant reaches 36.2 EM. The authors suggest BERT self-attention already performs this matching. - **OpenSQuAD test results:** Multi-passage BERT (Base) reaches 51.2 EM / 59.0 F1, and the Large version reaches 53.0 / 60.9. This is +21.4 EM over the best non-BERT system and +5.8 EM over the authors' BERT-RC baseline. - **Other benchmarks (Large):** TriviaQA 63.7 / 69.2, SearchQA 65.1 / 70.7, Quasar-T 51.1 / 59.1 (EM / F1). *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 80 papers of our library cite it (of the 140884 whose bibliographies we read); 9 of them have a page here, 71 not yet. - [Leveraging BERT Language Model for Arabic Long Document Classification](https://md.research.elamur.ai/free/en/papers/al-2023-qurishi-leveraging-bert-language-model-arabic/) - [Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering](https://md.research.elamur.ai/free/en/papers/asai-2019-learning-retrieve-reasoning-paths-over/) - [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://md.research.elamur.ai/free/en/papers/izacard-2021-leveraging-passage-retrieval-generative-models/) - [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://md.research.elamur.ai/free/en/papers/jin-2020-disease-this-patient-have-large/) - [Dense Passage Retrieval for Open-Domain Question Answering](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/) - [RealTime QA: What's the Answer Right Now?](https://md.research.elamur.ai/free/en/papers/kasai-2022-realtime-qa-s-answer-right/) - [RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering](https://md.research.elamur.ai/free/en/papers/qu-2021-rocketqa-optimized-training-approach-dense/) - [RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking](https://md.research.elamur.ai/free/en/papers/ren-2021-rocketqav2-joint-training-method-dense/) - [Big Bird: Transformers for Longer Sequences](https://md.research.elamur.ai/free/en/papers/zaheer-2020-big-bird-transformers-longer-sequences/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{wang2019multi, title = {{Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering}}, author = {Wang, Zhiguo and Ng, Patrick and Ma, Xiaofei and Nallapati, Ramesh and Xiang, Bing}, booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)}, year = {2019}, doi = {10.18653/v1/d19-1599}, url = {https://doi.org/10.18653/v1/d19-1599} } ``` - [← previous Multi-News: A Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model](https://md.research.elamur.ai/free/en/papers/fabbri-2019-multi-news-large-scale-multi/) - [next → Multi-Relational Script Learning for Discourse Relations](https://md.research.elamur.ai/free/en/papers/lee-2019-multi-relational-script-learning-discourse/) - [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/) ## References 21 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Linfeng Song, Zhiguo Wang, Mo Yu, Yue Zhang, Radu Florian, and Daniel Gildea. 2018. Exploring graph-structured passage representation for multi-hop reading comprehension with graph neural networks. arXiv preprint arXiv:1809.02040. — [arXiv:1809.02040](https://arxiv.org/abs/1809.02040) [find](https://scholar.google.com/scholar?q=Linfeng+Song%2C+Zhiguo+Wang%2C+Mo+Yu%2C+Yue+Zhang%2C+Radu+Florian%2C+and+Daniel+Gildea%2E+2018%2E+Exploring+graph%2Dstructured+passage+representation+for+multi%2Dhop+reading+comprehension+with+graph+neural+networks%2E+ar) 2. Chris Alberti, Kenton Lee, and Michael Collins. 2019. A bert baseline for the natural questions. arXiv preprint arXiv:1901.08634. — [arXiv:1901.08634](https://arxiv.org/abs/1901.08634) [find](https://scholar.google.com/scholar?q=Chris+Alberti%2C+Kenton+Lee%2C+and+Michael+Collins%2E+2019%2E+A+bert+baseline+for+the+natural+questions%2E+arXiv+preprint+arXiv%3A1901%2E08634%2E+%E2%80%94+%5BarXiv%3A1901%2E08634%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1901%2E08634%29) 3. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. In ACL 2017. [find](https://scholar.google.com/scholar?q=Danqi+Chen%2C+Adam+Fisch%2C+Jason+Weston%2C+and+Antoine+Bordes%2E+2017%2E+Reading+wikipedia+to+answer+open%2Ddomain+questions%2E+In+ACL+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2017-reading-wikipedia-answer-open-domain/) 4. Christopher Clark and Matt Gardner. 2018. Simple and effective multi-paragraph reading comprehension. In ACL 2018. [find](https://scholar.google.com/scholar?q=Christopher+Clark+and+Matt+Gardner%2E+2018%2E+Simple+and+effective+multi%2Dparagraph+reading+comprehension%2E+In+ACL+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/clark-2017-simple-effective-multi-paragraph-reading/) 5. Mostafa Dehghani, Hosein Azarbonyad, Jaap Kamps, and Maarten de Rijke. 2019. Learning to transform, combine, and reason in open-domain question answering. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, pages 681–689. ACM. [find](https://scholar.google.com/scholar?q=Mostafa+Dehghani%2C+Hosein+Azarbonyad%2C+Jaap+Kamps%2C+and+Maarten+de+Rijke%2E+2019%2E+Learning+to+transform%2C+combine%2C+and+reason+in+open%2Ddomain+question+answering%2E+In+Proceedings+of+the+Twelfth+ACM+Internation) 6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2018%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+arXiv+preprint+arXiv%3A1810%2E04805%2E) 7. Bhuwan Dhingra, Kathryn Mazaitis, and William W Cohen. 2017. Quasar: Datasets for question answering by search and reading. arXiv preprint arXiv:1707.03904. — [arXiv:1707.03904](https://arxiv.org/abs/1707.03904) [find](https://scholar.google.com/scholar?q=Bhuwan+Dhingra%2C+Kathryn+Mazaitis%2C+and+William+W+Cohen%2E+2017%2E+Quasar%3A+Datasets+for+question+answering+by+search+and+reading%2E+arXiv+preprint+arXiv%3A1707%2E03904%2E+%E2%80%94+%5BarXiv%3A1707%2E03904%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F) 8. Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017. Searchqa: A new q&a dataset augmented with context from a search engine. arXiv preprint arXiv:1704.05179. [find](https://scholar.google.com/scholar?q=Matthew+Dunn%2C+Levent+Sagun%2C+Mike+Higgins%2C+V+Ugur+Guney%2C+Volkan+Cirik%2C+and+Kyunghyun+Cho%2E+2017%2E+Searchqa%3A+A+new+q%26a+dataset+augmented+with+context+from+a+search+engine%2E+arXiv+preprint+arXiv%3A1704%2E05179%2E) [our card](https://md.research.elamur.ai/free/en/papers/dunn-2017-searchqa-new-q-dataset-augmented/) 9. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In ACL 2017. [find](https://scholar.google.com/scholar?q=Mandar+Joshi%2C+Eunsol+Choi%2C+Daniel+S+Weld%2C+and+Luke+Zettlemoyer%2E+2017%2E+Triviaqa%3A+A+large+scale+distantly+supervised+challenge+dataset+for+reading+comprehension%2E+In+ACL+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/) 10. Yankai Lin, Haozhe Ji, Zhiyuan Liu, and Maosong Sun. 2018. Denoising distantly supervised open-domain question answering. In ACL 2018. [find](https://scholar.google.com/scholar?q=Yankai+Lin%2C+Haozhe+Ji%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2018%2E+Denoising+distantly+supervised+open%2Ddomain+question+answering%2E+In+ACL+2018%2E) 11. Shuohang Wang and Jing Jiang. 2017. Machine comprehension using match-lstm and answer pointer. In ICLR 2017. [find](https://scholar.google.com/scholar?q=Shuohang+Wang+and+Jing+Jiang%2E+2017%2E+Machine+comprehension+using+match%2Dlstm+and+answer+pointer%2E+In+ICLR+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2016-machine-comprehension-using-match-lstm/) 12. Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerry Tesauro, Bowen Zhou, and Jing Jiang. 2018a. R3: Reinforced ranker-reader for open-domain question answering. In AAAI 2018. [find](https://scholar.google.com/scholar?q=Shuohang+Wang%2C+Mo+Yu%2C+Xiaoxiao+Guo%2C+Zhiguo+Wang%2C+Tim+Klinger%2C+Wei+Zhang%2C+Shiyu+Chang%2C+Gerry+Tesauro%2C+Bowen+Zhou%2C+and+Jing+Jiang%2E+2018a%2E+R3%3A+Reinforced+ranker%2Dreader+for+open%2Ddomain+question+answering%2E) 13. Shuohang Wang, Mo Yu, Jing Jiang, Wei Zhang, Xiaoxiao Guo, Shiyu Chang, Zhiguo Wang, Tim Klinger, Gerald Tesauro, and Murray Campbell. 2018b. Evidence aggregation for answer re-ranking in open-domain question answering. In ICLR 2018. [find](https://scholar.google.com/scholar?q=Shuohang+Wang%2C+Mo+Yu%2C+Jing+Jiang%2C+Wei+Zhang%2C+Xiaoxiao+Guo%2C+Shiyu+Chang%2C+Zhiguo+Wang%2C+Tim+Klinger%2C+Gerald+Tesauro%2C+and+Murray+Campbell%2E+2018b%2E+Evidence+aggregation+for+answer+re%2Dranking+in+open%2Ddomain+) 14. Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. 2017. Gated self-matching networks for reading comprehension and question answering. In ACL 2017. [find](https://scholar.google.com/scholar?q=Wenhui+Wang%2C+Nan+Yang%2C+Furu+Wei%2C+Baobao+Chang%2C+and+Ming+Zhou%2E+2017%2E+Gated+self%2Dmatching+networks+for+reading+comprehension+and+question+answering%2E+In+ACL+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2017-gated-self-matching-networks-reading/) 15. Zhiguo Wang, Haitao Mi, Wael Hamza, and Radu Florian. 2016. Multi-perspective context matching for machine comprehension. arXiv preprint arXiv:1612.04211. — [arXiv:1612.04211](https://arxiv.org/abs/1612.04211) [find](https://scholar.google.com/scholar?q=Zhiguo+Wang%2C+Haitao+Mi%2C+Wael+Hamza%2C+and+Radu+Florian%2E+2016%2E+Multi%2Dperspective+context+matching+for+machine+comprehension%2E+arXiv+preprint+arXiv%3A1612%2E04211%2E+%E2%80%94+%5BarXiv%3A1612%2E04211%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F16) 16. Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019. End-to-end open-domain question answering with bertserini. arXiv preprint arXiv:1902.01718. [find](https://scholar.google.com/scholar?q=Wei+Yang%2C+Yuqing+Xie%2C+Aileen+Lin%2C+Xingyu+Li%2C+Luchen+Tan%2C+Kun+Xiong%2C+Ming+Li%2C+and+Jimmy+Lin%2E+2019%2E+End%2Dto%2Dend+open%2Ddomain+question+answering+with+bertserini%2E+arXiv+preprint+arXiv%3A1902%2E01718%2E) [our card](https://md.research.elamur.ai/free/en/papers/yang-2019-end-end-open-domain-question/) 17. Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V Le. 2018. Qanet: Combining local convolution with global self-attention for reading comprehension. In ICLR 2018. [find](https://scholar.google.com/scholar?q=Adams+Wei+Yu%2C+David+Dohan%2C+Minh%2DThang+Luong%2C+Rui+Zhao%2C+Kai+Chen%2C+Mohammad+Norouzi%2C+and+Quoc+V+Le%2E+2018%2E+Qanet%3A+Combining+local+convolution+with+global+self%2Dattention+for+reading+comprehension%2E+In+ICLR) [our card](https://md.research.elamur.ai/free/en/papers/yu-2018-qanet-combining-local-convolution-global/) 18. Rodrigo Nogueira, Jannis Bulian, and Massimiliano Ciaramita. 2018. Learning to coordinate multiple reinforcement learning agents for diverse query reformulation. arXiv preprint arXiv:1809.10658. — [arXiv:1809.10658](https://arxiv.org/abs/1809.10658) [find](https://scholar.google.com/scholar?q=Rodrigo+Nogueira%2C+Jannis+Bulian%2C+and+Massimiliano+Ciaramita%2E+2018%2E+Learning+to+coordinate+multiple+reinforcement+learning+agents+for+diverse+query+reformulation%2E+arXiv+preprint+arXiv%3A1809%2E10658%2E+%E2%80%94+%5Bar) 19. Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Lixin Su, and Xueqi Cheng. 2019. Has-qa: Hierarchical answer spans model for open-domain question answering. In AAAI 2019. [find](https://scholar.google.com/scholar?q=Liang+Pang%2C+Yanyan+Lan%2C+Jiafeng+Guo%2C+Jun+Xu%2C+Lixin+Su%2C+and+Xueqi+Cheng%2E+2019%2E+Has%2Dqa%3A+Hierarchical+answer+spans+model+for+open%2Ddomain+question+answering%2E+In+AAAI+2019%2E) 20. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In EMNLP 2016. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+2016%2E+Squad%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+EMNLP+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/) 21. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In ICLR 2017. [find](https://scholar.google.com/scholar?q=Minjoon+Seo%2C+Aniruddha+Kembhavi%2C+Ali+Farhadi%2C+and+Hannaneh+Hajishirzi%2E+2017%2E+Bidirectional+attention+flow+for+machine+comprehension%2E+In+ICLR+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/seo-2016-bidirectional-attention-flow-machine-comprehension/)