---
title: "A Discrete Hard EM Approach for Weakly Supervised — Min 2019"
description: "Many QA datasets supply only the final answer string, not how it was derived."
authors: ["Sewon Min", "Danqi Chen", "Hannaneh Hajishirzi", "Luke Zettlemoyer"]
year: 2019
venue: "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)"
source: "https://aclanthology.org/D19-1284/"
doi: "10.18653/v1/d19-1284"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/min-2019-discrete-hard-em-approach-weakly/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/min-2019-discrete-hard-em-approach-weakly/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/) / A Discrete Hard EM Approach for Weakly Supervised Question Answering

# A Discrete Hard EM Approach for Weakly Supervised Question Answering

**Authors:** [Sewon Min](https://md.research.elamur.ai/free/en/authors/sewon-min/), [Danqi Chen](https://md.research.elamur.ai/free/en/authors/danqi-chen/), [Hannaneh Hajishirzi](https://md.research.elamur.ai/free/en/authors/hannaneh-hajishirzi/), [Luke Zettlemoyer](https://md.research.elamur.ai/free/en/authors/luke-zettlemoyer/) · **Year:** 2019 · **Venue:** Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D19-1284/) · [PDF](https://aclanthology.org/D19-1284.pdf) · [DOI 10.18653/v1/d19-1284](https://doi.org/10.18653/v1/d19-1284) · [OpenAlex](https://openalex.org/W2970168256)

**Citations:** 141 citations (openalex-snapshot · 2026-10-14) · [92](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary

Many QA datasets supply only the final answer string, not how it was derived. The same TriviaQA answer may appear in several document spans, and a DROP number can come from many different arithmetic expressions, most of which are spurious. Min et al. cast these weakly supervised tasks as discrete latent-variable learning. For each training example, they precompute a task-specific set Z of candidate solutions (spans or equations) that yield the gold answer, assuming exactly one of them is correct. Training uses a hard-EM objective: the current model picks its most probable candidate from Z and is then updated as if that candidate were the gold label. This replaces the sum in maximum marginal likelihood (MML) with a max. The method works with any model architecture. The authors tested it on six datasets covering multi-mention reading comprehension, open-domain QA, discrete reasoning and semantic parsing, using strong task-specific models. It beat heuristic supervision and MML by 2–10% absolute and reached state-of-the-art results on five datasets. The work shows that the training objective, and not only the architecture, matters greatly under weak supervision.

## Key points

- **Datasets:** TriviaQA, NarrativeQA, TriviaQA-open, Natural Questions-open, DROP-num and WikiSQL.
  - The average size of the candidate set Z ranges from 1.8 (Natural Questions-open) to 346.1 (WikiSQL).
- **WikiSQL comparison:** Hard EM outperforms the recent reward-driven semantic parsers proposed by Liang and colleagues (2018) and by Agarwal and colleagues (2019) by roughly 13% absolute. This suggests that a small precomputed solution set is a crucial ingredient.
- **Correctness of solutions, not just answers:**
  - On 50 manually checked development examples, 98% (NarrativeQA) and 92% (DROP-num) of correct answers came from the correct solution.
  - On WikiSQL, 7,110 of 8,421 development predictions returned the right answer, and 88.5% of those exactly matched the gold SQL query.
- **Robustness to noise:** When Z for NarrativeQA was made noisier (average size raised from 4.3 to 7.1):
  - MML dropped from 56.07 to 51.14.
  - Hard EM dropped only from 58.77 to 57.97.
- **Advantage grows with Z:** The gain over MML increases as the candidate set gets larger. The code is released at github.com/shmsw25/qa-hard-em.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

92 papers of our library cite it (of the 140884 whose bibliographies we read); 11 of them have a page here, 81 not yet.

- [Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering](https://md.research.elamur.ai/free/en/papers/asai-2019-learning-retrieve-reasoning-paths-over/)
- [Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks](https://md.research.elamur.ai/free/en/papers/asai-2022-evidentiality-guided-generation-knowledge-intensive/)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://md.research.elamur.ai/free/en/papers/asai-2023-self-rag-learning-retrieve-generate/)
- [Entities as Experts: Sparse Memory Access with Entity Supervision](https://md.research.elamur.ai/free/en/papers/fevry-2020-entities-as-experts-sparse-memory/)
- [TaPas: Weakly Supervised Table Parsing via Pre-training](https://md.research.elamur.ai/free/en/papers/herzig-2020-tapas-weakly-supervised-table-parsing/)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://md.research.elamur.ai/free/en/papers/izacard-2021-leveraging-passage-retrieval-generative-models/)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/)
- [UNIFIEDQA: Crossing Format Boundaries with a Single QA System](https://md.research.elamur.ai/free/en/papers/khashabi-2020-unifiedqa-crossing-format-boundaries-single/)
- [TAPEX: Table Pre-training via Learning a Neural SQL Executor](https://md.research.elamur.ai/free/en/papers/liu-2021-tapex-table-pre-training-learning/)
- [Generation-Augmented Retrieval for Open-domain Question Answering](https://md.research.elamur.ai/free/en/papers/mao-2020-generation-augmented-retrieval-open-domain/)
- [RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering](https://md.research.elamur.ai/free/en/papers/qu-2021-rocketqa-optimized-training-approach-dense/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{min2019discrete,
  title = {{A Discrete Hard EM Approach for Weakly Supervised Question Answering}},
  author = {Min, Sewon and Chen, Danqi and Hajishirzi, Hannaneh and Zettlemoyer, Luke},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1284},
  url = {https://doi.org/10.18653/v1/d19-1284}
}
```

- [← previous A Discourse Signal Annotation System for RST Trees](https://md.research.elamur.ai/free/en/papers/gessler-2019-discourse-signal-annotation-system-rst/)
- [next → A Dual Reinforcement Learning Framework for Unsupervised Text Style Transfer](https://md.research.elamur.ai/free/en/papers/luo-2019-dual-reinforcement-learning-framework-unsupervised/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/)

## References

44 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Daniel A Abolafia, Mohammad Norouzi, Jonathan Shen, Rui Zhao, and Quoc V Le. 2018. Neural program synthesis with priority queue training. arXiv preprint arXiv:1801.03526. — [arXiv:1801.03526](https://arxiv.org/abs/1801.03526) [find](https://scholar.google.com/scholar?q=Daniel+A+Abolafia%2C+Mohammad+Norouzi%2C+Jonathan+Shen%2C+Rui+Zhao%2C+and+Quoc+V+Le%2E+2018%2E+Neural+program+synthesis+with+priority+queue+training%2E+arXiv+preprint+arXiv%3A1801%2E03526%2E+%E2%80%94+%5BarXiv%3A1801%2E03526%5D%28https%3A%2F%2F)
2. Rishabh Agarwal, Chen Liang, Dale Schuurmans, and Mohammad Norouzi. 2019. Learning to generalize from sparse and underspecified rewards. In ICML. [find](https://scholar.google.com/scholar?q=Rishabh+Agarwal%2C+Chen+Liang%2C+Dale+Schuurmans%2C+and+Mohammad+Norouzi%2E+2019%2E+Learning+to+generalize+from+sparse+and+underspecified+rewards%2E+In+ICML%2E)
3. Chris Alberti, Kenton Lee, and Michael Collins. 2019. A BERT baseline for the Natural Questions. arXiv preprint arXiv:1901.08634. — [arXiv:1901.08634](https://arxiv.org/abs/1901.08634) [find](https://scholar.google.com/scholar?q=Chris+Alberti%2C+Kenton+Lee%2C+and+Michael+Collins%2E+2019%2E+A+BERT+baseline+for+the+Natural+Questions%2E+arXiv+preprint+arXiv%3A1901%2E08634%2E+%E2%80%94+%5BarXiv%3A1901%2E08634%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1901%2E08634%29)
4. Yoav Artzi and Luke Zettlemoyer. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. In ACL. [find](https://scholar.google.com/scholar?q=Yoav+Artzi+and+Luke+Zettlemoyer%2E+2013%2E+Weakly+supervised+learning+of+semantic+parsers+for+mapping+instructions+to+actions%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/artzi-2013-weakly-supervised-learning-semantic-parsers/)
5. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In ACL. [find](https://scholar.google.com/scholar?q=Mandar+Joshi%2C+Eunsol+Choi%2C+Daniel+S+Weld%2C+and+Luke+Zettlemoyer%2E+2017%2E+TriviaQA%3A+A+large+scale+distantly+supervised+challenge+dataset+for+reading+comprehension%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/)
6. Rudolf Kadlec, Martin Schmid, Ondrej Bajgar, and Jan Kleindienst. 2016. Text understanding with the attention sum reader network. In ACL. [find](https://scholar.google.com/scholar?q=Rudolf+Kadlec%2C+Martin+Schmid%2C+Ondrej+Bajgar%2C+and+Jan+Kleindienst%2E+2016%2E+Text+understanding+with+the+attention+sum+reader+network%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/kadlec-2016-text-understanding-attention-sum-reader/)
7. Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The NarrativeQA reading comprehension challenge. TACL. [find](https://scholar.google.com/scholar?q=Toma%CC%81s%CC%8C+Koc%CC%8Cisky%CC%80%2C+Jonathan+Schwarz%2C+Phil+Blunsom%2C+Chris+Dyer%2C+Karl+Moritz+Hermann%2C+Ga%CC%81bor+Melis%2C+and+Edward+Grefenstette%2E+2018%2E+The+NarrativeQA+reading+comprehension+challenge%2E+TACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/kocisky-2018-narrativeqa-reading-comprehension-challenge/)
8. Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gardner. 2017. Neural semantic parsing with type constraints for semi-structured tables. In EMNLP. [find](https://scholar.google.com/scholar?q=Jayant+Krishnamurthy%2C+Pradeep+Dasigi%2C+and+Matt+Gardner%2E+2017%2E+Neural+semantic+parsing+with+type+constraints+for+semi%2Dstructured+tables%2E+In+EMNLP%2E)
9. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In ACL. [find](https://scholar.google.com/scholar?q=Danqi+Chen%2C+Adam+Fisch%2C+Jason+Weston%2C+and+Antoine+Bordes%2E+2017%2E+Reading+Wikipedia+to+answer+open%2Ddomain+questions%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2017-reading-wikipedia-answer-open-domain/)
10. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red-field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Change, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natura… [find](https://scholar.google.com/scholar?q=Tom+Kwiatkowski%2C+Jennimaria+Palomaki%2C+Olivia+Red%2Dfield%2C+Michael+Collins%2C+Ankur+Parikh%2C+Chris+Alberti%2C+Danielle+Epstein%2C+Illia+Polosukhin%2C+Jacob+Devlin%2C+Kenton+Lee%2C+Kristina+Toutanova%2C+Llion+Jones%2C+Mat) [our card](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
11. Christopher Clark and Matt Gardner. 2018. Simple and effective multi-paragraph reading comprehension. In ACL. [find](https://scholar.google.com/scholar?q=Christopher+Clark+and+Matt+Gardner%2E+2018%2E+Simple+and+effective+multi%2Dparagraph+reading+comprehension%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/clark-2017-simple-effective-multi-paragraph-reading/)
12. Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In ACL. [find](https://scholar.google.com/scholar?q=Kenton+Lee%2C+Ming%2DWei+Chang%2C+and+Kristina+Toutanova%2E+2019%2E+Latent+retrieval+for+weakly+supervised+open+domain+question+answering%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/lee-2019-latent-retrieval-weakly-supervised-open/)
13. James Clarke, Dan Goldwasser, Ming-Wei Chang, and Dan Roth. 2010. Driving semantic parsing from the world’s response. In CoNLL. [find](https://scholar.google.com/scholar?q=James+Clarke%2C+Dan+Goldwasser%2C+Ming%2DWei+Chang%2C+and+Dan+Roth%2E+2010%2E+Driving+semantic+parsing+from+the+world%E2%80%99s+response%2E+In+CoNLL%2E)
14. Chen Liang, Jonathan Berant, Quoc Le, Kenneth D Forbus, and Ni Lao. 2017. Neural symbolic machines: Learning semantic parsers on Freebase with weak supervision. In ACL. [find](https://scholar.google.com/scholar?q=Chen+Liang%2C+Jonathan+Berant%2C+Quoc+Le%2C+Kenneth+D+Forbus%2C+and+Ni+Lao%2E+2017%2E+Neural+symbolic+machines%3A+Learning+semantic+parsers+on+Freebase+with+weak+supervision%2E+In+ACL%2E)
15. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In EMNLP. [find](https://scholar.google.com/scholar?q=Jonathan+Berant%2C+Andrew+Chou%2C+Roy+Frostig%2C+and+Percy+Liang%2E+2013%2E+Semantic+parsing+on+Freebase+from+question%2Danswer+pairs%2E+In+EMNLP%2E)
16. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+NAACL%2E)
17. Li Dong and Mirella Lapata. 2018. Coarse-to-fine decoding for neural semantic parsing. In ACL. [find](https://scholar.google.com/scholar?q=Li+Dong+and+Mirella+Lapata%2E+2018%2E+Coarse%2Dto%2Dfine+decoding+for+neural+semantic+parsing%2E+In+ACL%2E)
18. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In NAACL. [find](https://scholar.google.com/scholar?q=Dheeru+Dua%2C+Yizhong+Wang%2C+Pradeep+Dasigi%2C+Gabriel+Stanovsky%2C+Sameer+Singh%2C+and+Matt+Gardner%2E+2019%2E+DROP%3A+A+reading+comprehension+benchmark+requiring+discrete+reasoning+over+paragraphs%2E+In+NAACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/)
19. Pengcheng He, Yi Mao, Kaushik Chakrabarti, and Weizhu Chen. 2019. X-SQL: reinforce schema representation with context. arXiv preprint arXiv:1908.08113. — [arXiv:1908.08113](https://arxiv.org/abs/1908.08113) [find](https://scholar.google.com/scholar?q=Pengcheng+He%2C+Yi+Mao%2C+Kaushik+Chakrabarti%2C+and+Weizhu+Chen%2E+2019%2E+X%2DSQL%3A+reinforce+schema+representation+with+context%2E+arXiv+preprint+arXiv%3A1908%2E08113%2E+%E2%80%94+%5BarXiv%3A1908%2E08113%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1908%2E)
20. Chen Liang, Mohammad Norouzi, Jonathan Berant, Quoc V Le, and Ni Lao. 2018. Memory augmented policy optimization for program synthesis and semantic parsing. In NIPS. [find](https://scholar.google.com/scholar?q=Chen+Liang%2C+Mohammad+Norouzi%2C+Jonathan+Berant%2C+Quoc+V+Le%2C+and+Ni+Lao%2E+2018%2E+Memory+augmented+policy+optimization+for+program+synthesis+and+semantic+parsing%2E+In+NIPS%2E)
21. Percy Liang, Michael I Jordan, and Dan Klein. 2013. Learning dependency-based compositional semantics. Computational Linguistics, 39(2):389–446. [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Michael+I+Jordan%2C+and+Dan+Klein%2E+2013%2E+Learning+dependency%2Dbased+compositional+semantics%2E+Computational+Linguistics%2C+39%282%29%3A389%E2%80%93446%2E)
22. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. Text Summarization Branches Out. [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin%2E+2004%2E+ROUGE%3A+A+package+for+automatic+evaluation+of+summaries%2E+Text+Summarization+Branches+Out%2E)
23. Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019. Compositional questions do not necessitate multi-hop reasoning. In ACL. [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Eric+Wallace%2C+Sameer+Singh%2C+Matt+Gardner%2C+Hannaneh+Hajishirzi%2C+and+Luke+Zettlemoyer%2E+2019%2E+Compositional+questions+do+not+necessitate+multi%2Dhop+reasoning%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/min-2019-compositional-questions-do-not-necessitate/)
24. Niall Hurley and Scott Rickard. 2009. Comparing measures of sparsity. IEEE Transactions on Information Theory. [find](https://scholar.google.com/scholar?q=Niall+Hurley+and+Scott+Rickard%2E+2009%2E+Comparing+measures+of+sparsity%2E+IEEE+Transactions+on+Information+Theory%2E)
25. Kyosuke Nishida, Itsumi Saito, Kosuke Nishida, Kazu-toshi Shinoda, Atsushi Otsuka, Hisako Asano, and Junji Tomita. 2019. Multi-style generative reading comprehension. In ACL. [find](https://scholar.google.com/scholar?q=Kyosuke+Nishida%2C+Itsumi+Saito%2C+Kosuke+Nishida%2C+Kazu%2Dtoshi+Shinoda%2C+Atsushi+Otsuka%2C+Hisako+Asano%2C+and+Junji+Tomita%2E+2019%2E+Multi%2Dstyle+generative+reading+comprehension%2E+In+ACL%2E)
26. Wonseok Hwang, Jinyeung Yim, Seunghyun Park, and Minjoon Seo. 2019. A comprehensive exploration on WikiSQL with table-aware word contextualization. arXiv preprint arXiv:1902.01069. [find](https://scholar.google.com/scholar?q=Wonseok+Hwang%2C+Jinyeung+Yim%2C+Seunghyun+Park%2C+and+Minjoon+Seo%2E+2019%2E+A+comprehensive+exploration+on+WikiSQL+with+table%2Daware+word+contextualization%2E+arXiv+preprint+arXiv%3A1902%2E01069%2E)
27. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. [find](https://scholar.google.com/scholar?q=Adam+Paszke%2C+Sam+Gross%2C+Soumith+Chintala%2C+Gregory+Chanan%2C+Edward+Yang%2C+Zachary+DeVito%2C+Zeming+Lin%2C+Alban+Desmaison%2C+Luca+Antiga%2C+and+Adam+Lerer%2E+2017%2E+Automatic+differentiation+in+PyTorch%2E)
28. Mohit Iyyer, Wen-tau Yih, and Ming-Wei Chang. 2017. Search-based neural structured learning for sequential question answering. In ACL. [find](https://scholar.google.com/scholar?q=Mohit+Iyyer%2C+Wen%2Dtau+Yih%2C+and+Ming%2DWei+Chang%2E+2017%2E+Search%2Dbased+neural+structured+learning+for+sequential+question+answering%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/iyyer-2017-search-based-neural-structured-learning/)
29. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+2016%2E+SQuAD%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
30. Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends R in Information Retrieval. [find](https://scholar.google.com/scholar?q=Stephen+Robertson%2C+Hugo+Zaragoza%2C+et+al%2E+2009%2E+The+probabilistic+relevance+framework%3A+BM25+and+beyond%2E+Foundations+and+Trends+R+in+Information+Retrieval%2E)
31. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In ICLR. [find](https://scholar.google.com/scholar?q=Minjoon+Seo%2C+Aniruddha+Kembhavi%2C+Ali+Farhadi%2C+and+Hannaneh+Hajishirzi%2E+2017%2E+Bidirectional+attention+flow+for+machine+comprehension%2E+In+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/seo-2016-bidirectional-attention-flow-machine-comprehension/)
32. Swabha Swayamdipta, Ankur P Parikh, and Tom Kwiatkowski. 2018. Multi-mention learning for reading comprehension with neural cascades. In ICLR. [find](https://scholar.google.com/scholar?q=Swabha+Swayamdipta%2C+Ankur+P+Parikh%2C+and+Tom+Kwiatkowski%2E+2018%2E+Multi%2Dmention+learning+for+reading+comprehension+with+neural+cascades%2E+In+ICLR%2E)
33. Alon Talmor and Jonathan Berant. 2019. MultiQA: An empirical investigation of generalization and transfer in reading comprehension. In ACL. [find](https://scholar.google.com/scholar?q=Alon+Talmor+and+Jonathan+Berant%2E+2019%2E+MultiQA%3A+An+empirical+investigation+of+generalization+and+transfer+in+reading+comprehension%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/talmor-2019-multiqa-empirical-investigation-generalization-transfer/)
34. Yi Tay, Anh Tuan Luu, Siu Cheung Hui, and Jian Su. 2018. Densely connected attention propagation for reading comprehension. In NIPS. [find](https://scholar.google.com/scholar?q=Yi+Tay%2C+Anh+Tuan+Luu%2C+Siu+Cheung+Hui%2C+and+Jian+Su%2E+2018%2E+Densely+connected+attention+propagation+for+reading+comprehension%2E+In+NIPS%2E)
35. Wei Wang, Ming Yan, and Chen Wu. 2018. Multi-granularity hierarchical attention fusion networks for reading comprehension and question answering. In ACL. [find](https://scholar.google.com/scholar?q=Wei+Wang%2C+Ming+Yan%2C+and+Chen+Wu%2E+2018%2E+Multi%2Dgranularity+hierarchical+attention+fusion+networks+for+reading+comprehension+and+question+answering%2E+In+ACL%2E)
36. Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229–256. [find](https://scholar.google.com/scholar?q=Ronald+J+Williams%2E+1992%2E+Simple+statistical+gradient%2Dfollowing+algorithms+for+connectionist+reinforcement+learning%2E+Machine+learning%2C+8%283%2D4%29%3A229%E2%80%93256%2E)
37. Caiming Xiong, Victor Zhong, and Richard Socher. 2018. DCN+: Mixed objective and deep residual coattention for question answering. In ICLR. [find](https://scholar.google.com/scholar?q=Caiming+Xiong%2C+Victor+Zhong%2C+and+Richard+Socher%2E+2018%2E+DCN%2B%3A+Mixed+objective+and+deep+residual+coattention+for+question+answering%2E+In+ICLR%2E)
38. Xiaojun Xu, Chang Liu, and Dawn Song. 2018. SQL-Net: Generating structured queries from natural language without reinforcement learning. In ICLR. [find](https://scholar.google.com/scholar?q=Xiaojun+Xu%2C+Chang+Liu%2C+and+Dawn+Song%2E+2018%2E+SQL%2DNet%3A+Generating+structured+queries+from+natural+language+without+reinforcement+learning%2E+In+ICLR%2E)
39. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Peng+Qi%2C+Saizheng+Zhang%2C+Yoshua+Bengio%2C+William+W+Cohen%2C+Ruslan+Salakhutdinov%2C+and+Christopher+D+Manning%2E+2018%2E+HotpotQA%3A+A+dataset+for+diverse%2C+explainable+multi%2Dhop+question+answering%2E+) [our card](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/)
40. Adams Wei Yu, David Dohan, Quoc Le, Thang Luong, Rui Zhao, and Kai Chen. 2018a. Fast and accurate reading comprehension by combining self-attention and convolution. In ICLR. [find](https://scholar.google.com/scholar?q=Adams+Wei+Yu%2C+David+Dohan%2C+Quoc+Le%2C+Thang+Luong%2C+Rui+Zhao%2C+and+Kai+Chen%2E+2018a%2E+Fast+and+accurate+reading+comprehension+by+combining+self%2Dattention+and+convolution%2E+In+ICLR%2E)
41. Tao Yu, Zifan Li, Zilin Zhang, Rui Zhang, and Dragomir Radev. 2018b. TypeSQL: Knowledge-based type-aware neural text-to-sql generation. In NAACL. [find](https://scholar.google.com/scholar?q=Tao+Yu%2C+Zifan+Li%2C+Zilin+Zhang%2C+Rui+Zhang%2C+and+Dragomir+Radev%2E+2018b%2E+TypeSQL%3A+Knowledge%2Dbased+type%2Daware+neural+text%2Dto%2Dsql+generation%2E+In+NAACL%2E)
42. Luke S. Zettlemoyer and Michael Collins. 2005. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In UAI. [find](https://scholar.google.com/scholar?q=Luke+S%2E+Zettlemoyer+and+Michael+Collins%2E+2005%2E+Learning+to+map+sentences+to+logical+form%3A+Structured+classification+with+probabilistic+categorial+grammars%2E+In+UAI%2E)
43. Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2SQL: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103. [find](https://scholar.google.com/scholar?q=Victor+Zhong%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+2017%2E+Seq2SQL%3A+Generating+structured+queries+from+natural+language+using+reinforcement+learning%2E+arXiv+preprint+arXiv%3A1709%2E00103%2E)
44. Zhi-Hua Zhou, Yu-Yin Sun, and Yu-Feng Li. 2009. Multi-instance learning by treating instances as noni.i.d. samples. In ICML. [find](https://scholar.google.com/scholar?q=Zhi%2DHua+Zhou%2C+Yu%2DYin+Sun%2C+and+Yu%2DFeng+Li%2E+2009%2E+Multi%2Dinstance+learning+by+treating+instances+as+noni%2Ei%2Ed%2E+samples%2E+In+ICML%2E)