---
title: "Denoising Distantly Supervised Open-Domain Question Answering"
description: "Open-domain QA systems trained with distant supervision (DS-QA) find candidate paragraphs… 29 references, cited by 57 papers here."
authors: ["Yankai Lin", "Haozhe Ji", "Zhiyuan Liu", "Maosong Sun"]
year: 2018
venue: "Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/P18-1161/"
doi: "10.18653/v1/p18-1161"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lin-2018-denoising-distantly-supervised-open-domain/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lin-2018-denoising-distantly-supervised-open-domain/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/3/) / Denoising Distantly Supervised Open-Domain Question Answering

# Denoising Distantly Supervised Open-Domain Question Answering

**Authors:** [Yankai Lin](https://md.research.elamur.ai/free/en/authors/yankai-lin/), [Haozhe Ji](https://md.research.elamur.ai/free/en/authors/haozhe-ji/), [Zhiyuan Liu](https://md.research.elamur.ai/free/en/authors/zhiyuan-liu/) and [Maosong Sun](https://md.research.elamur.ai/free/en/authors/maosong-sun/) · **Year:** 2018 · **Venue:** [Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2018](https://md.research.elamur.ai/free/en/venues/acl/2018/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P18-1161/) · [PDF](https://aclanthology.org/P18-1161.pdf) · [DOI 10.18653/v1/p18-1161](https://doi.org/10.18653/v1/p18-1161) · [OpenAlex](https://openalex.org/W2799081691)

**Citations:** 142 citations (OpenAlex, 2026-10-14) · [57](https://md.research.elamur.ai/free/en/papers/lin-2018-denoising-distantly-supervised-open-domain/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Open-domain QA systems trained with distant supervision (DS-QA) find candidate paragraphs through retrieval and then run a reading-comprehension model over them. This setup has two weaknesses. Many retrieved paragraphs, and many answer-string matches inside them, are wrongly labeled. Earlier systems also usually read only the single top paragraph and discard useful evidence elsewhere. The authors address both problems with a coarse-to-fine model. First, a lightweight paragraph selector (MLP or single-layer BiLSTM encoders, a self-attentive question vector, max-pooling and softmax) assigns a relevance probability to every retrieved paragraph. Second, a multi-layer BiLSTM paragraph reader predicts answer spans within each paragraph. The answer probability is obtained by marginalizing over paragraphs, weighted by the selector's scores, and both parts are trained jointly. Experiments on Quasar-T, SearchQA and TriviaQA show consistent gains over baselines, including R^3. The selector is much better than plain IR at ranking useful paragraphs, and it lets the system read only about half of the paragraphs without losing accuracy. This makes DS-QA both more robust to noise and faster.

## Key points
- The model combines a fast paragraph selector that filters noise with a precise paragraph reader, then aggregates answers across all informative paragraphs instead of using only the best one. Code and data are released at github.com/thunlp/OpenQA.
- On paragraph selection, Hits@1 rises from 6.3 with IR to 27.7 with Our+FULL on Quasar-T, and from 13.7 to 58.9 on SearchQA.
- In the same comparison, Our+FULL reaches EM/F1 of 41.1/48.0 on Quasar-T and 58.8/64.5 on SearchQA. Joint training (Our+FULL) beats independent training (Our+INDEP).
- On both datasets, keeping the top half of paragraphs chosen by the selector causes no performance drop, whereas IR-based truncation degrades clearly.
- In a re-ranking upper-bound analysis on the dev sets, Our+FULL top-1 EM exceeds R^3 (42.2 vs 35.3 on Quasar-T; 58.8 vs 51.2 on SearchQA). Top-3/5 scores sit 10–20% above top-1, which indicates room for answer re-ranking.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 137 works · 83,262 citations
- [State Key Laboratory of Intelligent Technology and Systems](https://md.research.elamur.ai/free/en/labs/state-key-laboratory-of-intelligent-technology-and-systems/) — Research institute, Beijing, China · 21 works · 9,641 citations
- [Beijing National Research Center for Information Science and Technology](https://md.research.elamur.ai/free/en/labs/beijing-national-research-center-for-information-science-and-technology/) — Research institute, Beijing, China · 21 works · 5,843 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 523 works

🧭 Topics (1): [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 243 works

## Cite

Cite as: Yankai Lin, Haozhe Ji, Zhiyuan Liu and Maosong Sun. 2018. Denoising Distantly Supervised Open-Domain Question Answering. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{lin2018denoising,
  title = {{Denoising Distantly Supervised Open-Domain Question Answering}},
  author = {Yankai Lin, Haozhe Ji, Zhiyuan Liu and Maosong Sun},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1161},
  url = {https://doi.org/10.18653/v1/p18-1161}
}
```

- [← previous End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classification](https://md.research.elamur.ai/free/en/papers/libovicky-2018-end-end-non-autoregressive-neural/)
- [next → Neural Metaphor Detection in Context](https://md.research.elamur.ai/free/en/papers/gao-2018-neural-metaphor-detection-context/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/3/)

## References

29 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Table 6 shows two examples of our models, which illustrates that our model can make full use of informative paragraphs. From the table we find that: (1) For the question “Who directed the 1946 ‘It’s A Wonderful Life’?”, our model extracts the answer “Frank Capra” from both top-2 paragraphs ranked by… [find](https://scholar.google.com/scholar?q=Table+6+shows+two+examples+of+our+models%2C+which+illustrates+that+our+model+can+make+full+use+of+informative+paragraphs%2E+From+the+table+we+find+that%3A+%281%29+For+the+question+%E2%80%9CWho+directed+the+1946+%E2%80%98It%E2%80%99s+A)
2. Conclusion and Future Work Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013a. [Semantic parsing on freebase from question-answer pairs](https://md.research.elamur.ai/free/en/papers/berant-2013-semantic-parsing-freebase-question-answer/). In Proceedings of EMNLP. pages 1533–1544.
3. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013b. [Semantic parsing on Freebase from question-answer pairs](https://md.research.elamur.ai/free/en/papers/berant-2013-semantic-parsing-freebase-question-answer/). In Proceedings of EMNLP. pages 1533–1544.
4. Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. 2015. Large-scale simple question answering with memory networks. arXiv preprint arXiv:1506.02075 . [find](https://scholar.google.com/scholar?q=Antoine+Bordes%2C+Nicolas+Usunier%2C+Sumit+Chopra%2C+and+Jason+Weston%2E+2015%2E+Large%2Dscale+simple+question+answering+with+memory+networks%2E+arXiv+preprint+arXiv%3A1506%2E02075+%2E)
5. Christian Buck, Jannis Bulian, Massimiliano Ciaramita, Andrea Gesmundo, Neil Houlsby, Wojciech Gajewski, and Wei Wang. 2017. Ask the right questions: Active question reformulation with reinforcement learning. arXiv preprint arXiv:1705.07830 . [find](https://scholar.google.com/scholar?q=Christian+Buck%2C+Jannis+Bulian%2C+Massimiliano+Ciaramita%2C+Andrea+Gesmundo%2C+Neil+Houlsby%2C+Wojciech+Gajewski%2C+and+Wei+Wang%2E+2017%2E+Ask+the+right+questions%3A+Active+question+reformulation+with+reinforcement+l)
6. In this paper, we propose a denoising distantly supervised open-domain question answering system which contains a paragraph selector to skim over paragraphs and a paragraph reader to perform an intensive reading on the selected paragraphs. Our model can make full use of all informative paragraphs an… [find](https://scholar.google.com/scholar?q=In+this+paper%2C+we+propose+a+denoising+distantly+supervised+open%2Ddomain+question+answering+system+which+contains+a+paragraph+selector+to+skim+over+paragraphs+and+a+paragraph+reader+to+perform+an+intens)
7. Danqi Chen, Jason Bolton, and Christopher D. Manning. 2016. [A thorough examination of the cnn/daily mail reading comprehension task](https://md.research.elamur.ai/free/en/papers/chen-2016-thorough-examination-cnn-daily-mail/). In Proceedings of ACL. pages 2358–2367. \#\# Acknowledgments Bhuwan Dhingra, Kathryn Mazaitis, and William W Cohen. 2017b. Quasar: Datasets for question answering by sea…
8. This work is supported by the National Natural Science Foundation of China (NSFC No. 61572273, 61661146007 and 61572273). This paper is also partially funded by Microsoft Research Asia FY17-RES-THEME-017. [find](https://scholar.google.com/scholar?q=This+work+is+supported+by+the+National+Natural+Science+Foundation+of+China+%28NSFC+No%2E+61572273%2C+61661146007+and+61572273%29%2E+This+paper+is+also+partially+funded+by+Microsoft+Research+Asia+FY17%2DRES%2DTHEME%2D)
9. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. [Reading wikipedia to answer open-domain questions](https://md.research.elamur.ai/free/en/papers/chen-2017-reading-wikipedia-answer-open-domain/). In Proceedings of the ACL. pages 1870–1879.
10. Tongfei Chen and Benjamin Van Durme. 2017. Discriminative information retrieval for question answering sentence selection. In Proceedings of EACL. pages 719–725. [find](https://scholar.google.com/scholar?q=Tongfei+Chen+and+Benjamin+Van+Durme%2E+2017%2E+Discriminative+information+retrieval+for+question+answering+sentence+selection%2E+In+Proceedings+of+EACL%2E+pages+719%E2%80%93725%2E)
11. Jianpeng Cheng and Mirella Lapata. 2016. [Neural summarization by extracting sentences and words](https://md.research.elamur.ai/free/en/papers/cheng-2016-neural-summarization-extracting-sentences-words/). In Proceedings of ACL. pages 484–494.
12. Eunsol Choi, Daniel Hewlett, Jakob Uszkoreit, Illia Polosukhin, Alexandre Lacoste, and Jonathan Berant. 2017. Coarse-to-fine question answering for long documents. In Proceedings of ACL. pages 209– 220. [find](https://scholar.google.com/scholar?q=Eunsol+Choi%2C+Daniel+Hewlett%2C+Jakob+Uszkoreit%2C+Illia+Polosukhin%2C+Alexandre+Lacoste%2C+and+Jonathan+Berant%2E+2017%2E+Coarse%2Dto%2Dfine+question+answering+for+long+documents%2E+In+Proceedings+of+ACL%2E+pages+209%E2%80%93+22)
13. Yiming Cui, Zhipeng Chen, Si Wei, Shijin Wang, Ting Liu, and Guoping Hu. 2017. Attention-over-attention neural networks for reading comprehension. In Proceedings of ACL. pages 593–602. [find](https://scholar.google.com/scholar?q=Yiming+Cui%2C+Zhipeng+Chen%2C+Si+Wei%2C+Shijin+Wang%2C+Ting+Liu%2C+and+Guoping+Hu%2E+2017%2E+Attention%2Dover%2Dattention+neural+networks+for+reading+comprehension%2E+In+Proceedings+of+ACL%2E+pages+593%E2%80%93602%2E)
14. Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. 2017a. Gated-attention readers for text comprehension. In Proceedings of ACL. pages 1832–1846. [find](https://scholar.google.com/scholar?q=Bhuwan+Dhingra%2C+Hanxiao+Liu%2C+Zhilin+Yang%2C+William+Cohen%2C+and+Ruslan+Salakhutdinov%2E+2017a%2E+Gated%2Dattention+readers+for+text+comprehension%2E+In+Proceedings+of+ACL%2E+pages+1832%E2%80%931846%2E)
15. Matthew Dunn, Levent Sagun, Mike Higgins, Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017. [Searchqa: A new q&a dataset augmented with context from a search engine](https://md.research.elamur.ai/free/en/papers/dunn-2017-searchqa-new-q-dataset-augmented/). arXiv preprint arXiv:1704.05179 .
16. Anthony Fader, Luke Zettlemoyer, and Oren Etzioni. 2014. Open question answering over curated and extracted knowledge bases. In Proceedings of SIGKDD. pages 1156–1165. [find](https://scholar.google.com/scholar?q=Anthony+Fader%2C+Luke+Zettlemoyer%2C+and+Oren+Etzioni%2E+2014%2E+Open+question+answering+over+curated+and+extracted+knowledge+bases%2E+In+Proceedings+of+SIGKDD%2E+pages+1156%E2%80%931165%2E)
17. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. [Hierarchical attention networks for document classification](https://md.research.elamur.ai/free/en/papers/yang-2016-hierarchical-attention-networks-document-classification/). In Proceedings of NAACL. pages 1480–1489.
18. Bert F Green Jr, Alice K Wolf, Carol Chomsky, and Kenneth Laughery. 1961. Baseball: an automatic question-answerer. In Proceedings of IRE-AIEE-ACM. pages 219–224. [find](https://scholar.google.com/scholar?q=Bert+F+Green+Jr%2C+Alice+K+Wolf%2C+Carol+Chomsky%2C+and+Kenneth+Laughery%2E+1961%2E+Baseball%3A+an+automatic+question%2Danswerer%2E+In+Proceedings+of+IRE%2DAIEE%2DACM%2E+pages+219%E2%80%93224%2E)
19. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. [Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). In Proceedings of ACL. pages 1601–1611.
20. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Diederik+P+Kingma+and+Jimmy+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+Proceedings+of+ICLR%2E)
21. Cody Kwok, Oren Etzioni, and Daniel S Weld. 2001. Scaling question answering to the web. TOIS pages 242–262. [find](https://scholar.google.com/scholar?q=Cody+Kwok%2C+Oren+Etzioni%2C+and+Daniel+S+Weld%2E+2001%2E+Scaling+question+answering+to+the+web%2E+TOIS+pages+242%E2%80%93262%2E)
22. Yankai Lin, Shiqi Shen, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2016. Neural relation extraction with selective attention over instances. In Proceedings of ACL. pages 2124–2133. [find](https://scholar.google.com/scholar?q=Yankai+Lin%2C+Shiqi+Shen%2C+Zhiyuan+Liu%2C+Huanbo+Luan%2C+and+Maosong+Sun%2E+2016%2E+Neural+relation+extraction+with+selective+attention+over+instances%2E+In+Proceedings+of+ACL%2E+pages+2124%E2%80%932133%2E)
23. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. [Glove: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In Proceedings of EMNLP. pages 1532–1543.
24. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. [Bidirectional attention flow for machine comprehension](https://md.research.elamur.ai/free/en/papers/seo-2016-bidirectional-attention-flow-machine-comprehension/). In Proceedings of ICLR.
25. Yelong Shen, Po-Sen Huang, Jianfeng Gao, and Weizhu Chen. 2017. Reasonet: Learning to stop reading in machine comprehension. In Proceedings of SIGKDD. ACM, pages 1047–1055. [find](https://scholar.google.com/scholar?q=Yelong+Shen%2C+Po%2DSen+Huang%2C+Jianfeng+Gao%2C+and+Weizhu+Chen%2E+2017%2E+Reasonet%3A+Learning+to+stop+reading+in+machine+comprehension%2E+In+Proceedings+of+SIGKDD%2E+ACM%2C+pages+1047%E2%80%931055%2E)
26. Ellen M Voorhees et al. 1999. [The trec-8 question answering track](https://md.research.elamur.ai/free/en/papers/voorhees-2000-trec-8-question-answering-track/) report. In Proceedings of TREC. pages 77–82.
27. Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerald Tesauro, Bowen Zhou, and Jing Jiang. 2018a. R3: Reinforced ranker-reader for open-domain question answering. In Proceedings of AAAI. [find](https://scholar.google.com/scholar?q=Shuohang+Wang%2C+Mo+Yu%2C+Xiaoxiao+Guo%2C+Zhiguo+Wang%2C+Tim+Klinger%2C+Wei+Zhang%2C+Shiyu+Chang%2C+Gerald+Tesauro%2C+Bowen+Zhou%2C+and+Jing+Jiang%2E+2018a%2E+R3%3A+Reinforced+ranker%2Dreader+for+open%2Ddomain+question+answering)
28. Shuohang Wang, Mo Yu, Jing Jiang, Wei Zhang, Xiaoxiao Guo, Shiyu Chang, Zhiguo Wang, Tim Klinger, Gerald Tesauro, and Murray Campbell. 2018b. Evidence aggregation for answer re-ranking in open-domain question answering. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Shuohang+Wang%2C+Mo+Yu%2C+Jing+Jiang%2C+Wei+Zhang%2C+Xiaoxiao+Guo%2C+Shiyu+Chang%2C+Zhiguo+Wang%2C+Tim+Klinger%2C+Gerald+Tesauro%2C+and+Murray+Campbell%2E+2018b%2E+Evidence+aggregation+for+answer+re%2Dranking+in+open%2Ddomain+)
29. Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. 2017. [Gated self-matching networks for reading comprehension and question answering](https://md.research.elamur.ai/free/en/papers/wang-2017-gated-self-matching-networks-reading/). In Proceedings of ACL. pages 189–198.

*Updated 24 October 2026*