---
title: "Robust Distant Supervision Relation Extraction via Deep"
description: "Distant supervision labels training sentences automatically by aligning a knowledge base (Freebase)… 23 references, cited by 61 papers here."
authors: ["Pengda Qin", "Weiran Xu", "William Yang Wang"]
year: 2018
venue: "Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/P18-1199/"
doi: "10.18653/v1/p18-1199"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/qin-2018-robust-distant-supervision-relation-extraction/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/qin-2018-robust-distant-supervision-relation-extraction/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/2/) / Robust Distant Supervision Relation Extraction via Deep Reinforcement Learning

# Robust Distant Supervision Relation Extraction via Deep Reinforcement Learning

**Authors:** Pengda Qin, [Weiran Xu](https://md.research.elamur.ai/free/en/authors/weiran-xu/) and [William Yang Wang](https://md.research.elamur.ai/free/en/authors/william-yang-wang/) · **Year:** 2018 · **Venue:** [Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2018](https://md.research.elamur.ai/free/en/venues/acl/2018/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P18-1199/) · [PDF](https://aclanthology.org/P18-1199.pdf) · [DOI 10.18653/v1/p18-1199](https://doi.org/10.18653/v1/p18-1199) · [OpenAlex](https://openalex.org/W2962939608)

**Citations:** 236 citations (OpenAlex, 2026-10-14) · [61](https://md.research.elamur.ai/free/en/papers/qin-2018-robust-distant-supervision-relation-extraction/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Distant supervision labels training sentences automatically by aligning a knowledge base (Freebase) with raw text. This produces many wrongly labeled positives. Earlier neural extractors either keep one sentence per entity pair or soft-weight sentences with attention. Neither approach helps when every sentence for an entity pair is wrong.

The authors make a hard decision instead. They train one policy-gradient reinforcement learning agent per relation type. For each distantly labeled positive sentence, the agent chooses to remove it or retain it. Removed sentences are not discarded but moved into the negative set. The reward is the change in F1 of a relation classifier retrained on the cleaned data, measured on a held-out validation split.

The method does not depend on the choice of classifier. Applied to the NYT–Freebase (Riedel) dataset, it improved PR curves and AUC for both the extractor from Zeng et al. (2015) and the one from Lin et al. (2016). It works as a plug-in denoising step for any distant supervision pipeline.

## Key points
- **State, actions and reward:**
  - The state joins the current sentence vector (word and position embeddings), weighted more heavily, with the average vector of previously removed sentences, which approximates an MDP.
  - The actions are {remove, retain}.
  - The reward is α·(F1ᵢ − F1ᵢ₋₁) between epochs.
- **Training setup:**
  - Agents are pre-trained and then retrained.
  - A policy network was chosen over DQN because the reward only becomes available after a full pass over the training data.
  - Per-relation agents outperformed a single shared agent.
- **Evaluation:** Experiments used the New York Times (Riedel et al., 2010) dataset. Gains for both the PCNN-style model (Zeng et al., 2015) and the attention model (Lin et al., 2016) were statistically significant (t-test, p \< 0.05).
- **Where false positives occur:** The detected false positives cluster in entity pairs with few sentences. Pairs with 1–5 sentences account for 55.9%, 32.0%, 3.7%, 4.4% and 0.7% of them, respectively. The authors attribute this to imperfect alignment between Freebase and NYT.
- **Example:** The agent flagged (Sami Moubayed, Syria) as a false positive for /people/person/place_of_birth.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of California, Santa Barbara](https://md.research.elamur.ai/free/en/universities/university-of-california-santa-barbara/) — University, Santa Barbara, United States · 35 works · 9,287 citations
- [Beijing University of Posts and Telecommunications](https://md.research.elamur.ai/free/en/universities/beijing-university-of-posts-and-telecommunications/) — University, Beijing, China · 22 works · 32,364 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (1): [🔎 Information extraction](https://md.research.elamur.ai/free/en/topics/information-extraction/) · 109 works

## Cite

Cite as: Pengda Qin, Weiran Xu and William Yang Wang. 2018. Robust Distant Supervision Relation Extraction via Deep Reinforcement Learning. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{qin2018robust,
  title = {{Robust Distant Supervision Relation Extraction via Deep Reinforcement Learning}},
  author = {Pengda Qin, Weiran Xu and William Yang Wang},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1199},
  url = {https://doi.org/10.18653/v1/p18-1199}
}
```

- [← previous Paraphrase Generation with Deep Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/li-2018-paraphrase-generation-deep-reinforcement-learning/)
- [next → A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature](https://md.research.elamur.ai/free/en/papers/nye-2018-corpus-multi-level-annotations-patients/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/2/)

## References

23 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. 2017. A brief survey of deep reinforcement learning. arXiv preprint arXiv:1708.05866. — [arXiv:1708.05866](https://arxiv.org/abs/1708.05866) [find](https://scholar.google.com/scholar?q=Kai+Arulkumaran%2C+Marc+Peter+Deisenroth%2C+Miles+Brundage%2C+and+Anil+Anthony+Bharath%2E+2017%2E+A+brief+survey+of+deep+reinforcement+learning%2E+arXiv+preprint+arXiv%3A1708%2E05866%2E+%E2%80%94+%5BarXiv%3A1708%2E05866%5D%28https%3A%2F%2Farx)
2. Razvan Bunescu and Raymond J Mooney. 2005. Subsequence kernels for relation extraction. In NIPS, pages 171–178. [find](https://scholar.google.com/scholar?q=Razvan+Bunescu+and+Raymond+J+Mooney%2E+2005%2E+Subsequence+kernels+for+relation+extraction%2E+In+NIPS%2C+pages+171%E2%80%93178%2E)
3. Meng Fang, Yuan Li, and Trevor Cohn. 2017. Learning how to active learn: A deep reinforcement learning approach. arXiv preprint arXiv:1708.02383. [find](https://scholar.google.com/scholar?q=Meng+Fang%2C+Yuan+Li%2C+and+Trevor+Cohn%2E+2017%2E+Learning+how+to+active+learn%3A+A+deep+reinforcement+learning+approach%2E+arXiv+preprint+arXiv%3A1708%2E02383%2E)
4. Jun Feng, Minlie Huang, Li Zhao, Yang Yang, and Xiaoyan Zhu. 2018. Reinforcement learning for relation classification from noisy data. [find](https://scholar.google.com/scholar?q=Jun+Feng%2C+Minlie+Huang%2C+Li+Zhao%2C+Yang+Yang%2C+and+Xiaoyan+Zhu%2E+2018%2E+Reinforcement+learning+for+relation+classification+from+noisy+data%2E)
5. Jenny Rose Finkel, Trond Grenager, and Christopher Manning. 2005. [Incorporating non-local information into information extraction systems by gibbs sampling](https://md.research.elamur.ai/free/en/papers/finkel-2005-incorporating-non-local-information-into/). In Proceedings of the 43rd annual meeting on association for computational linguistics, pages 363–370. Association for Computational Linguistic…
6. Zhou GuoDong, Su Jian, Zhang Jie, and Zhang Min. 2005. [Exploring various knowledge in relation extraction](https://md.research.elamur.ai/free/en/papers/zhou-2005-exploring-various-knowledge-relation-extraction/). In Proceedings of the 43rd annual meeting on association for computational linguistics, pages 427–434. Association for Computational Linguistics.
7. Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2009. [Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals](https://md.research.elamur.ai/free/en/papers/hendrickx-2010-semeval-2010-task-8-multi/). In Proceedings of the Workshop on Seman…
8. Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel S Weld. 2011. Knowledge-based weak supervision for information extraction of overlapping relations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1… [find](https://scholar.google.com/scholar?q=Raphael+Hoffmann%2C+Congle+Zhang%2C+Xiao+Ling%2C+Luke+Zettlemoyer%2C+and+Daniel+S+Weld%2E+2011%2E+Knowledge%2Dbased+weak+supervision+for+information+extraction+of+overlapping+relations%2E+In+Proceedings+of+the+49th+A)
9. Guoliang Ji, Kang Liu, Shizhu He, Jun Zhao, et al. 2017. Distant supervision for relation extraction with sentence-level attention and entity descriptions. In AAAI, pages 3060–3066. [find](https://scholar.google.com/scholar?q=Guoliang+Ji%2C+Kang+Liu%2C+Shizhu+He%2C+Jun+Zhao%2C+et+al%2E+2017%2E+Distant+supervision+for+relation+extraction+with+sentence%2Dlevel+attention+and+entity+descriptions%2E+In+AAAI%2C+pages+3060%E2%80%933066%2E)
10. ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2-Volume 2, pages 1003–1011. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=ACL+and+the+4th+International+Joint+Conference+on+Natural+Language+Processing+of+the+AFNLP%3A+Volume+2%2DVolume+2%2C+pages+1003%E2%80%931011%2E+Association+for+Computational+Linguistics%2E)
11. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602. [find](https://scholar.google.com/scholar?q=Volodymyr+Mnih%2C+Koray+Kavukcuoglu%2C+David+Silver%2C+Alex+Graves%2C+Ioannis+Antonoglou%2C+Daan+Wierstra%2C+and+Martin+Riedmiller%2E+2013%2E+Playing+atari+with+deep+reinforcement+learning%2E+arXiv+preprint+arXiv%3A1312%2E)
12. Thien Huu Nguyen and Ralph Grishman. 2015. [Event detection and domain adaptation with convolutional neural networks](https://md.research.elamur.ai/free/en/papers/nguyen-2015-event-detection-domain-adaptation-convolutional/). In ACL (2), pages 365–371.
13. Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations and their mentions without labeled text. In Machine Learning and Knowledge Discovery in Databases, pages 148–163. Springer. [find](https://scholar.google.com/scholar?q=Sebastian+Riedel%2C+Limin+Yao%2C+and+Andrew+McCallum%2E+2010%2E+Modeling+relations+and+their+mentions+without+labeled+text%2E+In+Machine+Learning+and+Knowledge+Discovery+in+Databases%2C+pages+148%E2%80%93163%2E+Springer%2E)
14. Benjamin Roth, Tassilo Barth, Michael Wiegand, and Dietrich Klakow. 2013. A survey of noise reduction methods for distant supervision. In Proceedings of the 2013 workshop on Automated knowledge base construction, pages 73–78. ACM. [find](https://scholar.google.com/scholar?q=Benjamin+Roth%2C+Tassilo+Barth%2C+Michael+Wiegand%2C+and+Dietrich+Klakow%2E+2013%2E+A+survey+of+noise+reduction+methods+for+distant+supervision%2E+In+Proceedings+of+the+2013+workshop+on+Automated+knowledge+base+c)
15. Cicero Nogueira dos Santos, Bing Xiang, and Bowen Zhou. 2015. [Classifying relations by ranking with convolutional neural networks](https://md.research.elamur.ai/free/en/papers/santos-2015-classifying-relations-ranking-convolutional-neural/). arXiv preprint arXiv:1504.06580.
16. Yatian Shen and Xuanjing Huang. 2016. Attention-based convolutional neural network for semantic relation extraction. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. [find](https://scholar.google.com/scholar?q=Yatian+Shen+and+Xuanjing+Huang%2E+2016%2E+Attention%2Dbased+convolutional+neural+network+for+semantic+relation+extraction%2E+In+Proceedings+of+COLING+2016%2C+the+26th+International+Conference+on+Computational+L)
17. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. 2016. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484–489. [find](https://scholar.google.com/scholar?q=David+Silver%2C+Aja+Huang%2C+Chris+J+Maddison%2C+Arthur+Guez%2C+Laurent+Sifre%2C+George+Van+Den+Driessche%2C+Julian+Schrittwieser%2C+Ioannis+Antonoglou%2C+Veda+Panneershelvam%2C+Marc+Lanctot%2C+et+al%2E+2016%2E+Mastering+the)
18. Mihai Surdeanu, Julie Tibshirani, Ramesh Nallapati, and Christopher D Manning. 2012. Multi-instance multi-label learning for relation extraction. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 455– 46… [find](https://scholar.google.com/scholar?q=Mihai+Surdeanu%2C+Julie+Tibshirani%2C+Ramesh+Nallapati%2C+and+Christopher+D+Manning%2E+2012%2E+Multi%2Dinstance+multi%2Dlabel+learning+for+relation+extraction%2E+In+Proceedings+of+the+2012+Joint+Conference+on+Empiric)
19. Yankai Lin, Shiqi Shen, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2016. Neural relation extraction with selective attention over instances. In ACL (1). [find](https://scholar.google.com/scholar?q=Yankai+Lin%2C+Shiqi+Shen%2C+Zhiyuan+Liu%2C+Huanbo+Luan%2C+and+Maosong+Sun%2E+2016%2E+Neural+relation+extraction+with+selective+attention+over+instances%2E+In+ACL+%281%29%2E)
20. Kun Xu, Yansong Feng, Songfang Huang, and Dongyan Zhao. 2015. Semantic relation classification via convolutional neural networks with simple negative sampling. arXiv preprint arXiv:1506.07650. [find](https://scholar.google.com/scholar?q=Kun+Xu%2C+Yansong+Feng%2C+Songfang+Huang%2C+and+Dongyan+Zhao%2E+2015%2E+Semantic+relation+classification+via+convolutional+neural+networks+with+simple+negative+sampling%2E+arXiv+preprint+arXiv%3A1506%2E07650%2E)
21. Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the Dmitry Zelenko, Chinatsu Aone, and Anthony Richardella. 2003. Kernel methods for relation extraction… [find](https://scholar.google.com/scholar?q=Mike+Mintz%2C+Steven+Bills%2C+Rion+Snow%2C+and+Dan+Jurafsky%2E+2009%2E+Distant+supervision+for+relation+extraction+without+labeled+data%2E+In+Proceedings+of+the+Joint+Conference+of+the+47th+Annual+Meeting+of+the+)
22. Daojian Zeng, Kang Liu, Yubo Chen, and Jun Zhao. 2015. Distant supervision for relation extraction via piecewise convolutional neural networks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), Lisbon, Portugal, pages 17–21. [find](https://scholar.google.com/scholar?q=Daojian+Zeng%2C+Kang+Liu%2C+Yubo+Chen%2C+and+Jun+Zhao%2E+2015%2E+Distant+supervision+for+relation+extraction+via+piecewise+convolutional+neural+networks%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Metho)
23. Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, Jun Zhao, et al. 2014. [Relation classification via convolutional deep neural network](https://md.research.elamur.ai/free/en/papers/zeng-2014-relation-classification-convolutional-deep-neural/). In COLING, pages 2335–2344.

*Updated 24 October 2026*