---
title: "Revisiting DocRED - Addressing the False Negative Problem in"
description: "The authors show that this process left many true relations unlabeled, so false negatives are… 49 references, cited by 57 papers here."
authors: ["Qingyu Tan", "Lu Xu", "Lidong Bing", "Hwee Tou Ng", "Sharifah Mahani Aljunied"]
year: 2022
venue: "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2022.emnlp-main.580/"
doi: "10.18653/v1/2022.emnlp-main.580"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/tan-2022-revisiting-docred-addressing-false-negative/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tan-2022-revisiting-docred-addressing-false-negative/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/) / Revisiting DocRED - Addressing the False Negative Problem in Relation Extraction

# Revisiting DocRED - Addressing the False Negative Problem in Relation Extraction

**Authors:** [Qingyu Tan](https://md.research.elamur.ai/free/en/authors/qingyu-tan/), [Lu Xu](https://md.research.elamur.ai/free/en/authors/lu-xu/), [Lidong Bing](https://md.research.elamur.ai/free/en/authors/lidong-bing/), [Hwee Tou Ng](https://md.research.elamur.ai/free/en/authors/hwee-tou-ng/) and Sharifah Mahani Aljunied · **Year:** 2022 · **Venue:** [Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2022](https://md.research.elamur.ai/free/en/venues/emnlp/2022/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.emnlp-main.580/) · [PDF](https://aclanthology.org/2022.emnlp-main.580.pdf) · [DOI 10.18653/v1/2022.emnlp-main.580](https://doi.org/10.18653/v1/2022.emnlp-main.580) · [OpenAlex](https://openalex.org/W4385573505)

**Citations:** 94 citations (OpenAlex, 2026-10-14) · [57](https://md.research.elamur.ai/free/en/papers/tan-2022-revisiting-docred-addressing-false-negative/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
DocRED is a standard benchmark for document-level relation extraction. It was built by having annotators filter candidate triples suggested by Wikidata distant supervision and an early RE model. The authors show that this process left many true relations unlabeled, so false negatives are widespread. They trace the gaps to three causes: Wikidata's sparse coverage, the limited quality of the original recommender model, and missing inverse relations such as symmetric sibling links. Because the development and test sets share these gaps, existing evaluations are unreliable.

To address this, they re-annotated 4,053 DocRED documents in an iterative human-in-the-loop process. Several current document-level RE models proposed missing triples, and human annotators verified them. The resulting dataset, Re-DocRED, contains more than twice as many relation triples as the original. State-of-the-art models trained and tested on Re-DocRED score roughly 13 F1 points higher. This suggests that much of the apparent performance ceiling on DocRED came from incomplete labels rather than model weakness.

## Key points
- Re-DocRED raises the triple count for the covered documents from 50,503 to 120,664, and the share of no_relation entity pairs drops from 97.1% to 94.0%. Coreference errors were also corrected.
- The evaluation data went through two rounds of revision, while the training data received only one. The authors acknowledge that some relations may still be missing.
- Compared with the contemporaneous from-scratch annotation by Huang et al. (2022), which covers 96 documents, Re-DocRED is much larger, more precise, has more triples per document, and scales more easily to new relation types.
- Original DocRED has 5,053 Wikipedia documents, 96 relation types plus no_relation, and about 393.6 entity pairs per document. Re-annotation of sentence-level datasets (TACRED → Re-TACRED, Wiki20 → Wiki20m) shows a similar pattern of hidden false negatives.
- The dataset is public at github.com/tonytan48/Re-DocRED. Annotators were paid about 60 CNY per hour.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [National University of Singapore](https://md.research.elamur.ai/free/en/universities/national-university-of-singapore/) — University, Singapore, Singapore · 84 works · 32,204 citations
- [Alibaba Group](https://md.research.elamur.ai/free/en/companies/alibaba-group/) — Company, Hangzhou, China · 64 works · 14,113 citations
- [Singapore University of Technology and Design](https://md.research.elamur.ai/free/en/universities/singapore-university-of-technology-and-design/) — University, Singapore, Singapore · 33 works · 10,760 citations
- [Alibaba DAMO Academy](https://md.research.elamur.ai/free/en/companies/alibaba-damo-academy/) — Company, Hangzhou, China · 28 works · 6,115 citations
- [Alibaba Group (Cayman Islands)](https://md.research.elamur.ai/free/en/companies/alibaba-group-cayman-islands/) — Company, George Town, Cayman Islands · 15 works · 3,209 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (1): [🔎 Information extraction](https://md.research.elamur.ai/free/en/topics/information-extraction/) · 109 works

## Cite

Cite as: Qingyu Tan, Lu Xu, Lidong Bing, Hwee Tou Ng and Sharifah Mahani Aljunied. 2022. Revisiting DocRED - Addressing the False Negative Problem in Relation Extraction. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{tan2022revisiting,
  title = {{Revisiting DocRED - Addressing the False Negative Problem in Relation Extraction}},
  author = {Qingyu Tan, Lu Xu, Lidong Bing, Hwee Tou Ng and Sharifah Mahani Aljunied},
  booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.580},
  url = {https://doi.org/10.18653/v1/2022.emnlp-main.580}
}
```

- [← previous Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Text](https://md.research.elamur.ai/free/en/papers/dou-2022-is-gpt-3-text-indistinguishable/)
- [next → MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognition](https://md.research.elamur.ai/free/en/papers/adelani-2022-masakhaner-2-0-africa-centric/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/)

## References

49 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Christoph Alt, Aleksandra Gabryszak, and Leonhard Hennig. 2020. [TACRED revisited: A thorough evaluation of the TACRED relation extraction task](https://md.research.elamur.ai/free/en/papers/alt-2020-tacred-revisited-thorough-evaluation-tacred/). In Proceedings of ACL.
2. Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling. 2020. Do not have enough data? deep learning to the rescue! In Proceedings of AAAI. [find](https://scholar.google.com/scholar?q=Ateret+Anaby%2DTavor%2C+Boaz+Carmeli%2C+Esther+Goldbraich%2C+Amir+Kantor%2C+George+Kour%2C+Segev+Shlomov%2C+Naama+Tepper%2C+and+Naama+Zwerdling%2E+2020%2E+Do+not+have+enough+data%3F+deep+learning+to+the+rescue%21+In+Proceedi)
3. Lidong Bing, Sneha Chaudhari, Richard Wang, and William Cohen. 2015. Improving distant supervision for information extraction using label propagation through lists. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Lidong+Bing%2C+Sneha+Chaudhari%2C+Richard+Wang%2C+and+William+Cohen%2E+2015%2E+Improving+distant+supervision+for+information+extraction+using+label+propagation+through+lists%2E+In+Proceedings+of+EMNLP%2E)
4. Lidong Bing, Wai Lam, and Tak-Lam Wong. 2013. Wikipedia entity expansion and attribute extraction from the web using semi-supervised learning. In Proceedings of WSDM. [find](https://scholar.google.com/scholar?q=Lidong+Bing%2C+Wai+Lam%2C+and+Tak%2DLam+Wong%2E+2013%2E+Wikipedia+entity+expansion+and+attribute+extraction+from+the+web+using+semi%2Dsupervised+learning%2E+In+Proceedings+of+WSDM%2E)
5. Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of ACM SIGMOD. [find](https://scholar.google.com/scholar?q=Kurt+Bollacker%2C+Colin+Evans%2C+Praveen+Paritosh%2C+Tim+Sturge%2C+and+Jamie+Taylor%2E+2008%2E+Freebase%3A+a+collaboratively+created+graph+database+for+structuring+human+knowledge%2E+In+Proceedings+of+ACM+SIGMOD%2E)
6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). In Proceedings of NIPS.
7. Christian Buck, Kenneth Heafield, and Bas Van Ooyen. 2014. [N-gram counts and language models from the common crawl](https://md.research.elamur.ai/free/en/papers/buck-2014-n-gram-counts-language-models/). In Proceedings of LREC.
8. Jhih-wei Chen, Tsu-Jui Fu, Chen-Kang Lee, and Wei-Yun Ma. 2021. H-FND: hierarchical false-negative denoising for distant supervision relation extraction. In Findings of ACL. [find](https://scholar.google.com/scholar?q=Jhih%2Dwei+Chen%2C+Tsu%2DJui+Fu%2C+Chen%2DKang+Lee%2C+and+Wei%2DYun+Ma%2E+2021%2E+H%2DFND%3A+hierarchical+false%2Dnegative+denoising+for+distant+supervision+relation+extraction%2E+In+Findings+of+ACL%2E)
9. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin. [find](https://scholar.google.com/scholar?q=Joseph+L+Fleiss%2E+1971%2E+Measuring+nominal+scale+agreement+among+many+raters%2E+Psychological+Bulletin%2E)
10. Tianyu Gao, Xu Han, Yuzhuo Bai, Keyue Qiu, Zhiyu Xie, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2021. Manual evaluation matters: Reviewing test protocols of distantly supervised relation extraction. In Findings of ACL. [find](https://scholar.google.com/scholar?q=Tianyu+Gao%2C+Xu+Han%2C+Yuzhuo+Bai%2C+Keyue+Qiu%2C+Zhiyu+Xie%2C+Yankai+Lin%2C+Zhiyuan+Liu%2C+Peng+Li%2C+Maosong+Sun%2C+and+Jie+Zhou%2E+2021%2E+Manual+evaluation+matters%3A+Reviewing+test+protocols+of+distantly+supervised+rel)
11. Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2019. FewRel 2.0: Towards more challenging few-shot relation classification. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Tianyu+Gao%2C+Xu+Han%2C+Hao+Zhu%2C+Zhiyuan+Liu%2C+Peng+Li%2C+Maosong+Sun%2C+and+Jie+Zhou%2E+2019%2E+FewRel+2%2E0%3A+Towards+more+challenging+few%2Dshot+relation+classification%2E+In+Proceedings+of+EMNLP%2E)
12. Xu Han, Tianyu Gao, Yankai Lin, Hao Peng, Yaoliang Yang, Chaojun Xiao, Zhiyuan Liu, Peng Li, Jie Zhou, and Maosong Sun. 2020. More data, more relations, more context and more openness: A review and outlook for relation extraction. In Proceedings of AACL. [find](https://scholar.google.com/scholar?q=Xu+Han%2C+Tianyu+Gao%2C+Yankai+Lin%2C+Hao+Peng%2C+Yaoliang+Yang%2C+Chaojun+Xiao%2C+Zhiyuan+Liu%2C+Peng+Li%2C+Jie+Zhou%2C+and+Maosong+Sun%2E+2020%2E+More+data%2C+more+relations%2C+more+context+and+more+openness%3A+A+review+and+ou)
13. Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018. Fewrel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Xu+Han%2C+Hao+Zhu%2C+Pengfei+Yu%2C+Ziyun+Wang%2C+Yuan+Yao%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2018%2E+Fewrel%3A+A+large%2Dscale+supervised+few%2Dshot+relation+classification+dataset+with+state%2Dof%2Dthe%2Dart+evaluation%2E+In+Pr)
14. Kailong Hao, Botao Yu, and Wei Hu. 2021. Knowing false negatives: An adversarial training method for distantly supervised relation extraction. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Kailong+Hao%2C+Botao+Yu%2C+and+Wei+Hu%2E+2021%2E+Knowing+false+negatives%3A+An+adversarial+training+method+for+distantly+supervised+relation+extraction%2E+In+Proceedings+of+EMNLP%2E)
15. Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2010. [Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals](https://md.research.elamur.ai/free/en/papers/hendrickx-2010-semeval-2010-task-8-multi/). In Proceedings of the 5th International Wo…
16. Qiao Cheng, Juntao Liu, Xiaoye Qu, Jin Zhao, Jiaqing Liang, Zhefeng Wang, Baoxing Huai, Nicholas Jing Yuan, and Yanghua Xiao. 2021. HacRED: A large-scale relation extraction dataset toward hard cases in practical applications. In Findings of ACL. [find](https://scholar.google.com/scholar?q=Qiao+Cheng%2C+Juntao+Liu%2C+Xiaoye+Qu%2C+Jin+Zhao%2C+Jiaqing+Liang%2C+Zhefeng+Wang%2C+Baoxing+Huai%2C+Nicholas+Jing+Yuan%2C+and+Yanghua+Xiao%2E+2021%2E+HacRED%3A+A+large%2Dscale+relation+extraction+dataset+toward+hard+cases+)
17. Quzhe Huang, Shibo Hao, Yuan Ye, Shengqi Zhu, Yansong Feng, and Dongyan Zhao. 2022. Does recommend-revise produce reliable annotations? an analysis on missing instances in DocRED. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Quzhe+Huang%2C+Shibo+Hao%2C+Yuan+Ye%2C+Shengqi+Zhu%2C+Yansong+Feng%2C+and+Dongyan+Zhao%2E+2022%2E+Does+recommend%2Drevise+produce+reliable+annotations%3F+an+analysis+on+missing+instances+in+DocRED%2E+In+Proceedings+of+AC)
18. Yew Ken Chia, Lidong Bing, Soujanya Poria, and Luo Si. 2022. [RelationPrompt: leveraging prompts to generate synthetic data for zero-shot relation triplet extraction](https://md.research.elamur.ai/free/en/papers/chia-2022-relationprompt-leveraging-prompts-generate-synthetic/). Findings of ACL.
19. Varun Kumar, Ashutosh Choudhary, and Eunah Cho. 2020. [Data augmentation using pre-trained transformer models](https://md.research.elamur.ai/free/en/papers/kumar-2020-data-augmentation-using-pre-trained/). In Proceedings of the 2nd Workshop on Life-long Learning for Spoken Language Systems.
20. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzm%C3%A1n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+cross%2Dlingu)
21. Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, et al. 2015. [Dbpedia – a large-scale, multilingual knowledge base extracted from wikipedia](https://md.research.elamur.ai/free/en/papers/lehmann-2015-dbpedia-large-scale-multilingual-knowledge/). Semantic web.
22. Yangming Li, Lemao Liu, and Shuming Shi. 2021. Empirical analysis of unlabeled entity problem in named entity recognition. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Yangming+Li%2C+Lemao+Liu%2C+and+Shuming+Shi%2E+2021%2E+Empirical+analysis+of+unlabeled+entity+problem+in+named+entity+recognition%2E+In+Proceedings+of+ICLR%2E)
23. Linlin Liu, Bosheng Ding, Lidong Bing, Shafiq Joty, Luo Si, and Chunyan Miao. 2021. MulDA: A multilingual data augmentation framework for low-resource cross-lingual NER. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Linlin+Liu%2C+Bosheng+Ding%2C+Lidong+Bing%2C+Shafiq+Joty%2C+Luo+Si%2C+and+Chunyan+Miao%2E+2021%2E+MulDA%3A+A+multilingual+data+augmentation+framework+for+low%2Dresource+cross%2Dlingual+NER%2E+In+Proceedings+of+ACL%2E)
24. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+a)
25. Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Mike+Mintz%2C+Steven+Bills%2C+Rion+Snow%2C+and+Dan+Jurafsky%2E+2009%2E+Distant+supervision+for+relation+extraction+without+labeled+data%2E+In+Proceedings+of+ACL%2E)
26. Guoshun Nan, Zhijiang Guo, Ivan Sekulic, and Wei Lu. 2020. [Reasoning with latent structure refinement for document-level relation extraction](https://md.research.elamur.ai/free/en/papers/nan-2020-reasoning-latent-structure-refinement-document/). In Proceedings of ACL.
27. Tapas Nayak and Hwee Tou Ng. 2019. Effective attention modeling for neural relation extraction. In Proceedings of CoNLL. [find](https://scholar.google.com/scholar?q=Tapas+Nayak+and+Hwee+Tou+Ng%2E+2019%2E+Effective+attention+modeling+for+neural+relation+extraction%2E+In+Proceedings+of+CoNLL%2E)
28. Tapas Nayak and Hwee Tou Ng. 2020. Effective modeling of encoder-decoder architecture for joint entity and relation extraction. In Proceedings of AAAI. [find](https://scholar.google.com/scholar?q=Tapas+Nayak+and+Hwee+Tou+Ng%2E+2020%2E+Effective+modeling+of+encoder%2Ddecoder+architecture+for+joint+entity+and+relation+extraction%2E+In+Proceedings+of+AAAI%2E)
29. Sebastian Riedel, Limin Yao, Andrew McCallum, and Benjamin M. Marlin. 2013. Relation extraction with matrix factorization and universal schemas. In Proceedings of NAACL. [find](https://scholar.google.com/scholar?q=Sebastian+Riedel%2C+Limin+Yao%2C+Andrew+McCallum%2C+and+Benjamin+M%2E+Marlin%2E+2013%2E+Relation+extraction+with+matrix+factorization+and+universal+schemas%2E+In+Proceedings+of+NAACL%2E)
30. Peter West, Chandra Bhagavatula, Jack Hessel, Jena D Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2021. [Symbolic knowledge distillation: from general language models to commonsense models](https://md.research.elamur.ai/free/en/papers/west-2022-symbolic-knowledge-distillation-general-language/). arXiv preprint arXiv:2110.07178.
31. Chaojun Xiao, Yuan Yao, Ruobing Xie, Xu Han, Zhiyuan Liu, Maosong Sun, Fen Lin, and Leyu Lin. 2020. Denoising relation extraction from document-level distant supervision. In Proceedings EMNLP. [find](https://scholar.google.com/scholar?q=Chaojun+Xiao%2C+Yuan+Yao%2C+Ruobing+Xie%2C+Xu+Han%2C+Zhiyuan+Liu%2C+Maosong+Sun%2C+Fen+Lin%2C+and+Leyu+Lin%2E+2020%2E+Denoising+relation+extraction+from+document%2Dlevel+distant+supervision%2E+In+Proceedings+EMNLP%2E)
32. Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. 2020. [Generative data augmentation for commonsense reasoning](https://md.research.elamur.ai/free/en/papers/yang-2020-generative-data-augmentation-commonsense-reasoning/). In Findings of EMNLP.
33. Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019. DocRED: a large-scale document-level relation extraction dataset. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Yuan+Yao%2C+Deming+Ye%2C+Peng+Li%2C+Xu+Han%2C+Yankai+Lin%2C+Zhenghao+Liu%2C+Zhiyuan+Liu%2C+Lixin+Huang%2C+Jie+Zhou%2C+and+Maosong+Sun%2E+2019%2E+DocRED%3A+a+large%2Dscale+document%2Dlevel+relation+extraction+dataset%2E+In+Proceedi)
34. Ningyu Zhang, Xiang Chen, Xin Xie, Shumin Deng, Chuanqi Tan, Mosha Chen, Fei Huang, Luo Si, and Huajun Chen. 2021. Document-level relation extraction as semantic segmentation. Proceedings of IJCAI. [find](https://scholar.google.com/scholar?q=Ningyu+Zhang%2C+Xiang+Chen%2C+Xin+Xie%2C+Shumin+Deng%2C+Chuanqi+Tan%2C+Mosha+Chen%2C+Fei+Huang%2C+Luo+Si%2C+and+Huajun+Chen%2E+2021%2E+Document%2Dlevel+relation+extraction+as+semantic+segmentation%2E+Proceedings+of+IJCAI%2E)
35. Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware attention and supervised data improve slot filling. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Yuhao+Zhang%2C+Victor+Zhong%2C+Danqi+Chen%2C+Gabor+Angeli%2C+and+Christopher+D%2E+Manning%2E+2017%2E+Position%2Daware+attention+and+supervised+data+improve+slot+filling%2E+In+Proceedings+of+EMNLP%2E)
36. Evan Sandhaus. 2008. The New York Times annotated corpus. Linguistic Data Consortium. [find](https://scholar.google.com/scholar?q=Evan+Sandhaus%2E+2008%2E+The+New+York+Times+annotated+corpus%2E+Linguistic+Data+Consortium%2E)
37. Ran Zhou, Xin Li, Ruidan He, Lidong Bing, Erik Cambria, Luo Si, and Chunyan Miao. 2022. MELM: Data augmentation with masked entity language modeling for low-resource NER. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Ran+Zhou%2C+Xin+Li%2C+Ruidan+He%2C+Lidong+Bing%2C+Erik+Cambria%2C+Luo+Si%2C+and+Chunyan+Miao%2E+2022%2E+MELM%3A+Data+augmentation+with+masked+entity+language+modeling+for+low%2Dresource+NER%2E+In+Proceedings+of+ACL%2E)
38. Timo Schick and Hinrich Schütze. 2021. [Generating datasets with pretrained language models](https://md.research.elamur.ai/free/en/papers/schick-2021-generating-datasets-pretrained-language-models/). In Proceedings of EMNLP.
39. Wenxuan Zhou and Muhao Chen. 2021. [An improved baseline for sentence-level relation extraction](https://md.research.elamur.ai/free/en/papers/zhou-2022-improved-baseline-sentence-level-relation/). arXiv preprint arXiv:2102.01373.
40. George Stoica, Emmanouil Antonios Platanios, and Barnabás Póczos. 2021. Re-tacred: addressing shortcomings of the tacred dataset. In Proceedings of AAAI. [find](https://scholar.google.com/scholar?q=George+Stoica%2C+Emmanouil+Antonios+Platanios%2C+and+Barnab%C3%A1s+P%C3%B3czos%2E+2021%2E+Re%2Dtacred%3A+addressing+shortcomings+of+the+tacred+dataset%2E+In+Proceedings+of+AAAI%2E)
41. Wenxuan Zhou, Kevin Huang, Tengyu Ma, and Jing Huang. 2021. Document-level relation extraction with adaptive thresholding and localized context pooling. In Proceedings of AAAI. [find](https://scholar.google.com/scholar?q=Wenxuan+Zhou%2C+Kevin+Huang%2C+Tengyu+Ma%2C+and+Jing+Huang%2E+2021%2E+Document%2Dlevel+relation+extraction+with+adaptive+thresholding+and+localized+context+pooling%2E+In+Proceedings+of+AAAI%2E)
42. Qingyu Tan, Ruidan He, Lidong Bing, and Hwee Tou Ng. 2022. [Document-level relation extraction with adaptive focal loss and knowledge distillation](https://md.research.elamur.ai/free/en/papers/tan-2022-document-level-relation-extraction-adaptive/). In Findings of ACL.
43. Denny Vrandečić and Markus Krötzsch. 2014. Wikidata: a free collaborative knowledgebase. Communications of the ACM. [find](https://scholar.google.com/scholar?q=Denny+Vrandec%CC%8Cic%CC%81+and+Markus+Kr%C3%B6tzsch%2E+2014%2E+Wikidata%3A+a+free+collaborative+knowledgebase%2E+Communications+of+the+ACM%2E)
44. Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006. [find](https://scholar.google.com/scholar?q=Christopher+Walker%2C+Stephanie+Strassel%2C+Julie+Medero%2C+and+Kazuaki+Maeda%2E+2006%2E)
45. ACE 2005 multilingual training corpus. Linguistic Data Consortium. [find](https://scholar.google.com/scholar?q=ACE+2005+multilingual+training+corpus%2E+Linguistic+Data+Consortium%2E)
46. Zihao Wang, Kwunping Lai, Piji Li, Lidong Bing, and Wai Lam. 2019. Tackling long-tailed relations and uncommon entities in knowledge graph completion. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Zihao+Wang%2C+Kwunping+Lai%2C+Piji+Li%2C+Lidong+Bing%2C+and+Wai+Lam%2E+2019%2E+Tackling+long%2Dtailed+relations+and+uncommon+entities+in+knowledge+graph+completion%2E+In+Proceedings+of+EMNLP%2E)
47. Example 1 Error Type 1: Misunderstanding of Definition: creator vs. architect Error Cause: Annotator’s Misunderstanding of Wikidata Relation Definition Error Analysis: There are over 9,000 relation types in the Wikidata knowledge base, whereas DocRED only contains 84 of them. The relation between bu… [find](https://scholar.google.com/scholar?q=Example+1+Error+Type+1%3A+Misunderstanding+of+Definition%3A+creator+vs%2E+architect+Error+Cause%3A+Annotator%E2%80%99s+Misunderstanding+of+Wikidata+Relation+Definition+Error+Analysis%3A+There+are+over+9%2C000+relation+ty)
48. Wrong Triples: (Museo Jumex, creator, David Alan Chipperfield), (River and Rowing Museum, creator, David Alan Chipperfield), (Museum of Modern Literature, creator, David Alan Chipperfield), (Saint Louis Art Museum,creator, David Alan Chipperfield), (Hepworth Wakefield gallery, creator, David Alan Ch… [find](https://scholar.google.com/scholar?q=Wrong+Triples%3A+%28Museo+Jumex%2C+creator%2C+David+Alan+Chipperfield%29%2C+%28River+and+Rowing+Museum%2C+creator%2C+David+Alan+Chipperfield%29%2C+%28Museum+of+Modern+Literature%2C+creator%2C+David+Alan+Chipperfield%29%2C+%28Saint+Lou)
49. Wrong Triples: (Taking Over Me, series, Chapman Square Chapter II), (Brokenhearted, series, Chapman Square Chapter II), (When She Was Mine, series, Chapman Square), (Standing in the Dark, series, Chapman Square) [find](https://scholar.google.com/scholar?q=Wrong+Triples%3A+%28Taking+Over+Me%2C+series%2C+Chapman+Square+Chapter+II%29%2C+%28Brokenhearted%2C+series%2C+Chapman+Square+Chapter+II%29%2C+%28When+She+Was+Mine%2C+series%2C+Chapman+Square%29%2C+%28Standing+in+the+Dark%2C+series%2C+Chap)

*Updated 24 October 2026*